ParquetDestination<TInput>
Class ParquetDestination<TInput>
Implements
Inherited Members
Namespace: ETLBox.Parquet
Assembly: ETLBox.Parquet.dll
Syntax
Type Parameters
Name | Description |
---|---|
TInput |
Constructors
ParquetDestination()
Declaration
ParquetDestination(string)
Declaration
Parameters
Type | Name | Description |
---|---|---|
string | uri | The source parquet file name or uri |
ParquetDestination(string, ResourceType)
Declaration
Parameters
Type | Name | Description |
---|---|---|
string | uri | The source parquet file name or uri |
ResourceType | resourceType | Specifies if data is loaded from a file, a web endpoint or other storage types (e.g. Azure Blob Storage) |
Properties
AddCustomMetaData
Each parquet writer allows to have a set of custom meta data - use this function to set the custom meta data for every uri (only called multiple times when using HasNextUri / GetNextUri.
Declaration
Property Value
Type | Description |
---|---|
Func<StreamMetaData, Dictionary<string, string>> |
BatchSize
The batch size defines how many records needs to be in the Input buffer before data is written into the destination. The default batch size is 1000.
Declaration
Property Value
Type | Description |
---|---|
int |
HttpContentType
The content type used when sending the http request content.
Declaration
Property Value
Type | Description |
---|---|
string |
Overrides
ParquetColumns
List of mappings between columns names in parquet file and property name in object.
Declaration
Property Value
Type | Description |
---|---|
ICollection<ParquetColumn> |
ParquetOptions
Options used when creating the ParquetWriter
Declaration
Property Value
Type | Description |
---|---|
ParquetOptions |
SupportedDataTypes
Currently the ParquetDestination only supports 'basics' data types. Access this property to get all Clr types that are supported.
Declaration
Property Value
Type | Description |
---|---|
Type[] |
UseBufferBlock
Declaration
Property Value
Type | Description |
---|---|
bool |
Overrides
Methods
CheckParameter()
Declaration
Overrides
CloseStream()
Declaration
Overrides
FindParquetNameMapping(string)
Declaration
Parameters
Type | Name | Description |
---|---|---|
string | propertyName |
Returns
Type | Description |
---|---|
string |
FindPropNameMapping(string)
Declaration
Parameters
Type | Name | Description |
---|---|---|
string | columnName |
Returns
Type | Description |
---|---|
string |
InitStream()
Declaration
Overrides
PrepareParameterForCheck()
Declaration
Overrides
Reset()
Declaration
Overrides
WriteBatch(TInput[])
Declaration
Parameters
Type | Name | Description |
---|---|---|
TInput[] | data |
Overrides
WriteIntoStream(TInput)
Declaration
Parameters
Type | Name | Description |
---|---|---|
TInput | data |