IDataFlowStreamSource
Interface IDataFlowStreamSource
Implemented by data flow sources that allow reading data from a stream source
Inherited Members
Namespace: ETLBox
Assembly: ETLBox.dll
Syntax
Properties
AzureBlobStorage
Use this to pass your AzureBlobStorage connection information. There are different methods to connect with Azure Blob Storage: Option 1: provide an existing BlockBlobClient Option 2: create BlockBlobClient based on the Uri and optional credential tokens - the Uri needs to be the blob url Option 3: create BlockBlobClient with provided ConnectionString and ContainerName, the Uri is then the name of the blob
Declaration
Property Value
Type | Description |
---|---|
AzureBlobStorageConfiguration |
CollectUnparsedData
Indicates if data not parsed should be stored in the UnparsedData property (also accessable via the current StreamMetaData object when using the GetNextUri/HasNextUri pattern).
Declaration
Property Value
Type | Description |
---|---|
bool |
CreateStreamReader
By default, stream are created internally based on Uri & ResourceType. If you already have an existing stream, use this function to returns a StreamReader based on this stream. You can create a stream reader for each requested URI.
Declaration
Property Value
Type | Description |
---|---|
Func<string, StreamReader> |
Encoding
Encoding used to read data from the source file or web request.
Declaration
Property Value
Type | Description |
---|---|
Encoding |
GetNextUri
This function returns the next url that is used for reading data. It will be called until HasNextUri returns false. The incoming StreamMetaData holds information about the current progress and other meta data from the response, like unparsed json data that contains references to the next page of the response. This property can be used if you want to read multiple files or if you want to paginate through web responses.
Declaration
Property Value
Type | Description |
---|---|
Func<StreamMetaData, string> |
HasNextUri
This func determines if another request is started to read additional data from the next uri. StreamMetaData has information about the current progress and other meta data from the response.
Declaration
Property Value
Type | Description |
---|---|
Func<StreamMetaData, bool> |
HttpClient
The System.Net.Http.HttpClient uses for the request. Use this client if you want to add or change the http request data, e.g. you can add your authorization information here.
Declaration
Property Value
Type | Description |
---|---|
HttpClient |
HttpRequestMessage
The System.Net.Http.HttpRequestMessage used for the request in the HttpClient. Use this property to add additional header data type or to change the http method (e.g. for POST requests).
Declaration
Property Value
Type | Description |
---|---|
HttpRequestMessage |
HttpResponseMessage
The System.Net.HttpResponseMessage returned from the current request.
Declaration
Property Value
Type | Description |
---|---|
HttpResponseMessage |
RequestUri
The Uri used for the current request - this will only be different to the Uri if you use the GetNextUri and HasNextUri functions.
Declaration
Property Value
Type | Description |
---|---|
string |
ResourceType
Specifies the resource type. By default requests are made with HttpClient. Specify ResourceType.File if you want to read from a json file.
Declaration
Property Value
Type | Description |
---|---|
ResourceType |
SkipRows
Number of rows to skip before starting reading the header and csv data
Declaration
Property Value
Type | Description |
---|---|
int |
StreamReader
The stream reader used for the current request.
Declaration
Property Value
Type | Description |
---|---|
StreamReader |
UnparsedData
Contains all data from the stream that couldn't be processed.
Declaration
Property Value
Type | Description |
---|---|
string |
Uri
The Url of the webservice (e.g. https://test.com/foo) or the file name (relative or absolute).
Declaration
Property Value
Type | Description |
---|---|
string |
UseMulitpartContent
If the response is a multipart response, this function is used to determine which content part of the multipart response to use.
Declaration
Property Value
Type | Description |
---|---|
Func<HttpContent, bool> |