ETLBox ETLBox ETLBox Buy
API
Try Buy

LookupTransformation<TInput, TSource>

Class LookupTransformation<TInput, TSource>

The lookup transformation enriches the incoming data with data from the lookup source. Data from the lookup source is read into memory when the first record arrives. For each incoming row, the lookup tries to find a matching record in the loaded source data and uses this record to enrich the ingoing data.

Inheritance
object
LoggableTask
DataFlowComponent
DataFlowSource<TInput>
DataFlowTransformation<TInput, TInput>
LookupTransformation<TInput, TSource>
LookupTransformation
Implements
IDataFlowTransformation<TInput, TInput>
Inherited Members
Namespace: ETLBox.DataFlow
Assembly: ETLBox.dll
Syntax
    public class LookupTransformation<TInput, TSource> : DataFlowTransformation<TInput, TInput>, IDataFlowTransformation<TInput, TInput>, IDataFlowSource<TInput>, IDataFlowSource, IDataFlowDestination<TInput>, IDataFlowDestination, IDataFlowComponent, ILoggableTask
Type Parameters
NameDescription
TInput

Type of ingoing and outgoing data.

TSource

Type of data used in the lookup source.

Constructors

LookupTransformation()

Declaration
    public LookupTransformation()

LookupTransformation(IDataFlowExecutableSource<TSource>)

Declaration
    public LookupTransformation(IDataFlowExecutableSource<TSource> source)
Parameters
TypeNameDescription
IDataFlowExecutableSource<TSource>source

Sets the Source of the lookup.

LookupTransformation(IDataFlowExecutableSource<TSource>, Func<TInput, CachedData<TSource>, TInput>)

Declaration
    public LookupTransformation(IDataFlowExecutableSource<TSource> source, Func<TInput, CachedData<TSource>, TInput> applyRetrievedCacheToInput)
Parameters
TypeNameDescription
IDataFlowExecutableSource<TSource>source

Sets the Source of the lookup.

Func<TInput, CachedData<TSource>, TInput>applyRetrievedCacheToInput

Sets the ApplyRetrievedCacheToInput function.

Properties

ApplyRetrievedCacheForMultipleOutputs

This function describes how the ingoing data can be enriched with the already pre-read data from the Source, but can return multiple rows for every input row. This is necessary if PermitMultipleEntriesPerKey is set to true, which allows the cache lookup to load multiple rows for every key.

Declaration
    public Func<TInput, CachedData<TSource>, TInput[]> ApplyRetrievedCacheForMultipleOutputs { get; set; }
Property Value
TypeDescription
Func<TInput, CachedData<TSource>, TInput[]>

ApplyRetrievedCacheToInput

This function describes how the ingoing data can be enriched with the already pre-read data from the Source.

Declaration
    public Func<TInput, CachedData<TSource>, TInput> ApplyRetrievedCacheToInput { get; set; }
Property Value
TypeDescription
Func<TInput, CachedData<TSource>, TInput>

CacheMode

The cache mode used for loading data into the lookup cache. By default, all data is loaded into memory from the lookup source when the first records arrives at the lookup.

Declaration
    public CacheMode CacheMode { get; set; }
Property Value
TypeDescription
CacheMode

CachedBatchTransformation

Declaration
    public CachedBatchTransformation<TInput, TInput, TSource> CachedBatchTransformation { get; }
Property Value
TypeDescription
CachedBatchTransformation<TInput, TInput, TSource>

InputKeySelector

This function describe how the key is generated for an ingoing row. If the result of this function match with SourceKeySelector, both records can be identified as a match.

Declaration
    public Func<TInput, object> InputKeySelector { get; set; }
Property Value
TypeDescription
Func<TInput, object>

MatchColumns

This collection will be used to define the matching columns - will also work with ExpandoObject.

Declaration
    public ICollection<MatchColumn> MatchColumns { get; set; }
Property Value
TypeDescription
ICollection<MatchColumn>

OnNoMatchFound

Specifies the function to be invoked when a lookup operation fails to find a matching key in the source. Will only be invoked if ApplyRetrievedCacheToInput or ApplyRetrievedCacheForMultipleOutputs is not set! This action contains the current input row and the lookup key for which no match was found.

Declaration
    public Action<TInput, object> OnNoMatchFound { get; set; }
Property Value
TypeDescription
Action<TInput, object>

PartialCacheSettings

Only need if you set the CacheMode to PartialDbCache. This will only work if you use a DbSource as lookup source.

Declaration
    public LookupTransformation<TInput, TSource>.PartialDbCacheSettings PartialCacheSettings { get; set; }
Property Value
TypeDescription
LookupTransformation<TInput, TSource>.PartialDbCacheSettings

PermitMultipleEntriesPerKey

If set to true, multiple entries can be loaded into the cache for every key. Otherwise, only the first occurence will be loaded into the cache.

Declaration
    public bool PermitMultipleEntriesPerKey { get; set; }
Property Value
TypeDescription
bool

ProgressCount

Declaration
    public int ProgressCount { get; }
Property Value
TypeDescription
int

RemoveUnmatchedRows

If no match is found in the lookup source, the row will be removed from the output. Will only be invoked if ApplyRetrievedCacheToInput or ApplyRetrievedCacheForMultipleOutputs is not set!

Declaration
    public bool RemoveUnmatchedRows { get; set; }
Property Value
TypeDescription
bool

RetrieveColumns

This collection will be used to define the retrieve columns - will also work with ExpandoObject.

Declaration
    public ICollection<RetrieveColumn> RetrieveColumns { get; set; }
Property Value
TypeDescription
ICollection<RetrieveColumn>

ShouldSkipRow

Specifies if a row should be skipped based on the input row. If this function returns true, the row will be skipped and not be processed by the lookup. Will only be invoked if ApplyRetrievedCacheToInput or ApplyRetrievedCacheForMultipleOutputs is not set!

Declaration
    public Predicate<TInput> ShouldSkipRow { get; set; }
Property Value
TypeDescription
Predicate<TInput>

Source

The source component from which the lookup data is retrieved. E.g. a DbSource or a MemorySource.

Declaration
    public IDataFlowExecutableSource<TSource> Source { get; set; }
Property Value
TypeDescription
IDataFlowExecutableSource<TSource>

SourceBlock

SourceBlock from the underlying TPL.Dataflow which is used as output buffer for the component.

Declaration
    public override ISourceBlock<TInput> SourceBlock { get; }
Property Value
TypeDescription
ISourceBlock<TInput>
Overrides

SourceKeySelector

This function describe how the key is generated for a row from the lookup source. This key is used to store the data in the lookup dictionary.

Declaration
    public Func<TSource, object> SourceKeySelector { get; set; }
Property Value
TypeDescription
Func<TSource, object>

TargetBlock

TargetBlock from the underlying TPL.Dataflow which is used as input buffer for the component.

Declaration
    public override ITargetBlock<TInput> TargetBlock { get; }
Property Value
TypeDescription
ITargetBlock<TInput>
Overrides

UseExactNumericTypes

If set to true, data types are ignored when comparing data to identify matching records. E.g. property that stores the integer value 7 would match with a string "7".

Declaration
    public bool UseExactNumericTypes { get; set; }
Property Value
TypeDescription
bool

Methods

CheckParameter()

Declaration
    protected override void CheckParameter()
Overrides

CleanUpOnFaulted(Exception)

Declaration
    protected override void CleanUpOnFaulted(Exception e)
Parameters
TypeNameDescription
Exceptione
Overrides

CleanUpOnSuccess()

Declaration
    protected override void CleanUpOnSuccess()
Overrides

InitComponent()

Declaration
    protected override void InitComponent()
Overrides

LinkErrorTo(IDataFlowDestination<ETLBoxError>)

If an error occurs in the component, by default the component will throw an exception and stop execution. If you use the error linking, any erroneous records will be caught and redirected.

Declaration
    public override IDataFlowSource<ETLBoxError> LinkErrorTo(IDataFlowDestination<ETLBoxError> target)
Parameters
TypeNameDescription
IDataFlowDestination<ETLBoxError>target

The target for erroneous rows.

Returns
TypeDescription
IDataFlowSource<ETLBoxError>

The linked component.

Overrides

PrepareParameterForCheck()

Declaration
    protected override void PrepareParameterForCheck()
Overrides

Reset()

Declaration
    protected override void Reset()
Overrides

Implements