Data Transformation Pipeline
- by davenewza
I have create some kind of data pipeline to transform coordinate data into more useful information.
Here is the shell of pipeline:
public class PositionPipeline
{
protected List<IPipelineComponent> components;
public PositionPipeline()
{
components = new List<IPipelineComponent>();
}
public PositionPipelineEntity Process(Position position)
{
foreach (var component in components)
{
position = component.Execute(position);
}
return position;
}
public PositionPipeline RegisterComponent(IPipelineComponent component)
{
components.Add(component);
return this;
}
}
Every IPipelineComponent accepts and returns the same type - a PositionPipelineEntity. Code:
public interface IPipelineComponent
{
PositionPipelineEntity Execute(PositionPipelineEntity position);
}
The PositionPipelineEntity needs to have many properties, many which are unused in certain components and required in others. Some properties will also have become redundant at the end of the pipeline.
For example, these components could be executed:
TransformCoordinatesComponent: Parse the raw coordinate data into a Coordinate type.
DetermineCountryComponent: Determine and stores country code.
DetermineOnRoadComponent: Determine and store whether coordinate is on a road.
Code:
pipeline
.RegisterComponent(new TransformCoordinatesComponent())
.RegisterComponent(new DetermineCountryComponent())
.RegisterComponent(new DetermineOnRoadComponent());
pipeline.Process(positionPipelineEntity);
The PositionPipelineEntity type:
public class PositionPipelineEntity
{
// Only relevant to the TransformCoordinatesComponent
public decimal RawCoordinateLatitude { get; set; }
// Only relevant to the TransformCoordinatesComponent
public decimal RawCoordinateLongitude { get; set; }
// Required by all components after TransformCoordinatesComponent
public Coordinate CoordinateLatitude { get; set; }
// Required by all components after TransformCoordinatesComponent
public Coordinate CoordinateLongitude { get; set; }
// Set in DetermineCountryComponent, not required anywhere.
// Requires CoordinateLatitude and CoordinateLongitude (TransformCoordinatesComponent)
public string CountryCode { get; set; }
// Set in DetermineOnRoadComponent, not required anywhere.
// Requires CoordinateLatitude and CoordinateLongitude (TransformCoordinatesComponent)
public bool OnRoad { get; set; }
}
Problems:
I'm very concerned about the dependency that a component has on properties. The way to solve this would be to create specific types for each component. The problem then is that I cannot chain them together like this.
The other problem is the order of components in the pipeline matters. There is some dependency. The current structure does not provide any static or runtime checking for such a thing.
Any feedback would be appreciated.
