Lyftrondata pipeline allows to move and virtualize data between sources like Netsuite, Servicenow, Google Analytics etc and warehouse platforms like Snowflake, Google Bigquery, Redshift, S3 Data Lake, Azure Synapse etc with easy steps. Lyftrondata Pipeline follows the below architecture which is divided into six layers and known as Lyftrondata Lego Blocks as each layer is complementing one another and helping to perform certain actions.
Benefits
- Automatically convert any source API endpoints into the normalized structure.
- Automatically convert and map the data types of source platforms with the target platforms.
- Automatically build the replication and virtualize pipeline between the source and target in minutes and doesn't require any technical programming background.
Categories of Pipeline Supported
Realtime Data Pipeline
Lyftrondata real-time data pipeline enables data virtualization capabilities by creating the Lyftrondata View which uses in memory caching powered by Lyftrondata Spark engine.
Click here for more details
Replication Data ETL Pipeline
Lyftrondata replication data ETL pipeline enables data load capabilities between source and target platform and perform partial pushdown to the target platform to get data processing performance.
Click here for more details
Replication Data ELT Pipeline
Lyftrondata replication data ELT pipeline enables data load capabilities between source and target platform and perform full pushdown to the target platform and utilizes the compute power of the target platform to provide enhance performance for the data loads.
Click here for more details
Types of Pipeline Sync Supported
Full Sync
Lyftrondata pipeline full sync feature enables users to load the historical data from source to target platform. This readme describes Lyftrondata conventions around the "Full Sync" concept.
Click here for more details
Incremental Source
Lyftrondata pipeline incremental source sync feature enables users to load the incremental data from source to target platform based on the source table trigger field. This readme describes Lyftrondata conventions around the "Incremental Source" concept.
Click here for more details
Incremental Target
Lyftrondata pipeline incremental target sync feature enables to load the incremental data from source to target platform based on the target table trigger field. This readme describes Lyftrondata conventions around the "Incremental Target" concept.
Click here for more details
Lyftrondata Job Marker
Lyftrondata pipeline incremental job marker sync feature enables users to load the incremental data from source to target platform based on the Lyftrondata platform generated date value for source table trigger field. This readme describes Lyftrondata conventions around the "Incremental Lyftrondata Job Marker" concept.
Click here for more details
Types of Pipeline Load Supported
Insert
Lyftrondata pipeline insert feature enables users to append records from source to target platform as a new record. This readme describes Lyftrondata conventions around the "Insert" concept.
Click here for more details
Slowly Changing Dimension Type1
Lyftrondata pipeline SCD Type1 feature enables users to replace the change records from source to target platform. This readme describes Lyftrondata conventions around the "SCD Type1" concept.
Click here for more details
Slowly Changing Dimension Type2
Lyftrondata pipeline SCD Type2 feature enables to version the change records from source to target platform by adding Lyftrondata generated metadata fields. This readme describes Lyftrondata conventions around the "SCD Type2" concept.
Click here for more details
Types of Pipeline Actions Supported
Migrate Data
Lyftrondata pipeline migrate data feature enables users to create automatic pipeline based on the source table selection in bulk and load them into target platform quickly. This readme describes Lyftrondata conventions around the "Migrate Data" concept.
Click here for more details
Custom Sql
Lyftrondata pipeline custom sql feature enables users to create pipeline based on the custom sql query to perform data virtualization or data load transformation logic. This readme describes Lyftrondata conventions around the "Custom Sql" concept.
Click here for more details
Join
Lyftrondata pipeline join feature enables users to create pipeline based on the Lyftrondata ETL UI to perform data virtualization or data load transformation logic with drag and drop options to perform inner, left outer, right outer and full outer joins operations. This readme describes Lyftrondata conventions around the "Join" concept.
Click here for more details
Union
Lyftrondata pipeline union feature enables users to create a pipeline based on the Lyftrondata ETL UI to perform data virtualization or data load transformation logic with drag and drop options to perform union, intersect & union all operations. This readme describes Lyftrondata conventions around the "Union" concept.
Click here for more details
Data Vault
Lyftrondata pipeline data vault feature enables to create pipeline based on the Lyftrondata ETL UI to perform data virtualization or data load transformation logic with drag and drop options to perform data vault modeling operation. This readme describes Lyftrondata conventions around the "Data Vault" concept.
Click here for more details
Cube
Lyftrondata pipeline cube feature enables users to create a pipeline based on the Lyftrondata ETL UI to perform data virtualization or data load transformation logic with drag and drop options to perform data cube operation. This readme describes Lyftrondata conventions around the "Cube" concept.
Click here for more details