Lyftrondata realtime data pipeline is Data Virtualization concept which offer users a single interface often based on SQL to access data in multiple places or formats with the compute of Spark. By default Lyftrondata created realtime pipeline until target table in defined under Replication & Caching section.
Realtime Data Pipeline Types
- User Defined Views: This Realtime data pipeline features allows users to define the standardization rules, subquery, joins, unions or transformation on the source data in the Normalization section. This act as a prepping layer before the data is landed in the warehouse as shown below. You can also use this section to save your predefined queries that you want to run on the source platforms more often.
One of the live subquery and data filtering example of how Lyftrondata users utilizes user defined views section to prep the Paylocity source data before it's landed in the Warehouse.
- Realtime Data Pipeline Warehouse: This Pipeline feature allows to build virtual view over Data Vault, Data Warehouse, Data Marts and Semantic models for faster processing of records for BI consumption. Lyftrondata provides the power of spark caching compute to the realtime pipeline as shown below
