Create data warehouse pipeline
Step 4: Migrate data source data into Lyftron warehouse or any target
Please follow the below steps and screenshots for better understanding
Initial Steps:
Click on the Databases on the left panel → Search LyftronWarehouse -> Click on LyftronWarehouse -> Go to Data Pipeline
** if you want to import any tables that you want to use in your pipeline from the Lyftron Warehouse so you can go to table → import table
Or you can add your own Data Warehouse if you don’t want to use LyftronWarehouse.
Step 4.1: Data migration steps
Please follow the below steps and screenshots for better understanding
Steps:
Click on migrate data button on top right ->Choose the data source you want to migrate -> Click next -> Choose tables that you need to migrate -> Click import
Lyftron created the automatic pipeline for the selected sources and added the postfix as pipeline on the table names and also created the automatic schema on Lyftron with data_source_name to provide a better data lineage.
Step 4.2: Bulk Mode: Pipeline load steps
Please follow the below steps and screenshots for better understanding
Steps:
Sort schema name field of data pipeline → Select all pipelines of data_source schema that imported on Step 4.1 → Select replicate to a target table → Choose Lyftron Warehouse schema → Enter table names that you want to create on the warehouse → Click on update → you will get success screen that tables are created → Click on management tasks → Load Data → Click on the top bell icon and it will show the status that jobs are started.
Step 4.2.1: Sort schema name field of data pipeline
Step 4.2.2: Select all pipelines of data_source_name schema that you have imported
Step 4.2.3: Choose Target schema → Enter table names that you want to create on the warehouse → Click on update
Step 4.2.4: Click on the management tasks and select load data
Step 4.2.5: Click the job submission details from the bell icon
Step 4.3: Individual Mode: Full load Pipeline steps
Please follow the below steps and screenshots for better understanding
Steps:
Click on the pipeline that you want to load the data for → Go to Replication & Caching→ Click on edit → Select replicate to a target table → Choose Target Schema → Choose Target Table, if this is a new table so create new table by giving the name of the table → now you can all the fields are mapped, choose the fields which you don’t want to load → Click on Add Table → Choose Caching refresh mode as full → Click Save → Go to Management tasks → Click on load data
Step 4.4: Individual Mode: Incremental load Pipeline steps
Please follow the below steps and screenshots for better understanding
Steps:
Click on the pipeline that you want to load the data for → Go to Replication & Caching→ Click on edit → Select replicate to a target table → Choose Target Schema → Choose Target Table, if this is a new table so create new table by giving the name of the table → now you can all the fields are mapped, choose the fields which you don’t want to load → Click on Add Table → Choose Caching refresh mode as incremental → Choose Click Save → Go to Management tasks → Click on load data
Step 4.5: Adding Workflow & Pipeline Scheduler
Lyftron have two options to add workflow and we will cover both of the steps
Step 4.5.1: Adding workflow & scheduling from pipeline designer
Please follow the below steps to add the Workflow and job scheduler in your pipeline
Go to the Management Tasks in the pipeline section in warehouse → Click on Load data → Click define workflow → Now you run or Schedule → To schedule click on Schedule → Add Schedule → Select schedule frequency. Your workflow is ready and you can check it now in the workflow section
Step 4.5.2: Adding workflow & scheduling from workflow designer
Please follow the below steps to add the Workflow and job scheduler in your pipeline
Go to Workflow → Click on Add Workflow → Click Create -->Click on the workflow just created → Click on the steps → Click Add Step → Choose Stored Procedure from step type → Choose lyft_load_data as Job -->Type Data pipeline name → Choose your success and failure options → Choose your Datasource for the workflow → Click Save → Click on the schedule section → Add Schedule → Select schedule frequency. Your schedule is ready and you can check it now in the schedule section → If you need to check Workflow Execution history of the workflow so you can do that by clicking on the execution history tab → If you need Step execution history you can check by clicking step execution history from step overview section
Step 4.6: Monitor
Monitor section contain details about the job history and statement that pipeline have executed
Step 4.6.1: Statement history
Check the pipeline statement history by clicking on the monitor section -> statement history
Step 4.6.2: Workflow history
Check the pipeline job history to check status of the jobs
Step 4.6.3: Execution Plan
Check the pipeline health status with the execution plan → Click on each of the executed queries to check the details execution steps → Click on the each step to get the details about it
Step 4.7: Validate data load on Lyftron Warehouse
Go to the analyze section and drag and drop the table from LyftronWarehouse or your target data warehouse schema section
