Create data pipeline for target
Step 4: Migrate data source data into Lyftrondata warehouse or any target
Please follow the below steps and screenshots for a better understanding
Initial Steps:
Click on the Target on the left panel -> Add Target -> Give Name -> Choose Existing Connection -> In this example we are choosing Tgt_Snowflake connection
** if you want to import any tables that you want to use in your pipeline from the Lyftrondata Warehouse so you can go to the table → import table.
Step 4.1: Data migration steps
Please follow the below steps for a better understanding
Steps:
Click on migrate data button on the top right ->Choose the data source you want to migrate.(In this example it's Gmail) -> Click next -> Choose tables that you need to migrate -> Click import
Lyftrondata created the automatic pipeline for the selected sources and added the postfix as a pipeline on the table names and also created the automatic schema on Lyftrondata with data_source_name to provide a better data lineage.
Step 4.2: Bulk Mode: Pipeline load steps
Please follow the below steps and screenshots for a better understanding
Steps:
Sort schema name field of data pipeline → Select all pipelines of data_source schema that imported on Step 4.1 → Select replicate to a target table → Choose Target schema → table names will populate automatically → Click on create table → Click on management tasks → Load Data → Click on the top bell icon and it will show the status that jobs are started.
Step 4.2.1: Select all pipelines that you want to apply replication for
Step 4.2.2: Choose Target schema → Table names will auto-populate → Click on create table
Step 4.2.4: Click on the management tasks and select load data
Step 4.2.5: Click the job submission details from the bell icon
Step 4.3: Individual Mode: Full load Pipeline steps
Please follow the below steps and screenshots for a better understanding
Steps:
Click on the pipeline that you want to load the data for → Go to Replication & Caching→ Click on edit → Select replicate to a target table → Choose Target Schema → Choose Target Table if this is a new table so create a new table by giving the name of the table → now you can all the fields are mapped, choose the fields which you don’t want to load → Click on Add Table → Choose Caching refresh mode as full → Click Save → Go to Management tasks → Click on load data
Step 4.4: Individual Mode: Incremental load Pipeline steps
Please follow the below steps and screenshots for better understanding
Steps:
Click on the pipeline that you want to load the data for → Go to Replication & Caching→ Click on edit → Select replicate to a target table → Choose Target Schema → Choose Target Table, if this is a new table so create new table by giving the name of the table → now you can all the fields are mapped, choose the fields which you don’t want to load → Click on Add Table → Choose Caching refresh mode as incremental → Choose Click Save → Go to Management tasks → Click on load data
Step 4.5: Adding Workflow & Pipeline Scheduler
Lyftrondata have two options to add workflow and we will cover both of the steps
Step 4.5.1: Adding workflow & scheduling from pipeline designer
Please follow the below steps to add the Workflow and job scheduler in your pipeline
Go to the Management Tasks in the pipeline section in warehouse → Click on Load data → Click define workflow → Now you run or Schedule → To schedule click on Schedule → Add Schedule → Select schedule frequency. Your workflow is ready and you can check it now in the workflow section
Step 4.5.2: Adding workflow & scheduling from workflow designer
Please follow the below steps to add the Workflow and job scheduler in your pipeline
Go to Workflow → Click on Add Workflow → Click Create -->Click on the workflow just created → Click on the steps → Click Add Step → Choose Stored Procedure from step type → Choose lyft_load_data as Job -->Type Data pipeline name → Choose your success and failure options → Choose your Datasource for the workflow → Click Save → Click on the schedule section → Add Schedule → Select schedule frequency. Your schedule is ready and you can check it now in the schedule section → If you need to check Workflow Execution history of the workflow so you can do that by clicking on the execution history tab → If you need Step execution history you can check by clicking step execution history from step overview section
Step 4.6: Monitor
Monitor section contain details about the job history and statement that pipeline have executed
Step 4.6.1: Statement history
Check the pipeline statement history by clicking on the monitor section -> statement history
Step 4.6.2: Workflow history
Check the pipeline job history to check status of the jobs
Step 4.6.3: Execution Plan
Check the pipeline health status with the execution plan → Click on each of the executed queries to check the details execution steps → Click on the each step to get the details about it
Step 4.7: Validate data load on Lyftrondata Target
Go to the analyze section and drag and drop the table from LyftrondataWarehouse or your target data warehouse schema section
