Overview
The Lyftron Provider for Amazon DynamoDB offers the most natural way to access Amazon DynamoDB data from Lyftron with ease and also enables to connect with BI, MDM & ML tools, Data warehouses, Databases and other SAAS based applications with zero code and zero infrastructure requirements. The provider wraps the complexity of accessing Amazon DynamoDB data into easy-to-integrate relational fully managed ANSI Sql format. Make faster and better business decisions with Lyftron’s Amazon DynamoDB data provider and automatically build your data migration pipelines in minutes, not months
The provider hides the complexity of accessing data and provides additional powerful security features, smart caching, batching, socket management, and more.
Key Features
- Comprehensive Delta load mechanism.
- Real-time access to Amazon DynamoDB.
- Comprehensive full support of ANSI Sql to query data with ease.
- Collaborative query processing.
Prerequisites
The user must have credentials for Amazon DynamoDB, Lyftron and your destination data warehouse, lake or database to perform the data pipeline operation with Lyftron
Establishing a Connection with Lyftron's Quickstart Steps
Create your Amazon DynamoDB connection with Lyftron by following the 5 easy steps show below:
Step1. Add your connection
Click on Connect section on the left panel → Click on Add Connection button
Step2. Select your connector
In the connector selection panel, search and click Amazon DynamoDB for your connection
Step3. Enter your connection details
In the Connection String section enter the values of the below parameters. The following connection string is required to establish Amazon DynamoDB connection with Lyftron.
Access Key=xxx;Secret Key=xxx;Domain=amazonaws.com;Region=OREGON;| Key | Value | Field |
| Connection Name | Enter your connection details | Required |
| Access Key | Your DynamoDB access key | Required |
| Secret Key | Your DynamoDB secret key | Required |
| Domain | amazonaws.com | Required |
| Region | OREGON | Required |
| Logfile** | Use the logfile option to debug your job and provide your connection name to generate the log file. [ConnectionConfigurationPath]\Connection_name_log.tx | Optional |
| Verbosity** | Choose verbosity 1-5 based on the severity of debugging | Optional |
** For more information, check the Lyftron logging and debugging section.
If you want more detailed information about how to establish a connection with Lyftron, click on Lyftron Connection Quick Start guide.
Step4. Test your connection
Once you are done entering your connection details, simply click on the Test Connection button to test the connectivity. In case your connection fails, add Logfile and Verbosity parameters and check the Lyftron logging and debugging section, to debug the error.
Step5. Save your connection
Hurray! Now you have successfully connected with the Lyftron Amazon DynamoDB connector and can utilize the connector to Extract, Warehouse, Analyze, Visualize and Share your data.
Data Model
The list of tables is dynamically retrieved from your Amazon DynamoDB account. You can use the CreateTable stored procedure to create a new table, or you can create a table using the Amazon Web Services Admin Console.
Because DynamoDB tables are partitioned based on their key, you should take care in selecting a proper key based on the query requirements of your table. Refer to the documentation for DynamoDB for more information about using best practices to model data in DynamoDB tables. DynamoDB supports two types of primary keys:
- Hash Primary Key: This is a single-column key.
- Hash and Range Primary Key: This is a two-column key that includes a hash column and a range column.
The provider will model all key attributes in DynamoDB as key columns.
Table Columns
Since Amazon DynamoDB tables are schemaless, the provider offers the following two mechanisms to uncover the schema.
Dynamic Schemas
The columns of a table are dynamically determined by scanning data in the first few rows. You can adjust the number of rows that are used by modifying the RowScanDepth property. In addition to the name of the column, the row scan also determines the data type. The following table shows how the different data types supported by Amazon DynamoDB are modeled in the provider.
| Amazon DynamoDB Type | Modeled Type | Encoding | Sample Value |
| Boolean | Boolean | Not Required | USA |
| String | String | Not Required | USA |
| Blob | String | Not Required | |
| Number | Double | Not Required | 24 |
| String Array | String | JSON Array | ["USA","Canada","UK"] |
| Number Array | String | JSON Array | [20,200.5,500] |
| Blob Array | JSON Array | JSON Array | ["ABCD","EFGH"] |
| Document | JSON Object | JSON Object | {"Address":"123 Fake Street","City":"Chapel Hill","Zip":"27516"} |
| List | JSON Array | JSON Array | [{"S":"mystring"},{"NS":[1,2]},{"N":4}] |
Static Schemas
Instead of using dynamically discovered schemas, you can define your own schemas. This will give you more control over the projected columns and also enable you to use other data types such as boolean, datetime, etc. Refer to the CreateSchema Stored Procedure in order to create your own schema. You can simply specify the FileName (fullpath) and TableName of the new schema file, which should match with the name of the Amazon DynamoDB table, and edit the column listing to use it for your own table.
Schemaless Operations
While the schema of the table is necessary to report metadata, data may be selected, inserted, updated, or deleted from columns that do not exist in the schema. Columns that do not already exist in the table schema will have their data types dynamically determined based on the data that is specified. See DynamoDB Queries for more information.
Advanced Settings
To view a detailed advanced settings options, go to Amazon DynamoDB Advanced Settings. Complete list of the parameters you can configure in the connection string can be found by clicking Connection String Parameters.