Amazon S3

The Lyftron Provider for Amazon S3 offers the most natural way to access Amazon S3 data from Lyftron with ease and also enables to connect with BI, MDM & ML tools, Data warehouses, Databases and other SAAS based applications with zero code and zero infrastructure requirements. The provider wraps the complexity of accessing Amazon S3 data into easy-to-integrate relational fully managed ANSI Sql format. Make faster and better business decisions with Lyftron’s Amazon S3 data provider and automatically build your data migration pipelines in minutes, not months

The provider hides the complexity of accessing data and provides additional powerful security features, smart caching, batching, socket management, and more.

Key Features

• Comprehensive Delta load mechanism.
• Real-time access to Amazon S3.
• Comprehensive full support of ANSI Sql to query data with ease.
• Collaborative query processing.

Prerequisites

The user must have credentials for Amazon S3, Lyftron and your destination data warehouse, lake or database to perform the data pipeline operation with Lyftron

Establishing a Connection with Lyftron's Quickstart Steps

Create your Amazon S3 connection with Lyftron by following the 5 easy steps show below:

Step1. Add your connection

Click on Connect section on the left panel →  Click on Add Connection button

Step2. Select your connector

In the connector selection panel, search and click Amazon S3 for your connection 

Step3. Enter your connection details

In the Connection String section enter the values of the below parameters. The following connection string is required to establish Amazon S3 connection with Lyftron.

"AccessKey=a123;SecretKey=s123;"

Key	Value	Field
Connection Name	Enter your connection details	Required
AccessKey	a123	Required
SecretKey	a123	Required
Logfile**	Use the logfile option to debug your job and provide your connection name to generate the log file.  [ConnectionConfigurationPath]\Connection_name_log.tx	Optional
Verbosity**	Choose verbosity 1-5 based on the severity of debugging	Optional

** For more information, check the Lyftron logging and debugging section.

If you want more detailed information about how to establish a connection with Lyftron, click on Lyftron Connection Quick Start guide.

Step4. Test your connection

Once you are done entering your connection details, simply click on the Test Connection button to test the connectivity. In case your connection fails, add Logfile and Verbosity parameters and check the Lyftron logging and debugging section, to debug the error.

 Step5. Save your connection

Hurray! Now you have successfully connected with the Lyftron Amazon S3 connector and can utilize the connector to Extract, Warehouse, Analyze, Visualize and Share your data.

Data Model

This section describes how the provider models the Amazon S3 API as relational Views

Key Features 

• The provider models Amazon S3 buckets, bucket information, and objects as relational views, allowing you to write SQL to query Amazon S3 data.
• Stored procedures allow you to execute operations to Amazon S3, including downloading and uploading objects.
• Live connectivity to these objects means any changes to your Amazon S3 account are immediately reflected when using the provider.

Generally, querying Amazon S3 tables is the same as querying a table in a relational database. Sometimes there are special cases, for example, including a certain column in the WHERE clause might be required to get data for certain columns in the table. This is typically needed for situations where a separate request must be made for each row to get certain columns. 

Views are composed of columns and pseudo columns. Views are similar to tables in the way that data is represented; however, views do not support updates. Entities that are represented as views are typically read-only entities. Often, a stored procedure is available to update the data if such functionality is applicable to the data source.

Name	Type	Description
Buckets	Tables	Returns information for buckets. Buckets are used to store objects, which consist of data and its metadata.
Objects	Tables	Returns information for objects inside a specific bucket.
BucketsACL	Views	Returns the access control list (ACL) of a bucket. An ACL allows you to set permissions on each object in a specific bucket.
BucketsAnalytics	Views	Returns an analytics configuration (identified by the analytics configuration ID) from the bucket.
BucketsCORS	Views	Returns the CORS configuration information set for the bucket. CORS allows cross-domain communication.
BucketsInventory	Views	Returns an inventory configuration (identified by the inventory configuration ID) from the bucket.
BucketsLifecycle	Views	Returns the lifecycle configuration information set on the bucket.
BucketsReplication	Views	Returns a bucket's replication configuration.
ObjectsACL	Views	Returns the access control list (ACL) of an object.
PublicAccessBlock	Views	Retrieves the PublicAccessBlock configuration for an Amazon S3 bucket.