How Lyftron approaches its vision and goals from the technological standpoint:
- Provide a high performance columnar Modern Data Hub engine,
- Emulate most popular and successful enterprise-grade database engine on a protocol level, allowing for broad and fast adoption without additional changes to existing tools and infrastructure,
- Provide pure Modern Data Hub tool that is fully integrated with Microsoft technology stack,
- Take advantage of the fastest data processing engines on the market like Apache Spark and others, enabling users to easily switch engines and use them in parallel,
- Provide full web-based self-service and administration portal,
- Provide internal ETL pipeline that uses parallel columnar processing,
- Provide on-premise, cloud and hybrid deployment models.
Lyftron is a columnar Modern Data Hub engine internally. Incoming rows are converted into columnar format, for high parallelism and high performance processing. Outgoing results are converted into rows for compatibility with the SQL Server protocol.
Microsoft SQL Server emulation
Lyftron's emulation of Microsoft SQL Server is implemented from scratch. It involves compatibility with the following:
- Tabular Data Stream network protocol as described in Microsoft TDS documentation
- Data types and conversions - all data types are normalized automatically into equivalent SQL Server data types
- Metadata model exposed by SQL Server, including metadata catalog, system views, stored procedures, and functions
- SQL dialect supported by SQL Server
- Procedural programming constructs: IF, WHILE, DECLARE
- Temporary tables (stored in-memory)
- Selected SQL session environment variables
- Security model supported by SQL Server
- Authentication using Windows Integrated Authentication and SQL Server Standard Authentication (Mixed Mode)
- Job scheduling model and accompanying stored procedures
Client tools connectivity to Lyftron
Microsoft SQL Server emulation allows any client utilizing Microsoft-supported data access technology to connect to Lyftron. Connectivity to Lyftron was verified using example tools and technologies listed below.
Analytical, reporting or development tools:
- Microsoft Office
- Microsoft SQL Server Management Studio
Data access technologies:
- OLE DB
Lyftron's connectivity to data sources
Out-of-the-box Lyftron can connect to over 100 data source types like relational, MPP, NoSQL, SaaS, CRM's, or Social Media. For a full list of providers please consult the list of built-in providers.
Comparing to classic, flow-based ETL tools, Lyftron implements ETL responsibilities differently. It is assumed that most of processing requirements can be expressed using SQL and Views.
- Virtual Databases are wrappers over a data source to provide direct access to data (no caching allowed), or created over one of 10 supported data processing engines, that may be used for data integration and caching.
- Views may depend on each other and blend data using full power of SQL language.
- Views created in Lyftron's virtual databases are materializable (aka. cachable).
- Many processing-engine dependent strategies can be utilized, for example in-memory caching or persistent caching using table rotation.
- SQL Server restrictions of so called "indexed views" do not apply to Lyftron.
Lyftron can be deployed using the following models:
- on-premise, using physical or virtual machine with Windows (server and desktop)
- in-cloud, PaaS
- hybrid (mixed), on-premise and cloud