About Lakehouse :

Atlas lake house delivers a user-friendly and reliable data integration platform for organizations with growing data needs. You can use Atlas Lake house to automate collecting data from multiple applications and databases, loading it to a data warehouse, and making it analytic-ready. This allows your analysts to set up faster analysis and reporting. The Atlas Lakehouse product is delivered through an extremely intuitive user interface that eliminates the need for technical resources to set up and manage your data Pipelines. Atlas lake house’s design approach goes beyond data Pipelines. Its analyst-friendly data transformation features are well integrated into the platform to further streamline analytic tasks.

Features :

1. Data Source Registration Module :

• The first step in this module is to establish a connection with the source database. We will get server and port details from the user. If the connection is successful, we will proceed. 
• We will use Rest API if the data source is a SAAS application, such as Google Analytics.
• In the case of database Sources, you can load data from one, multiple, or all databases. For each selected object or table, you can specify how the records to be ingested must be queried, through the Query Mode. 
• Similarly, for SaaS Sources, you can select the objects or tables to be fetched or the report’s data to be retrieved and specify the query mode. 

2. Use Case Register Module :

We will take below use case details from the user :

• The size of the data.  
• Frequency of data change.
• Frequency of data load- daily or real-time etc.

3. Key-Partition Module :

• Our data will be placed on S3 once we receive it from different sources. 
• The partitioning of the keys will need to be considered while storing them on S3.
• Partitioning allows the use of different file systems to be installed for different kinds of files. Separating user data from system data can prevent the system partition from becoming full and rendering the system unusable. Partitioning can also make backing up easier. 

4. ETL Process Module :

• Atlas lake house ETL Pipelines, fetch data from your data Source, perform in-flight Transformations based on the settings you configure, and load it to your Destination. 
• In this module we will design an ETL process that will load data from S3 to the target. 
• In this module we will perform the transformation.
• We will make the data compatible with target and use GLUE script and load them data to target.
• ETL is a process in Data Warehousing, and it stands for Extract, Transform and Load. It is a process in which an ETL tool extracts the data from various data source systems, transforms it in the staging area, and then finally, loads it into the Data Warehouse system. 

5. Visualization Module  :

Data visualization brings data to life, making you the master storyteller of the insights hidden within your numbers. Through live data dashboards, interactive reports, charts, graphs, and other visual representations, data visualization helps users develop powerful business insight quickly and effectively.

We will perform visualization using the following BI tools: 

• Power BI 
• Tableau 
• Quick Sight.

Flexible for multiple sectors   :

• Banking sector 
• Financial services 
• Retail sector 
• Healthcare

We are evolving and creating the Atlas lake house app with new and enhanced functionalities to better address the business needs and experiences of our customers.

Components :

A data lake holds a vast amount of raw data in its native format until it is needed. 

1. Data Ingestion :

A highly scalable ingestion-layer system that extracts data from various sources, such as websites, mobile apps, social media, IoT devices, and existing Data Management systems, is required. It should be flexible to run in batch, one-time, or real-time modes, and it should support all types of data along with new data sources.

2. Data Storage :

A highly scalable data storage system should be able to store and process raw data and support encryption and compression while remaining cost-effective.

3. Data Security :

Regardless of the type of data processed, data lakes should be highly secure from the use of multi-factor authentication, authorization, role-based access, data protection, etc.

4. Data Analytics :

After data is ingested, it should be quickly and efficiently analyzed using data analytics and machine learning tools to derive valuable insights and move vetted data into a data warehouse.

5. Data Governance :

The entire process of data ingestion, preparation, cataloging, integration, and query acceleration should be streamlined to produce enterprise-level data quality. It is also important to track the changes to key data elements for a data audit.

Use Case: 01

Financial Services Ltd has embarked on an initiative for Data Lake Solution on AWS. 

Challenges :

• Making product cloud agnostic 
• Security of data in the cloud 
• Add dynamic ETL script based on user option 
• Create visualization on data as per user option. 

Proposed Solution to Customer :

• Quick Deployment for  Datalake creation 
• Easy to monitor and manage infrastructure 
• User friendly and customizable 
• Simplistic hence any type of user can operate 
• Sits on top on customer cloud and not hosted outside customer premises 
• Time Saving 
• cost and manual work optimization