Data Integration

Data Integration

Every organization requires data integration of some sort, from the simplest ERP data integration using Excels and CSVs to the most modern near real-time data integration. The use of Extract Load and Transform (E-LT) tools will simplify the process of using Excels, CSVs and accurately load all data to the required database for transformation. Generally, this process contains a data source and target. With modern day technology, performance enables us to use the E-LT approach rather than the old school ETL approach. E-LT leverages the power of the target database rather than transforming data in the ETL server which is extra cost. Moreover, E-LT usually happens as a night job or when source and target database is less busy which enables the database to be fully utilized.

Modern technology also enables us to have real time data analytics by using Change Data Capture (CDC) to synchronize data from source to target and directly transforming in the target database. Such technology allows users to react to the market in real-time and benefits organization to react to situations as well as new business strategies. Business strategies may require fine-tuning and these are done with daily reviews and without the correct real-time data, information is delayed. With lagging data, actions always have a cost.

Change Data Capture (CDC)

As long as you have multiple business locations and multiple databases around the world, you will require CDC to sync all data to a single data warehouse, simply to create a single source of truth. If you have been through this, you will understand that ETL window time during a night job is limited by data source night jobs, internet bandwidth especially when it is connected to China or India as well as the vast amount of data that needs to be transformed and loaded.

CDC eliminates all these hassles by doing Extract and Load near real-time 24/7 and centralizes data for further processing. The more data you have, the more savings you get. And when we talk about savings, this includes time, labor, maintenance and performance.

Let us know if time, labor, maintenance and performance is important to your data warehouse.

Extract, Load and Transform (E-LT)

Traditionally, ETL is the core technology for data integration. Since the 80s, processing and storage is expensive and complicated, hence ETL does not leverage current day computing power and storage. Modern design will leverage target database computing power as well as storage. Upgrading the target is much more beneficial than buying a new ETL server as this will improve efficiency 24/7.