Routing and Transformation

Johnston, Craig

Previous chapters covered the construction of databases, Data Lakes, and Data Warehouses, the foundational elements of a data platform, all from within Kubernetes, demonstrating a robust distributed services platform. This chapter focuses on the collection, extraction, movement, and processing of data, rounding out the majority of functionality required for any data-centric application. The ability to efficiently extract, transform, and load data from one diverse system into another is essential in harnessing the explosive growth of data from consumer and industrial IoT, social media, and digital transformation occurring in many organizations. The ability to quickly construct routes that move, transform, and process data is vital in leveraging the ever-advancing, data-driven trends such as Machine Learning based AI, technologies particularly hungry for large quantities of processed data. An effective data platform provides all the generalized mechanisms needed to extract, transform, and load data across data management systems and offers an application layer for supporting specialized processing and custom business logic.