Rethinking RDBMS Data Migration

June 23, 2017

Rethinking RDBMS Data Migration: Modernizing Traditional ETL Processes to Cloud-Native Event Driven Microservices [I] - Anupama Pradhan, Health Care Service Corporation & Jeff Cherng, Pivotal Extract, transform, load (ETL) has always been complex and expensive for moving massive data sets from one data source to another. This is especially true if the source system is a traditional RDBMS with complicated relationships between tables. Most of the time, traditional ETL processes are implemented with batch, monolithic, and tightly coupled approaches. As the result, traditional ETL processes are often considered fragile, hard to maintain, not easy to tune, and often introduce high data latency between source and destination systems. In this session, Anupama Pradhan and Jeff Cherng will cover how to modernize traditional RDBMS ETL processes to cloud-native event driven microservices pipelines by using Cloud Foundry, Spring Cloud Stream, and RabbitMQ/Kafka. The pipelines can handle high volume data sets and complex database queries, yet with low data latency between the source RDBMS and destination data store. In addition, the design is highly tunable and scalable. The session will also cover analysis of performance metrics based on implementations of real world use cases. Jeff Cherng Pivotal Software Inc. As a member of the Pivotal Data Engineering team, Jeff Cherng is responsible to work with customers to drive technical innovations on high performance data solutions. Jeff is very passionate about Spring Framework/Cloud Foundry and has worked on many distributed systems projects over the course of his career. His current focus is working on integrated solutions/patterns with Cloud Foundry, Spring, and Apache Geode (In Memory Data Grid). Anupama Pradhan Health Care Service Corporation Senior Technology Architect Anupama Pradhan is a Senior Technology Architect at HCSC with more than 20 years of experience in software engineering. As part of Enterprise Architecture team, she helps build reference implementations for various technologies and establish architecture patterns in the enterprise. Her current focus is Microservices on Cloudfoundry and monolith migration to Microservices Architecture.

Building Security Frameworks Into Your CI/CD Pipelines

Building Security Frameworks Into Your CI/CD Pipelines [I] - Nathan Gibson & David Brock, Allstate Insuranc...

Next Video

Core Project Updates Track Introduction

Core Project Updates Track Introduction - Colin Humphreys, Pivotal & Julian Friedman, IBM Julian Fried...

Rethinking RDBMS Data Migration

Previous

Next Video

Most Recent

Updated video of pushing the Spring Music Java application to Cloud Foundry provided by Tanzu Application Service. We push an app, bind it to a database, and scale it up in just a few minutes.

Streams, batches, and microservices. Oh My! Data integration and related workloads are at the foundation of all enterprise workloads. They ingest streams of events. They process large batches. How do

Streams, batches, and microservices. Oh My! Data integration and related workloads are at the foundation of all enterprise workloads. They ingest streams of events. They process large batches. How do

Day 2: Spring Tips: Configuration Hi, Spring fans! Spring is an inversion-of-control container. Inversion of control insulates objects from the knowledge of how a collaborating object on which they d

Hi, Spring fans! Spring is an inversion-of-control container. Inversion of control insulates objects from the knowledge of how a collaborating object on which they depend are created. It also insulate

Learn about the framework that enables enterprises to become wildly successful with Kubernetes based platforms in partnership with VMware Pivotal Labs. Learn more at tanzu.vmware.com/application-moder

With Kubernetes, everything from deployment to monitoring to scaling is more standardized and inherently easier to automate. This presents the possibility to achieve a more effective and comprehensive

With Kubernetes, everything from deployment to monitoring to scaling is more standardized and inherently easier to automate. This presents the possibility to achieve a more effective and comprehensive

Christopher Livermore, Head of Site Reliability Engineering, at Hive tells how Tanzu Observability by Wavefront helps Hive measure and deliver reliability, security, and efficiency for their customers

They call RabbitMQ the swiss army knife of messaging systems, but what impact do all those features have on performance? In this talk we'll review common features and their effect on end-to-end laten

Today, Bloomberg's many engineering teams around the globe have the luxury of access to a fully-managed RabbitMQ platform. This enables them to achieve scalability, flexibility, and maintainability,

It's time to talk about the current state and future for the most widely deployed open source message broker in the world. Selected guests takes part in a panel debate, including questions from the au

In order to detect anomalies and prevent runtime issues that can compromise the delivery of business services that depend on RabbitMQ, effective monitoring, alerting and analytics tools can make a big

CloudAMQP provides RabbitMQ clusters as a service, but what happens behind the scenes? CloudAMQP is powered by a number of RabbitMQ servers that provide various functionality in setting up, configuri

RabbitMQ is a multi-protocol messaging broker, which, on a vanilla installation supports AMQP-0.9-1 (and AMQP-0.8). Through its plugin architecture, RabbitMQ may also be configured to act as a messagi

In this session we explore a variety of modern event processing and streaming solutions built on top of RabbitMQ. We will demonstrate how Spring Cloud Streams and RabbitMQ provide abstractions for dev

Almost every introduction to publish-subscribe technology embraces a model where different message types appear on correspondingly distinct topics. Unfortunately this introduces a rather fundamental f