Four Real World Use Cases For An In-Memory Data Grid

February 18, 2016 Danielle Burrow



Big Data demands a technical architecture that can handle significant data flow, continuous availability and consistency, with flexible scalability across various data management systems and locations. Apache Geode (incubating) is based on over ten years of development work in some of the most demanding application environments as commercial software, Pivotal GemFire.

To celebrate Apache Geode’s 1.0 milestone release last week, this article will examine ways in-memory data grids can be leveraged to transform your customers’ experience, your decision-making capabilities, and your bottom line. Specifically, we’ve highlighted four use cases demonstrating how some of our enterprise customers use GemFire to support high-performance, real-time applications, without compromising speed or safety of valuable data.

Transaction Processing

Online shopping. Connected devices. Securities trading. These types of transactional applications demand extremely high-performance data management. And with more companies undergoing digital transformations, these types of applications are only increasing. Users (or semi-autonomous devices) won’t or can’t wait for slow transactions. In these cases, an in-memory data grid that supports high data consistency is a must.

Pivotal GemFire supports extremely large numbers of concurrent transactions involving terabytes of operational data, providing a faster, more reliable transactional experience for customers. It is also a more scalable data management system than traditional relational database management systems, able to elastically scale as demand increases, so the first user and the 10,000th user will process their requests with the same lightning speed. While speedy and secure transactions benefit every business, they are crucial for those hosting online shopping carts, payment processing, and financial trading where every second of wait time erodes the number of completed transactions, and depletes the bottom line. GemFire provides the lightening speed and flexible scalability that boosts customer retention and engagement by supporting fast transactions.

One company relying on GemFire for transaction processing is the world’s second largest railway network. Indian Railways carries over 23 million passengers each day on more than 12,000 trains. Using its legacy system, Indian Railways could handle no more than 40,000 concurrent internet users, many of whom spent up to 30 minutes trying to book tickets online. By switching to GemFire to support its e-ticketing application, the system can now handle more than 120,000 concurrent users. Completing a booking now takes customers mere seconds. GemFire helped transform the experience for millions of Indian Railway’s customers and has boosted the company’s daily revenue by INR$600 million.

Event Notification and Processing

Credit card fraud. Risk calculation. Industrial fleet tracking. Data tells the story of what is happening with your business on the ground, right now. With the Internet of Things as a potent data source across industries, the opportunity to take advantage of hot data is greater than ever. Yet, traditional data management systems can’t process big data fast enough to notify client applications of important events as they occur. Nor can a traditional system combine historical and live data for real-time analyses and predictions. But an in-memory data grid in conjunction with massively parallel processing data stores, like Pivotal Greenplum or the open source Greenplum Database, can process live and historical data sets quickly enough to make this kind of analysis possible. GemFire’s event notification and processing capabilities are used by industry leaders in banking, energy, telecom and more to track and respond to mission critical processes in real-time.

GemFire’s powerful in-memory data grid can manage tremendous amounts of incoming data (terabytes of data in-memory) and push notifications to client applications when changes occur within the server, across multiple clusters. This fast, continuous querying capability allows systems to quickly and easily access large amounts of incoming data and take action.

Industrial giant GE uses GemFire’s in-memory event processing and notification capabilities to track its fleet of gas turbines. GE Power and Water’s Remote Monitoring and Diagnostics Center is now able to process and store large streams of high velocity time series sensor data from turbines (more than 100,000 points per second, 10 terabytes in-memory). GemFire’s continuous query capability allows GE to closely monitor various sensor data for turbine failure signals. Furthermore, GE has implemented fast analytics that incorporates both live and historical data to make predictions on turbine maintenance needs to prevent equipment failure. Utilizing event processing and enabling analysis that incorporate live and historical data has streamlined the way GE monitors and maintains its fleet in ways that weren’t possible with its legacy system.

Highly Available, Distributed Caching

Still waiting for that product search page to load? Every. Second. Counts.

Data-driven applications that take too long to load are cumbersome and frustrating to use. And most users won’t wait more than three seconds for a page to load before giving up. GemFire’s in-memory data grid can serve as a caching layer for your existing applications, providing fast recall of frequently accessed data. Applications won’t get bogged down trying to access data on disk. And what’s more, GemFire is able to serve requests despite server outages and across WAN connections. This allows for continuous availability with very low-latency, supporting sophisticated caching solutions ranging from web session caching to mainframe offloading.

GemFire’s in-memory caching is utilized by a national business services provider for applications that require fast and frequent access to large subsets of data. Human resource and finance personnel using these applications to search and filter employee information need fast and reliable access to various data subsets, like employee location or pay grade. Generating this information using a search form to query the database is a slow and cumbersome user experience. To meet the demand for high-volume, low-latency data recall, a sophisticated local caching solution was created directly within the application, continuously updated by GemFire. This in-memory, local caching system functions similarly to a single page web application, enabling users to search and filter large amounts of data in a fraction of a second for a smooth and efficient user experience.

What’s worse: Waiting for a page to load or finding that all the information you typed into a form is gone? In another example of in-memory caching, a large insurance company faced high customer abandonment rates with a legacy system that was unable to save partially completed forms. GemFire was implemented to address this issue, caching customer information across three data centers in North America. Now customers can return to an application form without any data loss, dramatically increasing the number of insurance applications submitted online.

Compute/Data Grid

Monte Carlo simulations. Portfolio projections. Price and index performance comparisons. The financial services sector is a hot bed of Big Data use cases for GemFire’s in-memory data grid. Storing, processing and analyzing financial data using traditional database architecture requires moving massive amounts of data between separate systems. The legacy systems of many financial services companies struggle to keep up with the ever increasing amount of data they need to support their customers. GemFire’s ability to process very large data streams in-memory, without transferring the data to a separate analytics system, allows for fast access and rich data analysis.

A large financial services firm uses GemFire to manage terabytes of data in-memory to process results quickly on a single platform. The firm’s customers are able to quickly view compelling portfolio metrics that incorporate varied data sources, such as price data and index performance as points of comparison. Traditional database architecture requires an extra step of processing to provide these metrics which simply takes too long. GemFire’s in-memory data grid enables intense calculations without moving data between separate databases for analysis. This delivers analytics at speed and at scale for fund managers, investors and other platform users—instantly.

Let’s review!

Fast, high-volume concurrent transactions. Automatic event notifications in real-time. Intense, dynamic calculations of big data in milliseconds. In-memory caching for exceptionally fast data delivery, even across distributed WAN. These are some of the ways in-memory grids, like Geode and GemFire, have served as an essential, architectural component for transforming the way businesses use their data to do business.

Join Us for the Inaugural Geode Summit

Don’t miss the very first Geode Summit, March 9 in Palo Alto, CA. This event will bring together the incubating community of Apache Geode and GemFire users and developers. Meet with experts, core committers, and leading production users of Geode and GemFire. Learn how companies are using Geode and GemFire for low-latency, high concurrency data management and transaction processing in a variety of applications. Join us to see real-world uses cases, acquire in-depth knowledge of the technology, and help guide the future of Geode.
Register Now!

About the Author

Danielle Burrow

Danielle Burrow leads product marketing for VMware Tanzu Labs, the software consulting services branch of VMware Tanzu. Prior to VMware, Danielle worked in product marketing at Pivotal Software and in sales and digital marketing at Google. Danielle also has a background in the non-profit and healthcare sectors. Danielle has a BA in Art History from UCLA and an MA in Counseling Psychology from Santa Clara University.

Follow on Twitter More Content by Danielle Burrow
Preparing for RailsConf Component-Based Applications Workshop
Preparing for RailsConf Component-Based Applications Workshop

At this year’s RailsConf I am going to be teaching the workshop: Get started with Component-Based Rails app...

How To Write About PaaS and DevOps Without Once Typing “PaaS”
How To Write About PaaS and DevOps Without Once Typing “PaaS”

Many of us here at Pivotal have been eagerly contributing to the ongoing conversations around DevOps. What’...

SpringOne at VMware Explore 2023

Learn More