Are you caught up with the real-time (r)evolution?

This post originally appeared on InfoWorld on December 1, 2017.

Are you caught up with the real-time (r)evolution?

Real-time analytics have been around since well before computers. Here's where the evolution of real time is headed in 2018

People often think of “real time” as a new concept, but the need for real-time notification and decisioning has been around forever. Gartner defines real-time analytics as “the discipline that applies logic and mathematics to data to provide insights for making better decisions quickly.” This definition has always held true, it is just the meaning and measurement of real time that has evolved. As we approach the end of 2017, it’s a good time to look at the evolution of real-time decision making and ask yourself—are you caught up?

Real time B.C. (before computers)

Before there were computers, people were finding ways to use real-time information. For instance, in 500 BC, Philippides ran 25 miles from Marathon to Athens, Greece, to alert the magistrates of the victory at the Battle at Marathon, and in 200 BC, smoke signals were used on the Great Wall of China to warn those further down the wall of enemy size and attack direction.

Fast-forward to 1775 when Paul Revere developed a real-time alerting system using lanterns. We all know the famous phrase “one if by land, two if by sea,” and after initiating this real-time alert, he rode on horseback to deliver the information to towns around Boston along with 40 other horseback riders to spread the word throughout eastern Massachusetts.

Real-time A.D. (after digital)

With the invention of computers, we upgraded from “human speed” to “machine speed.” For example, telephone call routing was initially executed by human switchboard operators, but by 1983, computers automated this process, enabling real-time voice communication for almost everyone.

People also began interacting with real-time systems, typically through an intermediary. The stock market is a good example—if you wanted to buy or sell securities, you had to make a phone call to your stockbroker who would execute the trade on your behalf using the real-time stock exchange trading software.

From an organizational standpoint, computers enabled companies to gather transactions into a database and produce monthly, weekly or even daily reports. At this early stage, nightly batch processing of data was generally viewed as “real-time data.”

Today in real time

Every day most of us interact with real-time systems and execute real-time decisions, possibly without even realizing it. For instance, going back to our stock example, trades can be executed in seconds by the consumer without a broker.

Take credit card fraud as another example. What was previously fraud detection has evolved into fraud prevention thanks to advancements in real-time technology, and preventing the unauthorized transaction from being processed can save corporations millions of dollars.

Smartphones have helped advance the capabilities of real-time processing thanks to the personalized data they can ingest. Tracking the location of consumers in real time allows companies to deliver personalized offers based on what they are close to. Mobile traffic applications like Waze provide real-time incident alerts and traffic updates to redirect drivers to the fastest route.

Tomorrow in real time

We are witnessing the commoditization of real-time information, alerts, and decisions, and these capabilities will proliferate across all application domains, including online gaming, casino gaming, financial compliance and risk, fraud prevention, digital ad technology, telecom policy and billing, and more.

As we look ahead to 2018, we’ll see real-time advancements in the following areas:

• Self-driving cars will process data from countless sensors and ultimately get passengers to their destinations safely and efficiently, saving thousands of lives a year in the process.
• Real-time threat detection and remedy for data security breaches and DDoS attacks will alert consumers as they occur.
• Wearable health sensors (they already exist in your smartwatch) will not only monitor health statistics, but will also combine that telemetry with personal medical history to know when and how to notify doctors if necessary.
• Augmented reality will become part of our everyday lives. When traveling in a foreign country, AR on cell phones and tablets will provide “live” translations of street signs and shops and integrate reviews, recommendations and advertisements on whatever signposts or buildings you scan.

The value of real time is the ability to make decisions as soon as possible based on the most readily available information. Information that used to take days to receive can now be ingested and analyzed in fractions of a second, and 2018 will only cut that time down even more, empowering companies to leverage real-time information to increase corporate profit, protect resources and even save lives.

 

Streaming Data: Billions and Billions (of transactions)

 This post originally appeared on VoltDB.com in October, 2016.

Billions and billions (of transactions)

Billions and billions^[1] of transactions are executed each and every day by production-deployed VoltDB applications.

A back-of-the-napkin calculation puts the number north of 50 billion a day. The number is likely much, much higher. These billions of transactions are executed across a variety of innovative applications, applications not practical, or sometimes even possible, to build a decade ago. Chances are that you have touched VoltDB – without knowing it – today!

• When you call someone on your cell, there’s a good chance a VoltDB application is approving the call.
• If you have played some of the top 10 mobile games within the past few years, a VoltDB application likely processed your in-game events.
• If you’ve run a road race recently, there’s a good chance a VoltDB application tracked your progress.
• If you live in the UK and turn on a light switch, you are interacting with VoltDB as your electrical smart grid uses VoltDB to authorize and process grid and meter events.
• If you play baccarat in a casino, there’s a good chance the bets placed and movement of chips on each gaming table is validated by VoltDB via in-chip RFID technology.
• If you bought something in China with your credit card, it is likely that your transaction was verified as valid or detected as fraudulent by a VoltDB application.

The VoltDB engineering team has been building the world’s fastest in-memory database for over eight years now. It’s been a fun as well as challenging journey and we’ve learned a lot in the process. Some of the more notable learnings have been:

Batch is so 2000s. Legacy applications are moving from batch processing of feeds to real-time processing. New applications are being built to ingest, transact, and compute analytics on data streaming from all types of sources, such as IoT sensors, mobile devices, logs, etc. The world is moving to real-time and turning to products like VoltDB, which has the capability to handle high throughput (thousands to millions of events per second).

SQL [still] rules the roost.  VoltDB was founded around the same time as the NoSQL movement. At the time, building a distributed SQL data store seemed contrarian, given the NoSQL hype. Fast forward a half-dozen years and we now see many of the more popular NoSQL products offering SQL or SQL-like interfaces. Even the Hadoop ecosystem has added SQL interfaces to the data lake. SQL is here to stay(and, in fact, never left).

VoltDB’s commitment to SQL remains strong. Over the past year we’ve added windowing analytic SQL, geospatial SQL support (points and polygons), continuous queries via real-time materialized views, the ability to join streams of data to compute real time analytics, and the ability to very quickly generate an approximate count of the number of distinct values. More advanced SQL capabilities are made available nearly every month.

Expect the unexpected, operationally. When you build a highly-available, clustered database that runs 24×7 at high throughput, on bare metal or in the cloud, all types of error scenarios that could happen, dohappen. Disks fail, networks break down, power goes out, mistakes are made, lightning strikes.

Building a production-ready operational database is tough work. The bar is set high and not negotiable. The following features are table stakes for current and next generation applications:

• High availability: Machines, containers and virtual machines can and do break, crash or die. Networks pause or partition. The database must handle these situations properly.
• Correctness: Accuracy matters. Every client should see a consistent view of the data.
• Predictable latency: Response time should be predictable, within 99.999% of the time.
• Durability: Never lose data even in the face of nodes crashing or networks partitioning.
• Performance: The database needs to scale with your application. Adding additional nodes should scale throughput smoothly and linearly.

To maintain these requirements across all scenarios, VoltDB Engineering constantly pounds on the database in our QA lab. We focus on randomizing errors of all types. The more we learn, the more we feel we need to test more. That’s one of the reasons we reached out to Kyle Kingsbury, (aka Aphyr), to independently apply his dastardly Jepsen test suite and methodology to VoltDB. As part of this effort, we found a few issues in the product before they caused issues in customer deployments. This effort remains one of the team’s proudest accomplishments of 2016.

But we know there’s always more work to do, and we’re committed to doing it.

Streaming analytics and operational data stores are merging into a new real-time platform

The architecture required for real-time processing of streaming data is evolving rapidly. When we started building VoltDB, a developer had to cobble together numerous pieces of technology, (e.g. ZooKeeper, Kafka, Storm, Cassandra), to process real-time streams of data.

VoltDB Founding Engineer John Hugg stated at a recent Facebook @Scale conference, “OLTP at scale looks a lot like a streaming application.”  Today we’re seeing the result of that evolution: a convergence of real-time ingestion, streaming analytics, and operational interaction into a single platform. While creating the next generation of database, the VoltDB engineering team has assembled the core components for building streaming fast data applications in VoltDB:

• Real-time ingestion with in-process, highly-available importers reading from Kafka, Kinesis and other streaming sources;
• Streaming analytics on live data via ad hoc SQL analytics as well as continuous queries; and
• A blazingly fast in-memory storage and ACID transaction execution engine capable of accessing hot and historical data to make per-event decisions millions of times per second.

The VoltDB engineering team is not done building (or learning)! Our mission is to make building highly-reliable, high throughput fast data applications not only possible, as it is today, but also extremely easy. So stay tuned. Over the coming months we’ll be delivering additional geo-distributed and high availability features, additional importers and exporters for building fast data pipeline applications, and improving our real-time analytical SQL capabilities.

^[1] With a hat tip to Johnny Carson and Carl Sagan

In-Memory Database Sizing

 This post originally appeared on VoltDB.com in June, 2014.  

In-Memory Database Sizing

Sizing an in-memory database does not follow conventional database sizing rules.

For traditional databases, you buy a decent server machine, likely one with many CPU cores and reasonable memory, and then focus on application IOPS (I/O Operations per Second).  If you are really going to stress the database, you must choose disks that can support the I/O needs of your application, today and in the future. Because these systems often use many disks to achieve high I/O performance, capacity is usually an afterthought.

With in-memory databases, throw out everything you know about sizing databases. Performance is now the afterthought, but capacity must be carefully considered.  

To size your in-memory clustered database application you must consider the following factors:

●      First, compute the size of the data your application expects to host.  This means computing the size of all rows in all tables.  And also computing the size of all table indexes. VoltDB provides an easy-to-use interactive sizing tool in our application catalog report, described later in this post.

●      If you want a highly available database cluster, all data must be stored redundantly. You’ll need to multiply your logical data size by the replication factor to understand your memory requirements. For example, if you have 100GB of table and index data, and you want your database to be able to withstand losing one of the nodes in the cluster, you will want each datum to be stored twice, and on two different physical machines. This means that you need to size the data requirement of your database to be 200GB (100GB of data X 2 copies of the data).

●      Allow for some extra memory, perhaps an additional 25%, for database and operating system overhead. VoltDB uses extra memory for storing intermediate results, maintaining an undo log, caching SQL plans and buffering network data. Note that on a machine with 4GB, you’ll want to leave a larger fraction of memory free than on a machine with 256GB.

This tallied memory total can now help you decide how many machines your system requires.  Note at no time did we need to determine how much CPU we required, or how many IOPS were needed.  Because you are sizing your database based on data, memory size is the most important calculation you will likely use. Generally speaking, in-memory databases like VoltDB operate so fast that CPU utilization is not a problem - there is plenty of headroom for growth. As for disk I/O, most in-memory systems use sequential logs on disk, eschewing costly random I/O. With this reduced dependency on disk performance, VoltDB can achieve tremendous throughput, even on commodity spinning disks.

Often, you’ll be faced with a choice between many machines with less memory per machine or a small number of high-memory machines. There are lots of reasons to lean one way or another. For one, you should try to run with at least three nodes if availability or redundancy is a concern. Additionally, some administrator operations run faster with less per-machine data. At some point however, too many nodes can be a management hassle. Most VoltDB users run with 3 to 30 nodes per cluster for these reasons.

Planning for Data Capacity and Growth

Sizing VoltDB databases is fairly straight-forward.  Once you have settled on your application’s database schema, point your web browser to the database Catalog Report and choose the “Size Worksheet”, found at this URL: http://localhost:8080/#z (note the database must be running).

 

This sizing worksheet is automatically built from your application’s schema. It allows you to enter your expected number of rows for each table and then computes the amount of memory required for the database cluster as a whole.

Note that the footprint of your database is the size of the number of rows as well as the size of all indexes and materialized views. You must also leave additional memory for the operating system.  Note that this assumes that VoltDB is the only active application on these machines.  If you plan on sharing the machines with other applications, something we do not recommend, your memory requirements will naturally be larger.

Expanding your Database Capacity

A well behaved IT environment will monitor the performance and capacity of the applications running the business.

Because VoltDB processes transactions incredibly fast, it is often the case that there is plenty of transaction throughput headroom and there are lots of extra CPU cycles available. Monitoring the memory usage of the database cluster becomes much more important. VoltDB provides monitoring interfaces, as well as Nagios and New Relic plug-ins that make monitoring memory usage easy. These tools allow you to define capacity alerts, much the way you would define disk space usage alerts for legacy database systems. Should your in-memory database memory usage grow to a high level, signaling a capacity limit alert, it may be time to add more storage capacity to your database.

With VoltDB, adding capacity is easy: simply add more nodes.  VoltDB supports adding nodes to a running cluster without interrupting ongoing operations or sacrificing ACID transactional guarantees.

If you do not wish to add more nodes to your database cluster, doing a rolling memory upgrade of machines in a redundant VoltDB cluster is an alternative. Since VoltDB uses active replication within a cluster, removing and replacing a node can be done transparently to a user’s application with no data loss and often unnoticeable performance impact.

In-Memory Database provisioning… A New Way of Thinking

If you are provisioning and rolling out a new application using an in-memory database you will have to shift your thinking about how you size your deployment. With all data residing in memory you will need to provision enough memory in your database cluster to hold all of your data, and a bit more memory for regular operating system operations.

Disks are only used for durability and most in-memory systems like VoltDB work great with commodity hardware, disks included. Sizing for IOPS is a thing of the past!