Fast Data Pipeline Design: Updating Per-Event Decisions by Swapping Tables

VoltDB was one of the first companies to enable a new modern breed of applications, applications that combine streaming, or “fast data”, tightly with big data.We call these applications Fast Data Pipelines.

First, a quick high-level summary of the fast data pipeline architecture:

The first thing to notice is that there is a tight coupling of Fast and Big, although they are separate systems. They have to be, at least at scale. The database system designed to work with millions of event decisions per second is wholly different from the system designed to hold petabytes of data and generate Machine Learning (ML) models.

There are a number of critical requirements to get the most out of a fast data pipeline. These include the ability to:

• Ingest / interact with the data feed in real-time.
• Make decisions on each event in the feed in real time
• Provide visibility into fast-moving data with real-time analytics
• Seamlessly integrate into the systems designed to store Big Data
• Ability to deliver analytic results (mined “knowledge”) from the Big Data systems quickly to decision engine, closing the data loop. This mined knowledge can be used to inform per event decisions.

Hundreds of Fast Data Pipeline applications have been built and deployed using VoltDB as the fast operational database (the glue) between Fast and Big. These applications provide real-time decisioningengines in financial fraud detection, digital ad tech optimization, electric smart grid, mobile gaming and IoT industries, among others.

This blog is going to drill into how to implement a specific portion of this fast data pipeline, namely the last bullet: the ability to close the data loop, taking knowledge from a Big Data system and applying this knowledge, online, to the real-time decision engine (VoltDB).

Closing the Data Loop

“Per-event decisioning” means that an action is computed for each incoming event (each transaction).  Usually some set of facts informs the decision, often computed from historical data. These “facts” could be captured in machine learning models or consist of a set of generated rules to be executed on each incoming event. Or these facts could represented as rows in a database table, used to filter and generate optimized decisions for each event. This blog post will focus in on the latter, storing and updating facts represented in database tables.

When storing facts in database tables, each row corresponds to some bit of intelligence for a particular value or set of values.  For example, the facts might be a pricing table for airline flights, where each row corresponds to a route and service level.  Or the values might be list of demographic segmentation buckets (median income, marital status, etc) for browser cookies or device ids, used to serve up a demographic-specific ads.

Fact tables are application-specific, can be simple or sophisticated, and are often computed from an historical “big data” data set such as Spark, Hadoop, or commercial data warehouse, etc.  Fact tables can often be quite large and can be frequently recomputed, perhaps weekly, daily, or even hourly.

It is often important that the set of facts changes atomically.  In other words, if airline prices are changing for ten’s of thousands of flights, all the prices should change all at once, instantly. It is unacceptable that some transactions reference older prices and some newer prices during the period of time it takes to load millions of rows of new data.  This problem can be challenging when dealing with large fact tables as transactionally changing millions of values in can be a slow, blocking operation. Locking a table, thus blocking ongoing operations, is unacceptable when your application is processing hundreds of thousands of transactions per second.

VoltDB solves this challenge in a very simple and efficient manner.  VoltDB has the ability to transactionally swap tables in a single operation.  How this works is as follows:

1. Create an exact copy of your fact table schema, giving it a different name. Perhaps Facts_Table and Facts_Table_2.

2. Make sure the schemas are indeed identical (and neither is the source of a view).

3. While your application is running (and consulting rows in Facts_Table to make decisions), populate Facts_Table_2 with your new set of data that you wish future transactions to consult. This table can be populated as slowly (or as quickly) as you like, perhaps over the course of a day.

4. When your Facts_Table_2 is populated, and you are ready to make it “live” in your application, call the VoltDB System Procedure @SwapTables. This operation essentially switches the data for the table by swapping internal memory pointers. As such it executes in single to sub millisecond range.

5. At this point, all the data that was in Facts_Table_2 is now in Facts_Table, and the old data in Facts_Table now resides in Facts_Table_2.  You may consider truncating Facts_Table_2 in preparation for your next refresh of facts (and to reduce your memory footprint).

Let’s look at a contrived example using the VoltDB Voter sample application, a simple simulation of an ‘American Idol’ voting system. Let’s assume that each day you are going to feature different contestants for which callers can vote. Voting needs to occur 24x7, each day, with new contestants. The contestants change every day at midnight. We don’t want any downtime - no maintenance window, for example  - when changing our contestant list.

Here’s what we need to do to the Voter sample to effect this behavior:

1. First we create an exact copy of our CONTESTANTS table, calling it CONTESTANTS_2:

-- contestants_2 table holds the next day's contestants numbers -- (for voting) and names

CREATE TABLE contestants_2

(

 contestant_number integer     NOT NULL

, contestant_name   varchar(50) NOT NULL

, CONSTRAINT PK_contestants_2 PRIMARY KEY

 (

   contestant_number

 )

);

2. The schemas are identical, and this table is not the source of a materialized view.

3. The Voter application pre-loads the CONTESTANTS table at the start of benchmark with the following contestants:

1> select * from contestants;

CONTESTANT_NUMBER  CONTESTANT_NAME

------------------ ----------------

                1 Edwina Burnam   

                2 Tabatha Gehling

                3 Kelly Clauss    

                4 Jessie Alloway  

                5 Alana Bregman   

                6 Jessie Eichman  

$ cat contestants_2.csv

1, Tom Brady

2, Matt Ryan

3, Aaron Rodgers

4, Drew Brees

5, Andrew Luck

6, Kirk Cousins

$ csvloader contestants_2 -f contestants_2.csv

Read 6 rows from file and successfully inserted 6 rows (final)

Elapsed time: 0.905 seconds

$ sqlcmd

SQL Command :: localhost:21212

1> select * from contestants_2;

CONTESTANT_NUMBER  CONTESTANT_NAME

------------------ ----------------

                1 Tom Brady       

                2 Matt Ryan       

                3 Aaron Rodgers   

                4 Drew Brees      

                5 Andrew Luck     

                6 Kirk Cousins    

(Returned 6 rows in 0.01s)

4. Now that we have the new contestants (fact table) loaded and staged, when we’re ready (at midnight!) we’ll swap the two tables, making the new set of contestants immediately available for voting without interrupting the application. We’ll do this by calling the @SwapTables system procedure as follows:

$ sqlcmd

SQL Command :: localhost:21212

1> exec @SwapTables contestants_2 contestants;

modified_tuples

----------------

             12

(Returned 1 rows in 0.02s)

2> select * from contestants;

CONTESTANT_NUMBER  CONTESTANT_NAME

------------------ ----------------

                6 Kirk Cousins    

                5 Andrew Luck     

                4 Drew Brees      

                3 Aaron Rodgers   

                2 Matt Ryan       

                1 Tom Brady       

(Returned 6 rows in 0.01s)

5. Finally, we’ll truncate the CONTESTANTS_2 table, initializing it once again ready to be loaded with the next day’s contestants:

$ sqlcmd

SQL Command :: localhost:21212

1> truncate table contestants_2;

(Returned 6 rows in 0.03s)

2> select * from contestants_2;

CONTESTANT_NUMBER  CONTESTANT_NAME

------------------ ----------------

(Returned 0 rows in 0.00s)

Note that steps 3-5, loading, swapping, and truncating the new fact table, can all be done in an automated fashion, not manually as I have demonstrated with this simple example.

Running the Voter sample and arbitrarily invoking @SwapTables during the middle of the run yielded the following results:

A total of 15,294,976 votes were received during the benchmark...

- 15,142,056 Accepted

-   152,857 Rejected (Invalid Contestant)

-        63 Rejected (Maximum Vote Count Reached)

-         0 Failed (Transaction Error)

Contestant Name Votes Received

Tom Brady      4,472,147

Kirk Cousins 3,036,647

Andrew Luck      2,193,442

Matt Ryan      1,986,615

Drew Brees      1,963,903

Aaron Rodgers 1,937,391

The Winner is: Tom Brady

Apologies to those not New England-based! As you might have guessed, VoltDB’s headquarters are based just outside of Boston, Massachusetts.

Just the Facts, Ma’am

Leveraging big data intelligence to make per-event decisions is an important component of a real-time decision engine within your data pipeline. When building fast data pipeline applications using VoltDB, VoltDB provides tools and functionality to make this process easy and also painless to a running application. Two key tasks need to be performed: loading your new fact table into VoltDB, and atomically making that new data “live” to your business logic.

Loading data into VoltDB from an external data source can be done easily via a couple of approaches: you can use one of our loaders such as the CSV, Kafka or JDBC loader; or you can write an application to insert the data.

Swapping tables in VoltDB is a trivial exercise with the @SwapTable system procedure. And most importantly, swapping in new fact table data does not impact ongoing stream processing.

Scrum: It’s a full-time job.

“An adequate ScrumMaster can handle two or three teams at a time...A great ScrumMaster can handle one team at a time.”

-Michael James

One of the lessons we learned early in our Scrum roll-out was that the Product Owner and Scrum Master roles required much more involvement than originally thought.  Coming from a Waterfall development environment, where releases were long in duration and milestones spread out, people in the organization were used to having key roles on multiple projects, context switching as needed.   Juggling responsibilities in a Waterfall environment seemed to work well, or at least acceptable.  Our Product Managers, for example, could own several product lines, including associated customer meetings and field interactions, and easily satisfy occasional project touch-points such as requirements definition (at the beginning of the release) and Beta and FCS duties (at the end of the release).  These project touch-points required their full-time involvement periodically, a few times a year, making it relatively easy to context-switch periodically and satisfy the needs of each effort.



With Scrum, however, things turned out a bit different.  Though we “took the training” and “read the literature” which repeatedly said the roles of Scrum Master and Product Owner were full time jobs, we weren’t true believers.  Our “muscle memory” way of working had everyone juggling multiple roles and responsibilities, it was the way we were used to working. During the course of our Agile Pilot projects, however, we got convinced pretty quickly that Scrum roles could not be part-time.

Scrum is more intensive than waterfall development.   

With frequent “potentially shippable product increments” created every 2-4 weeks,  there’s essentially a full product release at least once a month if not twice.   During our Agile pilots, our Product Managers, filling the role of Product Owners, soon realized that creating and maintaining a Product Backlog required a significant amount of time.  Every few weeks a new set of fully-defined “product requirements” need to be ready for the team for the next Sprint. Product Owners found it challenging to satisfying their Scrum PO role and concurrently complete other project work.  Our Scrum Masters, too, found themselves booked full-time, facilitating team interactions, producing Sprint Burn Down charts, working impediments, and delivering frequent product releases at regular Sprint Review.  Team members, often with other-project responsibilities, found themselves spending more time delivering on Sprint Goals, often at the expense of their other responsibilities.  

As we scale Scrum within the organization, we’ve (obviously!) needed to identify new Scrum Masters and Product Owners.  We’ve held fast to the one project per Scrum Master recommendation. Identifying new dedicated Scrum Masters has been fairly easy - our Pilot teams was that the efforts produced several Scrum Master candidates, a nice side-benefit!   Finding full-time Product Owners for each team was more challenging.  Because we’re applying Scrum to large multi-team projects, our mapping of Product Manager positions to Product Owners simply didn’t scale.    Our solution was to look to our lead engineers. Most of our new teams have technical Product Owners, usually Software Architects, and they are dedicated full-time to the team.  Product Managers are involved at either a co-Product Owner (in an outward facing role) or at a higher level in the project effort, via a Product Council-type role that involves them as stakeholders.  

I expect our organization to fine tune (via inspect and adapt) these non-Scrum roles as we mature in our Agility.  But regardless of the organizational “noise” around the Scrum Teams, I believe that participation at the Scrum Team-level will continue to require full-time effort from all members in order to be successful.

Why Agile?

We’re at what I perceive to be a major “transition point” in our Agile transformation.  Most of our Agile pilot teams and projects have successfully completed and we are rolling out internal Scrum training throughout the Product Group.  We’re on the cusp of taking a big step forward with Scrum adoption within our 500 person organization.  It’s at transition points like this when I find myself reevaluating our direction: “Why are we doing this? Are we ready for this next big step?”   Our existing processes do, in fact, lead to shipping product and generating revenue, so: “Why Agile?”

Our original goal, defined we began this transformation, still rings true:

The goal is not to release things faster for the sake of it. It is to achieve nimbleness and agility to deliver on the company’s strategy as it moves forward and/or tactically changes.   It is about creating more business opportunities for ourselves rather than being cased into a model that requires our resource commitment for a time window that is longer than what the market we’re competing in requires.

Specifically, the software development group’s reason for moving from Waterfall to Agile is to achieve nimbleness and agility to deliver on the company’s strategy as it moves forward and/or tactically changes.  We want the ability to efficiently deliver strategic products to market in a timely manner.

Today we have challenges delivering on this goal.  As a mature software organizations our development practices have been hardened over many years of delivering software to tens of thousands of customers.   Our processes have been tempered with the successes (and often, the failures) of the past.    Because of this, it’s difficult to change the current way of doing things.  This is an understandable, and completely human, reaction - but it can dangerous over the long-term.  Fortifying the walls, via policies and procedures as well as organizational divisions, has the result of business being less reactive to change, less agile, which ultimately impacts corporate revenue.   New software releases often take longer to get out the door and may not fully meet customer needs.  Entering new markets in a timely manner becomes difficult, allowing younger more nimble companies a market advantage.   It is for these reasons that we’ve embarked on our development process transformation.

As part of our migration from Waterfall to Agile software development, we need to build a new way of working, including a new corporate culture: one that is nimble and responsive to ever changing business conditions. A culture that values responding to change over following a plan.  One that regularly inspects and adapts, and becomes better over time.  Scrum, with its “inspect and adapt” framework,  is one of the catalysts that can greatly assist us in this journey.

"Scrum is Like Chess"

One of my favorite Scrum quotes is “Scrum is like chess.”  

Like chess, the “rules” for Scrum are fairly straight forward. But also like chess, Scrum practitioners require continued practice and ongoing learning to be come good.




So why do it?  If it is hard to master, why change our development process?

Ken Schwaber, a co-author of the Scrum Guide, sums it up best:  “Those who play both games and keep practicing may become very good at playing the games. In the case of chess, they may become Grand Masters. In the case of Scrum, they may become outstanding development organizations, cherished by their customers, loved by their users, and feared by their competitors.”

Being loved by users and feared by competitors would mean that we’re delivering highly desired software products which are generating significant sales growth and, ultimately, more corporate value.  This is the state we, as a company, want to achieve, and it is one of the drivers for our transition from Waterfall to Agile.

So what can you do to become better at Scrum?   

Being part of a Scrum team and new to Scrum, you are likely devoted near-full time on learning new ways of working as well as actually delivering potentially shippable product every Sprint.  But this shouldn’t stop you from allocating some time to digging further into the nuances of Agile software development practices, communication and organizational behavior.

There’s a wealth of information, as well as tons of opinions from many of the great Agile industry experts, on two public Internet groups.  The first, and older, is the Yahoo! Group, Scrum Development (http://groups.yahoo.com/group/scrumdevelopment/).  The second, a Google Group called Agile Leaders (http://groups.google.com/group/agile-leaders?pli=1) was recently created and also features some lively discussions, often by the same cross-section of folks on the Scrum Development list.  By just becoming an observer to these lists you can glean many useful tidbits of information on all aspects of Scrum as well as Agile software development. While these are great resources for every Scrum team member, aspiring ScrumMasters and Product Owners should take particular note of these two valuable resources.

There’s also numerous blogs and webinars that can give you insight to different facets of Scrum.  I’d start there before buying too many books (Amazon lists over 300 books for “Scrum” and over 1300 for “Agile”).  There are many formal and informal Agile and Scrum gatherings, such as Agile New England (http://www.agilenewengland.org) that regularly host events and meet ups.  

And finally, as with any discipline, nothing beats practice coupled with an ongoing desire to learn.

Enjoy the journey!