Top 5 Ways to Better Use Your Data

This post originally appeared on VoltDB.com in May, 2016. 

 Top 5 Ways to Better Use Your Data

Sir Francis Bacon is said to have coined the phrase “scientia potentia est”, translated as ‘knowledge is power’. Four hundred years later, we might rewrite the phrase as “data potentia est”.  Data is power - so how can you better use your organization’s data?  Here are five suggestions to help you drive value from your organization’s data - quickly.

1)    Know what you have.

First and foremost, inventory your data. 

There are two types of data to identify. First is historical data. Historical data is data you’ve accumulated over years of doing business. This could include databases, files, spreadsheets, presentations, transactions, logs, etc. The second type of data is the data that is being created “right now” - this is real-time data. Real-time data potentially has immediate value, and then ultimately turns into historical data.

Catalog and prioritize the data you have. Ideally, you will also want to identify the sources of data. Knowing how data is created allows you to capture it, store it, and eventually extract value from it, at the least cost to the organization, and with the best ROI.

The value of each type of data is different. Historical data allows you to analyze and mine past events. Real-time data gives you the opportunity to calculate analytics, possibly compare them to historical trends, and perform business actions in real-time, to capture additional and immediate value.

By way of example, consider a fraud prevention offering. Fraudulent transaction patterns are mined from historical data. These historical patterns are applied to real-time transactions to identify and reject suspected fraudulent transactions.

2)    Architect a data strategy that handles both Big and Fast data.

Creating a historical archive, perhaps a data lake via a Hadoop cluster, to store your data is only one step. Today enterprises create data at a tremendous - and growing - rate.  Processing and ingesting data in batch mode overnight is no longer acceptable. Real-time responsive enterprises need to process and react to data in seconds to minutes. Many organizations, including mobile operators, telecom providers, financial services organizations and advertising technology providers must respond in milliseconds.

3)    Choose the appropriate technologies.

There are a plethora of big data tools, and most are designed around best practices, optimized to extract value from both historical and real-time data.

Minimally you will need technologies for these areas:

Big Data: Typically, the main data management platform for big data is Hadoop or a data warehouse or perhaps a combination of the two to handle both structured and unstructured data. They act as the repository for all your data, often called the “data lake”. The data lake stores historical data to be analyzed and mined. 

Fast Data: Data is being created at a dizzying rate every day. Fast data is data that is being created now and is streaming into your company now.  It could be user clicks on your corporate web page or product downloads or any operational event occurring in your organization.  To deliver this fast data to the systems that can act on it, consider a message queueing systems such as Kafka.  To eliminate batch processing (slow data!) this message queue needs to deliver event data to an operational data stores capable of handling and processing messages at web-scale speed, thousands to tens of thousands to even millions of events per second.  The operational data store’s role is to ingest the data and process it in real-time.  Real-time processing can include computing real-time analytics, such as counts, aggregations and leaderboards, issuing real-time alerts, deduping, enriching and aggregating events, and making transactional decisions on an event-by-event basis. Both NewSQL and NoSQL operational stores can provide horsepower for handling real-time processing of event streams.  Modern operational data stores range from strongly-consistent SQL databases to eventually consistent key/value and document stores.  Consider numerous factors when choosing, including transactions as well as query interface, important for your data visualization tooling.

Data Visualization: Dashboards, charts, leaderboards, pivot tables, and visualizations all play a key role in understanding your data, both historical and real-time.

Historical visualization helps you explore, understand patterns, and create predictive analytics. Real-time visualizations help you understand the current state of your business, usually in the form of a real-time dashboard.

You will want to evaluate tools from vendors such as Tableau, Qlik and MicroStrategy for dashboarding and ad hoc visualizations -- user experience is a critical factor with this kind of software so having your users try it out is essential.

Data Science: A growing number of tools can help you extract information and insight from your data. Machine learning packages provide data classification, clustering, and regression analysis, and allow software to “learn” to identify and make predictions on data. Consider popular open source offerings such as Spark (MLlib) or R to get started.

4)    Build a Data Pipeline that delivers Data as a Service (DaaS) to internal customers.

Define an architecture that serves data to your internal customers. Capturing and analyzing the data is great, but it is only the first step. Data and insights must be readily available to consumers (people and applications) across your enterprise. Consumers of your data must be able to tap into both historical data from the data lake as well as real-time fast data, along with the insights derived from both together.

5)    Begin building applications to extract value from the data  -  then iterate.

Start small and add incrementally. Identify opportunities for small quick wins that will prove you can capture value from your data. Realize that data evolves and new patterns will emerge. Foster an environment of experimentation, innovation and continuous improvement and iterate on your data analysis.

Data is valuable. Batch processing is so 1990s. Now you’ve got five ideas for how to extract more value from your data. Start now and iterate. Think Big, of course, but also Think Fast.

In-Memory Database Sizing

 This post originally appeared on VoltDB.com in June, 2014.  

In-Memory Database Sizing

Sizing an in-memory database does not follow conventional database sizing rules.

For traditional databases, you buy a decent server machine, likely one with many CPU cores and reasonable memory, and then focus on application IOPS (I/O Operations per Second).  If you are really going to stress the database, you must choose disks that can support the I/O needs of your application, today and in the future. Because these systems often use many disks to achieve high I/O performance, capacity is usually an afterthought.

With in-memory databases, throw out everything you know about sizing databases. Performance is now the afterthought, but capacity must be carefully considered.  

To size your in-memory clustered database application you must consider the following factors:

●      First, compute the size of the data your application expects to host.  This means computing the size of all rows in all tables.  And also computing the size of all table indexes. VoltDB provides an easy-to-use interactive sizing tool in our application catalog report, described later in this post.

●      If you want a highly available database cluster, all data must be stored redundantly. You’ll need to multiply your logical data size by the replication factor to understand your memory requirements. For example, if you have 100GB of table and index data, and you want your database to be able to withstand losing one of the nodes in the cluster, you will want each datum to be stored twice, and on two different physical machines. This means that you need to size the data requirement of your database to be 200GB (100GB of data X 2 copies of the data).

●      Allow for some extra memory, perhaps an additional 25%, for database and operating system overhead. VoltDB uses extra memory for storing intermediate results, maintaining an undo log, caching SQL plans and buffering network data. Note that on a machine with 4GB, you’ll want to leave a larger fraction of memory free than on a machine with 256GB.

This tallied memory total can now help you decide how many machines your system requires.  Note at no time did we need to determine how much CPU we required, or how many IOPS were needed.  Because you are sizing your database based on data, memory size is the most important calculation you will likely use. Generally speaking, in-memory databases like VoltDB operate so fast that CPU utilization is not a problem - there is plenty of headroom for growth. As for disk I/O, most in-memory systems use sequential logs on disk, eschewing costly random I/O. With this reduced dependency on disk performance, VoltDB can achieve tremendous throughput, even on commodity spinning disks.

Often, you’ll be faced with a choice between many machines with less memory per machine or a small number of high-memory machines. There are lots of reasons to lean one way or another. For one, you should try to run with at least three nodes if availability or redundancy is a concern. Additionally, some administrator operations run faster with less per-machine data. At some point however, too many nodes can be a management hassle. Most VoltDB users run with 3 to 30 nodes per cluster for these reasons.

Planning for Data Capacity and Growth

Sizing VoltDB databases is fairly straight-forward.  Once you have settled on your application’s database schema, point your web browser to the database Catalog Report and choose the “Size Worksheet”, found at this URL: http://localhost:8080/#z (note the database must be running).

 

This sizing worksheet is automatically built from your application’s schema. It allows you to enter your expected number of rows for each table and then computes the amount of memory required for the database cluster as a whole.

Note that the footprint of your database is the size of the number of rows as well as the size of all indexes and materialized views. You must also leave additional memory for the operating system.  Note that this assumes that VoltDB is the only active application on these machines.  If you plan on sharing the machines with other applications, something we do not recommend, your memory requirements will naturally be larger.

Expanding your Database Capacity

A well behaved IT environment will monitor the performance and capacity of the applications running the business.

Because VoltDB processes transactions incredibly fast, it is often the case that there is plenty of transaction throughput headroom and there are lots of extra CPU cycles available. Monitoring the memory usage of the database cluster becomes much more important. VoltDB provides monitoring interfaces, as well as Nagios and New Relic plug-ins that make monitoring memory usage easy. These tools allow you to define capacity alerts, much the way you would define disk space usage alerts for legacy database systems. Should your in-memory database memory usage grow to a high level, signaling a capacity limit alert, it may be time to add more storage capacity to your database.

With VoltDB, adding capacity is easy: simply add more nodes.  VoltDB supports adding nodes to a running cluster without interrupting ongoing operations or sacrificing ACID transactional guarantees.

If you do not wish to add more nodes to your database cluster, doing a rolling memory upgrade of machines in a redundant VoltDB cluster is an alternative. Since VoltDB uses active replication within a cluster, removing and replacing a node can be done transparently to a user’s application with no data loss and often unnoticeable performance impact.

In-Memory Database provisioning… A New Way of Thinking

If you are provisioning and rolling out a new application using an in-memory database you will have to shift your thinking about how you size your deployment. With all data residing in memory you will need to provision enough memory in your database cluster to hold all of your data, and a bit more memory for regular operating system operations.

Disks are only used for durability and most in-memory systems like VoltDB work great with commodity hardware, disks included. Sizing for IOPS is a thing of the past!