The New Rise of Time Series Databases
By Michael Risse, VP, Seeq Corporation
Time series data storage and management has long been an interesting but quiet market category. It’s been a multibillion-dollar business for years and a main stay in process manufacturing plants since the 1980s.
But recently, the category has been getting another look from investors and companies both large and small, due to huge time series data volumes. In 2010 manufacturing companies generated 1800 petabtyes of data per year, twice as much as the next closest vertical, and much of that is time series data. Manufacturing data volumes continue to grow, thanks to the new Internet of Things (IoT) and Industrial Internet of Things (IIoT) deployments.
These vast data volumes attract attention because “data has gravity”—which means that whoever stores the data will attract high value add-ons such as management, security, analytics, and consulting services. This has resulted in attracting these other business opportunities for the time series data storage platforms can be licensed for a comparatively lower cost.
Venture Interest and Open Source Options
On January 24, 2018, Timescale, an open source time series database (OSTSDB) company, secured $12.4 M Series, a funding led by Benchmark Capital. This was soon followed by InfluxData, which scored $35M in a Series C funding on February 12, 2018, led by Sapphire Ventures, bringing their total funding to $60M.
"With multibillion-dollar markets at stake, industry heavyweights are vying for a position in an once-quiet category"
Sapphire Ventures is the venture arm of SAP, and Benchmark Capital and Battery Ventures are both very successful venture funds. (Benchmark has nearly $3B under management and was an early-stage investor in companies ranging from Twitter to Dropbox to Instagram, and Battery Ventures has nearly $7B in assets.) The investors are likely looking at graphs (Figure 2) showing that time series databases have recently been the fastest growing segment in the database market. InfluxData, for example, claims 115,000 active sites using their product.
By virtue of their funding, Timescale and InfluxData are now separated from a pack of OSTSDB companies or open source efforts, including OpenTSDB, Prometheus, Druid and KairosDB . Timescale and InfluxData may be staging a repeat of the recent CloudEra/ Hortonworks battle among big data startups.
However Hortonworks (NASDAQ: HDP), a leader in Hadoop and big data implementations with process manufacturing companies, has itself been adding features and patterns to address time series database opportunities. Their added value is enabling manufacturers to analyze any type of data for batch, interactive, or real-time applications by unlocking siloed data sets from both operational technology and information technology systems. By centralizing customer process data into a single open source platform, Hortonworks is able to democratize industrial data analysis by providing a single view of operations for their customers.
May it be public companies or startup ventures, OSTSDB and big data vendors have become significant players in the time series storage market.
The Public Cloud Arrives
For storing large volumes of data, many companies are preferring cloud these days. Consequently, the big public cloud platforms are paying more attention to the largest sources of data.
For example, Microsoft recently introduced a Cassandra interface to Azure CosmosDB, in their NoSQL cloud data service, which brings them into the market for time series storage. What’s more, CosmosDB has a graph database interface, which means it has services required for modern historian functionality:-a Cassandra interface for time series storage and a graph database interface for defining and accessing asset models and hierarchies.
However, interfaces alone don’t make a historian or a time series database product successful. There are many other factors involved, and it remains to be seen how Microsoft prices their service and differentiates it from open source offerings, and how they work with partners offering historians on top of Azure. These and other decisions will go a long way to determining the success or failure of Microsoft’s foray into the market.
Honeywell’s recently announced Uniformance Cloud Historian that runs on Azure and leverages Microsoft data services, as its platform for distributed storage and management. These types of industry partnerships will be crucial for success within process manufacturing verticals.
Other companies like Amazon’s DynamoDB and Google’s BigTable are making their own arguments for using their NoSQL offerings for time series data storage. This list could go on and on: PTC/Thingworks has established partnerships to support their IIoT platform with time series storage options, plus there are time series storage services in GE Predix and Siemens Mindsphere. Beyond these offerings, second tier IIoT platform offerings supporting time series data could fill a dictionary.
As mentioned earlier, data historians (also called process historians) have been used by process manufacturing companies for decades. Every process automation vendor offers at least one historian, like DeltaV Continuous Historian from Emerson Process Automation, and others have multiple historians due to a history of acquisitions, like Schneider Electric. Some historians are sold separately by dedicated historian firms like Canary Labs, and others are offered in the context of the software company’s principle offering, like Inductive Automation’s Ignition SCADA system.
But for all the historians available for sale, one vendor stands apart in market share among high value oil & gas, chemical, power generation and other process industry customers which is OSIsoft with its PI infrastructure platform.
If emerging startups and clouds—are affecting OSIsoft’s business, it’s hard to see from the outside. As a private company, they don’t release earnings, but an investment in OSIsoft last year by Softbank suggests expectations of further growth. There are also public examples of OSIsoft’s momentum. Their upcoming user conference, rebranded as PI World, is expected to be their largest conference, with a doubling of space for partners and sponsors. With new investors, a growing partner ecosystem, and new efforts in edge and IIoT deployments—it would seem OSIsoft has opportunities for growth, despite challenges from new participants.
Certainly, OSIsoft’s established position is a point of confidence for customers, support for existing investments and IT requirements, and this position is validated by industry observers, including ARC—“ARC research indicates OSIsoft has been the market leader in process historians for many years,” commented Janice Abel, Principle Analyst at ARC Advisory. “The company has a well-established and loyal customer base, a large partner ecosystem, and the OSIsoft PI historian connects to data from more than 450 different sources, which to the best of our knowledge far exceeds any competitors’ products.”
Perhaps the actual impact on OSIsoft of the open source and cloud entrants to the time series database market is an increase in the awareness and need for a proven, enterprise-ready solution that’s delivered out of the box.
With incumbents and challengers using both open source and cloud services, the market for time series storage is shifting towards an interesting future. This market is attracting both top tier venture capital firms and the largest public cloud platforms. For now, it would seem all boats are rising on a tide of interest in the market segment, as IoT and IIoT interest and deployments continue to grow.
While it’s been interesting, it’s only the beginning of what promises to be a wild race, with multi-billion dollar prizes at stake.