The commodities industry has always been data intensive, but the volume, velocity, and variety of data generated today is staggering. It is estimated that 90 percent of the world's data was created in the last two years, much of it from data sources that did not exist just a few years ago.
The amount of data that organizations now have to process is growing at an exponential rate. In 1940, knowledge doubled every 25 years. In 2014 it doubled every 13 months. By 2020, experts predict that knowledge will double every 12 hours – primarily due to the explosive growth of internet of things (IoT) technology. Gartner forecasts that by 2020 more than 35 billion things will be connected to the Internet[i].
For commodities companies, IoT technology can be a game changer. IoT sensors on bulk-handling equipment can provide highly accurate, geo-located data about grain storage sites, ports, and mines. Sensors embedded in farms can provide data about the health of crops – and can remotely trigger equipment to water dry fields. Because IoT sensors collect and send real-time data, commodity traders can check the status of inventory as it moves through the value chain and more accurately predict the timing and availability of products.
Commodity data used to arrive a fixed intervals, allowing businesses to analyze predictable amounts of structured data using batch data processing. Today, markets move in a matter of seconds and the flow of data arriving in an organization is continuous. Extremely fast processing is needed to perform complex transactions on diverse data streams. Real-time or near real-time analytics are necessary to ensure everyone within the organization has access to the information they need at the time that they need it.
Commodities businesses use data from disparate internal and external sources, including CTRM, ERP, CRM, spreadsheets, market curves, and weather data. This information is traditionally highly structured in machine ready formats. The vast majority of new data being generated is predicted to be unstructured.[i] In addition to the information supplied by IoT sensors, this includes data available on social media, embedded in video and audio, emails and texts, or location and log files.
Just as a commodity trader looking at meteorological data to understand avocado production has an advantage over one who doesn’t, a trader following #avotoast on social media would have even greater insight.
Big data can be a game changer for commodity trading companies, but only if they can take the volume, velocity, and variety of data and convert it into valuable insight. That’s where solutions like Eka Analytics come in. Eka Analytics analyzes all available information – structured, unstructured, internal, and external – using advanced analytics to answer commodity traders’ most important questions. Eka’s app-based solution provides quick analytics on a computer or any mobile device, because changes occur whether you are at your desk or on the road.
The volume, velocity, and variety of data available today provides tremendous opportunities for commodity traders. The ability to synthesize, analyze, manage, and visualize it will be a key differentiating factor for successful commodities businesses in 2017 and beyond.
[i] EMC: The digital universe in 2020, 2012
[i] Gartner: Internet of Things, Endpoints and Associated Services, Worldwide, 2014