Data has grown at an exponential rate. My favorite example is to visualize the Amazon rainforest. It encompasses 1.4 billion acres of trees. Each acre has 500 trees so that aggregates to ~700 billion trees. Take all those trees and chop them down to paper (of course, I would not advise that as that would cause an ecological disruption) and scribe them into text; you get 1 Petabyte of data. An Exabyte is 1000x a Petabyte, and we’re projected to generate 40,900 Exabytes by 2020!
From my personal view, I have seen the incipient transition to the digital age from the startup I used to be a part of: an agriculture Farm-to-Store E-Commerce B2B Enterprise. Data was collected in fragmented Excel sheets, where lack of visibility and communication caused huge discrepancies in committing procurement and reconciling sales. This rudimentary way of running would not be able to differentiate us from an incumbent aggregator in the traditional market, with a pen and piece of paper.
Migrating to a unified cloud platform that would seamlessly connect the supply chain was absolutely paramount, as it was entrenched in the agriculture industry to place ad-hoc last-second orders attempting to anticipate the volatile daily market prices. On top of that, every minute is critical as once the perishable commodity is procured, it immediately starts depreciating in quality, weight, and even perceived value; this was arbitrary to every customer dealt with.
Additionally, ensuring our data was consistent, clean, and accurate was a separate daunting task by itself. It was hard to corroborate data to compare, such as our procurement cost at various locations. Each market pegged prices against erratic markets and their internal demand & supply.
One instance I clearly recall is redundancy challenges with vegetables/fruits that had different nomenclature based on the region, dialect, and customer requirements. For one variation of brinjal, the different names for the same SKU were Baigan/Balloon/Aubergine/Suphal/Bartha/Burdha/Bottle Brinjal. Without regular SKU Master control adherence, new SKUs would be created that imbalanced inventory valuation.
Another conundrum was reconciling vegetables/fruits that were procured and sold in multiple UOMs at different proportions. Applying a dynamic conversion factor would not suffice as the number of pieces of banana in a Kg was difficult to compute unless every banana was physically weighed piece-by-piece while doing Goods Receipt. This was not economically feasible.
With a myriad of data available at our fingertips, business decisions were able to be extrapolated from data generated. However, it led to a predisposition to succumb to impulsive and narrow-minded behavior by the insight the data gave. Once the Key Performance Indicators were succinctly defined as to what were most indicative of a successful day (Net Margin/Kg, Distribution Cost/Kg, Wastage/Kg, Customer Order Size vs. Rejection % etc), there was more alignment between various departments.
As our data matured, we partnered with an AI consulting company to come and take a stab at building a model for us to forecast demand and calibrate prices to our collection centers and customers respectively.
For instance, the purpose of the Sourcing model was to create an engine that gave foresight into how and where to source all primary attributed SKUs for each of the 5 Distribution Center across South India.
There are many key variables that make up this decision (Price trend, Forecast, Fill Rate, Rejection Rate, Delivery Timing, etc.) The solution involved running a multiple regression model for each Business Segment (Modern Retail, Kirana Stores(SME) & HoReCa (Hotels, Restaurants & Caterers). Each Business Segment had different dynamics based on the key variables.
For example, the HoReCa was more price and service level elastic, as vegetables was a component in their cooked meals instead of a finished good. Modern Retail stores have to sell the product on the shelf, so quality was the primary variable.
We would measure the model vs actual orders using historical data and reiterate the data to get an optimized output. However, building a model with variables that were significant for an extended period of time was difficult as the model would deteriorate quickly as we scaled the company and strategies changed.
After resilient effort, we started to involve the end-user to make the final decision based on their personal heuristics. Once we were able to build an interface where they could make final decisions based on what the model predicted, we observed a streamlined model vs actual outcome variation.
On the whole, I learnt data is a necessary tool to foreshadow decisions. However, beyond every model with an array of numbers, there are people who put their soul into working to provide the best service levels. Once stakeholders start effectively utilizing data to augment their collaborative insights on top of driving people to meet quantified outcomes, it brings a synergistic effect.