If there is one thing that managers don’t seem to understand about the the use of data, it’s that the path to developing a truly data-driven organization is littered with dead ends. This misunderstanding results in a mindset in which everything is seen as a scaling problem, and the consequence is that data endeavors fail.
This post focuses on the dead end that companies face when trying to develop their data infrastructure. When I get time I will write a second post about the dead end that companies run into at the point of actually utilizing the data.
Let’s talk about a hypothetical, mid-sized company, Saber Inc., that manufactures printers. Saber sources components from various suppliers, assembles them in a warehouse, and sells the finished printers directly to businesses and consumers both via a sales team and an e-commerce site.
Now let’s say Dwight is in charge of Saber’s data-driven transformation. He learns that despite Saber’s lack of a clear data strategy, people are already using data within the company. The team in charge of the website is tracking data using Google Analytics, and has added all sorts of custom dimensions with the help of Google Tag Manager. The warehouse manager is meticulous and keeps detailed logs of everything that comes and goes using Microsoft Excel, and he uses that data to produce charts and analyze the warehouse’s performance. The sales team is forced to log sales in an official system, but they don’t have access to that system’s data. As a result, they also keep an Excel sheet of the records, which they use to compare performance and track client orders so as to know which clients might need to order again in the future.
Dwight is happy about this and thinks he just needs to help the company to keep moving in this direction. In other words, Dwight thinks this is about scaling: we want to keep doing what we’re doing, but more. Unfortunately, this is totally wrong.
Each team is operating in a silo with its own system, and there is no way to link these systems together; the connections in the data are lost, as is most of the value. Dwight thinks he could hire a few people to build a system to connect the various pieces of data, but the quality of the data would be dubious, and a change in one system would break the connector and with it the whole dataset.
The only way to move forward in this case is to first go back. These systems need to die to make room for a new, coherent approach. However, killing those systems is no easy task. Dwight isn’t the CEO, so he probably can’t tell the warehouse what to do. Dwight’s team needs to build a new system before the other teams will stop using their own approach, but in the meantime the other teams are going to continue to add features to their hacky systems. There is no easy way back that will please the other teams.
The problem is that most companies are in a similar position to Saber Inc. and don’t even realize it. No one is standing at the starting line, because people within a company do not wait. In the absence of a good team with a clear data strategy, other teams will have developed their own hacks. They’ll have gone some distance forward, but the path will be a dead end because hacks don’t scale. Good data engineering does not come into existence naturally; it is the work of central planning.
The real lesson here is that if you’re a manager at a company that wants to transform itself into a data-driven organization, your real job is figuring out how to walk back, not forward.