Why Your Data-Driven Transformation Is Failing

I hate that I am writing this. The title alone screams “Generic LinkedIn Content”. It feels like the type of post that gets shared by corporate professionals who pretend to be robots online. On top of that, I’ve never worked for a big company, so what do I know? But here I am, writing about how companies are failing to realize the benefits of basic technology.

Most companies today hire for technical positions and non-technical positions; there are programming jobs and there are non-programming jobs. A company might claim to be “data-driven”, but if you ask the employees they’ll tell you that’s what the data science department is for.

Business analysts, operations managers, marketing associates, and compliance officers don’t know anything about programming. The employees might have received training on how to use some out-of-the-box data analysis software, but the software is costly, people don’t really use it, and each new employee has to receive training as well. This state of affairs results in a huge misallocation of resources.

My Background: Law, Finance, and Programming


There are two reasons why my background is relevant to this post. First, I want to make it clear that I’m not just a brogrammer that is annoyed that non-programmers have jobs. Second, many of my observations are related to challenges I’ve had applying for jobs, and I think that knowing my background will help you to understand these points.

Two years ago I wasn’t a programmer at all. I was a master’s student at Oxford, where I was studying law and finance. I had just graduated with the highest marks in my class from the University of Edinburgh, where I studied law, and wanted to do a master’s in law and finance before taking a job at a company doing… something. It’s not like graduates have any real skills; companies just hire the smart graduates because they’ll work hard, learn quickly, and eventually be worth something. I think I fit that description at the time.

But then I started programming. Encouraged by two of my friends, I got really into it. I started with Python, and mostly did projects that I found interesting related to data analysis. After only a few months I managed to scrape every share purchase agreement from the SEC website, and used a simple machine learning algorithm to identify the presence of certain types of clauses.

Pretty quickly I became aware of the ways in which programming was complementing my existing skills while also revealing my limitations. Every day I would have new ideas for how I could use programming to improve some task or strengthen some analysis. A number of my friends were working at the time, and on multiple occasions I used programming to automate some task they had been given at work.

Ultimately I didn’t go work for a big company, but rather started my own company in Texas with a friend. That isn’t especially relevant to this story, other than to say that I worked there for a year, improved my programming skills dramatically, and had some crazy experiences doing all of the things that go along with being a startup founder.

In August I decided to leave the startup and move to Norway to be with my girlfriend, and I have been looking for a job here since. With an excellent academic record in law and finance, strong data analysis programming skills, and experience as a startup founder, I never expected it would be this hard to find a job. But it’s not been easy, and the not-so-subtle undertone of this piece is that I’m frustrated by how difficult it has been. With that said, searching for a job here and talking to people has also taught me some interesting things about how most companies work.

Stepping Over the Low Hanging Fruit


Like I said at the start, companies have programming jobs and non-programming jobs, and nothing in between. This, in my opinion, is a huge mistake. I am usually hesitant to make general claims about business strategy, as strategy is rarely so easily generalizable. With that said, if a company wants to undergo a “digital transformation” and position itself as a “data-driven” company, it makes sense to integrate people with programming skills into every department and role.

Why? Because right now companies are stepping right over the low hanging fruit. A well-paid team of data scientists might be busy tuning a single predictive algorithm to squeak out slightly better performance, while Jim from Accounting spends 4 hours every Monday manually combining data contained in different Excel sheets. In one department the team is working tirelessly to improve the quality of projections, while in the department over they’re still eyeballing the figures.

If you read advice on undergoing a “data transformation” you’ll find all sorts of complicated plans and strategies, most of which I disagree with. Here is a quote from a BCG report:

Image Did Not Load

And sure, maybe this isn’t horrible advice. But if you want to transform your company into a modern data-driven enterprise, the logical first step isn’t “go build a team of super star data scientists to work on projects”. The first step is to start hiring people with basic programming skills into different roles. Executing a “data transformation” is way easier when everyone has a kid who sort of knows Python on their team.

“We Can’t All Be Programmers”


Not everyone is going to be a programmer, and it’d be a waste of time and money to teach Python to every employee. Unfortunately, this leads many organizations to adopt proprietary out-of-the-box analysis software and start spending on employee training.

This causes two problems. First, none of your employees really want to use the software; they were told to, and sent to training. Maybe they said they would like to be trained on the software and that’s why they were chosen to go to the training classes, but we all know they’re not really into it. In contrast, if you hire a kid who knows R or Python or some other programming language, he or she is probably actually interested in using it. I’ve never met a Python programmer who dislikes Python.

Second, the use of proprietary out-of-the-box analysis software actually makes it less likely you will hire the type of people you really need, either because they never apply or because you reject their application. By way of example, one of my strengths is data visualization. I use a JavaScript library called D3.js, which is the gold standard of data visualization. However, I’ve been rejected for multiple data visualization jobs because the company is looking for someone with Tableau skills. For those of you who don’t get why that’s a problem, Tableau visualization is to D3 what Little League is to Major League Baseball.

Proprietary out-of-the-box analysis software is meant as a halfway house. When a company starts relying on the software to the exclusion of better tools, and starts actively seeking to hire people with experience with that software to the exclusion of people with better technical skills, the potential for a true “data transformation” is gone.

Conclusion


If you want to make your company into a “data driven” company, the first step is to start hiring people who kind of know something about data analysis and programming. Don’t just hire these people into programming roles; hire them into every type of role. Plenty of people “sort of know some programming”, and I think that alone goes a long way. Finally, don’t do anything to turn these people away; whatever proprietary software you’re using doesn’t matter, and having experience with it isn’t a useful skill.

A data-driven transformation isn’t going to work if you think it will happen “somewhere else in the company”. It has to happen everywhere, or else you’ll step right over the low hanging fruit.