No more ad-hoc requests! The journey from data service to data product organizations

Data Services or Data Products? The Dilemma that shouldn’t be

Data teams represent an organizational transformation for most companies, some of which hire data scientists, data engineers, and machine learning engineers. But which problems do data folk solve? Most of them take care of urgent business requests, all day, every day. Do they work with engineering? Do they focus on R&D until they build something truly valuable? If this is you (or your team), are you doomed? Is there a way out?

These are common questions for companies of all sizes, but there is embarrassingly little consensus on what to do about them. This typically forces data teams to react to incoming ad-hoc requests. No roadmap, no direction: just SQL queries.

Luckily, there is a way to turn traditional data service teams into an integral part of your business, while adding value with every single contribution. It implies a shift toward becoming a data product team, listening to internal and external customers, and building tools that increase operational efficiency and reduce the volume of ad-hoc requests.

Data Services

Under the data services paradigm (a model inherited from Business Intelligence), data folk create a backlog of equally important and seemingly urgent business requests. These are usually tackled on a first-in-first-out basis.

Every single workday, the role of the data service team is to gather data, clean it, and deliver actionable insights to the business (this is easier said than done).

This approach has a few advantages:

• A few data pro’s in your organization can answer most business questions.
• Expertise compounds over time: Answering similar questions becomes easier with each additional one.
• Stakeholders have their go-to person/team: Folk in the organization know who to go for when they want their business questions answered. There is no ambiguity or ambivalence of who owns the responsibility of answering data questions.
• Role and duties are clear: Data folk under this paradigm have very clear roles since their objective is to minimize the time to respond to business requests while delivering high-quality answers.

Despite these advantages, the data services approach falls short in the following ways:

• Fails the “Winning the Lottery” test: If a member of your team wins the lottery and decides to leave, you’re stuck with a large knowledge void. Covering these knowledge gaps is extremely costly for companies of any size.
• The analyst learns, but the organization doesn’t: Congrats, you have an in-house data expert! However, are you sure the organization is learning as much as them? Do you keep proper documentation of this knowledge?. It seems that by solving a problem, you’ve just created a bigger one.
• Your company depends on your data analyst: The data team receives multiple emails or tickets from everyone in the organization: Finance, Marketing, Sales, and even the CEO partake. This rapid-fire of requests makes it challenging for your team to have a clear path to grow.
• Your data team is siloed: Everyone comes to your team for data, but this is likely a one-way street, which means fewer opportunities for members of your team to have a long-lasting impact on your business.

Fortunately, there’s a way to reap the benefits of this approach while reducing the downside of your organizational knowledge.

A few strategies to build a Data Products Team

In short, you need to set up an environment where your data team can build products that add value to your business. They shouldn’t be treated as a group of mercenaries standing by to fulfill data requests coming from every side of your organization.

Your data team needs the space to build tools to do the following:

• Answer their business questions: When a data person comes up with a question they should have the tools to answer it. By setting infrastructure to make this process fun and automated you’ll empower your team to answer questions faster and more accurately.
• Automate answering common and recurrent business questions: Frees a large percentage of your team’s time, and allows them to focus their analysis skills on the toughest business problems. This also creates a positive reinforcement loop with the business: Fast (and accurate) answers build trust and reliability.
• Build and deploy self-service data tools: Sounds like the holy grail, doesn’t it? In reality, you can (and should) start small. Find a concise and impactful business problem. Get to know the stakeholders, understand the frequency at which they need insights and data. Can you build a reliable solution? Is it going to make their jobs faster/easier/better? Prepare a framework to measure and quantify the impact of these self-service tools. If you do this right, your data team now develops and maintains a product that is constantly shaped and improved by feedback from your business counterparts.

Outlook: How to get there?

Every organization is a different beast: There are no magic recipes for success. But every organization needs high-quality data delivered to their stakeholders. Data product teams that work closely with business and product stakeholders have good chances of adding value to the business, answer critical business questions while building key internal products.