Product analytics with AI: a 7-step process that keeps the numbers honest

It’s a huge temptation: let AI analyze your data and discover hidden patterns and levers. But even if we put security aside, it is not as great an idea as it may seem.

At least not without a deep understanding of the data and a set of rules.

And of course, I did it. More than once.

The experiment

One of my clients insisted that I use Superset’s MCP to quickly analyze the company’s data and make a data-driven decision on it. At first I was excited to do so, and I had high expectations for this way of working with data, because in my own projects I’ve always hesitated to give AI access to my data. This was a great opportunity to run an experiment.

At first, everything looked great. I got all the data I needed very, very quickly. I could talk to the data like I would talk to a real person. Unlike a real analyst, Superset’s MCP returns data in minutes, not in days or weeks, and I was impressed.

But once I started to make a decision, I started to doubt the data, and I decided to recheck it in the source. It was a big surprise that a lot of the information presented so elegantly and smoothly in the AI’s storytelling wasn’t based on real data at all. For example, it found a price increase that never happened. It showed that the average price had jumped roughly by a third in November, which contradicted what the team and the founders were telling me. When I looked closely into the data, I found that the price had been calculated as an average across monthly and yearly subscriptions. It’s a classic newbie mistake when you’re working with averages, and it wasn’t what I had asked about. I hadn’t asked for a blended average price. I had asked about the monthly payment per user.

And the same combination, Superset’s MCP and an LLM, smoothed the data to fit the story several times. For example, it reported a steady decline in the user base from September to December, but when I looked at the real data, the number went up in November and only then went down again.

I just don’t trust its math

You may now suppose that I’m a strong anti-AI person, and you’d be surprised how much I actually use AI in analytics. I just don’t trust its math. I prefer to use AI to speed up the writing of analytical scripts, but I’d rather have traditional analytical scripts do the analysis, so that it’s repeatable and controllable.

Remember I said I don’t trust AI with my companies’ data? Yet I use a whole set of analytical script-writing skills, written by me and my colleague, throughout our analytics process. It has sped us up a lot, we manage without a dedicated analytics team for now, and we can trust our data. Here is what we are doing.

1. Describe the domain

First of all, we have described the domain knowledge. To do so, we use Codex or Claude Code connected to a GitHub repository containing all our code. It helps us describe what data we send to different systems: to our database, to the analytics system, and so on. Then we work out what this data means and how it is represented, and we store this domain knowledge in external files, like MD files, so our analytical skills can access it. At the same time, we analyze all the existing analytical scripts to determine how to work with the data. What connections do we have? What data sources do we have? Sometimes, for example, we can use an old analytics system to collect data for benchmarking, and we store that as an MD file for the next steps too.

2. Collect the inputs

Then we collect all the necessary input for the requirements. We gather the tribal knowledge: from a meeting, from the task in the task tracker, or wherever else it lives. We collect everything connected with the analytical task, and we use the task tracker to pull in even more history of the question.

3. Write and review the requirements

The next step is to create technical requirements for the analytical script. Here we use AI heavily. We add all the necessary information about the domain, about our analytical questions, and all of the history connected with them, so the AI can write correct requirements. And, of course, we check these requirements. There are two ways. We can do it ourselves: as a founder and CEO, I can read the requirements and understand whether they are correct, whether they are going to calculate the right numbers, or whether they need improvements. If I don’t have the time, or don’t know the systems and the analytical question well enough, I can also ask another AI, another model, to proofread, check, and review the requirements. In two or three iterations, we get pretty good requirements for the analytical script.

4. Write the tests first

When the requirements are ready, we write tests. Before we write a single line of code, we create a set of verification tests that will show us whether the data is correct and non-contradictory, and whether we can trust it. Or whether we need to take a few steps back, redo the script, redo the math, or recheck the data source.

5. Write the script

Now, finally, we can write the analytical script. This is the simplest task. We just ask the latest model to write it for the environment we use for notebooks, with the Django ORM and ClickHouse connections. It uses all the information from the technical requirements, which in turn use all the information from the domain knowledge and the rules for working with our analytics systems and our database. As a result, we have a final script that we can upload to Jupyter and run.

6. Run, debug, decide

The next step is the most boring one. We run the script, debug it, and discover that some data we expected doesn’t exist. We need to check and decide what to do next. This is a pretty long and pretty hard step, and boring at the same time, but we need to do it ourselves.

7. (Optional) Let AI summarize

The last step is optional. If the resulting data doesn’t contain any sensitive information, like users’ personal data or private business numbers, we can upload the result to AI and ask it to summarize it and generate insights from it.

Where it bit me anyway

We need to be careful here, because in one of my latest analytical tasks, there was a very bad mistake at exactly this step. I asked AI to prepare, for a data-based team brainstorm, some “how might we” questions in the design thinking style. One of the questions was based on data, but it looked suspicious to me. The sentence was “users click 4 to 7 cards before buying,” and I hadn’t seen this in the data myself.

I asked AI, “Where did you get this data?” It answered, “Oh, this is my mistake. They make 4 to 7 clicks and I had never seen data about how many cards”. It may be 4 to 7 clicks on the same card. It was wrong data that could have led us to the wrong decision and even a wrong way of thinking. I strongly recommend that, if you do this last step, you recheck every conclusion the AI draws, because it can hallucinate even there.

Can you do this without an analyst?

As I describe our way of working with data, I keep asking myself one question: can anyone without analytics knowledge do this, and do it right? I honestly don’t know, because on the one hand, a lot of the work here is delegated to AI. Yes, you can do something. But a lot of checking, verifying, and debugging is still performed by a person with an analytical background.

Anyway, whether you have analytical skills or not, whether you just use AI with an MCP to your data storage or analytics system, or you write analytical scripts yourself, never ever trust data generated by AI without questioning it. Always check that it corresponds to reality. Does it make sense to you? Does it agree with the other data you have? Always ask how it was calculated and how it should be interpreted, and, whenever you can, make the process repeatable and understand the structure of the data you’re working with.

It’s not only about analytics

And everything I wrote today isn’t only about analytics. I’ve been using AI systems since the first release of ChatGPT, and even a little earlier, and everything in my experience says that you should mix the deterministic approach with the probabilistic approach of AI, so that each does what it’s good at. I use this principle in all my products, in my new workforce solution, and in my consulting work. I believe we should take the best part of both approaches to working with information.