Summary

The problems appearing in AI systems today — data quality failures, measurement confusion, governance debt, integration assumptions nobody documented — are the same problems that analytics teams have been navigating for decades. The tools are different. The patterns are identical. Fifteen years of working on analytics infrastructure across FTSE 100 and S&P 500 organisations produced a set of hard-won lessons that transfer directly to AI systems. The organisations that already learned them in analytics will navigate AI faster. The ones that have not will pay for the same education twice.

I have been working on data and analytics infrastructure for a long time. Long enough to have watched the same patterns repeat across different technologies, different industries, different sizes of organisation. The specifics change. The underlying problems do not.

When AI workloads started appearing seriously inside the organisations I work with, the first thing I noticed was how familiar the failure modes felt. Not similar — identical. The same root causes, the same sequencing errors, the same underestimation of the work that happens upstream of the visible output. The tools had changed. The lessons had not.

What follows is not a history of analytics. It is a set of lessons that took years to learn in the analytics context and that transfer directly to AI systems — lessons that most AI projects are currently learning the slow way, which is to say by paying for them twice.

Lesson One: The Problem Is Always Upstream of Where You Think It Is

The single most consistent pattern in analytics work is the tendency to frame problems at the wrong layer. When a dashboard produces wrong numbers, the first instinct is to fix the dashboard. When the dashboard is not the problem, the next instinct is to fix the query. When the query is not the problem, attention moves reluctantly to the data pipeline. When the pipeline is not the problem, someone eventually, with some reluctance, starts asking questions about how the data was collected in the first place.

By that point, the cost of the original problem — which was always a data collection or definition problem — has been paid several times over in wasted diagnostic effort at the wrong layer.

AI systems reproduce this pattern with remarkable fidelity. When a model produces poor outputs, the first instinct is to improve the model. When the model is not the problem, attention moves to the prompt engineering. When the prompt is not the problem, someone starts looking at the training data or the retrieval pipeline. When those are examined, the root cause turns out to be in the data that fed them: inconsistent definitions, stale records, schema drift that accumulated over time.

The quality of an AI output is bounded by the quality of the data that produced it, regardless of how sophisticated the model is.

The lesson from analytics is to start the diagnostic at the data layer, not end there. Before investing in model improvement, characterise the data: is it fresh, is the schema consistent, are the definitions agreed, is the lineage clear? If any of those questions do not have clean answers, the model improvement will compound on an uncertain foundation.

Lesson Two: Confidence and Accuracy Are Different Things

Analytics tools produce outputs that look precise. A chart showing conversion rate to two decimal places looks more authoritative than a conversation. A dashboard that updates in real time looks more reliable than a spreadsheet. The precision and the production quality of the output convey confidence that the underlying data may not support.

I spent years learning to read past the production quality of analytics outputs to the data quality questions underneath. What are the event tracking gaps that the funnel chart cannot show? Which conversion events are double-counting because two tracking implementations overlap? What does “session” mean in this context, and is it the same across all the charts on this page?

AI outputs carry a stronger version of the same problem. Language model outputs are fluent, grammatically precise, and confident in register. They do not hedge visually in the way that a chart with error bars does, or signal uncertainty in the way that a dashboard note saying “data lag of 24 hours” does. A model trained on inconsistent data produces outputs with the same surface confidence as one trained on clean data. The visual cue that something might be wrong is absent.

A well-formatted analytics output and a fluent AI response carry identical confidence signals regardless of whether the underlying data supports them.

The discipline that analytics work builds — treating output confidence as a prompt to examine the underlying data rather than as confirmation of its reliability — transfers directly. The question “what data produced this output, and how confident am I in that data?” applies to AI outputs as precisely as it applies to dashboard numbers. The difference is that AI outputs are harder to trace back to their inputs, which makes the question harder to answer but more important to ask.

Lesson Three: Governance Is the Last Investment and Always Costs the Most

In every analytics engagement I have been part of, governance was the last piece to get funded. Teams built dashboards first. They integrated data sources. They deployed analytics platforms. When those investments started producing inconsistent outputs, they went looking for the problem and found it in the absence of governance: no agreed metric definitions, no data ownership, no documentation of how transformation logic was applied or why.

Retrofitting governance is expensive. The tables that existed before anyone asked “what does this column actually mean?” are the ones that produce the most difficult remediation work. The inconsistencies that accumulated before ownership was assigned are the ones that require the most careful untangling. The technical debt is not the cost of building governance. It is the cost of having deferred it.

AI is following the same sequence, compressed into a shorter timeframe. Organisations are deploying models and building RAG pipelines before establishing what the data those systems will consume is, who owns it, what it means, and how it will be maintained. The governance problems are accumulating. The remediation cost is accruing. The discovery will come when models start producing outputs that vary unexpectedly, or when an audit question arrives that the data lineage cannot answer.

The analytics lesson is not that governance must be complete before anything else is built. That is not feasible in practice and was never realistic in analytics either. The lesson is that governance investment should be proportional to the stakes of what is being built on top of it. Governance for a dashboard that three people read is a different investment than governance for a model whose outputs influence customer decisions at scale. AI raises the stakes. The governance investment should follow.

Lesson Four: Integration Surfaces the Assumptions You Did Not Know You Were Making

The moment in analytics projects that produces the most productive discomfort is when two data sources are integrated for the first time. The integration almost always reveals that the two sources embedded different assumptions about the same concepts: different definitions of “customer,” different granularity for “transaction,” different logic for handling refunds, different timestamp conventions, different interpretations of what “active” means.

Neither source was wrong in isolation. Each was built to serve a specific purpose, and the assumptions were appropriate for that purpose. The problem emerges when they are brought into contact, because the assumptions were never made explicit and are therefore invisible until the integration fails to reconcile.

Integration surfaces the assumptions embedded in how data was originally produced. AI pipelines that integrate across multiple sources inherit all of those assumption conflicts at once.

AI data pipelines are typically multi-source from the start. Training data comes from multiple systems. RAG retrieval pulls from multiple knowledge bases. Features are computed from multiple upstream tables. Every integration point is a potential conflict between implicit assumptions that were never resolved. The larger the pipeline and the more diverse its sources, the more assumption conflicts accumulate.

The analytical habit of making assumptions explicit before integrating — agreeing on canonical definitions, documenting the logic, surface the conflicts before they become runtime failures — is the right approach applied to AI pipelines. It is slower upfront. It produces significantly less expensive problems downstream.

Lesson Five: The Tool Is Never the Answer, But It Is Always Blamed First

Over fifteen years, the marketing analytics stack I worked on changed substantially: web analytics platforms, attribution tools, data warehouses, ETL frameworks, BI layers, CDPs. Each generation of tooling arrived promising to solve the problems the previous generation had not solved. Each generation of tooling was adopted with high expectations. Each generation eventually revealed that the problems were not in the tool but in the data, the definitions, and the processes that governed both.

This is not a criticism of the tools. Most of them were well-built products that did what they claimed to do. The problem was the sequencing of the investment: tooling adopted before the data layer was ready to support it, before the definitions were agreed, before the governance was in place to make the tool’s outputs reliable.

AI is receiving the same treatment at scale. Organisations are investing heavily in model capabilities, in AI platforms, in infrastructure for deploying and serving models. These are real investments in real capabilities. The organisations that will see the most return on those investments are the ones that have sequenced the data and governance work correctly — not as a subsequent project, but as the prerequisite. A capable model on poorly governed data performs below a less capable model on well-governed data. This was true for BI tools. It is true for AI.

What Analytics Experience Changes About How You Build AI

Working on analytics before working on AI systems changes the way I approach the AI layer. Not because the technologies are the same — they are not — but because the failure modes are the same, and recognising them early is worth something.

When I assess an organisation’s AI readiness, I start with the data layer questions that analytics work trained me to ask: where is data quality characterised, where are schema contracts enforced, where are metric definitions agreed and documented, where does lineage break down? These are not AI-specific questions. They are data infrastructure questions that determine the ceiling on what any system built on top of that infrastructure can reliably do.

The organisations that get the most out of AI quickly are, in my experience, the ones that already did the data work. Not because they were planning for AI — they were not. But because the discipline of building reliable analytics infrastructure requires solving the same problems that AI readiness requires. The data foundation is the same. The AI workload is new. The prerequisite work was already done.

For everyone else, the lesson from analytics is that the prerequisite work does not go away. It can be deferred, but deferral is a cost, not a saving. The organisations that paid for analytics governance ten years ago are paying that cost once. The ones that are now building AI on ungoverned data are paying it twice, in a higher-stakes context, with less time available than they had the first time.

Key Takeaways

FAQ

Do these lessons apply to smaller organisations building AI for the first time?
Yes, but the stakes and the remediation cost scale with the organisation. A small team with a small, well-characterised data set can move quickly and retrofit governance as the system grows. The pattern still holds — problems are upstream of where you expect them — but the cost of learning it the hard way is lower. The risk is that habits formed in a small-data context do not scale, and governance debt that was manageable at small scale becomes a structural problem as data volume and AI workload complexity grow.
Are there things AI governance needs that analytics governance did not?
Yes. Analytics governance governs what the data means and how it is produced. AI governance needs an additional layer: governing what AI systems are permitted to generate, not just what data they consume. This is the architectural governance layer — the constraints that prevent AI coding agents from violating architectural decisions, or AI customer-facing systems from producing outputs outside defined boundaries. Analytics experience provides the data governance foundation; AI adds the generation constraint layer on top.
How do you prioritise which data governance gaps to address before building AI?
Start with the data the AI system will actually consume first, not the entire estate. Characterise it: freshness, schema consistency, ownership, lineage, definition agreement. Then ask which gaps most directly affect the reliability of the specific AI output you are building. A recommendation engine has different sensitivity to governance gaps than a RAG pipeline for customer support. The priority order follows the sensitivity of the use case, not the completeness of the governance coverage.
Is there a stage where governance investment is ‘enough’ for AI?
No, in the sense that governance is an ongoing practice rather than a project with a completion state. But there is a threshold below which AI systems are not reliably useful — where data quality variance is high enough, or definition inconsistency is pervasive enough, that the AI outputs cannot be trusted without manual verification of every result. Getting above that threshold is the first governance objective. Improving coverage and consistency beyond it is a continuous investment, calibrated to the stakes of what the AI system is doing.