Before I moved into AI project management consulting and oncology, I spent years working as a data steward. That experience shaped how I think about data more than any course or certification has. You learn things in that role that are difficult to teach abstractly. You learn that data does not simply exist, ready and waiting to be used. It is produced, collected, stored, transformed, governed, and sometimes lost, by organisations that are often more focused on the business processes that generate the data than on the data itself. You learn the gap between what a data asset is called and what it actually contains. You learn how much time gets consumed by questions that should have been answered at the start of a project.
This post is about that foundational phase: data understanding. It is not glamorous. It does not generate the kind of excitement that model building or AI deployment does. But it is where projects are won or lost, often quietly and well before anyone realises it.
From raw data to wisdom: a useful way to think about value
One of the most useful frameworks I have encountered for thinking about what data actually is, and what it can become, is the DIKUW pyramid. It describes five levels through which raw information is progressively transformed into something of greater value:
Data is the raw material: individual facts, records, sensor readings, text strings, images. On its own, data has no inherent meaning. It simply exists.
Information is organised data. When you arrange data to answer the questions who, what, where, and when, you turn it into something a person can read and understand. A spreadsheet of sales transactions becomes a monthly report.
Knowledge is where patterns emerge. This is where machine learning, analytics, and data science operate. At this level you are not just reading data, you are learning from it: identifying trends, making predictions, and automating decisions. This is the sweet spot where AI creates its most practical value.
Understanding represents deeper reasoning, the ability to generalise from what is known to novel situations. This is emerging capability in AI, partially achievable but not reliably so.
Wisdom is human judgment informed by experience and values. We are not close to AI operating reliably at this level.
The lesson I take from this pyramid is practical rather than theoretical. Too many data projects stop at the information level. They produce reports, dashboards, and summaries. These have genuine value, but they leave the knowledge layer largely untouched. The real opportunity, and the harder work, is in moving upward. And that journey starts with understanding what data you actually have.
What we mean by big data
The term big data gets used loosely, but it has a precise meaning that is worth holding onto. Big data is not defined by a particular size threshold. It is defined by the challenges it presents. Even a relatively small dataset can qualify as a big data problem if those challenges are present in sufficient intensity.
Those challenges are traditionally described through four dimensions:
Volume refers to the sheer quantity of data. We have been in the zettabyte era since roughly 2012. One zettabyte is a billion terabytes. The volume of data being generated continues to grow exponentially, driven by IoT devices, social media, transactional systems, scientific instruments, and digital services of every kind. No single product or tool fully solves the volume problem. What you need is infrastructure that scales.
Velocity is about how fast data changes or needs to move. Stock market data updates every millisecond. Aircraft sensors generate readings continuously in flight. Social media produces millions of posts per minute. For many applications, data that arrives late is data that has already lost its value. The architecture that handles batch updates every night simply does not work when you need to act on data as it arrives.
Variety describes the heterogeneity of data formats, sources, and structures. Roughly 80 percent of an organisation's data is unstructured: documents, emails, images, audio, video, social media posts. Another portion is semi-structured, carrying some organisation through tags or metadata but without a rigid schema: JSON files, XML documents, email headers, invoice formats. Only a minority of organisational data is fully structured in the way a relational database requires. Any serious data project has to grapple with all three.
Veracity is the dimension that I think matters most in practice and gets the least attention. It refers to data quality: accuracy, consistency, completeness, and trustworthiness. The principle here is blunt. Garbage in, garbage out. No analytical method, however sophisticated, can compensate for data that is fundamentally unreliable. And in the real world, data is almost never as clean as the project plan assumes it will be.
The three types of data: why structure matters
Understanding the structure of data is not an academic exercise. It determines what tools you can use, how much preparation you need, and what kinds of insights are accessible.
Structured data follows a predefined schema. Rows and columns. Relational databases, spreadsheets, CSV files. It is the easiest data to work with analytically, and it is what most traditional business intelligence tools are built around. The limitation is that it represents a relatively small proportion of the data organisations actually generate.
Unstructured data has no predefined schema. Images, video files, audio recordings, free-text documents, social media content. It is difficult to analyse with traditional tools but holds enormous potential value. Natural language processing, computer vision, and audio analysis are all techniques for extracting structure and meaning from unstructured sources. The challenge is that this work is genuinely hard and requires careful data preparation before any analysis can happen.
Semi-structured data sits between the two. JSON, XML, HTML, log files, invoices, and similar formats carry some organisational information through tags, delimiters, or metadata, but do not conform to a rigid relational schema. Many modern data systems work heavily in this space.
When I was working as a data steward, the variety problem was constant. Data about the same underlying entity would arrive from three different systems in three different formats, with different field names, different date conventions, and different approaches to handling missing values. Reconciling these was not optional. It was the work.
Ground truth: the foundation of any reliable analysis
Ground truth data is a term that deserves more attention outside of machine learning circles. It refers to data gathered directly from real-world sources: actual users, real transactions, genuine events, actual physical measurements. It is the evidence base against which everything else is validated.
The challenge is that ground truth data, precisely because it comes from the real world, is messy. It is incomplete. It contains errors, inconsistencies, and missing values. Important fields may have never been captured, or were captured differently across time or geographies. Labelling and annotation, where required, introduces its own variability depending on who did it and how clear the guidelines were.
None of this means ground truth data should be avoided. It means it should be treated with respect and examined carefully before it is used. One of the most widely cited examples of this going wrong in AI is the ImageNet dataset, which was found to have a roughly five percent label error rate: images of deer tagged as magpies, mushrooms labelled as spoons. At scale, those errors propagate through training in ways that are difficult to trace and harder to fix.
Data quality: the most underestimated challenge
In my experience as a data steward, data quality issues were the single largest source of delay and rework across projects. Not because the data was unusable, but because no one had taken stock of its actual state before deciding what they wanted to build with it.
Data quality has several distinct dimensions, and each matters:
Accuracy is whether the data correctly represents the real-world entities or events it is supposed to describe.
Completeness is whether all the required data is present. Missing values are not just an inconvenience. In some contexts they are systematically missing in ways that introduce bias.
Consistency is whether the same entity is represented the same way across different systems and time periods.
Timeliness is whether the data is current enough to be useful for the intended purpose.
Validity is whether the data conforms to expected formats, ranges, and business rules.
Uniqueness is whether records are deduplicated appropriately.
Poor data quality does not just produce bad analysis. It produces bad decisions. Employees who cannot trust the data they are given create their own copies. Those copies diverge. The organisation ends up with multiple versions of the truth and no reliable way to resolve conflicts between them. The cost of managing this situation compounds over time, and it is almost always greater than the cost of investing in data quality upfront would have been.
Data governance: the policy framework
Data governance is the set of processes, procedures, standards, roles, and tools that an organisation puts in place to ensure its data is properly stored, managed, accurate, available, secured, and controlled across its life cycle.
The scope of effective data governance is broad. It needs to address security and privacy risks, uptime and availability, data quality standards, data ownership and intellectual property, data provenance, access control, data sharing, deletion and archival, and compliance with applicable regulation. None of these are purely technical problems. They require decisions about authority, accountability, and priorities that only the organisation can make.
A functional governance framework does not happen by accident. It requires someone to define the mission: why does governance exist here, and what outcomes is it trying to achieve? It requires decision rights: who has the authority to approve changes to data structures, access policies, or retention rules? It requires processes for managing conflict, because disagreements about data ownership and access are inevitable in any organisation of meaningful size.
The USGS Data Management Plan checklist captures this comprehensively. Before a project even starts, it asks: who has overall data management responsibility? What are the data formats and storage requirements? Who is responsible for preservation, quality control, and backup? How will metadata be created and maintained? What sharing agreements are in place? These are not bureaucratic formalities. They are the questions that, if left unanswered at the start, consume disproportionate time and energy in the middle of a project.
Data stewardship: where governance meets practice
Governance defines the what and the why. Stewardship defines the how.
A data steward is the person or role responsible for ensuring that governance policies are actually followed in practice. That means knowing what data the organisation holds and where it lives. It means ensuring that data is accessible to those who need it, secure from those who should not have it, and trustworthy for those who use it. It means enforcing data quality standards, maintaining data lineage documentation, and acting as an advocate for good data practice within the organisation.
The distinction between data stewards and data custodians is worth making clear. A data custodian manages the physical storage and infrastructure: servers, databases, backup systems. They keep the data safe and available but are not responsible for its content or use. A data steward, by contrast, is accountable for the data's integrity, quality, and appropriate use. The steward understands what the data represents, how it was produced, what its limitations are, and what decisions it should and should not be used to support.
In my time in that role, the most valuable thing I did was not any particular technical task. It was being the person who knew the data well enough to ask the right questions about it. When a project team assumed a field meant one thing and it actually meant another, when a dataset that looked comprehensive turned out to have a systematic gap, when a data source that was described as real-time was actually updated monthly: these were the situations where having someone who understood the data in depth prevented significant wasted effort.
Informational bias: the quiet distorter
One of the most important and underappreciated aspects of data understanding is the recognition that data is not neutral. The way data is collected, categorised, and recorded reflects the choices and limitations of the people and systems that produced it.
Informational bias takes several forms. Reporting bias occurs when only certain aspects of a situation are captured, leaving others invisible. If a company only records customer complaints that come through a formal channel, its complaint data underrepresents the actual volume of dissatisfaction. Recall bias occurs when certain time periods or contexts are systematically over or underrepresented, skewing trend analysis. Classification bias occurs when the categories used to organise data do not map cleanly onto the underlying reality, or when they group together entities that are meaningfully different.
These forms of bias are distinct from the statistical concept of bias in machine learning models, which refers to a model's tendency to underfit the data. Informational bias is a property of the data itself, not the analysis. And because it is embedded in the data, it is often invisible unless you actively look for it.
This is one of the reasons that data understanding cannot be delegated entirely to technical staff. The people who understand the business context, the history of how data was collected, the organisational incentives that shaped what got recorded and what did not, are essential contributors to any honest assessment of data quality.
The data life cycle: thinking longitudinally
Data does not just exist at a point in time. It has a life cycle, and understanding that life cycle is essential for managing it well.
Data is generated through some process, either captured from the world directly or produced by a system. It is collected and moved to wherever it will be stored. It is accessed by users and systems for various purposes. It is used in analyses, reports, models, and decisions. It is transferred between systems and geographies. It requires security measures at each stage. Eventually, it is either deleted, archived, or transformed into something else.
Each transition in this life cycle carries risk. Data can be corrupted in transfer. Access controls can be misconfigured. Retention policies can be ignored or misapplied. Regulatory obligations can be missed. The USGS checklist illustrates this life cycle approach well: it asks about data acquisition, processing, preservation, publication, metadata, quality management, and backup as distinct phases, each with its own responsibilities and considerations.
For AI and analytics projects specifically, the data life cycle has additional dimensions. Training data and inference data may have different provenance and quality characteristics. Models trained on historical data may perform poorly when the underlying data distribution changes over time. This is the phenomenon of data drift: the real world moves, and data that was representative when collected becomes progressively less representative as conditions change.
The data management plan: making intentions explicit
A Data Management Plan is a document that brings together all of the above into a coherent framework for a specific project or programme. It describes what data will be collected or used, how it will be managed, where it will be stored, who is responsible for what, and how data will be handled at the end of the project.
Its three core components are stewardship guidance, which provides direction on how data should be managed and used; a data sharing agreement, which defines who can access the data, under what conditions, and with what security measures in place; and access and compliance controls, which specify the security and regulatory requirements applicable to the data.
The value of a DMP is not primarily the document itself. It is the thinking it forces. When you sit down to write a DMP at the start of a project, you discover the questions you have not yet answered. You find that the data you assumed you would have access to is subject to restrictions no one mentioned. You find that no one has defined who is responsible for data quality. You find that there is no agreed approach to dealing with missing values. Finding these things before the project starts is far less costly than finding them during.
What good data understanding looks like in practice
After working in data stewardship and studying AI project methodology, I have come to think about data understanding as a set of honest questions that need honest answers before a project proceeds.
Do you know what data you actually have, not what you think you have? Are you clear on where it lives, who owns it, and how long it takes to access? Do you understand its quality well enough to know where it will let you down? Have you identified the sources of bias embedded in how it was collected? Do you know what the data does not contain, and whether those gaps are material to what you are trying to do?
These are not comfortable questions, because the honest answers are often that things are less certain, less clean, and more complicated than the project plan reflects. But asking them early is what separates data projects that deliver from those that produce polished presentations of results no one can trust.
Why this matters for AI
Everything above is relevant to any serious data project. It is especially relevant to AI projects, where the consequences of poor data understanding are amplified by the opacity of the models being trained.
Traditional software fails visibly. A bug produces an error. An AI model trained on poorly understood data fails quietly. It produces outputs that look plausible, that pass surface-level review, and that are only revealed as wrong when they hit the real world. The McDonald's AI voice ordering system, which was shut down after three years of development, almost certainly had this characteristic. The model worked in the test environment it was designed for and fell apart in the noisy, accented, variable reality of actual drive-through customers.
The structured AI project management methodology I work with addresses this directly by treating data understanding as a mandatory phase before model development begins. Not a checkbox. A genuine investigation. The question it asks is not "do we have data?" but "do we understand our data well enough to trust what a model will learn from it?" Those are very different questions, and the gap between them is where most AI project failures originate.
Closing thought
Data understanding is not a technical discipline that belongs only to data engineers and scientists. It is a practice that benefits from deep organisational knowledge, business context, and disciplined curiosity. The best data stewards I have known were not the ones with the most technical skill. They were the ones who asked the most uncomfortable questions about what the data actually represented, and who kept asking until they had answers they could trust.
That instinct, to understand before assuming, to investigate before building, is the most transferable thing I took from that chapter of my career. It applies whether you are designing an AI system, running a clinical data programme, or evaluating a business intelligence platform. The data is never as simple as it looks. Understanding it properly is where the real work begins.
This post draws on my experience as a data steward, course material from the PMI CPMAI certification programme, and the USGS Data Management Plan framework. It is written for a general professional audience interested in data management, AI project practice, and precision in how organisations handle information.