What Are the Most Common Data Problems and How Do You Fix Them?

Why Do the Same Data Problems Keep Showing Up?

Every data team has a version of the same story: pipelines that looked fine in development break in production, models accumulate technical debt faster than features, and nobody is sure when the data last matched reality. The Data Valentine Challenge — a five-day event where companies tackled real data problems live — confirmed that these issues are nearly universal.

Data reconciliation failures, untested AI agents, and sprawling dbt projects are not edge cases. They are the baseline state of most data platforms. The question is not whether your team has these problems, but whether you have a systematic way to find and fix them.

What Are the Five Most Common Data Problems?

The challenge surfaced five distinct categories, each representing a different failure mode in the data lifecycle:

Day	Problem	Root Cause	Fix
1	AI agent benchmarking gaps	No standardized evaluation for data-specific AI agents	Define repeatable test suites with known-good datasets
2	Cross-system data discrepancies	Data drifts between warehouses, files, and APIs	Use DuckDB for lightweight reconciliation without infrastructure
3	Fragile no-code pipelines	Visual pipeline builders hide complexity and failure modes	Add validation checkpoints and schema contracts at pipeline boundaries
4	dbt project sprawl	Models accumulate without cleanup or ownership	Audit with DAG lineage, remove orphaned models, enforce naming standards
5	Missing data versioning	No record of what data looked like at a given point in time	Implement snapshot strategies and change tracking on critical tables

Each problem was tackled live, with real teams showing their actual workflows — not sanitized demos.

How Do You Benchmark AI Agents on Data Tasks?

AI agent benchmarking for data tasks is still in its early stages. Unlike traditional software testing where inputs and outputs are deterministic, AI agents produce variable results that need evaluation against business-specific criteria.

The challenge revealed a practical approach:

1. Define a reference dataset with known correct answers
2. Run the agent against the dataset under controlled conditions
3. Measure three dimensions: accuracy (did it get the right answer?), latency (how long did it take?), and cost (what did it spend in tokens or compute?)
4. Track over time to detect regressions when models or prompts change

This matters for data review because teams increasingly rely on AI agents to validate data changes — and an agent that gives confident but wrong answers is worse than no agent at all. Understanding what an AI data review agent actually does is the first step toward benchmarking one effectively.

How Do You Reconcile Data Across Systems with DuckDB?

Data reconciliation — comparing datasets across different systems to identify discrepancies — traditionally required dedicated infrastructure. DuckDB changes this by running as an in-process analytical engine that can query CSV files, Parquet files, and database exports without a server.

A typical reconciliation workflow:

1. Export source data from your warehouse and target system
2. Load both into DuckDB (locally or in CI)
3. Run comparison queries — row counts, column distributions, value-level diffs
4. Flag discrepancies for investigation

This approach complements the data diff techniques that catch issues before they reach production. The key advantage is speed: you can run a reconciliation in seconds without provisioning anything.

How Do You Clean Up a Sprawling dbt Project?

dbt projects grow organically. Models get added for one-off analyses and never removed. Tests reference columns that no longer exist. Documentation covers models nobody uses. Over time, the project becomes a liability — every change risks breaking something nobody understands.

The cleanup process starts with visibility:

• Map your DAG lineage to understand which models feed which downstream consumers. DAG lineage analysis reveals the actual dependency graph, not just the intended one.
• Identify orphaned models — models with no downstream dependencies that exist only because nobody deleted them.
• Consolidate duplicate logic — look for CTEs or models that compute the same metric differently in different places.
• Enforce naming conventions going forward to prevent the same sprawl from recurring.

The challenge showed that teams who invested a single day in cleanup eliminated 15-30% of their models with zero impact on downstream consumers.

What Does Data Versioning Actually Look Like in Practice?

Data versioning means tracking what your data looked like at a specific point in time — not just what it looks like now. Without versioning, you cannot answer basic questions like “when did this metric change?” or “what did this table look like before the last deploy?”

Practical data versioning approaches include:

• dbt snapshots for slowly changing dimensions in your warehouse
• Git-based versioning for seed files and configuration
• Change data capture (CDC) for tracking row-level mutations in source systems
• Environment comparison — diffing development data against production before merging, as part of your CI checks beyond tests

The challenge confirmed that most teams have no versioning strategy at all. They discover data changed only when a stakeholder reports something wrong — days or weeks after the fact.

What Did the Data Valentine Challenge Prove?

The biggest takeaway was not any individual fix. It was confirmation that data problems are systematic, not accidental. Every team faces the same categories of failure, and the teams that manage them well share a common trait: they have processes for validating data changes before those changes reach production.

Whether that process involves AI agent benchmarking, DuckDB reconciliation, dbt cleanup, or data versioning, the principle is the same — validate the data, not just the code.