Jump to Content
Data Analytics

Frontier and Center: Why your data agents need a “wind tunnel”

November 20, 2026
Eva Sharma

Software Engineer, Data Cloud Frontier AI

Sunil Pedapudi

Technical Lead, Data Cloud Frontier AI

Try Gemini today

The front door to AI in the workplace

Try now

Imagine you're building a data agent to answer questions about an enterprise data warehouse. Ideally, you'd develop and test it against production data, but access restrictions and data sensitivity make that difficult – especially when the engineers building the agent sit outside the team that owns the data. Even with access, a live warehouse is a poor test track: Catalog descriptions are often incomplete or contradicted by institutional knowledge in docs and chat threads; there’s no verified answer key across its thousands of tables; and you can't systematically vary the complexity of the data and documents to see where the agent breaks. So teams often test on toy datasets and hope the agent holds up in production.

Other disciplines test differently. Automobile designers study aerodynamics by putting a scale model in a wind tunnel, which reproduces the airflow forces that matter while ignoring surface details like paint and badges. A shape that performs well in the tunnel performs well on the road because the tunnel preserves the physical properties the shape responds to.

Data agents need a wind tunnel too: a synthetic data estate (the databases, documents, and chat history where a company's knowledge lives) with the same structure and causal behavior as a real organization, while synthesizing every name, number, and document. It is designed to preserve what the agent reasons over: how tables relate, what business events create which records, where institutional knowledge lives, and how definitions get messy. When those properties hold, an agent's behavior in the tunnel tracks its behavior on production data, even though the two share no sensitive content.

At Google, we built an internal data simulator to serve as a wind tunnel for our enterprise data agents. Rather than copying proprietary schemas or cataloging every undocumented quirk in a live warehouse, an engineer starts with a short, plain-English description of organizational patterns, for example: "Simulate a two-plant brake manufacturer with an ERP, a shop-floor execution system, and a three-shift schedule where second-shift supervisors track downtime differently from operations and finance." From just a few sentences, the simulator invents a structurally analogous company ("Calder Brakes"), generating its schemas, rows, event history, documents, and chat threads alongside a benchmark of hard questions and a verified answer key.

https://storage.googleapis.com/gweb-cloudblog-publish/images/1_JHmwdEP.max-1100x1100.png

Figure 1. Production data and its synthetic counterpart share causal structure without sharing sensitive content. A short description of domain patterns seeds the wind tunnel, where the agent is evaluated over read-only tools and behavioral trends carry back to production.

Why clean test databases fall short

Why did we do this? Most data agents are evaluated on small, clean test databases with well-labeled tables, unambiguous metrics, and no conflicting documents. Those datasets are convenient, but they rarely test the judgment that enterprise data demands.

Once generated, a synthetic estate is served through a read-only interface — SQL execution, schema browsing, and document search — that any agent can call without seeing the blueprint or answer key. But consider what happens when an agent is asked either a qualitative onboarding question or a quantitative metric question from our simulated factory, "Calder Brakes Plant 2":

  • Qualitative: "A new analyst is joining the team. What should they know about how 'downtime' is handled here, before they start producing reports?"

  • Quantitative: "How many hours of downtime did Plant 2 log in Q3?"

A naive agent falls into the same trap on both questions. It searches for the most authoritative document and finds a policy memo from Bob, a second-shift supervisor, which states that downtime means unplanned stoppages only, and planned changeovers don't count. Bob is confident, articulate, and senior, so the agent summarizes his definition — and filters out planned changeovers when writing its SQL query.

Both answers are wrong. In fact, the plant's official metric counts changeovers as planned downtime. Bob sincerely believes his version because it flatters his shift's numbers, but weeks later the metric owner quietly corrects the record in a follow-up thread. Getting either question right requires looking past a document's tone, checking who owns the metric before writing SQL, and tracing later corrections. This is where data agents often fail in production.

How the simulator builds a trusted benchmark

A synthetic estate is only useful for evaluation if its answer key is accurate and reproducible. Our simulator achieves this by separating simulated company’s design from data generation across four stages (Figure 2).

https://storage.googleapis.com/gweb-cloudblog-publish/images/2_pMgxVcc.max-900x900.png

Figure 2. One pipeline, one source of truth. In the estate-generation pipeline, only ① Compile blueprint invents facts and ③ Render documents phrases those fixed facts as prose (blue boxes); ② Simulate & build tables (plus SQL answer computation) runs deterministically from the frozen blueprint and seed, before ④ Evaluate agent tests the agent over read-only tools.

First, the Compile blueprint stage uses a language model to turn the pattern prompt into a frozen blueprint specifying the company's systems, people, metrics, and individual beliefs. Those beliefs can be mistaken on purpose — Bob's flawed definition of downtime is recorded alongside the official metric, planting the trap and its solution together.

In the estate-generation pipeline, Compile blueprint is the only point where a language model decides what is true. Once the blueprint is frozen, Simulate & build tables runs deterministically without a model: employees act on their schedules, triggering tickets, shift handovers, and database rows, so that every record traces back to the event that created it. Organizational messiness emerges naturally from this timeline, such as outdated procedures or conflicting definitions, while row-level flaws that model real ingestion glitches (e.g., retry-storm duplicate rows, unmerged keys, or a mid-quarter unit change) are injected into the tables. Every flaw is recorded in a machine-readable messiness log so the benchmark generator can adjust the ground-truth SQL queries to account for the messy tables.

Given the same blueprint and random seed, the simulator produces identical tables and an identical SQL answer key on every run. The only subsequent step in building the estate that calls a model is Render documents, which turns the facts fixed in the blueprint into memos and chat threads in each person's writing style. Finally, Evaluate agent tests the agent over read-only tools.

Three build-time checks keep the benchmark honest:

  • Computed SQL and rubric gold answers: Every quantitative question ships with the SQL query that computes its expected ("gold") answer against the messy tables. Because the messiness log records injected retry duplicates on defect rows, the benchmark generator automatically pairs a defect-count question with the deduplicating COUNT(DISTINCT) query (240) – and runs decoy SQL queries representing common mistakes (like an un-deduplicated COUNT(*) or Bob's changeover filter) to verify that falling for a trap yields a different number. Qualitative questions ship with a rubric grounded in the blueprint's metric owner, official definition, and correction trail. If the estate lacks the data to verify a question, the generator drops the item rather than shipping an unverified question.

  • Verified documents: Every rendered memo and chat thread is checked against the blueprint to confirm it states its assigned facts without introducing unrecorded numbers, dates, or names.

  • Hallucination sentinels: The simulator plants questions about invented metrics like "Basalt Velocity Ratio" that appear nowhere in the estate; a reliable agent abstains and reports that no such metric exists.

Testing for behavioral trends and realism

A wind tunnel does not need to match the exact score an agent gets on production data; it needs to reproduce the same behavioral trends: direction (if longer reasoning chains lower accuracy on real data, they should lower accuracy in the tunnel) and rate (accuracy should degrade at the similar pace as contradictions accumulate). When those trends hold, you can confidently compare agent versions and dial up complexity step by step to find where reasoning breaks down.

We test how closely our simulated estates match reality — and whether evaluation signals carry over — at three levels:

  • Statistical properties: Every build produces a realism report measuring structural properties that affect SQL generation and joins — including row multiplication across table joins (join fan-out), key skew, null and duplicate rates, event durations, and day-of-week activity curves — alongside numeric distribution checks such as leading-digit frequencies (Benford's law).
  • Domain expert review: In early engagements with enterprise partners, we built synthetic estates modeled on their domains and had their analysts query the data to judge whether it behaved like their own systems.
  • Independent calibration and carry-over checks: We run the simulator's output through an independent calibration suite of more than a hundred checks covering statistical realism and answer-key SQL verification. In our internal evaluations, capability upgrades — such as switching an agent to a deeper reasoning mode or improving how it handles context — move accuracy in the same direction and rank agent configurations in the same order on both synthetic estates and human-authored benchmarks. Broader multi-agent studies across paired real and synthetic customer estates are now underway.

How our approach builds on prior work

Our simulator builds on two research threads. In database question answering, benchmarks like BIRD and Spider 2.0 have moved from clean schemas to messy databases, while agent environments like WebArena, tool-use benchmarks like τ-bench, and Generative Agents simulate websites or use language models to role-play human users. In contrast, our simulator's employees follow deterministic rules tied to their roles and schedules, so a single seed replays the exact same organizational history every time.

We also adapt an intuition from domain randomization in robotics, where simulations vary surface details like lighting and textures during training while keeping the underlying physics faithful. Applied to data agent evaluation, that means synthesizing every surface name, number, and document from a short pattern description, while keeping the causal structure, cross-document links, and institutional messiness faithful to enterprise reality.

Hardening agents without customer data

Generating a test estate on demand removes a bottleneck between enterprises that own sensitive data, and the engineering teams building data agents for them. Traditionally, developing and benchmarking an enterprise agent required access to production tables and a hand-labeled question set — the two assets hardest to share across organizational or compliance boundaries. We use this wind tunnel internally to stress-test and harden Google Cloud's data agents against realistic enterprise messiness without touching customer data, and the same blueprint-first architecture can be applied by any engineering team building evaluation suites for their own agents.

Synthetic evaluation still carries two risks: overfitting to the simulator's patterns of messiness, and drift as a live business evolves away from its blueprint. The discipline required is the same one that wind-tunnel engineers practice: Periodically calibrate the tunnel against real-world runs, keep benchmark difficulty aligned with production failure modes, and treat the simulation as a living model. Your agents — and their users — will thank you.

Posted in