Spider2 Benchmark: Why graph-guided SQL beats naive LLM SQL on real enterprise schemas

The Spider2 benchmark exposes what naive LLM-to-SQL approaches actually do on real enterprise schemas. The results aren't subtle: LLMs without structural grounding hallucinate joins, invent column names, and produce SQL that runs but returns the wrong answer. That last failure mode — plausible but wrong — is the worst kind for an enterprise.

What "join correctness" actually measures

Enterprise databases routinely span 500–3,000 tables. Homonymous column names (status, id, created_at) appear in dozens of tables with incompatible semantics. An engineer who writes joins by naming intuition produces SQL that is syntactically valid but semantically wrong — silently double-counting, missing rows, or correlating the wrong entities.

We measure correctness against a held-out ground-truth set of queries whose results are independently verified. A query is "correct" only if every row, column, and aggregate matches the reference within tolerance.

How the Curve-Context Join algorithm works

For every candidate join pair, Antcluster fits an information curve measuring mutual-information gain from the join. Only joins exceeding a per-cluster threshold τ are accepted. Accepted joins are stored as typed, confidence-weighted edges (JOINS, REF, INFERS) in the Topology Graph.

When the agent constructs SQL, it follows graph edges — not naming heuristics. Structurally invalid joins are never emitted. This is the difference between "looks plausible" and "actually correct."

Token economics

A 1,000-table database's full DDL consumes 300,000+ tokens — far beyond practical context-window limits. The Semantic Memory layer side-steps this: it stores LLM-generated cluster narratives (plain-English summaries of what each table cluster represents) and retrieves only the narratives and node sub-graphs relevant to the current query. Context payload: 800–3,000 tokens.

• Token consumption per query: −85% to −93% vs full-DDL injection
• SQL first-attempt success rate: +34 percentage points

Why this matters for production

Token economics and join correctness aren't academic concerns. They determine whether you can ship a text-to-SQL agent to actual business users without an analyst sitting between every question and every answer. A 61%-correct system shifts work onto the analyst (now they must verify every query). A 96%-correct system is self-service.

Read the full architecture or request the benchmark PDF.

For the general case of why text-to-SQL gets joins wrong on undocumented schemas, and what fixes it, read Text-to-SQL: why it gets joins wrong.