Atlas AI
Natural language to production dbt models, grounded in the catalog your organization already has.
How it holds together
Step 1 of 8
- Plannerintent · plan
- Metadata AnalystDataHub query
- Context reviewhuman approval
- Data EngineerSQL · dbt models
- QAvalidation · PII
- Documentationmodel + column docs
- Writeback approvalhuman approval
- Writebackdocs + lineage
Interprets the request and decomposes it into the work the downstream agents will do.
The problem
A business question — daily revenue by customer segment for the last ninety days — becomes a ticket, which sits in a queue, which becomes a data engineer hunting for the right tables, checking which columns are trustworthy, writing SQL and a dbt model, adding tests, writing docs and updating the catalog. Hours to days per request, and almost none of it creative.
Point a language model at it and you hit a specific wall: it hallucinates schemas. It invents column names that sound plausible and joins on keys that do not exist. Ungrounded generation is worse than useless, because the output looks correct.
The gap between an LLM writing SQL and an LLM writing SQL that runs in your warehouse is entirely a metadata problem — and organizations already solved half of it. DataHub holds the schemas, ownership, lineage and PII tags. That knowledge graph is exactly the context the model needs, sitting unused next to it.
What I built
A six-agent pipeline that turns a natural-language request into production-ready data artifacts — dbt models, SQL, tests, documentation and metadata — every one grounded in the organization's DataHub knowledge graph.
Two genuine human approval gates: one after metadata matching, before a line of SQL is generated, and one before anything is written back to the catalog.
The architecture
The pipeline splits into three resumable phases around the two checkpoints. Each phase reconstructs RunContext from persisted agent_runs.output_json rather than holding in-memory state across HTTP requests, so a pause survives a restart and each phase resumes independently.
Deliberately a modular monolith. The v1 design was enterprise microservices — Temporal, NATS, Kubernetes, service mesh — and collapsing it was correct for the scope. LLMProvider, DataHubGateway and Agent were preserved as clean interfaces, so any agent can later be extracted behind the same Agent.run() contract.
Every external dependency sits behind an interface. The system runs fully standalone in mock mode and switches to live backends purely through configuration.
Decisions
Early on, DataHub was load-bearing in execution and completely invisible in the interface — buried in output_json and one scrolling log line. For a tool whose entire pitch is grounding, that is a product failure regardless of correctness. It drove the context review panel: matched datasets, confidence scores, columns and PII tags sitting directly beside the generated SQL.
Stack
- Backend
- FastAPI · Python · asyncpg · SQLAlchemy 2.x async · WebSockets
- Model
- Groq · gpt-oss-120b · LLMProvider interface
- Metadata
- DataHub OSS · MCP · REST / GraphQL gateway
- Interface
- Next.js App Router · TypeScript · Tailwind v4 · Motion · Recharts · Zod v4
- Deploy
- Vercel · Render · Neon · Apache 2.0
What it demonstrates
- Retrieval grounded in a real organizational knowledge graph
- Human-in-the-loop gates that survive a restart
- PII-aware QA against field-level catalog tags
- Live execution streamed over WebSocket
- Interface-boundary design that keeps a monolith extractable