Launch one Genie Agent on a narrow data domain, gate it on the questions business users actually ask, and open it to a small pilot group before Genie One reaches the company. Databricks' documentation and teams' public accounts both point to this order.
When you need this
- The data team spends much of its week on recurring ad hoc questions from the business.
- Leaders want business users asking questions in Genie One, Slack, or Teams.
- A Genie pilot returned numbers people questioned, and the rollout stalled.
Evaluation criteria
Setup: scope the agent
Which tables answer this domain's questions?
Look forUse five or fewer tables, or one metric view, registered in Unity Catalog. Point the agent at pre-joined views. Keep raw joins out of its path and hide confusing columns. Make a domain expert who writes SQL the curator.
Setup: curate context
Where do business definitions live?
Look forWrite table and column descriptions in Unity Catalog, SQL expressions for metrics and filters, and example SQL for the most-asked questions. Use text instructions only for what SQL cannot express. One team found that 15 to 20 verified answers was the point where users stopped rechecking figures.
Governance: gate on benchmarks
What must the agent answer correctly before launch?
Look forWrite benchmark questions with SQL answers, two to four phrasings each, and run them before every change.
Rollout: pilot group first
Who tests it before everyone?
Look forPick a few business users who know the domain, and tell them their job is to rate answers and request reviews. Share the agent through Genie One with permissions set to CAN VIEW or CAN RUN.
Review practice: watch the Monitor tab
Who reads failed and flagged answers each week?
Look forRead the Monitor tab and review requests every week, save each fix as example SQL, and watch for believable wrong numbers. One sales team learned its agent counted Kentucky in the Midwest only when finance checked.
Cost control: budget before scale
What happens when the free promotion ends?
Look forCreate a Genie budget with an alert-only shared threshold during discovery, and add per-user thresholds once usage is known. Watch the SQL warehouse the agent uses separately.
Questions to ask vendors
- Which domain has the most repeated questions and the cleanest metric definitions to start with?
- Who owns each agent's context after launch, and how often do they review its Monitor tab?
- What benchmark score must an agent reach before it opens to a department?
- Do our metric definitions live in metric views, or only in dashboards?
Red flags
- An agent pointed at a whole schema with no descriptions or example SQL.
- An agent launched without benchmark questions, so the first sign of a wrong metric is a leader quoting it. Databricks' Summit pages say Virgin Atlantic's leaders rechecked early AI answers against dashboards until each metric had one signed-off metric view. GSK now holds every Genie space to a gold set of 20 to 50 SQL queries.
- Business definitions written only as text instructions.
- Genie One opened to everyone before any domain agent has passed its benchmarks.
Products in this piece
Sources
- Official documentation
- 11
- Public source
- 4
- Databricks docs: Create and manage a Genie Agent
- Databricks docs: Curate an effective Genie Agent
- Databricks docs: Tune Genie Agent quality
- Databricks docs: Test and monitor a Genie Agent
- Databricks docs: Troubleshoot Genie Agents
- Databricks docs: Genie Agents API
- Databricks docs: Manage budgets and cost controls for Genie
- Databricks customer story: BookMyShow (Sep 23, 2026)
- Databricks Data + AI Summit session: Atlassian's Genie journey from pilot to production (Jun 16, 2026)
- Databricks Data + AI Summit session: scaling Genie 10x at GSK Global Supply Chain (Jun 18, 2026)
- Databricks Data + AI Summit session: Virgin Atlantic's flight to trusted generative BI (Jun 17, 2026)
- Reddit post: notes from 8 months of a Genie space for 500+ sales users (Sep 9, 2026)
- Reddit comment: verified answers and when business users stopped rechecking (Jun 17, 2026)
- SunnyData: Databricks Genie, our experience and lessons learned (Feb 18, 2025)
- Mechanical Rock: Databricks Genie One opens the doors for business users (Aug 10, 2026)