A crew that works in a notebook needs four decisions before users reach it. Put crews inside Flows with persisted state, make every tool with side effects safe to run twice, and check CrewAI's token counts against the provider's bill. Then settle telemetry and where the crew runs.
When you need this
- A CrewAI prototype is moving from a laptop to a service that other people's requests reach.
- Agents call tools that send messages, move money, or change records in other systems.
- The team must decide between CrewAI AMP and running the library in its own service.
Evaluation criteria
Flows around crews
What decides which crew runs next?
Look forUse a Flow with start, listen, and router steps so code chooses the next crew, and keep each crew's task small. Builders report agents that collaborate awkwardly without a spec, and a delegated coworker that ran a tool the orchestrator never had.
Persistence and restore
What survives a crash or a deploy?
Look forAdd @persist to keep Flow state in SQLite or your own backend, and turn on checkpointing for long runs. Restore skips completed tasks and reruns the rest. Diagrid's CTO notes that persisted state does not resume by itself, and one issue shows a failed task skipped after restore, so test restore on a forced failure.
Tool idempotency
What happens when a tool runs twice?
Look forGive every tool with side effects an idempotency key. Builders report tools rerun on task retry, failed tool calls retried inside each outer attempt, and a model that reported a search it never ran.
Token accounting
Do CrewAI's counts match the bill?
Look forCompare crew.usage_metrics with the provider's billing for a week. Builders found counts doubled when agents shared one LLM instance and Bedrock counts that left out cache reads. Scope each task's context, since default context passing makes a five-agent crew send about six times the tokens of isolated tasks.
Models and memory
Which vendors see your data?
Look forSet each agent's model as provider/model-id and supply each provider's key. memory=True embeds with OpenAI text-embedding-3-large by default, so configure another embedder if memory content must stay off OpenAI. An agent with no model set falls back to gpt-4.1-mini in the library's code. Providers without a native integration run through LiteLLM; pin it, since CrewAI pinned LiteLLM 1.82.6 during the March 2026 compromise.
Telemetry and tracing
What leaves your systems by default?
Look forAnonymous telemetry is on by default; set CREWAI_DISABLE_TELEMETRY or OTEL_SDK_DISABLED to stop it, and keep share_crew off. AMP tracing needs CREWAI_TRACING_ENABLED=false. Builders reported telemetry still sent after disabling it and a tracing prompt that added about 20 seconds per run, so confirm with a network capture.
Hosting and governance
Where does the crew run, and who can change it?
Look forAMP Basic stops at 50 executions a month. Enterprise adds SSO with Microsoft Entra or Okta, role-based access control, PII redaction for traces, and deployment in your VPC, with execution overage quoted per customer. Self-hosting puts workers, storage, and scaling on your team.
Version pinning
Which release does production run?
Look forPin crewai and its extras. CrewAI shipped five releases between October 14 and October 27, 2026, and builders report upgrades that called tools repeatedly or sent Anthropic-only fields to other providers. Async guardrail retries and concurrent flow setup have open crash reports. Every release requires chromadb 1.5.9, which GitHub advisory GHSA-f4j7-r4q5-qw2c rates critical with no patched version, so clear it with your security team or isolate the service; Headroom and ai-agent-assembly dropped CrewAI from their lockfiles over it.
Questions to ask vendors
- Which tools have side effects, and does each carry an idempotency key?
- Has restore from a checkpoint been tested on a forced failure?
- Do crew.usage_metrics and the provider's bill agree for the same week?
- Is telemetry off, share_crew off, and AMP tracing set on purpose?
- Above 50 executions a month, is the plan an Enterprise quote or self-hosting, and who owns it?
Red flags
- Payment, email, or record-changing tools with no idempotency key.
- Token budgets set from CrewAI's own counts without checking the bill.
- memory=True on sensitive data with the default OpenAI embedder.
- Unpinned crewai and LiteLLM versions in production.
- A production workload planned on Basic's 50 executions.
- A crewai install that brings ChromaDB 1.5.9 into a service your security scanner covers, with no plan for the open advisory.
Products in this piece
Sources
- Official documentation
- 12
- Public source
- 17
- CrewAI docs: Flows (state and @persist)
- CrewAI docs: Checkpointing
- CrewAI docs: LLMs (model setup and providers)
- CrewAI docs: Memory (default embedder)
- CrewAI docs: Telemetry
- CrewAI docs: CrewAI Tracing
- CrewAI pricing (AMP Basic and Enterprise, executions, overage)
- CrewAI AMP docs: PII Redaction for Traces
- CrewAI AMP docs: Just-in-Time SSO provisioning
- CrewAI releases on GitHub (1.15.32, Oct 27, 2026)
- Hacker News comment on CrewAI agents collaborating awkwardly without a spec (Sep 9, 2026)
- Rafael Asor on dev.to: I delegated a review in CrewAI, and the reviewer ran an export tool the orchestrator never had (Sep 3, 2026)
- Yaron Schneider (Diagrid): Why Checkpoints Aren't Durable Execution, LangGraph, CrewAI, Google ADK, and Others (Feb 25, 2026)
- CrewAI issue #8718: a failed task skipped after checkpoint restore (Oct 28, 2026)
- CrewAI issue #5802: tool re-execution on task retry has no idempotency guard (May 14, 2026)
- CrewAI issue #7449: a failed tool call retried twice per outer attempt (Sep 14, 2026)
- CrewAI issue #3154: agent simulates tool use with fabricated output (Jul 14, 2025)
- CrewAI issue #7259: usage_metrics multiplied by agents sharing an LLM instance (Sep 4, 2026)
- CrewAI issue #8626: Bedrock prompt_tokens excludes cache reads (Oct 26, 2026)
- wartzarbee on dev.to: CrewAI's quadratic context problem, why a 5-agent crew costs 6x more (Jul 26, 2026)
- Hacker News comment on CrewAI pinning LiteLLM 1.82.6 during the March 2026 compromise (Mar 24, 2026)
- CrewAI issue #2945: telemetry still sent after turning it off (Jun 3, 2025)
- CrewAI issue #3789: tracing prompt adds about 20 seconds per run (Oct 24, 2025)
- CrewAI issue #7252: kickoff_async guardrail retry crashes, filed by an AB InBev engineer (Sep 3, 2026)
- CrewAI source: pyproject.toml, which requires chromadb~=1.5.9
- CrewAI source: constants.py, DEFAULT_LLM_MODEL gpt-4.1-mini
- GitHub advisory GHSA-f4j7-r4q5-qw2c: ChromaDB pre-authentication code injection, critical (May 18, 2026)
- Headroom pull request #3870: drop the crewai extra to remove chromadb from the lockfile (Sep 30, 2026)
- ai-agent-assembly pull request #133: remove crewai to fix a critical chromadb vulnerability (Jul 7, 2026)