Give each repository a working environment once, write the finish line into the task, keep each session small, and review the result against CI before anyone merges it. The steps follow Cognition's documentation and two public accounts, RunLLM's month with Devin and Answer.AI's month of testing.
When you need this
- Engineers have started handing Devin tickets, and the pull requests come back uneven.
- Sessions stall on setup or end without a change that passes CI.
- You want one team standard for which tickets go to Devin and how its pull requests get reviewed.
Evaluation criteria
Before work: the environment
Does every session boot with dependencies, tools, and secrets ready?
Look forAsk Devin to create an environment blueprint for each repository. Review it before building the snapshot. Cognition ranks environment setup first among the steps that improve results on your codebase. Store credentials in the blueprint's Secrets tab, never in a prompt.
Before work: repository instructions
Where do build commands, test commands, and conventions live?
Look forPut AGENTS.md at the repository root, with critical rules in its first 16 KiB. Devin injects that text automatically. Write repeatable procedures, such as testing before a pull request, in SKILL.md files. Knowledge is deprecated and migrating to Skills, so write new guidance as Skills.
Before work: the finish line
Can the ticket name where the work starts, where it ends, and the check that proves it?
Look forStart with a task an engineer would finish in about three hours, Cognition's rule of thumb. Define success with tests, lint, or CI. Use Ask Devin to scope larger work first. Split the result into sessions that can run in parallel.
Before work: a pattern to follow
Does Devin have an example of the change you want?
Look forInclude a similar merged pull request, or before-and-after examples, in the prompt so Devin has a pattern to follow.
During work: session size
Is the session staying small enough to succeed?
Look forSession Insights sizes finished sessions from XS to XL by usage and flags L and XL as unhealthy. Split a broad refactor into one session per module so each session stays below XL.
Before merge: review
Who checks the pull request, and against what?
Look forRequire checks through branch protection, name a human reviewer for every pull request, and use Devin Review for a first pass. Cognition's own security guidance says Devin can introduce bugs or insecure code. Answer.AI saw 3 of 20 tasks succeed in its January 2025 test. Cognition's November 2025 review of its own data reports 67% of Devin's pull requests merged, up from 34% a year earlier. Treat the first weeks as a measure of which tasks Devin finishes reliably on your code.
Questions to ask vendors
- Which of our repositories have a working blueprint, and who keeps each one current?
- What share of Devin pull requests merged last month without rework, by task type?
- Which session sizes produce our merged pull requests, and which sizes end abandoned?
- Which repeated corrections should become a Skill or an AGENTS.md rule?
Red flags
- Prompts that bundle several unrelated tasks into one session.
- Sessions that keep reaching usage limits, which Cognition reads as a sign the task is too large.
- Pull requests merged on a green build with no named human reviewer.
- Credentials pasted into chat and absent from Secrets.
Products in this piece
Sources
- Official documentation
- 12
- Public source
- 2
- Devin docs: When to use Devin
- Devin docs: Instructing Devin effectively
- Devin docs: Environment setup
- Devin docs: Environment blueprints
- Devin docs: AGENTS.md files
- Devin docs: Skills
- Devin docs: Knowledge (deprecated, migrating to Skills)
- Devin docs: Session Insights
- Devin docs: Devin Review
- Devin docs: GitHub integration
- Devin docs: Security at Cognition
- Vikram Sreekanti and Joseph E. Gonzalez (RunLLM), Frontier AI: One month of using Devin (Jan 23, 2025)
- Answer.AI: Thoughts On A Month With Devin (Jan 8, 2025)
- Cognition blog: Devin's 2025 Performance Review: Learnings From 18 Months of Agents At Work (Nov 14, 2025)