Scaling AI Agents: What We Learned From 9 Real-World Cases - The JetBrains Blog

Our brand-new report explores what it takes for software development leaders to put agents to work.
One organization reached 95% AI adoption without seeing an improvement in developer velocity. Another tripled its pull request throughput while reducing its defect backlog by 54%.
What explains the difference?
To find out what makes AI agents effective beyond individual developers, I spoke with engineering leaders and practitioners from Fin, Semrush, Skiller Whale, and other organizations.
The findings are now available in our new report, How Top Teams Scale AI Agents, featuring nine real-world case studies and practitioner accounts, including insights from our own engineering teams at JetBrains.
The report looks beyond adoption rates and individual productivity gains to explore what actually changes when teams start relying on AI agents – how they organize work, maintain code quality, share knowledge, manage costs, and measure results.
Here are three lessons that stood out to me.
At Fin, the initial goal was to double pull request throughput without compromising code quality, security, or performance.
Rather than simply rolling out more coding agents, Fin established a dedicated team, Team 2x, to manage the infrastructure around them, including shared skills, plugins, MCP servers, and evaluations.
The team tested agent skills against real examples, backtested its code-review skill against historical PRs, and analyzed agent sessions to identify recurring mistakes and missing context.
But increasing output created new problems. CI came under pressure, and code quality became inconsistent. Fin responded by investing in linting and standardization.
Over 16 months, Fin reported:
The especially interesting part is how much work Fin put into making the rest of its engineering system keep up.
Even capable agents struggle when they can’t reliably build a project, run tests, or find the context they need.
This is particularly challenging in large, complex repositories, where developers often rely on years of accumulated knowledge about build configurations, dependencies, and project conventions.
At JetBrains, we’ve been tackling this problem in our own large monorepo. Our experience points to several practical requirements: standardized build tools, fast tests, accessible APIs, and environments that agents can set up themselves.
The lesson is straightforward: Before investing in more agents, look at what prevents the ones you already have from completing tasks reliably.
In one organization Angie Jones discussed, AI adoption reached 95%, yet developer velocity remained unchanged.
The team shifted its focus from encouraging AI use to helping developers delegate meaningful work. Champions who built richer repository-level context were eventually opening pull requests at three to four times their previous pace.
Skiller Whale encountered a different version of the same problem. In one customer engagement, a group trained to supervise AI agents roughly doubled its PR throughput, while throughput declined across the rest of the organization.
These examples highlight something easy to overlook in AI adoption initiatives: Developers need to learn how to structure tasks, provide context, evaluate results, and share effective practices with colleagues. Otherwise, organizations risk ending up with a handful of highly productive AI power users rather than more productive engineering teams.
Across these conversations, one pattern kept emerging: The organizations seeing meaningful results were consistently changing the systems and practices around them.
That meant making development environments agent-ready, establishing shared standards, improving review processes, investing in team learning, and measuring outcomes beyond adoption.
There’s no single playbook that works for every engineering organization, but there are concrete approaches worth learning from, including the ones that didn’t work the first time.
Talking to these teams gave me plenty to think about, and I’m sure there are many more stories worth sharing. I’d love to hear about your experiences with AI agents, the lessons you’ve learned, or the challenges you’re still figuring out, in the comments section.
By submitting this form, I agree that JetBrains s.r.o. ("JetBrains") may use my name, email address, and location data to send me newsletters, including commercial communications, and to process my personal data for this purpose. I agree that JetBrains may process said data using third-party services for this purpose in accordance with the JetBrains Privacy Policy. I understand that I can revoke this consent at any time in my profile. In addition, an unsubscribe link is included in each email.
Thanks, we've got you!
Air Governance is the organization layer of JetBrains Air, a system of products for agentic development. It gives you one place to manage which AI agents, models, and providers your teams use, set spending limits, and see what that usage costs and delivers.
I've been mulling this over for about a year now. The developer landscape has shifted to the point where you can't scroll past the news without seeing someone declare "I never open my IDE anymore" or "I do everything in the terminal." And those workflows genuinely work for some people. But not for e…
AI can produce code. Organizations still have to produce software. Agentic development is changing how software gets made, but it hasn’t changed what it costs to be wrong. Six months ago, we began publicly experimenting with agentic development environments. Around the same time, we introduced Je…
We’re about to start rolling out a new set of AI capabilities that provide shared context, reusable agentic workflows, organization-level governance, and cost control for software production. Developers use different AI tools depending on the task – from JetBrains IDEs to terminal-based agents su…