Zapier built a benchmark that tests whether AI agents can actually run business workflows โ CRM updates, support tickets, invoices โ and Artificial Analysis just published independent results. The top score: 48.6%. Here's what that number means for a small team deciding how much to trust AI agents.
On July 6, benchmarking firm Artificial Analysis published AutomationBench-AA, an independent leaderboard built on a benchmark Zapier developed from real workflow patterns on its platform. It's one of the first serious, public attempts to answer a question small businesses have been asking for two years: can AI agents actually be trusted to run business workflows on their own?
The setup is closer to real life than most AI benchmarks. Models get 657 tasks spanning finance, HR, marketing, operations, sales, and support, and have to complete them inside 40 simulated app environments โ the kinds of tools a small business actually runs on. Tasks are graded programmatically on one thing: did the right data end up in the right systems. Nearly 12,000 individual checks cover both objectives (what the agent must accomplish) and guardrails (business rules it must not break, like touching records it shouldn't).
The strictest score โ completing a task fully without violating any guardrail โ tops out at 48.6% for Anthropic's Claude Fable 5, with Claude Opus 4.8 effectively tied at 48.5%. Google's Gemini 3.5 Flash scores 42.6% and OpenAI's GPT-5.5 hits 42.1%. Looking at raw objectives rather than full tasks, the leading model completes about 73% of what it's asked to do.
Two findings stand out beyond the leaderboard order. First, every single model breaks business rules โ guardrail violations range from roughly 0.5 to 1.3 per task depending on the model. Second, cost varies enormously for similar output: Gemini 3.5 Flash effectively matches GPT-5.5's score at around a third of the cost per task ($0.49 vs $1.32, per Artificial Analysis's figures).
Zapier, Make, HubSpot, and nearly every tool StackPilot covers now sells some flavor of AI agent โ software that plans and executes multi-step work instead of following a fixed recipe. The marketing tends to describe them as autonomous teammates. This benchmark, built by one of those vendors on its own workflow data, says something more sober: on realistic tasks, the best available agent gets the whole job done cleanly about half the time.
That's not a reason to ignore agents. Completing 73% of objectives is real work, and these scores will climb โ benchmarks like this exist precisely to drive improvement. But it is a reason to be deliberate about where you deploy them:
The emerging pattern โ and the one this data supports โ is a split: use fixed workflows (Zapier's classic Zaps, Make scenarios) for anything where a mistake is expensive, and use agents for judgment-heavy, lower-stakes work like drafting, summarizing, categorizing, and research, with a human checkpoint before anything irreversible happens.
It's also worth noting what Zapier's participation signals. Publishing a benchmark where every model โ including the ones powering your own agent products โ visibly breaks business rules is an unusual move for a vendor. It suggests the industry conversation is shifting from "look what agents can do" to "here's exactly how reliable they are," which is a healthier place for buyers to be.
AI agents in mid-2026 are useful assistants and unreliable employees. If you're evaluating agent features in Zapier, Make, HubSpot, or elsewhere, pilot them on support and ops tasks where errors are cheap and reversible, keep deterministic automation on your critical paths, and re-check the numbers in six months โ this is the kind of benchmark where the story can change fast.
See how the two automation platforms compare on pricing, complexity, and AI features before you commit.
Read Make vs Zapier โ