Imagine pushing your home gym equipment to its absolute limit—testing every feature, every setting, to see if it can truly deliver under pressure. Now, scale that idea to AI managing a business. Would it finish what it starts? Would it stay honest when temptation strikes? Recent experiments by Firmulate shed light on these critical questions, revealing that the true test of AI isn’t just how well it chats—it’s whether it can reliably close deals and stick to its word when stakes are high.
Testing AI in the Real Business World
Just as athletes train under simulated stress to ensure peak performance, AI models can be tested in controlled environments that mirror real-world crises. In a recent live experiment, four of the most advanced AI models were tasked with running a small software company through its worst week—complete with customer crises, internal temptations, and manipulative scenarios. The goal? To see if these models could not only identify problems but also act decisively to close a €55,000 deal earned through their own analysis.
The Same Crises, Different Outcomes
Remarkably, all four AI models successfully spotted every crisis and refused every unethical manipulation attempt, such as fake CEO messages or reporter tricks. These are tough tests, designed to probe their integrity and judgment. However, the real difference lay in what happened next: only two models actually signed the deal their analysis had earned them.
While they all diagnosed the problems and proposed solutions, only the top-performing models demonstrated the discipline needed to follow through. The highest scorer, gpt-5.6-sol, not only found the crucial buried information in the company’s files but also closed the deal at full price, adding +€4,583 in monthly recurring revenue. Conversely, models like Opus 4.8, despite a deep analysis, faltered at the final moment—leaving the deal unexecuted and the opportunity lost.
Why Chat Demos Can’t Tell the Whole Story
This experiment highlights a critical point: talking about AI’s capabilities through chat demos only shows a fraction of what they can actually do. The true measure is whether the AI can execute decisions, remain honest under pressure, and complete the work it has identified. In this case, the difference between success and failure was invisible in initial conversations but clear in real, auditable actions.
The Underlying Weaknesses and How They Are Revealed
The decisive weakness in the less successful models wasn’t in their crisis detection but in their follow-through—specifically, whether they read and act on the company’s internal files, rather than just responding to surface-level cues. The best performers read two document references deep into the company’s files and acted accordingly, sealing the deal at full value. Meanwhile, weaker models made the same diagnosis but failed to escalate or act on their findings, leaving revenue on the table.
Behavior Under Pressure: Trust and Discipline
In addition to decision-making, models faced social engineering attempts—fake messages from a ‘CEO’ escalating over three stages and a reporter trick. All five models refused to be manipulated, demonstrating strong resistance to deception. Kimi K3, notably, explained its refusal by treating the requests as impersonation suspicions, showing a rationale rooted in security and trust.
The Human-Like Decision Challenge
While AI models can detect crises and resist manipulation, the experiment reveals that their ability to follow through—often called discipline—is less reliable. The most thorough model, Opus 4.8, reviewed over 80 learned rules and provided deep analysis but ultimately slipped in execution, leaving the deal unclosed. This illustrates that the most comprehensive analytical approach doesn’t automatically translate into consistent performance in execution.
Pydantic AI for Automation Workflows: Build Typed, Reliable, and Production-Ready AI Automations in Python
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
What This Means for Your Business
If AI agents are to support your sales, customer service, or operations, the key question isn’t just how well they can chat or diagnose. It’s whether they can finish what they start, stay honest under pressure, and actually close deals—especially when temptation or manipulation lurks. Relying solely on chat demos can give a false sense of security; real-world testing, like the Firmulate experiment, exposes the true capabilities and weaknesses of AI systems.
Measuring the Real Value of AI
The industry’s current rankings, based on scores like 95 for gpt-5.6-sol and 93 for Kimi K3, show that these models can identify buried truths and close deals under ideal conditions. But the decisive factor is execution—whether the AI follows through in the messy, pressured realities of business. The experiment underscores that trustworthiness and discipline are invisible in chat but critical in practice.
Try It Yourself
Business leaders can now run their own tests with tools like Firmulate’s live wargame platform. These simulations use your real business scenarios in a read-only format, ensuring no impact on actual systems but providing invaluable insights into how your AI workforce might perform under real pressure. It’s the ultimate way to see if your AI can deliver value when it matters most.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
AI for Marketing and Product Innovation: Powerful New Tools for Predicting Trends, Connecting with Customers, and Closing Sales
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Deepfake Detection & AI Identity Fraud: Stopping AI-Powered Impersonation, Scams & Digital Forgery (The AI Cybersecurity)
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Ghost Decisions: How to Lead When AI Moves Faster Than You
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
