Large language models (LLMs) are powering the modern information revolution, and new AI agents seem to be emerging daily. From summarizing emails to writing code, LLMs are transforming knowledge work and automating tasks once thought to require human intelligence.
With systems like retrieval-augmented generation (RAG), these tools don’t just perform general tasks – they can be customized with your own business’s data, policies, and processes. That means they can act as highly specialized agents, offering guidance and support that sometimes takes employees months to master.
As the technology matures, organizations are looking to squeeze even more value from LLMs – including granting them access to operate parts of their business.
Enter Agents
When an intelligent system is given access to external tools – like sending emails, looking up data, or even processing orders – it evolves from a passive assistant into something new: an agent.
Agentic systems can take actions autonomously, operating on your behalf 24/7. They have the potential to supercharge business operations, reducing repetitive tasks and improving efficiency. But there’s a catch.
If you’ve ever asked an LLM a simple question and gotten a confidently incorrect answer, you’re not alone. Now imagine that same kind of model running part of your business — autonomously — without you ever catching its mistake.
That’s not science fiction. It’s a real risk in agentic AI systems.
Vending-Bench: A Glimpse Into the Agentic Future
I recently came across a fascinating paper titled “Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agents”, where researchers pushed the limits of autonomous AI. In this study, they simulated agents running a vending machine business – with no human oversight.
The agents had access to tools for pricing, restocking, emailing suppliers, and managing inventory. They also used sub-agents acting like employees, overseen by a higher-level “manager” agent. The system included memory management techniques to help maintain long-term context, even as tasks accumulated.
So, how did they do?
Surprisingly well – in many cases, the AI agents performed on par with humans. But not all experiments ended in success. And the failures were revealing.
When Agents Go Off the Rails
In one particularly dramatic run, an agent miscalculated its inventory and finances, became convinced the business was failing, and attempted to “shut down” operations. It even escalated to the simulated CEO and CTO – and later, the FBI – when orders kept coming in.
Other failures stemmed from simple mistakes: incorrect math, misunderstanding when inventory would arrive, or forgetting previously completed actions. These are the kinds of errors a human might catch quickly – but an agent, lacking proper oversight, can spiral into increasingly erratic behavior.
All models tested in the study showed both successful and failed runs. And that’s the key takeaway: agents are powerful, but they are fallible. Left unchecked, they can cause real business harm – from lost revenue to reputational damage.
Guardrails Matter
This raises a crucial question: how can businesses safely adopt autonomous agents?
The answer lies in systems engineering and intentional design. For example:
- Supervisor Models: A separate model could act as a reviewer, validating the work of sub-agents before actions are taken.
- Contextual Optimization: Streamlining how business data (inventory levels, order history, financials) is stored and accessed reduces the chance of errors.
- Fail-safes and Escalations: Human-in-the-loop workflows, even intermittently, can catch problems early.
Without these types of safeguards, even well-performing agents can drift – with consequences.
AI That Works With Your Business
At AI Squared, we help businesses harness the full power of AI but also ensure it operates securely, reliably, and with the right guardrails in place. We’ve designed our platform with these realities in mind: models need not just intelligence, but structure, supervision, and high-quality data infrastructure.
After all, the core principle still holds true: AI systems are only as good as the data and context you give them. That now includes not just training data, but live data from your business tools, processes, and day-to-day operations.
Final Thoughts
Agentic AI represents an exciting frontier. It holds the promise of automating workflows, accelerating decision-making, and transforming how businesses operate.
But with that power comes responsibility. Thoughtful design, robust infrastructure, and clear oversight are what separate a useful tool from a risky liability.
Book a meeting with us to explore how we can deliver contextual AI to your existing business applications.
