Artificial Intelligence has grown at a breathtaking pace in recent years, and tools like large language models (LLMs) have quickly become part of our everyday conversations. They often feel like super-smart helpers – able to write, explain, or analyze things in ways that leave us amazed. But there is also a flip side. Sometimes, the same system that gives a sharp and clever answer makes a silly mistake, like mixing up numbers or spelling a simple word wrong. These ups and downs are not just small slip-ups; they point to a deeper problem.
Andrej Karpathy, one of the co-founders of OpenAI, put it simply: we must keep AI “on the leash.” Without guidance, these systems can drift into strange errors that no one would make. His reminder is clear, if we want AI to be useful, we need not just intelligence but also discipline, structure, and control.
The Compounding Problem
The real difficulty begins when LLMs are asked to do not just one thing, but many things in sequence. Imagine you ask an AI to read an article, analyze it, extract insights, compare it with another source, and then summarize the key differences. At each step, there is a chance that the model makes a small error, perhaps misreading a sentence or misunderstanding the comparison.
One mistake on its own may not matter. But when you string several tasks together, those small slips build upon each other. By the time the final answer is presented, the original meaning may be distorted or even completely wrong. This is the compounding problem of LLMs as errors don’t stay isolated, they multiply, and the more you rely on long chains of reasoning, the bigger the gap between the truth and the output.
The Risk of Prompt and Pray
For enterprises, this becomes a real risk. Many companies today are excited about deploying AI to handle complex, multi-step processes of compliance checks, financial audits, customer reports, or policy reviews. If the AI gets even a few details wrong at the start, the result may look polished but will be misleading.
This is why the “prompt-and-pray” approach giving the AI a long instruction and hoping it produces the right answer is so dangerous. It may work when the task is simple, like writing an email or drafting a quick outline. But for serious workflows that affect money, compliance, or customer trust, the compounding of small errors can turn into costly failures. In that sense, the leash is not about limiting AI, but about protecting us from the risks of blind dependence.
AI with Guardrails
So how do we solve this? The answer lies in combining the free-flowing creativity of LLMs with the discipline of step-by-step logic. Instead of asking an AI to solve a huge problem all at once, we break it down into smaller parts. Each step is validated before moving forward.
For example, if the AI is asked to extract key figures from a financial report, those figures are first checked against the original source. Only then does the next step say, creating a comparison chart begin. By applying this kind of structure, we turn a black box into something more transparent, where errors can be caught early. Karpathy himself has argued for this kind of design, using the model where it is strong but surrounding it with guardrails that ensure reliability. This hybrid approach makes the system auditable, fixable and trustworthy in the long run.
The Case of Constrained Autonomy
The way forward is to rethink how we design AI systems. Instead of asking one big model to handle everything from start to finish, we need a mixed approach. The creative side of language models can be used for reasoning in small steps, while clear rules and checks in the background make sure the overall system stays reliable. This doesn’t limit the intelligence of AI, it simply gives it the right direction, like a guide on rope.
In practice, this means breaking big problems into smaller parts and creating a plan for how to solve them. The AI can look at the task, see what resources are available, and then set up a step-by-step workflow. At every stage, the outputs are checked against the requirements, and different options are compared before deciding which one to move forward with. This way, we reduce errors, save effort, and make sure the result is both accurate and trustworthy.
Keeping the Leash Tight but Flexible
LLMs are powerful but unpredictable. Small errors can snowball into big failures when tasks involve many steps. The answer is not blind trust or complete rejection, but careful design, breaking work into smaller parts, validating each step, and adding the right guardrails. That’s how we move from flashy demos to truly dependable AI.
At AI Squared, we focus on building guardrails that make AI both flexible and reliable so enterprises can use it with confidence and real value.
