Agentic AI

What is an agent? It is a computer program that uses an LLM as the decision-making engine, whereas a traditional program essentially uses a “decision-tree” that the programmer sets up. For example, a program may be determining adulthood or childhood. If the input is 18, then the output would be “adult,” but if the input was 17, then the output would be “child.”

Now, an agent is usually much more complex in that it has functions/methods/tools that the agent can use (based on the LLM decision). Before the agent invokes the method, it validates the decision output against programmed rules to make sure that the LLM output didn’t go off the rails. This is done to make sure it is consistent with the goal of the agent told to it by the user.

That is the central issue regarding how agents can be dangerous. If there isn’t good validation, then the agent can do something destructive accidentally. For example, if an agent uses an LLM to create a database query and it doesn’t have good validation after it receives the query but before it calls the method (to run this against a database, for example), the query from the LLM could have commands to delete data or drop a table or other problems. The bottom line is that an agent is unpredictable and can have unintended consequences.

Software is good when it is predictable, but when it isn’t we call it “buggy.” However, an agent is inherently unpredictable and as such there are going to be unintended consequences where the agent can do things that the user never wanted to begin with. This unpredictability is dangerous.