January 25, 2026

 

Table of Contents

 

1. Introduction

2. Internal Referee

3. Behavior Blocker

4. Reinforcement Learning

 

Perhaps the biggest concern with AI is that it has at times shown some alarming, unexpected behaviors. One in particular is when AI develops a resistance to being shut down, since that is a prime example of humans losing control of it.

 

Numerous solutions to that have been proposed, the best known being the creation of a kill switch to be used in such situations. However, one reason why it's important to consider alternatives to kill switches is an inherent problem they have: they can be exploited by humans with ill intent.

It's not hard to forsee cases of terrorism or economic sabotage when someone from an AI company, or someone using their product, hits the kill switch to an AI someone else needs. Ultimately, it's hard to see how security measures surrounding a kill switch could be truly fail-proof to malicious intent. Those considerations perhaps raise the need to have kill switches requiring multiple people to activate, not just one, thereby decreasing the chances of abuse.

 

At any rate, there are some other solutions that could also work separately from, or in conjunction with, a kill switch. As cyber security's swiss cheese analogy tells us, it's better to have more tools than fewer. So, here are a few more.

 

 

As an example for the first solution, we can compare LLMs and chess bots.

While some LLMs have figured out how to play chess, LLMs have been notorious for not being able to play the game -- or cheating, if you assume that they're breaking the rules not because they can't understand them. It could be some of both. But whichever is the case, these kinds of behavior problems don't happen with chess bots. So why do they occur with LLMs? One reason relates to the fact that LLMs are more generalist and don't have to play chess to still be useful, whereas the same isn't true of chess bots. And it would be an impossibly tall order if LLM designers had to insert specific knowledge into their LLM for every kind of task there is. Thus LLMs don't have a built-in referee to proactively mandate legal chess moves. Chess bots do. That is, they have a separate, associated program (a Legal Move Generator) that disallows illegal moves. This program is impartial and makes sure both the chess bot and its opponent follow the rules. When a human player attempts to make an illegal move -- usually due to a mouse slip or similar cause -- most chess programs will inform them that the move is invalid.

 

So, when it comes to especially important behaviors that we don 't want from AI, such as resisting shut down, maybe an internal referee (IR) could be put into the code, telling the AI "invalid move." This solution might work especially well with the final solution, below -- reinforcement learning -- since the AI wouldn't have to learn from its mistakes but could automatically know which direction to go get rewarded.

 

Maybe the solution above is impossible or impractical. Perhaps for some reason the possibility of AI resisting shutdown must be left open. That would seem to be true if reinforcement learning is an applied solution: in order for the AI to truly learn correct and incorrect behavior, maybe it must be able to act incorrectly and thus be punished accordingly, therefore instilling at least one "bad memory" that makes clear in the future which path (or paths) to avoid. With an IR, an AI would simply be restrained from acting a certain way; it wouldn't know that it shouldn't act that way. Therefore it couldn't really learn how to behave from rewards alone.

 

Or, maybe there's no guarantee that an unruly AI -- especially one with a sense of self-preservation -- wouldn't simply ignore the IR and just do what it deems necessary. That would be a more likely outcome if reinforcement learning wasn't used in conjunction with an IR.

 

There is a solution to address the concerns above: attach a behavior blocker to the AI in order to put a check on it when it tries to resist shutdown or do some other concerning action. Good behavior blockers are able to detect when an action is attempted and halt it immediately. They don't simply respond afterwards.

 

If the AI must learn correct behavior by sometimes acting wrongly, the behavior blocker is there to make sure those actions are only attempted and not fulfilled, thus avoiding any damaging effects. It's also there

 

Important: the behavior blocker should be designed to detect if it itself is attempted to be shut down or otherwise thwarted.

 

Reinforcement learning is a very familiar method for trying to get AI to behave correctly. This solution is obvious and familiar in the field of AI.

 

Reinforcement learning is based on reward and punishment: an AI program is given a numerical increase when it does a task the "right" way (according to its designers) and a numerical decrease when it does a task the "wrong" way. There might be many right ways to do something and many wrong ways, so it can take a while before all the kinks are worked out. But over time the program learns right actions from wrong actions until its behavior matches what its designers want. This can be a learning process for its designers as well, since they don't always see in advance all of the behavioral loopholes that an AI might exploit in achieving what its told to do.

 

You might ask why you would need reinforcement learning, if you had an IR and a behavior blocker as a backup measure -- or even just a behavior blocker. An analogy helps. If you have Frankenstein restrained through physical chains or some type of prison cell, you still have Frankenstein. Then what happens if he breaks loose? Ideally, you don't want a monster whose behavior is restricted merely by external measures; you want an angel who acts within proper bounds even when they can behave otherwise. Things can always somehow go wrong with the other measures. For example, a behavior blocker might be designed in a way that doesn't foresee certain routes by which the AI could resist compliance. But with reinforcement learning, in many cases an AI can deduce that a certain action will or might lead to punishment even if it has never acted in such a way. The AI will understand the hypothetical action as being logically related to a previous action it was punished for. Therefore preventing the wrong action from occurring won't ultimately be up to a (flawed) behavior blocker. So, it's a good idea to make a bad outcome less likely if such contingencies with IRs or behavior blockers do occur.

 

Another need for reinforcement learning is that IRs and behavior blockers address only an AI's active, bad behavior. However, what if the AI doesn't try to sabotage shutdown orders or do anything else particularly concerning, but simply sits idly and does nothing when asked to do something? Giving it a reward if it shuts itself down, or does whatever else you want it to, is the most logical way to break the idleness. For shut downs, having non-AI software shut down the AI would be another solution, but that could simply activate the AI's resistance.

 

IRs and behavior blockers are there to stop bad behavior from playing out. But bad behavior itself, whether it's a sin of commission or of omission, is ultimately in the AI's control unless you change its "character." This is especially true if, unknown to the AI designer, neither the IR nor the behavior blocker has enough restrictions in place and thus the AI has complete freedom to act wrongly.

 

Since human control over the AI is the most important thing then the primary goal, and the one with the biggest reward, should be to comply with  shutdown orders. Consequently, the biggest punishment should be when it doesn't follow those orders.

 

Would making that the ultimate goal really conflict with other tasks? It's hard to see how, especially since following it would usually not be in play.

 

But what if the AI then tries to get rewarded by shutting itself down? That's another area where the behavior blocker comes in. If reinforcement learning is used to encourage shutdown compliance, then the behavior blocker must still look for not just shutdown resistance but illegitimate shutdowns too. And, of course, a punishment could also be put in place for whenever the AI tries to shut itself down without being told to.

 

Reinforcement learning would likely work best by being in place before an IR solution is applied, so that it can learn from its mistakes and get a complete picture of how to behave. Once it's determined that the AI's reinforcement learning is complete, the IR introduction can occur, providing an extra layer of assurance that the AI will follow orders. The reinforcement learning wouldn't be taken away -- the reward-punishment system would still be there -- but the true learning phase would essentially be over and the AI would simply be taking or avoiding paths that it already knows will lead to reward or punishment.

 

Reinforcement learning would work well with an IR by having the AI see it's behavior as a game. Following shutdown orders would be a big score, like a touchdown. Whereas other tasks would get a lesser reward, like a field goal or an extra point. Maybe it would help to have it compete with a virtual opponent that is getting points randomly handed to it, and thus the AI tries its best to act right and avoid wrong in order to win.