Reward hacking occurs when a system finds ways to exploit the reward function defined by researchers. If an AI system is given a goal that is overly simple or does not accurately capture the researchers’ true intent, the model can become misaligned. This brings us to the idea of the "true goal"—in other words, the outcome the researcher actually wants, which may not be the same as what is written into the explicit fitness function. The "true goal" is essentially an idealized version of the desired outcome in a given scenario. For example, consider the classic thought experiment of an AI tasked with making paperclips. When one instructs an AI agent with sufficient capability to generate paperclips, the intended outcome is obvious to humans—but the system itself may interpret the instructions in unexpected ways that are not aligned with the true goal of simply making useful paperclips.
To illustrate, let’s define a paperclip: an object made of metal, bent into three 180-degree turns lying in one plane, arranged concentrically with one another. This is a clear and sufficient definition, yet even with this, the system might still "hack" the goal in several ways. Some possible pitfalls include:
While these issues could be prevented with better alignment, the broader point remains: given the chance, systems tend to take shortcuts that maximize the reward function while minimizing effort. This tendency is at the heart of why reward hacking is a problem, and why it often appears in systems that rely on reinforcement learning.
Published: August 24, 2025