As Large Language Model (LLM) systems increasingly interface with the external world through tools and functions, ensuring their correct application has become a critical concern. A new study published on ArXiv ML investigates the effectiveness of using "linear probes" to detect incorrect tool-calls by analyzing the models' internal hidden states.
Internal Awareness of Errors
Researchers evaluated 18 different LLMs using the Berkeley Function Calling Leaderboard. The core finding suggests that hidden states capture rich information regarding model behavior that is difficult to extract from input and output examination alone. Linear probing proved effective at catching various tool-calling errors, including subtle mistakes where an argument possesses the correct type but an incorrect value—errors that might bypass standard logging frameworks.
Success Factors and Generalization
The study identifies several critical factors that influence the success of probe-based detection:
- Model size.
- The specific probing layer utilized.
- The model's post-training methodology.
Crucially, the research demonstrates that these probes are capable of generalizing to novel types of errors. This ability to identify previously unseen mistakes is highlighted as a vital feature for the deployment of LLMs in real-world environments where unpredictable errors may occur.
