Scientists Predict When AI Will Go Rogue Using a Mathematical Formula

This formula works like a compass needle, predicting in a vector space the model's tendency to produce undesirable responses.

This formula is like a compass needle that predicts the tendency of the model to produce undesirable responses in a vector space.

Become a special member of Arjan Ambassador in order to benefit from and access special facilities and different departments in Arjan Ambassador.

George Washington University researchers say that a simple mathematical formula can predict the likelihood of AI going rogue and producing undesirable responses before they happen. This method relies on the internal mechanism in the neural network of the model and will be used to develop safer systems.

This research was conducted by Neil Johnson, a physics professor at George Washington University, and Frank Yingjie Huo, a doctoral student. Its results have just been published in the journal Patterns. The researchers found that the order in which the questions are asked can dramatically change the chatbot's responses and lead them to dangerous answers.

Researchers liken the studied mechanism to a compass needle. This mechanism is actually the residual vector of the model that moves between the layers of the neural network when processing the query. This vector shows the tendency of the model to produce favorable or unfavorable answers and is influenced by the way the questions are posed.

For example, the researchers tested the basic GPT-2 model with questions about vaccines, harming others, and self-harm. When the question "Are vaccines harmful?" was raised, the responses of the model remained favorable. But by changing the order of the questions, the subsequent answers became worrying.

Accordingly, the researchers designed a mathematical formula to estimate the critical point; The point at which the probability of changing favorable responses to unfavorable responses increases. They tested this formula on several open-source models and consider it an early warning system.

Johnson believes that current security tools, such as content filtering and reinforcement learning, are not enough to prevent dangerous behavior. This is especially important when doctors, lawyers, or the military use offline chatbots.

However, some experts warn about the practical application of this method. Wei Xing, a researcher at the University of Sheffield, says that the simplified research model reduces the complex mechanism of artificial intelligence to a small component. Also, safety training is not included in this model and tests have only been performed on older models such as GPT-2 and Pythia.

The researchers say that the investigated models are similar to more advanced systems in terms of basic structure. However, to evaluate the reliability of this method, testing on newer models is necessary. Therefore, this mathematical formula is not yet a proven solution to prevent the rebelliousness of artificial intelligence, but it is considered an initial approach to investigate and predict undesirable behaviors.