Solving AI’s black box

When we ask ChatGPT a question, we don’t really know how it comes to its answer. Sure, we can infer that a mix of data science, algorithms, and training went into its response. But there’s a divide between the user and the AI, a kind of a black box between our inputs and its output. Many artificial intelligence and machine learning experts admit that we simply don’t know what many models are doing in that black box.

This issue is broadly referred to as “AI interpretability,” the idea that AI models should be legible and transparent at every step. It may not be the highest priority when you’re asking an AI for a recipe based on what’s left in your fridge. But in science, where transparency, precision, and understanding are mandatory, it’s extremely important.

That’s where Perimeter Institute researcher Anindita Maiti comes in. She builds AI models that put interpretability at the forefront, and then uses them to study fundamental aspects of particle physics and quantum mechanics. It’s a challenge that’s interested her since she studied theoretical high energy physics at Northeastern University in Boston.

“I was working on a project where we were using reinforcement learning to analyze some aspects of the physical model,” she recalls. “I was plagued by these questions of interpretability. I wrote the code so there was no way it would give me something spurious. But I still wondered what was happening at each of the levels within the code.”

Maiti wanted to know why their program gave the answers it did. Not simply to see if it could pump out the right answers, but to use AI in specific situations that require a lot of pattern, symmetry, and other structural recognitions. This included data from particle colliders that held clues to subatomic constituents beyond the Standard Model. “When you’re looking at the signatures in the colliders, you are picking up a lot of signals from different fundamental particles. Many of the signals may be close to each other, so much so that you may not even be able to distinguish between two different fundamental particles,” she says. “AI and machine learning have a big advantage because they are extremely good with pattern recognition and picking up subtle differences between different kinds of signals.”

When more data means more problems

So how does one reliably separate signals from noise in an AI? How do you harness that power of pattern recognition machine to help with today’s biggest questions?

The prevailing approach is to simply feed an AI model more training data. Once it has enough data, the theory goes, it can make better guesses and provide more accurate answers. For Maiti, this is a problem.

“The thing that happens in AI and machine learning is that you get some data and the neural network learns the patterns and attributes associated with that data. But it’s more than that. The AI model learns what's hidden inside the data,” she explains. “When it comes to fundamental physics problems, there are so many hidden attributes and variables that the state-of-the-art AI models cannot learn all these hidden attributes intelligently.”

Essentially, there’s so much data, and so many similar things happening, that an AI cannot reliably uncover what’s hidden inside by increasing the volume of training data. It’s like putting out a fire with gasoline: the more you add, the wilder it gets.

It’s why Maiti proposes a different approach: instead of adding more data, why not constrain the AI to a theoretical framework that it operates within? That way, you know its parameters completely and it can be used to do advanced pattern recognition inside tightly controlled situations. This means more interpretability and reliability.

“If you take any AI model, you don't necessarily have to focus on how you are going to optimize it,” she says. “Just take the neural network – the black box with many knobs and switches – and take the theory that you want it to learn, and the theory will show you how you should constrain these knobs and switches within the neural network.”

The statistics of AI modelling

So what is the mathematical principle that makes this all work? The secret is in statistics. Let’s take, for example, quantum field theory, which is used in particle physics to construct physical models of subatomic particles. By using an AI constrained to a quantum field theory as its “knobs and switches,” you can use statistics to check the distribution of outputs, which should align with the original theoretical framework.

“Quantum field theory is statistics over a particular kind of quantum field. A particular kind of quantum field essentially represents a fundamental particle. Each fundamental particle has their own kinds of quantum fields, and quantum field theory is a statistical description of the fundamental particles through this paradigm,” Maiti explains.

“What you did was constrain the statistics of this neural network to be the statistics of the quantum field theory. All these different sets of numbers in the outputs will correspond to the one statistical distribution, and that statistical distribution would have same point of view as that theory.”

In other words, if you create a very specific theoretical framework within which the AI operates, then you can take its outputs and work backward to the theory and see how it came to its conclusions. You can figure out its logic.

A black box made more transparent.