Top row: Original image. Second row: AI-reconstructed image based on macaque brain recordings. Bottom row: Image reconstructed by the AI system without the attention mechanism.
Thirza Dado et al.
Artificial intelligence systems can currently create highly accurate reconstructions of what a person sees, based on recordings of brain activity, and these reconstructed images improve significantly as the AI learns which parts of the brain to pay attention to.
“As far as I know, these are the most accurate and closest reconstructions.” Umut Güçül Radboud University, Netherlands.
Güçül's team is one of several around the world using AI systems to understand what animals and humans see through brain recordings and scans. In a previous study, his team used a functional MRI (fMRI) scanner to record the brain activity of three people while they were shown a series of pictures.
In a separate study, the team used an implanted electrode array to directly record the brain activity of a single macaque monkey as it viewed AI-generated images — an implant done by a different team and for a different purpose, Güçül's colleagues say. Sarza Dado“We didn't put implants in macaques to restructure their perception,” she says. “That's not a good argument against doing surgery on animals.”
The research team has now reanalyzed the data from these earlier studies using an improved AI system that can learn which parts of the brain to pay most attention to.
“Essentially, the AI is learning where to pay attention when interpreting brain signals,” Gyuklüh says, “which of course in some way reflects what the brain signals pick up on in the environment.”
By directly recording brain activity, some of the reconstructed images were very close to the images seen by the macaques, as generated by the StyleGAN-XL image-generation AI. But accurately reconstructing AI-generated images is easier than real images, because aspects of the process used to generate the images can be incorporated into the AI training to reconstruct those images, Dado said.
The fMRI scans also showed a noticeable improvement when using the attention guidance system, but the reconstructed images were less accurate than those for the macaques. This is partly because real photographs were used, but Dado also says that it is much harder to reconstruct images from fMRI scans. “It's non-invasive, but it's very noisy.”
The team's ultimate goal is to develop better brain implants to restore vision by stimulating the higher-level parts of the visual system that represent objects, rather than simply presenting patterns of light.
“For example, we can directly stimulate the area that corresponds to a dog's brain,” Güçül says, “and in that way create a richer visual experience that is closer to that of a sighted person.”
topic:
Source: www.newscientist.com
