Google is developing the Pathways project, a new architecture for neural networks that will be able to perform many different tasks at once and master new ones.
Multitasking for Neural Networks
Google is developing an artificial intelligence-related project that will create “next-generation” neural networks capable of memorizing “thousands and millions of different tasks.” The project was named Pathways.
Today, neural networks are trained, as a rule, to perform a single task. For example, if a neural network is used to correct spelling errors, then, if it is overtrained to find grammatical errors, it will forget how to correct spelling.
“This is how most machine learning models are trained today. Instead of extending existing models to learn new tasks, we train new models from scratch to complete a single task,” Vice President Jeff Dean wrote in a blog post. “As a result, we develop thousands of models for thousands of individual tasks. As a result, learning a new task not only takes more time, but also requires more data … ”.
Google points out that the Pathways architecture is designed to change this: neural networks can be trained to perform many different tasks, according to the company.
The idea is to teach future AI systems at once a multitude of skills that it could use and combine in order to independently learn to perform new tasks.
“We would like to be able to train the model so that it can not only perform several different tasks at once, but also use and combine the skills already developed to teach new tasks – faster and more efficiently,” Dean wrote. “In particular, what the model learns from learning one task, such as how to identify elevations in the landscape from aerial photographs, can help accomplish another task — predicting how water flows through a given area. We want the model to have various capabilities that can be used as needed, and that can be combined to perform new, more complex tasks. This is a small step towards how the mammalian brain generalizes different tasks.”
Leopard Idea
Cnews points out that Pathways will enable multimodal models that can process and understand incoming visual, acoustic and language data – all at the same time. So that when the machine was dealing with the word “leopard,” or hearing someone say the word, or analyzing a video of a running leopard, the reaction would be the same in all three cases: the machine would recognize the idea of a leopard.
Another challenge that Pathway will need to help address is that today’s machine learning models involve all the neural nodes in the network, no matter how simple or complex the task is. Google believes that it is possible to achieve “sparse” activation, that is, to direct new tasks only to separate arrays of neural nodes. This approach is also much less energy intensive.
The concept of a sparse neural network has already been applied in the projects Google Switch Transformer, a model for understanding natural language, and Gshard. They consume only one tenth of the energy that would have to be spent on a more traditional neural network, where all neural nodes are activated at once, notes NIX Solutions.
In the future, Google expects Pathways to lead to neural networks capable of performing millions of different tasks, generalizing between them, combining them as needed, understanding and working with different types of data more efficiently than any network today. The goal of the project is to move “from the era of single-tasking models that simply recognize patterns, to multitasking intelligent systems that reflect a deeper understanding of our world and can adapt to new needs.”