Referential communication in heterogeneous communities of pre-trained visual deep networks

Mahaut, Matéo; Franzon, Francesca; Dessì, Roberto; Baroni, Marco

Referential communication in heterogeneous communities of pre-trained visual deep networks

Citation

Mahaut M, Franzon F, Dessi R, Baroni M. Referential communication in heterogeneous communities of pre-trained visual deep networks. In: AAMAS 2023. 22nd International Conference on Autonomous Agents and Multiagent Systems; 2023 May 29-Jun 2; London, United Kingdom. [Richland]: IFAAMAS; 2023. p. 2619-21.

Abstract

As large pre-trained image-processing neural networks are being embedded in autonomous agents such as self-driving cars or robots, the question arises of how such systems can communicate with each other about the surrounding world, despite their different architectures and training regimes. As a first step in this direction, we explore the task of referential communication in a community of state-of-the-art pre-trained visual networks, showing that they can develop a shared protocol to refer to a target image among a set of candidates. Such shared protocol, induced in a self-supervised way, can to some extent be used to communicate about previously unseen object categories. Finally, we show that a new neural network can learn the shared protocol developed in a community with remarkable ease, and the process of integrating a new agent into a community more stably succeeds when the original community includes a larger set of heterogeneous networks.

Description

Comunicació presentada a la 22nd International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2023), celebrada a Londres del 29 de maig al 2 de juny de 2023.