Boosting image captioning with an attentional mechanism = Boosting image captioning using diverse beam search

Benedicto Serrano, Javier

Boosting image captioning with an attentional mechanism = Boosting image captioning using diverse beam search

License

Citation

Benedicto Serrano, J. Boosting image captioning with an attentional mechanism = Boosting image captioning using diverse beam search. 2018. handle: http://hdl.handle.net/10230/35541

This citation was generated automatically.

Abstract

The task of automatically generating captions for arbitrary digital images involves both Computer Vision and Natural Language Processing. Popular approaches tackle the challenge by implementing neural networks based on frameworks capable of generating English captions of query images. Those architectures can be split into an image processing convolution neural network (CNN) encoder component transforming images to embedded vectors and a recurrent neural network (RNN) as a language model decoder component transforming embedded data to natural English sentences. In this undergraduate final project we implement and evaluate state-of-the-art image captioning algorithm upon reviewing how various architectures are used to generate captions for images. We propose the addition of an attentional based mechanism to the Long Short Term Memory (LSTM) network component of the captioning framework. In addition, we also propose the use of a Diverse Beam Search algorithm for the RNN inference module of the caption algorithm. Then the performance of the baseline implementation is compared with the new tunned captioning algorithm.