In-camera, photorealistic style transfer for on-set automatic grading

Zabaleta, Itziar; Bertalmío, Marcelo

doi:http://dx.doi.org/10.5594/M001835

In-camera, photorealistic style transfer for on-set automatic grading

Citation

Zabaleta I, Bertalmío M. In-camera, photorealistic style transfer for on-set automatic grading. In: Annual Technical Conference of the Society of Motion Picture and Television Engineers (SMPTE); 2018 Oct 22-25; Los Angeles, USA. New York: Society of Motion Picture & Television Engineers; 2018. DOI: 10.5594/M001835

Abstract

In professional cinema, the intended artistic look of the movie informs the creation of a static 3D LUT that is applied on set, where further manual modifications to the image appearance are registered as 10-parameter transforms in a color decision list (CDL). The original RAW footage and its corresponding LUT and CDL are passedon to the post-production stage where the fine-tuning of the final look is performed during color grading. — In many cases, the director wants to emulate the style and look present in a reference image, e.g. a still from an existing movie, or a photograph, or a painting, or even a frame from a previously shot sequence in the current movie. The manual creation of a LUT and CDL for this purpose may require a significant amount of work from very skilled artists and technicians, while the state of the art in the academic literature offers promising but partial solutions to the photorealistic style transfer problem, with limitations regarding artifacts, speed and manual interaction. — In this paper, we propose a method that automatically transfers the style, in terms of luminance, color palette and contrast, from a reference image to the source raw footage. It consists of three separable operations: global luminance matching, global color transfer and local contrast matching. As it just takes into account the statistics of source and reference images, no training is required. The total transform is not static but adapts to the changes in the source footage. The computational complexity of the procedure is extremely low and allows for real-time implementation in-camera, for on-set monitoring. While the method is proposed as a substitute for the need to specify a LUT and a CDL, it's compatible with further refinements performed via LUTs, CDLs and grading, both on-set and in post-production. The results are free from artifacts and provide an excellent approximation to the intended look, bringing savings in pre-production, shooting and post-production time.

Description

Comunicació presentada a la SMPTE 2018 Annual Technical Conference of the Society of Motion Picture and Television Engineers, celebrada a Los Angeles (Estats Units d'Amèrica) del 22 a 25 de octubre de 2018.