Editorial

This issue #2 of the NEXYAD FX newsletter deals with the concept of "image noise reduction" (grain for a silver film, electronic noise for a digital camera). We explain synthetically the very principles that lead to the construction of such a noise and explain what unwanted impact it may have for movies.

Even if this noise is considered as a grain effect that is supposed ti be a part of the artistic creation (as well as the movements of the camera), we explain why there may be technical and artistic interest to remove it before starting the calibration, and  then add it as a special effect in the end.

We explain the reasons why reduction of this noise is a complex task, and present the new methods main ideas.

We then focus on the application of  modern image denoising methods in the field of nighr shooting (night for night), and also on the compatibility of night for night with day for night shootings.. We show that image denoising significantly reduces production costs by allowing shooting with very little lighting (which also leads to not distort the nightlife by adding lights everywhere).





Moreover, in the "news area", you will find two tests of day for night  shootings -
  - one made in collaboration with the French cinematographer Philippe Piffeteau, (shooting under the sunlight without additional lighting)
  - The other directed by NEXYAD : a scene shot during the day on a pebbles beach, with characters around a campfire (day for night render of fire , reflections of light on faces, cast shadow problem, ...).



1 – images capture : principles

Movies uses two main principles of image capture:
- The silver film
- Digital video

It is interesting to notice that there are physical similarities in these principles that seem ti be initially very different, but which in any case involve :
- Photons: grains of light, they reflect the scene as an image capture
- Electrons: Electric Power grains used to manufacture image


Case of a silver film :


 
 A film is made of a plastic film coated with an emulsion containing silver halide crystals or silver bromide. These crystals containing silver ions and / or bromine ions. When photons from the scene arrive on the film, they are absorbed by the film and an electron is released and captured by an ion which is transformed into silver atom. The latter is transformed into a black particle in the operation of photographic development.




Case of a digital sensor CCD (Charge-Coupled Device)
 


A CCD array is composed of elements which release electrons by photoelectric effect when they are struck by photons from the scene. These electrons are stored (accumulated) on sites in numbers proportional to the illumination, then transferred to a component that turns it into an electrical voltage which is the pixel level.

NB: when a storing site is full of electrons, it can "overflow" and fill in the neighboring sites (blooming effect).



Case of  a digital sensor C-MOS


 
C-MOS sensors are composed of basic photosensitive  cells which are semiconductors (transistors) : they are insulators and do not let the electric current in the absence of light. When they  receive the photons from the scene they release electrons (and so allow electric current to get through). NB: As there is no accumulation of electric charges, C-MOS can therefore keep very high frame rates, and integration (miniaturization) can be very high (giving access to very high angular resolution). Similarly, each pixel is individually addressable.

The generated electric current is the pixel level.




2 - Concept of image noise

As explained above, the constitution of an image involves the intervention of the elementary particle of electricity: the electron.

Now it turns out that :
- electrons are "always on the move ' : they can jump from one ion / one site / one pixel to another ...
- other physical phenomena (that the light from the scene) also release electrons (eg ambient temperature). These extra electrons are superimposed on desired electrons produced by the illumination.

So we actually have two images overlap at all times:
- the desired image, constructed by electrons generated only by the light from the scene
- the noise image generated by other electrons

Dynamics of the noise image is independent of the dynamics of the desired image: that means that the noise image will be visibile  if the desired image has a  low dynamics (for instance : night shooting with poor lights) : it it called  “Signal to Noise Ratio” (SNR).

You can also visualize the noise when the camera captures an image in total darkness: the theoretical image is identically black while the captured image is made of small grains (known as dark noise level of camera or film).

We show below examples of noisy images (Image copyright ALCATRAZ Production, TV serie "Everything seriously harms you") :
 

Noisy image :


 
 
Sample of the noisy image (zoom) : white wall with noise





A calibration that would increase the contrast between highlights and middle lights would amplify the noise considerably. For this reason, a professional supervisor of calibration, wouldn’t accept to increase contrats- and he or she would be right – even if it is the aesthetic and / or artistic aspirations of the film director and of the cinematographer ... because quality of images wouldn’t lead to a good result. Then, you can see the potential of an efficient noise reduction system ...

 



 
Sample of the noisy image : hait with noise





A calibration work that would increase the luminance contrasts in this range or attempt to change some colors to bring out blonde highlights for instance, would significantly amplify the noise: so again, noise is a limitation of the image post processing.




3 – unwanted effects caused by image noise for a movie

The image noise is generally regarded as a problem to minimize. Note that it may still exist cases where this "grain" added to the image may be considered as a part of the artistic construction. Even in such a case, it is still preferable to withdraw the noise, to process and calibrate the images, and then add a chosen  grain at the end  (the original grain, or another one).

The main side effects caused by image noise are:

- Hiding movements: the grains generate random movement.
When watching the video (or film) the brain adjusts its level of vigilance and perception of movements at a level that allows to ignore as much as possible the Brownian motion of noise. This makes impossible to decode small movements in the scene.

- Hiding the details of the pictures : the noise is composed of small grains, randomly colored and lighted, which have the same size as the fine details of the image (like edges, elements of texture, ...). Again, to ignore the noise, the brain locks onto a level of vigilance that prevents it from correctly interpreting the fine details of the image.

- The limitation of low-light shooting : low-light shooting (eg night for night shooting) leads to an image whose dynamic range is rather low. To make it clearly visible, one need to amplify it : it also amplifies noise, which leads to images of very poor quality. To overcome this problem of poor signal to noise ratio, night shoots are often brightly lit, which changes their intrinsic nature.

- The limitation of depth of field effect  : filmmakers play with depth of field (blur effects) to focus our attention on certain elements of the scene. However, electronic noise is superimposed on the desiredl image, regardless of the settings of the optical : in other words, noise is "always clear”. This adds clear details (random noise) on the fuzzy areas, and it prevents to see the desired artistic effect.

- The limitation of the calibration: during an adbanced calibration (such as "The Matrix", "300", "Amelie Poulain", ...), colors and contrasts are strongly modified. This has the undesirable effect to bring out the noise (even if it was invisible on the rush). This leads calibration supervisors to stay within certain limits where the noise is not too visually disturbing (but the results in this case is necessarily more bland that if the effects of calibration could be pushed harder).

- It's not "realistic" at all: when looking ahead (with his eyes, without a camera), even when it's dark, there are no small grains in motion (or you should tell a Doctor) : the effect is therefore unrealistic, it is a product of the camera and not the scene.

All these effects of potentially unwanted image noise justifie the fact that we try as much as possible to mitigate it. And even when you want the film to have grain in the end, it is much more effective to remove at the beginning of post-production, then add it after.





4 - Noise reduction : a complex task

Let’s notice that in the case of a digital sensor, you can significantly reduce the noise by cooling the camera. Moreover, on blogs, many digital camera users often mention the need to cut the camera between shots (instead of leaving it on), or deport the display screen away from the camera. These remarks are based on empirical observations and do not usually offer no explanation. However, the explanation is very simple: when the camera if off, it cools, and the sensor has characteristics of signal to noise ratio significantly different in the first few seconds of power up and after. Similarly, the display screen heater warms the sensor, and it is therefore preferable to keep it away from the sensor.
 
We recommend the use of a cooling box.

But in any case, when the images were captured, one may wish to reduce the noise.

Noise can be characterized by two main features:

- It is composed of very thin grains:  so, reducing the thin details of the image, would reduce noise. But of course, in such a case, it also blurs the image. You lose details (contours, textures, ...).
- It is temporally random and is different for each image in a sequence: the flashing grains therefore takes place at the refresh rate of images (24 frames per second film, 25 frames per second video). By slowing the rapid changes from one image to another, we can reduce the noise. But of course it also slows down in this case all the rapid motions, those due to camera movement and proper motions of objects and actors, which creates a visual image ghost artifact (it seems that ghosts follow objects that have an apparent motion in the picture).

This explains the relative ineffectiveness of the integrated treatment of trade in software.

One could imagine using two more advanced features :
- The contours and textures are the joining of regions in the image that show a coherent structure, whereas the grains of noise does not define a coherent structure. If we can recognize the details that do not define structural elements, then we can reduce the noise without reducing the useful details of the image.
- The time trace of grain noise (of a given pixel, along with a film), generates a random signal, while the signals generated by movements in the scene or the camera movements are deterministic signals . If it possible to recognize "random process" (that is to say, if we know separate the random part of the deterministic part in a signal), then we can reduce the noise without creating ghost images.
 



5 - Implementation of advanced methods for image noise removing

NEXYAD company has developed advanced algorithms for image processing and signal processing that implement the advanced principles outlined above.

This allows to reduce image noise in a way that wasn’t imagined.

The applications are numerous, but obviously particularly impressive when the initial noise is very strong, which is the case, for example, when shooting with very little light (for night night).

We develop this application in the following paragraph.






6 - Application : night shooting (night for night)

During a night shooting (night for night), you have the choice :
 - almost no one shines to keep the night atmosphere (and then, the noise image is very stong),
 - the scene is illuminated to improve the quality of image (and then, the night atmosphere is destroyed). Note also that the strong lighting on night scenes adds a very important turning cost (cost of lighting, technical teams, and most importantly, time to implementation that significantly slows down the shooting ... see what we mean ? ...).

Similarly, day for night and night for night scenes are chained, the difference in picture quality is not acceptable.

We show three night for night sequences: 2 indoor and 1 outdoor. The camera is the Canon 5D MarkII.


A - indoor shooting under the light of 8 candles with no auxiliary lighting


(Click image to watch video)

B - shooting under the light of a lamp, without any additional lighting


(Click image to watch video)

C - shooting outside under streetlights, no additional lighting


(Click image to watch video)




7 - Conclusion

We explained how this noise reduction is essential in the particular case of night shooting, since the scenes are dark with poor signal to noise ratio. We pasted before/after videos to show the effectiveness of the denoising system.

Note that this denoising process can significantly reduce the cost of filming, because the imperfection of the camera is in part caught up, and because it becomes quite possible to shoot with very weak light sources (eg 8 candles, and nothing else !). It has also the advantage  of not distorting the night by bright lights scene.


Furthermore, we explained how this denoising process was one of the keys to the feasibility of a somewhat radical calibration (example: Amelie Poulain, 300, The Matrix, ...). Indeed, when increasing the contrast or if the color is saturated, the image noise quickly becomes apparent and very unsightly. This leads in practice to limit the effects of calibrators in order to maintain acceptable image quality. Our denoising process allows calibration supervisors to push their effects if they wish.

Finally, we explained why it is better to remove noise at the start of post production, even when it is desired that the film has a grain (for artistic reasons): the grain can be advantageously added as a last special visual effect.

We illustrate this with three examples of calibration (we talk about "special effect" side of the calibration, in practice, these sequences sould then put into the hands of the colorist to take into account "projection" aspects and to unify different sequences of the same film). These examples of calibrations would not possible without having previously denoised images.



(Click image to watch video)


(Click image to watch video)


(Click image to watch video)



Pour en savoir plus, contacter NEXYAD :  contact@nexyad.net