Pursuing the Ultimate in Imaging Reality Next-Generation Image-Processing Technology

Canon has devoted many years to the development of imaging technologies, including image-capture, object-recognition and data-processing technologies, for applying to a variety of fields. Among these, Canon's advanced image-processing technologies are recognized worldwide as among the industry's top level. Within this field, one area in which Canon is now focusing its R&D energies is on new image-processing technology enabling the reconstruction of still images and video, as well as the realistic automatic synthesis of such still images and video with computer graphics (CG) in accordance with user needs. In this way, Canon is pursuing a world of imaging reality that surpasses conventional imaging technologies.

Targeting New Frontiers for Image Processing

Making the Impossible Possible

photo: Relighting

Image-processing technologies are typically used to correct captured image data through the adjustment of such image parameters as resolution, contrast, color tone and gradation. Canon is now conducting research into new image-processing technologies that go beyond such adjustments by breaking down captured images into basic components and then reconstructing them.
Photography is made possible by capturing the light rays that travel through a camera lens. If, ideally, all of the light rays that existed within a given area (light field) could be captured, then image processing deemed unachievable using today's technologies would become possible. Changing the perspective or angle of view after an image has been captured, for example, or the location or angle of the light shining on the subject would enable the relighting of a subject, making possible changes in the distribution of light and shadow, as well as the shapes and densities of shadows along with other lighting conditions.
Such image processing requires technology that captures and recreates light rays and texture-acquisition technology that provides information about a subject's appearance, along with photorealistic CG technology capable of reconstructing highly detailed images based on texture information.
Canon is intensively researching these technologies with a view to creating a new realm of image processing.

Texture-Acquisition Technology

Acquiring Detailed Photo-Subject Information

When creating a CG image of an object, rendering is based on data for three aspects related to the object: its three-dimensional (3-D) geometry, its surface reflectance, and the light source. If these three data components can be obtained in sufficient detail, users can reconstruct or relight (e.g., move or adjust the light source) images based on principles similar to those of CG rendering.
To support research into complex textures, Canon has prepared a research environment supporting high-precision measurements to enable the direct acquisition of textural information. Texture depends on the reflectance properties of light on the surface of an object, properties that change with the incidence angle of light, the material and shape of the object, and the viewing direction. Accordingly, measurements are made by altering the angles of optical receivers and light sources to acquire data known as BRDF (Bidirectional Reflectance Distribution Function).
Looking ahead to the day when texture acquisition will be possible through image processing, Canon has begun looking into inverse-rendering technology. When realized, such technology would provide digital cameras and other consumer devices with texture-acquisition capabilities.

Measurement of Gloss
Samples with Different Levels of Glossiness

Photorealistic CG Technology

Reconstructing Realistic Images from Analytical Data

When reconstructing images, such as relighting or other user-specific adjustments based on the three extracted data components, the most important factor is image quality. Canon is now working on photorealistic CG technology with high-precision rendering capabilities to reconstruct photo images comparable in image quality to actual photographs, with none of the artificial appearances commonly associated with manipulated images.
One of the keys to developing photorealistic CG technology is the use of spectral color information. While conventional CG renders the visible light range in RGB (red, green and blue), the three primary colors of light, Canon's new technology enhances the precision of color reproduction by dispersing light rays in the visible region to render dozens of fundamental colors. This enables the faithful reproduction of the textures of real-world objects, the seamless synthesizing of actual photo images with CG, and the reproduction of an object as viewed within desired light-source settings.
Another key is the pursuit of high resolution. CG images are composed of triangular shapes called polygons. The more polygons there are in an image, the higher its resolution. Canon is developing a dispersion processing technology capable of handling one billion polygons, more than 100 times the number of polygons contained in an average motion picture.
The third key is high-speed processing. Speed is vital for processing the vast amounts of data required to achieve necessary color performance and high definition. Canon is conducting research into high-speed processing that is at least 100 times faster than conventional image-processing.
Canon's photorealistic CG technology is also effective in synthesizing photographic and CG images, enabling the smooth synthesis of photographic and CG images automatically, doing away with the troublesome process of manually manipulating each frame in order to achieve realistic results. The technology has the potential to easily create revolutionary composite images for major motion pictures, advertisements and other projects.

High-Definition Composite Image Created with Photorealistic CG Technology

Merging Real and Cyber Technologies

Combining the real (device-based) technologies in which Canon excels, such as texture-acquisition technology and photorealistic CG technology, with cyber (IT) technologies makes possible the creation of cutting-edge practical technologies, representing a new R&D concept for Canon: Advanced IRT. In the long term, Advanced IRT is expected to achieve expandability in various business domains. One application in which the technology shows particular promise is intelligent robotics employing Canon's sophisticated 3-D recognition technologies.

Launching R&D into Industrial Intelligent Robots Centered on 3-D Measurement and Recognition Technologies

In 2012, Canon introduced a new R&D concept under the banner "Advanced IRT." IRT is a hybrid term combining IT (Information Technology), from the world of cyber technology, and RT, or real-world technology. RT encompasses Canon's areas of technological expertise in interfacing with the real world, including the company's texture-acquisition and photorealistic CG technologies. IT refers to domains that are currently undergoing acceleration, such big data analysis through cloud computing and the practical application of machine intelligence techniques through networking. By merging IT and RT, Canon will develop applied technologies enabling the future expansion of the company's business domains.
The four domains expected to offer growth are: imaging and information, safety and security, health and healthcare, and industrial machinery and intelligent robots. Among these, intelligent robots for industrial use represents an area in which the company is currently focusing its energies. An intelligent robot comprises three basic elements: vision (sense of sight = real technology) to recognize its real world surroundings; intelligence (cyber technology) to think and act on its own accord; and movement (real technology) to carry out actions. Significant technological improvements must be realized in all three of these elements in order to create intelligent robots suitable for practical use.
Canon has already made use of various recognition technologies, such as character recognition, face recognition and image retrieval. Since 2005, Canon researchers have been working on the development of robotic eyes and Super Machine Vision, conducting research into machine vision technology through the application of optical, image-capture and recognition technologies. By adding the intelligent capabilities of IT to these advanced visual capabilities, Canon is moving ahead in the development of sophisticated intelligent robots. Intelligent robots capable of detecting and making decisions based on changes in their surroundings and situations, and learning and modifying their behavior based on experience, will likely play important roles in a variety of fields in the future.
As for the three growth domains aside from robotics (imaging and information, safety and security, and health and healthcare), Canon will synergistically combine the RT in which it excels with pioneering IT in pursuit of all-new applied technologies. Canon's Advanced IRT marks a new concept in R&D that will contribute widely to society in the near future.