Integrating the Real and Virtual Worlds: The Unlimited Potential of the Imaging World MR System

Canon's MR (Mixed Reality) System opens up a new realm in imaging possibilities. This technology displays computer-generated (CG) images within real-world environments, creating the sense that virtual objects actually exist. This all-new imaging technology developed by Canon is expected to be applied in a wide range of fields, from design and manufacturing to medical care, education, exhibition and entertainment.


Integrating the Real and Virtual Worlds Cutting-Edge Imaging Technologies

Virtual images emerge in a real-world environment before a user's eyes, creating the sense that the CG images actually exist. Mixed reality (MR) is an imaging technology that seamlessly merges the real and virtual worlds in real time. Canon's MR System allows users to interact with full-scale, realistic CG images from any point of view, an effect difficult to achieve using existing video systems. The MR System creates a realistic imaging world that allows users to enjoy an entirely new visual experience.
How does the MR System create this imaging experience?

illust: MR System basics

  1. (1) Two video cameras inside the head-mounted display (HMD), one each located in front of the left and right eyes, capture video from the real world, which is then sent to a computer.
  2. (2) After gauging the user's positioning and orientation based on registration markers captured by the video cameras and data from the built-in sensors, the system accurately aligns the real-world and virtual images and displays them on the small monitors located inside of the HMD.
  3. (3) The system's free-form prism, comprising cutting-edge optical technologies, enlarges the video displayed on the small monitors to enable users to experience high-impact three-dimensional images with low distortion.

The MR System is configured using Canon proprietary imaging technologies.


The Capabilities of the Free-Form Prism Integration of Canon Proprietary Optical Technologies

illust: Free-form prism basics

One of the challenges faced during the development of the MR System was minimizing the size of the head-mounted display (HMD). This was overcome through the development of the free-form prism.
The free-form prism, which is designed using a free-form surface shaped neither like a flat plane nor a sphere, enlarges the merged images displayed inside the HMD. The images shown on the display are refracted and reflected within the prism, helping to not only enlarge the images, but also deliver a clear image across the entire field of view without distortion or blur caused by optical aberrations. The free-form prism is also compact enough to substantially reduce both the size and weight of the HMD. Canon's proprietary optical technologies have made the development of the free-form prism possible.


photo: Marker

Registration Process Technology Creating a Realistic Mixed-Reality Environment

Another challenge faced during the development of the MR System was the realization of accurate alignment technology that enables the depiction of CG images in real-time at a specific location in the real-world environment based on the user's position and orientation. For example, when a CG object is rendered on top of a table, it may sink into the table or float above it if the real-world and virtual images are not aligned properly. The mixed reality world will appear unnatural if the MR System does not correct for changes in alignment.
Drawing from the company's image-processing technologies cultivated over many years, Canon overcame this issue by developing a high-precision, high-speed marker detection method. The video cameras employed in the HMD capture and read registration markers to calculate the exact position and orientation of the HMD. In addition, internal gyro sensors and optical sensors deliver further alignment accuracy. When creating a mixed reality environment, the MR System's dedicated calibration tool realizes greatly enhanced setting and calibration operational efficiency compared with previous methods, which were cumbersome and complicated.
This technological breakthrough has enabled substantial progress to be made in MR technology.


Reduced Product Development Times Using the MR System for Design and Prototyping

Canon began research and development for the MR System in 1997. The company has since been accumulating hardware and software engineering capabilities to achieve the practical application of mixed reality in society.
As product lifecycles grow progressively shorter, it has become necessary for the manufacturing industry to introduce products to the market in a timely manner. When used during the product design phase, the MR System allows users to view full-scale CG images that are instantly responsive to their position and orientation, enabling the evaluation of product design and usability. The system can also help to reduce the number of prototypes required and, in addition to shortening the amount of time spent on development, can contribute to minimizing costs and environmental impact.
The MR System can be used not only in the design and manufacturing sectors, but also in a wide range of industries and fields. Satisfied with the functionality and quality offered by the MR System, Canon launched the system in July 2012.

photo: MR System usage example

Moving toward a More Advanced World of Imaging Creating New Possibilities and Values

Canon keeps trying to further improve the MR System to make it more convenient for users and to expand the ways it enriches people's lives. The company is working on reducing the size and weight of the HMD, widening its angle of view and enhancing the image quality of merged visuals. Regarding alignment, technological advancements have enabled a reduction in the number of registration markers required, and progress is being made in creating a marker-less system.
As the MR System realizes further advancements, its scope of application will expand into a wide range of fields. For example, the system can support the medical industry by visualizing the state of diseases and treatments for patients, and can act as a presentation tool for new home and car showrooms. In the future, there will be more opportunities for the general public to experience the MR System.
With the potential to open up a new world of imaging possibilities, Canon's MR System takes advantage of the rich visual information of the real world and the flexibility offered by CG images, creating new possibilities and value across every industry and sector.
Canon is working toward the day when the MR System, as one of the company's core technologies, promotes innovation in society.

illust: The potential of MR System

Canon Technology

Top of Page