Transforming "seeing" into "experiencing," Mixed Reality (MR) technology merges the real world with virtual, computer-generated (CG) images in real time to create a new "mixed" reality. Canon's MREAL (pronounced em ree-AL) System is already supporting design and production processes in the manufacturing and construction industries.
Merging the Real World With Realistic CG Images
to Transform "Seeing" into "Experiencing"
Virtual reality (VR) and augmented reality (AR) are used in the devices we use every day, like computers and smartphones. Hopes are high for mixed reality (MR), a new technology with the potential to revolutionize manufacturing processes.
MR seamlessly merges the real world with CG images in real time. The computer-generated images are highly realistic, creating the illusion that they actually exist in front of the viewer. When the viewer approaches such images, they grow larger. When the viewer backs away, he is able to take in the entire scene. And since objects can be viewed from any angle, MR creates a sense of complete virtual reality-like immersion.
The Introduction of the MR System
Launching the Compact, Lightweight,
High-Image-Quality MREAL System
Canon's MR technology was established through years of research and development. The commercialization of the technology as a business began with the launch of MREAL in 2012.
Canon's MREAL System consists of a head-mounted display (HMD), underlying middleware, application software and various sensors.
When users put on the HMD and view the built-in displays, video cameras positioned at the left and right eyes capture video of the user's surroundings, which is sent to a computer via a controller and combined with CG images. The result is a mixed-reality world that users can experience in full scale.
Users can walk beside and behind the virtual objects that appear in their surroundings, and the system even allows multiple users to experience the same space at the same time.
The Free-Form Prism That is the Key to the HMD's Compact Size
The Canon HMD's most impressive feature, its compact size and light weight, was achieved through a free-form prism. The use of a compact free-form prism creates a space within the HMD that enables the optimal placement of the video cameras, producing a natural viewing experience that is no different from seeing with the naked eye.
Furthermore, positioning is another important factor in achieving the seamless merging of the real-world and CG images. Canon's proprietary markers, which are attached in advance to real-world objects, enable the MREAL System to perform three-dimensional positioning measurements. The company developed technologies that combine various sensors to accurately measure the positioning and orientation of the HMD while it is worn by a user, using this information to ensure images are displayed properly.
Canon's Solutions Business Employs MREAL at Manufacturing Sites
MREAL is not just changing the way we communicate with each other. It is also beginning to contribute to manufacturing processes.
In the automobile industry, for example, mixed-reality imaging is being used to confirm vehicle dimensions before prototypes are even created. This technology not only assists with the design of steering wheels, gauges, gearshifts, accelerators and other parts, but also makes it possible to visualize the results of air-resistance simulations and other tests. MREAL can also help to lower costs by such means as reducing rework during the design process and the number of manhours required for manufacturing preparation.
MREAL is also being used in the architectural design field, allowing users to employ realistic, full-scale simulations that create the feeling of actually being inside of a building. This enables the confirmation of such aspects as ceiling height and the amount of light a room receives. In this way, the technology is increasingly being used to share design concepts at the design stage.
Of particular note is the Canon MREAL's proposal-based business model, which enables the tailoring of not only the system's hardware and software, but also its applications and content based on user objectives and needs. Canon offers solutions that can be used across a wide range of fields, such as the construction, architecture and automotive industries.
Developing the MREAL System: Turning a Dream into Reality
Transforming Mixed Reality Technology into the MREAL System
MR Technology Began as a Collaboration Between
Industry and Academia
Canon Pursues Video Technology Innovations
Canon began development of MR technology in January 1997 with the establishment of the Mixed Reality Systems Laboratory Inc., working together with the former Japanese Ministry of International Trade and Industry.
Canon continuously pursues innovations in video technology. Based on the strong potential offered by MR technology, which integrates the real and virtual worlds, the company created a prototype system for industrial use. Canon's development division further refined the system based on feedback from companies that borrowed prototypes. Although the decision was made to commercialize this technology in 2007, which is when the overall product configuration took shape, Canon faced many hurdles that had to be overcome prior to the product's launch in 2012.
Achieving High Resolution in a Compact, Lightweight Design
Breakthroughs Lead to Commercialization of the MR System
One challenge faced by the company was making the system's headmounted display (HMD) compact and lightweight. The initial model, the MREAL HM-A1, weighed just 640 grams and featured optimal weight distribution, enabling users to wear it for 30 minutes without tiring.
The HMD's compact size is made possible by a free-form prism developed by Canon. Because typical optical systems consist of spherical lenses and other optical components aligned in a straight line, they jut outward, which would sacrifice balance in an HMD. Canon's uniquely shaped free-form prism, however, folds the optical path, reducing the degree to which the device extends in front. By employing the prism, the MREAL System's HMD realizes a sleek body design that is both compact and lightweight.
When developing the HMD, there was another major challenge that had to be overcome: gathering enough information to design a HMD that is comfortable to wear. During the mechanical design process, Canon collected a large data sample of head shapes of people of various races and ages. The product development team also visited a helmet manufacturer to gather such information as positive and negative comfort factors, and gravitational center positioning. As no two people have heads that are the same shape and size, the development team continued searching for the ideal HMD form that would offer a comfortable fit for all users.
High Positioning Accuracy
Producing Images Comparable to Those Seen With the Naked Eye
A computer-generated flower vase won't look realistic if it floats in the air or sinks into a table. Accurate positioning is essential to seamlessly merge real-world and computer-generated images, which is a longstanding challenge faced when developing MR technology.
Canon's software design team solved this problem by developing proprietary markers. The HMD's cameras capture these markers, which can be attached to walls, floors and elsewhere, enabling the system to accurately determine the user's location and orientation. Furthermore, the markers are used together with gyro sensors and other technologies to achieve a high level of positioning accuracy.
The HMD employs two cameras and two display panels, each located in front of the wearer's eyes, and performs high-speed data processing, assisted by a computer. Image data captured by the cameras is sent to the computer, which then transmits video merging the real-world images and computer graphics back to the HMD. A major challenge, however, was ensuring that video lag would not occur when transmitting the electrical signals. The development team was especially concerned about transmission loss caused by using the long 10-meter cable that connects the HMD to the computer. As a solution, the electrical design team devised a high-efficiency electrical control circuit that made possible lag-free video transmission.
Using MR in the Installation of Train Safety Doors
Expanding the Potential of MREAL
Before launching the MREAL System, Canon performed comprehensive testing based on applications in the B-to-B field. The system was tested under actual usage conditions to confirm its ability to withstand various applications. For example, a rail company used MREAL to investigate the placement of safety doors on a train station platform. Safety doors are used to prevent passengers from falling off the platform, collisions between trains and waiting passengers, and other accidents. Wearing HMDs, station personnel were able to confirm whether the safety doors would create any blind spots, checking safety and usability before actual construction even began.
Canon commercialized the MREAL brand in 2012. The engineering team worked to create a high-quality, high-performance product that could endure rough handling during the design of automobiles and manufacturing equipment, as well as when used on-location at construction and architectural sites. A wide range of industries have expressed interest in using the MREAL System.
Although the members of the MREAL engineering team have differing opinions about the future outlook of the system, all agree that it is essential that they pursue a creative vision that never falls behind that of users. MREAL has just taken the first step toward its great future potential.
Why are MREAL's markers hexagonal? It is because hexagons are a more easily recognizable shape when viewed at an angle than circles or squares. Hexagons can also be displayed in closer proximity than other shapes without wasted space. MREAL determines the HMD's positioning in a very simple way. Before using MR technology, the video cameras mounted inside the HMD capture the markers distributed in the real-world environment. This marker data then undergoes image processing to gauge the positioning and orientation of the user. The system uses this information to create composite images based on the movements of the user, displaying the mixed-reality video to the user in real-time.