Mixed Reality System The Story Behind the Development of MREAL

Employing Canon imaging technology to integrate the real and virtual worlds. Four engineers involved in the development of MREAL discuss the challenges they faced in this new imaging realm, Canon proprietary technologies, and their outlook for the future.

POINT 1

Introduction

Imagine looking at a brochure with pages that have neither text nor photos, only an assortment of peculiar-looking hexagonal markings. Upon putting on a head-mounted display (HMD), however, the brochure suddenly transforms into a remarkable picture book with three-dimensional photos of animals that literally pop out of the surface of each page. Or, say you'd like to tour the inside of a building that hasn't been built yet, or maybe go for a ride in a virtual car. While these may sound like scenes out of a movie, they are now very much a reality.
What makes such feats possible? The answer: Canon's MREAL System for Mixed Reality. MREAL (pronounced em ree-AL) is already supporting design and production efforts in the automobile industry as well as the building and construction industries.

Toshiyuki Yasuda

Area of technical expertise: Mechanical design

When did Canon start developing this MR System?

Toshiyuki Yasuda
Development of MR technology began in 1997 with the joint establishment of the Mixed Reality Systems Laboratory with the Japan Key Technology Center of the Ministry of International Trade and Industry (now the Ministry of Economy, Trade and Industry).

Was it your goal from the start to apply MR to manufacturing?

Ryoji Kondo
We decided to apply this technology in the industrial field, including manufacturing from the beginning. By around 2007, car manufacturers were using system prototypes and providing us with feedback.

Yasuda
It was in 2007 that, technologically speaking, we created a prototype of the current MREAL, with an HMD employing a free-form prism, alignment technology using markers, and external sensors.

Past and current head-mounted displays

POINT 2

Free-Form Prism:
Integrating Optical Technologies Cultivated over Many Years

Basic HMDs come in a variety of types. In terms of display method, they can be classified into three types: non-see-through, optical see-through and video see-through. Non-see-through HMDs don't allow wearers to see any of their surroundings while optical see-through type HMDs let users view directly through them by using half-silvered mirrors or other means. As for video see-through HMDs, although users can't directly see what's around them, built-in cameras and displays provide a view of the surroundings. Also, optical systems typically employed in video see-through HMDs may be simple in design, requiring users to peer through a lens at displays located in front of the eyes, or they may project images directly onto a user's retinas or make use of prisms. Canon, capitalizing on the optical technologies the company has accumulated over many years, developed a video see-through type HMD that employs a free-form prism. Despite its compact, lightweight body, Canon's HMD aligns the optical axes of the built-in cameras with the user's line of sight to seamlessly display both real and computer-generated images.

Ryoji Kondo

Area of technical expertise: Optical design

Why did you decide to use a video see-through type HMD employing a free-form prism for MREAL?

Kenji Hatori
With optical see-through type HMDs, the delay caused by CG rendering invariably results in a time lag between the real-world and computer-generated images. Such time lags, however, can be avoided by outputting images that have been previously synthesized by a PC, which results in the seamless merging of the real and virtual CG worlds. That's why video see-through HMDs are ideally suited for mixed reality.

Kondo
One of the reasons we used a free-form prism to project the images was to keep the HMD small. Another was to ensure a natural viewing experience. In order to achieve images that appear no different from those seen by the naked eye, MREAL captures real-world images by the HMD's built-in video cameras. If these video cameras aren't positioned correctly, the effect would be drastically different from what the user would usually see, which would feel quite strange. So, we went with a compact free-form prism, created some space, and then optimized the positioning of the video cameras to achieve a comfortable viewing experience. Although this approach isn't necessary for viewing just CG images, as is the case with VR (virtual reality), we currently believe this is the best way to provide users with a highly realistic MR experience.

Free-form prisms aren't used very much in other optical devices, are they?

Structure of the HMD's optical system

Kondo
Typical optical systems, such as those for cameras, consist of spherical lenses and other optical components aligned in a straight line. But, if you want to make effective use of limited space, you can fold the optical path by intentionally altering the axis of the optical components and compensate for the resulting drop in performance by using a free-form or other type of aspherical surface. While such an eccentric optical system may pose challenges in terms of design and manufacturing, for an HMD, it is an effective choice because it makes possible both a wide angle of view as well as a compact body design. But, using multiple optical components to fold the optical path results in a very complex structure, which is why we opted to use only a single prism to reflect and refract light.

How did you design a prism with such a complicated shape?

Kondo
In recent years, computer simulations have come to play an important role in designing optical instruments. When designing our free-form prism, we used computer simulations to calculate the optimal angles and distances between surfaces and to determine the optimal surface shape in order to achieve the necessary performance. Nevertheless, designing a special optical component like a free-form prism poses a distinct set of difficulties not usually faced when designing an ordinary lens, and the process can be tricky.

Why is that?

Kondo
Achieving the desired optical performance is naturally difficult, but even if everything goes well, the resulting shape may not be feasible, or it might be difficult to manufacture. That's why we impose design restrictions, but restrictions that are too severe could hinder the performance of the prism.

On top of that, creating an easy-to-use HMD requires more than simply pursuing high optical performance. The HMD is meant to be used by a variety of customers, and the optimal viewing position will vary from user to user.

This means that if the position at which images appear in proper focus is fixed at a pinpoint location, then attaining proper alignment when putting on the HMD will be problematic. Images must remain in focus throughout a certain range so they can promptly be seen by users as soon as they put on the HMD. It wouldn't do for the image quality to suffer every time the HMD shifted around a little. We put a lot of effort into striking a balance between ease of use and optical performance.

Our prism requires an exceptionally high level of accuracy not exceeding one hundredth of a millimeter (0.01 mm). But Canon has consistently worked with optical technology since the company's founding and has accumulated the necessary know-how. I believe this was the key to our success.

When I gave MREAL a try, I was able to see images as soon as I put on the HMD and hardly had to adjust it at all.

Yasuda
There are an infinite variety of head shapes, which makes trying to design a single product that fits all users quite difficult. During the mechanical design of the HMD, we referred to a database that contained the head shapes of people of various races, ages and sexes, but that wasn't enough. Through trial and error, we repeatedly created prototypes to get the right fit.

POINT 3

Creating Realism:
Alignment Technology

The MR System is more than just an HMD. Capturing real-world images and seamlessly merging them with CG images also requires precision alignment technology. To this end, Canon developed proprietary MREAL Markers incorporating hexagonal patterns. Furthermore, with the addition of various external sensors, such as gyro sensors and infrared sensors, the MR System can be applied to a variety of environments.

Kenji Hatori

Area of technical expertise: Software development

In order to seamlessly merge the real world with the virtual world of CG images, the MR System must accurately align both sets of images. To do this, I've heard that MREAL uses markers as well as Gyro sensors and other kinds of sensors.

Hatori
In the early stages of development, we used only sensors, but sensors are expensive. In order to provide our MR System to customers at the most reasonable price, we developed an alternative technology: the MREAL Marker. Because the built-in video cameras in the HMD capture real-world images, we came up with the idea of affixing markers on objects which, when viewed, enable the detection of each object's orientation and location.

MREAL Markers can be used not only as spatial coordinates by affixing them to walls or the floor, but also in the verification process of virtual tools by attaching them to mock-ups of the tools, enabling CG images to be overlaid on them. MREAL Markers make possible the identification of 2,048 ID patterns, enabling users to improve alignment accuracy by combining several markers.

The MREAL Markers consist of hexagonal patterns. Why did you choose hexagons?

Hatori
Although the system can recognize any shape, including circles, triangles and squares, in order to make the proportions of the markers as small as possible, we chose hexagons, which offer the greatest area efficiency. In actual use, the markers don't always face the cameras directly. Over the course of development, we found hexagons to be the most easily recognizable shape even when viewed at an angle.

When looking at the markers through the HMD, the CG images are displayed right in that location. What gave you the most trouble during development?

Hatori
I'd have to say speeding up processing. The system has to display natural images even when the user walks around wearing the HMD. It was a challenge for us to develop an algorithm (calculation procedure) to perform the high-speed processing of the entire series of necessary steps involved, namely the image-capture, recognition and position calculation of the markers, as well as the display of the CG images.
But it wasn't simply enough to speed up processing. Markers aren't always captured under ideal conditions. For example, they may be viewed at an angle, or viewed in a dark environment. As such, some kind of correction needs to be added depending on the situation, but that slows down the entire process, which made getting the proper balance quite difficult.

Yuji Takayama
I was responsible for the electrical design. With MREAL, images captured by the HMD's cameras are sent to a PC, where they are synthesized with CG images and sent back to the displays in the HMD. This means that the HMD sends and receives data over a total of four channels: one input signal per eye for the real-world images, and one output signal per eye for the displays. That's a lot of data. In addition, the HMD and the PC are connected by a cable that's about 10 meters long. With a cable that long, you're going to have signal degradation, so it was a real challenge to design an HMD that could accurately send and receive signals without any delays.

Yuji Takayama

Area of technical expertise: Electrical design

Using sensors along with markers was a unique approach.

Hatori
MREAL Markers were made to be recognizable even in less than ideal conditions but—although an extreme scenario—they cannot be seen in complete darkness. I believe one of the real advantages of MREAL is that markers can be combined with infrared sensors and gyro sensors in accordance with the environment in which the system is used.

Basic MREAL configuration

When I tried out MREAL just now, I put my hand into a computer-generated ring and it appeared just as though it was inside the ring. How did you get the system to recognize the relational positioning of objects?

Hatori
As a matter of fact, getting the system to recognize the relational positions of the real and CG images requires highly complicated processing. Since the real-world images captured by the video camera are planar, the CG image is always displayed on top of that plane. The accurate recognition of the positional relationship between real-world objects requires positional information to be obtained through such means as wearing special gloves. But this requires an extensive system and also doesn't permit the hands to be reproduced in a natural manner.

That's why we designed MREAL to recognize positional relationships. By allowing users to pre-register certain colors, the system can control whether or not CG images are displayed. In the future, we'd like to achieve the accurate recognition of positional relationships without having to register colors in advance.

A hand inserted into a CG ring
Left: Incorrect display of the positional relationship between the hand and the CG image
Right: Correct display of the positional relationship between the hand and the CG image

POINT 4

Future Prospects:
New Possibilities Brought to Life by MREAL

The automobile, building and construction industries are already making use of MREAL to support their design and production efforts. In the future, project team members may one day work together while sharing full-scale images of some enormous object.

In what fields is MREAL being used?

*Cooperation from Research & Development Center of
JR East Group Frontier Service Development Laboratory, Japan

Hatori
Primarily in the automobile, building and construction industries, MREAL is being used as a tool that contributes to reduced product development times and costs. As for other fields, for example, it has been used for exhibits in museums, to provide virtual housing tours, and to aid in the construction of safety doors on a train station platform. At the train station, we placed MREAL Markers on the platform after the last train of the day departed, which allowed us to verify the optimal placement of the doors. Even before actual construction began, we were able to confirm whether or not the safety doors would create any blind spots for the station's platform attendants.

I think one of the real advantages of MREAL is the ability to verify full-scale images of cars, buildings and other large-scale objects. Other benefits are that the system is highly portable and allows users to use it while walking around a wide area.

Because the system seamlessly merges the real and virtual computer-generated worlds, I find myself wanting to reach out and touch the CG images.

Hatori
When I'm absorbed in software development, I sometimes feel like touching the CG images myself.

Yasuda
You could call that an occupational hazard (laughs).

What kind of changes do you think the spread of MREAL and similar MR systems could bring about in the ways we work and live?

Hatori
In conventional design work, each engineer usually works alone on whatever he or she has been assigned. But, if MR systems were to come down in price and become more lightweight, we could very well see all engineers wearing HMDs, doing their work while discussing this aspect and that aspect of the project among themselves.

Takayama
That's because it makes things even easier to understand than using 3D CAD (computer-aided design) on a computer. You could say that it promotes understanding between engineers.

Hatori
For me, I view MREAL as a tool that promotes smooth communication among people who may have differing viewpoints or opinions.

Takayama
Although videophones already exist, communication can be further enhanced by seeing images that incorporate CG images. For example, you could remotely offer instruction to people in another location while taking part in a meeting.

Yasuda
If MR can make you feel as though you've taken a trip, who knows, maybe we'll see more people giving up travel (laughs).

Takayama
For people lacking freedom of movement, MR could help expand their world. I believe there are still many untapped needs for MR systems, which could change human behavior. By making the HMD easier to put on and wear, we could enhance not only business communication, but also the way we communicate with various people, which would make the world more enjoyable.

Now more compact and easier to transport, the MR System has begun to play an active role in manufacturing.
If HMDs can be made even smaller than they are now and the real and virtual worlds can be completely merged, MR systems could dramatically change not only the world of manufacturing, but also the way we live.
The enthusiasm of these Canon engineers is opening the door to the new and exciting world of the unknown.

Interview & composition
Tatsuya Yamaji
Born in 1970. After working as a magazine editor, he became a freelance writer / editor. He has been active as a researcher, interviewer, and writer in the fields of IT, science, and the environment.
His publications include The Age of the Inkjet Has Arrived (co-author), Japan's Eco-Technology is Changing the World, and Dangen (co-author).

Developers interviewed

Kenji Hatori

Area of technical expertise:
Software development

Toshiyuki Yasuda

Area of technical expertise:
Mechanical design

Ryoji Kondo

Area of technical expertise:
Optical design

Yuji Takayama

Area of technical expertise:
Electrical design

Previous Interviews

Related content