Augmented Reality

Background Of Augmented Reality

Augmented Reality (AR) is an enhanced version of the physical world and was first coined in 1992 by Thomas Caudell and David Mizell. AR shares a history with its technological cousin, Virtual Reality, replacing real-world Reality with a virtual computer-generated environment. Both virtual Reality or VR and AR share a common ancestor, “The Sword of Damocles.”

The Sword of Damocles was created in 1968 by computer scientist Ivan Sutherland with the help of his students. The prototype was so heavy that it needed to be suspended from the ceiling by a mechanical arm.

Augmented Reality prototype from 1968 — **AR prototype from 1968**

It was one of humankind's first experiments in replacing the real world with digital Reality. Today, mechanical arms are no longer needed to suspend heavy machines from the ceiling. Headsets are more powerful and advanced than the system Sutherland used with the Sword of Damocles from 1968. It can be worn on our faces like glasses such as Microsoft HoloLens (owned by Microsoft), Google Glass (owned by Google), and Ray-Ban Stories (owned by Ray-Ban in partnership with Facebook).

The hardware that one wears independently on their head like glasses, or a helmet, is referred to as a standalone headset or a head-mounted display (HMD). However, most people will access A.R. for the first time with smartphones.

Evolution And Advancement In Augmented Reality

While Augmented Reality technology has been in the picture since the 1960s, its evolution has sped up over the last couple of decades to the point at which it is pretty familiar to users nowadays. From wearing the Sword of Damocles to wearing Google Glass, AR technology has shown phenomenal growth.

Some notable milestones in Augmented Reality history:

Introduction

Augmented Reality enables the user to place virtual objects into the real world. Cameras, such as the one on a smartphone can help an app superimpose a three-dimensional (3D) virtual object on a real-world object. In this way, this immersive experience lets users think that these 3D objects exist in the real world. Users can reorient the device to explore their surroundings from different angles, interact with objects using gestures and movement, and even join other people in multiuser A.R. experiences. It is an enhanced version of the real world. The digital elements (or 3D objects) are placed in the virtual environment, and it seems as if the digital details are present in the real physical world.

Digital object in real-physical surrounding.

How Do AR Systems Work?

The critical components of the devices are:

Sensors and Cameras - It gathers real-world information (data) from the physical environment (surroundings).

2. A projection screen - It projects information about the environment.

3. A Processing unit - Process all the information gathered from the camera & different sensors.

4. Mirrors - There are one or more mirrors that take an active role by operating as the reflection system for the projected data. They are crucial as they can change the orientation of the images based on how the eye changes its position.

The AR ecosystem can be divided into five classes based on devices suitable for A.R. apps -

Mobile Based AR

Most users experience augmented reality for the first time with the help of A.R. apps on mobile or tablets. The exponential rise in smartphones has contributed to significant growth in the V.R. and A.R. industries. The same sensors which enable smartphones to work are also used by A.R. and V.R. headsets. These sensors are gyroscopes (senses angular orientation), accelerometers (senses axis orientation), and miniaturized high-resolution displays: the front and rear-facing cameras on the phone help create an immersive experience.

Mobile-based augmented Reality requires spatial mapping, depth mapping, light estimation, and feature points recognition. A feature point is a distinctive location in images, such as corners or T-junctions.

Google performed an A.R. experiment named the project Tango. Tango was a combination of custom software and additional hardware for unique depth-sensing smartphones. Tango-enabled phones could track the world around them. As a result, they were able to place unique digital overlays onto the actual environment. In 2017, Google announced AR Core. An evolution of the Tango platform. AR Core is an application programming interface or API for building and developing mobile AR experiences.

Glass-based A.R.

Smart glasses are lightweight and low-power wearables that provide first-person views, and glasses include products like Google Glass Enterprise Edition and Vuzix Blade. Like every other AR gadget, a powerful processor is required for Glass-based AR too. Google Glass Enterprise Edition 2 is equipped with a Snapdragon XR1 processor, enabling better support for Computer vision and Machine Learning use cases. It also supports Bluetooth and a wide range of wi-fi standards such as IEEE 802.11a/g/b/n/ac, dual-band. To sense different orientations and map surroundings, various sensors are included in Google Glass, such as 3-Axis Gyroscope, 3-Axis Accelerometer, and 3-Axis Magnetometer. Moreover, in September 2021, Facebook announced a partnership with Ray-ban to offer a new product called Stories.

The user below interacts with the environment using Google Glass where the orientation of images changes with the eye’s movement.

User wearing Google glass — User wearing google glass

Headset based AR

On the headset-based AR systems, as of early 2018, the most widely used HMD is the Microsoft HoloLens, a wireless device. It has a visor-like design and runs on a rechargeable battery, and its processing power is present inside its frame. As far as sensors in HoloLens are concerned, four visible light cameras for Head tracking, two I.R. cameras for eye tracking in addition to an accelerometer, gyroscope, and magnetometer are used. Like Google Glass, it also supports Bluetooth and Wi-fi {Wi-Fi 5 (802.11ac 2x2)}.

The defense personnel below is monitoring the environment using Microsoft HoloLens.

Use of HoloLens in Army — Army staff wearing HoloLens is surveying the surroundings

Limitations in AR

1.Low light condition.

Just like eyes, AR needs light to see its surrounding. The smartphone's camera and other sensors should be able to scan the feature points. In dim or dark environments, most Augmented Reality devices will struggle to get a clear understanding of the environment. Therefore, it is challenging for AR to perform under low light conditions. Such a condition is a problem for every AR tracking system that exists today.

2. Pure plane surface/environments.

AR tries to find distinct feature points to see, track, and remember for orienting the device and digital assets in an augmented experience. It means that the software will have more trouble with the plane surface like a white wooden coffee table than it would with a knotty wooden table with a coffee mug on it. The distinct texture is beneficial for providing the contrast needed to create feature points. The more differentiation/roughness on surfaces, the better the AR apps will function.

3. Mobile processors should be robust.

Augmented Reality software requires significant processing power, so only devices with a sufficiently powerful processor can handle the tasks.

Occlusion

A virtual object is projected on the person when they pass in front of the object. Ideally, to make the experience immersive, it should get eclipsed by the person. When this fails to happen, it breaks the illusion of the AR experience. Occlusion takes place when a real object blocks the view of a virtual object.

ARKit (Apple’s AR Framework) provides one type of occlusion when people are recognized within the viewing area. People occlusion is enabled to cover the app’s virtual content with people that ARKit perceives in the camera feed. The app can then render a virtual object behind people who pass in front of the camera. ARKit accomplishes the occlusion by identifying regions in the camera feed where people reside and preventing virtual content from drawing into that region’s pixels.

Immersive experience during occlusion — Immersive experience due to occlusion

Growth, Opportunities, & Future

Global Augmented/Virtual Reality (AR/VR) market revenue is estimated to reach $661.4 billion by 2025, increasing at a high compound annual growth rate (CAGR) of 86.3% from 2019 to 2025.

Several promising opportunities exist in gaming and consumer electronics. Healthcare will continue to be the primary focus as more emerging technologies become available. Opportunities will continue to grow for wearable sensors for healthcare that are configurable in smartwatches or armbands.

A supporting ecosystem will be required to adapt to the growth of new U.I. technologies. New interfaces require powerful computing capabilities. Meanwhile, neuromorphic computing approaches will be employed in training or running machine learning algorithms in cloud servers and on-device applications. Neuromorphic computing mimics neuro-biological architectures present in the nervous system.

For futuristic user interfaces to become ubiquitous, device security and user privacy become critical, and cybersecurity, business continuity plans, and risk assessments should be the primary concern. (Source)

Market Analysis

Competitive Landscape

The top players in this market have the most significant number of patents. Companies such as Sony Corp, Panasonic, Samsung Electronics, Microsoft Technology Licensing, Canon, Magic Leap, Intel, Qualcomm, Apple, Facebook Technologies, and L.G. Electronics have secured patents. Sony Corp has the most at 392 patent families. [Note: The data is extracted from the Orbit Intelligence patent analysis tool.]

Industrial Landscape

Augmented Reality is used in various industries such as the Gaming industry, Movie Industry, Medical Field, Tourism Industry, Education Sector, Military, and others. The top four sectors that are utilizing AR technologies are listed below.

Medical Industry

It is possible due to AR that in heart surgery, surgeons can use an AR display to see the various aspects of the heart upon which the surgery is being performed. An augmented reality system provides the doctor with computer-processed imaging data in real-time via dedicated hardware (Sensor, Camera, Projection Screen) and software. The projection is made possible by using displays, projectors, cameras, trackers, or other specialized equipment.

Gaming Industry

The augmented reality gaming market rose to a value of US$ 4.7 Billion in 2020 globally (Source). Pokémon G.O. is a well-known example of an AR app for gaming. It uses a smartphone camera, gyroscope, clock, and GPS to enable a location-based augmented reality.

As shown below, the illustration shows the digital object (Pokémon) placed in the real-world environment and is giving an immersive experience. This is what augmented reality is all about.

Pokemon Go is an example of AR in gaming industry — Pokémon is present in real-physical environment.

In another illustration, two players are playing Tic-Tac-Toe, indicating shared experience is possible in Augmented Reality.

Education Industry

Augmented Reality can also be used for training and educational purposes. It enables students and corporate employees to get well acquainted with new knowledge or acquire skills by experiencing and interacting with different forms of Reality, simulation, games, and others. The global augmented reality training and education market are expected to grow from $8.34 billion in 2020 to $10.23 billion in 2021 at a compound annual growth rate (CAGR) of 22.7%. (Source)

Real-time training with AR system — Real-Time Training with the help AR System.

Defense Industry

Augmented Reality makes the task of defense personnel a lot easier. The human eye cannot perceive everything, especially in low lighting conditions. For use in training and combat, U.S. Army closed a deal worth up to $21.88 billion over ten years to Microsoft for 1,20,000 augmented-reality headsets. The contract, first reported by CNBC on April 1, is the culmination of a years-long process for Microsoft adapting the HoloLens augmented-reality headset to military purposes. In military use, the headset is called the Integrated Visual Augmentation System (IVAS). The IVAS headset is based on HoloLens and is augmented by Microsoft Azure cloud services. The headset delivers enhanced situational awareness, helps in sharing information and decision-making in a variety of scenarios. Source

The US troops wearing Microsoft HoloLens — US troops wearing Microsoft HoloLens

Geographical Landscape

Today, the U.S. leads in terms of the total count of patent families for Augmented Reality. Other jurisdictions, such as China, Europe, Japan, and Korea, also have significant patent families, as seen in the graph depicted below.

Conclusion

Recently, the augmented reality domain has continued to demonstrate a mammoth growth and, the graph as shown below depicts the same. So, it is important and valuable to have patents in this space as 2019 reported a threefold sharp surge in the count of patent families (1330), showing around 200% rise over 2015 to 2019. [Note: The graph as shown below – Data for years 2020 and 2021 is provisional, as it takes ~18 months for a patent application to be published.]