Animated Emoji Technology
Customizable and animated emojis are fun and interesting to share with others; however, this appears to only be the start of technologies’ journey into the three-dimensional model generation area. With acquisitions and carrying algorithms that can read and analyze a user's facial expressions and emotions in real-time, we can expect to see a lot of innovation in this realm from Apple shortly.
Introduction
In one of our previous blogs, Facial Recognition Apps: Past, Present, and Future, we took a deep dive into facial recognition technology and how it works. An innovative and fun feature has recently emerged from facial recognition technology: Animated Emojis. Some of the most recognizable are those associated with Apple, i.e. iOS 12 and above.
A user can create and customize an emoji on an iPhone (iOS 12 and later) and iPadOS, known as a Memoji. These Memoji can then be sent through messages and animated to mimic a user's expressions and voice. While Apple is not the first or only company to use this technology, Apple has certainly made its mark on customizable and animated emojis. Let us examine how Apple's technology works and their journey of acquiring companies and patents through the years to get to where they are today.
The Technology
Apple's facial recognition technology, also used for Face ID, creates these animated emojis. Facial recognition uses hardware located in a small area at the top of the displays of compatible iPhones. This small area is known as Apple's TrueDepth Camera system.
This advanced technology within the TrueDepth Camera system can map the geometry of a user's face. The camera creates a depth map of a look by projecting and inspecting approximately 30,000 invisible dots. In addition, an infrared image of the user's face is taken. This depth map and captured infrared image are transformed into a mathematical representation. The transformation is completed by a part of the neural engine of the A11, A12 Bionic, A12X Bionic, and A13 Bionic chips. The latest Apple Chip A14 bionic has a 16-core neural engine, which when combined with machine learning accelerators and GPUs performs image recognition. Face ID uses this technology to compare the current data to the user's enrolled facial data. Animated emojis are created using the TrueDepth Camera system to capture and analyze over 50 muscles in a user's face.
History leading to Animated Emojis
Through the years, Apple has acquired many companies that are focused on facial expression recognition technologies:
Companies acquired by Apple over the years
(Infographic: Lumenci)
The technologies from PrimeSense, Faceshift, and RealFace can be seen in Apple's ARKit. For example, the class CIFaceFeature uses face/gaze tracking technology, allowing the tracking of facial positions and acting upon in the developer code. The combination of these technologies, along with those from Emotient, allow for dynamically animated emojis. Apple's acquisition of Emotient may be the most interesting of the group.
Emotient possessed algorithms for speedy face detection and expression analysis in real-time. In 2013, a team of scientists and researchers at the University of California - San Diego specializing in using machine learning for facial expression analysis began to create the technology that formed the basis of Emotient. Their work was built around the findings and research of Paul Ekman, a pioneer in the study of emotions and facial expressions. They used artificial intelligence to machine learn Dr. Ekman's research in micro-emotions, creating the FACET (Facial Action Coding System) System. Dr. Ekman later joined Emotient's Advisory board, where he helped to construct high-volume datasets. While this powerful technology will most likely be used for many purposes (voice-first systems, picture, and video analysis), it will also match a user's nuanced facial expressions to the animated emoji dynamically.
Patent around Apple's Animated Emojis
Apple has been granted many patents that deal with their Memojis and Animojis. On July 21, 2020, Apple was granted the US10719969 patent, named "Optimized avatar zones." This patent includes the creation of Apple's animated emojis and Memojis and the creation interface. The technology in the patent may initially be from the company Faceshift, with Guillaume P. Barlier listed as an inventor who joined Apple from Faceshift in 2015. The technology covered in the patent includes various customization options for the creation of a 3D avatar head. Also included are techniques that allow for dynamic changes to the avatar made by the user. Many of the patents granted to Apple involving Memoji have founders from the Faceshift team, demonstrating Apple's desires for acquiring the company.
The patent US10430642 was granted to Apple on October 1, 2019, titled "Generating animated three-dimensional models from captured images." This patent covers the generation of a three-dimensional model of a user from captured images or videos of the user. Tracking the user's expressions may be done via a machine learning network to refine the generated model. In doing so, the three-dimensional model will simulate the user's poses and presentations. Technologies from the companies Apple has acquired over the years are beginning to show in their emerging technology, including this patent.
Patent analysis of animated emoji technology
Paypal Inc is working on a patent for generating animated emoji mashups. According to this new method, an animated emoji is generated from the contextual information provided by the user. For eg; consider a scenario where two people are communicating over a mobile or laptop. If one of them sends/types “rock star”, the system would provide an emoji containing guitars or a rock & roll emoji combined with Stars instead of taking the literal meaning and providing a rock or a star emoji instead.
Conclusion
Apple has remained one of the top tech contenders for innovation, and their introduction of Memoji and Animoji is no exception. The tech giant has acquired companies that possessed innovative technology and algorithms over the years, which they have integrated into their products. Through their TrueDepth Camera system hardware, Apple can create a depth map and capture an infrared image of a user's face, which is used in Animoji functionality. According to a leading Analyst, Ming-Chi Kuo, the companies competing with Apple’s True depth camera system would require another two and a half years to replicate this technology. But, Xiaomi with its Mi 8 and Oppo with its Find X had tried to Mimic the Apple’s Animoji efforts. According to Oppo, the dot projector splashes 15,000 points of light on to the face unlike Apple which splashes 30,000 points of light.
Apple has been granted numerous patents regarding their Memoji and Animoji technology, mostly reflecting their acquired companies' original work and findings. Customizable and animated emojis are fun and interesting to share with others; however, this appears to only be the start of Apple's journey into the three-dimensional model generation area. With acquisitions such as Emotient, carrying algorithms that can read and analyze a user's facial expressions and emotions in real-time, we can expect to see a lot of innovation in this realm from Apple shortly.
Author
Ryan Gutierrez
Associate at Lumenci
Ryan Gutierrez is an Associate at Lumenci with experience in Software and Web/App Development. He holds a Bachelor of Science in Computer Science from the University of Texas at Austin. He is interested in patent litigation and researching new advancements in technology. He enjoys art, reading, and dancing in his spare time.