Course: intro to AR

google Coursera course: Introduction to augmented reality and ar core

We don't normally summarise course content.  Yet, this particular course offers a clear and accessible introduction to augmented reality technologies, so it seems helpful to flag it here.  The main points covered within the course are listed below:

The basics of augmented reality

• Humankind’s first foray into immersive reality through a head-mounted display was the “Sword of Damocles,” created by Ivan Sutherland in 1968.
• HMD is the acronym for “head-mounted display.”
• The term “Augmented Reality” was coined by two Boeing researchers in 1992.
• A standalone headset is a VR or AR headset that does not require external processors, memory, or power.
• Through the combination of their hardware and software, many smartphones can view AR experiences that are less immersive than HMDs.
• Many of the components in smartphones—gyroscopes, cameras, accelerometers, miniaturized high-resolution displays—are also necessary for AR and VR headsets.
• The high demand for smartphones has driven the mass production of these components, resulting in greater hardware innovations and decreases in costs.
• Project Tango was an early AR experiment from Google, utilizing a combination of custom software and hardware innovations that lead to a phone with depth-sensing cameras and powerful processors to enable high fidelity AR.
• An evolution of Project Tango, ARCore is Google’s platform for building augmented reality experiences.

AR functionality

• In order to seem real, an AR object has to act like its equivalent in the real world. Immersion is the sense that digital objects belong in the real world.
• Breaking immersion means that the sense of realism has been broken; in AR this is usually by an object behaving in a way that does not match our expectations.
• Placing is when the tracking of a digital object is fixed, or anchored, to a certain point in the real world.
• Scaling is when a placed AR object changes size and/or dimension relative to the AR device's position. For example, when a user moves away or towards an AR object, it feels like the object is getting larger or smaller depending on the distance of the phone in relation to the object. AR objects further away from the phone look smaller and objects that are closer look larger. This should mimic the depth perception of human eyes.
• Occlusion occurs when one object blocks another object from view.
• AR software and hardware need to maintain “context awareness” by tracking the physical objects in any given space and understanding their relationships to each other -- i.e. which ones are taller, shorter, further away, etc.

Inside-out vs. outside-in tracking

• There are two basic ways to track the position and orientation of a device or user: outside-in tracking and inside-out tracking.
• Outside-in tracking uses external cameras or sensors to detect motion and track positioning. This method offers more precision tracking, but a drawback is the external sensors lower the portability.
• Inside-out tracking uses cameras or sensors located within the device itself to track its position in the real world space. This method requires more hardware in the AR device, but offers more portability.
• On the AR headset side, the Microsoft HoloLens is a device that uses inside-out tracking. On the VR headset side, the HTC Vive is a device that uses outside-in tracking.
• On the AR mobile side, the Google Pixel is a smartphone that uses inside-out tracking for AR.

Fundamentals of ARCore

• ARCore integrates virtual content with the real world as seen through your phone's camera and shown on your phone's display with technologies like motion tracking, environmental understanding, and light estimation.
• Motion tracking uses your phone's camera, internal gyroscope, and accelerometer to estimate its pose in 3D space in real time.
• Environmental understanding is the process by which ARCore “recognizes” objects in your environment and uses that information to properly place and orient digital objects. This allows the phone to detect the size and location of flat horizontal surfaces like the ground or a coffee table.
• Light estimation in ARCore is a process that uses the phone’s cameras to determine how to realistically match the lighting of digital objects to the real world’s lighting, making them more believable within the augmented scene.
• Feature points are visually distinct features in your environment, like the edge of a chair, a light switch on a wall, the corner of a rug, or anything else that is likely to stay visible and consistently placed in your environment.
• Concurrent odometry and mapping (COM) is a motion tracking process for ARCore, and tracks the smartphone’s location in relation to its surrounding world.
• Plane finding is the smartphone-specific process by which ARCore determines where surfaces are in your environment and uses those surfaces to place and orient digital objects. ARCore looks for clusters of feature points that appear to lie on common horizontal or vertical surfaces, like tables or walls, and makes these surfaces available to your app as planes. ARCore can also determine each plane's boundary and make that information available to your app. You can use this information to place virtual objects resting on flat surfaces.
• Anchors “hold” the objects in their specified location after a user has placed them.
• Motion tracking is not perfect. As you walk around, error, referred to as drift, may accumulate, and the device's pose may not reflect where you actually are. Anchors allow the underlying system to correct that error by indicating which points are important.

Constraints with current AR

• Currently AR has a lack of user interface metaphors, meaning that a commonly understood method or language of human interaction has not been established.
• The purpose of the interface metaphor is to give the user instantaneous knowledge about how to interact with the user interface. An example is a QWERTY keyboard or a computer mouse.
• The details of what makes AR challenging from a technical standpoint are complex, but three influential factors are power, heat, and size.
• AR requires high processing power, batteries generate heat, and a current challenge is fitting all the necessary components into a small enough form factor to wear on your face comfortably for extended periods of time.
• Not everything in AR has to be 3D, but the vast majority of assets, applications, and experiences will require at least a little 3D design.
• Currently, there is a limited base of people with 3D design and interaction skills, such as professional animators, graphic designers, mechanical engineers, or video game creators. For AR to grow, the adoption of 3D design theory, skills, and language needs to become much more widespread. Later on in this course, we’ll be discussing a few programs that are helping overcome this challenge, like Sceneform or Poly API.
• Computer vision is a blend of artificial intelligence and computer science that aims to enable computers (like smartphones) to visually understand the surrounding world like human vision does. This technology needs to improve in terms of object detection and segmentation to make AR processes more effective.

Use cases and current powers/limitations of AR

• ARCore can be used to create dynamic experiences for businesses, nonprofits, healthcare, schools, and more.
• ARCore’s strengths are its phone-based spatial mapping capabilities and addressable user base. Approximately 85% of phones around the world run on the Android operating system.
• At the beginning of 2018, ARCore is already available on 100 million Android-powered smartphones and that number continues growing. ARCore requires a lot of processing power, so not all older Android models have the necessary specifications yet. ARCore is also available in China.
• Limitations to consider with contemporary AR technology include: low-light environments, a lack of featured surfaces, and the availability of powerful mobile processors in new phones.

Basic AR interaction options

1.     Drag and Drop
2.     Voice
3.     Tap
4.     Pinch and Zoom
5.     Slide
6.     Tilt

Think like a user

• User flow is the journey of your app's users and how a person will engage, step by step, with your AR experience.
• Planning your user flow needs to take into account the scene, the user interactions, any audio cues, and the final user actions.
• A user flow can be created with simple sketches and panels all collected into one cohesive diagram.
• UX and UI are complementary fields of product design, and generally speaking UX is the more technical of the two.
• When considering UX/UI, one good rule of thumb to remember with AR is to avoid cluttering the screen with too many buttons or elements that might be confusing to users.
• Choosing to use cartoonish designs or lighting can actually make the experience feel more realistic to the user, as opposed to photorealistic assets that fail to meet our expectations when they don't blend in perfectly with the real world.
• Users might try to “break” your experience by deliberately disregarding your carefully planned user flow, but your resources are better spent on improving your app’s usability rather than trying to prevent bad actors

Next steps on the AR journey

• Advanced 3D design tools like Maya, Zbrush, Blender, and 3ds Max are powerful professional tools.
• Google’s Poly can be a good starting resource for building your first ARCore experience.
• Poly by Google is a repository of 3D assets that can be quickly downloaded and used in your ARCore experience.
• The recommended guide for your AR experience is a design document that contains all of the 3D assets, sounds, and other design ideas for your team to implement.
• You may need to hire advanced personnel to help you build your experience, such as: 3D artists, texture designers, level designers, sound designers, or other professionals.

A closer look at mechanics of ARCore

• Surface detection allows ARCore to place digital objects on various surface heights, to render different objects at different sizes and positions, and to create more realistic AR experiences in general.
• Pose is the position and orientation of any object in relation to the world around it. Everything has its own unique pose: from your mobile device to the augmented 3D asset that you see on your display.
• Hit-testing lets you establish a pose for virtual objects and is the next step in the ARCore user process after feature-tracking (finding stationary feature points that inform the environmental understanding of the device) and plane-finding (the smartphone-specific process by which ARCore determines where horizontal surfaces are in your environment).
• Light estimation is a process that allows the phone to estimate the environment's current lighting conditions. ARCore is able to detect objects in suboptimal light and map a room successfully, but it’s important to note that there is a limit to how low the light can be for the experience to function.
• Occlusion is when one 3D object blocks another 3D object. Currently this is only possible with digital objects, and AR objects cannot be occluded by a real world object. For example, in an AR game the digital object would not be able to behind a real couch in the real world.
• Assets in multi-plane detection are scaled appropriately in relationship to the established planes, though only need to be placed on them (via anchor points) when it causes them to function like their real-world counterparts.
• Immersion can be broken by users interacting with AR objects as if they were physically real. Framing can be used to combat these immersion-breaking interactions.
• Spatial mapping is the ability to create a 3D map of the environment and helps establish where assets can be placed.
• Feature points are stationary and are used to further environmental understanding and place planes in an experience. ARCore assumes planes are unmoving, so it is inadvisable to attempt to anchor a digital object to a real world object that is in motion. In general, it’s best not to place an object until the room has been sufficiently mapped and static surfaces have been recognized and designated as feature points.

Using Poly and Unity to create ARCore assets

• Unity is a cross-platform game engine and development environment for both 3D and 2D interactive applications. It has a variety of tools, from the simple to the professionally complex, to allow for the streamlined creation of 3D objects and environments.
• Poly toolkit for Unity is a plugin that allows you to import assets from Poly into Unity at edit time and at runtime.
• Edit-time means manually downloading assets from Poly and importing them into your app's project while you are creating your app or experience.
• Runtime means downloading assets from Poly when your app is running. This allows your app to leverage Poly's ever-expanding library of assets.

Even despite Google's undeniable motivation to promote its own products via this course, the course explanation of the above summary points is helpful and clear, enhanced by videos, diagrams and graphics.

Coursera course content is free for University students.  Certification is optional for an extra fee.  

VR, Google AR &. "Introduction to Augmented Reality & AR Core." Coursera, accessed 22.9.20. https://www.coursera.org/learn/ar.

---