Tutorials

Abstract

This tutorial is a revised and updated edition of the OpenARK tutorial presented at ISMAR 2019 and 2020. The aim of this tutorial is to present an open-source augmented reality development kit, called OpenARK. OpenARK was founded at UC Berkeley in 2015. Since then, the project has received high-impact awards and visibility. Currently OpenARK is being used by several industrial alliances including HTC Vive, Siemens, Ford, and State Grid. In 2018, OpenARK won the only Mixed Reality Award at the Microsoft Imagine Cup Global Finals. In the same year in China, OpenARK also won a Gold Medal at the Internet+ Innovation and Entrepreneurship Competition, the largest such competition in China. OpenARK has received funding support from research grants by an Intel RealSense project, NSF and ONR.

OpenARK includes a multitude of core functions critical to AR developers and future products. These functions include multi-modality sensor calibration, depth-based gesture detection, depth-based deformable avatar tracking, and SLAM and 3D reconstruction. In addition to these functionalities, a lot of recent work has gone into developing a real time deep learning 3D object tracking module as a solution to the problem known as Digital Twin. The Digital Twin problem is the problem of overlaying a virtual augmentation over a real object with near perfect accuracy, enabling a wide variety of AR functionalities. All functions are based on state-of-the-art real-time algorithms and are coded to be efficient on mobile-computing platforms.

Another core component in OpenARK is its open-source depth perception databases. Currently we have made two unique databases available to the community, one on depth-based gesture detection and the other on mm-accuracy indoor and outdoor large-scale scene geometry models and AR attribute labeling. We would like to overview our effort in the design and construction of these databases that potentially could benefit the community at large.

Finally, we will discuss our effort in making depth-based perception easily accessible to application developers, who may not have and should not be forced to learn good understanding about 3D point cloud and reconstruction algorithms. The last core component of OpenARK is an interpreter of 3D scene layouts and its compatible AR attributes based on generative design principles first invented for creating architectural design layouts. We will discuss the fundamental concepts and algorithms of generative design and how it can be used to interpret common 3D scenes and their attributes for intuitive AR application development.

The teaching material of this tutorial will be drawn from a graduate-level advanced topics course on AR/VR offered at UC Berkeley for the past three years. Our teaching material can be downloaded from two websites: