Trends in Augmented Reality Tracking, Interaction and Display: A Review of Ten Years of ISMAR

ISMAR 2008
7th IEEE/ACM International Symposium on Mixed and Augmented Reality, 2008

http://www.augmented-reality.org/

Proceedings
State of the Art Report

Trends in Augmented Reality Tracking, Interaction and Display
: A Review of Ten Years of ISMAR
Feng Zhou (Center for Human Factors and Ergonomics, Nanyang Technological University, Singapore)
Henry Been-Lirn Duh (Department of Electrical and Computer Engineering/Interactive and Digital Media Institute, National University of Singapore)
Mark Billinghurst (The HIT Lab NZ, University of Canterbury, New Zealand)

Trends in Augmented Reality Tracking, Interaction and Displ.pdf

Tracking

1. Sensor-based tracking -> ubiquitous tracking and dynamic data fusion

2. Vision-based tracking: feature-based and model-based

1) feature-based tracking techniques:
- To find a correspondence between 2D image features and their 3D world frame coordinates.
- Then to find the camera pose from projecting the 3D coordinates of the feature into the observed 3D image coordinates and minimizing the distance to their corresponding 3D features.

2) model-based tracking techniques:
- To explicitly use a model of the features of tracked objects such as a CAD model or 2D template of the object based on the distinguishable features.
- A visual serving approach adapted from robotics to calculate camera pose from a range of model features (line, circles, cylinders and spheres)
- knowledge about the scene by predicting hidden movement of the object and reducing the effects of outlier data

3. Hybrid tracking
- closed-loop-type tracking based on computer vision techonologies
- motion prediction
- SFM (structure from motion)
- SLAM (simultaneous localization and mapping)

Interaction and User Interfaces

1. Tangible
2. Collaborative
3. Hybrid

Display

1. See-through HMDs

1) OST = optical see-through
: the user to see the real world with virtual objects superimposed on it by optical or video technologies
2) VST = video see-through
: to display graphical infromation directly on real objects or even daily surfaces in everyday life

2. Projection-based Displays
3. Handheld Displays

Limitations of AR

> tracking
1) complexity of the scene and the motion of target objects, including the degrees of freedom of individual objects and their represenation
=> correspondence analysis: Kalman filters, particle filters.
2) how to find distinguishable objects for "markers" outdoors

> interaction
ergonomics, human factors, usability, cognition, HCI (human-computer interaction)

> AR displays
- HMDs - limited FOV, image distortions,
- projector-based displays - lack mobility, self-occlusion
- handheld displays - tracking with markers to limit the work range

Trends and Future Directions

1. Tracking

1) RBPF (Rao-Blackwellized particle filters) -> automatic recognition systems
2) SLAM, ubiquitous tracking, sensor network -> free from prior knowledge
3) pervasive middleware <- information fusion algorithms

2. Interaction and User Interfaces

"Historically, human knowledge, experience and emotion are expressed and communicated in words and pictures. Given the advances in interface and data capturing technology, knowledge, experience and emotion might now be presented in the form of AR content."

3. AR Displays

pico projectors
http://gizmodo.com/tag/pico-projectors
http://en.wikipedia.org/wiki/Pico_projector

Studierstube Augmented Reality Project
: software framework for the development of Augmented Reality (AR) and Virtual Reality applications
Graz University of Technology (TU Graz)

Sharedspace project
The Human Interface Technology Laboratory (HITLab) at the University ofWashington and ATR Media Integration & Communication in Kyoto,Japan join forces at SIGGRAPH 99

The Invisible Train - A Handheld Augmented Reality Game

AR Tennis
camera based tracking on mobile phones in face-to-face collaborative Augmented Reality

Emmie - Environment Management for Multi-User Information Environments

VITA: visual interaction tool for archaeology

HMD = head-mounted displays

OST = optical see-through

VST = video see-through

ELMO: an Enhanced optical see-through display using an LCD panel for Mutual Occlusion

FOV
http://en.wikipedia.org/wiki/Field_of_view_(image_processing)

HMPD = head-mounted projective displays

The Touring Machine

MARS - Mobile Augmented Reality Systems
    
Klimt - the Open Source 3D Graphics Library for Mobile Devices

AR Kanji - The Kanji Teaching application


references  

---

Ronald T. Azuma  http://www.cs.unc.edu/~azuma/
A Survey of Augmented Reality. Presence: Teleoperators and Virtual Environments 6, 4 (August 1997), 355 - 385. Earlier version appeared in Course Notes #9: Developing Advanced Virtual Reality Applications, ACM SIGGRAPH '95 (Los Angeles, CA, 6-11 August 1995), 20-1 to 20-38.

Ronald Azuma, Yohan Baillot, Reinhold Behringer, Steven Feiner,Simon Julier, Blair MacIntyre
Recent Advances in Augmented Reality.IEEE Computer Graphics and Applications 21, 6 (Nov/Dec 2001),34-47.

Ivan E. Sutherland
The Ultimate Display, IFIP `65, pp. 506-508, 1965

Kato, H.   Billinghurst, M.   Poupyrev, I.   Imamoto, K.   Tachibana, K.   Hiroshima City Univ.
Virtual object manipulation on a table-top AR environment

Sandor, C., Olwal, A., Bell, B., and Feiner, S. 2005.
Immersive Mixed-Reality Configuration of Hybrid User Interfaces.
In Proceedings of the 4th IEEE/ACM international Symposium on Mixed and Augmented Reality(October 05 - 08, 2005). Symposium on Mixed and Augmented Reality. IEEEComputer Society, Washington, DC, 110-113. DOI=http://dx.doi.org/10.1109/ISMAR.2005.37

An optical see-through display for mutual occlusion with a real-time stereovision system
Kiyoshi Kiyokawa, Yoshinori Kurata and Hiroyuki Ohno
Computers & Graphics Volume 25, Issue 5, October 2001, Pages 765-779

Bimber, O., Fröhlich, B., Schmalstieg, D., and Encarnação, L. M. 2005.
The virtual showcase. In ACM SIGGRAPH 2005 Courses (Los Angeles, California, July 31 - August 04, 2005). J. Fujii, Ed. SIGGRAPH '05. ACM, New York, NY, 3. DOI= http://doi.acm.org/10.1145/1198555.1198713

Bimber, O., Wetzstein, G., Emmerling, A., and Nitschke, C. 2005.
Enabling View-Dependent Stereoscopic Projection in Real Environments. In Proceedings of the 4th IEEE/ACM international Symposium on Mixed and Augmented Reality (October 05 - 08, 2005). Symposium on Mixed and Augmented Reality. IEEE Computer Society, Washington, DC, 14-23. DOI= http://dx.doi.org/10.1109/ISMAR.2005.27

Cotting, D., Naef, M., Gross, M., and Fuchs, H. 2004.
Embedding Imperceptible Patterns into Projected Images for Simultaneous Acquisition and Display. In Proceedings of the 3rd IEEE/ACM international Symposium on Mixed and Augmented Reality (November 02 - 05, 2004). Symposium on Mixed and Augmented Reality. IEEE Computer Society, Washington, DC, 100-109. DOI= http://dx.doi.org/10.1109/ISMAR.2004.30

Ehnes, J., Hirota, K., and Hirose, M. 2004.
Projected Augmentation - Augmented Reality using Rotatable Video Projectors. In Proceedings of the 3rd IEEE/ACM international Symposium on Mixed and Augmented Reality (November 02 - 05, 2004). Symposium on Mixed and Augmented Reality. IEEE Computer Society, Washington, DC, 26-35. DOI= http://dx.doi.org/10.1109/ISMAR.2004.47

Arango, M., Bahler, L., Bates, P., Cochinwala, M., Cohrs, D., Fish, R., Gopal, G., Griffeth, N., Herman, G. E., Hickey, T., Lee, K. C., Leland, W. E., Lowery, C., Mak, V., Patterson, J., Ruston, L., Segal, M., Sekar, R. C., Vecchi, M. P., Weinrib, A., and Wuu, S. 1993.
The Touring Machine system. Commun. ACM 36, 1 (Jan. 1993), 69-77. DOI= http://doi.acm.org/10.1145/151233.151239

Gupta, S. and Jaynes, C. 2006.
The universal media book: tracking and augmenting moving surfaces with projected information. In Proceedings of the 2006 Fifth IEEE and ACM international Symposium on Mixed and Augmented Reality (Ismar'06) - Volume 00 (October 22 - 25, 2006). Symposium on Mixed and Augmented Reality. IEEE Computer Society, Washington, DC, 177-180. DOI= http://dx.doi.org/10.1109/ISMAR.2006.297811


Klein, G. and Murray, D. 2007.
Parallel Tracking and Mapping for Small AR Workspaces. In Proceedings of the 2007 6th IEEE and ACM international Symposium on Mixed and Augmented Reality - Volume 00 (November 13 - 16, 2007). Symposium on Mixed and Augmented Reality. IEEE Computer Society, Washington, DC, 1-10. DOI= http://dx.doi.org/10.1109/ISMAR.2007.4538852

Neubert, J., Pretlove, J., and Drummond, T. 2007.
Semi-Autonomous Generation of Appearance-based Edge Models from Image Sequences. In Proceedings of the 2007 6th IEEE and ACM international Symposium on Mixed and Augmented Reality - Volume 00 (November 13 - 16, 2007). Symposium on Mixed and Augmented Reality. IEEE Computer Society, Washington, DC, 1-9. DOI= http://dx.doi.org/10.1109/ISMAR.2007.4538830