Let the vision be augmented ... |

 

Indoor tracking

Precise indoor localisation is an important issue for Augmented Reality. The localized device is often held by an individual, allowing applications, to interact with the physical environment. Firstly, an accurate three dimensional model of the indoor environment is necessary for our applications. Surveying strategies are shown in the first section. Secondly, we attach heterogeneous tracking sensors to the handheld device to fuse the sensor data, which leads to a seamless tracking experience.

 

Towards the self-surveying building - Construction and maintenance of Augmented Reality Environments

 

Sentient Computing demands a detailed and up-to-date model of spatial relationships, which appear to reproduce the perceptions a user has of the world. Given a suitably large model it is possible to create AR scenarios that allow users to roam through a wide area interacting with a rich and responsive environment. Hence the two major ingredients necessary to experiment with and build ''Location-aware'' AR applications are tracking and modelling. We used the following approaches to obtain a 3D model of our building.

Total station

Earlier surveying techniques involved the use of reflectorless Total stations to survey the positions of fiducial markers.

Screenshot of the manual surveyed model of the building.

The data is stored as XML-encoded BAUML (Building Augmentation Language). Floor, walls, portals (doors) and ceiling are clearly visible in the model. The positions and orientations of the fiducial markers are represented as red cones. The accuracy of the model is in the area of millimetres. Consequently the model is very accurate compared to existing plans of building. On the other hand this manual approach is very time-consuming, and presents a serious barrier to the introduction of AR to new environments. Therefore automatic methods are necessary to speed up the process.

Robot surveying

We used an autonomously navigating mobile robot - PeopleBot (ActivMedia) - to detect and localise the fiducial markers and build a model that can be used by existing AR systems. The mobile robot is equipped with a laser range finder to localise the robot as well as with a digital camera. The images taken from the camera are used to detect the fiducial markers.


For generating a 3D model we use a hybrid approach in which measurements from both the robot and the Total station is combined. These points to a composite strategy in which, the Total station can be used to survey important rooms with significant resources and equipment that needs to be surveyed and calibrated. These rooms are often cluttered and do not lend themselves to automatic surveying. However, these manually surveyed "islands" can be chained together using the robot that can easily navigate between rooms through corridors.

 

SLAV (Simultaneous Localization And Visualization)

 

The vision of the project is a building that surveys itself and offers an updated 3D model of its geometry and tracking infrastructure to mobile users wirelessly. The Muddleware server stores the reconstructed model and offers them to AR clients.

The first step towards this goal is to develop an autonomous visual reconstruction system based on the Peoplebot platform. ArtoolKitPlus fiducial markers are detected with a single camera computer vision system and are integrated in a sparse 3D point reconstruction. This reconstruction is made using visual odometry while the robot is surveying the building. The data should be combined with measurements from the 2D laser range finder and optimized in an offline bundle adjustment batch process. In this batch process natural features can be used for loop-closing and they may be used later in AR-applications on the client side to provide additional information to the fiducial markers. The density of fiducial markers in the building may also be reduced.

Peoplebot The Peoplebot platform.

 

Handheld Multi-Sensor Tracking

 

For multi-sensor indoor tracking we use a handheld computer (Sony Vaio U70), and combine different tracking systems. These are Inertial (InertiaCube3), optical (ARToolKitPlus), and Ultra Wide Band (UWB Ubisense).


Additionally to single marker tracking we use multimarker tracking.

  • Inertial tracking for orientation tracking. Isense
  • UWB tracking for Position tracking.Ubisense
Each tracking technology has a dedicated working volume, with the exception of the inertial orientation tracker. The inertial tracker is used to assist other tracking systems with dead reckoning information, in particular the UWB system, which does not deliver any estimates of orientation. The USB Camera attached to the U70 provides video input to ARToolKitPlus and is also used to render the video background of the hand-held video see-through display.
As the core tracking software, we use an updated version of OpenTracker (OT), which implements a pipes-and-filters network for connecting producers and consumers of tracking information.

 

Augmented Reality Setups

 

Early work on mobile AR used bulky backpack prototypes. However, there is a trend towards smaller, discreet, lightweight hand-held setups that are much more socially acceptable in environments in which PDAs and smartphones are already commonplace.

Backpack system


User wearing a backpack setup equipped with a laptop, a helmet with Sony Glasston see-through glasses, a camera for vision tracking, and an inertial sensor. (2004).

Handheld setup with a VAIO U70

We have built a prototype hand-held system, consisting of a Sony VAIO U70 and a variety of different sensors attached to an acrylic mount. The sensors consist of a USB camera serving the dual purpose of providing images for an optical tracking system and also for providing the video required by the magic lens metaphor; a Ubitag, providing position estimates only; and an Intersense InertiaCube3 inertial tracker providing orientation estimates only. The latter two sensors are highly complementary as, when aggregated, they provide the full six degrees of freedom necessary for tracking rigid bodies.


Front and back view of AR platform.


User with the Sony VAIO U70 setup (2005).

Handheld setup with an UMPC Samsung Q1


Front and back view of Q1 equipped with tracking setup.


User with the UMPC Samsung Q1 (2006).

 

Navigation Application

 

Previous experiments with indoor navigation systems relied solely on widely distributed fiducial markers to provide a wide-area vision-based tracking capability of moderate accuracy. New tracking technologies, such as Ubisense's, robustly cover large areas without the visual clutter of visual markers, or the brittleness associated with natural-feature based vision trackers. This motivation lead us to explore how a widearea tracker, that can only sense position, lends itself to a hybrid approach whereby it is combined with complementary sensors to yield the pose estimates required for augmenting a user's view.
When a fiducial marker is visible then the pose is taken directly from the vision algorithms; however, when moving into an area where fiducials are either no longer present or are not visible due to occlusion, then the positional component of pose is taken from the Ubisense system and the orientational component of pose is taken from the inertial tracker. A real Ubicomp environment, its size notwithstanding, will be richly populated with objects both static and dynamic.
The following figures show the navigation system in action, with navigational cues, state information and current location visible using a "World in minature" view.

Navigating towards destination "corridor". Location can be determined from Ubisense wide-area tracker together with observations of fiducial markers. Necessary direction of travel indicated by compass pointer in topright.



Navigation to destination "corridor" completed. Full range of sensors, including fiducial markers and Ubisense wide-area tracker are utilised.


Overlay of 3D model on real world

The following pictures are screenshots made from the navigation application. They show the overlay of the 3D model on the real environment that is done in realtime. Also the world-in-miniature model is visualised which shows the position of the user in the building.



Ubitrack - Pervasive 2006

The following pictures were taken at PERVASIVE 2006 conference in Dublin, where a live demo of the handheld setup was shown at the Ubisense booth.


 

Related Publications

 

G. Schall, J. Newman, D. Schmalstieg, "Rapid and Accurate Deployment of Fiducial Markers for Augmented Reality", In Proceedings of the 10th Computer Vistion Winter Workshop, Zell an der Pram, Upper Austria, 2005

J. Newman, F. Fraundorfer, G. Schall, D. Schmalstieg, "Construction and Maintenance of Augmented Reality Environments using a Mixture of Autonomous and Manual Surveying Techniques", In Proceedings of the 7th conference on Optical 3-D Measurement Techniques, Vienna, 2005

J. Newman, G. Schall, Istvan Barakonyi, A. Schürzinger, D. Schmalstieg, "Wide-Area Tracking Tools for Augmented Reality", In Proceedings of the 4th International Conference on Pervasive Computing, Dublin, 2006

J. Newman, G. Schall, D. Schmalstieg, "Modelling and Handling Seams in Wide-Area Sensor Networks ", In Proceedings of the 10th IEEE International Symposium on Wearable Computers (ISWC`06), Montreaux, Swisserland, 2006

 

Group in Graz

 

 

Videos

 

 

copyright (c) 2006 Graz University of Technology