AR	Augmented Reality
CAAP	Computer-aided Assembly Planning Software
CAD	Computer-Aided Design Software
CAM	Computer-Aided Manufacturing Software
CNC	Computerized Numerical Control
HMD	Head-Mounted Display
VR	Virtual Reality

1 Augmented Reality

With Augmented reality (AR) the live view of the real world is merged with augmented computer-generated images. This augmentation is done in real time and in semantic context with environmental elements. All information that is hereby augmented to the real world becomes interactive and digitally usable. (Furth, 2011 p. VII).

Furthermore, AR is interactive and registered in 3D according to Carmigniani et al. It combines real and virtual objects and “aims at simplifying the user’s life by bringing virtual information not only to his immediate surroundings, but also to any indirect view of the real-world environment, such as live-video streams“(Carmigniani, et al., 2011 p. 3).

AR is oftentimes implemented with head-mounted displays (HMD), computers or portable devices like Smartphones, but it is not limited to improve the sense of sight. In fact, AR could be applied to all kinds of senses like smell, touch or hearing. Thus it could be used to augment or substitute users missing senses by sensory substitution like augmenting the sight of blind users or users with poor vision, and it could as well augment hearing of deaf users by the use of visual cues (Carmigniani, et al., 2011 p. 4).

While AR used to appear only in science fiction movies for a long time it has now been applied broadly in industrial environments (Ong, et al., 2011 p. 651). Nowadays AR can be applied in many fields of the product development cycle and in manufacturing processes, including:

· Product Design,

· Design Review and Design Evaluation,

· Material Processing,

· Assembly Planning and Verification,

· Assembly Training and Guidance,

· Robot Control and Programming for Industrial Applications,

· Factory Facility Layout Planning, as well as

· Maintenance

While computer-aided design software (CAD), computer-aided manufacturing software (CAM), as well as computer-aided assembly planning software (CAAP) have traditionally been used in these fields for a long time, virtual reality (VR) technologies were applied in addition to these programs to enhance the aforementioned processes recently (Ong, et al., 2011 p. 651).

Although VR and AR have many similarities, both technologies have different areas of application. Ong et al consider VR to outperform AR-based systems for “applications where process simulation and interactions using highly accurate models are vital while little intervention from the user is needed” (Ong, et al., 2011 pp. 651, 652). Reasons for this are the presence of tracking errors of currently available AR implementations. In contrast, AR is better suited to applications where manipulations and interactions with tangible objects are important.

2 AR Hardware Configurations

Object tracking is one of the most crucial tasks when implementing AR-based systems due to the high occurrence of tracking errors. The following section outlines several hardware settings and implementations of such systems.

2.1 Tracking and Registration Methods

All AR implementations rely on some sort of tracking and registrations methods. This is necessary either to determine the additional content that has to be displayed or, regarding more advanced scenarios, as a point of reference in physical space to determine where to display content on a screen.

Tracking Fiducials – commonly referred to as markers or patterns – with cameras are among the most common AR implementations and are applied widely in industrial applications. Figure 1 demonstrates camera tracking using the FLARToolkit library, a port from the widely used ARToolkit library to Adobe Flash:

Figure 1 - Marker Tracking using FLARToolKit to display three colored cubes

Tracking single markers is quite error-prone as tracking will fail easily if the marker is occluded in the tracked image. That’s why in most scenarios multiple markers are combined. Figure 2 demonstrates an approach of using clustered tags. Here the digital environment can still be displayed correctly even if some tags are hidden behind the users’ hand.

Figure 2 - Multiple Marker Tracking using ARTag

While recognizing and tracking marker is one of the best performing approaches of using camera tracking, it is not possible to put them on tangible objects in all situations.

Natural feature tracking can be used when attaching markers to real objects cannot be implemented. As mentioned by Ong et al, a contour-based markerless tracking method was applied for car engine maintenance. Here, contours are extracted from CAD models under different camera viewpoints and varied illuminations so that the objects can even be tracked with changing environmental influences (Ong, et al., 2011 p. 665).

2.2 User Interactions

While markers are used to find the reference to a point of origin in a digital image they are not solely used to add interaction to the application. Of course, it is possible to control the application by measuring the angle of the marker in relation to the camera or to change the application logic when a specific combination of markers is tracked concurrently, but such an implementation might not be reliable enough in advanced application scenarios. Sometimes it is sufficient to use AR mainly to display additional content and implement interaction capabilities only via keyboards or other kinds of tangible user interfaces.

There are, however, scenarios in which more advanced capabilities using marker-based interaction methods are implemented. Ong et al. mention implementations using marker cubes, pinch gloves attached with markers, as well as conductive materials and vibration motors for hand gesture detection and tactile feedback rendering. He also mentioned fingertip detection for determining the triggering of button functions in conjunction with AR (Ong, et al., 2011 p. 664). Figure 3 shows the appearance and the plan of an AR cube.

Figure 3 - Cube with six AR markers

2.3 Occlusion Handling

While occlusion handling in virtual 3D space is a common procedure, applying it to 3D content that is mixed with physical objects in the real world presents a non-trivial problem as the layer in which the 3D content is rendered usually covers the view to the real world.

One approach of implementing occlusion handling when mixing digital and virtual objects with AR is to calculate depth images for both virtual and physical elements and then compare them pixel by pixel. This only works, however, when CAD models of all physical elements are available (Ong, et al., 2011 p. 664)

While the aforementioned approach could be applied to a broad range of AR applications, it cannot be applied to all scenarios. A specialized example of an AR application would be a hand tracking application in which the hand of the user should be displayed on top of the actual virtual content. To accomplish this, it is possible to render the skin region of the user’s hand on top of the digital content and therefore pretend that the content is behind the hand (Ong, et al., 2011 p. 664). Figure 4 demonstrates the use of a depth-of-flight sensor to produce a 2D map of the distances to real physical objects in the environment and therefore calculates if the virtual objects are behind or in front of the physical object.

Figure 4 - Depth-of-flight sensor in AR application

3 AR Implementations

There already exists a wide range of AR applications that are used in different scenarios of production and manufacturing processes. This chapter gives some examples of how AR is used and how it is implemented.

3.1 Product Design

AR can be used in the early design stages and allows experts and consumers alike to provide feedback to designers in a collaborative way. While CAD-based and VR-based systems could be used as well for this purpose they are considered to lack the intuitiveness to let these people participate in the design process without having been trained first. Therefore, Ong et al claim that “AR-Based design spaces address these problems by introducing input devices and multimodal input approaches to promote natural and efficient geometry creation and manipulation using hands, speeches and gestures” (Ong, et al., 2011 pp. 652, 653).

For example, Chen et al. developed a 3D sketching system that is based on AR for product design and lets the user design 3D objects by sketching in 3D space without having to wear any devices. In the application, the user can draw wireframes using lines and curves as well as choosing color from a color-tipped wand and keyboard operations (Ong, et al., 2011 p. 653).

Another example is demonstrated by Haller et al, who have developed a collaborative tabletop environment that is designed for brainstorming and discussion meetings. They used an Anoto Digital Pen to draw sketches, manipulate menus and facilitate data transmission between the table and a wall or between other systems. This interactive table can be seen in Figure 5 (Ong, et al., 2011 p. 653).

Figure 5 - Interactive Table

Ong et al. developed a collaborative product design and development environment to help designers and experts design and review CAD models of physical products. Users can generate, modify, and combine solid models and can even include everyday objects by using marker-based AR and dataGloves. This procedure is described in Figure 6 (Ong, et al., 2011 p. 654):


Figure 6 Multiple-view AR based product development system

3.2 Material Processing

AR implementations are best suited to support processes in which a lot of human interaction is involved. That is why until recently few to none AR applications have been developed for material processing tasks. Ong argues that this is the case because these processing procedures are machine-centric and involve few human factors during this stage (Ong, et al., 2011 p. 656).

More recently, however, AR has been applied for training and process condition monitoring purposes. Ong states that by applying AR “users can accumulate knowledge and information when operating on real machines, and the switching of context with the use of computer-based and VR-based simulation systems can be avoided“ (Ong, et al., 2011 p. 656).

For example, Figure 7 shows a digital layer that is displayed during a - Computerized Numerical Control (CNC) machining simulation.

Figure 7 – CNC machining simulation system using AR

In another CNC machining system simulation, an ARToolkit marker was applied to track the movement of the cutter of a CNC wood cutter. The application then predicted collisions between the head stock and the workpiece (Ong, et al., 2011 p. 657).

3.3 Assembly Planning and Verification

While CAAP and CAD systems are mainly used to reduce assembly completion time and effort in the past, there have been some investigations about applying VR and AR based systems in this field for better considering human motions. According to Ong AR technologies save time and material that is needed to produce real assembly components to test the sequences as users can handle real objects directly and assemble virtual components with real parts. Furthermore product reuse and reverse engineering is simplified. To achieve this CAD models have to be reconstructed in real-time (Ong, et al., 2011 pp. 657, 658).

AR can even be used in training courses for assembly verification. For instance, Li et al. developed an AR based system in which participants use datagloves and inertial tracking to map the movement of the participant’s hand to a virtual assembly. The orientation of this virtual part is adjusted according to the inner-part constraints that are extracted from CAD models (Lie, et al., 2009).

In another project virtual parts are displayed according to the movement of fingertips. As a result the simulation of assembly operations can be achieved. This procedure is displayed in Figure 8 (Ong, et al., 2011 p. 658):

Figure 8 - Managing assembly operations with the movement of fingertips

3.4 Factory Facility Layout Planning

AR can be applied to parallel processing of layout planning processes. Here the technology promises to neglect shortcomings of VR-based systems as procedures become more realistic and user interaction can be implemented more easily. With using AR, newly proposed facility arrangement can be reviewed in real-time by using mobile platforms, by using computers with video cameras and inertial sensors, for example. Another approach to factory facility layout planning was used by Poh et al. who applied marker-based interaction methods to allocate physical constraints between mechanical components (Ong, et al., 2011 pp. 651, 652).

3.5 Maintenance Support

AR can be used for overlaying digital information for maintenance guidance. Thus, users can view instructions and do their work simultaneously. Furthermore AR can facilitate data logging and, as the view of the user can be reflected onto remote computer devices, remote collaboration and authoring can be achieved intuitively. In several scenarios users were guided by remote experts. This can be done in business scenarios and even, as mentioned by Ong, when a user needs guidance when repairing his or her car. This procedure is demonstrated in Figure 9. (Ong, et al., 2011 pp. 662, 663).

Figure 9 - Remote maintenance guidance using AR

4 Success factors for AR Business Models

The previous chapter showed that AR is evaluated and applied in a broad range of application scenarios in industry. While the importance of AR is already accepted in this field there has not been a real breakthrough for commercial applications in the mass consumer market (Kleef, et al., 2010 p. 1). The technology, however, seems to have great opportunity on the market, which is discussed this chapter.

This chapter first gives a short overview of AR implementations in the mass consumer market and then goes addresses AR business model propositions. Finally, the chapter sums up the results and final thoughts of a technology acceptance test carried out by Kleef et al.

4.1 Introduction to AR business models

Currently there is little competition in the field of AR applications. This might be changing once the technology is implemented in a way that a broad audience can benefit from it. Thus, Kleef et al. carried out a research to investigate success factors of augmented reality business models.

AR browsers are among the most successful commercial AR applications. They are mobile phone applications that use stream localized information to the device and display it above the camera image by taking into account both the current geographical location of the device and its orientation. The application Layar is one of the most popular AR browsers that are currently available. Figure 10 shows two views of this application both in the AR mode to the left and a representation on Google Maps to the right:

Webtermine

Figure 10 - Layer user interface in AR and Maps mode

Most AR browsers use the concept of layers, which gives Layar its name. With layers, a developer can structure related information and displays it above the camera image of the mobile phone. While this information used to be limited to text and images, several AR browsers recently added support to display and interact with 3D content (Kleef, et al., 2010 p. 1).

Kleef at al. believe that AR will be used in several commercial applications in the future including:

· Advanced location based store finders that let users brows the available products,

· Modeling of clothes by people at home to make internet purchasing of cloths easier, or

· Generating of 3D models of rooms and furniture so that people can better image their new home

A prerequisite for application development in general is to attract potential customers and to make profit. Therefore Kleef et al. assume that AR application success factors fall into three categories:

· Value creation and distribution of AR applications

· Acceptance of the technology by the user

· Technological factors such as the technical functionality of an AR application

Kleef et al. further point out that there are only a few business models that successfully exploit AR, which might be caused by hardware restrictions, costs of implementation and social acceptance (Kleef, et al., 2010 p. 3).

4.2 Success Factors of AR business models in detail

This chapter lists nine success factors of AR business models that provide a complete overview of the AR domain according to Kleef et al. They define these factors in due consideration to the Osterwalder business model ontology whose elements are outlined in Figure 11:

Figure 11 - Osterwalder Business Model Ontology

This chapter concludes by outlining some of these propositions.

4.2.1 Value Proposition

Value proposition includes bundles of services and products that a company offers which have value for its customers. Here, Lee et al. highlight 16 different types of value proposition that that can be implemented using AR (Kleef, et al., 2010).

In Situ

AR enables users to see products in the real environment before it is completed. This can be of benefit for furniture manufacturers when showing their products in the living rooms of potential customers. Figure 12 demonstrates the user interface of a web application that uses a camera to track users to display cloths over their bodies:

Figure 12 - AR overlaying cloths over the body of customers in Tobis‘ virtual dressing room

Utility

AR can be used to make things easier and can be used in any kind of application that displays information to the user. AR browser fit into this category.

Training

AR can be used in complex training and simulation scenarios to create situations that may be difficult to create in real life like training in surgery and bomb disposal.

Social Gaming

AR can provide games that are part of the real world. Thus, new and intuitive interaction capabilities can be implemented and social aspects can be more seamlessly as the game already takes place in the real world.

Location Layers

This category is similar to Utility, but focuses on location based information for travel guides.

Virtual demo

Users can see and manipulate products before they are available. This behavior is demonstrated in Figure 13 where image tracking is used to display an assembled Lego toy. In contrast to In Situ, this scenario does not show the product in its eventual location.

Figure 13 - Assembled lego toy is projected onto the product packaging

Experiential Education

With experiential education AR adds new experience to education. This can be done at an archaeological site where past processes or events are recreated and visualized.

Enhanced Classifieds

This type of application shows localized directories of offered products in the surrounding area.

3D Virals

This proposition deals with advertising in AR with some sort of interaction.

Personalized Shopping

Personalized Shopping is a special type of advertising in which shops can give customers advertisements on what to buy using AR. What is more, personal preferences of the user could be included.

Cooperation

AR can be used to improve remote cooperation with virtual services. One application could be for meetings to present external participants through AR.

Blended Branding

Virtual advertising space can be provided through AR.

Augmented Events

On augmented Events users could see relevant content, like how to buy the album of the band at a pop concert.

Intertainment

Kleef et al. define Intertainment as an enhancement of entertainment experience. This could be done by bringing TV and movies to the spectators through AR.

Understanding Systems

Users can have different views of an object or a system through AR. Thus it could be easier to increase the understanding of complex systems.

Recognizing and Targeting

This proposition deals with recognizing customers and knowing their backgrounds.

4.2.2 Key Partners

Key partners are organizations that partner with the company to create and deliver value to the customer. Technology providers and infrastructure providers are the major partners for companies that develop AR applications (Kleef, et al., 2010 p. 9).

Technology providers can provide tools to create value or some kind of middleware or hardware abstraction layer onto which the company builds its own application. AR browser like Layar and Junaio can be seen as such a middleware (Kleef, et al., 2010 p. 9).

Infrastructure providers help bringing the application to the customers. Internet Service Providers and companies related to mobile platforms like Google and Apple are among the most prominent providers for commercial AR applications (Kleef, et al., 2010 p. 9).

4.2.3 Key Activities

Key activities represent the processes that a company performs in order to create value to the customer. For AR the most important processes involve the creation of software. Kleef et al. further outline all key activities for AR application development, which can be seen in Figure 14. In fact, these key activities are common to all software development companies (Kleef, et al., 2010 p. 10).

Figure 14 - AR Application Development Key Activities

4.2.4 Key Resources

Key resources are all resources that processes and activities of a company need to perform and to create value for their customers. Kleef et al. identify technology and staff as the key resources AR application development and further subdivide them, as can be seen in Figure 15 (Kleef, et al., 2010 p. 11).

Figure 15 - Key resources for AR applications

4.2.5 Distribution Channels

Companies get in touch with their customers through distribution channels. The internet as well as application publisher and broker, like the Android Market, are the most prominent distribution channels for software development companies. As most available AR applications are targeting mobile and handheld devices, application broker are best suited as distribution channels. According to Kleef et al. the value propositions that are most likely to be delivered over this channel are in situ, location layers, enhanced classifieds, 3D virals, personalized shopping, blended branding and augmented events (Kleef, et al., 2010 pp. 11,12).

4.2.6 Customer Segments

According to Kleef et al. customer segments are “the types of customers the company wants to deliver value to”. They further list potential customers of all value propositions listed in chapter 4.2.1. This list can be seen in Table 1.

Value Proposition	Customer Segments
In Situ	Manufacturers, Architects
Utility	Smartphone users
Training	Students, Teachers
Social Gaming	Gamers
Location Layers	Smartphone users
Virtual Demo	Shop customers, shopkeepers
Experiential education	Museums, zoos, educational institutions
Enhanced classifieds	Smartphone users
3D Virals	Smartphone users
Personalized shopping	Smartphone users, shopkeepers
Cooperation	Business professionals
Blended branding	Smartphone users
Augmented events	Smartphone users
Intertainment	Home entertainment users
Understanding	Students, teachers, mechanics
Recognition and targeting	Marketers

Table 1 - Potential customer segments of value propositions

4.3 Survey on user acceptance of AR applications

Kleef et al. carried out a survey on user acceptance of AR applications. The target group consisted of people who might use AR for private use, entertaining, gaming, or personal assistance. Professional use of AR, however, was not investigated (Kleef, et al., 2010 p. 14).

Two AR applications were tested during the user acceptance survey. The first application was the Layar mobile AR browser that can be seen in Figure 10. The second application was the Tobi online virtual dressing room displayed in Figure 12.

According to Kleef et al. the results of the survey showed that important factors for AR applications are related to usability, usefulness, fun and productivity. They further argue that while hardly any AR business models are established at the time of writing, it is quite certain that AR applications will see more success in the near future (Kleef, et al., 2010 p. 30).

When it comes to success factors, latency, noise and orientation errors on tracking distant objects in AR applications matters most. Furthermore, contrast sensitivity and color perception of the AR displays also play a role as well as object recognition and the recognition of the boundaries between the physical world and digital objects and, when using marker tracking in AR applications, the environment needs to be prepared as well (Kleef, et al., 2010 p. 25).

5Literature

Carmigniani, Julie and Furht, Borko. 2011. Augmented Reality: An Overview. [book auth.] Borko Furht. Handbook of Augmented Reality. 2011.

Furth, Borko. 2011. Handbook of Augmented Reality. Florida Atlantic University : Springer Science + Business Media, 2011. 978-1-4614-0063-9.

Kleef, N., Noltes, J. and Spoel, S. J. van der. 2010. Success Factors for Augmented Reality Business Models. [Dokument] s.l. : Pixel, 2010.

Lie, S., et al. 2009. A MixedReality-Based Assembly Veri?cation and Training Platform. HCI International 2009 : s.n., 2009.

Ong, S.K., et al. 2011. Handbook of Augmented Reality. [book auth.] Borko Furth. Augmented Reality in Product Development and Manufacturing. 2011.

Homepage of Andreas Korat

My Ideas, My Work, My Life

Augmented Reality

Paper about the use of Augmented Reality in Manufacturing and Product Development

List of tables

List of figures

List of abbreviations