Implementation of automobile assembly system using AR technology.

Park, H.S. ; Choi, H.W. ; Park, J.W. 等

1. Introduction

Nowadays, the global market requires the variety and shorter life-cycle of products to fulfill the diverse demands of customers. To survive in the turbulent and competitive market, manufacturing companies must design and implement the manufacturing systems which respond rapidly to the market demands. For these reasons, VR(Virtual Reality) based digital manufacturing technologies are used to analyze the static and dynamic behavior of system at all stages for configuration of manufacturing system(Gunter et al., 2005; Mani et al., 1996; Park et al., 2006; Wolfgang, 2006; Zhu et al., 2007). However, most of methods and softwares for digital manufacturing require the perfect 3D models of the whole system in virtual environment to represent the target system and surrounding environment. That means this modeling work requires a lot of expenses and effort.

Some manufacturing companies try to introduce the AR technology which can remarkably reduce modeling work because AR technology uses the real manufacturing environment to design and plan manufacturing systems. It is also possible to obtain the information in real time from manufacturing field by superimposition of virtual objects to real scene(Ong et al., 2004; Wolfgang, 2004). Consequently, the application of AR technology is expected as an epochal method for implementing manufacturing systems in an efficient and user friendly manner.

Recently, the implementation of a manufacturing system for assembling new automobile body is done based on the prototype manufactured after the design of a new body model by changing a conventional system or newly designing. For this purpose, a system planner stops a current system and evaluates the manufacturability of a new model in that system as well as executes the work for changing it by putting a new model instead of the conventional automobile body to the current system. This traditional method leads to the critical problems such as reduction of system productivity and delay of the introduction of new product into market.

To remove these problems, the AR technology is applied in this paper. The cockpit assembly station is chosen as a research object because this process requires high accuracy in automobile assembly process. So, another station can be easily realized by the knowledge and experience acquired from the accomplishment of the cockpit assembly process.

For the development of an AR based assembly system, the architecture of AR browser is firstly introduced. Then, the optimal values of environment parameter for robust superimposition between virtual objects and real scene are proposed through lots of experiments. Through an initial test, the problems are derived for the application of AR system. For an implement of AR system through solving them, the methods for appropriate allocation of camera, avoidance of collision between virtual objects and marker structural configuration for recognizing in multi directions in consideration of robot behaviors are proposed. With these results and experiments, an operation program for cockpit assembly system is generated.

2. Augmented Reality system

2.1 Architecture of AR browser

AR browser as the core component for the interaction and visualization in an augmented environment consists of 3 modules for digital image processing and interfaces for communicating between devices(Fig. 1). The modules of AR browser are classified to tracking module to track positions of target markers, rendering module for generation and handling of virtual objects and measurement module to calculate the distance between objects.

For tracking target markers, lots of methods such as mechanic, magnetic and optical, are examined. For AR browser, optical method is widespread due to high precision. AR browser carries out digital image processing to recognize the position of marker using images obtained from the continuous marker tracking. The basic coordinate system for positioning virtual objects is established through the matching procedure between the information of marker DB and input data of UI(User Interface). The interior pattern of square markers which defininig the marker direction is configured by using the CyberCode type(Jun et al., 2000).

Rendering module executes functions to generate, remove and transform virtual objects. To get a virtual object in real scene, the relationship between marker and virtual object is defined as the input information through UI. Also, the coordinate transform of virtual object is executed by using 9 input data(3 translations, 3 rotations and 3 scales). Measurement module as auxiliary tool is used to measure the distance between two points of virtual objects, the distance between coordinate systems based on markers and to detect the collision among virtual objects. These modules are implemented by the C++ library to execute the mathematical function and algorithm for digital image processing and combined in the ActiveX-Control which is the standard for integration of visual software.

[FIGURE 1 OMITTED]

2.2 Camera calibration

AR system applies the camera to obtain the image of real environment. Most of cameras except the pin-hole type have distortion problems of an obtained image because of irregular manufacturing of lens(Roger, 1987). Therefore, to get the exact 3D image information, the camera must be calibrated before use. Generally, the main reasons of camera distortion come from radial distortion and decentering distortion. Radial distortion is caused by the curvature of a lens's surface. The equation for calibration of it is expressed by an odd-ordered polynomial series(Gruen et al., 2001). At the industrial application for machine vision system, it is known as reasonable to consider up to the second term of the equation. The another one, decentering distortion, is appeared by the misalignment of the optical axis and the center line of lens. That can be compensated by the displacement along x- and y direction of the optical axis.

All cameras used in this research are calibrated to compensate radial and decentering distortion(Fig. 2). For calibration of camera, the chessboard (18X12) was used as norm. Based on the information of the image acquired from camera, the distortion parameters were calculated with the above introduced algorithms(Tsai, 1987; Zhang, 2000). These experiments were carried out more than twenty times because they sensibly responded to the changes of environmental conditions. With the average of the experimented values except extraordinary things, the calibration of camera was done.

[FIGURE 2 OMITTED]

3. Environment variables for robust superimposition of virtual objects

AR system establishes the coordinate system for positioning virtual objects using 4 apexes of a square marker image obtained from the camera. While AR system internally executes the image processes, the positions of the four apexes of a square mark displayed in AR browser have errors in amount of 2 or 5 pixels according to the environmental conditions of camera. A rapidly ever-changing environment conditions change the origin of coordinate system. These lead to chatter of virtual object. In order to apply AR system to practice, it is important to find and maintain the condition to remove this chatter. To determine the appropriate conditions for an application of AR system, a lot of tests were carried out with the environmental variables such as the angle and distance between camera and marker, the intensity of light, the size of marker and the relationship between the light intensity and the marker size(Fig. 3).

[FIGURE 3 OMITTED]

3.1 Influence of the between marker and camera

To grasp how the angle between marker and camera influence the chatter of virtual objects, the relationship between the two angles [[theta].sub.C], [[theta].sub.M] and the chatter was examined. [[theta].sub.C] is the angle between the optical axis of camera and the line connected from principal point of lens to marker center and [[theta].sub.M] is the angle between the normal direction of marker plane and the line connected from principal point of lens to marker center.

In case that the marker is located in center of image obtained from camera(-15[degrees][less than or equal to][[theta].sub.C][less than or equal to]15[degrees]), virtual objects is generated robustly where the relative angle between marker and camera ([[theta].sub.C] + [[theta].sub.M]) is -60[degrees]~-10[degrees] or 10[degrees]~60[degrees]. Because marker pattern is not recognized clearly in the regions of -90[degrees]~-70[degrees] or 70[degrees]~90[degrees], virtual objects are not generated or chattered badly. Also, if the camera optical axis is perpendicular to the marker direction, virtual objects are chattered because of problems caused by formula for digital image processing. However, we can not recognize clear and perfect marker pattern in the regions of [[theta].sub.C]< -15[degrees] or [[theta].sub.C] > 15[degrees] because of the auto-focusing function of camera.

3.2 Influence of light intensity

The adequate light condition is required to recognize the marker effectively during digital image processing for AR system. However, the light condition of most fields is not enough to apply the AR system. In order to investigate the influence of light condition, tests are practiced with variables such as intensity of light and angle between marker and light([[theta].sub.L]). [[theta].sub.L] is changed from the left side of marker to the right side with an interval of 10[degrees]. And, the intensity of light is changed from 400LUX to 1800LUX. As the initial condition of tests, the angle between marker and camera is continued as 45[degrees] to avoid the influence of angle between marker and camera.

As the results of tests, an incidence angle is more important than the light intensity to obtain the robust condition. The incident light from the region of 120[degrees]~180[degrees] is reflected from marker which is rotated 45[degrees] based on optical axis. Then, the marker recognition is difficult because of the reflected light. Therefore, virtual objects are not generated or chattered seriously. consequently, in order to maintain the robustness of virtual objects, we must keep the light position to avoid the interference and reflected light.

3.3 Influence of the size of marker and distance between marker and camera

Because the robustness of virtual objects is influenced the quality of recognized marker, marker size represented AR browser is an important factor for robust superimposition. Generally, if the marker size is getting smaller and smaller, it is difficult to recognize the marker pattern. The marker size represented AR system is determined according to the real marker size and the distance between the marker and the camera(D). To propose the adequate marker size for robust superimposition, tests are carried out. The real marker sizes for tests are 50mm, 150mm and 300mm and we change the distance between marker and camera or use the zoom function of camera to change the marker size represented AR system.

As the results, the robust superimposition of virtual object is assured when the big marker and the camera moving method are used. In case of cameras used in test, we obtain the robust image when the marker size represented AR system is 35~40mm.

4. AR based cockpit module assembly system

4.1 Problems to be solved at implement of AR based assembly system and their solutions

To apply AR system to practice, there are some problems to be solved besides the environmental conditions for robust superimposition. For grasping them, a test bed of the H company's assembly station for cockpit module was realized with twelve times reduction model. Through lots of experiments with it, the problems occurring at implementing a AR based assembly system were examined(Fig. 4).

[FIGURE 4 OMITTED]

Problem 1 & 2: Finding the correct assembly positions

It is almost impossible for virtual object to reach the exact assembly location because the image acquired from one camera deliver only 2D information. So, it is necessary to get more than two image information obtained from different places. In case of the cockpit module assembly as the research object, two cameras are required in any case because this process should be done in two different directions such as assembly and insert direction. These two directions are perpendicular as shown in figure 4. Therefore, it is impossible to carry out this assembly process with one camera due to not covering two directions simultaneously. one camera is located for side view to observe insert process and the other to the front of automobile body to observe assembly process. With help of these two cameras, the system planner can generate operation program of the assembly, i.e. robot program.

Problem 3: Covering an assembly area by another virtual object

AR system can carry out as assembly process only with virtual objects. Because of that, it is possible to generate an operation program for assembling an object to the most seen assembly location of real object. In case of the assembly location covered by virtual object, the operation of assembly process cannot be programmed because a programmer dose not see the assembly location of virtual object(Fig. 5).

For viewing an assembly area, the virtual object was partitioned to several parts. For example, an automobile body is divided to side panel, bracket and the rest part and a cockpit module to a skin part and cockpit cross bar. Through that, the divided parts can be appeared or disappeared for supporting an execution of assembly process. To increase the adjusting degree of two objects, the alignment line was modeled to guide the correct approach of one object to his counter part.

[FIGURE 5 OMITTED]

Problem 4: Collision detection between two virtual objects

The camera located to the side of automobile body as shown in the above part of Fig. 6 is used to check the collision between cockpit and body during cockpit insert process. Due to the optical effect of camera, an object is seen to be small at the beginning of inserting processes. Along the proceeding with the process, a object is getting larger and larger. So, it is difficult at the end of inserting process whether a collision between objects happen or not.

To solve this problem, the clipping plane shown in the above part of figure 6 was generated to express the only information of the sectional surface for showing the possibility of collision.

[FIGURE 6 OMITTED]

Problem 5 & 6: Structural configuration of markers and their light intensity

The main virtual objects used for a cockpit module assembly system are an automobile body and a cockpit. So, the two coordinate systems consisted of markers for positioning the virtual objects are at least required. The marker for establishing the coordinate system for automobile body was located to the fixed position out of automobile body in consideration of the locations of the cameras. By this marker system, the virtual automobile body can be located to a transporter(Fig. 7).

The another marker system mounted on the robot gripper for generating a virtual cockpit was manufactured in the structure of a box type, because the system should be seen in every direction due to free movement of robot. The light intensity, i.e. the brightness, should be adjusted to an environmental condition. In case of entering a robot gripper into the inside of automobile body, the light condition is so dark that a camera cannot catch the marker system. This problem was solved by setting up a light inside of the box type marker system. In Fig. 7, the coordinate systems were also presented for the superimposition of automobile body and cockpit to the correct positions of real scene.

[FIGURE 7 OMITTED]

4.2 Realization of a test bed for generating an operation program for new model

For generating an operation program for assembling a new cockpit to a new body for a next model of automobile in a conventional assembly station by using AR technology, the implemented test bed is presented in Fig. 8.

The test bed was constructed in the twelve times reduced model of the conventional station for assembling a cockpit module. Base on the test bed, the camera and the marker systems were optimally allocated according to the results of the previous studies for applying AR technology to practical. With this test bed, the operation of the station for assembling a new cockpit of the next automobile model was planned and programmed by AR technology. The completeness of the generated operation program was proven by applying it to the conventional assembly station. The boundary conditions such as the position tolerance range within 10 mm were fulfilled.

[FIGURE 8 OMITTED]

5. Conclusion

AR in manufacturing system applications faces several technical challenges. This paper outlined some of the major working areas and highlighted new approaches and developed solutions.

One of major problems was the robust dynamic alignment of virtual objects with real view of the user. This was achieved by examining the environmental parameter. The partition of virtual object was done to correctly carry out assembly process by removing the covering area of assembly location. In addition, to detect a collision between objects, the section surface was putted into an operational view. The structural configuration and patterns of markers were designed so that they could be recognized in every direction independent of movement of robot.

These systematical solving approaches overcome the problems against the implement of an AR-based assembly system. With the experimental results of the developed system through the test bed, it is clear that AR system can help the system planer to analyze the actual assembly process in the early stages of the manufacturing process planning and save valuable costs and time for testing the real process with the prototype.

DOI: 10.2507/daaam.scibook.2009.57

6. References

Gruen, A. & Huang, T. S. (2001) Calibration and Orientation of Cameras in Computer Vision, Springer-VerlagBerlin Heidlberg

Gunter, W. & Emmerich, S. (2005). Digital Planning Validation in Automotive Industry, Computers in Industry, Vol. 56, pp. 393-405

Jun, R. & Yuji, A. (2000). CyberCode: Designing Augmented Reality Environments with Visual Tags, Designing Augmented Reality Environments, pp. 1-10.

Mani, J. & Raman, S. (1996). A methodology for Manufacturing Precedence Representation and Alternate Task Sequencing in CAPP, Transaction of NAMRI/SME, Vol. 24, pp. 251-256

Ong, S. K. & Nee, A. Y. C. (2004). Virtual and Augmented Reality Applications in Manufacturing, Springer-Verlag London Limited

Park, H. S. & Lee, G. B. (2007). Development of Digital Laser Welding System for Automobile Side Panels, International Journal of Automotive Technology, Vol. 8, No. 1, pp. 83-91

Roger, Y. T. (1987). A Versatile Camera Calibration Technique for High-Accuracy 3D Machine Vision Metrology Using Off-the-Shelf TV Cameras and Lenses, IEEE Journal of Robotics and Automation, Vol. RA-3, No. 6, pp. 323-344

Tasi, R. Y. (1987). A Versatile Camera Calibration Technique for High-Accuracy 3D Machine Vision Metrology Using Off-the-Shelf TV Cameras and Lens, IEEE Journal of Robotics and Automation, Vol. RA-3, No. 4, pp. 323-344

Wolfgang, F. (2004). ARVIKA: Augmented Reality fur Entwicklung, Produktion und Service, Publicis Corporate Publishing

Wolfgang, K. (2006). Digital Factory--Integration of Simulation Enhance the Product and Production Process toward operative Control and optimization, International Journal of Simulation, Vol. 7, No. 7, pp. 27-39

Zhang, Z. (2000). A Flexible New Technique for Camera Calibration, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 22, No. 11, pp. 1330-1334

Zhu, W. & Lee, Y. S. (2007). An Infrastructure Toward Haptic Virtual Assembly with Native 3D Model in Mainstream CAD system, Transaction of NAMRI/SME, Vol. 35, pp. 761-768

This Publication has to be referred as: Park, H[ong] S[eok]; Choi, H[ung] W[on] & Park, J[in] W[oo] (2009). Implementation of Automobile Assembly System Using AR Technology, Chapter 57 in DAAAM International Scientific Book 2009, pp. 587598, B. Katalinic (Ed.), Published by DAAAM International, ISBN 978-3-90150969-8, ISSN 1726-9687, Vienna, Austria

Authors' data: Prof. Dr.-Ing. Park, H[ong] S[eok]; Ph. D. Choi, H[ung] W[on]; Park, J[in] W[oo], University of Ulsan, Daehak-ro 102, Nam-gu, Ulsan, SouthKorea, [email protected], [email protected], [email protected]