A comprehensive survey of augmented reality assembly research

doi:10.1007/s40436-015-0131-4

Abstract

Abstract: In the past two decades, augmented reality(AR) has received a growing amount of attention by researchers in the manufacturing technology community, because AR can be applied to address a wide range of problems throughout the assembly phase in the lifecycle of a product, e.g., planning, design, ergonomics assessment, operation guidance and training. However, to the best of authors' knowledge, there has not been any comprehensive review of AR-based assembly systems. This paper aims to provide a concise overview of the technical features, characteristics and broad range of applications of ARbased assembly systems published between 1990 and 2015. Among these selected articles, two thirds of them were published between 2005 and 2015, and they are considered as recent pertinent works which will be discussed in detail. In addition, the current limitation factors and future trends in the development will also be discussed.

Key words: Augmented reality (AR), Assembly process, Assembly guidance, Assembly systems

X. Wang, S. K. Ong, A. Y. C. Nee. A comprehensive survey of augmented reality assembly research[J]. Advances in Manufacturing, 2016, 4(1): 1-22.

TrendMD

References

1. Whitney DE (2004) Mechanical assemblies: their design, manufacture, and role in product development. Oxford University Press, Oxford

2. Zhou F, Duh H-L, Billinghurst M (2008) Trends in augmented reality tracking, interaction and display: a review of ten years of ISMAR. Proceedings of the 7th IEEE international symposium on mixed and augmented reality (ISMAR 2008), Cambridge, pp 193-202

3. Sielhorst T, Feuerstein M, Navab N (2008) Advanced medical displays: a literature review of augmented reality. J Disp Technol 4(4):451-467

4. Henderson SJ, Feiner SK (2011) Exploring the benefits of augmented reality documentation for maintenance and repair. IEEE Trans Vis Comput Graph 17(10):1355-1368

5. Ridel B, Reuter P, Laviole J et al (2014) The revealing flashlight: interactive spatial augmented reality for detail exploration of cultural heritage artifacts. J Comput Cult Herit 7(2):1-18

6. Bower M, Howe C, McCredie N et al (2014) Augmented reality in education-cases, places and potentials. Educ Media Int 51(1):1-15

7. Google Glass. http://www.google.com/glass/start/. Accessed 10 July 2015)

8. Nee AYC, Ong SK, Chryssolouris G et al (2012) Augmented reality applications in design and manufacturing. CIRP Annal Manuf Technol 61(2):657-679

9. Ong SK, Yuan ML, Nee AYC (2008) Augmented reality applications in manufacturing: a survey. Int J Prod Res 46(10):2707-2742

10. Fite-Georgel P (2011) Is there a reality in industrial augmented reality? In: Proceedings of the 10th IEEE international symposium on mixed and augmented reality (ISMAR 2011), Basel, pp 201-210

11. Leu MC, ElMaraghy HA, Nee AYC et al (2013) CAD model based virtual assembly simulation, planning and training. CIRP Annal Manuf Technol 62(2):799-822

12. Boothroyd G, Dewhurst P, Knight WA (2011) Product design for manufacture and assembly. CRC Press, Boca Raton

13. Seth A, Vance JM, Oliver JH (2011) Virtual reality for assembly methods prototyping: a review. Virtual Reality 15(1):5-20

14. Cruz-Neira C, Sandin DJ, DeFanti TA et al (1992) The CAVE: audio visual experience automatic virtual environment. Commun ACM 35(6):64-72

15. Cruz-Neira C, Sandin DJ, DeFanti TA (1993) Surround-screen projection-based virtual reality: the design and implementation of the CAVE. Proceedings of the 20th annual conference on computer graphics and interactive techniques, Anaheim, California, pp 135-142

16. Kuehne R, Oliver J (1995) A virtual environment for interactive assembly planning and evaluation. Proceedings of ASME design automation conference

17. Pere E, Langrana N, Gomez D et al (1996) Virtual mechanical assembly on a PC-based system. Proceedings of ASME design engineering technical conferences and computers and information in engineering conference, Irvine, pp 18-22

18. Jayaram S, Connacher HI, Lyons KW (1997) Virtual assembly using virtual reality techniques. Computer Aided Design 29(8):575-584

19. Jayaram S, Jayaram U, Wang Y et al (1999) VADE: a virtual assembly design environment. Comput Graph Appl 19(6):44-50

20. Jayaram S, Jayaram U, Wang Y et al (2000) CORBA-based collaboration in a virtual assembly design environment. Proceedings of ASME design engineering technical conferences and computers and information in engineering conference

21. Jayaram U, Tirumali H, Jayaram S (2000) A tool/part/human interaction model for assembly in virtual environments. Proceedings of ASME design engineering technical conferences

22. Taylor F, Jayaram S, Jayaram U (2000) Functionality to facilitate assembly of heavy machines in a virtual environment. Proceedings of ASME design engineering technical conferences

23. Coutee AS, McDermott SD, Bras B (2001) A haptic assembly and disassembly simulation environment and associated computational load optimization techniques. J Comput Inform Sci Eng 1(2):113-122

24. Coutee AS, Bras B (2002) Collision detection for virtual objects in a haptic assembly and disassembly simulation environment. ASME design engineering technical conferences and computers and information in engineering conference, Montreal, pp 11-20

25. Seth A, Su HJ, Vance JM (2005) A desktop networked haptic VR interface for mechanical assembly. ASME 2005 International mechanical engineering congress & exposition, Orlando, pp 173-180

26. Seth A, Su HJ, Vance JM (2006) SHARP: a system for haptic assembly & realistic prototyping. ASME 2006 international design engineering technical conferences and computers and information in engineering conference in Philadelphia, Pennsylvania, pp 905-912

27. Graphics Processing Unit (GPU). http://www.nvidia.com. Accessed 10 July 2015

28. Caudell TP, Mizell DW (1992) Augmented reality: an application of heads-up display technology to manual manufacturing processes. Proceedings of the 25th Hawaii international conference on system sciences, Hawaii, pp 659-669

29. Sims D (1994) New realities in aircraft design and manufacture. Comput Graph Appl 14(2):91

30. Mizell D (2001) Boeing's wire bundle assembly project. In: Barfield W, Caudell T (eds) Fundamentals of wearable computers and augmented reality. CRC Press, Mahwah, pp 447-467

31. Feiner S, Macintyre B, Seligmann D (1993) Knowledge-based augmented reality. Commun ACM 36(7):53-62

32. Webster A, Feiner S, Maclntyre B et al (1996) Augmented reality in architectural construction, inspection and renovation. In: Proceedings of ASCE 3rd congress on computing in civil engineering, Anaheim, pp 913-919

33. Curtis D, Mizell D, Gruenbaum P et al (1998) Several devils in the details: making an AR application work in the airplane factory. In: Proceedings of the International Workshop Augmented Reality in San Francisco, California, pp 47-60

34. Schwald B, Figue J, Chauvineau E et al (2001) STARMATE: using augmented reality technology for computer guided maintenance of complex mechanical elements. In: Stanford-Smith B, Chiozza E (eds) E-work and e-commerce-novel solutions and practices for a global networked economy. IOS Press, Amsterdam, pp 196-202

35. Syberfeldt A, Danielsson O, Holm M et al (2015) Visual assembling guidance using augmented reality. In: International manufacturing research conference 2015, Charlotte, pp 1-12

36. Molineros J, Sharma R (2001) Computer vision for guiding manual assembly. In: Proceedings of the 2001 IEEE international symposium on assembly and task planning (ISATP 2001), Fukuoka, pp 362-368

37. Schwald B, Laval BD (2003) An augmented reality system for training and assistance to maintenance in the industrial context. In: International conference in central Europe on computer graphics, visualization and computer vision (WSCG), Plzen, pp 425-432

38. Regenbrecht H, Baratoff G, Wilke W (2005) Augmented reality projects in the automotive and aerospace industries. IEEE Comput Graph Appl 25(6):48-56

39. Saaski J, Salonen T, Hakkarainen M et al (2008) Integration of design and assembly using augmented reality. In: Ratchev S, Koelemeijer S (eds) Micro-assembly technologies and applications. Springer, London, pp 395-404

40. Salonen T, Saaski J (2008) Dynamic and visual assembly instruction for configurable products using augmented reality techniques. In: Yan XT, Jiang C, Eynard B (eds) Advanced design and manufacture to gain a competitive edge. Springer, London, pp 23-32

41. Salonen T, Saaski J, Hakkarainen M et al (2007) Demonstration of assembly work using augmented reality. In: Proceedings of the 6th ACM international conference on image and video retrieval (CIVR 2007), Amsterdam, pp 120-123

42. Salonen T, Saaski J, Woodward C (2009) Data pipeline from CAD to AR based assembly instructions. In: Proceedings of the ASME-AFM 2009 world conference of innovative virtual reality (WINVR09), Mediapole, pp 165-168

43. Tumkor S, Aziz ES, Esche SK et al (2013) Integration of augmented reality into the CAD process. In: Proceedings of the ASEE annual conference and exposition, Atlanta

44. Makris S, Pintzos G, Rentzos L et al (2013) Assembly support using AR technology based on automatic sequence generation. CIRP Annal Manuf Technol 62(1):9-12

45. Sanna A, Manuri F, Lamberti F et al (2015) Using handheld devices to support augmented reality based maintenance and assembly tasks. In: 2015 IEEE international conference on consumer electronics (ICCE), Berlin, pp 178-179

46. Kollatsch C, Schumann M, Klimant P et al (2014) Mobile augmented reality based monitoring of assembly lines. Procedia CIRP 23:246-251

47. Yuan ML, Ong SK, Nee AYC (2008) Augmented reality for assembly guidance using a virtual interactive tool. Int J Prod Res 46(7):1745-1767

48. Sukan M, Elvezio C, Oda O et al (2014) ParaFrustum: visualization techniques for guiding a user to a constrained set of viewing positions and orientations. In: Proceedings of the 27th annual ACM symposium on user interface software and technology, Honolulu, pp 331-340

49. Posada J, Toro C, Barandiaran I et al (2015) Visual computing as a key enabling technology for industry 4.0 and industrial internet. IEEE Comput Graph Appl 35(2):26-40

50. Zhang J, Ong SK, Nee AYC (2011) RFID-assisted assembly guidance system in an augmented reality environment. Int J Prod Res 49(13):3919-3938

51. Henderson SJ, Feiner SK (2011) Augmented reality in the psychomotor phase of a procedural task. In: Proceedings of the 10th IEEE international symposium on mixed and augmented reality (ISMAR 2011), Basel, pp 191-200

52. Neumann U, Majoros A (1998) Cognitive, performance, and systems issues for augmented reality applications in manufacturing and maintenance. In: Proceedings of IEEE virtual reality (VR 1998), San Francisco, pp 4-11

53. Andersen M, Andersen R, Larsen C et al (2009) Interactive assembly guide using augmented reality. In: Proceedings of the 5th international symposium on advances in visual computing: Part I, Las Vegas, pp 999-1008

54. Khuong BM, Kiyokawa K, Miller A et al (2014) The effectiveness of an AR-based context-aware assembly support system in object assembly. In: Proceedings of IEEE virtual reality, Minneapolis, pp 57-62

55. Friedrich W (2002) ARVIKA augmented reality for development, production, and service. In: Proceedings of the 1st IEEE international symposium on mixed and augmented reality (ISMAR 2002), Darmstadt, pp 3-4

56. Rentzos L, Papanastasiou S, Papakostas N et al (2013) Augmented reality for human-based assembly: using product and process semantics. In: Analysis, design and evaluation of human-machine systems, Las Vegas, pp 98-101

57. Zhu Z, Branzoi V, Wolverton M et al (2014) AR-Mentor: augmented reality based mentoring system. In: Proceedings of the 13rd IEEE international symposium on mixed and augmented reality (ISMAR 2014), Munich, pp 17-22

58. Radkowski R, Oliver J (2013) Natural feature tracking augmented reality for on-site assembly assistance systems. In: Shumaker R (ed) Virtual, augmented and mixed reality, systems and applications. Springer, Berlin, pp 281-290

59. Lowe DG (1999) Object recognition from local scale-invariant features. In: Proceedings of the 1999 international conference on computer vision, Kerkyra, pp 1150-1157

60. Chen CJ, Hong J, Wang SF (2015) Automated positioning of 3D virtual scene in AR-based assembly and disassembly guiding system. Int J Adv Manuf Technol 76(5-8):753-764

61. Damen D, Gee A, Mayol-Cuevas W et al (2012) Egocentric real-time workspace monitoring using a rgb-d camera. In: 2012 IEEE/RSJ international conference on intelligent robots and systems (IROS), Vilamoura, pp 1029-1036

62. Vignais N, Miezal M, Bleser G et al (2013) Innovative system for real-time ergonomic feedback in industrial manufacturing. Appl Ergon 44(4):566-574

63. Haringer M, Regenbrecht H (2002) A pragmatic approach to augmented reality authoring. In: Proceedings of IEEE 1st international symposium on mixed and augmented reality (ISMAR 2002), Darmstadt, pp 237-245

64. Zauner J, Haller M, Brandl A et al (2003) Authoring of a mixed reality assembly instructor for hierarchical structures. In: Proceedings of the 2nd IEEE international symposium on mixed and augmented reality (ISMAR 2003), Tokyo, pp 237-246

65. Servan J, Mas F, Menendez JL et al (2012) Using augmented reality in AIRBUS A400M shop floor assembly work instructions. In: The 4th manufacturing engineering society international conference, Cadiz, pp 633-640

66. Petersen N, Stricker D (2012) Learning task structure from video examples for workflow tracking and authoring. In: Proceedings of the 11th IEEE international symposium on mixed and augmented reality (ISMAR 2012), Atlanta, pp 237-246

67. Mura K, Petersen N, Huff M et al (2013) IBES: a tool for creating instructions based on event segmentation. Front Psychol 4:1-14

68. Petersen N, Pagani A, Stricker D (2013) Real-time modeling and tracking manual workflows from first-person vision. In: Proceedings of the 12th IEEE international symposium on mixed and augmented reality (ISMAR 2013), Adelaide, pp 117-124

69. Bhattacharva B, Winer E (2015) A method for real-time generation of augmented reality work instructions via expert movements. IS&T/SPIE Electronic Imaging: 93920G-1-93920G-13

70. Mohr P, Kerbl B, Donoser M et al (2015) Retargeting technical documentation to augmented reality. In: Proceedings of the 33rd annual ACM conference on human factors in computing systems, Seoul, pp 3337-3346

71. Hou L, Wang X (2013) A study on the benefits of augmented reality in retaining working memory in assembly tasks: a focus on differences in gender. Autom Constr 32:38-45

72. Odenthal B, Mayer MP, Kabub W et al (2014) A comparative study of head-mounted and table-mounted augmented vision systems for assembly error detection. Hum Fact Ergon Manuf Serv Ind 24(1):105-123

73. Hou L, Wang X, Truijens M (2013) Using augmented reality to facilitate piping assembly: an experiment-based evaluation. J Comput Civil Eng 29(1):05014007

74. Baird KM, Barfield W (1999) Evaluating the effectiveness of augmented reality displays for a manual assembly task. Virtual Reality 4(4):250-259

75. Tang A, Owen C, Biocca F et al (2003) Comparative effectiveness of augmented reality in object assembly. In: Proceedings of the SIGCHI conference on Human factors in computing systems (CHI 03), Fort Lauderdale, pp 73-80

76. Wiedenmaier S, Oehme O, Schmidt L et al (2003) Augmented reality (AR) for assembly processes design and experimental evaluation. Int J Hum Comput Interact 16(3):497-514

77. Odenthal B, Mayer MP, Kabub W et al (2011) An empirical study of disassembling using an augmented vision system. In: Proceedings of the 3rd international conference on digital human modeling (ICDHM'11), Orlando, pp 399-408

78. Boud AC, Haniff DJ, Baber C et al (1999) Virtual reality and augmented reality as a training tool for assembly tasks. In: Proceedings of 1999 IEEE international conference on information visualization, London, pp 32-36

79. Rios H, Hincapie M, Caponio A et al (2011) Augmented reality: an advantageous option for complex training and maintenance operations in aeronautic related processes. In: Proceedings of 2011 international conference on virtual and mixed reality—Part 1, Orlando, pp 87-96

80. Suarez-Warden F, Cervantes-Gloria Y, Gonzalez-Mendivil E (2011) Sample size estimation for statistical comparative test of training by using augmented reality via theoretical formula and OCC graphs: aeronautical case of a component assemblage. In: Proceedings of 2011 international conference on virtual and mixed reality-systems and applications—Part II, Orlando, pp 80-89

81. Gavish N, Gutierrez T, Webel S et al (2013) Evaluating virtual reality and augmented reality training for industrial maintenance and assembly tasks. Interact Learn Environ. doi:10.1080/10494820.2013.815221

82. Radkowski R, Herrema J, Oliver J (2015) Augmented realitybased manual assembly support with visual features for different degrees of difficulty. Int J Hum Comput Interact 31(5):337-349

83. Henderson SJ, Feiner SK (2009) Evaluating the benefits of augmented reality for task localization in maintenance of an armored personnel carrier turret. In: Proceedings of the 8th IEEE international symposium on mixed and augmented reality (ISMAR 2009), Orlando, pp 135-144

84. Gattullo M, Uva AE, Fiorentino M et al (2015) Legibility in industrial AR: text style, color coding, and illuminance. IEEE Comput Graph Appl 35(2):52-61

85. Hou L, Wang X (2011) Experimental framework for evaluating cognitive workload of using AR system for general assembly task. In: Proceedings of the 28th international symposium on automation and robotics in construction, Seoul, pp 625-630

[1]	S. K. Ong, X. Wang, A. Y. C. Nee. 3D bare-hand interactions enabling ubiquitous interactions with smart objects [J]. Advances in Manufacturing, 2020, 8(2): 133-143.
[2]	S. K. Ong, A. Y. C. Nee, A. W. W. Yew, N. K. Thanigaivel. AR-assisted robot welding programming [J]. Advances in Manufacturing, 2020, 8(1): 40-48.
[3]	D. Ni, A. W. W. Yew, S. K. Ong, A. Y. C. Nee. Haptic and visual augmented reality interface for programming welding robots [J]. Advances in Manufacturing, 2017, 5(3): 191-198.
[4]	H. C. Fang, S. K. Ong, A. Y. C. Nee. Novel AR-based interface for human-robot interaction and visualization [J]. Advances in Manufacturing, 2014, 2(4): 275-288.