Holorailway: an augmented reality system to support assembly operations in the railway industry

doi:10.1007/s40436-023-00479-5

Abstract

Abstract: During the last two decades, industrial applications of augmented reality (AR) have been incorporated in sectors such as automotive or aeronautics in tasks including manufacturing, maintenance, and assembly. However, AR’s potential has yet to be demonstrated in the railway sector due to its complexity and difficulties in automating tasks. This work aims to present an AR system based on HoloLens 2 to assist the assembly process of insulation panels in the railway sector significantly decreasing the time required to perform the assembly. Along with the technical description of the system, an exhaustive validation process is provided where the assembly using the developed system is compared to the traditional assembly method as used by a company that has facilitated a case study. The results obtained show that the system presented outperforms the traditional solution by 78% in the time spent in the localization subtask, which means a 47% decrease in the global assembly time. Additionally, it decreases the number of errors in 88% of the cases, obtaining a more precise and almost error-free assembly process. Finally, it is also proven that using AR removes the dependence on users’ prior knowledge of the system to facilitate assembly.

The full text can be downloaded at https://link.springer.com/article/10.1007/s40436-023-00479-5

Key words: Augmented reality (AR), Railway industry, Assembly, HoloLens 2

Clara Garcia, Mario Ortega, Eugenio Ivorra, Manuel Contero, Pau Mora, Mariano L. Alcañiz. Holorailway: an augmented reality system to support assembly operations in the railway industry[J]. Advances in Manufacturing, 2024, 12(4): 764-783.

TrendMD

References

[1] Schwab K (2017) The fourth industrial revolution, Hardcover edn, p. 192. Currency, Geneva
[2] Lasi H, Fettke P, Kemper HG et al (2014) Industry 4.0. Bus Inf Syst Eng 6(4):239-242
[3] Grieves M (2014) Digital twin: manufacturing excellence through virtual factory replication. White Pap 1:1-7
[4] Qi Q, Tao F (2018) Digital twin and big data towards smart manufacturing and Industry 4.0: 360 degree comparison. IEEE Access 6:3585-3593
[5] Carmigniani J, Furht B (2011) Augmented reality: an overview. In: Furht B (eds) Handbook of augmented reality. Springer, New York, pp 3-46
[6] De Pace F, Manuri F, Sanna A (2018) Augmented reality in Industry 4.0. Am J Comput Sci Inf Technol 6(1):1-7
[7] Santi GM, Ceruti A, Liverani A et al (2021) Augmented reality in Industry 4.0 and future innovation programs. Technologies 9(2):33. https://doi.org/10.3390/technologies9020033
[8] Bongomin O, Yemane A, Kembabazi B et al (2020) Industry 4.0 disruption and its neologisms in major industrial sectors: a state of the art. J Eng 2020:66. https://doi.org/10.1155/2020/8090521
[9] Langley A, Lawson G, Hermawati S et al (2016) Establishing the usability of a virtual training system for assembly operations within the automotive industry. Hum Factors Ergon Manuf Serv Ind 26(6):667-679
[10] Fraga-Lamas P, Fernández-Caramés TM, Blanco-Novoa O et al (2018) A review on industrial augmented reality systems for the Industry 4.0 shipyard. IEEE Access 6:13358-13375
[11] Regenbrecht H, Baratoff G, Wilke W (2005) Augmented reality projects in the automotive and aerospace industries. IEEE Comput Graph Appl 25(6):48-56
[12] Popov O, Iatsyshyn A, Sokolov D et al (2021) Application of virtual and augmented reality at nuclear power plants. In: Zaporozhets A, Artemchuk V (eds) Systems, decision and control in energy II, Springer, Cham, pp 243-260
[13] de Souza CLF, Mariano FCMQ, Zorzal ER (2020) A survey of industrial augmented reality. Comput Ind Eng 139:106159. https://doi.org/10.1016/j.cie.2019.106159
[14] Geng J, Song X, Pan Y et al (2020) A systematic design method of adaptive augmented reality work instruction for complex industrial operations. Comput Ind 119:103229. https://doi.org/10.1016/j.compind.2020.103229
[15] Dalle Mura M, Dini G (2021) Augmented reality in assembly systems: state of the art and future perspectives. In: International precision assembly seminar. Springer, pp 3-22
[16] Ashwini KB, Patil PN et al (2020) Tracking methods in augmented reality-explore the usage of marker-based tracking. In: Proceedings of the 2nd international conference on IoT, social, mobile, analytics & cloud in computational vision & bio-engineering (ISMAC-CVB 2020)
[17] Masood T, Egger J (2019) Augmented reality in support of Industry 4.0—implementation challenges and success factors. Robot Comput Integr Manuf 58:181-195
[18] Georgel PF (2011) Is there a reality in industrial augmented reality? In: 2011 10th IEEE international symposium on mixed and augmented reality. IEEE, pp 201-210
[19] Ortega M, Ivorra E, Juan A et al (2021) Mantra: an effective system based on augmented reality and infrared thermography for industrial maintenance. Appl Sci 11(1):385. https://doi.org/10.3390/app11010385
[20] Ungureanu D, Bogo F, Galliani S et al (2020) Hololens 2 research mode as a tool for computer vision research. arXiv preprint arXiv:2008.11239
[21] Vidal-Balea A, Blanco-Novoa O, Fraga-Lamas P et al (2020) Creating collaborative augmented reality experiences for Industry 4.0 training and assistance applications: performance evaluation in the shipyard of the future. Appl Sci 10(24):9073. https://doi.org/10.3390/app10249073
[22] Psarommatis F, May G, Dreyfus PA et al (2020) Zero defect manufacturing: state-of-the-art review, shortcomings and future directions in research. Int J Prod Res 58(1):1-17
[23] Psarommatis F, Sousa J, Mendonça JP et al (2022) Zero-defect manufacturing the approach for higher manufacturing sustain-ability in the era of Industry 4.0: a position paper. Int J Prod Res 60(1):73-91
[24] Wohlgemuth W, Triebfürst G (2000) Arvika: augmented reality for development, production and service. In: Proceedings of DARE 2000 on designing augmented reality environments, pp 151-152
[25] Lima JP, Roberto R, Simões F et al (2017) Markerless tracking system for augmented reality in the automotive Industry. Expert Syst Appl 82:100-114
[26] Egger J, Masood T (2020) Augmented reality in support of intelligent manufacturing—a systematic literature review. Comput Ind Eng 140:106195. https://doi.org/10.1016/j.cie.2019.106195
[27] Oh YJ, Park KY, Kim EK (2014) Mobile augmented reality system for design drawing visualization. In: 16th International conference on advanced communication technology. IEEE, pp 1296-1300
[28] Masoni R, Ferrise F, Bordegoni M et al (2017) Supporting remote maintenance in Industry 4.0 through augmented reality. Procedia Manuf 11:1296-1302
[29] Funk M, Kosch T, Schmidt A (2016) Interactive worker assistance: comparing the effects of in-situ projection, head-mounted displays, tablet, and paper instructions. In: Proceedings of the 2016 ACM international joint conference on pervasive and ubiquitous computing, pp 934-939
[30] Davies R (2015) Industry 4.0: digitalisation for productivity and growth. EPRS: European Parliamentary Research Service. https://policycommons.net/artifacts/1335939/industry-40/1942749/
[31] Masood T, Egger J (2020) Adopting augmented reality in the age of industrial digitalization. Comput Ind 115:103112. https://doi.org/10.1016/j.compind.2019.07.002
[32] Doshi A, Smith RT, Thomas BH et al (2017) Use of projector based augmented reality to improve manual spot-welding precision and accuracy for automotive manufacturing. Int J Adv Manuf Technol 89(5):1279-1293
[33] Mourtzis D, Zogopoulos V, Vlachou E (2018) Augmented reality supported product design towards Industry 4.0: a teaching factory paradigm. Procedia Manuf 23:207-212
[34] Alvarez H, Aguinaga I, Borro D (2011) Providing guidance for maintenance operations using automatic markerless augmented reality system. In: 2011 10th IEEE international symposium on mixed and augmented reality. IEEE, pp 181-190
[35] Fiorentino M, Uva AE, Gattullo M et al (2014) Augmented reality on large screen for interactive maintenance instructions. Comput Ind 65(2):270-278
[36] Mourtzis D, Zogopoulos V, Vlachou E (2017) Augmented reality application to support remote maintenance as a service in the robotics Industry. Procedia CIRP 63:46-51
[37] Hořejší P (2015) Augmented reality system for virtual training of parts assembly. Procedia Eng 100:699-706
[38] Füchter SK, Schlichting MS, Salazar G (2021) Aeronautic pilot training and augmented reality. Acta Imeko 10(3):66-71
[39] Fründ J, Gausemeier J, Matysczok C et al (2005) Using augmented reality technology to support the automobile development. In: computer supported cooperative work in design I: 8th international conference, CSCWD 2004, Xiamen, 26-28 May, Xiamen, China.
[40] Ceruti A, Marzocca P, Liverani A et al (2019) Maintenance in aeronautics in an Industry 4.0 context: the role of augmented reality and additive manufacturing. J Comput Des Eng 6(4):516-526
[41] Mourtzis D, Zogopoulos V, Xanthi F (2019) Augmented reality application to support the assembly of highly customized products and to adapt to production re-scheduling. Int J Adv Manuf Technol 105(9):3899-3910
[42] Frigo MA, da Silva E, Barbosa GF (2016) Augmented reality in aerospace manufacturing: a review. J Ind Intell Inf 4(2):66. https://doi.org/10.18178/jiii.4.2.125-130
[43] Jeffri NFS, Rambli DRA (2021) A review of augmented reality systems and their effects on mental workload and task performance. Heliyon 7(3):06277. https://doi.org/10.1016/j.heliyon.2021.e06277
[44] de Oliveira ME, Corrêa CG (2020) Virtual reality and augmented reality applications in agriculture: a literature review. In: 2020 22nd symposium on virtual and augmented reality (SVR). IEEE, pp 1-9
[45] Woll R, Damerau T, Wrasse K et al (2011) Augmented reality in a serious game for manual assembly processes. In: 2011 IEEE international symposium on mixed and augmented reality-arts, media, and humanities. IEEE, pp 37-39
[46] Barfield W, Caudell T (2001) Boeing’s wire bundle assembly project. In: fundamentals of wearable computers and augmented reality. CRC Press, pp 462-482
[47] Liu Y, Li S, Wang J et al (2015) A computer vision-based assistant system for the assembly of narrow cabin products. Int J Adv Manuf Technol 76(1):281-293
[48] Lai ZH, Tao W, Leu MC et al (2020) Smart augmented reality instructional system for mechanical assembly towards worker-centered intelligent manufacturing. J Manuf Syst 55:69-81
[49] Richardson T, Gilbert S, Holub J et al (2014) Fusing self-reported and sensor data from mixed-reality training
[50] Garrido-Jurado S, Muñoz-Salinas R, Madrid-Cuevas FJ et al (2016) Generation of fiducial marker dictionaries using mixed integer linear programming, pattern recognition. 51:481-491. https://doi.org/10.1016/j.patcog.2015.09.023
[51] Evans G, Miller J, Pena MI et al (2017) Evaluating the microsoft hololens through an augmented reality assembly application. In: degraded environments: sensing, processing, and display 2017, vol 10197. SPIE, pp 282-297
[52] Ariansyah D, Erkoyuncu JA, Eimontaite I et al (2022) A head mounted augmented reality design practice for maintenance assembly: toward meeting perceptual and cognitive needs of ar users. Appl Ergon 98:103597. https://doi.org/10.1016/j.apergo.2021.103597
[53] Vorraber W, Gasser J, Webb H et al (2020) Assessing augmented reality in production: remote-assisted maintenance with hololens. Procedia CIRP 88:139-144
[54] Brunzini A, Mandolini M, Caragiuli M, Germani M et al (2021) Hololens 2 for maxillofacial surgery: a preliminary study. In: international conference on design, simulation, manufacturing: the innovation exchange. Springer, pp 133-140
[55] Leu MC, ElMaraghy HA, Nee AY et al (2013) Cad model based virtual assembly simulation, planning and training. CIRP Ann 62(2):799-822
[56] Webel S, Becker M, Stricker D et al (2007) Identifying differences between cad and physical mock-ups using ar. In: 2007 6th IEEE and ACM international symposium on mixed and augmented reality. IEEE, pp 281-282
[57] Azpiazu J, Siltanen S , Multanen P et al (2011) Remote support for maintenance tasks by the use of augmented reality: the ManuVAR project. CARVI 2011: IX congress on virtualreality applications, 11-12 November, Alava, Spain
[58] Kwon HJ, Lee SI, Park JH et al (2021) Design of augmented reality train-ing content for railway vehicle maintenance focusing on the axle-mounted disc brake system. Appl Sci 11(19):9090. https://doi.org/10.3390/app11199090
[59] Yi B, Sun R, Long L et al (2022) From coarse to fine: an augmented reality-based dynamic inspection method for visualized railway routing of freight cars. Meas Sci Technol 33(5):055013. https://doi.org/10.1088/1361-6501/ac3c1c
[60] Seeliger A, Netland T, Feuerriegel S (2022) Aug-mented reality for machine setups: task performance and usability evaluation in a field test. Procedia CIRP 107:570-575
[61] Teruggi S, Fassi F (2022) Hololens 2 spatial map-ping capabilities in vast monumental heritage environments. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XLVI-2/W1-2022, pp. 489-496
[62] Unger M, Heinrich S, Rick M et al (2022) Hologram accuracy evaluation of hololens 2 for thermal imaging in medical applications. Curr Direct Biomed Eng 8(2):193-196
[63] Curless B, Levoy M (1996) A volumetric method for building complex models from range images. In: proceedings of the 23rd annual conference on computer graphics and interactive techniques, pp 303-312
[64] Marino E, Barbieri L, Colacino B et al (2021) An augmented reality inspection tool to support workers in Industry 4.0 environments. Comput Ind 127:103412. https://doi.org/10.1016/j.compind.2021.103412
[65] Lu Y, Zheng H, Chand S et al (2022) Outlook on human-centric manufacturing towards Industry 5.0. J Manuf Syst 62:612-627
[66] Cao A, Chintamani KK, Pandya AK et al (2009) Nasa tlx: software for assessing subjective mental workload. Behav Res Methods 41(1):113-117
[67] Loizeau Q, Danglade F, Ababsa F et al (2019) Evaluating added value of augmented reality to assist aeronautical maintenance workers—experimentation on on-field use case. In: international conference on virtual reality and augmented reality. Springer, pp 151-169
[68] Re GM, Oliver J, Bordegoni M (2016) Impact of monitor-based augmented reality for on-site industrial manual operations. Cognit Technol Work 18(2):379-392
[69] Jordan PW, Thomas B, McClelland IL et al (1996) Usability evaluation in industry, CRC Press, Boca Raton
[70] Fischini A, Ababsa F, Grasser M (2018) Usability of augmented reality in aeronautic maintenance, repair and overhaul. International conference on artificial reality and telexistence and Eurographics symposium on virtual environments, Nov 2018, Limassol, Cyprus.