Real-time K-TIG welding penetration prediction on embedded system using a segmentation-LSTM model

doi:10.1007/s40436-023-00437-1

Abstract

Abstract: Keyhole tungsten inert gas (K-TIG) welding is capable of realizing single-sided welding and double-sided forming and has been widely used in medium and thick plate welding. In order to improve the accuracy of automatic weld identification and weld penetration prediction of robot in the process of large workpiece welding, a two-stage model is proposed in this paper, which can monitor the K-TIG welding penetration state in real time on the embedded system, called segmentation-LSTM model. The proposed system extracts 9 weld pool geometric features with segmentation network, and then extracts the weld gap using a traditional algorithm. Then these 10-dimensional features are input into the LSTM model to predict the penetration state, including under penetration, partial penetration, good penetration and over penetration. The recognition accuracy of the proposed system can reach 95.2%. In this system, to solve the difficulty of labeling data and lack of segmentation accuracy, an improved LabelMe capable of live-wire annotation tool and a novel loss function were proposed, respectively. The latter was also called focal dice loss, which enabled the network to achieve a performance of 0.933 mIoU on the testing set. Finally, an improved slimming strategy compresses the network, making the segmentation network achieve real-time on the embedded system (RK3399pro).

The full text can be downloaded at https://link.springer.com/article/10.1007/s40436-023-00437-1

Key words: Keyhole tungsten inert gas (K-TIG) welding, Penetration state prediction, Segmentation-LSTM model, Embedded system, Focal dice loss, Improved LabelMe

Yong-Hua Shi, Zi-Shun Wang, Xi-Yin Chen, Yan-Xin Cui, Tao Xu, Jin-Yi Wang. Real-time K-TIG welding penetration prediction on embedded system using a segmentation-LSTM model[J]. Advances in Manufacturing, 2023, 11(3): 444-461.

TrendMD

References

1. Liu ZM, Chen SY, Liu S et al (2018) Keyhole dynamic thermal behaviour in K-TIG welding process. Int J Heat Mass Tran 123:54-66
2. Jarvis BL (2001) Keyhole gas tungsten arc welding: a new process variant. Dissertation, University of Wollongong
3. Fang ZJ, Weng WW, Wang WJ et al (2019) A vision-based robotic laser welding system for insulated mugs with fuzzy seam tracking control. Symmetry 11(11):1385. https://doi.org/10.3390/sym11111385
4. Zhu T, Shi YH, Cui SW et al (2019) Recognition of weld penetration during K-TIG welding based on acoustic and visual sensing. Sens Imaging 20(1):1-21
5. Wang ZS, Shi YH, Hong XB et al (2022) Weld pool and keyhole geometric feature extraction in K-TIG welding with a gradual gap based on an improved HDR algorithm. J Manuf Process 73:409-427
6. Zhan AW, Shi YH, Chen JR (2021) The effect of butt gap on the molten pool and keyhole of K-TIG welding 304 stainless steel. Hot Working Technology 50(23):139-145
7. Liu YK, Zhang YM (2013) Control of 3D weld pool surface. Control Eng Pract 21(11):1469-1480
8. Richardson RW, Gutow DA, Rao SH (1982) A vision based system for arc weld pool size control. Measurement and control for batch manufacturing pp 65-75
9. Zhang YM, Kovacevic R, Li L (1996) Characterization and realtime measurement of geometrical appearance of the weld pool. Int J Mach Tool Manuf 36(7):799-816
10. Shi FH, Huang XX, Duan Y et al (2010) Part-based model for visual detection and localization of gas tungsten arc weld pool. Int J Adv Manuf Tech 47(9):1097-1104
11. Wu D, Chen HB, Huang YM et al (2016) Weld penetration identification for UPPAW based on keyhole features and extreme learning machine. In 2016 IEEE workshop on advanced robotics and its social impacts (ARSO) pp 96-99. https://doi.org/10.1109/arso.2016.7736263
12. Wu D, Chen HB, Huang YM et al (2017) Monitoring of weld joint penetration during variable polarity plasma arc welding based on the keyhole characteristics and PSO-ANFIS. J Mater Process Tech 239:113-124
13. Wu D, Chen JS, Liu HB et al (2019) Weld penetration in situ prediction from keyhole dynamic behavior under time-varying VPPAW pools via the OS-ELM model. Int J Adv Manuf Tech 104(9):3929-3941
14. Liu XF, Wu CS, Jia CB et al (2017) Visual sensing of the weld pool geometry from the topside view in keyhole plasma arc welding. J Manuf Process 26:74-83
15. Liu XF, Jia CB, Wu CS et al (2017) Measurement of the keyhole entrance and topside weld pool geometries in keyhole plasma arc welding with dual CCD cameras. J Mater Process Tech 248:39-48
16. Li YF, Tian SS, Wu CS et al (2021) Experimental sensing of molten flow velocity, weld pool and keyhole geometries in ultrasonic-assisted plasma arc welding. J Manuf Process 64:1412-1419
17. Chen ZQ, Gao XD (2014) Detection of weld pool width using infrared imaging during high-power fiber laser welding of type 304 austenitic stainless steel. Int J Adv Manuf Tech 74(9/12):1247-1254
18. Gao XD, Zhang YX (2014) Prediction model of weld width during high-power disk laser welding of 304 austenitic stainless steel. Int J Precis Eng Man 15(3):399-405
19. Luo M, Yung CS (2015) Vision-based weld pool boundary extraction and width measure- ment during keyhole fiber laser welding. Opt Laser Eng 64:59-70
20. Kotecki DJ, Cheever DL, Howden DG et al (1972) Mechanism of ripple formation during weld solidification. WELD J, 51(8):368. https://doi.org/10.22486/iwj.v25i3.148316
21. Zhang WJ, Liu YK, Zhang YM (2012) Real-time measurement of three dimensional weld pool surface in GTAW. In Welding Processes. https://doi.org/10.5772/53753
22. Zhang K, Zhang YM, Chen JS et al (2017) Welding pool oscillation behaviors for pulsed GTA welding based on laser dot matrix sensing. In 2017 IEEE 7th Annual International Conference on CYBER Technology in Automation, Control, and Intelligent Systems (CYBER), pp 355-358. https://doi.org/10.1109/cyber.2017.8446232
23. Dong H, Cong M, Zhang YM et al (2018) Modeling and realtime prediction for complex welding process based on weld pool. Int J Adv Manuf Tech 96(5/8):2495-2508
24. Huang JK, Pan W, Chen JS et al (2018) The transient behaviours of free surface in a fully penetrated weld pool in gas tungsten arc welding. J Manuf Process 36:405-416
25. Li CK, Shi Y, Gu YF et al (2018) Monitoring weld pool oscillation using reflected laser pattern in gas tungsten arc welding. J Mater Process Tech 255:876-885
26. Liang ZM, Chang HX, Wang QY et al (2019) 3D reconstruction of weld pool surface in pulsed GMAW by passive biprism stereo vision. IEEE Robot and Autom Let 4(3):3091-3097
27. Gu ZN, Chen J, Wu CS (2021) Three-dimensional reconstruction of welding pool surface by binocular vision. Chin J Mech Eng-EN 34(1). https://doi.org/10.21203/rs.3.rs-19923/v1
28. Jiao WH, Wang QY, Cheng YC et al (2021) End-to-end prediction of weld penetration: a deep learning and transfer learning based method. J Manuf Process 63:191-197
29. Chen C, Xiao RQ, Chen HB et al (2021) Prediction of welding quality characteristics during pulsed GTAW process of aluminum alloy by multisensory fusion and hybrid network model. J Manuf Process 68:209-224
30. Knaak C, Kolter G, Schulze F et al (2019) Deep learning-based semantic segmentation for in-process monitoring in laser welding applications. Applications of Machine Learning 11139:1113905. https://doi.org/10.1117/12.2529160
31. wkentaro (2022) Labelme: image polygonal annotation with python (polygon, rectangle, circle, line, point and image-level flag annotation. https://github.com/wkentaro/labelme/ Accessed 6 March 2022
32. Yu CQ, Gao CX, Wang JB et al (2021) BiSeNet V2: bilateral network with guided aggregation for real-time semantic segmentation. Int J Comput Vision 129(11): 3051-3068
33. Long J, Shelhamer E, Darrell T (2015) Fully convolutional networks for semantic segmentation. In: proceedings of the IEEE conference on computer vision and pattern recognition, pp 3431-3440. https://doi.org/10.1109/CVPR.2015.7298965
34. Ronneberger O, Fischer P, Brox T (2015) U-Net: convolutional networks for biomedical image segmentation. In: international conference on medical image computing and computer-assisted intervention, pp 234-241. https://doi.org/10.1007/978-3-319-24574-4_28
35. Paszke A, Chaurasia A, Kim S et al (2016) ENet: a deep neural network architecture for real-time semantic segmentation. arXiv preprint. https://doi.org/10.48550/arXiv.1606.02147
36. Romera E, Alvarez J, Bergasa LM et al (2017) ERFNet: efficient residual factorized convnet for real-time semantic segmentation. IEEE T Intell Transp S 19(1):263-272
37. He KM, Zhang XY, Ren SQ et al (2016) Deep residual learning for image recognition. In: proceedings of the IEEE conference on computer vision and pattern recognition pp 770-778. https://doi.org/10.1109/cvpr.2016.90
38. Cordts M, Omran M, Ramos S et al (2016) The cityscapes dataset for semantic urban scene understanding. In: proceedings of the IEEE conference on computer vision and pattern recognition pp 3213-3223. https://doi.org/10.1109/CVPR.2016.350
39. Jadon S (2020) A survey of loss functions for semantic segmentation. In: 2020 IEEE Conference on Computational Intelligence in Bioinformatics and Computational Biology (CIBCB) pp 1-7. https://doi.org/10.1109/cibcb48159.2020.9277638
40. Ma YD, Liu Q, and Qian ZB (2004) Automated image segmentation using improved PCNN model based on cross-entropy. In Proceedings of 2004 International Symposium on Intelligent Multimedia, Video and Speech Processing pp 743-746. https://doi.org/10.1109/isimp.2004.1434171
41. Lin TY, Goyal P, Girshick R et al (2017) Focal loss for dense object detection. In: proceedings of the IEEE international conference on computer vision pp 2980-2988. https://doi.org/10.1109/iccv.2017.324
42. Sudre CH, Li WQ, Vercauteren T et al (2017) Generalised dice overlap as a deep learning loss function for highly unbalanced segmentations. In: deep learning in medical image analysis and multimodal learning for clinical decision support pp 240-248. https://doi.org/10.1007/978-3-319-67558-9_28
43. Berman M, Triki AR, Matthew BB (2018) The Lovasz-Softmax loss: a tractable surrogate for the optimization of the intersectionover-union measure in neural networks. In: proceedings of the IEEE conference on computer vision and pattern recognition pp 4413-4421. https://doi.org/10.1109/cvpr.2018.00464
44. Yu JQ, Blaschko MB (2015) The Lovász hinge: A convex surrogate for submodular losses. Stat 1050:24. https://doi.org/10.1109/tpami.2018.2883039
45. Fujishige S (2005) Submodular functions and optimization. Elsevier, Amsterdam
46. Shrivastava A, Gupta A, Girshick R (2016) Training region-based object detectors with online hard example mining. In: proceedings of the IEEE conference on computer vision and pattern recognition pp 761-769. https://doi.org/10.1109/cvpr.2016.89
47. Firefly (2022) Firefly: make technology simpler, make life smarter. https://www.t-firefly.com/doc/download/65.html/ Accessed 6 March 2022
48. Liu Z, Li JG, Shen ZQ et al (2017) Learning efficient convolutional networks through network slimming. In: proceedings of the IEEE international conference on computer vision pp 2736-2744. https://doi.org/10.1109/iccv.2017.298
49. Krishnamoorthi R (2018) Quantizing deep convolutional networks for efficient inference: a whitepaper. arXiv preprint. https://doi.org/10.48550/arXiv.1806.08342
50. Hecht-Nielsen R (1992) Theory of the backpropagation neural network. In: neural networks for perception pp 65-93. https://doi.org/10.1016/B978-0-12-741252-8.50010-8
51. Niranjan M (1999) Support vector machines: a tutorial overview and critical appraisal. IEE Colloquium on Applied Statistical Pattern Recognition. https://doi.org/10.1049/ic:19990359
52. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735-1780
53. Converti J (1981) Plasma-jets in arc welding. Dissertation, Massachusetts Institute of Technology
54. Wang B, Zhu XM, Zhang HC et al (2018) Characteristics of welding and arc pressure in the plasma-TIG coupled arc welding process. Metals 8(7):512-513