Multimodal hand gesture recognition using single IMU and acoustic measurements at wrist

English version

Autoři: Nabeel Siddiqui ^aff001; Rosa H. M. Chan ^aff001
Působiště autorů: Department of Electrical Engineering, City University of Hong Kong, Tat Chee Avenue, Kowloon, Hong Kong, China ^aff001
Vyšlo v časopise: PLoS ONE 15(1)
Kategorie: Research Article
doi: https://doi.org/10.1371/journal.pone.0227039

Souhrn

To facilitate hand gesture recognition, we investigated the use of acoustic signals with an accelerometer and gyroscope at the human wrist. As a proof-of-concept, the prototype consisted of 10 microphone units in contact with the skin placed around the wrist along with an inertial measurement unit (IMU). The gesture recognition performance was evaluated through the identification of 13 gestures used in daily life. The optimal area for acoustic sensor placement at the wrist was examined using the minimum redundancy and maximum relevance feature selection algorithm. We recruited 10 subjects to perform over 10 trials for each set of hand gestures. The accuracy was 75% for a general model with the top 25 features selected, and the intra-subject average classification accuracy was over 80% with the same features using one microphone unit at the mid-anterior wrist and an IMU. These results indicate that acoustic signatures from the human wrist can aid IMU sensing for hand gesture recognition, and the selection of a few common features for all subjects could help with building a general model. The proposed multimodal framework helps address the single IMU sensing bottleneck for hand gestures during arm movement and/or locomotion.

Klíčová slova:

Accelerometers – Acoustic signals – Acoustics – Body limbs – Nonverbal communication – Wrist – Microphones – Hands

Zdroje

1. Jiang X, Merhi L-K, Menon C. Force Exertion Affects Grasp Classification Using Force Myography. IEEE Trans Human-Mach Syst. 2018 Apr;48(2):219–26.

2. Savur C, Sahin F. Real-Time American Sign Language Recognition System Using Surface EMG Signal. In: 2015 IEEE 14th International Conference on Machine Learning and Applications (ICMLA). Miami, FL, USA: IEEE; 2015. p. 497–502.

3. Galka J, Masior M, Zaborski M, Barczewska K. Inertial Motion Sensing Glove for Sign Language Gesture Acquisition and Recognition. IEEE Sensors J. 2016 Aug;16(16):6310–6.

4. Jeong-Mook Lim, Dong-Woo Lee, Bae-Sun Kim, Il-Yeon Cho, Jae-Cheol Ryou. Recognizing hand gestures using wrist shapes. In: 2010 Digest of Technical Papers International Conference on Consumer Electronics (ICCE). Las Vegas, NV: IEEE; 2010. p. 197–8.

5. Xu C, Pathak PH, Mohapatra P. Finger-writing with Smartwatch: A Case for Finger and Hand Gesture Recognition using Smartwatch. In: Proceedings of the 16th International Workshop on Mobile Computing Systems and Applications—HotMobile ‘15. Santa Fe, New Mexico, USA: ACM Press; 2015. p. 9–14.

6. CoolSo. Available from: http://www.coolso.com.tw/ (accessed on 10 August 2019)

7. Zhu Y, Jiang S, Shull PB. Wrist-worn hand gesture recognition based on barometric pressure sensing. In: 2018 IEEE 15th International Conference on Wearable and Implantable Body Sensor Networks (BSN). Las Vegas, NV, USA: IEEE; 2018. p. 181–4.

8. Shull PB, Jiang S, Zhu Y, Zhu X. Hand Gesture Recognition and Finger Angle Estimation via Wrist-Worn Modified Barometric Pressure Sensing. IEEE Trans Neural Syst Rehabil Eng. 2019 Apr;27(4):724–32. doi: 10.1109/TNSRE.2019.2905658 30892217

9. Jiang S, Lv B, Guo W, Zhang C, Wang H, Sheng X, et al. Feasibility of Wrist-Worn, Real-Time Hand, and Surface Gesture Recognition via sEMG and IMU Sensing. IEEE Trans Ind Inf. 2018 Aug;14(8):3376–85.

10. Watakabe M, Mita K, Akataki K, Itoh Y. Mechanical behaviour of condenser microphone in mechanomyography. Med Biol Eng Comput. 2001 Mar;39(2):195–201. doi: 10.1007/bf02344804 11361247

11. Siddiqui N, Chan RHM. A wearable hand gesture recognition device based on acoustic measurements at wrist. In: 2017 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) [Internet]. Seogwipo: IEEE; 2017. p. 4443–6.

12. Zhang C, Starner T, Inan O, Abowd GD, Xue Q, Waghmare A, et al. FingerPing: Recognizing Fine-grained Hand Poses using Active Acoustic On-body Sensing. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems—CHI ‘18. Montreal QC, Canada: ACM Press; 2018. p. 1–10.

13. Beck TW, Housh TJ, Cramer JT, Weir JP, Johnson GO, Coburn JW, et al. Mechanomyographic amplitude and frequency responses during dynamic muscle actions: a comprehensive review. Biomed Eng Online. 2005 Dec 19;4:67. doi: 10.1186/1475-925X-4-67 16364182

14. Orizio C, Liberati D, Locatelli C, De Grandis D, Veicsteinas A. Surface mechanomyogram reflects muscle fibres twitches summation. J Biomech. 1996 Apr;29(4):475–81. doi: 10.1016/0021-9290(95)00063-1 8964777

15. Guo W, Sheng X, Liu H, Zhu X. Mechanomyography Assisted Myoeletric Sensing for Upper-Extremity Prostheses: A Hybrid Approach. IEEE Sensors J. 2017 May 15;17(10):3100–8.

16. Harrison AP. A more precise, repeatable and diagnostic alternative to surface electromyography—an appraisal of the clinical utility of acoustic myography. Clin Physiol Funct Imaging. 2018 Mar;38(2):312–25. doi: 10.1111/cpf.12417 28251802

17. Orizio C. Soundmyogram and EMG cross-spectrum during exhausting isometric contractions in humans. J Electromyogr Kinesiol. 1992;2(3):141–9. doi: 10.1016/1050-6411(92)90011-7 20719607

18. Hermens HJ, Freriks B, Disselhorst-Klug C, Rau G. Development of recommendations for SEMG sensors and sensor placement procedures. J Electromyogr Kinesiol. 2000 Oct;10(5):361–74. doi: 10.1016/s1050-6411(00)00027-4 11018445

19. Kim T-K, Shimomura Y, Iwanaga K, Katsuura T. Influence of force tremor on mechanomyographic signals recorded with an accelerometer and a condenser microphone during measurement of agonist and antagonist muscles in voluntary submaximal isometric contractions. J Physiol Anthropol. 2008 Jan;27(1):33–42. doi: 10.2114/jpa2.27.33 18239348

20. Repnik E, Puh U, Goljar N, Munih M, Mihelj M. Using Inertial Measurement Units and Electromyography to Quantify Movement during Action Research Arm Test Execution. Sensors. 2018 Aug 22;18(9):2767

21. Orizio C, Perini R, Veicsteinas A. Muscular sound and force relationship during isometric contraction in man. Eur J Appl Physiol Occup Physiol. 1989;58(5):528–33. doi: 10.1007/bf02330708 2759079

22. Fulcher BD, Jones NS. hctsa: A Computational Framework for Automated Time-Series Phenotyping Using Massive Feature Extraction. Cell Syst. 2017 22;5(5):527–531.e3. doi: 10.1016/j.cels.2017.10.001 29102608

23. Liu L, Wang S, Hu B, Qiong Q, Wen J, Rosenblum DS. Learning structures of interval-based Bayesian networks in probabilistic generative model for human complex activity recognition. Pattern Recognition. 2018 Sep;81:545–61.

24. Estevez PA, Tesmer M, Perez CA, Zurada JM. Normalized Mutual Information Feature Selection. IEEE Trans Neural Netw. 2009 Feb;20(2):189–201. doi: 10.1109/TNN.2008.2005601 19150792

25. Liu X, Vega K, Maes P, Paradiso JA. Wearability Factors for Skin Interfaces. In: Proceedings of the 7th Augmented Human International Conference 2016 on—AH ‘16. Geneva, Switzerland: ACM Press; 2016 p. 1–8.

26. Liu Y, Norton JJS, Qazi R, Zou Z, Ammann KR, Liu H, et al. Epidermal mechano-acoustic sensing electronics for cardiovascular diagnostics and human-machine interfaces. Sci Adv. 2016 Nov;2(11):e1601185. doi: 10.1126/sciadv.1601185 28138529

Multimodal hand gesture recognition using single IMU and acoustic measurements at wrist

Souhrn

Klíčová slova:

Zdroje

PLOS One

INSIGHTS from European Respiratory Congress

Současné pohledy na riziko v parodontologii

Svět praktické medicíny 3/2024 (znalostní test z časopisu)

Kardiologické projevy hypereozinofilií

Střevní příprava před kolonoskopií