Integrated Circuits and Systems >

2024 , Vol. 1 >Issue 4: 227 - 238

DOI: https://doi.org/10.23919/ICS.2024.3512503

Original article

A Multi-Modal System Featuring Wireless Flexible Sensor Patches and a Depth-Sensing Imager for Home-Based Monitoring of Rehabilitation Exercises

  • RUNTIAN YANG 1 ,
  • YUHAN HOU 1 ,
  • SAVANNA BLADE 1 ,
  • YINFEI LI 1 ,
  • VANSH TYAGI 1 ,
  • GLORIA-EDITH BOUDREAULT-MORALES 1 ,
  • JOSÉ ZARIFFA 1, 2, 3 ,
  • XILIN LIU , 1, 3
Expand
  • 1 The Edward S. Rogers Sr. Department of Electrical and Computer Engineering (ECE), University of Toronto, Toronto, ON M5S 1A1, Canada
  • 2 Institute of Biomedical Engineering, University of Toronto, Toronto, ON M58 1A1, Canada
  • 3 KITE Research Institute, Toronto Rehabilitation Institute - University Health Network (UHN), Toronto, ON M58 1A1, Canada
+ CORRESPONDING AUTHOR: XILIN LIU(e-mail: ).

Runtian Yang https://orcid.org/0009-0008-5889-4234

The Edward S. Rogers Sr. Department of Electrical and Computer Engineering (ECE), University of Toronto, Toronto, ON, Canada

Runtian Yang (Student Member,, IEEE) is currently working toward the undergraduate degree in computer engineering with the Edward S. Rogers Sr. Department of Electrical and Computer Engineering (ECE), University of Toronto (UofT), Toronto, ON, Canada. He was at XLab, UofT, under the supervision of Prof. Xilin Liu from May 2023 to August 2024 and started working with LinLab, UofT, under the supervision of Prof. Qian Lin in September 2024. His research interests include software and hardware design for brain computer interfaces and in-vivo neural imaging and stimulation.

Yuhan Hou https://orcid.org/0009-0006-6138-8573

The Edward S. Rogers Sr. Department of Electrical and Computer Engineering (ECE), University of Toronto, Toronto, ON, Canada

Yuhan Hou (Graduate Student Member,, IEEE) received the Bachelor of Science degree in electrical engineering from the University of Toronto, Toronto, ON, Canada, in 2023. She is currently working toward the Ph.D. degree with the Edward S. Rogers Sr. Department of Electrical and Computer Engineering, University of Toronto. She completed a 16-month internship with Alphawave Semi, Toronto, from May 2021 to August 2022, as a Hardware Validation Engineer. She has authored papers published at the International Solid-State Circuits Conference (ISSCC), in IEEE Transactions on Biomedical Circuits and Systems (TBioCAS), and others. Her research interests include analog and mixed-signal integrated circuits design and miniature system integration for neural interfaces, wearable devices, and other biomedical applications.

Savanna Blade https://orcid.org/0009-0004-9772-3543

The Edward S. Rogers Sr. Department of Electrical and Computer Engineering (ECE), University of Toronto, Toronto, ON, Canada

Savanna Blade (Student Member,, IEEE) received the B.A.Sc. (Hons.) degree in engineering science with specializations in robotics and artificial intelligence from the University of Toronto, Toronto, ON, Canada, in 2023, where she is currently working toward the M.A.Sc. degree in electrical and computer engineering. Her research interests include embedded systems design and edge-AI development for closed-loop brain-computer interfaces.

Yinfei Li

The Edward S. Rogers Sr. Department of Electrical and Computer Engineering (ECE), University of Toronto, Toronto, ON, Canada

Yinfei Li received the Bachelor of Science degree in physics and mechanical engineering from the University of North Carolina at Charlotte, Charlotte, NC, USA, in 2022, and the Master of Engineering degree in computer engineering from the University of Toronto, Toronto, ON, Canada, in 2024. He was at X-lab, University of Toronto, from 2023 to 2024. His expertise is embedded software development, Linux kernel configuration and management, and bioelectronics for brain-machine interfaces.

Vansh Tyagi https://orcid.org/0009-0004-5991-7005

The Edward S. Rogers Sr. Department of Electrical and Computer Engineering (ECE), University of Toronto, Toronto, ON, Canada

Vansh Tyagi received the Bachelor of Science degree in electrical and computer engineering from the University of Wisconsin—Madison, Madison, WI, USA, in 2022. He is currently working toward the Master of Engineering degree in electrical and computer engineering with the University of Toronto, Toronto, ON, Canada. His area of expertise is in real-time embedded system development.

Gloria-Edith Boudreault-Morales https://orcid.org/0009-0001-0421-1409

Institute of Biomedical Engineering, University of Toronto, Toronto, ON, Canada

Gloria-Edith Boudreault-Morales was born in Montreal, Canada, in 1999. She received the B.A.Sc degree in mechanical engineering and the M.A.Sc degree in biomedical engineering from the University of Toronto, Toronto, ON, Canada, in 2022 and 2024, respectively. She was a graduate student at KITE, Toronto Rehabilitation Institute -University Health Network, Toronto, under the supervision of Dr José Zariffa. The focus of her research was on lightweight monocular RGB human pose estimation models, where she investigated the effect of depth data and physical impairments on model performances.

José Zariffa https://orcid.org/0000-0002-8842-745X

The Edward S. Rogers Sr. Department of Electrical and Computer Engineering (ECE), University of Toronto, Toronto, ON, Canada

Institute of Biomedical Engineering, University of Toronto, Toronto, ON, Canada

KITE Research Institute, Toronto Rehabilitation Institute -University Health Network (UHN), Toronto, ON, Canada

José Zariffa (Senior Member,, IEEE) received the Ph.D. degree from the Department of Electrical and Computer Engineering and the Institute of Biomaterials and Biomedical Engineering, University of Toronto, Toronto, ON, Canada, in 2009. He completed Postdoctoral fellowships at International Collaboration On Repair Discoveries (ICORD), Vancouver, BC, Canada, and the Toronto Rehabilitation Institute, University Health Network, Toronto. He is currently a Senior Scientist with KITE, Toronto Rehabilitation Institute, University Health Network, Toronto, and an Associate Professor with the Institute of Biomedical Engineering, University of Toronto. His research focuses on neuroprosthetics and technology for upper limb neurorehabilitation, encompassing work in wearable sensors, neural interfaces, electrophysiology, and machine learning. Dr. Zariffa holds the KITE Chair in Spinal Cord Injury Research. In 2021, his team was awarded the Grand Prize in the Spinal Cord Rehab Innovation Challenge. He is an Associate Editor for the Journal of Neuroengineering and Rehabilitation.

Xilin Liu https://orcid.org/0000-0002-9547-3905

The Edward S. Rogers Sr. Department of Electrical and Computer Engineering (ECE), University of Toronto, Toronto, ON, Canada

KITE Research Institute, Toronto Rehabilitation Institute - University Health Network (UHN), Toronto, ON, Canada

Xilin Liu (Senior Member,, IEEE) received the Ph.D. degree from the University of Pennsylvania, Philadelphia, PA, USA, in 2017. He held industrial positions at Qualcomm Inc., San Diego, CA, USA. He is currently an Assistant Professor with the Edward S. Rogers Sr. Department of Electrical and Computer Engineering (ECE), University of Toronto, Toronto, ON, Canada. He is also an affiliated Scientist with the University Health Network (UHN), Toronto. His research interests include analog and mixed-signal IC design for biomedical circuits and systems, especially for neural interfacing and neuromodulation. He is an Associate Editor for IEEE Transactions of Biomedical Circuits and Systems (TBioCAS) and IEEE Transactions on Circuits and Systems II: Express Briefs (TCAS-II). He was on the committees of several CASS and SSCS conferences.

Received date: 2024-11-11

  Revised date: 2024-11-29

  Accepted date: 2024-12-02

  Online published: 2025-01-09

Fold

Abstract

Monitoring rehabilitation progress at home over the long term following a spinal cord injury (SCI) is crucial for maximizing therapeutic outcomes and enhancing the quality of life of affected individuals. Comprehensive monitoring requires collecting a range of physiological data, including surface electromyography (sEMG) and exercise motion data. Currently, assessments typically take place in clinical settings, which can be both costly and inconvenient for patients. There is a lack of accessible, user-friendly systems that allow individuals with SCI to independently gather this data at home. Additionally, video recordings may be necessary to verify that patients are positioning the sensors correctly and performing the exercises accurately. To bridge this gap, we have developed a self-contained, multi-modal sensor system that captures sEMG and motion data, along with depth-sensing video to track patient exercises while ensuring privacy by minimizing identifiable details. The system includes a configurable number of wireless, multisensor wearable patches that are easy to attach and comfortable for extended use, along with a time-of-flight depth-sensing camera. The multi-modal data is streamed and synchronized in real-time on a Raspberry Pi, establishing an innovative platform to support SCI rehabilitation and adaptable for various clinical monitoring applications.

Key words： Bluetooth low energy; embedded system; multimodal sensor system; rehabilitation monitoring; SEMG; spinal cord injury; wireless wearable device.

Cite this article

RUNTIAN YANG , YUHAN HOU , SAVANNA BLADE , YINFEI LI , VANSH TYAGI , GLORIA-EDITH BOUDREAULT-MORALES , JOSÉ ZARIFFA , XILIN LIU . A Multi-Modal System Featuring Wireless Flexible Sensor Patches and a Depth-Sensing Imager for Home-Based Monitoring of Rehabilitation Exercises[J]. Integrated Circuits and Systems, 2024 , 1(4) : 227 -238 . DOI: 10.23919/ICS.2024.3512503

Fund

This work was supported in part by the Faculty of Applied Science and Engineering, University of Toronto and the KITE Research Institute under the EHMSeed grant, in part by the Toronto Rehabilitation Institute—University Health Network, and in part by the Natural Sciences and Engineering Research Council of Canada (NSERC) under Discovery Grant RGPIN-2022-04957.
This work involved human subjects or animals in its research. Approval of all ethical and experimental procedures and protocols was granted by the Health Sciences Research Ethics Board (REB) at the University of Toronto under Application No. 44439.

ACKNOWLEDGMENT

The authors thank the Faculty of Applied Science and Engineering at the University of Toronto and the KITE Research Institute, Toronto Rehabilitation Institute-University Health Network, for funding this project through the EHMSeed grant.

References

Publishing order | Descend order by publishing year | Descend order by cited within
[1]
N. A. Silva, N. Sousa, R. L. Reis, and A. J. Salgado, “From basics to clinical: A comprehensive review on spinal cord injury,” Prog. Neurobiol., vol. 114, pp. 25-57, 2014.

[2]
AANS,“Spinal cord injury,” 2024. Accessed: Dec. 15, 2024. [Online].

[3]
World Health Organization,“Spinal cord injury,” 2024. Accessed: Dec. 15, 2024. [Online].

[4]
A. S. Burns and J. F. Ditunno, “Establishing prognosis and maximizing functional outcomes after spinal cord injury: A review of current and future directions in rehabilitation management,” Spine, vol. 26, no. 24S, pp. S137-S145, 2001.

[5]
A. S. Gorgey, “Exercise awareness and barriers after spinal cord injury,” World J. Orthop., vol. 5, no. 3, p. 158-162, Jul. 2014.

[6]
G. Balbinot et al., “The use of surface EMG in neurorehabilitation following traumatic spinal cord injury: A scoping review,” Clin. Neurio-physiol., vol. 138, pp. 61-73, 2022. [Online].

[7]
X. Liu, J. Sacks, M. Zhang, A. G. Richardson, T. H. Lucas, and J. Van der Spiegel, “The virtual trackpad: An electromyography-based, wireless, real-time, low-power, embedded hand-gesture-recognition system using an event-driven artificial neural network,” IEEE Trans. Circuits Syst. II, Exp. Briefs, vol. 64, no. 11, pp. 1257-1261, Nov. 2017.

[8]
V. J. K. Morinigo, S. Benatti, and L. Benini, “A high SNR, low-latency dry EMG acquisition system for unobtrusive HMI devices,” in Proc. 2022 IEEE Biomed. Circuits Syst. Conf. (BioCAS), 2022, pp. 544-548.

[9]
S. Blade et al., “An sEMG-controlled prosthetic hand featuring a tiny CNN-transformer model and force feedback,” in Proc. 2023 IEEE Biomed. Circuits Syst. Conf., 2023, pp. 1-5.

[10]
K. Arunganesh, G. Nagarajan, N. Sivakumaran, and P. Karthick, “Recognition of lower limb movements using machine learning methods and bispectral maps of wireless sEMG measurements,” IEEE Sensors Lett., vol. 7, no. 9, Sep. 2023, Art. no. 7004604.

[11]
C. Xie, Q. Yang, Y. Huang, S. W. Su, T. Xu, and R. Song, “A hybrid arm-hand rehabilitation robot with EMG-based admittance controller,” IEEE Trans. Biomed. Circuits Syst., vol. 15, no. 6, pp. 1332-1342, Dec. 2021.

[12]
Y. Hou, J. Ji, Y. Zhu, T. Dell, and X. Liu, “Flexible gel-free multi-modal wireless sensors with edge deep learning for detecting and alerting freezing of gait symptom,” IEEE Trans. Biomed. Circuits Syst., vol. 17, no. 5, pp. 1010-1021, Oct. 2023.

[13]
F. Chamberland et al., “Novel wearable HD-EMG sensor with shift-robust gesture recognition using deep learning,” IEEE Trans. Biomed. Circuits Syst., vol. 17, no. 5, pp. 968-984, Oct. 2023.

[14]
D. Zhang et al., “Stretchable and durable HD-sEMG electrodes for accurate recognition of swallowing activities on complex epidermal surfaces,” Microsystems Nanoengineering, vol. 9, no. 115, 2023.

[15]
E. Romero Avila, E. Junker, and C. Disselhorst-Klug, “Introduction of a sEMG sensor system for autonomous use by inexperienced users,” Sensors, vol. 20, no. 24, Dec. 2020, Art. no. 7348, doi: 10.3390/s20247348.

[16]
G. Biagetti, P. Crippa, L. Falaschetti, S. Orcioni, and C. Turchetti, “Human activity monitoring system based on wearable sEMG and accelerometer wireless sensor nodes,” BioMedical Eng. OnLine, vol. 17, no. S1, Nov. 2018, Art. no. 132.

[17]
OpenBCI,“Cyton + daisy biosensing boards (16-channels),” Accessed: Dec. 15, 2024. [Online].

[18]
D. Inc.,“Trigno wireless EMG systems,” 2024, Accessed:Nov. 9, 2024. [Online]. Available: https://delsys.com/products/

[19]
MyoVision,“Myovision scanvision EMG system,” 2024, Accessed: Nov. 9, 2024. [Online].

[20]
X. Liu, M. Zhang, B. Subei, A. G. Richardson, T. H. Lucas, and J. Van der Spiegel, “The PennBMBI: Design of a general purpose wireless brain-machine-brain interface system,” IEEE Trans. Biomed. Circuits Syst., vol. 9, no. 2, pp. 248-258, Apr. 2015.

[21]
D. C. Luvizon, D. Picard, and H. Tabia, “2D/3D pose estimation and action recognition using multitask deep learning,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2018, pp. 5137-5146.

[22]
K. Li, S. Wang, X. Zhang, Y. Xu, W. Xu, and Z. Tu, “Pose recognition with cascade transformers,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2021, pp. 1944-1953.

[23]
H. P. Jain, A. Subramanian, S. Das, and A. Mittal, “Real-time upper-body human pose estimation using a depth camera,” in Proc. Comput. Vis./Comput. Graph. Collaboration Techn., 5th Int. Conf., Oct. 10-11, 2011, pp. 227-238.

[24]
P. Speciale, J. L. Schonberger, S. B. Kang, S. N. Sinha, and M. Pollefeys, “Privacy preserving image-based localization,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2019, pp. 5493-5503.

[25]
R. Yang et al., “A multi-modal system with wireless flexible sensor patches and a depth-sensing imager for at-home monitoring of rehabilitation exercises,” in Proc. 2024 IEEE Biomed. Circuits Syst. Conf., 2024, pp. 1-5.

[26]
Arducam,“Time of flight (ToF) camera for raspberry pi,” Accessed: Dec. 15, 2024. [Online].

[27]
M. S. Momin, A. Sufian, D. Barman, P. Dutta, M. Dong, and M. Leo, “In-home older adults’ activity pattern monitoring using depth sensors: A review,” Sensors, vol. 22, no. 23, Nov. 2022, Art. no. 9067, doi: 10.3390/s22239067.

[28]
K. K. Balasubramanian et al., “Neural network-based bluetooth synchronization of multiple wearable devices,” Nat. Commun., vol. 14, no. 1, 2023, Art. no. 4472.

[29]
E. Ceolini, G. Taverni, L. Khacef, M. Payvand, and E. Donati, “Sensor fusion using EMG and vision for hand gesture classification in mobile applications,” in Proc. 2019 IEEE Biomed. Circuits Syst. Conf., 2019, pp. 1-4.

Options
Outlines

/

〈 〉