Eni Halilaj
CARNEGIE MELLON UNIVERSITY
Title: Video and inertial sensor fusion for accurate tracking of orthopaedic rehabilitation
Collaborators:
- Jessica Hodgins, PhD (CMU)
- Physical Therapists (University of Pittsburgh)
Abstract:
Physical therapy utilization is highly associated with functional outcomes in orthopaedics. Quantitative remote monitoring technologies aided by computer vision and wearable technologies could increase therapy utilization and improve personalization. We will use data fusion algorithms that take advantage of the strengths of these two different modalities with complementary strengths to improve pose estimation accuracy, and therefore physical therapy assessment. Such lightweight multimodal systems could be easily deployed to clinics, assisted living communities, and patient homes. We expect our automated approach using videos and inertial sensors to achieve greater agreement with adjudicated clinician assessments of exercise correctness than either modality alone. Ultimately, simple tools to track the quality of physical therapy will be impactful both for research and clinical practice. They could enable insight into what types of exercises are most related to long-term postsurgical outcomes, informing the delivery of selective feedback for in-clinic and at-home therapy.
Michelle Johnson
UNIVERSITY OF PENNSYLVANIA
Title: Computer vision to automatically analyze therapist-patient interactions
Collaborators:
- Nidhi Seethapathj, PhD
- Konrad Kording, PhD
Abstract:
With the growing population of older adults and the preference for automated low-cost home-based rehabilitation therapy, there is a need to develop tools to analyze therapist-patient interactions in the real world. In this pilot project, we firstly propose to build upon state-of-the-art computer vision models to develop a tool for three-dimensional tracking of patient-therapist interactions from videos. We will develop the tool by pre-training and modifying a three-dimensional tracking-based computer vision model called VIBE using publicly available labeled datasets of human-human interaction. Secondly, we will test the performance of the tool we develop on labeled domainspecific therapist-patient interaction videos and compare it to the performance of the more commonly-used skeletal and image-based OpenPose computer vision model. Finally, we will use these therapist-patient interaction data from a few hundred videos to train an LSTM and Dynamic Time Warping-based machine learning model. The goal of this model will be to predict the therapist’s intervention patterns such as the distance from the patient, degree and duration of physical contact with the patient, patient functional ability, and the presence of verbal cues, given the patient movements as input. This model could be incorporated into the programming of virtual and robotic therapists to provide them with a model of when and how to intervene when observing a patient’s movements. The tools, data, and models developed as a part of the project will be made publicly available to the community to further build upon.
Ryan Roemmich
KENNEDY KRIEGER INSTITUTE
Title: Video-based gait assessment for persons post-stroke
Collaborators:
- Jan Stenum, PhD (Kennedy Krieger Institute and Johns Hopkins University School of Medicine)
- Darcy Reisman, PT, PhD (University of Delaware)
Abstract:
The primary goal of this proposal is to develop and validate a video-based approach for post-stroke gait assessment. There is a need for methods that can be used directly in the home or clinic to provide more detailed, comprehensive information about walking than can be collected using currently available approaches like gait mats or wearables. We have developed a video-based pose estimation workflow that inputs simple digital videos recorded from household devices (e.g., smartphones, tablets, laptop computers) and outputs a wide range of spatiotemporal and kinematic gait parameters. We have validated our approach in healthy adults and demonstrated that the video-based gait assessment shows strong agreement with ground-truth measurements from three-dimensional motion capture; however, clinical impact will be limited until we understand how well this approach performs in clinical populations (e.g., persons post-stroke). The ultimate outcome of this project will be a freely-available video-based workflow for performing gait assessments in persons post-stroke directly in the home or clinic with minimal costs of time, money, or effort.
Na Jin Seo
MEDICAL UNIVERSITY OF SOUTH CAROLINA
Title: Development of a system to track upper extremity task practice at home for stroke survivors
Collaborators:
- Gabrielle Scronce, PT, PhD (Medical University of South Carolina)
- Nader Naghavi, PhD (University of Massachusetts Amherst)
- Feng Luo, PhD (Clemson University)
Abstract:
Upper extremity (UE) hemiplegia is a serious problem affecting the lives of many people post-stroke. Basic science research suggests that recovery requires high repetitions of task-specific practice. Sufficient practice cannot be completed during therapy sessions, requiring patients to perform additional task practices at home on their own. Adherence to these home task practices is often limited and is likely a factor reducing the effectiveness of rehabilitation post-stroke. However, home adherence is typically measured by self-reports that are known to be inconsistent with objective measurement. The objective of the proposed project is to develop a sensor-based system to accurately assess both repetitions and quality of UE task practice performed at home by stroke survivors. A state-of-the-art system has been tested only in the laboratory setting. Thus, the accuracy for home use is currently unknown. Therefore, for the proposed study, we will calibrate a sensor-based tracking system during the laboratory session, which will be given to the patients for their home use. We will enhance the existing state-of-the-art system by combining 3 approaches: Dynamic Time Warping to classify varying time series data, patient-specific modeling, and artificial augmentation to accommodate sensor misplacement by patients at home. Hypothesis: The system will track UE task practice in the home with accuracy ≥0.8 for repetition and >0.7 for quality. Impact: The quantity and quality feedback from the system is expected to optimize effective task practice at home by patients. The system is expected to also enable adherence- and progress-driven clinic visits to maximize efficiency of therapy service.