SoftPINCH

Real-time EMG decoding for control of a soft robotic hand exoskeleton using deep learning.

NOTE

This work is part of an ongoing master’s thesis project. Certain implementations, refinements, and code structure optimizations are still under development.

Overview

This project presents a EMG biosignal framework for real-time control of a soft hand exoskeleton. The framework investigates the contributions of EMG signals, for decoding hand motor movements.

Three deep learning architectures are explored:

• LSTM ($N_1$)
• CNN + LSTM ($N_2$)
• CNN + LSTM + Attention ($N_3$)

The system integrates:

• Real-time EMG acquisition
• EMG preprocessing pipelines
• Neural decoding networks
• Decision-level fusion

System Architecture

The framework and step-by-step overview:

1. EMG acquisition:
   EMG is recorded from the forearm

2. Signal preprocessing:
   The signal is filtered, trimmed, mitigating outliers, converted into an envelope, normalized, and segmented into periods with corresponding labels for supervised learning framework.

3. Deep neural decoding and feature representation:
   Decode processes signal into structured numerical vectors that machine learning algorithms can process. From one of the three networks: $N_1$, $N_2$, or $N_3$.

4. Motion classification:
   Forward feature representation through dense layers.

5. Exoskeleton actuation:
   Forward the classification via a TCP client/server connection to a ESP controller. Which actuates the tendon-driven exoskeleton and provide feedback from the fingertip sensor (MagSense).

Deep Learning Models

$N_1$ — LSTM

Baseline temporal sequence model for biosignal decoding.

$N_2$ — CNN + LSTM

Sequential 1D CNN layers extract local temporal and cross-channel features before the LSTM models temporal dependencies.

$N_3$ — CNN + LSTM + Attention

Attention mechanism enhances temporal feature weighting and improves discriminative representation learning.

Model Specifications

To ensure exact reproducibility, the details regarding our model architecture, data preprocessing pipeline, and decision logic are detailed below.

1. Data Preprocessing & Windowing

• Input Dimensions: (Batch Size, Time Steps, Channels)
• Windowing Choices:
  • RMS Envelope Window Size: [250 ms / 500 samples]
  • RMS Envelope Step Size: [25 ms / 50 samples / 90 % overlap]
  • Hampel Window Size: [100 samples]
  • Hamepl Sigma: [2 - Higher value gives more tolerance to outliers]
  • Trial Segmentation: [Extracted from 3.0s intervals within the same trial]
• Normalization Procedure: [e.g., Channel-wise z-score normalization calculated per subject]

2. Hyperparameter search space

Hyperparameter	Range	Type
Learning rate	(10⁻⁵, 10⁻³)	Continuous
Weight decay	(10⁻⁵, 10⁻³)	Continuous
Dropout	(0.1, 0.4)	Continuous
Batch size	(16, 32, 64, 128)	Ordinal
Dense ratio	(0.25, 0.5, 0.75, 1.0)	Ordinal
Hidden units	(32, 64, 128, 256)	Ordinal
Filters	(16, 32, 64)	Ordinal
Kernel size	(3, 5, 7, 9, 11)	Ordinal
Activation	(ReLU, ELU)	Choice
Bidirectional	(False, True)	Choice
Layers	(1, 2, 3)	Choice

Repository Structure

The repository contains the following elements:

• data_fusion:
  • data_fusion_manager : This is the key component for data aquistion of both EMG and EEG data given a protocol. EEG can be turned off by chancing the constant "METHOD" : METHOD = '_ _ EMG'.
  • EEG_collector : Used within data_fusion_manager to save EEG data.
  • EMG_collector : Used within data_fusion_manager to save EMG data.
• experiment:
  • experimental_protocol : Used within data_fusion_manager to handle the experimental protocol
  • metabolic_cost_exp : Holds the experimental protocol used in the muscular effort experiments
  • real_time_operation : Used for deployment of traning, real-time inference model
• models:
  • classification_pipeline : Handle every training senario of models.
  • loggings : Contain pretrained models of subject-independent classification and real-time inference systems.
• utilities:
  • Contrains additional functionality used by the other scripts.

Environment

Follow these steps to setup the virtual environment: REQUIREMENT : Python version: 3.11

Install: py -3.11 -m venv .venv source .venv/Scripts/activate pip install -r requirements.txt

OLD VERSION CONFLICT - change these to avoid sherpa shutdown erros:
Follow the path :
.venv/lib/python3.11/site-packages/GPyOpt/core/evaluators/batch_local_penalization.py

Do the following change in line 67:
minusL = res.fun[0][0] ->
minusL = res.fun if isinstance(res.fun, float) else res.fun[0][0]

Remarks

Trigno implementation only works in windows, at the time being.

data_fusion_manager script with listen_for_terminal_input func only works in windows given command: msvcrt.kbhit()

data_fusion_manager script with listen_for_terminal_input func only works in windows given command: msvcrt.kbhit()