Skip to content

Integrate Oculus Quest VR controls into lerobot-record as a first-class teleoperator - #2

Draft
Ac31415 with Copilot wants to merge 3 commits into
copilot/integrate-vr-controls-lerobotfrom
copilot/integrate-vr-controls-lerobot-again
Draft

Integrate Oculus Quest VR controls into lerobot-record as a first-class teleoperator#2
Ac31415 with Copilot wants to merge 3 commits into
copilot/integrate-vr-controls-lerobotfrom
copilot/integrate-vr-controls-lerobot-again

Conversation

Copilot AI commented Apr 14, 2026

Copy link
Copy Markdown

vr_aic.py was a standalone ROS2 node that couldn't participate in lerobot's record/replay loop. VR controller inputs were never captured into datasets.

New action type — VRMotionUpdateActionDict (types.py)

Absolute TCP pose + gripper, distinct from the existing velocity-based MotionUpdateActionDict:

pose.position.{x,y,z} | pose.orientation.{x,y,z,w} | gripper.position

New teleoperator — AICVRTeleop / AICVRTeleopConfig (aic_teleop.py)

  • Registered as "aic_vr" — drop-in with existing lerobot record CLI
  • 30 Hz ROS2 timer mirrors wrist pose (TF oculus_r → base_link) to absolute TCP target; same axis remapping, step-cap, sphere clamp, and position LPF as vr_aic.py
  • Full analogue stick support (yaw+Z / XY / pitch+roll with deadzone + power-law ramp)
  • Controller bindings: A = cycle IDLE→VR→ANALOGUE→IDLE · B = orientation follow · RJ = analogue frame · RG = gripper toggle
  • Thread-safe: 30 Hz timer updates _current_action under a lock; get_action() snapshots it
  • TF failures (startup / oculus_reader not running) caught as LookupException / ExtrapolationException and logged at DEBUG

Robot-side changes (aic_robot_aic_controller.py)

  • New teleop_target_mode="vr_cartesian" sends MotionUpdate with trajectory_generation_mode=MODE_POSITION (2) + absolute pose in base_link, instead of the velocity-mode path
  • TELEOP stiffness/damping (60/50) used for compliant operation, matching vr_aic.py
  • Gripper publisher (/gripper_commands → JointState) added to AICRos2Interface; send_action_vr_cartesian publishes both pose and gripper atomically per step

Usage

lerobot record \
  --robot.type=aic_controller \
  --robot.teleop_target_mode=vr_cartesian \
  --teleop.type=aic_vr

Copilot AI and others added 2 commits April 14, 2026 18:04
…t VR controls

Agent-Logs-Url: https://github.com/Ac31415/aic/sessions/54e9ed38-f1cb-40a3-81de-3af58a6e4d7b

Co-authored-by: Ac31415 <21079375+Ac31415@users.noreply.github.com>
… TF exceptions

Agent-Logs-Url: https://github.com/Ac31415/aic/sessions/54e9ed38-f1cb-40a3-81de-3af58a6e4d7b

Co-authored-by: Ac31415 <21079375+Ac31415@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants