Capstone with Arthrex · in progress

Hand tracking for surgical work in VR.

Arthrex regularly uses VR in its core surgical operations, from training surgeons to designing operating rooms. My capstone with Arthrex is a hand-tracking UI library in Unreal Engine for those surgical interactions in VR.

  • RoleInteraction design, user testing, Unreal Engine development
  • PartnerArthrex
  • Built inUnreal Engine
  • MeasuresNASA TLX, task time, errors
An operating room: surgeons at the table under surgical lights, an MRI scan on a wall display, an equipment tower and staff at workstations
An Arthrex operating room. Photo: Arthrex.

Why hands

In surgery, a surgeon's hands hold instruments, not controllers. A VR session for training or for planning a room should ask for the same hands. The library gives Arthrex's VR projects one shared set of hand interactions, so each project doesn't rebuild them.

How each version is tested

The library grows one surgical task at a time, and every version is tested with people before the next one starts.

  1. Builda gesture set for one surgical task
  2. Measuretask time, errors and NASA TLX
  3. Changewhat the scores point to
  4. Test againwith the next version

Why NASA TLX

The NASA Task Load Index asks people to rate a task on six scales: mental demand, physical demand, time pressure, performance, effort and frustration. A gesture can be fast and still tiring to hold. Timing alone would miss that; the workload scores catch it.

Where it started

The capstone came out of OrthoVision, the touch-free knee explorer my team built at EagleHacks 2026. It won 1st Place Overall and the Arthrex Challenge.