EagleHacks 2026 · 1st Place Overall and the Arthrex Challenge

Empowering surgeons with novel interfaces.

A hand-free surgery assistance interface, specializing in orthopedic surgery. It won 1st Place Overall and the Arthrex Challenge, and became my capstone with Arthrex.

  • RoleI designed all accessibility features, including hand tracking using MediaPipe, STT features, and scene controls.
  • Team3 people, one weekend
  • Built inTypeScript, React Three Fiber, Python, MediaPipe
  • Demoorthovision-seven.vercel.app
OrthoVision in surgeon mode: a 3D knee with muscles and tendons, beside a panel of toggles for bones, ligaments, soft tissue and muscles
OrthoVision in surgeon mode, from the live demo.

The constraint

Arthrex asked teams to help surgeons treat their patients better. Our answer was an interactive knee with a step-by-step ACL reconstruction walkthrough, separate surgeon and patient modes, and an AI assistant. One constraint shaped my part: in a sterile field, a surgeon can't reach for a mouse or a touchscreen.

Gestures that wait until they're meant

Raw hand tracking jitters, and a model that drifts while your hand rests feels broken. My Python sidecar (MediaPipe, OpenCV) streams gestures to the browser over WebSockets at 30 FPS, through three filters.

  1. Landmarks30 FPS, noisy
  2. Hysteresisa gesture must hold before it switches
  3. Dead zoneswallows small tremors
  4. EMA smoothingsteadies continuous motion
  5. Sceneorbit, zoom, select

Voice selects, hands navigate

Pointing at one of hundreds of small structures in mid-air is slow and error-prone. Naming one is instant. Saying a structure's name selects it and flies the camera to it, while hands handle orbit and zoom. A GPT-4o-mini assistant returns structured JSON that drives the scene.

Touch-free input has no click to confirm it worked, so the 452-mesh model answers every selection with an animated highlight. If it lands on the wrong structure, you see it right away.

What came next

The win turned into my capstone with Arthrex: a hand-tracking UI library for surgical VR.