Developing a Human-Machine Interface with Vision Using MediaPipe and Python

I built a real-time vision-based Human–Machine Interface that translates human hand motion into analog control signals, demonstrated through controlling a racing game using only a webcam.

Key Ideas

• Binary inputs (keyboard) are limiting • Analog control better matches human motion • Computer vision enables controller-free interaction

Tech Stack

• Python • OpenCV • MediaPipe Hands • vJoy / pyvjoy • Real-time smoothing & control logic

How It Works

1. Webcam captures frames

2. MediaPipe detects hand landmarks

3. Hand orientation is computed

4. Motion is smoothed temporally

5. Output mapped to a virtual joystick

Applications

• Human–robot interaction • Assistive tech • Vision-based control systems • AR/VR interfaces

Demo & Code

🎥 Demo video attached https://vimeo.com/1158672074?share=copy&fl=sv&fe=ci
📂 GitHub: https://github.com/santosh1231we/Cv-game-controller

Built as part of Science Quest 2026.