Signie

Role

XR developer

Timeline

March–Jun 2025

Tools
Unity6Unity6
BlenderBlender

Signie is an immersive ASL learning and real-time translation system powered by hand tracking, micro-gestures, and AI feedback.

Focus
Hand TrackingGesture RecognitionMicro-GesturesVoice-to-TextVirtual GuideAnimation State Machine
AWE USA present video
Fullplay video
CHAPTER 01

Ideation

Signie is an immersive ASL learning and real-time translation system powered by hand tracking, micro-gestures, and AI feedback. It evolved from concept validation to interactive learning experiences, and ultimately to AI-glasses-based live translation.

CHAPTER 02

Process

Stage 1 — Prototype: Solve animation of tutors (hand & full body)

1. record hand-guiding / full-body animation, all in Unity

Hand guiding animation
Hand guiding animation
Hand guiding animation with customized hand model
Hand guiding animation with customized hand model
Full body animation recorded by Meta Quest headset
Full body animation recorded by Meta Quest headset
I record via Unity recorder + FBX converter, all in Unity
I record via Unity recorder + FBX converter, all in Unity

2. add bubbles to guide hand movement

Bubble guide 1
Bubble guide 1
Bubble guide 2
Bubble guide 2

Stage 2 — Develop learning and testing function

1. mix words of different levels of familiarity

Pattern of learning and testing with mixed mechanism
Pattern of learning and testing with mixed mechanism
Manager prefab for learning a new word
Manager prefab for learning a new word
Follow the tutor animation to learn
Follow the tutor animation to learn
Test 1 — Hand gesture test
Test 1 — Hand gesture test
Test 2 — Rhythm-game testing of ASL alphabet
Test 2 — Rhythm-game testing of ASL alphabet
Test 3 — Pick correct lines based on animation played by tutor
Test 3 — Pick correct lines based on animation played by tutor

Stage 3 — AI Glasses: Live ASL Translation

1. Micro-gesture input for hands-free system control

Live Translation Pipeline

  • Voice → Text using Wit.ai
  • Text → Sign animation via animation state machine
Live ASL Translation Demonstration
CREDITS

My Contributions

01make hand interaction of learning process

02GameManager which manages the application experience, from learning to playful game testing

03add microgesture to integrate Speech-to-Text driving live translation of the tutor's animation

MORE WORK ↓

Explore more

I AND AI: MIRROR
I AND AI: MIRROR
Immersive AI Installation

I AND AI: MIRROR

AI can be a collaborator or a mirror—mentor, listener, lover. Before AI becomes anything, we assign it an "AI"dentity. What if identity itself is co-authored between "I" and "AI"? I & AI: Mirror is an immersive experience exploring human–AI intimacy. Audiences engage with a generative mirror-self that listens, learns, and reflects their presence, transforming interactions into real-time, shared expressions of identity.

The Shadow of Horizon(Motion Capture)
The Shadow of Horizon(Motion Capture)
Motion Capture

The Shadow of Horizon(Motion Capture)

A comprehensive video production showcasing motion capture and Metahuman animation.

The Tool Box
The Tool Box
AI Shopping Assistant

The Tool Box

AI assistant for customer purchasing