Projects | Haokun Wang

Fiery Hands: Thermal-Tactile Glove for VR Object Manipulation

Fri, 11 Oct 2024 00:00:00 +0000

Overview

How do you make a virtual fire feel real on your hands? Fiery Hands answers that question with a custom wearable thermal glove that delivers localized thermal and tactile sensations to the palm and all five fingertips — without blocking the hand or preventing natural object manipulation in VR.

Published at ACM UIST 2024 (the premier venue for novel interactive systems), this project represents a step change in how XR systems can deliver believable thermal touch.

The Problem

Existing haptic gloves either:

Cover the inner palm and fingertip surfaces, blocking touch and dexterity, or
Place thermal actuators only on the back of the hand, limiting localized feedback

The challenge: thermal actuators are physically large (Peltier modules), slow (seconds to change temperature), and need direct skin contact. Placing enough of them to cover a hand while preserving freedom of movement seemed contradictory.

Research Question: Can we achieve the perception of localized thermal feedback across the full hand using fewer actuators cleverly placed on non-obstructive body sites?

Research Approach

We leveraged two perceptual phenomena from psychophysics:

Thermal Referral — the brain attributes a thermal sensation to a nearby tactile stimulus site, not the actual thermal source. Heat felt elsewhere “moves” to where you’re touching.
Tactile Masking — a vibrotactile cue can suppress or redirect the perceived location of a thermal stimulus.

By combining strategically placed Peltier actuators on the outer palm and back of fingers with vibrotactile motors at fingertip contact points, we could generate perceived thermal sensations at the fingertips without physically touching them.

System Design

Hardware

Thermal actuators: 4 custom-fabricated Peltier modules (30 × 30 mm) mounted on the outer palm and finger dorsal surfaces
Tactile actuators: 5 coin-type LRA vibration motors placed at the inner fingertip
Controller: Arduino Mega with custom power amplifier board; Bluetooth LE to PC
Glove substrate: Thin spandex with 3D-printed actuator mounts — allows full grip

Unity VR Integration

Built in Unity 2022 LTS with OpenXR / XR Interaction Toolkit
Custom C# HapticFeedbackManager subscribes to XR physics collision events and maps contact surface temperature to actuator commands
Real-time thermal rendering: fire = sustained warm + rhythmic vibration; ice = sustained cool + gentle pulse; metal = rapid ramp-up on contact
Deployed on Meta Quest 2 via Quest Link (PC-tethered for full Peltier power budget)

User Evaluation

Study 1 — Thermal Localization

N = 12 participants, within-subject design
Task: identify which finger perceived the thermal stimulus while only the dorsal actuators were active
Conditions: palm-only thermal, 4× Peltier positions × 3 temperature levels (warm/hot/neutral)
Measure: accuracy of localization, JND (just-noticeable difference)

Study 2 — VR Interaction Plausibility

N = 16 participants
Task: interact with three virtual objects (glowing coal, ice block, metal rod) and rate realism
Conditions: thermal-only, tactile-only, thermal+tactile (Fiery Hands), and no-feedback baseline
Measures: NASA-TLX, immersion subscale, perceived temperature match (7-pt Likert)

Results & Key Findings

Localization accuracy: 84% — participants correctly identified the stimulated finger using only dorsal Peltier placement, validating the thermal referral strategy
Plausibility rating of thermal+tactile condition was significantly higher than any single-modality condition (F(3,45)=18.4, p<.001, η²=0.55)
Users reported the coal interaction as “surprisingly convincing” — qualitative themes: warmth buildup over time felt organic, not mechanical
Power consumption reduced by 60% vs. placing individual Peltiers at each fingertip while achieving comparable perceptual quality

Impact

📄 Published: ACM UIST 2024 — Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology
DOI:
Inspired follow-on work on thermally-integrated wearables for extended wear XR sessions

Let It Snow: Cross-Modal Cold & Touch for VR Snowfall

Wed, 15 May 2024 00:00:00 +0000

Overview

Let It Snow is a hands-free, wearable-free haptic experience: users hold their bare hands over a custom mid-air display that simultaneously fires focused ultrasound pressure points and directed cold airflow to simulate individual snowflakes landing — or rain drops splattering — on their palms.

Published in ACM IMWUT 2024 (Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies), the project explores how cross-modal cold–tactile pairing creates emergent sensory illusions greater than either cue alone.

The Problem

Simulating precipitation in VR is a classic immersion gap. Visually, snow and rain can look photorealistic. But without feeling the cold, the wet, the gentle impact — users never quite believe it. Existing approaches require worn devices, which break the “bare hand in the weather” fantasy entirely.

Core Questions:

Can cold airflow and ultrasound pressure co-localize in mid-air to synthesize a snowflake or raindrop percept?
Do cold and tactile cues mask each other, or can they be independently perceived at the same skin location?
How should aggregated stimuli be rendered for heavy snowfall / rainfall?

Research Approach

We drew on cross-modal sensory integration theory: cold and tactile channels are processed by separate neural pathways (thermoreceptors vs. mechanoreceptors), so two signals can coexist without mutual interference — unlike, say, two sounds at the same frequency.

Key hypothesis: a brief cold puff + simultaneous pressure focus = snowflake percept; a sharp cold burst + faster pressure = raindrop percept.

We also designed an aggregated haptic scheme for particle-dense scenes: rather than rendering every particle individually (physically impossible), we modulate cold intensity and pressure density proportionally to particle count, exploiting temporal summation in both sensory channels.

System Design

Hardware

Cold array: 6 Peltier modules (20 × 20 mm) mounted in a ring, each with a micro-fan to direct cold air toward the focus point; temperature range: 5°C–15°C above ambient
Ultrasound haptic display: Ultrahaptics STRATOS Inspire — 256 transducers at 40 kHz, creating mid-air pressure foci up to 200 mN at distances up to 22 cm
Depth tracking: Intel RealSense D435 hand tracking, integrated into Unity for palm position → focus point mapping
Control PC: Custom C++ driver for thermal timing; Unity handles audio, visuals, and hand tracking

Unity VR Integration

Built in Unity 2021 LTS, standalone VR scene with Oculus Integration SDK
Particle system drives two managers:
- SnowRenderer: handles visual particles with collision callbacks to trigger haptic events
- HapticAggregator: accumulates per-frame particle counts, applies transfer function to Peltier intensity and ultrasound amplitude
Snowflake percept: 150 ms cold puff + 40 Hz pressure burst; Raindrop: 60 ms sharp cold + 200 Hz single-pulse
Scene contains interactive environments: snowy mountain valley, rainstorm on a city rooftop

User Evaluation

Perceptual Study — Cold × Tactile Independence

N = 14 participants
Design: 2 (cold present/absent) × 2 (tactile present/absent) × 5 repetitions
Measure: detection accuracy per modality, reported interference rating
Finding: No significant cross-modal masking — participants detected cold and tactile independently (d’ > 2.5 for both modalities combined)

Experience Study — Aggregated Rendering Comparison

N = 20 participants, within-subject
Conditions: (1) no haptics, (2) tactile-only, (3) cold-only, (4) Snow (cold+tactile sparse), (5) Snow (cold+tactile aggregated)
Measures: presence subscale (IPQ), realism rating, preference ranking
Task: 3-minute free exploration of snowy mountain scene, 3-minute rainstorm scene

Results & Key Findings

Aggregated scheme rated significantly more realistic than sparse individual-particle scheme (p<.01) for heavy snowfall
Cold+tactile combination rated +1.8 points on 7-pt presence scale vs. tactile-only (p<.001)
18/20 participants preferred the full cross-modal condition; primary qualitative theme: “it actually felt cold and real, like being outside”
System achieved stable cold delivery at ±0.3°C variance across a 10-minute continuous session

Impact

📄 Published: ACM IMWUT 2024 — Proc. ACM Interact. Mob. Wearable Ubiquitous Technol.
DOI:
Framework for aggregated haptic rendering has been adopted in follow-on multi-particle VR haptics research

Thermal Masking: When the Illusion Takes Over the Real

Sat, 11 May 2024 00:00:00 +0000

Overview

Thermal Masking is a newly characterized perceptual illusion: when a vibrotactile stimulus is applied near a thermal stimulus, the perceived location of warmth completely jumps to the tactile site — the original thermal signal vanishes from conscious perception. This “masking” is distinct from previously known thermal referral and has profound implications for wearable haptic design.

Published at ACM CHI 2024 (the flagship venue for human–computer interaction research), this work provides the first systematic characterization of thermal masking on the human arm.

The Problem

Thermal feedback in wearables is expensive: Peltier modules are bulky, power-hungry, and slow. Covering a large body area (like the back or full arm) with thermal actuators is impractical. Prior work showed that thermal sensations can be “referred” — stretched toward a tactile stimulus. But we suspected a more dramatic effect existed.

Hypothesis: A single vibrotactile cue could completely suppress the original thermal percept, not just shift it — enabling sparse thermal + dense tactile arrays to cover large body areas affordably.

Research Approach

Three controlled psychophysical experiments on the forearm, each systematically varying one factor:

Experiment	Manipulated Variable	Key Question
1	Temperature level	Does masking occur more at warm vs. hot vs. cold?
2	Thermal-to-tactile distance	How far can masking propagate?
3	Actuator placement (same side vs. opposite side of arm)	Does masking cross body-part boundaries?

Participants reported where they felt the temperature (thermal site, tactile site, or both) on each trial. Masking was defined as reporting only the tactile site despite the thermal actuator being active elsewhere.

Apparatus

Thermal actuator: Single Peltier module (40 × 40 mm), range 20°C – 45°C (cold, neutral, warm, hot conditions)
Tactile actuator: ERM (eccentric rotating mass) motor, 180 Hz, 1.5 G amplitude
Placement rig: 3D-printed sliding rail on the forearm allowing 2–24 cm inter-actuator distance
All stimuli were synchronized via Arduino with 1 ms timing precision

User Evaluation

Total N = 48 participants (16 per experiment), all naïve to the hypothesis
Design: fully within-subject with counterbalanced ordering
Trial structure: 3 s baseline → simultaneous thermal + tactile onset for 5 s → localization report → 30 s ISI for thermal recovery
Measures: localization accuracy (thermal site / tactile site / both), response time, confidence rating

Results & Key Findings

Experiment 1 — Temperature Level:

Thermal masking rate: warm: 73%, hot: 41%, cold: 38%
Warm conditions produced significantly higher masking than hot or cold (χ²(2)=24.3, p<.001)
Implication: warm stimulation (≈35–38°C) is the optimal operating zone for masking-based designs

Experiment 2 — Distance:

Masking persisted up to 24 cm from the thermal actuator — nearly the full forearm length
Masking rate decayed logarithmically with distance (r²=0.91)
Practical finding: one thermal module can plausibly cover the entire forearm with tactile array assist

Experiment 3 — Opposite Side:

Masking also occurred when the tactile actuator was placed on the opposite side of the arm (dorsal vs. volar)
Rate: 58% — lower than same-side but still above chance (p<.001)
Opens door to through-limb sensing designs (e.g., armband with actuators only on outer surface)

Design Implications

These findings directly shaped the actuator placement strategy in and : by exploiting thermal masking, both projects placed thermal actuators exclusively on non-obstructive surfaces while delivering perceived localized warmth at the inner contact points.

Impact

📄 Published: ACM CHI 2024 — Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems
DOI:
Citation: Haokun Wang, Yatharth Singhal, Hyunjae Gil, Jin Ryong Kim. “Thermal Masking: When the Illusion Takes Over the Real.” CHI ‘24.

Fabric Thermal Display: Ultrasound-Heated Wearable for VR

Fri, 01 Sep 2023 00:00:00 +0000

Overview

Standard thermal wearables rely on Peltier thermoelectric modules — rigid, thick, and power-intensive. Fabric Thermal Display takes a different approach: weave thermally-conductive materials (copper, aluminum mesh) into a fabric glove and excite them with focused ultrasonic waves. The friction heats the conductive fibers, delivering warmth through the fabric itself.

Published at IEEE ISMAR 2023 (IEEE International Symposium on Mixed and Augmented Reality), this project delivers a proof-of-concept for ultrasound-driven textile thermal displays and demonstrates their use in VR object interaction scenarios.

The Problem

Peltier-based thermal gloves work, but they have hard constraints:

Thickness: modules are 3–5 mm, stiff, and change the hand’s natural shape
Power: each Peltier draws 3–10 W continuously
Scalability: covering all fingers requires 5+ modules, complicated wiring, and custom PCBs

Could fabric itself become the thermal actuator — flexible, lightweight, and able to conform to any body shape?

Research Question: Which fabric materials respond best to 40 kHz ultrasonic excitation, and can combinations with conductive materials achieve perceptually meaningful warmth for VR?

Research Approach

We started with a material science study before touching user testing:

Characterization phase: apply ultrasonic energy to 5 fabric types (polyester, cotton, nylon, Lycra, carbon-fiber blend), measure temperature rise over 30 s at three amplitude levels
Composite phase: integrate the best fabric (polyester) with copper mesh and aluminum foil, compare thermal curves
Perceptual phase: user study on thermal detection and level identification with the best material combination
Application phase: integrate into a glove form factor, demonstrate VR use cases

System Design

Ultrasound Setup

Ultrasound driver: Ultrahaptics STRATOS board, 40 kHz carrier, amplitude-modulated 0–100%
Focus geometry: single focal point directed at 15 cm standoff, corresponding to palm contact zone of glove
Thermal measurement: FLIR A315 thermal camera captured surface temperature maps at 9 Hz

Fabric Samples

Material	Peak Temp Rise (100% amp, 30s)	Flexibility	Notes
Polyester	+18.4°C	High	Best performance
Cotton	+9.1°C	High	Poor — high thermal mass
Nylon	+12.3°C	Medium	Acceptable
Lycra	+7.8°C	Very high	Too low output
Carbon fiber	+21.1°C	Low	Best thermal, too stiff

Winner: Polyester + Aluminum — +22.6°C peak, flexible, washable

Glove Design

Polyester base with 0.1 mm aluminum foil laminate on palm zone
Total glove weight: 28 g (vs. 95 g for Peltier glove baseline)
No wiring — ultrasound is contactless

Unity VR Integration

Unity 2021 LTS + Oculus Integration SDK (Quest 2)
Custom FabricHapticManager: maps virtual object surface temperature to ultrasound amplitude via lookup table
Demonstrated VR scenarios: picking up hot metal ingot, holding warm beverage, touching cold ice sculpture
Haptic rendering loop runs at 90 Hz, matching display refresh

User Evaluation

Study 1 — Detection Thresholds

N = 12 participants
Task: signal detection (yes/no) across 5 amplitude levels, 2-AFC paradigm
Measure: warm detection threshold (WDT)
Result: mean WDT = 38% amplitude (≈ +7.2°C skin surface delta)

Study 2 — Level Identification (Thermal JNDs)

N = 16 participants
Task: categorize warmth into 4 levels (none, low, medium, high) from ultrasound-heated glove
Condition: fabric-only vs. fabric+copper vs. fabric+aluminum
Result: fabric+aluminum achieved 78% accuracy for 4-level identification, significantly outperforming fabric-only (54%, p<.01) and fabric+copper (66%, p<.05)

Results & Key Findings

Polyester is the optimal base fabric for ultrasonic thermal generation among tested materials
Aluminum lamination provides +4.2°C improvement over copper at the same power setting
Users could reliably distinguish 4 thermal levels through the glove, meeting the threshold needed for meaningful VR thermal rendering
No participant reported discomfort over 20-minute continuous wear sessions

Impact

📄 Published: IEEE ISMAR 2023 — IEEE International Symposium on Mixed and Augmented Reality
DOI:
The material findings fed directly into the Fiery Hands glove substrate design and the broader thermal-wearables research program at MI Lab

Mid-Air Thermo-Tactile Fire: Ultrasound Haptic Display for VR

Wed, 01 Sep 2021 00:00:00 +0000

Overview

Imagine reaching toward a virtual campfire and actually feeling the heat wash over your hands — no gloves, no controllers, nothing on your skin. Mid-Air Thermo-Tactile Fire is a proof-of-concept system that delivers both thermal warmth and vibrotactile pressure to a free hand hovering above a custom device, using a combination of heated airflow channels and a 40 kHz ultrasound haptic array.

Published at ACM VRST 2021 (ACM Symposium on Virtual Reality Software and Technology), this was the first system to simultaneously characterize thermo-tactile mid-air feedback thresholds and demonstrate them in a VR fire interaction scenario.

The Problem

Mid-air haptics (ultrasound) had proven that focused pressure can be delivered without contact. Thermal mid-air feedback existed in industrial settings (heat lamps). But simultaneously combining both — localized, controllable, synchronized — for real-time VR had not been demonstrated.

Key unknowns at project start:

What temperature range can be achieved mid-air at realistic interaction distances (15–25 cm)?
Does the ultrasonic pressure signal interfere with thermal perception (or vice versa)?
What warm detection threshold (WDT) and heat-pain threshold (HPDT) apply to mid-air vs. contact thermal stimulation?

System Design

Hardware Architecture

Ultrasound display: 16×16 transducer array (256 elements), 40 kHz carrier, capable of focusing pressure at 10–25 cm above surface
Thermal channel: open-top acrylic chamber with 4 heating coils; a low-speed centrifugal fan directs warm air up through the focus zone
Temperature control: PID loop via Arduino — thermocouple at the focal plane feeds back to heater PWM, ±1°C stability
Integration: ultrasound focus point and warm airflow column co-aligned within ±5 mm

Measured System Specs

Parameter	Value
Peak achievable temperature at focal plane	54.2°C
Ultrasound pressure at focus	3.43 mN (100 Hz, 12 mm radius)
Temperature stability (mean error)	0.25% over 10 min
Interaction distance range	12–22 cm

Unity VR Integration

Unity 2020 LTS with SteamVR / OpenVR SDK (HTC Vive)
Custom C# bridge communicates over USB serial to Arduino controller
VR scene: virtual campfire with particle system; hand proximity triggers thermal ramp (further = cooler, closer = warmer) while fire flicker drives vibrotactile modulation at 4–12 Hz
Thermal latency from Unity event to onset at skin: ~120 ms (dominated by airflow thermal inertia)

User Evaluation

Threshold Study — WDT and HPDT

N = 14 participants
Protocol: method of limits (ascending/descending); 5 trials per direction, 3 interleaved staircases
Conditions: mid-air thermal only (no ultrasound) vs. mid-air thermal + ultrasound (thermo-tactile)
Measures: WDT (°C), HPDT (°C), response time to first detection

Haptic Pattern Recognition Study

N = 14 participants (same cohort, separate session)
Task: identify 4 spatial haptic patterns (dot, ring, horizontal bar, vertical bar) presented mid-air
Conditions: non-thermal (room temp) vs. thermal-on (heated airflow active)
Measure: identification accuracy, confusion matrix

VR Experience Study

N = 10 participants
Task: 5-minute campfire scene; ratings on warmth realism, presence, comfort

Results & Key Findings

WDT: mean 32.8°C (SD=1.12) — consistent with contact-based thermal WDT literature (validates mid-air stimulation as perceptually equivalent)
HPDT: mean 44.6°C (SD=1.64) — also matches contact norms; no elevated pain threshold from airflow delivery
Pattern accuracy: 98.1% (non-thermal) vs. 97.2% (thermal) — no significant degradation (p=.38); thermal channel does not interfere with tactile perception
Thermo-tactile condition received significantly higher VR realism ratings than tactile-only (p<.05)

Lessons & Evolution

This project established the core technical finding that underpins the entire MI Lab thermal haptics research line: thermal and tactile cues can coexist mid-air without masking each other, enabling richer multi-modal VR experiences. Every subsequent project (Snow, Fabric Thermal Display, Fiery Hands) built on these baseline thresholds and the dual-channel architecture proven here.

Impact

📄 Published: ACM VRST 2021 — Proceedings of the 27th ACM Symposium on Virtual Reality Software and Technology
DOI:
First paper characterizing mid-air thermo-tactile thresholds; foundational reference for the lab’s subsequent wearable thermal haptics work