Mid-Air | Haokun Wang

Let It Snow: Cross-Modal Cold & Touch for VR Snowfall

Wed, 15 May 2024 00:00:00 +0000

Overview

Let It Snow is a hands-free, wearable-free haptic experience: users hold their bare hands over a custom mid-air display that simultaneously fires focused ultrasound pressure points and directed cold airflow to simulate individual snowflakes landing — or rain drops splattering — on their palms.

Published in ACM IMWUT 2024 (Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies), the project explores how cross-modal cold–tactile pairing creates emergent sensory illusions greater than either cue alone.

The Problem

Simulating precipitation in VR is a classic immersion gap. Visually, snow and rain can look photorealistic. But without feeling the cold, the wet, the gentle impact — users never quite believe it. Existing approaches require worn devices, which break the “bare hand in the weather” fantasy entirely.

Core Questions:

Can cold airflow and ultrasound pressure co-localize in mid-air to synthesize a snowflake or raindrop percept?
Do cold and tactile cues mask each other, or can they be independently perceived at the same skin location?
How should aggregated stimuli be rendered for heavy snowfall / rainfall?

Research Approach

We drew on cross-modal sensory integration theory: cold and tactile channels are processed by separate neural pathways (thermoreceptors vs. mechanoreceptors), so two signals can coexist without mutual interference — unlike, say, two sounds at the same frequency.

Key hypothesis: a brief cold puff + simultaneous pressure focus = snowflake percept; a sharp cold burst + faster pressure = raindrop percept.

We also designed an aggregated haptic scheme for particle-dense scenes: rather than rendering every particle individually (physically impossible), we modulate cold intensity and pressure density proportionally to particle count, exploiting temporal summation in both sensory channels.

System Design

Hardware

Cold array: 6 Peltier modules (20 × 20 mm) mounted in a ring, each with a micro-fan to direct cold air toward the focus point; temperature range: 5°C–15°C above ambient
Ultrasound haptic display: Ultrahaptics STRATOS Inspire — 256 transducers at 40 kHz, creating mid-air pressure foci up to 200 mN at distances up to 22 cm
Depth tracking: Intel RealSense D435 hand tracking, integrated into Unity for palm position → focus point mapping
Control PC: Custom C++ driver for thermal timing; Unity handles audio, visuals, and hand tracking

Unity VR Integration

Built in Unity 2021 LTS, standalone VR scene with Oculus Integration SDK
Particle system drives two managers:
- SnowRenderer: handles visual particles with collision callbacks to trigger haptic events
- HapticAggregator: accumulates per-frame particle counts, applies transfer function to Peltier intensity and ultrasound amplitude
Snowflake percept: 150 ms cold puff + 40 Hz pressure burst; Raindrop: 60 ms sharp cold + 200 Hz single-pulse
Scene contains interactive environments: snowy mountain valley, rainstorm on a city rooftop

User Evaluation

Perceptual Study — Cold × Tactile Independence

N = 14 participants
Design: 2 (cold present/absent) × 2 (tactile present/absent) × 5 repetitions
Measure: detection accuracy per modality, reported interference rating
Finding: No significant cross-modal masking — participants detected cold and tactile independently (d’ > 2.5 for both modalities combined)

Experience Study — Aggregated Rendering Comparison

N = 20 participants, within-subject
Conditions: (1) no haptics, (2) tactile-only, (3) cold-only, (4) Snow (cold+tactile sparse), (5) Snow (cold+tactile aggregated)
Measures: presence subscale (IPQ), realism rating, preference ranking
Task: 3-minute free exploration of snowy mountain scene, 3-minute rainstorm scene

Results & Key Findings

Aggregated scheme rated significantly more realistic than sparse individual-particle scheme (p<.01) for heavy snowfall
Cold+tactile combination rated +1.8 points on 7-pt presence scale vs. tactile-only (p<.001)
18/20 participants preferred the full cross-modal condition; primary qualitative theme: “it actually felt cold and real, like being outside”
System achieved stable cold delivery at ±0.3°C variance across a 10-minute continuous session

Impact

📄 Published: ACM IMWUT 2024 — Proc. ACM Interact. Mob. Wearable Ubiquitous Technol.
DOI:
Framework for aggregated haptic rendering has been adopted in follow-on multi-particle VR haptics research

Mid-Air Thermo-Tactile Fire: Ultrasound Haptic Display for VR

Wed, 01 Sep 2021 00:00:00 +0000

Overview

Imagine reaching toward a virtual campfire and actually feeling the heat wash over your hands — no gloves, no controllers, nothing on your skin. Mid-Air Thermo-Tactile Fire is a proof-of-concept system that delivers both thermal warmth and vibrotactile pressure to a free hand hovering above a custom device, using a combination of heated airflow channels and a 40 kHz ultrasound haptic array.

Published at ACM VRST 2021 (ACM Symposium on Virtual Reality Software and Technology), this was the first system to simultaneously characterize thermo-tactile mid-air feedback thresholds and demonstrate them in a VR fire interaction scenario.

The Problem

Mid-air haptics (ultrasound) had proven that focused pressure can be delivered without contact. Thermal mid-air feedback existed in industrial settings (heat lamps). But simultaneously combining both — localized, controllable, synchronized — for real-time VR had not been demonstrated.

Key unknowns at project start:

What temperature range can be achieved mid-air at realistic interaction distances (15–25 cm)?
Does the ultrasonic pressure signal interfere with thermal perception (or vice versa)?
What warm detection threshold (WDT) and heat-pain threshold (HPDT) apply to mid-air vs. contact thermal stimulation?

System Design

Hardware Architecture

Ultrasound display: 16×16 transducer array (256 elements), 40 kHz carrier, capable of focusing pressure at 10–25 cm above surface
Thermal channel: open-top acrylic chamber with 4 heating coils; a low-speed centrifugal fan directs warm air up through the focus zone
Temperature control: PID loop via Arduino — thermocouple at the focal plane feeds back to heater PWM, ±1°C stability
Integration: ultrasound focus point and warm airflow column co-aligned within ±5 mm

Measured System Specs

Parameter	Value
Peak achievable temperature at focal plane	54.2°C
Ultrasound pressure at focus	3.43 mN (100 Hz, 12 mm radius)
Temperature stability (mean error)	0.25% over 10 min
Interaction distance range	12–22 cm

Unity VR Integration

Unity 2020 LTS with SteamVR / OpenVR SDK (HTC Vive)
Custom C# bridge communicates over USB serial to Arduino controller
VR scene: virtual campfire with particle system; hand proximity triggers thermal ramp (further = cooler, closer = warmer) while fire flicker drives vibrotactile modulation at 4–12 Hz
Thermal latency from Unity event to onset at skin: ~120 ms (dominated by airflow thermal inertia)

User Evaluation

Threshold Study — WDT and HPDT

N = 14 participants
Protocol: method of limits (ascending/descending); 5 trials per direction, 3 interleaved staircases
Conditions: mid-air thermal only (no ultrasound) vs. mid-air thermal + ultrasound (thermo-tactile)
Measures: WDT (°C), HPDT (°C), response time to first detection

Haptic Pattern Recognition Study

N = 14 participants (same cohort, separate session)
Task: identify 4 spatial haptic patterns (dot, ring, horizontal bar, vertical bar) presented mid-air
Conditions: non-thermal (room temp) vs. thermal-on (heated airflow active)
Measure: identification accuracy, confusion matrix

VR Experience Study

N = 10 participants
Task: 5-minute campfire scene; ratings on warmth realism, presence, comfort

Results & Key Findings

WDT: mean 32.8°C (SD=1.12) — consistent with contact-based thermal WDT literature (validates mid-air stimulation as perceptually equivalent)
HPDT: mean 44.6°C (SD=1.64) — also matches contact norms; no elevated pain threshold from airflow delivery
Pattern accuracy: 98.1% (non-thermal) vs. 97.2% (thermal) — no significant degradation (p=.38); thermal channel does not interfere with tactile perception
Thermo-tactile condition received significantly higher VR realism ratings than tactile-only (p<.05)

Lessons & Evolution

This project established the core technical finding that underpins the entire MI Lab thermal haptics research line: thermal and tactile cues can coexist mid-air without masking each other, enabling richer multi-modal VR experiences. Every subsequent project (Snow, Fabric Thermal Display, Fiery Hands) built on these baseline thresholds and the dual-channel architecture proven here.

Impact

📄 Published: ACM VRST 2021 — Proceedings of the 27th ACM Symposium on Virtual Reality Software and Technology
DOI:
First paper characterizing mid-air thermo-tactile thresholds; foundational reference for the lab’s subsequent wearable thermal haptics work