Multisensory | Haokun Wang

Let It Snow: Designing Snowfall Experience in VR

Wed, 15 May 2024 00:00:00 +0000

TL;DR

Stick your bare hand into a VR scene and feel snowflakes land and melt on your palm. Or rain droplets — cold, scattered, with a tiny tactile pulse for each one. Snow is a mid-air haptic system that pairs six Peltier modules with an ultrasound haptic display to project simultaneous cold + touch cues onto your skin without contact. Two studies confirmed that an aggregated haptic scheme (one strong cue summarizing many particles) feels more realistic than rendering every flake individually — a counter-intuitive but durable finding that should change how VR weather is designed.

Role. First author. Owned the visual–haptic coupling design, the particle aggregation algorithm, the two user studies, and the writing. Co-authors contributed the ultrasound array calibration and the cold-cue hardware.


Venue	IMWUT 2024 (ACM IMWUT / Ubicomp)
Team	Me (lead) · Yatharth Singhal · Prof. Jin Ryong Kim (advisor)
Timeline	~10 months — concept, hardware, two studies, journal-length write-up
Tools	Custom Peltier-cooled airflow rig, Ultraleap STRATOS array, Unity 2022, Meta Quest Pro, Python (data)
Methods	Within-subject UX study, realism + presence ratings, qualitative interviews

01 · Context

VR weather is visually good and somatically nothing. Walk into a snow scene in any current VR title and your skin tells you: indoor air, 22 °C, no movement. There’s a sensory gap between what your eyes accept and what your body can confirm.

Existing approaches don’t fix it well:

Wearables (gloves / sleeves). Heavy, charge-hungry, and they break the magic of a bare hand reaching into the scene.
Mid-air ultrasound alone. Gives a tactile pulse but no temperature — a snowflake without the cold reads as a tiny rubber bullet.
Cold-air-only rigs. Give temperature without spatial precision — feels like an air conditioner, not weather.

The gap I wanted to close: cold + touch + spatially registered to where the visual particle is. Without contact. With bare hands.

02 · Approach

I framed the design as a stack of three layers, each independently testable.

Layer 1 — the rig

Six Peltier modules cool a column of air. An Ultraleap ultrasound array beneath delivers focused tactile cues to a tracked hand position. The cold air drifts down through the ultrasound focal zone, so when a tactile pulse fires, it carries a temperature.

Layer 2 — the visual–haptic mapping

Each visible snowflake or raindrop in the VR scene is a particle with a position, velocity, and lifetime. When a particle’s path intersects the user’s palm, the system fires a coupled cue: a short tactile pulse paired with a cold magnitude scaled by particle size. (swap the bracketed values for your measured pulse durations when filling this in.)

Layer 3 — the aggregation question

Real snow doesn’t land flake-by-flake. It lands as a snowfall — many flakes at once, each barely perceptible individually, summing to an unmistakable feeling. Should the haptic rendering be 1-flake-per-pulse, or should it aggregate?

I built three rendering schemes:

Per-particle — one tactile + cold pulse per visible flake.
Sampled — random subsample of flakes generate cues.
Aggregated — cue strength scales with the number of flakes intersecting the hand in a window, but only one cue fires.

03 · User studies

(replace bracketed values with the real numbers from your IMWUT paper.)

Study 1 — modality contribution (N=[XX], within-subject). Visual-only, visual+tactile, visual+cold, visual+cold-tactile, and a no-stimulus baseline. Measured: realism, presence, and qualitative comments.

Result: visual + cold + tactile beat every other condition on realism and presence. Visual + tactile alone was worse than visual-only on realism — confirming that tactile-without-temperature reads as artificial.

Study 2 — rendering scheme (N=[XX], within-subject). Per-particle vs. sampled vs. aggregated, both for snow and rain.

Result: aggregated won on realism for both conditions. Per-particle felt “noisy” or “buzzy” in interviews. Several participants used the word “natural” for aggregated and “robotic” for per-particle.

04 · Outcome

Published in IMWUT — ACM’s premier journal for ubiquitous and wearable interaction.
Two design guidelines extracted for future VR weather systems:
1. Always co-render cold with tactile for “wet” or “frozen” particles.
2. Aggregate, don’t enumerate — a single well-designed cue beats N small cues for ambient sensations.

05 · Reflection

What I’d do differently. Wind. Real snow comes with airflow that telegraphs weather before the first flake lands. The current rig doesn’t move air — adding a small directional fan layer is the obvious next iteration.

Where this goes next. The aggregation finding probably extends well beyond weather — it should hold for any “many-small-particles” sensation: sand, embers, leaves blown across the palm. I’d love to test that.

What this project taught me about experience design. Realism is not fidelity. Per-particle is more faithful to the simulation, and it feels worse. Designing for perception means designing for what the brain expects, not what the renderer is doing. That’s a UX principle, not a haptics principle, and it generalizes everywhere.

Paper, video, and full methodology linked above. If you want to talk haptic experience design — .

Thermal Masking: When the Illusion Takes Over the Real

Sat, 11 May 2024 00:00:00 +0000

TL;DR

This is a perception study, not a product — but it’s the kind of study that unlocks products. We discovered that under the right conditions, a vibration on your arm can completely “steal” a heat sensation from a thermal source up to 24 cm away, even on the opposite side of the arm. For wearable haptic designers, that’s a fundamentally new degree of freedom: the heat source no longer has to live where the heat is felt. Three experiments, 60+ trials per participant, and a clean answer to when the illusion holds and when it breaks.

Role. First author. I designed the three experiments, ran the data collection, did the statistics in R, and led the writing. My collaborators contributed the actuator hardware and theoretical framing.


Venue	CHI 2024 (ACM Conference on Human Factors in Computing Systems)
Team	Me (lead) · Yatharth Singhal · Hyunjae Gil · Prof. Jin Ryong Kim (advisor)
Timeline	~7 months — formative studies, 3 main experiments, paper
Tools	Custom Peltier rig, voice-coil vibrotactile actuators, MATLAB control, R + JASP for stats
Methods	Within-subject psychophysics, mixed-effects models, IRB-approved

01 · Context

If you’re designing a thermal feedback device — a glove, a sleeve, a chair, a controller — the constraint that always wins is where you can mount the thermal cell. Peltiers are bulky. They need heat sinks. They draw current. So you mount them where you have room, and the user feels them where you mounted them. Tough.

But there’s a twin to thermal referral called thermal masking: when a vibrotactile cue is delivered next to a thermal stimulus, sometimes the thermal sensation doesn’t just move — it disappears at the source and reappears entirely at the vibration site. We knew the phenomenon existed in psychophysics literature. What we didn’t know was its envelope — the temperature range, the distance limit, the geometry it tolerates. Without those numbers, no designer can use it.

My question. What’s the design budget? How hot, how cold, how far away can I put the heat source and still make the illusion land?

02 · Approach

I broke the design budget into three orthogonal questions, each one a tightly controlled experiment:

Temperature. Does masking work the same for warm, hot, and cold stimuli?
Distance. How far apart can the thermal and tactile actuators be before the illusion collapses?
Geometry. Does the masking only work on the same side of the limb, or can it cross the arm?

Each experiment ran on the same custom rig — a Peltier and a voice-coil vibrotactile actuator on adjustable tracks along a forearm cradle — so we could isolate one variable at a time.

03 · What we found

Experiment 1 — temperature matters more than expected

Within-subject across warm, hot, and cold conditions. (replace bracketed numbers with the rates from your CHI paper.) Warm stimuli produced masking on [XX%] of trials, dropping to [XX%] for hot and [XX%] for cold. The takeaway for designers: warm is the sweet spot. If you’re trying to evoke “warm cup of coffee” or “ambient sun on skin,” masking is a generous tool. If you’re trying to evoke “open flame” or “ice cube,” you’ll need direct hardware.

Experiment 2 — masking carries 24 cm

Masking remained reliable up to 24 cm of separation between the thermal and tactile sites — roughly elbow to wrist on most adults. Beyond that, the illusion dropped sharply. For a sleeve designer, this means one thermal cell at the elbow can serve a tactile array all the way down the forearm.

Experiment 3 — it works across the arm

The most surprising result. We placed the thermal actuator on the outside of the forearm and the tactile actuator on the inside. Conventional wisdom predicts the illusion shouldn’t work — there’s no shared receptive field. It worked anyway, and was higher under warm conditions than under hot. The arm appears to integrate thermal and tactile signals at a higher level than the local skin patch.

Design takeaway. A wearable doesn’t need thermal cells on every surface. One well-placed warm source plus a tactile array gives you a continuous-feeling thermal field across the limb.

04 · Outcome

Accepted to CHI 2024 — flagship HCI venue.
The numbers from this paper became the design constants for our follow-on projects: Fiery Hands (thermal glove) and the upper-body thermal sleeve study both quote these distance and temperature bounds directly.

05 · Reflection

What I’d do differently. I’d add a temporal dimension. All three experiments measured the illusion at a steady state. In real wearable use, sensations come and go — does masking survive a 0.5 s tactile pulse? A 100 ms pulse? That paper is on my next-year list.

What I underestimated. How willing reviewers would be to accept the cross-arm result. I expected pushback — instead it became the most-cited finding. There’s a lesson there about leading with the most counterintuitive finding rather than burying it in Experiment 3.

What this taught me as a researcher. The most useful HCI papers aren’t the ones that show “we built X.” They’re the ones that quantify the envelope — here’s exactly when X works and when it breaks — because everything anyone builds afterwards uses your numbers. That’s what I tried to do here.

Full methodology and statistics in the paper. Happy to talk through the experimental design — .