Study: Mapping DSP Properties to Niagara Systems in Unreal Engine (2023)
By Timothy Gmeiner
About: This project supports my current research and artistic interest in chromesthesia (sight-sound synesthesia). Specifically, this project lends to research on subjective vs. objective perception of visualized sonic and musical properties by mapping amplitude and frequency output from a fixed Shephard’s tone to various particle system parameters in Unreal Engine. I chose Unreal’s Niagara System because I particle system’s offer a nuanced, detailed and 3-dimensional way to simulate the amplitudal, spatial and timbral qualities of time-varying signals. The above video illustrates the reasoning and process of these mappings.
Mappings:
Emitter state - life cycle of the particle is guided by the system. This allows multiple emitters to be spawned, each with their own unique particles (with unique lifespans, behaviors, meshes, etc)
Emitter Spawn Rate is affected by a non-real-time amplitude analysis user parameter. Amplitude has been normalized from 0 to 1. This dynamic value is multiplied by a set number and plugged into the value of this user parameter. This value constantly changes and therefore the spawn rate of this particle emitter does as well. As a result, when the amplitude from our descending sinusoid decays or drops, so will the spawn rate. As amplitude regains through a slow or sudden attack, the emitter spawn rate will increase.
Vortex Force - This module injects a velocity around a vortex axis. We saw in Sturn’s paper that velocity = Energy / Plank’s Constant. Since, per Debroglie’s Hypothesis, a particle’s energy is mapped to a sinusoid’s frequency, the amount of velocity added to the Vortex Force is determined by the dynamically descending frequency index multiplied by a static amount. This results in a slow-down of rotation as the sinusoid glides down the frequency spectrum.
Gravity Force - Applying a negative acceleration value to the z-axis of Gravity Force will result in particles falling downward. Frequency index was mapped to this parameter with conditions. The highest frequency index recognized in this example is 76, so a static subtraction operator of 76 was factored in, which was then multiplied by 1.5. The idea here is that as the sinusoid travels down the frequency spectrum, the particles ‘energy’ will accelerate downward faster. This is more so a metaphor than a real-life occurrence since a particle’s loss of energy will lower it’s temperature - more so exaggerating the visual relationship between sound wave frequency and particle energy with creative license.
Acceleration Force - mapped Z axis value to frequency index such that higher frequency bins cause faster initial acceleration upward at particle spawn before descending due to gravity force
Spring Force - Applies a spring-like force on the particle. The spring force has been mapped with the loudest recognized frequency index multiplied by its value normalized to 1 and that value multiplied by .0004. As a result, you should notice a flattening of the particle sphere at higher frequencies, specifically as the particles die off. As the frequency descends, as does frequency index, so the spring force lessens and the particle emitter maintains it’s more natural spherical shape of particles. This doesn’t necessarily practically apply to the relationship between sound and particle properties. I don’t have a strong enough grasp on spring force yet to accurate map it, but I liked exploring and the results were intriguing.
Scale mesh allows the particles to exponential shrink to 0 value for the life span of each particle
Color - The color is not yet mapped to anything, however a fade has been applied from bright yellow to dark orange to represent it’s life span.
Inspiration: This study was inspired by Bob Sturm’s “Sonification of Particle Systems” which builds somewhat artistically from the premise of quantum mechanics, which states that a particle can behave as either a particle or a wave. Sturm builds on this by equating a system of particles to a complex set of dynamic waveforms with the purpose of sonifying the various activities of particles (generations, movements, collisions, etc.) into a composed set of sinusoids. The goal in his research is to create a composition that uses properties of quantum physics to dictate a practical compositional structure with recognizable harmonic content and vice versa, use the results of this composition to engage conversation about the behaviors of particles. All this is directed at physics and music students alike, with the hope that each might find unexplored relationships between both fields and incite cross-discipline collaboration. My study inverts sonic and visual qualities such that the sinusoids of complex waveforms might be applied to particle parameters.