Seeing Music - devel

System Architecture

  • The audio-processor is a black-box that analyzes incoming sounds and generates OSC messages. A basic set of messages and parameters needs to be specified (those can be extended in the future).
    • volume / energy
    • tempo / rhythm
    • pitch / timbre
    • frequency spectrum and frequency spread
    • etc
    • additional msg (initial and periodical updates) configuration (number of channels, samplerate, etc)
  • Video-generator has an internal pipeline. e.g. shape-generator, texture-overlay, zoom, filter (eg. blur, brightness/contrast), encode.
    • every element in the pipeline may have configuration parameters
    • these parameters are either fixed or can be modified dynamically (interact with input)
    • dynamic parameters can be set by OSC (either from the sound-analyzer or via user-interaction)
    • a config file or database (aka preset) describes the pipeline and defines a map of parameters.
  • User interaction - also sends OSC messages to the video-generator (depending on the config, these are internally forwarded to one or more elements in the pipeline)
    • interaction can be via HID (mouse, keyboard, etc)
    • Joystick, Wii-mote, etc
    • Mixing-desk (faders or knobs)
    • advanced systems such as camera → processing → OSC

Implementation

  • Audio-processing: e.g. Max/MSP ; pure-data
  • Video-generation: Max/Jitter, processing, gstreamer, pure-data,…
 
wiki/seeing_music.txt · Last modified: 04.05.2012 11:43 by rgareus