A digital intermedia "instrument" that I've been working on lately, created in Max (a visual programming language) - version 1.1! The audio piece used is The Subtle From The Gross from my upcoming solo album Lingua Arcana, and the videos are snippets of the longer videos I filmed in Limerick, Galway and Cashel, Ireland (2023). Detailed description below:

One of the aims of my PhD research is to create an interactive intermedia experience and/or a mechanism that enables the visuals, sound, music and text to freely and fluidly transform into and merge with one another, in long (endless?) streams of iterations, converting back and forth into each other. As a result, the media would transcend their initial forms, and generate new, hybrid, unnamed media, or rather modalities, that are not visuals, sound, music, text, etc. from which they initially arose but all of those at once and nothing of those.

Using Max, I managed to create a multi-modal piece (or rather an instrument) which, to a certain extent, demonstrates this intermedia-like fluidity between sound, music, visuals and text/lyrics (shown in the video in the presentation mode only, i.e. without the processes behind it).

The piece/instrument consists of a playlist with 3-9 second long manually interchangeable videos (7 for now), and an audio track, separate from the videos. Only 1 audio track can be used at the time for now, until I find a practical way of controlling both the video and audio playlists via the same laptop keyboard or an external controller. The hue, saturation and overall visual content of the video playlist output are directly connected to and in turn altered by the changes/fluctuations in the frequency spectrum and amplitude of the audio.
The audio track is uploaded twice: one copy is connected with the frequency spectrum analysis outputs, and the other with the amplitude analysis outputs.

When the audio starts playing, the real-time detected frequency spectrum numerical values — outputting a number when a pitch/note is detected — are converted into a real-time visualization of the spectrum (the white lines moving in and out of the black background). A frequency shift function is included as well, so one can increase or decrease the frequency/pitch of the audio in real time (to the extremes of both positive and negative values, which gives interesting/unexpected results in both the audio and video outputs). The frequency can thus be manipulated with the frequency shift function, the result of which is shown in the real-time generated frequency spectrum visualization window. The frequency spectrum can be additionally manipulated by moving the frequency band sliders up and down (20 Hz-20 kHz) in the graphic equalizer, which further affects the audio output and through it the frequency spectrum visualization. The real-time detected frequency/pitch numerical values are connected with the Saturation dial in the HUSALIR (Hue, Saturation and Luminance) plug-in, so as the frequencies/pitches change, so does the saturation.
The frequency spectrum real-time detected pitches/notes are also converted into a real-time visualization of the changes in saturation and luminance of a monochromatic panel (blue).

The amplitude numerical values, i.e. the real-time detected amplitude peaks, are connected with the Hue dial in the HUSALIR plug-in, so as the amplitude changes so does the hue. The amplitude numerical values are also converted into a real-time visualization of the changes in saturation and luminance of another monochromatic panel (yellow).

The frequency spectrum visualization (white lines on the black background) is connected with the video playlist output (not visible in the presentation mode) so that they are blended/intertwined in the final audio-visual output. Importantly, it is possible to change the blending modes of this output in many different combinations, i.e. how one visual output (frequency spectrum visualization) affects/filters through the other (video playlist) in real time, affecting parameters such as hue, saturation, luminance and others.

The HUSALIR plug-in is thus connected with the final audio-visual output, in which we see the frequency spectrum visualization and the video playlist output being filtered through one another continuously in real time, affected by the changes in hue and saturation caused by the changes in the audio amplitude and frequency spectrum respectively. The final audio-visual output is also affected in real time by the changes made in the EQ frequency bands, and the frequency shifts, and this is both audible in the sound and visible in the visuals.

This is how audio/text becomes video in this piece, and how the video is affected/controlled by the changes in the audio/text.

Watch the videos below to learn how (I learned how) to build some of the parts of the intermedia "instrument": Max 8 Tutorial #33: Audio Becomes Video
Max 8 Tutorial #34: Video to Audio to Video To Audio To Video...
Audio Reactive: Amplitude to Color - Audiovisual Max/MSP Tutorial for Beginners