VISUAL MUSIC: 
AVIA Translation Project
Dr. Emilio Mendoza Guardia
Universidad Simón Bolívar, Caracas, ZKM - Zentrum für
Kunst und Medientechnologie, Karlsruhe

HOME

The "AVIA Translation" project was started in September, 2007, at the Simon Bolivar University, Caracas by Emilio Mendoza, Professor of Music, with the assistance of graduate student Gabriel Peraza of the Master's Degree Program in Composition. It was funded by the Dean of Research and Development, USB, and it has been developed in conjunction with the ZKM - Zentrum für Kunst und Medientechnologie, Karlsruhe, Germany, in three research internships as GastKünstler (Guest Artist) for the periods from February to August, 2008, (on sabbatical research leave from the USB), three weeks in July, 2009 funded by the USB + ZKM, and three months from October to December, 2013, funded by the Venezuelan research office FONACIT and by the Deutscher Akademischer Austausch Dienst (DAAD).

AVIA stands for "Audio-Visual Integrated Arts" as a fair, balanced name where both domains appear in equal importance. It was proposed as a wide genre that includes particularly visual music. The  AVIA Translation
project focuses on determining the possibilities of translation of time structures between both domains, music and moving visuals.

Title of  Research Project
Development and Perspectives of Visual Music: A historical outline and analysis of translation possibilities of music to moving visuals

General Objective
Produce analytical and artistic knowledge on Visual Music

Specific Objectives
  1. Research bibliographical and audio-visual references
  2. Complete writing a book with the project title
  3. Write a paper entitled “Rhythmic and spatial development in AVIA compositions”
  4. Write music compositions for space and rhythmic translation to visuals.
Author's previous translation attempts
In 1977 the author composed several pieces in a set of jokes and conceptual works by the name of La Caja de Juguetes, (The Box of Toys), treating the translation of musical ideas to other areas of expression such as dance, visuals, narrative and poetry, noise, and other concepts such as imagination, attention and perception, rituals, being and life itself. In the same year, Mendoza produced an interdisciplinary work within the  audiovisual field called Susurro (Whisper), a piece focused on the spatial composition for 6 synthesizers emitting white noise or for mixed choir of six voices as producers of white noise, with a spatial placement from left to right and mono. The author tried to perform it with 6 EMS synthesizers, the "Synthy" suitecase model, for its premiere at the 33rd Hauptarbeitstagung of the Institut für Neue Musik und Musikerziehung, 04/08/1979, Darmstadt, Germany. The six synthesizers were provided by the English company EMS, and the author leased six carousel slide-projectors to provide a visual version of the score. Unable to train six people to play the synthesizers live by producing white noise with filters and volume handling, the concert was finally performed by a group of friends performing the white noise with vocals, mostly medicine students from his college dorm at the Uni-Klinik in Düsseldorf. A sophisticated amplification was set up through six individual channels, assembled and controlled by the sound technician Michael Feller from the Robert Schumann Institut, Düsseldorf. The slide projectors were also discarded because it meant requiring six additional persons dedicated to controlling them live, following the score, and also because of the rental costs and transportation of the five screens for the projection...  a very expensive  production for a music student!

After this concert, the piece was recorded in the sound studios of the Robert Schumann Institute, conducted by Alfredo Rugeles, with the sound engineer Michael Feller, four composers doing each voice together (Ramón Ramos, Alfredo Marcano, Paco Estévez, Emilio Mendoza), and recording all six voice tracks. With this audio, the author concentrated again to run the A/V version of the piece with the six carousel projectors that could automate the change and dimmer of slides, but did not work.

In 1979, the author composed the work Secretos for wind sextet, with a structure exposed only through musical color, in order to translate it to live visuals again, with the same carousel slides equipment but with colors this time and not with space concepts. Secretos was premiered at the Kibbutz Shefayim, Israel, 1980, within the ISCM New World Music Days Festival and the recording was later produced by the WDR with the Düsseldorfer Bläßersextette, Alfredo Marcano, conducting. With recording in hand, much time was consumed with the projectors, painting slides and trying to set the visual score, but with very frustrating results.


Compositional failures of space and color
The two pieces, Susurro and Secretos proved to be complete failures in conveying musical time-structures through space relationships (Susurro) and through musical timbre (Secretos), although the author thoroughly composed the works with these two elements in mind. On listening to the recordings, there was no sense of perceiving structures, no trace in memory, and therefore no understanding of the music heard. That was one of the reasons that moved him to insist on trying to convey the similar time structure through visuals in real time: to help the music to be comprehended through its simultaneous visual version. The title Secretos specifically gives the sense concerning this frutration where color does not function to convey complex structures but just simple contrasts in music. The analysis of Secretos was included at the end of the printed score for the sake of revealing the unperceived work in it, until a visual version could be achieved.

The author did not abandon the A/V translation project completely, involving himself with the EMS Spectron Video Synthesizer in the late seventies, but access to this hardware was very limited and no progress was accomplished. He also worked on different projects dealing with music, space and visuals, (see Pasaje, 1976, Gaudeamus Prize), as well as with music for dance and video productions over the years. In November, 2012, a new version of Susurro was performed in the ISCM 2012 New World Music Days Festival in Antwerp, by the Aquarius Choir, with the inclusion of refinements in the “z” axis of the previous score.

New attempt since 2006 - AVIA Project
With this frustration kept at the back of his mind for several years, the author began the translation project again with a lecture within a science/art exhibition (VI Salón Arte-Ciencia, USB, 27-10-2006) on color-color translation, and then registered the AVIA Project at the Dean of Research and Development, USB in 2007. He realized how gradually the importance and omnipresence of audiovisual media had taken our lives and the strength of the trend of musical art to fold into the visual domain definitely. This was accounted by the technological development in this field (Internet, iPod, cell phones), reaching the people in the streets and other social A/V-phenomena as Napster, YouTube, iTunes, so he decided to use his sabbatical in 2008 to study the Music to Visuals translation thoroughly, reconnecting his previous frustrations in a second attempt. The project was initially focused on color/color translation due to the historical concentration on achieving this link, but changed strategy following the original ideas sketched 30 years ago: to compose the music to be translated first, taking in mind the possibilities of perceptual translation of only a selection of elements, removing other non-transferable elements, and searching with musical experiments to find which transposable elements and conditions are effective perceptually for music-to-visual translation.

Introduction
Within the audiovisual idiom, an ancient tendency to render music to visuals has re-emerged in the present decade as an attractive, expressive form to artists: visual music. The term may be often employed to name any audiovisual production that aims in a greater or lesser extent with the intention, yet unfulfilled, to provide the simultaneous perception of an organization of events in real time through sounds and images. This motivation has generated a constant challenge in Western culture from the Greeks to present day, that is, to achieve the connection between music and images and the close integration of these two sensory domains in a similar time structure. It is only in our digital today that this art form is reaching a state of being in itself with significant potential and social relevance because of the global audiovisual consumption of our times. Evidence of its placement in the arts today is accounted for by several symposia such as Seeing Sound 3 at Bath Spa University, UK, since 2009, the Visual Music exhibition at the Smithsonian Institution, Washington, DC, in conjunction with the Museum of Contemporary Art, Los Angeles in 2005, with its accompanying publication (Brougher, K., Strick, J., Wiseman, A. and J. Zilczer. Visual Music: Synaesthesia in Art and Music since 1900. Thames & Hudson, 2005), many specialized publications and centers, and a major yearly prize since 2007, the “Visual Music Award”.

Within the field of visual music, the on-going AVIA research project is concentrated in finding the effective and direct interconnection of the elements of both sensory domains and their perceptive behavior, to convey the same time structure simultaneously. This is not an end in itself and as such it would be counted as another attempt to achieve the “synaesthetic dream”. However, the final objective of the AVIA project in determining the translatable elements between music and moving visuals is to provide a fine-tuning mastery, a deeper knowledge of the audiovisual behavior for creative artists. The research has included a revision of the translation efforts and analysis of the production throughout the whole of the Western art history in music and in the visual arts, hinting at providing a different reading to its development based on this perennial pursue, specially at the turn of the twentieth century. More research on the present production is necessary due to the massive increase in the generation of audiovisual art in the last years. It is concentrated in the creation of music compositions specifically made for AVIA translation.

A constant problem encountered by many historical attempts in visualizing music has been the desire to translate works of music which may have not been created with this intention in mind, or by simply not having a menu at hand in the process of creation of music, of procedures which may or may not be translatable. Even synaesthetic composers may fall into this problem, since their synaesthetic logic may not be perceived nor understood in the end results. Other risk that could be mentioned with the recent developments in visual software, is that the complexity of images produced may make the music run short of its inter-relationship with the visual part and be forced to step back to its previous role as “soundtrack” again.  This is not a problem for open audiovisual creation, but it does not belong within the AVIA concept.

A historical deviation in visual music has been the quest for the analogy of musical/visual color, and the AVIA project, like many others, fell in the beginning into this unrewarding track. The author believes that the spatial and rhythmic compositions are ideally suited for AVIA translation, rather than any consideration of color/color translation, as it has been historically attempted for centuries.

Why Visual Music?

Outline of Visual Music
A challenge for Western arts 
since late 16th century

Mechanical Devices (1595–)

Arcimboldo - gravicembalo, 1595 Milano
Louis Bertrand Castel - ocular harpsichord, 1725
Krüger, Kastner, Bishop,
Alexander Wallace Rimington - Color organ, 1893
Thomas Wilfred - Clavilux, 1920-1
Daniel Vladimir Baranoff-Rossiné - Piano Optophonique, 1922-3

Theories of equivalence - Search mainly for the note-color relationship
Analogy of words for musical timbre (color) and visual color
Analogy of notes-colors (Newton, 1675)  [pict]
Law of Octaves – planets singing (Hans Cousto)  [pict]
“…disorganized nature in the color-pitch analogy…” (expo book)  [pict]

Technological A/V innovations, Manifestos & synaesthetic research at turn of 20th Cent.
Development of A/Vtechnology: Auguste & Lois Lumiére patent the cinematographe, 1885.
T. A. Edinson, invents the  light bulb.
First synaesthetic research publication (Galton, 1883)
Futurist Bruno Corra. Manifesto of visual music Cinema astratto–musica cromatica  with Arnaldo Ginna, 1912.
Rimington publishes Colour-Music The art of mobile colour, in NY, 1911.

Synaesthetic composers in search for atonality (abstraction?)
Scriabin - Prometheus, Schönberg Farben (Klangfarbenmelodie), Varèse, Gershwin, Messiaen, Ciurlionis, Bliss.
Composers+painters (Schönberg, Gershwin)  [Farben]
Synaesthetic composers & painters at the beginning of 20th Century were in paralleland reciprocal roads towards dissonance, non-rhythmic, non-linear melodic music & abstract non-figurative painting.
Arnold Schönberg Vision, Die glückliche Hand, 1910
George Gerschwin painting Schönberg

Still‐artists influenced by music in search for abstraction
(1900--)
 Die Blaue Reiter, Kandinsky,  Ciurlionis, Schönberg, Picabia, Kupka, Baranoff‐Rossiné, Russell, MacDonald‐Wright, Matiushin, O'Keeffe, Richter, Valensi, Hirschfeld‐Mack (Bauhaus), Klee

Rhythmic influence to painting by jazz themes, rhythmic enfasis
(1920--)
 Kupka, Dove
Music of Stravinsky, Bartók, Chávez, Roldán, Orff, Loius Amstrong, Scott Joplin

Harmonic visual rendering
Paul Klee

Abstract photography, Animation, Entertainment & Film

(1920--) Stieglitz, Bruguière, Wilfred
(1940--) 
Richter, Eggelin, Ruttmann, Fischinger, Disney, Lye, Hirsch,  Whitney brothers, Brakhage, Belson, MacLaren, Greenewalt, Klein, László, Bentham

Influence of World Fairs & Son et Lumière (Paris 1937, Brussels 1958) 

"Circus Complex" in Visual Music
Film: Disney’s Fantasia 1940 , Fantasia 2000

Changes in the graphic representation of music
(1950---) Brown, Cage, Logothetis, Feldman, Wolf, Kagel, Stockhausen
Feldman’s, A Year From Monday, 1950
Logothetis’s Agglomeration, 1960

Sound space, architecture, installations
Xenakis, Soto, June Paik, Rosalie, Steinkamp
Yannis Xenakis - Polytope de Cluny, 1972
Jesús Soto - Penetrables y Escultura en Movimiento, 1962
The person moving into the sculptures is the one producing the movement and the sounds
Nam June Paik - TV-Cello, 1971, Charlotte Moorman with cello and bra of tv monitors
Rosalie -  Helios, 2008 (ZKM) Music by Ludger Brümmer, Rosalie -  Chroma Lux, 2009 (ZKM)
Jennifer Steinkamp Swell, 1995

Artistic Movements & Psychedelia
(1960---) 
Cosmic cinema, Vortex, Psychedelia, light shows, artificially-induced synaesthesia by LSD
Mix of art, entertainment, painting, cinema, music, philosophy, hippie aesthetics
Concentration on music (rock), and through LSD on visual effects
Joshua’s Light Show & Franz Zappa, 1967

Development of Video‐clip, Music & Films
(1964---) Change from “musical” to visual-media promotion of pop music through video-clips & cinema from the Beatles in the Sixties.
Music promo-films: Hard Day’s Night, 1964
Music promo-clip: Paperback Writer, 1966
Live world recording: All your need is Love  (400M viewers, 1967)
Magical Mystery Tour, 1967, 
I am the Walrus, Flying, Blue Jay Way (MMT)
Yellow Submarine,1968 Eleonor Rigby
Stanley Kubrik, 2001: Space Odyssey, 1968
Michelangelo Antonioni,  Zabriskie Point, 1970
Ron Fricke, Baraka, 1992

End of “blind-music era”
Establishment of music consumption linked with visuals, following Beatles promos, established by  Music Television MTV (1981) 
Evolution of digital A/V technology and increase of transmission speed through the internet:
1. Sound files = mp3 revolution, 1997
2. On-line music – Napster, 2000
3. iTunes, iPod, 2001
4.Video files = YouTube revolution, 2005
3. iPad, smart mobile phones – the search for the “proper” screen…
Music visualizers: iTunes, Mmp, live laptop Vj’s

Sound design
Sound as signal language but not music nor speech:
Wall‐E, 2008

Visual Music Today
Modes of Visual Music

Tran
slation Processes
Data translation
Fractal Light Rock, Juan Antonio Lleó, 2005
Nano-World
Nanoanillos, Andrea Führer, 2007

Partial results of the AVIA Translation project
Articles:
Congresses attended:
Artistic residencies by the author:
Screenings/Lectures
Performances on the project’s theme