VISUAL MUSIC - AVIA Translation Project
Seeing Sounds: Visual Music and Abstraction in Video-Art
Dr. Emilio Mendoza, UNEARTE - USB, Caracas; ZKM | Hertz-Labor, Zentrum für Kunst und Medien, Karlsruhe
HOME
| Abstract  |  Introduction | Outline | Antecedents | Partial Results | References

BRIEF INTRO

 AVIA stands for "Audio-Visual Integrated Arts" as a fair, balanced name where both domains appear in equal importance. It was proposed as a wide genre that includes specifically and primarily Visual Music in the domain of artistic expression. The AVIA Translation Project focuses on determining the possibilities in translating time structures between both domains, music and moving visuals, being one way primarily, that is, music towards visuals. The final aim is focused in applying music composition knowledge into abstract moving visuals to render a dual art-form of the highest expressive possibilities. The "AVIA Translation Project" was started in September, 2007, at the Simon Bolivar University, Caracas, by Emilio Mendoza, with the assistance of graduate student Gabriel Peraza from the Master's Degree Program in Composition. It was originally funded by the Dean of Research and Development, USB, and it has been developed in conjunction with the ZKM - Zentrum für Kunst und Medien, Karlsruhe, Germany, in four research internships as GastKünstler (Guest Artist) at the ZKM. The title of the project "Seeing Sounds" is specifically chosen to honour an excellent platform of knowledge by the Visual Music Research Group, School of Music and Performing Arts, Bath Spa University, UK. In an excellent Seeing Sound meeting in 2013, I had the distinction to meet in person two legendary artists of Visual Music as Larry Cuba and Bret Battey. 

Title of Research & Creative Project
          Seeing Sounds: Visual Music and Abstraction in Video-Art


Abstract
The intended research will determine the closest translation possibilities of music into visuals and the application of music compositional procedures into the making of abstract video-art. Historical, bibliographic and audiovisual references will be analyzed, specifically the production that uses abstraction as a dominant visual language, including the still visual-arts at the turn of the 20th century, in the hypothetical assumption that perception of visual abstraction tends a link to music order. Visual Music is a field of today’s audiovisual art that focuses on creating the simultaneous perception of a time structure through sounds and moving visuals. It has been a constant challenge in Western culture for centuries but finally the digital audio-visual technology offers true possibilities of achieving the synaesthetic dream. The project will produce historical, artistic and technical knowledge on Visual Music in the form of papers and VM compositions.
General Objective
           To produce knowledge on visual music and a VM composition

Specific Objectives
  1. To research bibliographical and audio-visual references
  2. To establish the closest translation possibilities of time-structures between both sensory domains
  3. To analyze the relationship of still and moving visual abstraction to music perception
  4. To determine music composition procedures applicable to abstract moving visuals
  5. To compose visual music works
  6. To complete one article on abstraction in the part Flying of The Beatles' film Magical Mystery Tour (1967), for The Journal of Beatles Studies, Liverpool University Press.

INTRODUCTION
Within the audiovisual idiom, an ancient tendency to render music to visuals has re-emerged in the present decade as an attractive, expressive form to artists: visual music. The term may be often employed to name any audiovisual production that aims in a greater or lesser extent with the intention, yet unfulfilled, to provide the simultaneous perception of an organization of events in real time through sounds and images. This motivation has generated a constant challenge in Western culture, namely, to achieve the connection between music and images and the close integration of these two sensory domains in a similar time structure. It is only in our digital today that this art form is reaching a state of being in itself with significant potential and social relevance because of the global audiovisual consumption of our times, mainly through the screen of our cell phones. Evidence of its placement in the arts today is accounted for by several symposia such as Seeing Sound 3 at Bath Spa University, UK, since 2009, the Visual Music exhibition at the Smithsonian Institution, Washington, DC, in conjunction with the Museum of Contemporary Art, Los Angeles in 2005, with its accompanying publication (Brougher, K., Strick, J., Wiseman, A. and J. Zilczer. Visual Music: Synaesthesia in Art and Music since 1900. Thames & Hudson, 2005), many specialized publications and centers, and a major yearly prize since 2007, the “Visual Music Award”.

The on-going AVIA Research Project is concentrated in finding the effective and direct interconnection of the elements of both sensory domains and their perceptive behavior, to convey the same time structure simultaneously. This is not an end in itself and as such it would be counted as another attempt to achieve the “synaesthetic dream”. However, the final objective of the AVIA project in determining the translatable elements between music and moving visuals is to provide a fine-tuning mastery, a deeper knowledge of the audiovisual behavior for creative artists, knowledge which stems from the musical composition realm. The research has included a revision of the translation efforts and analysis of the production throughout the whole of the Western art history in music and in the visual arts, hinting at providing a different reading to its development based on this perennial pursue, specially at the turn of the twentieth century. More research on the present production is necessary due to the massive increase in the generation of audiovisual art in the last years. At this point, it is relevant for the project to focus on the marked tendency towards abstraction in film and video-art and the immediate connection with the musical domain when it occurs, as well as the need to research on the application of procedures of music composition to the field of moving visuals in time. Finally, the project will be concentrated in the creation of visual music compositions specifically made through AVIA translation.
A constant problem encountered by many historical attempts in visualizing music has been the desire to translate works of music which may have not been created with this intention in mind, or by simply not having a menu at hand in the process of creation of music, of procedures which may or may not be translatable. Even synaesthetic composers may fall into this problem, since their synaesthetic logic may not be perceived nor understood in the end results. Other risk that could be mentioned with the recent developments in visual software, is that the complexity of images produced may make the music run short of its inter-relationship with the visual part and be forced to step back to its previous role as “soundtrack” again.  This is not a problem for open audiovisual creation, but it does not belong within the AVIA concept. A historical deviation in visual music has been the quest for the analogy of musical/visual color, and the AVIA project, like many others, fell in the beginning into this unrewarding track. The author believes that the spatial and rhythmic compositions are ideally suited for AVIA translation, rather than any consideration of color/color translation, as it has been historically attempted for centuries.

Why Visual Music?

CONCEPTUAL OUTLINE
Visual Music main themes
A challenge for Western arts 
since late 16th century

Mechanical Devices (1595–)

Arcimboldo - gravicembalo, 1595 Milano
Louis Bertrand Castel - ocular harpsichord, 1725
Krüger, Kastner, Bishop,
Alexander Wallace Rimington - Color organ, 1893
Thomas Wilfred - Clavilux, 1920-1
Daniel Vladimir Baranoff-Rossiné - Piano Optophonique, 1922-3

Theories of equivalence - Search mainly for the note-color relationship
Analogy of words for musical timbre (color) and visual color
Analogy of notes-colors (Newton, 1675)  [pict]
Law of Octaves – planets singing (Hans Cousto)  [pict]
“…disorganized nature in the color-pitch analogy…” (expo book)  [pict]

Technological A/V innovations, Manifestos & synaesthetic research at turn of 20th Cent.
Development of A/Vtechnology: Auguste & Lois Lumiére patent the cinematographe, 1885.
T. A. Edinson, invents the  light bulb.
First synaesthetic research publication (Galton, 1883)
Futurist Bruno Corra. Manifesto of visual music Cinema astratto–musica cromatica  with Arnaldo Ginna, 1912.
Rimington publishes Colour-Music The art of mobile colour, in NY, 1911.

Synaesthetic composers in search for atonality (abstraction?)
Scriabin - Prometheus, Schönberg Farben (Klangfarbenmelodie), Varèse, Gershwin, Messiaen, Ciurlionis, Bliss.
Composers+painters (Schönberg, Gershwin)  [Farben]
Synaesthetic composers & painters at the beginning of 20th Century were in parallel and reciprocal roads towards dissonance, non-rhythmic, non-linear melodic music & abstract non-figurative painting.
Arnold Schönberg Vision, Die glückliche Hand, 1910
George Gerschwin painting Schönberg

Still‐artists influenced by music in search for abstraction
(1900--)
 Die Blaue Reiter, Kandinsky,  Ciurlionis, Schönberg, Picabia, Kupka, Baranoff‐Rossiné, Russell, MacDonald‐Wright, Matiushin, O'Keeffe, Richter, Valensi, Hirschfeld‐Mack (Bauhaus), Klee

Rhythmic influence to painting by jazz themes, rhythmic emphasis
(1920--)
 Kupka, Dove
Music of Stravinsky, Bartók, Chávez, Roldán, Orff, Louis Amstrong, Scott Joplin

Harmonic visual rendering
Paul Klee

Abstract photography, Animation, Entertainment & Film

(1920--) Stieglitz, Bruguière, Wilfred
(1940--) 
Richter, Eggelin, Ruttmann, Fischinger, Disney, Lye, Hirsch,  Whitney brothers, Brakhage, Belson, MacLaren, Greenewalt, Klein, László, Bentham

Influence of World Fairs & Son et Lumière (Paris 1937, Brussels 1958) 

"Circus Complex" in Visual Music
Film: Disney’s Fantasia 1940 , Fantasia 2000

Changes in the graphic representation of music

(1950---) Brown, Cage, Logothetis, Feldman, Wolf, Kagel, Stockhausen
Feldman’s, A Year From Monday, 1950
Logothetis’s Agglomeration, 1960

Sound space, architecture, installations, visual arts

Xenakis, Soto, June Paik, Rosalie, Steinkamp
Yannis Xenakis - Polytope de Cluny, 1972
Alejandro Otero - Coloritmos, 1955
Jesús Soto - Penetrables y Escultura en Movimiento, 1962
The person moving into the sculptures is the one producing the movement and the sounds
Nam June Paik - TV-Cello, 1971, Charlotte Moorman with cello and bra of tv monitors
Rosalie -  Helios, 2008 (ZKM) Music by Ludger Brümmer, Rosalie -  Chroma Lux, 2009 (ZKM)
Jennifer Steinkamp Swell, 1995

Artistic Movements & Psychedelia
(1960---) 
Cosmic cinema, Vortex, Psychedelia, light shows, artificially-induced synaesthesia by LSD
Mix of art, entertainment, painting, cinema, music, philosophy, hippie aesthetics
Concentration on music (rock), and through LSD on visual effects
Joshua’s Light Show & Franz Zappa, 1967

Development of Video‐clip, Music & Films
(1964---) Change from “musical” to visual-media promotion of pop music through video-clips & cinema from the Beatles in the Sixties.
Music promo-films: Hard Day’s Night, 1964
Music promo-clip: Paperback Writer, 1966
Live world recording: All your need is Love  (400M viewers, 1967)
Magical Mystery Tour, 1967, 
I am the Walrus, Flying, Blue Jay Way (MMT)
Yellow Submarine,1968 Eleonor Rigby
Stanley Kubrik, 2001: Space Odyssey, 1968
Michelangelo Antonioni,  Zabriskie Point, 1970
Ron Fricke, Baraka, 1992

End of “blind-music era” 

Establishment of music consumption linked with visuals, following Beatles promos, established by  Music Television MTV (1981) 
Evolution of digital A/V technology and increase of transmission speed through the internet:
 mp3 revolution, 197
2. On-line music – Napster, 200
3. iTunes, iPod, 201
4.Video files = YouTube revolution, 2005
. iPad, smart mobile phones – the search for the “proper” screen…
Music visualizers: iTunes, Mmp, live laptop Vj’s.

Sound design
Sound as signal language but not music nor speech:
Wall‐E, 2008

Visual Music Today
Modes of Visual Music

ANTECEDENTS
Author's previous translation attempts
In 1977 the author composed several pieces in a set of jokes and conceptual works by the name of La Caja de Juguetes, (The Box of Toys), treating the translation of musical ideas to other areas of expression such as dance, visuals, narrative and poetry, noise, and other concepts such as imagination, attention and perception, rituals, being and life itself. In the same year, Mendoza produced an interdisciplinary work within the  audiovisual field called Susurro (Whisper), a piece focused on the spatial composition for 6 synthesizers emitting white noise or for mixed choir of six voices as producers of white noise, with a spatial placement from left to right and mono (source form all sides and center simultaneously). The author tried to perform it with 6 EMS synthesizers, the "Synthy" suitcase model, for its premiere at the 33rd Hauptarbeitstagung of the Institut für Neue Musik und Musikerziehung, 04/08/1979, Darmstadt, Germany. The six synthesizers were provided by the English company EMS, and the author leased six carousel slide-projectors to provide a visual version of the score. Unable to train six people to play the synthesizers live by producing white noise with filters and volume handling, the concert was finally performed by a group of friends performing the white noise with vocals, mostly medicine students from his college dorm at the Uni-Klinik in Düsseldorf. A sophisticated amplification was set up through six individual channels, assembled and controlled by the sound technician Michael Feller from the Robert Schumann Institut, Düsseldorf. The slide projectors were also discarded because it meant requiring six additional persons dedicated to controlling them live, following the score, and also because of the rental costs and transportation of the five screens for the projection...  a very expensive  production for a music student!
After this concert, the piece was recorded in the sound studios of the Robert Schumann Institute, conducted by Alfredo Rugeles, with the sound engineer Michael Feller, four composers doing each voice together (Ramón Ramos, Alfredo Marcano, Paco Estévez, Emilio Mendoza), and recording all six voice tracks. With this audio, the author concentrated again to run the A/V version of the piece with six carousel projectors that could automate the change and dimmer of slides, but unfortunately did not work at all. In 1979, the author composed the work Secretos for wind sextet, with a structure exposed only through musical color, in order to translate it to live visuals again, with the same carousel slides equipment but with colors this time and not with space concepts. Secretos was premiered at the Kibbutz Shefayim, Israel, 1980, within the ISCM New World Music Days Festival and the recording was later produced by the West Deutsches Rundfunk (WDR) with the Düsseldorfer Bläßersextette, Alfredo Marcano, conducting. With recording in hand, much time was consumed with the projectors, painting slides and trying to set the visual score, but with very frustrating results, again.

Compositional failures of space and color
The two pieces, Susurro and Secretos proved to be complete failures in conveying musical time-structures through space relationships (Susurro) and through musical timbre (Secretos), although the author thoroughly composed the works with these two elements in mind. On listening to the recordings, there was no sense of perceiving structures, no trace in memory, and therefore no understanding of the music heard. That was one of the reasons that moved him to insist on trying to convey the similar time structure through visuals in real time: to help the music to be comprehended through its simultaneous visual version. The title Secretos specifically gives the sense concerning this frustration where color does not function to convey complex structures but just simple contrasts in music. The analysis of Secretos was included at the end of the printed score for the sake of revealing the unperceived work in it, until a visual version could be achieved.

The author did not abandon the A/V translation project completely, involving himself with the EMS Spectron Video Synthesizer in the late seventies, but access to this hardware was very limited and no progress was accomplished. He also worked on different projects dealing with music, space and visuals, (see Pasaje, 1976, Gaudeamus International Composition Prize), as well as with music for dance and video productions over the years. In November, 2012, a new version of Susurro was performed in the ISCM 2012 New World Music Days Festival in Antwerp, by the Aquarius Choir, with the inclusion of refinements in the “z” axis of the previous score.
New attempts since 2006 - AVIA Project
With this frustration kept at the back of his mind for several years, the author began the translation project again with a lecture within a science/art exhibition (VI Salón Arte-Ciencia, USB, 27-10-2006) on color-color translation, and then registered the AVIA Project at the Dean of Research and Development, USB in 2007. He realized how gradually the importance and omnipresence of audiovisual media had taken our lives and the strength of the trend of musical art to fold into the visual domain definitely. This was accounted by the technological development in this field (Internet, iPod, cell phones), reaching the people in the streets and other social A/V-phenomena as Napster, YouTube, iTunes, so he decided to use his sabbatical research leave in 2008 to work thoroughly in a research project on the Music to Visuals Translation, reconnecting his previous frustrations in a second attempt. The project was initially focused on color - color translation due to the historical concentration on achieving this link, but the strategy was changed following the original ideas sketched 30 years ago: to first compose the music to be translated, taking in mind the possibilities of perceptual translation of only a selection of elements, removing other non-transferable elements, and searching with musical experiments to find which transposable elements and conditions as well as compositional procedures are effective perceptually for music-to-visual translation.


PARTIAL RESULTS
Articles published

•   Mendoza, Emilio. "Mach Schau!: The contribution of The Beatles to the development of Visual Music in Magical Mystery Tour." In: Julia Merrill, ed. Popular Music Studies Today. Proceedings of the International Association for the Study of Popular Music 2017. Systematische Musikwissenschaft. Wiesbaden: Springer, 2017, 186-201. ISBN 978-3-658-17739-3.


Congresses attended
 Previous Artistic & Research residencies
 Screenings & Lectures given
 Performances on the project’s theme

SELECTED REFERENCES


Abbado, A. (2018). Masters of Visual Music -  Abstract Explorations of Past and Present Artists. London: Skira.
Almeida Rodríguez, P. A. (2021). La obra de Oskar Fischinger: Análisis de la relación entre música e imagen. Madrid: Editorial Académica Española.
Aussät, K. (1986). Paul Klee und die Musik. Hrsg. von H. Ole. Kunsthalle Frankfurt.
Becks-Malorny, U. (1999). Wassily Kandinsky The journey to abstraction. NY: Taschen.
Birren, F. (1987). Principles of Color. Atglen, PA: Schiffer Publishing Ltd.
Bosseur, J-Y. (1993). Sound and the Visual Arts. Paris: Dis Voir.
Boulton, A. (1994). Alejandro Otero. Caracas: O. Ascanio Editores.
________. (1966). Alejandro Otero. Colección Venezolanos. Caracas: OCI.
Brougher, K., Mattis, O., Strick, J., Wiseman, A. and J. Zilczer. Visual Music: Synaesthesia in Art and Music since 1900. NY: Thames & Hudson.
Chion, M. (1993). La Audiovisión: Introducción a un análisis conjunto de la imagen y el sonido. Madrid: Paidós Ibérica.
Cook, F. (2004). Analysing musical multimedia. New York: Oxford University Press.
Cytowic, R. E. (2002). Synesthesia: A Union of the Senses 2ed. Cambridge, Mass.: The MIT Press.
DeWitt, T. (1987). “Visual Music: Searching for an Aesthetic.” Leonardo 20 (115-122).
Dessauer-Reiners, C. (1996). Das Rhythmische bei Paul Klee. Eine Studie zum genetischen Bildverfahren. Worms.
Düchting, H. (2001). Paul Klee  Malerei und Musik. Munich: Prestel.
Evans, B. (2005). “Foundations of a visual music.” Computer Music Journal 29 (11-24).
Faulkner/D-Fuse, M., ed. (2006). VJ: Audio-Visual Art and VJ Culture. London: Lawrence King Ltd.
Feineman, N. and S. Reiss (2000). Thirty Frames Per Second: The Visionary Art of the Music Video. Harry N. Abrams.
Gramcko, I. (1958). "Coloritmos y Gracia Cinética." in Revista Shell (June 1958), 21-25.
Jones, R. and B. Nevile (2005). “Creating visual music in Jitter: Approaches and techniques.” Computer Music Journal 29, 55-70.
Kagan, A. (1983). Paul Klee: Art and Music. London.
Keefer, C. and J. Guldemond. (2013). Oskar Fischinger 1900-1967: Experiments in Cinematic Abstraction. Thames & Hudson.
Martin, S. (2006). Videoarte. Köln: Taschen.
Mendoza, E. (2017). "Mach Schau!: The contribution of The Beatles to the development of Visual Music in Magical Mystery Tour." Popular Music Studies Today. Proceedings of the International Association for the Study of Popular Music 2017. Systematische Musikwissenschaft. Julia Merrill, ed. Wiesbaden: Springer, 2017, 186-201.
Mendoza, E. (2008). "Fundamentals of Music Composition for AVIA Translation." System Research in the Arts and Humanities Symposium Proceedings – Vol. II. Canada: The International Institute for Advanced Studies in Systems Research and Cybernetics, 55-61.
Mösser, A. (1976). Das Problem der Bewegung bei Paul Klee. Heidelberg.
Otero. A. (1948). Alejandro Otero y la Crisis de la Pintura Moderna. Paris: Lévis Mano.
Overmeyer, G. (1982). Studien zur Zeitgestalt in der Malerei des 20. Jahrunderts. Robert Delaunay-Paul Klee. Hildesheim/Zürich/New York.
Peacock, K. (1988). “Instruments to perform color-music: Two centuries of technological experimentation.” Leonardo 21, 397-406.
Raynal, M. (1956). Modern Painting. Geneva: Skira.
Robertson, L. and N. Sagiv. (2005). Synesthesia perspectives from Cognitive Neuroscience. Oxford:  Oxford University Press.
Roeckelein, J. E. (2000). The Concept of Time in Psychology A Resource Book and Annotated Bibliography. Westport, CT: Greenwood Press.
Rudi, J. (2005). “Computer music video: A composer’s perspective.” Computer Music Journal 29, 36-44.
Runciman, J. F. “Noises, Smell and Colours.” The Musical Quaterly 1, Nº 2 (April 1915), 149-61.
Scoates, C. (2013). Brian Eno: Visual Music. NY: Chronicle Books.
Tüchtig, H. (1996). Farben am Bauhaus - Synthese und Synästhesie. Berlin.

van Campen, C. (2007). The Hidden Sense: Synesthesia in Art and Science. Cambridge, MA: The MIT Press.

Vernallis, C. (2004). Experiencing Music Video: Aesthetics and Cultural Context. Nueva York: Columbia University Press.

Von Maur, K. (1999). The Sound of Painting: Music in Modern Art. Prestel.

Wagler, S. R. (1974). "Sonovision: A visual display of sound," in Frank J. Malina [ed.], Kinetic Art, Theory and Practice, New York: Dover, 162-164.


Web References
Battey, B. (2017). Bret Battey/Bat Hat Media. <http://www.mti.dmu.ac.uk/~bbattey>.
Friedlander, P. (2008). What is Visual Music?. <http://www.paulfriedlander.com/text/visualmusic.html>.
Gallet, S. (2007). Fractal Animations by Silvie Gallet. <http://sylvie.gallet.free.fr>.
Griffiths, P. (2008). “Messiaen, Olivier.” Grove Music Online, ed. L. Macy. <http://www.grovemusic.com>.
Jewanski, J. (2008). “Colour and music 2. Music as related to colours.” Grove Music Online, ed. L. Macy. <http://www.grovemusic.com>.
Powell, J. (2007). “Skryabin, Aleksandr.” Grove Music Online, ed. L. Macy. <http://www.grovemusic.com>.
Malinowski, S. (2013). "Music Animation Machine". <http://www.musanim.com>.
Mendoza, E. (2020). "Visual Music: AVIA Translation Project." http://www.ozonojazz.com/emilio/avia/avia.html>.
Ox, J. (s/f). A Complex System for the Visualization of Music. <http://home.bway.net/jackox/complexity.html>.
Schubert, G. (2007). “Hindemith, Paul: Works Other dramatic.” Grove Music Online, ed. L. Macy. <http://www.grovemusic.com>. 

Other Associated Links
http://acg.media.mit.edu/people/golan/aves/
http://cimatics-masterclass.blogspot.com/
http://rhythmiclight.com
http://spotworks.com/
http://sylvie.gallet.free.fr
http://vispo.com/misc/ia.htm
http://visualmusic.blogspot.com/
http://www.centerforvisualmusic.org
http://www.generatorx.no/
http://www.iotacenter.org/
http://www.johnadamczyk.com/performance.html
http://www.openendedgroup.com/artworks/biped/biped.htm
http://www.paradise2012.com/visualMusic/musima/
http://www.pixelsumo.com/
http://www.solu.org
http://www.transmediale.de
http://www.visualmusic.org/
http://www.vjspain.com/
http://www.vjtheory.net
http://www.johnadamczyk.com/performance.html
http://www.paulfriedlander.com/text/visualmusic.html
http://sylvie.gallet.free.fr