ISTANBUL TECHNICAL UNIVERSITYF GRADUATE SCHOOL OF ARTS AND SOCIAL SCIENCES
PROPRIUS:
SONIFICATION OF ANIMAL BEHAVIOR
IN AN AUTONOMOUS INTERACTIVE MUSICAL ECOSYSTEM
Ph.D. THESIS Zeynep ÖZCAN ÇAMCI
Department of Music Music Programme
ISTANBUL TECHNICAL UNIVERSITYF GRADUATE SCHOOL OF ARTS AND SOCIAL SCIENCES
PROPRIUS:
SONIFICATION OF ANIMAL BEHAVIOR
IN AN AUTONOMOUS INTERACTIVE MUSICAL ECOSYSTEM
Ph.D. THESIS Zeynep ÖZCAN ÇAMCI
(409132010)
Department of Music Music Programme
Thesis Advisor: Assoc. Prof. Dr. Can Karado˘gan Co-advisor: Dr. Reuben de Lautour
˙ISTANBUL TEKN˙IK ÜN˙IVERS˙ITES˙I F SOSYAL B˙IL˙IMLER ENST˙ITÜSÜ
PROPRIUS:
OTONOM ETK˙ILE¸S˙IML˙I MÜZ˙IKAL B˙IR EKOS˙ISTEMDEK˙I HAYVAN DAVRANI¸SLARININ SESLE¸ST˙IR˙ILMES˙I
DOKTORA TEZ˙I Zeynep ÖZCAN ÇAMCI
(409132010)
Müzik Anabilim Dalı Müzik Programı
Tez Danı¸smanı: Assoc. Prof. Dr. Can Karado˘gan E¸s Danı¸sman: Dr. Reuben de Lautour
Zeynep ÖZCAN ÇAMCI, a Ph.D. student of ITU Graduate School of Arts and So-cial Sciences student ID 409132010 , successfully defended the dissertation entitled “PROPRIUS: SONIFICATION OF ANIMAL BEHAVIOR IN AN AUTONOMOUS INTERACTIVE MUSICAL ECOSYSTEM”, which she prepared after fulfilling the requirements specified in the associated legislations, before the jury whose signatures are below.
Thesis Advisor : Assoc. Prof. Dr. Can Karado˘gan ... Istanbul Technical University
Co-advisor : Dr. Reuben de Lautour ... University of Canterbury
Jury Members : Assoc. Prof. Dr. Yahya Burak Tamer ... Bahcesehir University
Asst. Prof. Dr. Taylan Özdemir ... Istanbul Technical University
Prof. Dr. Mehmet Can Özer ... Yasar University
Asst. Prof. Dr. Gökhan Deneç ... Istanbul Technical University
Dr. Emre Erkal ... Erkal Mimarlık Ltc.
Date of Submission : 1 October 2018 Date of Defense : 12 October 2018
FOREWORD
I would like to start with thanking my advisors Assoc. Prof. Dr. Can Karado˘gan and Dr. Reuben de Lautour. My research would have been impossible without their aid and support. I would also like to express my gratitude to the members of my jury, Assoc. Prof. Dr. Yahya Burak Tamer, Asst. Prof. Dr. Taylan Özdemir, Prof. Dr. Mehmet Can Özer, Asst. Prof. Dr. Gökhan Deneç, and Dr. Emre Erkal for their important suggestions and comments.
A very special gratitude goes out to all at M˙IAM and ˙ITÜ Development Foundation for funding my travels for conferences to present this study.
I also want to express my sincere thanks to Asst. Prof. Dr. Ay¸se Sevil Enginsoy Ekinci for introducing me to Iannis Xenakis eight years ago with a New York Times article which led me to pursue this study in sonic arts. Her friendship and guidance throughout more than five years of this study is not something easy to find.
I would like to thank my colleagues at MIAM Sonic Arts Department. The multidisciplinary environment that we have inspired me during these five years. It has been a very interesting journey to do research in such a challenging field with unforgettable memories.
I am also grateful to my close friend Müge Atala for her personality, emotional support, suggestions and comments. Without our phone talks I wouldn’t have finished this study as a sane person.
I would also like to thank my spouse Anıl Çamcı for backing me up with love, unfailing support and continuous encouragement during the many challenges of my dissertation. Thank you for being my best friend and making me happy every day.
I would also like to extend my respect to my grandfather who passed away during the writing of this dissertation. Ever since I was a kid he has encouraged me to be an intelligent and independent woman who doesn’t need a man’s financial support in her life. He doesn’t have any idea but he was one of the most feminist people I have ever met. I wish he could have seen me finish this research and become a PhD.
Thanks to my mother and father for their patience and understanding. This dissertation is dedicated to them. I would have never finished this study without their support. Thank you.
TABLE OF CONTENTS Page FOREWORD... ix TABLE OF CONTENTS... xi ABBREVIATIONS ... xiii LIST OF TABLES ... xv
LIST OF FIGURES ...xvii
SUMMARY ... xix
ÖZET ... xxi
1. INTRODUCTION ... 1
1.1 Structure and Contents of the Dissertation... 1
1.2 Motivations... 2
1.3 Aims and Objectives... 5
1.4 Methods ... 6 2. BACKGROUND ... 11 2.1 Animal Behavior... 11 2.2 Musical Ecosystems ... 13 2.2.1 Agent-based modelling... 13 2.2.2 Interactive approach ... 15 2.2.3 Emergence ... 20
2.3 Augmented Reality Approach ... 20
2.4 Related Work ... 21
3. DEVELOPMENT OF THE CASE STUDY: PROPRIUS... 35
3.1 Modeling of the Ecosystem ... 36
3.2 Modeling of Animal Behavior... 38
3.2.1 Optimal foraging model ... 38
3.2.2 Selection of behaviors ... 40
3.2.3 Movement of animals ... 42
3.3 Augmented Reality Audio Implementation... 42
3.3.1 Spatialization of organisms ... 46
3.3.2 Binaural spatialization ... 47
4. THE PROPRIUS COMPOSITION ... 51
4.1 Sonification of Organisms ... 53
4.1.1 Attribute related... 54
4.1.2 Behavior related... 56
4.2 Real-time Synthesis Methods ... 59
4.3 Compositional Structure... 61
4.3.1 Scene 1... 63
4.3.3 Scene 3... 64 4.3.4 Scene 4... 65 4.3.5 Scene 5... 66 4.4 Interaction... 67 5. CONCLUSION ... 69 5.1 Future Directions ... 71 5.2 Closing Remarks ... 73 APPENDICES... 87
APPENDIX A: Processing Code of Proprius ... 89
APPENDIX B: Complete List of Sound Examples (CD) ... 129
ABBREVIATIONS
AR : Augmented Reality
E.A.T. : Experiments in Art and Technology GUI : Graphical User Interface
FM : Frequency Modulation HCI : Human-Computer Interaction HMD : Head-Mounted Display
IRCAM : Institut de Recherche et Coordination Acoustique/Musique IVS : Interactive Virtual Soundscape
ICST : Institute for Computer Music and Sound Technology OOP : Object-oriented Programming
Mo-cap : Motion Capture OSC : Open Sound Control UDP : User Datagram Protocol
UPIC : Unité Polyagogique Informatique CEMAMu VR : Virtual Reality
LIST OF TABLES
Page Table 3.1 : The behaviors and attributes of organisms at each trophic level... 41 Table 4.1 : Parameter mappings of the attributes. ... 52 Table 4.2 : List of behaviors and related attributes. ... 58
LIST OF FIGURES
Page
Figure 1.1 : AR-based IVS use case... 3
Figure 1.2 : Users in the IVS system... 4
Figure 1.3 : Diagram of approaches and tools used in Proprius. ... 7
Figure 1.4 : Location Data Diagram. ... 9
Figure 1.5 : Psi pose... 9
Figure 2.1 : First example on human-computer interaction... 15
Figure 2.2 : Second example on human-computer interaction. ... 16
Figure 2.3 : Third example on human-computer interaction. ... 16
Figure 2.4 : Videoplace. Source: Ars Electronica Archive. ... 17
Figure 2.5 : User experiencing Proprius... 21
Figure 2.6 : A-Volve... 22
Figure 2.7 : Archipelago... 23
Figure 2.8 : Visualization of Gakki-mon Planet... 24
Figure 2.9 : White cube shows the position of the user in the world... 25
Figure 2.10: Eden... 26
Figure 2.11: Visuals of Listening Sky... 28
Figure 2.12: Infinite Game interaction guide ... 29
Figure 2.13: Infinite Game exhibition ... 30
Figure 2.14: Fluid Space exhibition... 31
Figure 2.15: Time of Doubles exhibition... 31
Figure 2.16: Endless Current ... 32
Figure 2.17: From left to right Flow, Membranes, Impacts... 33
Figure 3.1 : Layout of Microsoft Kinect Sensor. (Source: Valentini, 2012)... 36
Figure 3.2 : An example of ecological pyramid (Eltonian pyramid). ... 38
Figure 3.3 : An example of behavior selection for primary producers. ... 41
Figure 3.4 : Example image in Scene 5... 43
Figure 3.5 : Data flow across various hardware and software components of Proprius. ... 44
Figure 3.6 : Synapse screen on computer after psi pose. ... 44
Figure 3.7 : Max spat.virtualspeakers~ object outside insect poly~. ... 46
Figure 3.8 : Max ICST ambimonitors inside insect’s poly~ object number 21. ... 47
Figure 3.9 : Location data flow diagram. ... 48
Figure 3.11: Use of skeletal tracking data obtained from Kinect via Synapse. .... 49
Figure 4.1 : Sonification model in Proprius... 53
Figure 4.2 : Graphical representation of the frequency range of organism... 55
Figure 4.3 : Detailed structure with behaviors. ... 56
Figure 4.5 : Structure of the composition... 57 Figure 4.6 : Graphical representation of five layers... 59 Figure 4.7 : Graphical representation of five layers... 62 Figure 4.8 : Processing screen: Scene 1... 63 Figure 4.9 : Processing screen: Scene 2... 64 Figure 4.10: Processing screen: Scene 3... 65 Figure 4.11: Processing screen: Scene 4... 66 Figure 4.12: Processing screen: Scene 5... 66
PROPRIUS:
SONIFICATION OF ANIMAL BEHAVIOR
IN AN AUTONOMOUS INTERACTIVE MUSICAL ECOSYSTEM SUMMARY
Proprius is an interactive augmented reality composition based on the sonification of organisms in an artificial ecosystem. It is an autonomous sonic environment, where the sonification of an ecological simulation is used as a means to create an interactive augmented reality music composition. The consecutive layers of a food chain are used as movements that dictate the temporal progression of the composition. The listener explores a physical space augmented with Proprius to experience the compositional unfolding of the ecosystem. The listener’s presence in the system affects nearby organisms; the listener is thereby situated as an interactive agent that influences the progression of the work. The stochastic allocation of resources at the beginning of the work ensures that each instance of Proprius is a unique composition.
This dissertation presents the final report of Proprius, a creative research project that was conducted between 2015 and 2018. It describes the development of a biologically interactive sonic ecosystem in Processing, spatialization and interaction methods and techniques used for composition as well as a musical outcome of the interaction. In this research, an artificial ecosystem designed using the Processing platform simulates animal behaviors. The simulation data are then fed into a sonification engine designed with Max/MSP, which creates an immersive audio scene in real-time. The listener’s position in the scene is tracked with a Kinect sensor; as the listener explores the exhibition space, a binaural audio scene augments their physical environment. The overall structure of this dissertation is composed of five thematic chapters. The first chapter presents the structure and contents of the dissertation. It also explains motivations, aims and objectives as well as research methods employed by the research.
Chapter two begins by laying out the background information in biology within the scope of this research and focuses on the key themes of this study. It then presents an overview of the existing studies on the use of ecological models in music composition and sound art.
Chapter three further explains the theoretical details of the development of the case study focusing on the programming of organisms. It describes behavioral ecology and ethological models used in Proprius. It also clarifies how the methods are applied in the study and how certain tools and techniques are utilized in the implementation of the system.
Chapter four presents the compositional development and outcome of the research. It delves into approaches to sonifying animal behaviors, evaluates the musical characteristics of these behaviors, and structures the resulting sonifications in the context of an augmented reality composition.
The final chapter gives a brief summary and critique. It includes limitations, concluding remarks, suggestions for further research, and propositions.
The first appendix section includes the Processing code of Proprius. The second appendix section includes the complete list of audio examples given throughout this dissertation. You can listen to the individual sound examples given in the related section.
The PDF version of the dissertation provides hyperlinks to references, figures, tables and chapters, and to audio examples. Additionally, the document also provides hyperlinks to audio examples. The whole appendix section also can be found in the accompanying CD that is provided with the dissertation.
I would like to present this study as a modest artistic proposition in the field of sonic arts for future researchers. I believe that this study offers a strong contribution as a case study that blends music composition with ecological simulation, embodied interaction, and augmented reality audio. I expect that with recent advances in immersive media technologies, augmented reality compositions such as Proprius will gain further prominence.
PROPRIUS:
OTONOM ETK˙ILE¸S˙IML˙I MÜZ˙IKAL B˙IR EKOS˙ISTEMDEK˙I HAYVAN DAVRANI¸SLARININ SESLE¸ST˙IR˙ILMES˙I
ÖZET
Proprius yapay bir ekosistemde ya¸sayan organizmaların sesselle¸stirilmesine dayanan, etkile¸simli, arttırılmı¸s gerçeklik bestesidir. Bu çalı¸sma biyolojiden ilham alınarak yaratılmı¸s ve Processing programı kullanılarak tasarlanmı¸stır. Tezin içerisinde Processing kullanılarak tasarladıktan sonra, gerçek mekânı zenginle¸stirilmi¸s mekâna çevirirken kullanılan teknikler; seslerin mekânsalla¸stırılması; seyirci üzerinde kullanılan etkile¸sim metodları; organizmaların seyirciye göre mekânla¸stırılması sırasında kullanılan teknikler; ve bütün bunların sonucu olarak duyulan sesselle¸stirme anlatılmı¸stır.
Proprius içerindeki yapay ekosistemde ya¸sayan organizmalar otonom olma özelli˘gine sahiptirler ve kendi kendilerine avlanmaya ya da avcıdan kaçmaya karar verebilirler. Organizmalar sahip oldukları özellikler do˘grusunda beslenecekleri organizmaları seçer ve davranı¸slarını da bu do˘grultuda seçerler. Proprius içerisindeki organizmalar besin zincirinden yola çıkarak tasarlanmı¸slardır. Üreticilerden ba¸slayan besin zinciri, ot yiyenler, hem ot hem et yiyenler ve et yiyen organizmalar olmak üzere 4 ayrı kademeden olu¸sur. Bu ayrı katmanları temsil etmek üzere üreticiler için ye¸sil bitkiler; ot yiyenler için böcekler; hem et hem ot yiyenler için ku¸slar ve et yiyen canlılar için büyük kediler seçilmi¸stir.
Proprius içerisindeki organizmalar dinleyici için sessel ve etkile¸simli bir biçimde tecrübe edilecek müzikal bir ortam meydana getirirler. Bu müzikal ortamın içerisinde yaratılmı¸s olan farklı sessel katmanlar organizmalar tarafından meydana getirilir. Proprius içerisinde sözü edilen farklı sessel katmanları olu¸sturan organizmalar besteye daha önceden belirlenmi¸s zamana göre art arda eklenir. Organizmalar do˘gdukları andan itibaren ses çıkarmaya ba¸slar ve ölene kadar ses çıkarmaya devam ederler Proprius’un yapay ekosistemi Processing programı kullanılarak yaratıldıktan sonra, bu programdan elde edilen veriler sesselle¸stirmeyi gerçekle¸stirmek üzere Max/MSP programına gönderilmektedir. Max/MSP programı organizmaların seslerinin gerçek zamanda sentezlenmesi için kullanılmaktadır. Etkile¸simli ortamın yaratılması için dinleyicinin mekân içerisindeki konumunun takip edilmesi gerekir. Bunu sa˘glamak için Kinect sensörü kullanılmı¸stır. Kinect sensörü içerisinden dinleyicinin gerçek mekan içerisindeki üç boyutlu konumunu elde etmek için Sypnapse uygulaması kullanılmı¸stır. Dinleyici mekânı gezerken sesselle¸stirmeyi duymak için kulaklık takar. Dinleyicinin zenginle¸stirilmi¸s gerçeklik içerisindeki mekânı gezerken kulaklıkta duyaca˘gı sesler binaural audio tekni˘gi kullanılarak dinleyiciye dinletilir.
Processing ekranında yapay ekosistemin grafik animasyonu yaratılmı¸s olsa bile, dinleyici Proprius’u herhangi bir görsel olmadan sadece kulaklıklarla dinleyerek ve yapay ekosistem içerisinde yürüyerek tecrübe eder. Processing ekranında görselle¸sitirilen organizmalar, sadece bestecinin organizmaların hareketlerini ve
dinleyicinin yapay ekosistem içerisindeki konumunu ve organizmalara etkisini takip edebilmesi için yaratılmı¸stır.
Organizmaların sesselle¸stirilmesi için organizmaların özellikleri (boyut, ya¸s, v.b) ve o sıradaki mevcut olan davranı¸sları kullanılmı¸stır (avlanmak, kaçmak, v.b). Her organizmayı ayrı ayrı temsil etmek üzere farklı ses sentezleme yöntemleri kullanıl-mu¸stır. Ancak, türler arasında ortak bir e¸sle¸stirme modeli kullanılmı¸stır. Özetle, organizmaların sa˘glı˘gı ses dalgasının genli˘gine (amplitüd), organizmaların enerjisi sesin genel frekanssal zenginli˘gine, organizmaların boyutu sesin ana frekansına (F0), organizmaların ya¸sı sesin duyulma sıklı˘gına ve organizmaların pozisyonu sesin mekansalla¸stırılmasına denk gelecek biçimde e¸sle¸stirilmi¸stir. Proprius’taki organizmaların davranı¸sları, ses olaylarının uzunluk süresini belirler. Av/avcı etkile¸simlerine dayalı davranı¸slar (örne˘gin kaçma ve takip etme) daha kısa süreye sahipken, otlanma ve dinlenme gibi davranı¸slar daha uzun sürerler. Bu, parçanın sahneleri boyunca yapısal varyasyonlar yaratılmasına neden olur.
Bitkiler için yaratılan ses sentezmeler sırasında bir koro metaforundan yola çıkarak format sentezi kullanılmı¸stır. Bitkilerin boyutu bu sentezin temel frekansı olarak kullanılmı¸s ve tüm bitkiler arasında harmonik yapılar olu¸sturmak için ortak bir ölçe˘ge dayandırılmı¸stır. Böceklerin sesleri, cırcır böceklerinin sesleri dü¸sünülerek modellenmi¸stir. Bu sesi üretmek için filtrelenmi¸s gürültü sesi cırcır böce˘gi sesine göre modellenmi¸s amplitüd zarflarından (envelope) geçirilerek üretilir. Gürültü üzerindeki tepe filtrenin merkez frekansı, böce˘gin büyüklü˘güne göre e¸slenmi¸stir. Ku¸s sesleri, ta¸sıyıcı frekansının ku¸sun boyutuna e¸slendi˘gi FM sentezi ile üretilir. Modülatör frekansı, spektral zenginlik yaratmak için ku¸sun enerjisine e¸slenmi¸stir. Amplitüd zarfları do˘gal ku¸s seslerine göre modellenmi¸stir. Büyük kedilerin sesleri filtrelenmi¸s gürültü ve eklemeli sentezin birle¸stirilmesi ile yaratılır. Büyük kedinin büyüklü˘güne e¸sle¸stirilen sentezleyicinin temel frekansı, amplitüd zarfı her tetiklendi˘ginde LFO ile modüle edilir. Organizmaların boyutları frekanslarına göre ters orantılı olacak biçimde e¸sitlenmi¸stir. Bunu Sonucu olarak, büyük kediler hayvanlar arasında en dü¸sük frekansa sahiptirler. Farklı bir dinamik aralı˘ga sahip hayvanların tasarlanması, eserin son bölümünde sonik bir kaosun önlenmesine de yardımcı olur.
Zenginle¸stirilmi¸s gerçeklik içerisine yerle¸stirilen yapay ekosistem içindeki dinleyici, gerçek mekan içerisinden hareket ederken, organizmalar ile etkile¸sim halindedir. Organizmalar yapay ekosistem içerisinde var oldukları katmanda yaratılacakları zaman, sistem içerisine rastgele da˘gıtılırlar. Zenginle¸stirilmi¸s gerçeklik içersinde organizmalar ve dinleyici mekânda aynı noktalarda bulundukları zaman, dinleyici organizmaların sa˘glı˘gını bozarak ortaya çıkacak olan müzikal sonuca müdahalede bulunmu¸s olur. Dinleyici Proprius’un yapay ekosistemi içerisinde dola¸sıp eseri dinledi˘gi sırada organizmalar ile aynı pozisyonda oldu˘gu zaman organizmaların sa˘glıklarının normalden fazla bozulmasına neden olur. Normalde dinleyiciden gelen etkile¸sim olmasa daha organizmlar otonom olduklarından dolayı sa˘glıklarını belli bir zaman içerisinde kaybedecek davranı¸slarda bulunuyorlar. Ancak dinleyicinin ekosistem içerisindeki etkisi organizmların sa˘glı˘gını normalden çok fazla bozarak onların ölüm riskini arttıyor. Böylece normalde avlanmaya gidecek olan organizmanın beslenmeden ölme riski artmı¸s oluyor.
Bu çalı¸smayı olu¸sturan tezin tamamı be¸s bölümden olu¸smaktadır. ˙Ilk bölümü bütün tezin nasıl planlandı˘gını anlatan giri¸s bölümü olu¸sturur. Daha sonra ara¸stırma prati˘gi olan Proprius’u meydana getirmede katkıda bulunan motivasyonlar, tezin hedef ve
amaçları ile tezde yaratılmak istenen hedef ve amaçları geçekle¸stirmeye çalı¸sılırken kullanılan metodlar anlatılır.
Tezin ikinci bölümü, bu ara¸stırma kapsamında biyolojiden ilham alarak kullanılan hayvan davranı¸sları ile ilgili olan teorileri anlatarak ba¸slar. Tezin ikinci bölümü daha sonra bu çalı¸smanın ana temalarını içeren alanlara odaklanır ve tezin ara¸stırma prati˘gi sırasında kullanılan yakla¸sımları anlatır. Bu bölümün son kısmında, müzik kompozisyonunda ve ses sanatında ekolojik modellerin kullanımıyla ilgili mevcut çalı¸smalara genel bir bakı¸s sunar.
Üçüncü bölüm Proprius’un yaratılma a¸samasına odaklanır. Üçüncü bölümün ilk kısmında organizmaların besin zincirinden yararlanarak bir ekosistem içerisinde nasıl modellendirildikleri, bu modellendirme gerçekle¸stilirken kullanılan teoriler, hayvan davranı¸slarının seçiliminin nasıl kurgulandı˘gı ve hayvanların ekosistem içerisinde nasıl hareket etti˘gi anlatılır. Daha sonra zenginle¸stirilmi¸s gerçeklik uygulamasının kullanıldı˘gı teknikler ile organizmaların mekânda ve seyirciye göre nasıl yerle¸stirildi˘gi, binaural audio metodunun nasıl uygulandı˘gı da anlatılır.
Dördüncü bölüm, otonom etkile¸simli ekosistem içerisindeki hayvan davranı¸slarının nasıl sesselle¸stirildi˘gini anlatıyor. Bu bölüm hayvan davranı¸sları sesselle¸stirilirken kullanılan gerçek zamanlı sentezleme metodları, seslerin nasıl sentezlendi˘gini içerir. Son kısımda ise müzi˘gi olu¸sturan bölümlerin katmanları ayrı ayrı anlatılır. Bu bölüm seyircinin organizmalar ile olan etkile¸simi sonucu ortaya çıkan sesselle¸stirmeyi de anlatır.
Tezin son bölümü olan be¸sinci bölüm, bu çalı¸smanın gelecekte yönelebilece˘gi alanları farklı öneriler altında anlatır. Bölüm, bu çalı¸smanın genel özeti ve ele¸stiriler kısmından sonra yazarın bu alana faydalı olaca˘gını dü¸sündü˘gü kapanı¸s önerileri ile sonlanır. Tezde ayrıca iki adet ek bölümü bulunmaktadır. ˙Ilk ek bölümde, yapay ekosistemin kuruldu˘gu Processing kodu tamamıyla okuyucu ile payla¸sılmı¸stır. Bilginin payla¸sarak arttı˘gını savunan açık-kaynak topluluklarının da dahil oldu˘gu bu çalı¸sma, umuyorum ki gelecekte bu kodu geli¸stirecek ya da oldu˘gu gibi kullanacak ara¸stırmacı ve sanatçılar için bir ba¸slangıç kayna˘gı olur. Tezin ikinci ekinde ise tez içerisinde verilen sesselle¸stirme örnekleri toplu bir biçimde bulunup dinlenebilir. Ayrıca tezin basılı versiyonu içerisinde bulunan CD’de de bu kaynakları bulabilirsiniz.
Bu çalı¸smayı gelece˘gin ara¸stırmacıları ve sessel sanatçılarına mütevazı bir sanatsal yapıt olarak sunuyorum. Bu çalı¸smanın, müzik kompozisyonunu, ekolojik simülasyon, etkile¸simli sanat, artırılmı¸s gerçeklik içerisinde ses ile harmanlayan bir örnek ara¸stırma prati˘gi olarak, alanına güçlü bir katkıda bulundu˘guna inanıyorum. Umuyorum ki medya teknolojilerindeki son geli¸smelerle, Proprius gibi artırılmı¸s gerçeklik kompozisyonları daha da önem kazanacaklar.
1. INTRODUCTION
This study presents Proprius, a creative research project that was conducted between 2015 and 2018. It describes the development of a biologically interactive sonic ecosystem in Processing, spatialization and interaction methods and techniques used for composition as well as a musical outcome of the interaction.
Proprius is an autonomous sonic environment where the sonification of an ecological simulation is used as a means of creating an interactive augmented reality music composition.
In this research, an artificial ecosystem, designed using the Processing platform, simulates animal behaviors. The consecutive layers of a food chain are used as movements that dictate the temporal progression of the composition. The listener explores a physical space augmented with Proprius to experience the compositional unfolding of the ecosystem. The simulation data are then fed into a sonification engine designed with Max/MSP, which creates an immersive audio scene in real-time. The composition is presented to the listener through headphones. The listener’s position in the scene is tracked with a Kinect sensor: as the listener explores the exhibition space, a binaural audio scene augments the physical environment. The listener’s presence in the system affects nearby organisms; the listener is thereby situated as an interactive agent that influences the progression of the work. The stochastic allocation of resources at the beginning of the work ensures that each instance of Proprius is a unique composition.
1.1 Structure and Contents of the Dissertation
The overall structure of the study is composed of five thematic chapters including this introductory chapter.
Chapter two begins by laying out the background information in biology within the scope of this research and focuses on the key themes of this study. It then presents an
overview of the existing studies on the use of ecological models in music composition and sound art.
Chapter three further explains the theoretical details of the development of the case study focusing on the object-oriented programming of organisms. It describes behavioral ecology and ethological models used in Proprius. It also clarifies how the methods are applied in the study and how certain tools and techniques are utilized in the implementation of the system.
Chapter four presents the compositional development and outcome of the research. It delves into approaches to sonifying animal behaviors, evaluates the musical characteristics of these behaviors, and structures the resulting sonifications in the context of an augmented reality composition.
The final chapter gives a brief summary and critique. It includes limitations, concluding remarks and suggestions for further work.
There are two appendices included at the end of the thesis. The first one is the Processing code of Proprius. I am hoping that sharing this contribution will allow other researchers and composers to quickly impose their visions to their compositions by using the data of artificial ecosystems1.
The second appendix section includes the complete list of audio examples given throughout this dissertation. You can listen to the individual sound examples given in the related section. The PDF version of the dissertation provides hyperlinks to references, figures, tables and chapters, and to audio examples. Additionally, the document also provides hyperlinks to audio examples. When you see:I
object throughout the thesis, click on it to listen to sound examples. The whole appendix section also can be found in the accompanying CD that is provided with the dissertation.
1.2 Motivations
The early inspiration of this study lies in the idea of designing organisms/agents/sounds that can actually act the way I imagined. I started to imagine sounds as living entities 1If you would like to use the code please contact me ([email protected]). I am interested to hear
as I was beginning this study. For example, before composing a piece, I was imagining a compositional space where the sound entities can live. How big can it be? What can I do if I want to enlarge the space in the middle of the composition? What can I do to make the audience realize that now they are listening to a larger space than one minute ago?
I was also imagining what is going to happen if two sounds meet. What can happen if they collide? Which way are they going to move? Maybe they collide into one and start to live together. Or maybe these sounds are completely different entities that can’t see or hear each other.
I also thought of superimposing the compositional space into real space so that the audience can walk in the space that I designed. This way the audience can interact with different characteristics of sounds that I imagined.
My previous studies regarding the works of Iannis Xenakis inspired me to pursue research in electronic music. Since I started my PhD in Sonic Arts Program at MIAM (Center for Advanced Studies in Music), I have been very motivated to create interactive generative sound environments for the audience to explore compositions.
Figure 1.1 : AR-based IVS use case.
In this regard, I used different strategies of electronic music composing in my works that I composed between 2013 and 20162. I was interested in a wide range of sound-based art forms ranging from compositions to installations and sound spatialization. I was willing to explore the different methods of creating interactive generative sound environments.
During this process I was influenced by Le Monte Young’s concept of Dream House “in which a work would be played continuously and ultimately exist in time as a living organism with a life and tradition of its own” (Emmerson, 2014, p. 6).
My growing interest in interactivity eventually led to a collaborative research project with Damla Pehlevan and Anıl Çamcı titled Interactive Virtual Soundscapes (IVSs). In this project, we used augmented reality approach to IVS by “[superimposing] an aural virtual reality on a physical space that is to be explored by the user” (Çamcı et al., 2015). We also implemented Ambisonic Panning by using ICST Ambisonic Equivalent Panning library in Max. In this system, sound sources act as phantom objects through which the user can walk (See Figure 1.2).
With the AR system, we observed different bodily interactions that the audience used when they were exploring the environment (See Figure 1.1).
During the IVS experiments, I started to imagine sounds as different animals or natural phenomena. I was interested in exploring the artistic potential of biologically inspired models.
Figure 1.2 : Users in the IVS system.
After the IVS project, I started to design an artificial ecosystem that has organisms in it. The organisms would need to eat to survive, and different organisms would occupy different parts of the food chain. This artificial ecosystem is based on the idea
of biologically inspired musical system. This is the story of how Proprius came into being in a nutshell.
1.3 Aims and Objectives
This study describes Proprius an interactive sound environment where the sonification of animal behavior within an ecological simulation is used to create an interactive augmented reality music composition. It explores the systematic relationship between behavioral ecology and data sonification as a means to create complex compositional structures that rely on the interactions between the agents of our ecological model. In Proprius, attributes and behaviors of individual agents have been used within an ecological simulation to synthesize sounds in real-time to explore the generation of emergent sonic textures through sonification. This approach allows to utilize animal behaviors and how these behaviors permeate through a food chain to establish compositional structures.
In order to achieve this aim, I first examined autonomous interactive musical systems. Second, I designed an interaction between user/audience/listener and organisms/agents in the artificial ecosystem. Third, I performed a creative research practice titled Proprius, a composition that explores the artistic potential of biologically inspired models for electronic music composition and the interaction model of the audience in that environment. From the users’ perspective, what separates Proprius from a traditional composition is its navigability. The piece starts when a user enters the exhibition space. The user hears the sonification of various animal behaviors in the context of an interactive sound art piece without any visual feedback.
Starting from the late 1960s, the term “sound art” has been defined by several artists and extensively critiqued by vast amount of scholars (Demers, 2010; LaBelle, 2015; Licht and O’Rourk, 2007). Nick Collins defines sound art as “the idea of a publicly visited artwork; it may help to consider the art gallery setting as one common option, where an artist presents an installation work with a significant sound-based element, awaiting visitors to experience it” (Collins et al., 2013, p. 151). As explained by Leigh Landy, “sound art has traditionally been largely associated with fine and new media artists, but has also been associated with some musicians works” (Landy, 2007, p. 11). In Audio Culture: Readings in Modern Music sound art is described as a “general term
for works of art that focus on sound and are often produced for gallery or museum installation” (Cox and Warner, 2004, p. 415) LaBelle describes sound art as “a practice [that] harnesses, describes, analyzes, performs, and interrogates the condition of sound and the processes by which it operates” (LaBelle, 2015, p. ix).
Sound art usually tends to refer to significantly sound-related works like sound installations. The fundamental aspect of the definitions is that they all imply the space where these works are exposed to the audience. They are usually experienced in public interior spaces like art galleries or museums (Trimpin’s File Organ), or exterior public spaces other than concert halls (Annea Lockwood’s Piano Garden; Max Neuhaus’s Water Whistle; Bill Fontana’s Earth Tones; Andres Bosshard’s Klangturm).
Sound art is a very broad multidisciplinary field explored and used by experimental avant-garde composers, visual artists, sound artists, composers, performers, painters, poets, digital artists and so on. The term itself covers artworks that incorporate sound or use sound as the primary medium. The research and practice area of sound art is broadening according to the works of artists. Artists are blurring the lines between sound and whatever other media they are using in sound art. Although it is an accepted musical genre, majority of sound artworks are experienced or exhibited with visuals (La Monte Young’s Compositions 1960 10; Maryanne Amacher’s The Music Rooms: A Four Part Mini-Sound Series; Young and Marian Zazeela’s The Well-Tuned Piano in The Magenta Lights). Yet it is a field which is differentiated from visual, digital or fine arts. For that reason, I have chosen to represent Proprius without visuals as a more underrepresented form of sound art.
1.4 Methods
This study uses qualitative research methods. It follows a case study design. In practice-based research, “the creative artifact is the basis of the contribution to knowledge” (Candy, 2006, p. 1). I use practice as a case study and identify questions, problems, and needs that emerged during the development of the case study. Utilizing case study as the method of practice, this dissertation aims to make three main contributions to the field of sonic art: development of a biologically inspired interactive sonic ecosystem to compose; of techniques for the composition of spatially experienced interactive sound art driven by interactions as practiced in the Proprius
project; and of a design principle for imposing compositional structures. In this study I will also discuss how the experiment affects the creative work and the creative work affects the experiment. Figure 1.3 outlines the list of approaches during the body development of Proprius: real-time synthesis, body tracking, binaural encoding, interactive audience, augmented reality, and ecological models.
Kinect Proprius AR Approach Realtime Synthesis Interactive Binaural Audio Ecological Model/ Animal Behavior
Figure 1.3 : Diagram of approaches and tools used in Proprius.
In Proprius the artificial ecosystem is designed by using Processing3. This artificial ecosystem simulates animal behaviors. The simulation allows the composer to view the progression of the work in terms of the behaviors of animals that make up the ecosystem. The composer can monitor the location of the audience and the organisms on the Processing screen. The user, on the other hand, experiences this system aurally without such visual cues.
I use Processing, which is a class-based, object-oriented programming (OOP) and integrated development environment (IDE) language built in Java. I prefer to use Processing because it uses more simplified syntax and is predominantly used by creative coders. It is also an open source computer programming language. Discussing one’s creative ideas in an open source community is motivating for creative coders that
come from various interdisciplinary backgrounds. I believe that this helps to ease the learning curve.
The bases of a Processing code consist of classes and objects. The code is made of numerous different objects. Each of these objects can have different roles for the system. We can think of classes as blueprints of objects. Objects in Processing are defined with properties (e.g. data members, attributes) and behaviors (e.g. member functions) (Shiffman, 2009; Noble, 2009). While writing an object-oriented computer programming, we start to think about the things we are going to create like the way we think about the world.
To illustrate simply, we can think about ourselves as objects. How can we define a human? Our data members can be hair color, height, location, etc. What kind of functions does a human have? We can run, we can sing, we can code. When coding OOP, we can start simple with several data members and functions, and then make our code more complicated. We are basically defining any object that are similar to our real-life experience.
When coding Proprius, I followed this basic principle and created different classes that correspond to the steps in the food chain. More details of the modeling of organisms will be given in Chapter 34.
While Processing simulates animal behaviors, behavioral data are fed into Max (See Figure 1.4), a graphical programming environment for real-time musical and multimedia software applications (Noble, 2009; Puckette, 1991). Max creates an immersive audio scene in real-time. The extracted information is communicated to Max via Open Sound Control (OSC)5, a cross-platform networking protocol most commonly used in musical applications. Additionally, as McCartney highlights, “Max, which is quite a different kind of programming language, provides an interesting set of abstractions that enable many people to use it without realizing they are programming at all”, which helps to ease the learning curve for the newcomers (McCartney, 2002, p.30).
Motion-tracking data of the audience is important to locate them in the environment. This data is used for mapping the audience into the artificial ecosystem. Microsoft’s
4Refer to Appendix A for a detailed coding of Proprius. 5http://opensoundcontrol.org
Kinect Synapse Max Processing human
position animal position
Figure 1.4 : Location Data Diagram.
Kinect for Xbox 360 sensor is used to determine the position of the user in physical space. The sensor uses a motion capture technique called skeletal tracking which “allows Kinect to recognize people and follow their actions”6. In order to start extracting the skeletal tracking data from Kinect by doing the “psi pose” (See Figure 1.5), I use the Synapse7application (Bellona, 2012).
Figure 1.5 : Psi pose.
Simply put, I first use the human location data extracted from Kinect and send it to Synapse. Second, by using Synapse’s Max application, the location data is sent to Proprius’s Max application. Third, the same data is sent to Proprius’s Processing application in order to map and superimpose humans into an artificial ecosystem. Finally, the human location data is sent in Proprius’s Max application to the specialization of sounds (See Figure 1.4). The audience has a disease-like external
6https://msdn.microsoft.com/en-us/library/hh973074.aspx
factor in the system that lowers the health of nearby organisms. More details on the specialization of the composition will be given in Chapter 4.
When the listener explores the exhibition space, a binaural audio scene augments their physical environment. I used spat.virtualspeakers~ objects from IRCAM (Institute for Research and Coordination in Acoustics/Music) spatialization software. Spat8 was used in order to render multichannel input into a binaural mix.
I also implemented ICST Ambisonic tools for Max9 developed by Jan Schacher and Philippe Kocher at the Institute for Computer Music and Sound Technology (ICST) in Zurich (Schacher and Kocher, 2006). ICST Ambisonic tools10 allow encoding and decoding in three dimensions up to third order Ambisonics. They also include a graphical control module for real-time manipulation of the sound sources’ placement and algorithmic control in a variety of ways of source motion in 3D.
Depending on your project, ambisonic audio applications provide quite a lot of flexibility in terms of their use in virtual reality, surround formats, binaural, stereo or other forms of feedback11.
8Spat is available at http://forumnet.ircam.fr/product/spat-en/
9https://www.zhdk.ch/downloads-ambisonics-externals-for-maxmsp-5381 10The ICST Max/Msp externals are available at https://www.zhdk.ch/5381
11For another Ambisonic application, see HoaLibrary for Max and Pure data
2. BACKGROUND
This chapter presents background theory for the research presented in this thesis. It also explains various approaches used in this study. The final section of this chapter discusses the different approaches that have been used in the artificial ecological environments.
2.1 Animal Behavior
The modern field of ethology (i.e. study of animal behavior) is considered to have emerged in the 1930s with the works of biologists Nikolaas Tinbergen, Konrad Lorenz and Karl von Frisch. A subfield of biological sciences, ethology investigates animal decision-making, animal cognition, animal learning, and animal communication. Having brought behavior to biology, ethology tries to build a theoretical structure for interpreting animal behavior and classified behaviors as either innate and learned (Tinbergen, 1952).
Tinbergen is known for his studies on the courting behavior of the male three-spined stickleback and for the social behavior of herring gulls (Tinbergen, 1953). Lorenz’s research about instinctual behaviors in birds has led to the pioneering concept of imprinting in newly born ducklings (Lorenz, 1937). Frisch is recognized for his research on the behavior of bees, where he explains the dance-like movements of bees as a way of social communication to signal about the food sources or hostile environments (Von Frisch, 1967). In 1973, the three won a joint Nobel Prize and were cited “for their discoveries concerning organization and elicitation of individual and social behavior patterns” (Tinbergen et al., 1974). This was the first award given to a behavior-based study in the field of behavioral sciences. In those years, the Nobel Foundation was known to be biased and opposed to behavioral sciences (Dewsbury, 2003, p. 747).
In his 1963 article “On aims and methods of ethology”, Tinbergen left behind the innate or learned dichotomy and proposed four kinds of ways -cause, function (survival value), development (ontogeny), and evolution- to look at animal behavior.
Observing birds singing to each other, we can ask four questions about this behavior. First, we can ask questions about causation. For example, what’s the physiological bases of behavior of “all the way down to molecular biology” (Tinbergen, 1963, p. 416)? What is directing this behavior (e.g. hormones)? Second, we can look at the function of behavior and ask how it functions to promote the survival value of the animal. For example, what is the function of this behavior and for what purpose has it evolved? For Tinbergen, studying the survival value pertains to “find[ing] out, if possible by experimentation, how animal behaviour contributes to survival” (1963, p. 416). Third, we can ask questions of ontogeny (development), which Tinbergen describes as “change of behaviour machinery during development” of a lifetime (1963, p. 424). That is to say, what is the origin of the behavior? Finally, we can ask evolutionary questions. According to Tinbergen, evolutionary study has “two major aims: the elucidation of the course evolution must be assumed to have taken, and the unravelling of its dynamics” (Tinbergen, 1963, p. 428). For example, when we look at the close relatives of an animal, can we see similar kinds of behaviors?
Consequently, the idea of behavior can be a trait like eye color which forms as a result of natural selection and can evolve and adapt, leading us to the field called behavioral ecology1. Behavioral ecology has moved the study of behavior to studies concerning genetic and environmental influence on behavior. It provides an evolutionary and ecological framework regarding animal behavior (Krebs and Davies, 2009).
Behavioral ecology studies the ways in which animals adapt their behavior to maximize chances of survival and reproductive success. It tries to construct mathematical models and find a scientific framework of explanation regarding how animals have a propensity to behave in a certain way under different circumstances and environments and how they react when the circumstances change. By exploring such behaviors as hunting, eating, foraging, reproducing, and escaping from
predators, researchers in this field aim to reveal the underlying principles of animal decision-making (McFarland, 1977).
2.2 Musical Ecosystems
Biological systems have been used by many artists and researchers in creative applications that rely on the modeling of animal behavior in virtual environments. These applications, also referred to as computational ecosystems (Antunes, 2013), artificial ecosystems (Ji, 2012) or sonic ecosystems (Bown, 2009), take the form of audio (Dahlstedt and Nordahl, 2001), visual (Sommerer and Mignonneau, 1999), and multimodal (McCormack, 2001) art installations. Furthermore, the simulation of complex ecosystems in real-time enables the implementation of interactive works, where the user input affects the evolution of the system.
2.2.1 Agent-based modelling
The organisms in ecological environments are modeled with software entities called agents that show some degree of life-like intelligent behavior and autonomy. That is to say, an agent is identical to an organism in a natural ecosystem, which is a complex living that requires energy. Stanford Encyclopedia of Philosophy briefly defines an agent as “a being with the capacity to act, and agency denotes the exercise or manifestation of this capacity” (Schlosser, 2015). Following Franklin and Graesser, “an autonomous agent is a system situated within and a part of an environment that senses that environment and acts on it, over time, in pursuit of its own agenda and so as to effect what it senses in the future” (Franklin and Graesser, 1997). As described by Pattie Maes, “autonomous agents are computational systems that inhabit some complex dynamic environment, sense and act autonomously in this environment, and by doing so realize a set of goals or tasks for which they are designed” (Maes, 1995, p. 108). The term “autonomy” is used by many artists for explaining the ability of self-determination in software-based musical systems.
In life-like artificial environments, behaviors of the agents can be modeled biologically. Autonomous agents can interact with each other and with humans according to the simple or more complex rules that they follow. They have the ability to observe their environment and make decisions. Agents in computational ecosystems are “organized
in a hierarchical structure (of a food-chain) and trade token units (of energy and biomass) as a way of promoting community dynamics” (Antunes et al., 2014, p. 1). Like in real ecosystems, they are all part of the energy flow and a nutrient cycle. In artificial natures with a cybernetic approach, where autonomous agents and humans can communicate, the human participant can be introduced to the agents. Cybernetic systems, where human and computer mutually affect each other’s behaviors, create an action perception feedback loop between real and artificial environments (Chadabe, 1977, pp.5-6).
Originally developed in engineering, Cybernetics is a multidisciplinary research approach which is also used for creative purposes. It studies control and commu-nication, theory of automata, and information transmission in systems (Immelmann and Beer, 1989; Barrows, 2011). In the 1940s, Norbert Weiner described the term “Cybernetics” as the science of transmitting messages between man and machine, or from machine to machine. He suggested human communication as a model for human-computer interaction (Wiener, 1961). After that influential theory, as Hayles formulated, “humans were to be seen primarily as information processing entities who are essentially similar to intelligent machines” (Hayles, 2008, p. 7).
Later, Cybernetics became an important influence on not only science but art and “its involvement with electronic systems, networking, and transmission taking off throughout the 1960’s with the works of the art-technology groups such as Experiments in Art and Technology (E.A.T.) and Pulsa” (LaBelle, 2010). In 1968, Roy Ascott introduced cybernetic theory and cybernetic vision in art. He proposed a “behaviorist framework [to form cybernetic culture]” (Ascott, 1968, p. 190).
Within these systems, the output of the artificial environment affects the current state of the real environment or vice versa. Thus, both the human participant and the agents change the behaviors of each other mutually. We can compare this circular loop to our daily life communication. For example, when we see someone looking very angry, we might choose not to speak to this person. Or we might choose to speak more gently in order to calm this person down to avoid fight or any kind of danger. Based on our actions, the angry person can change their facial expressions or might look and sound angrier. Simply, we constantly transform ourselves to the environment according to the
information that we receive form other beings. In interactive music systems without any kind of visual feedback (e.g. performer, video, kinetic sculpture, projection), the feedback that the participant receives is a sound. At the same time, the feedback that the artificial agents receive is a data that the system affords.
2.2.2 Interactive approach
Composing interactive music for an interactive audience in a responsive artificial ecological environment involves some level of communication between humans and the agents of the ecological environment. Designing a meaningful interaction and a sonic feedback is an important part of the system (Schacher, 2009). The feedback loop between real and artificial environments depends on the role of the audience within the sonic ecosystem and the composers’s creative approach to human-computer interaction and sonification. For example, Figure 2.1 shows the use of linear relationship where the audience only experiences the work that comes from the artificial environment without interfering with the system. Also, the audience doesn’t contribute to the artwork itself (Dorin, 2009; Eldridge and Dorin, 2009; Dahlstedt and Nordahl, 2001).
Figure 2.1 : First example on human-computer interaction.
On the other hand, Figure 2.2 shows a feedback loop between the user and the artificial world. In addition to Figure 2.1, the audience interaction affects the behavior of the sonic ecosystem as well as the sonic output.
To give a further example, Figure 2.2 shows autonomous agents with more complex rules. On top of Figure 2.1, agents have the ability to listen to the sonic output and act accordingly.
In recent years, researchers have investigated a variety of approaches to develop and explore musical ecosystems. Interactive approach has a great potential to further
explore the creative output of Human Computer Interaction (HCI) (Eldridge and Bown, 2018).
Figure 2.2 : Second example on human-computer interaction.
In the field of interaction in music, from 1990 to present times, the term “interaction” implies the following:
[H]uman-computer musical interaction, or human-human musical interaction that is mediated through a computer, [it involves] the creation or programming of software that will respond to predetermined aspects of a live performance, ... determine other aspects of the music, either generating synthesized sound, or modifying in some way all or some of the live sound. (Weale, 2005)
Figure 2.3 : Third example on human-computer interaction.
Indeed, the beginnings of interaction date back to 1958 when Allan Kaprow developed the term “happening” where he uses active audience in his artworks. He aims to blur the boundaries between artwork and the audience. As he wrote in his premiere notes, “[t]he line between art and life should be kept as fluid, and perhaps indistinct, as possible” (Packer and Jordan, 2002, p. 308). Kaprow was interested in gaining creative response from the audience. In his work Happenings, which was held in physical environments, he encourages the audience to make their own decisions. Kaprow’s
approach allows the audience to make their personal choices that would affect the performance. This kind of approach requires the integration of art and technology. After his works, artists started to emerge more in the interactive artistic environments2. In early 1970’s, Myron Krueger created Videoplace to explore interaction between computer and the audience through video (See Figure 2.4). Krueger defines Videoplace as a “conceptual environment with no physical existence” (Packer and Jordan, 2002, p.113). In Videoplace, two participants in different rooms can communicate through their projected images on the screen. Krueger aims to connect images with participants and tries to enhance the connection between them by designating specific goals. For example, if two participants can cooperate, they can see the third image. Videoplace has a responsive environment that replies according to the participant’s position, voice volume, etc. Kruger developed interaction design based upon motion tracking before Microsoft launched Kinect in November 2010. That is to say, Kinect is not the earliest invention that makes motion tracking techniques possible. It has just made it accessible to everyone who wants to use it as a medium for an artwork.
Figure 2.4 : Videoplace. Source: Ars Electronica Archive.
Electronic music takes its part within all these interaction-related inspirational works in media arts. Composer Joel Chadabe describes this new era as “a fundamentally new way of functioning for a composer. It demands performance skill and
performance-oriented thinking, and it raises questions as to how well a composer can deal with real-time problems” (Chadabe, 1977, p. 7).
The aim of encouraging the audience to explore artistic space differently and to make themselves contribute to the artwork freely can be seen in Iannis Xenakis’s reputable works. His figure is important to mention in this context because he also shares the aim of enhancing audience experience in electronic music. As an engineer, architect and composer, Xenakis paid great attention to the specific combinations of media in his work. He was concerned with finding and constructing the ideal architectural space for a particular musical performance (Xenakis, 2008, pp.131-160). He believed that the architectural form in which a spectacle is experienced was of prime importance. Highlighting the audience-centric approach in artwork is rooted in his desire for new and immersive experiences. His attempts to unify various artistic forms bring about a consolidation of media.
In 1977, Xenakis introduced the image sonification system, Unité Polyagogique Informatique CEMAMu (UPIC). It was a computer drawing system, a product of acoustic and computer science. Compositions were made just by drawing with an electromagnetic pen on a special drawing board. The curves or waveforms of drawings were calculated by the computer and sent through a converter to make the results audible. He describes the process as follows: “[a] child may draw a fish, listen to it, [...] and learn by drawing how to think musically, compositionally without being tormented by solfege” (Xenakis, 2008). Mycène Alpha (1978) and La Légende d’Eer (1978) are selected examples that were composed with UPIC by Xenakis. Later, it has been used by several composers such as, Jean-Claude Risset (Saxatile, 1992), Takehito Shimazu, Curtis Roads, and Aphex Twin. Xenakis’s ideas about the graphic control of sound are still valid today.
UPIC has strongly inspired other music programs that use graphical user interface to compose music, such as IanniX3, Audiosculpt4, MetaSynth5, and Hyperscore6. More recently, an application is released by Rodolphe Bourotte in the Apple Store on 8 September 2018 called UPISketch7.
3https://www.iannix.org/en/
4http://anasynth.ircam.fr/home/english/software/audiosculpt 5http://www.uisoftware.com/MetaSynth/
6https://hyperscore.wordpress.com
Trying to connect the audience with the composition through selected media by the composer is important for enhancing the audience’s experience. This method also has been used by John Cage. Cage encourages interaction between the artist, performer and the audience in his own electronic theatre work Variations V (Cage, 1967). Interactive music systems also offered new ways of listening for the audience and blurred the boundaries between the composer and the audience, opening a new fluid relationship between them. Roy Ascott explains the new role of the participant in modern artworks as follows:
[The spectator] no longer expects to receive a ready-made experience, or the expression of an experience, but rather to participate at a deep level, either in his consciousness or, more physically, by immediate action. The artist no longer decides everything and projects it as a whole in some definitive and final composition; he now initiates a dialogue, or set of events, which, when taken up by the audience, whether in a group or individually, will be shaped into totally unpredictable and indeterminate forms and experiences. (Ascott, 1968)
Chadabe also claims that “the ultimate significance of interactive composing is that it represents a new way for composers and performers to participate in a musical activity“ (Chadabe, 1984, p. 27).
All these developments were followed by interaction design in multimedia arts as we discussed earlier. Proprius resonates under these modern multimedia practices. This research is examining the real-time interactive approach by tracking the location of the audience. Kinaesthetic sense, “proprioception”, is a sense of body position and movement. Henry C. Bastian coined the word “kinaesthetic” in 1880. Later, Charles S. Sherrington replaced the term “kinaesthetic sense” with “proprioception”. Without this sense we could not walk or could not touch our nose because we would not be able to find it. Proprioception is derived from the Latin word “proprius”, meaning individual, and “capio”, meaning to take or to grasp. This research, Proprius, is named after these definitions.
In Proprius, when listeners enter into an artificial environment augmented into real space,they become an element in the ecosystem. More specifically, they become a disease agent that lowers the health of organisms when they intersect in 3D real
environment. Biologically speaking, human agents are acting as a bioterrorism agents that can spread “virulence to harmless species” (Reece et al., 2014, p. 574). When a human agent lowers the health of a nearby organism, the latter runs away from the human agent and tries to find food in order to recover. However, the human agent lowers the health of an organism to a serious extent. This is why organisms have less time than usual to find food, which will eventually increase the chance of death of an organism.
2.2.3 Emergence
Generative artworks carry a certain degree of autonomy, which means that they are capable of working by themselves. Generative art uses algorithms for unexpected disordered behavior rather than usual randomness. Emergence is a central part of the generative art-making. There are a number of ways one can approach emergence. Collective behavior such as swarm-based algorithms (Shiffman, 2012); complex systems such as Cellular Automata (Conway, 1970); L-Systems (McCormack, 2004); evolutionary computation (e.g. generative algorithms; A-Life (Whitelaw, 2004) can be used to as methods to build environments.
Ecosystems are related to emergence because of their ability to generate randomness and unexpected new features in high level complex systems. Artists have created works out of biological emergent systems according to simple or more complex rules (A-Volve8, Life Spacies9, TechnoSphere10).
2.3 Augmented Reality Approach
The augmented reality (AR) approach relies on the superimposition of an aural virtual reality on a physical space that is to be explored by the user (e.g. Microsoft HoloLens11 and Meta 212). Virtual reality (VR) approach, on the other hand, aims to immerse the
8http://www.medienkunstnetz.de/works/a-volve/ 9http://www.medienkunstnetz.de/works/life-spacies/ 10http://v2.nl/archive/works/technosphere
11https://www.microsoft.com/en-us/hololens 12https://www.metavision.com
users into the virtual world by blocking all real-world connection (e.g. Oculus Rift13 and HTC Vive14).
This research uses AR approach which simply means virtual space within real space. The user doesn’t wear any headset and doesn’t fully get immersed into virtual reality. Proprius is fed back to the user through headphones with binaural spatialization (See Figure 2.5).
Figure 2.5 : User experiencing Proprius.
2.4 Related Work
This section gives an overview of modern interactive music systems that explore the use of different ecological principles as a model of creativity. A large and growing body of artists and researches has investigated this model. This section is mostly focused on the approaches to using autonomous interactive agents in music composition or sonification.
Ecological models have been extensively used in algorithmic art and music composition. Furthermore, with advances in digital computing, it has become possible to simulate complex ecological systems in real-time. This has enabled the implementation of interactive works where user input affects the evolution of a system. In one of the earliest examples of interactive artificial environments, Christa Sommerer and Laurent Mignonneau’s A-Volve allows visitors to design virtual creatures and insert
13https://www.oculus.com 14https://www.vive.com
these into an artificial ecosystem to interfere with the relationship between preys and predators (Sommerer and Mignonneau, 1997). Visitors can interact with the creatures projected into a glass pool by inserting their hands into the water (See Figure 2.6).
Figure 2.6 : A-Volve.
A-Volve didn’t use any sonification. However, a tactile interaction method was used in the project, later followed by the mixed reality works that use sound like Archipelago where participants explore the artificial world with the tactile user interface (See Figure 2.7) (Ji and Wakefield, 2012).
The majority of the artistic works that involve ecological simulations rely on either the visual or the audiovisual domain (Antunes et al., 2014). Like Proprius, there are, however, works that rely solely on the auditory domain as well. For instance, Living Melodies (1999) is a sonic ecosystem that has an evolutionary autonomous agent population who can sing to each other to generate musical compositions. This artificial environment of agents is capable of communicating between themselves in sound.
A simple genetic programming framework in a procedural language used by Palle Dahlstedt and Mats G. Nordahl allows the evolution of foraging behavior and movement to reproduce sonic and musical communication (Dahlstedt and Nordahl, 2001). The two are using an interactive evolution approach which is an evolutionary computation technique whose fitness function is replaced by a human user (Takagi, 2001). In biology, “the fitness of an organism is typically defined as the probability that the organism will live to reproduce or as a function of the number of offspring
Figure 2.7 : Archipelago.
the organism has” (Mitchell, 1998, p. 6). This method evaluates the choices of human participants (e.g. aesthetic preferences) and includes them into the system. The agents within the environment adapt, evolve, behave, and response according to the fitness function of a system. This way the human participant can be part of the reproduction and generation of the system. The agents can attain aesthetic value, and their survival value can be based on their aesthetic fitness.
In Living Melodies, Dahlstedt and Nordahl explore “coevolving musical communica-tion for artistic purposes” (Dahlstedt and Nordahl, 2001, p. 243). The chorus of the mating calls between these agents result in a musical composition, where only the sonic structures that the agents find to be musically pleasing according to their genetic code become audible by the listener (Dahlstedt and Nordahl, 2001). They implement mating calls used by animals to attract and find mates. The agents have to make a sound to find a mate and reproduce. They are designed to have an innate “listening pleasure” instinct to “sing” (2001, p. 245). The system uses note-based 12 tone interval mapping on piano sounds via MIDI. The first note that the creatures sing indicates their species. The creatures decide on the amplitude level of a sound based on their listening
pleasure, which eventually makes the audience hear or not hear the sound. Simply put, the audience can hear the sounds if the creatures like it or find it “musically interesting” (2001, p. 243).
Mating behavior of animals has been studied by many researchers. Like the agents of Living Melodies, the agents of Gakki-mon Planet also use sound for mating. Gakki-mon Planet (2000) is an audiovisual interactive musical environment designed by using a genetic algorithm by Rodney Berry and collaborators Palle Dahlstedt and Catherine Haw (Figure 2.8). The “musical creatures” follow the flow of energy principle (Berry et al., 2001). The creatures have a complex ecological behavior. They have to eat to survive, find a mate to reproduce, and they die if their energy level is negative.
Figure 2.8 : Visualization of Gakki-mon Planet.
In biology, any organism can be described with its genotypes and phenotypes (McFar-land, 1993). Genotype is the genetic code of an organism. The physical expression of a genotype is called phenotype. Phenotypes are the physical characteristics of organisms. When we look at an animal we can’t see its genes (genotype) but we can see its physical appearances (phenotype). In generative and evolutionary art systems that use genetic algorithms, a genotype and phenotype analogy is commonly used. Genotype simply refers to the software of an artwork or the part that is not experienced by the user. Phenotype refers to the observable features of the artwork (e.g. sound, visuals). In Gakki-mon Planet, genotypes of the organism determine the physical appearance and musical behavior of the agents. The creatures living in the environment
can control sound synthesis parameters with their DNA. Human participants navigate within the environment through the use of a joystick by the guidance of the cube (Figure 2.9).
Figure 2.9 : White cube shows the position of the user in the world.
Participants can follow, capture, and feed the creatures. Following creatures develops a musical narrative (2001). As an additional interaction parameter, users can also play a MIDI keyboard to accompany the composition. MIDI parameters are also inherited from the DNA of the parents. Keyboard-controlled sound is linked to the last selected genomes to make the participants hear different timbres when they move in the environment (Berry et al., 2001). By introducing another interaction, they aim “to extend this interaction so that creatures can hear and respond to the music introduced by the player. In this way, the player becomes another creature in the virtual sonic environment” (2001).
As was mentioned in 2.2 (Compare Figure 2.1, 2.2, and 2.3) the level of communication between the artwork and users varies. Unlike Living Melodies, Gakki-mon Planet offers a circular relationship between machines and humans, in other words between music and the audience. That is to say, the output result will change the behavior of the audience which in turn will change the behavior of creatures.
Artist Jon McCormack’s audiovisual installation Eden (2000-2011) is an evolutionary agent-based audiovisual sonic environment populated by “rocks, biomass and sonic agents” (McCormack, 2002, p. 4). The project explores “artificial life system for music
composition” (McCormack, 2001, p. 133). The user’s position determines where in the artificial ecosystem resources will be generated. As a result, instead of singing to each other like the creatures in Living Melodies, the artificial agents in Eden sing to the audience to attract their attention so as to increase the resources available to them. In Eden, creatures start with little knowledge about their environment when they are born. Later, they learn about their environment and pass the knowledge to their future generations. They learn how to eat, how to reproduce, and how to avoid predators while competing for limited food sources. They can mutate between their actions, such as from eating to singing. They can hear sounds in the environment made by other organisms. Agents can navigate, sing, feel, see, listen, mate in the virtual ecosystem, and communicate with the human participants.
The project involves multiple screens, projectors, speakers, and IR sensors (Figure 2.10). Sensors in the exhibition space track the movements of participants; when the audience stays still at a location, the system begins to produce resources at this location. Over time, the agents become aware that when they generate “interesting” sequences of sound, the audience tends to stay where they are to pay attention to it (McCormack, 2001, p. 136). After many iterations of the simulation, the agents begin to perform what is perceived by the audience to be more interesting sounds so that they can captivate them to generate resources (i.e. food).
