A Perception-Centric Framework for Digital Timbre Manipulation in Music Composition

MISSING IMAGE

Material Information

Title:
A Perception-Centric Framework for Digital Timbre Manipulation in Music Composition
Physical Description:
1 online resource (100 p.)
Language:
english
Creator:
Smock, Brandon
Publisher:
University of Florida
Place of Publication:
Gainesville, Fla.
Publication Date:

Thesis/Dissertation Information

Degree:
Doctorate ( Ph.D.)
Degree Grantor:
University of Florida
Degree Disciplines:
Computer Engineering, Computer and Information Science and Engineering
Committee Chair:
RANGARAJAN,ANAND
Committee Co-Chair:
WILSON,JOSEPH N
Committee Members:
GADER,PAUL D
BANERJEE,ARUNAVA
SAIN,JAMES P

Subjects

Subjects / Keywords:
composition -- instantaneous -- mds -- music -- perception -- space -- timbre
Computer and Information Science and Engineering -- Dissertations, Academic -- UF
Genre:
Computer Engineering thesis, Ph.D.
Electronic Thesis or Dissertation
bibliography   ( marcgt )
theses   ( marcgt )
government publication (state, provincial, terriorial, dependent)   ( marcgt )

Notes

Abstract:
In this work, a new framework is developed for the unrestricted manipulation of timbre in musical composition. Unlike other perceptual musical attributes such as pitch or loudness, timbre is not a one-dimensional property, and is not easily manipulated in a flexible yet precisely-controlled manner. The multi-dimensional nature of timbre has been previously studied using multi-dimensional scaling (MDS) techniques. The outcome of a study involving timbre and MDS is typically referred to as a timbre space. Previous timbre spaces have been suitable for a high-level description of some of the underlying dimensions of timbre, but not to a satisfying degree, and have not been suitable for subsequent synthesis of timbre. In this work, four design goals are proposed for a timbre space suitable for composition, along with ways in which to satisfy them. They are: perception, synthesis, interface, and modularity. The primary focus of this work is on the first goal of perception. Understanding the mental representation of timbre at a fundamental level is vital to achieving an effective and compact system for timbre manipulation. Due to the current lack of a satisfying model of timbre perception, we devise new methods and propose new ideas to develop one. As a result, contributions of this work include an improved experimental procedure for the study of perceptual spaces, theoretical contributions to the study of timbre, a novel timbre space suitable for synthesis, a new periodic waveform analysis technique, and a detailed blueprint for the implementation of a system for timbre manipulation in composition.
General Note:
In the series University of Florida Digital Collections.
General Note:
Includes vita.
Bibliography:
Includes bibliographical references.
Source of Description:
Description based on online resource; title from PDF title page.
Source of Description:
This bibliographic record is available under the Creative Commons CC0 public domain dedication. The University of Florida Libraries, as creator of this bibliographic record, has waived all rights to it worldwide under copyright law, including all related and neighboring rights, to the extent allowed by law.
Thesis:
Thesis (Ph.D.)--University of Florida, 2014.
Local:
Adviser: RANGARAJAN,ANAND.
Local:
Co-adviser: WILSON,JOSEPH N.
Statement of Responsibility:
by Brandon Smock.

Record Information

Source Institution:
UFRGP
Rights Management:
Applicable rights reserved.
Classification:
lcc - LD1780 2014
System ID:
UFE0046695:00001


This item has the following downloads:


Full Text

PAGE 1

APERCEPTION-CENTRICFRAMEWORKFORDIGITALTIMBREMANIPULATIONINMUSICCOMPOSITIONByBRANDONSMOCKADISSERTATIONPRESENTEDTOTHEGRADUATESCHOOLOFTHEUNIVERSITYOFFLORIDAINPARTIALFULFILLMENTOFTHEREQUIREMENTSFORTHEDEGREEOFDOCTOROFPHILOSOPHYUNIVERSITYOFFLORIDA2014

PAGE 2

c2014BrandonSmock

PAGE 3

Tomymom,thehardest-working,mostsupportive,andmostlovingpersonIknow

PAGE 4

ACKNOWLEDGMENTS First,ImustthankmyadvisorDr.AnandRangarajan.Withouthim,Ineverwouldhaveundertakenthisjourney.Hisencouragementandsupportwereabsolutelyvitaltomysuccess.ImustalsothankDr.JosephWilson,whoIthinkofasasecondmentor.Ourimpromptuconversationsinthelabwerenumerousandextremelyhelpful.Heoversawmynon-dissertationresearchandwentoutofhiswayprovidingguidancethathasgreatlycontributedtothescholarIamtoday.IwouldliketothankDrs.PaulGaderandArunavaBanerjee.EncouragementIreceivedfromyouasamaster'sstudentstuckwithmeasImovedontowardmyPh.D.IadmirebothofyougreatlyandwishIhadmoreopportunitiestoworkwithyou.IwouldalsoliketothankDr.JamesPaulSain,withwhomIhadnumerousconversations,andwhopatientlyandhappilylledingapsinmyformalknowledgeofmusicandtheory.TomyfellowlabmatespastandpresentintheComputationalScienceandIntelligencelab,Iwanttothankyouforallowingmetodrawuponyourvastknowledgeandtalents.Discussingnewideaswithyouwasextremelyhelpful,andIfeltthateverytimewecollaborated,goodthingshappened.ConversationswithTaylorGlennstandoutinmymindasbeingparticularlyhelpfulthroughoutmytimeasagradstudent.AlinaZaresetastrongexampletofollowearlyon.Thanks,ofcourse,gotomystudyparticipantsforvolunteeringandtakingthetimeoutoftheirbusyschedulestohelpmewithmyresearch.Finally,thankyoutomyfriends.Yoursupporthasbeeninvaluable,youhaveinspiredmewithyourdriveandabilities,andyouhavehelpedtomakemytimeasagradstudentsomeofthebestyearsofmylife! 4

PAGE 5

TABLEOFCONTENTS page ACKNOWLEDGMENTS ................................. 4 LISTOFTABLES ..................................... 7 LISTOFFIGURES .................................... 8 ABSTRACT ........................................ 10 CHAPTER 1INTRODUCTION .................................. 11 2LITERATUREREVIEW .............................. 17 2.1Timbre ...................................... 17 2.2Multi-DimensionalScaling(MDS) ....................... 22 2.2.1EuclideanorMetricMDS ........................ 24 2.2.2Non-MetricMDS ............................ 25 2.2.3WeightedEuclideanMDS ........................ 28 2.2.4SpecicitiesandLatentClasses .................... 30 2.2.5Isomap .................................. 30 2.3TimbreSpaces .................................. 32 2.3.1HistoryofTimbreSpaces ........................ 33 2.3.2ShortcomingsofTimbreSpaceResearch ................ 35 2.4RankingMethods ................................ 38 2.5Perceptually-BasedMusicalSoundSynthesis ................. 42 2.5.1ModelsofDynamicMusicalSoundSynthesis ............. 43 2.5.2InterfacesforTimbreControl ...................... 45 3METHODOLOGY .................................. 49 3.1DesignGoals .................................. 49 3.2Perception .................................... 51 3.2.1InstantaneousTimbreandDynamicTimbre ............. 52 3.2.2TimbreSpaceExperiment ....................... 54 3.2.3DeterminingandInterpretingPerceptualAxes ............ 60 3.2.4AnalyticalMethodsofPerceptualAxisExplanation ......... 61 3.2.5ExploratoryMethodsofPerceptualAxisExplanation ........ 63 3.2.6FindingAxesViaLeast-SquaresProjection .............. 64 3.3Synthesis ..................................... 65 3.3.1AdditiveSynthesisRepresentation ................... 66 3.3.2MaskedInverseFastFourierTransform ................ 68 3.4Interface ..................................... 71 3.4.1HierarchicalorLayeredApproach ................... 72 3.4.2OrthogonalAxesasControlParameters ................ 73 5

PAGE 6

4RESULTS ....................................... 75 4.1TimbreSpaceExperiment ........................... 75 4.2JointInstantaneousTimbreSpace ....................... 78 4.2.1ThePerceptualDimensionsofTimbre ................. 83 5CONCLUSIONS ................................... 90 REFERENCES ....................................... 96 BIOGRAPHICALSKETCH ................................ 100 6

PAGE 7

LISTOFTABLES Table page 3-1Alistofwordsusedbymusiciansandhowtheytintoaproposeddichotomyoftimbre.Inthisdichotomy,themostfundamentalpropertiesoftimbrewithnotemporalcomponentareknownasinstantaneousproperties,whilehigher-orderpropertieswithatemporalcomponentareknownasdynamic. .......... 54 3-2Adescriptionofthesoundsusedinthetimbrespaceexperiment.Mostofthesoundswerederivedfromanalysesofactualnoterecordings. ........... 57 3-3Thesevenacousticfeaturesandtheirformulas,basedontheadditivesynthesisrepresentationinEquation 3{1 ........................... 62 3-4Thevaluesofthesevenacousticfeaturesforeachofthe16sounds. ....... 62 4-1ThesquarerootsoftheweightscomputedbyINDSCALforsixsubjectsandfourdimensionsofinstantaneoustimbre,normalizedtosumto1. ........ 81 4-2ThesquarerootsoftheweightscomputedbyINDSCALforsixsubjectsandvedimensionsofinstantaneoustimbre,normalizedtosumto1. ......... 81 4-3ThesquarerootsoftheweightscomputedbyINDSCALforsixsubjectsandsixdimensionsofinstantaneoustimbre,normalizedtosumto1. ........... 82 4-4Thecorrelationbetweentheprincipalaxes~d1;~d2;:::;~d6determinedbyINDSCALandtheindividualembeddingsofthesixsubjects. ................. 84 4-5Thecorrelationsbetweentheprincipalaxes~d1;~d2;:::;~d6determinedbyINDSCALandsevenacousticproperties~p1;~p2;:::;~p7ofthesounds. ............. 85 4-6Thenewcorrelationsbetweentheprincipalaxes~d1;~d2;:::;~d6determinedbyINDSCALandsevenacousticproperties~p1;~p2;:::;~p7ofthesoundsafterremovingsounds3,6,and8fromthecorrelationcalculations. ................ 86 4-7AsummaryoftheinterpretationsofthedimensionsofinstantaneoustimbrefoundbyINDSCAL. ................................. 87 4-8Thecorrelationbetweensevenacousticproperties~p1;~p2;:::;~p7ofthesoundsandtheirbest-matchingaxesinthejointembedding. ............... 87 4-9Thecorrelationbetweensevenacousticproperties~p1;~p2;:::;~p7ofthesoundsandtheirbest-matchingaxesintheindividualembeddingsofeachsubject. ... 88 7

PAGE 8

LISTOFFIGURES Figure page 3-1Thisgureshowsthepipelineofderiveddataintheinstantaneoustimbrespaceexperiment.Foreachgroupofsoundspresented,thesubjectprovidessimilarityjudgments,whicheitherspecifywhichtwosoundsarethemostsimilarorleastsimilarsoundsinthegroup.Fromasimilarityjudgment,multiplebinarycomparisonsarederived.Allofthebinarycomparisonsareusedtoderivearankingofthedistancesbetweensounds,usingColley'smethod.Fromtherankingofdistances,aspatialembeddingisderivedusingKruskal'snon-metricMDSalgorithm.TheEuclideandistancesbetweenpointsarecalculatedfromthisspatialembedding.Finally,ajointembeddingisderivedforallofthesubjectsfromtheirindividualEuclideandistancesusingINDSCAL. ........................ 55 3-2L(x),theloudnessproportion,whichisaweightingoffrequenciesbytheirperceivedloudness,derivedfromISO226:2003[ 1 ],theequal-loudness-levelcontours.Thecurveisnormalizedtohaveameanvalueof1. ................... 63 3-3Illustrationoftheexpectedcorrelationbetweenarandomly-generatedsetofvaluesandthebest-matchingprojection(oraxis)foundinaspacewithrandomly-generatedpoints,asthenumberofpointsanddimensionsisvaried.Forthecaseof12pointsin10dimensions,theexpectedcorrelationis0.9.Thismeansthatinsuchaconguration,correlationsmustbeabove0.9tobemoremeaningfulthanrandom. ....... 66 3-4AnillustrationofdisjointsupportamongtherstthreeharmonicsofaC5trumpetnote,usingaportionoftheDiscreteFourierTransformofthesignalcorrespondingtothesound.Eachlargepeakinenergycorrespondstoaharmonic.Thedashedlinesrepresenttheboundariesofeachharmonic'ssupport.Notethattheenergyforeachharmonicisconcentratedwellwithintheboundariesofsupport,demonstratingthateachharmonichasessentiallydisjointsupport. ................ 69 3-5Aportionofthetime-domainsignalsfortherstsixharmonicsduringtheonsetofaC5trumpetnote,resolvedusingtheMaskedInverseFastFourierTransform(MIFFT)method. .................................. 70 3-6Theproposedhierarchyfortheentiresystem,servingasaninterfacefromhigh-leveltimbrespecicationtolow-levelsynthesis.Anynumberofhigh-levelobjectscanbeusedtointerfacewiththeinstantaneoustimbrespace.Eachobjectatthehighest-level,suchasagestureorverbaldescription,mapstoanumberofpointsintheinstantaneoustimbrespaceandspeciestheirorder.Thisabstractiongreatlysimpliesthespecicationfortheuser.Pointsininstantaneoustimbrespacethenmapdirectlytosynthesisparameters,preferablyinaone-to-onemannersothattheevolutionofasoundcouldbespeciedinreal-timeifnecessary. ... 73 8

PAGE 9

4-1TheP-by-PcomparisonmatrixforsubjectM.Eachrowandeachcolumnrepresentsapairofsounds.Eachsquareindicatestheoutcomeofthecomparisonbetweentherowpairandthecolumnpair.Awhitesquareindicatesthatthesoundsintherowpairarejudgedbythesubjecttobemoresimilarthanthesoundsinthecolumnpair.Ablacksquareindicatesthatthesoundsintherowpairarejudgedbythesubjecttobelesssimilarthanthesoundsinthecolumnpair.Agraysquareindicatesthatthereisnocomparisonbetweenthetwopairs. .... 76 4-2ThesamecomparisonmatrixasinFigure 4-1 exceptwiththepairsorderedfrommostsimilartoleastsimilarbyColley'smatrixmethod.Blacksquaresabovethemaindiagonalandtheircorrespondingwhitesquaresbelowthemaindiagonalrepresentviolationsoftransitivity,sincetheyindicatethatapairthatisrankedmoresimilarthananotherpairisjudgedbythesubjecttobelesssimilar. .... 77 4-3PlotofthestresscalculatedforthecongurationfoundbyKruskal'snon-metricMDSforeachsubjectforeachnumberofdimensions.Thereductioninstressappearstostopbeingsignicantafteradimensionalityofeither5or6. ..... 77 4-4Firstthreedimensionsoftheindividualinstantaneoustimbrespaceforsubject2. 78 4-5Firstthreedimensionsoftheindividualinstantaneoustimbrespaceforsubject6. 79 4-6Firstthreedimensionsofthe4-dimensionalINDSCALembeddingofjointinstantaneoustimbrespace. ..................................... 80 4-7Firstthreedimensionsofthe5-dimensionalINDSCALembeddingofjointinstantaneoustimbrespace. ..................................... 80 4-8Firstthreedimensionsofthe6-dimensionalINDSCALembeddingofjointinstantaneoustimbrespace. ..................................... 81 4-9Visualcomparisonofthersttwodimensionsofeachsubject'sweightedembeddingasproducedbyINDSCAL.Subjectsdonotexhibitmuchdierenceinthesedimensions 83 4-10Visualcomparisonofthethirdandfourthdimensionsofeachsubject'sweightedembeddingasproducedbyINDSCAL.Subjectsstarttoexhibitnoticeabledierencesinthesedimensions. ................................. 84 4-11Thex-axisvaluesarethelogfundamentalfrequenciesofthe16sounds.They-axisvaluesarethebest-matchingprojectionsintheindividualembeddingsofinstantaneoustimbrespaceforsubjects5and6.Thecorrelationsbetweenthesetsarelistedincolumn3ofTable 4-9 ....................... 88 4-12Theinharmonicityiscalculatedastheproportionally-weightedstandarddeviationoffi=(if0)foralloftheharmonics.Thisgraphshowshowthecorrelationofthisstatisticwithsubjects'individualinstantaneoustimbrespaceschangesasthevalueisraisedtodierentexponents.Thecorrelationpeaksatdierentexponentsforeachsubject. ............................. 89 9

PAGE 10

AbstractofDissertationPresentedtotheGraduateSchooloftheUniversityofFloridainPartialFulllmentoftheRequirementsfortheDegreeofDoctorofPhilosophyAPERCEPTION-CENTRICFRAMEWORKFORDIGITALTIMBREMANIPULATIONINMUSICCOMPOSITIONByBrandonSmockMay2014Chair:AnandRangarajanMajor:ComputerEngineeringInthiswork,anewframeworkisdevelopedfortheunrestrictedmanipulationoftimbreinmusicalcomposition.Unlikeotherperceptualmusicalattributessuchaspitchorloudness,timbreisnotaone-dimensionalproperty,andisnoteasilymanipulatedinaexibleyetprecisely-controlledmanner.Themulti-dimensionalnatureoftimbrehasbeenpreviouslystudiedusingmulti-dimensionalscaling(MDS)techniques.TheoutcomeofastudyinvolvingtimbreandMDSistypicallyreferredtoasatimbrespace.Previoustimbrespaceshavebeensuitableforahigh-leveldescriptionofsomeoftheunderlyingdimensionsoftimbre,butnottoasatisfyingdegree,andhavenotbeensuitableforsubsequentsynthesisoftimbre.Inthiswork,fourdesigngoalsareproposedforatimbrespacesuitableforcomposition,alongwithwaysinwhichtosatisfythem.Theyare:perception,synthesis,interface,andmodularity.Theprimaryfocusofthisworkisontherstgoalofperception.Understandingthementalrepresentationoftimbreatafundamentallevelisvitaltoachievinganeectiveandcompactsystemfortimbremanipulation.Duetothecurrentlackofasatisfyingmodeloftimbreperception,wedevisenewmethodsandproposenewideastodevelopone.Asaresult,contributionsofthisworkincludeanimprovedexperimentalprocedureforthestudyofperceptualspaces,theoreticalcontributionstothestudyoftimbre,anoveltimbrespacesuitableforsynthesis,anewperiodicwaveformanalysistechnique,andadetailedblueprintfortheimplementationofasystemfortimbremanipulationincomposition. 10

PAGE 11

CHAPTER1INTRODUCTION Themusicalvalueofthecomputerdoesnotlie,ofcourse,initsabilitytoduplicateexactlywhatarealinstrumentcando,butratherinyieldinganextendedrepertoryofsounds,includingandgoingbeyondtheclassesofsoundsofactualinstruments.|RissetandMathews,1969Music,itcouldbesaid,iswheresoundmeetspsychology.Whilesoundcanbeobjectivelymoved,measured,analyzed,andmultiplied,aperson'sresponsetomusicisanintenselysubjectiveexperience,anamalgamationoffeelingsaboutfrequencies.Despitethissubjectivity,therearecommonalitiestothehumanexperiencederivedfrompossessingsimilarperceptiveabilities.Thisallowsustoexpressourselvesthroughmusic,forinstance,andhaveareasonableexpectationthatwiththerightbackgroundandexperience,ourmessagecanbeunderstood.Musiciansdescribenotesashavingpitch,havingloudness,havingduration,andhavingtimbre.However,thesearenotphysicalquantitiesbelongingtothenotesthemselves.Theyarequalitativepropertiesbelongingtothenotes'perception.Buteveniftheyareexperiencedqualitatively,therearewaysinwhichwecanmeasureandquantifytheseresponsesinthebrain.Withoutmuchthought,anexperiencedlistenercanjudgewhenonenoteisperceivedtobelowerorlouderorlongerthananother.Theabilitytoordernotesbypitch,loudness,anddurationspeakstothefactthatthoughthesepropertieshaveasubjectivequality,theyalsohaveaquantitativeaspect,describablemathematicallyevenifourbrainsdonotgivesusaccesstotherawnumbers.Theabilitytoordernotesbyaperceptualproperty,however,doesnotapplytotimbre.Thisisbecauseonlyone-dimensionalquantitiescanbeorderedandtimbreisnotaone-dimensionalproperty.Butevenasamulti-dimensionalattribute,timbreisill-dened.Timbreisoftensaidtobethataboutanote,whichallotherthingsbeingequal,allowsonetodistinguishbetweentwomusicalinstruments,suchasatromboneandaclarinet.Butthisisadescriptionratherthanadenition.Inasense,timbreistheunconscious 11

PAGE 12

aggregationofmultiplesimultaneousperceptionsaboutthesourcecharacteristicsofanote.Buthowtheseperceptionsmanifestthemselvesinasingularrepresentationisnotentirelyclear.Acomposerdoesnothavetofullyunderstandthepsychologyoftimbretousetimbre.Allcomposersusetimbrebecausetheycomposeformusicalinstruments,andmusicalinstrumentsinvoketimbre.Butusingtimbreisnotnecessarilythesamethingasutilizingtimbre.Thedecisionmadebyacomposerthathasthegreatesteectonacomposition'stimbreislikelytobethechoiceofinstrumentitself.Thisishistoricallyfairlylimitedbecauseacomposerhastoconsiderhowwidespreadaninstrumentisifhewantstoensurethathispiececanbeperformed.Instrumentsrequireagreatinvestmentoftimetobelearned,thuslimitingthenumberofinstrumentsthatcanbewidespread.Somemeasureoftimbraldiversitycanbeachievedsimplybyswitchingbetweenmusicalinstrumentsinacomposition.Withinthesameinstrument,additionalmanipulationoftimbreisalsopossiblebecauseinstrumentstypicallypossessexibilityintheirsoundproduction.Butpitchmanipulationismuchmoreaccessibletocomposersandmusicians.Atypicalorchestralinstrument'srangeofpitchesmightcoverathirdofthetypicalrangeutilizedinallofmusic,whileitstimbralrangecoverssuchasmallfractionitcanhardlybequantied.Becausetimbremanipulationisnotasaccessibletocomposers,itisnotspeciedonanywherenearthesamelevelaspitch.Someoftheprimarywaysinwhichtimbremanipulationiscommunicatedvianotationarethroughtheuseofmarkingsindicatingstyleandarticulation.Stillmoremanipulationcanbesuggestedbynotatingnaturalharmonicsinplaceoftypicalnotes.Precisecontroloversoundproduction,however,israrelynotatedbycomposers.Often,anumberofchoicesthatdirectlyaecthowthetimbreofanoteisperceivedareleftuptotheperformer.Anumberofthetechnologicalandlogisticalrestrictionsontimbreproductionandspecication,however,arehistoricalinnature.Inmoderntimes,theopportunitiesfor 12

PAGE 13

acomposertoexertcontrolovertimbrearemuchgreater.Digitalrecordingtechnologyremovesmostbarrierstoacomposer'sworkbeingheard.Therefore,composerscanbemorediverseintheirselectionofmusicalinstruments|andthus,timbres|forwhichtowrite.AsRissetandMathewsnote,thecomputerispotentiallyanymusicalinstrument.Playingitisgreatlyaidedbyprogramsthatabstractmanyofthelow-leveldetailsofsoundproductionawayandreplacethemwithintuitiveinterfaces.Unliketheinterfacesoftraditionalinstruments,relyingonthephysicsofresonanceinacolumnofairoronastring,theseinterfacescantakeanyform.Thusthebarriertobecomingadigitalvirtuosoismuchlowerthanfortraditionalmusicalinstruments.Evenwithadvancesindigitalsoundsynthesisandreproduction,however,timbreremainsaqualityofmusicthatisnotconsciouslydirectedonanywherenearthesamelevelaspitchorloudness.Perhapsthisisduetothefactthattimbreitselfisstillnotperfectlyunderstood.Thestudyoftimbrehasbeenintimatelyrelatedtothestudyofharmonics,andinthissenseitcanbetracedatleastasfarbackasthe17thcentury,whenMarinMersennereportedhearingindividualharmonicsandobservedthatastringiscapableofvibratingatharmonicfrequenciessimultaneously[ 2 ].Inthe19thcentury,largerquestionsaboutauditoryperceptionwereposedanddebated.In1843,GeorgSimonOhmproposedthattheeardecomposesaperiodicwaveformintoitsconstituentharmonics[ 3 ].Inthesamedecade,AugustSeebeckarguedthatharmonicscontributetooursensationofpitchatthefundamentalfrequencyandthattheproportionsofharmonicsinasoundmustimparttimbre[ 3 ].In1863,HermannvonHelmholtzpublishedhisground-breakingworkOntheSensationsofToneasaPhysiologicalBasisfortheTheoryofMusic,applyinghisexperimentalacumenandknowledgeofphysicstoquestionsofauditoryand,inparticular,musicalperception.DespitethefoundationlaidbyHelmholtz,thesubjectivenatureoftimbreanditscomplexcorrespondencewithacousticsremainedbarrierstoafulltheoryoftimbre.It 13

PAGE 14

wasnotuntil1940thattimbrebegantobedescribedexplicitlyasamulti-dimensionalphenomenon.Modernstudyoftimbrebeganinearnestinthe1970'sfollowingthematurationofmulti-dimensionalscaling(MDS)techniques.Thesetechniquesallowedthemulti-dimensionalnatureoftimbretobestudieddirectlyinaspatialrepresentation.Studiesofthisnaturehaveconcludedrepeatedlythatbrightnessandspectraluxareoverwhelmingcontributorstooursenseoftimbraldierencebetweennotes.However,thestudiesthemselveswerenotsuccessfulinilluminatingtimbreatafundamentallevel,noraretheyapplicabletothesubsequentgenerationofnewsoundsfromthespatialrepresentation.Onemajorbarriercurrentlytotimbremanipulationinaperceptualframeworksuchasatimbrespaceisthelackofanexplicittemporalcomponent,sothatsynthesizedtimbrescanbemadeofarbitrarylengthsandevolveinresponsetochangingstimuli.Tothisend,weproposeanewtimbrespaceandisolateinstantaneousaspectsoftimbrefromdynamicaspectsoftimbre.Weassertthatsuchadichotomyexistsinvisualperceptionandarguethatstudyinginstantaneousanddynamicaspectsoftimbresimultaneously,ashasbeendoneinprevioustimbrespacestudies,muddlestheunderstandingofboth.Isolatedfromdynamicaspects,wecanexplicitlyinvestigatethemulti-dimensionalnatureofinstantaneoustimbrealone.DrawingonanumberofMDStechniques,wecanaskifaclearorthogonalrepresentationexists,whatthemeaningofitsaxesare,andperhapseveninvestigatemultiplecoherentorthogonal,orperhapsevennon-orthogonal,representationsofthesamespaceofinstantaneoustimbre.Allofthisismeanttoservethelargergoalofachievingunrestrictedtimbremanipulationinmusiccomposition.Thoughimplementationofsynthesisforsuchasystemmaybepossiblebyskirtingthenatureoftimbrealtogether,wearguethatachievingprecisecontroloveraninherentlyperceptualphenomenonisdonemosteectivelywithperceptionasaprimaryfocus.Thoughphysicsmaydrivemusicalproduction,perceptionistheonlythingthatmatters.Thoughnouniversalrepresentationmayexist,explicitly 14

PAGE 15

dealingwithperceptionoers,intheory,themostcompactandintuitiverepresentationpossible.Additionally,thebenetsofanorthogonalperceptualrepresentationareprecisionandclarity.Supposeauniqueperceptualrepresentationoftimbrecouldbeidentied.Methodsthatbypassperceptioncreatetheirownrepresentationthatisnotguaranteedtomaptothementalrepresentationoftimbreinaone-to-onemanner.Ifelementsoftherepresentationrelatetomentaltimbreasmany-to-one,theyareinecient,includingredundancyordistinctionswithnobearingonperception.Intheworstcase,theseleadtocontradiction,withmultipleelementsoftherepresentationtryingtoexertcontroloverthesameelementofperception.Ifelementsoftherepresentationareinaone-to-manyrelationshipwithtimbre,theyareambiguousandtherepresentationislossy.Thusaone-to-onemappingwiththeperceptionoftimbre,referredtohereasaperception-centricapproach,isessentialtoacompact,consistent,non-lossy,andpreciserepresentationoftimbre.Thismaynotbepossibleifhumansdiergreatlyintheirtimbralperceptions,butitmaybepossibletoachieveoptimallyonaverage.Thisworkpotentiallyrepresentsastepforwardinbothourunderstandingoftimbreandourimplementationofasystemforitsunrestrictedutilization.Butitisnotwithoutitslimitations.Indevelopingtherstinstantaneousspaceoftimbre,wehavehadtosimplifysomeofourexperimentstorstdemonstratetheirpossibleutility,sincetheexperimentalprocedureusedinthisworkisrelativelytime-consuming.Asaresult,thoughwebelievetherearenoinherentlimitationsonourexperimentalprocedure,thedatasetusedinthisworktolearnthespatialrepresentationdoesnotcovertheentirerangeofinstantaneoustimbres.Thislimitsustostudyingonlythosepropertiesthatcanbegleaneddirectlyfromourdataset.Afewpropertieswithsomepotentialimpactonperceptionthatcouldhavebeenstudied,suchasthephasesofharmonics,werealsoomittedduetotheiruncertaintyinasmalldataset.Withphaseinparticular,thereexistsnotheoryintheliteratureonhowitmightimpacttheperceptionspecicallyofpitched 15

PAGE 16

soundswithstationaryproperties,whicharetheonlysoundsusedintheexperimentinthiswork.Therefore,nodomainknowledgecouldbebroughtintoaidinitsanalysisforasmalldataset.Despitetheselimitations,webelievethereismuchtobelearnedfromourwork.InChapter2,wereviewindetailtheliteratureontimbrestudy,givingparticularattentiontoworktowardsmulti-dimensionalspatialrepresentations.InChapter3,weintroduceourtheoryoftimbre,whichisolatesinstantaneousaspectsfromthosethathaveatemporalcomponent.Weoutlineourexperimentalprocedureforlearningandinterpretingaspaceofinstantaneoustimbre.InChapter4,wedemonstrateourresults,includingasetofcommonperceptualaxesandtheirobjectiveandsubjectiveinterpretations.InChapter5,weanalyzetheresultsinmoredetailandplacethemintheirpropercontextwithinthelargereldoftimbrestudyandourgoalofimplementingasystemforunrestrictedtimbremanipulation.Wealsooutlinefuturedirectionsforourwork,detailinghowtoovercomethelimitationsarticiallyimposedbyourexperimentalprocedureandmoveevenclosertoachievingourprimarygoals.Contributionsofthisworkincludeanimprovedexperimentalprocedureforthestudyofperceptualspaces,theoreticalcontributionstothestudyoftimbre,anewtimbrespacesuitableforsynthesis,anewperiodicwaveformanalysistechnique,andadetailedblueprintfortheimplementationofasystemfortimbremanipulationincomposition. 16

PAGE 17

CHAPTER2LITERATUREREVIEWInthiswork,weproposeanewframeworkforachievingunrestrictedtimbremanipulationinmusicalcomposition.Achievingthisgoalrequiresknowledgeinanumberofareasincludingpsychoacoustics,signalprocessing,andmachinelearning.Asthesetofknowledgerequiredisratherdiverse,veryfewcomprehensiveframeworkshavebeenproposedforsynthesizingarbitrarytimbresfromaperceptualdescription,particularlyinmusicalcomposition.Although,anumberofpapershaveaddressedportionsofsuchaframeworkindepth.Wereviewthesehere.InSection 2.1 ,weoutlinethehistoryofformaltimbrestudy,beginningwithSeebeckandHelmholtzinthe19thcentury,andprogressinguptoanalysesinvolvingsophisticatedmachinelearningtechniquesintothepresentday.InSection 2.2 ,wereviewmulti-dimensionalscaling(MDS)techniquesindepth,asthesehaveformedthebasisfortimbrestudyformorethanthelast40years.InSection 2.3 ,weintroducetheconceptofthetimbrespace,theprimaryresultoftheapplicationofMDStechniquestoformaltimbrestudy.InSection 2.3.2 ,wedetailsomeofthelimitationsofpriorstudiesinvolvingtimbrespace,whichweattempttoimproveuponinthiswork.InSection 2.4 ,weintroducerankingmethods,theuseofwhichallowsustogeneralizekeyaspectsoftimbrespaceproductionandexpandthepotentialapproachesused.InSection 2.5.1 ,welookatgeneralmodelsofmusicalsoundsynthesis,focusingonadditivesynthesismethods,whicharetheoreticallygeneralenoughtosynthesizeanysound.Finally,inSection 2.5.2 ,weaddressapproachestotheproblemofinterface,whichisakeyissueforanysystemfortimbremanipulationincompositiontoactuallybeimplemented. 2.1TimbreTherstknowntheoryonthephysicalbasisoftimbreperceptionwasarguablyproposedbyAugustSeebeckin1844.AmongadisputewithGeorgOhmoverpitchperception,Seebeckproposedthatthehigherharmonicsofafundamentalfrequency, 17

PAGE 18

ratherthanthemselvesbeconsidereddistincttones,mustreinforcetheperceivedpitchofthefundamentaltoneanddetermineitstimbre[ 3 ].ButSeebeckdidlittletofurthertheidea.ThisislikelywhyitisoftencreditedtoHermannvonHelmholtz,whoinhis1863workdefendedOhm'snotionofpitchperceptionandaugmenteditwithahighly-developedtheoryofhowharmonicsariseandcontributetotimbre[ 4 ].Throughhisincredibleexperimentalacumen,itwasHelmholtz,notSeebeck,wholaidthefoundationforourunderstandingoftimbre.Muchhasbeendonetofurtherexplicateitsince.Yetourunderstandingoftimbrestillseemstoberatherincomplete.Timbreisdescribedbymusiciansusingwordssuchasbright,hollow,rich,dark,mellow,metallic,andwarm.Itiswhatgivesustheabilitytodistinguishthesamenoteplayedonatrumpetfromoneonaclarinet,orapianofromaviolin.Nomatteritsphysicalbasis,timbreisfundamentallyaperceptualproperty.Thefourprincipalperceptualpropertiesofanoteareconsideredtobepitch,loudness,duration,andtimbre,withtimbreencompassingallwhichisunclaimedbytherstthree[ 5 ].UnderthepropositionbySeebeckthattherelativeintensitiesoftheharmonicsarewhatdeterminetimbre,itistemptingtotheorizethatthefourperceptualpropertiesofanotehavedirectanaloguesinthefourphysicalpropertiesoffundamentalfrequency,amplitude,length,andovertoneseries.In1934,HarveyFletchershowedthattherelationshipbetweenthesepropertiesisnotone-to-one,however,andinfact,theperceivedtimbreofanoteisafunctionoffundamentalfrequency,amplitude,andovertoneseries[ 6 ].Helmholtzhimselfrecognizedthatourbasisfortheidenticationofmusicalinstrumentshasadditionalqualitiesnotcapturedstrictlybytherelativeintensitiesofpartials,includingtransientbehaviorattheonsetofanote.Buthesoughttofocusonjustthosestationarypropertiesthatareprimarilycapturedbytherelativeintensitiesofpartials,whathetermedmusicalquality,andinthisregardhewasparticularlyinsightful.Hedescribedtherelationshipbetweenseveralverbaltimbredescriptionsandtheircorrespondingdistributionsofenergyamongthevariousharmonics.Heconsidered 18

PAGE 19

howsomeinstrumentsproduceformants,muchlikethevocaltractintheproductionofvowels,andponderedthecontributiontomusicalquality.Heevennotedthattherstsixharmonicsofanoteformamajorchordandindicatedhowthepresenceofastrongseventhharmoniccouldcontributetoadissonantmusicalquality.Helmholtznotablywasalsothersttodemonstrateexperimentallythatphasehasnonoticeableeectonmusicalquality,althoughsomeresearchershaveobservedphaseeectsfortimbreingeneral[ 7 ].ItshouldbenotedthatHelmholtzwassuretopointoutthathislackofevidencethatphaseaectedmusicalqualitydidnotimplythatphasehasnobearingonotheraspectsoftimbre.Infact,hementionedthattheoreticalconsiderationsledhimtohypothesizethatphasealmostcertainlyhasaneectonaspectsoftimbreotherthanmusicalquality.PlompandSteenekennotethatHelmholtz'sconclusionsaboutphaseareoftenmisunderstood[ 7 ].Basedonthisandotherconsiderations,itisreasonabletoconcludethatanumberofresearchershaveoverlookedthedistinctionHelmholtzmadebetweenmusicalqualityandotheraspectsoftimbre.Thisisapointtowhichwereturnlaterinthissection.Manyconnectionshavebeenmadebetweenphysicalpropertiesofnotesandtheirperceptualeects.Shepardreferstothisasthepsychoacousticapproachtothestudyoftimbre[ 8 ].Forinstance,in1962,Fletcheretal.reportedtheirndingsthatthekeyfeatureindeterminingthewarmthofpianotoneswastheinharmonicityofthepartials[ 9 ].Discoveringhowsoundcanbemanipulatedphysicallytoproduceperceptualeectsisanimportantaspectoftimbreresearch.Butitstilldoesnotquiteexplaintheperceptualnatureoftimbreitself.Shepardreferstothestudyoftheinherentrelationshipsbetweenperceivedstimuliindependentoftheirrelationshiptophysicalparametersasthecognitivepsychologicalapproachtotimbrestudy.Indeterminingthefullperceptualbasisoftimbre,itisinsightfultodrawananalogywithcolorperceptioninvision.Helmholtznotesthatatleastasearlyas1686,itwasknownthatallcolorscanbeshowntobeacombinationofthreefundamentalcolors. 19

PAGE 20

WhileHelmholtzdrewananalogybetweencolordecompositionandharmonic,orfrequency,decomposition,itisjustaseasytodrawananalogybetweencolorandtonecolor,whichisatermoftenusedinplaceoftimbre.Thatallcolorscanbedecomposedintothreefundamentalcolorsimpliesthattherearethreeorthogonaldimensionsofcolorperception.Inotherwords,colorscannotbeorderedororganizedspatiallyinamannerconsistentwithourperceptionsofsimilaritywithoutaspaceofatleastthreedimensions.DonLewisandM.J.Larsenweretherst,in1940,toexplicitlyrefertotimbreasamulti-dimensionalphenomenon[ 10 ].Thisisincontrasttopitch,loudness,andduration,whichcanallbeexpressedasone-dimensional(althoughtheremaybereasontoconsideramoregeneralnotionofpitchtobemulti-dimensional[ 8 ],butwithaneasilyidentiableone-dimensionalprojectionthatwenormallyrefertoaspitch).Notescanbeorderedbytheirpitchortheirloudness,butnotbytheirtimbre.In1890,Stumpfenumeratedover20verbalaxesoftimbresuchassmoothversusrough,andwideversusnarrow[ 11 ].However,thisenumerationdoesnotmakeitclearhowmanyorthogonalaxesoftimbrethereare,norhowtheirrelativestrengthsmightbeperceivedinourperceptionofdissimilaritybetweendierenttimbres.Abreakthroughcamewiththematurationofmulti-dimensionalscaling(MDS)techniquesinthe1960s[ 12 { 16 ].Thesetechniquesallowedperceptualjudgmentsaboutthedissimilaritybetweenobjectstobeaggregatedandttedtoaspatialrepresentationthatbestexplainedthejudgments.Inotherwords,experimentsaboutperceivedtimbredierencesbetweenpairsofsoundscouldbeusedtodetermineorthogonaldimensionsofperceptualtimbre.TheseexperimentsarecoveredindetailinSection 2.3 .AreviewofMDStechniquesiscoveredinSection 2.2 .Themainproblemwiththeseexperimentsforfullyexplainingtimbreperceptionisthattheydonotdistinguishbetweenfundamentalandcompoundaspectsoftimbre.Ithasbeendemonstratedbynumerousresearchersthatdynamic,orcompound,attributesofsoundcontributegreatlytoouroverallperceptionoftimbreandidenticationofmusical 20

PAGE 21

instruments.Butinmyopinion,Helmholtzwasastuteinrecognizingtheneedforasimpleterm(hismusicalquality)todescribethemostfundamentalaspectsoftimbreperception.UnfortunatelyitappearsasifnumerousresearchershavemisrepresentedHelmholtz'stheoryoftimbreasoversimplied,asifHelmholtzintendedforhistheorytobeacomprehensivedescriptionoftimbre.KaiSiedenburgandChristophReuterdiscussasrecentlyas2012howthecomposerIannisXenakisexposedthe\insuciencies"ofHelmholtz'stheorywithhis1956compositionPithopratka[ 17 ].Onthecontrary,itseemsmorelikelythatHelmholtzonlyintendedtoexplicatethemostfundamentalaspectoftimbre,whattodaymightbereferredtobysomeastonecolor.Unfortunately,thereisnoprecisedichotomybetweenfundamentalandcompoundaspectsoftimbreinthepresentvocabulary,suchthattonecoloronlyreferstothisfundamentalconcept.Toclarifythis,weproposeanewtermthatisfreeofanyambiguityandwhichcapturestheessenceofthemostfundamentalaspectsoftonecolor:instantaneoustimbre.WementionitherebrieybutexpoundonitmoreinChapter3.Allthoseaspectsoftimbrethataremostfundamentalcanbedescribedasinstantaneousproperties,ordescribedasaquantityatasinglepointintime.Thisissimilartohow,incolorperception,thoseaspectsofcolorwhicharemostfundamentalcanbestudiedallwithinasinglepixel.Furthermore,tostudycolorperceptionatafundamentallevel,itcouldbeconfoundingtostudythedissimilaritybetweenobjectswithcompoundcolorproperties,suchasashinyblueballandamatteredblock.Thedegreetowhichanobjectismatteisacompoundpropertyofcolorperception,dependentontherelationshipsbetweencolorsoveranarea.Likewise,studyingnoteswithnon-stationarypropertiesmuddlesthedierencebetweeninstantaneoustimbreandtemporal,ordynamic,timbre.MuchofthemethodsIsetoutinChapter3willbemotivatedbythedesiretostudyinstantaneoustimbre,whichwillformthebasisforafundamentalspaceoftimbreontopofwhichcompositetimbreobjectscanbebuilt. 21

PAGE 22

2.2Multi-DimensionalScaling(MDS)Multi-dimensionalscaling(MDS)hasplayedamajorroleintimbreresearchsince1970.Beforeaddressingtimbrespacesinmoredetail,itseemsprudenttodiscusstheprecedingparalleldevelopmentoftheMDSmethodsthatunderlietheirestimation.TheoriginsofmodernMDScanbefoundinapaperfromYoungandHouseholder[ 18 ]from1938.TheyshowedthatacongurationofpointsinanN-dimensionalspacecouldbereconstructedgivenjustthesetofmutualdistancesbetweenthem.TheyalsoshowedthenecessaryconditionsforagivensetofsupposedmutualdistancestoactuallycorrespondtoacongurationofpointsinEuclideanspace,andhowtodeterminetheminimumdimensionalityofthespace.Acongurationofpointsisdenedtobeinvarianttorigidmotiontransformations.Thesearetranslation,rotation,andreection.Thusaninnitenumberofsetsofpointscorrespondtothesameconguration.Sinceonlyacongurationofpointscanberecoveredgiventheirmutualdistances,theabsolutecoordinatesoftheoriginalpointsarelostwhenrepresentedonlywiththeirmutualdistances.However,theirrelativecoordinatesarepreserved,whichisessentiallywhatYoungandHouseholdershowed.Afourthkindoftransformation,scaling,whichmultipliesallofthepointsbyaxednumber,doesnotpreservethedistancesbetweenthepointsbutdoespreservetheratioofthedistancesbetweenthepoints.Ifonlypreservingtheratiosisimportant,thenacongurationofpointscouldbedenedtobeinvarianttofourkindsoftransformations.Today,thetermmulti-dimensionalscalingreferstoanymethodthattriestondcoordinatesforpointsinanN-dimensionalspacethatbestpreserveasetofmutualdissimilarityvaluesbetweenthem.Pointscanrepresentanysetofobjectssuchaswords,faces,colors,orfoods.Dissimilarityvaluesareusuallythoughtofasmeasuresofdistancebuttheycanalsobemeasuresofcorrelationorevenjustassociation.MuchofthepromiseofMDSisthatoncepointsareplacedintoaspace,thespaceitselfcanbeanalyzedtodeterminetheunderlyingcausesoftheoriginaldissimilarityvalues.Inthisway,apurely 22

PAGE 23

mathematicalanalysiscanleadtoananalysisthatexplainswhatfactorscontributetothedissimilaritiesbetweenthepointsorobjectsintherstplace.Inpsychology,MDSiscommonlyusedtoestimatethedimensionsofaperceptualspace,suchasthementalrepresentationofhumanfaces.Thereareseveralmethodsusedtoestimatedissimilarityvaluesbetweenasetofobjects.Inthemoststraightforwardcase,subjectsareaskedtonumericallyratethesimilarityordissimilaritybetweeneachpairofobjectsinaset.Inthiscase,thesubjectsquantifythedissimilaritydirectly.However,theactofconsciouslyquantifyingwhatisnormallyaqualitativeperceptioncouldleadtoinconsistenciesinthevaluesprovidedbyanindividualsubject.Thisinconsistencycanbemitigatedbytheuseofpairedcomparisons.Ratherthanratingtheobjectsonanunfamiliarscalethatcanvaryovertime,objectsaremerelycomparedwitheachother.IfthereareNobjects,thereareN(N)]TJ /F1 11.955 Tf 12.32 0 Td[(1)=2possiblecomparisons.Whentheinformationisstrictlywhichobjectisgreater,withouttakingintoconsiderationbyhowmuch,eachcomparisonisabinarycomparison.Thesetofcomparisonscanthenbeanalyzedtodeterminearankingorratingofthestimuli.Arankingisanevenly-spacedrating,soitcontainsnoinformationonuncertaintyintherankings,whereasthenumbersinaratingcanhaveanyrelationship.Objectsthatareratedextremelyclosetogetherhavealotofuncertaintyintheirorderintheranking.Researchintoperceptualspacesdoesnotrankstimuli,butratherthepairwisedistancesbetweenallstimuli.SincethereareN(N)]TJ /F1 11.955 Tf 12.85 0 Td[(1)=2pairsgivenNobjects,thenumberofcomparisonsbetweenpairsisontheorderofN4.Onemethodthathasbeenusedtocompareandrankasetofdistancesisthemethodoftriads[ 19 ].Inthismethod,objectsarepresentedinsetsofthreeandthesubjectisaskedtodeterminewhichtwoobjectsarethemostsimilarandwhichtwoarethemostdissimilar.Amongasetofthreeobjects,thereare)]TJ /F4 7.97 Tf 5.48 -4.38 Td[(32=3waystopicktwoobjectsfromtheset,andthesethreepairsareorderedfrommostsimilartoleastsimilar.Quantitativedissimilarityvaluesarethenderivedfromtheentiresetofpairwisecomparisonsbysummingthenumberoftimesa 23

PAGE 24

pairwasjudgedmoresimilarthananotherpair.However,thiscoversonlyaportionofthetotalnumberofpossiblecomparisonsbetweenallpairsofpairs,andtheproportionitcoversgrowssmallerasthenumberofobjectsincreases.Furthermore,thesimplesummationoffavorablejudgmentsisimprecise,andevenwhenallofthecomparisonsobeytransitivity,tiesnaturallyarisebetweendierentpairs.Forthesereasons,weconsidertheuseofmoregeneralrankingmethods,coveredinSection 2.4 2.2.1EuclideanorMetricMDSYoungandHouseholder'smethodtoestimatethecoordinatesofpointsinaspaceisanexactprocedurethatworksfordistancesthatcanbeshowntocorrespondtoanactualcongurationofpointsinEuclideanspace|sometimescalledinfallibledata.Dissimilarityvaluesderivedfrompsychologicalexperimentshardlytthisdescription.EvenwithdistancevaluesthatareessentiallyEuclidean,noiseorotherslighterrorscancauseasetofdistancestonotcorrespondexactlytoanycongurationofpoints.In1952,TorgersonpublishedanextensiontoYoungandHouseholder'smethodthatcouldbeusedwithfallibledata[ 12 ].Inthiscase,acongurationisfoundthatminimizestheerrorbetweentheinputsetofdistancesandthedistancesbetweenthepointsintheoutputconguration.Evenallowingfornoisydata,however,Torgerson'smethodstillassumesdissimilaritiesthatareessentiallyEuclidean.AproblemariseswhenthedissimilaritiesareinsteadafunctionofEuclideandistances,likethoseestimatedfrompsychologicalexperiments.OnewaytodealwiththisproblemistorstconvertthedissimilaritiesintoEuclideandistances,usuallybyassumingtheformofthefunctionthatrelatesthetwoquantities.Torgersonoutlinedtwotypicalscenarioswhendissimilarityvaluesarenon-EuclideanandproposedmethodstotrytoconvertthemintoEuclideandistances.ThisworksaslongasthefunctionrequiredtoconvertasetofsimilaritiesintoasetofEuclideandistancesisknownorcanbedetermined.ButasShepard[ 13 ]pointedout,theformofthefunction 24

PAGE 25

cannotbeassumedingeneral.Sinceinmanysituationsofinterestdissimilarityvaluesarenon-Euclidean,thisposesamajorproblem. 2.2.2Non-MetricMDSNomatterwhattheformoftheunknownfunctionistoconvertfromdissimilaritytoEuclideandistance,though,thefunctionoughttobemonotonic.Thismeansthattheorder,orrank,ofthedissimilarityvaluesispreservedwhentheyareconvertedtoEuclideandistances.Shepard,therefore,proposedtousemonotonicityastheoptimalitycriterion.Inotherwords,thesetofpointsthatistobefoundinsomespaceofminimaldimensionalityistheonethatbestpreservestheorderofthedistances,ratherthanthedistancesthemselves.AccordingtoSheparditis\asurprisingoutcomethatthetwoconditionsofmonotonicityandminimumdimensionality(whichseemnonmetricorqualitativeinnature)aregenerallysucienttoleadtoauniqueandquantitativesolution"[ 13 ].Infact,Shepard'smethodevenestimatestheunknownmonotonicfunction.BecauseShepard'smethodonlyreliesontherankingofthedissimilarities,itwastherstmethodthatcouldbeuseddirectlyonanysetofdissimilarityvalueswithoutknowinganythingabouthowtheywerederived.Shepard'smethodisoutlinedasfollows.LetusassumethereareNobjectsandthusN(N)]TJ /F1 11.955 Tf 11.98 0 Td[(1)=2dissimilarityvalues,fijg,betweenalluniquepairsofobjects.First,createaninitialcongurationofNpointsin(N)]TJ /F1 11.955 Tf 12.26 0 Td[(1)-dimensionalspace.Shepardsuggestsstartingwithacongurationsuchthateachpointisequidistantfromallotherpoints(thisisstraightforwardtocreateforanyN).Rankthesetofdistances,fdijg,betweeneachpairofpointsandcomparetotherankingofthedissimilarities.Supposetherankingsareinascendingorderofdistance.Apairofpointswhoserankingistoohigh(aretoosimilar)comparedtotheirtargetrankingshouldbemovedawayfromeachother.Apairofpointswhoserankingistoolow(aretoodissimilar)shouldbemovedtowardeachother.Theamountbywhichtoadjustapairofpointsisproportionaltothedierencebetweentheiractualdistancerankandtheirtargetdissimilarityrank.Thuseachpointisactedonby 25

PAGE 26

\N)]TJ /F1 11.955 Tf 12.28 0 Td[(1forcesthataretendingtopullthatpointtowardsthoseotherpointsthataretoodistantandawayfromthosepointsthataretooclose."Afterthepointsareadjusted,therankingcomparisonisperformedagainandtheprocedureisiterated.AsShepardpointsout,foranyrankingofdistancesbetweenNpoints,acongurationofpointscanalwaysbefoundin(N)]TJ /F1 11.955 Tf 13.1 0 Td[(1)-dimensionalspacethatconformstotheserankings.Butthiscongurationmayhavelittleutilitysincethedimensionalityisonthesameorderasthenumberofpoints.Toinducethepointstosettleintoalower-dimensionalspace,Shepardperformsanotheradjustment,amonotonicwarping,duringeachiteration.Hetakespointsthatareclosetogetherandbringsthemevenclosertogether,andpushespointsthatarefarapartevenfartherapart.Thisisanalogoustotakingpointsthatlieonasemi-circleintwodimensionsandgraduallystretchingthesemi-circleintoaline,whichisone-dimensional.Afterconvergence,eventhoughthepointsmaylieinalower-dimensionalspace,theyarestillrepresentedusing(N)]TJ /F1 11.955 Tf 12.29 0 Td[(1)coordinates.Therefore,principalcomponentsanalysis(PCA)isperformedtondthelower-dimensionalsubspacethatthepointshavesettledinto.PCAidentiesorthogonalaxesthatsequentiallyaccountforthemostvarianceinthedata.Therstaxisrepresentsthedirectioninwhichthedatahasthemostvariance.Thesecondaxisrepresentsthedirection,orthogonaltotherst,whichhasthemostremainingvariance,andsoon.Whennearlyallofthevarianceinthedataisaccountedfor,therestoftheaxesmaybediscarded.Thisyieldsalower-dimensionalrepresentationthatstillaccountsfornearlyallofthevariance,orspatialextent,inthedata.ItmightseemasthoughonlyconstrainingtheMDSsolutiontohavethesamerankingofdissimilaritiescouldleadtotoowidearangeofpotentialsolutioncongurations.However,eachuniquepairofpointsrepresentsadissimilarityvalueandeachuniquepairofdissimilarityvaluesrepresentsaninequalityconstraintonthesolution.ThenumberofuniquepairsofasetisO(N2)sothenumberofinequalityconstraints,whichisuniquepairsofpairs,isO(N4).Inotherwords,itgrowsrapidlyasthenumberofpointsgrows. 26

PAGE 27

Therefore,asthenumberofpointssignicantlyexceedsthenumberoftruedimensions,theseeminglyweakconstraintofmonotonicitybecomesmuchstrongerandleadstoanessentiallyuniquesolution.Shepard'smethodisintuitiveandeectivebutperhapslacksamoreformaljustication.Kruskal[ 15 ]madeithisgoaltoimproveandformalizeShepard'smethods.Kruskalnotedthat\itdoesnotappearpossibletodescribe[Shepard's]procedureasonewhichminimizessomeparticularmeasurementofnonmonotonicity."Toremedythis,Kruskalproposedameasureofnon-monotonicity,asum-of-squareserrorhecalledthestress,givenbyEquation 2{1 ,S=vuut Pi
PAGE 28

toordertheotherincorrectlyrankedpairsofpointsfaster.Inthissense,Kruskal'smethodmaynotberadicallydierentbutmayconvergefaster.ThisisonewayinwhichamoreformalizedapproachledKruskaltoimproveuponShepard'smethod.Additionally,Kruskaldoesnotstretchthepointstoinducethemintoalower-dimensionalspace,amethodusedbyShepardthatisreasonablebutnotprincipled.Rather,hesuggestsrunninghisproceduremultipletimeswithdierentdimensionalitiesandlookingathowmuchthestressoftheminimum-stresscongurationgoesdownasthedimensionalityincreases.Wheneverincreasingthedimensionalityresultsinonlyamarginaldecreaseinstress,thentheproperdimensionalityhasbeendetermined.TheanswersproducedbyShepard'sandKruskal'smethodsarelikelytobesimilarbutnotexactlythesame.Kruskal'smethodcanbeviewedmoreeasilyintermsofoptimizingaprecisemeasureofmonotonicity.Infact,althoughnoformalproofisgiven,Kruskalevengivesareasontobelievehismethodmayproducethemaximum-likelihoodcongurationofpointsgivenjusttherankorderoftheirdistances.ItiswiththesekindsofconsiderationsthatKruskal'smethodisseenasaclear,ifmodest,improvementoverShepard's. 2.2.3WeightedEuclideanMDSWhileTorgerson'smethodisappropriateinanumberofcontexts,itsuseinpsychologicalapplicationscreatesanadditionalproblemtotheoneaddressedbyShepardandKruskal.Thatis,whenevermultiplesubjectsareaskedtoprovidedissimilarityvalues,thevaluesrarelyexhibitperfectagreement.Thequestionthenbecomes,howbestshouldonecombinetheconictinginformationfrommultiplesubjects?Horan[ 20 ],indescribingtheproblemandhowbesttoapproachit,usedthetermscommonattributespaceandindividualperceptualspace.Anattributespace,whichisindependentofanyindividualsubject,isaspaceofallthedimensions,orattributes,thatcontributetodissimilarityjudgmentofobjectsinthatspace.Anindividualperceptualspaceisanindividualsubject'sownversionoftheattributespace,whereanindividualmayweight 28

PAGE 29

somedimensionsasmoreimportantthanothersindissimilarityjudgments,orevenignoresomedimensionsaltogether.MoredetailonthismodelisgiveninSection 3.2.3 .TheindividualperceptualspaceiswhatwouldresultifMDSwereappliedtoasinglesubject'ssetofdissimilarityjudgments.ButrecoveringtheattributespaceisthelikelygoalofanyapplicationofMDSinapsychologicalexperiment.ThedefaultapproachtocombiningconictinginformationfrommultiplesubjectsmightbetoaveragethedissimilarityvaluesfromeachsubjectandthenperformMDSasusual.Horanshowedthatthisresultsinaspacethatdistortsboththeattributespaceandtheindividuals'perceptualspaces.In1970,CarrollandChang[ 21 ]proposedanalgorithmcalledINDSCALtorecovertheattributespaceandtheweightsoneachdimensionthatleadtotheindividualperceptualspacesofeachsubject.Itisinterestingtonotethattheweightsthemselvescanbeplottedasasubjectspacetoshowhowdierentsubjects'dissimilarityjudgmentscluster.Normally,MDSproducesacongurationofpointswhoseaxesarearbitrary,duetotherotationalinvarianceoftheconguration.Thishindersthetaskofattributingdissimilarityvaluestounderlyingcauses,aprocedurecalledfactoranalysis,becauseallsetsoforthogonalaxesmustbeconsidered.AninterestingoutcomeofthemodelusedbyCarrollandChangisthatintheoryitproducesasetofprincipalaxes.AsCarrollandChangnote,\themodelisbynomeanscompletelygeneral...Inparticularitdoesnotallowdierentialrotationaswellasdierentialweightingofdimensions."Butthemodelislikelyagoodstartingpointfordeterminingthemostmeaningfulsetofaxespossible.Thissimpliessubsequentanalysesthattrytodeterminetheunderlyingcausesofthedissimilarityvalues.Itisinterestingtonotethattheconictinginformationprovidedbymultiplesubjectsisactuallycomplementaryinthisframeworkandprovidesmoreinformationforchoosingthemostmeaningfulorientationoftheaxes. 29

PAGE 30

2.2.4SpecicitiesandLatentClassesIn1993,WinsbergandDeSoete[ 22 ]extendedtheweightedEuclideanmodeltoincorporatespecicitiesandlatentclasses,creatinganalgorithmcalledCLASCAL.SpecicitieswererstincorporatedintoaweightedEuclideanmodelbyWinsbergandCarrollin1989,butwereinothermodelsprior[ 23 ].Aspecicityisapropertyofastimulusthatcontributestoperceiveddistancesbetweenstimulibutisonlyexhibitedbyasinglestimulusintheset.Becauseitisonlycommontoonestimulus,thepresenceofspecicitiesinmultiplestimulihasthepotentialtoyieldahigh-dimensional,degeneratesolutionorwarpalow-dimensionalspacefoundfordimensionssharedbyallstimuli.Therefore,modelingspecicitiesexplicitlycanimprovethemodelofthedimensionssharedamongallstimuliwhenthestimuliarecomplexandhighlyvaried.Ontheotherhand,theuseofspecicitiesaddsanumberofadditionalparameters,therebyincreasingtheuncertaintyintheestimateofeachone.Therefore,onehastobecarefultoensurethattheuseofspecicitiesisreasonableandthatthereisenoughdatatoreliablyestimatethedimensionscommontoallstimuli.LatentclassesareasimplicationtotheweightedEuclideanmodelwhereeachsubjectisassumedtobelongtooneofamuchfewernumberofclasses,whereallsubjectsinaclassweightthedimensionsthesameway.Whensubjectsdoexhibitahighdegreeofsimilarityintheirdistancejudgments,theadditionoflatentclassescanbeseenasausefulwaytoreducethetotalnumberofparametersandincreasethecertaintyofeachone'sestimatedvalue.Ontheotherhand,ifsubjectsexhibitingsignicantdierencesintheirdissimilarityjudgmentsaregroupedtogether,thelatentclassmodelcouldpossiblybeaworsetthanonewithouttheuseoflatentclasses. 2.2.5IsomapMDSisusefulforidentifyinglower-dimensionalstructureembeddedinahigher-dimensionalspace.Usingthepairwisedistancescomputedbetweenallpointsinthehigher-dimensionalspace,MDScanattempttondalower-dimensionalrepresentationthatpreserves 30

PAGE 31

thesedistances.Thelower-dimensionalspacerepresentstheintrinsicdegreesoffreedominthedata.Sometimes,though,theobservedcongurationofdatapointsinthehigher-dimensionalspaceisanon-linearfunctionoftheunderlyingdegreesoffreedom.Inthiscase,thedistancesbetweenpointsinthehigh-dimensionalspacearenotEuclideanwithrespecttotheunderlyingparametersinthelower-dimensionalspace.However,ifthereareenoughpoints,thedistancesinthehigher-dimensionalspacearelikelytobelocallyEuclidean,meaningtheywouldagreewiththeirlower-dimensionalcounterpartswithinacertainsmallneighborhoodaroundeachpoint.Often,datainthiscasearesaidtolieonamanifold.Theshortestcurvebetweenpointssuchthatthecurvestaysonthemanifoldisreferredtoasthegeodesicdistancebetweenthepoints.Thegeodesicdistancebetweenpointsinthehigher-dimensionalspacecorrespondstotheEuclideandistancebetweenthepointsinthelower-dimensionalspace.In2000,Tenenbaum[ 24 ]cameupwithacleverextensionofEuclideanMDStoestimatethelower-dimensionalembeddingofpointsinthecaseofnon-linearormanifoldstructure.Sincethegeodesicdistancesinthehigher-dimensionalspaceshouldcorrespondtotheEuclideandistanceinthelower-dimensionalspace,MDScanbeusedtodiscoverthelower-dimensionalstructuresolongasthereisanestimateofthegeodesicdistancesbetweenallpoints.Tenenbaum'salgorithm,calledIsomap,isaschemeforestimatingthegeodesicdistancesbetweenallpointsandthenusingEuclideanMDStondthelower-dimensionalembedding.Toestimatethegeodesicdistances,agraphstructureiscreatedfortheentirecongurationofpoints.Eachpointisanodeinthegraph,andpointsareconnectedonlytotheirclosestneighborsinthehigher-dimensionalspace.TheedgeweightconnectingneighboringpointsistheEuclideandistancebetweenthepoints,whichlocallyshouldbethesameinboththehigherandlower-dimensionalspace.Thenanall-pairsshortestpathalgorithmiscomputedonthegraph,yieldinganestimateofthegeodesicdistancesbetweenallpoints. 31

PAGE 32

SincethecongurationofdatalyingonamanifoldislocallyEuclidean,Isomapisoftenreferredtoasamanifold-learningalgorithm,becauseitcandiscovertheunderlyinglower-dimensionalstructureofsuchdata.TheoutcomeofIsomapclearlydependsontheneighborschosentobeconnectedinthecorrespondinggraph,sinceonlyneighborswhosedistancesareroughlyequalinthehigherandlower-dimensionalrepresentationsshouldbechosen. 2.3TimbreSpacesTheideabehindtimbrespaceresearchistocollectdataontheperceiveddegreeofdissimilaritybetweenallpairsofasetofsoundsanduseMDStotthesoundsintoaspace,suchthatthedistancesbetweensoundscorrespondtothereporteddissimilarities.Fromthere,furtheranalysiscanbedonetouncovertheindividualdimensionsthatcontributedtotheperceiveddissimilarities.MostmethodsmaketheassumptionthatthespaceisEuclideanandthattherelevantdimensionswillbeorthogonal.TheorientationofthespaceisarbitraryinbasicMDSmethods,whichmeansitisindeterminateinwhichdirectionstheprincipalaxesgo.Although,byassumingthatthesameorthogonalaxesarerelevantforeveryonebutthatdierentpeoplewillweightdierentaxesdierently,moreadvancedmethodscansimultaneouslydeterminetheprincipalaxesandtheweightseachindividualplacesonthem.Section 2.3.1 outlinesthelastseveraldecadesofresearchintotimbrespaces.ThishistoryhascloselyfolloweddevelopmentswithintheMDScommunity,witheachnewMDSmodelappliedtoconrmorimproveupontheresultsachievedwiththepreviousmodels.DespitethedevelopmentswithinMDS,theparadigmforconductingtheseexperimentshasremainedremarkablyunchangedsincetherstapplicationsover40yearsago.Section 2.3.2 discussessomeoftheprimaryshortcomingsofconventionaltimbrespaces. 32

PAGE 33

2.3.1HistoryofTimbreSpacesPlomp[ 19 ]wasthersttoapplyMDStotimbre,usingKruskal'salgorithmonasetofdissimilaritiesderivedfromtriadiccomparisons.However,PlompdidnotreportmuchabouthisMDSndings,onlymentioningthatittookathree-dimensionalsolutiontoadequatelyexplainthedissimilarities.Insteadhewasinterestedintheroleoftheamplitudepatternoftheharmonicsindeterminingtimbre.Hisstimuliwerethesteadystateportionsofnotesofthesamepitchplayedondierentinstruments.Forthesamesetofstimuliusedtogetthedissimilarities,hecalculatedasimpledistancemeasurebasedonthedierenceinamplitudepatternoftheharmonics.Comparingthesetofperceptualdissimilaritieswiththecalculateddistances,hefoundacorrelationvaluebetween0.81and0.86.Fromthisheconcludedthatthetimbreofsteadystatetoneswasmostlydeterminedbytheamplitudepatternoftheharmonics.Wedin[ 25 ]wasthersttoreportthespatialresultsofapplyingMDStostudytimbre.Hisstimuliwerenotesfromninemusicalinstruments,allplayingthesamepitchandapproximatelyequalinloudnessandduration.Hepresentedstimuliinpairsandaskedsubjectstoratethesimilaritybetweenthetwostimulionascaleof1to10.UsingMDS,hefoundathree-dimensionalsolutionthataccountedfor76%ofthetotalvariance.Eachstimulustendedtofallpredominantlyalongsomedimension.Wedinexplainedthedimensionsintermsofthedierentspectralenvelopesofthestimuli,witheachdimensionhavingacharacteristicspectralenvelope.Miller[ 26 ]wasthersttoapplyMDStostudytimbreusingsyntheticstimuli,andthersttodosousingCarrollandChang'sINDSCALalgorithm[ 21 ].Thesyntheticstimuliweredesignedtoroughlyresembleactualmusicalinstruments(thoughnottothedegreethattheywouldbeconfusedfortherealthing)usingafewparametersthatweresystematicallyvaried.Foreachpossiblepairofstimuli,subjectswereaskedtoquantifythedissimilaritybetweenthetwostimulionascaleof1to9,with9representingthemostdissimilar.Intherstrunoftheexperiment,fundamentalfrequencywasincluded 33

PAGE 34

asaparameterthatwasvariedtocreatethestimuli.Interestingly,thisledtoreporteddissimilaritiesthatwereoverwhelminglydependentonthedierenceinfundamentalfrequency.ThisindicatedtoMillerthatheneededthestimulitoallhavethesamefundamentalfrequencysothattimbre-relateddierencescouldemerge.Inthesecondrunoftheexperiment,threetimbre-relatedparameterswerevariedtocreatethestimuli.UnlikegeneralMDSalgorithms,INDSCALproducesasetofprincipalaxesthataretbyassumingeachsubjectweightstheaxesdierentlyintheirpersonaltimbrespace.Inthiscasethenumberofharmonicsincluded,whichisrelatedtobrightness,wasfoundtobethemostsalientofthethreeparameters,followedbyenvelopetype,andlastly,rateofonsetoftheharmonics.ThestudybyGrey[ 27 ]isperhapsthemostwell-known.Heanalyzednotesfrom16dierentinstrumentsandcreatedsyntheticreproductionswherehecouldpreciselycontroltheunderlyingparameters.Heequalizedtheseforpitch,loudness,andduration.Thisequalizationwasconsideredtobenecessarytoisolatejustthoseattributescorrespondingtotimbre,andwouldprovetobeveryinuentialinthestudiesthatfollowed.Greyaskedlistenerstonumericallyratethesimilaritybetweenpairsofsyntheticnotes.LikeMiller,heusedINDSCALtondthespatialembeddingofthesoundsandfoundathree-dimensionalsolutionforthespaceoftimbre.McAdamsetal.[ 28 ]conductedatimbrespacestudyin1995withanextendedversionoftheCLASCALmodel.CLASCALdiersfromINDSCALbytheadditionofspecicitiesandlatentclasses.Modelingspecicitiesaddsmoreparameterstothemodeltoaccountfortimbrefactorsthatmaybepresentinonlyasinglestimulus,whichintheoryshouldpreventthespatialsolutionfrombeingdistortedbythesefactors.Ontheotherhand,assuminglatentclassesforthesubjectsisessentiallyawaytoremoveparametersfromthemodeltokeepitfrombeingimpracticaltot.Thisstudyused18dierentsynthetically-generatednotesandmaintainedtheconventionofequalizingforpitch,loudness,andduration.Subjectswereaskedtousea9-pointratingscaletorate 34

PAGE 35

thedierencesbetweendierentsounds.Thebestttothedatafoundbythestudywasasix-dimensionalsolutionwithnospecicities,buttheresearchersfounditdiculttointerprettheaxesphysically.Thestudyalsofoundagoodsolutionthatdidincludespecicitieswiththreeprimarydimensions.Thersttwodimensionsofthissolutionstronglycorrelatedwiththephysicalpropertiesoflog-attacktimeandspectralcentroid,respectively.Thethirddimensionwasnotaseasilyinterpretable,butfoundtolargelydependonspectralux.Lakatos[ 29 ]expandedtheuseofCLASCALin2000toamixofnotesfrompercussiveandnon-percussiveinstruments,andfoundalow-dimensionalspacehighly-correlatedwithspectralcentroidandattacktimeinthersttwodimensions.OnenalstudyworthnotingisthatofBurgoyneandMcAdams[ 30 ]who,in2008,combinedCLASCALwithIsomaptoredothreeoftheearliertimbrespacestudies.Isomapisamanifold-learningtechniquethatemphasizeslocalstructureindeterminingtheglobalcongurationofasetofpoints.Byemphasizingthelocalstructure,whichpresumablycanbeexplainedwithfewerdimensions,anoverallreductionincomplexitycanbeachievedintheglobalconguration.BurgoyneandMcAdamsfoundthatIsomapcollapsedtheCLASCALmodelintoasingledimensionforallthreepriorstudystimuli,thoughthehigh-correlatedphysicalfeaturewasnotthesameineachcase.Thereductionincomplexityisdiculttointerpretinaperceptuallymeaningfulway.Basedonthisresult,itdoesnotappearasifanybenetisderivedfromtheadditionaluseofIsomapinthesekindsoftimbrespacestudies. 2.3.2ShortcomingsofTimbreSpaceResearchResearchusingtimbrespaceshasyieldedsomeinsightintofeaturesthatplayaroleinourperceptionandcategorizationofsoundatthehighestlevel.Ingeneral,perceptualspacesseemlikeacoherentideawithmuchpromiseforlinkingphysicalphenomenatoourinternalrepresentationandperceptions.Butthetrueexplanatoryandgenerativecapacityoftimbrespaceshasbeenlimitedbyanumberofshortcomingswiththepriorresearchintothemuptothispoint. 35

PAGE 36

Firstofall,everymajorstudysinceMillerin1975hasbeendoneusingstimuliallwiththesamepitch.Theideathattostudytimbreinisolation,onemustequalizetheotherperceptualdimensionsofpitch,loudness,anddurationseemstohavebeenpopularizedbyGrey[ 27 ].Millerbeforehimfoundthatdierencesinfundamentalfrequencydominatedhisreportedresults,soheequalizedforpitch[ 26 ].Oneproblemwiththisistheideathatpitchandtimbrearetrulyindependent,anassumptionthatKrumhanslhascalledintoquestion[ 31 ].Byequalizingforpitch,oneremoveseventhepossibilitythatsignicantdimensionsoftimbrearedependentonfundamentalfrequency,whichiscountertothendingsofFletcher[ 6 ].Equalizingforpitcheliminatessomeofthepotentialforobservingtimbredierencesrelatedtoformants.Furthermore,ithasbeensuggested,forinstance,thattheclarinetpossessesenoughdierenceintimbreacrossitsentirerangeofnotesastofunctionallybeconsideredthreeseparateinstruments.WhileGreyperhapsdidnotintendforentireinstrumentstoberepresentedwithasinglepointintimbrespace,hisresultsdonotadequatelydistinguishbetweenthetimbreofaclarinet,whichhasbeendemonstratedtobeanincoherentconcept,andthetimbreofaclarinetplayedatacertainpitch(andpossiblywithotherplayingcharacteristics).Anadditionalproblemwithtimbrespacestudieshasbeentheuseofasmallnumberofdiscretepointsonascaletonumericallydescribetheperceiveddierencebetweendierentsounds.Suchascaleisunfamiliartosubjectspriortothestudy,soitsuseissubjecttoinconsistencyovertime.Furthermore,whenMillerobservedthatdierencesinfundamentalfrequencyoverwhelmedthereporteddierencesintimbre,itmayhavebeendueinparttothefactthathewasusingonlya9-pointratingscale,whichisessentiallythesamescaleusedineverytimbrespacestudy.Amoreconsistentmethodwithgreaterresolutionwouldbetheuseofbinarycomparisonsandasubsequentrankingalgorithm,whichcanthenbeusedwithnon-metricMDS.Wessel[ 32 ]in1979appearstoconsidertheuseofbinarycomparisonsandinsteadchoosesaratingscalelikeMillerbecausecollectinganentiresetofbinarycomparisonsistedious.Nopapersontimbrespaceappearto 36

PAGE 37

considerageneralrankingframeworkthatcanuncovertheperceiveddierenceamountsfromonlyafractionofthefullsetofbinarycomparisons.MoreconsiderationisgiventotheuseofarankingmethodinSection 2.4 andChapter3.Athirdmajorshortcomingistheuseofsinglepointstorepresentsoundsofarbitrarylength.Thismakeslearningthespacemoredicultandmakessynthesisimpractical.Thesoundscouldbeconstrainedtoallbethesamelength,butthenthisseverelylimitsthepossiblesoundsthatcanbegenerated.Additionally,sincespecifyingasoundwouldmeanchoosingapointinthespace,entiresoundswouldhavetobespeciedinadvance.Thiswouldplaceasevererestrictionontheexibilityofthesystemanditsabilitytosynthesizesoundsthatevolveinreal-time.Insomecontexts,itmaybeappropriatetorepresentcomplexphenomenawithsinglepoints,suchasfaces.Facesareusuallyconsideredtobeinvarianttoscaling,andcanallbenormalizedtothesamesize.Butsoundinherentlyisnot.Timingandlengthareveryimportanttohowsoundisperceived,whichisevenevidencedbytheimportanceofattacktimefoundintheperceptionofsoundsinprevioustimbrespacestudies.Takentotheextreme,wecouldtrytorepresententiresongsasisolatedpointsinsomeveryhigh-dimensionalspace.Butthiswouldmakeitextremelydiculttoconceiveofnewsongs.Ifnothingelse,astaticrepresentationofnotesassinglepointsseemslikeitlimitscreativity,astheentirespaceofpossiblenotesislikelytobevastlyunchartedanditisunclearhowtosynthesizeentirelynewtimbreswithoutalreadychartingthespacefromwhichtheyarise.Abetterrepresentationoftimbrespacewouldbeonewheretimeisanexplicitparameter.Inthisrepresentation,instantaneoustimbrepropertiesareencodedinthespace,andnotesarerepresentedastrajectoriesthroughthespace.Representingnotesastrajectoriesinaspacehasbeenproposedbeforebutnotwithperceptualfeatures,anditistherepresentationweadoptinthiswork.Analshortcomingoftimbrespaceresearchthusfaristhelow-dimensionalsolutionsthathavebeenproposed.Thattimbre,widelyknowntodependonanumberofspectral 37

PAGE 38

andtemporalfeatures,couldbeadequatelydescribedwiththreedimensionsorless,isaverysurprisingresult.EthingtonandPunch[ 33 ]havesaidthatthreedimensionsareclearlyinadequate,butagreethattheyareagoodstartingplace.Moststudiesndthatspectralcentroidandattacktimecanexplainthereporteddierencesfoundinthersttwodimensionsoftimbrespace.Butathirddimensionisconsistentlyfoundtoeludesimpleexplanation,sometimesbeingassociatedwithanimprecisefeaturesuchasspectralux.Caclinetal.[ 34 ]provideagoodoverviewofthevariousfeaturesthathavesometimesbeenfoundtocorrelatewithtimbrespacedimensions.Thatthethirddimensionisalwaysdiculttoexplainoughttoindicatethatathree-dimensionalrepresentationisnotsucient.Interestingly,Shepardfoundinonestudy[ 8 ]involvingMDSanddissimilarityratingsfromsubjectsthatpitchrequiresave-dimensionalspacetoadequatelyrepresentperceiveddistancesbetweendierentpitches.OnlyMcAdamsin1995evensuggestedatimbrespaceofatleastvedimensions,thoughnotmuchattentionwasgiventoitsanalysis.Givenallofthediscussionoftimbreasahighly-complexperceptualattribute,itwouldbeaverysurprisingresultifpitchwerefoundtobetrulyamorecomplexphenomenonthantimbre.InChapter3,weproposemethodstoaddressandimproveuponalloftheseshortcomings.Inthenextsection,welookatrankingmethods,whichwillformthebasisforoneofthekeywaysinwhichweproposetoimprovetimbrespaces.Afterthat,welookatcurrentmodelsofdynamicmusicalsoundsynthesis,sincewewouldliketobeabletosynthesizenewnotesfromtheirtimbrespacerepresentations. 2.4RankingMethodsNon-metricMDSrequiresarankingofthepairwisedistancesbetweenallpoints.Onepossiblewaytoacquiretheseforatimbrespaceapplicationistoasklistenerstonumericallyratethedissimilaritybetweendierentsounds.Butthereareafewreasonswhythisisnotpreferable.Oneofthereasonsisthatthescaleusedtonumericallyrate 38

PAGE 39

thedierencesisarbitraryandhasnotbeenlearnedinadvance,sojudgmentscouldbesubjecttolargeuctuationsfromonetrialtothenext.Toeliminatenon-stationarityinaccuraciesarisingfromtheuseofanarbitraryscale,onecaninsteadusethemethodofpairedcomparisons,coupledwitharankingalgorithm.Ingeneral,apairedcomparisonmeansaskinghowtwoitemstrelativetoeachotheronascale,ratherthanaskingexactlywhereaspecicitemtsonthescale.Inthiscaseweasklistenerstodecidewhetherapairofsoundsismoresimilarorlesssimilartoeachotherthananotherpairofsounds.Thisisanevenmorespecickindofpairedcomparison,abinarycomparison.Theuseofpairedcomparisonsmaystillbesubjecttosomenon-stationarity.Forinstance,inapairedcomparisonwherethedierencebetweenstimuliislessthanthejustnoticeabledierence,decisionscanstillbemade,buthavebeenobservedtochangeuponrepeatedtrials[ 35 ].Thurstonenotedthatinconsistenciesinpairedcomparisonjudgmentsbetweenstimulithataremoresimilarthanthejustnoticeabledierenceareindicativeofonewayinwhichthediscriminalprocessisnotentirelystationary,butsubjecttosmalluctuations.Oneimplicationofthisisthattheentiresetofpairedcomparisonsgeneratedbyasubjectforasetofstimulicannotbeguaranteedtoobeytransitivity,whichmeansitcouldcontainunintentionalcontradictions,particularlyasthedierencebetweenstimuliapproachesthejustnoticeabledierence.Whetherthecontradictionsareharmfulorhelpful,however,maybeamatterofcontextandperspective.Fromthesetofpairedcomparisons,thegoalistodeterminetherankingthatisinbestagreement,andapproachestothisgenerallylendthemselvestooneoftwoperspectives.Therstsetofapproachesviewstherankingasapredictorofcomparisonoutcomes[ 36 ].Whenastimulusiisrankedhigherthananotherstimulusj,therankingpredictsthatstimulusiwillbecomparedfavorablytostimulusj.Thesecondsetofapproacheshypothesizesthatthestimulieachhaveanintrinsicvalueorratingthatordersthem,andthepairedcomparisonsareevidenceofthetrueratings.Wheneverastimulus 39

PAGE 40

iiscomparedfavorablytoastimulusj,itisevidencethatstimulusihasahigherratingthanstimulusj.Itshouldbenotedthatonlyasetofrelativevaluescanberecovered,althoughthisisnotaconcernfortypicalapplications.Fromtheperspectiveoftherankingsasapredictorofcomparisonoutcomes,thedeviationbetweenpredictedcomparisonoutcomesandactualcomparisonoutcomesmeasurestheerroroftheranking.Thebestrankingistheonethatminimizesthiserror.Theerrorcanbemeasuredusingstrictlythenumberofincorrectpredictions,inwhichcasethesolutionisreferredtoastheminimumviolationsranking,oralsotakeintoconsiderationthemagnitudeoftheincorrectpredictionsusingthedierencesinrank.Notethatfromthisperspective,intransitivityamongthepairedcomparisonsisbad,asitincreasestheerror,oruncertainty,intheranking.Fromtheperspectiveoftheobjectshavinganintrinsicvalueorratingthatordersthem,violationsoftransitivityarenotincorrectpredictionsbutsimplyanexpectedresultofacomparisonprocessthatcanbeinconsistentwhenstimuliarenearthejustnoticeabledierence.Whileviolationsoftransitivitycanadduncertaintytorankings,itcouldbearguedthattheyarejustasmuchevidenceforthetruestimulusratingsasnon-violations,astheyareevidenceofextremesimilaritybetweenstimuli.Chartieretal.reportthat,asexpected,boththeMasseyandColleymethodsappliedtoaperfectlytransitivesetofbinarycomparisonsresultinanevenly-spacedsetofratings[ 37 ].Inthecaseofbinarycomparisons,onlyintransitivitycanprovideevidenceofirregularly-spacedratings.Manyrankingapproachescomewithlimitationsonthesettingsinwhichtheywillwork.TwoapproachesthatarecapableofoperatingonanincompletesetofbinarycomparisonsthatweconsideraretheColleymatrixmethod[ 38 ]andamodiedversionoftheminimumviolationsrankingmethodgivenbyAlietal.[ 39 ].Ingeneral,methodsthatminimizethenumberofviolationsareformulatedtooperateonacompletesetofbinarycomparisons.ThompsonandRemageshowedthatforthecaseofacompletesetofbinarycomparisons,minimizingthenumberofviolationsisjustied 40

PAGE 41

underamaximumlikelihoodframework[ 36 ].ThealgorithmproposedbyAlietal.,thoughformulatedtoworkonlyonacompletesetofbinarycomparisons,canbeadaptedforthecaseofmissingbinarycomparisons,asweexplaininChapter3.Still,thesolutiontotheminimumviolationsrankingproblemisequivalenttotheproblemofndingtheminimumfeedbackarcsetinadirectedgraph,whichisknowntobeNP-hard[ 40 ].Thisseverelylimitstheproblemsizestowhichitcanbeapplied.TheColleymatrixmethodistheleast-squaressolutiontoasetoflinearequations.Developedtobeappliedtocollegefootballrankings,itisabletosimultaneouslydeterminearatingforeveryteamthattakesintoaccounttheratingsoftheotherteamsithasplayedagainst.ColleyequateshismethodtothatofLaplace'smethodforestimatingthelocationofamarker'spositiononacrapstable.Laplacestudiedtheproblemgivenauniformrandomsamplingofpositionsonthetableandthenumberoftimesasampleistotheleftortotherightofthemarker.Themostlikelylocationcanbededucedusingsimplewinningpercentage,butLaplacealsoaddedaBayesianpriorof0.5becausethebestguessatthestartisthatthemarkerisinthemiddleofthetable.Insports,ifgameswithotherteamsarelikesamples,thenwithauniformsamplingofgamesbystrengthofopponent,winningpercentageisthebestguessatateam'sstrength.ButColleyusesLaplace'spriorandadjustsforstrengthofschedule,whichisanattractivefeaturethatmakesthemethodmorerobustwhenthenumberofteamsgreatlyexceedsthenumberofgamesplayedbyeachteam.OnepotentialproblemwithColley'smethodistheover-relianceonstrengthofschedule.Anundefeatedteamwithaweakstrengthofschedulecanhaveitsratingbroughtdownevenwhenthisintroducesintransitivitybetweenitandhigher-rankedteamsithasbeaten.Also,theuseofapriorbehavesbestwhenteamshaveallplayedthesamenumberofgamestoovercometheprior,whichistrueinmostsportsbutcannotbeassumedtobetrueingeneral.Asaresult,itmaybethatahybridapproachthat 41

PAGE 42

startswithColley'smethodandthenmakesiterativeadjustmentstoreducethenumberofviolationsoftransitivityismoredesirableincertaincontexts. 2.5Perceptually-BasedMusicalSoundSynthesisThetimbrespaceisthecorepieceoftheframeworkproposedinthisworkforachievingunrestrictedtimbremanipulation.Inthissectionweexaminepriorworkmadedirectlytowardssuchaframework.InSection 2.5 ,wereviewdynamicmodelsofsoundsynthesisaimedatrepresentinganyconceivabletimbre.InSection 2.5.2 ,wereviewproposedmethodsforinterfacingwithaperceptually-basedsynthesissystem.Averylimitedamountofworkhasbeendoneoncomprehensiveframeworks.Wessel[ 32 ]in1979wasoneoftheveryrsttooutlinehowmusicalsoundswitharbitrarytimbremightbesynthesizedinaperceptualframework.Heproposedworkingwithtimbrespaces,performingpsychoacousticanalysisonthedimensionstodiscovertheirunderlyingphysicalexplanations,andtreatingeachdimensionasacontrolparameterforchoosingbetweendierenttimbres.However,thetimbrespaceusedinhisworksuersfromalloftheproblemswementioninSection 2.3.2 ,whichmakesitunsuitableforunrestrictedtimbremanipulationincomposition.Wesselmentionshimselfthatheisonlyconsideringasystemcapableofproducingchangesintimbrebetweendierentnotes,notwithinasinglenote.Hegivesnomentionofhowtoproducenotesofarbitrarylength.Nicol[ 41 ]in2005proposedaframeworkforsynthesizingsoundswitharbitrarytimbrefromhigh-leveldescriptionsofthesounds.Henotedthatsoundsareoftenconsciouslyperceivedatalevelfarabovethesignallevelatwhichversatilesynthesissystemstendtooperate,thereforeamappingneedstobemadefromhigh-leveldescriptionstolow-levelsynthesis.Hecallsthefoundationforhisframeworkatimbrespace,butthisisamischaracterizationbecausehisspacehasnobasisinperception,insteadbeingjustatime-frequencyrepresentationofsoundderivedfromsignalprocessing(thoughitshouldbenotedthatafewothershaveusedthetermthisway,aswell).Heovercomessomeofthelimitationsofthetypicaltimbrespaceparadigmbyrepresentingsoundsaspaths 42

PAGE 43

andnotpointsinaspace.Buthisframeworkisnotanadequatemodelofperception,merelyattemptingtomapfromalimitednumberofhigh-levelperceptualdescriptionstosynthesisparameters.Thereforeitisnotsuitableforthelevelofunrestrictedtimbremanipulationthatwewouldliketoachieve.Thesubproblemscommontoallcomprehensiveframeworksareperception,synthesis,andinterface.WegivemuchattentiontoperceptioninSection 2.3 .Inthefollowingsections,welookatapproachestosynthesisandinterfaceinthecontextofunrestrictedtimbremanipulation. 2.5.1ModelsofDynamicMusicalSoundSynthesisTherearenumerousmethodstoachievedigitalsoundsynthesis.However,whilecapableofexertingsomecontrolovertimbre,mostofthesemethodsareunsuitablefortheunrestrictedmanipulationoftimbrethatwehopetoachieve.Nicol[ 41 ]givesanoverviewofvarioussynthesismethodsanddeemsFMsynthesis[ 42 ]andadditivesynthesisthemostsuitablemethodsforthecreationofarbitrarytimbre.TheprimaryadvantageofFMsynthesisistheabilitytospecifyrichtimbreswitharelativelysmallnumberofparameters.However,thisisonlyanadvantagefromtheperspectiveofcomputationalrequirements.Additivesynthesiscouldeasilybespeciedwithasmallnumberofparametersthatabstractcommonpatternsintheunderlyingparameters,butwithmoreexpressivecontrol.FMsynthesisinitsbasicformislimitedinitsexpressiveability.Ahybridadditive-FMsynthesismethodcouldbecreatedtoexpandtheexpressivecontrolofFMsynthesisifcomputationalconstraintssupersedeallotherconsiderations.However,duetotheirexibility,intuitivecorrespondencewithwhatweperceive,andthecontinuedexpansionofcomputingpoweravailabletoimplementthem,additivesynthesismethodsseemlikeabetterchoiceandaremuchmorecommonintheliterature.Therefore,weprimarilyfocusonmodelsofarbitrarytimbresynthesisthatuseadditivesynthesismethods. 43

PAGE 44

Everytimbrespacestudyperformedsofarhasrepresentedentiresoundsassinglepointsinaspace,whichisnotsuitableforgeneratingarbitrarysounds,particularlyofarbitrarylength,withoutaprohibitivelyhighnumberoftrainingsounds.Furthermore,itisnotclearhowtrulynovelsoundswouldbegenerated,sincetherepresentationisnotexibleenoughfortheunderlyingdimensionstobecombinedinnewways.Abetterrepresentationwithregardstosynthesiswouldbesomethinglikeaphasespace[ 43 ].Imagineasystemwhosecurrentstateisdescribedbyasetofparameters.Aphasespacerepresentsallofthepossiblestatesofthesystemaspointsinamulti-dimensionalspacewiththeparametersasaxes.Theevolutionofthesystemovertimeisrepresentedasatrajectoryinthisspace.Nolte[ 43 ]givesaverythoroughreviewofthecoinageofthetermphasespaceandtheearlieroriginationoftheconceptsinthe19thcentury.Inthecaseofsounds,theinstantaneouspropertiesofasoundcanberepresentedaspointsinaphasespace,andtheirevolutionovertimeasatrajectory.Thisdivisionofinstantaneousandtemporalpropertiesmakesitpossibletospecifyanysoundwithoutmodelingahighnumberofdimensions.Thedimensionalityiskeptlowbecausetheburdenisshiftedtothetrajectoryforcapturingasignicantportionoftheinformation.Additivesynthesistseasilywithintheframeworkofaphasespacewithtrajectories.Earlyworkonadditivesynthesisfocusedonmethodstoreducetheamountofdataandprocessinginvolvedtomakeitpracticaltoimplementinrealtime[ 44 45 ].Thisprimarilyinvolvedlinearlyinterpolatingbetweenafewexplicitlandmarkparameters.Thisdatareductionwasjustiedbynotingthathardlyanyperceptualdierencecouldbeobservedbetweenthefully-speciedversionandthepiecewiselinearversion.Latermethodshavetakenadvantageofadvancesinmachinelearningtolearncompactrepresentationscapableofcapturingtherangeofvariationforanentiremusicalinstrument.Burredetal.[ 46 ]chosetomodelmusicalinstrumentsoundsasaspectralenvelopethatevolvesoverthecourseofanote(inotherwords,overtime).Witheachspectralenvelopecorrespondingtoapointinalow-dimensionalspace,thisisatypicalrepresentationasaphasespace 44

PAGE 45

withtrajectories.Inadditiontocompactnessandrepresentativeness,theyciteaccuracyasadesigncriterionintheircompactrepresentation,buttreatthisasaccuracyfromasignalerrorperspectiveanddonotconsiderperceptualaccuracy.Therefore,theylikelymissoutonthedatareductionachievablebyonlyconsideringfeaturesthatarerelevanttoperception.Inthiswork,weadoptadditivesynthesisandaphasespacerepresentationwithtrajectories,butchoosetomodelthisprimarilyfromtheperspectiveofperception.Bymakingperceptiontheprimaryfocalpoint,therepresentationshouldbeascompactaspossible,onlycapturingdistinctionsthatarerelevantperceptually. 2.5.2InterfacesforTimbreControlInconsideringaninterfacefortimbremanipulation,theprimarygoalisintuitiveexpressionforcomposition.Real-timeperformancemaybeanothergoal,butusefulnessincompositiontakesprecedence.Thegoalisforacomposertobeabletoconceiveofanarbitrarysoundinhismindandthenbeabletoexpressiteectively.Thisisnotnecessarilythesameproblemasthatofperformance,whereduplicationandrangeofexpressionmaybemoreimportantthancreativity.Aneectiveinterfaceisusuallyonethatovertimeisnotnoticedbytheuser,easilyfacilitatingthetranslationfrommentalideatophysicalrealization.Timbreisacknowledgedtobeacomplexphenomenon,whichmakestheimplementationofaneectiveinterfacedicult.However,threeprimarytypesofinterfacesseemtohaveemergedthatcouldpotentiallyfacilitateunrestrictedtimbremanipulation.Thersttypeofinterfaceisadecompositionoftimbreintoorthogonaldimensionsthatcanbecontrolleddirectly.Everytimbrespacestudyinvolveslearninganorthogonalrepresentationoftimbreperception.Wessel[ 32 ]in1979proposedthatthedimensionsofatimbrespacebeassociatedwithacousticparametersandthatthedimensionsthemselvesbecontrolparametersforsynthesizingnoteswitharbitrarytimbre.Thoughhisoverallframeworkisnotsuitableforachievingunrestrictedtimbremanipulation,theideaof 45

PAGE 46

orthogonalcontrolimplicittohisproposedinterfaceisworthexamining.Thatdimensionsinatimbrespaceshouldbecontrolparametersseemstoimplytheideathattimbremanipulationoughttobelinearinthementalrepresentationoftimbre.Thedicultyliesinthemappingbetweenmentaltimbrespaceandphysicalsynthesisproperties.Thismappingcouldbequitecomplicatedandnon-linear.Butasageneralrule,itseemsreasonablethataninterfacetotimbremanipulationoughttobelinearintheperceptualdimensionsoftimbre,asWesselproposed.Thesecondtypeofinterfaceisare-mappingoftimbretoaspacethatcanbenavigatedvisuallyorgesturally.Musicalinstrumentsarethetraditionalinterfacebetweenhumansandmusicalsoundsynthesis.PerhapslikewithWessel,mostoftheactionsinvolvedwithplayingamusicalinstrumentcanbethoughtofasindependentparameterscontrollinghighlynon-linearchangesinsynthesisparameters.Butthecontrolinterfacedoesnothavetoinvolveindependent,orthogonaldimensions.HuntandWanderley[ 47 ]discussanumberofdierentmappingsfromgesturestosynthesisparameters,thoughtheirfocusisonperformance,notcomposition.Choietal.[ 48 ]conceivedofathree-dimensionalgesturecontrolspaceforspecifyingtimbre.Theycalledita\window"intoamuchhigher-dimensionalmanifoldspaceofsoundsandproposedtheuseofgeneticalgorithmstolearnthereductiontothreedimensions.Whatevertheassociationsbetweenthewindowandtheunderlyingsynthesisparametersendupbeing,theyproposedthattheycanbelearnedbyauserviatheprocessofexplorationandfeedback.Itisaninterestingcontrasttospecifyingindividualorthogonaldimensionsforcontrol,sincethehumanbrainisabletoprocessvisuallyuptothree-dimensionssimultaneously,andperhapsmoregesturally.Gesturalinterfacesmayskipoverexplicitlymappingperceptionandgodirectlytosynthesis,whichmaybeaweaknessintermsofexpressiveability.Ontheotherhand,itisconceivablethatwiththerightinterface,theproblemsofmappingfromgesturetoperceptionandperceptiontosynthesiscouldbeooadedtotheusertobelearnedthroughexperimentation.WesselandWright[ 49 ]giveathoroughoverviewof 46

PAGE 47

manydierenttypesofgesturalinterfaceforsynthesis.Theyencouragelearningviaexperimentationandlikentheactoflearningtouseagesturalinterfacewellwiththeactoflearningtouseanymusicalinstrumentwell,whichcanrequirealargeinvestmentoftime.However,intheirparticularproposaltheyaremorefocusedonperformancethanunboundedexpressionandlimitthemselvestothecreationofdiscretenoteeventswiththeirinterfaces,whichwouldnotachieveourgoalofunrestrictedtimbremanipulation.Goudeseune[ 50 ]elaboratesonmethodsofcontrolfordigitalmusicalinstruments,includingderivativecontrol,wheretherateofchangeofaparameteristhedriverratherthantheinstantaneousvalue.Healsogivesanextensivetreatmentoninterpolationbetweenhigh-dimensionalspaces,withparticularattentiontointerpolationpropertiesthataredesirableformusicapplications.Theproblemofmappingfromacontrolspacetoasynthesisspacegivenpairsofcorrespondingpointsinbothspacesistheproblemofinterpolation.Thisinterpolationisrelevantforanykindofmapping,notjustonesinvolvingavisualorgesturalinterface.Thethirdtypeofinterfaceisverbal,usinghigh-leveldescriptionstoencapsulateandsimplifytimbrespecicationthatwouldbetediousatthephysicallevel.Thistypicallyharnessesthevocabularyalreadyusedbymusicianstocategorizethenotestheyhear.Theideaistoreverseengineerthisprocessanddeterminethesynthesisparametersthatunderliethetimbreadjectives.EthingtonandPunch[ 33 ]proposedaninterestingapproachtothisin1994,wheretheylearnedassociationsbetweenverbalandsynthesistransformations.Insteadofmappingabsolutespecications,theylearnedhow,startingfromsomesoundalreadyconceivedof,itssynthesisparameterscouldbeadjustedtoresultinchangesinthedescriptionofthesoundsuchas\brighter"or\warmer".Thisavoidstheproblemofcreatinganexplicit,potentiallyhighlycomplexmappingbetweenhigh-leveldescriptionandlow-levelsynthesisparameters.Eachtypeofinterfacedoeshavethepotentialtoworkinaframeworkforunrestrictedtimbremanipulation,andwedonotseetheseasmutuallyexclusive.Intheframework 47

PAGE 48

proposedinthiswork,wefocusoncreatingtheorthogonalaxesinterfacetotimbremanipulationandsoundsynthesis,asenvisionedbyWessel[ 32 ],butdosowithaninterpolatedmappingunderlyingtheperceptualinterface,asenvisionedbyGoudeseune[ 50 ].Webelievethereisafundamentalgapthatexistsintheresearchintotheperceptionoftimbrethatcanbelledbythisinvestigation.Weproposethatverbaldescriptionsorahighlyexpressivegesturalinterfacebebuiltatalayerabovethisinterface.Thishierarchicalapproachgreatlysimpliesimplementationandseparatesanentiresystemfortimbremanipulationintonaturalsubproblemsthatshouldbesolvedindependently. 48

PAGE 49

CHAPTER3METHODOLOGYTheprimarygoaltobeaddressedbythisresearchisthedesignofaframeworkforunrestrictedtimbremanipulationinmusicalcomposition.Byunrestrictedtimbremanipulationincomposition,wemeanatahighleveltheabilitytospecifyamusicalnotewithanytimbralproperties.Atalowlevel,wemeantheabilitytoproducemusicalsoundofanarbitrarylengthandalteritstimbreovertimeaseasilyaswealteritsloudnessorpitch.InSection 3.1 ,weidentifyfourkeydesigngoalsthataframeworkfortheunrestrictedmanipulationoftimbreshouldmeet.Intheremainderofthechapterweexpandoneachofthegoalsandourproposalsformeetingthem. 3.1DesignGoalsTheprimarydesigngoalsforaframeworkforunrestrictedtimbremanipulationarealludedtoinChapter2butnotstatedasexplicitlyasinthissection.Thefourgoalscanbesummarizedasfollows: 1. Perception 2. Synthesis 3. Interface 4. ModularityTherstcomponent,perception,isincludedbecausetimbreisaninherentlyperceptualproperty,notanacousticorphysicalproperty.Perceptionreferstothementalrepresentationoftimbre.Theidealframeworkfortimbremanipulationcanfacilitateanecienttranslationfrommentalconceptiontoactualrealizationinsound.Thisseemsdiculttoachievewithoutarmunderstandingofhowtimbreisrepresentedinthebrain.Theinterpretationoftimbrebyalistenerisalsoimportantfromacompositionalpointofview.Therefore,facilitatingeectivemanipulationoftimbreincompositionrequiresanunderstandingofisbeingachievedthroughthemanipulation.Thisisaccomplished 49

PAGE 50

mosteectivelywithaperception-centricapproach,notanapproachcenteredaroundthemanipulationofphysicalpropertiesalone.Finally,thereisacompactnessofrepresentationthatisonlypossiblebyconsideringthementalrepresentationoftimbre.Thisrepresentationautomaticallydiscardsanypotentialpropertiesthathavenobearingonperception.Thesecondcomponentissynthesis.Thenotionofunrestrictedtimbremanipulationincompositionimpliesthenotionofunrestrictedtimbreproduction.Anaccuraterepresentationoftimbreisnotusefulincompositionwithoutameanstorealizeit.Therefore,anytimbrethatcanbeconceivedofmustbeabletobephysicallyrealized.Consequently,anextremelyexiblemethodofsynthesisisrequired.Thethirdcomponentisinterface.Agoodrepresentationoftimbreandameanstorealizeitaremuchlessusefulwithoutaneectiveinterface.Thechoiceofinterfacecoulddependonothergoalsfortheapplicationofthesystem.Thegoaloftheinterfaceinthisworkisprimarilytofacilitatecomposition,butperformanceisalsotakenintoconsideration.Thesetwogoalsarelikelytohavesubtledierencesintheinterfacesthatbestfacilitatethem.Thenalcomponentismodularity.Theidealframeworkshouldhavelayersthatacceptwell-denedinputandproducewell-denedoutput.Bydesigningaframeworkwithcomponentsthatcanbereplacedatonelayerwithoutaectingcomponentsatanotherlayer,thishelpsachieveasystemwhere,forexamples,oneinterfacecouldbedevelopedforcompositionandanotherforperformance.Intermsofsynthesis,itisconceivablethatdierentmethodscouldbepreferreddependingontheamountofcomputingresourcesavailable.Eventhegoalofperceptioncouldbenetfromamodularelement,assubjectiveelementsofperceptioncouldtheoreticallyleadtoapersonalizedtimbremanipulationsystem.Inthefollowingsections,wediscusshoweachofthesecomponentsisaddressedintheframeworkwepropose.Outofthethreelayersofperception,synthesis,andinterface,we 50

PAGE 51

giveparticularattentiontoperception,whichisthecomponentwechoosetodevelopforimplementation. 3.2PerceptionInChapter2,wereviewtheliteratureontheperceptionoftimbre.Anumberofattemptshavebeenmadetolearnthementalrepresentationoftimbre,whichisreferredtoasatimbrespace.Theaxesoftimbrespaceareusuallyassociatedwithacousticorphysicalpropertiesofsound.However,therepresentationsdevelopedsofarserveaspoorgenerativemodels,unsuitableasageneralmodelofsynthesis,particularlyinthecaseofnotesofarbitrarylength.Neitheraretheysatisfyingasmodelsofperception.Nearlyallofthetimbrespaceslearnedthusfarhavethreedimensionsorless,whichseemsoddconsideringhowmanyacousticalpropertiesareknowntohaveaneectontimbreperception.Todevelopagenerativespaceoftimbre,itseemsnecessarytohavearepresentationwithtimeasaparameter,sothatallotherparametersareexplicitlyassociatedwithpointsintime.Attemptshavebeenmadeatthiskindofrepresentation,butstrictlyfromanacousticperspective.Whatthisusuallyendsupbeingisamodelofadditivesynthesisthathasacompactrepresentation,determinedusingdimensionalityreductionmethods.Thiscanincludelossofperceptuallyrelevantinformation.Nomodelofperceptualtimbrewithtimeasanexplicitparameterhasbeendeveloped.Thebenetsofsuchamodelincludethepotentialforarepresentationthatiscompactaspossiblewithoutlossofinformationandapreciselycontrolledgenerationofanyconceivabletimbre.Sincesuchamodeldoesnotcurrentlyexist,itsdevelopmentisaprimarygoalofthiswork.Evenwithoutconsideringhowasystemcouldbebuiltforfacilitatingthemanipulationoftimbre,thecomprehensionoftimbreasanaspectofcompositionseemsunderdeveloped.Whileitmayberecognizedthattherearespectralandtemporalaspectstotimbre,timbrehasneverevenbeenfullybrokendownintoitsconstituenttemporalandnon-temporalaspects.Nomenclaturerelatingtotimbreisnotstandardized,either.Tonecolor,tone 51

PAGE 52

quality,andtextureareallsometimesusedinterchangeablywiththewordtimbre.Toremedysomeoftheseproblems,weproposesomenewnomenclatureandanewhierarchicalviewoftimbreproperties. 3.2.1InstantaneousTimbreandDynamicTimbreHelmholtzcoinedthetermmusicalqualitytodescribeasubsetoftimbreproperties,seeminglythosethatcouldbeassociatedwithsteadystatesounds.LaterresearchersseemnottohaveviewedHelmholtz'sworkascreatingadichotomybetweentheseandotheraspectsoftimbre,butratherasoversimpliedbecauseitdidnotcapturetemporalaspectsoftimbre.However,itseemsreasonabletoconcludethatHelmholtzdidintendtocreateadichotomyoftimbre.WhethertheideaforadichotomyoriginateswithHelmholtzorlatertheorists,webelievesuchabreakdownistheonlywaytomakeprogressonourunderstandingofsuchacomplexphenomenon.Weproposethataspectsoftimbreperceptionbebrokenintotwodistinctcategories:instantaneoustimbreanddynamictimbre.Instantaneoustimbrecaptureseverythingabouttimbrethatisconstant,stationary,orsteadystate.Thesearepropertiesthatcanbeassociatedwithadistinctpointintime,muchthesameasinstantaneousamplitudeorinstantaneousfrequency.Tonecolorseemslikeapotentialsynonymforinstantaneoustimbre,whichissomethingwewouldsuggest.Butthisequivalenceisnotcurrentlystandardized.Dynamictimbre,ontheotherhand,captureseverythingaboutchangesintimbreovertime.Thisincludestheevolutionofpitchandloudness,whichperhapsshouldnotnecessarilybeconsideredentirelydistinctfrominstantaneoustimbre.Includingamplitudeandfrequencywithtimbre,dynamictimbreisspeciedspecicallyastheevolutionofinstantaneoustimbre.Thusthisisahierarchicalrelationship,asdynamictimbrecanalwaysbebrokendownintoitsconstituentinstantaneousaspectsovertime.Notethatsuchabreakdownalreadyexistsinhumancolorvision.Pixelsrepresentthesmallestunitsofareaoverwhichcolorisperceived.Therearethreeorthogonaldimensionsofcolorvisionthatcanbeassociatedwithapixel.Onepossibleorientationof 52

PAGE 53

thisspaceistolabelthedimensionsasred,green,andblue.Anotherpossibleorientationishue,saturation,andlightness.Muchlikeinstantaneoustimbre,thesethreeorthogonaldimensionsareattributesofinstantaneouscolorandaretheonlydimensionsthatcanbeattributedtoasinglepixel.Complexpropertiesofcolorsuchasmatte-ness,oritsoppositeshiny-ness,thatcouldbethoughtofasdimensionsofcolorvision,cannotbeattributedtosinglepixelsandcanonlybedescribedwithacollectionofpixels,varyingeitherintimeorspace.Theseareelementsofdynamiccolor,whichisanalogoustodynamictimbre.Atanevenhigherlevelofcolorperception,imagesandobjectsemerge,andthisanalogyholdsforauditoryperception,aswell.Interestingly,thoughthereareonlythreeorthogonaldimensionsofcolor,wehavelisteduptosixperceptuallycoherentdimensions.Whetherornotthereareanalogousstructuresintimbreperceptionisofgreatinteresttous.Itmaybethattherearemultiplecoherentorientationsofinstantaneoustimbrespace.Thesearethekindsofquestionsthatcannotreallybeaddressedwithoutrstdistillingtimbredowntoitsmostfundamentalproperties.Therefore,wemakeitakeyaspectofthisworktodevelopatheoryoftimbreasadichotomybetweeninstantaneousanddynamicfeatures.EthingtonandPunch[ 33 ]describedasimilardichotomyoftimbreperceptionbutwithverbalattributes.Theyusethetermpresencetocategorizewordsthatareassociatedwithstationaryproperties,whilethetermsattackandcutoareusedtodescribedwordsassociatedwithnon-stationaryproperties.Whilethedichotomyproposedinthepresentworkisintendedtoapplytotimbreatthemostfundamentallevel,itispossibletoapplyittothesewords,aswell.Table 3-1 givesalistofwordsassociatedwithtimbreandtheircategorizationaseitherinstantaneousordynamic.Itisourhopethatsomeofthesewordscanbeadoptedtodescribefundamentalpropertiesoftimbre,sincetheyareinwidespreadusebymusiciansalready.But,thereisnouniversalstandardcurrentlyinuse.Asetoforthogonaldimensionscouldserveassuchastandard.Experimentsperformedtouncoverperceptualdimensionsoftimbre,however,attemptedtomodel 53

PAGE 54

Table3-1. Alistofwordsusedbymusiciansandhowtheytintoaproposeddichotomyoftimbre.Inthisdichotomy,themostfundamentalpropertiesoftimbrewithnotemporalcomponentareknownasinstantaneousproperties,whilehigher-orderpropertieswithatemporalcomponentareknownasdynamic. InstantaneousDynamic brightpluckednasalfadingdullvibratotonalstrummedinharmonicexpandinghollowechoinghissingblownbitingtremoloshrilltwangyrichwarbly bothinstantaneousanddynamicaspectsoftimbresimultaneously.Thisdidnotleadtoaclearunderstandingofeitheraspectintermsoforthogonaldimensions.Therefore,weproposetocreateaninstantaneoustimbrespace,whereeachpointrepresentsadistinctcombinationofinstantaneoustimbrefeaturesonly.Presumably,theaxesdeterminedbythiswouldshowsomeresemblancetoknownstationarypropertiesoftimbre,suchasbrightness.Dynamictimbrewouldthenberepresentedastrajectoriesinthisspace.Thisisaclassicphasespacesetup,toborrowatermfromphysicsthathasgainedwidespreadusethroughoutscience. 3.2.2TimbreSpaceExperimentThekeyaspectstobedeterminedexperimentallyinthisworkarethenumberofdimensionsofinstantaneoustimbreandtheirperceptualinterpretations.Thisdoesnotimplythatasinglesetofdimensionsiscoherentforallpeople.Wemayneedtodeveloparepresentationthatismostwidelyapplicableratherthanuniversal.Itmayalsobepossibletoconsidermultiplecoherentorientationsoftimbrespace.Wewishtointerpreteachdimensionsothatwecanlearntospecifynewtimbresperceptually,connectingthespatialrepresentationtoourownperceptions,ifpossible. 54

PAGE 55

. Binarycomp. . Binarycomp. . Binarycomp. . Binarycomp. . Binarycomp. . Binarycomp. . Binarycomp. . Binarycomp. Similarityjudgment Distancerankings Similarityjudgment Subjectembedding Euclideandistances Similarityjudgment Distancerankings Similarityjudgment Subjectembedding Euclideandistances . . Jointembedding Subject1 Subject2 Figure3-1. Thisgureshowsthepipelineofderiveddataintheinstantaneoustimbrespaceexperiment.Foreachgroupofsoundspresented,thesubjectprovidessimilarityjudgments,whicheitherspecifywhichtwosoundsarethemostsimilarorleastsimilarsoundsinthegroup.Fromasimilarityjudgment,multiplebinarycomparisonsarederived.Allofthebinarycomparisonsareusedtoderivearankingofthedistancesbetweensounds,usingColley'smethod.Fromtherankingofdistances,aspatialembeddingisderivedusingKruskal'snon-metricMDSalgorithm.TheEuclideandistancesbetweenpointsarecalculatedfromthisspatialembedding.Finally,ajointembeddingisderivedforallofthesubjectsfromtheirindividualEuclideandistancesusingINDSCAL. Toundertakethisanalysis,weproposetocreateaninstantaneoustimbrespaceusingsimilarmethodstothosedescribedinSection 2.3 .However,therearetwomaindierencesbetweentheproposedexperimentandpreviousattempts,whichwillhaveasignicanteectontheresult.Figure 3-1 containsadiagramthatoutlinestheexperimentalsteps.Therstprimarydierenceisthechoiceofdata.Weuse16sounds,each2secondsinlengthandsyntheticallygeneratedusingadditivesynthesiswithstationaryparameters,as 55

PAGE 56

givenbyEquation 3{1 ,s(t)=KXk=1akcos2 Fsfkt: (3{1)Weuse30sinusoidsforeachsound,witheachsinusoidkhavingconstantamplitudeakandfrequencyfk.Thefundamentalfrequencyofthesound,f0,couldbeconsideredadistinctparameter,althoughforourpurposeswewillassumef0=f1.Fsrepresentsthesamplingfrequency,takentobe44,100Hzforthesoundsusedinthiswork.Phaseisnotconsideredasaparameterinthepresentstudy.Mostofthesounds'parametersarederivedfromactualrecordingsofmusicalinstruments,analyzedwithanewperiodicwaveformanalysistechniquewedescribeinSection 3.3.2 .ThesesourcerecordingsarepartoftheUniversityofIowaMusicalInstrumentSamples[ 51 ].Table 3-2 givesalistanddescriptionofthe16sounds,includingwhatinstrumenteachisderivedfrom.Theinclusionofthesourcemusicalinstrumentinthedescriptionshouldnotbeinferredtomeanthatasoundisrepresentativeof,orevenclearlyidentiableascomingfrom,thatinstrument.Rather,itisameansofsuggestingwhatsomepropertiesofeachsoundmightbe.Thepurposeofusingsoundsderivedfromrealinstrumentsistoeasilygenerateavarietyofdataalreadyknowntohavenoticeabledierencesintimbre.Theparticularsoundsusedwerepickedtotryandsampleawiderangeofinstantaneoustimbrespace,butwithenoughsimilaritytoavoidanentirelyhollowspace.Somenotesofdierentpitchwerealsoincluded,whichtoourknowledgehasnotbeentriedsinceMillerandCarterette[ 26 ]in1975,whodeterminedthattheinclusionofpitchdierencesoverwhelmedtheperceptionoftimbredierences.However,itseemsimpossibletocharacterizethespaceofinstantaneoustimbrewithoutconsideringnotesofdierentpitch.Twonoteswiththeirotherpropertiesheldxedcouldbeperceivedasdieringbyadierentamountastheirpitchisaltered.Furthermore,itmakesforacleaneranalysistoconsiderallinstantaneousperceptualpropertiessimultaneouslyandsortthemoutduringtheanalysisitself,rather 56

PAGE 57

Table3-2. Adescriptionofthesoundsusedinthetimbrespaceexperiment.Mostofthesoundswerederivedfromanalysesofactualnoterecordings. No.InstrumentApprox.pitchActualfreq.(Hz) 1TrumpetA4442.72B-atclarinetG4392.33SyntheticstringE3164.84BassC3130.45SopranosaxophoneG4392.46PianoA3221.07TenortromboneA3220.98PianoA4440.09E-atclarinetA3218.110AltosaxophoneA3219.111ViolinC4259.412BassuteA3220.813BassclarinetA2109.514CelloA3220.115TrumpetB3246.516SquarewaveF]3185.0 thantotrytocreateadatasetcontainingonlythoseisolatedpropertiesconsideredtobetimbre,whichisill-denedtobeginwith.Thesecondprimarydierenceisthemethodoffeedbackforthelisteningexperiment.Intypicaltimbrespaceexperiments,subjectslistentoapairofsoundsandnumericallyratethedierencebetweenthetwoona9or10-pointscale.However,thisseemstoinviteinconsistencyintheresults,assuchascaleisunfamiliartothesubjectspriortotheexperimentandwouldbediculttoapplyconsistentlytotheirperceptions.Instead,weasksubjectstolistentosoundsingroupsof3or4andtodeterminewhichtwosoundsinthegrouparethemostsimilarandwhichtwoaretheleastsimilar.Thisrelativedeterminationhasthesamestationarity-of-perceptionassumptionasnumericalratings,butdoesnotrelyonasubjectrememberinganarbitraryandill-denedscale.Thismakesiteasiertostopthetestandre-startitlateriffatiguebecomesafactor.Thepurposeofchoosingthemostorleastsimilarsoundsinagroupistocreatebinarycomparisonsbetweenpairsofpairsofsounds.Thatis,wewanttoknowhowthedistancesbetweendierentpairsofsoundscompare.Makingbinarycomparisonsismore 57

PAGE 58

tediousthannumericallyratingthedierencebetweensounds.Butthemoresoundsthereareinagroup,themorebinarycomparisonsaremadebyprovidingfeedbackaboutthemostsimilarorleastsimilarpairsofsounds.Inagroupof3sounds,thereare)]TJ /F4 7.97 Tf 5.48 -4.38 Td[(32=3possiblepairs,sochoosingthemostsimilarpairprovidestwobinarycomparisons|itindicatesthatthechosenpairhasasmallerdistancebetweensoundsthanthesoundsintheothertwopairs.Inagroupof4sounds,thereare)]TJ /F4 7.97 Tf 5.48 -4.38 Td[(42=6possiblepairs,sothenumberofbinarycomparisonsprovidedbychoosingthemostsimilarpairincreasesto5.Althoughwedidnottestagroupof5sounds,thenumberofbinarycomparisonsinthiscasewouldbe9.Soincreasingthegroupsizecouldbeawaytosignicantlyreducetheamountoftimerequiredtocompletetheexperiment,althougheachindividualtrialdoesbecomemoredicult.Givenhowthisapproachcouldbecomeimpracticalasthenumberofsoundsinthedatasetincreases,itisimportanttoconsiderthemostecientwaytocollectusefulinformationfromasubject.Wedevelopedanapproachdesignedtoextractasmuchinformationaspossibleasquicklyaspossibleinthebeginning,andthentakemoretimetollingapsininformation,iftheyexisted,closertotheendoftheexperiment.Thus,themakeupofthetrialsintermsofthechosengroupingsofsoundswasnotxedatthebeginning.Sometrialshaveanswersthatareverycleartothesubjects,whileothertrialshaveanswersthatrequirerepeatedlistenstoarriveat.Insettinguptheexperiment,weemphasizedtrialsinvolvingsoundsingroupsof4inthebeginning,whicharemorediculttocompletelyassessthantrialsinvolvinggroupsof3.Tocompensate,wegavesubjectstheoptiontoskiptrialsthatwerediculttoassessquickly.Subjectswereinstructedtonotdwellondicultassessments,thoughtheywereaskedtoprovideinformationwheneverpossible.Thisservedtodecreasetheamountoftimefortheexperimentandincreasethereliabilityoftheinformationprovided. 58

PAGE 59

Ourmethodissimilartothemethodoftriadiccomparisonsusedinearlyresearchbuttheuseof4soundsinagroupoersadistinctadvantage.Sincethegoalistorankallpairsofsounds,intheoryweneedbinarycomparisonsbetweenallpairsofpairs.Thisisrepresentedwithamatrix,justliketheadjacencymatrixforaweightedgraph.Onlyusinggroupsof3whenchoosingthemostorleastsimilarsoundsinagrouplimitsthebinarycomparisonstobebetweenpairsthathaveonesoundincommon,whichrepresentsasubsetoftheentiresetofpossiblecomparisons.Infact,sinceanumberoftrialsinvolvinggroupsof3areusedinthisexperiment,thesubsetofthematrixcorrespondingtothesecomparisonsisreadilyapparentintheresultsshowninFigure 4-1 .Byaddingtheuseofgroupsizesabove3,weareabletomakecomparisonsbetweenpairsofsoundswithnosoundincommon,inwhichcasewecouldllouttheentirecomparisonmatrixifwewished,thoughthiswouldbetime-consumingisnotnecessary.Fromthebinarycomparisons,thegoalistodeterminearankingofallthepairwisedistancesbetweensounds.Thennon-metricMDS,whichisranking-based,isusedtodeterminetheN-dimensionalembeddingofallthesounds.Inthemethodoftriadiccomparisons,therankingisdeterminedbysummingthenumberoftimesapairofsoundsisjudgedtobeeithermoresimilarthananotherpairofsounds.Thisisequivalenttowinningpercentageinsports.Butwinningpercentageonlyprovidesanaccuraterankingiftheentirebinarycomparisonmatrixislledoutcompletely.Inthecaseoftriadiccomparisons,thisisalreadyimpossiblebecausethegroupsizeislimitedto3.However,evenwithagroupsizeof4,wewouldprefertonothavetollouttheentirebinarycomparisonmatrix,asthiswouldtakeaverylongtime.Therefore,wewanttoviewthisasageneralrankingproblemgivenapotentiallysparsematrixofbinarycomparisons.WechoosetosolvethisusingtheColleymatrixmethod[ 38 ],forreasonswediscussinSection 2.4 59

PAGE 60

3.2.3DeterminingandInterpretingPerceptualAxesGiventheinstantaneoustimbrespacesproducedbytheMDSmethods,thegoalistointerpretorexplainthedimensionsofthespacesperceptually.Fortheindividualsubjectembeddings,thisiscomplicatedbythefactthatthespaceshavenoxedorientation.Therefore,itisnotclearwhat,ifany,aretheprincipalaxesofthespaces.Norisitclearwhatperceptualcriteriaoughttobeusedtodeterminetheprincipalaxes.CarrollandChang[ 21 ]proposedanalgorithmtotamodelthatpresupposestheexistenceofasetofprincipalperceptualaxescommontoallsubjects.Inthismodel,individualsdierintheirpersonalperceptualspacesbyplacingdierentweightsonthecommonsetofaxes.ThedistancebetweentwopointsjandkinthepersonalspaceofsubjectiisgivenbyEquation 3{2 ,d(i)jk=vuut rXt=1wit(xjt)]TJ /F3 11.955 Tf 11.96 0 Td[(xkt)2; (3{2)whereristhenumberofdimensionsofthespaceandwitistheweightplacedondimensiontbysubjecti.Thecoordinatesofpointsfy(i)jgintheindividualspacearerelatedtothecoordinatesofpointsfxjginthecommonspacebyEquation 3{3 ,y(i)jt=w1=2itxjt: (3{3)CarrollandChangclaimthataxesfoundusingthisweightedEuclideanmodelforMDSshouldbeconsideredtobepsychologicallyimportant.Theydoprovidesomeevidencetosupporttheirassertion,buttheideathateachpersonusesthesamesetoforthogonalaxeswithdierentweightsonthemtocharacterizetheirownperceptualspaceisanassumptionthatappearstohavegoneunchallengedanduntestedintheeldoftimbreresearchforthelast40years.TheweightedEuclideanmodeliscertainlyaconvenientmethodtocombineMDSresultsfrommultiplesubjects.Itmayevenbemorereliablethanthespatialembeddingsdeterminedforsubjectsindividuallybecauseitusesmoreinformationandthuscanpotentiallyeliminatenoiseanderrors.Butthemodel 60

PAGE 61

shouldnotserveasadefactopsychologicaltheory.Therefore,weapproachthemodelasatoolthatmaybeabletodetermineasetofprincipalaxes,andusecomplementarymethodstovalidateourobservations.Supposingthatwehavedeterminedaperceptuallymeaningfulaxisininstantaneoustimbrespace,wewanttouseeverymethodatourdisposaltointerpretitsmeaning.Aninterpretationinvolveseitheraperceptualdescriptioninwordsorananalyticalexplanationintermsoftheacousticfeaturethatvariesalongtheaxis.Wedescribeseveralmethodsthatcouldbeusedtoobtainaninterpretation,eitherperceptuallyoracoustically. 3.2.4AnalyticalMethodsofPerceptualAxisExplanationWedescribetwoanalyticalmethodsofperceptualaxisexplanation.Therstmethodistocollectthevalues,fpjgforeachofthensoundsprojectedontoanaxisandthevaluesofsomeacousticfeature,fqjg,ofthesounds,andcomputethecorrelationrpqbetweenthetwosets,givenbyEquation 3{4 ,rpq=nPpjqj)]TJ /F8 11.955 Tf 11.95 8.97 Td[(PpjPqj q nPp2j)]TJ /F1 11.955 Tf 11.95 0 Td[((Ppj)2q nPq2j)]TJ /F1 11.955 Tf 11.95 0 Td[((Pqj)2: (3{4)Thisisarelativemeasureofhowwellanacousticfeatureexplainsanaxis.WeproposeseveralacousticorphysicalfeaturestotestaspossibleinterpretationsoftheaxesproducedbyINDSCAL.Theproposedacousticfeaturesareonesthoughttoberelevantperceptually.However,itisnotclearaprioritheprecisemannerinwhichsomeofthesefeaturesoughttobecomputedfromthesounds.Therefore,testsinvolvingtheacousticfeaturescouldbetestsofthefeaturesthemselves,andnotjusttheaxesthattheyarebeingusedtointerpret.Inthiswork,wetestsevenacousticfeatureswhichcanbeeasilydeterminedorcomputedfromthesounds'additivesynthesisrepresentation,givenbyEquation 3{1 .ThefeaturesarelistedinTable 3-3 .Thevaluesofeachofthefeatures,labeled~p1;~p2;:::;~p7,foreachofthe16soundsislistedinTable 3-4 61

PAGE 62

Table3-3. Thesevenacousticfeaturesandtheirformulas,basedontheadditivesynthesisrepresentationinEquation 3{1 AcousticfeatureLabelFormula Fundamentalfrequency~p1f1Logfundamentalfrequency(semitoneoset)~p2log2(f1=440)12Spectralcentroid~p3PKk=1akfkOddharmonicsproportion~p4PKk=1kisoddakLoudness~p5PKk=1akL(fk)Loudness-weightedspectralcentroid~p6PKk=1akfkL(fk)Inharmonicitystandarddeviation~p7q 1 KPKi=1(fi if1)]TJ /F4 7.97 Tf 14.89 4.71 Td[(1 KPKi=1fi if1)2 Table3-4. Thevaluesofthesevenacousticfeaturesforeachofthe16sounds. Sound~p1~p2~p3~p4~p5~p6~p7 1442.70.101884.60.531.502991.50.000042392.3-1.981308.50.891.291777.20.000633164.8-17.001629.90.791.031830.10.048344130.4-21.05687.00.730.901054.20.000495392.4-1.981382.70.391.442053.30.000106221.0-11.921346.10.511.301914.40.006407220.9-11.931350.00.521.392166.00.000148440.00.00881.50.771.281253.60.006759218.1-12.151122.60.911.221791.80.0005210219.1-12.061153.50.491.261842.10.0001911259.4-9.141472.60.601.402375.10.0011912220.8-11.93604.00.830.97767.20.0012513109.5-24.081329.80.731.192011.10.0002014220.1-11.99494.40.590.90579.90.0014315246.5-10.021592.00.531.472541.70.0000816185.0-15.00952.30.911.191559.60.00052 Tocalculateloudness,wedeneafunctioncalledtheloudnessproportion,L(x),showninFigure 3-2 .ThisfunctionisderivedfromISO226:2003[ 1 ],whichdenestherequireddecibellevelforasinewaveateachfrequencytobeheardatapre-denedloudness.Weinvertoneoftheseequal-loudnesscurvestoyieldafunctionthatweightseachfrequencybyitsperceivedloudness.SinceISO226:2003isonlydenedfordiscrete 62

PAGE 63

. . . 2;000 4;000 6;000 8;000 10;000 12;000 0:5 1 1:5 2 Frequency(Hz) L(x) Figure3-2. L(x),theloudnessproportion,whichisaweightingoffrequenciesbytheirperceivedloudness,derivedfromISO226:2003[ 1 ],theequal-loudness-levelcontours.Thecurveisnormalizedtohaveameanvalueof1. frequencyvalues,weinterpolateatintermediatefrequencyvaluesusingapiecewisecubicspline.Wenormalizethecurvetohaveameanvalueof1.Thesefeaturesarebasedonpropertiesthatareknowntoberelevanttoperception.However,theprecisemathematicalformmostrelevanttoperceptionisunknown.Thefunctionsherearebasedonthebestinformationavailable.Therefore,thesecondanalyticalmethodofaxisexplanationistolearnacousticfeaturesofthesoundsrelevanttoperceptiondirectlyfromtheirprojectionsininstantaneoustimbrespace.Thereareanumberofmachinelearningmethodscapableofdoingthis.Unfortunately,thisisnotpracticalinourcasebecausethenumberofsoundsinourstudyistoosmallcomparedtothenumberofpossibleacousticparameters.Thus,thiswouldleadtoovertting. 3.2.5ExploratoryMethodsofPerceptualAxisExplanationOnewaytoexplainaperceptualaxisistohavesoundsbedisplayedonanaxisattheirprojectedvaluesandlistenedtobyatestsubjecttotrytodeterminewhatperceptualfeaturesthesubjectcanassociatewiththevariationobservedalongtheaxis.Insomewaysthisisabetterexplanationofanaxisbecauseitremainspurelyinthedomainofperception.However,itmaybeadiculttaskinthecasethattheaxiscorrespondsto 63

PAGE 64

aperceptuallycoherentdimensionthatdoesnotalreadyhaveaverbaldescriptionthatthesubjectisawareof.Anotherexploratorymethodistoapplyknowledgeofthedomaintomanuallysearchforpatternsinthedatathattthevariationalongaperceptualaxis.Inourcasewithonly16sounds,knowledgeofthedomainiscrucialtoparedownthetotalnumberofacousticparametersthatcanpotentiallyvary.Assumingappropriatedomainknowledgeisapplied,apatternfoundmanuallyinthismannercanbeobjectivelyveriedusingthecorrelationmethoddescribedinSection 3.2.4 .Inasense,amanually-determinedpatternandapre-speciedacousticfeaturearenotmuchdierent,andthismethodcouldbethoughtofasanalyticalbecauseithasananalyticalvericationstep. 3.2.6FindingAxesViaLeast-SquaresProjectionTheprocedureofinterpretingaxesusingobjectivefeaturesofthesoundscanbeinvertedtolookfortheaxesthatbestcorrelatewithacousticfeatures.Thisismostusefulwhentheacousticfeaturescanbedemonstratedtoberelevanttoperception.Theprocedureworksasfollows.Anacousticpropertythathasalinearrelationshipwithaperceptualaxiscanbeconsideredtobeaprojectionofthesoundsontothataxis.Therefore,ifanyacousticpropertyhasalinearrelationshipwiththeperceptualspace,thereshouldbeanaxisalongwhichthesoundscanbeprojectedtorecoverthevaluesoftheacousticproperty.Thebestpossiblealignmentofanacousticproperty,treatedasanaxis,andaperceptualspacecanbedeterminedbyassumingthepropertyisaone-dimensionalprojectionandusingleast-squaresregressiontondasetofweightsthatbestreproducethisprojection.LetAbethematrixofpointsrepresentingsoundsininstantaneoustimbrespace.Let~pbeasetofvaluesforsomeacousticpropertyofthesoundsandlet~bbethemean-centeredversionofthesamevalues.ThenwewanttondtheprojectionA^x=^bsuchthatjj^b)]TJ /F3 11.955 Tf 11.51 3.16 Td[(~bjjisminimized.ThisisequivalenttominimizingjjA^x)]TJ /F3 11.955 Tf 11.51 3.16 Td[(~bjj,whichisminimizedusingthepseudo-inverseofA,andthevalueof^xwhichminimizesjjA^x)]TJ /F3 11.955 Tf 10.81 3.15 Td[(~bjjis 64

PAGE 65

givenbyEquation 3{5 ,^x=)]TJ /F3 11.955 Tf 5.48 -9.69 Td[(ATA)]TJ /F4 7.97 Tf 6.58 0 Td[(1AT~b: (3{5)So^xdenestheaxisthatbestmatchestheacousticfeature~b,^bistheprojectionofthesoundsontothataxis,andjj^b)]TJ /F3 11.955 Tf 10.77 3.16 Td[(~bjjistheerrorbetweenthefeatureandit'sbest-matchingprojection.Itshouldbenotedthatwhilethisisalinearmethodformatching~pwithanaxisinthespace,~pcanbeanynon-linearcombinationoftheacousticpropertiesofthesounds.Caredoesneedtobetakenwhenexaminingthecorrelationvaluesfoundusingthismethod.Asthenumberofdimensionsapproachesthenumberofpointsinthedimension,randomvaluesassignedtothepointscouldbefoundtocorrelatehighlywiththespace.ThisisillustratedinFigure 3-3 ,whichshowsthecorrelationbetweenarandomsetofNvaluesandthebest-matchingprojectioninarandomD-dimensionalspaceofNpoints,asNandDarevaried.Inourcase,with16pointsand5or6dimensions,theexpectedrandomcorrelationisaround0.5.Thisrepresentsabaselinethatanycorrelationmustbeaboveusingthismethodtobeconsideredatallmeaningful.WeusethisproceduretolookforprincipalaxesofinstantaneoustimbrespacethatmaynotbediscoveredbyINDSCAL.WealsousethisproceduretoverifythattheaxesofthejointembeddingdeterminedbyINDSCALhaveacorrespondencewithaxesintheindividualembeddingsofeachsubject,sincecorrespondencebetweenthetwoiscentraltothejusticationoftheprincipalaxesfoundbyINDSCALbeingmeaningful. 3.3SynthesisThemostimportantconsiderationforsynthesisinaframeworkforunrestrictedtimbremanipulationisexibility.Asynthesismethodmustbeabletorecreateawiderangeoftimbresanddoitinawaysuchthatonetimbrecanbeseamlesslymorphedintoanother.InChapter2,additivesynthesisisidentiedasthepreferredmethod. 65

PAGE 66

. . . 2 4 6 8 10 12 14 16 18 20 Numberofdimensions Numberofpoints . . 0:2 0:4 0:6 0:8 Figure3-3. Illustrationoftheexpectedcorrelationbetweenarandomly-generatedsetofvaluesandthebest-matchingprojection(oraxis)foundinaspacewithrandomly-generatedpoints,asthenumberofpointsanddimensionsisvaried.Forthecaseof12pointsin10dimensions,theexpectedcorrelationis0.9.Thismeansthatinsuchaconguration,correlationsmustbeabove0.9tobemoremeaningfulthanrandom. Theinverseofsynthesisisanalysis.Asynthesismethodisbestutilizedwhenthereisamethodtodecomposeasignalinto,ortasignalto,asetofsynthesisparameters. 3.3.1AdditiveSynthesisRepresentationInitsmostgeneralform,additivesynthesisrepresentsasignalassimplythesumofasetofsinewavesofdierentamplitudes,phases,andfrequencies.AsthesamecanbesaidoftheFouriertransform,additivesynthesisinitsmostgeneralformisclearlycapableofrepresentinganypossiblesignal(andtimbre).However,oneproblemwithsimplyusingtheFourierdomainisthereisnoexplicitrepresentationoftimeortheevolutionofparametersovertime.Eachsinusoidhasaconstantamplitudeandfrequency.Sincetheonlywaytoimpacthowasoundevolvesovertimeinthisrepresentationistoaltertheamplitudesandphasesofthesinewaves,anysoundtobegeneratedmustbespeciedinitsentiretypriortosynthesis.Tofacilitatethesynthesisofsoundthatcanevolveinreal-time,thesinusoidsmusthavetime-dependent,ratherthanconstant,amplitudesandfrequencies.Initsnaiveform, 66

PAGE 67

thisversionofadditivesynthesissuersfromahugeincreaseinstoragerequirements.Wecanimagineratherthantvaluestorepresentasignaloflengtht,wehaveontheorderoft2valuestorepresenttsinusoidsandtheirevolutionovertime.Butluckilyforasystemforwhichthisincreaseisprohibitive,thesituationisnotasdire.Allowingsinusoidstohavetime-dependentamplitudesdecreasesthenumberofsinusoidsneededtorepresentasignal.Infact,technicallyonlyonewell-craftedsinusoidisneeded,butthissinusoidwouldhavearapidlychangingamplitudetomatchthevaluesofthesignalitissupposedtorepresent.Asaresult,thissinglesinusoidwouldnotbeinformativeoftheunderlyingtimbre,sowewouldlosetheabilitytomorphbetweentimbresinanintuitiveway.Thesolution,then,istohavearelativelysmallnumberofslowly-evolvingsinusoids.Thisdenitionisnotterriblyprecisebutitmeansthatthenumberofsinusoidsshouldbemuchlessthanthelengthofthesignal,andthetime-dependentamplitudesandfrequenciesofthesinusoidsshouldbesmooth,withoutanydiscontinuitiesorotherwiseunpredictablejumps.TheformofthismodelisgivenbyEquation 3{6 ,s(t)=KXk=1ak(t)cos 2 FstXu=1fk(u)+k!; (3{6)wheretisthecurrenttimestep,Kisthenumberofsinusoids,Fsisthediscretesamplingrate,ak(t)istheamplitudeofsinusoidkattimet,fk(t)isthefrequencyofsinusoidkattimet,andkisthephaseofsinusoidk.OneconstraintonthemodelthatwasmentionedisthatKT,whereTisthetotalnumberoftimesteps.AnotherconstraintthatweadoptisgivenbyEquation 3{7 ,fk(t)kf0(t); (3{7)wheref0(t)isthefundamentalfrequencyattimet.Inusingthismodel,wewishtorestrictourselvestomostlytonalnotes,correspondingtoapproximatelyperiodicwaveforms.Thiscanbethoughtofasaquasi-harmonicmodelofadditivesynthesis. 67

PAGE 68

Weintendforthismodeltoincludesomeslightinharmonicityasexhibitedbycertainstrings,sothattheerrorbetweenfk(t)andkf0(t)canincreaseaskincreases.Itwouldnotbediculttoextendthismodeltoincludenoiseandmoresevereinharmonicityforsynthesispurposes,althoughitwouldbemorediculttoanalyzesoundstottosuchamodel.Anumberofmethodshavebeenproposedforttingtoharmonicmodelsofadditivesynthesis,butthesemethodsbecomelessapplicableastheamountofinharmonicityincreases.InSection 3.3.2 ,wepresentanewmethodsuitedtoanalyzequasi-periodicwaveformsthatactuallybecomesabetterestimateofthetime-varyingparametersinthepresenceofcertaintypesofinharmonicity. 3.3.2MaskedInverseFastFourierTransformInthissection,wepresentanewmethodcalledtheMaskedInverseFastFourierTransform(MIFFT)foranalyzingquasi-periodicwaveformsandttingthemtoourmodelforadditivesynthesis.Aspresented,themethodassumesthatthefundamentalfrequencyisalreadyknownorcanbeestimatedusinganumberofmethodsthathavebeenproposedelsewhere.Thefactthatthefundamentalfrequenciesofthesoundstobeanalyzedinthisworkareknowninadvancemeansthatwedonotneedtoincorporateasteptoestimatethefundamentalfrequency,sowedonotfocusonthisportionofthemethod.TheideabehindtheMIFFTissomewhatphilosophicalinnature.Insignalanalysis,timeandfrequencyareconsideredtobecomplementaryyetopposingdomainsinwhichthecertaintyofasignal'svalueinonedomainresultsinuncertaintyintheotherdomain.MethodssuchaswaveletsortheShort-TimeFourierTransform(STFT)trytostrikeabalancebetweenthetwo.Butneithermethodincorporatesadaptationtotheinherenttimeandfrequencyseparationthatmayalreadyexistinasignal.Inboththetimedomainandfrequencydomain,thevalueofasignalatindexiishighlycorrelatedwiththevaluesati)]TJ /F1 11.955 Tf 11.84 0 Td[(1andi+1.Itmayalsobethecasethatsomewhathighcorrelationsarealsoseenatsomemultipleofanoset,sothatthevalueatindexiiscorrelatedwiththevalueati+kTandi)]TJ /F3 11.955 Tf 11.97 0 Td[(kT,whereTisaxedoset,andk21;:::;K, 68

PAGE 69

. . . 0 200 400 600 800 1;000 1;200 1;400 1;600 1;800 2;000 2;200 0 2;000 4;000 6;000 8;000 10;000 Frequency(Hz) Relativeamplitude Figure3-4. AnillustrationofdisjointsupportamongtherstthreeharmonicsofaC5trumpetnote,usingaportionoftheDiscreteFourierTransformofthesignalcorrespondingtothesound.Eachlargepeakinenergycorrespondstoaharmonic.Thedashedlinesrepresenttheboundariesofeachharmonic'ssupport.Notethattheenergyforeachharmonicisconcentratedwellwithintheboundariesofsupport,demonstratingthateachharmonichasessentiallydisjointsupport. whereKisaninteger.Portionsofasignalthatareuncorrelatedaresaidtohavedisjointsupport.AnillustrationofdisjointsupportamongharmonicsforatrumpetnoteisshowninFigure 3-4 .Weusethetermdisjointsupportratherthanatermsuchasindependentbecausewewanttoemphasizethefactthatsignalsarerepresentedascombinationsofbasisfunctions.Itisinthebasisfunctionsthatportionsofasignalaresaidtohavedisjointsupport,meaningthevalueofonebasisfunctionforthatsignalisuncorrelatedwiththevalueofanother.Combiningtheanalysesoftwoportionsofasignalthatarecorrelatedhelpstoincreasethecertaintyofourestimateofbothportions.However,combiningtheanalysesoftwoportionsofasignalwithdisjointsupportprovidesnoadditionalhelptotheanalysis 69

PAGE 70

. . . 0:002 0:004 0:006 0:008 0:010 0:012 0:014 0:016 0:018 -0.15 -0.1 -0.05 0 0.05 0.1 0.15 Time(s) Amplitude . . Harmonic1 . Harmonic2 . Harmonic3 . Harmonic4 . Harmonic5 . Harmonic6 Figure3-5. Aportionofthetime-domainsignalsfortherstsixharmonicsduringtheonsetofaC5trumpetnote,resolvedusingtheMaskedInverseFastFourierTransform(MIFFT)method. ofeither,andmaybeevenweakenstheanalysisofboth.Therefore,theanalysesofportionsofasignalwithdisjointsupportoughttoproceedindependently.Weusethisnotionofdisjointsupportandindependentanalysistomotivateamethodforquasi-periodicwaveformanalysis.Inaquasi-periodicwaveform,thedierentharmonicsthatcomposethewaveformhaveapproximatelydisjointsupportinthefrequencydomain,anexampleofwhichisshowninFigure 3-4 .Therefore,wecanapplyasimplemasktothecoecientscorrespondingtoeachharmonicandseparatethemforindependentanalysis.Becausewewishtoanalyzetheirtime-dependentproperties,afterweseparatetheminthefrequencydomainwecanthentransformthembacktothetime-domainforfurtheranalysis,asisshowninFigure 3-5 .Oneinterestingconsequenceofthisideaisthatthesignalscorrespondingtostringswithinharmonicitywherefk(t)>kf0(t)haveevenlessoverlappingsupportinthe 70

PAGE 71

frequencydomain,makingtheassumptionofdisjointsupportevenmorevalid.Thisisaclearadvantageofourmethodovershort-timeanalysismethodswheretheanalysisisbasedonaxedwindowinthetimedomain.Methodsusingaxed-windowarejustiedfromtheperspectiveofdisjointsupportwhenfrequenciesaretime-invariantandperfectlyharmonic,butthisislessvalidasinharmonicityandtime-dependentfrequencyvariationincreases.IntheMIFFTmethod,eachharmonichasacorrespondingmaskindicatingwhichfrequenciesconstituteitssupport.Eachreconstructedharmonic,hk(t),isgivenbyEquation 3{8 ,hk(t)=1 NN)]TJ /F4 7.97 Tf 6.58 0 Td[(1Xn=0m(k)nXnei2nt=N; (3{8)wherethefXngaretheFouriercoecientsofthesignalinthefrequencydomainandm(k)nisthevalueofthemaskforthekthharmonicforthenthfrequency.Foreachmask,wehave,m(k)n=8>><>>:1iffrequencynisinharmonick'ssupport0iffrequencynisnotinharmonick'ssupport; (3{9)whereKXk=1m(k)n1; (3{10)sothatanyFouriercoecientcanbelongtothesupportofatmostoneharmonic. 3.4InterfaceInterfaceisanimportantconsiderationforunrestrictedtimbremanipulation.Inthisworkweprioritizefacilitatingcompositionoverperformance,whichisapotentiallyimportantdistinction.Bothrequirealargedegreeofexpressiveness,butotherconsiderationsdierbetweenthetwo.Real-timeperformancerequiresauidityandcompactnessthatisnotdemandedbycomposition.Ontheotherhand,compositionrequiresanintuitive 71

PAGE 72

interfaceforecientlytranslatingfromideastoimplementation.Oncetheideasareimplementedbycomposition,performanceneedstoimitatetheminreal-time,butnotre-createthemfromscratch. 3.4.1HierarchicalorLayeredApproachThedierentrequirementsfordierentgoalsareonereasonwhyweproposeamodularandhierarchicalapproachtointerface.AswediscussinSection 2.5.2 ,multipleinterfacesmakesensefortimbremanipulation.Averbalinterfaceisusefulbecauseitcorrespondstothelanguagemusicianscurrentlyusetospecifytimbre.Agesturalinterfaceisadvantageousbecauseitgivesapersonaccesstoanumberofmanipulabletimbredimensionssimultaneously,likeamusicalinstrumentbutwithamuchbroaderpalette.Aninterfacebasedonorthogonalaxesofperceptionisdesirablebecauseitwouldbethemostdirectandcompactrepresentation,withcontrolparametersthatareguaranteedtointeractindependently.Inahierarchicalapproach,thesemultipleinterfacescanbebuiltontopofonecommoninterface,extendingthefundamentalinterfacewithhigh-levelspecications.Thecommoninterface,then,istheonlyonethatneedstodealdirectlywiththementalrepresentationorsynthesis.Thismodularitymakesiteasiertosubstitutedierenthigh-levelinterfacesfordierentpurposes.OurproposedhierarchyisoutlinedinFigure 3-6 .Thesinglearrowsinthisdiagramrepresentaone-to-onecorrespondencebetweenobjectsinonelayerandobjectsinanotherlayer.Themultiplearrowsrepresentsingleobjectsatahigherlayermappingtoasequenceofobjectsatalowerlayer,buildingtheevolutionofparametersatahighlevelontopoftheirinstantaneousspecicationatalowerlevel.Forthedirectness,compactness,andindependenceofparameters,wechoosetheorthogonaldimensionsoftimbrespaceasthecommoninterfacewithperceptionandsynthesis.Inadditiontohigher-levelinterfacesbeingbuiltontopofit,thisinterfacecanbeuseddirectly.Thiscanbeuseful,especiallywhenonewantstoavoidtheambiguityorcontradictionthatcancomefromacombination 72

PAGE 73

. Verbaldescriptions Trajectories Gestures Orthogonalperceptualaxes Synthesisspace Figure3-6. Theproposedhierarchyfortheentiresystem,servingasaninterfacefromhigh-leveltimbrespecicationtolow-levelsynthesis.Anynumberofhigh-levelobjectscanbeusedtointerfacewiththeinstantaneoustimbrespace.Eachobjectatthehighest-level,suchasagestureorverbaldescription,mapstoanumberofpointsintheinstantaneoustimbrespaceandspeciestheirorder.Thisabstractiongreatlysimpliesthespecicationfortheuser.Pointsininstantaneoustimbrespacethenmapdirectlytosynthesisparameters,preferablyinaone-to-onemannersothattheevolutionofasoundcouldbespeciedinreal-timeifnecessary. ofverbalattributesorgesturalparameters.Trajectoriesarespecicationsofcoordinatesintheorthogonalaxesoveraperiodoftime,whichisjustawaytoencapsulateanentireinteractionwiththeorthogonaldimensionsascontrolparameters. 3.4.2OrthogonalAxesasControlParametersTheorthogonalaxesaretheprimaryinterfaceonwhichwefocusinthiswork.TheexperimentalprocedurefordevelopingtheaxesisgiveninSections 3.2.2 and 3.2.3 .Anindividual'stimbrespacehasanarbitraryorientation,soakeyquestionindevelopingthisinterfaceiswhetherornotacertainsetoforthogonalaxescanbeconsideredinsomesensemoreprincipalthananother,orevenwhethermultiplesetsofprincipalaxesexist.Acoordinateininstantaneoustimbrespacedirectlycorrespondstoasetofinstantaneoussynthesisparameters.Therefore,apath(ortrajectory)inthisspacespeciestheevolutionofinstantaneoustimbreovertime,whichinturnspeciesasound.Intheory,thisisthemostperceptuallycompactrepresentationpossible,sinceonlydimensionsrelevanttoperceptionareincluded,whilenodimensionrelevanttoperceptionisomitted. 73

PAGE 74

Therepresentationusedinourworkdoesomitsomedimensionsofperceptionthatarenotdemonstratedbythe16soundsusedintheexperiment.Thesedimensionsincludenoiseandsilence.Thesetwodimensionscouldeasilybeincorporatedinafutureversionoftheworkwithalargerdataset.However,somedimensionsofinstantaneoustimbremayonlyemergewithinthecontextofdynamicstimuli,inwhichcasetheywouldnotbediscoveredbyourmethod.Weplantoinvestigatethispossibilityinthefuture.Usingtheorthogonalaxescorrespondstothesimultaneousmanipulationofseverallinearparameters.Thiscanbediculttodoinreal-timewithoutaninterfaceatahigher-levelwhichcanincorporatemultipletypesofgestures.However,ifreal-timeinteractionisnotaconcern,likeperhapsforcomposition,theparameterscanbeprogrammed.Thejointspecicationofalloftheprogrammedparameterchangescanbethoughtofasapaththroughinstantaneoustimbrespace.Ontheotherhand,ahigher-levelinterfacecouldeasetheinteractionevenwhenreal-timemanipulationoftimbreisnotaconcern. 74

PAGE 75

CHAPTER4RESULTSTheprimarygoalofthisworkistodevelopaframeworkforutilizingtimbreasfreelyasloudnessorpitchinmusicalcomposition.Thefocusexperimentallyinthisworkisdevelopingtheperceptualcomponentoftheframeworkandincreasingourunderstandingoftimbreasacomplexperceptualphenomenon.Inthischapter,wereporttheprimaryexperimentalresultsandthedataderivedfromthem,andanalyzethemindetail. 4.1TimbreSpaceExperimentTheprocedureusedforthisexperimentisdescribedinSection 3.2.2 .Inall,sixsubjectswereaskedtolistentoasetof16soundsinsmallgroupsof3or4andidentifythetwomostsimilarandtwoleastsimilarsoundsineachgroup.Providinginformationonthetwomostsimilarsoundsinagroupof4soundssaysthatamongthe)]TJ /F4 7.97 Tf 5.48 -4.38 Td[(42=6uniquepairs,thetwosoundsinthemostsimilarpairarenearertoeachotherthanthetwosoundsinanyoftheother5possiblepairs.Thisprovidescomparisoninformationbetweenthispairandtheother5pairs.Takentogether,allofthetrialsofthissortlloutacomparisonmatrixthatisP-by-P,whereP=)]TJ /F4 7.97 Tf 5.48 -4.38 Td[(162=120.Thecomparisonmatrixcontains)]TJ /F4 7.97 Tf 5.48 -4.38 Td[(1202=7120uniqueentries.TheP-by-Pcomparisonmatrix,M,forsubject2isshowninFigure 4-1 .Awhitesquarecanbethoughtofasavictoryandindicatesthatthepairofsoundsintherowismoresimilarthanthepairofsoundsinthecolumn.Ablacksquarecanbethoughtofasalossandindicatesthatthepairofsoundsintherowislesssimilarthanthepairofsoundsinthecolumn.Agraysquarerepresentsthelackofanycomparisoninformation.Thematrixisanti-symmetricsothatM(i;j)=)]TJ /F3 11.955 Tf 9.3 0 Td[(M(j;i).Thecomparisonmatrixforeachsubjectissomewhatsparse,butthereisenoughinformationtoyieldarankingofthepairsfrommostsimilartoleastsimilar.Colley'smatrixmethodisusedtodotheranking.Thesamecomparisonmatrixorderedfrommost 75

PAGE 76

. . . 20 40 60 80 100 120 20 40 60 80 100 120 Pairtwo Pairone Figure4-1. TheP-by-PcomparisonmatrixforsubjectM.Eachrowandeachcolumnrepresentsapairofsounds.Eachsquareindicatestheoutcomeofthecomparisonbetweentherowpairandthecolumnpair.Awhitesquareindicatesthatthesoundsintherowpairarejudgedbythesubjecttobemoresimilarthanthesoundsinthecolumnpair.Ablacksquareindicatesthatthesoundsintherowpairarejudgedbythesubjecttobelesssimilarthanthesoundsinthecolumnpair.Agraysquareindicatesthatthereisnocomparisonbetweenthetwopairs. similartoleastsimilarisgivenbyFigure 4-2 .Thepairsatthetopoftherankinghavethemostevidenceinfavoroftheirbeingthemostsimilarpairs.Kruskal'snon-metricMDSalgorithmisusedtocomputethebestspatialembeddinginDdimensionsgiventherankingofdistancesbetweenpairsofpoints.TheerroroftheembeddinggiventherankingsisreferredtobyKruskalasthestress.Figure 4-3 showsthestressofeachembeddingforeachsubjectforarangeofdimensionalities.Wechoosetocompute5-dimensionalspaces.Oneofthereasonsforthisisthatthereductioninstressafter5dimensionsstartstobecomesoinsignicantthatbeyondthatoverttingtothenoiseinthedataislikelytostartoccurring.Anotherreasonisthatwithonly16sounds,usingmorethan5dimensionswouldstarttoimpairourabilityto 76

PAGE 77

. . . 20 40 60 80 100 120 20 40 60 80 100 120 Pairtwo Pairone Figure4-2. ThesamecomparisonmatrixasinFigure 4-1 exceptwiththepairsorderedfrommostsimilartoleastsimilarbyColley'smatrixmethod.Blacksquaresabovethemaindiagonalandtheircorrespondingwhitesquaresbelowthemaindiagonalrepresentviolationsoftransitivity,sincetheyindicatethatapairthatisrankedmoresimilarthananotherpairisjudgedbythesubjecttobelesssimilar. . . 2 4 6 8 0 0:1 0:2 0:3 0:4 0:5 Numberofdimensions Stress . . Subject1 . Subject2 . Subject3 . Subject4 . Subject5 . Subject6 Figure4-3. PlotofthestresscalculatedforthecongurationfoundbyKruskal'snon-metricMDSforeachsubjectforeachnumberofdimensions.Thereductioninstressappearstostopbeingsignicantafteradimensionalityofeither5or6. 77

PAGE 78

. . . )]TJ /F1 11.955 Tf 9.29 0 Td[(1 0 1 2 )]TJ /F1 11.955 Tf 9.29 0 Td[(2 )]TJ /F1 11.955 Tf 9.29 0 Td[(1 0 1 2 )]TJ /F1 11.955 Tf 9.3 0 Td[(2 )]TJ /F1 11.955 Tf 9.3 0 Td[(1 0 1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 Firstdimension Seconddimension Thirddimension Figure4-4. Firstthreedimensionsoftheindividualinstantaneoustimbrespaceforsubject2. distinguishstructureinthespacefromrandomvariation,anissuethatisdiscussedinSection 3.2.6 .Figure 4-4 showstherstthreedimensionsofthe5-dimensionalsolutionfoundforsubject2.Thesubjectwhosespatialembeddinghastheloweststressin5dimensionsissubject6.Figure 4-5 showstherstthreedimensionsofsubject6'sspatialembedding. 4.2JointInstantaneousTimbreSpaceINDSCALisusedtocombinespatialinformationfrommultiplesubjectsintoonecommonspace.Eventhoughwechoose5dimensionsfortheindividualembeddings,wearefreetochooseadierentnumberofdimensionsforthecommonspace.Havingmorethan5commondimensionswouldmakesenseifsubjectsdierinwhichdimensionstheyperceivemoststrongly,sincenosubjectwouldincludeallofthecommondimensionsintheirindividualembedding.Havinglessthan5commondimensionscouldbeusefulbecause,withtheenergymoreconcentrated,itmaycauseINDSCALtondclearerperceptualdimensionsthanwhentheenergyisspreadoutoveralargernumberof 78

PAGE 79

. . . )]TJ /F1 11.955 Tf 9.29 0 Td[(1 0 1 2 )]TJ /F1 11.955 Tf 9.3 0 Td[(2 )]TJ /F1 11.955 Tf 9.3 0 Td[(1 0 1 )]TJ /F1 11.955 Tf 9.29 0 Td[(2 )]TJ /F1 11.955 Tf 9.29 0 Td[(1 0 1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 Firstdimension Seconddimension Thirddimension Figure4-5. Firstthreedimensionsoftheindividualinstantaneoustimbrespaceforsubject6. dimensions.Toinvestigatethesepossibilities,weexamineINDSCALsolutionsfor4,5,and6dimensions.Figure 4-7 showstherstthreedimensionsofthe4-dimensionaljointembeddingcomputedbyINDSCAL.Figure 4-8 showstherstthreedimensionsofthe5-dimensionalINDSCALembedding,andFigure 4-6 showstherstthreedimensionsofthe6-dimensionalINDSCALembedding.INDSCALalsoproducesasetofweightsthateachsubjectapplies(actuallytheyapplythesquarerootsoftheweights)tothecommondimensionstoyieldtheirindividualspaces.Table 4-1 givesthesquarerootsoftheweightscomputedbyINDSCALforsixsubjectsandfourdimensionsofinstantaneoustimbre.Thevaluesarenormalizedtosumto1sothattheyareeasiertocompare.Table 4-2 givesthesquarerootsoftheweightsforthecaseofvedimensionsandTable 4-3 givesthesquarerootsoftheweightsforsixdimensions.Inallthreecases,theinputtoINDSCAListheEuclideandistancescomputedinthe5-dimensionalindividualembeddings. 79

PAGE 80

. . . )]TJ /F1 11.955 Tf 9.3 0 Td[(0:2 0 0:2 0:4 )]TJ /F1 11.955 Tf 9.3 0 Td[(0:4 )]TJ /F1 11.955 Tf 9.3 0 Td[(0:2 0 0:2 )]TJ /F1 11.955 Tf 9.3 0 Td[(0:4 )]TJ /F1 11.955 Tf 9.3 0 Td[(0:2 0 0:2 0:4 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 Firstdimension Seconddimension Thirddimension Figure4-6. Firstthreedimensionsofthe4-dimensionalINDSCALembeddingofjointinstantaneoustimbrespace. . . )]TJ /F1 11.955 Tf 9.29 0 Td[(0:2 0 0:2 0:4 )]TJ /F1 11.955 Tf 9.3 0 Td[(0:4 )]TJ /F1 11.955 Tf 9.3 0 Td[(0:2 0 0:2 )]TJ /F1 11.955 Tf 9.3 0 Td[(0:4 )]TJ /F1 11.955 Tf 9.3 0 Td[(0:2 0 0:2 0:4 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 Firstdimension Seconddimension Thirddimension Figure4-7. Firstthreedimensionsofthe5-dimensionalINDSCALembeddingofjointinstantaneoustimbrespace. 80

PAGE 81

. . . )]TJ /F1 11.955 Tf 9.3 0 Td[(0:2 0 0:2 0:4 )]TJ /F1 11.955 Tf 9.3 0 Td[(0:4 )]TJ /F1 11.955 Tf 9.3 0 Td[(0:2 0 0:2 )]TJ /F1 11.955 Tf 9.3 0 Td[(0:4 )]TJ /F1 11.955 Tf 9.3 0 Td[(0:2 0 0:2 0:4 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 Firstdimension Seconddimension Thirddimension Figure4-8. Firstthreedimensionsofthe6-dimensionalINDSCALembeddingofjointinstantaneoustimbrespace. Table4-1. ThesquarerootsoftheweightscomputedbyINDSCALforsixsubjectsandfourdimensionsofinstantaneoustimbre,normalizedtosumto1. Subjectp w1p w2p w3p w4Total 10.0560.0460.0470.0330.18020.0440.0530.0460.0300.17330.0500.0410.0490.0150.15440.0470.0500.0420.0210.16050.0570.0400.0120.0460.15560.0570.0360.0420.0430.178 Total0.3100.2660.2360.1871.000 Table4-2. ThesquarerootsoftheweightscomputedbyINDSCALforsixsubjectsandvedimensionsofinstantaneoustimbre,normalizedtosumto1. Subjectp w1p w2p w3p w4p w5Total 10.0480.0410.0410.0280.0170.17520.0400.0470.0400.0260.0140.16930.0430.0360.0430.0120.0110.14640.0390.0450.0370.0180.0440.18350.0520.0360.0090.0410.0200.15860.0510.0320.0360.0390.0120.170 Total0.2740.2360.2080.1640.1181.000 81

PAGE 82

Table4-3. ThesquarerootsoftheweightscomputedbyINDSCALforsixsubjectsandsixdimensionsofinstantaneoustimbre,normalizedtosumto1. Subjectp w1p w2p w3p w4p w5p w6Total 10.0430.0360.0360.0230.0250.0130.17520.0360.0420.0350.0270.0130.0130.16830.0350.0320.0410.0110.0240.0080.15240.0310.0410.0340.0080.0250.0390.17950.0510.0320.0050.0340.0140.0170.15360.0450.0280.0300.0370.0300.0060.174 Total0.2400.2110.1820.1400.1310.0961.000 Eachsubject'spersonalweightingofthejointembeddingshouldapproximatelyre-producetheirindividualembedding.Thusforeachsubjecttherearetwopersonalspaceswecanexamine,theoriginalindividualembeddingproducedbyKruskal'salgorithm,andtheweightedversionofthejointembeddingproducedbyINDSCAL.Theindividualembeddingsarediculttocomparedirectlybecausetheyhaveanarbitraryorientationandcannotbelinearlyttoeachotherusingleast-squaressincenoteverylineartransformpreservesthecongurationofpointsinthespace(shearingdoesnot).Therefore,theweightedembeddings,whichareanorientedapproximationoftheindividualembeddings,areusefulasawaytocompareeachsubject'spersonalinstantaneoustimbrespace.Figure 4-9 showsthersttwodimensionsofeverysubject's6-dimensionalweightedembedding.AscanbeseenfromthisgureandtheweightsinTable 4-2 ,thersttwodimensionsofjointinstantaneoustimbrespaceareexhibitedverysimilarlybyallsixsubjects.Table 4-10 showsthethirdandfourthdimensionsofthesameweightedembeddings.Inthesedimensions,dierencesaremorereadilyapparent.Whilesubjects1and2remainverysimilar,subjects3and5exhibitsignicantdierencesintheirspaces.Itcannotbeassumedthattheweightedembeddingsareaccuratereproductionsoftheindividualembeddingsforeachsubject.Theweightedembeddingsallrepresentanon-uniformbutlinearscalingofthedimensionsofthejointembeddingandhavethesameaxes.Wecantesttowhatextenttheseaxesexistintheindividualembeddingsusing 82

PAGE 83

. . . )]TJ /F1 11.955 Tf 9.3 0 Td[(0:3 )]TJ /F1 11.955 Tf 9.29 0 Td[(0:2 )]TJ /F1 11.955 Tf 9.3 0 Td[(0:1 0 0:1 0:2 0:3 0:4 )]TJ /F1 11.955 Tf 9.3 0 Td[(0:3 )]TJ /F1 11.955 Tf 9.3 0 Td[(0:2 )]TJ /F1 11.955 Tf 9.3 0 Td[(0:1 0 0:1 0:2 0:3 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 Firstdimension Seconddimension . . Subject1 . Subject2 . Subject3 . Subject4 . Subject5 . Subject6 Figure4-9. Visualcomparisonofthersttwodimensionsofeachsubject'sweightedembeddingasproducedbyINDSCAL.Subjectsdonotexhibitmuchdierenceinthesedimensions leastsquares.Table 4-4 showsthecorrelationbetweentheprincipalaxesdeterminedbyINDSCALandtheclosest-matchingaxisineachoftheindividualspaces. 4.2.1ThePerceptualDimensionsofTimbreOneofthecentralaimsofthisworkistoexplainperceptualdimensionsofinstantaneoustimbre.SinceINDSCALclaimstodiscovertheaxesthatarethemostmeaningfulperceptually,theseaxesareagoodplacetostart.AsisexplainedinSection 3.2.3 ,thereareanumberofwaysinwhichwecanattempttoexplaintheseaxes.Table 4-5 showsthecorrelationsbetweeneachaxisandsevendierentacousticpropertiesofthesounds.The 83

PAGE 84

. . . )]TJ /F1 11.955 Tf 9.3 0 Td[(0:3 )]TJ /F1 11.955 Tf 9.29 0 Td[(0:2 )]TJ /F1 11.955 Tf 9.3 0 Td[(0:1 0 0:1 0:2 0:3 )]TJ /F1 11.955 Tf 9.3 0 Td[(0:2 )]TJ /F1 11.955 Tf 9.3 0 Td[(0:1 0 0:1 0:2 0:3 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 Thirddimension Fourthdimension . . Subject1 . Subject2 . Subject3 . Subject4 . Subject5 . Subject6 Figure4-10. Visualcomparisonofthethirdandfourthdimensionsofeachsubject'sweightedembeddingasproducedbyINDSCAL.Subjectsstarttoexhibitnoticeabledierencesinthesedimensions. Table4-4. Thecorrelationbetweentheprincipalaxes~d1;~d2;:::;~d6determinedbyINDSCALandtheindividualembeddingsofthesixsubjects. Subject~d1~d2~d3~d4~d5~d6 10.960.960.880.720.840.3920.890.960.900.780.540.4330.870.880.930.400.870.3040.860.980.930.340.891.0050.980.890.340.900.610.6060.980.850.820.950.990.28 84

PAGE 85

Table4-5. Thecorrelationsbetweentheprincipalaxes~d1;~d2;:::;~d6determinedbyINDSCALandsevenacousticproperties~p1;~p2;:::;~p7ofthesounds. ~p1~p2~p3~p4~p5~p6~p7 ~d10.930.950.060.170.400.090.28~d20.300.300.720.320.530.600.54~d30.320.350.510.180.690.650.44~d40.010.100.150.380.360.280.36~d50.310.380.300.330.050.350.01~d60.030.080.040.300.150.020.03 acousticpropertiesare~p1=fundamentalfrequency,~p2=logfundamentalfrequency,~p3=spectralcentroid,~p4=oddharmonicsproportion,~p5=loudness,~p6=loudness-weightedspectralcentroid,and~p7=inharmonicityfactorstandarddeviation.ThesepropertiesaredescribedinmoredetailinTable 3-3 .Fromthecorrelationanalysis,therstaxisisclearlyrelatedtopitch,withthecorrelationbetween~d1and~p2being0.95.Theotheraxesarenotasclearfromthecorrelationanalysis.Theanalysissuggestsasignicantrelationshipbetweenthesecondaxisandspectralcentroid,indicatingthat~d2maybelinkedtobrightness.Thethirdaxishasapotentiallysignicantrelationshipwithloudness.Interestingly,thefourthandsixthaxesshownosignicantcorrelationwithanyofthesefeatures.Anothermethodofinvestigatinganaxisistosimultaneouslyobservethesoundsprojectedontotheaxisandlistentothesounds,thenattempttondaperceptualpatternthatexplainsthevariationalongtheaxis.Forthesecondaxis,brightnessdoesseemtohaveastrongcorrelationwiththeaxisperceptually.Interestingly,theinharmonicsounds,whicharenotnecessarilythebrightest,areatthefarendoftheaxiswiththebrightestsounds.Thisexplainsinpartwhyinharmonicityhasafairlystrongcorrelationwiththisaxisandwhyspectralcentroidmayhavehadaweakercorrelationthanitshouldhave.Thethirdaxis,too,seemsperceptuallytohaveastrongcorrelationwithbrightness,exceptthistimetheinharmonicsoundsareplacedattheendofthespectrumwiththeleastbrightsounds. 85

PAGE 86

Table4-6. Thenewcorrelationsbetweentheprincipalaxes~d1;~d2;:::;~d6determinedbyINDSCALandsevenacousticproperties~p1;~p2;:::;~p7ofthesoundsafterremovingsounds3,6,and8fromthecorrelationcalculations. ~p1~p2~p3~p4~p5~p6~p7 ~d10.910.940.300.250.370.220.19~d20.510.510.890.540.900.910.67~d30.580.580.830.230.900.810.64~d40.000.100.260.260.290.290.21~d50.240.330.290.380.090.320.15~d60.040.150.080.270.150.050.19 Asaresultoftheseobservations,themostinharmonicsounds,sounds3,6,and8(ascanbeseeninTable 3-4 ),wereremovedandthecorrelationswerere-calculated.Table 4-6 presentstheresults.Thesecondandthirddimensionsnowshowaverystrongcorrelationwithspectralcentroid,loudness,andloudness-weightedspectralcentroid.Thisindicatesthatthesetwodimensionsbothoughttobeinterpretedasbrightnessbutwithoutliers.Thissamemethodofinvestigationwiththefourthaxisalsoyieldedanobservablepattern.Inthedataset,therearearelativelylargenumberofAnotes.ThefourthaxisgroupsalloftheAnotes,alongwithoneBnote,ononesideoftheaxis,andplacestherestofthenon-Anotesontheothersideoftheaxis.GiventheprevalenceofAnotesinthedata,itisnotsurprisingthatAversusnon-Aemergedasadeningdistinction.Butitalsospeakstoaperceptualphenomenon,whichisthatnotesanoctaveapartareoftenconfusedforoneanother.Thus,thereisaninherentperceptualsimilaritybetweennotesanintegernumberofoctavesapart,andthisshowsupinthefourthaxis.ThefthINDSCALdimensionrequiredanumberoflistensbecauseitsvariationwasnotasapparentasthevariationofpitchandbrightnessobservedintherstfouraxes.Butafairlyclearperceptualaxisdideventuallyemerge.Onthelower(thoughthisisanarbitraryorientation)sideoftheaxis,soundscouldbedescribedassmoothorupbeat.Onthehighersideoftheaxis,soundscouldbedescribedasbitingandnegative.Thesetraitsarenotknowntobecorrelatedwithanyparticularacousticproperties,thereforeananalyticalvericationcannotbeperformedtodetermineacorrelationcoecient. 86

PAGE 87

Table4-7. AsummaryoftheinterpretationsofthedimensionsofinstantaneoustimbrefoundbyINDSCAL. DimensionInterpretationPrimarymethodused 1AbsolutepitchAnalytical2BrightnesswithoutliersPerceptualandanalytical3BrightnesswithoutliersPerceptualandanalytical4RelativepitchPerceptualandanalytical5Smoothness,positivity,bitePerceptual6Unknown Table4-8. Thecorrelationbetweensevenacousticproperties~p1;~p2;:::;~p7ofthesoundsandtheirbest-matchingaxesinthejointembedding. ~p1~p2~p3~p4~p5~p6~p7 0.981.000.920.750.930.940.84 ThesixthINDSCALdimensionwasfoundtobeverydiculttoexplainperceptuallyandnoconclusioncanbedrawn.AscanbeseenfromTable 4-4 ,however,thisdimensionseemstoprimarilyexistintheindividualinstantaneoustimbrespaceofonlyonesubject,withwhomthereisnearlyperfectcorrelation.Thereforeitisnotsurprisingthatthisdimensionisdiculttoexplain.Itwouldbeextremelyinterestingifthisonesubjectwereabletoprovideaperceptualexplanation,butthistestwasnotabletobedone.AsummaryoftheinterpretationsforthedimensionsofinstantaneoustimbrefoundbyINDSCALisgivenbyTable 4-7 .CarrollandChangarguethattheorientationfoundbyINDSCAListheonlyorientationthejointembeddingcanhave,sinceaccordingtothemodelthereisnolinearconnectionbetweenanyotheraxesinthejointembeddingandaxesintheindividualembeddings.However,itisstillinterestingtotestiftheacousticpropertiescorrelatewithaxesinthejointembeddingthatarenotalignedwiththeprincipalaxesfoundbyINDSCAL.Table 4-8 liststhecorrelationsfoundbetweentheacousticpropertiesofthesoundsandtheirbest-matchingaxesinthejointembedding.Wealsotesttheindividualspacestoseeifthereareaxesinthesespacesthatcorrelatewiththeacousticpropertiesthoughttoberelevanttoperception.ThesecorrelationsarelistedinTable 4-9 87

PAGE 88

Table4-9. Thecorrelationbetweensevenacousticproperties~p1;~p2;:::;~p7ofthesoundsandtheirbest-matchingaxesintheindividualembeddingsofeachsubject. Subject~p1~p2~p3~p4~p5~p6~p7 10.940.970.820.530.880.870.7620.960.950.920.670.950.950.8730.910.930.870.680.910.870.6940.870.910.920.500.850.890.7150.950.980.860.470.760.760.6960.970.980.830.650.900.890.66 . . )]TJ /F1 11.955 Tf 9.29 0 Td[(25 )]TJ /F1 11.955 Tf 9.3 0 Td[(20 )]TJ /F1 11.955 Tf 9.3 0 Td[(15 )]TJ /F1 11.955 Tf 9.3 0 Td[(10 )]TJ /F1 11.955 Tf 9.3 0 Td[(5 0 )]TJ /F1 11.955 Tf 9.29 0 Td[(25 )]TJ /F1 11.955 Tf 9.29 0 Td[(20 )]TJ /F1 11.955 Tf 9.29 0 Td[(15 )]TJ /F1 11.955 Tf 9.29 0 Td[(10 )]TJ /F1 11.955 Tf 9.3 0 Td[(5 0 Logfundamentalfrequency Projectedlogfundamentalfrequency . . Subject5 . Subject6 Figure4-11. Thex-axisvaluesarethelogfundamentalfrequenciesofthe16sounds.They-axisvaluesarethebest-matchingprojectionsintheindividualembeddingsofinstantaneoustimbrespaceforsubjects5and6.Thecorrelationsbetweenthesetsarelistedincolumn3ofTable 4-9 Figure 4-11 showsthelogfundamentalfrequencyofeachsoundplottedagainstthebest-matchingprojectionofthesoundsfoundintheindividualspacesofsubjects5and6.Outofallofthesubjects,thesetwohadthehighestcorrelationbetweentheirspacesandlogfundamentalfrequency.Theinharmonicityfeaturedoesnotappeartohaveaparticularlystrongcorrelationwiththeinstantaneoustimbrespaces,butduetotherelativelackofinharmonicsoundsinthedataset,thisdoesnotnecessarilymeanthefeaturecalculatedinthisworkisnotanappropriatemeasureofperceptualinharmonicity.Anobservationaboutthisfeature 88

PAGE 89

. . . 0 0:5 1 1:5 2 2:5 3 0:4 0:5 0:6 0:7 0:8 0:9 Exponent Correlation . . Subject1 . Subject2 . Subject3 . Subject4 . Subject5 . Subject6 Figure4-12. Theinharmonicityiscalculatedastheproportionally-weightedstandarddeviationoffi=(if0)foralloftheharmonics.Thisgraphshowshowthecorrelationofthisstatisticwithsubjects'individualinstantaneoustimbrespaceschangesasthevalueisraisedtodierentexponents.Thecorrelationpeaksatdierentexponentsforeachsubject. isshowninFigure 4-12 .Theinharmonicityiscalculatedasthestandarddeviationoffi=(if0)foreachharmonic,wherefiisthefrequencyofharmoniciandf0isthefundamentalfrequencyofthenote.Thestandarddeviationisnottheonlypossiblechoiceofsummarystatistic.Anotheristhemeanorthevariance.Sincethevarianceisthesquareofthestandarddeviation,anumberofpowersofthestandarddeviationwereexaminedtoseeiftheygaveanyimprovementincorrelationoverthestandarddeviationforeachofthesubjects'instantaneoustimbrespaces.Eachsubjecthasadierentpeakexponentvalue.Theexponentseachrepresentnon-linearwarpingsoftheinharmonicityfeaturewhenviewedasaprojectionintimbrespace.Thefactthateachsubjecthasafairlyhighpeakcorrelationsuggeststhattheparticularfeaturebeingusedforinharmonicitymayberankingtheinharmonicitiescorrectlybutnotmatchingperceptioninanoptimalway.Thusotherwaysofcalculatinginharmonicitymayneedtobeinvestigated. 89

PAGE 90

CHAPTER5CONCLUSIONSInthisworkwehaveproposedacomprehensive,perception-centricframeworkforthedigitalmanipulationoftimbreincomposition.Thishasbeenaninherentlyinter-disciplinaryapproach,drawingonknowledgefromdomainssuchascomputerscience,music,signalprocessing,andpsychology.Inadditiontoourcontributiontothedevelopmentofasystemforunrestrictedtimbremanipulation,wehavemadeanumberofpotentialcontributionstosignalprocessing,experimentalprocedure,andmusicology.Amongourprimarycontributionsisthedevelopmentofanewrepresentationfortimbre,whichisthersttosimultaneouslyrepresentperceptualdimensionsoftimbreandbesuitableforsynthesizingsoundsfromthisrepresentation.Previousattemptsatarepresentationoftimbreusingpsychologicalexperimentshavenotsucceededinidentifyingfundamentaldimensionsoftimbreandhavenotbeensuitableforsynthesis.Ontheotherhand,representationsoftime-varyingacousticpropertiesthataresuitableforsynthesishavechieybeendevelopedasmodelsofexistingmusicalinstrumentsandusedtheminimizationofsignalreconstructionerrorastheirprimarydevelopmentobjective.Webelievethisapproachisfundamentallyincapableofdevelopingacorrespondencebetweentime-varyingpropertiesandperceptualdimensions.Thusourapproachovercomesthelimitationsimposedbypreviousattempts.Likesomeoftherepresentationsbasedontime-varyingacousticparameters,wehaveproposedaphasespacerepresentation,inwhichallattributesatasinglemomentintimearetreatedaspointsinthespace,andpathsinthespacerepresenttheevolutionoftheseattributesovertime.Exceptinourcase,theseattributesareperceptualinnature.Inthiswork,thenotionofinstantaneoustimbre,whilehintedatinpriorwork,hasbeenmadeexplicitlydistinctfromdynamictimbre.Thiscreatesanewhierarchyoftimbrethathasimplicationsforbothasystemtoachieveitsmanipulationandforitsstudyingeneral. 90

PAGE 91

Tolearnaspaceofinstantaneoustimbre,wehaveproposedastudythatappliedmachinelearningtechniquestodatacollectedfromhumansubjects.Ashasbeenmentioned,thisgeneraltypeofpsychologicalstudyhasbeenundertakenbefore.Butthisisthersttimesuchastudyhasbeenattemptedexclusivelywithinstantaneoustimbre.Wehavealsoimproveduponearliermethodsinanumberofways.Weexpandedtherangeofvariationinthedatatoincludenotesofdierentpitch.Procedurally,wehavecollectedresponsesfromsubjectsasrelativecomparisonsratherthanabsolutejudgments,whichweargueeliminatesapotentiallylargesourceofinconsistencyintheobservations.Atthesametime,wehaveimproveduponthepriormethodoftriadiccomparisonsbyexpandingthegroupsizeinatrialtomorethanthreeatatime.Thisallowstheentirematrixofbinarycomparisonstobeaccessible,ratherthanasubsetofthematrixcorrespondingtopairswithonesoundincommon.Wehavecompensatedfortheincreasedtimeittakestocollectresponsesandimproveduponthemethodoftriadiccomparisonsbyframingtheproblemoforderingthepairsbytheirsimilarityasageneralrankingproblem,givenanincompletebinarycomparisonmatrix.Thisisamoreprincipledapproachthansimplewinningpercentage,asusedintriadiccomparisons,andleveragesadvancesinrankingtheory.Ingeneral,wehavebeenabletodrawanumberofconclusionsfromourexperiments.Investigationintothenatureofinstantaneoustimbreindicatesthatthereareatleast5coherentdimensionsreadilyapparentjustfromacollectionof16sounds.Priorstudiesinsistedthatpitchmustbeheldxedtostudytimbre.However,weproposethattheoppositemaybetrue,andwehaveshownthattimbredierencescanemergejustaseasilywhenpitchisincludedasasourceofvariation.Theprimarydimensionofperceptionindicatedinourstudyisbasedonabsolutepitch.Butbrightnessalsoemergedclearly,aswellasanewdimensionoftimbrenotmentionedinpriorliterature,thatisassociatedwithwordssuchassmoothness,positivity,andbite.Relativepitchemergedasadimension,aswell,whichisnotsurprisinggiventhattheperceptualqualityofharmony 91

PAGE 92

isbasedonintervals.Astudyindicatingtheexistenceofmultipledimensionsofpitchhasbeendoneprior,butneverhasadimensionofrelativepitchbeenobservedinthecontextoftimbre.Whileweuncoverednewdimensionsoftimbrethathavenotbeenobservedinpriorstudies,thereareadditionaldimensionsoftimbrethatwemayhavemissed.Someofthesecouldbeduetoasmalldatasetthatlackedenoughvariationinotherdimensionsforthemtoemergeclearly.Wealsoconsiderthepossibilitythatthereareinstantaneousdimensionsoftimbrethatcanonlybeobservedwithdynamicstimuli.Thispossibilityissomethingweplantoinvestigateinthefuture.Basedonourobservations,webelievewehaveprovidedfurtherindicationthatINDSCALisausefulmethodfordeterminingcoherentaxesofperception.However,wehavedonesobyexplicitlyexaminingtheroleofsubjectivityandobservingtheextenttowhichtheseaxesactuallyexistintheindividualspacesofthesubjects.ItisofgreatinteresttouswhethertheINDSCALspacecanbedeemedmorereliablethantheindividualspaces,byleveraginginformationfrommultiplesources,orwhethertheINDSCALspaceisamuddledversionoftheindividualspaces.TheclearutilityoftheINDSCALresultseemstoindicatetheformer,butwecannotdeterminethisconclusivelyfromthepresentstudy,andfutureworkmayberequiredtodoso.Thereareanumberofotherfuturedirectionssuggestedbyourworkthatwewouldliketoinvestigate.Ouroverarchinggoalofasystemfortimbremanipulationrequiredustorstproposenewconceptsandmethodsrelatedtothestudyoftimbreitself.Bothbasicresearchintothestudyoftimbreandthedevelopmentofasystemthatusesthisperceptualknowledgehavemanymorequestionstobeexplored.Tofacilitateresearchintoanumberofthese,webelievethatrstandforemostalargerstudysimilartotheoneproposedinthisworkbutincorporatingmoresoundsoughttobeundertaken.Increaseddiversityinthesoundswouldallowmoreperceptualdimensionstoemergeandallowustobemorecertainoftheformofthesedimensions. 92

PAGE 93

Forinstance,inharmonicitywasclearlypresentin3outofthe16soundsusedinthestudy.ThisfeaturehadamarkedinuenceontheabilityofINDSCALtoresolveadimensionofbrightness.Thisindicatestousthatinharmonicityisastrongperceptualfeature,butthatnotenoughvariationininharmonicitywasincludedforittoemergeasitsowndimension.Also,analyticalmethodsofinterpretingtheaxescouldbegreatlyimprovedwiththeuseofmoresounds.Wewereabletoexplainsomeaxesbycorrelatingthemwithacousticfeaturesthoughttoberelevanttoperception.Butwewerenotabletoautomaticallylearnthefeaturesthatbestexplainanaxisduetothelowernumberofsoundscomparedtothenumberofacousticfeatures.Discoveringtheacousticbasisforaperceptualaxiswouldmakeiteasiertosynthesizenewsoundsthatcorrespondtoaperceptualfeature.Butfutureworkinthisareaisonlypossiblewithmoredata.Perceptualinterpretationsofinstantaneoustimbredimensionswouldbeuseful,aswell.Wehaveusedsomeexploratorymethods,bolsteredbyknowledgeofthedomain,tointerpretaxesperceptually.Butwithmoredata,moresystematicinvestigationcouldbedone.INDSCALandrelatedmethodsofweightedcombinationarecurrentlythebestmethodsforlearningorientationsoftimbrespacewhereprincipalaxesemerge.ButitisnotcleariftheaxeslearnedbyINDSCALaretheonlysetofprincipalaxespossible.Incolorvision,atleasttwocompletelydierentcoherentsetsofaxesexist.Additionalperceptualexperimentscouldhelptoilluminatethis.Forinstance,aone-dimensionalMDSexperimentcouldbesetupwherespecicverbaldescriptionsofthesoundsareusedasthecriteriaforcomparison,ratherthananall-encompassingnotionofsimilarity.Thiswouldcreateperceptualaxesthatcouldthenbeanalyticallycorrelatedwithtimbrespaceembeddingstoseeiftheyarepresentandhowtheyareoriented.Conversely,anaxiscouldbetestedforitsperceptualcoherencebypresentingsoundsauditory,presentingtheirprojectionsvisually,andaskingsubjectstoidentifyapatternthatexplainsthevariationobservedintheprojectionsontotheaxis.Identicationofthepatterncouldbedonebyidentifyingwordstodescribeit,ashasbeen 93

PAGE 94

doneinthiswork,butitcouldalsobedoneimplicitlytosimplytestthecoherenceofanaxisbeforeattemptingtofullyinterpretit.Thiswouldentailleavingoutsomeofthesoundsfromtheprojectionandaskingsubjectstoplacethesesoundsontotheaxisbasedonthepatterntheyhaveidentied.Iftheaxiscorrespondstoaperceptuallycoherentdimension,apatternshouldemerge,andtheaccuracyinidentifyingthelocationsofthetestsoundsontheaxisshouldmeasureitscoherence.Thisisanotherexampleofhowmoreanalyticalanalysiscouldbedoneinthecontextofperception.Webelievewehavemadeapersuasivecasethatthehierarchyofinstantaneousanddynamictimbreproposedinthisworkisnecessaryandconsistentwithperception.Butmuchoftheformthatdynamictimbretakeswithinourmodelstillneedstobeinvestigated.Dynamictimbreisrepresentedinourmodelaspathsthroughinstantaneoustimbrespace.Butanenumerationofdynamictimbrepropertiesandtheirrepresentationsaspathsstillneedstobedone.Wehaveemphasizeddevelopingtheinstantaneousspatialrepresentationinthisworkbecauseitisanecessarypre-requisitetostudyingthedynamicrepresentation.Butassoonasasatisfyinginstantaneousspatialrepresentationisachieved,moreattentioncanbegiventodynamicconsiderations.Thehierarchybetweeninstantaneousanddynamictimbrehasimplicationsforthefuturedevelopmentoftheframeworkfortimbremanipulationingeneral.Developingthisframeworkwasidentiedasoneofourprimarymotivationsinundertakingthisresearch.Wehavelaidoutadetailedblueprinttoachievethisgoal,andtakenalargesteptowardimplementingthecenterpieceoftheframework.Inthefuture,wewouldliketoimplementtheentiresystem.Thisentailsdevelopingalow-levelmappingfrominstantaneoustimbrespacepointstosynthesisparametersandahigh-levelmappingfrompathsininstantaneoustimbrespacetosimpleinterfacesandabstractions.Developmentofthemappingtosynthesisparametersseemslikethenextlogicalstep.Todothiswell,morepointsintheinstantaneoustimbrespaceneedtobeassociatedwithsynthesisparameters.Currently,wehaveassociationsbetween16pointsonly,forthe16soundsinourexperiment.Thus 94

PAGE 95

weneedtoexpandthenumberofsoundsinourexperimentnotonlytoimproveourinstantaneoustimbrespacerepresentation,butalsotoimproveourabilitytomapfromthisspacetoalternaterepresentations.Itmaybethatasmorepointsareaddedtothespaceandwearemorecertainoftheaxes,itwillbecomenolongernecessarytoplacethepointsinthespaceviatheentireexperimentalprocedureoutlinedinthiswork.Instead,theremaybequickerwaystoaddpointstoapre-existingspace.Onceenoughpointsareinplace,anumberofmachinelearningmethodsareatourdisposaltomapbetweeninstantaneoustimbrespaceandsynthesisspace.Thenalpiecetoachievingasystemfordigitaltimbremanipulationincompositionisthentheinterface.Wehaveidentiedseveraltypesofinterfaceswethinkcouldbeuseful,andbelieveallofthemcouldbeimplemented.Futureworkinthisareaisneededtomapfromhigh-levelabstractionsandtimbreobjectstotheorthogonalperceptualrepresentation.Testsneedtobedonetoassesstheeectivenessofeachinterfacefordierentpurposes.Inall,webelievethissystemisnolongersofar-fetched,andlookforwardtocontinueworkinthisareatoachieveit,andtoincreaseourunderstandingoftimbreasaperceptualphenomenon. 95

PAGE 96

REFERENCES [1] ISO,\226:2003Normalequal-loudness-levelcontours",InternationalOrganizationforStandardization(2003). [2] C.Plack,A.Oxenham,R.Fay,andA.Popper,Pitch:NeuralCodingandPerception,volume24(SpringerNewYork)(2005). [3] R.Turner,\TheOhm-Seebeckdispute,HermannvonHelmholtz,andtheoriginsofphysiologicalacoustics",TheBritishJournalfortheHistoryofScience10,1{24(1977). [4] H.VonHelmholtz,OntheSensationsofToneasaPhysiologicalBasisfortheTheoryofMusic(Longmans,Green,andCompany)(1912). [5] S.McAdamsandA.Bregman,\Hearingmusicalstreams",ComputerMusicJournal26{60(1979). [6] H.Fletcher,\Loudness,pitchandthetimbreofmusicaltonesandtheirrelationtotheintensity,thefrequencyandtheovertonestructure",TheJournaloftheAcousticalSocietyofAmerica6,59{69(1934). [7] R.PlompandH.Steeneken,\Eectofphaseonthetimbreofcomplextones",TheJournaloftheAcousticalSocietyofAmerica46,409{421(1969). [8] R.N.Shepard,\Geometricalapproximationstothestructureofmusicalpitch.",PsychologicalReview89,305(1982). [9] H.Fletcher,E.D.Blackham,andR.Stratton,\Qualityofpianotones",TheJournaloftheAcousticalSocietyofAmerica34,749{761(1962). [10] D.LewisandM.Larsen,\Themeasurementofmaskedauditorythresholds.",JournalofExperimentalPsychology27,601(1940). [11] S.E.Olive,\Thepreservationoftimbre:Microphones,loudspeakers,soundsourcesandacousticalspaces",inAudioEngineeringSocietyConference:8thInternationalConference:TheSoundofAudio(AudioEngineeringSociety)(1990). [12] W.Torgerson,\Multidimensionalscaling:I.Theoryandmethod",Psychometrika17,401{419(1952). [13] R.Shepard,\Theanalysisofproximities:Multidimensionalscalingwithanunknowndistancefunction.I.",Psychometrika27,125{140(1962). [14] R.Shepard,\Theanalysisofproximities:Multidimensionalscalingwithanunknowndistancefunction.II.",Psychometrika27,219{246(1962). [15] J.Kruskal,\Multidimensionalscalingbyoptimizinggoodnessofttoanonmetrichypothesis",Psychometrika29,1{27(1964). 96

PAGE 97

[16] J.Kruskal,\Nonmetricmultidimensionalscaling:Anumericalmethod",Psychometrika29,115{129(1964). [17] K.SiedenburgandC.Reuter,\BeyondHelmholtz:150yearsoftimbralparadigms",in12thInternationalConferenceonMusicPerceptionandCognition(2012). [18] G.YoungandA.Householder,\Discussionofasetofpointsintermsoftheirmutualdistances",Psychometrika3,19{22(1938). [19] R.Plomp,\Timbreasamultidimensionalattributeofcomplextones",FrequencyAnalysisandPeriodicityDetectioninHearing397{414(1970). [20] C.Horan,\Multidimensionalscaling:Combiningobservationswhenindividualshavedierentperceptualstructures",Psychometrika34,139{165(1969). [21] J.CarrollandJ.Chang,\AnalysisofindividualdierencesinmultidimensionalscalingviaanN-waygeneralizationofEckart-Youngdecomposition",Psychometrika35,283{319(1970). [22] S.WinsbergandG.DeSoete,\AlatentclassapproachtottingtheweightedEuclideanmodel,CLASCAL",Psychometrika58,315{330(1993). [23] S.WinsbergandJ.D.Carroll,\Aquasi-nonmetricmethodformultidimensionalscalingviaanextendedEuclideanmodel",Psychometrika54,217{229(1989). [24] J.Tenenbaum,V.DeSilva,andJ.Langford,\Aglobalgeometricframeworkfornonlineardimensionalityreduction",Science290,2319{2323(2000). [25] L.WedinandG.Goude,\Dimensionanalysisoftheperceptionofinstrumentaltimbre",ScandinavianJournalofPsychology13,228{240(1972). [26] J.MillerandE.Carterette,\Perceptualspaceformusicalstructures",TheJournaloftheAcousticalSocietyofAmerica58,711(1975). [27] J.Grey,\Multidimensionalperceptualscalingofmusicaltimbres",TheJournaloftheAcousticalSocietyofAmerica61,1270{1277(1977). [28] S.McAdams,S.Winsberg,S.Donnadieu,G.Soete,andJ.Krimpho,\Perceptualscalingofsynthesizedmusicaltimbres:Commondimensions,specicities,andlatentsubjectclasses",PsychologicalResearch58,177{192(1995). [29] S.Lakatos,\Acommonperceptualspaceforharmonicandpercussivetimbres",Attention,Perception,&Psychophysics62,1426{1439(2000). [30] J.BurgoyneandS.McAdams,\Ameta-analysisoftimbreperceptionusingnonlinearextensionstoCLASCAL",ComputerMusicModelingandRetrieval.SenseofSounds181{202(2008). [31] C.L.Krumhansl,\Whyismusicaltimbresohardtounderstand",StructureandPerceptionofElectroacousticSoundandMusic9,43{53(1989). 97

PAGE 98

[32] D.Wessel,\Timbrespaceasamusicalcontrolstructure",ComputerMusicJournal45{52(1979). [33] R.EthingtonandB.Punch,\SeaWave:Asystemformusicaltimbredescription",ComputerMusicJournal30{39(1994). [34] A.Caclin,S.McAdams,B.Smith,andS.Winsberg,\Acousticcorrelatesoftimbrespacedimensions:Aconrmatorystudyusingsynthetictones",TheJournaloftheAcousticalSocietyofAmerica118,471(2005). [35] L.L.Thurstone,\Alawofcomparativejudgment.",PsychologicalReview34,273(1927). [36] W.ThompsonandR.Remage,\Rankingsfrompairedcomparisons",TheAnnalsofMathematicalStatistics739{747(1964). [37] T.P.Chartier,E.Kreutzer,A.N.Langville,andK.E.Pedings,\Sensitivityandstabilityofrankingvectors",SIAMJournalonScienticComputing33,1077{1102(2011). [38] W.N.Colley,\Colley'sbiasfreecollegefootballrankingmethod:TheColleymatrixexplained",PrincetonUniversity(2002). [39] I.Ali,W.D.Cook,andM.Kress,\Ontheminimumviolationsrankingofatournament",ManagementScience32,660{672(1986). [40] N.Alon,\Rankingtournaments",SIAMJournalonDiscreteMathematics20,137{142(2006). [41] C.A.Nicol,\Developmentandexplorationofatimbrespacerepresentationofaudio",Ph.D.thesis,UniversityofGlasgow(2005). [42] J.M.Chowning,\Thesynthesisofcomplexaudiospectrabymeansoffrequencymodulation",ComputerMusicJournal46{54(1977). [43] D.D.Nolte,\Thetangledtaleofphasespace",PhysicsToday63,32(2010). [44] L.H.SasakiandK.C.Smith,\Asimpledatareductionschemeforadditivesynthesis",ComputerMusicJournal4,22{24(1980). [45] K.W.Schindler,\Dynamictimbrecontrolforreal-timedigitalsynthesis",ComputerMusicJournal28{42(1984). [46] J.Burred,A.Robel,andT.Sikora,\Dynamicspectralenvelopemodelingfortimbreanalysisofmusicalinstrumentsounds",Audio,Speech,andLanguageProcessing,IEEETransactionson18,663{674(2010). [47] A.HuntandM.M.Wanderley,\Mappingperformerparameterstosynthesisengines",OrganisedSound7,97{108(2002). 98

PAGE 99

[48] I.Choi,R.Bargar,andC.Goudeseune,\Amanifoldinterfaceforahighdimensionalcontrolspace",inProceedingsofthe1995InternationalComputerMusicConference,385{392(1995). [49] D.WesselandM.Wright,\Problemsandprospectsforintimatemusicalcontrolofcomputers",ComputerMusicJournal26,11{22(2002). [50] C.Goudeseune,\Interpolatedmappingsformusicalinstruments",OrganisedSound7,85{96(2002). [51] L.Fritts,\UniversityofIowaelectronicmusicstudios",(2011),URL http://theremin.music.uiowa.edu/MIS.html 99

PAGE 100

BIOGRAPHICALSKETCH BrandonSmockwasbornin1983inWestPalmBeach,Florida.In2006,hegraduatedcumlaudefromtheUniversityofFloridawithaBachelorofSciencedegreeincomputerscience.In2008,heearnedhisMasterofScienceincomputerengineeringfromtheUniversityofFlorida.In2014,heearnedhisDoctorofPhilosophyincomputerengineeringfromtheUniversityofFlorida.Hisdoctoraldissertation,APerception-CentricFrameworkforDigitalTimbreManipulationinMusicComposition,wasdoneunderthesupervisionofDr.AnandRangarajan.Duringhistimeasadoctoralstudent,heworkedasaresearchassistantinmachinelearningintheComputationalScienceandIntelligencelaboratoryunderthesupervisionofDrs.PaulGaderandJosephWilson.HetwiceattendedtheIEEEGeoscienceandRemoteSensingSymposium,contributingproceedingspapersentitledReciprocalPointerChainsforIdentifyingLayerBoundariesinGround-PenetratingRadarDataandOptimalFusionofAlarmSetsfromMultipleDetectorsUsingDynamicProgramming.HealsoservedasateachingassistantfortheCISEdepartment'scourseondiscretemathematics.Hispost-graduationplansincludepursuingacareerasaresearchscientistorstartinghisowncompany.Heintendstoacceptapositionasapost-docintheComputationalScienceandIntelligencelabattheUniversityofFloridatocontinuehisresearchintheareaofoptimaldetectorfusion.Inthefuture,heenvisionscontinuinghisresearchinpsychoacousticsanddigitalsynthesisoftimbre,andexpandinghisresearchinareasrelatedtooptimalautomateddecision-making. 100