<%BANNER%>

The Transformational Complexity of Acyclic Networks of Neurons

Permanent Link: http://ufdc.ufl.edu/UFE0042775/00001

Material Information

Title: The Transformational Complexity of Acyclic Networks of Neurons
Physical Description: 1 online resource (95 p.)
Language: english
Creator: RAMASWAMY,VENKATAKRISHNAN
Publisher: University of Florida
Place of Publication: Gainesville, Fla.
Publication Date: 2011

Subjects

Subjects / Keywords: Computer and Information Science and Engineering -- Dissertations, Academic -- UF
Genre: Computer Engineering thesis, Ph.D.
bibliography   ( marcgt )
theses   ( marcgt )
government publication (state, provincial, terriorial, dependent)   ( marcgt )
born-digital   ( sobekcm )
Electronic Thesis or Dissertation

Notes

Abstract: Neurons are cells in the brain thought to be primarily responsible for information processing and cognition. Neurons communicate with each other via stereotypical electric pulses called action potentials or spikes. The exact computational mechanisms used by the brain for information processing are not yet well understood. The current work seeks to reduce this gap in our understanding, by theoretical analysis. Our thesis is that specific properties of neurons and the network's architecture constrain the computations it might be able to perform. This dissertation substantiates the thesis for the case of acyclic networks of neurons. In particular, the idea is to make precise the intuitive view that, acyclic networks are transformations that map input spike trains to output spike trains, to ask what transformations acyclic networks of specific architectures cannot accomplish. Our neurons are abstract mathematical objects that satisfy a small number of axioms, which correspond to basic properties of biological neurons. To begin with, we find that even a single neuron cannot be consistently viewed as a spike-train to spike-train transformation, in general (in a sense that we will make precise). However, under conditions consistent with spiking regimes observed in-vivo, we prove that the aforementioned notions of transformations are indeed well-defined and correspond to mapping finite-length input spike trains to finite-length output spike trains. Armed with this framework, we then ask what transformations acyclic networks of specific architectures cannot accomplish. We show such results for certain classes of architectures. While attempting to ask how increase in depth of the network constrains the transformations it can effect, we were surprised to discover that, with the current abstract model, every acyclic network has an equivalent acyclic network of depth two. What this suggests is that more axioms need to be added to the abstract model in order to obtain results in this direction. Finally, we study the space of spike-train to spike-train transformations and develop some more theoretical tools to facilitate this line of investigation.
General Note: In the series University of Florida Digital Collections.
General Note: Includes vita.
Bibliography: Includes bibliographical references.
Source of Description: Description based on online resource; title from PDF title page.
Source of Description: This bibliographic record is available under the Creative Commons CC0 public domain dedication. The University of Florida Libraries, as creator of this bibliographic record, has waived all rights to it worldwide under copyright law, including all related and neighboring rights, to the extent allowed by law.
Statement of Responsibility: by VENKATAKRISHNAN RAMASWAMY.
Thesis: Thesis (Ph.D.)--University of Florida, 2011.
Local: Adviser: Banerjee, Arunava.

Record Information

Source Institution: UFRGP
Rights Management: Applicable rights reserved.
Classification: lcc - LD1780 2011
System ID: UFE0042775:00001

Permanent Link: http://ufdc.ufl.edu/UFE0042775/00001

Material Information

Title: The Transformational Complexity of Acyclic Networks of Neurons
Physical Description: 1 online resource (95 p.)
Language: english
Creator: RAMASWAMY,VENKATAKRISHNAN
Publisher: University of Florida
Place of Publication: Gainesville, Fla.
Publication Date: 2011

Subjects

Subjects / Keywords: Computer and Information Science and Engineering -- Dissertations, Academic -- UF
Genre: Computer Engineering thesis, Ph.D.
bibliography   ( marcgt )
theses   ( marcgt )
government publication (state, provincial, terriorial, dependent)   ( marcgt )
born-digital   ( sobekcm )
Electronic Thesis or Dissertation

Notes

Abstract: Neurons are cells in the brain thought to be primarily responsible for information processing and cognition. Neurons communicate with each other via stereotypical electric pulses called action potentials or spikes. The exact computational mechanisms used by the brain for information processing are not yet well understood. The current work seeks to reduce this gap in our understanding, by theoretical analysis. Our thesis is that specific properties of neurons and the network's architecture constrain the computations it might be able to perform. This dissertation substantiates the thesis for the case of acyclic networks of neurons. In particular, the idea is to make precise the intuitive view that, acyclic networks are transformations that map input spike trains to output spike trains, to ask what transformations acyclic networks of specific architectures cannot accomplish. Our neurons are abstract mathematical objects that satisfy a small number of axioms, which correspond to basic properties of biological neurons. To begin with, we find that even a single neuron cannot be consistently viewed as a spike-train to spike-train transformation, in general (in a sense that we will make precise). However, under conditions consistent with spiking regimes observed in-vivo, we prove that the aforementioned notions of transformations are indeed well-defined and correspond to mapping finite-length input spike trains to finite-length output spike trains. Armed with this framework, we then ask what transformations acyclic networks of specific architectures cannot accomplish. We show such results for certain classes of architectures. While attempting to ask how increase in depth of the network constrains the transformations it can effect, we were surprised to discover that, with the current abstract model, every acyclic network has an equivalent acyclic network of depth two. What this suggests is that more axioms need to be added to the abstract model in order to obtain results in this direction. Finally, we study the space of spike-train to spike-train transformations and develop some more theoretical tools to facilitate this line of investigation.
General Note: In the series University of Florida Digital Collections.
General Note: Includes vita.
Bibliography: Includes bibliographical references.
Source of Description: Description based on online resource; title from PDF title page.
Source of Description: This bibliographic record is available under the Creative Commons CC0 public domain dedication. The University of Florida Libraries, as creator of this bibliographic record, has waived all rights to it worldwide under copyright law, including all related and neighboring rights, to the extent allowed by law.
Statement of Responsibility: by VENKATAKRISHNAN RAMASWAMY.
Thesis: Thesis (Ph.D.)--University of Florida, 2011.
Local: Adviser: Banerjee, Arunava.

Record Information

Source Institution: UFRGP
Rights Management: Applicable rights reserved.
Classification: lcc - LD1780 2011
System ID: UFE0042775:00001


This item has the following downloads:


Full Text

PAGE 1

THETRANSFORMATIONALCOMPLEXITYOFACYCLICNETWORKSOFNEURONSByVENKATAKRISHNANRAMASWAMYADISSERTATIONPRESENTEDTOTHEGRADUATESCHOOLOFTHEUNIVERSITYOFFLORIDAINPARTIALFULFILLMENTOFTHEREQUIREMENTSFORTHEDEGREEOFDOCTOROFPHILOSOPHYUNIVERSITYOFFLORIDA2011 1

PAGE 2

c2011VenkatakrishnanRamaswamy 2

PAGE 3

ToAppaandAmma 3

PAGE 4

ACKNOWLEDGMENTS IfeelacertaintingeofsadnessasIdrivethisnalnailinthedissertation,forIhavehadawonderfultimeingradschool.AlargemeasureofcreditforthisgoestomyadvisorArunavaBanerjee.Iowehimadebtofgratitudeforhiskindness,hiscollegialityandhisappetiteforthebigquestionsandambitiousattacksthatmadelifeinterestingandworthwhile.Idon'tknowhoworwhy,butIgotavirtualcarteblanche,rightfromthebeginning,topursuedirectionsthatinterestedme,evenduringtimeswhenprogresswasslowornonexistent.IusedtojokethathisoceistheoppositeofIHOPR 1 ;youcomehappyandleavehungry.Fewideassurvivedthecrucibleofhisintellect,butthosethatdidwereworthpursuing.Iamalsogratefulforhisamazingabilitytobeuplifting.Whenthechipsaredownandthingsseemhopeless,avisittohisocewassucienttogetyourmojoback.Healsohasacuriousmethodofaccomplishingthis;byrelatinganecdotesoftimeswhenhewasinevenmoredirestraits.IthankArunavaBanerjee,MingzouDing,AlirezaEntezari,JereyHo,MeeraSitharamandDonStehouwerforservingonmysupervisorycommittee.IamespeciallythankfultoJereyHoforbeingverygenerouswithhistimeacoupleofyearsagowhenIwaspursuinganattackthatdidnotbearfruit(yet).Thisworkwassupported,inpart,byagrant(IIS-0902230)fromtheNationalScienceFoundationtoArunavaBanerjee,whichIgratefullyacknowledge.IamalsothankfultotheCISEDepartmentforsupportingmethroughTeachingAssistantshipsovertheyears.Ialsobenetedfromanopportunitytoteachafewundergraduatecourses,forwhichIthankSteveThebaut.IamthankfultoJohnBowersandJoanCrismanfortheirkindnessandalltheirhelpwithmattersadministrative. 1 InternationalHouseofPancakes;arestaurantwiththeslogan,\Comehungry,leavehappy.R" 4

PAGE 5

IwishtothankMITPressforpermissiontoreproduceFigure 2-1 ,Figure 2-2 andFigure 2-3 thatappearin( Dayan&Abbott 2005 ).PamelaQuickmadetheprocessunbelievablyeasy.IwouldliketothankArunavaBanerjee,JereyHo,AnandRangarajan,MeeraSitharamandBabaVemurifortheirexcellentandmotivatingcoursesinComputerScienceandforbeingpatientwithmyquestions.IamalsothankfultoMikeJury,AlexandreTurullandPeterSinfortheirexcellentmathcourses.Iwishtothankmyhousematesovertheyears,AashishParikh,ShivkumarSwaminathan,VikasAggarwal,KarthikNagarajan,SwamiSankaran,GaneshViswanathan,SankethBhat,AvinashDeshpande,HarishBharadwaj,JigneshSoniandAbhijeetLawandefortoleratingmeandforinterestingconversations,foodandfun.SpecialthankstoVikasAggarwalforputtingupwithme(punintended)thelongest.Iamalsogratefultoallmyfriends.AjitRajwade,beingaco-suererrightthrough,helpedbluntsomeofthedespondencythatgoeswiththeterritory.MohsenAli,JasonChi,AmitDhurandhar,NekoFisher,KarthikGurumoorthy,RitwikKumar,ShahedNejhum,SubhajitSengupta,ManuSethiandNathanVanderkraatswerepeopleIbumpedintoeitherinthehallwayorinthelabonadailybasis.Thankyouforallthebanterandthebullsessions.And,abigthankyoutotherestofmyfriends,toonumeroustolisthere.Youknowwhoyouare!IwillalwaysremaingratefultoN.S.Narayanaswamy,foroeringmemyrsttasteofresearch.ThatyearworkingonSatisabilityalgorithmswithNariwasveryformativeandintroducedmetothesubtlejoysandunsubtledisappointmentsthatcomepackagedwithhardproblems.Onarecentmeeting,IrealizedhowmuchNarihasinuencedthewayIthinkaboutproblemsandthequestionsIask.Ihaveusedfreesoftwarealmostexclusivelyforanumberofyearsnow.Ithankthefreesoftware/opensourcecommunityforpreservingourfreedom. 5

PAGE 6

Butmostofall,Iamthankfultomyparents,C.V.RamaswamyandSaraswathiforallthatIam.Theirlove,aectionandpursuitofhighidealshavemadelifeworthliving.Iamgratefulforallthesacricestheyhavemadeovertheyearsforthesakeofmyeducationandwell-being.IamespeciallythankfultomyfatherforinstillinginmealoveforMathematicsfromanearlyageandmymotherforbeingmymoralcompass.IthankmysisterUshaforheraection,curiosityandteachingmeaboutcomputersandprogramming,backwhenIwasinhighschool.Paradoxically,ifnotforthatearlyexposure(andsubsequentsatiation),IwouldprobablystillbedoingcoreComputerScienceandnotthepresenttypeofwork.And,nally,Ithankyouthereader! 6

PAGE 7

TABLEOFCONTENTS page ACKNOWLEDGMENTS ................................. 4 LISTOFFIGURES .................................... 9 ABSTRACT ........................................ 11 CHAPTER 1INTRODUCTION .................................. 13 2BACKGROUND:NEURONALBIOPHYSICS ................... 17 2.1Introduction ................................... 17 2.2Neuron:BasicStructureandFunction .................... 17 2.3TheCellMembraneandtheMembranePotential .............. 20 2.4PassiveConductanceinDendrites-CableTheory .............. 22 2.5ActiveConductanceintheAxon-HodgkinHuxleyEquations ....... 24 2.6OtherVarietiesofNeuronalBehavior ..................... 26 3MODEL ........................................ 28 3.1NotationandPreliminaries ........................... 28 3.2AssumptionsUnderlyingtheModel ...................... 30 3.3FormalDenitionoftheModel ........................ 32 4ACYCLICNETWORKSASSPIKE-TRAINTOSPIKE-TRAINTRANSFORMATIONS ............................... 34 4.1OntheConsistencyofDeningInput/OutputTransformationsonNeurons 34 4.2GapCriteria ................................... 38 4.2.1GapLemma ............................... 38 4.2.2GapCriterionforaNeuron ....................... 41 4.2.3PracticalIssueswiththeGapCriteria ................. 45 4.3FlushCriterion ................................. 46 5TRANSFORMATIONALCOMPLEXITY:DEFINITIONSANDRESULTS .. 49 5.1MotivationandDenition ........................... 49 5.2Gap-FlushEquivalenceLemma ........................ 49 5.3ComplexityResultsfortheAbstractModel .................. 51 6COMPLEXITYRESULTSANDNETWORKDEPTH:ABARRIERATDEPTHTWO .................................... 60 6.1TechnicalStructureoftheProof ........................ 61 6.2Causal,Time-InvariantandResettableTransformations ........... 61 7

PAGE 8

6.3ConstructionofaDepthTwoAcyclicNetworkforeveryCausal,Time-InvariantandResettableTransformation ............. 65 6.4DirectionsforFurtherConstrainingtheAbstractModel ........... 76 6.5SomeAuxiliaryPropositions .......................... 76 7COMPLEXITYCLASSES,TRANSFORMATIONHIERARCHIESANDHIERARCHYCLASSES ............................... 79 7.1DenitionsandPreliminaries .......................... 80 7.2LowerBoundsontheHierarchyClassesofsomeArchitecturesinacertainTransformationHierarchy ........................... 85 8CONCLUSIONANDFUTUREWORK ...................... 90 8.1Contributions .................................. 90 8.2DirectionsforFutureWork ........................... 91 8.3Epilogue ..................................... 92 REFERENCES ....................................... 93 BIOGRAPHICALSKETCH ................................ 95 8

PAGE 9

LISTOFFIGURES Figure page 2-1Diagramsshowingmorphologyofthreeneurons. .................. 18 2-2Actionpotentialsandthesynapse. ......................... 19 2-3Aschematicdiagramofasectionofthelipidbilayerthatformsthecellmembraneoftheneuronwithtwoionchannelsembeddedinit. ......... 21 2-4Schematicdiagramoftheequivalentcircuitforapassivemembrane. ....... 23 4-1Counterexampletotheclaimthatthemembranepotentialofaneuroncanbemadeafunctionofaboundedwindowofinputspikesalone. ........... 35 4-2Counterexampletotheclaimthatthemembranepotentialofaneuroncanbemadeafunctionofinputspikesaloneintheinnitepast. ............. 36 4-3TheideabehindtheGapLemma. .......................... 38 4-4IllustrationaccompanyingtheGapLemma. .................... 39 4-5IllustrationdemonstratingthatforaninputspikecongurationthatsatisesaT-Gapcriterion,themembranepotentialatanypointintimeisdependentonatmostTsecondsofinputspikesinbeforeit. ................ 42 4-6SchematicdiagramillustratinghowtheGapcriterionworksforasimpletwo-neuronnetwork. ................................. 44 4-7IllustrationshowingthataninputspikecongurationsatisfyingaFlushCriterionalsosatisesaGapCriterion. ............................ 47 5-1Exampleofatransformationthatnoacyclicnetworkcaneect. ......... 52 5-2Atransformationthatnosingleneuroncaneect,thatanetworkwithtwoneuronscan. ...................................... 53 5-3Atwo-neuronnetworkeectingthetransformationprescribedinFigure 5-2 .. 55 5-4Atransformationthatnonetworkwithapath-disjointarchitecturecaneect. 56 5-5Atwo-neuronnetworkeectingthetransformationprescribedinFigure 5-4 .. 58 6-1Networkarchitectureform=2. ........................... 66 6-2ExampleillustratingtheoperationofI. ....................... 70 7-1VennDiagramillustratingthebasicnotionsofatransformationhierarchy,complexityclassandhierarchyclass. ........................ 80 7-2VennDiagramshowingcomplexityclasseswhenonesetofnetworksismore 9

PAGE 10

complexthananother. ................................ 81 7-3VennDiagramillustratingatransformationhierarchy. ............... 83 7-4VennDiagramdemonstratinghowupperboundsandlowerboundsonhierarchyclassesinatransformationhierarchycanbeusedtoestablishcomplexityresults. 84 7-5Diagramdepictingarchitectureofnetworksin2. ................. 87 7-6Diagramdepictingarchitectureofnetworksin3. ................. 88 10

PAGE 11

AbstractofDissertationPresentedtotheGraduateSchooloftheUniversityofFloridainPartialFulllmentoftheRequirementsfortheDegreeofDoctorofPhilosophyTHETRANSFORMATIONALCOMPLEXITYOFACYCLICNETWORKSOFNEURONSByVenkatakrishnanRamaswamyMay2011Chair:ArunavaBanerjeeMajor:ComputerEngineeringNeuronsarecellsinthebrainthoughttobeprimarilyresponsibleforinformationprocessingandcognition.Neuronscommunicatewitheachotherviastereotypicalelectricpulsescalledactionpotentialsorspikes.Theexactcomputationalmechanismsusedbythebrainforinformationprocessingarenotyetwellunderstood.Thecurrentworkseekstoreducethisgapinourunderstanding,bytheoreticalanalysis.Ourthesisisthatspecicpropertiesofneuronsandthenetwork'sarchitectureconstrainthecomputationsitmightbeabletoperform.Thisdissertationsubstantiatesthethesisforthecaseofacyclicnetworksofneurons.Inparticular,theideaistomakeprecisetheintuitiveviewthat,acyclicnetworksaretransformationsthatmapinputspiketrainstooutputspiketrains,toaskwhattransformationsacyclicnetworksofspecicarchitecturescannotaccomplish.Ourneuronsareabstractmathematicalobjectsthatsatisfyasmallnumberofaxioms,whichcorrespondtobasicpropertiesofbiologicalneurons.Tobeginwith,wendthatevenasingleneuroncannotbeconsistentlyviewedasaspike-traintospike-traintransformation,ingeneral(inasensethatwewillmakeprecise).However,underconditionsconsistentwithspikingregimesobservedin-vivo,weprovethattheaforementionednotionsoftransformationsareindeedwell-denedandcorrespondtomappingnite-lengthinputspiketrainstonite-lengthoutputspiketrains.Armedwiththisframework,wethenaskwhattransformationsacyclicnetworksofspecic 11

PAGE 12

architecturescannotaccomplish.Weshowsuchresultsforcertainclassesofarchitectures.Whileattemptingtoaskhowincreaseindepthofthenetworkconstrainsthetransformationsitcaneect,weweresurprisedtodiscoverthat,withthecurrentabstractmodel,everyacyclicnetworkhasanequivalentacyclicnetworkofdepthtwo.Whatthissuggestsisthatmoreaxiomsneedtobeaddedtotheabstractmodelinordertoobtainresultsinthisdirection.Finally,westudythespaceofspike-traintospike-traintransformationsanddevelopsomemoretheoreticaltoolstofacilitatethislineofinvestigation. 12

PAGE 13

CHAPTER1INTRODUCTIONNeuronsandtheirnetworks,fundamentally,aremachinesthattransformspiketrainsintospiketrains.Itisthesetransformationsthatformthebasisofinformationprocessing,indeedevencognition,inthebrain.Thisworkisbroadlymotivatedbythefollowingquestion:Whatconstraints,ifany,dolocalpropertiesofneuronsimposeonthecomputationalabilitiesofvariousnetworks?Atacoarselevel,wewouldliketosaysomethingaboutwhatspecicnetworks\cannot"do,byvirtueoftheirarchitecturealone.Nowtheanswermaydependonthespecicneuronmodelinuse.Tomitigatethispossibility,weinsteadassumethattheneuronisanabstractobjectthatsatisescertainaxioms,whichcorrespondwell-knownpropertiesofbiologicalneurons.Thisadmitsavarietyofmodelsaslongastheysatisfytheaxioms(includingonesthatmaybemorepowerfulthanwhatisbiologicallypossible).Wethenask,if(inspiteofthispossiblyadditionalpower),thereexistspecictransformationsthatcannotbeeectedbynetworksofcertainarchitecturesthatareequippedwiththeseabstractneurons.Inordertothenruleouttheprospectthatthetransformationinquestionissohardthatnonetworkcandoit,wealsoprovideanetwork(ofadierentarchitecture)comprisingsimpleneuronsthatcaninfacteectthistransformation.Itissuchconsiderationsthatmotivateourdenitionoftransformationalcomplexity.Thesenotionsaremadepreciseintheremainderofthedissertation.Sincethefunctionalroleofsingleneuronsandsmallnetworksinthebrainisnotyetwellunderstood,wedonotmakeanyassumptionsaboutparticular\tasks"thatthenetworkistryingtoperform;wearejustinterestedinphysicalspike-traintospike-traintransformations.Furthermore,sincethekindsofneuralcodeemployedisstillunclear,wemakenooverarchingassumptionsabouttheneuralcodeeither. 13

PAGE 14

Werestrictourstudytoacyclic 1 networksofneurons,i.e.networksthatdonothaveadirectedcycle.Whileevensingleneuronsarefunctionallyrecurrent 2 ,acyclicnetworksquicklysettledowntoquiescenceuponreceivingnoinput.Ontheotherhand,recurrentnetworkshavebeenknown( Banerjee 2006 )tohavecomplexdynamics,ingeneral,evenonreceivingnoinputforunboundedperiodsoftime.Also,wedonottreatsynapticplasticityeectshere.Inthepast,alargebodyofworkhasbeendoneassumingneuronsandtheirnetworksareentitiesforwhichonlytheinstantaneousrateofincomingspikesandoutgoingspikesisrelevant.Thetheoryofperceptrons( Rosenblatt 1988 ; Minsky&Papert 1969 )andmultilayerperceptrons( Rumelhartetal. 1986 ),forinstance,makessuchassumptions.However,thereiswidespreadevidence( Strehler&Lestienne 1986 ; Riekeetal. 1997 )thatprecisespiketimesplayaroleininformationprocessinginthebrain.Thereforerate-basedmodelsmaynotadequatelyexplainallaspectsofbrainfunction.Severalresearchershavestudiedrelatedquestions.In( Poirazietal. 2003 ),Poirazietal.,modelacompartmentalmodelofapyramidalneuronusingatwolayerneuralnetwork,assumingratecodes.Bohteetal.,( Bohteetal. 2002 )deriveasupervisedlearningruleforanetworkofspikingneurons,wheretheoutputisrestrictedtoasinglespikeinagivenperiodofobservation.GutigandSompolinsky( Gutig&Sompolinsky 2006 )describeamodeltolearnspiketimedecisions.Theyalsohaveataskwithtwooutcomes,whicharemappedtonotionsofapresenceorabsenceofspikes.Maass( Maass 1996 )investigatesthecomputationalpowerofnetworksofneuronsrelatingthemtowell-knownmodelsofcomputationsuchasTuringMachinesandRandomAccessMachines.FinallyBartlettand 1 whiletheterm\feedforward"networkiswidelyusedintheliteraturetorefertothistypeofnetwork,weprefertocalltheseacyclicnetworkstoemphasizethatthesenetworksarenotfeedforwardinthesystem-theoreticsense.2 owingtothemembranepotentialalsodependingonpastoutputspikestoaccountforeectsduringtheabsoluteandrelativerefractoryperiods 14

PAGE 15

Maass( Maass 1995 )analyzethediscriminativecapacityofapulse-codedneuronfromastatisticallearningtheoryperspective.Incontrasttoalltheaboveapproaches,weseektoinvestigatetherelativecomplexityofthephysicalspiketraintospiketraintransformationsthatareinstantiatedbysystemsofspikingneurons,withoutmakingoverarchingassumptionsabouttheunderlyingcomputationaldynamicsofthesystem.Ourresultsarethereforeatalevelmorefundamentalthanthecomputationalframeworkmostotherworkassumesandarethereforemorewidelyapplicable.Theremainderofthedissertationisorganizedasfollows.InChapter2,wereviewneurobiologyrelevanttounderstandingandevaluatingtheassumptionsofourmodel,whichiselaboratedinChapter3.Armedwiththismodel,inChapter4,weseektostudyacyclicnetworksasspike-traintospike-traintransformations.Tooursurprise,wendthatthenotionisnotalwayswell-dened.Weprovidecounterexamplesthatillustratesuchcases.Thispessimisticviewisthensurmountedbytherecognitionthatunderspikingregimesusuallyseeninthebrain,onecanconsistentlytreatacyclicnetworksasthesaidtransformations.Therestofthechapterisdevotedtorigorouslydeningsucientconditionswherethisisthecaseandprovingthatnotionsoftransformationsdoholdundertheseconditions.Wethenproceed,inChapter5,todenenotionsofcomplexitytoaddressquestionsofwhattransformationsspecicarchitecturescannotaccomplishthatnetworksofotherarchitecturescan.Wealsoproveanimportanttechnicallemmathatsignicantlyeasesthetaskofprovingcomplexityresults.Therestofthechapterthenestablishessomeresultsofthisformforsomespecicclassesofarchitectures.Whileattemptingtoaskhowthesetoftransformationseectedisconstrainedwhenthedepthofthenetworkisincreased,weweresurprisedtondthattheabstractmodelformulatedinChapter3doesnotsoconstrainitfornetworkswithdepthequaltotwoormore.Thus,Chapter6isdevotedtoprovingthatwiththecurrentabstractmodel,everyacyclicnetworkhasanequivalentacyclicnetworkofdepthtwo,thateectsexactlythesame 15

PAGE 16

transformation.InChapter7,wedevelopsomeadditionaltheoreticaltoolstoanswerquestionsofthistype.Specically,westudythespaceofallspike-traintospike-traintransformations,denenotionsofahierarchyofsetsinthatspaceandtrytorelatesetsinsuchhierarchiestothesetoftransformationsspannedbyspecicnetworkarchitectures.WeconcludeinChapter8withadiscussionanddescribesomedirectionsforfuturework. 16

PAGE 17

CHAPTER2BACKGROUND:NEURONALBIOPHYSICS 2.1IntroductionTheHumanBrainisoneofthemostremarkableentitiesintheUniverse.Thisapproximately3poundmassofsofttissuehasnotonlyensureditssurvival,butenabledittoruletheearth,exploretheuniverseandindeedattempttounderstanditself.Understandingthewaythebrainworksisoneofthelastgreatfrontiersinscience.Thetwopredominanttypeofcellsinthebrainareneuronsandglia.Gliaarebelievedprimarilytobesupportcellsthatprovidestructuralsupportandnourishmenttoneurons( Bearetal. 2007 ). 1 Neuronsarebelievedtobethecellsresponsibleforinformationprocessingandcognition.Thehumanbrainisestimatedtohavea100billionneuronsanda100trillionconnectionsbetweenthem.ThepurposeofthischapteristoprovidethereaderunfamiliarwithNeurobiology,thenecessarybackgroundtounderstandthecontextofthisworkandmorespecicallytoevaluatetheabstractmodeloftheneuronproposedinChapter 3 .Westartwithabriefdescriptionofneuronfunctionbeforedelvingintomoredetailedbiophysics.Theexpositionhereisbasedonmaterialfrom( Dayan&Abbott 2005 )and( Banerjee 2001b ). 2.2Neuron:BasicStructureandFunctionInthissection,webrieydescribehowaneuronworks.Thegoalistogivethereaderasenseofthebigpicture,tofacilitatethereadingofsubsequentsectionswherethebiophysicsisdescribedinmuchgreaterdetail.Aneuronisacellwhichreceivesinputfromcertainneuronsandcommunicatesitsoutputtocertainotherneurons.Figure 2-1 illustratesmorphologicalregionsinvarioustypesofneurons.Thesediagrams,circa1911,werehand-drawnbythepioneeringSpanishhistologistSantiagoRamonyCajal.Thethreeprimarymorphologicalregionsinaneuron 1 Interestingly,gliaoutnumberneuronsbyafactorof10. 17

PAGE 18

Figure2-1. Diagramsshowingmorphologyofthreeneurons.A)Acorticalpyramidalneuron.B)APurkinjecellofthecerebellum.C)Astellatecellofthecerebralcortex.[Reprintedwithpermissionfrom( Dayan&Abbott 2005 )(Page5,Figure1.1).] aredendrites,axonandthesoma.Thedendritesreceiveinputfromotherneurons,usuallythroughelaboratedendritictrees.Theneuroncommunicatesitsoutputtootherneuronsviatheaxonanditscollaterals.Axonsoftentendtocoverlargedistancesincludingentirelengthsofbodiesoftheorganism.Neuronscommunicatewithotherneuronsbyringstereotypicalpulsescalled\actionpotentials"or\spikes".Spikeshaveastereotypicalshape;itisonlytheirtimingthatisimportant.Neurons,thereforerepresentandcommunicateinformationviatemporalpatternsofspikes.Thegenerationofspikesismediatedbythepotentialdierencebetweentheinsideoftheneuronandoutside,calledthe\membranepotential".Themembranepotentialisaninternalvariable,inthesensethatitisnotexplicitlycommunicatedtootherneurons.Underrestingconditions(i.e.whentheneurondoesnotreceiveinputforsucientlylong),themembranepotentialismaintainedataround 18

PAGE 19

Figure2-2. Actionpotentialsandthesynapse.A)Anactionpotentialrecordedintracellularlyfromaculturedratneocorticalpyramidalcell.(RecordedbyL.RutherfordinthelabofG.Turrigiano).B)Diagramofasynapse.[Reprintedwithpermissionfrom( Dayan&Abbott 2005 )(Page6,Figure1.2).] )]TJ /F1 11.955 Tf 9.3 0 Td[(70mV.Thisiscalledthe\restingmembranepotential"orsimplytherestingpotentialandtheneuronissaidtobe\polarized"atthisstage.Inputspikesreceivedfromotherneuronsmodulatethispotential.Theymayeitherelevatethepotentialinwhichcasetheyarecalled\excitatory"ordepressthepotential,inwhichcasetheyarecalled\inhibitory".Elevationofthemembranepotentialabovetherestingpotentialiscalled\depolarization"anddepressingitbelowtherestingpotentialiscalled\hyperpolarization".Ifaneuronissucientlydepolarized,sothatitsmembranepotentialapproachesacertain\threshold"(frombelow),theneuronproducesaspikewhichtravelsdowntheaxonandiscommunicatedtoneuronsthatthepresentneuronisconnectedto.ThestereotypicalchangeinmembranepotentialthatanactionpotentialcausesisshowninFigure 2-2 (A).Oncetheneuronspikes,duetophysiologicalconstraints,itcannotspikeforacertainsmallperiodoftimecalledthe\absoluterefractoryperiod"whichistypicallyoftheorderofafewmilliseconds.Beyondthistime,foracertainintervaloftime,thethresholdtoreiselevated,i.e.morethanusualdepolarizationisrequiredtoelicitaspike.Thisperiodiscalledthe\relativerefractoryperiod"andisusuallytheorderoftensofmilliseconds.Actionpotentialsaretheonlytypeofmembrane 19

PAGE 20

uctuationthatarecommunicatedwithoutattenuationbyapositivefeedbackprocesswhichisaconsequenceofthetheuniqueelectrophysiologicalpropertiesofthecellmembraneontheaxon.Thisisexpoundedingreaterdetaillaterwhenwediscussthebiophysicsoftheactivemembrane.Axonscontain\synapses"atwhichdendritesofotherneuronsmakecontactwiththepresentneuron.Thoseneuronsaresaidtobe\postsynaptic"tothepresentneuron.AschematicdiagramofasynapseisprovidedinFigure 2-2 (B).Thegenerationofaspikeonthecurrentneuroniscommunicatedtopostsynapticneuronsviathesynapse.Asillustrated,thesynapseconsistsoftwoboutonsincloseproximitytoeachotherwithasmallgapinbetweencalledthe\synapticcleft".Theboutonsonthepresynapticsidehaveneurotransmittersstoredinsmallsacscalled\vesicles".Ongenerationofaspike,neurotransmittersarereleasedintothesynapticcleftwhichbindtoreceptorsonthepostsynapticsideandcausemodulationofthemembranepotentialofthepostsynapticneuron.Thisprocessiscalled\synaptictransmission".Armedwiththisbroadoutlineofhowneuronswork,wetreatitsbiophysicsinmoredetailintherestofthischapter.Moredetailedexpositionsofthesetopicsareavailablein( Tuckwell 1988 ; Koch 2004 ; Squireetal. 2003 ).ForarstlookatNeurobiology,thereaderisreferredto( Bearetal. 2007 ). 2.3TheCellMembraneandtheMembranePotentialThecellmembraneofaneuronisalipidbilayerwiththicknessapproximately3nm.Figure 2-3 providesanillustration.Thecellmembraneisimpermeabletomostions.Embeddedinthemembranearevariouswater-lledproteinmoleculesthatactas\ionchannels",whichallowspecicionstopassthroughthem.Theseionchannelsareusuallyveryselectiveinthattheyonlyallowcertaintypesofionstopassundercertainconditions.Theseconditionscouldincludespecicrangesofmembranepotentialwhenthechannelsareactivated,orpresenceofintracellularmessengerssuchasCa2+orextracellularentitiessuchasneurotransmitters.Atypicalneuronmayhavetensofdierenttypesofion 20

PAGE 21

Figure2-3. Aschematicdiagramofasectionofthelipidbilayerthatformsthecellmembraneoftheneuronwithtwoionchannelsembeddedinit.[Reprintedwithpermissionfrom( Dayan&Abbott 2005 )(Page154,Figure5.1).] channelsandoftheorderofmillionsofchannelsinit.Themembranealsocontains\ionpumps"thatconstantlyexpendenergytoexchangeionsinordertomaintainacertaindierenceinconcentrationbetweentheinsideandtheoutsideoftheneuron.Asmentionedearlier,therestingpotentialisabout)]TJ /F1 11.955 Tf 9.3 0 Td[(70mV,althoughthisnumbervariesdependingontheorganismandthetypeofneuronunderconsideration.Wenowexploretheunderlyingmechanismsthatareresponsibleformaintainingthispotentialdierence.TheprimarycauseisthedierentconcentrationofNa+andK+ionsacrossthemembrane.TheoutsideoftheneuronhasahigherconcentrationofNa+thantheinside.Likewise,theinsideoftheneuronhasahigherconcentrationofK+thantheoutside.ThisconcentrationgradientismaintainedbyNa+-K+ionpumpsthatsendout3Na+ionsforevery2K+takenin.Furthermore,themembraneissemi-permeabletotheseionsviaionchannels.Thisallowsfordiusionofionsacrossthemembrane,therateofdiusionbeingafunctionoftheconcentrationgradient,thepermeabilityoftheioninquestionandtheelectriceldacrossthemembrane.Therefore,K+ionstendtodiuseoutofthecellandNa+ionstendtodiuseintothecell.However,atrest,thepermeabilityofNa+ismuchlowerthanthatofK+andthereforemoreK+ionsareexpelledthanNa+ionsadmitted,whichleadstoanequilibriummembranepotentialinwhichtheinsideoftheneuronisnegativelychargedwithrespecttotheoutside.Also,Cl)]TJ /F1 11.955 Tf 10.98 -4.34 Td[(ionsarepresentin 21

PAGE 22

higherconcentrationoutside.Inthesecircumstances,theequilibriummembranepotentialcanbecomputedusingtheGoldman-Hodgkin-KatzEquation( Goldman 1943 ): Vm=RT FlnPK[K+]o+PNa[Na+]o+PCl[Cl)]TJ /F1 11.955 Tf 7.09 -4.34 Td[(]i PK[K+]i+PNa[Na+]i+PCl[Cl)]TJ /F1 11.955 Tf 7.08 -3.45 Td[(]o(2{1)whereRisthegasconstant,Ttheabsolutetemperature,FFaraday'sconstant,[]isand[]ostheconcentrationofthecorrespondingionsontheinsideandoutsiderespectivelyandPsthepermeabilityofthemembranetotheioninquestion. 2.4PassiveConductanceinDendrites-CableTheoryChangeinpotentialsinapostsynapticneuronduetoanincomingspikeareconductedpassivelyacrossthedendriteenroutetothesoma.ThisprocessismodeledusingCableTheory( Rall 1960 ).Thedendriteismodeledasacylinderofuniformradiuswithmembranepropertiessuchascapacitanceandresistivitybeinguniform.LetVmbethemembranepotentialalongtheaxisofthecable,r=ri+r0betheresistanceperunitlengthofcableandithecurrentowingacrosstheaxis.LetVmbethepotentialdroppedalongxofthecable.Then,byOhm'sLaw,wehave, Vm=)]TJ /F7 11.955 Tf 9.3 0 Td[(rix(2{2)Takingx!0,wehave, @Vm @x=)]TJ /F7 11.955 Tf 9.3 0 Td[(riandtherefore,@2Vm @x2=)]TJ /F7 11.955 Tf 9.3 0 Td[(r@i @x(2{3)Now,letimbethecurrentowingperpendiculartotheaxisofthecable,perunitlengthofcable.Letibethechangeincurrentforadistancexalongtheaxisofthecable.ByKircho'sFirstLaw,wehave i=)]TJ /F7 11.955 Tf 9.3 0 Td[(imx(2{4) 22

PAGE 23

Figure2-4. Schematicdiagramoftheequivalentcircuitforapassivemembrane.[Reprintedwithpermissionfrom( Banerjee 2001b )(Page28,Figure3.2).] Again,takingx!0,wehave, im=)]TJ /F7 11.955 Tf 11.82 8.09 Td[(@i @x(2{5)SubstitutingEquation 2{5 inEquation 2.4 ,wehave @2Vm @x2=rim(2{6)AsshowninFigure 2-4 ,aunitlengthofthemembraneismodeledasacircuitwhichhasacell(modelingtherestingpotential)andaresistanceconnectedinseries,whichareinturnconnectedinparalleltoacapacitor.Letcmbethemembranecapacitanceperunitlengthandrmisthemembraneresistanceperunitlength.WethereforehavetheEquation, im=cm@vm @t+vm rm(2{7)Letgm=1 rmrepresentthemembraneconductanceperunitlength.ThepreviousEquationthusbecomes, im=cm@vm @t+gmvm(2{8)ThePartialDierentialEquations 2{6 and 2{8 arethestartingpointofCableTheorywhichisusedtodescribepassiveconductanceofpostsynapticpotentialacrossdendrites. 23

PAGE 24

Solutionsoftheseequationshavebeenobtainedforvariousboundaryconditions,whichwewillnotdescribehere.Thereaderisreferredto( Dayan&Abbott 2005 )formore. 2.5ActiveConductanceintheAxon-HodgkinHuxleyEquationsTheaxonhasionchannelsthatallowforpassiveconductanceinthesamewayasthedendrites.However,inaddition,theyalsoallowforactiveconductance,whichiscrucialinproducingandpropagatingactionpotentials.Recallthatwhenthemembranepotentialiselevatedaboveacertainthreshold,theneuronproducesanactionpotentialwhichhasastereotypicalshapeasshowninFigure 2-2 (A).Themembranepotentialatwhichaspikeisinitiatedvariesbetween-55mVand-45mV,dependingonthetypeofneuroninquestion.Theactionpotentialcausesthemembranepotentialtodepolarizetoabout+55mVmomentarilyafterwhichitquicklyrepolarizestoavalueslightlybelowtherestingpotential;thelatterphenomenoniscalled\afterhyperpolarization(AHP)".Afterthis,themembraneslowlygetsbacktowardstherestingpotential.Asalludedtoearlier,unlikepassiveconduction,theactionpotentialistransmitteddowntheaxonwithoutanylossinamplitude.Themechanismofactionpotentialgenerationinvolvesasuddenchangeinthepermeabilityofthemembrane.Duetotheconcentrationgradient,thiscausesarapidchangeinthemembranepotential.Inwhatfollows,wediscusstheexactmechanism.Theaxonhillockandtheaxoncontainvoltage-gatedionchannels.Recallthationchannelsweresimplyproteinsembeddedinthecellmembrane.Voltage-gatedionchannelsareactivatedonlyatcertainvoltageranges.Thatis,atthosevoltageranges,theproteinsconstitutingthechannelundergoaconformationalchangethatmakesthempermeabletothecorrespondingion.Whenthemembranepotentialapproachesthethreshold,voltage-gatedNa+channelsstartgettingactivated.ThiscausesaninuxofNa+ionsfromoutsidethecell,whichfurtheractivatesmoreofthevoltage-gatedNa+channels.Thisprocess,calledNa+current\activation",momentarilyraisesthemembranepotentialtoabout+55mV.Atthispoint,thevoltage-gatedNa+channelsundergoanother 24

PAGE 25

conformationalchangethatmakesthemimpermeableagain.ThisiscalledNa+current\inactivation".Atthesametime,thereisafurtherincreaseintheK+permeabilityviavoltage-gatedK+channels,whichiscalled\delayedrectication".AfterhyperpolarizationistheconsequenceofNa+currentinactivationanddelayedrectication.Thisisfollowedby\deactivation"ofthevoltage-gatedK+channelsenablingthemembranepotentialtoreturntotherestingpotential.Thevoltage-gatedNa+channelsarealso\deinactivated"atthispoint,whichistosaythattheyundergoanotherconformationalchangetobringthembacktotheconformationtheywereinbeforethecurrentspikewasinitiated.Thismakesthemreadytoinitiatethenextspike.Recallthattheabsoluterefractoryperiodwasaperiodoftime(oftheorderofafewmilliseconds)immediatelyaftertheinitiationofaspikeduringwhichasubsequentspikecannotbeinitiated.ThereasonfortheabsoluterefractoryperiodisthedeactivationoftheNa+channels.Alsorecalltherelativerefractoryperiod,whichwasatimeperiodoftheorderoftensofmillisecondsafteraspikeduringwhichmorethanusualdepolarizationwasnecessarytoinitiateaspike.ThisisbecauseoftheoutwardK+currentwhichisaconsequenceofdelayedrectication.In1952,AlanHodgkinandAndrewHuxley,inNobelprizewinningwork,createdamathematicalmodelfortheaboveprocessthattdatafromthesquidgiantaxon.TheystartedwithamoregeneralversionofEquation 2{8 im=cm@vm @t+gNa(vm)]TJ /F7 11.955 Tf 11.96 0 Td[(VNa)+gK(vm)]TJ /F7 11.955 Tf 11.95 0 Td[(VK)+gL(vm)]TJ /F7 11.955 Tf 11.95 0 Td[(VL)(2{9)whereg'sarethemembraneconductancesforthecorrespondingionsandV'saretheirrespectiveequilibriumpotentials. 2 Theythendeterminedtherelationshipbetweenthevariousconductancesandthepotentialusingexperimentaldata,leadingtothefollowing 2 LstandsforleakandrepresentseectsofCl)]TJ /F1 11.955 Tf 10.98 -4.34 Td[(andotherionsputtogether. 25

PAGE 26

setofequations( Hodgkin&Huxley 1952b c d a ; Hodgkin&Katz 1952 ). im=cm@vm @t+gNam3h(vm)]TJ /F7 11.955 Tf 11.95 0 Td[(VNa)+gKn4(vm)]TJ /F7 11.955 Tf 11.95 0 Td[(VK)+gL(vm)]TJ /F7 11.955 Tf 11.95 0 Td[(VL);(2{10) dm dt=m(1)]TJ /F7 11.955 Tf 11.96 0 Td[(m))]TJ /F7 11.955 Tf 11.95 0 Td[(mm;(2{11) dh dt=h(1)]TJ /F7 11.955 Tf 11.96 0 Td[(h))]TJ /F7 11.955 Tf 11.95 0 Td[(hh;(2{12) dn dt=n(1)]TJ /F7 11.955 Tf 11.96 0 Td[(n))]TJ /F7 11.955 Tf 11.96 0 Td[(nn;(2{13)ThesearecalledtheHodgkin-HuxleyEquations.m;m,h;h,nandnarefunctionsofvmasbelow: m=0:1(vm+25) evm+25 10)]TJ /F1 11.955 Tf 11.96 0 Td[(1;m=4evm 18;(2{14) h=0:07evm 20;h=1 evm+30 10+1;(2{15) n=0:01(vm+10) evm+10 10)]TJ /F1 11.955 Tf 11.95 0 Td[(1;n=0:125evm 80:(2{16) 2.6OtherVarietiesofNeuronalBehaviorThebiophysicsdescribedthusfardoesnotaccountforseveralclassesofneuronalbehaviorthathavebeenseen.Primarilythisisbecausesuchbehaviorismediatedbyavarietyofionchannelsthatwerenottreatedpreviously.Inthissection,webrieydescribesomeofthesemodesandtheirelectrophyiologicalunderpinnings.\Spikefrequencyadaptation"isaslightmodicationofthebasicbehaviorwhereinoncontinuousdepolarizationoveraperiodoftime,theneuron'soutputspikerategoesdownor\adapts"tothesustainedinput.Corticalandhippocampalpyramidalneuronsexhibitthisbehavior( Shepherd 2004 ).ActivationofaslowK+hyperpolarizationcurrentthataddsupovertimehasbeendemonstratedtocausespikefrequencyadaptation.Someneuronsareinherently\bursty".Thatistheytendtoemitburstsofspikesratherthansinglespikes.ExamplesincludeThalamicrelayneurons,somepyramidalneurons,stellatecellsandneuronsintheinferiorolive( Shepherd 2004 ).Thesearecaused 26

PAGE 27

byspecializedCa2+currentswithslowkinetics,whichkeepthemembranedepolarizedforlongenough,sothattherecanbeasequenceofNa+-K+spikes.Inhibitoryinterneuronsinthecortex,thalamusandhippocampusreshort-durationspikesoflessthanamilliseconddurationatfrequenciesgreaterthan300Hz.Then,thereare\pacemakerneurons"thatspontaneouslygeneratelowfrequencyspiketrains(1-10Hz).Thesearefoundinthelocuscoeruleus,dorsalraphe,medialhabenulanucleiandcerebellarPurkinjecells( Shepherd 2004 ).ThesearecausedbypersistentNa+channels,wheresteadyinuxofNa+ionsdepolarizesthecelltoabovethreshold,consequentlytriggeringbaselineactivity. 27

PAGE 28

CHAPTER3MODELInthischapter,wedescribetheabstractmodelusedintherestofthework.Firstwedenesomenotationthatwillbeusedfrequently.Next,weinformallydiscusstheassumptionsunderlyingourmodel.Wethenproceedtoformallydenetheneuronasanabstractmathematicalobjectanddeneotherstructuresthatwewillndourselvesusing. 3.1NotationandPreliminariesAn\actionpotential"or\spike"isastereotypicaleventcharacterizedbythetimeinstantatwhichitisinitiated 1 ,whichisreferredtoasits\spiketime".Spiketimesarerepresentedrelativetothepresentbyrealnumbers,withpositivevaluesdenotingpastspiketimesandnegativevaluesdenotingfuturespiketimes 2 .A\spiketrain"~x=hx1;x2;:::;xk;:::iisanincreasingsequenceofspiketimes,witheverypairofspiketimesbeingmorethanapart,where>0is(called)theabsoluterefractoryperiod 3 andxiisthespiketimeofspikei.An\emptyspiketrain",denotedby~,isonewhichhasnospikes.LetSdenotethesetofallspiketrains.A\time-boundedspiketrain"(with\bound"[a;b])isonewhereallspiketimeslieintheboundedinterval[a;b],fora;b2R.Notethat,owingtotheabsoluterefractoryperiod,atime-boundedspiketrainisalsoanite-lengthsequence.LetS[a;b]denotethesetofalltime-boundedspiketrainswithbound[a;b].Aspiketrainissaidtohavea\gap"intheinterval[c;d],ifithasnospikesinthattimeinterval.Furthermore,thisgapissaidtobeof\length"d)]TJ /F7 11.955 Tf 12.67 0 Td[(c.Twospike 1 Theexactlocationofspikeinitiationisnotimportantaslongasthespiketimeismeasuredasthetimeatwhichthespikepassesthroughacertainxedpointintheaxon,uponinitiation.2 Notethatthisconventionimpliesthatlargeraspike'sspiketime,themoreancientthespikeis.3 Weassumeasinglexedabsoluterefractoryperiodforallneurons,forconvenience,althoughourresultswouldbenodierentifdierentneuronshaddierentabsoluterefractoryperiods. 28

PAGE 29

trainsaresaidtobe\identical"intheinterval[a;b],iftheyhaveexactlythesamespiketimesinthattimeinterval.A\spikeconguration"=h~x1;:::;~xmiisatupleofspiketrains.The\order"ofaspikecongurationisthenumberofspiketrainsinit.Forexample,aboveisaspikecongurationoforderm.A\time-boundedspikeconguration"(with\bound"[a;b])isonewhichhaseachofitsspiketrainstobetime-bounded(with\bound"[a;b]).Twospikecongurationsaresaidtobe\identical"intheinterval[a;b],iftheyhavethesameorderandtheircorrespondingspiketrainsareidenticalinthesametimeinterval.Aspikecongurationissaidhavea\gap"intheinterval[c;d],ifeachofitsspiketrainshasagapintheinterval[c;d].Next,wedenesomeoperatorstotime-shiftspiketrains/congurations,createbounded-timespiketrains/congurationsfromspiketrains/congurationsandtoassemble/disassemblespiketrainsinto/fromspikecongurations.Let~x=hx1;x2;:::;xk;:::ibeaspiketrainand=h~x1;:::;~xmibeaspikeconguration.Wedenesomeoperationsontheseobjects.The\shiftoperatorforspiketrains"isdenedast(~x)=hx1)]TJ /F7 11.955 Tf 13.62 0 Td[(t;x2)]TJ /F7 11.955 Tf 13.63 0 Td[(t;:::;xk)]TJ /F7 11.955 Tf 13.63 0 Td[(t;:::i.The\shiftoperatorforspikecongurations"isdenedast()=ht(~x1);:::;t(~xm)i.The\truncationoperatorforspiketrains"isdenedasfollows:[a;b](~x)isthetime-boundedspiketrainwithbound[a;b]thatisidenticalto~xintheinterval[a;b].(a;b](~x)and[a;b)(~x)aredenedlikewise.[a;1)(~x)isthespiketrainthatisidenticalto~xintheinterval[a;1)andhasnospikesintheinterval(;a).Similarly,(;b](~x)isthespiketrainthatisidenticalto~xintheinterval(;b]andhasnospikesintheinterval(b;1).The\truncationoperatorforspikecongurations"isdenedas[a;b]()=h[a;b](~x1);:::;[a;b](~xm)i;(a;b]()and[a;b)()aredenedlikewise.Similarly,[a;1)()=h[a;1)(~x1);:::;[a;1)(~xm)iand(;b]()=h(;b](~x1);:::;(;b](~xm)i.Furthermore,t()isshorthandfor[t;t]().The\projectionoperatorforspikecongurations"isdenedasi()=~xi,for 29

PAGE 30

1im.Let~y1;~y2;:::;~ynbespiketrains.The\joinoperatorforspiketrains"isdenedas~y1t~y2t:::t~yn=h~y1;~y2;:::;~yni.Forbrevity,~y1t~y2t:::t~ynisalsowrittenasnFi=1~yi. 3.2AssumptionsUnderlyingtheModelOurneuronsareabstractmathematicalobjectsthatareconstrainedtosatisfyasmallsetofaxioms.First,weinformallydescribetheassumptionsthatunderlietheaxioms.Notablecaseswheretheassumptionsdonotholdarealsopointedout.Thisisfollowedbyformaldenitions.Theapproachtakenherecloselyfollowstheonein( Banerjee 2001a ).Thefollowingareourassumptions: 1. Weassumethattheneuronisadevicethatreceivesinputfromotherneuronsexclusivelybyspikeswhicharereceivedviachemicalsynapses. 4 2. Theneuronisanite-precisiondevicewithfadingmemory.Hence,theunderlyingpotentialfunctioncanbedeterminedfromaboundedpast.Thatis,weassumethat,foreachneuron,thereexistrealnumbersand,sothatthecurrentmembranepotentialoftheneuroncanbedeterminedasafunctionoftheinputspikesreceivedinthepastsecondsandthespikesproducedbytheneuroninthepastseconds. 5 3. Weassumethatthemembranepotentialoftheneuroncanbewrittendownasareal-valued,everywhereboundedfunctionoftheformP(;~x0),where~x0isatime-boundedspiketrain,withbound[0;]and=h~x1;:::;~xmiisatime-boundedspikecongurationwithbound[0;].Informally,~xi,1im,isthesequenceofspikesaerentinsynapseiinthepastsecondsand~x0isthesequenceofspikeseerentfromthecurrentneuroninthepastseconds.ThefunctionP()characterizestheentirespatiotemporalresponseoftheneurontospikesincluding 4 Inthiswork,wedonottreatelectricalsynapsesorephapticinteractions( Shepherd 2004 ).5 correspondstothenotionof\relativerefractoryperiod." 30

PAGE 31

synapticstrengths,theirlocationondendrites,andtheirmodulationofeachother'seectsatthesoma,spike-propagationdelays,andthepostspikehyperpolarization. 4. Withoutlossofgenerality 6 ,weassumetherestingmembranepotential 7 tobe0.Let>0bethethresholdmembranepotential 8 5. TheneuronoutputsaspikewheneverP()=.Additionally,whenanewoutputspikeisproduced,weassumethattherstderivativeissucientlyhigh,soastokeepthemembranepotentialfromgoingabovethethreshold. 9 Thereforethemembranepotentialfunctionintheabstractmodeltakesvaluesthatareatmostthatofthethreshold. 6. Owingtotheabsoluterefractoryperiod>0,notwoinputoroutputspikescanoccurcloserthan.Thatis,suppose~x0=hx10;x20;:::;xk0i,wherex10<.ThenP(;~x0)<,forall\legal". 7. Pastoutputspikeshaveaninhibitoryeect,inthefollowingsense 10 :P(;~x0)P(;~),forall\legal"and~x0. 8. Finally,onreceivingnoinputspikesinthepastsecondsandnooutputspikesinthepastseconds,theneuronsettlestoitsrestingpotential.Thatis,P(h~;~;:::;~i;~)=0. 6 Thereisnolossofgeneralitysincethemembranepotentialcanberescaledtotthisconstraint,withoutaectingthespiketimes.7 saymeasuredatthesoma.8 Observethatthemodelallowsforvariablethresholds,aslongasthethresholditselfisafunctionofspikesaerentinthepastsecondsandspikeseerentfromthepresentneuroninthepastseconds.9 Notethatthisassumptionismadewithoutlossofgeneralityandhelpseasetheformulationoftheabstractmodel.10 Thisisnotablyviolatedincaseofneuronsthathaveapost-inhibitoryrebound. 31

PAGE 32

An\acyclicnetworkofneurons",informally,isaDirectedAcyclicGraphwhereeachvertexcorrespondstoaninstantiationoftheneuronmodel,withsomeverticesdesignatedinputvertices(whichareplaceholdersforinputspiketrains),andoneneurondesignatedtheoutputneuron.The\depth"ofanacyclicnetworkisthelengthofthelongestpathfromaninputvertextotheoutputvertex. 3.3FormalDenitionoftheModelNext,wepresentaformaldenitionofaneuronasanabstractmathematicalobject.Somemoredenitionsfollow. Denition1(Neuron). A\neuron"Nisa7-tupleh;;;;;m;P:Sm[0;]S[0;]![;]i,where;;;2R+with,2R)]TJ /F1 11.955 Tf 10.99 -4.34 Td[(andm2Z+.Furthermore, 1. If~x0=hx10;x20;:::;xk0iwithx10<,thenP(;~x0)<,forall2Sm[0;]andforall~x02S[0;]. 2. P(;~x0)P(;~),forall2Sm[0;]andforall~x02S[0;]. 3. P(h~;~;:::;~i;~)=0.Aneuronissaidto\generateaspike"wheneverP()=. Denition2(NetworkArchitecture). ANetworkArchitectureAisa5-tuplehGhV;Ei;I;o;#:f1;2;:::;jIjg!I;f#v:f1;:::;indegree(v)g!Vjv2(V)]TJ /F7 11.955 Tf 13.56 0 Td[(I)giwhereGhV;Eiisadirectedgraph 11 withinputverticesIV,an 12 outputvertexo2V,abijection#:f1;2;:::;jIjg!Ithatnumberstheinputverticesandabijection#v:f1;:::;indegree(v)g!V,foreachvertexv2(V)]TJ /F7 11.955 Tf 12.46 0 Td[(I),thatnumberstheverticesincidentonthein-edgesofv.Furthermore,foreachv2I,wehaveindegree(v)=0andoutdegree(o)=0. 11 Adirectededgerepresentsthefactthatthereis\atleast"onesynapsefromtheneuronrepresentedbytheoutgoingvertextotheneuronrepresentedbytheincomingvertex.TheP()functionoftheeerentneurongetsonlyonespiketrainfromtheaerentneuronandencodestheresponseofallthesynapsesfromtheaerentneuron.12 Inthiswork,weonlystudynetworkswithasingleoutputneuron. 32

PAGE 33

Denition3(AcyclicNetworkArchitecture). ANetworkArchitectureAhGhV;Ei;I;o;#:f1;2;:::;jIjg!I;f#v:f1;:::;indegree(v)g!Vjv2(V)]TJ /F7 11.955 Tf 12.26 0 Td[(I)giiscalledan\AcyclicNetworkArchitecture"ifGhV;Eiisanacyclicgraph. Denition4(Network). A\Network"Nisa3-tuplehA;N;L:(V)]TJ /F7 11.955 Tf 12.5 0 Td[(I)!NiwhereAhGhV;Ei;I;o;#:f1;2;:::;jIjg!I;f#v:f1;:::;indegree(v)g!Vjv2(V)]TJ /F7 11.955 Tf 12.33 0 Td[(I)giisanetworkarchitecture,NisasetofneuronsandL:(V)]TJ /F7 11.955 Tf 12.36 0 Td[(I)!Nisabijectionthatmapsnon-inputverticesofGtoneuronsinNsuchthatthefollowingistrue:Foreveryv2(V)]TJ /F7 11.955 Tf 12.76 0 Td[(I),thathasL(v)=Nh;;;;;m;P:Sm[0;]S[0;]![;]i,wehaveindegree(v)=m.Further,L(o)iscalledthe\outputneuron". Denition5(AcyclicNetwork). An\AcyclicNetwork"NhA;N;L:(V)]TJ /F7 11.955 Tf 12.02 0 Td[(I)!NiisonewhosenetworkarchitectureAisacyclic. Denition6. InanacyclicnetworkNhA;N;L:(V)]TJ /F7 11.955 Tf 12.13 0 Td[(I)!Ni,the\depth"ofaneuronN2NisthelengthofthelongestpathfromaninputvertextoL)]TJ /F6 7.97 Tf 6.59 0 Td[(1(N). Denition7. The\depth"ofanacyclicnetworkNhA;N;L:(V)]TJ /F7 11.955 Tf 12.08 0 Td[(I)!Niisthedepthofitsoutputneuron. 33

PAGE 34

CHAPTER4ACYCLICNETWORKSASSPIKE-TRAINTOSPIKE-TRAINTRANSFORMATIONSAsstatedintheintroduction,wewishtolookatacylicnetworksofneuronsastransformationsthatmapinputspiketrainstooutputspiketrains.Therefore,werstneedtodeneinwhatsense,ifatall,thesenetworksconstitutethesaidtransformations.Webeginbydemonstratingthatnotionsofinput/outputtransformationsarenotalwayswell-dened,evenforasingleneuron.Wethenshow,notwithstandingthisdismayingprognosis,thatsuchtransformationsareindeedwell-denedunderspikingregimesnormallyencounteredinthebrain.Inparticular,wederiveprecisesucientconditionsunderwhichthesetransformationsarewell-denedwhichcorrespondtomappingnite-lengthinputspiketrainstonite-lengthoutputspiketrains.Wethenobservethatwhilethissucientconditionisbiologicallywell-founded,itismathematicallyunwieldy.Wethereforeproceedtoestablishmoremathematicallytractablesucientconditions,albeitseeminglyatthecostofsomebiologicalrealism.However,inthenextchapter,wedemonstratethatinfactnobiologicalwell-foundednessislost.Weshowthisbyestablishinganequivalencetheorembetweenthesetwoconditions,insofarasprovingresultsinvolvingcomplexity 1 isconcerned. 4.1OntheConsistencyofDeningInput/OutputTransformationsonNeuronsLetusrstconsiderthesimplestacyclicnetwork,namelythesingleneuron.Giventhatourabstractneuronmodeldoesnotexplicitlyprescribeanoutputspiketrainforagiveninputspiketrain,weneedtoaskwhatitmeansforaneuronto\produce"anoutputspiketrain,whensuppliedwithaninputspiketrain.Recall,fromtheprevioussection,thatthemembranepotentialoftheneurondependsnotonlyontheinputspikesreceivedinthepastseconds,italsodependsontheoutputspikesproducedbyitin 1 whosedenitionwealsointroduceinthenextchapter. 34

PAGE 35

Figure4-1. Thisexampledescribesasingleneuronwhichhasjustoneaerentsynapse.Untiltimet0inthepast,itreceivednoinput.Afterthistime,itsinputwasspikesthatarrivedevery)]TJ /F7 11.955 Tf 11.96 0 Td[(=2seconds,where>0.Aninputspikealone(iftherewerenooutputspikesinthepastseconds)cancausethisneurontoproduceanoutputspike.However,iftherewereanoutputspikewithinthepastseconds,theAHPduetothatspikeissucienttobringthepotentialbelowthreshold,sothattheneurondoesnotspikecurrently.Wethereforeobservethatiftherstspikeisabsent,thentheoutputspiketraindrasticallychanges.Notethatthischangeoccursnomatterhowoftentheshadedsegmentinthemiddleisrepeated,i.e.itdoesnotdependonhowlongagotherstspikeoccured. thepastseconds.Therefore,knowledgeofjustinputspikesinthepastsecondsdoesnotuniquelydeterminethecurrentmembranepotential(andthereforetheoutputspiketrainproducedfromit).Itmightbetemptingtothensomehowusethefactthatthepastoutputspikesarethemselvesafunctionofinputandoutputreceivedinthemoredistantpast,andattempttomakethemembranepotentialafunctionofaboundedalbeitlarger\window"ofinputspikesalone.TheexampleinFigure 4-1 showsthatthisdoesnotwork.Inparticular,thecurrentmembranepotentialoftheneuronmaydependonthepositionoftheinputspikethathasoccuredarbitrarilylongtimeagointhepast(ifwe 35

PAGE 36

Figure4-2. TheexamplehereisverysimilartotheoneinFigure 4-1 ,exceptthat,insteadoftherebeingnoinputspikesbeforet0,wehaveanunboundedinputspikeconguration,withthesameperiodicinputspikesoccuringsincetheinnitepast.Observethatbothoutputspiketrainsareconsistentwiththisinput,foreacht2R. attempttocharacterizethemembranepotentialasafunctionofinputspikesalone).Oneisthenforcedtoaskifgiventheinnitehistoryofinputspikesreceivedbytheneuron,themembranepotentialisthenuniquelydetermined.Beforewecananswerthisquestion,weneedtorigorouslydenewhenwecanconsistentlyassociatean(unbounded)outputspiketrainwithan(unbounded)inputspikeconguration,forasingleneuron. Denition8. Anoutputspiketrain~xoissaidtobe\consistent"withaninputspikeconguration,withrespecttoaneuronNh;;;;;m;P:Sm[0;]S[0;]![;]i,ifthefollowingholds.Foreveryt2R,t2~x0ifandonlyifP([0;](t());[0;](t(~x0))=.Thequestionweposednowbecomesequivalenttothefollowing:Forevery(unbounded)inputspikeconguration,doesthereexistexactlyone(unbounded) 36

PAGE 37

outputspiketrain~xo,sothat~xoisconsistentwithforagivenneuronN?Interestingly,wendthattheanswerisstillinthenegative.TheexampleinFigure 4-2 describesaneuronandaninnitelylonginputspiketrain,whichhastwoconsistentoutputspiketrains.Itcouldbearguedthattheinputspiketraincannotpossiblybeinnitelylong,sinceeveryneuronbeginsexistenceatacertainpointintime.However,thisbegsthequestionofiftheneuronwasattherestingpotentialwhentherstinputspikesarrived 2 .Anassumptiontothiseectwouldbesignicant,particularlyifthecurrentmembranepotentialdependedonit.ItiseasytoconstructanexamplealongthelinesoftheexampledescribedinFigure 4-1 ,wherethecurrentmembranepotentialisdierentdependingonwhetherthisassumptionismadeornot.Assuminginnitelylonginputspikecongurations,ontheotherhand,obviatestheneedtomakeanysuchassumption.Weretainthisviewpointfortherestofthedissertationwiththeunderstandingthatthealternativeviewpointdiscussedatthebeginningofthisparagraphcanalsobeexpoundedalongsimilarlines.Nevertheless,theunderlyingdicultyindeningevensingleneuronsasspiketraintospiketraintransformations,withbothviewpointsdiscussedaboveisdependence,ingeneral,ofcurrentmembranepotentialto\initialstate".However,thisstillleavesopenthepossibilityofconsideringjustasubsetofinput/outputspiketrains,whichhavethepropertyofthecurrentmembranepotentialbeingindependentoftheinputspiketrainbeyondacertaintimeinthepast.Suchasubsetwouldexcludetheexamplesdiscussedinthissection.Thecaveat,ofcourse,isthatevenifsuchasubsetexists,unlessitisalsobiologicallywell-motivated,claimsofsubsequentresultshavingbiologicalrelevancearequestionable. 2 Notethatouraxiomaticdenitionofaneurondoesnotaddressthisquestion. 37

PAGE 38

Figure4-3. TheideabehindtheGapLemma. Inwhatfollows,wecomeupwithabiologically-realisticconditionthatimpliesindependenceasalludedtoabove;roughlyspeaking,theconditionisthatifaneuronhashadarecentgapinitsoutputspiketrainequaltoatleasttwiceitsrelativerefractoryperiod,thenitscurrentmembranepotentialisindependentoftheinputbeyondtherelativelyrecentpast.Weshowthatthisleadstothenotionofspike-traintospike-traintransformationstobewelldenedforacyclicnetworks. 4.2GapCriteriaInthissection,wecomeupwithabiologicallywell-motivatedconditionthatguaranteesindependenceofcurrentmembranepotentialfrominputspikesbeyondtherecentpast.Thisconditionisusedinconstructingacriterionforsingleneuronswhichwhensatised,guaranteesauniqueconsistent(unbounded)outputspiketrain.Next,similarcriteriaaredenedforacyclicnetworks.Foraneuron,thewayinputspikesthathappenedsucientlyearlieraectcurrentmembranepotentialisviaacausalsequenceofoutputspikes,causalinthesensethateachofthemhadaneectonthemembranepotentialwhilethesubsequentoneinthesequencewasbeingproduced.TheconditionintheGapLemmabasicallyseekstobreakthiscausalchain. 4.2.1GapLemmaToseethemainideathatleadstothecondition,seeFigure 4-3 .Suppose,thespikesintheshadedregion(whichisanintervaloflength)occurredattheexactsameposition 38

PAGE 39

Figure4-4. IllustrationaccompanyingtheGapLemma. forinputspikecongurationswithspikesoccurringatarbitrarypositionsolderthantimeinstantt0,thenthecurrentmembranepotentialdependsonatmosttheinputspikesintheinterval[t;t0].Letussupposewechoosetheshadedregiontohavenospikes,i.etobea\gap"oflength.Now,justrequiringthisdoesnotimplythatthisgapispreservedwheninputspikessucientlyoldarechangedasisclearfromFigure 4-1 whereaninputspikearbitrarilyoldwhenremoved,causesaspike,whereagapoflengthatleastexistedpreviously.Weneedsomeadditionalconstraints.Howaboutifthegapwerelonger?Surprisingly,wendthatagapof2insteadsuces,asthenextlemmashows.2isalsothesmallestgapforwhichthisworks.Thedetailsareinthefollowinglemma.Figure 4-4 accompaniesthelemma. Lemma1(GapLemma). ConsideraneuronNh;;;;;m;P:Sm[0;]S[0;]![;]i,aspikecongurationofordermandaspiketrain~x0whichhasagapintheinterval[t;t+2]sothat~x0isconsistentwith,withrespecttoN.Letbeanarbitraryspikecongurationthatisidenticaltointheinterval[t;t++].Then,everyoutputspiketrainconsistentwith,withrespecttoN,hasagapintheinterval[t;t+].Furthermore,2isthesmallestgaplengthin~xo,forwhichthisistrue. Proof. Since,ineach~xoconsistentwith,theinterval[t+2;t+3]of~xoandthe[t++;t++2]ofarearbitrary,thesequenceofspikespresentintheinterval[t+;t+2]of~xocouldbearbitrary.However,andareidenticalin[t;t++].Thus,itfollowsfromAxiom2inthedenitionofaneuronthatforeveryt02[t;t+], 39

PAGE 40

P([0;](t0());[0;](t0(~x0)))isatmostthevalueofP([0;](t0());[0;](t0(~x0))),because[0;](t0(~x0))is~.SinceP([0;](t0());[0;](t0(~x0)))islessthanforeveryt02[t;t+],P([0;](t0());[0;](t0(~x0)))islessthaninthesameinterval,aswell.Therefore,~xohasnooutputspikesin[t;t+].That2isthesmallestpossiblegaplengthin~xoforthistohold,followsfromtheexampleinFigure 4-1 ,wheretheconclusiondidnothold,when~xohadgapsoflength2)]TJ /F7 11.955 Tf 11.95 0 Td[(,forarbitrary>0. Corollary1. ConsideraneuronNh;;;;;m;P:Sm[0;]S[0;]![;]i,aspikecongurationofordermandaspiketrain~x0whichhasagapintheinterval[t;t+2]sothat~x0isconsistentwith,withrespecttoN.Then 1. Every~x0consistentwith,withrespecttoN,hasagapintheinterval[t;t+]. 2. Every~x0consistentwith,withrespecttoN,isidenticalto~x0intheinterval(;t+]. 3. Foreveryt0morerecentthan(t+),themembranepotentialatt0,ispreciselyafunctionofspikesin[t0;t++](). Proof. ( 1 )isimmediatefromthelemma,whenweset=.For( 2 ),theproofisbystronginductiononthenumberofspikessincet.Let~x0beanarbitraryspiketrainthatisconsistentwith,withrespecttoN.Noticethatfrom( 1 )wehavethatevery~x0isidenticalto~x0in[t;t+].Thebasecaseistoshowthatboth~x0and~x0havetheirrstspikeaftertatthesametime.Assume,withoutlossofgenerality,thattherstspikeof~x0att1
PAGE 41

identicalto[0;](tk+1(~x0))fromtheinductionhypothesissince(t+))]TJ /F7 11.955 Tf 12.2 0 Td[(tk+1.Thus,P([0;](tk+1());[0;](tk+1(~x0)))=P([0;](tk+1());[0;](tk+1(~x0)))andtherefore~x0alsohasits(k+1)thspikeattk+1.Thiscompletestheproofof( 2 ).( 3 )followsfromtheGapLemmaand( 2 ). TheupshotoftheGapLemmaanditscorollaryisthatwheneveraneurongoesthroughaperiodoftimeequaltotwiceitsrelativerefractoryperiodwhereithasproducednooutputspikes,itsmembranepotentialfromthenonbecomesindependentofinputspikesthatareolderthan+secondsbeforetheendofthegap.Largegapsintheoutputspiketrainsofneuronsseemtobeextensivelyprevalentinthehumanbrain.Inpartsofthebrainwheretheneuronsspikepersistently,suchasinthefrontalcortex,thespikerateisverylow(0.1Hz-10Hz)( Shepherd 2004 ).Incontrast,thetypicalspikerateofretinalganglioncellscanbeveryhighbuttheactivityisgenerallyinterspersedwithlargegapsduringwhichnospikesareemitted( Nirenbergetal. 2001 ). 4.2.2GapCriterionforaNeuronTheseobservationsmotivateourdenitionofacriterionforinputspikecongurationseerentonsingleneurons.Thecriteriondictatesthattherebeintermittentgapsoflengthatleasttwicetherelativerefractoryperiodinanoutputspiketrainconsistentwiththespikeconguration. Denition9(GapCriterionforasingleneuron). ForT2R+,aspikecongurationissaidtosatisfya\T-GapCriterion"foraneuronNh;;;;;m;P:Sm[0;]S[0;]![;]iifthefollowingistrue:isofordermandthereexistsaspiketrain~x0withatleastonegapoflength2ineveryintervaloftimeoflengthT)]TJ /F1 11.955 Tf 12.51 0 Td[(+2,sothat~x0isconsistentwithwithrespecttoN.Suchinputspikecongurationsalsohaveexactlyoneconsistentoutputspiketrain.Weprovethisinthenextproposition. 41

PAGE 42

Figure4-5. IllustrationdemonstratingthatforaninputspikecongurationthatsatisesaT-Gapcriterion,themembranepotentialatanypointintimeisdependentonatmostTsecondsofinputspikesinbeforeit. Proposition1. LetbeaspikecongurationthatsatisesaT-GapcriterionforaneuronNh;;;;;m;P:Sm[0;]S[0;]![;]i.Then,thereisexactlyonespiketrain~x0,suchthat~x0isconsistentwith,withrespecttoN. Proof. SincesatisesaT-Gapcriterion,thereexistsaspiketrain~x0withatleastonegapoflength2ineveryintervaloftimeoflengthT)]TJ /F1 11.955 Tf 12.51 0 Td[(+2,sothat~x0isconsistentwithwithrespecttoN.Forthesakeofcontradiction,assumethatthereexistsanotherspiketrain~x00,notidenticalto~x0,whichisconsistentwith,withrespecttoN.Lett0bethetimeatwhichonespiketrainhasaspikebutanotherdoesn't.Lett>t0besuchthat~x0hasagapintheinterval[t;t+].ByCorollary 1 totheGapLemma,itfollowsthat~x00isidenticalto~x0startingfromtimeinstantt+.Thiscontradictsthehypothesisthat~x00isdierentfrom~x0att0. ForaninputspikecongurationthatsatisesaT-Gapcriterion,themembranepotentialatanypointintimeisdependentonatmostTsecondsofinputspikesinbeforeit.ThiscanbeseenfromFigure 4-5 ,whichillustratesasectionoftheinputspikecongurationandtheoutputspiketrain.BecauseoftheT-Gapcriterionthedistancebetweenanytwogapsoflength2ontheoutputspiketrainisatmostT)]TJ /F1 11.955 Tf 12.23 0 Td[()]TJ /F1 11.955 Tf 12.22 0 Td[(2.Uptotheearlierhalfofa2gap(whoselatestpointisdenotedbyt0)isdependentoninput 42

PAGE 43

correspondingtotheprevious2gap.Themembranepotentialatt0dependsoninputspikesintheintervaloflengthT,asdepicted.WithinputsthatsatisfytheT-GapCriterion,hereiswhatweneedtodotophysicallydeterminethecurrentmembranepotential,eveniftheneuronhasbeenreceivinginputsincetheinnitepast:Startotheneuronfromanarbitrarystate,anddriveitwithinputthattheneuronreceivedinthepastTseconds.TheGapLemmaguaranteesthatthemembranepotentialweseenowwillbeidenticaltotheactualmembranepotential.TheGapCriterionwehavedenedforsingleneuronscanbenaturallyextendedtoacyclicnetworks.Thecriterionissimplythattheinputspikecongurationtothenetworkissuchthateveryneuron'sinputobeysaaGapcriterionforsingleneurons. Denition10(GapCriterionforanacyclicnetwork). ConsideranacyclicnetworkNhA;N;L:(V)]TJ /F7 11.955 Tf 12.54 0 Td[(I)!Ni.LetdbethedepthofN.LetNiN,for1id,bethesetofneuronsinNofdepthi.ForT2R+,aspikecongurationissaidtosatisfya\T-GapCriterion"fortheacyclicnetworkNifisoforderjIjandthefollowingaretrue: 1. ForeachN2N1,indegree(v)Fi=1#)]TJ /F4 5.978 Tf 5.75 0 Td[(1(#v(i))()satisesa(T d)-GapCriterionforN,wherev=L)]TJ /F6 7.97 Tf 6.59 0 Td[(1(N). 2. Fori=2;:::;drespectively,foreachN2Ni,theinputtoNsatisesa(T d)-GapCriterionforN.Aswiththecriterionforthesingleneuron,themembranepotentialoftheoutputneuronatanypointisdependentonatmostTsecondsofpastinput,iftheinputspikecongurationtotheacyclicnetworksatisesaT-Gapcriterion.ThesituationisillustratedinFigure 4-6 .Additionally,theoutputspiketrainisunique. Lemma2. ConsideranacyclicnetworkNhA;N;L:(V)]TJ /F7 11.955 Tf 10.94 0 Td[(I)!Ni.LetsatisfyaT-GapcriterionforN.ThenNproducesauniqueoutputspiketrainwhenitreceivesasinput.Furthermore,themembranepotentialoftheoutputneuronatanytimeinstantdependsonatmostthepastTsecondsofinputin. 43

PAGE 44

Figure4-6. SchematicdiagramillustratinghowtheGapcriterionworksforthesimpletwo-neuronnetworkontheleft.Themembranepotentialoftheoutputneuronattdependsoninputreceivedfromthe\intermediate"neuron,asdepictedinthedarkly-shadedregion,owingtotheGapLemma.Theoutputoftheintermediateneuroninthedarkly-shadedregion,inturn,dependsoninputitreceivedinthelightly-shadedregion.Thus,transitively,membranepotentialoftheoutputneuronattisdependentatmostoninputreceivedbythenetworkinthelightly-shadedregion. Proof. Weprovethattheoutputofthenetworkisuniquebystronginductionondepth.Asbefore,letNiN,for1id,bethesetofneuronsinNofdepthi.EachneuronN2N1receivesallinputsfromspiketrainsin.Since,NsatisesaGapcriterionwiththoseinputspiketrains,itsoutputisunique.Theinductionhypothesisthenisthatforallik
PAGE 45

penultimatelayerneuroninthepast(2T d)seconds.Similarargumentscanbeputforthuntil,foreachpath,onereachesaneuron,allofwhoseinputsdonotcomefromotherneurons.Sincethelongestsuchpathisoflengthd,itisstraightforwardtoverifythatthemembranepotentialoftheoutputneurondependsonatmostTsecondsofpastinput. Wearethusatajuncturewherequestionsweinitiallyaskedcanevenbeposedinacoherentmannerthatisalsobiologicallywell-motivated.Beforeweproceed,weintroducesomemorenotation.GivenanacyclicnetworkN,letGTNbethesetofallinputspikecongurationsthatsatisfyaT-GapCriterionforN.LetGN=ST2R+GTN.Therefore,everyacyclicnetworkNinducesatransformationTN:GN!SthatmapseachelementofGNtoauniqueoutputspiketraininS.SupposeG0GN.Then,letTNjG0:G0!SbethemapdenedasTNjG0()=TN(),forall2G0. 4.2.3PracticalIssueswiththeGapCriteriaTheGapCriteriaareverygeneralandbiologicallyrealistic.However,givenaneuronoranacyclicnetwork,theredoesnotappeartobeaneasywaytocharacterizealltheinputspikecongurationsthatsatisfyacertainGapCriterionforit.Whetheraspikecongurationsatisesacertaingapcriterionseemstobeintimatelydependentontheparticularformofthepotentialfunctionofeachneuron.Foranacyclicnetwork,thisisevenmorecomplex,sinceintermediateneuronsmustsatisfyGapCriteria,withtheinputstheygetbeingoutputsofotherneurons.EachneuronoracyclicnetworkcouldpotentiallyinduceadierentsetofspikecongurationsthatsatisfyaT-GapCriterionforit.Thisappearstomaketheproblemofcomparingthetransformationsperformedbytwodierentneurons/acyclicnetworksdicult,becauseofthedicultyinndingspikecongurationsthatsatisfyGapCriteriaforbothofthem.Thisbringsupthequestionoftheexistenceofanothercriterionaccordingtowhichthesetofspikecongurationsiseasiertocharacterizeandiscommonacrossdierentnetworks.Next,weproposeonesuchcriterionandweshowthatitinducesspike 45

PAGE 46

congurationswhichareasubsetofthoseinducedbytheGapcriteriaforallacyclicnetworks.Briey,theseareinputspikecongurationswhich,beforeacertaintimeinstantinthepast,havehadnospikes.Weemphasizethatthisisapurelytheoreticalconstructmadeformathematicalexpedience;thatis,nobiologicalrelevanceisclaimed.Thespikecongurationssatisfyingthesaidcriterion,whichwecalltheFlushcriterion,allowustosidestepthedicultissuesmentionedinthepreviousparagraph.Signicantly,inasubsequentsection,afterhavingdenednotionsofcomplexity,weshowthatthereisnolossbyrestrictingourselvestotheFlushcriterion.Thatis,notonlyisacomplexityresultprovedusingtheFlushcriterionapplicablewiththeGapcriterion,everycomplexityresulttrueforGapcriterioncanbeprovedbyusingtheFlushcriterionexclusively. 4.3FlushCriterionTheideaoftheFlushCriterionistoforcetheneurontoproducenooutputspikesforsucientlylongsoastoguaranteethataGapcriterionisbeingsatised.Thisisdonebyhavingasucientlylongintervalwithnoinputspikes.This\ushes"theneuronbybringingittotherestingpotentialandkeepsitthereforanappropriatelylongtime.Inanacyclicnetwork,theushispropogatedsothatallneuronshavehadasucientlylonggapintheiroutputspiketrains.NotethattheFlushCriterionisnotdenedwithreferencetoanyacyclicnetwork.Weformalizethisnotionbelow. Denition11(FlushCriterion). AspikecongurationissaidtosatisfyaT-FlushCriterion,ifallitsspikeslieintheinterval[0;T],i.e.ithasnospikesbeforetimeinstantTandaftertimeinstant0.First,weshowthataninputspikecongurationtoaneuronthatsatisesaFlushcriterionalsosatisesaGapcriterion.Figure 4-7 accompaniesthefollowinglemma. Lemma3. AninputspikecongurationforaneuronthatsatisesaT-FlushCriterionalsosatisesa(T+2+2)-GapCriterionforthatneuron. Proof. Theneurononbeingdrivenbycannothaveoutputspikesoutsidetheinterval[)]TJ /F1 11.955 Tf 9.3 0 Td[(;T].ThiseasilyfollowsfromAxiom2and3oftheneuronbecausetheneurondoes 46

PAGE 47

Figure4-7. IllustrationshowingthataninputspikecongurationsatisfyingaFlushCriterionalsosatisesaGapCriterion. nothaveinputspikesbeforetimeinstantTandintheinterval[)]TJ /F1 11.955 Tf 9.29 0 Td[(;0]andonwards.Nowtoseethatsatisesa(T+2+2)-GapCriterion,recallthatwithaT0-GapCriterion,distancebetweenanytwogapsoflength2ontheoutputspiketrainisatmostT0)]TJ /F1 11.955 Tf 10.22 0 Td[()]TJ /F1 11.955 Tf 10.21 0 Td[(2.With,weobservethatthedistancebetweenanytwo2gapsontheoutputspiketrainisatmostT+.Thus,T0)]TJ /F1 11.955 Tf 12.13 0 Td[()]TJ /F1 11.955 Tf 12.13 0 Td[(2=T+,whichgivesusT0=T+2+2.Theresultfollows. Next,itisshownthataninputspikecongurationtoanacyclicnetworksatisfyingaFlushcriterionalsosatisesaGapcriterionforthatnetwork. Lemma4. AninputspikecongurationforanacyclicnetworkthatsatisesaT-FlushCriterionalsosatisesa(dT+d(d+1)+2d)-GapCriterionforthatnetwork,where,areupperboundsonthesameparameterstakenoveralltheneuronsinthenetworkanddisthedepthofthenetwork. Proof. Followingtheproofofthepreviouslemma,weknowthatneuronsthatreceivealltheirinputsfromhavenooutputspikesoutsidetheinterval[)]TJ /F1 11.955 Tf 9.29 0 Td[(;T].Similarly,neuronsthathavedepth2withrespecttotheinputverticesofthenetworkhavenooutputspikesoutside[)]TJ /F1 11.955 Tf 9.3 0 Td[(2;T].Likewise,theoutputneuron,whichhasdepthd,hasnooutputspikesoutside[)]TJ /F7 11.955 Tf 9.3 0 Td[(d;T].Itfollowsthattheoutputneuronobeysa(T+(d+1)+2)-GapCriterion.Also,everyotherneuronfollowsthiscriterionbecausethedistancebetweenthe 47

PAGE 48

2outputgapsforeveryneuronisatmostthatoftheoutputneuron,sincetheirdepthisboundedfromabovebythedepthoftheoutputneuron.Thus,fromthedenitionoftheGapcriterionforacyclicnetworks,wehavethatsatisesa(dT+d(d+1)+2d)-GapCriterionforthecurrentnetwork. Weintroducesomemorenotation.LetthesetofspikecongurationscontainingexactlymspiketrainsthatsatisfytheT-FlushcriterionbeFTm.LetFm=ST2R+FTm.WhatwehaveestablishedinthissectionisthatFmGN,whereNhasexactlyminputvertices. 48

PAGE 49

CHAPTER5TRANSFORMATIONALCOMPLEXITY:DEFINITIONSANDRESULTS 5.1MotivationandDenitionInthissection,wedenenotionsofrelativecomplexityofsetsofacyclicnetworksofneurons,withrespecttotransformationseectedbythem.Forbrevity,werefertothesenotionsas\TransformationalComplexity".Whatwewouldliketocapturewiththedenitionisthefollowing:Giventwoclassesofnetworkswiththesecondclasssubsumingtherst,wewishtoaskiftherearetransformationsinthesecondclassthatcannotbeperformedbynetworksintherstclass.Thatis,dotheextranetworksinthesecondclassmakeitricherintermsoftransformationalpower?Theclassescouldcorrespondtonetworkarchitectures,althoughforthepurposeofthedenition,thereisnoreasontorequirethistobethecase.Whilecomparingasetofnetworks,werestrictourselvestoinputsforwhichallthenetworkssatisfyacertainGapCriterion(though,notnecessarilyforthesameT),sothatthenotionofatransformationiswell-denedontheinputset,forallnetworksunderconsideration. Denition12. Let1and2betwosetsofacyclicnetworks,eachnetworkbeingoforderm,with12.DeneG12=TN21[2GN.2issaidtobe\morecomplexthan"1,if9N022suchthat8N21;TN0jG126=TNjG12.NotethatG12isalwaysnonemptybecauseFmG12.Henceforth,anyresultthatestablishesarelationshipoftheformdenedaboveiscalleda\complexityresult". 5.2Gap-FlushEquivalenceLemmaNext,isthemainlemmaofthissection.Weshowthatifoneclassofnetworksismorecomplexthananother,theninputsthatsatisfytheFlushCriterionaresucienttoprovethis.Thatis,toprovethistypeofcomplexityresult,onecanworkexclusivelywithFlushinputswithoutlosinganygenerality.ThisisnotobviousbecauseFlushinputsformasubsetofthemorebiologically-realisticGapinputs. 49

PAGE 50

Lemma5(EquivalenceofFlushandGapCriteriawithrespecttoComplexity). Let1and2betwosetsofacyclicnetworks,eachnetworkbeingoforderm,with12.Then,2ismorecomplexthan1ifandonlyif9N022suchthat8N21;TN0jFm6=TNjFm. Proof. Weprovetheeasydirectionrst.If9N022suchthat8N21;TN0jFm6=TNjFm,thenitfollowsthatTN0jG126=TNjG12becauseFmGN.Fortheotherdirection,let9N022suchthat8N21;TN0jG126=TNjG12.WeconstructF0Fm,sothatTN0jF06=TNjF0.ThisimmediatelyimpliesTN0jFm6=TNjFm.ConsiderarbitraryN21.Fromthehypothesiswehave,TN0jG126=TNjG12.Therefore92G12suchthatTN0jG12()6=TNjG12().Additionally,thereexistT1;T22R+,sothatsatisesaT1-GapCriterionforNandaT2-GapCriterionforN0.LetT=max(T1;T2).LetTN0jG12()=~x00andTNjG12()=~x0.Let~F=St2R[0;2T](t()).Notethateachelementof~Fsatisesa2T-FlushCriterion.Theclaim,then,isthatTN0j~F6=TNj~F.Wehave[0;T](TN0([0;2T](t())))=[0;T](t(~x00))and[0;T](TN([0;2T](t())))=[0;T](t(~x0)).ThisfollowsfromthefactthatsatisestheT-GapCriterionwithbothNandN0andthereforewhenNandN0aredrivenbyanysegmentofoflength2T,theoutputproducedinthelastTsecondsofthatintervalagreeswith~x0and~x00respectively.Therefore,if~x06=~x00,itisclearthatthereexistsat,sothatTN0([0;2T](t()))6=TN([0;2T](t())).F0isobtainedbytakingtheunionofsuch~FforeveryN21.Theresultfollows. Assuredbythistheoreticalguaranteethatthereisnolossofgeneralitybydoingso,wewillhenceforthonlyworkwithinputssatisfyingtheFlushCriterionwhilefacedwiththetaskofprovingcomplexityresults.Thisbuysusagreatdealofmathematicalexpedienceatnocost.Fromnowon,unlessotherwisementioned,whenwesaya\transformationoforderm",wemeanamapT:Fm!S. 50

PAGE 51

5.3ComplexityResultsfortheAbstractModelInthissection,weprovesomecomplexityresultsfornetworkswhoseneuronsobeytheabstractmodeloftheneurondescribedinChapter 3 .Beforeweproceed,acoupleofremarksareinorder.First,theperceptivereadermightwonderifallowinginputssatisfyingtheT-Gap/FlushCriterionforarbitrarilylargeTiscontrived,particularlysincephysiologicallyitishardforaneurontosustainhighringratesforarbitrarilylargeperiodsoftime.Moreover,isthisunrealisticassumption,beingusedtopushthroughresults,therebymakingthemonlyoftheoreticalinterest?Suppose,instead,weputaglobalboundonT,thenanytransformationcanbeeectedevenbyasingleneuron,bymakingitssuitably(andunrealistically)high.Thenonemightproposeputtingaglobalboundon,whichbringsustothedicultquestionofhavingtoquantifytherelationshipbetweenthetwobounds,withcomplexityresultsdependingontheanswertothisquestion.Wesidesteptheseissueswithourcurrentformulation,butwheneverwepresentatransformationthatcannotbeeectedbyanetworkarchitecturewedothefollowing.Givenanupperboundonthevaluesofandoftheneurons,wewritedownTasafunctionofthesebounds,sothatforallT0T,inputsintheprescribedtransformationsatisfyingaT0-Flushcriterioncannot(provably)bemappedtotheprescribedoutputbyanynetworkwiththesamearchitecturewhichalsoobeystheboundsonand.Thesecondremarkconcernsourabstractmodeloftheneuron.Themodeladmitsawidevarietyofmembranepotentialfunctions,whicharemeanttosubsumetheclassofbiologically-realisticpotentialfunctions.Therefore,whenoneprovesanegativeresult 1 abouttheabstractmodel,thenegativeresultappliestoallofthebiologically-realistic 1 Byanegativeresult,wemeanaresultthatshowsthataparticulartransformationcannotbeaccomplishedbyacertainsetofnetworks.Likewise,byapositiveresultwemeanademonstrationthatamemberofacertainsetofnetworkscaneectthetransformationinquestion. 51

PAGE 52

Figure5-1. Exampleofatransformationthatnoacyclicnetworkcaneect.Theshadedregionisreplicatedover,toobtainmappingsforlargerandlargervaluesofT potentialfunctionsaswell.However,whenwewishtoproveapositiveresult,somecautionisrequired.So,forexample,supposewearetryingtocomeupwithanetworkthatcaneectatransformation,comingupwithanetworkwhoseneuronsmerelysatisfytheabstractmodel,leavesopenthepossibilitythatthepotentialfunctionsoftheneuronsofthenetworkarebiologicallyunrealistic.Therefore,whenweareprovingsuchpositiveresultswemustrestrictourselvestoassimpleamodelaspossible.WhatwedoisdescribetheneuronintheconstructionsothatitcancertainlybeeectedusingGerstner'sSpikeResponseModelSRM0 Gerstner&Kistler ( 2002 ).Analremarkaboutcausality.Wheneveroneprescribesatransformation,werequireittobecausal.Thatis,iftwoinputspikecongurationsareidenticaluptosometimet,theneachoftheirprescribedoutputsmustalsobeidenticaluptot.Thisistoforbidnon-causaltransformations,whichareobviouslyunrealistic.Wenowmentionsomecomplexityresults.First,wepointoutthatitisstraightforwardtoconstructatransformationthatcannotbeeectedbyanyacyclicnetwork.OneofitsinputspikecongurationswiththeprescribedoutputisshowninFigure 5-1 .ForlargerT,theshadedregionissimplyreplicatedoverandoveragain.Informally,thereasonthistransformationcannotbeeectedbyanynetworkisthat,foranynetwork,beyondacertainvalueofT,theshadedregiontendstoactasausherasingmemoryoftherstspike.Whenthenetworkreceivesanotherinputspike,itisintheexactsame\state"itwaswhenitreceivedtherstspike,andthereforeproducesnooutputspike.Whether 52

PAGE 53

Figure5-2. Atransformationthatnosingleneuroncaneect,thatanetworkwithtwoneuronscan. thereareothertypesoftransformationthatcannotbedonebyanyacyclicnetworkremainstobeinvestigated.Next,weprovethatthesetofnetworkswithatmosttwoneuronsismorecomplexthanthesetofsingleneurons.Theproofisbyprescribingatransformationwhichcannotbedonebyanysingleneuron.Wethenconstructanetworkwithtwoneuronsthatcaneectthistransformation. Theorem1. Form2,thesetofacyclicnetworkswithatmosttwoneuronswhichhaveminputverticesismorecomplexthanthesetofsingleneuronswithminputvertices. Proof. Werstprovetheresultform=2andindicatehowitcanbetriviallyextendedforlargervaluesofm.Thefollowingtransformationisprescribedform=2.Letthetwoinputspiketrainsineachinputspikeconguration,whichsatisesaFlushCriterionbeI1andI2.I1hasevenly-spacedspikesstartingattimeinstantTuntil0.Forthesakeofexposition,wecallthedistancebetweenconsecutivespikes,onetimeunitandwenumberthespikesofI1withtherstspikebeingtheoldestone.TheithinputspikecongurationintheprescribedtransformationsatisesaT-Flushcriterion,whereT=4i+3timeunits.Intheithconguration,I2hasspikesattimeinstantsatwhichspikenumbers2i+1and4i+3occurinI1.Finally,theoutputspiketraincorrespondingtothetheithinputspikecongurationhasexactlyonespikeatthetimeinstantatwhichI1hasspikenumber4i+3.Figure 5-2 illustratesthetransformationfori=2. 53

PAGE 54

Next,weprovethatthetransformationprescribedabovecannotbeeectedbyanysingleneuron.Forthesakeofcontradiction,assumeitcan,byaneuronwithassociatedand.Letmax(;)beboundedfromabovebyktimeunits.Weshowthatthek 2thinputspikecongurationandabovecannotbemappedbythisneurontotheprescribedoutputspiketrain.Considertheoutputoftheneuronatthetimeinstantscorrespondingtothe(k+1)thspikenumberand(2k+3)rdspikenumberofI1.Ateachofthesetimeinstants,theinputreceivedinthepastktimeunitsandtheoutputproducedbytheneuroninthepastktimeunitsarethesame.Therefore,theneuron'smembranepotentialmustbeidentical.However,thetransformationprescribesnospikeinthersttimeinstantandaspikeinthesecond,whichisacontradiction.Itfollowsthatnosingleneuroncaneecttheprescribedtransformation.WenowconstructatwoneuronnetworkwhichcancarryoutthetransformationprescribedinFigure 5-2 .Theideaistousethe\innite-timememory"mechanismofFigure 4-1 hereto\remember"iftherstspikeonI2hasoccurredornot,andtoswitchtheoutputspikepatternwhenitdoes.ThenetworkisshowninFigure 5-3 (A).I1andI2arriveinstantaneouslyatN2.I1arrivesinstantaneouslyatN1butI2arrivesatN1afteradelayof1timeunit.SpikesoutputbyN1takeonetimeunittoarriveatN2,whichistheoutputneuronofthenetwork.Thefunctioningofthisnetworkfori=2isdescribedinFigure 5-3 (B).Thegeneralizationforlargeriisstraightforward.Allinputsareexcitatory.N1isakintotheneurondescribedinexampleofFigure 4-1 ,inthatitwhilethedepolarizationduetoaspikeinI1causespotentialtocrossthreshold,if,additionally,thepreviousoutputspikehappenedonetimeunitago,theassociatedhyperpolarizationissucienttokeepthemembranepotentialbelowthresholdnow.However,ifthereisaspikefromI2alsoatthesametime,thedepolarizationissucienttocauseanoutputspike,irrespectiveofiftherewasanoutputspikeonetimeunitago.ThecorrespondingtoN2isshorterthan1timeunit.Further,N2producesaspikeifandonlyifallthreeofitsaerentsynapsesreceivespikesatthesametime.Inthegure, 54

PAGE 55

Figure5-3. Atwo-neuronnetworkeectingthetransformationprescribedinFigure 5-2 .A)ThenetworkthatcaneectthetransformationdescribedinFigure 5-2 .B)Figuredescribingtheoperationofthisnetwork. I1spikesattimes1;3;5.Itspikesat6becauseitreceivedspikesbothfromI1andI2atthattimeinstant.Subsequently,itspikesat8and10.TheonlytimewhereinN2receivedspikesatallthreesynapsesatthesametimeisat11,whichistheprescribedtime.Thegeneralizationforlargeriisstraightforward.Forlargermonecanjusthavenoinputontheextrainputspiketrainsandthesameproofgeneralizestrivially.Thiscompletestheproof. Theaboveproofalsosuggestsalargeclassoftransformationsthatcannotbedonebyasingleneuron.Informally,thesearetransformationsforwhichthereisnoxedbound,so 55

PAGE 56

Figure5-4. Atransformationthatnonetworkwithapath-disjointarchitecturecaneect. thatonecanalwaysdeterminewhetherthereisanoutputspikeornot,justbylookingawindowofpastinputandpastoutput,sothatthewindowhaslengthatmostthisbound.Thepreviousresultmightsuggestthatthemorethenumberofneuronsthelargerthevarietyoftransformationspossible.Thenextcomplexityresultdemonstratesthatthestructureofthenetworkarchitectureiscrucial.Thatis,wecanconstructnetworkarchitectureswitharbitrarilylargenumberofneuronswhichcannotperformtransformationsthatasimple2-neuronnetworkcan.Thisalsoshowsaclassofarchitecturesthatshareacertainabstractgraph-theoreticpropertyalsoshareintheirinabilityineectingaparticularclassoftransformations.First,wedescribeadenitionthatmakesprecisetheabstractgraph-theoreticpropertythatcharacterizesthisclassofarchitectures. Denition13(Path-disjointArchitecture). AnacyclicnetworkarchitectureAhGhV;Ei;I;o;#:f1;2;:::;jIjg!I;f#v:f1;:::;indegree(v)g!Vjv2(V)]TJ /F7 11.955 Tf 11.97 0 Td[(I)gi,wherejIj=miscalled\path-disjoint"ifforeverysetofmpaths,wheretheithpathstartsatinputvertexiandendsattheoutputvertex,theintersectionofthempathsisexactlytheoutputvertex. Theorem2. Form3,let1bethesetofallacyclicnetworkswhosearchitectureispath-disjoint.Let2betheunionof1withthesetofallnetworkswithatmost2neuronswhichhaveminputvertices.Then2ismorecomplexthan1. Proof. Weprovethetheoremform=3;thegeneralizationforlargermisstraightforward.Thefollowingtransformationisprescribedform=3.Letthethreeinputspiketrains 56

PAGE 57

ineachinputspikeconguration,whichsatisesaFlushCriterionbeI1,I2andI3.Asbefore,wewilluseregularlyspacedspikes;wecallthedistancebetweentwosuchconsecutivespikesonetimeunitandnumberthesespiketimeinstantswiththeoldestbeingnumbered1;wecallthisnumberingthespikeindex.ThetransformationisprescribedforinputsofvariouslengthswiththeithinputspikecongurationintheprescribedtransformationsatisfyingaT-FlushCriterionforT=4imtimeunits.Therst2itimeunitshavespikesonI2spacedonetimeunitapart,thenext2ionI3andsoforth.Inaddition,atspikeindex2im,Imhasasinglespike.Theinputspikepatternfromthebeginningisrepeatedonceagainforthelatter2imtimeunits.Theoutputspiketrainhasexactlyonespikeatspikeindex4im.Figure 5-4 illustratesthetransformationfori=2.Nextweprovethatthetransformationprescribedabovecannotbeeectedbyanynetworkin1.Forthesakeofcontradiction,assumethatN21caneectthetransformation.LetandbeupperboundsonthesameparametersoveralloftheneuronsinNandletdbethedepthofN.Byconstructionof1,everyneuroninNthatiseerentontheoutputneuronreceivesinputfromatmostm)]TJ /F1 11.955 Tf 11.28 0 Td[(1oftheinputspiketrains;for,otherwisetherewouldexistasetofmpaths,onefromeachinputvertextotheoutputneuron,whoseintersectionwouldcontaintheneuroninquestion.Theclaim,now,isthatfori>d 2+,theoutputneuronofNhasthesamemembranepotentialatspikeindex2imand4im,andthereforeeitherhastospikeatboththoseinstantsornot.Intuitively,thisissobecauseeachneuroneerentontheoutputneuronreceivesa\ush"atsomepointafter2im,sothattheoutputproducedbyitsecondsbeforetimeindex2imandsecondsbeforetimeindex4imarethesame.Wewillleavethetaskofverifyingthisasanexercisetothereader.Wenowconstructatwo-neuronnetworkthatcaneectthistransformation.TheconstructionissimilartotheoneusedinTheorem 1 .Form=3,thenetworkisshowninFigure 5-5 (A).I1,I2andI3arriveinstantaneouslyatN1andN2.SpikesoutputbyN1taketwotimeunitstoarriveatN2,whichistheoutputneuronofthenetwork.The 57

PAGE 58

Figure5-5. Atwo-neuronnetworkeectingthetransformationprescribedinFigure 5-4 .A)NetworkthatcaneectthetransformationdescribedinFigure 5-4 .B)Figuredescribingtheoperationofthisnetwork. functioningofthisnetworkfori=2isdescribedinFigure 5-5 (B).Thegeneralizationforlargeriisstraightforward.Allinputsareexcitatory.N1isakintothetheN1usedinthenetworkinTheorem 1 exceptthatthatperiodicinputmayarrivefromanyoneofI1,I2orI3.Asbefore,iftwoinputspikesarriveatthesametime,asinspikeindex2im,thedepolarizationissucienttocauseanoutputspike,irrespectiveofiftherewasanoutputspikeonetimeunitago.Again,thecorrespondingtoN2isshorterthan1timeunitandN2producesaspikeifandonlyifthreeofitsaerentsynapsesreceivespikesatthesametime.Asbeforetheideaisthatattime2im,N2,receivestwospikes,butnotaspikefromN1,sinceitis\outofsync".However,attime4im,additionally,thereisaspikefromN1arrivingatN2,whichcausesN2tospike. 58

PAGE 59

Thetheoremthereforedemonstratesthatitisfeasibletoattempttolinktransformationalpropertiesofnetworkstoabstractgraph-theoreticpropertiesoftheirarchitecture.Italsoestablishesthatnetworkstructureiscrucialindeterminingtransformationalability. 59

PAGE 60

CHAPTER6COMPLEXITYRESULTSANDNETWORKDEPTH:ABARRIERATDEPTHTWOWhile,inthepreviouschapter,wehaveprovedanumberofcomplexityresults,theperceptivereaderwillobservethatwedidnotexplicitlyaddressthequestionofcomplexityresultspertainingtodepth,ingeneral.Inotherwords,thequestionis,doesincreaseindepthofthenetwork,ingeneral,buyusalargervarietyoftransformationseectablebyit?Moreformally,isthesetof\all"acyclicnetworksmorecomplexthanthesetofacyclicnetworksofdepthk,fork2?Muchtooursurprise 1 ,itturnsoutthattheanswerisNo,evenfork=2,andwecanproveit.Thatis,givenanarbitraryacyclicnetwork(consistingofneuronsobeyingtheabstractmodeldescribedinChapter3,thereexistsanetworkofdepthtwo(equippedwithneuronsobeyingthesameabstractmodel),sothatthelatternetworkinduces\exactly"thesametransformationastheformer.TheimplicationofthisresultisthatoneneedstoaddmoreaxiomstotheabstractmodelofChapter3,inordertobreakthisbarriertothemanifestationofcomplexityresults. 2 Thedicultyinprovingthateveryacyclicnetwork,havingarbitrarydepth,hasanequivalentnetworkofdepthtwo,appearstobeindevisingawayof\collapsing"thedepthoftheformernetwork,whilekeepingtheeectedtransformationthesame.Ourproofactuallydoesnotdemonstratethishead-on,butinsteadprovesittobethecaseindirectly.Thebroadattackconsistsofstartingowithacertainsubsetofthesetofallpossibletransformationsandshowingthateverytransformationthatliesoutsidethissubsetcannotbeeectedby\any"acyclicnetwork.Thereafter,weprovethateverytransformation 1 Indeed,theauthorspentmanyamoonattemptingtoprovetheaforementionedtypeofcomplexityresults.Inhindsight,ofcourse,somethinglikethisshouldhavebeenexpectedatsomepoint,giventhattheabstractmodelassumedsolittleaboutsingleneurons.2 Notethatthecomplexityresultsprovedthusfarstillhold,ifthe\new"abstractmodelalsohasalltheaxiomsoftheonedescribedinChapter3. 60

PAGE 61

inthissubsetcaninfactbeeectedbyanacyclicnetworkofdepthtwo,byprovidingaconstruction. 6.1TechnicalStructureoftheProofThemaintheoremofthischapteristhefollowing. Theorem. IfT:Fm!Scanbeeectedbyanacyclicnetwork,thenitcanbeeectedbyanacyclicnetworkofdepthtwo.Theabovetheoremfollowsfromthefollowingtwolemmmaswhichareprovedinthetwosubsectionsthatfollow: Lemma. IfT:Fm!Scanbeeectedbyanacyclicnetwork,thenitiscausal,time-invariantandresettable. Lemma. IfT:Fm!Siscausal,time-invariantandresettable,thenitcanbeeectedbyanacyclicnetworkofdepthtwo. 6.2Causal,Time-InvariantandResettableTransformationsInthissection,werstdenenotionsofcausal,time-invariantandresettabletransformations 3 .Transformationsthatarecausal,time-invariantandresettableformastrictsubsetofthesetofalltransformations.Notethatthesenotionsexistindependentoftheexistenceofneuronsortheirnetworks.Wethenshowthattransformationslyingoutsidethissubsetcannotbeeectedbyanyacyclicnetwork.Thisistherelativelyeasypartoftheproof.Thenextsectionprovestheharderpart,namelythateverytransformationinthissubsetcanindeedbeeectedbyanacyclicnetworkofdepthequaltotwo.Asinsystemstheory,informally,a\causaltransformation"isonewhosecurrentoutputdependsonlyonitscurrentinputandpastinput(andnotfutureinput).Abstractly,itisconvenienttodeneacausaltransformationasonethatgiventwo 3 Recallthatwhenwesaytransformationwithoutfurtherqualication,wemeanone,oftheformT:Fm!S. 61

PAGE 62

dierentinputsthatareidenticaluptoacertainpointintime,alsohavetheiroutputs,accordingtothetransformation,beidenticalupto(atleast)thesamepoint. Denition14(CausalTransformation). AtransformationT:Fm!Sissaidtobe\causal"if,forevery1;22Fm,with[t;1)1=[t;1)2,forsomet2R,wehave[t;1)T(1)=[t;1)T(2).Again,asinsystemstheory,a\time-invarianttransformation"isoneforwhichaninputwhichisatime-shiftedversionofanotherinputhasasoutputthetime-shiftedversionoftheoutputcorrespondingtothelatterinput.Tokeepthedenitionkosher,wealsoneedtoensurethatthetime-shiftedinputinfactalsosatisestheushcriterion. Denition15(Time-InvariantTransformation). AtransformationT:Fm!Sissaidtobe\time-invariant"if,forevery2Fmandeveryt2Rwitht()2Fm,wehaveT(t())=t(T()).A\resettabletransformation"isoneforwhichthereexistsapositiverealnumberW,sothataninputgapoftheform(t;t+W]\resets"it,i.e.outputbeyondtisindependentofinputreceivedbeforeit.Again,abstractly,itbecomesconvenienttosaythattheoutputinthiscaseisidenticaltothatproducedbyaninputwhichhasnospikesbeforet,butisidenticaltothepresentinputthereafter. Denition16(W-ResettableTransformation). ForW2R+,atransformationT:Fm!Sissaidtobe\W-resettable"if,forevery2Fmwhichhasagapintheinterval(t;t+W],forsomet2R,wehave(;t]T()=T((;t]). Denition17(ResettableTransformation). AtransformationT:Fm!Sissaidtobe\resettable"ifthereexistsaW2R+,sothatitisW-resettable.Next,weprovethateverytransformationthatcanbeeectedbyanacyclicnetworkiscausal,time-invariantandresettable.Thisimpliesthateverytransformationthatisnotcausal,time-invariantandresettablecannotbeeectedbyanyacyclicnetwork. Lemma6. IfT:Fm!Scanbeeectedbyanacyclicnetworkthenitiscausal,time-invariantandresettable. 62

PAGE 63

Proof. LetNhA;N;L:(V)]TJ /F7 11.955 Tf 12.84 0 Td[(I)!NibeanetworkthateectsT:Fm!S.Werstshowthatitiscausal.Considerarbitrary1;22Fmwith[t;1)1=[t;1)2,forsomet2R.Wewishtoshowthat[t;1)T(1)=[t;1)T(2).LetNiN,for1id,bethesetofneuronsinNofdepthi,wheredisthedepthofN.EachneuronN2N1receivesallitsinputsfromspiketrainsin.Let01=indegree(v)Fi=1#)]TJ /F4 5.978 Tf 5.75 0 Td[(1(#v(i))(1)and02=indegree(v)Fi=1#)]TJ /F4 5.978 Tf 5.76 0 Td[(1(#v(i))(2),wherev=L)]TJ /F6 7.97 Tf 6.59 0 Td[(1(N).Whenthenetworkreceives1and2asinput,Nreceives01and02respectivelyasinput.Also,clearly[t;1)01=[t;1)02.Let~x01and~x02betheoutputproducedbyNonreceiving01and02respectively.Since01;022Fm,thereexistsaT2R+,sothat[T;1)01=[T;1)02=~m0,wherem0isthenumberofinputstoN.Therefore,byAxiom(3)oftheneuron,wehave[T;1)~x01=[T;1)~x02=~.Now,forallt02R,t02~x0jifandonlyifPN([0;N](t0(0j));[0;N](t0(~x0j))=N,forj=1;2.Itisimmediatethatfort0t,wehave[0;N](t0(01))=[0;N](t0(02)).Now,byaninductionargumentonthespikenumbersinceT,itisstraightforwardtoshowthatforallt0t,[0;N](t0(~x01))=[0;N](t0(~x02)).Thus,wehave[t;1)~x01=[t;1)~x02.Similarly,usingastraightforwardinductionargumentondepth,onecanshowthatforeveryneuroninthenetwork,itsoutputuntiltimeinstanttisidenticalineithercase.Wethereforehave[t;1)T(1)=[t;1)T(2).Next,weshowthatitistime-invariant.Considerarbitrary2Fmandt2Rwitht()2Fm.WewishtoshowthatT(t())=t(T()).Asbefore,letNiN,for1id,bethesetofneuronsinNofdepthi,wheredisthedepthofN.EachneuronN2N1receivesallitsinputsfromspiketrainsin.Let0=indegree(v)Fi=1#)]TJ /F4 5.978 Tf 5.76 0 Td[(1(#v(i))().Whenthenetworkreceivesandt()asinput,Nreceives0andt(0)respectivelyasinput.Let~x01and~x02betheoutputproducedbyNonreceiving0andt(0)asinputrespectively.Wewishtoshowthat~x02=t(~x01).Since02Fm,thereexistsaT2R+,sothat[T;1)0=[T)]TJ /F10 7.97 Tf 6.59 0 Td[(t;1)t(0)=~m0,wherem0isthenumberofinputstoN.Therefore,byAxiom(3)oftheneuron,wehave[T;1)~x01=[T)]TJ /F10 7.97 Tf 6.59 0 Td[(t;1)~x02=~.Now,forallt02R,t02~x01ifandonlyifPN([0;N](t0(0));[0;N](t0(~x01))=N.Itisthereforestraightforwardtomake 63

PAGE 64

aninductionargumentonthespikenumber,startingfromtheoldestspikein~x01toshowthat~x01hasaspikeatsomet0i~x02hasaspikeatt0)]TJ /F7 11.955 Tf 12.01 0 Td[(tandthereforewehave~x02=t(~x01).Similarly,usingastraightforwardinductionargumentondepth,onecanshowthatforeveryneuroninthenetwork,itsoutputinthesecondcaseisatime-shiftedversionoftheoneintherstcase.WethereforehaveT(t())=t(T()).Finally,weshowthatTisresettable.LetandbeupperboundsonthoseparametersoveralltheneuronsinN.If<,thenset=.TheclaimisthatforW=d(+)++,forarbitrary>0,TisW-resettable.Considerarbitrary2Fmsothathasagapintheinterval(t;t++d(+)+],forsomet2R+.Thushasagapintheinterval[t+;t++d(+)+].Wewillshowthat(;t+]T()=T((;t+])whichimpliestherequiredresult(;t]T()=T((;t]).Asbefore,letNiN,for1id,bethesetofneuronsinNofdepthi,wheredisthedepthofN.EachneuronN2N1receivesallitsinputsfromspiketrainsin.ThereforebyAxiom(3)oftheneuron,itisstraightforwardtoseethattheoutputofNhasagapintheinterval(t;t++(d)]TJ /F1 11.955 Tf 12.03 0 Td[(1)(+)+2].Bysimilararguments,wehavethatoutputofeachneuronN2Ni,for1idhasagapintheinterval(t;t++(d)]TJ /F7 11.955 Tf 12.12 0 Td[(i)(+)+(i+1)].Thusinparticulartheoutputneuronhasagapintheinterval[t+;t++(d+1)].Sinced1,theGapLemmaapplies,andattimeinstantt+theoutputoftheoutputneurondependsonspikesintheinterval[t+;t++(+)]ofitsinputs.Allinputstotheoutputneuronhaveagapintheinterval[t+;t++(+)+d],sincetheyhavedepthatmost(d)]TJ /F1 11.955 Tf 11.68 0 Td[(1).Sincethoseinputshaveagapintheinterval[t++(+);t++(+)+d],ford2,theGapLemmaappliesandtheoutputneuron'soutputattimeinstantt+dependsonoutputsofthe\penultimate"layerintheinterval[t+;t++2(+)].Thereforebysimilararguments,theoutputoftheoutputneuronattimeinstantt+atmostdependsoninputsfromintheinterval[t+;t++d(+)].ThatistosaythatT(0),forevery0identicaltointheinterval(;t+],hasthesameoutputasT()inthatsameinterval,followingthecorollarytotheGapLemma.Inparticular, 64

PAGE 65

(;t+]isonesuch0.Wethereforehave(;t+]T()=T((;t+])whichimplies(;t]T()=T((;t]).Thus,Tisresettable. 6.3ConstructionofaDepthTwoAcyclicNetworkforeveryCausal,Time-InvariantandResettableTransformationInthissection,wedescribeaconstructionofadepthtwoacyclicnetworkforeverycausal,time-invariantandresettabletransformation.Moreformally,weprovethefollowinglemma. Lemma. IfT:Fm!Siscausal,time-invariantandresettable,thenitcanbeeectedbyanacyclicnetworkofdepthtwo.Beforewediveintotheproofs,weoersomeintuition.SupposewehadatransformationT:Fm!Swhichiscausal,time-invariantandresettable.Forthemoment,pretenditsatisesthefollowingproperty:Foreveryinputspikecongurationsatisfyingaushcriterion,thereexistconstantinputandoutput\windows",sothatatanypointintime,justgivenknowledgeofspikesinthosewindowsofpastinputandoutput,onecanunambiguouslydetermineifthetransformationprescribesanoutputspikeornot.Intuitively,itseemsreasonablethatsuchatransformationcaninfactbeeectedbyasingleneuron 4 bysettingtheandoftheneurontothesizesoftheinputandoutputwindowmentionedabove.Ofcourse,oneeasilyseesthatnoteverytransformationthatiscausal,time-invariantandresettablesatisestheaforementionedproperty.Thatis,therecouldexisttwodierentinputinstances,whichhavepastinputsandoutputsbeidenticalintheaforementionedwindowsatsomepointsintime;yetinoneinstance,thetransformationprescribesanoutputspike,whereasitprescribesnoneintheother.Indeed,thetwoinputinstancesdodieratsomepointinthepast,forotherwisethetransformationwouldnotbecausal.Therefore,insuchasituation,itisnaturaltoaskifasingle\intermediate" 4 Presumingthatnoaxiomsoftheneuronareviolated. 65

PAGE 66

Figure6-1. Networkarchitectureform=2. neuroncan\breakthetie".Thatis,iftwoinputinstancesdieratsomepointinthepast,theoutputoftheintermediateneuronsincethen,inanyintervaloftimeoflengthU,mustbedierent.Thisissothatitcandisambiguatethetwo,wereanoutputspikedemandedforoneinstancebutnottheother.Unfortunately,thisexactpropertycannotbeachievedbyanysingleneuronbecausethetransformationT:Fm!Sisresettable,andsoiseverytransformationinducedbytheintermediateneuron.Inotherwords,theproblemisthat,supposetwoinputinstancesdieratacertainpointintime;however,sincethen,bothhavehadanarbitrarilylargeinputgap.Theinputgapservesto\erasememory"inanynetworkthatreceiveditandthereforeitcannotdisambiguatethetwoinputsbeyondthisgap.Fortunately,itdoesnothaveto,sincethisgapalsocausesa\reset"inthetransformation(whichisresettable).Thus,wecanmakedowithaslightlyweakercondition;thattheintermediateneuronisonlyguaranteedtodisambiguate,whenitisrequiredtodoso.Thatis,supposetherearetwoinputinstances,whoseoutputsaccordingtoT:Fm!Saredierentatcertainpointsintime.Then,thecorrespondinginputsmustbedierenttooatsomepointinthepastwithnoresetsintheinterveningtimeandthereforetheintermediateneuronshouldbreakthetie.Additionally,wesaythattheoutputsoftheintermediateneuronintheprecedingUsecondsareguaranteedtobedierent,onlyiftheinputsthemselvesinthepastUsecondsarenotdierent. 66

PAGE 67

ThenetworkwehaveinmindisillustratedinFigure 6-1 ,form=2.Inthefollowingproposition,weprovethatiftheintermediateneuronsatisestheweakerconditionalludedtoabove,thenthereexistsanoutputneuron,sothatthenetworkeectsthetransformationinquestion. Proposition2. LetT:Fm!Sbecausal,time-invariantandresettable.LetIbeaneuronwithTI:Fm!S,sothatforeach2Fm,TI()isconsistentwithwithrespecttoI.Further,supposethereexistsaU2R+sothatforallt1;t22Rand1;22Fmwith0t1(T(1))6=0t2(T(2)),wehave[0;U](t1(TI(1)t1))6=[0;U](t2(TI(2)t2)).ThenthereexistsaneuronO,sothatforevery2Fm,T()isconsistentwithTI()twithrespecttoO. Proof. Assumethatthehypothesisinthepropositionaboveistrue.LetT:Fm!SbeW-ResettableforsomeW2R+.WerstshowaconstructionfortheneuronO,provethatitobeysalltheaxiomsandthenshowthatithasthepropertythatforevery2Fm,T()isconsistentwithTI()twithrespecttoO.WerstconstructtheneuronOhO;O;O;O;O;mO;PO:SmO[0;O]S[0;O]![O;O]i.SetO=andO;O2R+,O2R)]TJ /F1 11.955 Tf 10.99 -4.34 Td[(arbitrarilywithOO.SetO=maxfU;WgandmO=m+1.ThefunctionPO:SmO[0;O]S[0;O]![O;O]isconstructedasfollows.For02SmO[0;O]and~x002S[0;O],setPO(0;~x00)=OandPO(0;~)=Oifandonlyifthereexists2Fmandt2RsothattT()=htiand0=[0;O](t(TI()t))and~x00=[0;O](t(T())).Everywhereelse,thevalueofthisfunctionissettozero.Next,weshowitobeysalloftheaxiomsofthesingleneuron.WeprovethatOsatisesAxiom(1)byshowingthatitscontrapositiveistrue.Let02SmO[0;O]and~x002S[0;O]bearbitrarysothatPO(0;~x00)=O.If~x00=~,Axiom(1)isimmediatelysatised.Thusconsiderthecasewhen~x00=hx100;x200;:::xk00i.Thenx100,otherwise,fromtheconstructionofPO(),itisstraightforwardtoshowthatthereexistsa2FmwithT()=2S. 67

PAGE 68

Next,weprovethatOsatisesAxiom(2).Let02SmO[0;O]and~x002S[0;O]bearbitrary.IfPO(0;~x00)=O,thenitisimmediatefromtheconstructionthatPO(0;~)=O.Onthecontrary,ifPO(0;~x00)6=O,fromtheconstructionofO,wehavePO(0;~x00)=0.ThentheconditioninthehypothesisimpliesthatPO(0;~)6=O.Therefore,PO(0;~)=0.Thus,Axiom(2)issatisedeitherway.WithAxiom(3),wewishtoshowPO(~m+1;~)=0.Here,wewillshowthatPO(~xIt~m;~x00)=0,forall~xI2S[0;O]and~x002S[0;O]whichimpliestherequiredresult.Assume,forthesakeofcontradiction,thatthereexistsa~xI2S[0;O]and~x002S[0;O],sothatPO(~xIt~m;~x00)=O.FromtheconstructionofO,thisimpliesthatthereexists2Fmandt2RsothattT()=htiand[0;O](t())=~m.Thatis,hasagapintheinterval[t;t+W],sinceOW.SinceT:Fm!Siscausal,time-invariantandW-resettable,byCorollary 3 ,wehavetT()=~,whichisacontradiction.Therefore,wehavePO(~xIt~m;~x00)6=OandbyconstructionofO,PO(~xIt~m;~x00)=0,forall~xI2S[0;O]and~x002S[0;O].ThisimpliesPO(~m+1;~)=0.Finally,wewishtoshowthatforevery2Fm,T()isconsistentwithTI()twithrespecttoO.Thatis,wewishtoshowthatforevery2Fmandforeveryt2R,0t(T())=h0iifandonlyifPO([0;O](t(TI()t));[0;O](t(T())))=O.Considerarbitrary2Fmandt2R.If0t(T())=h0i,thenitisimmediatefromtheconstructionofOthatPO([0;O](t(TI()t));[0;O](t(T())))=O.Toprovetheconverse,suppose0t(T())6=h0i.Then,fromtheconditioninthehypothesis,itfollowsthatforall~2Fmandforall~t2Rwith[0;O](~t(TI(~)t~))=[0;O](t(TI()t)),wehave0~t(T(~))=0t(T())6=h0i.Therefore,fromtheconstruction,wehavePO([0;](t(TI()t));[0;](t(T())))6=O. Wenowgiveanintuitivedescriptionofhowtheintermediateneuronisconstructed,soastosatisfythesaidcondition.Thebasicideaisto\encode",inthetimedierenceoftwosuccessiveoutputspikes,thepositionsofalltheinputspikesthatoccurredsincethelastoutputgapoftheform(t;t+W],whichwecalla\resetgap"fromnowon,forthe 68

PAGE 69

sakeofexposition.Suchpairsofoutputspikesoccuronceeverypseconds,withthetimedierencewithineachpairbeingafunctionofthetimedierencewithinthepreviouspairandtheinputspikesencounteredsince.Intuitively,itisconvenienttothinkofthisencodingasonefromwhichwecan\reconstruct"theentirepastinputspiketrainsincethelastreset.Werstdescribetheencodingfunctionforthecaseofasingleinputspiketrainafterwhichweremarkonhowitcanbegeneralized.So,supposethetimedierenceofthesuccessivespikesliesintheinterval[0;1).Denetheencodingfunctionas"0:[0;1)S(0;p]![0;1).pischosentobesuchthatthereareatmost8spikesinanyintervaloftheform(t;t+p].Wenowdescribehow"0(e;~x)iscomputed,givene2[0;1)and~x=hx1;x2;:::;xki,suchthateachspiketimeliesintheinterval(0;p].Letehaveadecimalexpansion,sothate=0:c1s1c2s2c3s3:::.Accordingly,letc=0:c1c2c3:::ands=0:s1s2s3:::.cisarealnumberthatencodesthenumberofspikesineachintervaloflengthpencountered,sincethelastreset.Sinceeachintervaloflengthphasbetween0and8spikes,thedigit9isusedasa\terminationsymbol".So,forexample,supposetherehavebeen4intervalsoflengthp,sincethelastresetwith5;0;8and2spikesapiecerespectively,thenc=0:8059andc0=0:28059,wherec0isthe\updated"valueofc.Likewise,sisarealnumberthatstoresthepositionsofallinputspikesencounteredsincethelastreset.Leteachspiketimebeoftheformxi=0:xi1xi2xi3:::10q,forappropriateq.Thentheupdatedvalueofsiss0=0:x11x21:::xk1s1x12x22:::xk2s2:::.Supposethec0ands0obtainedabovewereoftheformc0=0:c01c02c03:::ands0=0:s01s02s03:::,then"0(e;~x)=0:c01s01c02s02:::.Supposetheinputwereaspikecongurationoforderm,thenforeachspiketrainanencodingwouldbecomputedasaboveandinthenalstep,themrealnumbersobtainedwouldbeinterleavedtogether,soastoproducetheencoding.Givenknowledgeoftheencodingfunction,wenowdescribehowIexactlyworks.Figure 6-2 providesanillustration.Suppose2Fmisaninputspiketrain.LetitsoldestspikebeTsecondsago.ThenIproducesaspikeattimeT)]TJ /F7 11.955 Tf 12.4 0 Td[(pandateveryT)]TJ /F7 11.955 Tf 12.41 0 Td[(kp,for 69

PAGE 70

Figure6-2. ExampleillustratingtheoperationofI. k2Z+,unlessinthepreviouspsecondstowhenitistospikethereisagap 5 oftheform(t;t+W].Forthesakeofexposition,let'scallthesethe\clock"spikes.Now,supposethereisagapoftheform(t;t+W]intheinputandthereisaninputspikeattimet,thentheneuronspikesattimet)]TJ /F7 11.955 Tf 12.46 0 Td[(pandeverypsecondsthereaftersubjecttothesame\rules"asabove.Theseclockspikesarefollowedby\encoding"spikes,whichoccuratleastqsecondsaftertheclockspike,butatmostq+rsecondsafter,whereqisgreaterthantheabsoluterefractoryperiod.Asexpected,thepositionofthecurrentencodingspikeisafunctionofthetimedierencebetweenthepreviousencodingandclockspike 6 andthepositionsoftheinputspikesinthepastpseconds.Theoutputoftheencodingfunctionisappropriatelyscaledto\t"inthisintervaloflengthr;thedetailsareavailableintheproof.Theclaimthenisthatiftwoinputspiketrainsaredierentatsomepointwithnointervening\reset"gaps,thentheoutputofIinthepastUseconds,whereU=p+q+rwillbedierent.Intuitively,thisisbecausethedierencebetweenthelatestencodingandclockspikeineachcasewouldbedierent,astheyencodedierent\histories"ofinputspikes. 5 WesetW>ptoforceaspikeatT)]TJ /F7 11.955 Tf 11.96 0 Td[(p.6 unlessthepresentclockspikeistherstafteraresetgapintheinput. 70

PAGE 71

Finally,weremarkthattheaboveisjustaninformaldescriptionthatglossesoverseveraltechnicaldetailsthatthereaderisinvitedtoenjoyintheproof. Proposition3. LetT:Fm!Sbecausal,time-invariantandresettable.ThenthereexistsaneuronIandU2R+sothatforallt1;t22Rand1;22Fmwith0t1(T(1))6=0t2(T(2)),wehave[0;U](t1(TI(1)t1))6=[0;U](t2(TI(2)t2)),whereTI:Fm!Ssuchthatforeach2Fm,TI()isconsistentwithwithrespecttoI. Proof. Assumethatthehypothesisinthepropositionaboveistrue.LetT:Fm!SbeW0-ResettableforsomeW02R+.SetW=maxfW0;12g.OnereadilyveriesthatT:Fm!SisalsoW-resettable.WerstshowaconstructionfortheneuronI,provethatitobeysalltheaxiomsandthenshowthatithasthepropertythatthereexistsaU2R+sothatforallt1;t22Rand1;22Fmwith0t1(T(1))6=0t2(T(2)),wehave[0;U](t1(TI(1)t1))6=[0;U](t2(TI(2)t2)),whereTI:Fm!Ssuchthatforeach2Fm,TI()isconsistentwithwithrespecttoI.WerstconstructtheneuronIhI;I;I;I;I;mI;PI:SmI[0;I]S[0;I]![I;I]i.SetI=.Letp;q;r2R+,with 7 p=8;q=2andr=.SetI=p+q+r+W,I=p+q+randmI=m.LetI2R+,I2R)]TJ /F1 11.955 Tf 10.98 -4.34 Td[(bechosenarbitrarily.ThefunctionPI:SmI[0;I]S[0;I]![I;I]isconstructedasfollows.For2SmI[0;I]and~x02S[0;I],setPI(;~x0)=IandPI(;~)=Iifandonlyifoneofthefollowingistrue;everywhereelse,thefunctionissettozero. 1. (p;p+W]=~mI,p6=~mIand[0;p]~x0=~. 2. [0;p+q]~x0=hti,whereqt<(q+r)and(t)]TJ /F7 11.955 Tf 12.53 0 Td[(q)="(0;(0;p]t()).Moreover,(t+p;t+p+W]=~mIand(p+t)6=~mI. 7 Thechoiceofvaluesforp,q,randWwasmadesoastosatisfythefollowinginequalities,whichwewillneedintheproof:p2(q+r)andq>. 71

PAGE 72

3. [0;2p)]TJ /F6 7.97 Tf 6.59 0 Td[((q+r)]~x0=htx;tyiwith(p)]TJ /F1 11.955 Tf 12.5 0 Td[((q+r))tx(p)]TJ /F7 11.955 Tf 12.49 0 Td[(q)ty=p.Also,forallt02[0;p],(t0;t0+W]6=~mI. 4. [0;2p)]TJ /F10 7.97 Tf 6.59 0 Td[(r]~x0=ht;tx;tyiwithqt(q+r)<(p)]TJ /F7 11.955 Tf 12.41 0 Td[(r)txpty=p+tand(t)]TJ /F7 11.955 Tf 9.95 0 Td[(q)="((ty)]TJ /F7 11.955 Tf 9.94 0 Td[(tx)]TJ /F7 11.955 Tf 9.95 0 Td[(q);(0;p]t()).Furthermore,forallt02[0;p+t],(t0;t0+W]6=~mI.where":[0;r]SmI(0;p]![0;r]isasdenedbelow.Forconvenience,wedeneanoperatorkj:[0;1)![0;1),forj;k2Z+,thatconstructsanewnumberobtainedbyconcatenatingeveryithdigitofagivennumber,whereijmodk.Moreformally,forx2[0;1),kj(x)=1i=1((bx10j+(i)]TJ /F6 7.97 Tf 6.59 0 Td[(1)kc)]TJ /F1 11.955 Tf 20.39 0 Td[(10bx10j+(i)]TJ /F6 7.97 Tf 6.59 0 Td[(1)k)]TJ /F6 7.97 Tf 6.58 0 Td[(1c)10)]TJ /F10 7.97 Tf 6.59 0 Td[(i).Also,wedeneanotheroperatork:[0;1)k![0;1),fork2Z+which\interleaves"thedigitsofkgivennumbersinordertoproduceanewnumber.Moreformally,forx0;x1;:::;xk)]TJ /F6 7.97 Tf 6.58 0 Td[(12[0;1),k(x0;x1;:::;xk)]TJ /F6 7.97 Tf 6.59 0 Td[(1)=1i=0((bxk(i kbi kc)101+bi kcc)]TJ /F1 11.955 Tf 20.1 0 Td[(10bxk(i kbi kc)10bi kcc)10)]TJ /F6 7.97 Tf 6.59 0 Td[((i+1)).Letdbethelargestintegersothat,forallx02[0;r],wehavex010d<1.Forx02[0;r],letx=x010d.For2SmI(0;p],dene 8 "(x0;)=10)]TJ /F10 7.97 Tf 6.58 0 Td[(dmI("0(mI1(x);1());"0(mI2(x);2());:::;"0(mImI(x);mI())),where"0:[0;1)S(0;p]![0;1)isasdenedbelow.Letn2[0;1)and~x2S(0;p].Furthermore,letc=21(n)ands=22(n).Let~x=hx1;x2;:::;xki.Wehave0k8,becausep=8.Also,sincep=8r,wehavexi10d)]TJ /F6 7.97 Tf 6.59 0 Td[(1<1,for1ik.Lets0=k+1(x110d)]TJ /F6 7.97 Tf 6.59 0 Td[(1;x210d)]TJ /F6 7.97 Tf 6.58 0 Td[(1;:::;xk10d)]TJ /F6 7.97 Tf 6.58 0 Td[(1;s).Ifc=0,thenletc0=k 10+0:09elseletc0=k 10+c 10.Finally,dene"0(n;~x)=2(c0;s0).Next,weshowthatIsatisesalltheaxiomsoftheneuron.ItisimmediatethatIsatisesAxiom(1),sincealloutputspikesintheaboveconstructionareatleastqsecondsapart,andq=2. 8 Recallthatthe\projectionoperatorforspikecongurations"isdenedasi()=~xi,for1im,where=h~x1;~x2;:::;~xmi. 72

PAGE 73

WenowprovethatIsatisesAxiom(2).Let02SmI[0;I]and~x002S[0;I]bearbitrary.IfPI(0;~x00)=I,thenitisimmediatefromtheconstructionthatPI(0;~)=IwhichsatisesAxiom(2).Onthecontrary,ifPI(0;~x00)6=I,fromtheconstructionofI,wehavePI(0;~x00)=0.Also,fromtheconstructionwehaveeitherPI(0;~)=0orPI(0;~)=I.Axiom(2)issatisedineithercase.Also,IsatisesAxiom(3),sinceitisclearthat=~mIdoesnotsatisfyanyoftheconditionsenumeratedabove.WethereforehavePI(~mI;~)=0.Finally,weshowthatthereexistsaU2R+sothatforallt1;t22Rand1;22Fmwith0t1(T(1))6=0t2(T(2)),wehave[0;U](t1(TI(1)t1))6=[0;U](t2(TI(2)t2)),whereTI:Fm!Ssuchthatforeach2Fm,TI()isconsistentwithwithrespecttoI.FromProposition 5 ,itfollowsthatthereexistV1;V22R+sothat[0;V1](t1(1))6=[0;V2](t2(2)).LetU=p+q+r.If[0;U](t1(1))6=[0;U](t2(2)),itisimmediatethat[0;U](t1(TI(1)t1))6=[0;U](t2(TI(2)t2)).Itthereforesucestoprovethatif[U;V1](t1(1))6=[U;V2](t2(2))then[0;U](t1TI(1))6=[0;U](t2TI(2)).Proposition 5 impliesthat(V1;V1+W](t1(1))=~mandV1(t1(1))6=~m.Therefore,byCase(1)oftheconstruction,(V1)]TJ /F10 7.97 Tf 6.59 0 Td[(p)t1TI(1)=hV1)]TJ /F7 11.955 Tf 12.6 0 Td[(pi.Moreover,sinceProposition 5 impliesthatforallt012[0;V1),(t01;t01+W](t1(1))6=~m,fromCase(3)oftheconstruction,wehavethatforeveryk2Z+withV1)]TJ /F7 11.955 Tf 12.78 0 Td[(kp>0,(V1)]TJ /F10 7.97 Tf 6.59 0 Td[(kp)t1TI(1)=hV1)]TJ /F7 11.955 Tf 12.78 0 Td[(kpi.Letk1be 9 thesmallestpositiveinteger,sothatV1)]TJ /F7 11.955 Tf 12.56 0 Td[(k1pU.Fromthepreviousarguments,wehave(V1)]TJ /F10 7.97 Tf 6.59 0 Td[(k1p)t1TI(1)=hV1)]TJ /F7 11.955 Tf 12.44 0 Td[(k1pi.Also,itiseasytoseethatV1)]TJ /F7 11.955 Tf 12.44 0 Td[(k1p(q+r).Letk2besimilarlydenedwithrespectto2sothat(V2)]TJ /F10 7.97 Tf 6.58 0 Td[(k2p)t2TI(2)=hV2)]TJ /F7 11.955 Tf 12.25 0 Td[(k2piandV2)]TJ /F7 11.955 Tf 11.96 0 Td[(k2pU.Now,therearetwocases: 1. IfV1)]TJ /F7 11.955 Tf 12.15 0 Td[(k1p6=V2)]TJ /F7 11.955 Tf 12.15 0 Td[(k2p,wenowshowthat[0;U](t1TI(1))6=[0;U](t2TI(2)),whichistherequiredresult.Assume,withoutlossofgenerality,thatV1)]TJ /F7 11.955 Tf 12.22 0 Td[(k1pp. 73

PAGE 74

[0;U](t2TI(2)),becausebyCase(4)oftheconstructionTI(1)hasaspikeintheinterval[V1)]TJ /F7 11.955 Tf 12.07 0 Td[(k1p)]TJ /F1 11.955 Tf 12.07 0 Td[((q+r);V1)]TJ /F7 11.955 Tf 12.07 0 Td[(k1p)]TJ /F7 11.955 Tf 12.07 0 Td[(q]andbyCase(3)oftheconstruction,TI(2)hasnospikeintheinterval(V2)]TJ /F7 11.955 Tf 11.13 0 Td[(k2p;V2)]TJ /F7 11.955 Tf 11.12 0 Td[(k2p+p)]TJ /F1 11.955 Tf 11.13 0 Td[((q+r)].Inotherwords,thespikefollowingtheoneatV1)]TJ /F7 11.955 Tf 12.47 0 Td[(k1pinTI(1)hasnocounterpartinTI(2).Ontheotherhand,iftheyarelessthanpapartbutatmostp)]TJ /F7 11.955 Tf 12.6 0 Td[(rapart,bysimilararguments,itiseasytoshowthatthespikeatV2)]TJ /F7 11.955 Tf 12.34 0 Td[(k2pinTI(2)hasnocounterpartinTI(1).Finally,iftheyareatleastpapart,thenk2doesnotsatisfythepropertythatitisthesmallestpositiveinteger,sothatV2)]TJ /F7 11.955 Tf 11.95 0 Td[(k2pU,whichisacontradiction. 2. Onthecontrary,considerthecasewhenV1)]TJ /F7 11.955 Tf 11.96 0 Td[(k1p=V2)]TJ /F7 11.955 Tf 11.96 0 Td[(k2p.Wehavetwocases: (a) Supposek16=k2.Lett01bethelargestpositiveintegersothatt01t1TI(1)=ht01iandt01
PAGE 75

becausetheydierinthe(k1)]TJ /F7 11.955 Tf 13 0 Td[(k0+1)thdecimalplace 10 .Therefore,[0;U](t1TI(1))6=[0;U](t2TI(2)). ii. Nowconsiderthecasewhereforalljwith1jmIandk0k1,wehave(V1)]TJ /F10 7.97 Tf 6.58 0 Td[(k0p;V1)]TJ /F6 7.97 Tf 6.58 0 Td[((k0)]TJ /F6 7.97 Tf 6.59 0 Td[(1)p]j(t1(1))havethesamenumberofspikeswhencomparedto(V2)]TJ /F10 7.97 Tf 6.59 0 Td[(k0p;V2)]TJ /F6 7.97 Tf 6.59 0 Td[((k0)]TJ /F6 7.97 Tf 6.59 0 Td[(1)p]j(t2(2)).Now,byhypothesis,wehave[U;V1](t1(1))6=[U;V2](t2(2)).Thereforetheremustexista1jmIandk0k1,sothatthereisapointintimewhereoneofthespiketrains(V1)]TJ /F10 7.97 Tf 6.58 0 Td[(k0p;V1)]TJ /F6 7.97 Tf 6.58 0 Td[((k0)]TJ /F6 7.97 Tf 6.59 0 Td[(1)p]j(t1(1))and(V2)]TJ /F10 7.97 Tf 6.58 0 Td[(k0p;V2)]TJ /F6 7.97 Tf 6.58 0 Td[((k0)]TJ /F6 7.97 Tf 6.59 0 Td[(1)p]j(t2(2))hasaspike,whiletheotherdoesnot.Lett0bethelatesttimeinstantatwhichthisisso.Also,assumewithoutlossofgeneralitythat(V1)]TJ /F10 7.97 Tf 6.59 0 Td[(k0p;V1)]TJ /F6 7.97 Tf 6.59 0 Td[((k0)]TJ /F6 7.97 Tf 6.58 0 Td[(1)p]j(t1(1))=hx1;:::;xqihasaspikeattimeinstantt0while(V2)]TJ /F10 7.97 Tf 6.59 0 Td[(k0p;V2)]TJ /F6 7.97 Tf 6.59 0 Td[((k0)]TJ /F6 7.97 Tf 6.58 0 Td[(1)p]j(t2(2))doesnot.Letpbethenumbersothatt0=xp.Letn1;n2bedenedasbefore.Also,foreachhwith1hk1,letrhbethenumberofspikesin(V1)]TJ /F10 7.97 Tf 6.59 0 Td[(hp;V1)]TJ /F6 7.97 Tf 6.58 0 Td[((h)]TJ /F6 7.97 Tf 6.58 0 Td[(1)p]j(t1(1)).Eachrhcanbedeterminedfromn1.Then,itisstraightforwardtoverify 11 thatrk0prk0)]TJ /F4 5.978 Tf 5.76 0 Td[(1rk0)]TJ /F4 5.978 Tf 5.76 0 Td[(1:::r1r122mIjn16=rk0prk0)]TJ /F4 5.978 Tf 5.76 0 Td[(1rk0)]TJ /F4 5.978 Tf 5.76 0 Td[(1:::r1r122mIjn2.Therefore,n16=n2anditfollowsthat[0;U](t1TI(1))6=[0;U](t2TI(2)). Theprecedingtwopropositionsthusimplythefollowinglemma. Lemma7. IfT:Fm!Siscausal,time-invariantandresettable,thenitcanbeeectedbyanacyclicnetworkofdepthtwo. 10 Whichinn1andn2encodesthenumberofspikesintheinterval(V2)]TJ /F7 11.955 Tf 9.87 0 Td[(k0p;V2)]TJ /F1 11.955 Tf 9.87 0 Td[((k0)]TJ /F1 11.955 Tf 9.87 0 Td[(1)p]onthejthspiketrainof1and2respectively.11 Theexpressiononeithersideoftheinequalityisarealnumberthatencodesforthepthspiketimeinthespiketrains(V1)]TJ /F10 7.97 Tf 6.59 0 Td[(k0p;V1)]TJ /F6 7.97 Tf 6.59 0 Td[((k0)]TJ /F6 7.97 Tf 6.58 0 Td[(1)p]j(t1(1))and(V2)]TJ /F10 7.97 Tf 6.59 0 Td[(k0p;V2)]TJ /F6 7.97 Tf 6.59 0 Td[((k0)]TJ /F6 7.97 Tf 6.58 0 Td[(1)p]j(t2(2))respectively. 75

PAGE 76

Lemma 6 and 7 implythefollowingtheorem.Fromthisitfollowsthat,withtheabstractmodelcurrentlyunderconsideration,thesetofallacyclicnetworksisnotmorecomplexthanthesetofacyclicnetworksofdepthtwo. Theorem3. IfT:Fm!Scanbeeectedbyanacyclicnetwork,thenitcanbeeectedbyanacyclicnetworkofdepthtwo. Corollary2. Thesetofallacyclicnetworksisnotmorecomplexthanthesetofacyclicnetworksofdepthequaltotwo. 6.4DirectionsforFurtherConstrainingtheAbstractModelThemeticulousreaderwouldhaveobservedthattheproofaboveexploitedanumberofphenomenathattheabstractmodelallowedfor,butarenotparticularlybiologicallywell-founded.Thesearegoodcandidatesfornewaxiomsthatwehopewillovercomethe\depthbarrier"thatthissectiondescribes.Oneisthatspike-timesintheabstractmodelarerealnumbers,i.e.numberswithinniteprecision.Whennoiseistakenintoaccount,thisassumptionisnolongertrue.Theothereasyobservationisthatwedidnotassumemuchaboutthemembranepotentialfunctionhere,whichinnature,changessmoothlywithtime.And,nally,anassumptionconsistentwithDale'sprinciple,thateachneuronhaseitheranexcitatoryeectatallitspostsynapticneuronsoraninhibitoryeectcouldalsohelpinthisdirection. 6.5SomeAuxiliaryPropositions Proposition4. IfT:Fm!Sistime-invariant,thenT(~m)=~. Proof. Forthesakeofcontradiction,supposeT(~m)=~x0,where~x06=~.Thatis,thereexistsat2Rwitht~x0=hti.Let<.Clearly,(~m)=~m2Fm.SinceT:Fm!Sistime-invariant,T((~m))=(T(~m))=(~x0).Now,(~x0)6=~x0since(t)]TJ /F10 7.97 Tf 6.58 0 Td[()(~x0)=ht)]TJ /F7 11.955 Tf 12.13 0 Td[(iwhereas(t)]TJ /F10 7.97 Tf 6.59 0 Td[()~x0=~,forotherwise~x0=2S.Thisisacontradiction.Therefore,T(~m)=~. Corollary3. LetT:Fm!Sbecausal,time-invariantandW-resettable,forsomeW2R+.If2Fmhasagapintheinterval[t;t+W],thentT()=~. 76

PAGE 77

Proof. Assumethehypothesisoftheabovestatement.OnereadilyseesthattT()=[t;1)(;t]T().Now,sincehasagapintheinterval(t;t+W]andT:Fm!SisW-resettable,wehave[t;1)(;t]T()=[t;1)T((;t]).Further,sincet=~m,wehave[t;1)(;t]=[t;1)~m.Therefore,sinceT:Fm!Siscausal,itfollowsthat[t;1)T((;t])=[t;1)T(~m)=~,withthelastequalityfollowingfromthepreviousproposition. Proposition5. LetT:Fm!Sbecausal,time-invariantandW0-resettable,forsomeW02R+.ThenforallW2R+withWW0,t1;t22Rand1;22Fmwith0t1(T(1))6=0t2(T(2)),thereexistV1;V22R+sothatthefollowingaretrue. 1. [0;V1](t1(1))6=[0;V2](t2(2)) 2. (V1;V1+W](t1(1))=~m,V1(t1(1))6=~mand(V2;V2+W](t2(2))=~m,V2(t2(2))6=~m 3. Forallt012[0;V1),(t01;t01+W](t1(1))6=~mandforallt022[0;V2),(t02;t02+W](t2(2))6=~m. Proof. SinceT:Fm!Siscausal,wehave[t1;1)T(1)=[t1;1)T([t1;1)1).Thisimpliest1([t1;1)T(1))=t1([t1;1)T([t1;1)1))whichgivesus[0;1)t1(T(1))=[0;1)t1(T([t1;1)1)).SinceT:Fm!Sistime-invariantandt1([t1;1)1)=[0;1)t1(1)2Fm,wehave[0;1)t1(T([t1;1)1))=[0;1)T([0;1)t1(1)).Inshort,[0;1)t1(T(1))=[0;1)T([0;1)t1(1))whichimplies0t1(T(1))=0T([0;1)t1(1)).Similarly,0t2(T(2))=0T([0;1)t2(2)).Therefore,itfollowsfromthehypothesisthat0T([0;1)(t1(1)))6=0T([0;1)(t2(2))).LetV1;V22R+bethesmallestpositiverealnumberssothat[0;1)(t1(1))and[0;1)(t2(2))havegapsintheintervals(V1;V1+W]and(V2;V2+W]respectively.ThatsuchV1;V2existfollowsfromthefactthat1;22Fm.Since,T:Fm!SisW0-resettable,itisalsoW-resettableforWW0.Itthereforefollowsthat(;V1]T([0;1)(t1(1)))=T((;V1][0;1)(t1(1)))whichequalsT([0;V1](t1(1))).Thisimpliesthat0(;V1]T([0;1)(t1(1)))=0T([0;V1](t1(1)))duetowhichwe 77

PAGE 78

have0T([0;1)(t1(1)))=0T([0;V1](t1(1))).Likewise,0T([0;1)(t2(2)))=0T([0;V2](t2(2))).Wethereforehave0T([0;V1](t1(1)))6=0T([0;V2](t2(2))).Thisreadilyimplies[0;V1](t1(1))6=[0;V2](t2(2))and,fromtheconstruction,itfollowsthat(V1;V1+W](t1(1))=~m,V1(t1(1))6=~mand(V2;V2+W](t2(2))=~m,V2(t2(2))6=~m,forotherwiseV1orV2wouldnotbethesmallestchoiceofnumberswiththesaidproperty.Furthermore,forthesamereasons,forallt012[0;V1),(t01;t01+W](t1(1))6=~mandforallt022[0;V2),(t02;t02+W](t2(2))6=~m. 78

PAGE 79

CHAPTER7COMPLEXITYCLASSES,TRANSFORMATIONHIERARCHIESANDHIERARCHYCLASSESInthissection,wedevelopsomeadditionaltheoreticaltoolsusefulinprovingcomplexityresults.TheideaistostudythespaceofallpossibletransformationsoftheformT:Fm!SthatmapspikecongurationsofordermwhichsatisfytheFlushcriteriontooutputspiketrains.Specically,weareinterestedinaskinghowthesubsetoftransformationsspannedbycertainsetsofnetworksarerelatedtosubsetsofthespacethatcanbecharacterizedbyspecicmathematicalpropertiesoftransformations.Whiletheformertypesofsubsetsarerelatedtonetworksofneurons,thelattertypearerelatedtotransformationsaloneandmathematicalpropertiesthereof.Moreconcretely,usingmathematicalpropertiesofthetransformations(i.e.withoutanyreferencetonetworks),weidentifyasequenceofsubsetsofthisspace,witheachsubsetcontainedinthesubsequentoneinthesequence.Wethenattempttorelatesetsinthis\hierarchy"ofsubsetstosetsoftransformationsspannedbyspecicnetworkarchitectures.Wecallthissequenceofsubsetsa\TransformationHierarchy"andsubsetsoftransformationsspannedbyspecicnetworkarchitecturestobethe\ComplexityClasses"ofthosearchitectures.Thereareatleasttworeasonsfortryingtorelatecomplexityclassestotransformationhierarchies.First,complexityclassesthemselvesseemtobehardtocharacterizesuccinctlyintermsofpropertiesoftransformationstheycontain.Insteadwetrytounderstandcomplexityclassesofnetworkarchitecturesrelativetothesesetsinthehierarchywhichareeasiertocharacterizeusingmathematicalpropertiesoftransformations.Wedothisbyndingthe\smallest"setinthehierarchythatcontainsthecomplexityclassinquestionasasubset.Thissetiscalledthe\HierarchyClass"ofthearchitecturewithrespecttothehierarchyinquestion.Figure 7-1 providesanillustration.Thesecondreasonforthisapproachisthatitprovidesusanother,andpossiblymorewholesale,waytoprovethatonesetofnetworksismorecomplexthananother. 79

PAGE 80

Figure7-1. VennDiagramillustratingthebasicnotionsofatransformationhierarchy,complexityclassandhierarchyclass. Thesenotionsaremadeprecisenext.AsthereaderfamiliarwithTheoreticalComputerSciencewillundoubtedlyobserve,theapproachhereisreminiscentoftheoneinComputationalComplexityTheory. 7.1DenitionsandPreliminariesLetFmbethespaceofallpossibletransformationsoftheformT:Fm!SthatmapspikecongurationsofordermwhichsatisfytheFlushcriteriontooutputspiketrains.Eachacyclicnetworkoforderminducesonesuchtransformation.Asetofnetworksofordermthereforeinducesaclassofsuchtransformations,whichwecallthe\complexityclass"ofthatset. 80

PAGE 81

Figure7-2. VennDiagramshowingcomplexityclasseswhenonesetofnetworksismorecomplexthananother. Denition18(ComplexityClass). Letbeasetofacyclicnetworksoforderm.The\complexityclass"of,C,isdenedtobethesetSN2TNjFm.AsisclearfromLemma 6 ,nocomplexityclassspanstheentirespaceFm.Lemma 5 impliesthatquestionsofrelativecomplexityofsetsofnetworkscanbeposedintermsofquestionsaboutcontainmentoftheircomplexityclasses.Figure 7-2 illustratesthesituationandthenextpropositionformalizesit. Lemma8. Let1and2betwosetsofacyclicnetworks,eachnetworkbeingoforderm,with12.Further,letC1andC2bethecorrespondingcomplexityclasses.Then,2ismorecomplexthan1ifandonlyifC1C2. Proof. First,suppose2ismorecomplexthan1.ThatC1C2followsimmediatelyfromthefactthat12.Lemma 5 impliesthat9N022suchthat8N21;TN0jFm6=TNjFm.Thatis,TN0jFm2C2andTN0jFm=2C1.SinceC1C2,itfollowsthatC1C2. 81

PAGE 82

Toprovetheotherdirection,assumeC1C2.Therefore,9T:Fm!SsuchthatT2C2andT=2C1.BydenitionofC2,9N22,sothatTNjFm=T.SinceT=2C1,thedenitionofC1impliesthat8N021,TN0jFm6=T.Therefore,2ismorecomplexthan1. Next,wemakeprecisethenotionofaTransformationHierarchy.Informally,aTransformationHierarchyisasequenceofsubsetsofthisspace,witheachsubsetcontainedinthesubsequentoneinthesequence. Denition19(TransformationHierarchy). A\TransformationHierarchy"HinFmisasequenceofsubsetshH1;H2;:::;Hi;:::;FmiofFmwithHiHi+1;8i=1;2;:::.Notethattheabovedenitionexistsindependentoftheexistenceofnetworks.Thatis,ahierarchyisdenedonlyintermsofpropertiesoftransformationsinitsconstituentsets.Figure 7-3 providesanillustration.Thenextdenitionprovidesaconnectionbetweensetsofacyclicnetworksandhierarchies.Eachsetofacyclicnetworksisassociatedwithaspecicsetinthehierarchycalleditshierarchyclass,sothatthehierarchyclassisthesmallestsetinthehierarchythatcontainsthecomplexityclassof. Denition20(HierarchyClass). Letbeasetofacyclicnetworks,eachoforderm,andletH=hH1;H2;:::;Hi;:::;FmibeatransformationhierarchyinFm.The\HierarchyClass"HofinHisthesetHiwithCHiandC6Hi)]TJ /F6 7.97 Tf 6.59 0 Td[(1,ifsuchanHiexistsandFmotherwise.NotethattheHierarchyclassofasetofnetworksiswell-dened.ThisisbecauseCFm,sothereisatleastonesetinthehierarchythatcontainseverycomplexityclass.Also,thesetsinthehierarchyarewell-ordered 1 duetowhicheverycollectionofsetsfromthehierarchy(whichcontainC)hasasmallestset. 1 Theorderingisbysetinclusion. 82

PAGE 83

Figure7-3. VennDiagramillustratingatransformationhierarchy. Theabovedenitionsallowustocreateavarietyofhierarchiesbasedonspecicpropertiesoftransformations.Ifwecanthensaysomethingaboutthehierarchyclassesofspecicarchitecturesineachhierarchy,itenablesustogetabetterunderstandingofvariousaspectsofthetransformationseectedbynetworkswiththisarchitecture.Evenifwecannotidentifythehierarchyclassofagivenarchitectureinahierarchy,provingbounds 2 onthemmightgiveussomeinsight.Also,asalludedtobefore,theseboundscanbeusedtoestablishthatonesetofnetworksismorecomplexthananother.Asetinahierarchyisan\upperbound"onahierarchyclassifitcontainsthehierarchyclassasasubset.Likewise,asetinahierarchyisa\lowerbound"onahierarchyclassifthehierarchyclasscontainsthesetasasubset. 2 Again,formally,theboundsarewithrespecttothepartialorderinginducedbysetinclusion. 83

PAGE 84

Figure7-4. VennDiagramdemonstratinghowupperboundsandlowerboundsonhierarchyclassesinatransformationhierarchycanbeusedtoestablishcomplexityresults. Boundsonhierarchyclassesofspecicsetsofnetworkscanbeusedtoestablishcomplexityresults.Iftherearetwosetsofnetworks,therstcontainedinthesecondandifanupperboundonthehierarchyclassoftherstsetis\smaller"thanalowerboundonthehierarchyclassofthesecond(withrespecttothesamehierarchy),thenthesecondsetismorecomplexthantherst.Figure 7-4 givesapicture.Notethatthisisjustasucientcondition,notanecessaryone,foronesettobemorecomplexthantheother.Dependingonthehierarchyinquestion,itispossiblethatbothsetshavethesamehierarchyclass,yetoneismorecomplexthantheother.Thenextlemmaformalizestheaboveobservations. Lemma9. Let1and2betwosetsofacyclicnetworks,eachcomprisingnetworksoforderm,with12.Furthermore,letH1andH2bethecorrespondinghierarchyclassesinatransformationhierarchyH=hH1;H2;:::;Hi;:::;FmiinFm.Moreover,let 84

PAGE 85

HubeanupperboundonH1andHlbealowerboundonH2.IfHuHl,then2ismorecomplexthan1. Proof. LetC1andC2bethecomplexityclassesof1and2respectively.Byhypothesis,C1HuandHlC2.Since,HuHl,wehaveC1C2.FromLemma 8 ,itnowfollowsthat2ismorecomplexthan1. Indeed,thissuggestsaneconomicalwaytoprovecomplexityresults,sincetheupperboundsandlowerboundscouldapplytoseveralsetsofnetworks.Inthenextsection,weapplythesenotionstoexplicitlyconstructatransformationhierarchyandprovesomelowerboundsforsomearchitecturesaccordingtothehierarchy. 7.2LowerBoundsontheHierarchyClassesofsomeArchitecturesinacertainTransformationHierarchyInthissection,weconstructaspecicsequenceofsubsetsofFmandshowthattheyconstituteatransformationhierarchy.Next,weestablishsomelowerboundsonthehierarchyclassesofsomearchitecturesinthishierarchy.Toprovethatacertainsetinahierarchyisalowerboundonahierarchyclass,itsucestoshowatransformationthatisnotintheset,yetisinthecomplexityclassinquestion.Westartobydeningaclassoftransformationsparameterizedbyapositiveinteger.Forthesakeofexposition,wewillstartbydeninga\First-orderTransformation"whichwewillthengeneralizetoa\kth-orderTransformation".Wewillthenshowthatforalljieveryithordertransformationisalsoajthordertransformation.Intuitively,arst-ordertransformationhastheavorofanSRM0neuronmodel,inthateachsynapsehasa\kernel"functionsuchthateectsofinputsspikesaccordingtothiskernelaresummedoverallinputspikesacrossallsynapses.Thetransformationprescribesanoutputspikeifandonlyifthissumequalsacertain\threshold",whichisapositivenumber. 85

PAGE 86

Denition21(First-orderTransformation). AtransformationT:Fm!Sissaidtobea\First-orderTransformation"ifthereexistsa2R+andfunctionsfj:R!R,for1jm,sothatforevery2Fmandt2R,wehavetT()=htiifandonlyifwehavemj=1lji=1fj(xij)=,where[0;1)t()=h~x1;:::;~xmiwith~xj=hx1j;x2j;:::xljji,for1jm.Informally,akth-ordertransformationisageneralizationofarst-ordertransformationwithhigher-dimensionalkernelfunctions.Thus,asecond-ordertransformation,forexample,hasfunctionsthattakeeverypairofspikesandadduptheir\eects",inadditionto\rst-order"eects. Denition22(kth-orderTransformation). AtransformationT:Fm!Sissaidtobea\kth-orderTransformation"ifthereexistsa2R+andfunctionsfj1:R!R,fj1j2:R2!R,...,fj1j2:::jk:Rk!R,with1jpm,where1pk,sothatforevery2Fmandt2R,wehavetT()=htiifandonlyifwehavemj1=1lji1=1fj1(xij)+mj1=1mj2=1lji1=1lji2=1fj1j2(xi1j1;xi2j2)+:::+mj1=1:::mjk=1lji1=1:::ljik=1fj1:::jk(xi1j1;:::;xikjk)=,where[0;1)t()=h~x1;:::;~xmiwith~xj=hx1j;x2j;:::xljji,for1jm.Forallji,everyith-ordertransformationisalsoajth-ordertransformation.Thiscanbeseenbysettingthevalueofallthefunctionswhosedomainhasdimensionalitygreaterthanitobezeroeverywhere.Therefore,forallji,thesetofallith-ordertransformationsisasubsetofthesetofalljth-ordertransformations.ThisnaturallyinducesatransformationhierarchyinFm Proposition6. Fork2Z+,letOkmFmbethesetofallkth-ordertransformationsoftheformT:Fm!S.Then,Om=hO1m;O2m;:::;FmiisatransformationhierarchyinFm.Forcertainacyclicnetworkarchitectures,wenowestablishsomelowerboundsontheirhierarchyclassesintheabove-mentionedtransformationhierarchy. Theorem4. Let2bethesetthesetofallnetworkswiththearchitectureofthenetworkinFigure 7-5 .ThenO12isalowerboundonthehierarchyclassof2inO2. 86

PAGE 87

Figure7-5. Diagramdepictingarchitectureofnetworksin2. Proof. WeprovethatO12isalowerboundonthehierarchyclassof2inO2byshowingatransformationthatanetworkin2caneect,butwhichliesoutsideO12.Forthesakeofbrevity,wedescribethesalientresponsesoftheneuronsfromwhichitisstraightforwardtoconstructanSRM0modelforthem.Forthesakeofcontradictionassumethatthetransformationeectedbythenetworkisarst-ordertransformation.InFigure 7-5 ,neuronN12isaninhibitoryneuronandtheneuronN0isanexcitatoryneuron.Boththeinputspiketrainsprovideexcitatoryinputtobothneurons.Weassume,forthesakeofcontradiction,thatthepotentialoftheoutputneuronN0canbewrittendownasarst-ordertransformations.Theargumentismadeontwoinputspikesthatoccurredt1andt2secondsagointherstandsecondinputspiketrainrespectively.Considerthevaluesofthefunctionsf1(t1)andf2(t2),wheref1()andf2()arecomponentfunctionsoftheputativerst-orderfunctioninthetransformation.TheneuronN0issetupsothatitproducesaspikenow,ifaspikehappenseitheratt1aloneort2alone.Sincethetransformationisrst-order,thisgivesustwoequationsf1(t1)=andf2(t2)=.Whenthereisaspikebothatpositionst1andt2,N0wouldreachthresholdearlier.However,theoccurrenceofboththesespikescausesN12tospike.This,inturn,causesaninhibitoryeectonthemembranepotentialofN0,whichcompensatesfortheextraexcitation,sothatitspikesexactlyonce,now.Wethereforehavetheequationf1(t1)+f2(t2)=.Thesethreeequations(inthetwovariablesf1(t1)andf2(t2))formaninconsistentsystemoflinearequations,andthereforef1(t1)andf2(t2)donotexist,contradictingourhypothesis.Thereforethetransformationinducedbythecurrentnetworkisnota 87

PAGE 88

Figure7-6. Diagramdepictingarchitectureofnetworksin3. rst-ordertransformation.Thus,O12isalowerboundonthehierarchyclassof2inO2. Next,weapplyasimilarstrategytoderivealowerboundonthehierarchyclassofanothernetworkarchitecture.Itwillthenbeclearhowonecangeneralizethepresenttechnique. Theorem5. Let3bethesetthesetofallnetworkswiththearchitectureofthenetworkinFigure 7-6 .ThenO23isalowerboundonthehierarchyclassof3inO3. Proof. Asbefore,weprovethisbyshowingatransformationthatanetworkin3caneect,butwhichliesoutsideO23.Theargumentismadeonthreespikepositionst1,t2andt3inthepast,andthevaluesofthefunctionsf1(t1),f2(t2),f3(t3),f12(t1;t2),f23(t2;t3)andf31(t3;t1),whicharecomponentfunctionsoftheputativesecond-ordertransformation.Again,N0issetupsoitspikesoneachofthethreeindividualspikesoccurringalone.Thisgivesustheequationsf1(t1)=,f2(t2)=andf3(t3)=.N12worksexactlyasinthepreviousexample,andsodoN23andN31,soastomaketheoutputneuronspikewhenevereverypairofspikesoccur.Thisgivesustheequationsf1(t1)+f2(t2)+f12(t1;t2)=,f2(t2)+f3(t3)+f23(t2;t3)=andf3(t3)+f1(t1)+f31(t3;t1)=.Now,whenspikes 88

PAGE 89

occursimultaneouslyatallthreetimes,theinhibitionprovidedbyN12,N23andN23causesthemembranepotentialofN0toalwaysstaybelowthreshold.TheneuronN123nowprovidesenoughexcitationtoN0,inordertomakeitspikenow.Thisgivesustheequationf1(t1)+f2(t2)+f3(t3)+f12(t1;t2)+f23(t2;t3)+f31(t3;t1)=.Itisstraightforwardtoverifythatthissystemof7equationsin6variablesisinconsistent.Therefore,O23isalowerboundonthehierarchyclassof3inO3. Itisstraightforwardtoobtainsimilarresultsforhigher-ordertransformationswiththistechnique.Whileinthepreviouschapter,we,ineect,upper-boundthecomplexityclassofthesetofalltransformationseectedbyanacyclicnetwork,muchworkremainstobedoneinthisdirection. 89

PAGE 90

CHAPTER8CONCLUSIONANDFUTUREWORK 8.1ContributionsWeconcludebysummarizingourcontributionsinthisdissertation.Webeganbyndingthatacyclicnetworkscannot,ingeneral,beviewedasspike-traintospike-traintransformations.Thispessimisticoutlookwasthensurmountedbytheobservationthatunderspikingregimesusuallyobservedinthebrain,thisviewofacyclicnetworksasspike-traintospike-traintransformationsisinfactwell-founded.Indeed,werigorouslyformulatedpreciseconditionswhichweprovedfromrstprinciples.Thissignicantlyclariesandformalizestheinformalnotionoftenadvancedintheliteraturethatnetworksofneuronsarejustentitiesthattransformspiketrainstospiketrains.Wethendirectedourattentiontothequestionof,ifnetworkarchitectureconstrainsthetypeoftransformationsthatthesenetworkscando.Tothisend,wedenednotionsofrelativecomplexityandderivedausefultechnicallemmathatsignicantlyeasesthetaskofansweringtheaforementionedquestions.Finally,weprovedcomplexityresultsinvolvingsomeclassesofarchitecturesthatsuggestthefeasibilityofthisapproach.Next,weshowedthatthecurrentabstractmodelcannotrevealcomplexityresultsfornetworksbeyonddepthtwo.Weestablishedthisbyprovingthateveryacyclicnetworkhasanequivalentacyclicnetworkofdepthtwo,sothatboththenetworkseectexactlythesametransformation.Wethendevelopedsomeadditionaltheoreticaltoolsbystudyingthespaceofallspike-traintospike-traintransformations.Wedenednotionsoftransformationahierarchiesinthisspaceandrelatedthesetoftransformationsspannedbysomenetworkarchitecturestosetsinahierarchy.Notonlyaretheresultsthemselvesoftheoreticalsignicance,webelievethatintheseproofslietheseedsofcomputationinthebrain;thatis,thisforcesustodirectlyaddressquestionsaboutwhichpropertiesofneuronsarecrucialforwhichpropertiesofnetworksandhowlocalpropertiesofneuronsconstrainglobalnetworkbehavior. 90

PAGE 91

ThecurrentworkalsoassumesaddedimportanceinlightofrecenteortssuchastheHumanConnectomeProject,fundedbytheNIH,tomaptheneuronalcircuitryofthehumanbrain.Sincealsomeasuringthedynamicalpropertiesofeachoftheindividualneuronsappearstobeoutofreachofcurrentexperimentaltechnology,theneedfortheoreticaltechniquesagnostictodynamicalpropertiesthatcanhelpanalyzeandinterpretthisdatacannotbeoveremphasized.Thecurrentworkoerspromiseofprogresstowardssuchagoal. 8.2DirectionsforFutureWorkDuetotheresultsofChapter6,therstorderofthingsistoexplorenewaxiomstobeaddedtothecurrentabstractmodel,sothatcomplexityresultsbeyondthosefornetworksofdepthtwoemerge.Wehaveseveralcandidateaxiomsinmind.Forexample,thepresentmodeldoesnotassumethatthemembranepotentialchangessmoothlywithtime,eventhoughthiswouldbeabiologicallyreasonableassumptiontomake.Theotheroptionistobringnoiseintothepicture.Thiswouldalsonecessitatechangingthedenitions,foronewouldthenhavetomakeprobabilisticstatementsratherthandeterministiconesthatwehaveconsideredhere.Oncethisisdoneandwehavealargebodyofcomplexityresultsinourarsenal,wewouldliketoinvestigatetowhatextenttheseresultscanaordusinsightsintothelargerclassofrecurrentnetworks.Heretheproblemappearsmuchharderandthereforemoreinteresting.Inparticular,wearecurrentlyexploringwaysofaskingsimilarquestionsofrecurrentnetworkstoo.And,nally,oncewehavearoughsenseofwhattherightquestionsareandwhattheanswersmightbelike,weseektoreturntosquareone,andbeginonceagainwithaviewtotreatingbiologicalphenomenawhicharenotpresentlytreated.Synapticplasticity,forexamplewouldbeaprimecandidatetotreatinthenewmodel. 91

PAGE 92

8.3EpilogueWhenwebeganworkonthisdissertation,theideawastostartwithahighlyemaciatedabstractmodelofaneuronandseehowmuchstructureonecouldextractoutofasmallnumberofaxioms.Thiswasnotjustanexerciseinmathematicalvanity,butaconsequenceofthelargevarietyofneuronsandtheirmultifariousabilities;indeed,themoreoneassumesaboutneurons,thesmallertheclassofbiologicalneuronsonecovers.Theinitialattackledtothecounterexamplethatevenasingleneuroncannotbeconsistentlyviewedasaspike-traintospike-traintransformation.Whilethisseemedtobeasetback,wewerefortunatetohitupontheideathatledtotheGapLemmathatshowedthatbyconsideringarestrictedsubsetofspiketrains,whichluckilywasalsobiologicallywell-motivated,wecouldinfactredeemtheintuitiveviewofneuronsandindeedacyclicnetworksasaspike-traintospike-traintransformations.Thataneedforsuchabiologicallywell-foundedassumptionemergedfromthetheorywasveryencouraging.Thisgaveusalotoffaithintheexistenceofmuchstructurewhichwethensoughttouncover.Inhindsight,althoughweshowthattherearelimitstowhatwecouldinferfromthisemaciatedmodel,thatwecouldshowsomuchstructure,inouropinion,showsgreatpromiseinthistypeofaxiomaticapproachtoquestionsaboutnetworksofneurons.Whilethequestionswesoughttoanswerherewererelativelynarrow,thisexperienceleadsustobelievethatsuchanapproachcouldbeofvalueinaskingothertypesofquestionsaswell.Wethereforeendwithanoteofoptimismandhopethatthereaderwilljoinusinunearthingthemarvelsthatlieahead. 92

PAGE 93

REFERENCES Banerjee,A.(2001a).Onthephase-spacedynamicsofsystemsofspikingneurons.I:Modelandexperiments.NeuralComp.,13(1),161{193. Banerjee,A.(2001b).ThePhase-SpaceDynamicsofSystemsofSpikingNeurons.Ph.D.thesis,RutgersUniversity. Banerjee,A.(2006).Onthesensitivedependenceoninitialconditionsofthedynamicsofnetworksofspikingneurons.J.Comput.Neurosci.,20(3),321{348. Bear,M.F.,Conners,B.W.,&Paradiso,M.A.(2007).Neuroscience:Exploringthebrain.Baltimore,MD:LippincottWilliams&Wilkins,3rded. Bohte,S.,Kok,J.,&LaPoutre,H.(2002).Error-backpropagationintemporallyencodednetworksofspikingneurons.Neurocomputing,48(1-4),17{37. Dayan,P.,&Abbott,L.F.(2005).TheoreticalNeuroscience:ComputationalandMathematicalModelingofNeuralSystems.Cambridge,MA:MITPress. Gerstner,W.,&Kistler,W.(2002).Spikingneuronmodels:Singleneurons,populations,plasticity.NewYork,NY:CambridgeUniversityPress. Goldman,D.E.(1943).Potential,ImpedanceandRecticationinMembranes.J.Gen.Physiol.,27,37{60. Gutig,R.,&Sompolinsky,H.(2006).Thetempotron:aneuronthatlearnsspiketiming{baseddecisions.Nat.Neurosci.,9(3),420{428. Hodgkin,A.,&Huxley,A.(1952a).Aquantitativedescriptionofmembranecurrentanditsapplicationtoconductionandexcitationinnerve.J.Physiol.(Lond.),117,500{544. Hodgkin,A.,&Huxley,A.(1952b).CurrentscarriedbysodiumandpotassiumionsthroughthemembraneofthegiantaxonofLoligo.J.Physiol.(Lond.),116,449{472. Hodgkin,A.,&Huxley,A.(1952c).ThecomponentsofmembraneconductanceinthegiantaxonofLoligo.J.Physiol.(Lond.),116,473{496. Hodgkin,A.,&Huxley,A.(1952d).ThedualeectofmembranepotentialonsodiumconductanceinthegiantaxonofLoligo.J.Physiol.(Lond.),116,497{506. Hodgkin,A.,A.L.Huxley,&Katz,B.(1952).Measurementofcurrent-voltagerelationsinthemembraneofthegiantaxonofLoligo.J.Physiol.(Lond.),116,424{448. Koch,C.(2004).BiophysicsofComputation:InformationProcessinginSingleNeurons(ComputationalNeuroscienceSeries).NewYork,NY:OxfordUniversityPress. Maass,W.(1995).Vapnik-Chervonenkisdimensionofneuralnets.InM.Arbib(Ed.)Thehandbookofbraintheoryandneuralnetworks,(pp.1000{1003).Cambridge,MA:MITPress. 93

PAGE 94

Maass,W.(1996).Lowerboundsforthecomputationalpowerofnetworksofspikingneurons.NeuralComp.,8(1),1{40. Minsky,M.,&Papert,S.(1969).Perceptrons:AnIntroductiontoComputationalGeometry.Cambridge,MA:MITPress. Nirenberg,S.,Carcieri,S.,Jacobs,A.,&Latham,P.(2001).Retinalganglioncellsactlargelyasindependentencoders.Nature,411(6838),698{701. Poirazi,P.,Brannon,T.,&Mel,B.(2003).Pyramidalneuronastwo-layerneuralnetwork.Neuron,37(6),989{999. Rall,W.(1960).Membranepotentialtransientsandmembranetimeconstantofmotoneurons.Exp.Neurol.,2(5),503{532. Rieke,F.,Warland,D.,vanSteveninck,R.,&Bialek,W.(1997).Spikes:exploringtheneuralcode.Cambridge,MA:MITPress. Rosenblatt,F.(1988).Theperception:aprobabilisticmodelforinformationstorageandorganizationinthebrain,(pp.89{114).Cambridge,MA:MITPress. Rumelhart,D.E.,Hinton,G.E.,&Williams,R.J.(1986).Learninginternalrepresenta-tionsbyerrorpropagation,(pp.318{362).Cambridge,MA:MITPress. Shepherd,G.(2004).Thesynapticorganizationofthebrain.NewYork,NY:OxfordUniversityPress. Squire,L.R.,Bloom,F.E.,McConnell,S.K.,Roberts,J.L.,Spitzer,N.C.,&Zigmond,M.J.(2003).Fundamentalneuroscience.London:AcademicPress,2nded. Strehler,B.,&Lestienne,R.(1986).Evidenceonprecisetime-codedsymbolsandmemoryofpatternsinmonkeycorticalneuronalspiketrains.Proc.Nat.Acad.Sci.USA,83(24),9812. Tuckwell,H.C.(1988).IntroductiontoTheoreticalNeurobiology.Cambridgestudiesinmathematicalbiology,8.NewYork,NY:CambridgeUniversityPress. 94

PAGE 95

BIOGRAPHICALSKETCH VenkatakrishnanRamaswamywasborninBombay,Indiain1982.Aftergoingthroughschoolinginsevendierentschools,hereceivedhisundergraduatedegreeinInformationTechnologyfromtheUniversityofMadras,Indiain2004,graduatingFirstClasswithDistinction.HegraduateswithaPh.D.fromtheUniversityofFloridainMay2011. 95