<%BANNER%>

A Novel Conic Section Classifier with Tractable Geometric Learning Algorithms

Permanent Link: http://ufdc.ufl.edu/UFE0024624/00001

Material Information

Title: A Novel Conic Section Classifier with Tractable Geometric Learning Algorithms
Physical Description: 1 online resource (106 p.)
Language: english
Creator: Kodipaka, Santhosh
Publisher: University of Florida
Place of Publication: Gainesville, Fla.
Publication Date: 2009

Subjects

Subjects / Keywords: boundaries, class, classifier, concept, conic, constraints, design, discriminant, eccentricity, evaluation, geometry, large, learning, machine, margin, nonlinear, sections, stiffness
Computer and Information Science and Engineering -- Dissertations, Academic -- UF
Genre: Computer Engineering thesis, Ph.D.
bibliography   ( marcgt )
theses   ( marcgt )
government publication (state, provincial, terriorial, dependent)   ( marcgt )
born-digital   ( sobekcm )
Electronic Thesis or Dissertation

Notes

Abstract: Several pattern recognition problems in computer vision and medical diagnosis can be posed in the general framework of supervised learning. However, the high-dimensionality of the samples in these domains makes the direct application of off-the-shelf learning techniques problematic. Moreover, in certain cases the cost of collecting large number of samples can be prohibitive. In this dissertation, we present a novel concept class that is particularly designed to suit high-dimensional sparse datasets. Each member class in the dataset is assigned a prototype conic section in the feature space, that is parameterized by a focus (point), a directrix (hyperplane) and an eccentricity value. The focus and directrix from each class attribute an eccentricity to any given data point. The data points are assigned to the class to which they are closest in eccentricity value. In a two-class classification problem, the resultant boundary turns out to be a pair of degree 8 polynomial described by merely four times the parameters of a linear discriminant. The learning algorithm involves arriving at appropriate class conic section descriptors. We describe three geometric learning algorithms that are tractable and preferably pursue simpler discriminants so as to improve their performance on unseen test data. We demonstrate the efficacy of the learning techniques by comparing their classification performance to several state-of-the-art classifiers on multiple public domain datasets.
General Note: In the series University of Florida Digital Collections.
General Note: Includes vita.
Bibliography: Includes bibliographical references.
Source of Description: Description based on online resource; title from PDF title page.
Source of Description: This bibliographic record is available under the Creative Commons CC0 public domain dedication. The University of Florida Libraries, as creator of this bibliographic record, has waived all rights to it worldwide under copyright law, including all related and neighboring rights, to the extent allowed by law.
Statement of Responsibility: by Santhosh Kodipaka.
Thesis: Thesis (Ph.D.)--University of Florida, 2009.
Local: Adviser: Vemuri, Baba C.
Local: Co-adviser: Banerjee, Arunava.
Electronic Access: RESTRICTED TO UF STUDENTS, STAFF, FACULTY, AND ON-CAMPUS USE UNTIL 2010-02-28

Record Information

Source Institution: UFRGP
Rights Management: Applicable rights reserved.
Classification: lcc - LD1780 2009
System ID: UFE0024624:00001

Permanent Link: http://ufdc.ufl.edu/UFE0024624/00001

Material Information

Title: A Novel Conic Section Classifier with Tractable Geometric Learning Algorithms
Physical Description: 1 online resource (106 p.)
Language: english
Creator: Kodipaka, Santhosh
Publisher: University of Florida
Place of Publication: Gainesville, Fla.
Publication Date: 2009

Subjects

Subjects / Keywords: boundaries, class, classifier, concept, conic, constraints, design, discriminant, eccentricity, evaluation, geometry, large, learning, machine, margin, nonlinear, sections, stiffness
Computer and Information Science and Engineering -- Dissertations, Academic -- UF
Genre: Computer Engineering thesis, Ph.D.
bibliography   ( marcgt )
theses   ( marcgt )
government publication (state, provincial, terriorial, dependent)   ( marcgt )
born-digital   ( sobekcm )
Electronic Thesis or Dissertation

Notes

Abstract: Several pattern recognition problems in computer vision and medical diagnosis can be posed in the general framework of supervised learning. However, the high-dimensionality of the samples in these domains makes the direct application of off-the-shelf learning techniques problematic. Moreover, in certain cases the cost of collecting large number of samples can be prohibitive. In this dissertation, we present a novel concept class that is particularly designed to suit high-dimensional sparse datasets. Each member class in the dataset is assigned a prototype conic section in the feature space, that is parameterized by a focus (point), a directrix (hyperplane) and an eccentricity value. The focus and directrix from each class attribute an eccentricity to any given data point. The data points are assigned to the class to which they are closest in eccentricity value. In a two-class classification problem, the resultant boundary turns out to be a pair of degree 8 polynomial described by merely four times the parameters of a linear discriminant. The learning algorithm involves arriving at appropriate class conic section descriptors. We describe three geometric learning algorithms that are tractable and preferably pursue simpler discriminants so as to improve their performance on unseen test data. We demonstrate the efficacy of the learning techniques by comparing their classification performance to several state-of-the-art classifiers on multiple public domain datasets.
General Note: In the series University of Florida Digital Collections.
General Note: Includes vita.
Bibliography: Includes bibliographical references.
Source of Description: Description based on online resource; title from PDF title page.
Source of Description: This bibliographic record is available under the Creative Commons CC0 public domain dedication. The University of Florida Libraries, as creator of this bibliographic record, has waived all rights to it worldwide under copyright law, including all related and neighboring rights, to the extent allowed by law.
Statement of Responsibility: by Santhosh Kodipaka.
Thesis: Thesis (Ph.D.)--University of Florida, 2009.
Local: Adviser: Vemuri, Baba C.
Local: Co-adviser: Banerjee, Arunava.
Electronic Access: RESTRICTED TO UF STUDENTS, STAFF, FACULTY, AND ON-CAMPUS USE UNTIL 2010-02-28

Record Information

Source Institution: UFRGP
Rights Management: Applicable rights reserved.
Classification: lcc - LD1780 2009
System ID: UFE0024624:00001


This item has the following downloads:


Full Text

PAGE 1

1

PAGE 2

2

PAGE 3

3

PAGE 4

FortheuninchingsupportandmeticulousguidanceIreceivedfrommyadvisersProf.BabaC.Vemuri,andDr.ArunavaBanerjeethroughoutmylonggraduatestudy,Iamextremelygrateful.IoweittoDr.AnandRangarajanforthebroadexposureIgainedthroughhiscourseoeringsonvision,learningandshapeanalysis.IwouldliketothankDr.JereyHoforbeingveryforthcominginsharinghisinsightswhenIapproachedhimfornewideas.IalsothankDr.StephenBlackbandnotonlyforagreeingtobeonmycommittee,butalsoforhisvaluableinputsbeforeandaftermydissertationproposal,andduringtheEpilepsydiagnosisproject.IexpressmysinceregratitudetotheDepartmentofComputer&InformationSciences&Engineering,UniversityofFloridaandtheNIHsponsorsforgenerouslysupportingmygraduatestudies.ThisresearchwasinpartfundedbyNIHgrantRO1NS046812toProf.Vemuri.IalsoacknowledgethetravelgrantfromtheGraduateStudentCouncilatUF.AttheCenterforVision,GraphicsandMedicalImaging(CVGMI),Ihadtheprivilegeofworkinginahighspiritedresearchlabenvironment.IthankformerandcurrentCVGMIlabmembers:EricSpellman,HongyuGuo,FeiWang,NicholasLord,BingJian,AdrianPeter,AngelosBarmpoutis,O'NeilSmith,AjitRajwade,OzlemSubakan,RitwikKumar,andGuangCheng,forbeingveryapproachableingeneral,andenthusiasticwhenitcomestodiscussingstuatlength.Fortheircamaraderie,empathyandsupport,IthankmyfriendsVenu,Sudhir,Suri,Slash,Bala,Ashish,Shiva,Ashwin,Teja,Tapasvi,Jayendra,Muthu,Vishak,ArunandKalyan.SpecialthanksareextendedtoRaviJampaniforhisinvaluablefriendshipandofcourse,forproofreadingmydrafts. 4

PAGE 5

5

PAGE 6

page ACKNOWLEDGMENTS ................................. 4 LISTOFTABLES ..................................... 9 LISTOFFIGURES .................................... 10 ABSTRACT ........................................ 11 CHAPTER 1INTRODUCTION .................................. 12 1.1SupervisedLearning .............................. 13 1.1.1FeaturesforClassication ....................... 14 1.1.2ConceptLearning ............................ 16 1.1.3Generalization .............................. 17 1.1.4ModelSelection ............................. 18 1.1.5Application ............................... 22 1.2ConceptClasses ................................. 23 1.2.1Bayesianlearning ............................ 23 1.2.2FisherDiscriminant ........................... 24 1.2.3SupportVectorMachines ........................ 24 1.2.4KernelMethods ............................. 26 1.3OutlineoftheDissertation ........................... 27 1.4Summary .................................... 29 2CONICSECTIONCLASSIFIER .......................... 30 2.1Motivation .................................... 30 2.2Synopsis ..................................... 31 2.3TheConicSectionConceptClass ....................... 32 2.4LearningOverview ............................... 37 2.5RelatedWork .................................. 39 2.6Conclusions ................................... 41 3THEGEOMETRYOFCONSTRAINTS ...................... 42 3.1GeometricPrimitives .............................. 42 3.2IntersectionsofHyperspheres 45 3.3IntersectionsofShells 46 6

PAGE 7

................. 49 4.1LearningAlgorithm ............................... 49 4.1.1FindingClass-Eccentricitieshe1;e2i 50 4.1.2LearningMisclassiedPoints ...................... 51 4.1.3UpdatingTheFocus .......................... 52 4.1.4UpdatingTheDirectrix ......................... 55 4.1.5Initialization ............................... 56 4.1.6Discussion ................................ 57 4.2Results ...................................... 57 4.2.1Classiers ................................ 58 4.2.2SyntheticData ............................. 58 4.2.3EpilepsyData .............................. 58 4.2.4ColonTumor .............................. 59 4.2.5SheeldFaceDB ............................ 60 4.2.6CURETTextures ............................ 60 4.3SummaryandConclusions ........................... 61 5LEARNINGWITHLARGEMARGINPURSUIT ................. 62 5.1LearningAlgorithmOverview ......................... 62 5.2TheMarginComputation ........................... 63 5.2.1Overview ................................. 64 5.2.2SpanningtheDiscriminantBoundaryG 65 5.2.3NearestPointonGwithFixedDirectrixDistances .......... 65 5.2.4NearestPointonGwithFixedFocalDistances ........... 67 5.2.5LargeMarginPursuit .......................... 69 5.3Experiments ................................... 71 5.3.1EvaluatingMarginComputation .................... 71 5.3.2ClassicationResults .......................... 72 5.4Summary .................................... 74 6LEARNINGWITHINEQUALITYCONSTRAINTS ............... 75 6.1TheLearningAlgorithm ............................ 75 6.1.1LearninginEccentricitySpace ..................... 77 6.1.2ConstructingNullSpaces 80 6.1.3LearningMisclassiedPoints ..................... 87 6.1.4Initialization ............................... 89 6.1.5Discussion ................................ 90 6.2Experiments ................................... 90 6.2.1Datasets ................................. 91 6.2.2ClassiersandMethods ......................... 92 6.2.3ConicSectionClassier ......................... 93 6.2.4ClassicationResults .......................... 93 6.3Summary .................................... 95 7

PAGE 8

.................................... 97 APPENDIX:NOTATION ................................. 99 REFERENCES ....................................... 100 BIOGRAPHICALSKETCH ................................ 106 8

PAGE 9

Table page 3-1GeometricObjects .................................. 43 4-1ClassicationcomparisonsofCSC .......................... 61 5-1Accuraciesoflargemarginpursuits ......................... 71 5-2Detailsofdatausedinexperiments ......................... 72 5-3ClassicationcomparisonsofCSCwithlargemarginpursuit ........... 73 6-1Detailsofdatausedinexperiments ......................... 91 6-2ClassicationcomparisonsofCSCwithinequalityconstraints .......... 94 6-3ParametersforresultsreportedinTable 6-2 .................... 95 9

PAGE 10

Figure page 1-1Challengesinvolvedinsupervisedclassication ................... 13 1-2Connectionbetweenlearning,generalization,andmodelcomplexity ....... 18 2-1ConicSectionsin2D ................................. 33 2-2OverviewoftheConicSectionconceptclass .................... 34 2-3Discriminantsduetodierentcongurationsofclassconicsections ....... 35 2-4DiscriminantboundariesinR2forK=2 ...................... 36 2-5DiscriminantboundariesinR3forK=2 ...................... 36 3-1GeometricObjectsin1D,and2D .......................... 43 3-2FiberBundle ..................................... 44 3-3Intersectionoftwospheres .............................. 45 3-4Intersectionoftwoshells 47 4-1AlgorithmtoLearntheDescriptorsduetoEqualityConstraints ......... 50 4-2Classicationinecc-Space 51 4-3Intersectionoftwohyperspheres 53 4-4DeformationfeaturesintheEpilepsyData ..................... 59 4-5TextureData ..................................... 60 5-1LinearsubspaceHinwhichdirectrixdistancesareconstant ........... 66 5-2ThediscriminantboundaryinH 68 5-3MarginComputationAlgorithm ........................... 70 6-1AlgorithmtoLearntheDescriptorsduetoInequalityConstraints ........ 77 6-2Classicationinecc-Space 78 6-3Crosssectionoftwointersectinghyperspheres 82 6-4Cross-sectionoftwointersectingshells 85 6-5Cross-sectionofthedonut-liketoroidalNullSpace 87 10

PAGE 11

11

PAGE 12

12

PAGE 13

Challengesinvolvedinsupervisedclassication Inthischapter,webrieydiscussthevariousstagesandchallengesinvolvedinthedesignandapplicationofasupervisedlearningtechnique.WealsooutlineimportantclassicationtechniquesthatareusedtocomparewiththeCSC. 1 ].Inatypicalscenario,severalthousandsofgenesaresimultaneouslymonitoredforexpressionlevelseitherindierenttissuetypesdrawnfromthesameenvironmentorofthesametissuetypedrawnfromdierentenvironments.Asubsetofthesegenesarethenidentiedbasedontherelationoftheirexpressionlevelstothepathology.Thelearningproblemnowinvolvesidentifyingthepathologytypeofasample,givenitsgene-expressionlevelsastheinputfeaturesandasetoftrainingsampleswiththeirknownoutcomes.Moreinnocuousinstancesofproblemsthatcanbeposedinthisframeworkincludetheautomatedfacerecognition[ 2 ],diagnosisofvariouskindsofpathologies,forinstance,typesofEpilepsy[ 3 ]fromMRIscansofthebrain,andvariousformsofCancer[ 4 ]frommicro-arraygeneexpressiondata. 13

PAGE 14

1-1 .Inthisdissertation,weintroduceanovelclassierandhencefocusonthelearningaspectslistedinthegure. 5 ].However,ifthedistributionofintensitiesisobservedtobedierentbetweentheclasses,onecanusehistogramsortparametricdistributionstothedata.Infact,suchfeaturesareinvarianttoimagerotationandscale.Ifwefurtherlearnthateachimagehasonlyoneobjectofinterest,itcanthenbesegmented(intoforegroundandbackground)andfeaturescanbeobtainedfromtheobjectboundary.Inthismanner,themorediscriminablefeaturesweextract,theeasieritbecomestoclassifythem.Asurveyoffeatureextractionmethodsispresentedin[ 6 ].Generallyspeaking,toextractthebestfeatures,weneedtocharacterizethemanifoldonwhichthegivendataliesandemployrelatedmetricsandoperations.Inmostcases, 14

PAGE 15

7 ],rulebasedlearning[ 8 ]andstring-matchingmethodswithedit-distancesandkernelsdenedappropriatelyfortextdata. 9 ].Candidatefeaturesaretypicallyrankedbasedontheircorrelationtothetargetlabels.ThemeasureslikePearsoncorrelationcoecient[ 10 ],andmutualinformation[ 11 ],[ 12 ]areutilized.Asetoffeatureswithleadingranksarechosenforclassication,whiletherestaredeemedirrelevant.Inbio-medicalapplications,p-values(signicanceprobabilities)areusedtoperformfeatureselectiondespitestrongobjections[ 13 ],[ 14 ].Itcouldbeverymisleadingtousep-valuesinthiscontext,astheycannotbeconsideredasevidenceagainstthenullhypothesis[ 15 ].Especially,thenotoriousp<:05signicancelevelisnotagoldstandard,butisdatadependent.Certainsubsetsoffeaturescanhavemorepredictivepowerasagroup,asopposedtodiscardingthembasedontheirindividualranks.Thesubsetscanbescoredfortheirdiscriminatingabilityusinganygivenlearningmachine[ 16 ].Featuresubsetselectioncanalsobecombinedintothelearningprocess.Forinstance,theweightvectorinthelinear 15

PAGE 16

17 ],largemargincomponentanalysis[ 18 ],etc.Thecriteriaforprojectionaredatadelityandclassdiscriminability.TheUglyDucklingtheoremstatesthattherearenoproblem-domainindependentbestfeatures[ 19 ].Thusfeatureselectionhastobedomain-specicinordertoimproveperformanceofthelearningalgorithm.Forrecentliteratureonfeatureselection,wereferthereadertoaspecialissueonfeatureselection[ 20 ]. 21 ].Formulatedasabove,thelearningproblemisstillill-posedsincethereexistuncountablymanyfunctions,f's,thatyieldzeroerror.Tomakethisawellposedproblem,onefurtherrestrictsthef'stoaparticularfamilyoffunctions,sayF,knownastheconceptclass.Thebestmemberofthatclass,f2F,isidentiedastheonethatresultsinminimalexpectedclassicationerror,denedas: (1{1) 16

PAGE 17

1-2 .Generalizabilityoftheclassiersuerswhenthelearningclassieriseithertoosimpletopartitiontheinputspaceintotwoclassregions,orwhenitbecomestoocomplexandlearnsnoiseaswell.Now,weconsidersomecandiatecriteriathatcouldminimizethegeneralizationerror.Fisher'scriterion,foratwo-classcases,isdenedastheratioofdistancebetweentheclass-meansoverthesumofvarianceswithineachclass.Thegoalistoofndadirection,onwhichtheprojecteddatapointshavethemaximalseparationbetweenclassesand 17

PAGE 18

Connectionbetweenlearning,generalization,andmodelcomplexity,formemberclassiersofanygivenconceptclassingeneral.Thex-axisrepresentsmembersoftheconceptclassorderedbythecomplexityoftheirdiscriminants.Generalization(erroronunseendata)suerswhenthelearningmodeliseithertoosimpletocapturethediscriminabilityorwhenitbecomestoocomplex,therebylearningnoiseaswell.IllustrationisadaptedfromHastie,Tibshirani,&Friedman[ 22 ] minimalvariancewithineachclass.Aclassierisconsideredstableifitslabelassignmentremainsunchangedw.r.t.smallperturbationsinitsparametersand/orinthedatapoints.Thisintuitionisoneamongstmanytojustifypursuitoflargemargins,bothfunctionalandgeometricsoastondtheclassierwiththebestpossiblegeneralizationcapacitywithinaconceptclass.ThenotionofshatteringthetrainingsetcanbeusedtoboundRg[f],asexplainedintheSection 1.1.4.1 23 ].Further,noclassierissuperiorto(orinferiorto)anuniformlyrandomclassier,sinceeachisgaugedbyitsperformanceonunseendata.Thisistruewhentheperformanceisuniformlyaveraged 18

PAGE 19

24 ].InAkaikeInformationCriterion(AIC),BayesianInformationCriterion(BIC),andStochasticComplexity(SC),thetermpenalizingcomplexityisafunctionofthenumberoftrainingsamplesandthenumberofconceptclassparameters.Inthecontextofclassication,thesearenotpracticaleither.Inthissection,wereviewmodelselectionusingVCdimension,andcross-validationtechniques. 25 ].Inthisdissertation,whenwedealwithhighdimensionalsparsedata,withNM,ahyperplane(WTX+b=0)canhaveaVCdimensionof(M+1).ItdoesnotmeanthateveryeverycongurationofNsamplescanbeshattered.AtrivialcongurationtoconsideristheXORcongurationof4pointsinR2thatcannotbeseparatedbyalineardiscriminant.Thesamesetofpoints,whenmappedintoRMunderananetransform,continuetoremainnon-separable. 19

PAGE 20

25 ]. 24 ]: h(log(2N=d)+1)log(=4) 26 ],[ 24 ],whereinthemargincanbemaximizedbyplacingthepointsonaregularsimplexwhoseverticeslieonthesphere.Theactualproofispresentedin[ 27 ].ThismakestheVC-dimensionofthelargemarginhyperplanealmost 20

PAGE 21

28 ].Astratied10-foldCVisrecommended,especiallyforhigh-dimensionalsparsedatasets,asitensureseverysampleistestedonce,andcouldresultinreasonablecompromisebetweenerrorbiasandvariance.Thebestparametersforaconceptclassarechosenasthosethatyieldtheleastestimatedgeneralizationerror. 29 ]doesprovideformalboundsforthegeneralizationerrorofaclassier(i.e.,theexpectederroronasyetunseendata)asa 21

PAGE 22

30 ].However,learningproceduresthatusecross-validationtendtooutperformrandomguessing[ 31 ],asisthecaseinseveralsuccessfulapplications.Thecounterexamplesoftentendtobedegeneratecases,whichdonotrepresentthesamplesgenerallyseeninlearningapplications. 10 ])andspecicity(falsenegativeorType-IIerror)ratescanbeaccommodatedbyeitherselectingappropriatethresholdsforclassication,orbymodifyingthelearningalgorithmaccordingly.Aclassierfunctionusuallyinovlvesathresholdthatseparatestwoclassesonacontinuousvaluedoutputofadiscriminantfunction.Forinstanceinmedicaldiagnosisandsecurityapplications,lowerfalsenegative(patientclearedashealthy)rates 22

PAGE 23

32 ]curveisaplotofthetruepositiverateversusthefalsepositiverate,obtainedbyvaryingthethresholdforaclassier.Itcouldbeusedtoachieveddesiredpreferencebetweenthetwotypesoferrornotedabove. 8 ]isastatisticalapproachinwhichwecomputetheprobabilityofanunseenpatternbelongingtoaparticularclass(posterior),havingobservedsomelabeleddata.LetP(1);P(2)beaprioriprobabilitymassfunctionsoftwoclassestowhichacontinuousrandomsampleXbelongsto.Giventheclass-conditionalprobabilitydensitiesorlikelihoodsp(Xji),aposterioriprobabilityofXisgivenbytheBayesformula evidence 33 ]ifnecessary.Themodesofvariationwithintheobservedtrainingsampleassistindeningappropriatestatisticalmodels[ 34 ].Insuchscenarios,anaiveBayesclassierisreliableenough.ThelikelihoodsareusuallymodeledasNormaldistributionsinsimplercases,parameterizedbytheclasssamplemeanandcovariancematrix.Theresultant 23

PAGE 24

35 ]isbasedonprojectingdatainRNontoaline,onwhichclassseparationismaximalandvariancewithineachclassisminimal.Giventwosetsoffeaturesf1;2g,thevectorw w Then,theoptimaldirectionofprojectionisw 35 ].ThiscanalsobeinterpretedastheoptimalBayesclassier,whenthelikelihoodsp(X=i)inEq. 1{8 aremodeledasmultivariateNormaldensitieswitheachhavingthemeanmiandacommoncovariantmatrixasSW. 36 ],whoseVCdimensionprovidesalooseboundongeneralizationerror[ 29 ](Eq. 1{5 ).TheconnectionbetweenthemarginandVCdimensionisgiveninEq. 1{6 .Givenatrainingsetofpattern-labelpairshXi;yiiwherei2f1::Ng,Xi2RDandyi2f1;+1g,aseparating(canonical)hyperplaneissoughtsuchthatyi(WTXi+b)18iandthemargin,=1=kWk,ismaximal.Marginhereistheorthogonaldistancebetweenthenearestsamplesineachclassfromthehyperplane,assumingthatthetrainingdataislinearlyseparable.SincethismarginisinverselyproportionaltojjWjj2[ 29 ],wehaveto 24

PAGE 25

37 ]thatinvolvesSequentialMinimalOptimization(SMO).OurcomparisonresultsaredonewiththelibSVM[ 38 ]implementationofSMOfortheSVM.Thesparsityoftheweightvector,W,isusedtopruneoutprobableirrelevantfeaturesinthefeatureselectionphaseoflearningapplications.Recently,thisnotionhasbeenusedinmedicalimageanalysistoidentifydiscriminativelocations(voxels)acrossmultiplesubjectsfromfMRIdata[ 39 ],[ 40 ].Forhighdimensionalsparsedata(NM),allthesamplesendupbeingthesupportvectors,whichdoesnotbodewellforthegeneralizabilityoftheSVM.WebrieyreproducetherecentcommentsonSVMs,byBousquetandScholkopf[ 41 ],thataddressthepopularmisconceptionspersistentinthecommunityofpractitionersatlarge.Statisticallearningtheorycontainsanalysisofmachinelearningwhichisindependentofthedistributionunderlyingthedata.AsaconsequenceoftheNoFreeLunchtheorem,thisanalysisalonecannotguaranteeapriorithatSVMsworkwellingeneral.Amuchmoreexplicitassertionwouldbethatthelargemarginhyperplanescannotbeproventoperformbetterthanotherkindsofhyperplanesindependentlyofthedatadistribution.ThereasonsthatmakeSVMspecialarethefollowing:(i)theuseofpositivedenitekernels,(ii)regularizationusingthenormintheassociatedkernelHilbertspace,and(iii)theuseofconvexlossfunctionthatisminimizedbyaclassier. 25

PAGE 26

1{5 )andthehyperplanemargin(Eq. 1{6 ),theVCdimensionofalargemarginhyperplaneinthefeaturespacebecomesindependentofitsdimensionalityandcouldimproveupongeneralizability.However,itisoftennotpossibletoexplicitlydenesucha(non-linear)mapbetweentheinputspaceandthefeaturespace.Evenifitis,thecomputationalaspectsmightbeaectedbythedimensionalitycurse.Ifallthemathematicaloperationsinvolvedinthelearningalgorithm,oranytaskingeneral,dependonlyontheinnerproductsinthatfeaturespace,itissucienttodeneavalidinnerproductfunction,suchthatK(x;y)(x)T(y).InmostoftheabovelearningtechniquesliketheSVMsandtheFisherdiscriminants,aninnerproductintheEuclideanspaceRDcanbereplacedbyanyvalidkernelfunction,K(x;y),andisreferredtoasthekerneltrick. (MercerKernels[ 21 ])ThefunctionK:XX!RisaMercerkernel,ifandonlyifforeachnaturalnumberNandasetofpointsfXig1:::N,theNNkernelmatrixK,denedasKij=K(Xi;Xj),ispositivesemidenite.Lineardiscriminantsinthefeaturespacearenowequivalenttorichnon-linearboundariesintheoriginalinputspaceRD.Thisisachievedwithlittleornoadditionalcost.Someofthebasickernelslikepolynomial(Eq. 1{11 ),sigmoid(Eq. 1{12 )andradial 26

PAGE 27

1{13 )arelistedbelow,inwhichX;Xj2X. (1{12) (1{13) where;candthedegreedarescalarvaluedkernelparameters.Typically,theunknownhyperplaneorientation(theweightvectorW)ofthelineardiscriminantincanbeexpressedasalinearcombination(Eq. 1{14 )ofmappedinputpatterns. (1{15) Theresultantnon-lineardiscriminantg(X)isgiveninEq. 1{15 ,whereisavectorofunknownweightstobesolvedfor.TheformulationsinSections 1.2.3 1.2.2 canbetransparentlyadaptedtosolveforsuchalinearcombination,optimizingtheirrespectivecriteria.Now,wehaveamodel-selectionproblemgivenahostofkernelsalongwiththeirparameters,eachofwhichisaconceptclassbyitself.TheGaussianRBFkerneliswidelysuccessful,especiallyinapplicationswheretheEuclideandistanceininputspaceismeaningfullocally[ 41 ].Withregardstothepolynomialkernel,itishighlyprobablethattheresultantboundaryasthedegreeisincreasedwillbecomemorenon-linear,eventhoughtheysubsumelowerdegreeboundariesintheory.Inparticular,thiscouldbethecasefordatawithNM,sincetheresultantboundaryisalinearcombinationofdegree-dpolynomialsconstructedfromeachsamplepoint,asK(X;Xj). 2 ,wepresentanovelclassicationtechniqueinwhicheachclassisassignedaconicsectiondescribedbyafocus(point),adirectrix(plane)andan 27

PAGE 28

3 .Assuming,weupdateonedescriptoratatime,thelearningconstraintsaredenedsuchthatthevalueofthediscriminantfunctionisxedatallpointsexceptatonemisclassiedpoint.Thisimpliesthatthelabelsoftheotherpointsdonotchange.Sothealgorithmeitherlearnsamisclassiedpointorworkstowardsit.Hence,thelearningaccuracyisnon-decreasing.Thisapproachleadstoasetofquadraticequalityconstraintsonthedescriptors.InChapter 4 ,weconstructthefeasiblespaceinwhichtheseconstraintsaresatised.Wethenpickasolutioninthisspacethatcanlearnthemisclassiedpoint.InordertoimproveuponthegeneralizabilityoftheCSC,wedescribeamethodinChapter 5 toestimategeometricmargintothenon-linearboundariesduetotheCSC.Thecoreideaistoestimatedistanceofapointtonon-linearboundary,bysolvingfortheradiusofthesmallesthyperspherethatintersectstheboundary.Wefoundthatparticularsectionsoftheboundaryareatconstantdistancestoeachoftheclassdescriptorslikeafocusoradirectrix.Thisfactisusedtoestimatethemarginbycomputingexactdistancestothesesectionsinanalternatingmanner.InChapter 6 ,werelaxthelearningconstraintstopursuelargerfeasiblespacesontheclassdescriptors.First,wecanlimitthelearningconstraintstothecorrectlyclassiedpointsinagiveniteration,insteadofallbutonemisclassiedpoint.Adatapointisassignedtoaclass,amongtwoclasses,basedonthesignofthediscriminantfunctionevaluatedatthatpoint.Next,insteadofconstrainingthediscriminantfunctiontobexedinvalueatcorrectlyclassiedpoints,weconstrainitnottochangeitssign.Thisresultsinquadraticinequalityconstraintsonthedescriptors.Again,thesedescriptorconstraintshaveinterestinggeometrythatcanbeemployedtorepresentsub-regionsofthefeasible 28

PAGE 29

7 includesasummaryofourcontributionswithabriefdiscussionofourfuturework.AlistofsymbolsusedinthisdissertationaregivenintheAppendix. 42 ],Bishop[ 43 ],Haykin[ 44 ],andDuda,Hart,&Stork[ 8 ].Thestatisticallearningtheoryanditsrelatedconceptsaredealtwithindetail,inthetextsbyVapnik[ 29 ],[ 24 ],andHastie,Tibshirani,&Friedman[ 22 ]. 29

PAGE 30

3 ],thediagnosisofvarioustypesofCancerfrommicro-arraygeneexpressiondata[ 1 ],spokenletterrecognition[ 45 ],andobjectrecognitionfromimages[ 46 ],tonameonlyafew.ThesupervisedlearningproblemisseverelyunderconstrainedwhenoneisgivenNlabeleddatapointsthatlieinRMwhereNM.Thissituationariseswheneverthe\natural"descriptionofadatapointintheproblemdomainisverylarge,andthecostofcollectinglargenumberoflabeleddatapointsisprohibitive.Insuchscenarios,learningevenasimpleclassiersuchasalineardiscriminantisunder-constrainedbecauseonehastosolveforM+1parametersgivenonlyNconstraints. 30

PAGE 31

35 ],thelinearSupportVectorMachine(SVM),etc.Inthesecondapproach,thedimensionalityofthedatasetisreducedeitherbyapriorfeatureselection[ 47 ],[ 48 ]orbyprojectingthedataontodiscriminativesubspaces[ 17 ].Thecriterionforprojectionmayormaynotincorporatediscriminabilityofthedata,suchasinPCAversusLargeMarginComponentAnalysis[ 18 ],respectively.Theassumptionunderlyingthesecondapproachisthatthereisasmallersetofcompoundfeaturesthatissucientforthepurposeofclassication.Ourprincipalcontributioninthisdissertation,expandsthepoweroftherstapproachnotedabove,bypresentinganovelconceptclassalongwithatractablelearningalgorithm,wellsuitedforhigh-dimensionalsparsedata. 2.3 .Eachmemberclassinthedatasetisassignedaconicsectionintheinputspace,parameterizedbyitsfocuspoint,directrixplane,andascalarvaluedeccentricity.Theeccentricityofapointisdenedastheratiobetweenitsdistancetoaxedfocus,andtoaxeddirectrixplane.Thefocus,anddirectrixdescriptorsofeachclassattributeeccentricitiestoallpointsintheinputspaceRM.Adatapointisassignedtotheclasstowhichitisclosestineccentricityvalue.TheconceptclassisillustratedinFigure 2-2 .Theresultantdiscriminantboundaryfortwo-classclassicationturnsouttobeapairofpolynomialsurfacesofatmostdegree8inRM,andthushasniteVCdimension[ 25 ].Yet,itcanrepresentthesehighlynon-lineardiscriminantboundarieswithmerelyfourtimesthe 31

PAGE 32

4 5 6 ,wepresenttractablegeometricalgorithmsforbinaryclassicationthatupdatetheclassdescriptorsinanalternatingmanner.Thischapterexplainsthenovelconicsectionconceptclass,introducedin[ 49 ]. 2{1 )thatattributestoeachpointX2RMascalarvaluedeccentricitydenedas: 2-1 ,wedisplayvariousconicsectionsforthefollowingeccentricities:[0.5,0.8,1.0,2.0,6.0,-10,-3,-2-1.7].Ate=0,aconicsectioncollapsestothefocuspoint.Asjej!1,itbecomesthedirectrixplaneitself. 32

PAGE 33

ConicsectionsinR2forvariouseccentricityvalues.Eccentricityistheratioofapoint'sdistancetothefocus,F,overtothatofthedirectrixline Wearenowinapositiontoformallydenetheconceptclassforbinaryclassication.Eachclass,k2f1;2g,isrepresentedbyadistinctconicsectionparameterizedbythedescriptorset:focus,directrix,andeccentricity,asCk=fFk;(bk;Qk);ekg.ForanygivenpointX,eachclassattributesaneccentricity"k(X),asdenedinEq. 2{1 ,intermsofthedescriptorsetCk.Wereferh"1(X);"2(X)iastheclassattributedeccentricitiesofX.Welabelapointasbelongingtothatclassk,whoseclasseccentricityekisclosesttothesample'sclassattributedeccentricity"k(X),asinEq. 2{2 .ThelabelassignmentprocedureisillustratedinFigure 2-2 (2{2) Theresultantdiscriminantboundary(Eq. 2{3 )isthelocusofpointsthatareequidistanttotheclassrepresentativeconicsections,ineccentricity.ThediscriminantboundaryisdenedasGfX:g(X)=0gwhereg(X)isgivenbyEq. 2{3 .The 33

PAGE 34

Overviewoftheconceptclass.Circlesandsquaresaresamplesfromclasses1,&2.Aconicsection,"(X)=e,isthelocusofpointswhoseeccentricityisconstant.Eccentricityisdenedastheratiobetweenapoint'sdistancetoafocuspoint,andtoadirectrixline.Eachclassisassignedaconicsection.Boththeclassesshareacommondirectrixhere.Forapointineachclass,theclassattributedeccentricities,h"1(X);"2(X)i,areshown.Asampleisassignedtotheclasstowhichitisclosestto,ineccentricity,i.e.itbelongstoclass1ifj"1(X)1j
PAGE 35

B C DFigure2-3. DiscriminantboundariesfordierentclassconiccongurationsinR2:(A)Non-linearboundaryforarandomconguration.(B)Simplerboundary:directricesarecoincident.Signofe2isippedtodisplayitsconicsection.(C)Linearboundary:directricesperpendicularlybisectthelinejoiningfoci.(D)Linearboundary:directricesareparallelandeccentricitiesareequal.(SeeSection 2.3 ) 35

PAGE 36

DiscriminantboundariesinR2forK=2 Figure2-5. DiscriminantboundariesinR3forK=2 Dependinguponthechoiceoftheconicsectiondescriptors,fC1;C2g,theresultantdiscriminantcanyieldlowerorderpolynomialsaswell.TheboundariesduetodierentclassconiccongurationsinR2,areillustratedinFigure 2-3 .Whenthenormalstothedirectrices,Q1;Q2,arenotparallel,thediscriminantishighlynon-linear(Figure 2-3A ).Asimplerboundaryisobtainedwhendirectricesarecoincident,asinFigure 2-3B .Weobtainlinearboundarieswheneitherdirectricesperpendicularlybisectthelinejoiningfoci(Figure 2-3C )ordirectricesareparallelandeccentricitiesareequal(Figure 2-3D ).SomesimpletocomplexdiscriminantsurfacesinR2,andR3areillustratedinFigures 2-4 2-5 respectively.AlistofsymbolsusedinthisdissertationaregivenintheAppendix. 36

PAGE 37

37

PAGE 38

4 .Weconstructafeasiblespaceforeachdescriptor(focusordirectrix)relatedtotheconstraintthattheirlabelingofclassiedpointsremainsunchanged.Werepresentthefeasiblespace,referredtoastheNullSpace,asacompactgeometricobjectthatisbuiltincrementallyfromthegivenconstraintsinthelearningphase.Weintroduceastinesscriterionthatcapturestheextentofnon-linearityintheresultantdiscriminantboundary,giventheconicsectiondescriptors.ItisusedtopursuesimplerdiscriminantsbyselectingappropriatesolutionsfromthedescriptorNullSpaces,andtherebyimprovinguponthegeneralizabilityoftheclassier.However,ndingsuchoptimalfoci,anddirectricesisnon-trivialinRM.Hence,weintroducedlearningconstraintsonthesedescriptorssuchthatthedatapointsthatarecorrectlyclassiedinaniterationarenotmisclassiedinfutureiterations.Letbeafocusordirectrixdescriptorofaclassinagiveniteration,thatisbeingupdatedto.First,weareinterestedinndingallsuchthatforeachcorrectlyclassiedpointXi: (2{9) Whenisafocusordirectrix,weobtainEqs. 2{10 2{11 respectively.Here,thescalarvaluesri,andhiarethedistancestothefocus,anddirectrixdescriptorsinthepreviousiteration.Weconstructthefeasiblespaceofthatsatisestheseconstraints,inanincrementalmanner,andrepresentitasatractablegeometricobject.Anyinthefeasiblespacewillnotmis-classifythepointsthatarecorrectlyclassiedinapreviousiteration.Next,wepickoneparticularsolutionfromthisfeasiblespacethatcanlearnamisclassiedpoint.ThelearningalgorithminChapter 4 exploitstheconstraintslistedinEq. 2{9 .InChapter 6 ,wepursuelargerfeasiblespacesbyndingallsuchthatthesignofthediscriminantfunctionremainsunchangedateachcorrectlyclassiedpoint.This 38

PAGE 39

(2{12) ThisleadstoinequalityconstraintsrelatedtothoseinEqs. 2{10 2{11 .Equivalently,eachcorrectlyclassiedpointXirequiresafocusordirectrixtoliewithinacertainintervalofdistancesfromitself.Thegeometryofsuchconstraintsonthefocus,anddirectrixdescriptorsisdiscussedinChapter 3 .Thegeneralizationcapacityoftheclassiercanbeimprovedbypursuinglargegeometricmarginsfromthediscriminantboundary[ 27 ],[ 50 ].InChapter 5 ,weestimatedistancetothenon-lineardiscriminantboundaryduetotheConicSectionClassier(CSC).Eachmisclassiedpointresultsinadesiredupdateforagivendescriptor.Amongthesetofcandidateupdates,weusealargemargincriteriontopickthebestone. 51 ],[ 52 ],andrecoveringconicsfromimages[ 53 ]toinferstructurefrommotion,etc.Theprincipalreasonsforthisusageisthatalargevarietyofcurvescanberepresentedwithveryfewparameters,andthattheconicsectionsarequitecommoninoccurrence.Withinthedomainofsupervisedlearning,thereisoneinstanceinwhichconicsectionswereused.Onecanobtainaconicsectionbyintersectingaconewithaplaneatacertainangle.Theangleisequivalenttotheeccentricity.Whentheangleischanged,dierentconicsectionscanbeobtained.Thisnotionwascombinedwithneuralnetworksin[ 54 ]tolearnsuchanangleateachnode.However,theotherdescriptors,namelythefocus,anddirectrixarexedateachnodeunlikeourapproach.SupportVectorMachines(SVM),andKernelFisherDiscriminant(KFD)withpolynomialkernelalsoyieldpolynomialboundarieslikeourmethoddoes.Thesetofdiscriminantsduetotheseclassierscanhaveanon-emptyintersectionwiththoseduetotheconicsectionconceptclass.Thatis,theydonotsubsumetheboundariesthat 39

PAGE 40

55 ],withpolynomialkernelsastheyappeartoberelatedtoCSCinthetypeofdiscriminantsrepresented.Forboththeclassiers,thediscriminantboundarycanbedenedas: 2{7 )cannotbeexpressedintermsoftheboundarydueto(Eq. 2{13 ).Therecouldbeanon-emptyintersectionbetweenthesetofthepolynomialsurfacesrepresentedbyCSC,andthoseduetoEq. 2{13 .Wepointoutthatthereisnokernelwhichmatchesthisconceptclass,andtherefore,theconceptclassisnovel.KFDseeksboundariesthatmaximizeFishercriterion[ 35 ],i.e.maximizeinter-classseparationwhileminimizingwithinclassvariance.ThelearningcriterionusedinSVMistopursuelargefunctionalmargin,resultingislowerVCdimension[ 27 ],andtherebyimprovinggeneralizability.CSCusessimilarcriterion,andinfactgoesastepfurther.Thedegreeofpolynomialkernelisamodel-selectionparameterinkernelbasedclassierslikeSVM,andKernelFisherDiscriminants(KFD).AsthelearningalgorithminCSCwillinvolvearrivingatsimplerboundariesforthesameconstraintsonthetrainingsamples,thedegreeofpolynomialisalsobeinglearntineect(Figure 2-3 ).TheadvantageofSVMoverCSCisthatlearningintheformerinvolvesaconvexoptimization.TheoptimizationinKFDisreducedtothatofmatrixinverseproblemforbinaryclassication.However,theequivalentnumericalformulationforCSCturnsout 40

PAGE 41

56 ](Section 6.1.2.2 ).WhenSVMdealswithhigh-dimensionalsparsedatasets,mostofthedatapointsendupbeingsupportvectorsthemselves,leadingtoaboutN(M+1)parameters,whereasCSCemploysonly4(M+1)parameters.TheboundaryduetoKFDalsoinvolvessamenumberofparametersasSVM.Insummary,CSChassomeuniquebenetsoverthestate-of-the-arttechniquesthatmakeitworthexploring.Theseincludeincorporatingquasimodel-selectionintolearning,andshorterdescriptionofthediscriminantboundary.WealsofoundthatCSCout-performedSVM,andKFDwithpolynomialkernelinmanyclassicationexperiments. 41

PAGE 42

2{9 istheintersectionofasetofhyperspheres.Inthischapter,wedenethegeometricobjectsofinterestlikehyperspheres,hyperplanes,andregionsbetweenhyperspheres.Wewillalsoelaborateuponconstituentoperationslikeintersections,projections,andsamplingfromthefeasiblespaces.Then,wepresentgeometricintuitionstoincrementallyconstructthesefeasiblespacesforboththefocus,andthedirectrixdescriptors. 2{10 ),anddenedas 3-1 42

PAGE 43

Summaryofthegeometricobjects ObjectHyperplaneHypersphereBallShellDenitionXTQ+b=0kXPk=rkXPkrrlkXPkru Figure3-1. Theobjects:plane,sphere,ball,andshellareillustratedinR(toprow),andR2(bottomrow). 2{12 3-1 ).AnisitsgeneralizationtoRn.InFigure 3-1 ,thesegeometricobjectsinR,andR2areillustrated.TheirgeometryinRisimportanttounderstandcertainintersectionsofinterest.Forinstance,intersections 43

PAGE 44

Aberbundle.Allthebersareisomorphictoeachother.Thebundlecanbeunderstoodasaproductbetweenthebasemanifold,andaberstructure.Theillustrationisinspiredbyagurein[ 57 ]. ofshellscontaintractablesub-regionsthatcanbeexpressedasberbundleproducts.Wepresentthedenitionsforberbundle,givenin[ 57 ]. 3-2 illustratestheintuitionbehindaberbundle.Ashellcanalsobeinterpretedas 44

PAGE 45

BFigure3-3. (A)Intersectionoftwospheresisthecirclelyingtheplaneofintersection.(B)Intersectionofathirdspherewiththehyperplane.Thus,intersectionofthreespheresisreducedtothatoftwocirclesinthehyperplane. aberbundleoflinesegments(bers)overahyperspherebase,i.e. 2{10 )onafocusdescriptor,whenconsideredseparately,turnsouttobeahypersphere.Sincethefocusdescriptorinthepreviousiterationsatisedalltheconstraintsduetocorrectlyclassiedpoints,therespectivehyperspheresareguaranteedtointersect.Inthelimitingcase,theNullSpaceofalltheconstraintsputtogethercollapsestothepreviousfocusdescriptor.Weelaborateuponthegeometryinvolvedassumingthattheintersectionsalwaysexist. 45

PAGE 46

3.2.3 (k1)timestoobtainEq. 3{9 RecallthattheintersectionofkhyperspheresistheNullSpacewithinwhichalltheequalityconstraintsonafocusdescriptoraresatised.Now,theNullSpacecanberepresentedasasinglehypersphereinalowerdimensionalhyperplane.TheanalyticalformulaetocomputetheNullSpacearediscussedinSections 4.1.3.1 6.1.2 3-4 .TheintersectionfortheformercanalsobedenotedastheberbundleS0B2.Similarly,theirintersectioninR3istheberbundleS1B2,referredtoatoroid.Infact,torusistheberbundleS1S1.WewillnowgeneralizethesetoroidalregionslyingwithintheintersectionofshellstoRn. 46

PAGE 47

BFigure3-4. Ashell-shellintersectioninR2,andR3.In(A),wecanalwaysdenetwodiscsthatliewithintheintersection.In(B),onlythecross-sectionoftheshellsisshown.Similarly,wedeneatoroidalregionwithintheintersectiononshells. 3-4 ,wehave:T(0;2)=S0B2S1B1\S1B1T(1;2)=S1B2S2B1\S2B1 47

PAGE 48

1. FixthebasespaceasthehypersphereSn(k1)=\ki=1SnifromTheorem 3.2.4 2. TheberisaballBkwhoselargestradiuscanbedeterminedfromtheupper,andlowerboundsonthedistanceofeachcorrectlyclassiedpointtothenewfocusdescriptor(Eq. 2{12 ).AdetailedproofispresentedinSection 6.1.2 48

PAGE 49

49 ]tolearntheconicsectiondescriptors,Ck=fFk;fbk;Qkg;ekgfork=1;2,thatminimizetheempiricalerror(Eq. 4{1 ).WeassumeasetofNlabeledsamplesP=fhX1;y1i;:::;hXN;yNig,whereXi2RMandthelabelyi2f1;+1g,andthatthedataissparseinaveryhighdimensionalinputspace,i.e.,NM.Learninginvolvesarrivingatappropriateclassdescriptorsthatminimizetheempiricalrisk: 3 ,toarriveatatractablelearningalgorithm.Inouralgorithm,theclassdescriptorsarealternatelyupdatedtosimultaneouslylearnmisclassiedpointsandpursuesimpler(near-linear)boundarieswiththeconstraintthatthediscriminantvalues,g(Xi),forallthedatapointsexceptamisclassiedpointareheldxed. 4.1.5 ).Thelearningprocessisthencomprisedoftwostages.Intherststage,thefocianddirectricesofboththeclassesareheldxedandtheclasseccentricities,he1;e2i,areupdated.EachdatapointXiismappedintothespaceofclassattributedeccentricities,referredtoastheecc-Space,bycomputing,h"1(Xi);"2(Xi)i.Thepairofclasseccentricitieshe1;e2ithatminimizesthe 49

PAGE 50

LabeledSamplesP ConicSectionDescriptorsC1;C21:InitializefF1;b1;Q1g;fF2;b2;Q2g2:Computeh"1(Xi);"2(Xi)i8Xi2P3:Findclass-eccentricitiesh^e1;^e2iiflearningerrorisnotminimalthen4:Computethedesiredh"01i;"02ii5:Updatefoci&directricesalternately.6:Goto(2)untilconvergenceofdescriptors.end AlgorithmtoLearntheDescriptorsduetoEqualityConstraints Inthesecondstage,thefocifF1;F2gandthedirectricesffb1;Q1g;fb2;Q2ggareupdatedalternatelysoastoachievethedesiredattributedeccentricitiesforthosemisclassiedsamples,withoutaectingtheattributedeccentricitiesforthosesamplesthatarealreadycorrectlyclassied.Theprocessisrepeateduntilthedescriptorsconvergeortherecanbenofurtherimprovementinclassication.ThelearningprocedureislistedinFigure 4-1 4-2A )aroundtheoriginby45sothatanychoiceofthediscriminantswillnowbeparalleltothenewaxes,asinFigure 4-2B .Eachaxisisdividedinto(N+1)intervalsbyprojectingthepointsinecc-Spaceontothataxis.Consequently,ecc-Spaceispartitionedinto(N+1)22Dintervals.Wenowmakeacrucialobservation:withintheconnesofagiven2Dinterval,anychoiceofacross-hairclassiesthesetofsamplesidentically.(SeeFigure 4-2B ).Wecanthereforeenumerateallthe(N+1)2intervals 50

PAGE 51

BFigure4-2. (A)Shadedregionsbelongtooneclass.Thepairofdiagonallinesisthediscriminantboundary,j"1(X)e1j=j"2(X)e2j.Learninginvolvesshiftingmisclassiedpointsintodesiredregions.(B)ecc-Spacerotatedby45degrees.Whentheonlyunknownishe1;e2i,labelassignmentisinvariantwithina2Dinterval. andchoosetheonethatgivesthesmallestclassicationerror.Thecross-hairissetatthecenterofthis2Dinterval.Incaseswheretherearemultiple2Dintervalsthatgivethesmallestclassicationerror,thelargeroneischosen. 4-2A )bymovingitintothenearestquadrantassociatedwithitsclasslabel.ThismovementcanbeachievedbyupdatingafocusordirectrixinEq. 2{1 .Inordertokeepthelearningprocesssimple,weupdateonlyonedescriptorofaparticularclassateachiteration.Hence,wemovethemisclassiedpointsinecc-Spacebychanging"1ior"2ifortheclassofthedescriptorbeingupdated.ThelearningtasknowreducestoalternatelyupdatingthefocianddirectricesofC1andC2,sothatthemisclassiedpointsaremappedintothedesiredquadrantsinecc-Space,whilethecorrectlyclassiedpointsremainxed.Notethatwithsuchanupdate, 51

PAGE 52

2{1 wecanconcludethatallthe"kifork=1dependonlyontheclassdescriptorC1,andlikewisefork=2.Sinceweupdateonlyonefocusatatime,weshallhereafterdealwiththecasek=1.Theupdateproblemmaybeposedformallyasfollows.FindafocusF1thatsatisesthefollowingNquadraticconstraints.Letk:k2betheEuclideanL2norm. 4{3 .PcandPmcarethesetofclassiedandmisclassiedpointsrespectively.Theinequalitiesaboveimplythatthedesiredlocation"01icanlieinanintervalalonganaxisinecc-Space(SeeFigure 4-2A ).Inordertocloselycontrolthelearningprocess,welearnonemisclassiedpointatatime,whileholdingalltheothersxed.Thisleavesuswithonlyoneinequalityconstraint.WerefertothesetofallfeasiblesolutionstotheabovequadraticconstraintsastheNullSpaceofF1.Further,wehavetopickanoptimalF1inthisNullSpacethatmaximizesthegeneralizationcapacityoftheclassier.AlthoughthegeneralQuadraticProgrammingProblemisknowntobeNP-hard[ 58 ],theaboveconstraintshaveanicegeometricstructurethatcanbeexploitedtoconstructtheNullSpaceinO(N2M)time. 52

PAGE 53

Intersectionoftwohyper-sphereNullspaces.Thewhitecircleislocusoffocalpointssatisfyingcurrentdistanceconstraints. Notethatbyassumption,thenumberofconstraints,NM.TheNullSpaceofF1withrespecttoeachequalityconstraintinEq. 4{2 isahyper-sphereinRM.Hence,theNullSpaceforalltheconstraintscombinedissimplytheintersectionofallthecorrespondinghyper-spheresinRMwithcentersfX1;:::;XNgandradiifr11;:::;r1Ng.LetXNbethesinglepointbeingupdatedinecc-Space.Then,r1Ncantakeanyvaluewithinaninterval(rmin;rmax)correspondingtotherangeofdesired"01N.Thesolutiontothiscaseispresentednext. 4{2 .Itcanbeparameterizedasthehyper-sphereS1=(r1;X1),wherer1isthedesireddistanceofasolutionfromthesampleX1.Atthenextstep,thesecondequalityconstraintisintroduced,theNullSpaceforwhich,consideredindependently,isthehyper-sphere 53

PAGE 54

4-3 ,theintersectionoftwospheresinR3isacirclethatliesontheplaneofintersectionofthetwospheres.OurtechniqueisbasedonageneralizationofthissettinginRM.Wemaketwocriticalobservations:theintersectionoftwohyper-spheresisahyper-sphereofonelowerdimension,andthishyper-sphereliesontheintersectinghyper-planeoftheoriginalhyper-spheres.Were-parameterizethecombinedNullSpace,S1\S2,asalower-dimensionalhyper-sphereSf1;2g,lyinginthehyper-planeofintersectionHf1;2g.BasedonthegeometryoftheproblemandtheparameterizationofS1andS2,itistrivialtocompute,inO(M),theradiusandcenterofthenewhyper-sphereSf1;2g=(rf1;2g;Xf1;2g),aswellastheintersectinghyper-planeHf1;2grepresentedas(bf1;2g;Q12),therstparameterbeingthedisplacementofHf1;2gfromtheoriginandthesecondbeingtheunitnormaltoHf1;2g.Qf1;2gliesalongthelinejoiningX1andX2.Wenowsolvetheremainderoftheproblemonthehyper-planeHf1;2g.Thisisaccomplishedbyintersectingeachoftheremaininghyper-spheresS3;:;SNthatcorrespondtothesamplesX3;:;XN,withHf1;2g,inO(NM)time.Onceagain,basedonthegeometryoftheproblem,itistrivialtocomputethenewradiiandcentersofthecorrespondinghyper-spheres.Inshort,theintersectionoftheNhyper-spheresproblemisconvertedintotheintersectionof(N1)hyper-spheresandahyper-planeHf1;2gproblem. Theproblemisnowtransparentlyposedinthelowerdimensionalhyper-planeHf1;2gasaproblemequivalenttotheonethatwebeganwith,exceptwithonelesshyper-sphereconstraint.Theendresultofthisiterationforallthe(N1)equalityconstraintsisalowdimensionallinearsubspaceinwhichliestheNullSpaceSerepresentedasasinglehyper-sphere(parameterizedasaradiusandacenter),computedinO(N2M)time.It 54

PAGE 55

2-3D ,resultsinasimplerdiscriminantintheinputspace.IfSedoesnotintersectwithSN,thissimplymeansthatthechosenmisclassiedpointcannotbeshiftedentirelytothedesiredlocationinecc-Space.Insuchacase,wepickedanappropriatesolutiononSethatallowsthemaximumpossibleshift. (4{7) ThisproblemappearssimpleratrstsightsinceitiscomprisedofNlinearconstraints.However,thequadraticconstraintrequiringQ1tobeanunitnormalmakestheaboveaQuadraticProgrammingProblemwhichisagainNP-hardingeneral.Once 55

PAGE 56

4.1.3 ,welearnasinglemisclassiedpoint,sayXN,ineachiteration.Withaknownb1,wetranslateandscaleallremainingpointssuchthatthelinearconstraintsbecome: NowthenullspaceofQ1,foreachconstraintinEq. 4{8 consideredseparately,isahyper-planeHi2RMrepresentedasfv1i;Xig.ThenullspacecorrespondingtothequadraticconstraintonQ1isaunithyper-sphere,S12RM,centeredattheneworigin.Hence,thenalNullSpaceforQ1istheintersectionofalltheHi'sandS1.Wenowmaketwocriticalobservations.Theintersectionofahyper-planewithahyper-sphereisalower-dimensionalhyper-sphere.Sameisthecasewiththeintersectionoftwohyper-spheres.Wecanthereforeconvertthishyperplane-hypersphereintersectionproblemintoahypersphere-hypersphereintersectionproblem.Ineect,wecanreplaceeachhyper-planeHiwithasuitablehyper-sphereSisuchthatHi\S1=Si\S1.Owingtothegeometryoftheproblem,wecancomputeSifromHiandS1.TheNullSpaceforalltheconstraintscombinedisnowtheintersectionofallthehyper-spheresS1;S2;:::;SN.Theproblem,nowreducedtoahyperspheres-intersectionproblem,issolvedasinSection 4.1.3.1 56

PAGE 57

2{1 ,theNullSpacesaresmallorvanishingifthefociordirectricesareveryclosetothesamples.Wefoundthefollowinginitializationtobeconsistentlyeectiveinourexperiments.Thefociwererstplacedatthesampleclassmeansandthenpushedapartuntiltheywereoutsidethesampleclouds.Thenormalstothedirectriceswereinitializedasthelinejoiningthefoci.Thedirectrixplaneswerethenpositionedatthecenterofthislineoroneithersidesofthedata. 3 ],ColonTumorgene-expressiondata[ 4 ],theSheeld(formerlyUMIST)FaceDatabase[ 59 ],CURETTextureDatabase[ 60 ].Theresultswerecomparedagainstseveralstate-of-the-artlinearandnon-linearclassiers.Theclassicationaccuraciesbasedonleave-one-outcross-validationarepresentedinTable 4-1 57

PAGE 58

61 ].Thistechniqueexploitsthefactthathigh-dimensionaldatahassingularscattermatrices.Theydiscardthesub-spacewhichcarriesnodiscriminativeinformation.Kernelmethodsareknowntobeeectivenon-linearclassiersandtheSupportVectorMachines(SVM)[ 36 ]andKernelFisherDiscriminants(KFD)[ 62 ]broadlyrepresentedthenon-linearcategory.BothemploythekerneltrickofreplacinginnerproductswithMercerkernels.Amongthelinearclassiers,wechosetheLinearFisherDiscriminant(LFD)[ 8 ]andlinearSVM.[ 63 ]WeusedtheOSUSVMtoolboxforMATLABbasedonlibSVM[ 38 ].WeconsideredPolynomial(PLY)andRadialBasis(RBF)Kernels.Thebestparameterswereempiricallyexplored.Polynomialkernelsgavebestresultswitheitherdegree=1or2andthescalewasapproximatelythesamplevariance.TheRBFkernelperformedbestwhentheradiuswasthesamplevarianceorthemeandistancebetweenallsamplepairs.TheinitializationofthedescriptorsinthelearningisdescribedinSections 4.1.5 4.1.2 and 4.1.3.1 4-1 validateourclassier'seectivenessonsimple,linearlyseparabledata.Syntheticdataset-2wasgeneratedbysamplingfromtwointersectingparaboloids(relatedtothetwoclasses)inR3andplacingtheminR64.Thisinstanceshowsthatourclassierfavorsdatalyingonparaboloids.Itclearlyout-performedtheotherclassiers. 3 ]consistsofdisplacementvectoreldsbetweentheleftandrighthippocampifor31epilepsypatients.Thedisplacementvectorsarecomputedat762discretemeshpointsoneachofthehippocampalsurfaces,in3D.Thisvectoreldrepresentingthenon-rigidregistration,capturestheasymmetrybetweentheleftand 58

PAGE 59

Directionofdisplacementeldforepilepticswithfocusonleftandrighttemporallobe,respectively.Directioncomponentsin(x;y;z)arecolorcodedwithRed,Green,Bluecomponentsonthehippocampalsurface. righthippocampi.(SeeFigure 4-4 ).Hence,itcanbeusedtocategorizedierentclassesofepilepsybasedonthelocalizationofthefocusofepilepsytoeithertheleft(LATL)orrighttemporallobe(RATL).TheLATLvs.RATLclassicationisahardproblem.AsseeninTable 4-1 ,ourclassierout-performedalltheothersandwithasignicantmargin,exceptoverSVM-RBF.Infact,ourresultisbetterthanthatreportedin[ 3 ].ThebestRBFkernelparametersforSVMandKFDmethodswere600and1000,respectively.Thebestdegreeforthepolynomialkernelwas1forbothofthem. 4 ]comprisesof2000gene-expressionlevelsfor22normaland40tumorcolontissues.Eachgene-expressionscoreiscomputedfromalteringprocessexplainedin[ 4 ].ThebestparametersaredescribedinSection 4.2.1 .ThenormalstodirectrixdescriptorswereinitializedwiththeLFDdirectioninthiscase.Ourclassieryielded87%accuracyoutperformingtheotherclassiers.Interestingly,mostoftheotherclassierscouldnotout-performLFD,implyingthattheywerelearningthenoiseaswell.Terrence,etal.[ 64 ]wereabletocorrectlyclassifytwomoresampleswithalinearSVM,onlyafteraddingadiagonalfactoroftwotothekernelmatrix.Wehavetestedthisdatawithoutanyfurtherpre-processing. 59

PAGE 60

SandPaper,RoughPaper&Polyester.IntheresultsTable 4-1 ,wehaveclassicationfor6vs.12and6vs.2respectively. 59 ]has564pre-croppedfaceimagesof20individualswithvaryingpose.Eachimagehas92x112pixelswith256gray-levels.Sinceweonlyhaveabinaryclassiernow,theaverageclassicationperformanceoverallpossiblepairsofsubjectsisreported.Thisturnedouttobeaneasierproblem.Conicclassierachievedacomparableaccuracyofabout98%,whiletheotherswerenear100%.OurperformanceispoorerthanLFDherebecauseofthetradeobetweenlearningandgeneralization.illustratedinFigure 2-4 60 ]isacollectionof61textureclassesimagedunder205illuminationandviewingconditions.AVarmaetal.[ 65 ]havebuiltadictionaryof601textons[ 66 ]andproposedwaystogeneratefeaturesforclassicationbasedontextontextonfrequenciesinagivensampleimage.Thetextonfrequencyhistogramsobtainedfrom[ 67 ],areusedasthesamplefeaturevectorsforclassication.About47imageswerechosenfromeachclass,withoutapreferentialordersoastodemonstratetheecacyofourclassierforhigh-dimensionalsparsedata.Wereporttheresultsforaneasypairandarelativelytougherpairoftexturesforclassication.ThetwocasesareSandpapervs.Roughpaper(Pair1)andSandpapervs.Polyester(Pair2),respectively.SeeFigure 4-5 tonotehowsimilarthesandpaperandpolyestertexturesare.AsseeninTable 4-1 ,Pair1turnedouttobeeasiercaseindeed.KFDout-performedtheothersforthesecondpairandourclassierfaredcomparably. 60

PAGE 61

ClassicationaccuraciesfortheConicSectionClassier,(Linear&Kernel)FisherDiscriminantsandSVM. SyntheticData120x40100100100100100100SyntheticData232x6493.7587.5757581.2587.5Epilepsy31x228677.4267.7467.7461.2967.7474.19ColonTumor62x200087.185.4875.8182.2682.2685.48UMISTFaceDB575x1030497.7498.7299.9399.9199.399.06TexturePair195x601100100100100100100TexturePair295x60192.6398.9410010090.5282.10 4-1 61

PAGE 62

68 ].Thiscanthenbeusedtocomputeclassiermargin,givenalabeleddataset.ThemargincomputationisincludedinthelearningalgorithmsoastopursuelargemarginCSCandtherebyimproveuponthegeneralizationcapacityoftheclassier.First,webrieydescribetheincrementalConicSectionlearningalgorithmpresentedin[ 49 ],forthetwo-classclassicationprobleminthefollowingsection. 2{3 ).Foreachmisclassiedsample,onecanthenndadesiredpairofattributedeccentricities,denotedash"01i;"02ii,thatwouldcorrectlyclassifyit.Inthesecondstage,thefocusandthedirectrixdescriptorsarealternatelymodiedviaageometricalgorithmtolearnthemap.Themapisconstrainedtoachievethedesiredattributedeccentricitiesforthemisclassiedsamples,withoutaectingtheeccentricitiesofthosesamplesthatarealreadyclassiedcorrectly.Theprocessisrepeated 62

PAGE 63

4-1 .Thelearningrateisnon-decreasingsincethealgorithmdoesnotmisclassifyapreviouslycorrectlyclassiedpoint.Anotablefeatureofthetechniqueisitsabilitytotracktheentirefeasiblesetforadescriptor,labeledasNullSpace,thatwouldmapthedatapointstoxedattributedeccentricities,usingsimplegeometricconstructs. 2.1 .Wedenethemarginastheminimumoverthedistancesofallthecorrectlyclassieddatapoints,tothediscriminantboundary.Inordertocomputethemargin,wersthavetondtheshortestdistanceofapoint,sayP2RM,tothediscriminantboundaryG(Eq. 2{3 ).Whenformulatedasanoptimizationproblem(Eq. 5{1 )withthenon-linearconstraint(Eq. 2{3 ),thedistancecomputationisNP-hard[ 58 ]ingeneral. dist(P;G)=minkPXk2;subjecttog(X)=0 (5{1) AlthoughtheobjectivefunctionisquadraticinX,theequalityconstraintishighlynon-linear.ThenumericalsolutiontoEq. 5{1 thereforeworksouttobeveryexpensive.Notethatwehavetocomputetheshortestdistancefromtheboundarytoallthedatapointstodeterminethemargin.Foreachcompetingdescriptorupdate,wecomputeitsmarginsoastopickanupdatethatyieldsthelargestmargin.Thistaskbecomesespeciallydicultduetotworeasons.First,thediscriminantboundaryisapairof 63

PAGE 64

2{3 ).Existenceofsuchapointpairisguaranteedbyourinitialdiscriminantboundary.AbinarysearchonthelinejoiningthispairofdatapointsgivesusapointZ02Gandhenceanupperbound,sayrp,ontheshortestdistanceofagivenpointPtoG.ConsidernowsectionsoftheboundaryGforwhichthedistancestoeitherboththedirectrixplanesorboththefocalpointsareheldxed.Asaconsequenceofcertaingeometricobservations,weshalldemonstratethattheexistence/lackofintersectionofthehypersphereSwithanysuchsectionofthediscriminantboundarycanbedeterminedanalytically.TheshortestdistanceofapointtothesesectionsofthediscriminantboundarycanbecomputedinO(NM)time.Weproposeaniterativealgorithmthatalternatelyupdatestheshortestdistancestothesesectionssoastondapointonthediscriminantboundary,nearesttoP.AnoverviewofthetechniqueispresentedinFigure 5-3 .Next,wepresentalgorithmstoevaluateG\S(P;r). 64

PAGE 65

2{3 )canalsobewrittenastheimplicitfunction: ((r1e1h1)h2)((r2e2h2)h1)=0; (5{3) (5{4) whererk,andhkaredistancestothefocusanddirectrixdescriptorsrespectively.Inordertoevaluatetheintersection,G\S(P;r),weneedtoascertainifg(X)changessignonthehypersphere,S.Thedistancesrk,andhkareboundednow,sinceX2S(P;r).Asarstpass,wecansearchforsuchachangeinthesignofg(X)byevaluatingg(X)atdiscretizeddistancesinaboundedinterval,foreachvalidcombinationoffh1;h2;r1;r2g.ForanypointX2S,thedistancehk(X)isboundedtobewithin[hk(P)rp;hk(P)+rp]duetoEq. 5{4 .Similarly,rk(Eq. 5{3 )isboundedtobewithin[jrk(P)rpj;rk(P)+rp].IfwediscretizeeachparameteratO(N)locations,thecostofevaluatinganintersectionisO(N4M),whichisexpensive.Insubsequentsections,weintroduceafasterO(M)algorithmtocomputetheintersectionofthehypersphereSwithparticularsectionsoftheboundaryG. 2{3 )inHcanalsobewrittenas:

PAGE 66

5{6 ). SinceHRM2,letustrackthesectionofthediscriminantboundary,G\H.LetX2HandthedistancebetweenafocuspointFkanditsorthogonalprojectionF0kinH,besk=kFkF0kk.Given(h1;h2),thesectionofthediscriminantboundary,G\Hthenbecomes 5-1 illustratesthepointsfX;F01;F02g2H,relatedtoEq. 5{6 forG\H.AnypointP2RMcanbeorthogonallyprojectedintoHwiththeequationsbelow.Thecoecients(u;v)inEq. 5{7 areobtainedbysolvingtheconstraintsforHgiveninEq. 5{8 (5{8) Thus,G\HisfullyspeciedbyelementslyinginH.WearenowinterestedinndingoutifthehypersphereS(P;r)intersectswithG\H.Further,S\HcanberepresentedasahypersphereS02FcenteredattheprojecteddatapointP02Hwithradiusr0,derivedfromEq. 5{7 .Owingtosymmetry,theintersectionbetweenthesectionoftheboundary,G\H,andthehypersphereS0needstobecheckedonlyintheplanecomprisingfP0;F01;F02g.

PAGE 67

5{6 thatthediscriminantboundaryinHisradiallysymmetricaboutthelinejoiningF01andF02.LetbethelengthofthecomponentofXalong(F02F01)andbethelengthofitsorthogonalcomponentintheplanedenedbyfP0;F01;F02g,asillustratedinFigure 5-2 .Letf=kF02F01k.Aftertranslatingtheprojectedorigin,O02HtoF01,theboundarybecomesaquarticpolynomialinthetwovariables(;),asinEq. 5{9 .Also,theequationofthehypersphereS0(P0;r0)reducestothecircle,Eq. 5{10 .Here,(p;p)arethecomponentsofP0alongandacrossthelinejoiningF02andF01. (p)2+(p)2=(r0)2 UponintersectingthetwogeometricobjectsduetoEqs. 5{9 5{10 ,weobtainaquarticequationinaftereliminating.Foranyquarticpolynomialinonevariable,wecancheckfortheexistenceofrealroots[ 69 ]andcomputethemexplicitly[ 70 ],ifnecessary.Thus,wedeterminetheintersection,S\(G\H),inO(M)time.AssumethatwebeginwithahypersphereShavinganinitialradiusrothatisguaranteedtointersectwiththesectionofthediscriminantboundaryinH,i.eG\H.AllthatremainstobedoneistoconductabinarysearchontheradiusofSintheinterval(0;ro]tondtheshortestdistancebetweenPandthediscriminantsurfaceinH.Moreover,wecanexplicitlydeterminethenearestpointonthesectionG\H,sayZ,fromthepolynomialroots(;),andthepointsfP0;F01;F02g. 5{2 ).Thelocusofallthepointsthatareatxeddistancesfr1;r2;rgfromthepointsfF1;F2;Pgrespectively, 67

PAGE 68

ThediscriminantboundaryinthesubspaceHbecomesaquarticin(;)(seeEq. 5{9 ). canbeconstructedbycomputingtheintersection: whereS0isahypersphereinthelinearsubspaceFofco-dimension2inRM.ThiscanbeunderstoodfromananalogueinR3:TheintersectionoftwospheresinR3isacirclelyingwithintheplaneofintersection.AnO(K2M)algorithmwaspresentedin[ 49 ]tocomputetheintersectionofKhyperspheres.Wecomputetheintersectionofthreesphereshere,henceK=3.TheproblemisnowreducedtodeterminingtheintersectionG\S0.AftertranslatingtheorigintoC2F,anypointXinRMcanbeprojectedintoFas: where,fU1;U2garetwoorthonormalvectorsperpendiculartoF.ThesectionofthediscriminantboundaryGinFforagivenpairofxedfocaldistances(r1;r2),nowbecomes: (5{13) whichuponre-arrangementoftermsresultsinaquadraticimplicitsurfaceequationinX0.Eq. 5{13 representsG\FlyinginF.Theintersectionofthedirectrixhyperplane,denoted 68

PAGE 69

5{12 .NowG\FisfullyspeciedbyelementslyinginF.Wemakeacrucialgeometricobservationhere.WithinF,thediscriminantfunction(Eq. 5{13 )isinvarianttotranslationsnormaltotheplanespannedbyfQ01;Q02g.WenowexploitthefactthatG\FisafunctiononlyofthedistancestothedirectrixplanesinthelinearsubspaceF.SinceS0(C;r0)isahypersphere,itissucienttoinvestigatetheintersectionofinterest,i.e.S0\(G\F),intheplanespannedbyfQ01;Q02gandpassingthroughC,thecenterofthehypersphereS0.AnypointXinsuchaplanecanbeexpressedas: ThesectionoftheboundaryG\F(Eq. 5{13 ),inthisplane,reducestoaquadraticcurveinparameters(;).ThehypersphereS0(C;r0)inthisplanebecomesa(quadratic)circle,kX(;)Ck=r0,inparameters(;).Again,theintersectionofthesetwogeometricobjects,obtainedbyeliminating,yieldsaquarticpolynomialin.Wecananalyticallyndifrealrootsexistforagivenquartic[ 69 ]andcomputethemexactly[ 70 ].WedescribedanO(M)algorithmtondiftheintersection,S\(G\F),exists.TheshortestdistanceofapointPtothesectionoftheboundaryG,inwhichthefocaldistancesareconstant,iscomputedviaabinarysearchontheradiusoftheinitialhypersphereS(P;r)withintheinterval(0;r0].Attheendofthebinarysearch,wealsocomputethenearestpointonthesection,sayZ0(;),fromEq. 5{14 5-3 ,wealternatelyndtheshortestdistancetosectionsofthediscriminantboundary,G,withdistancestoeitherthefociordirectricesxed.Tobegin,thexeddistances,(h1;h2),tothedirectricesareobtainedfromtheinitialpointZ0onG.WethencomputethepointZ,thatisnearesttoPinthe 69

PAGE 70

DatapointssetZ,ConicDescriptorsC1;C2 MarginbetweenpointsetZandboundaryG1:FindapointZ0onGforeachPointP2Zdo2:InitialDistancemi=kPZ0k,Z0=Z0repeat3:FindclosestZ2Gforxedfh1(Z0);h2(Z0)g4:FindclosestZ02Gforxedfr1(Z);r2(Z)guntilZconverges5:PointDistancemi=kPZkend6:Margin=minfmig MarginComputationAlgorithm ThealternatingprocessisrepeateduntileitherthedistancetotheboundaryconvergesoranO(N)iterationlimitisreached.ThenumberofstepsinallthebinarysearchesislimitedtoO(N).ThecomplexityofcomputingtheshortestdistanceofapointtotheboundaryGinthismanner,isO(N2M).ThemarginforasetofatmostNpointsandagivenconiccongurationfC1;C2giscomputedinO(N3M)time.Similartothenumericaloptimizationtechnique(Eq. 5{1 ),themargincomputationcouldbepronetolocalminima.However,weobservedthatthecomputationtimesareseveralordersofmagnitudesmallerthanthosefortechniquesinvolvingeitherthenumericaloptimization(Eq. 5{1 )orthediscretizationapproach(Section 5.2.2 ).Inthelearningphase,amongasetofcompetingupdatesforaconicsectiondescriptor,wepicktheoneresultinginthelargestmargin.Wealsoavoidanupdateifitreducesthecurrentmarginwithoutimprovingthelearningaccuracy.Weupdatethedescriptorsinaconstrainedmannersothatonlythecorrectlyclassiedpointsdonotmoveintheecc-Space.ThisapproachensuresthattheNullSpaceforeachdescriptorupdateislarger. 70

PAGE 71

GivenapairofCSCdescriptorsets,theaccuracyofpickingtheonewithlargermarginiscomparedforvaryingboundarytypes.Thelasttwocolumnslisterrorsinmargincomputation. CSCBoundarySelectionMarginErrorTypesAccuracy% 4 ]thathas62sampleswith2000featureseach,andprojecteditintoR5usingawhiteningtransformsothatitscovariancematrixisidentity.Tocomputethetruemargins(Section 5.2.2 ),weperformedbrute-forcesearchforchangeinsignofg(X)onS(P;r)soastodetermineanintersection.Weconsidereddiscriminantboundariesofsuccessivelyhighercomplexity.IntheinitialcongurationfortheresultsinTable 5-1 ,thedirectricesarecoincident,thelinejoiningfociisparalleltothedirectrixnormal,sayQ,andtheclasseccentricitiesarebothzeros.Thisensuresthatthediscriminantboundaryisalwayslinear[ 49 ].Uponmakingtheeccentricitiesunequal,theboundaryturnsintoapairoflinearandnon-linearsurfaces.OncethelinejoiningthefociisnotparalleltoQ,theboundariesbecomenon-linear.Inthelastcase,thedirectricesarenotcoincident,resultinginhighlynon-linearboundaries. 71

PAGE 72

Detailsofdatausedinexperiments DatasetFeaturesSamplesClass1Class2 Epilepsy216441925Isolet-BC6171005050CNS7129602139ColonTumor2000624022Leukemia7129724725HWdigits35649400200200 Foreachboundarytype,20CSCdescriptorswererandomlygeneratedandtheerrorsinmargincomputationarelistedinTable 5-1 .Itcanbeseenthatourapproximationsarereliableforsimplerboundaries.Giveneachpossiblepairofcompetingdescriptors,weverifyifourmethodpickedthecongurationwithlargermargins.Weobservedthatfornear-linearandsimplerboundaries,weselectedthedesiredupdateinmorethan95%oftheinstances.Sincethediscriminantturnsouttobeacollectionofnon-linearsurfacesingeneral,ourmethodispronetolocalminima. 49 ],aswellasotherstate-of-the-artclassierssuchaslinearandkernelSVMs[ 29 ],andFisher[ 35 ],[ 62 ]discriminants.Weusedpolynomialkernelsforthekernelbasedclassiers.Thedegreeofthepolynomialwasempiricallydetermined.Unlessotherwisenoted,alltheclassicationexperimentswereperformedusingaleave-one-outcrossvalidationprotocol.ThediscriminantforCSC-MwasinitializedwiththatgeneratedbyeitherthelinearSVM[ 29 ]orLFD[ 35 ].Inthelearningphase,thecompetingconic-sectiondescriptorupdateswereallowedtoincreasethemarginwithoutlosingtrainingclassicationaccuracy.TheCSCintroducedin[ 49 ],pursuedsimplerboundariesinthelearningphasetoenhancegeneralizability.ThecharacteristicsofdatasetsusedinourexperimentsarelistedinTable 5-2 .Theclassicationresultsforsixhighdimensionalsparsedatasetspertainingtoapplicationsincomputervisionandmedicaldiagnosis,arelistedinTable 5-3 72

PAGE 73

ClassicationaccuraciesfortheConicSectionClassierwithlargemarginpursuit(CSC-M),CSC,(Linear&Kernel)FisherDiscriminantsandSVMswithpolynomial(PLY)kernels.Thedegreeofpolynomialkernelisinparentheses. DatasetCSC-MCSCLFDLin-SVMKFDPLYSVMPLY Epilepsy93.1888.6477.2756.1886.36(1)86.36(6)Isolet-BC92.0084.0081.0091.0091.00(2)91.00(1)CNS70.0073.3351.6768.3365.00(3)68.33(1)ColonTumor87.1087.1085.4880.5675.81(1)82.26(2)Leukemia97.2298.6197.2297.2298.61(1)97.22(1)HWdigits3596.0096.2585.2595.7597.75(3)95.75(1) TheEpilepsydataset[ 71 ]consistsof3DHistogramsofdisplacementvectoreldsrepresentingthenon-rigidregistrationbetweentheleftandrighthippocampiin3D.ThetaskistodistinguishbetweenLeftandRightAnteriorTemporalLobe(RATL)epileptics.Thedatasetincludesfeaturesfor19LATLand25RATLepilepticsubjects.Thefeaturevectorlength,whichdependedonthenumberof3Dbinsused,wasempiricallydetermined.Alltheclassiersperformedwellwitha666binningofthedisplacementeld,exceptKFDforwhichweuseda161616binning.AslistedinTable 5-3 ,CSCwithmarginpursuit(CSC-M)achievedanimpressive93%testingaccuracy,clearlyoutperformingalltheotherclassiers.TheIsolet-BCdatasetisapartoftheIsoletSpokenLetterRecognitionDatabase[ 72 ].[ 45 ],Weconsidered50speechsampleseach,foralphabetsBandCspokenby25speakers.EachsubjectspokethealphabetsBandCtwice,resultingin50samplesforeachalphabet.ThetaskhereistodiscriminatebetweenthespokenalphabetsBandC.Theresultsusing10-foldcross-validationtestingarereportedinTable 5-3 .CSC-Mclassieroutperformedalltheotherclassiers,withalargeimprovementoverCSCwithoutmarginpursuit.TheHWdigits35dataisapartoftheHand-writtenDigitsdataset[ 72 ].Ithas200samplesofeachdigit,andaround649featuresforeachsample.Weclassiedthedigitsthreeandve,astheyturnedouttobeadicultpair.Wereport10-foldcross-validationexperimentsforthesetwodatasets. 73

PAGE 74

73 ]containstreatmentoutcomesforcentralnervoussystemembryonaltumoron60patients,whichincludes21survivorsand39failures.Theexpressionscoresfor7129geneswereobtainedforeachpatient.TheColonTumordata[ 4 ]consistsof2000gene-expressionlevelsfor22normaland40tumorcolontissuefeatures.Eachgene-expressionscoreiscomputedfromalteringprocessexplainedin[ 4 ].IntheLeukemiacancerclassdiscovery[ 1 ],thetaskistodiscriminatebetween25acutemyeloidleukemia(AML)and47acutelymphoblasticleukemia(ALL)samples,given7129gene-expressionprobesforeachsubject.Overall,theperformanceofCSC-Mimprovedsignicantlyoverthatwithoutthemarginpursuit,fortheEpilepsyandIsolet-BCdatasets.CSC-MalmostmatchedCSCintheremainingfourexperimentsinTable 5-3 ,asboththemethodsaresusceptibletoachievinglocaloptimaw.r.t.learningtheoptimaldescriptors.Theperformanceoflinear-SVMiscomparabletothatofCSC-MforIsolet-BC,HWdigits35andLeukemiadatasets,sincethelargemargindiscriminantcanturnouttobenear-linear. 74

PAGE 75

49 ],wetrackedafeasiblespaceforeachdescriptor(focusordirectrix)relatedtotheconstraintduetoholdingthevaluesofthediscriminantfunctionattheclassiedpointstobexed.Inthischapter,weconstructamuchlargerfeasiblespaceforeachdescriptorrelatedtotheconstraintthattheirlabelingofcorrectlyclassiedpointsremainsunchanged.Werepresentthefeasiblespace,referredtoastheNullSpace,asacompactgeometricobjectthatisbuiltincrementallyinthelearningphase,asdescribedinSection 6.1.2 .InSection 6.1.3.1 ,weintroduceastinesscriterionthatcapturestheextentofnon-linearityintheresultantdiscriminantboundary,giventheconicsectiondescriptors.ItisusedtopursuesimplerdiscriminantsbyselectingappropriatesolutionsfromthedescriptorNullSpaces,andtherebyimprovinguponthegeneralizabilityoftheclassier.WedemonstratetheecacyofourtechniqueinSection 6.2 ,bycomparingittowellknownclassierslikeLinearandKernelFisherDiscriminantsandkernelSVMonseveralrealdatasets.OurclassierconsistentlyperformedbetterthanLFD,asdesired.Inthemajorityofcases,itout-performedstate-of-the-artclassiers.WediscussconcludingremarksinSection 6.3 .AlistofsymbolsusedtorepresentdierentgeometricentitiesinthisdissertationaregivenintheAppendix. 75

PAGE 76

6.1.4 .Thesubsequentlearningprocessiscomprisedoftwoprincipalstages.Intherststage,givenxedfocianddirectriceswecomputeattributedeccentricities(Eq. 2{1 )foreachpointXi,denotedash"1i;"2ii.Wethencomputeanoptimalpairofclasseccentricitieshe1;e2i,thatminimizestheempiricalriskLerrinO(N2)time,asdescribedinSection 6.1.1.1 .Forachosendescriptor(focusordirectrix)tobeupdated,wendfeasibleintervalsofdesiredattributedeccentricities,suchthatyig(Xi)>08Xi,i.e.,thesamplesarecorrectlyclassied(Section 6.1.1.2 ).Inthesecondstage,wesolvefortheinverseproblem.Givenclass-eccentricities,weseekadescriptorsolutionthatcausesattributedeccentricitiestolieinthedesiredfeasibleintervals.Thisresultsinasetofgeometricconstraintsonthedescriptor,duetoEq. 2{4 ,thataredealtwithindetailinSection 6.1.2 .WecomputetheentireNullSpace,denedastheequivalenceclassofthegivendescriptorthatresultsinthesamelabelassignmentforthedata.Foreachmisclassiedpoint,wepickasolutioninthisNullSpace,thatlearnsitwithalargemarginwhileensuringasimplerdecisionboundaryinRM.InSection 6.1.3.1 ,weintroduceastinesscriteriontoquantifytheextentofthenon-linearityinadiscriminantboundary.Amongthecandidateupdatesduetoeachmisclassiedpoint,anupdateischosenthatyieldsmaximumstiness;i.e.,minimal 76

PAGE 77

LabeledSamplesP ConicSectionDescriptorsC1;C21:InitializetheclassdescriptorsfFk;fbk;Qkgg;k2f1;2grepeat2:quadComputeh"1(Xi);"2(Xi)i8Xi2P3:Findthebestclass-eccentricitieshe1;e2iforeachdescriptorinfF1;F2;fb1;Q1g;fb2;Q2ggdo4:Determinetheclassifyingrangeforh"1i;"2ii5:Finditsfeasiblespaceduetotheseconstraints.foreachmisclassiedpointXmcdo6:ComputeadescriptorupdatetolearnXmcend7:Pickupdateswithleastempiricalerror8:ThenpickanupdatewithlargestStinessenduntilthedescriptorsC1;C2converge AlgorithmtoLearntheDescriptorsduetoInequalityConstraints non-linearity.ThesecondstageisrepeatedtoupdatethefocifF1;F2gandthedirectricesffb1;Q1g;fb2;Q2gg,oneatatime.Thetwostagesarealternatelyrepeateduntileitherthedescriptorsconvergeortherecanbenofurtherimprovementinclassicationandstiness.Notethatallthroughtheprocess,thelearningaccuracyisnon-decreasingsinceapreviouslyclassiedpointisnevermisclassiedduetosubsequentupdates.ThelearningalgorithmislistedinFigure 6-1 .Wediscusstherstphaseindetailinthefollowingsection. 2{1 ,fromtheinputspaceintoR2,as: 6-2A ,thexandyaxesrepresentthemaps"1(X)and"2(X)respectively,asdenedinEq. 2{1 .Foranygivenchoiceofclasseccentricities,e1ande2, 77

PAGE 78

BFigure6-2. (A)Shadedregionsinthisecc-Spacebelongtotheclasswithlabel1.Thepairofdiagonallinesisthediscriminantboundary,j"1(X)e1j=j"2(X)e2j.Learninginvolvesshiftingmisclassiedpointsintodesiredregions.(B)Thediscriminantboundaryisthepairofthicklines.Withintheshadedrectangle,anychoiceofclasseccentricitydescriptors(thepointofintersectionoftwothicklines)resultsinidenticalclassication. thediscriminantboundaryequivalentinecc-Space,j"1(X)e1jj"2(X)e2j=0,becomesapairofmutuallyorthogonallineswithslopes+1;1,respectively,asillustratedinthegure.Theselinesintersectathe1;e2i,whichisapointinecc-Space.Thelinesdivideecc-Spaceintofourquadrantswithoppositepairsbelongingtothesameclass.Itshouldbenotedthatthisdiscriminantcorrespondstoanequivalentnon-lineardecisionboundaryintheinputspaceRM.Weuseecc-Spaceonlyasameanstoexplainthelearningprocessintheinputspace.Thecrucialpartofthealgorithmistolearntheeccentricitymaps,Eq. 2{1 foreachclassbyupdatingthefocianddirectrices.Inthissection,werstpresentanO(N2)timealgorithmtondtheoptimalclasseccentricities.Next,wedetermineresultantconstraintsonthefocusanddirectrixdescriptorsduetotheclassiedpoints. 6{1 ),givenxedfocianddirectrices.Thediscriminantboundary(Eq. 2{3 )inecc-Spaceiscompletelydenedbythelocationof 78

PAGE 79

6-2B .Consequently,thesepairsoflinespartitiontheecc-Spaceinto(N+1)22Drectangularintervals.Wenowmakethecriticalobservationthatwithintheconnesofsucha2Dinterval,anychoiceofapointthatrepresentsclasseccentricitiesresultsinidenticallabelassignments(seeFigure 6-2B ).Therefore,thesearchislimitedtojustthese(N+1)2intervals.Theintervalthatgivesthesmallestclassicationerrorischosen.Thecross-hairissetatthecenterofthechosen2Dintervaltoobtainlargefunctionalmargin.Whenever,therearemultiple2Dintervalsresultingintheleastempiricalerror,thelargerintervalischosensoastoobtainalargermargin. 6{3 .Assumingthatthedescriptorsofclass2arexed,theconstrainton"1(Xi)duetoEq. 6{3 isderivedinEq. 6{4 .Now,ifweareinterestedinupdatingonlyfocusF1,theconstraintsonF1intermsofdistancestopoints,Xi,aregivenbyEq. 6{5 Here"2i=j"2(Xi)e2j,andh1iisthedistancetothedirectrixhyperplaneofclass1(Eq. 2{4 ).InEq. 6{5 ,theonlyunknownvariableisF1.Similarly,wecanobtainconstraintsforalltheotherdescriptors.Whenevertheintervalsduetotheconstraintsareunbounded,weapplyboundsderivedfromtherangeofattributedeccentricitiesinthepreviousiteration.Themargin,,wassetto1%ofthisrange. 79

PAGE 80

6{5 )relatedtothecurrentlyclassiedpoints.Next,wepickasolutionfromtheNullSpacethatcanlearnamisclassiedpointbysatisfyingitsconstraintonF1,ifsuchasolutionexists.Thelearningtasknowreducestoupdatingthefocianddirectricesofboththeclassesalternately,sothatthemisclassiedpointsaremappedintotheirdesiredquadrantsinecc-Space,whilethecorrectlyclassiedpointsremainintheirrespectivequadrants.Notethatwithsuchupdates,ourlearningrateisnon-decreasing.Inthenextsection,weconstructNullSpacesforthefocusanddirectrixdescriptors.InSection 6.1.3 wedealwithlearningmisclassiedpoints. 6{5 ,: 80

PAGE 81

6.1.3 ).WhilethiswouldotherwisehavebeenanNP-hard[ 58 ]problem(likeageneralQPproblem),thegeometricstructureofthesequadraticconstraintsenablesustoconstructtheNullSpaceinjustO(N2M)time.Notethatbyassumption,thenumberofconstraintsNM.TheNullSpaceofFwithrespecttoeachconstraintinEq. 6{6 isthespacebetweentwoconcentrichyperspheresinRM,referredtoasashell.Hence,theNullSpaceforalltheconstraintsputtogetheristheintersectionofallthecorrespondingshellsinRM.Thisturnsouttobeacomplicatedobject.However,wecanexploitthefactthatthefocusinthepreviousiteration,denotedasF,satisesalltheconstraintsinEq. 6{6 sinceitresultedinNcclassiedpoints.Tothatend,werstconstructthelocusofallfocuspoints,F0,thatsatisfythefollowingequalityconstraints: 6{6 thatalsohasamuchsimplergeometry. 6{7 ;i.e.,thelocusofallfociF0thatareatdistanceritotherespectiveclassiedpointXi.TheNullSpaceisinitializedasthesetoffeasiblesolutionsfortherstequalityconstraintinEq. 6{7 .ItcanbeparameterizedasthehypersphereS1=(r1;X1),centeredatXiwithradiusri.Next,thesecondequalityconstraintisintroduced,theNullSpaceforwhich,consideredindependently,isthehypersphereS2=(r2;X2).ThenthecombinedNullSpaceforthetwoconstraintsistheintersectionofthetwohyperspheres,S1\S2. 81

PAGE 82

BFigure6-3. (A)IntersectionoftwohypersphereNullSpaces,S1(r1;X1)andS2(r2;X2)lyinginahyperplane.AnypointonthenewNullSpace(bright-circle)satisesboththehypersphere(distance)constraints.(B)S1\S2canbeparameterizedbythehypersphereSf1;2gcenteredatX1;2withradiusR1;2. AsillustratedinFigure 6-3A ,theintersectionoftwospheresinR3isacirclethatliesontheplaneofintersectionofthetwospheres.ThefollowingsolutionisbasedontheanalogueofthisfactinRM.Wemaketwocriticalobservations:theintersectionoftwohyperspheresisahypersphereofonelowerdimension,andthishypersphereliesontheintersectinghyperplaneoftheoriginalhyperspheres.Eachiterationofthealgorithminvolvestwosteps.Intherststep,were-parameterizethecombinedNullSpace,S1\S2,asahypersphereSf1;2gofonelowerdimensionlyinginthehyperplaneofintersectionHf1;2g.BasedonthegeometryoftheproblemandtheparameterizationofS1andS2,showninFigure 6-3B ,wecancomputetheradiusandthecenterofthenewhypersphereSf1;2g=(rf1;2g;Xf1;2g)inO(M)time,givenbyEqs. 6{8 6{9 6{10 .WecanalsodeterminetheintersectinghyperplaneHf1;2grepresentedasfbf1;2g;Q12g.Therstdescriptorbf1;2gisthedisplacementofHf1;2gfromtheoriginandtheotherdescriptoristheunitvectornormaltoHf1;2g.Infact,Qf1;2gliesalongthelinejoiningX1andX2.Theparametersof 82

PAGE 83

(6{9) Inthesecondstep,theproblemfortheremainingequalityconstraintsisreposedonthehyperplaneHf1;2g.ThisisaccomplishedbyintersectingeachoftheremaininghyperspheresS3;:;SNcthatcorrespondtothesamplesX3;:;XNc,withHf1;2g,inO(NM)time.Onceagain,basedonthegeometryoftheproblem,thenewcentersofthecorrespondinghyperspherescanbecomputedusingEq. 6{9 andtheirradiiaregivenby: TheproblemisnowtransparentlyposedinthelowerdimensionalhyperplaneHf1;2gasaproblemequivalenttotheonethatwebeganwith,exceptwithonelesshypersphereconstraint.Theendresultofrepeatingthisprocess(Nc1)timesyieldsaNullSpacethatsatisesalltheequalityconstraints(Eq. 6{7 ),representedasasinglehyperspherelyinginalowdimensionallinearsubspaceandcomputedinO(N2M)time.ItshouldbeobservedthatalltheintersectionsthusfararefeasibleandthatthesuccessiveNullSpaceshavenon-zeroradiisincetheequalityconstraintshaveafeasiblesolutionapriori,i.e.,F. 83

PAGE 84

6{8 6{9 overiterations,wenoticethatthecomputationofthenormaltothehyperplaneofeachintersectionandthatofthenewcentercanbeequivalentlyposedasaGram-Schmidtorthogonalization[ 56 ].TheprocessisequivalenttotheQRdecompositionofthefollowingmatrix: where;~Xi=(XiX1)Ce=(X1)=X1+QQT(FX1) (6{13) 6{13 projectsanypointintoalowdimensionallinearsubspace,H,normaltotheunitvectorsinQ(Eq. 6{12 )andcontainsF.HcanbedenedasthelinearsubspacefX2RM:QT(XF)=0g.WeuseexistingstableandecientQRfactorizationroutinestocomputeSeinO(N2M)time.ItisnoteworthythatthenalNullSpaceduetotheequalityconstraintsinEq. 6{7 canberepresentedasasinglehypersphereSe=(re;Ce)H.ThecenterandradiusofSecanbecomputedusingEq. 6{13 .ThegeometryofSeenablesustogeneratesamplefocuspointsthatalwayssatisfytheequalityconstraints.Inthenextsection,wepursuelargerregionswithinwhichtheinequalityconstraintsonthefocusduetoEq. 6{6 aresatised. 6-4 .Werstcomputethelargestdisccenteredatthepreviousfocus,F,whichisguaranteedtoliewithintheintersection.Next,werevolvethediscaboutthelinejoiningX1andX2.Theresultantobjectisadoughnutliketoroidalregion,thatcanbedenedasaproductofacircleandadiscinR3,asillustratedinFigure 6-4A .ThecircletracedbyFisthelocusofallfociFthatareatthesamedistancestopointsX1;X2,asF.WepursuethisideainRM.ThenalNullSpace,saySf,thatsatisestheinequality 84

PAGE 85

BFigure6-4. Cross-sectionofshell-shellintersectioninR3.Thereddiscin(A)isthelargestdisccenteredatFthatiswithintheintersection.ThethickcirclepassingthroughFandorthogonaltothecross-sectionplane,isthelocusofallfociwithsamedistanceconstraintsasF.Aproductofthediscandthethickcircleresultsinatoroid,asin(B),whichisatractablesubregionlyingwithintheintersectionofshells. constraintsonF(Eq. 6{6 )canbedenedas: 6{7 ),and>0istheradiusofthelargestsolidhypersphere(ball)atallF0thatsatisestheinequalityconstraints.Inthismanner,weintendtoaddacertainthicknesstothelocusofF0.Wecancomputetheradius,,ofthelargestdiscatFfromEqs. 6{6 6{14 .Letri=kXiFk,F=F+UbeapointontheballatF.DuetotriangleinequalitybetweenF,FandanypointXi,82[0;)wehave: 85

PAGE 86

6{6 .WecanthuscomputefromEq. 6{15 injustO(N)timegiventhedistanceboundsrli,andrui.Withverylittlecomputationalexpense,wecantrackalargerNullSpacethanthatduetotheequalityconstraints(Eq. 6{7 ). 6{4 .First,welistconstraintsonthedirectrixsothattheclassiedpointsremainintheirquadrantswhenmappedintoecc-Space.Second,wereducetheproblemofconstructingtheresultantfeasiblespaceintothatofafocusNullSpacecomputation.Theconstraintsonthehyperplanedescriptorsetfb;Qg,duetothoseonattributedeccentricitiesinEq. 6{4 ,8i=1:::Nc,are: (6{16) wherehli,andhuiarelowerandupperboundsonthedistanceofXitothehyperplanefb;Qg.Weassumeeveryotherdescriptorisxedexcepttheunknownfb;Qg.UponsubtractingtheconstraintonX1fromtheotherconstraints,weobtainEq. 6{17 ,where~Xi=(XiX1),~hli=(hlihu1),and~hui=(huihl1).Wecannowconvertthedistancestohyperplaneconstraintstothoseofdistancestoapoint,byconsideringkQ~XikasinEq. 6{18 .Letzi=(1+k~Xik2).TheresultantshellconstraintsonQ,8i=1:::NcareinEq. 6{19 (6{19) ThusgivenNcinequalityconstraintsonpointdistancestoQandgiventhepreviousQ,weusethesolutionfromthefocusupdateproblemtoconstructaNullSpaceforQ.The 86

PAGE 87

Cross-sectionofthe(donutlike)toroidalNullSpace,Sf,lyingintheplanespannedbyfXmc;(Xmc);Ceg.Theexpressionsfordistancesdnanddfintermsofp,s,andrearegiveninEq. 6{20 onlyunknownleftisbwhichliesinanintervalduetoX1inEq. 6{16 ,givenQ.Wechoosebtobethecenterofthatinterval,asb=(hu1+hl1)=2QTX1,sothatthemarginforX1islargerinecc-Space. 6{6 )ofXmc,sayrl
PAGE 88

where(re;Ce)aretheradiusandcenterofSe.SeeFigure 6-5 tointerprettheserelations.NowtheintersectionofSfandtheshellcorrespondingtoXmcisreducedtothatoftheintervals:(dn;df+)\(rl;ru).Iftheydon'tintersect,wecaneasilypicka(focus)pointonSfthatiseithernearestorfarthesttotheshellrelatedtoXmc,soastomaximallylearnXmc,i.e.,changer(Xmc)whichinturnmaximizesymcg(Xmc).Iftheydointersect,thereareonlyafewcasesofintersectionspossibleduetothegeometryofSfandtheshell.EachsuchintersectionturnsouttobeorcontainsanotherfamiliarobjectlikeSf,ashell,orasolidhypersphere(ball).Whenevertheintersectionexists,wepickasolutioninitthatmaximizesPyig(Xi)fortheothermisclassiedpoints,i.e.,asolutionclosertosatisfyingalltheinequalityconstraintsputtogether.Inthismanner,wecomputeanewupdateforeachmisclassiedpoint.Next,wedescribeacriteriontopickanupdate,fromthesetofcandidateupdates,thatcanyieldsimplestpossiblediscriminant. (C1;C2)=jQT1Q2j+jQT1F1;2j+jQT2F1;2j(6{22)where,F1;2istheunitnormalalongthelinejoiningthefoci.ThemaximumofthismeasurecorrespondstothecongurationofconicsectionsthatcanyieldalineardiscriminantasdiscussedinSection 2.3 .Themeasureisdenedforfocusanddirectrixdescriptors.Itcanbeusedtopickanupdateyieldingthelargeststinesssothattheresultantdiscriminantcanbemostgeneralizable.Wenoticedthatthenon-linearityof 88

PAGE 89

2-3 ,thestinessmeasuresfordierenttypesofboundariesarelisted.Thediscriminantboundarybecomessimplerwhenthedirectrixnormalsaremadeidentical.Moreover,thefunctionaldependenciesbetweenthedistanceconstraintswhichdeterminethediscriminant,inEq. 2{4 ,becomesimplerifthelinejoiningthefociisparalleltothedirectrixnormals.Itstemsfromtheobservationthateverypointinthehyperplanethatperpendicularlybisectsthelinejoiningthefoci,isatequaldistancefromthefoci(r1(X)r2(X)).Lineardiscriminantscanbeguaranteedwhenthestinessismaximumandtheclasseccentricitiesareeitherequalinmagnitudeornearzero.TwocongurationsthatyieldedlineardiscriminantsareillustratedinFigures 2-3C 2-3D .FromthenalNullSpaceofthedescriptorbeingupdated,wedetermineadescriptorupdatefavoredbyeachmisclassiedpoint.Amongthecompetingupdates,wechoosetheupdatewiththeleastempiricalerrorandmaximumstiness.Thus,thelargestinesspursuitincorporatesaquasimodelselectionintothelearningalgorithm. 2{1 ),theNullSpacesbecomesmall(attimes,vanishinglysmall)ifthefocusordirectrixdescriptorsareplacedveryclosetothesamples.Wefoundthefollowingdata-driveninitializationstobeconsistentlyeectiveinourexperiments.TheinitializationofalldescriptorsweredonesoastostartwithalineardiscriminantobtainedfromeitherlinearSVMorFisher[ 35 ]classierorwiththehyperplanethatbisectsthelinejoiningtheclassmeans.Theclasseccentricitiesweresettoh0;0i.Thefociwereinitializedtoliefarfromtheirrespectiveclasssamples.WeinitializednormalsQ1=Q2withthatinthe 89

PAGE 90

6{13 .Also,theorderofclassiedsamplesprocesseddoesnotaectthenalNullSpace.Akeycontributionofourlearningtechniqueisthetrackingofalargesetoffeasiblesolutionsasacompactgeometricobject.FromthisNullSpacewepickasetofsolutionstolearneachmisclassiedpoints.Fromthisset,wepickasolutionbiasedtowardsasimplerdiscriminantusingastinesscriterionsoastoimproveupongeneralization.Thesizeofthemargin,,inecc-Spacealsogivesamodicumofcontrolovergeneralization. 90

PAGE 91

Detailsofdatausedinexperiments DatasetFeaturesSamplesClass1Class2 Epilepsy216441925ColonTumor2000624022Leukemia7129724725CNS7129602139ETH-Objects16384201010TechTC3816831437469Isolet-BC6171005050 classier(CSC).Next,wediscussimplementationdetailsofCSCfollowedbyareviewoftheresults. 6-1 .Epilepsydata[ 3 ]consistsoftheshapedeformationbetweentheleftandrighthippocampifor44epilepsypatients.First,wecomputedthedisplacementvectoreldin3Drepresentingthenon-rigidregistrationthatcapturestheasymmetrybetweentheleftandrighthippocampi.Wefoundthejoint3Dhistogramofthe(x;y;z)componentsofthedisplacementvectoreldtobeabetterfeaturesetforclassication.Weuseda666binningofthehistogramsin3D.Thetaskwastocategorizethelocalizationofthefocusofepilepsytoeithertheleft(LATL)orrighttemporallobe(RATL).TheColonTumor[ 4 ],theLeukemia[ 1 ],andtheCNS[ 73 ]datasetsareallgenearrayexpressiondatasets.Thegene-expressionscorescomputationisexplainedintheirrespectivereferences.IntheColonTumordatathetaskistodiscriminatebetweennormalandtumortissues.Theleukemiadataconsistsoffeaturesfromtwodierentcancer 91

PAGE 92

46 ]objectdataset,wechose10proleviewsofadogandahorse.Theviewsarebinaryimagesofsize128128.FromtheTechTC-38[ 74 ]text-categorizationdataset,weclassifythedocumentsrelatedtoAlabamavs.Michiganlocalities(id:38[ 74 ]),givenwordfrequenciesastheirfeatures.Wedroppedfeatures(words)thatareeitherveryinfrequentortoocommon.TheIsolet-BCdatasetisapartoftheIsoletSpokenLetterRecognitionDatabase[ 45 72 ],whichconsistsoffeaturesextractedfromspeechsamplesofBandCfrom25speakers. 35 ]asdescribedinYu&Yang[ 61 ].Thistechniqueexploitsthefactthathigh-dimensionaldatahassingularscattermatricesanddiscardsthesub-spacethatcarriesnodiscriminativeinformation.SupportVectorMachines(SVM)[ 36 ]andKernelFisherDiscriminants(KFD)[ 62 ]broadlyrepresentedthenon-linearcategory.BothemploythekerneltrickofreplacinginnerproductswithMercerkernels.Amonglinearclassiers,wechoseLFDandlinearSVM.WeusedlibSVM[ 75 ],aC++implementationofSVMusingSequentialMinimalOptimization[ 37 ]andourownMATLABimplementationofKFD.Polynomial(PLY)andRadialBasis(RBF)KernelswereconsideredforSVMandKFD.Weperformedstratied10-foldcrossvalidation(CV)[ 28 ]inwhichthesamplesfromeachclassarerandomlysplitinto10partitions.Partitionsfromeachclassareputintoafold,sothatthelabeldistributionoftrainingandtestingdataissimilar.Ineachrun,onefoldiswithheldfortestingwhiletherestisusedfortrainingandtheprocessisrepeated10times.Theaveragetest-errorisreportedasthegeneralizationerrorestimateoftheclassier.TheexperimentalresultsarelistedinTable 6-2 .Thebestparametersforeachclassierwereempiricallyexploredusinggridsearch.Wesearchedforthebest 92

PAGE 93

6-1 .TheparametersofCSCinvolvethechoiceofinitialization,describedinSection 6.1.4 ,andthefunctionalmargin.Thevalueisxedtobeatacertainpercentageoftheextentofattributedeccentricities,"(X).Wesearchedforthebestpercentageamong[1%;:1%;:01%]. 6-2 .ConicSectionClassier(CSC)performedsignicantlybetterthantheotherclassiersfortheEpilepsydata.Inthegene-expression 93

PAGE 94

ClassicationresultsforCSC,(Linear&Kernel)FisherDiscriminantsandSVM. DatasetCSCLFDKFD-PLYKFD-RBFSVM-PLYSVM-RBF Epilepsy91.5078.0080.5085.0084.5084.50ColonTumor88.3385.2478.5783.5788.3383.81Leukemia95.7192.8698.5797.3297.1495.71CNS71.6750.0063.3375.0065.0065.00ETH-Objects90.0085.0090.0055.0085.0090.00TechTC3874.1467.0571.2471.9072.1060.24Isolet-BC95.0091.0094.0094.0094.0094.00 datasets,CSCwascomparabletootherswithLeukemiaandColonTumordata,butnotwiththeCNSdatawhichturnedoutbeatougherclassicationtask.CSCfaredslightlybetterthantheotherclassiersintext-categorizationandspokenletterrecognitiondata.Infact,CSChasconsistentlyperformedsubstantiallybetterthantheLFD,asitisdesirablebydesign.ItisalsointerestingtonotethatnoneoftheSVMclassiersactuallybeatCSC.ThisempiricallyprovesthatCSCisabletorepresentmoregeneralizableboundariesthanSVMforallthedataconsidered.TheparametersusedintheexperimentsarelistedinTable 6-3 .ThecolumnrelatedtoQ,liststhatthenormalstodirectriceswereinitializedfromthoseduetoeitherlinearSVM,orLFD,orthelinejoiningthemeansofthesamplesfromthesameclass.Thecolumnrelatedtodenotesthemarginaspercentageoftherangeofclassattributedeccentricities.WereporttheaveragestinessofCSCover10-foldsineachexperiment.NotethatthisisnotaparameterofCSC.WeincludedthisinTable 6-3 forcomparisonwiththedegreeofthepolynomialkernels.Thestinessvaluesforallexperimentswerenearitsmaximum(3:0).Thestandarddeviationofthestinesswaslessthat0:02forallthedata,exceptforCNS(:08),andIsolet-BC(:1)datasets.Hence,stinessdoesguidedescriptorupdatestowardsyieldingsimplerdiscriminantsforthesameorhigherlearningaccuracy.Thebestdegreeforotherpolynomialkernelclassiersalsoturnedouttobe1,inseveralcases.ExceptfortheLeukemiadata,CSCperformedbetterthanlinearSVMandlinearFisherclassiersinallcases. 94

PAGE 95

ParametersforresultsreportedinTable 6-2 EpilepsyMeans12.9997435.006ColonTumorLFD.012.9955152.5LeukemiaLFD12.979317013CNSMeans.12.94545.071.001ETH-ObjectsLFD.012.9884330012TechTCSVM12.905715001300Isolet-BCSVM13.04.31.07 *Notaparameter. 61 ],ittookunderasecondforallthedatasets.TheruntimesforCSCwerebetterthantheotherclassierswithRBFkernels.However,thesearchfortheoptimalmodelparametersaddsafactorof5to10.FromtheexperimentalresultsinTable 6-2 ,itisevidentthatConicSectionclassierout-performedormatchedtheothersinamajorityofthedatasets. 56 ]orthogonalization.Weintroducedastinesscriterionthatquantiestheextentofthenon-linearityinthediscriminantboundary.IneachdescriptorNullSpace,wepicksolutionsthatlearnmisclassiedpointsandyieldsimplerdiscriminantboundaries,duetothestinesscriterion.Thusstinessenablestheclassiertoperformmodelselectioninthelearningphase.Asthelearningproblemisequivalenttoanon-linearoptimizationproblemthatisnotconvex,ourmethodispronetolocalminimaaswell. 95

PAGE 96

6-2 .Theclassierinitspresentformusesaxiallysymmetricconicsections.Theconceptclass,bydenitionappliestothemulti-classcaseaswell.Thelearningalgorithmneedstoincorporateequivalentboundaryrepresentationinecc-Space,amongotheraspects.Infuturework,weintendtoextendthistechniquetomulti-classclassication,toconicsectionsthatarenotnecessarilyaxiallysymmetric,andexplorepursuitofconicsinkernelspaces. 96

PAGE 97

56 ].Wehaveintroducedastinesscriterionthatquantiestheextentofnon-linearityinthediscriminantboundary.IneachdescriptorNullSpace,wepicksolutionsguidedbythestinesscriterion,thatcanlearnmisclassiedpointsaswellasyieldsimplerdiscriminantboundaries.Thusstinessenablestheclassiertoperformanad-hocmodelselectioninthelearningphase.WehavealsopresentedanovelgeometricalgorithmtocomputemargintothediscriminantboundariesresultingfromtheConicSectionclassier.Thistechniquewasthenusedtomaximizemarginwhilelearningconicsectiondescriptorsforeachclass,soastoimproveuponthegeneralizabilityoftheclassier.Wehavedemonstratedtheversatilityoftheresultantclassierbytestingitagainstseveralstate-of-the-artclassiersonmanypublicdomaindatasets.Ourclassierwasabletoclassifytougherdatasetsbetterthanothersinmostcases.Asthelearningproblemisequivalenttoanon-linearoptimizationproblemthatisnotconvex,ourmethodisalsopronetolocalminima.FutureWork 97

PAGE 98

98

PAGE 99

6{14 99

PAGE 100

[1] T.Golub,D.Slonim,P.Tamayo,C.Huard,M.Gaasenbeek,J.Mesirov,H.Coller,M.Loh,J.Downing,M.Caligiuri,C.Bloomeld,andE.Lander,\Molecularclassicationofcancer:Classdiscoveryandclasspredictionbygeneexpressionmonitoring,"Science,vol.286,no.5439,pp.531{537,1999. [2] W.Zhao,S.Corporation,A.Rosenfeld,andP.J.Phillips,\Facerecognition:Aliteraturesurvey,"ACMComputingSurveys,vol.35,no.4,pp.399{458,2003. [3] N.Vohra,B.C.Vemuri,A.Rangarajan,R.L.Gilmore,S.N.Roper,andC.M.Leonard,\Kernelsherforshapebasedclassicationinepilepsy,"MICCAI,pp.436{443,2002. [4] U.Alon,N.Barkai,D.A.Notterman,K.Gish,S.Ybarra,D.Mack,andA.Levine,\Broadpatternsofgeneexpressionrevealedbyclusteringanalysisoftumorandnormalcolontissuesprobedbyoligonucleotidearrays,"inProc.Nat.Acad.Sc.USA,vol.96,1999,pp.6745{6750. [5] A.M.MartinezandM.Zhu,\Wherearelinearfeatureextractionmethodsapplicable?"PatternAnalysisandMachineIntelligence,IEEETransactionson,vol.27,no.12,pp.1934{1944,2005. [6] Due,A.K.Jain,andT.Taxt,\Featureextractionmethodsforcharacterrecognition-asurvey,"PatternRecognition,vol.29,no.4,pp.641{662,April1996. [7] R.J.Quinlan,\Inductionofdecisiontrees,"MachineLearning,pp.81{106,1986. [8] R.Duda,P.Hart,andD.Stork,PatternClassication.WileyInterscience,2001. [9] I.GuyonandA.Elissee,\Anintroductiontovariableandfeatureselection,"JournalofMachineLearningResearch,vol.3,pp.1157{1182,2003. [10] G.CasellaandR.Berger,StatisticalInference.DuxburyResourceCenter,June2001. [11] N.KwakandC.-H.Choi,\Inputfeatureselectionbymutualinformationbasedonparzenwindow,"PatternAnalysisandMachineIntelligence,IEEETransactionson,vol.24,no.12,pp.1667{1671,2002. [12] H.Peng,F.Long,andC.Ding,\Featureselectionbasedonmutualinformation:criteriaofmax-dependency,max-relevance,andmin-redundancy."IEEETransPatternAnalMachIntell,vol.27,no.8,pp.1226{1238,August2005. [13] J.A.Sterne,G.D.Smith,andD.R.Cox,\Siftingtheevidence|what'swrongwithsignicancetests?anothercommentontheroleofstatisticalmethods,"BMJ,vol.322,no.7280,pp.226{231,January2001. 100

PAGE 101

J.HorraandM.Rodriguez-Bernal,\Posteriorpredictivep-values:whattheyareandwhattheyarenot,"TEST:AnOcialJournaloftheSpanishSocietyofStatisticsandOperationsResearch,vol.10,no.1,pp.75{86,June2001. [15] T.Hsing,E.Dougherty,P.Sebastiani,I.S.Kohane,andM.F.Ramoni,\Relationbetweenpermutation-testpvaluesandclassiererrorestimates,"inMachineLearn-ing,SpecialIssueonMachineLearningintheGenomics,vol.52,2003,p.1130. [16] R.KohaviandG.H.John,\Wrappersforfeaturesubsetselection,"ArticialIntelligence,vol.97,no.1-2,pp.273{324,1997. [17] K.-C.Lee,J.Ho,andD.J.Kriegman,\Acquiringlinearsubspacesforfacerecognitionundervariablelighting,"IEEETransactionsonPatternAnalysisandMachineIntelligence,vol.27,no.5,pp.684{698,2005. [18] L.TorresaniandK.chihLee,\Largemargincomponentanalysis,"inAdvancesinNeuralInformationProcessingSystems19.Cambridge,MA:MITPress,2007,pp.1385{1392. [19] S.Watanabe,Patternrecognition:humanandmechanical.NewYork,NY,USA:JohnWiley&Sons,Inc.,1985. [20] JMLR,SpecialIssueonVariableandFeatureSelection,2003,vol.3. [21] R.Herbrich,LearningKernelClassiers:TheoryandAlgorithms(AdaptiveComputa-tionandMachineLearning).TheMITPress,December2001. [22] T.Hastie,R.Tibshirani,andJ.H.Friedman,TheElementsofStatisticalLearning.Springer,August2001. [23] D.H.Wolpert,\Thelackofaprioridistinctionsbetweenlearningalgorithms,"NeuralComputation,vol.8,no.7,pp.1341{1390,October1996. [24] V.N.Vapnik,TheNatureofStatisticalLearningTheory(InformationScienceandStatistics).Springer,November1999. [25] C.Burges,\Atutorialonsupportvectormachinesforpatternrecognition,"DataMiningandKnowledgeDiscovery,vol.2,no.2,pp.121{167,1998. [26] V.N.Vapnik,\Estimationofdependencesbasedonempiricaldata(translatedbys.kotz)."NewYork:Springer-Verilag,1982. [27] D.HushandC.Scovel,\Onthevcdimensionofboundedmarginclassiers,"Ma-chineLearning,vol.45,no.1,pp.33{44,2001. [28] R.Kohavi,\Astudyofcross-validationandbootstrapforaccuracyestimationandmodelselection,"inInternationalJointConferenceonAriticialIntelligence,1995,pp.1137{1145. [29] V.Vapnik,StatisticalLearningTheory.JohnWileyandSons,NewYork,1999. 101

PAGE 102

H.ZhuandR.Rohwer,\Nofreelunchforcross-validation,"NeuralComputation,vol.8,pp.1421{1426,1996. [31] C.Goutte,\Noteonfreelunchesandcross-validation,"NeuralComputation,vol.9,no.6,pp.1245{1249,1997. [32] T.Fawcett,\Anintroductiontorocanalysis,"PatternRecognitionLetters,vol.27,no.8,pp.861{874,June2006. [33] D.M.Gavrila,\Abayesian,exemplar-basedapproachtohierarchicalshapematching,"IEEETransactionsonPatternAnalysisandMachineIntelligence,vol.29,no.8,pp.1408{1421,2007. [34] Y.Zhou,L.Gu,andH.-J.Zhang,\Bayesiantangentshapemodel:Estimatingshapeandposeparametersviabayesianinference,"ComputerVisionandPatternRecognition,IEEEComputerSocietyConferenceon,vol.1,p.109,2003. [35] R.Fisher,\Theuseofmultiplemeasurementsintaxonomicproblems,"inAnnalsofEugenics(7),1936,pp.111{132. [36] C.CortesandV.Vapnik,\Support-vectornetworks,"MachineLearning,vol.20,no.3,pp.273{297,1995. [37] J.C.Platt,\Sequentialminimaloptimization:Afastalgorithmfortrainingsupportvectormachines,"inAdvancesinKernelMethods-SupportVectorLearning,B.Scholkopf,C.Burges,andA.Smola,Eds.MITPress,Cambridge,MA,USA,1999,pp.185{208. [38] C.-C.ChangandC.-J.Lin,\Libsvm:alibraryforsupportvectormachines(version2.31),"2001. [39] J.Mouro-Miranda,K.J.Friston,andM.Brammer,\Dynamicdiscriminationanalysis:Aspatial-temporalsvm,"NeuroImage,vol.36,no.1,pp.88{99,2007. [40] X.Wang,R.Hutchinson,andT.M.Mitchell,\Trainingfmriclassierstodiscriminatecognitivestatesacrossmultiplesubjects,"inInNIPS,2004. [41] O.BousquetandB.Scholkopf,\Commenton"supportvectormachineswithapplications","StatisticalScience,vol.21,p.337,2006. [42] T.M.Mitchell,MachineLearning.McGraw-Hill,1997. [43] C.M.Bishop,NeuralNetworksforPatternRecognition.NewYork,NY,USA:OxfordUniversityPress,Inc.,1995. [44] S.Haykin,NeuralNetworks:AComprehensiveFoundation(2ndEdition),2nded.PrenticeHall,1998. [45] R.Cole,Y.Muthusamy,andM.Fanty,\TheISOLETspokenletterdatabase,"OregonGraduateInstitute,Tech.Rep.CSE90-004,1990. 102

PAGE 103

B.LeibeandB.Schiele,\Analyzingappearanceandcontourbasedmethodsforobjectcategorization,"inComputerVisionandPatternRecognition,vol.2,2003,pp.409{415. [47] A.JainandD.Zongker,\Featureselection:Evaluation,application,andsmallsampleperformance,"IEEETransactionsonPatternAnalysisandMachineIntelligence,vol.19,no.2,pp.153{158,1997. [48] P.Somol,P.Pudil,andJ.Kittler,\Fastbranchandboundalgorithmsforoptimalfeatureselection,"IEEETransactionsonPatternAnalysisandMachineIntelligence,vol.26,no.7,pp.900{912,2004. [49] A.Banerjee,S.Kodipaka,andB.C.Vemuri,\Aconicsectionclassieranditsapplicationtoimagedatasets,"ComputerVisionandPatternRecognition,IEEEComputerSocietyConferenceon,vol.1,pp.103{108,2006. [50] A.J.SmolaandP.J.Bartlett,Eds.,AdvancesinLargeMarginClassiers.Cambridge,MA,USA:MITPress,2000. [51] F.L.Bookstein,\Fittingconicsectionstoscattereddata,"ComputerGraphicsandImageProcessing,vol.9,no.1,pp.56{71,1979. [52] T.Pavlidis,\Curvettingwithconicsplines,"ACMTransactionsonGraphics,vol.2,no.1,pp.1{31,1983. [53] L.Quan,\Conicreconstructionandcorrespondencefromtwoviews,"IEEETrans-actionsonPatternAnalysisandMachineIntelligence,vol.18,no.2,pp.151{160,1996. [54] G.Dorner,\AuniedframeworkforMLPsandRBNFs:Introducingconicsectionfunctionnetworks,"CyberneticsandSystems,vol.25,no.4,1994. [55] D.DeCoste,M.Burl,A.Hopkins,andN.Lewis,\Supportvectormachinesandkernelsherdiscriminants:Acasestudyusingelectronicnosedata,"FourthWorkshoponMiningScienticDatasets,2001. [56] L.N.TrefethenandD.Bau,NumericalLinearAlgebra.SIAM:SocietyforIndustrialandAppliedMathematics,1997. [57] E.W.Weisstein,CRCConciseEncyclopediaofMathematics,2nded.BocaRaton,FL:CRCPress,2003. [58] K.G.MurtyandS.N.Kabadi,\Somenp-completeproblemsinquadraticandnonlinearprogramming,"Math.Program.,vol.39,no.2,pp.117{129,1987. [59] D.GrahamandN.Allinson,\Characterizingvirtualeigensignaturesforgeneralpurposefacerecognition,"FaceRecognition:FromTheorytoApplications,NATOASISeriesF,ComputerandSystemsSciences,vol.163,pp.446{456,1998. 103

PAGE 104

K.J.Dana,B.vanGinneken,S.K.Nayar,andJ.J.Koenderink,\Reectanceandtextureofreal-worldsurfaces,"ACMTrans.onGraph.,vol.18,no.1,pp.1{34,1999. [61] H.YuandJ.Yang,\Adirectldaalgorithmforhigh-dimensionaldatawithapplicationtofacerecognition,"PatternRecognition,vol.34,no.12,pp.2067{2070,2001. [62] S.Mika,G.Ratsch,J.Weston,B.Scholkopf,andK.-R.Muller,\Fisherdiscriminantanalysiswithkernels,"NeuralNetworksforSignalProcessingIX,pp.41{48,1999. [63] B.Scholkopf,C.Burges,andA.Smola,AdvancesinKernelMethods-SupportVectorLearning.MITPress,Cambridge,1999. [64] T.S.Furey,N.Duy,N.Cristianini,D.Bednarski,M.Schummer,andD.Haussler,\Supportvectormachineclassicationandvalidationofcancertissuesamplesusingmicroarrayexpressiondatar,"Bioinformatics,vol.16,no.10,pp.906{914,2000. [65] M.VarmaandA.Zisserman,\Textureclassication:Arelterbanksnecessary?"inComputerVisionandPatternRecognition,vol.2,2003,pp.691{698. [66] T.LeungandJ.Malik,\Recognizingsurfacesusingthree-dimensionaltextons,"inProceedingsoftheInternationalConferenceonComputerVision(ICCV),vol.2.Washington,DC,USA:IEEEComputerSociety,1999. [67] E.Spellman,B.C.Vemuri,andM.Rao,\UsingtheKL-centerforecientandaccurateretrievalofdistributionsarisingfromtextureimages,"ComputerVisionandPatternRecognition,pp.111{116,2005. [68] S.Kodipaka,A.Banerjee,andB.C.Vemuri,\Largemarginpursuitforaconicsectionclassier."inIEEEConf.onComputerVisionandPatternRecognition,2008. [69] I.Z.EmirisandE.P.Tsigaridas,\Comparingrealalgebraicnumbersofsmalldegree."inEuropeanSymposiaonAlgorithms,2004,pp.652{663. [70] S.S.Rao,Mechanicalvibrations,4thed.Addison-Wesley,2004. [71] S.Kodipaka,B.C.Vemuri,A.Rangarajan,C.M.Leonard,I.Schmallfuss,andS.Eisenschenk,\Kernelsherdiscriminantforshape-basedclassicationinepilepsy,"MedicalImageAnalysis,vol.11,no.2,pp.79{90,2007. [72] A.AsuncionandD.Newman,\UCImachinelearningrepository,"2007. [73] S.L.Pomeroyetal.,\Predictionofcentralnervoussystemembryonaltumouroutcomebasedongeneexpression,"Nature,vol.415,no.6870,pp.436{442,2002. [74] E.GabrilovichandS.Markovitch,\Textcategorizationwithmanyredundantfeatures:usingaggressivefeatureselectiontomakesvmscompetitivewithc4.5,"inICML'04:Proceedingsofthetwenty-rstinternationalconferenceonMachinelearning.ACM,2004. 104

PAGE 105

R.-E.Fan,P.-H.Chen,andC.-J.Lin,\Workingsetselectionusingsecondorderinformationfortrainingsupportvectormachines,"JournalofMachineLearningResearch,vol.6,pp.1889{1918,December2005. 105

PAGE 106

SanthoshKodipakareceivedBachelorofTechnologyincomputerscienceandengineering,withhonorsinvisualinformationprocessingfromtheInternationalInstituteofInformationTechnology,Hyderabad,IndiainMay2003.HereceivedhisMasterofScienceincomputerengineeringfromtheUniversityofFloridainDecember2007.HegraduatedwithaPh.DincomputerengineeringinAugust2009,advisedbyProf.VemuriandDr.Banerjee.Inhisdissertation,heintroducedanovelConicSectionClassieralongwithtractablegeometriclearningalgorithms.Duringhisgraduatestudy,healsoworkedondiagnosisofepilepsyinhumansgiven3DhippocampalshapessegmentedfromMRscans,andclassicationoflongitudinalrathippocampalscansforseizureincidencefrom3Dvolumedeformationtensorbasedmorphometry.HisresearchinterestsincludeMachineLearning,MedicalImageAnalysis,ComputerVision,GraphicsandVisualization. 106