<%BANNER%>

Combinatorial and Nonlinear Optimization Techniques in Pattern Recognition with Applications in Healthcare

Permanent Link: http://ufdc.ufl.edu/UFE0024768/00001

Material Information

Title: Combinatorial and Nonlinear Optimization Techniques in Pattern Recognition with Applications in Healthcare
Physical Description: 1 online resource (169 p.)
Language: english
Creator: Kundakcioglu, Omer
Publisher: University of Florida
Place of Publication: Gainesville, Fla.
Publication Date: 2009

Subjects

Subjects / Keywords: bound, branch, cancer, data, discrimination, integer, machine, mining, multiple, optimization, pattern, programming, support, svm, toxic, vector
Industrial and Systems Engineering -- Dissertations, Academic -- UF
Genre: Industrial and Systems Engineering thesis, Ph.D.
bibliography   ( marcgt )
theses   ( marcgt )
government publication (state, provincial, terriorial, dependent)   ( marcgt )
born-digital   ( sobekcm )
Electronic Thesis or Dissertation

Notes

Abstract: The first main contribution of this dissertation is the application of standard support vector machine (SVM) classifiers for death cell discrimination. SVMs are a set of machine learning algorithms widely used for classification and regression in data mining, machine vision, and bioinformatics. In this study, Raman spectroscopy is employed to assess the potential toxicity of chemical substances and SVM classifiers successfully assess the potential effect of the test toxins. The second main contribution is the formulation, complexity result, and an efficient heuristic for Selective SVM classifiers that consider a selection process for both positive and negative classes. Selective SVMs are compared with other standard alignment methods on a neural data set that is used for analyzing the integration of visual and motor cortexes in the primate brain. The third main contribution of this dissertation is the extension of SVM classifiers for multiple instance (MI) data where a selection process is required for only positive bags. Different formulations, complexity results, and an exact algorithm are presented with computational results on publicly available image annotation and molecular activity prediction data sets. MI pattern recognition methods are then further extended to support vector regression (SVR) and an exact algorithm is presented for the problem. Computational results are presented for a well established breast cancer prognosis data set that is added artificial noise to create synthetic MI regression data. Finally, two open complexity results on feature selection for consistent biclustering and sparse representation for hyperplane clustering are presented.
General Note: In the series University of Florida Digital Collections.
General Note: Includes vita.
Bibliography: Includes bibliographical references.
Source of Description: Description based on online resource; title from PDF title page.
Source of Description: This bibliographic record is available under the Creative Commons CC0 public domain dedication. The University of Florida Libraries, as creator of this bibliographic record, has waived all rights to it worldwide under copyright law, including all related and neighboring rights, to the extent allowed by law.
Statement of Responsibility: by Omer Kundakcioglu.
Thesis: Thesis (Ph.D.)--University of Florida, 2009.
Local: Adviser: Pardalos, Panagote M.
Electronic Access: RESTRICTED TO UF STUDENTS, STAFF, FACULTY, AND ON-CAMPUS USE UNTIL 2011-08-31

Record Information

Source Institution: UFRGP
Rights Management: Applicable rights reserved.
Classification: lcc - LD1780 2009
System ID: UFE0024768:00001

Permanent Link: http://ufdc.ufl.edu/UFE0024768/00001

Material Information

Title: Combinatorial and Nonlinear Optimization Techniques in Pattern Recognition with Applications in Healthcare
Physical Description: 1 online resource (169 p.)
Language: english
Creator: Kundakcioglu, Omer
Publisher: University of Florida
Place of Publication: Gainesville, Fla.
Publication Date: 2009

Subjects

Subjects / Keywords: bound, branch, cancer, data, discrimination, integer, machine, mining, multiple, optimization, pattern, programming, support, svm, toxic, vector
Industrial and Systems Engineering -- Dissertations, Academic -- UF
Genre: Industrial and Systems Engineering thesis, Ph.D.
bibliography   ( marcgt )
theses   ( marcgt )
government publication (state, provincial, terriorial, dependent)   ( marcgt )
born-digital   ( sobekcm )
Electronic Thesis or Dissertation

Notes

Abstract: The first main contribution of this dissertation is the application of standard support vector machine (SVM) classifiers for death cell discrimination. SVMs are a set of machine learning algorithms widely used for classification and regression in data mining, machine vision, and bioinformatics. In this study, Raman spectroscopy is employed to assess the potential toxicity of chemical substances and SVM classifiers successfully assess the potential effect of the test toxins. The second main contribution is the formulation, complexity result, and an efficient heuristic for Selective SVM classifiers that consider a selection process for both positive and negative classes. Selective SVMs are compared with other standard alignment methods on a neural data set that is used for analyzing the integration of visual and motor cortexes in the primate brain. The third main contribution of this dissertation is the extension of SVM classifiers for multiple instance (MI) data where a selection process is required for only positive bags. Different formulations, complexity results, and an exact algorithm are presented with computational results on publicly available image annotation and molecular activity prediction data sets. MI pattern recognition methods are then further extended to support vector regression (SVR) and an exact algorithm is presented for the problem. Computational results are presented for a well established breast cancer prognosis data set that is added artificial noise to create synthetic MI regression data. Finally, two open complexity results on feature selection for consistent biclustering and sparse representation for hyperplane clustering are presented.
General Note: In the series University of Florida Digital Collections.
General Note: Includes vita.
Bibliography: Includes bibliographical references.
Source of Description: Description based on online resource; title from PDF title page.
Source of Description: This bibliographic record is available under the Creative Commons CC0 public domain dedication. The University of Florida Libraries, as creator of this bibliographic record, has waived all rights to it worldwide under copyright law, including all related and neighboring rights, to the extent allowed by law.
Statement of Responsibility: by Omer Kundakcioglu.
Thesis: Thesis (Ph.D.)--University of Florida, 2009.
Local: Adviser: Pardalos, Panagote M.
Electronic Access: RESTRICTED TO UF STUDENTS, STAFF, FACULTY, AND ON-CAMPUS USE UNTIL 2011-08-31

Record Information

Source Institution: UFRGP
Rights Management: Applicable rights reserved.
Classification: lcc - LD1780 2009
System ID: UFE0024768:00001


This item has the following downloads:


Full Text

PAGE 1

1

PAGE 2

2

PAGE 3

3

PAGE 4

First,IwouldliketothankmysupervisorycommitteechairDr.PanosM.Pardalosfortheencouragement,guidance,andopportunitieshehasprovided.IamgratefultoDr.J.ColeSmith,Dr.JosephGeunes,andDr.MyT.Thaiforservingonmycommitteeandtheirvaluablefeedback.IamalsoappreciativeforthegreatcontributionfromDr.OnurSeref,Dr.WilcovandenHeuvel,Dr.H.EdwinRomeijn,Dr.GeorgiosPyrgiotakis,andArdaYenipazarli.IwouldliketothankmybelovedwifeAysanforhercaringlove,understanding,support,andencouragement.IamalsogratefultomymotherReyhan,myfatherTurgut,andmysisterGozdewhoseloveandsupporthavebeeninvaluable.IreservemymostspecialappreciationformyhighschoolmathteacherMehmetUz.Iamforeverindebtedtohimforhisenthusiasm,inspiration,andgreateortstoexplainthingsclearlyandsimply. 4

PAGE 5

page ACKNOWLEDGMENTS ................................. 4 LISTOFTABLES ..................................... 8 LISTOFFIGURES .................................... 9 ABSTRACT ........................................ 11 CHAPTER 1OPTIMIZATIONINPATTERNRECOGNITIONANDHEALTHCARE .... 13 1.1Introduction ................................... 13 1.2UnsupervisedLearning ............................. 14 1.3LinearClassication .............................. 20 1.3.1SupportVectorMachineClassiers .................. 20 1.3.2ApplicationsinNeuroscience ...................... 28 1.3.2.1Magneticresonanceimaging ................. 29 1.3.2.2Otherimagerytypes ..................... 30 1.3.2.3Featureselection ....................... 32 1.3.2.4Braincomputerinterface ................... 33 1.3.2.5Cognitiveprediction ..................... 36 1.3.2.6Othermodelingtechniques .................. 37 1.3.3SVMExtensionsandGeneralizations ................. 38 1.3.4OtherClassicationTechniques .................... 42 1.4LinearRegression ................................ 46 1.5BiomedicalTreatmentandOtherApplications ................ 49 1.6ConcludingRemarks .............................. 55 2CELLDEATHDISCRIMINATIONWITHRAMANSPECTROSCOPYANDSUPPORTVECTORMACHINES ......................... 56 2.1Introduction ................................... 56 2.2Methods ..................................... 61 2.2.1CellCultureProtocols ......................... 61 2.2.2ToxicAgentDosing ........................... 62 2.2.3ToxicAgentsStandards ......................... 63 2.2.4RamanSpectroscopyProtocolsandProcedures ............ 63 2.2.5SupportVectorMachines ........................ 65 2.3ResultsandDiscussion ............................. 67 2.3.1Triton-X100andEtoposideInducedCellularDeathDiscrimination 67 2.3.2CaseStudy:HeatInducedCellularDeath ............... 68 2.4ConcludingRemarks .............................. 69 5

PAGE 6

................. 71 3.1Introduction ................................... 71 3.2ACombinatorialSelectiveSVMProblem ................... 73 3.3AnAlternativeSelectiveSVMProblem .................... 76 3.4SelectionMethods ................................ 79 3.4.1IterativeElimination .......................... 81 3.4.2DirectSelection ............................. 82 3.5ComputationalResults ............................. 82 3.5.1SimulatedDataandPerformanceMeasure .............. 83 3.5.2IterativeEliminationvs.NaveElimination .............. 84 3.5.3DirectSelection ............................. 84 3.5.4AnApplicationtoaVisuomotorPatternDiscriminationTask .... 85 3.6Conclusion .................................... 89 4MULTIPLEINSTANCELEARNINGVIAMARGINMAXIMIZATION .... 91 4.1Introduction ................................... 91 4.2MarginMaximizationforMultipleInstanceData .............. 94 4.2.1ProblemFormulationforClassicationofMultipleInstanceData .. 94 4.2.2ComplexityoftheProblem ....................... 97 4.3ABranchandBoundAlgorithmforMIL ................... 101 4.3.1BranchingScheme ............................ 102 4.3.2BoundingScheme ............................ 105 4.4ComputationalStudy .............................. 106 4.5ConcludingRemarks .............................. 111 5SUPPORTVECTORREGRESSIONWITHMULTIPLEINSTANCEDATA .. 113 5.1Introduction ................................... 113 5.2ProblemFormulation .............................. 117 5.3SolutionApproach ............................... 121 5.3.1LowerBoundingScheme ........................ 121 5.3.2BranchingScheme ............................ 122 5.3.3HeuristicAlgorithm ........................... 123 5.4ComputationalResultsonBreastCancerDataSet ............. 124 5.5ConclusionsandFutureWork ......................... 128 6OTHERPATTERNRECOGNITIONTECHNIQUES ............... 129 6.1Thecomplexityoffeatureselectionforconsistentbiclustering ....... 129 6.1.1Introduction ............................... 129 6.1.2ComplexityResults ........................... 133 6.2SparseRepresentationbyHyperplanesFitting ................ 137 6.2.1Introduction ............................... 137 6.2.2ProblemFormulation .......................... 139 6.2.3ComplexityResults ........................... 141 6

PAGE 7

..................... 141 6.2.5Approximationresults ......................... 143 7CONCLUDINGREMARKSANDFUTUREWORK ............... 145 REFERENCES ....................................... 148 BIOGRAPHICALSKETCH ................................ 169 7

PAGE 8

Table page 4-1SizeinformationfortheMolecularActivityPredictionandtheImageAnnotationDataSets ....................................... 106 4-2Time(inseconds)toachievetheoptimalsolutionforOurBranchandBoundSchemevs.CPLEXDefaultBranchandBoundAlgorithmfortheImageAnnotationData .......................................... 107 4-3ComputationalResultsforOurBranchandBoundSchemevs.CPLEXDefaultBranchandBoundAlgorithmfortheMolecularActivityPredictionData(Musk1)with3minutestimelimit. .............................. 108 4-4ComputationalResultsforOurBranchandBoundSchemevs.CPLEXDefaultBranchandBoundAlgorithmfortheMolecularActivityPredictionData(Musk1)with30minutestimelimit. ............................. 108 4-5ComputationalResultsforOurBranchandBoundSchemevs.CPLEXDefaultBranchandBoundAlgorithmfortheImageAnnotationDatawith3minutestimelimit. ....................................... 109 4-6ComputationalResultsforOurBranchandBoundSchemevs.CPLEXDefaultBranchandBoundAlgorithmfortheImageAnnotationDatawith30minutestimelimit. ....................................... 110 4-7Benchmarkresultsfortestswithtimelimits. .................... 111 5-1Eectoffreeslackincreasefor100articialinstanceswithdierentdeviations. 125 5-2ComputationalResultsforOurBranchandBoundSchemevs.CPLEXDefaultBranchandBoundAlgorithmfor32features .................... 126 5-3ComputationalResultsforOurBranchandBoundSchemevs.CPLEXDefaultBranchandBoundAlgorithmfor10features .................... 127 8

PAGE 9

Figure page 1-1Anexampleofcheckerboardpatternafterbiclustering. .............. 17 1-2MaximalMarginClassier .............................. 22 1-3SoftMarginClassier ................................. 22 1-4ExamplesofnonlinearclassicationusingSVMwithGaussianKernel. ..... 28 2-1ThebasicprinciplesofRamanspectroscopy.a)Aphotonofacertainenergyandfrequencyinducesvibrationaltransitionsontheexaminedmolecule,bygivingaportionofitsenergy.Thetransitionoccursthroughavirtualstate,createdduetothepolarizabilityofthestudiedmolecule.Thescatteredphotonhaslowerenergythantheincidentandtheenergydierencein-betweenismeasuredbythedetector.ThisisreferredtoastheRamanShift.b)ThemicroRamanutilizesamicroscopeandfocusesthelaserthroughtheobjectivelensonthesample.ThescatteredphotonsarecollectedbythesameobjectivelensandtraveltheRamanspectrometer,wheretheyareanalyzedbyagratingandaCCDdetector. 57 2-2(a)Spectraacquiredfrom10dierentcellsafter24hrsonMgF2crystal.(b)Theaveragespectrumandstandarddeviationof30A549cellsspectra,after24hrsontheMgF2. ................................... 60 2-3DemonstrationofthepatternrecognitionbasedonSVMclassication.(a)Theclassicationoftheetoposideinducedapoptoticdeathafter24hrsexposure.(b)TheTritonX-100inducedapoptosisontheMgF2. .............. 64 2-4Theclassicationoftheheatingeect.(a)Theheatingincomparisonwiththehealthyandtheapoptotic,(b)theheatingincomparisonwiththehealthyandthenecrotic,(c)theheatingincomparisontothenecroticandtheapoptotic. .. 67 3-1Exampleshowingtherelationshipbetweenpenalizedslackandfreeslack .... 80 3-2Distributionofrestrictedfreeslackshowninthethirddimensiononatwodimensionaldata:(a)Topview,(b)Frontview ......................... 80 3-32-Ddatawithseparability(a)c=0,(b)c=r=2,(c)c=r 83 3-4NormalizeddierencebetweenIterativeEliminationandNaveeliminationmethods 84 3-5Eectoftheamountoffreeslackondatawithseparability(a)c=0,(b)c=r=2,(c)c=r 85 3-6Comparisonofiterativeeliminationanddirectselectionmethods ......... 86 9

PAGE 10

..................................... 88 3-8Rasterplotsfortheadaptivescalingfeatureselectionmethod(a):afterDTWapplied,(b):afterselectiveSVMapplied. ...................... 89 4-1Anexampleofcriticalbag. .............................. 104 5-1The"-insensitivebandforalinearregressionproblem. .............. 114 10

PAGE 11

Therstmaincontributionofthisdissertationistheapplicationofstandardsupportvectormachine(SVM)classiersfordeathcelldiscrimination.SVMsareasetofmachinelearningalgorithmswidelyusedforclassicationandregressionindatamining,machinevision,andbioinformatics.Inthisstudy,RamanspectroscopyisemployedtoassessthepotentialtoxicityofchemicalsubstancesandSVMclassierssuccessfullyassessthepotentialeectofthetesttoxins. Thesecondmaincontributionistheformulation,complexityresult,andanecientheuristicforSelectiveSVMclassiersthatconsideraselectionprocessforbothpositiveandnegativeclasses.SelectiveSVMsarecomparedwithotherstandardalignmentmethodsonaneuraldatasetthatisusedforanalyzingtheintegrationofvisualandmotorcortexesintheprimatebrain. ThethirdmaincontributionofthisdissertationistheextensionofSVMclassiersformultipleinstance(MI)datawhereaselectionprocessisrequiredforonlypositivebags.Dierentformulations,complexityresults,andanexactalgorithmarepresentedwithcomputationalresultsonpubliclyavailableimageannotationandmolecularactivitypredictiondatasets.MIpatternrecognitionmethodsarethenfurtherextendedtosupportvectorregression(SVR)andanexactalgorithmispresentedfortheproblem.ComputationalresultsarepresentedforawellestablishedbreastcancerprognosisdatasetthatisaddedarticialnoisetocreatesyntheticMIregressiondata. 11

PAGE 12

12

PAGE 13

Partofthischapterispresentedin( KundakciogluandPardalos 2009b )and( Serefetal. 2008a ).Basedon( Pyrgiotakisetal. 2009 ),Chapter 2 presentstheapplicationofstandardsupportvectormachine(SVM)classiersfordeathcelldiscrimination.Chapter 3 presentsSelectiveSVMclassiersandmainndingsofthischapterarepublishedin( Serefetal. 2009 ).Chapters 4 and 5 onmultipleinstancegeneralizationofsupportvectortechniquesarebasedon( Kundakciogluetal. 2009b )and( Kundakciogluetal. 2009a ),respectively.OneofthetwocomplexityresultsinChapter 6 isalsopresentedin( KundakciogluandPardalos 2009a ). Theremainderofthischapterisorganizedasfollows:Section 1.2 presentsthemostcommonlyusedunsupervisedlearningtechniques.Theclassicationtechniques,particularlySVMsareexplainedinSection 1.3 .Section 1.4 presentslinearregressionproblems.InSection 1.5 ,treatmentplanningandotheroptimizationapplicationsinbiomedicalresearcharementioned.Finally,Section 1.6 concludesthischapter. Werstexploreunsupervisedlearningtechniquesandproceedwithsupervisedlearningtechniques(i.e.,classicationandregression).Unsupervisedlearningisthecase 13

PAGE 14

Ontheotherhand,supervisedlearningreferstothecapabilityofasystemtolearnfromasetofinput/outputpairs.Theinputisusuallyavectoroffeaturesforanobject,andtheoutputisthelabelfortheclassthisobjectbelongsto.Thesetofobjectswithafeaturevectorandaclasslabeliscalledatrainingset.Basedonthisinformation,afunctionisderivedandappliedonatestset.Theoutputoftheregressionfunctionisacontinuousnumberwhichisusefultoforecastalabelthatcantakeanyvalue.Inclassication,ontheotherhand,theoutputisadiscreteclasslabelthatisusedforcategoricaldiscrimination.Thetermsupervisedoriginatesfromthefactthatthelabelsfortheobjectsinthetrainingsetareprovidedasinput,andthereforearedeterminedbyanoutsidesourcethatcanbeconsideredasthesupervisor. Oneofthemostwidelyusedclusteringtechniquesisk-meansclustering.Underlyingoptimizationproblemfork-meansclusteringforadatasetoffx1;:::;xNgcanbeformulatedas minJ=NXn=1KXk=1rnkkxnkk2 14

PAGE 15

Formulation( 1{1 )isusuallysolvedwithk-meansalgorithm,whichessentiallyisanexpectation-maximization(EM)appliedtomixturesofGaussians.Althoughthealgorithmdoesnotguaranteeoptimality,itssimplicity,eciency,andreasonablesolutionqualitymakesitdesirable.Themostcommonformofthek-meansalgorithmusesaniterativerenementheuristicknownasLloyd'salgorithm( Lloyd 1982 ).EMalgorithmconsistsofsuccessiveoptimizationswithrespecttornkandk.FirstsomeinitialvaluesofkarechosenandJisminimizedwithrespecttornkintheE(expectation)step.Inthesecondphase,whichistheM(maximization)step,Jisminimizedwithrespecttokkeepingrnkxed.SinceeachphasereducesthevalueoftheobjectivefunctionJ,theconvergenceofthealgorithmisassured. Luetal. 2004 )). Leeetal. ( 2008 )presentdetailsonclusteringapplicationsingenomics.Next,weexploreanotherclusteringmethodthatutilizeoptimizationtechniques. 15

PAGE 16

Biclusteringisappliedbysimultaneouspartitioningofthesamplesandfeatures(i.e.,columnsandrowsofmatrixA,respectively)intokclasses.LetS1;S2;:::;Skdenotetheclassesofthesamples(columns)andF1;F2;:::;Fkdenotetheclassesoffeatures(rows).BiclusteringcanbeformallydenedasacollectionofpairsofsampleandfeaturesubsetsB=f(S1;F1);(S2;F2);:::;(Sk;Fk)gsuchthatS1;S2;:::;Skfajgj=1;:::;n;k[r=1Sr=fajgj=1;:::;n;Sv\Su=;,v6=u;F1;F2;:::;Fkfaigi=1;:::;m;k[r=1Fr=faigi=1;:::;m;Fv\Fu=;,v6=u; Apair(Sk;Fk)iscalledabicluster.Theultimategoalinabiclusteringproblemistondapartitioningforwhichsamplesfromthesameclasshavesimilarvaluesforthatclass'characteristicfeatures.Thevisualizationofareasonablebiclusteringshouldrevealablock-diagonalor\checkerboard"patternasinFig. 1-1 .Adetailedsurveyonbiclusteringtechniquescanbefoundin( MadeiraandOliveira 2004 )and( Busyginetal. 2008 ). 16

PAGE 17

Anexampleofcheckerboardpatternafterbiclustering. Thecriteriausedtorelateclustersofsamplesandfeaturesmayhavedierentproperties.Mostcommonly,itisrequiredthatthesubmatrixcorrespondingtoabiclusteriseitheroverexpressed(i.e.,mostlyincludesvaluesaboveaverage),orhasalowervariancethanthewholedataset.However,biclusteringingeneralmayrelyonanykindofcommonpatternsamongelementsofabicluster. DivinaandAguilar-Ruiz ( 2006 )addressthebiclusteringofgeneexpressiondatawithevolutionarycomputation.Theirapproachisbasedonevolutionaryalgorithmsandsearchesforbiclustersfollowingasequentialcoveringstrategy.Toavoidoverlappingamongbiclusters,aweightisassignedtoeachelementoftheexpressionmatrix.Weightsareadjustedeachtimeabiclusterisfound.Thisisdierentfromothermethodsthatsubstitutecoveredelementswithrandomvalues.Experimentalresultsconrmthequalityoftheproposedmethodtoavoidoverlappingamongbiclusters. 17

PAGE 18

bioNMFestimatesbiclustersusinganovelmethodbasedonamodiedvariantoftheNon-negativeMatrixFactorizationalgorithm( Pascual-Montanoetal. 2006 ).Thisalgorithmproducesasuitabledecompositionasaproductofthreematricesthatareconstrainedtohavenon-negativeelements. Biclusteringisusedtoanalyzeoneorseveralofsixexpressionmatricescollectedfromyeast(see( Tanayetal. 2002 ; Segaletal. 2001 ))andtoanalyzeoneormoreofelevendierentexpressionmatriceswithhumangeneexpressionlevels(see( Tanayetal. 2002 ; Klugeretal. 2003 ; Busyginetal. 2002 )).Almostallthesedatasetscontainexpressiondatarelatedtothestudyofcancer.Somecontaindatafromcanceroustissuesatdierentstagesofthedisease;othersfromindividualssueringfromdierenttypesofcancer;andtheremainingdatasetscontaindatacollectedfromindividualswithaparticularcancerorhealthypeople.Thesedatasetsareusedtotesttheapplicabilityofbiclusteringapproachesinthreemajortasks:Identicationofcoregulatedgenes,genefunctionalannotation,andsampleclassication. Biclusteringtechniquesarealsoappliedtotheproblemofidenticationofcoregulatedgenes(seee.g.,( Segaletal. 2001 ; Ben-Doretal. 2002 )).Morespecically,theobjectiveistoidentifysetsofgenesthat,underspecicconditions,exhibitcoherentactivationsthatindicatecoregulation.Theseresultsareusedtosimplyidentifysetsofcoregulatedgenesor,moreambitiously,toidentifyspecicregulationprocesses.Alessobviousapplication 18

PAGE 19

Tanayetal. 2002 ; Segaletal. 2001 )).Theideaunderlyingthisapproachistousebiclusterswherealargemajorityofgenesbelongtoaspecicclassinthegeneontologytoguesstheclassofnonannotatedgenes. Anothersignicantareaofapplicationisrelatedwithsampleand/ortissueclassication( Busyginetal. 2002 ; Tanayetal. 2002 ; Klugeretal. 2003 ).Anexampleisthediagnosisofleukemiawherethegoalistoidentifydierentresponsestotreatmentandgroupofgenestobeusedasthemosteectiveprobe( Shengetal. 2003 ). Theapplicationsofbiclusteringmentionedaboveanalyzedatafromgeneexpressionmatrices.However,biclusteringcanalsobeusedintheanalysisofotherbiologicaldata. LiuandWang ( 2003 )applybiclusteringtoadrugactivitydataset.Thegoalinthisstudyistondgroupsofchemicalcompoundswithsimilarbehaviorswhensubsetsofcompounddescriptorsaretakenintoaccount. LazzeroniandOwen ( 2002 )analyzenutritionaldatatoidentifysubsetsoffoodswithsimilarpropertiesonasubsetoffoodattributes. In( Genkinetal. 2002 ),itisshownhowseveralproblemsindierentareasofdataminingandknowledgediscoverycanbeviewedasndingtheoptimalcoveringofaniteset.Manysuchproblemsariseinbiomedicalandbioinformaticsresearch.Forexample,proteinfunctionalannotationbasedonsequenceinformationisanubiquitousbioinformaticsproblem.Itconsistsofndingasetofhomolog(highsimilarity)sequencesofknownfunctiontoagivenaminoacidsequenceofunknownfunctionfromthevariousannotatedsequencedatabases.Thesecanthenbeusedascluesinsuggestingfurtherexperimentalanalysisofthenewprotein. Genkinetal. ( 2002 )showtheseoptimizationproblemscanbestatedasmaximizationofsubmodularfunctionsonthesetofcandidatesubsets.Thisgeneralizationmaybeespeciallyusefulwhenconclusionsfromdataminingneedtobeinterpretedbyhumanexpertsasindiagnostichypothesisgeneration,logicalmethodsofdataanalysis,conceptualclustering,andproteinsfunctionalannotations. 19

PAGE 20

( 1998 )presentanovelelectroencephalogram(EEG)-based,brain-stateidenticationmethod,whichcouldformthebasisforforecastingageneralizedepilepticseizure.25ratsareexposedtohyperbaricoxygenuntiltheappearanceofageneralizedEEGseizure.EEGsegmentsfromthepreexposure,earlyexposure,andtheperioduptoandincludingtheseizureareprocessedbythefastwavelettransform.Featuresextractedfromthewaveletcoecientsareinputtotheunsupervisedoptimalfuzzyclustering(UOFC)algorithm.TheUOFCisusefulforclassifyingsimilardiscontinuoustemporalpatternsinthesemistationaryEEGtoasetofclusterswhichmayrepresentbrain-states.Theunsupervisedselectionofthenumberofclustersovercomestheaprioriunknownandvariablenumberofstates.Theusuallyvaguebrainstatetransitionsarenaturallytreatedbyassigningeachtemporalpatterntooneormorefuzzyclusters. Next,westartsupervisedlearningtechniqueswithinclassicationframework. Inatypicalbinaryclassicationproblem,eachpatternvectorxi2Rn,i=1;:::;lbelongstooneoftwoclassesS+andS.Avectorisgiventhelabelyi=1ifxi2S+oryi=1ifxi2S.Thesetofpatternvectorsandtheircorrespondinglabelsconstitutethetrainingset.Theclassicationproblemconsistsofdeterminingwhichclassnewpatternvectorsfromthetestsetbelongto. Vapnik ( 1995 ),SVMsarethestate-of-the-artsupervisedmachinelearningmethods.SVMclassiersclassifypatternvectorswhichareassumedtobelongtotwolinearlyseparablesetsfromtwodierentclasses.Althoughthereareinnitelymanyhyperplanesthat 20

PAGE 21

SVMssolvebinaryclassicationproblembyndingahyperplane(;b)thatseparatesthetwoclassesinthetrainingsetfromeachotherwiththemaximummargin. Theunderlyingoptimizationproblemforthemaximalmarginclassierisonlyfeasibleifthetwoclassesofpatternvectorsarelinearlyseparable.However,mostofthereallifeclassicationproblemsarenotlinearlyseparable.Nevertheless,themaximalmarginclassierencompassesthefundamentalmethodsusedinstandardSVMclassiers.Thesolutiontotheoptimizationprobleminthemaximalmarginclassierminimizestheboundonthegeneralizationerror( Vapnik 1998 ).Thebasicpremiseofthismethodliesintheminimizationofaconvexoptimizationproblemwithlinearinequalityconstraints,whichcanbesolvedecientlybymanyalternativemethods( BennetandCampbell 2000 ). Ahyperplanecanberepresentedbyh;xi+b=0,whereisthen-dimensionalnormalvectorandbistheosetparameter.Thereisaninherentdegreeoffreedominspecifyingahyperplaneas(;b).Acanonicalhyperplaneistheonefromwhichtheclosestpatternvectorhasadistance1=kk,i.e.,mini=1;:::;mjh;xii+bj=1. Considertwopatternvectorsx+andxbelongingtoclassesS+andS,respectively.Assumingthesepatternvectorsaretheclosesttoacanonicalhyperplane,suchthath;x+i+b=1andh;xi+b=1,itiseasytoshowthatthegeometricmarginbetweenthesepatternvectorsandthehyperplanearebothequalto1=kk.Maximizingthegeometricinterclassmarginwhilesatisfyingthecanonicalseparatinghyperplaneconditionforthepatternvectorsresultsinthefollowingoptimizationproblem: 21

PAGE 22

2kk2 Fromthesolutionto 1{2 ,anewpatternvectorxcanbeclassiedaspositiveifh;xi+b>0,andnegativeotherwise. Mostreallifeproblemsarecomposedofnonseparabledatawhichisgenerallyduetonoise.Inthiscase,slackvariablesiareintroducedforeachpatternvectorxiinthetrainingset.TheslackvariablesallowmisclassicationsforeachpatternvectorwithapenaltyofC=2.InFig. 1-3 ,softmarginclassierisdemonstratedthatincurspenaltyformisclassiedpatternvectors. Figure1-2. MaximalMarginClassier Figure1-3. SoftMarginClassier Themaximummarginformulationcanbeaugmentedtosoftmarginformulationasfollows. 22

PAGE 23

2kk2+C In( 1{3 ),itisunnecessarytoenforcenonnegativityoftheslackvariablesexplicitlysinceasolutioncannotbeoptimalwheni<0foranypatternvector.Itshouldbenotedthatthe2-normoftheslackvariablesarepenalizedintheobjectiveof( 1{3 ).Analternativeformulationinvolvespenalizationofthe1-normofslackvariables.Inthiscase,nonnegativityconstraintsontheslackvariablesarenecessaryasfollows: min1 2kk2+ClXi=1i Itshouldalsobenotedthat( 1{3 )and( 1{4 )areessentiallyminimizationofaconvexfunctionswithlinearinequalityconstraints.Theseproblemscanbesolvedecientlybynumerousmethods(see( BennetandCampbell 2000 )).See( CristianiniandShawe-Taylor 2000 )forfurtherdetailsonformulationandimplementationdetailsofSVMs. Platt ( 1999 )solvesSVMproblemsbyiterativelyselectingsubsetsonlyofsize2andoptimizingthetargetfunctionwithrespecttothem.ThistechniqueiscalledtheSequentialMinimalOptimization(SMO).Ithasgoodconvergencepropertiesanditiseasilyimplemented.Thekeypointisthatforaworkingsetof2,theoptimizationsubproblemcanbesolvedanalyticallywithoutexplicitlyinvokingaquadraticoptimizer. 23

PAGE 24

2kk2+C (1{5) DierentiatingLwithrespecttotheprimalvariablesandb,andassumingstationarity,@L @=nXi=1yiixi=0 (1{6)@L @b=nXi=1yii=0 (1{7)@L @i=Cii=0 (1{8) SubstitutingtheexpressionsbackintheLagrangianfunction,thefollowingdualformulationisobtainedwhichrealizesthehyperplane=Pni=1yiixiwithgeometricmargin=1=kk. maxnXi=1i1 2nXi=1nXj=1yiyjijhxi;xji1 2CnXi=12i (1{9b)i0i=1;:::;n NotethatfromKarush-Kuhn-Tuckercomplementarityconditions,theconstraintsintheprimalproblemarebindingforthosewiththecorrespondingdualvariablei>0.The 24

PAGE 25

1{9b ),bcanbecalculatedas Alternatively,bcanbecalculatedusing (1{11) Thederivationforthe1-normdualformulationisverysimilartothatof2-norm.TheLagrangianfunctionforthe1-normSVMclassicationproblemisgivenasfollows. 2kk2+ClXi=1ilXi=1i[yi(h;xii+b)1+i]lXi=1rii @=lXi=1yiixi=0@L @b=lXi=1yii=0@L @i=Ciri=0: 25

PAGE 26

2lXi=1lXj=1yiyjijhxi;xji (1{12b)0iCi=1;:::;l Kerneltrickcanbeappliedandthe1-normformulationbecomes, maxlXi=1i1 2lXi=1lXj=1yiyjijhxi;xji (1{13b)0iCi=1;:::;l Thisproblemisequivalenttothemaximalmarginhyperplane,withtheadditionalconstraintthatalltheiareupperboundedbyC.Thisgivesrisetotheboxconstraintsthatisfrequentlyusedtorefertothisformulation,sincethevectorisconstrainedtolieinsidetheboxwithsidelengthCinthepositiveorthant.Thetrade-oparameterbetweenaccuracyandregularizationdirectlycontrolsthesizeofthei.Thatis,theboxconstraintslimittheinuenceofoutliers,whichwouldotherwisehavelargeLagrangemultipliers.Theconstraintalsoensuresthatthefeasibleregionisboundedandhencetheprimalalwayshasanon-emptyfeasibleregion. NotethatKarush-Kuhn-Tuckercomplementarityconditionscanbeusedtoobtainbsimilartothe2-normcase.However,in1-normcasewelookforbothconstraintstobebinding,i.e.,i>0;ri>0. 26

PAGE 27

(1{14) Thedecisionrulesgn(f(x))isequivalenttothehyperplanef(x)=Pni=1yiihx;xii+bandbcanalsobecalculatedusingyif(xi)=1forthosepatternvectorswith0
PAGE 28

where,isreferredtoasthebandwidth.Smallerbandwidthsarebetterinclassifyingintricatepatterns,butworseingeneralization. Figure1-4. ExamplesofnonlinearclassicationusingSVMwithGaussianKernel. ManyofthemedicalSVMapplicationsfocusonimageprocessingofmagneticresonanceimaging(MRI)datatodetectstructuralalterationsinthebrainovertime.Magneticresonancespectroscopy(MRS)datahasalsobeenanalyzedwithSVMsfora 28

PAGE 29

NumerousneurosciencestudieshaveutilizedSVMstoclassifyneuralstates.fMRIisausefulimagingmodalityforneuralstateclassicationduetoitsabilitytotrackchangesinbloodoxygenationleveldependentsignal,whichiscorrelatedwithbloodow.Inaddition,theelectroencephalogram(EEG)isahighlyusefulmeasureforSVMneuralstateclassicationduetoitsabilitytoquantifybrainelectricalactivity(e.g.,voltagedierencebetweenaregionofinterestandareferenceregiononthescalporinthebrain)withexceptionallyhightemporalresolution. Theremainderofthischapterwillprovideanoverviewofthestate-of-the-artSVMapplicationtodatafromvariousneuroimagingmodalitiesforthepurposesofmedicaldiagnosis,understandingthephysiologyofcognition,andclassicationofneuralstates. Leeetal. ( 2005 )usedSVMsandanewSVMbasedmethoddevelopedearliercalledsupportvectorrandomeldsforsegmentingbraintumorsfromMRimages. Rinaldietal. ( 2006 )classiedbraininammationinmultiplesclerosispatientsbasedontheperipheralimmuneabnormalitiesfromMRimagesusingnonlinearSVMs.Theydeterminedthatbraininammationinpatientswithmultiplesclerosisisassociatedwithchangesinsubsetsofperipherallymphocytes.Thus,SVMclassicationhelpeddetectapotentialbiomarkercandidatefor 29

PAGE 30

Quddusetal. ( 2005 )combinedSVMsandboosting( Schapire 2001 ),anothermachinelearningmethod,toperformsegmentation(vianonlinearclassication)onwhitematterlesionsintheMRIscans.Theircompositeclassicationmethodwasshowntobefasterandjustasreliableasmanualselection.Inanotherstudyby Martinez-Ramonetal. ( 2006 ),asimilarapproachwasusedtocreatesegmentsofthebrainwithrespecttotheirfunctions.Later,thesesegmentswereaggregatedusingboosting,whichisusedformulti-classSVMclassicationofanfMRIgroupstudywithinterleavedmotor,visual,auditory,andcognitivetaskdesign. KotropoulosandPitas ( 2003 )usedSVMsforsegmentationofultrasonicimagesacquirednearlesionsinordertodierentiatebetweenlesionsandbackgroundtissue.TheradialbasisfunctionSVMsoutperformedtheprocessofthresholdingofL2meanlteredimagesforvariouslesionsundernumerousrecordingconditions. Darbellayetal. ( 2004 )usedSVMandotherclassicationtechniquestodetectsolidorgaseousembolibytranscranialDoppler(TCD)ultrasound.Sincetheleadingcauseofcerebralinfarctionisduetotheextracranialatherosclerosis,rapidassessmentofthephysicalcharacteristicsofsolidobjectsinthebloodowisimportant.Darbellayet.al.demonstratedthatSVMscoulddistinguishbetweensolidandgaseousembolismsfromultrasonicmeasuresofthebloodstream.Amedicaldiagnosticdevicebasedonthistechnologymaybeabletopreventbraindamagebyallowingameanstoexpeditethediagnosisandtreatmentofembolisms. Devosetal. ( 2005 )devisedasystemthatcanautomaticallydiscriminatebraintumorsbasedondatafromMRIandMRSI,whichisafunctionofMRimagingthatproducesaspectroscopicproleofthescannedbrainregion.Inthisstudy,MRspectrafromMRSIwasusedforcomparisonwithlinearandnonlinearLeastsquaresSVM(LS-SVM)( SuykensandVandewalle 1999 ).Asimilarstudywas 30

PAGE 31

Devosetal. 2004 ). Lukasetal. ( 2004 ),onthecontraryusedlongechoHMRSsignalstoclassifybraintumorsintofourclasses:meningiomas,gliobastomas,astro-cytomasgradeIIandmetastases,withanumberofclassiersincludingstandardSVM,andLS-SVM.ThestudydemonstratedthatkernelbasedSVMswereabletodetecttumorswithoututilizingdimensionalityreductionandstillproduceaccuracycomparabletolineardiscriminantanalysis.Automatedtumordetectionalgorithmsareasought-aftertoolforassistingphysicianstomakemoreaccurateandrapiddetectionoftumors.Supportvectormachineclassiershavecontributedsignicantlyinthisarea.Menzeet.al.utilizedSVMclassicationofMRIimagestoserveasanautomateddiagnostictoolforthedetectionofrecurrentbraintumors( Menzeetal. 2006 ).TheyreportthatSVMamongothermethodswasableruleoutlipidandlactatesignalsasbeingtoounreliable,andthatcholineandN-acetylaspartatearethemainsourcesofinformation(mostimportantfeatures). Kelmetal. ( 2007 )performedanevaluationofnumerousautomatedprostatetumordetectionmethods.Theirstudydeterminedthatthepatternrecognitionmethods,suchasSVMclassication,wereabletooutperformquantizationmethodssuchasQUEST,AMARES,andVARPROforprostatetumordetection. Rapiddiagnosisofstrokeinpatientsisdesirableaspunctualtreatmentcanreducethechanceofpermanentbraindamage.Onepotentialmethodforrapidlydiagnosingstrokeistoexaminethecontentsofapotentialstrokepatientsbloodforbiomarkersindicativeofastroke. Pradosetal. ( 2004 )utilizedsupportvectormachinestohelpidentify14potentialbiomarkerswhichcouldbeusedtodistinguishthechemicalproleofacontrolsubject'sbloodfromthechemicalproleofanischemicorhemorrhagicstrokepatient.Surfaceenhancedlaserdesorption/ionizationmassspectometryisusedwithSVMsforfeatureselectiontondasmallsubsetofpotentialbiomarkersforearlystrokediagnosis.SomeimagesuseddonotdirectlycomefromMRscanningofthebrain. Glotsosetal. ( 2005a )and Glotsosetal. ( 2005b )useddigitizedimagesofbiopsiesofastrocytomastodetectbrain 31

PAGE 32

Laoetal. ( 2004 )toclassifybetweenmaleandfemalebrainandagedierentiationforoldadults.Althoughbrainimagesaretheprimarysourcesfordetectingbrainabnormalities,electricalbrainsignalscanalsobeused,suchasin( Lehmannetal. 2007 ).TheycomparedanumberofclassicationmethodsincludingSVMsforthedetectionAlzheimer'sdiseasefromtheEEGrecordingsanddiscoveredthattheSVMsperformancewassuperiortoothermethods. Fanetal. ( 2007 )introducedamethodforclassicationofschizophreniapatientsandhealthycontrolsfrombrainstructureswhosevolumetricfeaturesareextractedfromprocessedMRimages.ThebestsetofsuchfeaturesaredeterminedusinganSVM-basedfeatureselectionalgorithm,whichinreturnsignicantlyimprovedtheclassicationperformance. Yoonetal. ( 2007 )extractedprincipalcomponentsderivedfromcorticalthicknesstodierentiatebetweenhealthycontrolsandschizophrenicpatientsusingSVMsforuseasadiagnostictool. Yushkevichetal. ( 2005 )investigatedtheeectofabnormaldevelopmentandbrainstructureinpatientswithschizophreniawithrespecttothemorphologicalcharacteristicsandagerelatedchanges.TheyuseddeformedbraintemplatesofavarietyofsubjectimagesandusedSVMsforclassicationandfeatureselectiontoclassifybetweenpathologicalcasesfromthehealthycontrols.Asimilarstudy 32

PAGE 33

Liuetal. ( 2004 )forautomatedschizophreniaandAlzheimer'sdiseasedetection. FungandStoeckel ( 2007 )alsousedanSVMfeatureselectionalgorithmappliedtoSPECTperfusionimagingtodetectAlzheimer'sdisease.Theyuseda1-normlinearSVMclassier,whichisknowntogivesparsesolutions,whichinturnisusedforfeatureselection. Diabetesmellitus(DM)isacommondiseaseintheindustrializedcountriesanditisaprominentriskfactorforischemiccerebrovascularaccidents.Diabetesaloneisresponsiblefor7%ofdeathsinstrokepatients.Diabetesmellitusoftenresultsinbrainmicro-bloodowdisordersthatmaycausecerebralinfarction.However,assessingthefunctionofcerebralmicro-vesselsisdicult,sincetheyarelocatedwithinthebonyskull. Kalatzisetal. ( 2003 )performedastudywhereSVMwasappliedtodistinguishbetweenbloodowdatainpatientswithdiabetesversuscontrolsubjectsusingSPECTimagesfromcerebralabnormalities. Lietal. ( 2006b )usedSVMswithoatingsearchmethodtondrelevantfeaturesforassessingthedegreeofmalignancyinbraingliomafromMRIndingsandclinicaldatapriortooperations. Lietal. ( 2006a )furtherdevelopedanovelalgorithmthatcombinesbaggingofSVMswithembeddedfeatureselectionfortheindividualobservationsandcomparedthenewalgorithmusingpubliclyavailabledatasets. BCIdevicestypicallyutilizeneurophysiologicmeasureswhichcanbeacquiredforextendeddurationsandwithhightimetimeresolution.ThoughfMRIcanprovidehighlyusefulinformationaboutthetemporalhemodynamicresponsetochangesin 33

PAGE 34

Themajorityoftheapplicationsfocusonprostheticsforpatientssueringfromconditionssuchasamyotrophiclateralsclerosis(ALS),brainstemstroke,andbraininjury. Guigueetal. ( 2006 )developedanewgraphbasedmethodtoclassifynon-stationarysignals,eachwithadiscriminantwaveformwithrandomtimeandlocation.ThegraphbasedrepresentationwasusedtodeneaninnerproductbetweengraphstobeusedwithSVMs,whichincreasedtheaccuracyoftheBCIsystem. ManystudieshaveutilizedtheP300evokedpotentialasanSVMinputforclassifyingtextwhichisreadbytheuser(see( Thulasidasetal. 2006 )).TheP300evokedpotentialisanevent-relatedelectricalpotentialwhichappearsapproximately300msafteraninfrequenteventisperceived.AP300spellingdevicecouldprovideameansofcommunicationfordisabledindividualswhowouldotherwisebeunabletocommunicatewiththeworld.ThistechniqueisfrequentlyusedtoassesstheperformanceofBCIrelatedmethods. KaperandRitter ( 2004 )and Kaperetal. ( 2004 )usedSVMsonEEGrecordingsfromtheP300spellerBCIparadigmtoreachhighratesofdatatransferandgeneralization.Inthesestudiesthesubjectsweregivena6by6matrixwithashingsymbolsandwereinstructedtoattendtoonlyonesymboltocounthowmanytimesitappears.TheSVMclassierwasusedtodetectthisP300componentintheEEG,andwasshowntoperformwithhighaccuracy. Guanetal. ( 2005 )usedasimilarmentalspellerparadigmwithatargetandnon-targetsymbolsmovingfromrighttoleftinasmallwindowonacomputerscreenanddetectedsignicantdierencesintheEEGusingSVMs. 34

PAGE 35

Laletal. ( 2004 )investigatedthefeatureselectionandEEGsignalclassicationproblemswiththeSVM-basedRecursiveFeatureEliminationandZero-NormOptimizationmethods. Althoughnonlinearmethodscanprovidebetterresults,linearmethodsmaybepreferredwhereverpossible.However,complexcasesstillrequireecientmethodsinBCIwhichcanhandlenonlinearclassicationsuchasSVMs,asitwasshownin( Mulleretal. 2003 ). Garrettetal. ( 2003 )alsousedSVMstoclassifyEEGsignalsfromawell-knownEEGdataset(see( KeirnandAunon 1990 )),whichinvolvevedierentmentaltasks,andshowedthatlinearandnonlinearmethodsmayperformsimilarly. Liangetal. ( 2006 )usedtheExtremeLearningMachine(ELM)algorithmtoclassifyEEGsignalsfromthesamedatasetandshowedthatELMhassimilarperformancetoSVMs. SomeBCIsystemsaredevelopedusingnon-humansubjects.Ratsarethemostcommonsubjectsforthiskindofresearch.ABCIsystemadaptedforratswasdevelopedby Huetal. ( 2005 ),whoshowedthatSVMclassiersandprincipalcomponentanalysiscombinedwithaBayesianclassiermayperformequallywellforclassication.TheyalsoshowedSVMclassicationofneuronalspiketrainsallowidenticationofindividualneuronsassociatedwiththedecisionmakingprocess. Jakuczunetal. ( 2005 )appliedSVMstoclassifyhabituatedfromarousedstatesusingevokedpotentialsfromasinglebarrelcolumnoftherat'ssomatosensorycortex. Olsonetal. ( 2005 )usedspiketrainsfromratstopredictleftandrighthandcommandsinabinarypaddlepressingtaskperformedbyrats. OpticalmeasurementmethodshavealsodemonstratedsuccessinSVMBCIsystems. Sitarametal. ( 2007 )usednearinfraredspectroscopytodetectoxygenationinthelefthandversusrighthandmotorimageryofhumansubjectsfroma20-channelNIRSsystem. 35

PAGE 36

Astudyby AcirandGuzelis ( 2005 )investigatedtheutilityofSVMsforidentifyingEEGsleepspindles,anEEGpatternfoundinstage2ofsleep.ThestudydemonstratedthatradialbasisSVMsdetectedEEGsleepspindleswithhighaccuracy.ThisapplicationofSVMsmaybeusefulinanautomatedsleepstagingalgorithm. Epilepsyistheconditionofrecurrentseizures.Overthepastfewdecades,theareaofseizuredetectionandseizurepredictionusingquantitativeEEGanalysishasdrawngreatinterest. Chaovalitwongseetal. ( 2006 )developedaseizurepredictionalgorithmusingSVMswhichwasabletosuccessfullyclassifybetweenEEGpatternsassociatedwithaninterictal(\normal")brainstateandEEGpatternsassociatedwithapre-ictal(\seizureprone")state.Suchanalgorithmcouldbedevelopedtobecomethebasisforabedsideorimplantableseizurecontroldevice. BrouwerandvanEe ( 2007 )usedSVMsonfunctionalfMRIdatatopredictthevisualperceptualstatesfromtheretinotopicvisualcortexandmotion-sensitiveareasinthebrain. CoxandSavoy ( 2003 )investigatedvisualpresentationofvariouscategoriesofobjects.TheyusedSVMstoclassifytheimagesbasedonsimilarityfrompredeterminedregionsofvoxels(volumeelements)overashortperiodoftime.ThismethodwasshowntoproducesimilarresultsusingmuchlessdatathantraditionalfMRIdataanalysis,whichrequiresnumeroushoursofdataacrossmanysubjects. PessoaandPadmala ( 2007 )alsousedfMRIimagestopredictperceptualstates.SVMsareusedtodetectnear-thresholdfeardetection,andconcludedthatmultipleregionsofthebrainareinvolvedandthatbehavioralchoiceisdistributedacrosstheseregionstohelpmanagetheemotionalstimuliandpreparetheappropriateresponse. Shokeretal. ( 2005 )introducedahybridalgorithmwhichcombines 36

PAGE 37

Serefetal. ( 2007 )usedintracraniallocaleldpotentialrecordingsfrommacaquemonkeysanddevelopedaselectiveSVM-basedclassicationmethodinconjunctionwithSVM-basedfeatureselectionmethodstodetectcategoricalcognitivedierencesinvisualstimulibasedonsingle-trialsfromavisuomotortask. ( 2004 )studiedbrainanatomyandmodeledbrainfunctionfromMRimages.SVMscombinedwithmethodsfrominformationtheoryareusedinclusteringofvoxelsinthestatisticalmodelingofthefMRIsignals. LaConteetal. ( 2005 )usedSVMsinblockdesignfMRIandcomparedthemtocanonicalvarianceanalysis(CVA). Mour~ao-Mirandaetal. ( 2006 )investigatedtheperformanceofSVMswithtimecompressiononsingleandmultiplesubjects,andshowedthatthetimecompressionofthefMRIdataimprovestheclassicationperformance.Inasimilarstudy, Mour~ao-Mirandaetal. ( 2007 )introducedtimeseriesembeddingintotheclassicationframework.Inthisworkspatialandtemporalinformationwascombinedtoclassifydierentbrainstatesincognitivetasksinpatientsandhealthycontrolsubjects.Inastudyby Wangetal. ( 2003 ),anonlinearframeworkforfMRIdataanalysisisintroduced,whichusesspatialandtemporalinformationtoperformsupportvectorregressioninordertondthespatio-temporalautocorrelationsinthefMRIdata.Finally, Parraetal. ( 2005 )presentanarrayofmethodsas\recipes"forlinearanalysisofEEGsignals,amongwhichperformanceofSVMsiscomparedwithlogisticregression. See( Serefetal. 2008a )foradetailedsurveyonapplicationsofSVMinneuroscienceand( Leeetal. 2008 )forclassicationapplicationsingenomics.Next,wediscusssomegeneralizationsofthelinearclassicationproblem. 37

PAGE 38

Dietterichetal. ( 1997 )inthecontextofdrugactivitypredictionanddevelopedin( Auer 1997 ; LongandTan 1998 ).Theproblemconsistsofclassifyingpositiveandnegativebagsofpointsinthen-dimensionalrealspaceIRnwhereeachbagcontainsanumberofpoints.Patternsx1;:::xlaregroupedintobagsX1;:::XmwithXj=fxi:i2Ijg,Ijf1;:::;ng,andSjIj=f1;:::;ng.EachbagXjisassociatedwithalabelyj2f1;1g.Classicationisperformedsuchthatatleastonepointforeachpositivebagisclassiedaspositive,andallthepointsforallnegativebagsareclassiedasnegative.In( Dietterichetal. 1997 ),ahypothesisclassofaxis-parallelrectanglesareassumed,andalgorithmsaredevelopedtodealwiththedrugactivitypredictionproblem.Anecientalgorithmisdescribedin( LongandTan 1998 )forlearningaxis-alignedrectangleswithrespecttoproductdistributionsfromMIexamplesinthePACmodel. Auer ( 1997 )givesamoreecientalgorithm. BlumandKalai ( 1998 )showthatlearningfrommultiple-instanceexamplesisreducibletoPAC-learningwithtwosidednoiseandtothestatisticalquerymodel.Integerprogramming,expectationmaximization,andkernelformulationsarealsoproposedforMIclassicationproblem(seee.g.,( WangandZucker 2000 ; ZhangandGoldman 2001 ; Gartneretal. 2002 ; Andrewsetal. 2002 ; MangasarianandWild 2008 )). RayandCraven ( 2005 )provideabenchmarkofseveralmultipleinstanceclassicationalgorithmsandtheirnon-multiple-instancecounterparts. KundakciogluandPardalos ( 2008 )formulateMIclassicationproblemasthefollowingmixed0{1quadraticprogrammingproblem. min;b;;1 2kk2+ClXi=12i subjecttoh;xii+b1iM(1i)8i:i2Ij^yj=1 (1{16b) (1{16c) (1{16d) 38

PAGE 39

(1{16e) InthisformulationMisasucientlylargenumberthatensuresthattheconstraintisactiveifandonlyifi=1.iisabinaryvariablethatis1ifithinstanceisoneoftheactualpositiveexamples.In( KundakciogluandPardalos 2008 ),abranchandboundalgorithmisproposedforthisproblemthatoutperformsacommercialsolverforlargescaleproblems.MIclassicationhasverysuccessfulimplementationsinapplicationareassuchasdrugdesign(seee.g.,( Jainetal. 1994 ; Dietterichetal. 1997 ))andproteinfamilymodeling(seee.g.,( Taoetal. 2004 )). Serefetal. ( 2008c )introduceageneralizedsupportvectorclassicationframework,calledtheSelectiveSupportVectorMachine:LetSi,i=1;:::;lbemutuallyexclusivesetsofpatternvectorssuchthatallpatternvectorsxi;k,k=1;:::;jSijhavethesameclasslabelyi.Thegoalistoselectonlyonepatternvectorxi;kfromeachsetSisuchthatthemarginbetweenthesetofselectedpositiveandnegativepatternvectorsaremaximized.Thisproblemisformulatedasaquadraticmixedintegerprogrammingproblem,whichisageneralizationofthestandardsupportvectorclassiersandmultipleinstanceclassiers. min1 2kk2+ClXi=1jSijXk=12i;k Thisquadraticmixed0{1programmingproblemisshowntobeNP-hard(see( Serefetal. 2009 )).Analternativeapproachisproposedwiththefreeslackconceptasfollows: 39

PAGE 40

2kk2+C Dualformulationfor( 1{18 )isderivedfornonlinearclassication.Formulationswithfreeslackprovideexibilitytotheseparatinghyperplanetoidentifythepatternvectorswithlargerinterclassmargin.Iterativeeliminationanddirectselectionmethodsaredevelopedtoselectsuchpatternvectorsusingthealternativeformulations.Thesemethodsarecomparedwithanavemethodonsimulateddata. TheiterativeeliminationmethodforselectiveSVMisalsoappliedtoneuraldatafromavisuomotorcategoricaldiscriminationtasktoclassifyhighlycognitivebrainactivitiesin( Serefetal. 2007 ).Standardandnovelkernelbasednonlinearclassicationmethodsareappliedonaneuraldatarecordedduringavisuomotortaskperformedbyamacaquemonkey.Thestagesofthevisuomotortaskaretheinitialresponseofthevisualcortex,thecategoricaldiscriminationofthevisualstimuliandtheappropriateresponseforthevisualstimuli.AstandardSVMclassierandanSVMbasedadaptivescalingmethodareusedforfeatureselectioninordertodetectrelevanttimeintervalsandtheirspatialmappingonthebrain.TherstandthethirdstagesofthevisuomotortaskaredetectablewiththestandardSVMclassier.However,forthesecondstage,SVMclassierperformspoorly.DynamicTimeWarping(DTW)isalsoappliedinordertoreducethetemporalvariances.MotivatedbytheimprovementintherststageafterDTW,selectiveSVMisapplied.ItisshownthattheresultsobtainedafterselectiveSVMareexceptionallybettercomparedtoDTWforbothclassicationaccuracyandfeatureselection.Theresults 40

PAGE 41

Thereareanumberofbiomedicalstudieswhichintroducedierentmethodsforimagesegmentation.Segmentationistheprocessofpartitioningadigitalimageintodierentsectionsinordertochangetherepresentationoftheimage.Thisnewrepresentationmayinvolvecertaincharacteristicsintheimagesuchascurves,edges,color,intensityortexture.Segmentedimagesareusuallyusedtodeterminebrainabnormalitiesusingimageclassication,anditisshownthatSVMsperformverywell. Leeetal. ( 2005 )useSVMsandanewSVMbasedmethodcalledsupportvectorrandomeldsforsegmentingbraintumorsfromMRimages. Rinaldietal. ( 2006 )classifybraininammationinmultiplesclerosispatientsbasedontheperipheralimmuneabnormalitiesfromMRimagesusingnonlinearSVMs.Theydeterminethatbraininammationinpatientswithmultiplesclerosisisassociatedwithchangesinsubsetsofperipherallymphocytes.Thus,SVMclassicationhelpsdetectapotentialbiomarkercandidatefortheprognosisofpatientsintheearlystagesofmultiplesclerosis. Quddusetal. ( 2005 )combineSVMsandboosting,anothermachinelearningmethod,toperformsegmentation(vianonlinearclassication)onwhitematterlesionsintheMRIscans.Theircompositeclassicationmethodisshowntobefasterandjustasreliableasmanualselection.Inanotherstudyby Martinez-Ramonetal. ( 2006 )asimilarapproachisusedtocreatesegmentsofthebrainwithrespecttotheirfunctions.Later,thesesegmentsareaggregatedusingboosting,whichisusedformulti-classSVMclassicationofanfMRIgroupstudywithinterleavedmotor,visual,auditory,andcognitivetaskdesign. Automatedtumordetectionalgorithmsareasought-aftertoolforassistingphysicianstomakemoreaccurateandrapiddetectionoftumors.Supportvectormachineclassiershavecontributedsignicantlyinthisarea. Menzeetal. ( 2006 )utilizeSVMclassicationofMRIimagestoserveasanautomateddiagnostictoolforthedetectionofrecurrentbraintumors.TheyreportthatSVMamongothermethodsisabletoruleoutlipidand 41

PAGE 42

Diabetesmellitus(DM)isacommondiseaseintheindustrializedcountriesanditisaprominentriskfactorforischemiccerebrovascularaccidents.Diabetesaloneisresponsiblefor7%ofdeathsinstrokepatients.Diabetesmellitusoftenresultsinbrainmicro-bloodowdisordersthatmaycausecerebralinfarction.However,assessingthefunctionofcerebralmicro-vesselsisdicult,sincetheyarelocatedwithinthebonyskull. Kalatzisetal. ( 2003 )applySVMstodistinguishbetweenbloodowdatainpatientswithdiabetesversuscontrolsubjectsusingSPECTimagesfromcerebralabnormalities. Lietal. ( 2006b )useSVMswithoatingsearchmethodtondrelevantfeaturesforassessingthedegreeofmalignancyinbraingliomafromMRIndingsandclinicaldatapriortooperations. Lietal. ( 2006a )developanovelalgorithmthatcombinesbaggingofSVMswithembeddedfeatureselectionfortheindividualobservationsandcomparethenewalgorithmusingpubliclyavailabledatasets.Next,wecontinuediscussingSupportVectormethodwithintheregressionframework. Givenaclassicationofthesamples,Sr,letS=(sjr)nkdenotea0{1matrixwheresjr=1ifsamplejisclassiedasamemberoftheclassr(i.e.,aj2Sr),andsjr=0otherwise.Similarly,givenaclassicationofthefeatures,Fr,letF=(fir)mkdenotea0{1matrixwherefir=1iffeatureibelongstoclassr(i.e.,ai2Fr),andfir=0otherwise.Constructcorrespondingcentroidsforthesamplesandfeaturesusingthesematricesasfollows 42

PAGE 43

Theelementsofthematrices,cSiandcFj,representaverageexpressionofthecorrespondingsampleandfeatureinclass,respectively.Inparticular, Notethattheconstructedclassicationofthefeatures,^Fr,isnotnecessarilythesameasclassicationFr.Similarly,onecanusetheelementsofmatrixCFtoclassifythesamples.Samplejisassignedtoclass^rifcFj^r=maxfcFjg,i.e., Asbefore,theobtainedclassication^SrdoesnotnecessarilycoincidewithclassicationSr. BiclusteringBisreferredtoasaconsistentbiclusteringifrelations( 1{21 )and( 1{22 )holdforallelementsofthecorrespondingclasses,wherematricesCSandCFaredenedaccordingto( 1{19 )and( 1{20 ),respectively. Adatasetisbiclustering-admittingifsomeconsistentbiclusteringforthatdataexists.Furthermore,thedatasetiscalledconditionallybiclustering-admittingwithrespecttoagiven(partial)classicationofsomesamplesand/orfeaturesifthereexistsaconsistentbiclusteringpreservingthegiven(partial)classication. 43

PAGE 44

Italsofollowsfromtheconicseparabilitythatconvexhullsofclassesdonotintersect(see( Busyginetal. 2005 )). Bydenition,abiclusteringisconsistentifFr=^FrandSr=^Sr.However,agivendatasetmightnothavetheseproperties.Thefeaturesand/orsamplesinthedatasetmightnotclearlybelongtoanyoftheclassesandhenceaconsistentbiclusteringmightnotbeconstructed.Insuchcases,onecanremoveasetoffeaturesand/orsamplesfromthedatasetsothatthereisaconsistentbiclusteringforthetruncateddata.Selectionofarepresentativesetoffeaturesthatsatisescertainpropertiesisawidelyusedtechniqueindataminingapplications.Thisfeatureselectionprocessmayincorporatevariousobjectivefunctionsdependingonthedesirablepropertiesoftheselectedfeatures,butonegeneralchoiceistoselectthemaximalpossiblenumberoffeaturesinordertoloseminimalamountofinformationprovidedbythetrainingset. Givenasetoftrainingdata,constructmatrixSandcomputethevaluesofcSiusing( 1{19 ).Classifythefeaturesaccordingtothefollowingrule:featureibelongstoclass^r(i.e.,ai2F^r),ifcSi^r>cSi,86=^r.Finally,constructmatrixFusingtheobtainedclassication.Letxidenoteabinaryvariable,whichisoneiffeatureiisincludedinthecomputationsandzerootherwise.Consistentbiclusteringproblemisformulatedasfollows. maxxmXi=1xi 44

PAGE 45

1{23 )isprovedtobeNP-hard(see( KundakciogluandPardalos 2009a ))andisaspecictypeoffractional0{1programmingproblem,whichcanbesolvedusingtheapproachdescribedin( Busyginetal. 2005 ).Whenithasafeasiblesolution,thecorrespondingfeatureselectionmakesthedatasetconditionallybiclustering-admittingwithrespecttothegivenclassicationofsamples. Twogeneralizationsof( 1{23 )andanimprovedheuristicprocedureisproposedin( Nahapetyanetal. 2008 ).Inthismethod,alinearprogrammingproblemwithcontinuousvariablesissolvedateachiteration.Numericalexperimentsonthedata,whichconsistsofsamplesfrompatientsdiagnosedwithacutelymphoblasticleukemia(ALL)oracutemyeloidleukemia(AML)diseases(see( Golubetal. 1999 ; Ben-Doretal. 2000 2001 ; Westonetal. 2000 ; XingandKarp 2001 )),conrmthatthealgorithmoutperformsthepreviousresultsinthequalityofsolutionaswellascomputationtime. Busyginetal. ( 2007a )applybiclusteringtoanalyzetheelectroencephalogram(EEG)data.SomebiomedicalapplicationsofbiclusteringareDNAmicroarrayanalysisanddrugdesign(seee.g.,( Busyginetal. 2008 ; MadeiraandOliveira 2004 ; Tanayetal. 2004 )).However,biclusteringisshowntobealsousefulforfeatureselectionwhichisthemajorconcernofmanybiomedicalstudies(see( Busyginetal. 2007a )).RevealingsubsetsofchannelswhoseLyapunovexponentsconsistentlychangewithswitchingtheVNSstimulationONandOFFisclaimedtobeverymuchinlinewithdiscoveringupregulatedanddownregulatedgenesinamicroarraydataset.Therefore,eachEEGchannelisrepresentedasafeatureanddatasamplestakenwithinthestimulationperiodsversussamplestakenoutsideoftheseperiodsareanalyzed.Itisshownthatthemethodofbiclusteringisabletoperformsuccessfulfeatureselection.Anotherstudywhereepilepsytreatmentwithvagusnervestimulationisby Uthmanetal. ( 2007 ).See( Chaovalitwongseetal. 2007 ; Sabesanetal. 2008 )forotherapplicationsofoptimizationtoepilepticbraindisorders. 45

PAGE 46

BertsimasandShioda ( 2007 )introducemixed-integeroptimizationmethodstotheclassicalstatisticalproblemsofclassicationandregressionandconstructasoftwarepackagecalledCRIO(classicationandregressionviaintegeroptimization).CRIOseparatesdatapointsintodierentpolyhedralregions.Inclassication,eachregionisassignedaclass,whileinregressioneachregionhasitsowndistinctregressioncoecients.ComputationalexperimentationswithgeneratedandrealdatasetsshowthatCRIOiscomparabletoandoftenoutperformsthecurrentleadingmethodsinclassicationandregression.Theseresultsillustratethepotentialforsignicantimpactofintegeroptimizationmethodsoncomputationalstatisticsanddatamining. LogicalAnalysisofData(LAD)isatechniquethatisusedforriskpredictioninmedicalapplications(see( Alexeetal. 2003 )).Thismethodisbasedoncombinatorialoptimizationandbooleanlogic.ThegoalisessentiallyclassifyinggroupsofpatientsatlowandhighmortalityriskandLADisshowntooutperformstandardmethodsusedbycardiologists. Anothersupervisedlearningmethodisby Mammadovetal. ( 2007a )whereamulti-labelclassierisconsidered.See( LeeandWu 2007 ; Lee 2008 )forsurveysonclassicationanddiseasepredictionmethodsthatusemathematicalprogrammingtechniques. TheSupportVectormethodcanalsobeappliedtothecaseofregression,maintainingallthemainfeaturesthatcharacterizethemaximalmarginalgorithm.Thismethodis 46

PAGE 47

Aswiththeclassication,wemotivatetheapproachbyseekingtooptimizethegeneralizationboundsgivenforregression.Theserelyondeningalossfunctionthatignoreserrorswithinacertaindistanceofthetruevalue.Thistypeoffunctionisreferredtoasan-insensitivelossfunction.Withmanyreasonablechoicesoflossfunction,thesolutionischaracterizedastheminimumofaconvexfunctional.Anothermotivationforconsideringthe-insensitivelossfunctionisthatitwillensuresparsenessofthedualvariablessimilartotheclassicationcase.Theideaofrepresentingthesolutionbymeansofasmallsubsetoftrainingpointshasenormouscomputationaladvantages.-insensitivelossfunctionhasthatsparsenessadvantage,whilestillensuringexistenceofaglobalminimumandtheoptimizationofareliablegeneralizationbound. Inthissection,werstdescribethe-insensitivelossandthenderivetwoapproachesfromtheboundsrelatingtothe1-normor2-normofthelossvector. Thelinear-insensitivelossfunctionL1(x;y;f)isdenedas andthequadratic-insensitivelossfunctionL2(x;y;f)isdenedas 47

PAGE 48

2kk2+ClXi=1(2i+^2i) (1{24a)subjectto(h;xii+b)yi+ii=1;:::;l TheSVRproblemforthelinear-insensitivelossfunctionis min1 2kk2+ClXi=1(i+^i) (1{25a)subjectto(h;xii+b)yi+ii=1;:::;l Linearregressionisusedinidenticationofadirectlyproportionalrelationshipbetweentwophysicochemicalpropertiesanddrugactivityprediction(see( Jones 2002 )).Breastcancerprognosisisstudiedextensivelyusinglinearprogrammingandaregressionframeworkin( Streetetal. 1995 )and( Mangasarianetal. 1995 ). FortheSupportVectorRegression,thederivationofthedualissimilartothatofSVMclassiers.Forthesakeofcompleteness,weonlypresentthedualforSVR. 2-normdualforSVRisasfollows: maxlXi=1yiilXi=1jij1 2lXi=1lXj=1ijK(xi;xj)+1 (1{26b) 48

PAGE 49

2lXi=1lXj=1ijK(xi;xj) (1{27a)subjecttolXi=1i=0 (1{27b)CiCi=1;:::;l Letasystembesetby 49

PAGE 50

ForshorttermmaximalLyapunovexponent(STLmax)wecantakereasonabletinsteadofexternallimit.Inreallifeweoftendealwithonedimensionaltimeseriesofnoisydata(suchasEEGsignal)insteadofexplicitsystemofequations. Wolfetal. ( 1985 )suggestanalgorithmforLyapunovExponentcalculationfromtimeseries. Pardalosetal. ( 2004 )and Chaovalitwongseetal. ( 2006 )usemodicationofWolfsalgorithmdescribedin( Iasemidis 1991 )forSTLmaxcalculationthathandlesnoisynon-stationarydata. Sincethebrainisanonstationarysystem,algorithmsusedtoestimatemeasuresofthebraindynamicsshouldbecapableofautomaticallyidentifyingandappropriatelyweighingexistingtransientsinthedata.Inachaoticsystem,orbitsoriginatingfromsimilarinitialconditions(nearbypointsinthestatespace)divergeexponentially(expansionprocess).RateofdivergenceisanimportantaspectofthesystemdynamicsandisreectedinthevalueofLyapunovexponents.Duringthelastdecade,advancesinstudyingbrainareassociatedwithextensiveuseofEEGwhichcanbetreatedasthequantitativerepresentationofthebrainfunction.EEGdataessentiallyrepresenttimeseriesrecordedfromtheelectrodeslocatedindierentfunctionalunitsofbrain.WeutilizetheconceptofT-indextomeasureentrainmentoftwobrainsitesatatimemoment.T-indexattimetbetweenelectrodesitesiandjisdenedas 50

PAGE 51

Oneaspectoftheanalysisoftheepilepticbrainisndingamaximumcliqueinthisgraph.Itprovidesuswiththelargestsetofcriticalelectrodesitesmostentrainedduringtheseizure.Ifthenumberofcriticalsitesissetequaltok,wecanformulatetheproblemofselectingtheoptimalgroupofcriticalsiteasamulti-quadratic0{1programmingasfollows. Letxi2f0;1gdenoteifsiteiisselected.aijistheT-indexbetweensitesiandjduringtheseizurepoint.bijistheT-indexbetweensitesiandj10minutesaftertheonsetofseizure. minxTAx (1{30c)x2f0;1gn Pardalosetal. ( 2004 )developanovellinearizationtechniquetoreformulateaquadraticallyconstrainedquadratic0{1programmingproblemasanequivalentmixedintegerprogramming(MIP)problem.Thepracticalimportanceofthisreformulationisthatnumberof0{1variablesremainsthesameandnumberofadditionalcontinuousvariablesisO(n),wherenisthenumberof0{1variables. 51

PAGE 52

Manyoptimizationalgorithmsaredevelopedforthetreatmentplanninginradiationtherapywhichemploytechniquessuchasmultiobjectiveoptimization(see( Lahanasetal. 2003a b )),investigatingtradeosbetweentumorcoverageandcriticalorgansparing(see( Craftetal. 2006 )),linearprogramming(see( Lodwicketal. 1999 )),mixed-integerprogramming(see( LeeandZaider 2003 ; Leeetal. 2001 )),non-linearprogramming(see( BillupsandKennedy 2001 ; Ferrisetal. 2001 )),simulatedannealing(see( Webb 1991 )),andinverseplanningwithageneticalgorithm-basedframework(see( Bevilacquaetal. 2007 )). Recently, Menetal. ( 2007 )considertheproblemofintensity-modulatedradiationtherapy(IMRT)treatmentplanningusingdirectapertureoptimization.Incontrasttotheheuristicapproaches,anexactapproachisusedthatexplicitlyformulatestheuencemapoptimization(FMO)problemasaconvexoptimizationproblemintermsofall 52

PAGE 53

Forareviewonoptimizationmethodsinradiationtherapy,thereaderisreferredto( Shepardetal. 1999 ; Ehrgottetal. 2008 ). Acostaetal. ( 2008 )studytheinuenceofdosegridresolutiononbeamselectionstrategiesinradiotherapytreatmentdesign. Censoretal. ( 2006 )studyauniedmodelforhandlingdoseconstraintsandradiationsourceconstraintsinasinglemathematicalframeworkbasedonthesplitfeasibilityproblem.See( Brandeauetal. 2004 )fordescriptionofothertreatmentproblems.Futureresearchdirectionsforradiationtherapyarediscussedin( Leeetal. 2001 ). Anotherproblemthathasbeenextensivelystudiedisthenon-uniqueprobeselection.Thisproblemconsistsofselectingoligonucleotideprobesforuseinhybridizationexperimentsinwhichtargetvirusesorbacteriaaretobeidentiedinbiologicalsamples.Thepresenceorabsenceofthesetargetsisdeterminedbyobservingwhetherselected 53

PAGE 54

Ragleetal. ( 2007 )presenttherstexactmethodforndingoptimalsolutionstothenon-uniqueprobeselectionproblemwithinpracticalcomputationallimits,withouttheapriorieliminationofcandidateprobes.Previousmethodshaveemployedheuristicstondapproximatesolutionsthatarenotprovablyoptimal,andasaresult,noknowledgehasbeenobtainedregardingthequalityofthosesolutionsrelativetooptimality.Thecomputationalresultsshowthatthemethodcanndtheoptimalsolutionwithin10minutes,andiscapableofreducingthenumberofprobesrequiredoverstate-of-the-artheuristictechniquesbyasmuchas20%. Usingd-disjunctmatrix, Thaietal. ( 2007b )presenttwo(1+(d+1)logn)-approximationalgorithmstoidentifyatmostdtargetsforthenon-uniqueprobeselectionproblem.Basedontheirselectednon-uniqueprobes,thedecodingalgorithmswithlineartimecomplexityarealsopresented.Theproposedalgorithmswithfaulttolerantsolutionscanidentifyatmostdtargetsinthepresenceofexperimentalerrors. OtheroptimizationbasedstudiesinbiomedicineareinDNAmicroarrayexperiments(see( UgurandWeber 2007 ; Kochenbergeretal. 2005 ; Busyginetal. 2007b )),intensitymodulatedprotontherapy(see( Pugfelderetal. 2008 )),ultrasound-mediatedDNAtransfection(see( ZarnitsynandPrausnitz 2004 )),proteindesignandgenenetworks(see( Menesesetal. 2007 ; Balasundarametal. 2005 ; Fungetal. 2005 ; Strickleretal. 2006 ; McAllisteretal. 2007 ; Donahueetal. 2007 ; Thaietal. 2007a )),humanmotionanalysis(see( Dariush 2003 )),imaging(see( Dubeetal. 2007 ; CarewandYuan 2007 ; Louis 2008 )),ultrasoundsurgery(see( Huttunenetal. 2008 )),cornealrotation(see( KarpouzasandPouliquen 1991 )),drugdesign(see( Mammadovetal. 2007b ; Pardalosetal. 2005 )),vaccineformularies(see( Halletal. 2008 ))andqueryoptimizationindatabaseintegration(see( Sujansky 2001 )). Marchuk ( 1997 )developsmathematicalmodelsofinfectiousdiseases,antiviralimmuneresponseandantibacterialresponse.Thesemodels 54

PAGE 55

Greenbergetal. 2004 ). 55

PAGE 56

Inthepresentstudy,Ramanspectroscopyisemployedtoassessthepotentialtoxicityofchemicalsubstances.Havingseveraladvantagescomparedtoothertraditionalmethods,Ramanspectroscopyisanidealsolutionforinvestigatingcellsintheirnaturalenvironment.Inthepresentwork,wecombinethepowerofspectralresolutionofRamanwithoneofthemostwidelyusedmachinelearningtechniques.Supportvectormachines(SVMs)areusedinthecontextofclassicationonawellestablisheddatabase.Thedatabaseisconstructedonthreedierentclasses:healthycells,Triton-X100(necroticdeath),andetoposide(apoptoticdeath).SVMclassierssuccessfullyassessthepotentialeectofthetesttoxins(TritonX-100,etoposidestaurosporine).Thecellsthatareexposedtoheat(45oC)aretestedusingtheclassicationrulesobtained.Itisshownthattheheateectresultsinapoptoticdeath,whichisinagreementwithexistingliterature. Kanducetal. 1999 2003 2005 ),andincertainpathogenicinfections( NavarreandZychlinsky 2000 ).Usuallyapoptosisismarkedbycaspaseactivation,chromatincondensation,andtheformationofapoptoticbodies.Autophagicismarkedbyautophagicengulfmentoforganellesandparticles.Cellsdyingbynecrosisdisplayorganelleswellingwiththeeventuallossofplasma,membraneintegrity,andsubsequentinammation.Monitoringthecelldeathprocess,therefore,isanimportantstepinunderstandingthe 56

PAGE 57

Figure2-1. ThebasicprinciplesofRamanspectroscopy.a)Aphotonofacertainenergyandfrequencyinducesvibrationaltransitionsontheexaminedmolecule,bygivingaportionofitsenergy.Thetransitionoccursthroughavirtualstate,createdduetothepolarizabilityofthestudiedmolecule.Thescatteredphotonhaslowerenergythantheincidentandtheenergydierencein-betweenismeasuredbythedetector.ThisisreferredtoastheRamanShift.b)ThemicroRamanutilizesamicroscopeandfocusesthelaserthroughtheobjectivelensonthesample.ThescatteredphotonsarecollectedbythesameobjectivelensandtraveltheRamanspectrometer,wheretheyareanalyzedbyagratingandaCCDdetector. 57

PAGE 58

Jaeschkeetal. 2004 ),whichmakescharacterizingcelldeathevenmoredicult.Awiderangeofcytotoxicityassaysarepresentlyinuseforthedeterminationofcellviability;however,thesetechniqueshaveshortcomings.Theyaredestructive,timeconsuming,andexpensive.Currentassaysdependonlargepopulationsandcannotmeasurethehealthofindividualcells.Furthermore,manyfactorsmustbeconsideredwheninterpretingresults.Becausecytotoxicityassaysrelyonchemicalsandbiomarkers,problemsmayariseduetounwantedinteractionsduringpharmaceuticaltesting.Furthermore,inthecasewhereassaysaredependentuponenzymaticreactions(e.g.,MTT,LDH),resultsmaybeskewedbypromiscuousenzymaticinhibitors.Specicityissuescanalsoleadtocomplicationsintheinterpretationofresults. Kanducetal. ( 2002 )comparedmanyoftheconventionalcytotoxicityassaysandndthatthereportedviabilityoftreatedcellsdiereddependingontheassayused.Moreover,alargenumberofcellsisrequiredtodeterminetheexactcellulardeathandtoconcludeonthetoxicologicalassessment. Ramanspectroscopy,awellestablishedanalyticaltool,isbeingemployedasanalternativeforstudyingcellhealth.Itdoesnotsharemanyofthedisadvantagesinherentintraditionalcytotoxicityassaysdescribedabove( Notingheretal. 2002 2003 ).Ramanspectroscopyreliesontheinelasticscatteringoflightonmatter.ItisacomplementarytechniquetotheInfraRed(IR)spectroscopy(FTIR,DRIFTetc.).ThebasicdierenceliesonthepolarizabilityofthemoleculethatisrequiredbyRamanvs.thepolaritythatisrequiredbytraditionalIRspectroscopy.Inbothcases,thematerialisradiatedwithalightofspecicfrequencythatinducesanelectrontransitiontoadierentvibrationalstate,withanenergylossofthephoton.InthecaseofRamanspectroscopy,duetothepolarizabilityofthemolecule,thetransitionoccursthroughanintermediatestate, 58

PAGE 59

Verrieretal. 2004 ).Ithasbeensuccessfullyusedtoevaluatethetoxicityofpharmaceuticals( Owenetal. 2006 ),toxins( Notingheretal. 2004 ),andmorerecentlythetoxiceectofparticles( Pyrgiotakisetal. 2008 ). WhileRamanspectroscopyhasmanyadvantages,thereexistsonelargedrawback;highlycomplexspectra.Becausethespectrumofacellcontainsinformationfromallcellularcomponents,detectingminutechangesfromonespectrumtothenextcanbeadauntingtask.Traditionally,peakttinghasbeenusedtoanalyzeRaman(andFTIR)spectra.Peakttingreliesontherecognitionofpeaksrepresentingcertaincellularcomponentsandcorrelatingtheirrelativepeakintensitiestotheirbiochemicalconcentrationswithinthecell.Therelativechangesinpeakintensityovertimeareindirectresponsetothechangingbiochemicalandbiophysicalfactorsthatarerelatedtothehealthviability,andeventuallytothecelldeathtypeandprocess.However,duetothelargenumberofoverlappingpeaks,thistaskbecomesverytediousandtimeconsuming.Thetraditionalmethodologyforanalyzingthespectraincludesanelaborateseriesofalgorithms.Aseriesofspectraisobtained(seeFigure 2-2 (a))andaseriesofmathematicalproceduresisfollowedtoremovethebaseline,theuorescence,tonormalizethespectra,tocalculatetheaverageandthestandarddeviation(seeFigure 2-2 (b)).Furthermore,theanalysisdependsonthepresumptionthatonealreadyknowswhich 59

PAGE 60

(b) Figure2-2. (a)Spectraacquiredfrom10dierentcellsafter24hrsonMgF2crystal.(b)Theaveragespectrumandstandarddeviationof30A549cellsspectra,after24hrsontheMgF2. peaksarediscriminant,andthosepeaksmustbeprevalentspectralfeatureswithlimitedinterferencefrombackgroundnoiseandoverlappingpeaks.Thus,itiscriticaltodevelopamethodthatisapplicableforhighthroughputscreening,issimplerthanpeakttingtoexecute,andutilizestheentirespectruminsteadofpredeterminedsections.Moreover,anautomatedmethodisdesiredthatcanderiveresultswithoutanymanualspectraprocessing. 60

PAGE 61

Theremainderofthechapterisorganizedasfollows:Section 2.2 presentsthemethodsusedandthedetailsfortheexperiments.ComputationalresultsarepresentedinSection 2.3 .Section 2.4 givesconcludingremarksanddirectionsforfutureresearch. 2.2.1CellCultureProtocols Giardetal. ( 1973 )throughexplantscultureoflungcarcinomatoustissuefroma58-year-oldCaucasianmale. Thegrowthmediaismadeby89%RPMI-1640withL-glutamine(fromCellgro;Cat#:25-053-CI),10%FetalBovineSerum(fourtimeslteredthrough0.1mlter,fromHyclone;Cat.#:SH30070.03)and1%antibiotic-antimycoticsolution(fromCellgro;Cat.#:30-004-CL).Thecellsaregrownwithcompletegrowthmediaina25cm2cell 61

PAGE 62

62

PAGE 63

YogalingamandPendergast 2008 ).TheRPMI1640providesallthenecessarygrowthhormonesandsugarsessentialforthecellviability. Karpinichetal. 2002 ).Ithasalsobeenshowntoupregulatep53,aninitiatorofapoptosis( HuangandPlunkett 1992 ; Solovyanetal. 1998 ).TritonX-100isusedasabenchmarkinvariousassays,sinceitcanrapturethecellularmembraneandresultsinthenecroticdeathofthecells.TritonX-100exposureisreportedtoincreasetheexpressionofapoptosisinhibitorsandisknowntosolubilizeanddestabilizethecellmembrane( Boesewetteretal. 2006 ). Thetoxinconcentrationsareselectedbasedontheliteraturethatsuggestthatthesevalueswillimpactthecells,butnotcatastrophically.Fortheexperiments,theagentsconcentrationis100MforTriton-X( Notingheretal. 2003 )and80M( YogalingamandPendergast 1997 ; Karpinichetal. 2002 ; Owenetal. 2006 )fortheetoposide.Theseconcentrationsareexpectedtoinducedamageinthecellswithoutcompletelylysingthecellsintherst24hrsoftheexperiment.Thesolutionispreparedimmediatelypriortodosing.Theetoposideisinsolubleinwater,soastocksolutionispreparedwith100mMofetoposideindi-Methyl-sulfo-oxide(DMSO). 63

PAGE 64

Figure2-3. DemonstrationofthepatternrecognitionbasedonSVMclassication.(a)Theclassicationoftheetoposideinducedapoptoticdeathafter24hrsexposure.(b)TheTritonX-100inducedapoptosisontheMgF2. mW)produceslaserlightof785nmanddoesnotcauseanydamagetothecellsevenafter40minexposuretime.TheMgF2plateafterrinsingwiththeHBSSismovedonaDeltaTCultureDish(fromBiotechs;Cat#:04200415C),and2mlofRPMI1640isadded.Thedishisplacedontoaheatingstage(DeltaT4CultureDishController,Biotechs,Butler,PA,USA)tomaintain37oCthroughtheentiremeasurementandinducetherequiredheating.Thelaserisfocusedoverthecenterofthecell,withthehelpofthecrosshair,throughtheLeicamicroscope.Thespotsizeis2040mwhenfocusedondrySiwaferand2030mwheninwaterbasedliquid.Itcanbeassumedthereforethatthelaserspotcancoverthewholecell(2020mwhen80%conuent,4040mwhenisolated).Althoughthelaserspotcanbelargerthanthecell,sincetheintensityofthelaserfollowsaGaussiandistributionaroundthegeometriccenter,thepartsthatarenotfromthemeasuredcell,arenotcontributingsignicantly.However,fortheisolatedcellstherelativepositionofthelasercanpotentiallyeectthespectrumandthereforetheyarenotincludedinthisstudy.The785nmlaserbeampassesthroughthe63xwater 64

PAGE 65

TheRPMImediawithorwithoutthepresenceofthevarioustoxinsdoesnotinuencethespectra.Inpreviouspublications,wehavedevelopedanalgorithmthattakesthebackground,theuorescence,andthenormalizationofthespectraintoaccount( Maquelinetal. 1999 ; Bhowmicketal. 2008 ).Inthepresentwork,thebackgroundisobtainedandsubtractedfromthespectrafollowingnonlinearsubtraction.Thespectrumbeforeandafterareusedforclassication,butthereisnosignicantdierenceinthenalresults.Thereforeweomitthisstepsinceitislikelythattheseprocesseshinderorremoveinformation,essentialfortheclassicationtechniques. Shawe-TaylorandCristianini 2004 ). SVMshaveawidespectrumofapplicationareassuchaspatternrecognition( LeeandVerri 2002 ),textcategorization( Joachims 1998 ),biomedicine( Brownetal. 2000 ; CifarelliandPatrizi 2007 ; Noble 2004 ; Serefetal. 2008b ),brain-computerinterface( Laletal. 2004 ; Garciaetal. 2003 ),andnance( Huangetal. 2004 ; TrafalisandInce 2000 ).Thetrainingisperformedbyminimizingaquadraticconvexfunctionthatissubjecttolinearconstraints.Quadraticprogramming(QP)isanextensivelystudiedeldofoptimizationtheoryandtherearemanygeneralpurposemethodstosolveQP 65

PAGE 66

BennetandCampbell 2000 ).Thesegeneralpurposemethodsaresuitableforsmallsizeproblems.Inordertosolvelargeproblems,fastermethodsarerequired.ForSVMclassiers,thesefastermethodsinvolvechunking( OsunaandGirosi 1997 )anddecomposition( Platt 1999 )techniques,whichusesubsetsofpointstondtheoptimalhyperplane.SVMLight( Joachims 1999 )andLIBSVM( Hsuetal. 2004 )areamongthemostfrequentlyusedsoftwareapplicationsthatusechunkinganddecompositionmethodseciently. Theexperimentalprocedurestartsbyconstructingabasic561301matrixbasedonthetwoclassesthedatamustbediscriminatedto.Thediscriminationisdonealwaysamongtwodierentclasses.The56columnsconsistof25fromclass1,25fromclass2,3testsubjectsfromclass1,and3testsubjectsfromclass2.Therowsrepresentthedierentfrequencies(600cm1-1800cm1withstep0.92cm1),whilethecolumnsarespectraofdierentcellsindierentenvironmentalconditions.Therearethreedierentmatricesstudied;Necrotic(NC):TritonX-100andControl,Apoptotic(AC):EtoposideandControl,andNecroticvs.Apoptotic(NA):TritonX-100andEtoposide.Forthevalidationofclassicationalgorithm,inadditiontothe50datainstancesofthelibrary,weuse3controlcellsand7cellswithtoxins. Torepresenttheresults,weplotthepointswithx-axistobethesampleIDandy-axisthedistancefromthehyperplanethatseparatesthetwoclasses.SVMlight( Joachims 1999 )isusedtotrainthedatainthisstudy.Linearclassiersareusedandthetrade-oparameterCissetafterleave-one-outcrossvalidationtechniqueisemployed.Whenusingtheleave-one-outmethod,SVMistrainedmultipletimes,usingallbutoneoftheinstancesinthetrainingsetthatisselectedrandomly.ThehighestpredictionaccuracyisachievedforC=1000fortrainingsetsofallexperiments.Therefore,wesetparameterCto1000inourcomputationalstudies. 66

PAGE 67

(c) Figure2-4. Theclassicationoftheheatingeect.(a)Theheatingincomparisonwiththehealthyandtheapoptotic,(b)theheatingincomparisonwiththehealthyandthenecrotic,(c)theheatingincomparisontothenecroticandtheapoptotic. 2.3.1Triton-X100andEtoposideInducedCellularDeathDiscrimination 67

PAGE 68

Ideally,thedataisexpectedtohaveafunctionalmarginofatleast1.However,sincethecellsarenotfromthesamepassageandthereareotherconditions(humidity,smallalterationsatthefullgrowthmedia)thatcaninducevariations,itisnotalwayspossibletokeepthedistancemorethan1(orlessthan-1).Furthermore,theinteractionofeachcellindividuallywiththetoxinisnotthesame,duetothecomplexityofitsnature.AsitcanbeseeninFigures 2-3 (a)and 2-3 (b),SVMclassierssuccessfullydiscriminatethecontrolcellsfrometoposideandTritonX-100,respectively.Thedistancefromtheseparatinghyperplaneandsmallvariationshowcasetheclassicationandprovetheabilityofthealgorithmtoclassifytheobtainedspectraintwoclasses. Robinsonetal. 1974 ; Gerneretal. 1975 )andchemotherapy( Hildebrandtetal. 2002 ; Robinsonetal. 1974 ).Ithasbeenestablishedthatelevatedtemperaturesalonecausecelldeathinapredictablemannerthatislinearlydependentonexposuretimeandisnon-linearlydependentontemperature( SaparetoandDewey 1984 ; Dewhirstetal. 1984 ).Avarietyofcelllines,includingA549,havebeenreportedtoundergoapoptosis( Hayashietal. 2005 ; Armouretal. 1993 )duringmildheattreatmentandnecrosisduringprolongedorintensiedexposure( Tadashietal. 2004 ; Prasadetal. 2007 ; Hildebrandtetal. 2002 ).Inthisstudy,heattreatmentat45oCover30minutesisusedtotestthepredictivestrengthofthemodelbyusingadierentcelldeathtrigger 68

PAGE 69

Assumingthattheeectoftheheatistheunknownsample,wetrytoattemptclassication,amongallthethreeclasses,healthy,apoptotic,andnecrotic.Sincetherearemanydrawbacksofhyperplane-basedmulti-classlearningtechniques( Bishop 2006 ),pairwiseexaminationisperformedacrossallthepossiblecombinations.Sointhisparticularcase,weexamineHealthyNecrotic,HealthyApoptotic,andApoptoticNecrotic.InFigure 2-4 (a)aretheresultsoftheheatingexperimentasitisattemptedforapoptoticdeathvs.healthycells.Theheatingexperimentisclassiedasapoptoticdeath.Asitcanbeseeninthegure,mostofthesamplesarelyingbetween0.3-1.0inregardstothedistancefromthehyper-plane.Thenextstepistocheckthecaseofthenecroticcelldeathvs.healthycells.Inthiscase,theresultsoftheclassicationappeartobescatteredamongbothclasses,whilethetestinstancesareclassiedcorrectly(seeFigure 2-4 (b)).Thisisaninconclusiveresultsincethereisnoparticulartrend.Thiscanhappen,eitherbecausetheclassicationiswrong,orbecausesomeoftheinstancesareindeednecrotic.Ifthesecondistrue,thenaclassicationamongapoptoticvs.necroticwillclassifythemagainasnecrotic.Thereforethelastclassicationisperformedamongthenecroticandapoptoticcells.Figure 2-4 (c)showsthatalltheheatingspectraareclassiedagainasapoptotic.Sointhecaseswheretheapoptoticdeathisusedasoneofthetwoclasses,theheatexposedcellsareclassiedasapoptotic. Widjajaetal. 69

PAGE 70

, a )thatcombinethesetwoelds,itistherstknownattempttowardstheissueofcelldeathidentication.TheclassicationmodelsbuiltwithRamanspectraldatacanbeusedtodiscriminatebetweenminutebiochemicaldierenceswithincellsrapidly,inrealtime,andinanondestructiveandnoninvasivemanner.Averyimportantaspect,furtherhighlightingtheresults,isthesuccesstoclassifybiologicalsamplesthatcanpresentalteration,anddierencesintheirsignalduetoexternal(orinternal)parameters.Thosealterationsaremanifestedtothecurrentprojectbythevariationsinthedistancefromtheseparatinghyperplane.Cases,however,inrealbiologicalsystemsalwaysexhibitminutevariationsandalteration.Thesuccessofthistechnique(Raman-SVM)isshowcasedbythefactthatalthoughitisabletodetecttheseminutechanges,itdoesnotpreventthealgorithmfromcorrectlyclassifyingtheresults. Thisstudysetsthefoundationfordevelopingdiagnostictoolsforcancerorothergeneticdiseases,thecellularresponsetochemotherapyandthetoxicityassessmentofdrugsandparticles.Futureworkwillexplorethesensitivityofthistechniqueintermsofitsabilitytodistinguishnerbiochemicalorbiophysicalprocessesrelatedtocelldeathsuchascaspaseactivationorchromatincondensation.Itiscriticaltoexpandthismethodologytoincludemorethantwoclasseswithoutpairwisecomparisonandthereforebeingabletodistinguishimmediatelybetweenvariousstagesofthecell. 70

PAGE 71

Inthisstudy,weintroduceageneralizedsupportvectorclassicationproblem:LetXi,i=1;:::;nbemutuallyexclusivesetsofpatternvectorssuchthatallpatternvectorsxi;k,k=1;:::;jXijhavethesameclasslabelyi.Selectonlyonepatternvectorxi;kfromeachsetXisuchthatthemarginbetweenthesetofselectedpositiveandnegativepatternvectorsaremaximized.Thisproblemisformulatedasaquadraticmixed0-1programmingproblem,whichisageneralizationofthestandardsupportvectorclassiers.Thequadraticmixed0-1formulationisshowntobeNP-hard.Analternativeapproachisproposedwiththefreeslackconcept.Primalanddualformulationsareintroducedforlinearandnonlinearclassication.Theseformulationsprovideexibilitytotheseparatinghyperplanetoidentifythepatternvectorswithlargemargin.Iterativeeliminationanddirectselectionmethodsaredevelopedtoselectsuchpatternvectorsusingthealternativeformulations.Thesemethodsarecomparedwithanavemethodonsimulateddata.Theiterativeeliminationmethodisalsoappliedtoneuraldatafromavisuomotorcategoricaldiscriminationtasktoclassifyhighlycognitivebrainactivities. 3{1 ,isthequadraticoptimizationproblemthatmaximizesthemarginbetweenpositiveandnegativepatternvectors.ThestandardSVMproblemcanbeconsideredasaspecialcaseofselectiveSVMclassicationwheret=1. 71

PAGE 72

Dietterichetal. 1997 ).However,MILinvolvesclassifyingpositiveandnegativebagsofpatternvectors,whereeachbagcontainsanumberofpatternvectorssharingthesamelabel.GivenaclassicationfunctionforMILproblem,atleastonepatternvectorinapositivebagshouldbeclassiedcorrectlyforthatbagtobecountedascorrectlyclassied.Foranegativebagtobecorrectlyclassied,allofthepatternvectorsinitshouldbeclassiedcorrectly.TheMILproblemistondaclassicationfunctionthatobtainsahighclassicationaccuracyforthebags.Theobjectiveinselectiveclassicationisnotclassifyingthebags.Itis,rather,toselectasinglepatternvectorfromeachset(bag)tomaximizethemarginbetweentheselectedpositiveandnegativepatternvectors. Theselectiveclassicationproblemposesahardcombinatorialoptimizationproblem.Inthischapter,weshowthattheselectiveSVMproblemisNP-hard.Weprovidealternativeapproachestothehardselection.Weintroducetherestrictedfreeslackconcept,whichprovidesexibilitytothehyperplanebydecreasingtheinuenceofthepatternvectorsthataremisclassiedorveryclosetothehyperplane.Theresultingoptimizationproblemisalsoconvexandquadraticwithlinearconstraints,andthereforecanbekernelizedthroughitsLagrangiandual.Wepresenttheoreticalresultsonhowtherestrictedfreeslackisdistributedamongthepatternvectors.Weintroducealgorithmsbasedontheseresults.Thesealgorithmsaretestedonsimulateddataandcomparedwithnaivemethods.ThisalgorithmisalsotestedonaneuraldatabasetoimprovetheclassicationaccuracyandtheperformanceofanSVMbasedfeatureselectionmethod. Theremainderofthechapterisorganizedasfollows.WeintroducetheconceptofselectiveclassicationinSection 3.2 ,wherethecombinatorialselectiveclassicationproblemisshowntobeNP-hard.ThealternativeformulationsarediscussedinSection 3.3 .InSection 3.4 ,dierentalgorithmsbasedontheselectiveclassicationformulationsarepresented.InSection 3.5 ,computationalresultsfromtheapplicationoftheproposed 72

PAGE 73

3.6 min1 2kk2+C Notethatthisformulationissimilarto( 1{3 ),exceptfortheextratermM(1i;k)in( 3{1b )andthenewconstraints( 3{1c )and( 3{1d ).Misasucientlylargepositivenumber.Binaryvariablesi;kindicatewhetherkthpatternvectorfromsetiisselectedornot.Notethatwheni;k=0,therightsideof( 3{1b )becomessucientlysmallsuchthattheconstraintisalwayssatised,whichisequivalenttoremovingthepointfromthe 73

PAGE 74

3{1c )ensuresthatonlyonepatternvectorisincludedfromeachset. ItisclearthatforsucientlyhighpenaltyC,theselectiveSVMformulationcanbeconsideredasahardselectionproblemwithouttheslackvariablesi,whosesolutionwouldprovideahyperplanethatcancompletelyseparatetheselectedpositiveandnegativepatternvectors.Now,considerthefollowingdecisionproblem: LetXi=fxi;jgdenoteasetofd-dimensionalvectors,wherej=1;:::;t.Assumethattherearensuchsetsandallvectorsxi;jineachsetXiarelabeledwiththesamelabelyi2f+1;1g.Letdenoteaselectionwhereasinglevectorxi;jisselectedfromeachsetXi.Isthereaselectionsuchthatallpositiveandnegativepatternvectorscanbeseparatedbyahyperplane(;b)? Proof. 2nXi=1si?(3{2) ThisproblemisknowntobeNP-complete( GareyandJohnson 1979 ).Now,letusconsiderthefollowingequivalentformulationofthePARTITIONproblem:GivenasetofnpositiveintegersS=fs1;s2;:::;sng,doesthereexistavectorw2f1;+1gn,suchthatPni=1siwi=0? SupposewearegivenaninstanceofthePARTITIONproblem.Letd=n+1.Leteibead-dimensionalvectorwhosecomponentsarezerosexceptforcomponenti,whichis 74

PAGE 75

(i) Fori=1;:::;naddthesetsofvectors,fei;eigwithpositivelabels,fei;eigwithnegativelabels. (ii) Addthesetsofvectorsfen+1;en+1gwithpositivelabels,fen+1;en+1gwithnegativelabels. (iii) Addthesetsofvectorsfs+;s+gwithpositivelabels,fs;sgwithnegativelabels. Notethat,regardingitem i oftheconstruction,followingarethecorrespondinginequalitiesintheselectiveSVMformulation. (3{3a)wi+b1M(1i;2) (3{3b)i;1+i;2=1 (3{3c)wib1M(10i;1) (3{3d)wib1M(10i;2) (3{3e)0i;1+0i;2=1 (3{3f) Itcanbeveriedthat( 3{3a )-( 3{3b )and( 3{3d )-( 3{3e )haveafeasiblesolutionifandonlyif (3{4a)i;1=0i;1=0andi;2=0i;2=1: Fromitem ii oftheconstructionwehave (3{5a)wn+1b1 (3{5b) 75

PAGE 76

iii oftheconstructionwehave (3{6a)nXi=1siwi+wn+1b1 (3{6b) Takingintoaccountourobservationsabove,from( 3{6a )-( 3{6b )wecanconcludethattheobjectivePdi=1w2iisequaltodifandonlyifPni=1siwi=0. Thepresentedreductionispolynomial,therefore,thedecisionversionoftheselectiveSVMproblemisNP-complete. 3{1 )isNP-hard. WeproviderestrictedfreeslackamountofVforallpatternvectors.NotethataverysmallamountoffreeslackwouldmakeaverysmalldierencecomparedtothestandardSVMformulation,whereasaverylargefreeslackwouldyieldtrivialsolutions.Dependingontheselectionscheme,theamountoftotalfreeslackmayvary.Thecorrespondingformulationisgivenasfollows. 76

PAGE 77

2kk2+C NotethatthisformulationissimilartothestandardSVMformulationwithaconvexquadraticobjectivefunctionandlinearconstraints.TheLagrangiandualofthisformulationcanalsobederivedfornonlinearclassication.Thedualformulationisgivenasfollows. max(nXi=1tXk=1i;k1 2nXi=1tXk=1nXj=1tXl=1yiyji;kj;lhxi;k;xj;li1 2CnXi=1tXk=12i;kV) (3{8b)0i;ki=1;:::;n;k=1;:::;t: Fromcomplementaryslackness,wecandirectlyndbfromaconstraintthatsatises0
PAGE 78

3{7 ),withanobjectivefunctionvaluez. 3{7c )isbinding,i.e.,P(i;k)2Di;k=Vintheoptimalsolution. Proof. 3{8 ),whichforcesthedualobjective,andthustheprimalobjectivetobe0.Thisimplies=0,thusacontradiction. 3{7b )isbinding,i.e.,yi(h;xi;ki+b)=1i;ki;k. Proof. 1 ). 2 ).Let, Notethat0max,0i;kand0i;kvaluessatisfyLemmas( 1 )and( 2 ),Theorem( 2 ),anddonotviolateanyoftheconstraintsin 3{7 .ItiseasytoverifythatP(i;k)2Di;k=0. LetSDbethesetofindiceswith0i;k=0,andz0=kk2+P(i;k)2D0i;k2.Theobjectivefunctionvalue,z=kk2+P(i;k)2D2i;k,canbewrittenas, 78

PAGE 79

Notethati;k08(i;k)2D,bydenition,and0i;k=0max8(i;k)2DnS.Since,X(i;k)2S0i;ki;k0maxX(i;k)2Si;k; Fromexpression( 3{10 ),zcanonlybeoptimalifandonlyifi;k=0,andthusi;k=0i;kandi;k=0i;kforall(i;k)2D. Theorem( 2 )basicallystatesthatallpatternvectorswithafunctionalmargindi;k=yi(h;xi;ki+b)<1incurpenaltyfori;k=minf1di;k;maxg.Forpatternvectorsi;k=maxthefreeslackisequaltoi;k=1maxdi;k,thesumofwhichisalwaysequaltoV.ExamplesaredemonstratedinFigure 3-1 Thisresultimplies,withoutlossofgenerality,thefreeslackforapositivepatternvectorisdistributedlinearlyproportionaltoitsdistancefromthehyperplaneh;xi;ki+b=1max,asshowninFig. 3-2 .Inthisgure,freeslackforeachpointisshowninthethirddimension.Thegureontheleftisthetopviewshowingtheoriginaldata.Theguresontherightarefrontviews,onlyshowingtheamountofslackassigned. Thisresultleadstoafewpossiblemethodstomaximizethemarginbetweentheselectedpoints,whicharediscussedinthenextsection. 79

PAGE 80

Exampleshowingtherelationshipbetweenpenalizedslackandfreeslack (a)(b) Figure3-2. Distributionofrestrictedfreeslackshowninthethirddimensiononatwodimensionaldata:(a)Topview,(b)Frontview remainderofthechapter.Themethodsintroducedinthissectionarebasedonthesoftselectionformulationandtheresultwhichstatesthattheamountoffreeslackacquiredbyeachpatternvectorislinearlyproportionaltoitsdistancefromthehyperplane.Twomethodsareproposed:aniterativeeliminationmethod,andadirectselectionmethod. 80

PAGE 81

1 1 ,X(0)istheoriginalinputofpatternvectors,yisthevectoroflabelsforeachsetXi,nistotalthefreeslackamountprovidedforthesoftselectionproblem,(;b)isthehyperplane,theamountyi(h;xi;ki+b)isthedistanceofxi;kfromthehyperplane(;b),t(0)istheinitialnumberofpatternvectorsineachset,andristhesetofpatternvectorstoberemovedateachiteration.Notethatthisdistancecanbenegativeifthepatternvectorismisclassied. 81

PAGE 82

2 .ThenotationissimilartothatofAlgorithm 1 3.5ComputationalResults 3.4 .Westartwiththedescriptionofhowthedataisgeneratedandhowtheperformancesofthemethodsarecompared.Then,wepresentcomparativeresultsoftheiterativeeliminationmethod,directselectionmethodandthenaveeliminationmethod. 82

PAGE 83

3-3 threeinstanceswithdierentseparabilityvalues(a)c=0(b)c=r=2and(c)c=rareshownford=2. (a)(b)(c) Figure3-3. 2-Ddatawithseparability(a)c=0,(b)c=r=2,(c)c=r 83

PAGE 84

3.5.1 ford=2;4;:::;20andc=0;r=2;r.Notethatt=6andfreeslackparameterp=1(perset).Foreachcombinationoftheparameters100instancesofsimulateddatasetsaregeneratedandtestedusingiterativeeliminationandnaveelimination.TheresultsarenormalizedasexplainedinSection 3.5.1 .LetzPFSandzNdenotetheaveragenormalizedobjectivefunctionvaluesobtainedfromiterativeeliminationandnaveelimination. InFig. 3-4 ,thevalueszNzPFS,ford=2;4;;20areplottedforeachcvalue.Itisclearfromthegurethatasthedimensionalityincreasestheiterativeeliminationissignicantlysuperiortothenaveeliminationmethod.Thedierencebecomesmoreapparentforhigherlevelsofdataseparation.Thisresultclearlyshowsthesuccessoftheiterativeeliminationduetotheexibilityoftheseparatinghyperplaneincorporatedbytherestrictedfreeslack. Figure3-4. NormalizeddierencebetweenIterativeEliminationandNaveeliminationmethods 3.5.1 ford=2;4;;20andc=0;r=2;rfortotalslackparameterp=1;;5with100instanceseach.Therearet=6patternvectorsineachset.InFig. 3-5 ,theeectoftheincreaseintotalslackisshown.Thethreegraphsinthegureareintheorderofincreasingseparationinthedata.Ineachgraph,theobjectivefunctionvaluesforthehighesttotalslackparameter 84

PAGE 85

(a)(b)(c) Figure3-5. Eectoftheamountoffreeslackondatawithseparability(a)c=0,(b)c=r=2,(c)c=r 3-6 ,theperformancesofiterativeeliminationanddirectselectionareshownwiththevalueszDSzIE,wherezIEandzDSarethenormalizedobjectivefunctionvaluesobtainedfromiterativeeliminationanddirectselectionmethods,respectively.Theresultsuctuateandthereisnosignicantdominanceofonemethodovertheother.However,weobservefromthegurethat,ontheaverage,theiterativeeliminationmethodperformsslightlybetterthanthedirecteliminationmethod. 85

PAGE 86

Comparisonofiterativeeliminationanddirectselectionmethods multiplechannelsimplantedindierentcorticalareasofamacaquemonkeyduringavisualdiscriminationtask.Thistaskinvolvesrecognizingavisualgostimuliwhichisfollowedbyamotorresponse.Thevisuomotortaskisrepeatedmultipletimesforthesameexperimentwithdierentstimuli-responsecombinations.Thesedierencesaregroupedasdierentclassesofdataforclassication.Themainobjectiveistobeabletodetectthesedierencesoverthetimecourseofthetask,whichrequiresextensivecomputationaleorttoachieverobustresultsfromthemulti-dimensionalandhighlynonlinearneuraldata. Thevisualstimuliaredesignedtocreatelinesanddiamonds.Thegostimuliischosentobeeitherlinesordiamondsfromonesessiontoanother.Weareinterestedindetectingdierentcognitivestagesofthevisualdiscriminationtaskoverthetimeline.Wedistinguishdierentsetsoflabelsforeachcognitivestage.Threedierentstagesareanticipated:i)thedetectionofthevisualstimulus,ii)thecategoricaldiscriminationofthestimulus,andiii)themotorresponse.Therstandthethirdstagesarerelativelyeasytodetect,howeverthesecondstagehasnotbeendetectedinpreviousstudies( Ledbergetal. 2007 ).Thisstageinvolvesacomplexcognitiveprocesswhoseonsetandlengthvaryovertime. TheclassicationisperformedwiththepattervectorscollectedataspecictimeTfromeachtrial.Theclassicationaccuracyobtainedfromeachtimepointshowsthetimeintervalswhenthetwoobservedstatesofthemonkeybrainaredierent.However,therearetemporalvariationsineachtrialregardingthetimingoftheobservedstages. 86

PAGE 87

Thedataconsistsofaround4000trials.Becauseofthecomputationallimitationsoftheoptimizationsoftware(CPLEX10.1),theentiredatacouldnotbeprocessedsimultaneously.Thereforeweconsider200trialsatatimewithequalnumbersofpositiveandnegativerecordings.Nonlineariterativeeliminationmethodisappliedwithawindowof3recordingsfromeachtrialforeachtimepoint.Thiswindowcorrespondto15milliseconds.Therecordingswiththeminimumdistanceiseliminatedfromeachsetateachiteration.Thisisrepeateduntilthereisonlyonepatternvectorremainsfromeachtrial. Eachindependentbatchof200trialsresultedinaconsistentlyseparatedcumulativesetofselectedrecordings.TheclassicationaccuracyoftheselectedrecordingsfromeachtimewindowisevaluatedwiththestandardSVMclassierusing10-foldclassication.InFig. 3-7 (a),thecomparisonoftheclassicationaccuracyresultsfromiterativeeliminationandtheresultsfromthestandardSVMclassication.Theiterativeeliminationshowsaconsistentincreasearound10%.Thisincreasecanbeadjustedbythebaselineapproach.Inordertocreateabaseline,werandomlyassignclasslabelstopatternvectorsandapplytheiterativeeliminationmethods,sothatwecandetecttheincreaseintheaccuracyforrandomdataandsubtractitfromtheoriginalaccuracyresults.ThebaselineisalsogiveninFig. 3-7 (a).Thedierencebetweentheoriginalaccuracyresultsandthebaseline 87

PAGE 88

3-7 (b).Thepeakaround160millisecondsinthisgraphisveryclear.Thisresultmatchestheanticipatedintervalofthecategoricaldiscriminationstage.Thesecondpeakaround275millisecondsistoolateforthecategoricaldierentiation,howeverwouldprobablyberelatedtopostprocessingofthecategoricaldierence. (a)(b) Figure3-7. Comparativeclassicationaccuracyresults.(a):StandardSVM,baselineandafterapplyingselectiveSVM.(b):DierencebetweenthebaselineandselectiveSVMresults. InFig. 3-8 theresultsforthefeature(channel)selectionarepresented.WeusedanSVMbasedadaptivescalingmethodforfeatureselection.ThismethodndsthechannelsthatcontributetoSVMclassication.Whenadaptivescalingmethodisappliedoverthetimeline,itproducesnormalizedweightvectorsforeachtimepointthatcanbetransferedintoarasterplot. InFig. 3-8 (a)theresultsobtainedwithoutiterativeeliminationarepresented.Inthisplot,channelsaresignicantlyintermittentovertimeandtheoverallpictureisnotconclusive.TherasterplotinFig. 3-8 (b)showstheresultsobtainedbyiterativeelimination.Duetothesparsenessinuenceoftheadaptivescalingmethod,wecanclearlyseetheinuenceofthreemajorchannelsonthedata.Wefocusonthetimeintervalsaroundthepeaksobservedintheclassicationaccuracygraphs.Therstpeakcorrespondstoelectrode3,whichisaroundthesuperiortemporalgyrus.Physicaldamageintemporallobeisknowntoimpairvisualdiscrimination( HorelandMisantone 1976 ; MendolaandCorkin 1999 )andourresultsagreewiththeliterature.Thesecondpeak 88

PAGE 89

EacottandGaan 1991 ). (a)(b) Figure3-8. Rasterplotsfortheadaptivescalingfeatureselectionmethod(a):afterDTWapplied,(b):afterselectiveSVMapplied. 89

PAGE 90

Themotivationforthedevelopmentofselectiveclassicationmethodscomesfromtheclassicationofcognitivestatesinavisuomotorpatterndiscriminationtask.Duetothetemporalnoiseinthedata,theclassicationresultsobtainedarepoorwithstandardSVMmethods.Aslidingsmalltimewindowofrecordingsareconsideredassetsofpatternvectorsinselectiveclassication.Wellseparatedrecordingsareselectedbytheiterativeeliminationmethod.TheselectedrecordingsareevaluatedwithstandardSVMmethods,whichresultinasignicantincreaseintheclassicationaccuracyovertheentiretimelineofthetask.Theincreaseisadjustedbyabaselinemethodwhichisolatestheactualimprovementpeaks.Thesepeaksclearlymarkthecategoricaldiscriminationstageofthevisuomotortask,whichinvolvesacomplexcognitiveprocessthathasnotbeendetectedbypreviousstudies.Thisresultsuggestthattheproposedselectiveclassicationmethodsarecapableofprovidingpromisingsolutionsforotherclassicationproblemsinneuroscience. 90

PAGE 91

Inthischapter,weconsidertheclassicationproblemwithinthemultipleinstancelearning(MIL)context.Trainingdataiscomposedoflabeledbagsofinstances.Despitethelargenumberofmarginmaximizationbasedclassicationmethods,thereareonlyafewmethodsthatconsiderthemarginforMILproblemsintheliterature.WerstformulateacombinatorialmarginmaximizationproblemformultipleinstanceclassicationandprovethatitisNP-hard.Wepresentawaytoapplythekerneltrickinthisformulationforclassifyingnonlinearmultipleinstancedata.Wealsoproposeabranchandboundalgorithmandpresentcomputationalresultsonpubliclyavailablebenchmarkdatasets.Ourapproachoutperformsaleadingcommercialsolverintermsofthebestintegersolutionandoptimalitygapinthemajorityofimageannotationandmolecularactivitypredictiontestcases. Jainetal. 1994 ; Dietterichetal. 1997 ),harddrivefailureprediction( Murrayetal. 2005 ),textcategorization( Browetal. 2005 ), 91

PAGE 92

Carneiroetal. 2007 ; QiandHan 2007 ; ChenandWang 2004 ; Chuangetal. 2005 ). ThereisanarrayofmethodsproposedfortheMILproblem,mostofwhicharehybridsofotherwell-knownmethods.AcombinationoflazylearningandHausdordistanceisusedfortheMILproblemin( WangandZucker 2000 )withtwoextensionsofk-nearestneighbor(k-NN)algorithmandapplicationsonthedrugdiscoverybenchmarkdata.EM-DDtechnique,whichcombinesexpectationmaximization(EM)withthediversedensity(DD)algorithm,isproposedin( ZhangandGoldman 2001 ).EM-DDisrelativelyinsensitivetothenumberoffeaturesandscalesupwelltolargebagsizes.In( Doolyetal. 2002 ),extensionsofk-NN,citation-kNN,andDDalgorithmareproposedwithapplicationstobooleanandrealvalueddata. Marginmaximizationisthefundamentalconceptinsupportvectormachine(SVM)classiers,whichisshowntominimizetheboundonthegeneralizationerror( Vapnik 1998 ).AnincreasingnumberofmethodsthatinvolveSVMshavebeenproposedtosolveMILproblems.AgeneralizationofSVMforMILisintroducedin( Andrewsetal. 2003 ).ThismethodisbasedonaheuristicthatiterativelychangesthelabelsofinstancesinpositivebagsandusesstandardSVMformulation,untilalocaloptimalsolutionisfound.AnovelautomaticimageannotationsystemthatintegratesanMIL-basedSVMformulationtogetherwithaglobal-feature-basedSVMisproposedin( QiandHan 2007 ).Forregion-basedimagecategorization,acombinationofDDandSVMisusedin( ChenandWang 2004 ).Inthismethod,aDDfunctionisusedtocreateinstanceprototypesthatrepresenttheinstanceswhicharemorelikelytobelongtoabagwithaspeciclabel.InstanceprototypesareclassiedusingastandardSVMformulation.In( Chenetal. 2006 ),aninstancesimilaritymeasureisusedtomapbagstoafeaturespace.Thismethodliftstherequirementfortheexistenceofatleastonepositiveinstancetolabelapositivebagandusesa1-normSVMtoeliminateredundantandirrelevantfeatures.Aformulationwithlinearobjectiveandbilinearconstraintsisproposedtosolvemultiple 92

PAGE 93

MangasarianandWild 2008 ).Bilinearconstraintsarehandledbyanalternatingmethodthatusessuccessivefastlinearprogramsthatconvergetoalocalsolutioninafewiterations.Thelinearclassiersfoundbythismethodaresubstantiallysparse. Recently,afasttrainingalgorithm,MIL-boost,isproposedtodetectobjectsinimages( Violaetal. 2006 ).ThismethodcombinesacascadedetectormethodoptimizedforMILwithinaboostframework.ABayesianMILmethodisintroducedin( Raykaretal. 2008 ),whichautomaticallyidentiesrelevantfeaturesandusesinductivetransfertolearnmultipleclassiers.In( Fungetal. 2007 ),amethodthatusesaconvexhullrepresentationofmultipleinstancesisshowntoperformsignicantlyfasterandbetteronunbalanceddatawithfewpositivebagsandverylargenumberofnegativebags.TheconvexhullframeworkappliestomosthyperplanebasedMILmethods. ThischaptermainlyfocusesonthemaximalmarginclassiersforMIL.Ourgoalistondahyperplanethatmaximizesthemarginbetweenaselectionofinstancesfromeachpositivebagandalloftheinstancesfromnegativebags.TheformulationproposedfortheselectionofactualpositiveinstancesrendersthisproblemtobeNP-hard.Ageneralizationofthisformulationisproposedin( Serefetal. 2009 ),wheretheselectionconceptappliestobothpositiveandnegativeinstances.Thisselectivelearningmethodisusedtoclassifyneuraltime-seriesdata.Anothersimilarformulationisintroducedwithinanewsupervisedlearningproblemthatinvolvesaggregateoutputsfortraining( Musicantetal. 2007 ).Ourmaincontributioninthisstudyistointroducethemarginmaximizationformulationanditsdualformultipleinstanceclassication,discussthecomplexityoftheproblemandproposeabranchandboundalgorithmtosolvetheproblem. Theremainderofthischapterisorganizedasfollows:Section 4.2 presentsthemathematicalformulationwithsomeinsightsregardingthekerneltrickanddemonstratesNP-hardnessofmarginmaximizationformultipleinstancedata.Section 4.3 givestheimplementationdetailsofoursolutionapproachandSection 4.4 presentsthe 93

PAGE 94

4.5 ,weprovideconcludingremarksanddirectionsforfutureworkonthisclassofproblems. Basedonthisdenition,themaximummarginformulationcanbegeneralizedasthefollowingMixed0{1QuadraticProgrammingproblem. min;b;;1 2kk2+C s.t.h;xii+b1iM(1i)i2I+ Inthisformulation,I+=fi:i2Ij^yj=1g,I=fi:i2Ij^yj=1g,andJ+=fj:yj=1g.Notethat,Misasucientlylargenumberthatensuresthatthecorrespondingconstraintisactiveifandonlyifi=1.iisabinaryvariablethatis1ifi-thinstanceisoneoftheactualpositiveexamplesofitsbag. 94

PAGE 95

4{1 )as minPi2Iji1i2f0;1gmin;b;1 2kk2+C s.t.h;xii+b1iM(1i)i2I+ Inthisformulation,theouterminimizationsetsthebinaryvariables,andtheinnerminimizationsolvesregular2-normsoftmarginproblembasedonthesebinaryvalues.ThereforewecanwritetheLagrangianfunctionfortheinnerminimizationas 2kk2+C DierentiatingLwithrespecttotheprimalvariables,b,and,andusingstationarity,weobtain @=nXi=1yiixi=0; (4{4a) @b=nXi=1yii=0; (4{4b) @i=Cii=0: Wecansubstitutetheexpressionsin( 4{4 )backintheLagrangianfunctiontoobtainthedualformulation,whichwillgiveamaximizationprobleminsidetheminimization 95

PAGE 96

Mangasarian 1994 ).Instead,wesubstitutetheconditions( 4{4 )inside( 4{2 )directly: minPi2Iji1i2f0;1gmin;b1 2nXi=1nXj=1yiyjijhxi;xji+1 2CnXi=12is.t.nXj=1yjjhxj;xii+b1i=CM(1i)i2I+ (4{5c) Wenalizethediscussionbyapplyingthekerneltrickon( 4{5 )andtheresultingformulationis min;b;1 2nXi=1nXj=1yiyjijK(xi;xj)+1 2CnXi=12i s.t.nXj=1yjjK(xj;xi)+b1i=CM(1i)i2I+ (4{6d) 96

PAGE 97

4{6b )or( 4{6c ). Nextwepresentthecomplexityresultsonmarginmaximizationformultipleinstancedata. Serefetal. 2009 ).Selectivelearningisoriginallydevelopedtoecientlysolveatimeseriesalignmentprobleminneuraldata.However,theproblemdenitioninselectivelearningisslightlydierent;thepatternsarechosenfromeachpositiveandnegativesetinsuchawaythatthemarginbetweentheselectedpositiveandnegativepatternvectorsismaximized.Selectivelearning,whichisageneralizationofMIL1,isprovedtobeNP-hard( Serefetal. 2009 ).However,thisisnotenoughtoprovethecomplexityofMIL.Tothebestofourknowledge,thereisnoformalproofonthecomplexityofclassifyingmultipleinstancedataandthissectionintendstollthisgap. ItisclearthatforsucientlyhighpenaltyC,formulation( 4{1 )willprovideaseparatinghyperplanewherei=0;i=1;:::;n,ifdataislinearlyseparable.Therefore,thedecisionversionoftheoptimizationproblemin( 4{1 )isdenedasfollows: 97

PAGE 98

2kk2n? Proof. TheclassicalPARTITIONproblemisdescribedasfollows:GivenasetofpositiveintegersS=fs1;s2;:::;sng,doesthereexistasubsetS0Ssuchthat 2nXi=1si?(4{7) ThisproblemisknowntobeNP-complete( GareyandJohnson 1979 ).Next,weconsiderthefollowingvariantofthePARTITIONproblem. GivenasetofnpositiveintegersS=fs1;s2;:::;sng,doesthereexistavector2f1;+1gd,suchthat SupposewearegivenaninstanceofthePARTITIONproblem.Wewilladdndummyfeaturesandsetthedimensionofthespaced=2nandconstructaninstanceoftheMILDproblemasfollows: Leteibead-dimensionalvectorwhosecomponentsarezeroexceptcomponenti,whichisequalto1. (i) Addthepattern(s1;s2;;sn;1;0;;0)Twithpositivelabel. (ii) Addthepattern(s1;s2;;sn;1;0;;0)Twithnegativelabel. (iii) Addpatternsen+1;en+2;:::;e2nwithpositivelabels. (iv) Addpatternsen+1;en+2;:::;e2nwithnegativelabels. (v) Addnbagswithpositivelabelswherebagiconsistsofpatternseiandeifori=1;:::;n. Afterthisreduction,thecorrespondinginequalitiesin( 4{1 )become 98

PAGE 99

(4{9a)nXi=1sii+n+1b1 (4{9b)i+b1i=n+1;:::;2n Notethat,Cisasucientlylargenumberandahyperplanethathasthemaximuminterclassmarginwithi=0;i=1;:::;n,isdesired. Letusassertthatb=0andprovetheconstraintsin( 4{9 )ensureaYESanswerforMILDifandonlyifPARTITIONhasaYESanswer. Itisapparentfrom( 4{9c )and( 4{9d )thati=1;i=n+1;:::;2n,andfrom( 4{9e )thati2f1;+1g;i=1;:::n,sincethegoalistominimizekk2andsatisfy1 2kk2n.Usingthisfactwith( 4{9a 4{9b ),theanswerforMILDisYESifandonlyifPni=1sii=0(i.e.,PARTITIONhasaYESanswer). Next,weprovebycontradictionthatb=0inthemaximummarginsolution.Notethat,whenb=0,thesolutiondescribedaboveisfeasiblewithi2f1;1g;i=1;:::;n,andi=1;i=n+1;:::;2n,providedthatPARTITIONhasaYESanswer.Thisseparationgivesanobjectivefunctionofn.Assumethatthereisabettersolutionwithb=6=0.Then( 4{9c 4{9d )forcei1+jj;i=n+1;:::;2n,and( 4{9e )forcesjij1jj;i=1;:::;n.Evenif( 4{9a 4{9b )areignored,theobjectivefunctionvalueisatleastn+njj2whichisstrictlymorethann,thusaworsesolutionandacontradiction. Thepresentedreductionispolynomial.HenceMILDisNP-completeforbagsofsizeatleast2. 99

PAGE 100

4{1 ))isNP-hardforbagsofsizeatleast2. Proof. Theclassical3SATproblemisdescribedasfollows:GivenacollectionC=fc1;c2;:::;cmgofclausesonanitesetUofvariablessuchthatjcij=3for1im,isthereatruthassignmentforUthatsatisesalltheclausesinC? IfuisavariableinU,thenuanduareliteralsoverU.ThisproblemisknowntobestronglyNP-complete( GareyandJohnson 1979 ). Supposewearegivenaninstanceofthe3SATproblem.Wewillsetthedimensionofthespaced=2nandconstructaninstanceoftheMILDproblemasfollows: Notethat,eiisad-dimensionalvectorwhosecomponentsarezerosexceptforcomponenti,whichisequalto1. (i) Addmbagswithpositivelabelsforeachclausethatconsistsofvectorseiforliteralsuiandeiforliteralsuiinthecorrespondingclause. (ii) Addpatternsen+1;en+2;:::;e2nwithpositivelabels. (iii) Addpatternsen+1;en+2;:::;e2nwithnegativelabels. (iv) Addnbagswithpositivelabelswherebagiconsistsofpatternseiandeifori=1;:::;n. Afterthisreduction,thecorrespondinginequalitiesin( 4{1 )become (ili+b1)OR(jlj+b1)OR(klk+b1)l=1;:::;m (i+b1)OR(i+b1)i=1;:::;n 100

PAGE 101

Notethat,Cisasucientlylargenumberandahyperplanethathasthemaximuminterclassmarginwithi=0;i=1;:::;n,isdesired. Letusassertthatb=0andprovetheconstraintsin( 4{10 )ensureaYESanswerforMILDifandonlyif3SAThasaYESanswer. Itisobviousfrom( 4{10a )thatiareeithergreaterthan1orlessthan1andtheobjectiveofminimizingkk2ensuresiaresettoeither1or1,respectively.Itiseasytoseethattheanswerfor3SATisYESifandonlyif,i=1forvariablesthataresettoTRUEandi=1forthosethatareFALSE. Next,weprovebycontradictionthatb=0inthemaximummarginsolution.Assumethatthereisabettersolutionwithb=6=0.Then( 4{10b 4{10c )forcei1+jj;i=n+1;:::;2n,and( 4{10d )forcesjij1jj;i=1;:::;n.Theobjectivefunctionvalueisatleastn+njj2whichisstrictlymorethann,thusaworsesolutionandacontradiction. Thepresentedreductionispolynomial.HenceMILDisstronglyNP-completeforbagsofsizeatleast3. 4{1 ))isstronglyNP-hardforbagsofsizeatleast3. Wolsey 1998 ). 101

PAGE 102

Whentheupperboundisequaltothelowerbound,anodeisprunedbyoptimality,sincetheoptimalsolutionforthisdecompositionisknownandfurtherdecompositionisredundant.Anodecanalsobeprunedbybound,whichimpliesthatitdoesnotsuggestabettersolutionthancurrentbestsolution. Upperboundsareobtainedfromtheobjectivefunctionvalueoffeasiblesolutions.Ifafeasiblesolutionisbetterthantheincumbentsolution,incumbentissettothatsolution.Lowerboundsontheotherhand,arenotnecessarilyfeasiblebuttheygiveameasureofhowpromisingthedecompositionis.Tightboundsleadtomorepruningandfasterconvergence.Goodbranchingstrategiesarealsocrucialinasuccessfulbranchandboundalgorithm.Next,weexploreourboundingandbranchingschemesforMILproblem. 102

PAGE 103

2kk2+C s.t.h;xii+b1ii2I+^ci=1 (4{11b) (4{11c) 0i1i2Ij^j2J0^ci6=0 (4{11e) whereJ0isthesetofpositivebagswhoseactualpositiveinstancesarenotdiscovered,i.e.,J0=fj:yj=1^ck6=1;8k2Ijg.Itiseasytoseethatwhenconstraint( 4{1d )ischangedtoequality,theoptimalobjectivefunctionvaluewillnotchangefor( 4{1 ).Ontheotherhand,selectionofexactlyonedatainstanceperpositivebagwillsignicantlyreducethesizeofthefeasibleregion.Therefore,weusetheequalityconstraintforourlowerboundingformulation( 4{11 ).Whenaninstanceisselectedforadecomposition,constraint( 4{11d )willautomaticallyignoreremaininginstancesthatsharethesamebag,thusavoidredundantcomputationalwork. Iftheobtainedsolutionisintegerfeasible(i.e.,i2f0;1g;8i:i2Ij^yj=1)thenwecanprunethenodesinceupperandlowerboundsareequal(i.e.,theoptimalsolutionforthatdecompositionisknown).However,weobservethatwithoutacarefulselectionofparameterM,theaboveformulationignores( 4{11c )bysetting0
PAGE 104

4{12 )isnotsatised,thenbranchingisperformedonkwhere and Theproblemisdecomposedintotwosubproblemswithadditionalconstraintsk=1andk=0,respectively.TheaimhereistobranchonthecriticalbagIj0thatiscurrentlymisclassiedorclosesttobeingmisclassiedbasedon(;b).( 4{14 )selectsthecriticalbagwhereas( 4{13 )selectsthemostpromisinginstancefromthatbag. Figure4-1. Anexampleofcriticalbag. ConsidertheexampleinFig. 4-1 .Thealgorithmstartsbysolvingtherelaxationin( 4{11 ).Thereisone(circled)instanceinoneofthepositivebagswhichshouldbeselectedandthatsolutiondenesthelowerbound.Theseparatinghyperplanefortherelaxationisshownasadottedline.Thebagwhosebestinstanceisthemostmisclassiedisconsiderednext.Branchingisperformedonthemostpromisinginstanceinsquare.Fortherstdecompositionwheretheinstanceinsquareisselected,thecorrespondingnodecanbeprunedbyoptimalitysince( 4{12 )issatised.Whenotherinstancesinthisbagareconsideredasactualpositive,thelowerboundsarelarger,thustheoptimalsolutionis 104

PAGE 105

min;b;1 2kk2+C s.t.h;xii+b1ii2I+^ci=1 (4{15b) (4{15c) Foreachundecidedbag,weselecttheinstancethatisfurthestawayfromtheoptimalhyperplaneobtainedfrom( 4{15 ).SetSofselectedinstancesisdenedas where(;b;)denetheoptimalsolutionfor( 4{15 ). Thesecondphasecomputestheupperboundbysolvingthemarginmaximizationproblembasedonthistemporaryselection. 2kk2+C s.t.h;xii+b1ii2I+^ci=1 (4{17b) 105

PAGE 106

AsuncionandNewman 2007 )and( Andrewsetal. 2002 ).Twodatasetsfrom( AsuncionandNewman 2007 )representthemolecularactivitypredictiondatasets.Moleculesjudgedbyhumanexpertsarelabeledasmusksornon-musks.ThegoalforMIListodiscriminatethesetwocategoriesgiventheexactshapeandconformationofeachmolecule.Threedatasetsfrom( Andrewsetal. 2002 )correspondtoanimageannotationtaskwherethegoalistodeterminewhetherornotagivenanimalispresentinanimage.ColorimagesfromCoreldatasetaresegmentedwithBlobworldsystem.Setofsegmentsineachpicturearecharacterizedbycolor,shape,andtexturedescriptors.ThesizesofthesedatasetsarepresentedinTable 4-1 Features(Nonzero) +Bags +Instances -Bags -Instances 166 47 207 45 269 Musk2 166 39 1017 63 5581 Elephant 230(143) 100 762 100 629 Fox 230(143) 100 647 100 673 Tiger 230(143) 100 544 100 676 SizeinformationfortheMolecularActivityPredictionandtheImageAnnotationDataSets Allcomputationsareperformedona3.4GHzPentiumIVdesktopcomputerwith2.0GbRAM.ThealgorithmsareimplementedinC++andusedinconjunctionwithMATLAB7.3environmentinwhichthedataresides.Inouralgorithm,wesolvedtheconvexminimizationproblems(i.e.,formulations( 4{11 ),( 4{15 ),and( 4{17 ))usingCPLEX10.1( ILOG 2008 ).Forbenchmarkingpurposes,formulation( 4{1 )issolvedusingCPLEX10.1withdefaultsettings.Inallexperiments,trade-oparameterCbetweentrainingerrorandmarginissetto(Phx;xi=n)1,whichisscaledbasedontheinputvector. 106

PAGE 107

4{1 ),wereportthebestintegersolutionobtained(i.e.,UB),optimalitygap(i.e.,UB-LB)andsolutiontimesinsteadofthepredictionaccuracyresultsforgeneralization.Incaseswhereanalgorithmterminateswithoptimalityinthegiventimeframe,thelowerboundisequaltotheupperbound(i.e.,incumbentsolution),thuszerooptimalitygap. Inordertoshowthecomputationallimitationsofexactalgorithms,allinstancesareobtainedbyarandomfeatureandbagselection.Becausethenumberofinstancesisrestricted,thelastbagselectedmightnothaveallinstancesfromtheoriginaldataset.Theresultsshowthatwhenthenumberofinstancesincreases,ouralgorithmoutperformsCPLEXintermsofthebestobjectivefunctionvalue.However,whenthenumberoffeaturesincreases,thereisadditionalcomputationaltaskateachnodeofbranchandboundtreethatmightdeterioratetheperformanceofourimplementation.Nevertheless,featureselectioncanbeusedtoscaletheproblemwhereastheinstancesarecrucial. CPLEX10.1 ELEPHANT 2 10 0.04 0.01 20 2 5 0.01 0.01 40 3 10 0.14 0.03 40 3 5 0.20 0.03 80 6 10 259.29 1.95 80 6 5 91.56 3.00 CPLEX10.1 FOX 2 10 0.17 0.01 20 2 5 0.14 0.01 40 3 10 0.89 0.06 40 3 5 0.45 0.01 80 6 10 231.81 9.29 80 6 5 618.01 86.87 CPLEX10.1 TIGER 2 10 0.20 0.01 20 2 5 0.03 0.01 40 4 10 0.26 0.01 40 4 5 0.20 0.05 80 8 10 265.71 12.18 80 8 5 399.95 36.23 Time(inseconds)toachievetheoptimalsolutionforOurBranchandBoundSchemevs.CPLEXDefaultBranchandBoundAlgorithmfortheImageAnnotationData Table 4-2 showstheperformanceofexactalgorithmsforsmalltestinstances.Thecomputationtimestoachieveoptimalsolutionsarepresentedwithdierentdatasetsandimplementations.Asseenonthistable,CPLEXoutperformsourbranchandbound 107

PAGE 108

Next,weconsiderlargerproblemsets.Tables 4-3 and 4-4 presentbenchmarkresultsforourbranchandboundimplementationandCPLEXdefaultimplementationwithtimelimitsof3and30minutes,respectively.Inthesetests,allinstancesfromthemolecularactivitypredictiondatasetareusedandarandomfeatureselectionisperformed.Numberoffeaturesselectedisdenotedbyd. Tables 4-3 and 4-4 showthatouralgorithmachievesbettersolutionsthanCPLEXinalltests.However,thelowerboundsobtainedbyCPLEXaretighter.Musk2isnotusedinourcomputationalstudiesbecauseonlynonlinearclassiersareusedonthisdatasetintheliterature. CPLEX10.1 UB-LB Time UB UB-LB Time 5 180 11263.03 10 10801.06 12259.66 11082.57 180 ComputationalResultsforOurBranchandBoundSchemevs.CPLEXDefaultBranchandBoundAlgorithmfortheMolecularActivityPredictionData(Musk1)with3minutestimelimit. CPLEX10.1 UB-LB Time UB UB-LB Time 5 1800 13305.71 10 1800 11691.09 ComputationalResultsforOurBranchandBoundSchemevs.CPLEXDefaultBranchandBoundAlgorithmfortheMolecularActivityPredictionData(Musk1)with30minutestimelimit. Next,westudytheimageannotationdata.Inordertoobservehowthealgorithmsscaleup,instanceselectionisperformedaswellasfeatureselection.NumberofinstancesisdenotedbynandnumberofpositivebagsisdenotedbyjJ+j. Table 4-5 showsthatouralgorithmscalesupwellandobtainsgenerallybettersolutionsthanCPLEXforlargerproblemsin3minutes.TherearecaseswhereCPLEX 108

PAGE 109

CPLEX10.1 UB UB-LB Time UB UB-LB Time ELEPHANT 26 20 767.45 986.23 787.63 180 400 26 10 3064.59 3425.26 3230.09 180 400 26 5 180 3305.80 800 50 20 3792.22 4397.07 4295.40 180 800 50 10 6272.77 6757.60 6563.97 180 800 50 5 6557.27 7501.58 7308.48 180 1200 78 20 6585.39 9637.13 9637.13 180 1200 78 10 10062.24 11072.95 11072.95 180 1200 78 5 9952.44 11821.95 11631.28 180 FOX 33 20 180 3388.99 400 33 10 4751.69 4548.62 180 3999.80 400 33 5 180 4558.33 800 63 20 8792.20 8792.20 180 8429.70 800 63 10 10216.73 10050.18 180 9321.82 800 63 5 10045.32 9878.48 180 9485.00 1200 93 20 13034.06 15440.33 15417.24 180 1200 93 10 15395.31 15395.22 180 14309.68 1200 93 5 15547.77 15380.59 180 14456.20 TIGER 33 20 1699.07 1699.01 180 1484.61 400 33 10 180 3058.04 400 33 5 180 3422.92 800 71 20 4761.77 5472.10 5345.19 180 800 71 10 6946.20 7353.32 6953.16 180 800 71 5 180 8898.57 1144 100 20 7453.07 10433.51 10176.86 180 1144 100 10 10250.09 12190.93 11805.41 180 1144 100 5 11605.10 12774.59 11997.72 180 ComputationalResultsforOurBranchandBoundSchemevs.CPLEXDefaultBranchandBoundAlgorithmfortheImageAnnotationDatawith3minutestimelimit. performsbetterbutinthesecasesthedierencesaresubtle.Table 4-6 showsthatwhenthetimelimitisincreasedto30minutes,ouralgorithmstillachievesbettersolutionsinthemajorityoftests.Theremightbecaseswherethebestsolutionfoundbyanalgorithmisoptimalbutthereareactivenodesthathavelowerboundslessthantheincumbentsolution,thereforeoptimalityisnotguaranteed.Wedonotreportthenumberofremainingactivenodesexplicitly.However,itshouldbenotedthatCPLEXhassignicantlymorenumberofactivenodesthanouralgorithmontheaverage.Itshould 109

PAGE 110

CPLEX10.1 UB UB-LB Time UB UB-LB Time ELEPHANT 26 20 711.05 711.05 293.41 1800 400 26 10 2956.12 2954.31 1800 2482.72 400 26 5 3037.73 3037.57 1800 2442.02 800 50 20 3482.11 4379.25 4193.66 1800 800 50 10 6272.58 6594.63 6397.13 1800 800 50 5 6540.20 7092.43 6707.46 1800 1200 78 20 6585.39 7637.07 7470.57 1800 1200 78 10 10062.24 10564.25 10370.84 1800 1200 78 5 9874.41 11599.74 11402.11 1800 FOX 33 20 3130.62 2919.98 1800 2553.51 400 33 10 4115.66 3886.59 1800 3468.40 400 33 5 1800 4543.71 800 63 20 1800 8406.36 800 63 10 1800 9402.43 800 63 5 1800 9539.56 1200 93 20 13034.06 13588.41 13293.90 1800 1200 93 10 14532.07 14531.79 1800 14222.29 1200 93 5 14849.72 14650.02 1800 14246.02 TIGER 33 20 1429.96 1429.68 1800 1208.82 400 33 10 2785.38 2589.39 1800 2061.83 400 33 5 2971.33 3381.63 2973.43 1800 800 71 20 1800 4813.83 800 71 10 1800 7156.77 800 71 5 1800 8307.19 1144 100 20 1800 7973.10 1144 100 10 1800 11225.02 1144 100 5 10803.19 12202.86 11174.82 1800 ComputationalResultsforOurBranchandBoundSchemevs.CPLEXDefaultBranchandBoundAlgorithmfortheImageAnnotationDatawith30minutestimelimit. alsobenotedthat,lowerboundsobtainedbyCPLEXaregenerallybetterthanthatofourimplementation. Thebagsarehardertoseparatewhenthenumberoffeaturesdecreases.Therefore,theoptimalitygapwithlessnumberoffeaturesisusuallylarger.Tables 4-5 and 4-6 showthatouralgorithmusuallyndsbettersolutionsthanCPLEXdespitelargeroptimalitygap. Table 4-7 summarizestheresultsforcaseswhereanoptimalsolutionisnotachieved.#denotesthenumberoftestsanalgorithmoutperformstheother.Averageandlargest 110

PAGE 111

OptimalityGap OurB&B CPLEX10.1 OurB&B CPLEX10.1 # 25 3.19% 7.07% BEST 8.77% 25.59% Benchmarkresultsfortestswithtimelimits. improvementsachievedbyanalgorithmovertheotheraredenotedbyAVGandBEST,respectively.Asseenonthetable,ouralgorithmachievessignicantlybettersolutionsthanCPLEXingeneral.AlthoughoptimalitygapforCPLEXissmallerthanouralgorithmin33of58tests,theaverageimprovementisrelativelysmall.Ontheotherhand,whenouralgorithmhasasmalleroptimalitygap,theimprovementoverCPLEXismuchmoresignicant. Tosumup,whenthenumberofprobleminstancesissmallandnumberoffeaturesislarge,CPLEXdefaultimplementationcanbemoresuitablebecauseofitspreprocessingpower.Ouralgorithm,ontheotherhand,outperformsCPLEXforpracticalcases,wherenumberofinstancesislargeandfeatureselectionisapplied. AninterestingfuturestudymightbetheselectionofMinformulation( 4{1 )basedoninputdata.Thisnumbershouldsatisfytheselectioncriteria,butitshouldbesmallenoughtohavetightlowerboundswiththerelaxationsaswell.Alternatively,Mselection 111

PAGE 112

112

PAGE 113

Thischapterpresentsalinearregressionframeworkandasolutionapproachformultipleinstance(MI)data.Introducedinthecontextofdrugactivityprediction,MIlearningisageneralizationofsupervisedlearningmethods.Inthissetting,learningmethodsareperformedoverthebagsofpatternvectorsinsteadofindividualinstances.Thissettingisparticularlyusefulwhenthereisambiguityinthedatasetsuchasnoiseinclinicalmeasurementsoruncertaintyonthebindingconformationinadrug. Yoonetal. 2003 )).AdierentstudyapplieslocalregressiontoassessEsophagealPressureinGastroesophagealReuxDisease(GERD).Theresultsfrombothextensivesimulationsandrealdatademonstrateabilityoflocalregressiontocharacterizethepressure,whichisconsistentwiththeclinicalobservation(see( LiangandChen 2005 )).Inanotherbiomedicalstudy,regressionanalysisisusedtoevaluatesmokecarcinogendepositioninamulti-generationhumanreplica(see( Robinsonetal. 2006 )).Also,inastudyofFractionalBrownianMotion(FBM),regressionmethodsarecomparedforestimationaccuracyonsynthesizeddatasets(see( RussellandAkay 1996 )).Advancedtechniques,suchasmultipleregression,permituseofmorethanoneinputvariableandallowforthettingoffurthercomplexmodels(e.g.,quadraticequations). 113

PAGE 114

Vapnik 1995 )). SVRapproachisbasedonestimationofalinearfunctioninakernelinducedfeaturespace.Theobjectiveistooptimizeacertainboundarytotheoptimalregressionline,therefore,errorswithinacertaindistance(")ofpredictedvaluearedisregarded.Thelearningalgorithmminimizesaconvexfunctionalwithsparsesolutioncomparabletoclassicationtechnique.Forimprovedillustration,thiscanbeconsideredahyper-tube(insensitiveband)aboutalinearfunctioninthekernelinducednonlinearspace,suchthatpatternvectorsinthistubeareassumednottocontributeanyerror.Fig. 5-1 showstheinsensitivebandforaonedimensionallinearregressionproblem. Figure5-1. The"-insensitivebandforalinearregressionproblem. Thisformofregressioniscalled"-insensitivebecauseanypointinthe"oftheanticipatedregressionfunctiondoesnotcontributeanerror.Animportantmotivationforconsideringthe"-insensitivelossfunctionisthesparsenessofthedualvariablessimilartothecasewithSVMclassiers.Theideaofrepresentingthesolutionbymeansofasmallsubsetoftrainingpointshasenormouscomputationaladvantages.Furthermore,itensures 114

PAGE 115

CristianiniandShawe-Taylor 2000 )). SVRhasvariousapplicationsinnumeroustechnology(seee.g.,( SakhanenkoandLuger 2006 ),( Bergeronetal. 2005 )),analytical(seee.g.,( LauerandBloch 2008 ),( Hyunsooetal. 2005 )),andscienticelds(seee.g.,( Sunetal. 2004 ),( Yamamotoetal. 2006 )).( Wuetal. 2007 )performslocationestimationusingtheGlobalSystemforMobilecommunication(GSM)basedonanSVRapproachwhichdemonstratespromisingperformances,especiallyinterrainswithlocalvariationsinenvironmentalfactors.SVRmethodisalsousedinagriculturalschemesinordertoenhanceoutputproductionandreducelosses(seee.g.,( Xieetal. 2008 ),( Lietal. 2007 ),( PaiandHong 2007 ),( ChoyandChan 2003 )).Basedonstatisticallearningtheory,SVRhasbeenusedtodealwithforecastingproblems.Performingstructuralriskminimizationratherthanminimizingthetrainingerrors,SVRalgorithmshavebettergeneralizationabilitythantheconventionalarticialneuralnetworks(see( HongandPai 2007 )). Occasionallyallpointswithinadatasetcannotdeterminetheregressionfunctiondistinctively.Forexample,oneoftheseveralfeaturevectorencodingsmaybeknowntocontributeacertainoutcome,however,itmaynotbepossibletoidentifywhichone.Therefore,itisbenecialtodiscoveraregressionfunctionthatconsiderbagsofdatapoints. Themainapproachistoforecastvalueofadependentvariable,usingregressionfacts,meantfordatasetsinwhichmultipleinstancefeaturesareathand.Forinstance,inadrugthatisknowntobehelpfulforacertaindisease,itisdesiredtodiscriminatethemoleculesthatbindthetargetfromuselessones.Numerousmoleculecongurationsmaysharesimilarmoleculesinadynamicbalance.Experimentalactivitywillbeafunctionofoneormoreofthesecongurations;however,itisusuallynotviabletoestablishwhichone.Additionally,seldomistheconditionthatallcongurationscontributetotheexperimental 115

PAGE 116

Ray 2005 )). Multipleinstancelearning(MIL)problemsareintroducedby( Balasundarametal. 2005 )inthecontextofdrugactivityprediction.Theseproblemsareanalyzedandstudiedusingvariousproposedalgorithmsintheliterature.SupportVectorMachinesaremodiedtoexpressmultipleinstanceproblemsbyalteringthekernelsortheobjectivefunction(seee.g.,( Andrewsetal. 2003 ),( Gartneretal. 2002 )).GaussiannotionsarestudiedusingaDiverseDensityapproach(see( Maron 1998 )).FurtheralgorithmsintendedforextendedMILproblemsareintroducedwithashiftingtimewindowapproachforharddrivefailureprediction(referredtoas"RegularSupervisedLearningTechniques")(see( Murrayetal. 2005 )).( Serefetal. 2007 )employedasimilarshiftingtimewindowapproachandaselectivelearningtechniquetodetectcategoricaldiscriminationinavisuomotortaskperformedbyamacaquemonkey.ThisselectivelearningtechniqueisageneralizationofMILframeworkwherethenegativebagrepresentationsaredierentinthatatleastoneinstancefromeachnegativebagistruenegative(see( Serefetal. 2009 )). Multipleinstanceregressionproblemsoccurinanarrayofnewareas.Numerousfunctionsofmultipleinstancestudiespreferrealnumbersasforecastvalues.Toexemplify,indrugactivityprediction,drugdesignersdesireforecastedactivitystagesofthemoleculestobearticulatedasrealnumbervaluesratherthananticipatingactiveorinactivecategorizationofthesemolecules. Studiesarepreparedtounderstandcomputationalintricacyinnatetomultipleinstanceregressionproblems.Examplesofsuchstudiesincludeproteinfamilymodeling(see( Taoetal. 2004 )),stockprediction(see( Maron 1998 )),content-basedimageretrieval(see( MaronandRatan 1998 )),andtextclassication(see( Andrewsetal. 2003 )). Theremainderofthechapterisorganizedasfollows.Section 5.2 describestheformulationforthemultipleinstancesupportvectorregression(MI-SVR)problem.Section 5.3 presentstheexactsolutionapproachtondtheregressionfunctioninthissetting. 116

PAGE 117

5.4 demonstratescomputationalresultsforcomparisonpurposes.Section 5.5 revealstheconclusionandfutureresearchdirections. MI-SVRproblemreducestoselectingexactlyonepatternvectorfromeachbagsuchthatthesumofthe"-insensitiveerrorsbetweentheselectedpatternvectorsandtheregressionfunctionisminimized.Themultipleinstancesupportvectorlinearregressionproblemcanbeformulatedasaquadraticmixed0{1programmingproblemasfollows: min1 2kk2+C (5{1a)subjectto(h;xii+b)yj"+i+M(1i)8i:i2Ij Intheaboveformulation,quadratic"-insensitivelossisconsidered.Misasucientlylargenumber,suchthatforthosepointswithi=0,therelatedconstraintisalwayssatised,andthus,doesnothaveanyinuenceontheproblem.Thisisequivalenttoremovingthispatternvectorfromtheproblem.Constraints( 5{1b 5{1c )accountforthecaseifapatternvectorisbeloworabovetheregressionfunction.Finally,constraint( 5{1d )ensuresthatonlyoneofpatternvectorfromeachsetisselected. 117

PAGE 118

min1 2kk2+C (5{2a)subjectto(h;xii+b)yj"+i+M(1i)8i:i2Ij ThisproblemisknowntobestronglyNP-hardforbagsizesofatleast3(see( Ray 2005 )).Whileensuringtheconstraintsdropwhen=0,settingMassmallaspossibleiscrucialtoobtaingoodlowerbounds.However,givenasetofpatternvectorswithclasslabels,Mcannotevenbeconvenientlysettothemaximumdistancebetweentwopairsofpatternvectors.Considerthecasewhere"=0whichimpliesMmaxi:i2Ijjh;xii+byjj.AssumethatCislargeenoughthatthegoalistondaregressionfunction(ifpossible)withnoerror.Next,consideronedimensionaldatagivenasx1=0,x2=1,x3=2,x4=(>0)andassociatedlabelsy1=0,y2=2,y3=4,y4=4.Inotherwords,therearetwobagswithsingleinstances(labeled0and2)andonebagwithtwoinstances(labeled4).Clearly,0-insensitiveregressionwillselectinstance3astheprimaryinstancewith=2andb=0.ThissolutiondirectlyimpliesthatM>2whereasthelargestdistanceismax(2;).Inourcomputationalexperiments,weempiricallysetMsacricingthequalityofthelowerbound. InordertoapplythekerneltrickforMI-SVR,thedotproductsoftheinputpatternsareneeded.Werewriteformulation( 5{1 )asfollows: 118

PAGE 119

2kk2+C (5{3a) subjectto(h;xii+b)yj"+i+M(1i)8i:i2Ij Inthisformulation,theouterminimizationsetsthebinaryvariables,andtheinnerminimizationsolvesquadratic"-insensitivelossversionofSVRproblembasedonthesebinaryvalues.TheLagrangianfunctionfortheinnerminimizationis 2kk2+C DierentiatingLwithrespecttotheprimalvariables,b,,and^,andusingstationarityoftheinnerminimizationproblem,weobtain @=nXi=1(i+^i)xi=0;@L @b=nXi=1(i^i)=0;@L @i=Cii=0;@L @^i=C^i^i=0:(5{5) 119

PAGE 120

5{5 )aresubstitutedbackintheLagrangianfunction,amaximizationprobleminsidetheminimizationproblemisobtained.Instead,wesubstitutetheconditions( 5{5 )inside( 5{1 )directly. minPi2Iji1i2f0;1gmin;b1 2nXi=1nXj=1(i+^i)(j+^j)hxi;xji+1 2CnXi=1(2i+^2i) (5{6a) s.t.nXj=1(j+^j)hxj;xii+byj"+i=C+M(1i)8i:i2Ij (5{6d) Thekerneltrickisappliedbyreplacingthedotproductswithkernelfunctionsin( 5{6 ). min;b1 2nXi=1nXj=1(i+^i)(j+^j)K(xi;xj)+1 2CnXi=1(2i+^2i) (5{7a) subjecttonXj=1(j+^j)K(xi;xj)+byj"+i=C+M(1i)8i:i2Ij (5{7d) Nonlinearregressionfunctioncanbeobtainedformultipleinstancedatausing( 5{7 )forquadratic"-insensitiveloss.Thekerneltrickisappliedsimilarlyforlinear"-insensitive 120

PAGE 121

5.3.1LowerBoundingScheme 2kk2+C (5{8a) s.t.(h;xii+b)yi+i8i:i2Ij^ci=1 (5{8b) (5{8c) (h;xii+b)yi+i+M(1i)8i:i2Ij^0
PAGE 122

5{8 ). If( 5{9 )isnotsatised,thefeasiblespaceneedtobedecomposedfurther.Thefollowingisthebranchingschemeweemployed. Inourscheme,branchingisperformedonkwhere and Theproblemisdecomposedintotwosubproblemswithadditionalconstraintsk=1andk=0,respectively.Theaimhereistobranchonthecriticalbagthatiscurrentlyoutoftheinsensitiveband.( 5{11 )selectsthecriticalbagfromI0whereas( 5{10 )constructsI0,thesetofbagsoutoftheinsensitiveband. 122

PAGE 123

Serefetal. 2009 ).Inthesecondstep,were-optimizebasedonatemporaryselectionofactualpositiveinstanceswhichareclosesttothehyperplane.Formally,therstphasesolvesthefollowingproblem. min;b;1 2kk2+C (5{12a) subjectto(h;xii+b)yi+i8i:i2Ij^ci=1 (5{12b) (5{12c) (h;xii+b)yi+i+vi8i:i2Ij^0
PAGE 124

2kk2+C (5{14a) subjectto(h;xii+b)yi+i8i:i2Ij^ci=1 (5{14b) (5{14c) (h;xii+b)yi+i8i:i2S Next,wepresentcomputationalresultsonpubliclyavailablebreastcancerdatasets.Thealgorithmdescribedinthissectioniscomparedwithacommercialsolver. AsuncionandNewman 2007 ).Breastcancerprognosisisstudiedextensivelyin( Streetetal. 1995 )and( MangasarianandWild 2008 ). Eachrecordinthisdatasetrepresentsfollow-updataforonebreastcancercase.Theseareconsecutivepatientsseensince1984,andincludeonlythosecasesexhibitinginvasivebreastcancerandnoevidenceofdistantmetastasesatthetimeofdiagnosis.Thereare32featuresforeachrecord.Thesefeaturesarethesize(diameteroftheexcisedtumorincentimeters),lymphnodestatus(numberofpositiveaxillarylymphnodesobservedattimeofsurgery),and30featuresthatarecomputedfromadigitizedimageofaneneedleaspirate(FNA)ofabreastmass.These30featuresdescribecharacteristicsofthecellnucleipresentintheimageandincludethefollowinginformationforeachcellnucleus:radius,texture(standarddeviationofgray-scalevalues),perimeter,area,smoothness(localvariationinradiuslengths),compactness,concavity,numberofconcavepoints,symmetry,andfractaldimension. 124

PAGE 125

Allcomputationsareperformedona3.4GHzPentiumIVdesktopcomputerwith2.0GbRAMandtheWindowsXPoperatingsystem.ThealgorithmsareimplementedinC++andusedinconjunctionwithMATLAB7.3environmentinwhichthedataresides.Inouralgorithm,wesolvedtheconvexminimizationproblems(i.e.,formulations( 5{8 ),( 5{12 ),and( 5{14 ))withILOGCPLEX10.1.Forbenchmarkingpurposes,formulation( 5{1 )issolvedusingCPLEX10.1withdefaultsettings.Ifanalgorithmterminateswithoptimalityin180seconds,thelowerboundisequaltotheupperbound(i.e.,incumbentsolution).InallexperimentsparameterCissetto100. 1 21354271185440389755 2 20667271184005188991 5 24335329414005148418 20 244472744560545119150 Table5-1. Eectoffreeslackincreasefor100articialinstanceswithdierentdeviations. Table 5-1 showshowthechangeineectsthequalityofsolutions.TheheuristicalgorithmdescribedinSection 5.3 isusedwithdefaultbranchingandlowerboundingschemeofCPLEX.Whenthedeviationbetweentheinstancesofabagislarger(i.e.,issmaller),formulation( 5{12 )needsmoreslacktoignoretheconstraintsofnon-primaryinstances.However,sincethealgorithmusestheheuristicfornumerousdecompositions,thedierenceinthesolutionqualitymightbesubtlefordierentvalues.Inour 125

PAGE 126

#art. CPLEXB&BAlgorithm LBUBTime 1829901829904.91 1829901829901.45 505 123.3820954180 507.2222479180 10 66.79118540180 197.98145540180 20 855.4971615180 2697.4117970180 50 2050189191180 28550100030180 1005 4.7925080180 5.4632929180 10 0.1757269180 0.3490237180 20 9.8931400180 47.53104750180 50 7.3746707180 22.0380442180 1505 38.2718511180 28.7337345180 10 0.0016750180 0.0089518180 20 0.0039806180 0.0084676180 50 0.0060837180 0.0080025180 2005 0.0011096180 0.0025012180 10 0.0017407180 0.0034628180 20 0.0070435180 0.0061281180 50 0.0059730180 0.00135130180 Table5-2. ComputationalResultsforOurBranchandBoundSchemevs.CPLEXDefaultBranchandBoundAlgorithmfor32features InTable 5-2 ,rstcolumnshowsthenumberofarticialinstancesthatareaddedtotheoriginaldata.ThesecondcolumnisthevaluethatadjuststhedeviationoftheGaussiannoiseforthearticialinstances.WecompareourbranchandboundschemewithCPLEXsolverdefaultoptionsintermsofthelowerboundachievedbythetimeofterminationandthebestsolutionobtained.Thelastcolumnshowsthetimespentinsecondsforthealgorithmstoterminateeitherbyoptimalityorbytimelimit. Table 5-2 showsthatbothCPLEXandouralgorithmndtheoptimalsolutionin180secondsinsmalltestcasewherenoarticialdataisadded.Notethatinstancesthatsharethesamelabelareassumedtobeinthesamebag.Therefore,theoriginaldatasetisalsosolvedinamultipleinstanceframework.CPLEXperformsbetterintermsofsolutiontime 126

PAGE 127

ThetimespentineachnodeofthebranchandboundtreeissmallerforCPLEX,hencethelargenumberofexploredandactivenodes.ThisleadstoacasewhereCPLEXndsbettersolutionsthanouralgorithm(200articialinstanceswith=20).Fortherestofthedatasets,ouralgorithmoutperformsCPLEXintermsofthebestintegersolutionandtheoptimalitygap. Next,werandomlyselect10featuresandtesttwoalgorithmsonthisdataset.Theideahereistoseehowouralgorithmperformswhenrelativelyeasierdecompositionswithlessfeaturesaresolvedineachnodeofthetree. #art. CPLEXB&BAlgorithm LBUBTime 4132684132682.62 4132684132681.21 505 121090186360180 161810186370180 10 90080225600180 70418248130180 20 156340294470180 158830350020180 50 135410372270180 134490377770180 1005 19048124760180 20695235420180 10 27636223070180 33314236980180 20 14967214360180 24662266300180 50 88788371710180 95028383210180 1505 227174579180 1669152050180 10 29734192280180 31376228060180 20 1247279140180 1695397850180 50 2554350130180 4061351910180 2005 0.3396907180 1.30114800180 10 214.99104060180 943.67129350180 20 11.29217340180 0.82351560180 50 36.04315930180 44.58361940180 Table5-3. ComputationalResultsforOurBranchandBoundSchemevs.CPLEXDefaultBranchandBoundAlgorithmfor10features 127

PAGE 128

5-3 showsthatouralgorithmobtainedbettersolutionsthanCPLEXinalltestcases.Itcanbeobservedthatwhenthedeviationbetweenthearticialinstancesarelarger(i.e.,issmaller),ourintuitionofbranchingworksbetter.When=50,ontheotherhand,thedierencebetweenthesolutionsobtainedbyouralgorithmandCPLEXissubtle. Weobservethattheemployedheuristicgivestightupperbounds.ThelowerboundingschemeshouldbeimprovedthroughacarefulselectionofM.ThisnumbershouldsatisfytheselectioncriteriabutitshouldbesmallenoughtohavetightlowerboundswiththeLP-relaxationsaswell.Adierentlowerboundingapproachmightbeaninterestingfuturestudy.AsimilarframeworkcanalsobeappliedforthedualformulationstoobtainnonlinearMIregression. 128

PAGE 129

Thischapterconsistsoftwocomplexityresultsondierentpatternrecognitiontechniques.First,weconsiderthecomplexityoffeatureselectionforconsistentbiclusteringinSection 6.1 .Next,weprovethecomplexityresultonhyperplanesttingprobleminSection 6.2 Biclusteringisappliedbysimultaneousclassicationofthesamplesandfeatures(i.e.,columnsandrowsofmatrixA,respectively)intokclasses.LetS1;S2;:::;Skdenotetheclassesofthesamples(columns)andF1;F2;:::;Fkdenotetheclassesoffeatures(rows). 129

PAGE 130

Hartigan 1972 ),whichisknownasblockclustering.GivenabiclusteringB,thevariabilityofthedataintheblock(Sr;Fr)isusedtomeasurethequalityoftheclassication.Alowervariabilityintheresultingproblemispreferable.Thenumberofclassesshouldbexedinordertoavoidatrivial,zerovariabilitysolutioninwhicheachclassconsistsofonlyonesample.Amoresophisticatedapproachforbiclusteringwasintroducedin( ChengandChurch 2000 ),wheretheobjectiveistominimizethemeansquaredresidual.Inthissetting,theproblemisproventobeNP-hardandagreedyalgorithmisproposedtondanapproximatesolution.Asimulatedannealingtechniqueforthisproblemisdiscussedin( Bryan 2005 ). Anotherbiclusteringmethodisdiscussedin( Dhillon 2001 )fortextminingusingabipartitegraph.Inthegraph,thenodesrepresentfeaturesandsamples,andeachfeatureiisconnectedtoasamplejwithalink(i;j),whichhasaweightaij.Thetotalweightofalllinksconnectingfeaturesandsamplesfromdierentclassesisusedtomeasurethequalityofabiclustering.Alowervaluecorrespondstoabetterbiclustering.Asimilarmethodformicroarraydataissuggestedin( Klugeretal. 2003 ). In( Dhillonetal. 2003 ),theinputdataistreatedasajointprobabilitydistributionbetweentwodiscretesetsofrandomvariables.Thegoalofthemethodistonddisjointclassesforbothvariables.ABayesianbiclusteringtechniquebasedontheGibbssamplingcanbefoundin( Shengetal. 2003 ). Theconceptofconsistentbiclusteringisintroductedin( Busyginetal. 2005 ).Formally,abiclusteringBisconsistentifineachsample(feature)fromanysetSr(setFr),theaverageexpressionoffeatures(samples)thatbelongtothesameclassrisgreaterthantheaverageexpressionoffeatures(samples)fromotherclasses.Themodelforsupervisedbiclusteringinvolvessolutionofaspecialcaseoffractional0-1programmingproblemwhoseconsistencyisachievedbyfeatureselection.Computationalresultsonmicroarray 130

PAGE 131

Busyginetal. 2005 )fortheproofofTheorem 5 .Italsofollowsfromtheprovenconicseparabilitythatconvexhullsofclassesdonotintersect. Aproblemwithselectingthemostrepresentativefeaturesisthefollowing.Assumethatthereisaconsistentbiclusteringforagivendataset,andthereisafeature,i,suchthatthedierencebetweenthetwolargestvaluesofcSirisnegligible,i.e., min6=^rfcSi^rcSig; 1{21 )canbeviolatedbyaddingaslightlydierentsampletothedataset.Inotherwords,ifisarelativelysmallnumber,thenitisnotstatisticallyevidentthatai2F^r,andfeatureicannotbeusedtoclassifythesamples.Thesignicanceinchoosingthemostrepresentativefeaturesandsamplescomeswiththedicultyofproblemsthatrequirefeaturetestsandlargeamountsofsamplesthatareexpensiveandtimeconsuming.Somestrongeradditiveandmultiplicativeconsistentbiclusteringscanreplacetheweakerconsistentbiclustering.Additiveconsistentbiclusteringisintroducedin( Nahapetyanetal. 2008 )byrelaxing( 1{21 )and( 1{22 )as and 131

PAGE 132

Anotherrelaxationin( Nahapetyanetal. 2008 )ismultiplicativeconsistentbicluster-ingwhere( 1{21 )and( 1{22 )arereplacedwith and respectively,whereFj>1andSi>1. Supervisedbiclusteringusesaccuratedatasetsthatarecalledthetrainingsettoclassifyfeaturestoformulateconsistent,-consistentand-consistentbiclusteringproblems.Then,theinformationobtainedfromthesesolutionscanbeusedtoclassifyadditionalsamplesthatareknownasthetestset.Thisinformationisalsousefulforadjustingthevaluesofvectorsandtoproducemorecharacteristicfeaturesanddecreasethenumberofmisclassications. Givenasetoftrainingdata,constructmatrixSandcomputethevaluesofcSiusing( 1{19 ).Classifythefeaturesaccordingtothefollowingrule:featureibelongstoclass^r(i.e.,ai2F^r),ifcSi^r>cSi,86=^r.Finally,constructmatrixFusingtheobtainedclassication.Letxidenoteabinaryvariable,whichisoneiffeatureiisincludedinthecomputationsandzerootherwise.Consistent,-consistentand-consistentbiclusteringproblemsareformulatedasfollows. CB:

PAGE 133

In( 6{5 ),xi;i=1;:::marethedecisionvariables.xi=1ifi-thfeatureisselected,andxi=0otherwise.fik=1iffeatureibelongstoclassk,andfik=0otherwise.Theobjectiveistomaximizethenumberoffeaturesselectedand( 6{5b )ensuresthatthebiclusteringisconsistentwithrespecttotheselectedfeatures. 6{5 )isaspecictypeoffractional0-1programmingproblemwhichisdenedas 133

PAGE 134

ThisproblemisNP-hardsincelinear0-1programmingisaspecialclassofProblem( 6{8 )whensji=0andsj0=1forj=1;:::;ns,i=1;:::mands=1:::;S.Atypicalwaytosolveafractional0-1programmingproblemistoreformulateitasalinearmixed0-1programmingproblem,andsolvenewproblemusingstandardlinearprogrammingsolvers(see( T.-H.Wu 1997 ; Tawarmalanietal. 2002 )). In( Busyginetal. 2005 ),alinearizationtechniqueforageneralizedNP-hardformulation( 6{8 )isappliedtosolve( 6{5 ).In( Nahapetyanetal. 2008 )heuristicsareproposedfor( 6{5 )andgeneralizations.TheseattemptsareappropriateiftheproblemisNP-hard.However,whether( 6{5 )itselfisNP-hardornotwasanopenquestion.ThischapterintentstollthisgapbyprovingtheNP-hardnessof( 6{5 ). 6{5 ))isNP-hard. Proof. 6{5b )becomes 134

PAGE 135

GareyandJohnson 1979 )).Thedecisionversionoffeatureselectionforconsistentbiclusteringproblemis D-CB:Isthereasetoffeaturesthatensuresbiclusteringisconsistent,i.e.,satises( 6{9 )-( 6{10 )? Clearly,D-CBisinNPsincetheanswercanbecheckedinO(m)timeforagivensetoffeatures. Next,theKNAPSACKproblemwillbereducedtoD-CBinpolynomialtimetocompletetheproof. Inaknapsackinstance,anitesetU1,asizes(u)2Z+andavaluev(u)2Z+foreachu2U1,asizeconstraintB2Z+,andavaluegoalK2Z+aregiven.Thequestionis KNAPSACK:IsthereasubsetU0U1suchthatPu2U0s(u)BandPu2U0v(u)K. Wecanmodifytheknapsackproblemas :IsthereasubsetU0Usuchthat (6{11) (6{12) Obviously,remainsNP-complete,sinceKNAPSACKcanbereducedtoitsmodiedvariantifwedeneU=U1[t,s(t)=B,andv(t)=K. Denings0(u)=s(u)+,v0(u)=v(u)+foreachu2Uanditcaneasilybeseenthat 135

PAGE 136

Theinequalitysignsin( 6{13 )-( 6{14 )canbechangedtostronginequalityasfollows where0<1
PAGE 137

Classicalclusteringtechniquesintheliterature(e.g.,k-means)generateclustercentersaspointsthatminimizethesumofsquaresofdistancesofeachgiveninstancetoitsnearestclustercenter. BradleyandMangasarian ( 2000 )introducedthenotionofclustercenterhyperplane.Thejusticationforthisapproachisthatdatacanbegroupedaround 137

PAGE 138

Georgiev ( 2008 )laterextendedthisnotiontoclustercentersubspaceandnonlinearanalogsofthembyreproducingKernelHilbertSpaces. ConsidertheproblemoflinearrepresentationofadatasetXmN: Inthisdecomposition,theunknownmatricesA(dictionary)andS(sourcesignals)havecertainpropertiesunderdierentproblemsettings.Someofthemostwidelystudiedproblemsandtheircorrespondingpropertiesare: (i) IndependentComponentAnalysis(ICA):therowsofSareconsideredasdiscreterandomvariablesthatarestatisticallyindependentasmuchaspossible. (ii) SparseComponentAnalysis(SCA):Scontainsasmanyzerosaspossible. (iii) NonnegativeMatrixFactorization(NMF):theelementsofX;AandSarenonnegative. Theselinearrepresentationshaveseveralapplicationsincludingdecompositionofobjectsinto\natural"componentsandlearningtheelementsofeachobject(e.g.,fromasetoffaces,learningafaceconsistsofeyes,nose,mouth,etc.),redundancyanddimensionalityreduction,micro-arraydatamining,enhancementofimagesinnuclearmedicine(seee.g.,( LeeandSeung 1999 ),( Chenetal. 1998 )). TherearenumerousstudiesdevotedtoICAproblemsintheliteraturebutthesestudiesoftenconsiderthecompletecase(m=n)(seee.g.,( CichockiandAmari 2002 ),( Hyvarinenetal. 2001 )).Wereferto( BollandZibulevsky 2001 ),( Georgievetal. 2005 ),( Georgievetal. 2004 ),( ZibulevskyandPearlmutter 2001 )forSCAandovercompleteICA(m
PAGE 139

UnderthetermsparserepresentationofX2RmNweunderstandtherepresentationX=AS+E; 6.2.3 weshowthattheproblemisNP-hardandconcludethisChapter. RubinovandUdon 2003 ).LetXbeanitesetofpointsrepresentedbythecolumnsofX.Wecandescribethissetbyacollectionofhyperplanes. Thesolutionofthefollowingminimizationproblem minNXj=1min1ikjnTixjbij subjecttoknik=1i=1;:::;k 139

PAGE 140

Thesolutionofthefollowingminimizationproblem minNXj=1min1ikjnTixjbij2 subjecttoknik=1i=1;:::;k denesk(2)-skeletonofX(rstconsideredin( BradleyandMangasarian 2000 )). OurcrucialobservationisthattherepresentationX=AS Now,letXbeanarbitrarydatamatrix,andUbetheunionofthekhyperplanes,whichbesttthecolumnsofX.LetX1bethematrix,whichcolumnsaretheprojectionsofthecolumnsofXoverU(i.e.theclosestpointinUtothecolumnsofX).Then,obviously,theskeletonofthecolumnsofX1isexactlyU,sowehavetherepresentationX1=AS,forsomeA1andsparseS1(eachcolumnofS1containsatmostm1nonzeroelements),andwehavethefollowingsparserepresentationoftheoriginalX,as wherethematrixEhasaminimalnorm.Thisisexactlythesparserepresentationwhichwearelookingfor,usinghyperplanesttingalgorithms.Theuniquenessofsuch 140

PAGE 141

Georgievetal. 2005 ),( Georgievetal. 2007 ).Notethatsuchidentiabilityconditionsaremild,sotheyaresatisedalmostsurelyinpracticalsituations. Averysuitablealgorithmforclusteringdatanearanehyperplanes(i.e.,ndingk(2)skeletonofdatapoints)isthek-planeclusteringalgorithm( BradleyandMangasarian 2000 ).However,thisalgorithmhasaseriousdisadvantagethatitstopsinlocalminimum,evenkissmall.Wehaveperformedextensiveexperimentswiththisalgorithmandnotedthatifk7,thealgorithminalmostallrunsstopsinlocalminimum.So,aglobaloptimizationalgorithmisneeded. Next,weprove( 6{22 )and( 6{23 )areNP-hardandreformulatetheproblemasabilinearprogrammingproblem.Wealsoshowdirectionstoapplysomeglobaloptimizationtechniquestosolvetheproblem. minNXj=1min1ikj~nTi~xjjl subjecttok~nik=1;i=1;:::;k where~ni=nibiand~xj=xj1.Itssolution~ni;i=1;:::;kdeneshyperplaneskeletoninRm+1,consistingofaunionofkhyperplanes.Anehyperplanek(l)-skeletonofX2RmN,introducedforl=1in( RubinovandUdon 2003 )andforl=2in( BradleyandMangasarian 2000 ),isobtainedfrom( 6{25 )as(ni=(1jbij);bi=(1jbij))fori=1;:::;k. 141

PAGE 142

UsingareductionfromtheSETCOVERproblemtothisdecisionproblem,wenextshowthefollowingresult: Proof. TheclassicalSETCOVERproblemisdescribedasfollows:GivenacollectionC=fc1;c2;:::;cmgofsubsetsofanitesetS=fs1;s2;:::;sng,positiveintegerKjCj,doesCcontainacoverforSofsizeKorless?ThisproblemisknowntobeNP-complete( GareyandJohnson 1979 ). SupposethatwearegiveaninstanceofSETCOVERproblem.WewillconstructthedatamatrixXmnasfollowsandsetk=K: SelectionofacollectioniinSETCOVERimpliesahyperplaneeiasananswerofHCD.eiisthestandardbasiscolumnvectorwhoseelementsare0exceptithelementwhichis1.ThishyperplaneensuresthatPj:sj2cij~nTi~xjjl=0duetotheconstructiondescribedabove. WhenthereexistnormalhyperplanesthatsatisfyPNj=1min1ikj~nTi~xjjl=0butarenotstandardbasisvectors,wecanmakethefollowingtransformation: 142

PAGE 143

Consequently,HCDhasaYESanswerfordatamatrixXifandonlyiforiginalSETCOVERproblemhasaYESanswer.ThepresentedreductionispolynomialandHCDisNP-complete. 6{25 )isNP-hard. 3 isNP-hard. 7 infactimpliesthatitisunlikelythatthesolutiontothehyperplaneclusteringproblemcanbeapproximatedeciently.Analgorithmiscalleda(1+)-approximationalgorithmif,foranyminimizationprobleminstance,thealgorithmndsasolutionwithanobjectivefunctionvalueAthatsatisesA(1+);where0istheoptimalobjectivefunctionvalueand>0. Proof. 7 .Recallthatanysolutionforthehyperplaneclusteringprobleminstancewithobjectivefunctionvalue=0correspondstoafeasiblecoveringwithYESanswerfortheSETCOVERinstanceandthatasolutionwith>0correspondstoaNOanswer.Assumethereexistsapolynomial-time(1+)-approximationalgorithmAforsome>0.If>0,thenthe 143

PAGE 144

Inthisstudy,weexplorehyperplanesttingproblemandprovethatthisproblemisNP-hard.Asaconsequence,acorrespondingsparserepresentationproblemisNP-hardtoo.SuchsparserepresentationproblemcanbeconsideredasageneralizationoftheBlindSignalSeparationproblembasedonsparsityassumptionsofthesourcematrix.Wealsoproposedanewglobaloptimizationalgorithmforndingthebesthyperplaneskeleton,basedonabilinearreformulationandcuttingplanemethod.ItisabaseforanewalgorithmforsparserepresentationandBlindSignalSeparationproblemsfordemixingunknownmixtureofsourcesignalsundermildsparsityassumptions. 144

PAGE 145

Ourdiscussiononmathematicalprogrammingproblemsinpatternrecognitionstartswithanintroductorysurveyongeneraloptimizationbasedmachinelearningtechniqueswithapplicationsinhealthcare.Thischapterisbasedon( KundakciogluandPardalos 2009b )and( Serefetal. 2008a ).Next,weconsiderlinearclassicationproblemsindeathcelldiscrimination.Thisaimofthisstudyistodevelopdiagnostictoolsforcancerandquantifythecellularresponsetochemotherapyandthetoxicityassessmentofdierentdrugs.Thisstudyisbasedon( Pyrgiotakisetal. 2009 ). Basedon( Serefetal. 2009 ),weintroduceanovelselectiveclassicationmethodwhichisageneralizationofthestandardSVMclassiers.Setsofpatternvectorssharingthesamelabelaregivenasinput.Onepatternvectorisselectedfromeachsetinordertomaximizetheclassicationmarginwithrespecttotheselectedpositiveandnegativepatternvectors.Theproblemofselectingthebestpatternvectorsisreferredtoasthehardselectionproblem.ThehardselectionproblemisshowntobeNP-hard.Weproposealternativelinearandnonlinearapproacheswithtractableformulations,whichwecallsoftselectionproblems.Theselectivenatureofthetheseformulationsismaintainedbytherestrictedfreeslackconcept.Theintuitionbehindthisconceptistoreversethecombinatorialselectionproblembydetectinginuentialpatternvectorswhichrequirefreeslacktodecreasetheireectontheclassicationfunctions.Iterativelyremovingsuchpatternvectors,wecanndthosepatternvectorswithalargermargin.Aniterativeeliminationmethodisproposedforthispurpose.Anotheralternativeapproachistoprovideenoughfreeslacktoidentifyallt1outoftpatternvectorstoberemovedatonce,whichleadstothedirectselectionmethod.Theiterativeeliminationandthedirectselectionmethodsarefoundtoproducesimilarresults.IterativeeliminationmethodisalsocomparedwithanaveeliminationmethodwhichusesstandardSVMtoeliminate 145

PAGE 146

Chapter 4 presentsthemathematicalformulation,kerneltrickapplication,complexityresults,andanexactalgorithmforlinearmultipleinstanceclassicationthroughmarginmaximization.Experimentalresultsshowadditionalbenetsofintelligentboundingandbranchingschemes.Weobservethattheemployedheuristicgivestightupperboundsbutthelowerboundingschemeneedstobeimproved.Thelowerboundingtechniqueweproposehelpsmostlywithpruningbyoptimalitybutrarelywithpruningbybound.Thischapterisbasedon( Kundakciogluetal. 2009b ).Chapter 5 extendstheexactalgorithmforregressionandisbasedon( Kundakciogluetal. 2009a ). Nextisabriefcomplexityresultonfeatureselectionforconsistentbiclustering.Theaiminthissettingistoselectasubsetoffeaturesintheoriginaldatasetsuchthattheobtainedsubsetofdatabecomesconditionallybiclustering-admittingwithrespecttothegivenclassicationoftrainingsamples.Theadditiveandmultiplicativevariationsoftheproblemareconsideredtoextendthepossibilitiesofchoosingthemostrepresentativesetoffeatures.ItisshownthatthefeatureselectionforconsistentbiclusteringisNP-hard.Thisstudyispublishedin( KundakciogluandPardalos 2009a ).Inthesamechapter,weconsiderthehyperplanesttingproblem,wherethegoalistondhyperplanesthatminimizethesumofsquaresofthedistancesbetweeneachdatapointandthenearesthyperplane.WeprovethatthisproblemisNP-hard. Here,wepresentalternativeformulationsforsupportvectorclassierswithmultipleinstancedata.Thesenonconvexformulationsdonotutilizeintegervariables.Acomparisonofdierentformulationswithdierentcommercialsolverswouldalsobeaninterestingfuturestudy. min;b;;1 2kk2+C 146

PAGE 147

min;b;;1 2kk2+C subjecttoXi2Ij(ih;xii)+b1^jj2J+ Formulation( 7{1 )and( 7{2 )considertheconvexcombinationofallpointsinabagandtrytopenalizethemisclassicationforthispoint.Theobjectiveofminimizingtotalmisclassicationensuresthatthisconvexcombinationistheactualpositiveforthebag. Ourfutureworkisonnewexactmethodsforcombinatorialclassicationandregressionproblems.NonlinearextensionsareonlyconsideredforSelectiveSVMsbutexactmethodscanalsobeexploredfornonlinearclassicationwithmultipleinstancedata.AsfarastheproblemsinChapter 6 areconcerned,exactandheuristicmethodsforhyperplanesttingproblemwouldbeinterestingfuturestudies. 147

PAGE 148

Acir,N.,C.Guzelis.2005.AutomaticrecognitionofsleepspindlesinEEGviaradialbasissupportvectormachinebasedonamodiedfeatureselectionalgorithm.NeuralComputingandApplications14(1)56{65. Acosta,R.,M.Ehrgott,A.Holder,D.Nevin,J.Reese,B.Salter.2008.OptimizationinMedicine,chap.Theinuenceofdosegridresolutiononbeamselectionstrategiesinradiotherapytreatmentdesign.Springer,1{23. Alexe,S.,E.Blackstone,P.Hammer,H.Ishwaran,M.Lauer,C.Snader.2003.Coronaryriskpredictionbylogicalanalysisofdata.AnnalsofOperationsResearch11915{42. Andrews,S.,T.Hofmann,I.Tsochantaridis.2002.Multipleinstancelearningwithgeneralizedsupportvectormachines.EighteenthNationalConferenceonArticialIntelligence.AmericanAssociationforArticialIntelligence,MenloPark,CA,USA,943{944. Andrews,S.,I.Tsochantaridis,T.Hofmann.2003.AdvancesinNeuralInformationProcessingSystems,vol.15,chap.Supportvectormachinesformultiple-instancelearning.MITPress,Vancouver,BritishColumbia,Canada,561{568. Armour,E.P.,D.McEachern,Z.Wang,P.M.Corry,A.Martinez.1993.Sensitivityofhumancellstomildhyperthermia.CancerResearch53(12)2740{2744. Asuncion,A.,D.J.Newman.2007.UCImachinelearningrepository.URL Auer,P.1997.Onlearningfrommulti-instanceexamples:Empiricalevaluationofatheoreticalapproach.Proceedings14thInternationalConferenceonMachineLearning.21{29. Balasundaram,B.,S.Butenko,S.Trukhanov.2005.Novelapproachesforanalyzingbiologicalnetworks.JournalofCombinatorialOptimization10(1)23{39. Ben-Dor,A.,L.Bruhn,N.Friedman,I.Nachman,M.Schummer,Z.Yakhini.2000.Tissueclassicationwithgeneexpressionproles.JournalofComputationalBiology:AJournalofComputationalMolecularCellBiology7559{583. Ben-Dor,A.,B.Chor,R.Karp,Z.Yakhini.2002.Discoveringlocalstructureingeneexpressiondata:Theorder-preservingsubmatrixproblem.RECOMB'02:ProceedingsoftheSixthAnnualInternationalConferenceonComputationalBiology.49{57. Ben-Dor,A.,N.Friedman,Z.Yakhini.2001.Classdiscoveryingeneexpressiondata.RE-COMB'01:ProceedingsoftheFifthAnnualInternationalConferenceonComputationalBiology.ACMPress,NewYork,NY,USA,31{38. Bennet,K.,C.Campbell.2000.Supportvectormachines:Hypeorhallelujah?SIGKDDExplorations2(2)1{13. 148

PAGE 149

Bertsimas,D.,R.Shioda.2007.Classicationandregressionviaintegeroptimization.OperationsResearch55(2)252{271. Bevilacqua,V.,G.Mastronardi,G.Piscopo.2007.Evolutionaryapproachtoinverseplanningincoplanarradiotherapy.ImageandVisionComputing25(2)196{203. Bhowmick,T.K.,G.Pyrgiotakis,K.Finton,A.K.Suresh,S.G.Kane,J.R.Bellare,B.M.Moudgil.2008.RamanspectroscopystudyoftheeectofJBparticlesonsaccharomycescerevisiae(yeast)cellsbyRamanspectroscopy.JournalofRamanSpectroscopy391859{1868. Billups,S.,J.Kennedy.2001.Minimum-supportsolutionsforradiotherapyplanning.AnnalsofOperationsResearch119229{245. Bishop,C.M.2006.PatternRecognitionandMachineLearning.Springer. Blum,A.,A.Kalai.1998.Anoteonlearningfrommultiple-instanceexamples.MachineLearning30(1)23{29. Boesewetter,D.,J.Collier,A.Kim,M.Riley.2006.AlterationsofA549lungcellgeneexpressioninresponsetobiochemicaltoxins.CellBiologyandToxicology22(2)101{108. Boll,P.,M.Zibulevsky.2001.Underdeterminedblindsourceseparationusingsparserepresentation.SignalProcessing81(11)2353{2362. Bradley,P.S.,O.L.Mangasarian.2000.k-planeclustering.JournalofGlobalOptimiza-tion16(1)23{32. Brandeau,M.L.,F.Sainfort,W.P.Pierskalla,eds.2004.HandbookofOperationsResearchandHealthCare:MethodsandApplications.KluwerAcademicPublishers. Brouwer,G.J.,R.vanEe.2007.Visualcortexallowspredictionofperceptualstatesduringambiguousstructure-from-motion.TheJournalofNeuroscience27(5)1015{1023. Brow,T.,B.Settles,M.Craven.2005.Classifyingbiomedicalarticlesbymakinglocalizeddecisions.ProceedingsoftheFourteenthTextRetrievalConference(TREC). Brown,M.,W.Grundy,D.Lin,N.Cristianini,C.Sugne,T.Furey,M.Ares,D.Haussler.2000.Knowledge-baseanalysisofmicroarraygeneexpressiondatabyusingsupportvectormachines.ProceedingsoftheNationalAcademyofSciences97(1)262{267. 149

PAGE 150

Busygin,S.,N.Boyko,P.M.Pardalos,M.Bewernitz,G.Ghacibeh.2007a.BiclusteringEEGdatafromepilepticpatientstreatedwithvagusnervestimulation.O.Seref,O.E.Kundakcioglu,P.M.Pardalos,eds.,DataMining,SystemsAnalysis,andOptimizationinBiomedicine.AmericanInstituteofPhysics,220{231. Busygin,S.,G.Jacobsen,E.Kramer.2002.Doubleconjugatedclusteringappliedtoleukemiamicroarraydata.ProceedingsoftheSecondSIAMInternationalConferenceonDataMining,WorkshoponClusteringHighDimensionalData. Busygin,S.,O.A.Prokopyev,P.M.Pardalos.2005.Featureselectionforconsistentbiclusteringviafractional0{1programming.JournalofCombinatorialOptimization10(1)7{21. Busygin,S.,O.A.Prokopyev,P.M.Pardalos.2007b.Anoptimization-basedapproachfordataclassication.OptimizationMethods&Software22(1)3{9. Busygin,S.,O.A.Prokopyev,P.M.Pardalos.2008.Biclusteringindatamining.Comput-ers&OperationsResearch35(8)2964{2987. Carew,J.D.,M.Yuan.2007.Nonparametricsmoothinganditsapplicationsinbiomedicalimaging.O.Seref,O.E.Kundakcioglu,P.M.Pardalos,eds.,DataMining,SystemsAnalysis,andOptimizationinBiomedicine.AmericanInstituteofPhysics,85{105. Carneiro,G.,A.B.Chan,P.J.Moreno,N.Vasconcelos.2007.Supervisedlearningofsemanticclassesforimageannotationandretrieval.IEEETransactionsonPatternAnalysisandMachineIntelligence29(3)394{410. Censor,Y.,T.Bortfeld,B.Martin,A.Tromov.2006.Auniedapproachforinversionproblemsinintensity-modulatedradiationtherapy.PhysicsinMedicineandBiology51(10)2353{2365. Chaovalitwongse,W.,P.M.Pardalos,L.D.Iasemidis,W.Suharitdamrong,D.-S.Shiau,L.K.Dance,O.A.Prokopyev,V.L.Boginski,P.R.Carney,J.C.Sackellares.2007.DataMininginBiomedicine,OptimizationandItsApplications,vol.7,chap.DatamininginEEG:Applicationtoepilepticbraindisorders.Springer,459{481. Chaovalitwongse,W.,O.Prokopyev,P.M.Pardalos.2006.Electroencephalogram(EEG)timeseriesclassication:Applicationsinepilepsy.AnnalsofOperationsResearch148(1)227{250. Chen,S.,D.Donoho,M.Saunders.1998.Atomicdecompositionbybasispursuit.SIAMJournalonScienticComputing20(1)33{61. 150

PAGE 151

Chen,Y.,J.Z.Wang.2004.Imagecategorizationbylearningandreasoningwithregions.JournalofMachineLearningResearch5913{939. Cheng,Y.,G.M.Church.2000.Biclusteringofexpressiondata.ProceedingsoftheEighthInternationalConferenceonIntelligentSystemsforMolecularBiology.AAAIPress,93{103. Choy,K.Y.,C.W.Chan.2003.Modellingofriverdischargesandrainfallusingradialbasisfunctionnetworksbasedonsupportvectorregression.InternationalJournalofSystemsScience34(14{15)763{773. Chuang,S.C.,Y.Y.Xu,H.-C.Fu.2005.Neuralnetworkbasedimageretrievalwithmultipleinstanceleaningtechniques.LectureNotesinComputerScience36821210{1216. Cichocki,A.,S.Amari.2002.AdaptiveBlindSignalandImageProcessing.JohnWiley. Cifarelli,C.,G.Patrizi.2007.Solvinglargeproteinfoldingproblembyalinearcomplementarityalgorithmwith0{1variables.OptimizationMethodsandSoftware22(1)25{49. Cox,D.D.,R.L.Savoy.2003.Functionalmagneticresonanceimaging(fMRI)\brainreading":DetectingandclassifyingdistributedpatternsoffMRIactivityinhumanvisualcortex.NeuroImage19261{270. Craft,D.L.,T.F.Halabi,H.A.Shih,T.R.Bortfeld.2006.Approximatingconvexparetosurfacesinmultiobjectiveradiotherapyplanning.MedicalPhysics33(9)3399{3407. Cristianini,N.,J.Shawe-Taylor.2000.AnIntroductiontoSupportVectorMachines.CambridgeUniversityPress,Cambridge,UK. Darbellay,G.A.,R.Du,J.M.Vesin,P.A.Despland,D.W.Droste,C.Molina,J.Serena,R.Sztajzel,P.Ruchat,T.Karapanayiotides,A.Kalangos,J.Bogousslavsky,E.B.Ringelstein,G.Devuyst.2004.Solidorgaseouscirculatingbrainemboli:Aretheyseparablebytranscranialultrasound?JournalofCerebralBloodFlow&Metabolism24860{868. Dariush,B.2003.Humanmotionanalysisforbiomechanicsandbiomedicine.MachineVisionandApplications14(4)202{205. Devos,A.,L.Lukas,J.A.K.Suykens,L.Vanhamme,A.R.Tate,F.A.Howe,C.Majos,A.Moreno-Torres,M.vanderGraaf,C.Arus,S.VanHuel.2004.Classicationofbraintumoursusingshortechotime1HMRspectra.JournalofMagneticResonance170164{175. 151

PAGE 152

Dewhirst,M.W.,D.A.Sim,S.Sapareto,W.G.Connor.1984.Importanceofminimumtumortemperatureindeterminingearlyandlong-termresponsesofspontaneouscanineandfelinetumorstoheatandradiation.CancerResearch44(1)43{50. Dhillon,I.S.2001.Co-clusteringdocumentsandwordsusingbipartitespectralgraphpartitioning.KDD'01:ProceedingsoftheseventhACMSIGKDDinternationalconferenceonKnowledgediscoveryanddatamining.ACMPress,NewYork,NY,USA,269{274. Dhillon,I.S.,S.Mallela,D.S.Modha.2003.Information-theoreticco-clustering.KDD'03:ProceedingsoftheninthACMSIGKDDinternationalconferenceonKnowledgediscoveryanddatamining.ACMPress,NewYork,NY,USA,89{98. Dietterich,T.G.,R.H.Lathrop,T.Lozano-Perez.1997.Solvingthemultipleinstanceproblemwithaxis-parallelrectangles.ArticialIntelligence8931{71. Divina,F.,J.S.Aguilar-Ruiz.2006.Biclusteringofexpressiondatawithevolutionarycomputation.IEEETransactionsonKnowledgeandDataEngineering18(5)590{602. Donahue,M.M.,W.Zhang,M.L.Harrison,J.Hu,A.E.Rundell.2007.EmployingoptimizationandsensitivityanalysestoolstogenerateandanalyzemathematicalmodelsofT-cellsignalingevents.O.Seref,O.E.Kundakcioglu,P.M.Pardalos,eds.,DataMining,SystemsAnalysis,andOptimizationinBiomedicine.AmericanInstituteofPhysics,43{63. Dooly,D.R.,Q.Zhang,S.A.Goldman,R.A.Amar.2002.Multiple-instancelearningofreal-valueddata.JournalofMachineLearningResearch3651{678. Dube,S.,J.J.Corso,T.F.Cloughesy,S.El-Saden,A.L.Yuille,U.Sinha.2007.AutomatedMRimageprocessingandanalysisofmalignantbraintumors:Enablingtechnologyfordatamining.O.Seref,O.E.Kundakcioglu,P.M.Pardalos,eds.,DataMining,SystemsAnalysis,andOptimizationinBiomedicine.AmericanInstituteofPhysics,64{84. Eacott,M.J.,D.Gaan.1991.Theroleofmonkeyinferiorparietalcortexinvisualdiscriminationofidentityandorientationofshapes.BehaviouralBrainResearch46(1)95{98. Ehrgott,M.,H.W.Hamacher,M.Nubaum.2008.OptimizationinMedicine,chap.Decompositionofmatricesandstaticmultileafcollimators:Asurvey.Springer,25{46. 152

PAGE 153

Faugeras,O.,G.Adde,G.Charpiat,C.Chefd'Hotel,M.Clerc,T.Deneux,R.Deriche,G.Hermosillo,R.Keriven,P.Kornprobst,J.Kybic,C.Lenglet,L.Lopez-Perez,T.Papadopoulo,J.-P.Pons,F.Segonne,B.Thirion,D.Tschumperle,T.Vieville,N.Wotawa.2004.Variational,geometric,andstatisticalmethodsformodelingbrainanatomyandfunction.NeuroImage2346{55. Ferris,M.,J.Lim,D.Shepard.2001.Radiosurgerytreatmentplanningvianonlinearprogramming.AnnalsofOperationsResearch119247{260. Fung,G.,M.Dundar,B.Krishnapuram,R.B.Rao.2007.Multipleinstancelearningforcomputeraideddiagnosis.B.Scholkopf,J.Platt,T.Homan,eds.,AdvancesinNeuralInformationProcessingSystems,vol.19.MITPress,Vancouver,BritishColumbia,Canada,425{432. Fung,G.,J.Stoeckel.2007.SVMfeatureselectionforclassicationofSPECTimagesofAlzheimer'sdiseaseusingspatialinformation.KnowledgeandInformationSystems11(2)243{258. Fung,H.K.,S.Rao,C.A.Floudas,O.Prokopyev,P.M.Pardalos,F.Rendl.2005.Computationalcomparisonstudiesofquadraticassignmentlikeformulationsfortheinsilicosequenceselectionproblemindenovoproteindesign.JournalofCombinatorialOptimization10(1)41{60. Garcia,G.N.,T.Ebrahimi,J.M.Vesin.2003.Jointtime-frequency-spaceclassicationofEEGinabrain-computerinterfaceapplication.JournalonAppliedSignalProcessing713{729. Garey,M.R.,D.S.Johnson.1979.ComputersandIntractability:AGuidetotheTheoryofNP-Completeness.W.H.Freeman&Co. Garrett,D.,D.A.Peterson,C.W.Anderson,M.H.Thaut.2003.Comparisonoflinear,nonlinear,andfeatureselectionmethodsforEEGsignalclassication.IEEETransactionsonNeuralSystemsandRehabilitationEngineering11(2)141{144. Gartner,T.,P.A.Flach,A.Kowalczyk,A.J.Smola.2002.Multiinstancekernels.Proceedingsofthe19thInternationalConferenceonMachineLearning.179{186. Genkin,A.,C.A.Kulikowski,I.Muchnik.2002.Setcoveringsubmodularmaximization:Anoptimalalgorithmfordatamininginbioinformaticsandmedicalinformatics.JournalofIntelligent&FuzzySystems125{17. Georgiev,P.,F.Theis,A.Cichocki.2004.Blindsourceseparationandsparsecomponentanalysisofovercompletemixtures.ProceedingsofICASSP2004.Montreal,Canada. 153

PAGE 154

Georgiev,P.,F.Theis,A.Ralescu.2007.IdentiabilityconditionsandsubspaceclusteringinsparseBSS.LectureNotesComputerScience4666357{364. Georgiev,P.G.2008.Nonlinearskeletonsofdatasetsandapplications{methodsbasedonsubspaceclustering.P.M.Pardalos,P.Hansen,eds.,DataMiningandMathematicalProgramming,CRMProceedingsandLectureNotes,vol.45.AmericanMathematicalSociety,95{108. Gerner,E.W.,W.G.Connor,M.L.Boone,J.D.Doss,E.G.Mayer,R.C.Miller.1975.Thepotentialoflocalizedheatingasaadjuncttoradiationtherapy.Radiology116(02)433{439. Geva,A.B.,D.H.Kerem.1998.ForecastinggeneralizedepilepticseizuresfromtheEEGsignalbywaveletanalysisanddynamicunsupervisedfuzzyclustering.IEEETransactionsonBiomedicalEngineering45(10)1205{1216. Giard,D.J.,S.A.Aaronson,G.J.Todaro,P.Arnstein,J.H.Kersey,H.Dosik,W.P.Parks.1973.Invitrocultivationofhumantumors:Establishmentofcelllinesderivedfromaseriesofsolidtumors.JournaloftheNationalCancerInstitute51(5)1417. Glotsos,D.,P.Spyridonos,D.Cavouras,P.Ravazoula,P.ArapantoniDadioti,G.Nikiforidis.2005a.Animage-analysissystembasedonsupportvectormachinesforautomaticgradediagnosisofbrain-tumourastrocytomasinclinicalroutine.MedicalInformaticsandtheInternetinMedicine30(3)179{193. Glotsos,D.,J.Tohka,P.Ravazoula,D.Cavouras,G.Nikiforidis.2005b.Automateddiagnosisofbraintumoursastrocytomasusingprobabilisticneuralnetworkclusteringandsupportvectormachines.InternationalJournalofNeuralSystems151{11. Golub,T.R.,D.K.Slonim,P.Tamayo,C.Huard,M.Gaasenbeek,J.P.Mesirov,H.Coller,M.L.Loh,J.R.Downing,M.A.Caligiuri,C.D.Bloomeld,E.S.Lander.1999.Molecularclassicationofcancer:Classdiscoveryandclassprodictionbygeneexpressionmonitoring.Science286(5439)531{537. Greenberg,H.,W.Hart,G.Lancia.2004.Opportunitiesforcombinatorialoptimizationincomputationalbiology.INFORMSJournalonComputing16211{231. Guan,J.,Y.Chen,J.Lin.2005.Single-trialestimationofimitating-natural-readingevokedpotentialsinsingle-channel.Proceedingsofthe2005IEEEEngineeringinMedicineandBiology27thAnnualConference.2052{2055. Guigue,V.,A.Rakotomamonjy,S.Canu.2006.Translation-invariantclassicationofnon-stationarysignals.Neurocomputing69743{753. 154

PAGE 155

Hartigan,J.A.1972.Directclusteringofadatamatrix.JournaloftheAmericanStatisticalAssociation67(337)123{129. Hayashi,S.,M.Hatashita,H.Matsumoto,Z.H.Jin,H.Shioura,E.Kano.2005.ModicationofthermosensitivitybyamrubicinoramrubicinolinhumanlungadenocarcinomaA549cellsandthekineticsofapoptosisandnecrosisinduction.InternationalJournalofMolecularMedicine16(3)381{387. Hildebrandt,B.,P.Wust,O.Ahlers,A.Dieing,G.Sreenivasa,T.Kerner,R.Felix,H.Riess.2002.Thecellularandmolecularbasisofhyperthermia.CriticalReviewsinOncology/Hematology43(1)33{56. Hong,W.C.,P.F.Pai.2007.Potentialassessmentofthesupportvectorregressiontechniqueinrainfallforecasting.WaterResourcesManagement21(2)495{513. Horel,J.A.,L.J.Misantone.1976.Visualdiscriminationimpairedbycuttingtemporallobeconnections.Science193(4250)336{338. Hsu,C.W.,C.C.Chang,C.J.Lin.2004.Apracticalguidetosupportvectorclassication.http://www.csie.ntu.edu.tw/cjlin/papers/guide/guide.pdf. Hu,J.,J.Si,B.P.Olson,J.He.2005.Featuredetectioninmotorcorticalspikesbyprincipalcomponentanalysis.IEEETransactionsonNeuralSystemsandRehabilitationEngineering13(3)256{262. Huang,P.,W.Plunkett.1992.AquantitativeassayforfragmentedDNAinapoptoticcells.AnalyticalBiochemistry207(1)163{167. Huang,Z.,H.Chen,C.J.Hsu,W.H.Chenb,S.Wuc.2004.Creditratinganalysiswithsupportvectormachinesandneuralnetworks:Amarketcomparativestudy.DecisionSupportSystems37543{558. Huttunen,T.,J.P.Kaipio,M.Malinen.2008.OptimizationinMedicine,chap.Optimalcontrolinhighintensityfocusedultrasoundsurgery.Springer,169{195. Hyunsoo,K.,Z.X.Je,M.C.Herbert,P.Haesun.2005.Athree-stageframeworkforgeneexpressiondataanalysisbyL1-normsupportvectorregression.InternationalJournalofBioinformaticsResearchandApplications1(1)51{62. Hyvarinen,A.,J.Karhunen,E.Oja.2001.IndependentComponentAnalysis.JohnWiley&Sons. Iasemidis,L.1991.Onthedynamicsofthehumanbrainintemporallobeepilepsy.Ph.D.thesis,UniversityofMichigan,AnnArbor. 155

PAGE 156

Jaeschke,H.,J.S.Gujral,M.L.Bajt.2004.Apoptosisandnecrosisinliverdisease.LiverInternational:OcialjournaloftheInternationalAssociationfortheStudyoftheLiver24(2)85{89. Jain,A.N.,T.G.Dietterich,R.H.Lathrop,D.Chapman,R.E.Critchlow,B.E.Bauer,T.A.Webster,T.Lozano-Perez.1994.Ashape-basedmachinelearningtoolfordrugdesign.JournalofComputer-AidedMolecularDesign8(6)635{652. Jakuczun,W.,E.Kublik,D.K.Wojcik,A.Wrobel.2005.Localclassiersforevokedpotentialsrecordedfrombehavingrats.Actaneurobiologiaeexperimentalis65425{434. Joachims,T.1998.Textcategorizationwithsupportvectormachines:Learningwithmanyrelevantfeatures.ClaireNedellec,CelineRouveirol,eds.,ProceedingsoftheEuropeanConferenceonMachineLearning.Springer,Berlin,137{142. Joachims,T.1999.Makinglarge{scaleSVMlearningpractical.B.Scholkopf,C.J.C.Burges,A.J.Smola,eds.,AdvancesinKernelMethods:SupportVectorLearning.MITPress,Cambridge,MA,169{184. Jones,D.S.2002.PharmaceuticalStatistics.PharmaceuticalPress. Kalatzis,I.,D.Pappas,N.Piliouras,D.Cavouras.2003.SupportvectormachinesbasedanalysisofbrainSPECTimagesfordeterminingcerebralabnormalitiesinasymptomaticdiabeticpatients.MedicalInformaticsandtheInternetinMedicine28(3)221{230. Kanduc,D.,P.Bannasch,E.Farber.1999.Acriticalperspectiveincancerresearch(review).InternationalJournalofOncology15(6)1213{1220. Kanduc,D.,F.Capuano,S.A.Capurso,J.Geliebter,D.Guercia,A.Lucchese,A.Mittelman,S.M.Simone,A.A.Sinha,R.Tiwari,E.Farber.2003.Cancerpreventionandtherapy:strategiesandproblems.JournalofExperimentalThera-peutics&Oncology3(3)108{114. Kanduc,D.,J.Geliebter,A.Lucchese,R.Mazzanti,A.Mittelman,L.Polimeno,A.Ponzetto,R.Santacroce,S.Simone,E.Sinigaglia,A.A.Sinha,L.Tessitore,R.K.Tiwari,E.Farber.2005.Genetherapyincancer:Themissingpoint.JournalofExperimentalTherapeutics&Oncology5(2)151{158. Kanduc,D.,A.Mittelman,R.Serpico,E.Sinigaglia,A.A.Sinha,C.Natale,R.Santacroce,M.G.DiCorcia,A.Lucchese,L.Dini,P.Pani,S.Santacroce,S.Simone,R.Bucci,E.Farber.2002.Celldeath:Apoptosisversusnecrosis(review).InternationalJournalofOncology21(1)165{170. Kaper,M.,P.Meinicke,U.Grossekathoefer,T.Lingner,H.Ritter.2004.BCIcompetition2003{datasetiib:SupportvectormachinesfortheP300spellerparadigm.IEEETransactionsonBiomedicalEngineering51(6)1073{1076. 156

PAGE 157

Karpinich,N.O.,M.Tafani,R.J.Rothman,M.A.Russo,J.L.Farber.2002.Thecourseofetoposide-inducedapoptosisfromdamagetoDNAandp53activationtomitochondrialreleaseofcytochromec.JournalofBiologicalChemistry277(19). Karpouzas,I.,Y.Pouliquen.1991.Modellingandnumericaloptimizationofcornealrotation.MathematicalMedicineandBiology8(1)73{82. Keirn,Z.A.,J.I.Aunon.1990.Anewmodeofcommunicationbetweenmanandhissurroundings.IEEETransactionsonBiomedicalEngineering371209{1214. Kelm,B.M.,B.H.Menze,C.M.Zechmann,K.T.Baudendistel,F.A.Hamprecht.2007.Automatedestimationoftumorprobabilityinprostatemagneticresonancespectroscopicimaging:Patternrecognitionvs.quantication.MagneticResonanceinMedicine57150{159. Kluger,Y.,R.Basri,J.T.Chang,M.Gerstein.2003.Spectralbiclusteringofmicroarraydata:coclusteringgenesandconditions.GenomeResearch13(4)703{716. Kochenberger,G.,F.Glover,B.Alidaee,H.Wang.2005.Clusteringofmicroarraydataviacliquepartitioning.JournalofCombinatorialOptimization10(1)7{21. Kotropoulos,C.,I.Pitas.2003.Segmentationofultrasonicimagesusingsupportvectormachines.PatternRecognitionLetters24715{727. Kundakcioglu,O.E.,S.M.Nasseri,P.M.Pardalos.2009a.Supportvectorregressionwithmultipleinstancedatasubmitted. Kundakcioglu,O.E.,P.M.Pardalos.2008.Abranchandboundalgorithmformultipleinstanceclassication.H.R.Arabnia,Y.Mun,eds.,Proceedingsofthe2008Inter-nationalConferenceonMachineLearning;Models,TechnologiesandApplications(MLMTA),vol.2.865{869. Kundakcioglu,O.E.,P.M.Pardalos.2009a.ClusteringChallengesinBiologicalNetworks,chap.Thecomplexityoffeatureselectionforconsistentbiclustering.WorldScientic,257{266. Kundakcioglu,O.E.,P.M.Pardalos.2009b.LecturesonGlobalOptimization,FieldsInstituteCommunicationsSeries,vol.55,chap.Optimizationinbiomedicalresearch.AmericanMathematicalSociety,155{182. Kundakcioglu,O.E.,O.Seref,P.M.Pardalos.2009b.Multipleinstancelearningviamarginmaximization.AppliedNumericalMathematicsdoi:10.1016/j.apnum.2009.05.013. LaConte,S.,S.Strother,V.Cherkassky,J.Anderson,X.Hu.2005.SupportvectormachinesfortemporalclassicationofblockdesignfMRIdata.NeuroImage26317{329. 157

PAGE 158

Lahanas,M.,E.Schreibmann,D.Baltas.2003b.Multiobjectiveinverseplanningforintensitymodulatedradiotherapywithconstraint-freegradient-basedoptimizationalgorithms.PhysicsinMedicineandBiology48(17)2843{2871. Lal,T.N.,M.Schroeder,T.Hinterberger,J.Weston,M.Bogdan,N.Birbaumer,B.Scholkopf.2004.SupportvectorchannelselectioninBCI.IEEETransactionsonBiomedicalEngineering51(6)1003{1010. Lao,Z.,D.Shen,Z.Xue,B.Karacali,S.M.Resnick,C.Davatzikos.2004.Morphologicalclassicationofbrainsviahigh-dimensionalshapetransformationsandmachinelearningmethods.NeuroImage2146{57. Lauer,F.,G.Bloch.2008.Incorporatingpriorknowledgeinsupportvectorregression.MachineLearning7089{118. Lazzeroni,L.,A.Owen.2002.Plaidmodelsforgeneexpressiondata.StatisticaSinica12(1)61{86. Ledberg,A.,S.L.Bressler,M.Ding,R.Coppola,R.Nakamura.2007.Large-scalevisuomotorintegrationinthecerebralcortex.CerebralCortex1744{62. Lee,C.-H.,M.Schmidt,A.Murtha,A.Bistritz,J.Sander,R.Greiner.2005.Segmentingbraintumorswithconditionalrandomeldsandsupportvectormachines.ComputerVisionforBiomedicalImageApplications.469{478. Lee,D.D.,H.S.Seung.1999.Learningthepartsofobjectsbynon-negativematrixfactorization.Nature40788{791. Lee,E.K.2008.OptimizationinMedicine,chap.Optimization-basedpredictivemodelsinmedicineandbiology.Springer,127{151. Lee,E.K.,T.-L.Wu.2007.Classicationanddiseasepredictionviamathematicalprogramming.O.Seref,O.E.Kundakcioglu,P.M.Pardalos,eds.,DataMining,SystemsAnalysis,andOptimizationinBiomedicine.AmericanInstituteofPhysics,1{42. Lee,E.K.,T.Fox,I.Crocker.2001.Integerprogrammingappliedtointensity-modulatedradiationtreatmentplanningoptimization.AnnalsofOperationsResearch119165{181. Lee,E.K.,M.Zaider.2003.Mixedintegerprogrammingapproachestotreatmentplanningforbrachytherapy{applicationtopermanentprostateimplants.AnnalsofOperationsResearch119147{163. Lee,J.K.,P.D.Williams,S.Cheon.2008.Dataminingingenomics.ClinicsinLabora-toryMedicine28(1)145{166. 158

PAGE 159

Lehmann,C.,T.Koenig,V.Jelic,L.Prichep,R.E.John,L.-O.Wahlund,Y.Dodge,T.Dierks.2007.Applicationandcomparisonofclassicationalgorithmsforrecognitionofalzheimer'sdiseaseinelectricalbrainactivity(EEG).JournalofNeuroscienceMethods161342{350. Li,G.-Z.,T.-Y.Liu,V.S.Cheng.2006a.ClassicationofbraingliomabyusingSVMsbaggingwithfeatureselection.BioDM.124{130. Li,G.-Z.,J.Yang,C.-Z.Ye,D.-Y.Geng.2006b.Degreepredictionofmalignancyinbraingliomausingsupportvectormachines.ComputersinBiologyandMedicine36313{325. Li,Y.K.,P.L.Yang,Y.J.Jian,S.M.Ren,H.X.Zhao.2007.Applicationofsupportvectorregressionmethodinpredictingsoilerosionintensityofsmallwatershedintheinsensitiveerosionareas.JournalofBeijingForestryUniversity29(3)93{98. Liang,H.,J.D.Z.Chen.2005.Assessmentoftheesophagealpressureingastroesophagealreuxdiseasebythelocalregression.AnnalsofBiomedicalEngineering33(6)847{853. Liang,Nan-Ying,ParamasivanSaratchandran,Guang-BinHuang,NarasimhanSundararajan.2006.ClassicationofmentaltasksfromEEGsignalsusingextremelearningmachine.InternationalJournalofNeuralSystems16(1)29{38. Liu,J.,W.Wang.2003.OP-Cluster:Clusteringbytendencyinhighdimensionalspace.ThirdIEEEInternationalConferenceonDataMining.187{194. Liu,Y.,L.Teverovskiy,O.T.Carmichael,R.Kikinis,M.Shenton,C.S.Carter,V.A.Stenger,S.Davis,H.Aizenstein,J.Becker,O.Lopez,C.Meltzer.2004.DiscriminativeMRimagefeatureanalysisforautomaticschizophreniaandalzheimer'sdiseaseclassication.LectureNotesComputerScience3216393{401. Lloyd,S.1982.LeastsquaresquantizationinPCM.IEEETransactionsonInformationTheory28(2)129{137. Lodwick,W.A.,S.McCourt,F.Newman,S.Humphries.1999.ComputationalRadiologyandImaging:TherapyandDiagnostics,chap.OptimizationMethodsforRadiationTherapyPlans.Springer,229{248. Long,P.M.,L.Tan.1998.PAClearningaxisalignedrectangleswithrespecttoproductdistributionsfrommultipleinstanceexamples.MachineLearning307{22. Louis,A.K.2008.OptimizationinMedicine,chap.Optimalreconstructionkernelsinmedicalimaging.Springer,153{168. Lu,Y.,S.Y.Lu,F.Fotouhi,Y.P.Deng,S.J.Brown.2004.Incrementalgenetick-meansalgorithmanditsapplicationingeneexpressiondataanalysis.BMCBioinformatics5(172). 159

PAGE 160

Madeira,S.C.,A.L.Oliveira.2004.Biclusteringalgorithmsforbiologicaldataanalysis:Asurvey.IEEETransactionsonComputationalBiologyandBioinformatics124{45. Mammadov,M.,A.Rubinov,J.Yearwood.2007a.DataMininginBiomedicine,Opti-mizationandItsApplications,vol.7,chap.Anoptimizationapproachtoidentifytherelationshipbetweenfeaturesandoutputofamulti-labelclassier.Springer,141{167. Mammadov,M.A.,A.M.Rubinov,J.Yearwood.2007b.Thestudyofdrug-reactionrelationshipsusingglobaloptimizationtechniques.OptimizationMethods&Software22(1)99{126. Mangasarian,O.L.1994.NonlinearProgramming.SIAM,Philadelphia. Mangasarian,O.L.,W.N.Street,W.H.Wolberg.1995.Breastcancerdiagnosisandprognosisvialinearprogramming.OperationsResearch43570{577. Mangasarian,O.L.,E.W.Wild.2008.Multipleinstanceclassicationviasuccessivelinearprogramming.JournalofOptimizationTheoryandApplications137(3)555{568. Maquelin,K.,L.P.Choo-Smith,T.vanVreeswijk,H.P.Endtz,B.Smith,R.Bennett,H.A.Bruining,G.J.Puppels.1999.Ramanspectroscopicmethodforidenticationofclinicallyrelevantmicroorganismsgrowingonsolidculturemedium.AnalyticalChemistry72(1)12{19. Marchuk,GuriI.1997.MathematicalModellingofImmuneResponseinInfectiousDiseases.KluwerAcademicPublishers. Maron,O.1998.Learningfromambiguity.Tech.rep.,DepartmentofElectricalEngineeringandComputerScience,MassachusettsInstituteofTechnology,Cambridge,MA.Ftp://publications.ai.mit.edu/ai-publications/pdf/AITR-1639.pdf. Maron,O.,A.L.Ratan.1998.Multiple-instancelearningfornaturalsceneclassication.ProceedingsoftheFifteenthInternationalConferenceonMachineLearning.MorganKaufmannPublishersInc.,SanFrancisco,CA,USA,341{349. Martinez-Ramon,M.,V.Koltchinskii,G.L.Heileman,S.Posse.2006.fMRIpatternclassicationusingneuroanatomicallyconstrainedboosting.NeuroImage311129{1141. McAllister,S.R.,R.Rajgaria,C.A.Floudas.2007.Globalpairwisesequencealignmentthroughmixed-integerlinearprogramming:atemplate-freeapproach.OptimizationMethods&Software22(1)127{144. 160

PAGE 161

Mendola,J.D.,S.Corkin.1999.Visualdiscriminationandattentionafterbilateraltemporal-lobelesions:Acasestudy.Neuropsychologia37(1)91{102. Meneses,C.N.,C.A.S.Oliveira,P.M.Pardalos.2007.DataMininginBiomedicine,Op-timizationandItsApplications,vol.7,chap.Mathematicalprogrammingformulationsforproblemsingenomicsandproteomics.Springer,275{290. Menze,B.H.,M.P.Lichy,P.Bachert,B.M.Kelm,H.-P.Schlemmer,F.A.Hamprecht.2006.Optimalclassicationoflongechotimeinvivomagneticresonancespectrainthedetectionofrecurrentbraintumors.NMRinBiomedicine19(5)599{609. Mour~ao-Miranda,J.,K.J.Friston,M.Brammer.2007.Dynamicdiscriminationanalysis:Aspatial-temporalSVM.NeuroImage3688{99. Mour~ao-Miranda,J.,E.Reynaud,F.McGlone,G.Calvert,M.Brammer.2006.TheimpactoftemporalcompressionandspaceselectiononSVManalysisofsingle-subjectandmulti-subjectfMRIdata.NeuroImage331055{1065. Muller,KlausRobert,CharlesW.Anderson,GaryE.Birch.2003.Linearandnonlinearmethodsforbrain-computerinterfaces.IEEETransactionsonNeuralSystemsandRehabilitationEngineering11(2)165{169. Murray,J.F.,G.F.Hughes,K.Kreutz-Delgado.2005.Machinelearningmethodsforpredictingfailuresinharddrives:Amultiple-instanceapplication.TheJournalofMachineLearningResearch6783{816. Musicant,D.R.,J.M.Christensen,J.F.Olson.2007.Supervisedlearningbytrainingonaggregateoutputs.ICDM'07:Proceedingsofthe2007SeventhIEEEInternationalConferenceonDataMining.IEEEComputerSociety,Washington,DC,USA,252{261. Nahapetyan,A.,S.Busygin,P.M.Pardalos.2008.MathematicalModellingofBiosystems,chap.Animprovedheuristicforconsistentbiclusteringproblems.Springer,185{198. Navarre,W.W.,A.Zychlinsky.2000.Pathogen-inducedapoptosisofmacrophages:Acommonendfordierentpathogenicstrategies.CellularMicrobiology2(4)265{273. Noble,W.S.2004.KernelMethodsinComputationalBiology,chap.Supportvectormachineapplicationsincomputationalbiology.MITPress,71{92. Notingher,I.,C.Green,C.Dyer,E.Perkins,N.Hopkins,C.Lindsay,L.L.Hench.2004.DiscriminationbetweenricinandsulphurmustardtoxicityinvitrousingRamanspectroscopy.JournaloftheRoyalSociety,Interface1(1)79{90. 161

PAGE 162

Notingher,I.,S.Verrier,H.Romanska,A.E.Bishop,J.M.Polak,L.L.Hench.2002.InsitucharacterisationoflivingcellsbyRamanspectroscopy.Spectroscopy{AnInternationalJournal16(2)43{51. Olson,B.P.,J.Si,J.Hu,J.He.2005.Closed-loopcorticalcontrolofdirectionusingsupportvectormachines.IEEETransactionsonNeuralSystemsandRehabilitationEngineering13(1)72{80. Osuna,R.F.E.,F.Girosi.1997.Animprovedtrainingalgorithmforsupportvectormachines.IEEEWorkshoponNeuralNetworksforSignalProcessing.276{285. Owen,C.A.,J.Selvakumaran,I.Notingher,G.Jell,L.L.Hench,M.M.Stevens.2006.InvitrotoxicologyevaluationofpharmaceuticalsusingRamanmicro-spectroscopy.JournalofCellularBiochemistry99(1)178{186. Pai,P.F.,W.C.Hong.2007.Arecurrentsupportvectorregressionmodelinrainfallforecasting.HydrologicalProcesses21(6)819{827. Pardalos,P.M.,V.L.Boginski,O.A.Prokopyev,W.Suharitdamrong,P.R.Carney,W.Chaovalitwongse,A.Vazacopoulos.2005.EssaysandSurveysinGlobalOptimiza-tion,chap.Optimizationtechniquesinmedicine.Springer,211{232. Pardalos,P.M.,W.Chaovalitwongse,L.D.Iasemidis,J.C.Sackellares,D.-S.Shiau,P.R.Carney,O.A.Prokopyev,V.A.Yatsenko.2004.Seizurewarningalgorithmbasedonoptimizationandnonlineardynamics.MathematicalProgramming101(2)365{385. Parra,L.C.,C.D.Spence,A.D.Gerson,P.Sajda.2005.RecipesforthelinearanalysisofEEG.NeuroImage28326{341. Pascual-Montano,A.,P.Carmona-Saez,M.Chagoyen,F.Tirado,J.M.Carazo,R.D.Pascual-Marqui.2006.bioNMF:Aversatiletoolfornon-negativematrixfactorizationinbiology.BMCBioinformatics7(366). Pessoa,L.,S.Padmala.2007.Decodingnear-thresholdperceptionoffearfromdistributedsingle-trialbrainactivation.CerebralCortex17691{701. Pugfelder,D.,J.J.Wilkens,U.Oelfke.2008.Worstcaseoptimization:Amethodtoaccountforuncertaintiesintheoptimizationofintensitymodulatedprotontherapy.PhysicsinMedicineandBiology531689{1700. Platt,J.1999.FasttrainingofSVMsusingsequentialminimaloptimization.B.Scholkopf,C.J.C.Burges,A.J.Smola,eds.,AdvancesinKernelMethods:SupportVectorLearning.MITPress,Cambridge,MA,185{208. 162

PAGE 163

Prasad,K.V.,A.Taiyab,D.Jyothi,U.K.Srinivas,A.S.Sreedhar.2007.Heatshocktranscriptionfactorsregulateheatinducedcelldeathinarathistiocytoma.JournalofBiosciences32(3)585{593. Pyrgiotakis,G.,T.K.Bhowmick,K.Finton,A.K.Suresh,S.G.Kane,J.R.Bellare,B.M.Moudgil.2008.Cell(A549)-particle(JasadaBhasma)interactionsusingRamanspectroscopy.Biopolymers89(6)555{64. Pyrgiotakis,G.,O.E.Kundakcioglu,K.Finton,P.M.Pardalos,K.Powers,B.M.Moudgil.2009.CelldeathdiscriminationwithRamanspectroscopyandsupportvectormachines.AnnalsofBiomedicalEngineering37(7)1464{1473. Qi,X.,Y.Han.2007.IncorporatingmultipleSVMsforautomaticimageannotation.PatternRecognition40728{741. Quddus,A.,P.Fieguth,O.Basir.2005.AdaboostandsupportvectormachinesforwhitematterlesionsegmentationinMRimages.Proceedingsofthe2005IEEEEngineeringinMedicineandBiology27thAnnualConference.463{466. Ragle,M.A.,J.C.Smith,P.M.Pardalos.2007.Anoptimalcutting-planealgorithmforsolvingthenon-uniqueprobeselectionproblem.AnnalsofBiomedicalEngineering35(11)2023{2030. Ray,S.2005.Learningfromdatawithcomplexinteractionsandambiguouslabels.Ph.D.thesis,UniversityofWisconsin-Madison. Ray,S.,M.Craven.2005.Supervisedversusmultipleinstancelearning:Anempiricalcomparison.ICML'05:Proceedingsofthe22ndinternationalconferenceonMachinelearning.ACMPress,NewYork,NY,USA,697{704. Raykar,V.C.,B.Krishnapuram,J.Bi,M.Dundar,R.B.Rao.2008.Bayesianmultipleinstancelearning:Automaticfeatureselectionandinductivetransfer.ICML'08:Proceedingsofthe25thinternationalconferenceonMachinelearning.ACM,NewYork,NY,USA,808{815. Rinaldi,L.,P.Gallo,F.Ranzato,D.Luise,D.Colavito,M.Motta,A.Guglielmo,E.DelGiudice,C.Romualdi,E.Ragazzi,A.Darrigo,M.DalleCarbonare,B.Leontino,A.Leon.2006.Longitudinalanalysisofimmunecellphenotypesinearlystagemultiplesclerosis:distinctivepatternscharacterizeMRI-activepatients.Brain1291993{2007. Robinson,J.E.,M.J.Wizenberg,W.A.McCready.1974.Combinedhyperthermiaandradiationsuggestandalternativetoheavyparticletherapyforreducedoxygenenhancementratios.Nature251(5475)521{522. 163

PAGE 164

Rubinov,A.M.,J.Udon.2003.Skeletonsofnitesetsofpoints.Tech.rep.,CentreforInformaticsandAppliedOptimizationoftheUniversityofBallarat. Russell,F.,M.Akay.1996.Acomparisonofanalyticalmethodsforthestudyoffractionalbrownianmotion.AnnalsofBiomedicalEngineering24(4)1{3. Sabesan,S.,L.Good,N.Chakravarthy,K.Tsakalis,P.M.Pardalos,L.Iasemidis.2008.OptimizationinMedicine,chap.Globaloptimizationandspatialsynchronizationchangespriortoepilepticseizures.Springer,103{125. Sakhanenko,N.A.,G.F.Luger.2006.Shockphysicsdatareconstructionusingsupportvectorregression.InternationalJournalofModernPhysics17(9)1313{1325. Sapareto,S.A.,W.C.Dewey.1984.Thermaldosedeterminationincancertherapy.InternationalJournalofRadiationOncology,Biology,Physics10(6)787{800. Schapire,R.2001.Theboostingapproachtomachinelearning:Anoverview. Scholkopf,B.,A.J.Smola.2002.LearningwithKernels.MITPress,Cambridge. Segal,E.,B.Taskar,A.Gasch,N.Friedman,D.Koller.2001.Richprobabilisticmodelsforgeneexpression.Bioinformatics17243{252. Seref,O.,C.Cifarelli,O.E.Kundakcioglu,P.M.Pardalos,M.Ding.2007.Detectingcategoricaldiscriminationinavisuomotortaskusingselectivesupportvectormachines.H.R.Arabnia,M.Q.Yang,J.Y.Yang,eds.,Proceedingsofthe2007InternationalConferenceonBioinformatics&ComputationalBiology(BIOCOMP),vol.2.580{587. Seref,O.,O.E.Kundakcioglu,M.Bewernitz.2008a.EncyclopediaofHealthcareInfor-mationSystems,chap.Supportvectormachinesinneuroscience.IDEAGroupInc.,1283{1293. Seref,O.,O.E.Kundakcioglu,P.M.Pardalos,eds.2008b.DataMining,SystemsAnalysis,andOptimizationinBiomedicine.953,AmericanInstituteofPhysics. Seref,O.,O.E.Kundakcioglu,P.M.Pardalos.2008c.Selectivelinearandnonlinearclassication.P.M.Pardalos,P.Hansen,eds.,CRMProceedingsandLectureNotes,vol.45.211{234. Seref,O.,O.E.Kundakcioglu,O.A.Prokopyev,P.M.Pardalos.2009.Selectivesupportvectormachines.JournalofCombinatorialOptimization17(1)3{20. Shawe-Taylor,J.,N.Cristianini.2004.Kernelmethodsforpatternanalysis.CambridgeUniversityPress,Cambridge,UK. 164

PAGE 165

Shepard,D.M.,M.C.Ferris,G.H.Olivera,T.R.Mackie.1999.Optimizingthedeliveryofradiationtherapytocancerpatients.SIAMReview41(4)721{744. Shoker,L.,S.Sanei,A.Sumich.2005.DistinguishingbetweenleftandrightngermovementfromEEGusingSVM.Proceedingsofthe2005IEEEEngineeringinMedicineandBiology27thAnnualConference.5420{5423. Sitaram,R.,H.Zhang,C.Guan,M.Thulasidas,Y.Hoshi,A.Ishikawa,K.Shimizu,N.Birbaumer.2007.Temporalclassicationofmultichannelnear-infraredspectroscopysignalsofmotorimageryfordevelopingabrain-computerinterface.NeuroImage341416{1427. Solovyan,V.,Z.Bezvenyuk,V.Huotari,T.Tapiola,T.Suuronen,A.Salminen.1998.Distinctmodeofapoptosisinducedbygenotoxicagentetoposideandserumwithdrawalinneuroblastomacells.MolecularBrainResearch62(1)43{55. Street,W.N.,O.L.Mangasarian,W.H.Wolberg.1995.Aninductivelearningapproachtoprognosticprediction.inMachineLearning:ProceedingsoftheTwelfthInternationalConference.MorganKaufmann,522{530. Strickler,S.S.,A.V.Gribenko,A.V.Gribenko,T.R.Keier,J.Tomlinson,T.Reihle,V.V.Loladze,G.I.Makhatadze.2006.Proteinstabilityandsurfaceelectrostatics:Achargedrelationship.Biochemistry45(9)2761{2766. Sujansky,W.2001.Heterogeneousdatabaseintegrationinbiomedicine.JournalofBiomedicalInformatics34(4)285{298. Sun,Y.F.,Y.C.Liang,C.G.Wu,X.W.Yang,H.P.Lee,W.Z.Lin.2004.Estimateoferrorboundsintheimprovedsupportvectorregression.ProgressinNaturalScience14(4)362{364. Suykens,J.A.K.,J.Vandewalle.1999.Leastsquaressupportvectormachineclassiers.NeuralProcessingLetters9293{300. T.-H.Wu.1997.Anoteonaglobalapproachforgeneral0-1fractionalprogramming.EuropeanJournalOfOperationalResearch16220{223. Tadashi,K.,K.Takao,N.Takeo,A.Hiroshi,E.Shin,N.Masaaki,T.Hideaki,Y.Tadashi,K.Seiji,T.Ryuichi.2004.Mildheatshockinducesautophagicgrowtharrest,butnotapoptosisinu251-mgandu87-mghumanmalignantgliomacells.JournalofNeuro-Oncology68101{111. Tanay,A.,R.Sharan,R.Shamir.2002.Discoveringstatisticallysignicantbiclustersingeneexpressiondata.Bioinformatics18136{144. 165

PAGE 166

Tao,Q.,S.Scott,N.V.Vinodchandran,T.T.Osugi.2004.SVM-basedgeneralizedmultiple-instancelearningviaapproximateboxcounting.ICML'04:ProceedingsoftheTwenty-rstInternationalConferenceonMachineLearning.ACMPress,NewYork,NY,USA,799{806. Tawarmalani,M.,S.Ahmed,N.V.Sahinidis.2002.Globaloptimizationof0{1hyperbolicprograms.JournalofGlobalOptimization24(4)385{416. Thai,M.T.,Z.Cai,D.Z.Du.2007a.Geneticnetworks:processingdata,regulatorynetworkmodellingandtheiranalysis.OptimizationMethods&Software22(1)169{185. Thai,M.T.,P.Deng,W.Wu,T.Znati.2007b.Approximationalgorithmsofnon-uniqueprobesselectionforbiologicaltargetidentication.O.Seref,O.E.Kundakcioglu,P.M.Pardalos,eds.,DataMining,SystemsAnalysis,andOptimizationinBiomedicine.AmericanInstituteofPhysics,174{184. Thulasidas,M.,C.Guan,J.Wu.2006.RobustclassicationofEEGsignalforbrain-computerinterface.IEEETransactionsonNeuralSystemsandRehabilitationEngineering14(1)24{29. Trafalis,T.B.,H.Ince.2000.Supportvectormachineforregressionandapplicationstonancialforecasting.InternationalJointConferenceonNeuralNetworks,vol.6.Como,Italy,348{353. Ugur,O.,G.W.Weber.2007.Optimizationanddynamicsofgene-environmentnetworkswithintervals.JournalofIndustrialandManagementOptimization3(2)357{379. Uthman,B.,M.Bewernitz,C.-C.Liu,G.Ghacibeh.2007.Optimizationofepilepsytreatmentwithvagusnervestimulation.O.Seref,O.E.Kundakcioglu,P.M.Pardalos,eds.,DataMining,SystemsAnalysis,andOptimizationinBiomedicine.AmericanInstituteofPhysics,308{315. Vapnik,V.1995.TheNatureofStatisticalLearningTheory.Springer-Verlag. Vapnik,V.1998.StatisticalLearningTheory.Wiley,NewYork. Verrier,S.,I.Notingher,J.M.Polak,L.L.Hench.2004.InsitumonitoringofcelldeathusingRamanmicrospectroscopy.Biopolymers74(1-2)157{162. Viola,P.,J.C.Platt,C.Zhang.2006.Multipleinstanceboostingforobjectdetection.NeuralInformationProcessingSystems,vol.18.MITPress,Vancouver,BritishColumbia,Canada,1419{1426. Wang,J.,J.Zucker.2000.Solvingthemultiple-instanceproblem:Alazylearningapproach.Proc.17thInternationalConf.onMachineLearning.MorganKaufmann,SanFrancisco,CA,1119{1125. 166

PAGE 167

Webb,S.1991.Optimizationbysimulatedannealingofthree-dimensionalconformaltreatmentplanningforradiationeldsdenedbyamultileafcollimator.PhysicsinMedicineandBiology36(9)1201{1226. Weston,J.,S.Mukherjee,O.Chapelle,M.Pontil,T.Poggio,V.Vapnik.2000.FeatureselectionforSVMs.NIPS.668{674. Widjaja,E.,G.H.Lim,A.An.2008a.AnovelmethodforhumangenderclassicationusingRamanspectroscopyofngernailclippings.TheRoyalSocietyofChemistry133493{498. Widjaja,E.,W.Zheng,Z.Huang.2008b.Classicationofcolonictissuesusingnear-infraredRamanspectroscopyandsupportvectormachines.InternationalJournalofOncology32(3)653{662. Wolf,A.,J.B.Swift,H.L.Swinney,J.A.Vastano.1985.Determininglyapunovexponentsfromatimeseries.PhysicaD16285{317. Wolsey,L.A.1998.IntegerProgramming.Wiley-Interscience,NewYork. Wu,Z.-L.,C.-H.Li,J.K.-Y.Ng,K.R.P.H.Leung.2007.Locationestimationviasupportvectorregression.IEEETransactionsonMobileComputing6(3)311{321. Xie,X.S.,W.T.Liu,B.Y.Tang.2008.Spacebasedestimationofmoisturetransportinmarineatmosphereusingsupportvectorregression.RemoteSensingofEnvironment112(4)1846{1855. Xing,E.P.,R.M.Karp.2001.CLIFF:Clusteringofhigh-dimensionalmicroarraydataviaiterativefeaturelteringusingnormilizedcuts.BioinformaticsDiscoveryNote17306{315. Yamamoto,K.,F.Asano,T.Yamada,N.Kitawaki.2006.Detectionofoverlappingspeechinmeetingsusingsupportvectormachinesandsupportvectorregression.IEICETransactionsonFundamentalsofElectronicsCommunicationsandComputerSciencesE89A(8)2158{2165. Yogalingam,G.,A.M.Pendergast.1997.Serumwithdrawalandetoposideinduceapoptosisinhumanlungcarcinomacelllinea549viadistinctpathways.Apoptosis2(2)199{206. Yogalingam,G.,A.M.Pendergast.2008.ABLkinasesregulateautophagybypromotingthetrackingandfunctionoflysosomalcomponents.JournalofBiologicalChemistry283(51)35941{35953. 167

PAGE 168

Yoon,U.,J.M.Lee,J.J.Kim,S.M.Lee,I.Y.Kim,J.S.Kwon,S.I.Kim.2003.Modiedmagneticresonanceimagebasedparcellationmethodforcerebralcortexusingsuccessivefuzzyclusteringandboundarydetection.AnnalsofBiomedicalEngineering31(4)441{447. Yushkevich,P.,A.Dubb,Z.Xie,R.Gur,R.Gur,J.Gee.2005.Regionalstructuralcharacterizationofthebrainofschizophreniapatients.AcademicRadiology12(10)1250{1261. Zarnitsyn,V.G.,M.R.Prausnitz.2004.Physicalparametersinuencingoptimizationofultrasound-mediatedDNAtransfection.UltrasoundinMedicine&Biology30(4)527{538. Zhang,Q.,S.Goldman.2001.EM-DD:Animprovedmultiple-instancelearningtechnique.NeuralInformationProcessingSystems,vol.14.MITPress,Vancouver,BritishColumbia,Canada,1073{1080. Zibulevsky,M.,B.A.Pearlmutter.2001.Blindsourceseparationbysparsedecompositioninasignaldictionary.NeuralComputation13(4)863{882. 168

PAGE 169

O.ErhunKundakcioglureceivedhisPh.D.degreeinIndustrialandSystemsEngineeringattheUniversityofFlorida.Hisresearchfocusesonoptimizationtechniquesforpatternrecognitionandmachinelearning.Mr.Kundakciogluisalsointerestedinproductionandinventoryplanningproblems.Heisthe2008recipientoftheFloridaChapterScholarshipgivenbytheHealthcareInformationandManagementSystemsSocietyFoundation.HiscontributiontothedepartmenthasalsobeenrecognizedbyrewardinghimtheGraduateStudentAwardforExcellenceinResearchandTeachingin2008and2009,respectively. 169