<%BANNER%>

New Optimization Methods and Applications in Kernel-Based Machine Learning

xml version 1.0 encoding UTF-8
REPORT xmlns http:www.fcla.edudlsmddaitss xmlns:xsi http:www.w3.org2001XMLSchema-instance xsi:schemaLocation http:www.fcla.edudlsmddaitssdaitssReport.xsd
INGEST IEID E20101219_AAAADE INGEST_TIME 2010-12-19T18:10:47Z PACKAGE UFE0017560_00001
AGREEMENT_INFO ACCOUNT UF PROJECT UFDC
FILES
FILE SIZE 1150 DFID F20101219_AABRCA ORIGIN DEPOSITOR PATH seref_o_Page_050.txt GLOBAL false PRESERVATION BIT MESSAGE_DIGEST ALGORITHM MD5
4d66fcd83a56fe1aeae6234245b80a90
SHA-1
88c8761d8e3fb303ae9611a839d01c179d86ac0f
2403 F20101219_AABRBM seref_o_Page_018.txt
d040427fc50d265b4b2595e46438d815
0c7eb886a57356406f5e96272d7b1c9cbc51e98c
1051953 F20101219_AABQWG seref_o_Page_047.jp2
f1932c293fcc173ac17d3bcc07bb1ac8
637cdc044241efbd8203d51b0dd2791b3f84267d
62382 F20101219_AABRAY seref_o_Page_109.pro
fb6e09441b1772b0a821614321aba978
6761199dd3385751425e89f9f6a3f7369f3ac30c
120576 F20101219_AABQVS seref_o_Page_012.jp2
d43c65e86b1c3abba8dcde529963eb7a
d372cfee2664405bb56b3bc88ca985eb33b569da
1162 F20101219_AABRCB seref_o_Page_051.txt
82ca2cf3d1acb5561e2ebdca7351f11e
0808ff1a6cbf242b42437cf86e99b257a3aea205
2160 F20101219_AABRBN seref_o_Page_019.txt
deacf726e7957ef5067e5574a3665a0c
efaa8ed4a40077406d942d7bee3d373fa3bea1e8
492771 F20101219_AABQWH seref_o_Page_049.jp2
12ac0315efa2ac0438f0cbf8788ff785
ccaa3b1d6ce5268f6ac6114ff295353f7a43677b
63989 F20101219_AABRAZ seref_o_Page_111.pro
c8ff30a7af0fd3067c48f24617b4b1bf
add3b033e07f01eada74f53c6937a748cdc60ad8
2275 F20101219_AABRBO seref_o_Page_024.txt
c8b3ca7707ba7fe41afd1f50e368ae75
0a3dabe58aa78cb39964f75c820183e647e586e3
514969 F20101219_AABQWI seref_o_Page_050.jp2
12602f4a4c972b779c6165eba75edb94
94d1d94eb599c080fa41fcc275f98d4dbd46648f
113942 F20101219_AABQVT seref_o_Page_019.jp2
a944f49e23055116f2400641ccfd62d9
533ad1dabbffc5b8ee959265d3e81d18917509da
1947 F20101219_AABRCC seref_o_Page_053.txt
4ead915298d7345a719e6c580e635cf9
808641c1c9ea281644d4f647aa4e383f17e14b44
2029 F20101219_AABRBP seref_o_Page_025.txt
0bb7b01769d5422ec63a4081e284012d
8e02dd0a761505ad9925bf2bc3b6fc7643ba2f4b
587720 F20101219_AABQWJ seref_o_Page_052.jp2
b4ed8244027d4f8426a98fd51863677a
aff0a30d0c0620d6ee613467529ea8226edcd5ec
653923 F20101219_AABQVU seref_o_Page_021.jp2
a133c020810ec446f311f3fa17f13627
198d370bff27698ef7fbecb43343df98e993cfbe
1038 F20101219_AABRCD seref_o_Page_058.txt
eeba0bb0024e920155b5d94f6fe7a555
95b66d56361edf67b07157ab422f779dc3df5209
1796 F20101219_AABRBQ seref_o_Page_027.txt
81f6322da2cf2da7ef839684bd95643e
c18e3d1d3c759a0fa371b48b4bca295aab624c00
1042005 F20101219_AABQWK seref_o_Page_053.jp2
6e994b620f2598af632d02851b7c1dae
6ba8418227e8b8ba5ef383d5596af3d1fee29d98
883396 F20101219_AABQVV seref_o_Page_022.jp2
cfae60aef0504bd944b8f0159354baf1
2cfbebdc57f4931d8c78beceb11e35d5d598e658
1052 F20101219_AABRCE seref_o_Page_059.txt
4cc3e1bf04d6c1b26d24fe8ef33c6bf1
86051ca821409cfb0e1c76b74920b7e4881af0b6
1660 F20101219_AABRBR seref_o_Page_029.txt
44a64df2c162321cb008dcbf6f0afa3b
93af4b427656a595f7b91daa793f938bfb7d68d7
1051703 F20101219_AABQWL seref_o_Page_054.jp2
d452e0bc604c4e28f495febdf8e6c52b
7466e4e1e233997a127148b5214a0b7675d9e3f5
778854 F20101219_AABQVW seref_o_Page_023.jp2
04d0851c98e73b837389d82a0378cd5c
41f66134986c4a482019638be3a9f28446b50d10
1746 F20101219_AABRCF seref_o_Page_062.txt
85a4ae45a0dd3086c4d468bd4264a015
074b9b6e372f2c11bc460ea801160e04412d22d0
393764 F20101219_AABQXA seref_o_Page_084.jp2
85a07c2b21fca0417bf5367621c36d4a
c0f3334a23ba2a4fcb09a07b52f33744ce00fcbf
2167 F20101219_AABRBS seref_o_Page_031.txt
396a26bac0248e68af84adabe7b9dcaf
1a72d0c546081502d32f18971cb7012ac4356aba
917098 F20101219_AABQWM seref_o_Page_056.jp2
49b414e1bb8d51d0151f25e99a5c1b08
72f58999ba07b450eff57f4b673bd06489965305
1051975 F20101219_AABQVX seref_o_Page_024.jp2
aa02747b1ed592ed2f7c4cb0a0466639
9b9fc3766197d339d7e704bd4c8f3f16fd45324d
343 F20101219_AABRCG seref_o_Page_063.txt
d5b0757df5b16852ef61e706bb3c986e
1ecc9a63b5f816f85537f337d88c16da92e5cb7d
1051980 F20101219_AABQXB seref_o_Page_085.jp2
840829951cc600f760eba0cb1976416f
45d1e222dde0b6c2e54537a395fd4406de3056ff
1000 F20101219_AABRBT seref_o_Page_032.txt
d21b3803dfe9d017fd4ee50d3673beaf
994201fb609b1692ffebda71454f3716d72cbef2
535583 F20101219_AABQWN seref_o_Page_059.jp2
07d6d41ac9c527d4d0ad88a34ec54274
f53b3bf6ccb1ff1bf075f05d70b978e3ce18ad26
1023062 F20101219_AABQVY seref_o_Page_025.jp2
ca77f5d8d8409bd3755d17b549dc15fd
6597decbac030b390fd9083ea2188c5ec51c8f80
2231 F20101219_AABRCH seref_o_Page_064.txt
44fcc850a01be2d23658b828bbbb4d98
9e1c1cf274aa621d4256fb6f74af3aa665270a2b
899061 F20101219_AABQXC seref_o_Page_087.jp2
13314b8ff40eb6e67c6967b988b0aeec
5df2baa7a780698af4c55c3032a23ce7bf50aac6
1344 F20101219_AABRBU seref_o_Page_036.txt
a071fff6b814a8080ea7e56e3aebd719
d5cd1a244ff026a5328f13c9748476c88a60e798
835556 F20101219_AABQWO seref_o_Page_062.jp2
3f42b5ab2d60dc773000d502c9a23b13
57b50d6d5ff87c660299653f348bcd6286dc45f2
728865 F20101219_AABQVZ seref_o_Page_028.jp2
625b740efb9c0fd61f87628bbe67a40e
daa66686e5022c827a6a386a8948e7e4634c7ee8
1683 F20101219_AABRCI seref_o_Page_065.txt
b16b75c3662d3b3ea8e8ee28946e3b5e
500f0f46d07af6a0864b0d0ab22c469c6d05e05d
817205 F20101219_AABQXD seref_o_Page_089.jp2
e924fbdc3d8be01d92186a6c0b2dbb81
3ee0498ba1d7c064b9f7424154d9399e128637cc
1431 F20101219_AABRBV seref_o_Page_038.txt
fe65ac6548c6ef83b5006373a0d0b5e2
86e6d150863345928951ef538e9adbfacb771e04
1051939 F20101219_AABQWP seref_o_Page_065.jp2
6fb69b15ae3aafaca5d9f2f62152a919
181cb22f8c35e99b18856c8e637c2462c7060153
1593 F20101219_AABRCJ seref_o_Page_066.txt
9861a7973d606cde7bdbe5999e81ac8b
f1af39aa84715cb3cf429d33820c3c69c3acf63e
766179 F20101219_AABQXE seref_o_Page_090.jp2
b71894fc02e71be1673e15adc0f4f135
48f28641e3645d52139255ce6490b18100f06621
1613 F20101219_AABRBW seref_o_Page_039.txt
d1eeef82a5f552a24afc411c784082c4
77bbeb318ca4123b8120be7ee10ff981a90dde51
1051964 F20101219_AABQWQ seref_o_Page_067.jp2
b308ea39d226ca4f308e2ce1ecf93b39
2ab00a691e1552d0d7e9134d219623ac8e5a54d2
2371 F20101219_AABRCK seref_o_Page_070.txt
5526ae6baf09238b24f1c7193293b8c9
bc2967bb52f6b49e607b13097a5cb170e4b1337b
889124 F20101219_AABQXF seref_o_Page_091.jp2
90703b09e76bf60cc657ac488e39904d
f952413ef330398802d79838f9d36e426ea6f05b
1587 F20101219_AABRBX seref_o_Page_040.txt
24c7a32934630c04000cad04eb9fd3f9
c4b5dec05edaee2abd0ade680b49434c73fc97b1
1051984 F20101219_AABQWR seref_o_Page_068.jp2
29127ed756687492a49e579fd0a38313
e256e3a6d6df5c6bcf80a1116fe0ae1ea22b9d92
2198 F20101219_AABRCL seref_o_Page_071.txt
1844fa36d486c391a7e3833f04a66210
7668003a5cd43859d1b71645dcfaafa302f97486
904184 F20101219_AABQXG seref_o_Page_097.jp2
e8b50f3365f552316f9063d4801185e8
74d41e1f098fe0ccafd6e66d90a6df78086651c8
1254 F20101219_AABRBY seref_o_Page_041.txt
e320afa1ead39f03691a6b99f32bc14d
515f79417197bab57432bf0222668014b6040723
104515 F20101219_AABQWS seref_o_Page_069.jp2
109b61120019d71ad2b1f52f21a1ccf2
f65edab4c4181c50b66c0f69ba50810162719970
1733033 F20101219_AABRDA seref_o.pdf
108944bbaec29ea6d71c739a491080e7
32ca7a0a9aa5bf585ec39c91ebc8515bde24817d
2086 F20101219_AABRCM seref_o_Page_073.txt
e2db067bd1f6ae0cdb54482f937b67d0
2131a7d3d14d3866ea4a305d31c5ba0d416c587c
978893 F20101219_AABQXH seref_o_Page_104.jp2
6f21548ec2b2ac4067ac379431b459c8
392976662457a64af9e110addba32469112337b6
1378 F20101219_AABRBZ seref_o_Page_049.txt
8f46d7e1bf2bdb6a69df0a920ce4b846
63b26b95feca52dc66a75b1e55b1dc86ffb60be8
122819 F20101219_AABQWT seref_o_Page_070.jp2
f8e04c7e359e3794aba83969f2ff7809
ae2f365ab6cafa88749a5421a347c93d420aa770
1833 F20101219_AABRDB seref_o_Page_001thm.jpg
4afc6714958126f8ee29849dbf6a9220
754b9cda81fff7cfe939b912d6c82a110929d629
2369 F20101219_AABRCN seref_o_Page_076.txt
b06b28c4e25699efe5aa42b2c371c023
f837734abd38adee0052e928fddcd96401805017
906313 F20101219_AABQXI seref_o_Page_105.jp2
6cf73f9997a857795b67bf12115a510d
9af8511229219c57d8b934839d058fd6592a5211
F20101219_AABRDC seref_o_Page_002.QC.jpg
0d4661201288ba61d522c1877d91c185
09cd18fd80086061a931c2581001d82f8f33b794
515 F20101219_AABRCO seref_o_Page_077.txt
bd47a0f381b61a044249e76b4af61f50
0aec5d611a54d3485056b1332089ce6acd35315b
53950 F20101219_AABQXJ seref_o_Page_106.jp2
35d70aecccde67724a62099b86e7d472
be902cc48c2af41562aa83882e70235165022c14
1051973 F20101219_AABQWU seref_o_Page_072.jp2
740ac5c60cde620fcadf966d682a8ab5
582a4097a8f36d2fc2614021844ef328303afe24
540 F20101219_AABRDD seref_o_Page_002thm.jpg
9231052830267caf8305cb45bbce94ed
2fabc8bd816a30c683c67b4bd4f05887cee03ffc
551 F20101219_AABRCP seref_o_Page_083.txt
dd290f502a48933374875dbdec004520
012f9f39af1be6c3b4f317c237ca740fb7b02b35
123062 F20101219_AABQXK seref_o_Page_107.jp2
fa082b46368d8e61cfdac91212f3431d
42b43b81e8ff21aad5e92f3f5487c6fefc70468b
1051949 F20101219_AABQWV seref_o_Page_073.jp2
f5111964da807016051c304c4cebba41
e7e29b0c0f89431e996cc676c902e884ad77c043
567 F20101219_AABRDE seref_o_Page_003thm.jpg
de60dd714cc92656a55b7aa767bf440d
938e3221f24d65b99b34e1127220d4cb16fafa31
340 F20101219_AABRCQ seref_o_Page_084.txt
80cfdc3187133ec6b7ca5d3de07e314b
5f9bfff6c8e0ccb52471396bc6b75905e3cf015e
116355 F20101219_AABQXL seref_o_Page_110.jp2
174c8ac7aa4e21214c54f3935a51969a
00313ec477d64dadf083fdb298eec32b638823da
1051937 F20101219_AABQWW seref_o_Page_074.jp2
a0434b4e3b546c88afcd11349dae3b6d
887f3f7ccaa0a5335185fc80c1669ea7ef1f1a98
5524 F20101219_AABRDF seref_o_Page_004thm.jpg
cf05b44cd0c326dc79778e236b83e568
a0091888de1c225b38c00f6e2005bb7b3758e43e
2127 F20101219_AABRCR seref_o_Page_086.txt
2556841aa47fb21ad1c4a0d65bf279b1
7feaa4b7060b456ae58e6850d9dd44d13268bd15
130695 F20101219_AABQXM seref_o_Page_113.jp2
7f6e3ee282cf9d60f1557456005ea30f
654b6a5d828f8d8c6ac7520b2e4533bb8aab7168
743694 F20101219_AABQWX seref_o_Page_078.jp2
ae17da7fa273c694f93d999fc359603d
9b4555fb05e3d19c21975a5c77fed47378cb6457
15947 F20101219_AABRDG seref_o_Page_006.QC.jpg
31086966ed19e4b9784058fdf2f160bc
ff1ac730f2a240f39927af4f377beb245c47167b
25271604 F20101219_AABQYA seref_o_Page_026.tif
438120c79ce72b3a15fdbff6a5b0ae35
04cb44e0f44c4aa7aeaceb346eab09f93f296084
1714 F20101219_AABRCS seref_o_Page_090.txt
72e0b31325d5319f810da7c965e0dc65
40103e9beab467267354fd3b2e4bf9b980313cf3
126997 F20101219_AABQXN seref_o_Page_114.jp2
af4b7af64b1a83aa056a55eb6ca0eefa
b400c0986d851a7468b21af5a40132f06e311a53
497480 F20101219_AABQWY seref_o_Page_080.jp2
70f2bd881af6eabb510d59a0b07c5d4a
8fa0e7d127caca2f1cd3353298f250e38809f3ce
16318 F20101219_AABRDH seref_o_Page_007.QC.jpg
9c159762a4c906941ce4c03863a9f140
051107cb05a53404c3769308e09d80f0b22111e0
F20101219_AABQYB seref_o_Page_028.tif
e8cf6706e7cda54812c404a0adee779d
d63bfe53f936d1717b8c53e91702e9768ba0a9f1
2120 F20101219_AABRCT seref_o_Page_095.txt
3eb20128610282f382f2ebde8438fc33
8f3b4a069e9f93c8500925938b450383e42ca77c
125018 F20101219_AABQXO seref_o_Page_115.jp2
b7d3799b9f6ddab79b652f24341c96c7
6a081ddcd38bab36be1de65da2e126b1c16eeea4
775515 F20101219_AABQWZ seref_o_Page_081.jp2
6f3d2743a0db482eae259f29bbe85322
bd0205e373b9a588f411220efe817ff5734a0b49
7655 F20101219_AABRDI seref_o_Page_008thm.jpg
1cd888f50d75045e98683221b5c81e00
c8fd17ecdd6c35d261aa2b87d7925499c7c1aad3
F20101219_AABQYC seref_o_Page_029.tif
95edd4f15c4a91968cc6b4df4c6c8c1f
81b6ccc9bfa7e8752da51fc7e1747d672baff6c5
1803 F20101219_AABRCU seref_o_Page_097.txt
05080c98ebdda27943b5d317f5b1536e
c1a22dd4011faf3554cf501e4dde65e68155efc7
1053954 F20101219_AABQXP seref_o_Page_004.tif
abae7fc6c15f8b24d26dc8130b63a8a1
39ba73cd8564be3221bf3061987ed56423fcb32c
8419 F20101219_AABRDJ seref_o_Page_009.QC.jpg
995fc608b9234d807673c4eb5c3d4e93
f696930a4bad13e5f5fd3f88518dcaa108871d9e
F20101219_AABQYD seref_o_Page_033.tif
41b250b15b329c1c57f3cfb6f3e4ea81
06a07e368359b6564eea3197f9f150ac87a3cce5
2279 F20101219_AABRCV seref_o_Page_099.txt
4493217de85da384e508fa4e774f4967
33609e67c91e1d0c0a8629c179a31fd3f94d5e5e
F20101219_AABQXQ seref_o_Page_006.tif
c995c882810fd6c481d20de8cb4f8f31
c251f5b00533c5b34c146bf0d9d0ad27541b8ef9
2133 F20101219_AABRDK seref_o_Page_009thm.jpg
7aca26044745006f8f9b95caa28d0233
4fa8f37237cd1968de20b717ff88fa5757087d16
F20101219_AABQYE seref_o_Page_038.tif
47818521fe2855b8d1b96fe0c43be7f8
59f0329c4d8b42eb9dd464a07c7b7d672d8f1ee9
1317 F20101219_AABRCW seref_o_Page_103.txt
133702f48314905049b5daf4261b4a97
6aa33207ff802df89e8222075698593e0fdab89c
F20101219_AABQXR seref_o_Page_007.tif
4c8081c0594771d92691de9603c569a6
e7f1e7d935dd48c3c193296614446f79b84914be
33695 F20101219_AABREA seref_o_Page_026.QC.jpg
b58024140cc0fc8e4accf439ebefc784
5b770d55d38cf90f496e9428e0e85a28cba40e0b
34592 F20101219_AABRDL seref_o_Page_012.QC.jpg
a54822e67943970998f18d2d03080a05
9746784b79fe71ef8863e21ca90d28a65694faec
F20101219_AABQYF seref_o_Page_043.tif
ece94f036adff41bbed8c13ed7a9f6be
35235eabf644cc3526e7c6663d7c9bf715130255
947 F20101219_AABRCX seref_o_Page_106.txt
1b54e6b09de9e075bfbe73f9bf565cf8
19598b1822faac332ae408e930ab0343303b36b9
F20101219_AABQXS seref_o_Page_009.tif
55c0ca0b3b0532120852c93290954b41
894ef795f3aa27beecd467cf337b920dd0335556
36359 F20101219_AABRDM seref_o_Page_014.QC.jpg
5871c4a7c228cf8d09341add734aec08
70d873f64f1cc942e88601c1e5d48672c6afae2b
F20101219_AABQYG seref_o_Page_049.tif
c13f874f28611cd0450fbc8287502932
8d3501026633cd075f475269b838de9b7de274bd
2282 F20101219_AABRCY seref_o_Page_108.txt
4850328fe4cd8224cc54973c468db2a7
af76bd72b54a431d8b669b4668c96315ad9f0355
F20101219_AABQXT seref_o_Page_011.tif
e94d9657d414528a029b69b84054ea15
f6270b3022c4c9293a55ccd31222eaa81ae76a48
8181 F20101219_AABREB seref_o_Page_026thm.jpg
d72a30aa01e2a5f379fb77d147b5b7b0
a31b966ce924879e31246de9e80dfdceb8869203
8224 F20101219_AABRDN seref_o_Page_014thm.jpg
3d840d48d1b39527436d2d502ca442ee
6cd0c8bb3668ca6f9ad9f5eca1d34c0a4ae52d5b
F20101219_AABQYH seref_o_Page_050.tif
39843f5e68088b57408d334c9543030d
968e7a6a750b54c46c0737b8c56e04d70e1c440a
2570 F20101219_AABRCZ seref_o_Page_111.txt
563d7cf5d75f4a7a79193a6d3d7c6670
1d3ecf6bd80451907f44f595dad3c8161027bc00
F20101219_AABQXU seref_o_Page_013.tif
7a173b0e759e757c7569f0f30f75d927
a93ea59cf383a91ad2e941b0f91dac231ba4c2f9
6228 F20101219_AABREC seref_o_Page_027thm.jpg
81b116e5749bc8618db376229499e951
975824b536790c47548cbe26cc5f5156087c1459
8205 F20101219_AABRDO seref_o_Page_015thm.jpg
ad8b53069f7313367d2b856953ea3500
f945d4969bdfb0d6d830ed364360fc4b1bc61b94
F20101219_AABQYI seref_o_Page_052.tif
f58e9275bb88acb93af766f78b108ef2
05f4f10d609c0f8164074f47987569bb333fb5cb
28549 F20101219_AABRED seref_o_Page_030.QC.jpg
b231fe4130d829d39c937ac33d8d4f81
c162d8977e4cb4f6749b56200121e1022d3d73f0
36517 F20101219_AABRDP seref_o_Page_017.QC.jpg
4408f4f3e73e9814191129d414740d68
8b03f09743fa6a7934e6dd53878444f95caea975
F20101219_AABQYJ seref_o_Page_068.tif
b1f477dd005ed667ad4e2c83ad81b5bd
01d34a5334ad5d70a3b0640c4133bbb883e6ea70
F20101219_AABQXV seref_o_Page_014.tif
0ba5ac817ee285fdad4eb08ae930dfac
2732f6c6816a67fad57bd0f657164e80d644d871
33003 F20101219_AABREE seref_o_Page_031.QC.jpg
492f5e213573108e40ea58f157586205
271ec74781062ae34ace4efe1befbe0cc8d82e58
8360 F20101219_AABRDQ seref_o_Page_018thm.jpg
0399a01c0a68e1aec5cc2caad05bb955
4b75c08e70c9748330e97a09e4a75fa867ed5b2e
F20101219_AABQYK seref_o_Page_069.tif
1ad22a72d3c25244ac95146c3dcc0825
2cef7ba6befcad1394dcd4dea4f839a9b3b3e89c
F20101219_AABQXW seref_o_Page_015.tif
dccce424c268e22e8ae1802dab39bca2
0812faf40024ed9c8d7121a3753ea989d0d861f6
F20101219_AABQBD seref_o_Page_030.tif
485fa0c90e80c14eabb5c78af59f7892
3be8d380b927f8942061ed250d50cb53b60cab04
5102 F20101219_AABREF seref_o_Page_033thm.jpg
dd43047d13664a61a037d42e7f4eb2cd
00ce33428897076bd72e486167d681fb83513037
7202 F20101219_AABRDR seref_o_Page_019thm.jpg
00f53392d28918b1f3a6befd9129a985
3f644d60ee17f9ffb0a3331c086f05bdeb34f94c
F20101219_AABQYL seref_o_Page_072.tif
004ff9e066a24ce190c09abae59ca8aa
a867de55e79f9020739b1d1ca3e44bdf701f9dcf
F20101219_AABQXX seref_o_Page_016.tif
a05cffdf56fdf662f5316c93a9e6099e
0e32d4f4c3c3df08e1e3da904c8f4ba7e7623d39
F20101219_AABQBE seref_o_Page_099.tif
e03989b331483e6003c0adc18ec44a02
e8fdcf9edb161e6810c5bc31c52ad944769b7cd4
20316 F20101219_AABREG seref_o_Page_034.QC.jpg
1f68a36f4d90764023f9cbfa62a3aa4b
6bca0cb9f009efe3fd36c026301f3321b3a3c765
F20101219_AABQZA seref_o_Page_098.tif
35d4de6ea268567f4a250f2b0133f5b3
955e57c27a35ee9544e6e92e534a4fae98c02c1e
29848 F20101219_AABRDS seref_o_Page_020.QC.jpg
31c3825ea38e865b1bc84821c0d17c18
57e52fbd13bc5f0cb4d478adaf5331caf753b978
F20101219_AABQYM seref_o_Page_073.tif
64084998394d4fe924a3c3800c13b769
f34bb79d484669713b93e636d28e9a9b360a5ba0
F20101219_AABQXY seref_o_Page_021.tif
43307774194982c0eb343e164a04d28c
646ac07cffc39cb6d83ead0fed39c3de3e2c4c68
29068 F20101219_AABQBF seref_o_Page_104.QC.jpg
fcb068475b25d8e7556c32915f615c89
98fe10f9577eea763106025b3b6e06718822f16d
19710 F20101219_AABREH seref_o_Page_035.QC.jpg
a57149e57e34816c61a7c99e509784ff
b9a2ed72a882c2c6d31670132cbc91d744bd1b76
F20101219_AABQZB seref_o_Page_102.tif
76ac46e3d7c2bdb66e2c967e87634eff
7f107d96dd9dd5f4ae80c9bda4fd3ae797b2a243
7187 F20101219_AABRDT seref_o_Page_020thm.jpg
4cb24255c8e9fcb10bfb6a780901b517
9f6ed8537d20b36f109795fa3975143296af0bee
F20101219_AABQYN seref_o_Page_074.tif
832fd78426ef0cbd87292194f7ea15cc
0c4a7856fb010ee587f0bdd3bfab4a77a3fbf4db
F20101219_AABQXZ seref_o_Page_025.tif
9aecbeb12e5cda6e255beddeebb08831
c8e8dfeef376b7ce79872f28703530bad8dd0561
16508 F20101219_AABQBG seref_o_Page_116.pro
680f0f093bf680c35f1458e7e8b3e8fc
21aa2fe8e5002c790a6caf46e174856beb55f77e
5465 F20101219_AABREI seref_o_Page_035thm.jpg
02f6f076cc4a5ec5d734975469fed685
237ced36e6ba2aa384a0c6e5c7a0f4fa25f76910
F20101219_AABQZC seref_o_Page_103.tif
b1f522239efd981679ec2e88fb18238f
3bda5757146d93d6862227501311bad780fe7086
14564 F20101219_AABRDU seref_o_Page_021.QC.jpg
a2516bf3f35d86eeaddbeb0b140fbfec
6700a493101884bae82b56131f1aea74d5f7d11a
F20101219_AABQYO seref_o_Page_077.tif
19f1c54821e8e572a57a0d6c562adab9
755e3480fb3a490892e89a8cc478009aa2680752
F20101219_AABQBH seref_o_Page_108.tif
e15fc91fab82ee66dbe8c20dd9fe4945
4f7641e2d431b8c22bfc6b4111c496dfaee205a3
5470 F20101219_AABREJ seref_o_Page_038thm.jpg
147dd832f19ffc15782e56df6dd3f22e
a34b5123a3820ddca5a039b6ae27e143113aa43e
F20101219_AABQZD seref_o_Page_105.tif
b9397299c659cc9106e1dcd652902a77
a226091e383b2eb60793f274dcedb13b28f931c1
4458 F20101219_AABRDV seref_o_Page_021thm.jpg
625e0c3cca55f2c4d6fa212e20b43c5c
96cbe90443f317e350b0182af7a49b9656171a52
F20101219_AABQYP seref_o_Page_078.tif
91ab5647949a7c35828bdc4d7cb5342f
509b275b20bb901e1e35607b68393112111ffa86
37212 F20101219_AABQBI seref_o_Page_040.pro
f9d53f2eca214da38349da3de9db8797
fe272fed872e54d0faeffcce0ce9144c807941c2
25262 F20101219_AABREK seref_o_Page_039.QC.jpg
ea2c7e05ee604a4779fa839e924ecedb
073dd1ebd691cebac43b62b7538a8c0ad5cd6c4b
F20101219_AABQZE seref_o_Page_109.tif
95c8a15fc55f09da058ef44f3140ae79
cd72c027ad60f25aca866ca7d3ffbfa54273e563
6690 F20101219_AABRDW seref_o_Page_022thm.jpg
2905a9caa5e983be5ba9d9b6c8aaf55b
1d5754933d33d36874f2a77339413736256b5e45
F20101219_AABQYQ seref_o_Page_081.tif
e4a56dd1f72651c2dede3d7ed4fa16c9
efaf373edff5d5a3e955122cc2db47b438bb03de
113162 F20101219_AABQBJ seref_o_Page_114.jpg
12ef1aa13de3173486bc83ef08841209
e4e5720a8db031ea6498292e5a5a60dbf97f3ca3
6161 F20101219_AABREL seref_o_Page_039thm.jpg
12a6fa3e8b59ac5cdd8e47404133b3a7
0488ef89e10e021172d3d7a018f39beff6b41630
F20101219_AABQZF seref_o_Page_110.tif
0924c524ec02099fdd8398200068a0bd
57f3e30176992f9aa8760b6610c68cec8585b0de
21636 F20101219_AABRDX seref_o_Page_023.QC.jpg
a15337fb7e77c4c9a4e1237a65ebbf9b
d67e5a17cc007a9878cd15149a4e68c73a3617eb
F20101219_AABQYR seref_o_Page_082.tif
97218c023b059423afbd93c7e99d5007
a37e68e89f77310919f7097e337e9a62ed63917c
8049 F20101219_AABRFA seref_o_Page_063.QC.jpg
e3b2d9c4e0c963661aa2be5a0e72b6a3
0c0975f713cf9a7b632ef80aeff8b472544819f3
1114 F20101219_AABQBK seref_o_Page_061.txt
e359ea54f2f2b1fe8930b2d096e432fc
c2720b53782dfe67bcfcb29cbc9a7026c14ef028
34897 F20101219_AABREM seref_o_Page_044.QC.jpg
29747fecbea47e994edf67f015b1a180
d481d4d136762581334b8db62bd6608966736896
F20101219_AABQZG seref_o_Page_111.tif
9eaa3eb51469b5cb6967207200cef0dc
4dc824898896f0ef339fd283f4144689799557bb
6433 F20101219_AABRDY seref_o_Page_023thm.jpg
968520b97b09b9c13e8f2ba604bb9553
b9e31a1b813ea5ee08125f6b1fa18ddb35f3b305
F20101219_AABQYS seref_o_Page_083.tif
a8f3ffaaae9191e44814d21661cbf7a3
193328e68a753a397eaebd4d92e7eac44e43fe98
33987 F20101219_AABRFB seref_o_Page_064.QC.jpg
f09bd9e453ddfb69b666d9993a0cc681
6dd711caa47508f4c654c7903152c45a5c30b026
102394 F20101219_AABQBL seref_o_Page_069.jpg
a816e19d56aaea2c5b1d5f9bbb827585
0cab1d15bbecd823025eb24b5e19c55754b2bd63
8064 F20101219_AABREN seref_o_Page_046thm.jpg
21601a395e4f92d983cc4f2c7be7aff5
4e512b83da5bede086874deb4ed5c26e6e2b9767
56823 F20101219_AABQZH seref_o_Page_005.pro
1282fbcdfaa69ba8bae6c32d18c754f7
7318387e9a4c0f3010e9d420f7591f25e45fe088
33172 F20101219_AABRDZ seref_o_Page_024.QC.jpg
727a9396c0a4ed629fa328744763cc6f
8814e52df7ccf791863108cbede251535316baac
F20101219_AABQYT seref_o_Page_084.tif
f15a9b3a3467e3cfa11bae10e7dc065f
67b4244c1c9604520e6b60e0589b7684f894a4b6
2574 F20101219_AABQBM seref_o_Page_112.txt
41f61dc04f2d405ae0b28004d851911c
5eabd0fc95cb5f95b33d71cca655bd5e53c2c9d0
15570 F20101219_AABREO seref_o_Page_050.QC.jpg
62c2219a3e85de494d9686ae183c384d
3c8eed8a80c7ba1eba20ef445b77e78ac56d7f06
25737 F20101219_AABQZI seref_o_Page_006.pro
8803f71178a43a173f2f71abce234929
8f4d89999a62dbfe4b39d08892ced09e10631d6d
F20101219_AABQYU seref_o_Page_085.tif
f0a766168322fbca87a1436b4618102d
3f5513789b10792bc6eb7aa30fe3619a34109537
22010 F20101219_AABQCA seref_o_Page_088.QC.jpg
fe080f780e820ff9985b8a6fc8d71044
1d2beb85fe97f9a94cca769f370bac4561a07db3
8046 F20101219_AABRFC seref_o_Page_064thm.jpg
db219f2e813e082cf2f821fed9206f6c
54858b87c38986ed82ef3a963ba71f4379827b24
477 F20101219_AABQBN seref_o_Page_081.txt
8080266852b5cf4de126d5165ce2d03d
cb5217950dc2c3076a9ebc0e91aeee58941511c7
4484 F20101219_AABREP seref_o_Page_050thm.jpg
c8894383e0959d3139b2690eed8efc29
79a7273702062d5919b9ff85398d9cb8f46be6a7
21385 F20101219_AABQZJ seref_o_Page_007.pro
77dacf4c51977bf4a4365559a408c278
55400ce025ab43c88d81bd1685bebd53d88a9487
F20101219_AABQYV seref_o_Page_086.tif
7cabb8bff022024324ab84ef3dad2616
9d94a236f78fec1bfda9ed254614156723407b5e
F20101219_AABQCB seref_o_Page_091.tif
75808382eb66292959a774b1ee20b0bd
dbc2e23a788115faaada1da4b1fac73d8820152c
31124 F20101219_AABRFD seref_o_Page_065.QC.jpg
1884ee835e49fecb3879677fe964b058
c4ef754a8f356cdcd5304a2d11a4c63c135d9bfb
F20101219_AABQBO seref_o_Page_057.tif
30db0f29211beb15647131ff044c8ca2
6895f6b35bf9659e203a8f838a468ce3669b2df2
5591 F20101219_AABREQ seref_o_Page_052thm.jpg
120dfb229704382d650287af875a841a
748c9b0abf76516af338118324df25c85ceccb15
63835 F20101219_AABQZK seref_o_Page_013.pro
48e2eb6ef65501fb73ed8155f80415bd
f6a6f93fca1273d07160d882013226d0856dde9d
1717 F20101219_AABQCC seref_o_Page_100.txt
4186edd3707aa6f4667caf5b589225ff
8c251053c0439dd9ab2ea27fade38521b275856e
6038 F20101219_AABRFE seref_o_Page_066thm.jpg
9bc5d0ecc165a26ccb48da22ebb8e950
1043d78faeb2426283f86177aec4f7379d5c9c79
7929 F20101219_AABQBP seref_o_Page_095thm.jpg
1904a0b5ea50772b4001314051050d70
a19b50002191b103f8c95fd150f01b33f43f4ecc
7208 F20101219_AABRER seref_o_Page_053thm.jpg
d66f9fe081aef13972e4159c35328f98
455314a5ea059f973b337f0ea2e9f78c6e67b626
56330 F20101219_AABQZL seref_o_Page_016.pro
7a5df059bd1636df72c04ba0d1d642f9
9a91074a28d7924b32ef752f3907b997df1d2010
F20101219_AABQYW seref_o_Page_088.tif
0d11097580e854088ca3baf236717645
0aa2dcbadf3adc7d2d68d7eec19fbf14965f21fc
7487 F20101219_AABQCD seref_o_Page_047thm.jpg
950e1b2c9508fbf9a9d11c6f2b7e2d0e
4b25283c356b78838615d4f1215e6df3668a4c01
8151 F20101219_AABRFF seref_o_Page_068thm.jpg
82197f766e6b45947d2ea14c98090261
dafe2b7ee40160cc02d03e699b556fd274435f26
6877 F20101219_AABRES seref_o_Page_054thm.jpg
8adcc9d4ca724ca305c8c200f6b36342
089e8ebbedb558e5f5ccf16e0b98045b1c4e29ea
60552 F20101219_AABQZM seref_o_Page_017.pro
85dea2019419525b074cf50f331c8cc4
c38bcd8db68b649466107081a6bcef3eac759a89
F20101219_AABQYX seref_o_Page_089.tif
0d3f45cc7e963d8baf700a2f5f129b98
6230c890ac9558098b54b9d9ef875abe9923464b
7887 F20101219_AABQCE seref_o_Page_016thm.jpg
398e44db80c81684bf2f0d83c6455bc3
61fe16dd912bec70cfdd76820bceb98ad172548b
36319 F20101219_AABRFG seref_o_Page_070.QC.jpg
7330e61ff39570af10d5c72b41d6a143
e68726a9c187557fc8895c83ddfd0c3983d45d98
20114 F20101219_AABQBQ seref_o_Page_048.QC.jpg
eaa824a5ab2d3207f5a4764af93b335b
2ce829b4e54dd720016cbc384e1fc197e5a05801
26864 F20101219_AABRET seref_o_Page_056.QC.jpg
948f9b01f896cf35c8459bebcd0d7e26
40804f0ca82268d81092f54fac011b119f9ee8e8
54612 F20101219_AABQZN seref_o_Page_019.pro
e3056ddae8cb604f7fcef39187edd903
a5fb55abead3cfb5104ecc8c0ff8039722eabc02
F20101219_AABQYY seref_o_Page_095.tif
50350866869b1c4bd2b2b7fe89e9d2a8
5c2fed72760f526ca1335bf7247b2e6b91fbb6dc
1051985 F20101219_AABQCF seref_o_Page_098.jp2
e0a5d43ee256c3a07939548fd1317b6c
44da7fd1acb0deea7f2a749198f9fc8972875d81
33333 F20101219_AABRFH seref_o_Page_071.QC.jpg
e620e33dcef62bc5e0005462bcf7051c
580463a91e59f35339dbabc54e263cf715c06bcc
4924 F20101219_AABQBR seref_o_Page_063.pro
4b20a0cbf4be50c78918115bc0aa9403
8bbef15ff367ed44f1f80fdf05acc4c1c7b02e46
19916 F20101219_AABREU seref_o_Page_057.QC.jpg
aa091f3c50c0384d55bffbc7a4c111d6
481c561a2f413f05a51c71161e6e61edeb7fdeb0
38697 F20101219_AABQZO seref_o_Page_027.pro
9acd2f55057f3cd71493d018e7e75903
7e101411a65e22717beca44f254adae82c120e17
F20101219_AABQYZ seref_o_Page_096.tif
feb19c2b6d502b41f09228ad585ae089
69d58fb216ad1b70dac85ecaf893598bf52e2b82
6630 F20101219_AABQCG seref_o_Page_055thm.jpg
1ee9e98353bcea838a41fadfc83800c0
60ac087dec3fc3dad9b40343086880bbc14a08be
34625 F20101219_AABRFI seref_o_Page_072.QC.jpg
51c1fb6fff7ed11d857e6591a95215f9
35bbe31ba0e89d5a554bb3ba1e3776df8af7fed5
F20101219_AABQBS seref_o_Page_037.tif
04dd9c455f3bd81834cbfa24300e04be
77c16c336a5ae545adb467cd5b4104934d1a098b
31052 F20101219_AABRFJ seref_o_Page_073.QC.jpg
dbaed61456b5013d3142a78b9d5d5dfe
8acf7d1fbfa3c629988353139791ca054d5e451f
5493 F20101219_AABREV seref_o_Page_057thm.jpg
2ff85dd4fa7916644768df4c27a9a813
f31e48e83b2b2ade7226ae67da4122ef386357de
35776 F20101219_AABQZP seref_o_Page_029.pro
21a322bd348a3e3771ec501ff0f79ea1
c4c3594699bdb56eb5e475c61d55cb042217eea8
73693 F20101219_AABQCH seref_o_Page_027.jpg
90e0cafe4bd6717704275fe5aa31dd6b
24f1addc8209e049caa64859e43019b471f85d67
7996 F20101219_AABQBT seref_o_Page_024thm.jpg
628d7087e91bc6231634ed4083bf893f
3c38e8590434f20f8fa941c345c6365b02e987f6
7488 F20101219_AABRFK seref_o_Page_073thm.jpg
f201f92209d2681438917badcdd01837
b6bb1a301be23bba3eb2b1c160a9526c81602172
16733 F20101219_AABREW seref_o_Page_059.QC.jpg
27e1d7088771873ed7b0f7d7a739a8db
13d33e4051ea9c5d0d84a576b63c179ac0f14737
43956 F20101219_AABQZQ seref_o_Page_030.pro
e2ad0254eb717d6cf07fa79a19c4dcd2
abf36761835ffeaea23a5679eae9c75d7b889d4f
2331 F20101219_AABQCI seref_o_Page_044.txt
fd1412e8d43572a5ed222e98cac9a249
5626e146c5da1e0e1cbab9e5b14dc120e7303437
F20101219_AABQBU seref_o_Page_008.tif
d16013740ffb24dd2a91aa3b337b5773
25b9fa1b511502d294f7d94801d8027c702f1777
33656 F20101219_AABRGA seref_o_Page_092.QC.jpg
6ba641efbb8d7c8051627726593b89a0
8df25d6ac5f2380b292cf912bcf6f770880eb8b5
8226 F20101219_AABRFL seref_o_Page_074thm.jpg
3c69e1be81230b1fcb0e674646ef457b
b2689efe4966d4c7ae9ab73e6f6c09788cb19271
17276 F20101219_AABREX seref_o_Page_061.QC.jpg
1e7e4f0ef20604df821d12ee4712c15d
c6a2f0bbdabb5d1d97ed2646c3d5d0bc0abbdc08
53905 F20101219_AABQZR seref_o_Page_031.pro
3c5549fa4f3b5d76417a4a08e7b3e5aa
b806b4459490f42732db58e39ad305793812cf5b
8914 F20101219_AABQCJ seref_o_Page_079.pro
602c39784c318ec31361419c980a74ab
c400efa46dfaf1b81c631ee9f3c1786dc9c93f73
34687 F20101219_AABQBV seref_o_Page_108.QC.jpg
51d309857dff502762a36441e76af391
6af6500d1f882e4ebaca97fa2742d74315a4e078
21960 F20101219_AABRGB seref_o_Page_093.QC.jpg
c76a979da06aeb1499505df041c9f834
0bec1a98741ef0d0acb2463cdb122c07f8bdda43
35678 F20101219_AABRFM seref_o_Page_075.QC.jpg
63d48250e5047a614267e25a6aabf2c8
0034bf1e8bbd44c24175ff11227e628441afe94e
25506 F20101219_AABREY seref_o_Page_062.QC.jpg
2a35a3a2b68e72b9db863202d664baa6
ec01087ee0ecc53a955f42772091783d8724af70
24880 F20101219_AABQZS seref_o_Page_033.pro
46da661e9196e36d06953c61ac539194
d01fd10ba1e7e246e48d6e68a3bf8e842bfb24e4
118369 F20101219_AABQCK seref_o_Page_108.jp2
a6f4e9d0165ce289852cfa75dd823f9b
0031cb1b2e6bd8f983a293b1d7dd616c40760873
60043 F20101219_AABQBW seref_o_Page_076.pro
7558e3b61e6b0598999c8001fd7bad75
e09694bc31e1386d08cc288ccea507ce5b49b951
32959 F20101219_AABRGC seref_o_Page_094.QC.jpg
9a002f0f99de29d5f221a07ac5317240
3aede9e51676055d07db975ca76125c3f76e0aa0
5661 F20101219_AABRFN seref_o_Page_077thm.jpg
8bbf83c69e3286f647e6369bc881394f
a05af76cac5e277e90242517d25706537f984aff
6306 F20101219_AABREZ seref_o_Page_062thm.jpg
7395d393f91e0a968b5a3a3680f51681
3ffdeee3ce0bbd08c32396c4ea0e9f1eb5c3ce10
31752 F20101219_AABQZT seref_o_Page_034.pro
cefee1adacb5a08172ffe1cd8cf4cf4c
7ff97e87e4a649492363c37ca768145960565cec
F20101219_AABQDA seref_o_Page_112.tif
30f99f024e06326fd8cd6aaf7f2a41ee
47423359bd20c31f0787f17a7e302b1dd67b9385
F20101219_AABQCL seref_o_Page_002.tif
6251dc2f214e22dd175cebe78451c144
8420d9f0a27a9f4e55b96806d9864a456cc3ee45
1957 F20101219_AABQBX seref_o_Page_091.txt
7bff19a2019d16ef625ce0bd106d338b
58bcca1639a5c5d37deab42870205024d1d9a722
5888 F20101219_AABRFO seref_o_Page_079thm.jpg
5b296c1c35e673e8fadbc619d5c347ff
a66886aa8b82ceb53cedb3e709d73ef97534be23
29261 F20101219_AABQZU seref_o_Page_037.pro
40474c37cd4d4207177fa5c71a721527
5bd263fe24b81ab5bc16d88ae16d62dedb5b4de3
39510 F20101219_AABQCM seref_o_Page_022.pro
ea05efd4bb97be058d614bab0534cbce
217fbea6664c9aa0f6820e4abea6c1da8e71c7a2
25055 F20101219_AABQBY seref_o_Page_097.QC.jpg
4f32c4850b8252cc6315e61c4e5f2576
4e45defe236b3e200ad2a9753fbad72a6534da29
7972 F20101219_AABRGD seref_o_Page_096thm.jpg
4ba37dcab34b6b9edc3a19f4bc37926c
21995b8cdb2b4aeba144b1b74631de22bb9d0911
4411 F20101219_AABRFP seref_o_Page_080thm.jpg
7fca19ff04292d5886a2d820fa26223d
12cbd92a4bb917814e4d5574ca637180472d30e9
24173 F20101219_AABQZV seref_o_Page_041.pro
5e0f25bbddb46aebd45da67d152a3938
1eed94695155ae04f3f12d6c9b258ab29967e160
1051875 F20101219_AABQDB seref_o_Page_016.jp2
1503a4a1920f898b890490ed05386f0e
d9e18bf2f63aab7435895fc2c39fb7195fa8f8a3
32290 F20101219_AABQCN seref_o_Page_019.QC.jpg
a451139f9faa6c95f3bdd0b328d8ac29
35576dace620aef6337f6ef779c3564de97ecfd0
47336 F20101219_AABQBZ seref_o_Page_053.pro
cd9ac551149818e607f089b706fec5d9
a363cddc54aff6c7db3045eac8a71b6b723f041f
8139 F20101219_AABRGE seref_o_Page_099thm.jpg
f56c2a5b65697267a104adec4cd4aed9
47655240cc2e0d468a25c79d2b03d8ebbcee35d8
4681 F20101219_AABRFQ seref_o_Page_082thm.jpg
8e9a5bb16ddb37d9c63e5f387470eea7
16d5345e3f3a760692d388a4c347e3d06bd39132
24166 F20101219_AABQZW seref_o_Page_045.pro
6cad88163cd414444e0731df37c13e6a
f952c560f2ad240436b1f8df46640fe7afa57572
497347 F20101219_AABQDC seref_o_Page_058.jp2
c8ed0b6906a618fe1b434969f308c0e8
24877d77ef0b2857fc24e093cc64d13c08164c7c
5762 F20101219_AABQCO seref_o_Page_037thm.jpg
276ad4b7243910f7ee186cfde72d70f9
88b83a3917b4106c3f4e59c582c2ebefc3b99d21
6773 F20101219_AABRGF seref_o_Page_100thm.jpg
0cf95ecf03d74e041a40c47f760fcec7
168e4eac1d97b3258ab3443f6805768af77f77d4
6242 F20101219_AABRFR seref_o_Page_083thm.jpg
2235a446e798e0116e5626048d93476b
5917b6737f168762cd229b97eb5cdaa20dc7336c
5558 F20101219_AABQDD seref_o_Page_093thm.jpg
ac88eaf89eacaf47b9dc58a4ad41d17c
fca931405fe2bff12a6deb789c4db53ac84518ac
58556 F20101219_AABQCP seref_o_Page_060.jp2
814d66ba42dcd170a13717251f375e55
8551ff9c1d7a82758210937d6519224a0c8206e6
16261 F20101219_AABRGG seref_o_Page_106.QC.jpg
bc64bb0db1b6d106949ea919e78cbdf1
cb68b215bd8d2958a59fc624acc9b48ce88bf5cf
3117 F20101219_AABRFS seref_o_Page_084thm.jpg
cb4bc32621031fba64cedefd6d64a478
7a7019f976a4a82f2459260d5ae067f814a5cd5c
59824 F20101219_AABQZX seref_o_Page_046.pro
71aaa8850ff1742761a6bbc4679d86cb
99286fc010c8e79542e07b796f7eee1c3cd80c54
57319 F20101219_AABQDE seref_o_Page_078.jpg
e1f9be6267ce116b3441230910b2e7da
130eec2579e94e8644b402be2882e24cd2327cc5
5375 F20101219_AABQCQ seref_o_Page_002.jp2
a5d694fef6181484eb1c5ca6e9e09dfb
9ea79b4257c5b01c3cf0de890340c10ddb7b35ab
35178 F20101219_AABRGH seref_o_Page_107.QC.jpg
44a676a0c08cfbda9c67d04195b47543
36e6dddc17b5206dada56b86ca9e52f5b24cdaaa
F20101219_AABRFT seref_o_Page_085thm.jpg
24d9c18b624cb3161bc17164b996f35e
d8584304ab6216670f4bddaa21d496d007e2cea8
20989 F20101219_AABQZY seref_o_Page_048.pro
a9bef591332089a698c0d061dc62a25a
c7be70e92f8131c38eb902251fa84bd40fdc4a48
69962 F20101219_AABQDF seref_o_Page_023.jpg
807daaf837e7a324ec9d8812727dcda7
3db1ff3516ac6e65b986b41d826ba5dde60ec876
700163 F20101219_AABQCR seref_o_Page_035.jp2
4679a78f61228694a8dedee4f4609323
bd11af3a4cc446ffdecf47b867513e4c765d6906
8015 F20101219_AABRGI seref_o_Page_111thm.jpg
8314947601de722a9381670aad51184a
2a4eb64a41f1b4c4c4842d66823aea8db40ce6d8
7326 F20101219_AABRFU seref_o_Page_086thm.jpg
ca9f4b6190825de4d34836becc230ff8
0e74aede3060f0fb310f43ed6dae36424ff003a7
22871 F20101219_AABQZZ seref_o_Page_049.pro
9cae9208cd101b75c3afb99a31ce31ec
db06748b337aaa85246e3bd876ca817146fbd785
6052 F20101219_AABQDG seref_o_Page_105thm.jpg
6d79c67f77194bc94defb0eb259a243e
ca1540be5ce054b6d6ae9766b8da2a3e0dbf9448
1663 F20101219_AABQCS seref_o_Page_101.txt
8f46c26939fe53e7b625d708e792336e
b92a160ed8bd9b3d8ba742a0186ab1931c8ba3d1
33374 F20101219_AABRGJ seref_o_Page_113.QC.jpg
fc40fc22850f3b3a0324a12b10c4a5c2
5f1f7a616dc04045f73df912f5c9f1811403c62c
25906 F20101219_AABRFV seref_o_Page_087.QC.jpg
9620cd9492872d5cc05fb8da0384a693
ddc3ff4824f1e7a28cc25df9d3fb78c2bb229021
55540 F20101219_AABQDH seref_o_Page_096.pro
1031fd49d0517bafb17d67c5b71bb6a5
9248c12c4722dad642aca6bf30c0dd3790a421ea
7500 F20101219_AABQCT seref_o_Page_069thm.jpg
660d5cacbb32d44f752418e728e3bf74
716df1a528844da76face9a7a8bc3f2048a86ea9
7955 F20101219_AABRGK seref_o_Page_113thm.jpg
672759482da4f9d77e2ec98e6c296aa8
c3133fc921c1bfe9d014cd6fc687e09482a1f5dc
23430 F20101219_AABRFW seref_o_Page_089.QC.jpg
cff1bed0281d1c3f425af19e0092e05c
36a95a7fd0a52b30f1286b055ff366fc49e94bf6
6689 F20101219_AABQDI seref_o_Page_005thm.jpg
ee8fcdc49ec08cade153fbdbd97b099b
a4d1710e4bf0d0c52e43989e1a47193e5cd155c4
F20101219_AABQCU seref_o_Page_063.tif
63349ced207a0a9b4bf74c5caf78c601
472e4c2b1f93b505514ae3f965b186ed6d9212c1
33818 F20101219_AABRGL seref_o_Page_114.QC.jpg
c72a3bece610139864f678ef7d77b305
05d9534d86d8fac56e932f8224a1afb88c09b506
22904 F20101219_AABRFX seref_o_Page_090.QC.jpg
5259ddd86b7ab198149f1a555a711ad7
0c4fa98af736986945b1dff0fa638b534b06910c
1802 F20101219_AABQDJ seref_o_Page_056.txt
47b40044900cbdf218e06585fe492b11
3809d460120bbc6d1b333b003d5b638a68a1df43
109031 F20101219_AABQCV seref_o_Page_085.jpg
514feb97500be6faf35a737d897f5e79
c95e54c995717b19f72ecd96f4011bb1fd6fe067
2644 F20101219_AABRGM seref_o_Page_116thm.jpg
b38d7989c6b9faaa0bfb2244ea0bc7d3
d3fcacdf64613092e7b7e097d4a1bb023d2bb1a0
6064 F20101219_AABRFY seref_o_Page_090thm.jpg
685409654022e11c1d7a3bfd0e82da28
7eb4524bfd8b92f08353dd3b01a4c89a45745b05
72864 F20101219_AABQDK seref_o_Page_034.jp2
665578b6bf7e8ba63b57efae8d23dbe1
3d017e37e66706d27bab962dd081643a9fea1e98
24263 F20101219_AABQCW seref_o_Page_029.QC.jpg
d9b20359c4a8fa67b266107becd6a5c7
37b83f4ce75fcdd0ee922d2447c4caf0beddca17
133961 F20101219_AABRGN UFE0017560_00001.mets FULL
6e72e1a15ede44de2a4cef00973b8eb6
e6b8e412ae87dffb47423d3b505431447d6cf0c1
25615 F20101219_AABRFZ seref_o_Page_091.QC.jpg
5c16001f6449a640052d832902ff07cc
42c289cc70b87b891ce3b484052d51fb1996b04e
408418 F20101219_AABQDL seref_o_Page_063.jp2
2111d5adf282b80d1b1cab30d918f7ea
6a7c08fefd7cafc42d4218f6a24bc6c08894ad7e
F20101219_AABQCX seref_o_Page_022.tif
3e372e2a2789ed2af7f17f8151f00a6b
f23019b882291f49d41bcd13650221229be27c62
20855 F20101219_AABQEA seref_o_Page_083.QC.jpg
5aca227f649d12410688ddeaa3fd192e
f6c7fb7263ca4fa18bb185174f30b3c3b53c8ac0
6922 F20101219_AABQDM seref_o_Page_030thm.jpg
2c5b034757c55bf168a20621e8b69413
90fbf660417e8b40d01491177e5b36e7d9af01e8
8149 F20101219_AABQCY seref_o_Page_075thm.jpg
3193190da01003e07c5b86f0337658a3
49326f8c74a0348718e9874940d6df9ca74e918b
95744 F20101219_AABQEB seref_o_Page_053.jpg
b7a893f69bfd3768ed690b3a6ebe655f
db301e15474df6d1295bfef233445374cb85cebb
58946 F20101219_AABQDN seref_o_Page_098.pro
1fe58cda493ccfeaba3ea13e798c51d2
7ccdc19176b689ec085dc2c1b02aba5698f61094
25320 F20101219_AABQCZ seref_o_Page_055.QC.jpg
8ee344360845d990803c97e42513f33d
a34ef4447f3e108e2258f91e112895f8adc5ba30
7710 F20101219_AABQDO seref_o_Page_071thm.jpg
37f4ac6b1019dc5eb92e2780312cb0cd
5206fc7d41393b9fba56b6046ac8c8db3ee7c469
12673 F20101219_AABQEC seref_o_Page_080.QC.jpg
b325c0aae409a193b3f1e423c4cf45dc
c18992993ce93920c3aafbc612eb142d1c712b02
1042 F20101219_AABQDP seref_o_Page_082.txt
25118dc7bca127cf1b189b9c62c6baaa
67075b5ecab5238a0314b5f6172076a2dca0ba4d
49508 F20101219_AABQED seref_o_Page_069.pro
1a6a22833b980d4d1915ac797a95886e
95601925cb492cc5a3314c783e9bee140bbb4c3b
1051982 F20101219_AABQDQ seref_o_Page_026.jp2
1760c1b25eb9efaa5f2b165a137bc373
e5011d002c9cb2ce65f4f325821b0af74b33d206
669 F20101219_AABQEE seref_o_Page_021.txt
f8137fd04cfc0e725733da67aacc45a0
9dc603e3a4c8e8214d9eb77a001ff480982361eb
F20101219_AABQDR seref_o_Page_044.tif
f8c74b6d23e93ba24dc22bc62d846f81
2dca49b0474018dd20670ff757789a0572da3ed9
1051920 F20101219_AABQEF seref_o_Page_102.jp2
8f50d5121dcfcdef019397d46dd510f5
dc73cec363d91de47559b6a8f6cebc2e9c6f5a5d
8329 F20101219_AABQDS seref_o_Page_067thm.jpg
f2113ef13ce9b549875f1253bf645d34
225d6727e8c79ecad447034e5222abae878463cb
24043 F20101219_AABQEG seref_o_Page_027.QC.jpg
2ee67378004979525da42a7a9e05f982
05858877ae7daf4f1d5322ff704b9a68875957a8
59522 F20101219_AABQDT seref_o_Page_115.pro
a95b8f7cd89205e5d0bf26d9ac5b4e07
296c0981b8c378ed17856382ba47d4e1f6981f7d
F20101219_AABQEH seref_o_Page_005.tif
1d4d63f0fc2b54e92b444be06d993b0e
c5262db9b403d42d6a0e940cb9ada7096568943c
F20101219_AABQDU seref_o_Page_045.tif
0c5c69dd390e0f2fef2f67f33802aa32
7032d202884d19f87e6ac2ae9d8f0f3eb42d140d
8183 F20101219_AABQEI seref_o_Page_103thm.jpg
23d686bb644af0e3255806c3d43d1fde
e2dd30875e4528e1b7c833986417af3f030ae037
1051927 F20101219_AABQDV seref_o_Page_064.jp2
0932725090cbccb41b968a6fdb692dcc
76a56411bb7c83ff26fd53b6c774b5d04abdac2a
43491 F20101219_AABQEJ seref_o_Page_082.jpg
75f44962f09138e9de3ed5912451b77d
53cd6c9c6ab0bfd8d82e097ad63407df87585181
7988 F20101219_AABQDW seref_o_Page_012thm.jpg
f857b591ad6e33c496b27afeeec1439d
20eab2aee87201d8d5cbab0ec569d8978dde2bcc
917830 F20101219_AABQEK seref_o_Page_045.jp2
6fcf2a61132f633f769b84904813beef
8261e057679d3666bd234f430602c18dcb1e63be
638374 F20101219_AABQDX seref_o_Page_032.jp2
925d9ecfa4047f56bce18331c906850d
5bc511da662c1c1a502b74611003fb99eeb9f475
8293 F20101219_AABQFA seref_o_Page_109thm.jpg
27a572aa4934a356c3a5e9558b25e6e1
bbd0052a1942d5493b6f8d77ac2e053bed39f946
126836 F20101219_AABQEL seref_o_Page_109.jp2
bf135a3beda3c3f01bef291acd9b9606
659f555f654aa7b610c7f9fdfaa5e36563434104
38800 F20101219_AABQDY seref_o_Page_004.pro
eae7d6a8c82f861a66eaf07b07266761
a98f28a46ba849e4904c6539dafca219f3c28fbf
7035 F20101219_AABQFB seref_o_Page_104thm.jpg
405c9e7b610a97c2cc472ef698542ed5
fbeb9d9d3900856bf87cfe3da68d2c36d5b4c9a8
803682 F20101219_AABQEM seref_o_Page_029.jp2
887d91d4d3c52fca8f576c33825c2f79
5df428b93d70b9b2172cad51a2a3cbc0e6f1e28e
50225 F20101219_AABQDZ seref_o_Page_092.pro
0b9ecc7293aea50f4df3b8d1e550a90c
648391a0db158673ac2e8b7e1a56ba0ea7b269aa
8365 F20101219_AABQFC seref_o_Page_070thm.jpg
287da02e66a5d54f80e4c8198eaafb75
3aa3da5b5bf8325d0d2a1efdba54b2f80e703bd9
117929 F20101219_AABQEN seref_o_Page_113.jpg
f910719f751c722f24d2a6ee65a5d842
48c2466940a1f4113c60d02a0bd245b28a7c0128
4661 F20101219_AABQEO seref_o_Page_061thm.jpg
6a1d49750ac453a7163667268899db15
3a9a476c216b87fc28f8425d5c27b5301dfaf49c
62028 F20101219_AABQFD seref_o_Page_042.jp2
fea08e393bc573ed30a93b399a9d8ddf
6c33d420ebc1bee580bdec761bc1ce7e76f6d233
F20101219_AABQEP seref_o_Page_115.tif
ae53b4e8e339ce49a81dfb03ac214047
51e0b26d9addaf5086145009ebc5e316df542779
F20101219_AABQFE seref_o_Page_106.tif
e436bafd5076b7f165640b6cfcb0f81b
2a93b2fcdefd38485fea05d8bb01a6c8b1df03ad
34244 F20101219_AABQEQ seref_o_Page_023.pro
36e72f04c13eaaee9855761cb5b8ac46
39ee520c0dcea1a69951f332a50e57af89d08bd6
119681 F20101219_AABQFF seref_o_Page_018.jpg
b54ddc888fadd6c7fbf703a7d60a2bbb
b4d7b98b7a9c10bb25a7c7a5e5a4935f9d9c4857
F20101219_AABQER seref_o_Page_041.tif
76133a4f211fd859acc0d742b12aaaed
18a316fae8dfe6dc7829b46109b0950e277a113e
F20101219_AABQFG seref_o_Page_062.tif
4d97e6d67225becc07ffbc5807994904
ce83557325369c5040c3a5d20830cef832e744e2
545 F20101219_AABQES seref_o_Page_080.txt
6d02abf3eaa5397c2c68ff2bc66a1d61
59428437778dfcc7d1d4758f450c73be121a7a59
6739 F20101219_AABQFH seref_o_Page_087thm.jpg
15103305a57721dccc44e3eb76b6d3e3
a7d15b99e65ec003c0ca9c00f821fe31ddf52a2e
15352 F20101219_AABQET seref_o_Page_082.QC.jpg
edb92bf8cb49733aae1e33eb14fe4810
2b503ac989abe370bcdd0cb1dcb6546f05e45d0e
35752 F20101219_AABQFI seref_o_Page_067.QC.jpg
89a71c67d939c08c72f7db4e2a63fd06
f8a29b5a3a70a464a612547113d227e4c53e644d
1605 F20101219_AABQEU seref_o_Page_042.txt
c3572cb6985445a63e7cf73c6f8f21f4
c56686b8c68c07ae547a289d76bf2fef45db0cdc
18768 F20101219_AABQFJ seref_o_Page_079.QC.jpg
59592420c404794a06fc259206d844f2
5160fb16c9308871b7a6ff933a745d1d6ea4a70d
1051916 F20101219_AABQEV seref_o_Page_095.jp2
5bea7b4ce1c5e962b2db8f63c4b1558e
f2e99b45be7469bf5d347b2e82df9307a2549f39
108905 F20101219_AABQFK seref_o_Page_092.jpg
97c6f3d3277b32dbd714dc9b6a5bc058
b56e4b6c72b61256bf8e4456d3e1c07a43f5f4df
F20101219_AABQEW seref_o_Page_015.jp2
61cec030d6d8e1d0e7dce135a1eaf710
1a651cf732ce8b6ed92ccc73b8d8c58e6d5b1617
F20101219_AABQGA seref_o_Page_036.tif
1e5fe549c81afc317e177eb334b071c7
9d417b5a44b4f45361d0fc91d61099f7091b37e1
F20101219_AABQFL seref_o_Page_034.tif
97c025af98aaadad3f61bf712ed8eae7
630071987d54f505b0024e6328fe0aa235a4ce81
1621 F20101219_AABQEX seref_o_Page_034.txt
9d53f1a311470c05b0ab961461648dce
c62aaf816d4a4adc70e908756de72bbf676e3feb
593978 F20101219_AABQGB seref_o_Page_051.jp2
9b29453dbbebae6e7bd56f62f97e12a5
f809205aac339c36293291b18ce7af6ce4adb98d
18438 F20101219_AABQFM seref_o_Page_052.QC.jpg
8f74644f83b7ace901b1ac729841dbf6
0af56831587a7ef27728d4db202740503476b37b
1051976 F20101219_AABQEY seref_o_Page_014.jp2
35ca08855cc9cbca87467b0cae8dbad5
828da8c939e05a91e3d1eaff242787972f013946
81114 F20101219_AABQGC seref_o_Page_027.jp2
9c2d3723cfded6e314ff2f623a288622
69ea22bf1734c36f8cf9d501c9cd3c8ea2496617
F20101219_AABQFN seref_o_Page_114.tif
1ace22ff2f544d902d836872f31d0c94
fab4096f0acd1b2c4e5e6d9f680f480b4a440797
5315 F20101219_AABQEZ seref_o_Page_041thm.jpg
a7fe829902f030f08d6f39a803e30316
008534db74a9b6bfe034089b36c1a63baa902e35
5974 F20101219_AABQGD seref_o_Page_088thm.jpg
4131aef1e7954fc9ae18348d0cc94304
5f46eaa4e6f00870e5686a4e19de23ba60450282
847800 F20101219_AABQFO seref_o_Page_007.jp2
120b4f583b5e8071b2d20c3083cf249a
5cdeab5e09369096229f4366c25c75b322c1ed91
31511 F20101219_AABQFP seref_o_Page_010.QC.jpg
f45fee9a5f8b893728946870a646a79d
4f7e0ba5089b431571316ca8e57ca762ddcd27ef
1184 F20101219_AABQGE seref_o_Page_035.txt
cca22617ac9b328f614415eb59266693
f501e433644a0c6c992749bd06628fbe8cd9dc0c
F20101219_AABQFQ seref_o_Page_087.tif
2ed48b46615815b43b7b19e040882090
65df3fdc6b34c0d9c766484ad0536c1b6718ad2b
2296 F20101219_AABQGF seref_o_Page_102.txt
9a9f08b758cb8602f9464a6027d1386c
de10f7531aec4218d909c3075601385b084ecde0
1324 F20101219_AABQFR seref_o_Page_011thm.jpg
b7f6ac0f38829b0d08c6a051a9cb2742
e6805f858cb8c93708e578671c6ef731e063ff8a
104121 F20101219_AABQGG seref_o_Page_019.jpg
b714b16f3e1022f06babd7417bc82509
f18a263cf1e300d194bb8328c3bfb370af439266
94449 F20101219_AABQFS seref_o_Page_020.jpg
99e0be47ec8e0a9ba7ade176374c4fda
df56a165b4456d57aa9dd36e8c5afd4c3cb3f719
5887 F20101219_AABQGH seref_o_Page_078.pro
affb7a0195f5cfdcbdfe48d4762dd1f0
2f9a5ae4f089560c3f08fa8a22953db2cec4f760
2502 F20101219_AABQFT seref_o_Page_013.txt
34b8e85ae3b3a5058c7194d1fbc1539e
5b24a01da4367e1afaa242ca01fa5aceb8227571
2053 F20101219_AABQGI seref_o_Page_092.txt
fcea64a5fd0f014a436a409a22007c29
a3e42792b36154653f8b3d96bb1594b76fe79494
1051956 F20101219_AABQFU seref_o_Page_020.jp2
bfbd593466f57b04b1daa88b8de2c8b1
9d4cbacd29c93cfbb85c6aa81dbaf0b5ff2146cf
F20101219_AABQGJ seref_o_Page_001.tif
30c5bfa6f987ea856fa4afb0f86f4b88
3c82d536fe1ac3fc73f12e517d1056bef4b22f40
18508 F20101219_AABQFV seref_o_Page_078.QC.jpg
e806a836da60ea1dbb7f3fbd4c0e9dd0
b04cb82a78082bf1d3d01aa10dc871fc542a5feb
2357 F20101219_AABQGK seref_o_Page_067.txt
35b27a532073c45e8ed666f643db2bea
5971da40a6174d8f51276b605bc08ec9bd9da12e
64412 F20101219_AABQFW seref_o_Page_112.pro
b82a6630c1aa4cc5db31e47a0047f62b
21c19fd13a8dd1c0673a8a8efb8a19d844c2b09a
111419 F20101219_AABQGL seref_o_Page_012.jpg
45028cc1c423b0197ae2033b02bf8c6b
23e13546764daa6dffd175366cff7290cf0e2c57
6060 F20101219_AABQFX seref_o_Page_036thm.jpg
1bb6e395fba0389ea6b197f42285cd8f
0ce5d1ec02ed710abf759eafc23844eda9bde5f1
1051938 F20101219_AABQHA seref_o_Page_075.jp2
c12a9ff2bae30d06d8c428d2780d7dd9
90a5532ecad4c7b37e02c59b0ca3dff6869ef906
57905 F20101219_AABQGM seref_o_Page_015.pro
8b38db374f78c3ab3c085cf78bae83e6
ffb8ff6e02affefd934658202d8eb6dce95ed72f
131548 F20101219_AABQFY seref_o_Page_013.jp2
143a60faf27aabd3b16a96dd525318a2
fc27ab4826b40f58985d4ecbc1728eee86a8a220
1016069 F20101219_AABQHB seref_o_Page_048.jp2
64feef3194e44aa8cc59139554759eb0
321c8271daca33cb5f40ec2053a6c43a073fac0e
36300 F20101219_AABQFZ seref_o_Page_068.QC.jpg
65f692fe45b8c0ec747c3363778999aa
528fa8078abcd2eaf9fa53508f4554239819db79
2329 F20101219_AABQHC seref_o_Page_074.txt
b27d3d35a123c2bfe787cd966b4be225
226a27eb722b4c9526c268b2419e450f72c32ae7
26415 F20101219_AABQGN seref_o_Page_057.pro
0acabd84104325cfca159a9428de6863
fb336f93f65592e96d740aa87b60635e28437a29
1051829 F20101219_AABQHD seref_o_Page_103.jp2
41525eb2cd7ca0ba7b0e55aad96fa725
f58fc62c4c4172861914ca1a055276621ee23bd9
35788 F20101219_AABQGO seref_o_Page_039.pro
ee6f03905415058ef24624384b2b64ad
5a16485beccc80e3a899b16ccfe15228da3bfdcc
53195 F20101219_AABQHE seref_o_Page_061.jp2
4fa29844409b4e89d832f4040fad2db8
c7d0a62fe3602e964b00522e1bb69af4ddd93ca7
6542 F20101219_AABQGP seref_o_Page_056thm.jpg
470e00ed3707a4b7302d6a51cbace81d
aa1cb0dbbbb91827dcffe7dc5de7fb9167468b5a
63992 F20101219_AABQGQ seref_o_Page_066.jpg
14516ae3d809e4c9c6e5faaf84018877
a1f63126c868bde06781c6b9fcabacd689d89cea
104950 F20101219_AABQHF seref_o_Page_071.jpg
adb37dfd1ed607879f605a03392c5e54
d8043873b14f4a9bbe3de957bcc342e4d9178287
4980 F20101219_AABQGR seref_o_Page_081thm.jpg
08aae349c4c305b38f4201493666b269
6fc572af5f3ef8a9702cef58729e2622430fcfb9
7547 F20101219_AABQHG seref_o_Page_094thm.jpg
e6a4724d6273a29ffd5425cda874bc25
13c704d7f0b2dbe60ebf7a05308414fc6f7f6d20
1644 F20101219_AABQGS seref_o_Page_104.txt
852917ccb17bf06bf4ade0655195d540
8ed799cab860fa9aa9273fe4a0839764a23af571
7964 F20101219_AABQHH seref_o_Page_108thm.jpg
0704ba2a5280e0a00385ffec71224bc8
0a3c26a4a69ae7c6057c31d83113ce44b784362e
77224 F20101219_AABQGT seref_o_Page_045.jpg
dc2ff247c6f6a054831ac61fbc8a522a
2c6ef2a460fb8f343c5133b25e7b513555eea75d
F20101219_AABQHI seref_o_Page_060.tif
e04a3694531362b01e7653a5c531cb48
da32d4297d03983dab2f88fcb3d77d2a26008b69
2337 F20101219_AABQGU seref_o_Page_068.txt
f3223692af3b1b5f331dbdca745aeca5
9f814808e8f4171aa6e455ac349ba20315d920b4
30526 F20101219_AABQHJ seref_o_Page_053.QC.jpg
c274439d024436c1826a542f80e28186
c6bf2d3d9d26ff0ac96fa55296b9c0d0e39f00a2
17325 F20101219_AABQGV seref_o_Page_087.pro
add402abf050ab20c356b70b80a55e4f
294bb534c97706cbb07759b718ff8268542d52ad
1051919 F20101219_AABQHK seref_o_Page_055.jp2
eb8a2a8c1089bdbdd1efb11e8f899388
5962d18246990a123f02c8f83e9239fc1b96c599
54607 F20101219_AABQGW seref_o_Page_064.pro
48654b80006df1884463966965545650
55e04e5b0b22ea831fe036657738bb3d1f93dee3
F20101219_AABQIA seref_o_Page_046.tif
1eec268e7c9292f584aed42d6dd9c51e
03e9af8fc5c08095415b0a6435302ddc463ad327
25256 F20101219_AABQHL seref_o_Page_055.pro
ff6a0e1efbfaad1a627150dddc464a25
05eea1e08509a8b9beb7712baa31623f515a5af6
37190 F20101219_AABQGX seref_o_Page_080.jpg
0de7ec09f4632c5e579cad8f73ac32b6
a6d5eb1e82a5e045d97b12b206301ca8ce7d2e5d
33464 F20101219_AABQIB seref_o_Page_111.QC.jpg
426e661f66ffb2642173b9c5b046496e
a3fe1d2bdd253cce324c9a686841193239e5e15a
F20101219_AABQHM seref_o_Page_048.tif
94eda4d275e533bbe4a05b80b425fd0d
31a164bf4bb94c873105d5239b0580064e7c3664
1557 F20101219_AABQGY seref_o_Page_003.QC.jpg
e45c69fb11bff1361c938ce3aa5d2fb7
d2a0f9413f30dc49099e58228a521528fc3e4c21
8062 F20101219_AABQIC seref_o_Page_072thm.jpg
b4f67297f4be98a072a55b0ae25b9d50
a0eb9a928a91bb92edcb422962062c30c24cc1a7
40314 F20101219_AABQHN seref_o_Page_065.pro
5078a4491d730ef1739c3775b090ae54
38219cf04b2d313bc0c945bd81f1954f3d4bdeb0
85899 F20101219_AABQGZ seref_o_Page_056.jpg
217233cc86d9f36e5dcc89c710a8250d
912c356029f7dc3d10cf1fc551de2d06b216179a
F20101219_AABQID seref_o_Page_019.tif
3f3098b80b9b55c475859e0b4c0ef695
c176756f54f3b45283eb455a13e4245a01fb7811
25889 F20101219_AABQHO seref_o_Page_035.pro
2a4c8713713cb11abb5f4a9c561933cb
19a434b3318ecee111bc18719db3434a0c0ea233
31112 F20101219_AABQIE seref_o_Page_103.QC.jpg
4a8b53358971acb2148228948bee44d8
9e6ed2399e9a287d9245dd8ccdcfee819806d7ef
1474 F20101219_AABQHP seref_o_Page_047.txt
807f475332b7c9ad1e62b5c7717bf623
2cfba7d944694ad67095f8bfcca584144f1f4b8b
720314 F20101219_AABQIF seref_o_Page_088.jp2
ec0885441b5a78d91897d562eef915c7
6a659a407d0f783d5d1193c95aae6c7f70cc1198
27209 F20101219_AABQHQ seref_o_Page_040.QC.jpg
7b06d22c0970a4f02d3e823d67111944
98ed7ff7bf5e16d5542880b037cef57424bddc8b
24928 F20101219_AABQHR seref_o_Page_022.QC.jpg
35e106b4a24796399d3d1b6f32d6fe9d
85a33d54c511d01217fb4332aab3984e0721b39e
1390 F20101219_AABQIG seref_o_Page_003.pro
82335c72c9d64229d9cd45a058fc9f16
fe909a8d8680187e7b3dc23670d92845713fb41f
5345 F20101219_AABQHS seref_o_Page_042thm.jpg
6f760c463f64fa2ab629e66954b51326
ff2dba19e37f62bf470d4f971b2cf7c521eb11f6
27106 F20101219_AABQIH seref_o_Page_038.pro
d5e216f5c1dec4c4529defee54c20146
307dfd5c60d22d93e770a66b6ee4f4f3d3fbc11c
34940 F20101219_AABQHT seref_o_Page_096.QC.jpg
12caed6bac8439c6a0f829a89fbfbbbf
0fdab61a178136992e778c8656ac44770cb78b63
113815 F20101219_AABQII seref_o_Page_070.jpg
2b8037d618bbf50937b0255bbb265807
9a2f9c58b3defc4a25d783025eae32b483950dff
6378 F20101219_AABQHU seref_o_Page_097thm.jpg
ac0de248547b784a5fa2264b5f5fe457
3db92ae96bf3724cea6c1796f91576f8d372e5f3
8265 F20101219_AABQIJ seref_o_Page_017thm.jpg
aae89b500758f46f86b31982bb77cf61
111c6b4f451dea4b26243ed28b42769413bd63e0
F20101219_AABQHV seref_o_Page_045.txt
f75893deb0fb976cf6c537e715b46817
cad6a3fbf22a22e6d4aad83b5151fbce4d6e67b7
82705 F20101219_AABQIK seref_o_Page_004.jp2
caec762f583575d237ab0f08f54c00fc
86d11c0bd713588cdce89efafbb75b088c3570cc
81049 F20101219_AABQHW seref_o_Page_054.jpg
4eee28dfdbba0dfee42c65ff177331fe
7a0b5769cfb3591aefb1952d741752cfe2fbac0e
40402 F20101219_AABQIL seref_o_Page_105.pro
ab5fef69740b17402ff2ac9eeaea92b1
6db4f51e9c2807a41a57faa559566f7cb3dcbe69
5050 F20101219_AABQHX seref_o_Page_049thm.jpg
43f8f3e6112f170066f7d7be9db01d37
00d2505825b1fdf7f4c9f34c05ee4bff91c7a5cb
90520 F20101219_AABQJA seref_o_Page_100.jpg
ff907a4e3fe60d1bd61e0bd03be6f043
9ab34cc478fef6056fcc9309ef02f23623b15522
57288 F20101219_AABQIM seref_o_Page_077.jpg
07fed0b2053dec3f159ab2ce5d3eda00
cc56b261aee176fd304682d9724eebf84ccfbda4
56058 F20101219_AABQHY seref_o_Page_024.pro
5312899356232d24b146799b10fbddd9
a7bc38d6f8c2795dfba329e2db56cf88655515fb
24426 F20101219_AABQJB seref_o_Page_004.QC.jpg
c3164eab6c0276d6f28fe13428206a3e
1a60ff8649f10e71f4b488284d33fb90620b3d6c
8052 F20101219_AABQIN seref_o_Page_001.QC.jpg
177a36acd5dafc3df4639e225f5cd444
2d74617d8f986725a747eac39b3d67361ad98d6a
46518 F20101219_AABQHZ seref_o_Page_021.jpg
5f24d29347be0efe508aee0058edd5e0
1e17bc65e425c2e292ad53118967b353232d7a04
7700 F20101219_AABQJC seref_o_Page_115thm.jpg
3ee5bab5cf28ecc2914aee560507fba9
826d9a339403f1fa9abae8f1b10bac167c75f9e9
60548 F20101219_AABQIO seref_o_Page_070.pro
d5bdb053bf7c8a507ba99f3871a27d86
1f76341d3759eb0080ba95c4066871ba8c0f79fb
33853 F20101219_AABQJD seref_o_Page_047.pro
218f55175c079a30a6b7b0e993830925
9a635f83c7904e0f4faa58f1341802a0b3b48e12
2404 F20101219_AABQIP seref_o_Page_107.txt
ae00c4609abef5ce2a283e9747fae482
3c26c533fe9ca602df5134fb8e48365050c43af4
73909 F20101219_AABQJE seref_o_Page_093.jpg
31cd5ae4618aea2c9a930e6e73dc77ac
c218a28bab22f60b5289b2a7223f5381e54c3aee
1715 F20101219_AABQIQ seref_o_Page_089.txt
a020a81c3e246b8f058cea8302cd62b6
892b81004d981c884951fe8bae431a13933307fd
6866 F20101219_AABQJF seref_o_Page_040thm.jpg
6a098ee67082925f5d29584dd46d509e
160730af167997ba8c44118407349ef99beda7a9
35910 F20101219_AABQIR seref_o_Page_074.QC.jpg
e61ef49f6f044e9036f0ca2bc5a10791
1577dc90159d698769c1eed5e45195acf5b04212
1051960 F20101219_AABQJG seref_o_Page_046.jp2
a9a524d413c5a7b3ca11f63b1334316e
e489d80c85919d15ad5b31f7b8685453bad3fada
10716 F20101219_AABQIS seref_o_Page_083.pro
bacca11a0c01a41a2a29c9031f59933a
091af4e57be2b4c4c2eb2e2d8839d2a45614b08f
1721 F20101219_AABQIT seref_o_Page_022.txt
81a46e2946a05321b28dd79b8fa50c1c
b58c23b49bbe3042238583c52d3cff970a2fbc33
29703 F20101219_AABQJH seref_o_Page_086.QC.jpg
397feff4a29eb31801f0a236302e1d7b
ba6ffb1ec56eededbdbd365f4a74df3d2a186245
34939 F20101219_AABQIU seref_o_Page_095.QC.jpg
a8844b6d8eb56a4b706de7f71166c08b
bf3c0971bb34d58e1c221537b0c8a8a080b77840
79794 F20101219_AABQJI seref_o_Page_022.jpg
07bcb69d1ae0873738b714ed989c2075
e8350df51ef1cae25c9c7faaa6fd0124e10bc540
6449 F20101219_AABQIV seref_o_Page_011.pro
20128bfa43bc66c58961f96ccd3477cb
933e7414d32f48413016e770ee8ff98e43587caa
F20101219_AABQJJ seref_o_Page_100.tif
a93e89d7816a09cba321b45f21a79944
f42eaac69828d26bfd26171f246c2838c2e473b4
3294 F20101219_AABQIW seref_o_Page_006thm.jpg
60c591dcc3f9042f0e59f1f123c9fe30
6ce0ee1adeee2ce0bd14a39e00ee4c4c552e8d7b
F20101219_AABQJK seref_o_Page_042.tif
2e54d70a3280b9e52a31dc8ff1b519bc
c63dc3c8b89667b0bdbf8088c120e36313036eac
31925 F20101219_AABQIX seref_o_Page_069.QC.jpg
a0a61e22d40e251c0e62d0cb72575ad5
ea7fd1097b61efc6d64f0a140562870c527092eb
5842 F20101219_AABQKA seref_o_Page_028thm.jpg
34a9b70ef7a3ba1b3729e545dbd27747
7bd7ede0c9aa687fbcdd7e3254b00acdab5c15ac
1507 F20101219_AABQJL seref_o_Page_037.txt
4f09f53cfed118f7dd02115ac2dec03e
f2e03f4a4b4ee982744cd132f986548928ed1cf0
2223 F20101219_AABQIY seref_o_Page_016.txt
47b44ce816357f1e0c6513f424983886
d167842f337d24cd8d75dae45bc8d29437122d68
32731 F20101219_AABQKB seref_o_Page_005.QC.jpg
5f77f37395f973c0dd9551a563504400
4cdbe3d38fef62bbdabbf2b801edb48b88da24a2
111527 F20101219_AABQJM seref_o_Page_015.jpg
62261ef266f3cbcb56933801f4ea329f
82884d2967dd38737c69427c6c4eb8098aaa6043
115088 F20101219_AABQIZ seref_o_Page_071.jp2
38345433fe49e79ae5b03cbcb0850033
1ecfddd266840b599da2584e34cd2c7d5b9c9a26
17648 F20101219_AABQKC seref_o_Page_060.QC.jpg
bc6906e14cd3cc6928fdfa8af67b5756
0f7ec22685c0376e989590f5cc59e94d486a83c1
8267 F20101219_AABQJN seref_o_Page_112thm.jpg
a55ac95f70bbd81e42b89f2f412d4b0c
8842051c99fb050a48bc33bcf6d25d30de5c670e
51398 F20101219_AABQKD seref_o_Page_061.jpg
2cdf23e3a99113fd3f15b54c8227bad5
0ed4c126cfd2ff1f98eab0e228e41e03055e31d9
23862 F20101219_AABQJO seref_o_Page_001.jp2
fbeffb653fd0db815d85fdab12e4a1c0
255bfa8dc6a1f46b627ea817eaf14239473b9cee
1323 F20101219_AABQKE seref_o_Page_060.txt
1a46c75e8959e7d5f1bc0c465b7e9587
e66db30f003c064492877e69188c00edd08d2a83
74312 F20101219_AABQJP seref_o_Page_089.jpg
c42d9d8b0a253450ca1e0708a9107f77
219c2b18692c30f38330857632372df61ab86b89
58530 F20101219_AABQKF seref_o_Page_012.pro
14862a88dce91ad7159268b404ed10e7
955b5c261559085296fea38a57e246ac8e5cc9c9
1041 F20101219_AABQJQ seref_o_Page_002.pro
028c23974633346181de5a38cd8d3528
ff97a5574fd82f3b4e42f49d96ce212423ddcc09
20549 F20101219_AABQKG seref_o_Page_028.QC.jpg
4a6284136e8abdae19fc33fef38f7cfa
973802a9808b112cd46ad5cf6c58ee04ccfdf934
16464 F20101219_AABQJR seref_o_Page_049.QC.jpg
11c2d49d9f2f73d5965c6a1793d9c4d7
717ba152ed9d1369058c3f46adc926517b90dd1f
26185 F20101219_AABQKH seref_o_Page_105.QC.jpg
05fb9e0439414af1aabb27d39a5a1b8f
6ae32c8d0c6896d883d9777a16c3bc73f7ee3a0b
F20101219_AABQJS seref_o_Page_027.tif
6c30804dbe43c1ea916ff54df4c67634
4a17070c4f58d16c78f65ddd8dd8db04629575a1
F20101219_AABQJT seref_o_Page_055.tif
6796185d3736678c7853c41d4458008f
be4c1fc4f65e0eaab8e24e530d69dbd8475ebc88
93636 F20101219_AABQKI seref_o_Page_086.jpg
b5c84334016260e06cee38dabffed19d
99d9356c766b44cc6e0ba65107f4f38b571e305e
22503 F20101219_AABQJU seref_o_Page_032.pro
ebbeeb38f07c3919fa75b4a696c73fd5
0cbfa8076c2dcc363ed9ebbac9ae75aede5694e1
11172 F20101219_AABQKJ seref_o_Page_116.QC.jpg
8a65d74d9c1c6ef6f1c601b495ab1de5
6dd8e71af4da289a7d191745d36e839f0c0393c1
550443 F20101219_AABQJV seref_o_Page_082.jp2
936d13a4745bf5cd7d5f6719837313bc
0ce650910ec90a513d07aa1363467fc7ddd189c8
4591 F20101219_AABQKK seref_o_Page_058thm.jpg
b560d931cc0a75dcc8724da79bdf180a
1f1ac80c1f4027aec021a8a1569a7a3076a07bb2
27484 F20101219_AABQJW seref_o_Page_042.pro
766a7a6f6d4a88fc49ca337e9ffb6186
fd6a2fe46a84e8a39b8f2c90293815d774738b87
F20101219_AABQLA seref_o_Page_017.tif
cfd20fce32dd629bad01dc56d18895b8
57b131c5451156dd8e67d2a28dd21721745ba6f3
45944 F20101219_AABQKL seref_o_Page_025.pro
33e81e746cb299987122dbf39bbff8f8
c09fd513fbe32597d06d212bc32d74418ab1632f
125925 F20101219_AABQJX seref_o_Page_112.jpg
e6a9fe00ff0078e2874f60bd67df2bf4
0695e5987b1b027208084389a6cbe97c89c7c828
118788 F20101219_AABQLB seref_o_Page_013.jpg
1edae0def4288bfbda769256bc0b0b72
634410f0ac0f8862fead2d2112a73e7dd283d83f
F20101219_AABQKM seref_o_Page_047.tif
95dc700cab69dac910038e73184d0839
19a8961077f2f48e1fca7ea21a067f6000b68559
700648 F20101219_AABQJY seref_o_Page_036.jp2
e2ca66fede4a81ace48b0ddbee32296d
6a571970ca0832f4343c4950aeb7292704d4a728
83202 F20101219_AABQLC seref_o_Page_105.jpg
1699519ddf60a09c7a9947793d16b05a
612dd570c7225cf29367856e06546ebed112b9a6
59026 F20101219_AABQKN seref_o_Page_075.pro
543c311debc4abb294e3630c9514a85d
9fed7aa0cee0b11cede33334ec5696be989cd1a2
7475 F20101219_AABQJZ seref_o_Page_110thm.jpg
7a8c7f24d84760b9e620e0d278529d60
b27a5afc7adc1a870f425620af2c98233ed1eaa5
F20101219_AABQLD seref_o_Page_075.tif
5f9af2486621dd933e2c39a7ef651b16
1ed5b4e373d3f1a11ff9470bcd0341215b87b65b
6261 F20101219_AABQKO seref_o_Page_091thm.jpg
9af78a0dee2cd39267aefcbbfbf4767b
ce431c154f311d6717eccf89c662557bbe5b38a1
35905 F20101219_AABQLE seref_o_Page_076.QC.jpg
288469273d1f100c48c1a59cc1619096
94b6f2fd403be0928fde24977671c5c39fa547b0
1615 F20101219_AABQKP seref_o_Page_023.txt
231491479a6076f2125ca348e956ef4c
cf40886f6684818c8540cb7f94f44ef16a09f099
F20101219_AABQLF seref_o_Page_071.tif
fc5f1518719d2218c24284a9892a9429
3f474e9cf3723c2fec7115e68404dc411e38b93d
5097 F20101219_AABQKQ seref_o_Page_059thm.jpg
af7e88ff06a25d27bc7d7eb35fe6fb7a
5129d40021128f73862711ab384751d7c3e5d471
5316 F20101219_AABQLG seref_o_Page_051thm.jpg
cb8743c72a324b06511dfeaa63ead867
705bc28ec40e48a499931cc2030c2a8306659420
37565 F20101219_AABQKR seref_o_Page_018.QC.jpg
9d02dff89b3cdf5d5cf39db0ff8b0214
0dc1f2af8b7b30b87903c09d7a75e0524b88aa25
33762 F20101219_AABQLH seref_o_Page_016.QC.jpg
50afb143837489c75e89422fd4fceab8
ff860029d2f2e1758fa7b0ae759df90b2bf075e8
81416 F20101219_AABQKS seref_o_Page_093.jp2
f267e8353e6880712d66a808ddc8db65
e4d900b6c4ce742f0d7f70763f62a5e3789d3468
1051947 F20101219_AABQLI seref_o_Page_008.jp2
47519ec932f651559835e14bbc6c3d17
785f06b90eff8ce3715ec2181c678de3fc8a32ee
111236 F20101219_AABQKT seref_o_Page_044.jpg
b27bb016430c9fe9363dac9b9ccbb85a
ff4da95acfcef24533f6d996da7258dbcd2f670f
7216 F20101219_AABQKU seref_o_Page_077.pro
82b267c59e8a5a7d208ca249809574c8
82f0bfbf9093e8934e22016189c44da6b4a332e7
3798 F20101219_AABQLJ seref_o_Page_106thm.jpg
7cb08e22d4665dd453b68429d328a9c1
0bbfce12adcb8ce15317bdd0a8420eb3fa3fe633
25416 F20101219_AABQKV seref_o_Page_051.pro
4abca56f16ba270d8a70014e36718920
d3a576596a092498fa93f963cf7cff967864b5db
F20101219_AABQKW seref_o_Page_070.tif
f9f0327bd67dae4eb27b95e74627bbc6
a6809e70777522831c2201897cbce18419f6ce4b
37268 F20101219_AABQLK seref_o_Page_089.pro
19ba60d887a1e406b1b7cf7649a9b735
c9b7f9cfe6baa2e1166fc0fb221d9d0ce639bf4e
28464 F20101219_AABQKX seref_o_Page_025.QC.jpg
6daeaebbe5c39728c0558f211bd28af0
b14f7609d9e718d73bf582611b7c7a10ae5ed378
134496 F20101219_AABQMA seref_o_Page_112.jp2
6e479392d4874f8efda18d3f3110a338
fee49a5c633ae64fa247c235660bd771a387daa4
2320 F20101219_AABQLL seref_o_Page_075.txt
5cf3ea54b12b3b1c765abf486c3478e2
5479b536a80662c9556301cc0cc7ef73995a24f4
52723 F20101219_AABQKY seref_o_Page_007.jpg
dfa2a5bec3b43f6f5cdecfd7cf31fdc5
47e829c761000161c90b0924d01d49a328c0362b
13234 F20101219_AABQMB seref_o_Page_082.pro
97206b0dc3920c4ce79a8cc23cf900fb
7ce9d8765cddc3d0e5e453d70610c94d291aab4f
114864 F20101219_AABQLM seref_o_Page_075.jpg
2921ba9cc950842cac8ed6d0443423c7
87748accc31367bb145ca4ed12f4a587b3d21f5b
F20101219_AABQKZ seref_o_Page_051.tif
a58cb78eac3cfb729db1677b3e663b1a
5e50eb7132cfe70dba8a8980d608bc4617e8e679
40691 F20101219_AABQMC seref_o_Page_091.pro
1f3090c1518cb0380cd2970d93abe7b6
6f70cf42324e63f99fa5e484a958e508e2961261
32398 F20101219_AABQLN seref_o_Page_103.pro
328bb5e6e12db9df55719c832813c8b4
796682f13e9115632943f59f77c1f8a24358f46f
6384 F20101219_AABQMD seref_o_Page_043.QC.jpg
fcd319f110869d26ef2c6741775bf741
811321c7c34500ffea72f4e56e172794dc0604c9
F20101219_AABQLO seref_o_Page_097.tif
1b31651dfaf5fbf50025567d461b622f
196af0caee9e98e9a39cfcb1a89ee1f694723541
F20101219_AABQME seref_o_Page_061.tif
05d8ff706ca549a421848e85b880d846
745a6e7618d5994a120dfe60d441decd37f556f1
F20101219_AABQLP seref_o_Page_064.tif
609b20ff7394504243d92eca91a86f90
4713fe259d796d0a6397be487026731a395d6041
2036 F20101219_AABQMF seref_o_Page_020.txt
09c8c9af03b925ea327258e4cb526cbf
ca80401e848b4691cddec7eadd0414de5fdf9916
812465 F20101219_AABQLQ seref_o_Page_083.jp2
7ae47325fc253c501a1c732633ef6e1b
312648f46d47dff63f87279174509cf9ef777154
2711 F20101219_AABQMG seref_o_Page_005.txt
2b6b95e14a781d05ef01005a451b1c20
0e3ab24c0afc9aa7081cabd36b5175f8113a5e02
2372 F20101219_AABQLR seref_o_Page_114.txt
de4c7bd772920cc1a20753d8581f2739
4d4fb46364a89dcadce5a3d967ab4e887d59debb
1051965 F20101219_AABQMH seref_o_Page_017.jp2
2f8f8f305b5a3d960b9d7195255dcd77
d8ee1dc33f34bfa1b43a19f69f5e21e471c2e001
5801 F20101219_AABQLS seref_o_Page_080.pro
d6d65535ab2ff9c65fb13b06e7fd5beb
ac115a04733c1dac51c7772dd5882a653d5fa563
F20101219_AABQMI seref_o_Page_003.tif
7ad01b67ab665cbe7438c13b25b2f8c6
b23bdd06f66dbf132a9b305aaae28b769059a00d
6349 F20101219_AABQLT seref_o_Page_029thm.jpg
ae281643d613b8be94637d19c5c5d08f
d86839983a4d9c566033bfec6de6e2d2cfbbb042
35517 F20101219_AABQMJ seref_o_Page_102.QC.jpg
f0f57c75f59042931cbe940c349727e0
b034a1ad16b7147e821ee37e7d3962dc519742ba
934169 F20101219_AABQLU seref_o_Page_100.jp2
5693834e32cb550fa30949bfd6c7e4a5
0dba75abe49e4b455bbbc336b4f192ce77af5a2c
F20101219_AABQLV seref_o_Page_093.tif
aaf2e4be7a1f74a15e492d7f2b038400
d15c89e8f8a202609a36344c8333dd51f319a382
106934 F20101219_AABQMK seref_o_Page_026.jpg
c45c44e086a8b577ff86ac4b1095bd22
cc7cffcdd2e727c3fe7af95051eebd1405989d12
20789 F20101219_AABQLW seref_o_Page_043.jpg
f5f895aa7e17e2eef47c0bbc91d93708
a3c5e3923912ace0c7defe2624209b2ab32f688d
81471 F20101219_AABQNA seref_o_Page_091.jpg
ebdf4915e441e3e0b397a70411c90a82
dd41d8c89b5fbce7be3b549591242d5455f7454b
9146 F20101219_AABQML seref_o_Page_084.QC.jpg
9cf0a309e95e92494b0af65d65108e0d
14aedf15be45fd8dd6b121222c7d22170d6560bd
6364 F20101219_AABQLX seref_o_Page_089thm.jpg
1d760aab254e151575eb91936c2f1f2c
8b7254760c0cbc065f566bd2f5bd3e385c86fb8d
17991 F20101219_AABQNB seref_o_Page_033.QC.jpg
aac95b6783da91e0a77704e732bda63e
c58106c7c1c83ba5a079c5e2624cdc424184f7d4
F20101219_AABQMM seref_o_Page_092.jp2
481487a694fcc4e191edeaa028eb62ae
b94c628498cd559f21e1c0ddf1e0d7eb4cd36ca7
F20101219_AABQLY seref_o_Page_035.tif
29db76bf0ce41f06d503632f5e487841
0b4931083a67143d3806c0b5a6dba757dc0d353e
26104 F20101219_AABQNC seref_o_Page_054.QC.jpg
5da471ae059181052929bfc20cdd6ff4
10ffe1d43b70a46cae0bc65815b7b15fcece6696
1968 F20101219_AABQMN seref_o_Page_030.txt
5fb2cff63eef90169a09b3b4800a9386
c1f94179d4dea9dd9fd624a899d148c467e42b51
3934 F20101219_AABQLZ seref_o_Page_007thm.jpg
b27d118d66bfb50c50cee0380e4c74e6
749c5a6c9a5ee6a502d0185bcaa3a23feece35fb
608 F20101219_AABQND seref_o_Page_079.txt
90e057837da73e7ad9e244bafe4bb1ab
74e5cf6ddb1c030f61740dfe37e0bb3ba561ffd7
F20101219_AABQMO seref_o_Page_053.tif
c7f5d67e4df1ac50957ff9d52197d0d9
80624c25b7cfea1bab4ad4b9e909711acfbca6b3
F20101219_AABQNE seref_o_Page_116.tif
4929eebb8df16b9abae64bf176504951
b6afa56a77bef2cd5a65641200bb970834b20760
109524 F20101219_AABQMP seref_o_Page_072.jpg
ec12c522c6fb5ec456211d025c8ad689
5cb53e9b70a1e65eabf7d37e1c287a45f2e757eb
7499 F20101219_AABQNF seref_o_Page_025thm.jpg
b517a7c745ae8581a6d2d53e7be62f77
dda0b86e5b787329da82e72a9942b048b92cda56
34408 F20101219_AABQMQ seref_o_Page_099.QC.jpg
5745ccd72300bc506e8c154c3e27cfa0
97ba4b12d9a6c382ddf9d8cff7203ad8e8d4b55c
F20101219_AABQNG seref_o_Page_039.tif
f819f17273bf6bf32b80471fe52ac9ea
fb528b55f103a0fedb15689e17c59f7f59e7c7a4
824 F20101219_AABQMR seref_o_Page_087.txt
486d5bb72b00f39d45587586ebf846ae
5add763fd74cdb340c847061d96909d136e644ca
1051959 F20101219_AABQNH seref_o_Page_094.jp2
a02f314e1ee1f8c7f784a6a72124154a
7edfa11883620659f2722b3e556d30507bbb9e23
2099 F20101219_AABQMS seref_o_Page_069.txt
ed11ab1ea6b82935204c1cce01bf2d2e
4415ff9f9f23aebead51eb888c911ee6f9f92c2d
8092 F20101219_AABQNI seref_o_Page_001.pro
b0b0f0ad6253b26af9b939ccb8cb6317
3232b87e89acfe374a400ce011804e7ddd106162
1039890 F20101219_AABQMT seref_o_Page_086.jp2
33623d4522657eebae1f5b7e822dcb29
006dbca7c755f0b8b2f2a2d0afe3ee5ce021981c
18849 F20101219_AABQNJ seref_o_Page_032.QC.jpg
bdca79612e8473e5d4d81d12051ebf86
17cf4b2096fa969fddb0a579702f94b779fe049c
2681 F20101219_AABQMU seref_o_Page_063thm.jpg
aeef7bb1a2a6547fb12952e4e913333a
40818a222e625b19ee40e62a70ac6b6e3b1643bb
2192 F20101219_AABQNK seref_o_Page_096.txt
c0da135dcf91840d12dada3b13c45b55
462a03beaab1fa2a096d1f7a0ace180edc1492af
18945 F20101219_AABQMV seref_o_Page_077.QC.jpg
821a042107c7da9b3ab512cc860b3bf8
4743bed7a550b4fde955bf8bba1b332714a9b22e
8221 F20101219_AABQMW seref_o_Page_102thm.jpg
5c3f1b77099222dcff5ff41fa9fd3e5e
48314a3b7715d95646ca1f30e472391a6bd5a55e
35665 F20101219_AABQNL seref_o_Page_015.QC.jpg
b28f9d3ddaf2d12ffdcadd786dd656ea
e4740e74f66d29eff9533a2ba2fe34fe23616f41
1018 F20101219_AABQMX seref_o_Page_055.txt
28cc5bc6390559b26cb083fd46d7b553
6ad77bc5dcc59e9d025492917f007b76431d0149
F20101219_AABQOA seref_o_Page_056.tif
be2f0885fdea5ce005815da2fabfb529
a4ca5afb1f73fa97735a12279498ee5a0a759a1c
F20101219_AABQNM seref_o_Page_040.tif
8f66e66002d3950b9ee7f9777ec3d6db
80a7cf22e782fdc9bcafddcc39e404d61bd94bcc
F20101219_AABQMY seref_o_Page_076.tif
6874d74b8e893e72a360c597cd7aaf0b
1d9dea1bf407760dfd1b464fce0515e2ef805260
16308 F20101219_AABQOB seref_o_Page_058.QC.jpg
6067c401532804b0a3c7f8938ccaf613
9e2e9285f15cacf1a8a598d865b26e7636c1ebdc
1051940 F20101219_AABQNN seref_o_Page_099.jp2
001cd7aea3b5461af46c6b611756039a
79c73a49d8b173aa8d5dcb6fac7733b5dae15836
F20101219_AABQMZ seref_o_Page_032.tif
a9c1f9ad5288be31b019fd7b0498bfb5
36c9ff6d7352c698369fc03d954f3be6f5712778
117029 F20101219_AABQOC seref_o_Page_005.jpg
600e341bfa2d0c72c6c2e3973e83a289
e717909794efa8ab10e31bf8652d833a803e64ae
36274 F20101219_AABQNO seref_o_Page_046.QC.jpg
2872d20aa48e871d8d655d80447cff4d
eca2b052e3c1dd1dd87d440a69a056e4206daaac
51721 F20101219_AABQOD seref_o_Page_010.pro
d3f6c0f187b8e401dfb9be7755252a6a
0e1776f1056b96314afe7b85cddefa2e3fd57184
4854 F20101219_AABQNP seref_o_Page_060thm.jpg
7a046900c36342bda2a945684dd4bed1
0fcf24f10480ed894dbeb50070eb9e7307058fbe
6386 F20101219_AABQOE seref_o_Page_084.pro
c68c3f92e2869360f1b07e2efc63befc
7114ed2392a1f4b0c93c325539a5263bcea47d6b
F20101219_AABQNQ seref_o_Page_010.tif
995d9e6f3534c9ce405e00b0c1b48662
ab29be3c400135542e5e9cc2f8466141157d0fc1
122884 F20101219_AABQOF seref_o_Page_111.jpg
984d7ddd9975dc4761968f991b438e3f
dc8f86dab3f5d1c8648c14cd87dcabd4e0fe536f
F20101219_AABQNR seref_o_Page_092.tif
df9f48dd700c6090e598a2aa29eafda5
f20e5f5c92f6014bc3aee3ab48d7fc803caac99f
7361 F20101219_AABQOG seref_o_Page_043.pro
9c99f114bb30a2c878492c239c8d68d7
90efd2c92dabb30a3084cc55416820fc01fc7b27
36284 F20101219_AABQNS seref_o_Page_109.QC.jpg
83d136a2d66c59ca6a419c14c0a177f4
6662324ed6c49ecbdc0f9ab39e0d43b31952d4c0
F20101219_AABQOH seref_o_Page_113.tif
9e8043d6d6c465df80aa6a3a74887891
a6e9d09fe14d1d8b4b552b1af14dc11b19aa1c6c
58987 F20101219_AABQNT seref_o_Page_074.pro
94b7ec51872244e98c81d1288433ab05
4063d1a69d078b7157b5c6c6ac7c4a080c34dc99
7767 F20101219_AABQOI seref_o_Page_107thm.jpg
df09a914b6bb270c299ccc4e04612e20
9d68b953030487cbc5695614819f8529bfde51ac
F20101219_AABQNU seref_o_Page_043thm.jpg
8c7e2ad7e27066ff56a71da9c4e9a0a3
49429236a19f745b1ad4bc1de0e311b4409bb815
F20101219_AABQOJ seref_o_Page_054.tif
b68cd4010c7fa465e857f4ab9fb723fb
280f7a1f1f3c42f48862f215fece17b355050c7d
8172 F20101219_AABQNV seref_o_Page_013thm.jpg
0e92341e634b0676f70389fcb3f64e7f
9c9ed15684d2bfb22b585e8110ca954219936d13
695 F20101219_AABQOK seref_o_Page_116.txt
59f999ff4f9f365ca9aed3a8372755c4
62f8d21039372892c1162c92a4e36eb3a35bfeff
F20101219_AABQNW seref_o_Page_024.tif
87b77fbfeec7721cad439326f7300798
a2f6771c3ec7877dd16eec71bbba374d0f72a0dc
23679 F20101219_AABQOL seref_o_Page_045.QC.jpg
104409e3dca48e0c5e4b556cd1f8334e
8c081748479620842f9b8768e6a7965acf2d9c74
7167 F20101219_AABQNX seref_o_Page_010thm.jpg
9da6302d8aa1be8dc134771d90ddad77
4d7e6e2296642df92b469ae7a5e8a5bd4104d42f
2207 F20101219_AABQPA seref_o_Page_072.txt
06f5a2bbf17a1dd4b18ac67b478a292c
46392197b40e463bb78209fbcd449e8569d1a9a7
621825 F20101219_AABQNY seref_o_Page_057.jp2
fac4e8356fa4ff9eaa8c85c776da796f
bde81e267406fa6e7feb30d587c12653e1d19424
78029 F20101219_AABQPB seref_o_Page_055.jpg
011af0abaa1b650b97b2eb09de8f4b9b
ebbf1d4940632d799c59ac04ea99acc8102f9ca7
620066 F20101219_AABQOM seref_o_Page_038.jp2
ac1e6528c13fa977360cbfe2d85c5464
8249315d467af77d22359a116fe07012cc0aa93b
F20101219_AABQNZ seref_o_Page_094.tif
e8a7bed1c04da559e3258644dabb7d92
f3d61d5656da76d063bd04aed97108ff59413c01
469572 F20101219_AABQPC seref_o_Page_009.jp2
88e9a156882667be2837077f79ef60bf
e4005f397766afae56da921578510b60dfd05966
21737 F20101219_AABQON seref_o_Page_036.QC.jpg
b61fc44e0a0253ec1bde605c1e3c80d1
a78b95373e2c88946c72e47a04c77a241a395337
8097 F20101219_AABQPD seref_o_Page_044thm.jpg
e0af359d034b161fd23f9410368bfc25
87ea8d3f0595afc60a1585133b9962e01710e4c9
21155 F20101219_AABQOO seref_o_Page_066.QC.jpg
c8f3d8697050030952dd8e1cd17dcdda
27bf516258d8bc82802220911eb6e22974823ef6
8257 F20101219_AABQPE seref_o_Page_098thm.jpg
cc166868c6a75fa3fbd4acebfc597d70
54d76bb1fc57fec7e67a4126c5e0fa785300ecf6
6126 F20101219_AABQOP seref_o_Page_045thm.jpg
1751f673c77152ec5ca40b0e8aebf2b9
9e86994788e72d1d4bb893a41cc0057a3f8fd464
47569 F20101219_AABQPF seref_o_Page_058.jpg
9bcfa4cfae0cc0b876489638b874af7f
ebad941989347a753af7cbf0c6953891b524e891
8218 F20101219_AABQOQ seref_o_Page_114thm.jpg
94862548693b2099135a1964688f3dec
654c5fa057ee23e7d10e6ab47288fdd29c0abbeb
31528 F20101219_AABQPG seref_o_Page_047.QC.jpg
4ff13e7861ffb368c53cd93a096b3474
981d657cec9bbbc515ad385217ad88b99a0db354
103764 F20101219_AABQOR seref_o_Page_010.jpg
5dedd10652e769bf0e49f0833a0cc718
8c87632f3bec151ec928ebcbbc4d66c6a153acb2
2352 F20101219_AABQPH seref_o_Page_046.txt
0150d0b0326dc4c54450f28aad113a6b
f86ff843c07ab76f2d1d84f621d2375857fa42a9
37323 F20101219_AABQOS seref_o_Page_013.QC.jpg
a7891a8384f5be54ab93f57251bf5de7
1317e15cdb42e2cd1996f8fad3f88fe9926311a4
55926 F20101219_AABQPI seref_o_Page_071.pro
d2b08a19a4ae967ffe5d0a884eb84a9c
1756dc1e21d3a7e65a682372131592d1f52179dc
1645 F20101219_AABQOT seref_o_Page_105.txt
99879882c7144148fca3b23515ed7341
352b7cfcf3fff4ea17e1004b182188553ae95e72
2397 F20101219_AABQPJ seref_o_Page_115.txt
0d0c52d2e7aaaef90ee74cc2cd0a0b07
ff119ec5cb118f525ac98c42868bde3093912b36
2338 F20101219_AABQOU seref_o_Page_098.txt
0fbec2ed45c6dedddda091080c5f7bd3
8fbfe76973687f5252f7949b12b010091f3d1d1a
7938 F20101219_AABQPK seref_o_Page_101thm.jpg
4207394fae765bb8bc9208a24f3348ce
ffc30b77fb4abce451c98432bebb393288885be8
F20101219_AABQOV seref_o_Page_023.tif
af1c1a9f9e374ee087bd0d11f769591d
41a4bae5ecb9df011bd2e7e8ebce3c452db1792d
5777 F20101219_AABQPL seref_o_Page_034thm.jpg
0c02f9758ce6da0f0ca7610d8b14e8fa
7431f30c4b6e35211a37620a140a1d8f7870127b
F20101219_AABQOW seref_o_Page_067.tif
ffd30db8cc88bb514812535bedba07a5
f6b4384b3fe0f1f650b262e83c43aa0c16218a2c
61141 F20101219_AABQQA seref_o_Page_018.pro
b412b9280557767f19c062b9b72455e1
586abef332a37f959829b287214ef20d14100519
47746 F20101219_AABQPM seref_o_Page_020.pro
599a3aadf100b770801ddf27a56a8527
5ab3ddc86dca9c71d84113aa3fbc961088e4e329
8238 F20101219_AABQOX seref_o_Page_076thm.jpg
9ae41a2db9aa7e6b5b237ec68be601e6
a3214f4fecd5a033368a86d832216b04d96d7108
32447 F20101219_AABQQB seref_o_Page_115.QC.jpg
b86bdc20be2d264da9b6a191fecb2e66
025c5459f5cac298d4a9b69c784b5f7b067114ed
855474 F20101219_AABQOY seref_o_Page_006.jp2
d862a97681d07c08d4d01f6682a1ad56
0e8bebd571f9b9d032a5ebe52c735d6bee4c0f7c
F20101219_AABQQC seref_o_Page_065.tif
970e163862f32fe6ca78591057b18427
d14a38558d6eeac5639a374ff25c4a1b0bd61379
2451 F20101219_AABQPN seref_o_Page_109.txt
405123670ba0742f7130786469cd6db8
c7277f6310d7a137f088348ed3b3d6230bdf535b
18934 F20101219_AABQOZ seref_o_Page_081.QC.jpg
cc66fee5e50b08155550a189843865eb
faafe61a401e9f2d107ad2e95ca5ef65741932fd
29823 F20101219_AABQQD seref_o_Page_084.jpg
fd69247a04909c905df38c8eeef763fd
d8a89a864ee72c949de759d7bb5af98b180f2e5a
4599 F20101219_AABQPO seref_o_Page_011.QC.jpg
5677e44ffa7550d9c86b597e6844d790
53569844648f4528dd287f3e3889cccc8c15dd1f
86483 F20101219_AABQQE seref_o_Page_040.jpg
fe7b129a46e8f2402b21246150f96c0d
a0ed7f4e3846f0b0b0af06737c8cb50e2b05b1af
35193 F20101219_AABQPP seref_o_Page_008.QC.jpg
52fc038f239d43f75d17f786dd26f9b0
54bb1f798eebc8b0ceecb6007866c5c07f012739
F20101219_AABQQF seref_o_Page_090.tif
551bc4599ddadb923fb22b986c877c62
1fc533e6410b4a1ab58509636e4005c70204c90e
134939 F20101219_AABQPQ seref_o_Page_111.jp2
84c2ed10f6c4e9169e560c2dfe25d9d6
731c365ce339c599685475e5fa3ecf7d776bd98f
2208 F20101219_AABQQG seref_o_Page_085.txt
45659993690ab2ad3cc0ac41a75f0efa
b3959616c00798a25ea9fcce2797c44e2e6046e0
F20101219_AABQPR seref_o_Page_076.jp2
f2a6765ba1ce1a0312686a9f64c3e736
7f7153b1741b4cd58dcc67cbdaa055e6917bdf4d
1705 F20101219_AABQQH seref_o_Page_093.txt
c51487bc8a5f134316d47500c69c52a5
0614c9896a347150c4db3830b7dd2a3fc8f9e856
17982 F20101219_AABQPS seref_o_Page_041.QC.jpg
5ac8b3b187c8c2e4f33e5441f55a3a93
2424d979161d52fdede2d9034c1282de7ceb102f
F20101219_AABQPT seref_o_Page_079.tif
bd219dc6fd1465111eab1ad66263d87e
9eebacd2dc358bcebdc623ee3e2a4de76927e190
14966 F20101219_AABQQI seref_o_Page_021.pro
6ac2e649de761587a3e20d9ae9e657de
4f2d36b4f7af8937bec9fd83f899909959e813c2
F20101219_AABQPU seref_o_Page_028.txt
b915b3e4b2041241e4c346ffdaf0137a
2c64327d2858155af7cf8ed4737b8d378cab9bf4
F20101219_AABQQJ seref_o_Page_080.tif
acf78e504333c5766d7af03f3b855a6f
bc59adc02f8868d89c7fafa8f1cb958bae27502a
118161 F20101219_AABQPV seref_o_Page_008.jpg
9fb9f88e2d1254615de4c77334ce250c
46014aeb434abc8031a1d4acf96a42bf76d30fdb
F20101219_AABQQK seref_o_Page_012.tif
a7098063b7ee5aa6fc779586e2281482
1f7229326aba1e0247956310593cefdbe1caa10c
5339 F20101219_AABQPW seref_o_Page_048thm.jpg
6d98894ca2219632c4d110dd7d4565a9
7d1fe365c9942d9d302d1f8d0412494c87e30d97
1322 F20101219_AABQQL seref_o_Page_033.txt
0f43d4fa99593eca01585736d2b9c0a3
a1585ef84494bc2dba77abbd62e7c624ccfc9745
97621 F20101219_AABQPX seref_o_Page_065.jpg
f59b1e0fce48ac0680f0e04fa1a3214f
3d689d01daed244eb08b933c1ceb99f8b284ab81
2517 F20101219_AABQRA seref_o_Page_113.txt
96348feca58d83487b127760ac418cee
22ee2d3d9edc580abcaec329d0165d486839976f
7735 F20101219_AABQQM seref_o_Page_065thm.jpg
1dab709e5f53b47122f624674506bbdd
8a08413b5f980a535522313f4cc69d12fb2e4aa7
768904 F20101219_AABQPY seref_o_Page_077.jp2
cb8d0472116392c7ec500a38a3c75974
f380ba4d222770f0dfda7f8c490b79502dfe3438
1188 F20101219_AABQRB seref_o_Page_052.txt
d875bdf79634a2b6c9f7b1b04dbe1bcf
cb921ceed49bcb26337126b186ba82ba320807b2
F20101219_AABQQN seref_o_Page_020.tif
42687dfedbe360da8ac7cdd855fdafaf
3b75f64eb3d728391ba2488385fefd2e1a9af87b
1308 F20101219_AABQPZ seref_o_Page_088.txt
47c1bcd9ea47c4295d4ee6e248a1bc89
fb1a553fe75c39ba5fe91a16684bb5d625bc165d
17975 F20101219_AABQRC seref_o_Page_051.QC.jpg
5d2547f65dc01cb2bf36d53c56cb0a6e
5cac93eb6c289737f70c72502399c389a1f5e5d6
109944 F20101219_AABQRD seref_o_Page_064.jpg
655b422dd34ee4aa5955690082254a95
025c6132960aa65d708463f162790d85fc169800
54413 F20101219_AABQQO seref_o_Page_072.pro
43d16d241c3f935c5013fbea8455872f
bc4ab1bd4f0ee31482089c9ea4a8a325f7b1da79
F20101219_AABQRE seref_o_Page_031.tif
0f1055376c99b7dfa3b5b48999db29a7
505275aa5253b7d292ae8a5014606e97abb0706d
30639 F20101219_AABQQP seref_o_Page_036.pro
63d38a9564f0bd7b50a3507bacf8bfa1
f635ab92e844eb019d1aa53aee1474422af5837b
5404 F20101219_AABQRF seref_o_Page_078thm.jpg
6b1345d8cad2f962fd6300de8fcb1fb3
f395a36a4fb526ef82ccc061c068d78a7298bf2c
31450 F20101219_AABQQQ seref_o_Page_028.pro
a0804d1b7429635f9adeb67bd5ea894d
80735bc10f90a4e21e0e62f3430afa2049243c84
113142 F20101219_AABQRG seref_o_Page_098.jpg
32278d867b3739cf26dcde15da8f5493
5a560cb7d4575d15a77ed55aff96e109f8a2919c
F20101219_AABQQR seref_o_Page_110.txt
365d8cd24e23afd82a46061b9454a369
49afe286bd60e955cfd05ecdd41f9c0725b125f8
48608 F20101219_AABQRH seref_o_Page_050.jpg
e8549700649b35b055eac4fac86df154
6199e19c42576d2fd26ecb2bdfecdebdc74d5014
F20101219_AABQQS seref_o_Page_101.tif
c01f7bf2df569ef6f3eb763e62c20813
23468a6628790d46ab1b5c91c08595cfcc9a07ee
1203 F20101219_AABQRI seref_o_Page_057.txt
5aad70fb2017c90514f29a56c63716cd
2546e549a66aa3d0fc2df7b342939d2e043e921e
39281 F20101219_AABQQT seref_o_Page_116.jp2
8adb004954b2df3085dacafe8564bf29
300ce577b965f88d6843c1fab2ad191ea043f728
1861 F20101219_AABQRJ seref_o_Page_094.txt
9ec01b42fd319713c5b54ced5932f501
5001f931672eb8c53d05bff7277b4c8794f0ffc3
582605 F20101219_AABQQU seref_o_Page_033.jp2
661588a99116b9ed70d63474a454cdbd
17ac7efdfa7eec3bcb751aa17552ce63033dcd63
57054 F20101219_AABQRK seref_o_Page_044.pro
beec3ae71993b4768a40dfee86277968
d39ff19c7c3fc8dbb2546b51445da6c1c5666538
33829 F20101219_AABQQV seref_o_Page_085.QC.jpg
1c2672863efe4e9b0da86e404e1712d9
305bd752154bc51310c3f603950d0d18b5c68f75
F20101219_AABQRL seref_o_Page_018.tif
b429ebc32661a857464e2922730eba39
ed338761424aefc77c54185f017bb18c5edb971b
F20101219_AABQQW seref_o_Page_107.tif
c9de8bfca6b927fb89f5483b27825181
20c727aa46389cafd919c148773fc4ce94bc1d69
798629 F20101219_AABQSA seref_o_Page_079.jp2
8cff54fe8e058d58f38342648d827b03
684cd9b933ed582183a1ef3a3bce503a7c33e180
1051951 F20101219_AABQRM seref_o_Page_044.jp2
837dcdba1981f2269e93b17d4495fe9b
c86d8f891ba6670af5f1f22f3436b00ca5d840e4
94676 F20101219_AABQQX seref_o_Page_101.jpg
421ce38043d8476bdab5b0dcc171bc97
e50ada11e6da9e3acdcb2b647a0884b1c71c952e
1051977 F20101219_AABQSB seref_o_Page_018.jp2
99109d6675c9fd4af6a331464cdf6bbf
e23fc9ac710a52dd6e04a9db98760c7ca9fe6455
53699 F20101219_AABQRN seref_o_Page_095.pro
fa52c2eb14e1d67716b2e928ca765009
eba8580fa097f3a71728a6580d997650a5cdf8c8
33264 F20101219_AABQQY seref_o_Page_110.QC.jpg
6cc8bafa877ff5e8d8fc8b6fbc6d4bb2
6267199a28cf917755657df8e2d19dde3d8be9b7
35554 F20101219_AABQSC seref_o_Page_098.QC.jpg
6983d226ca9f4d7bba877329ee6df969
3f9e20b89497a00b170a32703c02d39a2a4e765e
10649 F20101219_AABQRO seref_o_Page_009.pro
0b8366c821da73e5aae8c847ca733f92
a4dc2b24f2054b781b76cb9400ae20cf97393ad2
116395 F20101219_AABQQZ seref_o_Page_014.jpg
af3b21822f07242e8fdaf62a6f0edb2d
a1d48ee13e2a84ade063c792d9a1cea1d3ad9e4c
116807 F20101219_AABQSD seref_o_Page_017.jpg
09d0cb74ef7dc70e9a6a2a6234f7ea6f
2062ac2d9b1b4c43737dcee7aad6df23c56546a2
28170 F20101219_AABQSE seref_o_Page_100.QC.jpg
e722bb954c1a8de597b154434af3a707
35d865d5dfde43c4e528b01d2545e5cee65b4c29
2230 F20101219_AABQRP seref_o_Page_010.txt
de96936d290bd432ad642c020ab0a4e4
cfbe3630ef9561858c4678f1d844d132d9c8af17
31817 F20101219_AABQSF seref_o_Page_088.pro
cf34edd21819f5846ba9d80fb0fc7d84
9bdc05a08f1feb8c8b4b9050f4343e83ba8d4ffb
1073 F20101219_AABQRQ seref_o_Page_054.txt
c81053ca720ff8f22175a4334584d329
e4964c57aac8b09cc4d8dbef974cfba36194938f
20670 F20101219_AABQSG seref_o_Page_037.QC.jpg
491fe9b3c6fdf81448b518832765fdf9
345e676c79e5d0da0c6f4676783fd2b30c87d179
19053 F20101219_AABQRR seref_o_Page_042.QC.jpg
2f4937c7ceabc6f6c037241c862d9f1c
ef2966f8230f5c77b0b793fec5e23030862f533d
48917 F20101219_AABQSH seref_o_Page_049.jpg
0127d73441af45a09f3707e197934e92
404ab365dc4f76ee2956f7dd3238c25d64fce998
53949 F20101219_AABQRS seref_o_Page_085.pro
6491d01c95b81b57f6ae7cb49ce08ad7
6eef54140d4adc6db022c706a7c7937a233ca384
111087 F20101219_AABQSI seref_o_Page_096.jpg
c2b184ec7875c89f701a178e16442beb
c7e25021be397eeb6a8343307a9c230591c2b818
970415 F20101219_AABQRT seref_o_Page_030.jp2
bb46a9e4f5a686eff3c6574302167c9c
20878bdf917540cc5302e1b0a600c6aedf0c4577
827 F20101219_AABQSJ seref_o_Page_048.txt
7b369730d0583bfbf6c86b97df50d08e
fcea0ee5e9014b23d650f0f12c7f388f30f20d3d
56063 F20101219_AABQRU seref_o_Page_008.pro
8559ebe535efec2b1af4f5372f17ec99
7b34e3ad9ef0dfa3cd7929bab169e8ee0aad0dfb
447 F20101219_AABQSK seref_o_Page_009.txt
2e7e5d193a02364d2500be0d9db9dcde
3d040c224b4238e1044c673e2abbb170d52793dd
2235 F20101219_AABQRV seref_o_Page_026.txt
d02a89c86c6f76e8e850e6212fe96095
ca6f8cff088ee42ee1a03009a7c9169c759c5d49
66998 F20101219_AABQSL seref_o_Page_088.jpg
366ca690d37a4f9da74af45b31674d4a
89b247940efd33ace721556b232c7bfcaf3ddc6e
667674 F20101219_AABQRW seref_o_Page_066.jp2
47f1d09e6fcffdf574a227fc587bb7ab
cddd13f5ea4776e26a73c18184848e74748f6ac9
61944 F20101219_AABQSM seref_o_Page_037.jpg
60f8c217e1f27231d85a9000f788fb7d
2911d37414a6dc8404e965dce2f5b67a0fe1922f
7837 F20101219_AABQRX seref_o_Page_092thm.jpg
d1709cac28fed3820c5c4d34aa3c2747
2e5a3342bc15a89b90f3f5795e35e9289db0a6fd
35124 F20101219_AABQTA seref_o_Page_112.QC.jpg
969ae42ef448a2de8b3206288a6d1097
c2b1fd863723e7d34f5ad6a959c3688b0e5f6c4c
1051983 F20101219_AABQSN seref_o_Page_096.jp2
a5a888c8d21e8f172789f7f37f09dc54
498daa62e73d41590dae390cf408972d95b32ab8
109961 F20101219_AABQRY seref_o_Page_095.jpg
741e4da30a9c4fd767a2a351a3d3205f
bc6fe9d97b780540ea1f133abab92eb9f85616ea
449 F20101219_AABQTB seref_o_Page_001.txt
e650ff77e616b59b287e73dbf547f419
d19d03131e69fea7d73bbefedf85e3a7f20cb4dc
5038 F20101219_AABQSO seref_o_Page_032thm.jpg
e1d57af6df0472f742d251528e9a4a75
3ae6d73f049b5323707906f9b1872016774bee71
19957 F20101219_AABQRZ seref_o_Page_038.QC.jpg
d29df2721e8313b7b4f088312b07b3f4
132fb5c83c97267761e06e5bc122f02523fddda1
31023 F20101219_AABQTC seref_o_Page_066.pro
00c88d58ce980e3597c7ebcc43e26eec
be3d7c8c5d05683c4713f102f6e24c461fcb55a0
F20101219_AABQSP seref_o_Page_058.tif
f80af336ae25ad66a598f28caea7ddb6
3f288c058804f9c3fbdc4c1947b225a2dd476529
30621 F20101219_AABQTD seref_o_Page_101.QC.jpg
004fe3d4df6550eebb47ceefde85d1d9
62c6a513a65dcb83efdf5ded3fc1f77ea82528e4
518 F20101219_AABQTE seref_o_Page_043.txt
d76d31f9ea1598b66463ec9d038b75b6
35268c240cd523eef8a9b20c42279879b6605891
409 F20101219_AABQSQ seref_o_Page_078.txt
ad58ae09df61400a5b669e535ad668dc
f767ec4f8cacee36780cebab3eef72df1565e490
7663 F20101219_AABQTF seref_o_Page_031thm.jpg
5c7bf7f5f965af5aa141e81fc432a511
a65d411e3e4570384a667b728777d72e7eae16c0
F20101219_AABQSR seref_o_Page_066.tif
7463c64283d149598615fccd9af8ccb9
90afaaf3453d44dcfad885fa6c089af282872ee1
58660 F20101219_AABQTG seref_o_Page_014.pro
c126011588e6e8ed8b9edf2d9674a07c
581413b12a841916780bd7bbf4f679eaaac8704d
261 F20101219_AABQSS seref_o_Page_011.txt
1828465aac1410bef747380a548cdfdf
0cb92107b3f427ddb3df58482868fd9cb4618ffc
F20101219_AABQTH seref_o_Page_104.tif
55149df5cd99fcc4d852c8aaa474484f
d0ce7e668556369a2c0d3651383e1716a47962f0
55753 F20101219_AABQST seref_o_Page_099.pro
05ddd6c3f268be4e1789956b55b4587d
7fbfc225f612f8c0cf4da6a844928e2adb35a8bb
54271 F20101219_AABQTI seref_o_Page_026.pro
5f83c71f8a75890b0268f5982c7f490f
362f13a3b8e32ef40534e14f98219d3be3c3d5bc
19538 F20101219_AABQSU seref_o_Page_058.pro
3d1f3e2570877389330cf86e8d53c56f
ed8afaa104fe9c0c7c779465587cffb17cde658f
1021647 F20101219_AABQTJ seref_o_Page_101.jp2
a35961ddea83eada0f5741cf703dea93
99d6e963ec6fd173e13892fee0bad10f8a31e2c4
25969 F20101219_AABQSV seref_o_Page_001.jpg
7e3a9d21344eca9f20b6be07f1f8c89d
598a07d4a5128d2c8a4321e3830e3cbb6b3fcd41
173513 F20101219_AABQTK UFE0017560_00001.xml
0d31578eeedc651c8e22ee4bb433f3c6
4d136ebeebc16cbdf71cb236fe9b7351c3b099cf
54591 F20101219_AABQSW seref_o_Page_041.jpg
546e09c750ac57577cad559560cec53c
851a582a58e3215b29ca797d833faed1de8e1958
55612 F20101219_AABQSX seref_o_Page_110.pro
0de9cebb04954f583fd60f865c543d60
061297f42d6ce3430fd6b7eb714a1f8eb2fcee21
59676 F20101219_AABQUA seref_o_Page_032.jpg
8223c9c7ad55c12be662ea564d1ca290
72c827651fa7c0e8df046f87768e740149f75cc3
F20101219_AABQSY seref_o_Page_059.tif
b1c53c3f2f6a11e5fd68b008ae26d72e
dcce5c542c066940fd10b413fc2af3dee3bcda97
55789 F20101219_AABQUB seref_o_Page_033.jpg
a33d7e81c85ea1cb66b013d76469449b
04032b539991f510202839735761f3c48d81f9d7
4187 F20101219_AABQTN seref_o_Page_002.jpg
875110a51cdbefdf7e858f102fb5aefd
9d5121b3b2df5cdbfb526a01eb2c864cee9c9181
109159 F20101219_AABQSZ seref_o_Page_099.jpg
9410c1e7e4eca3572660b9a5bf894671
3bff9524badfb796ccfae7f25a365fe037398e45
63740 F20101219_AABQUC seref_o_Page_034.jpg
9d75aed1a27a6cb5f0f55d6aea05a909
230400d2c4cdb003f660df76e43f358ced4af1ae
4803 F20101219_AABQTO seref_o_Page_003.jpg
45e9e6ed1606b137c751e32d28255935
bac76e5a7168fa693bdd4080cef9c80f1d33b801
62986 F20101219_AABQUD seref_o_Page_035.jpg
02248ed9bcff71fb221da9ac05e98f38
af470ce5661ebca258ee68ad62fd87bb25a4df9f
79003 F20101219_AABQTP seref_o_Page_004.jpg
e8a10c65c90c945beef4420baa970d59
dbdc076beb8472dbdfa035b9fe0317262fb9a036
66480 F20101219_AABQUE seref_o_Page_036.jpg
df01d7a513002c0e06e86947f0fdc9ee
d06aa1c73640b712fb3901ef6c8ecf4613f14b10
55757 F20101219_AABQTQ seref_o_Page_006.jpg
b371652cf340b3a6262b5ec9ba45d503
31f0a67aedc606d3e7153e8e54594769e7c268c5
61081 F20101219_AABQUF seref_o_Page_038.jpg
2efef3cf3e8f44bde2913d281fe66b2a
6747bcbec720de496ef98fba3f315b8d9b91fad6
81448 F20101219_AABQUG seref_o_Page_039.jpg
f399c24cd034ff97bacfa788ecfbb1ca
b4199c79692ce936b561737d7025136646da77e1
26450 F20101219_AABQTR seref_o_Page_009.jpg
fb1eea78649eb332597e98832f936f86
4e3c003aaeeb0e690f1cab0a579ab66f909e9c6c
21483 F20101219_AABRAA seref_o_Page_050.pro
e3afd55deb2dcee09a3bf7b66727dfb6
0c67bd88064066d161bbca7c31d56e32364bf6b6
60018 F20101219_AABQUH seref_o_Page_042.jpg
7a3462e43cf5b3f38f6ee16e47475c5d
8e36dbf8174799deffe99ce2ff4eb84a7dae33d5
13475 F20101219_AABQTS seref_o_Page_011.jpg
35a9fc8014d1b123bb4009c80e7f3d96
6428323b53798419ce2ea02a290926b5286b9dd5
26445 F20101219_AABRAB seref_o_Page_052.pro
0ba0b400837527394acd4efa137f31fa
14a8a5a14e1fad049fbb0f94293f30ebc9311416
115186 F20101219_AABQUI seref_o_Page_046.jpg
faeddb96d86de8473bd0ffb7bfae7c57
bc9570208be4a5f3195ca229b9fecae0ff26c6eb
109225 F20101219_AABQTT seref_o_Page_016.jpg
87a70bd36f9c21456de4440a50f68ae3
7adf0d23d2d7a14c0f271ad79229235feba78673
24752 F20101219_AABRAC seref_o_Page_054.pro
139204f69b561f19ea2a626974a6e3f7
765bbfeeb577f2100698bdf78c3da716ff182598
103566 F20101219_AABQUJ seref_o_Page_047.jpg
9c755bff56fe14f229f5c33998a8d028
2d0e377eefafe4d70a9f5c3a245af077ac71f730
107343 F20101219_AABQTU seref_o_Page_024.jpg
a8358085cabcf2b2d3beb8fafbdcf226
d671d6f090ed732906e38bcca9912541c92cd136
42879 F20101219_AABRAD seref_o_Page_056.pro
761f33d74fb1871667de21cb8925c6af
539326d22ede1ee1c7d23c4f77764db05821388e
65501 F20101219_AABQUK seref_o_Page_048.jpg
2880a5b6b2e1b3bdc706f2b87af7a273
2628c64b0fccaa614a931711209b73e659f04398
90536 F20101219_AABQTV seref_o_Page_025.jpg
2e69d9270a8a31c1c5a05fa5f5dbeabb
83aea5d6c0d09a1eb100288787770eb5537a6599
22300 F20101219_AABRAE seref_o_Page_059.pro
0adc43c544caa1bdcc9995a5047f4321
ee6d3952101bf76b3498a939e76bec308dcd9183
56653 F20101219_AABQUL seref_o_Page_051.jpg
0be32f660cd85da73b70e6426e721588
f8c14804de57749f6d849c26040964585ef0d3a2
67646 F20101219_AABQTW seref_o_Page_028.jpg
a2fed3c393cd0f98d97f4f0e809e1805
4ab75a8017516aebec346712e303ae7223b99fbe
26943 F20101219_AABRAF seref_o_Page_060.pro
278a914635b8866ae74273b16a8a5cac
b66dde55c75b43d6f6861f40269e5827a5634de5
81936 F20101219_AABQVA seref_o_Page_087.jpg
2dcc523cbf21910098a554234445125b
5c1f44ac513c9bb8439aca7c8218d6bdb84a8378
54322 F20101219_AABQUM seref_o_Page_052.jpg
7bcc6bbe2134b4516f3af02a91a43e7a
ac85b4caf22bf44e4db45c0f392475358c2c25af
72252 F20101219_AABQTX seref_o_Page_029.jpg
af389984bc59427edb0844575374c47f
abbb7a5ba15c3832be9b83e7ce815c2a8baf393f
22531 F20101219_AABRAG seref_o_Page_061.pro
ed7c1a32d751dd69b534fbd9d1ec9824
95bc4d13811f436dca81f53cc0c4c7e407260165
72148 F20101219_AABQVB seref_o_Page_090.jpg
f77704110966ac85a1975b0a64fa562b
d304dc0e4d8f1da55a1cb7859947838248a8c8fa
60742 F20101219_AABQUN seref_o_Page_057.jpg
0aca938f53d668b6e41641574df5b725
dfb466a2c45259c0e26c8e5d82650550ad54e29a
88288 F20101219_AABQTY seref_o_Page_030.jpg
09b6690fb96dce279e594aaefe5fce46
4b382fdf0991f9c46f350291f910b753afb324a7
38528 F20101219_AABRAH seref_o_Page_062.pro
888a911dca58c40ba42ba6e7b54e5cb9
292013b600332b59703ee47b61b4a832ba1be9bb
107155 F20101219_AABQVC seref_o_Page_094.jpg
d9eb3e12c62dadf89317ea6a8439567a
8bd95526d700d1c3a4fa8175b04968284140ffd2
52768 F20101219_AABQUO seref_o_Page_059.jpg
d90bf75ca9ec2962d47e6c4be887b34f
c5e14360748e913f3267d74a9c1283d30ac9095b
102690 F20101219_AABQTZ seref_o_Page_031.jpg
ccf54c2be8f7f6f45f15fa21790b58a9
1a71f1bb76475069d93901d28fdcb5c1f9091d04
58420 F20101219_AABRAI seref_o_Page_067.pro
932686b2d048df9d326d5c9844f35092
8bf20b0450ee8a187aa8ff75f6b993537cff2009
81630 F20101219_AABQVD seref_o_Page_097.jpg
28b8d3c0af2c78e9120a8f1cd29d47ae
fa5bfe49078330e7b24d3bb7a9788cf156f16f99
56544 F20101219_AABQUP seref_o_Page_060.jpg
521af1676ca7d52d8ce6cca19d99cc40
6b199601bfae1c9a882627f3e9cbb69f93ee16c7
57824 F20101219_AABRAJ seref_o_Page_068.pro
0ed6df63a8d2ba26ba17fd169f528ab2
0f7e0483121d45adae30dacbed6d165c7d796b9a
111141 F20101219_AABQVE seref_o_Page_102.jpg
3f1b4ee885146af545429c566fe0c053
aa528610373abed04a5dd56ea52f5780e7666447
79302 F20101219_AABQUQ seref_o_Page_062.jpg
36b28b5ce6884377b7a8431d3cfb6956
531ed30dfae6e7d82479b1dbca75dd2aa0936d80
50102 F20101219_AABRAK seref_o_Page_073.pro
75cd70c710bdaf8c0bff4a77e3ff370c
e6f9136e9d328c932f4784bebedd7b4b1feb04e5
25714 F20101219_AABQUR seref_o_Page_063.jpg
9333dc7ecc7f2dcb5146e7e5cf7c7552
1083b5d355ce7bef158d8602d9820d66cb92fc34
7524 F20101219_AABRAL seref_o_Page_081.pro
aeef3a1eade97b74c6c00dac746ae830
df2debe0389db87d2f339c923a315b2ea0d8cac9
104320 F20101219_AABQVF seref_o_Page_103.jpg
ebe77434e207ef6ea3220995126f611d
2147770f4b30e3a223ce2b881558ffb391c75cca
62617 F20101219_AABRBA seref_o_Page_113.pro
9b7fb18b3d31295eae61a2810b195443
86b6fa590b0f1a32d8b43c2990f4e1949250a58a
47514 F20101219_AABRAM seref_o_Page_086.pro
bc7fa3cd5094cb6481a01c3a9a3b73be
54b7caa7bb6b12bde24d02c6ae2f5f1ba0e9d3e9
90360 F20101219_AABQVG seref_o_Page_104.jpg
ee506eddb32b5bdde521d19e21a47735
7f1146b59d893408c9d55c5c415facbf2f9ebd03
112549 F20101219_AABQUS seref_o_Page_067.jpg
18f032dfb967eb527982493f7201c874
c39f13ffc5a5341e358708a09a2e527bba1c89fe
59128 F20101219_AABRBB seref_o_Page_114.pro
6c55b32c199ef3aba68e0fe5f6fa44fb
683a1f57136f8a6fb75172479c5a0da7693d6b8f
34291 F20101219_AABRAN seref_o_Page_090.pro
e2b088f09dc0888876f104e643118938
03c51223013ab0ffedfc46af79cefcb14ff9e8c7
55704 F20101219_AABQVH seref_o_Page_106.jpg
d850e924373a569c0354b209806861b7
8497b715ed25e00d544ecc737e97edbe78ecfb1f
112793 F20101219_AABQUT seref_o_Page_068.jpg
8e9a6f098bab4f5a5b74211e7e5e9003
67f287380fc9a2fdda3067f258935aa409d1b063
103 F20101219_AABRBC seref_o_Page_002.txt
2c5c6d778395b4c7708aede9a603d129
97bb08e8e30d8503e5b101e12e3c5eeddfa74ed7
34558 F20101219_AABRAO seref_o_Page_093.pro
a97d08a21fdf789eb2dd599dc5c94c48
1161039d31e754c95fe0ac6c5aa6b17e37645c81
112324 F20101219_AABQVI seref_o_Page_107.jpg
9077f9eec6e438ad158aeb50c9cc40d7
ac7515b8843d2e251af6298b5011ef802e0f04a2
98357 F20101219_AABQUU seref_o_Page_073.jpg
ec5f9ecad9cd0e333e7b2f2b1ea3e383
8560c19fbde204526989fd7633beb729ab150bc5
118 F20101219_AABRBD seref_o_Page_003.txt
2396793f184e8f329fc7a5c3841225cb
efc47a38b63f08bb99f668d0c169ca78b4295e7f
45192 F20101219_AABRAP seref_o_Page_094.pro
a0aef5232ef2ee41ceeaf282d0924635
2ce1bc6b322428443f8353d84044ba2672a13b8f
110024 F20101219_AABQVJ seref_o_Page_108.jpg
baeabd7a2b662fcd640beef6776dfd9b
539f613e0d23c04525b10389bb4d58c0f1c16b57
113388 F20101219_AABQUV seref_o_Page_074.jpg
853539a577f968a00f4d3f95fb22617b
9a45f6989668275e8c089c418d9663069dd46125
1586 F20101219_AABRBE seref_o_Page_004.txt
35069b50018e1161020a7da8866de061
21b3e603c6b99060299925227c97dce263fe414b
41855 F20101219_AABRAQ seref_o_Page_097.pro
c6a02e5d6da2cf877dcefdea59fe8489
ed3d13bf888011321f8d9972d2abca7c140ff1e0
116839 F20101219_AABQVK seref_o_Page_109.jpg
4758123a175747e0f9f9704a216a7cee
3d4ec3e42fba913d9d1ca0dbcebaeb6e3619b02f
116577 F20101219_AABQUW seref_o_Page_076.jpg
fa645fcfc05509e9befc75245ccf8a56
bf4997ef493f0856189def9978ed5fe1c4a8c2c6
1155 F20101219_AABRBF seref_o_Page_006.txt
6d565bebaadac53bdac7b99e2e430438
00bdb754d4e8ae0435e08a6459eb761736c81bde
43047 F20101219_AABRAR seref_o_Page_100.pro
bb86608491935f4d7e0888b6cadf6a8e
357c9dd95c72f39c43016f6f84f6b0941c51b561
107917 F20101219_AABQVL seref_o_Page_110.jpg
d8fef75ff94827030dc7f33422a9d4a4
870b9ad6918fd799da722814f560558112cc0788
57238 F20101219_AABQUX seref_o_Page_079.jpg
8db1ca4f4b22a0f5ceb9113e16d45d77
1ed2e0df586e4163351adaac4531ae2fe99b27a1
F20101219_AABRBG seref_o_Page_007.txt
6b3d450c013c5ef4961501dbe9af3be3
2afeb555a64fd1f21030cb29138798bf4a2a3328
1051962 F20101219_AABQWA seref_o_Page_031.jp2
dd112fefead49b71c974f21b0f081b5a
e1b2c181d7cccd6daa3f423a44c622526104709c
41350 F20101219_AABRAS seref_o_Page_101.pro
077358c19fa7a2494ee8549ff3c33d1d
d74e4c57c747d1085a9e1f8f12672fcd9b5a8eeb
112930 F20101219_AABQVM seref_o_Page_115.jpg
b85f0261cbbf284150e211ba63dd1d9c
2fbdd0b80d5b30d9fa237cbc282fd621b5c377d8
63601 F20101219_AABQUY seref_o_Page_081.jpg
466280509ce5999e68e13806854b391f
c917e5a8f206eb94d5f43679e77ae040329da3ae
F20101219_AABRBH seref_o_Page_008.txt
6171d4eb4853d8cf90f4f32bd11fd698
5958c96f44858096eb48d7c3b89c70b99eb9e039
664575 F20101219_AABQWB seref_o_Page_037.jp2
adebcdc395e05ae416afafb35f341bf7
ecdaad12ddbdb3ea3ed1adf78bf2e268493c5ef7
58135 F20101219_AABRAT seref_o_Page_102.pro
83824307bc372fe8808e0fcf4c4c6c4f
893b3645c67ed65969091132992e1d9f24129ed8
34739 F20101219_AABQVN seref_o_Page_116.jpg
98380998bbc67be31ed3bcb60f78f11e
4dcbbc00fbf69e88653b64ab4578890988bedc3f
57604 F20101219_AABQUZ seref_o_Page_083.jpg
fa89d7f4c57c1d6c88a11c87dca303d8
16f1c590bef1587ec4307d7b1ae1dede622d6577
2376 F20101219_AABRBI seref_o_Page_012.txt
fa273dc6d9ff8214ccbf12e4047bb29a
a12c366c45db551204279e4b1b17ce3bcfed867c
848992 F20101219_AABQWC seref_o_Page_039.jp2
d3789befc16f4de9b37f8079b40b7c19
81f6f99686c145e064053e2be569f3b4345fec87
41134 F20101219_AABRAU seref_o_Page_104.pro
3988c2567b47ecfe652f38648d8db806
6f6f4ef3cef60bf5e43dfb0a7b24dd7dc8c90c5e
6017 F20101219_AABQVO seref_o_Page_003.jp2
28692aed9bdea1356bdda389c7747083
4a6e25626d0bcf667ce134564c1d9101bdbc49be
2327 F20101219_AABRBJ seref_o_Page_014.txt
e57b9ad39f004246978df3221ed35649
69070b301de94eb7b5f0f5a24928595b70c4c98b
929065 F20101219_AABQWD seref_o_Page_040.jp2
08252d2d6c348598d657ef8a645addec
22b19fa4635280cbfff8656d701b5559692f31db
21735 F20101219_AABRAV seref_o_Page_106.pro
c6e4af26f0a9603062faa841093b052b
8e10469c0c267a7f0211a49488fa625f7357e9c6
1051923 F20101219_AABQVP seref_o_Page_005.jp2
943e105acb90c4e07943a6ba9c413b8b
e4d71025c47f771d351071cc27de6036f51f32c8
2307 F20101219_AABRBK seref_o_Page_015.txt
74a2e800ed066a3030d1012219e9a3eb
d78ea9f2fb38729dc2dee4d5c8c7279948aa2b65
585563 F20101219_AABQWE seref_o_Page_041.jp2
0398ae0eaea7728e0e6f0e9b60377c82
5ab488740e2bb6f53dd6fceca43b64d6c38e7294
59417 F20101219_AABRAW seref_o_Page_107.pro
9410cdf72d2a5385417e701124c8e415
684a461c8da71cb0355134da043688924aceaf44
109709 F20101219_AABQVQ seref_o_Page_010.jp2
9bcba6c0d87613b70d3d67a11a7a0209
1bd4d925eb6b449fd08ec237be9061ee8ed88b07
2390 F20101219_AABRBL seref_o_Page_017.txt
4909f9ad225f1c4b4a0b5ebc382c13e6
33b048aa7f8a51019da6ae9ea4e126d2a29ff212
187723 F20101219_AABQWF seref_o_Page_043.jp2
09d2cf3233df541ec6a941acd234388d
2ef2031dfa6a0b525899b0d0bb1f468077f6581d
57986 F20101219_AABRAX seref_o_Page_108.pro
e8e58b27d9824f21105265dec2b0f916
d9b9e5811081d21807c108f0f2e4bcddc9b6e86e
17038 F20101219_AABQVR seref_o_Page_011.jp2
f0170a9255ff86e5a5f65a36170a2ca4
606dd82f4f7f679239489d8e43735779f2f48b98



PAGE 4

First,IwouldliketothankmysupervisorycommitteechairDr.PanosPardalosforhiscontinuousencouragementandguidance.Dr.Pardalosprovidedmanyopportunitiesthatallowedmetobecomeaself-condentandindependentacademician.IwouldliketoextendmygratitudetomyothercommitteemembersDr.RavindraAhuja,Dr.EdwinRomeijnandDr.TamerKahvecifortheirvaluablefeedback.IwouldliketoincludeallIndustrialandSystemsEngineering(ISE)faculty,especiallyDr.SuleymanTufekciandDr.ColeSmithfortheirhelp.IamverythankfulforthegreatcontributionfromDr.MingzhouDingandotherresearchersfromhislabintheBiomedicalEngineeringDepartment.IwouldliketothankallmycolleaguesintheCenterforAppliedOptimizationLabfortheirsupport,especiallyErhunKundakciogluforhisextrahelp.MybelovedwifeMichellehasbeenmybiggestsupportwithhercaringloveandmydailyinspiration,towhomIowemyeternalgratitude.IwanttothankmymotherEsma,mysisterEzgiandmybrotherKvancwhoseloveandencouragementbroughtmetoGainesvilleandhelpedmegetmyPh.D.IalsowouldliketothankMichelle'sfamily,Magdi,RoblynandTimothy,fortheirsupportandlove.IreservemymostspecialappreciationformybelovedfatherAbdullahSeref,whorecentlypassedaway.Hishonesty,humility,philosophyandhispassionforsciencehavealwaysguidedmeandalwayswill. 4

PAGE 5

page ACKNOWLEDGEMENTS ................................ 4 LISTOFTABLES ..................................... 7 LISTOFFIGURES .................................... 8 ABSTRACT ........................................ 10 CHAPTER 1INTRODUCTION .................................. 12 1.1Kernel-BasedMachineLearning ........................ 14 1.2ApplicationstoNeuroscience .......................... 15 1.3RegularizedGeneralizedEigenvalueClassiers ................ 17 2KERNEL-BASEDLEARNINGMETHODS .................... 20 2.1Kernels ...................................... 20 2.2GeneralizationTheory ............................. 24 2.3OptimizationTheory .............................. 26 2.4SupportVectorMachines ............................ 30 2.4.1MaximalMarginClassier ....................... 31 2.4.2SoftMarginClassier .......................... 34 2.5GradientBasedAdaptiveScaling ....................... 39 2.6SupportVectorRegression ........................... 40 2.6.1Quadratic"-SensitiveLoss ....................... 40 2.6.2Linear"-SensitiveLoss ......................... 42 3SELECTIVEKERNEL-BASEDMETHODS .................... 44 3.1SelectiveSupportVectorMachines ...................... 44 3.1.1Selective2-NormSupportVectorMachines .............. 48 3.1.2Selective1-normSupportVectorMachines .............. 51 3.1.3SelectingPatternVectorswithLargeMargin ............. 53 3.2SelectiveSupportVectorRegression ...................... 54 3.2.1Selective2-norm"-InsensitiveRegression ............... 57 3.2.2Selective1-norm"-InsensitiveRegression ............... 60 4KERNELMETHODSAPPLIEDTONEURALDATA .............. 64 4.1VisualDiscriminationTask ........................... 64 4.2Methods ..................................... 67 4.3ClassicationandChannelSelectionResults ................. 67 4.4TimeSeriesAlignmentwithDynamicTimeWarping ............ 72 4.5SelectiveSupportVectorMachinesonNeuralData ............. 74 5

PAGE 6

.................. 85 5.1GeneralizedEigenvalueClassiers ....................... 85 5.2ANewRegularizedFastClassicationMethod ................ 87 5.2.1TheNewRegularizationMethod .................... 88 5.2.2ComputationalResults ......................... 91 5.3AParallelImplementationoftheFastClassicationMethod ........ 94 5.3.1ImplementationDetails ......................... 95 5.3.2ComputationalResults ......................... 96 5.4AnIncrementalClassicationAlgorithm ................... 98 5.4.1IncrementalSubsetSelectionAlgorithm ................ 99 5.4.2InitialPointsSelection ......................... 101 5.4.3ComputationalResults ......................... 104 6CONCLUSION .................................... 107 REFERENCES ....................................... 111 BIOGRAPHICALSKETCH ................................ 116 6

PAGE 7

Table page 4-1Dierentstagesofthevisuomotortask ....................... 68 4-2ChannelnamesandlocationsforTI,LUandGE ................. 69 5-1Classicationaccuracyusinglinearkernel. ..................... 92 5-2Classicationaccuracyusinggaussiankernel. ................... 93 5-3Elapsedtimeinsecondsusinglinearkernel. .................... 93 5-4Elapsedtimeinsecondsusinggaussiankernel. ................... 93 5-5Executiontimesfortheparallelimplementation .................. 97 5-6Eciencyfortheparallelimplementation ...................... 97 5-7Datasetscharacteristics ................................ 105 5-8ClassicationaccuracyforReGEC,I-ReGECandSVM .............. 106 5-9IncrementaldatasetusingI-ReGECandpercentageofthetrainingset ..... 106 7

PAGE 8

Figure page 2-1Kernelmappingofcirculardatain2-Dto3-D. ................... 21 2-2Maximalmarginclassier .............................. 32 2-3Softmarginclassier ................................. 35 2-4StandardSVRand"-Insensitivetubearoundtheregressionfunction ....... 40 3-1Setofpatternvectors,3patternvectorsineachset ................ 45 3-2Classicationofaseparablesetinthepresenceofrestrictedfreeslack ...... 47 3-3Distributionofpooledfreeslack(PFS) ....................... 48 3-4Distributionoffreeslackperset(FSS) ....................... 48 3-5StandardSVMonaseparable3Dexamplefordierentkernelbandwiths .... 54 3-6SelectiveSVMonaseparable3Dexample:boldpointsareselected ....... 55 3-7SelectiveSVRcomparedtoanavemethodfordierentbandwidths ....... 63 4-1Approximateplacementofelectrodesinthetreemonkeys ............. 65 4-2Visualstimuliusedintheexperiment ........................ 66 4-3Experimentalparadigm ............................... 66 4-4ClassicationandchannelselectionresultsforGE ................. 77 4-5ClassicationandchannelselectionresultsforLU ................. 78 4-6ClassicationandchannelselectionresultsforTI ................. 79 4-7Brainsnapshots .................................... 80 4-8a)Uncodedscoresmatrixb)Sortedscoresc)Codedmatrixd)Sortedmatrix 81 4-9a)comparativeclassicationresultsb)comparativechannelselectionresults ... 82 4-10Improvementclassicationforline-diamond(left)andgo-nogo(right) ...... 83 4-11Improvementinthefeatureselectionforline-diamond(left)andgo-nogo(right) 83 4-12Baselineapproachtoemphasizeimprovement(top),dierencebetweenactualimprovementandbaseline(bottom) ......................... 84 5-1Separationobtainedwithgeneralizedeigenvectors. ................. 87 5-2ReGECalgorithm ................................... 91 8

PAGE 9

....... 94 5-4ClassicationsurfacesproducedbyReGEC(left)andI-ReGEC(right) ..... 101 5-5ClassicationconsistencyofI-ReGECforrandomselection(left)andk-menas(right) ......................................... 103 5-6PerformanceofI-ReGECwithrespecttothenumberofstartingpoints ..... 104 9

PAGE 10

Inthisstudy,newoptimizationmethodsareintroducedonkernel-basedmachinelearning.Thesenovelmethodssolvereallifeclassicationproblems,especiallythosearisinginthebiomedicalarea.Therstmaincontributionofthisstudyistheselectivesupportvectormachine(SelSVM)classiers.SelSVMclassiersaremotivatedbythenoisytemporalvariationsintherecordingsofrepeatedcognitiveprocesses,whichaecttheperformanceofstandardsupportvectormachine(SVM)classiers.IntheSelSVMclassicationproblemtherearesetsofpossiblepatternvectorsinsteadofindividualpatternvectors.SelSVMclassiersselectthosepatternvectorsfromeachsetthatwouldmaximizethemarginbetweenthetwoclassesofselectedpatternvectors.SelSVMiscomparedwithotherstandardalignmentmethodsonaneuraldatasetthatisusedforanalyzingtheintegrationofvisualandmotorcortexesintheprimatebrain.Selectivekernel-basedmethodsarethenfurtherextendedtoselectivesupportvectorregression(SelSVR). Thesecondmaincontributionofthisstudyisafastclassierbasedonthestandardgeneralizedeigenvalueclassiers(GEC).TheregularizedGEC(ReGEC)usesanewregularizationtechniquewhichreducesthesolutionoftwoeigenvalueproblemsintheoriginalGECtoasingleeigenvalueproblem.AparallelimplementationofReGECisdevelopedtostudylargescalegenomicproblems.Finally,anincrementalversionI-ReGECisdevelopedtotrainlargeamountsofdataeciently.I-ReGECincrementallybuildsa 10

PAGE 11

11

PAGE 12

Learningisonethemostdistinctiveinnateabilityofanimals.Acombinationofexperiencesasinputforthelearningprocessincreasesthechanceofsurvival.Theseexperiencesintroducemoresamplesofsimilarconditionsfromwhichtoavoid,suchasthepresenceofapredator,orthosetoseek,suchasndingamate.Humans,themostadvancedlearnersamonganimals,usetheirknowledgetocreateabstractmodelsforthoseconditionstheyhavenotexperienced,butcanrespondappropriatelyspecictoanobjective.Formallyspeaking,thisabstractioncanbeperceivedasasetofruleswhichcanmapaninputconditionintoasetofpossibleresponses.Intelligencecanbedenedastheinteractionofallsuchrulesthatconstitutecomplexcognitivetaskssuchasreasoning,planning,problemsolving,andcreatingfurtherrulesformorecomplexscenarioswithadynamicandrecursivenature. Inaworld,wheremachinesdomostofthehardwork,eventually,thequestionofmakingthemintelligentcameintopicture.Theinitialtheoreticalconceptswereintroducedinthersthalfofthetwentiethcentury,andimplementationsthrivedwiththeintroductionofcomputers,inparallelwiththedevelopmentofformalsystemsinlogicandneuralnetworks.Thetermarticialintelligencewascoinedbythebeginningofthesecondhalfofthecentury.Oneoftherstexamplesofarticialintelligencewereimplementationsofachess-playingprogramdevelopedin1951.Deep-Blue,asuccessorofsuchprogramsrunningonpowerfulcomputersbeattheworldchesschampionGarryKasparov46yearslater,in1997.ThereweremajoraccomplishmentsinmachinelearningtowardstheendofthecenturywithcarsequippedwithAIsystemswhichcandriveforthousandsofmilesintracwithoutadriver. OneofthemajorareasinAIismachinelearning,whichdevelopedsubstantiallyduring1990sand2000s,mostlyasaresultoftheintroductionofprobabilitytheoryandstatisticalmethodsinAI.Machinelearningcanbedenedasthemethodsdevelopedtoenablecomputerstolearn.Therehavebeenverysuccessfulimplementationsrangingfrom 12

PAGE 13

Machinelearningwithintheclassicationframeworkcanbecategorizedintotwomainclasses.Supervisedlearningreferstothecapabilityofasystemtolearnfromasetofexamples,whichisasetofinput/outputpairs,wheretheinputisusuallyavectoroffeaturesofanobject,andtheoutputisthelabelfortheclassthisobjectbelongsto.Asetofobjectswithafeaturevectorandaclasslabeliscalledatrainingset.Thissetisusedtoderiveclassicationfunctions.Thetrainedsystemiscapableofpredictingthelabelofanobject.Thetermsupervisedoriginatesfromthefactthatthelabelsfortheobjectsinthetrainingsetareprovidedasinput,andthereforeweredeterminedbyanoutsidesource,whichcanbeconsideredasthesupervisor.Onthecontrary,unsupervisedlearningisthecasewheretheobjectsarenotlabeledwithanyclassinformation,andlearningisaboutformingclassesofobjectsbasedonsimilaritiesbetweentheirfeatures. Supervisedlearningsystemsapplicationscanbefoundinmanyelds.Financialcompaniespreferstoclassifyloanrequestsdependingonthefeaturesthatcharacterizesloaner'sabilitytopayback.Suchfeaturesarelearnedfromthehistoricaldata.AsimilarexampleistheInternalRevenueService'spredictingtaxevadersbasedonthefeaturesoftaxevaderstheydetectedpreviously.Applicationsmaydierincludingcasessuchasapredictionsystemthatmaywarndriversaboutpedestriansattemptingtocrossthestreet.Therearemanyapplicationsinbiologyandmedicinesuchasdetectionofcancerprone 13

PAGE 14

1 ],SupportVectorMachine(SVM)algorithmsarethestate-of-the-artamongtheclassicationmethodsintheliterature.Thesemethodsclassifypatternvectorswhichareassumedtobelongtotwolinearlyseparablesetsfromtwodierentclasses.Theclassicationfunctionisdenedwithahyperplanethatseparatesbothclasses.Althoughthereareinnitelymanyhyperplanesthatseparatethetwoclasses,SVMmethodndsthehyperplanethatmaximizesthethedistancefromtheconvexhullsofbothclassesbysolvingaquadraticconvexoptimizationproblem.ThesuccessandrobustnessofSVMclassiersareduetotheirstrongfundamentalsonthestatisticallearningtheorywhichisbasedongeneralizationboundsforSVMclassiers.Thesemethodscanbeextendedtothenonlinearcasesbyembeddingthedatainanonlinearspaceusingkernelfunctions[ 2 ]. SVMclassiershavebeenoneofthemostsuccessfulmethodsinsupervisedlearningwithapplicationsinawidespectrumofapplicationareas,rangingfrompatternrecognition[ 3 ]andtextcategorization[ 4 ]tobiomedicine[ 5 { 7 ],brain-computerinterface[ 8 9 ],andnancialapplications[ 10 11 ].Thetrainingpartreliesonoptimizationofaquadraticconvexcostfunctionsubjecttolinearconstraints.Quadraticprogramming(QP)isanextensivelystudiedeldofmathematicsandtherearemanygeneralpurposemethodstosolveQPproblemssuchasquasi-newton,primal-dualandinterior-pointmethods.Thegeneralpurposemethodsaresuitableforsmallsizeproblems.Forlargeproblemsfastermethodsarerequired.Thesefastermethodsusuallyinvolvechunking[ 12 ]anddecomposition[ 13 ]techniques,whichusesubsetsofpointstondtheoptimalhyperplane.SVMLight[ 14 ]andLIBSVM[ 15 ]areamongthemostfrequentlyusedimplementationsthatusechunkinganddecompositionmethodseciently. 14

PAGE 15

16 ]thatapproximatethetwoclasseswithtwohyperplanesinsteadofseparatingthem.Thesupportvectormachinemethodhasbeenextendedtoperformregression.Insupportvectorregression(SVR),classlabelsarereplacedwithscalarquantitiesofadependentvariable.Theideaofmaximizingamarginisinvertedbyrequiringthepatternvectorstobewithinaspecieddistancefromtheregressionfunction.SVRmethodwasalsodevelopedbyV.Vapnik[ 17 ],followedbysomeotherecientimplementations[ 18 { 20 ].Thevarietyoftheoptimizationmethodsandtheirapplicationshaveleadtotheevaluationofthefeaturesthatdenetheclassicationofobjects.Featureselectionmethodsweredevelopedinordertondthemostrelevantfeaturesthatcontributetothedierentclassesofelements[ 21 { 24 ]. Oneofthemaincontributionsinmydissertationisthenovelconceptofselectivesupportvectormachines(SelSVM).Thismethodhasnsetsofpatternvectorswithatmostkpatternvectorsineachset,representingnsinglepossiblepatternvectors.Allpatternvectorsinasetsharethesamelabel.ThestandardSVMproblemcanbeconsideredasaspecialcaseofSelSVMwhenk=1.SelSVMpicksanoptimalpointfromeachsetinordertomaximizetheclassicationmargin.Theotherpatternvectorsdonothaveaneectontheclassicationfunction.Thesameideacanbeappliedtosupportvectorregressionwiththeoutcomeofselectedvectorsformingabetterregressioncurvewithlesserror.However,theselectionprocessisahardoptimizationproblem.Therefore,relaxationsofthisproblemarestudiedandshowntoworkverywellontoyproblemsaswellasinreallifeproblems. 15

PAGE 16

Thespatiotemporaldynamicsofthevisualcorticalprocessingandtheinteractionbetweenthevisualsensorycortexandmotorcortexhasbeenextensivelystudiedincognitiveneurophysiology.[ 25 { 31 ].Inrecentstudies,alargenumberofvisualcorticalareashasbeenshowtobeactivatedrapidlyasafeedforwardstepaftertheonsetofthevisualstimuli[ 32 33 ].Thismechanismservesastoprimethemotorsystemtopreparefortheprocessingoftheresponsefollowingthedeterminationofthestimuluscategory.Theventralvisualpathwayisknowntobeinvolvedinthevisualpatterndiscrimination[ 34 35 ],whereasthecategoricaldiscriminationofthestimuliisexpectedtobeperformedinthedorsolateralprefrontalcortex[ 36 { 39 ].Theresponserelatedpotentialsarerecordedinsimilarstudies,forevennoresponsecasesinmonkeys[ 40 41 ]andavisualcategorizationtaskinhumans[ 42 43 ]. Themajorityofthesestudiesareonvisualonsetlatencyfocusingoninformationprocessingwithinthevisualsystemandignoringtheintegrationwiththemotorsystem.Thetechniquesusedintheanalysisofcorticalactivityusuallyinvolvesrstorderstatisticalmeasuresandlinearstatisticalmodels.However,itiswellknownthatthecorticalactivitiesandtheirinteractionshaveahighlynon-linearnature.Furthermore,theubiquityoftheactivitiesinthenervoussystemandthehighlyconnectedstructureofmassiveneuralnetworksinthebrainrequiremethodsthatcanincorporatesimultaneousrecordingsinanappropriatemultidimensionaldomain,ratherthanstudyingindividualrecordingsindependently. 16

PAGE 17

44 ],whichrequireecientcomputationalprocedurestorespondtothedynamicsoflargedatabases.Asmachinelearningbecomesapartofdataintensivecomputationsystems,updatingthelearningsystemsbecomeintractable.Therefore,incrementalmethodsthatrequireminimalcomputationalburdenarepreferred.Forthispurposeseveralmethods,especiallyinkernel-basednonlinearclassication,havebeenproposedtoreducethesizeofthetraining 17

PAGE 18

45 { 49 ].Allofthesemethodsshowthatasignicantdatareductionispossiblewhilemaintainingacomparablelevelofclassicationaccuracy. Thebinaryclassicationproblemcanbeformulatedasageneralizedeigenvalueproblem[ 16 ].ThisformulationdiersfromSVMssince,insteadofndingonehyperplanethatseparatesthetwoclasses,itndstwohyperplanesthatapproximatethetwoclasses.Thepriorstudiesrequirethesolutionoftwodierenteigenvalueproblems.TheRegular-izedGeneralEigenvalueClassier(ReGEC)isintroduced,whichusesanewregularizationtechniquethatrequiresonlyoneeigenvalueproblemtobesolved.Thisclassierreducesthecomputationaltimebyhalfcomparedtothestandardeigenvalueclassiers. Duetothesizeandeciencyproblems,verylargedatabasescouldonlybeprocessedorminedusingagroupofconnectedcomputers(multicomputers)thatruninparallelandcommunicateamongthemselves.Standarddataminingalgorithmsdonotachieveagoodperformanceonmulticomputers,ingeneral.Therefore,specialalgorithmsmustbedesignedinordertoexploittheirstrongcomputationalinfrastructure.Thereareanumberofcomprehensivesurveysonparallelimplementationsofwidelyuseddataminingandknowledgediscoverymethodsandtheirapplicationspectrum[ 50 { 53 ].AparallelimplementationofReGECisintroducedtoperformcomputationallyhardclassicationtasksongenomicdatabases. Anothermajorprobleminclassicationistheamountoftrainingexamplesavailable.Althoughitlooksintuitivethatthemoreexamplealearningsystemshas,thebetterpredictionsitcanmake,thecomputationalburdenoflearningmaymakethesysteminconvenient.Oneimmediatesolutionistoselectasubsetofpointsthatwouldretainthecharacteristicsofthetrainingset.Asecondproblemariseswhenanewtrainingdatabecomesavailablefortraining.Adesirablemethodasasolutiontothesecondproblemshouldecientlyevaluatethecontributionofthenewdatatotheclassicationfunction,ratherthanacompletetrainingoftheincrementallyaugmentedtrainingset.Anewmethod,whichisreferredtoasIncrementalReGEC(I-ReGEC),isintroduced.I-ReGEC 18

PAGE 19

Therestofthedissertationisorganizedasfollows.Chapter2discussesthefundamentaltopicsonkernel-basedlearningsuchasgeneralization,kernels,optimizationtheory,supportvectorclassicationandsupportvectorregression.InChapter3,theseclassicationandregressionproblemsarefurtherdevelopedintoselectiveclassicationandregressionproblemsthatareintendedtondthosepatternvectorswhichprovideabetterseparationoftwoclassesofdata,orvectorsthatdenearegressionfunctionwithlowererror.InChapter4,thestandardsupportvectorclassication,regressionandfeatureselectionmethodsareappliedtoneuraldatafromavisuomotortaskperformedbymacaquemonkeysinordertostudytheintegrationofvisualandmotorsystemsintheprimatebrain.Theresultsarefurtherimprovedbyapplyingselectivesupportvectorclassication.InChapter5,extensionstogeneralizedeigenvalueclassiersarepresentedinvolvinganewregularizationtechnique,aparallelimplementationoftheregularizedclassier,andafastincrementalclassicationalgorithmthatusestheregularizedclassiertondasmallsubsetofthetrainingsetwithbettergeneralizationcapabilitiesandcomparableclassicationaccuracy.Finally,theconclusionsregardingthekernel-basedmethodsandtheirapplicationsaredrawninChapter6. 19

PAGE 20

Thischaptercoversthefundamentaltopicsonmachinelearning,specicallykernelmethodsinmachinelearning,whicharebuiltonthestrongtheoreticalbackgroundofthestatisticallearningtheory[ 17 ].Althoughkernelmethodsarerelativelynewtopicsinmachinelearningtherearecomprehensivebooksavailableonkernelmachinelearning[ 54 55 ]andmostofthematerialpresentedinthischapterarecompiledfromthesebooks. Thesequenceoftopicscoveredinthischapterareasfollows.First,theconceptofkernelsisexplaninedandhowtheyareusedasnonlinearmappingtoolsinordertoenhancesimilarityamongdatapointsisshown.Then,thegeneralizationtheoryisreviewedbrieytomentionboundsoftheempiricalriskbasedontheVapnik-Chervonenkistheory.Somefundamentalconceptsinoptimization,morespecically,someelementsoftheLagrangiantheoryareintroducedtoexplainnecessaryandsucientconditionsforconvexoptimizationproblemswithlinearinequalityconstraints.Thesethreetopicspavethewaytosupportvectorclassicationandregressionmodelswithdierentvariationsofprimalanddualformulationswithrespecttopenalizationschemesontheclassicationerror. Themainconceptisfocusedonthedotproductoftwomappedpatternvectors.Thedotproductofmappedpatternvectorsmaybecomecomputationallyintractable,while 20

PAGE 21

55 ](pg.26) 2-1 .Anothermappingfrom2-Dinputspacetoa4-Dfeatureforthesamekernelisasfollows Figure2-1. Kernelmappingofcirculardatain2-Dto3-D. 21

PAGE 22

56 ]. Theresultsonunorderedproductfeaturesarealsogivenin[ 55 ]. Nowwecaninvestigatetheconditionshowkernelscancreatefeaturemaps.Moregenerally,givenakernel,howcanweconstructafeaturespacesuchthatthekernelcomputesthedotproductinthatkernelspace?Firstwedeneapositivesemidenitekernel. (Tkf)(x)=ZXK(x;x)f(x)dx:(2{3) Thefunction( 2{3 )aboveiscalledthekernelofTk. 22

PAGE 23

Intheliteratureitwasshownthatanyalgorithmthatworksondotproductscanbekernelizedthroughthekerneltrick,asexplainedbelow: 55 ](pg.34)). 54 ](pg.35)). 23

PAGE 24

ThedatausedintrainingandtestingareassumedtobegeneratedfromthesameunknowndistributionDoverthepairs(xi;yi).Theprobablyapproximatelycorrect(PAC)boundonlearningrequiresastrongboundthatisnotlikelytofail.PAChastheform"="(n;H;),whichmeansthat,withprobabilityof1overrandomlygeneratedsetsS,thegeneralizationerrorofselectedhypothesishSwillbeboundedby whereHistheclassofallpossiblehypothesisandnisthenumberofexamplesinthetrainingset.Beforepresentingfurtherresults,wementionthatthequantityknownasVapnik-Chervenonkis(VC)dimensionshowstheexibilityorcapacityofafunctionclasstoclassifyarandomtrainingsetofpoints.Here,wegivethefundamentaltheoremoflearning,detailsofwhichcanbefoundin[ 54 ].

PAGE 25

d+log2 54 ]. nR2+kk22 25

PAGE 26

nkwk22+kk21log(1=) 2.4.1 thelatterformulationprovidesthemeansforthekernelmethodstobeapplicableinsupportvectorclassiers.ThetransformationfromtheprimalproblemtothedualproblemisexplainedthroughLagrangiantheory,whichisalsoawellstudiedsubjectinmathematics.Westartbythedenitionoftheprimaloptimizationproblem.

PAGE 27

(2{16) Thetheoptimizationproblemthatisrenderedbythesupportvectorclassicationisaconvexfunctiononaconvexset.Firstwedenetheconvexfunctionandconvexset. ContinuousfunctionsthataretwicedierentiablearealsoconvexiftheirHessianmatrixispositivesemi-denite. Lagrangiantheoryisbasedoncharacterizinganoptimizationproblemwithoutinequalityconstraints.TheLagrangianfunctionandtheLagrangianmultipliersarethemainconceptsinthistheory.Karush-Kuhn-Tucker,furtherimprovedthischaracterizationtodeneoptimalityconditionswhentheproblemhasinequalityconstraints.Wegiveformaldenitionsoftheseconceptsbelow.Westartwiththemostgeneralcaseof 27

PAGE 28

minimizef(w)w2 2{18 ),andsetsalowerboundontheoptimalsolutionvalueoftheoriginalproblem.Beforeweshowthisrelation,letusdenetheLagrangiandualformulation. 2{18 )isdenedasfollows. Thefollowingtheoremestablishestherelationshipbetweenthesolutiontotheprimalproblemandthedualproblem.

PAGE 29

Proof. 2.3.6 Thesolutionstotheprimalanddualproblemsarenotguaranteedtobeequal,inwhichcasethedierenceisreferredasthedualitygap.Thefollowingtheoremstatesthestrongdualitywhencertainconditionsaresatisedforanoptimizationproblem. (2{22) 2.3.8 )canbeappliedinpractice. 29

PAGE 30

2.3.8 ,wheref(w)isconvex,thenecessaryandsucientconditionsforapointwtobeoptimumaretheexistenceofandsuchthat ThemainconvenienceoftheKuhn-Tuckertheoremistransformingtheprimalproblemintoadualproblemwhichiseasiertosolve.ThisismainlydonebydierentiatingtheLagrangianfunctionwithrespecttotheprimalvariables,andsubstitutingtheequivalentexpressionsbackintheLagrangianfunctiontoachieveasimplerformulationwithoutinequalityconstraints.Thesolutiontothedualproblemshowswhichconstraintsintheprimalproblemareactive.Theactiveconstraintsindicatesupportvectors,whicharegenerallymuchlessinnumbercomparedtothetotalnumberofallinequalityconstraints.Inmachinelearning,thisresultwillbecomemoreclearinthenextsection. 30

PAGE 31

2.3 Firstweintroducetheproblembyformaldenitionsofahyperplane,acanonicalhyperplane,andthemarginofahyperplane.Ahyperplaneisad1dimensionallinearsurfaceinaddimensionalspaceH,anditcanberepresentedas: In 2{23 ,wisorthogonaltothehyperplaneandhxigivesthelengthofpatternvectorxalongthedirectionwwhenwisaunitvector.Notethatwehaveinherentdegreeoffreedominspecifyingthesamehyperplaneas(w;b)for6=0.

PAGE 32

(2{24) (2{26) Thisimpliesamarginof Weassumethatthetrainingdataandthetestdatacomefromthesamedistribution.Duetothenoiseinthedata,theseparatinghyperplanecanstillclassifynewpatternvectorscorrectlyifthemarginismaximized.Maximizingthemargin1=kwkforthecanonicalhyperplaneisequivalenttominimizingkwk.Inthefollowingoptimizationproblem,eachpointxiwithalabelyisatisesthecanonicalityconditionintheconstraints 2{28 ,whilemaximizingthemarginbyminimizingthenormofthenormalvectorwofthehyperplane,asshowninFigure 2-2 Figure2-2. Maximalmarginclassier 32

PAGE 33

2kwk2+C subjecttoyi(hwxii+b)1i Problem 2{28 isreferredastheprimalproblem.Inordertoapplykernelmethods,thedualproblemcanbederivedfromtheLagrangianfunction 2{30 L(w;b;;^)=1 2kwk2+C (2{31) DierentiatingL(w;b;)withrespecttotheprimalvariableswandb,andassumingstationarity,wegetthefollowingexpressions,whichwecansubstitutedbackintheLagrangianfunctiontoobtainthedualformulationwithonlydualvariables. Thefollowingpropositionshowsthedualformulationandhowthesolutiontothedualformulationrealizesthemaximalhyperplane.

PAGE 34

2nXi=1nXj=1yiyjijhxixji; Notethatthedistanceofapatternvectortothehyperplaneisgivenbythefunction sincethedistanceoneithersideofthehyperplanewouldhaveoppositesigns,thefollowingfunctioncanclassifyapatternvectorxattherightsideofthehyperplane. ThisclassicationmethodcaneasilybeextendedtothefeaturespacebyreplacingthelineardotproducthxixjiwithakernelK(xi;xj).Sincethelineardotproductisalinearkernel,weuseagenerickernelinfurtherformulations. 34

PAGE 35

2-3 softmarginisdemonstratedwhichincurspenaltyformisclassiedpatternvectors.Now,wecanmodifythemaximummarginclassierwiththeintroductionofsackvariables,asfollows. Figure2-3. Softmarginclassier min1 2kwk2+C subjecttoyi(hwxii+b)1i Notethati<0cannotbetrueforanyoptimalsolutionto 2{39 ,sincethecorrespondingconstraintwouldstillbesatisedwhileincurringlesscostifi=0.Therefore,wedonotexplicitlyneedapositivityconditionontheslackvariables. 35

PAGE 36

2{39 the2-normoftheslackarepenalizedintheobjective.Analternativeformulation 2{39 involvespenalizationofthe1-normslackvariablesintheobjective.However,weneedtoimposepositivityontheslackvariablesbeabletopenalizethem. min1 2kwk2+C subjecttoyi(hwxii+b)1ii0i=1;:::;n: Nowwecanderivethedualformulationforboth1-normand2-normsupportvectormachineformulations.Thesignicanceoftheseformulationsarethattheydonotinvolveinequalityconstraints,andtheyallowkernelstobeintroducedintheobjectivefunctionwhereadotproductoccurs.ThestandardmethodistodierentiatetheLagrangianfunctionwithrespecttotheprimalvariablesandsubstitutethembackintheLagrangianfunction.TheLagrangianfunctionforthe2-normSVMprimalproblemisasfollows. L(w;b;;)=1 2kwk2+C (2{42) DierentiatingtheLagrangianfunctionwithrespecttow,b,andeach,weget (2{44) 36

PAGE 37

2{43 2{45 )in( 2{41 ),weobtain, maxnXi=1i1 2nXi=1nXj=1yiyjiihxixji1 2CnXi=12i Wecansubstitutethedotproductintheobjectivewithamoregeneralkernelofchoice.Thefollowingpropositionsummarizesthedualformulationfor2-normsoftmarginSVM. 2nXi=1nXj=1yiyjijK(xi;xl)1 2nXi=12i Thederivationforthe1-normdualformulationisverysimilartothatof2-norm.TheLagrangianfunctionisslightlydierent,withthesumofslackvariables,andtheadditionaltermforthepositivityconstraints. 37

PAGE 38

2kwk2+C Dierentiatingthelagrangianfunctionwithrespecttow,band,andimposingstationarity,weobtain, (2{52) Substitutingtheseexpressionsbackin 2{49 ,weobtain, maxnXi=1i1 2nXi=1nXj=1yiyjiihxixji 2{53 andri,thedualvariablesishouldbein[0;C],whichisoftenreferredtoastheboxconstraint.Thefollowingpropositionsummarizestheresultfor1-normdualformulationforclassication. 2nXi=1nXj=1yiyjijK(xi;xl) (2{54)

PAGE 39

max;;`Xi=1i1 2`Xi=1`Xj=1yiyjijK(^xi;^xj)+C`Xi=1i However,thisproblemishardtosolve.Therefore,aniterativetwostagealgorithmisproposed[ 22 ].Intherststageofanyiterationiwiththestartingscalevector(i1),theproblemisoptimizedwithrespecttovectorasinthestandardSVMmethodtondasolution(i).Inthesecondstage,theproblemissolvedwithrespecttothescalevector 39

PAGE 40

2-4 Figure2-4. StandardSVRand"-Insensitivetubearoundtheregressionfunction Sinceanythinginthe"oftheestimatedregressionfunctiondoesnotcontributeaserror,thistypeofregressioniscalled"-Insensitive.Asintheclassicationcase,therearetwoversionsofthe"denedbythenormoftheerror,whicharecalledQuadraticandLinear"Insensitivesupportvectorregression,respectively. 2{57 ).Theconstraintsimplythattheactualpattern 40

PAGE 41

2{58 )orbelow(constraint 2{59 )theregressionfunctionwithoutpenalty.Allpatternvectorsoutsidethe"rangearestillallowed,howevertheyincuracostofC. min1 2kwk2+C (2{57) subjectto(hwxii+b)yi"+i Inordertoapplythekerneltrick,weneedtoderivethedualformulationfromtheLagrangianfunctionfor( 2{57 ),whichisgivenas, L(w;b;;^)=1 2kwk2+C (2{60) DierentiatingtheLagrangianfunctionwithrespecttotheprimalvariablesw,b,andeachand^,weget (2{61) (2{62) (2{63) (2{64) Substitutingthepartialderivatives( 2{61 2{64 )in( 2{60 ),wecanwritethefollowingdualformulation, 41

PAGE 42

2CnXi=1(2i+^2i)1 2nXi=1nXj=1(^ii)(^ii)hxixji where,bischosensuchthatf(xi)yi="(^)=Cforanyiwith(^)>0. min:1 2kwk2+C (2{67) subjectto(hwxii+b)yi"+iyi(hwxii+b)"+^i:i;^i0fori=1;:::;n: 42

PAGE 43

2nXi=1nXj=1(^ii)(^ii)hxixji 2{66 ),howeverbischosensuchthatf(xi)yi="foranyiwith0<(^)
PAGE 44

Themotivationfortheselectivekernel-basedmethodscomesfromareallifeproblem,inwhichtimeseriesaretobealignedforimprovingclassicationresults.Theindividualpatternvectorsandcorrespondinglabelsorvaluesfromthemachinelearningcontextareextendedtosetsofpatternvectorssharingthesamelabelorvalue.First,westudytheproblemofchoosingasinglepatternvectorfromeachsetofpossibilitiesinordertondanoptimumseparationbetweentheselectedpositiveandnegativepatternvectors.Thisproblem,whichwecallhardselectionishighlycombinatorialinnature.Therefore,weintroducerelaxationsofthisproblem.Thesesoftselectionproblems,incontrast,slightlydierfromtheoptimizationproblemswehavestudiedinChapter 2 .However,thegeneralstructureofthealteredproblemsisquitesimilartotheoriginaloneswithaconvexobjective,andlinearconstraintsonaconvexdomain.Thisconvenienceallowsustoderivedualformulationsinwhichwecanapplythekerneltricktoobtainnonlinearclassiers.Thischapterincludesformulationsfortheselectiveversionsofsupportvectormachinesandsupportvectorregressionandtheirvariations.Theresultsontoyproblemsarepresentedforthehardselectionmethods,aswellasthesoftselectionmethods,whichincludeincrementaleliminationandincrementalinclusionschemes. 3-1 Thersttypeofproblemwewanttosolveisthehardselectionproblem,inwhichasinglepatternxi;k2Xiisselectedforeachsetsuchthatthemarginfortheselectedpatternsismaximized.Wegivetheformaldenitionbelow. 44

PAGE 45

Setofpatternvectors,3patternvectorsineachset minimize1 2kwk2+C 3{1 ),intherstsetofconstraints( 3{1 ),Misalargepositivenumber.ThiscanbesetasthemaximumdistancebetweenanypairofpatternvectorsinX.Whenthecorrespondingi;k=0,then,basically,thisconstrainthasnoeectontheproblem,and 45

PAGE 46

3{1 )ensurethatonlyonepatternvectorischosenfromeachsetofpoints.Althoughthisproblemndstheoptimalsubsetofpatternvectorsforthepositiveandthenegativeclass,itishighlycombinatorial. Werelaxthebinaryvariablesinthehardselectionprobleminsuchawaythattheywouldgiveusameasureonhowmuchslackdoesapatternvectorrequireifitismisclassied.Thefollowingquestionis,howcanwediminishtheeectofmisclassiedpatternvectorsontheobjectivefunctioninordertohavealargemargin?Weattempttoanswerthesequestionswithdierentrelaxationsofthehardselectionproblem,inwhichbinaryvariablesarereplacedwithcontinuousvariablesservingasfreeslackvariablesforpatternvectorsthataremisclassied. WeknowfromSection 2.4.2 thatinthesoftmarginclassication,eachpatternvectorhasaslackvariable,butthisslackispenalizedintheobjectivefunction.Here,weintroducetheconceptofrestrictedfreeslackforthosepatternvectorswhichareclosetotheseparatinghyperplane,ormisclassied.Themainmotivationbehindthisrelaxationisthatsuchpatternvectorsaretheoneswhichimposetheorientationandplacementoftheseparatinghyperplane.Wewanttodiminishtheireectofthesepatternvectorsandletthepatternvectorswhicharemoreseparateddeterminetheorientationandplacementoftheseparatinghyperplane. InFigure 3-2 ,therearepairsofredpatternvectorsandbluepatternvectors.Ifweignorethepairsandconsidertheproblemasastandardclassicationproblem,sinceblueandreadpatternsareseparable,astandardsupportvectormachinealgorithmwouldsolvethisproblemwithoutusinganyslack,howeverwithasmallmargin.Itisintuitiveandstraightforwardthatifthepatternvectorstowardsthemiddleweretoberemoved,theremainingpatternvectorswouldhavealargemargin.Byintroducingfreeslackvariableswecandiminishtheeectofthemiddlegroupofpatternvectors,howeverwerestrictthe 46

PAGE 47

Figure3-2. Classicationofaseparablesetinthepresenceofrestrictedfreeslack Therearetwoproblemstobeaddressed.Therstoneis:howmuchslackshouldbeprovided?,andthesecondoneis:howtodistributethetotalslack?Fortherstproblem,anintuitiveapproachcanbeadapted.Inthestandardsupportvectormachineformulation,afunctionalmarginof1isrequired.Sincewehaveatleasttwopatternvectorsperset,weprefertoprovideatotalfreeslackofn,whichisafreeslackof1persetontheaverage.Forasetoftwopatternvectors,therearethreepossibilities:i)bothareclassiedright,ii)oneofthemismisclassied,iii)botharemisclassied.Intherstcasethepatternvectorthatisclosertotheseparatinghyperplane,inthesecondcasethemisclassiedpatternvector,andinthethirdcasethepatternvectorfurtherfromthehyperplanewouldrequiremorefreeslackcomparedtotheotherpatternvector. Fordistributingtherestrictedfreeslack,weintroducetwoapproaches.Intherstapproach,thereisatotalslackofn,andeachpatternvectorisallowedtoreceiveslackirrespectiveofthesettheybelongto.Werefertothiscaseaspooledfreeslack(PFS).Figure 3-3 showsthedistributionforPFS.Inthesecondapproach,however,eachsetof 47

PAGE 48

3-4 .Inthefollowingsubsections,wepresentmodiedoptimizationproblemstogetherwiththeirdualrepresentationsinordertostudyselectivekernelmethodsinlinearandnonlinearclassicationproblems. Figure3-3. Distributionofpooledfreeslack(PFS) Figure3-4. Distributionoffreeslackperset(FSS) 48

PAGE 49

2kwk2+C subjecttoyi(hwxii+b)1i;ki;ki=1;:::;n;k=1;:::;tPni=1Ptk=1i;kn 3{1 )isgivenasfollows, L(w;b;;^)=1 2kwk2+C (3{4) (3{5) (3{6) (3{7) Substitutingthepartialderivatives( 3{5 3{8 )in( 3{3 ),wecanwritethefollowingdualformulation: 49

PAGE 50

2nXi=1tXk=1nXj=1tXl=1yiyji;kj;lhxi;kxj;li 2CnXi=1tXk=12i;knsubjecttonXi=1tXk=1yii;k=00i;kfori=1;:::;n;k=1;:::;t: 3{9 )withakernelK(xi;k;xj;l). FortheFSScase,theconstraint( 3{2 )becomes, Consequently,thischangeintroducesaseparateLagrangianmultiplieriforeachconstraint,andthereforetherelevantcomponent( 3{4 )intheLagrangianfunctionchangesto, FollowingthedierentiationoftheLagrangefunction,imposingstationarityandsubstitutingtheexpressionsbackintheLagrangian,wecanobtainthedualformulationforthefreeslackpersetcaseas, 50

PAGE 51

2nXi=1tXk=1nXj=1tXl=1yiyji;kj;lhxi;kxj;li 2CnXi=1tXk=12i;knXi=1isubjecttonXi=1tXk=1yii;k=00i;kifori=1;:::;n;k=1;:::;t: 2{39 ).Herewepresenttheselectiveversionofthe1-normSVMformulations.Thederivationoftheprimalanddualoptimizationproblemaresimilartothosediscussedforthe2-normcase,thereforewebrieypresenttheseformulations.TheprimalforthePFSapproachisasfollows. minimize1 2kwk2+CnXi=1tXk=1i;k 51

PAGE 52

maximizenXi=1tXk=1i;k1 2nXi=1tXk=1nXj=1tXl=1yiyji;kj;lhxi;kxj;lin 3{15 ),thedotproducthxixjicanbereplacedwithakernelK(xi;xj)foranonlinearmaptothekernelinducedfeaturespace. Theprimalanddualformulationsforthe1-normFSScaseareslightlydierentfromthegeneralfreeslackcase,wherethegeneralfreeslackofninconstraint( 3{14 ),nowimpliesthatthefreeslackisrestrictedto1foreachset,whichisgivenas, minimize1 2kwk2+CnXi=1tXk=1i;k (3{17)i;k0i=1;:::;n;k=1;:::;t:i;k0i=1;:::;n;k=1;:::;t: 52

PAGE 53

2nXi=1tXk=1nXj=1tXl=1yiyji;kj;lhxi;kxj;linXi=1i Throughempiricaltestsontoyexamples,eliminateworstmethodoutperformedchoosebestmethod.Eliminateworstmethodapproachestoalinearseparationforseparablesets.Foranappropriateparametervalue,forexampleintheGaussiankernel,theseparatingcurveturnsintoalinearhyperplaneforsucientlylargevalue.InFigure 3-5 therearethreepatternvectorsineachsetandthegroupsthemselvesareseparate. 53

PAGE 54

3-5 .SelectiveSVM,ontheotherhandcreatesaseparatinghyperplanewhichalignitselfsuchthatthemarginismaximizedwithrespecttothepatternschosenattheendoftheeliminationprocess.NotethatthesepatternvectorsaremostseparatedfromeachotherinFigure 3-6 Figure3-5. StandardSVMonaseparable3Dexamplefordierentkernelbandwiths 2 suggestanaturalextensionofselectiveSVMmethodstoSVMregressionmethods.ThestructureoftheregressionproblemsareverysimilartotheclassicationproblemscoveredinSection 3.1 Instandardsupportvectorregression,theinputisasetofpairsofapatternvectorandadependentvalue,S=f(x1;y1);:::;(xn;yn)g.Intheselectivesupportvector 54

PAGE 55

SelectiveSVMonaseparable3Dexample:boldpointsareselected regression,wehavesetsXi=fxi;1;:::;xi;kgofpatternvectorswithcorrespondingdependentvaluesforeachsetS=f(X1;y1);:::;(Xnyn)g.Themainobjectiveisselectingasinglepatternvectorfromeachsetsuchthattheobjectivefunctionoftheoptimizationproblem( 2{57 )isminimized.AsintheSelSVM,westartbydeninghardselectionforsupportvectorregression. 55

PAGE 56

2kwk2+C (3{19)subjectto(hwxi;ki+b)yi"+i;k+M(1i;k)i=1;:::;n;k=1;:::;t (3{22)vi;k2f0;1gi=1;:::;n;k=1;:::;t: Intheaboveformulation,Misalargepositivenumber,suchthatforthosepointswithvi;k=0,therelatedconstraintisalwayssatised,andthus,doesnothaveanyinuenceontheproblem.Thisisequivalenttoremovingthispatternvectorfromtheproblem.Mcanconvenientlybesettothemaximumdistancebetweenthedistancesofanytwopairsofpatternvectors.Constraints( 3{20 )and( 3{21 )accountforthecaseifapatternvectorisaboveorbelowtheregressionfunction,respectively.Finally,constraint( 3{22 )ensuresthatonlyoneofpatternvectorfromeachsetisselected.Thehardselectionproblem,likeinSelSVRcase,isahighlycombinatorialproblemandthereisnoecientsolutionknown. Weconsiderrelaxationsofthehardselectionprobleminordertoobtainanecientandeectivesolution.Insteadofthebinaryvariables,weintroducecontinuousvariablesforeachpatternvector,totalvalueofwhichisbounded.ThisisthesameframeworkofrelaxationsthatwediscussinSection 3.1 .Likewise,weincludetwoapproachesforthefreeslackvalues.Inthepooledfreeslack(PFS)case,eachpatternvectorcanbeassignedfreeslackregardlessoftheotherpatternvectorsintheirset,whereasfreeslackperset(FSS)boundsthefreeslackvaluesforeachsetwith1. 56

PAGE 57

minimize1 2kwk2+C (3{24)subjectto(hwxi;ki+b)yi"+i;k+i;ki=1;:::;n;k=1;:::;t:yi(hwxi;ki+b)"+^i;k+^i;k:i=1;:::;n;k=1;:::;t:nXi=1tXk=1(i;k+^i;k)n" 3{24 )isgivenas, L(w;b;;^)=1 2kwk2+C (3{26) 57

PAGE 58

3{24 )w,b,andeachi,^i,i,^i,andimposingstationarity,wegetthefollowingexpressions: (3{27) (3{28) (3{29) (3{30) (3{31) (3{32) Substitutingtheexpressions( 3{27 3{32 )in( 3{26 ),weobtainthefollowingdualformulation: maximize1 2CnXi=1tXk=1(2i;k+^2i;k) (3{33)1 2nXi=1tXk=1nXj=1tXl=1(^i;ki;k)(^j;lj;l)hxi;kxj;li"nXi=1tXk=1(i;k+^i;k)+nXi=1tXk=1yi;k(^i;ki;k)n"subjecttonXj=1tXk=1(i;k^i;k)=00i;kfori=1;:::;n;k=1;:::;t0^i;kfori=1;:::;n;k=1;:::;t:

PAGE 59

Thefreeslackpersetapproachresultsinasimilarformulation,whereconstraint( 3{39 )isreplacedbythefollowingconstraintforeachseti,whichlimitsthefreeslackto"perset. ThischangereectsontheLagrangianfunctionsuchthattheterm isreplacedwith, Theresultingdualformulationforthefreeslackpersetisgivenas, maximize1 2CnXi=1tXk=1(2i;k+^2i;k) (3{37)1 2nXi=1tXk=1nXj=1tXl=1(^i;ki;k)(^j;lj;l)hxi;kxj;li"nXi=1tXk=1(i;k+^i;k)+nXi=1tXk=1yi(^i;ki;k)"nXi=1isubjecttonXj=1tXk=1(i;k^i;k)=00i;kifori=1;:::;n;k=1;:::;t0^i;kifori=1;:::;n;k=1;:::;t:

PAGE 60

minimize1 2kwk2+C (3{38)subjectto(hwxi;ki+b)yi"+i;k+i;ki=1;:::;n;k=1;:::;t:yi(hwxi;ki+b)"+^i;k+^i;k:i=1;:::;n;k=1;:::;t:nXi=1tXk=1(i;k+^i;k)n" 60

PAGE 61

2nXi=1tXk=1nXj=1tXl=1(^i;ki;k)(^j;lj;l)hxi;kxj;li"nXi=1tXk=1(i;k+^i;k)+nXi=1tXk=1yi(^i;ki;k)n"subjecttonXj=1tXk=1(i;k^i;k)=00i;kCfori=1;:::;n;k=1;:::;t0^i;kCfori=1;:::;n;k=1;:::;t: minimize1 2kwk2+C (3{40)subjectto(hwxi;ki+b)yi"+i;k+i;ki=1;:::;n;k=1;:::;t:yi(hwxi;ki+b)"+^i;k+^i;k:i=1;:::;n;k=1;:::;t:tXk=1(i;k+^i;k)"i=1;:::;n"i;k;^i;k0i=1;:::;n;k=1;:::;t:i;k;^i;k0i=1;:::;n;k=1;:::;t: 61

PAGE 62

2nXi=1tXk=1nXj=1tXl=1(^i;ki;k)(^j;lj;l)hxi;kxj;li"nXi=1tXk=1(i;k+^i;k)+nXi=1tXk=1yi(^i;ki;k)"nXi=1isubjecttonXj=1tXk=1(i;k^i;k)=00i;kiCfori=1;:::;n;k=1;:::;t0^i;kiCfori=1;:::;n;k=1;:::;t: WetestedtheeciencyofselectiveSVMregressionona3Dsurfacewhichisgivenbypoints(x;y;z)suchthatz=sin(r)=r,wherer=p 3-7 62

PAGE 63

SelectiveSVRcomparedtoanavemethodfordierentbandwidths 63

PAGE 64

Inthepreviouschapterswereviewedthekernel-basedmachinelearningalgorithms,morespecicallysupportvectormachine(SVM)methods,whichincludeclassication,featureselectionandregressionmethods.Weintroducednovelselectiveclassicationandregressionmethods.Thischapterinvolvestheneuraldatasetonwhichthemethodswehavedevelopedsofarareapplied.Inafact,thisdatasetisthemainmotivationtoapplykernel-basedmachinelearningalgorithmsonneuraldata. Thedatasetconsistoflocaleldpotentials(LFP)fromprimatebrainsduringavisualdiscriminationtask.Wegivefurtherdetailonhowtheseexperimentswereperformed,includingthetechnicaldetailsandtheexperimentalparadigm.ThenweintroducetheSVMmethodsforclassicationandfeatureselection.Theresultsfromtheapplicationofthesemethodsarereportedandillustratedforthreemacaquemonkeys.Theselectiveclassicationmethodsareintroducedtoaccountforthetemporalvariationsamongthemanyrepetitionsofthesamecognitivetaskwhicharereferredtoassingletrials.TheimprovedresultsaftertheapplicationofselectiveSVMmethodsarealsopresented. 57 { 60 ].Allthreemonkeysarewelltrainedtorespondtoago/nogovisualstimulibeforetheelectrodesareplacedinthesemonkeys'brains.Surgeriestoplacetheelectrodesareperformedundersterileconditionswiththemonkeyundergeneralanesthesia.Teoncoatedplatinum-iridiumwirebipolarelectrodeswitha0.125mmdiameterareinsertedinthehemispherecontralateraltothehandusedinthetask,withthelessadvancedtipoftheelectrodesextended0.5mmintothedura 64

PAGE 65

60 ].InFigure 4-1 theelectrodelocationsthatareusedinthisstudyareshown. Figure4-1. Approximateplacementofelectrodesinthetreemonkeys Localeldpotential(LFP)dataarecollectedfromeachmonkeyduringanumberofsessions.Eachsessionconsistsaround1000trials.Thedataareband-passlteredbetween1and100Hzanddigitizedat200Hz.Foreachtrial,dataarerecordedfor900msafterthemonkeyinitiatesthetrial.Thetimebetweenthetrialsarearoundasecond. ThestimuliaredesignedtocreatelinesanddiamondsusingeightsquaresasshowninFigure 4-2 .Thestimuliarereferredasrightslantedlineandrightslanteddiamondintherstrow,andleftslantedlineandleftslanteddiamondinthesecondrow.Notethatlinesaregivenintherstcolumnanddiamondsaregiveninthesecondcolumn.Thegostimuliischosentobeeitherbothlinesorbothdiamonds,andneveranyotherpossiblecombinationofthefourstimulustypes.Notethatwhenthetwolinestimuliandthetwodiamondstimuliaresuperimposedontoeachotherseparately,theresultingtwoshapesareidentical.Therefore,themonkeyhastodistinguishalinefromadiamondbycorrectlyrealizingtheplacementofatleasttwosquares.Thetotalarea,contrast,edgelengthandbrightnessareconstantacrossthestimulitypesandtrials.Thestimuliisplaced57cmfromthesubjectanditsoverallsizeis6degreesbetweentheoutercornersquares.Thepresentationtimeiscomputercontrolledwithapiezoelectricshutter. Atrialisinitiatedbythemonkeypressingaleverandkeepingitpressed.Theinitiationisfollowedbyarandomamountoftime,uniformlydistributedbetween1200 65

PAGE 66

Visualstimuliusedintheexperiment and2200ms,beforethestimulusappearsonthescreenforexactly100ms.Themonkeyisexpectedtorespondtoastimuluswithinthe500msfollowingthestimulusonset.Thereleaseoftheleveristhecorrectgoresponse,andkeepingitpressedisthecorrectnogoresponse.Thecorrectgoresponsesarerewardedwithasmallamountofwater,whereascorrectnogoresponsesarenotrewarded.TheoutlineofthetaskispresentedinFigure 4-3 Figure4-3. Experimentalparadigm Inallsessions,therearetwodierentstimulus-responsecontingencies:line-go/diamond-nogoanddiamond-go/line-nogo.Inthisstudy,onlycorrectgoandcorrectnogotrialsareconsidered.ForeachtrialLFPdatarecordingstarts90mspriortothestimulusonset 66

PAGE 67

8 9 ].Consideringthemulti-dimensionalnatureoftheproblemandthenonlinearbehaviorofthebrain,SVMclassicationcouldbeusedoneachtimepointtodetectdierentstagesofearlyeectsofvisualstimulus,discriminationofthestimulitype,andpreparationandexecutionoftheresponse.Moreover,thecontributionofthechannelstotheclassicationateachtimepointwouldrevealvaluableinformationonthespatiotemporalinteractionbetweenthevisualsystemandthemotorsystem.InthefollowingsubsectionstheSVMmethodisbrieyintroduced,followedbytheSVMbasedfeatureselectiontechniqueappliedinthisstudy. Thereasoninassociatingtheselabelstothedierenteectsareasfollows:1)Fortheearlyvisualeect,thedetectionofthedirectionofthestimuli(leftorright).2)Forthediscriminationofthevisualstimuli,themonkeyhastodistinguishbetweenalineanda 67

PAGE 68

Dierentstagesofthevisuomotortask Label Leftvs.Rightstimuli Discriminationofstimulus Linevs.Diamondstimuli Responsepreparationandexecution Govs.Nogo diamondstimuli.Inthemotionpartthemaineectisthepreparationoftheimpulsesentfrommonkey'sbraintoitshandanditsexecution. Weareconsidertwosetsofdata,whereintherstonelinesaregostimuliandthediamondsarenogostimuli.Inthesecondonediamondsaregostimuliandlinesarenogostimuli.Foreachmonkeywehavearound10,000trials.Sincethecombinatorialresultoftheclassicationscheme,andthenumberoftimepointsiscomputationallydemanding,foreachmonkey,between20%and30%oftheavailabletrialswereusedtoperformtheclassication.Thetrialsarerandomlyselectedfromacombinedsetofdatainvolvingallthetrialsfromallthesession.Wegroupthetrialsinvariouswayswithrespecttotheirstimulustype.Ifweletatrialwithrightline,rightdiamond,leftlineandleftdiamondbemarkedassi1;si2;si3andsi4,respectively,comingfromdataseti=1;2,thenwecanformthreedierentsetsofdatawithrespecttoourinterest.Thepossiblecombinationsare: 1. RIGHTvsLEFT(S+=s11[s12[s21[s22vs.S=s13[s14[s23[s24). 2. LINEvsDIAMOND(S+=s11[s13[s12[s14vs.S=s21[s23[s22[s24), 3. GOvsNOGO(S+=s11[s13[s22[s24vs.S=s12[s14[s21[s23), InFigures 4-4 4-5 ,and 4-6 ,theclassicationandthefeatureselectionresultsaregivenforthemonkeysGE,LUandTI.ThethreecolumnsofgraphsineachgurecorrespondtothethreedierentstagesweareinterestedaslistedinTable 4.3 .Intherstrowofgraphsineachgure,theoverallaccuracyresultsarepresented.Inthesecondrow,thecontributionofeachchanneltotheoverallclassicationaregiven.werefertotheseplotsaschannelselectionplots.ThechannelnamesandcorrespondingcorticalregionsaregiveninTableF.Knowingthecontributionofeachchannelforatimepointt,wecansortthechannelsindescendingorderoftheircontribution.Intheplotsinthethird 68

PAGE 69

ChannelnamesandlocationsforTI,LUandGE LU GE Abbr.Names Abbr.Names Abbr.Names Supr StrAStriateA2 StrFStriate StrBStriateB3 PstSPrestriate StrCStriateC4 ParEParietal PstAPrestriateA5 AudB PstBPrestriateB6 FrnDFrontal PstCPrestriateC7 SomVSomatosensory FefAFEF8 SomMSomatosensory MotBMotor9 MotHMotor SomASomatosensory10 StrBStriate ParDParietal11 StrCStriate ParEParietal12 IftAInferotemporal PrmAPremotor13 AudA PrmCPremotor14 FrnAFrontal FrnBFrontal15 // FrnGFrontal row,foreachtimepointweshowthechangeinclassicationaccuracyasweincrementallyintroducethesortedchannels.Thehorizontalaxisisthetime,andtheverticalaxisshowsthenumberofchannelsincluded.Werefertothisplotastheincrementalaccuracyplot. 69

PAGE 70

70

PAGE 71

71

PAGE 72

4-7 comparativesnapshotsofpeakactivitiesaregivenforthreemonkeystogetherwiththetimes.Thelocationandthetimeareconsistentforallthreestages. Singletrialsarecomposedofsimultaneousrecordingsfrommultiplechannels.Assumingthattherearedchannels,therecordedvaluesattimetfromallchannelsfromanindividualtrialcanbeconsideredasapointz(t)=[z1(t);:::;zd(t)]TinRd,andtheentiretrialasadiscreterepresentationofa1-dimensionalcurveinRd.AsimplemeasuretondthesimilaritybetweenthesetwocurveszaandzboflengthTistondthesumofthedierencevectorsforeachtimepoint,i.e., Usingdynamicprogramming,thedistancebetweenthetwocurvescanbeminimizedbystretchingthesecurvestogetabettermatch.whichimpliesthatsomepointsmaybedeletedorgapsmaybeinsertedineithercurve.DynamicTimeWarping(DTW)isoneoftheavailablematchingmethodsthatcanbeappliedonourdata[ 61 ].Theoriginalalgorithmisproposedforonedimensionalcurves.Itisfairlyeasytoextendthemethodtod-dimensionsforcurveszaandzbasfollows, 72

PAGE 73

whereci,cdandcmaretheassociatedcostsforinsertion,deletionandmatch.Inthecurrentapplicationci=cd=1andcm=0.Thematchinganditsscorecanbefoundbybacktrackingandaddingthecostsup.Theresultingcostiscalledtheeditdistancebetweenzaandzb. Smallerdistancesbetweentrialsmeanthatthetrialsareverysimilartoeachother.Asubsetofsimilartrialsmaysignicantlyreducethevariationofthetrialswithrespecttoagiventimepointt.Reducedvariation,inreturn,reducesthenoise,andincreasestheclassicationaccuracyandthesignicanceofthefeatureselectionprocess. Theeditdistancebetweentwotrials,whichisasimilaritymeasure,canbefoundusingdynamictimewarping(DTW).InFigure 4-8 ,arasterplotofthepair-wisedistancescoresfor110single-trialrecordingsregardingrightslantedlinestimuliareshown.Wesortedall110110scoresinthematrixandthenplottedthesortedscoresontheright.Thisplotshowstwolevelsofsimilarityamongtrials.Wegroupthetrialswithscoresbelow90andabove90.Thispatternisveryconsistentoverallothertrialsregardingdierentstimulitypeswithathresholdbetween90and100.Whenthesescoresaredividedintolow(0)andhigh(1)values,itiseasytondsimilartrialsbysummingthescoresforeachtrialandsortingthetrialsbytheirtotalscore. Asubsetofabout35trialsfromthetopofthelistcanbeconsideredtobeverysimilartoeachother.Therefore,theclassicationandfeatureselectionalgorithmsareexpectedtogivebetterresultsduetoreducednoise.Thesamemethodisappliedtoeachdierentsetoftrialsregardingdierentstimulianddierentsessionstondsubsetsofsimilartrials.In 73

PAGE 74

4-9 TheimprovementafterusingDTWisveryclearforstimulus,categoricalandresponserelateddierences. 4.4 ,thevariationinthestagesofthevisuomotortaskovertimeprecludesusinganalignmentmethodtodecreasethevariationacrosstrialsforagiventimepoint.Althoughtheresultsfromthestandardkernel-basedmachinelearningmethodsareintuitiveandconclusive,asshowninChapter 3 ,betterresultsmaybeachievedusingselectivekernel-basedmachinelearningmethods.However,oneshouldnotethatDTWusesthedataintheoriginalinputspace,andworkswitheuclidiandistances,whereassimilaritiesbetweenpatternvectorsmaybebetterexplainedinanonlinearfeaturespace.SelectiveSVMmethodsachievethesimultaneousgoalofclassicationinanonlinearspaceandalignmentofpatternvectorsatthesametime.WeshowinthissectionthattheresultsobtainedwiththeselectiveSVMmethodsdominatetheresultsusingstandardmethodslikeDTW. ThemotivationbehindthedevelopmentofselectiveSVMisthevariationintheunderlyingprocessesthataretobedistinguishedbyclassication.Thebasicideaisasfollows.LetSjbetheamatrixwhereeachrowiofnrowsisavectorofrecordingsfromtrialiattimepoint,j.NowconsidertheappendedmatrixjSTtSTj+1:::Sj+t1j,whichinvolvesntrowsofvectorpatterns,suchthatthecorrespondingrowsofSjandSj+1aretwoconsecutiverecordingsoverthetimelinefromthesametrial.Thismatrixiscomposedofallpatternvectorswithinawindowofsizetfromjtoj+t1.OurobjectiveistoprovideSVMclassierwithmorepatternvectorstoincreasethechanceofbetterpointselection.However,theclassicationalgorithmwillbeeectedbythenoisypoints.Weimplicitlyassumethatonepointoftfromeachtrialwillbeallowedtobechosen.Infact,thepatternvectorchosenisconsideredtobeagoodpoint,whichisexpectedto 74

PAGE 75

Sinceselectingthebestpointfromeachwindowandeachtrialisaveryhardproblem,weusethepooledfreeslack(PFS)toeliminatetheworstpointsonebyonetoachievethedesiredpointsfromeachwindow.Althoughthisisaheuristicapproachatarstglance,itgivesverygoodresultsinclassicationandfeatureselection. ThecodeforselectiveSVMiswritteninCPLEX9.0anddynamiclibrarylesareproducedtobeusedinaccordancewithMATLABenvironment.Becauseofthecomputationallimitationsofthesesoftware,theentiredatacouldnotbesolved.Thedataconsistsofaround4000trials.Thestrategyfollowedistoconsider200trialsatatime(observedlimitsofthesoftware)withalmostequaldistributionofpositiveandnegativelabels.SelectiveSVMalgorithmisrunoverawindowof3patternvectors,whichisequivalentto10milliseconds.Thepatternvectorwiththehighestslackiseliminatedfromeachset.Thisisrepeateduntilthereisonlyonepatternvectorpereachcell.Althougheachbatchof200trialsareindependentofeachother,thecumulativesetoftheselectedpointsareconsistent.Theclassicationandfeatureselectionresultssignicantlydominatedtheimprovementsobtainedfromdynamictimewarpingmethod.SinceselectiveSVMtriestondbetterpoints,thereisanincreaseinaccuracyatthepartoftherecordingbeforethevisualstimuli.Thisincreaseposesthequestionwhetherthisisjustanoverallincreaseateverytimepoint.Ifthisisthecase,wearenotactuallydetectinganythingabouttheprogressionofthevisuomotortaskprogressionovertime.However,whenwegraphthepercentageincrease,itisclearfromFigure 4-10 thatforline-diamondseparation,weachievednotonly80%accuracyresultsbetweentime130msand200ms,butalso,almosttwicetheamountofthepercentincreasecomparedtotherestofthetimescale.Thisconrmsthat40-50msafterstimulusarrivalstartsadierentiationinthebrainregardingtotheabstractcategorizationoftheconceptofa 75

PAGE 76

Thego-nogoseparationissomewhatconfusingbecausethehighestpercentincreaseisbeforetheonsetofstimulus.However,thisisduetothebiasintroducedinthebrainprobablyfromtheunevendistributionofgotrialswhichmaybeperformedbacktobackandthuscarryingoverthebrainstatefromonetrialtothenext.However,itcanalsobeobservedthatsuchseparationdecreasesbythetimestimuliisarrivestothestriatecortex.Thispointonthetimelinecanbeconsideredasaprimingpointforthisartifactafterwhichajumpinthepercentincreasecanbeobserved.Thisjumpfrom200msto260ms,beingtooearlyforthemotorresponsemaycorrespondtothedecisiontorespondornot. Theresultsonfeatureselectionisalsoquitesignicant.InFigure 4-11 ,theleftcolumnisline-diamond,rightcolumnisgo-nogo,upperrowisfeatureselectionresultsafterDTW,lowerrowisfeatureselectionresultsafterselectiveSVM.Notethattheinteractionofchannel8,whichshowsthemotorresponseisnotobservedprominentlyontheline-diamondcase.Insteadstriateandprestriatecortexescontributetoclassicationfortheaforementionedtimeperiods,forwhichcategoricalandresponserelateddierencescanbeobserved.However,theassumptionthatthepossibilityofback-to-backgotrialsareconrmedbytheexcitedactivityonthemotorcortex,evenbeforethestimulusonset. SinceselectiveSVMtriestoseparatethetwoclassesforeverytimepointthereisanaverageincreaseof10%intheaccuracy.Thisincreasecanbecounteractedbycreatingabaseline.Forthisapproachwerandomlyassignclasslabelstopatternvectorssothatwecandeducetheincreaseintheaccuracyforrandomdata.InFigure 4-12 ,theimprovementisveryclearafterthebaselineapproach.Itisalsoworthnotingthatthereisanactualdecreaseinthebaselinewherethereisaprominentincreaseintheaccuracy.Thiscase,whichhasnotbeendetectedbyanypreviousstudiesisveryprominentlydetectedusingselectiveSVMmethodandcreatingabaseline. 76

PAGE 77

ClassicationandchannelselectionresultsforGE 77

PAGE 78

ClassicationandchannelselectionresultsforLU 78

PAGE 79

ClassicationandchannelselectionresultsforTI 79

PAGE 80

Brainsnapshots 80

PAGE 81

(c)(d) Figure4-8. a)Uncodedscoresmatrixb)Sortedscoresc)Codedmatrixd)Sortedmatrix 81

PAGE 82

(b) Figure4-9. a)comparativeclassicationresultsb)comparativechannelselectionresults 82

PAGE 83

Improvementclassicationforline-diamond(left)andgo-nogo(right) Figure4-11. Improvementinthefeatureselectionforline-diamond(left)andgo-nogo(right) 83

PAGE 84

Baselineapproachtoemphasizeimprovement(top),dierencebetweenactualimprovementandbaseline(bottom) 84

PAGE 85

InthisChapterwediscussadierentclassofSupportVectorMachineclassiersusuallyreferredasProximalSupportVectorMachines,whichwaspopularizedbyMangasarianetal.Inthisfamilyofclassiers,thenotionofaseparatinghyperplaneturnsintotwohyperplanes,eachapproximatingitsownclassbyminimizingthetotaldistancefromitspatternvectorswhilemaximizingthetotaldistancefromtheoppositesetofpatternvectors.Theadvantageoftheproximalclassiermodelisthattheunderlyingminimizationproblemreducestoaneigenvalue/eigenvectorproblem,whosesolutioniseasytond. Ourmaincontributioninthischapterarethree-fold.Therstoneisanimprovementthatwouldreducetherunningtimeofthecurrentmethodintheliterature.ThemethoddevelopedbyMangasarianrequirestwoeigenvalueproblemstobesolved,oneforeachoftheapproximatinghyperplane.However,inSection 5.2 ,weshowthatsolvingonlyoneproblemwouldsuce,usinganewregularizationtechniquetondbothhyperplanes.Weshowcomparativecomputationalresultsofourmethodwiththecurrentone.InSection 5.3 ,wediscussanimplementationoftheproposedmethodonparallelcomputersandpresentscomputationalresults.Consideringadynamictrainingset,weintroduceanincrementalversionoftheproposedmethodinSection 5.4 thatusesasignicantlysmallkerneltoreducecomputationalcomplexityofclassifyingnewtrainingpoints,andimprovethegeneralizationcapabilityoftheclassier.Similarly,computationalresultsarepresentedfortheincrementalmethod. 85

PAGE 86

16 ]proposestoclassifythesetwosetsofpointsAandBusingtwohyperplanesinsteadofasingleseparatinghyperplaneasinsupportvectormachines.InMangasarian'sapproacheachhyperplaneisclosesttoonesetofpoints,andfurthestfromtheother.LetxTw=0beahyperplaneinRd.InordertosatisfythepreviousconditionforthepointsinA,thehyperplanefortherstclassofpatternvectorscanbeobtainedbysolvingthefollowingoptimizationproblem: minw;6=0kAwek2 Here,eisavectorof1's.ThehyperplaneforBcanbeobtainedbyminimizingtheinverseoftheobjectivefunctionin( 5{1 ).Now,let thenequation( 5{2 ),becomes: minz2RmzTGz zTHz:(5{3) Theexpressionin( 5{3 )istheRaleighquotientofthegeneralizedeigenvalueproblemGx=Hx.Thestationarypointsareobtainedatandonlyattheeigenvectorsof( 5{3 ),wherethevalueoftheobjectivefunctionisgivenbytheeigenvalues.WhenHispositivedenite,theRaleighquotientisboundedanditrangesovertheintervaldeterminedbyminimumandmaximumeigenvalues[ 62 ].Hispositivedeniteundertheassumptionthatthecolumnsof[Be]arelinearlyindependent.Theinverseoftheobjectivefunctionin( 5{3 )hasthesameeigenvectorsandreciprocaleigenvalues.Letzmin=[w11]andzmax=[w22]betheeigenvectorsrelatedtotheeigenvaluesofsmallestandlargestmodulo,respectively.ThenxTw11=0istheclosesthyperplanetothesetofpointsinAandthefurthestfromthoseinBandxTw22=0istheclosesthyperplanetothesetofpointsinBandthefurthestfromthoseinA.ThisisdepictedintheexamplesshowninFigure 5-1 86

PAGE 87

Separationobtainedwithgeneralizedeigenvectors. AstandardtechniqueinSVMstoobtainagreaterseparabilitybetweensetsistoembedthepatternvectorsintoanonlinearspace,viakernelfunctions.InthisworkweusetheGaussiankernel, In( 5{4 ),xiandxjdenotetwopatternvectorsinthefeaturespace.Thistechniqueusuallygivesbetterresults,asshowninseveralstudies.Resultsregardingnonlinearlyseparableproblems[ 63 64 ]stillholdandaformulationfortheeigenvaluesproblemcaneasilybederived.Thisformulationisgiveninthenextsection. 5{2 ).NotethatevenifAandBarefullrank,matricesGandHarealwaysrank-decient.ThereasonisthatGandHarematricesoforderd+1,andtheirrankcanbeatmostd.Theaddedcomplexityduetosingularityofthematricesmeansthatspecialcarehastobegiventothesolutionofthegeneralizedeigenvalueproblem.Indeed,ifthenullspacesofGandHhaveanontrivialintersection,i.e.Ker(A)TKer(B)6=0,then 87

PAGE 88

Mangasarianetal.proposestouseTikhonovregularizationappliedtoatwo-foldproblem: minw;6=0kAwek2+kzk2 and minw;6=0kBwek2+kzk2 5{3 ).Thesolutions(wi;i);i=1;2to( 5{5 )and( 5{6 )representthetwohyperplanesapproximatingthetwoclassesoftrainingpoints. Inpractice,ifGHisnonsingularforeveryand,itispossibletotransformtheproblemintoanotherproblemthatisnonsingularandthathasthesameeigenvectorsoftheinitialone.Westartwiththefollowingtheorem[ 65 ] 88

PAGE 89

5.2.1 canbeapplied.Bysetting1=2=1and^1=1;^2=2,theregularizedproblembecomes minw;6=0kAwek2+^1kBwek2 If^1and^2arenonnegative,isnon-degenerate.Thespectrumisnowshiftedandinvertedsothattheminimumeigenvalueoftheoriginalproblembecomesthemaximumoftheregularizedone,andthemaximumbecomestheminimumeigenvalue.Choosingtheeigenvectorsrelatedtothenewminimumandmaximumeigenvalue,westillobtainthesameonesoftheoriginalproblem. Thisregularizationworksforthelinearcaseifwesupposethatineachclassofthetrainingsetthereisanumberoflinearlyindependentrowsthatisatleastequaltothenumberofthefeatures.Thisisoftenthecaseand,sincethenumberofpointsinthetrainingsetismuchgreaterthanthenumberoffeatures,Ker(G)andKer(H)havebothdimension1.Inthiscase,theprobabilityofanontrivialintersectioniszero. Inthenonlinearcasethesituationisdierent.Usingthekernelfunction( 5{4 ),eachelementofthekernelmatrixis Let 89

PAGE 90

5{1 )becomes: minu;6=0kK(A;C)uek2 Now,theassociatedeigenvalueproblemhasmatricesofordern+1andrankatmostd.Thismeansaregularizationtechniqueisneeded,sincetheproblemcanbesingular. Weproposetogeneratethefollowingtwoproximalsurfaces: bysolvingthefollowingproblem minu;6=0kK(A;C)uek2+k~KBuek2 where~KAand~KBarediagonalmatriceswiththediagonalentriesfromthematricesK(A;C)andK(B;C).Theperturbationtheoryofeigenvalueproblems[ 66 ]providesanestimationofthedistancebetweentheoriginalandtheregularizedeigenvectors.Ifwecallzaneigenvectoroftheinitialproblemandz()thecorrespondingoneintheregularizedproblem,thenjzz()j=O(),whichmeanstheirclosenessisintheorderof. Asmentionedintheprevioussection,theminimumandthemaximumeigenvaluesobtainedfromthesolutionof( 5{13 )providetheproximalplanesPi,i=1;2toclassifythenewpoints.Apointxisclassiedusingthedistance andtheclassofapointxisdeterminedas 90

PAGE 91

Figure5-2. ReGECalgorithm TheproposedclassicationmethodisoutlinedinFigure 5-2 .Here,K(A;C;)isthekernelmatrixwhoseentriesareK(i;j)=expf(jjaicijj2=)g,whereaiistheithrowofmatrixAandcjisthejthrowofmatrixC,andistheshapeparameterofthekernel.Functionones(nrow;ncol)isamatrixofsizenrowncolwithallentries1,anddiag()returnsthemaindiagonalofasquarematrix. 67 ],Odewahnetal.[ 68 ],andIDArepository[ 69 ].Theserepositoriesarewidelyusedtocomparetheperformanceofnewalgorithmstotheexistingmethods.Theresultsregardingthelinearkernelhavebeenobtainedusingthersttworepositories.Thethirdonehasbeenusedinthenon-linearkernelimplementation.Foreachdataset,thelatterrepositoryoers100predenedrandomsplitsintotrainingandtestsets.Forseveralalgorithms,results 91

PAGE 92

Classicationaccuracyusinglinearkernel. obtainedfromeachtrial,includingSVMs,arerecorded.TheaccuracyresultsforthelinearkernelSVMsandGEPSVMaretakenfromMangasarianetal.[ 16 ]andforthenonlinearkernelfrom[ 69 ].ExecutiontimesandtheotheraccuracyresultshavebeencalculatedusinganIntelXeonCPU3.20GHz,6GBRAMrunningRedHatEnterpriseLinuxWSrelease3withMatlab6.5,duringnormaldaylightoperations.MatlabfunctioneigforthesolutionofthegeneralizedeigenvalueproblemhasbeenusedforGEPSVMandReGEC.ThelatestreleasesforLIBSVM[ 15 ]andSVMlight[ 70 ]havebeenusedtocomparethesemethodswithSVMs. Intables 5-1 and 5-2 ,classicationaccuracyusinglinearandgaussiankernelshavebeenevaluated.Tablescolumnsrepresent:datasetname,thenumberofelementsinthetrainingset(n+k),thenumberofelementsinthetestsetandtheaccuracyresultsforReGEC,GEPSVMandSVMs.InTable 5-1 ,theaccuracyresultshavebeenevaluatedusingtenfoldcrossvalidation.Intable 5-2 ,therandomsplitsofIDArepositoryhavebeenused.Inthelinearcasecomparableaccuracyresultshavebeenobtainedbythethreemethods.Usingthegaussiankernel,ReGECandGEPSVMshowsimilarbehavioryieldingalwaysresultsslightlylowerthanSVMs. InTables 5-3 and 5-4 ,elapsedtimeisreported.InthelinearcaseReGECandGEPSVMoutperformSVMsimplementations(LIBSVMandSVMlight)inallcases.FurthermoreReGECisatleasttwicefasterthenGEPSVM.Whenthegaussiankernelisused,SVMsimplementationsachievebetterperformanceswithrespecttotheeigenvaluesbasedmethods.Inallcases,ReGECisfasterthanGEPSVM. 92

PAGE 93

Classicationaccuracyusinggaussiankernel. Table5-3. Elapsedtimeinsecondsusinglinearkernel. Table5-4. Elapsedtimeinsecondsusinggaussiankernel. 93

PAGE 94

SeparationsurfacesobtainedwithReGEC,GEPSVMandLIBSVM Finally,agraphicalrepresentationoftheclassicationsurfacesobtainedbyReGEC,GEPSVMandSVMsisgiveningure 5-3 relativelytoBananadataset.Thethreemethodsshowsimilarclassregions.SVMsobtainsmootherbordersandmoreregularregions.ThesedierencesdependuponthefactthatinSVMsthesurfacesarecharacterizedbythesupportvectorsandthepenaltiesterms,whileintheeigenvaluesmethodsallthepointscontributetothesolutionsurfaces.Thisbehaviordependsonthefactthateigenvaluesmethodsalwaysmaximizetheclassicationaccuracyonthetrainingsetwithrespecttokernelandregularizationparameters. 94

PAGE 95

5-2 ,linearalgebraoperationsareessentiallymatrix-matrixmultiplicationsandageneralizedeigenvalueproblemsolution.Inordertoobtainanecient,portableandscalableparallelimplementationofReGECwedecidedtousestandardmessagepassinglibraries,i.e.BLACSandMPI,anddefactostandardnumericallinearalgebrasoftware,PBLASandScaLAPACK.Sincematricesinvolvedinthealgorithmaredistributedamongprocessingnodes,memoryisusedecientlyandnoreplicationofdataoccurs.Onsinglenode,theuseofoptimizedlevel3BLASandLAPACKroutinesenablesbothitsecientuseandafavorablecomputation/communicationratio. ThemainroutineofPBLASusedintheimplementationofFigure 5-2 isPDGEMMtoevaluatematrix-matrixmultiplications.ThecurrentmodelimplementationofthePBLASassumesthematrixoperandstobedistributedaccordingtotheblockscatterdecompositionofPBLASandScaLAPACK.RoutinesforeigenvaluesproblemsarenotincludedinPBLAS,buttheyarecoveredbyScaLAPACK.TheevaluationofthegeneralizedeigenvalueproblemGx=HxthenperformedbyusingtheroutinePDSYGVX.Werequiredmachineprecisioninthecomputationofeigenvaluesand,dynamicallyallocatedmemoryforreorthogonalizationofeigenvectors.CurrentversionofScaLAPACKdoesnotpermittoreorthogonalizeeigenvectorsagainstthoseindierentprocessorsmemory,whichcanleadtoslightlydierentresults,withrespecttosequentialcomputation. Wedevelopedtheauxiliaryroutinesforparallelkernelcomputation,andfordiagonalmatricesoperations.ParallelkernelroutineisderivedbythedistributionroutinePDMATDISimplementedinHPEC,whichloadsmatricesfromlesanddistributestoprocessors,accordinglytotheblockscattereddecomposition.Itpermitstoappropriately 95

PAGE 96

Finally,theoperationcountofparallelReGECisexactlythesameasthesequentialone.Thankstocomputationalcharacteristicsoflinearalgebrakernels,theparallelimplementationofthealgorithmdescribedinFigure 5-2 hasacomputationalcomplexityonpnodesthatisexactly1=pofthesequentialone,andacommunicationcomplexityofoneordermagnitudelessthancomputationalone.Thisisusuallyatargetintheimplementationofparallellinearalgebrakernels,becauseitassuresscalableimplementations. 71 ].ThepredictionofTISinagenomicsequenceisanimportantissueinbiologicalresearch.Thisproblemcanbestatedasaclassicationproblemand,althoughsometechniquesexist,thereisagreatpotentialfortheimprovementoftheaccuracyandspeedofthesemethods.Moreover,itprovidesasignicantcasestudyfortheanalysisofgenomicsequences.TheaforementionedmethodhasbeentestedonbenchmarkdatasetsobtainedfromtheTIS.Resultsregardperformanceintermsofexecutiontimeandeciency.ExecutiontimesandtheotheraccuracyresultshavebeencalculatedusingaBeowulfclusterof16Pentium41.5GHz,with512MBRAM,connectedwithaFastEthernetnetwork.EachnoderunsaLinuxkernel2.4.20,gcccompiler2.96,mpich1.2.5,BLACS1.1,ScaLAPACK1.7,LAPACK3.0,BLASwithATLASoptimization.Testshavebeenperformedonidleworkstations;thetimereferstowallclocktimeoftheslowerexecutingnodeandithasbeenmeasuredwithfunctionMPI WTIME()providedbympich.Themaximummemoryavailableoneachnodeledtotheimpossibilitytorunsometestcasesonasmallnumberofprocessors. TheexecutiontimesandparalleleciencyareshowninTables 5-5 and 5-6 ,usingdierentnumberofeithertrainingelementsandCPU.Testshavebeenperformedon 96

PAGE 97

Table5-5. Executiontimesfortheparallelimplementation 500 Intable 5-6 theeciencyiscalculatedusingthefollowingformula: wheret#istheexecutiontimeusing#numberofcpu.Inallcasesforwhichwecouldnotevaluatesequentialorparallelexecutiontimeonasmallnumberofnodes,weseteciencyto1ontheminimumnumberofprocessorsonwhichwecouldruntheapplication. Table5-6. Eciencyfortheparallelimplementation 500 Resultsshowthat,foranincreasingnumberofprocessors,theexecutiontimedecreasesproportionally,iftheproblemtobesolvedhassucientcomputational 97

PAGE 98

Datasetsinalmosteveryapplicationareaareevergrowingandarecontinuouslyupdated.Moreover,numerousapplicationsonmassivedatasetsareemerging[ 44 ],whichrequireecientcomputationalprocedurestorespondtothedynamicsoflargedatabases.Asmachinelearningbecomesapartofdataintensivecomputationsystems,updatingthelearningsystembecomesintractableinmanycases.Therefore,incrementalmethodsthatrequiresomeminimalcomputationalburdenarestronglypreferred.Forthispurposeseveralmethods,especiallyinthekernel-basednonlinearclassicationcases,havebeenproposedtoreducethesizeofthetrainingset,andthus,therelatedkernel[ 45 { 49 ].Allofthesemethodsshowthatasensibledatareductionispossiblewhilemaintainingacomparablelevelofclassicationaccuracy. Inthisstudy,anewmethodthatndsasmallsubsetofthetrainingdatasetisintroduced.Theamountofreductioninthetrainingsetcanbeaslargeas98%withcomparableclassicationaccuracyandimprovedconsistencywithrespecttotheoriginaltrainingset.Theproposedsubsetselectionmethodstartswithaninitialsetofpointsandincrementallyexpandsthissetbyaddingthosepointswhichcontributetoimproving 98

PAGE 99

5{13 )isequalton,thenumberofpointsinthetrainingset,plus1.SincethecomputationalcomplexityoftheoperationisintheorderofO((n+k)3),itisimportanttodevelopmethodsthatarecapableofndingasmallandrobustsetofpointsthatretainsthecharacteristicsoftheentiretrainingsetandprovidescomparableaccuracyresults.Akernelbuiltfromasmallersubsetiscomputationallymoreecientinpredictingnewpointscomparedtokernelsthatusetheentiretrainingset.Furthermore,asmallersetofpointsreducestheprobabilityofover-tting.Finally,asnewpointsbecomeavailable,thecostofretrainingthealgorithmdecreasesiftheinuenceofthenewpointsontheclassicationfunctionisonlyevaluatedbythesmallsubset,ratherthanthewholetrainingset.Themainideaistoexploittheeciencyofsolvingasmalleigenvalueproblem.Therefore,weuseReGECastheinternalmethodtoevaluatetheclassicationaccuracyontheentiretrainingset. ThealgorithmtakesaninitialsetofpointsC0andtheentiretrainingsetCasinput,suchthatCC0=A0[B0,andA0andB0aresetsofpointsinC0thatbelongtothetwoclassesAandB.WerefertoC0astheincrementalsubset.Let0=CnC0betheinitialsetofpointsthatcanbeincludedintheincrementalsubset.ReGECclassiesallofthepointsinthetrainingsetCusingthekernelfromC0.LetPA0andPB0bethehyperplanesfoundbyReGEC,r0betheclassicationaccuracyandM0bethepointsthataremisclassied.Then,amongthepointsin0\M0thepointthatisfarthestfromitsrespectivehyperplaneisselected,i.e. 99

PAGE 100

5-4 agraphicalexampleofthisapproachisshown.Theclassicationsurfacesofthetwoclasses(darkandwhite),generatedusing400trainingpointsoftheBananadataset[ 69 ],clearlydenetheaimofourstrategy.Indeed,whentheReGECalgorithmistrainedonallofthetrainingpointstheclassicationboundariesaresignicantlyaectedbynoisypoints(left).Ontheotherhand,I-ReGECmethodachievesclearlydenedboundaries(right).Furthermore,thenumberofpointsneededintheexampletogeneratetheclassicationhyperplaneareonly23inI-ReGECcomparedto400pointsinReGEC. 100

PAGE 101

ClassicationsurfacesproducedbyReGEC(left)andI-ReGEC(right) 5.4.1 ,weassumedthatwehaveastartingsetofpointsforI-ReGEC.However,wehavenotmentionedthebiasthisinitialsetintroduces.Sincetheinitialpointspermanentlybecomeapartoftheincrementalsubset,itisintuitivethatsuchpointsshouldbechosencarefully.Inthissectionweshowhowtheinitialsetofpointsinuencetheperformanceoftheincrementalselectionalgorithm.Clusteringtechniquescanbeadaptedtoobtainbetterdatarepresentations[ 72 ].Forthispurpose,wecomparekrandomlyselectedstartingpointsforeachclass,andasetofpointsdeterminedbyasimplek-meansmethod[ 73 ],alsoforeachclass.Weshowthatitispossibletoreachhigherclassicationaccuracyandamoreconsistentrepresentationofthetrainingsetusingk-meansmethod. Thetwodatasetsusedforthecomparisonhave2dimensions,inordertoshowtheconsistencyofthek-meansmethodoverrandomselection,graphically.Fromeachclass,kpointsarechosenforbothrandomandk-meansmethods.TherstdatasetistheBananadatasetwith400trainingpointsand4900testpoints.ThesecondsetofpointsistheChessboarddataset.Itcontains16squares,withatotalof1000trainingand5400testpoints. First,classicationparametersaredeterminedusingaten-foldcross-validationusingthetrainingandtestpoints.Aninitialsetofstartingpointsischosena)randomly,and 101

PAGE 102

InFigure 5-5 ,whitecolorisassociatedtothepointsforwhich^y=1andblackfor^y=0:5.Thelighterregionsaremoreconsistentcomparedtodarkregions,wherethepointshavethesameprobabilitytobeclassiedinoneofthetwoclasses.Theinuenceofthestartingpointsontheresultingclassicationcanbeseenclearly.TheBananadatasethasfewclustersofdataandconsequently,forachoiceofk=5,theaverageclassicationaccuracyslightlychangesbetweenrandominitialpoints,whichproduceaclassicationaccuracyof84.5%,andk-meansinitialpoints,withaccuracyof85.5%.Inordertocomparetheconsistencyofthetwoinitialpointsselectionstrategies,wemeasurethestandarddeviationofthe^yvaluesforthepointsintherectangle.Thek-meansmethodacievesastandarddeviationof0.01comparedtothestandarddeviationof0.05fromtherandommethod,whichmeansthatk-meansmethodhasahigherclassicationconsistencythanrandomselection. FortheChessboarddataset,theclustersareclearlyseparatedforeachclasswhenk=8.Thedierenceismorepronouncedbothintermsofclassicationaccuracyandconsistency.Randomselectionofinitialpointscouldonlyreachaclassicationaccuracy 102

PAGE 103

ClassicationconsistencyofI-ReGECforrandomselection(left)andk-menas(right) of72.1%,whereask-meansreaches97.6%accuracy.ThedierenceinclassicationconsistencyisfarmoreevidentcomparedtotheBananadataset,withastandarddeviationof1.45forrandomselectionand0.04fork-means.Wecanempiricallyinferfromtheresultsthataknowledgeregardingthedatasetandthechoiceofinitialpointsinuencesbothclassicationaccuracyandclassicationconsistency.Thisinuencemaybegreaterasthenumberofclustersincreases. Wealsoinvestigatedtheeectofthenumberofinitialpointskforeachclassusingthek-meansmethodontheChessboarddataset.InFigure 5-6 ,thegraphontopistheclassicationaccuracyversusthetotalnumberofinitialpoints2kfrombothclasses.Itreachesitspeakat16(fork=8),afterwhichitslightlydecreasesandcontinuesatasteadystateofaccuracyforhighervaluesofk.Thisresultempiricallyshowsthatthereisaminimumk,withwhichwereachhighaccuracyresults.Althoughthedecreaseintheaccuracyisnotsignicantforlargervaluesofk,thekerneltobeusedinI-ReGEC 103

PAGE 104

5-6 whichshowsthenumberofpointsselelctedbyI-ReGECversusthenuberofinitialpoints.Again,noadditionalpointsareaddedtotheinitial16(fork=8),andthenumberofpointsaddedarealmostthesamebeyond.Thismeansthattheinitialsetofpointsreachesaminimumatanidealnumberofkanditgrowslinearlywithk.Onesimpleandpracticalwayofndingagoodkistoincreasekincrementallyanddetectingthelowestvalueofkwithhigherclassicationaccuracy. Figure5-6. PerformanceofI-ReGECwithrespecttothenumberofstartingpoints 69 ]andfromUCI[ 67 ]repositories,bothofwhicharewidelyusedtocomparetheperformanceofnewalgorithmstoexistingmethods.Theaccuracyresultsforthenonlinearkernelaretakenfrom[ 69 ].AccuracyresultsarecalculatedusinganIntelXeonCPU3.20GHz,6GBRAMrunningRedHatEnterpriseLinuxWSrelease3withMatlab6.5.MatlabfunctioneigforthesolutionofthegeneralizedeigenvalueproblemisusedforReGEC. InTable 5-7 ,foreachdataset,name,dimensionofthetrainingandtestsets,andthenumberoffeaturesarereported.InTable 5-8 ,classicationaccuracyisevaluatedusingGaussiankernelforReGEC,I-ReGEC,andSVM,usingten-foldcross-validationto 104

PAGE 105

Datasetscharacteristics traintestm 40049002 German 70030020 Diabetis 4683008 Haberman 275314 Bupa 310356 Votes 3914416 WPBC 991132 Thyroid 140755 Flare-solar 6664009 determineparameters.AGaussiankernelisusedforeachclassierandthevalueofthebestkernelparametertogetherwiththekvalueforthek-meansmethodforI-ReGECarealsoincludedinthetable.Thekvalueforeachdatasetisempiricallydeterminedasfollows:rst,thebestvalueisdeterminedfork=2usingten-foldcross-validation;then,thebestkvalueisdeterminedbygraduallyincreasingitsvalue. I-ReGECisnearlyalwaysmoreaccuratethanReGEC.TheslightdierenceinaccuracyforthetwodatasetswhereReGECgivesbetterresultscouldbeduetothecrossvalidationprocedure.WehavealsocomparedtheaccuracyresultsofI-ReGECwithSVM.ResultsarealwaysslightlylowerthanSVM,exceptforonedataset.Therelativedierenceofaccuracy,i.e.,theabsolutedierenceoftheaccuraciesofI-ReGECandSVM,dividedbythemaximumvalue,islessthen8.2%,exceptthecaseofFlare-solar(11.50%)andBupadataset(15.55%). InTable 5-9 thedimensionofincrementaldatasetsandthepercentagewithrespecttothedimensionofthetrainingsetisgiven.Inallcases,I-ReGECproducedasubsetcomposedoflessthen8.85%ofthetrainingsetwithacomparableclassicationaccuracyonthetestsetswithrespecttotheoriginalReGECmethod. 105

PAGE 106

ClassicationaccuracyforReGEC,I-ReGECandSVM ReGEC I-ReGEC SVMs chunkkacc acc 0.284.44 15.750.285.49 89.15 50070.26 29.0981073.50 75.66 50074.56 16.63540074.13 76.21 120073.26 7.5922000073.45 71.70 20059.03 15.28480063.94 69.90 5095.09 25.901010093.41 95.60 100058.36 4.2025060.27 63.60 0.892.76 12.4051.5094.01 95.20 358.23 9.673365.11 65.80 IncrementaldatasetusingI-ReGECandpercentageofthetrainingset I-ReGEC 106

PAGE 107

Theaimofthisdissertationwastostudycoreproblemsinrecentlydevelopedveryecientandsuccessfulmachinelearningmethods.Thecommoncharacteristicforthesemachinelearningmethodsisthattheycanaccommodateakernelfunctionwhichimplicitlydenesamapfromtheoriginalinputspacetoafeaturespace.Themappedfeaturespacecanhandlenonlinearsurfacesforclassicationandforndingnonlinearregressionfunctions.Asidefromthestandardsupportvectorclassicationandsupportvectorregressionmethods,someotherkernel-basedfeatureselectionalgorithmswerealsoreviewed,suchastheadaptivescalingforSVMs,whichusesaiterativecombinationofstandardsupportvectormachinesandaconjugategradienttechniquetondtherelativecontributionofeachfeaturetoclassication.Thestandardmethodshavebeenperfectedoveryearsanditishardtocomeupwithimprovementmethods,buttheotheralgorithms,suchasthefeatureselectionmethodcanbecomparedwithasimplerfeatureselectiontoseehoweectiveitis. Selectivesupportvectormachineshavebeenshownonvisualexamplestoworkonndingthemostseparatedpointsoutofmanypointsperset.Thesimilarresultsalsoapplyforregression,althoughtheresultsarenotasprominentasSelSVM.Thepatternvectorsarechosenasfarfromeachotheraspossiblefordierentclasses,regardlesswhetherthedatasetisperfectlyseparableornot,whichshowsrobustness.Infact,forhighervaluesofthebandparametere(),thedistributiontakestheshapeofalinearclassier.Fromatheoreticalpointofview,selectivesupportvectormachines,beinganovelmethodofselectiveclassication,havealottoexplore,startingwiththeeortstosolvethehardselectionproblemusingintegerprogrammingandcombinatorialoptimizationtechniques.Otherkernel-basedmethods,whoseselectiveversionsarenotpresentedherecanbedevelopedbasedonthesamepremise.Forexample,selectiveproximalsupportvectormachinescanbedevelopedeasily,studiesofwhichareongoing.Asfarasthecurrentrelaxationgoes,dierentschemescanbeintroducedtodistribute 107

PAGE 108

Westudiedtheintegrationofvisualandmotorcortexesinmacaquemonkeysusingkernel-basedmachinelearningalgorithms.Themaincontributionofthestudyisthattherecordingsfromalargesetofcortexesinvolvedinthevisuomotortaskareconsideredsimultaneouslyusingnon-linearmapping,asopposedtothepreviousstudiesthatrevolvedaroundmethodsthatstudyindividualchannels. Throughclassication,wedetectedtheonsettimesfortheautonomousprocessingofthevisualstimuliintheprimaryvisualcortex(V1).Wehaveclearlydetectedthedierencesbetweenstimuliwhichhasdierentalignment(rightorleftslanted)aseachstimulihasadetailedmapofthespatialinformationinvision.Inourexperimentalparadigm,thevisualdiscriminationtaskinvolvesdiscriminationoflineanddiamondstimuliregardlessoftheiralignment.Weobservedthatthediscriminationistime-lockedtotheearlystimulusonsetinthestriatecortex.TheresponserelateddierencesinvolvemotorplanningandexecutionoftheresponsetotheGOstimuli.Theexecutionisdependentonthediscriminationofthestimulus,andtheGOcueforthespecicsession.WeobservedthatdierencesintheGOvsNOGOresponsesistime-lockedtothediscriminationphase. Theonsetandthedurationofthestimuluscomefromseparatedistributionsforeachofthevisual,categoricalandresponserelatedstages.Thestagesarehypothesizedtobesequentiallydependent.Therefore,attheindividualtriallevel,adelayontheonsetorosetofastagewouldcausefurtherdelaysinthesubsequentstages.Weuseddynamictimewarpingalgorithmtondthosesingletrialswithineachsessionandstimulustypetoreducevariability,whichresultedinsignicantimprovementsinbothclassicationandchannelselectionmethods.Thisshowsthat,withanappropriatemethod,thesingletrialscanbealigned.Moreoverthisalignmentmayoerinsightstothevariabilityoftheunderlyingstagesofthevisualdiscriminationtask. 108

PAGE 109

TheselectiveSVMstudymayeasilybeextendedtoothermacaquemonkeysandsimilarpatternsmaybesearchedtounderstandtheintegrationofthevisualandmotorcortexesinabroadersense.Infact,selectivesupportvectorregressioncanalsobeusedtomakeapredictionontheresponsetimesforgotrialsoveraslidingwindow.Apossiblescenarioistocomparetheresultsofclassicationandregressionforthesametimepointsandwindowsize. SelectiveSVMscaneasilybeimplementedwhenevertherearerepetitivetimeseriestobecategorizedforeachtimepoint.Suchapplicationsmayinvolverecordingsfromepilepticpatientswhosenervoussystemsareperiodicallystimulated.Inasense,therepeatedepochscanbetreatedassingletrialdata,anddynamicsofthebrainasaresponsetothisstimulationcanbestudiedtoanalyzeandunderstandhowepilepticseizuresdevelop,andwhatcanbedonetostopthem.Briey,anydatabaseoftimeseriesthatarerepetitions 109

PAGE 110

Researchactivitiesrelatedtosupervisedlearninghaveanimportantroleinmanyscienticandengineeringapplications.InthepresentworkanovelregularizationtechniqueregularizedgeneralizedeigenvalueclassierReGECanditsapplicationhasbeenproposedandtestedagainstothermethodsonanumberofdatasets.Resultsshowthattheproposedmethodhasaclassicationaccuracycomparabletoothermethods,hasacomputationalperformancecomparabletomostoftheothermethods,andismuchfasterthentheothersinthelinearcase.TheseadvantagesarestrongmotivationsforReGECtobeusedinaparallelcomputationalplatforms.Parallelimplementationofthisalgorithmprovidetimeeciencyandcomputationalaccuracy.Theproposedimplementationistestedonalargescalegenomicdatabase.Thepreliminaryresultsshowtheproposedimplementationtobeecientandscalable.Thefutureworkmayincludetestingthisimplementationonlargescaledatasetsfromdierentresearchareas,mostlyinthebiomedicaldomain,andcomparingitsperformancewiththeotherparallelclassicationmethods. Forfurthercomputationaladvantagewithoutsubstantialsacriceontheresults,I-ReGEC,anovelincrementalclassicationtechniqueisintroduced,withdramaticresultsinreducingthecardinalityoftrainingsets,whenappliedtogeneraleigenvalueclassiers.TheproposedmethodachievesahighclassicationconsistencyandclassicationaccuracycomparablewithotherSVMmethods.Furthermore,itallowsecientonlineupdatingoftheclassicationfunctionwhennewtrainingpointsbecomeavailable.I-ReGECmethodcanbeimprovedbyadaptivetechniquesfortheselectionoftheinitialpoints,inordertondbetterstrategiestobuildtheincrementalsubset.Furthermore,newcriteriaforincludingnewpointstotheincrementalsubsetorremovinglesspromisingpointsfromtheincrementalsubsetmaybeconsidered. 110

PAGE 111

[1] V.Vapnik.TheNatureofStatisticalLearningTheory.Springer-Verlag,1995. [2] J.Shawe-TaylorandN.Cristianini.Kernelmethodsforpatternanalysis.CambridgeUniversityPress,Cambridge,UK,2004. [3] S.LeeandA.Verri.Patternrecognitionwithsupportvectormachines.InSVM2002,NiagaraFalls,Canada,2002.Springer. [4] T.Joachims.Textcategorizationwithsupportvectormachines:Learningwithmanyrelevantfeatures.InClaireNdellecandClineRouveirol,editors,ProceedingsoftheEuropeanConferenceonMachineLearning,pages137{142,Berlin,1998.Springer. [5] C.CifarelliandG.Patrizi.Solvinglargeproteinfoldingproblembyalinearcomplementarityalgorithmwith0-1variables.OptimizationMethodsandSoftwares,2005.Submittedforpublication. [6] W.GrundyM.Brown,D.Lin,N.Cristianini,C.Sugne,T.Furey,M.Ares,andD.Haussler.Knowledge-baseanalysisofmicroarraygeneexpressiondatabyusingsupportvectormachines.PNAS,97(1):262{267,2000. [7] W.S.Noble.KernelMethodsinComputationalBiology,chapterSupportvectormachineapplicationsincomputationalbiology,pages71{92.MITPress,2004. [8] G.N.Garcia,T.Ebrahimi,andJ.M.Vesin.Jointtime-frequency-spaceclassicationofeeginabrain-computerinterfaceapplication.JournalonAppliedSignalProcessing,pages713{729,2003. [9] T.N.Lal,M.Schroeder,T.Hinterberger,J.Weston,M.Bogdan,N.Birbaumer,andB.Schlkopf.Supportvectorchannelselectioninbci.IEEETransactionsonBiomedicalEngineering,51(6):1003{1010,2004. [10] Z.Huang,H.Chen,C.J.Hsu,W.H.Chenb,andS.Wuc.Creditratinganalysiswithsupportvectormachinesandneuralnetworks:amarketcomparativestudy.DecisionSupportSystems,37:543{558,2004. [11] T.B.TrafalisandH.Ince.Supportvectormachineforregressionandapplicationstonancialforecasting.InInternationalJointConferenceonNeuralNetworks(IJCNN'02),Como,Italy,2002.IEEE-INNS-ENNS. [12] R.F.E.OsunaandF.Girosi.Animprovedtrainingalgorithmforsupportvectormachines.InIEEEWorkshoponNeuralNetworksforSignalProcessing,pages276{285,1997. [13] J.Platt.AdvancesinKernelMethods:SupportVectorLearning,chapterFasttrainingofSVMsusingsequentialminimaloptimization,pages185{208.MITpress,Cambridge,MA,1999. 111

PAGE 112

[14] T.Joachims.Makinglarge{scaleSVMlearningpractical.InB.Scholkopf,C.J.C.Burges,andA.J.Smola,editors,AdvancesinKernelMethods|SupportVectorLearning,pages169{184,Cambridge,MA,1999.MITPress. [15] C.W.Hsu,C.C.Chang,andC.J.Lin.Apracticalguidetosupportvectorclassication.http://www.csie.ntu.edu.tw/cjlin/papers/guide/guide.pdf,2004. [16] O.L.MangasarianandE.W.Wild.Multisurfaceproximalsupportvectorclassicationviageneralizedeigenvalues.TechnicalReport04-03,DataMiningInstitute,September2004. [17] V.N.Vapnik.Thenatureofstatisticallearningtheory.SpringerVerlag,NewYork,1995. [18] J.B.Gao,S.R.Gunn,andC.J.Harris.Meaneldmethodforthesupportvectormachineregression.Neurocomputing,50:391{405,2003. [19] J.B.Gao,S.R.Gunn,andC.J.Harris.Svmregressionthourghvariationalmethodsanditssequentialimplementation.Neurocomputing,55:151{167,2003. [20] O.L.MangasarianandD.R.Musicant.Largescalekernelregressionvialinearprogramming.MachineLearning,46:255{269,2002. [21] N.Cristianini,C.Campbell,andJ.Shawe-Taylor.Dynamicallyadaptingkernelsinsupportvectormachines.InMITPress,editor,AdvancesinNeuralInformationProcessingSystems11,1999. [22] Y.GrandvaletandS.Canu.Adaptivescalingforfeatureselectioninsvms.InMITPress,editor,AdvancesinNeuralInformationProcessingSystems15,2003. [23] J.Weston,S.Mukherjee,O.Chapelle,M.Pontil,T.Poggio,andV.Vapnik.FeatureselectionforSVMs.InNIPS,pages668{674,2000. [24] P.S.Bradley,O.L.Mangasarian,andW.N.Street.Featureselectionviamathematicalprogramming.JournalonComputing,10(209{217),1998. [25] W.Bair,J.R.Cavanaugh,M.A.Smith,andJ.A.Movshon.Thetimingresponseonsetadosetinmacaquevisualneurons.JournalofNeuroscience,22:3189{3205,2002. [26] J.BullierandL.G.Nowak.Parallelversusserialprocessing:newvistasonthedistributedorganizationofthevisualsystem.CurrOpinNurobiol,5:494{503,1995. [27] L.Fogassi,P.F.Ferrari,B.Gesierich,S.Rozzi,F.Chersi,andG.Rizzolati.Parietallobe:fromactionorganizariontointentionunderstanding.Science,308:662{667,2005. [28] L.G.NowakandJ.Bullier.CerebralCortex,volume12,chapterThetimingofinformationtransferinthevisualsystem.PlenumPress,NewYork,1997.

PAGE 113

[29] D.L.RobinsonandM.D.Rugg.Latenciesofvisuallyresponsiveneuronsinvariousregionsofrhesusmonkeybrainandtheirrelationtohumavisualresponses.BioPsychol,26:111{116,1988. [30] M.T.Schmolesky,Y.Wang,D.P.Hanes,K.G.Thompson,S.Leutgeb,J.D.Schall,andA.G.Leventha.Signaltimingacrossthemacaquevisualsystem.JournalofNeurophysiology,79:3272{3278,1998. [31] S.Vanni,M.Dojat,J.Warnking,C.Delon-Martin,C.Segerbarth,andJ.Bullier.Timingofinteractionsacrossthevisualledinthehumancortex.JournalCognNeurosci,21:818{828,2004. [32] J.JFoxeandG.v.Simpson.FlowofactivationfromV1tofrontalcortexinhumans.aframeworkfordening"early"visualprocessing.ExpBrainRes,142:139{150,2002. [33] C.E.Schroeder,A.D.Mehta,andS.J.Givre.Aspatiotemporalproleofvisualsystemactivationrevealedbycurrentsourcedensityanalysisintheawakemacaque.CerebCortex,8:575{592,1998. [34] M.MishkinandL.G.Ungerleider.Contributionofstriateinputstothevisuospatialfunctionsofparieto-preoccipitalcortexinmonkeys.BehavBrainRes.,6:57{77,1982. [35] K.Tanaka.Representationofvisualfeaturesofobjectsintheinferotemporalcortex.NeuralNetw.,9:1459{1475,1996. [36] F.G.AshbyandB.J.Spiering.Theneurobiologyofcategorylearning.BehavCognNeuroscieRev,3:101{113,2004. [37] D.J.Freedman,M.Riesenbuger,T.Poggio,andE.K.Miller.Acomparisonofprimateprefrontalandinferiortemporalcorticesduringvisualcategorization.JNeurosci,23:5235{5246,2003. [38] D.J.Freedman,M.Riesenhuber,T.Poggio,andE.K.Miller.Categoricalrepresentationofvisualstimuliintheprimatepreforontalcortex.Science,291:312{316,2001. [39] D.J.Freedman,M.Riesenhuber,T.Poggio,andE.K.Miller.Visualcategorizationandtheprimateprefrontalcortex:neurophysiologyandbehavior.JNeurophysiol.,88:929{941,2002. [40] K.SasakiandH.Gemba.Electricalactivityintheprefrontalcortexspecictono-goreactionofconditionedhandmovementwithcolourdiscriminationinthemonkey.ExpBrainRes,64:603{606,1986. [41] K.SasakiandH.Gemba."No-GoPotential"intheprefrontalcortexofmonkeys.,pages290{301.SpringerSeriesinBrainDynamics(E.BasarandT.H.Bullock,eds.).Springer-Verlag,Berlin,1989.

PAGE 114

[42] S.J.ThorpeandM.Fabre-Thorpe.Seekingcategoriesinthebrain.Neuroscience,291:260{263,2001. [43] R.VanRullenandS.J.Thorpe.Thetimecourseofvisualprocessing:fromearlyperceptiontodecision-making.JCognNeurosci,13:454{461,2001. [44] J.Abello,P.M.Pardalos,andM.G.C.Resende,editors.Handbookofmassivedatasets.KluwerAcademicPublishers,Norwell,MA,USA,2002. [45] G.CauwenberghsandT.Poggio.Incrementalanddecrementalsupportvectormachinelearning.InNIPS,pages409{415,2000. [46] C.DomeniconiandD.Gunopulos.Incrementalsupportvectormachineconstruction.InFirstIEEEInternationalConferenceonDataMining(ICDM'01),pages589{593,2001. [47] Y.J.LeeandO.L.Mangasarian.RSVM:Reducedsupportvectormachines.InFirstSIAMInternationalConferenceonDataMining,2001. [48] K.LinandC.Lin.Astudyonreducedsupportvectormachines.IEEETransactionsonNeuralNetworks,6(14):1449{1459,2003. [49] L.Ralaivola.Incrementalsupportvectormachinelearning:Alocalapproach.LectureNotesinComputerScience,2130:322{330,2001. [50] M.Cannataro,D.Talia,andP.K.Srimani.Paralleldataintensivecomputinginscienticandcommercialapplications.ParallelComput.,28(5):673{704,2002. [51] F.ProvostandV.Kolluri.Asurveyofmethodsforscalingupinductivealgorithms.DataMin.Knowl.Discov.,3(2):131{169,1999. [52] D.Skillicorn.Strategiesforparalleldatamining.IEEEConcurrency,7(4):26{35,1999. [53] A.Srivastava,E.Han,V.Kumar,andV.Singh.Parallelformulationsofdecision-treeclassicationalgorithms.DataMin.Knowl.Discov.,3(3):237{261,1999. [54] N.CristianiniandJ.Shawe-Taylor.AnIntroductiontoSupportVectorMachines.CambridgeUniversityPress,Cambridge,UK,2000. [55] B.ScholkopfandA.J.Smola.LearningwithKernels.MITPress,Cambridge,2002. [56] T.Poggio.Onoptimalnonlinearassociativerecall.BiologicalCybernetics,19:201{209,1975. [57] S.L.Bressler.Large{scalecorticalnetworksandcognition.BrainResBrainResRev,20:288{304,1995. [58] S.L.Bressler.Interarealsinchronizationinthevisualcortex.BehavBrainRes,76:37{49,1996.

PAGE 115

[59] S.L.Bressler,R.Coppola,andR.Nakamura.Episodicmultiregionalcorticalcoherenceatmultiplefrequenciesduringvisualtaskperformance.Nature,366:153{156,1993. [60] S.L.BresslerandR.Nakamura.ComputationandNeuralSystems,chapterInterareasinchronizationinmacaqueneocortexduringvisualpatterndiscriminationtask,pages515{522.Kluver,Boston,1993. [61] D.J.BerndtandJ.Cliord.Usingdynamictimewarpingtondpatternsintimeseries.InProc.ofAAAIWorkshop:KnowledgeDiscoveryinDatabases,pages359{370,Seattle,Washington,1994. [62] B.N.Parlett.TheSymmetricEigenvalueProblem,page357.SIAM,Philadelphia,PA,1998. [63] K.BennetandC.Campbell.Supportvectormachines:Hypeorhallelujah?SIGKDDExplorations,2(2):1{13,2000. [64] K.BennettandO.Mangasarian.Robustlinearprogrammingdiscriminationoftwolinearlyinseparablesets.OptimizationMethodsandSoftware,1:23{34,1992. [65] Y.Saad.NumericalMethodsforLargeEigenvalueProblems.HalstedPress,NewYork,NY,1992. [66] J.Wilkinson.TheAlgebraicEigenvalueProblem.ClarendonPress,1965. [67] D.J.Newman,S.Hettich,C.L.Blake,andC.J.Merz.UCIrepositoryofmachinelearningdatabases.http://www.ics.uci.edu/mlearn/MLRepository.html,1998. [68] S.Odewahn,E.Stockwell,R.Pennington,R.Humphreys,andW.Zumach.Automatedstar/galaxydiscriminationwithneuralnetworks.AstronomicalJour-nal,103(1):318{331,1992. [69] S.Mika,G.Rtsch,J.Weston,B.Schlkopf,,andK.R.Mller.Fisherdiscriminantanalysiswithkernels.IEEENeuralNetworksforSignalProcessing,IX:41{48,1999. [70] T.Joachims.Makinglarge-ScaleSVMLearningPractical.AdvancesinKernelMethods-SupportVectorLearning.MIT-Press,1999. [71] J.LianH.Liu.Kentridgebiomedicaldatasetrepository.http://sdmc.i2r.a-star.edu.sg/rp/. [72] L.KaufmanandP.J.Rousseeuw.FindingGroupsinData:AnIntroductiontoClusterAnalysis.Wiley,NewYork,1990. [73] J.MacQueen.Somemethodsforclassicationandanalysisofmultivariateobservations.InProceedingsofBerkeleySymposiumonMathStatProbability,1965.

PAGE 116

OnurSerefisadoctoralcandidateinindustrialandsystemsengineeringattheUniversityofFlorida.Hisresearchfocusesondataminingandoptimizationmethodsinbiomedicalapplications.Onurisalsointerestedinproblemsonnetworkowoptimization.OnurreceivedhisbachelorsandmastersdegreesinindustrialengineeringfromMiddleEastTechnicalUniversity(METU)inAnkara,TurkeyinJuly1998andJuly2001,respectively.HeplanstograduatewithhisPh.D.degreeinDecember2006andpursueacareerinacademia,wherehecancontinuehisresearchandotheracademicinvolvement. 116


Permanent Link: http://ufdc.ufl.edu/UFE0017560/00001

Material Information

Title: New Optimization Methods and Applications in Kernel-Based Machine Learning
Physical Description: Mixed Material
Copyright Date: 2008

Record Information

Source Institution: University of Florida
Holding Location: University of Florida
Rights Management: All rights reserved by the source institution and holding location.
System ID: UFE0017560:00001

Permanent Link: http://ufdc.ufl.edu/UFE0017560/00001

Material Information

Title: New Optimization Methods and Applications in Kernel-Based Machine Learning
Physical Description: Mixed Material
Copyright Date: 2008

Record Information

Source Institution: University of Florida
Holding Location: University of Florida
Rights Management: All rights reserved by the source institution and holding location.
System ID: UFE0017560:00001


This item has the following downloads:


Full Text





NEW OPTIMIZATION METHODS AND APPLICATIONS IN K(ERNEL-BASED
MACHINE LEARNING,


















By
ONUR SEREF


A DISSERTATION PRESENTED TO THE GRADUATE SCHOOL
OF THE UNIVERSITY OF FLORIDA IN PARTIAL FULFILLMENT
OF THE REQUIREMENTS FOR THE DEGREE OF
DOCTOR OF PHILOSOPHY

UNIVERSITY OF FLORIDA


2006
































Copyright 2006

by
Onur Seref


































to the memory of my father,

Abdullah Seref









ACK(NOWLEDGEMENTS

First, I would like to thank my supervisory coninittee chair Dr. Panos Pardalos for

his continuous encouragement and guidance. Dr. Pardalos provided many opportunities

that allowed me to become a self-confident and independent acadentician. I would

like to extend my gratitude to my other coninittee nienters Dr. Ravindra AlMll I

Dr. Edwin Romeijn and Dr. Tanter K~ahveci for their valuable feedback. I would

like to include all Industrial and Systems Engineering (ISE) faculty, especially Dr.

Suleynian Tufekci and Dr. Cole Smith for their help. I am very thankful for the great

contribution from Dr. Mingfzhou Ding and other researchers front his lah in the Biomedical

Engineering Department. I would like to thank all my colleagues in the Center for Applied

Optimization Lab for their support, especially Erhun K~undakcioglu for his extra help.

My beloved wife Michelle has been nly 'I;---- -1 support with her caring love and

my daily inspiration, to whom I owe my eternal gratitude. I want to thank my mother

Esnia, my sister Ezgi and my brother K~lvanC whose love and encouragement brought me

to Gainesville and helped me get my Ph.D. I also would like to thank Michelle's family,

Magfdi, Roblyn and Timothy, for their support and love.

I reserve my most special appreciation for my beloved father Abdullah Seref, who

recently passed away. His honesty, humility, philosophy and his passion for science have

ahr-l- .- guided me and ahr-l- .- will.











TABLE OF CONTENTS

page

ACK(NOWLEDGEMENTS ......... . .. 4

LIST OF TABLES ......... .... .. 7

LIST OF FIGURES ......... .. . 8

ABSTRACT ......... ..... . 10

CHAPTER

1 INTRODUCTION ......... .. .. 12

1.1 K~ernel-Based Machine Learning . .... .. 14
1.2 Applications to Neuroscience ....... .. 15
1.3 Regularized Generalized Eigfenvalue Classifiers .. .. .. .. 17

2 K(ERNEL-BASED LEARNING METHODS ..... .. 20

2.1 K~ernels ......... ... . 20
2.2 Generalization Theory ......... .. 24
2.3 Optimization Theory ......... ... 26
2.4 Support Vector Machines ......... .. 30
2.4.1 Maximal Margin Classifier . ..... 31
2.4.2 Soft Margin Classifier ........ .. 34
2.5 Gradient Based Adaptive Scaling . ..... 39
2.6 Support Vector Regression ........ ... .. 40
2.6.1 Quadratic E-Sensitive Loss . ... .. 40
2.6.2 Linear E-Sensitive Loss . .... .. 42

3 SELECTIVE K(ERNEL-BASED METHODS .... .. .. 44

3.1 Selective Support Vector Machines ..... ... .. 44
3.1.1 Selective 2-Norm Support Vector Machines ... .. .. 48
3.1.2 Selective 1-norm Support Vector Machines ... .. .. 51
3.1.3 Selecting Pattern Vectors with Large Margin .. .. .. .. 53
3.2 Selective Support Vector Regression ..... ... .. 54
3.2.1 Selective 2-norm E-Insensitive Regression .. .. . .. 57
3.2.2 Selective 1-norm E-Insensitive Regression .. .. . .. 60

4 KERNEL METHODS APPLIED TO NEURAL DATA .. .. .. 64

4.1 Visual Discrimination Task ........ ... .. 64
4.2 Methods ........... ..... ....... 67
4.3 Classification and C'I .Ill., I Selection Results ... .. .. 67
4.4 Time Series Alignment with Dynamic Time Warping .. .. 72
4.5 Selective Support Vector Machines on Neural Data ... .. .. 74











5 GENERALIZED EIGENVALUE CLASSIFIERS .... .... .. 85

5.1 Generalized Eigenvalue Classifiers ..... .. . 85
5.2 A New Regularized Fast Classification Method .. .. .. .. 87
5.2.1 The New Regfularization Method . .. 88
5.2.2 Computational Results .. .. .. ... .. .. 91
5.3 A Parallel Implementation of the Fast Classification Method .. .. .. 94
5.3.1 Implementation Details . ..... .. 95
5.3.2 Computational Results . ...... .. 96
5.4 An Incremental Classification Algorithm .... ... .. 98
5.4.1 Incremental Subset Selection Algorithm .. . .. 99
5.4.2 Initial Points Selection . ...... .. 101
5.4.3 Computational Results . ...... .. 104

6 CONCLUSION ......... . .. 107

REFERENCES ......... . .. . 111


BIOGRAPHICAL SKETCH ...........











LIST OF TABLES

Table

4-1 Different stages of the visuomotor task .....

4-2 C'!I. .ill. I names and locations for TI, LU and GE ...

5-1 Classification accuracy using linear kernel. .....

5-2 Classification accuracy using gaussian kernel. .....

5-3 Elapsed time in seconds using linear kernel. .....

5-4 Elapsed time in seconds using gaussian kernel. .....

5-5 Execution times for the parallel implementation ....

5-6 Efficiency for the parallel implementation .....

5-7 Datasets characteristics ....

5-8 Classification accuracy for ReGEC, I-ReGEC and SV1\l


pagfe

. 8

. . 6;9

. 92

. 9:3

. 9:3

. 9:3

. . 97

. 97

.. 105

. . 106


5-9 Incremental dataset usingf I-ReGEC and percentage of the training set










LIST OF FIGURES

Figure page

2-1 K~ernel mapping of circular data in 2-D to 3-D. .... .. 21

2-2 Maximal margin classifier ......... . 32

2-3 Soft margin classifier ......... . . 35

2-4 Standard SVR and E-InSenSitive tube around the regression function .. .. .. 40

3-1 Set of pattern vectors, 3 pattern vectors in each set ... .. .. .. 45

3-2 Classification of a separable set in the presence of restricted free slack .. .. 47

3-3 Distribution of pooled free slack (PFS) ...... .. . 48

3-4 Distribution of free slack per set (FSS) ...... .. . 48

3-5 Standard SVM on a separable 3D example for different kernel bandwiths .. 54

3-6 Selective SVM on a separable 3D example: bold points are selected .. .. .. 55

3-7 Selective SVR compared to a naive method for different bandwidths .. .. .. 63

4-1 Approximate placement of electrodes in the tree monkeys .. .. .. .. 65

4-2 Visual stimuli used in the experiment . .... .. 66

4-3 Experimental paradigm ......... . .. 66

4-4 Classification and channel selection results for GE ... ... .. 77

4-5 Classification and channel selection results for LU ... ... .. 78

4-6 Classification and channel selection results for TI .... .. 79

4-7 Brain snapshots ......... . .. 80

4-8 a) Uncoded scores matrix b) Sorted scores c) Coded matrix d) Sorted matrix .81

4-9 a) comparative classification results b)comparative channel selection results .. 82

4-10 Improvement classification for line-diamond (left) and go-nogo (right) .. .. 83

4-11 Improvement in the feature selection for line-diamond (left) and go-nogo (right) 83

4-12 Baseline approach to emphasize improvement (top), difference between actual
improvement and baseline (bottom) . ..... .. 84

5-1 Separation obtained with generalized eigenvectors. .. . .. 87

5-2 ReGEC algorithm ......... . .. 91










5-3 Separation surfaces obtained with ReGEC, GEPSVM and LIBSVM .. .. .. 94

5-4 Classification surfaces produced by ReGEC (left) and I-ReGEC (right) .. .. 101

5-5 Classification consistency of I-ReGEC for random selection (left) and k-menas
(right) .... ........ .......... ... 103

5-6 Performance of I-ReGEC with respect to the number of starting points .. .. 104









Abstract of Dissertation Presented to the Graduate School
of the University of Florida in Partial Fulfillment of the
Requirements for the Degree of Doctor of Philosophy

NEW OPTIMIZATION METHODS AND APPLICATIONS IN K(ERNEL-BASED
MACHINE LEARNING,

By

Onur Seref

December 2006

C'I I!r: Panos 31. Pardalos
Major Department: Industrial and Systems Engineering

In this study, new optimization methods are introduced on kernel-based machine

learning. These novel methods solve real life classification problems, especially those

arising in the biomedical area. The first main contribution of this study is the selective

support vector machine (SelSVAI) classifiers. SelSVAI classifiers are motivated by the

noisy temporal variations in the recordings of repeated cognitive processes, which affect

the performance of standard support vector machine (SVAI) classifiers. In the SelSVAI

classification problem there are sets of possible pattern vectors instead of individual

pattern vectors. SelSVAI classifiers select those pattern vectors front each set that would

nmaxintize the margin between the two classes of selected pattern vectors. SelSVAI is

compared with other standard alignment methods on a neural data set that is used for

analyzing the integration of visual and motor cortexes in the primate brain. Selective

kernel-based methods are then further extended to selective support vector regression

(SelSVR) .

The second main contribution of this study is a fast classifier based on the standard

generalized eigenvalue classifiers (GEC). The regularized GEC (ReGEC) uses a new

regfularization technique which reduces the solution of two eigfenvalue problems in the

original GEC to a single eigfenvalue problem. A parallel intplenientation of ReGEC is

developed to study large scale genontic problems. Finally, an incremental version I-ReGEC

is developed to train large amounts of data efficiently. I-ReGEC incrementally builds a










substantially small subset of the training data with more consistent generalization results.

These classifiers are shown to perform comparably with the best classification methods on

publicly available benchmark classification datasets.









CHAPTER 1
INTRODUCTION

Learning is one the most distinctive innate ability of animals. A combination of

experiences as input for the learning process increases the chance of survival. These

experiences introduce more samaples of similar conditions from which to avoid, such as

the presence of a predator, or those to seek, such as findings a mate. Humans, the most

advanced learners among animals, use their knowledge to create abstract models for

those conditions they have not experienced, but can respond appropriately specific to an

objective. Formally -In' I1:;0s this abstraction can he perceived as a set of rules which

can map an input condition into a set of possible responses. Intelligence can he defined as

the interaction of all such rules that constitute complex cognitive tasks such as 1 -.!i:

pI l .l..ilr_ problem solving, and creating further rules for more complex scenarios with a

dynamic and recursive nature.

In a world, where machines do most of the hard work, eventually, the question

of making them intelligent came into picture. The initial theoretical concepts were

introduced in the first half of the twentieth century, and implementations thrived with the

introduction of computers, in parallel with the development of formal systems in logic and

neural networks. The term
half of the century. One of the first examples of artificial intelligence were implementations

of a chess-pll li-ing program developed in 1951. Deep-Blue, a successor of such programs

running on powerful computers beat the world chess champion Garry K~asparov 46 years

later, in 1997. There were us! lin i accomplishments in machine learning towards the end of

the century with cars equipped with AI systems which can drive for thousands of miles in

traffic without a driver.

One of the us! in i- areas in Al is machine learning, which developed substantially

during 1990s and 2000s, mostly as a result of the introduction of probability theory and

statistical methods in AI. 1\achine learning can he defined as the methods developed to

enable computers to learn. There have been very successful implementations ranging from










language processing to medical diagnosis and financial analysis. Most of these applications

rely on pattern recognition which is mostly concerned with classifying objects based on

their characteristics. C'l. I) Il:teristics of an object are the qualitative and quantitative

measures that distinguish it from other objects, which are also referred to as features.

Similarity between two objects can be evaluated as a function of the differences in a set of

features they possess. Based on their similarity, objects can be grouped into classes. These

classes may be represented in different v-a--s~ such as approximating functions or functions

that define boundaries between classes. Arranging objects into such classes based on their

position relative to these functions is called I Ar-0.:l;H r.>n.

Machine learning within the classification framework can be categorized into two

main classes. Supervised learning refers to the capability of a system to learn from a set

of examples, which is a set of input/output pairs, where the input is usually a vector of

features of an object, and the output is the label for the class this object belongs to. A set

of objects with a feature vector and a class label is called a training set. This set is used

to derive classification functions The trained system is capable of predicting the label of

an object. The term supervised originates from the fact that the labels for the objects in

the training set are provided as input, and therefore were determined by an outside source,

which can be considered as the supervisor. On the contrary, unsupervised learning is the

case where the objects are not labeled with any class information, and learning is about

forming classes of objects based on similarities between their features.

Supervised learning systems applications can be found in many fields. Financial

companies prefers to classify loan requests depending on the features that characterizes

loaner's ability to p li- back. Such features are learned from the historical data. A similar

example is the Internal Revenue Service's predicting tax evaders based on the features of

tax evaders they detected previously. Applications may differ including cases such as a

prediction system that may warn drivers about pedestrians attempting to cross the street.

There are many applications in biology and medicine such as detection of cancer prone










tissues, or remote protein homology detection. Protein folding hased on the DNA sequence

provides important information on its expression level. More examples are available

related to numerical interpolation, handwriting recognition and Montecarlo methods.

1.1 Kernel-Based Machine Learning

developed by V. Vapnik [1], Sup~port V/ector M~achine (SVAI) algorithms are the

state-of-the-art among the classification methods in the literature. These methods classify

pattern vectors which are assumed to belong to two linearly separable sets from two

different classes. The classification function is defined with a hyperplane that separates

both classes. Although there are infinitely many hyperplanes that separate the two classes,

SVAI method finds the hyperplane that maximizes the the distance from the convex

hulls of both classes by solving a quadratic convex optimization problem. The success

and robustness of SVAI classifiers are due to their strong fundamentals on the statistical

learning theory which is based on generalization bounds for SVAI classifiers. These

methods can he extended to the nonlinear cases by embedding the data in a nonlinear

space using kernel functions [2].

SVAI classifiers have been one of the most successful methods in supervised

learning with applications in a wide spectrum of application areas, ranging from pattern

recognition [3] and text categorization [4] to biomedicine [5-7], brain-computer interface

[8, 9], and financial applications [10, 11]. The training part relies on optimization of a

quadratic convex cost function subject to linear constraints. Quadratic programming

(QP) is an extensively studied field of mathematics and there are many general purpose

methods to solve QP problems such as quasi-newton, primal-dual and interior-point

methods. The general purpose methods are suitable for small size problems. For large

problems faster methods are required. These faster methods usually involve chunking

[12] and decomposition [13] techniques, which use subsets of points to find the optimal

hyperplane. SVAI Light [14] and LIBSVAI [15] are among the most frequently used

implementations that use chunking and decomposition methods efficiently.










There are also alternative methods such as Generalized Proxinial SVAI (GEPSVAI)

[16] that approximate the two classes with two hyperplanes instead of separating them.

The support vector machine method has been extended to perform regression. In support

vector regression (SVR), class labels are replaced with scalar quantities of a dependent

variable. The idea of nmaxintizing a margin is inverted by requiring the pattern vectors

to be within a specified distance front the regression function. SVR method was also

developed by V. Vapnik [17], followed by some other efficient intplenientations [18-20].

The variety of the optimization methods and their applications have lead to the evaluation

of the features that define the classification of objects. Feature selection methods were

developed in order to find the most relevant features that contribute to the different

classes of elements [21-24].

One of the main contributions in my dissertation is the novel concept of selective

support vector machines (SelSVAI). This method has n sets of pattern vectors with

at most k pattern vectors in each set, representing n single possible pattern vectors.

All pattern vectors in a set share the same label. The standard SVAI problem can he

considered as a special case of SelSVAI when k = 1. SelSVAI picks an optimal point front

each set in order to nmaxintize the classification margin. The other pattern vectors do not

have an effect on the classification function. The same idea can he applied to support

vector regression with the outcome of selected vectors forming a better regression curve

with less error. However, the selection process is a hard optimization problem. Therefore,

relaxations of this problem are studied and shown to work very well on toy problems as

well as in real life problems.

1.2 Applications to Neuroscience

The goal directed behavior develops through the activities in the integrated

brain cortexes that are involved in sensory and motor processes. Understanding the

structure and dynamics of this integration is one of the usl I i ur goals of cognitive

neuroscience. A great 1 its .ini~y of the studies in the neurophysiology area are limited










to focusing on isolated cortical areas and their associated functions. 1\oreover, the

differences in the experimental designs, recording techniques and analysis methods pose

difficulties in making general inferences on the temporal coordination across cortical areas.

Simultaneous recordings of cortical activity front visual, motor and executive cortical

areas provide the essential information to explore the integration of the visual and motor

processes on a large scale.

The spatioteniporal dynamics of the visual cortical processing and the interaction

between the visual sensory cortex and motor cortex has been extensively studied in

cognitive neurophysiology. [25-31]. In recent studies, a large number of visual cortical

areas has been show to be activated rapidly as a feedforward step after the onset of

the visual stimuli [:32, :33]. This niechanisni serves as to prime the motor system to

prepare for the processing of the response following the determination of the stimulus

category. The ventral visual pathway is known to be involved in the visual pattern

discrimination [:34, :35], where as the categorical discrimination of the stimuli is expected to

be performed in the dorsolateral prefrontal cortex [:36-39]. The response related potentials

are recorded in similar studies, for even no response cases in monkeys [40, 41] and a visual

categorization task in humans [42, 4:3].

The majority of these studies are on visual onset latency focusing on information

processing within the visual system and ignoring the integration with the motor system.

The techniques used in the analysis of cortical activity usually involves first order

statistical measures and linear statistical models. However, it is well known that the

cortical activities and their interactions have a highly non-linear nature. Furthermore,

the ubiquity of the activities in the nervous system and the highly connected structure of

massive neural networks in the brain require methods that can incorporate simultaneous

recordings in an appropriate nmultidintensional domain, rather than studying individual

recordings independently.










The general purpose SVM and the feature selection methods pIli wa critical role

in my dissertation. The main data under study is the local field potentials collected

from multiple channels implanted in different cortical areas of macaque monkeys while

they perform a visual discrimination task that require recognition of a visual pattern

followed by an appropriate response. This task is repeated to create many instances of

the same experiment with different emphasis on different stimuli response combinations.

These differences are grouped as different classes of data for classification. The main

objective is to be able to detect when and where these differences are observed between

classes. An extensive computational effort is required to achieve robust results on a

very complex, multi-dimensional and highly nonlinear data. Support vector machines

are highly capable of delivering these qualities as confirmed by a large number of recent

implementations. The SVM classifiers provide the temporal measure for the differentiation

between the classes of different stages of the visuomotor task. In parallel with these

stages, a feature selection method would provide cortical location of the differentiation

for those intervals with a significant separation between different classes of recordings.

The feature selection method preferred is naturally a kernel-based adaptive scaling

algorithm in order to maintain consistency. This method uses support vector machines as

a sub-procedure together with a combination of a conjugate gradient technique to find the

relative contribution of each channel.

1.3 Regularized Generalized Eigenvalue Classifiers

Datasets in almost every application area are ever growing and are continuously

updated. Moreover, numerous applications on massive datasets are emerging [44], which

require efficient computational procedures to respond to the dynamics of large databases.

As machine learning becomes a part of data intensive computation systems, updating the

learning systems become intractable. Therefore, incremental methods that require minimal

computational burden are preferred. For this purpose several methods, especially in

kernel-based nonlinear classification, have been proposed to reduce the size of the training










set, and thus, the related kernel [45-49]. All of these methods show that a significant data

reduction is possible while maintaining a comparable level of classification accuracy.

The binary classification problem can he formulated as a generalized eigenvalue

problem [16]. This formulation differs from SV1\s since, instead of finding one hyperplane

that separates the two classes, it finds two hyperplanes that approximate the two classes.

The prior studies require the solution of two different eigfenvalue problems. The R.> c;,lar,-

ized General F.:II ,..arl;,.- Cl. ;--. .; r (ReGEC) is introduced, which uses a new regularization

technique that requires only one eigenvalue problem to be solved. This classifier reduces

the computational time by half compared to the standard eigenvalue classifiers.

Due to the size and efficiency problems, very large databases could only be processed

or mined using a group of connected computers (multicomputers) that run in parallel

and communicate among themselves. Standard data mining algorithms do not achieve

a good performance on multicomputers, in general. Therefore, special algorithms must

he designed in order to exploit their strong computational infrastructure. There are a

number of comprehensive surveys on parallel implementations of widely used data mining

and knowledge discovery methods and their application spectrum [50-53]. A parallel

implementation of ReGEC is introduced to perform computationally hard classification

tasks on genomic databases.

Another inl r ~ problem in classification is the amount of training examples available.

Although it looks intuitive that the more example a learning systems has, the better

predictions it can make, the computational burden of learning may make the system

inconvenient. One immediate solution is to select a subset of points that would retain

the characteristics of the training set. A second problem arises when a new training data

becomes available for training. A desirable method as a solution to the second problem

should efficiently evaluate the contribution of the new data to the classification function,

rather than a complete training of the incrementally augmented training set. A new

method, which is referred to as Incremental ReGEC' (I-ReGEC), is introduced. I-ReGEC










finds a small representative subset of the training data, which in fact, provides better

generalization results. For some publicly available benchmark classification problems,

the amount of reduction in the training set can he as large as I' with comparable

classification accuracy and improved consistency. The proposed subset selection method

starts with an initial set of pattern vectors and incrementally expands this set by adding

new pattern vectors which contribute to improving classification accuracy. The main idea

is to use a small subset of points to solve the generalized eigfenvalue problem to evaluate

the contribution of the new pattern vectors using ReGEC.

The rest of the dissertation is organized as follows. Chapter 2 discusses the

fundamental topics on kernel-based learning such as generalization, kernels, optimization

theory, support vector classification and support vector regression. In Chapter 3, these

classification and regression problems are further developed into selective classification

and regression problems that are intended to find those pattern vectors which provide

a better separation of two classes of data, or vectors that define a regression function

with lower error. In OsI Ilpter 4, the standard support vector classification, regression and

feature selection methods are applied to neural data from a visuomotor task performed

by macaque monkeys in order to study the integration of visual and motor systems in

the primate brain. The results are further improved by applying selective support vector

classification. In C'!s Ilter 5, extensions to generalized eigenvalue classifiers are presented

involving a new regularization technique, a parallel implementation of the regularized

classifier, and a fast incremental classification algorithm that uses the regularized classifier

to find a small subset of the training set with better generalization capabilities and

comparable classification accuracy. Finally, the conclusions regarding the kernel-based

methods and their applications are drawn in OsI Ilpter 6.









CHAPTER 2
K(ERNEL-BASED LEARNING 1\ETHODS

This chapter covers the fundamental topics on machine 1. Ill1.11. specifically kernel

methods in machine 1. Illrf11, which are built on the strong theoretical background of

the statistical learning theory [17]. Although kernel methods are relatively new topics

in machine learning there are comprehensive books available on kernel machine learning

[54, 55] and most of the material presented in this chapter are compiled from these books.

The sequence of topics covered in this chapter are as follows. First, the concept

of kernels is explained and how they are used as nonlinear mapping tools in order

to enhance similarity among data points is shown. Then, the generalization theory is

reviewed briefly to mention bounds of the empirical risk hased on the Vapnik-Ch!~~ onenkis

theory. Some fundamental concepts in optimization, more specifically, some elements of

the Lagrangian theory are introduced to explain necessary and sufficient conditions for

convex optimization problems with linear inequality constraints. These three topics pave

the way to support vector classification and regression models with different variations of

primal and dual formulations with respect to penalization schemes on the classification

error.

2.1 Kernels

Kernels are introduced in classification to provide enhanced similarity measures

between pattern vectors. They basically transform the, so called, input space, X, in which

the original pattern vectors reside, to a usually higher dimensional dot-product space N~

called the feature space, via a map # : X i'F, such that



K(x,, x) = (#(xi) #(xj)). (2-1)

The main concept is focused on the dot product of two mapped pattern vectors. The

dot product of mapped pattern vectors may become computationally intractable, while











finding the dot product implicitly has same complexity as in the linear case, in general. As

an example, consider the following dot product in 2-D. [55] (pg. 26)





S : RW2 i'~ I3





where x(i) is the ith component of the pattern vector x. Note that for two pattern vectors

xi ad x, ((xe (xy) =((x xj))2. A visuanlization1 of this mapping is given on a

small toy example in Figure 2-1. Another mapping from 2-D input space to a 4-D feature

for the same kernel is as follows


gs


u

r.


100


Figure 2-1. Kernel mapping of circular data in 2-D to 3-D.












S : RW2 i'~ I4

(x(1),X(2)j H (X 1), X 2), X(1)X(2)! X(2L)X(1))

The following proposition demonstrates a generalization on polynomial approximation

for any dimensional input vectors, and any degree of polynomial kernel [56].

Proposition 2.1.1. D.li~ It.. Gk map X 6 IRd t0 the UcCLOT k (X) whOSe enrfieS aTC ill

possible kth degree ordered products of the entries of x. Then, the corresponding kernel

.., ind1'1:': the dot product of vectors mapped by #k iS



K(x,, xy) = (#kXi> k(Xj) = (Xi Xj k. (2-2)

The results on unordered product features are also given in [55].

Now we can investigate the conditions how kernels can create feature maps. More

generally, given a kernel, how can we construct a feature space such that the kernel

computes the dot product in that kernel space? First we define a positive semidefinite
kernel.

Definition 2.1.2 (Positive definite Kernel). Let X be a not.. pilul set. A function K on

Sx X which for all xl,. .,x, E X gives rise to a positive 7. It../.~: matrix: KC such that

CijcicjKij > 0 for all ci E RW, is called a positive 7 t.. enl

In general, if we relax the semi-definite positivity condition on K, the resulting matrix
is called Gram matrix or kernel matrix;. The term kernel stems from the first use of

integral operators, studied by the famous mathematician David Hilbert for functions (Tkf

that give rise to an operator Tk, Such that



(Tk f)(x) = K(x, x*l() f(x)d* (2-3)

The function (2-3) above is called the kernel of Tk~.









It is worth noting that a positive definite kernel implies positivity on the diagonal and

symmetry. That is,



K (x, x) > 0 for all x EX

K(xi, xj) = K(xj, xi). (2-4)

In the literature it was shown that any algorithm that works on dot products can be

kernelized through the kernel trick, as explained below:

Remark 2.1.3 (K~ernel Trick [55] (pg. 34)). Given an rll' -r ithm which is formulated

in terms of a positive 7. It...:I~ kernel K, one can construct an alternative rlly. rithm by

replacing K by another positive 7. It../.~: kernel K.

In the machine learning literature, this kernel trick is introduced by Mercer's theorem

and explains the geometry of feature spaces. It can be considered as the characterization

of a kernel function K(x, b fx*).

Theorem 2.1.4 (11. rcer [54] (pg. 35)). Let X be a compact subset of RW". Suppose K is a

continuous ;; iii/ .i~ function such that the :.J~ yndr operator TK : L2(X) L2(X),



(Tk )(- = K-, x f (~dx(2-5)

is positive, that is



/~~~~~~~~~( K~* )f(* xd~x>0 26)
for all f E L2(X). Then we can expand K(x*,x) in a unifont ilti; convergent series in

terms of TK 'S .hl :t. functiOns, phij E L2(X), normalized in such a ;,.rt;, that ||~ l||g = 1,

and positive associated .:II ,:; al;,. 4 Ag > 0,



K(x*, x) = i(x*) (x). (2-7)
j= 1










The conditions for 1\ercer's theorem are equivalent to the requirement that the

corresponding matrix is positive semi-definite for any finite subset of X. The term kernel

will be used for the rest of the text to refer to those functions that satisfy this property,

although it is not the general definition of a kernel. This concludes the discussion on

kernels to be able to develop kernel-based methods within the intended scope of the study.

2.2 Generalization Theory

In this section, we review the factors that affect good generalization results and give

and overview of the Vapnik-Ch.~ Ii.onenkis (VC) theory which provides robust bounds on

linear classifiers in kernel spaces. The results of this theorem are quite fundamental to

machine learning since the flexibility introduced with kernels would lead to overfitting,

which is contrary to the generalization of the classification function. The generalization

bounds for maximal margin classifiers, soft margin classifiers and generalization for

regression are discussed briefly.

The data used in training and testing are assumed to be generated from the same

unknown distribution ~D over the pairs (xi, Yi). The J.e. J..rld I, 'i'i'''..t:nor/. let correct (PAC)

hound on learning requires a strong bound that is not likely to fail. PAC has the form

E = E(n, H, 6), which means that, with probability of 1 6 over randomly generated sets S,

the generalization error of selected hypothesis hs will be bounded by



errv(hs)l
where H is the class of all possible hypothesis and n is the number of examples in the

training set. Before presenting further results, we mention that the quantity known as

Vapnik-Ch.~ i..; ~~Isu:!- (VC) dimension shows the flexibility or capacity of a function class

to classify a random training set of points. Here, we give the fundamental theorem of

learning, details of which can he found in [54].

Theorem 2.2.1 (Vapnik-Ch.! Ii .onenkis). Let H be
dimension of d. For ar,:; r, .e~.:.1sl...;i distribution on X x {-1,1}., with r, go~l.:.:sl..;i 1 over









n random ex~amp~les S, r t.;, bi,./;,l. -I, 4 h eH that is consistent with S has no error more

than
errv(hs) < E(n, H, ) =, 2n dlog +log 29


provided that d < n and n > 2/E.

This theorem is followed by the error bounds for the maximal margin classifiers, soft

margin classifier and the error bound for support vector regression. The details on how

these theorems are derived can be found in [54].

Theorem 2.2.2 (Error bound on maximal margin classifier). Consider thresholding real

values linear functions with unit weight vectors on an inner product space X and ~fix;

y ER I. For ar,:; I4. e~r.:.1sl..;i distribution ~D on X x {-1, 1} with support in a ball of radius

R around the origin, with Igo~l~l.:.:sl...;i 1 6 over a random ex~amp~les S, ; i-l

f E that has I,,tryl.:. ms(f) > 7 on S has error no more than,

2,,, eny 128nR2 4
ery EnL 6 7 = 6R22 0g 1g 2 +log (2-10)
n 4R Y2 6/

provided n > 2/E and 64R2/y 2

Theorem 2.2.3 (Error bound on soft margin classifier). Consider thresholding real-valued

linear functions L: with unit weight vectors on an inner product space X and fix: y E R+

There is a constant c, such that for r,:; I, 4.e..1st:l..7.;, distribution ~D on X x {-1, 1} with

support in a ball of radius R around the origin, with Igo~l~l.:.:sl...;i 1 6 over a random

ex~amp~les S, r,.;, bi,./;,l.l, .:4 f E has no more than


c R2 1112
erry, < Y 210g2, +logf (2-11)

where ( = ((f, S, y) is the Ina~ryl.:. slack vector with respect to f and y.

The generalization error bound for the 1-norm soft margin classification is very similar

to that of the 2-norm with an additional term log(1/y) in the numerator of the first term

in parenthesis, which is multiplied by ||(|| .










Theorem 2.2.4 (Error bound on regression). Consider performing regression with linear

functions on an inner product space X and ~fix; y I 8 E R+. There is a constant c,

such that for n..;;, Igo~l~l.:.:sl..;i distribution ~D on X x RW with support in a ball of radius R

around the origin, with I4.e..l.,th.7.1:;i 1 6 over a random ex~amp~les S, the I,~ l~l.:l..J...iz that a

/ i,./;,l -I, .: w E has output more the 8 ton; r; from its true value is bounded by,

erz c ||w|| + |(|jlg(/ 1\
erry < 2 1g2 n +log ,(2-12)

where ( = ((w, S, 0, ) is the Ina~ryl.:. slack vector with respect to w, 8, and y.

These bounds provide good generalization results on the support vector machine

classification and regression methods. The only remaining topic, before the introduction

of SVM methods is optimization theory which explains the conditions under which we

can solve the problem of finding a hyperplane that maximizes the margin, or a regression

function that minimizes the error.

2.3 Optimization Theory

Optimization theory pIIl us an important role in machine learning. The machine

learning problems can be formulated as optimization problems, for which well studied

efficient solution methods are known. The mathematical framework provides necessary and

sufficient conditions for the optimum solution. These conditions, in turn, help determine

the desired classification function that provide a good generalization on the data. The

p~rimal optimization problem is presented in a general form, which has an associated dual

problem. As it will be explained in Section 2.4.1 the latter formulation provides the means

for the kernel methods to be applicable in support vector classifiers. The transformation

from the primal problem to the dual problem is explained through Lagrangian theory,

which is also a well studied subject in mathematics. We start by the definition of the

primal optimization problem.

Definition 2.3.1 (Primal problem). Given functions f, gi, i=1, ..., k, and hi, i=1, ...

m, I. Is... .1 on a s domain 2CR I~,













vainimize f (w), wE 02, (2-13)

.subject to gi(w) < 0, ,, (2-14)

h,(w) = 0, i = 1,... m, (2-15)

(2-16)


The the optimization problem that is rendered by the support vector classification is a

convex function on a convex set. First we define the convex function and convex set.

Definition 2.3.2 (Convex function). A real valued function f(w) is called convert for

we R I if for w, u E R~d. and for r,.;, 8 in[O, 1].


f (0w + (1 8) u) < f (0) + (1 8) f(u). (2-17)


Continuous functions that are twice differentiable are also convex if their Hessian

matrix is positive semi-definite.

Definition 2.3.3 (Convex set). A .set is convex if for it..;; w, u E U., the point


r = (0w + (1 8)u) E n for <<11 0 E [0, 1].


Now, we restrict the optimization problem such that the objective function is

quadratic, the constraints are linear, and the domain 02 = Ra". Next, we review the

Lagrangian theory and K~arush-K~uhn-Tucker (KKET) optimality conditions for the dual

transformation.

Lagrangian theory is based on characterizing an optimization problem without

inequality constraints. The Leegr t r ,:I.:. r. function and the Leegr t r ,:I.:i r, multip~liers are the

main concepts in this theory. K~arush-K~uhn-Tucker, further improved this characterization

to define optimality conditions when the problem has inequality constraints. We give

formal definitions of these concepts below. We start with the most general case of









the Lagrangian function, where the optimization function includes both equality and

inequality constraints.

Definition 2.3.4 (Lagrangian function and multipliers). The Lagrtr,:l.:;e.' function of an

optimization problem with domain 2ER I~,



minimize f (w) w E n (2-18)

subject to gi(w) I 0 =1..k

As (w) = 0 ,...,m






L(w,~ a ) =f (w) + agg4(w) + ,Sh,(w), (2-19)
i= 1 i= 1
and the coefficients asi and ,S are called the ~llaye:l.:r,:..; multip~liers.

Note that the Lagrangian function is a relaxation of the original optimization problem

(2-18), and sets a lower bound on the optimal solution value of the original problem.

Before we show this relation, let us define the Lagfrangfian dual formulation.

Definition 2.3.5 (Lagrangian dual problem). The lagr tr,:ll:e.' dual problem of the problem

(2-18) is 1, I s... .1 as follows.


m~ax 0 (a, p) =(inf gL (w,~ a, ) (2-20)

subject to a > 0. (2-21)


The following theorem establishes the relationship between the solution to the primal

problem and the dual problem.

Theorem 2.3.6. Let w E O be a feasible solution of the primal problem and (a, P) a

feasible solution of the dual problem. Then f(w) > 0(a, P).










Proof. By definition and the feasibility of w, a and P,


O(a, p) =inf L(u, a, a)
u602
< L(w, a, P)

=f (w) + a g(w) + p h(w) < f (w).




A useful corollary follow, basically stating that the optimality of the solutions to the

primal and dual problems can he concluded when the objective functions of two problems

are equal.

Corollary 2.3.7. If f(w*) = 0(a*, P*). and w. and (a, P) are feasible for the primatl

and dwel problems. then they are optimatl .solutions to the primatl and dwel problems.

'' i'.~ 1.:; le; in which case of gi(w*) = 0.

Proof. The results follow from the inequalities in the proof of Theorem 2.3.6. O

The solutions to the primal and dual problems are not guaranteed to be equal, in

which case the difference is referred as the I;,.el./l~ i prl. The following theorem states the

strong duality when certain conditions are satisfied for an optimization problem.

Theorem 2.3.8 (Strong duality). Given



mmn f(w) (2-22)

.subject to gi(w) < 0, i=1 .,k

hi(w)= 0, i ,.,,


where ye and hi are
vector b. the I;,.el./ ~i 11~'1 is zero.

This theorem is followed by the K~uhn-Tucker theorem, which states the conditions

such that the results of strong duality theorem (Theorem 2.3.8) can he applied in practice.










Theorem 2.3.9. Given
convert. the necessary and .sufficient conditions for a point w* to be optimum are the

existence of a and p .such that

8L(w,a,P)
8w
8L(w,a,P)


of ge(w*) = 0, i ,.,k

gi(w) < =1,.,k

at > 0i ,..k


The relation asysi(w) = 0 is known as the K~arush-K~uhn-Tucker complementarity

condition, and implies that asi > 0 for active constraints, and asi = 0 for inactive

constraints. This means that small perturbations of the inactive constraints have no effect

on the optimal solution.

The main convenience of the Kouhn-Tucker theorem is transforming the primal

problem into a dual problem which is easier to solve. This is mainly done by differentiating

the Lagfrangfian function with respect to the primal variables, and substituting the

equivalent expressions back in the Lagrangian function to achieve a simpler formulation

without inequality constraints. The solution to the dual problem shows which constraints

in the primal problem are active. The active constraints indicate support vectors,

which are generally much less in number compared to the total number of all inequality

constraints. In machine 1. Illr11.-- this result will become more clear in the next section.

2.4 Support Vector Machines

In this section, we coalesce all of the results we reviewed before into building efficient

and effective classification methods. Namely, the dot products and kernel spaces provide

the foundation for linear and non linear mapping of the pattern vectors, where as

generalization theory explains the classification error bounds, and finally, optimization










theory introduces the tools and techniques for efficient implementation of the nonlinear

methods. We first discuss the Iales~imrul Iabrgin Clr.,-m6. r, which is the simplest form of

SVAI that work for separable classes. Next, we allow some misclassification together some

penalty in Soft Iabrgin Cl~r.;--. rs.

2.4.1 Maximal IVargin Classifier

We introduce maximal margin classifier as the simplest form of SVAI classification.

The underlying optimization problem is only feasible for separable cases, which does

not fit well with most of the real life classification problems. However, the maximal

margin classifier demonstrates the fundamental technique used in the more commonly

used versions of SVAI classifiers. The key factor is that the solution to the optimization

problem minimizes the bound on the generalization error by maximizing the margin with

respect to the training set. Since this bound does not depend on the dimensionality of the

space the same method can he used in a kernel induced space. The basic premise of this

method lies in the minimization of a convex optimization problem with linear inequality

constraints, which can efficiently be implemented based on the well-studied optimization

theory we have reviewed in Section 2.3.

First we introduce the problem by formal definitions of a hyperplane, a canonical

hyperplane, and the margin of a hyperplane. A hyperplane is a d 1 dimensional linear

surface in a d dimensional space N~, and it can he represented as:


{:r E |~(w x) + b = 0}, we E N, be R (2-23)


In 2-23, w is orthogonal to the hyperplane and (:r-) gives the length of pattern vector

:r along the direction w when w is a unit vector. Note that we have inherent degree of

freedom in specifying the same hyperplane as (Aw, Ab) for A / 0.

Definition 2.4.1 (Canonical Hyperplane). Al i,;l* ,'lr,:t... (w, b) is a called a canon-

ical bi,;ll' 'i'Ar,: if the closest point front the hi,;ll' 'i'Ar,: heas a distance 1/||w ||. i. e.

mini=1,...m, |(w xi) + b| = 1.









Now, consider two points, each belongfingf to one class, x- and x+. We want to find a

canonical hyperplane (w, b) such that


(w x+) + b

(w x-) + b


(2-24)

(2-25)

(2-26)


This implies a margin of


(w/||w|| (x+ x+))
= 1/|w||.(2-27)

We assume that the training data and the test data come from the same distribution.

Due to the noise in the data, the separating hyperplane can still classify new pattern

vectors correctly if the margin is maximized. Maximizing the margin 1/||w|| for the

canonical hyperplane is equivalent to minimizing ||w||. In the following optimization

problem, each point xi with a label yi satisfies the canonicality condition in the constraints

2-28, while maximizing the margin by minimizing the norm of the normal vector w of the

hyperplane, as shown in Figure 2-2.


Figure 2-2. Maximal margin classifier














i= 1


(2-28)


mmn:

subject to


ye ((w xi) + b) > 1 (4

(2-29)


Problem 2-28 is referred as the primal problem. In order to apply kernel methods, the

dual problem can be derived from the Lagrangian function 2-30.


i= 1


L(w, b, E, )


(2-30)


-jn a [y((w xi)+b) 1 + ]
i= 1
(2-31)


Differentiating L(w, b, a) with respect to the primal variables w and b, and assuming

stationarity, we get the following expressions, which we can substituted back in the

Lagrangian function to obtain the dual formulation with only dual variables.


w yealx, = 0,
i= 1

yeas = 0.
i= 1


8L(w, b,a)
8w

8L(w, b,a)
iib


(2-32)


The following proposition shows the dual formulation and how the solution to the

dual formulation realizes the maximal hyperplane.

Proposition 2.4.2. Consider a linearly separable training sample S = {(xl, yl),...,(x,,y,}}

and suppose a* solve the following quadratic optimization problem.














i= 1 i= 1 j= 1

subject to ya =0
i= 1
a > 0.


Then w* = CE yiafxi realizes the maximal I,,rtyl.:. i';i'' 'id1'"..: with geometric



Note that from K~arush-K~uhn-Tucker complementarity conditions, the corresponding

constraints in the primal problem are active for those with the dual variable of5 = 0.

K~nowingf w*, we can find b* as follows:



b*= ,(w* xi). (2-34)
i:af >0
Note that the distance of a pattern vector to the hyperplane is given by the function



f (x, a*, b*)= yeaf :(xi x) +B b. (23)
i= 1
since the distance on either side of the hyperplane would have opposite signs, the

following function can classify a pattern vector x at the right side of the hyperplane.



class(x) = agn {f (x, a~*, b*)}. (2-36)

This classification method can easily be extended to the feature space by replacing

the linear dot product (xi xj) with a kernel K(xi, xj). Since the linear dot product is a

linear kernel, we use a generic kernel in further formulations.

2.4.2 Soft Margin Classifier

Most real life problems have non separable data, even in the feature space. This

is caused by the noise in the data. However, the maximal margin classifier can find a










separation in the feature space with the help of suitable kernels. This usually results in

overfitting. Slack variables can tolerate the noise in the data well, and better generalization

results can be achieved. The slack variables (4 allow misclassification for each pattern

vector if needed. However, the slack variables, when greater than zero, are subject to

a penalty of C. In Figure 2-3 soft margin is demonstrated which incurs penalty for

misclassified pattern vectors. Now, we can modify the maximum margin classifier with the

introduction of sack variables, as follows.

*$ *




W


IW1


Figure 2-3. Soft margin classifier


1 C"
mm 2||2 2
i= 1
subject to yi((w xi) + b) > 1 (4


(2-37)


(2-38)


Note that (4 < 0 cannot be true for any optimal solution to 2-39, since the

corresponding constraint would still be satisfied while incurring less cost if (4=

Therefore, we do not explicitly need a positivity condition on the slack variables.










In 2-39 the 2-norm of the slack are penalized in the objective. An alternative

formulation 2-39 involves penalization of the 1-norm slack variables in the objective.

However, we need to impose positivity on the slack variables be able to penalize them.


21 C2

i= 1
subject to yi((w xi) + b) > 1 (4

~i> 0 ,. .


(2-39)


(240)


Now we can derive the dual formulation for both 1-norm and 2-norm support vector

machine formulations. The significance of these formulations are that they do not involve

inequality constraints, and they allow kernels to be introduced in the objective function

where a dot product occurs. The standard method is to differentiate the Lagrangian

function with respect to the primal variables and substitute them back in the Lagrangian

function. The Lagfrangfian function for the 2-norm SVM primal problem is as follows.


i= 1

-jn a [y((w -xi) +b) 1 + (4]
i= 1


Li(w,b,(, a)


(2-41)


(2-42)


Differentiatingf the Lagfrangfian function with respect to w, b, and each I, we get


w yeagxi = 0
i= 1


i= 1

C(4 asi = 0.


8L(w, 6,(, a)
8w

8L(w, 6,(, a)
iib

8L(w, 6,(, a)


(2-43)


(2-44)


(2-45)










Substituting the partial derivatives (2-43-2-45) in (2-41), we obtain,


max as -C ygmgii- a
i= 1 i= 1 j= 1 i= 1
(2-46)


We can substitute the dot product in the objective with a more general kernel of

choice. The following proposition summarizes the dual formulation for 2-norm soft margin

SVM.

Proposition 2.4.3. Consider 1~. Ifying a training sample S = {(x y ),..., (x,, y,)}

using a feature space induced by a kernel K(-, -), and suppose a* is a solution to the

following problem:




max a ysyyagayinK~xx, x) ~af (2-47)
i= 1 i= 1 j= 1 i= 1

subject to y =0
i= 1
asi > 0 i = 1,..., n.


Then, the decision rule agn( f(x)) is equivalent to the hyperp~lane in the feature space

induced by K (-, -) where



f x)= yicaf K(x, xi) + b* (2-48)
i= 1
and b* is chosen such that ye f(xi)= 1 af*/C for, thnos patterr n vectors with of* > 0

The derivation for the 1-norm dual formulation is very similar to that of 2-norm.

The Lagrangian function is slightly different, with the sum of slack variables, and the

additional term for the positivity constraints.













2||w|| 2+ C
i= 1


L~(w,b,(,a,r)


(2-49)


-jn a [y((w xi)+b)-1+(4]- rs(.
i= 1 i= 1
(2-50)


Differentiating the lagrangian function with respect to w, b and (, and imposing

stationarity, we obtain,


w yeagxi = 0
i= 1


i= 1

C asi ri = 0.


8L(w, 6,(, a)
8w

8L(w, 6,(, a)
iib

8L(w, 6,(, a)


(2-51)


(2-52)


(2-53)


Substitutingf these expressions back in 2-49, we obtain,


max as ygymagf~i xi xy)
i= 1 i= 1 j= 1

The only difference between the 1-norm and 2-norm are that from 2-53 and ri '>, the

dual variables asi should be in [0, C], which is often referred to as the box: constraint. The

followingf proposition summarizes the result for 1-norm dual formulation for classification.

Proposition 2.4.4. Consider ,1 Ir.fying a training sample S = {(xl, yl),...,(x,, y,)}

using a feature space induced by a kernel K(-, -), and suppose a* is a solution to the

following problem:


max as 21i
i= 1 i= 1

subject to yeas = 0,
i= 1
0

ysy p71ri01K(xi, xl)
j= 1


(2-54)










Then, the decision rule agn( f(x)) is equivalent to the hyperp~lane in the feature space

induced by K (-, -) where



f (x) =.-:~I yef KIx xe +b (2-55)
i= 1
and b* is chosen such that, yefxe) 1 for those,~,, patter vectrs wih 0 < af* < C

2.5 Gradient Based Adaptive Scaling

This approach introduces a global optimization problem and finds certain parameters

regarding the separating hyperplane and linear scaling factors for the the input space.

This linear scaling consist in determining a diagonal matrix E, where each element on

the vector of diagonal entries a is the weight of the corresponding feature. The purpose

of the scaling is to obtain a linear transformation of the input space that improves the

classification rate. In this method the following global optimization problem is defined:





i= 1 i= 1 j= 1 i= 1



~i> 0 i = 1, ...,



ak '> k =1, ..., a

where ~ii = E xi, and the parameter p has the role to encourage the sparsity of the

solution for E.

However, this problem is hard to solve. Therefore, an iterative two stage algorithm is

proposed [22]. In the first stage of any iteration i with the starting scale vector a(i 1),

the problem is optimized with respect to vector a~ as in the standard SVM method to find

a solution a~(i). In the second stage, the problem is solved with respect to the scale vector










a to find a steepest ascend a(i). However ca(i) are fixed to simplify the problem assuming

that co(i + 1) will not he affected by a(i) substantially.

2.6 Support Vector Regression

The structure of the support vector regression is very similar to the classification

method. Basically, we try to estimate a linear function in a kernel induced nonlinear

space. The objective is to minimize the generalization bounds while ignoring errors

within a given value E. This can he thought as a hyper-tube around a linear function in

the kernel induced feature space, such that the pattern vectors that are in this tube are

assumed not to contribute any error as in Figure 2-4.





r----













Figure 2-4. Standard SVR and E-Insensitive tube around the regression function


Since anything in the E of the estimated regression function does not contribute as

error, this type of regression is called E-In~sensitive. As in the classification case, there are

two versions of the E defined by the norm of the error, which are called Quadratic and

Litear E- Insensitive support vector regression, respectively.

2.6.1 Quadratic E-Sensitive Loss

The main idea is to create a linear function in the kernel induced space such that the

loss function for regression from the generalization theory is minimized. The corresponding

primal problem is given as in (2-57). The constraints imply that the actual pattern









vectors are allowed to be E above (constraint 2-58) or below (constraint 2-59) the

regression function without penalty. All pattern vectors outside the E Tange are Still

allowed, however they incur a cost of C.


min ||w||2 3+ )
i= 1
ct to ((w xi) + b) yi < E +

yi ((w xi) + b) < E +


(2-57)

(2-58)

(2-59)


subj e


i

li.


In order to apply the kernel trick, we need to derive the dual formulation from the

Lagrangian function for (2-57), which is given as,


1' ~C"s


L(wb,(,()


(2-60)


Ea (E Us W Xi) -b)
i= 1

-~~~ & (E + (W -Xi) + b)
i= 1

Differentiating the Lagrangian function with respect to the primal variables w, b, and

each ( and (, we get




8L/8 = w- (4 )xi= 0(2-61)


i= 1
SC(4 asi = 0


8L/8b
= d~

8L/8(4


(2-62)

(2-63)

(2-64)


Substituting the partial derivatives (2-61 2-64) in (2-60), we can write the following

dual formulation,















i= 1 i= 1 j= 1


i= 1 i= 1

subject to (a- )=0
i= 1
asi, 64 > 0 for i = 1,..., n.


Fr-om the solution a* and <^*, the resulting regression function is given as,



f () = (d af )K(x, xi) +b*, (2-66)
i= 1
where, b* is chosen such that f(xi) yi = -E (di* a*)/C for any i with (di* a~*) > 0.

2.6.2 Linear E-Sensitive Loss

The linear E-SenSitiVe lOSS VerSiOn is Slightly different from the quadratic loss

formulation, where we impose positivity on the slack variables while penalizing them

in the objective function. The formulation is given as follows,



21 C"ic lI 27
mmn : 2 ||w ||2 2 f i)(-7
i= 1
subject to

((w-xi) +b) yi
yi ((w xi) + b) < E + i

(4,(4 '> 0 for i= 1,...,n.


The dual can be found by finding the Lagrangian function for the primal problem,

differentiating this function with respect to the primal variables, and substituting

equivalent expression for the primal variables in the Lagfrangfian function. The resulting

dual formulation is given as,














max -( -a)( a)xi-x)(2-68)
i= 1 j= 1



i= 1 i= 1

subject to i(as & ) =
i= 1
0


Fr-om the solution a* and <^*, the regression function is as in (2-66), however b* is

chosen such that f(xi) yi = -E for any i with 0 < (&* a~*) < C.









CHAPTER 3
SELECTIVE K(ERNEL-BASED METHODS

The motivation for the selective kernel-based methods comes from a real life problem,

in which time series are to be aligned for improving classification results. The individual

pattern vectors and corresponding labels or values from the machine learning context

are extended to sets of pattern vectors sharing the same label or value. First, we study

the problem of choosing a single pattern vector from each set of possibilities in order to

find an optimum separation between the selected positive and negative pattern vectors.

This problem, which we call hard selection is highly combinatorial in nature. Therefore,

we introduce relaxations of this problem. These soft selection problems, in contrast,

slightly differ from the optimization problems we have studied in C'!s Ilter 2. However,

the general structure of the altered problems is quite similar to the original ones with a

convex objective, and linear constraints on a convex domain. This convenience allows us

to derive dual formulations in which we can apply the kernel trick to obtain nonlinear

classifiers. This chapter includes formulations for the selective versions of support vector

machines and support vector regression and their variations. The results on toy problems

are presented for the hard selection methods, as well as the soft selection methods, which

include incremental elimination and incremental inclusion schemes.

3.1 Selective Support Vector Machines

In the standard support vector machine framework, the input for the classifier is the

set of labeled pattern vectors in the training set a = { (x x, yl), .. ,(x, y,) }. Now, we

consider that instead of each pattern vector, there is a set Xi = {xi,1,...,xt} of t pattern

vectors with the same label yi, where the set Xi can be considered as a tx d matrix, each

row being a pattern vector in R d. An example is shown in Figure 3-1.

The first type of problem we want to solve is the hard selection problem, in which

a single pattern Xi~k* E Xi is selected for each set such that the margin for the selected

patterns is maximized. We give the formal definition below.




















m Ae



Fiue31 eto atr vetrs 3 atr etr nec e

Definitio 3.1. (Hr eeto rbe) e ={i..X}b eso at
vectors wi t atr etr ei x nec stXadlty={i e}b h
corepodig ablsfo echst it echpaten ecori X hvig hesae abl :







seecedpaten ectr with lael +1 pand -1 ietos maximized.vcosinec







The hard selection problem can be formulated as a mixed integer quadratic

optimization problem as follows, which involves a modification over the standard support

vector machine optimization problem.




minimize |w || = = s

subject to yi((w -xi) b) > 1 i,k -M(1 -vi~fk) i = 1,..., n; k = 1,...,t,


Chi,k; =01} i= 1,...,n; =1..t



In (3-1), in the first set of constraints (3-1), M is a large positive number. This can

be set as the maximum distance between any pair of pattern vectors in X. When the

corresponding vui~k = 0, then, basically, this constraint has no effect on the problem, and










it is equivalent to removing the pattern vector from the training set. When I = 1, the

constraint reduces to a regular constraint in the standard SV1\ formulation. The second

set of constraints (3-1) ensure that only one pattern vector is chosen from each set of

points. Although this problem finds the optimal subset of pattern vectors for the positive

and the negative class, it is highly combinatorial.

We relax the binary variables in the hard selection problem in such a way that

they would give us a measure on how much slack does a pattern vector require if it is

misclassified. The following question is, how can we diminish the effect of misclassified

pattern vectors on the objective function in order to have a large margin? We attempt to

answer these questions with different relaxations of the hard selection problem, in which

binary variables are replaced with continuous variables serving as free slack variables for

pattern vectors that are misclassified.

We know from Section 2.4.2 that in the soft margin classification, each pattern

vector has a slack variable, but this slack is penalized in the objective function. Here, we

introduce the concept of restricted free .slack for those pattern vectors which are close to

the separating hyperplane, or misclassified. The main motivation behind this relaxation is

that such pattern vectors are the ones which impose the orientation and placement of the

separating hyperplane. We want to diminish their effect of these pattern vectors and let

the pattern vectors which are more separated determine the orientation and placement of

the separating hyperplane.

In Figure 3-2, there are pairs of red pattern vectors and blue pattern vectors. If we

ignore the pairs and consider the problem as a standard classification problem, since blue

and read patterns are separable, a standard support vector machine algorithm would solve

this problem without using any slack, however with a small margin. It is intuitive and

straightforward that if the pattern vectors towards the middle were to be removed, the

remaining pattern vectors would have a large margin. By introducing free slack variables

we can diminish the effect of the middle group of pattern vectors, however we restrict the











total slack to avoid trivial solutions. The reason we call them free slack is because they are

not penalized in the objective function. We still keep the original slack variables which are

penalized for the problem to stay feasible for any training set. we would like to note that,

given a hyperplane, if a pattern vector is misclassified, then the free slack will be used first

depending on how much free slack is available.










Figure~~~~~~~----- :32.Clssfiatonofa epralese i te reene f esrite feesl

Ther ar tw prbles t beaddessd. he irs n s o uh.lc sol
be proided?,and te secod oneisI: howto dstrbutethetota .sackForthe irs
problem, ~ ~ -- aninutieaprac anh aate.Inth tadrdsppr vcormchn
forulaio, afuctina magi of1 s rquied Sicewe av atlest wopater

vecor pr stwepreertoproid atotl re slckofn, hih s afre lac o 1pe

set on he averge. Fo a set f two a------tternvecors thee ae tree ossbiltie i)hot
ar casifedrih, i)on o temi mscasifed ii hthar mscasiie. n hefis
case the matter vctor thatis close to the sepa----rtighprln nthseodae
the msclasifiedpattrn vetor, nd i the hird ase he patern ecto furter frm th







irepcivue of th Clset ithe eongo to We arbefser to this prsenc as poolied free .lk(PS).c


Figu e re 3-3 showls the ditibto fodr PSs. Inthe fise ond aproch however, elach setofl










pattern vector receive an equal amount of 1 regardless. We refer to the second case as

free slack per set (FSS). The distribution of FSS is more restrictive therefore its effect

on the separating hyperplane is not as much as the PFS, as seen in Figure 3-4. In the

following subsections, we present modified optimization problems together with their

dual representations in order to study selective kernel methods in linear and nonlinear

classification problems.


Figure 3-3. Distribution of pooled free slack (PFS)


Figure 3-4. Distribution of free slack per set (FSS)


3.1.1 Selective 2-Norm Support Vector Machines

Assume that there are sets Xi, i = 1,..., n of vectors, xi~k, k = 1,..., t. The following

optimization problem finds the maximal margin with PFS approach.














minimize |w|2= 1s


subject to yi((w -xi) +b) >1 i,k i,k i =1i,..., n;

C~=1 ~=1 Uij < n

vi,k, > 0i ,. ;


The Lagrangian function for (3-1) is given as follows,


(31)


k=1,...,t


(3-2)


k= 1,...,t.


a t
|w||2 2 sE2k
i= 1 = 1
a t
t~k i i~k +b) -1 +(~
i= 1 = 1
a t

i= 1 = 1


L(wl,b,(,()


(33)


(34)


a t

CCiTli,ki;k


In this equation, a, p and r are Lagrangian multipliers. Taking partial derivatives

with respect to the primal variables w, b, and each (4 and I and imposing stationarity,

we get the following expressions:




8L/8w =w yeaixi = 0 (3-5)
i= 1

8L/8b = wn yes (3-6)
i= 1
8L/8(i,k Ci,k t~i = 0 (3-7)

8L/8vi,k = ~ir,k ri~k* (3 8


Substituting the partial derivatives (3-5 3-8) in (3-3), we can write the following

dual formulation:












u t n t nt
max CC i,k, C C gi2JJaikjX,kk;lX l Xj,1) (3-9)
i= 1 = 1 i= 1 = 1 j= 1 l= 1




subject to




0

Note that kernel induced maps to nonlinear feature spaces can be used by replacing

the linear dot product (Xi,k -Xj,1) in (3-9) with a kernel K(Xi,k, Xj,1).

For the FSS case, the constraint (3-2) becomes,




vik 1 i=1,. ., n. (3-10)


Consequently, this change introduces a separate Lagrangian multiplier Pi for each

constraint, and therefore the relevant component (3-4) in the Lagrangian function changes

to,




Si 1 vi)
i=1 k= 1

Following the differentiation of the Lagrange function, imposing stationarity and

substituting the expressions back in the Lagrangian, we can obtain the dual formulation

for the free slack per set case as,












u t n t nt
max CC i,,k CCC gi2J~aikjX,kk;lX l Xj,1)
i= 1 = 1 i= 1 = 1 j= 1 l= 1



ii= 1 =1 i= 1
subject to



O i ti,k IP o = 0 ,,L


(3-12)


3.1.2 Selective 1-norm Support Vector Machines

An alternative formulation for support vector machines is to penalize 1-norm penalty

term, as shown in (2-39). Here we present the selective version of the 1-norm SVM

formulations. The derivation of the primal and dual optimization problem are similar to

those discussed for the 2-norm case, therefore we briefly present these formulations. The

primal for the PFS approach is as follows.


minimize |||| IW' i,k
ii 1 = 1


(313)


subject to


n< n

i= 1 = 1

~i,k > 0


i=1...n; k= 1,...,t


(3-14)


1,...,n;

1,...,n;


1,...,t.

1,...,t.


Di,k, > 0


As in the 2-norm case, we derive the Lagrangian function for the primal problem,

differentiate it with respect to the primal variables, impose stationarity and substitute










the resulting expressions back in the Lagfrangfian function to obtain the following dual

formulation.


n t
maximize C ai~k
i= 1 = 1


ntnt

2 CCCCiyjain,ktj,/Xi~xk Xj,1) n
i= 1 k=1 j= 1 l= 1


(3-15)


subject to





0

In (3-15), the dot product (xi xj) can be replaced with a kernel K(xi, xj) for a

nonlinear map to the kernel induced feature space.

The primal and dual formulations for the 1-norm FSS case are slightly different from

the general free slack case, where the general free slack of a in constraint (3-14), now

implies that the free slack is restricted to 1 for each set, which is given as,


ik
k= 1


(3-16)


subject to


yi((w -xi + b)





(i,k >0

vi,k, > 0


>1-(ik-Uik


i=1,...,n;


k=1,...,t


(317)


1,...,n;

1,...,n;


:1,...,t.

I 1,..., t.


Through standard Lagrangian function, differentiation and substitution method, we

can obtain the following corresponding dual problem:


minimize || w||2 i
i= 1













maximize C ai,k 2 CC iZJJai,ktj,Xi~x k Xj,1) iW(38
i= 1 = 1 i= 1 = 1 j= 1 l= 1 i= 1

subject to





0

Havingf introduced 2-norm and 1-norm formulations of selective support vector

machines with the PFS and FSS case, we can move on how we use these optimization

problems .

3.1.3 Selecting Pattern Vectors with Large Margin

Our objective may differ with respect to minimum or maximum values of the free

slack assigned to pattern vectors. Based on such values we can adopt two immediate

approaches for findings the desired points. The first one is to choose the pattern vector

with the lowest free slack from each seat. This requires the optimization problem to

be solved only once. We refer to this approach as choose best vectors. The alternative

approach is to identify those pattern vectors that are very close to the separating

hyperplane, or those which are misclassified, and possibly incurring a high penalty.

From the solution to the optimization problem we remove the pattern vectors with the

highest free slack from each set, and resolve the problem with the reduced set. We refer to

this approach as eliminate worst vectors. This approach requires the optimization problem

to be solved t 1 times, until there is one pattern vector remaining from each set.

Through empirical tests on toy examples, eliminate worst method outperformed

choose best method. Eliminate worst method approaches to a linear separation for

separable sets. For an appropriate parameter value, for example o- in the Gaussian kernel,

the separating curve turns into a linear hyperplane for sufficiently large o- value. In Figure

3-5 there are three pattern vectors in each set and the groups themselves are separate.









In both standard SVM and selective SVM, lower values of Gaussian bandwidth causes

overfitting. However, it is clear that for the higher values of Gaussian kernel bandwidth

that standard SVM creates the hyperplane that separates all red and all blue pattern

vectors from each other as shown in 3-5. Selective SVM, on the other hand creates a

separating hyperplane which align itself such that the margin is maximized with respect to

the patterns chosen at the end of the elimination process. Note that these pattern vectors

are most separated from each other in Figure 3-6.











Fiue35 tadr V nal sea abl Deapefo ifrn ereadih



3.2 Seeciv Supr etrRgeso
Th smlait etee heSV lasiiato ad V rgrsio, hchissude
indti nChpe 0----antral exeso fseetv V mtost V








FInu s-. standard supor vcor ersin teipti a setaal D xl fof paifrsn ofapterne vecdwtor

and deendet vlue S={(xy),...,xy)}.Ithselective support vector Rgeso





Figur 3-.SlcieSM nasprbe3Deape odponsaeslce
regrssin, w hae sts X = xe,,...Xi~} ofpaternvectrs ithcorrspodin


dependen vausfrec e ={(1 1,..., X ,.Temi ojciei eetn
a singl patten vectr fromeach se such hat th objecivefnto fteotmzto





problem (2-57 smniie.Asih SelSVMv wV nasprb e st eartl by l defiing s ard selectionfo




support vector regression.
Definition 3.2.1 (SelSVR Hard Selection). Let X = {X1,...,X,} be sets of pattern
vectors with t pattern vectors xeil, .. .,xi~t in each set Xi, and let y = {yl, .. ,ye} be the
corresponding values for each set with each pattern vector in Xi having the same value
yi. Choose ira. 110 one pattern vector from each set such that the sum of the E-inSCRSitive
errors between the pattern vectors and the regression function is minimized.
The hard selection support vector regression problem can be formulated as a
quadratic mixed integer programming problem as follows,













minimize 2 ||w ||2 2 k sk)(-9
i= 1

subject to

((w Xi,k) + b) yi < E + i,k + M~(1 vi,k) i= 1.,;k= 1,..., (3-20)

yi ((w Xi,k) + b) < E + i,k + M~(1 vi,k) i = 1,... ; k = 1,..., (3-21)


Di~~ Uiik= 1 (3-22)


vi,k E {0, 1} =1..) k= 1,..., t. (3-23)


In the above formulation, M~ is a large positive number, such that for those points

with vi~k = 0, the related constraint is ah-li-w satisfied, and thus, does not have any

influence on the problem. This is equivalent to removing this pattern vector from the

problem. M~ can conveniently be set to the maximum distance between the distances of

any two pairs of pattern vectors. Constraints (3-20) and (3-21) account for the case if a

pattern vector is above or below the regression function, respectively. Finally, constraint

(3-22) ensures that only one of pattern vector from each set is selected. The hard selection

problem, like in SelSVR case, is a highly combinatorial problem and there is no efficient

solution known.

We consider relaxations of the hard selection problem in order to obtain an efficient

and effective solution. Instead of the binary variables, we introduce continuous variables

for each pattern vector, total value of which is bounded. This is the same framework of

relaxations that we discuss in Section 3.1. Likewise, we include two approaches for the free

slack values. In the pooled free slack (PFS) case, each pattern vector can be assigned free

slack regardless of the other pattern vectors in their set, where as free slack per set (FSS)

bounds the free slack values for each set with 1.











We cover two formulations based on the penalization of the slack variable, namely

selective i) quadratic, and ii) linear E-inSenSitiVe lOSS TegreSSion. For each formulation, we

consider both the general and free slack per set approaches.

3.2.1 Selective 2-norm E-InSenSitiVe RegreSSIOn

Assume that there are sets Xi, i = 1,..., n of vectors, Xi,k, k = 1,... t. Then,

the following optimization problem finds the regression function that minimizes the loss

function, given that the total general free slack for all pattern vector iS nE.


n t
minimize ||w |2 3k s )
i= 1 =


(3-24)


subject to


((w Xi,k) + b) yi < E + i,k U Ii,k

yi ((w Xi,k) + b) < E + i,k U Ii,k.
u t

CCiVik i'ik) < RE
i= 1 = 1


\* RI

\* RI


l, *, *

l, *, *


(3-25)


i= 1,..,n;k= 1,..., t.


The Lagrangian function for (3-24) is given as,


i= 1 =

a t

ai~~k E Lilek Ui,k ik ii) -b
i= 1 = 1
a t

i~k E i~k i,k (W -Xik) + b]
i= 1 = 1


att


i= 1 k=


(3-26)










Differentiating this Lagfrangfian function with respect to the decision variables

from (3-24) w, b, and each (4, (4, vei, I^ and imposing stationarity, we get the following

expressions :


(&ik -tih)Xi~k = 0
i= 1 = 1


8L/8w = w


(3-27)


(3-28)

(3-29)

(3-30)

(3-31)

(3-32)


~(ti~k ti~) = 0
i= 1
Ci,k t~i,k = 0

i't~k tii~kl = 0

= t~i,k rli = 0


8L/8b =

8L/8(i,k

8L/8(i,k

8L/8vi,k




Substituting the expressions (3

formulation:


-27 3-32) in (3-26), we obtain the following dual


maiximize -C~ I (a ,k ,

ntnt

ii= 1 =1 j= 1 l= 1
at at
E ti~~~~k i,k il ck -tik
i= 1 k= 1 i= 1 k= 1

subject to


(333)


n~ t

j=1 k=1

0I < i,k, < P


tirk) = 0

for i = 1


,...,8 ; k

,...,8 ; k


1,...,t.










Fr-om the standard kernel trick, the dot product (xi, xj) can be replaced with a kernel

K(xi, xj) for nonlinear regression.

The free slack per set approach results in a similar formulation, where constraint

(3-39) is replaced by the following constraint for each set i, which limits the free slack to a

per set.


D~i~k U'i; < E i ., n.


This change reflects on the Lagrangian function such that the term


(-84)


/7 na


i= ~1 k=


(3-35)


is replaced with,


E C ii iUiV,~k U Ci~f).

formulation for the free slack per set is given as,



a t

-C (a .,k ",kl

ntnt


(3-36)


i= 1 k=1 j= 1 l= 1
at at

i= 1 k= 1 i= 1 k= 1


maximize


(337)


i= 1


t~i,k)


subject to
a t

(t~i,k in)
j=1 k=1

0
0<<^1ii~k Ii for i


0


=1,. n k

=1,. n k


1,...,t.


The resulting dual










The dot product (xi, xj) can be replaced with a kernel K(xi, xj) for nonlinear

regression.

3.2.2 Selective 1-norm E-InSenSitiVe RegreSSIOn

The second alternative as in the SelSVR problem is to penalize the 1-norm of the

non-free slack for each pattern vector. Because of the similarity to the 2-norm case, we

present only the primal and dual formulations, and skip the intermediate steps. The

primal problem for selective 1-norm e-insensitive regression is given as,




minimize ||w||2 i+ i(k 8


subject to

((w -Xi,k) +b) yi
yi ((w -Xi,k) +b)

(vi~kUi~k)< RE(3-39)
ii 1 = 1

(i~k i~k 0 i 1,..,n; k= 1,..., t.

vi~kUi~ > i= ,..,n;k= 1,..., t.


Deriving the Lagrangian function, differentiating it with respect to the decision

variables in the primal problem, imposing stationarity and substituting the resulting

expressions back in the Lagfrangfian function we obtain the following dual problem.













ntnt
maximize -(ik- ik jl- j1 Xk Xj1
ii= 1 =1 j= 1 l= 1
at at


i= 1 k= 1 i= 1 k= 1

subject to


a t

C(ti7,k ti) =
j= 1 k= 1

0

0<&~ii,k IP

1,...,n;k


1,...,n;k


1,...,t.


In the free slack per set approach, we allow a free slack of E for each set. This means

that the constraint regarding the free slack needs to be separate for each set of points. The

primal problem for this variant is given as,


a t
minimize ||lw||2 i2 ~ )
ii= 1 =1


(3-40)


subject to


((w -Xi,k) + b) yi <


Yi ((w Xi,k) + b) <




i~k, iik > 0


+ (ik Uik


+ (ik Uik*


- *


* *


* *.


* *.


i ., ni


1,...,n;


1,...,n;


= ,...,t.


= ,...,t.


Through standard Lagrangian function, differentiation, optimality

substitution, we obtain the dual formulation given as,


conditions and













maximize -C C~i, ( ilk ti~k jl -t,) ik- j
ii= 1 =1 j= 1 l= 1
at at n
-C E ti,k t iL~k) Ii CZ.tiL,k t~ik) -C E i
i= 1 k= 1 i= 1 k= 1 i= 1

subject to



j=1 k=1

0
0<&~ii,k Ii

In both of the dual formulations for the 1-norm case, the dot product (xi xj) can be

replaced by the kernel K(xi, xj) for nonlinear regression.

We tested the efficiency of selective SVM regression on a 3D surface which is given

by points (x,: y, z) such that z = sin(r)/r, where r = 2 2.. The selective SVR

is compared with a native method in which regular SVR is used and whichever point

was closer to the predicted surface, was included in the winning set. For selective SVR

winning points are those chosen by the elimination method. The two classes of points

come from the underlying function plus noise. In the first group of pattern vectors, x and

y coordinates were distorted with a uniform noise within a box of 1 by 1 unit, whereas the

second set of pattern vectors' coordinates were distorted uniformly within a box of 3 by 3

units. The results are encouraging in the sense that although there were inconclusive cases

between which point to be picked, after settling a threshold value on the amount of slack

required by a pair of points, the ratio of blue point (with less noise) selected to red points

selected ahr-l-w dominated the native method, as given in Figure 3-7.


































[-0.5, 0.5] [-1.5,1.5]








51/3 = 14 4/14 2.8 7150 =I.4





59/62 =cl 095 7150=1.2 853=

Figue 3-. Seectie SV copare to naie mehodfor iffeent andwdth














rr~l rr63









CHAPTER 4
K(ERNEL METHODS APPLIED TO NEURAL DATA

In the previous chapters we reviewed the kernel-based machine learning algorithms,

more specifically support vector machine (SVAI) methods, which include classification,

feature selection and regression methods. We introduced novel selective classification and

regression methods. This chapter involves the neural data set on which the methods we

have developed so far are applied. In a fact, this data set is the main motivation to apply

kernel-based machine learning algorithms on neural data.

The data set consist of local field potentials (LFP) from primate brains during

a visual discrimination task. We give further detail on how these experiments were

performed, including the technical details and the experimental paradigm. Then we

introduce the SVAI methods for classification and feature selection. The results from the

application of these methods are reported and illustrated for three macaque monkeys.

The selective classification methods are introduced to account for the temporal variations

among the many repetitions of the same cognitive task which are referred to as single

trials. The improved results after the application of selective SVAI methods are also

presented.

4.1 Visual Discrimination Task

The experiments to acquire the data used in this study takes place in the Laboratory

of Neuropsychology at the NIMH between 1984-1988. The data are collected from

three young adult macaque monkeys (11 II II I mulatta) GE, LU and TI. This data has

previously been used in a number of studies [57-60]. All three monkeys are well trained

to respond to a go/nogo visual stimuli before the electrodes are placed in these monkeys'

brains. Surgeries to place the electrodes are performed under sterile conditions with the

monkey under general anesthesia. Teflon coated platinum-iridium wire bipolar electrodes

with a 0.125 mm diameter are inserted in the hemisphere contralateral to the hand used

in the task, with the less advanced tip of the electrodes extended 0.5 mm into the dura










and the more advanced tip extended 2.5 mm into the cortex. For more details, see [60]. In

Figure 4-1 the electrode locations that are used in this study are shown.

GE LU TI





Figur 4-1 Aprxmt lcmn feetoe ntete oky

Loalfil po tal (LFP dat are colc rmeahmne uin ubro
sson.Eahsssio onits aron 10trlsThdta ar adps ilee ewe
1 n 10H addgiie a 0 H.Fr eac tildtarerCore o 00m fe






in Fiure 42. Te stimul arereferred pas right of lantode line and trigh molnted imdi


the a firs rwad loeft.lntedl (lie dand lref colated diamon iahmn tey sod row. Notber th

liess. reac gien ion thefirst clm arond diamondias T dt are gie i hescnd-ps folumnd The go

stml isd chos to bed eiithzer both line Hor both driamondsa and never any othr po ssibler

cobnto fthe foure itaes tetrimulus types Notwee thahen thilae two line stimlind te w

diaon stimuli are dsurimpoed cet o into eanhote sepaately, th sing twotsuaesa shapes

re Fiduentica. Therefore, thre monkrey has rgto dsltinuih line fndromh sadimntd byan coretl

realizing the pndlaemet of ate lie ast two squartes. Then tot area scontatedgew loengthand

righness are gonstnt acos the stimul typesn and timn re ial. inThe stimuli is laed 57e gm


from inthen subjet an isoveal sizulsyeis 6 degres bt ween the tour corner sqiuliars The w






presentation time is computer controlled with a piezoelectric shutter.

A trial is initiated by the monkey pressing a lever and keeping it pressed. The

initiation is followed by a random amount of time, uniformly distributed between 1200












































~1
*;;;;;;;;;;;;;;;;;;;;;;;;~;il


........................................
::::::::::::::::::::::::::::::::::::::::


O HO

M HO
MO1 0 [

57 D 0r








Figure 4-2. Visual stimuli used in the experiment


and 2200 ms, before the stimulus appears on the screen for exactly 100 ms. The monkey

is expected to respond to a stimulus within the 500 ms following the stimulus onset. The

release of the lever is the correct go response, and keeping it pressed is the correct nogo

response. The correct go responses are rewarded with a small amount of water, whereas

correct nogo responses are not rewarded. The outline of the task is presented in Figfure

4-3.


D8taen~dy~is
Rpward
FIggpb~ wind~rv
Lever press, NOG(5 .---------------------------------------
~Blrer pe4;s BLI ~
SDnauC~s

Time (me)l
-f30D to -9~0 ioa


rr


0 ,,,,,,,,,


100 2100 300 4010 500 6000


Figure 4-3. Experimental paradigm



In all sessions, there are two different stimulus-response contingencies: line-go/diamond-nogo

and diamond-go/line-nogo In this study, only correct go and correct nogo trials are

considered. For each trial LFP data recording starts 90 ms prior to the stimulus onset










and continues for 600 nis. After the digfitization at 200 Hz, each electrode produces 120

discrete recordings with 5 nis between two consecutive recordings, for each trial.

4.2 Methods

Support Vector Machines (SVA~s) are the state-of-the-art methods in machine

learning literature. Although it is a relatively new methodology that emerged within the

last decade, its application spectrum covers a rich variety of problems. The most general

application of SVA~s is classification. There are recent studies on the classification of

neural data using SVA~s [8, 9]. Considering the niulti-dintensional nature of the problem

and the nonlinear behavior of the brain, SVAI classification could be used on each time

point to detect different stages of early effects of visual stimulus, discrimination of the

stimuli type, and preparation and execution of the response. Moreover, the contribution

of the channels to the classification at each time point would reveal valuable information

on the spatioteniporal interaction between the visual system and the motor system. In the

following subsections the SVAI method is briefly introduced, followed by the SVAI hased

feature selection technique applied in this study.

4.3 Classification and Channel Selection Results

In this work we are interested in detecting the spatioteniporal effects in the visual

and motor cortexes of a macaque nionkey's brain during a visual discrimination task. To

obtain these results, we have distinguished different sets of labels, considering a different

effect for each label. In the following table different labels are associated with different

effects that are to be detected. We distinguish between three effects: i)early effect of

visual stimulus, ii) discrimination of stimulus, and iii) response preparation and execution.

For each effect type, binary SVAI classification has been applied by labeling the trials in

different combinations.

The reason in associating these labels to the different effects are as follows: 1) For the

early visual effect, the detection of the direction of the stimuli (left or right). 2) For the

discrimination of the visual stimuli, the monkey has to distinguish between a line and a









Table 4-1. Different stages of the visuomotor task


Stage Label
Early effect of visual stimulus Left vs. Right stimuli
Discrimination of stimulus Line vs. Diamond stimuli
Response preparation and execution Go vs. Nogo


diamond stimuli. In the motion part the main effect is the preparation of the impulse sent

from monkey's brain to its hand and its execution.

We are consider two sets of data, where in the first one lines are go stimuli and the

diamonds are nogo stimuli. In the second one diamonds are go stimuli and lines are nogo

stimuli. For each monkey we have around 10,000 trials. Since the combinatorial result of

the classification scheme, and the number of time points is computationally demanding,

for each monkey, between 211' and 311l' of the available trials were used to perform the

classification. The trials are randomly selected from a combined set of data involving all

the trials from all the session. We group the trials in various owsi~ with respect to their

stimulus type. If we let a trial with right line, right diamond, left line and left diamond be

marked as si, sj, s6 and s}, respectively, coming from data set i = 1, 2, then we can form

three different sets of data with respect to our interest. The possible combinations are:

1. RIGHT vs LEFT (S+ = 1U s, U s2 U vs. S-= UsUs Us)

2. LINE vs DIAMOND (S+ = 1U s, U sl U sl s S- =s iU Us)

3. GO vs NOGO (S+ = s] U s' Us U .s2 s S- = s U s' Us U sj)

In Figures 4-4, 4-5, and 4-6, the classification and the feature selection results are

given for the monkeys GE, LU and TI. The three columns of graphs in each figure

correspond to the three different stages we are interested as listed in Table 4.3. In the first

row of graphs in each figure, the overall accur r. ;, results are presented. In the second row,

the contribution of each channel to the overall classification are given. we refer to these

plots as channel selection plots. The channel names and corresponding cortical regions

are given in Table F. K~nowingf the contribution of each channel for a time point t, we

can sort the channels in descending order of their contribution. In the plots in the third












TI LU GE
N ehn Abbr. \.,.4Abbr. n Abbr. 1,
1 StrB Striate Supr StrA Striate A
2 PstA Prestriate StrF Striate StrB Striate B
3 PstB Prestriate PstS Prestriate StrC Striate C
4 AudA ParE Parietal PstA Prestriate A
5 IftB Inferotemporal AudB PstB Prestriate B
6 MotH Motor FrnD Frontal PstC Prestriate C
7 Fr-nB Fr-ontal SomV Somatosensory FefA FEF
8 StrF Striate SomM Somatosensory MotB Motor
9 IftA Inferotemporal MotH Motor SomA Somatosensory
10 SupT StrB Striate ParD Parietal
11 FrnD Fr-ontal StrC Striate ParE Parietal
12 FefA FEF IftA Inferotemporal PrmA Premotor
13 InPr Inferior-Parietal AudA PrmC Premotor
14 Supr Fr-nA Frontal FrnB Frontal
15 / / / / FrnG Frontal


row, for each time point we show the change in classification accuracy as we incrementally

introduce the sorted channels. The horizontal axis is the time, and the vertical axis shows

the number of channels included. We refer to this plot as the incremental accur r. ;, plot.

Results for GE.

RIGHT vs LEFT: The early visual effect, i.e. the right-left separation, is experienced

around 110 ms after the stimulus onset with a very sharp peak. The separation between

trials labeled as right and left diminishes slowly, reaching a dip at 240 ms, and makes

another peak around 300 ms, which diminishes again back to an undetectable level. This

behavior can clearly be observed from the feature selection plot, in which channel 5 is

clearly very significant wherever there are peaks in the accuracy plot in A. Also, the

plot, in which the sorted features are added one by one to the classification looks almost

not effected starting with only one feature. This means that only one channel would

be enough to do the classification in this case, which supports the results of the feature

selection plot.


Table 4-2. C'!I. .ill. I names and locations for TI, LU and GE










L/INE v~s DIAM~OND: The line-diamond case is almost in parallel with the right-left case

except for a small shift of the dip between the two peaks, and the second peak occurring

around 210 ms and 280 ms, respectively. However, the feature selection for this case is

not dependent on a single channel. The spots that have a high contrast for a short time

interval are from channel 11 and :3 corresponding to the first peak, and channels 5 and 6

corresponding to the second peak. This observation is also supported by the increasing

number of features included in the classification.

GO v~s NOGO: Although the go-nogo case does not have as sharp peaks as in the previous

two cases, there is a sharp increase with a first peak at 200 ms, followed by two other

dips at 250 ms and :360 ms, and with slight dips in between the peaks (A, go/nogo).

After :360 ms it diminishes and continues as a plateau at a significantly high classification

percentage rate. The significant channels that contribute to the classification are 9 and

15 almost throughout the time line, and 5 and 6 for the first peak, 9, 10 and 12 for the

second peak, and 9, 10 and 15 for the last peak (B, go/nogo). Including very few number

of best channels performs well in classification, however, as more channels are included

the accuracy drops slightly, possibly because of the increased noise. As more channels are

added, a combination of channels help classification and increase the accuracy again.

Results for LU.

RIGHT v~s LEFT: The early visual effect can he seen starting at 100 ms. with a sharp

peak at 110 ms and drops back to a low level around 200 ms. ('I! ,Ill., I 11 contributes to

the classification the most and it is the only significant channel at the feature selection

plot. This is also verified by the incremental best features plot the figure such that the

addition of new features does not change the classification accuracy.

LINE v~s DIAM~OND: There is no peak at around 100 ms but the accuracy starts to

increase, reaching its first peak at around 160 ms. There are two other peaks with higher

accuracy at 270 ms and 420 ms. ('I! ,Ill., I 8 is the most significant channel in the feature

selection plot with high intensities around the peaks in the overall accuracy plot. From the










bottom plot, it is observed that increasing the number of best features introduces noise

and decrease the classification accuracy.

GO v~s NOGO: In this case, there is a classification rate around 68 percent between the go

and nogo cases even prior to the visual stimulus onset. The first increase in the accuracy

appears at around 280 ms, followed by a dip at 240 ms, and a high peak at :350 ms. The

most significant channel is 8 in this case. ('I! .Ill., I 2 and 14 creates noise and decrease the

accuracy between the interval that corresponds to the dip in the overall accuracy graph.

This effect is also verified by the incremental features graph; the accuracy is the highest

with the best channels, and it starts dropping as more channels are included, however

starts increasing as even more channels are introduced.

Results for TI.

RIGHT v~s LEFT: The only sharp increase is at around 110 ms followed by a steady

decrease. C'l! l~! .1.I 2 is the only significant channel in the feature selection plot. Therefore,

adding more channels does not change the classification accuracy as seen in the incremental

best features graph.

L/INE v~s DIAM~OND: The accuracy increases slightly at 100 ms and cr li-- the same until

240 ms, at which point there is another slight increase, reaching its highest value, until 280

ms. The only significant channel is 6, whose highest interval corresponds to the highest

point in the overall accuracy graph. The significance of the other channels over any time

interval are indistinguishable from the feature selection graph.

GO v~s NOGO: In this case, there are two highly significant peaks. The first one starts

at 150 ms and reaches its maximum at 200 ms, followed by a dip with the lowest point

at 260 ms. The second peak starts at :300 ms and reaches its highest point at :350 ms.

C'll Ia!l., I 8 is the most significant channel, except for the interval where the dip occurs.

The classification accuracy decreases as more channels are included because of the noise

introduced, however further increasing the number of channels increases the accuracy

slightly.










In Figure 4-7 comparative snapshots of peak activities are given for three monkeys

together with the times. The location and the time are consistent for all three stages.

4.4 Time Series Alignment with Dynamic Time Warping

Brain is a very complex and highly dynamic system. Although the result of the

go-nogo experimentation we study is identical in the sense that the desired outcome

is met, the progression of stages involved in this visuomotor task may vary over time.

This phenomenon can directly be observed in the variation of the response times. More

formally, the start time of a stage, as well as its duration may differ from one trial to

another. Since each stage is dependent on the previous stages, the variations are expected

to increase over time. The applications of the standard SVM methods across trials for

each time point ignore the effect of this variation in the stages over the time line. In this

section, we use a method to align single trials with each other in order to decrease the

variation across the trials for a given time point t.

Single trials are composed of simultaneous recordings from multiple channels.

Assuming that there are d channels, the recorded values at time t from all channels

from an individual trial can be considered as a point z(t) = [zz (t), .. ., za(t)]T in R d,

and the entire trial as a discrete representation of a 1-dimensional curve in R d. A simple

measure to find the similarity between these two curves z' and zb of length T is to find the

sum of the difference vectors for each time point, i.e.,




t=1
Using dynamic programming, the distance between the two curves can be minimized

by stretching these curves to get a better match. which implies that some points may

be deleted or gaps may be inserted in either curve. D;, i' Time War-ping (DTW) is

one of the available matching methods that can be applied on our data [61]. The original

algorithm is proposed for one dimensional curves. It is fairly easy to extend the method to

d-dimensions for curves za and zb aS follows,











DTW(z'(t 1),zxb t)) Ciq

DTW(z"(t), zb(t)) = DTW(z'(t), zb t 1)) C d (4-2)

DTW(z'(t 1), zb t 1)) Cm

where cs, cd and cm are the associated costs for insertion, deletion and match. In the

current application ci = ca = 1 and cm = 0. The matching and its score can be found

by backtracking and adding the costs up. The resulting cost is called the edit distance
between z' and zb

Smaller distances between trials mean that the trials are very similar to each other.

A subset of similar trials may significantly reduce the variation of the trials with respect

to a given time point t. Reduced variation, in return, reduces the noise, and increases the

classification accuracy and the significance of the feature selection process.



The edit distance between two trials, which is a similarity measure, can be found

using dynamic time warping (DTW). In Figure 4-8, a raster plot of the pair-wise distance

scores for 110 single-trial recordings regarding right slanted line stimuli are shown. We

sorted all 110 x 110 scores in the matrix and then plotted the sorted scores on the right.

This plot shows two levels of similarity among trials. We group the trials with scores

below 90 and above 90. This pattern is very consistent over all other trials regarding

different stimuli types with a threshold between 90 and 100. When these scores are

divided into low (0) and high (1) values, it is easy to find similar trials by summing the

scores for each trial and sorting the trials by their total score.



A subset of about 35 trials from the top of the list can be considered to be very similar to

each other. Therefore, the classification and feature selection algorithms are expected to

give better results due to reduced noise. The same method is applied to each different set

of trials regarding different stimuli and different sessions to find subsets of similar trials. In










Figure .the classification and feature selection results are compared between a randomly

selected set of trials and the set of similar trials determined by DTW. In Figure 4-9 The

improvement after using DTW is very clear for stimulus, categorical and response related

differences.

4.5 Selective Support Vector Machines on Neural Data

As explained in Section 4.4, the variation in the stages of the visuomotor task over

time precludes using an alignment method to decrease the variation across trials for a

given time point. Although the results from the standard kernel-based machine learning

methods are intuitive and conclusive, as shown in (I Ilpter 3, better results may be

achieved using selective kernel-based machine learning methods. However, one should note

that DTW uses the data in the original input space, and works with euclidian distances,

whereas similarities between pattern vectors may be better explained in a nonlinear

feature space. Selective SVM methods achieve the simultaneous goal of classification in

a nonlinear space and alignment of pattern vectors at the same time. We show in this

section that the results obtained with the selective SVM methods dominate the results

using standard methods like DTW.

The motivation behind the development of selective SVM is the variation in the

underlying processes that are to be distinguished by classification. The basic idea is as

follows. Let Sj be the a matrix where each row i of a rows is a vector of recordings from

tribal i: at time point, j. Now consider th~e a~ppended maitrix |STS zz ... S,+t-1|, which?

involves n xt rows of vector patterns, such that the corresponding rows of Sj and Sj I are

two consecutive recordings over the time line from the same trial. This matrix is composed

of all pattern vectors within a window of size t from j to j + t 1. Our objective is to

provide SVM classifier with more pattern vectors to increase the chance of better point

selection. However, the classification algorithm will be effected by the noisy points. We

implicitly assume that one point of t from each trial will be allowed to be chosen. In

fact, the pattern vector chosen is considered to be a good point, which is expected to










appear, ;?i, at the middle of the time window, however some jitter towards each side of

the window caused it to appear within a close neighborhood.

Since selecting the best point from each window and each trial is a very hard problem,

we use the pooled free slack (PFS) to eliminate the worst points one by one to achieve the

desired points from each window. Although this is a heuristic approach at a first glance, it

gives very good results in classification and feature selection.

The code for selective SVM is written in CPLEX 9.0 and dynamic library files

are produced to be used in accordance with MATLAB environment. Because of the

computational limitations of these software, the entire data could not be solved. The

data consists of around 4000 trials. The strategy followed is to consider 200 trials at

a time (observed limits of the software) with almost equal distribution of positive and

negative labels. Selective SVM algorithm is run over a window of 3 pattern vectors,

which is equivalent to 10 milliseconds. The pattern vector with the highest slack is

eliminated from each set. This is repeated until there is only one pattern vector per each

cell. Although each batch of 200 trials are independent of each other, the cumulative

set of the selected points are consistent. The classification and feature selection results

significantly dominated the improvements obtained from dynamic time warping method.

Since selective SVM tries to find better points, there is an increase in accuracy at the

part of the recording before the visual stimuli. This increase poses the question whether

this is just an overall increase at every time point. If this is the case, we are not actually

detecting anything about the progression of the visuomotor task progression over time.

However, when we graph the percentage increase, it is clear from Figure 4-10 that for

line-diamond separation, we achieved not only MI' accuracy results between time 130

ms and 200 ms, but also, almost twice the amount of the percent increase compared to

the rest of the time scale. This confirms that 40-50 ms after stimulus arrival starts a

differentiation in the brain regarding to the abstract categorization of the concept of a










line and a diamond. We want to make a note that the previous studies using standard

statistical methods failed to detect any difference for the line-diamond separation.

The go-nogo separation is somewhat confusing because the highest percent increase

is before the onset of stimulus. However, this is due to the bias introduced in the brain

probably from the uneven distribution of go trials which may be performed back to back

and thus carrying over the brain state from one trial to the next. However, it can also be

observed that such separation decreases by the time stimuli is arrives to the striate cortex.

This point on the time line can he considered as a priming point for this artifact after

which a jump in the percent increase can he observed. This jump from 200 ms to 260 ms,

being too early for the motor response may correspond to the decision to respond or not.

The results on feature selection is also quite significant. In Figure 4-11, the left

column is line-diamond, right column is go-nogo, upper row is feature selection results

after DTW, lower row is feature selection results after selective SVAI. Note that the

interaction of channel 8, which shows the motor response is not observed prominently on

the line-diamond case. Instead striate and prestriate cortexes contribute to classification

for the aforementioned time periods, for which categorical and response related differences

can he observed. However, the assumption that the possibility of back-to-back go trials are

confirmed by the excited activity on the motor cortex, even before the stimulus onset.

Since selective SVAI tries to separate the two classes for every time point there is an

average increase of 10I' in the accuracy. This increase can he counteracted by creating a

baseline. For this approach we randomly assign class labels to pattern vectors so that we

can deduce the increase in the accuracy for random data. In Figure 4-12, the improvement

is very clear after the baseline approach. It is also worth noting that there is an actual

decrease in the baseline where there is a prominent increase in the accuracy. This case,

which has not been detected by any previous studies is very prominently detected using

selective SVAI method and creating a baseline.


































rm9h9~t
8~90~edQ


w-








a
sr


PNnt~mhmm~Cn~Pn
~rrrC11


I
PN~tnmh~m~Pn~T~




















g,
r~nrnmF~a~rn~r~


i

c



il ;::

~ L




clJ('4~"~r
99~eaB

~

r
c,


Figure 4-4. Classification and channel selection results for GE


r~


C-l
r~nr~mcmm~rnnan




uu



















i







%~C19YI
~b~C


L
*,


MI










"I LtJ o~""


."-i --~c~s


LYJ


Figure 4-5. Classification and channel selection results for LU


~1
111



i

~-nm+so~cmaro--~nr

------rr




19
~


i::::: :::::!:::: ::::~


% m:E








































%~a~a~


~Nmu~c~~m~=Nm~


-Nn~nmh~~"'"""


O1OPh~q~
ac~acs~e~


J



rr k~i

ri
icl



-r

r~msn~cm~o-~~v


Q1~q'qqr
r,,,,,,

rrl

PY,


rNmlPshql~.Qr~~r)TI
rrrrr

r
,IC1
Y
"-)

,ul


Figure 4-6. Classification and channel selection results for TI


~;j~s~,44

i;iY I

-Nnrn~c~~~-9~Z


I ~82;




,nnrn~n~~~r~~f




Iu 1
u,
Le





























I
n

bl
,esra


a~i

m
f


as


ed


h
P1


r(

d?
nSi
P A


~O -v"


dl
rQ
rt,
P
A


rsa
3r

,a


(s,



p

u~9a


1? h
t ~e



U
p

~t ~0,
mN


ir


c B


,g








j

E
E


Figure 4-7. Brain snapshots














80
























































** p I lita*FI FI'* *.g p*2









P=F=.F II i. ll= -1I 4 = II r(




me.:.en -ce--n:- .ur
to 10 a 40 0 D 6U 0 BU fa 10


1001


/i


LCu ;rm no qui iua rrn mou um naan *u limu

(b)


5



3


Figure 4-8. a) Uncoded scores mratrix b) Sorted scores c) C~oded matrix d) Sorted matrix






































0 100 211 311 400 5D0






0.5


(a)




sig ii i -I e I I1 r
a III CZHlie I I il 1 i II :i1 lo 1l I

0 100 23 3]0 4DD SE ( I 1 J0 1..d. I


it II I I'l WA '1 iA 1111 I gi III

Random II li Al II II I HAI1
II I: I I :
'41 14i 14
0 11Im 2 ll 411 mo 1 m 10 211) 30 0 40il 91 0 ICo :00 DEC 410 5

(b)


Figure 4-9. a) comparative classification results b)comparative channel selection results


































I? I
1 .1Il5


10 1 li ll I I
ClII II
0 no 200 alo 400 saon



18 II 311 I I IS


ggg1





O tool 2001 me00 rso nD





II IIi i II
0i' In 29 30 s0050


Figure 4-10. Improvement classification for line-diamond (left) and go-nogo (right)


O a20 14 19 M 4


Figure 4-11.


Improvement in the feature selection for line-diamond (left) and go-nogo
(right)


~~


ji~~l 'i I lll






























:L
'



'''i

i' -.i li;i :i' ilrj i'ir


ine~ vs Diamond


SelSVME vs Baseline


SelSVMR mninus Baseine


Figure 4-12. Baseline approach to emphasize improvement (top), difference between actual
improvement and baseline (bottom)









CHAPTER 5
GENERALIZED EIGENVALUE CLASSIFIERS

In this C'!s Ilter we discuss a different class of Support Vector Machine classifiers

usually referred as Proximal Support Vector Machines, which was popularized by

Mangasarian et al. In this family of classifiers, the notion of a separating hyperplane

turns into two hyperplanes, each approximating its own class by minimizing the total

distance from its pattern vectors while maximizing the total distance from the opposite set

of pattern vectors. The advantage of the proximal classifier model is that the underlying

minimization problem reduces to an eigenvalue/eigenvector problem, whose solution is

easy to find.

Our main contribution in this chapter are three-fold. The first one is an improvement

that would reduce the running time of the current method in the literature. The method

developed by Mangasarian requires two eigenvalue problems to be solved, one for each of

the approximating hyperplane. However, in Section 5.2, we show that solving only one

problem would suffice, using a new regularization technique to find both hyperplanes.

We show comparative computational results of our method with the current one. In

Section 5.3, we discuss an implementation of the proposed method on parallel computers

and presents computational results. Considering a dynamic training set, we introduce

an incremental version of the proposed method in Section 5.4 that uses a significantly

small kernel to reduce computational complexity of classifying new training points, and

improve the generalization capability of the classifier. Similarly, computational results are

presented for the incremental method.

5.1 Generalized Eigenvalue Classifiers

In this section we are going to switch to a slightly different notation. Now, consider

that Anz xd = [X1 X2 X,] iS a matrix formed by all pattern vectors xi ERI~ from the

first class, and similarly Based is a matrix formed by all pattern vectors from the second

class, and the total number of pattern vectors is nl + n2 = n










Mangasarian et al. [16] proposes to classify these two sets of points A and B using

two hyperplanes instead of a single separating hyperplane as in support vector machines.

In Mangasarian's approach each hyperplane is closest to one set of points, and furthest

from the other. Let xTw y = 0 be a hyperplane in R d. In order to satisfy the previous

condition for the points in A, the hyperplane for the first class of pattern vectors can be

obtained by solving the following optimization problem:


||Aw er||2
mm (5-1)
w,ywo ||Bw ey||2'

Here, e is a vector of 1's. The hyperplane for B can be obtained by minimizing the

inverse of the objective function in (5-1). Now, let



G = [A e]" [A e], H = [B e] [B e], z = [wT y] (5-2)

then equation (5-2), becomes:
z'Gz
mmn (5-3)
zEpnm zTHz

The expression in (5-3) is the Raleigh quotient of the generalized eigenvalue problem

Gx = AHx. The stationary points are obtained at and only at the eigenvectors of (5-3),

where the value of the objective function is given by the eigenvalues. When H is positive

definite, the Raleigh quotient is bounded and it ranges over the interval determined by

minimum and maximum eigenvalues [62]. H is positive definite under the assumption that

the columns of [B e] are linearly independent. The inverse of the objective function

in (5-3) has the same eigenvectors and reciprocal eigenvalues. Let zmin = [wl y,] and

zmax = [W2 y2] be the eigenvectors related to the eigenvalues of smallest and largest

module, respectively. Then x~wl yl = 0 is the closest hyperplane to the set of points in

A and the furthest from those in B and x"W2 y2 = 0 is the closest hyperplane to the set

of points in B and the furthest from those in A. This is depicted in the examples shown in

Figure 5-1.

















Fiur 5-1. Sepraio oband ihgeeaiedegnvcos


A stndr tehiu nSoootanagetrsprbliybtenst st
embed the patr etr noannina pcvakre uctos nti okw s
the ~~ Gasin nl

KOe xy = e ,x2 54

In(-),x n xy deot two pater vc ors intefauesac Ti ehiu
usually gie betrrsls sso ni eea tuis eut eadn olnal
sepaabl prolem 63 64 stil hodadafruainfrteegnausp obe ca
easilyo be deie.Ti omlto sgvni h etscin
5.2 sA` New Reuaie atCasiainMto
Realta n Baetemtiescnann h two clase of traiigpons







carehas o begiven to1 thesolution of thie geeaized egenvaluzed problvemr. Ideih




nulld spaes of G and He r have a nontial inter saecto, via e Ker(A fu Ker(B) I /hi 0r then










the problem is singular and a regfularization technique is needed to solve the eigfenvalue

problem.

1\angasarian et al. proposes to use Tikhonov regularization applied to a two-fold

problem:


(5-5)



(5-6)


and


5.2.1 The New Regularization Method

where 6 is the regfularization parameter and the new problems are still convex. The

minimum eigenvalues-eigenvectors of these problems are approximations of the minimum

and the maximum eigenvalues-eigenvectors of equation (5-3). The solutions (wi, Yi), i=

1, 2 to (5-5) and (5-6) represent the two hyperplanes approximating the two classes of

training points.

In practice, if /3G cOH is nonsingular for every c0 and /3, it is possible to transform

the problem into another problem that is nonsingular and that has the same eigenvectors

of the initial one. We start with the following theorem [65]

Theorem 5.2.1. Consider the generalized ~.:II. ,..rl;,n- problem Gx = AHx and the

transformed G*x = AH*x I. I;,.. l by:


G* = -rG 6,H, H* = -r2H 62G,


(5-7)


for each choice of .scathers T-r~ 2-r 61 (End 62. .such that the 2 x 2 matrix:


(5-8)


||Aw er||2 + Z 2ll
mm
wag~o ||Bw ey||2


||Bw er||2 sl Z 2
mm
wag~o ||Aw ey||2










is no,....:..yalar. Then the problem G*x = AH*x hass the .state eigenvectors of the problem

Gx = AHx. An associatedd .:II e..; al;,. A* of the ',r,,, ,-Ormed problem is related to
,:I ':y. e.tle.- A of the any.:l.::al problem by

-r72 61
Ti~ + 62X



In the linear case, Theorem 5.2.1 can he applied. By setting -rl = -r2 = 1 and

bl -61, 62 -2, the regularized problem becomes


||Aw eq||2 61|| IBw -y e|2
mmn (5-9)
wyO o |Bw ey ||2 2||~Aw ey ||2

If Gland 62 are 1101 negative, R is non-degfenerate. The spectrum is now shifted and

inverted so that the minimum eigfenvalue of the original problem becomes the maximum

of the regularized one, and the maximum becomes the minimum eigenvalue. Choosing the

eigenvectors related to the new minimum and maximum eigenvalue, we still obtain the

same ones of the original problem.

This regfularization works for the linear case if we suppose that in each class of the

training set there is a number of linearly independent rows that is at least equal to the

number of the features. This is often the case and, since the number of points in the

training set is much greater than the number of features,Ker(G) and Ker(H) have both

dimension 1. In this case, the probability of a nontrivial intersection is zero.

In the nonlinear case the situation is different. Using the kernel function (5-4), each

element of the kernel matrix is


K(A, B)i, = e ||iB|2(510)


Let











C=I:


then, problem (5-1) becomes:

||K(A, C)u ey||2
mm (5-11)
neg~o ||K(B, C)u ey||2'

Now, the associated eigenvalue problem has matrices of order n + 1 and rank at most d.
This means a regfularization technique is needed, since the problem can be singular.

We propose to generate the following two proximal surfaces:


K(x, C)ul y, = 0, K(x, C)u2 y2 = 0 (5-12)

by solving the following problem


||K(A, C)u ey||2 +6||Ka;u ey||2
mmn (5-13)
neg~o ||K(B, C)u ey||2 +6||K;Au ey7 2
where K~A and KsB are diagonal matrices with the diagonal entries from the matrices

K(A, C) and K(B, C). The perturbation theory of eigenvalue problems [66] provides an
estimation of the distance between the original and the regularized eigfenvectors. If we call

z an eigenvector of the initial problem and z(5) the corresponding one in the regularized

problem, then |z z(5)|I = O(5), which means their closeness is in the order of 6.
As mentioned in the previous section, the minimum and the maximum eigenvalues

obtained from the solution of (5-13) provide the proximal planes Pi, i = 1, 2 to classify the

new points. A point x is classified using the distance


||K(x, C>U r||2
dist(x, Pi) ||2(5-14)

and the class of a point x is determined as


class(x) = argmini=1,2{dist(x, Pi)}. (5-15)










Let Ae Rw Inx and Be E -.xd be
the training points in each class.
C!....--- appropriate by, 62 ERI and a

Build G and H matrices
S= [K(A, C, a), -ones(nl, 1)];
h =[kernel(B, C, a), -ones(nR211
G = gT I g;
H = hT I h;

R.~~I, I~ rrize the problem
G* = G +6*1..()
H* = H + 62 diag(G);

Compute the I 1 .:;~ r. ..;H.n i,;i'' cl.r: .
[V, D] = eig(G*, H*);


Figure 5-2. ReGEC algorithm


The proposed classification method is outlined in Figure 5-2. Here, K(A, C, a) is the

kernel matrix whose entries are K(i, j) = exp{-(||ai ci||2 U) Where ai is the ith row of

matrix A and cj is the je" row of matrix C, and a is the shape parameter of the kernel.

Function ones(now, neoz) is a matrix of size nrow x acol with all entries 1, and diag(-)

returns the main diagonal of a square matrix.

5.2.2 Computational Results

The aforementioned methods have been tested on benchmark data sets publicly

available. Results regard their performance in terms of classification accuracy and

execution time. We used data from different repositories: UCI repository [67], Odewahn

et al. [68], and IDA repository [69]. These repositories are widely used to compare the

performance of new algorithms to the existing methods. The results regarding the linear

kernel have been obtained using the first two repositories. The third one has been used

in the non-linear kernel implementation. For each data set, the latter repository offers

100 predefined random splits into training and test sets. For several algorithms, results









Table 5-1. Classification accuracy using linear kernel.

dataset n+k dim ReGEC GEPSVM SVMs
NDC 300 7 87.60 86.70 89.00
Cleveland Heart 297 13 86.05 81.80 83.60
Pima Indians 768 8 74.91 73.60 75.70
Galaxy Bright 24632 14 98.24 98.60 98.30


obtained from each trial, including SVMs, are recorded. The accuracy results for the linear

kernel SVMs and GEPSVM are taken from Mangasarian et al. [16] and for the non linear

kernel from [69]. Execution times and the other accuracy results have been calculated

using an Intel Xeon CPU 3.20GHz, 6GB RAM running Red Hat Enterprise Linux WS

release 3 with Matlab 6.5, during normal d w-light operations. Matlab function eig for the

solution of the generalized eigenvalue problem has been used for GEPSVM and ReGEC.

The latest releases for LIBSVM [15] and SVMlight [70] have been used to compare these

methods with SVMs.

In tables 5-1 and 5-2, classification accuracy using linear and gaussian kernels have

been evaluated. Tables columns represent: data set name, the number of elements in the

training set (n+k), the number of elements in the test set and the accuracy results for

ReGEC, GEPSVM and SVMs. In Table 5-1, the accuracy results have been evaluated

using ten fold cross validation. In table 5-2, the random splits of IDA repository have

been used. In the linear case comparable accuracy results have been obtained by the three

methods. Using the gaussian kernel, ReGEC and GEPSVM show similar behavior yielding

alr-ws- results slightly lower than SVMs.

In Tables 5-3 and 5-4, elapsed time is reported. In the linear case ReGEC and GEPSVM

outperform SVMs implementations (LIBSVM and SVM light) in all cases. Furthermore

ReGEC is at least twice faster then GEPSVM. When the gaussian kernel is used, SVMs

implementations achieve better performances with respect to the eigfenvalues based

methods. In all cases, ReGEC is faster than GEPSVM.













Table 5-2. Classification accuracy using gaussian kernel.


dataset
Breast-cancer
Diabetis
German
Thyroid
Heart
Waveform
Flare-solar
Titanic
Banana


n+k
200
468
700
140
170
400
666 b
150
400


test
77
300
300
75
100
4600
400
2051
4900


5
1.e-03
1.e-03
1.e-03
1.e-03
1.e-03
1.e-03
1.e-03
1.e-03
1.e-05


ReGEC
73.40
74.56
70.26
92.76
82.06
88.56
58.23
75.29
84.44


GEPSVM
71.73
74.75
69.36
92.71
81.43
87.70
59.63
75.77
85.53


SVM
73.49
76.21
75.66
95.20
83.05
90.21
65.80
77.36
89.15


Table 5-3. Elapsed time in seconds using linear kernel.


dataset
NDC
Cleveland Heart
Pima Indians
Galaxy Bright


ReGEC
0.1e-03
1.92e-04
1.21e-04
0.3e-3


GEPSVM
0.2e-03
3.58e-04
2.36e-04
0.5e-3


LIBSVM
0.8991
0.0099
15.8737
1.2027


SVM Light
22.0020
0.3801
48.8092
21.1280


Table 5-4. Elapsed time in seconds usingf gaussian kernel.


ReGEC
0.0698
1.1474
3.8177
0.0243
0.0316
0.5962
1.8737
0.0269
0.4989


GEPSVM
0.3545
5.8743
25.2349
0.1208
0.2139
4.4090
16.2658
0.1134
3.1102


LIBSVM
0.0229
0.1323
0.2855
0.0053
0.0172
0.0916
0.1429
0.0032
0.0344


SVM Light
0.1188
0.2022
0.4005
0.0781
0.1372
0.2228
4.4524
7.1953
1.3505


Dataset
Breast-cancer
Diabetis
German
Thyroid
Heart
Waveform
Flare-solar
Titanic
Banana









ReGEC GEPSVM hbSVM


o~a o" loo





Figure 5-3. Separation surfaces obtained with ReGEC, GEPSVAI and LIBSVAI


Finally, a graphical representation of the classification surfaces obtained by

ReGEC, GEPSVAI and SVA~s is given in figure 5-3 relatively to Banana dataset. The

three methods show similar class regions. SVA~s obtain smoother orders and more

regular regions. These differences depend upon the fact that in SVA~s the surfaces are

characterized by the support vectors and the penalties terms, while in the eigenvalues

methods all the points contribute to the solution surfaces. This behavior depends on the

fact that eigenvalues methods ahr-l-~ 0 naxintize the classification accuracy on the training

set with respect to kernel and regfularization parameters.



5.3 A Parallel Implementation of the Fast Classification Method

.Our aim has been to realize an efficient, portable and scalable parallel intplenientation

of ReGEC to be used on different MINID distributed nienory architectures. As well

known, these are multiprocessor computers, in which each node has local nienory

and coninunicates with the others through message passing. Let us suppose that each

processor executes the same program and the same operations on different data (SPMD).

Given the algorithm structure, a flexible connection topology is supposed to exist among

the nodes, that is, point-to-point coninunications are allowed, as well as the broadcast

and gather of data. Finally, we suppose to have a network in which the processors are in

a niesh topology. With this environment in mind, it is natural to develop a program in

terms of loosely synchronous processes, executing the same operations on different data,










and synchronizing each other through message passing. To clarify the exposition, we

suppose that each node is driven by a single process.

5.3.1 Implementation Details

In Figure 5-2, linear algebra operations are essentially matrix-matrix multiplications

and a generalized eigfenvalue problem solution. In order to obtain an efficient, portable and

scalable parallel implementation of ReGEC we decided to use standard message passing

libraries, i.e. BLACS and MPI, and de facto standard numerical linear algebra software,

PBLAS and ScaLAPACK(. Since matrices involved in the algorithm are distributed among

processing nodes, memory is used efficiently and no replication of data occurs. On single

node, the use of optimized level 3 BLAS and LAPACK( routines enables both its efficient

use and a favorable computation/communication ratio.

The main routine of PBLAS used in the implementation of Figure 5-2 is PDGEMM

to evaluate matrix-matrix multiplications. The current model implementation of the

PBLAS assumes the matrix operands to be distributed according to the block scatter

decomposition of PBLAS and ScaLAPACK(. Routines for eigenvalues problems are

not included in PBLAS, but they are covered by ScaLAPACK(. The evaluation of the

generalized eigenvalue problem G*x = AH*x then performed by using the routine

PDSYGVX. We required machine precision in the computation of eigenvalues and,

dynamically allocated memory for reorthogonalization of eigenvectors. Current version

of ScaLAPACK( does not permit to reorthogonalize eigenvectors against those in different

processors memory, which can lead to slightly different results, with respect to sequential

computation.

We developed the auxiliary routines for parallel kernel computation, and for diagonal

matrices operations. Parallel kernel routine is derived by the distribution routine

PDMATDIS implemented in HPEC, which loads matrices from files and distributes to

processors, accordingly to the block scattered decomposition. It permits to appropriately










load the matrices A and B and to evaluate the elements of the kernel matrix needed by

each process.

Finally, the operation count of parallel ReGEC is exactly the same as the sequential

one. Thanks to computational characteristics of linear algebra kernels, the parallel

implementation of the algorithm described in Figure 5-2 has a computational complexity

on p nodes that is exactly 1/p of the sequential one, and a communication complexity

of one order magnitude less than computational one. This is usually a target in the

implementation of parallel linear algebra kernels, because it assures scalable implementations.

5.3.2 Computational Results

The dataset used in this study consists of the genomic sequences of Translation

Initiation Site (TIS), which is publicly available [71]. The prediction of TIS in a genomic

sequence is an important issue in biological research. This problem can be stated as a

classification problem and, although some techniques exist, there is a great potential

for the improvement of the accuracy and speed of these methods. Moreover, it provides

a significant case study for the analysis of genomic sequences. The aforementioned

method has been tested on benchmark data sets obtained from the TIS. Results regard

performance in terms of execution time and efficiency. Execution times and the other

accuracy results have been calculated using a Beowulf cluster of 16 Pentium 4 1.5 GHz,

with 512MB RAM, connected with a Fast Ethernet network. Each node runs a Linux

kernel 2.4.20, gcc compiler 2.96, mpich 1.2.5, BLACS 1.1, ScaLAPACK( 1.7, LAPACK( 3.0,

BLAS with ATLAS optimization. Tests have been performed on idle workstations, the

time refers to wall clock time of the slower executing node and it has been measured with

function MPLWTIME() provided by mpich. The maximum memory available on each

node led to the impossibility to run some test cases on a small number of processors.

The execution times and parallel efficiency are shown in Tables 5-5 and 5-6, using

different number of either training elements and CPU. Tests have been performed on










logical 2D nieshes of 1(1), 2(1 x 2), 4 (2 x 2), 8(2 x 4) and 16(4 x 4) processors. The

training sets have dimensions ranging between 500 and 9000 points.

Table 5-5. Execution times for the parallel intplenientation


8 16


500
1000
2000
3000
4000
5000
6000
7000
8000
9000


2.99 :3.59
21.90 17.79
162.12 89.79
5:32.42 260.39
1487.87 562.70
2887.51 1050.02
1921.1:3
3414.97


:3.07
12.29
55.95
14:3.9:3
290.02
265.92
812.64
1298.75
1875.02
27:33.95


:3.51
12.61
46.59
109.6:3
205.95
:342.22
52:3.99
75:3.6:3
1046.08
1421.28


4.00
12.4:3
40.54
87.30
155.39
247.36
:365.92
514.66
69:3.84
91:3.16


In table 5-6 the efficiency is calculated using the following formula:


(5-16)


aff


#cpu t


where t# is the execution time using # number of cpu. In all cases for which we could not

evaluate sequential or parallel execution time on a small number of nodes, we set efficiency

to 1 on the nmininiun number of processors on which we could run the application.

Table 5-6. Efficiency for the parallel intplenientation


1 2 4 8 16


500
1000
2000
3000
4000
5000
6000
7000
8000
9000


0.4175
0.6157
0.9027
1.022:3
1.3221
1.3750
1
1


0.2442
0.4458
0.7244
0.9248
1.2825
2.7146
1.1820
1.3147
1
1


0.1066
0.2172
0.4:349
0.6071
0.90:31
1.0547
0.9166
1.1:328
0.896;2
0.9618


0.0468
0.1102
0.2499
0.3812
0.5984
0.7296
0.656:3
0.8294
0.6756
0.7485


Results show that, for an increasing number of processors, the execution time

decreases proportionally, if the problem to be solved has sufficient computational










complexity. Moreover, time reduction increases for larger problems, with a consistent

gain in performance. We note that, in some cases efheciency is above 1, due to limited

nienory on each cluster node; nevertheless a sensible execution time reduction is obtained

when the number of processors increases. We can conclude that parallel ReGEC is efficient

and scalable on the target architecture.

5.4 An Incremental Classification Algorithm

Classification problems may involve a large number of training points. One ininediate

solution is to select a subset of points that would retain the characteristics of the training

set. A second problem arises when a new training data point becomes available for

training. A desirable method as a solution to the second problem should be based on an

efficient evaluation of how the new point may influence the classification function, rather

than a complete training of the incrementally augmented training set.

Datasets in almost every application area are ever growing and are continuously

updated. Moreover, numerous applications on massive datasets are emerging [44], which

require efficient computational procedures to respond to the dynamics of large databases.

As machine learning becomes a part of data intensive computation systems, updating

the learning system becomes intractable in many cases. Therefore, incremental methods

that require some nmininmal computational burden are strongly preferred. For this purpose

several methods, especially in the kernel-based nonlinear classification cases, have been

proposed to reduce the size of the training set, and thus, the related kernel [45-49]. All

of these methods show that a sensible data reduction is possible while maintaining a

comparable level of classification accuracy.

In this study, a new method that finds a small subset of the training dataset is

introduced. The amount of reduction in the training set can he as large as I' with

comparable classification accuracy and improved consistency with respect to the original

training set. The proposed subset selection method starts with an initial set of points

and incrementally expands this set by adding those points which contribute to improving










classification accuracy. The main idea is to use the small subset of points to solve the

general eigenvalue problem, and therefore the evaluation of the contributions for new

points is performed in conjunction with ReGEC. Thus, we refer to our method as Incre-

mental ReGEC' (I-ReGEC).

5.4.1 Incremental Subset Selection Algorithm

The dimension of generalized eigenvalue problem (5-13) is equal to n, the number of

points in the training set, plus 1. Since the computational complexity of the operation is

in the order of O((n + k)3), it is important to develop methods that are capable of finding

a small and robust set of points that retains the characteristics of the entire training

set and provides comparable accuracy results. A kernel built from a smaller subset is

computationally more efficient in predicting new points compared to kernels that use

the entire training set. Furthermore, a smaller set of points reduces the probability of

over-fitting. Finally, as new points become available, the cost of retraining the algorithm

decreases if the influence of the new points on the classification function is only evaluated

by the small subset, rather than the whole training set. The main idea is to exploit the

efficiency of solving a small eigenvalue problem. Therefore, we use ReGEC as the internal

method to evaluate the classification accuracy on the entire training set.

The algorithm takes an initial set of points Co and the entire training set C as input,

such that C > Co = Ao U Bo, and Ao and Bo are sets of points in Co that belong to

the two classes A and B. We refer to Co as the incremental subset. Let To = C \ Co be

the initial set of points that can he included in the incremental subset. ReGEC classifies

all of the points in the training set C using the kernel from Co. Let PA. and PB. he the

hyperplanes found by ReGEC, ro be the classification accuracy and Mo be the points that

are misclassified. Then, among the points in To n Mo the point that is farthest from its

respective hyperplane is selected, i.e.



x1 = xi : max disx P ssx)} (5-17)
xEfronivo}










where class(x) returns A or B depending on the class of x. This point is the candidate

point to be included in the incremental subset. This choice is based on the idea that a

point very far from its plane may be needed in the classification subset in order to improve

accuracy. We update the incremental set as C1 = Co U {xl}. Then, we classify the entire

training set C using the points in C1 to build the kernel. Let the classification accuracy

be rl. If rl > ro then we keep the new subset, otherwise we reject the new point, that is

C1 = Co. In both cases rl = To \ {xl}. The algorithm repeats until the condition |Tk| = 0

is reached at some iteration k. The algorithm can be summarized as follows:

Algorithm 1 I-ReGEC(Co, C)
1: To = C \ Co
2: {ro, Mo} = Classi fy(C, Co)
3: k~ = 1
4: while |Tk| > do
5: X, =x : minxe{Manr, } {dist(x, Peass,(x)) }
6: {rk, Mk) ClsitSyi C, (k-1 U {Xk ))
7: if rk > k-1 then
8: Ck -_1 U {Xk
9: k -1 X
10: k=k + 1
1:end if
12: end while





In Figure 5-4 a graphical example of this approach is shown. The classification

surfaces of the two classes (dark and white), generated using 400 training points of

the Banana dataset [69], clearly define the aim of our strategy. Indeed, when the

ReGEC algorithm is trained on all of the training points the classification boundaries

are significantly affected by noisy points (left). On the other hand, I-ReGEC method

achieves clearly defined boundaries (right). Furthermore, the number of points needed in

the example to generate the classification hyperplane are only 23 in I-ReGEC compared to

400 points in ReGEC.