<%BANNER%>

Implementation patterns for parallel program and a case study

University of Florida Institutional Repository
xml version 1.0 encoding UTF-8
REPORT xmlns http:www.fcla.edudlsmddaitss xmlns:xsi http:www.w3.org2001XMLSchema-instance xsi:schemaLocation http:www.fcla.edudlsmddaitssdaitssReport.xsd
INGEST IEID E20110109_AAAAWQ INGEST_TIME 2011-01-09T14:13:37Z PACKAGE UFE0000552_00001
AGREEMENT_INFO ACCOUNT UF PROJECT UFDC
FILES
FILE SIZE 50233 DFID F20110109_AABHTI ORIGIN DEPOSITOR PATH kim_e_Page_059.pro GLOBAL false PRESERVATION BIT MESSAGE_DIGEST ALGORITHM MD5
9566f4cdd821cb91ee9a7595fb90586f
SHA-1
afec5879632665c3f7d63db2c3afb81740ea66f2
31914 F20110109_AABHSU kim_e_Page_039.pro
2cc0670efd50952cb40959c683ee532c
2998e99c281c224f6d57049faf848ad09b7dbc87
33585 F20110109_AABHTJ kim_e_Page_062.pro
2a6c69ec413de16f0e945ecd10446df1
c887de80dfad31ad53940f8a7eaeb9dd79adc0cf
44424 F20110109_AABHSV kim_e_Page_040.pro
8bd161c69823e13d42ef0d9ea4f40c96
0025c0903db363ded549c4600a33b0422c0e87be
21219 F20110109_AABHTK kim_e_Page_063.pro
12512548b7337c4a2baf12160397820a
2f8d3ec57b9e0f5b70175003ad6ac8fae4b5c2fd
21031 F20110109_AABHSW kim_e_Page_041.pro
600e99b75136b772dab6cae8efde5bbc
f6bb2b21907a6ff5b9afa205e048a65ea8599c59
20910 F20110109_AABHTL kim_e_Page_064.pro
b709942d6f9b9ad86bb8644b2b9b3062
cc8212b2f8d332f47088881b2778546a0da42987
39672 F20110109_AABHSX kim_e_Page_043.pro
80a994cd5d5dfc006ee0378c5df3b864
5280cbc27137c5f0d090003da8ca528c34c900b1
33946 F20110109_AABHSY kim_e_Page_044.pro
f36baa41cedcacc2cf381d0a6123fff4
6de0d393934bb5dd15a0a39054650bcc8dcb7429
46831 F20110109_AABHUA kim_e_Page_081.pro
b77fe5be2f08abf0b573ee916377a5ac
d03b95db2b62f212e8216c7ac6068b86671028b2
25818 F20110109_AABHTM kim_e_Page_065.pro
9a73a83c0eaf287d18d11d1378dcf9a1
f1b225b1f89296362122e5486c8ed0dfdd12b3c5
34114 F20110109_AABHSZ kim_e_Page_047.pro
f5e113be799b64fbd20f1b3a16665d97
9c256b8f629942b8f0562cd6f2fd5901e014589f
48926 F20110109_AABHUB kim_e_Page_082.pro
6760d6afdd9456af88a421fa75c54c88
9a45b6e4d4c1487d4830d03e21be9722f3602ff9
45146 F20110109_AABHTN kim_e_Page_066.pro
0a1f4c157dd822da48ba38d58175b1c8
4908ba502184a1fe77fca78022e61cb1b2c49c7f
37216 F20110109_AABHUC kim_e_Page_083.pro
a2e2726212ff488d0c318b697d687e77
a5f2d8ca4260df91e4091ccb4d94d6e834dda0be
19430 F20110109_AABHTO kim_e_Page_068.pro
a8ca107bcffd4debbc3c7aa1f18173af
6ab4ef96f4cc3f285724708aba412913355d5ef4
12753 F20110109_AABHUD kim_e_Page_084.pro
6bc4bb20b4ca80b7a26557ae1d389958
4ff261d83165a4df588787ee0938260d1aba98c8
33049 F20110109_AABHTP kim_e_Page_069.pro
c26fb9a39cd0ccbeacba7710d0cccddf
0f47a1e48309633ed6d17c278c169dc05c7f13ee
39117 F20110109_AABHUE kim_e_Page_085.pro
bead1f0a91740a7c13184388e2d6c7b7
21cf9c345b5e2f8a3c52a0d4e403e87f35b49cdb
28121 F20110109_AABHTQ kim_e_Page_070.pro
977a1916ef43c10e5d6f995262913331
89c264aa7a05aa7c7be3607d53ca0c7e7c4d0fc4
97175 F20110109_AABIAA kim_e_Page_005.QC.jpg
febea239178971129ffe94afa0a90675
7adab5b7dece19acb89c6f3e200559319173fcea
53911 F20110109_AABHUF kim_e_Page_086.pro
e0c4b63cd47b7ab8daac9c1645188d59
de5a02edcd77cf0b58bcf64cb2180ec4ae84d33f
25827 F20110109_AABHTR kim_e_Page_071.pro
95f4a5d2beea119d5a7f9d0c50995ae4
157f841c2d1787a1686f1685157c7e2d066ab5b8
47819 F20110109_AABIAB kim_e_Page_005thm.jpg
b5ce186a362d9013e401e4c30701a1d2
d78b24557518d341fda8847c5f8aee42d265afa2
27565 F20110109_AABHUG kim_e_Page_087.pro
ddc0a61397b2ef7dc33d81c23c801b8f
444d533f60e5ed6dfccb708fa663aa84caf5c90b
36416 F20110109_AABHTS kim_e_Page_072.pro
01335f29d2074e84fd342ed3f535491f
ecd99764bd2936b2df34be07f236c0fabbef5ce0
122985 F20110109_AABIAC kim_e_Page_006.QC.jpg
4e747e9834f342a0b9c81e0ebb41ed25
57cc4c5faf86e9196dab49bbbc3bf45f2740b24e
12151 F20110109_AABHUH kim_e_Page_088.pro
f24a59cc98abf684985d472d845c689f
abc80b016ee2e7a52a409289eb28024a85325d64
44073 F20110109_AABHTT kim_e_Page_073.pro
0a65773b9a2457f37b542b2ca61885ee
c4d30d6bc57b4adbcf19279e7a1b7054e2f2358b
54178 F20110109_AABIAD kim_e_Page_006thm.jpg
273cefa69847148155c9292189c22870
abe14af7895a297422be211159bd55563d9def83
45067 F20110109_AABHUI kim_e_Page_090.pro
f98436cf2a5a708e066342330232104e
38425ede9882ea13fe7fcd68250e502db0e0222e
40652 F20110109_AABHTU kim_e_Page_074.pro
b1c42706c282ae4491e7242e15ab2c17
470127a795d6547c9d0470ef205c69108839dce0
114963 F20110109_AABIAE kim_e_Page_007.QC.jpg
b0dfb2998636a9804136f42e26739e05
13249d85084a54e27c7cc809bf337562c1228c16
48464 F20110109_AABHUJ kim_e_Page_091.pro
699f6cee2ea23b36cd55638109a2e5ca
fa6c7fb5d54868a292224b14f6430c0d91f100cd
35242 F20110109_AABHTV kim_e_Page_075.pro
b37bc0856d1c1d9a23e38bf056617618
33f6d65667977e4195aca7b987b69a2dd217ce89
54802 F20110109_AABIAF kim_e_Page_007thm.jpg
9b4e53f848edff28e9ad9f5f7cf6e23f
79b4d172eb942659c27f4a3c6663c318cb394f6a
38355 F20110109_AABHUK kim_e_Page_094.pro
bfe5b87492c1284b7f8352b237274836
af8bb59f9eac723b9d9f1a9c10854e52c7d623b7
9810 F20110109_AABHTW kim_e_Page_076.pro
82c6e5a4d7b3e17c9ca2d7a7e0b67389
92b05b6dd2b84505a57f1022a0a4837ff477a930
57474 F20110109_AABIAG kim_e_Page_008.QC.jpg
2997cc26e268b7a23f924962d98c2fbd
f6824c11b6588944b04d0a6cc601db92e98a2310
36732 F20110109_AABHUL kim_e_Page_095.pro
6b9e3b5f7c8e9e45fe74ab1670fa7ce5
322ddc40ffcb729c7dc247672591d608099f2e1b
43374 F20110109_AABHTX kim_e_Page_077.pro
c15a7498ce0ba67a3e97cd8b140d2e37
5f7ac185106da08f3f4f5e0006fd7a6915fda24c
47513 F20110109_AABIAH kim_e_Page_009.QC.jpg
053d5e0dfa37fa15351850898191360c
602be154f6ad744bfdaac31058bbc2c4e5f0492a
23528 F20110109_AABHVA kim_e_Page_115.pro
b66299802f6c4f0cc69c52455b1433f3
46cdf7e85a25faec16485ce385229bfe9879b1d5
39104 F20110109_AABHUM kim_e_Page_096.pro
db9ebb0bd1801e7ee55a520ce451224b
c53692a447a502918f094fa7c48e5a04e5b09ef9
40637 F20110109_AABHTY kim_e_Page_078.pro
b27b40c38aa62d3aa79bc5810d0772f6
d5f58e216ebd0a0dcc6edd448ef257bfff301583
22130 F20110109_AABHVB kim_e_Page_117.pro
961dee47f1d7dbaa1f6c1b3f883bc522
3f1df497c1d3a69cbf6102d32dec25258e9ff453
51374 F20110109_AABHUN kim_e_Page_097.pro
b7d442d197e5daab3ccdebd62f5d2a0b
9b58ea5f51b2a7b92251d1c1e5407f91be6de096
50087 F20110109_AABHTZ kim_e_Page_080.pro
fa4d33b0fd123d74bdf077afc88c360c
c541a6f6c682da98e7106046a0d59f3ef9a8957d
58790 F20110109_AABIAI kim_e_Page_010.QC.jpg
94c1435d37d29a51321f90ac862b2c5c
5c039908368e52e03e0ff923940936ca53292766
31969 F20110109_AABHVC kim_e_Page_119.pro
f05927620b87f7a225c34a39525f0cf3
299caebea2e2cb40b9dd5716c041c40bffc0d3f2
44742 F20110109_AABHUO kim_e_Page_098.pro
9a0f8bac45c074eeab0f91e4e66ac4da
c7ef305fc1865a9d80abcdb1496909dca5974692
28673 F20110109_AABHVD kim_e_Page_120.pro
ce64de92fd01cd7357d1972d5620d35a
ce99011ee734bed955b06f6b52184102bad86738
32358 F20110109_AABHUP kim_e_Page_099.pro
9e69f33269fab09ecb56b4229d1198e0
9be9c3ea4adc2aef0f12670d7486a72c2ddca205
61693 F20110109_AABIAJ kim_e_Page_012.QC.jpg
f8b8db98cc5dab03274f6c3c7d23bf9c
32b87d891595dd3aa8ce0377c93f60002d4d65e9
36373 F20110109_AABHVE kim_e_Page_121.pro
982b80821b57f1dee406453e4ab9ffbb
4dec07a4233ec000dccbfb4a234ee2407ed4bbc3
13964 F20110109_AABHUQ kim_e_Page_100.pro
dbfd7f6f1712f3de4ca811b89229f924
e292cf6c5616fd6afd7808bda6ad5cbd8f275356
76327 F20110109_AABIAK kim_e_Page_013.QC.jpg
91c89333256662b15f04dc92bb5c83ea
e298d556997b1bd3949548196202122828ecd701
31772 F20110109_AABHVF kim_e_Page_122.pro
ee1017637201af7d82758d55047cefaf
2b8f87c0767edcbcb8ec948b9fb938b40fd47810
26470 F20110109_AABHUR kim_e_Page_101.pro
d52d7522ce8ab5407a535c9f3e0d606d
6a7ba9dcea107af5483386fee199f4e0fe8a3f3d
66000 F20110109_AABIBA kim_e_Page_022.QC.jpg
b760fef89bd83c89e4ec7c1d54b9fd30
6facbf6051ba3de1af0b85ffd596d750fa32b90c
24288 F20110109_AABIAL kim_e_Page_013thm.jpg
9dcd401660fd3b0c6f704d37c9edea9f
c1e886d06f12fbcf1ca325bceaa2c132b75d6185
15270 F20110109_AABHVG kim_e_Page_123.pro
076fc27cc85f243db8cae8f2ae8bc856
bcfa55d061dbfbf7ac0a446aa7cfafb9901d59da
25540 F20110109_AABHUS kim_e_Page_102.pro
1472a54150ad8e02dc2805c3863ad5cd
fbd9dfaaf2197134161ad781f62292ed895eeb51
63086 F20110109_AABIBB kim_e_Page_023.QC.jpg
011d650f811b517c245e4c1e7eaf9c3f
c6b35923129b11c948f3aee38584623cacb632c8
35768 F20110109_AABIAM kim_e_Page_014.QC.jpg
4f0db9ec9ab560aa4f85f64831915c73
2e9337bd1a0502003ea3cbb14dd3a6513eabd469
2972 F20110109_AABHVH kim_e_Page_124.pro
3ce6f3e6b8c99679e024b83c4674099a
9ca6729220d10fa9f56073c2ca9fbd0ca0624f63
40535 F20110109_AABHUT kim_e_Page_103.pro
985894dd5f84a12db69498d84f3d616b
19c2a3949551d476cfb67699a04769ad911f2e6a
54349 F20110109_AABIBC kim_e_Page_024.QC.jpg
b967ddecdaca580a4c27d7c2afdd42ca
fa788a28c531b4f0b597c966a12ab945f77bca9c
12253 F20110109_AABIAN kim_e_Page_014thm.jpg
d1c8011ad0170c518a52006d3594e7eb
ebc1a9095d7800c832519380f186d105e6ed24be
12587 F20110109_AABHVI kim_e_Page_125.pro
e83d18c095e56ac7439e2091348ea186
45fe9fd254dbdf09840399e8db19a9cd40fab24f
18307 F20110109_AABHUU kim_e_Page_106.pro
9d3991090b2108596747d028238e2911
16d6816a80a85cd4416ba10dd21e2ed7091969c4
18925 F20110109_AABIBD kim_e_Page_024thm.jpg
b773e51f448fa1854acb9288a66f85e6
da657ffc9c5f174130cb0bfd513ef17fdba4b81a
56052 F20110109_AABIAO kim_e_Page_015.QC.jpg
f896dcca86daf0481792cf4e4d98e99a
3153a4f5382c5c1fbffdf5e479aa74aca418ce2c
20216 F20110109_AABHVJ kim_e_Page_126.pro
ed710c1c81d0d44074c2e78e9553c4a8
1ed16cd03e7b75604fbd636a92df2001a9af791d
27276 F20110109_AABHUV kim_e_Page_107.pro
3208b76cef4b813d16d4631a6a815538
60899f35a9886909927ed9c3516cd3dec10bbcd6
38601 F20110109_AABIBE kim_e_Page_025.QC.jpg
ab783c7a52fe4fde0ff7a3de7576dbed
66c7510b41ca1212618384f88d24a0e896c9e320
19151 F20110109_AABIAP kim_e_Page_015thm.jpg
157fd910c7147a6f5dd7b3170d9fd2c8
954be1d12e526d08c0892093e33e044afa579b94
27692 F20110109_AABHVK kim_e_Page_127.pro
58164d6b16425a271e8582c1c449d3d0
75e5e38ed203026a6e60933b878d9b606c5aefee
25822 F20110109_AABHUW kim_e_Page_108.pro
e908efc31b3d3d7fcb5dc5dbee0edb28
cd3ac8cc6ff1062e3d30e13b5aafae0341717fac
12784 F20110109_AABIBF kim_e_Page_025thm.jpg
c3100d01535421f7f3f0dad9e2c185ef
6df673ab84a249e6f4ef62dec7ef46eee3f53340
61648 F20110109_AABIAQ kim_e_Page_016.QC.jpg
12c9e96780f8d6cd4277561b3f8187cd
e332bed45a0407890c6098fc8beb4e51afd87bc7
22358 F20110109_AABHVL kim_e_Page_128.pro
96fe315e24a5f6c9c57d198c5e6d6645
1c28eb703d6eb3bf2b01f1960921877596708750
23236 F20110109_AABHUX kim_e_Page_110.pro
8ecfceb1f67954460a85103bcfc39997
8b8fa24100d213d79bcc5ac07592570191a1d0d1
18350 F20110109_AABIBG kim_e_Page_026thm.jpg
ecb4bd0fd0893a6df661c88228026e23
53534e7f92d35f7a29ec312ce45486de4c61a28c
21424 F20110109_AABIAR kim_e_Page_016thm.jpg
5dd32e612fd3df113bf89117d7284647
9635d763d0719c9fb41219eafd223ce3372535cc
22701 F20110109_AABHVM kim_e_Page_129.pro
46bb86a903e28c5a6cf4a4e824bf9d1d
1ffd4996612191a758873b0a090b5e1e0872af53
19417 F20110109_AABHUY kim_e_Page_111.pro
6ab17bebb3c3fb6483033ccb94345048
994dd3047b05aca08fd14f3b09f6183eca47273a
61383 F20110109_AABIBH kim_e_Page_027.QC.jpg
8ad3cada62fcba132bd54a4af1a09f48
5d9c8556b10efacd12cd78d4bbe222d8603f65f3
1699 F20110109_AABHWA kim_e_Page_012.txt
4b6ebf8cec4f498240649e50bf4dba91
7e5497b6d6b7f5309e075666b08399ef3ea14fde
67249 F20110109_AABIAS kim_e_Page_017.QC.jpg
4a9344b666fd44bb8b6dc92b2598b37e
f9d6858794b2f96b2080ba5e3fe0757ffed9dbce
22670 F20110109_AABHVN kim_e_Page_130.pro
d30ca26dd31811ae8b06362f0363feea
acafbaacdf398bf34c45c6a344f500b836c98c18
10927 F20110109_AABHUZ kim_e_Page_112.pro
bdb06c8149d4306fd8e2e9e7b45de738
88b91cfe870c4f41dc71d609d1ccca53b7e6312e
19992 F20110109_AABIBI kim_e_Page_027thm.jpg
dd5c93d2c92709850d380656dfa2179e
cbaed9a4afbc9e6ef3622605b002ff6bb0bc4261
1955 F20110109_AABHWB kim_e_Page_013.txt
f53bf5bdc85a93c81ce53ddcfdc2042d
c7e58da8eeec177c610c41e820e8f0e0d62c4eaf
22958 F20110109_AABIAT kim_e_Page_017thm.jpg
4e41d5714620315d3f4496085552c692
b191265086865768f9f83af1361bd833676feab8
22866 F20110109_AABHVO kim_e_Page_131.pro
544975b6185d87fface15e806e225b9f
2173f9093521266f4d7919ab0c28857e7e52af5f
895 F20110109_AABHWC kim_e_Page_014.txt
704343567a759702810f13cefbf5fc00
0b0f17d902caf63115ccbf45495b37fbf493c187
21633 F20110109_AABIAU kim_e_Page_018thm.jpg
27e6f318e8a7ce6a5cedcbf44549336b
5e5aadae63d2e077d3da93497a98bad6697094f2
61387 F20110109_AABIBJ kim_e_Page_028.QC.jpg
589463f3fac0d2ae40c333c3c15f7214
2cf9502f5decca8cf265f2175e9f8d032eca7bf5
1550 F20110109_AABHVP kim_e_Page_133.pro
9a50edde37a9a9e86199c9b91bf624e3
6a11db978a77a3cba0ca8e6e071457b190d83dbf
1539 F20110109_AABHWD kim_e_Page_015.txt
83dfb8cc7d93af61ba10fd7b34478d33
086a84b37bbc9a9d88815dda2266e65ef619bfec
69488 F20110109_AABIAV kim_e_Page_019.QC.jpg
b349e55e65e5ac34d028796021787859
0891fcfb6fb743b2f620e388b95f3372011ff4ef
19500 F20110109_AABIBK kim_e_Page_028thm.jpg
ee030d483c24256d41f433dd262e6bca
bc92ce18ef01eea07ac60cb723973c07a37ce840
50130 F20110109_AABHVQ kim_e_Page_134.pro
362e4ac8ab887142e3b544e8b4493fc4
571134405d6c8665bfa3ee40107772cac7803248
1569 F20110109_AABHWE kim_e_Page_016.txt
55f6df29cd900b4fb8a00b0f459bcc22
926ab98745e6e13496eb72aee24dbb6f6530ad0b
75721 F20110109_AABIAW kim_e_Page_020.QC.jpg
1cb1e9aafa576c96fbc64929105c2f31
0bf15309ae19604adad590403ced41cc92f04661
22555 F20110109_AABICA kim_e_Page_036thm.jpg
c7e392d3ad7c090a907e06ddf6e2859d
86d3a9c0548f6559a4874e24ee7950580299925a
63341 F20110109_AABIBL kim_e_Page_029.QC.jpg
f49f1457839d3fe9ae5aa8aa3505f32a
d1c51677862c64ad84981989c6f183349a04f027
49805 F20110109_AABHVR kim_e_Page_135.pro
3133554bda8d99e9ac5aa3fd4d8cd45e
59b44cdfb43a6526dac28ca7dfebbe0d5e893c5b
1741 F20110109_AABHWF kim_e_Page_017.txt
be4d7931c41f09d611bbadbbac393173
64992bf2f17f7ad7ae4a975ab04c7148e1ce8176
44566 F20110109_AABIAX kim_e_Page_020thm.jpg
3c930c61f23ec77b983a3f6a59241132
89dca95ba9d78970d7f178c2e67ab8901f7ce546
55467 F20110109_AABICB kim_e_Page_037.QC.jpg
ae3cbcec35df9b03cfc8f1c75a5978af
d31a97f30c55b82c6c10db999df795fce1ee3613
21489 F20110109_AABIBM kim_e_Page_029thm.jpg
69cf7678f2d9b8acbfed90012b6ac404
4cd357da4a472003974f6d73e84093ee7c83cb5c
29194 F20110109_AABHVS kim_e_Page_136.pro
d3f4e11d7ee3d1d8be16812658ec927c
956369f0612bb8c2bfcac4bfe8a4854c795c19ec
1935 F20110109_AABHWG kim_e_Page_020.txt
dd5b77983193d51a4d9ca7ca8bd98440
370af0a397533e55073e2b41e7b63cfbc1bb89b4
62952 F20110109_AABIAY kim_e_Page_021.QC.jpg
d98bdf5cb2a52a658629d08195bbf665
edd3012992bdbbaa8d64da46e98ae0c1859761a8
59655 F20110109_AABICC kim_e_Page_038.QC.jpg
ca99319bdde5415f39289bcfae1fe193
0d6f742384dd6b37f8d78bcb7be64f2af6b73cfc
74514 F20110109_AABIBN kim_e_Page_030.QC.jpg
fa9c9a5737761e297482d6b9d2e0e327
14c2a0b69051b8f8117c31f44266ca4b10e5f097
446 F20110109_AABHVT kim_e_Page_001.txt
75734497746c5cc9f39fd550c8d9af41
aa942cca1c07b181dd22a1c8cb419b49620c986d
1712 F20110109_AABHWH kim_e_Page_021.txt
e4783c2e301108a97fe33f67f8f13042
15ff078c37b54ef4a8ce95a8000cd167db7b1577
21644 F20110109_AABIAZ kim_e_Page_021thm.jpg
a8000a6616f9fd85ea724dad1be24b5c
522460af13d1ce97404b04c97932dfeec3f72690
21425 F20110109_AABICD kim_e_Page_038thm.jpg
7cc8bdbd2d9014e0de49974302976cfe
4e7522fe7b9f883059517685ed2861c9c063d5a5
24574 F20110109_AABIBO kim_e_Page_030thm.jpg
9c0a6e6a0e418bd4111b234ffb840408
579d44b376173b6019497f8486e67a6323ae6300
101 F20110109_AABHVU kim_e_Page_002.txt
f0526bb82e1fac5462c17eb9a1e87276
48ce3a2ff3ab5b0d70132842429e202fb87d37b0
1710 F20110109_AABHWI kim_e_Page_022.txt
c397f81b036501386a92e4c4773bb22b
c2a5055b3b35837924ab3e64d51bf4d704039f1a
17145 F20110109_AABICE kim_e_Page_039thm.jpg
fc5bbb5a3ca1f7f1d1b4051c43b181e7
0e3b535bcfa2e034e24d4d80f239ab265778b315
57795 F20110109_AABIBP kim_e_Page_031.QC.jpg
d894c01a4b0213f722805e4538f8f772
c1564579c921af59e1fb525984efcee34e1b1a88
620 F20110109_AABHVV kim_e_Page_004.txt
08a0cf8fc876763831206b4d0da388b2
cd92ca60549ad87d358343243ed3b1be55c7460a
1691 F20110109_AABHWJ kim_e_Page_023.txt
f02e78cb74bd3ccd36800543142151e8
850b25b8863e2930824c15d32465ec4643562f16
70446 F20110109_AABICF kim_e_Page_040.QC.jpg
1a4b10008e137163c9e23cf76e46443b
8ac90676b244ccb4a928525c7affee59af9947fc
19309 F20110109_AABIBQ kim_e_Page_031thm.jpg
ba0d2be5f0c6d042950fb5490b48ce50
f792ad1f1e019c644c0994bda955dd75da197c20
2994 F20110109_AABHVW kim_e_Page_005.txt
bf73013f1a8a3e8712ee1c8d277a91f7
5aee922b80bf56d54a1b777182705db9798b8827
926 F20110109_AABHWK kim_e_Page_025.txt
d2bd787089b7de434c015e0294722db4
15816c1b692c36857dd2cfeadb9b14a799a47c5c
23612 F20110109_AABICG kim_e_Page_040thm.jpg
e532932e9208ffbba1d2b661ccab0a5d
7ef66f12e51ee02ebb18c2ae1e15608f9c5e7b1d
48284 F20110109_AABIBR kim_e_Page_032.QC.jpg
4e92330299deff2ceee6ab7a9b1c4e3a
e23b5fae95134ac5c07f1c7cc28bb1c2c5e485db
4923 F20110109_AABHVX kim_e_Page_006.txt
265d8bd0d84b552dc460858c63c8de78
36e30a93ca2efb28bba3cfb3bab349d60ec15114
1406 F20110109_AABHWL kim_e_Page_026.txt
e0eb169809cf2c69ea32ae8b2dd848de
16777103f699fdaf43004b424b5e0f413fc86f35
37102 F20110109_AABICH kim_e_Page_041.QC.jpg
c465ab6d69271acea78d59e4862b71c8
a1bd4ef583b2ffe7398a1cdca9281fba6ca3aaeb
15857 F20110109_AABIBS kim_e_Page_032thm.jpg
dc179024ff04d18c11f7f416baa004c2
0eb5bb0fcbdbb864087880e0c1869eaf43fc80b8
712 F20110109_AABHVY kim_e_Page_008.txt
bf2297e4d4ae7430a45b02c29f25fe4e
04367ffb382ad4cfc5ea76201a34784ac166a57a
1702 F20110109_AABHXA kim_e_Page_043.txt
5e882297d381383a43ac921d1a934271
ebdecf5d23868bc11c3391d3cb57201c73c0bca3
1581 F20110109_AABHWM kim_e_Page_027.txt
392bae3df559056a54f540e9e74f770c
e6639f0d0d63937dd3f5aea0c7342a46da3adc71
12811 F20110109_AABICI kim_e_Page_041thm.jpg
f53a5c946983387f81a4c6a23dc6a133
498844f81ed009d26bfbfd2d0f069524679032a7
56024 F20110109_AABIBT kim_e_Page_033.QC.jpg
99669c692a7309d2b8ca050a8c4cc5f1
01428ec88653289a41341265593a1d086664be58
1257 F20110109_AABHVZ kim_e_Page_011.txt
cd94936b9e55a26bd145cfc4f033c796
1019c5af964e807f2b5c190a281deb75d06a9997
1176 F20110109_AABHXB kim_e_Page_045.txt
93ace3805e53fad410bb6f529eda13af
a96557c25f608a41e671cf35812cfe0692f34080
1812 F20110109_AABHWN kim_e_Page_028.txt
e78f3fff507b973a5bcb1d7db4d67043
0a1f365d25d2c47e3f0f0575a56f20a8035bc2fb
37533 F20110109_AABICJ kim_e_Page_042.QC.jpg
1d0642f5fc7f084d30767c81f0eedc90
5805450cc0dd7ecd1863042ee6b2903adad9f4aa
18037 F20110109_AABIBU kim_e_Page_033thm.jpg
33dc224ee995abb22be256d96500fe62
8d3a8eae99492ddfac290452e3e9676af7c41cce
1767 F20110109_AABHXC kim_e_Page_047.txt
c2b3aeb015d70040f776d96287de2174
d59ab7297147036451898cef6b0eea7176689fa4
1626 F20110109_AABHWO kim_e_Page_029.txt
6445152042dedee4d63fa1e1629e9b04
79a837a8c32935ef7a51b41f8ba7d5fa9466c143
62849 F20110109_AABIBV kim_e_Page_034.QC.jpg
4a59ab3f0ba4c33139c577ec8ba5832a
0392410e2d580a8b9d4d2097565cbb3d3947f88a
1070 F20110109_AABHXD kim_e_Page_049.txt
a38384e76d9723ca306e500de0518441
2ba30a8f45839826913465430eca0ae55834bc9b
1848 F20110109_AABHWP kim_e_Page_030.txt
c9b8349356e0470201ecb09a8e5d0af3
aa9ff3c859dd28f1ec983678234544ab740c81a3
12883 F20110109_AABICK kim_e_Page_042thm.jpg
1e4e5eb9373b1e17eedd998799788a43
028b280a41515424e439ca6d7e451af08f342fdb
22046 F20110109_AABIBW kim_e_Page_034thm.jpg
f0d49ae18bb7f2faf4bd21715c553706
30b9eed933adeb251c6121f90da9affbaa00b77a
1191 F20110109_AABHXE kim_e_Page_051.txt
7d61c3a5ce754f3f7e763ae8d0b1c9ab
92b0c1229cea18f23250d010ace996b3c19611d2
F20110109_AABHWQ kim_e_Page_031.txt
ec452de12e7ff0162d7fa4c501972171
a814e7e25a9fa9441a3dcaad008bdee9028f44ce
52553 F20110109_AABIDA kim_e_Page_052.QC.jpg
deee21512b87608fb71c5378ca65c9ba
5ae5c0989e6ff4f095a93a4b4c0b832cb917f07c
54449 F20110109_AABICL kim_e_Page_043.QC.jpg
db63229bb0c69c6d5a06a3d14c0f024f
b0ae011bc2378ace5e45b258a8de54ae24c772aa
62383 F20110109_AABIBX kim_e_Page_035.QC.jpg
8335e1b7d7a2150d87ca3926546f59f4
0e98b910d56f513c17fcfb3270fd96c64177a3f2
1414 F20110109_AABHXF kim_e_Page_052.txt
d1236156542d141d3d27fb0023efb04e
a0ea8fbfef46761bf20687022ff15e1bf29e459e
1196 F20110109_AABHWR kim_e_Page_032.txt
8488751f9c613bd1f4d3a82711eff1e1
adc3c3434e44db49826c2dea791c0d3a0d9d2c6a
17850 F20110109_AABIDB kim_e_Page_052thm.jpg
0d25b058a4606bc6cd88106beda86c13
fdd1318c8bf69b966097639e38c15757b6689c79
17793 F20110109_AABICM kim_e_Page_043thm.jpg
16c0b3ce69530d82388d899611ad6908
817e93d26b10681f9f6029be903542665a79da13
20010 F20110109_AABIBY kim_e_Page_035thm.jpg
39c7a41cb3e323d0e42e32fbfd4545c5
9e3abdfea986731c0377368b270be38ea5a24a04
1975 F20110109_AABHXG kim_e_Page_053.txt
00e25563f7875e43809d85b53d48c5ea
ead7d0b5d4fc5592201913d5f2771104cad0b465
1579 F20110109_AABHWS kim_e_Page_033.txt
940ab5f2fee45bd30814b3c3a98416d6
a0b4d45f22e39d59de2f71da6a1e6bbbd8e1388c
60927 F20110109_AABIDC kim_e_Page_053.QC.jpg
d4f6f444fe19b16f84dbe82e92202334
57307c9200e5693a9217d428c53febb5aac16918
14503 F20110109_AABICN kim_e_Page_044thm.jpg
7aa8d465b084195f174c7d15b73b0d7b
4c53a777dd19e9d2f62d7d36bf006422e0810ed4
67058 F20110109_AABIBZ kim_e_Page_036.QC.jpg
d50cd61f968ddf51a814f4b06256c457
834d342dd8cde807dc28894b7e7c2c923d95071e
2099 F20110109_AABHXH kim_e_Page_054.txt
e11a608a7640997aa7cda500cc28fa47
c7d741c51b1ba627d4d86787d064f4ddf8876c7e
1631 F20110109_AABHWT kim_e_Page_034.txt
66a1ec45b37d8d15cbec6194905b685b
b2687fd9154d883bec08e932d8f7be857b42a738
253 F20110109_AABHAA kim_e_Page_104.txt
bed79f006f95c85cd91591d7df8f3538
42cfcccdfc72f40f89f40bfe321ff1d27c2cfd32
19254 F20110109_AABIDD kim_e_Page_053thm.jpg
59a1377f6022ef3211dca3e972535df3
10ad0c8f5174b2ee99d52776fe328e5edd12f232
46710 F20110109_AABICO kim_e_Page_045.QC.jpg
e2b0461e5b981f910254d9daf5a3c7fa
8752c9e31345af690388a8d6fa9a56f03c7fe429
1921 F20110109_AABHXI kim_e_Page_056.txt
d4f5f23f47445eecea289b2bf879d5f6
3c481fb0d1f6d3584d71f647a0ed388214ea8640
1575 F20110109_AABHWU kim_e_Page_035.txt
adf64eec332b6225c8d905cc75de3b20
d4e691314e4eafeddd3fb85b532b852f642ea4c3
57633 F20110109_AABHAB kim_e_Page_083.QC.jpg
62f4b7abbdbed5b1e89cc51d72f463d9
df91583bdbee83c37e684470481984d05f2a20b0
59733 F20110109_AABIDE kim_e_Page_054.QC.jpg
9f4deaa88884a2ff3bd73426a74a7aad
2a7b57fbb3af1f0fb4f3755ba695abda72747f69
15284 F20110109_AABICP kim_e_Page_045thm.jpg
71417b9fc46b92107c27c4db85530d9a
fb7d7741ce420f62df25e7efbf8775420ea13301
1372 F20110109_AABHXJ kim_e_Page_058.txt
cfa869b89fc856fefbe88d5c6620b466
c982a4fa04de6b8b7206246b3b9b735d1c65e47d
1818 F20110109_AABHWV kim_e_Page_036.txt
971c276a81d7cc7bac7cadc3685d7278
4d2549a200297d86c25b1526fe96731713932c3d
1486 F20110109_AABHAC kim_e_Page_057.txt
d65cb297d072ed22125508aa9daf5944
0f7769cfb049522abc7940f3968754f484cd106d
19816 F20110109_AABIDF kim_e_Page_054thm.jpg
1831d597de0ed49a7d8526202c49feec
ab567412a05bfebf578add696eb945305b422988
50326 F20110109_AABICQ kim_e_Page_046.QC.jpg
9e729a970942752be988c2b07c19cffd
e2afaf73911595c9f05b7cb3c728e60877dc1fa8
2028 F20110109_AABHXK kim_e_Page_059.txt
7c1554a4f76e355cf2167aad018c1370
d5bd5a39d0fe1f9897de94fc0edb688e5eb1123b
2176 F20110109_AABHWW kim_e_Page_038.txt
c68594ccc48825a0daed6f08df00b55b
83292b4fc808e5fd5d7d3917074f773636c15087
27022 F20110109_AABHAD kim_e_Page_116.jp2
cfd0db61ff07e5e053b3e2560d93c08b
de38f8d0a06c16b8d5893a467b06c30580969387
63181 F20110109_AABIDG kim_e_Page_055.QC.jpg
a3fcad41b8ea0eb9c0b13c4c964341bd
2f285f43ad7338b5d82463fd2906602115c8fdd1
15600 F20110109_AABICR kim_e_Page_046thm.jpg
7fffcbde13e6ad0e2ccc5a21fbf69585
bed0d2606ec3d1a9abea207ea418d39681fd28ee
1950 F20110109_AABHYA kim_e_Page_082.txt
2544e974fa9b63f19f0add98386a500e
385e7e1acb57e4e23205389c77159689839e2291
1629 F20110109_AABHXL kim_e_Page_060.txt
80ba9b442489f66cb544c2180fb28f10
8c2bca83afe19814a0293db20fd51c40683a1437
1770 F20110109_AABHWX kim_e_Page_040.txt
f2b1dae1f4a981957a532fb32f621535
20c9e20533eabbb92a8412fcc0e2aded1a498a56
1859 F20110109_AABHAE kim_e_Page_019.txt
24c1c2edecfe88bd027322391706fc54
6f6a8adbbc13454a97af2d0e702d6643b1b7a296
20977 F20110109_AABIDH kim_e_Page_055thm.jpg
bf87a36b8a8f51488c1b9186ac43206b
2bbc7514ab9f9c7b46a5c833987d3c0471c9c0ab
48914 F20110109_AABICS kim_e_Page_047.QC.jpg
6901694f50e4d11183704a41286c7700
727a489d397104668f5e08802acd7ba53cfa46b7
1082 F20110109_AABHXM kim_e_Page_061.txt
b2230f52395df204b2c248bcca7080a1
01bc3cf0ccbb63c0d562d33e77f31be8b5cdf0eb
909 F20110109_AABHWY kim_e_Page_041.txt
627d1a3c64df3a8711c6b1211feee7ad
4e492f77845f2d09983ecae00f4959a8532f2b3e
10790 F20110109_AABHAF kim_e_Page_123thm.jpg
52d6d5c7bba97f3420999751b963f4e8
96e72f0f462e50fdca0acfbbf915d684ec537b5b
F20110109_AABIDI kim_e_Page_056.QC.jpg
cd182a10d5985ce5f53d917ed41689d4
277fe5e19aa16860657ae6a5efd262ceb246a2f7
16188 F20110109_AABICT kim_e_Page_047thm.jpg
ad2fb7fbde0e0b084890dcddc2f30236
d2447513b6aab3d82b7fef7a892a5fc628c933b5
1611 F20110109_AABHYB kim_e_Page_083.txt
cc60d93e985d8377386f6520713402da
dafee6d990de3f92e97ab7f2057d1539c1e3992e
1370 F20110109_AABHXN kim_e_Page_062.txt
e82fc67cc6d735f0b74f6cf83929e9a4
33d7c22c7ee85b67efb53633650877aad91eaaa6
869 F20110109_AABHWZ kim_e_Page_042.txt
ff341299dc336fb54d0c096f4d84c58b
cf8b6ee039226ccc4e57ac1cb27436a560ce0ddd
52825 F20110109_AABHAG kim_e_Page_001.jpg
17328b513a804a83fb4c0482f9ddec69
16f060c448dbfdf2b736e00cc911a4ed3361f44a
19516 F20110109_AABIDJ kim_e_Page_056thm.jpg
1de82dc52ae69dd8fdc843922fb160c1
a923d67ff9420adf1376e568ded2e8c68bd54573
53948 F20110109_AABICU kim_e_Page_048.QC.jpg
23b1c2e32c7e3d436e0ad1d8dd41f1d5
fabb1124f9cabfb3b6cc975ccf73a8478792153e
591 F20110109_AABHYC kim_e_Page_084.txt
079c05b37f809227b16ad06da7ce7612
a4021898df73466806ba2ba771f23e7fbc30787c
1040 F20110109_AABHXO kim_e_Page_063.txt
7df48bfcf686552cbc04fee1db505503
fde1e6b20ea66f91002afb6e2de7abf5685b1290
1089068 F20110109_AABHAH kim_e_Page_050.tif
5f7b13a5bfd52e93b53cd2be899d0115
553a71b2956fb2ec4e988137d1d988f980814a27
16127 F20110109_AABIDK kim_e_Page_057thm.jpg
bbd3425ec899a09e5bcb7b7639f1ba9d
b4ba31b3ac7e9bcd4eb26696233af8b12e65c4cd
17910 F20110109_AABICV kim_e_Page_048thm.jpg
80fa01062bc651cda56cf762def08e1e
1e51444e65e21b22b68aee11ab4f07e93cc26c60
1658 F20110109_AABHYD kim_e_Page_085.txt
c30991b350e1b4f44d8982b696af3b63
45cd5d2e92d145c712b9b96c2c2b7505f6e09500
994 F20110109_AABHXP kim_e_Page_064.txt
94bda520dee14f7ed84c13d49a4d4ae0
1cee6f0e4113dcacfcf6b0d0904a125e771f7703
106 F20110109_AABHAI kim_e_Page_003.txt
e9a10be6a1c8d80e2924336f75006110
252d55c4bc882af74911787efca5f15733ca73dd
39178 F20110109_AABICW kim_e_Page_049.QC.jpg
fde7589840ec746af744dbe6ec28ad36
9bdffd5eabf448e1250f789a395857a5c14a102b
2526 F20110109_AABHYE kim_e_Page_086.txt
036cf298241a5d78f319985a498fb624
fc4a56fe2e44e4c08a749b02d76a74d25b311082
1786 F20110109_AABHXQ kim_e_Page_066.txt
233d90b79e73d42c0d171364943fe7f5
f7d7be376b2fe38cfb74cc2661a35e000a9d10d7
12554 F20110109_AABIEA kim_e_Page_068thm.jpg
425aeae8102364970c1d50d23b6052b5
9109e7a6afa85599ef6728a23b9638fe15e5a8f7
41252 F20110109_AABIDL kim_e_Page_058.QC.jpg
97674fc6ce19f1a937b85e388c1c4487
055b1ab0595cab864994c1424567cf31da1b0de7
13409 F20110109_AABICX kim_e_Page_049thm.jpg
7114b49b5c94c054c7a95f262bc4ca2b
fed3f3addcd2b770a4bb3668f9fbf4276026bf79
1495 F20110109_AABHYF kim_e_Page_087.txt
a187ba71ceaf590057a34fbd780a3052
8b70b924fb36da727088e77762a0796b873d9d69
1294 F20110109_AABHXR kim_e_Page_067.txt
1bfc7039e6ae01497c5d6652edcb8edb
69e6398c5f7a03e0b07e449c32f5f68f6b755887
20187 F20110109_AABHAJ kim_e_Page_112.QC.jpg
95d4425ae8b63d9eb5d59d59e55c1048
893240554dcdf443ba89850c5eaf21b7903d58e1
16780 F20110109_AABIEB kim_e_Page_070thm.jpg
90634a7058c271e02288929b4c2abc82
e2974fbd1a90aa623c72cb19316de51f91fb55c2
14405 F20110109_AABIDM kim_e_Page_058thm.jpg
2140305068c19c5855ca42815979b6e5
2ea70de16dad878e0a8ea522bfce3144f8ee3c1c
22663 F20110109_AABICY kim_e_Page_050thm.jpg
74cb20ca9d431f0c8e7f65494bb7f492
f5137455bd7060e0c3c99b0a25b8240e93d0aa6e
529 F20110109_AABHYG kim_e_Page_088.txt
c9b38bc7aeacfc65988c609bf52661b9
9bfcbdf82fe7ad07fc975bdba6e18c613d95f37d
842 F20110109_AABHXS kim_e_Page_068.txt
50a0a6e9961d34d40a5af476fd134487
20a1a1e709e62f2970196e4ec0542ae898fc8370
41157 F20110109_AABHAK kim_e_Page_065.QC.jpg
1bc8fa61177e0dbe6181e6fb101840fa
1b1da7611945144270c080f00df6ae433237e6a8
39806 F20110109_AABIEC kim_e_Page_071.QC.jpg
dae2a2413f26e21949bffced090b3786
7fab79f5e2b0d0eb5946d214764780c41a5d762c
71349 F20110109_AABIDN kim_e_Page_059.QC.jpg
d3f22b4602fa3156e772b4aec7599684
2e4ae2b3fb8bda9890adb2280ec23bec0c16f10f
57967 F20110109_AABICZ kim_e_Page_051.QC.jpg
2fd779982fffe2dc358113287fc2cb5a
b488779f40985c36b1282798965e4a683f662152
1795 F20110109_AABHYH kim_e_Page_089.txt
3230a33bb6b3706eea9a04b9d477536e
10b177996b8c3f3e54b24b0ce7821ede0896f391
1448 F20110109_AABHXT kim_e_Page_069.txt
4a1d4351f9390b6bf553da7a7eaac9ba
da2e820ef296b4215160a604cfa1af6407efbc2c
2074 F20110109_AABHBA kim_e_Page_135.txt
b13fb93d914239a27b2b20933512c68e
45491a0856420c125818f1e7dacbe07615955a3c
61859 F20110109_AABHAL kim_e_Page_088.jpg
554cb422fc574f47b8e7ab0ce88651e4
e42d7be34ceb45cb1bb8b4e56cc83a803f4f09fd
14178 F20110109_AABIED kim_e_Page_071thm.jpg
0dc2aeeae941f7d81ba06a0e1d6d3aad
0b4b9f60384bea889fffa57ad00a83e91481d850
61617 F20110109_AABIDO kim_e_Page_060.QC.jpg
2105ca784a1fb3793d52ab884257b171
e36ff3cc79e8c2796dde813ab51b5b01e8353db7
1790 F20110109_AABHYI kim_e_Page_090.txt
bc529cf302e48ec66f769777778b90a3
7d4a2ed693455032734ab049c4e37094c20d15fd
1775 F20110109_AABHXU kim_e_Page_073.txt
54e36b5848582905854d0b2be78f3131
9cadc525cae8fb429e72a400d0aa916b139d752d
1083 F20110109_AABHBB kim_e_Page_071.txt
3b2b2c1ad2aa5fc4eaea573edcd4ba36
0c9d056b957c8e9a85ba574dbd75aaa62fa4bc84
987 F20110109_AABHAM kim_e_Page_115.txt
82f4b84e707fda81935c7c80114da5d7
604585dea11854ae87ebb86fa4bdffb4cd837205
54870 F20110109_AABIEE kim_e_Page_072.QC.jpg
123b3c99451aa78bb020cfe97908671a
95553c22d55c0f38371ac24c8b6f41bd9a17128d
21919 F20110109_AABIDP kim_e_Page_060thm.jpg
0e08a8a64a7db321ce18c4d3e540f6e4
5ae1c80e577f3cad4b71f6193477d50af0d7ca2e
1721 F20110109_AABHXV kim_e_Page_074.txt
6882b5235ac9ac5c801756cf2eb015f9
9da2c2611a8937e6db6d07d15d918b88942aca53
79319 F20110109_AABHBC kim_e_Page_094.jp2
234bc14dee932b07457c3faf140d46fb
5bb1fda0e09797fe35d3b79a301465aea270e910
8362 F20110109_AABHAN kim_e_Page_137.pro
e06672f556b4ee43e4f2be07a39a14a6
7ecb8b38c9c7a49770cc9e1e87fc73240d1344c0
1913 F20110109_AABHYJ kim_e_Page_091.txt
ed5089f91f5b1fdd917cd7d8b53a6192
fac19db150af4a579f7a809cc215d0704d5c7840
18030 F20110109_AABIEF kim_e_Page_072thm.jpg
107f41a259590cd752f41cd9f5ec609b
5758a1ac700db10c7706858cc601c34ce1e7abce
49728 F20110109_AABIDQ kim_e_Page_061.QC.jpg
0032c8e00115ecfdf52e79d9fc3e18be
cbe0621c0b6cdc6959b9a329c055a3ff2f6848ef
1791 F20110109_AABHXW kim_e_Page_077.txt
266cadd76a29658b043cb50c99eb2ed8
dd09350cf7de15bced029d118474477ba20ee862
F20110109_AABHBD kim_e_Page_098.tif
a85f352d91c84d143d1990edf2f0ddd9
6f123af5f585a0778085e8f4003da445f1fd9019
8813 F20110109_AABHAO kim_e_Page_100thm.jpg
5ac9096a43e66ab627d3d46427153c93
160b8842342755129f168220ae1f0d9261dcee1e
1977 F20110109_AABHYK kim_e_Page_092.txt
63163e71fa5020031cc0cab27872369a
ee756ebe0eedeec9e2326c7cbfc0b5cad1bd431a
62301 F20110109_AABIEG kim_e_Page_073.QC.jpg
199ece2f8c5dee31c9f68e2814941f2d
0f71bc2b85c036114cf091df23b88d2c676f0eee
22556 F20110109_AABIDR kim_e_Page_061thm.jpg
63220f14c141337885baef226d88b9f0
a50200daf348ca5135a7213bceb3f964c60269f5
1725 F20110109_AABHXX kim_e_Page_079.txt
251d7779c5604b2738fd1bda7906101e
f0cb085e644208229ff9420c452a715b173a4aa8
20784 F20110109_AABHBE kim_e_Page_012thm.jpg
9f528b381fb9f6e48df8cc4b645de8bf
b6145b467281234ccf16166aedc6d19205eb94b4
35067 F20110109_AABHAP kim_e_Page_118.pro
58028a186ee60e22bc2693d947247f72
2c06a8cf5858384dfc9d48dd4c4c907b94bd7b3d
727 F20110109_AABHZA kim_e_Page_113.txt
9434d2b1cc089a97c560d949ae46d22a
1512bc17731b01ab2761653bab1edd894edb9859
1058 F20110109_AABHYL kim_e_Page_093.txt
a973711e56f7e70323391808dd873bcc
0d06d00adc9579aa0de838144e3a3fa143ca5ece
20967 F20110109_AABIEH kim_e_Page_073thm.jpg
78dfd863584535a2a9fbdd9e966db648
9f6d46621e1c34173c38604f343516ae7e4b27ea
53212 F20110109_AABIDS kim_e_Page_062.QC.jpg
9bbf62cc995f9ab2090e3ce440a97777
1a73ce4da91aa6a6fdd97da23e04304ab6295453
1983 F20110109_AABHXY kim_e_Page_080.txt
694780f47ddda3b8243263e8d6bc0706
ac26a450aadd8bd5f801b21821352d79012175b4
100483 F20110109_AABHBF kim_e_Page_130.jpg
3157d5f201653d6482070640b2abb7e6
0907cc253add701c88ae0d1226ef3b1faba6a620
73795 F20110109_AABHAQ kim_e_Page_052.jp2
b3676050ca1d8ec13c3bbc67543ce92a
08dad4422ff041043f21baf8a470843f55887208
1158 F20110109_AABHZB kim_e_Page_114.txt
8f491bd9b6a34404e585105ab9b351b3
d30476b60976036a7632049f192a8809670c6223
1667 F20110109_AABHYM kim_e_Page_094.txt
547aecd5e0841adb4d982fd4368be746
eb4bed02574617ab2622b9b79aff59cd9a779ac1
66728 F20110109_AABIEI kim_e_Page_074.QC.jpg
6e2c7e7c7712ca2c61dcce6a738821e5
9f39b69070c3062ecc03d8367b7e5c7aa7e6972f
18130 F20110109_AABIDT kim_e_Page_062thm.jpg
a9e73e6a8e144462fccc606a1bf61559
4d71e522617e889cdddf4737667c7e108fb9e2a1
1872 F20110109_AABHXZ kim_e_Page_081.txt
cad9c77af335e958121f1f5bca864276
ee6d407a45c3b3e4cd194640bd2d72b9b5d7d84f
62177 F20110109_AABHBG kim_e_Page_103.QC.jpg
0b6f283bcb65be346880d448d131e0f4
5d22a4d6fd6f56e63f0d192de4ff751ba738279c
137348 F20110109_AABHAR kim_e_Page_118.jpg
7f1a21c9241d72bcdbfa2fdf63c148eb
f000fea8e646cfe9cb79687cc2d10bfe79cac490
1623 F20110109_AABHYN kim_e_Page_096.txt
6a7066b75b4afcba82c010bd4ba69c34
85de2b498cab667d1db2b1eada463d90682a7d06
22239 F20110109_AABIEJ kim_e_Page_074thm.jpg
ecafbc152b19db583a3eb7da30d05894
866bebe37d66f572f4ebb45db38ad5cf662f8248
34638 F20110109_AABIDU kim_e_Page_063.QC.jpg
fe926d1364954d9dfa3d85ba5478c2b9
af36f8f4cf1e494a87339e0eaf241489eaadd2fe
152314 F20110109_AABHBH kim_e_Page_119.jpg
4b3cc105e07a78bb1d7a8390df73f394
4c5360467db2736b3b994df2ac300918b2186709
88976 F20110109_AABHAS kim_e_Page_022.jp2
8d686d566332cdd18954544b7c78130a
30e6a2390d5ca6db438b3e57ec7b61ff3406ee33
468 F20110109_AABHZC kim_e_Page_116.txt
be0142ff75b25457c49c4a73a25495f2
8f861d5beccbe4ae78eb3d6d2b812e37049a6c38
2013 F20110109_AABHYO kim_e_Page_097.txt
eab43ae4c4ca4c91fb0bae971fd602aa
9ac8fbf95ea5947c04eb280f474f3fc1c1369e78
55572 F20110109_AABIEK kim_e_Page_075.QC.jpg
76535837ba41e295a32b75c75a70ed33
5e81f942d3e00e6e6532814516450eaf7d7c3806
11855 F20110109_AABIDV kim_e_Page_063thm.jpg
20ebbd449cb73a4ff992660110368303
a3306d053a07d1f721c8fd47ac6afefcdf7ce00d
7223 F20110109_AABHBI kim_e_Page_137thm.jpg
00035fd43319e90490360ff241ec0420
7159c4d8676ad37ac0469c5813083588aca2bb65
70428 F20110109_AABHAT kim_e_Page_039.jp2
ee1ca75cb65fa44238c6ee107bbec282
6e51b93a06757fcd6518b87c7e9a3b02ce649ccc
925 F20110109_AABHZD kim_e_Page_117.txt
439f3533445456de91b06b4eb556d779
420e58d5f251372ae18bad7d8f920f6bc681377d
1827 F20110109_AABHYP kim_e_Page_098.txt
18072abd0197de71a8af0dc5ded7cb5c
f45a1e548872ec23e2fe664e4a6f5e51661cd73c
20203 F20110109_AABIEL kim_e_Page_075thm.jpg
0b5b28d6dda8e4a84cb9a4bd0ef75a36
554edf16d0e39e4e9c01984a25445f3fcfde69e1
33731 F20110109_AABIDW kim_e_Page_064.QC.jpg
d02d9ef5eb1f91bd23444e65bc7883f2
5f9ed45e706b4df3ab1aee98099ba29e44ec4eb6
14624 F20110109_AABHBJ kim_e_Page_011thm.jpg
82fdbce6803381305ac9aea36061199d
20b5fb488a2851d8e410719571f32c1cd1f74500
F20110109_AABHAU kim_e_Page_113.tif
6febf05bfbc50c6ee1caaaaf768b90b4
f363de0a51e2d5a6d6bc7836392fd62774382dcd
1519 F20110109_AABHZE kim_e_Page_118.txt
51a4cee701fbbe158b6bac111fc240a8
8f3b46c72d99d564270f918986227c658889c157
1522 F20110109_AABHYQ kim_e_Page_099.txt
6a461a5d050d61b2ea4457384a82a3e7
260c5c4ab2cc7cb5fc85359edec2d3d3120da858
20151 F20110109_AABIFA kim_e_Page_083thm.jpg
83dc60202938f5f6b098ea55f677b658
35dba094f3dd1be8ecc152fbeefee15f17b9b903
23063 F20110109_AABIDX kim_e_Page_066thm.jpg
8aaea68c9288f2645cdd07c006fe3584
822fcc0a3b5d8855d62b2a19566c482e59eff0ab
F20110109_AABHAV kim_e_Page_112.tif
fd5c8e553641467494fe6b43302d4868
19c38af1f8e199361920d79f4c322e8f8f993969
1377 F20110109_AABHZF kim_e_Page_119.txt
4d92912850a837711cfe43290d693244
8314b395944db1dd8f66c772f959cab8eb3dc460
687 F20110109_AABHYR kim_e_Page_100.txt
1ae0a8ea5fc1a508252ef48ef6b1e416
b65a1fb4b3a7b3409f946f2a210fd5c6a93d2e40
23057 F20110109_AABIFB kim_e_Page_084.QC.jpg
40cd5c0d2cdcefca39692ec0402a057d
f67eefd3c1fbac78c377ceae2560461c97520fc8
16267 F20110109_AABIEM kim_e_Page_076.QC.jpg
7ee71cf408f8087566decb0793f5f3e3
7744ae1885b675f91a669a649a70207720790a0c
67690 F20110109_AABIDY kim_e_Page_067.QC.jpg
71792aa57613acf3d5aeba117c956083
9b1edfc19ab966ae0eb6d5bc812afaf577dd1ef9
97897 F20110109_AABHBK kim_e_Page_081.jp2
1e22d2bb5db8c02483b7a4e19b0a44c2
a88b546f1007d7ddb316e8f210aedc692ca1b2f5
11278 F20110109_AABHAW kim_e_Page_113thm.jpg
8371c4293071cd49e4dcb8a9adb33edd
681d541b33b1e58fa4ca9ff7f6c48fcd3d7dc2aa
1201 F20110109_AABHZG kim_e_Page_120.txt
d6ffca3a47262863eefe2cceaf899f17
0b0010b5a4a19f7bada890991535bfe5e17e652e
1254 F20110109_AABHYS kim_e_Page_101.txt
d8e19303ed729fcf4d6d87dad8373834
fefe7e33680e55a2db952594b33734aab31800c0
7763 F20110109_AABIFC kim_e_Page_084thm.jpg
da0dc75ba7a800a0de2e30e138358b59
c2f58c1c7b204947e08700a1fded16e7ea16bccb
6602 F20110109_AABIEN kim_e_Page_076thm.jpg
76313b55f420e9faff07a3a6a737eacb
7f441bb92f9408020fde5d0edeb88ffe783ddd0e
33372 F20110109_AABIDZ kim_e_Page_068.QC.jpg
cb64fa18926d39268950fa9c3f146d65
e1c6ec7bf35388432ecbb2cfa9a4e11f6f46cef5
20383 F20110109_AABHBL kim_e_Page_042.pro
28e169eea5af84f12d7b37a37a7c738f
150dba7fee41beb6d684f8b37d4f78752e264a55
89735 F20110109_AABHAX kim_e_Page_028.jp2
d66f15aeb8e320d83b38ee0d8cb9bb6c
9f80902fb98e982dee2d877e4e0f2740a39901b2
1627 F20110109_AABHZH kim_e_Page_121.txt
03752ab2ff0766cedd237f1dd85513c0
6cc6bb6c985ad2ea9204acc04b2b5ceae879ca69
1182 F20110109_AABHYT kim_e_Page_102.txt
6d2b220bb328494f92f7cb9a99bc612e
aed1e1cf7aec5fea062c0fa1b7fd0c1695483b5a
F20110109_AABHCA kim_e_Page_002.tif
3d5575628d973188823ae8b360171255
f4bebeea2b5fbe327d11aa75860f7ab766884657
63572 F20110109_AABIFD kim_e_Page_085.QC.jpg
1478f975e2f8965b435557262b804bf2
89307f3d645a86789c985f8378d7e27d4455d470
69410 F20110109_AABIEO kim_e_Page_077.QC.jpg
c17a9c888326954a6cf501898879235e
ba9656ae0eaf2f2a26ab7dd4c7ee765089ed3b61
186710 F20110109_AABHBM kim_e_Page_098.jpg
f2f2ef00ac105656f7466bc80255c999
f73907c82b2dfa7dc4733c73f9e59e34f0db0ac1
209499 F20110109_AABHAY kim_e_Page_013.jpg
65237dab1bc571978c2f7391df857d03
3f4d4b5df99072eba840f3727fb73dfced0f0610
1438 F20110109_AABHZI kim_e_Page_122.txt
29a4d9e60e7afafb2591f3429ccef88c
94b993133c54e4bb39ca667d28a829b79fd96fe4
1000 F20110109_AABHYU kim_e_Page_105.txt
be4586721d47eea2b429b2d7d22ad4eb
7cc993eb76cd41b56cbf1874cd434e96b8d45ca8
606 F20110109_AABHCB kim_e_Page_009.txt
b92346cef5a5774fd75e9ad2d6dc2295
5ab29eaeb7c8008cf2ac95f66a42a80c5e756662
21025 F20110109_AABIFE kim_e_Page_085thm.jpg
288fa0b61ca980fc30306573a0237ec1
8c1092a107dba5cdc7ff470fa0e1195d0cffd19b
21671 F20110109_AABIEP kim_e_Page_077thm.jpg
7978359403d15018f697ef833fe8dcb4
e64c09b4a24eb29f6a976c14078b29b40eebbb02
50441 F20110109_AABHBN kim_e_Page_063.jp2
7401f4e0a5a2e3f9b95e3f99a0c13619
b4102f50f569e5ec59bec93aa46c609eae0d0b73
16076 F20110109_AABHAZ kim_e_Page_101thm.jpg
5a949e1cc76d4c08c69680df9177517f
1dc6cb3d4bae6d94b63a3b38b9efe8511099c135
680 F20110109_AABHZJ kim_e_Page_123.txt
ed6d536f553cbef09fb4f041ae00db3f
b215139c75d6e89c03996ec4b71784dfa7cfc09c
768 F20110109_AABHYV kim_e_Page_106.txt
22e01f23bccefaf93a9953e76a2c683b
2e601d05116e1037886f669b25b538bae05f741c
22920 F20110109_AABHCC kim_e_Page_105.pro
5d61dd9affea72e2f22ec49c92b9df91
44baeca42aedcbdd763524017192b69508b50d28
75384 F20110109_AABIFF kim_e_Page_086.QC.jpg
c13e276ab53813558d4074181dea4868
18de1dca1c3f6053c5b321bfa6a6686c089e1c67
61110 F20110109_AABIEQ kim_e_Page_078.QC.jpg
c797d2e68e3879852dce814c36744cee
f27fa34cc70a7f21812045a1cea0ac4fbfa56e71
36742 F20110109_AABHBO kim_e_Page_130.QC.jpg
414d1fb92ad44952a4f143c7ed197ebc
36fecd0b69bd85f64f2e96bd65aea27f42c79d0e
187 F20110109_AABHZK kim_e_Page_124.txt
5e6932ea383a42dd7df299f382aa51bf
e5a627ca5be72c7cc6da41319997931af72062ba
1112 F20110109_AABHYW kim_e_Page_107.txt
78073a8f892c5ab73024f871d2212ca7
6268061e0a89b160eaa70264d7a7baf3dd9f699d
165370 F20110109_AABHCD kim_e_Page_087.jpg
9100a36e0d11b274046e612106b4a846
9ee568e283159541641db6903343ed07c64f8c71
25487 F20110109_AABIFG kim_e_Page_086thm.jpg
887ae70427b3dcca07956c19bb59ce92
714a14de767170c5cc813a35c38d536a58072ecd
20053 F20110109_AABIER kim_e_Page_078thm.jpg
b7684eb7bf8b7a93762d0712e8dcf128
910e63f76624532aa41614e5dfba742d455ed94a
25180 F20110109_AABHBP kim_e_Page_049.pro
6522e2c351bc14427d582dfcbdc02cf6
45b9436f12d625d536d980ac9745fd908ce26648
581 F20110109_AABHZL kim_e_Page_125.txt
7321f039a6f95588d1e13a8adbea6ab2
35c6c0e82fcf5cb7f58d0b1e26e6a72c131226b2
1057 F20110109_AABHYX kim_e_Page_108.txt
b8042f3d0ae1c59b4de4cb9bbd779559
2ef149fa691c6f6bb11dda361adae6bd90046759
14937 F20110109_AABHCE kim_e_Page_065thm.jpg
c3c4fb0d7a6cac9db40a344e33ef2c5f
648a3f313abff5cbe37d6cb9093b46b5ab3ef0cf
77451 F20110109_AABIFH kim_e_Page_087.QC.jpg
c2209f45f7e7f2f57b2ce3cfdb4dc669
ae93806a2b5e8d7b350ecab2298aa4bccd5b468c
65469 F20110109_AABIES kim_e_Page_079.QC.jpg
1b87b66f175a50c90e54bc73b815abb6
1f945474d37f497bca6f297955e586aa62f9d36e
1289 F20110109_AABHBQ kim_e_Page_070.txt
185831c55b0f0d26dcbc1e1b498b4b4a
3e3b10e3eca05053169776a372d8036fb3b2e25a
1210 F20110109_AABHZM kim_e_Page_127.txt
29926fc2091811d3d14cca836fc440fb
eda1b27fbaa5f6dbc3c380cbb63fdd2cb586f80f
967 F20110109_AABHYY kim_e_Page_110.txt
98d98a5c456f0ed01c70ea12d83527b4
a7b4a25dcb7e5963a220eec198800b54d56b647e
63658 F20110109_AABHCF kim_e_Page_084.jpg
9254bdb6cad381d693bebfefa153e1c6
a54dec890dca67a520390923dc28151807e1da9c
44270 F20110109_AABIFI kim_e_Page_087thm.jpg
e05095b3f705eb54edb7b1c35216c6da
ab350787f2a2585af87e2b28882724546bb990c7
21245 F20110109_AABIET kim_e_Page_079thm.jpg
e21d2da935b6d33fa69c5d25129c3d0a
e83daa3eb0788ee23b0c0a6d17c4bc3839fa564e
17624 F20110109_AABHBR kim_e_Page_037thm.jpg
d2de5279c340937da3af830c377c9aa1
c27ac879cc87ad8ccc644bbc8d47599d91fa8149
1016 F20110109_AABHZN kim_e_Page_128.txt
9bba9de4710c4d52e66a5dda7fa74e2a
07168191b7d491f8bfc978055c9793e8e51f17bd
502 F20110109_AABHYZ kim_e_Page_112.txt
f5d7ee81bcc5b1dfa651817926fec520
55a98275812adb615cc284455e06090995c4f8e0
111669 F20110109_AABHCG kim_e_Page_010.jpg
58d7bb197fdeb906a2133425681bbd0d
c1a2fdd3ed2821a13972d0eae621b13a073cb79e
66973 F20110109_AABIFJ kim_e_Page_089.QC.jpg
83bc8e1d4e8773cfdb54d14faf7d437e
74cffc346821cede0b1f80b1ff779a0e66ee452e
76017 F20110109_AABIEU kim_e_Page_080.QC.jpg
5d351727ccab5e24d3558e014d2586cc
59d47d57c0db702563946655c1c279512eeaeda5
208882 F20110109_AABHBS kim_e_Page_030.jpg
60a78b02881176cfa3f871107407474a
c2d847efc2c20d049a10244f0dd63dc64bd4bb12
1003 F20110109_AABHZO kim_e_Page_130.txt
5b1381ccc3125d42100d313791a977f6
b29afba3210b904b69e1490342c4c49c796de1cd
1087895 F20110109_AABHCH kim_e_Page_135.jp2
06947a3e4da0feff886c8d10498135bb
ddc87276d2f296c72e31ffd2b08e4d81309b9a7e
22695 F20110109_AABIFK kim_e_Page_089thm.jpg
eec4c04e06700c81606f2ab83e720eb8
dbe2f4be0c294be8478d647e5dc0c1152c028e46
24909 F20110109_AABIEV kim_e_Page_080thm.jpg
7b825f7fb3849efab25ef7b15560d8af
fe42e3e2ccd95d9b92d3d109b28319d018849b52
43422 F20110109_AABHBT kim_e_Page_089.pro
4288dda99e097a20ad82c48890fb8d50
0c3ceb3f5e494c469c161944de1e5178188b86e6
960 F20110109_AABHZP kim_e_Page_131.txt
e1c3db77725c9d9e84d99f769bd3bef4
60340adf33faafea91928865d90fd1dbf57407bf
F20110109_AABHCI kim_e_Page_119.tif
c4cba7f3557ea3a69683257feca5b649
03d30b3f95cda446621a55dfe624c45beee1a81d
71552 F20110109_AABIFL kim_e_Page_090.QC.jpg
dbd43cec4c22e918552f8f219d394098
630b1d318bdbb530c676c5bde04b31e367dcfdef
72430 F20110109_AABIEW kim_e_Page_081.QC.jpg
f096e04ead76a06fc272b93df7aa2e8c
0fac31fdd773de4b65b1b078d060907b1bda5234
106832 F20110109_AABHBU kim_e_Page_086.jp2
8fb220b25dbe25633683a6b0cc92f263
f8840cfb36c78e007450160461e1301b381c2c6e
1039 F20110109_AABHZQ kim_e_Page_132.txt
6b05f1223470016f604ce4aa93d6974a
2fe21764c01f634bad96dac0147bc254243d33c8
4059 F20110109_AABHCJ kim_e_Page_007.txt
a7dcc2c0c88544db0b30b98575610975
a91cb27976300ff3db71de16b1c314aed2dd9f3c
48555 F20110109_AABIGA kim_e_Page_099.QC.jpg
977443e3d4b0828c20538fb291baec93
e1fe8fadae4d37c132ab61c93e5d71a02af2bc44
22982 F20110109_AABIFM kim_e_Page_090thm.jpg
5edf8cb6f8e4b414ee976c792c3ee3f0
bb1815fe697835074b451de47c6f8e379e58cbf9
24112 F20110109_AABIEX kim_e_Page_081thm.jpg
987cf48d193adba5e148dc46e557bb3b
24c9a8dcc1582ca2d992cdd4d7591b5fd1b2fdd5
27218 F20110109_AABGXD kim_e_Page_045.pro
66b8a7902473bf8b66b543f8af4db17d
f1fab44a750333a91ad18649c15bd85c2cfaa192
42344 F20110109_AABHBV kim_e_Page_011.QC.jpg
bc3f3d4d85dc59f2a045b571b2d0438e
92d630b0c78c0c4216cbabfb951dd69bf19fb580
86 F20110109_AABHZR kim_e_Page_133.txt
2e93912310b8a10f73e82260ccce6414
479bb1c72c9294edb2da00492a878451e4b10c21
488 F20110109_AABHCK kim_e_Page_076.txt
be4b91d5533b87262b7288e8fe1ca83b
8c572227ab826017baf76568960166c558a49ca2
17520 F20110109_AABIGB kim_e_Page_099thm.jpg
cdaca267b16dbb3d33e76d0cca5ab626
e5c3b6a35d878bea7bb038f2048e404b50cbde50
73414 F20110109_AABIEY kim_e_Page_082.QC.jpg
41b152aaefd965734c6de005ba62b67c
ee8eb0cf7ae8ffdd15b4e8da5aa01c68387cd2d8
13542 F20110109_AABGXE kim_e_Page_110thm.jpg
a94ff9e19fcfa988375b8fe80bb2887f
145d7c5f3ad00ac35e38ef58a51e43e42cb5b04f
9354 F20110109_AABHBW kim_e_Page_004thm.jpg
6dfb6eb9625b8e901fb4eaa2d04a74ba
680f60ca4eb86e58e2fd5f4330627683d745b664
2092 F20110109_AABHZS kim_e_Page_134.txt
2eaf2c0a2500f7a92accc3a850e17e05
b971a01bfe17cc3425649a2a135d0a63e58df2e3
24766 F20110109_AABIGC kim_e_Page_100.QC.jpg
07399f5a05b8c40019ecaf9fe883f1b9
f9f8f76d5be87bf72d504adff902cfc53fa74ad6
75358 F20110109_AABIFN kim_e_Page_091.QC.jpg
d56bf7fadba1e4a25ca0ae74967a9d22
d19b13cc4857272c102cafa05b5fe791257bd209
24613 F20110109_AABIEZ kim_e_Page_082thm.jpg
52c30f7111f4a4638d2f9213b547760a
ac1c49f2b11e93530d037a162711050b9c5532df
948 F20110109_AABGXF kim_e_Page_126.txt
0ecd09d076c24040961366e39c407e95
ec798efa609d831a3455d701f3de194914ca7069
110544 F20110109_AABHBX kim_e_Page_126.jpg
21126789f7cb3ee4ec1b20415db5e586
1a6a13ef0265eb50bf63d306ad9280ade255baa9
356 F20110109_AABHZT kim_e_Page_137.txt
0835949f5bcd6d8e99b34d2c71be02dc
a2f15ac7f05a25823de929177061927ecf2bb8c9
25775 F20110109_AABHDA kim_e_Page_051thm.jpg
45ed4d47c3a94e8889ee2eee7a9f36d0
b57cd63282430dd3c660188940a3e3177622446d
721583 F20110109_AABHCL kim_e_Page_136.jp2
d76f5ee93e0f4c3c7f5c796be4ebdc73
a189b0358d5958bc048aea46cd8003894c1f317a
45409 F20110109_AABIGD kim_e_Page_101.QC.jpg
ce6db286b0be12070dcd7670c976892c
405ab094f5e147be6c717d8fa729163dd8827484
23943 F20110109_AABIFO kim_e_Page_091thm.jpg
5fae87c96c8e4e92a73ec0879afb1c49
7c01c080ca2f89c04e0145501214f4f798a3a937
F20110109_AABGXG kim_e_Page_017.tif
adb01f17699eb9056ad7691be89f851c
9e5d2bace217ad04e7efd305935b158819a066da
22375 F20110109_AABHBY kim_e_Page_061.pro
b28239715df0a81174bd67ed2921faac
f2a6617f7cff26468558d58ee9b45066bc20e568
1928918 F20110109_AABHZU kim_e.pdf
c9c312170a3b474f1656608e7329b65d
6336a1b04eb6104b41f9d19ccce7800880dd4c8c
F20110109_AABHDB kim_e_Page_111.txt
6ec5db4b1ef8fd421c1db2241ed60be6
b14156d18e4287753b4ff03a8155336036ac32cb
F20110109_AABHCM kim_e_Page_106.tif
b480c5755a7e55b4e05592030d4f0204
6beb2ca23e776568200711b348701c1128046ae4
41074 F20110109_AABIGE kim_e_Page_102.QC.jpg
b6e1bda3ec6832b07b0b90ae7d905deb
7db78af52f0b1f788709f8b5ddcf0004ff990047
78073 F20110109_AABIFP kim_e_Page_092.QC.jpg
51c7934df302d4b65c910ad5876af6af
eeeea435b9e40f78afe1bbe08ca9a6836fdf87c7
1418 F20110109_AABGXH kim_e_Page_024.txt
2dc471cab68aa4468030ea3b6bb9d336
988497f86bb976c74cf52f45caf7815c62bc7f71
F20110109_AABHBZ kim_e_Page_025.tif
87bfd408dece6ae87bfa1af89865aeb3
224dae09b2c361600b9c46a7f60d13b99331329a
16875 F20110109_AABHZV kim_e_Page_001.QC.jpg
8f64df67afe7d4f737561c056af130ea
c00ba90fdfe792d150f7d90298023b73c9e57bd2
33811 F20110109_AABHDC kim_e_Page_009thm.jpg
56f7483e043906af184d9f9da6da540c
d3eac6237e208a8f79c539afd05ce16c069af650
119339 F20110109_AABHCN kim_e_Page_042.jpg
693d6d0a51f81f434f03eb3360d395d9
75638f4bb24a225d7848c2bab7453fa11ecd585a
14967 F20110109_AABIGF kim_e_Page_102thm.jpg
f987db4257a0315e3d9b252a4c556386
075ec7536347ed4bf8d0b2d2bb8ac69f7428a0b0
24617 F20110109_AABIFQ kim_e_Page_092thm.jpg
bb5fea3f296fe1eef5df284f2a245588
7636ff30dc2f45e9bfc5e51bb1865bb74443a31d
1027 F20110109_AABGXI kim_e_Page_129.txt
4fd7ae449bfd4fd0359d9e7dbefa1d16
1d823c5c17610e4580718953734f9624bbad33bb
6763 F20110109_AABHZW kim_e_Page_001thm.jpg
43046a381135600bda63a4692112f81c
a15d8548348f06e1a88af4cbd6e60a6f2323227d
72978 F20110109_AABHDD kim_e_Page_005.pro
dbc604fbe027c2e0014af1019d364ce5
eb19adf5f5cde1bf695d7264e38e8db8743112b8
170049 F20110109_AABHCO kim_e_Page_103.jpg
2ba3c4eeffa3c5bb2b88e56737314e37
bc3f981f85f0644d989a234cd4b7265bab65540a
10748 F20110109_AABIGG kim_e_Page_104.QC.jpg
6815eae62b60b4f63323a8654ff3a050
001bb74fde18c579d4420fce53cb7e28f87157c8
39211 F20110109_AABIFR kim_e_Page_093.QC.jpg
5471d852d8dcc9d115b73c87c643aa83
20d05af1d9b00540b019ad365e6446cde4f95969
1564 F20110109_AABGXJ kim_e_Page_095.txt
8296c2a4db39062dfb0ea112839ddc2a
b54ccf481426401d48afeb117233a1023f5891ef
2974 F20110109_AABHZX kim_e_Page_002thm.jpg
61ec365ab293e58bf744bd2311e6354c
4de90721eb70ce5a9744b68e0447b50d3ae775ae
F20110109_AABHDE kim_e_Page_093.tif
b710a445784789a1758adfdd289c7432
c2afdfea3e0f713d31be29a93c8f1ccc1139f88e
1240 F20110109_AABHCP kim_e_Page_136.txt
5e2a0b6cd864d94db9f26c05291c3133
38499e9ac7e46da286adc70ea77161ab4c8037ac
5172 F20110109_AABIGH kim_e_Page_104thm.jpg
0dfe5a14ebb33d8383b468dedd05f106
06ee4e86914241702910b9ee81b425436e2c1fec
14097 F20110109_AABIFS kim_e_Page_093thm.jpg
dc5dade9a08190720ecc9360ac60e4c0
65691c09cfe1f9faebf3c46529c8aa730f9ad200
26135236 F20110109_AABGXK kim_e_Page_087.tif
5e09af64980950b2777c09a0557faf5b
de3361b3b71927782a8caf1865e502952b453517
5760 F20110109_AABHZY kim_e_Page_003.QC.jpg
e190bde8077a60fc0eb82df9530b9d9e
d2b28589ed0d316995636395fd718bb65f43e2fc
22658 F20110109_AABHDF kim_e_Page_103thm.jpg
cbdde3cebcf6e1e48f27c3f10268b658
b550326a8c4c38dbdd6a0a40a4c78dfb261b5729
F20110109_AABHCQ kim_e_Page_103.tif
d41367eb12f618f1eb1e6d946d0fa8dc
1310f1d2bf46b26d5c640a537076b017f533ceaf
41377 F20110109_AABIGI kim_e_Page_105.QC.jpg
8d3e207403e8207681858b3157eb5b80
7e7d811543bcfd28406b0ba15663d6c98b0cbc21
60409 F20110109_AABIFT kim_e_Page_094.QC.jpg
3cea8b76886dc746e5b38d9ee4904f30
aa46774e76d32a07e2ff14c3d33d65480503a1e7
91669 F20110109_AABGXL kim_e_Page_017.jp2
33a5054684f4c9495d99ffb3ddf317d0
fd94b2b095abec8503ac00c8315f64cbfbfe6d64
25661 F20110109_AABHZZ kim_e_Page_004.QC.jpg
dc13dd96fb9f60388ef855214512fe01
140d13a32a7583a46db423cabbae184b519b4ff0
1444 F20110109_AABHDG kim_e_Page_044.txt
046829169c59b492358cbc61abf4dde8
e0fb8becd1065539e98be7f4866caf2c0d83450d
F20110109_AABGYA kim_e_Page_075.tif
ca3908b8e0529ac3ee0ff8221e56e8b8
7fc5fbb0dca1524dbb59ddd88bb776107a10a182
F20110109_AABHCR kim_e_Page_110.tif
41355ca299a3b101452fc6a7c5a755d8
4b970b9ed502a04a7a0cfe44c16ed724a9b1cbfa
15296 F20110109_AABIGJ kim_e_Page_105thm.jpg
dfb62b0c3f790d2790aa8f3af62f012b
6077873c42803f6a9a4835bde3bf194a3cc79e09
21605 F20110109_AABIFU kim_e_Page_094thm.jpg
0196b53bae58680e1c5ae54e26d5eb6c
317c41572c27a3105039f3d67e6cb964eb0f6321
169592 F20110109_AABGXM kim_e_Page_060.jpg
8ff93da99cb1482d97a4892d8b38061a
9b566e81325c6b6a321a06745f3233c0d552b2b7
90419 F20110109_AABHDH kim_e_Page_077.jp2
e1af97a5d539ed61b6aa0c0057f69d30
ca3fa96bdd52005f27c42a21f58bff662be2122c
15126 F20110109_AABGYB kim_e_Page_122thm.jpg
8d9c762c301ab4abff968d628055d844
6b94c521d3ec1e217d1c9bdb890c5b005727c3cd
74538 F20110109_AABHCS kim_e_Page_050.QC.jpg
bd6f0c730d2432d6f0bfc7172abc293b
a35c24b360c9770b1b6fbc4a5c90632cf260d874
33878 F20110109_AABIGK kim_e_Page_106.QC.jpg
6a64e9ff2eb1770f6717424512a4b5da
d37c999e857f89aeff85fd74d425a5e68e11d988
56810 F20110109_AABIFV kim_e_Page_095.QC.jpg
bb86e13a65be90e11db58ed8cd9c627b
5d656a241597e63cf0daaa09a48183ae9f61d6b3
135152 F20110109_AABGXN kim_e_Page_120.jpg
f69605c152853b0d6fdeeb1e7d30158f
dc35c72223e30da269cb88697445892ea1e3c38d
32047 F20110109_AABHDI kim_e_Page_113.QC.jpg
9e9766a22f492911b68b30f1a7fa6f42
218f0493522be12a12112ce3cb83d23999521f02
47138 F20110109_AABHCT kim_e_Page_107.QC.jpg
53557deb8d863092947a614a46a168b4
d4623633350d1843ed9808b33f80dfb4e3da5b07
16445 F20110109_AABIGL kim_e_Page_107thm.jpg
4a47c82901f83846ff862cd288b8265f
170f62d5737c088ea53db8aa92613560123261ba
60526 F20110109_AABIFW kim_e_Page_096.QC.jpg
eb35715b1f4eb38191796392554b7cac
cbedb799c81ceb8c3e923f01f51ef8c09029da13
F20110109_AABGXO kim_e_Page_057.tif
df665ce5cf094978603e8e57950a011b
a3a60b579ef8259a9b5ced079aa4ee3cf3435b14
6562 F20110109_AABHDJ kim_e_Page_003.jp2
ea7e3f7e2f4eab393c76e6d60140ef37
bf6f8a185d7066ec37990fa306833bf7c1cf3662
191128 F20110109_AABGYC kim_e_Page_066.jpg
51d8d06fb0b57ce10cdf2c46855ab5ec
7e9e3146b4ce90abf6adfc6a1d31f1a9739c1403
12121 F20110109_AABHCU kim_e_Page_106thm.jpg
b38d6e89346ae5aa451ba2375859c5a5
91d23e44757088424c56bf8272e6347780b85746
17787 F20110109_AABIHA kim_e_Page_119thm.jpg
b1705c819faf4090120fbb6fa2799872
a70571d91c65ecffe02adc7a428001c5374fd2d6
47350 F20110109_AABIGM kim_e_Page_108.QC.jpg
b13fcc82d8bd9a396b0eff53c22dbae8
0a0cd3872b56932cdfaea75e11b4d2d791f490fe
22285 F20110109_AABIFX kim_e_Page_096thm.jpg
2da8a5d618319111a09e3bd3f086bee2
69708a3446eab44a3e22dc43554edad5235cf27d
F20110109_AABGXP kim_e_Page_133.tif
ed9bb9a00b424b448054f8fb86621b5f
d8eef1cae3bf1adfe5394e87ffeec66ee67b7deb
26145 F20110109_AABHDK kim_e_Page_109.pro
642102b0742af07aca492c05dc3ed1b0
2bd4b988a8fbcbf3f5a9f3ca517a5b625659e680
1776 F20110109_AABGYD kim_e_Page_046.txt
e099a05b50ca66a42b27dc49e2b61a62
0a30b6ad4686e9b3bbafecf57c02804c8d1b78cb
F20110109_AABHCV kim_e_Page_090.tif
6db3a1aa1fe0b065d6eefdaed3ada50d
2bdd5fa8ce07613fbe245f9cd48f22cc3b7231c2
47336 F20110109_AABIHB kim_e_Page_120.QC.jpg
ffc2dd95d16b203849f251bdd4b1f274
d27092160fdc9b0af8dbe33f934b129e967b8c2b
14972 F20110109_AABIGN kim_e_Page_108thm.jpg
61b061f3d1f3c1dfe5f642ba98c76c41
48dcdf9fe527792ab8b87b03c545c0a5ff56b364
25963 F20110109_AABIFY kim_e_Page_097thm.jpg
571369492192438c67279f64af0e8192
1e1ba53eb5a149a2d559d53d9728d271a6974c77
54612 F20110109_AABGXQ kim_e_Page_102.jp2
71ce4f885e5e2b47ca1ecda95bd3581f
2ba3cf0ca6c6ba8680a532fd394bafb09a98dd16
8418 F20110109_AABHDL kim_e_Page_088thm.jpg
114bbeda1faed89ee115dde31a26900f
11fa43e1fd0fc628af820061d7cf01a6b425a9d8
46524 F20110109_AABGYE kim_e_Page_070.QC.jpg
14c5f844c0a56e69599d6a31cbfdaa1e
ef5e57b6d0e0d7aef0b78d6b63d1d650c5377602
51063 F20110109_AABHCW kim_e_Page_069.QC.jpg
30a67fa04a56b6fef8e8b2aa87446913
635ed0bd6c7df1c75b0dcbe33b0446a1b4f7bdb6
50682 F20110109_AABIHC kim_e_Page_121.QC.jpg
c5de0895d2e4905da110c55db7fde9ca
3be54ffca93e416317214f1407b5ae909abdd250
64776 F20110109_AABIFZ kim_e_Page_098.QC.jpg
6653b954dae00ece7650570550811e6e
6bef0826e1b7c6d7474b2bb3d2ad8cfb51873560
22760 F20110109_AABGXR kim_e_Page_098thm.jpg
5d375b2cfb1db44e91c6fcd8deb45215
0887581ee062722e4222e27369000418a7712c74
10088 F20110109_AABHEA kim_e_Page_116.pro
318a874e57c28815464666ea0e956367
a2487448e958b5f64737cacfe05c60a16804044e
11440 F20110109_AABGYF kim_e_Page_064thm.jpg
7382225459d115b77f499fc86ca54b96
ae54f38a9ebdb8df087a3c0fcbfc9efc873f9baf
25915 F20110109_AABHCX kim_e_Page_011.pro
3066526ae16cdff98da386e6895b0c4b
aa6bcd812bb1be27dd1a48f470106a45cb5eac85
16793 F20110109_AABIHD kim_e_Page_121thm.jpg
6fa22299e648601f1d15e685d9c098b4
2b0023f80425a511159ed1791015f218c52efc3d
46660 F20110109_AABIGO kim_e_Page_109.QC.jpg
3482149364af643bcf00b7f83a4340b1
4fcbed2ee66300952abe7d268f63c296ff2ca7cb
21368 F20110109_AABGXS kim_e_Page_095thm.jpg
8540e51e7a16aec0b22b173fc339a2ae
6188a533f3e1c0ae1ec93b1419a7489356d2ce36
70186 F20110109_AABHEB kim_e_Page_026.jp2
5d8ed6dbd5e3e6e6f0babcc356fe5197
e378188015cd92ce7fc9c2e45a261ec90d5d1707
38250 F20110109_AABHDM kim_e_Page_008thm.jpg
c6f9829b42fa84fe2ebf59dfdb5e6b44
bdbc75891806d5766c06ab67b1996d34637c6226
16128 F20110109_AABGYG kim_e_Page_118thm.jpg
5b85891a16107642f7d7caecef97f530
d51c0332a7e1e99e13beb22c9e1f8b3aeb263a68
197145 F20110109_AABHCY kim_e_Page_081.jpg
b25255d135785c32ad2c594f6b1ad157
ede242de4acbb086b7f8fd60a566408d1b08a86e
45608 F20110109_AABIHE kim_e_Page_122.QC.jpg
8af56e0f066e1be92b1b41f1f6e02ac5
adc05032931239138e6aa95892949330d2594b7b
13866 F20110109_AABIGP kim_e_Page_109thm.jpg
71952b2a8298664b31da8618f873845d
411457e64f92493e520348db8065ac31530782b3
7336 F20110109_AABGXT kim_e_Page_112thm.jpg
326c5e044a7e898d0e18fd23b0a85c99
0290239ce0632685bc88185fd3c88efbc6a3f25e
F20110109_AABHEC kim_e_Page_026.tif
137a95cc529a7127e7cc7703a53be516
36dc7c114e6925b5d004a823615bc1ff1139aaa8
49151 F20110109_AABHDN kim_e_Page_092.pro
411091c225db7ff5e246e8b557ba819c
ceec3d2cd8680b487fdcd6f0e0525d96dcdefc7e
39748 F20110109_AABGYH kim_e_Page_128.QC.jpg
c2f09b6994354802698ab425743f44a0
3c7c98d28b7da0366b659ed33806c8de5ff7c16c
1439 F20110109_AABHCZ kim_e_Page_039.txt
d4c9ae7a71a9cdcb372aa973f3fa56f6
6a63346435049fd67506c48b0ebc2fa69abae5c9
29224 F20110109_AABIHF kim_e_Page_123.QC.jpg
061d9de7f99faef175b755c934301699
01455c31b3b3c3e7bc0f2caef98be7bb89283481
42733 F20110109_AABIGQ kim_e_Page_110.QC.jpg
196b3edf15b94e87858c5704224ac156
fc1f714e158141bf64c84f7f4b9710cac03cd948
48875 F20110109_AABGXU kim_e_Page_118.QC.jpg
22c9a16b727dd4e754a69a5541837e31
4d0bf1a9de47cd8af515757b13b7888b33b2d746
39495 F20110109_AABHED kim_e_Page_035.pro
187e51ec93ce4bcdd45c22e48046bb66
e6cff2da3ef3c596c0fcbb20fa84d6db0788f043
205614 F20110109_AABHDO kim_e_Page_036.jpg
31d40e814bb012e76c3a9ef662ba5798
cf56afb03ddccfff8a581dacf8489d5e5073d43d
1672 F20110109_AABGYI kim_e_Page_078.txt
b4710c924d7e0cfc6ed54c5c60d07209
5a62031553416f9a6676c2415c8a0923930fe199
8644 F20110109_AABIHG kim_e_Page_124.QC.jpg
ff99097ac72c0169069768cc0fd05242
67d6bf89095d7bf2eaaf4e4fb5116a547b17703d
36560 F20110109_AABIGR kim_e_Page_111.QC.jpg
1aaad82510f434451900288cd2c3e654
dbe9879057c037ba2cbe0b38d9c2718a8fe39134
F20110109_AABGXV kim_e_Page_027.tif
d4b48f1fe252f92cb8be951de75562d8
4ec6d02365b93edd8f8afa8b407097b2f88d2930
162194 F20110109_AABHEE kim_e_Page_062.jpg
6f25212c7f63c6036b947d87ef773ced
25c02ed5d87f04252263ccb2d39fa392c66185b0
3188 F20110109_AABHDP kim_e_Page_003thm.jpg
3608132095b886d84bda003afc424970
7c96f4dff40fe577d549c2d4aa88ee1de6ab1e7d
5164 F20110109_AABGYJ kim_e_Page_104.pro
97583e783426eb4c1f2b5c4c2b48f763
d03b0610b05fc537481e310e5389fc83bba72f74
4087 F20110109_AABIHH kim_e_Page_124thm.jpg
d69d06e6273809c2873e6ff0dcf41688
ff09fce76a884d536c75c15ae4fc9802a2455523
11695 F20110109_AABIGS kim_e_Page_111thm.jpg
acaf30e10c49ca5c1669ec133af777d0
64a9c71247908c94cc029fabb009640306aa89ea
F20110109_AABGXW kim_e_Page_135.tif
dfec13b308286a742e08cb0958fcb76f
0095da1b2e8abaea4d46afe055abacc58aad729f
17342 F20110109_AABHEF kim_e_Page_113.pro
e0f6a77908077ad7e51d4077d991c6f6
6c84402adf860350b568a5b8dd8db064ca9c1bcc
F20110109_AABHDQ kim_e_Page_127.tif
263a0fa1a3e2c0a85f082ac71139ff2a
58f8d688bdb4837a9bcae9660b6b4a3a3081dc3c
43691 F20110109_AABGYK kim_e_Page_017.pro
dc45900a363694b87dcbca556cdb9bd8
351195f2731712deb47558fdb91ab4ff9f2976a0
28466 F20110109_AABIHI kim_e_Page_125.QC.jpg
0d26e03fe508728b25c6d02a4729a320
1e6bb28b1bdbe5ebdb90977b89be7ab84bd7261f
51543 F20110109_AABIGT kim_e_Page_114.QC.jpg
f42df319efeb52691293c721a8b8e4dd
ce2f8379eb25a8380bad25eba3c5734acbc01a74
163295 F20110109_AABHEG kim_e_Page_020.jpg
1341bd242265064a4a9065cadacdc77b
158daaa89fc6b147e71d04199515f1c7797da46f
F20110109_AABGZA kim_e_Page_053.tif
e85fe77e0c7d24974276485ac0bace4f
a5e0451ba5e27660a19763614417591c2f7ede11
99458 F20110109_AABHDR kim_e_Page_064.jpg
4197e8f02e55f810c04ccd34548717df
4c3d5f08e89aa01baf2250aa53a6694e29e82303
78825 F20110109_AABGYL kim_e_Page_015.jp2
5c74e0c0ea4f91946dd2eb8d72de7c71
145b84d70cb4ef9d4c43f5cf72352e95e38a52f0
21734 F20110109_AABGXX kim_e_Page_022thm.jpg
a925d27c1371d1004982e0586862fd76
908cdf53cf7046b63433a69563aab2cca559b11d
9828 F20110109_AABIHJ kim_e_Page_125thm.jpg
c46686b4dbc8feed9a148c52a1e7e09b
39ed97560e5eeb934ef5f443f36f2907bc8998bb
16045 F20110109_AABIGU kim_e_Page_114thm.jpg
c684c0d3b6239b62ae3d3b54a272ca7f
93867bd5e607484af1cc7a01c5b4d5698bb72eba
56241 F20110109_AABHEH kim_e_Page_026.QC.jpg
7b47fc7a29482f23d079253ec010c792
19b5ec365133c4d34c2dde38c2ff189966aa96de
145676 F20110109_AABGZB kim_e_Page_026.jpg
269f95cdfddbb7e8b432d40e9603ae63
4dccfd6f2ddf2fb28bb03a0fb7419825d7bcf07a
160305 F20110109_AABHDS kim_e_Page_015.jpg
1cd7ea526f3d3677a652e20f315add32
3f55c6615d690ec5e8f8eff03f13539465e091c4
22778 F20110109_AABGYM kim_e_Page_059thm.jpg
49b2b2a07c0696f47da45502eb2ce23d
3921156b7d568ac7af92878852b5103af2cbb751
23976 F20110109_AABGXY kim_e_Page_132.pro
6ee0de7a7ae407d9890f0e7b490d4ca1
5ca733aacd8837e4585d59901f9d1e51283f815e
38580 F20110109_AABIHK kim_e_Page_126.QC.jpg
86f0469db6e51d3fcad2f68261555f65
5301b2709253ba05d6f6f4f0f9a65c7f503e0a46
14349 F20110109_AABIGV kim_e_Page_115thm.jpg
ec9d7f8f0b767ed3564f8bfbd33f2ed9
3c99babdb2f5db851d515407adf4143ac26b4f74
211432 F20110109_AABHEI kim_e_Page_050.jpg
eb486859b4122462dba0660302fd6da1
c624aaa2764fbabb3bab784cc99e337913bda5a8
29287 F20110109_AABGZC kim_e_Page_114.pro
2fb824d61783d0c6e0e251221d6da251
2b3dbdafb6e55ad8d698123c2a46f750447b907c
F20110109_AABHDT kim_e_Page_018.tif
80563f75f21c49244102f3579acc8815
18b2323ad5298c19df3cc2d8313206168ff68ee1
54108 F20110109_AABGYN kim_e_Page_039.QC.jpg
37e88fcf8b2e6327be5c93f5b3640bc3
4e7ac6d6e35adfc43dde1be26ec91c46afda0623
1065 F20110109_AABGXZ kim_e_Page_109.txt
ffe17f8eb4bdca934d3b23843933719c
455cc56c39467660acd0a70e71ad0ab5d5ff0b4e
13242 F20110109_AABIHL kim_e_Page_126thm.jpg
d82ef6217e0600a0b5b985412e07e72d
ce097fa2b8c9fec7c1ca8bb88c88195422be96f1
21074 F20110109_AABIGW kim_e_Page_116.QC.jpg
845c6d6cf0408e4b5e7716454ca68c9e
4def26b6b272b75fde36603b72a8949805f8a5fc
33618 F20110109_AABHEJ kim_e_Page_004.jp2
2b70269b56ac3d924c9f7b4001ddc6bd
25d9646e4837bfd39469a904917944803090e9bd
43184 F20110109_AABHDU kim_e_Page_036.pro
8220662c1ddaf009335ba24c984a882e
4bb0aed866f771a6e167393d553ddadb646178f7
40765 F20110109_AABGYO kim_e_Page_060.pro
2f706a42ac6ecd4d994d6d34f8319d2d
6231ae31fd13a683c81b4bd3dd5493c1a8b9a8ac
101840 F20110109_AABIIA kim_e_Page_135.QC.jpg
ca3b43cf763959b0e221e394c9b82396
80256888691d1499ca9f821432cfbf6e963652f7
44294 F20110109_AABIHM kim_e_Page_127.QC.jpg
5bea82028beb49c059758f190087bbe6
25dbec04be302deb66f313cb3fdbc0bfdea1e26a
34944 F20110109_AABIGX kim_e_Page_117.QC.jpg
bdf5efd127f272907b9e5cd3bc0de6a3
f3a90bcf0cc30ccd4c2f7465ff88ccc241725cd8
729590 F20110109_AABHEK kim_e_Page_067.jp2
fcda879de0952463d52eb824035cc468
a9ddb3fe12b7f3fe20aec914adb0c16f4475673c
16341 F20110109_AABGZD kim_e_Page_069thm.jpg
1c46f80410baf3a0121d5112b1150702
43822d54cd1a20c54643669ce1d7c5fadf402508
21891 F20110109_AABHDV kim_e_Page_088.QC.jpg
591d94a587879f9a0177e4f019693eb4
1731d66de8a6539f5e6928cd948ad80f8851d44e
51884 F20110109_AABGYP kim_e_Page_135thm.jpg
29016e29e7c90105fe9136ca0582f300
1b0172bbf8a6f7ba6350df893456839087cb755b
75234 F20110109_AABIIB kim_e_Page_136.QC.jpg
b0e1b82c2a7337b4c9261ad2391438d0
eb39e3ffe8a5dc1736be2c331c04fdd8014b522f
15128 F20110109_AABIHN kim_e_Page_127thm.jpg
ffb46992cf7c4697eec03fc0e85622f2
8aa1024911ffc99dc0223a54906baf39eabe07b2
12225 F20110109_AABIGY kim_e_Page_117thm.jpg
86eb32069a6e58ec34a6e238f1614ca2
32b820ab57b17cca82eca3834f4a12f36fc1eccf
1535 F20110109_AABHEL kim_e_Page_037.txt
1f20078c584755acf63930ebda62f6c4
f4983b25d703c90d5ba0d8c090cfc804e1dcb019
7794 F20110109_AABGZE kim_e_Page_116thm.jpg
3b315e6b55bfac2ccd8acb8516d39934
68f314ffb2351010fd2b1fd10bda2ee1e0a94dea
8711878 F20110109_AABHDW kim_e_Page_051.tif
cb0ca7865c0d63e4475c85faf20de675
c35569d406eac75221f9f8ff1e5918a22ad7b811
F20110109_AABGYQ kim_e_Page_094.tif
14a126c1d829a472cb5eace9e1b99db4
fb0a9c96481d971ff07026896521a24de1c88683
42655 F20110109_AABIIC kim_e_Page_136thm.jpg
606a17d421833f5aa2fb0ff496d9fb0b
8a92657c270efdf7f7f4630c4270c998670d2f56
13300 F20110109_AABIHO kim_e_Page_128thm.jpg
d1b9085ddda67152aadff6c6d22e211d
6e082bbe836351f5b8b8ac744bd1d3216e14bcde
51634 F20110109_AABIGZ kim_e_Page_119.QC.jpg
2306d664fc28503d290d600886ef6f7c
360bbacae738260c10393f68bea6d68495139005
F20110109_AABHEM kim_e_Page_036.tif
a639d741b2d86a530612bc83d2506b9c
84787e92f1139d358edd390442c5c5e68befc509
2031 F20110109_AABGZF kim_e_Page_050.txt
ec1b7ee77061b73823e4a830f03332a6
5eb96da3608d1b4dae18976eb4afbe9f160940c5
5516 F20110109_AABHDX kim_e_Page_002.QC.jpg
ba4c02fcb7324adf960613dd044c712d
99d54aacc3175d85b301290a7a92460910578868
64214 F20110109_AABGYR kim_e_Page_109.jp2
bfd9caee91103de6d5d0d8341468f9bd
89364ae4a121e5f156fefd71d8f496914690c68f
105484 F20110109_AABHFA kim_e_Page_097.jp2
d00af014fd117fc73f206c3c84816ca8
9062c77941b26f4e459c2f825c9c1cf4c465bb8c
18220 F20110109_AABIID kim_e_Page_137.QC.jpg
70d42856f2a2396de1f90c70ddf9d7e0
23f3f1c2ee435d1ac50598fc023253e3a95eb6fc
1259 F20110109_AABGZG kim_e_Page_065.txt
1823d101ff675f71d8074110d3f17135
cf5a51d4a9a598dccbfe46bfdc97c1581de27b32
43326 F20110109_AABHDY kim_e_Page_079.pro
4021834cc9c3b4817f9330f98b72486b
095f90b08609e507ac6d1297549fedbefe1f494b
78534 F20110109_AABGYS kim_e_Page_095.jp2
0f97122f45aeb035a967b2574e0edb6d
1594ff8f594b404573fdee09ab50df2bffb70dab
1647 F20110109_AABHFB kim_e_Page_018.txt
de70507429a8e351bdf58c9029d22651
95f12ffb1c11d99413b2df9620e29725cbde2cb1
156078 F20110109_AABIIE UFE0000552_00001.mets FULL
e75354f5db3ebaf73875e17da0c17c05
6852494669252434cc795a89398ea7f93eae943b
37535 F20110109_AABIHP kim_e_Page_129.QC.jpg
cfa8c6f2568a71693aa3676ffeaf812c
60bcf599ddc276af28c66f906c3062c3a65c6be0
148060 F20110109_AABHEN kim_e_Page_114.jpg
30a32a0022c9afd3b64a61caa2908ce7
4fe3081887d04bcead3e9d8ac36cc9fbbdbb05cb
97503 F20110109_AABGZH kim_e_Page_025.jpg
e231cac998ab578f573385a3186799f8
9958beb6f734ed01b88f79713e624c89c9d79fa1
F20110109_AABHDZ kim_e_Page_028.tif
b08160436f2526f6e82581936618eeae
25a6e3f85e73e2b108648165073152ebdf4bfc33
1492 F20110109_AABGYT kim_e_Page_072.txt
07389a41e486b19091fd4edab48e8a07
92d253d643459d6e2f39bc9571183e8485b56a4e
36387 F20110109_AABHFC kim_e_Page_048.pro
8bd8b74abc329cd1a24fb3533498ed6a
b7bbfc9dfda0c3e5fb0edd4bb1b0ee61d28fcd1b
13123 F20110109_AABIHQ kim_e_Page_129thm.jpg
bf102ad6ed5d3ca21f685d35cb3f5095
c1c35e2a634fc00fddc1762fb8c82c9f4dce18d3
25231 F20110109_AABHEO kim_e_Page_093.pro
edc080475ee3804831b5cc81394fcc99
a48e9b08bd94e0f69566fcd08c9485e186b55bf3
186260 F20110109_AABGZI kim_e_Page_089.jpg
81de155e86833291c7802d1b6f1b6a4e
edcc9210d75aec18574f06902a87a7670287d98c
28817 F20110109_AABGYU kim_e_Page_067.pro
64e0725194135621e42bb70aad6dd65d
2821d732248522c26d7639e83b525a53bf6e25f6
63008 F20110109_AABHFD kim_e_Page_018.QC.jpg
4429e1ce93ee76c902644e388ea19ee3
7a2e7d1728ef38bb4440f0f3aea8f296fd316087
12991 F20110109_AABIHR kim_e_Page_130thm.jpg
ca701c83bc2fc4031a3120fb1f809a54
5c38e342d6812527a979963014641c40b04106b5
F20110109_AABHEP kim_e_Page_103.txt
93667a75dd6cc7b6940d6a288e080f96
e70da62ab29b4d9551640d3701392eecbf8049cc
81015 F20110109_AABGZJ kim_e_Page_078.jp2
7272296d081a919f8daff0ceaa67cc22
78011859be02d58aea035394660a6da6095db0b9
F20110109_AABGYV kim_e_Page_122.tif
e31843fb843da47ea11c3401f46fceed
ba6590b88598bbd0d4faf03c0582352d47ac706a
121657 F20110109_AABHFE kim_e_Page_006.pro
479a07e9e94bb91139dff7299c6d9fc5
c62b064a0721afbef24a05047dce5e993e0edfe8
37495 F20110109_AABIHS kim_e_Page_131.QC.jpg
938e98affaacbb554205b35e90f6f22b
15537b6061914cad150e1564d056537b7c6636e1
85582 F20110109_AABHEQ kim_e_Page_043.jp2
8b30ccb06e2ecf11b9e96a1b7c67bec4
2cfc2c0a2cb2191bd2c0d8269de07ae83ace6051
175035 F20110109_AABGZK kim_e_Page_028.jpg
64471ef9df9507d84d517a570bdb2af8
5ef13bdcccab9235da42bcb975f34cb31cee8db9
27475 F20110109_AABGYW kim_e_Page_067thm.jpg
96dec3173574b258ea6578cd056f8ad6
66077f34172b2a3a2c8c6af404a109a4eac7988a
978 F20110109_AABHFF kim_e_Page_010.txt
e475f67dbf09c5fcaf19e26fed1fb1b4
5cd5c8edadca3952afb1b9d72f7955a47e14aaa9
12561 F20110109_AABIHT kim_e_Page_131thm.jpg
8542da56ec574d485c9cd09d0909cf73
b224c7777159d1fd090744d5ef46fa15b9036a65
F20110109_AABHER kim_e_Page_101.tif
13df6b84ab0ffe5e60c5699b9cf0b587
45e3f8c4945727b10f1cf80fa5a25bdf4c017db2
16066 F20110109_AABGZL kim_e_Page_120thm.jpg
36826faf81d5f1772e15719c71a5f131
ee8565fcb284114bab7c0912769c506031a14700
46834 F20110109_AABGYX kim_e_Page_057.QC.jpg
0d4e8f4819d489d0d1698b1ef997d1ac
7084f680e98652f282faef9fc758ffcc68a9ac25
F20110109_AABHFG kim_e_Page_031.tif
e0590e698ab4eb67cb9fc2d73ff9559a
9fe9555ed6c37e8ca0939fba5c89ef4c9bd2e87b
42194 F20110109_AABIHU kim_e_Page_132.QC.jpg
313076d81653f93c468169e5f861013c
19f0a3a033aa6593b5e1c91dc250b2ce553d37d6
69397 F20110109_AABHES kim_e_Page_121.jp2
1f3c4c942585c09f11d97c9a9c8c076f
2d94a11f3ac3111a7a686f0fa172112ac6761725
46338 F20110109_AABGZM kim_e_Page_044.QC.jpg
d86a0360690dad1be7824f7682ef35c9
41db9e14681e71a2e42aba436d137124c7a83ed7
112325 F20110109_AABGYY kim_e_Page_128.jpg
cd62b67942c8c02b074f01fa7ff810a3
9609e28802aae18d8cc1b7ab267f4405c73d36a9
202249 F20110109_AABHFH UFE0000552_00001.xml
d346b4a03f778ed1e3e30b5f8ecbd750
2973256022a66c774d7e24d5c844c9f962a9b5c2
13677 F20110109_AABIHV kim_e_Page_132thm.jpg
715747eb88ab8a4bd5811d3ef8514d7f
f1ad4b1086b82ddcabeb5c46c45706e1358724f7
595353 F20110109_AABHET kim_e_Page_010.jp2
13aadbc24ea52ec6ade571f4273c2b31
e4b69fe5f62b97ada4e88b545bbcb565f1d93717
58888 F20110109_AABGZN kim_e_Page_115.jp2
a06a207b366269913cbc2330fb0db956
1dd6b9e97437809156b14b3fc48d70591b5f3680
22543 F20110109_AABGYZ kim_e_Page_019thm.jpg
a443c9a97c4a9a748bf550907e40f6fc
d61eb6f1417a2a00af00e5ff2d67ff89fb6c3766
6887 F20110109_AABIHW kim_e_Page_133.QC.jpg
f9b1189f67b62ac378be49e83eda714a
33626d55a2585d93675d379c571a82c9ea17a551
F20110109_AABGZO kim_e_Page_064.tif
cc1679ddcf8936eff0406ab663b2e786
acda053624796368ed63dda8f928359501adb2fc
1517 F20110109_AABHEU kim_e_Page_075.txt
982bd38b8cf264a0666dc2f09a78c03d
61b8d8f41ef9f165e5a8f9068c2fb47fbf3bcb72
3341 F20110109_AABIHX kim_e_Page_133thm.jpg
6bdeacfa219a566301d9cfa2a8405862
2efb224955ab0e822ce0fba18b6ca191dca5bf87
2165 F20110109_AABGZP kim_e_Page_055.txt
253479e2871586901bcf64ca0d23a716
94721832022f86fa730c922a4f0c3000c6f01c91
16448 F20110109_AABHFK kim_e_Page_003.jpg
855f902cc2229b6d051aff594e2efefe
b0d3d7c6cea52706de4308b97a5fc1dc6c53f26a
205870 F20110109_AABHEV kim_e_Page_091.jpg
b7be48f46562553955ebbf5c8558306d
deda4b0cda8e4ffa16a9bd6b247d19696ecdd093
101323 F20110109_AABIHY kim_e_Page_134.QC.jpg
e74de9e97c4ca1a0fdfa2307018d1983
dd25f0e0df1be1ddff08a1ffdb7bab8e4684361a
36211 F20110109_AABGZQ kim_e_Page_010thm.jpg
c9afc16d1b6b68394965aece99abf121
097aacb4e7052549877d0aa4db12afe0842520cc
70513 F20110109_AABHFL kim_e_Page_004.jpg
c1326aae797214fb70cc5f73b2eb90e3
06121d10ef3727083baf86652ee519bea5378fb0
10139 F20110109_AABHEW kim_e_Page_124.jp2
b5d7f8a1b1709b4b485b513509bb3932
fb966e7d7fbe30cf7d6e8b75df6c567b54ab1ac4
50348 F20110109_AABIHZ kim_e_Page_134thm.jpg
99b664eccfef80766ad6f4250bb63a91
188dcf6febc36a64544445d7d8862e25f3ac4fd0
22258 F20110109_AABGZR kim_e_Page_023thm.jpg
8509f5e80f2479515880f4941666e5ac
e5dc6e379d60b38bfac9d0a84e1c8eaa11387749
175908 F20110109_AABHGA kim_e_Page_023.jpg
22f6e3ca520d32fb24aad31d5260e402
605705d7e2f6fe4e6be727505e9286f5437709d1
249074 F20110109_AABHFM kim_e_Page_005.jpg
78241ef7c4171f360f6b85d0433437df
16d9a16661e3c082fd6b666c3d38b29a65340271
F20110109_AABHEX kim_e_Page_109.tif
c772da06971ff376cca747d22e62537c
a3a987e28c860cda12fb77b2c4ec017dd1cde371
35518 F20110109_AABGZS kim_e_Page_046.pro
a4fb5bb366c3b61de570265cb148602a
4b9c8b2aee4a6cfa92c5cc780aba01c1bc35e926
146607 F20110109_AABHGB kim_e_Page_024.jpg
3c12afb00a3941a1fc89fce6740bb596
c3ce50d765a95922e6ea0afe235c75885d9ab8de
370652 F20110109_AABHFN kim_e_Page_006.jpg
d1d7ab718609d02116cdea4aa8c1cc23
d9800ed30f9187d8e40fba4fe21e41498a5692a8
25885 F20110109_AABHEY kim_e_Page_051.pro
4440827f3cbd1a13ea23285f6dd6781b
3810ef1a7e9260912de38202bf07f3bc73730cf7
F20110109_AABGZT kim_e_Page_038.tif
b33c17175d1b93baefbd30f90308ff4b
55189de5010d7a9e14f2d4969f8f8be13ba9fed9
172422 F20110109_AABHGC kim_e_Page_027.jpg
c74c2deb13e327f812cb7c4b0e690bed
ac6363b7b59adadb8663f06e851e7d4eb985f8e2
192159 F20110109_AABHEZ kim_e_Page_040.jpg
5f63720388f7e5b780f4a1b5d61c7236
afa569645748881b436f7979cc97d8af4edfda34
78531 F20110109_AABGZU kim_e_Page_097.QC.jpg
71081d867c2900c4c016cba37022bd01
f07fca8ccd7f568cb35dde420afba596615c5edb
174450 F20110109_AABHGD kim_e_Page_029.jpg
7835b180fef7b38fbd6327ed69ad60d1
d43f5b22a3a34dc6b313036ad5a8ef9b54fe4c21
336449 F20110109_AABHFO kim_e_Page_007.jpg
61115a12530b8ef7c5fcb38395661163
c8453e4ff511d5da31365084643b2fc92bbd37e5
56697 F20110109_AABGZV kim_e_Page_093.jp2
a90fdace17b57be034d3a6ff87a4f13c
6a71485dd1fba2d12f3b7e94a396575507aab71b
172336 F20110109_AABHGE kim_e_Page_031.jpg
c7d0e36001e7347ee873d07f19e46783
ee98bb151823b92c1027cfa8435c981ffbf5842f
112247 F20110109_AABHFP kim_e_Page_008.jpg
56c752196f2269146c42c221a5f395e6
ae7af77d6583cf6663303ae3b12b4b6c0f729822
14528 F20110109_AABGZW kim_e_Page_002.jpg
4520a03b8653567605ee8091bbdf882b
cd6fb27085580bcd955213a3bb7ce6a339a79409
140916 F20110109_AABHGF kim_e_Page_032.jpg
1d3379c4578d036c524803e84342a062
54ae83cc051d7d90b1899f30ed910d02e86de075
84269 F20110109_AABHFQ kim_e_Page_009.jpg
710c3f35a6b3368a5b16c501d9398a8b
c9822274eb6a9fa7b27eedc7ee487024a2213cbe
72593 F20110109_AABGZX kim_e_Page_066.QC.jpg
0639c1aa6d55ebf5e971917cc327363b
76179da056c5638aad802254ae4dd1eb1715fc9c
153348 F20110109_AABHGG kim_e_Page_033.jpg
2bc283b27bebea236a145f9e3f35eaa3
6615c92a9ab5385cab05efcf2f6f5328c81fb8f4
123844 F20110109_AABHFR kim_e_Page_011.jpg
b06c17ca7f098f563e75844e42165713
694f3ab31e8484a90f5bdaecc2e1a18319b38c0c
1640 F20110109_AABGZY kim_e_Page_048.txt
1b937ae0f81921293d85395d46d9a957
85b42b86ceefdee03d7398dd16cd45080359e86f
176564 F20110109_AABHGH kim_e_Page_034.jpg
37f1ad538309377d9257abf6f39eedbe
806d9a22d904c5d7210408dff5bead8a3a26827f
163824 F20110109_AABHFS kim_e_Page_012.jpg
5604b11d4da925ed5a16f086e97e6c24
2ca82b8e62fd5adbc88b21dbd0285e15ad87b550
40468 F20110109_AABGZZ kim_e_Page_115.QC.jpg
c0778dbc3246c6a553cb4392f78d528d
304faab81fcb9761cd7944f785c98d8cd505d134
167120 F20110109_AABHGI kim_e_Page_035.jpg
b83c4e37abb372e6cfb7b6123e39b4d1
d837a1f639ee173c9550d0861ac8812caa25755d
94979 F20110109_AABHFT kim_e_Page_014.jpg
f34880fb8172b5946ff19c72a5fbd471
a446994b325ec2dc0b3b3407d7add997889f280e
166707 F20110109_AABHGJ kim_e_Page_037.jpg
25ae255cedfe68dae310c85f2d290acd
23f59c04f5c896b331492e1b3a4bd65968eca139
168974 F20110109_AABHFU kim_e_Page_016.jpg
120e30c5bc1b759a808debe0dede0303
3641eede548f7cec52c62efa297dc491458a4a8b
184331 F20110109_AABHGK kim_e_Page_038.jpg
7ba2544ce8e992aca6446970a981c317
c92af1ec86dc6f3b59fbb7adaa1b55fe3fbd0eb2
181223 F20110109_AABHFV kim_e_Page_017.jpg
f235e7ac3b1c6984ebdb0c41228b5970
91b7a710663c749f6bfd4e3bdda80f9b832a0658
141775 F20110109_AABHGL kim_e_Page_039.jpg
113a9a8cf374c61a098de5b3682e3a31
0f281794ba16226a3fec56b832bee861dd78a6e9
168534 F20110109_AABHFW kim_e_Page_018.jpg
5ec5c20471b3ba06d9530886775fae2c
b1fa669a2bd27c55fe923cd0099a79a2baceeee7
116058 F20110109_AABHGM kim_e_Page_041.jpg
ba9bd64ef83509d61c7ef65ad62e4a63
a7fa6b77af9e19e33e817d440947750edc87fb9d
193611 F20110109_AABHFX kim_e_Page_019.jpg
1a4ee6a8f99cfdea6b891b3e76b4e49c
262819c321b84a9c54a4a389f76c05720465fceb
144714 F20110109_AABHHA kim_e_Page_057.jpg
d1f2f5c69915429d32ff0c34c3bf2a1c
e284012728c94fd4d9935578c9d0d84ba9e4165c
177284 F20110109_AABHGN kim_e_Page_043.jpg
c89e6e13fc15c0707445fe53ba06c776
a1e32132fb7852f2be2f73959138ddcb80ecf379
177399 F20110109_AABHFY kim_e_Page_021.jpg
3ef35cf6c7696a5c3d1dbc2067373a8f
d16961ff2199fb118e27de2797900700143e0505
127095 F20110109_AABHHB kim_e_Page_058.jpg
b7c7d9745aa56979216d601f803e0d73
667feead374d97866cf1047a32b0e959f8d0d3bc
149487 F20110109_AABHGO kim_e_Page_044.jpg
8c71a596df0fa34e57219b8d7a228a04
eec8bf53b3b237c138d76d31b92c8b206a4f44a1
171136 F20110109_AABHFZ kim_e_Page_022.jpg
08effba57cf6a7321497e046a1778a04
9cba9a42a985bc890d5e626b8fa0350e7e06abc3
214662 F20110109_AABHHC kim_e_Page_059.jpg
b18c2542e6121997c8154e8a653b2097
0505f6171b6f0b2df8e681bb8f67cc08804556fc
124284 F20110109_AABHHD kim_e_Page_061.jpg
22c605a93f016b799d813319500bd34c
3bb3635792d5e2d038f8c9b24d35fa4cafef3cbe
142244 F20110109_AABHGP kim_e_Page_045.jpg
10ec06eca014cf500ea2be0afd0185c3
6abdfac785ca96c9d65cc52a28b2bbe588f6e996
92524 F20110109_AABHHE kim_e_Page_063.jpg
98ea2bbaafae440898a943976b1127df
fdb739b4f9c0a5c20282d0e2cfeea9a1ca9d8726
150035 F20110109_AABHGQ kim_e_Page_046.jpg
6e68cae3fbe658cfc42901cbf587792d
6f4641b80b38aae86189f3df63626bccce9a66e2
117518 F20110109_AABHHF kim_e_Page_065.jpg
1658b145eaf842a1573ce256c3bcd623
963a038a224df81cf990325202d99bb34e6d29af
149642 F20110109_AABHGR kim_e_Page_047.jpg
a7c7dc323bbeaacd73e7476e0a775efc
f6ec54774ec73336b3e0f9785e0ea222e257251f
172527 F20110109_AABHHG kim_e_Page_067.jpg
2d5128d350d7dc11da098cb38fd09ba9
f323bd5aa90fa3c8ea5c95888de9569414e344d6
171781 F20110109_AABHGS kim_e_Page_048.jpg
a8102f8e6f0cd25a477372457c0f8e6b
24333cdbac15b48cd76607bf6ab4edced6b6eadd
104215 F20110109_AABHHH kim_e_Page_068.jpg
bb73cb847e82d4a5d63be59cac596d23
d86ba82f2ec17cb44e838f5eb8a4692d4cdd64d2
108763 F20110109_AABHGT kim_e_Page_049.jpg
b61ea83f7f9f19d3b429edac4c4c1e52
b424b60c6e18b769d80bb5dfedfc36d67d63e8f0
157640 F20110109_AABHHI kim_e_Page_069.jpg
3c7ab5bb5fcedc359270ad0e42b62f7f
e6268e2f772465922d1b48a817d606bb39563cb7
142274 F20110109_AABHGU kim_e_Page_051.jpg
d8326348b44d05d964c49fc5ca31b91a
d221c5b95540dbf4cab7f0a5a9d194684ead62ac
152177 F20110109_AABHHJ kim_e_Page_070.jpg
6e1946a93e2a45b94b4d5ecaadbfd53c
05266b59ced7e95e21af9a6036e8153a77dbc195
144838 F20110109_AABHGV kim_e_Page_052.jpg
4a286ca1ac18a1b01f61c86a298f0068
db080b93e74cae47f79904e95f97ed980600af6c
127298 F20110109_AABHHK kim_e_Page_071.jpg
d03212d0371ca109fceeeb3ae1c470b7
7fc89271dcfc7aa66627997b5effd94ebe6c454b
185227 F20110109_AABHGW kim_e_Page_053.jpg
a657885febb3d266e80f0f66a977014f
3017e1323ebd4c9810cd7c91fff2378ef18425f5
164481 F20110109_AABHHL kim_e_Page_072.jpg
9f225e90e5a10b0257f9df216f03175a
87645322da09e4ed81c2726a9ac5037f129408be
196296 F20110109_AABHGX kim_e_Page_054.jpg
4aacbdd78a4578aeb2ca864384387e14
e34d1f86115f73551d96c6e7a24f8d8716b18665
113622 F20110109_AABHIA kim_e_Page_093.jpg
4ef33aa3808b33ccefdd6670e19bb162
b9e9b7b46afc30873efd5216f740d4761caf537e
187828 F20110109_AABHHM kim_e_Page_073.jpg
3c23871ad9d37b609471b17aca1dd178
5a7e4edf5089b58b605f413101de836cb9903359
197698 F20110109_AABHGY kim_e_Page_055.jpg
e385c17d3635c6cdeabb68c298e0fd0c
467e88dfee83416c0e9f2d073dbf4a4e54aab33f
159799 F20110109_AABHIB kim_e_Page_094.jpg
22b56f825c3edfe027c3a699ce70259a
b315a6bec0a4f22c95938d49731e1947cbe86499
178288 F20110109_AABHHN kim_e_Page_074.jpg
38d216de0cb92ad113bc39e7c234357b
01ed510559e19d84960b0f75fade8daf44df55ff
180824 F20110109_AABHGZ kim_e_Page_056.jpg
4d57633300a25c455cb1fe0bdbfbdb65
c932a9b6551de0b22ea9e103d4d6c37d6ebfe50a
159816 F20110109_AABHIC kim_e_Page_095.jpg
5d4537a4a49b1bfceafa0016c48a609c
4361f9aa571038331dd4c64dc6fa49d5f6c3e2bd
148030 F20110109_AABHHO kim_e_Page_075.jpg
29ecb79eda2dd55a63d523aad29f8736
6e4212f98cfc17dce9ab1631660a643001672936
164322 F20110109_AABHID kim_e_Page_096.jpg
7ca65b194a2286217f5dbc6b538b25d3
ab47d3a6b2a9536a2ab6d74626c8e98121aea42e
48297 F20110109_AABHHP kim_e_Page_076.jpg
dfe35769f3880ca8424d2327e58afb10
09f983fd77837765b609f30fa067587738cf5010
218868 F20110109_AABHIE kim_e_Page_097.jpg
f10af517269a150c6f41eace0250713d
134cf9fcd9ddcd6facec9f985c5808977efeb15b
144533 F20110109_AABHIF kim_e_Page_099.jpg
1828d677f344267be3784f722b83a7cc
088b02018c1d14d81524833475ab6aad0b6f01d9
182453 F20110109_AABHHQ kim_e_Page_077.jpg
7de4262bc1d72778c1a44542d14e9638
45dc7ed979366318a201ffa999bd59c4bf0f7e9d
65881 F20110109_AABHIG kim_e_Page_100.jpg
b00a76734054e7a1546bb6e1b1618007
97a7b57fb378ac827f399844c728e017ac9995d3
173071 F20110109_AABHHR kim_e_Page_078.jpg
59b35900a24fd07382df51f4fcdfd6d5
44f7c8f30a25f2e577feee56d68d9ec649176709
122352 F20110109_AABHIH kim_e_Page_101.jpg
8cef19e7be90f13dfec19f918e9ee726
ebd511a4696d723f94978decc919c7927f572dc4
182494 F20110109_AABHHS kim_e_Page_079.jpg
e7fc43eed6100d093782e75a5454b042
327ea1e262a2ea44ba9e6c622a51ae33d90644d8
107098 F20110109_AABHII kim_e_Page_102.jpg
2be27144628073f472633a6d6588e3f3
8c2abe03fa647320bd88edd934f7bed452a3529d
207187 F20110109_AABHHT kim_e_Page_080.jpg
2e87cb72dedc3495afdde55d87216391
1a3bac5576d52e98547be175d42e50c149422952
30970 F20110109_AABHIJ kim_e_Page_104.jpg
edcd3f1a3bc3b234df1f99d7482b7682
26d5320a53e79c964c2d603d1c9edf7c981bf79c
205546 F20110109_AABHHU kim_e_Page_082.jpg
8ec4f2b294545a6e0de84a02149e3fec
742f173910864ee2e5c57a55c3e393a81c870232
119575 F20110109_AABHIK kim_e_Page_105.jpg
9a5caeff2bcdcb35938128e697c15733
77bf2f221d594cb46e570bd961d50b2d0d62e3ca
159998 F20110109_AABHHV kim_e_Page_083.jpg
e2e342cd9c81f3ce38b1a567fdc709b9
f8a8a17ce5ead279cc7a8f67e8fdd26161505c0d
95744 F20110109_AABHIL kim_e_Page_106.jpg
644b536745fa179bef9948dd62c34ff8
bbf6971c2411d07bd8ef9f59008f8277c2a5616f
168289 F20110109_AABHHW kim_e_Page_085.jpg
b41d5b328dab43c96d7fd442908644dc
8f56b5205d31a88fc128fb785fc6d37947ecceb1
73467 F20110109_AABHJA kim_e_Page_125.jpg
dff0f5b8b46c2f7b0912f95c684d10b8
f2b0636b1b5afa5de2d5bd5fd7f1af01cc63b3c9
127097 F20110109_AABHIM kim_e_Page_107.jpg
511f94eac122a91afbbc16e7444b3ac0
1c26a94d00c3626e56174821dad3ddaddfffc61c
226939 F20110109_AABHHX kim_e_Page_086.jpg
7b879e0090c342334cc4c04195917864
8bcd1bd64daff7374908f1489a8f90827636d834
126715 F20110109_AABHJB kim_e_Page_127.jpg
28729d8404620984ace12f4758c26050
1cabf2f5483473670bdf1b8544a219485943e680
130319 F20110109_AABHIN kim_e_Page_108.jpg
7ed9ad4e53a158a400358756a6effe21
bd19c6b6d75214ca7aa4264385b5d1e17201e2ab
194896 F20110109_AABHHY kim_e_Page_090.jpg
c4424494085539e8c3d82ebac67aeb15
184e504c09fe4125981166909b789e37560d2a7e
106033 F20110109_AABHJC kim_e_Page_129.jpg
2b1aaa534b7c1f95dd7a4bf976aee8d1
4bdd922b23bb4ca1f71966a674c51cdc8617b23c
130642 F20110109_AABHIO kim_e_Page_109.jpg
06b115502113c1c5d58dbb404e666d72
ea46ea3421515b4919455743a09b04a6519e6a23
210453 F20110109_AABHHZ kim_e_Page_092.jpg
84b883197b8d07c72cd2d55eb4195b1c
adca872da1c63855fb9967a410291f5b281c9acd
102073 F20110109_AABHJD kim_e_Page_131.jpg
9e3806632cee78533818130f3f093ce5
12e4940101144f9afdf8bde492f3d634cf6937a1
118025 F20110109_AABHIP kim_e_Page_110.jpg
66c22e363ef418447ca827b57aeba33b
fd0a81887622110cfdae77e706560e2f1b739ece
122580 F20110109_AABHJE kim_e_Page_132.jpg
88342b2912746304ac3867440f381ffd
2cf2add214afe671ec953bcafba9ddfab1ff4774
97309 F20110109_AABHIQ kim_e_Page_111.jpg
f9b32198353e39008f7f8a67fabe2994
2af92ed43c98f52b3caabe10ee6aa30ba53e4637
17129 F20110109_AABHJF kim_e_Page_133.jpg
7e4b1dc40f341e41db0a951c37ec469e
be51c23b6c7cfca39d8810564ae23c02f58d2c0e
258480 F20110109_AABHJG kim_e_Page_134.jpg
99fb8af259ff4f09aca9f6db2167fc46
a312cef599b879bd08e4a797f7b739f120bb8588
56054 F20110109_AABHIR kim_e_Page_112.jpg
ff65f4698c8c06527c3a3268ee070685
cf1d493bcc9f0558323d520d0af9152c4b6ac855
256746 F20110109_AABHJH kim_e_Page_135.jpg
b15b46d53dae803b345a90d9274724ea
80ce5bf5c9772d25c8cc8da1e8b834cdae8521a2
93718 F20110109_AABHIS kim_e_Page_113.jpg
32611af6b1b1cbffabca3045dbb73bf7
eda5ef73c4813940275659a5b9a9030dbf6ffca4
174339 F20110109_AABHJI kim_e_Page_136.jpg
e35b3eecc1f4c85b008f97b6d3f2abaf
3ea33b6aea1d952adf6ad6615efb20c80d32c455
120056 F20110109_AABHIT kim_e_Page_115.jpg
462fb46a73b3bb67dec54db252f0cf44
74ae881a246f7a497c19644d69a020c6b704822d
47592 F20110109_AABHJJ kim_e_Page_137.jpg
ad84426aca2614f89e8b134c450b59f2
21ba0184b467e5715aa3d418614018cc887e9a94
58341 F20110109_AABHIU kim_e_Page_116.jpg
778a26cda3a0a1328567a784b58bf8d3
e3b2ed9854e57c9abb578a255fd4912c9a3b1436
21897 F20110109_AABHJK kim_e_Page_001.jp2
9bec6206faf11b8ed81e0c557e433f0f
a99d43b5acf1e47cab77c70412bbcb8c92a0ce8e
95883 F20110109_AABHIV kim_e_Page_117.jpg
107b06df51e601d2fa3930203318a890
104c57ca91289527321f45707e859f5df8061acc
5338 F20110109_AABHJL kim_e_Page_002.jp2
7bd469e60002ba119562d7a59aff828c
fd361aed5a3d34253463f8aac0a74c8618ad2f3c
143753 F20110109_AABHIW kim_e_Page_121.jpg
a040f1ca181ae2c3818002eaaf960edb
818ca5229306736670783bb14469bdead047e2bc
1087889 F20110109_AABHJM kim_e_Page_005.jp2
8890bf362adf8767a227ea9cdd9de20f
a5cc223f022901aca827bfc893f38cd5b30873b4
129612 F20110109_AABHIX kim_e_Page_122.jpg
cbd0648e2dbd9a95e4d231c4e3ea4c66
9c3c36b2203604e1c6f4f1355d990e2da6356c91
88224 F20110109_AABHKA kim_e_Page_023.jp2
ccc80f53762116ad074a3a8ed154fb9b
036a162b5bac954b246c05c64669f52382ba5e12
1087891 F20110109_AABHJN kim_e_Page_006.jp2
8b0e830b9de09cc883c0f639b5b697bf
4d7e617b9c8acb5cd9bfbbb75b4cb4e4db334ffb
80961 F20110109_AABHIY kim_e_Page_123.jpg
627c3f7187c331b663a74d66ac0da13c
27211cf88fc521e0eaf40ff7695fd8727329b26a
77117 F20110109_AABHKB kim_e_Page_024.jp2
ce38c6883dfe6ccfd964c8e3479cb7c1
0fde295b1984438a0bac6a30d01b5f80c208ed0c
F20110109_AABHJO kim_e_Page_007.jp2
9b231f2efccf67154cb30cd3c335ee36
218eda6c1bd7bd383c0189e0451c64d7e85880b5
23474 F20110109_AABHIZ kim_e_Page_124.jpg
7d3b243b043eae7cae3864469aa2a67e
ce69c24aacae386bb03d53a44e7d7f7ba59144a2
50102 F20110109_AABHKC kim_e_Page_025.jp2
dfc21cb24ae289332bb1079819c68b2d
ac0b514125b0e16dabd95cef86046656de8c715b
623807 F20110109_AABHJP kim_e_Page_008.jp2
b0db8087235db0c0afee90f20d7d1281
cfc67ca61eae2a1d83f0dfdcc2a4e99e171e8c33
88423 F20110109_AABHKD kim_e_Page_027.jp2
d6cc471c6a96efd8634d5daf2b322dd1
3f068abdf05a1da97ee3c3af1c8ca2c416017f3f
366906 F20110109_AABHJQ kim_e_Page_009.jp2
a19a55e4ee1a1a2d9965d14b24e201a5
330273b640441bbc431393f9c5712f4286b416c1
87942 F20110109_AABHKE kim_e_Page_029.jp2
ac0851b9df61a40ea70930dcb23135cf
c6850b45a726c2c5cc918d075aaa0db7919836bb
59636 F20110109_AABHJR kim_e_Page_011.jp2
8b46413406a490df1223ad0ff3b0c543
3d1bd5a74bea93523430de32c2ba20e831e0a120
101882 F20110109_AABHKF kim_e_Page_030.jp2
c10b0c4b7d7cf6a478122a029fcc1530
a80097dc9a48afe0b86d2411c71f53a5df326477
85568 F20110109_AABHKG kim_e_Page_031.jp2
515a66457f55159c6eafc23fc7445ef6
89380fe9997d96e49337ed18e124b83d44769a30
73547 F20110109_AABHKH kim_e_Page_032.jp2
6ea71fd22d8c9e55a15e1553fd73f770
03e1bcc2487d4e55e4b6d11fd99368ed1f7c7805
82232 F20110109_AABHJS kim_e_Page_012.jp2
e87ced69e517033d2cf3ab43f2fc38d2
8580ca7bedf764191f9625e49fde4b932c9013d7
75991 F20110109_AABHKI kim_e_Page_033.jp2
e02b713a71187f3fe26e83620e296a29
c53b8b01057565f40502ead11128422be2170e99
103018 F20110109_AABHJT kim_e_Page_013.jp2
e7d9114de1073fb1208ddf509cd95ea9
72d8ddac5325c350ae63b5783051b52cf317fa3d
86323 F20110109_AABHKJ kim_e_Page_034.jp2
d9ca2321e1845fc9e2fc26107359e0ca
cbef528363e7a230debfbd660b1d1b61e29b7684
48328 F20110109_AABHJU kim_e_Page_014.jp2
8a767cff7cd78aa5dc2ef69cd2e69e99
b30597c90d186ec81bb3d606d16e93fc3d353726
83202 F20110109_AABHKK kim_e_Page_035.jp2
9d9b6b7f87186cfdd1e47fbc28c1e63c
6e1bec130d4c1e94a5070aab23450cf3f34c8cd3
83691 F20110109_AABHJV kim_e_Page_016.jp2
5451a4da83111ad297ffae799d8f7638
804110ba9172756ce14f28b5383f9f3d52ef6001
101273 F20110109_AABHKL kim_e_Page_036.jp2
9cd5379ae2f4941884c3c28650f74630
fa461e9238d6507555952ae8b4226277cdb23139
85269 F20110109_AABHJW kim_e_Page_018.jp2
5b9c46b2f3683bab9b86f324ba42ee95
7f1c6c45bd218c3028c59296c9f2757977c76310
95256 F20110109_AABHLA kim_e_Page_054.jp2
bc682e86d67eef98cd1d0118ebcc3a0d
afdc18b1f79e1be95c9133e87ac9334e18ffb7d5
87259 F20110109_AABHKM kim_e_Page_037.jp2
e1dae3ebec557d1e9c1dd3fd5874294a
740d4702b78ca1b64fe74918c0f7e00d1611458f
95832 F20110109_AABHJX kim_e_Page_019.jp2
0e0812c84f44d0ba2c2c01a551e55041
d76d051eb469ee514b36b5fdfdf1d044159a9490
97405 F20110109_AABHLB kim_e_Page_055.jp2
8b10bb0797129fcd89adb8936fadb4bb
12eb47ab6faee3cd564097c37d60b8101f13329d
90811 F20110109_AABHKN kim_e_Page_038.jp2
09f3e8faf1f89501b23800b162151e79
bed22350c44d4f4f78513a34f1a1d14afa7e88f3
670069 F20110109_AABHJY kim_e_Page_020.jp2
7530675793369f402d0de6ab3bce9eb5
64c69c84f2ff10bdda53a2395817b0e7c67cc2e7
88460 F20110109_AABHLC kim_e_Page_056.jp2
425564972f2b18b1dfc2c21e67b133ce
726dc8048af0b920fc227a08b40de83b3f259241
95437 F20110109_AABHKO kim_e_Page_040.jp2
ef4e33b74e3d0ff9f618d49564c915a2
416d6dd2ee947ae89662d50b8f9b046133f01f7c
89312 F20110109_AABHJZ kim_e_Page_021.jp2
ace5fa85cd1df66b4f533001fdc99e63
716ecf0c4564e20d7753dd06bbb78db919bca780
72317 F20110109_AABHLD kim_e_Page_057.jp2
89fe847103c011f18c947725ec1af23e
0b59ef253ae3864aba71be01aefb19f8fcff0138
57456 F20110109_AABHKP kim_e_Page_041.jp2
e0a3597e0b69a6663daab648c122b7d4
7667589e4330498279dfa4ab540923fc49dd5287
64673 F20110109_AABHLE kim_e_Page_058.jp2
93bae5974640c77fdd118adff0047ca6
ce3a5284d75e1c4b240aada680d455154030ecb2
60233 F20110109_AABHKQ kim_e_Page_042.jp2
e241ee31d8e61bac7def1c97731c4e9f
94f1f9d9dbe19407052eb5223558bc1265dee774
104566 F20110109_AABHLF kim_e_Page_059.jp2
aa910d3703714720010d357d67610d74
ed2f09f54b70868e7334b6ff2bb7772e5cc225a3
70873 F20110109_AABHKR kim_e_Page_044.jp2
7db3ddb3ebce8579f25bff4685bca080
326d4f7fab74e6c3431d719586a08e5862fe4b69
88497 F20110109_AABHLG kim_e_Page_060.jp2
4d915aebbcaf4a66114433b54de1e3b0
19bbec790092e276f9381f620f20e73545191660
69093 F20110109_AABHKS kim_e_Page_045.jp2
7ed2cac3a49080874956507e795441a7
a8d17c8c9e2057b53c124de41e0dfcb1511cb6b6
538789 F20110109_AABHLH kim_e_Page_061.jp2
e636465d580cf8eae401a39e7fc4c1f2
fc710e3c8f91c655eaf6b0587e2cd366434646bb
82282 F20110109_AABHLI kim_e_Page_062.jp2
416e41c15acf6a27cbca891522b12f96
42b01e9532753c96463b3ebb053e3ae33bfc1545
73902 F20110109_AABHKT kim_e_Page_046.jp2
3187f898373f0b6b596194740b3378d4
bc2e23c8e2151fa316ec25a5b405d1429f74f16a
50198 F20110109_AABHLJ kim_e_Page_064.jp2
c7e3fe2b258f0c1823bcdda416679951
db13db0325d6a04913c3b4fb155581d4672d43a1
74004 F20110109_AABHKU kim_e_Page_047.jp2
a6430724f9fe40e07ce66bfd5073e206
5a2851a547c0f03580e93cbbdddc4e8b56cf5299
56535 F20110109_AABHLK kim_e_Page_065.jp2
76aedad4ae957dacc7036659ba07f5dc
16655154c143d5486d9a90e49df6dbb5776e5443
89915 F20110109_AABHKV kim_e_Page_048.jp2
0154f8adb6f8f5c541f7815abf7e0377
72232b5d8eef9a01a43b0eb63b20aeb14c060152
94899 F20110109_AABHLL kim_e_Page_066.jp2
dd2243f2ad1b1b8ad91ee3879b4333a0
6ccd63fb1c7fa7b0492577eb12630b51c03819d6
56293 F20110109_AABHKW kim_e_Page_049.jp2
8f43b1fc240d3cb14dff57be3a2cd77a
3ff7e86fdb839fa488c778cb99286179dbfc238e
53984 F20110109_AABHLM kim_e_Page_068.jp2
95fd553efd0bec9d9b9ecd6811659aab
66393d59d4a348d57dd8143e0ea6fcf329e916ce
102933 F20110109_AABHKX kim_e_Page_050.jp2
291dc10bc31d09eb8562ce85713474d8
bcecbeb374c416d8a0ba7e847987cd5ca1493f33
84629 F20110109_AABHMA kim_e_Page_085.jp2
d5a498d5fc5e1cfe2113ed655eae8553
790a42e659b37f6c2a0b776146f3fc688b9dd268
78423 F20110109_AABHLN kim_e_Page_069.jp2
23f716f446c6151f653dc3417c70753a
b58a647257f3af05b125c6f550e7ff96b5355a32
619226 F20110109_AABHKY kim_e_Page_051.jp2
7a6f8b29a7195d5038f09c511aed2b5f
2373f34600ab4f4db6f2f82b8bb62fa960294923
691940 F20110109_AABHMB kim_e_Page_087.jp2
d4317761ba6eb8a27871011dca19d2b6
b9ff4c95833a625a131d79888debb2d55bbc48c9
70759 F20110109_AABHLO kim_e_Page_070.jp2
6448ec71f5b1af477196ef81b1b993b6
b0d22b748d8d11553c69b5d89487a10c343d51c5
90210 F20110109_AABHKZ kim_e_Page_053.jp2
5d8e2b824deb9d2997ac9b34a510d29d
259058c592a6a5b867727124659d1d8abbabe158
29514 F20110109_AABHMC kim_e_Page_088.jp2
9c8a1157230b7ea649c8d7c7b7971832
8d9d8207b3eb2281c386e190e3efdff6fe32bfd7
63370 F20110109_AABHLP kim_e_Page_071.jp2
5275d8bb526120628bf632582632fd02
ef458b2ad06ca33790aa0d33a8bf08c615be9f67
92807 F20110109_AABHMD kim_e_Page_089.jp2
885e7d384695c9fc5bafa69671bdb6f2
5745066de68abd7c6c0d3b836878439774b28908
75587 F20110109_AABHLQ kim_e_Page_072.jp2
95c70bc88afd74ad2bd9894592856b43
169fc3dfcd9fe14fc9ff43f504df2e17da380b78
98039 F20110109_AABHME kim_e_Page_090.jp2
f13df305dff3944f267e3a3562cb72f7
aee4bde91933990a2f30623d6dd807a5700b124b
92504 F20110109_AABHLR kim_e_Page_073.jp2
88056a58c4767ca20950327ff0585df6
5b863a1cdc91ba440aa2d88c6257e01a46ae140c
101808 F20110109_AABHMF kim_e_Page_091.jp2
d5d70c4b904f29803d79ef2853b2391c
7b5de29d751f0371d428a6ae29c977cb191a8857
86687 F20110109_AABHLS kim_e_Page_074.jp2
565191a80a7c3f15aa05e125fd76e682
fa2035d95fe64ed3296db535349d1a336c2879ae
105259 F20110109_AABHMG kim_e_Page_092.jp2
9f15e29a34300064aaad95f115b8e9f9
5d9b1bd77067749efa5e6219eff642ce010a9006
74481 F20110109_AABHLT kim_e_Page_075.jp2
da29465f349e89f904b40cc1c3c1ac28
8dff786641beac8efdcf5b8baa1818249ee28c2f
81392 F20110109_AABHMH kim_e_Page_096.jp2
0592f5dfc08ed46bcb95218fe90f6588
d353545a7dd5619e40d5466b76995e1860984686
92379 F20110109_AABHMI kim_e_Page_098.jp2
ad2c910f78748598d8ce35bd300b067e
f4ef7dc50ef25606d7671a9b28cf234341a29158
21839 F20110109_AABHLU kim_e_Page_076.jp2
ecb2b8e94e7a6f2c0a8c9672e52d4771
7c9a0f4069c09132443d6bcebbae36e04a84e662
68445 F20110109_AABHMJ kim_e_Page_099.jp2
f4305b83700659a4d3477daf739e72c0
9fa819d1e3780632375fa4f7ae2d749dd1535a62
93114 F20110109_AABHLV kim_e_Page_079.jp2
ebd79b00fdeee2f62942059de6d68446
ef803f97aea60984909a54e2a657dd7ce5a04287
30500 F20110109_AABHMK kim_e_Page_100.jp2
ffc4773cc7f50e30eb1c37d2e37407d6
14463d337c7d987e05b062e174e1ca8abad7b456
104543 F20110109_AABHLW kim_e_Page_080.jp2
7c275043ae76d6df6e2bc2d673d3df2e
592dbb343ca7e888fea800cd9c2810f89edaa005
58282 F20110109_AABHML kim_e_Page_101.jp2
a899cf740c0cd14739ff03945fc8eec3
4acca6978f96d2e745154e6d9ba9111b3943bb25
102423 F20110109_AABHLX kim_e_Page_082.jp2
9c428c85b74918f0597b5e3575202019
649a215500246f7a15f6597490a59dbda78bcaf9
67680 F20110109_AABHNA kim_e_Page_120.jp2
2feefef05160f2fad936c1bc6f2e6210
ceedff2f8738e33daa938f80bb490297e9665810
86571 F20110109_AABHMM kim_e_Page_103.jp2
40c6180bd1ee1940a46a8a3c5a352cff
daef7946b4da755138e4e75fd18bc48ba4b9f139
79430 F20110109_AABHLY kim_e_Page_083.jp2
98541030a2fd6cd9f776aa9ecfb68013
909c48bd1287358e44f764739e46b5a43b9def01
63143 F20110109_AABHNB kim_e_Page_122.jp2
dc488b4bf300f0b7d454dd4b9e01b7f3
a4c3bc5c13fffc1c3f3858a1306b537837f42583
15081 F20110109_AABHMN kim_e_Page_104.jp2
de3b7ef200f162c262e6395b245111a3
e9eb5013bd01386bbf85bdd9086366965a34455a
29767 F20110109_AABHLZ kim_e_Page_084.jp2
52e2b8a3fa9250a4710cfc803a9e6af6
d9e64af278981aec55d8c359bcd081b6b943a2bb
40523 F20110109_AABHNC kim_e_Page_123.jp2
df5aec464437ebaa42c08fb922136695
be7f5d73ea12a7ce2d6f9d353adc617ea4184e85
58207 F20110109_AABHMO kim_e_Page_105.jp2
0b9b24ccf122e496eee1eb363e44fba7
79eb852aed3787f3c1e3d55b51891f8fbdfd590f
34848 F20110109_AABHND kim_e_Page_125.jp2
26e187d19f2b99840d1573a8216b4e56
771216cbf5b4cd6dce7a59f2fe9f1d45cec6e91a
50401 F20110109_AABHMP kim_e_Page_106.jp2
a9ad16dfa3d32e9ad9d2b1bdc083c98b
a7b8d0d943b8e30050978d36fb56e1f1c8b4804e
54595 F20110109_AABHNE kim_e_Page_126.jp2
2821eb58106aa5cf25fb5dc625ebb390
5344c3c181ca842f0d5c636d3ce0382c80d92949
68002 F20110109_AABHMQ kim_e_Page_107.jp2
9fa25398888a3ca676a368dfc67e6d8d
12514fdd4c408395ae48a8df2f917d245ac27ec5
63568 F20110109_AABHNF kim_e_Page_127.jp2
e81be0c26aabde4e94f32d9e5bad57a7
772ce1d5da2699cfb17da57ed264e5251481badd
63541 F20110109_AABHMR kim_e_Page_108.jp2
62f1dabccf5909985568091f29d7620b
22ef2aee194f5a00abb0df99f418a52288ed882d
53108 F20110109_AABHNG kim_e_Page_128.jp2
285a5cc2eb7f9d01970c35125482348b
e3f032429f7393d08d784d63d5cbd6d1925e66c0
57139 F20110109_AABHMS kim_e_Page_110.jp2
bfa4a856cadfca92ef6bda90d41ed8c1
08e24ffb600f9bb6a16dc06e8c0f9c99fe6291b9
49050 F20110109_AABHNH kim_e_Page_129.jp2
43094a592f676711a597bd1f923ef21f
02779aa092212e44788dd39de883ab8adb0d60c0
50187 F20110109_AABHMT kim_e_Page_111.jp2
b714ee7c15f58e16d86b8af8279d8ea6
cb9d443e1de35e01845eaca8b9355ca4f662ad4c
52043 F20110109_AABHNI kim_e_Page_130.jp2
da138999530ca482118a8d0f17f2cb13
690edd0ff7b64710b9248e7f4e2f77ecc8be8ff6
24194 F20110109_AABHMU kim_e_Page_112.jp2
0172b9ecf476f2f9ae25d833a67c0464
a6b0204585792bd5d22cd33be7627f537f8e4c92
49463 F20110109_AABHNJ kim_e_Page_131.jp2
af9fdaedf2d8a66be284b65a183d41f8
64e119cb9d24b96e8c3e1db972776eac7338f1d8
58310 F20110109_AABHNK kim_e_Page_132.jp2
38f032f6debdd469c1606390c1e92fa4
438ff60f3d03bc3fff40d54884690a7374bdd857
44557 F20110109_AABHMV kim_e_Page_113.jp2
bd91fbedce32ddaae51f1f1f49c3128a
8cf3d32f7a9b1ca970d818244ef7c2264c479107
6960 F20110109_AABHNL kim_e_Page_133.jp2
d5cbc3d44b746851a538e804c48cc261
d376ce7f15a2a279fe02efaf60ba8cc53b063ad1
71786 F20110109_AABHMW kim_e_Page_114.jp2
1786b6d2820e5ff2beb36e73f12d9cf9
5eafe6a4ec51a9da6fdef15134e72a6957fdaa27
F20110109_AABHOA kim_e_Page_014.tif
b0b2a69849c1cb69c67f9560806e081e
f4dc1a66be03484f717a45abe56e01573cbd2e06
1087840 F20110109_AABHNM kim_e_Page_134.jp2
ab653c09c44e448abd4de6718b86e078
fe66615b773638b13fc6e0f8d56ff19c044aae6b
48750 F20110109_AABHMX kim_e_Page_117.jp2
a955620302359b048fda51a9a41d5448
8b81227709bdf3fbaf44022c145187133ffde667
F20110109_AABHOB kim_e_Page_015.tif
389edd527e0ce9f9d111c3fc223aa8c6
2ad6547d89f6cbfe7e0eb8bd81897dac19d5ea94
22557 F20110109_AABHNN kim_e_Page_137.jp2
9d8d176f83a368460c6aee38f9d6926c
2fd391922ccfeaedfca219b6ce155a6ea6d8501a
67731 F20110109_AABHMY kim_e_Page_118.jp2
5577cc23e7c64e0ae0dd7284a253874d
2ababe49a5de488dab204efd27355a7490db6ef2
F20110109_AABHOC kim_e_Page_016.tif
5e60186e8f12df56cfcdd73418922a6f
0b71e7d14314352547b9bf1dd193ef4a6a433d9c
F20110109_AABHNO kim_e_Page_001.tif
396173d9405c996b6e5425a0d24247d2
16c6be76dceff198587ee68e2088c0220b623358
74271 F20110109_AABHMZ kim_e_Page_119.jp2
6872407f5a83d193e01ab86da7eb8fec
0af62509c0ac873381bea2a2ad0aef1949c04255
F20110109_AABHOD kim_e_Page_019.tif
4728b9b37c4770782071ef23879e4780
5c54871e4743ccecafc5611c7135e6be2490ed29
F20110109_AABHNP kim_e_Page_003.tif
45f957b75c41452f5e9d006189979d89
3365cfd67199a7f3f10dd950fbb9773961a2947b
F20110109_AABHOE kim_e_Page_020.tif
0d31d8d00b61254b7a3af1e108a6b76c
29979f7cf6109c5ac90a38d7b7fedeb1496c2372
F20110109_AABHNQ kim_e_Page_004.tif
e4ef0557d2992b62567ad79dc958769f
a162c73e20600c1e9ca84f16b80b7ad644d8d394
F20110109_AABHOF kim_e_Page_021.tif
282d3eb740ea9cd742f812994c1c54bf
42fb99f5b539ff84a2f79586571ade328d1246bc
F20110109_AABHNR kim_e_Page_005.tif
12f98941746f6600b6617ecaaf0c559c
461c2b58e5e8e3ffdc3234d3690278817c4f7d7c
F20110109_AABHOG kim_e_Page_022.tif
5ac1828ceb5882e0f9606f74fe916613
8fce1ce7137d24ed7fa9c1afc55e566580dbdd2d
F20110109_AABHNS kim_e_Page_006.tif
b2fb2c4a83b28ed99ef5176304ceb2f7
0c9278e927ec73754d7d873579030b8edebc6716
F20110109_AABHOH kim_e_Page_023.tif
99046d5c3a0978dc7808f4f1c0b62e80
5de7bb276408c76406ba3d04d09df2fb6c5fffcc
F20110109_AABHNT kim_e_Page_007.tif
fb31338cef4866e990b1ecbcfee9b52a
f6ad6ce19199b665cc9bfa71d48b52502b600dc7
F20110109_AABHOI kim_e_Page_024.tif
83d2731991dfa3f32924e66f6cc15d69
6063ef732fda4f4e9dc26c17eb4610632132a4d0
F20110109_AABHNU kim_e_Page_008.tif
9546fc0dcbc08d3bc037bfb722c8cbda
0a240ed216664236751bfc8506ba2cf3583c1137
F20110109_AABHOJ kim_e_Page_029.tif
8d790619788caf13542becf0a0ae2986
51650204836d1242b10f302be8ab19f3e22c334d
F20110109_AABHNV kim_e_Page_009.tif
ed6ea01917e95772ed9ad1d0181aafcf
7ca840186f453f9a6f8767562afe2aa386a63b23
F20110109_AABHOK kim_e_Page_030.tif
5b94d802c212f351345afdc1fb3aa4c1
7df1f27a8ccd5952ee22a5fb174c232593abeb1c
F20110109_AABHOL kim_e_Page_032.tif
a5b8c4c74c9e94117882d86275f96ba6
f612b0310481b89a1a74af5e627f217f18e2f020
F20110109_AABHNW kim_e_Page_010.tif
da6f133855a68059661566b2e39a4e53
9c86c5e76640b516238ba1e460d13b6bc9159262
F20110109_AABHOM kim_e_Page_033.tif
4e9b1b2f7b08f750a5bd05c622150d7f
8ac5da2ca94449d8ff2374e0d95e3b33d8417d9d
F20110109_AABHNX kim_e_Page_011.tif
cfecaa30e298b5fddc41498578776ad8
545e913a90427f2fb34c537fb682bf61cef797da
F20110109_AABHPA kim_e_Page_049.tif
4e709eddc3470814ce508c563809273e
8019bdf86221eb4bc52ede24cc82d4542561be80
F20110109_AABHON kim_e_Page_034.tif
22f36e7281857452a4604cd8f27372b8
85a5415122789185ef5aba7fa283e0260d59fef6
F20110109_AABHNY kim_e_Page_012.tif
8b7b44d198a9472e5db03912cdbb4ec8
0118508d6ea1de51c0e86cd5a176f1b6317d6e9f
F20110109_AABHPB kim_e_Page_052.tif
8b7fb2d1e02eeec0ffbc50148636098e
93e069b3c6b540731e1a61be93381545a385d391
F20110109_AABHOO kim_e_Page_035.tif
77e65cb5c782e0e2ecfbc5b1d98240b1
38b7729820e67a00ec2f2ec0f339064419857ba2
F20110109_AABHNZ kim_e_Page_013.tif
e98858adcf8ead05925cdc2224afaeee
9a97f7f2c0c6d50fd821c8fb4622b9558aacce63
F20110109_AABHPC kim_e_Page_054.tif
059b84fa6fb4b4b731d0922916b2e09e
d0fbc33b5f241e32714b20c05d495dad822b59e7
F20110109_AABHPD kim_e_Page_055.tif
4b017869a17b7eae0ad314185829a53f
980a5dfed77fa159375c727e03fe6cbdb1b627ab
F20110109_AABHOP kim_e_Page_037.tif
e8a21b0495ef7978db873662f04ec630
7fc196847761c3a296e6261cea51a6c207b4d2c3
F20110109_AABHPE kim_e_Page_056.tif
df964b1bcecd67ff5d0832729050c23d
2140cce4c19b1e2238efcc041599ece2faac1878
F20110109_AABHOQ kim_e_Page_039.tif
f0feb1478b1d8539c67da7e1e7e096f0
7c994bb3d0b7216d2e3b2235072149ec5c7a0a8d
F20110109_AABHPF kim_e_Page_058.tif
4c9bd0841a1986086edec775dbb6863b
91f902b90804900fdf96a93412469f59a2d9b74e
F20110109_AABHOR kim_e_Page_040.tif
f4a7428ce7baa10f1124c3952d85f9fa
5bed9635a0f842f73b953c63b6350f930fd89c8e
F20110109_AABHPG kim_e_Page_059.tif
3727f94a7f2dba63e3e98b0c238e80b4
646c11298e8d604e4d76a7e0694f5f83cb290b6c
F20110109_AABHOS kim_e_Page_041.tif
9d6392d86b5f3dc8a55e689fd0da6c80
97c9c7f345df306cc243e5540cc2cdd76951009d
F20110109_AABHPH kim_e_Page_060.tif
c9c6b18fee6a569eb0ec370bd5538c31
6326a5841ce57cf01bdcc10bdcaceb430717ba9b
F20110109_AABHOT kim_e_Page_042.tif
46c567a1250821e554d5f96da8a94483
4b124ab0a63faef85ca39965114f159c6701d8d2
F20110109_AABHPI kim_e_Page_061.tif
7c39f40438a8045099912ee671a7db04
43fc46ca8af74104d5d3decc7f397f0d73969747
F20110109_AABHOU kim_e_Page_043.tif
6a1ed451d787bedc61bb02560ea98d7b
8e13112425fadae108fd21fea8f225d258da9f07
F20110109_AABHPJ kim_e_Page_062.tif
a0e7108d3a6d670f8c2958e714331f9c
380c1b474e32f14beeb966fae48cc9651577d027
F20110109_AABHOV kim_e_Page_044.tif
48ad7e3aaeb91416c9e069c82766081e
815544c30b12212f8ebc307072aa00138e34e4ac
F20110109_AABHPK kim_e_Page_063.tif
cf8c3276effda14443112ab0a7681b8c
83760546cb59a1393410941eee878bfe0ffec6b8
F20110109_AABHOW kim_e_Page_045.tif
ddaa758322a62ea8dd80856aee1b6eb4
e147f8a3711b12455a180f8b69d69321e9d1b943
F20110109_AABHPL kim_e_Page_065.tif
baa1479ba93528d4fc820a5a8cf8c3bd
79332aa1f27c9329e264acbf9fff6b9dc3dd0063
F20110109_AABHQA kim_e_Page_081.tif
3155b148d74bcaf8db47e32f6eaed757
29a2c6ba8033eb88d22fb21f6be848584ea2ac4a
F20110109_AABHPM kim_e_Page_066.tif
8c69b6700092278db18f0937d365571c
1e08a7b811d206adb8ad9fb3aa1d82719333dd80
F20110109_AABHOX kim_e_Page_046.tif
9739888b3480c480128f4935044d4e2b
94c56f79bd57d6549fe8217f9a2326f03b154e83
F20110109_AABHQB kim_e_Page_082.tif
2a8e316f86771505ac2e10d2e853e82f
39819a099e2f728e9babba321494b8c74c3c0f2d
F20110109_AABHPN kim_e_Page_067.tif
74c943a67ffcf0d6e9e8343bd3fb09f6
f8deea138321d60555ead80f4180f0d2b8754fdf
F20110109_AABHOY kim_e_Page_047.tif
d70595fe58151fa45bfee92f400b5c0a
c117dbd18e93d67035c428eb837387761dc445d9
F20110109_AABHQC kim_e_Page_083.tif
ca28a4c3b03da18235ae61f3fc2ddaf2
2b087faf959bf9f0f7589d6f496b8d610f7f4c93
F20110109_AABHPO kim_e_Page_068.tif
6ddc70cd057fc40e04fec0d4a0e49976
5f334f1936afd18dd59533aaefcc20470fb70409
F20110109_AABHOZ kim_e_Page_048.tif
45740949607e4d501b9ce34ffb798db6
1762f6e7580d39c582c29bbbcc4e1766a8bef868
F20110109_AABHQD kim_e_Page_084.tif
d23d448fec3b8ab32fc04b8eccb47f34
f024b0b93edf41985ba28f961b31e1981f4669d3
F20110109_AABHPP kim_e_Page_069.tif
b2c60dcc7a311a6d5e83e2577a0415b4
c17b85d5eb4337c9ea3b39372d997484ae1de66f
F20110109_AABHQE kim_e_Page_085.tif
780ccd0001d69b505a28042fff1cfc24
6b41f9935c41bc47615796c67a50a617d36cb25e
F20110109_AABHPQ kim_e_Page_070.tif
8decf9dbf242d25f869be540dcc0328a
27ffa35ca227bea5a3267e7b50f7d178c9542a88
F20110109_AABHQF kim_e_Page_086.tif
40d80bc7b5f58ac83dcfbe7c3c253cd6
31c9af63327c384a8347cc202f1f832362e2166f
F20110109_AABHPR kim_e_Page_071.tif
aeab1a57c233615679b91073aaa9b1d6
3b238e505e19ab6f68632d60bf096f153e1d6957
F20110109_AABHQG kim_e_Page_088.tif
24b2c6ace7103f106db66264e3086de1
c591d7b7aa362722a622415d8aa1c9229d95105f
F20110109_AABHPS kim_e_Page_072.tif
a4f211b9fbae31d30f0d2cc42e5cbc29
b919a562faf4820e4b6087a417fc25d0ec65a539
F20110109_AABHQH kim_e_Page_089.tif
6172eb4d2f5d20d82f5dc02a5c75f993
bd7fdb225a80b6cd6610e36433a867f2d2d6d85e
F20110109_AABHPT kim_e_Page_073.tif
b64f7324507fa6f061a557b6de9f4f57
2d523bdab07296857a7b9fa8e351ad1c076fcf0c
F20110109_AABHQI kim_e_Page_091.tif
ed91442c074cb58e4987d2ac55479df8
18c084c7e70ed84520fd1f8d26869a0fca8ca948
F20110109_AABHPU kim_e_Page_074.tif
69e28fe5989d5343d3ffa58e653a1be1
8930248548f7f9befb1f48cb7d23501017f85683
F20110109_AABHQJ kim_e_Page_092.tif
94074e115a79aab0e640966e237dfe85
57a736bca1b4113143da7718eb42150ef406ec88
F20110109_AABHPV kim_e_Page_076.tif
bce0751c1af4ede7560ed59d377378c8
5d47726d81b9de9bdce7eb129203ad15eaed7bfd
F20110109_AABHQK kim_e_Page_095.tif
d860d793d14f6acf652b05f06f66b720
8d1a4541fd4a509aea974c61fb69f88889ba382e
F20110109_AABHPW kim_e_Page_077.tif
e2d0400d6d9e5a06008483a297485b14
c5fe1b51b31503b1308f106dd7e17c43582ac525
F20110109_AABHQL kim_e_Page_096.tif
8895d5ea74b0cf5cba8eeeaf41db5b3e
5ee46fa84f50a00547307be82d23e7671ba4145c
F20110109_AABHPX kim_e_Page_078.tif
7e1f0990ebe53925400dbac5bb028e40
5d6762f2d0a038ee134cfe4802afdc1358d70430
F20110109_AABHQM kim_e_Page_097.tif
129b71b589339ce148e715e2eabae7eb
7f8c3772df70ca312525cf9d2c5ace0029097c27
F20110109_AABHRA kim_e_Page_120.tif
d52fa3e9721e309162e5f1886d48dea6
e80c409cf1bce41ee8aa8aaa840788a162cd175b
F20110109_AABHQN kim_e_Page_099.tif
78009c1027e9010079819d42bf21ebb8
986f1c6fd448e7819b0d2b50b3448b475f10439d
F20110109_AABHPY kim_e_Page_079.tif
b673f41312b0c44af9770edece1a253b
c858183bd60c203721bf08732f1d5eac08c4c40d
F20110109_AABHRB kim_e_Page_121.tif
3d6a3a77ad104b85929f58633253d96e
620a88e15083dc435722d1727c9e247e98056422
F20110109_AABHQO kim_e_Page_100.tif
65401a094c851273eaebfbb2cf701c9d
19c2c49305c7bee55e731ec42e81d689deb4aa07
F20110109_AABHPZ kim_e_Page_080.tif
595d679b7df2e221630b422e5351b60e
574b525c43831519048ea2b6908ace1f807dcb2f
F20110109_AABHRC kim_e_Page_123.tif
f8642629264befa38ef4489be5be13d3
4b745aeb82e1dad7e940c2e133d82dfab2247dd8
F20110109_AABHQP kim_e_Page_102.tif
337684fe67b8bab09d0b2f956883d6af
ec42fc4a8e9e3c2afe1796979c8e874b305dea95
F20110109_AABHRD kim_e_Page_124.tif
02bdd1eb79b4b43c12dab42a70e895d2
8e34d9c959c5022ed8be59717da0834db68170c4
F20110109_AABHQQ kim_e_Page_104.tif
96271d2244abc5e2e538ff88314e30db
107510dbe1e07e428d38e5761b6e3517f372cf29
F20110109_AABHRE kim_e_Page_125.tif
063fb02241d3e20613adbac21c178ac8
6ae756e73a61efe77ff2f564c9611df5bbbcffd0
F20110109_AABHQR kim_e_Page_105.tif
3594b9465237d676875bfeb12fc8dde0
70b450d1972343a5f5e19651d25646d77f444fdb
F20110109_AABHRF kim_e_Page_126.tif
1acc9b88392f5047a644c2aaaee6fe58
46ea798954bf34fed32667bedd2e3d65d25d2638
F20110109_AABHQS kim_e_Page_107.tif
09e55c400db530a33aac4de642628743
d5b7305939d2a99d3db67b8215d90dc5ed94f1fb
F20110109_AABHRG kim_e_Page_128.tif
1cef73b566fcab7e0196dabe944c7654
f35d9520e826a91a25107a2aee1ce3895980a322
F20110109_AABHQT kim_e_Page_108.tif
c83c678716a05b4c8bef5ac2485d80c3
6000eaaec093c54ce5ece009f6b9821ca6840a5e
F20110109_AABHRH kim_e_Page_129.tif
9bc3af6bdc775ed03f4bf8541f100664
cbc8217237ea0670c5ee5f5b5469cef0682b296e
F20110109_AABHQU kim_e_Page_111.tif
1ecd343d25f5d902040864654d7c3488
2c86d4b604852de23ebd14b4675430d01f9cea27
F20110109_AABHRI kim_e_Page_130.tif
6f8e63b01b8be016faedb2605f34f438
c3b897da3b4e1e29084e689eea1a07296b55a9a2
F20110109_AABHQV kim_e_Page_114.tif
2dfca2c0bc5c8cb4c332020add59c079
5fd6f746fbce9688d4fe141692e3ce5ce58657ed
F20110109_AABHRJ kim_e_Page_131.tif
b93dff4031db6395085a3e1282ea3a27
e74163db1f4a864039bac80ad588b654c0871d03
F20110109_AABHQW kim_e_Page_115.tif
866088bf78b2693e0bbcf84e8cb499b5
ff4d53a608a79b187f5dd781fdecc97417fd83af
F20110109_AABHRK kim_e_Page_132.tif
6220881fc8f3d15d1f2c5b323422c935
b572442b0cdd3faca2fe65b18e0c458be9a2d607
F20110109_AABHQX kim_e_Page_116.tif
20b82579e45e3f027f17b8fb82c48507
0df7687dda343ecb6286d5db762f18692ffdd364
F20110109_AABHRL kim_e_Page_134.tif
feb859b3748ed49c8c01883b74654554
382d135a81aa348c6eb60abde63e2b3a6c835ad6
F20110109_AABHQY kim_e_Page_117.tif
d247e62909e1bec72b9ad27a6bd52ce4
e5725083b638f43a30f7a7bbe51761842f610408
39190 F20110109_AABHSA kim_e_Page_016.pro
2e50fb5724afedabc337a40a803510cb
3969c84e938e893cbe85ed4a321d6c637e53be83
F20110109_AABHRM kim_e_Page_136.tif
4dd4158a4c0fa13adeafab9b91734adf
bfd80fb72b6b25dc3e45da215e1700d2f78b7459
40893 F20110109_AABHSB kim_e_Page_018.pro
c6450e42abb64ab4fd9785c2bd772016
35bd2d12ac3907ca453ee3d18d3e2ec781205fa2
F20110109_AABHRN kim_e_Page_137.tif
82addbe1e2fb28b891c87c28ec474828
51db334b6bdc2b8353b9eafb4545878fe6968331
F20110109_AABHQZ kim_e_Page_118.tif
d66e5d14e0b7448df2ae8c3345f0c079
41475cf7ecd7b9f2e7ccf6e4792cd2c02feea783
46179 F20110109_AABHSC kim_e_Page_019.pro
22d520755cea278dfbe30f0e57432f17
9859a14a30790be9bd0f75b7dba539f917af7322
7377 F20110109_AABHRO kim_e_Page_001.pro
fba5bcb6ca7ce40d46d6d7ad9d494244
e00baf004b00ddf0381d42da9b8eeb075355ef44
33653 F20110109_AABHSD kim_e_Page_020.pro
75846c66c9e04fe69cfb224de1f98f82
f54645ebfafd24c3e4962a6eed6efbf361dd5d72
1041 F20110109_AABHRP kim_e_Page_002.pro
7c0d3a27a914ef2c1656107206f4d63b
0e87dc3771ad0bb4f3650b608ad0a206bfe05bc4
42547 F20110109_AABHSE kim_e_Page_021.pro
8e1fb2152a37b5ad07ea2b108e96f2b8
ba500703c16896a33aa7c00728e0f440582de20a
1505 F20110109_AABHRQ kim_e_Page_003.pro
12e8183c8cfcaea7d201eab79c9a12b3
3229179fb2c439ce4f2513d4d89431ccc8f9957c
42682 F20110109_AABHSF kim_e_Page_022.pro
4700f9e9b5df4fc24be489ea3a7454d7
8f22625e69d301bb7bde3555b677eddf708e32fc
14027 F20110109_AABHRR kim_e_Page_004.pro
5acf83efaebf64d20e63d2fb1f957c44
e5142f8324ccb7e5382b9e55f08e07298ddcb339
41613 F20110109_AABHSG kim_e_Page_023.pro
7f9bb82028b6d67386e430190b7122fc
ed8478ce621da2920e2caabb9148ce688e7c31ff
102572 F20110109_AABHRS kim_e_Page_007.pro
38609d0ddcdcecffe68ecdaf87fef048
8230cb80963be09610e5f0a2ae4ed81844b10a49
35332 F20110109_AABHSH kim_e_Page_024.pro
3f8edb1e13d0bb0bd2625cc35a5b679b
f75793a9efac88f967369de242fe1b9039d2eaf9
17333 F20110109_AABHRT kim_e_Page_008.pro
63b6690424f4bbb32be1ae87c1b8dc9d
20a85cff4f35ae39f18d0771ed1744faf95f504c
21992 F20110109_AABHSI kim_e_Page_025.pro
6fd3728174d8b5a65bbbfe411ebb4c7e
047c67da24a3dc52706b1f8a0a7b956feb000237
13250 F20110109_AABHRU kim_e_Page_009.pro
eb29ba55be39f73fc2adefa48de8fd9c
2b293ce23a44c4e4dd66ae36391ae43f2bbfddfa
32924 F20110109_AABHSJ kim_e_Page_026.pro
81b8552d47a8acf4fcf6d5deaef6dd82
024f9a690bf2d078d7c04d2b0a4b3a5c685650ba
22881 F20110109_AABHRV kim_e_Page_010.pro
84637d5eb6a7ac54521ca1de110064f8
527f2f249bcea9b133f0f945bb93d3ede4babbf9
39541 F20110109_AABHSK kim_e_Page_027.pro
40b3f8dd5fef524996441d56f932870f
07b3ce61700182077ddbba8762620001a94abfba
39515 F20110109_AABHRW kim_e_Page_012.pro
8724cc7260c02ea1735bd011e1ff12ce
4dae773a5529f277cfb84f6b7fedd4ee1ee33a53
41766 F20110109_AABHSL kim_e_Page_028.pro
306f12354a5269b7fba981bb3844b037
0fe7fe20459ead53ece615c9db2d1e1cceb154c7
48278 F20110109_AABHRX kim_e_Page_013.pro
3553ab2b29fab634be72e940b75ea1c8
0057f4f9fd55a1b81687d49efd63e61fb9c341fd
50412 F20110109_AABHTA kim_e_Page_050.pro
3d4a4437e1e22fe6bcd9d9acc56d52a3
d0960630476e7699561199a02527bcaaa44823a6
40869 F20110109_AABHSM kim_e_Page_029.pro
ae0404afafa065a0f1e7e6f1f0058eec
fece8687469d829453f3a1e301ba5b744ffa1144
21207 F20110109_AABHRY kim_e_Page_014.pro
52eb624e6e0d3ff89036d8b45c824487
528d193b3f626ff1ec3950efaf89b0414f94dd4d
34571 F20110109_AABHTB kim_e_Page_052.pro
ed0920d58490e5fc1d56e97a447239f1
9caa2fc309272f8bcb5fa7dbaf361597004c24ec
46623 F20110109_AABHSN kim_e_Page_030.pro
80ba18106a3888b0177a62612af667b4
ae9ed726a357b563f70aabc75d6bd8c720c9212b
36458 F20110109_AABHRZ kim_e_Page_015.pro
7d81a10be0e169334c3aa7ac7503f869
1d2d644fbd01b4c139cb60b9e14025341838cde4
42969 F20110109_AABHTC kim_e_Page_053.pro
19fd05bcebdb1c3bf196756222e700a8
5961c0dfad9e3d4d1649af4c965bfe5805be78f4
35110 F20110109_AABHSO kim_e_Page_031.pro
d3fd20195b151c53814831070abe361e
4eacf0580c90c6f6708894d51321067fd7b1009b
45923 F20110109_AABHTD kim_e_Page_054.pro
73904f1a28e1ddec09a17964655a2602
01b8683dcb6e6f410e74c6a2ce74f1ea818b865d
29319 F20110109_AABHSP kim_e_Page_032.pro
936f43f8cec9b112d5eccd45e4c13b33
755011d91be04e8a29a8eac427bf203aaf314a00
46565 F20110109_AABHTE kim_e_Page_055.pro
3dfb7d92515172787f51938fc06f4cf2
b7901d8bbe9f464cbc00313a661a50cce6f9a7fb
34689 F20110109_AABHSQ kim_e_Page_033.pro
b24baa06db2746430c4dc9a04fe000f8
bbb5b855a028b1f39e21903828144b697e74e606
41320 F20110109_AABHTF kim_e_Page_056.pro
a0156e7293def32e903355515facb07b
7e5c5c43510b328aa0ba0546b32824073ef037f9
40567 F20110109_AABHSR kim_e_Page_034.pro
4c70ab26aa50294567c71f29e1195153
75adaec9ed3696322c8e098ed2a466c768a047c1
32663 F20110109_AABHTG kim_e_Page_057.pro
74254573da0a0713b18905655a99318b
60356b2aa00905e0dc932e12a58eb639a4a185f2
36300 F20110109_AABHSS kim_e_Page_037.pro
c0521e6ff5f7928d7711835279c14bcf
9118bf410fafd59d4c86f861800002f66be94dcf
31649 F20110109_AABHTH kim_e_Page_058.pro
800ff7dbff14797c78fe7c35c6bf9c85
c77b4b43a1f101257580df047078ac552d85b331
44554 F20110109_AABHST kim_e_Page_038.pro
1afaded0260bcc2c51ef425b8ac76e7c
8559b6d6a626a19d69daabe114fedc1b3b698d4b



PAGE 1

IMPLEMENTATION PATTERNS FOR PARALLEL PROGRAM AND A CASE STUDY By EUNKEE KIM A THESIS PRESENTED TO THE GRADUATE SCHOOL OF THE UNIVERSITY OF FLORIDA IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF SCIENCE UNIVERSITY OF FLORIDA 2002

PAGE 2

Copyright 2002 by Eunkee Kim

PAGE 3

Dedicated to Grace Kim, Jineun Song and parents

PAGE 4

iv ACKNOWLEDGMENTS I would like to express my sincere gratitude to my adviser, Dr. Beverly A. Sanders, for providing me with an opportunity to work in this exciting area and for providing feedback, guidance, support, and encouragement during the course of this research and my graduate academic career. I wish to thank Dr. Joseph N. Wilson and Dr. Stephen M. Thebaut for serving on my supervisory committee. Finally I thank Dr. Berna L. Massingill for allowing me to use machines in Trinity University and for technical advice.

PAGE 5

v TABLE OF CONTENTS page ACKNOWLEDGMENTS.................................................................................................ivLIST OF TABLES.............................................................................................................ixLIST OF FIGURES............................................................................................................xABSTRACT....................................................................................................................... xiCHAPTER 1 INTRODUCTION.........................................................................................................11.1 Parallel Computing...................................................................................................11.2 Parallel Design Patterns............................................................................................11.3 Implementation Patterns for Design Patterns...........................................................21.4 Implementation of Kernel IS of NAS Parallel Benchmark Set Using Parallel Design Patterns...................................................................................................21.5 Organization of the Thesis........................................................................................32 OVERVIEW OF PARALLEL PATTERN LANGUAGE.............................................42.1 Finding Concurrency Design Sp ace..........................................................................42.1.1 Getting Started................................................................................................42.1.2 Decomposition Strategy..................................................................................42.1.3 Task Decomposition.......................................................................................52.1.4 Data Decomposition........................................................................................52.1.5 Dependency Analysis......................................................................................62.1.6 Group Tasks....................................................................................................62.1.7 Order Tasks.....................................................................................................72.1.8 Data Sharing....................................................................................................72.1.9 Design Evaluation...........................................................................................72.2 Algorithm Structure Design Space...........................................................................82.2.1 Choose Structure.............................................................................................82.2.2 Asynchronous Composition..... ................ ................ ................ ............. ..........92.2.3 Divide and Conquer........................................................................................9

PAGE 6

vi 2.2.4 Embarrassingly Parallel................................................................................102.2.5 Geometric Decomposition............................................................................102.2.6 Pipeline Processing.......................................................................................112.2.7 Protected Dependencies................................................................................112.2.8 Recursive Data..............................................................................................112.2.9 Separable Dependency..................................................................................122.3 Supporting Structures Design Space.......................................................................122.3.1 Program-Structuring Group..........................................................................122.3.1.1 Single program and multiple data.......................................................122.3.1.2 Fork join..............................................................................................132.3.1.3 Master worker.....................................................................................132.3.1.4 Spawn..................................................................................................132.3.2 Shared Data Structures Group.......................................................................132.3.2.1 Shared queue.......................................................................................132.3.2.2 Shared counter....................................................................................142.3.2.3 Distributed array.................................................................................142.4 Chapter Summery...................................................................................................143 PATTERNS FOR IMPLEMENTATION.....................................................................153.1 Using Message Passing Interface (MPI).................................................................153.1.1 Intent................................................................................................................... .153.1.2 Applicability........................................................................................................153.1.3 Implementation....................................................................................................153.1.3.1 Simple message passing.............................................................................163.1.3.2 Group and communicator creation.............................................................173.1.3.3 Data distribution and reduction..................................................................223.2 Simplest Form of Embarrassingly Parallel.............................................................233.2.1 Intent.............................................................................................................233.2.2 Applicability ................................................................................................. 233.2.3 Implementation.............................................................................................243.2.4 Implementation Example..............................................................................253.2.5 Example Usage..............................................................................................273.3 Implementation of Embarrassingly Parallel............................................................283.3.1 Intent.............................................................................................................283.3.2 Applicability..................................................................................................283.3.3 Implementation.............................................................................................283.3.4 Implementation Example..............................................................................293.3.5 Usage Example ............................................................................................. 383.4 Implementation of Pipeline Processing..................................................................393.4.1 Intent.............................................................................................................393.4.2 Applicability.................................................................................................. 393.4.3 Implementation ............................................................................................. 393.4.4 Implementation Example..............................................................................413.4. 5 Usage Example ............................................................................................. 473.5 Implementation of Asynchronous-Composition.....................................................483.5.1 Intent.............................................................................................................48

PAGE 7

vii 3.5.2 Applicability..................................................................................................493.5.3 Implementation.............................................................................................493.5.4 Implementation Example..............................................................................503.6 Implementation of Divide and Conquer.................................................................543.6.1 Intent.............................................................................................................543.6.2 Motivation.....................................................................................................543.6.3 Applicability..................................................................................................553.6.4 Implementation.............................................................................................553.6.5 Implementation Example..............................................................................573.6.6 Usage Example.............................................................................................614 KERNEL IS OF NAS BENCHMARK........................................................................634.1 Brief Statement of Problem....................................................................................634.2 Key Generation and Memory Mapping..................................................................634.3 Procedure and Timing.............................................................................................645 PARALLEL PATTERNS USED TO IMPLEMENT KERNEL IS.............................665.1 Finding Concurrency..............................................................................................665.1.1 Getting Started..............................................................................................665.1.2 Decomposition Strategy................................................................................675.1.3 Task Decomposition.....................................................................................685.1.4 Dependency Analysis....................................................................................685.1.5 Data Sharing Pattern.....................................................................................685.1.6 Design Evaluation.........................................................................................695.2 Algorithm Structure Design Space.........................................................................705.2.1 Choose Structure...........................................................................................705.2.2 Separable Dependencies................................................................................715.2.3 Embarrassingly Parallel................................................................................715.3 Using Implementation Example.............................................................................715.4 Algorithm for Parallel Implementation of Kernel IS..............................................726 PERFORMANCE RESULTS AND DISCUSSIONS..................................................746.1 Performance Expectation........................................................................................746.2 Performance Results...............................................................................................746.3 Discussions.............................................................................................................767 RELATED WORK AND CONCLUSIONS AND FUTURE WORK.........................787.1 Related work...........................................................................................................787.1.1 Aids for Parallel Programming.....................................................................787.1.2 Parallel Sorting..............................................................................................807.2 Conclusions.............................................................................................................817.3 Future Work............................................................................................................82APPENDIX

PAGE 8

viii A KERNEL IS OF THE NAS PARALLEL BENCHMAKRK.....................................83B PSEUDORANDOM NUMBER GENERATOR.......................................................90C SOURCE CODE OF THE KERNEL IS IMPLEMENTATION...............................94D SOURCE CODE OF PIPELINE EXAMPLE..........................................................107E SOURCE CODE OF DIVIDE AND CONQUE R ...................................................115LIST OF REFERENCES................................................................................................124BIOGRAPHICAL SKETCH..........................................................................................127

PAGE 9

ix LIST OF TABLES Table page 4-1 Parameter Values to be used for Benchmark...............................................................65 6 -1 Performance Results for Class A and B ......................................................................75 A -1 Values to be used for Partial Verification................................................................... 8 8 A-2 Parameter Values to be used for Benchmark ........................................................... ...... 89

PAGE 10

x LIST OF FIGURES Figure page 3-1 The Algorithm Structure Decision Tree..........................................................................93-2 Usage of Message Passing ...............................................................................................403-3 Irregular Message Handling ............................................................................................503-4 Message Passing for Invocation of the Solve and Merge Functions.............................564-1 Counting Sort...................................................................................................................676-1 Execution Time Comparison for Class A........................................................................766-2 Execution Time Comparison for Class B........................................................................76

PAGE 11

xi Abstract of Thesis Presented to the Graduate School of the University of Florida in Partial Fulfillment of the Requirements for the Degree of Master of Science IMPLEMENTATION PATTERNS FOR PARALLEL PROGRAM AND A CASE STUDY By Eunkee Kim December 2002 Chairman: Beverly A. Sanders Major Department: Computer and Information Science and Engineering Design patterns for parallel programming guides the programmer through the entire process of developing a parallel program. In this thesis, implementation patterns for parallel program are presented. These patterns help programmers to implement a parallel program after designing it using the parallel design patterns. Parallel Integer sorting, one of the kernels of Parallel Benchmark set of the Numerical Aerodynamic Simulation Program (NAS), has been designed and implemented using the parallel design pattern as a case study. How the parallel design pattern was used in implementing Kernel IS and its performance are discussed

PAGE 12

1 CHAPTER 1 INTRODUCTION 1.1 Parallel Computing Parallel computing is what a computer does when it carries out more than one computation at a time using many processors. An example of parallel computing (processing) in our daily life is an automobile assembly line: at every station somebody is doing part of the work to complete the product. The purpose of parallel computing is to overcome the limit of the performance of a single processor. We can increase the performance of a sequential program by parallelizing exploitable concurrency in a sequential program and using ma ny processors at once. Parallel programming has been considered more difficult than sequential programming. To make it easier for programmers to write a parallel program, a parallel design pattern has been developed by B.L. Massingill, T.G. Mattson, and B.A Sanders.1-4 1.2 Parallel Design Patterns The Parallel pattern language is a collection of design patterns for parallel programming.5-6 Design patterns are a high level description of a solution to a problem.7 Parallel pattern language is written down in a systematic way so that a final design for a parallel program can result by going through a sequence of appropriate patterns from a pattern catalog. The structure of the patterns is designed to parallelize even complex problems. The top-level patterns help to find the concurrency in a problem and decompose it into a collection of tasks. Th e second-level patterns help to find an

PAGE 13

2 appropriate algorithm structure to exploit concurrency that has been identified. The parallel design patterns are described in more detail in Chapter 2. 1.3 Implementation Patterns for Design Patterns Several implementation patterns for design patterns in the algorithm structure design space were developed as a part of the parallel pattern language and presented in this thesis. These implementation patterns are solutions of problems mapping high-level parallel algorithms into programs using a Message Passing Interface (MPI) and programming language C.8 The patterns of the algorithm structure design space capture recurring solutions to the pr oblem of turning problems into parallel algorithms. These implementation patterns can be reused and provide guidance for programmers who might need to create their own implementations after using the parallel design pattern. The implementation patterns of design patterns in the algorithm structure design space are in Chapter 3. 1.4 Implementation of Kernel IS of NA S Parallel Benchmark Set Using Parallel Design Patterns The Numerical Aerodynamic Simulation (NAS) Program, which is based at NASA Ames Research Center, developed parallel benchmark set for the performance evaluation of highly parallel supercomputers. The NAS Parallel Benchmark set is a Paper and Pencil benchmark.9 All details of this benchmark set are specified only algorithmically. The Kernel IS of NAS Parallel Benchmark set is a parallel sorting over large numbers of integers. A solution to the Kernel IS is de signed and implemented using parallel pattern language as a case study. The used patterns are following. Getting Start, Decomposition Strategy, Task Decomposition, Dependency Analysis, Data Sharing, and Design Evaluation patterns of Find Concurrency Design Space are

PAGE 14

3 used. Choose Structure pattern, Separable Dependency, and Embarrassingly Parallel patterns of the algorithm structure design space are also used. The details of how these patterns are used and the final design of the parallel program for Kernel IS are in Chapter 5. 1.5 Organization of the Thesis The research that has been conducted as part of this thesis and its organization are as follows: Analysis of the parallel design patterns (Chapter 2) Implementation patterns for patterns in Algorithm structure design space (Chapter 3) A description of Kernel IS of the NAS Parallel Benchmark set (Chapter 4) Design and implementation of Kernel IS through the paralle l design patterns (Chapter 5) Performance results and discussions (Chapter 6). Conclusions and future work (Chapter 7).

PAGE 15

4 CHAPTER 2 OVERVIEW OF PARALLEL PATTERN LANGUAGE Parallel pattern language is a set of design patterns that guide the programmer through the entire process of developing a parallel program.10 The patterns of parallel pattern language, as developed by Masingill et al., are organized into four design spaces: Finding Concurrency Design Space, Algorithm Structure Design Space, Supporting Structure Design Space, and Implementation Design Space. 2.1 Finding Concurrency Design Space The finding concurrency design space includes high-level patterns that help to find the concurrency in a problem and decompose it into a collection of tasks. 2.1.1 Getting Started The getting started pattern helps to start designing a parallel program. Before using these patterns, a user of this pattern needs to be sure that the problem is large enough or needs to speed up and understand the problem. The user of this pattern needs to do the following tasks: Decide which parts of the problem require most intensive computation. Understand the tasks that need to be carried out and data structure that are to be manipulated. 2.1.2 Decomposition Strategy This pattern helps to decompose the problem into relatively independent entities that can execute concurrently. To expose the concurrency of the problem, the problem can be decomposed along two different dimensions:

PAGE 16

5 Task Decomposition: Break the stream of instructions into multiple chunks called tasks that can execute simultaneously. Data Decomposition: Decompose the problem’s data into chunks that can be operated relatively independently. 2.1.3 Task Decomposition The Task Decomposition pattern addresses the issues raised during a primarily taskbased decomposition. To do task decomposition, the user should try to look at the problem as a collection of distinct tasks. And these tasks can be found in the following places: Function calls may correspond to tasks. Each iteration of a loop, if the iterations are independent, can be the tasks. Updates on individual data chunks decomposed from a large data structure. The number of tasks generated should be flexible and large enough. The tasks should have enough computation. 2.1.4 Data Decomposition This pattern looks at the issues involved in decomposing data into units that can be updated concurrently. The first point to be considered is whether the data structure can be broken down into chunks that can be operated on concurrently. An array-based computation and recursive data structures are examples of this approach. The points to be considered in decomposin g data are as follows: Flexibility in the size and number of data chunks to support the widest range of parallel systems The size of data chunks large enough to offset the overhead of managing dependency Simplicity in data decomposition

PAGE 17

6 2.1.5 Dependency Analysis This pattern is applicable when the proble m is decomposed into tasks that can be executed concurrently. The goal of a dependency analysis is to understand the dependency among the tasks in detail. Data-sharing dependencies and ordering constraints are the two kinds of dependencies that should be analyzed. The dependencies must require little time to manage relative to computation time and easy to detect and fix errors. One effective way of analyzing dependency is following approach. Identify how the tasks should be grouped together. Identify any ordering constraints between groups of tasks. Analyze how tasks share data, both within and among groups. These steps lead to the Group Tasks, the Order Tasks, and the Data Sharing patterns. 2.1.6 Group Tasks This pattern constitutes the first step in analyzing dependencies among the tasks of problem decomposition. The goal of this pattern is to group tasks based on the following constraints. Three major categories of constraints among tasks are as follows: A temporal dependency: a constraint placed on the order in which a collection of tasks executes. An ordering constraint that a collection of tasks must run at the same time. Tasks in a group are truly independent. The three approaches to group tasks are as follows: Look at how the problem is decomposed. The tasks that correspond to a high-level operation naturally group together. If the tasks share constraints, keep them as a distinct group. If any other task groups share the same constraints, merge the groups together. Look at constraints between groups of tasks.

PAGE 18

7 2.1.7 Order Tasks This pattern constitutes the second step in analyzing dependencies among the tasks of problem decomposition. The goal of this pattern is to identify ordering constraints among task groups. Two goals are to be met in defining this ordering: It must be restrictive enough to satisfy all the constraints to be sure the resulting design is correct. It should not be more rest rictive than it need be. To identify ordering constraints, consid er the following ways tasks can depend on each other: First look at the data required by a group of tasks before they can execute. Once this data have been identified, fi nd the task group that crea ted it, and you will have an order constraint. Consider whether external services can impose ordering constraints. It is equally important to note when an order constraint does not exit. 2.1.8 Data Sharing This pattern constitutes the third step in analyzing dependencies among the tasks of problem decomposition. The goal of this pattern is to identify what data are shared among groups of tasks and how to manage access to shared data in a way that is both correct and efficient. The following approach can be used to determine what data is shared and how to manage the data: Identify data that are shared between tasks. Understand how the data will be used. 2.1.9 Design Evaluation The goals of this pattern are to evaluate the design so far and to prepare for the next phase of the design space. This pattern therefore describes how to evaluate the design

PAGE 19

8 from three perspectives: suitability for the target platform, design quality, and preparation for the next phase of the design. For the suitability for the target plat form, the following issues included: How many processing elements are available? How are data structures shared among processing elements? What does the target architecture imply about the number of units of execution and how structures are shared among them? On the target platform, will the time spent doing useful work in a task be significantly greater than the time taken to deal with dependencies? For the design quality, simplicity, flexibility, and efficiency should be considered. To prepare the next phase, the key issues are as follows: How regular are the tasks and their data dependencies? Are interactions between tasks (or task groups) synchronous or asynchronous? Are the tasks grouped in the best way? 2.2 Algorithm Structure Design Space The algorithm structure design space contains patterns that help to find an appropriate algorithm structure to exploit the concurrency that has been identified. 2.2.1 Choose Structure This pattern guides the algorithm designer to the most appropriate AlgorithmStructure patterns for the problem. Consideration of Target Platform, Major Organizing Principle and Algorithm-Structure Decision Tr ee are the main topics of this pattern. The two primary issues of Consideration of Target Platform are how many units of execution (threads or processes) the target system will effectively support and the way information is shared between units of execution. The three major organizing principles are organization by ordering, organization by tasks, and organization by data. The

PAGE 20

9 Algorithm-Structure Decision Tree is in Figure 3.1. We can select an algorithm-structure using this decision tree. Figure 3-1 The Algorithm Structure Decision Tree 2.2.2 Asynchronous Composition This pattern describes what may be the most loosely structured type of concurrent program in which semi-independent tasks interact through asynchronous events. Two examples are the Discrete-Event Simulation and Event-Driven program. The key issues in this pattern are how to define the tasks/en tities, how to represent their interaction, and how to schedule the tasks. 2.2.3 Divide and Conquer This pattern can be used for the parallel application program based on the well-known divide-and-conquer strategy. This pattern is particularly effective when the amount of

PAGE 21

10 work required to solve the base case is large compared to the amount of work required for the recursive splits and merges. The key elem ents of this pattern are as follows: Definitions of the functions such as “solve,” “split,” “merge,” “baseCase,” “baseSolve” A way of scheduling the tasks that exploits the available concurrency This pattern also includes Correctness issues and Efficiency issues. 2.2.4 Embarrassingly Parallel This pattern can be used to describe concurrent execution by a collection of independent tasks and to show how to organize a collection of independent tasks so they execute efficiently. The key element of this pattern is a mechanism to define a set of tasks and schedule their execution and also a mechanism to detect completion of the tasks and terminate the computation. This pattern also includes the correctness and efficiency issues. 2.2.5 Geometric Decomposition This pattern can be used when the concurrency is based on parallel updates of chunks of a decomposed data structure, and the update of each chunk requires data from other chunks. Implementations of this pattern include the following key elements: A way of partitioning the global data structure into substructures or “chucks” A way of ensuring that each task has access to all the data needed to perform the update operation for its chunk, including da ta in chunks corresponding to other tasks A definition of the update operation, whether by points or by chunks A way of assigning the chunks among units of execution (distribution), that is a way of scheduling the corresponding tasks

PAGE 22

11 2.2.6 Pipeline Processing This pattern is for algorithms in which data flow through a sequence of tasks or stages. This pattern is applicable when the problem consists of performing a sequence of calculations, each of which can be broken do wn into distinct stages on a sequence of inputs. So for each input, the calculations must be done in order, but it is possible to overlap computation of different stages for different inputs. The key elements of this pattern are as follows: A way of defining the elements of the pipeline where each element corresponds to one of the functions that makes up the computation A way of representing the dataflow among pipeline elements, i.e., how the functions are composed A way of scheduling the tasks 2.2.7 Protected Dependencies This pattern can be used for task-based decompositions in which the dependencies between tasks cannot be separated from the execution of the tasks, and the dependencies must be dealt with in the body of the task. The issues of this pattern are as follows: A mechanism to define a set of tasks and sc hedule their execution onto a set of units of executions Safe access to shared data Shared memory available when it is needed 2.2.8 Recursive Data This pattern can be used for parallel applications in which an apparently sequential operation on a recursive data is reworked to make it possible to operate on all elements of the data structure concurrently. The issues of this pattern are as follows: A definition of the recursive data structure plus what data is needed for each element of the structure

PAGE 23

12 A definition of the concurrent operations to be performed A definition of how these concurrent operations are composed to solve the entire problem A way of managing shared data A way of scheduling the tasks onto units of executions A way of testing its termination condition, if the top-level structure involves a loop. 2.2.9 Separable Dependency This pattern is used for task-based decompositions in which the dependencies between tasks can be eliminated as follows: Necessary global data are replicated and (partial) results are stored in local data structures. Gl obal results are then obtained by reducing results from the individual tasks. The key el ements of this pattern are as follows: Defining the tasks and scheduling their execution Defining and updating a local data structure Combining (reducing) local ob jects into a single object 2.3 Supporting Structures Design Space The patterns at this level represent an intermediate stage between the problem-oriented patterns of the algorithm structure design space and the machine-oriented patterns of the Implementation-Mechanism Design Space. Patterns in this space fall into two ma in groups: Program-Structuring Group and Shared Data Structures Group. 2.3.1 Program-Structuring Group Patterns in this group deal with constructs of structuring the source code. 2.3.1.1 Single program and multiple data The computation consists of N units of execution in parallel. All N UEs (Generic term for concurrently executable entity, usually e ither process or thread) execute the same

PAGE 24

13 program code, but each operates on its own set of data. A key feature of the program code is a parameter that differentiates among the copies. 2.3.1.2 Fork join A main process or thread forks off some number of other processes or threads that then continue in parallel to accomplish some portion of the overall work before rejoining the main process or thread. 2.3.1.3 Master worker A master process or thread sets up a pool of worker processes or threads and a task queue. The workers execute concurrently with each worker repeatedly removing a task from the task queue and processing it until all tasks have been processed or some other termination condition has been reached. In so me implementations, no explicit master is present. 2.3.1.4 Spawn A new process or thread is created, which then executes independently of its creator. This pattern bears somewhat the same relation to the others as GOTO bears to the constructs of structured programming. 2.3.2 Shared Data Structures Group Patterns in this group describe commonly used data structures. 2.3.2.1 Shared queue This pattern represents a "thread-safe" implementation of the familiar queue abstract data type (ADT), that is, an implementation of the queue ADT that maintains the correct semantics even when used by concurrently executing units of execution.

PAGE 25

14 2.3.2.2 Shared counter This pattern, as with the Shared Queue pattern, represents a “thread-safe” implementation of a familiar abstract data ty pe, in this case a counter with an integer value and increment and decrement operations. 2.3.2.3 Distributed array This pattern represents a class of data st ructures often found in parallel scientific computing, namely, arrays of one or more dimensions that have been decomposed into subarrays and distributed among processes or threads. 2.4 Chapter Summery This chapter describes an overview of the parallel pattern languages for an application program which guides programmers from the design process of the program to the implementation point. The Next chapter illustrates how these design patterns have been used to design and implement the Kern el IS of the NAS Parallel Benchmark.

PAGE 26

15 CHAPTER 3 PATTERNS FOR IMPLEMENTATION In this chapter, we will introduce patterns for the implementation of patterns in the algorithm structure design space. Each pattern contains Implementation Example for each pattern in the algorithm structure design space. The Implementation Examples are implemented in an MPI and C environment. The MPI is standards for "Message Passing Interface" in Distributed Memory Environments. The Implementation Examples may need modification occasionally, but will be reus able and helpful for an implementation of the parallel program designed using parallel pattern language. 3.1 Using Message Passing Interface (MPI) 3.1.1 Intent This pattern is an introduction of Message Passing Interface (MPI). 3.1.2 Applicability This pattern is applicable when the user of parallel pattern language has finished the design of his parallel program and considers implementing it using MPI. 3.1.3 Implementation The MPI is a library of functions (in C) or subroutines (in Fortran) that the user can insert into a source code to perform data communication between processes. The primary goals of MPI are to provide source code portability and allow efficient implementations across a range of architectures.

PAGE 27

16 3.1.3.1 Simple message passing Message passing programs consist of multip le instances of a serial program that communicates by library calls. The elementary communication operation in MPI is "point-to-point" communication, that is, direct communication between two processes, one of which sends and the other receives. This message passing is the basic mechanism of exchanging data among processes of a parallel program using MPI. This message passing is also a good solution about how to order the executions of tasks belongs to a group and how to order the executions of groups of tasks. To communicate among processes, communicators are needed. Default communicator of MPI is MPI_COMM_WORLD which includes all processes. The common implementation of MPI executes same programs concurrently at each processing element (CPU or workstation). The following program is a very simple example of using MPI_Send (send) and MPI_Recv (receive) functions. /* simple send and receive */ /* Each process has the same copy of the following program and executes it */ #include #include #define BUF_SIZE 100 int tag =1; void main (int argc, char **argv) { int myrank; /* rank (identifier) of each process*/ MPI_Status status; /* status object */ double a[BUF_SIZE]; /*Send(Receive) buffer */ MPI_Init(&argc, &argv); /* Initialize MPI */ MPI_Comm_rank(MPI_COMM_WORLD, &myrank); /* Get rank of process*/ if( myrank == 0 ) { /* code for process 0 */ /* send Message */ MPI_Send(

PAGE 28

17 a, /*initial address of send buffer */ BUF_SIZE, /*number of elements in send buffer */ MPI_DOUBLE, /*datatype of each send buffer element */ 1, /*rank of destination */ tag, /* message tag */ MPI_COMM_WORLD /*communicator */ ); } else if( myrank == 1 ){ /* code for process 1 */ /*receive message */ MPI_Recv( a, /*initial address of receive buffer */ BUF_SIZE, /*number of elements in send buffer */ MPI_DOUBLE, /*datatype of each receive buffer element */ 0, /*rank of source */ tag, /*message tag */ MPI_COMM_WORLD, /*communicator */ &status /*status object */ ); } /* more else if statement can be added for more processes */ /* switch statements can be used instead of if else statement s*/ MPI_Finalize(); /* Terminate MPI */ } 3.1.3.2 Group and communicator creation A communicator in MPI specifies the communication context for communication operations and the set of processes that share this communication context. To make communication among processes in a program using MPI, we must have communicators. There are two kinds of communicators in MPI: intra-communicator and intercommunicator. Intra-communicators are for message passing among processes belong to the same group. Inter-communicators are for message passing between two groups of processes. As mentioned previously, the execution order of tasks belonging to one group can be implemented using intra-communicator and message-passing. The execution order of groups of tasks can be implemented using inter-communicator and message-passing.

PAGE 29

18 To create a communicator, a group of processes is needed. MPI does not provide mechanism to build a group from scratch, but only from other, previously defined groups. The base group, upon which all other groups can be created, is the group associated with the initial communicator MPI_COMM_WORLD. A group of processes can be created by the five steps as follows: 1. Decide the processing units that will be included in the groups 2. Decide the base groups to use in creating new group. 3. Create group. 4. Create communicator. 5. Do the work. 6. Free communicator and group. The first step is decide how many and which processing units will be included in this group. The points to be considered are the number of available processing units, the number of tasks in one group, the number of groups that can be executed concurrently, etc., according to the design of the program. If only one group of tasks can executed because of dependency among groups than the group can use all available processing units and may include them in that group. If there are several groups that can be executed concurrently, then the available processing units may be divided according to the number of groups that can be executed concurrently and the number of tasks in each group. Since MPI does not provide a mechanism to build a group from scratch, but only from other, previously defined groups, the group constructors are used to subset and superset existing groups. The base group, upon which all other groups can be defined, is the group

PAGE 30

19 associated with the initial communicator MPI_COMM_WORLD, and processes in this group are all the processes available when MPI is initialized. There are seven useful group constructors in MPI. Using these constructors, groups can be constructed as designed. The first three constructors are similar to the union and intersection of set operation in mathematics. The seven group constructors are as follows: The MPI_GROUP_UNION creates a group which contains all the elements of the two groups used. The MPI_GROUP_INTERSECTION creates a group which contains all the elements that belong in both groups at the same time. The MPI_GROUP_DIFFERENCE creates a group which has all the elements that do not belong in both groups at the same time. The MPI_GROUP_INCL routine creates a new group that consists of the specified processes in the array from the old group. The MPI_GROUP_EXCL routine creates a new group that consists of the processes not specified in the array from the old group. The MPI_GROUP_RANGE_INCL routine includes all processes between one specified process to another process, and the specified processes themselves. The MPI_GROUP_RANGE_EXCL routine excludes all processes not between the specified processes, and also excludes the specified processes themselves. For the creation of communicators, there are two types of communicators in MPI: intra-communicator and inter-communicator. The important intra-communicator constructors in MPI are as follows: The MPI_COMM_DUP function will duplicate the existing communicator. The MPI_COMM_CREATE function creates a new communicator for a group. For the communication between two groups, the inter-communicator for the identified two groups can be created by using the communicator of each group and peer-

PAGE 31

20 communicator. The routine for this purpose is MPI_INTERCOMM_CREATE. The peercommunicator must have at least one selected member process from each group. Using duplicated MPI_COMM_WORLD as a dedicated peer communicator is recommended. The Implementation Example Code is as follows: /******************************************/ /* An Example of creating intra-communicator */ /******************************************/ #include main(int argc, char **argv) { int myRank, count, count2; int *sendBuffer, *receiveBuffer, *sendBuffer2, *receiveBuffer2; MPI_Group MPI_GROUP_WORLD, grprem; MPI_Comm commSlave; static int ranks[] = { 0 }; /* ... */ MPI_Init(&argc, &argv); MPI_Comm_group(MPI_COMM_WORLD, &MPI_GROUP_WORLD); MPI_Comm_rank(MPI_COMM_WORLD, &myRank); /* Build group for slave processes */ MPI_Group_excl( MPI_GROUP_WORLD, /* group */ 1, /* number of elements in array ranks */ ranks, /* array of integer ranks in group not to appear in new group */ &grprem); /* new group derived from above */ /* Build communicator for slave processes */ MPI_Comm_create( MPI_COMM_WORLD, /* communicator */ grprem, /* Group, which is a subset of the group of above communicator */ &commSlave); /* new communicator */ if(myRank !=0) { /* compute on processes other than root process */ /* ... */ MPI_Reduce(sendBuffer, receive Buffer, count, MPI_INT, MPI_SUM, 1, commSlave);

PAGE 32

21 /* ... */ } /* Rank zero process falls through immediately to this reduce, others do later */ MPI_Reduce(sendBuffer2, receiveBuffer2, count2, MPI_INT, MPI_SUM, 0, MPI_COMM_WORLD); MPI_Comm_free(&commSlave); MPI_Group_free(&MPI_GROUP_WORLD); MPI_Group_free(&grprem); MPI_Finalize(); } /******************************************/ /* An example of creating inter-communicator */ /******************************************/ #include main(int argc, char **argv) { MPI_Comm myComm; /* intra-communicator of local sub-group */ MPI_Comm myInterComm; /* inter-communicator between two group */ int membershipKey; int rank; MPI_Init(&argc, &argv); MPI_Comm_rank(MPI_COMM_WORLD, &rank); /* User generate membership Key for each group */ membershipKey = rank % 3; /* Build intra-communicato r for local sub-group */ MPI_Comm_split( MPI_COMM_WORLD, /* communicator */ membershipKey, /* Each subgroup contains all processes of the same membershipKey */ rank, /* Within each subgroup, the processes are ranked in the order defined by this rank value*/ &myComm /* new communicator */ ); /* Build inter-communicators. Tags are hard-coded */ if (membershipKey == 0) {

PAGE 33

22 MPI_Intercomm_create( myComm, /* local intra-communicator */ 0, /* rank of local group leader */ MPI_COMM_WORLD, /* "peer" intra-communicator */ 1, /* rank of remote group leader in the "peer" communicator */ 1, /* tag */ &myInterComm /* new inter-communicator */ ); } else { MPI_Intercomm_create(myComm, 0, MPI_COMM_WORLD,0,1, &myInterComm); } /* do work in parallel */ if(membershipKey == 0) MPI_Comm_free(&myInterComm); MPI_Finalize(); } 3.1.3.3 Data distribution and reduction There are several primitive functions for distribution of data in MPI. The three basic functions are as follows: The MPI_BROADCAST function replicates data in one root process to all other processes so that every process has the same data. This function can be used for a program which needs replication of data in each process to improve performance. The MPI_SCATTER function of MPI can scatter data from one processing unit to every other processing unit with the same amount (or variant amounts) of data, and each processing unit will have a diffe rent portion of the original data. The Gather function of MPI is the inverse of Scatter. For the reduction of data distributed among processes, the MPI provides two primitive MPI_REDUCE and MPI_ALL_REDUCE functions. These reduce functions are useful

PAGE 34

23 when it is needed to combine locally computed subsolutions into one global solution. The MPI_REDUCE function combines the elements provided in the input buffer of each process in the group using the specified operation as a parameter, and returns the combined value in the output buffer of the process with rank of root. The MPI_ALL_REDUCE function combines the data in the same way as the MPI_REDUCE, but every process has the combined value in the output buffer. This function is beneficial when the combined solution can be used as an useful information for the next phase of computation. The Implementation Example Code is not provided in this section, but the Implementation Example Code of 3.2.4 can be used as an Implementation Example Code. The source code of the Kernel IS implementation is also a good Implementation Example Code. 3.2 Simplest Form of Embarrassingly Parallel 3.2.1 Intent This is a solution to the problem of how to implement the simplest form of Embarrassingly Parallel pattern in the MPI and C environment. 3.2.2 Applicability This pattern can be used after the program is designed using patterns of the finding conc urrency design space and patterns of algorithm structure design space. And, the resulted design is a simplest form of the Embarrassingly Parallel pattern. The simplest form of Embarrassingly Parallel pattern satisfies the conditions as follows: All the tasks are independent. All the tasks must be completed. Each task executes same algorithm on a distinct section of data.

PAGE 35

24 3.2.3 Implementation The common MPI implementations have a stat ic process allocation at initialization time of the program. It also executes same program at each processing unit (or workstation). Since all the tasks executes same algorithm on a different data, the tasks can be implemented as one high level function which will be executed in each process. The simplest form of Embarrassingly Parallel pattern can be implemented by the following five steps. 1. Find out the number of available processing units. 2. Distribute or replicate data to each process. 3. Execute the tasks. 4. Synchronize all processes. 5. Combine (reduce) local re sults into a single result. In MPI, the number of available processing units can be found by calling the MPI_COMM_SIZE function and passing the MPI_WORLD_COM communicator as a parameter. In most cases, after finding out the number of available processing units, data can be divided by that number and distributed into each processing unit. There are several primitive functions for distribution of data in MPI. The three basic functions are Broadcast Scatter and Gather These functions are introduced in 3.1.3.3. After computing local results at each process, synchronize all processes to check that all local results are computed. This can be implemented by calling the MPI_BARRIER function after each local result is solved at each processing unit because the MPI_BARRIER blocks the caller until all group members have called it.

PAGE 36

25 The produced local results can be combined to get the fina l solution of the problem. The Reduce functions of MPI are useful in combining subsolutions. The Reduce function of MPI combines the elements, which are provided in the input buffer of each process in the group, using the specified operation as a parameter, and returns the combined value in the output buffer of the process with rank of root. The operations are distributed to each process in many MPI implementations so they can improve overall performance of the program if it takes considerable time to co mbine subsolutions in one processing unit. 3.2.4 Implementation Example #include #define SIZE_OF_DATA 2000 /* size of data to be distributed into each process */ int root = 0; /* The rank(ID) of root process */ int myRank; /* process ID */ /*= Modification1========================================*/ /*= The data type of each variable should be modified =*/ int* data; /*= The data to be distributed into each process =*/ int* localData; /*= distributed data that each process will have =*/ int* subsolution; /*= subsolutaion data =*/ int* solution; /*= solution data =*/ /*===================================================*/ int sizeOfLocalData; /* number of elements in local data array */ int numOfProc; /* number of avaiable process */ int numOfElement; /* number of element in subsolution */ /***************************************************/ /** Highest level function which solves subproblem */ /***************************************************/ void solve() { /*= Implementation1==============================*/ /*=The code for solving subproblem should be implemented =*/ /*============================================*/ } /****************************************************/ /* Main Function which starts program */ /****************************************************/

PAGE 37

26 main(argc, argv) int argc; char **argv; { MPI_Status status; /* MPI initialization */ MPI_Init(&argc, &argv); /* Finding out the rank (ID) of each process */ MPI_Comm_rank(MPI_COMM_WORLD,&myRank); /*Finding out the number of available processes */ MPI_Comm_size(MPI_COMM_WORLD,&numOfProc); /* Dividing the data by the number of processes */ sizeOfLocalData = SIZE_OF_DATA/numOfProc; /* Dynamic memory allocation for local data */ localData=(int *)mall oc(sizeOfLocalData*sizeof(int)); /*****************************************************************/ /* Distribute data into local data of each process */ /*****************************************************************/ MPI_Scatter( data, /* Address of send buffer (data) to distribute */ sizeOfLocalData, /* Number of element sent to each process */ MPI_INT, /*MODIFICATION2=================*/ /*= data type of the send buffer =*/ /*===============================*/ localData, /* Address of receive buffer (local data) */ sizeOfLocalData, /* Number of element in receive buffer (local */ /* data) */ MPI_INT, /*= MODIFICATION2===============*/ /*= Data type of receive buffer =*/ /*===============================*/ root, /* Rank of sending process */ MPI_COMM_WORLD /* Communicator */ ); /*solve sub-problem in each process */ solve(); /*Synchronize all processes to check all subproblems are solved */ MPI_Barrier(MPI_COMM_WORLD); /******************************************************** *****/

PAGE 38

27 /* Combine sub-solutions to get the solution */ /*************************************************************/ MPI_Reduce( subsolution, /* address of send buffer (subsolution) */ solution, /* address of receive buffer (solution) */ numOfElement, /* number of elements in send buffer (subsolution) */ MPI_INT, /*= MODIFICATION3============*/ /*= data type of the send buffer =*/ /*=============================*/ MPI_MULT, /*= MODIFICATION4 ============*/ /*= reduce operation =*/ /*============================*/ root, /* rank of root process */ MPI_COMM_WORLD /* communicator */ ); MPI_Finalize(); 3.2.5 Example Usage The steps to reuse the Implementation Example are as follows: Implement the blocks labeled as IMPLEMENTATION1. This block should contain code for the “solve” function. Modify the data type of the variables which are in the block labeled as MODIFICATION1. Modify the data type parameters which are in the block labeled as MODIFICATION2. Each data type parameter must be one of MPI data type which matches with the type of the data to receive and send. Modify the data type parameters which ar e in the block labeled as MODIFICATION3. Each data type parameter must be one of MPI data type which matches with the type of the data to receive and send Modify the reduce operator parameter which are in the block labeled as MODIFICATION4. This parameter should be one of MPI operators. Consider a simple addition of each element in an integer array with size 1024. What should be modified is the solve function and fifth parameter of MPI_Reduce function as follows: void solve()

PAGE 39

28 { for(i=0;i
PAGE 40

29 If each task has known amount of computations, the implementation of a parallel program is straightforward in MPI and C environment. Each task can be implemented as a function and can be executed at each process using process ID (rank) and if else statement or switch statement. Since the amount of computation of each task, the load balance should be achieved by distributing the tasks among processes when the program is implemented. If the amount of computation is not known, there should be some dynamic tasks scheduling mechanism for the load balance. The MPI does not provide primitive process scheduling mechanism. One way of simulating dynamic tasks scheduling mechanism is using a task-name queue and primitive message passing of MPI. We implemented this mechanism as follows: Each task is implemented as a function. The root process (with rank 0) has a task-name queue which contains the names of tasks (functions). Each process sends message to the root process to get task name, whenever it is idle or finished its task. The root process with rank 0 waits for messages from other processes. If the root process receives messages and the task queue is not empty, it sends back the name of task in the task queue to the sender process of that message. When the process, which sent message for a task name, receives the task name, it executes the task (function). The Implementation Example is provided in 3.3.4. The MPI routines, which is useful for the combining the locally computed subsolutions into a global solution, are described in 3.1.3.3 3.3.4 Implementation Example #include #include #define EXIT -1 /* exit message */ #define TRUE 1

PAGE 41

30 #define FALSE 0 #define ROOT 0 #define NUM_OF_ALGO 9 int oneElm =1; /* one element */ int tag =0 ; int algoNameSend= -1; int idleProc = -1; int algoNameRecv= -1; /* Size of Data to send and Receive */ int sizeOfData; /* number of tasks for each algorithm */ int numOfTask[NUM_OF_ALGO] ; int isAlgorithmName = TRUE; int moreExitMessage = TRUE; MPI_Status status; /*******************************************************/ /* Simple Queue implementation. */ /*******************************************************/ int SIZE_OF_Q = 20; int* queue; int qTail = 0; int qHead = 0; /******************************/ /* insert */ /******************************/ void insert(int task){ if(qHead-qTail==1) { /* when queue is full, increase array */ int* temp = (int *) malloc(2*SIZE_OF_Q*sizeof(int)); int i; for(i=0;i2){

PAGE 42

31 /* queue is not full and there is more than one space */ qTail++; queue[qTail]= task; } else{ /* there is just one more space in queue */ qTail=0; queue[qTail]= task; } } /***************************************/ /* check whether or not queue is empty */ /***************************************/ int isEmpty() { if(qTail==qHead){ /* queue is empty */ return 1; } else{ return 0; } } /*************************/ /* remove */ /*************************/ int remove() { if(qHead
PAGE 43

32 /*= the data which will be used by this algorithm in local process =*/ /*= More of this block can be added on needed basis =*/ int* data; /* Modify the data type*/ /* receive the size of data to receive */ MPI_Recv(&sizeOfData, oneElm,MPI_INT,ROOT, tag,MPI_COMM_WORLD,&status); /* dynamic memory allocation */ data = (int* /* Modify the data type*/ ) malloc(sizeOfData*sizeof(int /* Modify the data type*/ )); /* Receive the input data */ MPI_Recv(&data,sizeOfData, MPI_INT, /*=
PAGE 44

33 /*=IMPLEMENTATION1 ==============================*/ /*= code for this algorithm should be implemented =*/ /*=================================================*/ /* Memory deallocation ex) free(data);*/ } void algorithm3(){ /*= DATA TRANSFER 1 =================================*/ /*= the data which will be used by this algorithm in local process =*/ /*= More of this block can be added on needed basis =*/ int* data; /* Modify the data type*/ /* receive the size of data to receive */ MPI_Recv(&sizeOfData, oneElm,MPI_INT,ROOT, tag,MPI_COMM_WORLD,&status); /* dynamic memory allocation */ data = (int* /* Modify the data type*/ ) malloc(sizeOfData*sizeof(int /* Modify the data type*/ )); /* Receive the input data */ MPI_Recv(&data,sizeOfData, MPI_INT, /*=
PAGE 45

34 MPI_Init(&argc, &argv); /*find local rank*/ MPI_Comm_rank(MPI_COMM_WORLD, &myRank); /*find out last rank by using size of rank*/ MPI_Comm_size(MPI_COMM_WORLD,&mySize); /*****************************************************************/ /* This process distributes tasks to other processing elements */ /*****************************************************************/ if(myRank == 0) { /* code for root process. This root process receives messages from other processes when they are idle. Then this process removes an algorithm name for a task from task name queue and send it back to sender of the message. If the task queue is empty, this process sends back an exit message */ /*=MODIFICATION1=================================*/ /*= the data for each algorithm =*/ /*= the data type should be modified =*/ int* algo1Data; int* algo2Data; int* algo3Data; int* algo4Data; int* algo5Data; /*= =*/ /*=================================================*/ int numOfSentExit = 0; /* This array is a task queue. */ queue = (int *) malloc(SIZE_OF_Q*sizeof(int)); /* */ for(i=0;i
PAGE 46

35 insert(i); } } /* Receive message from other processes */ while(moreExitMessage) { int destination= ROOT; /* Receive message from any process */ MPI_Recv(&algoNameRecv,oneElm,MPI_INT,MPI_ANY_SOURCE, MPI_ANY_TAG,MPI_COMM_WORLD,&status); destination = algoNameRecv; if(!isEmpty()) { /* If the task queue is not empty, send back an algorihtm name for a task to the sender of received message */ algoNameSend = queue[remove()]; MPI_Send(&algoNameSend,oneElm,MPI_INT, destination,tag,MPI_COMM_WORLD); switch(algoNameSend) { case 1: /*=IMPLEMENTATION2===========================*/ /*= code for calculating the size of data to send and =*/ /*= finding the starting address of data to send =*/ /*= should be implemented =*/ /*=============================================*/ /*=DATA TRANSFER2=============================*/ /*= More of this block can be added on needed basis =*/ /*= =*/ MPI_Send(&sizeOfData,oneElm,MPI_INT, destination,tag,MPI_COMM_WORLD); MPI_Send( &algo1Data, /*=
PAGE 47

36 /*= code for calculating the size of data to send and =*/ /*= finding the starting address of data to send =*/ /*= should be implemented =*/ /*===========================================*/ /*=DATA TRANSFER2=============================*/ /*= More of this block can be added on needed basis =*/ /*= =*/ MPI_Send(&sizeOfData,oneElm,MPI_INT, destination,tag,MPI_COMM_WORLD); MPI_Send( &algo1Data, /*=
PAGE 48

37 algoNameSend = EXIT; MPI_Send(&algoNameSend,oneElm,MPI_INT, destination,tag,MPI_COMM_WORLD); /* keep tracking the number of exit message sent. If the number of exit messages is same with the number of processes, root process will not receive any more message requesting tasks.*/ numOfSentExit++; if(numOfSentExit==mySize-1) { moreExitMessage=FALSE; } } /* computation and/or message passing for combining subsolution can be added here */ } } /***************************************************************/ /* Code for other processes, not root. These pr ocesses send message */ /* requesting next task to execute to root process when they are idle. */ /* These processes execute tasks received from root */ /***************************************************************/ else { idleProc = myRank; /*Send message to root process, send buffer contains the rank of sender process whenever it is idle*/ MPI_Send(&idleProc,oneElm,MPI_INT,ROOT,tag,MPI_COMM_WORLD); while(isAlgorithmName) /*while message contains an algorihtm name */ { /* Receive message from root which contains the name of task to execute in this process */ MPI_Recv(&algoNameRecv,oneElm,MPI_INT,ROOT,tag, MPI_COMM_WORLD,&status); /* each process executes tasks using the algorihtm name and data received from root */ switch(algoNameRecv) { /* More case statements should be added or removed, according to the number of tasks */ case EXIT : isAlgorithmName = FALSE;

PAGE 49

38 break; case 1: algorithm1(); break; case 2: algorithm2(); break; case 3: algorithm3(); break; case 4: algorithm4(); break; case 5: algorithm5(); break; default: break; } if(algoNameRecv != EXIT) { /*Send message to root process, send buffer contains the rank of sender process whenever it is idle*/ idleProc = myRank; MPI_Send(&idleProc,oneElm,MPI_INT,ROOT,tag,MPI_COMM_WORLD); } } } /*============================================*/ /*= Codes for collecting subsolution and =*/ /*= computing final solution can be added here. =*/ /*============================================*/ MPI_Finalize(); } 3.3.5 Usage Example The steps to reuse this Implementation Example are as follows: Implement the blocks labeled as IMPLEMENTATION1. Each block should contain codes for each algorithm. Implement the block labeled as IMPLEMENTATION2. What should be implemented are codes for calculating the initial address and the number of elements of the send buffer for each algorithm.

PAGE 50

39 Modify the data type of the variables and data type parameters which are in the blocks labeled as DATA TRANSFER 1. The data type of each variable should match with the data type of the input data for ea ch algorithm. The purpose of this send function is to send input data to the destination process. More of this block can be added on needed basis. Modify the data type and data type parameters which are in the block labeled as DATA TRANSFER2. Each data type parameter must be one of MPI data type which matches with the type of the data to receive. The purpose of these receive function is to receive input data for the algorithm execution on local proc ess. More of this block can be added on needed basis Assume that there is a parallel program design such that the amount of computation for each task is unknown. If some of the tasks share same algorithm, these tasks should be implemented as one algorithm function in one of the block labeled as IMPLMENTATION1. To execute the tasks, the same number of algorithm name with the number of tasks that share same algorithm should be put in the queue. 3.4 Implementation of Pipeline Processing 3.4.1 Intent This is a solution to the problem of how to implement the Pipeline Processing pattern in the MPI and C environment. 3.4.2 Applicability This pattern can be used after the program is design ed using patterns of finding concurrency design space and patterns of the algorithm structure design space and the resulted design is a form of the Pipeline Processing pattern. 3.4.3 Implementation Figure3.1 illustrates the usage of messa ge passing to schedule tasks in the Implementation Example. The arrows represent the blocking synchronous send and receive in MPI. The squares labeled C1, C2, etc., in Figure3.1, represent the elements of calculations to be performed which can overlap each others. The calculation elements are

PAGE 51

40 implemented as functions in the Implementation Example. The calculation elements (functions) should be filled up to do comput ation by the user of this Implementation Example. Adding more functions for more calculation is trivial because message passing calls inside functions are very similar between functions. Each Stage (Stage1, Stage2, and etc.), in Figure3.1, corresponds to each process. In the Implementation Example, each Stage calls single function (calculation element) sequentially. Figure 3-2 Usage of Message Passing One point to note is that the first pipe line stage do not receive message and last element do not send message for scheduling of tasks. Scheduling of tasks is achieved by blocking and synchronous mode point-to-point communications (MPI_SSEND, MPI_RECEIVE). C1 C2 C3 C4 C1 C2 C3 C4 C1 C2 C3 C4 C1 C2 C3 C4 Process 1 Process 2 Process 3 Process 4 time

PAGE 52

41 In MPI, if the Send mode is blocking and synchronous and the Receive mode is blocking, then the Send can complete only if the matching Receive has started. The statements after the matching receive will not be executed until the receive operation is finished: MPI provides synchronous communication semantics. A duplicate of MPI_COMM_WORLD (initial intra-communicator of all processes) is used to provide separate communication space and safe communication. If we change the duplicate of MPI_COMM_WORLD into appropriate communicator for a group of processes then, this structure can be used for a portion of a program where the Pipeline Processing pattern can be used. 3.4.4 Implementation Example #include #include #include int numOfCalElem = 4; char startMsg[7]; int myRank; int mySize; int tag = 1; char* sendBuf, recvBuf; int sendBufSize, recvBufSize; MPI_Status status; /* Only four pipeline stages are implem ented in this Implementation Example. More elements can be added or removed, according to the design of the parallel program */ /******************/ /*First Pipeline Stage */ /******************/ void firstPipelineSta ge(MPI_Comm myComm) { /*=IMPLEMENTATION1========================================*/ /*= =*/

PAGE 53

42 /*= code for this Pipeline Stage should be implemented here =*/ /*= =*/ /*==========================================================*/ /* send message to the next Pipeline Stage of process with myRank+1 */ MPI_Ssend(startMsg,strlen(startMsg),MPI_CHAR ,myRank+1,tag,MPI_COMM_WORLD); /*=DATA TRANSFER 1======================================*/ /*= More send function, which has same structure, can be added =*/ /*= to transfer data =*/ MPI_Ssend( /*= Modify the following parameters =*/ sendBuf, /*
PAGE 54

43 /* send message to the next Pipeline Stage of process with myRank+1 */ MPI_Ssend(startMsg,strlen(startMsg),MPI_CHAR,myRank+1,tag, MPI_COMM_WORLD); /*=DATA TRANSFER 1======================================*/ /*= More send function, which has same structure, can be added =*/ /*= to transfer data =*/ MPI_Ssend( /*= Modify the following parameters =*/ sendBuf, /*
PAGE 55

44 /*= More send function, which has same structure, can be added =*/ /*= to transfer data =*/ MPI_Ssend( /*= Modify the following parameters =*/ sendBuf, /*
PAGE 56

45 MPI_CHAR, /*
PAGE 57

46 /********************/ /* Last Pipeline Stage */ /********************/ void lastPipelineStage(MPI_Comm myComm) { /* Receive message from previous Pipeline Stage of process with myRank-1 */ MPI_Recv(startMsg,strlen(s tartMsg),MPI_CHAR,myRank-1, tag, MPI_COMM_WORLD,&status); /*=DATA TRANSFER 2====================================*/ /*= More receive f unctions, which has same structure, can be added =*/ /*= to transfer data =*/ MPI_Recv( /*= Modify the following parameters =*/ recvBuf, /*=
PAGE 58

47 switch(myRank) { case 0 : for(i=0;i
PAGE 59

48 Add or remove the case statements, in the block labeled as MODIFICATION2, according to the number of pipeline stages of the program design. Implement the blocks labeled as IMPLEMENTATION1. Each block should contain codes for each calculation element. Modify the initial address of send buffer, the number of elements in send buffer, and the data type parameters which are in the block labeled as DATA TRANSFER1. The data to be sent is the input for the next calculation element. More send functions, which have same structure, can be added according to the need. Modify the initial address of receive buffer, the number of elements in receive buffer, and the data type parameters which are in the block labeled as DATA TRANSFER2. The data to be received is the input for this calculation element. More receive functions, which have same structure, can be added according to the need. Consider following problem as an example problem to parallelize: There are four SAT scores of four schools. Find out standard deviation for each class. But the main memory of one processor barely contains the scores of one school. To solve this problem, the pipeline stages can be defined as follows: The first pipeline stage computes the sum and average of SAT scores of each school. The second pipeline stage computes the differences between each individual score and the average. The third pipeline stage computes squares of each difference. Forth pipeline stage computes the sum of the computed squares. This problem can be easily implemented by following the above steps. The implementation of this is provided in appendix. 3.5 Implementation of As ynchronous-Composition 3.5.1 Intent This is a solution of the problem how to implement the parallel algorithm which resulted from the Asynchronous -Composition pattern in the algorithm structure design space in MPI (Message Passing Interface)?

PAGE 60

49 3.5.2 Applicability Problems are represented as a collection of semi-independent entities interacting in an irregular way. 3.5.3 Implementation Three points that need to be defined to implement the algorithm resulting from Asynchronous-Composition pattern using MP I are tasks/entities, events, and task scheduling. A task/entity, which generates an event and processes them, can be represented as a process in an implementation of an Algorithm using MPI. An event corresponds to a message sent from the event generating task to the event processing task. All tasks can be executed concurrently. In this Implementation Example, each case block in a switch statement should contain code for each semi-independent entity (process) which will be executed concurrently. An event corresponds to a message sent from the event generating task (process) to the event processing task (process) in MPI and C environment. Therefore, event handling is a message handling. For safe message passing among processes, the creation of groups of processes which need to communicate is necessary. Creating communicators for the groups in MPI is also necessary. Because of that, we added groups and communicators creations in this Implementation Example. This Implementation Example is also an example of event handling in MPI environment. In a situation that an entity (process) receives irregular events (messages sent) from other known entities (processes), we can implement it using defined constant of MPI, MPI_ANY_SOURCE and MPI_ANY_TAG.

PAGE 61

50 Assume that process A, B and C sends messa ges in an irregular way to process D and the process D handles (processes) events (message s). It is same situation as the car/driver example of the Asynchronous-Composition pattern. To use the MPI_ANY_SOURCE and MPI_ANY_TAG in the receive routine of MPI, we need to create an dedicated group and communicator for these entities (processes) to prevent entity (process) D receives messages from other entities (processes) that is not intended to send message to the entity (process) D. Figure 3-3 Irregular Message Handling 3.5.4 Implementation Example #include #include #include main(argc, argv) int argc; char **argv; { /* More group variables can be added or removed on need basis*/ Process B Process D Process A Process C

PAGE 62

51 MPI_Group MPI_GROUP_WORLD,group_A,group_B; /*More communicator variables can be added or removed on need basis */ MPI_Comm comm_A,comm_B; /*This varilable will hold the rank for each process in MPI default communicator MPI_COMM_WORLD */ int rank_in_world; /*This variable will hold the rank for each process in group A. Variables can be added or removed on need basis.*/ int rank_in_group_A; int rank_in_group_B; /* ranks of processes which will be used to create subgroups. More array of ranks can be added or removed on need basis.*/ int ranks_a[]={1,2,3,4}; int ranks_b[]={1,2,3,4}; MPI_Init(&argc, &argv); /*Create a group of processes and communicator for a group More groups and communicator can be created or removed on need basis*/ MPI_Comm_group(MPI_COMM_WORLD,&MPI_GROUP_WORLD); /*Create group and communicator */ MPI_Group_incl(MPI_GROUP_WORLD,4,ranks_a,&group_A); MPI_Comm_create(MPI_COMM_WORLD,group_A,&comm_A); /*Create group and communicator */ MPI_Group_incl(MPI_GROUP_WORLD,4,ranks_b,&group_B); MPI_Comm_create(MPI_COMM_WORLD,group_B,&comm_B); MPI_Comm_rank(MPI_COMM_WORLD, &rank_in_world); switch(rank_in_world) { /*This case 1 contains codes to be executed in process 0 */ case 0: { /* events can be generated or processed */ /* work */ break;

PAGE 63

52 } /*This case contains codes to be executed in process 1 */ case 1: { char sendBuffer[20]; int isOn=1; int i=0; while(isOn) { /* works that need to be done before generating event */ strcpy(sendBuffer,"event"); /* an example */ /* Generate Event (message passing) */ MPI_Send(sendBuffer,strlen(sendBuffer),MPI_CHAR,3,1,comm_B); printf("sent message"); i++; /* break loop */ if(i==10) /* this should be changed according to the problem */ { isOn = 0; } } break; } /*This case contains codes to be executed in process 2 */ case 2: { char sendBuffer[20]; int isOn=1; int i=0; while(isOn) { /* works that need to be done before generating event */ strcpy(sendBuffer,"event");/* an example */ /* Generate Event (message) */ MPI_Send(sendBuffer,strlen (sendBuffer),MPI_CHAR,3,1,comm_B); i++;

PAGE 64

53 /* break loop */ if(i==10) /* this should be changed according to the problem */ { isOn = 0; } } break; } /*This case contains codes to be executed in process 3 */ case 3: { char sendBuffer[20]; int isOn=1; int i=0; while(isOn) { /* works that need to be done before generating event */ strcpy(sendBuffer,"event"); /* an example */ /* Generate Event (message) */ MPI_Send(sendBuffer,strlen (sendBuffer),MPI_CHAR,3,1,comm_B); i++; /* break loop */ if(i==10) /* this should be changed according to the problem */ { isOn = 0; } } break; } /*This case contains codes to be executed in process 4 */ case 4: { char receiveBuffer[20]; int isOn = 1; int messageCount=0; MPI_Status status; while(isOn) { MPI_Recv(receiveBuffer,20 ,MPI_CHAR,MPI_ANY_SOURCE, MPI_ANY_TAG,comm_B,&status);

PAGE 65

54 messageCount++; if(0==strncmp(r eceiveBuffer,"event",3)) { /* work to process the event (message) */ printf("\nreceived an event at process 4"); if(messageCount==30) { isOn = 0; } } break; } /* more cases(processes) can be added or removed. */ default: break; } MPI_Finalize(); } 3.6 Implementation of Divide and Conquer 3.6.1 Intent For the completeness of the parallel pattern language, it would be beneficial for a programmer to show an implementation example of a top-level program structure for the Divide-and-Conquer pattern in the Algorithm Design space. 3.6.2 Motivation The top-level program structure of the divide-and-conquer strategy that is stated in the Divide-and-Conquer pattern is as follows Solution solve(Problem P){ if (baseCase (P)) return baseSolve(P); else { Problem subProblems[N]; Solution subSolutions[N]; subProblems = split(P); for (int i=0;i
PAGE 66

55 } This structure can be mapped onto a design in terms of tasks by defining one task for each invocation of the solve function. The common MPI implementations have a st atic process allocation at initialization time of the program. We need a mechanism to invoke the solve function at another processor whenever the solve function calls the split function, and the split function splits the problem into sub-problems so that subproblems can be solved concurrently. 3.6.3 Applicability This pattern can be used after the program is designed using patterns of the parallel pattern language and the resulting design is a form of the Divide-and-Conquer pattern and if we want to implement the design in an MPI and C environment. This structure can be used as an example of how to implement a top-level program structure in an MPI and C environment or directly adopting this structure as an implementation of the program by adjusting control parameters and adding a computational part of the program. 3.6.4 Implementation We are trying to implement a top-level program structure of the divide-and-conquer strategy in MPI and C so that parallel program design resulting from the Divide-andConquer pattern can be implemented by filling up each function or/and adjusting the structure. The basic idea is using message passing to invoke the solve functions at other processing elements when needed. In a standard MPI and C environment, each processing element has the same copy of the program with other processing elements, and same program executes at each processing element communicating with each other on a needed basis. When the problem splits into subproblems, we need to call the solve

PAGE 67

56 function at other processes so that subproblems can be solved concurrently. To call the solve functions, the split function send messag es to other processing elements and the blocking MPI_Receive function receives messages before the first solve function at each process. This message passing can contain data/tasks divided by the split function. In this structure, every solve function calls the merge function to merge subsolutions. Figure 3-4 Message Passing for Invocation of the Solve and Merge Functions For simplicity, we split a problem into tw o subproblems and used problem size to determine whether or not the subproblem is a base case. Figure3.3 shows the sequence of CPU CPU2 CPU1 CPU5 CPU6 CPU4 CPU3 CPU7 solve split solve solve split split solve solve solve solve split split split split solve solve solve solve solve solve solve solve merge merge merge merge merge merge merge merge merge merge merge merge merge merge Message sent from split to solve

PAGE 68

57 function calls in each process and message passing to invoke the solve function at remote process for new sub-problem and to merge sub-solutions. 3.6.5 Implementation Example #include #include #include #define DATA_TYPE int int numOfProc; /*number of available processes*/ int my_rank; int ctrlMsgSend; int ctrlMsgRecv; int* localData; int dataSizeSend; int dataSizeRecv; /********************/ /* Solve a problem */ /********************/ void solve(int numOfProcLeft) { if(baseCase(numOfProcLeft)) { baseSolve(numOfProcLeft); merge(numOfProcLeft); } else { split(numOfProcLeft); if(numOfProcLeft!=numOfProc) { merge(numOfProcLeft); } } } /*****************************************************/ /* split a problem into two subproblems */ /*****************************************************/ int split(int numOfProcLeft)

PAGE 69

58 { /*=IMPLEMENTATION2 ==============================*/ /*= Code for splitting a problem into two su bproblems =*/ /*= should be implemented =*/ /*= =*/ /*================================================*/ ctrlMsgSend = numOfProcLeft/2; /* invoke a solve function at the remote process */ MPI_Send(&ctrlMsgSend,1,MPI_INT, my_rank+numOfProc/numOfProcLeft, 0,MPI_COMM_WORLD); /*=DATA TRANSFER 1 ==============================*/ /*= More of this block can be added on needed basis =*/ MPI_Send(&dataSizeSend,1,MPI_INT, my_rank+numOfProc/numOfProcLeft, 0, MPI_COMM_WORLD); MPI_Send( &localData[dataSizeLeft], /*= numOfProc/(numOfProcLeft*2)) { ctrlMsgSend = 1; /* Send a subsolution to the process from which this process got the subproblem */ MPI_Send(&ctrlMsgSend,1,MPI_INT, my_rank numOfProc/(numOfProcLeft*2), 0,MPI_COMM_WORLD); /*=DATA TRANSFER 3=============================*/

PAGE 70

59 /*= More of this block can be added on needed basis =*/ MPI_Send(&dataSizeSend,1,MPI_INT, my_rank numOfProc/(numOfProcLeft*2), 0,MPI_COMM_WORLD); MPI_Send( localData, /*<-modify address of data */ dataSizeSend, MPI_INT, /*<-modify data type */ my_rank numOfProc/(numOfProcLeft*2), 0,MPI_COMM_WORLD); /*= =*/ /*=============================================*/ } else { MPI_Status status; /* Receive a subsolution from the process which was invoked by this process */ MPI_Recv(&ctrlMsgRecv,1,MPI_INT, my_rank+numOfProc/(numOfProcLeft*2), 0,MPI_COMM_WORLD,&status); /*=DATA TRANSFER 3===============================*/ /*= More of this block can be added on needed basis =*/ MPI_Recv(&dataSizeRecv,1,MPI_INT, my_rank+numOfProc/(numOfProcLeft*2), 0,MPI_COMM_WORLD,&status); MPI_Recv( &localData[dataSizeLeft], /*
PAGE 71

60 /***********************************************/ /* Decide whether a problem is a "base case" */ /*that can be solved without further splitting */ /***********************************************/ int baseCase(int numOfProcLeft) { if(numOfProcLeft == 1) return 1; else return 0; } /*****************************/ /* Solve a base-case problem */ /*****************************/ int baseSolve(int numOfProcLeft) { /*=IMPLEMENTATION1=================================*/ /*== Code for solving base case problem should be implemented ==*/ /*== ==*/ /*===================================================*/ return 0; } main(argc, argv) int argc; char **argv; { int i; MPI_Status status; MPI_Init(&argc, &argv); MPI_Comm_rank(MPI_COMM_WORLD, &my_rank); MPI_Comm_size(MPI_COMM_WORLD, &numOfProc); count = (int *)malloc((maxInt+1)*sizeof(int)); if(my_rank == 0){ ctrlMsgSend=numOfProc; /* number of processes must be power of 2 */ } ctrlMsgRecv = numOfProc; solve(ctrlMsgRecv); } else{ /* Every process waits a message before calling solve function */

PAGE 72

61 MPI_Recv(&ctrlMsgRecv,1,MPI_INT,MPI_ANY_SOURCE,MPI_ANY_TAG, MPI_CO MM_WORLD,&status); /*=DATA TRANSFER 2===============================*/ /*= More of this block can be added on needed basis =*/ MPI_Recv(&dataSizeR ecv,1,MPI_INT,MPI_ANY_SOURCE, MPI_ANY_TAG,MPI_COMM_WORLD,&status); MPI_Recv( localData, /*
PAGE 73

62 TRANSFER1. The data to be sent is the input for the solve function of remote process. More of this block can be added on needed basis. Find out and modify the initial address of receive buffer, the number of elements to receive, and the data type parameters which are in the block labeled as DATA TRANSFER2. The data to receive is the inpu t for the solve function of local process. More of this block can be added on needed basis. Find out and modify the initial address of receive buffer, the number of elements to receive, and the data type parameters which are in the block labeled as DATA TRANSFER3. The data to receive and send are the input for the merge function of local process. More of this block can be added on needed basis. One point to note about using the Implementation Example is that the “baseCase” of the Implementation Example is when there are no more available processes. Merge sort algorithm is well known problem which uses divide and conquer strategy. If we consider the merge sort over N integers as the problem to parallelize, it can be easily implemented using the Implementa tion Example. To use the Implementation Example, the merge sort algorithm should be modified as follows: The “baseSolve” is a counting sort over the local data (an array of integers). The “split” algorihtm divides received data (an array of integers) into two contiguous subarrays. The “merge” algorithm merges two sorted array into one sorted array. The “BaseCase” of this problem is when there are no more available processes. The problem can be implemented by following the implementation steps above. The implementation of this problem is provided in appendix.

PAGE 74

63 CHAPTER 4 KERNEL IS OF NAS BENCHMARK The Numerical Aerodynamic Simulation (NAS) Program, which is based at NASA Ames Research Center, developed a set of benchmarks for the performance evaluation of highly parallel supercomputers. These benchmarks, NPB 1 (NAS Parallel Benchmarks 1), consist of five parallel kernels and three si mulated application benchmarks. The principle distinguishing feature of these benchmarks is their “pencil and paper” specification. All details of these benchmarks are specified only algorithmically. The Kernel IS (Parallel Sort over small integers) is one of the kernel benchmark set. 4.1 Brief Statement of Problem Sorting N integer keys in parallel is the problem. The number of keys is 232 for class A and 252for class B. The range of keys is max, 0 B where maxB is 192 for class A and 212 for class B. The keys must be generated by the key Generation algorithm of the NAS benchmark set. The initial distribution of the key must follow the specification of memory mapping of keys. Even though the problem is sorting, what will be timed is the time needed to rank every key, and the pe rmutation of keys will not be timed. 4.2 Key Generation and Memory Mapping The Keys must be generated using the pseudorandom number generator of the NAS Parallel Benchmark. The numbers generated by this pseudorandom number generator will have range (0, 1) and have very nearly uniform distribution of the unit interval.11-12 The keys will be generated using this number in the following way. Let fr

PAGE 75

64 be a random fraction uniformly distributed in the range (0,1), and let iK be the thi key. The value of iK is determined as 4 / ) (3 4 2 4 1 4 0 4 max i i i i ir r r r B K for 1 ,..., 1 0 N i iK must be an integer and indicates truncation. maxB is 192 for class A and is 212 for class B. For the distribution of keys among memory, all keys initially must be stored in the main memory units and each memory unit must have same amount of keys in a contiguous address space. If the keys cannot be evenly divisible, the last memory unit can have a different amount of keys, but it must follow specification of NAS Benchmark. See Appendix A for details. 4.3 Procedure and Timing The implementation of Kernel IS must follow this procedure. The partial verification of this procedure tests the ranks of five unique keys where each key has unique rank. The full verification rearranges the sequence of keys using the computed rank of each key and tests that the keys are sorted. 1. In a scalar sequential manner and using the key generation algorithm described above, generate the sequence of N keys. 2. Using the appropriate memory ma pping described above, load the N keys into the memory system. 3. Begin timing. 4. Do, for 1 i to maxI (a) Modify the sequence of keys by making the following two changes: i Ki ) (maxmaxi B KI i

PAGE 76

65 (b) Compute the rank of each key. (c) Perform the partial verification test described above. 5. End timing. 6. Perform full verification test described previously. Table 4-1: Parameter Values to Be Used for Benchmark Parameter Class A Class B N 232 252 maxB 192 212 seed 314159265 314159265 maxI 10 10

PAGE 77

66 CHAPTER 5 PARALLEL PATTERNS USED TO IMPLEMENT KERNEL IS In this chapter, we will explain how we used the parallel pattern language to design the parallel program and implement it in an MPI and C environment. 5.1 Finding Concurrency 5.1.1 Getting Started As advised by this pattern, we have to understand the problem we are trying to solve. According to the specification of Kernel IS of NAS Parallel Benchmark, the range of the keys to be sorted is ) 2 0 [19 for class A and a range of ) 2 0 [21 for class B. Because of this range, bucket sort, radix sort, and counting sort can sort the keys in O(n) time.13 These sequential sorting algorithms are good candidate algorithms to parallelize because of their speeds. Because of following reasons, Counting Sort is selected as a target algorithm to parallelize. According to the specification of the benchmark, what will be timed is the time needed to find out the rank of each keys, and the actual permutation of keys occurs after the timing. This means that it is impo rtant to minimize the time needed to find out rank of each key. One interesting characteristic of the three candidate sorting algorithms is following. The Radix Sort and Bucket Sort make permutations of keys to find out rank of each key, in other words, the keys should be sorted to know the rank of every key. But, the Counting Sort does not need to be sorted to find out the rank of every key. This means that the Counting Sort takes less ti me in ranking every key than the others. Because of this reason, the Counting Sort is selected. Another reason is the simplicity of the Counting Sort algorithm.

PAGE 78

67 The basic idea of the Counting Sort algorithm is to determine, for each element x, the number of elements less than x. This information can be used to place element x directly into its position in the output array and the number of elements less than x is 1 ) ( x rank. Counting Sort Algorithm is as follows: Counting-Sort(A, B ,k) // A is an input array. B is an output array. C is an intermediate //array. k is an biggest key in the input array. for 1 i to k do 0 ] [ i C for 1 j to ] [ A length do 1 ]] [ [ ]] [ [ j A C j A C // ] [ i C now contains the number of elements equal to i for 2 i to k do ] 1 [ ] [ ] [ i C i C i C // ] [j C now contains the number of elements less than or equal to i for ] [ A length j down to 1 do ] [ ]]] [ [ [ j A j A C B 1 ]] [ [ ]] [ [ j A C j A C Algorithm 4-1 Counting Sort 5.1.2 Decomposition Strategy The decision to be made in this pattern is whether or not to follow data decomposition or task decomposition pattern. The key data structure of the Counting Sort algorithm is an array of integer keys, and the most compute-intensive part of the algorithm is counting each occurrence of same keys in the integer keys array and summing up the counts to determine the rank of each key. The task-based decomposition is a good starting point if it is natural to think about the problem in terms of a collection of independent (or nearly independent) tasks. We can think of the problem in terms of tasks that count each occurrence of the same keys in key array. So we followed the task decomposition pattern.

PAGE 79

68 5.1.3 Task Decomposition The Task Decomposition pattern suggests findin g tasks in functions, loop, and updates on data. We tried to find tasks in one of th e four loops in the Counting Sort algorithm. We found, using this pattern, that there ar e large enough independent iterations in the second for loop of the counting sort algorithm. These iterations can be divided into many enough relatively independent tasks which find out rank of each key. We can think that each task concurrently counts each occurrence of the same keys on array A. The array A is shared among tasks right now and it is not divided yet. Because of dependency among tasks, this pattern recommends using Dependency Analysis pattern. 5.1.4 Dependency Analysis Using this pattern, data sharing dependencies are found on array A. Each task is sharing the array A because the array A is accessed by each task to read a number in each cell of the array. If one task has read and counted the integer number of a cell, that number should not be read and counted again. Array C is shared, too, because each task access this array to accumulate the count for each number, and one cell must not be accessed or updated concurrently by two or more tasks at the same time. We follow Data-Sharing Pattern since data sharing dependencies are found. 5.1.5 Data Sharing Pattern Array A can be considered read only data among the three categories of shared data according to this pattern, because it is not updated. Array A can be partitioned into subsets, each of which is accessed by only one of the tasks. Therefore Array A falls into effectively-local category.

PAGE 80

69 The shared data, array C, is one of the special cases of read/write data. It is an accumulate case because this array is used to accumulate the counts of each occurrence of the same keys. For this case, this pattern su ggests that each task has a separate copy so that the accumulations occur in these local copies. The local copies are then accumulated into a single global copy as a final step at the end of the computation. What has been done until this point is a parallelization of second loop of Counting Sort algorithm. The fourth for loop can be parallelized as follows: Each task has its own ranks of keys in its own data set so the keys can be sorted in each process. To sort these locally sorted keys, redistribute keys after finding out the range of keys for each task so that each task has approximately same amount of keys and the ranges of keys in ascending order among tasks. Then the tasks of sorting redistributed keys have a dependency on locally sorted keys. But this dependency is read only and effectively local so the locally sorted keys can be redistributed among processes without complication, according to the ranges, because those are already sorted. Then each task can merge the sorted keys in its own range with out dependency using the global count. Therefore, the final design of parallel sorting (the implementation of Kernel IS) follows. First, each task counts each occurrence of the same keys on its own subset data, accumulates the results on its own output array, and then reduces them into one output array. Second, each task redistributes the keys into processes, according to the range of keys of each process, and merges the redistributed keys. 5.1.6 Design Evaluation Our target platform is Ethernet-connected Unix workstations. It is a distributed memory system. This Design Evaluation pattern says that it usually makes it easier to

PAGE 81

70 keep all the processing elements busy by having many more tasks than processing elements. But using more than one UE per PE is not recommended in this pattern when the target system does not provide efficient support for multiple UEs per PEs (Processing Element. Generic term used to reference a hardware element in a parallel computer that executes a stream of instructions), and the Design can not make good use of multiple UEs per PE, which is our case because of reduction of output array (if we have more local output array than the number of PEs, then th e time needed to reduce to one output will take much longer). So our program adjusts the number of tasks into the same number of workstations. This pattern questions simplicity, flexibility, and efficiency of the design. Our design is simple because each generated task will find out rank of each key on its own subset data (keys) and then reduce them into global ranks. The next steps of sorting locally and redistribution of keys and merging the redistributed local key arrays are also simple because we already know the global ranks. It is also flexible and efficient because the number of tasks is adjustable and the computational load is evenly balanced. 5.2 Algorithm Structure Design Space 5.2.1 Choose Structure The Algorithm-Structure decision tree of this pattern is used to arrive at an appropriate Algorithm-Structure for our problem. The major organizing principle of our design is organization by tasks because we used the loop-splitting technique of the Task-D ecomposition pattern, so we arrived at the Organized-by-Tasks branch point. At that branch point, we take a partitioning branch because our ranking tasks can be gathered into a set linear in any number of dimensions.

PAGE 82

71 There are dependencies between tasks and it is an associative accumulation into shared data structures. Therefore we arrived at the Separable-Dependencies pattern. 5.2.2 Separable Dependencies A set of tasks that will be executed concurrently in each processing unit corresponds to iterations of a second for loop and third for loop of Algorithm 4.1. Each task will be independent of each other because each task will use its own data. This pattern recommends balancing the load at each PE. The size of each data is equal because of data distribution specification of Kernel IS so the load is balanced at each PE. The specification of keys distribution of Kernel IS is also satisfied. The fact that all the tasks are independent leads to the Embarrassingly Parallel Pattern. 5.2.3 Embarrassingly Parallel Each task of finding all the ranks of distinct key array can be represented as a process. Because each task will have almost same amount of keys, each task will have same amount of computation. So the static schedu ling of the tasks will be effective as advised by this pattern. For the correctness considera tions of this pattern, each tasks solve the subproblem independently, solve each subproblem exactly once, correctly save subsolutions, and correctly combine subsolutions. 5.3 Using Implementation Example The design of the parallel in teger sort problem satisfies the condition of simplest form of parallel program as follows: All the tasks are independent. All the tasks must be completed. Each task executes same algorithm on a distinct section of data. Therefore the resulted design is a simplest form of Embarrassingly Parallel Pattern. The Implementation example is provided in Chapter 3. Using the Implementation Example of the Embarrassingly Parallel pattern, the Kern el IS of NAS Parallel Benchmark set can be

PAGE 83

72 implemented. The basic idea of Implementation Example of Embarrassingly Parallel pattern is that of scattering data to each process, compute subsolutions, and then combine subsolutions into a solution for the problem. If we apply the Implementation Example two times, one time for finding rank of each key and one for sorting the keys in local processes, the Kernel IS can be impelemented. Another method of implementation is to use Implementation Example of Divide Conquer pattern. But this Implementation Example is not chosen because it is hard to measure the elapsed time for ranking all the keys. 5.4 Algorithm for Parallel Implementation of Kernel IS The following algorithm illustrates the parallel design for implementation of Kernel IS (parallel sort over small integer) that has been obtained through the parallel design patterns. 1. Generate the sequence of N keys using key generation algorithm of NAS Parallel Benchmarks. 2. Divide the keys by the number of PEs and distribute to each memory of PEs. 3. Begin timing. 4. Do, for 1 i to maxI (a) Modify the sequence of keys by making the following two changes: i Ki ) (maxmaxi B KI i (b) Each task (process) finds out the rank of each key in its own data set (keys) using ranking algorithm of counting sort. (c) Reduce the arrays of ranks in its own local data into an array of ranks in global data. (d) Do partial verification.

PAGE 84

73 5. End timing. 6. Perform permutation of keys according to the ranks. (a) Each task sorts local keys accord ing to the ranks among its local keys. (b) Compute the ranges of keys each process will have so that each process will have nearly same amount of keys and the ranges of keys are in ascending order (c) Redistribute keys among processes a ccording to the range of each process. (d) Each task (process) merges its redistributed key arrays. 7. Perform full verification.

PAGE 85

75 CHAPTER 6 PERFORMANCE RESULTS AND DISCUSSIONS 6.1 Performance Expectation An ideal execution time for a parallel program can be expressed as T/N where T is the total time taken with one processing element and N is the number of processing Elements used. Our implementation of Kernel IS of NAS Parallel Benchmarks will not have an ideal execution time because of overhead that comes from several sources. A source of overhead comes from the computations needed to reduce local arrays of ranks into one array of global ranks. The more local arrays of ranks and processing elements we use, the more computation time will be needed. The gap between the ideal execution time and the actual execution time will be increased. Another source of overhead will be the communication because MPI uses message transfer, which typically involves both overhead due to kernel calls and latency due to the time it takes the message to travel over the network. 6.2 Performance Results The Kernel IS implementation was executed on top of Ethernet-connected workstations and LAM 6.3.2, which is an implementation of MPI.14 These workstations are Sun Blade 100 with 500-MHz UltraSPARC-IIe cpu, 256-KB L2 External Cache, 256MB DIMM memory, Ethernet/Fast Ethernet, and twisted pair standard (10BASE-T and 100BASE-T) self-sensing network. Figure 6.1 shows the performance results for class A and B. The rows show the total number of processing elements (workstations) used to

PAGE 86

75 execute Kernel IS implementation. The column s show the execution time of Kernel IS implementation and an ideal execution time in milliseconds for each class A and B. The performance result of NPB2.2 (NAS Parallel Benchmark 2.2. These are MPI-based source-code implementations written and distri buted by NAS. They are intended to be run with little or no tuning, and they appr oximate the performance a typical user can expect to obtain for a portable parallel prog ram. They supplement, rather than replace, NPB 1. The NAS solicits performance results from all sources) for class A and B are shown for comparison purpose.15 Ta ble 6-1 Performance Results for Class A and B Number of Processing Elements Actual Execution Time for class A Ideal Execution Time for Class A Execution Time of NPB2.2 for Class A Actual Execution Time for Class B Ideal Execution Time for Class B Execution Time of NPB2.2 for ClassB 1 30940 30940 20830 147220 147220 N/A 2 16500 15470 25720 76100 73610 1446270 3 12160 10313 N/A 55640 49073 N/A 4 10600 7735 16870 46500 36805 103110 5 9200 6188 N/A 41150 29444 N/A 6 8150 5156 N/A 36550 24536 N/A 7 7600 4420 N/A 33350 21031 N/A 8 7080 3867 9720 30200 18402 42760 9 7470 3437 N/A 31790 16357 N/A 10 7040 3094 N/A 30860 14722 N/A 11 6820 2812 N/A 28740 13383 N/A 12 6750 2578 N/A 27900 12268 N/A 13 6690 2380 N/A 27580 11324 N/A 14 6320 2210 N/A 26630 10515 N/A 15 6000 2062 N/A 25560 9814 N/A 16 5780 1933 16680 25230 9201 62080 The execution times for Class B when the number of processing elements is 1 and 2 are not shown in Figure 6.2 because execution time for Class B is too big to show in Figure 6.2. The reason for that long execution time is that NPB2.2 consumes much more

PAGE 87

76 memory than the physical memory, which leads to many I/O between the hard disk and main memory of workstations. Figure 6-1 Execution Time Comparison for Class A Figure 6-2 Execution Time Comparison for Class B 6.3 Discussions As seen from the previous graphs, the following conclusions can be made about the performance of the parallel implementation of Kernel IS of NAS Parallel Benchmarks. The most increase in performance is achieved by using one or two additional processing

PAGE 88

77 elements, that is when the total number of pr ocessing elements is 2 or 3 respectively and there are smaller improvements by using more processing elements. The more processing elements we use for computation, the more overhead was created and the gap between the ideal execution time and the actual execution time increased, as we expected. The overall performance is acceptable because it has better performance compared to the performance of NPB2.2

PAGE 89

78 CHAPTER 7 RELATED WORK AND CONCLUSIONS AND FUTURE WORK 7.1 Related work 7.1.1 Aids for Parallel Programming Considerable research has been done to ease the tasks of designing and implementing efficient parallel program. The related work can be categorized into a program skeleton, a program framework, and design patterns. An algorithmic skeleton was introduced by M. Cole as a part of his proposed parallel programming systems (language) for parallel machines.16 He presented four independent algorithmic skeletons which are fixed degree divide and conquer, iterative combination, clustering, and task queues. Each of skeletons describes the structure of a particular style of algorithm in terms of abstraction. These skeletons capture very high-level patterns and can be used as an overall program structure. The user of this proposed parallel programming system must choose one of these skeletons to describe a solution to a problem as an instance of the appropriate skeleton. Because of this restriction, these skeletons can not be applied to every problem. These skeletons are similar to the patterns in the algorithm structure design space of parallel pattern language in a sense that both are an overall program structure and provide algorithmic frameworks. But these skeletons provide less guidance for an inexperienced parallel programmer about how to arrive at one of these skeletons in comparison with parallel pattern language. Program Frameworks, which can address overall program structure but are more detailed and domain specific, are widely used in many areas of computing.17 In a parallel

PAGE 90

79 computing area, Parallel Object Oriented Methods and Applications (POOMA) and Parallel Object-oriented Environment and Toolkit (POET) are examples of frameworks from Los Alamos and Sandia, respectively.18-19 The POOMA is an object-oriented framework for a data-parallel programming of scientific applications. It is a library of C++ classes designed to represent common abstractions in these applications. Application programmers can use and derive from these classes to express the fundamental scientific content and/or numerical methods of their problem. The objects are layered to represent a data-parallel programming interface at the highest abstraction layer whereas lower, implementation layers encapsulate distribution and communication of data among processors. The POET is a framework for encapsulating a set of user-written components. The POET framework is an object model, implemented in C++. Each user-written component must follow the POET template interface. The POET provides services, such as starting the parallel application, running components in a specified order, and distributing data among processors. In recent years, Launay and Pazat developed a framework for parallel programming using Java.20 This framework provides a parallel programming model embedded in Java by a framework and intended to separate co mputations from control and synchronizations between parallel tasks. Doug Lea provided design principles to build concurrent applications using the Java parallel programming facilities.21 Design patterns that can address many levels of design problem have been widely used with object oriented sequential programming. Recent work of Douglas C. Schmidt, et al

PAGE 91

80 addresses issues associated with concurrency and networking but mostly at a fairly low level.22 Design pattern, frameworks, and skeleton share the same intent of easing parallel programming by providing libraries, classes, or a design pattern. In comparison with parallel pattern language, which provides a systemic design patterns and implementation patterns for parallel program design and implementation, frameworks and skeletons might have more efficient implementations or better performance in their specialized applicable problem domain. But parallel pattern language could be more helpful for inexperienced parallel programmer in designing and solving more general application problems because of its systematic structure in exploiting concurrency and providing frameworks and libraries down to implementation level. 7.1.2 Parallel Sorting Parallel sorting is one of the most widely studied problems because of its importance in wide variety of practical applications. Various parallel sorting algorithms have also been proposed in the literature. Guy E. Blelloch, et al. analyzed and evaluated many of the parallel sorting algorithms proposed in the literature to implement as fast a general purpose sorting algorithm as possible on the Connection Machine Supercomputer model CM-2.23 After the evaluation and analysis, the researchers selected the three most promising alternatives for implementation: bitonic sort that is a parallel merge sort, parallel version of counting-based radix sort, and a theoretically efficient randomized algorithm, sample sort. According to their e xperiments, sample sort was the fastest on large data sets. Andrea C. Dusseau, et al. analyzed these three parallel sorting algorithms and column sort using a LogP model, which characterizes the performance of modern parallel

PAGE 92

81 machines with a small set of paramete rs: the communication latency, overhead, bandwidth, and the number of processes.24 They also compared performances of Split-C implementation of the four sorting algorithms on message passing, distributed memory, massively parallel machine, CM-5. In their comparison, radix and sample sort was faster than others on a large data set. To understand the performance of parallel sorting on hardware cache-coherent shared address space (CC-SAS) multiprocessors, H. Shan et al. investigated the performance of two parallel sorting algorithms (radix, samp le sort) under three major programming models (a load-store CC-SAS, message passing, and the segmented SHMEM model) on a 64 processor SGI Origin2000 (A scalable, hardware-supported, cache-coherent, nonuniform memory access machine).25 In their investigation, the researchers found that sample sort is generally better than radix sort up to 64k integers per processor, and radix sort is better after that point. The best combination of algorithm and programming models are sample sort under the CCSAS for smaller data sets and radix sort under SHMEM for larger data sets, accordin g to their investigation. Communication is fundamentally required in a parallel sorting problem as well as computation. Because of this characteristic and the importance of sorting in applications, parallel sorting problem has been selected as one of kernel benchmarks for the performance evaluation of various parallel computing environments.26 The Kernel IS of NAS Parallel Benchmark set has been implemented and reported its performance on various parallel supercomputers by its vendors.27-28 7.2 Conclusions This thesis shows how the parallel design patterns were used to develop and implement a parallel algorithm for Kernel IS (Parallel sort over large integers) of NAS

PAGE 93

82 Parallel Benchmarks as a case study. And it presented reusable frameworks and examples for implementations of patterns in the algorithm structure design space of parallel pattern language. Chapter 6 showed the performance results for Kernel IS. As the result shows, the parallel design patterns help developing a parallel program with relative ease, and it is also helpful in reasonably improving the performance. 7.3 Future Work Parallel pattern language is an ongoing project. Mapping design patterns with various parallel computing environments and developing frameworks for object oriented programming systems can be considered as future research. Testing the resulted implementation of Kernel IS using parallel pattern language on various supercomputers, comparing this algorithm as a full sorting, not just ranking, and more case studies of the parallel application program using parallel pattern language can also be included in the future work.

PAGE 94

83 APPENDIX A KERNEL IS OF THE NAS PARALLEL BENCHMAKRK A.1 Brief Statement of Problem Sort N keys in parallel. The keys are generated by the sequential key generation algorithm given below and initially must be uniformly distributed in memory. The initial distribution of the keys can have a great impact on the performance of this benchmark, and the required distribution is discussed in detail below. A.2 Definitions A sequence of keys,} 1 ,..., 1 0 | { N i Ki, will be said to be sorted if it is arranged in non-descending order, i.e., ...2 1 i i iK K K The rank of a particular key in a sequence is the index value I that the key would have if the sequence of keys were sorted. Ranking, then, is the process of arriving at a rank for all the keys in a sequence. Sorting is the process of permuting the keys in a sequence to produce a sorted sequence. If an initially unsorted sequence, 1 1 0,..., NK K K has ranks ) 1 ( ),... 1 ( ), 0 ( N r r r, the sequence becomes sorted when it is rearranged in the order ) 1 ( ) 1 ( ) 0 (,..., N r r rK K K. Sorting is said to be stable if equal keys retain their original relative orde r. In other words, a sort is stable only if ) ( ) ( j r i r whenever ) ( ) ( j r i rK K and j i Stable sorting is not required for this benchmark. A.3 Memory Mapping The benchmark requires ranking an unsorted sequence of N keys. The initial sequence of keys will be generated in an unambiguous sequential manner as described below. This

PAGE 95

84 sequence must be mapped into the memory of parallel processor in one of the following ways depending on the type of memory syst em. In all cases, one key will map to one word of memory. Word size must be no less than 32 bits. Once the keys are loaded onto the memory system, they are not to be removed or modified except as required by the procedure described in the Procedure subsection. A.4 Shared Global Memory All N keys initially must be stored in a contiguous address space. If iA is used to denote the address of the thiword of memory, then the address space must be ] [1 N i iA A. The sequence of keys,1 1 0,..., NK K K initially must map to this address space as ) (j j iK MEM A for 1 ,..., 1 0 N j (A.1) where ) (jK MEM refers to the address of jK A.5 Distributed Memory In a distributed memory system with p distinct memory units, each memory unit initially must store pNkeys in a contiguous address space, where p N Np/ (A.2) If iA is used to denote the address of the thi word in a memory unit, and if jP is used to denote the thj memory unit, then i jA P will denote the address of the thi word in the thj memory unit. Some initial addressing (or ordering) of memory units must be assumed and adhered to throughout the benchmark. Note that the addressing of the memory units is left completely arbitrary. If N is not evenly divisible by p then

PAGE 96

85 memory units } 2 ,..., 1 0 | { p j Pj will store pN keys, and memory unit 1 pP will store ppN keys, where now 5 0 / p N Np p ppN p N N) 1 ( In some cases (in particular if p is large) th is mapping may result in a poor initial load balance with p ppN N. In such cases it may be desirable to use 'p memory units to store the keys, wherep p '. This is allowed, but the storage of the keys still must follow either equation 2.2 or equation 2.3 with 'p replacing p In the following we will assumeN is evenly divisible by p The address space in an individual memory unit must be 1, pN i iA A. If memory units are individually hierarchical, then pN keys must be stored in a contiguous address space belonging to a single memory hierarchy and iA then denotes the address of the thi word in that hierarchy. The keys cannot be distributed among different memory hierarchies until after timing begins. The sequence of keys,1 1 0,..., NK K K, initially must map to this distributed memory as ) (j kN j i kpK MEM A P for 1 ,..., 1 0 pN j and 1 ,..., 1 0 p k where ) (j kNpK MEM refers to the address of j kNpK. If N is not evenly divisible by p then the mapping given above must be modified for the case where 1 p k as ) () 1 ( 1j N p j i ppK MEM A P for 1 ,..., 1 0 ppN j (A.3) A.6 Hierarchical Memory All N keys initially must be stored in an address space belonging to a single memory hierarchy which will here be referred to as the main memory. Note that any memory in

PAGE 97

86 the hierarchy which can store all N keys may be used for the initial storage of the keys, and the use of the term main memory in the description of this benchmark should not be confused with the more general definition of this term in section 2.2.1. The keys cannot be distributed among different memory hierarchies until after timing begins. The mapping of the keys to the main memory must follow one of either the shared global memory or the distributed memory mappings described above. The benchmark requires computing the rank of each key in the sequence. The mappings described above define the initial ordering of the keys. For shared global and hierarchical memory systems, the same mapping must be applied to determine the correct ranking. For the case of a distributed memory system, it is permissible for the mapping of keys to memory at the end of the ranking to differ from the initial mapping only in the following manner: The number of keys mapped to a memory units at the end of the ranking may differ from the initial value, pN. It is expected, in a distributed memory machine, that good load balancing of the problem will require changing the initial mapping of the keys and for this reason a different mapping may be used at the end of the ranking. If kpN is the number of keys in memory unit kP at the end of ranking, then the mapping which must be used to determine the correct ranking is given by )) ( (j kN r MEM A Pkp j i k for 1 ,..., 1 0 kpN j and 1 ,..., 1 0 p k where ) (j kN rkp refers to the rank of key j kNk pK. Note, however, this does not imply that the keys, once loaded into memory, may be moved. Copies of the keys may be maid and moved, but the original sequence must remain intact such that each time the ranking process is repeated (Step 4 of Procedure) the original sequence of keys exist (except for the two modification of Step 4a) and the same algorithm for ranking is applied.

PAGE 98

87 Specifically, knowledge obtainable from the communications pattern carried out in the first ranking cannot be used to speed up subsequent rankings and each iteration of Step 4 should be completely independent of the previous iteration. A.7 Key Generation Algorithm The algorithm for generating the keys makes use of the pseudorandom number generator described in section 2.2. The keys will be in the range max, 0B. Let fr be a random fraction uniformly distributed in the range 1 0 and let iK be the thi key. The value of iK is determined as 4 / ) (3 4 2 4 1 4 0 4 max i i i i ir r r r B K for 1 ,..., 1 0 N i. (A.4) Note that iK must be an integer and indicates truncation. Four consecutive pseudorandom numbers from the pseudorandom number generator must be used for generating each key. All operations before the truncation must be performed in 64-bit double precision. The random number generator must be initialized with 314159265 s as a starting seed. A.8 Partial Verification Test Partial verification is conducted for each ranking performed. Partial verification consists of comparing a particular subset of ranks with the reference values. The subset of ranks and the reference values are given in table 2.1. Note that the subset of ranks is selected to be invariant to the ranking algorithm (recall that stability is not required in the benchm ark). This is accomplished by selecting for verification only the ranks of unique keys. If a key is unique in the sequence (i.e., there is no other equal key), then it will have a unique rank despite an unstable ranking algorithm. The memory mapping described in the Memory Mapping subsection must be applied.

PAGE 99

88 Table A-1 : Values to be used for partial verification Rank (full) Full scale Rank (sample) Sample code ) 2112377 (r i 104 ) 48427 (r i 0 ) 662041 (r i 17523 ) 17148 (r i 18 ) 5336171 (r i 123928 ) 23627 (r i 346 ) 3642833 (r i 8288932 ) 62548 (r i 64917 ) 4250760 (r i 8388264 ) 4431 (r i 65463 A.9 Full Verification Test Full verification is conducted after the last ranking is performed. Full verification requires the following: 1. Rearrange the sequence of keys, } 1 ,..., 1 0 | { N i Ki, in the order )} 1 ( ),..., 1 ( ), 0 ( | { N r r r j Kj, where ) 1 ( ),..., 1 ( ), 0 ( N r r r is the last computed sequence of ranks. 2. For every iK from 2 ... 0 N i test that 1 i iK K If the result of this test is true, then the keys are in sorted order. The memory mapping described in the Memory Mapping subsection must be applied. A.10 Procedure 1. In a scalar sequential manner and using key generation algorithm described above, generate the sequence of N keys. 2. Using the appropriate memory mapping described above, load the N keys into the memory system. 3. Begin timing. 4. Do, for 1 i to maxI (a) Modify the sequence of keys by making the following two changes: i Ki ) (maxmaxi B KI i

PAGE 100

89 (b) Compute the rank of each key. (c) Perform the partial verification test described above. 5. End timing. 6. Perform full verification test described above. Table A-2: Parameter values to be used for benchmark Parameter Class A Class B N 232 252 maxB 192 212 seed 314159265 314159265 maxI 10 10 A.11 Specifications The specifications given in table A.2 shall be used in the benchmark. Two sects of values are given, one for Class A and one for Class B.

PAGE 101

90 APPENDIX B A PSEUDORANDOM NUMBER GENERATOR FOR THE PARALLEL NAS KERNELS B.1 Pseudorandom Number Generator Suppose that n uniform pseudorandom numbers are to be generated. Set 135 a and let s x0 be a specified initial “seed,” i.e., an integer in the range 462 0 s. Generate the integers kx for n k 1 using the linear congruential recursion ) 2 (mod46 1 k kax x and return k kx r462 as the result. Thus 1 0 kr, and the kr are very nearly uniformly distributed distribution on the unit interval. See [2], beginning on page 9 for further discussion of this type of pseudorandom number generator. Note that any particular value kx of the sequence can be computed directly from the initial seed s by using the binary algorithm for exponentiation, taking remainders modulo 462 after each multiplicati on. To be specific, let m be the smallest integer such that km 2, set s b anda t. Then repeat the following for i from 1 to m: 2 /k j ) 2 (mod46bt b if k j 2 ) 2 (mod46 2t t j k

PAGE 102

91 The final value of b is s a xk k (mod 462). See [2] for further discussion of the binary algorithm for exponentiation. The operation of multiplying two large integers modulo 462 can be implemented using 64 bit floating point arithmetic by splitting the arguments into two words with 23 bits each. To be specific, suppose one wishes to compute ) 2 (mod46ab c. Then perform the following steps, where int de notes the greatest integer: ) 2 int(23 1a a 1 23 22a a a ) 2 int(23 1b b 1 23 22b b b 1 2 2 1 1b a b a t ) 2 int(1 23 2t t 2 23 1 32t t t 2 2 3 23 42b a t t ) 2 int(4 46 5t t 5 46 42t t c An implementation of the complete pseudorandom number generator algorithm using this scheme produces the same sequence of results on any system that satisfies the following requirements: The input multiplier a and initial seed s as well as the constants 232, 232, 462 and 462, can be represented exactly as 64 bit floating point constants.

PAGE 103

92 The truncation of a nonnegative 64 bit floating point value less than 242 is exact. The addition, subtraction and multiplication of 64 bit floating point values, where the arguments and results are nonnegative whole numbers less than 472, produce exact results. The multiplication of a 64 bit floating point value, which is a nonnegative whole number less than 472, by the 64 bit floating point value m2, 46 0 m, reduces and exact result. These requirements are met by virtually all scientific computers in use today. Any system based on the IEEE-754 floating point arithmetic standard [1] easily meets these requirements using double precision. However, it should be noted that obtaining an exact power of two constant on some systems requires a loop rather than merely an assignment statement with. B.2 Other Features The period of this pseudorandom number generator is 13 4410 76 1 2 and it passes all reasonable statistical tests. This calculation can be vectorized on vector computers by generating results in batches of size equal to the hardware vector length. By using the scheme described above for computing kx directly, the starting seed of a particular segment of the sequence can be quickly and independently determined. Thus numerous separate segments can be generated on separate processors of a multiprocessor system. Once the IEEE-754 floating point arithmetic standard gains universal acceptance among scientific computers, the radix 462 can be safely increased to 522 although the

PAGE 104

93 scheme described above for multiplying two such numbers must be correspondingly changed. This will increase the period of the pseudorandom sequence by a factor of 64 to approximately.

PAGE 105

94 APPENDIX C SOURCE CODE OF THE KE RNEL IS IMPLEMENTATION /* This program is an implementation of Kernel IS(sorting over small integers of NAS Parallel benchmark set. */ #include #include #include #include #define MAX_KE Y 524288 /*maximum value of key */ #define NUM_ KEYS 8388608 /*number of keys to be sorted */ #define TEST_ARRAY_SIZE 5 int partialVerifyVals[TEST_ARRAY_SIZE]; int testIndexArray[TEST_ARRAY_SIZE] = {2112377,662041, 5336171,3642833,4250760}; int testRankArray[TEST_ARRAY_ SIZE] ={104,17523 ,123928,8288932,8388264}; int passedVerification = 0; /* the number of key s each processing element has */ int numOfLocalKeys; /* rank of each key */ int* countGlobal; /* rank of each key in local key subset */ int count[MAX_KEY+1]; /* a subset of keys th at each process has*/ int* localKeyArray;

PAGE 106

96 /* Sorted subset keys before redistribution */ int* tempResultArray; /* Final sored keys */ int* finalResultArray; /* rank of each process */ int myRank; /* total number of processes */ int mySize; /*************************************/ /* RANKING */ /*************************************/ void ranking(){ long i; long j; for(i=0;i
PAGE 107

97 2. Test whether the keys are in sored order. *******************************/ void full_verify(){ int i, j; int temp; /* The number of keys th at each process will have at last */ int lastNumOfKeysLocal; /* max value in each process before key redistributtion */ int maxValueInPE[mySize]; /* number of keys afte r key redistribution */ int newNumOfKeysLocal[mySize]; /* number of keys that will be send to each process */ int numOfKeysRedis[mySize]; /* number of keys that each process will r eceive from other proc esses and itself */ int numOfKeysToRecv[mySize]; /* Entry i specifies the disp lacement relative to sendbuffer from which to take the outgoing data destined for process j*/ int senddispls[mySize]; /* Entry j specifies the displacement rela tive to receivebuffer at which to place the incoming data from process i*/ int recvdispls[mySize]; int arrayTemp[mySize]; int *lastArray; int accum[mySize]; MPI_Bcast(countGlobal,MAX_KE Y+1,MPI_INT,0,MPI_COMM_WORLD); /* Sorting keys locally in process */ for(j=numOfLocalKeys-1;j>=0;j--){

PAGE 108

98 tempResultArray[count[localKeyArray[j]]-1]=localKeyArray[j]; count[localKeyArray[j]]=count[local KeyArray[j]]-1; } /* KEY REDISTRIBUTION */ /* finding out maximum number and number of k eys that each process will have afte r key redistribution */ if(myRank == 0){ i=0; int tempN = 0; for(j=1;j= nu mOfLocalKeys+tempN){ maxValueInPE[j-1]=i;/* maxValueInPE */ newNumOfKeysLocal[j-1] = countGlobal[i-1]-tempN; tempN = tempN + newNumOfKeysLocal[j-1]; break; } } } maxValueInPE[j-1]= MAX_KEY; newNumOfKeysLocal[j-1]=NUM_KEYS tempN; tempN=0; for(i=0;i
PAGE 109

99 lastNumOfKeysLocal= newNumOfKeysLocal[myRank]; lastArray = (int *) malloc(lastNumOfKeysLocal*sizeof(int)); /* compute the number of keys to send to each process and displacement(starting point of send buffer for each pr ocess) */ i=0; temp=0; for(i;i
PAGE 110

100 for(i=0;i=0;j--){ lastArray[countGlobal [finalResultArray[j]] -accum[myRank]]=fin alResultArray[j]; countGlobal[finalResultArray[j ]]=countGlobal[finalR esultArray[j]]-1; } /* Test wheterher the keys are in sorted order*/ j = 0; for( i=1; i lastArray[i] ){ j++; } if( j != 0 ){ printf( "Full_verify: number of keys out of so rt: %d\n", j ); } else printf("Full verifica tion was successful\n"); } /*************************************/ /* Partial Verification Test */ /*************************************/ partialVerify(int itera tion,int* countGlobal){ int i=0; int k=0; int j=0; passedVerification=0; for( i=0; i
PAGE 111

101 printf( "Failed partial verification: "iteration %d, test key %d ,\n", iteration, i ); printf("%d %d\n", countGlobal[k-1],testRankA rray[i]+iteration-1); } else passedVerification++; } else{ if( countGlobal[k-1] != te stRankArray[i] -iteration+1){ printf( "Failed partial verification: "iteration %d, test key %d\n", iteration, i ); printf("%d %d\n", countGlobal[k-1],testRankA rray[i]-iteration+1); } else passedVerification++; } if(passe dVerification == 5){ printf("Partial verifica tion was successful\n"); } } } } /**************************************************/ /* Genterate uniform ly distributed random number */ /**************************************************/ double randlc(X, A) double *X; double *A; { static int KS=0; static double R23, R46, T23, T46; double T1, T2, T3, T4; double A1;

PAGE 112

102 double A2; double X1; double X2; double Z; int i, j; if (KS == 0){ R23 = 1.0; R46 = 1.0; T23 = 1.0; T46 = 1.0; for (i=1; i<=23; i++){ R23 = 0.50 R23; T23 = 2.0 T23; } for (i=1; i<=46; i++){ R46 = 0.50 R46; T46 = 2.0 T46; } KS = 1; } T1 = R23 *A; j = T1; A1 = j; A2 = *A T23 A1; T1 = R23 *X; j = T1; X1 = j; X2 = *X T23 X1; T1 = A1 X2 + A2 X1; j = R23 T1; T2 = j;

PAGE 113

103 Z = T1 T23 T2; T3 = T23 Z + A2 X2; j = R46 T3; T4 = j; *X = T3 T46 T4; return(R46 *X); } /****************************************/ /* Generate a sequence of integer keys */ /****************************************/ void create_seq( doub le seed, double a,in t* key_array ){ double x; int i, j, k; k = MAX_KEY/4; for (i=0; i< NUM_KEYS; i++){ x = randlc(&seed, &a); x += randlc(&seed, &a); x += randlc(&seed, &a); x += randlc(&seed, &a); key_array[i] = k*x; } } /*************************************/ /* MAIN */ /*************************************/ void main(int argc ,char* argv[]){ int i = 0; int j = 0; int* key_array; int* resultArray; /* sorted integer */ clock_t start; clock_t end;

PAGE 114

104 double timecounter; double maxtime; int numOfKeyLastNode; MPI_Status status; MPI_Request request; MPI_Init(&argc,&argv); MPI_Comm_rank(MPI_ COMM_WORLD,&myRank); MPI_Comm_size(MPI_COMM_WORLD,&mySize); /* compute the numb er of keys that each process will have according to the memo ry mapping of Kernel IS */ numOfLocalKeys=( int)((double)(NUM_KEYS /mySize) + 0.5); numOfKeyLastNode = NUM_KEYS-(numOfLocalKeys*(mySize-1)); if(myRank == mySize -1){ numOfLocalKeys = numOfKeyLastNode; } countGlobal= (int *) mall oc((MAX_KEY+1)*si zeof(int)); localKeyArray =(int *) malloc(numOfLocalKeys*sizeof(int)); tempResultArray = (int *) malloc(numOfLocalKeys*sizeof(int)); /* Process 0 (root proc ess) will distribute k eys to each process */ if(myRank==0){ int i = 0; int j = 0; key_array = (int *) malloc(NUM _KEYS*sizeof(int)); resultArray= (int *) malloc(NUM_KEYS* sizeof(int)); /****Generating Keys to be sorted *********************************/ printf("Generatin g Random Numbers...\n"); create_seq( 314159265.00, /* Random number gen seed */ 1220703125.00,key_array ); printf("Creat ed Sequence\n"); }

PAGE 115

105 /*** SCATTER *******************************************/ MPI_Scatter(key_array,numOfLocalKeys,MPI_INT,localKeyArray, numOfLocalKeys,MPI_INT ,0,MPI_COMM_WORLD); /*********************************************************/ /* Synchronize all pr ocesses before timing*/ MPI_Barrier(MPI_COMM_WORLD); start=clock(); for(i=1;i<11;i++){ /* Modify the sequence of keys */ if(myRank==0){ localKeyArray[i] = i; localKeyArray [i+10] = MAX_KEY i; for( j=0; j< TEST_ARRAY_SIZE; j++ ) partialVerifyVals[j] = key_array[testIndexArray[j]]; } /* Compute the rank of each key */ ranking(); /*** REDUCE **************************************************/ MPI_Reduce(coun t,countGlobal,MAX_KEY+1,MPI_INT,MPI_SUM, 0,MPI_COMM_WORLD); /*** REDUCE **************************************************/ /* Partial Verification */ if(myRank==0) partialVerify(i,countGlobal); } MPI_Barrier(MPI_COMM_WORLD);

PAGE 116

106 /* End Timing*/ end = clock(); timecounter = ((double) (end start)) 1000/CLO CKS_PER_SEC; MPI_Reduce( &timecounter, &maxtime, 1, MPI_DOUBLE, MPI_MAX, 0, MPI_COMM_WORLD ); if(myRank==0){ printf("star t:%ld\n",start); printf("en d:%ld\n",end); printf("Sorting pro cess took %f mill iseconds to execute\n",maxtime); } /* Full verification */ full_verify() ; MPI_Finalize(); }

PAGE 117

107 APPENDIX D SOURCE CODE OF PIP ELINE EXAMPLE PROBLEM #include #include #include #include int numOfCalElem = 20; char startMsg[7]; int myRank; int mySize; int tag = 1; char* sendBuf, recvBuf; double sendBufSize, recvBufSize; int numOfStudent = 1000; MPI_Status status; /* Only four pipeline stag es are implemented in this Example Implementation Code. More elements can be added or removed, according to the design of the para llel program */ /***********************/ /*First Pipeline Stage */ /***********************/ void firstPipelineStage(MPI_Comm myComm) { /*=IMPLEMENTATION1==============================*/ /*= =*/ /*= code for this Pipeline Stage should be impl emented here =*/

PAGE 118

108 double scores[numOfStudent]; int i=0; double sum =0; double av erage = 0; for(i=0;i
PAGE 119

109 /*=========================================*/ myRank+1,tag ,MPI_COMM_WORLD); } /********************/ /* Pipeline Stage 2 */ /********************/ void pipelineStage2(MPI_Comm myComm) { double scores[numOfStudent]; double average=0; /* Receive message from the previous Pipeline Stage of process with myRank-1 */ MPI_Recv(startMsg,strlen(startMsg),MPI_CHAR,myRank-1, tag, MPI_COMM_WORLD,&status); /*=DATA TRANSFER 2================================*/ /*= More receive functions, whic h has same structure, can be add ed =*/ /*= to transfer data =*/ MPI_Recv( /*= Modify the following paramete rs =*/ scores, /*=
PAGE 120

110 /*= =*/ /*= code for this Pipeli ne Stage should be implemented here =*/ int i=0; for(i=0;i
PAGE 121

111 /*= More receive functions which has same structure, can be added =*/ /*= to transfer data =*/ MPI_Recv( /*= Modify the following para meters =*/ scores, /*=
PAGE 122

112 /* Last Pipeline Stage */ /***********************/ void lastPipelineStage(MPI_Comm myComm) { double scores[numOfStudent]; double stdDeviation; /* Receive messag e from previous Pipeline Stage of process with myRank-1 */ MPI_Recv(startMsg,strlen(startMsg),MPI_CHAR,myRank-1, tag, MPI_COMM_WORLD,&status); /*=DATA TRANSFER 2==================================*/ /*= More receive functions, which ha s same structure, can be added =*/ /*= to transfer data =*/ MPI_Recv( /*= Modify the following pa rameters =*/ scores, /*=
PAGE 123

113 } void main(argc,argv ) int argc; char **argv; { int i = 0; MPI_Comm myComm; MPI_Init(&argc, &argv); MPI_Comm_dup(MPI_COMM_WORLD, &myComm); /*find rank of this process*/ MPI_Comm_rank(myComm, &myRank); /*find out rank of last process by usin g size of rank*/ MPI_Comm_size(myComm,&mySize); strcpy(startMsg,"start"); switch(myRank) { case 0 : for(i=0 ;i
PAGE 124

114 for(i=0 ;i
PAGE 125

115 APPENDIX E SOURCE CODE OF DIVIDE A ND CONQUER EXAMPLE PROBLEM #include #include #include #define DATA_TYPE int int numOfProc; /*number of available processes*/ int my_rank; int ctrlMsgSend; int ctrlMsgRecv; int* localData; int mergedInt[200]; int dataSizeSend; int dataSizeRecv; int maxInt = 200; int dataSizeLeft; int numOfIntTo Sort = 200; int* integer int* count; int* temp; /********************/ /* Solve a proble m */ /********************/ void solve(int numOfProcLeft) {

PAGE 126

116 if(baseCase(numOfProcLeft)) { baseSolve(numOfProcLeft); merge(numOfProcLeft); } else { split(numOfProcLeft); if(numOfProcLeft!=numOfProc) { merge(numOfProcLeft); } } } /*****************************************************/ /* split a problem into two subproblems */ /*****************************************************/ int split(int numOfProcLeft) { /*=IMPLEMENTATION2 ============================*/ /*= Code for splitting a problem into two subproblem s =*/ /*= should be implemente d =*/ dataSizeSend = dataSizeRecv/2; dataSizeLeft = dataSi zeRecv-dataSizeSend; dataSizeRecv = dataSizeLeft; ctrlMsgSend = numOfProcLeft/2; /*= =*/ /*==========================================*/ /* invoke a solve f unction at the remo te process */ MPI_Send(&ctrlM sgSend,1,MPI_INT, my_rank+numOfP roc/numOfProcLeft, 0,MPI_COMM_WORLD); /*=DATA TRANSFER 1 ==========================*/

PAGE 127

117 /*= More of this bloc k can be added on needed ba sis =*/ MPI_Send(&dataSizeSend,1,MPI_INT, my_rank+numOfP roc/numOfProcLeft, 0, MPI_COMM_WORLD); MPI_Send( &localData[dataSizeLeft], /*= numOfP roc/(numOfProcLeft*2)) { ctrlMsgSend = 1; dataSizeSend = dataSizeLeft; /* Send a subsolution to the process from which this pro cess got the subproblem*/ MPI_Send(& ctrlMsgSend,1,MPI_INT, my_rank numOfProc/(numOfProcLeft*2), 0,MPI_COMM_WORLD); /*=DATA TRANSFER 3=========================*/ /*= More of th is block can be added on needed basis =*/

PAGE 128

118 MPI_Send(&da taSizeSend,1,MPI_INT, my_rank nu mOfProc/(numOfProcLeft*2), 0,MPI_COMM_WORLD); MPI_Send( localData, /* <-modify address of data */ dataSizeSend, MPI_INT, /*<-modify data type */ my_rank numOfProc/(numOfProcLeft*2), 0,MPI_COMM_WORLD); /*= =*/ /*=======================================*/ } else { MPI_Status status; int i1; int i2; int iResult; int t; /* Receive a subsolution from the process which was invoked by this process */ MPI_Recv(&ctrlMsgRecv,1,MPI_INT, my_rank+nu mOfProc/(numOfProcLeft*2), 0,MPI_COMM_WORLD,&status); /*=DATA TRANSFER 3===========================*/ /*= More of this block can be added on needed basis =*/ MPI_Recv(&dataSizeRecv,1,MPI_INT, my_rank+nu mOfProc/(numOfProcLeft*2), 0,MPI_COMM_WORLD,&status); MPI_Recv( &localData[dataSizeLeft], /*
PAGE 129

119 dataSizeRecv, MPI_INT, /*dataSizeLeft-1){ for(t=i2;t<=dataSizeLeft+dataSizeRecv-1;t++){ mergedInt[iResult+t-i2] = localData[t]; } } else{ for(t=i1;t
PAGE 130

120 for(t=0;t
PAGE 131

121 count[localData[j]] = count[localData[j]]+1; } for(i=1;i=0;j--){ temp[count[localData[j]]]=localData[j]; count[localData[j]] =count[localData[j]]-1; } for(i=0;i
PAGE 132

122 localData[i]= ((int)rand())%maxInt; } /* dataSizeS end = numOfIntToSort; */ dataSizeRecv = numOfIntToSort; ctrlMsgRecv = numOfProc; dataSizeLeft = dataSizeRecv; solve(ctrlMsgRecv); } else{ /* Every process waits a message before calling so lve function */ MPI_Recv(&ctrl MsgRecv,1,MPI_INT,MPI_A NY_SOURCE,MPI_ANY_TAG, MPI_COMM_WORLD,&status); /*=DATA TRANSFER 2=============================*/ /*= More of this bloc k can be added on needed basis =*/ MPI_Recv(&dataSizeRe cv,1,MPI_INT,MPI_ANY_SOURCE, MPI_ANY_TAG,MPI_COMM_WORLD,&status); localData = (int *)mallo c(dataSizeRecv*sizeof(int)); MPI_Recv( localData, /*
PAGE 133

123 printf(" %d",localData[i]); } } MPI_Finalize(); }

PAGE 134

124 LIST OF REFERENCES 1. Massingill BL, Mattson TG, Sanders BA. Patterns for Parallel Application Programs. Proceedings of the Sixth Pa ttern Languages of Programs Workshop (PLoP 1999) http://jerry.cs.uiuc.edu/~plop/plop99/ proceedings/massingill/massingill.pdf Accessed August 2002. 2. Massingill BL, Mattson TG, Sanders BA. A Pattern Language for Parallel Application Programs. Proceedings of the Sixth International Euro-Par Conference (Euro-Par 2000) Springer, Heidelberg Germany 2000; pages 678-681 3. Massingill BL, Mattson TG, Sanders BA. Patterns for Finding Concurrency for Parallel Application Program. Proceedi ngs of the Eighth Pattern Languages of Programs Workshop (PLoP 2000) Washington University Technical Report Number WUCS-00-29. 4. Massingill BL, Mattson TG, Sanders BA More Patterns for Parallel Application Programs. Proceedings of the Eighth Patt ern Languages of Programs Workshop (PLoP 2001) http://jerry.cs.uiuc.edu/~plop/plop2001/accepted_submissions/PLoP2001/bmassin gill0/PLoP2001_bmassingill0_1.pdf Accessed August 2002. 5. Massingill BL, Mattson TG, Sanders BA Parallel Programming with a Pattern Language. International Journal on Software Tools for Technology Transfer 2001;volume 3 issue 2 pages 217-234. 6. Coplien JO, Schmidt DC, editors. Pattern Languages of Program Design. Addison-Wesley, Reading MA 1995. 7. Gamma E, Helm R, Johnson R, Vlissides J. Design Patterns: Elements of Reusable Object-Oriented Software. Addison-Wesley, Reading MA1995. 8. Message Passing Interface Forum. A Message Passing Interface Standard. http://www.mpi-forum.org/docs/docs.html Accessed August 2002. 9. Bailey D, Barszcz E, Barton J, Browning D, Carter R, Dagum L, Fatoohi R, Fineberg S, Frederickson P, Lasinski T, Schreiber R, Simon H, Venkatakrishnan V, Weeratunga S. The NAS Parallel Benchmark RNR Technical Report RNR-94007 1994.

PAGE 135

125 10. Massingill BL, Mattson TG, Sanders BA. A Pattern Language for Parallel Application Program. http://www.cise.ufl.edu/research/Parallel Patterns/PatternLanguage/Background/P DSE99_long.htm Accessed August 2002. 11. Knuth DE. The Art of Computing Programming: volume 2. Addison-Wesley, Reading MA 1981. 12. IEEE Standard for Binary Floating Point Numbers. ANSI/IEEE Standard 7541985. IEEE New York 1985. 13. Cormen TH, Leiserson CE, Rivest RL, Stein C. Introduction to Algorithms. The MIT Press, Cambridge MA1998. 14. Squyer M, Lumsdaine A, George WL, Hagedorn JG, Devaney JE. The Interoperable Message Passing Interfac e (IMPI) Extensions to LAM/MPI. MPI Developer's Conference (M PIDC), Ithaca NY 2000. 15. Saphir W, Wijngaart RV, Woo A, Yarr ow M. New Implementations and Results for the NAS Benchmark 2. 8th SIAM Conference on Parallel Processing for Scientific Computing, Minneapolis MN 1997. 16. Cole M. Algorithmic Skeletons: Structured Management of Parallel Computation. MIT Press, Cambridge MA1989. 17. Coplien JO, Schmidt DC, editors. Pattern Languages of Program Design. Addison-Wesley, Reading MA1995. 18. Reynders JV, Hinker PJ, Cummings JC, Atlas SU, Banerjee S, Humphrey W, Karmesin SR, Keahey K, Srikant M, Tholburn M. POOMA: A Framework for Scientific Simulation on Parallel Architectures. SuperComputing, San Diago CA 1995. 19. Armstrong R. An Illustrative Example of Frameworks for Parallel Computing. Proceedings Of Parallel Object Oriented Methods and Applications (POOMA) http://www.acl.lanl.gov/Pooma96/abstracts/rob-armstrong/pooma96.htm Accessed August 2002 20. Launay P, Pazat JL. A Framework for Parallel Programming in Java. Proceedings of the High-Performance Computing and Networking (HPCN Europe) Springer, Heidelberg Germany 1998; pages 628-637 21. Lea D. Concurrent Programming in Java. Addison-Wesley, Reading MA 1996.

PAGE 136

126 22. Douglas D, Schmidt C, Stal M, Rohnert H, Buschmann F. Pattern-Oriented Software Architecture: Patterns for Concurrent and Networked Objects. Wiley & Sons, New York NY 2000. 23. Blelloch GE, Plaxton CG, Leiserson CE Smith SJ, Zagha M. A Comparison of Sorting Algorithms for the Connection Machine CM-2. ACM Symposium on Parallel Algorithms and Architectures, Hilton Head SC 1991. 24. Dusseau AC, Culler DE, Schauser KE, Martin RP. Fast Parallel Sorting under LogP: Experience with the CM-5. IEEE Transactions on Parallel and Distributed Systems August 1996; pages 791-805 25. Shan H, Singh JP. Parallel Sorting on Cache-Coherent DSM Multiprocessors. Proceedings of the 1999 conference on Supercomputing http://www.supercomp.org/sc99/proceedings/papers/shan.pdf Accessed August 2002. 26. Al AE. A Measure of Transaction Proc essing Power. Datamation 1985; volume 32 issue 7 pages 112-118 27. Saini S, Bailey DH. NAS Parallel Benchm ark (Version 1.0) Results 11-96. Report NAS-96-018 December 1996. 28. Saini S, Bailey DH. NAS. Parallel Be nchmark Results 12-95. Report NAS-95-021 December 1995.

PAGE 137

127 BIOGRAPHICAL SKETCH Eunkee Kim was born 1970 in Dea-Gu, Republic of Korea. He received a B.S in physics at Yeung-Nam Universtiy in Keung-Sa n, Republic of Korea, in 1997. He joined the graduate program of the Computer and Information Science and Engineering Department in 1999 to pursue his master’s degree.


Permanent Link: http://ufdc.ufl.edu/UFE0000552/00001

Material Information

Title: Implementation patterns for parallel program and a case study
Physical Description: Mixed Material
Creator: Kim, Eunkee ( Author, Primary )
Publication Date: 2002
Copyright Date: 2002

Record Information

Source Institution: University of Florida
Holding Location: University of Florida
Rights Management: All rights reserved by the source institution and holding location.
System ID: UFE0000552:00001

Permanent Link: http://ufdc.ufl.edu/UFE0000552/00001

Material Information

Title: Implementation patterns for parallel program and a case study
Physical Description: Mixed Material
Creator: Kim, Eunkee ( Author, Primary )
Publication Date: 2002
Copyright Date: 2002

Record Information

Source Institution: University of Florida
Holding Location: University of Florida
Rights Management: All rights reserved by the source institution and holding location.
System ID: UFE0000552:00001


This item has the following downloads:


Full Text














IMPLEMENTATION PATTERNS FOR PARALLEL PROGRAM AND A CASE
STUDY














By

EUNKEE KIM


A THESIS PRESENTED TO THE GRADUATE SCHOOL
OF THE UNIVERSITY OF FLORIDA IN PARTIAL FULFILLMENT
OF THE REQUIREMENTS FOR THE DEGREE OF
MASTER OF SCIENCE

UNIVERSITY OF FLORIDA


2002































Copyright 2002

by

Eunkee Kim






























Dedicated to Grace Kim, Jineun Song and parents


















ACKNOWLEDGMENTS

I would like to express my sincere gratitude to my adviser, Dr. Beverly A. Sanders, for

providing me with an opportunity to work in this exciting area and for providing

feedback, guidance, support, and encouragement during the course of this research and

my graduate academic career.

I wish to thank Dr. Joseph N. Wilson and Dr. Stephen M. Thebaut for serving on my

supervisory committee.

Finally I thank Dr. Berna L. Massingill for allowing me to use machines in Trinity

University and for technical advice.


















TABLE OF CONTENTS
page


A C K N O W L E D G M E N T S ................................................................................................. iv


LIST O F TA B LES ........................................................ .......... ....... .............. ix


L IS T O F F IG U R E S ............................................ ......................................................... x


A B S T R A C T ................................................... ........ .......... ...... x i


CHAPTER


1 IN TROD U CTION ................ ..................... ................... .. ............. ............. 1

1.1 P parallel C om putting ........................................... ........................ ................... 1
1.2 P parallel D design P patterns ...................... .. .. ......... .. ...................... .................... 1
1.3 Implem entation Patterns for D esign Patterns ..................................... ................... 2
1.4 Implementation of Kernel IS of NAS Parallel Benchmark Set Using Parallel
D design P patterns. .................................................... 2
1.5 O organization of the Thesis ......... ...................................................... .............. 3

2 OVERVIEW OF PARALLEL PATTERN LANGUAGE ............... .............. 4

2.1 Finding Concurrency Design Space......................................... ........................ 4
2.1.1 G getting Started ..................... .................. ............ ............ .............. 4
2.1.2 Decomposition Strategy....... .......................................... 4
2.1.3 Task D ecom position ........................................................ .............. 5
2.1.4 Data Decomposition.................................. .............. ... .............. 5
2.1.5 D ependency A nalysis...................... ...................................... .............. 6
2.1.6 Group Tasks ......... ........ .............................. ...... ..... ......... 6
2.1.7 O order Tasks ........................................ ........... .. .. ............. .............. 7
2 .1.8 D ata Sharing ..................................................... 7
2.1.9 Design Evaluation .................. .............. ..... .............. 7
2.2 Algorithm Structure Design Space ............... ......................... ... .............. 8
2.2.1 Choose Structure ........... .... ................................... ... 8
2.2.2 Asynchronous Composition........ ............................................... 9
2.2.3 Divide and Conquer .......... .......... ........... ...... .............. 9













2.2.4 E m barrassingly P arallel ................................................................................ 10
2.2.5 G eom etric D ecom position ........................................ .................. ..... 10
2.2.6 P pipeline Processing ........... ............................................................... 11
2.2.7 Protected Dependencies ..................................................... 11
2.2.8 Recursive Data ............ ......... ..................... 11
2.2.9 Separable Dependency ........... .. ................................. 12
2.3 Supporting Structures D esign Space................................................. .... .. .............. 12
2.3.1 Program -Structuring G roup ........................................ ................. ..... 12
2.3.1.1 Single program and multiple data.............. .... .................. 12
2.3.1.2 Fork join ................... ......... ......... 13
2.3.1.3 M aster w worker ......... .............................. ................................... 13
2.3.1.4 Spawn ..................................................................... ........ ........ ......... 13
2.3.2 Shared D ata Structures G roup................................................. .... .. .............. 13
2 .3 .2 .1 Shared queue ............... .................... ........... ............. .... ..... ..... 13
2.3.2.2 Shared counter ..................... ..... ................... .... .............. 14
2.3.2.3 D distributed array .............. .................................... ... ........... 14
2.4 Chapter Sum m ery ................ .... ..... ............. ......... ............ ... 14

3 PATTERNS FOR IMPLEMENTATION.................. ...... ........................ 15

3.1 Using M message Passing Interface (M PI)............................. .... ............ ... ..... 15
3.1.1 Intent ........... .. .............................. ................... ......... 15
3 .1.2 A p p licab ility .......................................................................... 15
3.1.3 Im plem entation ............... .......... ...... .......... .... .. ...... .......... ... 15
3.1.3.1 Sim ple m message passing ................................................... .... .. .............. 16
3.1.3.2 Group and com m unicator creation........................................ .................... 17
3.1.3.3 Data distribution and reduction............... .................................... 22
3.2 Simplest Form of Embarrassingly Parallel ............ .. ........... .. ................ 23
3 .2 .1 In te n t ......... ...................................................................................... 2 3
3.2.2 Applicability .......................................... 23
3.2.3 Im plem entation .............. ......................................... ............ ........ ..... 24
3.2.4 Im plem entation Exam ple ......... .............. ...................................... .......... 25
3.2.5 E xam ple U sage..................... ..... .... ....... ...... ................ ...... ...................... 27
3.3 Implementation of Embarrassingly Parallel.................. ...... ................ 28
3 .3 .1 In te n t ......... ...................................................................................... 2 8
3.3.2 Applicability.................. .. ............................... 28
3.3.3 Implementation .... .............................. ..... 28
3.3.4 Implementation Example ..................... ............. .................. 29
3.3.5 U sage E xam ple ............... ........................ ......................... ... ...... ..... 38
3.4 Implementation of Pipeline Processing ................ ................ ............. 39
3 .4 .1 In te n t ......... ...................................................................................... 3 9
3.4.2 Applicability.................. ... ............................... 39
3.4.3 Implementation .... .............. ............. ......... 39
3.4.4 Im plem entation Exam ple ....... .. .............. .......................... .. .... ........ 41
3.4.5 Usage Example ............... ........... .... ..... .......................... ....... 47
3.5 Implementation of Asynchronous-Composition............................... ............... 48
3 .5 .1 In ten t ......... .................................................................................... 4 8


vi













3.5.2 Applicability.................. .. ............................... 49
3.5.3 Implementation ........... ...................................... ..................... 49
3.5.4 Im plem entation Exam ple ......................................................... ........... ... 50
3.6 Implementation of Divide and Conquer .................. ..................................... 54
3.6.1 Intent ..................................................................... ........ 54
3.6.2 M otivation ............................ ............... ..... 54
3.6.3 Applicability.................. .. ............................... 55
3.6.4 Im plem entation .............. ............................................... ... ...... 55
3.6.5 Im plem entation Exam ple ......................................................... ........... ... 57
3.6.6 U sage E xam ple ............... ................................................. ... ...... ..... 6 1

4 KERNEL IS OF NAS BENCHMARK .............................................. .............. 63

4.1 B rief Statem ent of P problem .............................................................. .............. 63
4.2 Key Generation and Memory Mapping .............................................. ............ 63
4.3 Procedure and Tim ing ................... ......................................... ........... .............. 64

5 PARALLEL PATTERNS USED TO IMPLEMENT KERNEL IS .......................... 66

5.1 F finding C oncurrency ............................................. ...... ............. 66
5.1.1 G getting Started ....................................... .............................. 66
5.1.2 Decomposition Strategy ................ ......... .............. 67
5.1.3 Task D ecom position .................................. ............... .............. 68
5.1.4 Dependency Analysis ................ ......... .............. 68
5.1.5 D ata Sharing Pattern .................................. ............... .............. 68
5.1.6 D esign Evaluation ................................................... ............. ..... 69
5.2 A lgorithm Structure D esign Space .................................. .................................... 70
5.2.1 C hoose Structure ........................................................ ........... ..... 70
5.2.2 Separable Dependencies .................. ......... ............. .... ...... ........ 71
5.2.3 E m barrassingly P arallel ................................................................................ 7 1
5.3 U sing Im plem entation Exam ple ................................................................ 71
5.4 Algorithm for Parallel Implementation of Kernel IS....................................... 72

6 PERFORMANCE RESULTS AND DISCUSSIONS......................... .............. 74

6.1 P perform ance E expectation ......... .......................................................................... 74
6.2 P perform ance R results ................................................... ........................... ..... 74
6.3 D discussions ................................................ 76

7 RELATED WORK AND CONCLUSIONS AND FUTURE WORK....................... 78

7 .1 R e la te d w o rk .............. .... ............. ........................................................................ 7 8
7.1.1 Aids for Parallel Programming ........................... .... .............. 78
7.1.2 Parallel Sorting.......... ................. ...... ............ ........ ... 80
7.2 Conclusions ..................... ......... 81
7.3 Future W ork .................................................................................................... 82

APPENDIX













A KERNEL IS OF THE NAS PARALLEL BENCHMAKRK................................. 83


B PSEUDORANDOM NUMBER GENERATOR................................................ 90


C SOURCE CODE OF THE KERNEL IS IMPLEMENTATION............................... 94


D SOURCE CODE OF PIPELINE EXAMPLE................................. ................... 107


E SOURCE CODE OF DIVIDE AND CONQUER ............................. 115


L IST O F R E FE R E N C E S ................................................................................................ 124


BIO GRAPHICAL SKETCH ............................................... ................... .............. 127


















LIST OF TABLES
Table page


4-1 Parameter Values to be used for Benchmark.....................................................65

6-1 Performance Results for Class A and B ........................................... ............... 75

A-1 Values to be used for Partial Verification ............................................ .............. 88

A-2 Param eter Values to be used for Benchmark............................ ............................. 89


















LIST OF FIGURES
Figure page


3-1 The A lgorithm Structure D decision Tree.................................. ..................................... 9

3-2 U sage of M message Passing ................................................................ ....... .................. 40

3-3 Irregular M message H handling ................................................ ................................ 50

3-4 Message Passing for Invocation of the Solve and Merge Functions...........................56

4-1 C counting S ort ...........................................................................67

6-1 Execution Time Comparison for Class A............................................. ............... 76

6-2 Execution Tim e Comparison for Class B .............................................. .................. 76


















Abstract of Thesis Presented to the Graduate School
of the University of Florida in Partial Fulfillment of the
Requirements for the Degree of Master of Science

IMPLEMENTATION PATTERNS FOR PARALLEL PROGRAM AND A CASE
STUDY

By

Eunkee Kim

December 2002

Chairman: Beverly A. Sanders
Major Department: Computer and Information Science and Engineering

Design patterns for parallel programming guides the programmer through the entire

process of developing a parallel program. In this thesis, implementation patterns for

parallel program are presented. These patterns help programmers to implement a parallel

program after designing it using the parallel design patterns.

Parallel Integer sorting, one of the kernels of Parallel Benchmark set of the Numerical

Aerodynamic Simulation Program (NAS), has been designed and implemented using the

parallel design pattern as a case study. How the parallel design pattern was used in

implementing Kernel IS and its performance are discussed



















CHAPTER 1
INTRODUCTION

1.1 Parallel Computing

Parallel computing is what a computer does when it carries out more than one

computation at a time using many processors. An example of parallel computing

(processing) in our daily life is an automobile assembly line: at every station somebody is

doing part of the work to complete the product. The purpose of parallel computing is to

overcome the limit of the performance of a single processor. We can increase the

performance of a sequential program by parallelizing exploitable concurrency in a

sequential program and using many processors at once.

Parallel programming has been considered more difficult than sequential programming.

To make it easier for programmers to write a parallel program, a parallel design pattern

has been developed by B.L. Massingill, T.G. Mattson, and B.A Sanders.1-4

1.2 Parallel Design Patterns

The Parallel pattern language is a collection of design patterns for parallel

programming.5-6 Design patterns are a high level description of a solution to a problem.7

Parallel pattern language is written down in a systematic way so that a final design for a

parallel program can result by going through a sequence of appropriate patterns from a

pattern catalog. The structure of the patterns is designed to parallelize even complex

problems. The top-level patterns help to find the concurrency in a problem and

decompose it into a collection of tasks. The second-level patterns help to find an












appropriate algorithm structure to exploit concurrency that has been identified. The

parallel design patterns are described in more detail in Chapter 2.

1.3 Implementation Patterns for Design Patterns

Several implementation patterns for design patterns in the algorithm structure design

space were developed as a part of the parallel pattern language and presented in this

thesis. These implementation patterns are solutions of problems mapping high-level

parallel algorithms into programs using a Message Passing Interface (MPI) and

programming language C.8 The patterns of the algorithm structure design space capture

recurring solutions to the problem of turning problems into parallel algorithms. These

implementation patterns can be reused and provide guidance for programmers who might

need to create their own implementations after using the parallel design pattern. The

implementation patterns of design patterns in the algorithm structure design space are in

Chapter 3.

1.4 Implementation of Kernel IS of NAS Parallel Benchmark Set Using Parallel
Design Patterns

The Numerical Aerodynamic Simulation (NAS) Program, which is based at NASA

Ames Research Center, developed parallel benchmark set for the performance evaluation

of highly parallel supercomputers. The NAS Parallel Benchmark set is a "Paper and

Pencil" benchmark.9 All details of this benchmark set are specified only algorithmically.

The Kernel IS of NAS Parallel Benchmark set is a parallel sorting over large numbers of

integers. A solution to the Kernel IS is designed and implemented using parallel pattern

language as a case study. The used patterns are following.

Getting Start, Decomposition Strategy, Task Decomposition, Dependency Analysis,

Data Sharing, and Design Evaluation patterns of Find Concurrency Design Space are












used. Choose Structure pattern, Separable Dependency, and Embarrassingly Parallel

patterns of the algorithm structure design space are also used. The details of how these

patterns are used and the final design of the parallel program for Kernel IS are in Chapter

5.

1.5 Organization of the Thesis

The research that has been conducted as part of this thesis and its organization are as

follows:

* Analysis of the parallel design patterns (Chapter 2)

* Implementation patterns for patterns in Algorithm structure design space (Chapter 3)

* A description of Kernel IS of the NAS Parallel Benchmark set (Chapter 4)

* Design and implementation of Kernel IS through the parallel design patterns
(Chapter 5)

* Performance results and discussions (Chapter 6).

* Conclusions and future work (Chapter 7).



















CHAPTER 2
OVERVIEW OF PARALLEL PATTERN LANGUAGE

Parallel pattern language is a set of design patterns that guide the programmer through

the entire process of developing a parallel program.10 The patterns of parallel pattern

language, as developed by Masingill et al., are organized into four design spaces: Finding

Concurrency Design Space, Algorithm Structure Design Space, Supporting Structure

Design Space, and Implementation Design Space.

2.1 Finding Concurrency Design Space

The finding concurrency design space includes high-level patterns that help to find

the concurrency in a problem and decompose it into a collection of tasks.

2.1.1 Getting Started

The getting started pattern helps to start designing a parallel program. Before using

these patterns, a user of this pattern needs to be sure that the problem is large enough or

needs to speed up and understand the problem. The user of this pattern needs to do the

following tasks:

* Decide which parts of the problem require most intensive computation.

* Understand the tasks that need to be carried out and data structure that are to be
manipulated.

2.1.2 Decomposition Strategy

This pattern helps to decompose the problem into relatively independent entities that

can execute concurrently. To expose the concurrency of the problem, the problem can be

decomposed along two different dimensions:












* Task Decomposition: Break the stream of instructions into multiple chunks called
tasks that can execute simultaneously.

* Data Decomposition: Decompose the problem's data into chunks that can be operated

relatively independently.

2.1.3 Task Decomposition

The Task Decomposition pattern addresses the issues raised during a primarily task-

based decomposition. To do task decomposition, the user should try to look at the

problem as a collection of distinct tasks. And these tasks can be found in the following

places:

* Function calls may correspond to tasks.

* Each iteration of a loop, if the iterations are independent, can be the tasks.

* Updates on individual data chunks decomposed from a large data structure.

The number of tasks generated should be flexible and large enough. The tasks should

have enough computation.

2.1.4 Data Decomposition

This pattern looks at the issues involved in decomposing data into units that can be

updated concurrently. The first point to be considered is whether the data structure can be

broken down into chunks that can be operated on concurrently. An array-based

computation and recursive data structures are examples of this approach. The points to be

considered in decomposing data are as follows:

* Flexibility in the size and number of data chunks to support the widest range of
parallel systems

* The size of data chunks large enough to offset the overhead of managing dependency

* Simplicity in data decomposition












2.1.5 Dependency Analysis

This pattern is applicable when the problem is decomposed into tasks that can be

executed concurrently. The goal of a dependency analysis is to understand the

dependency among the tasks in detail. Data-sharing dependencies and ordering

constraints are the two kinds of dependencies that should be analyzed. The dependencies

must require little time to manage relative to computation time and easy to detect and fix

errors. One effective way of analyzing dependency is following approach.

* Identify how the tasks should be grouped together.

* Identify any ordering constraints between groups of tasks.

* Analyze how tasks share data, both within and among groups.

These steps lead to the Group Tasks, the Order Tasks, and the Data Sharing patterns.

2.1.6 Group Tasks

This pattern constitutes the first step in analyzing dependencies among the tasks of

problem decomposition. The goal of this pattern is to group tasks based on the following

constraints. Three major categories of constraints among tasks are as follows:

* A temporal dependency: a constraint placed on the order in which a collection of
tasks executes.

* An ordering constraint that a collection of tasks must run at the same time.

* Tasks in a group are truly independent.

The three approaches to group tasks are as follows:

* Look at how the problem is decomposed. The tasks that correspond to a high-level
operation naturally group together. If the tasks share constraints, keep them as a
distinct group.

* If any other task groups share the same constraints, merge the groups together.

* Look at constraints between groups of tasks.












2.1.7 Order Tasks

This pattern constitutes the second step in analyzing dependencies among the tasks of

problem decomposition. The goal of this pattern is to identify ordering constraints among

task groups. Two goals are to be met in defining this ordering:

* It must be restrictive enough to satisfy all the constraints to be sure the resulting
design is correct.

* It should not be more restrictive than it need be.

To identify ordering constraints, consider the following ways tasks can depend on

each other:

* First look at the data required by a group of tasks before they can execute. Once this
data have been identified, find the task group that created it, and you will have an
order constraint.

* Consider whether external services can impose ordering constraints.

* It is equally important to note when an order constraint does not exit.

2.1.8 Data Sharing

This pattern constitutes the third step in analyzing dependencies among the tasks of

problem decomposition. The goal of this pattern is to identify what data are shared

among groups of tasks and how to manage access to shared data in a way that is both

correct and efficient. The following approach can be used to determine what data is

shared and how to manage the data:

* Identify data that are shared between tasks.

* Understand how the data will be used.

2.1.9 Design Evaluation

The goals of this pattern are to evaluate the design so far and to prepare for the next

phase of the design space. This pattern therefore describes how to evaluate the design












from three perspectives: suitability for the target platform, design quality, and preparation

for the next phase of the design.

For the suitability for the target platform, the following issues included:

* How many processing elements are available?

* How are data structures shared among processing elements?

* What does the target architecture imply about the number of units of execution and
how structures are shared among them?

* On the target platform, will the time spent doing useful work in a task be significantly
greater than the time taken to deal with dependencies?

For the design quality, simplicity, flexibility, and efficiency should be considered.

To prepare the next phase, the key issues are as follows:

* How regular are the tasks and their data dependencies?

* Are interactions between tasks (or task groups) synchronous or asynchronous?

* Are the tasks grouped in the best way?

2.2 Algorithm Structure Design Space

The algorithm structure design space contains patterns that help to find an appropriate

algorithm structure to exploit the concurrency that has been identified.

2.2.1 Choose Structure

This pattern guides the algorithm designer to the most appropriate Algorithm-

Structure patterns for the problem. Consideration of Target Platform, Major Organizing

Principle and Algorithm-Structure Decision Tree are the main topics of this pattern.

The two primary issues of Consideration of Target Platform are how many units of

execution (threads or processes) the target system will effectively support and the way

information is shared between units of execution. The three major organizing principles

are "organization by ordering," "organization by tasks," and "organization by data." The















Algorithm-Structure Decision Tree is in Figure 3.1. We can select an algorithm-structure


using this decision tree.



Start


OrgarnzeByOrdenng OrgarnzeByData
OrganzeByTasks
] Regular u ] Iegular / \ Linear I \ Recursive


PipelineProcessing Asynchr~onus | Linear Recursive C, Geometric Recursve
Composition -- -- ----- -- Decomposition Data

Partitorng DivideAnd
S- Co nuer

| Independent Depender d


Embarrassingly Separable Inseparable
Parallel Dependencies Dependencies


SeparableDependenaes ProtectedDependencies

Key Decastl anchpo

Terminal pattern




Figure 3-1 The Algorithm Structure Decision Tree

2.2.2 Asynchronous Composition

This pattern describes what may be the most loosely structured type of concurrent


program in which semi-independent tasks interact through asynchronous events. Two


examples are the Discrete-Event Simulation and Event-Driven program. The key issues


in this pattern are how to define the tasks/entities, how to represent their interaction, and


how to schedule the tasks.


2.2.3 Divide and Conquer

This pattern can be used for the parallel application program based on the well-known


divide-and-conquer strategy. This pattern is particularly effective when the amount of












work required to solve the base case is large compared to the amount of work required for

the recursive splits and merges. The key elements of this pattern are as follows:

* Definitions of the functions such as "solve," "split," "merge," "baseCase,"
"baseSolve"

* A way of scheduling the tasks that exploits the available concurrency

This pattern also includes Correctness issues and Efficiency issues.

2.2.4 Embarrassingly Parallel

This pattern can be used to describe concurrent execution by a collection of

independent tasks and to show how to organize a collection of independent tasks so they

execute efficiently.

The key element of this pattern is a mechanism to define a set of tasks and schedule

their execution and also a mechanism to detect completion of the tasks and terminate the

computation. This pattern also includes the correctness and efficiency issues.

2.2.5 Geometric Decomposition

This pattern can be used when the concurrency is based on parallel updates of chunks

of a decomposed data structure, and the update of each chunk requires data from other

chunks. Implementations of this pattern include the following key elements:

* A way of partitioning the global data structure into substructures or "chucks"

* A way of ensuring that each task has access to all the data needed to perform the
update operation for its chunk, including data in chunks corresponding to other tasks

* A definition of the update operation, whether by points or by chunks

* A way of assigning the chunks among units of execution (distribution), that is a way
of scheduling the corresponding tasks












2.2.6 Pipeline Processing

This pattern is for algorithms in which data flow through a sequence of tasks or stages.

This pattern is applicable when the problem consists of performing a sequence of

calculations, each of which can be broken down into distinct stages on a sequence of

inputs. So for each input, the calculations must be done in order, but it is possible to

overlap computation of different stages for different inputs. The key elements of this

pattern are as follows:

* A way of defining the elements of the pipeline where each element corresponds to
one of the functions that makes up the computation

* A way of representing the dataflow among pipeline elements, i.e., how the functions
are composed

* A way of scheduling the tasks

2.2.7 Protected Dependencies

This pattern can be used for task-based decompositions in which the dependencies

between tasks cannot be separated from the execution of the tasks, and the dependencies

must be dealt with in the body of the task. The issues of this pattern are as follows:

* A mechanism to define a set of tasks and schedule their execution onto a set of units
of executions

* Safe access to shared data

* Shared memory available when it is needed

2.2.8 Recursive Data

This pattern can be used for parallel applications in which an apparently sequential

operation on a recursive data is reworked to make it possible to operate on all elements of

the data structure concurrently. The issues of this pattern are as follows:

* A definition of the recursive data structure plus what data is needed for each element
of the structure












* A definition of the concurrent operations to be performed

* A definition of how these concurrent operations are composed to solve the entire
problem

* A way of managing shared data

* A way of scheduling the tasks onto units of executions

* A way of testing its termination condition, if the top-level structure involves a loop.

2.2.9 Separable Dependency

This pattern is used for task-based decompositions in which the dependencies between

tasks can be eliminated as follows: Necessary global data are replicated and (partial)

results are stored in local data structures. Global results are then obtained by reducing

results from the individual tasks. The key elements of this pattern are as follows:

* Defining the tasks and scheduling their execution

* Defining and updating a local data structure

* Combining (reducing) local objects into a single object

2.3 Supporting Structures Design Space

The patterns at this level represent an intermediate stage between the problem-oriented

patterns of the algorithm structure design space and the machine-oriented patterns of the

Implementation-Mechanism Design Space.

Patterns in this space fall into two main groups: Program-Structuring Group and

Shared Data Structures Group.

2.3.1 Program-Structuring Group

Patterns in this group deal with constructs of structuring the source code.

2.3.1.1 Single program and multiple data

The computation consists of N units of execution in parallel. All N UEs (Generic term

for concurrently executable entity, usually either process or thread) execute the same












program code, but each operates on its own set of data. A key feature of the program

code is a parameter that differentiates among the copies.

2.3.1.2 Fork join

A main process or thread forks off some number of other processes or threads that

then continue in parallel to accomplish some portion of the overall work before rejoining

the main process or thread.

2.3.1.3 Master worker

A master process or thread sets up a pool of worker processes or threads and a task

queue. The workers execute concurrently with each worker repeatedly removing a task

from the task queue and processing it until all tasks have been processed or some other

termination condition has been reached. In some implementations, no explicit master is

present.

2.3.1.4 Spawn

A new process or thread is created, which then executes independently of its creator.

This pattern bears somewhat the same relation to the others as GOTO bears to the

constructs of structured programming.

2.3.2 Shared Data Structures Group

Patterns in this group describe commonly used data structures.

2.3.2.1 Shared queue

This pattern represents a "thread-safe" implementation of the familiar queue abstract

data type (ADT), that is, an implementation of the queue ADT that maintains the correct

semantics even when used by concurrently executing units of execution.












2.3.2.2 Shared counter

This pattern, as with the Shared Queue pattern, represents a "thread-safe"

implementation of a familiar abstract data type, in this case a counter with an integer

value and increment and decrement operations.

2.3.2.3 Distributed array

This pattern represents a class of data structures often found in parallel scientific

computing, namely, arrays of one or more dimensions that have been decomposed into

subarrays and distributed among processes or threads.

2.4 Chapter Summery

This chapter describes an overview of the parallel pattern languages for an application

program which guides programmers from the design process of the program to the

implementation point. The Next chapter illustrates how these design patterns have been

used to design and implement the Kernel IS of the NAS Parallel Benchmark.



















CHAPTER 3
PATTERNS FOR IMPLEMENTATION

In this chapter, we will introduce patterns for the implementation of patterns in the

algorithm structure design space. Each pattern contains Implementation Example for

each pattern in the algorithm structure design space. The Implementation Examples are

implemented in an MPI and C environment. The MPI is standards for "Message Passing

Interface" in Distributed Memory Environments. The Implementation Examples may

need modification occasionally, but will be reusable and helpful for an implementation of

the parallel program designed using parallel pattern language.

3.1 Using Message Passing Interface (MPI)

3.1.1 Intent

This pattern is an introduction of Message Passing Interface (MPI).

3.1.2 Applicability

This pattern is applicable when the user of parallel pattern language has finished the

design of his parallel program and considers implementing it using MPI.

3.1.3 Implementation

The MPI is a library of functions (in C) or subroutines (in Fortran) that the user can

insert into a source code to perform data communication between processes. The primary

goals of MPI are to provide source code portability and allow efficient implementations

across a range of architectures.












3.1.3.1 Simple message passing

Message passing programs consist of multiple instances of a serial program that

communicates by library calls. The elementary communication operation in MPI is

"point-to-point" communication, that is, direct communication between two processes,

one of which sends and the other receives. This message passing is the basic mechanism

of exchanging data among processes of a parallel program using MPI. This message

passing is also a good solution about how to order the executions of tasks belongs to a

group and how to order the executions of groups of tasks.

To communicate among processes, communicators are needed. Default communicator

of MPI is MPI_COMMWORLD which includes all processes. The common

implementation of MPI executes same programs concurrently at each processing element

(CPU or workstation). The following program is a very simple example of using

MPI_Send (send) and MPIRecv (receive) functions.

/* simple send and receive */

/* Each process has the same copy of the following program and executes it *
#include
#include

#define BUF SIZE 100
int tag =1;
void main (int argc, char **argv) {

int myrank; /* rank (identifier) of each process*/
MPIStatus status; /* status object */
double a[BUF_SIZE]; /*Send(Receive) buffer *

MPIInit(&argc, &argv); /* Initialize MPI */
MPI_Commrank(MPI_COMM WORLD, &myrank); /* Get rank ofprocess*

if( myrank == 0 ) {
/* code for process 0 *
/* send Message *
MPI_Send(












a, /*initial address of send buffer */
BUF_SIZE, / muin,/r of elements in send buffer *
MPI_DOUBLE, / ,tla,/lj, of each send buffer element *
1, /*rank of destination *
tag, /* message tag */
MPI COMM WORLD *communicator *

}
else if( myrank == 1 )
/* code for process 1 *
/ I'L tL'V1' message */
MPIRecv( a, /*initial address of receive buffer *
BUF_SIZE, / nuinl'r of elements in send buffer *
MPI_DOUBLE,/ tJIa, Ij' of each receive buffer element *
0, /*rank of source */
tag, / inu'\\ %S tag */
MPI COMM WORLD, *communicator *
&status /*status object *

}
/* more else if statement can be addedfor more processes *
/* switch statements can be used instead of if else statement s*/

MPIFinalize(); /* Terminate MPI */
}

3.1.3.2 Group and communicator creation

A communicator in MPI specifies the communication context for communication

operations and the set of processes that share this communication context. To make

communication among processes in a program using MPI, we must have communicators.

There are two kinds of communicators in MPI: intra-communicator and inter-

communicator. Intra-communicators are for message passing among processes belong to

the same group. Inter-communicators are for message passing between two groups of

processes. As mentioned previously, the execution order of tasks belonging to one group

can be implemented using intra-communicator and message-passing. The execution order

of groups of tasks can be implemented using inter-communicator and message-passing.












To create a communicator, a group of processes is needed. MPI does not provide

mechanism to build a group from scratch, but only from other, previously defined groups.

The base group, upon which all other groups can be created, is the group associated

with the initial communicator MPICOMMWORLD. A group of processes can be

created by the five steps as follows:

1. Decide the processing units that will be included in the groups

2. Decide the base groups to use in creating new group.

3. Create group.

4. Create communicator.

5. Do the work.

6. Free communicator and group.

The first step is decide how many and which processing units will be included in this

group. The points to be considered are the number of available processing units, the

number of tasks in one group, the number of groups that can be executed concurrently,

etc., according to the design of the program.

If only one group of tasks can executed because of dependency among groups than the

group can use all available processing units and may include them in that group. If there

are several groups that can be executed concurrently, then the available processing units

may be divided according to the number of groups that can be executed concurrently and

the number of tasks in each group.

Since MPI does not provide a mechanism to build a group from scratch, but only from

other, previously defined groups, the group constructors are used to subset and superset

existing groups. The base group, upon which all other groups can be defined, is the group












associated with the initial communicator MPICOMMWORLD, and processes in this

group are all the processes available when MPI is initialized.

There are seven useful group constructors in MPI. Using these constructors, groups

can be constructed as designed. The first three constructors are similar to the union and

intersection of set operation in mathematics. The seven group constructors are as follows:

The MPI_GROUPUNION creates a group which contains all the elements of the two

groups used. The MPI_GROUPINTERSECTION creates a group which contains all the

elements that belong in both groups at the same time. The MPIGROUP_DIFFERENCE

creates a group which has all the elements that do not belong in both groups at the same

time. The MPIGROUPINCL routine creates a new group that consists of the specified

processes in the array from the old group. The MPIGROUPEXCL routine creates a

new group that consists of the processes not specified in the array from the old group.

The MPIGROUPRANGEINCL routine includes all processes between one specified

process to another process, and the specified processes themselves. The

MPI_GROUPRANGE_EXCL routine excludes all processes not between the specified

processes, and also excludes the specified processes themselves.

For the creation of communicators, there are two types of communicators in MPI:

intra-communicator and inter-communicator. The important intra-communicator

constructors in MPI are as follows: The MPI_COMMDUP function will duplicate the

existing communicator. The MPI_COMMCREATE function creates a new

communicator for a group.

For the communication between two groups, the inter-communicator for the identified

two groups can be created by using the communicator of each group and peer-












communicator. The routine for this purpose is MPIINTERCOMIMCREATE. The peer-

communicator must have at least one selected member process from each group. Using

duplicated MPI_COMMWORLD as a dedicated peer communicator is recommended.

The Implementation Example Code is as follows:


/* An Example of creating intra-communicator */

#include
main(int argc, char **argv)


int myRank, count, county;
int *sendBuffer, *receiveBuffer, *sendBuffer2, *receiveBuffer2;
MPI Group MPI_GROUP WORLD, grprem;
MPI Comm commSlave;
static int ranks[] = { 0 };
/* */
MPI_Init(&argc, &argv);
MPI_Comm_group(MPI_COMMWORLD, &MPI_GROUPWORLD);
MPI_Commrank(MPI_COMM WORLD, &myRank);

/* Build group for slave processes *
MPI Group_excl(
MPIGROUP_WORLD, /* group *
1, /* number of elements in array ranks *
ranks, /* array of integer ranks in group not to appear
in new group */
&grprem); /* new group derived from above *

/* Build communicator for slave processes *
MPI_Commcreate(
MPI COMM WORLD, /* communicator *
grprem, /* Group, which is a subset of the group of
above communicator *
&commSlave); /* new communicator */


if(myRank !=0)
{
/* compute on processes other than root process *
/* */


MPIReduce(sendBuffer, receiveBuffer, count, MPIINT, MPI_SUM, 1,
commSlave);












/* */
}

/* Rank zero process falls ilith r gh immediately to this reduce,
others do later **
MPI_Reduce(sendBuffer2, receiveBuffer2, county, MPIINT, MPI_SUM, 0,
MPICOMM WORLD);
MPI_Comm_free(&commSlave);
MPI_Group_free(&MPIroreeGROUPWORLD);
MPIGroupfree(&grprem);
MPIFinalize();


/* An example of creating inter-communicator */

#include

main(int argc, char **argv)
{
MPI_Comm myComm; /* intra-communicator of local sub-group */
MPI_Comm myInterComm; /* inter-communicator between two group *
int membershipKey;
int rank;

MPI_Init(&argc, &argv);
MPI_Commrank(MPI_COMM WORLD, &rank);

/* User generate membership Key for each group */
membershipKey = rank % 3;


/* Build intra-communicatorfo
MPI_Commsplit(
MPI COMM WORLD,
membershipKey,


rank,


&myComm


r local sub-group */

/* communicator */
/* Each subgroup contains all processes
of the same membershipKey */
/* Within each subgroup, the processes are
ranked in the order defined by
this rank value */
/* new communicator *


/* Build inter-communicators. Tags are hard-coded *
if (membershipKey == 0)
{












MPI_Intercommcreate(
myComm, /* local intra-communicator */
0, /* rank of local group leader */
MPICOMM WORLD,/* "peer" intra-communicator *
1, /* rank of remote group leader
in the "peer" communicator */
1, /* tag */
&myInterComm /* new inter-communicator */

}
else
{
MPI_Intercommcreate(myComm, 0, MPICOMMWORLD,0,1,
&myInterComm);
}

/* do work in parallel */

if(membershipKey == 0)
MPI_Commfree(&myInterComm);
MPIFinalize();
}

3.1.3.3 Data distribution and reduction

There are several primitive functions for distribution of data in MPI.

The three basic functions are as follows: The MPIBROADCAST function replicates

data in one root process to all other processes so that every process has the same data.

This function can be used for a program which needs replication of data in each process

to improve performance.

The MPI_SCATTER function of MPI can scatter data from one processing unit to

every other processing unit with the same amount (or variant amounts) of data, and each

processing unit will have a different portion of the original data. The Gather function of

MPI is the inverse of Scatter.

For the reduction of data distributed among processes, the MPI provides two primitive

MPI REDUCE and MPI ALL REDUCE functions. These reduce functions are useful












when it is needed to combine locally computed subsolutions into one global solution. The

MPIREDUCE function combines the elements provided in the input buffer of each

process in the group using the specified operation as a parameter, and returns the

combined value in the output buffer of the process with rank of root. The

MPIALLREDUCE function combines the data in the same way as the MPIREDUCE,

but every process has the combined value in the output buffer. This function is beneficial

when the combined solution can be used as an useful information for the next phase of

computation.

The Implementation Example Code is not provided in this section, but the

Implementation Example Code of 3.2.4 can be used as an Implementation Example Code.

The source code of the Kernel IS implementation is also a good Implementation Example

Code.

3.2 Simplest Form of Embarrassingly Parallel

3.2.1 Intent

This is a solution to the problem of how to implement the simplest form of

Embarrassingly Parallel pattern in the MPI and C environment.

3.2.2 Applicability

This pattern can be used after the program is designed using patterns of the finding

concurrency design space and patterns of algorithm structure design space. And, the

resulted design is a simplest form of the Embarrassingly Parallel pattern. The simplest

form of Embarrassingly Parallel pattern satisfies the conditions as follows: All the tasks

are independent. All the tasks must be completed. Each task executes same algorithm on

a distinct section of data.












3.2.3 Implementation

The common MPI implementations have a static process allocation at initialization

time of the program. It also executes same program at each processing unit (or

workstation). Since all the tasks executes same algorithm on a different data, the tasks

can be implemented as one high level function which will be executed in each process.

The simplest form of Embarrassingly Parallel pattern can be implemented by the

following five steps.

1. Find out the number of available processing units.

2. Distribute or replicate data to each process.

3. Execute the tasks.

4. Synchronize all processes.

5. Combine (reduce) local results into a single result.

In MPI, the number of available processing units can be found by calling the

MPICOMM SIZE function and passing the MPI_WORLDCOM communicator as a

parameter.

In most cases, after finding out the number of available processing units, data can be

divided by that number and distributed into each processing unit. There are several

primitive functions for distribution of data in MPI. The three basic functions are

Broadcast, Scatter, and Gather. These functions are introduced in 3.1.3.3.

After computing local results at each process, synchronize all processes to check that

all local results are computed. This can be implemented by calling the MPIBARRIER

function after each local result is solved at each processing unit because the

MPIBARRIER blocks the caller until all group members have called it.












The produced local results can be combined to get the final solution of the problem.

The Reduce functions of MPI are useful in combining subsolutions. The Reduce function

of MPI combines the elements, which are provided in the input buffer of each process in

the group, using the specified operation as a parameter, and returns the combined value in

the output buffer of the process with rank of root. The operations are distributed to each

process in many MPI implementations so they can improve overall performance of the

program if it takes considerable time to combine subsolutions in one processing unit.

3.2.4 Implementation Example

#include
#define SIZEOFDATA 2000 /* size of data to be distributed into each process */
int root = 0; /* The rank(ID) of root process */
int myRank; /* process ID */

/*= Modificationl ======================================== */
/*= The data type of each variable should be modified =
int* data; /*= The data to be distributed into each process =*/
int* localData; /*= distributed data that each process will have =*/
int* subsolution; /*= subsolutaion data =
int* solution; /*= solution data =


int sizeOfLocalData; /* number of elements in local data array */
int numOfProc; /* number ofavaiable process */
int numOfElement; /* number of element in subsolution */



/* Highest levelfunction which solves subproblem */

void solve()
{/*= Implementation ==============================/
/*=The code for solving subproblem should be implemented =/

}



Main Function which starts program */
*******************************












main(argc, argv)
int argc;
char **argv;
Status status;
MPI Status status;


/* MPI initialization */
MPI_Init(&argc, &argv);

/* Finding out the rank (ID) of each process */
MPI_Comm_rank(MPI_COMM WORLD,&myRank);

I id rin1g out the number ofavailable processes */
MPI_Comm_size(MPI_COMM WORLD,&numOfProc);

* Dividing the data by the number ofprocesses */
sizeOfLocalData = SIZEOFDATA/numOfProc;

/* Dynamic memory allocation for local data */
localData=(int *)malloc(sizeOfLocalData*sizeof(int));


/* Distribute data into local data of each process */
/*****************************************************************/


MPI_Scatter(


data,
sizeOfLocalData,
MPI INT,


localData,
sizeOfLocalData,

MPI INT,


/*Address of send buffer (data) to distribute */
/* Number of element sent to each process */
/*MODIFICATION2=================*/
/*= data type of the send buffer
/*---------------------------------7/
/* Address of receive buffer (local data) */
/* Number of element in receive buffer (local */
/* data) */
/*= MODIFICATION2=============== *
/*= Data type of receive buffer =


root, /* Rank of sending process *
MPI COMM WORLD /* Communicator *


%/ \,/ sub-problem in each process */
solve();

*Synchronize all processes to check all subproblems are solved */
MPIBarrier(MPI_COMM WORLD);












/* Combine sub-solutions to get the solution */

MPIReduce( subsolution, /* address of send buffer (subsolution) */
solution, /* address of receive buffer (solution) */
numOfElement, /* number of elements in send buffer
(subsolution) */
MPI INT, /*= MODIFICATION3============ /
/*= data type of the send buffer =
7*-------------------------------*7
----------------------------------------*/
MPI MULT, /*= MODIFICATION4 ===-=========/
/*= reduce operation = *
/*. */-------------------------
root, /* rank of root process */
MPI COMM WORLD /* communicator */


MPIFinalize();
}

3.2.5 Example Usage

The steps to reuse the Implementation Example are as follows:

* Implement the blocks labeled as IMPLEMENTATION1. This block should contain
code for the "solve" function.

* Modify the data type of the variables which are in the block labeled as
MODIFICATION.

* Modify the data type parameters which are in the block labeled as MODIFICATION2.
Each data type parameter must be one of MPI data type which matches with the type
of the data to receive and send.

* Modify the data type parameters which are in the block labeled as MODIFICATION3.
Each data type parameter must be one of MPI data type which matches with the type
of the data to receive and send

* Modify the reduce operator parameter which are in the block labeled as
MODIFICATION4. This parameter should be one of MPI operators.

Consider a simple addition of each element in an integer array with size 1024. What

should be modified is the solve function and fifth parameter of MPIReduce function as

follows:


void solve()













for(i=0;i {
subSolution=subSolution+localData[i];
}
}

MPIReduce( subsolution,
solution,
numOfElement,
MPIINT, /*Modified */
MPISUM, /* Modified */
root,
MPI COMM WORLD


3.3 Implementation of Embarrassingly Parallel

3.3.1 Intent

This pattern shows how to implement Embarrassingly Parallel pattern in MPI and C

environment

3.3.2 Applicability

This pattern can be used after the program is designed using patterns of finding

concurrency design space and patterns of the algorithm structure design space. A simple

form of the Embarrassingly-Parallel-Algorithm-Structure pattern is that the order of

priority of all the tasks is known and all tasks must be completed and all the tasks are

independent. The Implementation Example is for the case that the resulted design is a

simple form of the Embarrassingly Parallel Algorithm Structure pattern.

3.3.3 Implementation

The common implementation of MPI executes same programs concurrently at each

processing element and makes message passing on a needed basis. Since the design of the

parallel program has independent tasks, those tasks can be executed without restriction on

the order of tasks execution.












If each task has known amount of computations, the implementation of a parallel

program is straightforward in MPI and C environment. Each task can be implemented as

a function and can be executed at each process using process ID (rank) and if else

statement or switch statement. Since the amount of computation of each task, the load

balance should be achieved by distributing the tasks among processes when the program

is implemented.

If the amount of computation is not known, there should be some dynamic tasks

scheduling mechanism for the load balance. The MPI does not provide primitive process

scheduling mechanism. One way of simulating dynamic tasks scheduling mechanism is

using a task-name queue and primitive message passing of MPI. We implemented this

mechanism as follows: Each task is implemented as a function. The root process (with

rank 0) has a task-name queue which contains the names of tasks (functions). Each

process sends message to the root process to get task name, whenever it is idle or finished

its task. The root process with rank 0 waits for messages from other processes. If the root

process receives messages and the task queue is not empty, it sends back the name of task

in the task queue to the sender process of that message. When the process, which sent

message for a task name, receives the task name, it executes the task (function). The

Implementation Example is provided in 3.3.4.

The MPI routines, which is useful for the combining the locally computed

subsolutions into a global solution, are described in 3.1.3.3

3.3.4 Implementation Example

#include
#include

#define EXIT -1 /* exit message *
#define TRUE 1












#define FALSE 0
#define ROOT 0
#define NUM OF ALGO 9

int oneElm =1; /* one element */
int tag =0 ;
int algoNameSend= -1;
int idleProc = -1;
int algoNameRecv= -1;

/* Size of Data to send and Receive */
int sizeOfData;

/* number of tasks for each algi riilui */
int numOfTask[NUM OF ALGO];

int isAlgorithmName = TRUE;
int moreExitMessage = TRUE;
MPI Status status;



/* Simple Queue implementation. *


int SIZEOF_Q = 20;
int* queue;
int qTail = 0;
int qHead = 0;


/* insert *

void insert(int task) {
if(qHead-qTail==l)
{ /* when queue is full, increase array */
int* temp = (int *) malloc(2*SIZEOFQ*sizeof(int));
int i;
for(i=0;i {
temp[i]=remove();
}
queue=temp;
SIZEOF_Q = SIZEOF_Q*2;
}
else if(qTail2) {











/* queue is not full and there is more than one space *
qTail++;
queue[qTail]= task;

}
else{
/* there is just one more space in queue *
qTail=0;
queue[qTail]= task;
}
}

/* check whether or not queue is empty *

int isEmpty()
{
if(qTail==qHead) {/* queue is empty *
return 1;
}
else{
return 0;
}
}

* remove *

int remove()
{
if(qHead qHead++;
return queue[qHead];
}
else { qHead =0;
return queue[SIZEOF_Q 1];
}
return 0;
}


/* Each function should correspond to an algritiihn for tasks. *
/* If several tasks have same alg 'riltin, then one function for them *
/* More algorihtm for tasks can be added as a function *


void algorithml() {
/*= DATA TRANSFER 1 =================================*/












/*= the data which will be used by this algorithm in local process =/
/*= More of this block can be added on needed basis

int* data; /* Modify the data type*/
/* receive the size of data to receive *
MPIRecv(&sizeOfData,oneElm,MPI_INT,ROOT,
tag,MPI_COMM WORLD,&status);

/* dynamic memory allocation */
data = (int*/* Modify the data type*/)
malloc(sizeOfData* sizeof(int/* Modify the data type*/));

/* Receive the input data */
MPI_Recv(&data,sizeOfData,
MPIINT, /*= <- Modify the data type =*/
ROOT,tag,MPI_COMM WORLD,&status);
/*= =*/


/*=IMPLEMENTA TION1 ============================== *
*= code for this algorithm should be implemented


/* Memory deallocation ex) free(data); *

void algorithm2() {

7*=DAA RANFE 1-----------------------------------/
/*=DATA TRANSFER 1 =================================*/
/*= the data which will be used by this algorithm in local process =*/
*= More of this block can be added on needed basis
int* data; /* Modify the data type*/
/* receive the size of data to receive *
MPIRecv(&sizeOfData,oneElm,MPI_INT,ROOT,
tag,MPI_COMM WORLD,&status);

/* dynamic memory allocation */
data = (int*/* Modify the data type*/)
malloc(sizeOfData* sizeof(int/* Modify the data type*/));

/* Receive the input data */
MPI_Recv(&data,sizeOfData,
MPI INT, /*= <- Modify the data type =*
ROOT,tag,MPI_COMM WORLD,&status);
/*= =*/
/* ---------------------------------------- /












/*=IMPLEMENTA TION =============================
/*= code for this algorithm should be implemented


/* Memory deallocation ex) free(data); */
}

void algorithm3) {
7* DT TANFR ----------------------------------------------*7
/*=DATA TRANSFER 1 ================================= /
/*= the data which will be used by this algorithm in local process =/
/*= More of this block can be added on needed basis

int* data; /* Modify the data type*/
/* receive the size of data to receive */
MPIRecv(&sizeOfData,oneElm,MPI_INT,ROOT,
tag,MPI_COMM WORLD,&status);

/* dynamic memory allocation */
data = (int*/* Modify the data type*/)
malloc(sizeOfData*sizeof(int/* Modify the data type*/));

/* Receive the input data */
MPI_Recv(&data,sizeOfData,
MPI INT, /*= <- Modify the data type =*
ROOT,tag,MPI_COMM WORLD,&status);
/*= =*/
7*------------------------------------------------*

/*=IMPLEMENTA TIONJ--------------------------------*7k
=IMPLEMENTATION ============================/
*= code for this algorithm should be implemented


/* Memory deallocation ex) free(data); */}

****************************
/* main method *
****************************
void main( argc, argv )
int argc;
char **argv;
{
int myRank;
int mySize;

int i;
intj;














MPI_Init(&argc, &argv);

/*find local rank*/
MPI_Commrank(MPI_COMM WORLD, &myRank);

/*find out last rank by using size of rank*/
MPI_Comm_size(MPI_COMMWORLD,&mySize);


/* This process distributes tasks to other processing elements */


if(myRank == 0)
{/* code for root process.
This root process receives messages from other processes when they are idle.
Then this process removes an alg ,rilth name for a task from task name queue
and send it back to sender of the message.
If the task queue is empty, this process sends back an exit message */

/*=MODIFICA TION1================================= */
/*= the datafor each algorithm
/*= the data type should be modified =*
int* algolData;
int* algo2Data;
int* algo3Data;
int* algo4Data;
int* algo5Data;



int numOfSentExit = 0;

/* This array is a task queue. */
queue = (int *) malloc(SIZEOF_Q*sizeof(int));

/* */
for(i=0;i {
numOfTask[i] = mySize;
}
for(i=0;i f = for(j=0;j











insert(i);
}
}

/* Receive message from other processes *
while(moreExitMessage)
{
int destination= ROOT;
/* Receive message from any process *
MPIRecv(&algoNameRecv,oneElm,MPI_INT,MPIANY_SOURCE,
MPIANYTAG,MPICOMM WORLD,&status);
destination = algoNameRecv;

if(!isEmpty())
{
/* If the task queue is not empty, send back an algorihtm name for a task
to the sender of received message *
algoNameSend = queue[remove()];
MPI_Send(&algoNameSend,oneElm,MPIINT,
destination,tag,MPI_COMMWORLD);

switch(algoNameSend)
{
case 1:
implementationION2===========================/
/*= code for calculating the size of data to send and
/*=finding the starting address of data to send
*= should be implemented =*
7*-____________________________---------------------------------------------------------*7

/*=DA TA TRANSFER2============================= *
/*= More of this block can be added on needed basis
/*= =*/
MPI_Send(&sizeOfData,oneElm,MPIINT,
destination,tag,MPI_COMMWORLD);
MPI_Send(
&algo 1Data, /*= <- Modify the initial address of data =*/
sizeOfData,
MPI INT /*= <- Modify the data type =*/
,destination,tag,MPI_COMMWORLD);
/*= =*/
7*-----------------------------------------------*

break;
case 2:
/*=IMPLEMENTA TION2========================= */













/*= code for calculating the size of data to send and =
/*=finding the starting address of data to send =*
*= should be implemented =*


/*=DATA TRANSFER2=============================*/
*= More of this block can be added on needed basis
/*= =*/
MPI_Send(&sizeOfData,oneElm,MPIINT,
destination,tag,MPI_COMM_WORLD);
MPI_Send(
&algo 1Data, /* <- Modify the initial address of data =*
sizeOfData,
MPI INT *= <- Modify the data type =*/
,destination,tag,MPI_COMMWORLD);
/*= =*/

break;
case 3:
implementationION2============================= /
/*= code for calculating the size of data to send and
/*=finding the starting address of data to send
*= should be implemented =*
7*----------------------------------------------- /

/*=DATA TRANSFER2=============================*/
*= More of this block can be added on needed basis
/*= =*/
MPI_Send(&sizeOfData,oneElm,MPI_INT,
destination,tag,MPI_COMMWORLD);
MPI_Send(
&algo 1Data, /* <- Modify the initial address of data =*
sizeOfData,
MPI INT *= <- Modify the data type =*/
,destination,tag,MPI_COMMWORLD);
/*= =*/



break;
default:
break;



else
{ *If the task queue is empty, send exit message *












algoNameSend = EXIT;
MPI_Send(&algoNameSend,oneElm,MPIINT,
destination,tag,MPI_COMM WORLD);

/* keep tracking the number of exit message sent.
If the number of exit messages is same with the number ofprocesses,
root process will not receive any more message requesting tasks. *

numOfSentExit++;
if(numOfSentExit==mySize-1)

moreExitMessage=FALSE;



/* computation and/or message passingfor combining subsolution can be
added here *



/**************************************************************
/* Code for other processes, not root. These processes send message *
/* requesting next task to execute to root process when they are idle. *
/* These processes execute tasks received from root *

else

idleProc = myRank;
/*Send message to root process, send buffer contains the rank of
sender process whenever it is idle */
MPI_Send(&idleProc,oneElm,MPI_INT,ROOT,tag,MPI_COMM WORLD);
while(isAlgorithmName) / i I//l' message contains an algorihtm name *

/* Receive message from root which contains the name of task
to execute in this process */

MPIRecv(&algoNameRecv,oneElm,MPI_INT,ROOT,tag,
MPICOMM WORLD,&status);


/* each process executes tasks using the algorihtm name
and data received from root *
switch(algoNameRecv)
S/* More case statements should be added or removed, according to
the number of tasks */
case EXIT : isAlgorithmName = FALSE;












break;
case 1:
algorithm ();
break;
case 2:
algorithm2();
break;
case 3:
algorithm3(;
break;
case 4:
algorithm4();
break;
case 5:
algorithmS();
break;
default:
break;
}
if(algoNameRecv != EXIT)

/*Send message to root process, send buffer contains the rank of
sender process whenever it is idle*/
idleProc = myRank;
MPI_Send(&idleProc,oneElm,MPIINT,ROOT,tag,MPICOMM WORLD);


/*= Codes for collecting subsolution and =
/*= computing final solution can be added here. =
/* =====================================/
MPIFinalize();


3.3.5 Usage Example

The steps to reuse this Implementation Example are as follows:

* Implement the blocks labeled as IMPLEMENTATION1. Each block should contain
codes for each algorithm.

* Implement the block labeled as IMPLEMENTATION2. What should be implemented
are codes for calculating the initial address and the number of elements of the send
buffer for each algorithm.












* Modify the data type of the variables and data type parameters which are in the
blocks labeled as DATA TRANSFER 1. The data type of each variable should match
with the data type of the input data for each algorithm. The purpose of this "send"
function is to send input data to the destination process. More of this block can be
added on needed basis.

* Modify the data type and data type parameters which are in the block labeled as
DATA TRANSFER2. Each data type parameter must be one of MPI data type which
matches with the type of the data to receive. The purpose of these "receive" function
is to receive input data for the algorithm execution on local process. More of this
block can be added on needed basis

Assume that there is a parallel program design such that the amount of computation

for each task is unknown. If some of the tasks share same algorithm, these tasks should

be implemented as one algorithm function in one of the block labeled as

IMPLMENTATION1. To execute the tasks, the same number of algorithm name with the

number of tasks that share same algorithm should be put in the queue.

3.4 Implementation of Pipeline Processing

3.4.1 Intent

This is a solution to the problem of how to implement the Pipeline Processing pattern

in the MPI and C environment.

3.4.2 Applicability

This pattern can be used after the program is designed using patterns of finding

concurrency design space and patterns of the algorithm structure design space and the

resulted design is a form of the Pipeline Processing pattern.

3.4.3 Implementation

Figure3.1 illustrates the usage of message passing to schedule tasks in the

Implementation Example. The arrows represent the blocking synchronous send and

receive in MPI. The squares labeled Cl, C2, etc., in Figure3.1, represent the elements of

calculations to be performed which can overlap each others. The calculation elements are












implemented as functions in the Implementation Example. The calculation elements

(functions) should be filled up to do computation by the user of this Implementation

Example. Adding more functions for more calculation is trivial because message passing

calls inside functions are very similar between functions.

Each Stage (Stagel, Stage2, and etc.), in Figure3.1, corresponds to each process. In

the Implementation Example, each Stage calls single function (calculation element)

sequentially.

time


Process Cl C2 C3 C4
1



Process Cl C2 C3 C4
2



Process Cl C2 C3 C4
3



Process Cl C2 C3 C4
4


Figure 3-2 Usage of Message Passing

One point to note is that the first pipeline stage do not receive message and last

element do not send message for scheduling of tasks.

Scheduling of tasks is achieved by blocking and synchronous mode point-to-point

communications (MPI_SSEND, MPIRECEIVE).












In MPI, if the Send mode is blocking and synchronous and the Receive mode is

blocking, then the Send can complete only if the matching Receive has started. The

statements after the matching receive will not be executed until the receive operation is

finished: MPI provides synchronous communication semantics.

A duplicate of MPI_COMMWORLD (initial intra-communicator of all processes) is

used to provide separate communication space and safe communication.

If we change the duplicate of MPICOMMWORLD into appropriate communicator

for a group of processes then, this structure can be used for a portion of a program where

the Pipeline Processing pattern can be used.

3.4.4 Implementation Example

#include

#include
#include

int numOfCalElem = 4;
char startMsg[7];
int myRank;
int mySize;
int tag = 1;

char* sendBuf, recvBuf;
int sendBufSize, recvBufSize;

MPI Status status;

/* Only four pipeline stages are implemented in this Implementation Example.
More elements can be added or removed, according to the design
of the parallel program */

/******************/
/*First Pipeline Stage */
/******************/
void firstPipelineStage(MPI_Comm myComm)
{
/*=IMPLEMENTA TION1========================================*/
/*= =*/













*= code for this Pipeline Stage should be implemented here
/*= =*/



/* send message to the next Pipeline Stage of process with myRank+ *
MPI_Ssend(startMsg,strlen(startMsg),MPI_CHAR
,myRank+l ,tag,MPI_COMM WORLD);


/*=DA TA TRANSFER 1======================================
*= More send function, which has same structure, can be added =*
*= to transfer data =*/
MPI_Ssend( *= Modify the following parameters =/
sendBuf, /* <- starting address of buffer
sendBufSize, /* <- the number of elements in buffer
MPI_CHAR, /* <- the data type =*

myRank+1,tag,MPI_COMM WORLD);


*********************
/* Pipeline Stage 2 *
*********************
void pipelineStage2(MPI_Comm myComm)

/* Receive message from the previous Pipeline Stage ofprocess with myRank-1 *
MPI_Recv(startMsg,strlen(startMsg),MPI_CHAR,myRank-1,
tag, MPICOMM WORLD,&status);

/*=DA TA TRANSFER 2====================================*/
*= More receive functions, which has same structure, can be added =*
*= to transfer data
MPI_Recv( *= Modify the following parameters =*/
recvBuf, *= <- starting address of the buffer
recvBufSize, *= <- number of elements to receive =/
MPI_CHAR, *= <- data type =
7*----------------------------------*7
myRank-1,tag, MPI_COMM WORLD,&status);


/*=IMPLEMENTA TION1========================================*/
/= =
/*= code for this Pipeline Stage should be implemented here =/
/*= =*/
/7*... *............................/.......













/* send message to the next Pipeline Stage of process with myRank+ *
MPI_Ssend(startMsg,strlen(startMsg),MPI_CHAR,myRank+l,tag,
MPICOMM WORLD);

/*=DA TA TRANSFER 1======================================
*= More send function, which has same structure, can be added
*= to transfer data =*/
MPI_Ssend( *= Modify the following parameters =/
sendBuf, /* <- starting address of buffer =*
sendBufSize, /* <- the number of elements in buffer
MPI_CHAR, /* <- the data type =*
/ --------------------------------------------*7
myRank+1,tag,MPI_COMM WORLD);


*********************
/* Pipeline Stage 3 *
*********************
void pipelineStage3(MPI_Comm myComm)

/* Receive message from the previous Pipeline Stage ofprocess with myRank-1 *
MPI_Recv(startMsg,strlen(startMsg),MPI_CHAR,myRank-1,
tag, MPICOMM WORLD,&status);

/*=DA TA TRANSFER 2====================================*/
*= More receive functions, which has same structure, can be added =*
*= to transfer data =*/
MPI_Recv( *= Modify the following parameters =/
recvBuf, *= <- starting address of the buffer
recvBufSize, *= <- number of elements to receive
MPI_CHAR, /*= <- data type =
7*-----------------------------------*7
myRank-1,tag, MPI_COMM WORLD,&status);

/*=IMPLEMENTA TION1======================================= *
/*= =*/
*= code for this Pipeline Stage should be implemented here
/*= =*/



/* send message to the next Pipeline Stage of process with myRank+ *
MPI_Ssend(startMsg,strlen(startMsg),MPI_CHAR,myRank+l,tag,

MPICOMM WORLD);

/*=DATA TRANSFER 1======================================













*= More send function, which has same structure, can be added
/*= to transfer data =*/
MPI_Ssend( *= Modify the following parameters =/
sendBuf, /* <- starting address of buffer
sendBufSize, /* <- the number of elements in buffer
MPI_CHAR, /* <- the data type =*
/ --------------------------------------------*7
myRank+1,tag,MPI_COMM WORLD);



/* Pipeline Stage 4 *

void pipelineStage4(MPI_Comm myComm)


/* Receive message from the previous Pipeline Stage ofprocess with myRank-1 *
MPI_Recv(startMsg,strlen(startMsg),MPI_CHAR,myRank-1,
tag, MPICOMM WORLD,&status);

/*=DA TA TRANSFER 2====================================*/
*= More receive functions, which has same structure, can be added =/
*= to transfer data =*/
MPI_Recv( /*= Modify the following parameters =/
recvBuf, /= <- starting address of the buffer
recvBufSize, /*= <- number of elements to receive
MPI_CHAR, /*= <- data type =
7*-----------------------------------*7
myRank-1,tag, MPI_COMM WORLD,&status);

/*=IMPLEMENTA TION1=======================================*/
/*= =*/
*= code for this Pipeline Stage should be implemented here
/*= =*/



/* send message to the next Pipeline Stage of process with myRank+ *
MPI_Ssend(startMsg,strlen(startMsg),MPI_CHAR,myRank+l,tag,
MPICOMM WORLD);

/*=DATA TRANSFER 1======================================
*= More send function, which has same structure, can be added
*= to transfer data =*/
MPI_Ssend( *= Modify the following parameters =/
sendBuf, /* <- starting address of buffer
sendBufSize, /* <- the number of elements in buffer













MPI_CHAR, /* <- the data type =*

myRank+1,tag,MPI_COMM WORLD);



/* Pipeline Stage 5 *

void PipelineStage5(MPI_Comm myComm)

/* Receive message from the previous Pipeline Stage of process with myRank-1 *
MPI_Recv(startMsg,strlen(startMsg),MPI_CHAR,myRank-1,
tag, MPICOMM WORLD,&status);

/*=DA TA TRANSFER 2====================================*/
*= More receive functions, which has same structure, can be added =*
*= to transfer data =*/
MPI_Recv( /*= Modify the following parameters =/
recvBuf, /*= <- starting address of the buffer
recvBufSize, /*= <- number of elements to receive
MPI_CHAR, *= <- data type =
7*-----------------------------------*7
myRank-1,tag, MPI_COMM WORLD,&status);



/*=IMPLEMENTA TION1======================================= /
/*= =*/
/*= code for this Pipeline Stage should be implemented here
/*= =*/



/* send message to the next Pipeline Stage of process with myRank+ *
MPI_Ssend(startMsg,strlen(startMsg),MPI_CHAR,myRank+l,tag,
MPICOMM WORLD);

/*=DA TA TRANSFER 1======================================
/= More send functions, which has same structure, can be added
/= to transfer data = *
MPI_Ssend( *= Modify the following parameters =/
sendBuf, /* <- starting address of buffer =*
sendBufSize, /* <- the number of elements in buffer
MPI_CHAR, /* <- the data type =*

myRank+1-,tag,MPICOMM--WORLD);
myRank+1 ,tag,MPICOMMWORLD);














/* Last Pipeline Stage *
*********************

void lastPipelineStage(MPI_Comm myComm)
{
/* Receive message from previous Pipeline Stage ofprocess with myRank-1 *
MPI_Recv(startMsg,strlen(startMsg),MPI_CHAR,myRank-1,
tag, MPICOMM WORLD,&status);

/*=DA TA TRANSFER 2====================================*/
/*= More receive functions, which has same structure, can be added =*/
/*= to transfer data =*/
MPI_Recv( *= Modify the following parameters =/
recvBuf, /*= <- starting address of the buffer
recvBufSize, /*= <- number of elements to receive
MPI_CHAR, *= <- data type =
7*-----------------------------------*7
myRank-1,tag, MPI_COMMWORLD,&status);

/*=IMPLEMENTA TION1======================================= /
/*= =*/
*= code for this Pipeline Stage should be implemented here
/*= =*/




void main(argc,argv)
int argc;
char **argv;

int i = 0;
MPI_Comm myComm;

MPI_Init(&argc, &argv);

MPI_Commdup(MPI_COMM WORLD, &myComm);

/*find rank of this process*/
MPI_Comm_rank(myComm, &myRank);

/*find out rank of last process by using size of rank*/
MPI_Comm_size(myComm,&mySize);


strcpy(startMsg," start");












switch(myRank)
{
case 0 :
for(i=0;i firstPipelineStage(myComm);
break;
case 1 :
for(i=0;i pipelineStage2(myComm);
break;
case 2 :
for(i=0;i pipelineStage3 (myComm);
break;
case 3 :
for(i=0;i pipelineStage4(myComm);
break;
case 4 :
for(i=0;i pipelineStage5(myComm);
break;

/*=MODIFICA TION2======================= /
/*= More case statement can be added or removed =/
/= on needed basis. =/
/*__ ______________________-- ------------------------------------*7

case 5 :
for(i=0;i lastPipelineStage(myComm);
break;
default:
break;
}

MPIFinalize();

3.4.5 Usage Example

The steps to use this frame are as follows:

*Add or remove the pipeline stage functions according to the number of pipeline
stages of the program design. Each pipeline stage function has same structure of
message passing, so it can be coped and pasted to implement more pipeline stages.
But the first and last pipeline stage should not be removed because the first pipeline
stage does not receive messages and the last pipeline stage does not send messages.












* Add or remove the case statements, in the block labeled as MODIFICATION2,
according to the number of pipeline stages of the program design.

* Implement the blocks labeled as IMPLEMENTATION1. Each block should contain
codes for each calculation element.

* Modify the initial address of send buffer, the number of elements in send buffer, and
the data type parameters which are in the block labeled as DATA TRANSFER1. The
data to be sent is the input for the next calculation element. More send functions,
which have same structure, can be added according to the need.

* Modify the initial address of receive buffer, the number of elements in receive buffer,
and the data type parameters which are in the block labeled as DATA TRANSFER2.
The data to be received is the input for this calculation element. More receive
functions, which have same structure, can be added according to the need.

Consider following problem as an example problem to parallelize: There are four SAT

scores of four schools. Find out standard deviation for each class. But the main memory

of one processor barely contains the scores of one school.

To solve this problem, the pipeline stages can be defined as follows: The first pipeline

stage computes the sum and average of SAT scores of each school. The second pipeline

stage computes the differences between each individual score and the average. The third

pipeline stage computes squares of each difference. Forth pipeline stage computes the

sum of the computed squares.

This problem can be easily implemented by following the above steps. The

implementation of this is provided in appendix.

3.5 Implementation of Asynchronous-Composition

3.5.1 Intent

This is a solution of the problem how to implement the parallel algorithm which

resulted from the Asynchronous-Composition pattern in the algorithm structure design

space in MPI (Message Passing Interface)?












3.5.2 Applicability

Problems are represented as a collection of semi-independent entities interacting in an

irregular way.

3.5.3 Implementation

Three points that need to be defined to implement the algorithm resulting from

Asynchronous-Composition pattern using MPI are tasks/entities, events, and task

scheduling.

A task/entity, which generates an event and processes them, can be represented as a

process in an implementation of an Algorithm using MPI. An event corresponds to a

message sent from the event generating task to the event processing task. All tasks can be

executed concurrently.

In this Implementation Example, each case block in a switch statement should contain

code for each semi-independent entity (process) which will be executed concurrently.

An event corresponds to a message sent from the event generating task (process) to the

event processing task (process) in MPI and C environment. Therefore, event handling is a

message handling. For safe message passing among processes, the creation of groups of

processes which need to communicate is necessary. Creating communicators for the

groups in MPI is also necessary. Because of that, we added groups and communicators

creations in this Implementation Example.

This Implementation Example is also an example of event handling in MPI

environment. In a situation that an entity (process) receives irregular events (messages

sent) from other known entities (processes), we can implement it using defined constant

of MPI, MPI ANY SOURCE and MPI ANY TAG.












Assume that process A, B and C sends messages in an irregular way to process D and

the process D handles (processes) events (messages). It is same situation as the car/driver

example of the Asynchronous-Composition pattern. To use the MPIANY_SOURCE and

MPIANYTAG in the receive routine of MPI, we need to create an dedicated group and

communicator for these entities (processes) to prevent entity (process) D receives

messages from other entities (processes) that is not intended to send message to the entity

(process) D.



Process

B







Process Process Process

D C




Figure 3-3 Irregular Message Handling
3.5.4 Implementation Example

#include
#include
#include


main(argc, argv)
int argc;
char **argv;
{


/* More group variables can be added or removed on need basis*/












MPIGroup MPIGROUP_WORLD,groupA,group B;

/*More communicator variables can be added or removed on need basis */
MPIComm comm A,comm B;

/*This varilable will hold the rank for each process in MPI default communicator
MPI COMM WORLD */
int rank in world;

/*This variable will hold the rank for each process in group A.
Variables can be added or removed on need basis. */
int rankin_group A;
int rankin_group B;

/* ranks ofprocesses which will be used to create subgroups.
More array of ranks can be added or removed on need basis. */
int ranksa[= {1,2,3,4};
int ranksb[]= {1,2,3,4};

MPI_Init(&argc, &argv);

/*Create a group ofprocesses and communicator for a group.
More groups and communicator can be created or removed on need basis*/

MPI_Comm_group(MPI_COMMWORLD,&MPIGROUPWORLD);

/*Create group and communicator */
MPI_Group_incl(MPI_GROUPWORLD,4,ranksa,&groupA);
MPI_Comm_create(MPI_COMM WORLD,group A,&comm A);

/*Create group and communicator */
MPI_Group_incl(MPI_GROUPWORLD,4,ranksb,&groupB);
MPI_Comm_create(MPI_COMM WORLD,group B,&comm B);

MPI_Comm_rank(MPI_COMM WORLD, &rank in world);

switch(rankin world)
{

/*This case 1 contains codes to be executed in process 0 */
case 0:
{
/* events can be generated or processed */

/* work */
break;














/*This case contains codes to be executed in process 1 *
case 1:
{
char sendBuffer[20];
int isOn=l;
int i=0;

while(isOn)
{
/* works that need to be done before generating event *
strcpy(sendBuffer,"event");/* an example *

/* Generate Event (message passing) */
MPI_Send(sendBuffer,strlen(sendBuffer),MPI_CHAR,3,1 ,commB);
printf("sent message");
i++;

/* break loop *
if(i==10) /* this should be changed according to the problem *
{
isOn = 0;
}


break;



/*This case contains codes to be executed in process 2 *
case 2:
{
char sendBuffer[20];
int isOn=l;
int i=0;

while(isOn)
{
/* works that need to be done before generating event *
strcpy(sendBuffer,"event");/* an example */

/* Generate Event (message) */
MPI_Send(sendBuffer,strlen(sendBuffer),MPI_CHAR,3,1 ,commB);
i++;












/* break loop *
if(i==10) /* this should be changed according to the problem *


isOn = 0;


break;
}

/*This case contains codes to be executed in process 3 *
case 3:
{
char sendBuffer[20];
int isOn=l;
int i=0;

while(isOn)
{
/* works that need to be done before generating event *
strcpy(sendBuffer," event"); /* an example *

/* Generate Event (message) */
MPI_Send(sendBuffer,strlen(sendBuffer),MPI_CHAR,3,1 ,commB);
i++;

/* break loop *
if(i==10) /* this should be changed according to the problem *


isOn = 0;


break;
}

/*This case contains codes to be executed in process 4 *
case 4:
{
char receiveBuffer[20];
int isOn = 1;
int messageCount=0;
MPI Status status;

while(isOn)
{
MPI_Recv(receiveBuffer,20,MPI_CHAR,MPI ANY_SOURCE,
MPIANYTAG,commB,&status);












messageCount++;
if(0==strncmp(receiveBuffer," event",3))
{
/* work to process the event (message) */
printf("\nreceived an event at process 4");

if(messageCount==30)
{
isOn = 0;
}
}
break;
}
/*
more cases(processes) can be added or removed.
*/
default: break;
}
MPIFinalize();
}
3.6 Implementation of Divide and Conquer

3.6.1 Intent

For the completeness of the parallel pattern language, it would be beneficial for a

programmer to show an implementation example of a top-level program structure for the

Divide-and-Conquer pattern in the Algorithm Design space.

3.6.2 Motivation

The top-level program structure of the divide-and-conquer strategy that is stated in the

Divide-and-Conquer pattern is as follows

Solution solve(Problem P){
if (baseCase (P))
return baseSolve(P);
else {
Problem subProblems[N];
Solution subSolutions[N];
subProblems = split(P);
for (int i=0;i subSolutions[i] = solve(subProblems[i]);
return merge(subSolutions);














This structure can be mapped onto a design in terms of tasks by defining one task for

each invocation of the solve function.


The common MPI implementations have a static process allocation at initialization

time of the program. We need a mechanism to invoke the solve function at another

processor whenever the solve function calls the split function, and the split function splits

the problem into sub-problems so that subproblems can be solved concurrently.

3.6.3 Applicability

This pattern can be used after the program is designed using patterns of the parallel

pattern language and the resulting design is a form of the Divide-and-Conquer pattern

and if we want to implement the design in an MPI and C environment. This structure can

be used as an example of how to implement a top-level program structure in an MPI and

C environment or directly adopting this structure as an implementation of the program by

adjusting control parameters and adding a computational part of the program.

3.6.4 Implementation

We are trying to implement a top-level program structure of the divide-and-conquer

strategy in MPI and C so that parallel program design resulting from the Divide-and-

Conquer pattern can be implemented by filling up each function or/and adjusting the

structure. The basic idea is using message passing to invoke the solve functions at other

processing elements when needed. In a standard MPI and C environment, each

processing element has the same copy of the program with other processing elements,

and same program executes at each processing element communicating with each other

on a needed basis. When the problem splits into subproblems, we need to call the solve














function at other processes so that subproblems can be solved concurrently. To call the

solve functions, the split function send messages to other processing elements and the

blocking MPI_Receive function receives messages before the first solve function at each

process. This message passing can contain data/tasks divided by the split function. In this

structure, every solve function calls the merge function to merge subsolutions.

CPU CPUz CPU2 CPU3 CPU4 CPU5 CPU6 CPU7


solve


split
Message sent from split to solve

solve solve


split split


solve solve solve solve
solvesolve solvesove


split split split split


solve solve solve solve solve solve solve solve


merge merge [merge merge merge merge merge merge




merge merge merge merge



merge merge



Figure 3-4 Message Passing for Invocation of the Solve and Merge Functions

For simplicity, we split a problem into two subproblems and used problem size to

determine whether or not the subproblem is a base case. Figure3.3 shows the sequence of











function calls in each process and message passing to invoke the solve function at remote

process for new sub-problem and to merge sub-solutions.

3.6.5 Implementation Example

#include

#include
#include

#define DATA TYPE int
int numOfProc; / imuiinlr of available pt i,'L l', ,'

int my_rank;
int ctrlMsgSend;
int ctrlMsgRecv;
int* localData;
int dataSizeSend;
int dataSizeRecv;


/* Solve a problem */

void solve(int numOfProcLeft)
{
if(baseCase(numOfProcLeft))
{
baseSolve(numOfProcLeft);
merge(numOfProcLeft);
}
else
{
split(numOfProcLeft);

if(numOfProcLeft! =numOfProc)
{
merge(numOfProcLeft);
}
}
}

/* split a problem into two subproblems */

int split(int numOfProcLeft)













/*=IMPLEMENTA TION2 ==============================
/*= Code for splitting a problem into two subproblems
/*= should be implemented =*/
/*= =*/
7*--------------------------------------------------------*7

ctrlMsgSend = numOfProcLeft/2;
/* invoke a solve function at the remote process *
MPI_Send(&ctrlMsgSend, 1,MPIINT,
my_rank+numOfProc/numOfProcLeft,
0,MPI_COMM WORLD);

/*=DATA TRANSFER 1 ==============================*/
/*= More of this block can be added on needed basis =
MPI_Send(&dataSizeSend, 1 ,MPIINT,
myrank+numOfProc/numOfProcLeft,
0, MPI_COMM WORLD);
MPI_Send(
&localData[dataSizeLeft], /*<- modify address of data *
dataSizeSend,
MPIINT, /*<-modify data type*/
my_rank+numOfProc/numOfProcLeft,
0, MPI_COMM WORLD);


/* invoke a solve function at a local process *
solve(numOfProcLeft/2);
return 0;
}


Merge two subsolutions into a solution *

void merge(int numOfProcLeft)
{
if(my_rank >= numOfProc/(numOfProcLeft*2))
{
ctrlMsgSend = 1;

/* Send a subsolution to the process from which this process got the subproblem *
MPI_Send(&ctrlMsgSend, 1 ,MPIINT,
myrank numOfProc/(numOfProcLeft*2),
0,MPI_COMM WORLD);


/*=DA TA TRANSFER 3=============================*/












/*= More of this block can be added on needed basis
MPI_Send(&dataSizeSend, 1,MPI_INT,
myrank numOfProc/(numOfProcLeft*2),
0,MPI_COMM WORLD);
MPI_Send(
localData, /*<-modify address of data */
dataSizeSend,
MPIINT, /*<-modify data type */
myrank numOfProc/(numOfProcLeft*2),
0,MPI_COMM WORLD);


}
else
{
MPI Status status;
/* Receive a subsolution from the process which was invoked by this process *
MPI_Recv(&ctrlMsgRecv, 1,MPIINT,
myrank+numOfProc/(numOfProcLeft*2),
0,MPICOMM WORLD,&status);

/*=DA TA TRANSFER 3===============================*/
/*= More of this block can be added on needed basis
MPIRecv(&dataSizeRecv, 1,MPIINT,
myrank+numOfProc/(numOfProcLeft*2),
0,MPICOMM WORLD,&status);
MPIRecv(
&localData[dataSizeLeft], /*<- modify address of data */
dataSizeRecv,
MPIINT, /*<- modify data type*/
myrank+numOfProc/(numOfProcLeft*2),
0,MPICOMM WORLD,&status);



/*=IMPLEMENTA TION3========================== /
*= Code for merging two subsolutions into one solution =*/
*= should be implemented


/ -------------------













/* Decide whether a problem is a "base case" */
/*that can be solved without further splitting */

int baseCase(int numOfProcLeft)
{
if(numOfProcLeft == 1)
return 1;
else
return 0;
}


/* Solve a base-case problem */

int baseSolve(int numOfProcLeft)
{
/*=IMPLEMENTA TION1=================================*/
/*== Code for solving base case problem should be implemented ==/
/*...... /==*
7*-------------------------------------------------*
/*=====================================*/
return 0;
}


main(argc, argv)
int argc;
char **argv;
{
int i;
MPI Status status;
MPI_Init(&argc, &argv);
MPI_Commrank(MPI_COMM WORLD, &myrank);
MPI_Commsize(MPI_COMM WORLD, &numOfProc);

count = (int *)malloc((maxlnt+l)*sizeof(int));
if(my_rank == 0) {
ctrlMsgSend=numOfProc; /* number ofprocesses must be power of 2 */
}


ctrlMsgRecv = numOfProc;
solve(ctrlMsgRecv);
}
else{
/* Every process waits a message before calling solve function */












MPI_Recv(&ctrlMsgRecv, 1 ,MPIINT,MPI ANY_SOURCE,MPIANYTAG,
MPI_COMM WORLD,&status);

/*=DA TA TRANSFER 2============================ =*/
/*= More of this block can be added on needed basis =
MPI_Recv(&dataSizeRecv, 1 ,MPIINT,MPI_ANY_SOURCE,
MPIANYTAG,MPICOMM WORLD,&status);
MPI_Recv(
localData, /*<- modify address of data */
dataSizeRecv,
MPI INT, /*<- modify data type*/
MPI ANY SOURCE,MPI ANY TAG,
MPI_COMM WORLD,&status);
/* = ====================================*/
------------------------------------*7___________ ^


dataSizeLeft = dataSizeRecv;
solve(ctrlMsgRecv);
}
if(myrank==0) {

for(i=0;i
printf(" %d",localData[i]);


MPIFinalize();
}

3.6.6 Usage Example

To use the previous framework, the number of available processes must be power of 2.

The steps to use this Implementation Example Code are as follows:

* Implement the block labeled as IMPLEMENTATION1. This block should contain
codes for "baseSolve" algorithm.

* Implement the block labeled as IMPLEMENTAION2. This block should contain
codes for "split" algorithm.

* Implement the block labeled as IMPLEMENTATION3. This block should contain
codes for "merge" algorithm.

* Find out and modify the initial address of send buffer, the number of elements in send
buffer, and the data type parameters which are in the block labeled as DATA












TRANSFER1. The data to be sent is the input for the solve function of remote
process. More of this block can be added on needed basis.

* Find out and modify the initial address of receive buffer, the number of elements to
receive, and the data type parameters which are in the block labeled as DATA
TRANSFER2. The data to receive is the input for the solve function of local process.
More of this block can be added on needed basis.

* Find out and modify the initial address of receive buffer, the number of elements to
receive, and the data type parameters which are in the block labeled as DATA
TRANSFER3. The data to receive and send are the input for the merge function of
local process. More of this block can be added on needed basis.

One point to note about using the Implementation Example is that the "baseCase" of

the Implementation Example is when there are no more available processes.

Merge sort algorithm is well known problem which uses divide and conquer strategy.

If we consider the merge sort over N integers as the problem to parallelize, it can be

easily implemented using the Implementation Example. To use the Implementation

Example, the merge sort algorithm should be modified as follows: The "baseSolve" is a

counting sort over the local data (an array of integers). The "split" algorihtm divides

received data (an array of integers) into two contiguous subarrays. The "merge"

algorithm merges two sorted array into one sorted array. The "BaseCase" of this problem

is when there are no more available processes. The problem can be implemented by

following the implementation steps above. The implementation of this problem is

provided in appendix.

















CHAPTER 4
KERNEL IS OF NAS BENCHMARK

The Numerical Aerodynamic Simulation (NAS) Program, which is based at NASA

Ames Research Center, developed a set of benchmarks for the performance evaluation of

highly parallel supercomputers. These benchmarks, NPB 1 (NAS Parallel Benchmarks 1),

consist of five parallel kernels and three simulated application benchmarks. The principle

distinguishing feature of these benchmarks is their "pencil and paper" specification. All

details of these benchmarks are specified only algorithmically. The Kernel IS (Parallel

Sort over small integers) is one of the kernel benchmark set.

4.1 Brief Statement of Problem

Sorting N integer keys in parallel is the problem. The number of keys is 223 for class

A and 225 for class B. The range of keys is [0, Bma ) where Bmax is 219 for class A and

221 for class B. The keys must be generated by the key Generation algorithm of the NAS

benchmark set. The initial distribution of the key must follow the specification of

memory mapping of keys. Even though the problem is sorting, what will be timed is the

time needed to rank every key, and the permutation of keys will not be timed.

4.2 Key Generation and Memory Mapping

The Keys must be generated using the pseudorandom number generator of the

NAS Parallel Benchmark. The numbers generated by this pseudorandom number

generator will have range (0, 1) and have very nearly uniform distribution of the unit

interval.1112 The keys will be generated using this number in the following way. Let rf












be a random fraction uniformly distributed in the range (0,1), and let K, be the ith key.

The value of K, is determined as K, <- Bmax (r4, + r4, + + r4, 2 + r4, ) / 4 for

i = 0,1,..., N -1. K, must be an integer and L[- indicates truncation. Bmax is 219 for class

A and is 221 for class B. For the distribution of keys among memory, all keys initially

must be stored in the main memory units and each memory unit must have same amount

of keys in a contiguous address space. If the keys cannot be evenly divisible, the last

memory unit can have a different amount of keys, but it must follow specification of

NAS Benchmark. See Appendix A for details.

4.3 Procedure and Timing

The implementation of Kernel IS must follow this procedure. The partial verification

of this procedure tests the ranks of five unique keys where each key has unique rank. The

full verification rearranges the sequence of keys using the computed rank of each key and

tests that the keys are sorted.

1. In a scalar sequential manner and using the key generation algorithm described above,

generate the sequence of N keys.

2. Using the appropriate memory mapping described above, load the N keys into the

memory system.

3. Begin timing.

4. Do, for i= to Imx

(a) Modify the sequence of keys by making the following two changes:

K -i

K,+ <-- (Bmax -i)












(b) Compute the rank of each key.

(c) Perform the partial verification test described above.

5. End timing.

6. Perform full verification test described previously.

Table 4-1: Parameter Values to Be Used for Benchmark

Parameter Class A Class B

N 223 225
Bmax 219 221
seed 314159265 314159265
Imax 10 10

















CHAPTER 5
PARALLEL PATTERNS USED TO IMPLEMENT KERNEL IS

In this chapter, we will explain how we used the parallel pattern language to design

the parallel program and implement it in an MPI and C environment.

5.1 Finding Concurrency

5.1.1 Getting Started

As advised by this pattern, we have to understand the problem we are trying to solve.

According to the specification of Kernel IS of NAS Parallel Benchmark, the range of the

keys to be sorted is [0,219) for class A and a range of [0,221) for class B. Because of this

range, bucket sort, radix sort, and counting sort can sort the keys in O(n) time.13 These

sequential sorting algorithms are good candidate algorithms to parallelize because of their

speeds. Because of following reasons, Counting Sort is selected as a target algorithm to

parallelize. According to the specification of the benchmark, what will be timed is the

time needed to find out the rank of each keys, and the actual permutation of keys occurs

after the timing. This means that it is important to minimize the time needed to find out

rank of each key. One interesting characteristic of the three candidate sorting algorithms

is following. The Radix Sort and Bucket Sort make permutations of keys to find out rank

of each key, in other words, the keys should be sorted to know the rank of every key. But,

the Counting Sort does not need to be sorted to find out the rank of every key. This

means that the Counting Sort takes less time in ranking every key than the others.

Because of this reason, the Counting Sort is selected. Another reason is the simplicity of

the Counting Sort algorithm.












The basic idea of the Counting Sort algorithm is to determine, for each element x, the

number of elements less than x. This information can be used to place element x directly

into its position in the output array and the number of elements less than x is rank(x) -1.

Counting Sort Algorithm is as follows:

Counting-Sort(A, B ,k) // A is an input array. B is an output array. C is an intermediate
//array. k is an biggest key in the input array.
for i <- 1 to k
do C[i] <- 0
for j 1 to le'gtl[.4]
do C[A[j]] C[A[j]] +1
/ C[i] now contains the number of elements equal to i
for i 2 to k
do C[i] C[i] + C[i 1]
/ C[j] now contains the number of elements less than or equal to i
for j <- leLgtl[.4] down to 1
do B[C[A[j]]] A[j]
C[A[j]] <- C[A[j]] -1

Algorithm 4-1 Counting Sort
5.1.2 Decomposition Strategy

The decision to be made in this pattern is whether or not to follow data decomposition

or task decomposition pattern. The key data structure of the Counting Sort algorithm is an

array of integer keys, and the most compute-intensive part of the algorithm is counting

each occurrence of same keys in the integer keys array and summing up the counts to

determine the rank of each key. The task-based decomposition is a good starting point if

it is natural to think about the problem in terms of a collection of independent (or nearly

independent) tasks. We can think of the problem in terms of tasks that count each

occurrence of the same keys in key array. So we followed the task decomposition pattern.












5.1.3 Task Decomposition

The Task Decomposition pattern suggests finding tasks in functions, loop, and updates

on data. We tried to find tasks in one of the four loops in the Counting Sort algorithm.

We found, using this pattern, that there are large enough independent iterations in the

secondfor loop of the counting sort algorithm. These iterations can be divided into many

enough relatively independent tasks which find out rank of each key. We can think that

each task concurrently counts each occurrence of the same keys on array A. The array A

is shared among tasks right now and it is not divided yet.

Because of dependency among tasks, this pattern recommends using Dependency

Analysis pattern.

5.1.4 Dependency Analysis

Using this pattern, data sharing dependencies are found on array A. Each task is

sharing the array A because the array A is accessed by each task to read a number in each

cell of the array. If one task has read and counted the integer number of a cell, that

number should not be read and counted again.

Array C is shared, too, because each task access this array to accumulate the count for

each number, and one cell must not be accessed or updated concurrently by two or more

tasks at the same time. We follow Data-Sharing Pattern since data sharing dependencies

are found.

5.1.5 Data Sharing Pattern

Array A can be considered read only data among the three categories of shared data

according to this pattern, because it is not updated. Array A can be partitioned into

subsets, each of which is accessed by only one of the tasks. Therefore Array A falls into

effectively-local category.












The shared data, array C, is one of the special cases of read/write data. It is an

accumulate case because this array is used to accumulate the counts of each occurrence

of the same keys. For this case, this pattern suggests that each task has a separate copy so

that the accumulations occur in these local copies. The local copies are then accumulated

into a single global copy as a final step at the end of the computation.

What has been done until this point is a parallelization of second loop of Counting

Sort algorithm. The fourth for loop can be parallelized as follows: Each task has its own

ranks of keys in its own data set so the keys can be sorted in each process. To sort these

locally sorted keys, redistribute keys after finding out the range of keys for each task so

that each task has approximately same amount of keys and the ranges of keys in

ascending order among tasks.

Then the tasks of sorting redistributed keys have a dependency on locally sorted keys.

But this dependency is read only and effectively local so the locally sorted keys can be

redistributed among processes without complication, according to the ranges, because

those are already sorted. Then each task can merge the sorted keys in its own range with

out dependency using the global count.

Therefore, the final design of parallel sorting (the implementation of Kernel IS)

follows. First, each task counts each occurrence of the same keys on its own subset data,

accumulates the results on its own output array, and then reduces them into one output

array. Second, each task redistributes the keys into processes, according to the range of

keys of each process, and merges the redistributed keys.

5.1.6 Design Evaluation

Our target platform is Ethernet-connected Unix workstations. It is a distributed

memory system. This Design Evaluation pattern says that it usually makes it easier to












keep all the processing elements busy by having many more tasks than processing

elements. But using more than one UE per PE is not recommended in this pattern when

the target system does not provide efficient support for multiple UEs per PEs (Processing

Element. Generic term used to reference a hardware element in a parallel computer that

executes a stream of instructions), and the Design can not make good use of multiple UEs

per PE, which is our case because of reduction of output array (if we have more local

output array than the number of PEs, then the time needed to reduce to one output will

take much longer). So our program adjusts the number of tasks into the same number of

workstations.

This pattern questions simplicity, flexibility, and efficiency of the design. Our design

is simple because each generated task will find out rank of each key on its own subset

data (keys) and then reduce them into global ranks. The next steps of sorting locally and

redistribution of keys and merging the redistributed local key arrays are also simple

because we already know the global ranks. It is also flexible and efficient because the

number of tasks is adjustable and the computational load is evenly balanced.

5.2 Algorithm Structure Design Space

5.2.1 Choose Structure

The Algorithm-Structure decision tree of this pattern is used to arrive at an appropriate

Algorithm-Structure for our problem.

The major organizing principle of our design is organization by tasks because we used

the loop-splitting technique of the Task-Decomposition pattern, so we arrived at the

Organized-by-Tasks branch point. At that branch point, we take a partitioning branch

because our ranking tasks can be gathered into a set linear in any number of dimensions.












There are dependencies between tasks and it is an associative accumulation into shared

data structures. Therefore we arrived at the Separable-Dependencies pattern.

5.2.2 Separable Dependencies

A set of tasks that will be executed concurrently in each processing unit corresponds

to iterations of a secondfor loop and thirdfor loop of Algorithm 4.1. Each task will be

independent of each other because each task will use its own data. This pattern

recommends balancing the load at each PE. The size of each data is equal because of data

distribution specification of Kernel IS so the load is balanced at each PE. The

specification of keys distribution of Kernel IS is also satisfied. The fact that all the tasks

are independent leads to the Embarrassingly Parallel Pattern.

5.2.3 Embarrassingly Parallel

Each task of finding all the ranks of distinct key array can be represented as a process.

Because each task will have almost same amount of keys, each task will have same

amount of computation. So the static scheduling of the tasks will be effective as advised

by this pattern. For the correctness considerations of this pattern, each tasks solve the

subproblem independently, solve each subproblem exactly once, correctly save

subsolutions, and correctly combine subsolutions.

5.3 Using Implementation Example

The design of the parallel integer sort problem satisfies the condition of simplest form

of parallel program as follows: All the tasks are independent. All the tasks must be

completed. Each task executes same algorithm on a distinct section of data. Therefore the

resulted design is a simplest form of Embarrassingly Parallel Pattern. The

Implementation example is provided in Chapter 3. Using the Implementation Example of

the Embarrassingly Parallel pattern, the Kernel IS of NAS Parallel Benchmark set can be












implemented. The basic idea of Implementation Example of Embarrassingly Parallel

pattern is that of scattering data to each process, compute subsolutions, and then combine

subsolutions into a solution for the problem. If we apply the Implementation Example

two times, one time for finding rank of each key and one for sorting the keys in local

processes, the Kernel IS can be implemented.

Another method of implementation is to use Implementation Example of Divide

Conquer pattern. But this Implementation Example is not chosen because it is hard to

measure the elapsed time for ranking all the keys.

5.4 Algorithm for Parallel Implementation of Kernel IS

The following algorithm illustrates the parallel design for implementation of Kernel IS

(parallel sort over small integer) that has been obtained through the parallel design

patterns.

1. Generate the sequence of N keys using key generation algorithm of NAS Parallel
Benchmarks.

2. Divide the keys by the number of PEs and distribute to each memory of PEs.

3. Begin timing.

4. Do, for i= to Ima

(a) Modify the sequence of keys by making the following two changes:

K<-i

K.im <- (Bmax -i)

(b) Each task (process) finds out the rank of each key in its own data set (keys)
using ranking algorithm of counting sort.

(c) Reduce the arrays of ranks in its own local data into an array of ranks in global
data.

(d) Do partial verification.









73


5. End timing.

6. Perform permutation of keys according to the ranks.

(a) Each task sorts local keys according to the ranks among its local keys.

(b) Compute the ranges of keys each process will have so that each process will
have nearly same amount of keys and the ranges of keys are in ascending order

(c) Redistribute keys among processes according to the range of each process.

(d) Each task (process) merges its redistributed key arrays.

7. Perform full verification.



















CHAPTER 6
PERFORMANCE RESULTS AND DISCUSSIONS

6.1 Performance Expectation

An ideal execution time for a parallel program can be expressed as T/N where T is the

total time taken with one processing element and N is the number of processing Elements

used. Our implementation of Kernel IS of NAS Parallel Benchmarks will not have an

ideal execution time because of overhead that comes from several sources. A source of

overhead comes from the computations needed to reduce local arrays of ranks into one

array of global ranks. The more local arrays of ranks and processing elements we use, the

more computation time will be needed. The gap between the ideal execution time and the

actual execution time will be increased. Another source of overhead will be the

communication because MPI uses message transfer, which typically involves both

overhead due to kernel calls and latency due to the time it takes the message to travel

over the network.

6.2 Performance Results

The Kernel IS implementation was executed on top of Ethernet-connected

workstations and LAM 6.3.2, which is an implementation of MPI.14 These workstations

are Sun Blade 100 with 500-MHz UltraSPARC-IIe cpu, 256-KB L2 External Cache, 256-

MB DIMM memory, Ethernet/Fast Ethernet, and twisted pair standard (10BASE-T and

100BASE-T) self-sensing network. Figure 6.1 shows the performance results for class A

and B. The rows show the total number of processing elements (workstations) used to












execute Kernel IS implementation. The columns show the execution time of Kernel IS

implementation and an ideal execution time in milliseconds for each class A and B. The

performance result of NPB2.2 (NAS Parallel Benchmark 2.2. These are MPI-based

source-code implementations written and distributed by NAS. They are intended to be

run with little or no tuning, and they approximate the performance a typical user can

expect to obtain for a portable parallel program. They supplement, rather than replace,

NPB 1. The NAS solicits performance results from all sources) for class A and B are

shown for comparison purpose.1

Table 6-1 Performance Results for Class A and B

Number Actual Ideal Execution Actual Ideal Execution
of Execution Execution Time of Execution Execution Time of
Processing Time for Time for NPB2.2 for Time for Time for NPB2.2
Elements class A Class A Class A Class B Class B for ClassB
1 30940 30940 20830 147220 147220 N/A
2 16500 15470 25720 76100 73610 1446270
3 12160 10313 N/A 55640 49073 N/A
4 10600 7735 16870 46500 36805 103110
5 9200 6188 N/A 41150 29444 N/A
6 8150 5156 N/A 36550 24536 N/A
7 7600 4420 N/A 33350 21031 N/A
8 7080 3867 9720 30200 18402 42760
9 7470 3437 N/A 31790 16357 N/A
10 7040 3094 N/A 30860 14722 N/A
11 6820 2812 N/A 28740 13383 N/A
12 6750 2578 N/A 27900 12268 N/A
13 6690 2380 N/A 27580 11324 N/A
14 6320 2210 N/A 26630 10515 N/A
15 6000 2062 N/A 25560 9814 N/A
16 5780 1933 16680 25230 9201 62080


The execution times for Class B when the number of processing elements is 1 and 2

are not shown in Figure 6.2 because execution time for Class B is too big to show in

Figure 6.2. The reason for that long execution time is that NPB2.2 consumes much more













memory than the physical memory, which leads to many I/O between the hard disk and

main memory of workstations.


60000

50000
-Actual Execution Time
40000
o -- Ideal Execution Time
c 30000

S20000 \A Execution Time of
NPB2.2
10000

0
1 3 5 7 9 11 13 15
Number of Processing Elements

Figure 6-1 Execution Time Comparison for Class A


160000
140000
120000 -*-Actual Execution
S100000 A Time
o -- Ideal Execution Time
o 80000
60000 A A Execution Time of
40000 A NPB2.2
20000
0
1 3 5 7 9 11 13 15
Number of Processing Elements

Figure 6-2 Execution Time Comparison for Class B

6.3 Discussions

As seen from the previous graphs, the following conclusions can be made about the

performance of the parallel implementation of Kernel IS of NAS Parallel Benchmarks.

The most increase in performance is achieved by using one or two additional processing









77


elements, that is when the total number of processing elements is 2 or 3 respectively and

there are smaller improvements by using more processing elements. The more processing

elements we use for computation, the more overhead was created and the gap between

the ideal execution time and the actual execution time increased, as we expected. The

overall performance is acceptable because it has better performance compared to the

performance of NPB2.2

















CHAPTER 7
RELATED WORK AND CONCLUSIONS AND FUTURE WORK

7.1 Related work

7.1.1 Aids for Parallel Programming

Considerable research has been done to ease the tasks of designing and implementing

efficient parallel program. The related work can be categorized into a program skeleton, a

program framework, and design patterns.

An algorithmic skeleton was introduced by M. Cole as a part of his proposed parallel

programming systems (language) for parallel machines.16 He presented four independent

algorithmic skeletons which are "fixed degree divide and conquer," iterativee

combination," "clustering," and "task queues." Each of skeletons describes the structure

of a particular style of algorithm in terms of abstraction. These skeletons capture very

high-level patterns and can be used as an overall program structure. The user of this

proposed parallel programming system must choose one of these skeletons to describe a

solution to a problem as an instance of the appropriate skeleton. Because of this

restriction, these skeletons can not be applied to every problem. These skeletons are

similar to the patterns in the algorithm structure design space of parallel pattern language in

a sense that both are an overall program structure and provide algorithmic frameworks.

But these skeletons provide less guidance for an inexperienced parallel programmer

about how to arrive at one of these skeletons in comparison with parallel pattern language.

Program Frameworks, which can address overall program structure but are more

detailed and domain specific, are widely used in many areas of computing.17 In a parallel












computing area, Parallel Object Oriented Methods and Applications (POOMA) and

Parallel Object-oriented Environment and Toolkit (POET) are examples of frameworks

from Los Alamos and Sandia, respectively.18-19

The POOMA is an object-oriented framework for a data-parallel programming of

scientific applications. It is a library of C++ classes designed to represent common

abstractions in these applications. Application programmers can use and derive from

these classes to express the fundamental scientific content and/or numerical methods of

their problem. The objects are layered to represent a data-parallel programming interface

at the highest abstraction layer whereas lower, implementation layers encapsulate

distribution and communication of data among processors.

The POET is a framework for encapsulating a set of user-written components. The

POET framework is an object model, implemented in C++. Each user-written component

must follow the POET template interface. The POET provides services, such as starting

the parallel application, running components in a specified order, and distributing data

among processors.

In recent years, Launay and Pazat developed a framework for parallel programming

using Java.20 This framework provides a parallel programming model embedded in Java

by a framework and intended to separate computations from control and synchronizations

between parallel tasks. Doug Lea provided design principles to build concurrent

applications using the Java parallel programming facilities.21

Design patterns that can address many levels of design problem have been widely used

with object oriented sequential programming. Recent work of Douglas C. Schmidt, et al












addresses issues associated with concurrency and networking but mostly at a fairly low

level.22

Design pattern, frameworks, and skeleton share the same intent of easing parallel

programming by providing libraries, classes, or a design pattern. In comparison with

parallel pattern language, which provides a systemic design patterns and implementation

patterns for parallel program design and implementation, frameworks and skeletons

might have more efficient implementations or better performance in their specialized

applicable problem domain. But parallel pattern language could be more helpful for

inexperienced parallel programmer in designing and solving more general application

problems because of its systematic structure in exploiting concurrency and providing

frameworks and libraries down to implementation level.

7.1.2 Parallel Sorting

Parallel sorting is one of the most widely studied problems because of its importance

in wide variety of practical applications. Various parallel sorting algorithms have also

been proposed in the literature. Guy E. Blelloch, et al. analyzed and evaluated many of

the parallel sorting algorithms proposed in the literature to implement as fast a general

purpose sorting algorithm as possible on the Connection Machine Supercomputer model

CM-2.23 After the evaluation and analysis, the researchers selected the three most

promising alternatives for implementation: bitonic sort that is a parallel merge sort,

parallel version of counting-based radix sort, and a theoretically efficient randomized

algorithm, sample sort. According to their experiments, sample sort was the fastest on

large data sets.

Andrea C. Dusseau, et al. analyzed these three parallel sorting algorithms and column

sort using a LogP model, which characterizes the performance of modern parallel












machines with a small set of parameters: the communication latency, overhead,

bandwidth, and the number of processes.24 They also compared performances of Split-C

implementation of the four sorting algorithms on message passing, distributed memory,

massively parallel machine, CM-5. In their comparison, radix and sample sort was faster

than others on a large data set.

To understand the performance of parallel sorting on hardware cache-coherent shared

address space (CC-SAS) multiprocessors, H. Shan, et al. investigated the performance of

two parallel sorting algorithms (radix, sample sort) under three major programming

models (a load-store CC-SAS, message passing, and the segmented SHMEM model) on a

64 processor SGI Origin2000 (A scalable, hardware-supported, cache-coherent, non-

uniform memory access machine).25 In their investigation, the researchers found that

sample sort is generally better than radix sort up to 64k integers per processor, and radix

sort is better after that point. The best combination of algorithm and programming models

are sample sort under the CC-SAS for smaller data sets and radix sort under SHMEM for

larger data sets, according to their investigation.

Communication is fundamentally required in a parallel sorting problem as well as

computation. Because of this characteristic and the importance of sorting in applications,

parallel sorting problem has been selected as one of kernel benchmarks for the

performance evaluation of various parallel computing environments.26 The Kernel IS of

NAS Parallel Benchmark set has been implemented and reported its performance on

various parallel supercomputers by its vendors.2728

7.2 Conclusions

This thesis shows how the parallel design patterns were used to develop and

implement a parallel algorithm for Kernel IS (Parallel sort over large integers) of NAS












Parallel Benchmarks as a case study. And it presented reusable frameworks and examples

for implementations of patterns in the algorithm structure design space of parallel pattern

language.

Chapter 6 showed the performance results for Kernel IS. As the result shows, the

parallel design patterns help developing a parallel program with relative ease, and it is

also helpful in reasonably improving the performance.

7.3 Future Work

Parallel pattern language is an ongoing project. Mapping design patterns with various

parallel computing environments and developing frameworks for object oriented

programming systems can be considered as future research. Testing the resulted

implementation of Kernel IS using parallel pattern language on various supercomputers,

comparing this algorithm as a full sorting, not just ranking, and more case studies of the

parallel application program using parallel pattern language can also be included in the

future work.


















APPENDIX A
KERNEL IS OF THE NAS PARALLEL BENCHMAKRK

A.1 Brief Statement of Problem

Sort N keys in parallel. The keys are generated by the sequential key generation

algorithm given below and initially must be uniformly distributed in memory. The initial

distribution of the keys can have a great impact on the performance of this benchmark,

and the required distribution is discussed in detail below.

A.2 Definitions

A sequence of keys, {K, I i = 0,1,..., N -1}, will be said to be sorted if it is arranged in

non-descending order, i.e., K,
is the index value I that the key would have if the sequence of keys were sorted. Ranking,

then, is the process of arriving at a rank for all the keys in a sequence. Sorting is the

process of permuting the keys in a sequence to produce a sorted sequence. If an initially

unsorted sequence, K0,K ,..., KN, has ranks r(0),r(1),...r(N- 1), the sequence becomes

sorted when it is rearranged in the order K,(O), K(1) ,..., K(N1) Sorting is said to be stable

if equal keys retain their original relative order. In other words, a sort is stable only if

r(i) < r(j) whenever K,(, = K,() and i < j. Stable sorting is not required for this

benchmark.

A.3 Memory Mapping

The benchmark requires ranking an unsorted sequence of N keys. The initial sequence

of keys will be generated in an unambiguous sequential manner as described below. This












sequence must be mapped into the memory of parallel processor in one of the following

ways depending on the type of memory system. In all cases, one key will map to one

word of memory. Word size must be no less than 32 bits. Once the keys are loaded onto

the memory system, they are not to be removed or modified except as required by the

procedure described in the Procedure subsection.

A.4 Shared Global Memory

All N keys initially must be stored in a contiguous address space. If A, is used to

denote the address of the ith word of memory, then the address space must be [A,, A,, ,].

The sequence of keys, K0, K1,..., K,, initially must map to this address space as

A,, <+-MEM(Kj) for j = 0,1,...,N-1 (A.1)

where MEM(K ) refers to the address of K .

A.5 Distributed Memory

In a distributed memory system with p distinct memory units, each memory unit

initially must store N, keys in a contiguous address space, where

N, =Nlp (A.2)

If A, is used to denote the address of the ith word in a memory unit, and if P is used

to denote the jh memory unit, then P, n A, will denote the address of the ith word in the


jt memory unit. Some initial addressing (or "ordering") of memory units must be

assumed and adhered to throughout the benchmark. Note that the addressing of the

memory units is left completely arbitrary. If N is not evenly divisible by p, then












memory units {P I j = 0,1,..., p 2} will store N, keys, and memory unit P 1 will store

N, keys, where now

Np =LN/ p+0.51

N, = N (p 1)Np

In some cases (in particular if p is large) this mapping may result in a poor initial load

balance with Np >> N In such cases it may be desirable to use p' memory units to

store the keys, where p'< p. This is allowed, but the storage of the keys still must follow

either equation 2.2 or equation 2.3 with p' replacing p. In the following we will

assume N is evenly divisible by p. The address space in an individual memory unit must

be [A,, A 1 If memory units are individually hierarchical, then Np keys must be

stored in a contiguous address space belonging to a single memory hierarchy and A, then

denotes the address of the ith word in that hierarchy. The keys cannot be distributed

among different memory hierarchies until after timing begins. The sequence of

keys, K0, K1,..., KN initially must map to this distributed memory as

Pk A,+j <- MEM(KkNj ) for j = 0,1,..., N -1 and k = 0,,..., p -1

where MEM(KkN j ) refers to the address of Kk, If N is not evenly divisible by p,

then the mapping given above must be modified for the case where k = p -1 as

Pp nAf +-MEM(K(p 1)N ) for j= ,1,...,N -1. (A.3)

A.6 Hierarchical Memory

All N keys initially must be stored in an address space belonging to a single memory

hierarchy which will here be referred to as the main memory. Note that any memory in












the hierarchy which can store all N keys may be used for the initial storage of the keys,

and the use of the term "main memory" in the description of this benchmark should not

be confused with the more general definition of this term in section 2.2.1. The keys

cannot be distributed among different memory hierarchies until after timing begins. The

mapping of the keys to the main memory must follow one of either the shared global

memory or the distributed memory mappings described above.

The benchmark requires computing the rank of each key in the sequence. The

mappings described above define the initial ordering of the keys. For shared global and

hierarchical memory systems, the same mapping must be applied to determine the correct

ranking. For the case of a distributed memory system, it is permissible for the mapping of

keys to memory at the end of the ranking to differ from the initial mapping only in the

following manner: The number of keys mapped to a memory units at the end of the

ranking may differ from the initial value, N It is expected, in a distributed memory

machine, that good load balancing of the problem will require changing the initial

mapping of the keys and for this reason a different mapping may be used at the end of the

ranking. If Np is the number of keys in memory unit Pk at the end of ranking, then the

mapping which must be used to determine the correct ranking is given by

Pk n +, MEM(r(kN + j)) for j = 0,1,...,N -1 and k = 0,1,...,p -1 where

r(kNp + j) refers to the rank of key KkNP + Note, however, this does not imply that the

keys, once loaded into memory, may be moved. Copies of the keys may be maid and

moved, but the original sequence must remain intact such that each time the ranking

process is repeated (Step 4 of Procedure) the original sequence of keys exist (except for

the two modification of Step 4a) and the same algorithm for ranking is applied.












Specifically, knowledge obtainable from the communications pattern carried out in the

first ranking cannot be used to speed up subsequent rankings and each iteration of Step 4

should be completely independent of the previous iteration.

A.7 Key Generation Algorithm

The algorithm for generating the keys makes use of the pseudorandom number

generator described in section 2.2. The keys will be in the range [0, Bmax). Let rf be a

random fraction uniformly distributed in the range [0,1], and let K, be the ith key. The

value of K, is determined as

K, <- Bmax(r4, + r4, + r4,2 + r4+3)/4J for i= 0,1,...,N-1. (A.4)

Note that K, must be an integer and L[j indicates truncation. Four consecutive

pseudorandom numbers from the pseudorandom number generator must be used for

generating each key. All operations before the truncation must be performed in 64-bit

double precision. The random number generator must be initialized with s = 314159265

as a starting seed.

A.8 Partial Verification Test

Partial verification is conducted for each ranking performed. Partial verification

consists of comparing a particular subset of ranks with the reference values. The subset of

ranks and the reference values are given in table 2.1.

Note that the subset of ranks is selected to be invariant to the ranking algorithm (recall

that stability is not required in the benchmark). This is accomplished by selecting for

verification only the ranks of unique keys. If a key is unique in the sequence (i.e., there is

no other equal key), then it will have a unique rank despite an unstable ranking algorithm.

The memory mapping described in the Memory Mapping subsection must be applied.












Table A-1 : Values to be used for partial verification

Rank (full) Full scale Rank (sample) Sample code
r(2112377) 104 + i r(48427) 0+i
r(662041) 17523 +i r(17148) 18+i
r(5336171) 123928 + i r(23627) 346 +i
r(3642833) 8288932 -i r(62548) 64917 -i
r(4250760) 8388264 -i r(4431) 65463 i

A.9 Full Verification Test

Full verification is conducted after the last ranking is performed. Full verification

requires the following:

1. Rearrange the sequence of keys, {K, I i = 0,1,...,N 1 in the order

{K, I j= r(0),r(),...,r(N 1), where r(0),r(1),...,r(N -1) is the last computed

sequence of ranks.

2. For every K, from i = O...N 2 test that K, < K, 1

If the result of this test is true, then the keys are in sorted order. The memory mapping

described in the Memory Mapping subsection must be applied.

A.10 Procedure

1. In a scalar sequential manner and using key generation algorithm described above,
generate the sequence of N keys.

2. Using the appropriate memory mapping described above, load the N keys into the
memory system.

3. Begin timing.

4. Do, for i= to Ima
(a) Modify the sequence of keys by making the following two changes:

K,-i

K+. <- (Bma -i)












(b) Compute the rank of each key.
(c) Perform the partial verification test described above.
5. End timing.

6. Perform full verification test described above.

Table A-2: Parameter values to be used for benchmark

Parameter Class A Class B

N 223 225
Bmax 219 221
seed 314159265 314159265
Imax 10 10

A.11 Specifications

The specifications given in table A.2 shall be used in the benchmark. Two sects of

values are given, one for Class A and one for Class B.