<%BANNER%>

A Distributed File System (DFS)

xml version 1.0 encoding UTF-8
REPORT xmlns http:www.fcla.edudlsmddaitss xmlns:xsi http:www.w3.org2001XMLSchema-instance xsi:schemaLocation http:www.fcla.edudlsmddaitssdaitssReport.xsd
INGEST IEID E20101119_AAAABA INGEST_TIME 2010-11-19T08:14:28Z PACKAGE UFE0014040_00001
AGREEMENT_INFO ACCOUNT UF PROJECT UFDC
FILES
FILE SIZE 20940 DFID F20101119_AAAOJS ORIGIN DEPOSITOR PATH jayaraman_p_Page_69.QC.jpg GLOBAL false PRESERVATION BIT MESSAGE_DIGEST ALGORITHM MD5
f9d52068595d9933d46376207a09a0b8
SHA-1
3304eca0fac1e8226e5f49198e0944cacf059973
460 F20101119_AAAOEV jayaraman_p_Page_55.txt
1991166e9bbe6a9edb30ddf2c8c5bc81
1b2a5b28b695771f3e129a35ecdbe3046b59c611
91222 F20101119_AAANXX jayaraman_p_Page_51.jp2
70e83a69b04240e3e113c9a622ff4a66
32a157569756a0bc93af1be12e699a95e9095880
59412 F20101119_AAANQC jayaraman_p_Page_18.jpg
9aa6008ddb96708f314ac264cd365225
749736e38cf5d6dc018afee0039f75783a9aa279
18781 F20101119_AAAOJT jayaraman_p_Page_70.QC.jpg
f38cd1ea324d7cf0e5ebf1ed4e28f7ee
1e167807c22e2f1ed3f819f7feb6416e8f2c48bc
356 F20101119_AAAOEW jayaraman_p_Page_56.txt
0b6a6879e962d10995b41403e2fa310e
8b53efaf10a192c2e548196c775c6bd5cb4bc416
90793 F20101119_AAANXY jayaraman_p_Page_52.jp2
6ddd5dfb9b8eedfef1ddf16c7311b357
3b4500f901c31ddd3db5ef61b333655ef39c3f2e
38242 F20101119_AAANQD jayaraman_p_Page_13.pro
8ad540fd65a7a6828f5ec08a905320c0
4ec2e3fee46bdf6bfd2b450f6c6398e808b2bf0d
60926 F20101119_AAANVA jayaraman_p_Page_23.jpg
317bb2ab8947a5086042e27972fd03d5
4fd5b2a6095e95be45fc8da3e39174aa3e0835ce
10162 F20101119_AAAOJU jayaraman_p_Page_72.QC.jpg
72552c256f82e1259b11aa133f5a0f19
08ed53cae17b3d51016ef1a33387a492c66aaf3c
206 F20101119_AAAOEX jayaraman_p_Page_57.txt
2f93ca03f9543f64d49035dced304818
90c835ffea69505c9ba40d702c4065f6d61ad0e3
95134 F20101119_AAANXZ jayaraman_p_Page_54.jp2
a7058ea29e8eb09e0b67f5904c28a1d8
c1024aafc4d13162415c9ec4ea838bf553130666
60201 F20101119_AAANQE jayaraman_p_Page_79.pro
b5d0ccbca081979488f091d0e5646801
f2d0930bbb58a09db5a4db45ed0660af59bd822b
65491 F20101119_AAANVB jayaraman_p_Page_25.jpg
3637c36f2e8e99be643e8caff07f90c4
01fddc14710e6d53df05a96234f16a40d59f2367
3496 F20101119_AAAOJV jayaraman_p_Page_72thm.jpg
b4c194445e3208b3347536c261c13e4e
03d3bbe7009b4c1df6e98e1300854298f62a1058
832 F20101119_AAAOEY jayaraman_p_Page_58.txt
c41a0461bffec3519e5078053785235a
456e41dfa84350e9166dedbe2afd190649eff5e2
102158 F20101119_AAANQF jayaraman_p_Page_46.jp2
ed3fdf29164e3583980668731d59d9cb
fcc7857c28ce0eebb31876f4c12aebdf588a1c96
63470 F20101119_AAANVC jayaraman_p_Page_26.jpg
fe27168b11388119b11167cf494d5fe8
51c50e8914aac19cb49d9154cc244d9919ad98be
10317 F20101119_AAAOJW jayaraman_p_Page_77.QC.jpg
8045774c8504b37129914ebc25feac9e
0530e7daa68c10ba63c1b9c93f7603ddcfb626a8
1534 F20101119_AAAOEZ jayaraman_p_Page_59.txt
590b1254e37d982b4afc4405da6ddbf9
7a24627c22018ef19cc47cc48c08b85cf588faf1
1053954 F20101119_AAANQG jayaraman_p_Page_34.tif
c2becfc382ab4bd2745aa59777998d18
c9e42d8138c7c21216c694ad52b3eb83e0e87589
13325 F20101119_AAAOCA jayaraman_p_Page_31.pro
638d281c72817170bf654a3e728e10c7
ea069bd7846d47dfd6c86236bcc747e04363dbc7
66801 F20101119_AAANVD jayaraman_p_Page_28.jpg
cb6b4db305a6cf6477f41bb2f2e3f121
7486d849409e93ddb0dec48f16abcfa6c723fa3d
3251 F20101119_AAAOJX jayaraman_p_Page_77thm.jpg
5a48415a88686980d6ecdd7f9f4aa8eb
1ad19083a58ae2536e61c567965772e6554cd98e
40170 F20101119_AAANQH jayaraman_p_Page_70.pro
8cf7df6dea3c67e3f5f359fa1ffdf538
3b4152764de42a03b250c7e520656597b5b1cdc9
38506 F20101119_AAAOCB jayaraman_p_Page_32.pro
7bfc1d34cf3580e0b59ca5794931dd84
c1f73a817ee65dc5f0e4bd1b00e452dc28c1a917
69353 F20101119_AAANVE jayaraman_p_Page_29.jpg
188bee067405b68a085aa44fbcff1261
03712ce0f1597c356e63540cc59289587b5eab5f
25821 F20101119_AAAOJY jayaraman_p_Page_79.QC.jpg
589e634c886be552dd2deac5a7a88667
4a8938abec8bfaba27cb72faebf954f00f40adcd
5949 F20101119_AAAOHA jayaraman_p_Page_60thm.jpg
d7d6cf09381b406636a2ee64beee667f
0cce36199c3d8a6cedc2054235f6d74f859fa8b3
29060 F20101119_AAANQI jayaraman_p_Page_08.jpg
8d4a0fbe222aeccb0bdf4661a22f757d
45e8f0883188f2b132464611c0eaf7b4a7f114fb
40298 F20101119_AAAOCC jayaraman_p_Page_33.pro
d664d131cce4b7f85e6bb55451be44c9
0194175dba890a43f08c61fb53e0f4aa05622f15
31648 F20101119_AAANVF jayaraman_p_Page_31.jpg
1744cea08e320a5efb07f6ba0c7e1ba7
9de25fb7dc9b37dfc57103d3c1fec6a630aba2ca
12713 F20101119_AAAOJZ jayaraman_p_Page_80.QC.jpg
21b11d78f2ae6a955b8ff6dad621af10
0cb41630b8693998bc825284f5ef88bc0d24b38b
5975 F20101119_AAAOHB jayaraman_p_Page_66thm.jpg
d70d7124a940f3ce70c15a1134061183
5d992e420e0b0c028bdb836548311da9d87b9fdd
5874 F20101119_AAANQJ jayaraman_p_Page_11thm.jpg
d1a031b95419e6a2d9215e1690cc4773
d045a7a470c6846e332b91b5a2900d691b1945ea
41191 F20101119_AAAOCD jayaraman_p_Page_34.pro
5f67844e55035f70c9912e4c5cf9ca9b
78c41901631371db6b290558a3954e85c3b81772
59267 F20101119_AAANVG jayaraman_p_Page_33.jpg
1bc7e1ad6f377776bf1831bb0c0e9015
8ab3823da65f416a40402828c96c4654fe45ceef
4029 F20101119_AAAOHC jayaraman_p_Page_71thm.jpg
89d87787c9cdb79b662b97f228253203
3dac3b7500f4ee358fb7f951b1e8c079183491df
1984 F20101119_AAANQK jayaraman_p_Page_29.txt
46d1f43e4feb7263f2538bbe33e84bc1
7d6bb1ab7926634a49ea3ba9c4b092e8e675035c
44721 F20101119_AAAOCE jayaraman_p_Page_35.pro
16d7d55d9f0eeb74aecdf34222da4a98
21a5839dd53c29af1aa8fec7bb3c6317c96b4d15
61371 F20101119_AAANVH jayaraman_p_Page_34.jpg
8a3b7e2dae6cdadc1a27b3c936a9f954
661d167b06cfeb314e880813f597b01d8999fcf4
20797 F20101119_AAAOHD jayaraman_p_Page_66.QC.jpg
11ca651b44cb66cc5c6e370517df525e
795ca84cab4f09514b28a20c0d2c48bea32c2834
6234 F20101119_AAANQL jayaraman_p_Page_75thm.jpg
25ddbc5c478cde243c5e5476453076ad
11f5252dfcd0974106df53e1f7deb9a7223619a1
50492 F20101119_AAAOCF jayaraman_p_Page_36.pro
e90116caba9754ea6365667686d745da
3e33577413c622130c6f5e19ad81d91e4fc61ae7
65079 F20101119_AAANVI jayaraman_p_Page_35.jpg
c1fe5492da689a9606254b826b17cfb2
f8058fb28d96cc7488d0e7e49c7d3072ed1fe0c2
6209 F20101119_AAAOHE jayaraman_p_Page_54thm.jpg
0f6a43db0a6083be33cbddd1c80d377f
a7bc4c8340f33b488fd48e1d156a2b49b4c269bf
44672 F20101119_AAANQM jayaraman_p_Page_80.jpg
df998ba2f331d061294b784cb46ebc8a
d4d6760877b2cef5071566fe787789d29c3f0390
47691 F20101119_AAAOCG jayaraman_p_Page_37.pro
cce68981e63605b01b20ce19789e004c
2ab38f6a2b7470e77de0aa1a1ec92ca0b74a1846
71267 F20101119_AAANVJ jayaraman_p_Page_36.jpg
37be1a73cb29516a15db85d2e863527f
813016f0c635efd7738d6a803fa9a6da8e50bb84
24179 F20101119_AAANQN jayaraman_p_Page_16.QC.jpg
b6e9330c5168da2351707c6e51b49fee
bb9b937ab0584a5b4d7bc648b8fa865f72dc4dd5
49146 F20101119_AAAOCH jayaraman_p_Page_39.pro
f6c7da19211579ef1be35b9b98cf1af3
bd4419f43009b563b9e4b821e4fa5eb8edef0909
66456 F20101119_AAANVK jayaraman_p_Page_37.jpg
917ea02850a65a602841865fd5a061f2
0b57ef54351e3d3a7806d97c6da2813bd6fb68cd
5593 F20101119_AAAOHF jayaraman_p_Page_15thm.jpg
6e82e97a9bb5fd77e5243afb18c106f0
08a76830dd757f364f12ecf3a8a6cdfe20344474
6023 F20101119_AAANQO jayaraman_p_Page_67thm.jpg
1ab647247c9c780a86710a37bd53af12
7b7647497a0a50aea0f7179dc5cf2fcb8f1ffdf6
29667 F20101119_AAAOCI jayaraman_p_Page_41.pro
c6565bc247926b04983a2d0628a51b84
ce2e9455ef6a20783b17ed30723acd9f98b039e2
65130 F20101119_AAANVL jayaraman_p_Page_38.jpg
a4ccd8be1f79714a3e1317a6d5875b20
3c495df54abf6a9103763d42e89529f98597b5b0
6376 F20101119_AAAOHG jayaraman_p_Page_39thm.jpg
5036c0cbeff06a6b70563926e5b67b62
74f8e7ae0d53b36f99436136e6977fce143f0310
48262 F20101119_AAAOCJ jayaraman_p_Page_42.pro
1dc3ed06e4a9ed28bda3a2ed0ccc94f2
7549f0e5915d1c282a62c2909b9f1a2cb44e3530
43501 F20101119_AAANVM jayaraman_p_Page_41.jpg
864cbb524ff384bc7e41cd2893343f30
8cc269eb7746ac838a2db9bb2d5b046f6ae82ac1
6432 F20101119_AAAOHH jayaraman_p_Page_37thm.jpg
a49ccf662db3e77b62adc4d0eb1c2e9e
5241acba6cd1912dd765203223f20ffb19a998bb
7153 F20101119_AAANQP jayaraman_p_Page_55.QC.jpg
e9c196cacf6dbcad70206add7e2aebc7
d32092224414cc3982f6f0bcc6981c530d66c6b7
44944 F20101119_AAAOCK jayaraman_p_Page_43.pro
e9da046808b233a975fd20c660b5cc1c
75dd76a45830fa0cdf9e8407be2e82a968560819
67916 F20101119_AAANVN jayaraman_p_Page_42.jpg
c5c55247e771a10bbacb27a91101681c
6cdba603274a5e03ded6ce136567c206afb8440a
124007 F20101119_AAAOHI UFE0014040_00001.xml FULL
c02937a392144d9c085ad2c1dd3e4958
f50499682afef50108fee2d92272c6e2fc2fad6d
87730 F20101119_AAANQQ jayaraman_p_Page_15.jp2
c48782f04a3ac31ec4e22dd4a3b846ae
f92ecb14fe1f96b01e85e8678928175d4118bd94
41611 F20101119_AAAOCL jayaraman_p_Page_44.pro
0bb300f24fe13edbb115fac591421197
8d14d046b9f22398add81ef5607bcdf42fbf59bc
58923 F20101119_AAANVO jayaraman_p_Page_44.jpg
9dcdc94f3a246cc88e5e3a08de106e19
92fe260f49c97c8dcc0015a1ef36d55d9f509175
4531 F20101119_AAAOHJ jayaraman_p_Page_05thm.jpg
f81d04584d597b6715af4b07266679db
809c6436878177f34c4ff6fc39407973c409f91f
49231 F20101119_AAANQR jayaraman_p_Page_40.pro
df112a3683a08a677d253633623e4f0b
045bb60d0a09f6511e29d7685549969ef41b0e08
46357 F20101119_AAAOCM jayaraman_p_Page_45.pro
7baa5b2b94ddb98efe770730300b2b29
a212b73db8ccc8c487a2c311ba540db621ab6f54
66091 F20101119_AAANVP jayaraman_p_Page_47.jpg
39ea3ff85a7ad3bfdbbc72bca0d352ce
869bed23452d824627a0d9de94628cd11a45831a
18803 F20101119_AAAOHK jayaraman_p_Page_06.QC.jpg
0b263c1b6b004847961492586e3608ca
083589bb9e748f22da87caa5b257a5eab3e7ed89
117173 F20101119_AAANQS jayaraman_p_Page_76.jp2
292adf75f0119b1240e11c30563c46ee
298bc74e007bd2b65dd08bafaff7e797b07c0ba2
49657 F20101119_AAAOCN jayaraman_p_Page_46.pro
c464a816ee1869696477c4c8b76d464f
990c39a0bbc3c79b99ef73acc7621f143b93a090
66438 F20101119_AAANVQ jayaraman_p_Page_48.jpg
aff7202d31d08c3870440e2b6687b9e4
3b30f74414c6cd97477605e1ea57bb35d3e53c5d
4465 F20101119_AAAOHL jayaraman_p_Page_07thm.jpg
eb3f2ddc9977124ad70905ec99a3c7d8
9020b04c865abc23ca81b3368497df6af11b616a
20260 F20101119_AAANQT jayaraman_p_Page_51.QC.jpg
19df1270bf60579d88dbe8d3e93fbaf1
75167100c3d5ed236862bd9fbd614d96efbe434d
48287 F20101119_AAAOCO jayaraman_p_Page_47.pro
85ef6e1c5d17eecacb3edb6467896d7e
f552f84f76f8a28679a5e06239864983b88600fc
65693 F20101119_AAANVR jayaraman_p_Page_49.jpg
61f28a90724c23887788a4d3d00d9e92
f7b858bfd14237e0887e7699d0eb19b347221edc
8628 F20101119_AAAOHM jayaraman_p_Page_08.QC.jpg
188655e4ab68346417c18c9b22ac679c
5f5784eef9e4bda60ffedb2a1a8a88035cfff93b
F20101119_AAANQU jayaraman_p_Page_27.tif
7585c066e5bf7662d33fb67ec4cb7d2d
2627365840eb333a3ba092fa2ea2b079f3189182
47942 F20101119_AAAOCP jayaraman_p_Page_48.pro
45e089e0d69d8878968bfae7953ee048
a57bf7054b991a22a46db5bed6d9281161262f77
61040 F20101119_AAANVS jayaraman_p_Page_50.jpg
6488ef47f439f8c3a8b5a187ae413de2
4eddc71be55d338bc6a2b086d04b59d6a0e9ac84
2817 F20101119_AAAOHN jayaraman_p_Page_08thm.jpg
5c5f9d32c88c2db5a40dd69d068516cd
aa4956101294b3d8b18b9a910385b4d7340efd74
1829 F20101119_AAANQV jayaraman_p_Page_60.txt
d6b0aace8b40cfca57f968a759cb3565
acff6ef4837332691a79c0df8f222483f1519aff
47409 F20101119_AAAOCQ jayaraman_p_Page_49.pro
117bebcde4a650cf8f8fe9b77a082d4f
64dfefde2a94ece1e14e46fc2b0d54a46060175d
61111 F20101119_AAANVT jayaraman_p_Page_51.jpg
97c451e8253db1744725d1c759beac89
05f3d36949e68b4dbb0f1c70f8116d87634d7704
17624 F20101119_AAAOHO jayaraman_p_Page_09.QC.jpg
89053759961c489be5a990afe7bb9f9a
56f90ec523496b8f7ca74a72db07700e13d44c67
46556 F20101119_AAANQW jayaraman_p_Page_38.pro
e3f8473be47dd6919841f13b1923f947
3d4a5d1e866ae6efb5858655e9a0fb0354948439
44251 F20101119_AAAOCR jayaraman_p_Page_50.pro
0ca3b4d5e15179a5beb8d79cdf48443c
c7fac6f8aaceab834305906241805e0f8035b9c1
5145 F20101119_AAAOHP jayaraman_p_Page_09thm.jpg
6e88e4829dff88ecd8d86da72ad01e2b
2be8e11b2a2154a994a2e1f60775fb622c57746c
46950 F20101119_AAANQX jayaraman_p_Page_17.jpg
1d59c1d584211483ff6794727e28b47d
d91f8a59a195938129e23c6cbc1e37c10b90f1db
43705 F20101119_AAAOCS jayaraman_p_Page_51.pro
2c969e9424e2bc58afcd7c6321287c8e
ed7464a542d75b775beb62df58a352d8b31ceb32
61065 F20101119_AAANVU jayaraman_p_Page_52.jpg
6c3fd8848094346709860b35c4de365b
a3691d5b0903e0e2700ab3951f2b60385bf5b19e
18732 F20101119_AAAOHQ jayaraman_p_Page_15.QC.jpg
b00096c2815ca2d17e223fa672c5293d
14a8fa16d981c17c489b32b78f3774b64e6d7525
43836 F20101119_AAANQY jayaraman_p_Page_69.pro
32822c8872fd15707e701f07b23c7173
46d0c86c4f797ba126ab056b7a9918efd9dff3dc
47703 F20101119_AAAOCT jayaraman_p_Page_53.pro
ae8f908b04ad4d41ccd1fbf34f96ab73
904d8b25739937a3829a1f59d5861f13c5dd358d
66514 F20101119_AAANVV jayaraman_p_Page_53.jpg
9a4378af832745da0df4a61533f02518
814f339aa928db892d7f3e37356f701de0f4dea3
3695 F20101119_AAAOHR jayaraman_p_Page_17thm.jpg
c519f891e5ba26d21a8bedfee29076c6
9b171df95ed0837fb675ec8a3358c8c1a0a372d7
80443 F20101119_AAANQZ jayaraman_p_Page_76.jpg
adf87b282e0d1c8d145ae4c5c16ea104
b8048a3fcb2199cc35e120e6fad0bab148d39eb5
9399 F20101119_AAAOCU jayaraman_p_Page_55.pro
858171d52b8ada05a8ce5bcd439e6aef
6ed33c962673c4369490619a9a267bfcc58a9b37
23832 F20101119_AAANVW jayaraman_p_Page_56.jpg
d9a6ea704a9ad0922f789a3abe38c74e
410c2218d5513b7daaf7aae8f462702f48c43bee
6473 F20101119_AAAOHS jayaraman_p_Page_19thm.jpg
ae7de60490dad59a5bcf3c345f1ba5ea
a0e704125fecb963f0d327ee016d93d6e96b4c90
7722 F20101119_AAAOCV jayaraman_p_Page_56.pro
36a35c50463a7a606aed789dd9114dc4
af3f9529c0ed430d5a07092d3ccf29b7f2c5bd28
31140 F20101119_AAANVX jayaraman_p_Page_57.jpg
a4b62d95eb45756c991b67f13d7fcfc8
50548b9b67f73a59ae843e3eb1f805ff71763fb1
42179 F20101119_AAANOC jayaraman_p_Page_24.pro
28d7402bea81713eab7418068895e545
95856275ad547f4cf8a2b54ff2ab1a612659f3a6
19993 F20101119_AAAOHT jayaraman_p_Page_20.QC.jpg
fd46f393775431bbc26ac8da76d34351
98012ec1a7e66b44ccc7ebdd66c75395b2470bcd
3862 F20101119_AAAOCW jayaraman_p_Page_57.pro
23caa38612af418b4142f7e579b62405
b1c9f78f79564f5db7e7a3d59efaf1b8ea67888e
21386 F20101119_AAANTA jayaraman_p_Page_64.QC.jpg
26b2c681c3eb898fca3a4754dc0006da
47a01c09f272416b87cbb180274449d234f44d8b
34257 F20101119_AAANVY jayaraman_p_Page_58.jpg
b9e42d0d65c59ddbe2341c94e1de7231
a4464865e9ca0e304e6e51aad38dc742a57da4d1
F20101119_AAANOD jayaraman_p_Page_43.tif
c7170e7ab4d56b095a78df5f27dc4e56
a095cd8f72d86660e8547d26f963d2663c46fc52
6175 F20101119_AAAOHU jayaraman_p_Page_21thm.jpg
03570dce6caf6fb15da036378175121b
c6fc711a0b4db27a4ef9580525a98d94b98bc260
35474 F20101119_AAAOCX jayaraman_p_Page_59.pro
83ff5bbc77eff632ebb24cbaee49e7ee
157c25c7bf2884a476a9cf88698b08763972c4dc
16991 F20101119_AAANTB jayaraman_p_Page_05.QC.jpg
fc71a52e7d153fc65ca71fbbd51b63c2
2d2ce5821b72262d8aefca93bb7d861a0e370fd0
55825 F20101119_AAANVZ jayaraman_p_Page_59.jpg
a9f9ee08d123ff9f254795986478f8d4
50c3010e7aecf2d6dcdf80ec64f918249d8536d4
6830 F20101119_AAANOE jayaraman_p_Page_01.pro
a05bb234fa447ca5924b03dee7d711b1
34c84c02d41668bdf3e7016955efb48112a9011e
20711 F20101119_AAAOHV jayaraman_p_Page_22.QC.jpg
7a8fda2ffc2445d20a9a07ac0b54d241
f6a4082781cd48df5faeef0f85cad1a59889cda8
31663 F20101119_AAAOCY jayaraman_p_Page_61.pro
635cd3b89b45a38fc79f8b01aaae13f9
952e30ca22413ed25cb39d95423de274bfc64be5
32245 F20101119_AAANTC jayaraman_p_Page_57.jp2
afe511d1679ac728a9710fa7f7ae687c
5d95ce675aec204f63bf9a13a6ebfd3f1b66f379
84998 F20101119_AAANOF jayaraman_p_Page_32.jp2
404138a0298312efae27da2ea30a5b8c
ef257cb3525f48c29ef5736c5cdfafb5ff95aa07
6117 F20101119_AAAOHW jayaraman_p_Page_22thm.jpg
757e7709178379cc900c51121a4ffa7b
d0fb741c2074e187843728097f48f8da3a480098
F20101119_AAAOAA jayaraman_p_Page_48.tif
862a6cecf00444e3cc119e6f1fa7c55f
bad8f39caa44480cdfe7da1fdb7ef8f5523eb7c2
44073 F20101119_AAAOCZ jayaraman_p_Page_64.pro
859b8836053601e6c9882f1c8b79ceef
218abe250f53baa3725d109271a1180d0c81cc0c
6162 F20101119_AAANTD jayaraman_p_Page_38thm.jpg
211d99d9881a984b7b6a6a4c7d36f04d
639e6f9ca0a6f873cff71f91c984ae92a74ee40d
88893 F20101119_AAANOG jayaraman_p_Page_10.jp2
f450c68bd59dd93b9f868cec29f9b25b
7dbce3efd6f71abba960049d6fd5bb0b5034911b
22876 F20101119_AAANYA jayaraman_p_Page_55.jp2
38221c3e27bbf84933700b324cbe6a8e
10d3add6d0f8c9527767c3fcea94a7b33fe17db9
5835 F20101119_AAAOHX jayaraman_p_Page_23thm.jpg
7fe545cba31775eb2d2bf27cbe328b9e
9844c880866a9f2d4f314d1e60df641066e8e012
F20101119_AAAOAB jayaraman_p_Page_49.tif
cd76b2d1e83b836df70f85dc0938398b
ca885ea6ef7c360ba544a2386c01dd777ab2b3d6
62408 F20101119_AAANTE jayaraman_p_Page_27.jpg
efa5cc6e29e791aec1a5a0738937b252
bcb2ae967fdec180345321843395f21a5786ee05
21572 F20101119_AAANOH jayaraman_p_Page_37.QC.jpg
05d5853b4b257a58c11bdf21dd51224a
6e26845737af41da1cfb5783bafde2c70fd25289
79757 F20101119_AAANYB jayaraman_p_Page_59.jp2
cc135f656e67e6c0d44b7a80e4b8accb
797b97ad569d374635c537d1250f4ccc3efd43de
5908 F20101119_AAAOHY jayaraman_p_Page_24thm.jpg
3918b22aa1f7419cadd40264f6017c23
cbfcc9f8d5bf8fafd59916cada16a69127b1c618
1943 F20101119_AAAOFA jayaraman_p_Page_62.txt
732a9b7e0bc4689102e0b7d8b0b2e994
1341d162fc997f3604455458e54e896e8eb52fdf
F20101119_AAAOAC jayaraman_p_Page_50.tif
dd7129f0329ff43578a2b647e7860913
2fb91e7ddaeb2bc377a38380000b88b4cd17d9f4
46682 F20101119_AAANTF jayaraman_p_Page_63.pro
4ff566e97d847185d2b393650d5e9e1f
b095db6f13ba940e064e04eda4c80181b9030f33
46488 F20101119_AAANOI jayaraman_p_Page_54.pro
3b3db0144b8df3dff6cbd86bf0fe3a7c
d0a58252e06155f4134e3ceefe34a785ef3bcd30
99645 F20101119_AAANYC jayaraman_p_Page_60.jp2
ece82de5e5cab8e931bd2496010404a4
fdafd5ca11606ec11d5dd1d1802dfdcbca59c619
21521 F20101119_AAAOHZ jayaraman_p_Page_26.QC.jpg
68e78c63b4e182d717732874becabad6
4bb261762e82e8425fa4de21abbd0832019d9e30
1934 F20101119_AAAOFB jayaraman_p_Page_63.txt
c17ac53d14996c1c7cb96e9297737542
89f49cf61a8cb1d324d186ecc04d73f42a37741e
22589 F20101119_AAANTG jayaraman_p_Page_46.QC.jpg
15ef3699743ebf0fdc66458f0dce1ef5
0c647f93439b22733e45caa34ba1fc44bab93479
6164 F20101119_AAANOJ jayaraman_p_Page_69thm.jpg
42ea0809af268b057b210fdfced83af2
f1ab231067c455952208a7a020861d80f2a43f6a
66552 F20101119_AAANYD jayaraman_p_Page_61.jp2
4318674f1e0dae9f3b13b1c3ad66096f
c1d0bfc4b8c12c43aeee6c5142a7244be2ecb13a
F20101119_AAAOAD jayaraman_p_Page_51.tif
3d1d84631d77bd5071a0223fd00d0690
c60add6b9ae6834cb4c473e5aa741c1f98014d85
1676 F20101119_AAAOFC jayaraman_p_Page_65.txt
8e7cf9f2e5010e588096321d74d65104
0b37da7105f7ea9d298116d77c3ae9a7ba2cfff7
98211 F20101119_AAANTH jayaraman_p_Page_68.jp2
c4f7ba919ed92ae9c2087ac86139e6b4
3e7ad1a7c8b37005afd83e16fdc64f305614e327
F20101119_AAANOK jayaraman_p_Page_21.tif
2469d8f4c5172d637727fbab51ed5db9
baf35fe73e5424dba526f08deeb91c1acd60ca89
99898 F20101119_AAANYE jayaraman_p_Page_62.jp2
d87adbf65f06e6567ea282f845390a9e
09d4baa02eaf423e184ba06182975ee456cd4257
F20101119_AAAOAE jayaraman_p_Page_52.tif
22b9d65eadc091adf55507371cf41fe6
de67a6e55c9e78696ddb0a367dfc7fa1a842a6f0
3543 F20101119_AAAOKA jayaraman_p_Page_80thm.jpg
b910481822932265fcc329693be089a4
5f822bf07023ebabacd08f6a8b1e0eaf592b9d4c
19301 F20101119_AAANTI jayaraman_p_Page_10.QC.jpg
d84ec669a9fda088b599dd0b56d68fcd
86ed6b4eb5ab05e3b5de61b58b492edbc048ec41
1386 F20101119_AAANOL jayaraman_p_Page_02thm.jpg
92ad4e527f0b66c64d39641488a63aeb
c5be2f88433bfa87435dd10ded917fe9faa78749
100182 F20101119_AAANYF jayaraman_p_Page_63.jp2
6664bca29c8297919a7ef7c74106321e
f385f8bcfd3c192f50711b0e442756ead5e63b2c
F20101119_AAAOAF jayaraman_p_Page_54.tif
bc57990a3c2180eb991762d0a1cd88ac
7b2b08821ef85747dc34385256f9a9475a50acf3
2718 F20101119_AAAOKB jayaraman_p_Page_81thm.jpg
1e859c9b16063410c7a488c1b9b0ce1f
ff41a6b9faf536cca88897ea658e2bdea3f8a22b
1850 F20101119_AAAOFD jayaraman_p_Page_67.txt
6d1c20caae5181b11c39e52548031001
bffa83457a387a78a72e0fa730b4d28c0f28d8b9
F20101119_AAANTJ jayaraman_p_Page_32.tif
64e7b904c9a5ff364d5c7a7c92bacb30
e1b999b54c9430b20ca8bf97df04f06de1fb0eaa
20513 F20101119_AAANOM jayaraman_p_Page_27.QC.jpg
232834ef791a1f12a87f1455d80627db
429321d2f0a2db1fbc5b839d5694bc1a397b6923
95566 F20101119_AAANYG jayaraman_p_Page_64.jp2
a74d55535b4dff7fbd13c0b65e69b1c6
517b3f15e204e9d9b88087c76d98e7ca28dcd7fb
F20101119_AAAOAG jayaraman_p_Page_56.tif
d3e598f36e6627cd83717bd2af569b03
5561e7cea24cd9b16a3adc2fe28588cad1f61360
2007 F20101119_AAAOFE jayaraman_p_Page_68.txt
fd70ebea0b6cd5f14e56b3d97df82a81
7872f31f9b2164f8aa240a322ae841f78635ba26
F20101119_AAANTK jayaraman_p_Page_47.tif
cbfd0cd028f519e51d32725ebd9b8752
1b78c7def217096be93cb7c48d619b84434b6349
94091 F20101119_AAANYH jayaraman_p_Page_66.jp2
31f9c3344e38ca04240a1772567a6600
92d023653fa5c18ead9ef3b7dda9d0d8365b09fe
F20101119_AAAOAH jayaraman_p_Page_59.tif
1b2dba7f040d49b9ff38ab9e19055a91
8642d9504b8110227268d00ff0634c112ae71c87
F20101119_AAAOFF jayaraman_p_Page_70.txt
551d1ee8de2d52ea50fa25e9677b9c0b
d90d40c276a3cbfd30a3e7c1bd171aa77f5a6d6a
4992 F20101119_AAANTL jayaraman_p_Page_74thm.jpg
59b75c414a06351644b81e04ebe7dc2a
2a437e170cfa82b4afc11f2b2e28ed34035812af
40127 F20101119_AAANON jayaraman_p_Page_11.pro
53b69f4a819b766a167a00cab61d5adb
0c8a41047dbf65bd9b43a7b17cf1303142c26f24
93837 F20101119_AAANYI jayaraman_p_Page_67.jp2
f07aaf3dbd1fafaa54b934a6c5e93017
2be2376d374420fbb27e83d66b1b29647158e1fc
F20101119_AAAOAI jayaraman_p_Page_60.tif
bc16519025bf0117467e614238f27166
1b3ef9997edc3da1bedea813c3f3861ec1f90c2a
1074 F20101119_AAAOFG jayaraman_p_Page_71.txt
e89b5f609d6c25a9bfdb96f663f55155
78a5ca80810a0f80f2d8b81e58a8db1f6b7d2897
93090 F20101119_AAANTM jayaraman_p_Page_23.jp2
8cb0da124848c3d6b5fce75f108213d2
e27221426e0cc244e2e6edb6f3e7457201ca369d
2712 F20101119_AAANOO jayaraman_p_Page_12thm.jpg
17446e2e5c2c85567d4ccb78b28f470d
f9e393a0d3d0e58a57ca26b70ae052fcec344798
92117 F20101119_AAANYJ jayaraman_p_Page_69.jp2
acf1456b17cfc370c8b1c6c38971e60a
7caf1a7b1b675092ceba8a4c6709932ca602182f
F20101119_AAAOAJ jayaraman_p_Page_61.tif
ca0fa2f67ba63614a456a4df9544f662
6057909b737f340a7d465d806f54d738daf6161f
108 F20101119_AAAOFH jayaraman_p_Page_73.txt
8dd61efd6e3e05d20f7ec2fcbf98a506
5dee5767389382b850806a91cf8dbe2ae3c64eaf
21657 F20101119_AAANTN jayaraman_p_Page_75.QC.jpg
abc5cffc2b80eb541474623ee24d95ed
57f75c9c02a404995a563881dfe7ec5e84568096
2005 F20101119_AAANOP jayaraman_p_Page_19.txt
ccd314351def77abfabc2d51704ff9ba
3cc933bfc78a9eec17919db1b407a11833ce2d47
83569 F20101119_AAANYK jayaraman_p_Page_70.jp2
052172a0fb3fb17b9ce08b51a9e547cf
c474fa4ddff48c67de6dded48b6566f1ea4161dc
F20101119_AAAOAK jayaraman_p_Page_62.tif
a6d05706934bbcb444671b58cccfa69f
0227e05510b068482f8b6853ffc81646b650478b
1915 F20101119_AAAOFI jayaraman_p_Page_75.txt
f63620653591f3ac099a949adb8fd19b
1cbb63fe79f863c236a2fb24f8b007467d563760
20541 F20101119_AAANTO jayaraman_p_Page_78.QC.jpg
849c178ade33acd0a539447763853846
a1ccfa000a556cf395423d13164b935a714d6b58
60738 F20101119_AAANOQ jayaraman_p_Page_24.jpg
859e01734649208d33e1d1e1039d4595
22b3b8312c8af5278ee34900406ce83cf2271deb
55633 F20101119_AAANYL jayaraman_p_Page_71.jp2
cc6dee06a5d9f83bc187e80585ea8faf
923289a4b98cd8715691c54a1f809745e054c006
F20101119_AAAOAL jayaraman_p_Page_63.tif
ace5c9046da4e5baf7e568692550195c
feaa019fea1138a9df92011502270fc5ee57bcfe
2389 F20101119_AAAOFJ jayaraman_p_Page_76.txt
2867c2b6c0057f9e7b7e1648cb5e6c0d
44cea5190cff0dca61366ed2a24d246994cb0241
47575 F20101119_AAANTP jayaraman_p_Page_75.pro
2822c9275c481a81d95b75843102a744
0fd60781a971236820bba2cf961f73e47fc2706b
122805 F20101119_AAANOR jayaraman_p_Page_16.jp2
74e4d940225a9dc3cd5700803a6ac934
849b5903a77f59d55b2e192ab47e1f2646df5d86
17777 F20101119_AAANYM jayaraman_p_Page_73.jp2
da978684658a4b67ce906e207d85ba92
0ae6b71417166c69bfb46c5979d2915e6f269d54
F20101119_AAAOAM jayaraman_p_Page_64.tif
d1e6c6b6bd84b0ade7c969ba64bf489e
f07fdf8b6dd264f16630735010ec3fcc4249baf5
739 F20101119_AAAOFK jayaraman_p_Page_77.txt
a57d4fd65e4aa5298cae57029e74c26e
b96ee5ef9d7411d2a4871f32d9a1b1c20bf1eee7
1494 F20101119_AAANTQ jayaraman_p_Page_74.txt
c6f8135e9091a14d694b79ed9541f3f9
0b003f67e968150f9d76e02c324acf560f6dcf9d
63434 F20101119_AAANOS jayaraman_p_Page_21.jpg
cba047bdb8125d31b74c142a5f5360d1
0cea996f8878aa5a61dabec028652162e1fdd7c6
43395 F20101119_AAANYN jayaraman_p_Page_77.jp2
3ac2a75612aa852886bef636518d58c2
df29dcb88ac542cffb623d7417ec390b53267094
F20101119_AAAOAN jayaraman_p_Page_65.tif
f0de5b930b09e3e5e1245d8a173dc577
76a27d6965e3493517c35aca3fb7ab69b90cf35b
1939 F20101119_AAAOFL jayaraman_p_Page_78.txt
c4ca1e7876c39b4e24ef5526dd8ab289
657362bc64223073106bc0f551aae99495bd2e66
1855 F20101119_AAANTR jayaraman_p_Page_38.txt
56eecfa97195e9f722c040bc58a79e4a
86036bed3fa84cafe3b7cde08c136d73f398560a
5776 F20101119_AAANOT jayaraman_p_Page_27thm.jpg
6d75ecd78ed1c8c208f85e50c5d6edbd
5d12bef1c3411e4a22a2424b9ecd23094d822869
102689 F20101119_AAANYO jayaraman_p_Page_78.jp2
8f8832491ba7cb681ee8a72fa2a38d81
f915d4ed16cf8c4bf3585887ff08c9fc4cc8d9d4
F20101119_AAAOAO jayaraman_p_Page_66.tif
7052742e6d9bd066f6224cd2721e9435
9a69baf9364b97e4efaa714f8c6e714a607bbdb4
2446 F20101119_AAAOFM jayaraman_p_Page_79.txt
2909d9e18531b37717e7df9fc9848387
f32897b7f7443dcd1b75f9f20abcd50814492335
21209 F20101119_AAANOU jayaraman_p_Page_38.QC.jpg
15652953b059ae0eba892a19d58a1c3d
d7d72c433154117b3ec79527c6eac0b99ea59e88
131004 F20101119_AAANYP jayaraman_p_Page_79.jp2
638981c26958cee1dad2dbe195aabe39
d58808645ed8cba67f5632a6c7dda9bd8a8d8e2b
F20101119_AAAOAP jayaraman_p_Page_67.tif
6c00dd41c233ef077697ffcb6d0a9fce
6350fbe45fdaa88447cda7774576771de0e96941
1123 F20101119_AAAOFN jayaraman_p_Page_80.txt
6eec7ae1c9a5cb55e9338252c8511e2a
0a3603fe2712cde6558495479ca7c19b774fe207
65716 F20101119_AAANTS jayaraman_p_Page_54.jpg
73a14632c4f96fa0ecfcc5ebb7ff9b5f
734d89c84f6616649bf62bdc1297898cef2b2812
F20101119_AAANOV jayaraman_p_Page_33.tif
c0ff3b9b40344b95cfff2cc335e77579
10261b557a8d4893fadf8aa9227f2cdd715e0bd9
62745 F20101119_AAANYQ jayaraman_p_Page_80.jp2
2040d39af86035392511e8d530a422ae
7094ccdd909125dee3bbb6c46cae2eb66df768a0
F20101119_AAAOAQ jayaraman_p_Page_68.tif
b6b38d31d4c554c0ed717437920c0ade
34aa1818cc6aa5d06d316831523b585d10045aeb
349672 F20101119_AAAOFO jayaraman_p.pdf
380b149e01f615e470a8ed0be5a90572
158fd8acf61b76d0fbbb32bb3284a23c3cc45b2f
75885 F20101119_AAANTT jayaraman_p_Page_06.jpg
9c75c5cf94dda537de15f5ce7310de29
385ac303b882cf1ffe548a6797db6940e78bd892
24917 F20101119_AAANOW jayaraman_p_Page_56.jp2
e0a30a7eaa162d7baf39537a44836a46
3c0c95666a5d671456fdf05869e0de058b06f814
32562 F20101119_AAANYR jayaraman_p_Page_81.jp2
78a8bb202c5123cffaffe7e6a2f21b77
a34edb3277b6bac27278c82f56caf8f34e75e371
F20101119_AAAOAR jayaraman_p_Page_69.tif
0188cbd6cde68e9a7d29bc8142e9b733
15e3bd60fae639c257a162f03c0b3fa1c60dff44
6441 F20101119_AAAOFP jayaraman_p_Page_01.QC.jpg
49c9a54429e082cd44339980570ec242
419f275cdbaa2d72750295401a81034642c6ce9d
2334 F20101119_AAANTU jayaraman_p_Page_55thm.jpg
2bf3730eb92f7871895cd9fe58100058
3f116ee4ca77c2a881bf8f3acac8801b717cd96d
94380 F20101119_AAANOX jayaraman_p_Page_22.jp2
4e83817856b58a91eb6ce53735abf605
0eb9fc799ed54c894ca9e5b9395bd8b25f34b116
F20101119_AAANYS jayaraman_p_Page_01.tif
5aea51a8a24e91b885878602eb683244
87e4d4eb3b232093a8b9c0a6e06f538f7657695b
F20101119_AAAOAS jayaraman_p_Page_70.tif
a01ee917d0248afaae2943b4bff4af22
b08d15183ee7707e9761e4fdf9dc9fef799805c8
21191 F20101119_AAAOFQ jayaraman_p_Page_25.QC.jpg
2e3d6f50b335d81db605dc69bfa40bf4
6888168f771a0219bd8e09092b98da2ed5bfab9c
F20101119_AAANTV jayaraman_p_Page_72.tif
27585b0012734d121b8371fa49379497
da502f0c88fabad2002b18de7953352eff4bf0ec
59127 F20101119_AAANOY jayaraman_p_Page_15.jpg
ad0a87faf1b52e862a3f5c149ada53ae
600fb6af3f665812973df554fa811fd75af38819
F20101119_AAANYT jayaraman_p_Page_02.tif
14a6efe12f065c82e1475cabcb44f0ee
d7723b27c40a938a7293df8d97ec32d1cb6f81d1
F20101119_AAAOAT jayaraman_p_Page_71.tif
8553d949998b22f0948105893532656c
fbfd7811262e429d809d5d38dffc59c4ec524a0b
19395 F20101119_AAAOFR jayaraman_p_Page_65.QC.jpg
4f97e42836aaa856beca8c6ea77b5e38
0f9e67564bea23e92b932102c772a4738c428b4c
67027 F20101119_AAANTW jayaraman_p_Page_39.jpg
e66fad427443c5347fe2f03577469f6e
5d88ba8be91ee4659c822deb99b36634b2ae475f
34955 F20101119_AAANOZ jayaraman_p_Page_72.jp2
3a552c0163c4d11841959e323a80faba
cbd3e975ca55dc10df777c2585b6940a106eda3c
F20101119_AAANYU jayaraman_p_Page_03.tif
8b61727958fa142521ec7967205ded3d
06b1f4426c1d510a776d85a70101678b3ca2822f
F20101119_AAAOAU jayaraman_p_Page_73.tif
2a076e377d23c986fd2bf61a59d5ee9d
f6ab5f4a5cd738351049f70e1956ea87e44c2ff9
8104 F20101119_AAAOFS jayaraman_p_Page_12.QC.jpg
4a8d35e841bfbaca7f8749046da2f32f
f41902fc78ed7243fe02ca6aec8a1b820189b1d4
2002 F20101119_AAANTX jayaraman_p_Page_36.txt
bc7863ac0a864cbfc2b0cce1e2a8513f
91635dcbd1205b18feccd947e89f89e2e9269cfb
F20101119_AAANYV jayaraman_p_Page_04.tif
4fa9a9aabfa829487616d99f2f84e5e7
cd6d2941e6c32ba987d2ccc74d93dc558d5fd2b9
F20101119_AAAOAV jayaraman_p_Page_74.tif
ab6aeeb14a7102dc52a36435904e2120
bd77b57e02c50eef6c8d919284d8e85fdd16218a
5652 F20101119_AAAOFT jayaraman_p_Page_35thm.jpg
10c6ee860ce9f6fc51a1d5ae096dafd9
1df7e6bc57420365f54e24b14560487e8e0a5b5b
25271604 F20101119_AAANYW jayaraman_p_Page_05.tif
f2394a7ede14b335db092be42fcec14b
6c64ffdcd82d15b9159f408cc51ac04da144c884
F20101119_AAAOAW jayaraman_p_Page_75.tif
1e94045dba7d4a846311167893ef8536
bfaf66e2a24361defd637b2bddf198730459afc0
2615 F20101119_AAANRA jayaraman_p_Page_73thm.jpg
1945ee7d2c68fa8eff664cc9cd032552
320ef5e062cc6b6b2846fe3d600dc335d67d7dd3
86843 F20101119_AAANTY jayaraman_p_Page_65.jp2
82f4dd29cb7f2630c022067199e6dc33
4c4941a260330589c51c5c93798e4b2a2a47bd8b
22807 F20101119_AAAOFU jayaraman_p_Page_19.QC.jpg
d239ad47fbe7333e4b938f1c2ee8ee94
ebe8b65fdc04c05261d1387fba72455b3acce5fb
F20101119_AAAOAX jayaraman_p_Page_76.tif
1b5cd7602f9737a4c6c37f8df5cc0db5
b09ac54875aa946b8a96e8c067af10fc9692f32e
116 F20101119_AAANRB jayaraman_p_Page_02.txt
44a60fc10ac35a8756d691633de9c98e
6d199878bb49d8a8f33250f4cd6e91b1df33650d
70116 F20101119_AAANTZ jayaraman_p_Page_40.jpg
83e62930a62d8bbf1b3f8f63a96429ff
5e981ac14593ab2d718ec83a7e7e1159d59315c9
12865 F20101119_AAAOFV jayaraman_p_Page_71.QC.jpg
bf1e2712c57aca7011ebfcb369ee813f
c8c2b4c76fbe6731cbdeffc07522f6a4bac21910
F20101119_AAANYX jayaraman_p_Page_06.tif
4720aa54e772d81dcc32c57aabeced4a
d71ad2f7a89e25bbe1b8a9f8f3cbb82127183dc7
F20101119_AAAOAY jayaraman_p_Page_77.tif
7bc620bfd63eee490ba56a9c9cf7f2d4
7153c30fe307b76656a4b8f8640c14e2bc96dc2b
95491 F20101119_AAANRC jayaraman_p_Page_43.jp2
93269fde6eaade33321a5a245ccb3bc3
360ae8ee29e1a7cda12bedffcbb3846c1edb6fe0
5605 F20101119_AAAOFW jayaraman_p_Page_18thm.jpg
120a0f5cca0f112932f50f7b98603d44
58acce1b750d88ef79ec08704d7ffeb148db1d2f
67525 F20101119_AAANWA jayaraman_p_Page_62.jpg
6607c06c493e0c19ec6bd2507dc73771
8be77d01ba7f6c1c1354c17d3a484878fd9b8371
F20101119_AAANYY jayaraman_p_Page_09.tif
8806fd857b0e94942449d7df3e33e5d2
3e66b023f2adb8b146e9b6cc4d19e79ebc6795ca
F20101119_AAAOAZ jayaraman_p_Page_78.tif
f844818c3b528d612a8b6d6ddfb6f4ca
764cc26bbf3046391f337f53da9faa6cd1d58b2b
4878 F20101119_AAANRD jayaraman_p_Page_61thm.jpg
d218400e95811723b320506892550b7e
7193bd7bd0e73cfcf5ed85125c97f62e07554645
8754 F20101119_AAAOFX jayaraman_p_Page_81.QC.jpg
0da9434025c766d5a4ab4e1d2d382e21
ffc972fe6fd7d8fb571298b2383679d47adcafd7
67908 F20101119_AAANWB jayaraman_p_Page_63.jpg
61d2793c8574441abcea7d6600ff991d
c6468edeae3ad0c1a6a0cf8badd4ffcbb02f2caf
F20101119_AAANYZ jayaraman_p_Page_10.tif
07f8babdec9c7a71973c2f6a1fba3bf2
1121b2d2ab24e2a7de3d5534b8ba1a6a73f60acc
11050 F20101119_AAANRE jayaraman_p_Page_31.QC.jpg
6242ce7b5c3dcaafeb00bb41e9ce4f61
8514fa99af2866fbbd428b04b6144ffe2c261a5f
3480 F20101119_AAAOFY jayaraman_p_Page_31thm.jpg
a517aa5de7cc140591bce7fc9c4bb9b7
e25edad85890f0b7f2ede606070ff420fe7abac2
63394 F20101119_AAANWC jayaraman_p_Page_64.jpg
ba9fd94f620b5f26c39c470b971e1a73
70f89be4a64f80d40130ff5d1f292b200eacba05
4638 F20101119_AAANRF jayaraman_p_Page_41thm.jpg
f634a649611e5898f9f27e487829339a
198db60c0ec366a4832994448f6a3c0adbc23a5f
41304 F20101119_AAAODA jayaraman_p_Page_65.pro
184ff3ba236b1120bba7c71f6587e884
7f4d0f0330f8869cde99bb72db95004aea662bf9
22866 F20101119_AAAOFZ jayaraman_p_Page_76.QC.jpg
22c86d3e77bce4dcd907971bc82bf501
9af10f34d6a3cffec1ce034cf5561b1168b730fd
58675 F20101119_AAANWD jayaraman_p_Page_65.jpg
27b58bd48aada67d9fef9849764b761a
e3241523ff5a284e07290f30d1b82001d4661b13
1704 F20101119_AAANRG jayaraman_p_Page_52.txt
eec82ed4ad6469ac91aec114bc6cf1dc
59e029f837f5820c571be09d2cb158b86a0e9cfd
63784 F20101119_AAANWE jayaraman_p_Page_67.jpg
94b64c3643128065ec6dea1464cf1f5f
a0c09867eb46be9ec8ad11f26b29e1b92760408d
1234 F20101119_AAANRH jayaraman_p_Page_41.txt
d2a400baecc26bde8c4520d063324967
4f0aab14ff949362c5167ca86eb58e17929c1fae
44476 F20101119_AAAODB jayaraman_p_Page_66.pro
e433dcb8a1ef3c95329ff12abf0a352a
6f53157d2e0474625774831646dd10fef13f92ea
6089 F20101119_AAAOIA jayaraman_p_Page_26thm.jpg
bd34cff158ba80520903999ee646f577
f0755b6b9b7590f6143674994439db93b6cc9e52
66850 F20101119_AAANWF jayaraman_p_Page_68.jpg
c88664af2dc3cc5f2512cf9140ccd33b
15d5f253fa0dcf529b245178392ac9ade1ed8c44
F20101119_AAANRI jayaraman_p_Page_08.tif
0ae5cbdd11b5e17962fb6f216dd25d72
43a125136756bb91de5aba5f4e10d7009f5fcc30
44861 F20101119_AAAODC jayaraman_p_Page_67.pro
7b6492a2ee36392b85983c1300be5e4c
d3292243f795084f4c18e707fee35d6000066c15
21780 F20101119_AAAOIB jayaraman_p_Page_28.QC.jpg
d1c3fcf84539b93da3e051ec08895727
26f31d33b7d18e11c4ed190f2a1da7712cfb401f
58563 F20101119_AAANWG jayaraman_p_Page_70.jpg
e7a2d64ae5af7120889ba5283e3bcedd
36a84641bd5990df530dd095cb58fc0a127ffac2
F20101119_AAANRJ jayaraman_p_Page_57.tif
1f7cf266c8fb9d8efdcde24ef21e596a
49695e6d0c07571dd1aa1360c113ac6c3b9529ac
48636 F20101119_AAAODD jayaraman_p_Page_68.pro
758550c83b125592c5f2c8bf385c3d66
05912bf1c8a33c77165c3dbfbb099b1bd9363a7b
22906 F20101119_AAAOIC jayaraman_p_Page_29.QC.jpg
8bd0d62ced19305d049191f3c492d773
05ef40033340e6f67f541137313b7299f2fb75d1
39836 F20101119_AAANWH jayaraman_p_Page_71.jpg
9249762fec02825e971b794ade487846
e060157fcb0b261456d1b70546f1b63f437f6514
21618 F20101119_AAANRK jayaraman_p_Page_45.QC.jpg
dce41d7eae0da63cb033eab5957246a5
693cb83991b97fa8b877165d4bcf6648ce6f8fb1
25194 F20101119_AAAODE jayaraman_p_Page_71.pro
e48db3dbd2f576d128ae4701471fc2aa
fd843562a6287205437972420a3d95074b9a8608
10215 F20101119_AAAOID jayaraman_p_Page_30.QC.jpg
9e32ac28956bf2ccec897f993e861fdc
f9eaf588382d9fea87b682554a5b7819eefa75ab
28747 F20101119_AAANWI jayaraman_p_Page_72.jpg
dcdc029499713b69ad68488256512a24
bd97bf725b00cf85892fd5615fa5450956ad4c13
7058 F20101119_AAANRL jayaraman_p_Page_79thm.jpg
afceb43937391a3139d9c2a2f57189d3
f551e8ca1a962ab8eb9e6c785314379a4e034ba0
5905 F20101119_AAAODF jayaraman_p_Page_72.pro
4ad8a999cd4c8cfac1c44103cc31637a
f0dae574cbff12d2d8edeb0d57752e03a2242f26
3293 F20101119_AAAOIE jayaraman_p_Page_30thm.jpg
09de878e542b9f2ecb6a8cc1ab8f99a0
ec5fd9d6b39113e27c44761bdbdb174339b290d0
20569 F20101119_AAANWJ jayaraman_p_Page_73.jpg
d4703c02047f23f7944833a87266985b
a582c39a73634a8ab643313cae887846785c23c6
1925 F20101119_AAANRM jayaraman_p_Page_42.txt
4a7ea00c5cebb0d0a9a24021223191d1
309e173884780a71fd4ca79d8a258737f1200a4c
1094 F20101119_AAAODG jayaraman_p_Page_73.pro
950830c00228dc26000f5bc4cab45551
551558e16cecfc45482bdd9d423aa72e4022b58c
5570 F20101119_AAAOIF jayaraman_p_Page_32thm.jpg
44e91da42fb89568bcd877b6b065433a
994beb9a219c513fb6d97a18d457c3309dd63ddd
51093 F20101119_AAANWK jayaraman_p_Page_74.jpg
cd740ab385223cb4131952f3a9111c1f
ad7119880a9abfff9a3553eccdbe42a8be2d8791
22041 F20101119_AAANRN jayaraman_p_Page_42.QC.jpg
03295978d1cb321a8da14eb19ea0a5c7
885e6b99c0217d6d8a65399ca5523918a9c1e1e5
33210 F20101119_AAAODH jayaraman_p_Page_74.pro
d0d13c8dd2d759cb08a10afd5e77a3fb
d1b7ef804026d7fc23156babe019bcd7e6f0e5ea
67223 F20101119_AAANWL jayaraman_p_Page_75.jpg
1160e8629438df19ded1bff262ddd4a9
872c3893c1da9531bfa67abb23d3a0723a06db86
1842 F20101119_AAANRO jayaraman_p_Page_28.txt
d0bd1531d47f92198bf27bd700fc0aad
2eb35a418eba06aa583c984013aac27b935c6feb
56498 F20101119_AAAODI jayaraman_p_Page_76.pro
283c43be91b544eee4761748075486d4
bfb8295e415a2295a262e24bb3d91db5c87a586f
19682 F20101119_AAAOIG jayaraman_p_Page_33.QC.jpg
df4083e1513cb5630b47d5b699cfea90
d656a3d326fae9556beca53c7b9b0d27e81d26ff
66398 F20101119_AAANWM jayaraman_p_Page_78.jpg
0bffc1a8cb65317d637531e115519daf
bbf98262c571d71f280ed44b25c2f79b5115b989
31700 F20101119_AAANRP jayaraman_p_Page_30.jpg
801664ec5d81e59ac679b44fe94658bb
e02feb284febea1ceddf812a7b50a6854ba92350
18556 F20101119_AAAODJ jayaraman_p_Page_77.pro
4ca7e0d6bb537c65efdaf65e57fa0a66
10bb397d4ca95a2ad6432120a364f8b79ad68b4b
5641 F20101119_AAAOIH jayaraman_p_Page_33thm.jpg
7faf7c9ceed61b69aad3b44878398dbf
a47f89d303492a5ad3a4323972de4dba0cd646fc
88521 F20101119_AAANWN jayaraman_p_Page_79.jpg
1bfaf8e90ef671a86cca8f68f8756ee5
c658724a26d410746ea7fc3e8c8db0cf5ab3f87c
47326 F20101119_AAAODK jayaraman_p_Page_78.pro
9f3de5100716d48050229a10a5d76b2b
724c92806235a950fcdf3dc674c3ffcf040616d3
19619 F20101119_AAAOII jayaraman_p_Page_34.QC.jpg
1231c7ef47ccf26f296797491909f98e
200caf4077efbce0613db137443c1b6ef3166234
20304 F20101119_AAANWO jayaraman_p_Page_01.jp2
db6ad2751e934643f61583a717652287
d3dbdba22791abb8a3b9c76fce404748fb22f77c
14660 F20101119_AAANRQ jayaraman_p_Page_58.pro
31c0679af499b9eea11fcafcf29db5f3
1cf08f8163444943206fa11c548f88167d7b9a96
27383 F20101119_AAAODL jayaraman_p_Page_80.pro
79ea0100b8f874ed376219d84b10bacf
d811b3660fa7a839f3b654e8be42bd02d07847fd
5667 F20101119_AAAOIJ jayaraman_p_Page_34thm.jpg
15d42d6abbd429308ef202e104814b8c
66a64bcc7a6ed741d6253386ea6f62ee1045c20f
5872 F20101119_AAANWP jayaraman_p_Page_02.jp2
4f96303253f84ea3ba40e26b0ab93501
8243fe1750887ee60ca254ca8deaa1e5179fecfe
63651 F20101119_AAANRR jayaraman_p_Page_66.jpg
eaa4a46e483c4a06931680b0234df71e
3fd75d9ee9448dfcec3cde7105375f30e976f81d
13300 F20101119_AAAODM jayaraman_p_Page_81.pro
b2815da8308d463491fe2c639c5a633b
e1585d1f4a3f87ecc9157e5ff5e29ab9ffdaac8f
23323 F20101119_AAAOIK jayaraman_p_Page_36.QC.jpg
5d4671921e77e7538c0a98624a598fbd
419c4d01cbcb1b36917467a6646f37f92a1cacfa
5771 F20101119_AAANWQ jayaraman_p_Page_03.jp2
92cb0ac36609e01a80e68ec575d46cb0
1fd473576c8a0d4aa44557e716f44f2c063609fa
45272 F20101119_AAANRS jayaraman_p_Page_60.pro
a79c0b0922ed5d15d1efc380aefb8248
b2dec996590fd8e334330611a89f9df4ad6b52eb
380 F20101119_AAAODN jayaraman_p_Page_01.txt
8a7d039ae3f4300c36f9473ed44d5ba4
177b65a7005ce0dfbde35a67b3e9da6f24866ee1
6647 F20101119_AAAOIL jayaraman_p_Page_36thm.jpg
9f583625b18f759a06ac83658bd662f5
04aeb283368c2ea118ec35c26ec3d76d34e68b6a
103 F20101119_AAAODO jayaraman_p_Page_03.txt
67bc58c45762364a8d4b86f26745ecd0
e13a4dd4f926a6055c01a0b5b6550c0102b3952c
27499 F20101119_AAANWR jayaraman_p_Page_04.jp2
f30d0947408afa0d445ab3607be693d6
601f593e4b64626c713d78402b0236b517531e9b
6014 F20101119_AAANRT jayaraman_p_Page_28thm.jpg
5a1e7c27aac6aa4a0702216ad11fd1f3
95007cf94930575a3de932710e9aab2dfd1544bf
21815 F20101119_AAAOIM jayaraman_p_Page_39.QC.jpg
7f65c5b7197dcd79baccc327ab8ce454
cf5856168ca68d869b6a661df14dd663ef29348e
467 F20101119_AAAODP jayaraman_p_Page_04.txt
24a1ce688a8915a225b4e84daf4fa82c
2aa9cac95356ced90becf82707dc051741a395d2
1051976 F20101119_AAANWS jayaraman_p_Page_05.jp2
11b3f8ef99929c0d1c73d69b502a02d1
dcc50b06f19aab1fee6eb02d4499b134e53b4544
9019 F20101119_AAANRU jayaraman_p_Page_14.QC.jpg
da3736103a2439d2eea0f9d6d80c1983
381f540b46ded6ceb181dab49cd6ae5997fc8fc6
14737 F20101119_AAAOIN jayaraman_p_Page_41.QC.jpg
af509fb919fb390289236ff380e9f3d3
90e58e6fad1695abb299bb2481d06f7943d57ca8
3335 F20101119_AAAODQ jayaraman_p_Page_05.txt
c9c4ae1e913dd401443cf868f72095e4
2c3fdc7546e7fd2c84a7ef250b0be1970d975846
1051981 F20101119_AAANWT jayaraman_p_Page_07.jp2
6184df5321c2d62a7f2b1fda16c1fc99
33ccba0081c76ec096917aa558738d6e85c33e18
1785 F20101119_AAANRV jayaraman_p_Page_64.txt
ae29ab4dfb4c76c420494aef1fdb4b6c
4e08315d494d70108a1f7f0d2ad453ffa731bb9e
21148 F20101119_AAAOIO jayaraman_p_Page_43.QC.jpg
5fbe975dcdaeec9658b948ed3973c8e0
50428fd996696940742dca0c8c9b607441e7373d
4310 F20101119_AAAODR jayaraman_p_Page_06.txt
fab51a367ace089db0a31e0746a59f8a
1e1bc393b7fb813861a086e3b056fc699e65915e
629756 F20101119_AAANWU jayaraman_p_Page_08.jp2
9ce26acebca263298a12cde6e748440a
40b0bd406ee1699499d2526886e0da5de806b3bd
22436 F20101119_AAANRW jayaraman_p_Page_47.QC.jpg
f90ed3ee69c7de13bd47351ecc2d3967
484fc03e863709a5cda86146a996c04fc858a101
5572 F20101119_AAAOIP jayaraman_p_Page_44thm.jpg
16b4893f36cb71ea46da28e5f8979b3e
c50a83038ca83b4047bf5d2b8c083dfd11f61360
3342 F20101119_AAAODS jayaraman_p_Page_07.txt
d80e075c4eae3cfbd7079b2cf13ad1c8
9dfcf910a7247b43575bef311465b2ad0495a258
21967 F20101119_AAANRX jayaraman_p_Page_40.QC.jpg
b3b9cfd740d24b52bf0f347c6f011e36
be33d02b5f33f29506b7aac0b0e564659964e01f
6262 F20101119_AAAOIQ jayaraman_p_Page_46thm.jpg
92d393c498de38b3e2e6cd129659b6c8
762a08beb7340fc57c85c0995525d3f006305d33
1016 F20101119_AAAODT jayaraman_p_Page_08.txt
1ffe4935e388f1ddfbda6c57f040bfbf
5821b7b4c21750faac88cb82d89a7f16539b3f0c
58341 F20101119_AAANPA jayaraman_p_Page_32.jpg
e4930bec8aa5c44d8557f8495bb4d1f7
225db309ecb00bb5884bfc4534f47a1bd3b1b469
517 F20101119_AAANRY jayaraman_p_Page_72.txt
4052b58c2a68834fbe84f82a05d2e89c
3752f65fe62b5e30527f0dcf0b26da3980b276cf
76387 F20101119_AAANWV jayaraman_p_Page_09.jp2
d53c38ed55ef7fc633a02560539af94b
8d9c676c21be2f5921a4c404d3d20d765b268852
6195 F20101119_AAAOIR jayaraman_p_Page_47thm.jpg
924f6ed5e0d1be784e7960a290a9462d
40d59bcb3544d67375db819bd362b7c5c6891e88
1640 F20101119_AAAODU jayaraman_p_Page_10.txt
f147b41fe04ee3a23e278855ccf3bb2a
6dad954f60687667daf251556d8f5fadfb79d40d
48033 F20101119_AAANPB jayaraman_p_Page_62.pro
f102d10b76ce16910f14562aa2e4ca45
f865312e013f8937ac86772ae4cc9562959a4656
5754 F20101119_AAANRZ jayaraman_p_Page_10thm.jpg
2ff690bc3a505ff845b708aa5f23a5b8
3de86bcb78c23a44fb732fc6cdc81462bbc7dd6e
32642 F20101119_AAANWW jayaraman_p_Page_12.jp2
14ee8a066f19a997fc4e202a4406473e
e609ad1a17f1e1ec251f45f1d7a43decefcf392b
21900 F20101119_AAAOIS jayaraman_p_Page_48.QC.jpg
4e2e99fd81b5d467571bfb428a4c1e66
65c327767b8898606c078f4fe8d24250f2d24571
567 F20101119_AAAODV jayaraman_p_Page_12.txt
b8e35a5a97a981d70aeaad665f146c40
31dc9c0808ef28d55d0196a6b3cd28d76ab28c9c
1701 F20101119_AAANPC jayaraman_p_Page_13.txt
054e4855a992fb5a3b7f1e60728144e8
8d272c6d7b001594b764be684ad53ce2be9ae31c
83307 F20101119_AAANWX jayaraman_p_Page_13.jp2
238314a7a25a1cdd87d253ecc715d608
cbad257009460faa4dc15e14b41e09efff3c9682
6334 F20101119_AAAOIT jayaraman_p_Page_48thm.jpg
e6a25bf693c7bed906712157d4cffd61
4f90af10a99b55ac4ff8610edd03cd1fa364763a
605 F20101119_AAAODW jayaraman_p_Page_14.txt
168f0909c00e824a6d917e0355ea4418
b66339195e103ca82dfdddea8ab7ef601da195ee
F20101119_AAANPD jayaraman_p_Page_58.tif
1a9db5761fadfc4242f16eab9c1b2178
a75a5f8ac6387531d2fc268886b1bad78310a80e
1648 F20101119_AAANUA jayaraman_p_Page_11.txt
2461d8255ddff0238b4bb9d20d9370e1
7b3c288dac40d07cd3b43ac00507fee2b94cf95b
65696 F20101119_AAANWY jayaraman_p_Page_17.jp2
856acf12e66badc4dabd00c2da817163
9984232121bd72b95d35738a4b8b52059f210e13
21499 F20101119_AAAOIU jayaraman_p_Page_49.QC.jpg
38eefe7ebe3601af7ec3e6f79595d1c3
9dfe275dda0e275edf159a20214befb639b18ba3
1720 F20101119_AAAODX jayaraman_p_Page_15.txt
6e2e61361fbdf486caf236527c07a799
d61a37780a1a068ef32ec7f23dfa9ff2feed9bfd
1672 F20101119_AAANPE jayaraman_p_Page_44.txt
7b133ddc17b0032a4403f9b7f741896b
8f097a693c9c991f68cde044bc2d7136e53657f0
1816 F20101119_AAANUB jayaraman_p_Page_69.txt
8f2a44b4377f50b0121504e03602d779
4697ae909fa4c0a884618b4fbbecb2a15d555244
84496 F20101119_AAANWZ jayaraman_p_Page_18.jp2
398da6f2d9f6ecb40f02c46ac0041a2d
23d8c7157b248b6a19c77be7f0e9503625fdd750
19881 F20101119_AAAOIV jayaraman_p_Page_50.QC.jpg
fdf839a0ed08f7a1ada953fde637ca0a
35240e271f19f7825f79ea509506722041b22a32
2417 F20101119_AAAODY jayaraman_p_Page_16.txt
34244d5ba470dd39dc3edee7b08416d1
66d53d8c5421322e7530e015cbf869449033504f
47800 F20101119_AAANPF jayaraman_p_Page_61.jpg
7b22d9026ebf9b02c73f64bfe77ca537
4391538217494fad7203a727db973b60e01df7bc
F20101119_AAANUC jayaraman_p_Page_55.tif
151e0ae03e68e95c2c83be789cb05868
8667651d34a36c3dc03090f6f965552fe56ff017
6060 F20101119_AAAOIW jayaraman_p_Page_50thm.jpg
6f1358fd3f108a212b6fb76a9eca2e9d
6eab831333e1746c983f311b1c555bf844943b5a
1265 F20101119_AAAODZ jayaraman_p_Page_17.txt
9baefe2c6d5af72db05f425627f9d916
c0dc3271353388cfa45a3aa713a7ebead715af30
F20101119_AAANPG jayaraman_p_Page_07.tif
6faac2ebb299b1204a715f0d554b4a21
138edfc062935e67352ab8231e1ea2b5526df580
F20101119_AAANZA jayaraman_p_Page_11.tif
f7a17501543b3a4f90ea74ced0f4f05d
cd9ffc4eac377f5ad9e7ebd38d1ac3b312f7b12d
F20101119_AAAOBA jayaraman_p_Page_79.tif
2d3cf13326f1ce82e876a9216c3e6e3f
e7a3cc9bdaf6dcb1c4ae61445113b21f45aebb72
63490 F20101119_AAANUD jayaraman_p_Page_43.jpg
0a1ba45df33df2fcfbcf0a18ba807af6
ad00bd691f7276e8fec90e524befc18984da23c5
19819 F20101119_AAAOIX jayaraman_p_Page_52.QC.jpg
a568a0073540d73d12cfcdeade0685d7
8942d0ff441c4a60429cd6eae887f42573938768
F20101119_AAANPH jayaraman_p_Page_06.jp2
89f4a969963d40ca13bac461add76db0
64d41edcd794eb47713181784a5ad93f56123311
F20101119_AAANZB jayaraman_p_Page_12.tif
c91761533ddbd610d32cf9074534367c
552b71f937992140864162a95b62df3f1942c99b
F20101119_AAAOBB jayaraman_p_Page_80.tif
ca7c0f53db748ef30b70a020c172d804
d84e05720cdd31dd9138e599f0523c93e9bac81b
99220 F20101119_AAANUE jayaraman_p_Page_38.jp2
687a7867709481a54abfbd7de48b9119
97f208610b6361387bf3a19aa51e69061bad9f46
5766 F20101119_AAAOIY jayaraman_p_Page_52thm.jpg
843cdfa861ba32ba1848ff0268c093fe
6de16236250be99350d16b90c89f7dbdca60967a
10895 F20101119_AAAOGA jayaraman_p_Page_57.QC.jpg
4c077372d814bab0bce6903d568de3a5
122956df2b2ebe3963b2204f18ebb33559168312
F20101119_AAANZC jayaraman_p_Page_13.tif
9991bdcfd8e31d7d57dfd12058c42ff9
b30df977aa01580ac0c835aefe57ebd586940d5c
F20101119_AAAOBC jayaraman_p_Page_81.tif
09205c69842d829d07ef8bf279dd20b2
bf3e68a5a74d8cc07923f52f73ec561d643762c9
F20101119_AAANUF jayaraman_p_Page_53.tif
3c177f98555c07da0d1687d1c98677c2
139c43a1eb16eeee8ec97b843505750dd4663f05
26410 F20101119_AAANPI jayaraman_p_Page_81.jpg
f14e2be3a92522f70d11d23f0d7d28b4
ac85ece37d808ec588c8b87782bb0876494e8cd8
21277 F20101119_AAAOIZ jayaraman_p_Page_53.QC.jpg
c1e17d96bba56a48a7e3426f0020d3ff
26e68978e2487a3ad95e6e8ed13ee74e1780efa6
5986 F20101119_AAAOGB jayaraman_p_Page_25thm.jpg
167a0e3f8a70be762037ec957f094904
22eed240c7dd5bad16109993ca32c3b029cd84f3
F20101119_AAANZD jayaraman_p_Page_14.tif
fedc45c0c2545e6e7f229313464a20bf
97d343cf7e447a73208addfc2700e51c77d82581
1254 F20101119_AAAOBD jayaraman_p_Page_02.pro
3c745366a4a51623e117b2ddd570ec24
7775a179f0fa3ae53c8d8b4214e2c3245b6f70de
20265 F20101119_AAANUG jayaraman_p_Page_55.jpg
82071ee476e5a647bbaa9d3b26a28358
9553f28cab691d52db2d8a9ced8486eb525135a5
F20101119_AAANPJ jayaraman_p_Page_41.tif
ffe34c40bdc0eb315fb9643af8a6aadd
8446d4015a676896960a065977c8991fb4e659be
2549 F20101119_AAAOGC jayaraman_p_Page_04thm.jpg
e8d25a5d3a028358387159bfbef94293
912122d5d139b6cf353e267fb592a45b2e7a3ca0
F20101119_AAANZE jayaraman_p_Page_15.tif
ae61233b3bed881f779040a24427764b
d1b6238eccd5f59b654b0d26fe99885da6f09fc6
1276 F20101119_AAAOBE jayaraman_p_Page_03.pro
c82f3c308b34174c12096ff6eb89c7e1
2885ff8749c942e3f9eb35877f8fb3482a43e596
68878 F20101119_AAANUH jayaraman_p_Page_46.jpg
6fedde0857d7a74b28890cb9e8c155a5
2cf5aa661b00627e4e468aff27d2a54b7c2109ac
76977 F20101119_AAANPK jayaraman_p_Page_05.pro
df2c2bd3e711855073c2c1fc6115f233
f5243202f57a83ed85c8c8b76a8319a6fb3198b6
20501 F20101119_AAAOGD jayaraman_p_Page_35.QC.jpg
b880eeca26e9dcaa1d7caf7460be429b
8120ea9d51344c8a13f36bb96b8b7ebde3c27e0f
F20101119_AAANZF jayaraman_p_Page_16.tif
9041f7f277db09f5e75a89663ecac3c1
4f2f85f32d92eba5f13d43e6ff3159e49b3e793d
10529 F20101119_AAAOBF jayaraman_p_Page_04.pro
0c122c0daa4cfd0d78c6b6ad95cf8f29
4bacef19ce22fa91229b2c00a53e8feef6aa08c7
24393 F20101119_AAANUI jayaraman_p_Page_12.jpg
de60f0d0b947feff1a83a0aac80ca95a
d61c398697a9764429276d61ac19f7d88730bb8d
F20101119_AAANPL jayaraman_p_Page_35.tif
0b64417939ec6d33b152bc050133cef9
ebf70345541d0966e92e19e7aead245d8773d875
F20101119_AAANZG jayaraman_p_Page_17.tif
40a6c98487e3cedd0b75f04271d26f50
6ad05f903c656a74daef6ab2a49f9b221b8ff3a9
96149 F20101119_AAAOBG jayaraman_p_Page_06.pro
b54d6d94b0b454fb0a3c4970f8b235fb
134568b3671db5931067ba99bbe8137372a54bf9
1852 F20101119_AAANUJ jayaraman_p_Page_45.txt
607488e062cea4959d78a41ada6e64e3
88313f698b4d4206a5d75deac1cdb2591bda65e6
1360 F20101119_AAANPM jayaraman_p_Page_61.txt
18345bfeb6bb24c3d14dbbc0609f4892
ca02d7b4beae1d827f0d33a22f52ed239cd6e520
6106 F20101119_AAAOGE jayaraman_p_Page_20thm.jpg
85b62f4cd0844ae801ef433a6e52b8d6
650915804ef1f2eefc861ffd4ad86fc62827ff4d
F20101119_AAANZH jayaraman_p_Page_18.tif
d3392c061df1b08e50447074a8cbc543
88637195a65d85f253c2b76c6dd62a1b3f5dd33f
79569 F20101119_AAAOBH jayaraman_p_Page_07.pro
c09e48ed9acce48df148a38e82a1f559
38f784bfb0e51585ede1bb0adb04e9f301a800e8
1798 F20101119_AAANUK jayaraman_p_Page_66.txt
f785e0e39db7d01c51095b74f179e755
423ca1724191c6c74e4206eceda5113996862453
18100 F20101119_AAANPN jayaraman_p_Page_13.QC.jpg
8f85d1c24d20211a21d627f16137a6c0
bc01a2304eb55ae3c6a530b512443062e21897cf
7293 F20101119_AAAOGF jayaraman_p_Page_73.QC.jpg
dd0fe6529c44aead923d54e75bfb4b77
54a4d60629cb7c2d360576913b21b302a5b1ea0d
F20101119_AAANZI jayaraman_p_Page_19.tif
811f777a37a17f75942fb68d516e68bd
cd4057822cfd26852e925a59cfd8cdca634fa77c
23500 F20101119_AAAOBI jayaraman_p_Page_08.pro
d7362f7ec446cdb705ac938d0a66a376
0717f45bb57a744e5c99ea26afa4a58af654a120
95867 F20101119_AAANUL UFE0014040_00001.mets
e3d1e44844952fe12b058fa760c02b40
bebd277bda756780da2377fad81f73641838ac61
4928 F20101119_AAAOGG jayaraman_p_Page_06thm.jpg
5d4ad6121397327669d542c1426871ba
8aff543f8b0b9f77ec912db61e343013219c915c
F20101119_AAANZJ jayaraman_p_Page_20.tif
21f9cc9fb08065f54a052204b7eaa7a8
facf7945bd06d3ed2d04e890d3326b7723c4ae99
34348 F20101119_AAAOBJ jayaraman_p_Page_09.pro
0a46c24ce684b80782cab72e7153bb21
d44895eb1d06620894be4464411b56dac32c9cbb
75110 F20101119_AAANPO jayaraman_p_Page_74.jp2
34056e6cce3f090689997b3f8407da6e
296da81704a41d016e9ed89c8fbc79d18f884856
5832 F20101119_AAAOGH jayaraman_p_Page_51thm.jpg
e82e34269819ed79b5ee298f0d2a0d71
a37e8220a463ebfeaccbd1dddd21bc70a5fe5177
F20101119_AAANZK jayaraman_p_Page_22.tif
238c775cb787a4333bc824b53a7cba85
48b9c9029ff231704f2382a6cdc8db74774307e4
13523 F20101119_AAAOBK jayaraman_p_Page_12.pro
a8717ed0e1fa8d84193176037e4ce7cf
62df9bf32d6b8c710d57b9cfb41801aa845ddc7a
3153 F20101119_AAANPP jayaraman_p_Page_03.QC.jpg
860ba062071723d781a16b981a921075
da992054d4484d58fb9a12394a66a71f778956df
5768 F20101119_AAAOGI jayaraman_p_Page_43thm.jpg
64bda680b94bd85492ef5da8b4c14524
40e644a6eb48627284541315b69ac5326d68142c
F20101119_AAANZL jayaraman_p_Page_23.tif
53ce1557c586e8e230a98e4f6402f109
dbe208859a556e45d88d8e5d0dc7241b741b6ae4
15042 F20101119_AAAOBL jayaraman_p_Page_14.pro
688c2ab090d0550ffaf8c9fc1676673e
82fe07c79a77a2aafd4c47dc907923f86d6b5d34
20589 F20101119_AAANUO jayaraman_p_Page_01.jpg
9743a9a05ff61e81fdb8f7ba0bcf19c6
175406deb8c61a3180ffdaf45f4c25927474f977
1841 F20101119_AAANPQ jayaraman_p_Page_26.txt
89b62fbfecbc9cabe70aa5cb093318b3
1591148d014ec3dddddc681b4a02fe0c99b8ebfe
18457 F20101119_AAAOGJ jayaraman_p_Page_07.QC.jpg
9b73fc019321bf63de8c11f468f8945a
a271dc0854210d3ef1293284a023ef21da389073
F20101119_AAANZM jayaraman_p_Page_25.tif
f683e9ebe65866ef81f32b93ed557aea
11883acb00fe3a4fc9e5f7367f3dd684cd3f8b2e
40427 F20101119_AAAOBM jayaraman_p_Page_15.pro
2eac593012d01cafad426a38dd993add
e672812543625c7ddc1addb1d92fd3d877623ce9
10377 F20101119_AAANUP jayaraman_p_Page_03.jpg
5c84443c49d226ab4705a7e9d788d748
6f62ae4eb67766fa53fd010973c371ae80a46154
5594 F20101119_AAANPR jayaraman_p_Page_70thm.jpg
ac86b823c66bfc2d64122c832af438d0
2ab5b9884a3a7e91381b2c4763d6718102919d74
20177 F20101119_AAAOGK jayaraman_p_Page_24.QC.jpg
43b56d8c261e8f4c1cf8396439819aae
8f6e3084a502ceae3c8e0e31c4410e1fc39f414c
F20101119_AAANZN jayaraman_p_Page_26.tif
cd1f2a2d10844c66e71dda72d91668c2
769b1778a672343d3e797e87daf682cb9b85a3b4
59105 F20101119_AAAOBN jayaraman_p_Page_16.pro
acfc9aff7d745219911f8d13c19dc6b5
f3f09bacd9ac64bd266d73dd20c0db92f0449fcf
23138 F20101119_AAANUQ jayaraman_p_Page_04.jpg
4e98b0094bb9fe877640eef74436f4af
92e1ff9878a0fcfd77457efd650ad0d0bf6d01e0
F20101119_AAANPS jayaraman_p_Page_24.tif
54b4c8cbbdb94dee51302e938760649b
116cc31c699ae545d4c9aa2a6bec079a3b91586d
5728 F20101119_AAAOGL jayaraman_p_Page_78thm.jpg
35ca786c7034067e3f191328c86b7aa8
d76b8c47b898278ec589aeaacd08baf9dbef6ba4
F20101119_AAANZO jayaraman_p_Page_28.tif
958452704e2b1b3fe85ddf8ee0438f90
e7cc15ebbebf27ca4f4a98c58e6ac4c9f0a21c57
30686 F20101119_AAAOBO jayaraman_p_Page_17.pro
0e70ef7fd27957ab16ae82c4a55174e5
5e9e3d8c952f9e8867257700343848e38cfb1272
64566 F20101119_AAANUR jayaraman_p_Page_05.jpg
4254cbeac0ac358a175d2260e2f2c96e
247de0cf51b8b84675d4078aff3f1e6c5b068c97
19402 F20101119_AAANPT jayaraman_p_Page_44.QC.jpg
76281e2f81b1a741df6565bcc74c2918
95d5d879a9f45065724963f6ebd7d62cf758f4af
5164 F20101119_AAAOGM jayaraman_p_Page_13thm.jpg
b1edb877faa5fb857680b3f27197cb26
ec46c1e18b718fb253aa819a31dc752ae4bad33f
F20101119_AAANZP jayaraman_p_Page_29.tif
95b2e935c31bbe2f1163ac593e3df65f
798d248928626d3d0c94ac45ba16057c4978872a
38565 F20101119_AAAOBP jayaraman_p_Page_18.pro
a14642ad2d57f8966a7d54e53abe5e7c
e868dda145865312203bc6a00aa9fcdee2751812
71965 F20101119_AAANUS jayaraman_p_Page_07.jpg
60735488deceeb1d650fccd02695a330
e3456a7fd06f5cc6dbbf2062612d4b366481a2da
20035 F20101119_AAANPU jayaraman_p_Page_23.QC.jpg
b3e801bf8a9d8347bc5c125c7dc460bd
58cab11629005b6f5d0a20cf959703af753308fb
6026 F20101119_AAAOGN jayaraman_p_Page_45thm.jpg
4515fb48a5ceb63c9acb1cab60196cae
bf0dfe5e39146a8d467f229df4ce9201020136ba
F20101119_AAANZQ jayaraman_p_Page_30.tif
6e0827ab0eac95ef9b799040cae640e8
69fc614c2e12ba4e0279447f0d86a70f4fa44438
50690 F20101119_AAAOBQ jayaraman_p_Page_19.pro
1574e1a4f98fc6f78dfbea110ed2a017
4d70d17b5515e1e7b3589407177aa2fec171e66a
1797 F20101119_AAANPV jayaraman_p_Page_27.txt
ba1de59faa8d90cb774fef1838e9f001
aff629ef7e7d8bdb274b847446ee3c9631ba4665
19187 F20101119_AAAOGO jayaraman_p_Page_18.QC.jpg
25c578b4191f940b433f810e006b44d0
4a491b6c0d73b2698e4a73464461593aec5aded7
43294 F20101119_AAAOBR jayaraman_p_Page_20.pro
fe9a88094e40109b875d61cdf59d7b04
beeba699bbf195eae9209aaed7595cc1ed917851
52410 F20101119_AAANUT jayaraman_p_Page_09.jpg
740b762448e5ea63b630b77631d4cb91
ef90082115be13e730729e02070976967ba3cdc6
34702 F20101119_AAANPW jayaraman_p_Page_14.jp2
8aa0a15b2c3df61f41fb5adbb99fd130
5e4864c77c6bd0b6836b0c85ff1b369fb30cf72c
F20101119_AAANZR jayaraman_p_Page_31.tif
43ccaf14455a593488ff6e96126caf0f
2058cf26d61af967f2eb4e286bfb947e5efe8f0d
21068 F20101119_AAAOGP jayaraman_p_Page_21.QC.jpg
a76498c9365ff58a0ce81ea573009152
3930f6d0b8963a91fba834302a1a13fd0a8ca7c6
44375 F20101119_AAAOBS jayaraman_p_Page_21.pro
eb57729cca89cc60a0fffe772a3de17e
717f2206ca062ce40a32b6572e5f84a285037aa5
58574 F20101119_AAANUU jayaraman_p_Page_11.jpg
b8b0ee217a0b536e1841c22b958eecee
31072b0d6583e4268edb587f6567f5a2af9147f2
42549 F20101119_AAANPX jayaraman_p_Page_52.pro
8b6e08aee8185a36a8bf2690bee98a9b
a390abde191be5bc73401ae4f163bd5d442cd965
F20101119_AAANZS jayaraman_p_Page_36.tif
0f5d821a86f08fc094a72024fb15d724
f683b936e7d2523c57c278b23627f08dae85f96d
7579 F20101119_AAAOGQ jayaraman_p_Page_04.QC.jpg
ecbbffa66bf04da74d659b32b0603b30
e54946e925bfb6f398b65752a7c35eaae4b9c15d
42792 F20101119_AAAOBT jayaraman_p_Page_22.pro
dd83c582cb6abbaaf0fd494db6e5a89f
131313aa7001f0cbc867770561f7e752b0c765f2
26915 F20101119_AAANUV jayaraman_p_Page_14.jpg
55dd23b7c34772987bc1403b0a2e6f3e
1569086107609126277e8ad16e978342764b7097
31846 F20101119_AAANPY jayaraman_p_Page_77.jpg
bb3e3eb46528699a4de6c60aaa10c76b
3749e76566c743096b1bf9dfdcd6fff823d7dd12
F20101119_AAANZT jayaraman_p_Page_37.tif
7871c04475cc1d20f0ce14c1734ad5d6
239b3ac8c92e8e861d8291aaeb8234989b8f13a5
6479 F20101119_AAAOGR jayaraman_p_Page_29thm.jpg
904324458999e711f0f27dc8549ffbe7
f2fb0002a418f844f7b9b9642cd8063b75f55c4c
42983 F20101119_AAAOBU jayaraman_p_Page_23.pro
5de25acebc03c8c508cbbc07180eb0e8
2eb521e7be6dc5df6f832f636123efbd93fceef3
80506 F20101119_AAANUW jayaraman_p_Page_16.jpg
821356475f2a26aa64fd7a25fef7e473
21072fbe12cbf7afe98ef33e8ea9515931ea601e
95374 F20101119_AAANPZ jayaraman_p_Page_27.jp2
67eb0e89d3e4504b9c21bc46cb700d80
97c9b66387d4b24a36eaa01ad47d7aed08c99db8
F20101119_AAANZU jayaraman_p_Page_39.tif
f375fc94d0a376e07d07ec123db44be2
4d01734859e08126835240725c7fcfd8fdb626f9
6698 F20101119_AAAOGS jayaraman_p_Page_16thm.jpg
97a603902cc6477c3ca1053e9ea188d8
ee05315ec5fb5b985e0b13414104998b64c35d2e
45329 F20101119_AAAOBV jayaraman_p_Page_25.pro
0542e8b61b81a3dcba8d0a7fcfd0afe7
a346a11844e106903e14269018a628d579e69ad4
70616 F20101119_AAANUX jayaraman_p_Page_19.jpg
e005053e0b88555096abeaff5e4cb0b5
db10dbe2add582bdd9a560fb20022fa887b779a5
F20101119_AAANZV jayaraman_p_Page_40.tif
bd138753eac7c5582e3ffa3c9d485038
74fa9c0c94021380dff915a1d6f29e8291118aaf
16486 F20101119_AAAOGT jayaraman_p_Page_74.QC.jpg
383c106ece98f27818ce7f8978d950c3
2e78b222e83086e1854f37800aba00103605931b
44875 F20101119_AAAOBW jayaraman_p_Page_26.pro
81739d1bcb0671d864b85bfd6c175910
94507635d5307dd277b2175491a16126fa03b9b2
2083 F20101119_AAANSA jayaraman_p_Page_01thm.jpg
a50e3e34cce15a9f28ae0c8504b59737
80b191c3cae0b321c0b73edfcc15d434df2270a5
63192 F20101119_AAANUY jayaraman_p_Page_20.jpg
980505915d14280437fd382fcaa1d3e7
f73c9ebe05acf57e48b48a1e84eec372157c302b
F20101119_AAANZW jayaraman_p_Page_42.tif
211697ef70a1f47abbbdf16bca649ad3
d6529692f99e422a4594acc7722c606c4093f2d1
18634 F20101119_AAAOGU jayaraman_p_Page_32.QC.jpg
bbfa59818388955456ce6c795ffeb952
6b1151ad9dfe8d37e00ccbc76b4f10103c2a899a
43793 F20101119_AAAOBX jayaraman_p_Page_27.pro
2a87667e91bd2b94d9a11d71b6c6f9a6
c78a74784090be50b4e9a4202540776cae509a91
6019 F20101119_AAANSB jayaraman_p_Page_49thm.jpg
778ede1732b58bda991eaf1885ff6c59
cc4e742f23091b4fca86b1292664fc965163f6b2
63890 F20101119_AAANUZ jayaraman_p_Page_22.jpg
4ba44f23cf96e798ed7a9e62bf82fcab
49c9fbb3230922c52582ebe40b83e3a3b4e34c95
F20101119_AAANZX jayaraman_p_Page_44.tif
211c906f028705629bb9fd8cc583c6de
99e0c4d5034ef4bc3e264d3eb0b5722e0db7f7de
20755 F20101119_AAAOGV jayaraman_p_Page_62.QC.jpg
7d54b6e18b49e3d3eef4c7d6d22981f6
4582b1cdbb0813d4c0cbe7bb9b85d6d3d945fbe0
102265 F20101119_AAANSC jayaraman_p_Page_75.jp2
e16bfe893d1bebe7ea83b9e924ac306b
8e1cc98e8a90bdb19665a0e51762b4098e4bb512
45860 F20101119_AAAOBY jayaraman_p_Page_28.pro
24a71890775bba9f7ddb57264b6ee46f
29c6e22383ebc73e81f17cb95a59d18c988a0876
3314 F20101119_AAAOGW jayaraman_p_Page_02.QC.jpg
6b8cf11bfb7395db41857a364cd78c4e
97d7722b41477c8e6e092153a78ff2414b831516
65301 F20101119_AAANSD jayaraman_p_Page_45.jpg
5e8f34699fcf8bd055ffa7f4ebeac035
4a5da8173a7cf2053ba5f49eba80f92791d47f52
107060 F20101119_AAANXA jayaraman_p_Page_19.jp2
f24b346e8810b3feee4a4eb517839ce3
a64f01166edfdc29b02d7568f25a0bacb130dad3
F20101119_AAANZY jayaraman_p_Page_45.tif
e2638540165bc6fc7a8a96207dd4a811
8d5ee154b1976f5c13a0000889a5fe7057a7ca9a
18868 F20101119_AAAOBZ jayaraman_p_Page_30.pro
f8ea271dcd0bf66724fad40ff5bc92ea
9a1a4671d850005cb561907cb2369d1ea0f92e98
6302 F20101119_AAAOGX jayaraman_p_Page_40thm.jpg
d636296569c8c023ec408b41d62f15f1
47fb6d9af5ecad2bb8b8f86b295a06da0439a30a
92633 F20101119_AAANXB jayaraman_p_Page_20.jp2
5c1b03adca3768af6e5ca4a39c594860
1ee6a990e986a3e1a7525c5efb2da938cd61610f
F20101119_AAANZZ jayaraman_p_Page_46.tif
a2f5107047e7c149a9e1ff8af01e97be
f5dbaeb938a1fc3b83da74ca954de0c37275cf0a
2743 F20101119_AAANSE jayaraman_p_Page_14thm.jpg
dae2fb9ce08f72923cd42e5c95dda93b
9b71368833b85896e6cb5a72b18af8712735f881
15594 F20101119_AAAOGY jayaraman_p_Page_61.QC.jpg
14d6564e79843fa68af5db5c16488d62
8e9673e78875f7623a44c9538583df931af5ca15
1661 F20101119_AAAOEA jayaraman_p_Page_18.txt
9c3288dab348c7893db80534b941de26
7aa632e80da5699d53230d124fe0deba822ee574
97058 F20101119_AAANXC jayaraman_p_Page_21.jp2
fd0c1c3f24e5a4d4bd12a2c9fe166bf4
1d1d2434b8c949545e79cff2680aeed25119cc6f
F20101119_AAANSF jayaraman_p_Page_38.tif
de557a49b93a1e75be6130ee2dac9134
e6a65a289dbcce280c60c1831b16de4b6864fb46
18992 F20101119_AAAOGZ jayaraman_p_Page_11.QC.jpg
ce7c71dbbd70f685fa56e711c2006506
e911e4da0740c95ee2248caafe115fc98c74a145
1734 F20101119_AAAOEB jayaraman_p_Page_20.txt
350da303b88d3d7dd37c96d70931abc9
f41c8c3043c399a51be04f733240a185523f0fa0
92051 F20101119_AAANXD jayaraman_p_Page_24.jp2
e97b0080c141133393c459ec6cc89b71
f11a1b3883c2490d915b9e99c46c1b8054d77b2c
49066 F20101119_AAANSG jayaraman_p_Page_29.pro
90719574d05b8af768aadd6875422618
fdabba456b79289b5023a0c321e616220f4bb99c
97364 F20101119_AAANXE jayaraman_p_Page_25.jp2
b58019d9bd990761f77bc0d410284cd9
a7f39a4e22cc2f52f436ad3ae47d6a85aaeab070
66720 F20101119_AAANSH jayaraman_p_Page_60.jpg
30909d634ec1ee7f7013c5da8b7f583f
5365d8ab5eb1a79331c37ede7ca4120ce91d9af2
6069 F20101119_AAAOJA jayaraman_p_Page_53thm.jpg
4e40f8a0f4f4cbc2890eed944616b7e8
46b21eee6bba7acddabe7e7104f5a0fa88046998
1777 F20101119_AAAOEC jayaraman_p_Page_21.txt
e1e3bd97abacbb468de45707806562b8
67a224dca9c6883270b1e2a7a4e44eb4daf88672
95561 F20101119_AAANXF jayaraman_p_Page_26.jp2
a516cf5a4edbf25b4c68c5e9a2d46fbc
a45a67191d979ef0ccd78ec8dfcbf9e2224ecc5b
1652 F20101119_AAANSI jayaraman_p_Page_34.txt
1b53cf8914ec5e3c6067f1f4bafadb3a
7344d0a29c3371a90ae62278df1f2714403ab803
21428 F20101119_AAAOJB jayaraman_p_Page_54.QC.jpg
973b5986d72260899efd3f785a0ef434
41cbd403a8f859044fe6b8a1920dbcab32b13ce6
1733 F20101119_AAAOED jayaraman_p_Page_22.txt
4e6cedea98dbc0104b40275d98c4dbba
c5b389a75b01e336e7cddfc9adc2a332ed3d2c38
97777 F20101119_AAANXG jayaraman_p_Page_28.jp2
ef5067ea873757d07f95d6c8b6c0a10d
254ce4589d76d8c8f001f204a61add959c0fb2ae
1793 F20101119_AAANSJ jayaraman_p_Page_43.txt
56491ff1c40e391d4b09a15bfd40f866
8197acd00b3beccbb1388255318d2d1554d3e3b6
8526 F20101119_AAAOJC jayaraman_p_Page_56.QC.jpg
5dfe8c09e6c9ac2782541543dbcbfdb9
76ca38c00583fb52cd176f9881fff059afbb02f6
1712 F20101119_AAAOEE jayaraman_p_Page_23.txt
19ef43a73de3e854b0ab5314e32b6de5
177e796b825eb243b3636b689c5fbf436b31abb7
105312 F20101119_AAANXH jayaraman_p_Page_29.jp2
10b23ef49a83b57d635b13b0aae673b7
3f93fc7711fbfe301c043c8d561126c36d37f23e
86497 F20101119_AAANSK jayaraman_p_Page_11.jp2
3d25e11c9dc1d2656a16b8787aa2e6a8
1e7e49e27bc50a984b137b1ef2e746c60cb11dbd
2768 F20101119_AAAOJD jayaraman_p_Page_56thm.jpg
c1963373e0398d6badd65765abbfb06e
6bf91e06e20e173e5b020c1353dec25d445c058f
1694 F20101119_AAAOEF jayaraman_p_Page_24.txt
4b46841112fe20a7fd15bab87cf125ff
f13660c5dc1980addb57634b20503fe21e59656c
42189 F20101119_AAANXI jayaraman_p_Page_30.jp2
592a51e1c902d6a7905b831625e2d004
e7fd2e6b5d5fb1ede78450cc5d17831759731f8e
36216 F20101119_AAANSL jayaraman_p_Page_58.jp2
dc6e912550443c63e5c03df658ce2b7b
fa41afe1546a71e0dbb6191b8ada33f4bb8cf0cf
3650 F20101119_AAAOJE jayaraman_p_Page_57thm.jpg
008987b1eedbf3d0fab08e0fa354d9ad
a15d7459cdd3653bb840cddc20026b74bcef1399
847 F20101119_AAAOEG jayaraman_p_Page_31.txt
f9f295578c352af737407e06587beb33
fc692425c86096666ca38a8ae0233be3be3aa856
34996 F20101119_AAANXJ jayaraman_p_Page_31.jp2
7cb1cd1eeff68a69b2e374d7e3b8a871
ba838cbb49694b2acb8cf4f0dcba0c5c2e53ceae
1872 F20101119_AAANSM jayaraman_p_Page_25.txt
c661819e000557b0618ad5cbad1f8542
948dd16a88d93d5f29bcf91a612456e4751a0e3b
11333 F20101119_AAAOJF jayaraman_p_Page_58.QC.jpg
36bc39fd63b012e673d99204a3c4a103
c0bd45b0d11e31f6cf8762c52db6b8d73d612f58
1643 F20101119_AAAOEH jayaraman_p_Page_32.txt
1277a55a8323b97141b705e387cb9059
50492fb614c135488effaacf0851c7637da4fd31
88890 F20101119_AAANXK jayaraman_p_Page_33.jp2
6cfeb4eb8c2e5061d27495524facdc54
6b75889101b1928039b8246ef64f2abb9e862f4c
57322 F20101119_AAANSN jayaraman_p_Page_13.jpg
8f919520e536e29b1b0b30d46bd33cd5
dd7d146eaf60e793a9d1e4862773c07209e223b2
3542 F20101119_AAAOJG jayaraman_p_Page_58thm.jpg
a6f859c946ea0c7fff69f49add5e013d
49271704608f6d9eb010249fcbeaaa28dc054e49
1625 F20101119_AAAOEI jayaraman_p_Page_33.txt
b43ab8b4daba9a1407b0acaf9a294cdd
070a169c7580e4d568880a6329885fbbd97c111b
90808 F20101119_AAANXL jayaraman_p_Page_34.jp2
aa5a79ee44dcc908437ab0c1b5080bcc
56231d036cede886c248e6c248907909bf0c9bf4
1447 F20101119_AAANSO jayaraman_p_Page_09.txt
ff3680a3d90aa68f38cf3e5e2862d494
3d3239f7f9a54bbffd132c7a9894ceeb35ff50fe
1827 F20101119_AAAOEJ jayaraman_p_Page_35.txt
64729b03ff703fd32d751a2791116ba5
eaf7c982ba0b39b6b351d08c1776ed1e954c177a
94196 F20101119_AAANXM jayaraman_p_Page_35.jp2
19c10bc077484e0eed4557f3dc101293
9f0bf3be5b29937b10e45376d90338c80c032962
576 F20101119_AAANSP jayaraman_p_Page_81.txt
b9dca6e28ba09dbaf5cea215a3dfa63f
2cad17d503cfc81ce51b1f293595b8cef74ceab3
17691 F20101119_AAAOJH jayaraman_p_Page_59.QC.jpg
f1147db5658b5e4193de3d582517a348
9fa110eb5055ca15f40608c12d1d3ed056efb9e0
1892 F20101119_AAAOEK jayaraman_p_Page_37.txt
a68590ff9ecfd5098cf948bd734eb2ae
6de007af3e9583aed266232f35c12bd02bd2e59c
107603 F20101119_AAANXN jayaraman_p_Page_36.jp2
a6f5ab2a322566569c4a5bbeff98ad00
3f28876dbf352d4744c7ca44401942959d161b7f
13197 F20101119_AAANSQ jayaraman_p_Page_17.QC.jpg
2ca92f2caf9bd272677ad8820257b8fc
da6900776459153f367436e242119a591e2d2a13
5388 F20101119_AAAOJI jayaraman_p_Page_59thm.jpg
066d7d2246e434ea686d62cb26115ec1
a8035f9e63f6ecb006974a07e3e4f70789f3be6f
1972 F20101119_AAAOEL jayaraman_p_Page_39.txt
25b1882baf7769638cf4dd54ca8acb29
0e578182c38b1b3302f75dd5fcbb27c16f4b2105
101256 F20101119_AAANXO jayaraman_p_Page_37.jp2
7129ed30e597f5794802dd9429e66277
f3df509e87c953ead5f3325bf9a01e81c2c5e519
21228 F20101119_AAAOJJ jayaraman_p_Page_60.QC.jpg
fdc4efa215dbcfde8d3635db6b89cd71
eca961706333b2bde098ff33dccc844421b9cf68
1957 F20101119_AAAOEM jayaraman_p_Page_40.txt
f20a9ba81529f72a9043b4a8e4195e76
3dce16b1929803fb3dfb9f1b53770458a9534934
98914 F20101119_AAANXP jayaraman_p_Page_39.jp2
2a0d79327ef2962d122f5f3245577d18
5ef3708364cc2caca483cb1aaa435b151b963740
99397 F20101119_AAANSR jayaraman_p_Page_53.jp2
58add7985c826f968a7decac8a319c98
8988be20f8745807026b6783c2ee52262101646d
6219 F20101119_AAAOJK jayaraman_p_Page_62thm.jpg
64f60cdd45d402aa3547ad557ad7b9e2
626b3500826773cf3e5fee3c7dcfebb6cab97051
1966 F20101119_AAAOEN jayaraman_p_Page_46.txt
b71f037025b14d6f7465da819b259949
deb3937da98f5df50a900ba2b7eefb926c0279a7
104929 F20101119_AAANXQ jayaraman_p_Page_40.jp2
9bc6e057a90ab681ea18a2bb88f1f74b
f38f2047d0ff0ce65155aeda04dd44fc9f11fb5c
40294 F20101119_AAANSS jayaraman_p_Page_10.pro
bfd4e60bd4cfbff4334b016099192186
fb0be289cc85a45b60aa1119b1f734a6e52aca67
22233 F20101119_AAAOJL jayaraman_p_Page_63.QC.jpg
2a369aa6f683b43e546ef0e3bdabb28f
f31ebc8f09d16eb9c87fb89df5bcb171a4a0ea48
1914 F20101119_AAAOEO jayaraman_p_Page_47.txt
39cf6b860b7b891a9d9254763af1ddb9
0cef1f2d27e263ae7e2a1a422daa6281d2e56278
98572 F20101119_AAANXR jayaraman_p_Page_42.jp2
0b1d5ce6c601775b75fbf16225bef8c6
629f1b58f6e5ceb3f813bc4cb8ac4759294900bc
6291 F20101119_AAANST jayaraman_p_Page_76thm.jpg
2ff250d2e8a5723a61667f5c23680d9b
41b3f4ac8c4cf823546e21136381b3f85061a821
6480 F20101119_AAAOJM jayaraman_p_Page_63thm.jpg
23b6901ca8e5beb5bce98dbe7a4f7b59
8233da5a6fb5021b3fac4b564648e2d35b4e8cda
1921 F20101119_AAAOEP jayaraman_p_Page_48.txt
9795ae4753fd18ddaafcf9fadb30c44c
d3da7a0be2aa9d0334117fc96a4ed43d6a42df4d
86973 F20101119_AAANXS jayaraman_p_Page_44.jp2
22b7f36c0d0c2fc8966c91cb50a3b3ef
92b7b685ae6c5aaedf0e9afd9b087b1598eb363e
1345 F20101119_AAANSU jayaraman_p_Page_03thm.jpg
d2ddea1069f741626359fe8d80beb23d
6c69e6cea6ac38c5288046e736f32cec56d6ab22
6141 F20101119_AAAOJN jayaraman_p_Page_64thm.jpg
9bc0a498e217769ff1be740fba5928ab
8194a4db66e0369f120b8d8d9284bfd0a1d31128
1897 F20101119_AAAOEQ jayaraman_p_Page_49.txt
81fa75887e16b7321b73a270b71a635c
e412b4a7dd976cfb096af68e0c18fbe981c8cf55
98582 F20101119_AAANXT jayaraman_p_Page_45.jp2
8935219baaad816df85f3a990fec7ce0
d7eed5df547262f1f50cfd73f17871018a1c485a
6371 F20101119_AAANSV jayaraman_p_Page_42thm.jpg
a73591b6b1d15efd220233fced8b3c2f
1c294488dc3725cb4b00c6e15c5b882a8dcc55f2
5700 F20101119_AAAOJO jayaraman_p_Page_65thm.jpg
ed88f499b192ed9eec7df3c241449402
ef75d3b521a985ab86a6c29c3d6f707055249fc6
1779 F20101119_AAAOER jayaraman_p_Page_50.txt
00dd7bed70fd2a46d9d3549e60aeee6f
8a0166e5a1bac3067159589fa7962c645289e931
100255 F20101119_AAANXU jayaraman_p_Page_47.jp2
a501acb655d73f3ff6319b85f1c9f6f9
2ee4546f0814cffcfcd14e071b905e27c6731a95
63313 F20101119_AAANSW jayaraman_p_Page_69.jpg
f31949b797ef38c3ecef815b743913d9
f7d6f7f112451c876eb024660eebaccbc599cb8a
20903 F20101119_AAAOJP jayaraman_p_Page_67.QC.jpg
6ecc8694b461c73c58ee659b490123f0
5ca3c7bd1506200ad69bd29546073915b88c3170
1742 F20101119_AAAOES jayaraman_p_Page_51.txt
933504d7df2f9e3d75952863cc2a3f91
f570ac622d73241530e2d1c2f29256933d5f4602
100059 F20101119_AAANXV jayaraman_p_Page_49.jp2
5eb015a41592f0b8d08121b964ba6625
64a47e2c5b7477201a293b9461b0178dc737bac1
58853 F20101119_AAANSX jayaraman_p_Page_10.jpg
8b0a037a9dff9c4a067f90f4ab358230
fc2596248b2c5844e6cefb29f77419abb7e94ebe
20631 F20101119_AAAOJQ jayaraman_p_Page_68.QC.jpg
ce0c6ed8eecfc35bb9a8f7e27ed1b080
bfcc6a19c1d4a8a97c8b420751ab1b394c783095
F20101119_AAAOET jayaraman_p_Page_53.txt
34ed6fa2900caaee97e5cea8d20d3e55
72350e323c1c08b9d3dec0b189210a11b3c2c035
100492 F20101119_AAANQA jayaraman_p_Page_48.jp2
678f4deeb545db1281abc33efad94201
995f53ea45cafefd076e058018d3b0bc7848f380
763 F20101119_AAANSY jayaraman_p_Page_30.txt
df8ff9dd15c5c74d97804c21936ceb82
c4ffcd2c9ef1c3c8246b572d6c1680b2a6337b8e
6433 F20101119_AAAOJR jayaraman_p_Page_68thm.jpg
d04d87318a1cd8bcb2f9f1448310aa69
ee5b1f60167d7e040c53d6313bb49deda2cb3955
1930 F20101119_AAAOEU jayaraman_p_Page_54.txt
a525aa4119d15ce28b7492a90f0ca9bc
0599d8f193d4b2b7660f45dfd27299bdc83024cc
92685 F20101119_AAANXW jayaraman_p_Page_50.jp2
2eedbc0f85be36ada8600cce6605a258
61eea4b63944fddb2e794ee8888b7472a81fbcc2
61636 F20101119_AAANQB jayaraman_p_Page_41.jp2
9178bf9bcca099e91586e20527cd578d
19208b5bcd78ff76616d0a13d0dfe6f6581735a3
10396 F20101119_AAANSZ jayaraman_p_Page_02.jpg
849a15295aaebbdb47212413bd591c20
41df5d43168c55b6d7a699f9feca26146b4e163d



PAGE 1

A DISTRIBUTED FILE SYSTEM (DFS) By PRASHANT JAYARAMAN A THESIS PRESENTED TO THE GRADUATE SCHOOL OF THE UNIVERSITY OF FLOR IDA IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF SCIENCE UNIVERSITY OF FLORIDA 2006

PAGE 2

Copyright 2006 by Prashant Jayaraman

PAGE 3

To my parents and to my brother Vivek.

PAGE 4

ACKNOWLEDGMENTS I thank Dr. Richard Newman for his help and guidance. I would like to express my gratitude to Dr. Randy Chow and Dr. Shigang Chen for serving on my supervisory committee and reviewing my work. I also acknowledge the contributions of all members of the Distributed Conferencing System group. Special thanks are due to my family and friends for their encouragement and support. iv

PAGE 5

TABLE OF CONTENTS page ACKNOWLEDGMENTS .................................................................................................iv LIST OF FIGURES .........................................................................................................viii LIST OF ABBREVIATIONS ............................................................................................ix ABSTRACT .....................................................................................................................xiii CHAPTER 1 INTRODUCTION...................................................................................................1 Introduction..............................................................................................................1 Distributed Conferencing System Services.............................................................2 2 STUDY OF DISTRIBUTED FILE SYSTEMS......................................................4 Introduction..............................................................................................................4 Network File System (NFS).....................................................................................4 Introduction..................................................................................................4 Protocol........................................................................................................4 Naming and Transparency...........................................................................5 Concurrency Control and Consistency........................................................6 Replication and Migration...........................................................................6 Caching........................................................................................................7 Security........................................................................................................7 Andrew File System (AFS)......................................................................................8 Introduction..................................................................................................8 Architecture..................................................................................................8 Naming and Transparency...........................................................................9 Concurrency Control and Consistency........................................................9 Replication and Migration...........................................................................9 Caching......................................................................................................10 Security......................................................................................................10 Common Internet File System (CIFS)...................................................................11 Coda ....................................................................................................................11 Plan 9 ....................................................................................................................12 xFS ....................................................................................................................13 v

PAGE 6

Low Bandwidth File System (LBFS)....................................................................14 Freenet....................................................................................................................15 3 REQUIREMENTS AND DESIGN.......................................................................18 Requirements.........................................................................................................18 Naming and Transparency.........................................................................18 Concurrency Control and Consistency......................................................19 Replication and Migration.........................................................................19 Caching......................................................................................................20 Security......................................................................................................20 Design ....................................................................................................................21 Communication Architecture.....................................................................21 Stateful/Stateless Server.............................................................................21 File Access Model......................................................................................21 Concurrency Control and Consistency......................................................22 Versioning..................................................................................................23 Forward deltas................................................................................23 Reverse deltas................................................................................23 Previous design..............................................................................24 Current design................................................................................24 Naming and Transparency.........................................................................26 Storing Metadata........................................................................................26 Symbolic links...............................................................................28 File Information Table (FIT).........................................................28 Previous design..............................................................................29 Proposed design.............................................................................30 Cog Policy..................................................................................................33 User Policy.................................................................................................34 Migration....................................................................................................35 Replication.................................................................................................36 Replicate entire file........................................................................36 Replicate portions of file................................................................36 Replicate on create.........................................................................36 Replicate on open...........................................................................37 Design decision..............................................................................37 Caching......................................................................................................38 Client caching................................................................................38 Server caching................................................................................38 Security......................................................................................................39 File System Interface.................................................................................39 Conclusion.............................................................................................................41 vi

PAGE 7

4 IMPLEMENTATION............................................................................................45 Requirements.........................................................................................................45 Interaction with Distributed Conferencing System Services.................................45 Interaction with Conference Control Service............................................45 Interaction with Access Control Service....................................................46 Interaction with Secure Communication Service......................................46 Communication......................................................................................................46 Distributed File System Architecture.....................................................................47 File Information Table Manager (FITM)...............................................................47 Version Naming Manager (VNM).........................................................................48 Cog Policy Manager (CPM)..................................................................................49 User Policy Manager (UPM).................................................................................49 Version Control Manager (VCM)..........................................................................50 Creating a Delta.........................................................................................51 Recreating a Version..................................................................................52 Replication/Migration Manager (RMM)...............................................................52 Local Filestore Manager (LFM)............................................................................53 Distributed File System Server..............................................................................53 Client-Server Interface...............................................................................54 Server-Server Interface..............................................................................55 Distributed File System Client...............................................................................55 Applications...........................................................................................................56 Shell ........................................................................................................56 Distributed Conferencing System Text Editor...........................................57 5 EVALUATION......................................................................................................60 Testing....................................................................................................................60 Evaluation..............................................................................................................60 Future Work...........................................................................................................62 Conclusion.............................................................................................................62 LIST OF REFERENCES...................................................................................................64 BIOGRAPHICAL SKETCH.............................................................................................67 vii

PAGE 8

LIST OF FIGURES Figure page 2-1. Example of NFS clients mounting the same directories at different mount points..17 3-1. Forward deltas..........................................................................................................42 3-2. Reverse deltas...........................................................................................................42 3-3. Actual location of files in a conference....................................................................43 3-4. Virtual view of conference from site 1.....................................................................43 3-5. Sample FIT...............................................................................................................44 4-1. DFS architecture.......................................................................................................58 4-2. Version tree in DFS..................................................................................................59 viii

PAGE 9

LIST OF ABBREVIATIONS Causally related: Two events are causally related if the order of their occurrence affects their result. Causal consistency: A model specifying that operations that are potentially causally related must be seen by everyone in the same order. The order of other operations is unimportant and they are assumed to be concurrent. Client: Process that accesses a (remote) service on another computer through a network A user uses a client to access the services provided by the servers. Coherence: A property of distributed systems in which the file is consistent across all its replicas. The consistency is defined by the consistency model that is used. Concurrency control: The manner in which concurrent access/modification of objects is handled. Time stamping, locking and optimistic concurrency control are forms of concurrency control. Consistency: A system is consistent if operations on objects follow specific rules. The system guarantees that upon following the rules, the objects will be consistent and the results will be predictable. Consistency model: The set of rules that specify when an object can be classified as consistent. DCS aware application: An application that can take advantage of DCS services. It is designed specifically with DCS in mind. ix

PAGE 10

DCS conference: Set of users, with a set of roles, a set of objects, and a set of applications. Typically a conference has one specific purpose or goal and the members of the conference work together using the objects and applications to achieve this goal. A conference can also have one or more sub-conferences that are used to accomplish a specific aspect of the larger conferences goal. A conference may be spread over several sites. Each site contributes resources to the conference. DCS file space: File space managed by Distributed File System presented in this thesis. It is only accessible through the services provided by this module. DCS instance: A collection of sites and conferences. DCS server: Server modules of the various DCS services. A site may run several servers belonging to different DCS conferences. DCS site: Collection of servers running DCS services. A site may be a part of several conferences. DCS sites are composed of a Local Area Network (LAN) and have a shared file system. The communication within a site is assumed to be reliable and highly efficient. There is also the notion of correlated accessibility, which means that the accessibility of the servers and objects on a site are related. If one is inaccessible, there is a great probability that all will be inaccessible. DCS unaware application: Any application that is not DCS aware. Almost all normal applications fall in this category. Delta: A file containing the difference between one file and other. It is also called a diff. x

PAGE 11

Delta compression: A way of storing data in form of differences between sequential data rather than as complete files. A file is stored as the difference between itself and a base file. Using the base file and the difference, the given file can be rebuilt. Immutable semantics: A model in which all objects are immutable. Any change to an object is stored as a new object. Location independence: A resource can be moved about the system without changing its name. Location transparency: Names used to identify resources do not reveal the location of the user or the resource. Object: Any file that exists within DCS and is managed by DFS. Role: A named set of capabilities. Roles can also be viewed as a collection of job functions. Each role has a set of actions it can take on an object. A user that is bound to a role can perform the set of actions available for that role on a specific object. Session semantics: A consistency model that dictates that changes to a file are visible only after the process closes the file. Stateful server: A server that maintains information about its clients. It is more efficient than a stateless server. Stateless server: A Server that does not store information about its clients. This allows for a very robust design. Transparency: Refers to hiding irrelevant system dependent details from the user. Uniform name space: Everybody invokes a resource using the same name. In the context of DFS, this means that every client uses the same path to access the file. xi

PAGE 12

Unix semantics: A model specifying that operations on a file are immediately visible to all users/processes. User: An entity (usually human) in the system that performs actions on objects through services provide by the servers. User file space: File space that exists on the users local system. It is not managed by DCS. However, a DCS client often acts as a go-between. Version chain: A sequence of file versions in which, the first version is stored as a full file but all others are stored as deltas. xii

PAGE 13

Abstract of Thesis Presented to the Graduate School of the University of Florida in Partial Fulfillment of the Requirements for the Degree of Master of Science A DISTRIBUTED FILE SYSTEM (DFS) By Prashant Jayaraman May 2006 Chair: Richard E. Newman Major Department: Computer and Information Science and Engineering Distributed Conferencing System (DCS) is a platform that supports collaboration in a distributed environment. Distributed File System (DFS) provides the file system of DCS. Distributed file systems face some peculiar challenges. They must provide a uniform name space for users, along with location independence and location transparency. A file can be edited simultaneously at different locations. If the changes are not coordinated, the file may become inconsistent and the resulting unpredictability can be frustrating to users. An important characteristic of a distributed system is that at almost all times, some node or another will be unavailable. DFS must ensure that the system is reliable and highly fault-tolerant. Currently available distributed file systems cannot support DCS. They are not designed for an environment where a high level of concurrency is common. They assume that simultaneous writes are unlikely and do not support an adequate level of concurrency control. Most file systems make use of primary copy replication which has a large overhead. Others use read-only replication. xiii

PAGE 14

DFS supports a high level of concurrency. It uses an immutable versioning scheme where, every time a file is closed, it is saved as a new version of the file. This allows multiple users to edit and save the file simultaneously, without the danger of interleaving file blocks. At the same time, it makes sure that no changes are lost. DFS provides fault tolerance and ensures high reliability by replicating files across multiple sites. In addition, it supports file migration. The immutable versioning scheme also gets rid of the problem of synchronizing replicas. xiv

PAGE 15

CHAPTER 1 INTRODUCTION Introduction In the modern world, there is a growing need for collaboration among geographically separated groups. Platforms and services that assist such collaboration are in great demand. Distributed Conferencing System (DCS) is an effort to provide a platform for users to work cooperatively in real time in a distributed environment. The system is based on a robust, hierarchically distributed architecture and provides access control, file service, notification, secure communication and mechanisms for group decision-making [1]. DCS provides basic distributed collaborative tools (text editors, graphics editors, etc) and can also support third-party applications. The architecture of DCS is designed to be scalable. Another major design goal is fault-tolerance. The system is designed to handle network failures and system crashes by restoring from backups and contacting other sites for the most up-to-date information. File management is important in any distributed system. File systems in distributed environments have to tackle issues not seen elsewhere. A key question is how are files and directories presented to the user [2]? Another question is what happens when several users modify the same file at the same time [2]. It is also important to ensure that data is not rendered unavailable by the failure of a few systems [3]. File permissions are also significantly more complicated [2]. Finally, performance should be one of the goals of a distributed file system [2]. 1

PAGE 16

2 Distributed File System (DFS) provides file-handling services for DCS users. It is designed to allow file sharing and concurrent access of files. It also provides transparency [2]. In addition, it uses the Access Control Service provided by DCS to enforce file access permissions [4]. Distributed Conferencing System Services The first version of DCS (DCS v1) provided only limited services for focused tasks. One of its drawbacks was that it had very few roles and only users in the voter role could vote on decisions [1]. Also, it relied on UNIX permissions for security [5]. The second version (DCS v2) provides more services and better communication primitives. In addition, it supports more voting mechanism and limited Role Based Access Control (RBAC) [4]. DCS v2 modules have better support for file/object types. The major services provided by DCS are described below [6]. Conference Control Service (CCS): This module manages the Collaborative Group (CoG). It is the first service to be activated and instantiates all other modules. It uses secure messaging to allow users to login and interact with services. This module also handles operations like splitting and merging instances/CoGs/sites, access control requests and add user requests. It is responsible for creating and deleting CoGs. Database Service (DBS): DBS maintains all tables in DCS space. It makes use of a Database Managements System (DBMS) as the backend. Tables are stored as partially replicated, distributed databases and group multicast is used to ensure eventual consistency. Notification Service (NTF): NTF provides asynchronous event notification to registered users. In addition to built-in events, NTF allows users to define new events. NTF maintains a global and local database to match events to registered users, along with the delivery method. Decision Support Service (DSS): DSS facilitates the resolution of issues by a group of people with the joint responsibility for them. It maintains decision templates for that purpose. It allows creation, modification and execution of templates. If a template requires a vote among a set of users, DSS will contact the users, get their votes and return the result.

PAGE 17

3 Access Control Service (ACS): ACS is responsible for maintaining access rights and protection information for all objects in DCS. Instead of just maintaining attributes, ACS supports the use of decision templates to allow members to decide on actions. Applications can register new access types for more fine grained control. ACS makes use of DBS to maintain state. Secure Communication Service (SCS): SCS allows DCS and its users to communicate securely and reliably. It ensures authenticity, confidentiality and integrity of all messages. It is responsible for the creation and maintenance of keys and certificates for sites and users. Application Manager (AM): The Application Manager is responsible for registering and invoking applications that are available for each CoG. The applications could be DCS aware or DCS unaware. AM maintains a list of applications for each CoG and makes them available to the user. Distributed File System (DFS): This is the focus of the thesis. It manages all files in the CoG. It provides concurrency control using a versioning scheme with immutable files. It also provides replication and migration services.

PAGE 18

CHAPTER 2 STUDY OF DISTRIBUTED FILE SYSTEMS Introduction This chapter surveys some of the popular distributed file systems. A distributed file system is a file system that supports sharing of files and resources in the form of persistent storage over a network [2]. The first distributed file systems were developed in the 1970s. The most popular distributed file systems are Suns Network File System (NFS) and the Andrew File System (AFS). In addition to a discussion on these file systems, this chapter will also provide an overview of some less popular systems. Network File System (NFS) Introduction The NFS protocol was developed by Sun Microsystems in 1984 [7]. It was meant to be a distributed file system which allows a computer to access files stored remotely, in the same manner as a file on the local hard drive. NFS is built on the Open Network Computing Remote Procedure Call (ONC RPC), described in RFC 1831 [8]. Version 2 of NFS was stateless and ran over UDP [9]. Version 3 introduced support for TCP [10]. Finally, version 4 introduced a stateful protocol [11, 12]. Though NFS is strongly associated with UNIX, it can be used on Windows, Mac and Novell Netware. Protocol NFS provides its services through a client-server relationship. The client and server communicate through RPCs. The client issues an RPC request for information and the server replies with the result. NFS was designed to be machine, operating system, 4

PAGE 19

5 network architecture and protocol independent. External Data Representation (XDR) is used to translate between machines with different byte representations. The computers that make their file systems, or directories, and other resources available for remote access are called servers. The act of making file systems available is called exporting. The computers, or the processes they run, that use a server's resources are considered clients. Once a client mounts a file system that a server exports, the client can access the individual server files (access to exported directories can be restricted to specific clients). It must be noted that a computer can be both a server and a client at the same time. An NFS server is stateless. It does not have to remember any transaction information about its clients. In other words, NFS transactions are atomic and a single NFS transaction corresponds to a single, complete file operation. NFS requires the client to remember any information needed for later NFS use. This has meant that NFS has poor performance over Wide Area Networks (WAN). However, Version 4 has introduced stateful protocols and compound procedures to improve this situation [2]. Naming and Transparency A client can mount the exported directory (from the server) anywhere on its local file system. The purpose of this is to provide clients transparent access to the servers file system. Figure 2-1 illustrates the fact that different clients can mount the exported directories at different locations. In this case, the client Client 1, mounts the directory Dir A (from Server 1) at /mnt and the directory Dir B (from Server 2) at /mnt/remote. By contrast, Client 2 mounts Dir B at /mnt and Dir A at /mnt/remote. An NFS server can mount directories that are exported by other NFS servers. However, it cannot export these to its clients. The client will have explicitly mount it

PAGE 20

6 from the other server. On the other hand, a file system can be mounted over another file system that is not a local one, but rather is a remotely mounted one. NFS supports location transparency but not location independence. This is due to its decision not to enforce a common namespace. This makes it harder to share files. Concurrency Control and Consistency The NFS protocol does not provide concurrency control mechanisms. A single read or write call may be broken down into several RPC read or writes because each RPC call can contain up to 8KB of data. As a result, two clients writing to the same remote file may get their data intermixed. It is hard to characterize the consistency semantics of NFS. New files created on a machine may not be visible anywhere for half a minute. It is indeterminate whether writes to a file are visible to other sites that have the file open for reading. New opens of a file observe only the changes that have been flushed to the server. Thus, NFS does not provide either UNIX semantics or session semantics. Replication and Migration NFS version 4 provides limited support for file replication. It requires that an entire file system be replicated, including the files, directories and attributes. One of the attributes of a file, contains the list of alternate locations (DNS name or IP address) for the file system containing the file. In order to access files from a server, the client must switch over to that server. It must be noted that the NFS protocol does not specify how replicas are kept up-to-date, each implementation must decide for itself on the best manner to achieve this.

PAGE 21

7 As explained previously, NFS does not provide location independence. This means that migrating a file system from one server to another would invalidate all mounts of that file system. Caching NFS version 3 mainly left caching out of the protocol specification. Most implementations never guaranteed consistency. The time lag, between the time a file was written to the server, and the time the file was invalidated on the client cache varied from a few seconds to almost half a minute. NFS version 4 does try to alleviate this problem. Most clients cache file data and attributes. On a file open, the client contacts the server to check whether the attributes need to be fetched or revalidated. The cached file blocks are used only if the corresponding cached attributes are up-to-date. Cached attributes are generally discarded after 60 seconds. NFS uses both read ahead caching and delayed write, to improve performance. Security Security, in distributed file systems has two parts secure communication between the computers and controlling access to files. NFS is built on top of RPCs. Hence, secure communication boils down to the establishment of a secure RPC. Older versions of NFS had support for Diffie-Hellman key exchange for establishment of a secure key. Recent versions support Kerberos (version 4) for establishing secure RPCs. NFS version 4 also provides a general framework that can support numerous security mechanisms to ensure authenticity, integrity and confidentiality. NFS provides a much richer set of access controls than the UNIX file system. Access control is implemented in the form of access control entries, where each entry

PAGE 22

8 specifies the rights for a user or a group of users. Unlike UNIX, it is possible to specify the rights of several users and groups. Andrew File System (AFS) Introduction The Andrew File System (AFS) was initially developed by Carnegie Mellon University (CMU) as part of the Andrew Project [13, 14, 15]. It is named for Andrew Carnegie and Andrew Mellon, the founders of CMU. Its biggest advantages are security and scalability. AFS has expanded to over 1000 servers and 20000 clients worldwide [16]. It is believed that AFS can scale up to support twice as many clients. AFS heavily influenced Version 4 of Sun Microsystems NFS. Additionally, a variant of AFS was adopted by the Open Software Foundation in 1989 as part of their distributed computing environment. There are three major implementations of AFS Transarc (IBM), OpenAFS and Arla. AFS is also the predecessor of the Coda file system. Architecture Like NFS, AFS uses RPCs as the basis of communication between clients and servers. AFS can be divided into two components. The first is a group of relatively small number of centrally administered file servers, called Vice and the second consists of a large number of workstations called Virtue. Virtue workstations provide access to the file system. Every client (Virtue workstation) has a process running on it, called Venus. This allows the client to access the shared file system in the same manner as the local one. AFS defines a volume as a tree of files and subdirectories. Volumes can be created by administrators and linked to specific paths (similar to mount in NFS). Volumes can be

PAGE 23

9 located anywhere and replicated or moved without informing the user. In fact, a volume can be moved while its files are in use. Naming and Transparency AFS provides its clients with a global name space. In addition, clients have a local name space to store temporary data. From the clients point of view, the file system appears as one global resource. It is not possible to deduce the location of the file from its name. In addition, all files have the same path from every client. Thus, AFS provides both location transparency and location independence. Concurrency Control and Consistency Like NFS, AFS uses the Ostrich approach (it is widely and incorrectly believed that when threatened, the Ostrich buries its head in the sand, hoping that its attacker would go away). It does not make any effort to resolve concurrency, believing that shared writes are extremely rare. When two users close the same file at the same time, the results are unpredictable. AFS supports session semantics. This means that once client A has saved the file, any client opening the file after A has closed it, will see the modifications made by A. This is weaker than UNIX semantics but much easier to support in a distributed environment. Replication and Migration AFS provides replication to read-only backup copies. Even an entire volume can be replicated. A client system looks to retrieve the read-only file from a particular copy. In case the copy is not available, it switches to a different copy. This allows both better fault tolerance and improved performance. AFS allows the creation and migration of copies as

PAGE 24

10 needed. By ensuring that the copies are read-only, replication consistency problems are avoided. Caching AFS divides files into chunks of up to 64 KB, for caching. A large file may have several chunks, while a small one may occupy just one. The cache tries to establish a working set of files (copied from the server) on the client. In case of server failures, the local copy is still accessible. Read and write operations are performed on the local, cached copy. When the file is closed, the modified chunks are copied back to the file server. Callback is a mechanism used to maintain cache consistency. When the client caches a file, the server makes a note of it (stateful) and informs the client when a change is made to the file. The cache is believed to be consistent while the callback mechanism is operational. Callbacks must be reestablished after client, server and network failures. Security Security is one of the strengths of AFS. It uses a secret key cryptosystem to establish a secure channel between Vice and Virtue. The secret key is used by the two machines to establish a session key, which is used to setup the secure RPC. The process of authenticating the user is more complex. The protocol used for this purpose, is derived from the Needham-Schroeder protocol [17]. An Authentication Server (AS) provides the user with appropriate tokens. These are used to establish his/her identity. Access control lists are used to control permissions to files and directories. The Vice file servers associate lists with directories only. Files do not have access control

PAGE 25

11 lists. This makes it simpler and more scalable. AFS supports both users and groups. It is also possible to list negative rights. Common Internet File System (CIFS) In its original incarnation as Server Message Block (SMB), it was invented by IBM to provide a genuine networked file system. Microsoft modified it heavily, adding support for symbolic links, hard links and large files and launched an initiative to rename it as CIFS. CIFS can run on top of both NetBIOS and TCP/IP protocols. In the manner of other distributed file systems, CIFS uses a client-server model for communication. In contrast to other file systems discussed, CIFS also provides Inter-process communication (named pipes). In addition, SMB servers can share other resources besides file systems. CIFS cannot really be described as a legitimate distributed system. It merely provides a way to access resources over a network. It does not provide either location transparency or location independence. Resources are identified as //server-name/path-to-resource. Unlike other distributed file systems, it makes little effort to remain agnostic of the underlying operating system. There is no concurrency control or caching. It does not support file replication or migration. It does provide UNIX consistency semantics because the file has only one copy. Coda Coda is another distributed file system, developed at Carnegie Mellon University [18]. It is based on the Andrew File System. Some of the areas that it improves over AFS are listed below. In contrast to almost all other distributed file systems, Coda allows the client to work, even when it is unable to contact any server. In this case, the client uses the local

PAGE 26

12 (cached) copy of the file. It is possible to ensure that the client fills its cache with the appropriate files before disconnecting from the server. This is called hoarding. Replicated volumes do not have to be read-only. Coda uses a replicated-write protocol to ensure that the modified parts are transferred to all servers at the same time. In case server failures, different servers will have different versions of the file. Version vectors (similar to vector timestamps) are used to resolve this inconsistency. The security mechanisms of Coda are interoperable with Kerberos. Also, it has native support in the Linux 2.6 kernel. Plan 9 Plan 9 is based on the idea of using a few centralized servers and numerous clients [19]. Plans 9 allows computer to share any resource in the form of files. For example, the network interfaces of a machine can be exported as a bunch of directories. Each TCP connection is represented as a subdirectory of the corresponding network interface. Plan 9's naming philosophy is similar to that of NFS with some interesting deviations. Multiple name spaces can be mounted at the same mount point. In this case, the contents of the two remote file systems are Boolean ORed. Such a mount point is called a union directory. Plan 9 implements UNIX file semantics by ensuring that all updates are always sent to the server. Concurrent writes will happen in an order determined by the server but unlike session semantics, the writes will never be lost. However, this reduces performance. All files have four integers identified with them. Two of these type and device number, identify the server or device hosting the file. Two others, path and version,

PAGE 27

13 identify the file relative to the server or device. Version is incremented after every modification. Plan 9 provides some support for replication and caching. Clients can cache files and use their version numbers to ensure that the files have not been modified. Users are authenticated using a protocol similar to the Needham-Schroeder protocol. File permissions are identical to those in UNIX. Plan 9 has the notion of group leaders. Only group leader can modify the groups permissions on the file. xFS xFS is an unusual serverless file system [20, 21], designed to operate on a local area network. It distributes storage, cache and control over cooperating workstations. This arrangement is believed to yield higher performance, reliability and scalability. xFs defines three types of processes. Storage servers stored parts of data. Metadata managers store information on locating data and clients form the interface between users and the file system. The storage servers together, implement a virtual RAID device. xFS uses system wide identifiers to locate metadata managers. Clients cache blocks of data rather than files and the cache consistency scheme is similar to AFS. Each data block belongs to a manager. When a client wishes to modify a block, it obtains write permission from the manager. The manager invalidates all other cached copies of the file. Write permission is revoked when another client wants to modify it. This also requires that the block be flushed and written. xFS does not emphasize security. It provides the usual access control lists and requires that clients, managers and storage servers run on trusted machines.

PAGE 28

14 Low Bandwidth File System (LBFS) LBFS is designed for networks with low bandwidth and high latencies [22]. LBFS works on the concept that a version of a file has much in common with its previous version. It also believes that there are similarities between files created by the same application. The techniques used by LBFS can be used in conjunction with the techniques used by other distributed file systems (like Coda) to improve tolerance to network vagaries. LBFS works by maintaining a large cache at both the client and the server. Files are divided into variable size chunks. The chunk boundaries are determined by Rabin fingerprints [23], subject to lower and upper bounds. A Rabin fingerprint is the polynomial representation of the data modulo a predetermined irreducible polynomial. When the low-order 13 bits of a region's fingerprint equal a chosen value, the region constitutes a breakpoint. Assuming random data, the expected chunk size is 2 l3 = 8192 = 8 KBytes This scheme ensures that modifications to a chunk affect only that chunk and its neighbours. LBFS indexes chunks using their SHA-1 hash [24]. In case the chunk is present in both the server and the client, only its hash is sent. Otherwise, the entire block is transmitted. All data sent on the network is compressed using gzip. LBFS provides the same session semantics as AFS. Files are flushed to the server on close. LBFS uses temporary files to implement atomic updates. All writes to the file happen on a temporary file which is then renamed. This ensures that concurrent writes can never result in inconsistent files. This is in marked contrast to NFS. The client that closed the file last will overwrite the changes of the others.

PAGE 29

15 Every server has a public key, which is specified on the client when it mounts the server. Future versions look to incorporate self authenticating pathnames [25], to facilitate auto-mounting. All communication between the client and the server is encrypted with the help of a session key. Freenet Freenet is a decentralized, anonymous, censorship resistant, distributed data store [26]. Unlike most distributed file systems, it is based on the peer to peer communication model. Freenet allows all participating computers to contribute bandwidth and storage space. It allows users to publish and retrieve data anonymously. Files have a descriptive string associated with them. This yields a public/private key pair. The private key is used to sign the file and the hash of the public key is called the Keyword Signed Key (KSK). Signed Subspace Key (SSK) allows the use of personal namespaces and is created in a manner similar to KSK, using usernames instead of file descriptions. A combination of the username and descriptive string is sufficient to access the file. Content Hashed Keys (CHK) are created by hashing the contents of a file. A Content Hashed Key can also be used to access a file. A CHK points directly to a file, but the KSK and SSK combination uses an indirection mechanism. Freenet uses a routing table similar to the Internet Protocol. The client retrieves the file by specifying its binary key. It then checks whether it has knowledge of the node that should be contacted to get this file. If no node is found, it looks up the nearest key (lexicographic) in its table and forwards the request to that node. This proceeds until the key is located or the TTL expires. In order to preserve anonymity, nodes along the successful path randomly designate themselves as sources of the file and may cache a copy of the file.

PAGE 30

16 To insert a file, the user computes a binary key for the file and makes a request for a file with that key. In case there is no collision, the file is inserted into the node and all nodes along the path of the previous query are informed of the new key. Again, nodes can randomly re-designate themselves as hosts. Each file in Freenet is encrypted using a random key that is published along with the binary key. The routing heuristic and insertion algorithm ensure that files with similar keys are clustered. This also ensures better performance of both of them. Freenet is closer to an anonymous bulletin board that a distributed file system. It has been included here because of its unusual design and goals.

PAGE 31

17 Server 1 Server 2 D1 D2 Dir A D3 Dir B Client 1 Client 2 remote Dir A D1 D2 D3 Dir B mnt mnt remote D3 Dir B D1 D2 Dir A Figure 2-1. Example of NFS clients mounting the same directories at different mount points.

PAGE 32

CHAPTER 3 REQUIREMENTS AND DESIGN This chapter presents the requirements of Distributed File System (DFS) in detail and goes on to describe the design of DFS. In each section, it details the approach taken in DFS along with the reasons for it. Requirements DCS is designed to simplify collaboration among geographically separated users. DFS provides the underlying file system for DCS. Other modules of DCS interact with DFS to provide the complete user experience. So, the requirements were formed on the basis of discussion among the members of the DCS Research Group Naming and Transparency DCS users can reside in different locations and the file system should allow users to share files. This requires a conference-wide shared name space. In addition, clients and servers will often store data related to them. This requires that there be a local name space, in addition to the conference-wide name space. So, DFS must provide both local and conference-wide name spaces. Users join DCS conferences using Conference Control Services (CCS) [27]. It is possible for a user to join multiple conferences through CCS. DFS must present the user with a hierarchical view of the file system. DCS conferences span several sites. It is very likely that files are spread across multiple sites. DFS should ensure that the user does not have to inform it about the location of the file. In other words, filenames should be location transparent. 18

PAGE 33

19 The name of the file should not depend on the client that is being used to access the file. This ensures that users have the freedom to connect from any client and that they dont have to remember the mount points of that client. Concurrency Control and Consistency Concurrency control is very important in a distributed file system. DFS should be designed to allow concurrent access as best as possible. Users must be able to access and write files concurrently. It is up to DFS to ensure that changes made by one user do not interfere with the changes made by others. Most distributed file systems provide an approximation of session semantics. It would be highly desirable for DFS to provide the same consistency semantics. New files should be seen when the client closes the file. Replication and Migration Fault tolerance and high availability are two of the goals of DFS. In order to achieve these goals, DFS must provide file replication. If a site is unavailable, DFS should be able to retrieve the file from its replicas. DFS should also enforce replica transparency [28]. This means that normally, the client should be unaware of the existence of replicas. DCS is designed to spread across Wide Area Networks (WANs) and different administrative domains. Replicating a file on a server close to the client can decrease access times dramatically. Migration also helps DFS improve performance and availability. DFS should allow migration from far away or unreliable servers, to closer, more reliable servers.

PAGE 34

20 Caching As mentioned previously, DCS is designed for WANs. This introduces the problem of network latency. DFS must use caching aggressively to reduce access times. DFS should also be aware of the fact that while servers are connected through WANs, the connection between a client and a server is often over a Local Area Network (LAN). If both the server and the client maintain a working set of the files that they use, it is possible to reduce the traffic over WANs. Security DCS is spread across multiple administrative domains. DFS must ensure that it makes it as hard as possible for an intruder to play havoc with the system. This requires that all communication between the client and the server (and between the servers themselves) be protected. Authentication, integrity verification and confidentiality must be provided by the communication service. Another part of security is to regulate the behaviour of legitimate DCS users. DFS must have very flexible and robust permission schemes. In addition, it should allow users to create new permission types. This ensures that DFS is able to handle scenarios that were not envisaged during design and development. As described in Vijay Manians work on DCS access control, clients are bound to roles [4]. Each role has specific privileges. One of the desired privileges should be viewing the contents of a directory. Only those files, for which the user has the privileges, should be shown in the directory listing. This can be used to give each a user a different view of the same directory.

PAGE 35

21 Design Communication Architecture The design of DCS was made around the idea of separate clients and servers. DFS, as a module of DCS has to maintain that architecture. DFS has dedicated servers for file management and file services. Client processes run on client machines and pass on requests to DFS servers. It must be noted that DCS does not preclude the client and server being on the same machine and DFS must not impose this requirement. It only requires that the client and server be different processes. DFS servers and clients must be able to run on the same machines as the clients and servers of other DCS modules. The current design of DFS allows a site to run multiple instances of DFS, one for each conference that it participates in. It also allows servers to connect to a cog on the fly. Stateful/Stateless Server As previously mentioned, stateless servers are simpler to design and have better fault tolerance as compared to stateful servers. They can be easily plugged in and taken out of a network. Stateful servers are often shutdown in a dirty state and must restore themselves to a consistent state. For these reasons, DFS servers do not maintain state information. File Access Model File Access Model refers to the manner in which the file system lets the client access the file. Distributed file systems offer one of the following file access models Remote access: In this model, all operations are performed directly on the remote file. NFS is one file system that offers this type of access. Upload/download: All files are downloaded by the client when opened and are uploaded back to the server when the client closes the file.

PAGE 36

22 The upload/download model lends itself more easily to caching and is quite popular among distributed file systems. DFS uses the upload/download model. All files are downloaded to the client for use and are uploaded back to the server when they are saved or closed. Concurrency Control and Consistency Concurrency is very important in a distributed file system like DFS, which is meant for Wide Area Networks (WANs). NFS does not deal with concurrency at all and other file systems like AFS and Coda have inadequate concurrency control. DFS manages concurrency with the help of an immutable versioned file system. All files are immutable. Changes made to a file are stored as a new version of the file. This ensures that concurrent writes by multiple clients do not interleave and make the file inconsistent. Another part of concurrency control is to ensure that writes by all clients are incorporated. When multiple clients open a file and write to it, it should not be the case that only the changes made by the user who closed the file last, are incorporated while all others are lost. This is tacked by the versioning system. Every time a change is made to a file and stored, DFS stores the modified file as a new version. Changes made to the file by different users are stored as different versions. Eventually, DFS will offer tools for automatic merging and conflict resolution. Versioning in DFS is different from versioning systems such as CVS and Perforce. Versioning systems are designed to function as depots (usually code). They do not provide transparent access. All files must be explicitly checked out and check in. They are not optimized for the high number of requests that a file system must handle. In DFS, for a user to see changes made by others, the user has to specify the version, after it has been saved by the other user. This peculiarity of DFS makes it harder

PAGE 37

23 to evaluate its consistency semantics. Other immutable file systems do not maintain a versioning scheme and so, they seem to follow session semantics. In DFS, if the version is specified properly, it is possible to look at the changes made by the other user. Thus, from the users point of view, the file system follows session semantics. Versioning It has been discovered with versioning systems that consecutive versions of a file have a lot in common. It is more economical to store just the changes between consecutive versions, rather than the versions themselves. All versioning systems maintain their files as deltas. The delta of two files is the set of changes which, when applied to the first file, yield the second file. The use of a delta scheme will greatly reduce the storage demands of DFS. A delta compression scheme could use forward or reverse deltas. These are discussed next. At the end of the section, the final design decision is presented with reasons why it is chosen. Forward deltas If forward deltas are stored, only the original file is stored in its entirety. All other versions are constructed from this version using a string of deltas. The major disadvantage of this scheme is that every time the latest version is needed, it must be reconstructed step by step from all the deltas. This makes the retrieval of new files painfully slow. The SCCS versioning system uses this approach. Reverse deltas In a reverse delta, the latest version is stored in its entirety and all other versions are stored as deltas. The older versions can be recreated by applying the deltas on the latest version. This scheme requires that the latest version of each branch be stored as a complete file. There can be multiple ways to generate a file and the paths might be of

PAGE 38

24 different lengths. This scheme uses storage space less efficiently than forward deltas. However, retrieval of the latest version is quick. Previous design Previous versions of DFS used a combination of forward and reverse deltas. The latest version on the main branch was kept as a complete file. All other versions could be obtained from it. Other versions on the main branch were stored as reverse deltas of their subsequent versions. In case of branches in the versioning tree, forward deltas were used. This scheme allowed the system to quickly retrieve the latest version of the main branch. It also reduced storage space by storing only one full version. On the other hand, there were a few problems with the scheme. It was hard to define which branch was the main branch. Also, this scheme made file replication and migration very complex. These problems were tackled in the new design. Current design The new versioning system is designed with two goals in mind. The first is to reduce server load and the second is to facilitate file migration. All branches are born equal, none more equal than the others. The main branch of a version is simply the first child of the version. The tips of all branches are stored as complete files. At the same time, all deltas are forward ones. It is likely that the latest versions of a branch will be accessed far more often than the older ones. By keeping them as complete files, DFS reduces the load on the server. Also, for this scheme, both forward and reverse deltas will work equally well. However, forward deltas are more intuitive and so, are used in the scheme. Some files may be stored complete, at all times. The versioning system designates such files based on the following rules:

PAGE 39

25 The version 0 is a full file Any version that is not on the main branch, i.e. if it is not the first child of its parent is stored as a complete file Any file that not stored in exactly the same servers as its parent is stored as a complete file If the version chain exceeds its maximum length with this file, the file is always complete. The first rule is obvious and needs no explanation. The second rule is to facilitate migration. Let the term last full ancestor of a version denote the last version (in the same branch), older than the given version, that has been stored as a full file. In case this rule is not applied, it would mean that besides the tip of a branch, only the root of the tree is a full file. In this case, to migrate an interior version, from one server to another, it would be necessary to migrate all versions except the tips of branches. This rule makes sure that no two branches have the same last full ancestor. The third rule helps in keeping the replication procedure simple. Consider the consequences of not having this rule. Now, it is possible that a delta will be stored on a server and its parent will be on another server. In order to retrieve the file, both the delta and the parent are necessary. This means that both servers should be alive and this decreases availability. In order to avoid these situations, this rule is applied. The last rule ensures that there are no long series of deltas. In the course of operations, it is likely that a complete file will be followed by a long series of deltas. To retrieve the last version, all preceding versions have to be reconstructed. Also, migration will require moving an enormous amount of data because migrating a version involves not only moving that version, but also all versions that it depends on and all versions that depend on it.

PAGE 40

26 Naming and Transparency DFS supports a conference-wide name space and a local name space. The conference-wide name space allows users to share files and directories on the file system. The local name space is used to store site specific information and temporary files. DFS provides location transparency and location independence. It is not possible to learn the files location from its name. In fact, the only way to ascertain the location of the file is to check the metadata. This makes it is possible to migrate a file from one server to another, without changing its name. Location Independence and location transparency allow DFS to provide a centralized view of DCS files, even when they are on totally different sites. DFS provides hierarchically structured directories. The path of each file starts with /DCS/. This allows client side applications to provide a global view of all conferences. It is possible to populate /DCS/ with all the conferences that the user is connected to and make it seem like /DCS/ is one directory on the local file space that provides access to conference files. Thus, client side applications can provide a global name space that merges multiple conferences. These conferences dont have to have any site in common. The files in each conference will be widely dispersed as well. Storing Metadata Location transparency and location independence are provided with the help of a file table, which maps a filename to its metadata and stores file metadata. This structure is also essential in order to present a uniform name space. A uniform name space also requires that the metadata on various sites be consistent. DFS maintains multiple versions of the same file. For the sake of simplicity, let the term physical file refer to each version of a file. This term is used because, on the

PAGE 41

27 operating system of the site, each version appears as a file. Another layer of the file table groups different versions together. This will be referred to, as a virtual file. Virtual files have no existence on the host operating system. A virtual file is just a collection of metadata related to every version of a file. The attributes of a physical file are listed below: Location of the file within DCS directory space Version of the file Type of file Owner of the file Size of the file Time the file was created Mode at which the file can be opened Location of replicas Is this a full file Should this file always be a full file Has this file been deleted Parent version of this file. Currently, a virtual file maintains only four attributes Type of virtual file (File or directory) Latest version References to all versions (file) References to other logical files (directory). The design of the file table is very important in ensuring performance, scalability and error recovery. Two options will be considered below for this purpose.

PAGE 42

28 Symbolic links One of the ways of representing this information is analogous to symbolic links. This is illustrated in Figure 3-3 and Figure 3-4. Figure 3-3 shows the physical location of conference files. Figure 3-4 illustrates the manner in which directories and links are used to provide a uniform view of the file system. In this example, the conference Conf1 is split across two sites Site 1 and Site 2. Figure 3-3 shows the distribution of the files among the sites. Site 1 has the files File1 and File2 along with the directory Dir1 and its contents. Similarly, Site 2 has File3, File4 and the directory Dir2. Figure 3-4 presents the uniform namespace, as seen from the site Site 1. Site 1 does not have the files File3 and File4. In order to keep the location of the files transparent, it creates symbolic links File3 and File4. These symbolic links are presented in bold in the figure. File3 and File4 contain information that identifies them as symbolic links and also contain the location of the server that actually stores the file. In order to access the file, the server opens the file, realizes that it is a symbolic link and obtains the location of the server that stores the file. It then contacts the server to obtain the file. It can be seen that the directory Dir2 is not shown in bold. This is to indicate that Site 1 does create the directory Dir2, even though all its contents are symbolic links. File Information Table (FIT) It is also possible to organize metadata and present a uniform name space using a File Information Table (FIT). This table stores all the attributes of the file, including its path and physical location. Each site has a copy of the FIT. Upon receiving a request for a file, the server searches through the FIT to locate the file and return its physical

PAGE 43

29 location. In case the file is on the same site, it returns the data. If the file is on another server, it requests the other server for the data. The actual arrangement of files on a site does not bear any relation to the view presented to the clients. It is the task of the FIT to map the logical view to the physical location. It is not necessary to create directories or symbolic links. The FIT can be retained in memory for fast access. A site can be brought up to speed by just sending it the FIT. Previous design Previous implementations of DFS chose the FIT approach for its simplicity. The scheme was designed with the Databases Service (DBS) in mind [29]. DBS would keep the FIT consistent by serializing all changes. DBS uses causal order protocol to implement reliable atomic multicast, ensuring that concurrent changes are serialized in some order. After a file is created, few of its attributes would change. The static and dynamic attributes would be maintained in different tables. Either the site that created the file or the home site of the conference can own the rows in DBS and serialize changes. There are several advantages in using DBS to maintain FIT. First, it takes a great load off the hands of DFS. Besides, DBS already maintains tables for other modules. Unfortunately, DBS was not fully functional at that point. This required DFS designers to implement their own methods for FIT consistency and a FIFO scheme was finally agreed upon. Here, every entry in the FIT has an owner and the owner is responsible for communicating changes to all servers. Servers must ensure that they process messages from a site, in the same order in which they were sent.

PAGE 44

30 Instead of a causal order protocol, only FIFO consistency is required. The current implementation of FIT is substantially similar. Proposed design It can be easily seen that both DBS and FIFO have their disadvantages. Their biggest problem is that neither is particularly scalable. DBS is too restrictive and in the case of the FIFO scheme, the number of messages required to maintain consistency increases nearly exponentially. A more scalable FIT scheme is presented next. In this scheme, every server maintains the root directory contents. The contents of other directories are maintained only if they lie in the path of a file located at this server. This means that if a server has version 2a.0 of the file /abc/def/xyz/pqr.txt, it has the contents of /abc, /abc/def, and /abc/def/xyz in its local FIT along with information about the different versions of /abc/def/xyz/pqr.txt. In order to get the listings of other directories, the server contacts other servers. In this scheme, the virtual files (collection of versions) will have to maintain these properties. Type of virtual file (file or directory) Contents of the virtual file Is the virtual file maintained locally List of servers maintaining this virtual file. If a directory is maintained on this server, then the attributes of all its virtual files are valid. Any change in the directory must be propagated to the list of servers. If the directory is not maintained locally, the attributes of the directory must be obtained from the list of servers maintaining this directory.

PAGE 45

31 This leads to a situation in which, the directories whose contents are more widely distributed are maintained on many servers. Normally, this means that directories with more files are more heavily replicated. This does make sense because the FIT should be at least as reliable as the file replication system. Another characteristic of this scheme is that higher level directories have higher replication factors. The directory /games/pc is on at least as many servers as /games/pc/Microsoft. Operations on the FIT are complicated by the fact that the FIT must be fetched from other servers in some cases. The various operations supported by the FIT are described below. Getting the attributes of a directory/file. If the given directory is maintained locally, it is trivial to obtain the information from the local FIT. Obtaining the attributes from another server is more challenging. Consider a directory /books/fiction/humour/20thCentury/English/Wodehouse. Let the server (Server A) have the directory listing up to /books/fiction. On receiving a query for the entire path, the local FIT realizes that it can resolve up to /books/fiction. As described previously, it will also have the list of servers that maintain the directory /books/fiction/humour. It passes on the path to one of them and asks that server to resolve it as much as it can. This server (Server B) might resolve it till /books/fiction/humour/20thCentury and ask Server C to resolve the rest and pass the result to Server A. Server C will be one of the servers on the list of servers maintaining /books/fiction/humour. It will be much faster if Server A could directly contact Server C for the information. However, Server A cannot know that one server is better than

PAGE 46

32 another for resolving this path. In theory, this information could be cached, so that from now on, Server C is contacted for /books/fiction/humour/20thCentury/English/Wodehouse. The interaction between servers could be iterative or recursive. In the iterative case, Server B gives Server A the address of Server C and asks it to contact Server C for resolution. In the recursive implementation, Server B queries Server C on behalf of Server A and passes on the result to Server A. The recursive version has a cleaner interface than the iterative version. However Server A might timeout on long paths. Creating a directory. The first step in creating a directory is to obtain the attributes of the parent directory. Then, all servers having the parent are informed that a new directory is being created and the list of servers maintaining the new directory is left empty (to signify that this is an empty directory). Creating a new DFS file. Creating a file is much more complicated than creating a directory. The server must first check whether it has the entire path on its local FIT. If it does not have it locally, it has to obtain information on all directories in the path and then inform other servers that it will also maintain all the directories in the path of the new file. Next, it needs to store the file (version 0 of the DFS file) and inform all servers maintaining that directory, of the new file. Replication is one of the services provided by DFS. The server (Server A), which is connected to the client creating the file, is responsible for the file. In this case, after creating and storing the file on all servers, Server A informs other servers about the existence of the file and the location of its replicas. All files with replicas must update their FIT, to include information about all directories in the path of the file. It must be

PAGE 47

33 noted that Server A may not be one of the servers which store the file. This makes it significantly complicated. Creating a new version. Normally, DFS stores a new version as a delta relative to the previous version. However, a version might be stored as a complete file if the version chain length is reached or if it is the first version of a new branch. In case a version is stored as a delta, it has to be stored on the same servers as its parent. It is fairly simple to do this because FIT of these servers already has information on the file. On the other hand, creating a version that is to be stored as a full file is just as complicated as creating a new file. Deleting a file. All servers that maintain the directory (the file is in) are informed to remove the file from the FIT. Next, each server that stores the file is asked to delete it. Now, these servers determine whether they need to maintain any of the directories in the path of the file. This involves determining whether the server has replicated a file in any of the directories (or their children) in the path of the deleted file. Once a server determines the directories it doesnt need to maintain, it informs the other servers maintaining those directories to remove it from the list. Cog Policy All DCS services operate at the cog level. It is imperative that they provide means to set policies that affect the entire cog. This gives administrators a way to broadly define policy. DFS allows users to set CoG-wide policies, which allow a coarse level control over its operations. Cog policies can also be used to set conference specific environment variables. It is possible for clients to set and query for the values of specific strings. This is analogous to environment variables in Windows and UNIX. Interestingly enough, its

PAGE 48

34 initial purpose was to help store replication policies and was simply adapted to this new role. The default cog policy strings are VersionChainLength: The maximum length of a version chain in the cog. Version chains are a sequence of file versions, where only the oldest version is stored in file system as a full file. All other versions can be obtained by applying deltas. WriteInterval: This specifies how often the FIT and cog policy are written to disk. It is possible to specify replication policy for the cog. This policy defines the number of replicas that a version should have. The number of replicas depends on the type of the version. For example, it might be specified that all ASCII files have 5 replicas. The cog policy manager ensures that cog policy changes are propagated to all sites. User Policy DFS also allows the user to set policies that apply only to him/her. In a manner similar to cog wide environment variables, applications can also take advantage of user specific environment variables. User policies are a fine grained way of controlling DFS operations. As described previously, DFS uses a versioning scheme with immutable files. One of the personal policies that a user can set is, to decide which version he/she sees when a file is opened. The user can specify either of the two options listed below. Get the latest version that was seen by the user (default) Get the latest version created. It is possible for the user to override the policy by specifying the version while asking for the file. For the sake of clarity, consider the following example. Let user A and user B, be working on the same version of a file. After a while, user A closes the file. This file is now saved on the server. User B is not affected right now and proceeds to

PAGE 49

35 finish his/her work and then save the file. When user A opens the file again, one of the three things might happen The user policy can be set to get the last file that the user opened. In this case, the user sees the file that he/she saved The user could be given a copy of the latest version of the file. Here, the user sees the version created by B The user could specify the version number to avoid any ambiguity. DFS also has a personal history manager which keeps track of the latest version of every file that was accessed by the user. The server can consult the history manager to obtain the version number, in case the user wishes to see the last accessed version. User policies also allow users to specify replication policies for files created by them. In a manner similar to cog replication policies, the user can specify the number of replicas for every type of file. Migration DFS is designed to operate over Wide Area Networks (WANs). WANs are characterized by long latencies and low bandwidth. In addition, one characteristic of a distributed system is continuous partial operation. The entire system is almost never operational simultaneously [28]. File migration is one of the approaches taken by DFS to mitigate these problems. DFS can move files from one server to another, to increase availability and decrease latency. Migration is complicated by the fact that files are often stored as deltas. Moving just the delta does not accomplish anything because DFS will need to access the earlier server to reconstruct the file. So, DFS migrates a version along with all versions that depend on it and all versions that this version requires. This list of versions is called a version chain. The first version in a version chain is a full file.

PAGE 50

36 Migration is also simplified by versioning rules that place limits on the length of a version chain. This reduces the amount of data that must be transmitted to migrate a version. Another rule ensures that no two branches have the same last full ancestor. Replication File replication is one of the ways of improving performance and availability of the system. If a server that maintains a file is unavailable, DFS looks to get the file from another replica. As long as at least one replica is available, DFS will retrieve the file. DFS hides replication detail from clients and maintains replica transparency. Since files are stored as immutable versions, no replica consistency issues have to be addressed. Some design options are considered below. Replicate entire file In this option, the file (a delta or a full file) is considered as the unit of replication. The entire file is replicated at a site. The advantage of this scheme is its simplicity. Replicate portions of file In this case, a file (a delta or a full file) is split into blocks. The size of the blocks can be fixed or variable. Each block is a unit of replication and is spread among the sites. The file is reconstructed by obtained the blocks from the replicas. This is analogous to RAID devices. RAID allows the use of error correction codes. It is not necessary to obtain all pieces to reconstruct the file. It is also far more complex and requires the system to maintain a lot of metadata. Replicate on create Replication can be performed as soon as a file is created. The server queries user and cog policies to obtain the number of replicas. This is determined by the file type. An

PAGE 51

37 important file can be replicated at a lot of sites to increase availability. A less important file can have fewer replicas, to save space. Replicate on open It is also possible to replicate a file on a site, when the site needs it. In this case, when the site obtains the file from another site, it checks whether the file has as many replicas as required for a file of that type. In case it has fewer replicas, a replica is created at this site and the FIT is updated. Design decision In DFS, the entire file is replicated when the file is created. Replication of entire files was chosen because of the simplicity of the scheme. It also makes other operations (like migration) easier. Splitting the file in several locations could force the server to connect to more remote sites. Simplicity is also one of the reasons for choosing to replicate the file on creation. In the replicate on demand scheme, it does not make sense to replicate the delta. All replicas should be complete files. So, there would be only one delta and all other replicas would be full files. This has two side effects. The first is that it increases storage requirements and the second is that it creates a situation where all replicas are not the same. All this can be avoided by replicating a file while creation. It also ensures that a file is not lost, even if it has not been accessed and the site that created it has crashed. Replication on creation ensures that the replication policy at the time of creation is followed. In case of replication on access, the decision to create a file will depend on the policy at the time of access. This is more unpredictable.

PAGE 52

38 Caching DFS is designed to use caching aggressively to improve performance. This is very significant over WANs. DFS supports caching on both the client and the server. Client caching DCS client process maintains a cache on the client machine. When asked for a file, the client obtains the file from the server and stores it in the cache directory. All operations are performed on the file. When the file is closed, the new file is pushed to the server, which stores it. The advantage of an immutable versioned file system is that a version of the file will never change. It can be cached on the client as long as necessary, with hard disk space as the only constraining factor. Clients do not cache attributes because they might change over time. It is preferable to get the attributes from the server. Server caching In contrast to most distributed file systems, DFS allows both replication and server caching. Server caching is similar to replicate on demand. When a file is obtained from another site, the server caches the full file for future use. The biggest difference between caching and replication is that, when a file is replicated on a server, the server must have the means to retrieve the full file without having to contact any other server. The server cannot remove the file from its local storage unless the file has migrated to another server. On the other hand, the contents of server cache do not concern any other server. A server can clear its cache when it needs space. Server and client caching are used to establish a working set of files on the respective machines.

PAGE 53

39 Security DFS security has two components. The first is, to provide secure communication between the client and the server. Secure Communication Service (SCS) is a DCS component designed to ensure authenticity, integrity and confidentiality of all messages. It also provides ways of authenticating users. Using SCS for this purpose will greatly simplify DFS design. The second component is to ensure that users are not allowed to perform unauthorized actions. Modern distributed file systems use access control lists for this purpose. In DCS, Access Control Service (ACS) is responsible for maintaining rights and protection information. ACS is extremely flexible and can even allow voting. DFS uses ACS to maintain the access control lists. ACS also allows users to create new attributes. File System Interface DFS provides an interface to the file system located at the server. This interface is used by clients and other servers. The server-client interface is described below: create: If the function is provided with data, it stores the data as a new file. In the event that no data is provided, a blank file is created. readVersion: This command is used by clients to obtain the contents of a DCS file. The version is determined by the versioning policy. The user can also specify a version explicitly. save: It saves the given data as a new version of the file. The parent version must be specified. deleteVersion: This marks a given version of a file as deleted. The file is not physically deleted. It is marked in the FIT as unavailable. undeleteVersion: The function reverses the action of DeleteVersion. It marks the version as available. deleteFile: It physically removes all versions of a file from all sites. It is not possible to retrieve the file.

PAGE 54

40 copy: This command asks the server to copy a version of a file as another file. The new file is not just a delta. It is the full contents of that version. move: This command renames a file or directory. All versions are renamed. makeDirectory: It creates a new directory in the file system. This operation merely updates the FIT. No directory is created on disk. removeDirectory: This command removes an empty directory. Again, this operation modifies only the FIT. listFile: This command lists the attributes of all versions of a file. listDirectory: It lists the contents of a directory. A directory can contain files or other directories. migrate: This command migrates the given version to a server. Along with the version, its entire version chain is moved. It is possible to specify the server from which the file is to be moved. getCogPolicy: This command allows the user to view cog policy. setCogPolicy: This command allows the user to modify cog policy. getCogReplicationPolicy: This command allows the user to view cog replication policy. setCogReplicationPolicy: This command allows the user to modify cog replication policy. getPersonalReplicationPolicy: This command allows the user to view personal replication policy. setPersonalReplicationPolicy: This command allows the user to modify personal replication policy. logout: This command flushes user related information to disk. isDirectory: It returns true if a given path is a directory. isPresent: This command returns true if a given path exists. The server: server interface is described next. getSiteList: This command returns the list of sites in the cog. readFile: This command reads a version of a file.

PAGE 55

41 writeFile: This command writes a version of a file. deleteFile: This command deletes a version of a file. Conclusion The chapter presents the requirements and the design of DFS. All options have been documented, even those which have not been used. This allows future designers to modify the design if they are not applicable any longer.

PAGE 56

42 1 3.1 3 2 4 Figure 3-1. Forward deltas. The dashed boxes represent delta files and the solid box represents a complete file. 1 2 3 3.1 4 Figure 3-2. Reverse deltas. The dashed boxes represent delta files and the solid boxes represent complete files.

PAGE 57

43 Figure 3-3. Actual location of files in a conference Site 1 DCS/Conf1 File1 File2 Dir1 files Dir1 DCS/Conf1 File3 File4 Dir2 files Dir2 Site 2 File2 File3 Dir1 Dir1 files File1 Dir2 File4 Dir2 files DCS/Conf1 Site 1 Figure 3-4. Virtual view of conference from site 1.

PAGE 58

44 Root Directory Dir A Not Local Server B Server C Server F Dir B Local Dir B1 Dir B2 File BT3 File BT4 Server Z Server L Server I Dir C Not Local Server B Server I Server M Dir B1 Local Dir B1.A Dir B1.B Server Z Server L Dir B2 Not Local Server I File BT4 File Attributes File BT3 File Attributes Dir B1.B Not Local Server Z Server L Dir B1.A Not Local Server Z Figure 3-5. Sample FIT

PAGE 59

CHAPTER 4 IMPLEMENTATION This chapter describes the implementation of DFS. It starts with a discussion of the minimum requirements and the interaction of DFS with other DCS services. Details of DFS architecture, with a description of the server, client and DFS applications conclude the chapter. Requirements DFS has been implemented in Java. J2SE 5.0 or greater should be installed on the system. DFS code is in pure java and can be run on any operating system and architecture without changes to the code. However, the versioning system uses BSDiff, BSPatch and Bzip for delta compression. These tools are available for Windows and all flavours of UNIX. Interaction with Distributed Conferencing System Services Interaction with Conference Control Service Conference Control Service is the lynchpin of DCS. Like most DCS services, CCS has been designed using client-server architecture [27]. CCS provides a graphical tool the User Manager (UM) which functions as a CCS client. User Manager allows the user to log in to the conference and use the services provided by the CCS server. The DFS client cannot connect to the server until the users credentials have been verified by CCS. On the server side, requests to the DFS server are often routed through CCS. Current work on DCS is focusing on integrating DFS with CCS. 45

PAGE 60

46 Interaction with Access Control Service ACS is used by DFS to ensure that users do not perform any unauthorized action. When a file is created, an entry is made in ACS tables, identifying the file and specifying the permissions of different roles. Also, when an operation on a file or its metadata is to be performed, ACS is consulted to determine whether the user has the specific permissions. Interaction with Secure Communication Service SCS is a DCS service that allows other DCS modules to communicate safely. It uses cryptographic techniques to guarantee message authenticity, integrity and confidentiality. SCS also maintains client and server certificates and changes keys periodically. DFS uses SCS to provide secure communication between DFS servers and between DFS clients and servers. Communication DFS has not been completely integrated with SCS. This has forced DFS to implement its own communication module. All communication between the DFS client and the DFS server takes place as a Java Remote Method Invocation (RMI). DFS Server object is registered with RMI registry and the client binds to it. The client invokes the appropriate methods for various operations. Communication between servers also takes place using RMI. Distributed File System Server helps in initializing new servers that wish to join the conference. The FIT Manager also binds to the registry. This helps it contact other sites and maintain consistency. The Replication/Migration Manager registers with the RMI registry so that it can deposit and retrieve replicas on other sites. Both User Policy Manager and Cog Policy Manager bind to the registry to ensure policy consistency throughout the cog.

PAGE 61

47 Distributed File System Architecture The implementation architecture of DFS is shown in Figure 4-1. The components in bold communicate with the corresponding modules on other servers to maintain synchronization. The details of the modules are given next. File Information Table Manager (FITM) File Information Table Manager maintains and updates the FIT. It is also responsible for synchronizing the FIT with other servers. It ensures that the site is consistent and has an up-to-date list of files. When the server is started, it contacts other servers to get the latest FIT and when the server is shutdown, it writes the FIT back to the disk. The FIT is stored in a file on the directory DFS_ROOT/system. The filename is specified in the property file. For the sake of convenience, the list of attributes that it maintains is reproduced here. Location of the file within DCS directory space Type of virtual file (file or directory) Version of the file Type of file Owner of the file Size of the file Time the file was created Mode at which the file can be opened Location of replicas Is this a full file Should this file always be a full file

PAGE 62

48 Has this file been deleted Parent version of this file. The first attribute stores the DCS path of the file. The /DCS/ prefix is stripped out of the path name. They next attribute identifies whether the virtual file is a file or a directory. Directories have only a few attributes associated with them. Files have all attributes and can have multiple versions. There is a similar attribute, type of version. Most DFS policies are set according to the type of version. Different versions of the same file can be of different types. For example, a document with the format of a letter might be of template type but after it has been filled up, its type can become letter. This is set by application at the time the new version is written. The FIT does not maintain file permissions. The last accessed time is not maintained because this places an unnecessary synchronization load on the servers. The attribute mode is not currently implemented. It can specify the mode in which the file should be opened read-only, binary, can cache, etc. One copy of the FIT is maintained in memory for fast access. Hash tables are used because they have an order of O(1) access time. The FIT is periodically written to disk. This interval is configurable. Version Naming Manager (VNM) Version Naming Manager generates version names for new versions of a file. The generated version name depends on the parent version and the number of its siblings. Version naming in DFS has the following characteristics. The version zero is the first version of a file. Siblings are versions that are derived from the same parent. Siblings can be identified by the fact that they have everything in common until the last sequence of alphabets. In Figure 4-2, V2, V2a.0 and V2b.0 are siblings.

PAGE 63

49 Once all alphabets are used up, the next sibling is V2aa.0 and so on. To obtain the parent of a version, decrement number after the last period. However, if the number is 0, then remove the 0, period and the alphabetic portion and then decrement the number, to obtain the parent. The parent of V2b.1a.0 is V2b.0. Figure 4-2 shows a version tree in DFS. Versions marked in bold are the first versions of their sub tree. DFS stores all these file in full. Version Control Manager (VCM) obtains these file names from VNM and then stores the file. Version Naming Manager also has routines to determine whether a child is on the main branch of its parent. Using this information, VCM decides whether the file should be stored as a delta or as a complete file. Cog Policy Manager (CPM) The Cog Policy Manager is responsible for maintaining general cog policy. In addition, it also maintains specific policies like the cog replication policy. Cog policy values can be queried by applications. It is possible for applications to modify the policy as well. Replication policy determines the number of replicas of a file. This information is used by the Replication/Migration Manager (RMM). Policy values can be modified using the setPolicy interface which is exposed to the client. The getPolicy interface is used to obtain a particular policy. Cog Policy Manager is also responsible for maintaining synchronization with other sites. When cog policy is modified by a user, CPM communicates this to other sites so that cog policy is uniform across all sites. User Policy Manager (UPM) User Policy Manager (UPM) is quite similar to CPM. It maintains policies regarding specific users. This includes the general user policy and user replication policy.

PAGE 64

50 They are quite analogous to their cog wide counterparts. They can be queried and modified by applications. In case of conflicts between cog and user policies, the application can choose to follow either. One user policy is to specify which version of a file is seen by default. This can either be the latest version of the file or the last version seen by the user. User Policy Manager needs to keep track of the last version of every file seen by the user. This component of UPM is called Personal History Manager (PHM). Whenever the user opens a file, PHM enters the file name and the version. This can be retrieved by querying UPM. Cog policies must be immediately communicated to other sites in the cog. By contrast, it is unlikely that a user will be connected to more that one site in a conference. That makes it unnecessary to update every site when a change is made to user policy. When the user logs out of the system, the server contacts all other sites and updates their user profile. Cogs have a large number of users. It is prohibitively expensive for sites to load the profile of every user. Instead sites load profiles into memory on demand. Once the user logs out, the profile is dumped to disk and that memory is reclaimed. Version Control Manager (VCM) Version Control Manager (VCM) is responsible for the versioning aspects of DFS. It gets the version name from VNM, along with the number of replicas from CPM and UPM. It then hands the file over to Replication/Migration Manager (RMM) which replicates the file on different sites. VCM then checks whether a delta of the parent version needs to be created and replicates it as necessary.

PAGE 65

51 Its other task is to rebuild files from deltas. For this purpose, it gets the list of versions that are required to reconstruct the file. Then, it applies the deltas repeatedly and rebuilds the file. Creating a Delta The new version is always stored as a complete file. It is the parent version that must now be converted into a delta. In some cases however, the parent file remains as a complete file. These cases are recapitulated below. The version 0 is a full file. Any version that is not on the main branch, i.e. if it is not the first child of its parent is stored as a complete file. Any file that not stored in exactly the same servers as its parent is stored as a complete file. If the version chain exceeds its maximum length with this file, the file is always complete. If the parent version does not fall under any of these categories and is a complete file, it must be converted to a delta. First, the grandparent version number is obtained from FIT. Version Control Manager reconstructs the grandparent file. This may not be necessary if the grandparent is a full file. Now, a delta is created between the grandparent and the parent files. Due to the lack of a delta compression package is java (current language of implementation), DFS uses the utility BSDiff to create deltas. BSDiff was chosen because it is available for a large number of platforms and can handle both text and binary files. The delta file is replicated on the same sites and the FIT is modified to show that the parent version is no longer a full file.

PAGE 66

52 Recreating a Version The FIT stores the version number of the parent of every version. When VCM asks for the list of versions that are required to recreate this version, the FIT traverses up the versioning tree to find the latest ancestor that was stored as a full file. This full file is copied to a temporary file. The delta of its child version is applied to recreate the child version. The next delta is applied to get the next version and so on. The deltas are applied till the required version is recreated. The data in the temporary file is then returned. BSPatch is a utility which can recreate file that were diffed using BSDiff. Replication/Migration Manager (RMM) File replication and migration are handled by Replication/Migration Manager (RMM). When a request is made to migrate a file, RMM checks whether a copy of the file already exists on the destination site. If there is a replica of the file on the destination server, no action is taken. If the file is not present on the site, RMM contacts its corresponding modules in the source and destination sites. In the first step, a copy of the file is transferred from the source to the destination server. The FIT is then updated to reflect this change. In the final step, the file is deleted from the source server. Replication/Migration Manager uses Local Filestore Manager (LFM) to handle the actual storage and retrieval. When a file is created, it is replicated on the required number of sites. Replication/Migration Manager identifies the servers which can host the file. It connects to RMM on these servers and passes the data to them. FIT is then updated to include the location of these servers.

PAGE 67

53 Local Filestore Manager (LFM) Local Filestore Manager (LFM) is the component of DFS that interacts directly with the OS file system. When VCM stores a file, it passes the data and asks LFM to store the data as a file on disk. It handles the retrieval of files too (both delta and complete files). As far as LFM is concerned, both are identical. It just reads the data and passes it on. It is up to the Version Control Manager to rebuild the file. Local Filestore Manager uses the underlying OS file system for file and disk operations. This eliminates the need to implement file system manipulation functions, which have already been correctly and efficiently implemented. Distributed File System Server Distributed File System Server glues all the components together, and coordinates their activities. It also interacts with the client and with other servers when necessary. When the server is started, it loads the DCS property files and if necessary, creates the DCS directory. It then reads the list of servers that belong to the conference. At this point, one of the servers is contacted and a fresh list of cog sites is obtained. Now, all servers in the cog are asked to add the location of this new server. The next step is to synchronize the server with other sites in the conference. It tries to obtain the up-to-date FIT table from another server. In case it is not successful, it looks for the FIT file on disk and loads the FIT. If there is no FIT file, the server starts with a clean FIT. Cog policies are requested from other servers in the cog. Again, if it is unsuccessful, the server tries to read it from disk. If there is no policy file, a blank policy is loaded.

PAGE 68

54 The final step is to register the necessary server objects on the local RMI registry. All modules that require synchronization with other sites are registered. These include DFS Server, FITM, RMM and both UPM and CPM. The interfaces presented by the server to the client and to other servers are reproduced below for convenience. Client-Server Interface create: If the function is provided with data, it stores the data as a new file. In the event that no data is provided, a blank file is created. readVersion: This command is used by clients to obtain the contents of a DCS file. The version is determined by the versioning policy. The user can also specify a version explicitly. save: It saves the given data as a new version of the file. The parent version must be specified. deleteVersion: This marks a given version of a file as deleted. The file is not physically deleted. It is marked in the FIT as unavailable. undeleteVersion: The function reverses the action of DeleteVersion. It marks the version as available. deleteFile: It physically removes all versions of a file from all sites. It is not possible to retrieve the file. copy: This command asks the server to copy a version of a file as another file. The new file is not just a delta. It is the full contents of that version. move: This command renames a file or directory. All versions are renamed. makeDirectory: It creates a new directory in the file system. This operation merely updates the FIT. No directory is created on disk. removeDirectory: This command removes an empty directory. Again, this operation modifies only the FIT. listFile: This command lists the attributes of all versions of a file. listDirectory: It lists the contents of a directory. A directory can contain files or other directories.

PAGE 69

55 migrate: This command migrates the given version to a server. Along with the version, its entire version chain is moved. It is possible to specify the server from which the file is to be moved. getCogPolicy: This command allows the user to view cog policy. setCogPolicy: This command allows the user to modify cog policy. getCogReplicationPolicy: This command allows the user to view cog replication policy. setCogReplicationPolicy: This command allows the user to modify cog replication policy. getPersonalReplicationPolicy: This command allows the user to view personal replication policy. setPersonalReplicationPolicy: This command allows the user to modify personal replication policy. logout: This command flushes user related information to disk. isDirectory: It returns true if a given path is a directory. isPresent: This command returns true if a given path exists. Server-Server Interface getSiteList: This command returns the list of sites in the cog. readFile: This command reads a version of a file. writeFile: This command writes a version of a file. deleteFile: This command deletes a version of a file. Distributed File System Client The DFS Client exposes the DFS server interface to DFS aware applications. Any application that seeks to exploit the features of DFS must include a reference to the DFS client. Prior implementations mandated that each user have only one DCS Client, which has been done away with, in this implementation. Applications can use multiple DCS Clients at the same time. It is also possible for multiple applications to use the same client

PAGE 70

56 and interleave their calls to the DFS server. Most methods on the client side, merely call the same methods on the server. Applications Shell In order to take advantage of the capabilities of DFS, it is necessary to implement applications. The shell is an application designed to perform operations similar to UNIX shells. It connects to the server using the DFS Client and manipulates DCS files and directories. The shell can accept both absolute and relative paths (starting with ./ or ../). It implements the following commands. ls: If the command is applied on a directory, all its contents are listed. These might be files or other directories. If the command is applied on a file, information is displayed on all versions of the file. cd: This command changes the current directory. cp: This command copies a given version of a file as another file. mv: This command renames / moves a file or directory. delfile: This command deletes a file permanently. delver: This command marks a version of a file as deleted. undelver: This command restores a deleted version. mkdir: This command creates a new directory. rmdir: This command removes an empty directory. If a directory is not empty, an error message is displayed. migrate: This command migrates a file version to a server. The location of the source site can be optionally specified. logout: This command closes the connection to the server. pwd: This command displays the current working directory.

PAGE 71

57 chpol: This command changes a specific policy. The next argument must specify the policy which is to be changed. Current options are: o repl: User replication policy. o gen: General user policy. o coggen: Cog general policy. o cogrepl: Cog replication policy. getpol: This command gets a specific policy. It requires one of the options listed above. Distributed Conferencing System Text Editor The DCS Text Editor is an application which is specifically designed to demonstrate the functionality of DFS. The text editor uses the DCS Client to connect to the server. When the text editor is asked to open a file, it checks to see if the file exists. If the file does not exist, it asks the user whether a new file should be created and then creates the file if needed. In case of an existing file, the file is retrieved and its contents are displayed in the editor. The editor allows the user to modify and then save the file as a new version.

PAGE 72

58 Version Namin g FIT A pp User policy Serve r Client Version Control Cog policy Replication / Migration Server Local Filestore Figure 4-1. DFS architecture. The modules in bold interact with their counterparts on other servers.

PAGE 73

59 V0 V2 V2a. 0 V2b. 0 V2b.1 V2b.1a.0 V1 V3 V2a.1 V4 V2a.2 V2b.2 V2b.1a.1 Figure 4-2. Version tree in DFS

PAGE 74

CHAPTER 5 EVALUATION Testing DFS is one of the modules of DCS and depends on some of the services provided by DCS. Unfortunately, the components of DCS have not been completely integrated. In order to test DFS, specific test classes have been created to simulate the missing DCS services. DFS was tested on both Windows and Linux and was found to be satisfactory. The test bed had three machines running DFS server process and multiple clients. One of the aspects tested, was the ability of DFS components to maintain synchronization. Tests were conducted to verify all components of DFS. Replication/Migration Manager, Version Control Manager, User Policy Manager and Cog Policy Manager were the focus of the tests. Additional tests were made to ensure that servers could be added on the fly, to increase space. Evaluation The overall objectives of DFS were stated previously in Chapter 3. These are reproduced below for convenience. The file system should provide a uniform name space. Files and directories should be location transparent and location independent. Concurrency control should allow users to modify the same files, without losing changes made by any user. The consistency semantics should be predictable. 60

PAGE 75

61 File migration and replication must be supported to improve availability and performance. New servers must be allowed to join the conference without having to shut it down. Addition of servers should not change user experience negatively. The file system should provide an interface to allow users and applications to interact efficiently. DFS provides a uniform name space for all files in the conference. In fact, it is possible to extend it to provide a global namespace. In this respect, it is similar to Coda. The second objective is achieved by providing a File Information Table (FIT). The FIT acts as a mapping table and translates between the logical name of the file and its actual location. The immutable versioning system in DFS ensures that concurrent changes made to a file do not overwrite each other. Changes made by users are saved as new versions. This ensures that writes do not interfere with each other and files never lose changes. DFS file consistency is predictable and configurable. Users can see the latest changes by specifying their policy. The semantics are quite close to session semantics. Replication and migration are supported by Replication/Migration Manager. This increases system availability and fault-tolerance greatly and improves performance as well. DFS implementation allows a new server to join the conference as long as it knows the location of one server in the cog. The other servers in the cog bring the server up to speed. A user does not even know that a new server has joined the cog. DFS provides interfaces to both DFS Clients as well as other Servers. It can be seen from the summary that the main objectives of DFS were achieved. The next section puts forth some of the ideas that could be examined in the next version of DFS.

PAGE 76

62 Future Work The current implementation of DFS is the first step towards implementing a flexible distributed file system with high availability and great ease of use. It has not been possible to design and implement all aspects of a distributed file system. However, DFS provides a foundation for future work. Some of the ideas that can be examined are given below. Integrate DFS with other DCS modules. DFS is designed to interact closely with Conference Control Service. It also depends on Secure Communication Service and Access Control Service for security. A good deal of work will be required to modify calls in all these services. Allow DFS to raise events. The Notification Service (NTF) is responsible for registering events and listeners. It also informs the listeners when the event takes place. DFS should use NTF to raise events. One scenario could be to inform specific users when a new version of a file is available. Current implementation of the File Information Table Manager (FITM) is not very scalable. Implementing the new design would improve scalability. Replace the current command line delta creation and rebuilding utility with a Java package. One of the biggest strengths of Java is platform independence. The use of command line tools has severely restricted the operating systems that DFS can run on. A delta compression package must be identified or developed for use in DFS. DFS client should be integrated with the operating system. This requires modification of OS system calls. Tight integration with Windows is virtually ruled out due to source code unavailability. Linux system calls can be modified to support DFS. Support file merging and reconciliation. Coda is a distributed file system that supports automatic file merging and reconciliation. Similar work can be done on DFS to allow users to merge files. Conclusion Distributed Conferencing System provides support for users to collaborate efficiently and securely. Distributed File System is an important part of DCS and provides the file system for it. Distributed File System must provide concurrency control,

PAGE 77

63 uniform naming, location transparency and location independence, high performance and availability along with security. Other file systems provide some of these features but no other distributed file system supports all of them. Most do not deal very well with concurrency, believing that concurrent writes on the same file are extremely rare. Distributed File System handles concurrency well and allows each user to preserve his/her changes as immutable versions. Another feature of DFS is the ability to specify cog wide properties and replication policies. Though DFS is designed specifically for DCS, the ideas are applicable in any environment where concurrency and availability are important.

PAGE 78

LIST OF REFERENCES [1] R. E. Newman, C. L. Ramire, and H. Pelimuhandiram, M. Montes, M. Webb, and D. L Wilson, A Brief Overview of the DCS Distributed Conferencing System, in Proceedings of the Summer USENX Conference, Nashville, TN, June 1991, pp. 437-452. [2] A. S. Tanenbaum and M. Steen, Distributed Systems: Principles and Paradigms, Prentice Hall, Upper Saddle River, NJ, 2003. [3] R. Chow and T. Johnson, Distributed Operating Systems & Algorithms, Addison-Wesley, Reading, MA, 1997. [4] V. Manian, Access Control Model for Distributed Conferencing System, M.S. thesis, University of Florida, Gainesville, FL, 2002. [5] L. Li, File Services for a Distributed Conferencing System (DCS), M.S. thesis, University of Florida, Gainesville, FL, 1995. [6] P. S. Yeager, A Distributed File System for Distributed Conferencing System, M.S. thesis, University of Florida, Gainesville, FL, 2003. [7] R. Sandberg, D. Coldberg, S. Kleiman, D. Walsh, and B. Lyon, "Design and Implementation of the Sun Network Filesystem," in Proceedings of the Summer USENX Conference, Portland, OR, 1985, pp. 119-130. [8] R. Srinivasan, RPC: Remote Procedure Call Protocol Specification Version 2, Sun Microsystems, Santa Clara, CA, Request for Comments 1831, August 1995. [9] B. Nowicki, NFS: Network File System Protocol Specification, Sun Microsystems, Santa Clara, CA, Request for Comments 1094, March 1989. [10] B. Callaghan, B. Pawlowski and P. Staubach, NFS: Network File System Protocol Specification, Sun Microsystems, Santa Clara, CA, Request for Comments 1813, June 1995. [11] B. Callaghan, D. Robinson, R. Thurlow, C. Beame, M. Eisler, S. Shepler and D. Noveck, NFS version 4 Protocol, Sun Microsystems, Santa Clara, CA, Request for Comments 3010, December 2000. 64

PAGE 79

65 [12] B. Callaghan, D. Robinson, R. Thurlow, C. Beame, M. Eisler, S. Shepler and D. Noveck, Network File System (NFS) version 4 Protocol, Sun Microsystems, Santa Clara, CA, Request for Comments 3530, April 2003. [13] M. Satyanarayanan, A Survey of Distributed File Systems, Annual Review of Computer Science, vol. 4, pp. 73-104, August 1990. [14] M. Satyanarayanan, Scalable, Secure, and Highly Available Distributed File Access, IEEE Computer, vol. 23, pp. 9-21, May 1990. [15] M. Satyanarayanan, and M. Spasojevic, AFS and the Web: Competitors or Collaborators, in Proceedings of the Seventh ACM SIGOPS European Workshop, Connemara, Ireland, 1996, pp. 89-94. [16] M. Spasojevic and M. Satyanarayanan, An Empirical Study of a Wide Area Distributed File System, ACM Transactions on Computer Science, vol. 14, pp. 200-222, May 1996. [17] R. Needham and M. Schroeder, Using Encryption for Authentication in Large Networks of Computers, Communications of the ACM, vol. 21, pp. 993-999, June 1978. [18] M. Satyanarayanan, J. Kistler, P. Kumar, M. Okasaki, E. Siegel, and D. Steere, Coda: A Highly Available File System for a Distributed Workstation Environment, IEEE Transactions on Computers, vol. 39, pp. 447-459, April 1990. [19] R. Pike, D. L. Presotto, S. Dorward, B. Flandrena, K. Thompson, H. Trickey and P. Winterbottom, Plan 9 from Bell Labs, Computing Systems, vol. 8, pp. 221-254, February 1995. [20] R. Y. Wang and T. E. Anderson, xFS: A Wide Area Mass Storage File System, in Proceedings of Fourth Workshop on Workstation Operating Systems, Napa, CA, 1993, pp. 71-78. [21] T. Anderson, M. Dahlin, J. Neefe, D. Patterson, D. Roselli and R. Wang, Serverless Network File Systems, in Proceedings of ACM SOSP, Ashville, NC, 1995, pp. 109-127. [22] A. Muthitacharoen, B. Chen and D. Mazieres, A Low-Bandwidth Network File System, in Proceedings of ACM SIGOPS Operating Systems Review, Newport, RI, 2001, pp. 174-187. [23] M. O. Rabin, Fingerprinting by Random Polynomials, Harvard University, Cambridge, MA, Technical Report TR-15-81, 1981. [24] U.S. Department of Commerce/National Institute of Standards and Technology, Secure Hash Standard, National Technical Information Service, Springfield, VA, FIPS PUB 180-1, April 1995.

PAGE 80

66 [25] D. Mazieres, M. Kaminsky, M.F. Kaashoek and E. Witchel, Separating Key Management from File System Security, in Proceedings of ACM SIGOPS Operating Systems Review, Charleston, SC, 1999, pp. 124-139 [26] I. Clarke, O. Sandberg, B. Wiley, and T. W. Hong, Freenet: A Distributed Anonymous Information Storage and Retrieval System, in Proceedings of the ICSI Workshop on Design Issues in Anonymity and Unobservability, Berkeley, CA, 2000, pp. 46-66. [27] A. Bhalani, Implementation of Conference Control Service in Distributed Conferencing System, M.S. thesis, University of Florida, Gainesville, FL, 2002. [28] R. G. Guy, T. W. Page, J. S. Heidemann, and G. J. Popek, Name Transparency in Very Large Scale Distributed File Systems, in Proceedings of Second IEEE Workshop on Experimental Distributed Systems, Huntsville, AL, 1990, pp. 20-25. [29] A. V. Date, Implementation of Distributed Database and Reliable Multicast for Distributed Conferencing System Version 2, M.S. thesis, University of Florida, Gainesville, FL, 2001.

PAGE 81

BIOGRAPHICAL SKETCH Prashant Jayaraman was born in Thanjavur, India in 1983, the son of Raji and S. Jayaraman and the younger brother of Vivekanand Jayaraman. He completed his Bachelor of Technology degree in computer science and engineering from Regional Engineering College, Tiruchirappalli, India in June 2004. He then decided to join the University of Florida in 2004 to pursue his masters degree. After graduation, he plans to pursue a career in industry but hopes to eventually earn a Ph.D. 67


Permanent Link: http://ufdc.ufl.edu/UFE0014040/00001

Material Information

Title: A Distributed File System (DFS)
Physical Description: Mixed Material
Copyright Date: 2008

Record Information

Source Institution: University of Florida
Holding Location: University of Florida
Rights Management: All rights reserved by the source institution and holding location.
System ID: UFE0014040:00001

Permanent Link: http://ufdc.ufl.edu/UFE0014040/00001

Material Information

Title: A Distributed File System (DFS)
Physical Description: Mixed Material
Copyright Date: 2008

Record Information

Source Institution: University of Florida
Holding Location: University of Florida
Rights Management: All rights reserved by the source institution and holding location.
System ID: UFE0014040:00001


This item has the following downloads:


Full Text












A DISTRIBUTED FILE SYSTEM (DFS)


By

PRASHANT JAYARAMAN













A THESIS PRESENTED TO THE GRADUATE SCHOOL
OF THE UNIVERSITY OF FLORIDA IN PARTIAL FULFILLMENT
OF THE REQUIREMENTS FOR THE DEGREE OF
MASTER OF SCIENCE

UNIVERSITY OF FLORIDA


2006

































Copyright 2006

by

Prashant Jayaraman

































To my parents and to my brother Vivek.















ACKNOWLEDGMENTS

I thank Dr. Richard Newman for his help and guidance. I would like to express my

gratitude to Dr. Randy Chow and Dr. Shigang Chen for serving on my supervisory

committee and reviewing my work. I also acknowledge the contributions of all members

of the Distributed Conferencing System group. Special thanks are due to my family and

friends for their encouragement and support.
















TABLE OF CONTENTS

page

A C K N O W L E D G M E N T S ......... ...................................................................................... iv

LIST OF FIGURES ................ ...... .. .......................................... viii

LIST OF A BBREV IA TION S........ ......... .......... .......... ..................... .............. ix

A B S T R A C T .......................................... .................................................. x iii

CHAPTER

1 IN TR O D U C TIO N ........................ ............................ ........ ...... ................

Introduction ................................ .. ............................... .. ............. .
D distributed Conferencing System Services .............................................................2

2 STUDY OF DISTRIBUTED FILE SYSTEMS ............................................... 4

In tro d u ctio n ............................................................... ................ .. 4
N etw ork File System (N F S)................................................................ ...............4
Introduction ............... ......... ......... ......... 4
Protocol ............................................................................ ...... .................. 4
N am ing and Transparency ........................................ ........ ............... 5
Concurrency Control and Consistency ....................... ................ ........... 6
R eplication and M igration ........................................ ........ ............... 6
C a c h in g ..................................................... ................ 7
S e cu rity ..................................................... ................ 7
A ndrew File System (A F S)........................................................... ............... 8
In tro d u ctio n ............. .. .............. .......................................... 8
A architecture .................. ............ .............. .................... .......... 8
N am ing and Transparency ........................................ ........ ............... 9
Concurrency Control and Consistency .............. ............ .....................9
R eplication and M igration ........................................ ........ ............... 9
Caching ...... .......... ....... .................. 10
Security ...... .......... ........ .................... 10
Common Internet File System (CIFS) .... .......... ..................... .................11
C o d a .................................................................................. 1 1
P la n 9 ...............................................................................1 2
xFS ................................................. 13


v









Low Bandwidth File System (LBFS) ........................................ ............... 14
F reen et......................................................................................... .. .. ..... 15

3 REQUIREMENTS AND DESIGN ........... ................................. ...............18

R equ irem en ts ..................................................... ................ 18
N am ing and Transparency ........................................ ...... ............... 18
Concurrency Control and Consistency ............................................... 19
R eplication and M igration ........................................ ...... ............... 19
C ach in g ......................................................................2 0
S ecu rity ......................................................................2 0
D design ................... ...................................... ........ 21
Communication Architecture...................................... 21
Stateful/Stateless Server..................................... ............................. ........ 2 1
File Access M odel............. ........ .. ..... .......................21
Concurrency Control and Consistency ............................................... 22
V ersioning ........................................................................... 23
Forw ard deltas........................... .......... ........ .. .......... ....23
R everse deltas ............................ ....... ...... ...... .. ...... ... 23
Previous design ................................................ .... .................. .. 24
Current design .......... ............................. ............ .................. 24
N am ing and Transparency ........................................ ...... ............... 26
Storing M etadata ......................................... .. ........................26
Sym b olic link s ........................................... ........ ........ 2 8
File Inform ation Table (FIT) ............................... ............... .28
Previous design ........ .... ...... ................ .......... ... ............ 29
Proposed design ............................................... .. .............. 30
C o g P o licy .............................................. ................ 3 3
U ser Policy .................................... .......................... .. .........34
M ig ration ................................................................................. .. 3 5
Replication ................................... .......................... .. .........36
R eplicate entire file ................................. ............. .................. 36
Replicate portions of file..... .......... ...................................... 36
R eplicate on create ..................................................................... 36
Replicate on open................. ........... ..... ........................37
D esign decision .......... ........................ ........ .......... ........ 37
C ach in g ..................................................................... 3 8
Client caching ..................................... ............ .............. 38
Server caching........................... .......... ........ .. .......... .. ..38
S ecu rity ......................................................................3 9
File System Interface ........................................ ........................... 39
C o n c lu sio n ....................................................... ................ 4 1









4 IM PL E M E N T A TIO N ................................................................. .....................45

Requirements ........................................................... 45
Interaction with Distributed Conferencing System Services............................45
Interaction with Conference Control Service ........................................45
Interaction with Access Control Service............................ .....................46
Interaction with Secure Communication Service ......................................46
Com m unication....... .................. ........ ......... ...... ............. 46
D distributed File System Architecture................................... ...................... 47
File Information Table M manager (FITM ).................................. ............... 47
Version Naming M manager (VNM )..................................................................... 48
Cog Policy M manager (CPM ) ............................................................................ 49
U ser Policy M manager (U PM ) ........................................ ........................... 49
V version Control M manager (V CM )...................................... ........................ 50
C creating a D elta ....................... ........................ .. ...... .. .......... 5 1
Recreating a Version................ ............................... 52
Replication/M igration M manager (RM M ) .................................... ............... 52
Local Filestore M manager (LFM ) ........................................ ........................ 53
D distributed File System Server ........................................................... ... .......... 53
C lient-Server Interface.......................................................... ................54
Server-Server Interface ................................................... ................. 55
D distributed File System Client......................................... .......................... 55
Applications .................................... ................................ .........56
S h ell ........................ ........... .................................. 5 6
Distributed Conferencing System Text Editor................. ..................57

5 EV ALU A TION .... .............................. ........ ............ ............... .. 60

T e stin g ..........................................................................................6 0
E valuation .................................................................... 60
Future Work ............................................... 62
C o n clu sio n ....................................................... 6 2

LIST OF REFEREN CE S ................................................................................................... 64

B IO G R A PH IC A L SK E T C H ....................................................................................... 67
















LIST OF FIGURES

Figure page

2-1. Example of NFS clients mounting the same directories at different mount points..17

3-1. Forward deltas........................................................... ..... ......... 42

3-2 R ev erse deltas ............................................... ............................ ...... 42

3-3. Actual location of files in a conference.................... ...... ...............43

3-4. Virtual view of conference from site 1.......................................... ............... 43

3-5. Sam ple F IT .................................................................................................. 44

4-1. DFS architecture .............................. ....................... .... ..... 58

4-2. V version tree in D F S .................. ................... ....................... 59















LIST OF ABBREVIATIONS

Causally related: Two events are causally related if the order of their occurrence

affects their result.

Causal consistency: A model specifying that operations that are potentially

causally related must be seen by everyone in the same order. The order of other

operations is unimportant and they are assumed to be concurrent.

Client: Process that accesses a (remote) service on another computer through a

network A user uses a client to access the services provided by the servers.

Coherence: A property of distributed systems in which the file is consistent across

all its replicas. The consistency is defined by the consistency model that is used.

Concurrency control: The manner in which concurrent access/modification of

objects is handled. Time stamping, locking and optimistic concurrency control are forms

of concurrency control.

Consistency: A system is consistent if operations on objects follow specific rules.

The system guarantees that upon following the rules, the objects will be consistent and

the results will be predictable.

Consistency model: The set of rules that specify when an object can be classified

as consistent.

DCS aware application: An application that can take advantage of DCS services.

It is designed specifically with DCS in mind.









DCS conference: Set of users, with a set of roles, a set of objects, and a set of

applications. Typically a conference has one specific purpose or goal and the members of

the conference work together using the objects and applications to achieve this goal. A

conference can also have one or more sub-conferences that are used to accomplish a

specific aspect of the larger conference's goal. A conference may be spread over several

sites. Each site contributes resources to the conference.

DCS file space: File space managed by Distributed File System presented in this

thesis. It is only accessible through the services provided by this module.

DCS instance: A collection of sites and conferences.

DCS server: Server modules of the various DCS services. A site may run several

servers belonging to different DCS conferences.

DCS site: Collection of servers running DCS services. A site may be a part of

several conferences. DCS sites are composed of a Local Area Network (LAN) and have a

shared file system. The communication within a site is assumed to be reliable and highly

efficient. There is also the notion of correlated accessibility, which means that the

accessibility of the servers and objects on a site are related. If one is inaccessible, there is

a great probability that all will be inaccessible.

DCS unaware application: Any application that is not DCS aware. Almost all

normal applications fall in this category.

Delta: A file containing the difference between one file and other. It is also called a

diff









Delta compression: A way of storing data in form of differences between

sequential data rather than as complete files. A file is stored as the difference between

itself and a base file. Using the base file and the difference, the given file can be rebuilt.

Immutable semantics: A model in which all objects are immutable. Any change to

an object is stored as a new object.

Location independence: A resource can be moved about the system without

changing its name.

Location transparency: Names used to identify resources do not reveal the

location of the user or the resource.

Object: Any file that exists within DCS and is managed by DFS.

Role: A named set of capabilities. Roles can also be viewed as a collection of job

functions. Each role has a set of actions it can take on an object. A user that is bound to a

role can perform the set of actions available for that role on a specific object.

Session semantics: A consistency model that dictates that changes to a file are

visible only after the process closes the file.

Stateful server: A server that maintains information about its clients. It is more

efficient than a stateless server.

Stateless server: A Server that does not store information about its clients. This

allows for a very robust design.

Transparency: Refers to hiding irrelevant system dependent details from the user.

Uniform name space: Everybody invokes a resource using the same name. In the

context of DFS, this means that every client uses the same path to access the file.









Unix semantics: A model specifying that operations on a file are immediately

visible to all users/processes.

User: An entity (usually human) in the system that performs actions on objects

through services provide by the servers.

User file space: File space that exists on the user's local system. It is not managed

by DCS. However, a DCS client often acts as a go-between.

Version chain: A sequence of file versions in which, the first version is stored as a

full file but all others are stored as deltas.















Abstract of Thesis Presented to the Graduate School
of the University of Florida in Partial Fulfillment of the
Requirements for the Degree of Master of Science

A DISTRIBUTED FILE SYSTEM (DFS)

By

Prashant Jayaraman

May 2006

Chair: Richard E. Newman
Major Department: Computer and Information Science and Engineering

Distributed Conferencing System (DCS) is a platform that supports collaboration in

a distributed environment. Distributed File System (DFS) provides the file system of

DCS. Distributed file systems face some peculiar challenges. They must provide a

uniform name space for users, along with location independence and location

transparency. A file can be edited simultaneously at different locations. If the changes are

not coordinated, the file may become inconsistent and the resulting unpredictability can

be frustrating to users. An important characteristic of a distributed system is that at

almost all times, some node or another will be unavailable. DFS must ensure that the

system is reliable and highly fault-tolerant.

Currently available distributed file systems cannot support DCS. They are not

designed for an environment where a high level of concurrency is common. They assume

that simultaneous writes are unlikely and do not support an adequate level of concurrency

control. Most file systems make use of primary copy replication which has a large

overhead. Others use read-only replication.









DFS supports a high level of concurrency. It uses an immutable versioning scheme

where, every time a file is closed, it is saved as a new version of the file. This allows

multiple users to edit and save the file simultaneously, without the danger of interleaving

file blocks. At the same time, it makes sure that no changes are lost. DFS provides fault

tolerance and ensures high reliability by replicating files across multiple sites. In addition,

it supports file migration. The immutable versioning scheme also gets rid of the problem

of synchronizing replicas.














CHAPTER 1
INTRODUCTION

Introduction

In the modem world, there is a growing need for collaboration among

geographically separated groups. Platforms and services that assist such collaboration are

in great demand.

Distributed Conferencing System (DCS) is an effort to provide a platform for users

to work cooperatively in real time in a distributed environment. The system is based on a

robust, hierarchically distributed architecture and provides access control, file service,

notification, secure communication and mechanisms for group decision-making [1]. DCS

provides basic distributed collaborative tools (text editors, graphics editors, etc) and can

also support third-party applications. The architecture of DCS is designed to be scalable.

Another major design goal is fault-tolerance. The system is designed to handle network

failures and system crashes by restoring from backups and contacting other sites for the

most up-to-date information.

File management is important in any distributed system. File systems in distributed

environments have to tackle issues not seen elsewhere. A key question is how are files

and directories presented to the user [2]? Another question is what happens when several

users modify the same file at the same time [2]. It is also important to ensure that data is

not rendered unavailable by the failure of a few systems [3]. File permissions are also

significantly more complicated [2]. Finally, performance should be one of the goals of a

distributed file system [2].









Distributed File System (DFS) provides file-handling services for DCS users. It is

designed to allow file sharing and concurrent access of files. It also provides transparency

[2]. In addition, it uses the Access Control Service provided by DCS to enforce file

access permissions [4].

Distributed Conferencing System Services

The first version of DCS (DCS vl) provided only limited services for focused

tasks. One of its drawbacks was that it had very few roles and only users in the voter role

could vote on decisions [1]. Also, it relied on UNIX permissions for security [5].

The second version (DCS v2) provides more services and better communication

primitives. In addition, it supports more voting mechanism and limited Role Based

Access Control (RBAC) [4]. DCS v2 modules have better support for file/object types.

The major services provided by DCS are described below [6].

* Conference Control Service (CCS): This module manages the Collaborative
Group (CoG). It is the first service to be activated and instantiates all other
modules. It uses secure messaging to allow users to login and interact with services.
This module also handles operations like splitting and merging
instances/CoGs/sites, access control requests and add user requests. It is
responsible for creating and deleting CoGs.

* Database Service (DBS): DBS maintains all tables in DCS space. It makes use of
a Database Managements System (DBMS) as the backed. Tables are stored as
partially replicated, distributed databases and group multicast is used to ensure
eventual consistency.

* Notification Service (NTF): NTF provides asynchronous event notification to
registered users. In addition to built-in events, NTF allows users to define new
events. NTF maintains a global and local database to match events to registered
users, along with the delivery method.

* Decision Support Service (DSS): DSS facilitates the resolution of issues by a
group of people with the joint responsibility for them. It maintains decision
templates for that purpose. It allows creation, modification and execution of
templates. If a template requires a vote among a set of users, DSS will contact the
users, get their votes and return the result.









* Access Control Service (ACS): ACS is responsible for maintaining access rights
and protection information for all objects in DCS. Instead of just maintaining
attributes, ACS supports the use of decision templates to allow members to decide
on actions. Applications can register new access types for more fine grained
control. ACS makes use of DBS to maintain state.

* Secure Communication Service (SCS): SCS allows DCS and its users to
communicate securely and reliably. It ensures authenticity, confidentiality and
integrity of all messages. It is responsible for the creation and maintenance of keys
and certificates for sites and users.

* Application Manager (AM): The Application Manager is responsible for
registering and invoking applications that are available for each CoG. The
applications could be DCS aware or DCS unaware. AM maintains a list of
applications for each CoG and makes them available to the user.

* Distributed File System (DFS): This is the focus of the thesis. It manages all files
in the CoG. It provides concurrency control using a versioning scheme with
immutable files. It also provides replication and migration services.














CHAPTER 2
STUDY OF DISTRIBUTED FILE SYSTEMS

Introduction

This chapter surveys some of the popular distributed file systems. A distributed file

system is a file system that supports sharing of files and resources in the form of

persistent storage over a network [2]. The first distributed file systems were developed in

the 1970's. The most popular distributed file systems are Sun's Network File System

(NFS) and the Andrew File System (AFS). In addition to a discussion on these file

systems, this chapter will also provide an overview of some less popular systems.

Network File System (NFS)

Introduction

The NFS protocol was developed by Sun Microsystems in 1984 [7]. It was meant

to be a distributed file system which allows a computer to access files stored remotely, in

the same manner as a file on the local hard drive. NFS is built on the Open Network

Computing Remote Procedure Call (ONC RPC), described in RFC 1831 [8]. Version 2 of

NFS was stateless and ran over UDP [9]. Version 3 introduced support for TCP [10].

Finally, version 4 introduced a stateful protocol [11, 12]. Though NFS is strongly

associated with UNIX, it can be used on Windows, Mac and Novell Netware.

Protocol

NFS provides its services through a client-server relationship. The client and server

communicate through RPCs. The client issues an RPC request for information and the

server replies with the result. NFS was designed to be machine, operating system,









network architecture and protocol independent. External Data Representation (XDR) is

used to translate between machines with different byte representations.

The computers that make their file systems, or directories, and other resources

available for remote access are called servers. The act of making file systems available is

called exporting. The computers, or the processes they run, that use a server's resources

are considered clients. Once a client mounts a file system that a server exports, the client

can access the individual server files (access to exported directories can be restricted to

specific clients). It must be noted that a computer can be both a server and a client at the

same time. An NFS server is stateless. It does not have to remember any transaction

information about its clients. In other words, NFS transactions are atomic and a single

NFS transaction corresponds to a single, complete file operation. NFS requires the client

to remember any information needed for later NFS use. This has meant that NFS has poor

performance over Wide Area Networks (WAN). However, Version 4 has introduced

stateful protocols and compound procedures to improve this situation [2].

Naming and Transparency

A client can mount the exported directory (from the server) anywhere on its local

file system. The purpose of this is to provide clients transparent access to the server's file

system.

Figure 2-1 illustrates the fact that different clients can mount the exported

directories at different locations. In this case, the client Client 1, mounts the directory Dir

A (from Server 1) at /mnt and the directory Dir B (from Server 2) at /mnt/remote. By

contrast, Client 2 mounts Dir B at /mnt and Dir A at /mnt/remote.

An NFS server can mount directories that are exported by other NFS servers.

However, it cannot export these to its clients. The client will have explicitly mount it









from the other server. On the other hand, a file system can be mounted over another file

system that is not a local one, but rather is a remotely mounted one.

NFS supports location transparency but not location independence. This is due to

its decision not to enforce a common namespace. This makes it harder to share files.

Concurrency Control and Consistency

The NFS protocol does not provide concurrency control mechanisms. A single read

or write call may be broken down into several RPC read or writes because each RPC call

can contain up to 8KB of data. As a result, two clients writing to the same remote file

may get their data intermixed.

It is hard to characterize the consistency semantics ofNFS. New files created on a

machine may not be visible anywhere for half a minute. It is indeterminate whether

writes to a file are visible to other sites that have the file open for reading. New opens of

a file observe only the changes that have been flushed to the server. Thus, NFS does not

provide either UNIX semantics or session semantics.

Replication and Migration

NFS version 4 provides limited support for file replication. It requires that an entire

file system be replicated, including the files, directories and attributes. One of the

attributes of a file, contains the list of alternate locations (DNS name or IP address) for

the file system containing the file. In order to access files from a server, the client must

switch over to that server. It must be noted that the NFS protocol does not specify how

replicas are kept up-to-date, each implementation must decide for itself on the best

manner to achieve this.









As explained previously, NFS does not provide location independence. This means

that migrating a file system from one server to another would invalidate all mounts of

that file system.

Caching

NFS version 3 mainly left caching out of the protocol specification. Most

implementations never guaranteed consistency. The time lag, between the time a file was

written to the server, and the time the file was invalidated on the client cache varied from

a few seconds to almost half a minute. NFS version 4 does try to alleviate this problem.

Most clients cache file data and attributes. On a file open, the client contacts the

server to check whether the attributes need to be fetched or revalidated. The cached file

blocks are used only if the corresponding cached attributes are up-to-date. Cached

attributes are generally discarded after 60 seconds. NFS uses both read ahead caching and

delayed write, to improve performance.

Security

Security, in distributed file systems has two parts secure communication between

the computers and controlling access to files.

NFS is built on top of RPC's. Hence, secure communication boils down to the

establishment of a secure RPC. Older versions of NFS had support for Diffie-Hellman

key exchange for establishment of a secure key. Recent versions support Kerberos

(version 4) for establishing secure RPC's. NFS version 4 also provides a general

framework that can support numerous security mechanisms to ensure authenticity,

integrity and confidentiality.

NFS provides a much richer set of access controls than the UNIX file system.

Access control is implemented in the form of access control entries, where each entry









specifies the rights for a user or a group of users. Unlike UNIX, it is possible to specify

the rights of several users and groups.

Andrew File System (AFS)

Introduction

The Andrew File System (AFS) was initially developed by Carnegie Mellon

University (CMU) as part of the Andrew Project [13, 14, 15]. It is named for Andrew

Carnegie and Andrew Mellon, the founders of CMU. Its biggest advantages are security

and scalability. AFS has expanded to over 1000 servers and 20000 clients worldwide

[16]. It is believed that AFS can scale up to support twice as many clients. AFS heavily

influenced Version 4 of Sun Microsystem's NFS. Additionally, a variant of AFS was

adopted by the Open Software Foundation in 1989 as part of their distributed computing

environment. There are three major implementations of AFS Transarc (IBM), OpenAFS

and Arla. AFS is also the predecessor of the Coda file system.

Architecture

Like NFS, AFS uses RPC's as the basis of communication between clients and

servers. AFS can be divided into two components. The first is a group of relatively small

number of centrally administered file servers, called Vice and the second consists of a

large number of workstations called Virtue. Virtue workstations provide access to the

file system. Every client (Virtue workstation) has a process running on it, called Venus.

This allows the client to access the shared file system in the same manner as the local

one.

AFS defines a volume as a tree of files and subdirectories. Volumes can be created

by administrators and linked to specific paths (similar to mount in NFS). Volumes can be









located anywhere and replicated or moved without informing the user. In fact, a volume

can be moved while its files are in use.

Naming and Transparency

AFS provides its clients with a global name space. In addition, clients have a local

name space to store temporary data. From the client's point of view, the file system

appears as one global resource. It is not possible to deduce the location of the file from its

name. In addition, all files have the same path from every client. Thus, AFS provides

both location transparency and location independence.

Concurrency Control and Consistency

Like NFS, AFS uses the Ostrich approach (it is widely and incorrectly believed that

when threatened, the Ostrich buries its head in the sand, hoping that its attacker would go

away). It does not make any effort to resolve concurrency, believing that shared writes

are extremely rare. When two users close the same file at the same time, the results are

unpredictable.

AFS supports session semantics. This means that once client A has saved the file,

any client opening the file after A has closed it, will see the modifications made by A.

This is weaker than UNIX semantics but much easier to support in a distributed

environment.

Replication and Migration

AFS provides replication to read-only backup copies. Even an entire volume can be

replicated. A client system looks to retrieve the read-only file from a particular copy. In

case the copy is not available, it switches to a different copy. This allows both better fault

tolerance and improved performance. AFS allows the creation and migration of copies as









needed. By ensuring that the copies are read-only, replication consistency problems are

avoided.

Caching

AFS divides files into chunks of up to 64 KB, for caching. A large file may have

several chunks, while a small one may occupy just one. The cache tries to establish a

"working set" of files (copied from the server) on the client. In case of server failures, the

local copy is still accessible.

Read and write operations are performed on the local, cached copy. When the file is

closed, the modified chunks are copied back to the file server. Callback is a mechanism

used to maintain cache consistency. When the client caches a file, the server makes a note

of it (stateful) and informs the client when a change is made to the file. The cache is

believed to be consistent while the callback mechanism is operational. Callbacks must be

reestablished after client, server and network failures.

Security

Security is one of the strengths of AFS. It uses a secret key cryptosystem to

establish a secure channel between Vice and Virtue. The secret key is used by the two

machines to establish a session key, which is used to setup the secure RPC.

The process of authenticating the user is more complex. The protocol used for this

purpose, is derived from the Needham-Schroeder protocol [17]. An Authentication Server

(AS) provides the user with appropriate tokens. These are used to establish his/her

identity.

Access control lists are used to control permissions to files and directories. The

Vice file servers associate lists with directories only. Files do not have access control









lists. This makes it simpler and more scalable. AFS supports both users and groups. It is

also possible to list negative rights.

Common Internet File System (CIFS)

In its original incarnation as Server Message Block (SMB), it was invented by IBM

to provide a genuine networked file system. Microsoft modified it heavily, adding

support for symbolic links, hard links and large files and launched an initiative to rename

it as CIFS.

CIFS can run on top of both NetBIOS and TCP/IP protocols. In the manner of other

distributed file systems, CIFS uses a client-server model for communication. In contrast

to other file systems discussed, CIFS also provides Inter-process communication (named

pipes). In addition, SMB servers can share other resources besides file systems.

CIFS cannot really be described as a legitimate distributed system. It merely

provides a way to access resources over a network. It does not provide either location

transparency or location independence. Resources are identified as //server-name/path-to-

resource. Unlike other distributed file systems, it makes little effort to remain agnostic of

the underlying operating system. There is no concurrency control or caching. It does not

support file replication or migration. It does provide UNIX consistency semantics

because the file has only one copy.

Coda

Coda is another distributed file system, developed at Carnegie Mellon University

[18]. It is based on the Andrew File System. Some of the areas that it improves over AFS

are listed below.

In contrast to almost all other distributed file systems, Coda allows the client to

work, even when it is unable to contact any server. In this case, the client uses the local









(cached) copy of the file. It is possible to ensure that the client fills its cache with the

appropriate files before disconnecting from the server. This is called hoarding.

Replicated volumes do not have to be read-only. Coda uses a replicated-write

protocol to ensure that the modified parts are transferred to all servers at the same time.

In case server failures, different servers will have different versions of the file. Version

vectors (similar to vector timestamps) are used to resolve this inconsistency.

The security mechanisms of Coda are interoperable with Kerberos. Also, it has

native support in the Linux 2.6 kernel.

Plan 9

Plan 9 is based on the idea of using a few centralized servers and numerous clients

[19]. Plans 9 allows computer to share any resource in the form of files. For example, the

network interfaces of a machine can be exported as a bunch of directories. Each TCP

connection is represented as a subdirectory of the corresponding network interface.

Plan 9's naming philosophy is similar to that of NFS with some interesting

deviations. Multiple name spaces can be mounted at the same mount point. In this case,

the contents of the two remote file systems are Boolean ORed. Such a mount point is

called a union directory.

Plan 9 implements UNIX file semantics by ensuring that all updates are always sent

to the server. Concurrent writes will happen in an order determined by the server but

unlike session semantics, the writes will never be lost. However, this reduces

performance.

All files have four integers identified with them. Two of these type and device

number, identify the server or device hosting the file. Two others, path and version,









identify the file relative to the server or device. Version is incremented after every

modification.

Plan 9 provides some support for replication and caching. Clients can cache files

and use their version numbers to ensure that the files have not been modified. Users are

authenticated using a protocol similar to the Needham-Schroeder protocol. File

permissions are identical to those in UNIX. Plan 9 has the notion of group leaders. Only

group leader can modify the group's permissions on the file.

xFS

xFS is an unusual serverless file system [20, 21], designed to operate on a local

area network. It distributes storage, cache and control over cooperating workstations.

This arrangement is believed to yield higher performance, reliability and scalability.

xFs defines three types of processes. Storage servers stored parts of data. Metadata

managers store information on locating data and clients form the interface between users

and the file system. The storage servers together, implement a virtual RAID device.

xFS uses system wide identifiers to locate metadata managers. Clients cache blocks

of data rather than files and the cache consistency scheme is similar to AFS. Each data

block belongs to a manager. When a client wishes to modify a block, it obtains write

permission from the manager. The manager invalidates all other cached copies of the file.

Write permission is revoked when another client wants to modify it. This also requires

that the block be flushed and written.

xFS does not emphasize security. It provides the usual access control lists and

requires that clients, managers and storage servers run on trusted machines.









Low Bandwidth File System (LBFS)

LBFS is designed for networks with low bandwidth and high latencies [22]. LBFS

works on the concept that a version of a file has much in common with its previous

version. It also believes that there are similarities between files created by the same

application. The techniques used by LBFS can be used in conjunction with the techniques

used by other distributed file systems (like Coda) to improve tolerance to network

vagaries.

LBFS works by maintaining a large cache at both the client and the server. Files are

divided into variable size chunks. The chunk boundaries are determined by Rabin

fingerprints [23], subject to lower and upper bounds. A Rabin fingerprint is the

polynomial representation of the data modulo a predetermined irreducible polynomial.

When the low-order 13 bits of a region's fingerprint equal a chosen value, the region

constitutes a breakpoint. Assuming random data, the expected chunk size is 213 = 8192 =

8 KBytes This scheme ensures that modifications to a chunk affect only that chunk and

its neighbours.

LBFS indexes chunks using their SHA-1 hash [24]. In case the chunk is present in

both the server and the client, only its hash is sent. Otherwise, the entire block is

transmitted. All data sent on the network is compressed using gzip.

LBFS provides the same session semantics as AFS. Files are flushed to the server

on close. LBFS uses temporary files to implement atomic updates. All writes to the file

happen on a temporary file which is then renamed. This ensures that concurrent writes

can never result in inconsistent files. This is in marked contrast to NFS. The client that

closed the file last will overwrite the changes of the others.









Every server has a public key, which is specified on the client when it mounts the

server. Future versions look to incorporate self authenticating pathnames [25], to

facilitate auto-mounting. All communication between the client and the server is

encrypted with the help of a session key.

Freenet

Freenet is a decentralized, anonymous, censorship resistant, distributed data store

[26]. Unlike most distributed file systems, it is based on the peer to peer communication

model. Freenet allows all participating computers to contribute bandwidth and storage

space. It allows users to publish and retrieve data anonymously.

Files have a descriptive string associated with them. This yields a public/private

key pair. The private key is used to sign the file and the hash of the public key is called

the Keyword Signed Key (KSK). Signed Subspace Key (SSK) allows the use of personal

namespaces and is created in a manner similar to KSK, using usernames instead of file

descriptions. A combination of the username and descriptive string is sufficient to access

the file. Content Hashed Keys (CHK) are created by hashing the contents of a file. A

Content Hashed Key can also be used to access a file. A CHK points directly to a file, but

the KSK and SSK combination uses an indirection mechanism.

Freenet uses a routing table similar to the Internet Protocol. The client retrieves the

file by specifying its binary key. It then checks whether it has knowledge of the node that

should be contacted to get this file. If no node is found, it looks up the nearest key

lexicographicc) in its table and forwards the request to that node. This proceeds until the

key is located or the TTL expires. In order to preserve anonymity, nodes along the

successful path randomly designate themselves as sources of the file and may cache a

copy of the file.









To insert a file, the user computes a binary key for the file and makes a request for

a file with that key. In case there is no collision, the file is inserted into the node and all

nodes along the path of the previous query are informed of the new key. Again, nodes can

randomly re-designate themselves as hosts. Each file in Freenet is encrypted using a

random key that is published along with the binary key.

The routing heuristic and insertion algorithm ensure that files with similar keys are

clustered. This also ensures better performance of both of them. Freenet is closer to an

anonymous bulletin board that a distributed file system. It has been included here because

of its unusual design and goals.











Server 1


Server 2


Dir A
--A--- --
'>




-I I


Client 1


Dir B
...........



D3

:D3 i



Client 2



mnt






Dir B :remote





...--------I


----_ ......... ...... Dir A








D3 I I
1 ," D2 : DirB : : D3 : ,



D -I- -- -I I

....... | D2 |




Figure 2-1. Example of NFS clients mounting the same directories at different mount
points.


---
D


\














CHAPTER 3
REQUIREMENTS AND DESIGN

This chapter presents the requirements of Distributed File System (DFS) in detail

and goes on to describe the design of DFS. In each section, it details the approach taken

in DFS along with the reasons for it.

Requirements

DCS is designed to simplify collaboration among geographically separated users.

DFS provides the underlying file system for DCS. Other modules of DCS interact with

DFS to provide the complete user experience. So, the requirements were formed on the

basis of discussion among the members of the DCS Research Group

Naming and Transparency

DCS users can reside in different locations and the file system should allow users

to share files. This requires a conference-wide shared name space. In addition, clients and

servers will often store data related to them. This requires that there be a local name

space, in addition to the conference-wide name space. So, DFS must provide both local

and conference-wide name spaces.

Users join DCS conferences using Conference Control Services (CCS) [27]. It is

possible for a user to join multiple conferences through CCS. DFS must present the user

with a hierarchical view of the file system.

DCS conferences span several sites. It is very likely that files are spread across

multiple sites. DFS should ensure that the user does not have to inform it about the

location of the file. In other words, filenames should be location transparent.









The name of the file should not depend on the client that is being used to access the

file. This ensures that users have the freedom to connect from any client and that they

don't have to remember the mount points of that client.

Concurrency Control and Consistency

Concurrency control is very important in a distributed file system. DFS should be

designed to allow concurrent access as best as possible. Users must be able to access and

write files concurrently. It is up to DFS to ensure that changes made by one user do not

interfere with the changes made by others.

Most distributed file systems provide an approximation of session semantics. It

would be highly desirable for DFS to provide the same consistency semantics. New files

should be seen when the client closes the file.

Replication and Migration

Fault tolerance and high availability are two of the goals of DFS. In order to

achieve these goals, DFS must provide file replication. If a site is unavailable, DFS

should be able to retrieve the file from its replicas. DFS should also enforce replica

transparency [28]. This means that normally, the client should be unaware of the

existence of replicas.

DCS is designed to spread across Wide Area Networks (WANs) and different

administrative domains. Replicating a file on a server close to the client can decrease

access times dramatically. Migration also helps DFS improve performance and

availability. DFS should allow migration from far away or unreliable servers, to closer,

more reliable servers.









Caching

As mentioned previously, DCS is designed for WANs. This introduces the problem

of network latency. DFS must use caching aggressively to reduce access times. DFS

should also be aware of the fact that while servers are connected through WANs, the

connection between a client and a server is often over a Local Area Network (LAN). If

both the server and the client maintain a working set of the files that they use, it is

possible to reduce the traffic over WANs.

Security

DCS is spread across multiple administrative domains. DFS must ensure that it

makes it as hard as possible for an intruder to play havoc with the system. This requires

that all communication between the client and the server (and between the servers

themselves) be protected. Authentication, integrity verification and confidentiality must

be provided by the communication service.

Another part of security is to regulate the behaviour of legitimate DCS users. DFS

must have very flexible and robust permission schemes. In addition, it should allow users

to create new permission types. This ensures that DFS is able to handle scenarios that

were not envisaged during design and development.

As described in Vijay Manian's work on DCS access control, clients are bound to

roles [4]. Each role has specific privileges. One of the desired privileges should be

viewing the contents of a directory. Only those files, for which the user has the privileges,

should be shown in the directory listing. This can be used to give each a user a different

view of the same directory.









Design

Communication Architecture

The design of DCS was made around the idea of separate clients and servers. DFS,

as a module ofDCS has to maintain that architecture. DFS has dedicated servers for file

management and file services. Client processes run on client machines and pass on

requests to DFS servers.

It must be noted that DCS does not preclude the client and server being on the same

machine and DFS must not impose this requirement. It only requires that the client and

server be different processes. DFS servers and clients must be able to run on the same

machines as the clients and servers of other DCS modules. The current design of DFS

allows a site to run multiple instances of DFS, one for each conference that it participates

in. It also allows servers to connect to a cog on the fly.

Stateful/Stateless Server

As previously mentioned, stateless servers are simpler to design and have better

fault tolerance as compared to stateful servers. They can be easily plugged in and taken

out of a network. Stateful servers are often shutdown in a 'dirty' state and must restore

themselves to a consistent state. For these reasons, DFS servers do not maintain state

information.

File Access Model

File Access Model refers to the manner in which the file system lets the client

access the file. Distributed file systems offer one of the following file access models

* Remote access: In this model, all operations are performed directly on the remote
file. NFS is one file system that offers this type of access.

* Upload/download: All files are downloaded by the client when opened and are
uploaded back to the server when the client closes the file.









The upload/download model lends itself more easily to caching and is quite popular

among distributed file systems. DFS uses the upload/download model. All files are

downloaded to the client for use and are uploaded back to the server when they are saved

or closed.

Concurrency Control and Consistency

Concurrency is very important in a distributed file system like DFS, which is meant

for Wide Area Networks (WANs). NFS does not deal with concurrency at all and other

file systems like AFS and Coda have inadequate concurrency control. DFS manages

concurrency with the help of an immutable versioned file system.

All files are immutable. Changes made to a file are stored as a new version of the

file. This ensures that concurrent writes by multiple clients do not interleave and make

the file inconsistent. Another part of concurrency control is to ensure that writes by all

clients are incorporated. When multiple clients open a file and write to it, it should not be

the case that only the changes made by the user who closed the file last, are incorporated

while all others are lost. This is tacked by the versioning system. Every time a change is

made to a file and stored, DFS stores the modified file as a new version. Changes made to

the file by different users are stored as different versions. Eventually, DFS will offer tools

for automatic merging and conflict resolution.

Versioning in DFS is different from versioning systems such as CVS and Perforce.

Versioning systems are designed to function as depots (usually code). They do not

provide transparent access. All files must be explicitly checked out and check in. They

are not optimized for the high number of requests that a file system must handle.

In DFS, for a user to see changes made by others, the user has to specify the

version, after it has been saved by the other user. This peculiarity of DFS makes it harder









to evaluate its consistency semantics. Other immutable file systems do not maintain a

versioning scheme and so, they seem to follow session semantics. In DFS, if the version

is specified properly, it is possible to look at the changes made by the other user. Thus,

from the user's point of view, the file system follows session semantics.

Versioning

It has been discovered with versioning systems that consecutive versions of a file

have a lot in common. It is more economical to store just the changes between

consecutive versions, rather than the versions themselves. All versioning systems

maintain their files as deltas. The delta of two files is the set of changes which, when

applied to the first file, yield the second file.

The use of a delta scheme will greatly reduce the storage demands of DFS. A delta

compression scheme could use forward or reverse deltas. These are discussed next. At the

end of the section, the final design decision is presented with reasons why it is chosen.

Forward deltas

If forward deltas are stored, only the original file is stored in its entirety. All other

versions are constructed from this version using a string of deltas. The major

disadvantage of this scheme is that every time the latest version is needed, it must be

reconstructed step by step from all the deltas. This makes the retrieval of new files

painfully slow. The SCCS versioning system uses this approach.

Reverse deltas

In a reverse delta, the latest version is stored in its entirety and all other versions are

stored as deltas. The older versions can be recreated by applying the deltas on the latest

version. This scheme requires that the latest version of each branch be stored as a

complete file. There can be multiple ways to generate a file and the paths might be of









different lengths. This scheme uses storage space less efficiently than forward deltas.

However, retrieval of the latest version is quick.

Previous design

Previous versions of DFS used a combination of forward and reverse deltas. The

latest version on the main branch was kept as a complete file. All other versions could be

obtained from it.

Other versions on the main branch were stored as reverse deltas of their subsequent

versions. In case of branches in the versioning tree, forward deltas were used.

This scheme allowed the system to quickly retrieve the latest version of the main

branch. It also reduced storage space by storing only one full version.

On the other hand, there were a few problems with the scheme. It was hard to

define which branch was the main branch. Also, this scheme made file replication and

migration very complex. These problems were tackled in the new design.

Current design

The new versioning system is designed with two goals in mind. The first is to

reduce server load and the second is to facilitate file migration. All branches are born

equal, none more equal than the others. The main branch of a version is simply the first

child of the version. The tips of all branches are stored as complete files. At the same

time, all deltas are forward ones. It is likely that the latest versions of a branch will be

accessed far more often than the older ones. By keeping them as complete files, DFS

reduces the load on the server. Also, for this scheme, both forward and reverse deltas will

work equally well. However, forward deltas are more intuitive and so, are used in the

scheme. Some files may be stored complete, at all times. The versioning system

designates such files based on the following rules:









* The version 0 is a full file

* Any version that is not on the main branch, i.e. if it is not the first child of its parent
is stored as a complete file

* Any file that not stored in exactly the same servers as its parent is stored as a
complete file

* If the version chain exceeds its maximum length with this file, the file is always
complete.

The first rule is obvious and needs no explanation. The second rule is to facilitate

migration. Let the term last full ancestor of a version denote the last version (in the same

branch), older than the given version, that has been stored as a full file. In case this rule is

not applied, it would mean that besides the tip of a branch, only the root of the tree is a

full file. In this case, to migrate an interior version, from one server to another, it would

be necessary to migrate all versions except the tips of branches. This rule makes sure that

no two branches have the same last full ancestor.

The third rule helps in keeping the replication procedure simple. Consider the

consequences of not having this rule. Now, it is possible that a delta will be stored on a

server and its parent will be on another server. In order to retrieve the file, both the delta

and the parent are necessary. This means that both servers should be alive and this

decreases availability. In order to avoid these situations, this rule is applied.

The last rule ensures that there are no long series of deltas. In the course of

operations, it is likely that a complete file will be followed by a long series of deltas. To

retrieve the last version, all preceding versions have to be reconstructed. Also, migration

will require moving an enormous amount of data because migrating a version involves

not only moving that version, but also all versions that it depends on and all versions that

depend on it.









Naming and Transparency

DFS supports a conference-wide name space and a local name space. The

conference-wide name space allows users to share files and directories on the file system.

The local name space is used to store site specific information and temporary files.

DFS provides location transparency and location independence. It is not possible to

learn the file's location from its name. In fact, the only way to ascertain the location of

the file is to check the metadata. This makes it is possible to migrate a file from one

server to another, without changing its name. Location Independence and location

transparency allow DFS to provide a centralized view of DCS files, even when they are

on totally different sites.

DFS provides hierarchically structured directories. The path of each file starts with

/DCS/. This allows client side applications to provide a global view

of all conferences. It is possible to populate /DCS/ with all the conferences that the user is

connected to and make it seem like /DCS/ is one directory on the local file space that

provides access to conference files. Thus, client side applications can provide a global

name space that merges multiple conferences. These conferences don't have to have any

site in common. The files in each conference will be widely dispersed as well.

Storing Metadata

Location transparency and location independence are provided with the help of a

file table, which maps a filename to its metadata and stores file metadata. This structure

is also essential in order to present a uniform name space. A uniform name space also

requires that the metadata on various sites be consistent.

DFS maintains multiple versions of the same file. For the sake of simplicity, let the

term physical file refer to each version of a file. This term is used because, on the









operating system of the site, each version appears as a file. Another layer of the file table

groups different versions together. This will be referred to, as a virtual file. Virtual files

have no existence on the host operating system. A virtual file is just a collection of

metadata related to every version of a file. The attributes of a physical file are listed

below:

* Location of the file within DCS directory space

* Version of the file

* Type of file

* Owner of the file

* Size of the file

* Time the file was created

* Mode at which the file can be opened

* Location of replicas

* Is this a full file

* Should this file always be a full file

* Has this file been deleted

* Parent version of this file.

Currently, a virtual file maintains only four attributes

* Type of virtual file (File or directory)

* Latest version

* References to all versions (file)

* References to other logical files (directory).

The design of the file table is very important in ensuring performance, scalability

and error recovery. Two options will be considered below for this purpose.









Symbolic links

One of the ways of representing this information is analogous to symbolic links.

This is illustrated in Figure 3-3 and Figure 3-4. Figure 3-3 shows the physical location of

conference files. Figure 3-4 illustrates the manner in which directories and links are used

to provide a uniform view of the file system.

In this example, the conference Confl is split across two sites Site 1 and Site 2.

Figure 3-3 shows the distribution of the files among the sites. Site 1 has the files Filel

and File2 along with the directory Dirl and its contents. Similarly, Site 2 has File3, File4

and the directory Dir2.

Figure 3-4 presents the uniform namespace, as seen from the site Site 1. Site 1 does

not have the files File3 and File4. In order to keep the location of the files transparent, it

creates symbolic links File3 and File4. These symbolic links are presented in bold in the

figure. File3 and File4 contain information that identifies them as symbolic links and also

contain the location of the server that actually stores the file.

In order to access the file, the server opens the file, realizes that it is a symbolic link

and obtains the location of the server that stores the file. It then contacts the server to

obtain the file.

It can be seen that the directory Dir2 is not shown in bold. This is to indicate that

Site 1 does create the directory Dir2, even though all its contents are symbolic links.

File Information Table (FIT)

It is also possible to organize metadata and present a uniform name space using a

File Information Table (FIT). This table stores all the attributes of the file, including its

path and physical location. Each site has a copy of the FIT. Upon receiving a request for

a file, the server searches through the FIT to locate the file and return its physical









location. In case the file is on the same site, it returns the data. If the file is on another

server, it requests the other server for the data.

The actual arrangement of files on a site does not bear any relation to the view

presented to the clients. It is the task of the FIT to map the logical view to the physical

location. It is not necessary to create directories or symbolic links. The FIT can be

retained in memory for fast access. A site can be brought up to speed by just sending it

the FIT.

Previous design

Previous implementations of DFS chose the FIT approach for its simplicity. The

scheme was designed with the Databases Service (DBS) in mind [29]. DBS would keep

the FIT consistent by serializing all changes. DBS uses causal order protocol to

implement reliable atomic multicast, ensuring that concurrent changes are serialized in

some order.

After a file is created, few of its attributes would change. The static and dynamic

attributes would be maintained in different tables. Either the site that created the file or

the home site of the conference can own the rows in DBS and serialize changes.

There are several advantages in using DBS to maintain FIT. First, it takes a great

load off the hands of DFS. Besides, DBS already maintains tables for other modules.

Unfortunately, DBS was not fully functional at that point.

This required DFS designers to implement their own methods for FIT consistency

and a FIFO scheme was finally agreed upon. Here, every entry in the FIT has an owner

and the owner is responsible for communicating changes to all servers. Servers must

ensure that they process messages from a site, in the same order in which they were sent.









Instead of a causal order protocol, only FIFO consistency is required. The current

implementation of FIT is substantially similar.

Proposed design

It can be easily seen that both DBS and FIFO have their disadvantages. Their

biggest problem is that neither is particularly scalable. DBS is too restrictive and in the

case of the FIFO scheme, the number of messages required to maintain consistency

increases nearly exponentially. A more scalable FIT scheme is presented next.

In this scheme, every server maintains the root directory contents. The contents of

other directories are maintained only if they lie in the path of a file located at this server.

This means that if a server has version 2a.0 of the file /abc/def/xyz/pqr.txt, it has the

contents of/abc, /abc/def, and /abc/def/xyz in its local FIT along with information about

the different versions of /abc/def/xyz/pqr.txt. In order to get the listings of other

directories, the server contacts other servers.

In this scheme, the virtual files (collection of versions) will have to maintain these

properties.

* Type of virtual file (file or directory)

* Contents of the virtual file

* Is the virtual file maintained locally

* List of servers maintaining this virtual file.

If a directory is maintained on this server, then the attributes of all its virtual files

are valid. Any change in the directory must be propagated to the list of servers. If the

directory is not maintained locally, the attributes of the directory must be obtained from

the list of servers maintaining this directory.









This leads to a situation in which, the directories whose contents are more widely

distributed are maintained on many servers. Normally, this means that directories with

more files are more heavily replicated. This does make sense because the FIT should be

at least as reliable as the file replication system. Another characteristic of this scheme is

that higher level directories have higher replication factors. The directory /games/pc is on

at least as many servers as /games/pc/Microsoft.

Operations on the FIT are complicated by the fact that the FIT must be fetched

from other servers in some cases. The various operations supported by the FIT are

described below.

Getting the attributes of a directory/file. If the given directory is maintained

locally, it is trivial to obtain the information from the local FIT. Obtaining the attributes

from another server is more challenging.

Consider a directory /books/fiction/humour/20thCentury/English/Wodehouse. Let

the server (Server A) have the directory listing up to /books/fiction. On receiving a query

for the entire path, the local FIT realizes that it can resolve up to /books/fiction. As

described previously, it will also have the list of servers that maintain the directory

/books/fiction/humour. It passes on the path to one of them and asks that server to resolve

it as much as it can. This server (Server B) might resolve it till

/books/fiction/humour/20thCentury and ask Server C to resolve the rest and pass the

result to Server A.

Server C will be one of the servers on the list of servers maintaining

/books/fiction/humour. It will be much faster if Server A could directly contact Server C

for the information. However, Server A cannot know that one server is better than









another for resolving this path. In theory, this information could be cached, so that from

now on, Server C is contacted for

/books/fiction/humour/20thCentury/English/Wodehouse.

The interaction between servers could be iterative or recursive. In the iterative case,

Server B gives Server A the address of Server C and asks it to contact Server C for

resolution. In the recursive implementation, Server B queries Server C on behalf of

Server A and passes on the result to Server A. The recursive version has a cleaner

interface than the iterative version. However Server A might timeout on long paths.

Creating a directory. The first step in creating a directory is to obtain the

attributes of the parent directory. Then, all servers having the parent are informed that a

new directory is being created and the list of servers maintaining the new directory is left

empty (to signify that this is an empty directory).

Creating a new DFS file. Creating a file is much more complicated than creating

a directory. The server must first check whether it has the entire path on its local FIT. If it

does not have it locally, it has to obtain information on all directories in the path and then

inform other servers that it will also maintain all the directories in the path of the new

file. Next, it needs to store the file (version 0 of the DFS file) and inform all servers

maintaining that directory, of the new file.

Replication is one of the services provided by DFS. The server (Server A), which is

connected to the client creating the file, is responsible for the file. In this case, after

creating and storing the file on all servers, Server A informs other servers about the

existence of the file and the location of its replicas. All files with replicas must update

their FIT, to include information about all directories in the path of the file. It must be









noted that Server A may not be one of the servers which store the file. This makes it

significantly complicated.

Creating a new version. Normally, DFS stores a new version as a delta relative to

the previous version. However, a version might be stored as a complete file if the version

chain length is reached or if it is the first version of a new branch.

In case a version is stored as a delta, it has to be stored on the same servers as its

parent. It is fairly simple to do this because FIT of these servers already has information

on the file. On the other hand, creating a version that is to be stored as a full file is just as

complicated as creating a new file.

Deleting a file. All servers that maintain the directory (the file is in) are informed

to remove the file from the FIT. Next, each server that stores the file is asked to delete it.

Now, these servers determine whether they need to maintain any of the directories in the

path of the file. This involves determining whether the server has replicated a file in any

of the directories (or their children) in the path of the deleted file. Once a server

determines the directories it doesn't need to maintain, it informs the other servers

maintaining those directories to remove it from the list.

Cog Policy

All DCS services operate at the cog level. It is imperative that they provide means

to set policies that affect the entire cog. This gives administrators a way to broadly define

policy. DFS allows users to set CoG-wide policies, which allow a coarse level control

over its operations. Cog policies can also be used to set conference specific environment

variables. It is possible for clients to set and query for the values of specific strings. This

is analogous to environment variables in Windows and UNIX. Interestingly enough, its









initial purpose was to help store replication policies and was simply adapted to this new

role. The default cog policy strings are

* VersionChainLength: The maximum length of a version chain in the cog. Version
chains are a sequence of file versions, where only the oldest version is stored in file
system as a full file. All other versions can be obtained by applying deltas.

* Writelnterval: This specifies how often the FIT and cog policy are written to disk.

It is possible to specify replication policy for the cog. This policy defines the

number of replicas that a version should have. The number of replicas depends on the

type of the version. For example, it might be specified that all ASCII files have 5

replicas. The cog policy manager ensures that cog policy changes are propagated to all

sites.

User Policy

DFS also allows the user to set policies that apply only to him/her. In a manner

similar to cog wide environment variables, applications can also take advantage of user

specific environment variables. User policies are a fine grained way of controlling DFS

operations.

As described previously, DFS uses a versioning scheme with immutable files. One

of the personal policies that a user can set is, to decide which version he/she sees when a

file is opened. The user can specify either of the two options listed below.

* Get the latest version that was seen by the user (default)

* Get the latest version created.

It is possible for the user to override the policy by specifying the version while

asking for the file. For the sake of clarity, consider the following example. Let user A and

user B, be working on the same version of a file. After a while, user A closes the file.

This file is now saved on the server. User B is not affected right now and proceeds to









finish his/her work and then save the file. When user A opens the file again, one of the

three things might happen

* The user policy can be set to get the last file that the user opened. In this case, the
user sees the file that he/she saved

* The user could be given a copy of the latest version of the file. Here, the user sees
the version created by B

* The user could specify the version number to avoid any ambiguity.

DFS also has a personal history manager which keeps track of the latest version of

every file that was accessed by the user. The server can consult the history manager to

obtain the version number, in case the user wishes to see the last accessed version.

User policies also allow users to specify replication policies for files created by

them. In a manner similar to cog replication policies, the user can specify the number of

replicas for every type of file.

Migration

DFS is designed to operate over Wide Area Networks (WANs). WANs are

characterized by long latencies and low bandwidth. In addition, one characteristic of a

distributed system is continuous partial operation. The entire system is almost never

operational simultaneously [28].

File migration is one of the approaches taken by DFS to mitigate these problems.

DFS can move files from one server to another, to increase availability and decrease

latency. Migration is complicated by the fact that files are often stored as deltas. Moving

just the delta does not accomplish anything because DFS will need to access the earlier

server to reconstruct the file. So, DFS migrates a version along with all versions that

depend on it and all versions that this version requires. This list of versions is called a

version chain. The first version in a version chain is a full file.









Migration is also simplified by versioning rules that place limits on the length of a

version chain. This reduces the amount of data that must be transmitted to migrate a

version. Another rule ensures that no two branches have the same last full ancestor.

Replication

File replication is one of the ways of improving performance and availability of the

system. If a server that maintains a file is unavailable, DFS looks to get the file from

another replica. As long as at least one replica is available, DFS will retrieve the file.

DFS hides replication detail from clients and maintains replica transparency. Since files

are stored as immutable versions, no replica consistency issues have to be addressed.

Some design options are considered below.

Replicate entire file

In this option, the file (a delta or a full file) is considered as the unit of replication.

The entire file is replicated at a site. The advantage of this scheme is its simplicity.

Replicate portions of file

In this case, a file (a delta or a full file) is split into blocks. The size of the blocks

can be fixed or variable. Each block is a unit of replication and is spread among the sites.

The file is reconstructed by obtained the blocks from the replicas. This is analogous to

RAID devices.

RAID allows the use of error correction codes. It is not necessary to obtain all

pieces to reconstruct the file. It is also far more complex and requires the system to

maintain a lot of metadata.

Replicate on create

Replication can be performed as soon as a file is created. The server queries user

and cog policies to obtain the number of replicas. This is determined by the file type. An









important file can be replicated at a lot of sites to increase availability. A less important

file can have fewer replicas, to save space.

Replicate on open

It is also possible to replicate a file on a site, when the site needs it. In this case,

when the site obtains the file from another site, it checks whether the file has as many

replicas as required for a file of that type. In case it has fewer replicas, a replica is created

at this site and the FIT is updated.

Design decision

In DFS, the entire file is replicated when the file is created. Replication of entire

files was chosen because of the simplicity of the scheme. It also makes other operations

(like migration) easier. Splitting the file in several locations could force the server to

connect to more remote sites.

Simplicity is also one of the reasons for choosing to replicate the file on creation. In

the replicate on demand scheme, it does not make sense to replicate the delta. All replicas

should be complete files. So, there would be only one delta and all other replicas would

be full files. This has two side effects. The first is that it increases storage requirements

and the second is that it creates a situation where all replicas are not the same. All this

can be avoided by replicating a file while creation. It also ensures that a file is not lost,

even if it has not been accessed and the site that created it has crashed. Replication on

creation ensures that the replication policy at the time of creation is followed. In case of

replication on access, the decision to create a file will depend on the policy at the time of

access. This is more unpredictable.









Caching

DFS is designed to use caching aggressively to improve performance. This is very

significant over WANs. DFS supports caching on both the client and the server.

Client caching

DCS client process maintains a cache on the client machine. When asked for a file,

the client obtains the file from the server and stores it in the cache directory. All

operations are performed on the file. When the file is closed, the new file is pushed to the

server, which stores it.

The advantage of an immutable versioned file system is that a version of the file

will never change. It can be cached on the client as long as necessary, with hard disk

space as the only constraining factor.

Clients do not cache attributes because they might change over time. It is preferable

to get the attributes from the server.

Server caching

In contrast to most distributed file systems, DFS allows both replication and server

caching. Server caching is similar to replicate on demand. When a file is obtained from

another site, the server caches the full file for future use. The biggest difference between

caching and replication is that, when a file is replicated on a server, the server must have

the means to retrieve the full file without having to contact any other server. The server

cannot remove the file from its local storage unless the file has migrated to another

server. On the other hand, the contents of server cache do not concern any other server. A

server can clear its cache when it needs space. Server and client caching are used to

establish a working set of files on the respective machines.









Security

DFS security has two components. The first is, to provide secure communication

between the client and the server. Secure Communication Service (SCS) is a DCS

component designed to ensure authenticity, integrity and confidentiality of all messages.

It also provides ways of authenticating users. Using SCS for this purpose will greatly

simplify DFS design.

The second component is to ensure that users are not allowed to perform

unauthorized actions. Modern distributed file systems use access control lists for this

purpose. In DCS, Access Control Service (ACS) is responsible for maintaining rights and

protection information. ACS is extremely flexible and can even allow voting. DFS uses

ACS to maintain the access control lists. ACS also allows users to create new attributes.

File System Interface

DFS provides an interface to the file system located at the server. This interface is

used by clients and other servers. The server-client interface is described below:

* create: If the function is provided with data, it stores the data as a new file. In the
event that no data is provided, a blank file is created.

* readVersion: This command is used by clients to obtain the contents of a DCS file.
The version is determined by the versioning policy. The user can also specify a
version explicitly.

* save: It saves the given data as a new version of the file. The parent version must
be specified.

* deleteVersion: This marks a given version of a file as deleted. The file is not
physically deleted. It is marked in the FIT as unavailable.

* undeleteVersion: The function reverses the action of DeleteVersion. It marks the
version as available.

* deleteFile: It physically removes all versions of a file from all sites. It is not
possible to retrieve the file.









* copy: This command asks the server to copy a version of a file as another file. The
new file is not just a delta. It is the full contents of that version.

* move: This command renames a file or directory. All versions are renamed.

* makeDirectory: It creates a new directory in the file system. This operation merely
updates the FIT. No directory is created on disk.

* removeDirectory: This command removes an empty directory. Again, this
operation modifies only the FIT.

* listFile: This command lists the attributes of all versions of a file.

* listDirectory: It lists the contents of a directory. A directory can contain files or
other directories.

* migrate: This command migrates the given version to a server. Along with the
version, its entire version chain is moved. It is possible to specify the server from
which the file is to be moved.

* getCogPolicy: This command allows the user to view cog policy.

* setCogPolicy: This command allows the user to modify cog policy.

* getCogReplicationPolicy: This command allows the user to view cog replication
policy.

* setCogReplicationPolicy: This command allows the user to modify cog replication
policy.

* getPersonalReplicationPolicy: This command allows the user to view personal
replication policy.

* setPersonalReplicationPolicy: This command allows the user to modify personal
replication policy.

* logout: This command flushes user related information to disk.

* isDirectory: It returns true if a given path is a directory.

* isPresent: This command returns true if a given path exists.

The server: server interface is described next.

* getSiteList: This command returns the list of sites in the cog.

* readFile: This command reads a version of a file.






41


* writeFile: This command writes a version of a file.

* deleteFile: This command deletes a version of a file.

Conclusion

The chapter presents the requirements and the design of DFS. All options have

been documented, even those which have not been used. This allows future designers to

modify the design if they are not applicable any longer.

























rI .1I
I 3.1
L _


Figure 3-1. Forward deltas. The dashed boxes represent delta files and the solid box
represents a complete file.


r-----
I 1
L__- __


Figure 3-2. Reverse deltas. The dashed boxes represent delta files and the solid boxes
represent complete files.































Figure 3-3. Actual location of files in a conference


Filel File2 File3


Figure 3-4. Virtual view of conference from site 1.










Root Directory






Dir A Dir B Dir C
Not Local Local Not Local
Server B -- ir B1 Server B
Server C Dir B2 Server I
Server F File BT3 Server M
File BT4
Server Z
Server L
Server 1




Dir Bl File BT3 File BT4
Local File File N
Dir Bl.A-- Attributes Attributes


Figure 3-5. Sample FIT


Dir B2
Tot Local
Server I


Dir B 1.BI
Server Z
Server L


r
Dir B1.B
Not Local
Server Z
Server L


-







Dir B .A
Not Local
Server Z














CHAPTER 4
IMPLEMENTATION

This chapter describes the implementation of DFS. It starts with a discussion of the

minimum requirements and the interaction of DFS with other DCS services. Details of

DFS architecture, with a description of the server, client and DFS applications conclude

the chapter.

Requirements

DFS has been implemented in Java. J2SE 5.0 or greater should be installed on the

system. DFS code is in pure java and can be run on any operating system and architecture

without changes to the code. However, the versioning system uses BSDiff, BSPatch and

Bzip for delta compression. These tools are available for Windows and all flavours of

UNIX.

Interaction with Distributed Conferencing System Services

Interaction with Conference Control Service

Conference Control Service is the lynchpin of DCS. Like most DCS services, CCS

has been designed using client-server architecture [27]. CCS provides a graphical tool -

the User Manager (UM) which functions as a CCS client. User Manager allows the user

to log in to the conference and use the services provided by the CCS server. The DFS

client cannot connect to the server until the user's credentials have been verified by CCS.

On the server side, requests to the DFS server are often routed through CCS. Current

work on DCS is focusing on integrating DFS with CCS.









Interaction with Access Control Service

ACS is used by DFS to ensure that users do not perform any unauthorized action.

When a file is created, an entry is made in ACS tables, identifying the file and specifying

the permissions of different roles. Also, when an operation on a file or its metadata is to

be performed, ACS is consulted to determine whether the user has the specific

permissions.

Interaction with Secure Communication Service

SCS is a DCS service that allows other DCS modules to communicate safely. It

uses cryptographic techniques to guarantee message authenticity, integrity and

confidentiality. SCS also maintains client and server certificates and changes keys

periodically. DFS uses SCS to provide secure communication between DFS servers and

between DFS clients and servers.

Communication

DFS has not been completely integrated with SCS. This has forced DFS to

implement its own communication module. All communication between the DFS client

and the DFS server takes place as a Java Remote Method Invocation (RMI). DFS Server

object is registered with RMI registry and the client binds to it. The client invokes the

appropriate methods for various operations.

Communication between servers also takes place using RMI. Distributed File

System Server helps in initializing new servers that wish to join the conference. The FIT

Manager also binds to the registry. This helps it contact other sites and maintain

consistency. The Replication/Migration Manager registers with the RMI registry so that it

can deposit and retrieve replicas on other sites. Both User Policy Manager and Cog

Policy Manager bind to the registry to ensure policy consistency throughout the cog.









Distributed File System Architecture

The implementation architecture of DFS is shown in Figure 4-1. The components

in bold communicate with the corresponding modules on other servers to maintain

synchronization. The details of the modules are given next.

File Information Table Manager (FITM)

File Information Table Manager maintains and updates the FIT. It is also

responsible for synchronizing the FIT with other servers. It ensures that the site is

consistent and has an up-to-date list of files. When the server is started, it contacts other

servers to get the latest FIT and when the server is shutdown, it writes the FIT back to the

disk.

The FIT is stored in a file on the directory DFS_ROOT/system. The filename is

specified in the property file. For the sake of convenience, the list of attributes that it

maintains is reproduced here.

* Location of the file within DCS directory space

* Type of virtual file (file or directory)

* Version of the file

* Type of file

* Owner of the file

* Size of the file

* Time the file was created

* Mode at which the file can be opened

* Location of replicas

* Is this a full file

* Should this file always be a full file









* Has this file been deleted

* Parent version of this file.

The first attribute stores the DCS path of the file. The /DCS/

prefix is stripped out of the path name. They next attribute identifies whether the virtual

file is a file or a directory. Directories have only a few attributes associated with them.

Files have all attributes and can have multiple versions. There is a similar attribute, type

of version. Most DFS policies are set according to the type of version. Different versions

of the same file can be of different types. For example, a document with the format of a

letter might be of template type but after it has been filled up, its type can become letter.

This is set by application at the time the new version is written. The FIT does not

maintain file permissions. The last accessed time is not maintained because this places an

unnecessary synchronization load on the servers. The attribute mode is not currently

implemented. It can specify the mode in which the file should be opened read-only,

binary, can cache, etc.

One copy of the FIT is maintained in memory for fast access. Hash tables are used

because they have an order of 0(1) access time. The FIT is periodically written to disk.

This interval is configurable.

Version Naming Manager (VNM)

Version Naming Manager generates version names for new versions of a file. The

generated version name depends on the parent version and the number of its siblings.

Version naming in DFS has the following characteristics.

* The version zero is the first version of a file.

* Siblings are versions that are derived from the same parent. Siblings can be
identified by the fact that they have everything in common until the last sequence
of alphabets. In Figure 4-2, V2, V2a.0 and V2b.0 are siblings.









* Once all alphabets are used up, the next sibling is V2aa.0 and so on.

* To obtain the parent of a version, decrement number after the last period.

* However, if the number is 0, then remove the 0, period and the alphabetic portion
and then decrement the number, to obtain the parent. The parent of V2b.la.0 is
V2b.0.

Figure 4-2 shows a version tree in DFS. Versions marked in bold are the first

versions of their sub tree. DFS stores all these file in full. Version Control Manager

(VCM) obtains these file names from VNM and then stores the file. Version Naming

Manager also has routines to determine whether a child is on the main branch of its

parent. Using this information, VCM decides whether the file should be stored as a delta

or as a complete file.

Cog Policy Manager (CPM)

The Cog Policy Manager is responsible for maintaining general cog policy. In

addition, it also maintains specific policies like the cog replication policy. Cog policy

values can be queried by applications. It is possible for applications to modify the policy

as well. Replication policy determines the number of replicas of a file. This information

is used by the Replication/Migration Manager (RMM).

Policy values can be modified using the setPolicy interface which is exposed to the

client. The getPolicy interface is used to obtain a particular policy. Cog Policy Manager

is also responsible for maintaining synchronization with other sites. When cog policy is

modified by a user, CPM communicates this to other sites so that cog policy is uniform

across all sites.

User Policy Manager (UPM)

User Policy Manager (UPM) is quite similar to CPM. It maintains policies

regarding specific users. This includes the general user policy and user replication policy.









They are quite analogous to their cog wide counterparts. They can be queried and

modified by applications. In case of conflicts between cog and user policies, the

application can choose to follow either.

One user policy is to specify which version of a file is seen by default. This can

either be the latest version of the file or the last version seen by the user. User Policy

Manager needs to keep track of the last version of every file seen by the user. This

component of UPM is called Personal History Manager (PHM). Whenever the user opens

a file, PHM enters the file name and the version. This can be retrieved by querying UPM.

Cog policies must be immediately communicated to other sites in the cog. By

contrast, it is unlikely that a user will be connected to more that one site in a conference.

That makes it unnecessary to update every site when a change is made to user policy.

When the user logs out of the system, the server contacts all other sites and updates their

user profile.

Cogs have a large number of users. It is prohibitively expensive for sites to load the

profile of every user. Instead sites load profiles into memory on demand. Once the user

logs out, the profile is dumped to disk and that memory is reclaimed.

Version Control Manager (VCM)

Version Control Manager (VCM) is responsible for the versioning aspects of DFS.

It gets the version name from VNM, along with the number of replicas from CPM and

UPM. It then hands the file over to Replication/Migration Manager (RMM) which

replicates the file on different sites. VCM then checks whether a delta of the parent

version needs to be created and replicates it as necessary.









Its other task is to rebuild files from deltas. For this purpose, it gets the list of

versions that are required to reconstruct the file. Then, it applies the deltas repeatedly and

rebuilds the file.

Creating a Delta

The new version is always stored as a complete file. It is the parent version that

must now be converted into a delta. In some cases however, the parent file remains as a

complete file. These cases are recapitulated below.

* The version 0 is a full file.

* Any version that is not on the main branch, i.e. if it is not the first child of its parent
is stored as a complete file.

* Any file that not stored in exactly the same servers as its parent is stored as a
complete file.

* If the version chain exceeds its maximum length with this file, the file is always
complete.

If the parent version does not fall under any of these categories and is a complete

file, it must be converted to a delta. First, the grandparent version number is obtained

from FIT. Version Control Manager reconstructs the grandparent file. This may not be

necessary if the grandparent is a full file. Now, a delta is created between the grandparent

and the parent files. Due to the lack of a delta compression package is java (current

language of implementation), DFS uses the utility BSDiff to create deltas. BSDiff was

chosen because it is available for a large number of platforms and can handle both text

and binary files. The delta file is replicated on the same sites and the FIT is modified to

show that the parent version is no longer a full file.









Recreating a Version

The FIT stores the version number of the parent of every version. When VCM asks

for the list of versions that are required to recreate this version, the FIT traverses up the

versioning tree to find the latest ancestor that was stored as a full file.

This full file is copied to a temporary file. The delta of its child version is applied

to recreate the child version. The next delta is applied to get the next version and so on.

The deltas are applied till the required version is recreated. The data in the temporary file

is then returned. BSPatch is a utility which can recreate file that were diffed using

BSDiff

Replication/Migration Manager (RMM)

File replication and migration are handled by Replication/Migration Manager

(RMM). When a request is made to migrate a file, RMM checks whether a copy of the

file already exists on the destination site. If there is a replica of the file on the destination

server, no action is taken. If the file is not present on the site, RMM contacts its

corresponding modules in the source and destination sites. In the first step, a copy of the

file is transferred from the source to the destination server. The FIT is then updated to

reflect this change. In the final step, the file is deleted from the source server.

Replication/Migration Manager uses Local Filestore Manager (LFM) to handle the actual

storage and retrieval.

When a file is created, it is replicated on the required number of sites.

Replication/Migration Manager identifies the servers which can host the file. It connects

to RMM on these servers and passes the data to them. FIT is then updated to include the

location of these servers.









Local Filestore Manager (LFM)

Local Filestore Manager (LFM) is the component of DFS that interacts directly

with the OS file system. When VCM stores a file, it passes the data and asks LFM to

store the data as a file on disk.

It handles the retrieval of files too (both delta and complete files). As far as LFM is

concerned, both are identical. It just reads the data and passes it on. It is up to the Version

Control Manager to rebuild the file.

Local Filestore Manager uses the underlying OS file system for file and disk

operations. This eliminates the need to implement file system manipulation functions,

which have already been correctly and efficiently implemented.

Distributed File System Server

Distributed File System Server glues all the components together, and coordinates

their activities. It also interacts with the client and with other servers when necessary.

When the server is started, it loads the DCS property files and if necessary, creates the

DCS directory. It then reads the list of servers that belong to the conference. At this point,

one of the servers is contacted and a fresh list of cog sites is obtained. Now, all servers in

the cog are asked to add the location of this new server.

The next step is to synchronize the server with other sites in the conference. It tries

to obtain the up-to-date FIT table from another server. In case it is not successful, it looks

for the FIT file on disk and loads the FIT. If there is no FIT file, the server starts with a

clean FIT. Cog policies are requested from other servers in the cog. Again, if it is

unsuccessful, the server tries to read it from disk. If there is no policy file, a blank policy

is loaded.









The final step is to register the necessary server objects on the local RMI registry.

All modules that require synchronization with other sites are registered. These include

DFS Server, FITM, RMM and both UPM and CPM.

The interfaces presented by the server to the client and to other servers are

reproduced below for convenience.

Client-Server Interface

* create: If the function is provided with data, it stores the data as a new file. In the
event that no data is provided, a blank file is created.

* readVersion: This command is used by clients to obtain the contents of a DCS file.
The version is determined by the versioning policy. The user can also specify a
version explicitly.

* save: It saves the given data as a new version of the file. The parent version must
be specified.

* deleteVersion: This marks a given version of a file as deleted. The file is not
physically deleted. It is marked in the FIT as unavailable.

* undeleteVersion: The function reverses the action of DeleteVersion. It marks the
version as available.

* deleteFile: It physically removes all versions of a file from all sites. It is not
possible to retrieve the file.

* copy: This command asks the server to copy a version of a file as another file. The
new file is not just a delta. It is the full contents of that version.

* move: This command renames a file or directory. All versions are renamed.

* makeDirectory: It creates a new directory in the file system. This operation merely
updates the FIT. No directory is created on disk.

* removeDirectory: This command removes an empty directory. Again, this
operation modifies only the FIT.

* listFile: This command lists the attributes of all versions of a file.

* listDirectory: It lists the contents of a directory. A directory can contain files or
other directories.









* migrate: This command migrates the given version to a server. Along with the
version, its entire version chain is moved. It is possible to specify the server from
which the file is to be moved.

* getCogPolicy: This command allows the user to view cog policy.

* setCogPolicy: This command allows the user to modify cog policy.

* getCogReplicationPolicy: This command allows the user to view cog replication
policy.

* setCogReplicationPolicy: This command allows the user to modify cog replication
policy.

* getPersonalReplicationPolicy: This command allows the user to view personal
replication policy.

* setPersonalReplicationPolicy: This command allows the user to modify personal
replication policy.

* logout: This command flushes user related information to disk.

* isDirectory: It returns true if a given path is a directory.

* isPresent: This command returns true if a given path exists.

Server-Server Interface

* getSiteList: This command returns the list of sites in the cog.

* readFile: This command reads a version of a file.

* writeFile: This command writes a version of a file.

* deleteFile: This command deletes a version of a file.

Distributed File System Client

The DFS Client exposes the DFS server interface to DFS aware applications. Any

application that seeks to exploit the features of DFS must include a reference to the DFS

client. Prior implementations mandated that each user have only one DCS Client, which

has been done away with, in this implementation. Applications can use multiple DCS

Clients at the same time. It is also possible for multiple applications to use the same client









and interleave their calls to the DFS server. Most methods on the client side, merely call

the same methods on the server.

Applications

Shell

In order to take advantage of the capabilities of DFS, it is necessary to implement

applications. The shell is an application designed to perform operations similar to UNIX

shells. It connects to the server using the DFS Client and manipulates DCS files and

directories. The shell can accept both absolute and relative paths (starting with ./ or ../). It

implements the following commands.

* Is: If the command is applied on a directory, all its contents are listed. These might
be files or other directories. If the command is applied on a file, information is
displayed on all versions of the file.

* cd: This command changes the current directory.

* cp: This command copies a given version of a file as another file.

* my: This command renames / moves a file or directory.

* delfile: This command deletes a file permanently.

* delver: This command marks a version of a file as deleted.

* undelver: This command restores a deleted version.

* mkdir: This command creates a new directory.

* rmdir: This command removes an empty directory. If a directory is not empty, an
error message is displayed.

* migrate: This command migrates a file version to a server. The location of the
source site can be optionally specified.

* logout: This command closes the connection to the server.

* pwd: This command displays the current working directory.









* chpol: This command changes a specific policy. The next argument must specify
the policy which is to be changed. Current options are:

o repl: User replication policy.

o gen: General user policy.

o coggen: Cog general policy.

o cogrepl: Cog replication policy.

* getpol: This command gets a specific policy. It requires one of the options listed
above.

Distributed Conferencing System Text Editor

The DCS Text Editor is an application which is specifically designed to

demonstrate the functionality of DFS. The text editor uses the DCS Client to connect to

the server. When the text editor is asked to open a file, it checks to see if the file exists. If

the file does not exist, it asks the user whether a new file should be created and then

creates the file if needed. In case of an existing file, the file is retrieved and its contents

are displayed in the editor. The editor allows the user to modify and then save the file as a

new version.

















-1- ^U sei I
SClient ------- Server policyN














local Replication

Server
VFilestore



Figure 4-1. DFS architecture. The modules in bold interact with their counterparts on
other servers.







































Figure 4-2. Version tree in DFS















CHAPTER 5
EVALUATION

Testing

DFS is one of the modules of DCS and depends on some of the services provided

by DCS. Unfortunately, the components of DCS have not been completely integrated. In

order to test DFS, specific test classes have been created to simulate the missing DCS

services.

DFS was tested on both Windows and Linux and was found to be satisfactory. The

test bed had three machines running DFS server process and multiple clients. One of the

aspects tested, was the ability of DFS components to maintain synchronization.

Tests were conducted to verify all components of DFS. Replication/Migration

Manager, Version Control Manager, User Policy Manager and Cog Policy Manager were

the focus of the tests. Additional tests were made to ensure that servers could be added on

the fly, to increase space.

Evaluation

The overall objectives of DFS were stated previously in Chapter 3. These are

reproduced below for convenience.

* The file system should provide a uniform name space.

* Files and directories should be location transparent and location independent.

* Concurrency control should allow users to modify the same files, without losing
changes made by any user.

* The consistency semantics should be predictable.









* File migration and replication must be supported to improve availability and
performance.

* New servers must be allowed to join the conference without having to shut it down.
Addition of servers should not change user experience negatively.

* The file system should provide an interface to allow users and applications to
interact efficiently.

DFS provides a uniform name space for all files in the conference. In fact, it is

possible to extend it to provide a global namespace. In this respect, it is similar to Coda.

The second objective is achieved by providing a File Information Table (FIT). The FIT

acts as a mapping table and translates between the logical name of the file and its actual

location. The immutable versioning system in DFS ensures that concurrent changes made

to a file do not overwrite each other. Changes made by users are saved as new versions.

This ensures that writes do not interfere with each other and files never lose changes.

DFS file consistency is predictable and configurable. Users can see the latest changes by

specifying their policy. The semantics are quite close to session semantics.

Replication and migration are supported by Replication/Migration Manager. This

increases system availability and fault-tolerance greatly and improves performance as

well. DFS implementation allows a new server to join the conference as long as it knows

the location of one server in the cog. The other servers in the cog bring the server up to

speed. A user does not even know that a new server has joined the cog. DFS provides

interfaces to both DFS Clients as well as other Servers.

It can be seen from the summary that the main objectives of DFS were achieved.

The next section puts forth some of the ideas that could be examined in the next version

of DFS.









Future Work

The current implementation of DFS is the first step towards implementing a

flexible distributed file system with high availability and great ease of use. It has not been

possible to design and implement all aspects of a distributed file system. However, DFS

provides a foundation for future work. Some of the ideas that can be examined are given

below.

* Integrate DFS with other DCS modules. DFS is designed to interact closely with
Conference Control Service. It also depends on Secure Communication Service and
Access Control Service for security. A good deal of work will be required to
modify calls in all these services.

* Allow DFS to raise events. The Notification Service (NTF) is responsible for
registering events and listeners. It also informs the listeners when the event takes
place. DFS should use NTF to raise events. One scenario could be to inform
specific users when a new version of a file is available.

* Current implementation of the File Information Table Manager (FITM) is not very
scalable. Implementing the new design would improve scalability.

* Replace the current command line delta creation and rebuilding utility with a Java
package. One of the biggest strengths of Java is platform independence. The use of
command line tools has severely restricted the operating systems that DFS can run
on. A delta compression package must be identified or developed for use in DFS.

* DFS client should be integrated with the operating system. This requires
modification of OS system calls. Tight integration with Windows is virtually ruled
out due to source code unavailability. Linux system calls can be modified to
support DFS.

* Support file merging and reconciliation. Coda is a distributed file system that
supports automatic file merging and reconciliation. Similar work can be done on
DFS to allow users to merge files.

Conclusion

Distributed Conferencing System provides support for users to collaborate

efficiently and securely. Distributed File System is an important part of DCS and

provides the file system for it. Distributed File System must provide concurrency control,









uniform naming, location transparency and location independence, high performance and

availability along with security. Other file systems provide some of these features but no

other distributed file system supports all of them. Most do not deal very well with

concurrency, believing that concurrent writes on the same file are extremely rare.

Distributed File System handles concurrency well and allows each user to preserve

his/her changes as immutable versions. Another feature of DFS is the ability to specify

cog wide properties and replication policies. Though DFS is designed specifically for

DCS, the ideas are applicable in any environment where concurrency and availability are

important.















LIST OF REFERENCES


[1] R. E. Newman, C. L. Ramire, and H. Pelimuhandiram, M. Montes, M. Webb, and
D. L Wilson, "A Brief Overview of the DCS Distributed Conferencing System," in
Proceedings of the Summer USENX Conference, Nashville, TN, June 1991, pp.
437-452.

[2] A. S. Tanenbaum and M. Steen, Distributed Systems: Principles and Paradigms,
Prentice Hall, Upper Saddle River, NJ, 2003.

[3] R. Chow and T. Johnson, Distributed Operating Systems & Algt i/thi,, Addison-
Wesley, Reading, MA, 1997.

[4] V. Manian, "Access Control Model for Distributed Conferencing System," M.S.
thesis, University of Florida, Gainesville, FL, 2002.

[5] L. Li, "File Services for a Distributed Conferencing System (DCS)," M.S. thesis,
University of Florida, Gainesville, FL, 1995.

[6] P. S. Yeager, "A Distributed File System for Distributed Conferencing System,"
M.S. thesis, University of Florida, Gainesville, FL, 2003.

[7] R. Sandberg, D. Coldberg, S. Kleiman, D. Walsh, and B. Lyon, "Design and
Implementation of the Sun Network Filesystem," in Proceedings of the Summer
USENX Conference, Portland, OR, 1985, pp. 119-130.

[8] R. Srinivasan, "RPC: Remote Procedure Call Protocol Specification Version 2,"
Sun Microsystems, Santa Clara, CA, Request for Comments 1831, August 1995.

[9] B. Nowicki, "NFS: Network File System Protocol Specification," Sun
Microsystems, Santa Clara, CA, Request for Comments 1094, March 1989.

[10] B. Callaghan, B. Pawlowski and P. Staubach, "NFS: Network File System Protocol
Specification," Sun Microsystems, Santa Clara, CA, Request for Comments 1813,
June 1995.

[11] B. Callaghan, D. Robinson, R. Thurlow, C. Beame, M. Eisler, S. Shepler and
D. Noveck, "NFS version 4 Protocol," Sun Microsystems, Santa Clara, CA,
Request for Comments 3010, December 2000.









[12] B. Callaghan, D. Robinson, R. Thurlow, C. Beame, M. Eisler, S. Shepler and
D. Noveck, "Network File System (NFS) version 4 Protocol," Sun Microsystems,
Santa Clara, CA, Request for Comments 3530, April 2003.

[13] M. Satyanarayanan, "A Survey of Distributed File Systems," Annual Review of
Computer Science, vol. 4, pp. 73-104, August 1990.

[14] M. Satyanarayanan, "Scalable, Secure, and Highly Available Distributed File
Access," IEEE Computer, vol. 23, pp. 9-21, May 1990.

[15] M. Satyanarayanan, and M. Spasojevic, "AFS and the Web: Competitors or
Collaborators," in Proceedings of the Seventh ACM SIGOPS European Workshop,
Connemara, Ireland, 1996, pp. 89-94.

[16] M. Spasojevic and M. Satyanarayanan, "An Empirical Study of a Wide Area
Distributed File System," ACM Transactions on Computer Science, vol. 14, pp.
200-222, May 1996.

[17] R. Needham and M. Schroeder, "Using Encryption for Authentication in Large
Networks of Computers," Communications of the ACM, vol. 21, pp. 993-999, June
1978.

[18] M. Satyanarayanan, J. Kistler, P. Kumar, M. Okasaki, E. Siegel, and D. Steere,
"Coda: A Highly Available File System for a Distributed Workstation
Environment," IEEE Transactions on Computers, vol. 39, pp. 447-459, April 1990.

[19] R. Pike, D. L. Presotto, S. Dorward, B. Flandrena, K. Thompson, H. Trickey and
P. Winterbottom, "Plan 9 from Bell Labs," Computing Systems, vol. 8, pp. 221-
254, February 1995.

[20] R. Y. Wang and T. E. Anderson, "xFS: A Wide Area Mass Storage File System,"
in Proceedings of Fourth Workshop on Workstation Operating Systems, Napa, CA,
1993, pp. 71-78.

[21] T. Anderson, M. Dahlin, J. Neefe, D. Patterson, D. Roselli and R. Wang,
"Serverless Network File Systems," in Proceedings ofACMSOSP, Ashville, NC,
1995, pp. 109-127.

[22] A. Muthitacharoen, B. Chen and D. Mazieres, "A Low-Bandwidth Network File
System," in Proceedings ofACM SIGOPS Operating Systems Review, Newport,
RI, 2001, pp. 174-187.

[23] M. O. Rabin, "Fingerprinting by Random Polynomials," Harvard University,
Cambridge, MA, Technical Report TR-15-81, 1981.

[24] U.S. Department of Commerce/National Institute of Standards and Technology,
"Secure Hash Standard," National Technical Information Service, Springfield, VA,
FIPS PUB 180-1, April 1995.









[25] D. Mazieres, M. Kaminsky, M.F. Kaashoek and E. Witchel, "Separating Key
Management from File System Security," in Proceedings ofACMSIGOPS
Operating Systems Review, Charleston, SC, 1999, pp. 124-139

[26] I. Clarke, O. Sandberg, B. Wiley, and T. W. Hong, "Freenet: A Distributed
Anonymous Information Storage and Retrieval System," in Proceedings of the ICSI
Workshop on Design Issues in Anonymity and Unobservability, Berkeley, CA,
2000, pp. 46-66.

[27] A. Bhalani, "Implementation of Conference Control Service in Distributed
Conferencing System," M.S. thesis, University of Florida, Gainesville, FL, 2002.

[28] R. G. Guy, T. W. Page, J. S. Heidemann, and G. J. Popek, "Name Transparency in
Very Large Scale Distributed File Systems," in Proceedings of SecondlEEE
Workshop on Experimental Distributed Systems, Huntsville, AL, 1990, pp. 20-25.

[29] A. V. Date, "Implementation of Distributed Database and Reliable Multicast for
Distributed Conferencing System Version 2," M.S. thesis, University of Florida,
Gainesville, FL, 2001.















BIOGRAPHICAL SKETCH

Prashant Jayaraman was born in Thanjavur, India in 1983, the son of Raji and S.

Jayaraman and the younger brother of Vivekanand Jayaraman. He completed his

Bachelor of Technology degree in computer science and engineering from Regional

Engineering College, Tiruchirappalli, India in June 2004. He then decided to join the

University of Florida in 2004 to pursue his master's degree. After graduation, he plans to

pursue a career in industry but hopes to eventually earn a Ph.D.