<%BANNER%>

Performance Analysis of Dynamic Sparing and Error Correction Techniques for Fault Tolerance in Nanoscale Memory Structures

xml version 1.0 encoding UTF-8
REPORT xmlns http:www.fcla.edudlsmddaitss xmlns:xsi http:www.w3.org2001XMLSchema-instance xsi:schemaLocation http:www.fcla.edudlsmddaitssdaitssReport.xsd
INGEST IEID E20110114_AAAACA INGEST_TIME 2011-01-14T11:48:18Z PACKAGE UFE0007163_00001
AGREEMENT_INFO ACCOUNT UF PROJECT UFDC
FILES
FILE SIZE 16650 DFID F20110114_AABFKL ORIGIN DEPOSITOR PATH jeffery_c_Page_074.QC.jpg GLOBAL false PRESERVATION BIT MESSAGE_DIGEST ALGORITHM MD5
cf515b9a7d12dac2566c3f6beb2f8c62
SHA-1
6b166f1fa0b7ed85d06c91e8b77589db358dc8c3
4964 F20110114_AABFJW jeffery_c_Page_066thm.jpg
2d0ab2da7c38d34a907df7f3506544a3
5150cfc3dba0acc8d278381c29c8040996cb9eea
14101 F20110114_AABFLA jeffery_c_Page_082.QC.jpg
6a4fca020c2f6068d8fb1bdad6f561aa
0046590a960129c6928af5eb49662379dabe657a
4678 F20110114_AABFKM jeffery_c_Page_074thm.jpg
e053a0498e13c23eedc077c670c0d19d
ef8f23e786e89f3152453d1271a833e8d711e1dc
9600 F20110114_AABFJX jeffery_c_Page_067.QC.jpg
b6c4abfd63214159419b5d0358e3f019
9b7075a18340ac9458bcf35be4dd687bc14158cb
4057 F20110114_AABFLB jeffery_c_Page_082thm.jpg
1d11e34944264b538f5b02416fadb28b
7565d94fb7c7f441d230fa01835fbc57025a0322
19063 F20110114_AABFKN jeffery_c_Page_075.QC.jpg
f5d44ef4f33d2cca89f266952898727e
60f967fa7df0873a0c9732337e644ff867b67cf1
3174 F20110114_AABFJY jeffery_c_Page_067thm.jpg
ea927a3e36ce4b86c35480942f9d48db
b89498c10255552c1efbfc7136eaa1bbd05e30ce
15728 F20110114_AABFLC jeffery_c_Page_083.QC.jpg
dbf8896a05c6a45465b3fa612204ce2c
9066538521fae6fb32e0dba258b25cd28df38379
5177 F20110114_AABFKO jeffery_c_Page_075thm.jpg
4e3cbcd30cf04bd85b0105d059157cea
5868a5e0f7df521e91d2362d1862b92202cefcd1
14607 F20110114_AABFJZ jeffery_c_Page_068.QC.jpg
4d531ea4c5394a523c83addb4e8bd0c6
7ed44051bd085b2515af0830a1c48853deaf9912
4374 F20110114_AABFLD jeffery_c_Page_083thm.jpg
42f0749bd5b3fd596be811acaa390978
f5c8e1a5e9d7415c6ba4c2e403167aa9500e0928
16915 F20110114_AABFKP jeffery_c_Page_076.QC.jpg
29345a259219e26204ce78860e9c276f
e8ce5c0f2bef2303f3d5cf143b13bf317d6e00e0
14332 F20110114_AABFLE jeffery_c_Page_084.QC.jpg
8e030fcb323a4a61ce4d51cfd8271973
8eca8ac6012aa8931e4b5448c1ed01fa518d4994
4670 F20110114_AABFKQ jeffery_c_Page_076thm.jpg
3b8fc68c5849d0096b3bf72d593ddccf
819ce679bb1eb9999e8a1a0a73a5bfebaa87139f
3928 F20110114_AABFLF jeffery_c_Page_084thm.jpg
89db633958396fb0d194b0b683a6fded
772c25df54bef7d359a574ed90a1e405ba6de5fa
12922 F20110114_AABFKR jeffery_c_Page_077.QC.jpg
71fed174c3da63f5d9ebe8b9b220f6f7
55af7994223f6f56e3b130bb60a8c80264201149
13107 F20110114_AABFLG jeffery_c_Page_085.QC.jpg
26b1297416ec1e9d63a4e94135369b72
64ddc5e868a91d35d2e93459c88f276a76d70c10
3655 F20110114_AABFKS jeffery_c_Page_077thm.jpg
5c206b9a8ca5e8ebb809779be5ed3123
b02371c3720373cfdc1171d37cdf5c259b8a422f
3847 F20110114_AABFLH jeffery_c_Page_085thm.jpg
c36e6b73d76b2021c1fc4c979279a2d9
863ec0714cf644311093b8322bc3c6fdc2639a78
4658 F20110114_AABFKT jeffery_c_Page_078thm.jpg
2b36bca05f8bcfb5e462d2f45683394d
a0bb5fc1f8c96102536db19330a44dc72005c911
15732 F20110114_AABFLI jeffery_c_Page_086.QC.jpg
97fc88fd0cdc5a520886287a145a6b76
b127d3f9d803955e6c7ec4c58d9bc9f7756bc122
14051 F20110114_AABFKU jeffery_c_Page_079.QC.jpg
13792d8807094d8a3e239545e4c9ee75
bf9d3eff172e6ff59303884b956569ea3ede2ad5
4604 F20110114_AABFLJ jeffery_c_Page_086thm.jpg
c2cbcb3992ff1c7d3bd1d142f4ab538e
9fa2cad6ed1397e62bd730c629d471db98702675
4028 F20110114_AABFKV jeffery_c_Page_079thm.jpg
aca7cbf1bfae8e06cfe8d7f1e1fac938
e40649c43b5862d61fcc7f1a0584d4f02f465439
17283 F20110114_AABFLK jeffery_c_Page_087.QC.jpg
8d44ad1fccc79c88f6fc01174a10a989
be98755e3933b23d77db5719cd7922b6e23f4b8e
16971 F20110114_AABFKW jeffery_c_Page_080.QC.jpg
b96334bfe34f51ec243de8616b08c256
c12d0e33b07a4ea84a6a83895a72563c46f5487b
4514 F20110114_AABFLL jeffery_c_Page_087thm.jpg
4374be4b274cb835be274d8c8dbeae85
1123879636163f7089b31d70fdbc6a323d09f901
4369 F20110114_AABFKX jeffery_c_Page_080thm.jpg
84534a04760a8cee9cfbece4e3a830bd
bc909829544e5f20daa27ad855ec7ba35b1b17ca
3774 F20110114_AABFMA jeffery_c_Page_095thm.jpg
f24789b6768784ed73fb423c5e32d9fa
08264d8cd9c632581ac035a8fba267926bf5b1be
18807 F20110114_AABFLM jeffery_c_Page_088.QC.jpg
c821863c1b6635db741dc4a158cfccc2
0117e4a1200a9eaf7edfb46e11fb42c6fc22c852
16402 F20110114_AABFKY jeffery_c_Page_081.QC.jpg
ef0ba3fab028fd63ccd8f27d35654c65
5fd9c8bbf1157b557434754b3be96f43d186512f
12025 F20110114_AABFMB jeffery_c_Page_096.QC.jpg
f2124e7f663e95d464c35c0a6b2620a6
cfea2175ccf62d303b97f11aba22fb97c64222b0
4800 F20110114_AABFLN jeffery_c_Page_088thm.jpg
29bedce2ac4decd1dee306ba0198390d
e1a16d48b8852770c06c1139a8692d8cb036c7f8
4434 F20110114_AABFKZ jeffery_c_Page_081thm.jpg
ea3d41df6dc33fd4d392049c3e216ea6
e50963974366aeef8ad2ad44ed67f0ba181df9ca
3631 F20110114_AABFMC jeffery_c_Page_096thm.jpg
0e54f75c4f5ef00f75735b0b2c391fcb
001ff03c58808d4be75d61a4834d882561ff448b
16332 F20110114_AABFLO jeffery_c_Page_089.QC.jpg
24a925d0560703f20f7353a245e3649a
fd5cda299f6b90647cfd1f9e3b9df0a66e905ae4
10138 F20110114_AABFMD jeffery_c_Page_097.QC.jpg
ec5cdb7a3435fdc85efaba5951f5085f
4023dbc2f3093da3f59abca74f121f39be0537c7
4567 F20110114_AABFLP jeffery_c_Page_089thm.jpg
224a2e53d8645d1a47d91e9191e033e2
22a2fdad32041cc11db8be0c4b1b790a7dea1691
3169 F20110114_AABFME jeffery_c_Page_097thm.jpg
d9f5e3fcc01a7f78c1291238495406d4
d271fc291ac03adce5d0fd25fb6424ad0b4376af
8176 F20110114_AABFMF jeffery_c_Page_098.QC.jpg
49cc32d1665a071896afc18d882ac682
495245cf2d6234365fb79e246d35b4459145242c
10091 F20110114_AABFLQ jeffery_c_Page_090.QC.jpg
86d7a1195ed1e9ec88e4e203a24c5c81
f8d0d926725c94d32e3fdfbbcafebc1fc9c6ceb1
2744 F20110114_AABFMG jeffery_c_Page_098thm.jpg
a63e2ebb7507430c6991b578646138b1
29fa89d7991ecb96f19a7fb86f34d42ec8605202
3011 F20110114_AABFLR jeffery_c_Page_090thm.jpg
e44a75022e69cb49e7271cd583b3ad57
032544ae1a1e9a002cf43dd324ed0e0b3d9ec403
22248 F20110114_AABFMH jeffery_c_Page_099.QC.jpg
95eca3c8272a63100ed2963f33f04226
cfc8f4c33d93749bed89689144dbe9b6f16e470b
17638 F20110114_AABFLS jeffery_c_Page_091.QC.jpg
856bf3617b245f79365f628e597532eb
83d396557aac95aeb8269a28ba3cf6b5a57ab508
26773 F20110114_AABFMI jeffery_c_Page_100.QC.jpg
9680e7ef67ebdd3135f4d6ec4f6eadfb
06369831a0d4e063e55181fc30dd4ad46ddc5d6e
4732 F20110114_AABFLT jeffery_c_Page_091thm.jpg
9335aa83681af9b5d2ef76434f66387f
2c3f0cf25027015f99d8f57f8cadd7a05f033f06
7001 F20110114_AABFMJ jeffery_c_Page_100thm.jpg
775c2c29d2ace521043a2c59d6c5be32
ed49b5bf7053792ac081bf6a8517c9be811a1d19
14701 F20110114_AABFLU jeffery_c_Page_092.QC.jpg
873b4f5c677ae9b78569343439332660
a213afec68c8067e50f52103ed88d2c1547d0c3c
26808 F20110114_AABFMK jeffery_c_Page_101.QC.jpg
653b694a779ad040f11b5508fee2c6db
4d399b7595c88891cc24a3da006a43fca1916ba4
4099 F20110114_AABFLV jeffery_c_Page_092thm.jpg
016325fb2b23554b839ab1fb25e110b4
8b4ca1e1acd46edb39cbc1652acbf63c5b98b5d0
25100 F20110114_AABFML jeffery_c_Page_102.QC.jpg
084af480dd05161cc0f6601a159aee39
c85778dc905f09e9831140b8ae04756f1c6a9d2c
3374 F20110114_AABFLW jeffery_c_Page_093thm.jpg
f4c8b208a30fef17f1a0a13508db03a2
a71edecc902dbfd9c9129ae6d42f77b685b568b3
6582 F20110114_AABFMM jeffery_c_Page_102thm.jpg
9030b3f5494a29d032c00b61de44229a
9f411161f7d4be5f017a4458a76989879f1b3098
14188 F20110114_AABFLX jeffery_c_Page_094.QC.jpg
4a3001b8b2a16c8dd0e58d81090495a7
286358ea019cbfa51eeca47dc77e721fb3ebdbef
12577 F20110114_AABFMN jeffery_c_Page_103.QC.jpg
09aafb232ec2267068391002a3f22250
55f2be73b53a50d1eff348a876285f88d3a8f071
4037 F20110114_AABFLY jeffery_c_Page_094thm.jpg
f1e54e038275b88366cc6f9b703ecc12
ff7fe34078efda0ebce328d45fe33a7b422a6fe2
3529 F20110114_AABFMO jeffery_c_Page_103thm.jpg
efa869ef1e06c3ccbf05fa0fc599dac0
0519f507f55eba7a9bb592db200719e0b648da86
12294 F20110114_AABFLZ jeffery_c_Page_095.QC.jpg
d56e19a3a968ba8b908dd95791188da0
61fb8d0f2757365476b17572b8a0a9b95bda2738
13112 F20110114_AABFMP jeffery_c_Page_104.QC.jpg
4f60081a3cfd301f1f73b7d48de95597
b4b4fc3225f4dc3cb49803c553cf3410adff174b
3912 F20110114_AABFMQ jeffery_c_Page_104thm.jpg
a8166f414be0d59d8347d092e0e7dedf
b86894b4cfe59760cd2a2f2a11fa4f09eb803842
121602 F20110114_AABFMR UFE0007163_00001.mets FULL
f5b48a0567b012d0eb1e3a82f2b485f3
a589a00a8284c7c724b1776e474021c6fb4f95ff
6611 F20110114_AABELA jeffery_c_Page_023thm.jpg
4539e72477a8ea9260bce618f7eb9291
57625c732b1572a691e45c602f109d80bf10b048
44168 F20110114_AABEKN jeffery_c_Page_025.pro
46756ba357ada67db6021efbd748ae52
e0c9e13fae0f3a0a853b697957c278fbe7e7f3ae
6313 F20110114_AABELB jeffery_c_Page_013thm.jpg
70c70407e7c6463761c7f0f820456941
1ee5819be618bb6db2292b147e8c8cfecd0ffe5f
1053954 F20110114_AABEKO jeffery_c_Page_004.tif
c16dfb2c26de10ee20a7d63b6e394bc9
931aeb2acc4dfb5ed5d4f322c5176170fe735b4e
10923 F20110114_AABELC jeffery_c_Page_093.QC.jpg
5914fdd86065db389f50b489d2705de8
70dc83cb9e6a3124c228433b05c8664f08f993ca
F20110114_AABEKP jeffery_c_Page_077.tif
cff9e03dcf101ecb482a4c91535759c9
17de07b08c8e1e914838140e9f9edebbf4d55f08
36484 F20110114_AABELD jeffery_c_Page_043.pro
c8e9154b1909541510832b1b19c55a08
3bc9e617a907b02d9d4adbad327571f085d0bf36
48938 F20110114_AABELE jeffery_c_Page_065.pro
df3503f2349148d8d6f85776ddb98e15
5030432a03444b6d495d35dd1348361fb1b18f68
6224 F20110114_AABEKQ jeffery_c_Page_035thm.jpg
f30d2c78819356c099224d3b7a3c3a8b
36ac01799cdb7e3424d61deee62fd31b7cfc6fda
F20110114_AABELF jeffery_c_Page_069.tif
e982f5df3a1bab7ef1028504bd4087ed
734bfe18a9ce4b7876cfcad983b7dbebfeebcdf5
F20110114_AABELG jeffery_c_Page_094.tif
04d98fa061c450389099fbd7b74bd750
52a97ba540ebf90ba1f544daf0b645c3a6d4caba
51263 F20110114_AABEKR jeffery_c_Page_033.pro
f2d01e0aaba06f639dde54ae0ee5f231
0b9aeed2529fe5a105a838993607b7b1683a7c0c
6568 F20110114_AABELH jeffery_c_Page_065thm.jpg
f03327cd13453f69962957e71c9b6f01
6b87812225388d0511cb8f47fed7beada0e4c94f
59562 F20110114_AABEKS jeffery_c_Page_095.jp2
ea8bad719bd2298e7de9911e466dc6dd
29472f1e3b12c60dd05186af0852b6d4a0ed4c6e
69229 F20110114_AABELI jeffery_c_Page_014.jp2
f53626cf89e02cee7553a4acfec43343
06482deb067dea52d85e870195b2549f944a0947
59576 F20110114_AABEKT jeffery_c_Page_016.jpg
4326be2e5d1643c268c1bf665d8f4cb5
dd5bf40a6c6fa0a38272f66c9b78530cdf49ca1f
5973 F20110114_AABELJ jeffery_c_Page_099thm.jpg
7fac9170fec73515f8866d91e06495d5
ed3db917e9ed2fea5d2f1e1dbc37a4c09269c4f0
96463 F20110114_AABEKU jeffery_c_Page_072.jp2
c679c9bb822dd65e412ef5ebeedbacf1
12c58ad492c0b4f17285fca7255ff44cfefef871
2245 F20110114_AABELK jeffery_c_Page_099.txt
cbe49fcd3b67ae2af44a011ac9bc5e71
ad0965f19429caf721f6ba3b14d5376745e79735
1836 F20110114_AABEKV jeffery_c_Page_040.txt
381ff27ba4cd95f360732a82d5ec876f
9e03ab291383ce37478ed89a2e8fc6e1fb4439d1
71538 F20110114_AABELL jeffery_c_Page_032.jpg
ab739db8d95fac35218fa77bf140de57
95da08a6f8d6b13f1f39a9b21ba75e2a4355c86f
49513 F20110114_AABEKW jeffery_c_Page_092.jpg
38201554b81fe9e3904d5fa18f2f36a4
a567af8bee315d91e1a922a6d35f170b5b93e209
26497 F20110114_AABEMA jeffery_c_Page_008.jpg
227ef585b90b7c6e1a03b9873792ad23
6ec0593dc47825a97661d6051e4b613d72cc15bc
1412 F20110114_AABELM jeffery_c_Page_096.txt
57666efa35680cc84569dc335b227668
b6da70ddc8f1f29c3b04d268c6481aceda0bd748
F20110114_AABEKX jeffery_c_Page_079.tif
28273662094e9d2266c024e989372a71
7081d55b40c5c0c6c5ce6648c4f4a1d688eb8f5d
49662 F20110114_AABEMB jeffery_c_Page_023.pro
e7bdf89e3a05e128b02664192559ca76
a5e4ef9fe2ff9342f0771d9c7129cf89c96e0110
40718 F20110114_AABELN jeffery_c_Page_055.pro
43e1a20b71d3680ca422d213c274dc34
bc5bfa32ddad5aef4abc65e2a67a3148b473951f
1984 F20110114_AABEKY jeffery_c_Page_032.txt
ffbeb2f5318624f8dba30f46849f4945
6ad15ed8e0b82bb89392fb5dc9a98037aac986db
45974 F20110114_AABEMC jeffery_c_Page_052.pro
c88a10ce501d52ee810829ead53f42e6
17a1a3a92ac8ae971263698f303f3feaa5406ce3
906 F20110114_AABELO jeffery_c_Page_042.txt
c70b15c61aac50ec6d43f09d2ff53b96
be2ae257b704123fd371b04c86c3d1594d34d1c9
20604 F20110114_AABEKZ jeffery_c_Page_029.jp2
05447dabf0c5d3f66a85da162fe5b7b8
38b78736e8352edd372b475e3f4ca9b0aa23f661
60955 F20110114_AABEMD jeffery_c_Page_085.jp2
e8c873755d32bd282ca664dab4904982
a5137f0ef4ccae826e4070a0424e79dd8a1baf84
1868 F20110114_AABELP jeffery_c_Page_083.txt
c760f7edbbfc1e81837a46e5abc01854
2a234fd85ac0ec4a7377199c3fc717532def2167
20731 F20110114_AABEME jeffery_c_Page_040.QC.jpg
85b96a9f9fda63efa2a928ef62a81cc6
f8f4866bd96e63289c7a872fe5e0ea5161677df2
20353 F20110114_AABELQ jeffery_c_Page_061.QC.jpg
f1ebd9c881e15cf4d4869c5a034f4a51
464b669f82bbad450672bc13d932d6c3e986bdae
40052 F20110114_AABEMF jeffery_c_Page_016.pro
b4bbc510900f834f8b5b257e47725011
7870e692743072dc280c76a5269f81f90e42748b
5710 F20110114_AABEMG jeffery_c_Page_061thm.jpg
0196870d0a50927a58dc2f92677baf86
7af174ae167f1e5684c89f7bfdc8b1958a3adc4c
21536 F20110114_AABELR jeffery_c_Page_042.QC.jpg
307b089d6422d7d7b1d96496f9243969
b1e35690a81a0736dc1915d188c75d2519b20cfb
50393 F20110114_AABEMH jeffery_c_Page_048.pro
9ff3554257f57ec045082e8240781ac3
9d4c6e41209a039dd111e269ca8d11034062f294
57979 F20110114_AABELS jeffery_c_Page_060.jpg
6d7fd2ba667d35ec5a403a10fb270762
0f4752e1431ca1c129db6da4f5bc45c54cac88c8
23230 F20110114_AABEMI jeffery_c_Page_054.QC.jpg
80661a23c0a2e728f6ea9985c4dc260d
63c4641a5993c15043ae69cdb86751f6c81dbf43
27282 F20110114_AABELT jeffery_c_Page_001.jp2
e4c54b4c45c5d2514e482709b92bb9a2
c36506df099ed62f3419a6a85f5a21133c4a192e
1912 F20110114_AABEMJ jeffery_c_Page_087.txt
e550f13bfa1e49362b534ffaef963908
bbf560fbf0069f71448b8f64a48d95702704c169
47163 F20110114_AABELU jeffery_c_Page_027.pro
508c0e4354bcca3444cddb739aeb0742
f84ebe5b58d641b9cff0ec73920a3ce3a03488b9
17181 F20110114_AABEMK jeffery_c_Page_010.QC.jpg
aa6b30a60d284bddf04aba01da6348f9
6296b18b0b15b037167e6424177766d8851fd49c
63111 F20110114_AABELV jeffery_c_Page_071.pro
32076a0e0e106fe09769de04c196e5b1
fa0bfed0913dc59538766a972091ca154a411609
45112 F20110114_AABEML jeffery_c_Page_085.jpg
e11fe94198002ec8f7166eb21c6d02c0
bb10dc4bf1c0bc50865580c65c70f3db189fc81d
F20110114_AABELW jeffery_c_Page_026.tif
8aaaf8b95e0d91913769403333d2f10c
612ac9b449e574ff7ad02c5a28a22cbbefb5a8d0
4617 F20110114_AABEMM jeffery_c_Page_049thm.jpg
7a817c751fd2741ea8e2d5c8decd573a
af0bbcc6a966ee8d6a703d20da0acd075b69380f
25074 F20110114_AABELX jeffery_c_Page_039.jpg
e5b5bb37aadced24c2d108a6ccee428d
7ed93211c1a9633e0bf1319c6af2fef3eb171ede
3674 F20110114_AABENA jeffery_c_Page_011thm.jpg
da3e580a5c88ffd79bb38a446296dc13
614869a40c99c39ae776223e6d20500b1e88a68a
1528 F20110114_AABEMN jeffery_c_Page_037.txt
431e4fc0936c312017fb6dc6810a7985
58b46be87391e71c669194456363497ac476f10b
51857 F20110114_AABELY jeffery_c_Page_009.jpg
9ac4a9ae3fce2a22b7e7d7219cd8b29a
13eaad212c74f419cf286d1e7105daca32072b17
979525 F20110114_AABENB jeffery_c_Page_062.jp2
762d6d8e6e9dd758f11c9ec51ec08625
7b20978cfb93cb513638c27e2943b55488d1d1e4
67735 F20110114_AABEMO jeffery_c_Page_028.jpg
950f07c114174e0785f40cbf1f8e9ea0
5881295804f2e9000705d2058ddeab4ba2da28dd
13856 F20110114_AABELZ jeffery_c_Page_046.QC.jpg
f90bc30bf0f3ccea35a90652c4058bc0
6b06a233ba423d58ee6f7483e12b74ac9ba4898e
598965 F20110114_AABENC jeffery_c_Page_049.jp2
faf9e7e9809390b337198d887df61a7a
cacf8c91a2979ded65608cf988600fcb34dd5e93
65015 F20110114_AABEMP jeffery_c_Page_051.jpg
204f25d1fccf5d580e1ddb0459ba1119
fba6e25199d779364b392890e97cd6cf16e0a849
17279 F20110114_AABEND jeffery_c_Page_078.QC.jpg
22b252d7566a1a188fa9ecce1b6019dd
7664486fa4edbba97d67e8f005b8cd171a35ab8c
98473 F20110114_AABEMQ jeffery_c_Page_059.jp2
cf379b7c1bc7cd3b38967f197ade6979
8bbab784769881dd6d64713755395c67e68f71d6
49853 F20110114_AABENE jeffery_c_Page_078.pro
1c13d23c6285db78b8a7cb7976a2fc50
3dfb5b89d4eb721483d3aa91b7196bbc5911a6b4
78206 F20110114_AABEMR jeffery_c_Page_034.jpg
f0d4670456aa7793b129d0fb918991b1
569f87e5e1ae6d0d978ce8aafb68605453ddbf51
1945 F20110114_AABENF jeffery_c_Page_017.txt
20017db21f9b80881d8b3d299b38639d
2a429c6ad9486cfe9887d3e57a6ce76c2d672cad
1989 F20110114_AABENG jeffery_c_Page_048.txt
f6cbe44fb05db9005dc2f40ef36e631d
544c8384761e284b59f6c8311f021c7e8aa759de
84554 F20110114_AABEMS jeffery_c_Page_074.jp2
ced3ab0ebb335bae2aaba92c72e50854
452ed5e59ed03bef74934cd0949bbab71890a61a
60263 F20110114_AABENH jeffery_c_Page_102.pro
bb51bddb0db70c152046d318c1d1292e
05d4b470ad464ad5c5141c528227a925f920dd9c
25271604 F20110114_AABEMT jeffery_c_Page_062.tif
78e6df527c8f1edf1162a55bd7771645
a2c1176e54e95717c7c5c18e2987828b107d70a9
21587 F20110114_AABENI jeffery_c_Page_090.pro
56ebaf25c43de1440e0cdaf43dcf67d5
99fa31d69319d06bd9c111ef51fa2ccb48cfefe4
2833 F20110114_AABEMU jeffery_c_Page_071.txt
f77c261fd19602442abb46985bb93077
8060f38da68bf6670ab4bed2de07276648a0c900
107685 F20110114_AABENJ jeffery_c_Page_088.jp2
1341d03828471eef04da9df48ad99fef
47f624166016dc335331f510b40a9a9433cc9d6a
48331 F20110114_AABEMV jeffery_c_Page_019.pro
047a626e682b372ae09b19a9fa9f289e
757ff3af139ea1e73fbed14b9236d5dbe09ddd6e
1560 F20110114_AABENK jeffery_c_Page_060.txt
8187f6952234c932f946e741085a9170
dd358dcdb81fccbe19cc19afd8b9e087428bd9fd
F20110114_AABEMW jeffery_c_Page_084.tif
3c6ecfc36c78e04a9bf97de9fa95c2a9
ce0891c071cf014682a0219202da3dd3d44f5deb
76063 F20110114_AABENL jeffery_c_Page_083.jp2
4d96c2d2621ef5e1513f35fd944a6320
49ef7ec7a8273b9a0c2b1e5cc54a1d7344dc9310
57108 F20110114_AABEMX jeffery_c_Page_046.jp2
e9b72051dea4dda843a3241517506d23
fd81f63a75322668d59ab8f0dcd93f55fa025b07
10869 F20110114_AABEOA jeffery_c_Page_003.jpg
3165f661975e119960e664f4c1966cab
9745d68aaf7049a7a1655b955c65f522a3e26b42
7220 F20110114_AABENM jeffery_c_Page_101thm.jpg
e9b9fd70873980eabb5b6c77fb32ef36
7093767b13e0061b26b46ba3aeb43e616a0740e0
3181 F20110114_AABEMY jeffery_c_Page_063thm.jpg
094d1d4f46f38bd34ff867edf0c5f0eb
7ffa58e7ecb9741e2c3922cac2b0883c17e371d7
24412 F20110114_AABEOB jeffery_c_Page_004.jpg
11330307262d9b0a7a387c9e7b9fa65c
be50a2e735dcef4dcdb626ad2b4a563398565a34
43508 F20110114_AABENN jeffery_c_Page_076.pro
9a77b1c3fbd91207c541b2d0d91d5575
9fe421e4d8292174b0893a06a89f324f7809e3ae
48407 F20110114_AABEMZ jeffery_c_Page_013.pro
ffb589b22404844f67ae48cae637ef69
cffff9b0a10966d05ff865d403d4b29be5437fcb
65026 F20110114_AABEOC jeffery_c_Page_005.jpg
cc6f01bfcf58eee4e0ea92e9fea467af
abd37274082fa7cd2972ea49776c5098c3a4a536
4598 F20110114_AABENO jeffery_c_Page_014thm.jpg
b6d7cba914e02744ae700ec572389e19
1ba10462c6ede02f1a32b72a9e40ebe80d17f706
88845 F20110114_AABEOD jeffery_c_Page_006.jpg
7c2bac50568538ea5b900c9d5f7c9238
fe4b9f5d3142b1aca5970ea279428d5c2da15c9a
2934 F20110114_AABENP jeffery_c_Page_072.txt
77ef7c795551c2f09f8a8a36cc02e125
d16e3f2da14936edb8712a64eeaad56ffc19f3ba
13095 F20110114_AABEOE jeffery_c_Page_007.jpg
099a129909fbef6b393da346c8233ce1
fb12870551faf5f76f7a3b52d1e89e10bb0e5e4a
72748 F20110114_AABENQ jeffery_c_Page_092.jp2
7a4e287da2c384c1e04c60b0ec4ce0f0
4fc98709e662dff74586bbe2f02cc74df337df9d
55854 F20110114_AABEOF jeffery_c_Page_010.jpg
5f0a9ed46184a4ca9d357a381500c107
8f3b20db809456c9430a444557f3202086fbed71
6290 F20110114_AABENR jeffery_c_Page_038thm.jpg
03a9fea30edc12ced385410292959d0a
231c9d838f57bcbdbc87799a0fa896ac29fd741b
39246 F20110114_AABEOG jeffery_c_Page_011.jpg
a494fc78a40d59c5a34c8283f91ca758
9c953f0114d6196fb09c11421d875f9cdb0f4699
22991 F20110114_AABENS jeffery_c_Page_050.QC.jpg
4d7aad8ceb56bf14d2afc34652563e3b
47014e1343c3d99d6356ee49a97dfd37ff2ed760
63144 F20110114_AABEOH jeffery_c_Page_012.jpg
98ed04246187dd36e93ec40076df6464
aaf91ba421f8f8bcfd44ae2243c0ed6533809a09
69088 F20110114_AABEOI jeffery_c_Page_013.jpg
c053e643301dbeb74f931a578fe142cb
bd69cc3fc489595cc4d031079699fd5ed153bde6
29435 F20110114_AABENT jeffery_c_Page_095.pro
c2333a9e25fa1138571426172623044c
9e3e7888ca897bbd4896fd328bfddd21dc991d73
47680 F20110114_AABEOJ jeffery_c_Page_014.jpg
508229b3d09e5cf3513516ed493e8c0c
4ce1eaeb51814256023a8c73b0c6014a24ef769b
61000 F20110114_AABENU jeffery_c_Page_015.jpg
8eb233c81ca88c88a4592268719f8473
a4052bee2226f358c5f771daa0ceb840cdc0d895
69356 F20110114_AABEOK jeffery_c_Page_017.jpg
9f73696773b1144e19b83c18de44e3fd
6c260a41dfa32c8f03bc25acb6e2f4f349f7488d
157379 F20110114_AABENV UFE0007163_00001.xml
f8b66f12d86b4044aa98440daa8d7f35
5db6b3e96ece14eb3a729d1671e167d5b1def81b
72243 F20110114_AABEOL jeffery_c_Page_018.jpg
dfa7dc5f2eb656008cdf95d71a028b0e
82ad2c148c1c4b4d2b64ba16679023ea29b2ded8
70210 F20110114_AABEPA jeffery_c_Page_036.jpg
a18b972bb121af6378cdd9fe652ae1c8
fedc4517e069bbc4dda0982e4ac458f6eb03b0b7
67561 F20110114_AABEOM jeffery_c_Page_019.jpg
33dfb31af3c2602f38e07ff311ccf7b4
c0920a197f8a3ff88f83e396c90d6e8631669a63
80496 F20110114_AABEPB jeffery_c_Page_037.jpg
70e1eb4da706ce226a4c86a3345e355c
253f14d8904f1b0d2e0b784fec92a407861f9d72
53486 F20110114_AABEON jeffery_c_Page_020.jpg
0a23b2f33a8b0b8b2a4c0ad99044a5a9
cf59d75ff22f415f57bd4946ac73dada561b6675
26019 F20110114_AABENY jeffery_c_Page_001.jpg
69c9c778450c2528c95d50e2e899b07f
7c04f790b6d6b69caaded57f33fe3f5e21e03c4d
68376 F20110114_AABEPC jeffery_c_Page_038.jpg
8469d6809e2501d6f7075ac5a86076ae
404cd9b45f6fa3694d0bdfc5998366bc9250bb7f
45689 F20110114_AABEOO jeffery_c_Page_021.jpg
28bd94301c91e420ccd8e9b0f9ad4f70
b4094a434f1126060b395177e3e95382987457c0
10323 F20110114_AABENZ jeffery_c_Page_002.jpg
762c5e1cbe870af36a6823faa81017f2
c1741e52404a6f8d763bfbea97beda43d5879254
63859 F20110114_AABEPD jeffery_c_Page_040.jpg
ae12828c8c63340aad6d0e7b642138f6
0c489fb15546cfb5d83f13535fe122197bb0ac3a
69810 F20110114_AABEOP jeffery_c_Page_022.jpg
cafd85ed5c67230e867d41dfe728bc3a
ef9d5539130a84665e47c158da7d5eba787e38c3
66015 F20110114_AABEPE jeffery_c_Page_041.jpg
e37204ca311b4f3a1587934621e31d80
4bd30041a1dfb217b364f8c71b218213992a8be0
68835 F20110114_AABEOQ jeffery_c_Page_023.jpg
358970af8e6529db11133123643fbe01
8a0cb4cb046c8213f1c3c0672cdab74f4f44c900
68723 F20110114_AABEPF jeffery_c_Page_042.jpg
d5169226b03aba4ce382ab8ba0c9dcfd
aaaed7c131a4f858259c64c9c4816be30146457a
71431 F20110114_AABEOR jeffery_c_Page_024.jpg
9358f5ebefaa4a6da0702c81e52ff12b
bab4fa5f4f1fd5b3542156d2a1f01232725c176e
70860 F20110114_AABEPG jeffery_c_Page_043.jpg
b802c86b8e2bc886d4fc2266d8d06297
bf31033db386f673f23abc4f6d48772fae371b6f
64745 F20110114_AABEOS jeffery_c_Page_025.jpg
7a0c92eb13194c879c48904f5402fbdd
17dd4fab39c46f31d3e23efc3347ebffd2ca544d
64043 F20110114_AABEPH jeffery_c_Page_044.jpg
6125b1a448b7b53e9df0bc2d5fdd46cd
6a87d9a3cf22da69afd23e1114b64e9fa8670372
68713 F20110114_AABEOT jeffery_c_Page_026.jpg
089b864ab4a5aa10fd54f26e9e84bd91
8603203223c87606e8cca76e11d4f9f47ccd70c9
73411 F20110114_AABEPI jeffery_c_Page_045.jpg
e2fe9fefbf03ca37ac76bb0934d59efb
1efba0b2201df72e18276443d94b74db9919637b
43458 F20110114_AABEPJ jeffery_c_Page_046.jpg
a78b746db2a337f954d7e7264d0cf440
e20f33eb7f69ba6d8038b86b9785ade6a85efd88
67023 F20110114_AABEOU jeffery_c_Page_027.jpg
eef085264ceaa731f2a32c28b79b62fa
23fbcab258ac822158e3beaec1483b2c2f1326d2
62720 F20110114_AABEPK jeffery_c_Page_047.jpg
15e7b9e79c4646898cc6eec57b795d23
981a6fb639f2dfe2786cf74b0009e0f04d0468cc
20955 F20110114_AABEOV jeffery_c_Page_029.jpg
ce30a245ec90f4a3f6a79ba17006ef3e
73d7063e51484236ffea3613ebee797f0ac3d173
70312 F20110114_AABEPL jeffery_c_Page_048.jpg
6ed07720646f0972763b8767ae52fe03
c03b8e24108fab7dfd9cb04af23710e4faffb978
55568 F20110114_AABEOW jeffery_c_Page_030.jpg
67d58fdb4e4a3e693a4ba7988a517f51
c74ad97fa2ed524c6725ca4e8d8195985ee7f5e1
46781 F20110114_AABEPM jeffery_c_Page_049.jpg
9cdcadf0938883a1f731052129fad722
9cd4842b37f216ec06f859370fd6db55c3d45528
71188 F20110114_AABEOX jeffery_c_Page_031.jpg
fe1a37f5d5a9ff0aeef92a05f3b54a50
7b76c2b459f9041039b86de8acca56f20bb04522
69241 F20110114_AABEQA jeffery_c_Page_065.jpg
0bb06cce50633794d847a0c6962077aa
3b53188539c366e7d64cfea6eb5d1c189f5e787f
76252 F20110114_AABEPN jeffery_c_Page_050.jpg
e392bbcdad4db855d2287d4ec657036a
6e2957b84e7ae94961781c80203d11d248e847a5
74428 F20110114_AABEOY jeffery_c_Page_033.jpg
85100c4b69e165660b0b39e3455e202d
2aa35b7fe0af3398906a4d830acefd4048215dfa
53451 F20110114_AABEQB jeffery_c_Page_066.jpg
eb864f7e69a47cb8878b786f10f99489
49b6c9012e0f21c4a92643fc4ec33c10e1be8e58
76985 F20110114_AABEPO jeffery_c_Page_052.jpg
20addb60d9ccc60ea65a6adfdbe52425
cf1fe8cd5da143c3183e2826336817ea046618cd
67226 F20110114_AABEOZ jeffery_c_Page_035.jpg
afd294eefbf09e53f4f8e8606732914f
ff437ef7983a3563d596705406519dc057c216a7
29625 F20110114_AABEQC jeffery_c_Page_067.jpg
5b19b9966bfb495368dc0ffea8569c4e
2af690676340dd32e6a6e4c125c1db7084f3fd6a
55002 F20110114_AABEQD jeffery_c_Page_068.jpg
57aca19ab72ddfae160ae6d66686a522
0209b4e9e8ff7fee69e20df212c971b02824376b
59494 F20110114_AABEPP jeffery_c_Page_053.jpg
f876bb9c3d769b5910dc195ab7122fa8
148444b641e9fb025298144f1cbff5dac4070775
73817 F20110114_AABEQE jeffery_c_Page_069.jpg
439c47f893d6728a76066bcbff4fafdd
673dfe956702c8f7bc864b1966ac9dfe31693ceb
69206 F20110114_AABEPQ jeffery_c_Page_054.jpg
70907852445ab2e51752f1fa3765b467
b590fdecfb2937ed561c7a633d386ce1401ce058
65877 F20110114_AABEQF jeffery_c_Page_070.jpg
9567ef4966e4a6f2f8f41ebb85a3ca80
8009d791f0f9e07e5b4a93e6c6c492a4e15d6355
65980 F20110114_AABEPR jeffery_c_Page_055.jpg
bd24d4d1b88615f4958560c324cfe335
4573b860cd2a55d8605a943d4b4c9d8c834efc0d
74635 F20110114_AABEQG jeffery_c_Page_071.jpg
d5afdb8779930d906933639c69eaa92a
c7a80dabc92da47a7fb84aae623ebd6aa9339416
71964 F20110114_AABEPS jeffery_c_Page_056.jpg
a1ec2d9232d965602c043b2cd5b6c2fc
8ccd7185cb22fad42ad9ba86ceec5f50701814b4
68105 F20110114_AABEQH jeffery_c_Page_072.jpg
0adc4c151e33cac76133dd1801fa6678
2244bb18d2775b10fa2730454f9d3bb60a4414c4
73455 F20110114_AABEPT jeffery_c_Page_057.jpg
37c1fd02d56452a59f8264a6e48fe55d
893d10aae06294790002d432329e8aaabe9e7cc8
64097 F20110114_AABEQI jeffery_c_Page_073.jpg
d32d0c127997f9b65338f8c286b5a473
8773d685b74f1de32aabdf9f06e4dc6c888fa830
71659 F20110114_AABEPU jeffery_c_Page_058.jpg
f89423d7ec06615506a9bb90f7e76b88
07bbbfca68b0559cb139eaf2d90e73814ded9437
59690 F20110114_AABEQJ jeffery_c_Page_074.jpg
eb8402b51115a05d0ce5dc6621d5579a
dda68d14043b5a055aea5e5f177e69e2835c0041
66313 F20110114_AABEQK jeffery_c_Page_075.jpg
f14a62a18c7d3e14dbc9189880c10d6b
ddc479fc9c1961aca0ac73036c7d2796ebf563ab
68184 F20110114_AABEPV jeffery_c_Page_059.jpg
37533bdd9b8e06cecb1bf88e6f922b79
4352437e9894ecc1749b2c359e081d2bf56df5f7
58443 F20110114_AABEQL jeffery_c_Page_076.jpg
f47179768bcc10bbbfde95f01a48c2cb
49a73902794168d6c983190d7b373be7c761105f
61926 F20110114_AABEPW jeffery_c_Page_061.jpg
a0524a709b1154e1b6d0dbd7bd129f40
9a07e21c8aefa0901f9dba0d25c6cee9ae2d8ac1
36742 F20110114_AABERA jeffery_c_Page_093.jpg
11fe9305d2fa4a5db95f283557995dbd
6ab7a2c952873884a6cdebbaa4190a6b85c5668b
44736 F20110114_AABEQM jeffery_c_Page_077.jpg
0190880188083857789b1bf5e32ef27d
a50f0c539dd239b56473715ace7d93ae9c250d68
53096 F20110114_AABEPX jeffery_c_Page_062.jpg
a8be6dbc336ddac2c90dad6efbcf7871
a588bad5c2a42c13f3372c022c57fb34747508e0
45488 F20110114_AABERB jeffery_c_Page_094.jpg
83a3848861b704d1a14343af9670613b
ab42de8fc001ca797653a8e23d1f1eb8d2d08b10
63324 F20110114_AABEQN jeffery_c_Page_078.jpg
d91dc046164660388bf6c09d073867a5
75d11aa9165eb2ea5bc94c61599dde1734d85bec
30550 F20110114_AABEPY jeffery_c_Page_063.jpg
71107f8ccad892404a0cccb57f3caa43
3f2667e1f5329ee07abd435de67706cdb54d8fd0
42048 F20110114_AABERC jeffery_c_Page_095.jpg
db96b89a8aca43d672ac5dcba7b905e7
d9f6e793b3a90e0bf6c3582a3d3bf25dcfb2061c
49769 F20110114_AABEQO jeffery_c_Page_079.jpg
eda972a3b6ad56aa7b7f5151ffefa54f
542ea601614ada0be103b5feef08e2a75bd36b7e
58144 F20110114_AABEPZ jeffery_c_Page_064.jpg
bd18c75dbe0b2b4a7c527a5bcd57a939
ce13db8b6b8ba78e1599b2c5f4aa95e549cbe6b3
42900 F20110114_AABERD jeffery_c_Page_096.jpg
46a961d2dadf5c168d909718009b5ec6
cecffdd4df743852c81faa938d67e99cef848e99
62870 F20110114_AABEQP jeffery_c_Page_080.jpg
7d79edd052552663d454c88205f3bbcf
6247992882ac674052f332f9871f051e7427ed5b
33810 F20110114_AABERE jeffery_c_Page_097.jpg
6c05df3ab3a17190b63acef3cdeb9acd
67678f94378b00e6328569f63993575999e5743c
59260 F20110114_AABEQQ jeffery_c_Page_081.jpg
0bc7f15a04dbe04e6b325ef3c29e3b8b
b74b524e38edd4daf53fee5d4a02839c97bc3886
27325 F20110114_AABERF jeffery_c_Page_098.jpg
dd6c240b601a30e8ae97928932518a0b
3b41c70878042dc7f57ce1ac94cc030fdc972b6c
51252 F20110114_AABEQR jeffery_c_Page_082.jpg
cfba487b952338a34a50a55cebdcf0fc
ebbcfdff287462db39d6bcec522e975fddac723d
75868 F20110114_AABERG jeffery_c_Page_099.jpg
c056f730135bfa0e167637fb6e90d0ef
65b5a4900ebb993e2acc6adac206230f914be6a2
55290 F20110114_AABEQS jeffery_c_Page_083.jpg
cc25b706c5360bb16aad0cb1c5ded963
81281a0f0875e35c965c5c324b31405a80cd8567
91870 F20110114_AABERH jeffery_c_Page_100.jpg
7148b42dedac9db24880eecbe66c0faa
3291cae88f27240f4dfa4d736511141e2f74b135
52430 F20110114_AABEQT jeffery_c_Page_084.jpg
8c258e7b335ebf819ccde7d3ea2583cd
6e1812d4939bfe7627ebb365e69412f22f26a117
95823 F20110114_AABERI jeffery_c_Page_101.jpg
8d53ac3f87aa57cbb93b7cda2716f528
67e6b5fbc5078b5e954fc42c437ee8440c868ebf
55982 F20110114_AABEQU jeffery_c_Page_086.jpg
da4f7a05df5bc5aa66d169a217357f92
f010469db92b798129c0cd5e449e82b61e420f17
85595 F20110114_AABERJ jeffery_c_Page_102.jpg
3fba11da684592355e10874ed252c8c1
ef4afb087b75c96a560da5f5d3092a4ad263333b
60999 F20110114_AABEQV jeffery_c_Page_087.jpg
065457888b723b2e0ff4ef852e8ca99e
42dbf7219213929b963fc2a39c629ec680a04818
43345 F20110114_AABERK jeffery_c_Page_103.jpg
fc8b6966af05cc82ebf17a30f6182418
f3ac32248f339c1fc29617092ed0665cf90e1267
39676 F20110114_AABERL jeffery_c_Page_104.jpg
6384a9929f57d68b11e9e0c6e5b2bdd3
18b61b8c4e0aaa9694414f4deb6b9d02db485895
73811 F20110114_AABEQW jeffery_c_Page_088.jpg
14e48bc266786fb6ba89b0decf293d83
a37e1c041eec2430d2b8eac77749d106a6a74fab
5666 F20110114_AABERM jeffery_c_Page_002.jp2
4b37245a0e609c4be76e632eb22f8e5d
2ef4614fd48478be82a96ec9f5077c54431612f7
57717 F20110114_AABEQX jeffery_c_Page_089.jpg
6f049cf1386fc3991e655d4f4e409c4a
9c58566da4566d9d947b0034389274752a389845
967103 F20110114_AABESA jeffery_c_Page_017.jp2
12605c6d47a7c3e8ea603420f47e3abd
a6db32a63b81fbc81ae5b5a4a83808b91ff1d17b
6534 F20110114_AABERN jeffery_c_Page_003.jp2
3662dce2ac5fbfc402f6ec97935fc53a
4cbcf81f64a1c86f219efa7d83ccf2bd5d0d25cd
31809 F20110114_AABEQY jeffery_c_Page_090.jpg
ceb10a5c9ca8c7b3a64260f94acb49ad
059d58edcc3b38c3f22ac10c6015d44d5ed19d0f
936338 F20110114_AABESB jeffery_c_Page_018.jp2
ed226b2b39bda38987cf662f73264d4b
9324a7b5b878bd1757b2e7e70f1557d0c1d55b1f
29837 F20110114_AABERO jeffery_c_Page_004.jp2
8847b550b5132435659717aad9d24540
06b3be0cfab7f4f882028dc02a4aff41c9ff8bd8
63937 F20110114_AABEQZ jeffery_c_Page_091.jpg
55a4007ddcf1c7a3eb97c96230a7f253
79bd4f212da49defc2004d29fc6c0325169c31dc
103490 F20110114_AABESC jeffery_c_Page_019.jp2
988d6bfa36bd559a26d3e00b71f5d4c2
73a7cdca6fa2b182ed14af18c582c05212e4a4ce
1051961 F20110114_AABERP jeffery_c_Page_005.jp2
946f5368f53f52d32ac042d3f2587fdf
0dce57fcdfb7875ec45c316548800470763b75cc
77845 F20110114_AABESD jeffery_c_Page_020.jp2
0c0c4d056edb44aeec20581ebba09b9c
6fd37cdbb86359b2aff5bc92b2bc6dcf0497e9d7
1051972 F20110114_AABERQ jeffery_c_Page_006.jp2
7e60b4c297dd45738f0b024ad26b24e4
10902077431599d023de7ae8bcd3c9ec0ea54ec5
544263 F20110114_AABESE jeffery_c_Page_021.jp2
dbf3958bcde87a0229468a57be5eb9cf
876caa857b6565cd11229552bfbcf9ce450c227e
155294 F20110114_AABERR jeffery_c_Page_007.jp2
29e6b198f81fc9401a6a101fb3cbdf0a
fa200fa8600912e9a8b9c38e2495405ec94c5c1f
105946 F20110114_AABESF jeffery_c_Page_022.jp2
f88fdfef1dc6a9da9ef2c5ba1882af57
e86986535a90549ac4b87ef6e6a4a09238691cc4
568355 F20110114_AABERS jeffery_c_Page_008.jp2
1215e7972fc66ff37434d6b29928c628
b5b795f32f58f18387168d812b8a31107517d661
103180 F20110114_AABESG jeffery_c_Page_023.jp2
a9ec5874c6f4c6600fde6e8ac41d1159
9f0a4ea28c89ea6f33e8a86907bf9fca5322ebe1
1051971 F20110114_AABERT jeffery_c_Page_009.jp2
146e4e1ba8e84eb3eaf971c35a0e293d
68fb3b13c6b8108a28d1b5a247693901f3305d62
107442 F20110114_AABESH jeffery_c_Page_024.jp2
29b342fc8e3689dce80f69e9732f2f64
06f78e13bc2dc0535c2059588012341ca06bd37f
81992 F20110114_AABERU jeffery_c_Page_010.jp2
719bec4fdb010cd0397beeec46bb2e08
5d6886ed510ab7d8a9015f64b3e6ba6cd5a14ef3
96466 F20110114_AABESI jeffery_c_Page_025.jp2
0a596ceb2b52d87446c88a2293007cf8
d6f2bf75900fe88add0ed969bc6737262bf1cdd2
54982 F20110114_AABERV jeffery_c_Page_011.jp2
dbceccd832f1e9f4fdc8086af0587ed9
49ca3ded91963abf4b9d51616622bcc081968b90
103084 F20110114_AABESJ jeffery_c_Page_026.jp2
9439cd8eaf384572a2f192376d03b2e3
08f973fc06aa45c3596c1971df5aab08dbcbd943
94706 F20110114_AABERW jeffery_c_Page_012.jp2
35a1511d180259d0c67617a319adc9c2
15b5835ed80ed6adc71db0157dcc648e4e806313
100073 F20110114_AABESK jeffery_c_Page_027.jp2
3e4106d5599445434385c3951953331f
3d8a6a9b33f64a8a552e6315bdffddb2396137a6
100275 F20110114_AABESL jeffery_c_Page_028.jp2
b12a71e1ccfbb7c3b14daabb40ffdbdf
36def261302f14bb896d0dd7dc84d8c139a308a8
103696 F20110114_AABERX jeffery_c_Page_013.jp2
7eb0efd7ec9f523bd621d9b37582e694
2e1f086ad289463af8627550694d585226c70e07
96905 F20110114_AABETA jeffery_c_Page_044.jp2
fb545339744376292aba70241f57281e
eb893ffa672da989cbe6188959403ed6adeda7d3
82277 F20110114_AABESM jeffery_c_Page_030.jp2
631ed55e16b8b0fbd089be225495e3bb
b2f45fe7c639af80d770eff3bc2e38c2985a1ac4
90111 F20110114_AABERY jeffery_c_Page_015.jp2
8da56b3a7aac42b56778ee84df3b99fd
f5ba0906ed7b1022b4b9da27b94c634f0920d8a5
110523 F20110114_AABETB jeffery_c_Page_045.jp2
1cf6001f0da942506b699d1cc6a4de0f
ebe044ee0f13ed9cfd3c2a49fa1d6577538c75d4
108109 F20110114_AABESN jeffery_c_Page_031.jp2
c3dac3cbf8ca692308b8666d5b669ba0
b1d65c0304f9b9627578980fd747ab7d56dbca3e
87640 F20110114_AABERZ jeffery_c_Page_016.jp2
6092ac511f552b08b60bcaec2d72d9ef
c783d8cef16a17111ca9cf092e01998fda0679c3
93214 F20110114_AABETC jeffery_c_Page_047.jp2
a94719316e9c810a70c7146823104bdf
8e4135e43112af198f3be5dcdf42a72deb6089e9
103412 F20110114_AABESO jeffery_c_Page_032.jp2
f9224aa88350fcf16fc35ef931fb284b
12f150ff2199d6eed0c4eab21ed17b99a20a4b3e
106410 F20110114_AABETD jeffery_c_Page_048.jp2
a941b33a8c6de8f2287bceebf04103f4
27e5a46dbecbe4de7a7e8deaecf4dbabbda1fdfb
110501 F20110114_AABESP jeffery_c_Page_033.jp2
b6da0886de32dabe9be330799295bfaf
b5b4e02aa34b3496323df4deaf9a9c958b030d86
1051978 F20110114_AABETE jeffery_c_Page_050.jp2
5bb66c9a21dd10b3c2e8cc3c08ae4942
10e49fbcdf70f4c96f0ef10e41b08edbc408206f
1051945 F20110114_AABESQ jeffery_c_Page_034.jp2
39972e87a20343904f6ca6563b85bbfe
a50e6d81a3d20860c4c2b9bdf67ec20bd81f249c
97174 F20110114_AABETF jeffery_c_Page_051.jp2
935031a5efaa65d6fc8bd5daef350e8c
f3beaca95d52d6ebbbf1a6bdefd1cd6ecc10c981
102181 F20110114_AABESR jeffery_c_Page_035.jp2
825123beb6b7dc61e780188cb82126bb
2d28c055ff1c995057dca1cb8f4fc608d19117a8
97125 F20110114_AABETG jeffery_c_Page_052.jp2
0c9777d89ef1d312b29599826fe744e2
94aca606f85acf458dd87492d38172ffb65b0f09
106912 F20110114_AABESS jeffery_c_Page_036.jp2
6968916ac0619143db5a213d2f13f1b1
7314e98012f057f0a4b3c9bb2d682d7de5d8b75a
86231 F20110114_AABETH jeffery_c_Page_053.jp2
e3b6e3da19f288913f923e420d8c65be
a3320b1fc1a9a03608690f9aacb8e6a1bfa4182f
1045251 F20110114_AABEST jeffery_c_Page_037.jp2
26dbeb24121068b8ef63e00144a8b512
401339086858ce41b7dc573a53ae8c548ed388a1
100043 F20110114_AABETI jeffery_c_Page_054.jp2
747c823d0f2e41a7392da73ef5628d75
e968f75c72f3eb2686b104758d5205aeddba97f5
102732 F20110114_AABESU jeffery_c_Page_038.jp2
c78561ace7af305ba4fdb5650ebdbd17
57b366a81a21b3c0f35e18e01c54220ab9f0dc14
827882 F20110114_AABETJ jeffery_c_Page_055.jp2
831079e2b47242dd869f8cfcadceee89
ddeebfbc40955c773e8f2839390517b6285a2ccc
31247 F20110114_AABESV jeffery_c_Page_039.jp2
3a785c309d2c1064ca3274b89a230118
812666e08d24240aa74c17eb5c9ba0ddee8604e5
105896 F20110114_AABETK jeffery_c_Page_056.jp2
3f108b7dc771f71009cdb02eef57b889
4fe650ba23c1b62f962b9b47faf8c1d53a1aa044
93530 F20110114_AABESW jeffery_c_Page_040.jp2
9bf8bfa2203f32e640d3b4c835dcdb13
196ebc5fc65c798be2b2dcb27f6fae37c8883952
1051969 F20110114_AABETL jeffery_c_Page_057.jp2
d891687c0e0b6296e3bd1600c89ec957
794bc8b6f48c58911b3d8c920d5bc1f9452e886f
100200 F20110114_AABESX jeffery_c_Page_041.jp2
9af6da0461adc1f575553fbbb3ea2f34
c9a7fb9101cfda168825d6fdc7212598b6ff9477
80843 F20110114_AABEUA jeffery_c_Page_076.jp2
007fb58a07f4ffa8bc6ef7a4fa776877
37bead3d5b599b63eb7e72138671adc3731dc193
108144 F20110114_AABETM jeffery_c_Page_058.jp2
682af7d5ff4e0576aff946533f43cc00
17151a6d4a8dbd498d466d470c9206e7f3125258
60536 F20110114_AABEUB jeffery_c_Page_077.jp2
33176df93845e41ec2eb40d08a37e4e3
6b94e4bb5035d04dab4d4b397a40ab97926da01b
819431 F20110114_AABETN jeffery_c_Page_060.jp2
5f94a29fdfc484aa061a03dcb10e90fc
248a651371efa7ce78f2c0595995689f77f70c99
930437 F20110114_AABESY jeffery_c_Page_042.jp2
9f1e19e189f43f0be49687315fcc2e71
842d12c6ff81ff2396f073b22e739eb9bbf529b4
86832 F20110114_AABEUC jeffery_c_Page_078.jp2
4d88ef7ee6b6fc4f67163e18383ce37a
903b510d3395e9d2c87e2dfc9d61476d34e173ad
92210 F20110114_AABETO jeffery_c_Page_061.jp2
4b09b9552a718c970a8ee37057004248
a51ff6f75f86bbd354bc394c4ff3dff5470dd6e6
1051927 F20110114_AABESZ jeffery_c_Page_043.jp2
6012baecaddc4d9ee17872cbf41eb37b
1cba01f6aa522cc75030166dceb47503abb0aacb
69196 F20110114_AABEUD jeffery_c_Page_079.jp2
d6759935e87888ece325a36cae0d79c2
0394ff7c17a3345ad990581dc7ce66356079f434
565295 F20110114_AABETP jeffery_c_Page_063.jp2
9407d3f8decf669b1760a3f697655902
f9fd09c765854abf468ee7549798a7d57212ef03
88006 F20110114_AABEUE jeffery_c_Page_080.jp2
7f2aa46f9faf96adf063a77c3ea336bd
83d5160bcce4bd78743067b5c566e9fb5903b5ac
84802 F20110114_AABETQ jeffery_c_Page_064.jp2
53ddf511947b41646bc936997c3edf18
8a2cec1f084b17d8073521354cee41e3dbb33d66
82824 F20110114_AABEUF jeffery_c_Page_081.jp2
f7be09e8e1aa5940cff9922779a10b4d
11e2f9b1b437564982d94de147a34c937ef4f4a1
105316 F20110114_AABETR jeffery_c_Page_065.jp2
439ef301a31d9921c7936be475704c00
0497c942232b54e50a896e6d6567b01add65b71f
52482 F20110114_AABFAA jeffery_c_Page_045.pro
27243b6b213a9a1a66c4f872171db31c
46f1016d644417a3cb3f420f21d5068c55dc51af
69577 F20110114_AABEUG jeffery_c_Page_082.jp2
09ab0d6927d07daa18362e7cfaa70f2c
49465415159721e8cd162d82e6b59bf94284f90f
78502 F20110114_AABETS jeffery_c_Page_066.jp2
8edc570e04978128bcaed31091f463c8
7a90447d8b76101bd7fb467a6bf6bbbb00a8e17c
25956 F20110114_AABFAB jeffery_c_Page_046.pro
7d7c0d70900347dba807b33616f3c3c6
865448a1603610afa91fbd7a209b0d4a592d659e
70498 F20110114_AABEUH jeffery_c_Page_084.jp2
c098dfd6f2e0ed55730a85d68b58539c
539ad2a87d78448f4f4b53cc8e7ff2a6b33371f7
293251 F20110114_AABETT jeffery_c_Page_067.jp2
dafb27c614c930b867f1307eb7321352
a3452d8c120550f1be1b10c4ec5264e906e19611
42796 F20110114_AABFAC jeffery_c_Page_047.pro
487dd0cec33928df7604f2d4a73ed1d2
360e9c753bef73f00638be9ed570a1786a9f722b
79863 F20110114_AABEUI jeffery_c_Page_086.jp2
3b08220c48cf5b6e98c90e4d62679dda
db3b7a8d9abd0b3f6baf44f1e18b9e0d41205436
78099 F20110114_AABETU jeffery_c_Page_068.jp2
13b3f9f8dca2316418d743ecca810ed0
29ae814bd92b0a1a21d08004b7db535a9b008922
25689 F20110114_AABFAD jeffery_c_Page_049.pro
d6bac43f80b6f321de69731326a39f18
8ca968463970d34eac2a799bbd161bfb7c7e0439
83612 F20110114_AABEUJ jeffery_c_Page_087.jp2
7e64bb14de8785e43ef3b54bc8e527f3
103fb451cfec55ce3a06ecf73fd7de25920cf9b6
105392 F20110114_AABETV jeffery_c_Page_069.jp2
191e9ff63c926f7e51628b2b90e87246
d42380e419402b304ce3d1207a3cf3c8331cf8d7
56353 F20110114_AABFAE jeffery_c_Page_050.pro
26991a425c22c846c5b6bf3ee0366158
68df64da97dddb5da2c56c6892c670a277735200
80964 F20110114_AABEUK jeffery_c_Page_089.jp2
3ab77e915917fd26402517dc1a8f4393
3d9de24cf1e582fe967f063991f7620674ecafbf
94489 F20110114_AABETW jeffery_c_Page_070.jp2
28a1b542cf13931070f766ea50264721
f60e0d81af61659e92448728c007113fd449e9b8
41646 F20110114_AABEUL jeffery_c_Page_090.jp2
9b77d110eecc24215485e6083a358fd7
b3a33ddb6cc9e695d1a0f4e72df25a97e169cd28
101423 F20110114_AABETX jeffery_c_Page_071.jp2
921e2b294b0f395065258d5fd75586de
fa80afd66ed039a19aadbf1541e41b8f209f8fea
45310 F20110114_AABFAF jeffery_c_Page_051.pro
8588fc0aa3e5ca6b0e736b4f4a2bb28c
29c6407079419cd2f9ab87bcb52f86d09d4e6c54
90102 F20110114_AABETY jeffery_c_Page_073.jp2
363a3f622a5519ef8136e3c6ffbed691
c82feb66c044bd2f560ac3c6a151f44fc9b40ada
41719 F20110114_AABFAG jeffery_c_Page_053.pro
d5fd753242ff913743f3d6f066d8aa21
7a4269b25eec9bf4a0ce721ac3468e4e0a85f763
F20110114_AABEVA jeffery_c_Page_003.tif
bf6caed45b5c7c02e12cd546a4a6a8f9
3e6866dec1b30ed2c5d37000a2a0885cd254f420
92592 F20110114_AABEUM jeffery_c_Page_091.jp2
6e889cfa1c3ddfa8e4a0b3d5f96a4af8
8b52a3811a5fa6a4d33982a67c41fceeeb22768c
51485 F20110114_AABFAH jeffery_c_Page_054.pro
207bc61f37eeac0d5a20f1ffe72f1e70
c347f08390704719e513c8ed22e8992bab1332f3
F20110114_AABEVB jeffery_c_Page_005.tif
f03006d968938a6095f43f134fd38869
cb8aad7a073d65b7965abd1365172b97e428217f
51512 F20110114_AABEUN jeffery_c_Page_093.jp2
4705a4d01cf43f6794dab66be39fe82d
dbc54492e77bfac2c5b7afb19e03029b766adb1a
91060 F20110114_AABETZ jeffery_c_Page_075.jp2
aacbafe25310bafc68cc430175630bea
74da026895ea3b9ee43c483730c3534ae4aaae2e
52894 F20110114_AABFAI jeffery_c_Page_056.pro
71eebd606638d9ec82e63fbca71d4409
c1990bfc5c646e5684f7f9c5319c538db6709ae3
F20110114_AABEVC jeffery_c_Page_006.tif
34bb60000f439d674322bac606afffca
33a5be293109eb090bb9eaa0700e34c600dea33a
61920 F20110114_AABEUO jeffery_c_Page_094.jp2
6568c4c9f4b961b63b310524f52b4f41
6da37f722a084e1042b27ec5307d1d6f6be08922
43084 F20110114_AABFAJ jeffery_c_Page_057.pro
18cf9448746847bef63e336b03a660f6
ec822b042989649ad4998c4f56bf2ad86dba4892
F20110114_AABEVD jeffery_c_Page_007.tif
9ed2451f58841be139d011f781732e4e
7189e34f96e664dd0a8bec959a5e6c55441ebf0b
59974 F20110114_AABEUP jeffery_c_Page_096.jp2
d09fe774e1a885458b63d03843f35545
cf0dba1097c9e5edcc5671aec9f7c1a2435669ca
51653 F20110114_AABFAK jeffery_c_Page_058.pro
11960a54b3a705c79df1a48ea6f2857c
455853adf88d3ce2f0cd55b2ed0d958432023594
F20110114_AABEVE jeffery_c_Page_008.tif
c62d97b669bb03d517ca22e511e86ece
21718b83e5daae6126fec3a88da9fe875aa5a286
45838 F20110114_AABEUQ jeffery_c_Page_097.jp2
70e4d28403c0577835866d92928871f9
fd1a6eedbf59f4516f65ac5748d642766073007d
31961 F20110114_AABFBA jeffery_c_Page_077.pro
415a6ec3f245216bf96d79e74f29bf9e
d064fcdaba23b75eb4b72da9b35ef89b554876cf
46155 F20110114_AABFAL jeffery_c_Page_059.pro
f0fc2d6170d3154c58e12c24d7321748
3d538d4a3d9d35365219823bbb1c878454344aff
F20110114_AABEVF jeffery_c_Page_009.tif
29802a12d2223284e3a6213d353d6f34
9e05f0427eb43ec5d38051e501e026f7a3cb28d2
35849 F20110114_AABEUR jeffery_c_Page_098.jp2
6dadc475148f600e35554b770f135456
baa3417e6a626f3d1c15488363c3bffdc7205fd1
36867 F20110114_AABFBB jeffery_c_Page_079.pro
4f441fa4ed4a990c74ed917168dcc2fd
5e1e08dfd37797a043ca779c26067dbbe7ea2070
32228 F20110114_AABFAM jeffery_c_Page_060.pro
5352877f0d784caf0b2300f13d13798a
780220fd7e4382c6523ef8054e81b4c1c4da3823
F20110114_AABEVG jeffery_c_Page_010.tif
bb2a31617fb3599c35671e7c7c632953
b32198a3cd00d5697ef7cfb4b306763d0ee9729a
116489 F20110114_AABEUS jeffery_c_Page_099.jp2
31fb5b568e4e88279955e877f45668de
1b9461312379e00bcc5cd131d13c7a1053ebfb41
49423 F20110114_AABFBC jeffery_c_Page_080.pro
ec7e588e71cfcc11986bde556e1c7776
52508e601a1279afbb38a963825a1cc64dfd3f72
42014 F20110114_AABFAN jeffery_c_Page_061.pro
60a4e463330de532034d71e9d58e4ffd
e59b8302c92bd9be7abc5b43caf1539f931619b9
F20110114_AABEVH jeffery_c_Page_011.tif
38b85f3f7962532f32eb44ba742ef31a
86a89a45143f2ffe7ec9455a656c46299973cbe2
134147 F20110114_AABEUT jeffery_c_Page_100.jp2
9dce184c9d060d7299f9b1db56422317
04cfe2768b42cc12ec7f8936ef2c46c5457de083
44683 F20110114_AABFBD jeffery_c_Page_081.pro
865095413337ef37d29fcf4adfc9e689
95c8c43b8b0c12d8a16a0a5d978c26d180e63044
25257 F20110114_AABFAO jeffery_c_Page_062.pro
7fc6d4edb69f9226c7671bebdaec033c
2735a79aab6aed98d6083c1cd674e3dc07073d4b
F20110114_AABEVI jeffery_c_Page_012.tif
3bc7bdee21ec742535f01dcc487a97f9
808727c8763db769e30a4312ba01b3cd6c0898eb
1051981 F20110114_AABEUU jeffery_c_Page_101.jp2
c0906ea2a4f91573689f63908032477f
d7a5c8eb2c8036a84e83f73731cca4c242d127e7
38008 F20110114_AABFBE jeffery_c_Page_082.pro
deb688045fba97619c1bbc6464f73b0d
872d4294f747025fb1511e23f9a000329181d137
9419 F20110114_AABFAP jeffery_c_Page_063.pro
16c2e4814f20a67357d8aedfc438d04f
e297b80ea54be5ef0f49955c5b3984294abeaf85
F20110114_AABEVJ jeffery_c_Page_013.tif
8851dc80562f165399991fca8d50e1be
b26aca8efbc6c43488a91d4dd3884397cd3278f5
124718 F20110114_AABEUV jeffery_c_Page_102.jp2
c00e215f053ad3aec7b2dbda105a7ecf
3bee4d7686fd6279af802e98c6dce97752d1aa28
41583 F20110114_AABFBF jeffery_c_Page_083.pro
0d06fd2cf89c23608203e35000457488
42b43e6919eda76fe38c3a20c4b4670528b97d3c
38685 F20110114_AABFAQ jeffery_c_Page_064.pro
97d7b7d9dc438d379743b702f0d75881
fa51d3b28662279cfb7f3bdd221ba1b762e5f312
F20110114_AABEVK jeffery_c_Page_014.tif
d1b72a08cfe27ecebc90cf1c237b5e8a
e98e5182061038a213cc8e9033e696d47ab19c75
58212 F20110114_AABEUW jeffery_c_Page_103.jp2
d228335c1930314556b410a38b4b910a
e03a36c7cdf7ef4a761b87924f00c33d5eaf4b82
35405 F20110114_AABFAR jeffery_c_Page_066.pro
a6668924426aceff7fe3b7eb3def2d87
6459f59dc930f845740ade30ad455f9285ed3d32
F20110114_AABEVL jeffery_c_Page_015.tif
047430e34a33608e8bcc0a197fe75f5b
bb291bdd65261034b84c80d63da8c0db6b787eff
54718 F20110114_AABEUX jeffery_c_Page_104.jp2
c745098cf9f346576a4aeac9530e8279
d011811ca7079b337002b3acebfda081b9b23017
37343 F20110114_AABFBG jeffery_c_Page_084.pro
9bb8fee375ad584f2c3be54b0e41d271
bbcf7d6316b504ba8a6f3f4fca8a9ae20b903f47
F20110114_AABEWA jeffery_c_Page_031.tif
1826c791e4af587ce56595d205301106
d14eb063784a59f8c3e98706668597594404133b
2409 F20110114_AABFAS jeffery_c_Page_067.pro
88983882884937124c19f3d753210c43
10b2db46177b55d85ac14f99574e7d0fd7e0bf8b
F20110114_AABEVM jeffery_c_Page_016.tif
0c03c05caae09a8a1011ae53360eb84e
bb2b78184208479d2c6902e261fd87837f0855dd
F20110114_AABEUY jeffery_c_Page_001.tif
ef8b1f38631e9985674459369adf1a66
63207d3220d01fb7cb580a85408a911b8dc97ce0
33262 F20110114_AABFBH jeffery_c_Page_085.pro
3a110bfe7d83b6584f1b641233e90882
b3fd13100ee56ea72aa34690313a77ec69450646
F20110114_AABEWB jeffery_c_Page_032.tif
1d54926c47abc3e5a5dbf5db2218c81b
d2b0fc116ac1abf0a01d54cb4965ae77fb1ab86d
40171 F20110114_AABFAT jeffery_c_Page_068.pro
bfd04fd2b62691c3076fb9e9ea27a3c6
be3d864ae77b3d11060dce0d7a1638b9501b408c
F20110114_AABEVN jeffery_c_Page_017.tif
2fb52bd79a57018c3237ffc22c323117
65951ad29e41c72b24fa94f7d82259d703cc8373
F20110114_AABEUZ jeffery_c_Page_002.tif
153b7fbaa2b6d880d7d548675df953b9
d76e0ae33a35d2f9307dc7bd21f8df91dafaed32
42378 F20110114_AABFBI jeffery_c_Page_086.pro
c7be9aef498381e6a3615ed0f211188c
6bd9decb2eb8385d4408667ee9ccf2aca509b8b2
F20110114_AABEWC jeffery_c_Page_033.tif
f2266226420efc50af200e361353d864
5167c53482419f2709df45bdcebd09b2a7f02ee4
58089 F20110114_AABFAU jeffery_c_Page_069.pro
58062488a20952769076f64e8eb800de
20fb7bcf9c65c70c81800ce0b30b9b6ed1f90108
F20110114_AABEVO jeffery_c_Page_018.tif
0259bffdabc5afd8e09929030e453d32
c33a05a93e44e1d4b80c1def5926ac2d566e582e
F20110114_AABEWD jeffery_c_Page_034.tif
733dd8261583c778d73a3090d1c74166
1211916af9dcbd08f6d85962651ddc5cc34b3ab8
53129 F20110114_AABFAV jeffery_c_Page_070.pro
14e3ab97865f7865cb5b918213f78a71
bc753a90048af36fa730672008f727e3ec3700cb
F20110114_AABEVP jeffery_c_Page_019.tif
d753a49da2d245a2dd67a3a29e324ecd
105987546bff2ef2c259f3e80e7321fbdd85441b
45178 F20110114_AABFBJ jeffery_c_Page_087.pro
f728def7119544e546dcf4c16645d516
70afe85ab209ace1af95786a10afaee7f4f93b6e
F20110114_AABEWE jeffery_c_Page_035.tif
9fb45a63f71ba9641e3893f8a950acf3
5951f100e98b55fb78c8cae6cf9b84222a2aa0ea
53394 F20110114_AABFAW jeffery_c_Page_072.pro
ee14c7fb9faa5a81823f419b132e2d18
208dc9443ef3ccbb1ca0e390747e2e77100bb9de
F20110114_AABEVQ jeffery_c_Page_020.tif
986872ba8cb9678a614a3cce920ba6b1
7056a34abc604841dcf43cebb58f6183e8625219
59240 F20110114_AABFBK jeffery_c_Page_088.pro
80ba6f3d5769e117795a12cb7c945c9f
2b30bea6fd3dc6fa1ef2285f256b8da5890051b9
F20110114_AABEWF jeffery_c_Page_036.tif
d077a07dd4765ec62fd1fd3f92edfb4b
47be3b6861fa0985493db2d5c51a93f7bb49abb6
51216 F20110114_AABFAX jeffery_c_Page_073.pro
4121033d93dabc243435f610eb86355e
fcd000d88126f2759399827830ae5b222093890b
F20110114_AABEVR jeffery_c_Page_021.tif
003c7afe0d74f8cb9fb6a496e083fff1
5e351e66ec4e96e1ab4c94efb2d7ce86794182db
115 F20110114_AABFCA jeffery_c_Page_003.txt
e09740047fb6398f6bdba7483230f260
5fdc19c7e7a78dc04dbcd91d4f40321856dca413
43087 F20110114_AABFBL jeffery_c_Page_089.pro
89689b7f45d38b7e3e4729c2a5d34bdf
197740971a86608a87b3242a06902e5f4c9a5589
F20110114_AABEWG jeffery_c_Page_037.tif
dd50de0ca2a6b79546d42013a5245b0e
2eaa2dd2053068a2bd0fca20575af6bb3329f162
45836 F20110114_AABFAY jeffery_c_Page_074.pro
d29944daaf8df95251d0c85ed2cc63d1
ae6032117ab7a20f01915253fe099d7802484faf
F20110114_AABEVS jeffery_c_Page_022.tif
f5b98960cbfe6154163105fa18f00d13
ffc371258b9775d399a015c0040f66b0f76a3527
533 F20110114_AABFCB jeffery_c_Page_004.txt
7512e9d07bda3b082fa577415560eaef
8dda02e58f092f6e13d982cbc4aeb3d67110e433
48891 F20110114_AABFBM jeffery_c_Page_091.pro
3af3224b9aa5d7a8eb69db2be8ce6031
e69a68c1b518f18c90f7511cdb1c154dd252486f
F20110114_AABEWH jeffery_c_Page_038.tif
74d05bc57618cbc13f6f4612354993f3
212f821b05d1acf39b7b2fdbd40bdf38d06854e4
51842 F20110114_AABFAZ jeffery_c_Page_075.pro
86e925523c9305361b6dd0a53d653602
2b11afdd4976f1e1ace69cf57e03fd2a869d1e45
F20110114_AABEVT jeffery_c_Page_023.tif
3cbd5e9369a8d381d7014e01125eafee
73e69c1f90caca3dc721c5a33002c2704482e825
3133 F20110114_AABFCC jeffery_c_Page_005.txt
6264b1341944fda07181e9b92c633c4d
9e7ba6bcc657d1a966a938ea3e2ecb8adaf3d6c1
36462 F20110114_AABFBN jeffery_c_Page_092.pro
3881ae017d5ab5128a35fd96c61cbf15
a6f9c2abe3431d64cb94429cac09900825694f46
F20110114_AABEWI jeffery_c_Page_039.tif
954d883de147575199fc674316594c18
28d4bae7d8db6cc9d1103d39cdfb7d1c2603400f
F20110114_AABEVU jeffery_c_Page_024.tif
9ad577d23f0d6438c130166b45c1559b
0450cc4936d9cfc47509147f0d1fa159c3b6398c
3831 F20110114_AABFCD jeffery_c_Page_006.txt
7e8681ac81aa649ebf03099ccec4cfa8
2ae5eeeca6fead9f2272b5149bab4290c27dce2e
25352 F20110114_AABFBO jeffery_c_Page_093.pro
018888008b2d9ea47f7b2b43347c69be
738260e1c1416a8ddbbe3e1c7e6360434d69a56e
F20110114_AABEWJ jeffery_c_Page_040.tif
4a0572471135a11e615bc55b0c27a167
4e0aef0d68f71d91c7ef8a4e862ece9f9ee66310
F20110114_AABEVV jeffery_c_Page_025.tif
12a3af86d722baa0816c9b5896035d24
1b23bdc12969ccc981d26586709d6354c4cce57d
275 F20110114_AABFCE jeffery_c_Page_007.txt
dca43238a8adb0603396231d461f4771
d09f8f8bcce9e5c650133756a5364c98cb7f7bbc
32012 F20110114_AABFBP jeffery_c_Page_094.pro
2f37ac24a632293e98c8c23cec195a91
7c87ca29cc373eb16abcbaac28339b48ae5c1547
F20110114_AABEWK jeffery_c_Page_041.tif
956382c39a8440df7fd05ac1a42235be
b8a7953ae6efbeeda08f99a91130a789958e3deb
F20110114_AABEVW jeffery_c_Page_027.tif
386e55b9bb5f9733ad39b3f09915ae75
e52933b3c2a57218b87d13070edf38423ab5690d
791 F20110114_AABFCF jeffery_c_Page_008.txt
39bedc0f91794031834280ba57728947
8b738aa92e8db3a7f9dee4412011e8b76e2c148b
29756 F20110114_AABFBQ jeffery_c_Page_096.pro
616df8f3ad19d060522eb4b740a09a63
8a551891f48a9fe9e596a832f1ee2986cbbd796b
F20110114_AABEWL jeffery_c_Page_042.tif
5c9fb521e057cb3c867a1d9d4c8091f5
845a2ee59d95f9e4e702bef5803565e7b9dc28cd
F20110114_AABEVX jeffery_c_Page_028.tif
8840b3907df138cbc93645b15085d243
e63e7dd276bcdb765bee029823572335f7eb25ce
1799 F20110114_AABFCG jeffery_c_Page_009.txt
7936435318b461b199c5c6ce7903b77a
1dfb1f14fe2517a82976f96a4b1c4caf294c28d9
22224 F20110114_AABFBR jeffery_c_Page_097.pro
241b250c4f243ea8dbba9612139e7f37
51136f6ddb54dfd5c741ea583aa24c255844960f
F20110114_AABEWM jeffery_c_Page_043.tif
57fe2991a76a394d514d7ee90d6a9d28
da7e31fb306719c2742e39334b41c2023e0b9e99
F20110114_AABEVY jeffery_c_Page_029.tif
80f6f9395d97084651af63c634846b16
f0f6de439eedadbbb042e51b4e0a80e552c9edd6
F20110114_AABEXA jeffery_c_Page_057.tif
9fc260fe0cb45ab2049e442016b7f13d
7a60df7a9e5bc56de8962f4f550a5af5b508a68f
16619 F20110114_AABFBS jeffery_c_Page_098.pro
b70842589d49963e303c267dafbed0c4
ea34e4156ca46ab08826c0565efda127fac4bfde
F20110114_AABEWN jeffery_c_Page_044.tif
bacf01dea44e7081efbe61a05f8060cb
183b9aac79ae1cd13982fe897a308df55c93395a
F20110114_AABEVZ jeffery_c_Page_030.tif
87e91323ca17f605f33e41de1dec1280
15f8879289fe5cba85588865e28d08f57b0cb59e
1629 F20110114_AABFCH jeffery_c_Page_010.txt
5900384978ac3b0fa32c17c06534f192
e5715addc3962ec44ce688be6e6ec676b125c85c
F20110114_AABEXB jeffery_c_Page_058.tif
da87dfcb65684f127e19d40f6e18d538
063ba2f6e373c4f40a72d15bb64f1cb1e4e48510
54283 F20110114_AABFBT jeffery_c_Page_099.pro
16d8ce307623709fe406299269c39a17
ecc274b0ae1386e6ac6008ed1ba28878ab29c2d0
F20110114_AABEWO jeffery_c_Page_045.tif
25b04bcc1ce3ef3a2cfbd9ac4caf72f0
b018e58b1cd6de2710b0c46f56d15b85bf453b7e
958 F20110114_AABFCI jeffery_c_Page_011.txt
7e342d38a4cec952feeb3262b92dc29d
7b13d49b348d8b195a371eef77528ba539f53d37
F20110114_AABEXC jeffery_c_Page_059.tif
52b5cac362ba1132bbd68c0acef67973
a2fe122b8267bc610d2c4bd89aa9a760896ffd4b
65659 F20110114_AABFBU jeffery_c_Page_100.pro
c5f07054da69e241de43e6740b8ee2af
69b740aea11995620b0e90487a091c27a53e7688
F20110114_AABEWP jeffery_c_Page_046.tif
519e63351319b8c7d50eed1aa20e58b2
9d6fa6145a86c6739493c3d05c418fcf0d947cee
1860 F20110114_AABFCJ jeffery_c_Page_012.txt
c31ed1b1c012335ed72c02bfb4fb930f
bbcd55f1718646d590c94aecb92718f5428aad36
F20110114_AABEXD jeffery_c_Page_060.tif
213aed15c39a19330b74d5c8195a7b4b
56f627534b14ba60368bd5f2583b8f88a435efb7
62762 F20110114_AABFBV jeffery_c_Page_101.pro
e4e72a560732eeef1c778bf8dd36abfb
392b5ceb5eb2e23d80d164f90cd277c983287475
F20110114_AABEWQ jeffery_c_Page_047.tif
168db73b9c93e15b71f8f7961b24fec0
190072a02265af1066ef00fbe8900a80cc59a22c
1919 F20110114_AABFCK jeffery_c_Page_013.txt
42c32a4f243b84a1303afa61b78b385d
6f29776e7a2d34c3f693efa94fe3e68c9d9cc3d7
F20110114_AABEXE jeffery_c_Page_061.tif
7aaa0fd41654f0e3d2f1008e82b01124
afbe9ce84c7a4cb8051266dc1ff0fe984ab7e736
27210 F20110114_AABFBW jeffery_c_Page_103.pro
4fa8fa1757f884b2e479cc4c893428f9
ba9fba14a6e0ca1535317aec1290d6a97a36a14e
F20110114_AABEWR jeffery_c_Page_048.tif
cb99d4400e7150e38898813ea3624fa8
3507a6070ad37ff9a3abc5e650d8898c4a57daf4
1551 F20110114_AABFDA jeffery_c_Page_030.txt
50b4826ccfbe9bc98a57c0efd5b1b2d9
21670c8bf466a285018c5a42af595bd062e1005b
1228 F20110114_AABFCL jeffery_c_Page_014.txt
259ebee25549bc781eb3b4ca82cedcd1
980ea8d8353cadf75b074464253b26d31eb186d7
F20110114_AABEXF jeffery_c_Page_063.tif
3d4957f09fd59d1eb97d54a8d806f9e6
eb2690e5f302986cdaa2b588f53686d769f9d24f
23515 F20110114_AABFBX jeffery_c_Page_104.pro
f580bb7aa88141f4d5513509dd29f373
efce634cd4bf72c39168ae63dce488e01e5ce969
F20110114_AABEWS jeffery_c_Page_049.tif
31fc7d6acc830a7bc38ccb12530c1ed4
c4fcfec93e7bfecafbbe4defa6d65e0690d6ccbd
1967 F20110114_AABFDB jeffery_c_Page_031.txt
09e126845b07a08da68a5d4bfef0851a
8b21e0873543d9076d08f4d143aba2d7ab44ced3
1802 F20110114_AABFCM jeffery_c_Page_015.txt
b747df9fa835b10de576ecc7d42096d7
248284107b6237909d9f3176924e93ba805faf78
F20110114_AABEXG jeffery_c_Page_064.tif
8d0e65c9a3ee1f17cf99f658c6c7bdeb
29d7965b1f8f04c52eb1c82595bfdfc7497454d2
506 F20110114_AABFBY jeffery_c_Page_001.txt
aab3e1cecdb11dd55c919db0cb103c6f
704d04783ce93e992c9b4f7c24d54cc34927a1af
F20110114_AABEWT jeffery_c_Page_050.tif
dbc182fa04bcc477adffa80cf38ba2f1
071ffda52ab63fb98f12296470fb1efdd7a6bad6
2019 F20110114_AABFDC jeffery_c_Page_033.txt
7953bd09076ebddd2af7a2249be76dd8
25928704cdbd1eebe962185949dbaa3b86d022a3
1636 F20110114_AABFCN jeffery_c_Page_016.txt
78bd6f7a01cb359ae921c89b2addf31c
8565fd4975c9e2eb3c53a9c62e16cf57326afc7a
F20110114_AABEXH jeffery_c_Page_065.tif
270f417b773ce78a962446032a72f402
55ec75cfe502cefc76684c83f9371c85e6f41261
111 F20110114_AABFBZ jeffery_c_Page_002.txt
2bf8074c3e18c0b7a0b329d594fe0d8f
8b6c558d9abfb031868a529753c1e69e79384c13
F20110114_AABEWU jeffery_c_Page_051.tif
bcc8e37794105c15ddebbb4a3e196f87
667bbb9664c076c070531ae9f7a7a83dd9a76831
1765 F20110114_AABFDD jeffery_c_Page_034.txt
e8e21eb754ef7eb317dab81bcc9f9660
c17fd893b6141787ba4f5a7b18eeaa7e74b0f237
1964 F20110114_AABFCO jeffery_c_Page_018.txt
e5a7149b14e0d541f0e0b529ba399141
4f99cc6785211fc8c8b5130e827471fca33bb038
F20110114_AABEXI jeffery_c_Page_066.tif
dadabe37e440b8a42c4b608be7e40b01
03148af5455ac8257077534309c440d1e67ff453
F20110114_AABEWV jeffery_c_Page_052.tif
5243921bd12898978667b59ba387e46e
af30fd150ad768fc2aab8f622255e364ac08160b
1880 F20110114_AABFDE jeffery_c_Page_035.txt
2c685e4bc4ccb9adfd1e50c41696716b
b8134cba7d40a7f77785789613988e890ca9eaa9
1911 F20110114_AABFCP jeffery_c_Page_019.txt
a39c87ea3224c2c3e5a31347ce1f5a37
cd7483e593b0d9bae229a19ff1c4ea33822a58a4
F20110114_AABEXJ jeffery_c_Page_067.tif
b446c31cac687331dd776686338ce6dc
c8c4c5615c85014c4513ec20aec5615d84a58bfb
F20110114_AABEWW jeffery_c_Page_053.tif
c30b019c358531e105b29fec8fac8665
28de8725008ce663f79f96cb16b5b5ec5d7e5738
F20110114_AABFDF jeffery_c_Page_036.txt
459c4bdf76695c2bb2354185626b47eb
74fc30d8d28932c3c402446c4b76215296b1c56b
1732 F20110114_AABFCQ jeffery_c_Page_020.txt
fc35da1c465fa4a81bccaa36f7ed1ef3
4343dfbcefbe8f346a576526c181fd623650a7af
F20110114_AABEXK jeffery_c_Page_068.tif
6732cb3a007c401f93ceb3e1f034654a
3dd3e5cdb04e1848fb13f35877e03d3e24850062
F20110114_AABEWX jeffery_c_Page_054.tif
33d788d608208ad3292c7c70f7843baf
9466b080aa55e2217b0667bd7a702cea815dfffe
1908 F20110114_AABFDG jeffery_c_Page_038.txt
c828546f4acdb2acb5a91907522e3aa4
1d621f80c165f2d6d2bfb5a4b91b8e07a8af0ea7
965 F20110114_AABFCR jeffery_c_Page_021.txt
c53d825999b25c6f8d592c2ef6112e48
bb23538a46fb4581fb1301a500373b867c49d28a
F20110114_AABEXL jeffery_c_Page_070.tif
69a0c454c6057c1befca01ef2ceba792
d85c9020b746bd04c9b411d9fe90d620447d2651
F20110114_AABEWY jeffery_c_Page_055.tif
26a42f1174ff144c8c1e2e0f157d1db3
bd3414119cb5b4857f434930b571ceeb598d436c
588 F20110114_AABFDH jeffery_c_Page_039.txt
89ee02181346c8c436425ef67279de61
1d0be7316db4b28e9219fc008464bb92f0be9758
F20110114_AABEYA jeffery_c_Page_088.tif
b33de2372e371c3f33b163e6168147c5
a6fcc8829dfa622278ab7730971675ce9a069ed4
2032 F20110114_AABFCS jeffery_c_Page_022.txt
5d4ac9dfa92d01af4d6cc177a2d739af
044b8f715a7ed7c13402859542f7c1bc0f864254
F20110114_AABEXM jeffery_c_Page_071.tif
859f6b74fa1b6c9875fabbbd8e53f17b
82a2d543010c5e2481db0f70ef55a2c75661bc31
F20110114_AABEWZ jeffery_c_Page_056.tif
ddca466cb883980010ffeb11c83de686
da46e6be843a12be482a885110bd9a7d5e085c29
F20110114_AABEYB jeffery_c_Page_089.tif
ed2b8400ab5e79a127254f711eb585d6
7c78b2e23950ea67a2a60bcf8345a7c1a1408b73
2086 F20110114_AABFCT jeffery_c_Page_023.txt
d3a1a29da074093e58da0506b4d8adbd
66c2e59788b9c2f8529995e3296f4f7d06ba0004
F20110114_AABEXN jeffery_c_Page_072.tif
d2895378128b72fd322fa96eb7c6cb89
294910a95bb70dd7ae3a9fe33cc703b4d2447e5a
1916 F20110114_AABFDI jeffery_c_Page_041.txt
d8cc91cb47235ba1ec325f911d651e94
eb0e3a76b9cec51a00afedd36e6c6108f02b0b6a
F20110114_AABEYC jeffery_c_Page_090.tif
e9baf76a1390c46d115729abcdab7e02
87b905f77b8bc46232a745f70e924aa67ec4eb8b
2011 F20110114_AABFCU jeffery_c_Page_024.txt
c1ffe91c3b2441736296a23878de1cd0
7dc28c9b3c6ea30a5eb332da7d448b4cf701cd1b
F20110114_AABEXO jeffery_c_Page_073.tif
c2b81bd3d77f316c0e3eb78a1e63fccd
5be842883b729fb73f28b686c32ab9b78f4349fa
1623 F20110114_AABFDJ jeffery_c_Page_043.txt
60e500c08e5c73916024d2407b538457
b7ab564c26185ab586f67d538d508f926922b8ea
F20110114_AABEYD jeffery_c_Page_091.tif
d8dc08c784755072fcd1e30be603e27c
0ffeb41be5acff864e65adf203abed078e78f6b3
1782 F20110114_AABFCV jeffery_c_Page_025.txt
526856a9ce962ad338608955e085ceca
8584933f8f9783d7ce89fb8002bd6c890ecdb913
F20110114_AABEXP jeffery_c_Page_074.tif
e8408e45d19c3ab049d7cff2ae0dbf18
d037be9fef54a78e93438ee37f69a735a4b8dc08
1803 F20110114_AABFDK jeffery_c_Page_044.txt
694f8d0e38fefe340ce7fcf4fb6f87dd
0e24506a704c31d14f7576950959b74027086eb1
F20110114_AABEYE jeffery_c_Page_092.tif
52b9a1506aff8ebb1e0c195021da01fd
a815d700e1bd9c3b2ab0558d29f589f87bb8b49e
1892 F20110114_AABFCW jeffery_c_Page_026.txt
d3d0bbbd0f968d41771fe0ab62e64b1a
356aff52a25028b8d7341b77c67dcc5be81ea2f2
F20110114_AABEXQ jeffery_c_Page_075.tif
4478438a3e6523bbd5637f0e334cbd2a
e58a21867063e0066a1966cff6cbf591ffeb8a1d
2066 F20110114_AABFDL jeffery_c_Page_045.txt
fef22af33c20cd9f62178466e237097d
5c7f817947d8f714742718d90d00510fb027ef49
F20110114_AABEYF jeffery_c_Page_093.tif
5da2ef3be5f234d56346ead64d0e8413
f5f8703ef032e29bf51c3cd0b39b96654f9ae647
1871 F20110114_AABFCX jeffery_c_Page_027.txt
ddce49c6b1fc2305dd6795f4cf57cead
889f6596d3e757b6a40dfbb4c78614802b53fd4c
F20110114_AABEXR jeffery_c_Page_076.tif
22e56250196129dc0a1fb90aac422f29
047ec16d8da627ae8f3bdd9e2e57f8bea54ecbde
1376 F20110114_AABFEA jeffery_c_Page_062.txt
f7699df70b28cc92323dd66f0e6aca3e
d27325a413fa9e352ff84c9dd09f12fd76859932
1084 F20110114_AABFDM jeffery_c_Page_046.txt
cef2e125456d937cc86ca83bfee225b7
fe58a7b28fd58f7cb6ff7ed4a172cbddb02d3f08
F20110114_AABEYG jeffery_c_Page_095.tif
9a887f559132c85a2d089b5fd75e56f8
82fbf9b0611aab37cd7b387cdbb5e95c2f119492
1917 F20110114_AABFCY jeffery_c_Page_028.txt
0c618295b58eb5717702868361c9aa32
0bc072ec0f2274234d1b2562ebade278ef5a3ae5
F20110114_AABEXS jeffery_c_Page_078.tif
ff5a2652ce9dcc59230845c10ea4141c
886c606dd4ac4148c38838cd9137d8adad85abbc
541 F20110114_AABFEB jeffery_c_Page_063.txt
53704ba711c2c9270efba9651688e2f8
fb524b8df032779861554b24456f6cb456d8e596
1787 F20110114_AABFDN jeffery_c_Page_047.txt
528fc4a2928fb482bcf5fd0fa09baf1e
adc673b08a242a4a6c6605c0bda7cccc5b7c208b
F20110114_AABEYH jeffery_c_Page_096.tif
ef391d7ee5ed255ef93c175e6c83c230
73bbbce21134c9d3127ca7fd33130f0b6cb8d360
444 F20110114_AABFCZ jeffery_c_Page_029.txt
671f8b9d79b7670ef88fa1099135e4c9
6a4174d35617cf5d3550f960b7c7345cfbf3b118
F20110114_AABEXT jeffery_c_Page_080.tif
a04e8073324850dd85ee2d52c7efb8e9
6630ee95b4d66a723ef99fcf0ea76dfd84d6cdc9
1655 F20110114_AABFEC jeffery_c_Page_064.txt
041695a027ec00762ad2fc4efcc62ab9
a9bac8b0a4a1d2d820e614e8d8bb77df4fe31775
1189 F20110114_AABFDO jeffery_c_Page_049.txt
dd62abf6c2b288ee81689f7ab62fd046
d3898dc0fded79de4397aec8e4ba6ed84434a7c3
F20110114_AABEYI jeffery_c_Page_097.tif
041d37ec8e61a337a41a0d35521e0093
046c756a0c13dd8e3436aea3b50164f30695634b
F20110114_AABEXU jeffery_c_Page_081.tif
8e2b494f9811a32a481816916912d54b
dc6d1ca8ec3ea72fe830599d69930f5b55671a9f
1970 F20110114_AABFED jeffery_c_Page_065.txt
690f2417fc7f470618f256362f90d39f
e3372a4802aad40b7315ddcc7b7ce7f5f79bdde0
2563 F20110114_AABFDP jeffery_c_Page_050.txt
a32b31f4e901d569bbbed0e237de2849
395a14adf9da249e0909de49f6d648820beefdae
F20110114_AABEYJ jeffery_c_Page_098.tif
25396f3d3bc218d6a6a0b5e845ac69d7
d4e17c82b597d6898386bd661bf0d8826e405c1c
F20110114_AABEXV jeffery_c_Page_082.tif
b35f21c932486cb66a4ce1b4b202dff4
45239935857fa5ab112e7255de708d8a9421e364
1410 F20110114_AABFEE jeffery_c_Page_066.txt
93ccdb05deb7c95c1f6c73d54234ebe8
5081eae77ff4823c9305a803685f06d45f635acc
1830 F20110114_AABFDQ jeffery_c_Page_051.txt
1a69d52f14118ead252933ec76fa1d4f
860a6aba699508e136e9b474c8b1fbbc1788137b
F20110114_AABEYK jeffery_c_Page_099.tif
c4f36b33e08d62e5d098dcf4b66135f9
0ef9a9ddc18084c49ca6e3284580d5030cf4706a
F20110114_AABEXW jeffery_c_Page_083.tif
bee0b497cdc314323a54dbd5c6abc1b3
d73d7635ed24b9cd6869d734aad41dd68d215101
133 F20110114_AABFEF jeffery_c_Page_067.txt
c3b9295ac9f8c60d691885aafc609922
c946bba7c166042436254529be6548ad7f711c0a
1869 F20110114_AABFDR jeffery_c_Page_052.txt
961206223f3241b88f85d75088e73982
fb27499d0769a8fd5596cfa6df97e33e930bb6f2
F20110114_AABEYL jeffery_c_Page_100.tif
3cabd5cb9a9ab4d2b0d4f200073ee74e
6eade5c2f764656932b93063693efa7b26b4fbfd
F20110114_AABEXX jeffery_c_Page_085.tif
7916a4d774a3f9cc4b068d72fc1f2548
162f6df40b7c787e798a805a0a49180433f74e83
1743 F20110114_AABFEG jeffery_c_Page_068.txt
8d61eb913726f0baceb2f4724d10d9ba
5d7396ee3d664f2be401554b95409064c9dbefce
23749 F20110114_AABEZA jeffery_c_Page_011.pro
8070b9ea3434729f6065eb145638167b
b1b4f3041924d0ca1c3bb367693d0a083ce355c2
1756 F20110114_AABFDS jeffery_c_Page_053.txt
cb0d2e5072e03dc7335bd84be2deb20e
ef9e07854fa86219f3e5b0d4177f4befd609cb51
F20110114_AABEYM jeffery_c_Page_101.tif
3cd394e1ac2babfbde2e7bda56673355
6ea6a3a0a40c6cbd1b05671f6da70b5ab02c4734
F20110114_AABEXY jeffery_c_Page_086.tif
6fef8971912344cf8d42c15c28be2043
30d79fa70a8d613ba9a3207912e19dc1533ff571
2459 F20110114_AABFEH jeffery_c_Page_069.txt
2e61fb42adb60b34331a27c992488168
ed13654c494abc191560838bcf66ff704c859e3b
43922 F20110114_AABEZB jeffery_c_Page_012.pro
0ffd2f59eb2c9ec12923464ffaa223da
427a0b1deaecc9e2443373cf46e3c829901a085a
2265 F20110114_AABFDT jeffery_c_Page_054.txt
d58bc07e710acdad93b353ed6d9092f2
36326d131520e25cd132d66e85adcbafdd512f9b
F20110114_AABEYN jeffery_c_Page_102.tif
c2e9177569b0ad45ce36c2eb7082f8c4
e56e76631fe5e8d3b1cdef726e476ae8058fafc1
F20110114_AABEXZ jeffery_c_Page_087.tif
8f70739f5b98b660a3dbea2acc923da5
a30c8ee403f90d332d5189f7279d175e2319a06f
2436 F20110114_AABFEI jeffery_c_Page_070.txt
aed9d76b16ea73fdbe73caf799a20ced
d616340f1fd37a695a1a8684ab8d87e0d23e6f94
30736 F20110114_AABEZC jeffery_c_Page_014.pro
60a9d6096dfb5fb852aef5729e883b5d
bb366619ad0509bbaf8732d866188ae94c7ec7f1
2211 F20110114_AABFDU jeffery_c_Page_055.txt
ea7ff4233d12aa73680ac4cec3ae06b9
af214a8324e07cf43cedecd4428c714eed6417b6
F20110114_AABEYO jeffery_c_Page_103.tif
1dddeaca93afe2f06bf054e5547728d2
7aab11922965b2d5350a8b5b646f9d625effeecf
42429 F20110114_AABEZD jeffery_c_Page_015.pro
6838642d5d54d35b0ac6e38271be0597
509580bcd00b5ef0632251673d99fefbf015241f
2075 F20110114_AABFDV jeffery_c_Page_056.txt
97ce65b8d10c3c50df3c2c871bc4d482
3f1a8d87866f738b0996debae6e2e8aa40893ebd
F20110114_AABEYP jeffery_c_Page_104.tif
95b8337707109c3d56f328bc75862959
c4c83ed1af7f75d6be0ecc3c07d6e17c31027879
2659 F20110114_AABFEJ jeffery_c_Page_073.txt
089433a966cd15b1921f57f77fae3b59
a429bae0813571d8a9eb1cecdc98bc31b6009580
40991 F20110114_AABEZE jeffery_c_Page_017.pro
f52a1fe0bc8bd9658f2a8fe96d3fd051
90a6ae3931ac0922a49dd9b34589a98a89fe47ab
F20110114_AABFDW jeffery_c_Page_057.txt
6548c9ec5c353bfef1d3951d7c659e34
e12c41a58b1e0f3a67ac7f8ba8c7de703215668c
9259 F20110114_AABEYQ jeffery_c_Page_001.pro
97e333c6d8ca3722337ed0cac4869f70
c98c89f85b78268e230af202352b2039c2153c1f
2899 F20110114_AABFEK jeffery_c_Page_074.txt
1e16fc5d4fdda39a976ffc2ced1ec77f
68ba4aacff776f489a93a5171cbcde879d7f29c5
42840 F20110114_AABEZF jeffery_c_Page_018.pro
4c9cae0ab213fed2ed6eb71ef9ea8b34
45cd049cec614ece71b9c8e63aaa23b52516ca66
2034 F20110114_AABFDX jeffery_c_Page_058.txt
9fe79d6f88a80ae9e1e03eabea72f2bb
5d018f64f367c5e1071da9c049d87359c693df6b
1201 F20110114_AABEYR jeffery_c_Page_002.pro
720113dc97cb48f342f88c12706e4349
afa7f4235626b44995f2ba31df66c090d33eae93
1568 F20110114_AABFFA jeffery_c_Page_092.txt
b934337780b3a1838af73e2d1115c0b4
86abb3294ab8c18a104e1d37e6b1d314a52ec927
2879 F20110114_AABFEL jeffery_c_Page_075.txt
320081e9d628ee9c20de0ec5bd2fe433
badb93378823039e30e641e224d722e09ce389f8
F20110114_AABEZG jeffery_c_Page_020.pro
3aacaac8d03403ec93a45d6e9d3f09c7
56272f54d1d9f81da0d6821342fd8bced3b40b56
1933 F20110114_AABFDY jeffery_c_Page_059.txt
e82a9d89ba158269237da1297a7ee5db
1e03b640d2117dc399b66e487910b894b8e4d625
1587 F20110114_AABEYS jeffery_c_Page_003.pro
892b60cfb6b1ad5d285cfeb0f3cbaba4
68238a75eae39a3b0a2d32a5f8174ea2aff462f8
1195 F20110114_AABFFB jeffery_c_Page_093.txt
750b4dd2cf7bea82f026006abe5c24d5
c820fe0d12f2bb3971dd733406da46efc608bd58
2085 F20110114_AABFEM jeffery_c_Page_076.txt
3472b7184993fae0acf8747f3f83827d
385a391bb86052a48388a2a8e69c17100b0f8fba
22892 F20110114_AABEZH jeffery_c_Page_021.pro
c8c052ef827c42fcdbecb67c00fe693d
47922c391ae2a06141597746aa7b6f42882de0cb
1679 F20110114_AABFDZ jeffery_c_Page_061.txt
b625ed87d4d0a7cc310397963f4397e1
b8ab1386729222028f596dbac9b4401ab05601ce
12077 F20110114_AABEYT jeffery_c_Page_004.pro
17c0c2166492b411033af6c990360489
734005dd875d64bc7114474392f42e9b8af7b8a4
1776 F20110114_AABFFC jeffery_c_Page_094.txt
ad6e34a6ea08548533c3a2809dcd9756
09d63293e1ef5f4a85f31d059093c071f89b1acf
1433 F20110114_AABFEN jeffery_c_Page_077.txt
7aceeec9a85bcf6390be6a85e443719d
bf4cfc4e8ce3ebb512b9da33507031403a8e137e
51273 F20110114_AABEZI jeffery_c_Page_022.pro
770bbe1e8665231c904701f227a37977
4e0c9214093750c35b98fe7d8beaa517404ef424
74262 F20110114_AABEYU jeffery_c_Page_005.pro
b9f85e309fd70108530996c5dad8aff2
40a365e9ba17ce5cad8261f063847dcba19ef237
1408 F20110114_AABFFD jeffery_c_Page_095.txt
c7a422cc657165de92a55a05591ba5f0
b9efd9051997c88613d772fbab1f09ae2d5a87bc
2174 F20110114_AABFEO jeffery_c_Page_078.txt
8364adb1163e67c66e4ca03c3c1a9c90
d7cb9c3aa303e8adbff01f83601638de06761d70
94486 F20110114_AABEYV jeffery_c_Page_006.pro
277a732f4c6d1c52b2d686d488e4e420
f657f472572b11fc4b3e306ed6262f990c17380f
981 F20110114_AABFFE jeffery_c_Page_097.txt
43db419134be84332f5eef0eafa0580f
a240aabaf174597e3417871513700afb89d4f1a0
1935 F20110114_AABFEP jeffery_c_Page_079.txt
6ebd1245b2c37df49b21993cfa491192
d360ed2c450bbe9bc271cb5a0902e570cdcb4c8c
50921 F20110114_AABEZJ jeffery_c_Page_024.pro
8ac90e8915fb6757cf68f696e0ea0ff5
a7d780d449c61f4e910a112631e5ccf0c3e47663
6924 F20110114_AABEYW jeffery_c_Page_007.pro
fede2dbc61fc4dba80a0351ec8ff10cf
357e4a8e8c4f4326f4d343afdf2122db3c615cb0
765 F20110114_AABFFF jeffery_c_Page_098.txt
3eb6465ae5c28c1a65407076360b3891
a8da0b5133013a123c4302c208f1865056fa4373
2140 F20110114_AABFEQ jeffery_c_Page_080.txt
6c6a0dc5c11b994e7ba7adfb182e9358
0049207b9cd08ab1c7a20ce86895b826204c36f1
47741 F20110114_AABEZK jeffery_c_Page_026.pro
12a2c78376c3f3045e543f6439f18192
5178f4466e601df38569dbedd4a51e54686bdf2f
18114 F20110114_AABEYX jeffery_c_Page_008.pro
31a918f4086f2c8e49047a8a1d7abb18
5c3601dddd36607f80d06a374ffca5cfc0a27a8f
2714 F20110114_AABFFG jeffery_c_Page_100.txt
158c03a2c652859c49dc785506a57473
21e3cd8090d7e7b896497d0dffd0b129faf429e0
2025 F20110114_AABFER jeffery_c_Page_081.txt
99ba2c30ebe408832f2a514bb683e3fa
044cfbe698d6764e7f79043ed13ca53bdda99181
47117 F20110114_AABEZL jeffery_c_Page_028.pro
f0e0e713ca6088d1d96c577afcda9eda
0f7b251cac3e76453b9718ac3d5b3c888d735d98
43728 F20110114_AABEYY jeffery_c_Page_009.pro
abdc8df78fd653b1d7d57523ad0b0da3
c11e13b41a23590138864ec7acc4305fe3e99c6d
2603 F20110114_AABFFH jeffery_c_Page_101.txt
27ab6f8e353d872a8163f3323f5cd610
5fa0c7752e17ce6ec6d95e96c311b7d98cbd70f9
1834 F20110114_AABFES jeffery_c_Page_082.txt
4efb97594e0fe7e9e8f80d0c75e0ef4a
f13b334eb2bf1834dc85ee36a816dfd3c8138cc5
8920 F20110114_AABEZM jeffery_c_Page_029.pro
a4d0549749f78372ce7baa6abda7cb74
a7d6f47593a1ddfa5d84672f22fd601373c525c5
36151 F20110114_AABEYZ jeffery_c_Page_010.pro
b88ad24b95b67dbd19eddb780c7856f9
6f9367aac48dec751ad47decafdc0f032b6d563c
2509 F20110114_AABFFI jeffery_c_Page_102.txt
4e941394373974a65018b67ac900abb2
2e4340244839d8f993974cc965aef8633fa17321
1759 F20110114_AABFET jeffery_c_Page_084.txt
8dc41392549f3a70e5594f8490cee849
b6924cf5fdc7ba4c4217636cd9d1c4f2ce574193
36554 F20110114_AABEZN jeffery_c_Page_030.pro
de614442f88b7218e1b5631dc415c936
56528f93188eb1988011021b540ac1881a2aaf2a
1135 F20110114_AABFFJ jeffery_c_Page_103.txt
35f3f626ef57b955433d63c61828dcfd
a93e77fdd88eb2bc9da69acdfa314a629c0d762b
F20110114_AABFEU jeffery_c_Page_085.txt
94cf2a5fa58014dbbbf62bcb2b69771c
908fa29ceda124df79cd41cd32bbda79fff0db73
49770 F20110114_AABEZO jeffery_c_Page_031.pro
a6b6e06e8a6f2baf660e7e8badab2257
d8cc32fd5b7493e7dfa344e8bf924b06c94911ae
1914 F20110114_AABFEV jeffery_c_Page_086.txt
e6368da97e9adfc512026f13372e0ebd
3141a1b253bda4540321851bda1a644c22299361
48125 F20110114_AABEZP jeffery_c_Page_032.pro
3a80df51d8d3b28a716812a3a21126d3
2d12d0f5291a708fd8b7cd0e209a5d01bdd799bd
989 F20110114_AABFFK jeffery_c_Page_104.txt
6119a141f8c076ec76f013df00796c0b
acbfb2237c817f064311efe54c48d93eefb66374
2485 F20110114_AABFEW jeffery_c_Page_088.txt
48a36c32a8a73b016d5103e061a779be
bc56df2d5d0368c17323a74c2b592241c2c048df
42480 F20110114_AABEZQ jeffery_c_Page_034.pro
60b50dedb99ee2ce575d792375474451
9b6c0fd9aacdde8caf53005ea2bc6adcc3d413d7
8209 F20110114_AABFGA jeffery_c_Page_008.QC.jpg
33a2b332956d1028ce0c72e27d984925
84c3df0d184a59e25515c40f470f68b561daa90e
2471 F20110114_AABFFL jeffery_c_Page_001thm.jpg
121ee8a8f0c910398d06dfa5a836d4e5
936a54fd0e0cbf3e3dfde3dd71b2821f053e5b63
1936 F20110114_AABFEX jeffery_c_Page_089.txt
80e19457f727b0f8ca9cdf386d8e8ea8
5c4e00792b8a9e0a4b69aa493bdcdcabbb7b7256
47392 F20110114_AABEZR jeffery_c_Page_035.pro
83a6c78c6438c9a8e3494b1a3032d6e1
2a6a27a75cce57724e12763d3bb43d58038ccb69
2716 F20110114_AABFGB jeffery_c_Page_008thm.jpg
844111285fb3a3f4910889f83726e398
c8e035f66841de832002270fd694804e71b81472
2937772 F20110114_AABFFM jeffery_c.pdf
f5b385dd4246e1e49dadce4173a12710
789dfc3bc93081adc41ded39e492420cad3d0b7b
945 F20110114_AABFEY jeffery_c_Page_090.txt
1b41dcda7d19b19f9f47013598785118
b04bad7e5b73c566a2febc1c6dcbadd92585114f
49881 F20110114_AABEZS jeffery_c_Page_036.pro
98dcb3be0b7eccccc928ff8f1d48e522
7bbd8f5ea1f51bfbfba2ab23dd26b22d265141f4
15352 F20110114_AABFGC jeffery_c_Page_009.QC.jpg
eb6b27cabfbe7e7d962bed8bdde935fb
92b5e492d70891b35368b29d1606d013242f3969
7943 F20110114_AABFFN jeffery_c_Page_001.QC.jpg
4800c2090425f0b8d86f12b01371cb9c
2d575e5dceadb2e4ad67fd9027672a425ad239d5
2281 F20110114_AABFEZ jeffery_c_Page_091.txt
d34928656a67814a9ce4c282dde2b6c8
458555251ac22b2d8fa547614f2d59efafe7c4ef
38689 F20110114_AABEZT jeffery_c_Page_037.pro
6e4dad54cebe554c0077e1512d632f7e
d09c908aa6b3683129569bbaee311be9ae7a26a7
4432 F20110114_AABFGD jeffery_c_Page_009thm.jpg
dd2f3ef1e2996be587b71db7dc2c4e11
87a80fb727790038bd810845e565891450078810
3306 F20110114_AABFFO jeffery_c_Page_002.QC.jpg
7b119bca386feffcbbd7a336a3f61302
b51f61af13937323eb546e7e9b3db282a3addc03
48193 F20110114_AABEZU jeffery_c_Page_038.pro
af806c379b52980f710d33659828145f
a91aa8921119cb82e9743be6a095f216b148e650
5091 F20110114_AABFGE jeffery_c_Page_010thm.jpg
536e0b03dcdab3103e7c75610e3765e1
32c52f94b0f2ac90a277c070a3ad8db236d7b78f
1387 F20110114_AABFFP jeffery_c_Page_002thm.jpg
0378bdd8564c398179a546044baeb648
9cce88a891d03e9d18468a49f25d4893d88f17ea
13604 F20110114_AABEZV jeffery_c_Page_039.pro
30cdac7e6a37793363ff9d9b7e07556d
ae125bc858adbdfb9b5ce66dbc0d3878d7060aee
13087 F20110114_AABFGF jeffery_c_Page_011.QC.jpg
29342a99e68a3620b223beca9d5dc977
68d50b677f11ec1847f6fa03d0b7a5c4bad9e68c
3281 F20110114_AABFFQ jeffery_c_Page_003.QC.jpg
ec7b2dbdaf9d58e7e9b4d2fab018dd16
bad6c0b83abc2341309fcc17b6612a7462db5f1d
43397 F20110114_AABEZW jeffery_c_Page_040.pro
4468270e7c2bd4c5756db5b03cd8b897
6fd28668c7edb1a9e0a485cfab2dd29b8e252526
20460 F20110114_AABFGG jeffery_c_Page_012.QC.jpg
e2e0293ec22a0598087ce87370043e3c
d2495c25bf3187875371e767c58a339adeee3696
1372 F20110114_AABFFR jeffery_c_Page_003thm.jpg
36b9c870d66495e61b3084b814216275
000b7ab6afb9d24e597c87bba8516cf871a748ac
46869 F20110114_AABEZX jeffery_c_Page_041.pro
ef676eaf8fbe83b7650e6b2d16c587d2
c29481958d7e4c15651acce6850f5aa54bcc0180
8141 F20110114_AABFFS jeffery_c_Page_004.QC.jpg
c474c40a84ab3cb8eafa42a26ce92d54
a30e8ce6eb854a6103a05f0ea2fa3b0c3c4f17d4
18475 F20110114_AABEZY jeffery_c_Page_042.pro
6bc6d228c199ca6c11167f25a45c1703
01223b06c006cfd4287a9bfebaf515c6e3ba15ba
5911 F20110114_AABFGH jeffery_c_Page_012thm.jpg
b0f089db25a310061747cf528dae3f27
7c9bf4be7097fb08d054cf9902d5b271debaaf74
2582 F20110114_AABFFT jeffery_c_Page_004thm.jpg
11b7268baedc4d63749da2d27e42600e
beff6970d15b0769fd64c50bdcb96218effbafe6
F20110114_AABEZZ jeffery_c_Page_044.pro
c425d42e7b29198256409751b2291ebe
c0ef19f414f67cb909c8549fb25b3d63dac700b0
22764 F20110114_AABFGI jeffery_c_Page_013.QC.jpg
8581dc18f6f146b31ee3ed72465f0377
16e6e96322f7775c9dcffe850ccef4784398cab7
16843 F20110114_AABFFU jeffery_c_Page_005.QC.jpg
a83b66054da31fafbe9038a578552033
e906ef6cbeb23b78c6f1bb9da815cac73035b9c5
15699 F20110114_AABFGJ jeffery_c_Page_014.QC.jpg
6708abb7f6f9f065da2e1073f155e97c
402ae82cd75f4cca2c609225022458f5ae8b89b1
4462 F20110114_AABFFV jeffery_c_Page_005thm.jpg
3f36258289bc330a3d47b631dad4cf76
5918c28f073af9ce8e3c5708bd4afe857e8664c5
19643 F20110114_AABFGK jeffery_c_Page_015.QC.jpg
e024bf0b576cf6481f7d64f504fe0642
e724c1c657242408b1698c2875608f6479532b27
22891 F20110114_AABFFW jeffery_c_Page_006.QC.jpg
7a953f7bfc6de43d50e7b8d9374810ad
6204e529d35d7073f11d108767656721a2b23ce2
5565 F20110114_AABFFX jeffery_c_Page_006thm.jpg
375e92d9eef05fa9ead7694e88a98eac
7221ffdb26acf30f3a884c7a6867afa974b13cc3
22590 F20110114_AABFHA jeffery_c_Page_023.QC.jpg
a45bd362d969954b49bdf6852156538b
4575343e24aa061f0a35346e6da62167a2ded800
5499 F20110114_AABFGL jeffery_c_Page_015thm.jpg
e04cd92f71e48aad1c8638c263cb3c4c
7aeda86242b7dd4e677c0c9601dcb5280f3faa40
3933 F20110114_AABFFY jeffery_c_Page_007.QC.jpg
0046e8fd0689c07edd69c93413fcc757
c6095cf63f9e9059594da8f6346e1f8fda8c1956
23554 F20110114_AABFHB jeffery_c_Page_024.QC.jpg
887434d24c157bc57a80054f3c63dc04
1bb1215de1285a2be975ba0e2da3ed826df36c12
18965 F20110114_AABFGM jeffery_c_Page_016.QC.jpg
f752dc11173d2bc082886902c08bf020
3688ee0310cb10c4cc7c7d18ce7f49c81c935400
1578 F20110114_AABFFZ jeffery_c_Page_007thm.jpg
009f94b8918a1addabdd02855d7717ba
316b76024c37327a95426b34b2f8714d4fcd22bb
6572 F20110114_AABFHC jeffery_c_Page_024thm.jpg
a1bedffd49a1298bf039a1534672f75f
98df6b1e6f9a70e812d43e7be9a559f141b77caa
5395 F20110114_AABFGN jeffery_c_Page_016thm.jpg
21dd97d92cd8467b153c3faf0177985c
29a316fb62496760ae356cea4f75fe501226e7ce
21484 F20110114_AABFHD jeffery_c_Page_025.QC.jpg
f03e8ffd912e15ec0ee7347473ebe964
75e9abd112e6473ecf9f2c17fecb388a3cc9b828
22085 F20110114_AABFGO jeffery_c_Page_017.QC.jpg
c4d8f8df3818ed26192df4d4300f4f93
83a5100be6b222913e6b88ef022513db424a095b
6090 F20110114_AABFHE jeffery_c_Page_025thm.jpg
23ea660fa22ddaacf385da9dc78ad525
ccb865011014834785556b92729edfa70fcf76ae
6450 F20110114_AABFGP jeffery_c_Page_017thm.jpg
de2f87d53a60a1aca8070533db932167
b91014ef6e7b9be731864df0cee114c6796ae34b
22338 F20110114_AABFHF jeffery_c_Page_026.QC.jpg
cab5666b5ccf4dda00b1314c52d380f9
015542c4e4ef296a03085774b75baeddf0371ff8
23056 F20110114_AABFGQ jeffery_c_Page_018.QC.jpg
a2105327ac6990f16ae5fc20e0c5e6e3
19f158e57706359c7ff7d2cf5dd02d3dc1e9380f
6398 F20110114_AABFHG jeffery_c_Page_026thm.jpg
0050636ffdd2270bdc7d605b51f0a73c
3d27ff0709acef357027b3b150f1fe06d093b0bc
6586 F20110114_AABFGR jeffery_c_Page_018thm.jpg
bc391fc3a4e00454edfee173efe4f4a8
b92026c726704923bcb51d7b17a820eda87a22ee
21570 F20110114_AABFHH jeffery_c_Page_027.QC.jpg
01551de578384e355574506b26eda38c
b43c89252bb05faa82e19dbc61cb8eba96db51c3
22064 F20110114_AABFGS jeffery_c_Page_019.QC.jpg
f2269b1391a8250c5d144cf15f1e8f73
985c6b4e2f67c19dd32adcb758a3bb4cf7116d4d
6186 F20110114_AABFHI jeffery_c_Page_027thm.jpg
12345363c922bc8162415574e1559768
2cd6df1d261045747fc778704d7940a7726f3e43
6314 F20110114_AABFGT jeffery_c_Page_019thm.jpg
b8895ca23abe74fbe051768e2c3f1c72
fb9775b9836936acf792974d93cd66559ac8ee05
21943 F20110114_AABFHJ jeffery_c_Page_028.QC.jpg
fc0f0c61022321466d2722810b009815
89468f973abe613945ad4a7c36d0879be485663a
17757 F20110114_AABFGU jeffery_c_Page_020.QC.jpg
3abaf11e9905185e87eee362dde6baa1
4606fe75774f21136a39bf721cfbdd0367abc513
6268 F20110114_AABFHK jeffery_c_Page_028thm.jpg
a0960226375b216c9c7c8071f12db13f
9d820e745919fdbcad83d3efa4161133b8a0528c
5266 F20110114_AABFGV jeffery_c_Page_020thm.jpg
cbf86f2bf7172a354bf6fd0cb50abf84
fc8403c5c627e5704f92883bedbd2ad0e628d6a0
6718 F20110114_AABFHL jeffery_c_Page_029.QC.jpg
79ed6759d0e2c6494e025968ce99cb76
6985182bee684252a0771b93b1cda7a569780bb9
14870 F20110114_AABFGW jeffery_c_Page_021.QC.jpg
306dd6090b11f589b390ffc251e07289
13493cdb8d89602b4b37f0336a4f6967559369a1
24304 F20110114_AABFIA jeffery_c_Page_037.QC.jpg
f30b4f5204a7e96c4b655ecd234e5cd6
3c3e59ac8482532359b15362e6cfde0b9e5f6d34
4546 F20110114_AABFGX jeffery_c_Page_021thm.jpg
c5442450c08873cd3347ad564e0d110b
4b8504917b067f336a76c9cbc7b1aee85051cea7
6634 F20110114_AABFIB jeffery_c_Page_037thm.jpg
a52b21076265e1c37555e1df1766e17c
7bbd2f7a2d84bfd541f351815efa4b42abd16e61
2241 F20110114_AABFHM jeffery_c_Page_029thm.jpg
5a6803281dffe96187804c082ce31a0f
62fee6e965875e20ad5a3bc5dd2e7f1906766c5f
23409 F20110114_AABFGY jeffery_c_Page_022.QC.jpg
b00dd9fa81f6f60e99ab133b613a43e2
d63d84d315f3e503b556b905dfdd88a0f33e3d11
22310 F20110114_AABFIC jeffery_c_Page_038.QC.jpg
3869b509fafd25630e0837992c1c3fcc
064dac6b4482dc8d2735158e99b08243df400b46
18011 F20110114_AABFHN jeffery_c_Page_030.QC.jpg
3e8747879b95187c9ab5947c78c530e0
f96d3acb78c9d2726f4c002980eded91eb08684e
6487 F20110114_AABFGZ jeffery_c_Page_022thm.jpg
5aa7202f192a6c4a6b1056607b0f3c88
4bb717365513dc98627f984e17c43c42714a6223
8381 F20110114_AABFID jeffery_c_Page_039.QC.jpg
e0d0f6cb5c107a4cdd4548fb0e60a83e
e926585ba17aff49977cd3b04314d431c2b75623
5189 F20110114_AABFHO jeffery_c_Page_030thm.jpg
fc3a6e0ca94aad86654bce714cfac4aa
c1b9c92482b31ac28231c609078b669c9a76a47b
2594 F20110114_AABFIE jeffery_c_Page_039thm.jpg
330beba872790a87e33a9bc1cd73ed3d
6b2e8b00e6b01369bb64cab40dd3a129ba508d5c
23076 F20110114_AABFHP jeffery_c_Page_031.QC.jpg
ac34d6825ebdf8003f67b6f605480d15
fc9f95a20b11eff12023bab1212da547a2df16a6
5982 F20110114_AABFIF jeffery_c_Page_040thm.jpg
8de563c24db612de6687286abf6caf84
5d7dce1cb1888abfe62f67074832117f8ba0ee85
6371 F20110114_AABFHQ jeffery_c_Page_031thm.jpg
affafcfca4d20d4f82e40650b5d583d5
c561b62003a3b1db41b916c30a0ff0aa4629bfe5
21243 F20110114_AABFIG jeffery_c_Page_041.QC.jpg
a33c6060307832d15372edd6a83145ac
1fc0bf474ae3a4d57a1dc95d46210b73256186e4
22971 F20110114_AABFHR jeffery_c_Page_032.QC.jpg
e4da7cbf879169746243391fe5194e63
45df2c5bd54f7d2f3242e097bad879ae8aa2db65
6278 F20110114_AABFIH jeffery_c_Page_041thm.jpg
8b18e35af30abeee4172c63d42a5ac18
1efb5850db53d6a26911031db54750ed21b1f28b
6251 F20110114_AABFHS jeffery_c_Page_032thm.jpg
e2f766dd54ade2ff14a607711553d483
d332e919bc2b43db3581074e9b87e31087e23c66
6165 F20110114_AABFII jeffery_c_Page_042thm.jpg
00410353815db89f0a87c8512f99dc64
d0b49b4f0eddf8801de3bb2501c1da24e06cf01c
24130 F20110114_AABFHT jeffery_c_Page_033.QC.jpg
23602eeab73a9f8b9e7dd63f4220972e
9bcd8f135293de168e3330e07e62b6c36e1a512f
22402 F20110114_AABFIJ jeffery_c_Page_043.QC.jpg
771abfd8d2adf064b0b56d36dc3600e3
f19ee0a76534e23c4e6fd72cf2a257a27a48e6c5
6677 F20110114_AABFHU jeffery_c_Page_033thm.jpg
8d3c8b86a8d61d54f97706b18473cc0b
d8733c22229cddf731d9a3202ca42977544c6f5d
6241 F20110114_AABFIK jeffery_c_Page_043thm.jpg
2ddd50934dbc3c034557a7cc61e18afa
53c1b13d9bb0b6472ce856bce3156edc07b4b639
24542 F20110114_AABFHV jeffery_c_Page_034.QC.jpg
c6805be83a6b6f04a490c3d6a8f6fe88
5aca2e1b99c8d959073a663e91b630ebca5ad964
21560 F20110114_AABFIL jeffery_c_Page_044.QC.jpg
848990994ddf392f5c4bb5b81ea73dca
a57665c05dffbcf795c19a8e725d1bceca1208e6
6880 F20110114_AABFHW jeffery_c_Page_034thm.jpg
3278abf622a42151ec30ad46ed5eee4b
799a3557ed577a097b39639ba36352584c1085d1
6131 F20110114_AABFIM jeffery_c_Page_044thm.jpg
c42b8cfe1356929307442e08ed3c9669
f1d7211714911b29dea15bcffde7ae8168fc3e06
22517 F20110114_AABFHX jeffery_c_Page_035.QC.jpg
be350780624d11051e8f0071dc16c226
eb73a382b9c83856ca1dc0f10ebfb131cd0cf444
20015 F20110114_AABFJA jeffery_c_Page_053.QC.jpg
ca77fda162e1fd058a6c45334961fe9d
2477a6c69ff519be41669ff3d32f50848a398fe6
23149 F20110114_AABFHY jeffery_c_Page_036.QC.jpg
d6abacddf93ec03c5de4dd4a86952534
feace8efbbd32ba220ba7a95553d2f955266460f
5471 F20110114_AABFJB jeffery_c_Page_053thm.jpg
4f93e314c3fecd6bf2c218eab01b9c6a
a05ba0d6ebd60d797ce1ac951ed061a43398f382
24422 F20110114_AABFIN jeffery_c_Page_045.QC.jpg
388ba647943e195a11d6cb5f3d8a00fe
b8abc750c27f2ac635de923a914a3f35e020efe7
6480 F20110114_AABFHZ jeffery_c_Page_036thm.jpg
3b37eb3b1bba13ee69b73a084d697a6c
bdf7841f4e7110a36331aa4ebb15cd073c86e3fe
6486 F20110114_AABFJC jeffery_c_Page_054thm.jpg
5e928352b2becb264875769aaa167758
db219a39c8c17e995edb4d543cab99a9b048122e
6696 F20110114_AABFIO jeffery_c_Page_045thm.jpg
234266e5aefe4f3f333300612ee9eec5
3d0ba904433867f60ee83704d889b2a6fef14f93
20629 F20110114_AABFJD jeffery_c_Page_055.QC.jpg
17cd5925ed2b44e7984f617f2a1e3eac
0e70238212fc440beeca8943d055cb2a5f7e375e
4218 F20110114_AABFIP jeffery_c_Page_046thm.jpg
8d6e8974c28f3bc1d034c68273766485
5d926e9ccea2c04e6abb794f308c725f6cd7e669
5845 F20110114_AABFJE jeffery_c_Page_055thm.jpg
11da04bd8acadb1d5ae64240e963c08e
3d518cfd8e3a077fd6d17ab72935b3724d54ff38
20237 F20110114_AABFIQ jeffery_c_Page_047.QC.jpg
551cfd2b6eaf5c0e7fea996005955341
db05173dbb1a23ac9fea96fd6a3c5eb94f9f0b6f
23353 F20110114_AABFJF jeffery_c_Page_056.QC.jpg
4a38f72d5c44c83e67f12f1d19d9fe62
30fdeff0fddd81582684b6563ceb18d2c54765c9
5887 F20110114_AABFIR jeffery_c_Page_047thm.jpg
5ce81e659aefea7970f02291e9eef164
abaec5e2bdc0b13336d8908116401a5db39d286d
6471 F20110114_AABFJG jeffery_c_Page_056thm.jpg
612451136cd2004995e50ff16078406c
f9346a23079aeac0da96f70268861e17e7e2353e
23235 F20110114_AABFIS jeffery_c_Page_048.QC.jpg
85d88466535968305dc912c78bcdcea1
e48cb1c5d8bd481bbb0b13d097613651ba0e6a80
22355 F20110114_AABFJH jeffery_c_Page_057.QC.jpg
c2e63483a527e53f828b975f68d4477d
292bfc4d7a50cdbbac9c6498b2f7de59611a0423
F20110114_AABFIT jeffery_c_Page_048thm.jpg
dc55548918331a6f04e364e58854f7af
5e8aa8a408c6649fd62b25fdbe969bf2a70bb845
6185 F20110114_AABFJI jeffery_c_Page_057thm.jpg
1f834d9ae1636ac8372f8ea2c2e5164e
b91539b01a73ed3a9fc074842aa3d9ff232af3c2
14921 F20110114_AABFIU jeffery_c_Page_049.QC.jpg
c59471f689887d1b9c513406c5925610
a0a1935e92c3dc91d3106c332ebccd490f08a24b
23573 F20110114_AABFJJ jeffery_c_Page_058.QC.jpg
8adcbc1003cafac4e9b5c9bf40b8eadb
48199404ccc043ea7d8f18b24e6c83d722dddd24
6638 F20110114_AABFIV jeffery_c_Page_050thm.jpg
7bb2b4dca390ad6121a0e843e9d00f92
ddb31f63b8cf1427f1f7f6647758899c735ffc51
6527 F20110114_AABFJK jeffery_c_Page_058thm.jpg
f1b6b41a72bbe84467d69490d263faad
4b0e3a291f54cdfc510ba9f35eb4c1adb7bf1702
21450 F20110114_AABFIW jeffery_c_Page_051.QC.jpg
db109ffeb6c9de844d6d9c33ff00b4cd
f33f5f965fff9f48ace276bb922851a4174ea97d
22480 F20110114_AABFJL jeffery_c_Page_059.QC.jpg
19af650f9d3e9d3697f81bd2b74c75ff
1b25d799d656ab6eb74e3c5a77942c1917ff1d08
5994 F20110114_AABFIX jeffery_c_Page_051thm.jpg
33c235839f7a61568c826bd9f7812d65
055092eba6c7c4bd69714d6656cee7dff468918a
4073 F20110114_AABFKA jeffery_c_Page_068thm.jpg
411407053bb95206b09f0b16a1b0113f
e13570f97a9a5dc85b3779380e10dccec917c701
6157 F20110114_AABFJM jeffery_c_Page_059thm.jpg
a66976085b24493bc338f9b0cabfb0e4
6e6d5080b9ed6683f83de1d7adc00491461a7af7
23728 F20110114_AABFIY jeffery_c_Page_052.QC.jpg
14d33ab03436909b95fd80dced0734dd
944155e1136edca82b57672a959b0a0a490fab03
19504 F20110114_AABFKB jeffery_c_Page_069.QC.jpg
17944f795d6830d82917ce9fe99aa97c
13015f1deeb8388ddfe4c1719ff9b64460812a80
18579 F20110114_AABFJN jeffery_c_Page_060.QC.jpg
684ecf486416c5efd88e4caf415fb42b
df2aa6e6df824943929bbf837230c553ac0a23fa
F20110114_AABFIZ jeffery_c_Page_052thm.jpg
e2dcf59e947186e62aa8040fce8093b3
291db2185bdb52cbc84780ed93c26e1ed7e65d5f
4979 F20110114_AABFKC jeffery_c_Page_069thm.jpg
72d11a19d12046273df7fa3da92eba24
23333449c067feb292149d3962a7bb677c35e3c6
18078 F20110114_AABFKD jeffery_c_Page_070.QC.jpg
79b2f9ceca7f2231db1739662e5e552c
4cb73e625887b574037dfc1fe550768c4edd7776
5397 F20110114_AABFJO jeffery_c_Page_060thm.jpg
ead32c560b4fff0b032a739b5b6d3b94
9d0f5df75e42bfcbf0aa7f5f9c820705ffdf0c30
4808 F20110114_AABFKE jeffery_c_Page_070thm.jpg
450b881186484bf524f28292416f652d
3e492161f58474160a0e232509a2b51218181649
16859 F20110114_AABFJP jeffery_c_Page_062.QC.jpg
27400c9d22f8f01e97d15e79ffca6eaf
cb634ceb38c612ef6231cf27bfee5e7a8113911b
19875 F20110114_AABFKF jeffery_c_Page_071.QC.jpg
64069dbe9799a68e1496bebce8a22282
f70564397cc4d78ddf3aaf39890dd7edd63f7f26
4875 F20110114_AABFJQ jeffery_c_Page_062thm.jpg
503f102daa0afa57e02b7863ea836ecc
3593d5e1c8c25a62515d27c658ca345e9c6e1d3c
5069 F20110114_AABFKG jeffery_c_Page_071thm.jpg
4edad0c635ec34427f5162dd9e7e069d
48e6d7e86ff7380618053d7024a3836908de9437
9435 F20110114_AABFJR jeffery_c_Page_063.QC.jpg
40ccc59ae7dbde60e620eade97c128ac
e1c0ffa9cb7ff8b6e26c00b8edebc8b83430e81b
18306 F20110114_AABFKH jeffery_c_Page_072.QC.jpg
e0367fac6b78877790f029554187ae92
36e487b4dfdb4da1b21c415590cac6ee94aa6fa6
18572 F20110114_AABFJS jeffery_c_Page_064.QC.jpg
654a777174e5069a68e6978c8664f0a1
00ef3d5876ce9282f9cd40de52a8f4acfd92d0a4
4909 F20110114_AABFKI jeffery_c_Page_072thm.jpg
2465dcfe68ab003bbb7728145452095f
6ab33380e67df2b4937990843be3f1474b9e6bec
5332 F20110114_AABFJT jeffery_c_Page_064thm.jpg
17eeed4116b8ebe6d7fee2ce177a9e31
e432545e955e003e5b0236068ee8ce620bb951f5
18012 F20110114_AABFKJ jeffery_c_Page_073.QC.jpg
b47843c48e465d12d3c0c5b2e2812aea
350993961fe186013357a97da3cde32a20bcc427
23031 F20110114_AABFJU jeffery_c_Page_065.QC.jpg
b805fa79f5cae3aef795a215d6d6198a
69183502fef331e8b7448623ba69bb9896d0857e
4816 F20110114_AABFKK jeffery_c_Page_073thm.jpg
92f77b09b13f367d3707abba54f09975
9145ee5e6afc94bc8dd779e83b04b3c97da7b13d
17706 F20110114_AABFJV jeffery_c_Page_066.QC.jpg
1d60319785cb8a7853e43b3f714e002f
39ae591fe04243591a16e7286a3f50798c90c05b



PAGE 1

PERFORMANCE ANALYSIS OF DYNAMIC SPARING AND ERROR CORRECTION TECHNIQUES FOR FA ULT TOLERANCE IN NANOSCALE MEMORY STRUCTURES By CASEY MILES JEFFERY A THESIS PRESENTED TO THE GRADUATE SCHOOL OF THE UNIVERSITY OF FLOR IDA IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF SCIENCE UNIVERSITY OF FLORIDA 2004

PAGE 2

Copyright 2004 by Casey M. Jeffery

PAGE 3

This document is dedicated to my new wife, Mariah.

PAGE 4

ACKNOWLEDGMENTS It is essential that I thank my advisor, Dr. Renato Figueiredo. He has given a great deal of time and assistance in the culmination of this degree. His enthusiasm and guidance helped see me through the completion of this project. I would also like to thank my family and friends who have supported me in this endeavor. Completion of the research and the writing of this thesis would not have been possible without their encouragement. iv

PAGE 5

TABLE OF CONTENTS page ACKNOWLEDGMENTS .................................................................................................iv LIST OF TABLES ...........................................................................................................viii LIST OF FIGURES ...........................................................................................................ix ABSTRACT .........................................................................................................................x CHAPTER 1 INTRODUCTION........................................................................................................1 2 BACKGROUND..........................................................................................................4 2.1 Modern Memory Devices.......................................................................................5 2.1.1 Memory Architecture...................................................................................6 2.1.2 Memory Fabrication.....................................................................................7 2.2 Reliability...............................................................................................................9 2.2.1 Characterization of Reliability.....................................................................9 2.2.2 Defects, Faults, and Errors.........................................................................11 2.2.3 Error Rates..................................................................................................11 2.3 Fault Tolerance in Memory..................................................................................14 2.3.1 Device Redundancy....................................................................................14 2.3.2 Error Correction Codes...............................................................................17 3 NANOSCALE MEMORY DEVICES.......................................................................19 3.1 Present and Future Scaling Challenges.................................................................19 3.2 Nanoscale Building Blocks...................................................................................21 3.2.1 Nanoscale Wires.........................................................................................21 3.2.2 Transistors, Diodes, and Switches..............................................................22 3.3 Molecular Electronic Architectures......................................................................23 3.3.1 Crossbar Circuits........................................................................................24 3.3.2 Micro/Nano Interface.................................................................................25 3.3.3 Hierarchical Architectures..........................................................................27 3.3.4 New Reliability Concerns...........................................................................27 v

PAGE 6

4 HIERARCHICAL FAULT TOLERANCE FOR NANOSCALE MEMORIES........29 4.1 Benefits of a Hierarchical Approach....................................................................29 4.2 Proposed Memory Organization...........................................................................30 4.2.1 Architecture Overview...............................................................................30 4.2.2 Possible Implementation Strategy..............................................................32 4.3. Fault Tolerance Mechanisms...............................................................................33 4.3.1 Hierarchical Built-in Self-Test and Self-Repair.........................................33 4.3.2 Advanced Error Correction Codes.............................................................34 5 MEMORY SIMULATION AND RESULTS............................................................36 5.1 Simulator Architecture..........................................................................................36 5.1.1 Simulator Components...............................................................................37 5.1.2 Simulation Interface...................................................................................38 5.2 Simulation Component Details.............................................................................40 5.2.1 Parameter Generation.................................................................................40 5.2.2 Simulation...................................................................................................41 5.2.3 Results Analysis.........................................................................................47 5.3 Simulator Validation.............................................................................................48 5.4 Simulation Results................................................................................................49 6 CONCLUSION AND FUTURE WORK...................................................................53 6.1 Conclusions...........................................................................................................53 6.2 Future Work..........................................................................................................54 APPENDIX A MODERN MEMORY DEVICES..............................................................................56 B SIMULATOR SOURCE CODE................................................................................57 B.1 Initialization and Variable Declaration................................................................57 B.2 Graphical User Interface......................................................................................59 B.3 Simulation Subroutines........................................................................................65 B.3.1 Generate Failures.......................................................................................65 B.3.2 Simulate.....................................................................................................67 B.3.3 Device Redundancy...................................................................................75 B.3.4 Run Simulation..........................................................................................75 B.4 Graphical User Interface Subroutines..................................................................78 B.4.1 Commify....................................................................................................78 B.4.2 Plot Results................................................................................................78 B.4.3 Update Memory Size.................................................................................80 B.4.4 Menubar Subroutines.................................................................................81 B.4.5 Text Editor Subroutines.............................................................................86 vi

PAGE 7

LIST OF REFERENCES...................................................................................................88 BIOGRAPHICAL SKETCH.............................................................................................93 vii

PAGE 8

LIST OF TABLES Table page 2-1. Reliability metric summary........................................................................................10 2-2. Hamming (7,4) parity and syndrome generation equations.......................................18 3-1. ITRS DRAM production product generations............................................................21 4-1. ECC summary for the proposed architecture.............................................................35 5-1. Simulator input parameters.........................................................................................41 5-2. Simulator validation to SEC analytical model for small memory sizes.....................48 viii

PAGE 9

LIST OF FIGURES Figure page 2-1. Memory hierarchy representation.................................................................................6 2-2. The basic memory architectures...................................................................................7 2-3. The bathtub curve used to illustrate reliability over time...........................................10 3-1. Nanoscale FET implementations................................................................................23 3-2. Demultiplexor designs................................................................................................26 4-1. Proposed hierarchical memory architecture...............................................................31 4-2. A diagram of the nanoFabric architecture..................................................................32 5-1. High-level simulator data flow diagram.....................................................................38 5-2. Simulation interface with sample results....................................................................39 5-4. Hierarchical hash data structure used in the simulator...............................................44 5-5. Example of failure chaining process in simplified fail list.........................................46 5-6. Simulator validation to SEC analytical model for large memory sizes......................49 5-7. HUEs for 1-Gbit memory with 64-bit word size.......................................................51 5-8. HUEs for 1-Gbit memory with 128-bit word size.....................................................51 5-9. HUEs for 1-Gbit memory with 256-bit word size.....................................................52 ix

PAGE 10

Abstract of Thesis Presented to the Graduate School of the University of Florida in Partial Fulfillment of the Requirements for the Degree of Master of Science PERFORMANCE ANALYSIS OF DYNAMIC SPARING AND ERROR CORRECTION TECHNIQUES FOR FAULT TOLERANCE IN NANOSCALE MEMORY STRUCTURES By Casey Miles Jeffery December 2004 Chair: Renato J. O. Figueiredo Major Department: Electrical and Computer Engineering Continued advances in the fabrication of molecular structures have led to speculation that nanoelectronic devices may be incorporated with CMOS technology in the realm of ultra-dense logic and memory systems. Many proposed molecular systems involve self-assembly techniques ideally suited for large, regular structures such as memory. This method of fabrication, however, is predicted to result in an increased hard error rate due to physical defects. This effect will be compounded by an increased sensitivity of the ever-smaller bit storage units to soft errors (e.g., resulting from noise and radiation effects). Current memory devices rely on fault tolerance schemes such as modified Hamming error correcting codes and static spare swapping to cope with these errors. It is conceivable, however, that such solutions may not be sufficient in systems that are larger and built using less reliable components and/or manufacturing techniques. x

PAGE 11

This thesis describes a memory organization that supports hierarchical, dynamic fault tolerance mechanisms applicable to heterogeneous CMOS/molecular systems. It has been projected that, in such systems, programmable mapping circuitry becomes necessary when the interface between microscale and nanoscale address lines is formed in a non-deterministic manner. This mapping circuitry may double as a means of implementing more advanced forms of reconfiguration and error correction codes useful in detecting and recovering from runtime faults. In lieu of the ability to formulate an analytical model for such complex systems, computer simulation is used to estimate the effectiveness of various fault tolerance configurations. Results show that for a given percentage of redundancy overhead, these methods may allow for substantial improvements in reliability over standard error correction codes. xi

PAGE 12

CHAPTER 1 INTRODUCTION The size of the devices used to construct electronic components has decreased at an astonishing pace over the past fifty years. It has followed the very familiar Moores Law, which states that the number of devices doubles approximately every 18 months. As of 2004, the state of technology is still several orders of magnitude away from even the conservative limits imposed by physics (i.e., 1 bit/ 3 ) [1], and it is in the foreseeable future when human ingenuity will allow for the integration of huge numbers of devices in a single part (i.e., 10 12 /cm 2 ) [2]. There will be many new challenges associated with this level of integration that will require complete new approaches to designing, fabricating, and testing devices. The circuits will likely require construction using self-assembly and self-alignment techniques at the molecular level, as it will almost certainly be infeasible to accurately order and connect such a large number of devices using the current lithography-based technology. The relative inexactness inherent in the self-assembly method of construction will make the technology most applicable to the manufacture of devices with large, regular patterns such as programmable logic and memory arrays at the lowest level. The arrays of nanoscale devices may then be used hierarchically as building blocks for more complex circuits [3, 4]. Even in these very regular processes, it is predicted that defect densities on the order of 5% can be expected, and the small size of the devices will make them more sensitive to external disturbances such as radiation and electromagnetic interferences [2, 1

PAGE 13

2 5, 6]. Off-line testing to find the faults will be impractical due to the sheer number of devices. Instead, the circuits will be required to test themselves to a high degree and detect defects in a reasonable amount of time. They must also be capable of dealing with the defects by employing some form of fault tolerance. The defects that occur at runtime are referred to as dynamic defects and are the focus of this thesis. In the past, dynamic defects have all but been ignored. Only high-end workstation and server memory currently make use of dynamic fault tolerance mechanisms in the form of error correcting codes (ECCs). The codes have typically been capable of logically correcting at most a single bit in each block of data accessed. This will not be sufficient, however, for memories with orders of magnitude higher error rates. In addition to the increase in bit error rates, an increase in the rate of row, column, chip, and other periphery-related defects is expected. Defects of this type often cannot be handled even when ECCs are implemented and ultimately lead to the failure of the entire memory. The most likely solution to combat these problems in future memory designs will be to implement a form of fault tolerance that is more advanced than has been done in the past. This new method of fault tolerance involves more powerful ECCs, as well as dynamic reconfiguration. In dynamic reconfiguration built-in self-test (BIST) and built-in self repair (BISR) logic are used to detect and repair faults either by swapping the defective devices with spare devices or by marking the faulty devices so they will not be accessed. In this thesis, dynamic fault tolerance configurations are introduced that implement such advanced ECCs and dynamic reconfiguration in a hierarchical fashion. A simulator

PAGE 14

3 is then used to estimate the performance of the systems in terms of tolerance to faults throughout the usable lifetime of the parts. Finally, the simulation results are utilized to suggest configurations capable of providing the highest degree of synergism and correspondingly make the most efficient use of a given percentage of overhead in terms of redundant bits. Chapter 2 gives a brief overview of the current state of memory devices, as well as an introduction to fault tolerance mechanisms and the metrics used in determining the reliability of such devices. Chapter 3 covers many of the most promising nanoscale devices and architectures that have recently been developed and the corresponding reliability concerns associated with them. Chapter 4 introduces the proposed hierarchical fault tolerance mechanisms and goes on to show how they can be applied to the architectures from the previous chapter. Chapter 5 details the simulation methodology used to model the fault tolerance mechanisms and includes the results that have been obtained. Finally, Chapter 6 summarizes the findings and gives some insight into future work that may be done in the area.

PAGE 15

CHAPTER 2 BACKGROUND Fault tolerance in electronic devices involves the use of redundant circuitry to detect and possibly correct faults. This is not a new notion, as it has been in place since the 1940s when it was proposed by John von Ne umann. Before this discovery, it was thought that large-scale circuits (i.e., millions of devices) could not be constructed to function reliably. Von Neumann showed, however, that a very high degree of reliability can be obtained, albeit at a high cost in terms of redundancy overhead [7]. To date, von Neumanns ideas have garnered little attention in industry as circuits with hundreds of millions, and recently billions of devices have been constructed to function reliably with little or no need to implement such fault-to lerance mechanisms. These feats require extremely low error rates that are only possible with advanced silicon fabrication techniques conducted in the cleanest facilities in the world. It does seem reasonable to reevaluate the usefulness of fault tolerance, however, as device sizes approach atomic levels and the ability to build entire components with near perfection becomes too difficult and costly. This chapter first gives a brief overview of the current state of memory device technology and fabrication. The terminology related to memory reliability is then introduced, and the basic mechanisms used to deal with defects are discussed. The topics covered in this chapter will be used as the foundation for describing more advanced fault-tolerance architectures applicable to future generations of devices. 4

PAGE 16

5 2.1 Modern Memory Devices It is well-known that memory devices in computer systems are used in a hierarchical fashion that can be represented in a simplified form by the pyramid structure shown in Figure 2-1. The base of the pyramid is composed of very large devices such as tape backups and networked hard-disk storage devices that have a low cost per bit, slow access times, and large data access chunks. The top of the pyramid represents the smallest, fastest devices, which are the registers in the core of the microprocessor. The registers move data in small blocks, have very fast access times, and a high cost per bit. In recent years, there has been a continued growing and splitting up of the pyramid. The number of bits at each of the levels has grown steadily and new levels have emerged. There has also been a great deal of growth outside the pyramid. The use of non-volatile flash memories in devices such as cameras, cellular phones, and personal digital assistants is on the rise. There has also been a push towards entirely new architectures that integrate logic processing units into memory devices [8]. It is conceivable that the memories constructed from nanoscale devices may fill any of these roles, as well as others that have yet to be discovered. Because of this uncertainty, the fault tolerance architectures to be presented in this thesis are relatively generic and applicable to a wide range of memories, including random access devices in the levels that have been highlighted in Figure 2-1.

PAGE 17

6 Figure 2-1. Memory hierarchy representation. Highlighting is placed over levels most applicable to the fault tolerance architectures described in this thesis 2.1.1 Memory Architecture Modern memory devices come in many forms and are categorized based on their function and access pattern. The function is defined as whether the memory is read-only or rewritable, as well as if it is volatile or non-volatile. The access pattern of the memory classifies whether the data can be randomly accessed, as in a common DRAM device, or if the accesses must be done serially, which is the case for video memory. There are also memories that use parts of the data instead of an address for memory accesses. These are known as associative memories and are common in cache structures. An overview of a wide variety of memory devices is included in Appendix A. The layout of most memory devices is approximately the same regardless of the type. They generally follow a square shape and make use of row and column decoders to control the selection of word and bit lines. Figure 2-2A shows the simplest example in which the decoded row signals select multiple words and the decoded column signals choose which of those words is returned. Figure 2-2B expands on this slightly by adding a block decoder that determines in which block the word line will be activated. The

PAGE 18

7 design is known as divided word line (DWL), and the extra level of decoding has the benefit of reduced power consumption at the cost of additional decoding latency and area overhead. Figure 2-2. The basic memory architectures. A) The simplest example with a square memory array and row and column decoders. B) Example of a DWL structure with an extra level decoding at the block level. 2.1.2 Memory Fabrication Virtually all modern semiconductor memory devices are manufactured on silicon wafers using fabrication processes that have been fine-tuned for decades. The fabrication starts with silicon dioxide ore (sand) which is converted through several stages before ending in the form of a doped silicon crystal ingot. The ingot is sliced into wafers, planed to be perfectly flat, and put through nearly 200 major layering, patterning, doping, and heating steps [9]. Each of these steps has some probability of introducing a defect into the device, although some are much more likely than others. The patterning process, also known as photolithography or masking, is the most critical as it define the exact size, shape, location, and orientation of the circuit devices on the wafer. It is on the bleeding edge of technology, and with current processes having more than two dozens mask steps and

PAGE 19

8 device feature sizes of less than 100 nanometers, there is a tremendous possibility for the introduction of defects. Defects are caused by four major types of contamination: particles, metallic ions, chemicals, and bacteria [9]. When particles are transferred to the wafer surface, either through the air or the growth of bacteria, they can cause opens by not allowing features to form correctly, or shorts by connecting wires together. Metallic ions cause problems by disrupting the doping and altering device electrical characteristics. Chemicals interfere in wet processing steps and can affect etching or interact to change the properties of the material. Defects have been avoided historically by manufacturing the wafers in some of the cleanest facilities in the world. Clean room technology makes use of high-efficiency particulate attenuation (HEPA) filters and laminar air flow to circulate the air and filter out all particles down to sub-micron sizes. The introduction of particles is kept to a minimum by automating as much as possible and keeping people and tools out of the environment. The tools are kept in separate bays called chases and have only their interfaces exposed. When it is necessary for workers to enter the clean room, special suits are worn to reduce the release of contaminants. There is a class rating that has been associated with the level of cleanliness that has been achieved. A Class-1 clean room has particles with a maximum size of approximately 100 nm. This is based on the ru le of thumb that pa rticles must be 10x smaller than the smallest feature size to maintain an acceptable level of physical defects. Currently, the highest density memory devices are manufactured in clean rooms with a rating of Class-0.1 or better and must be almost fully automated to avoid the

PAGE 20

9 contamination brought in by humans. This is likely approaching the limit of the level of cleanliness that can be achieved, yet device feature size continues to shrink [10]. 2.2 Reliability The reliability of a device is defined by the IEEE as, the ability of a system or component to perform its required functions under stated conditions for a specified period of time [11]. It is an important metric in memory devices since it defines the probability that a device will function correctly throughout its usable lifetime, which is typically considered to be on the order of 100,000 hours (~11 years). This section will introduce the terminology related to reliability, much of which is specific to electronic devices. 2.2.1 Characterization of Reliability The reliability of a device is a function of its failure rate, In memory devices, the failure rate is often modeled as a constant, which implies that the intervals between failures follow an exponential distribution with a probability density function of e t The inverse of the failure rate gives the mean time to failure (MTTF), also known as mean time between failures (MTBF). This value is the expected time for a failure to occur. The derivation of the reliability is shown in Equation 1, and a complete summary of the terminology is provided in Table 2-1 [12, 13, 14]. (1) tee11te1R(t)ttt0

PAGE 21

10 Table 2-1. Reliability metric summary Metric Function Symbol Failure Rate h(t) Mean time to failure MTTF 1/ Exponential PDF f(t) e t Exponential CDF F(t) 1-e t Reliability R(t) e t The failure rate is not really constant over the lifetime of devices as was assumed in the derivation of reliability. It is typically represented as a collection of three failure curves that sum to form what is referred to as a bathtub curve (Figure 2-3). The relatively flat bottom of the bathtub defines the usable lifetime and allows for the assumption in the derivation. Figure 2-3. The bathtub curve used to illustrate reliability over time. The bottom of the bathtub represents the usab le lifetime of components.

PAGE 22

11 The steep failure rate early in the lifetime is called infant mortality and is the result of the failing of marginally functional devices. A process known as burn-in is used to accelerate the state of the devices past this point before they are distributed to customers. The second steep curve at the end of the lifetime is from the wear-out of the devices. Wear-out is of little concern as most devices are out of commission by that time. The constant curve throughout the lifetime is the effect of single, random events such as latchup or heavy ion radiation strikes. 2.2.2 Defects, Faults, and Errors The distinction between defects, faults, and errors is not immediately clear. In this thesis the term defect is used to reference a physical imperfection in a device such as the opens or shorts described in the previous section. The ability of a part to work correctly when defects are present is called defect tolerance. If a defect occurs in a section of memory that is being accessed and results in incorrect data being returned, it is said to result in a permanent fault. If the fault can be corrected either by the part or through the use of software, then the part is said to have fault tolerance. Finally, if the fault cannot be corrected it will cause an error that will be seen by the user [12]. It is possible for a fault to occur without being a result of a physical defect. The extraordinarily small sizes of modern devices make them susceptible to environmental effects such as cosmic rays, alpha particles, and neutrons that can change the value of a stored bit. This type of fault is not permanent and is referred to as a transient fault. The next time the bit is written to it will be restored to the correct value [15]. 2.2.3 Error Rates There are typically two bit-level error rates given for a memory device, a hard error rate (HER) and a soft error rate (SER). The HER represents the frequency that

PAGE 23

12 permanent faults occur, and the SER represents the rate at which transient faults arise. The rates are measured in units of failures in time (FIT) where 1 FIT = 1 failure/10 9 device hours [15, 16]. The device-level error rate can be calculated by simply multiplying the bit-level rate by the number of bits in the device. For example, a 1-Gbit memory with a bit-level HER of one FIT is calculated in Equation 2. hourfailure 1hours device101failurebits10HER991Gb (2) The HER of a device is affected somewhat by the level of maturity of the process technology, but it is most directly correlated to the reduction is size of devices [17]. The HER has increased slightly with each generation, but it has remained relatively low overall at approximately one to two percent of the total error rate [18]. This is due in large part to the precision with which silicon devices are fabricated. Soft errors represent the majority of the errors that occur. Unlike the HER, the SER is mostly dependent on the operating voltage, charge collection efficiency, and cycle time of the device, as well as the level of radiation present in the device. The last several generations of SRAM and DRAM memories have seen reductions in charge collection efficiency and the impact of radiation through improvements in fabrication processes, both of which help to lower the SER. They have also had continuous reductions in the operating voltages and cycle times, both of which result in an increased SER. The outcome of these counteracting influences has been a slight decrease in the bit-level SER [18]. In devices such as DRAM, the effect of the cycle time on the SER must be averaged over both the refresh cycle time and the read/write cycle time. In a 16-MB DRAM chip the difference in the refresh cycle time of 15 s and read/w rite cycle time of

PAGE 24

13 200 ns leads to a 50x increase in SER [16]. To correctly calculate the overall SER, the ratio of each cycle time must be estimated and multiplied by the corresponding SER. The cycle time at which the memory has the data refreshed, or scrubbed, also plays an important role in determining the overall error rate in the case that fault tolerance is implemented. If a correctable number of bits have become faulty as a result of permanent faults and are not rewritten, they have the possibility of lining up with transient faults to cause an error. The rate at which the soft errors are removed from the system is referred to as the scrubbing interval and must be accounted for in determining overall system reliability. The level of radiation present has a significant impact on the SER. Experiments have shown more than a 10x increase in the SER of DRAM devices in going from sea-level to an altitude of 10,000 feet and another 10x increase in going to commercial airline flight altitude [15, 19]. The increase is the result of reduced protection from the atmosphere. For this reason, any devices that will be used at high altitudes, and especially in space, must have much more protection than those used at terrestrial levels. At the device level, both the HER and SER have shown some cause for concern as the size of memories continues to expand at a rate of up to 60% per year [20]. Even if the bit-level error rates were to remain constant going forward, the error rate at the device level would become unacceptable. For example, an SRAM memory manufactured on 90 nm technology has a bit-level HER on the order of 10 -5 FIT and a SER of 10 -3 FIT at terrestrial levels. For a 1-MB array, now common as a cache in microprocessors, the device-level error rate is 8472 FIT with an acceptable MTTF of 13.5 years. If the cache size were to increase to 128-MB with the bit-level error rate unchanged, however, the

PAGE 25

14 MTTF would drop to 38 days, which is clearly unacceptable and would require some form of fault tolerance [18]. 2.3 Fault Tolerance in Memory As the size of memories grew to millions of bits, it became impractical for manufacturers to fabricate fully functional parts with a high enough yield to be profitable. This led to the introduction of static redundancy where redundant rows and columns were included in the design and could be switched in to repair defects before shipping the part to the customer. In todays memories of a billion or more bits, static redundancy has become a necessity for producing functional units. To handle the errors that occur after the part has shipped, dynamic redundancy is used. The most common form of dynamic redundancy is the use of error correction codes (ECC). ECCs allow errors to be detected and corrected before the data are returned from the part. These and other forms of fault tolerance will be introduced in this section and will form the basis for the hierarchical fault tolerance mechanisms that will be proposed for future memories. 2.3.1 Device Redundancy Device redundancy is the inclusion of spare devices in the design that can be swapped with faulty devices to maintain functionality. As mentioned above, it has become an important technique for producing modern memories with an acceptable yield. Even with the precision of lithography-based fabrication, defects caused by contaminants and tiny variations in process steps have become inevitable. All types of memory from DRAM to microprocessor cache to flash are built with extra rows and/or columns that can be switched in when defects are found.

PAGE 26

15 The first step in making a repair is to determine the location of the defects in the memory through the use of fault diagnosis algorithms. The specific types of defects that can occur vary by memory type, but the algorithms used are similar in all cases. Fortunately, the test cases are much simpler than those used for logic parts and work by performing a sequence of reads and writes that vary in direction, bit pattern, and timing. The goal is to locate bits that are stuck at high or low levels, lines that are bridged together as a result of a short circuit, lines that are open circuited, and bits or lines that are coupled together [12]. The patterns are applied with automated test equipment (ATE) and/or BIST circuitry integrated into the memory. ATE are very expensive tools capable of applying comprehensive sets of diagnostic test patterns to the memories at high speeds. As the size of the memories has grown, the ability to test for all faults with ATE has become too costly in many cases, and impossible in others, as some embedded memories cannot be accessed directly through the external pins. It has been estimated that for 1-Gbit memories, the test cost is roughly 85% of the manufacturing cost when done on standard ATE machines, and the ratio continues to increase [21]. To reduce the test costs to profitable levels and to gain access to embedded memories, many designs have been developed that allow for BIST. With BIST the memory can test itself by applying the necessary patterns and returning the results, which allows for much simpler and less expensive ATE. There are many techniques used for generating the patterns, but most make use of linear feedback shift registers (LFSRs) or microcode programming. Advance compilers have been designed to assist in the design

PAGE 27

16 of the BIST architectures, which typically require less than one percent of the total chip area [22-24]. When the faults have been detected, another algorithm is used to determine which devices need to be repaired and if there is enough redundancy available. The complexity of an optimal repair algorithm has been shown to be NP-hard, so heuristic approaches are often used. The ATE is usually capable of determining optimal or near-optimal repair strategies. In cases where the repair is done on the part with BISR circuitry, however, greedy algorithms may be necessary to reduce test time and circuit complexity. The redundant devices are swapped in by one of two reconfiguration methods. If the reconfiguration is done by the manufacturer, it is known as a hard repair. A hard repair is done by physically blowing fuses on the device through the use of lasers or electrical currents. If the repair is done in the field, such as in a BISR circuit, it is known as a soft repair. A soft repair is done by programming non-volatile devices on the parts such as EPROMs or flash cells to connect the redundant devices. Device redundancy does have some limitations. First, when a fault occurs, the data that are in error cannot be corrected. The repair process must also be done non-concurrently, or when the memory is not in use, which means the memory must be cleared of its current contents before it can be tested. This requires that the current memory contents be copied to spare arrays before the testing if it is not to be lost. The process is also relatively slow as it is necessary to scan the entire memory, devise a repair strategy, and program in the redundant devices. Overall, device redundancy is an important means of fault tolerance, but it is not sufficient.

PAGE 28

17 2.3.2 Error Correction Codes ECCs are another important form of fault tolerance. The term is typically used to refer to linear block codes that encode redundant information about the data stored in the memory into what are referred to as codewords. The number of valid codewords is then a subset of the possible codewords. The benefit to this method is that it can be used concurrently, or when the memory is in use. It can also be integrated in many devices for a tolerable overhead in term of latency and area, and in some implementations it is capable of correcting errors. The simplest form of ECC is the parity check code in which an extra bit is added to each word and set to give either an even (even parity) or and odd (odd parity) number of 1s. This allows for the detection of all errors in which an odd number of bits are at fault. More advanced ECCs have been developed that are capable of not only detecting errors, but also correcting them. This class of codes requires that the valid codewords be separated by a minimum number of bits given in Equation 3, where d is the minimum distance, and y is the number of errors that are correctable. 1y2d (3) The most common implementation is a class of Hamming code known as a single-error correcting, double-error detecting (SEC-DED) code. The Hamming SEC-DED code uses total bits to encode information bits. An example is the Hamming (7, 4) code in which three parity bits are added to each 4-bit word to form a 7-bit codeword ( i 12n nn12 1 +i 2 +i 3 +i 4 +p 1 +p 2 +p 3 ). The parity bits are generated in such a way as to distinguish the location of the bit in error after decoding of the received codeword in what is referred to as the syndrome. The equations for the Hamming parity and

PAGE 29

18 syndrome generation are given in Table 2-2 [12]. A complete overview of ECCs is given in [25, 26]. Table 2-2. Hamming (7,4) parity and syndrome generation equations Parity generation Syndrome generation 3211iiip 32111iiips 4322iiip 43222iiips 4213iiip 42133iiips

PAGE 30

CHAPTER 3 NANOSCALE MEMORY DEVICES The term nanoscale is generally used to refer to structures that have dimensions smaller than 100 nm. Nanoelectronics, or equiva lently molecular electronics, then refers to any electronic device with critical dimensions of less than 100 nm. There is one exception to this; the terms are limited to novel technologies of this scale. The restriction is used to exclude CMOS technology, which is technically below the 100 nm mark in modern processes. This chapter presents the goals, both near-term and long-term of the semiconductor industry, as well as the upcoming challenges in scaling silicon-based CMOS devices to ever smaller sizes. Several promising molecular electronic technologies are then introduced that are most applicable to memory devices and may go on to become successors to CMOS technology. 3.1 Present and Future Scaling Challenges The International Technology Roadmap for Semiconductors (ITRS) is a document that is based on predictions of the state of semiconductor technology for approximately 15 years into the future [27]. It is compiled by a cooperative effort of organizations in every aspect of semiconductor technology from manufacturers to universities. The document tracks the device characteristics expected at each generation of technology and defines the basic requirements necessary to achieve them. 19

PAGE 31

20 The requirements are grouped into thirteen different areas related to the physical properties, manufacturability, and testability of the devices. The document also outlines any roadblocks foreseen in each of the areas that may inhibit the realization of each generation of technology. The roadblocks are categorized as being yellow, meaning that a manufacturing solution is known but not yet implemented, or red, which means that no manufacturing solution is currently known. The challenges in the red category form what is known as the Red Brick Wall that represents the major hurdles that stand in the way of continuing the exponential growth of Moores Law. In the 2001 edition of the ITRS the Red Brick Wall was predicted to occur in the 2005 to 2006 timeframe. According to the update provided in 2003, some of these obstacles have been overcome, but only enough to push the wall out to about 2007. Of the dozens of near-term challenges that remain, some of the hardest are dealing with the exponential increases in leakage current, making masks for the lithography process, developing metrology tools to perform critical measurements, and building the interconnect networks on the chips. Assuming that engineers are able to work quickly to develop the solutions necessary to maintain the growth predicted by the ITRS, the critical dimension in devices will be on the order of 20 to 30 nm in only a d ecade (Table 3-1). Devices of this scale will likely be fabricated quite differently than they are today. They will presumably be built using a bottom-up approach in which large, regular arrays are self-assembled rather than lithographically drawn. If lithography-based CMOS processes are still implemented, they will be integrated with nanoelectronics in what has been referred to as mixed CMOS/nano designs or NoC. NoC circuits will consist of CMOS as the

PAGE 32

21 underlying structure that connects blocks of nanoscale structures together while providing only a small degree of functionality [28]. Table 3-1. ITRS DRAM production product generations Year of Production 2004 2007 2010 2013 2016 DRAM Pitch (nm) 90 65 45 32 22 Cell Area (mm) 0.065 0.028 0.012 0.0061 0.0025 Functions/chip (Gbits) 1.07 2.15 4.29 8.59 34.36 Gbits/cm at production 0.97 2.22 5.19 10.37 24.89 3.2 Nanoscale Building Blocks To date, there have been no large-scale memory devices constructed from molecular electronics, but many of the necessary building blocks are coming into place. There have been prototypes demonstrated of nanoscale wires, field-effect transistors, junction transistors, diodes, and switches [2]. There has also been some success in small-scale integration of these devices into work ing circuits. For example, Hewlett Packard demonstrated a 64-bit memory array that was constructed from a cross-bar configuration of platinum wires 40 nm in diameter. Much research is still necessary to continue refining the building blocks a nd discovering ways to integrate before usable products can be produced. 3.2.1 Nanoscale Wires There first useful building block was the discovery of nanoscale wires. The two most promising are the carbon nanotube (CNT) and the semiconductor nanowire (SNW). The CNT is composed of a cylinder of carbon atoms. The wall of the cylinder can be as thin as a single monolayer, or can consist of several layers. The diameters range from a single nanometer to several nanometers, and the length can be on the order of millimeters. It is also possible for the tube to have either metallic or semiconductor properties, depending on the helicity of the carbon lattices. CNTs are typically formed

PAGE 33

22 using catalyzed chemical vapor deposition (CVD), and unfortunately, the type of structures produced is a random event. It is not currently possible to uniquely construct metallicor semiconductor-type nanotubes [29, 30]. SNWs refer to nanoscale wires composed of semiconductor materials in general, but the most common are Si, Ge, and GaAs. They have many of the same properties as CNTs. It is possible to construct the wires with diameters as small as 3-nm and lengths on the order of microns. The greatest benefit the SNWs have over their carbon counterparts is that, although the same method of CVD fabrication is used, it is possible to precisely control the composition, diameter, length, and doping of the structures. It is also much easier to control the parallel production of SNWs [4, 29]. 3.2.2 Transistors, Diodes, and Switches In addition to nanoscale wires, it is also necessary to have active devices as building blocks. A wide variety of transistors, diodes, and switches have been constructed, albeit on small scales. The devices have been limited to proof-of-concept using fabrication techniques far too complex for mass production, but they are the first steps towards characterizing the devices and developing necessary models. Diodes and FETs have been constructed by using the CNTs and SNWs described above, which can be easily doped to be either p-type or n-type [29, 30]. In one implementation, a single-walled CNT or a SNW is used as the channel of a FET and is placed between two metal electrodes. The molecular FETs, benefited by ballistic transport properties, have been shown to be comparable in performance to similar CMOS devices [31]. In another example, a multi-walled CNT is side-bonded to electrodes and an FET is formed by selectively thinning portions of the tube by removing the outer shells of the CNT [30]. Both diodes and FETs have been fabricated by simply crossing

PAGE 34

23 oppositely doped CNTs or SNWs to form p-n junctions at the intersections [2, 29]. Figure 3-1 graphically depicts each of these constructs. Switches have been developed in much the same way as the diodes and the FETs. Most implementations involve crossing nanoscale wires and using the intersection as a simple two-terminal switch. The benefit of such switches is that the storage area for a bit is merely the intersection of two wires. No periphery is necessary to program or access the bit. A conventional device such as an SRAM or a DRAM requires as much as 100x larger footprint for a programmable crosspoint when built with equivalent feature sizes [2]. Figure 3-1. Nanoscale FET implementations. A) depicts a CNT being used as a FET channel. B) shows a multi-walled CNT with various number of shells removed. C) represents an oxidized SNW gated by a CNT. 3.3 Molecular Electronic Architectures Currently, the greatest challenge in nanoelectronics is the ability to efficiently build and configure reliable architectures out of the core building blocks discussed in the previous section. There are many issues related to this including designing and simulating an architecture composed of billions of gates, fabricating a device that adheres to the design, and testing the circuits and repairing defects. The design and simulation of the nanoscale circuits is beyond the scope of this thesis, but the regularity of memory devices should make this challenge manageable and certainly much simpler than it will be for logic devices. The complexity of the

PAGE 35

24 fabrication process will require tradeoffs with the testability of the design. If the designs remain fully deterministic, as they are today, the testing process will remain relatively simple, although steps would have to be taken to find methods of testing such a large number of bits in a reasonable amount of time. If the designs become completely random in structure, the fabrication process will be greatly simplified and the testing process will become complex. It will be necessary to map out the device post-fabrication and determine if it is a usable part. 3.3.1 Crossbar Circuits The crossbar architecture appears ideally suited to memory devices made from nanoscale wires. In this design a simple 2-D mesh of wires is constructed by lining up a number of wires in parallel to form a plane, rotating the plane by 90 and repeating the process. There have been a number of techniques developed for doing this, including fluid flow directed, electric field directed, biologically assisted, catalyzed growth, and nanoimprinting assembly methods, among others [2, 28, 29]. In the fluid flow and e-field methods, the wires are lined up by the flow of fluids or current, respectively, and the spacing is determined by the rate of the flow. Biologically assisted processes use complementary DNA strands as smart glue to selectively bind components and move them in place. In the catalyzed growth methods, a laser ablation or CVD is used to direct the growth of nanodroplets in one direction. Finally, in nanoimprinting, a mold is constructed using e-beam lithography and then wires are cast into the mold. Nanoimprinting is the only known method of making irregular arrays of wires in a controlled fashion. The main drawback is that it has a larger pitch requirement than the

PAGE 36

25 other methods. Notably, it was the method used by HP to make the prototype 64-bit memory mentioned above. There have been several methods devised to store the information at the wire intersections of the crossbars. HP patented a method in which molecules are sandwiched between the wires. The molecules can be electrochemically modified to vary the tunneling barrier between the two wires [32]. IBM patented a similar idea except that magnetic structures are placed between the devices [33]. In a method developed at Harvard, there is nothing between the wires. The wires are suspended slightly apart from one another at the intersection. To program the device, electromechanical forces are used to attract the wires closer together. They are then held in this position by van der Waals forces at a second potential energy minimum [34]. Using crossbar circuits and storing the state of the bits at the intersections does have some drawbacks. First is the inability to create aperiodic structures, which would have to be configured into the meshes post-fabrication. Also, the diode-resistor logic available does not allow for signal gain. The lack of gain can be coped with by including molecular latches in the periphery circuitry and by keeping the size of the arrays small and using them as building blocks for larger circuits [28]. 3.3.2 Micro/Nano Interface At present, the only way to make use of the crossbar arrays is to connect them to a microscale interface at some point. This is not a trivial task because even if a microscale wire could be attached to each nanoscale wire, such a design would defeat the purpose by requiring microscale pitch between wires. Most designs have dealt with this requirement by including demultiplexor circuits in which a logarithmic number of microscale lines are used to interface with the given number of nanoscale lines.

PAGE 37

26 The approach taken by HP in making a demultiplexor was to make connections between the microscale and nanoscale lines with a stochastic chemical process that has a 50% probability of making a connection. It was shown that by increasing the number of address lines necessary to address n data lines from to approximately there is a reasonable probability that each bit can be uniquely addressed [35]. n2log n2log5 Another approached suggests that a decoder may be imprinted lithographically during fabrication across which the nanoscale wires are placed. This design also made use of a sparser decoding scheme based on two-hot coding to reduce the portions of the array that would be lost in the case of an error in the decoding circuitry [4]. A third method has also been proposed in which the decoder is actually integrated into the crossbar architecture. The outer edge of the array is fabricated to allow connections to be made to microscale devices and is programmed at higher voltages than the interior of the array. The decoder is then programmed into the periphery during the testing phase of the circuit [28]. A diagram of each of the demultiplexor designs in shown in Figure 3-2. Figure 3-2. Demultiplexor designs. A) Random connections made between microscale address lines and nanos cale data lines. B) Lithographically drawn decoder pattern. C) Demux being encoded into the nanoscale wires on the periphery of the memory.

PAGE 38

27 3.3.3 Hierarchical Architectures There are a number of compelling reasons for designing the overall architecture in a hierarchical fashion where smaller arrays exhibit locality and work as independent pieces of larger constructs. As mentioned above, the size of the arrays is limited by the lack of gain in the circuits and requires that multiple smaller arrays be connected to allow for signal latching and restoration. This is also true for interconnect purposes as switching frequency continues to increase and the latency in routing the signals becomes a limiting factor. A final argument for smaller arrays is a result of the lack of control in fabricating self-assembled circuits. Smaller design should lead to greater accuracy and fewer defects. It also makes sense to use a hierarchical architecture from a design perspective. It will simplify the process greatly to have multiple levels of abstraction. Again, this is most applicable to logic designs, but it will play a part in memories as more logic is integrated either for computational purposes or to simply perform all the necessary mapping, testing, and repair processes on the arrays. 3.3.4 New Reliability Concerns It has been implied throughout this chapter that self-assembled molecular electronics will have a number of new concerns with regard to defects during fabrication. The first problems stemmed from the inability to control the properties of CNTs. This was exacerbated by the further inability to deterministically control the alignment and spacing of the nanoscale wires in the construction of the crossbar arrays. The limits of cleanroom technology will also come into play as devices sizes shrink to on a few nanometers. The consequence of all these factors is a defect rate expected to be on the order of 5% [2, 5, 28].

PAGE 39

28 Even if the initial defects are dealt with during the manufacturing process and a working part is completed, there will inevitably be a much higher rate of both permanent and transient faults during the runtime of the devices. The extremely small size of the structures will make them more susceptible to the random single-event effects from high-energy particle strikes. The only way to create reliable parts will be to integrate fault tolerance circuits into the designs to deal with the dynamic faults.

PAGE 40

CHAPTER 4 HIERARCHICAL FAULT TOLERANCE FOR NANOSCALE MEMORIES This chapter introduces hierarchical memory architectures with several possible forms of fault tolerance at each of the levels. The goal is to develop generic memory architectures with new, more powerful combinations of fault tolerance mechanisms that take advantage of the intrinsic properties of molecular electronics to overcome what is expected to be a very high defect rate. Several possible implementations will also be suggested based on previous work in the area. 4.1 Benefits of a Hierarchical Approach It makes sense to use a fault tolerance scheme that is hierarchical given that the underlying memory structure is expected to be hierarchical. In fact, it has been found to be beneficial to do so. The defects that occur in a memory device have been related to the birthday surprise problem [36]. The problem calculates the expected number of independent trials necessary to have one of a set of equally likely results occur twice. In the case of birthdays, the outcome is two persons sharing the same birthday. The expected value in this case is a surprisingly low 24 people. The value is also low for memories if only single level fault tolerance is used. It was shown in [37] that a 16-Mbit memory chip with 1,000 random failures has greater than a 98% chance of having two errors line up. The benefit to having a second level of fault tolerance is that it effectively removes the first sets of matches. For example, if a SEC-DED code is used as a first-level mechanism and row redundancy as a second, then when two bits line up in a word to 29

PAGE 41

30 cause a failure the row can be replaced by the second level of fault tolerance. It is as if the persons with matching birthdays were taken out of the room. The effects of each fault tolerance mechanism work constructive to perform better together than either would individually. This was termed synergism in [37] and is the basis for the configurations to be presented in the remainder of the chapter. 4.2 Proposed Memory Organization The architecture used in this thesis is based on the specific set of conditions applicable to molecular electronics that were specified in Chapter 3, and which can be summarized as follows. The memories will be built hierarchically from relatively small, regular arrays of nanoscale wires. The arrays will be self-assembled with a high degree of randomness and an error rate on the order of 5% [2, 5, 28]. The arrays must function even with defects present. Locality is important; the fault tolerance mechanisms should work independently at the array level and only signal higher-level structures when local resources are exhausted. A high degree of configuration and testing will be necessary post-fabrication that will necessitate logic structures in the memory arrays. 4.2.1 Architecture Overview The basic structure of the memory organization to be considered is shown in Figure 4-1. Much as the DWL architecture added an extra level of word line decoding, this hierarchical architecture extends it further to allow for more sublevels. The lowest level, which will be referred to as level-0, consists of small, regular arrays of nanoscale wires such as CNTs or SNWs. There is also a logic component that will be called a processing nodePN0 in this case. The processing nodes are used to map out the

PAGE 42

31 structure of the arrays during the testing and configuration phases, and later they are used to implement the fault tolerance mechanisms in the array. PN0 PN0 Figure 4-1. Proposed hierarchical memory architecture. The lowest level consists of arrays of nanoscale wires with a processing node used for both static testing and dynamic fault tolerance. An arbitrary number of higher levels can exist, each consisting of arrays of lower level structures and a processing node. The next higher level, level-1, is composed of an array of level-0 structures. Again a processing node, PN1, is present to carry out the mapping necessary during the testing and configuration of the part. It is then used at runtime to implement fault tolerance mechanisms throughout the life of the part. If there were another level present, it would be referred to as level-2 and would consist of arrays of level-1 structures. The hierarchy

PAGE 43

32 can continue in this fashion, which is similar to previous hierarchical memory designs [38, 39]. It has been shown that in terms of timing latency the optimal number of levels in the hierarchy increases slightly with memory size and inversely with minimum feature size, but is on the order of five to ten [40]. 4.2.2 Possible Implementation Strategy As mentioned in Chapter 3, the higher level structures need not be implemented with nanoscale devices. It may be the case, at least initially, that NoC architectures are employed and lithographically-based CMOS structures are used in conjunction with nanoscale arrays. The CMOS structure would provide the logic functionality necessary for the mapping and fault tolerance, as well as basic infrastructure such as power, ground, and the clock signal. Figure 4-2. A diagram of the nanoFabric architecture Examples of such a NoC approach are the nanoFabric architecture [3], shown in Figure 4-2, and the similar array-based approach in [4]. In these designs, the lowest level consists of arrays of nanoscale wires. The next level then connects these arrays into a larger array called a cluster. The top level is an array of clusters with long nanoscale wires connecting the clusters. Inside each cluster, the sub-arrays allow for both

PAGE 44

33 programming and routing to be done entirely at the nanoscale. Through simple routing techniques, arrays can not only be used as a memory device but can be configured to implement full logic functionality. 4.3. Fault Tolerance Mechanisms The basic static and dynamic fault tolerance mechanisms in use today were introduced in Chapter 2. It is assumed, however, that these will not be sufficient for nanoscale memories. More advanced techniques are available and are presented in this section. 4.3.1 Hierarchical Built-in Self-Test and Self-Repair The need for BIST and BISR circuits is suggested by all of the conditions outlined in the previous section. The underlying assumptions that molecular electronic circuits will be very large, have high bit error rates, and require a great deal of functionality to be configured post-fabrication suggest that ATE alone will not be capable of testing and configuring the structures. The integrated mechanisms for test and repair are also necessary to contend with the dynamic defects that occur throughout the lifetime of the devices. They function synergistically with ECCs and other forms of fault tolerance by replacing defective devices that in many cases do not cause faults individually, but put a portion of memory at high risk of faults by lining up with future defects. In the proposed design, the reconfiguration processes are similar both during the device testing and at runtime. The BIST/BISR circuitry used to map out the functionality of the arrays is integrated into the PNs and continues to be employed at runtime when defects are detected either through error detection circuits or at scheduled servicing intervals.

PAGE 45

34 The reconfiguration options of the repair circuits are limited by the speed of the interconnections in the devices. It is not feasible to swap in a spare row from a distant array, for example, as it would require a decrease in the system clock rate. The hierarchical approach handles this by allocating spare devices to each array and allowing the local PN to conduct the sparing process. It is not until the local PN has exhausted the available resources that a signal is sent to the next level up in the hierarchy. When this occurs, the entire lower-level structure may be swapped with a spare. 4.3.2 Advanced Error Correction Codes ECCs were introduced as a method of dynamic fault tolerance that has been implemented in some modern, high-reliability memories. The most common codes are capable of either detecting errors or possibly correcting one error. There are much more powerful linear block codes possible, but because of the complexity of the encoding process double-error-correcting and triple-error-detecting is the most powerful one considered. A summary of the original codes, as well as several additional codes that are considered in this thesis is given in Table 4-1. There are other forms of ECCs that do not fall into the category of linear block codes, such as Reed-Solomon, Golay, Turbo, and RAID codes. Normally, these forms of error correction are not applied to memory devices. The one exception is RAID parity, which has been used by IBM in high-reliability Chipkill server memory [43, 44]. The benefit of RAID is that, unlike typical ECCs, it is designed for inter-block redundancy. When used in conjunction with ECCs, it allows for one more than the number of bits correctable by the ECC to be tolerated. The memory is configured with an additional level-1 array in each row or column of level-1 arrays. The additional array is used to store the parity of each bit in the other arrays. In this way, even if an entire array were to

PAGE 46

35 fail, the data in error could be reconstructed by taking the parity of the other arrays so long as the other arrays do not contain uncorrectable errors at the same relative location. Since the RAID parity is the only alternate form of dynamic redundancy typically applied to random access memory devices, it is the only one that will be included in this thesis. The additional requirement for the use of RAID redundancy is knowledge of more than the number of correctable bits being in error. This requires not only that an ECC be implemented in addition to the RAID code, but that the ECC is capable of detecting at least one more error than it is able to correct. For this reason, only the codes in Table 4-1 that are marked with an asterisk* can be used in conjunction with the RAID parity. Table 4-1. ECC summary for the proposed architecture. Code # Redundant Bits Parity* 1 SEC 1log2 N SEC-DED* 2log2 N DEC 2log22 N DEC-TED* 3log22 N

PAGE 47

CHAPTER 5 MEMORY SIMULATION AND RESULTS The complexity of the fault tolerance mechanisms implemented in the proposed architecture ruled out the derivation of analytical solutions to describe the performance of the systems. The size of the memories involved also made the Monte-Carlo simulation of the systems non-trivial. This chapter describes how the simulations of the various architectures have been accomplished and validated. The results of the simulations are included and analyzed with the goal of providing insight into optimal memory configurations in terms of reliability for a given amount of redundancy overhead. 5.1 Simulator Architecture The concept of developing simulation tools for analyzing the performance of memory architectures is not new. A variety of software packages are available for running many types of simulations. These packages are limited, however, in their support for simulating the states of very large memories. The requirement that multi-gigabit and even terabit sized memories need to be simulated in a reasonable amount of time, not once, but for thousands of iterations, rules out the generic simulation packages that are designed more for modeling the states of tools in a factory than the states of billions of nodes. To obtain the level of efficiency necessary, a very specialized simulator that implements only the desired functionality has been developed. The Perl programming language was chosen for the development environment since it offers the benefit of efficient, easy-to-use data structures, while maintaining adequate speed at runtime. Perl 36

PAGE 48

37 also provides a means of easily implementing a graphical interface with the Tk expansion module and eliminates the need to port and/or recompile the simulator code to various platforms that might execute the simulations. The individual components are described in detail in the following sections, and the source code has been included for reference in Appendix B. 5.1.1 Simulator Components The process of simulating the memories can be broken down into three steps. The first is the generation of memory configuration parameters for the simulator, including the size and arrangement of the memory, the hard and soft error rates, and the fault tolerance mechanisms to be supported. The next step is the simulation of the memory for a reasonable lifetime, which is typically considered to be 100,000 hours. The final step is the analysis of the simulation results. The bulk of the complexity lies in the simulator itself, which can be further broken into three steps. The first of these is the generation of failures throughout the lifetime of the memory. It would be intractable to store the complete state of all bits being simulated. Instead, a form of statistics called order statistics is used to generate, in order, the occurrence of failures for the entire lifetime of the part before any simulation takes place. The second step is the process of simulating and recording the actions taken by the memory controller to each of the failures. The final step is integrated into the simulation process and involves the check-pointing of the state of the memory at predefined polling intervals for off-line analysis during the results analysis. The simulator is constructed in a modular fashion based on the categories just described. The programs for input parameter generation, simulation, and analysis are implemented as shown in Figure 5-1. Each module is a standalone program and

PAGE 49

38 communication occurs through the use of files, which allows for the output of each to be saved and reused in subsequent analyses. Figure 5-1. High-level simulator data flow diagram. 5.1.2 Simulation Interface There are two interfaces provided to the simulator. One is a command line version available for use in running batch processes of several configurations for which input parameters have already been chosen and inserted into a batch file. The other is a graphical version that provides tools for configuring the parameters, executing the simulation, and analyzing the results. A screenshot of the graphical interface is shown in Figure 5-2. The top section of the interface provides a text area that is used for displaying results of simulations and

PAGE 50

39 configuring the batch execution scripts. Basic editing and searching capabilities are supported, as well as a tabbed notebook feature that allows for multiple files to be opened. Figure 5-2. Simulation interface with sample results The lower portion of the interface is also tabbed. The first tab contains settings widgets used for configuring the architecture of the memory to be simulated, which includes all of the input parameters given in Table 5-1 (see 4.2.1 for the Architecture Overview). There is an additional option for enabling verbose output, which provides an increased level of detail during the simulation. There are also a range of labels that are dynamically updated to reflect the overall memory structure described by the input parameters. The labels display the distribution of bits in the memory, as well as the memory-level MTTF based on the hard and soft error rates that have been defined. The tab for simulation settings is much simpler; the only inputs are for the lifetime of the

PAGE 51

40 memory, the polling interval, and the number of complete trials to be executed. The final tab is used for displaying summary information in the form of statistical box-plots, which are useful for comparing the results of multiple different configurations. 5.2 Simulation Component Details This section covers each of the simulation components in greater detail, including the algorithms implemented and the inputs and outputs. 5.2.1 Parameter Generation The first program in the simulator is used for input parameter generation. It was designed to ease in the generation of the many input parameter values to the memory simulator. It takes as input a listing of possible simulator inputs from Table 5-1 along with a range of values for each. Then, based on criteria such as percentage of redundancy overhead and possible fault tolerance mechanisms, the program generates all possible input configurations. The algorithm simply iterates through all combinations of the values for all parameters. Then, for each combination found that is within the search boundaries, a line is written to the output file that contains all the parameter information. The output file from the program shows all possible combinations of the parameter values that fit the search criteria. The file is typically quite large with a high degree of redundancy and so requires some manual filtering by the user to narrow down the list of possibilities to a representative few. The file is composed of tab-separated values so it can be imported into a spreadsheet program such as Excel and sorted and filtered to assist in choosing the test cases. The final selections are then copied into a batch file for use in automated executions.

PAGE 52

41 Table 5-1. Simulator input parameters Input Parameter Description Level 2 Row The number of rows of level-2 devices Level 2 Column The number of columns of level-2 devices Level 2 Redundancy The number of spare level-2 devices Level 1 Row The number of rows of level-1 devices Level 1 Column The number of columns of level-1 devices Level 1 Redundancy The number of spare level-1 devices Level 0 Row The number of rows of level-0 devices (bit level) Level 0 k Value The number of columns representing a word (bit level) Level 0 Plane The number of words in a row Level 0 Row Red. The number of spare level-0 rows Level 0 Column Red. The number of spare level-0 columns Errors Correctable The number of errors correctable with ECCs RAID/Chipkill Whether or not parity implemented across level-1 rows Usable Lifetime The lifetime for which to simulate (hours) Poll Interval The interval at which to record the memory state (hours) Hard Error Rate The bit-level hard error rate (FIT) Soft Error Rate The bit-level soft error rate (FIT) 5.2.2 Simulation The second program performs the simulations of the device configurations developed by the parameter generation. The simulation program is quite complex and can be broken down into three subcategories. The first of these deals with the generation of the failure times, types, and locations throughout the lifetime of the memory. Once generated, these failures are fed into the memory controller simulation which records the actions taken by the controller in response to each of the failure events. While the simulation is in progress, the memory state is recorded at predetermined polling intervals.

PAGE 53

42 Failure generation. In the initial versions of the simulator, an attempted was made to model the memory at a bit level. The hard and soft failures for each bit were generated and ordered in a linked-list structure. This method required the entire memory state to be kept and a sorting algorithm be used to ensure that the errors were processed in order. As the total number of bits increased to over one million, this method was soon determined to be impractical. Several alternative methods of simulation were considered, such as those based on Markov models and order statistics [45-47]. The current failure generation module makes use of order statistics to generate the hard bit failure times for the entire lifetime of the memory at the start of the simulation. The equation for the i th failure time, T i is given in (4), where F -1 (x) is the percent point function of the time to fail, U is a sequence of uniformly distributed random variables on the interval (0,1), n is the size of the memory, and r is the desired number of ordered failures to be generated [47]. rjijjnjiUFT,..,111111 (4) Each of the failure times is randomly associated with a failure type based on the probability distribution of failures in the type of memory being simulated. Each failure is also assigned to a random location in memory. The result is a list of failures with the location parameters for each of the levels defined by the number of the row or the column. A zero is used to represent an entire row, column, or device failure. This is illustrated in Figure 5-3.

PAGE 54

43 Index Type Fail Time Physical Location Level 2 Level 1 Level 0 Row Col Row Col Row Plane Col 1 B 755.73910 1 1 1 1 4 1 1 2 R 1587.40267 1 1 1 1 7 0 0 3 C 2619.78800 1 1 1 1 0 3 2 Figure 5-3. Sample failure generation output The occurrence of soft bit failures need not be simulated at all as they can be estimated from the number of hard uncorrectable errors in the system at each of the polling intervals. The equation for the SUE is given in (5), where is the SER, t is the time in hours, G is the number of good bits in words with the number of correctable bits in error, and B nec (t) is the number of words in the memory with the number of correctable bits in error at time t. The equation finds the probability that the soft errors in the system will affect one of the good bits in a word that already contains nec bad bits. tBGttSUEnec910)( (5) Simulation. The memory is simulated by iterating through each of the failures in the fail list and determining whether the fault tolerance mechanisms in the memory are sufficient to avoid uncorrectable errors of the data were to be accessed. This requires the interactions of the failures be determined to ascertain when and how the defects line up. It is also necessary during the simulation process to store information about the system. This is done in the data structure shown in Figure 5-4. The top level of the structure is a hash with the key determined by the level-2 row and level-2 column values. The only information stored at this level is the status of the level-2 device, such as whether it is a spare that has been swapped in or if it is has been swapped out and is considered dead. The next level consists of two hashes, the first has a key derived from the level-1 row and level-1 column and the other is just the level-1 row. In the case of the

PAGE 55

44 row and column, information local to each of the level-1 arrays is stored, which includes the status of the array, the number of spare rows and columns that have been used, the number of words with 1 to nec bad bits, and the number of uncorrectable errors. The entry that stores the start of the chain indexes the first failure in the fail list for the array and will be discussed in more detail shortly. The hash that is indexed by only the level-1 row is used only in the case that RAID parity is implemented across level-1 arrays and stores additional information necessary for calculating the number of HUEs. Figure 5-4. Hierarchical hash data structure used in the simulator The interaction of the failing bits is modeled by chaining them together by level-1 array. The chaining process, which is similar to that used in Libson and Harvey [48], avoids having to search through all previous failures in the fail list for each new failure. Chaining is done by first accessing the start of the chain in the simulation data structure to determine if there has been a previous failure in the same level-1 array as the current

PAGE 56

45 failure. If the current failure is the first one in the array, then its index is recorded as the start of the chain for the array and it is examined to see if it results in a HUE. If the current failure is not the first failure in the array, then the index for the first failure is retrieved and the two failures are checked to see if they line up to cause a HUE. The index for the current failure is also entered into the chain entry in the fail list at the first failure. If at a later time, another failure occurs in the same level-1 array, the index to the first failure is retrieved from the data structure and the second failure is retrieved by looking at the first failures chain entry. The current failures index is then inserted into the second failures chain entry and the three failures are checked for the possibility of lining up to cause a HUE. The process continues in this fashion for all failures with each only being processed with those in the same level-1 array without the need to ever scan the entire fail list. An example is given in Figure 5-5, which represents a somewhat simplified failure list. In this case, the memory is configured with eight rows with four words of four bits each at level-0. The level-1 is composed of an array of three rows and two columns of level-0 arrays. The level-2 has been reduced to a single instance for simplicity and all errors only affect a single bit. Highlighting has been used throughout the list to identify the level-1 array that each error affects. It has been repeated at the top of the figure to represent the start of the chain that would be stored in the simulators data structure. A chaining column has also been inserted to represent how the failures are connected. In this case, there is only a single case in which more than one error occurs in a word. It is in the level-1 array at the location (2,1) in row three and word four; it has been designated by circling and connecting the failures in the figure.

PAGE 57

46 Figure 5-5. Example of failure chaining process in simplified fail list. Highlighted chain shows two bits in the same word lining up to be in error. The simulation process becomes somewhat more complicated in the case that RAID redundancy is enabled across level-1 arrays. The first difference is in the determination of the HUEs within an array. In the previous case, the HUEs could be determined locally within each level-1 array. When RAID is enabled, however, the failures that line up within the arrays are tracked but not recorded as HUEs. Instead, there is a second chaining process that chains together the low-level failures by level-1 row. The chaining process used in the case of RAID is the same as that used for the bit-level errors except the failures are compared on a word level to see if multiple words are uncorrectable in the same relative position across a row of level-1 arrays. It is only when

PAGE 58

47 two uncorrectable words line up that the RAID parity cannot be used to regenerate the words in error and HUEs are recorded. Only the last failure to line up to cause a HUE in each word needs be chained together, and there are two HUEs for the first occurrence and then each subsequent word that lines up results in another single HUE. The HUE information is tracked in the hash structure indexed by the level-1 row and supersedes the information in the other hash when calculating the number of HUEs for the system. Memory state recording. The recording process logs the number of hard and soft uncorrectable errors in the memory. The result of the logging is an overview of the number of hard and soft uncorrectable errors present throughout the lifetime of the memory, as well as a log of the number and type of spares used. The logging is done by iterating through the data structure that maintains the number of hard uncorrectable errors in the system and calculating a total for the entire memory. It is also during this polling process that the number of soft uncorrectable errors is estimated using (5). When multiple iterations are run, which is necessary to obtain accurate results, statistical measures such as the mean, minimum, maximum, and standard deviation of the UE values is calculated and recorded. The goal for this thesis was to run 5,000 iterations of each simulation to assure statistical significance based on the law of large numbers [47, 48]. 5.2.3 Results Analysis The results of the simulations are derived from the log file kept during the polling process, as well as optional verbose logs kept during the simulation. The log file kept during the polling process gives values for the number of hard and soft uncorrectable errors in the system, and optionally the number and type of spares used. A portion of a sample log file is displayed in Figure 5-2. The name of the file is prefixed with "time_"

PAGE 59

48 followed by the values for the parameters that define the simulation and as described in Table 5-1. In the example shown, the first lines of the file give the parameter values for the memory in an easy-to-read format. This is followed by the results for the number of HUEs and SUEs in the system at every 1,000 hours up to the memory lifetime of 100,000 hours. The mean, minimum, maximum, and standard deviation are given for cases that the simulation is executed for multiple trials. When the graphical version of the simulator is used, it is possible to plot the statistical summaries in a box-plot format. 5.3 Simulator Validation It is not possible to obtain or derive analytical models for the proposed memory architecture model in its full complexity, so the validation of the simulator has been approached as follows. First, a memory configuration that implements ECC without sparing has been evaluated against an analytical model for memories with SEC [14]. The results in Table 5-2 are averaged over 5,000 trials and show small differences in the MTTF for small memory sizes for which the analytical model was initially used. Table 5-2. Simulator validation to SEC analytical model for small memory sizes Memory Size (b) Simulator MTTF Analytical MTTF Difference (%) 256x39 2.025x10 12 2.010x10 12 0.739 1024x39 1.014x10 12 1.000x10 12 1.37 256x72 1.107x10 12 1.100x10 12 0.657 1024x72 5.455x10 11 5.490x10 11 0.640 A comparison was also made against the analytical model when extended to larger memory sizes. These results are shown in Figure 5-6, and although there is a larger percentage difference in the results, it is clear that the simulator closely tracks the model.

PAGE 60

49 It should also be noted that the model was admittedly not entirely accurate for large memory sizes and should not be used for memories over 1-Gbit [14]. 0200400600800100012001400160018000102030405060Memory Size (Mb)MTTF (h) 70 Analytical 32-bit words Simulator 32-bit words Analytical 64-bit words Simulator 64-bit words Figure 5-6. Simulator validation to SEC analytical model for large memory sizes Configurations with ECC and sparing have also been evaluated though the analysis of simulation trace outputs and by comparison against a previously developed simulator that implemented different algorithms, data structures (event-driven vs. chaining), random number generation (per-bit Poisson distribution vs. ordered statistics), and programming language (C vs. Perl). The results for a 1-Mb memory configuration implementing SEC and DEC run in each of the simulators showed an average difference of 6.64 and 8.94%, respectively, over 5,000 trials. 5.4 Simulation Results The results shown in Figures 5-7 to 5-9 are from a set of experiments used to compare the performance of various fault tolerance mechanisms based on the percentage

PAGE 61

50 of bits used for redundancy. The simulations are based on multi-level memory organizations with 1-Gbit of usable memory size, which are composed of 1-Mbit modules at the lowest level. The results are the average of 5,000 trials and show the number of HUEs in the system after 100,000 hours (~11.4 years) of system life. It is immediately apparent that without the use of a Hamming ECC code, the performance is relatively poor, even if RAID and spare devices are used. It can be seen that for small word sizes SEC with spares has better performance than DEC initially, but the benefit is lost later in the memory lifetime as the spares are exhausted. It is also notable that the SEC implementations using RAID have a significant performance advantage for a given amount of overhead, although this is at the tradeoff in complexity and possible latency increase of implementing inter-block redundancy. The only configurations to protect the memory from all errors regardless of word size are DEC with a small number of spares, which requires a significant overhead. The next closest configuration in terms of performance is a combination of SEC with both RAID and spare devices implemented. Again, this configuration requires a high degree of overhead but it is less than that needed by DEC. This provides a strong case for the use of hierarchical fault tolerance mechanisms as a way of optimizing the reliability of devices at the lowest cost in terms of the percentage of devices used for redundancy. The synergism allows for multiple, simpler mechanisms to outperform more complex techniques.

PAGE 62

51 1.E+001.E+011.E+021.E+031.E+041.E+051.E+060100002000030000400005000060000700008000090000100000Time (hours)Hard Uncorrectable Errors NEC-RAID-spare-11% NEC-RAID-spare-22% SEC-11% SEC-spare-22% SEC-RAID-22% SEC-RAID-spare-22% DEC-22% DEC-spare-22.4% Figure 5-7. The HUEs for 1-Gbit memory with 64-bit word size using various percentages of fault tolerance 1.E+001.E+011.E+021.E+031.E+041.E+051.E+060100002000030000400005000060000700008000090000100000Time (hours)Hard Uncorrectable Errors SEC-6.25% SEC-spare-12.5% SEC-RAID-12.5% SEC-RAID-spare-12.5% DEC-12.5% DEC-spare-16% Figure 5-8. The HUEs for 1-Gbit memory with 128-bit word size using various percentages of fault tolerance

PAGE 63

52 1.E+001.E+011.E+021.E+031.E+041.E+051.E+060100002000030000400005000060000700008000090000100000Time (hours)Hard Uncorrectable Errors NEC-RAID-spare-7% SEC-3.5% SEC-spare-7% SEC-RAID-7% SEC-RAID-spare-7% DEC-7% DEC-spare-12.5% Figure 5-9. The HUEs for 1-Gbit memory with 256-bit word size, using various percentages of fault tolerance

PAGE 64

CHAPTER 6 CONCLUSION AND FUTURE WORK 6.1 Conclusions Continued advances in the fabrication of molecular structures have led to speculation that molecular electronic devices may be incorporated with or even supplant CMOS technology in the realm of ultra-dense logic and memory systems. These systems will be manufactured using self-assembly techniques, which will require entirely new methods of design, simulation, fabrication, and testing. One of the consequences of the self-assembly process is an increased hard error rate due to physical defects and soft error rate from an enhanced sensitivity to noise and radiation effects. In this thesis a generic, hierarchical fault tolerance architecture has been proposed as a method of dealing with the increased error rates. The architecture incorporates many of the attributes predicted to be inherent in molecular electronics, such as a hierarchical design, randomness in device assembly and interface, and the need for locality of communication. The key concepts from this architecture are as follows: Reuse at runtime the logic circuitry necessary for static address mapping Perform dynamic reconfiguration at multiple levels in a localized, hierarchical fashion Incorporate intraand inter-device ECCs with reconfiguration The performance of the architecture has been modeled against existing architectures through a specially constructed simulation tool. The results of the analysis 53

PAGE 65

54 show that there is an opportunity for the advancement of fault tolerance techniques through the synergistic effect of multiple, hierarchical mechanisms. 6.2 Future Work There are a number of possibilities for future work in this area molecular electronics. One of the first steps that must be accomplished is the development of accurate models from the device level through the architectural level of abstraction. As models are developed and validated by empirical data, then the requirements in terms of fault tolerance can be fully understood. There are some additional metrics necessary to completely model the performance of these new fault tolerance mechanisms that will only become available with the fabrication of larger systems and the development of accurate models. One of the most important is the overhead of the mechanisms in terms of latency. Unless molecular electronics ushers in a major shift in the computer architecture paradigm, this will remain a critical factor as there is an ever increasing performance gap between memory and logic devices [20]. It will also be necessary to examine more complex fault models than are currently implemented in the simulator. Examples include fault clustering [49], infant mortality, and vintage learning [47, 48]. It is also useful to model the devices from a software perspective. If the faults occur in memory but are never accessed or are overwritten with new data before being accessed, then they are not observed as errors [50]. There are several other forms of fault tolerance that have not been covered in this thesis that may prove to be beneficial. One of the most researched is based on the redundancy methods originally proposed by von Neumann. In his work, von Neumann suggested a multiplexing technique in which each wire is replaced with a bundle of wires

PAGE 66

55 and a majority vote is used to determine the value of the wire [6, 7]. This concept has been expanded upon to allow for multiple, redundant components in what is termed R-fold modular redundancy A special case is Triple Modular Redundancy (TMR), which uses three redundant components [7]. Again, a majority vote is used to determine the final value. Other techniques involve the use of asynchronous cellular arrays [51], fault alignment exclusion [52], error masking [53], and layout modification [54]. Another avenue of exploration is the area of defect tolerance, or the ability to create functional systems in the presence of defects. The location and type of all the defects in the system is determined through a testing process and the system is designed to avoid using those devices. The most famous implementation of defect tolerance is the HP Teramac supercomputer, which is fully functional with a 3% defect rate [41, 42]. The logic for the entire computer system is encoded in memory and functions through the use of simple look-up tables (LUT). The LUTs are connected hierarchically in a fat-tree topology. The fat-tree allows for a very high communication bandwidth and gives the compiler the ability to easily map a wide variety of custom architectures onto the system, including a BIST architecture that can map around defects.

PAGE 67

APPENDIX A MODERN MEMORY DEVICES 56

PAGE 68

APPENDIX B SIMULATOR SOURCE CODE B.1 Initialization and Variable Declaration use Cwd; #provide cross-platform file access use Fcntl; use GD::Graph::boxplot; use Math::Random qw(random_uniform random_uniform_integer); use Math::Round qw(nhimult nearest); use Statistics::Descriptive; use strict; use Tk; #use Tk GUI tools use Tk::BrowseEntry; #use Tk Combobox widgets use Tk::Dialog; #use dialog boxes use Tk::NoteBook; #make use of tabbed notebook widget use Tk::TextUndo; #use textboxes with undo/redo ########################################################## # Variable declaration ########################################################## my ($currentPage, $execCnt, $fileNum, $fName); my ($b1Acc, $b2Acc, $crAcc, $raAcc, $sueAcc, $ueAcc); my (@fileName, @files_t); my (%pageName); # Common variables for simulator my ($her, $kVal, $L0Col, $L0Plane, $L0Row, $L0RedCol); my ($L0RedRow, $L1Col, $L1Key, $L2Key, $L1Red, $L1Row); my ($L2Col, $L2Red, $L2Row, $lifetime, $maxTrials); my ($nec, $path, $pollInterval, $raid, $rVal, $ser); my ($verbose); # Variables reset for each simulation my (%SST); #used to track array status my ($count); #counts number of failures generated my (@fails); #array with all failure info # Statistic arrays for determining when error rates # are calculated to desired degree of accuracy my (@herL0StatData, @serL0StatData); my (@herL1StatData, @serL1StatData); my (@herL0PlotData, @serL0PlotData); my (@herL1PlotData, @serL1PlotData); $path = cwd; # End variable declaration 57

PAGE 69

58 # Default values $L2Row = 1; #rows of L2 devices (cards) $L2Col = 1; #columns of L2 devices (cards) $L2Red = 0; #number of spare L2 devices (cards) $L1Row = 8; #rows of L1 devices (chips) $L1Col = 8; #columns of L1 devices (chips) $L1Red = 0; #number of spare L1 devices (chips) $L0Row = 1024; #rows of L0 devices per plane (cells) $kVal = 128; #cols of L0 devices per plane (word size in bits) $L0Plane = 8; #number of words the L0 arrays are split in to $L0RedRow = 0; #number of spare rows of L0 devices (cells) $L0RedCol = 0; #number of spare columns of L0 devices (cells) $nec = 1; #how many bits are correctable with ECC $raid = 0; #extra col of arrays added for RAID across blocks $lifetime = 1e5; #how much system life should be generated $pollInterval = 1e3; #freq the state of the memory is taken $her = 2; #hard error rate in FIT/bit $ser = 100; #soft error rate in FIT/bit $maxTrials = 1; $execCnt = 0; #number of complete executions that have completed # Determine number of bits used for error correcting codes # 'k' data bits and 'r' check bits are calculated $rVal = $nec (log($kVal)/log(2) + 1); # add one more check bit for RAID with NEC or SEC ($nec <= 2 and $raid ? $rVal += 1: 0); $L0Col = $kVal + $rVal; # Total memory size is size of all usable and spare memory my $L0UsableSize = $kVal $L0Row $L0Plane; my $L0ECCSize = $rVal $L0Row $L0Plane; my $L0SpareSize = ($L0RedRow $L0Col $L0Plane) + ($L0RedCol $L0Row); my $L0MemSize = $L0UsableSize + $L0ECCSize + $L0SpareSize; my $L1UsableSize = $L0UsableSize $L1Row $L1Col; my $L1ECCSize = ($L0ECCSize $L1Row $L1Col) + ($raid $L0MemSize $L1Row); my $L1SpareSize = ($L0SpareSize $L1Row $L1Col) + ($L0MemSize $L1Red); my $L1MemSize = $L1UsableSize + $L1ECCSize + $L1SpareSize; my $totalUsableSize = $L1UsableSize $L2Row $L2Col; my $totalECCSize = $L1ECCSize $L2Row $L2Col; my $totalSpareSize = ($L1SpareSize $L2Row $L2Col) + ($L1MemSize $L2Red); my $totalMemSize = $totalUsableSize + $totalECCSize + $totalSpareSize; #Calculate MTTF from hard and soft errors my $herMTTFMem = nearest(0.001, (1e9 / ($her $totalMemSize))); my $serMTTFMem = nearest(0.001, (1e9 / ($ser $totalMemSize))); my $herMTTFBit = 1e9 / $her; $verbose = 0;

PAGE 70

59 B.2 Graphical User Interface # Toplevel window initialization for GUI my $top_w = MainWindow->new(); #create new toplevel my $sw = $top_w->screenwidth; my $sh = $top_w->screenheight; #center toplevel on screen $top_w->geometry("965x650+" int($sw/2 965/2) "+" int($sh/2 675/2)); $top_w->title("Fault Tolerant Memory Simulator"); #title it $top_w->raise(); my $top_f = $top_w->Frame(); ## Create frames for top and bottom tabbed notebooks #the top one will contain open files and the bottom #will contain coverage metrics and other information #also create frame for script selection buttons my $files_f = $top_f->Frame(-relief => "raised", -borderwidth => 2) ->pack(-fill => 'both', -expand => 1); my $main_f = $top_f->Frame(-relief => "raised", -borderwidth => 2) ->pack(-fill => 'both'); ##Create the top tabbed notebook my $files_nb = $files_f->NoteBook() ->pack(-fill => 'both', -expand => 1); ##Create the bottom tabbed notebook my $main_nb = $main_f->NoteBook() ->pack(-fill => 'both', -expand => 1); ##Create pages for information about settings and batches my $archSettings_pg = $main_nb ->add(0,-label => 'Architecture Settings'); my $simSettings_pg = $main_nb ->add(1, -label => 'Simulation Settings'); my $summSettings_pg = $main_nb ->add(2, -label => 'Summary Settings'); ## Create the frames for the "Main" settings ############################################ # Architecture settings frames my $archSettings_f = $archSettings_pg->Frame(-relief => "groove", -borderwidth => 2) ->pack(-fill => 'both', -expand => 1); my $archSettingsLeft_f = $archSettings_f->Frame() ->pack(-side=> 'left', -fill => 'both', -expand => 1); my $archSettingsMiddle_f = $archSettings_f->Frame(-relief =>'groove', -borderwidth => 2) ->pack(-side=> 'left', -fill => 'both', -expand => 1); my $archSettingsRight_f = $archSettings_f->Frame() ->pack(-side=> 'left', -fill => 'both', -expand => 1); my $dimenTop_f = $archSettingsLeft_f->Frame(-relief =>'groove', -borderwidth => 2) ->pack(-side => 'top', -fill => 'both', -expand => 1);

PAGE 71

60 my $plane_f = $dimenTop_f->Frame() ->pack(-fill => 'both', -side => 'bottom'); my $colSize_f = $dimenTop_f->Frame(-label => '# Cols') ->pack(-fill => 'both', -expand => 1, -side => 'right'); my $rowSize_f = $dimenTop_f->Frame(-label => # Rows') ->pack(-fill => 'both', -expand => 1, -side => 'left'); my $L2Col_f = $colSize_f->Frame() ->pack(-fill => 'both', -expand => 1, -side => 'top'); my $L1Col_f = $colSize_f->Frame() ->pack(-fill => 'both', -expand => 1, -side => 'top'); my $L0Col_f = $colSize_f->Frame() ->pack(-fill => 'both', -expand => 1, -side => 'top'); my $L2Row_f = $rowSize_f->Frame() ->pack(-fill => 'both', -expand => 1, -side => 'top'); my $L1Row_f = $rowSize_f->Frame() ->pack(-fill => 'both', -expand => 1, -side => 'top'); my $L0Row_f = $rowSize_f->Frame() ->pack(-fill => 'both', -expand => 1, -side => 'top'); my $redunL2_f = $archSettingsMiddle_f->Frame() ->pack(-fill => 'both', -expand => 1, -side => 'top'); my $redunL1_f = $archSettingsMiddle_f->Frame() ->pack(-fill => 'both', -expand => 1, -side => 'top'); my $redunL0Row_f = $archSettingsMiddle_f->Frame() ->pack(-fill => 'both', -expand => 1, -side => 'top'); my $redunL0Col_f = $archSettingsMiddle_f->Frame() ->pack(-fill => 'both', -expand => 1, -side => 'top'); my $memSpecs_f = $archSettingsRight_f->Frame() ->pack(-fill => 'both', -expand => 1); my $memSpecsLeft_f = $memSpecs_f->Frame() ->pack(-side => 'left', -fill => 'both', -expand => 1); my $memSpecsRight_f = $memSpecs_f->Frame() ->pack(-side => 'right', -fill => 'both', -expand => 1); my $errorRate_f = $memSpecsLeft_f->Frame(-relief =>'groove', -borderwidth => 2) ->pack(-side => 'top', -fill => 'both', -expand => 1); my $ecc_f = $memSpecsLeft_f->Frame(-relief => 'groove', -borderwidth => 2, -label => 'Error Correcting Code') ->pack(-side => 'top', -fill => 'both', -expand => 1); my $info_f = $memSpecsLeft_f->Frame() ->pack(-side => 'top', -fill => 'both', -expand => 1); my $algoSel_f = $memSpecsRight_f->Frame(-relief => 'groove', -borderwidth => 2, -label => 'Other Options') ->pack(-side => 'top', -fill => 'both', -expand => 1); my $memSize_f = $memSpecsRight_f->Frame(-relief => 'groove', -borderwidth => 2, -label => 'Memory Size') ->pack(-side => 'top', -fill => 'both', -expand => 1); my $ser_f = $errorRate_f->Frame(-label => 'Soft Error Rate') ->pack(-side=> 'left', -fill => 'both', -expand => 1);

PAGE 72

61 my $her_f = $errorRate_f->Frame(-label => 'Hard Error Rate') ->pack(-side=> 'right', -fill => 'both', -expand => 1); my $mttf_f = $info_f->Frame(-label => 'MTTF(h)', -relief => 'groove', -borderwidth => 2) ->pack(-side=> 'left', -fill => 'both', -expand => 1); my $chkBits_f = $info_f->Frame(-label => 'Word Size', -relief => 'groove', -borderwidth => 2) ->pack(-side=> 'right', -fill => 'both', -expand => 1); # Create updating text box with memory size info my $spareSizePrnt = commify($totalSpareSize); my $memSizePrnt = commify($totalMemSize); my $usableSizePrnt = commify($totalUsableSize); my $eccSizePrnt = commify($totalECCSize); my $herMTTFMemPrnt = commify($herMTTFMem); my $serMTTFMemPrnt = commify($serMTTFMem); my $overheadPrnt = commify(nearest(0.01,($totalMemSize $totalUsableSize) / $totalUsableSize 100)); my $memSize_l = $memSize_f->Label(-text => "Total: $memSizePrnt bits\n Spare: $spareSizePrnt bits\n ECC: $eccSizePrnt bits\n Usable: $usableSizePrnt bits\n Overhead: $overheadPrnt%", -width => 30)->pack(); my $mttf_l = $mttf_f->Label(-text => "H: $herMTTFMemPrnt\n S: $serMTTFMemPrnt", -width => 15)->pack(); my $chkBits_l = $chkBits_f->Label(-text => "Data: $kVal\nCheck: $rVal", -width => 15)->pack(); # Create comboboxes my @L2Choices = qw(1 2 4 8 16 32 64 128 256 512); my @L1Choices = qw(1 2 4 8 12 16 22 32 42 46 64 128 256 512 1024); my @L0RChoices = qw(32 64 128 256 512 1024 2048 4096 8192); my @L0CChoices = qw(32 64 128 256 512); my @L0PlaneChoices = qw(1 2 4 8 16 32 64); my $L2_l = $L2Row_f->Label(-text => Level 2:") ->pack(-side => 'left'); my $L2Col_cb = $L2Col_f->BrowseEntry(-listwidth => 25, -variable => \$L2Col, -choices => \@L2Choices, -width => 8, -browsecmd => \&updateMemSize) ->pack(-side => 'right', -fill => 'both', -padx => 5); my $L2Row_cb = $L2Row_f->BrowseEntry(-listwidth => 25, -variable => \$L2Row, -choices => \@L2Choices, -width => 8, -browsecmd => \&updateMemSize) ->pack(-side => 'right', -fill => 'both', -padx => 5); my $L1_l = $L1Row_f->Label(-text => Level 1:") ->pack(-side => 'left');

PAGE 73

62 my $L1Col_cb = $L1Col_f->BrowseEntry(-listwidth => 25, -variable => \$L1Col, -choices => \@L1Choices, -width => 8, -browsecmd => \&updateMemSize) ->pack(-side => 'right', -fill => 'both', -padx => 5); my $L1Row_cb = $L1Row_f->BrowseEntry(-listwidth => 25, -variable => \$L1Row, -choices => \@L1Choices, -width => 8, -browsecmd => \&updateMemSize) ->pack(-side => 'right', -fill => 'both', -padx => 5); my $L0_l = $L0Row_f->Label(-text => Level 0:") ->pack(-side => 'left'); my $L0Col_cb = $L0Col_f->BrowseEntry(-listwidth => 25, -variable => \$kVal, -choices => \@L0CChoices, -width => 8, -browsecmd => \&updateMemSize) ->pack(-side => 'right', -fill => 'both', -padx => 5); my $L0Row_cb = $L0Row_f->BrowseEntry(-listwidth => 25, -variable => \$L0Row, -choices => \@L0RChoices, -width => 8, -browsecmd => \&updateMemSize) ->pack(-side => 'right', -fill => 'both', -padx => 5); my $L0P_l = $plane_f->Label(-text => Planes:") ->pack(-side => 'left'); my $L0Plane_cb = $plane_f->BrowseEntry(-listwidth => 25, -variable => \$L0Plane, -choices => \@L0PlaneChoices, -width => 8, -browsecmd => \&updateMemSize) ->pack(-side => 'bottom', -fill => 'y', -padx => 5); my @redunChoices = qw(0 1 2 4 8 16 32 64 104 128 256 512 1024 2048 4096 8192 16384); my $redunL2_l = $redunL2_f->Label(-text => Redundant L2: ") ->pack(-side => 'left'); my $redunL2_cb = $redunL2_f->BrowseEntry(-listwidth => 25, -variable => \$L2Red, -choices => \@redunChoices, -width => 8, -browsecmd => \&updateMemSize) ->pack(-side => 'right', -fill => 'both', -padx => 20); my $redunL1_l = $redunL1_f->Label(-text => Redundant L1: ") ->pack(-side => 'left'); my $redunL1_cb = $redunL1_f->BrowseEntry(-listwidth => 25, -variable => \$L1Red, -choices => \@redunChoices, -width => 8, -browsecmd => \&updateMemSize) ->pack(-side => 'right', -fill => 'both', -padx => 20); my $redunL0Row_l = $redunL0Row_f->Label(-text => Redundant L0 Rows: ") ->pack(-side => 'left');

PAGE 74

63 my $redunL0Row_cb = $redunL0Row_f->BrowseEntry(-listwidth => 25, -variable => \$L0RedRow, -choices => \@redunChoices, -width => 8, -browsecmd => \&updateMemSize) ->pack(-side => 'right', -fill => 'both', -padx => 20); my $redunL0Col_l = $redunL0Col_f->Label(-text => "Redundant L0 Cols: ") ->pack(-side => 'left'); my $redunL0Col_cb = $redunL0Col_f->BrowseEntry(-listwidth => 25, -variable => \$L0RedCol, -choices => \@redunChoices, -width => 8, -browsecmd => \&updateMemSize) ->pack(-side => 'right', -fill => 'both', -padx => 20); # Create HER/SER entry boxes my $ser_en = $ser_f->Entry(-validate => 'focusout', -validatecommand => \&updateMemSize, -textvariable => \$ser, -justify => 'right', -width => 10) ->pack(-padx => 20, -pady => 9); my $her_en = $her_f->Entry(-validate => 'focusout', -validatecommand => \&updateMemSize, -textvariable => \$her, -justify => 'right', -textvariable => \$her, -width => 10, ) ->pack(-padx => 20, -pady => 9); # Create error correction selection buttons my $eccNEC = $ecc_f->Radiobutton(-text => 'None', -value => 0, -variable => \$nec, -command => \&updateMemSize) ->pack(-side => 'left', -padx => 5); my $eccSEC = $ecc_f->Radiobutton(-text => 'SEC', -value => 1, -variable => \$nec, -command => \&updateMemSize) ->pack(-side => 'left', -padx => 5); my $eccDEC = $ecc_f->Radiobutton(-text => 'DEC', -value => 2, -variable => \$nec, -command => \&updateMemSize) ->pack(-side => 'left', -padx => 5); # Create fault tolerance algorithm selection buttons my $L1ECCSel = $algoSel_f->Checkbutton(-text => 'L1 ECC (RAID)', -variable => \$raid, -command => \&updateMemSize) ->pack(-side => 'left', -padx => 5); my $verbSel = $algoSel_f->Checkbutton(-text => 'Verbose', -variable => \$verbose) ->pack(-side => 'left', -padx => 5);

PAGE 75

64 # Simulation settings frames my $simSettings_f = $simSettings_pg->Frame(-relief => 'groove', -borderwidth => 2) ->pack(-fill => 'both', -expand => 1); my $simSettingsLeft_f = $simSettings_f->Frame(-label => 'Time (h)', -relief => 'groove', -borderwidth => 2) ->pack(-side=> 'left', -fill => 'both', -expand => 1); my $simSettingsRight_f = $simSettings_f->Frame(-label => 'Execute', -relief =>'groove', -borderwidth => 2) ->pack(-side=> 'left', -fill => 'both', -expand => 1); my $pollTime_f = $simSettingsLeft_f->Frame(-relief => 'groove', -borderwidth => 2) ->pack(-fill => 'both', -expand => 1); my $endTime_f = $simSettingsLeft_f->Frame(-relief => 'groove', -borderwidth => 2) ->pack(-fill => 'both', -expand => 1); my $executeSim_f = $simSettingsRight_f->Frame(-relief =>'groove', -borderwidth => 2) ->pack(-fill => 'both', -expand => 1); my $maxTrials_f = $executeSim_f->Frame() ->pack(-fill => 'x', -expand => 1); my $pollTime_l = $pollTime_f->Label(-text => "Polling Interval(h): ") ->pack(-side => 'left'); my $pollTime_en = $pollTime_f->Entry(-justify => 'right', -textvariable => \$pollInterval, -width => 10) ->pack(-side => 'left'); my $endTime_l = $endTime_f->Label(-text => "Max Run Time(h): ") ->pack(-side => 'left'); my $endTime_en = $endTime_f->Entry(-justify => 'right', -textvariable => \$lifetime, -width => 10) ->pack(-side => 'left'); my $maxTrials_l = $maxTrials_f->Label(-text => "Maximum Trials: ") ->pack(-side => 'left'); my $maxTrials_en = $maxTrials_f->Entry(-justify => 'right', -textvariable => \$maxTrials, -width => 10) ->pack(-side => 'left', -pady => 5); my $executeSim_b = $executeSim_f->Button(-text => 'Execute 0', -width => 20, -command => \&runSimulation) ->pack(-pady => 5); my $plotSim_b = $executeSim_f->Button(-text => 'Plot', -width => 20, -command => \&plotResults) ->pack(-pady => 5); # Summary settings frames my $summSettings_f = $summSettings_pg->Frame(-relief => "raised", -borderwidth => 2) ->pack(-fill => 'both', -expand => 1);

PAGE 76

65 my $summSettingsLeft_f = $summSettings_f->Frame(-relief => 'groove', -borderwidth => 2) ->pack(-side=> 'left', -fill => 'both', -expand => 1); my $summSettingsMiddle_f = $summSettings_f->Frame(-relief => 'groove', -borderwidth => 2) ->pack(-side=> 'left', -fill => 'both', -expand => 1); my $summSettingsRight_f = $summSettings_f->Frame(-relief =>'groove', -borderwidth => 2) ->pack(-side=> 'left', -fill => 'both', -expand => 1); # Create menubar $top_w->configure(-menu => my $menubar_mb = $top_w->Menu); # Create menu items in menubar map {$menubar_mb->cascade(-label => '~' $_->[0], -menuitems => $_->[1])} ['File', &file_menuitems], ['Edit', &edit_menuitems], ['Search', &search_menuitems], ['Help', &help_menuitems]; ## Pack all main frames to toplevel $top_f->pack(-side=>'top', -fill=>'both', -expand=>1); $files_f->pack(-side => "top", -fill => 'both', -expand => 1); $archSettings_f->pack(-side=>"top", -fill=>'both'); MainLoop(); #Display Tk widgets B.3 Simulation Subroutines B.3.1 Generate Failures sub generate { # Variable declarations my ($failType, $failL0Col, $failL0Plane, $failL0Row, $failL1Col); my ($failL1Row, $failL1RowCalc, $failL2Col, $failL2Row); my ($failL2RowCalc, $index, $inverse, $minTime, $power); my ($rand, $Uprod); $Uprod = 1; #initialize Uprod to 1 since it will be mult by power while ($minTime <= $lifetime) { #use IBM FTMS equations to calculate failure times in order $count += 1; $rand = random_uniform(); $power = $rand ** (1 / ($totalMemSize $count + 1)); $Uprod *= $power; $inverse = -1 $herMTTFBit log($Uprod); $minTime = $inverse; ##determine type and location of each failure #generation of fail type #$rand = random_uniform();

PAGE 77

66 if ($rand < 0.35) { $failType = 'B'; $failL0Row = random_uniform_integer(1, 1, ($L0Row + $L0RedRow)); #if error is in spare row, set the flag if ($failL0Row > $L0Row) { #failure can't be in both a spare row and a spare col $failL0Col = random_uniform_integer(1, 1, $L0Col); $failL0Plane = random_uniform_integer(1, 1, $L0Plane); } } elsif ($rand < 0.67) { $failType = 'R'; $failL0Col = 0; $failL0Plane = 0; $failL0Row = random_uniform_integer(1, 1, ($L0Row + $L0RedRow)); } elsif ($rand < 0.85) { $failType = 'C'; $failL0Row = 0; #choose one of the cols including the spares, then determine #col/plane recieved the error-spares are in "extra" plane $failL0Col = random_uniform_integer(1, 1, (($L0Col $L0Plane) + $L0RedCol)); $failL0Plane = nhimult(1, $failL0Col / $L0Col); $failL0Col = $failL0Col % $L0Col; } else { $failType = 'A'; $failL0Row = 0; $failL0Col = 0; } $failL1Row = random_uniform_integer(1, 1, $L1Row); $failL1Col = random_uniform_integer(1, 1, ($L1Col + $raid)); $failL2Row = random_uniform_integer(1, 1, $L2Row); $failL2Col = random_uniform_integer(1, 1, $L2Col); @{$fails[$count]} = ($count, $failType, $minTime, $failL2Row, $failL2Col, $failL1Row, $failL1Col, $failL0Row, $failL0Plane, $failL0Col); } print "COUNT: $count\n"; }

PAGE 78

67 B.3.2 Simulate #--------------------------------------------------------------------# # KEY TO LABELS: # 'A' => Label for L0 CHIP (entire array) error # 'B' => Label for L0 BIT error # 'BD' => Track BAD COUNT for words with < NEC bad bits # 'C' => Label for L0 COLUMN error # 'CH' => CHAIN of L0 or L1 fails linking those in similar devices # 'DD' => Marks L0 array as DEAD # 'DE' => Track L0 rows that have been DEALLOCATED # 'R' => Label for L0 ROW error # 'SC' => Tracks number of SPARE COLUMNS used # 'SR' => Tracks number of SPARE ROWS used # 'ST' => STATUS information for array # 'UE' => Track count of UNCORRECTABLE ERRORS # 'UU' => Mark L0 spare array as UNUSED #--------------------------------------------------------------------# sub simulate { # Variable declarations my ($errorCnt, $failTime, $flag, $goodBits); my ($index, $key, $keyC, $keyR); my ($L0ColKey, $L0PlaneKey, $L0RowKey, $L1ColKey, $L1RowKey); my ($L0TempColKey, $L0TempPlaneKey, $L0TempRowKey, $L1TempColKey); my ($numWords, $redunSuccess, $tempIndex, $tempL1Key, $time); my ($totalL0SUE, $totalL1SUE, $totalL0UE, $totalL1UE); my (@colChain, @rowChain, @totalBadBit, @totalBadWord); my (%bitErrors, %wordErrors); # End variables #read fail info in from file and put in a 2D array $time = $pollInterval; #calculate parameters used in the estimation of number of SUEs #total number of L1 devices in memory $numWords = $L2Row $L2Col $L1Row $L1Col $L0Row $L0Plane; #fraction of good bits in words with NEC bad bits #these are the ones succeptable to errors $goodBits = $L0Col $nec; #go through each failure generated by order stats FAILLOOP: while ($index <= $count) { $index += 1; #go on to next fail in generated list $flag = 0; #set to only go through hash once $failTime = $fails[$index][2]; while ($time <= $failTime) { if ($flag == 0) { $flag = 1; #reset variables @totalBadBit = @totalBadWord = (); $totalL0UE = $totalL0SUE = 0; $totalL1UE = $totalL1SUE = 0;

PAGE 79

68 #go through all L0 arrays foreach $key (keys %SST) { #only count the errors if they are in the main memory: #they must not be marked 'dead' and must either not be from #a spare device or be a spare device that is marked 'used' if ($SST{$key}{'ST'} ne 'DD' and ($key !~ /S/ or $SST{$key}{'ST'} eq 'UU')) { #go through each row key and, if it is a number then #go through all corresponding columns associated with it foreach $keyR (keys %{$SST{$key}}) { if ($keyR =~ /\d+/) { #key is for L0 failure info if ($keyR =~ /-/) { if ($SST{$key}{$keyR}{'ST'} ne 'DD' and ($key !~ /S/ or $SST{$key}{$keyR}{'ST'} eq 'UU') ) { $totalBadBit[1] += $SST{$key}{$keyR}{'BD'}[1]; $totalBadBit[2] += $SST{$key}{$keyR}{'BD'}[2]; $totalL0UE += $SST{$key}{$keyR}{'UE'}; #print "totalL0UE: $totalL0UE\n"; } } #if RAID enabled, then key is for L1 failure info elsif ($raid) { #record total num rows with 2 to L0Col word failures #as well as the overall number of L1 (word) HUEs for my $i (2..($L1Col + 1)) { if ($SST{$key}{$keyR}{'UE'}[$i]) { $totalBadWord[$i] += $SST{$key}{$keyR}{'UE'}[$i]; $totalL1UE += $SST{$key}{$keyR}{'UE'}[$i] $i; } } } } } } } } #Calc SUE by multiplying SER by the num good bits in a word with #nec bad bits x total words w/ nec bad bits and finally by time #div by 1e9 to account for FIT given as fails in 1e9 bit-hours. if ($nec > 0) { $totalL0SUE = ($ser / 1e9 $time) $goodBits $totalBadBit[$nec]; }

PAGE 80

69 else { $totalL0SUE = ($ser / 1e9 $time) $goodBits $numWords; } #calculate L1 SER if RAID is enabled if ($raid) { #when RAID is enabled, the total L1SUE is the total number of #L0 soft errors multiplied by the ratio of words susceptible to #lining up to cause a word error this needs to be done for the #number of rows with k = 1..(L0Col -1) word errors the number #of words susceptible in each case is (L0Col k) and the case #of k == 1 needs to be mult. by 2 to account for two new UEs #the total number of rows with 1 UE in them multiplied by the #number of words susceptible a soft UE lining up with these #would result in two UEs $totalL1SUE = ($totalL0UE $totalL1UE) ($L1Col 1) 2; foreach my $i (2 .. ($L1Col 1)) { $totalL1SUE += $totalBadWord[$i] ($L1Col $i); } #currently have the total number of words susceptible to #a soft error lining up with them divide these by total #number of words to get a ratio and multiply by the total #number of soft errors in the system $totalL1SUE = ($totalL1SUE / $numWords) $totalL0SUE; } #add each failure amount to an array to be used later #in a statistical analysis of multiple trials push @{$serL0StatData[$time]}, $totalL0SUE; push @{$herL0StatData[$time]}, $totalL0UE; if ($raid) { push @{$serL1StatData[$time]}, $totalL1SUE; push @{$herL1StatData[$time]}, $totalL1UE; } $time += $pollInterval; } #generate key values from fail locations $L2Key = $fails[$index][3].'-'.$fails[$index][4]; $L1Key = $fails[$index][5].'-'.$fails[$index][6]; $L1RowKey = $fails[$index][5]; $L1ColKey = $fails[$index][6]; $L0RowKey = $fails[$index][7]; $L0PlaneKey = $fails[$index][8]; $L0ColKey = $fails[$index][9]; #Don't process failure if it is in a part of memory that has #already been killed or if it is in a L0 row that has been #deallocated if ($SST{$L2Key}{'ST'} eq 'DD' or $SST{$L2Key}{$L1Key}{'ST'} eq 'DD') { next FAILLOOP; }

PAGE 81

70 elsif ($L0RowKey > $L0Row) { #keep track of all spare rows that are no longer available $SST{$L2Key}{$L1Key}{'SR'} += 1; next FAILLOOP; } #check to see if failure is of a bit in an L1 Device (chip) elsif ($fails[$index][1] eq 'B') { #initialize bit errors to the current error %bitErrors = (); $bitErrors{$L0Col} = 1; #reset arrays to be used to track all bit errors in the same #row/col as the current one these failure indexes will be #disabled if the row/col is replaced with a spare. @colChain = (); @rowChain = (); #FIXME: put back in when spare columns used #push @colChain, $index; push @rowChain, $index; #check to see if key is present in system status table #the SST points to the start of each chain of bit failures if ($SST{$L2Key}{$L1Key}{'CH'} == 0) { #SST row entry was blank, so set SST row to current index value $SST{$L2Key}{$L1Key}{'CH'} = $index; } else { #look through previous failures in same L0 array matrix #set the index to the start of the chain $tempIndex = $SST{$L2Key}{$L1Key}{'CH'}; #cycle through all errors chained together while(1) { $L0TempRowKey = $fails[$tempIndex][7]; $L0TempPlaneKey = $fails[$tempIndex][8]; $L0TempColKey = $fails[$tempIndex][9]; #bit causes error if it is in the same row and plane but a #different column also check that the index has not been #disabled by swapping in a spare if ($fails[$tempIndex][12] != 1 and $L0RowKey == $L0TempRowKey) { #this is the same row as the current bit failure, so keep #track of it in case this row is replaced with a spare push @rowChain, $tempIndex; if ($L0PlaneKey == $L0TempPlaneKey) { #only count this error if it is a new error #and it is in a word that hasn't been deallocated; #otherwise go on to the next failure

PAGE 82

71 if ($L0ColKey == $L0TempColKey) { next FAILLOOP; } elsif ($fails[$tempIndex][12] == 2) { next FAILLOOP; } else { #this is the same plane as the current failure, as long #as it is a different col (i.e., not the same bit), it #lines up with another failure $bitErrors{$L0TempColKey} = 1; } } } #if failure entry in the chain is itself chained, then #continue with the next link if ($fails[$tempIndex][10] > 0) { #set the index variable to next fail index in the chain $tempIndex = $fails[$tempIndex][10]; } #otherwise, add the current failure to the chain else { $fails[$tempIndex][10] = $index; #last error for this row, so break from the while() loop last; } } } } #count the number of unique bit errors that lined up in the word $errorCnt = keys(%bitErrors); ($verbose > 1 ? print SIMFILE "ERRORCNT: $errorCnt\n" : 0); #if the number of errors is less than NEC, they won't be corrected #so they need to be tracked to see how they line up with soft #errors to create SUE's if ($errorCnt < $nec) { $SST{$L2Key}{$L1Key}{'BD'}[$errorCnt] += 1; #the next lowest value is reduced if ($nec > 1) { $SST{$L2Key}{$L1Key}{'BD'}[($errorCnt 1)] -= 1; } } #if the total number of bit errors in the row is equal to the #number that can be corrected, then need to replace this row with a #spare if possible elsif ($errorCnt == $nec) { #need to replace this row if possible #if length of array is less than total spares, then there is a #spare L0 row available

PAGE 83

72 if ($SST{$L2Key}{$L1Key}{'SR'} < $L0RedRow) { $SST{$L2Key}{$L1Key}{'SR'} += 1; #disable all index is current chain with the same row, #since this entire row was replaced foreach $keyR (@rowChain) { #set disable flag $fails[$keyR][12] = 1; } } #need to swap in a spare at the L1 level (chip) else { $SST{$L2Key}{$L1Key}{'BD'}[$errorCnt] += 1; #the next lowest value is reduced if ($nec > 1) { $SST{$L2Key}{$L1Key}{'BD'}[($errorCnt 1)] -= 1; } } } #if the number of errors is greater than the number that are #correctable, then there wasn't a spare available to replace the #device previously so UE's occur if ECC is not available across the #L1 blocks. If ECC is available, then need to track the words with #UE's to find when two of them line up to cause a L1 UE elsif ($errorCnt > $nec) { #if RAID not implemented, a UE occurs $SST{$L2Key}{$L1Key}{'UE'} += 1; #deallocate this word if another bit error occurs in this word, #then it will see this word in the chain and know to go on to the #next error $fails[$index][12] = 2; if (!$raid) { #if error correction is avail., then all spares have been used #and any errors are permanent UE's the L0 rows containing these #errors will be deallocated if ($nec) { ##deallocate row #$SST{$L2Key}{$L1Key}{'DE'}{$L0RowKey} = 1; ##record the uncorrectable error } else { #if no error correction, then make use of spares even though #a UE occurs this will reduce the amount of deallocation that #occurs. ECC allows for spares to be used up before #bitErrors > nec if length of array is less than total #spares, then there is a spare L0 row available

PAGE 84

73 if ($SST{$L2Key}{$L1Key}{'SR'} < $L0RedRow) { $SST{$L2Key}{$L1Key}{'SR'} += 1; #disable all index is current chain with the same row, #since this entire row was replaced foreach $keyR (@rowChain) { #set disable flag $fails[$keyR][12] = 1; } } } } else { #if RAID ECC is implemented across blocks, then for a given L1 #row, go through L1 Col's. Then for each of these L1 arrays, #look through all other errors that were uncorrectable at the #L0 level. If any of these are in the same L0 row as the #current failure, then an L1 uncorrectable error occurs. #if no previous L0 UE's in this L1 row, insert current index #value and go on if($SST{$L2Key}{$L1RowKey}{'CH'} == 0) { $SST{$L2Key}{$L1RowKey}{'CH'} = $index; } else { #reset array each time chain is traversed %wordErrors = (); #record current error $wordErrors{$L1ColKey} = 1; #set tempIndex to the index of the first failure that caused #an L0 UE. This index is used to determine which word #(plane) had the failure. $tempIndex = $SST{$L2Key}{$L1RowKey}{'CH'}; while(1) { $tempL1Key = $fails[$tempIndex][5].'-'. $fails[$tempIndex][6]; $L1TempColKey = $fails[$tempIndex][6]; $L0TempRowKey = $fails[$tempIndex][7]; $L0TempPlaneKey = $fails[$tempIndex][8]; #word failure must be in the same L0 row and L0 plane #(aka relative word location) #and be in a different L1 device if ($L0RowKey == $L0TempRowKey and $L0PlaneKey == $L0TempPlaneKey) { if ($L1ColKey != $L1TempColKey) {

PAGE 85

74 if ($SST{$L2Key}{$tempL1Key}{'ST'} ne 'DD') { #if L1 chip status is not dead, then count it $wordErrors{$L1TempColKey} = 1; ($verbose > 1 ? print SIMFILE "WORD ERROR IN SAME RELATIVE LOCATION IN DIFFERENT WORD!\n" : 0); } else { ($verbose > 1 ? print SIMFILE "NO ERROR\n" : 0); } } else { #current error is in the same word as a previous error #don't chain this failure; just go on to next failure next FAILLOOP; } } else { ($verbose > 1 ? print SIMFILE "NO ERROR\n" : 0); } #if failure entry in the chain2 is itself chained, #then continue with the next link if ($fails[$tempIndex][11] > 0) { #set the index variable to next fail index in the chain2 $tempIndex = $fails[$tempIndex][11]; } #otherwise, add the current failure to the chain else { $fails[$tempIndex][11] = $index; #last error in current chain2, so break from loop last; } } #if there was an uncorrectable word error, then record it $errorCnt = keys (%wordErrors); ($verbose ? print SIMFILE "ERROR CNT: $errorCnt\n" : 0); #if two or more errors, then that num of UEs occur if ($errorCnt >= 2) { $SST{$L2Key}{$L1RowKey}{'UE'}[$errorCnt] += 1; #the next lowest value is reduced if ($errorCnt > 2) { $SST{$L2Key}{$L1RowKey}{'UE'}[($errorCnt 1)] -= 1; } } } } } } }

PAGE 86

75 B.3.3 Device Redundancy sub deviceRedundancy { #check all the possible spares for each key and find the first one #that isn't swapped in or marked for being bad from a chip failure foreach my $i (1 .. $L1Red) { #if a spare is found that is usable, mark it and the current #bad L1 device (chip) as swapped if ($SST{$L2Key}{"S-$i"}{'ST'} ne 'UU') { $SST{$L2Key}{"S-$i"}{'ST'} = 'UU'; $SST{$L2Key}{$L1Key}{'ST'} = 'DD'; # probably redundant #return success at swapping in spare L1 device return 1; } } #need to swap in spare at a higher level (entire L2 card) #in future, might let errors build up before doing this foreach my $i (1 .. $L2Red) { #if a spare is found that is usable, mark it #and the current bad L2 device (card) as swapped if ($SST{"S-$i"}{'ST'} ne 'UU') { $SST{"S-$i"}{'ST'} = 'UU'; $SST{$L2Key}{'ST'} = 'DD'; #return success at swapping in spare L2 device return 2; } } return 0; } B.3.4 Run Simulation sub runSimulation { my ($diffTime1, $diffTime2, $endTime1, $endTime2, $startTime); #variables for calculating accuracy #my ($acc, $average, $stdev); my $filename = join '-', $L2Row, $L2Col, $L2Red, $L1Row, $L1Col, $L1Red, $L0Row, $L0Col, $L0Plane, $L0RedRow, $L0RedCol, $nec, $raid, $lifetime, $pollInterval, $her, $ser, $maxTrials; updateMemSize(); #make sure all parameters are updated #insert commas intoall numbers before printing them out my $spareSizePrnt = commify($totalSpareSize); my $memSizePrnt = commify($totalMemSize); my $usableSizePrnt = commify($totalUsableSize); my $eccSizePrnt = commify($totalECCSize); $herMTTFMemPrnt = commify($herMTTFMem); $serMTTFMemPrnt = commify($serMTTFMem);

PAGE 87

76 #reset variables on each execution @herL0StatData = (); @serL0StatData = (); @herL1StatData = (); @serL1StatData = (); foreach my $i (1..$maxTrials) { $startTime = time(); #reset variables %SST = (); $count = 0; #counts number of failures generated @fails = (); #array with all failure info #generate all failure times and locations sysopen(GENFILE, "gen_$filename.out", O_RDWR | O_TRUNC | O_CREAT) or die "Can not open the file gen.out in $path\n"; generate(); #record end time to calc amount of time necessary to generate fails $endTime1 = time(); #output for first simulation if verbose variable set selected #verbose==1 gives one level of output #verbose==2 gives more detailed if ($verbose and $i == 1) { sysopen(SIMFILE, "sim_$filename.out", O_RDWR | O_TRUNC | O_CREAT) or die "Can not open the file sim_$filename.out in $path\n"; } simulate(); if ($verbose and $i == 1) { close SIMFILE or die "Closing: $!"; } close GENFILE or die "Closing: $!"; } #at given intervals print current failure data to a file sysopen(TIMEFILE, "time_$filename.out", O_RDWR | O_TRUNC | O_CREAT) or die "Can not open the file time_$filename.out in $path\n"; print TIMEFILE "L2Row: $L2Row\tL2Col: $L2Col\tL2Red: $L2Red\n"; print TIMEFILE "L1Row: $L1Row\tL1Col: $L1Col\tL1Red: $L1Red\n"; print TIMEFILE "L0Row: $L0Row\tL0Plane: $L0Plane\tL0Col: $kVal\t"; print TIMEFILE "L0RRow: $L0RedRow\tL0RCol: $L0RedCol\n"; print TIMEFILE "MEM SIZE: $memSizePrnt\tECC SIZE: $eccSizePrnt\t"; print TIMEFILE "SPARE SIZE: $spareSizePrnt\tUSABLE SIZE: $usableSizePrnt\n"; print TIMEFILE "OVERHEAD: $overheadPrnt\nRAID: $raid\nNEC: $nec\n"; if (!$raid) { print TIMEFILE "\tL0 SUE\t\t\t\tL0 HUE\n"; print TIMEFILE "TIME\tMEAN\tMIN\tMAX\tSTDV\tMEAN\tMIN\tMAX\tSTDV\n"; }

PAGE 88

77 else { print TIMEFILE "\tL0 SUE\t\t\tL0 HUE\t\t\tL1 SUE\t\t\tL1 HUE\n"; print TIMEFILE "TIME\tMEAN\tMN\tMAX\tSTDV\tMEAN\tMN\tMAX\tSTDV\t"; print TIMEFILE "MEAN\tMN\tMAX\tSTDV\tMEAN\tMN\tMAX\tSTDV\n"; } #add failures at each polling interval, for all memory trials to a #stats data structure this will allow the statistical results of all #trials to be printed to the TIMEFILE for (my $i = 0; $i <= $lifetime; $i += $pollInterval) { if (!$raid) { my $herL0Stat = Statistics::Descriptive::Sparse->new; my $serL0Stat = Statistics::Descriptive::Sparse->new; $herL0Stat->add_data(@{$herL0StatData[$i]}); $serL0Stat->add_data(@{$serL0StatData[$i]}); #print stat results for the total series of executions to a file printf TIMEFILE "$i\t%d\t%d\t%d\t%d\t%d\t%d\t%d\t%d\n", $serL0Stat->mean, $serL0Stat->min, $serL0Stat->max, $serL0Stat->standard_deviation, $herL0Stat->mean, $herL0Stat->min, $herL0Stat->max, $herL0Stat->standard_deviation; } else { my $herL0Stat = Statistics::Descriptive::Sparse->new; my $serL0Stat = Statistics::Descriptive::Sparse->new; my $herL1Stat = Statistics::Descriptive::Sparse->new; my $serL1Stat = Statistics::Descriptive::Sparse->new; $herL0Stat->add_data(@{$herL0StatData[$i]}); $serL0Stat->add_data(@{$serL0StatData[$i]}); $herL1Stat->add_data(@{$herL1StatData[$i]}); $serL1Stat->add_data(@{$serL1StatData[$i]}); #print stat results for the total series of executions to a file printf TIMEFILE "$i\t%d\t%d\t%d\t%d\t%d\t%d\t%d\t%d\t %d\t%d\t%d\t%d\t%d\t%d\t%d\t%d\n", $serL0Stat->mean, $serL0Stat->min, $serL0Stat->max, $serL0Stat->standard_deviation, $herL0Stat->mean, $herL0Stat->min, $herL0Stat->max, $herL0Stat->standard_deviation, $serL1Stat->mean, $serL1Stat->min, $serL1Stat->max, $serL1Stat->standard_deviation, $herL1Stat->mean, $herL1Stat->min, $herL1Stat->max, $herL1Stat->standard_deviation; } } #Generate data structure for plotting with GDGraph::boxplot #The first line is the label for each of the memory configurations #Each memory configurations will have been executing maxTrials #num times and the statistical results will be plotted push @{$serL0PlotData[0]}, "Trial $execCnt"; push @{$herL0PlotData[0]}, "Trial $execCnt"; push @{$serL1PlotData[0]}, "Trial $execCnt"; push @{$herL1PlotData[0]}, "Trial $execCnt";

PAGE 89

78 #copy error counts for trials at a given time in the memory lifetime for my $i (0..$maxTrials) { push @{$serL0PlotData[1][$execCnt]}, $serL0StatData[$lifetime][$i]; push @{$herL0PlotData[1][$execCnt]}, $herL0StatData[$lifetime][$i]; push @{$serL1PlotData[1][$execCnt]}, $serL1StatData[$lifetime][$i]; push @{$herL1PlotData[1][$execCnt]}, $herL1StatData[$lifetime][$i]; } #track number of memory configs executed since the script was started $execCnt += 1; #update the Execute button to represent the num configs executed $executeSim_b->configure(-text => "Execute $execCnt"); close TIMEFILE or die "Closing: $!"; openFile("time_$filename.out"); } B.4 Graphical User Interface Subroutines B.4.1 Commify sub commify { my $text = reverse $_[0]; $text =~ s/(\d\d\d)(?=\d)(?!\d*\.)/$1,/g; return scalar reverse $text; } B.4.2 Plot Results sub plotResults { my $serL0graph = new GD::Graph::boxplot(); my $herL0graph = new GD::Graph::boxplot(); my $serL1graph = new GD::Graph::boxplot(); my $herL1graph = new GD::Graph::boxplot(); $serL0graph->set( x_label => 'Memory Configuration', y_label => 'Number of L0 Soft Uncorrectable Errors', title => 'Memory Simulation L0 SUE Results', ); $herL0graph->set( x_label => 'Memory Configuration', y_label => 'Number of L0 Hard Uncorrectable Errors', title => 'Memory Simulation L0 HUE Results', ); $serL1graph->set( x_label => 'Memory Configuration', y_label => 'Number of L1 Soft Uncorrectable Errors', title => 'Memory Simulation L1 SUE Results', ); $herL1graph->set( x_label => 'Memory Configuration', y_label => 'Number of L1 Hard Uncorrectable Errors', title => 'Memory Simulation L1 HUE Results', );

PAGE 90

79 my $serL0gd = $serL0graph->plot( \@serL0PlotData ); my $herL0gd = $herL0graph->plot( \@herL0PlotData ); my $serL1gd = $serL1graph->plot( \@serL1PlotData ); my $herL1gd = $herL1graph->plot( \@herL1PlotData ); if ($serL0gd) { open(IMG, '>serL0.png') or die $!; binmode IMG; print IMG $serL0gd->png; close IMG; } else { print "EMTPY SET: SER L0\n"; } if ($herL0gd) { open(IMG, '>herL0.png') or die $!; binmode IMG; print IMG $herL0gd->png; close IMG; } else { print "EMTPY SET: HER L0\n"; } if ($serL1gd) { open(IMG, '>serL1.png') or die $!; binmode IMG; print IMG $serL1gd->png; close IMG; } else { print "EMTPY SET: SER L1\n"; } if ($herL1gd) { open(IMG, '>herL1.png') or die $!; binmode IMG; print IMG $herL1gd->png; close IMG; } else { print "EMTPY SET: HER L1\n"; } }

PAGE 91

80 B.4.3 Update Memory Size sub updateMemSize { # Determine number of bits used for error correcting codes # 'k' data bits and 'r' check bits are calculated $rVal = $nec (log($kVal)/log(2) + 1); # add one more check bit for RAID with NEC or SEC ($nec <= 2 and $raid ? $rVal += 1: 0); $L0Col = $kVal + $rVal; # Total memory size is size of all usable and spare memory $L0UsableSize = $kVal $L0Row $L0Plane; $L0ECCSize = $rVal $L0Row $L0Plane; $L0SpareSize = ($L0RedRow $L0Plane $L0Col) + ($L0RedCol $L0Row); $L0MemSize = $L0UsableSize + $L0ECCSize + $L0SpareSize; $L1UsableSize = $L0UsableSize $L1Row $L1Col; $L1ECCSize = ($L0ECCSize $L1Row $L1Col) + ($raid $L0MemSize $L1Row); $L1SpareSize = ($L0SpareSize $L1Row $L1Col) + ($L0MemSize $L1Red); $L1MemSize = $L1UsableSize + $L1ECCSize + $L1SpareSize; $totalUsableSize = $L1UsableSize $L2Row $L2Col; $totalECCSize = $L1ECCSize $L2Row $L2Col; $totalSpareSize = ($L1SpareSize $L2Row $L2Col) + ($L1MemSize $L2Red); $totalMemSize = $totalUsableSize + $totalECCSize + $totalSpareSize; $spareSizePrnt = commify($totalSpareSize); $memSizePrnt = commify($totalMemSize); $usableSizePrnt = commify($totalUsableSize); $eccSizePrnt = commify($totalECCSize); $overheadPrnt = commify(nearest(0.01, ($totalMemSize $totalUsableSize) / $totalUsableSize 100)); #Calculate MTTF from hard and soft errors #print "HER: $her\tSER: $ser\tTOTALMEM: $totalMemSize\n"; $herMTTFMem = nearest(0.001, (1e9 / ($her $totalMemSize))); $serMTTFMem = nearest(0.001, (1e9 / ($ser $totalMemSize))); $herMTTFBit = 1e9 / $her; $memSize_l->configure(-text => "Total: $memSizePrnt bits\n Spare: $spareSizePrnt bits\n ECC: $eccSizePrnt bits\n Usable: $usableSizePrnt bits\n Overhead: $overheadPrnt%"); $mttf_l->configure(-text => "H: $herMTTFMem\nS: $serMTTFMem"); $chkBits_l->configure(-text => "Data: $kVal\nCheck: $rVal"); return 1; }

PAGE 92

81 B.4.4 Menubar Subroutines #create EDIT menu item in menubar sub edit_menuitems { [ [qw/command Undo -accelerator Ctrl-Z -command/ => \&tUndo], [qw/command Redo -accelerator Ctrl-Y -command/ => \&tRedo], '', [qw/command Copy -accelerator Ctrl-C -command/ => \&tCopy], [qw/command Cut -accelerator Ctrl-X -command/ => \&tCut], [qw/command Paste -accelerator Ctrl-V -command/ => \&tPaste], '', [qw/command/,'Select All',qw/-accelerator Ctrl-A -command/ => \&tSelectAll], [qw/command/,'Unselect All',qw/-command/ => \&tUnSelectAll], ] } #create FILE menu item in menubar sub file_menuitems { [ [qw/command New -command/ => \&newFile], [qw/command Open -command/ => \&openFile], '', [qw/command Save -command/ => \&saveFile], [qw/command/,'Save As',qw/-command/ => \&saveAs], [qw/command/,'Save All',qw/-command/ => \&saveAll], '', [qw/command Close -command/ => \&closeFile], [qw/command/,'Close All',qw/-command/ => \&closeAll], '', [qw/command Exit -command/ => \&exitProg] ]; } #create HELP menu item in menubar sub help_menuitems { [ [qw/command/,'Help Topics',qw/-command/ => sub {print "Help Topics\n"}], '', [qw/command About -command/ => \&showAbout], ]; } #create SEARCH menu item in menubar sub search_menuitems { [ [qw/command Find -command/ => \&tFind], [qw/command/,'Find Next',qw/-command/ => \&tNext], [qw/command/,'Find Previous',qw/-command/ => \&tPrev], [qw/command/,'Replace',qw/-command/ => \&tReplace], ]; }

PAGE 93

82 #close all programs, saving if necessary sub closeAll { #Save all pages before closing for (sort keys %pageName) { if($pageName{$_}) { $files_nb->raise($pageName{$_}); &closeFile(); } } } #standard CLOSE FILE function sub closeFile { $currentPage = $files_nb->raised(); my $page2delete = $currentPage; $files_nb->delete($pageName{$currentPage}); delete $pageName{$page2delete}; } #close all files and exit program sub exitProg { &closeAll; exit 1; } #create a new notebook page with text box and insert data into it sub newFile { #Are there any open pages? if(($files_nb->raised()) eq "") { $fileNum = 1; } else { $fileNum += 1; } #Create a new page #if a filename was defined by open file, use it #otherwise, name Untitled # if($fName) { $fileName[$fileNum] = $fName; $fName = ""; } else { $fileName[$fileNum] = "Untitled $fileNum"; } $pageName{$fileNum} = $fileNum; my $files_pg = $files_nb->add($pageName{$fileNum}, -label=> $fileName[$fileNum]);

PAGE 94

83 $files_nb->Resize(); # Create a scrolled text widget and place inside the notebook page $files_t[$fileNum] = $files_pg->Scrolled("Text", -height => 8, -width => 70, -wrap => 'none', -background => 'white', -scrollbars => "osoe", -font => 'Courier 11'); $files_t[$fileNum]->pack(-side => 'bottom', -fill => 'both', -expand => 1); $files_nb->raise($pageName{$fileNum}); $top_w->update(); } #open a file and then send it to a new notebook page sub openFile { #if parameter was passed to open, open that file; #otherwise, give open dialog box if(@_) { $fName = shift(@_); } else { #define acceptable types of files to be read in my $types = [ ['All Files', '*', ], ['Text Files', ['.out', '.txt', 'TEXT']], ]; #return user selection to filename $fName = $top_f->getOpenFile(-filetypes=>$types, -initialdir => $path); } if ($fName =~ /(.*)[\/\\](.+\.?.*)/) { $path = $1; $fName = $2; chdir "$path" or die "Can't cd to $path: $!\n"; } #Open the file in a new page if one was defined and the file is #not already opened #if same file is open, close current version and reopen new version for (sort keys %pageName) { if($fileName[$pageName{$_}] eq $fName) { $files_nb->raise($pageName{$_}); &closeFile(); } } if($fName) { &newFile; }

PAGE 95

84 else { return 0; } sysopen(INFILE, "$fileName[$fileNum]", O_RDONLY) or die "Can not open the file $fileName[$fileNum] in $path\n"; #use filename to title notebook tab $files_nb->pageconfigure($pageName{$fileNum}, -label=> "$fileName[$fileNum]"); while () { $files_t[$fileNum]->insert('end', $_); } close INFILE or die "Closing: $!"; } #standard SAVE ALL function sub saveAll { #save current page that is raised $currentPage = $files_nb->raised(); #loop through each page to save them for (sort keys %pageName) { $files_nb->raise($pageName{$_}); &saveFile(); } $files_nb->raise($pageName{$currentPage}); } #standard SAVE AS function sub saveAs { $currentPage = $files_nb->raised(); $files_t[$currentPage]->FileSaveAsPopup(); #retrieve filename of saved file my $saveAsName = $files_t[$currentPage]->FileName(); if($saveAsName eq "") #canceled save { return 0; } else { if($saveAsName =~ /.*[\/\\](.*)/) { $fileName[$currentPage] = $1; } else { $fileName[$currentPage] = $saveAsName; } # Show the file name in the page tab $files_nb->pageconfigure($pageName{$currentPage}, -label=>$fileName[$currentPage]); } }

PAGE 96

85 #standard SAVE function sub saveFile { $currentPage = $files_nb->raised(); my $infoBox = $top_f->messageBox(-title => 'Are You Sure?', -message => "Do you want to overwrite the existing file(s)?", -type => 'YesNo', -icon => 'question', -default => 'no' ); if($infoBox =~ /yes/i) { #display Save dialog box $files_t[$currentPage]->Save($fileName[$currentPage]); # Retrieve filename my $saveName = $files_t[$currentPage]->FileName(); if($saveName eq "") #canceled save { return 0; } else { #only capture the filename and not the path if($saveName =~ /.*[\/\\](.*)/) { $fileName[$currentPage] = $1; } else { $fileName[$currentPage] = $saveName; } # Set label of page to filename. $files_nb->pageconfigure($pageName{$currentPage}, -label=>$fileName[$currentPage]); } } } #check to see if file should be saved before close if it has been changed sub shouldSave { $currentPage = $files_nb->raised(); my $file = $files_nb->pagecget($pageName{$currentPage}, -label); my $save_d = $top_f->Dialog(-title=>"Save File?", -text=>" Do you want to save $file? ", -bitmap=>'question', -default_button=>'No', -buttons=>[qw/Yes No Cancel/]); return $save_d->Show(); }

PAGE 97

86 B.4.5 Text Editor Subroutines #copy text selection sub tCopy { $currentPage = $files_nb->raised(); #$files_t[$currentPage]->clipboardColumnCopy(); $files_t[$currentPage]->insertControlCode(); } #cut text selection sub tCut { $currentPage = $files_nb->raised(); $files_t[$currentPage]->clipboardCut(); } #paste from clipboard sub tPaste { $currentPage = $files_nb->raised(); $files_t[$currentPage]->clipboardPaste(); } #undo last action sub tUndo { $currentPage = $files_nb->raised(); $files_t[$currentPage]->undo(); } #redo last action sub tRedo { $currentPage = $files_nb->raised(); $files_t[$currentPage]->redo(); } #select all text sub tSelectAll { $currentPage = $files_nb->raised(); $files_t[$currentPage]->selectAll(); } #unselect all text sub tUnSelectAll { $currentPage = $files_nb->raised(); $files_t[$currentPage]->unselectAll(); }

PAGE 98

87 #find text sub tFind { $currentPage = $files_nb->raised(); $files_t[$currentPage]->FindPopUp(); } #find next match sub tNext { $currentPage = $files_nb->raised(); $files_t[$currentPage]->FindSelectionNext(); } #find prev match sub tPrev { $currentPage = $files_nb->raised(); $files_t[$currentPage]->FindSelectionPrevious(); } #search and replace sub tReplace { $currentPage = $files_nb->raised(); $files_t[$currentPage]->findandreplacepopup(); } #show About information box sub showAbout{ my $aboutString = "Fault Tolerant Memory Simulator v0.5\n\n"; $top_w->messageBox(-title => 'About', -message => "$aboutString", -type => 'OK');

PAGE 99

LIST OF REFERENCES 1. M. P. Frank, Reversibility for efficient computing, Ph.D. dissertation, Dept. Elect. Eng. and Comp. Sci., Massachusetts Institute of Technology, Cambridge, MA, 1999. 2. M. Butts, A. DeHon, and S. C. Goldstein, Molecular electronics: devices, systems and tools for gigagate, gigabit chips, in Proc. IEEE/ACM International Conference on Computer Aided Design, pp. 433-440, Nov. 2002. 3. S. C. Goldstein and M. Budiu, NanoFabrics: spatial computing using molecular electronics, in Proc. 28 th Annual International Symposium on Computer Architecture, pp. 178-189, July 2001. 4. A. DeHon, Array-based architecture for FET-based nanoscale electronics, IEEE Trans. on Nanotechnology, vol. 2, no. 1, March 2003. 5. M. Mishra and S. C. Goldstein, Scalable defect tolerance for molecular electronics, in Workshop on Non-Silicon Computing, 8th International Symposium on High-Performance Computer Architecture, (Cambridge, MA), Feb. 2002. 6. K. Nikoli, A. Sadek, and M. Forshaw, Fault-tolerant techniques for nanocomputers, Nanotechnology, vol. 13, pp. 357-362, May 2002. 7. J. von Neumann, Probabilistic logic and the synthesis of reliable organisms from unreliable components, in Automata Studies, C. Shannon and J. McCarthy, Eds. Princeton, NJ: Princeton University Press, 1956. 8. C. E. Kozyrakis, S. Perissakis, D. Patterson, T. Anderson, K. Asanovi, N. Cardwell, R. Fromm, J. Golbus, B. Gribstad, K. Keeton, R. Thomas, N. Treuhaft, and K. Yelick, Scalable processors in the billion-transistor era: IRAM, Computer, vol. 30, no. 9, pp. 75-78, Sept. 1997. 9. P. Van Zant, Microchip Fabrication: A Practical Guide to Semiconductor Processing. 4 th ed., New York, NY: McGraw-Hill, 2000, pp. 49-110. 10. H. Kitajima and Y. Shiramizu, Requirements for contamination control in the gigabit era, IEEE Trans. on Semiconductor Manufacturing, vol. 10, no. 2, pp. 267-272, May 1997. 11. IEEE Standard Computer Dictionary: A Compilation of IEEE Standard Computer Glossaries. New York, NY: 1990. 88

PAGE 100

89 12. K. Chakraborty and P. Mazumder, Fault-Tolerance and Reliability Techniques for High-Density Random-Access Memories. Upper Saddle River, NJ: Prentice Hall, 2002, pp. 2-18. 13. D. J. W. Noorlag, L. M. Terman, and A. G. Konheim, The effect of alpha-particle-induced soft errors on memory systems with error correction, IEEE Journal of Solid-State Circuits, vol SC-15, no. 3, pp. 319-325, June 1980. 14. A. M. Saleh, J. J. Serrano, and J. H. Patel, Reliability of scrubbing recovery-techniques for memory systems, IEEE Trans. on Reliab., vol. 39, no. 1, pp. 114-122, April 1998. 15. P. Hazucha, C. Svensson, and S. A. Wender, Cosmic-ray soft error rate characterization of a standard 0.6-m CMOS process, IEEE Journal of Solid-State Circuits, vol. 35, no. 10, pp. 1422-1429, Oct. 2000. 16. S. Schaefer, DRAM Soft Error Rate Calculations, Micron Design Line, vol. 3, no. 1, 1994. 17. A. H. Johnston, G. M. Swift, and D. C. Shaw, Impact of CMOS scaling on single-event hard errors in space systems, in Proc. IEEE Symposium on Low Power Electronics, pp. 88-89, Oct. 1995. 18. R. Baumann, The impact of technology scaling on soft error rate performance and limits to the efficacy of error correction, IEEE International Electron Devices Meeting, pp. 329-332, Dec. 2002. 19. J. F. Ziegler, H. W. Curtis, H. P. Muhlfeld, C. J. Montrose, B. Chin, M. Nicewicz, et al., IBM experiments in soft fails in computer electronics (1978-1994), IBM J. Res. Develop., vol. 40, no. 1, pp. 3-17, Jan. 1996. 20. J. L. Hennessy and D. A. Patterson, Computer Architecture: A Quantitative Approach. 3 rd ed., New York, NY: Morgan Kaufmann, 2003, pp. 11-56. 21. N. Sakashita, F. Okuda, K. Shimomura, H. Shimano, M. Hamada, T. Tada, S. Komori, K. Kyuma, A. Yasuoka, and H. Abe, A built-in self-test circuit with timing margin test function in a 1Gbit synchronous DRAM, in Proc. International Test Conference, pp. 319-324, 1996. 22. K. Chakraborty, S. Kulkarni, M. Bhattacharya, P. Mazumder, and A. Gupta, A physical design tool for built-in self-repairable RAMs, IEEE Trans. on VLSI Systems, vol. 9, no. 2, pp. 352-364, April 2001. 23. K. Zarrineh and S. J. Upadhyaya, On programmable memory built-in self test architectures, in Proc. Design, Automation and Test, pp. 708-713, March 1999. 24. R. Treuer and V. K. Agarwal, Built-in self-diagnosis for repairable embedded RAMs, IEEE Design & Test of Computers, vol 10, no. 2, pp. 24-33, June 1993.

PAGE 101

90 25. C. L. Chen and M. Y. Hsiao, Error-correcting codes for semiconductor memory applications: a state-of-the-art review, IBM J. Res. Develop., vol. 28, no. 2, pp. 124-134, March 1984. 26. K. Furutani, K. Arimoto, H. Miyamoto, T. Kobayashi, K. Yasuda, and K. Mashiko, A built-in hamming code ECC Circuit for DRAMs, IEEE Journal of Solid-State Circuits, vol. 24, no. 1, pp. 50-56, Feb. 1999. 27. Sematech. (July 1, 2003) International Technology Roadmap for Semiconductors. [Online] Available: http://public.itrs.net 28. M. R. Stan, P. D. Franzon, S. C. Goldstein, J. C. Lach, and M. M. Ziegler, Molecular elctronics: from devices and interconnect to circuits and architecture, in Proc. of the IEEE, vol. 91, no. 11, pp. 1940-1957, Nov. 2003. 29. X. Duan, Y. Huang, and C. Lieber, Nanowire nanoelectronics assembled from the bottom-up, in Molecular Nanoelectronics, M. A. Reed and T. Lee, Eds. Stevenson Ranch, CA: American Scientific Publishers, 2003, pp. 199-227. 30. P. Avouris, J. Appenzeller, R. Martel, and S. J. Wind, Carbon nanotube electronics, in Proc. of the IEEE, vol 91, no. 11, pp. 1772-1784, Nov. 2003. 31. J. Guo, M. Lundstrom, and S. Datta, Performance projections for ballistic carbon nanotube field-effect transistors, Applied Physics Letters, vol. 80, no. 17, pp. 3192-3194, April 2002. 32. J. R. Heath, R. S. Williams, and P. J. Kuekes, Chemically synthesized and assembled electronics devices, U.S. Patent 6 459 095, Oct. 1, 2002. 33. W. J. Gallagher, J. H. Kaufman, S. S. Papworth, and R. E. Scheuerlein, Magnetic memory array using magnetic tunnel junction devices in the memory cells, U.S. Patent 5 640 343, June 17, 1997. 34. T. Rueckes, K. Kim, E. Joselevich, G. Y. Tseng, C. Cheung, and C. M. Lieber, Carbon nanotube-based nonvolatile random access memory for molecular computing, Science, vol. 289, pp. 94-97, July 2000. 35. P. J. Kuekes and R. S. Williams, Demultiplexer for a molecular wire crossbar network, U.S. Patent 6 256 767, July 3, 2001. 36. G. Yang and T. Fuja, The reliability of systems with two levels of fault tolerance: the return of the birthday surprise, IEEE Trans. on Computers, vol. 41, no. 11, pp. 1490-1496, Nov. 1992. 37. C. H. Stapper and H. S. Lee, Synergistic fault-tolerance for memory chips, IEEE Trans. on Computers, vol. 41, no. 9, pp. 1078-1086, Sept. 1992.

PAGE 102

91 38. T. Chen and G. Sunada, Design of a self-testing and self-repairing structure for highly hierarchical ultra-large capacity memory chips, IEEE Trans. on VLSI Systems, vol. 1, no. 2, pp. 88-97, June 1993. 39. T. Hirose, H. Kuriyama, S. Murakami, K. Yuzuriha, T. Mukai, K. Tsutsumi, Y. Nishimura, Y. Kohno, and K. Anami, A 20-ns 4-Mb CMOS SRAM with hierarchical word decoding architecture, IEEE Journal of Solid-State Circuits, vol. 25, no. 5, pp. 1068-1074, Oct. 1990. 40. T. Chen, D. Louderback, and G. Sunada, Optimization of the number of levels of hierarchy in large-scale hierarchical memory systems, in Proc. IEEE Symposium on Circuits and Systems, 1992, vol. 2, pp. 2104-2107. 41. W. B. Culbertson, R. Amerson, R. J. Carter, P. Kuekes, and G. Snider, Defect tolerance on the Teramac custom computer, in Proc. 5 th Annual IEEE Symposium on FPGAs for Custom Computing Machines, pp. 116-123. 42. J. R. Heath, P. J. Kuekes, G. S. Snider, and R. S. Williams, A defect-tolerant architecture: opportunities for nanotechnology, Science, vol. 280, pp. 1716-1721, June 1998. 43. T. J. Dell, A white paper on the benefits of Chipkill-correct ECC for PC server main memory, IBM Microelectronics Division, Nov. 1997. 44. D. A. Patterson, G. Gibson, and R. H. Katz, A Case for Redundant Arrays of Inexpensive Disks (RAID), ACM SIGMOD Record, vol. 17, no. 3, pp. 109-116, 1988. 45. K. A. Mehdi and J. M. Kontoleon, Design and analysis of a self-testing-and-self-repairing random access memory STAR-RAM with error correction, Microelectron. Reliab., vol. 38, no. 4, pp. 605-617, 1998. 46. K. A. Mehdi and J. M. Kontoleon, A rapid SMART approach for reliability and failure mode analysis of memory and large systems, Microelectron. Reliab., vol. 36, no. 10, pp. 1547-1556, 1996. 47. C. L. Chen and R. A. Rutledge, Fault-tolerant memory simulator, IBM J. Res. Develop., vol. 28, no. 2, pp. 184-194, March 1984. 48. M. R. Libson and H. E. Harvey, A general-purpose memory reliability simulator, IBM J. Res. Develop., vol. 28, no. 2, pp. 196-205, March 1984. 49. D. M. Blough, Performance evaluation of a reconfiguration-algorithm for memory arrays containing clustered faults, IEEE Trans. on Reliab., vol. 45, no. 2, pp. 274-284, June 1996.

PAGE 103

92 50. A. Messer, P. Bernadat, G. Fu, D. Chen, Z. Dimitrijevic, D. Lie, D. D. Mannaru, A. Riska, D. Milojicic, Susceptibility of modern systems and software to soft errors, HP Labs Tech. J., vol. 43, March 2001. 51. F. Peper, J. Lee, F. Abo, T. Isokawa, S. Adachi, N. Matsui, and S. Mashiko, Fault-tolerance in nanocomputers: a cellular array approach, IEEE Trans. on Nanotechnology, vol. 3, no. 1, pp, 187-201, March 2004. 52. D. C. Bossen, C. L. Chen, and M. Y. Hsiao, Fault alignment exclusion for memory using address permutation, IBM J. Res. Develop., vol. 28, no. 2, pp. 170-176, March 1984. 53. P. Shivakumar, M. Kistler, S. W. Keckler, D. Burger, and L. Alvisi, Modeling the effect of technology trends on the soft error rate of combinational logic, in Proc. IEEE International Conference on Dependable Systems and Networks, pp. 389-398, June 2002. 54. I. Koren and Z. Koren, Defect tolerance in VLSI circuits: techniques and yield analysis, in Proc. of the IEEE, vol. 86, no. 9, pp. 1819-1836, Sept. 1998.

PAGE 104

BIOGRAPHICAL SKETCH Casey Jeffery is a graduate student in the Electrical and Computer Engineering Department at the University of Florida. He will graduate in December 2004 with a Master of Science degree in computer engineering and will continue working toward a Ph.D. degree with a focus on computer architecture. Casey completed his Bachelor of Science degree in the same area at the South Dakota School of Mines and Technology. At present, Casey has completed four internships with Intel Corporation and is working on his fifth for a total of more than two-and-a-half years with the company. He has worked at four different sites throughout the United States in a variety of areas, including microprocessor architecture R&D, product engineering, and software engineering. He also worked as a teaching assistant for juniorand senior-level undergraduate digital design laboratories. 93


Permanent Link: http://ufdc.ufl.edu/UFE0007163/00001

Material Information

Title: Performance Analysis of Dynamic Sparing and Error Correction Techniques for Fault Tolerance in Nanoscale Memory Structures
Physical Description: Mixed Material
Copyright Date: 2008

Record Information

Source Institution: University of Florida
Holding Location: University of Florida
Rights Management: All rights reserved by the source institution and holding location.
System ID: UFE0007163:00001

Permanent Link: http://ufdc.ufl.edu/UFE0007163/00001

Material Information

Title: Performance Analysis of Dynamic Sparing and Error Correction Techniques for Fault Tolerance in Nanoscale Memory Structures
Physical Description: Mixed Material
Copyright Date: 2008

Record Information

Source Institution: University of Florida
Holding Location: University of Florida
Rights Management: All rights reserved by the source institution and holding location.
System ID: UFE0007163:00001


This item has the following downloads:


Full Text












PERFORMANCE ANALYSIS OF DYNAMIC SPARING AND ERROR
CORRECTION TECHNIQUES FOR FAULT TOLERANCE IN NANOSCALE
MEMORY STRUCTURES
















By

CASEY MILES JEFFERY


A THESIS PRESENTED TO THE GRADUATE SCHOOL
OF THE UNIVERSITY OF FLORIDA IN PARTIAL FULFILLMENT
OF THE REQUIREMENTS FOR THE DEGREE OF
MASTER OF SCIENCE

UNIVERSITY OF FLORIDA


2004

































Copyright 2004

by

Casey M. Jeffery

































This document is dedicated to my new wife, Mariah.














ACKNOWLEDGMENTS

It is essential that I thank my advisor, Dr. Renato Figueiredo. He has given a great

deal of time and assistance in the culmination of this degree. His enthusiasm and

guidance helped see me through the completion of this project.

I would also like to thank my family and friends who have supported me in this

endeavor. Completion of the research and the writing of this thesis would not have been

possible without their encouragement.
















TABLE OF CONTENTS

page

A C K N O W L E D G M E N T S ................................................................................................. iv

LIST OF TABLES ......... .... ............... .......... .. ... ........ ........... .. viii

LIST OF FIGURES ......... ......................... ...... ........ ............ ix

A B ST R A C T ................. .......................................................................................... x

CHAPTER

1 IN TRODU CTION ................................................. ...... .................

2 B A C K G R O U N D .................. .................................... .......... ........ .......... .. ....

2 .1 M odem M em ory D evices .......................................................................... ...... 5
2.1.1 M emory Architecture ........................................................6
2.1.2 M em ory Fabrication ............................................................................. 7
2.2 Reliability .........................................9........9
2.2.1 Characterization of Reliability .................................................. ...............9
2.2.2 D effects, Faults, and Errors ...................................................................... 11
2 .2 .3 E rro r R ates .................................................................................... 1 1
2.3 Fault Tolerance in M em ory ........................................................................... 14
2 .3 .1 D evice R edundancy ........................................................................ ... ... 14
2.3.2 Error C orrection C odes......................................... .......................... 17

3 NANOSCALE MEMORY DEVICES ............................ .................... 19

3.1 Present and Future Scaling Challenges...................................... ............... 19
3.2 N anoscale B building B locks........................................................ ............... 21
3.2 .1 N anoscale W ires ................................................. .. .................. ... .. 2 1
3.2.2 Transistors, Diodes, and Switches............... ...........................................22
3.3 Molecular Electronic Architectures .......................................... ...............23
3.3.1 Crossbar Circuits ............................ ..... ...... .... ............... .24
3.3.2 M icro/N ano Interface ........................................ ........................... 25
3.3.3 H ierarchical A rchitectures................................... ..................................... 27
3.3.4 N ew R liability Concerns.................................... ........................... 27




v









4 HIERARCHICAL FAULT TOLERANCE FOR NANOSCALE MEMORIES........29

4.1 B benefits of a H hierarchical A approach ........................................ .....................29
4.2 Proposed M em ory Organization....................................... ......... ............... 30
4.2.1 A architecture O verview ........................................ .......................... 30
4.2.2 Possible Im plem entation Strategy ................................... ............... ..32
4.3. Fault Tolerance M echanisms ......... ................................33
4.3.1 Hierarchical Built-in Self-Test and Self-Repair.............................33
4.3.2 Advanced Error Correction Codes .................................. ............... 34

5 MEMORY SIMULATION AND RESULTS .................................... ...............36

5 .1 Sim u lator A rchitectu re............................................................... .....................36
5.1.1 Sim ulator C om ponents ........................................ ......................... 37
5.1.2 Simulation Interface ............ .... ......... ........................ 38
5.2 Simulation Component Details........................................................40
5.2.1 Parameter Generation ......... ....................................... 40
5.2.2 Simulation ........... ............... ......... 41
5.2.3 R results A analysis ................... .... .......... ............. .... ........47
5.3 Sim ulator V alidation .................................................. .....................................4 8
5.4 Sim ulation R results ......... ..... .... .................... ........ ........ .. .. ..........49

6 CONCLUSION AND FUTURE WORK ....................................... ............... 53

6 .1 C o n clu sio n s................................................. .................. 5 3
6 .2 F u tu re W ork ................................................... ................ 54

APPENDIX

A M ODERN M EM ORY DEVICES ........................................ ......................... 56

B SIM ULATOR SOURCE CODE ........................................ .......................... 57

B. 1 Initialization and Variable Declaration................................... .........................57
B .2 G raphical U ser Interface ........................................................... .....................59
B.3 Sim ulation Subroutines ......... .................................... ................ ............... 65
B.3.1 Generate Failures ..................................... ............................ 65
B .3 .2 S im u la te ............................................................................................... 6 7
B.3.3 D evice Redundancy ...........................................................................75
B .3.4 R un Sim ulation .............................. ................ .............. ............. 75
B.4 Graphical User Interface Subroutines......................... ........ ... ............ 78
B .4 .1 C om m ify .....................................................................78
B .4 .2 P lot R esu lts ............................................................7 8
B .4.3 U pdate M em ory Size ........................................ ...........................80
B .4.4 M enubar Subroutines ....................................................... ................81
B.4.5 Text Editor Subroutines ...... ..... .......................... ... ................86









L IST O F R E F E R E N C E S ........................................................................ .. ....................88

B IO G R A PH IC A L SK E T C H ...................................................................... ..................93















LIST OF TABLES
Table page

2-1. R liability m etric sum m ary ............................................... ............................. 10

2-2. Hamming (7,4) parity and syndrome generation equations ....................................18

3-1. ITRS DRAM production product generations......................................21

4-1. ECC summary for the proposed architecture. ................................. .................35

5-1. Sim ulator input param eters............................................... .............................. 41

5-2. Simulator validation to SEC analytical model for small memory sizes ...................48















LIST OF FIGURES
Figure page

2-1. M em ory hierarchy representation....................................................... .............. 6

2-2. The basic m em ory architectures. ......................................................... ....................7

2-3. The bathtub curve used to illustrate reliability over time.........................................10

3-1. N anoscale FET im plem entations....................................................... .............. 23

3-2. D em ultiplexor designs ......... .................... ......... ....................... ............... 26

4-1. Proposed hierarchical m em ory architecture........................................................... 31

4-2. A diagram of the nanoFabric architecture ..............................................................32

5-1. High-level simulator data flow diagram ........................................ ............... 38

5-2. Simulation interface with sample results ......................................... ...............39

5-4. Hierarchical hash data structure used in the simulator..............................................44

5-5. Example of failure chaining process in simplified fail list ...................................46

5-6. Simulator validation to SEC analytical model for large memory sizes....................49

5-7. HUE's for 1-Gbit memory with 64-bit word size....................... ...............51

5-8. HUE's for 1-Gbit memory with 128-bit word size .......... .......... .............. 51

5-9. HUE's for 1-Gbit memory with 256-bit word size .......... .......... .............. 52














Abstract of Thesis Presented to the Graduate School
of the University of Florida in Partial Fulfillment of the
Requirements for the Degree of Master of Science

PERFORMANCE ANALYSIS OF DYNAMIC SPARING AND ERROR
CORRECTION TECHNIQUES FOR FAULT TOLERANCE IN NANOSCALE
MEMORY STRUCTURES

By

Casey Miles Jeffery

December 2004

Chair: Renato J. O. Figueiredo
Major Department: Electrical and Computer Engineering

Continued advances in the fabrication of molecular structures have led to

speculation that nanoelectronic devices may be incorporated with CMOS technology in

the realm of ultra-dense logic and memory systems. Many proposed molecular systems

involve self-assembly techniques ideally suited for large, regular structures such as

memory. This method of fabrication, however, is predicted to result in an increased hard

error rate due to physical defects. This effect will be compounded by an increased

sensitivity of the ever-smaller bit storage units to soft errors (e.g., resulting from noise

and radiation effects).

Current memory devices rely on fault tolerance schemes such as modified

Hamming error correcting codes and static spare swapping to cope with these errors. It is

conceivable, however, that such solutions may not be sufficient in systems that are larger

and built using less reliable components and/or manufacturing techniques.









This thesis describes a memory organization that supports hierarchical, dynamic

fault tolerance mechanisms applicable to heterogeneous CMOS/molecular systems. It

has been projected that, in such systems, programmable mapping circuitry becomes

necessary when the interface between microscale and nanoscale address lines is formed

in a non-deterministic manner. This mapping circuitry may double as a means of

implementing more advanced forms of reconfiguration and error correction codes useful

in detecting and recovering from runtime faults.

In lieu of the ability to formulate an analytical model for such complex systems,

computer simulation is used to estimate the effectiveness of various fault tolerance

configurations. Results show that for a given percentage of redundancy overhead, these

methods may allow for substantial improvements in reliability over standard error

correction codes.














CHAPTER 1
INTRODUCTION

The size of the devices used to construct electronic components has decreased at an

astonishing pace over the past fifty years. It has followed the very familiar Moore's Law,

which states that the number of devices doubles approximately every 18 months. As of

2004, the state of technology is still several orders of magnitude away from even the

conservative limits imposed by physics (i.e., 1 bit/A3) [1], and it is in the foreseeable

future when human ingenuity will allow for the integration of huge numbers of devices in

a single part (i.e., 1012/cm2) [2].

There will be many new challenges associated with this level of integration that

will require complete new approaches to designing, fabricating, and testing devices. The

circuits will likely require construction using self-assembly and self-alignment techniques

at the molecular level, as it will almost certainly be infeasible to accurately order and

connect such a large number of devices using the current lithography-based technology.

The relative inexactness inherent in the self-assembly method of construction will make

the technology most applicable to the manufacture of devices with large, regular patterns

such as programmable logic and memory arrays at the lowest level. The arrays of

nanoscale devices may then be used hierarchically as building blocks for more complex

circuits [3, 4].

Even in these very regular processes, it is predicted that defect densities on the

order of 5% can be expected, and the small size of the devices will make them more

sensitive to external disturbances such as radiation and electromagnetic interference [2,

1









5, 6]. Off-line testing to find the faults will be impractical due to the sheer number of

devices. Instead, the circuits will be required to test themselves to a high degree and

detect defects in a reasonable amount of time. They must also be capable of dealing with

the defects by employing some form of fault tolerance.

The defects that occur at runtime are referred to as dynamic defects and are the

focus of this thesis. In the past, dynamic defects have all but been ignored. Only high-

end workstation and server memory currently make use of dynamic fault tolerance

mechanisms in the form of error correcting codes (ECC's). The codes have typically

been capable of logically correcting at most a single bit in each block of data accessed.

This will not be sufficient, however, for memories with orders of magnitude higher error

rates. In addition to the increase in bit error rates, an increase in the rate of row, column,

chip, and other periphery-related defects is expected. Defects of this type often cannot be

handled even when ECC's are implemented and ultimately lead to the failure of the entire

memory.

The most likely solution to combat these problems in future memory designs will

be to implement a form of fault tolerance that is more advanced than has been done in the

past. This new method of fault tolerance involves more powerful ECC's, as well as

dynamic reconfiguration. In dynamic reconfiguration built-in self-test (BIST) and built-

in self repair (BISR) logic are used to detect and repair faults either by swapping the

defective devices with spare devices or by marking the faulty devices so they will not be

accessed.

In this thesis, dynamic fault tolerance configurations are introduced that implement

such advanced ECC's and dynamic reconfiguration in a hierarchical fashion. A simulator









is then used to estimate the performance of the systems in terms of tolerance to faults

throughout the usable lifetime of the parts. Finally, the simulation results are utilized to

suggest configurations capable of providing the highest degree of synergism and

correspondingly make the most efficient use of a given percentage of overhead in terms

of redundant bits.

Chapter 2 gives a brief overview of the current state of memory devices, as well as

an introduction to fault tolerance mechanisms and the metrics used in determining the

reliability of such devices. Chapter 3 covers many of the most promising nanoscale

devices and architectures that have recently been developed and the corresponding

reliability concerns associated with them. Chapter 4 introduces the proposed hierarchical

fault tolerance mechanisms and goes on to show how they can be applied to the

architectures from the previous chapter. Chapter 5 details the simulation methodology

used to model the fault tolerance mechanisms and includes the results that have been

obtained. Finally, Chapter 6 summarizes the findings and gives some insight into future

work that may be done in the area.
















CHAPTER 2
BACKGROUND

Fault tolerance in electronic devices involves the use of redundant circuitry to

detect and possibly correct faults. This is not a new notion, as it has been in place since

the 1940s when it was proposed by John von Neumann. Before this discovery, it was

thought that large-scale circuits (i.e., millions of devices) could not be constructed to

function reliably. Von Neumann showed, however, that a very high degree of reliability

can be obtained, albeit at a high cost in terms of redundancy overhead [7].

To date, von Neumann's ideas have garnered little attention in industry as circuits

with hundreds of millions, and recently billions of devices have been constructed to

function reliably with little or no need to implement such fault-tolerance mechanisms.

These feats require extremely low error rates that are only possible with advanced silicon

fabrication techniques conducted in the cleanest facilities in the world. It does seem

reasonable to reevaluate the usefulness of fault tolerance, however, as device sizes

approach atomic levels and the ability to build entire components with near perfection

becomes too difficult and costly.

This chapter first gives a brief overview of the current state of memory device

technology and fabrication. The terminology related to memory reliability is then

introduced, and the basic mechanisms used to deal with defects are discussed. The topics

covered in this chapter will be used as the foundation for describing more advanced

fault-tolerance architectures applicable to future generations of devices.

4









2.1 Modern Memory Devices

It is well-known that memory devices in computer systems are used in a

hierarchical fashion that can be represented in a simplified form by the pyramid structure

shown in Figure 2-1. The base of the pyramid is composed of very large devices such as

tape backups and networked hard-disk storage devices that have a low cost per bit, slow

access times, and large data access chunks. The top of the pyramid represents the

smallest, fastest devices, which are the registers in the core of the microprocessor. The

registers move data in small blocks, have very fast access times, and a high cost per bit.

In recent years, there has been a continued growing and splitting up of the pyramid.

The number of bits at each of the levels has grown steadily and new levels have emerged.

There has also been a great deal of growth outside the pyramid. The use of non-volatile

flash memories in devices such as cameras, cellular phones, and personal digital

assistants is on the rise. There has also been a push towards entirely new architectures

that integrate logic processing units into memory devices [8].

It is conceivable that the memories constructed from nanoscale devices may fill any

of these roles, as well as others that have yet to be discovered. Because of this

uncertainty, the fault tolerance architectures to be presented in this thesis are relatively

generic and applicable to a wide range of memories, including random access devices in

the levels that have been highlighted in Figure 2-1.










Re sters bit
lOf1s B

Nanoscale Meirry Cache Memory
Application J cls 5 KB

Main Memory
1 D'rs MB

Hard Disk


Tape Backup
100's TB Spee


Figure 2-1. Memory hierarchy representation. Highlighting is placed over levels most
applicable to the fault tolerance architectures described in this thesis

2.1.1 Memory Architecture

Modem memory devices come in many forms and are categorized based on their

function and access pattern. The function is defined as whether the memory is read-only

or rewritable, as well as if it is volatile or non-volatile. The access pattern of the memory

classifies whether the data can be randomly accessed, as in a common DRAM device, or

if the accesses must be done serially, which is the case for video memory. There are also

memories that use parts of the data instead of an address for memory accesses. These are

known as associative memories and are common in cache structures. An overview of a

wide variety of memory devices is included in Appendix A.

The layout of most memory devices is approximately the same regardless of the

type. They generally follow a square shape and make use of row and column decoders to

control the selection of word and bit lines. Figure 2-2A shows the simplest example in

which the decoded row signals select multiple words and the decoded column signals

choose which of those words is returned. Figure 2-2B expands on this slightly by adding

a block decoder that determines in which block the word line will be activated. The










design is known as divided word line (DWL), and the extra level of decoding has the

benefit of reduced power consumption at the cost of additional decoding latency and area

overhead.


2 !"-1 2;- 1 2-1-'_L1
-Word line
RAt RowP x, Block Block, Mc
Decoder Memory Array A_ .o 0N Decoder A,

"=- Br~tline A 0-
Controll Control

Column Decoder Column Deoder
A Uo B vo


Figure 2-2. The basic memory architectures. A) The simplest example with a square
memory array and row and column decoders. B) Example of a DWL
structure with an extra level decoding at the block level.

2.1.2 Memory Fabrication

Virtually all modem semiconductor memory devices are manufactured on silicon

wafers using fabrication processes that have been fine-tuned for decades. The fabrication

starts with silicon dioxide ore (sand) which is converted through several stages before

ending in the form of a doped silicon crystal ingot. The ingot is sliced into wafers,

planed to be perfectly flat, and put through nearly 200 major layering, patterning, doping,

and heating steps [9].

Each of these steps has some probability of introducing a defect into the device,

although some are much more likely than others. The patterning process, also known as

photolithography or masking, is the most critical as it define the exact size, shape,

location, and orientation of the circuit devices on the wafer. It is on the bleeding edge of

technology, and with current processes having more than two dozens mask steps and









device feature sizes of less than 100 nanometers, there is a tremendous possibility for the

introduction of defects.

Defects are caused by four major types of contamination: particles, metallic ions,

chemicals, and bacteria [9]. When particles are transferred to the wafer surface, either

through the air or the growth of bacteria, they can cause opens by not allowing features to

form correctly, or shorts by connecting wires together. Metallic ions cause problems by

disrupting the doping and altering device electrical characteristics. Chemicals interfere in

wet processing steps and can affect etching or interact to change the properties of the

material.

Defects have been avoided historically by manufacturing the wafers in some of the

cleanest facilities in the world. Clean room technology makes use of high-efficiency

particulate attenuation (HEPA) filters and laminar air flow to circulate the air and filter

out all particles down to sub-micron sizes. The introduction of particles is kept to a

minimum by automating as much as possible and keeping people and tools out of the

environment. The tools are kept in separate bays called chases and have only their

interfaces exposed. When it is necessary for workers to enter the clean room, special

suits are worn to reduce the release of contaminants.

There is a class rating that has been associated with the level of cleanliness that has

been achieved. A Class-1 clean room has particles with a maximum size of

approximately 100 nm. This is based on the rule of thumb that particles must be 10x

smaller than the smallest feature size to maintain an acceptable level of physical defects.

Currently, the highest density memory devices are manufactured in clean rooms with a

rating of Class-0.1 or better and must be almost fully automated to avoid the










contamination brought in by humans. This is likely approaching the limit of the level of

cleanliness that can be achieved, yet device feature size continues to shrink [10].

2.2 Reliability

The reliability of a device is defined by the IEEE as, "the ability of a system or

component to perform its required functions under stated conditions for a specified period

of time [11]." It is an important metric in memory devices since it defines the probability

that a device will function correctly throughout its usable lifetime, which is typically

considered to be on the order of 100,000 hours (~11 years). This section will introduce

the terminology related to reliability, much of which is specific to electronic devices.

2.2.1 Characterization of Reliability

The reliability of a device is a function of its failure rate, X. In memory devices, the

failure rate is often modeled as a constant, which implies that the intervals between

failures follow an exponential distribution with a probability density function of e -t.

The inverse of the failure rate gives the mean time to failure (MTTF), also known as

mean time between failures (MTBF). This value is the expected time for a failure to

occur. The derivation of the reliability is shown in Equation 1, and a complete summary of

the terminology is provided in Table 2-1 [12, 13, 14].


R(t) = 1- jfe -6t
0
= -[1-e't] (1)
t
=e












Table 2-1. Reliability metric summary
Metric Function Symbol

Failure Rate h(t) A

Mean time to failure MTTF 1/

Exponential PDF f(t) ke- t
Exponential CDF F(t) 1-e

Reliability R(t) e-


The failure rate is not really constant over the lifetime of devices as was assumed in

the derivation of reliability. It is typically represented as a collection of three failure

curves that sum to form what is referred to as a bathtub curve (Figure 2-3). The

relatively flat bottom of the bathtub defines the usable lifetime and allows for the

assumption in the derivation.


Time


- Random Failures
S- Infant Mortality


............. Wearout
_ All Failures


Figure 2-3. The bathtub curve used to illustrate reliability over time. The bottom of the
bathtub represents the usable lifetime of components.









The steep failure rate early in the lifetime is called infant mortality and is the result

of the failing of marginally functional devices. A process known as burn-in is used to

accelerate the state of the devices past this point before they are distributed to customers.

The second steep curve at the end of the lifetime is from the wear-out of the devices.

Wear-out is of little concern as most devices are out of commission by that time. The

constant curve throughout the lifetime is the effect of single, random events such as

latchup or heavy ion radiation strikes.

2.2.2 Defects, Faults, and Errors

The distinction between defects, faults, and errors is not immediately clear. In this

thesis the term defect is used to reference a physical imperfection in a device such as the

opens or shorts described in the previous section. The ability of a part to work correctly

when defects are present is called defect tolerance. If a defect occurs in a section of

memory that is being accessed and results in incorrect data being returned, it is said to

result in a permanent fault. If the fault can be corrected either by the part or through the

use of software, then the part is said to have fault tolerance. Finally, if the fault cannot

be corrected it will cause an error that will be seen by the user [12].

It is possible for a fault to occur without being a result of a physical defect. The

extraordinarily small sizes of modern devices make them susceptible to environmental

effects such as cosmic rays, alpha particles, and neutrons that can change the value of a

stored bit. This type of fault is not permanent and is referred to as a transient fault. The

next time the bit is written to it will be restored to the correct value [15].

2.2.3 Error Rates

There are typically two bit-level error rates given for a memory device, a hard

error rate (HER) and a soft error rate (SER). The HER represents the frequency that









permanent faults occur, and the SER represents the rate at which transient faults arise.

The rates are measured in units of failures in time (FIT) where 1 FIT = 1 failure/109

device hours [15, 16]. The device-level error rate can be calculated by simply

multiplying the bit-level rate by the number of bits in the device. For example, a 1-Gbit

memory with a bit-level HER of one FIT is calculated in Equation 2.

bits Ifailure 1 failure
HER1Gb =109bits x = (2)
109 device hours hour

The HER of a device is affected somewhat by the level of maturity of the process

technology, but it is most directly correlated to the reduction is size of devices [17]. The

HER has increased slightly with each generation, but it has remained relatively low

overall at approximately one to two percent of the total error rate [18]. This is due in

large part to the precision with which silicon devices are fabricated.

Soft errors represent the majority of the errors that occur. Unlike the HER, the

SER is mostly dependent on the operating voltage, charge collection efficiency, and cycle

time of the device, as well as the level of radiation present in the device. The last several

generations of SRAM and DRAM memories have seen reductions in charge collection

efficiency and the impact of radiation through improvements in fabrication processes,

both of which help to lower the SER. They have also had continuous reductions in the

operating voltages and cycle times, both of which result in an increased SER. The

outcome of these counteracting influences has been a slight decrease in the bit-level SER

[18].

In devices such as DRAM, the effect of the cycle time on the SER must be

averaged over both the refresh cycle time and the read/write cycle time. In a 16-MB

DRAM chip the difference in the refresh cycle time of 15 ats and read/write cycle time of









200 ns leads to a 50x increase in SER [16]. To correctly calculate the overall SER, the

ratio of each cycle time must be estimated and multiplied by the corresponding SER.

The cycle time at which the memory has the data refreshed, or scrubbed, also plays

an important role in determining the overall error rate in the case that fault tolerance is

implemented. If a correctable number of bits have become faulty as a result of permanent

faults and are not rewritten, they have the possibility of lining up with transient faults to

cause an error. The rate at which the soft errors are removed from the system is referred

to as the scrubbing interval and must be accounted for in determining overall system

reliability.

The level of radiation present has a significant impact on the SER. Experiments

have shown more than a 10x increase in the SER of DRAM devices in going from sea-

level to an altitude of 10,000 feet and another 10x increase in going to commercial airline

flight altitude [15, 19]. The increase is the result of reduced protection from the

atmosphere. For this reason, any devices that will be used at high altitudes, and

especially in space, must have much more protection than those used at terrestrial levels.

At the device level, both the HER and SER have shown some cause for concern as

the size of memories continues to expand at a rate of up to 60% per year [20]. Even if the

bit-level error rates were to remain constant going forward, the error rate at the device

level would become unacceptable. For example, an SRAM memory manufactured on

90 nm technology has a bit-level HER on the order of 10-5 FIT and a SER of 10-3 FIT at

terrestrial levels. For a 1-MB array, now common as a cache in microprocessors, the

device-level error rate is 8472 FIT with an acceptable MTTF of 13.5 years. If the cache

size were to increase to 128-MB with the bit-level error rate unchanged, however, the









MTTF would drop to 38 days, which is clearly unacceptable and would require some

form of fault tolerance [18].

2.3 Fault Tolerance in Memory

As the size of memories grew to millions of bits, it became impractical for

manufacturers to fabricate fully functional parts with a high enough yield to be profitable.

This led to the introduction of static redundancy where redundant rows and columns were

included in the design and could be switched in to repair defects before shipping the part

to the customer. In today's memories of a billion or more bits, static redundancy has

become a necessity for producing functional units.

To handle the errors that occur after the part has shipped, dynamic redundancy is

used. The most common form of dynamic redundancy is the use of error correction

codes (ECC). ECC's allow errors to be detected and corrected before the data are

returned from the part. These and other forms of fault tolerance will be introduced in this

section and will form the basis for the hierarchical fault tolerance mechanisms that will

be proposed for future memories.

2.3.1 Device Redundancy

Device redundancy is the inclusion of spare devices in the design that can be

swapped with faulty devices to maintain functionality. As mentioned above, it has

become an important technique for producing modern memories with an acceptable yield.

Even with the precision of lithography-based fabrication, defects caused by contaminants

and tiny variations in process steps have become inevitable. All types of memory from

DRAM to microprocessor cache to flash are built with extra rows and/or columns that

can be switched in when defects are found.









The first step in making a repair is to determine the location of the defects in the

memory through the use of fault diagnosis algorithms. The specific types of defects that

can occur vary by memory type, but the algorithms used are similar in all cases.

Fortunately, the test cases are much simpler than those used for logic parts and work by

performing a sequence of reads and writes that vary in direction, bit pattern, and timing.

The goal is to locate bits that are stuck at high or low levels, lines that are bridged

together as a result of a short circuit, lines that are open circuited, and bits or lines that are

coupled together [12].

The patterns are applied with automated test equipment (ATE) and/or BIST

circuitry integrated into the memory. ATE are very expensive tools capable of applying

comprehensive sets of diagnostic test patterns to the memories at high speeds. As the

size of the memories has grown, the ability to test for all faults with ATE has become too

costly in many cases, and impossible in others, as some embedded memories cannot be

accessed directly through the external pins. It has been estimated that for 1-Gbit

memories, the test cost is roughly 85% of the manufacturing cost when done on standard

ATE machines, and the ratio continues to increase [21].

To reduce the test costs to profitable levels and to gain access to embedded

memories, many designs have been developed that allow for BIST. With BIST the

memory can test itself by applying the necessary patterns and returning the results, which

allows for much simpler and less expensive ATE. There are many techniques used for

generating the patterns, but most make use of linear feedback shift registers (LFSR's) or

microcode programming. Advance compilers have been designed to assist in the design









of the BIST architectures, which typically require less than one percent of the total chip

area [22-24].

When the faults have been detected, another algorithm is used to determine which

devices need to be repaired and if there is enough redundancy available. The complexity

of an optimal repair algorithm has been shown to be NP-hard, so heuristic approaches are

often used. The ATE is usually capable of determining optimal or near-optimal repair

strategies. In cases where the repair is done on the part with BISR circuitry, however,

greedy algorithms may be necessary to reduce test time and circuit complexity.

The redundant devices are swapped in by one of two reconfiguration methods. If

the reconfiguration is done by the manufacturer, it is known as a hard repair. A hard

repair is done by physically blowing fuses on the device through the use of lasers or

electrical currents. If the repair is done in the field, such as in a BISR circuit, it is known

as a soft repair. A soft repair is done by programming non-volatile devices on the parts

such as EPROM's or flash cells to connect the redundant devices.

Device redundancy does have some limitations. First, when a fault occurs, the data

that are in error cannot be corrected. The repair process must also be done non-

concurrently, or when the memory is not in use, which means the memory must be

cleared of its current contents before it can be tested. This requires that the current

memory contents be copied to spare arrays before the testing if it is not to be lost. The

process is also relatively slow as it is necessary to scan the entire memory, devise a repair

strategy, and program in the redundant devices. Overall, device redundancy is an

important means of fault tolerance, but it is not sufficient.









2.3.2 Error Correction Codes

ECC's are another important form of fault tolerance. The term is typically used to

refer to linear block codes that encode redundant information about the data stored in the

memory into what are referred to as codewords. The number of valid codewords is then

a subset of the possible codewords. The benefit to this method is that it can be used

concurrently, or when the memory is in use. It can also be integrated in many devices for

a tolerable overhead in term of latency and area, and in some implementations it is

capable of correcting errors.

The simplest form of ECC is the parity check code in which an extra bit is added to

each word and set to give either an even (even parity) or and odd (odd parity) number of

l's. This allows for the detection of all errors in which an odd number of bits are at fault.

More advanced ECC's have been developed that are capable of not only detecting errors,

but also correcting them. This class of codes requires that the valid codewords be

separated by a minimum number of bits given in Equation 3, where d is the minimum

distance, and y is the number of errors that are correctable.

d>2y+l (3)

The most common implementation is a class of Hamming code known as a single-

error correcting, double-error detecting (SEC-DED) code. The Hamming SEC-DED

code uses 2" -1 total bits to encode 2" -1- n information bits. An example is the

Hamming (7, 4) code in which three parity bits are added to each 4-bit word to form a

7-bit codewordl(iti2+i3+i4+p +p2+p3). The parity bits are generated in such a way as to

distinguish the location of the bit in error after decoding of the received codeword in

what is referred to as the syndrome. The equations for the Hamming parity and






18

syndrome generation are given in Table 2-2 [12]. A complete overview of ECC's is

given in [25, 26].

Table 2-2. Hamming (7,4) parity and syndrome generation equations
Parity generation Syndrome generation

P2 = i + i 3 S+ i4 = pf + if + i, + i
P2 = + + 4 S = p3 + i + i + i
P3 = + 2 + 3 = p + i3 + 1 + i4

















CHAPTER 3
NANOSCALE MEMORY DEVICES

The term nanoscale is generally used to refer to structures that have dimensions

smaller than 100 nm. Nanoelectronics, or equivalently molecular electronics, then refers

to any electronic device with critical dimensions of less than 100 nm. There is one

exception to this; the terms are limited to novel technologies of this scale. The restriction

is used to exclude CMOS technology, which is technically below the 100 nm mark in

modern processes.

This chapter presents the goals, both near-term and long-term of the semiconductor

industry, as well as the upcoming challenges in scaling silicon-based CMOS devices to

ever smaller sizes. Several promising molecular electronic technologies are then

introduced that are most applicable to memory devices and may go on to become

successors to CMOS technology.

3.1 Present and Future Scaling Challenges

The International Technology Roadmap for Semiconductors (ITRS) is a document

that is based on predictions of the state of semiconductor technology for approximately

15 years into the future [27]. It is compiled by a cooperative effort of organizations in

every aspect of semiconductor technology from manufacturers to universities. The

document tracks the device characteristics expected at each generation of technology and

defines the basic requirements necessary to achieve them.









The requirements are grouped into thirteen different areas related to the physical

properties, manufacturability, and testability of the devices. The document also outlines

any roadblocks foreseen in each of the areas that may inhibit the realization of each

generation of technology. The roadblocks are categorized as being yellow, meaning that

a manufacturing solution is known but not yet implemented, or red, which means that no

manufacturing solution is currently known.

The challenges in the red category form what is known as the "Red Brick Wall"

that represents the major hurdles that stand in the way of continuing the exponential

growth of Moore's Law. In the 2001 edition of the ITRS the "Red Brick Wall" was

predicted to occur in the 2005 to 2006 timeframe. According to the update provided in

2003, some of these obstacles have been overcome, but only enough to push the wall out

to about 2007. Of the dozens of near-term challenges that remain, some of the hardest

are dealing with the exponential increases in leakage current, making masks for the

lithography process, developing metrology tools to perform critical measurements, and

building the interconnect networks on the chips.

Assuming that engineers are able to work quickly to develop the solutions

necessary to maintain the growth predicted by the ITRS, the critical dimension in devices

will be on the order of 20 to 30 nm in only a decade (Table 3-1). Devices of this scale

will likely be fabricated quite differently than they are today. They will presumably be

built using a bottom-up approach in which large, regular arrays are self-assembled rather

than lithographically drawn. If lithography-based CMOS processes are still

implemented, they will be integrated with nanoelectronics in what has been referred to as

mixed CMOS/nano designs or NoC. NoC circuits will consist of CMOS as the









underlying structure that connects blocks of nanoscale structures together while providing

only a small degree of functionality [28].

Table 3-1. ITRS DRAM production product generations
Year of Production 2004 2007 2010 2013 2016
DRAM /2 Pitch (nm) 90 65 45 32 22
Cell Area (mm2) 0.065 0.028 0.012 0.0061 0.0025
Functions/chip (Gbits) 1.07 2.15 4.29 8.59 34.36
Gbits/cm2 at production 0.97 2.22 5.19 10.37 24.89


3.2 Nanoscale Building Blocks

To date, there have been no large-scale memory devices constructed from

molecular electronics, but many of the necessary building blocks are coming into place.

There have been prototypes demonstrated of nanoscale wires, field-effect transistors,

junction transistors, diodes, and switches [2]. There has also been some success in

small-scale integration of these devices into working circuits. For example, Hewlett

Packard demonstrated a 64-bit memory array that was constructed from a cross-bar

configuration of platinum wires 40 nm in diameter. Much research is still necessary to

continue refining the building blocks and discovering ways to integrate before usable

products can be produced.

3.2.1 Nanoscale Wires

There first useful building block was the discovery of nanoscale wires. The two

most promising are the carbon nanotube (CNT) and the semiconductor nanowire (SNW).

The CNT is composed of a cylinder of carbon atoms. The wall of the cylinder can be as

thin as a single monolayer, or can consist of several layers. The diameters range from a

single nanometer to several nanometers, and the length can be on the order of

millimeters. It is also possible for the tube to have either metallic or semiconductor

properties, depending on the helicity of the carbon lattices. CNT's are typically formed









using catalyzed chemical vapor deposition (CVD), and unfortunately, the type of

structures produced is a random event. It is not currently possible to uniquely construct

metallic- or semiconductor-type nanotubes [29, 30].

SNW's refer to nanoscale wires composed of semiconductor materials in general,

but the most common are Si, Ge, and GaAs. They have many of the same properties as

CNT's. It is possible to construct the wires with diameters as small as 3-nm and lengths

on the order of microns. The greatest benefit the SNW's have over their carbon

counterparts is that, although the same method of CVD fabrication is used, it is possible

to precisely control the composition, diameter, length, and doping of the structures. It is

also much easier to control the parallel production of SNW's [4, 29].

3.2.2 Transistors, Diodes, and Switches

In addition to nanoscale wires, it is also necessary to have active devices as

building blocks. A wide variety of transistors, diodes, and switches have been

constructed, albeit on small scales. The devices have been limited to proof-of-concept

using fabrication techniques far too complex for mass production, but they are the first

steps towards characterizing the devices and developing necessary models.

Diodes and FET's have been constructed by using the CNT's and SNW's described

above, which can be easily doped to be either p-type or n-type [29, 30]. In one

implementation, a single-walled CNT or a SNW is used as the channel of a FET and is

placed between two metal electrodes. The molecular FET's, benefited by ballistic

transport properties, have been shown to be comparable in performance to similar CMOS

devices [31]. In another example, a multi-walled CNT is side-bonded to electrodes and

an FET is formed by selectively thinning portions of the tube by removing the outer

shells of the CNT [30]. Both diodes and FET's have been fabricated by simply crossing









oppositely doped CNT's or SNW's to form p-n junctions at the intersections [2, 29].

Figure 3-1 graphically depicts each of these constructs.

Switches have been developed in much the same way as the diodes and the FET's.

Most implementations involve crossing nanoscale wires and using the intersection as a

simple two-terminal switch. The benefit of such switches is that the storage area for a bit

is merely the intersection of two wires. No periphery is necessary to program or access

the bit. A conventional device such as an SRAM or a DRAM requires as much as 100x

larger footprint for a programmable crosspoint when built with equivalent feature sizes

[2].

A B C







Figure 3-1. Nanoscale FET implementations. A) depicts a CNT being used as a FET
channel. B) shows a multi-walled CNT with various number of shells
removed. C) represents an oxidized SNW gated by a CNT.

3.3 Molecular Electronic Architectures

Currently, the greatest challenge in nanoelectronics is the ability to efficiently build

and configure reliable architectures out of the core building blocks discussed in the

previous section. There are many issues related to this including designing and

simulating an architecture composed of billions of gates, fabricating a device that adheres

to the design, and testing the circuits and repairing defects.

The design and simulation of the nanoscale circuits is beyond the scope of this

thesis, but the regularity of memory devices should make this challenge manageable and

certainly much simpler than it will be for logic devices. The complexity of the









fabrication process will require tradeoffs with the testability of the design. If the designs

remain fully deterministic, as they are today, the testing process will remain relatively

simple, although steps would have to be taken to find methods of testing such a large

number of bits in a reasonable amount of time. If the designs become completely random

in structure, the fabrication process will be greatly simplified and the testing process will

become complex. It will be necessary to map out the device post-fabrication and

determine if it is a usable part.

3.3.1 Crossbar Circuits

The crossbar architecture appears ideally suited to memory devices made from

nanoscale wires. In this design a simple 2-D mesh of wires is constructed by lining up a

number of wires in parallel to form a plane, rotating the plane by 900 and repeating the

process. There have been a number of techniques developed for doing this, including

fluid flow directed, electric field directed, biologically assisted, catalyzed growth, and

nanoimprinting assembly methods, among others [2, 28, 29].

In the fluid flow and e-field methods, the wires are lined up by the flow of fluids or

current, respectively, and the spacing is determined by the rate of the flow. Biologically

assisted processes use complementary DNA strands as "smart glue" to selectively bind

components and move them in place. In the catalyzed growth methods, a laser ablation

or CVD is used to direct the growth of nanodroplets in one direction. Finally, in

nanoimprinting, a mold is constructed using e-beam lithography and then wires are cast

into the mold.

Nanoimprinting is the only known method of making irregular arrays of wires in a

controlled fashion. The main drawback is that it has a larger pitch requirement than the









other methods. Notably, it was the method used by HP to make the prototype 64-bit

memory mentioned above.

There have been several methods devised to store the information at the wire

intersections of the crossbars. HP patented a method in which molecules are sandwiched

between the wires. The molecules can be electrochemically modified to vary the

tunneling barrier between the two wires [32]. IBM patented a similar idea except that

magnetic structures are placed between the devices [33]. In a method developed at

Harvard, there is nothing between the wires. The wires are suspended slightly apart from

one another at the intersection. To program the device, electromechanical forces are used

to attract the wires closer together. They are then held in this position by van der Waals

forces at a second potential energy minimum [34].

Using crossbar circuits and storing the state of the bits at the intersections does

have some drawbacks. First is the inability to create periodic structures, which would

have to be configured into the meshes post-fabrication. Also, the diode-resistor logic

available does not allow for signal gain. The lack of gain can be coped with by including

molecular latches in the periphery circuitry and by keeping the size of the arrays small

and using them as building blocks for larger circuits [28].

3.3.2 Micro/Nano Interface

At present, the only way to make use of the crossbar arrays is to connect them to a

microscale interface at some point. This is not a trivial task because even if a microscale

wire could be attached to each nanoscale wire, such a design would defeat the purpose by

requiring microscale pitch between wires. Most designs have dealt with this requirement

by including demultiplexor circuits in which a logarithmic number of microscale lines are

used to interface with the given number of nanoscale lines.








The approach taken by HP in making a demultiplexor was to make connections
between the microscale and nanoscale lines with a stochastic chemical process that has a
50% probability of making a connection. It was shown that by increasing the number of

address lines necessary to address n data lines from log2 n to approximately 5 log2 n

there is a reasonable probability that each bit can be uniquely addressed [35].
Another approached suggests that a decoder may be imprinted lithographically
during fabrication across which the nanoscale wires are placed. This design also made
use of a sparser decoding scheme based on two-hot coding to reduce the portions of the
array that would be lost in the case of an error in the decoding circuitry [4].
A third method has also been proposed in which the decoder is actually integrated
into the crossbar architecture. The outer edge of the array is fabricated to allow
connections to be made to microscale devices and is programmed at higher voltages than
the interior of the array. The decoder is then programmed into the periphery during the
testing phase of the circuit [28]. A diagram of each of the demultiplexor designs in
shown in Figure 3-2.


Figure 3-2. Demultiplexor designs. A) Random connections made between microscale
address lines and nanoscale data lines. B) Lithographically drawn decoder pattern.
C) Demux being encoded into the nanoscale wires on the periphery of the memory.


fttt~t~









3.3.3 Hierarchical Architectures

There are a number of compelling reasons for designing the overall architecture in

a hierarchical fashion where smaller arrays exhibit locality and work as independent

pieces of larger constructs. As mentioned above, the size of the arrays is limited by the

lack of gain in the circuits and requires that multiple smaller arrays be connected to allow

for signal latching and restoration. This is also true for interconnect purposes as

switching frequency continues to increase and the latency in routing the signals becomes

a limiting factor. A final argument for smaller arrays is a result of the lack of control in

fabricating self-assembled circuits. Smaller design should lead to greater accuracy and

fewer defects.

It also makes sense to use a hierarchical architecture from a design perspective. It

will simplify the process greatly to have multiple levels of abstraction. Again, this is

most applicable to logic designs, but it will play a part in memories as more logic is

integrated either for computational purposes or to simply perform all the necessary

mapping, testing, and repair processes on the arrays.

3.3.4 New Reliability Concerns

It has been implied throughout this chapter that self-assembled molecular

electronics will have a number of new concerns with regard to defects during fabrication.

The first problems stemmed from the inability to control the properties of CNT's. This

was exacerbated by the further inability to deterministically control the alignment and

spacing of the nanoscale wires in the construction of the crossbar arrays. The limits of

cleanroom technology will also come into play as devices sizes shrink to on a few

nanometers. The consequence of all these factors is a defect rate expected to be on the

order of 5% [2, 5, 28].






28


Even if the initial defects are dealt with during the manufacturing process and a

working part is completed, there will inevitably be a much higher rate of both permanent

and transient faults during the runtime of the devices. The extremely small size of the

structures will make them more susceptible to the random single-event effects from high-

energy particle strikes. The only way to create reliable parts will be to integrate fault

tolerance circuits into the designs to deal with the dynamic faults.














CHAPTER 4
HIERARCHICAL FAULT TOLERANCE FOR NANOSCALE MEMORIES

This chapter introduces hierarchical memory architectures with several possible

forms of fault tolerance at each of the levels. The goal is to develop generic memory

architectures with new, more powerful combinations of fault tolerance mechanisms that

take advantage of the intrinsic properties of molecular electronics to overcome what is

expected to be a very high defect rate. Several possible implementations will also be

suggested based on previous work in the area.

4.1 Benefits of a Hierarchical Approach

It makes sense to use a fault tolerance scheme that is hierarchical given that the

underlying memory structure is expected to be hierarchical. In fact, it has been found to

be beneficial to do so. The defects that occur in a memory device have been related to

the "birthday surprise" problem [36]. The problem calculates the expected number of

independent trials necessary to have one of a set of equally likely results occur twice. In

the case of birthdays, the outcome is two persons sharing the same birthday. The

expected value in this case is a surprisingly low 24 people. The value is also low for

memories if only single level fault tolerance is used. It was shown in [37] that a 16-Mbit

memory chip with 1,000 random failures has greater than a 98% chance of having two

errors line up.

The benefit to having a second level of fault tolerance is that it effectively removes

the first sets of "matches." For example, if a SEC-DED code is used as a first-level

mechanism and row redundancy as a second, then when two bits line up in a word to

29









cause a failure the row can be replaced by the second level of fault tolerance. It is as if

the persons with matching birthdays were taken out of the room. The effects of each

fault tolerance mechanism work constructive to perform better together than either would

individually. This was termed synergism in [37] and is the basis for the configurations to

be presented in the remainder of the chapter.

4.2 Proposed Memory Organization

The architecture used in this thesis is based on the specific set of conditions

applicable to molecular electronics that were specified in Chapter 3, and which can be

summarized as follows.

* The memories will be built hierarchically from relatively small, regular arrays of
nanoscale wires.

* The arrays will be self-assembled with a high degree of randomness and an error
rate on the order of 5% [2, 5, 28]. The arrays must function even with defects
present.

* Locality is important; the fault tolerance mechanisms should work independently at
the array level and only signal higher-level structures when local resources are
exhausted.

* A high degree of configuration and testing will be necessary post-fabrication that
will necessitate logic structures in the memory arrays.

4.2.1 Architecture Overview

The basic structure of the memory organization to be considered is shown in

Figure 4-1. Much as the DWL architecture added an extra level of word line decoding, this

hierarchical architecture extends it further to allow for more sublevels. The lowest level,

which will be referred to as level-0, consists of small, regular arrays of nanoscale wires

such as CNT's or SNW's. There is also a logic component that will be called a

processing node-PNO in this case. The processing nodes are used to map out the








structure of the arrays during the testing and configuration phases, and later they are used

to implement the fault tolerance mechanisms in the array.



t7





and dynamic fault tolerance. An arbitrary number of higher levels can exist,








PN o o














and configuration of the part. It is then used at runtime to implement fault tolerance
is presen t---r out | mpi I drng

PN2 : "1 SPARE-2 'r :











and configuration of the part. It is then used at runtime to implement fault tolerance

mechanisms throughout the life of the part. If there were another level present, it would

be referred to as level-2 and would consist of arrays of level-1 structures. The hierarchy









can continue in this fashion, which is similar to previous hierarchical memory designs

[38, 39]. It has been shown that in terms of timing latency the optimal number of levels

in the hierarchy increases slightly with memory size and inversely with minimum feature

size, but is on the order of five to ten [40].

4.2.2 Possible Implementation Strategy

As mentioned in Chapter 3, the higher level structures need not be implemented

with nanoscale devices. It may be the case, at least initially, that NoC architectures are

employed and lithographically-based CMOS structures are used in conjunction with

nanoscale arrays. The CMOS structure would provide the logic functionality necessary

for the mapping and fault tolerance, as well as basic infrastructure such as power, ground,

and the clock signal.

long-liner


IoniXIim,

nanoBlock
'MV ..... W

m e n @ switch-block




Figure 4-2. A diagram of the nanoFabric architecture

Examples of such a NoC approach are the nanoFabric architecture [3], shown in

Figure 4-2, and the similar array-based approach in [4]. In these designs, the lowest level

consists of arrays of nanoscale wires. The next level then connects these arrays into a

larger array called a cluster. The top level is an array of clusters with long nanoscale

wires connecting the clusters. Inside each cluster, the sub-arrays allow for both









programming and routing to be done entirely at the nanoscale. Through simple routing

techniques, arrays can not only be used as a memory device but can be configured to

implement full logic functionality.

4.3. Fault Tolerance Mechanisms

The basic static and dynamic fault tolerance mechanisms in use today were

introduced in Chapter 2. It is assumed, however, that these will not be sufficient for

nanoscale memories. More advanced techniques are available and are presented in this

section.

4.3.1 Hierarchical Built-in Self-Test and Self-Repair

The need for BIST and BISR circuits is suggested by all of the conditions outlined

in the previous section. The underlying assumptions that molecular electronic circuits

will be very large, have high bit error rates, and require a great deal of functionality to be

configured post-fabrication suggest that ATE alone will not be capable of testing and

configuring the structures. The integrated mechanisms for test and repair are also

necessary to contend with the dynamic defects that occur throughout the lifetime of the

devices. They function synergistically with ECC's and other forms of fault tolerance by

replacing defective devices that in many cases do not cause faults individually, but put a

portion of memory at high risk of faults by lining up with future defects.

In the proposed design, the reconfiguration processes are similar both during the

device testing and at runtime. The BIST/BISR circuitry used to map out the functionality

of the arrays is integrated into the PN's and continues to be employed at runtime when

defects are detected either through error detection circuits or at scheduled servicing

intervals.









The reconfiguration options of the repair circuits are limited by the speed of the

interconnections in the devices. It is not feasible to swap in a spare row from a distant

array, for example, as it would require a decrease in the system clock rate. The

hierarchical approach handles this by allocating spare devices to each array and allowing

the local PN to conduct the sparing process. It is not until the local PN has exhausted the

available resources that a signal is sent to the next level up in the hierarchy. When this

occurs, the entire lower-level structure may be swapped with a spare.

4.3.2 Advanced Error Correction Codes

ECC's were introduced as a method of dynamic fault tolerance that has been

implemented in some modern, high-reliability memories. The most common codes are

capable of either detecting errors or possibly correcting one error. There are much more

powerful linear block codes possible, but because of the complexity of the encoding

process double-error-correcting and triple-error-detecting is the most powerful one

considered. A summary of the original codes, as well as several additional codes that are

considered in this thesis is given in Table 4-1.

There are other forms of ECC's that do not fall into the category of linear block

codes, such as Reed-Solomon, Golay, Turbo, and RAID codes. Normally, these forms of

error correction are not applied to memory devices. The one exception is RAID parity,

which has been used by IBM in high-reliability Chipkill server memory [43, 44]. The

benefit of RAID is that, unlike typical ECC's, it is designed for inter-block redundancy.

When used in conjunction with ECC's, it allows for one more than the number of bits

correctable by the ECC to be tolerated. The memory is configured with an additional

level-1 array in each row or column of level-1 arrays. The additional array is used to

store the parity of each bit in the other arrays. In this way, even if an entire array were to









fail, the data in error could be reconstructed by taking the parity of the other arrays so

long as the other arrays do not contain uncorrectable errors at the same relative location.

Since the RAID parity is the only alternate form of dynamic redundancy typically

applied to random access memory devices, it is the only one that will be included in this

thesis. The additional requirement for the use of RAID redundancy is knowledge of

more than the number of correctable bits being in error. This requires not only that an

ECC be implemented in addition to the RAID code, but that the ECC is capable of

detecting at least one more error than it is able to correct. For this reason, only the codes

in Table 4-1 that are marked with an asterisk* can be used in conjunction with the RAID

parity.

Table 4-1. ECC summary for the proposed architecture.
Code # Redundant Bits

Parity* 1
SEC log2 N+1
SEC-DED* log2N+2
DEC 2og2 N+ 2

DEC-TED* 2og2 N+3














CHAPTER 5
MEMORY SIMULATION AND RESULTS

The complexity of the fault tolerance mechanisms implemented in the proposed

architecture ruled out the derivation of analytical solutions to describe the performance of

the systems. The size of the memories involved also made the Monte-Carlo simulation of

the systems non-trivial. This chapter describes how the simulations of the various

architectures have been accomplished and validated. The results of the simulations are

included and analyzed with the goal of providing insight into optimal memory

configurations in terms of reliability for a given amount of redundancy overhead.

5.1 Simulator Architecture

The concept of developing simulation tools for analyzing the performance of

memory architectures is not new. A variety of software packages are available for

running many types of simulations. These packages are limited, however, in their

support for simulating the states of very large memories. The requirement that multi-

gigabit and even terabit sized memories need to be simulated in a reasonable amount of

time, not once, but for thousands of iterations, rules out the generic simulation packages

that are designed more for modeling the states of tools in a factory than the states of

billions of nodes.

To obtain the level of efficiency necessary, a very specialized simulator that

implements only the desired functionality has been developed. The Perl programming

language was chosen for the development environment since it offers the benefit of

efficient, easy-to-use data structures, while maintaining adequate speed at runtime. Perl









also provides a means of easily implementing a graphical interface with the Tk expansion

module and eliminates the need to port and/or recompile the simulator code to various

platforms that might execute the simulations. The individual components are described

in detail in the following sections, and the source code has been included for reference in

Appendix B.

5.1.1 Simulator Components

The process of simulating the memories can be broken down into three steps. The

first is the generation of memory configuration parameters for the simulator, including

the size and arrangement of the memory, the hard and soft error rates, and the fault

tolerance mechanisms to be supported. The next step is the simulation of the memory for

a reasonable lifetime, which is typically considered to be 100,000 hours. The final step is

the analysis of the simulation results.

The bulk of the complexity lies in the simulator itself, which can be further broken

into three steps. The first of these is the generation of failures throughout the lifetime of

the memory. It would be intractable to store the complete state of all bits being

simulated. Instead, a form of statistics called order statistics is used to generate, in order,

the occurrence of failures for the entire lifetime of the part before any simulation takes

place. The second step is the process of simulating and recording the actions taken by the

memory controller to each of the failures. The final step is integrated into the simulation

process and involves the check-pointing of the state of the memory at predefined polling

intervals for off-line analysis during the results analysis.

The simulator is constructed in a modular fashion based on the categories just

described. The programs for input parameter generation, simulation, and analysis are

implemented as shown in Figure 5-1. Each module is a standalone program and










communication occurs through the use of files, which allows for the output of each to be

saved and reused in subsequent analyses.


In s Input
Parameter Input Paiamgters




Inputs




Selected
Inputs

C onlroller











Figure 5-. Hig-level simuls at aa l
Analysis ntervals MertW status
Of Results / S' ?^


Figure 5-1. High-level simulator data flow diagram.

5.1.2 Simulation Interface

There are two interfaces provided to the simulator. One is a command line version

available for use in running batch processes of several configurations for which input

parameters have already been chosen and inserted into a batch file. The other is a

graphical version that provides tools for configuring the parameters, executing the

simulation, and analyzing the results.

A screenshot of the graphical interface is shown in Figure 5-2. The top section of

the interface provides a text area that is used for displaying results of simulations and













configuring the batch execution scripts. Basic editing and searching capabilities are


supported, as well as a tabbed notebook feature that allows for multiple files to be


opened.



File Edit Search Help
time l-1-0-8-8-0-1024-36-0-0-01-0-100000-1000-2-10-10out
L2Row: 1 L2Col: 1 L2Red: 0
LiRow: 8 LICol: 8 L1Red: 0
LORow: 1024 LOPlane: 8 LOCol: 128 LORRow: 0 LORCol: 0
MEM SIZE: 71,303,168 ECC SIZE: 4,194,304 SPARE SIZE: 0 USABLE SIZE: 67,108,864
OVERHEAD: 6.25
RAID: 0
NEC: 1
LO SUE LO HUE
TIME MEAN MIN MAX STDV MEAN MIN MAX STDV
0 0 0 0 0 0 0 0 0
1000 1 1 2 0 0 0 0 0
2000 7 7 8 0 0 0 0 0
3000 17 16 18 0 0 0 1 0
4000 31 29 33 1 0 0 3 0
5000 48 46 52 1 0 0 3 0
6000 70 66 76 2 1 0 3 0
7000 95 91 102 3 1 0 3 1
8000 124 120 132 3 1 0 3 1
9000 157 151 167 5 1 0 4 1
10000 193 187 207 6 2 0 4 1
11000 234 228 248 6 2 0 4 1
12000 279 269 295 7 3 1 5 1
1 3nnn 37 31 344 q1 1



i.... | i. F _, [ .. [ -- '- .... -

:" ,, I, :,





Figure 5-2. Simulation interface with sample results


The lower portion of the interface is also tabbed. The first tab contains settings


widgets used for configuring the architecture of the memory to be simulated, which


includes all of the input parameters given in Table 5-1 (see 4.2.1 for the Architecture


Overview). There is an additional option for enabling verbose output, which provides an


increased level of detail during the simulation. There are also a range of labels that are


dynamically updated to reflect the overall memory structure described by the input


parameters. The labels display the distribution of bits in the memory, as well as the


memory-level MTTF based on the hard and soft error rates that have been defined. The


tab for simulation settings is much simpler; the only inputs are for the lifetime of the









memory, the polling interval, and the number of complete trials to be executed. The final

tab is used for displaying summary information in the form of statistical box-plots, which

are useful for comparing the results of multiple different configurations.

5.2 Simulation Component Details

This section covers each of the simulation components in greater detail, including

the algorithms implemented and the inputs and outputs.

5.2.1 Parameter Generation

The first program in the simulator is used for input parameter generation. It was

designed to ease in the generation of the many input parameter values to the memory

simulator. It takes as input a listing of possible simulator inputs from Table 5-1 along

with a range of values for each. Then, based on criteria such as percentage of redundancy

overhead and possible fault tolerance mechanisms, the program generates all possible

input configurations. The algorithm simply iterates through all combinations of the

values for all parameters. Then, for each combination found that is within the search

boundaries, a line is written to the output file that contains all the parameter information.

The output file from the program shows all possible combinations of the parameter

values that fit the search criteria. The file is typically quite large with a high degree of

redundancy and so requires some manual filtering by the user to narrow down the list of

possibilities to a representative few. The file is composed of tab-separated values so it

can be imported into a spreadsheet program such as Excel and sorted and filtered to assist

in choosing the test cases. The final selections are then copied into a batch file for use in

automated executions.









Table 5-1. Simulator inpu parameters
Input Parameter Description

Level 2 Row The number of rows of level-2 devices
Level 2 Column The number of columns of level-2 devices
Level 2 Redundancy The number of spare level-2 devices
Level 1 Row The number of rows of level-1 devices
Level 1 Column The number of columns of level-1 devices
Level 1 Redundancy The number of spare level-1 devices
Level 0 Row The number of rows of level-0 devices (bit level)
Level 0 'k' Value The number of columns representing a word (bit level)
Level 0 Plane The number of words in a row
Level 0 Row Red. The number of spare level-0 rows
Level 0 Column Red. The number of spare level-0 columns
Errors Correctable The number of errors correctable with ECC's
RAID/Chipkill Whether or not parity implemented across level-1 rows
Usable Lifetime The lifetime for which to simulate (hours)
Poll Interval The interval at which to record the memory state (hours)
Hard Error Rate The bit-level hard error rate (FIT)
Soft Error Rate The bit-level soft error rate (FIT)



5.2.2 Simulation

The second program performs the simulations of the device configurations

developed by the parameter generation. The simulation program is quite complex and

can be broken down into three subcategories. The first of these deals with the generation

of the failure times, types, and locations throughout the lifetime of the memory. Once

generated, these failures are fed into the memory controller simulation which records the

actions taken by the controller in response to each of the failure events. While the

simulation is in progress, the memory state is recorded at predetermined polling intervals.









Failure generation. In the initial versions of the simulator, an attempted was made

to model the memory at a bit level. The hard and soft failures for each bit were generated

and ordered in a linked-list structure. This method required the entire memory state to be

kept and a sorting algorithm be used to ensure that the errors were processed in order. As

the total number of bits increased to over one million, this method was soon determined

to be impractical. Several alternative methods of simulation were considered, such as

those based on Markov models and order statistics [45-47].

The current failure generation module makes use of order statistics to generate the

hard bit failure times for the entire lifetime of the memory at the start of the simulation.

The equation for the ith failure time, T, is given in (4), where F'(x) is the percent point

function of the time to fail, Uis a sequence of uniformly distributed random variables on

the interval (0,1), n is the size of the memory, and r is the desired number of ordered

failures to be generated [47].


= F-1 I lU (4)
I J=1, ,r

Each of the failure times is randomly associated with a failure type based on the

probability distribution of failures in the type of memory being simulated. Each failure is

also assigned to a random location in memory. The result is a list of failures with the

location parameters for each of the levels defined by the number of the row or the

column. A zero is used to represent an entire row, column, or device failure. This is

illustrated in Figure 5-3.









Index Type Fail Time Physical Location
Level 2 Level 1 Level 0
Row Col Row Col Row Plane Col
1 B 755.73910 1 1 1 1 4 1 1
2 R 1587.40267 1 1 1 1 7 0 0
3 C 2619.78800 1 1 1 1 0 3 2

Figure 5-3. Sample failure generation output

The occurrence of soft bit failures need not be simulated at all as they can be

estimated from the number of hard uncorrectable errors in the system at each of the

polling intervals. The equation for the SUE is given in (5), where a is the SER, t is the

time in hours, G is the number of good bits in words with the number of correctable bits

in error, and Bnec(t) is the number of words in the memory with the number of correctable

bits in error at time t. The equation finds the probability that the soft errors in the system

will affect one of the good bits in a word that already contains nec bad bits.

art
SUE(t) = xGx B,,c (t) (5)


Simulation. The memory is simulated by iterating through each of the failures in

the fail list and determining whether the fault tolerance mechanisms in the memory are

sufficient to avoid uncorrectable errors of the data were to be accessed. This requires the

interactions of the failures be determined to ascertain when and how the defects line up.

It is also necessary during the simulation process to store information about the

system. This is done in the data structure shown in Figure 5-4. The top level of the

structure is a hash with the key determined by the level-2 row and level-2 column values.

The only information stored at this level is the status of the level-2 device, such as

whether it is a spare that has been swapped in or if it is has been swapped out and is

considered dead. The next level consists of two hashes, the first has a key derived from

the level-1 row and level-1 column and the other is just the level-1 row. In the case of the










row and column, information local to each of the level-1 arrays is stored, which includes

the status of the array, the number of spare rows and columns that have been used, the

number of words with 1 to nec bad bits, and the number of uncorrectable errors. The

entry that stores the start of the chain indexes the first failure in the fail list for the array

and will be discussed in more detail shortly. The hash that is indexed by only the level-1

row is used only in the case that RAID parity is implemented across level-1 arrays and

stores additional information necessary for calculating the number of HUE's.


System Status Table
%SST { $L2Row: $L2Col



Status Level 1 Key r Level 1 Rw Key
{'ST'} {$LlRow:T:$LCol} {LlRTow) j

Status I I Chain Start
'ST } 'CH' I

Used Spare Rovs I ard Uncorrectadel
{ 'SR' } I Errors I
{, UE }[]
Used Spare Cols
{ 'SC' }

Chain Start
( 'fCH' }

Bits Bad Count
{,'ED' }[ ]

Hard Uncorrectale
Errors
( 'UE' }
Figure 5-4. Hierarchical hash data structure used in the simulator

The interaction of the failing bits is modeled by "chaining" them together by level-

1 array. The chaining process, which is similar to that used in Libson and Harvey [48],

avoids having to search through all previous failures in the fail list for each new failure.

Chaining is done by first accessing the start of the chain in the simulation data structure

to determine if there has been a previous failure in the same level-1 array as the current









failure. If the current failure is the first one in the array, then its index is recorded as the

start of the chain for the array and it is examined to see if it results in a HUE. If the

current failure is not the first failure in the array, then the index for the first failure is

retrieved and the two failures are checked to see if they line up to cause a HUE. The

index for the current failure is also entered into the chain entry in the fail list at the first

failure. If at a later time, another failure occurs in the same level-1 array, the index to the

first failure is retrieved from the data structure and the second failure is retrieved by

looking at the first failure's chain entry. The current failure's index is then inserted into

the second failure's chain entry and the three failures are checked for the possibility of

lining up to cause a HUE. The process continues in this fashion for all failures with each

only being processed with those in the same level-1 array without the need to ever scan

the entire fail list.

An example is given in Figure 5-5, which represents a somewhat simplified failure

list. In this case, the memory is configured with eight rows with four words of four bits

each at level-0. The level-1 is composed of an array of three rows and two columns of

level-0 arrays. The level-2 has been reduced to a single instance for simplicity and all

errors only affect a single bit. Highlighting has been used throughout the list to identify

the level-1 array that each error affects. It has been repeated at the top of the figure to

represent the start of the chain that would be stored in the simulator's data structure. A

chaining column has also been inserted to represent how the failures are connected. In

this case, there is only a single case in which more than one error occurs in a word. It is

in the level-1 array at the location (2,1) in row three and word four; it has been designated

by circling and connecting the failures in the figure.














Index
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20


1 m =i 6=2 FM=3


Level 1


Fail Time
26.60175
108.60437
303.70921
416.11924
432.50310
514.62416
596.21003
641.83216
710.29693
776 60665
819 03183
927.44709
1045.64624
1087.35102
1249 80435
1280 70762
1316.46148
1510.99605
1611.46515
1740.21415


Level 0
Row Plane Col
4 2 2
5 2 2
5 3 2
3 4 3
5 4 3
8 1 1
3 2 4
5 4 3
3 2 1
4 4 1
2 2 4
3 4 1
6 3 3
6 1 3
4 3 3
7 4 2
7 3 2
8 4 1
3 1 2
2 3 3


Figure 5-5. Example of failure chaining process in simplified fail list. Highlighted chain
shows two bits in the same word lining up to be in error.

The simulation process becomes somewhat more complicated in the case that

RAID redundancy is enabled across level-1 arrays. The first difference is in the

determination of the HUE's within an array. In the previous case, the HUE's could be

determined locally within each level-1 array. When RAID is enabled, however, the

failures that line up within the arrays are tracked but not recorded as HUE's. Instead,

there is a second chaining process that chains together the low-level failures by level-1

row.

The chaining process used in the case of RAID is the same as that used for the bit-

level errors except the failures are compared on a word level to see if multiple words are

uncorrectable in the same relative position across a row of level-1 arrays. It is only when


Start of chain:

~=m q Pp=%l


Chain
10
6
11
7
15
8
12

13
17
--
14
18

16
19

20


r


0PS 0









two uncorrectable words line up that the RAID parity cannot be used to regenerate the

words in error and HUE's are recorded. Only the last failure to line up to cause a HUE in

each word needs be chained together, and there are two HUE's for the first occurrence

and then each subsequent word that lines up results in another single HUE. The HUE

information is tracked in the hash structure indexed by the level-1 row and supersedes the

information in the other hash when calculating the number of HUE's for the system.

Memory state recording. The recording process logs the number of hard and soft

uncorrectable errors in the memory. The result of the logging is an overview of the

number of hard and soft uncorrectable errors present throughout the lifetime of the

memory, as well as a log of the number and type of spares used.

The logging is done by iterating through the data structure that maintains the

number of hard uncorrectable errors in the system and calculating a total for the entire

memory. It is also during this polling process that the number of soft uncorrectable errors

is estimated using (5). When multiple iterations are run, which is necessary to obtain

accurate results, statistical measures such as the mean, minimum, maximum, and

standard deviation of the UE values is calculated and recorded. The goal for this thesis

was to run 5,000 iterations of each simulation to assure statistical significance based on

the law of large numbers [47, 48].

5.2.3 Results Analysis

The results of the simulations are derived from the log file kept during the polling

process, as well as optional verbose logs kept during the simulation. The log file kept

during the polling process gives values for the number of hard and soft uncorrectable

errors in the system, and optionally the number and type of spares used. A portion of a

sample log file is displayed in Figure 5-2. The name of the file is prefixed with "time_"









followed by the values for the parameters that define the simulation and as described in

Table 5-1. In the example shown, the first lines of the file give the parameter values for

the memory in an easy-to-read format. This is followed by the results for the number of

HUE's and SUE's in the system at every 1,000 hours up to the memory lifetime of

100,000 hours. The mean, minimum, maximum, and standard deviation are given for

cases that the simulation is executed for multiple trials. When the graphical version of

the simulator is used, it is possible to plot the statistical summaries in a box-plot format.

5.3 Simulator Validation

It is not possible to obtain or derive analytical models for the proposed memory

architecture model in its full complexity, so the validation of the simulator has been

approached as follows. First, a memory configuration that implements ECC without

sparing has been evaluated against an analytical model for memories with SEC [14]. The

results in Table 5-2 are averaged over 5,000 trials and show small differences in the

MTTF for small memory sizes for which the analytical model was initially used.

Table 5-2. Simulator validation to SEC analytical model for small memory sizes
Memory Size (b) Simulator MTTF Analytical MTTF Difference (%)

256x39 2.025x1012 2.010x1012 0.739
1024x39 1.014x1012 1.000x1012 1.37
256x72 1.107x1012 1.100x1012 0.657
1024x72 5.455x1011 5.490x1011 0.640



A comparison was also made against the analytical model when extended to larger

memory sizes. These results are shown in Figure 5-6, and although there is a larger

percentage difference in the results, it is clear that the simulator closely tracks the model.







49



It should also be noted that the model was admittedly not entirely accurate for large


memory sizes and should not be used for memories over 1-Gbit [14].





1800

1600

1400

1200

1000

S800

600

400

200


0 10 20 30 40 50 60 70
Memory Size (Mb)
-+-Analytical 32-bit words Simulator 32-bit words
---Analytical -64-bit words Simulator 64-bit words

Figure 5-6. Simulator validation to SEC analytical model for large memory sizes

Configurations with ECC and sparing have also been evaluated though the analysis


of simulation trace outputs and by comparison against a previously developed simulator


that implemented different algorithms, data structures (event-driven vs. chaining),


random number generation (per-bit Poisson distribution vs. ordered statistics), and


programming language (C vs. Perl). The results for a 1-Mb memory configuration


implementing SEC and DEC run in each of the simulators showed an average difference


of 6.64 and 8.94%, respectively, over 5,000 trials.


5.4 Simulation Results

The results shown in Figures 5-7 to 5-9 are from a set of experiments used to


compare the performance of various fault tolerance mechanisms based on the percentage









of bits used for redundancy. The simulations are based on multi-level memory

organizations with 1-Gbit of usable memory size, which are composed of 1-Mbit

modules at the lowest level. The results are the average of 5,000 trials and show the

number of HUE's in the system after 100,000 hours (-11.4 years) of system life.

It is immediately apparent that without the use of a Hamming ECC code, the

performance is relatively poor, even if RAID and spare devices are used. It can be seen

that for small word sizes SEC with spares has better performance than DEC initially, but

the benefit is lost later in the memory lifetime as the spares are exhausted. It is also

notable that the SEC implementations using RAID have a significant performance

advantage for a given amount of overhead, although this is at the tradeoff in complexity

and possible latency increase of implementing inter-block redundancy.

The only configurations to protect the memory from all errors regardless of word

size are DEC with a small number of spares, which requires a significant overhead. The

next closest configuration in terms of performance is a combination of SEC with both

RAID and spare devices implemented. Again, this configuration requires a high degree

of overhead but it is less than that needed by DEC. This provides a strong case for the

use of hierarchical fault tolerance mechanisms as a way of optimizing the reliability of

devices at the lowest cost in terms of the percentage of devices used for redundancy. The

synergism allows for multiple, simpler mechanisms to outperform more complex

techniques.









51







1 E+06



1 E+05-



1 E+04



1 E+03



1 E+02 -



1 E+01



1E+00 '
0 10000 20000 30000 40000 50000 60000 70000 80000 90000 100000
Time (hours)

---NEC-RAID-spare-11% ---NEC-RAID-spare-22% SEC-11% SEC-spare-22%
-W-SEC-RAID-22% -*-SEC-RAID-spare-22% -'-DEC-22% DEC-spare-22 4%

Figure 5-7. The HUE's for 1-Gbit memory with 64-bit word size using various

percentages of fault tolerance






1 E+06



1 E+05



!" 1 E + 0 4 _----------- ---------------------- ----------------------------





2 1 E+02





1 E+01




1 E+00
0 10000 20000 30000 40000 50000 60000 70000 80000 90000 100000
Time (hours)

SEC-625% SEC-spare-125% -k-SEC-RAID-125% -*-SEC-RAID-spare-125% --DEC-125% DEC-spare-16%

Figure 5-8. The HUE's for 1-Gbit memory with 128-bit word size using various

percentages of fault tolerance









































0 10000 20000 30000 40000 50000 60000 70000 80000 90000 100000
Time (hours)
NEC-RAID-spare-7% SEC-3 5% SEC-spare-7% -- SEC-RAID-7%
-*-SEC-RAID-spare-7% -4-DEC-7% DEC-spare-12 5%



Figure 5-9. The HUE's for 1-Gbit memory with 256-bit word size, using various
percentages of fault tolerance














CHAPTER 6
CONCLUSION AND FUTURE WORK

6.1 Conclusions

Continued advances in the fabrication of molecular structures have led to

speculation that molecular electronic devices may be incorporated with or even supplant

CMOS technology in the realm of ultra-dense logic and memory systems. These systems

will be manufactured using self-assembly techniques, which will require entirely new

methods of design, simulation, fabrication, and testing. One of the consequences of the

self-assembly process is an increased hard error rate due to physical defects and soft error

rate from an enhanced sensitivity to noise and radiation effects.

In this thesis a generic, hierarchical fault tolerance architecture has been proposed

as a method of dealing with the increased error rates. The architecture incorporates many

of the attributes predicted to be inherent in molecular electronics, such as a hierarchical

design, randomness in device assembly and interface, and the need for locality of

communication. The key concepts from this architecture are as follows:

* Reuse at runtime the logic circuitry necessary for static address mapping

* Perform dynamic reconfiguration at multiple levels in a localized, hierarchical
fashion

* Incorporate intra- and inter-device ECC's with reconfiguration

The performance of the architecture has been modeled against existing

architectures through a specially constructed simulation tool. The results of the analysis









show that there is an opportunity for the advancement of fault tolerance techniques

through the synergistic effect of multiple, hierarchical mechanisms.

6.2 Future Work

There are a number of possibilities for future work in this area molecular

electronics. One of the first steps that must be accomplished is the development of

accurate models from the device level through the architectural level of abstraction. As

models are developed and validated by empirical data, then the requirements in terms of

fault tolerance can be fully understood.

There are some additional metrics necessary to completely model the performance

of these new fault tolerance mechanisms that will only become available with the

fabrication of larger systems and the development of accurate models. One of the most

important is the overhead of the mechanisms in terms of latency. Unless molecular

electronics ushers in a major shift in the computer architecture paradigm, this will remain

a critical factor as there is an ever increasing performance gap between memory and logic

devices [20].

It will also be necessary to examine more complex fault models than are currently

implemented in the simulator. Examples include fault clustering [49], infant mortality,

and vintage learning [47, 48]. It is also useful to model the devices from a software

perspective. If the faults occur in memory but are never accessed or are overwritten with

new data before being accessed, then they are not observed as errors [50].

There are several other forms of fault tolerance that have not been covered in this

thesis that may prove to be beneficial. One of the most researched is based on the

redundancy methods originally proposed by von Neumann. In his work, von Neumann

suggested a multiplexing technique in which each wire is replaced with a bundle of wires









and a majority vote is used to determine the value of the wire [6, 7]. This concept has

been expanded upon to allow for multiple, redundant components in what is termed

R-foldmodular redundancy. A special case is Triple Modular Redundancy (TMR), which

uses three redundant components [7]. Again, a majority vote is used to determine the

final value. Other techniques involve the use of asynchronous cellular arrays [51], fault

alignment exclusion [52], error masking [53], and layout modification [54].

Another avenue of exploration is the area of defect tolerance, or the ability to create

functional systems in the presence of defects. The location and type of all the defects in

the system is determined through a testing process and the system is designed to avoid

using those devices. The most famous implementation of defect tolerance is the HP

Teramac supercomputer, which is fully functional with a 3% defect rate [41, 42]. The

logic for the entire computer system is encoded in memory and functions through the use

of simple look-up tables (LUT). The LUT's are connected hierarchically in a fat-tree

topology. The fat-tree allows for a very high communication bandwidth and gives the

compiler the ability to easily map a wide variety of custom architectures onto the system,

including a BIST architecture that can map around defects.













APPENDIX A
MODERN MEMORY DEVICES




r J, i, 1'

-F PM F ,^" '.. ,.- **




















APPENDIX B
SIMULATOR SOURCE CODE

B.1 Initialization and Variable Declaration

use Cwd; #provide cross-platform file access
use Fcntl;
use GD::Graph::boxplot;
use Math::Random qw(random uniform random uniform integer);
use Math::Round qw(nhimult nearest);
use Statistics::Descriptive;
use strict;
use Tk; #use Tk GUI tools
use Tk::BrowseEntry; #use Tk Combobox widgets
use Tk::Dialog; #use dialog boxes
use Tk::NoteBook; #make use of tabbed notebook widget
use Tk::TextUndo; #use textboxes with undo/redo

##########################################################
# Variable declaration
##########################################################
my ($currentPage, $execCnt, $fileNum, $fName);
my ($blAcc, $b2Acc, $crAcc, $raAcc, $sueAcc, $ueAcc);
my (@fileName, @files t);
my (%pageName);

# Common variables for simulator
my ($her, $kVal, $LOCol, $LOPlane, $LORow, $LORedCol);
my ($LORedRow, $L1Col, $LlKey, $L2Key, $LlRed, $LlRow);
my ($L2Col, $L2Red, $L2Row, $lifetime, $maxTrials);
my ($nec, $path, $pollInterval, $raid, $rVal, $ser);
my ($verbose);

# Variables reset for each simulation
my (%SST); #used to track array status
my ($count); #counts number of failures generated
my (@fails); #array with all failure info

# Statistic arrays for determining when error rates
# are calculated to desired degree of accuracy
my (@herLOStatData, @serLOStatData);
my (@herLlStatData, @serLlStatData);
my (@herLOPlotData, @serLOPlotData);
my (@herLlPlotData, @serLlPlotData);

$path = cwd;
# End variable declaration











# Default values
$L2Row = 1; #rows of L2 devices (cards)
$L2Col = 1; #columns of L2 devices (cards)
$L2Red = 0; #number of spare L2 devices (cards)
$LiRow = 8; #rows of L1 devices (chips)
$LlCol = 8; #columns of L1 devices (chips)
$LiRed = 0; #number of spare L1 devices (chips)
$LORow = 1024; #rows of LO devices per plane (cells)
$kVal = 128; #cols of LO devices per plane (word size in bits)
$LOPlane = 8; #number of words the LO arrays are split in to
$LORedRow = 0; #number of spare rows of LO devices (cells)
$LORedCol = 0; #number of spare columns of LO devices (cells)
$nec = 1; #how many bits are correctable with ECC
$raid = 0; #extra col of arrays added for RAID across blocks
$lifetime = le5; #how much system life should be generated
$pollInterval = le3; #freq the state of the memory is taken
$her = 2; #hard error rate in FIT/bit
$ser = 100; #soft error rate in FIT/bit
$maxTrials = 1;
$execCnt = 0; #number of complete executions that have completed

# Determine number of bits used for error correcting codes
# 'k' data bits and 'r' check bits are calculated
$rVal = $nec (log($kVal)/log(2) + 1);
# add one more check bit for RAID with NEC or SEC
($nec <= 2 and $raid ? $rVal += i: 0);
$LOCol = $kVal + $rVal;

# Total memory size is size of all usable and spare memory
my $LOUsableSize = $kVal $LORow $LOPlane;
my $LOECCSize = $rVal $LORow $LOPlane;
my $LOSpareSize = ($LORedRow $LOCol $LOPlane)
+ ($LORedCol $LORow);
my $LOMemSize = $LOUsableSize + $LOECCSize + $LOSpareSize;

my $LiUsableSize = $LOUsableSize $LiRow $LlCol;
my $LlECCSize = ($LOECCSize $LiRow $LlCol)
+ ($raid $LOMemSize $LiRow);
my $LlSpareSize = ($LOSpareSize $LiRow $LlCol)
+ ($LOMemSize $LiRed);
my $LiMemSize = $LiUsableSize + $LlECCSize + $LiSpareSize;


my $totalUsableSize
my $totalECCSize =
my $totalSpareSize

my $totalMemSize =


= $LiUsableSize $L2Row $L2Col;
$LiECCSize $L2Row $L2Col;
= ($LlSpareSize $L2Row $L2Col)
+ ($LiMemSize $L2Red);
$totalUsableSize +
$totalECCSize +
$totalSpareSize;


#Calculate MTTF from hard and soft errors
my $herMTTFMem = nearest(0.001, (le9 /
($her $totalMemSize)));
my $serMTTFMem = nearest(0.001, (le9 /
($ser $totalMemSize)));


my $herMTTFBit
$verbose = 0;


le9 / $her;







59



B.2 Graphical User Interface

# Toplevel window initialization for GUI
my $top w = MainWindow->new(); #create new toplevel
my $sw = $top w->screenwidth;
my $sh = $top w->screenheight;
#center toplevel on screen
$top w->geometry("965x650+" int($sw/2 965/2) "+"
int($sh/2 675/2));
$top w->title("Fault Tolerant Memory Simulator"); #title it
$top w->raise();
my $top f = $top w->Frame();

## Create frames for top and bottom tabbed notebooks
#the top one will contain open files and the bottom
#will contain coverage metrics and other information
#also create frame for script selection buttons
my $files f = $top f->Frame(-relief => "raised",
-borderwidth => 2)
->pack(-fill => 'both', -expand => 1);
my $main f = $top f->Frame(-relief => "raised",
-borderwidth => 2)
->pack(-fill => 'both');

##Create the top tabbed notebook
my $files nb = $files f->NoteBook()
->pack(-fill => 'both', -expand => 1);


##Create the bottom
my $main nb = $main
->pack(-fill =>


tabbed notebook
f->NoteBook()
'both', -expand


##Create pages for information about settings and batches
my $archSettings pg = $main nb
->add(0,-label => 'Architecture Settings');
my $simSettings pg = $main nb
->add(l, -label => 'Simulation Settings');
my $summSettings pg = $main nb
->add(2, -label => 'Summary Settings');

## Create the frames for the "Main" settings
############################################
# Architecture settings frames
my $archSettings f = $archSettings pg->Frame(-relief => "groove",
-borderwidth => 2)
->pack(-fill => 'both', -expand => 1);
my $archSettingsLeft f = $archSettings f->Frame()
->pack(-side=> 'left', -fill => 'both', -expand => 1);
my $archSettingsMiddle f = $archSettings f->Frame(-relief =>'groove',
-borderwidth => 2)
->pack(-side=> 'left', -fill => 'both', -expand => 1);
my $archSettingsRight f = $archSettings f->Frame()
->pack(-side=> 'left', -fill => 'both', -expand => 1);
my $dimenTop f = $archSettingsLeft f->Frame(-relief =>'groove',
-borderwidth => 2)
->pack(-side => 'top', -fill => 'both', -expand => 1);


> 1);











my $plane f = $di
->pack(-fill
my $colSize f = $
->pack(-fill
my $rowSize f = $
->pack(-fill
my $L2Col f = $co
->pack(-fill
my $L1Col f = $co
->pack(-fill
my $LOCol f = $co
->pack(-fill
my $L2Row f = $ro
->pack(-fill
my $L1Row f = $ro
->pack(-fill
my $LORow f = $ro
->pack(-fill
my $redunL2 f = $
->pack(-fill
my $redunL1 f = $
->pack(-fill
my $redunLORow f
->pack(-fill
my $redunLOCol f
->pack(-fill
my $memSpecs f =
->pack(-fill
my $memSpecsLeft
->pack(-side
my $memSpecsRight
->pack(-side
my $errorRate f


menTop f->Frame()
=> 'both', -side => 'bottom');


dimenTop f->Frame(-
=> 'both', -expand
dimenTop f->Frame(-
=> 'both', -expand
lSize f->Frame()
=> 'both', -expand
lSize f->Frame()
=> 'both', -expand
lSize f->Frame()
=> 'both', -expand
wSize f->Frame()
=> 'both', -expand
wSize f->Frame()
=> 'both', -expand
wSize f->Frame()
=> 'both', -expand
archSettingsMiddle
=> 'both', -expand
archSettingsMiddle
=> 'both', -expand


-label
=> 1,
-label
=> 1,


:> '#
side
:> '
side


Cols')
=> 'right');
# Rows')
=> 'left');


> 1, -side => 'top');

> 1, -side => 'top');

> 1, -side => 'top');

> 1, -side => 'top');

> 1, -side => 'top');

> 1, -side => 'top');


f->Frame()
=> 1, -side
f->Frame()
=> 1, -side


=> 'top');

=> 'top');


= $archSettingsMiddle f->Frame()
=> 'both', -expand => 1, -side => 'top');
= $archSettingsMiddle f->Frame()
=> 'both', -expand => 1, -side => 'top');
$archSettingsRight f->Frame()
=> 'both', -expand => 1);
f = $memSpecs f->Frame()
=> 'left', -fill => 'both', -expand => 1);
f = $memSpecs f->Frame()
=> 'right', -fill => 'both', -expand => 1);
$memSpecsLeft f->Frame(-relief =>'groove',


-borderwidth => 2)
->pack(-side => 'top', -fill => 'both', -expand => 1);
my $ecc f = $memSpecsLeft f->Frame(-relief => 'groove',
-borderwidth => 2,
-label => 'Error Correcting Code')
->pack(-side => 'top', -fill => 'both', -expand => 1);
my $info f = $memSpecsLeft f->Frame()


->pack(
my $algoSel


-side => 'top', -fill
f = $memSpecsRight f


->pack(-side => 'top', -fill
my $memSize f = $memSpecsRight f


->pack(
my $ser f =
->pack(


-side => 'top', -fill
$errorRate f->Frame(
-side=> 'left', -fill


=> 'both', -expand => 1);
->Frame(-relief => 'groove',
-borderwidth => 2,
-label => 'Other Options')
=> 'both', -expand => 1);
->Frame(-relief => 'groove',
-borderwidth => 2,
-label => 'Memory Size')
=> 'both', -expand => 1);
-label => 'Soft Error Rate')
=> 'both', -expand => 1);







61


my $her f = $errorRate f->Frame(-label => 'Hard Error Rate')
->pack(-side=> 'right', -fill => 'both', -expand => 1);
my $mttf f = $info f->Frame(-label => 'MTTF(h)',
-relief => 'groove',
-borderwidth => 2)
->pack(-side=> 'left', -fill => 'both', -expand => 1);
my $chkBits f = $info f->Frame(-label => 'Word Size',
-relief => 'groove',
-borderwidth => 2)
->pack(-side=> 'right', -fill => 'both', -expand => 1);

# Create updating text box with memory size info
my $spareSizePrnt = commify($totalSpareSize);
my $memSizePrnt = commify($totalMemSize);
my $usableSizePrnt = commify($totalUsableSize);
my $eccSizePrnt = commify($totalECCSize);
my $herMTTFMemPrnt = commify($herMTTFMem);
my $serMTTFMemPrnt = commify($serMTTFMem);
my $overheadPrnt = commify(nearest(0.01,($totalMemSize
$totalUsableSize)
/ $totalUsableSize 100));
my $memSize 1 = $memSize f->Label(-text => "Total: $memSizePrnt bits\n
Spare: $spareSizePrnt bits\n
ECC: $eccSizePrnt bits\n
Usable: $usableSizePrnt bits\n
Overhead: $overheadPrnt%",
-width => 30)->pack();
my $mttf 1 = $mttf f->Label(-text => "H: $herMTTFMemPrnt\n
S: $serMTTFMemPrnt",
-width => 15)->pack();
my $chkBits 1 = $chkBits f->Label(-text => "Data: $kVal\nCheck: $rVal",
-width => 15)->pack();

# Create comboboxes
my @L2Choices = qw(1 2 4 8 16 32 64 128 256 512);
my @LlChoices = qw(1 2 4 8 12 16 22 32 42 46 64 128 256 512 1024);
my @LORChoices = qw(32 64 128 256 512 1024 2048 4096 8192);
my @LOCChoices = qw(32 64 128 256 512);
my @LOPlaneChoices = qw(1 2 4 8 16 32 64);

my $L2 1 = $L2Row f->Label(-text => Level 2:")
->pack(-side => 'left');
my $L2Col cb = $L2Col f->BrowseEntry(-listwidth => 25,
-variable => \$L2Col,
-choices => \@L2Choices,
-width => 8,
-browsecmd => \&updateMemSize)
->pack(-side => 'right', -fill => 'both', -padx => 5);
my $L2Row cb = $L2Row f->BrowseEntry(-listwidth => 25,
-variable => \$L2Row,
-choices => \@L2Choices,
-width => 8,
-browsecmd => \&updateMemSize)
->pack(-side => 'right', -fill => 'both', -padx => 5);
my $L1 1 = $LlRow f->Label(-text => Level 1:")
->pack(-side => 'left');











my $L1Col cb = $L1Col f->BrowseEntry(-





->pack(-side => 'right', -fill =>
my $LlRow cb = $LlRow f->BrowseEntry(-





->pack(-side => 'right', -fill =>
my $L0 1 = $LORow f->Label(-text => "
->pack(-side => 'left');
my $LOCol cb = $LOCol f->BrowseEntry(-





->pack(-side => 'right', -fill =>
my $LORow cb = $LORow f->BrowseEntry(-





->pack(-side => 'right', -fill =>


-listwidth => 25,
-variable => \$L1Col,
-choices => \@LlChoices,
-width => 8,
-browsecmd => \&updateMemSize)
'both', -padx => 5);
-listwidth => 25,
-variable => \$L1Row,
-choices => \@LlChoices,
-width => 8,
-browsecmd => \&updateMemSize)
'both', -padx => 5);
Level 0:")

-listwidth => 25,
-variable => \$kVal,
-choices => \@LOCChoices,
-width => 8,
-browsecmd => \&updateMemSize)
'both', -padx => 5);
-listwidth => 25,
-variable => \$LORow,
-choices => \@LORChoices,
-width => 8,
-browsecmd => \&updateMemSize)
'both', -padx => 5);


my $LOP 1 = $plane f->Label(-text => Planes:")
->pack(-side => 'left');
my $LOPlane cb = $plane f->BrowseEntry(-listwidth => 25,
-variable => \$LOPlane,
-choices => \@LOPlaneChoices,
-width => 8,
-browsecmd => \&updateMemSize)
->pack(-side => 'bottom', -fill => 'y', -padx => 5);

my @redunChoices = qw(0 1 2 4 8 16 32 64 104 128 256
512 1024 2048 4096 8192 16384);
my $redunL2 1 = $redunL2 f->Label(-text => Redundant L2: ")
->pack(-side => 'left');
my $redunL2 cb = $redunL2 f->BrowseEntry(-listwidth => 25,
-variable => \$L2Red,
-choices => \@redunChoices,
-width => 8,
-browsecmd => \&updateMemSize)
->pack(-side => 'right', -fill => 'both', -padx => 20);
my $redunLl 1 = $redunLl f->Label(-text => Redundant LI: ")
->pack(-side => 'left');
my $redunLl cb = $redunLl f->BrowseEntry(-listwidth => 25,
-variable => \$L1Red,
-choices => \@redunChoices,
-width => 8,
-browsecmd => \&updateMemSize)
->pack(-side => 'right', -fill => 'both', -padx => 20);
my $redunLORow 1 = $redunLORow f->Label(-text => Redundant LO Rows:
>pack(side > 'left');
->pack(-side => 'left');








63


my $redunLORow cb = $redunLORow f->BrowseEntry(-listwidth => 25,
-variable => \$LORedRow,
-choices => \@redunChoices,
-width => 8,
-browsecmd => \&updateMemSize)
->pack(-side => 'right', -fill => 'both', -padx => 20);
my $redunLOCol 1 = $redunLOCol f->Label(-text => "Redundant LO Cols: ")
->pack(-side => 'left');
my $redunLOCol cb = $redunLOCol f->BrowseEntry(-listwidth => 25,
-variable => \$LORedCol,
-choices => \@redunChoices,
-width => 8,
-browsecmd => \&updateMemSize)
->pack(-side => 'right', -fill => 'both', -padx => 20);

# Create HER/SER entry boxes
my $ser en = $ser f->Entry(-validate => 'focusout',
-validatecommand => \&updateMemSize,
-textvariable => \$ser,
-justify => 'right',
-width => 10)
->pack(-padx => 20, -pady => 9);
my $her en = $her f->Entry(-validate => 'focusout',
-validatecommand => \&updateMemSize,
-textvariable => \$her,
-justify => 'right',
-textvariable => \$her,
-width => 10,
->pack(-padx => 20, -pady => 9);

# Create error correction selection buttons
my $eccNEC = $ecc f->Radiobutton(-text => 'None',
-value => 0,
-variable => \$nec,
-command => \&updateMemSize)
->pack(-side => 'left', -padx => 5);
my $eccSEC = $ecc f->Radiobutton(-text => 'SEC',
-value => 1,
-variable => \$nec,
-command => \&updateMemSize)
->pack(-side => 'left', -padx => 5);
my $eccDEC = $ecc f->Radiobutton(-text => 'DEC',
-value => 2,
-variable => \$nec,
-command => \&updateMemSize)
->pack(-side => 'left', -padx => 5);

# Create fault tolerance algorithm selection buttons
my $L1ECCSel = $algoSel f->Checkbutton(-text => 'LI ECC (RAID)',
-variable => \$raid,
-command => \&updateMemSize)
->pack(-side => 'left', -padx => 5);
my $verbSel = $algoSel f->Checkbutton(-text => 'Verbose',
-variable => \$verbose)
->pack(-side => 'left', -padx => 5);








64


# Simulation settings frames
my $simSettings f = $simSettings pg->Frame(-relief => 'groove',
-borderwidth => 2)
->pack(-fill => 'both', -expand => 1);
my $simSettingsLeft f = $simSettings f->Frame(-label => 'Time (h)',
-relief => 'groove',
-borderwidth => 2)
->pack(-side=> 'left', -fill => 'both', -expand => 1);
my $simSettingsRight f = $simSettings f->Frame(-label => 'Execute',
-relief =>'groove',
-borderwidth => 2)
->pack(-side=> 'left', -fill => 'both', -expand => 1);
my $pollTime f = $simSettingsLeft f->Frame(-relief => 'groove',
-borderwidth => 2)
->pack(-fill => 'both', -expand => 1);
my $endTime f = $simSettingsLeft f->Frame(-relief => 'groove',
-borderwidth => 2)


->pack(-fill
my $executeSim f

->pack(-fill
my $maxTrials f
->pack(-fill

my $pollTime 1 =
->pack(-side
my $pollTime en


=> 'both', -expand => 1);
= $simSettingsRight f->Frame(-relief =>'groove',
-borderwidth => 2)
=> 'both', -expand => 1);
= $executeSim f->Frame()
=> 'x', -expand => 1);

$pollTime f->Label(-text => "Polling Interval(h): ")
=> 'left');
= $pollTime f->Entry(-justify => 'right',
-textvariable => \$pollInterval,
-width => 10)


->pack(-side => 'left');
my $endTime 1 = $endTime f->Label(-text => "Max Run Time(h): ")
->pack(-side => 'left');
my $endTime en = $endTime f->Entry(-justify => 'right',
-textvariable => \$lifetime,
-width => 10)
->pack(-side => 'left');


my $maxTrials 1
->pack(-side
my $maxTrials en


->pack(-side
my $executeSim b


$maxTrials f->Label(-text => "Maximum Trials: ")
=> 'left');
= $maxTrials f->Entry(-justify => 'right',
-textvariable => \$maxTrials,
-width => 10)
=> 'left', -pady => 5);
S$executeSim f->Button(-text => 'Execute 0',
-width => 20,
-command => \&runSimulation)


->pack(-pady => 5);
my $plotSim b = $executeSim f->Button(-text => 'Plot',
-width => 20,
-command => \&plotResults)
->pack(-pady => 5);

# Summary settings frames
my $summSettings f = $summSettings pg->Frame(-relief => "raised",
-borderwidth => 2)
->pack(-fill => 'both', -expand => 1);











my $summSettingsLeft f = $summSettings f->Frame(-relief => 'groove',
-borderwidth => 2)
->pack(-side=> 'left', -fill => 'both', -expand => 1);
my $summSettingsMiddle f = $summSettings f->Frame(-relief => 'groove',
-borderwidth => 2)
->pack(-side=> 'left', -fill => 'both', -expand => 1);
my $summSettingsRight f = $summSettings f->Frame(-relief =>'groove',
-borderwidth => 2)
->pack(-side=> 'left', -fill => 'both', -expand => 1);

# Create menubar
$top w->configure(-menu => my $menubar mb = $top w->Menu);
# Create menu items in menubar
map {$menubar mb->cascade(-label => '~' ->[0],
-menuitems => $ ->[1])}
['File', &file menuitems],
['Edit', &edit menuitems],
['Search', &search menuitems],
['Help', &help menuitems];

## Pack all main frames to toplevel
$top f->pack(-side=>'top', -fill=>'both', -expand=>l);
$files f->pack(-side => "top", -fill => 'both', -expand => 1);
$archSettings f->pack(-side=>"top", -fill=>'both');

MainLoop(); #Display Tk widgets

B.3 Simulation Subroutines

B.3.1 Generate Failures

sub generate
{
# Variable declarations
my ($failType, $failLOCol, $failLOPlane, $failLORow, $failLlCol);
my ($failLlRow, $failLlRowCalc, $failL2Col, $failL2Row);
my ($failL2RowCalc, $index, $inverse, $minTime, $power);
my Randnd, $Uprod);

$Uprod = 1; #initialize Uprod to 1 since it will be mult by power
while ($minTime <= $lifetime)
{
#use IBM FTMS equations to calculate failure times in order
$count += 1;
Randd = random uniform();
$power = Randd ** (1 / ($totalMemSize $count + 1));
$Uprod *= $power;
$inverse = -1 $herMTTFBit log($Uprod);
$minTime = $inverse;


##determine type and location of each failure
#generation of fail type
Randnd = random uniform();











Randnd < 0.35)


$failType
$failLORow


'B';
random uniform integer(1, 1, ($LORow +
$LORedRow));


#if error is in spare row, set the flag
if ($failLORow > $LORow)


#failure can't be in both a spare row and a spare col
$failLOCol = random uniform integer(l, 1, $LOCol);
$failLOPlane = random uniform integer(l, 1, $LOPlane);


elsif Randnd < 0.67)


$failType =
$failLOCol =
$failLOPlane
$failLORow =


random uniform integer(1, 1, ($LORow +
$LORedRow));


elsif Randnd < 0.85)
{
$failType = 'C';
$failLORow = 0;
#choose one of the cols including the spares, then determine
#col/plane recieved the error-spares are in "extra" plane
$failLOCol = random uniform integer(1, 1,
(($LOCol $LOPlane) + $LORedCol));
$failLOPlane = nhimult(l, $failLOCol / $LOCol);
$failLOCol = $failLOCol % $LOCol;


else


$failType
$failLORow
$failLOCol


'A';
0;
0;


$failLlRow
$failLlCol
$failL2Row
$failL2Col


random
random
random
random


@{$fails[$count]}


uniform integer(l, 1, $LlRow);
uniform integer(1, 1, ($L1Col + $raid));
uniform integer(1, 1, $L2Row);
uniform integer(l, 1, $L2Col);

($count, $failType, $minTime, $failL2Row,
$failL2Col, $failLlRow, $failLlCol,
$failLORow, $failLOPlane, $failLOCol);


print "COUNT: $count\n";







67


B.3.2 Simulate

# ------------------------------------------------------
# KEY TO LABELS:
# 'A' => Label for LO CHIP (entire array) error
# 'B' => Label for LO BIT error
# 'BD' => Track BAD COUNT for words with < NEC bad bits
# 'C' => Label for LO COLUMN error
# 'CH' => CHAIN of LO or L1 fails linking those in similar devices
# 'DD' => Marks LO array as DEAD
# 'DE' => Track LO rows that have been DEALLOCATED
# 'R' => Label for LO ROW error
# 'SC' => Tracks number of SPARE COLUMNS used
# 'SR' => Tracks number of SPARE ROWS used
# 'ST' => STATUS information for array
# 'UE' => Track count of UNCORRECTABLE ERRORS
# 'UU' => Mark LO spare array as UNUSED
# ------------------------------------------------------

sub simulate
{
# Variable declarations
my ($errorCnt, $failTime, $flag, $goodBits);
my ($index, $key, $keyC, $keyR);
my ($LOColKey, $LOPlaneKey, $LORowKey, $LlColKey, $LiRowKey);
my ($LOTempColKey, $LOTempPlaneKey, $LOTempRowKey, $LiTempColKey);
my ($numWords, $redunSuccess, $tempIndex, $tempLiKey, $time);
my ($totalLOSUE, $totalLiSUE, $totalLOUE, $totalLiUE);
my (@colChain, @rowChain, @totalBadBit, @totalBadWord);
my (%bitErrors, %wordErrors);
# End variables

#read fail info in from file and put in a 2D array
$time = $pollInterval;
#calculate parameters used in the estimation of number of SUEs
#total number of L1 devices in memory
$numWords = $L2Row $L2Col $LiRow *
$LlCol $LORow $LOPlane;
#fraction of good bits in words with NEC bad bits
#these are the ones succeptable to errors
$goodBits = $LOCol $nec;

#go through each failure generated by order stats
FAILLOOP: while ($index <= $count) {
$index += 1; #go on to next fail in generated list
$flag = 0; #set to only go through hash once
$failTime = $fails[$index][2];
while ($time <= $failTime)
{
if ($flag == 0)
{
$flag = 1;
#reset variables
@totalBadBit = @totalBadWord = ();
$totalLOUE = $totalLOSUE = 0;
$totalLiUE = $totalLiSUE = 0;







68


#go through all LO arrays
foreach $key (keys %SST)
{
#only count the errors if they are in the main memory:
#they must not be marked 'dead' and must either not be from
#a spare device or be a spare device that is marked 'used'
if ($SST{$key}{'ST'} ne 'DD' and
($key !~ /S/ or $SST{$key}{'ST'} eq 'UU'))

#go through each row key and, if it is a number then
#go through all corresponding columns associated with it
foreach $keyR (keys %{$SST{$key}})
{
if ($keyR =~ /\d+/)
{
#key is for LO failure info
if ($keyR =~ /-/)
{
if ($SST{$key}{$keyR}{'ST'} ne 'DD' and
($key !~ /S/ or $SST{$key}{$keyR}{'ST'} eq 'UU')

{
$totalBadBit[l] += $SST{$key}{$keyR}{'BD'}[1];
$totalBadBit[2] += $SST{$key}{$keyR}{'BD'}[2];
$totalLOUE += $SST{$key}{$keyR}{'UE'};
#print "totalLOUE: $totalLOUE\n";
}
}
#if RAID enabled, then key is for L1 failure info
elsif ($raid)
{
#record total num rows with 2 to LOCol word failures
#as well as the overall number of L1 (word) HUEs
for my $i (2.. ($LCol + 1))
{
if ($SST{$key}{$keyR}{'UE'}[$i])
{
$totalBadWord[$i] += $SST{$key}{$keyR}{'UE'}[$i];
$totalLiUE += $SST{$key}{$keyR}{'UE'}[$i] $i;
}
}
}
}
}



#Calc SUE by multiplying SER by the num good bits in a word with
#nec bad bits x total words w/ nec bad bits and finally by time
#div by le9 to account for FIT given as fails in le9 bit-hours.
if ($nec > 0)
{
$totalLOSUE = ($ser / le9 $time) $goodBits *
$totalBadBit[$nec];







69


else
{
$totalLOSUE = ($ser / le9 $time) $goodBits $numWords;
}

#calculate L1 SER if RAID is enabled
if ($raid)
{
#when RAID is enabled, the total L1SUE is the total number of
#LO soft errors multiplied by the ratio of words susceptible to
#lining up to cause a word error this needs to be done for the
#number of rows with k = 1..(LOCol -1) word errors the number
#of words susceptible in each case is (LOCol k) and the case
#of k == 1 needs to be mult. by 2 to account for two new UEs

#the total number of rows with 1 UE in them multiplied by the
#number of words susceptible a soft UE lining up with these
#would result in two UEs
$totalLlSUE = ($totalLOUE $totalLlUE) ($L1Col 1) 2;
foreach my $i (2 .. ($L1Col 1))
{
$totalLlSUE += $totalBadWord[$i] ($L1Col $i);
}
#currently have the total number of words susceptible to
#a soft error lining up with them divide these by total
#number of words to get a ratio and multiply by the total
#number of soft errors in the system
$totalLlSUE = ($totalLlSUE / $numWords) $totalLOSUE;
}
#add each failure amount to an array to be used later
#in a statistical analysis of multiple trials
push @{$serLOStatData[$time]}, $totalLOSUE;
push @{$herLOStatData[$time]}, $totalLOUE;
if ($raid)
{
push @{$serLlStatData[$time]}, $totalLlSUE;
push @{$herLlStatData[$time]}, $totalLlUE;
}
$time += $pollInterval;
}
#generate key values from fail locations
$L2Key = $fails[$index][3].'-'.$fails[$index][4];
$LlKey = $fails[$index][5] .'-'.$fails[$index] [6];
$LlRowKey = $fails[$index] [5];
$LlColKey = $fails[$index][6];
$LORowKey = $fails[$index] [7];
$LOPlaneKey = $fails[$index] [8];
$LOColKey = $fails[$index][9];

#Don't process failure if it is in a part of memory that has
#already been killed or if it is in a LO row that has been
#deallocated
if ($SST{$L2Key}{'ST'} eq 'DD' or
$SST{$L2Key}{$L1Key}{'ST'} eq 'DD')
next FAILLOOP;
next FAILLOOP;







70


elsif ($LORowKey > $LORow)
{
#keep track of all spare rows that are no longer available
$SST{$L2Key}{$LlKey}{'SR'} += 1;
next FAILLOOP;
}

#check to see if failure is of a bit in an L1 Device (chip)
elsif ($fails[$index][1] eq 'B')
{
#initialize bit errors to the current error
%bitErrors = ();
$bitErrors{$LOCol} = 1;
#reset arrays to be used to track all bit errors in the same
#row/col as the current one these failure indexes will be
#disabled if the row/col is replaced with a spare.
@colChain = ();
@rowChain = ();
#FIXME: put back in when spare columns used
#push @colChain, $index;
push @rowChain, $index;

#check to see if key is present in system status table
#the SST points to the start of each chain of bit failures
if ($SST{$L2Key}{$LlKey}{'CH'} == 0)
{
#SST row entry was blank, so set SST row to current index value
$SST{$L2Key}{$LlKey}{'CH'} = $index;
}
else
{
#look through previous failures in same LO array matrix
#set the index to the start of the chain
$tempIndex = $SST{$L2Key}{$LlKey}{'CH'};

#cycle through all errors chained together
while (1)
{
$LOTempRowKey = $fails[$tempIndex][7];
$LOTempPlaneKey = $fails[$tempIndex][8];
$LOTempColKey = $fails[$tempIndex][9];
#bit causes error if it is in the same row and plane but a
#different column also check that the index has not been
#disabled by swapping in a spare
if ($fails[$tempIndex][12] != 1 and
$LORowKey == $LOTempRowKey)
{
#this is the same row as the current bit failure, so keep
#track of it in case this row is replaced with a spare
push @rowChain, $tempIndex;
if ($LOPlaneKey == $LOTempPlaneKey)
{
#only count this error if it is a new error
#and it is in a word that hasn't been deallocated;
#otherwise go on to the next failure







71


if ($LOColKey == $LOTempColKey)
{
next FAILLOOP;
}
elsif ($fails[$tempIndex][12] == 2)
{
next FAILLOOP;
}
else

#this is the same plane as the current failure, as long
#as it is a different col (i.e., not the same bit), it
#lines up with another failure
$bitErrors{$LOTempColKey} = 1;
}
}

#if failure entry in the chain is itself chained, then
#continue with the next link
if ($fails[$templndex] [10] > 0)
{
#set the index variable to next fail index in the chain
$tempIndex = $fails[$tempIndex][10];
}
#otherwise, add the current failure to the chain
else
{
$fails[$tempIndex] [10] = $index;
#last error for this row, so break from the while() loop
last;
}
}
}
}
#count the number of unique bit errors that lined up in the word
$errorCnt = keys(%bitErrors);
($verbose > 1 ? print SIMFILE "ERRORCNT: $errorCnt\n" : 0);
#if the number of errors is less than NEC, they won't be corrected
#so they need to be tracked to see how they line up with soft
#errors to create SUE's
if ($errorCnt < $nec)
{
$SST{$L2Key}{$LlKey}{'BD'}[$errorCnt] += 1;
#the next lowest value is reduced
if ($nec > 1)
{
$SST{$L2Key}{$L1Key}{'BD'}[($errorCnt 1)] -= 1;
}
}
#if the total number of bit errors in the row is equal to the
#number that can be corrected, then need to replace this row with a
#spare if possible
elsif ($errorCnt == $nec)
{
#need to replace this row if possible
#if length of array is less than total spares, then there is a
#spare LO row available







72


if ($SST{$L2Key}{$LlKey}{'SR'} < $LORedRow)
{
$SST{$L2Key}{$LlKey}{'SR'} += 1;
#disable all index is current chain with the same row,
#since this entire row was replaced
foreach $keyR (@rowChain)
{
#set disable flag
$fails[$keyR][12] = 1;
}
}
#need to swap in a spare at the L1 level (chip)
else
{
$SST{$L2Key}{$LlKey}{'BD'}[$errorCnt] += 1;
#the next lowest value is reduced
if ($nec > 1)
{
$SST{$L2Key}{$LlKey}{'BD'}[($errorCnt 1)] -= 1;
}
}
}
#if the number of errors is greater than the number that are
#correctable, then there wasn't a spare available to replace the
#device previously so UE's occur if ECC is not available across the
#L1 blocks. If ECC is available, then need to track the words with
#UE's to find when two of them line up to cause a L1 UE
elsif ($errorCnt > $nec)
{
#if RAID not implemented, a UE occurs
$SST{$L2Key}{$LlKey}{'UE'} += 1;
#deallocate this word if another bit error occurs in this word,
#then it will see this word in the chain and know to go on to the
#next error
$fails[$index][12] = 2;
if (!$raid)
{
#if error correction is avail., then all spares have been used
#and any errors are permanent UE's the LO rows containing these
#errors will be deallocated
if ($nec)
{
##deallocate row
#$SST{$L2Key}{$LlKey}{'DE'}{$LORowKey} = 1;
##record the uncorrectable error
}
else
{
#if no error correction, then make use of spares even though
#a UE occurs this will reduce the amount of deallocation that
#occurs. ECC allows for spares to be used up before
#bitErrors > nec if length of array is less than total
#spares, then there is a spare LO row available







73


if ($SST{$L2Key}{$LlKey}{'SR'} < $LORedRow)
{
$SST{$L2Key}{$LlKey}{'SR'} += 1;

#disable all index is current chain with the same row,
#since this entire row was replaced
foreach $keyR (@rowChain)
{
#set disable flag
$fails[$keyR][12] = 1;
}
}
}
}
else
{
#if RAID ECC is implemented across blocks, then for a given L1
#row, go through L1 Col's. Then for each of these L1 arrays,
#look through all other errors that were uncorrectable at the
#LO level. If any of these are in the same LO row as the
#current failure, then an L1 uncorrectable error occurs.

#if no previous LO UE's in this L1 row, insert current index
#value and go on
if($SST{$L2Key}{$LlRowKey}{'CH'} == 0)
{
$SST{$L2Key}{$LlRowKey}{'CH'} = $index;
}
else
{
#reset array each time chain is traversed
%wordErrors = ();
#record current error
$wordErrors{$LlColKey} = 1;

#set tempIndex to the index of the first failure that caused
#an LO UE. This index is used to determine which word
#(plane) had the failure.
$tempIndex = $SST{$L2Key}{$LIRowKey}{'CH'};
while (1)
{
$tempLiKey = $fails[$tempIndex] [5].'-'
$fails[$tempIndex] [6];
$LiTempColKey = $fails[$tempIndex][6];
$LOTempRowKey = $fails[$tempIndex][7];
$LOTempPlaneKey = $fails[$tempIndex][8];
#word failure must be in the same LO row and LO plane
#(aka relative word location)
#and be in a different L1 device
if ($LORowKey == $LOTempRowKey and
$LOPlaneKey == $LOTempPlaneKey)
{
if ($LlColKey != $LlTempColKey)
{







74


if ($SST{$L2Key}{$tempLlKey}{'ST'} ne 'DD')
{
#if L1 chip status is not dead, then count it
$wordErrors{$LiTempColKey} = 1;
($verbose > 1 ? print SIMFILE "WORD ERROR IN SAME
RELATIVE LOCATION IN DIFFERENT WORD!\n" : 0);
}
else
{
($verbose > 1 ? print SIMFILE "NO ERROR\n" : 0);
}
}
else
{
#current error is in the same word as a previous error
#don't chain this failure; just go on to next failure
next FAILLOOP;
}
}
else
{
($verbose > 1 ? print SIMFILE "NO ERROR\n" : 0);
}
#if failure entry in the chain2 is itself chained,
#then continue with the next link
if ($fails[$templndex] [11] > 0)
{
#set the index variable to next fail index in the chain2
$tempIndex = $fails[$tempIndex][11];
}
#otherwise, add the current failure to the chain
else
{
$fails[$tempIndex] [11] = $index;
#last error in current chain2, so break from loop
last;
}
}
#if there was an uncorrectable word error, then record it
$errorCnt = keys (%wordErrors);
($verbose ? print SIMFILE "ERROR CNT: $errorCnt\n" : 0);
#if two or more errors, then that num of UEs occur
if ($errorCnt >= 2)
{
$SST{$L2Key}{$L1RowKey}{'UE'I[$errorCnt] += 1;
#the next lowest value is reduced
if ($errorCnt > 2)
2K{RowKey'UE errorCnt 1;
$SST{$L2Key}{$L1RowKey}{'UE'}[($errorCnt 1)] -= 1;
}
}
}
}
}











B.3.3 Device Redundancy

sub deviceRedundancy
{
#check all the possible spares for each key and find the first one
#that isn't swapped in or marked for being bad from a chip failure
foreach my $i (1 .. $LiRed)
{
#if a spare is found that is usable, mark it and the current
#bad L1 device (chip) as swapped
if ($SST{$L2Key}{"S-$i"}{'ST'} ne 'UU')
{
$SST{$L2Key}{"S-$i"}{'ST'} = 'UU';
$SST{$L2Key}{$LlKey}{'ST'} = 'DD'; # probably redundant
#return success at swapping in spare L1 device
return 1;
}
}
#need to swap in spare at a higher level (entire L2 card)
#in future, might let errors build up before doing this
foreach my $i (1 .. $L2Red)
{
#if a spare is found that is usable, mark it
#and the current bad L2 device (card) as swapped
if ($SST{"S-$i"}{'ST'} ne 'UU')
{
$SST{"S-$i"}{'ST'} = 'UU';
$SST{$L2Key}{'ST'} = 'DD';
#return success at swapping in spare L2 device
return 2;
}
}
return 0;
}

B.3.4 Run Simulation

sub runSimulation
{
my ($diffTimel, $diffTime2, $endTimel, $endTime2, $startTime);
#variables for calculating accuracy
#my ($acc, $average, $stdev);
my $filename = join '-', $L2Row, $L2Col, $L2Red, $LiRow, $LlCol,
$LiRed, $LORow, $LOCol, $LOPlane, $LORedRow,
$LORedCol, $nec, $raid, $lifetime,
$pollInterval, $her, $ser, $maxTrials;

updateMemSize(); #make sure all parameters are updated
#insert commas intoall numbers before printing them out
my $spareSizePrnt = commify($totalSpareSize);
my $memSizePrnt = commify($totalMemSize);
my $usableSizePrnt = commify($totalUsableSize);
my $eccSizePrnt = commify($totalECCSize);
$herMTTFMemPrnt = commify($herMTTFMem);
$serMTTFMemPrnt = commify($serMTTFMem);











#reset variables
@herLOStatData =
@serLOStatData =
@herLlStatData =
@serLlStatData =


on each execution
();
();
();
();


foreach my $i (1..$maxTrials)
{
$startTime = time();
#reset variables
%SST = ();
$count = 0; #counts number of failures generated
@fails = (); #array with all failure info

#generate all failure times and locations
sysopen(GENFILE, "gen $filename.out", O RDWR i O TRUNC O CREAT)
or die "Can not open the file gen.out in $path\n";
generate();


#record end time to calc amount of time
$endTimel = time();

#output for first simulation if verbose
#verbose==l gives one level of output
#verbose==2 gives more detailed
if ($verbose and $i == 1)


necessary to generate fails


variable set selected


sysopen(SIMFILE, "sim $filename.out", O RDWR 0O TRUNC i O CREAT)
or die "Can not open the file sim $filename.out in $path\n";
}
simulate();
if ($verbose and $i == 1)


close SIMFILE or die "Closing: $!";
}
close GENFILE or die "Closing: $!";


#at given intervals print current failure data to a file
sysopen(TIMEFILE, "time $filename.out", O RDWR 0O TRUNC I O CREAT)
or die "Can not open the file time $filename.out in $path\n";
print TIMEFILE "L2Row: $L2Row\tL2Col: $L2Col\tL2Red: $L2Red\n";
print TIMEFILE "LiRow: $LlRow\tLlCol: $LlCol\tLlRed: $LlRed\n";
print TIMEFILE "LORow: $LORow\tLOPlane: $LOPlane\tLOCol: $kVal\t";
print TIMEFILE "LORRow: $LORedRow\tLORCol: $LORedCol\n";
print TIMEFILE "MEM SIZE: $memSizePrnt\tECC SIZE: $eccSizePrnt\t";
print TIMEFILE "SPARE SIZE: $spareSizePrnt\tUSABLE SIZE:
$usableSizePrnt\n";
print TIMEFILE "OVERHEAD: $overheadPrnt\nRAID: $raid\nNEC: $nec\n";

if (!$raid)
{
print TIMEFILE "\tLO SUE\t\t\t\tLO HUE\n";
print TIMEFILE
"TIME\tMEAN\tMIN\tMAX\tSTDV\tMEAN\tMIN\tMAX\tSTDV\n";













TIMEFILE "\tLO SUE\t\t\tLO HUE\t\t\tLl SUE\t\t\tLl HUE\n";
TIMEFILE "TIME\tMEAN\tMN\tMAX\tSTDV\tMEAN\tMN\tMAX\tSTDV\t";
TIMEFILE "MEAN\tMN\tMAX\tSTDV\tMEAN\tMN\tMAX\tSTDV\n";


#add failures at each
#stats data structure
#trials to be printed
for (my $i = 0; $i <=
{
if (!$raid)


polling interval, for all memory trials to a
this will allow the statistical results of all
to the TIMEFILE
$lifetime; $i += $pollInterval)


my $herLOStat = Statistics::Descriptive::Sparse->new;
my $serLOStat = Statistics::Descriptive::Sparse->new;
$herLOStat->add data(@{$herLOStatData[$i] );
$serLOStat->add data(@{$serLOStatData[$i] );
#print stat results for the total series of executions to a file
printf TIMEFILE "$i\t%d\t%d\t%d\t%d\t%d\t%d\t%d\t%d\n",
$serLOStat->mean, $serLOStat->min, $serLOStat->max,
$serLOStat->standard deviation,
$herLOStat->mean, $herLOStat->min, $herLOStat->max,
$herLOStat->standard deviation;


else


my $herLOStat =
my $serLOStat =
my $herLlStat =
my $serLlStat =
$herLOStat->add
$serLOStat->add
$herLlStat->add
$serLlStat->add


Statistics::Descriptive::Sparse-
Statistics::Descriptive::Sparse-
Statistics::Descriptive::Sparse-
Statistics::Descriptive::Sparse-
data(@{$herLOStatData[$i]});
data(@{$serLOStatData[$i] );
data(@{$herLlStatData[$i]});
data(@{$serLlStatData[$i] );


#print stat results for the total series of executions to a file
printf TIMEFILE "$i\t%d\t%d\t%d\t%d\t%d\t%d\t%d\t%d\t
%d\t%d\t%d\t%d\t%d\t%d\t%d\t%d\n",
$serLOStat->mean, $serLOStat->min, $serLOStat->max,
$serLOStat->standard deviation,
$herLOStat->mean, $herLOStat->min, $herLOStat->max,
$herLOStat->standard deviation,
$serLlStat->mean, $serLlStat->min, $serLlStat->max,
$serLlStat->standard deviation,
$herLlStat->mean, $herLlStat->min, $herLlStat->max,
$herLlStat->standard deviation;
}
}
#Generate data structure for plotting with GDGraph::boxplot
#The first line is the label for each of the memory configurations
#Each memory configurations will have been executing maxTrials
#num times and the statistical results will be plotted
push @{$serLOPlotData[0]}, "Trial $execCnt";
push @{$herLOPlotData[0]}, "Trial $execCnt";
push @{$serLlPlotData[0]}, "Trial $execCnt";
push @{$herLlPlotData[0]}, "Trial $execCnt";


else


print
print
print


->new;
->new;
->new;
->new;







78




#copy error counts for trials at a given time in the memory lifetime
for my $i (0..$maxTrials)
{
push @{$serLOPlotData[l] [$execCnt]}, $serLOStatData[$lifetime] [$i];
push @{$herLOPlotData[l] [$execCnt]}, $herLOStatData[$lifetime] [$i];
push @{$serLiPlotData[l] [$execCnt]}, $serLlStatData[$lifetime] [$i];
push @{$herLlPlotData[l] [$execCnt]}, $herLlStatData[$1lifetime] [$i];
}
#track number of memory configs executed since the script was started
$execCnt += 1;
#update the Execute button to represent the num configs executed
$executeSim b->configure(-text => "Execute $execCnt");
close TIMEFILE or die "Closing: $!";
openFile("time $filename.out");



B.4 Graphical User Interface Subroutines


B.4.1 Commify

sub commify
{
my $text = reverse $ [0];
$text =~ s/(\d\d\d)(?=\d) (?!\d*\.)/$1,/g;
return scalar reverse $text;


B.4.2 Plot Results


sub plotResults

my serLgrap new
my $serLOgraph = new
my $herLOgraph = new
my $serLlgraph = new
my $herLlgraph = new
$serLOgraph->set(
x label
y label
title

$herLOgraph->set(
x label
y label
title

$serLlgraph->set(
x label
y label
title

$herLlgraph->set(
x label
y label
title
) ;


GD::Graph::boxplot();
GD::Graph::boxplot();
GD::Graph::boxplot();
GD::Graph::boxplot();


'Memory
'Number
'Memory


'Memory
'Number
'Memory


'Memory
'Number
'Memory


'Memory
'Number
'Memory


Configuration',
of LO Soft Uncorrectable Errors',
Simulation LO SUE Results',


Configuration',
of LO Hard Uncorrectable Errors',
Simulation LO HUE Results',


Configuration',
of L1 Soft Uncorrectable Errors',
Simulation L1 SUE Results',


Configuration',
of L1 Hard Uncorrectable Errors',
Simulation L1 HUE Results',












my $serLOgd =
my $herLOgd =
my $serLlgd =
my $herLlgd =

if ($serLOgd)


$serLOgraph-
$herLOgraph-
$serLlgraph-
$herLlgraph-


->plot(
->plot(
->plot(
>plot(


\@serLOPlotData );
\@herLOPlotData );
\@serLlPlotData );
\@herLlPlotData


open(IMG, '>serLO.png') or die $!;
binmode IMG;
print IMG $serLOgd->png;
close IMG;

else
else


print "EMTPY SET: SER LO\n";


if ($herLOgd)
{
open(IMG, '>herLO.png') or die $!;
binmode IMG;
print IMG $herLOgd->png;
close IMG;
}
else
{
print "EMTPY SET: HER LO\n";
}

if ($serLlgd)
{
open(IMG, '>serLl.png') or die $!;
binmode IMG;
print IMG $serLlgd->png;
close IMG;

else
else


print "EMTPY SET: SER Ll\n";


if ($herLlgd)
{
open(IMG, '>herLl.png') or die $!;
binmode IMG;
print IMG $herLlgd->png;
close IMG;
}
else

print "EMTPY SET: HER L\n";
print "EMTPY SET: HER Ll\n";











B.4.3 Update Memory Size

sub updateMemSize
{
# Determine number of bits used for error correcting codes
# 'k' data bits and 'r' check bits are calculated
$rVal = $nec (log($kVal)/log(2) + 1);
# add one more check bit for RAID with NEC or SEC
($nec <= 2 and $raid ? $rVal += 1: 0);
$LOCol = $kVal + $rVal;

# Total memory size is size of all usable and spare memory
$LOUsableSize = $kVal $LORow $LOPlane;
$LOECCSize = $rVal $LORow $LOPlane;
$LOSpareSize = ($LORedRow $LOPlane $LOCol) +
($LORedCol $LORow);
$LOMemSize = $LOUsableSize + $LOECCSize + $LOSpareSize;

$LlUsableSize = $LOUsableSize $LlRow $L1Col;
$L1ECCSize = ($LOECCSize $LlRow $L1Col) +
($raid $LOMemSize $LlRow);
$LlSpareSize = ($LOSpareSize $LlRow $L1Col) +
($LOMemSize $LlRed);
$LlMemSize = $LlUsableSize + $L1ECCSize + $LlSpareSize;

$totalUsableSize = $LlUsableSize $L2Row $L2Col;
$totalECCSize = $L1ECCSize $L2Row $L2Col;
$totalSpareSize = ($LlSpareSize $L2Row $L2Col) +
($LlMemSize $L2Red);
$totalMemSize = $totalUsableSize + $totalECCSize + $totalSpareSize;

$spareSizePrnt = commify($totalSpareSize);
$memSizePrnt = commify($totalMemSize);
$usableSizePrnt = commify($totalUsableSize);
$eccSizePrnt = commify($totalECCSize);
$overheadPrnt = commify(nearest(0.01, ($totalMemSize -
$totalUsableSize) /
$totalUsableSize 100));

#Calculate MTTF from hard and soft errors
#print "HER: $her\tSER: $ser\tTOTALMEM: $totalMemSize\n";
$herMTTFMem = nearest(0.001, (le9 / ($her $totalMemSize)));
$serMTTFMem = nearest(0.001, (le9 / ($ser $totalMemSize)));
$herMTTFBit = le9 / $her;

$memSize l->configure(-text => "Total: $memSizePrnt bits\n
Spare: $spareSizePrnt bits\n
ECC: $eccSizePrnt bits\n
Usable: $usableSizePrnt bits\n
Overhead: $overheadPrnt%");
$mttf l->configure(-text => "H: $herMTTFMem\nS: $serMTTFMem");
$chkBits l->configure(-text => "Data: $kVal\nCheck: $rVal");
return 1;











B.4.4 Menubar Subroutines

#create EDIT menu item in menubar
sub edit menuitems


[qw/command Undo
[qw/command Redo


-accelerator Ctrl-Z
-accelerator Ctrl-Y


command/ => \&tUndo],
command/ => \&tRedo],


[qw/command Copy -accelerator Ctrl-C -command/ => \&tCopy],
[qw/command Cut -accelerator Ctrl-X -command/ => \&tCut],
[qw/command Paste -accelerator Ctrl-V -command/ => \&tPaste],

[qw/command/,'Select All',qw/-accelerator Ctrl-A
-command/ => \&tSelectAll],
[qw/command/,'Unselect All',qw/-command/ => \&tUnSelectAll],

}
#create FILE menu item in menubar
sub file menuitems


[qw/command New -command/
[qw/command Open -command/
ii


=> \&newFile],
=> \&openFile],


[qw/command Save -command/ => \&saveFile],
[qw/command/,'Save As',qw/-command/ => \&saveAs],
[qw/command/,'Save All',qw/-command/ => \&saveAll],

[qw/command Close -command/ => \&closeFile],
[qw/command/,'Close All',qw/-command/ => \&closeAll],

[qw/command Exit -command/ => \&exitProg]
];
}
#create HELP menu item in menubar
sub help menuitems
{

[qw/command/,'Help Topics',qw/-command/ => sub {print "Help
Topics\n"}],

[qw/command About -command/ => \&showAbout],
];
}
#create SEARCH menu item in menubar
sub search menuitems
{

[qw/command Find -command/ => \&tFind],
[qw/command/,'Find Next',qw/-command/ => \&tNext],
[qw/command/,'Find Previous',qw/-command/ => \&tPrev],
[qw/command/,'Replace',qw/-command/ => \&tReplace],







82


#close all programs, saving if necessary
sub closeAll
{
#Save all pages before closing
for (sort keys %pageName)
{
if($pageName{$ })
{
$files nb->raise($pageName{$ _);
&closeFile();
}
}
}

#standard CLOSE FILE function
sub closeFile
{
$currentPage = $files nb->raised();
my $page2delete = $currentPage;
$files nb->delete($pageName{$currentPage});
delete $pageName{$page2delete};
}

#close all files and exit program
sub exitProg
{
&closeAll;
exit 1;
}

#create a new notebook page with text box and insert data into it
sub newFile
{
#Are there any open pages?
if(($files nb->raised()) eq "")
{
$fileNum = 1;
}
else
{
$fileNum += 1;
}
#Create a new page
#if a filename was defined by open file, use it
#otherwise, name Untitled #
if($fName)
{
$fileName[$fileNum] = $fName;
$fName = ""
}
else
{
$fileName[$fileNum] = "Untitled $fileNum";
}
SpageName{$fileNum} = $fileNum;
my $files pg = $files nb->add($pageName{$fileNum},
-label=> $fileName[$fileNum]);







83


$files nb->Resize();
# Create a scrolled text widget and place inside the notebook page
$files t[$fileNum] = $files pg->Scrolled("Text", -height => 8,
-width => 70,
-wrap => 'none',
-background => 'white',
-scrollbars => "osoe",
-font => 'Courier 11');
$files t[$fileNum]->pack(-side => 'bottom',
-fill => 'both',
-expand => 1);
$files nb->raise($pageName{$fileNum});
$top w->update();
}

#open a file and then send it to a new notebook page
sub openFile
{
#if parameter was passed to open, open that file;
#otherwise, give open dialog box
if(@)
{
$fName = shift(@ );
}
else

#define acceptable types of files to be read in
my $types = [
['All Files', '*', ],
['Text Files', ['.out', '.txt', 'TEXT']],
1;
#return user selection to filename
$fName = $top f->getOpenFile(-filetypes=>$types,
-initialdir => $path);
}
if ($fName = /(.*)[\/\\] (.+\.?.*)/)
{
$path = $1;
$fName = $2;
chdir "$path" or die "Can't cd to $path: $!\n";
}

#Open the file in a new page if one was defined and the file is
#not already opened
#if same file is open, close current version and reopen new version
for (sort keys %pageName)

if($fileName[$pageName{$ }] eq $fName)
{
$files nb->raise($pageName{$ });
&closeFile();
}

if($fName)
{
&newFile;







84


else
{
return 0;
}
sysopen(INFILE, "$fileName[$fileNum]", O RDONLY)
or die "Can not open the file $fileName[$fileNum] in $path\n";

#use filename to title notebook tab
$files nb->pageconfigure($pageName{$fileNum},
-label=> "$fileName[$fileNum]");
while ()
{
$files t[$fileNum]->insert('end', $ );
}
close INFILE or die "Closing: $!";
}

#standard SAVE ALL function
sub saveAll
{
#save current page that is raised
$currentPage = $files nb->raised();
#loop through each page to save them
for (sort keys %pageName)
{
$files nb->raise($pageName{$ _);
&saveFile();
}
$files nb->raise($pageName{$currentPagel);
}

#standard SAVE AS function
sub saveAs
{
$currentPage = $files nb->raised();
$files t[$currentPage]->FileSaveAsPopup();
#retrieve filename of saved file
my $saveAsName = $files t[$currentPage]->FileName();
if($saveAsName eq "") #canceled save
{
return 0;
}
else
{
if($saveAsName =~ /.*[\/\\](.*)/)
{
$fileName[$currentPage] = $1;
}
else
{
$fileName[$currentPage] = $saveAsName;
}
# Show the file name in the page tab
$files nb->pageconfigure($pageName{$currentPage},
-label=>$fileName[$currentPage]);
}












#standard SAVE function
sub saveFile
{
$currentPage = $files nb->raised();
my $infoBox = $top f->messageBox(-title => 'Are You Sure?',
-message => "Do you want to overwrite the existing file(s)?",
-type => 'YesNo',
-icon => 'question',
-default => 'no'


if($infoBox
{


/yes/i)


#display Save dialog box
$files t[$currentPage]->Save($fileName[$currentPage]);
# Retrieve filename
my $saveName = $files t[$currentPage]->FileName();
if($saveName eq "") #canceled save


return 0;


else


#only capture the filename and not the path
if($saveName =~ /.*[\/\\](.*)/)


$fileName[$currentPage]


else


$fileName[$currentPage]


$saveName;


# Set label of page to filename.
$files nb->pageconfigure($pageName{$currentPage},
-label=>$fileName[$currentPage]);


#check to see if file should be saved before close if it has been
changed
sub shouldSave
{
$currentPage = $files nb->raised();
my $file = $files nb->pagecget($pageName{$currentPage}, -label);
my $save d = $top f->Dialog(-title=>"Save File?",
-text=>" Do you want to save $file? ",
-bitmap=>'question',
-default button=>'No',
-buttons=>[qw/Yes No Cancel/]);
return $save d->Show();







86


B.4.5 Text Editor Subroutines

#copy text selection
sub tCopy
{
$currentPage = $files nb->raised();
#$files t[$currentPage]->clipboardColumnCopy();
$files t[$currentPage]->insertControlCode();
}

#cut text selection
sub tCut
{
$currentPage = $files nb->raised();
$files t[$currentPage]->clipboardCut();
}

#paste from clipboard
sub tPaste
{
$currentPage = $files nb->raised();
$files t[$currentPage]->clipboardPaste();
}

#undo last action
sub tUndo
{
$currentPage = $files nb->raised();
$files t[$currentPage]->undo();
}

#redo last action
sub tRedo
{
$currentPage = $files nb->raised();
$files t[$currentPage]->redo();
}

#select all text
sub tSelectAll
{
$currentPage = $files nb->raised();
$files t[$currentPage]->selectAll();
}

#unselect all text
sub tUnSelectAll
{
$currentPage = $files nb->raised();
$files t[$currentPage]->unselectAll();







87


#find text
sub tFind
{
$currentPage = $files nb->raised();
$files t[$currentPage]->FindPopUp();
}

#find next match
sub tNext
{
$currentPage = $files nb->raised();
$files t[$currentPage]->FindSelectionNext();
}

#find prev match
sub tPrev
{
$currentPage = $files nb->raised();
$files t[$currentPage]->FindSelectionPrevious();
}

#search and replace
sub tReplace
{
$currentPage = $files nb->raised();
$files t[$currentPage]->findandreplacepopup();
}

#show About information box
sub showAbout{
my $aboutString = "Fault Tolerant Memory Simulator v0.5\n\n";
$top w->messageBox(-title => 'About',
-message => "$aboutString", -type => 'OK');















LIST OF REFERENCES


1. M. P. Frank, "Reversibility for efficient computing," Ph.D. dissertation, Dept.
Elect. Eng. and Comp. Sci., Massachusetts Institute of Technology, Cambridge,
MA, 1999.

2. M. Butts, A. DeHon, and S. C. Goldstein, "Molecular electronics: devices, systems
and tools for gigagate, gigabit chips," in Proc. IEEE/ACM International
Conference on Computer Aided Design, pp. 433-440, Nov. 2002.

3. S. C. Goldstein and M. Budiu, "NanoFabrics: spatial computing using molecular
electronics," in Proc. 28th Annual International Symposium on Computer
Architecture, pp. 178-189, July 2001.

4. A. DeHon, "Array-based architecture for FET-based nanoscale electronics," IEEE
Trans. on Nanotechnology, vol. 2, no. 1, March 2003.

5. M. Mishra and S. C. Goldstein, "Scalable defect tolerance for molecular
electronics," in Workshop on Non-Silicon Computing, 8th International Symposium
on High-Performance Computer Architecture, (Cambridge, MA), Feb. 2002.

6. K. Nikolic, A. Sadek, and M. Forshaw, "Fault-tolerant techniques for
nanocomputers," Nanotechnology, vol. 13, pp. 357-362, May 2002.

7. J. von Neumann, "Probabilistic logic and the synthesis of reliable organisms from
unreliable components," in Automata Studies, C. Shannon and J. McCarthy, Eds.
Princeton, NJ: Princeton University Press, 1956.

8. C. E. Kozyrakis, S. Perissakis, D. Patterson, T. Anderson, K. Asanovic, N.
Cardwell, R. Fromm, J. Golbus, B. Gribstad, K. Keeton, R. Thomas, N. Treuhaft,
and K. Yelick, "Scalable processors in the billion-transistor era: IRAM,"
Computer, vol. 30, no. 9, pp. 75-78, Sept. 1997.

9. P. Van Zant, Microchip Fabrication: A Practical Guide to Semiconductor
Processing. 4th ed., New York, NY: McGraw-Hill, 2000, pp. 49-110.

10. H. Kitajima and Y. Shiramizu, "Requirements for contamination control in the
gigabit era," IEEE Trans. on Semiconductor Manufacturing, vol. 10, no. 2, pp. 267-
272, May 1997.

11. IEEE Standard Computer Dictionary: A Compilation of EEE Standard Computer
Glossaries. New York, NY: 1990.









12. K. Chakraborty and P. Mazumder, Fault-Tolerance andReliability Techniquesfor
High-Density Random-Access Memories. Upper Saddle River, NJ: Prentice Hall,
2002, pp. 2-18.

13. D. J. W. Noorlag, L. M. Terman, and A. G. Konheim, "The effect of alpha-particle-
induced soft errors on memory systems with error correction," IEEE Journal of
Solid-State Circuits, vol SC-15, no. 3, pp. 319-325, June 1980.

14. A. M. Saleh, J. J. Serrano, and J. H. Patel, "Reliability of scrubbing recovery-
techniques for memory systems," IEEE Trans. on Reliab., vol. 39, no. 1, pp. 114-
122, April 1998.

15. P. Hazucha, C. Svensson, and S. A. Wender, "Cosmic-ray soft error rate
characterization of a standard 0.6-[um CMOS process," IEEE Journal of Solid-State
Circuits, vol. 35, no. 10, pp. 1422-1429, Oct. 2000.

16. S. Schaefer, "DRAM Soft Error Rate Calculations," Micron Design Line, vol. 3,
no. 1, 1994.

17. A. H. Johnston, G. M. Swift, and D. C. Shaw, "Impact of CMOS scaling on single-
event hard errors in space systems," in Proc. IEEE Symposium on Low Power
Electronics, pp. 88-89, Oct. 1995.

18. R. Baumann, "The impact of technology scaling on soft error rate performance and
limits to the efficacy of error correction," IEEE International Electron Devices
Meeting, pp. 329-332, Dec. 2002.

19. J. F. Ziegler, H. W. Curtis, H. P. Muhlfeld, C. J. Montrose, B. Chin, M. Nicewicz,
et al., "IBM experiments in soft fails in computer electronics (1978-1994)," IBMJ.
Res. Develop., vol. 40, no. 1, pp. 3-17, Jan. 1996.

20. J. L. Hennessy and D. A. Patterson, Computer Architecture: A Quantitative
Approach. 3rd ed., New York, NY: Morgan Kaufmann, 2003, pp. 11-56.

21. N. Sakashita, F. Okuda, K. Shimomura, H. Shimano, M. Hamada, T. Tada, S.
Komori, K. Kyuma, A. Yasuoka, and H. Abe, "A built-in self-test circuit with
timing margin test function in a 1Gbit synchronous DRAM," in Proc. International
Test Conference, pp. 319-324, 1996.

22. K. Chakraborty, S. Kulkarni, M. Bhattacharya, P. Mazumder, and A. Gupta, "A
physical design tool for built-in self-repairable RAMs," IEEE Trans. on VLSI
Systems, vol. 9, no. 2, pp. 352-364, April 2001.

23. K. Zarrineh and S. J. Upadhyaya, "On programmable memory built-in self test
architectures," in Proc. Design, Automation and Test, pp. 708-713, March 1999.

24. R. Treuer and V. K. Agarwal, "Built-in self-diagnosis for repairable embedded
RAMs," IEEE Design & Test of Computers, vol 10, no. 2, pp. 24-33, June 1993.