<%BANNER%>

Medication Use Performance Indicator Evaluation: A Systems Perspective

xml version 1.0 encoding UTF-8
REPORT xmlns http:www.fcla.edudlsmddaitss xmlns:xsi http:www.w3.org2001XMLSchema-instance xsi:schemaLocation http:www.fcla.edudlsmddaitssdaitssReport.xsd
INGEST IEID E20110114_AAAABS INGEST_TIME 2011-01-14T10:54:40Z PACKAGE UFE0006469_00001
AGREEMENT_INFO ACCOUNT UF PROJECT UFDC
FILES
FILE SIZE 8423998 DFID F20110114_AABASI ORIGIN DEPOSITOR PATH sauer_b_Page_100.tif GLOBAL false PRESERVATION BIT MESSAGE_DIGEST ALGORITHM MD5
035fd2699f704b0272fb31e519f8fcbb
SHA-1
6f518dc9ce2825ec3ad341c909b9842401b6ae81
F20110114_AABARU sauer_b_Page_080.tif
f8ada89aeb6e0154d61506a195611d76
3c227a197c9c2658c872645c8922f4bb07e89f18
F20110114_AABASJ sauer_b_Page_101.tif
6f90c21295957cd54965df95a6a1a2f5
f07969c0bcf7c28b6b9710a863efd7ee677fece9
F20110114_AABARV sauer_b_Page_081.tif
66be24b2424008c2d892a4fbaf01384c
3f52e87d0a88d774f1cede3ae30640f16d44fa31
F20110114_AABASK sauer_b_Page_102.tif
3f5f2b62655fc5ae623fdd5a2916affe
a6a9b3d29ee03ece82da9ab480dd1b387dec7296
F20110114_AABARW sauer_b_Page_083.tif
5e43463b130a7583637405615c58a74c
fd3ff0b18afa2ed5afb765844e30a1b015fa63b3
F20110114_AABASL sauer_b_Page_103.tif
420e317addf99a49d06ad511ba633d3a
4e8182139477d9de28ec5ce9835d156577f0a363
F20110114_AABARX sauer_b_Page_085.tif
c96336805a28d05a5cc4a8f415bca348
b1db076605d0c85e37b9fd64819578dde9a3d95e
F20110114_AABATA sauer_b_Page_122.tif
bc112b50485038418e45e7996bbb0092
441062e8597a3ceda909ecd019a9d9ed172dd16c
F20110114_AABASM sauer_b_Page_105.tif
889e58ff485aaf15cd58cdac12afa0ff
0a42855a24d20a4e5a79226b9a2d59c3df4815f6
F20110114_AABARY sauer_b_Page_086.tif
67cbf512bed36a543ab6ec6117a3810f
17a79e1878e34e89385fe148aed2544045fc5ca0
F20110114_AABATB sauer_b_Page_123.tif
025e20587156d84170fa62387a82654a
e7965105c7ca7c98e5bea19598098a01d7f6f7d6
F20110114_AABASN sauer_b_Page_106.tif
9e2432f561ff7d4c15e2742dc7e35a63
2793cfb9408322db2fde8090facab34e61650b43
F20110114_AABARZ sauer_b_Page_088.tif
2a35f433c204878c20d03125efe4e3e2
1fb2b4725075c66171ad0136033d9a88ff64c85d
F20110114_AABATC sauer_b_Page_126.tif
946b54a4f3a1cf2c82d5fc99a307551a
a28aeb8327f8f87c1aeda0c854e87e3c093c5ed4
F20110114_AABASO sauer_b_Page_107.tif
dcc11eb0dcd96a9631efce53b3ddf17d
44e465e35fb830f4ae949c964de3a2b55925445f
74270 F20110114_AAAZUA sauer_b_Page_160.pro
a751ed074eb76943510b60f8e9db04d5
6da87d0acb0c75d052b60f955a3f22bafb3d30ba
F20110114_AABATD sauer_b_Page_127.tif
b73bb65f6e604c65877f4b2d90ec4b7a
55e92c68e879aee6f8d81f0f84aba24e514a5669
8425398 F20110114_AABASP sauer_b_Page_108.tif
f684426707844a38ac0eb07c0818b134
4785d30d6e4c3dcfc7b5395e374ee268fd1934b6
F20110114_AAAZUB sauer_b_Page_152.tif
98ca28f6758ca16d73a6d3bef0c0c72c
8ae9dad703418454a524a45ee5b70f64355b2cce
F20110114_AABATE sauer_b_Page_128.tif
cdc1d08fdb3195ef2bb556c3609a013d
422fd214409ccfa1fdcc7c52341ffa83fa449784
F20110114_AABASQ sauer_b_Page_110.tif
9faf1938337e8bad35973381cdfa32b4
6462d3157bc7edbe9530e6e653d71e6551d76064
44100 F20110114_AAAZUC sauer_b_Page_156.pro
378b68a71ec6bc55d47bcefb7ce9e48a
e8a78df4c2f16fb0c4ec960789dca8c8a41be39c
F20110114_AABATF sauer_b_Page_129.tif
00b3db3cc85737d30d80c0d04e021a9e
a9b69396e59cea537fef8c13ae79a4c2cf72f361
F20110114_AABASR sauer_b_Page_111.tif
8060fb4a776ca741492237b29c83cac4
d0078954a0fa820188f716d9b742f88afd68b120
29787 F20110114_AAAZUD sauer_b_Page_018thm.jpg
f3302c254209a191cb218d7dc8279751
d4d6441d517edf94eac3a03052594a5fe523399d
F20110114_AABATG sauer_b_Page_130.tif
d79e83f9a22537963d445dc9ed649312
e49833b314ac7b2125214ea644fed2a93c536b09
F20110114_AABASS sauer_b_Page_112.tif
ce75b84fbb863de40efe8b0b279565f5
fdba3bc365e45d73871df62659d718fc6abee21a
F20110114_AABATH sauer_b_Page_131.tif
7cb4383f08ea2183ea3ee4599b784f40
ebcef20d9ed226bf9077d5128ac38a299519bc5b
F20110114_AABAST sauer_b_Page_114.tif
764ba15884917ccd0a4bf9833919fbb7
71d17c2484c92468950bfc779a858c17cc479b8a
1730 F20110114_AAAZUE sauer_b_Page_129.pro
df0d710322d2f699cfffb16fbec5a059
eceed5ac944fd186d525eaee110ffa9878b15207
F20110114_AABATI sauer_b_Page_133.tif
3533eb794b9f5463d68b4ec609ab56b3
5ba08412f13dcf377484ff8258d31a5b47727904
F20110114_AABASU sauer_b_Page_115.tif
10b66cc27bd0213498cf5854a8f5566f
f6c030268308070d93d0ce574dc4461bf36bcd78
88038 F20110114_AAAZUF sauer_b_Page_063.QC.jpg
149c26850ba42f0092b2e85aa7b89441
8eec2d0a633a0a0cbdf9d3aaf8a3c1fdb5590ea2
F20110114_AABATJ sauer_b_Page_135.tif
3a188910c7d6db4fbcfba5c13b6ee353
1730bf3e9b23868905bdccfe3af1cb7e72fab4bc
F20110114_AABASV sauer_b_Page_116.tif
c5721f279abef770a02fa3d8ead54f6b
215a954b29bef18897ecf6bfea0e7b6b5a1f861d
231778 F20110114_AAAZUG sauer_b_Page_062.jpg
ffb343b431400dca0116fec22e299c64
b98cdad7471f0752354fc8a1035c24563740d806
F20110114_AABATK sauer_b_Page_136.tif
3011deb074d48a95fee2c7764a1bd388
8dfc6cb64a20906ebf9eec30ac71fe310202702b
F20110114_AABASW sauer_b_Page_117.tif
43826dfb5999b55118099e02d0fe5f4b
2481fa23601baa7294af136ea5812925a1bbf82b
47171 F20110114_AAAZUH sauer_b_Page_069.pro
717664659639d1bc7d8f8caacd2eb50d
41bb94b123931331b082038a2ea9a0872cd78320
F20110114_AABATL sauer_b_Page_137.tif
b6516b93f96236382da34c19f8a8f2d6
515d11d7977f6d6bff2e423a66c3648c2af27057
F20110114_AABASX sauer_b_Page_118.tif
06142941d078c0131df07cef0e95c797
53ad127096488e8a61fe7e1afab4b52b7bc94cb7
F20110114_AAAZUI sauer_b_Page_144.tif
e7a4713acb217d13cc783640a1873118
da3ef893ef9d0a969aba3a423068b445f865da0a
F20110114_AABATM sauer_b_Page_138.tif
359175d927b5c078d41228cdff43818b
4fa9281bd8ec63e14b532d324ac4bdd2b93bbcfe
F20110114_AABASY sauer_b_Page_120.tif
5ed527745b333f6850f85b60a1622f1a
7942a0d05be742c3c0453953f761e105b4fcd690
32060 F20110114_AAAZUJ sauer_b_Page_098thm.jpg
4c566f61c3dc4ea23782f8781d57d3b5
265edcc5404888eff0ee5cdb0944ad40eb227ba1
F20110114_AABAUA sauer_b_Page_158.tif
96cf7a8f23280b344066ae8d7a66672a
10c86206ab083c4d7c1fa9d6559bb0bcfe573260
F20110114_AABATN sauer_b_Page_139.tif
8ef7c566a169da6f654e1e50230acce1
3ea16fbb40bc5045017985b3a05ee00cd5f5d80c
F20110114_AABASZ sauer_b_Page_121.tif
9f6e1071696faa9624bdcdfe6ca63ac6
4182d44ecaa238492cfe020b9616f474345b993d
65883 F20110114_AAAZUK sauer_b_Page_012.QC.jpg
fbeeb0ca5d944f6d64e28d62fa102b65
64d189782009ad5ebcdcbede58011ef2b070faff
F20110114_AABAUB sauer_b_Page_159.tif
4d94923ab6c00a93bff2e9b0f6801c91
fbaf0f84440bfc7f663c9816238bf2db38978c91
F20110114_AABATO sauer_b_Page_140.tif
2764a49cea1c9173402804e9a5fd9caa
de1365ca043585c0617d595e0a135e30a17bb340
2101 F20110114_AAAZVA sauer_b_Page_099.txt
d26b8ffa0a997abe2c524c11a9ec9587
72bcf1479c4feefabc6c5568198cfbb638797c5a
F20110114_AAAZUL sauer_b_Page_064.tif
56d5a9f41a2fd493874866581a3a0389
7ea84bf67243829aa8e9c84fbb816cd1702b832c
F20110114_AABAUC sauer_b_Page_160.tif
cc26cdc5417708b0fb66771847c0861a
d4be7b6882cb0bfc7550397e73045048cc7fbf40
F20110114_AABATP sauer_b_Page_141.tif
0c82b0280fa2301d997a1b63f9680822
0b352ac1eb65d641234d506ca6d859cc8ff33f2c
31682 F20110114_AAAZVB sauer_b_Page_022thm.jpg
4cd2b9fc3434d2175146309b6fb93a2d
f1760be3869f3b20cde8bbdbfe6f077e86c19af9
F20110114_AAAZUM sauer_b_Page_119.tif
10b8536fcab3aed6f4131197ebb5cc40
a0026d415f236e12d1a2445b965939cc758af53e
F20110114_AABAUD sauer_b_Page_162.tif
40a0382839ff62c322ab2b1023df6769
cbe68a1049ae6defd1f0d9808a0458305bdb0308
233058 F20110114_AAAZTZ sauer_b_Page_107.jp2
993bcf03eb909e3454e4f3c65c900656
7fc757fd675e96e82e4c2f6a940e1f66800a9717
F20110114_AABATQ sauer_b_Page_143.tif
06ca28c0495e9b2ea72e319d2f096c57
f4ec060fbf7bfdefd4e2ec7ef08ccf8bda824b12
7808 F20110114_AAAZVC sauer_b_Page_001.pro
8e8c93f66174aacab53a226d05a4e486
94be92a27f3ae97bb03e70624d6b81e80149312b
211999 F20110114_AAAZUN sauer_b_Page_078.jpg
305846e085d1e35f8ca9b2cc0419f37e
1846ef92db35c3afc2604e672af2d6cc40920e14
F20110114_AABAUE sauer_b_Page_165.tif
a851cbd87263631eb33515c83c890a92
bc492a4d4d4d002de6c8e7920009e77986b091cf
1995 F20110114_AABBAA sauer_b_Page_019.txt
ecf50925a6ef4b492b98e7ba6f370773
14212d995b2330c1f8c55b375a5e0818df7d54c7
F20110114_AABATR sauer_b_Page_147.tif
6a78981b7d65da032149aaf19b372ee4
6199e20ce546841364878f2b888791df0750d196
F20110114_AAAZVD sauer_b_Page_073.tif
08755074602ee7649380755203a9eb9d
99822a1542ca0d88d2032f50e4823644c786bea8
811395 F20110114_AAAZUO sauer_b_Page_072.jp2
30e845f868ab7812e1837a824f2fa677
622d1c947314196e52ce16f605cb0e78dc87adcd
F20110114_AABAUF sauer_b_Page_166.tif
57e270f424c69ad186e3172e2b29fe5d
041c094762e7c52e4b191c333fbdb2c48b35c6fb
2072 F20110114_AABBAB sauer_b_Page_020.txt
9740a847ba00f66b91d9f98896c2901f
0473cfd457d54ea37c36ad8b0df07090f1f79b35
F20110114_AABATS sauer_b_Page_148.tif
f0e97689a3a793e0a9dc55756e47e358
4d93218fdada9771fe5498e7e6f7b68369b7d31c
202742 F20110114_AAAZVE sauer_b_Page_110.jpg
1dbf17c63a5e4821108f8383612705a6
c3a21c0eb8ce9bac3c3d46a764f926d10a519c7d
35671 F20110114_AAAZUP sauer_b_Page_004.pro
268399cd93b66da614fdce5c15e04cb7
8c877d0c206097410d4b7d09cde8b6b5aaf98bf3
F20110114_AABAUG sauer_b_Page_167.tif
b382af26ea316b07814482a82870e67e
d130a8021e40967388fa655c277ddc6ba6d9e29d
1933 F20110114_AABBAC sauer_b_Page_021.txt
088a4df2dc690056c6aef6a37528b508
251295158a6bdb9b0baa333f3d9ce9b56b7c2209
F20110114_AABATT sauer_b_Page_149.tif
a9177b50b67f1132301fd1983e2df142
9ddb5f34fca1a1c71b6274f9b07c656166c0802c
23369 F20110114_AAAZUQ sauer_b_Page_009.QC.jpg
aca6e8aded027562ada33f7f26f0b09b
edc29cea83d140cec71fc58ebe18e89e7fe13582
F20110114_AABAUH sauer_b_Page_168.tif
29a494a44065b6801f08aaf4dfe87cc1
5d0596a4a7865573a25398e45e7e98c74b5d464c
2118 F20110114_AABBAD sauer_b_Page_022.txt
9ec801872495719b2a0b39597523c698
c40d301e42b17369d50634943fdd25599dcba86a
F20110114_AABATU sauer_b_Page_150.tif
0b7b52397bfd225286571bf22f33fc8f
cf1a868753f493869e6083d0ee0d5fcb9db99c94
50242 F20110114_AAAZVF sauer_b_Page_124.pro
8094150fabb86a2b197007e8c3015e82
9d9f10e1447859824088f57e5d4c6ee945d1326a
2271 F20110114_AAAZUR sauer_b_Page_014.txt
7e3cac428eaaedbb7375b9dbb56c2d50
e4fbec7d1d47f11ebc52d3f0fd2db71f30d1b6b3
F20110114_AABAUI sauer_b_Page_169.tif
dce26b3ee56c3170469564062b9dfff8
11d459fb53e0eda8e548b3ee183e3b3c467b5227
1917 F20110114_AABBAE sauer_b_Page_023.txt
7e9acf4a1148cfb61978c657287cfcc5
c5e947718556608550dc5adaf58ae3425aa123c7
F20110114_AABATV sauer_b_Page_151.tif
d1f579788b2bb46c7eee013371c3e294
12a96a0b71bc34c8c9c265faa6766f44465772d4
29041 F20110114_AAAZVG sauer_b_Page_044thm.jpg
cd67910be4d46fdf68cd6eb36df5e047
673a614563720474b123f10b829e39b29a2c1b43
1916 F20110114_AAAZUS sauer_b_Page_096.txt
73c2310deee025c688140d929dc5798d
f1d44f1b9b1feb9253461df3fbf16c8b02d664a0
F20110114_AABAUJ sauer_b_Page_170.tif
f10dd57f8aa0c097b4ec869a0b3bbc7f
125eecaa2ff6fb01b69a2e348124ab08995d9525
2008 F20110114_AABBAF sauer_b_Page_024.txt
49c830f35d5e1f98974301bb293f9b00
43e9480e51f45c4dce5203e7e7d88eba3a6c24bc
F20110114_AABATW sauer_b_Page_153.tif
241c21041f0527327054a4274a0b3405
bae7efccf76fe575b185643d0218a83cf275c39d
2204 F20110114_AAAZVH sauer_b_Page_062.txt
549f8c10fc4e593e33f47f8cfaebf646
5df1f3f9c73cf8a3498cf4724a12516f98e217bf
F20110114_AAAZUT sauer_b_Page_002.tif
c5531ba4bac04f2ed1a61818a38a28e1
ded9af578d32aebcac6f0cc60b12a0b589cb625c
F20110114_AABAUK sauer_b_Page_171.tif
d058b4abbc93c861bad5d5112bb57714
6ba43f838b27e169be259fa9258196daaa9b884b
2069 F20110114_AABBAG sauer_b_Page_025.txt
90c3bd76a2331297af906740f3ff02e1
349b35b0fb4517c9f99242f18e0ba5ac777be76c
F20110114_AABATX sauer_b_Page_154.tif
dc31bdacdefe834c05e43c92e86dd2b2
700a0e29b16e6abf14182e3c1b5b571e5f010795
2137 F20110114_AAAZVI sauer_b_Page_089.txt
1553d5d42d544e56fbbeebfb8832f0e4
aa1ca74b22c77000a7b68a4d064e5bf15fe91e70
32679 F20110114_AAAZUU sauer_b_Page_111thm.jpg
63bcc2e4338650361ca6a8b32e2c0bae
89680009b5e43cceec5959201103ea3f8b9e8dba
1147 F20110114_AABAUL sauer_b_Page_002.pro
3c59669552cc02f94d2df410ee90d9fb
5527cc8f07fc0d2bbe75c3f2090dd295ff2debd7
1875 F20110114_AABBAH sauer_b_Page_026.txt
3e49d19313eac1a6aca51e9ae7da3e38
65b8b742cae203c56ec7d8f5eb49d293f3246fa2
F20110114_AABATY sauer_b_Page_156.tif
c18f1ed9f410d7db452fc55c61026931
c2d5d7f7fcd9d25a369ed8a4ae79607433e71732
28248 F20110114_AAAZVJ sauer_b_Page_026thm.jpg
e587b87c4da48883b2f9fb797e91b52c
43919ecddc769ab7a5c7706c94d943c0a37e3668
47994 F20110114_AABAVA sauer_b_Page_021.pro
916928462a97c2e71ddefdfce6f12f0f
eeac0e8a1449826eb09d5990a1564962a10ad68e
140766 F20110114_AAAZUV sauer_b_Page_085.jpg
bf11c7f35d4bd832bfe009ba3e2df410
de18f6f6b8f4f4a092939f848befa8f1401c1f6a
69257 F20110114_AABAUM sauer_b_Page_005.pro
11ce8b307cb29bb8f56e1d18b65bce10
5aeab482f138dae88dd215b3ba504672f221edfa
822 F20110114_AABBAI sauer_b_Page_027.txt
6489d1e87f533e2a74fcb634d0353122
9633899f7a56f07b315735abfd40ab71292af27b
F20110114_AABATZ sauer_b_Page_157.tif
7b544b6d181585699b93f631517db232
c0ccc524eb55522916670b0cb52b2e4e62bb26f5
12280 F20110114_AAAZVK sauer_b_Page_140.pro
c23eb40406aca03399d0d01ad451b0d1
1507b0606288185450abcd059ad156e9f460a24f
53160 F20110114_AABAVB sauer_b_Page_022.pro
dc82b07c1db92baf9e8fa75b3c63ae53
d7e766dd0ad98c7138b3bf68ee3ed4fa906476bb
1707 F20110114_AAAZUW sauer_b_Page_011.txt
7294460980714b8a82e19fc658653de6
502ae2815a876b064a416347cdaaf650cbcb0281
114333 F20110114_AABAUN sauer_b_Page_006.pro
a63f0c669db65b01c20ce16561180634
ffad19cf3376caaf26a7643d519c18a968cfeabf
3325 F20110114_AAAZWA sauer_b_Page_057.txt
a153c0ba17dc8774ca88f974b0241375
0fd7ecd25e4b7822d66b4477d049551db3bf8be8
F20110114_AAAZVL sauer_b_Page_132.tif
965151f043cc5bc9a584fc006e95b944
7c19cb531e839a4e084df200eb620ca994cd55b1
48408 F20110114_AABAVC sauer_b_Page_023.pro
5fd36bcbbecbaaa0e6dabb815545d904
2d814adf3b28f40b2945da737c5ccb92470e1325
24886 F20110114_AAAZUX sauer_b_Page_008thm.jpg
b7d5aa7bdf27852fde710f0ef75e1009
422fb195b89e2673c06b2daea2c93688e76993ac
80337 F20110114_AABAUO sauer_b_Page_007.pro
17c9b80ab32fe017e9e307231ee672f6
5960613dd8ebf594fb42ffa8da06e533c06f194b
1547 F20110114_AABBAJ sauer_b_Page_028.txt
27fd44cfc50cc6680f0966d94e6df3d4
cdfc84ae54ea88cdddd4c77da658d499357be817
1051983 F20110114_AAAZWB sauer_b_Page_073.jp2
4f2a89ceb840e158f3fddae63b301232
512fa15535762eb57029d58918ac74bbfe9c4e25
F20110114_AAAZVM sauer_b_Page_136.txt
e8fa04e9026b21ec671e6330c9119f30
c6d195d152104da8a5c0096475515f4f314907d5
50690 F20110114_AABAVD sauer_b_Page_024.pro
8e19b1c41a7cfe7df4fb801c192bba25
4e4692feb7eb86be36cb04fc000f0136c6fc576f
33040 F20110114_AAAZUY sauer_b_Page_124thm.jpg
1eaf9e1335fe30f05eda1c569a925459
519e7e0651e63e20653bf8d1e755bd8294a3f581
55094 F20110114_AABAUP sauer_b_Page_008.pro
2250d9b59476457d11c13799afb05259
08c76edc531074623440b64dcd5fb86824522c2b
91 F20110114_AABBAK sauer_b_Page_029.txt
f09d19a68e078205b049af6a4291ca0c
abf9eca1feab55e8e0b341a70205c763afe0730b
1878 F20110114_AAAZWC sauer_b_Page_069.txt
b20f41d069c7e7b13f50df1d51d34a0a
273ca22f21e5130840a15391ad7079924364f625
1642 F20110114_AAAZVN sauer_b_Page_147.txt
438b1694219081e590dfae68a101784b
27834116a7ecbee9ed74f6142d805f1ea968f9bc
51468 F20110114_AABAVE sauer_b_Page_025.pro
461f28f79ecfbc337e1f2bff103d1b8b
744082b74f72607dd93ab14b68b5a1a53ae6c302
F20110114_AAAZUZ sauer_b_Page_035.tif
68d3f9ebdbe8b37702002891e354598f
5f3690ac3134f7f1976b7f70fa182b983c002418
10590 F20110114_AABAUQ sauer_b_Page_009.pro
68d100e08aba29c2ac0847374abf4d6b
1167b54088a3b6882443ba5814b570aa8059da6a
1271 F20110114_AABBBA sauer_b_Page_050.txt
291d6afd0e456ea700ee5c49e8fde29f
c562bfb9a85e1f419987c28701d5bafba9a2cffd
1844 F20110114_AABBAL sauer_b_Page_030.txt
a5afc6533658c411cf3568c7b3f5b79d
2f033e658e877bfded3831e20d2dabe5aa10c546
133977 F20110114_AAAZWD sauer_b_Page_153.jpg
765f6014c03b4192bcb8fab1bca59d72
0f6ef319b1169b7777b4aeb7c98d2334311af844
32350 F20110114_AAAZVO sauer_b_Page_168thm.jpg
dc62b810cf64f58b5c70d4e3b0867365
553e5e49d95d94c781b91657a23d650a911f406b
38354 F20110114_AABAVF sauer_b_Page_026.pro
f8d4e44de767a99d10cd87f02f8569cd
3a79e9841af2c80bc0e4d5c3c87665c9028d959a
55768 F20110114_AABAUR sauer_b_Page_010.pro
0affc171d5a97fd1b9d53f59ff9e3c83
e65dde96f51b023b782c6909aefefba4949112e6
1915 F20110114_AABBBB sauer_b_Page_052.txt
09ad55e9dbacfd76d33bd3b474766b14
a0f0134b9372d436883f2af2bc220986466623df
1911 F20110114_AABBAM sauer_b_Page_031.txt
047a50bbb0eecd22707c2bd9a83d9e4e
e61e0fe72bbf5baf11da2e62e83fe505951ef164
31940 F20110114_AAAZWE sauer_b_Page_086thm.jpg
e53448baf779919a2ed178c26a422b98
ab2f330712a652bf9df9957250abfc24505e805a
84685 F20110114_AAAZVP sauer_b_Page_096.QC.jpg
63ed790da1f37a6bab9f53ae055a7a93
c8d29e30f746f4fb9a875efd4258e6b4fce63349
20897 F20110114_AABAVG sauer_b_Page_027.pro
814b8f89b2a92bb531984a9f6495f233
410fd7dc50bad1e95da9a97cbc0efe8f76bf8ac3
38249 F20110114_AABAUS sauer_b_Page_011.pro
d0b75271632a123630c6fff525fd611c
cb3ff3811ccadb598f8bcacdcaddf5d2d9c0046b
2039 F20110114_AABBBC sauer_b_Page_053.txt
b9c34c5ad2df9376c9d8c302fa128b00
d2eb830a23c46a09896e3967b4f76d7ca82f9d85
1926 F20110114_AABBAN sauer_b_Page_032.txt
3191ab3a879fcec83c14281f834227b7
c4c9132cd3f2b815a169fa4a7c97a61505be790c
68166 F20110114_AAAZWF sauer_b_Page_106.QC.jpg
be08a9fddf80db5a0771384757bf5684
311029735868580107ab5b3d89dedcb6c760f402
F20110114_AAAZVQ sauer_b_Page_155.tif
9debf4824cfa629ff5975a25fd437221
771425d991f0e8dafce78eccb35b1f2c99174183
37873 F20110114_AABAVH sauer_b_Page_028.pro
d76948639bf4ae5db9fc16d133338464
f4f4ceb905ccee6153ffd47a8fedafc350fca492
35095 F20110114_AABAUT sauer_b_Page_012.pro
81c0b72e0a786b5d4a9ea7d8805a7f3b
e948040b3d26569854423b0761336419758f3b36
1879 F20110114_AABBBD sauer_b_Page_054.txt
77c45630944416355fadd128e6a4d944
df403e16009922da62c28213bbce524871f78e61
1804 F20110114_AABBAO sauer_b_Page_033.txt
909aa7b45c37aa8067c775e2764693dc
6e4865d99b7fd66d9ea211fe332266e7c1711061
31780 F20110114_AAAZVR sauer_b_Page_076thm.jpg
01a593bf0f760f66d5707741ce166974
b6e149292316c4cc4d94e0f638b6179f31c97ab1
1533 F20110114_AABAVI sauer_b_Page_029.pro
dc593688590019ced96dd063021e9a31
d47e41f7d601a6d0bab764eddfc807b8d3822bba
35683 F20110114_AABAUU sauer_b_Page_013.pro
16494f5d5e1e064d9233f0862337c58e
06398e8ed02d23cf14751086a616bdb94822dd60
1610 F20110114_AABBBE sauer_b_Page_055.txt
c50d3efb5e35aef3601e9bcff52552e6
30f7c3fac5ebefc6aac081bfc33e07591dc68a7b
1842 F20110114_AABBAP sauer_b_Page_035.txt
fd66f51a34ab716939470ad5faa9e5a9
927145d20ef90b80d50db19b417a60763f95c1df
181341 F20110114_AAAZWG sauer_b_Page_081.jpg
08f550f68a53dc562e957313e2d1a835
ab64c4431759a32f5be5faa3f23e3b4e1c152578
508 F20110114_AAAZVS sauer_b_Page_163.txt
51278dc65890c58ef1d650d124cd842e
9d1d5b7784ba465eab3a9b3f0f64e73c0edbf732
44606 F20110114_AABAVJ sauer_b_Page_030.pro
2b04b340e82156513e61622d2d1e49c5
aa02c4257dcdcb2e5524aa60601a39f7b4f173b0
52752 F20110114_AABAUV sauer_b_Page_014.pro
f6f808a6f83c775e148e807894878299
eff08e79647807ab3f2ee7f2ec3b2bd0e983921f
2046 F20110114_AABBBF sauer_b_Page_056.txt
537a9a4c75bd6a5e2f631510ae3a35f4
1f06697793da6e48032dc6c44efeeaede924343e
1973 F20110114_AABBAQ sauer_b_Page_036.txt
2a505c2cad6ba85f18b4cbc004ee2616
97be12557863a342dae1d82f5282371686338bb4
2538 F20110114_AAAZWH sauer_b_Page_168.txt
bb3461332c993dd6a119baaf598f57f9
8e87f400d126d9c95294c8ee356727116a90b3e0
772591 F20110114_AAAZVT sauer_b_Page_010.jp2
9cfd33bb3f9f6e74a286dc12b52a2e6d
f3e7a5101fb2fef148fbde5ee49ca6e008dd3e8d
48751 F20110114_AABAVK sauer_b_Page_032.pro
de7b8616e0e385b7730cd4c3098a70a8
b2ca412b35046a733ead09c8a9a9ed22a941c67a
45173 F20110114_AABAUW sauer_b_Page_016.pro
dd43e401cd0a06559cfc3cf717501ace
eb9cb8291d6ca212ee32320bca813a26da87324a
2358 F20110114_AABBBG sauer_b_Page_058.txt
34c6d4cc4e8bb02f28a2b77746cc532b
86499f228d7ce8c32738cef9174228acfbc32bf8
3108 F20110114_AABBAR sauer_b_Page_037.txt
1c251d0f5ffcd4aa71d845adcb16896e
795074bc953c58fa03b186bd0c01936836f195ad
18009 F20110114_AAAZWI sauer_b_Page_139thm.jpg
facb8db59450136a54443e9707c09ae3
8ae210985f41af5c8045ef3e19a028d5b320253c
51801 F20110114_AAAZVU sauer_b_Page_157.pro
735a0547e25058fa80e217534651832f
f655ca6f7f285983074fb307d0f90aaf77b02481
45568 F20110114_AABAVL sauer_b_Page_033.pro
7c089971d8ee02c6ca3dfb0fa378bf73
ff26f3fbe532b12bea39550cf25b4cad04279244
44756 F20110114_AABAUX sauer_b_Page_018.pro
7ef08f6adf60fbc5c0de9c916cae6c67
4a601ac316e9b89d0ef8344d101575af0adfd0e1
1732 F20110114_AABBBH sauer_b_Page_059.txt
19e7440a113d3d2798648d682992a93e
f051e6c5cdb4ac3de1122db3ff78ee6dbc76ff32
2097 F20110114_AABBAS sauer_b_Page_038.txt
2f14462d309e8d5e6ea17faa2359a7ac
8a0b2ee93349f39e9a015a7f3dfdadb88be58660
F20110114_AAAZWJ sauer_b_Page_082.tif
1b433e159a70c82e4880d5d4cd8962c0
8ba9dc9e17cdf693dea9de92172d9522674bad38
49056 F20110114_AABAWA sauer_b_Page_049.pro
886cd9cf36bafc6a34599f5c30201c40
3c357b1847e1727dc14742dc7356450f0d0985a2
33135 F20110114_AAAZVV sauer_b_Page_138.pro
63971581813ba0240f092bea3deece07
b4e9fdcda120c9e63d229f01b2167df9bd588a9b
49396 F20110114_AABAVM sauer_b_Page_034.pro
7db9766b726f1a64d69f18b2ba25d50c
559ef9dcc140d6612065e5b7fcd06d412e339738
50576 F20110114_AABAUY sauer_b_Page_019.pro
b5456588e1e7c1d0424e2a667e0862f2
0b083a758e7a7d022a29c5132ae9b7a085984f04
1770 F20110114_AABBBI sauer_b_Page_060.txt
36afc83ebe27453b8785334a3a2cb5a9
534227b75bed07a94029d1dca0593e0824c59704
1808 F20110114_AABBAT sauer_b_Page_039.txt
f4507f5c445e50ca033e122d51553f29
02f7cdc0c28b347011120a5fcbb21d41d930cbee
205871 F20110114_AAAZWK sauer_b_Page_007.jpg
3d650ea9e108a529ca6464eb45d8ff52
47cccc7bab2052cfffdc0a0fddecbe1ebab0efcf
47507 F20110114_AABAWB sauer_b_Page_054.pro
39b97b2e8e0682a60bd5ef726a36a2d7
49a7481c817c9ec58d2c4f396eafe0b84f8ec319
F20110114_AAAZVW sauer_b_Page_161.tif
3dda2e34972560e99c5d99c8c2188f8e
e5895792f1d002208f7eed98cfa2f8e01b8a7065
46427 F20110114_AABAVN sauer_b_Page_035.pro
2272529feb183981a2b9564c7a79d12f
e3291c484e3bc627f2ab70af93ae402541606fce
51543 F20110114_AABAUZ sauer_b_Page_020.pro
cc2f9393bbe605da4deb07d3bcc7046a
b0f246f1b6416cff2eafa11dc81e3a91d01db215
661 F20110114_AABBBJ sauer_b_Page_061.txt
87aa5dba0918a1b75dc2805ab40b8548
300e6c35725e0d19724ea5ecb04f541403d30bf8
1816 F20110114_AABBAU sauer_b_Page_040.txt
f3d9e995f7c6406c604f915b00063f33
fc273d50c8de53f2d0feeef8121241a59b048b38
F20110114_AAAZWL sauer_b_Page_109.tif
e2803287553cf1b425532c981f0d6180
ce97bffb57cecd3397200f3ab909989c41b8eaf3
39622 F20110114_AABAWC sauer_b_Page_055.pro
8039a965dee8e47047f6a9a84a2c3495
7b7c3a41aa171998c58e05aaa6bc4e4534ee3c96
F20110114_AAAZVX sauer_b_Page_084.tif
dcd032121e2b1fe472cffa36cbc37413
b0152ebdc247df22ee5df656615ec834e36e734d
49808 F20110114_AABAVO sauer_b_Page_036.pro
7d5af4c4f938978b3a6ad5ac8bec4390
4acc5ccd6ee5cc4799bcb2a8c51827cc60bb7ba8
F20110114_AAAZXA sauer_b_Page_023.tif
4fe2687f26ae0555587397ed8e4bd64c
45391c4b4facad5fb561e3be25379fdc8a6953a5
1988 F20110114_AABBAV sauer_b_Page_043.txt
f9efff963fb68bc7d7b4c44e330c6118
0cd95a9b297d13c416f9cd6979b013c40558fce8
1924 F20110114_AAAZWM sauer_b_Page_134.txt
0a5d13a5f4709127565f3106deb6183a
4cb554a26dde2fc601b3d429c4b5819419516e04
42383 F20110114_AABAWD sauer_b_Page_056.pro
8a815459bc7fc5e5973a73c67ba9bf05
9d6c998f755aeb2cb6f4955d6effea06c7039efe
48549 F20110114_AAAZVY sauer_b_Page_121.pro
1f00a7026683cd88957825a6f3d2ed94
be771f794151e6b03058e3e563efd60356fb1f35
48234 F20110114_AABAVP sauer_b_Page_038.pro
29aebc759f134a5407f5402816799ea8
3e97f3c1f64b84d24c970b4c4bfefba3d543bd78
43490 F20110114_AAAZXB sauer_b_Page_091.pro
33483fe7e0e332e5d3deb42a5c2e2f6d
07dcb1fde38eab562b569398d1dab5c10bc78e5e
2032 F20110114_AABBBK sauer_b_Page_063.txt
fae99d03fd541e837a4fa2e727b7c4ef
452930c91ebc7bedec73ff6fa6925a9b1435a5db
1765 F20110114_AABBAW sauer_b_Page_045.txt
3df6f3ab58d7c0de6f9493af1b539713
d4d480e466e38c11fbcd5d4bc7f5f046d8a58dcf
85749 F20110114_AAAZWN sauer_b_Page_113.QC.jpg
0c5ae28d08b030e515d635249943ba63
262e735cad1c75ba731b1310ed59c296f5775a90
67715 F20110114_AABAWE sauer_b_Page_057.pro
c26ba2f0be6da791acceb185ae8acf73
7d44c7c9ce0bc543debe50c361debf6fc32a599f
F20110114_AAAZVZ sauer_b_Page_037.tif
7bca73fd29cf1abb5a3f08ca7b2bbba7
1521d04bb489911614544723e6da549c27c1671a
45364 F20110114_AABAVQ sauer_b_Page_039.pro
03cc49e972327cacf714a2bcc92af826
e992e5d456e5ab6a56c6e97bd773b7ffcecba73b
66571 F20110114_AAAZXC sauer_b_Page_145.QC.jpg
1dd7a33501c987fc78f2cdcc81e4c785
ebd1ce20cbb6df54c6be5086ad30169f9c8be172
564 F20110114_AABBCA sauer_b_Page_082.txt
4228e9b3cd17edbdc59e41e3637eba83
4fa465fbcb0fa6f4b7d672bea1480b85ac2e8842
1867 F20110114_AABBBL sauer_b_Page_064.txt
8c3e449852316be1376ec4debb0d7b69
ae5d00d2767d01c21a052bdddcdba7fe4c6799ca
1939 F20110114_AABBAX sauer_b_Page_046.txt
797c01395e32ce64d7b60a1dc7bde7eb
03c0de984196a0ee75f772f054d60afa308bb320
14655 F20110114_AAAZWO sauer_b_Page_127.QC.jpg
10d1c5c3731444461b2542dd2236b661
ad183cf19ae982bc58d854ee626c4c0dca085e2d
60838 F20110114_AABAWF sauer_b_Page_058.pro
d645d9ed596b779ca8e4997d4e9c4225
7c2c3c6d93ef9f0c31c779db1b916c14f3c5bea2
45885 F20110114_AABAVR sauer_b_Page_040.pro
3a4b34dc8b6c4de37411d1ef13816570
9006e202e24bfb1b704cfed8f3a203fb532fc318
F20110114_AAAZXD sauer_b_Page_164.tif
231bca1d7ef0588642be32d55a80af6f
4625366721e9a08bf0210539cbf9ed65866c5e96
1766 F20110114_AABBCB sauer_b_Page_083.txt
23246295e424921c7222c7dd19f00f7f
d087316aa9ec3774c90469b30b4acc0bc205ec0d
1964 F20110114_AABBBM sauer_b_Page_065.txt
e086ee096e2352e0a72b9c9858c6cf0f
07551ed75ac5bf08d51dd9b9f9d980de1a2d6da8
1923 F20110114_AABBAY sauer_b_Page_047.txt
1384defbd7eea247a33478324fcaa23f
b16009227d04987548b3847c895aa897168ba3d3
67461 F20110114_AAAZWP sauer_b_Page_011.QC.jpg
a1134cd9c1c33a5ba670c8f3d55a6e6f
4e053d7bf5033638752fe7d24cb2a33f364720a5
43570 F20110114_AABAWG sauer_b_Page_059.pro
ff4ce577e11a62ed964eb088f50e53a8
aff790c4b68b8b3ac4ee356f08d9e94ddd75c55c
45753 F20110114_AABAVS sauer_b_Page_041.pro
b4618709120cd44d271874d980fc3ed7
380094dc0e79841ff1db2237ad1f6f11a193f9a6
F20110114_AAAZXE sauer_b_Page_113.tif
922e1b1e697f64ee24ed923325d879c3
59f72386e39ce934cfa4daff766b17f02762a7aa
1044 F20110114_AABBCC sauer_b_Page_084.txt
bc8aa53cece70949e3617e74076dc382
15085ecd60fc21c2d5309324d2946710bb7d3bda
1145 F20110114_AABBBN sauer_b_Page_066.txt
dd5e191d7b98578935bc6f0e2be3ea16
fa111343573cdb37a826b3abcb9932df1332df90
1943 F20110114_AABBAZ sauer_b_Page_049.txt
56eef118a83f5633a0b8844da5546c7a
dd4835ab6f3ea1bdb99c6a0933a776b7ea7e3dd1
1857 F20110114_AAAZWQ sauer_b_Page_051.txt
75a36436e12ae8862bc3368b6247e736
e1fd84e9a1ec01e303a8c6d625269a75f8c7a388
43930 F20110114_AABAWH sauer_b_Page_060.pro
559096dad3e43c97de078b414a7dbbf1
a786f54b3a014cfebf4007cc8b886876f41de3b0
65505 F20110114_AABAVT sauer_b_Page_042.pro
5d826aceb61b41a2507ff3bc539b92dc
0c00f93e4808f7ad9f8bba2d1220bc88b7ad7332
197538 F20110114_AAAZXF sauer_b_Page_123.jpg
863ee32feba987ae8aa5d75e91a660da
81aa4ad26e05aca892f26f49b69883315caaf6f5
2149 F20110114_AABBCD sauer_b_Page_086.txt
67a3ea8d3d48d5f4a2e2072d22a5a63f
514ce7902aff05e5743c7c65440e2db40b767159
1660 F20110114_AABBBO sauer_b_Page_067.txt
e7982ce1261a1edd4060285b5a5c8cba
74c8402138d77ea9e4ddc61c52d6793536640dd9
167171 F20110114_AAAZWR sauer_b_Page_130.jpg
0ee9ba6c1c5320f3fb0ff45de84eabf1
497bbe7765f83d0a07d30c300cd41691090d8c89
13319 F20110114_AABAWI sauer_b_Page_061.pro
aced1cfb9728c878c16a5222502f7883
5e1564c53c2bb1aad405564a96e61bbb6888af61
49409 F20110114_AABAVU sauer_b_Page_043.pro
c98b3612dfe90d54e2177564331e994b
228419e8809602a2ebb48ea1f6ef6e2ffd4ad8ba
84741 F20110114_AAAZXG sauer_b_Page_112.QC.jpg
0e4a7c6f92711e51139d63ee6c98801f
28debd2414dc1d0de2076f1f923028b0b25db70a
1725 F20110114_AABBCE sauer_b_Page_087.txt
bf430d18678e29ad4c3c4f0f1617e92e
4080149fc2b114d1372f08a007af819e297fa9d2
1962 F20110114_AABBBP sauer_b_Page_068.txt
38f9cc91aa127170a4d40a60223cdc12
334d689ea33f633e9cdce6b3f8170d00a3728be9
14003 F20110114_AAAZWS sauer_b_Page_029thm.jpg
f463e160389482a975e4ba696ecf440c
746cbf20d3e84c5c71148a569676815e0734df75
56329 F20110114_AABAWJ sauer_b_Page_062.pro
28b7598a5a58896602525c5eaba0e87e
1cfd13909281c999eae94b3fbc058e989a86c958
46249 F20110114_AABAVV sauer_b_Page_044.pro
4a6c40abd9c1fbecf56f69692fdbe7ed
6b89e915a7c968752bf60201a9ebd4c11e0c6bdd
3536 F20110114_AABBCF sauer_b_Page_090.txt
d3c85013b824eae588a9faee40fad63a
a6a5829a21dca5d2a76aa4f1c499b5862496a021
F20110114_AABBBQ sauer_b_Page_070.txt
2b6cc77f823abd53dfc7f11ad0320897
ad9bc3ff0758032bb20758262bcf2740f959689b
726591 F20110114_AAAZWT sauer_b_Page_143.jp2
8e637db491ef78bd71a107b048e35408
2bb2cf559172092aac91a1401363e10f36ace399
49826 F20110114_AABAWK sauer_b_Page_065.pro
3f579726b18bea4f32c0b33a254222f6
5ddf9c0d17b480d9cf00ad69239ddaf0f1216f45
44351 F20110114_AABAVW sauer_b_Page_045.pro
6b249f27f38c46cbeda5dd048cd1d0f9
ff995695410c6d17728ee9a920cd747fdfba4626
24376 F20110114_AAAZXH sauer_b_Page_010thm.jpg
2d3c452429337583c25d8564dd326413
42f1ca5a343d9bacd7f93f1e75123956d2a28964
2051 F20110114_AABBCG sauer_b_Page_091.txt
44f78a3744ca0a1f1f31230370ae531c
ee109235469a4ce6438dc16b6d23bc86ec4b7dc1
1681 F20110114_AABBBR sauer_b_Page_071.txt
19c98d547edcf16f10e6fb4e9b6ba27e
e65e7a69ea1b5aeeab17b068c4eee12c98ad6d70
82836 F20110114_AAAZWU sauer_b_Page_097.QC.jpg
fc4d1400f2d21a780d652927860bc14e
af644441a3ae71912f4f6f6a0840bc374bb4e07b
24016 F20110114_AABAWL sauer_b_Page_066.pro
b56fc3ec62dfed5d1c9afd4f68b3a64a
8e127a6fca2bf96c0cb6c1cd12dd65283c79cecb
48953 F20110114_AABAVX sauer_b_Page_046.pro
8f59a284c0eb073af4e7cecec98ddb77
6f5d8b38e109256e2b6dbe771ed114b136ac8874
F20110114_AAAZXI sauer_b_Page_094.tif
0992ad26bf0dee7976cc9011b5025790
8bd04e5a4d5c464691e57a0dd9f8858c57433a44
43613 F20110114_AABAXA sauer_b_Page_087.pro
0b08adb7f69e681c68238f0a6bebd123
08da6295c9c19b47ac826029afaff4f92de28dd4
1544 F20110114_AABBCH sauer_b_Page_092.txt
7a76738b7ddbeb2684967d50d625b7a9
3463d9e854655047dba13d7cecfdb0bb475ccc6b
2026 F20110114_AABBBS sauer_b_Page_073.txt
21faed2ca45b3724d107eec60672eee5
0933bd6e58865702e93f93323f3cba6796db2376
47476 F20110114_AABAVY sauer_b_Page_047.pro
09a53d5f46e9888cb48d711bcc13bb08
57f4bfe6c0120e274351df061d4d320fed240d34
32163 F20110114_AAAZXJ sauer_b_Page_048thm.jpg
d95d973e630408ac73f56967db81c239
edb3bf180e7f3869aa494571500ed6731e12e3d3
440 F20110114_AAAZWV sauer_b_Page_001.txt
a5f971c0d7ca1b0090c8a8d88ab391ef
ac5a53c25a5db2226778c74ad36138560c71ebde
38016 F20110114_AABAWM sauer_b_Page_067.pro
af53bcb8e5b72b2c10345899cbeafb11
df3429067bf058ba3c3a40920e2d89f65ad43b0c
975 F20110114_AABBCI sauer_b_Page_093.txt
fc968de348e7661b381637858f64df92
ace955b4cc54d24b2fd372a54d04563609544f77
1876 F20110114_AABBBT sauer_b_Page_074.txt
225190029092578912cb0f52c4e658b1
1b6a01cfb3fc507b8cb4fc444848e60ee76e43e1
49268 F20110114_AABAVZ sauer_b_Page_048.pro
9582e14a380fe158d7422f627d94ef23
99d0569b8511ad4d860a2085d453b1b31cd17bb6
1353 F20110114_AAAZXK sauer_b_Page_126.txt
db4de7d03d57a46d74f6a91324348faf
b0c9a7579a559852af3cc0c0d18b4f17da267ecd
32566 F20110114_AABAXB sauer_b_Page_088.pro
ef5457eeb9494127a1d5bf78a04e1490
459bc7013b073385ae423df40c1536cb8153d493
27303 F20110114_AAAZWW sauer_b_Page_004thm.jpg
b68d839967afc53ca0153df2543857d3
2bf27d01102edbf785532b073097341cfb9d8921
49948 F20110114_AABAWN sauer_b_Page_068.pro
75278d9422b375026d8b4cd89f2ad76a
b0bfed2967e22c40e214a7dba7580e9b4cdd8ff8
349 F20110114_AABBCJ sauer_b_Page_094.txt
c4903a3ca3bc26ff2e8f7381aed49234
0c8f16fe1cd0b54a7c77ff2bc71033998b1bcbf9
1720 F20110114_AABBBU sauer_b_Page_075.txt
b64d173db11da4ce9e37b0902004d747
a4b5f970ad5225317b53352347bef77e71c935c6
255780 F20110114_AAAZYA sauer_b_Page_006.jpg
06d1b0d9419b2697cc2f83f92e8c4e33
5d45951028881425320afa6f0f12a137afbd3ab4
F20110114_AAAZXL sauer_b_Page_038.tif
16558704ec4957a2519f1df0e5b988f8
4011b978913d09c3d5613717c4a46c674f54e62a
43745 F20110114_AABAXC sauer_b_Page_089.pro
a09cfa689fec8a943ffe2f34a31628c9
ff1008f2da47aba0f33c1c6955aa51fb24db89a5
189868 F20110114_AAAZWX sauer_b_Page_136.jpg
7035d9336ffb831c8039aa4848902d69
0569c4ed55a30511273ba302a0660ec8597bac5e
44908 F20110114_AABAWO sauer_b_Page_070.pro
fbd4718e751bd2f767d6b8bcd95be5a0
8e5ce7ebce2280295f17c7ef48a69fb1ab2254b9
1296 F20110114_AABBCK sauer_b_Page_095.txt
8421244820e593de5a073272e8620df1
4eb08b8ec83e472b30df23b386c4d8884caa1319
2070 F20110114_AABBBV sauer_b_Page_076.txt
33f2f3ebcd93856f6c954d252ba24fee
bc548616d6a3e10141dace4e8ccfd1f7612d763c
F20110114_AAAZYB sauer_b_Page_031.tif
769c0da8b5e8f483baf34f9752cc7eb6
447e48abd51a598058dfb141cc6e8b7fd3d40862
1031499 F20110114_AAAZXM sauer_b_Page_110.jp2
4acc085d55ec8f1fa238210396806f38
8466b1c33fc08654eef7828d953e99ddf6a09755
35389 F20110114_AABAXD sauer_b_Page_092.pro
abc1267096d8eba5503e07578aa14db0
87e98cf9120492873d483a724442c86b65228e25
42113 F20110114_AAAZWY sauer_b_Page_083.pro
47bc5d94a3308198877c6b4f2ba7939e
c7f387263ec8e73622083419be6c2a718ad91517
36119 F20110114_AABAWP sauer_b_Page_072.pro
0fd32155e54134560417d5eb61f5d42c
de8dcd3c45ca1fe94d1a240756b391e2cb420cbe
1789 F20110114_AABBBW sauer_b_Page_077.txt
fa4e571eccc604864b1c1cf03d198e1c
f89ad1ac5775873aabdc197dc339ab30fdb252ed
F20110114_AAAZYC sauer_b_Page_093.tif
8f5cd77c343700223d1b7f9889c27377
998a346e99c38a301b405ef8a174f13b68ef025f
94023 F20110114_AAAZXN sauer_b_Page_169.QC.jpg
ac0765c498b992afb9ea2782f7881d39
ebff39f544fe6b3228fc7e3c9a1e0908b30189bd
22009 F20110114_AABAXE sauer_b_Page_093.pro
4a12f526f8f6a3cc229a86c0d952e208
c52e511e852591f1079a29978cffbed391b6f628
F20110114_AAAZWZ sauer_b_Page_056.tif
496a3f6b4f91851d2fb2b8a6c06ad72d
17dbf173be1d3f709d2ab270cf0711872449b4ad
48947 F20110114_AABAWQ sauer_b_Page_073.pro
a162ccbdd25ccded0c150e452da5e9ae
1697e7311121dbe15dad27b69ad5ef11eab4fc1d
1977 F20110114_AABBDA sauer_b_Page_120.txt
8a31cb0168ebe546ce0efdce2d5c4156
8945a2ff0187ca147f2c86bc1b1bf0774b4cf9db
2715 F20110114_AABBCL sauer_b_Page_098.txt
bd718dd17d2983aec66eb3fbbbade920
9e762bdf1327afc683e6ad7356856977969bbc41
1716 F20110114_AABBBX sauer_b_Page_079.txt
9d8a4a5fa3e0d7297bbe386e0bf44f74
9bbc2040db761676ff22a636a8db4b93ff237451
24536 F20110114_AAAZYD sauer_b_Page_029.QC.jpg
586a9af69b5c8cba1af5767163361001
e0f0b611f5a2e15dc85d16b6737c5f40fa4c2769
24968 F20110114_AAAZXO sauer_b_Page_093thm.jpg
ddc823e4baba40e66f03e3de58cb7fb4
9e0f57cb7011a2f7bb1e5174a30b1d68fe415bec
7273 F20110114_AABAXF sauer_b_Page_094.pro
c36a6c4891f159476d1c6d5854202565
29bf3899f27505c46c21d30ccc79478d7e283934
46602 F20110114_AABAWR sauer_b_Page_074.pro
20ef3aafd820776cc3fb6b112747721f
0a55bfb692fc6c311f69c341d0811d24000f1344
1947 F20110114_AABBDB sauer_b_Page_121.txt
5709706df84bfbe9c62a2840ba02110d
561872837670f34eba5b375f24ab1bde75ea9650
1904 F20110114_AABBCM sauer_b_Page_100.txt
277c13059f7e39f38da9a0bcbe844733
62a7363c9005f26b75e674dda79d25d057b0eac9
1940 F20110114_AABBBY sauer_b_Page_080.txt
d5f19a8061c977b0fc27c4d46018f97c
49699249a7d29137450205bff83b77ab781f6847
F20110114_AAAZYE sauer_b_Page_125.tif
669383ff7e0c92df6803dc3637b25267
d990115df00980b0c476a824ce10c364f6ae22ad
109604 F20110114_AAAZXP sauer_b_Page_159.jpg
26b8829f7dcf1eada1716d779b308d11
9954db56dc02e91005902e96f72ff13040b3be77
48419 F20110114_AABAXG sauer_b_Page_096.pro
fb8386cf7d897ba2b77c42be87709a45
2f527c3a90c86ba3a8b82073770bbf55e668f8e9
42015 F20110114_AABAWS sauer_b_Page_075.pro
193a12bab0d6defcb5888140626d8a1d
6bec3a7522770560550ed3f58a0278ad77bb7008
1870 F20110114_AABBDC sauer_b_Page_122.txt
cec62076067cf057c47291dfd568b161
9921af55cd6ae5afc56e1aba8054e825fd27a127
2052 F20110114_AABBCN sauer_b_Page_101.txt
758259225fc9e980af2635b7a610bba2
2622633d915588836ad8b3b0cdefd5817f46b029
1679 F20110114_AABBBZ sauer_b_Page_081.txt
2e04a991aa4b336569875bb74bcfaff2
d75bbecd6d8e812b0f3a944ed1dbc48914358b9a
82159 F20110114_AAAZYF sauer_b_Page_054.QC.jpg
cb4898ec6c6f1ec93c151a4815c681d9
857265256c5c5d604182c193192bd05ccdf5b8d6
2289 F20110114_AAAZXQ sauer_b_Page_166.txt
7e7988371bc94e28c0baa028c370e64b
e713b0a35fc9f8103896fee7fcd5c1a46613a27f
48876 F20110114_AABAXH sauer_b_Page_097.pro
286474f2717daa30324a3930d5f4d28b
d36c990a55b7a77d1bde7a05ca1351add33a0ec3
45052 F20110114_AABAWT sauer_b_Page_077.pro
d27573023e0734da0075bc07e789c8ad
650168eaa7a0fbcce990cd1792af57d8c6ce8212
88051 F20110114_AABAAA sauer_b_Page_170.QC.jpg
ca4b12d03e525295f6de17beef359875
5a6d22df4cce3b70f3fe4deeed0d02d2b70744cc
1981 F20110114_AABBDD sauer_b_Page_124.txt
2e3501ea19cf4444dfcb6dd2c7edada2
5c508a87bd7b691f472325a996f852315f0488c9
1960 F20110114_AABBCO sauer_b_Page_103.txt
b4c9bc9bfbc44ea2ec8bf522e6c30206
86661c8b09724ba9c486a52658c574562eb96ae8
53206 F20110114_AAAZYG sauer_b_Page_153.QC.jpg
2759069dd5f9d7796d5e4ea0d24274b6
9867ea4b4ecbf85571a1b71cd1ef91bb70dd93da
52629 F20110114_AAAZXR sauer_b_Page_076.pro
57f2b85f4e4e3c70072acadd3f7affbe
d15c48a5564f56bd6266a2d8c9f2733e66586e59
60249 F20110114_AABAXI sauer_b_Page_098.pro
5dd3e9b8efc73e82d70f9dae6a4b1a09
c0bd26a8bd2bf146ff4b6e08f6b7fa03ef7261d3
50218 F20110114_AABAWU sauer_b_Page_078.pro
d264fed7014bfdbdb49f0838a5c9578c
ac31dd4e03ce37d8b310bee8a63e6cb3ca7c3435
49806 F20110114_AABAAB sauer_b_Page_116.pro
88875f217417598ea021d6bc9c318fef
188876df334f15c798bfc4bf7379a4b0e2ccc320
2031 F20110114_AABBDE sauer_b_Page_125.txt
e1291b4f28e1eb46bb29ea9aaf0f37fc
cabe5067f6b315b6e2902b7a124156db53a9a2be
2033 F20110114_AABBCP sauer_b_Page_104.txt
3889cf9d215cfb9568cdefeb17feeb2e
6b36ba12869dbd417766126ce9c0ac3c8cc16652
44682 F20110114_AAAZYH sauer_b_Page_051.pro
e1b4fc75d96a92946a389b0be91a300d
0efb2010d7912401e9e47114143f5860b8343685
36874 F20110114_AAAZXS sauer_b_Page_071.pro
1ae869030a58ec27bb5d5000ae3c45ae
3b422855c0fbcb81b62a4cf0884e2b8d1042e3e9
43759 F20110114_AABAXJ sauer_b_Page_099.pro
adaa51c8c0decee8105a9e3c82374d15
d12825952c75256375bb6c145d886f7d92ea65e9
36911 F20110114_AABAWV sauer_b_Page_081.pro
40f97c445cef484098bd11d28df19b32
49adee6e204ff2d609d541443b639152e281df89
84236 F20110114_AABAAC sauer_b_Page_073.QC.jpg
b14c8c003bc8d0469bff07832aefe5a4
9a24e9232970bde0c69b0c330138f82ef0f47da6
88 F20110114_AABBDF sauer_b_Page_127.txt
d0bab0676ea604f8c2a966cb036c7147
84d118bc6ffcebb8cc1a84fbe81a4869d7e1a6fb
1738 F20110114_AABBCQ sauer_b_Page_106.txt
df7473d0ef4f15d59e687fcc287c0859
cd32917ef484703477de9321216a1e515ca14407
F20110114_AAAZXT sauer_b_Page_027.tif
0307b0978f68140da766b1f50034c454
20dde5d656286dd16c4f1985ef07a1efd11264e8
47545 F20110114_AABAXK sauer_b_Page_100.pro
23faa5a7617cb041837d2846c1e65371
e8ed010b073d9323ed4d04571a43483fa9efa940
11112 F20110114_AABAWW sauer_b_Page_082.pro
cea75b4779036bf7e04d5ce0a93fce7d
c78b2326cd2fcb8266e0dd2d9fb0723e06b73056
25070 F20110114_AABAAD sauer_b_Page_137thm.jpg
b009ada7828eb241c2a0f2bfa64d3def
cc053e5ed41ff16929a22ace1c209601990bbcc8
5392 F20110114_AABBDG sauer_b_Page_128.txt
847a44dbeb671a4cf5307d30af147e1e
541e546370059b2b242912e25cb550c42be9cb5d
447 F20110114_AABBCR sauer_b_Page_107.txt
6287060340dcb43f531e49995c09ca89
d55f0a8d545c36f4adbb1aac91a436c1606dac88
44982 F20110114_AAAZYI sauer_b_Page_015.pro
b7ef07a7a8e0365418e125507515df9c
41fa27c1aadec54de7fabb8bf885cc32f5567af7
49232 F20110114_AABAYA sauer_b_Page_120.pro
03e7ac56f22f562f4a515bf3821149c2
7ec327388c784c2777dcef25fc007751663fcb76
46591 F20110114_AAAZXU sauer_b_Page_114.pro
3c57ab83dc684e31864773a1794369a7
95a1d61d3d5fe6391abcbe98ac97b8a86769be8e
42148 F20110114_AABAXL sauer_b_Page_101.pro
204fdd1e3fc8bc6e73e23f6c46d9ebb7
1c7e56fa69ae92b3d5279cefa9d5bcb7dd97f18c
21164 F20110114_AABAWX sauer_b_Page_084.pro
7ffbc613f3c778261a3398938b01059e
d1a95189144e5ac327ff5dbef92e4dec61f0c2f9
1112 F20110114_AABAAE sauer_b_Page_131.txt
1af106a82f3b441b7a4864ed2c0756eb
4a6e15831a262257608dccd203ef6aa7413bee13
100 F20110114_AABBDH sauer_b_Page_129.txt
c147d0096ee3510780d0c87b389479d0
34ec9352c23f24d32181058e449d8597267ac241
3657 F20110114_AABBCS sauer_b_Page_108.txt
10342befd87d9fffabbc1733dcec357d
7523656cd38dcf89c173f0a002093dbf6a5f744a
2003 F20110114_AAAZYJ sauer_b_Page_044.txt
8a7f56acebf9cfd6bd9b52a232aef93b
ec3cd720a34ade719ac89d754cc31394b5ab018c
46571 F20110114_AABAYB sauer_b_Page_122.pro
d2570ca570c9e1fb9d0329508e128bc5
ae07198225c983f8af7089b4efbf55cacc817fa8
963327 F20110114_AAAZXV sauer_b_Page_044.jp2
fb2948190ea50327df8e40cd5967df89
7c794cdee5812d8965460ced9b6b81be3a5c168f
82447 F20110114_AABAXM sauer_b_Page_102.pro
7c5ab218bf1218dcad5c5c9eec57b3a1
df02d3291436ea8c1df10ae36b1a9c58a57939e0
18591 F20110114_AABAWY sauer_b_Page_085.pro
9e6bb50731ce2e440d2dceecef5bc816
5f1930fde9cf20fc020808ee1f119def5fa0617f
53188 F20110114_AABAAF sauer_b_Page_108.QC.jpg
9bf95a54a259b7e2114da7dfb44b54b3
670bda523c6f63fa74b8df03a65af56dc681850e
4367 F20110114_AABBDI sauer_b_Page_130.txt
837269b524a5e05d1fa712671e468499
5835f9ea9abd6675705617bacb613b2d0e7e0fe3
1711 F20110114_AABBCT sauer_b_Page_109.txt
88e3bce1c181c9eb0d1dbeab1dfca404
ba6c9d5eb80be0af93d0b5a33a98dcb6f2efe275
31314 F20110114_AAAZYK sauer_b_Page_043thm.jpg
f5046eabbc7ea2681409dfad64aac21d
a8df9b7e43e3f34b096bd19accf938aa9558cdf0
F20110114_AAAZXW sauer_b_Page_015.tif
b451e8b624b2ed44976afae3ed990117
0e3f34a512ad3ec342f4a3b07c6c30ea19cf6754
49432 F20110114_AABAXN sauer_b_Page_103.pro
fe6642d651f1e4680668c4686969640a
1561204c40779588053529305e59ac9b6df12b00
50778 F20110114_AABAWZ sauer_b_Page_086.pro
e5e393fc9ff58c3626bc9bc3ddfea0e0
22d731b01d1d4ee2daf57b0e2d8078f83d6420bc
37889 F20110114_AABAAG sauer_b_Page_131.QC.jpg
9b1ed3e76250de8de1770830dd998482
bb18c5cf774d6a57baab9ee5ef13288a8e805a00
1596 F20110114_AABBDJ sauer_b_Page_132.txt
eb4a41807b5dd2623d4cc7a0f11b4e43
a0af3922351a0c451150c4bc0b69a4521e34183a
F20110114_AABBCU sauer_b_Page_110.txt
028d593fc967c455562ed28252e42a63
4a25df9e50d964b682c7fe1fe55f206f83f21819
2049 F20110114_AAAZYL sauer_b_Page_078.txt
d2807677ea91175ccc1d5fde0dc89aad
b64a5471a55fd85c5cf7b1352c43c458d741bbfc
45018 F20110114_AABAYC sauer_b_Page_123.pro
fde7fc321c216f1637745d5927331090
733b73de4cbbfcaacdcd8ce895151adee0299be1
201440 F20110114_AAAZXX sauer_b_Page_077.jpg
23bd9921f8323e16cc4d440a648fb1e6
80c694cdb914ab55ffda139e54a2d7b271ac59be
48719 F20110114_AABAXO sauer_b_Page_104.pro
a0c938396cbfead6d8c423f7e0ed7cc2
ff40f5a37ac7a284d851141491ee3650f3b3fb42
29518 F20110114_AABAAH sauer_b_Page_075thm.jpg
a0c9ccf920fab4ed7f985cd6e8753d37
0f1eea700d6ea7aa6e5eb08cf3140d6f6e828267
F20110114_AAAZZA sauer_b_Page_145.tif
dea99537a1f3107ee1aae89a8a647030
a2a357afc7e26dcca93457f7ea137f8a0d78ec5c
1683 F20110114_AABBDK sauer_b_Page_133.txt
b5c4fb3fecaf5160ad388c7cc2be6235
0513c87e3bf860aa0ad73bb83b4100f4fd93c898
1982 F20110114_AABBCV sauer_b_Page_112.txt
ef94a7249482c7d55da32efb77a147c0
678010c9f258a2afae4050774da0a9e8727e5312
197006 F20110114_AAAZYM sauer_b_Page_045.jpg
67a26ff7e2827326c90c7117760da8fd
e8499ddd26e2aa8fb8eced2bdd2322731a21876a
51740 F20110114_AABAYD sauer_b_Page_125.pro
c51ba405840c872866a2164caca77a66
e0dc89d6282a3b4b0f01e3d6d574801a290fee7b
203492 F20110114_AAAZXY sauer_b_Page_057.jpg
23861b1e694ed78339b349aa16253aa0
055bf5a379be6554d0816f1caa5606995d4eb7e6
52825 F20110114_AABAXP sauer_b_Page_105.pro
1f65e5b713e0f2f281e4e08555fa1dc7
34c598bccfb7cadded7369839362eaeb0f584d42
941944 F20110114_AABAAI sauer_b_Page_091.jp2
7eb26094419043e683a07c5a9b9dac6f
d7635624c4ff2b5952a02f8828222abe41bd8d91
201723 F20110114_AAAZZB sauer_b_Page_018.jpg
44bc21827618e34a180f8f412e27d4d8
998be4cdc01b0c6d31d24ead2c562dd0f153e669
2011 F20110114_AABBDL sauer_b_Page_135.txt
4309a2e6068a6273ecc82c3016d225f6
449115d4cf00988f5a71202b16f3aa3117f626be
1950 F20110114_AABBCW sauer_b_Page_113.txt
ba3bc475a7b3ea48f5faebc5f05140d1
df7e0b58474c8692071e3b955e5a86b7882eec07
1979 F20110114_AAAZYN sauer_b_Page_097.txt
52f9737cdf50c668b48b3decaf27dd45
67a3ec870af49fe0df25ec7cad62b00f6ef928d0
33702 F20110114_AABAYE sauer_b_Page_126.pro
fca2cd87d299f476ae2a9255741d7f12
f06b181959f71e0f8cd13bdfb30a13fd37c3211d
82256 F20110114_AAAZXZ sauer_b_Page_078.QC.jpg
b843719d524210c3be6123acbf0c6252
7db402b8be4893ca585cf5a14732b7de6b669639
41717 F20110114_AABAXQ sauer_b_Page_106.pro
326cbf5407518ece344b4ee44263a989
bcdbf1ae193ce5acd5ef5fcaaebc96ff2e7d89fe
76050 F20110114_AABAAJ sauer_b_Page_091.QC.jpg
e24a54eb4a9df26a847e46363f5f132a
9abcc06015ca86e9a3d488263df1ef7e6f76b135
22863 F20110114_AAAZZC sauer_b_Page_159.pro
e2b1f838bfde133f6b29a8439ec637e7
53a6b978b8176e62d37932e8e97274e1ab714484
1740 F20110114_AABBEA sauer_b_Page_154.txt
6b3e3aa3ba631b60d53a263c1a61120d
d75efc50ab3acfd0597e760954fa50ebde785a94
1884 F20110114_AABBCX sauer_b_Page_115.txt
437eb90dcc2ef0f4a2f80f5577ece04e
12ef4899ac6f5d81121e1e2246f3d91d6ce071af
74349 F20110114_AABAYF sauer_b_Page_128.pro
27ae64ab7f21477e792ac07ce580051b
fc258cd7d2b9ee70a4edc603d43fbba6a47553c5
1812 F20110114_AAAZYO sauer_b_Page_123.txt
751d02953d368b35b8b0b866cc2f8e2d
62fb0139fe5f5a4c2f75bf47ff6f753a67a5ea3b
70459 F20110114_AABAXR sauer_b_Page_108.pro
57975c27ac0210572308bf575cf05cdd
8108d18779d3bf77af49f3ce8b1176e381780503
38880 F20110114_AAAZZD sauer_b_Page_017.pro
0be0884d51f3b7471ac5cc0fb7e05eba
0b18b3f3cf75d5c90ff230eeb502baf68f0ac956
1802 F20110114_AABBEB sauer_b_Page_155.txt
c03f82f88ca5421704e460a8a72a9b9a
a9c974e78b4ba962147d84e578d5d17e9d2aa481
1443 F20110114_AABBDM sauer_b_Page_138.txt
2852af6296fbd0442d1bc1959f5b0689
0e10cbca13c32658b49c9e7c5961551a708f76c0
F20110114_AABBCY sauer_b_Page_118.txt
04aaa177cc44dc6dc9b605a3fcfb2708
6af00b4ef81807695ec1a7293090ed4eadae2a50
79131 F20110114_AABAYG sauer_b_Page_130.pro
9ea6a5892bf81452f1531f9c706ea30d
05f064ba5893337213bd56d5234e363d23d6bb14
4391 F20110114_AAAZYP sauer_b_Page_102.txt
6b6a8af52ba309a68ebf2251b70a31b5
a3a72e2239ddd847e8630f3c0b414dce0daf231d
40008 F20110114_AABAXS sauer_b_Page_109.pro
6b979386a3ac57662a299746f7e57224
48163c3cc41f79c721c6e872d1041cf50e14cfb8
28609 F20110114_AABAAK sauer_b_Page_095thm.jpg
eee68ac4d1a6613f55deb7784b7e5f16
1300c48c4bbb6d6edc47a53834499b944415c873
73351 F20110114_AAAZZE sauer_b_Page_081.QC.jpg
d76002789434dc071ed0b5fd12ea7df7
7c9198757d395534c807dc264b84426f9c63dece
1954 F20110114_AABBEC sauer_b_Page_156.txt
0307c333aa1c5ecf704309d69bbb6ec4
d28177692dfae711202c7e2d35779e5b5518ab68
1156 F20110114_AABBDN sauer_b_Page_139.txt
b78f3b035e2a3f0df9d8f3a45fab8dec
47a47b2c020a6acb31c354a5379472e01d37083b
1925 F20110114_AABBCZ sauer_b_Page_119.txt
3d6cc4e540fc713af4c9d907dab24d9c
249c59bc12a55d2bdd1d4e2007e70c8f9f97ea41
26075 F20110114_AABAYH sauer_b_Page_131.pro
8bfbe681ce8bee9b1dc22b0a9f351691
f1f9fde23f03065361f4c317fcfba4f852354ba1
178701 F20110114_AAAZYQ sauer_b_Page_137.jpg
71af90aa06ef6f60d42c31548b82788a
af79c184746a3cb191d9ae91a98578736614c0d5
46466 F20110114_AABAXT sauer_b_Page_110.pro
3d238b1691d7dd8416b62257e3ab6a08
880703296a06f42aecc84a3697dc4e738d89e636
83416 F20110114_AABABA sauer_b_Page_031.QC.jpg
add8361dff27681f2824e90070ee0f27
25b0324343dda7f793e14a9b92ce647fb672af08
24074 F20110114_AABAAL sauer_b_Page_133thm.jpg
e141b1fa462ca0fa349ccba8a1947928
2b396787a8360f2f969b863a3a6d214ef7af854f
81081 F20110114_AAAZZF sauer_b_Page_118.QC.jpg
456bfb9b3365f1483704a7e9d8104c58
e5d25a27a250e811989eef23c215fcd38d97fe6d
2281 F20110114_AABBED sauer_b_Page_157.txt
2b2c7212d4dc3ba1be382c88ee48e6de
ab8b13c4a7a123dc4278abfb4774fa87a3f60d4e
556 F20110114_AABBDO sauer_b_Page_140.txt
6d054b9e25653d3a3cada15251963a3c
b2084dac2422a5e76a4b1cd1aa419181d5b93e54
36130 F20110114_AABAYI sauer_b_Page_132.pro
103f2290148f31e5c823ff8e49b85196
9589a4b08dfbb02137828010b425d11f32e32b6d
3166 F20110114_AAAZYR sauer_b_Page_007.txt
dbf7aad518f1d7308bac993ea2b87c9c
5dbe9ae56135cdffa1300cefe4a926b220acef94
51387 F20110114_AABAXU sauer_b_Page_111.pro
7464c49b2aa88c52c1f6531a329d9365
5cb5c95254a07252708b3792d57abd17372d33c5
30789 F20110114_AABABB sauer_b_Page_157thm.jpg
6c3e60fb42d5633a69d43a918476a81b
2138db424c232608ee1fe08d9589a554e0be5a59
153426 F20110114_AABAAM sauer_b_Page_126.jpg
8c8255f8450764be08fadfbd438c4afb
afe43f355a998f45bf6c91c68d366360d9e0360c
27005 F20110114_AAAZZG sauer_b_Page_001.QC.jpg
c6f78e9136edeaf02b5089295a6e2cfc
647fae324f271669626ff050f3b96890414f10e8
765 F20110114_AABBEE sauer_b_Page_158.txt
fa823c819e16d03f8b091ae39aa8aabe
a620fd3cd6536bdba141764396a696a71a8bf72b
2830 F20110114_AABBDP sauer_b_Page_141.txt
2e5353451f4013e4b3609681d5ebbf2a
18704f00ece6c11e0d5df88319a1cdc0f1ab92fb
38650 F20110114_AABAYJ sauer_b_Page_133.pro
827bb4d9423d042a21c61b507946dc3a
5e7d755ad012ff5bc9ac8adaa2d850460ca69cc0
51608 F20110114_AAAZYS sauer_b_Page_063.pro
874f34db16735e60d0e90cda583c0cb6
dae3d5e5591b6ac04a36ad7aeab8f2153af2f288
50332 F20110114_AABAXV sauer_b_Page_112.pro
693c960da3a6bdc7af8bd021767d59cb
9dc7bbd5b0123ab6bea9368e24f22175b61ff03b
1003420 F20110114_AABABC sauer_b_Page_123.jp2
fbdf9add1d66caf7c143dfb01fbe7b1f
fb0a748a389780b2a1c34f7a28f3217a9d5ba66b
747463 F20110114_AABAAN sauer_b_Page_101.jp2
7018c90744e647a9562703bf60c4227c
ca49b3e783fdb6800f2d92fce6837db791469731
212655 F20110114_AAAZZH sauer_b_Page_049.jpg
2b99f3518e7d6b1408204641c7036000
7ff59c67e207cfa88ac20ec25044f80d35854e6e
930 F20110114_AABBEF sauer_b_Page_159.txt
7f9dac7138b31c914273613dbefb1aa9
92fc50c0a1ff6ed2d1ec8e097933f59c48c8531f
605 F20110114_AABBDQ sauer_b_Page_142.txt
08ec439a25b69c6f1e4cda8c14ee0ea6
6605215aed216990fc79a896bf6f1b34bbb056c8
45273 F20110114_AABAYK sauer_b_Page_134.pro
3e25e423cf5e3d3b7e0c2c80b4d8445c
271337fa0dc280465bb0f8540cd7a59de99b7e39
122479 F20110114_AAAZYT sauer_b_Page_088.jpg
6fc0cace8304c46631510653b3e3e123
8b459367533401405fdf9f1c045073e405fa93c8
49491 F20110114_AABAXW sauer_b_Page_113.pro
738539d0920eb8a14a25ac26bd8534ac
7b54ed217a948ce14b5dfeecc55fa6fb1f38e2e7
41029 F20110114_AABABD sauer_b_Page_146.pro
9a71af81bb5fb2713fdc7ba8f33ab96e
f4d82800dc2d865f9dd56584b651534e3b2b503a
912570 F20110114_AABAAO sauer_b_Page_145.jp2
df7f93cf31116f411d57cef05ccda19c
79ff3d47e2432a20a213c06114625844848bae51
79841 F20110114_AAAZZI sauer_b_Page_035.QC.jpg
0693d6869d758a973077b5ff5468511f
80bc74085aef78057c386545118ae626991cc3e0
3685 F20110114_AABBEG sauer_b_Page_160.txt
d409803cbb453eb384b3a683daf51d71
572fec95076eba6d366962cf7131833a13559a6a
1527 F20110114_AABBDR sauer_b_Page_143.txt
62cd339fb6d135600c4bae70cea2b650
dd3cf8b357d9127f3445c0b2f7dde7114ea309b5
39537 F20110114_AABAZA sauer_b_Page_154.pro
ed855be90a0015944e0e2516c2596a0e
52c7e5b9a4c2c1c7215b3a18f6e7017759be11a1
47249 F20110114_AABAYL sauer_b_Page_135.pro
62abb056cb52027bb8a97be2a015dba9
746cc36e80266b22b10c4d3574ea8303f22d0ce1
53804 F20110114_AAAZYU sauer_b_Page_084.QC.jpg
e99a3eb65c05b823a53734c59880e8b5
ee23bdc115de9c4254c03a5996a6c62d50a9fe2d
46639 F20110114_AABAXX sauer_b_Page_115.pro
836bcd12dad22c05eb0fb248795b6da8
e098a68c733d0a470a2046f652900560f75f281f
F20110114_AABABE sauer_b_Page_163.tif
09bf0292f859142447e3e7dd32154907
8dab8597c740062b5ab19a5d42e41da4a212a889
F20110114_AABAAP sauer_b_Page_116.txt
89fc1efc7739ec028bc8cb6cc9a05726
672ab820c7aa8afe8ba9a051af49eb545d2bc43f
1617 F20110114_AABBEH sauer_b_Page_162.txt
0be2b822a2488f7b3ce89e1ee75ed0bc
f002ff81571bc6f4d4f953ab0a7f0c9dac242196
2021 F20110114_AABBDS sauer_b_Page_144.txt
8b263780d1ff2d5f7729781e308a0942
b330dd4e3f90742b6c90e146de99671cc89a9104
39043 F20110114_AABAZB sauer_b_Page_155.pro
37fd18ad06c52c3ec8ac4de58eaefa45
0f7970c351235840166660ae0dc0536e4c5505e0
43535 F20110114_AABAYM sauer_b_Page_136.pro
f7543d74bedb226c1bd651bfc0ed2927
60ff0649412e5744bee66c2fa56cecc0c4e06d57
81849 F20110114_AAAZYV sauer_b_Page_103.QC.jpg
23a6377e96d09b65826c58c12977a1ad
a779103658a82e7468d4ac4179e6e171a73edd0c
49573 F20110114_AABAXY sauer_b_Page_117.pro
d25c891b7b550a9628bb5d40a73ced8a
bf50e5cbf57fa55a097eee98856cd452d79dcb50
F20110114_AABABF sauer_b_Page_039.tif
9720c30a76caf3b2e90934de529eb26b
bec8d4685dd8e3c06f1e2c992512a6fc1df559d3
2020 F20110114_AABAAQ sauer_b_Page_111.txt
6053476c730474902df6eff699a7f56d
ac8f2f2c24a3c6ab10dde3532a6cb731e7d29867
29556 F20110114_AAAZZJ sauer_b_Page_059thm.jpg
637a18b26289e136779b01a47bc211a7
c1060ada7f864c17b8738d7bee2be20c0639c849
359 F20110114_AABBEI sauer_b_Page_164.txt
016158b03ed44087f8e10c83e56db3a0
e85c469da3cad1d1f0f17be79cc95177fad577ca
2082 F20110114_AABBDT sauer_b_Page_145.txt
9820ee341acc6191e8ada451cb667367
b882efb956802dd35c5901353b75430f2ee2aaec
19249 F20110114_AABAZC sauer_b_Page_158.pro
b776140d34af31bc2fedbffdc0ab0cc3
392ae4736f3443279677dd1a8feb323e273e6ade
42097 F20110114_AABAYN sauer_b_Page_137.pro
c2d3fd639d6cf4061734d7d762187d91
c0df99b3aaa4f59f059b254c271dff86e730519d
78122 F20110114_AAAZYW sauer_b_Page_059.QC.jpg
a602cd28f92db734ac50b03f6332f1f0
11d128c239d10e6e1499054d355469777ab414d6
44497 F20110114_AABAXZ sauer_b_Page_118.pro
2af101293a9a4dcdc74b9249261f1848
dd9223b38331aefbf7c9992f69714280b2a11a89
19656 F20110114_AABABG sauer_b_Page_003.jpg
7c3f14b16470611d7dc424c877cca09d
50284feb56e3e25c14156e8170f07f876f8b9c22
61591 F20110114_AABAAR sauer_b_Page_037.pro
336a707d2352fef42759b1a9220fc6dd
57a9abf9b37f0264dc76092b06d4e82ea27fb0e3
1049180 F20110114_AAAZZK sauer_b_Page_115.jp2
6385d89a8a5bde1757be6236e5fddd03
6d73ad119bd9ba075d57eb0ccb06cbcf969e6b4b
128 F20110114_AABBEJ sauer_b_Page_165.txt
813dae9ee3e530619536290119356c55
a92921ec379777b38917d365722c5c232bfd4b95
1895 F20110114_AABBDU sauer_b_Page_146.txt
ccdb5993818744d7fdd667e4d4e551d4
961eb15fe1b2d5ef7c979751f7a82b8196159060
26262 F20110114_AABAYO sauer_b_Page_139.pro
e358ddbc8088d668775aeadb042faae1
041cbdaaec635997b11cb58631a636439615cb16
32456 F20110114_AAAZYX sauer_b_Page_078thm.jpg
4ffa0c97bfb79cd957f8f437b6d84bdf
9f251ff83b6593249b3341fc6ff1322d900381e6
1051882 F20110114_AABABH sauer_b_Page_031.jp2
0db1e3d8144e5c8486425eca15aa2d59
004875f9de6992e262db0704e0542f14c2852866
25972 F20110114_AABAAS sauer_b_Page_147thm.jpg
4c9874e77a5f955eb3b6fa9ce0e68f1f
f444b6020fff75634e34fc138d75421708181747
1051984 F20110114_AAAZZL sauer_b_Page_037.jp2
ca3d2e77f6834ce0192e61ae3602b749
75a8180319731a115ea6a2c25fdf8d8622a758b5
2540 F20110114_AABBEK sauer_b_Page_167.txt
0a7eed446ce0b901660725ed00c858f6
044faabebc8764b598c7b4cc8fc24d3fe5a6ae72
1424 F20110114_AABBDV sauer_b_Page_148.txt
53e7049ec15ac276ebaaac27fbc4acf6
5a0f15f87e5ad76cacf42ce0a54e65d9f160bd57
23732 F20110114_AABAZD sauer_b_Page_161.pro
2d4e18c5d356b44199f4b66bd69c5440
8c1dfb2cd25ccbf3064ef0a780215f89d50a3263
70560 F20110114_AABAYP sauer_b_Page_141.pro
54905c6e63f11e75cdb67e6f4b0d539a
de07eb59fe6609e9875b9325e9a17daa1d35b2cd
623 F20110114_AAAZYY sauer_b_Page_003.pro
9a449b658516fbb4f9853b770cd9dfdd
661e340cee59d6cd23070776760bdcd980d2f6ec
48965 F20110114_AABABI sauer_b_Page_080.pro
15b1e090f22cce155661baf6f88f1b5c
1ba196b30f9a666c6e00f92d2afc01569736ce72
82020 F20110114_AABAAT sauer_b_Page_115.QC.jpg
72cc3bf3162e47cb306b1f82346c2601
bd76d4ffeca70965856fecfe75df5e9bfeea1529
1966 F20110114_AAAZZM sauer_b_Page_048.txt
9e57cacd80b433e35da77e6cf8215b49
41a334fc602ed8fd56b38a8e50e4eeb6d106ec0b
2412 F20110114_AABBEL sauer_b_Page_169.txt
aa69a98ad1fe1a49ab182808773d0996
7760e604173a67ce195174aa86bce448d5e10653
1390 F20110114_AABBDW sauer_b_Page_150.txt
88bd55d04f706890974950879a5f2cf6
2b9c3ae63a0645ef2049345b7812195415231f63
34805 F20110114_AABAZE sauer_b_Page_162.pro
49c6dea413838e7d74c0f348e523e193
b5a54cb9a1e23fe7c2acf19dfd8e7b92c1fda718
11972 F20110114_AABAYQ sauer_b_Page_142.pro
72f6ca067236ad5138058eaa51bc3563
e3a4b24abbf3d1bb17a45488cf4178e5659cf686
22800 F20110114_AAAZYZ sauer_b_Page_138thm.jpg
32a69810b7e5ea7c9b847626c044526e
5c8e2faa59e57ecb93c93216b5a36509c6cf8b64
32120 F20110114_AABABJ sauer_b_Page_035thm.jpg
008c17c732d66737bd427b5bcafbcddf
f5c071eafc60707a22e9266638c956a7c93f1ee0
232930 F20110114_AABAAU sauer_b_Page_058.jpg
9061dc6aa82b893ea156fc685bb4d7d6
4b2cf791b10a1d2f3b296c1d892125f2a5519d7e
F20110114_AAAZZN sauer_b_Page_124.tif
d8d326c7a7dd47d70add5051e5a9d78c
923f7981186da363f6e35da7a04bf223dee1221a
31786 F20110114_AABBFA sauer_b_Page_100thm.jpg
1cc19ac31396db895d8f841e24724719
e60986937dcf884943525528b9a780fabf90a35c
382 F20110114_AABBEM sauer_b_Page_171.txt
9da3124285f423d65e8c51b3d1b7a2a5
3cfdd5b54077eb90ee967ef58c2186e7b63a0d11
1426 F20110114_AABBDX sauer_b_Page_151.txt
0b4f487745df661fff1632f70b29df54
94640dae48cdb411ad9e8b3b59f9e944b64aa3b0
9181 F20110114_AABAZF sauer_b_Page_163.pro
16104ba914d51cdfb42e91eeaccfa9c5
1669958c4618e706bf3640ce836a8772afc7e08e
35825 F20110114_AABAYR sauer_b_Page_143.pro
ec3b566224289d890f882b92920cd54f
860cbc8f96dbbebe7cddb113278b7f92e582c42d
F20110114_AABABK sauer_b_Page_042.tif
f3a805ab2adccaed2ecd546faaaa4cb4
78f5fc9f1670ce4e3f9b55f34733e52fe2545204
79252 F20110114_AABAAV sauer_b_Page_090.pro
93df530490dec05be4f6894569a67591
5b1888352fb545fb8cc5db7d9e36b5de1be823fc
F20110114_AAAZZO sauer_b_Page_055.tif
bfa700553bfdc0c6318f51a342bc7389
be67f12cc64f483a8c3cc0a09be615d80f02eec7
75819 F20110114_AABBFB sauer_b_Page_141.QC.jpg
2daf37add7392ad488fb7ae811898722
f47ced7fe08bad9bff036fd3848be1a8d9ede54b
990 F20110114_AABBDY sauer_b_Page_152.txt
bc8c2095379376938ede67eea3ccef9f
3b45d18f042ef12851e876f0c1de910e362d878a
6694 F20110114_AABAZG sauer_b_Page_164.pro
06b9eefcd2c3520c3adb99f87c91718c
1a20004f0db46293fc588bd86cf9d4da6aaf7ce9
40250 F20110114_AABAYS sauer_b_Page_145.pro
94715d61677b85ae0f7093c976f11f77
c30ce3f7e95c3bb0c90c872ecc124dfc41b6ab71
83989 F20110114_AABAAW sauer_b_Page_048.QC.jpg
7ade8ed6bb1005e7ca5acc2d8cf58b07
b78f508cc042f51ea360064cb0355e49e7001ca0
143514 F20110114_AAAZZP sauer_b_Page_093.jpg
6656372df83847ae72c32e683c676b28
367c0d9b37d0ea856de078a59bab22e679f7f608
65604 F20110114_AABBFC sauer_b_Page_154.QC.jpg
4142d54a7a5d936e3f66f03d32c9bac2
54d2d33ed1d9f0a18f911e959f7f72cd46ed6fc0
616 F20110114_AABBEN sauer_b_Page_172.txt
a51c6b486f08faaf85fb95717e6a44d5
959fcb00bafb690f4f9c5b2f2759decb41b6bd4f
1237 F20110114_AABBDZ sauer_b_Page_153.txt
a7e54303b40ac3756c142548237a68ef
62adca95453bf772849d6cee8cfed24aec266b50
1342 F20110114_AABAZH sauer_b_Page_165.pro
79caac6437adb8c264733b92cd25da55
cb50df49c8109c24b5b7d5d31ba8aa1afa9f1eec
37604 F20110114_AABAYT sauer_b_Page_147.pro
468cde712e238a76bd8bc83e052b1a8f
c07a9008e0055aa1afd1b55bb53034a7106f2aa1
1814 F20110114_AABACA sauer_b_Page_137.txt
8bc7a5d5b288b5ae44bb0417b89fce1c
3591a0a719a2f1d6cbb15faa096c0cab58bed340
10103 F20110114_AABABL sauer_b_Page_107.pro
ab9605006f0ab085cfff64619fbdd204
d9b72279a68f54e59232dd8d212f121926f20c82
F20110114_AABAAX sauer_b_Page_146.tif
5d0d6aa28fea2aec7c97d1b438bc5238
9f40560e83d81602f9daef02a5a979774e8ce4d5
13480 F20110114_AAAZZQ sauer_b_Page_009thm.jpg
e744763c1e0def2a9222cb482525fe8f
3a73b45d92a8b8dc2e048ce014591abea525f02a
79586 F20110114_AABBFD sauer_b_Page_018.QC.jpg
0f5bcbd2c6ec8a92a7dbb25635eb221e
d76a1928db792068f718ad4d130cc3f2261ed20a
1196007 F20110114_AABBEO sauer_b.pdf
8d95f069043981049a8c480771884ffd
17e73e82a3b09b015cc73db742479b21a29384ed
55425 F20110114_AABAZI sauer_b_Page_166.pro
e137c1f21e37cabdce543a2857316cd2
6d4db26de8ed87cb237734397b0fb90b26528522
29708 F20110114_AABAYU sauer_b_Page_148.pro
7ba9b2f18859f67b5a1bc53925ffbbfd
500add3a8d45e772a72986cfcdd6c6e6b217c14a
32992 F20110114_AABACB sauer_b_Page_125thm.jpg
f3e3884343d6b0142635e4d4f1f6060b
07ee53697438d5bb445051e8c6b9aaeaa8bceb04
26983 F20110114_AABABM sauer_b_Page_136thm.jpg
1d169597ed761eb67743ca17f0bf54a6
a89a681b14d27bafec9460ee79f3ff5a9a779318
29769 F20110114_AABAAY sauer_b_Page_102thm.jpg
3aaba8a4af3faa36f0de33a6f74871f6
e72ee34bb91ab16d9c18fd04398dda30fbec9d2c
29528 F20110114_AAAZZR sauer_b_Page_166thm.jpg
2c4cc36d2374df652bd0df9d94de1d10
cb5097201d195e16eae5094feecad03eea4f1b39
31575 F20110114_AABBFE sauer_b_Page_120thm.jpg
dc0ce65a1db01adca1a70cb044df17d7
d279b3fc2b918d6afea992a5d7081892007e489a
20685 F20110114_AABBEP sauer_b_Page_061thm.jpg
9a7aff935a330a7ee32e96f713e23d57
d4052fc0505fcb41699c0b58f1b6e06a9ff0e9ea
61136 F20110114_AABAZJ sauer_b_Page_168.pro
e5a0b2b8725cf45eaf745eb638234921
398e42b5d421ebab22b102f0432c9525d7d98136
35466 F20110114_AABAYV sauer_b_Page_149.pro
c819dce183bdaed20aa8faf840d682e6
775b8db3f66758fcf260041446faaf7ca5bc99ec
30795 F20110114_AABACC sauer_b_Page_033thm.jpg
60eb2126d4618b209e7e9a3f410ef4e7
310fdbfbc2339611f82dcd9b4f46e4c005e273e9
32110 F20110114_AABABN sauer_b_Page_023thm.jpg
87e3cc2755a93a0371e4bdd5035f0f2b
24b1ce4647fe92732351f41849390f797bb334c0
F20110114_AABAAZ sauer_b_Page_134.tif
d0da9f3fb393d27e6edb6a7eeceb9b8a
667e1a6992b07baa80d49b9d357dc45484d11fac
32306 F20110114_AAAZZS sauer_b_Page_169thm.jpg
1cfa27e9afcdee37f7d40c5a4d875791
4b98d2db8ece521147e8d11752e5ec689a82991d
39322 F20110114_AABBFF sauer_b_Page_061.QC.jpg
f69a9c82926e98c8b438f9a19a704903
be7c0b512d49ee28b1f819398d49c77636ec2e12
82925 F20110114_AABBEQ sauer_b_Page_121.QC.jpg
f72e2dda84e2e67631b121358585a4c0
0097dd828075b4a4929dce0f579b9caebb75aaa8
58189 F20110114_AABAZK sauer_b_Page_169.pro
e2b31039be2c405c637b4c4dd2ffa0c5
95c19b7b0122160d5a27b9a4cdfa26f699973b3b
26150 F20110114_AABAYW sauer_b_Page_150.pro
7ae7f0ec3669fa4e9af17e4c0a1c7210
53461180fdad59b9477e2edc18f062bb78366c1c
F20110114_AABACD sauer_b_Page_087.tif
0acc70840e4e4ef1729c2099ac455802
259812e1429164fc01a7610d771b2e37bf07a02a
2329 F20110114_AABABO sauer_b_Page_170.txt
34ff001e8b3466ce5efe8f1d21794634
28f5c7e825033a22023ecc3581fef0807484f5c3
F20110114_AAAZZT sauer_b_Page_013.tif
fd1b8f5cec3cedfac32ec4345633b92e
8fc9fa06106db3779b1724511e7d5d1f53e7dbae
14384 F20110114_AABBFG sauer_b_Page_140thm.jpg
388f179aec703633172ffbe7af98a753
073e0ebf39b532574153340a91e01f414fdec22e
74162 F20110114_AABBER sauer_b_Page_044.QC.jpg
7e18db68fc46ecebe9a3f882141b9701
a42afb357170524a1f9a2b2132315ef9374d08f3
55928 F20110114_AABAZL sauer_b_Page_170.pro
09992e96321d8f6b7d3b6f5025bdecd8
98cdae7206f49e6782dddb25331fede16d319e1d
29516 F20110114_AABAYX sauer_b_Page_151.pro
8404c88292d0a591275fb8d86d47964d
9854e943a172bb8e22e24be12d3cc3b9b0806fe4
51260 F20110114_AABACE sauer_b_Page_144.pro
b370d80b83e54600cf9dc46baf0a4fc1
cb8a958065968ce4922868ad3bcac90861e3a7da
F20110114_AABABP sauer_b_Page_117.txt
58cba6bab7cc9736144900e8353ca9fd
06b9eaf1646ab59db30d8704e007e6a58ed0f77d
187318 F20110114_AAAZZU sauer_b_Page_056.jpg
f73c1e624434477c9c70bf4e8d084346
502cf34ce7f56508d7b08c30b443ac999cec0796
80038 F20110114_AABBFH sauer_b_Page_076.QC.jpg
d8b860fab939b5e5d9058f6061245490
1414776e89468c8dd24e76b89f4edd6b4979515f
29487 F20110114_AABBES sauer_b_Page_014thm.jpg
7b7c2deacef6fd30fdaa0cb7bb5b5403
631130dd5f7ecae5f985017113a617fde1d8de72
8233 F20110114_AABAZM sauer_b_Page_171.pro
0c1fd9a32b514c4ddf8bf528fdb30456
af71fb6dd9ebfa143c6abd0d6bc2df9407236164
21173 F20110114_AABAYY sauer_b_Page_152.pro
d1a90ad978b6063e0fe2284d46c188a0
0ccd26b701b610144740ee48adc187db2d9ab2ef
1967 F20110114_AABACF sauer_b_Page_034.txt
0e6cc255b7aeb0c956a2a7335770f490
139c902597ac1d1dc1017e01de91c1951e7b0846
330341 F20110114_AABABQ sauer_b_Page_172.jp2
f0911b26dacc86f58d960bec13034038
c499dbc0bb3b190f44ea9d0cf7cb8b9f1e39723a
77364 F20110114_AAAZZV sauer_b_Page_030.QC.jpg
0899be34ce3cb07ecfd6764a189ad209
3343e305fdee0a92ff5ae4357a2150e241c6c632
86486 F20110114_AABBFI sauer_b_Page_024.QC.jpg
c34fe3ed312db0f64d59954ba9ecb9ec
2d7f955d0e514a436aa2ab6ac6879351e14a6bca
84204 F20110114_AABBET sauer_b_Page_032.QC.jpg
00e337258c211797159bb7f60573ec93
399d059797d5fae192550086c3481f6783b6274f
14169 F20110114_AABAZN sauer_b_Page_172.pro
138ac90513f922baca195003ba4538f8
41289388f08f12f1929483615de34c6f199540e2
28944 F20110114_AABAYZ sauer_b_Page_153.pro
804f1e7055d1341f805d9aefac974a25
36fd6f71754c6fb6276faf151a8b2ff5a67bff58
F20110114_AABACG sauer_b_Page_072.tif
f8bd20175c25d4bb7b5eb1bf17ba83e4
f415a428195ad8d84986f175cde907c9f66b8496
48355 F20110114_AABABR sauer_b_Page_031.pro
3857ef3439899a814e93b73b63e52751
9510041d859991a812cfd5d246039620e8c06130
1051981 F20110114_AAAZZW sauer_b_Page_025.jp2
f91f6609e9e82b74bd4095741de9b28c
190bf88e10065304d6e97eb4725b70f93338bb62
30896 F20110114_AABBFJ sauer_b_Page_025thm.jpg
022287d8c83f96f68189bae97ed6706a
b327085b35bd90bcef8e9d3550da81392358f0ba
61022 F20110114_AABBEU sauer_b_Page_085.QC.jpg
8ef4666c375834976c071ddba485aecf
e6b51c97fa37e3ac63fd3e94ba8b3f48f7d81679
109 F20110114_AABAZO sauer_b_Page_002.txt
214ccf3168f2a5aeb5cae517b83a52c7
64904e64c2e2dd6d7663ab4a1d1077632acb276d
42185 F20110114_AABACH sauer_b_Page_164.QC.jpg
ec9d44cb9afe7ab5b7664dba89186db3
614731c3404b8227fcec63390da59f47fc207c8c
1852 F20110114_AABABS sauer_b_Page_041.txt
1eea6df6cdf4179603a169aba059127b
8bf3e84dcf55cc25b4a2eebacb0d6c2b8261045d
13473 F20110114_AAAZZX sauer_b_Page_171thm.jpg
7643cdcc1d78225a9c2c801f6d76555d
291ea6ad4c235a95c686c44e1523100619567fd2
76603 F20110114_AABBFK sauer_b_Page_083.QC.jpg
949f2a489c8c67af1b9d728dcc7283b5
e9894e6420826f4c304ca64c155ba8aede6da0cb
27274 F20110114_AABBEV sauer_b_Page_028thm.jpg
ef9eafdea6b51e26cdd818ad72262758
409071da6eb38e71041005f9cc03f1bf6f17d48f
82 F20110114_AABAZP sauer_b_Page_003.txt
c0f82858de8a262db7ff805ef75df187
6ae7301f65cb9a1997bb0c25fd0a47e57186d869
204136 F20110114_AABACI sauer_b_Page_074.jpg
eec4b55877a2b3a917dc5214ceba4b77
28a5f5872585b383fbd2c63fdb14c330d17d02de
F20110114_AABABT sauer_b_Page_025.tif
3db48f4749cbed147c9574a2e0a6b3bf
ca54fa8407179a948cf8683c261fc4bd29130c30
27570 F20110114_AAAZZY sauer_b_Page_158thm.jpg
2ed789acbcc9f79b09d099ec8d548646
d716834a49ce2eb0e6469a3bb3330d69449e6e10
15208 F20110114_AABBFL sauer_b_Page_165.QC.jpg
9b6a564a8cafd977b974e0d725834513
883528a0cdcf4d5cd587849617b2f273c9caaadb
68677 F20110114_AABBEW sauer_b_Page_134.QC.jpg
70c0489bd85d9fa6a6933242c4846ab7
8c36bad16a8713ece2cd6c4b2f2330579a4b0d04
1474 F20110114_AABAZQ sauer_b_Page_004.txt
9fb3fd6ebf32eba40360576ed4809210
166cc4f7751c82b4ed41557cc1af6bdaf82e07a8
F20110114_AABACJ sauer_b_Page_045.tif
4eb5499ef431dbf562652de4e76b662e
9927b7d4e9f455b549195e661ec8b2850a4d13e2
871 F20110114_AABABU sauer_b_Page_085.txt
8d3819ca9efc884ef35ea03a8cfe6a2e
b0b89986322b8de69514a140a475f9c2b55efa54
F20110114_AAAZZZ sauer_b_Page_076.tif
56a9f39531fab2663f60da8ca921d7d2
141b9ce0558239d34a8949b8c94df260da90ff6b
32051 F20110114_AABBGA sauer_b_Page_046thm.jpg
5f5519531fe87c4c77f72b9511f1d577
ffa504b269758102d89a0b41dfeaf6e48052bff0
82706 F20110114_AABBFM sauer_b_Page_080.QC.jpg
276905fd43eb85785da1a4081a144ed0
f1f97c1c1eb95c22e30915b48b28fd8ac72910c7
72839 F20110114_AABBEX sauer_b_Page_071.QC.jpg
389840d66d5b5e1770276aa866564ee2
f169b56d8839dcc6cfc537a5f843b65039d3cdf5
2807 F20110114_AABAZR sauer_b_Page_005.txt
fcc3da43adba83c17c7cd54e1f4e76c4
4808b2c9de8fe506e49dceb8c638947bcb2eecd9
819755 F20110114_AABACK sauer_b_Page_147.jp2
9dd8796ee4316b2b6df39902b059446e
63234be158de6622400c79041988c2f3253e3984
1033297 F20110114_AABABV sauer_b_Page_041.jp2
4ab05ac45f99b7affbab39b118c9ea2a
b8ecd5a489e038571749204e5db3964f52496760
25126 F20110114_AABBGB sauer_b_Page_140.QC.jpg
bb51234c01460e95ae45f89242b79d71
cebdc16a665bdb655e1351ba013b32a7760a21df
88175 F20110114_AABBFN sauer_b_Page_090.QC.jpg
e8dcbfd2d13eacd8880236bd73a652eb
62a8ab48393c3c6f90c95931af20292b2967faeb
26976 F20110114_AABBEY sauer_b_Page_106thm.jpg
f809bbf76eb5164c6309a252ca414f3b
3bee26d63ea2a4a23a47f75c50b8b6cdfb8b529d
4687 F20110114_AABAZS sauer_b_Page_006.txt
2f58f3eeae5f11bf452c95a1b57c9b36
cf5e13c6c39270de5ce35392d68061926d3929ed
1496 F20110114_AABACL sauer_b_Page_127.pro
7aaf10e6001e907c81add78c5273fd8a
baf3d857300f599aa18b2c117654df3de2e299df
41844 F20110114_AABABW sauer_b_Page_064.pro
2148b8fb2fe3b7a94546ad3b80bbcaa5
39654557d05fdab90dbdc93483b773f208f32c21
63787 F20110114_AABBGC sauer_b_Page_013.QC.jpg
cf28587d9dee14127ae319e53abc6fe2
e615697b0d87e424dc39f2335168b36da051bd07
24916 F20110114_AABBEZ sauer_b_Page_101thm.jpg
c46abc55832310482d946e898002636a
23566e34a1b90ce59d87220a2b43061e9c13765f
2190 F20110114_AABAZT sauer_b_Page_008.txt
88092c23aebba2974695c473766e739e
5c237a2b25baa6e69ccbe1cb1afe6235359cbb00
226514 F20110114_AABABX sauer_b_Page_141.jpg
65ef9d6cf3057b170d0ab34eae2469b7
ad15958893a19ef499af5181abc2e1ead0485e6e
1910 F20110114_AABADA sauer_b_Page_072.txt
c24e86daee07d3e585408c0a1416b8ed
f39f1c3265f330c90da548697485fb1b1e2f89a2
27162 F20110114_AABBGD sauer_b_Page_134thm.jpg
a2edb8380b2caeb88013457ba2f0dc0d
18704b05866bf43328e883c6a560752a879f18cc
29775 F20110114_AABBFO sauer_b_Page_037thm.jpg
35de0840f5b05781bb1a6e104424b796
e42b793094917b9c175195ca8969cac9f6ddc93b
420 F20110114_AABAZU sauer_b_Page_009.txt
c9556d8f4f471bc0b51c7c2f9554d6f2
8a07717f822b7bac595ce6ab3c30f9e20039d405
48189 F20110114_AABACM sauer_b_Page_009.jpg
898a86a11b20c6b813ba0d3c33bfe217
a024c383b8947018ed22725976934b2c7e641bab
F20110114_AABABY sauer_b_Page_104.tif
a87f5a38ff701b25e98d1d6748d47a32
7d2de38e5e1dc4f42576251793d8866c182782e9
535838 F20110114_AABADB sauer_b_Page_150.jp2
bce26f1f14452833a297fba20e6988dc
0742f13b0318f2be259191bc34bc9aaeaeab4f81
19658 F20110114_AABBGE sauer_b_Page_163thm.jpg
beb29d833ff044899ed45e37da824372
5734320e0fcb3a1a409789754e60e5a4eb27425d
32570 F20110114_AABBFP sauer_b_Page_167thm.jpg
56cac2949510e8ce72e33ee3bc691e1c
2db3495b75361cd3224beab25c7f5ffaf960e016
2182 F20110114_AABAZV sauer_b_Page_010.txt
998f89e58599b3f49f009a320d78df27
302e8ff8e083f03822fa3749cd47ca60b1590f93
30684 F20110114_AABACN sauer_b_Page_095.pro
77a7b2b595b16e781310a56117524a30
986057094591d1bc8b9ff4e3245e261fee157b73
42215 F20110114_AABABZ sauer_b_Page_079.pro
054ddc47c9ede96cd6309ad69d27b9a1
c979df42c3fe93e95cd61a50ab695a579af487ad
48694 F20110114_AABADC sauer_b_Page_119.pro
82dd6c6032953dd92b57d0f150f3c7bf
3e8acb60b45be6f5340e6e144ab3245b81504694
46125 F20110114_AABBGF sauer_b_Page_088.QC.jpg
2ce5bb195140edccbefb4542bc4c36cb
2331fe6aeaadebcd7ff55330fd3e5da3c681ab1d
83074 F20110114_AABBFQ sauer_b_Page_052.QC.jpg
808a11a4eca851e950cf9567fceb49a0
513f9a192d066f9955ffea4ad7d63175321365e9
1419 F20110114_AABAZW sauer_b_Page_012.txt
d1dd364617475c41a0f7463669801c85
2c4cea734bef17934f23ab49ab976c2aefd44c23
999 F20110114_AABACO sauer_b_Page_161.txt
7c356dda9ad5d9ce2d621d3abb1dad19
a2bad188386e2523aa771b8c7946df34088ffab0
1855 F20110114_AABADD sauer_b_Page_018.txt
735f66dbf32249ecb8407b90a5e2a4c0
a02d4dcb60d554da1fbd5eba80b4596de10262ce
57176 F20110114_AABBGG sauer_b_Page_162.QC.jpg
d5cd56c1e5becd4c3b524e9ebcb8a238
e0b3946dbfcb543d291c200f0172ab6929721824
79658 F20110114_AABBFR sauer_b_Page_057.QC.jpg
2cf21c174af8378a0e898282a1090a40
ac3122483331fc08a68a0fb465b87e06c7804793
1823 F20110114_AABAZX sauer_b_Page_015.txt
09628477fa9cd23701f7f71fde737a06
d1d1ab28490bedaa4b09c3bdcb5efeb497135501
33152 F20110114_AABACP sauer_b_Page_024thm.jpg
f61f2a23e3db8716c8210243a673f8de
61ebb3a2d77c3cc885b428e8611d60a3039ed42b
132280 F20110114_AABADE sauer_b_Page_148.jpg
02067824e8c37e2a568d3c59c7b43e7a
9eee908f907052967411e1ac1735a813daf87c29
31880 F20110114_AABBGH sauer_b_Page_080thm.jpg
12556b2e5ddfe0698930e7518dbed1b2
78926d3b441172c4d81c020ee00898f227175b88
31440 F20110114_AABBFS sauer_b_Page_041thm.jpg
215e359390cc4ca1294dae249741acd4
96b2e28875ab0e0563cb9d4118ce7bb72b8762b8
1791 F20110114_AABAZY sauer_b_Page_016.txt
18580b06477a07b45e72b92e1438f5fa
d5ab7d28b52ff2002bdfc8fea4b10a6a0c8c7be2
30100 F20110114_AABACQ sauer_b_Page_123thm.jpg
8ee5e3f3fad62117ef3e15096c7948fd
2fcf010885f9574c4b09a5891d9052e1ea53a57c
30885 F20110114_AABADF sauer_b_Page_060thm.jpg
c1fc56117d46b6aee37710c774885750
cf117be1f63970633f6a0a6a0c10198a59428ea3
30883 F20110114_AABBGI sauer_b_Page_087thm.jpg
620fd4ac7a0b0dfd1d80d8784c3d2325
15dd7b21df3ce26dab6ec626cfd3ea3c50dd9498
71515 F20110114_AABBFT sauer_b_Page_026.QC.jpg
97b3f597d67280f5a654e682601af8f1
231c266ae9f5a442120ee1ea31747af233cfedc4
1563 F20110114_AABAZZ sauer_b_Page_017.txt
12aa4edc71790edc62cb56c94376d578
048ac064b28aeb32801f96bf986be70a678a06c1
31524 F20110114_AABACR sauer_b_Page_074thm.jpg
c53856594fe225ae98c130862bdac5af
87910defbef071ae93a0b8dd339d80bd79a8ad6d
F20110114_AABADG sauer_b_Page_172.tif
a42e89b2c0708f62fa7da5638d82a58f
122447e32ff4cb2d27a9a23a1bea21f49c9cf56c
30469 F20110114_AABBGJ sauer_b_Page_039thm.jpg
93c22d2332715faacd7ac51848862f35
211717f8bd752d8de810fd89b14cb291e33535f5
50155 F20110114_AABBFU sauer_b_Page_150.QC.jpg
ec410d71910b2a5e9ff11c79a8d9a06a
63d7aa5bd4bd422fd0af3fe8d328a7b2129fa38f
1767 F20110114_AABACS sauer_b_Page_149.txt
f1b4206694119867068622dc231a8ad6
bc5a387ae6f0015e87bc6cad89b81837bd90b66a
26442 F20110114_AABADH sauer_b_Page_013thm.jpg
0b340cd0d7ba3358d16d232b77d04c3c
768150221232358ee491c4e787bade1c5d90e621
17068 F20110114_AABBGK sauer_b_Page_161thm.jpg
b905cf0dc87d397715044be47699a86e
e21d7c36c8f93ca339b4a9caad34ac0bb066a543
86409 F20110114_AABBFV sauer_b_Page_124.QC.jpg
4f022e408e58f3c3ff0cc5b7a542e59e
6bc0c23be79fa97da9a2f1b621e794674f6e2f04
31789 F20110114_AABACT sauer_b_Page_050.pro
61a67b561ca2a795368d7b742a14cfaa
c7af9fb2fdbe4827d6256787107292c4bc1d2cdc
193465 F20110114_AABADI sauer_b_Page_092.jpg
10928b8ad9fa658b76997684936ee01e
cf2927f544bbce560fa994ea916a37e0c840895e
24319 F20110114_AABBGL sauer_b_Page_132thm.jpg
201bbf3b2c6bbb807c219b3245e0bcdf
ed9781bb0ce8dc80d1c60bb66ea285ca3fa179ca
11273 F20110114_AABBFW sauer_b_Page_165thm.jpg
eabe831637317242484cfa14712740bc
b3896d9719a4c39b8559f15f7970166dbff80f6d
27310 F20110114_AABACU sauer_b_Page_011thm.jpg
e6ab39b2e50a802cc51cbfa2788be861
ab17e8f5f0fd12ce47b05b86888cee6c9c1ae6e9
90832 F20110114_AABADJ sauer_b_Page_125.QC.jpg
4378de8281c3564047a10a2a1be1a2bd
76e2f000efa67a5817e4e3397f402f643c864fc4
24331 F20110114_AABBHA sauer_b_Page_143thm.jpg
e2155949fb4820122b2ae6a3efa2eec1
da6c4597080df943c7d4e6b00972cafaeb26f362
31857 F20110114_AABBGM sauer_b_Page_090thm.jpg
67d1c6e337d8e80485debcba6c677b4c
e61ef84301d8fef7aa5f7043c4cf62c23ea9672b
15214 F20110114_AABBFX sauer_b_Page_129.QC.jpg
93c5671c51a8270c18e55e0c91cc4630
30f02d72977709b2e348e16e8349a85182e0d83f
2998 F20110114_AABACV sauer_b_Page_042.txt
c62b6372c95eab8e97d255ef5a5b0185
3a44b839e5a9bd0a06d18d03ca56a10acb5cb4ba
1549 F20110114_AABADK sauer_b_Page_013.txt
0b500a35e1f4e3206e96316dfe0dad16
690a9ebfafdad62c1fd599aed6f8ee5d642b0fc8
68827 F20110114_AABBHB sauer_b_Page_072.QC.jpg
6368ea238de810b4699c21252359ea32
7c2f6c6910ec379a34df3b351e082dc79d331faf
22483 F20110114_AABBGN sauer_b_Page_151thm.jpg
3d5d1733f46199d1c2d11c09fe985ca5
59d1c8c0cd347a6d9d1a5113263371b55eb760d8
F20110114_AABBFY sauer_b_Page_041.QC.jpg
272b7ca33aeeeec3554c2dab82d37220
174018293801dcaf05a2b33f18e89e0ca476f7ab
51078 F20110114_AABACW sauer_b_Page_053.pro
d86e922e924a225c70477ab689df18d0
176adcd31545663d08a887d03fd69c37dfefcce4
220556 F20110114_AABADL sauer_b_Page_125.jpg
329861b2248f1dbaebfb164dfcd9233f
ed816ecac50dd0ce5f750b118d96b3ed134858b4
22624 F20110114_AABBHC sauer_b_Page_005thm.jpg
a1aa33ce6729ed5878cb1e6b86d9f08e
021427434fbfcf11def52b8d0a2f22c120163da7
28811 F20110114_AABBGO sauer_b_Page_079thm.jpg
de8da86af633892f06aeb34881997d56
d54f2f2bc1e2cb160a63370cafdf8a8f1f0ee2c4
58989 F20110114_AABBFZ sauer_b_Page_050.QC.jpg
4e3d96c7d69245158a73f6a279362c60
93dc7c930fcd67baf65002799ff6859c34f1d9cf
48245 F20110114_AABAEA sauer_b_Page_159.QC.jpg
3dc8ef56eb1936f80d4328e8a4dd8e1b
e88db87ea801c5eff5b0f3282160e55ac0e0561b
1783 F20110114_AABADM sauer_b_Page_088.txt
c8609c439218f6593cd0179cd60cfce0
cb4bb37d28a75439aeb1b3f06520c4784a82f8ac
68394 F20110114_AABACX sauer_b_Page_004.QC.jpg
7e1bb40bba5cac647cd3e208a0d88b72
456ac210c359618c73416444dffef3ed4fec544e
78302 F20110114_AABBHD sauer_b_Page_087.QC.jpg
2c059c9d6648e1db361dafa4fe4ce402
66848310c03b684170d5802e96e799ccef05e833
48523 F20110114_AABAEB sauer_b_Page_052.pro
c28a9de62b52dcf3effafc3fc189cff0
65b0a02ae778dfce1b1005c09748b136ef182b62
100236 F20110114_AABACY sauer_b_Page_152.jpg
4676d04dc35039a8d2db1bcacf6dfa21
5e6a1a6d1ad0b6c7108e3933225abf8745c4ed9f
85948 F20110114_AABBHE sauer_b_Page_117.QC.jpg
f735949a0712d9de8539186d463f75f6
8755a92c7d59a13e111e32ff599963d8faf733d8
31910 F20110114_AABBGP sauer_b_Page_040thm.jpg
d2291bd030e0d075cd369a1ff616702c
0f979e5b0d98f4d29f9191b53974846422c1a1ae
61223 F20110114_AABAEC sauer_b_Page_167.pro
272a3030b110df36080add3015f09134
1f49ca1ada3475dbeb760a535a857ddcf7a885c7
1033194 F20110114_AABADN sauer_b_Page_038.jp2
ad1ddd76a61c604dd3a474408de892b4
0c1842818be4a87ba29341000a0e18fb0321c932
76205 F20110114_AABACZ sauer_b_Page_095.QC.jpg
a2e8796e5b4d1f081e06d03385d6665e
b9bf05d32d35a54c24b1e59d0b15b5fb7b33710e
82757 F20110114_AABBHF sauer_b_Page_074.QC.jpg
8e8ddd6708f87222503ee21a735ee299
f1f221b4395451270cc48561f9620fa0b85842c4
27583 F20110114_AABBGQ sauer_b_Page_055thm.jpg
bd923cef7caaed37e630a87cfd436d35
5a80556e77c553ab4ca0316a713de96f8abb4b42
29250 F20110114_AABAED sauer_b_Page_071thm.jpg
eb0ff7b7b53f417ecf01fb1aa10609b2
ad0803f7d37b7155b3b056f64d91746b1d7a7888
30575 F20110114_AABADO sauer_b_Page_016thm.jpg
5b3879f643ecb7f1ad1634a6fa13ef1b
d6f0a0ea4b4d8834560430d516047dba3497f511
31557 F20110114_AABBHG sauer_b_Page_097thm.jpg
c21e0318972f89d2c33e8bd68fc184ae
01ad8f0befeb9ddbb5e075dca26eead7bf203728
71050 F20110114_AABBGR sauer_b_Page_055.QC.jpg
92c02d84fbd3e1be2b4b8437f326a9f3
cb5d5cc146f08f3d6496b89d86ca00cb941a6500
2353 F20110114_AABAEE sauer_b_Page_105.txt
377442bee0a4a51a8d2d32262d4e195f
3304a33a7ea8c036a2300d3f8b9a4ea42f4543ce
F20110114_AABADP sauer_b_Page_095.tif
ea6c564399192237cf3a57cbaa42b88f
d4938295f7ef3fee5815220834adc3074b7ee1ed
31479 F20110114_AABBHH sauer_b_Page_119thm.jpg
750f64d770d683d055ba10724479c7b9
8b235e27d9f1b21217c8758370c94f4e717ea98d
31802 F20110114_AABBGS sauer_b_Page_058thm.jpg
ddb6b257c295dfebbcf62cc8d9ff6093
b90e4a4f071a2c0ed9f6e15fca049369e3252ca3
197179 F20110114_AABAEF UFE0006469_00001.mets FULL
3dc7760af3328a487af4576695076bad
9a2fb410ad64b045f31b8e0132cb8b3797285c9a
83315 F20110114_AABADQ sauer_b_Page_023.QC.jpg
e91d91c281e67158def4ca8de35ea8b7
40fc7fd8108911ea75e63548fa39d3703ab894ef
26199 F20110114_AABBHI sauer_b_Page_126thm.jpg
c32783002e203c9623c5e37f5a4c06fe
46642b97042b5341fc8db64e3cf7ead71db38a9f
30344 F20110114_AABBGT sauer_b_Page_038thm.jpg
0ec3c45e8b075eb0a5bf0c618b7ae000
1b424fbc4fe584576ff1ec2ee5d3eccd45d0fa40
1051952 F20110114_AABADR sauer_b_Page_068.jp2
41fd361e7a0fb8f26a0ba3b5ecb5323b
5e45e37128cbdbe2605154b84bc438feace9634a
72002 F20110114_AABBHJ sauer_b_Page_135.QC.jpg
b60b9285860af99c3add84fe00d4d690
4cb38aba9a16dc348611208ec12c47c0d20bc527
62965 F20110114_AABBGU sauer_b_Page_008.QC.jpg
d83c50a2dbbd61953ffba15411498536
cc40fb57f644c970be05eea2bfcebe1477828373
25362 F20110114_AABADS sauer_b_Page_165.jpg
512cfdf4ad77d9a796bc033dd705c68d
f5f980a6992fd1c9fe99f92a0a30394a7cdb3a3c
64131 F20110114_AABBHK sauer_b_Page_010.QC.jpg
42f17c38478b2c8350ab3a45a2ef420c
272e81fcf2edb812fee0ddd1fca2c32fe59efda7
54189 F20110114_AABBGV sauer_b_Page_128.QC.jpg
3dc7b00fecb9e06db808f369ec9ba6de
d997d12978969a2dcb4df6e9e0cc869263eb6e33
60322 F20110114_AABAEI sauer_b_Page_001.jpg
04762aae95c705d975f2fd0a5c24fa72
d42f0b12b29eb2f24205f197c574e5ed56cd4ffe
F20110114_AABADT sauer_b_Page_142.tif
b31fc0e9b24df88910bea453c309f069
1f70c4ee9e8a066d2b73b265ee60c768aef9076f
24110 F20110114_AABBHL sauer_b_Page_149thm.jpg
3eee0c2120ab64e3ba080dbb4889d67a
b0808875f2cb24f2fb1b03f27c59967cd57dd8de
74834 F20110114_AABBGW sauer_b_Page_104.QC.jpg
2c5b3cc907628040c63c1c123507c8a4
2efca0844a9f10a3aacef9e44183339154a32a3e
22150 F20110114_AABAEJ sauer_b_Page_002.jpg
49578177457e11b7aa419c4c0e862968
7763e23b57e0add37bbed694f323e648c6409e0a
F20110114_AABADU sauer_b_Page_114.txt
e575a9cc9dfe5cdf186655739b5744c1
f45c462c12b74ccb9d2388441f69a62ed8ad2d1c
70050 F20110114_AABBIA sauer_b_Page_007.QC.jpg
43399a5d8249d8474dc916bbc4bf09d1
b4b6b505b2e21e7c33d88008eccced04b98ce919
79863 F20110114_AABBHM sauer_b_Page_086.QC.jpg
784600415af008df18abf5a3b64b1cdc
61bf852e1242879690009c1c2699c4386eabf3f3
28938 F20110114_AABBGX sauer_b_Page_091thm.jpg
777b6d22c861a9d9d1d1e5a90a2d59e4
0033d18d8aefdd3e1cd104b46f856bca589086a9
162656 F20110114_AABAEK sauer_b_Page_004.jpg
903cb1c755ee215417555eaa179da7f5
4ed98332453016f63421516312895bee85165844
70855 F20110114_AABADV sauer_b_Page_109.QC.jpg
0865f214da2b50c07981bdcb21ae0b5f
dd95ee6fcc993e2820c07cbd131d900729dae74e
25706 F20110114_AABBIB sauer_b_Page_012thm.jpg
eb9739c27a729161506cb32f7112d9fa
ac17b4643963972bb9ea892d80faa20a2f311968
32341 F20110114_AABBHN sauer_b_Page_116thm.jpg
26859a7a8b3494c8d6d27d307e803dbc
578471626e1867e091d4860ba0f7035df00736ec
65819 F20110114_AABBGY sauer_b_Page_102.QC.jpg
35ef07044181c00ccc4abc5a5dce67b1
f7881af532a6b385d3c1d38cf65bf4a73c29db6a
159372 F20110114_AABAEL sauer_b_Page_005.jpg
5e83d4615d6ac108d0f65a977261a5a1
1ac6a08bbf868607bfe3e4ec2b241581c55a0d34
23742 F20110114_AABADW sauer_b_Page_171.QC.jpg
656f00b9f23c92928672bb25fdd531f7
71d5fc9ad54486a1bc38010b1b98dfd3e0ca43d0
77966 F20110114_AABBIC sauer_b_Page_014.QC.jpg
5b52cf299101deed59d0108a8e64be06
4e33db7478c6ddab907c74acf94681ff3d896748
66355 F20110114_AABBHO sauer_b_Page_146.QC.jpg
fecd5808108d1824897a7a03b7281cbd
baebea3a338af357fa2b7404ed640f7b32d7f662
22161 F20110114_AABBGZ sauer_b_Page_164thm.jpg
195f8869daf1b59a7555862b4c1180f3
c84e1da5ac958c4102dd36789ac12e62363a34f9
160950 F20110114_AABAEM sauer_b_Page_008.jpg
bc33e4e13ebbaaa160f2060f363c7a74
eed001eee455347e282d817fae01bd51c3274dca
67176 F20110114_AABADX sauer_b_Page_136.QC.jpg
3432dc7e918a9f54f73dd8c8d30d2742
a0942e78f8c8892303d10651cd314edc85e04d4a
216567 F20110114_AABAFA sauer_b_Page_024.jpg
c20655fb2260a9c5c9befa69a4ccbd08
3470721fab97f6085ef4ebc2611c3f12137b6917
30256 F20110114_AABBID sauer_b_Page_015thm.jpg
abd914442eabec6e66ab94c109fd70de
03513d5d8803ad9c4b1feb03f93c9446ec795d03
79851 F20110114_AABBHP sauer_b_Page_110.QC.jpg
8c63a461bb7eff4ec2ad16e09746aeb8
5aee8ad0af9f3941895bbed60f6feaefb2be76ec
160814 F20110114_AABAEN sauer_b_Page_010.jpg
fedd15cb6ccdaf6f446045bdb77229b4
7b63501c99db76f25c45eb8dd3bbdaf5b82e648f
32610 F20110114_AABADY sauer_b_Page_054thm.jpg
e7218ebd52467dc1859c2c5f449b9a9f
40b80747870deae531eb8030262e359366e5e0f5
217301 F20110114_AABAFB sauer_b_Page_025.jpg
8f4c079a434677de13d8d8abc8aae384
ba8a3a05bf07b85f363f68509fad10cf89a8f657
79098 F20110114_AABBIE sauer_b_Page_015.QC.jpg
81bc3a61e0f031363236c8f6f60ea7a2
e7cdb06309315e4dabe59c03b85add461f149a09
998458 F20110114_AABADZ sauer_b_Page_045.jp2
70bb92a3983aeacb7ade4f2eec6ad7fc
cd24b4a7f65fe82b621bc86579718ab6c6d066e0
176820 F20110114_AABAFC sauer_b_Page_026.jpg
cb3690565fd6fe8db5f916f7e6311938
1e7ec6581c6b86d28ed961cc004fe0f144e6c55b
80248 F20110114_AABBIF sauer_b_Page_016.QC.jpg
38231289a2bc6870bd1324cf3b44403a
042c34e2113014964f2580153428c5f8bc39e390
255672 F20110114_AABBHQ UFE0006469_00001.xml
9694b6f3766078d4b2e89b0e5e6d2431
10e464ec2d68df059198bae39e99cf0cb890189d
176065 F20110114_AABAEO sauer_b_Page_011.jpg
091c90a5fee01a0e623b62de46ccf331
cb3c15cd7f31f9c26e004f22452623c2bb5a541d
173631 F20110114_AABAFD sauer_b_Page_027.jpg
e1b0aa021b7455944ce6ee213fbb0314
5ef4d79fd926203820a9ab91b6419a457383c3d1
27098 F20110114_AABBIG sauer_b_Page_017thm.jpg
e6c47d6efa3ab4ffac6218bbbb08c66a
dff8e19a76d16502fd8cc8ff010aa7bcbc2f5678
14740 F20110114_AABBHR sauer_b_Page_001thm.jpg
3595942c886b3385633e482acabd5f23
5320183af705a965b1d5b1d04d3526d04bae9c2c
161361 F20110114_AABAEP sauer_b_Page_012.jpg
ef17caf385d3ad50bafe664996cdf982
ba1f88d90bbda064ae9cb05732483c81170ee9c9
168734 F20110114_AABAFE sauer_b_Page_028.jpg
58d8baa280224b3d40397da5fca5c250
5886761b4ae24974828d56c6c73524827fbb4c35
72616 F20110114_AABBIH sauer_b_Page_017.QC.jpg
ed7711f7c0a1853bafb7a29f2a37ca92
37a2329d307f42dae577a69f9f47b0ec3a3af5cd
10835 F20110114_AABBHS sauer_b_Page_002thm.jpg
eb7f6c5a26dafe6893c04e655188a69c
9aa3c6b4ba8a3147b4db485c7b6515b029abb211
160230 F20110114_AABAEQ sauer_b_Page_013.jpg
5a54b8faf1a452fa7a88847a7a777942
96f502bc82847d6cc1ea10e95bf2bbe29b10f927
49875 F20110114_AABAFF sauer_b_Page_029.jpg
85b7463699540b3f6f69d5023b13ffb3
9ff3d317d97eee84cdb37853f3fbd20f95d592fb
33101 F20110114_AABBII sauer_b_Page_019thm.jpg
3b0d60ba295b2625a5fea15eae2a238c
707ee2ce9f7e7fbf8bb14be2a01b96bc619f3a07
13455 F20110114_AABBHT sauer_b_Page_002.QC.jpg
2e41c02008c9f600ff0825f4e8febcd1
4b4c11ea87918cf99ac89b6198acf5437ec3501a
220137 F20110114_AABAER sauer_b_Page_014.jpg
9377411e61926e027846e2b270715f97
920f3a1565be6c042fda51a93044cfc27d493de1
196639 F20110114_AABAFG sauer_b_Page_030.jpg
a6aa4e9e0f97426fda9f0bf0f79951bf
e4d7cab0f331798e99f25f22e8b87c1e9867434c
87102 F20110114_AABBIJ sauer_b_Page_019.QC.jpg
3bc4f28d3334c96a073799bdf791f924
21c6916a4b928aeb87d91754872ef38162d38bb5
10326 F20110114_AABBHU sauer_b_Page_003thm.jpg
3c66b97273cc1e79d6989794c2097e15
5eae2c9cae6ade27891c206b83dec5aa30dcd402
197128 F20110114_AABAES sauer_b_Page_015.jpg
3ad3b20df9dfda0b2b096e3380b2f32f
374546339468cdf98ba076abc9b7bbc5debdf1ee
213955 F20110114_AABAFH sauer_b_Page_031.jpg
40f3a9a64ad02954101f9d5010245604
cdd6f417ea482993c86c88cdb4139cdee6014f52
31493 F20110114_AABBIK sauer_b_Page_020thm.jpg
6724008d5dcffa6e0bb8a87c5412f88a
020d702334ab7d740e6f6fbbc2077148b2897fe3
12179 F20110114_AABBHV sauer_b_Page_003.QC.jpg
0f54db2a7404123a09e209fd6bb8a35c
13a2f0caa1272cf194c15a764dee183de78429b6
196802 F20110114_AABAET sauer_b_Page_016.jpg
680dd7b4758ccff0993827a1be4ae7bb
cbf53894e211ca3fd220698fce779f61c6be14be
214473 F20110114_AABAFI sauer_b_Page_032.jpg
f64e3d666701f860ccd340f8f2bd6464
fa3f2fb58cdf724d35b922de98c7259d58c9bb95
85342 F20110114_AABBIL sauer_b_Page_020.QC.jpg
381cd2d2fc1f5ac734fd3e0674df55b5
8d16258b7eb9641d1b9f78afa17ac3f23de28fc4
57073 F20110114_AABBHW sauer_b_Page_005.QC.jpg
41cd94e21cd067ec8232a667132ef5a7
8134f354de1a131b64413652b3676ebb213d0ab7
173771 F20110114_AABAEU sauer_b_Page_017.jpg
4e141b2bc71985ae88c9f38d5f711877
ea074fb7a79b62dd1980266fd032d5a44885fa2e
208058 F20110114_AABAFJ sauer_b_Page_033.jpg
b472d43dcadb0d9be04a18f801ca800d
09b35fff3ae2ddf75761f5e8599992aded117f79
86963 F20110114_AABBJA sauer_b_Page_036.QC.jpg
7cc757b55968845839a5b60c50de767e
a898c22b59a174d852c7a51c6c214c89ff383f23
32145 F20110114_AABBIM sauer_b_Page_021thm.jpg
f2c1cfbb50695554685bb5647ffbc332
09e98f245fffd6b46c7cedcb88292771e1316f7e
28255 F20110114_AABBHX sauer_b_Page_006thm.jpg
c8c703c53197a573e5e6935c760fe5c4
0d5ccf6706bc7c16474a6b8cf1ffd9c7e92daa7a
222191 F20110114_AABAEV sauer_b_Page_019.jpg
c7d68040ba11c6754695e7e6e78b9e81
a8ac9e1388e4c09fc810353136feb29e4adcab44
217623 F20110114_AABAFK sauer_b_Page_034.jpg
5127d7d19476aa5a083fe9242da19996
63040916fc63289f0fb38b588db41f36f034a4ef
78544 F20110114_AABBJB sauer_b_Page_037.QC.jpg
6abeae0daf2a74df4e2af13d354865e3
ae9271e0418038a83055f037b76cb10799127603
84512 F20110114_AABBIN sauer_b_Page_021.QC.jpg
64863cdb1af203a2911d806a3d71d0dc
53ff4888777aa593e3ab15c314bdaf265aaf9a93
79947 F20110114_AABBHY sauer_b_Page_006.QC.jpg
e5481e8bc6a9732d516d1b32c49ad84d
a9bdb8b669c66c5730951c24476463e38bc52bb5
222364 F20110114_AABAEW sauer_b_Page_020.jpg
dd0751e689acbcbf5cb1631236ceeec1
5541bed2f498fe91485430716e8a6ab996e6bfea
205200 F20110114_AABAFL sauer_b_Page_035.jpg
b60f75735a63816b79ef0b5a3bfafc92
8cb7e1c136abe3e13080db7b8de785ee8dcfa201
82641 F20110114_AABBJC sauer_b_Page_038.QC.jpg
72612cca5dca2ea64b9edd45154c6562
36bd90d8d6a378c30ae04d812e39842e0a25e659
88378 F20110114_AABBIO sauer_b_Page_022.QC.jpg
396e24e9ee3268de27693f2b39901e66
4a9bbb31910a39d88bc20f9b528227c0cfd010bd
26930 F20110114_AABBHZ sauer_b_Page_007thm.jpg
201c8e984f768290c8282ca00ac2d9b3
95442e0ffc00d0c62a9f37d947d7e9995ee9b896
213680 F20110114_AABAEX sauer_b_Page_021.jpg
57d48340ad073c0e38d0a92f82ef7080
1ed64b9173dd8b7f3fd8756555f828968551d495
212986 F20110114_AABAGA sauer_b_Page_052.jpg
920622c3b5a4cbe3098929bfef22b552
ee8ac31cc695bd398e36d575ea06d404a69cca1f
220760 F20110114_AABAFM sauer_b_Page_036.jpg
5d26a81debfd7d2737159444cfa724ab
2420a649804eedf5e32c1558b06b26dd82ebcc00
79889 F20110114_AABBJD sauer_b_Page_039.QC.jpg
801dbe2850c4a9fb1b66dcdc153d1b05
e47ca92a99bb4732fdb25e8e6b4460d99885b663
82179 F20110114_AABBIP sauer_b_Page_025.QC.jpg
396cb92e8de1ad2b51df46ef313878c1
e67ad6f00838ea207c8a701a921339ec1a9efce6
224779 F20110114_AABAEY sauer_b_Page_022.jpg
a8399fc85fd7d395910bea937e229041
50a87a03a80d27e10d80c571b5641bffddbc8669
220808 F20110114_AABAGB sauer_b_Page_053.jpg
b7b4610052e129e0df6976912401722a
995482fa2d9f5cc4a41683413f95bff8690bdb97
214218 F20110114_AABAFN sauer_b_Page_037.jpg
9b00549b3868206a977a45523f9f6bba
b26a043b97d1a48d50ec1bb9d1a92beacd3eddd8
80443 F20110114_AABBJE sauer_b_Page_040.QC.jpg
73c99a074a8ea6d6c8da9bda3179fc25
38dcf2f3ff7fe5b7ddff2284341bda58f1e625e5
28470 F20110114_AABBIQ sauer_b_Page_027thm.jpg
78f82739921fd8171982f92dca47fcec
ccaf94e275018ac16f4f5371b94bf6ea206bea73
213330 F20110114_AABAEZ sauer_b_Page_023.jpg
98559f6892d8d7083377f9e90e445bf0
73656bc8352eac27c886341ea2d90d385ca92426
205726 F20110114_AABAGC sauer_b_Page_054.jpg
37988b64aafa6c3bbdb6f5b5ded913ee
8a3cda5de3ba25b750ad64ca06601908cf4a6d36
203106 F20110114_AABAFO sauer_b_Page_038.jpg
d5fc5a6ce3deaae59afef70c897f9382
ad36133fb10df1839f63bba46269ea66fb6930f8
30223 F20110114_AABBJF sauer_b_Page_042thm.jpg
f3429d073c1708ae58c7639fe5b68caf
ff23394be70b1d92032699ac083e3f8aa52840bf
177248 F20110114_AABAGD sauer_b_Page_055.jpg
e0815ac21ee3c7330af0441aada6d367
245ab18013bf91b30f51cbebb4c047a981da8853
77563 F20110114_AABBJG sauer_b_Page_042.QC.jpg
ccf81090dfdd18e9f36916d8b925aa6b
d721c80e107449f62969920998af726174c8586a
70231 F20110114_AABBIR sauer_b_Page_027.QC.jpg
104bbb65a56f02165f35dbe9018563e4
fcf195bd8dce3c40021ec4cff74e916489315a6c
189993 F20110114_AABAGE sauer_b_Page_059.jpg
39103757fadcf807274b4303ddf9d5aa
d9a4bd27f48b1464248b2e666b2e64d31a6c8fba
196814 F20110114_AABAFP sauer_b_Page_039.jpg
0e1db3da5e9571794ff91f5e2ed7d359
d27f01d2df3888b9a4bf71988016cbab0c74b135
82045 F20110114_AABBJH sauer_b_Page_043.QC.jpg
3b1301981f70ab7818138be776f9a9c8
9b9695bba3380efd20ab8144ce7ca51dfbd045cc
65436 F20110114_AABBIS sauer_b_Page_028.QC.jpg
d3f12cc85c2f2b4f8b9913ccc7881463
dfd6f6515cd851a638b44d79b9ae2a2c25fbe8bf
196020 F20110114_AABAGF sauer_b_Page_060.jpg
da07d7c0ed3e1ef7572bcde9cd47232b
401643a7ea82c5f4368a9b8b90378cd62cfcca94
203721 F20110114_AABAFQ sauer_b_Page_040.jpg
94a07c02570bbc2f81de472983b9ff5e
a2c2c0c461fc28fa40e14ccc5b5b2134071dc6cd
30904 F20110114_AABBJI sauer_b_Page_045thm.jpg
c66d81748eeec86bb9f2bb15641e2a07
777be54212df3f5b369acbdf0f66f77d49dd8d4e
30078 F20110114_AABBIT sauer_b_Page_030thm.jpg
3e276087fe881cb9349714e508fd49d6
f096cbe640cf7bd9739fba2f4910f86754d68cd0
95705 F20110114_AABAGG sauer_b_Page_061.jpg
c0582c66a6fefb7c843010a76ea67080
d9df0c7ba3df989f9743266c6abf0248ae33c749
202014 F20110114_AABAFR sauer_b_Page_041.jpg
96f3cc5df5bf4424181baccfce722564
00e9ca75e435f78178d1cd36ed4c572395996950
79710 F20110114_AABBJJ sauer_b_Page_045.QC.jpg
f23d634a076c4904a0f555ef0d2a0a07
fb10218873143a9e33983fe194d314ac923062a0
31901 F20110114_AABBIU sauer_b_Page_031thm.jpg
4085686ac49429e3763682d11f65a1c7
7b598e85095829bed05dd6489af502a055ae5af6
224437 F20110114_AABAGH sauer_b_Page_063.jpg
fc39aa3305d07839749f0b05a35b01cd
50fc4d85137f7e125cefc432f4f04657e84ddfed
218221 F20110114_AABAFS sauer_b_Page_042.jpg
e0a3f85437c1c70e63a70cac33767bce
57e0f283787915d3214429c7714797d8dc0ef304
83146 F20110114_AABBJK sauer_b_Page_046.QC.jpg
63b33d77e6b9a4588d40960c6b206749
69947009013be5d74840d95082ba9a9602e5bc1b
31610 F20110114_AABBIV sauer_b_Page_032thm.jpg
ce61d107d789f784ffd7a8bf3be433ec
68899dcfa20b8d4ff871c48c63314292ff183e5d
185756 F20110114_AABAGI sauer_b_Page_064.jpg
f358b02b8b0bd0e7994cfe936791a0c4
80eb8883284055a9b955f13d75b170ae61ecfea5
216663 F20110114_AABAFT sauer_b_Page_043.jpg
d14043d09583c5d1d33e4014d7aef1a5
2bee406f7239f49697e750b3bbf3b11569b36150
31394 F20110114_AABBJL sauer_b_Page_047thm.jpg
a263eb1ca5a72311a16adaf487fc1f87
af7b77458e621ec1251892da88b6c476e2d4082c
80779 F20110114_AABBIW sauer_b_Page_033.QC.jpg
0ce90ff6569736376848efb4948ab48d
8664d2a8ddbd30fb1f6c5d2a254d7d1710aa6d02
219068 F20110114_AABAGJ sauer_b_Page_065.jpg
77e863ac350162349c3ebbfc665a538c
89239e20a84bd9754931b0d54edad8dad3de7328
189794 F20110114_AABAFU sauer_b_Page_044.jpg
1aa40576c6ebabd28dbf5c6282abecec
32e896bbf085d8387954eabeefaba8d75d64d393
32285 F20110114_AABBKA sauer_b_Page_062thm.jpg
506d2afc9318710cd61c6d7aac4f723d
796f431e63b70f4943538effc36df2342b338c2f
82837 F20110114_AABBJM sauer_b_Page_047.QC.jpg
0ba6f62dc08bfb5a8712afde96ccf39a
a179ca986ce3541bed568592bb245e1a6cb5c509
31937 F20110114_AABBIX sauer_b_Page_034thm.jpg
fc730d637ab8941fb5fb9b40e12b0859
227467ef23074882f61022dc1d79882e6ff09d41
117537 F20110114_AABAGK sauer_b_Page_066.jpg
75ed66d8757b6ef8568bff0c842b0f01
b845a33cda6dabb576f873a36020446ce8f1e736
213061 F20110114_AABAFV sauer_b_Page_046.jpg
57010dc04334b10e18801ae3e17db087
b7f9f6dd5e17622d88abad62ee53a0ff5bfdf1ec
90820 F20110114_AABBKB sauer_b_Page_062.QC.jpg
18338e8cc2f059562774349e707ca6c7
39452d2808bebd9bc9e9c964984e43f0ade4bb3f
31948 F20110114_AABBJN sauer_b_Page_049thm.jpg
68888ecbef84b684aec75cfc1a37e97b
d11d846a3c13ea38e5d4fc6ba94656f88c9b902a
85395 F20110114_AABBIY sauer_b_Page_034.QC.jpg
8da4680b92a51cc196151e093bf9d977
9d9ff0a31898ebadeb1da85f4becd9536bc9d1f0
172772 F20110114_AABAGL sauer_b_Page_067.jpg
d12808d1ec7feaccc0d35540b5b3e920
5a2918cda657430823fe661ae70067b16d9d0baa
205768 F20110114_AABAFW sauer_b_Page_047.jpg
12dce5a76dca578f99d765089ef899fd
950403972d9e88ea13e166e7b6e5af84df20c39f
33055 F20110114_AABBKC sauer_b_Page_063thm.jpg
861df1b5bc9b3b4d53a36926f5c45a70
36a06884e87763f0914ccfd152feb177b54d4b73
85118 F20110114_AABBJO sauer_b_Page_049.QC.jpg
bbe021f087b67e70a7536df96f404d46
4670cdff64fd0fc99d0b9d9ec4de54f15519e40b
32690 F20110114_AABBIZ sauer_b_Page_036thm.jpg
a89fa5793e5ebec9b7bb5c95e1576d74
59fc791f7d061b8e074e61947feb381e39742091
197677 F20110114_AABAHA sauer_b_Page_087.jpg
97b386607cbf380e47ae365ae5374026
2486dcbff394bde21efa73dc67c9a6a395c24f8a
209820 F20110114_AABAGM sauer_b_Page_068.jpg
ca3f14102979c14d89bcd42363a00cb8
0d0c724c57d3ee36239fda7f626654a739d37fb8
209374 F20110114_AABAFX sauer_b_Page_048.jpg
2bbfa21fea72293c66ab506c0ebb5a15
3a35fc8bfff98856ca7282135e8dcfbc7adb1baa
29697 F20110114_AABBKD sauer_b_Page_064thm.jpg
cbd9378cdabc44835138f84812607054
a17b5ef513c900851dc585e21c3ea6c8ab2371c7
25106 F20110114_AABBJP sauer_b_Page_050thm.jpg
143f791616b42597fbadcec4db5b71e9
9fc7b730591598fab08524d2bd5f5c427329d70e
166623 F20110114_AABAHB sauer_b_Page_089.jpg
d5658b7871a4d733a41a4eb77aa911b1
cc954bdfd00da5bfae1828d6bf614976c0352bce
207931 F20110114_AABAGN sauer_b_Page_069.jpg
d5222b2825239f37f1afd5755ea8e3a9
f4ec4599daff8206c558bb5831aeb92843eb6537
143625 F20110114_AABAFY sauer_b_Page_050.jpg
56cadb7c0ca284af2e7243b9db7e4096
f534308f4ca8b4944b2c1608e328e539b9f9d860
72885 F20110114_AABBKE sauer_b_Page_064.QC.jpg
e9502f1eb29f5e0fd8adb97e7838350d
5718f2c9be5ce3705f08b3b6214b454e31549544
29554 F20110114_AABBJQ sauer_b_Page_051thm.jpg
e587df212aad680d8c1c0da9c900568f
c67230c64a5c2281865cbc609fadd19c98c6b8a9
251191 F20110114_AABAHC sauer_b_Page_090.jpg
c70860f12c34849c389820a67ec711a8
3495e389007eb7d1dae58a96761f4937dd77ec81
198158 F20110114_AABAGO sauer_b_Page_070.jpg
e66ef45cdec544c923ca9003879cfc69
0902be0197c0f857c12718327a317608d83a09f0
191697 F20110114_AABAFZ sauer_b_Page_051.jpg
bd1e9f67e888b87a9542c7a1a0fa74ea
4712a33f8c332f20a6f844bbe22f490b0babb56c
33332 F20110114_AABBKF sauer_b_Page_065thm.jpg
fdadff1a4fa41de2df132dac46d8dd2e
fdc6e78d304ef57881a5c906d4b779e8c357881f
75594 F20110114_AABBJR sauer_b_Page_051.QC.jpg
d98ded083b5a81d2387a2800926087d2
de800d2cb82e21541697779c6efc4cdf5cc01d6a
187161 F20110114_AABAHD sauer_b_Page_091.jpg
b4b1daf28223807f53c314b287b55a7f
30359bb410ab8937c8546d585eecd0c779f5ff05
168521 F20110114_AABAGP sauer_b_Page_071.jpg
efa14d681aa99a591abc53c9033edef9
00ddf523f61ba5e60098db07deecd8649ab9bf44
87291 F20110114_AABBKG sauer_b_Page_065.QC.jpg
181ded2b2744d28264c0cdb5e6b7134e
091b377f03c14ac11152c8195ceac80c4bfc2082
138697 F20110114_AABAHE sauer_b_Page_094.jpg
2ea5faf5f002e7c5c487e5c5b3b4d23c
0710bdbf513e229fbcc760059476ec3707d8ed39
22212 F20110114_AABBKH sauer_b_Page_066thm.jpg
7378869d72e550431764819afe3bb547
cc3680e32ca7a0a1cc4fe3c30e7ec265d0f94120
32293 F20110114_AABBJS sauer_b_Page_052thm.jpg
cd0be231e103f7391cc4faf9afa99332
3d839f00d0074a29f27cb8ad7008f1c16d493bc5
191933 F20110114_AABAHF sauer_b_Page_095.jpg
7c87e568217e70c7c79d6f5330be6c37
da1c51d66e597249fc0520992eb400953bf6df5f
167678 F20110114_AABAGQ sauer_b_Page_072.jpg
309449f63cff69793b2f25ac0ea214a0
1301865c32bd7f80c6eb4f22f01c68ef30499df6
49051 F20110114_AABBKI sauer_b_Page_066.QC.jpg
85abc5bc71097c5174d7302fb742fd38
e66ac4eac87ac3fd9a23faadf4aa76de9fd5d0b0
31923 F20110114_AABBJT sauer_b_Page_053thm.jpg
7261a83019c242481a01b623cb5191d0
1e53a0eafa1ea16911dca497fda8200bc33fa014
212328 F20110114_AABAHG sauer_b_Page_096.jpg
9de254ffce098e051edbeb213bd9688d
11f5af9e996df66bd6b96931a827da9613a92d66
214594 F20110114_AABAGR sauer_b_Page_073.jpg
51db5dce6a965302098901b758b6284f
00dc17c31a9259856ae9711f7ae43fe9073800c0
28892 F20110114_AABBKJ sauer_b_Page_067thm.jpg
6a40e9c1c278646f6a06c448cdee0fd6
d55abd48ba8e55fa76574ca36efe12ed677d4820
85175 F20110114_AABBJU sauer_b_Page_053.QC.jpg
6f78956fae096f1d7817e82cbe9307d0
80727ce6bc65f9aa08d624fd90fecf9634171280
214077 F20110114_AABAHH sauer_b_Page_097.jpg
cbb1c095860d32e78cdd7dcacec08464
d1318c18c253fb3486b2212224f401099ee47ddf
187946 F20110114_AABAGS sauer_b_Page_075.jpg
98690ceedfacf105216e653d21c43427
40fac01fe87106b16984673395e4eaefc6a3e0eb
68246 F20110114_AABBKK sauer_b_Page_067.QC.jpg
c846283a5b84df8af3e58930a9fd2d60
3faa20362883496391ca7c0a0bf9b94599e920d4
29246 F20110114_AABBJV sauer_b_Page_056thm.jpg
e3bb49b7dc09f4ddb00451e7de444154
a7cbf49d124bcc43e1accd8f91c074a88af52b18
234563 F20110114_AABAHI sauer_b_Page_098.jpg
706d84484bb3d06fe6a9021973f3b020
d9adefa6899c9ce63a146ac2566acb695374aa01
214269 F20110114_AABAGT sauer_b_Page_076.jpg
61f8b2a456563acebc977df8a90b83f0
0bb4ba9c5fce600dd821f12110e687f38f03ae1d
30627 F20110114_AABBKL sauer_b_Page_068thm.jpg
d53025255114a5d3b0e745b72d105329
8434f817171243908aa3e4735d667566b05f8b94
70100 F20110114_AABBJW sauer_b_Page_056.QC.jpg
e98a7e9c54773e93be6f516c4d61a6aa
b0f3ce66663423810f572dcc9310e9b9290ce1b2
188181 F20110114_AABAHJ sauer_b_Page_099.jpg
ad23516d0e1caec1fb8ef3b8ca3eedc1
5d4216cf03295df52155ff9411fae0c1b351ff9b
189653 F20110114_AABAGU sauer_b_Page_079.jpg
74bb316967942793811b4be57b1ffc37
a6aab8cdcdecef9fa5fc58b0ecc3c20f4c92d55f
28761 F20110114_AABBLA sauer_b_Page_083thm.jpg
34a88bc358058638e549e26bf6941fbd
93db853ac7d52e29a4096d9b4d44dd1bf413356c
82381 F20110114_AABBKM sauer_b_Page_068.QC.jpg
435919c0142a5fb252606fb47fa98368
1ddd69fc970019d517e3daca9376253f5b973703
31132 F20110114_AABBJX sauer_b_Page_057thm.jpg
4288c14b3f6cfaf064640ede779a4607
3315d462f95ad3bcba3fbc680d8f1cd479a94488
205934 F20110114_AABAHK sauer_b_Page_100.jpg
11db6d68b82b993a21145f125ebfd376
2db4929dab1e674289d4f4acd26706aa69e94edd
211210 F20110114_AABAGV sauer_b_Page_080.jpg
944f57baf8eeba588d0a502771015adb
997a2359781579ed1a416dacda4724241953d429
24662 F20110114_AABBLB sauer_b_Page_084thm.jpg
a524e406af975bae20155cab35795ad8
8f8a99f613ece6f2cd96798e3e1ea67164d4c96f
31666 F20110114_AABBKN sauer_b_Page_069thm.jpg
155931e52df8b05672e7047da164fa49
9876192a0b0f13f15e45ab1c03b122fd72a83704
85709 F20110114_AABBJY sauer_b_Page_058.QC.jpg
ac0b00aed7990de51e998f58c7a30b74
ad9c3adb80127779e5518a26495f96029644f218
158295 F20110114_AABAHL sauer_b_Page_101.jpg
312ee9933a1ee2d5c561af6c0f8b2f25
3fbd0fbcf09c6dbf8f2f98467c63657145042571
64923 F20110114_AABAGW sauer_b_Page_082.jpg
948d41570b8fe19cd04e1cbad8e8b4e1
9fea768bc23cf62a9b820b0a559f3932ce308676
24931 F20110114_AABBLC sauer_b_Page_085thm.jpg
24727cd167a5ddb67eae093d752d721d
abc4be5ab684afb07785f0ca2a784c56ffb80aaa
81638 F20110114_AABBKO sauer_b_Page_069.QC.jpg
c897d581e2567b37d149a5fed44f59e9
e636528199a4e170dbe66d4e8fee7eb029b7c14b
76202 F20110114_AABBJZ sauer_b_Page_060.QC.jpg
b977e4b747add21fe1d026f2642ce678
0955205dd00634e712e405f5d37b95a9694f3861
164980 F20110114_AABAHM sauer_b_Page_102.jpg
724aca706f60a8af7a83314db159d94c
039c29a6fb0242a76574f5eecf40e60d53a5646f
188695 F20110114_AABAGX sauer_b_Page_083.jpg
1b4d5c3e7578653d92d72dee1901814e
dd8e8d832eca97749b7954d3910d69a27f07e2d3
216233 F20110114_AABAIA sauer_b_Page_117.jpg
cbeea530a7f29deca062074f6e5452be
c821a6d98b4461e964365cdf519e0a84708c35e9
22102 F20110114_AABBLD sauer_b_Page_088thm.jpg
2ee71781abc997eb0add9b6a60be3f95
a6feeef6882bb70b5678fb512eaf565f4b3720f6
30834 F20110114_AABBKP sauer_b_Page_070thm.jpg
3927646e4d7f45cfb23daee714709978
16cbc5b763e4f84e0e03d3e8c0abee7e6ca56dc3
216487 F20110114_AABAHN sauer_b_Page_103.jpg
a81f97ec2ccd333a40496e6b11a61231
3699c1621fc04f428b4c9d4c5c56a3a0a0350b7f
139996 F20110114_AABAGY sauer_b_Page_084.jpg
669a8bf5d5cab9c8bb9b624544a9560e
ac0d0b50689deb8f31c4119b7847091af86a362c
193178 F20110114_AABAIB sauer_b_Page_118.jpg
ab3785159cfd7b2f908bb499e3bfdc80
d09e5b424eb4fc788c0ff1ca62e281c32296d300
26821 F20110114_AABBLE sauer_b_Page_089thm.jpg
ffb2c5a71f5d6ebf0348ded23523ee12
480791ee457cea8c7e535cf22eb172efc8e18146
79830 F20110114_AABBKQ sauer_b_Page_070.QC.jpg
9d61cb9b7c7cb923eb30c25d16752738
562dfa4f10645fc57d9f511d6833a82a2d2201c4
195502 F20110114_AABAHO sauer_b_Page_104.jpg
b9a7945c3e519f3295fe5fb0429c3df5
4961592d481f738bf67065c458aac14cc1c68b0a
216853 F20110114_AABAGZ sauer_b_Page_086.jpg
075cdf66f40a7674b50b7eb2e0e60e58
545d080541667c6541e0a6241c044c24683d3639
212430 F20110114_AABAIC sauer_b_Page_119.jpg
ab01ec1f0011b37e9790b3f29801a199
8209083f801930d441371eab47248d7fec035843
63239 F20110114_AABBLF sauer_b_Page_089.QC.jpg
aa95351b9fbd4bc9bed0562056a3bd65
9ed07968e021e2905b0e698dd7d9e0fd33bd4df7
28748 F20110114_AABBKR sauer_b_Page_072thm.jpg
4df51b9f4af387f0573a5ab085e9c0ba
fde036b106215cbbd6f4e03ed8ec752b129c9aa6
193562 F20110114_AABAHP sauer_b_Page_105.jpg
57ff2404c2eb6a1abcd5119df79469a1
a0d7e1f5e22d28e0a2e8c75f81bcb2479ccfbea3
215312 F20110114_AABAID sauer_b_Page_120.jpg
00ea0ca81b4bb0f13fe8c88aa191e177
ede41ff0ab31999db05d5710e952b751de60d706
29284 F20110114_AABBLG sauer_b_Page_092thm.jpg
5008916ba037c8e171cfd061808db7af
12eb53a43471ae7011dc8b26041a63f56d34af5a
31718 F20110114_AABBKS sauer_b_Page_073thm.jpg
3d74eebd157529e62b8ef736324f0777
2c139ec72ce8d016929f0013abd076cffd052837
173331 F20110114_AABAHQ sauer_b_Page_106.jpg
f3d57e846b90dbf16eef1194e8db224f
964dea17f18bdef6f8a1ed2d6fc0b67b53f42f2e
208737 F20110114_AABAIE sauer_b_Page_121.jpg
147cf25245a9cb0f7cfb0221b36676a6
438b554a4337198a6fe4f00eb2ddaaa906bfc424
75555 F20110114_AABBLH sauer_b_Page_092.QC.jpg
123fc286bf347e941ae0259011b214b9
a3efe870aae0e6e90cc3368ec9e5ddea31291525
204122 F20110114_AABAIF sauer_b_Page_122.jpg
cbf760b658e8207d5c9919549f00f553
1c3a0c10c8782ff829cfeb4ef3892722179a0c85
59331 F20110114_AABBLI sauer_b_Page_093.QC.jpg
db2063f1badb7d3390e1fa158f75b93a
d3a1325a86d9ea55f15dec7bd9dfbe4c24096490
74387 F20110114_AABBKT sauer_b_Page_075.QC.jpg
3979f451f2fa4b93f69c71cbfa0a58df
719c2f7bea0215d5c8075e206d44f114eed5d720
56240 F20110114_AABAHR sauer_b_Page_107.jpg
d2e35e232770bab1b4c675f7786ec5ff
01d5ec5bab13b96e80c0180a6a2f410f031ff0ca
220948 F20110114_AABAIG sauer_b_Page_124.jpg
2b02543d1d83a5f5030a76a03d122c42
b9cf68c4d8d65fb03541dcd082ab82d059025521
22042 F20110114_AABBLJ sauer_b_Page_094thm.jpg
3060d444ba4cb2c9e0a178a1356a315b
d5ed404c402d137ae56e4fb09ac0290e04eb788d
31689 F20110114_AABBKU sauer_b_Page_077thm.jpg
d16a9d931422ed29b7d4bc850f3b21e4
523e9bb358283ec0fe6f1dbb67d186f9b7285649
152951 F20110114_AABAHS sauer_b_Page_108.jpg
61789551a6d2c3d7fbe3a18807b7adb0
d7f417bd9b5710dd9b114788adf4e20e7c69fe40
25080 F20110114_AABAIH sauer_b_Page_127.jpg
ef6102af15b072a0ab4a224ead73c885
91b1dcaee5e79e2be0f19c49ae0a8c8cbf47e04c
51983 F20110114_AABBLK sauer_b_Page_094.QC.jpg
440577a0fef1c8432fce0a6ecc1d7806
e21d78bb43b97d23b9a540185877e01a1222917c
79378 F20110114_AABBKV sauer_b_Page_077.QC.jpg
6a1c147877f0cf2ac44c570eb711aa96
1f536eca1ae89c054d5838b0c3c3d7d1e5f15883
176580 F20110114_AABAHT sauer_b_Page_109.jpg
a7666d591502897717e91c72534df2bf
e20a3b30a86a0da7393dade07baed49bc0528d84
161331 F20110114_AABAII sauer_b_Page_128.jpg
082eee0216c1348b65356485d65725b0
83e17a95760c76e8a0fcd682156baff2167c7aa8
31305 F20110114_AABBLL sauer_b_Page_096thm.jpg
c8a94543d50ec449d389a7f1fdbec52e
91d127ae966a1511c1e3e5422a52ae0de15c742a
75977 F20110114_AABBKW sauer_b_Page_079.QC.jpg
dd2d2861b78a1aceb639e39447df6c09
5ca20b4a1108c3c91038666a877947e4d2b25459
26303 F20110114_AABAIJ sauer_b_Page_129.jpg
66f52ac9c63245c0da240c53aa6c90f8
e089fc352cbf623d7d26e7410d48efb4e298e976
224728 F20110114_AABAHU sauer_b_Page_111.jpg
96c33ddf4804bcc5fc218a19692b8518
b011487341aee988129d7f2cf58236d7fb042fab
88806 F20110114_AABBMA sauer_b_Page_111.QC.jpg
bd959a0fd0841cd70c215097cd95adab
18e4862fdbe29140e2ac9efee809c8693b1dbb32
86009 F20110114_AABBLM sauer_b_Page_098.QC.jpg
1f4d430c9554a5e07008f86234cb7c2f
4da7208e5a318b48a4ec547d4eaaa710dafe1d38
28394 F20110114_AABBKX sauer_b_Page_081thm.jpg
3be2c1edebb9768f9caaf4f1a1469b53
4e82dd859f3a3af1072ccbb6f0c57c0020fa28df
95264 F20110114_AABAIK sauer_b_Page_131.jpg
032bf201152d391614e1f3ea21a6bcb8
1dd5788035c15a4396e2ae0bcbe35f80e098b971
211136 F20110114_AABAHV sauer_b_Page_112.jpg
0986e6164d5417d826e921116393b668
ea67c679d5c6cd686155e1e0cee55c7d9c2445c0
32650 F20110114_AABBMB sauer_b_Page_112thm.jpg
62f08ae0f3bdbd7d6bfb7cfbd8b70f59
6366830b7251e33047a1bdf378f48964388b603d
30387 F20110114_AABBLN sauer_b_Page_099thm.jpg
0d11d69b0327f9d03b139584e3677858
75bf27d6f16c2abd848bc4fbff8fc516c08dc67b
15868 F20110114_AABBKY sauer_b_Page_082thm.jpg
2c22245df4ce85122a25dc9e09a68f88
260aadf177d1d72fec86b2053866111efc0fafa3
157402 F20110114_AABAIL sauer_b_Page_132.jpg
5b78814c84d93a1f898b3a9f58b3b6ce
4c704a63dd2ec37041a4108fbc0e803473067221
217269 F20110114_AABAHW sauer_b_Page_113.jpg
1112e223c78af8f0b416112523780ca4
6e8d742c4aabf5d95e64a8460cf23c73529a08fe
32018 F20110114_AABBMC sauer_b_Page_113thm.jpg
33c72f519018825fd47ec2a2bf87b272
8b5296454b895e757bb5d51a6c3ea91b98897af2
75462 F20110114_AABBLO sauer_b_Page_099.QC.jpg
5a301ed0bd68f6c68b73411c6b67b234
eccd9cc4be7c58075d78922ee30fb24ee132865a
28671 F20110114_AABBKZ sauer_b_Page_082.QC.jpg
c05e2576524b320895e92d4319e72301
10915c5698a0c602a997e7c5f35c9f9a4393c0a3
132852 F20110114_AABAJA sauer_b_Page_151.jpg
19ef0c74fb0d5f14afb5e555446a42fe
9ef65ede77aa25205d52fa243c6760d83a6dac23
165529 F20110114_AABAIM sauer_b_Page_133.jpg
42fd74dbf63dcd0f998bbf570c372359
36e0b484a2136220bf477a8766cd6a6c4d33017a
202771 F20110114_AABAHX sauer_b_Page_114.jpg
3fc582596dfcc9e7dedfa9a6332ca48d
46ed8b769c675657ffd596fa6c8dbbfe848b782d
30873 F20110114_AABBMD sauer_b_Page_114thm.jpg
8a66a3cfb30aa08af06a8c5e96d070dc
46529a0ce00b62bb035ab6acfda28f1ba41edf9d
79301 F20110114_AABBLP sauer_b_Page_100.QC.jpg
2ff871b65d5842287e8a2b7cb60a5779
73f4ca04a277e6caf8d20c2c6fca5d6fe578f087
173259 F20110114_AABAJB sauer_b_Page_154.jpg
3da446eca2b85e7c79947f2b423dd4b1
4f784b9e1602579efd5fdea2af1634518ba2dfb4
191472 F20110114_AABAIN sauer_b_Page_134.jpg
170df3d66fc2dae99ceb9bd5bcd8c1f0
ab6e977603eb4cebf29861c1e61649e5ec33ca3b
207036 F20110114_AABAHY sauer_b_Page_115.jpg
7b44ffc1df190cd98af13952b4d5491f
5b2334c6418ec5ef5175710195a3f6432c2f68bd
83465 F20110114_AABBME sauer_b_Page_114.QC.jpg
d48cf7019880300f5503acd791c51577
a783c35b7df8e61abf7e90b025a906472470e378
61187 F20110114_AABBLQ sauer_b_Page_101.QC.jpg
e7b2d6405fd1ecc088eaec575ac0b6e5
52b269843542577b35ad1bc74e3ebaf7f66757a5
157922 F20110114_AABAJC sauer_b_Page_155.jpg
6d1579b301e76b967d6bf5a0648ec91a
1d2801edec522e24375d98fbc6884329239ea187
200666 F20110114_AABAIO sauer_b_Page_135.jpg
e6149dbe6e59118c24fdb4021b03ffa7
9b27e6b33d5c13bca91338e572d7f9707eca0fdb
217279 F20110114_AABAHZ sauer_b_Page_116.jpg
54cc77887931460e76ef574638b41bec
46e221090df932cb30b24b86c8d796a07cf822fa
31412 F20110114_AABBMF sauer_b_Page_115thm.jpg
d1e10d83288319496441d7095c8c4b43
86cd15ff9d05d09f0f3eed5ab3f1c3436178e11d
32043 F20110114_AABBLR sauer_b_Page_103thm.jpg
2aefb0d18562b6461a0d81122c11a7cc
d5b469e8f93a2956a61ae7389a828db9e72ca12a
181370 F20110114_AABAJD sauer_b_Page_156.jpg
514f4b913661d4a42b95e31ea04f1961
9b1c5dceeaa1a2ed62c884466dbc2ad49c87c1b1
150301 F20110114_AABAIP sauer_b_Page_138.jpg
518389a510e2967c64d7d5319ad414c8
0e5eae420eb51e1366e8633052c524a3c35ea291
86994 F20110114_AABBMG sauer_b_Page_116.QC.jpg
e1e54e0995c174408d6758a56400cfb9
a529bd24d3d4d1b7419d0e94eef77b0bb4fa71f0
29436 F20110114_AABBLS sauer_b_Page_104thm.jpg
a0a6714ac8bbe72caa657cc40987db2a
ec23a856d69f99f100889545c39bf8a29f066a1c
199769 F20110114_AABAJE sauer_b_Page_157.jpg
13c7859b6719c7e2963720b57704fafe
41d3a1570548a15d8a69733cd48529c70695a4f3
114631 F20110114_AABAIQ sauer_b_Page_139.jpg
4d37dd901857ba43e780a341b1a1367a
d783f4222188c0f07a799faf3b7f6355b4033f7e
33157 F20110114_AABBMH sauer_b_Page_117thm.jpg
7a1af2bd516e70790238f41ff8c97c03
f6648e842e0760d1e54fae772a5c18bb83ee6170
28538 F20110114_AABBLT sauer_b_Page_105thm.jpg
0e2833987f7464dd576f7e4125350b22
c3d4b2964e3f7632f2a4003e238c15c00a7e4a7e
168143 F20110114_AABAJF sauer_b_Page_158.jpg
628bfc33ddf7900f547d4df90293d22b
0883c21a31d26b265b8dd75ec054e2f5b0293271
62450 F20110114_AABAIR sauer_b_Page_140.jpg
4433513398aaeb8a6d89dee329c1f640
91118270bdf38a4cef35b9c2f40b5fd590d52aac
31028 F20110114_AABBMI sauer_b_Page_118thm.jpg
68eed0326dc1a651ba29f50f675fd5e4
20e37aca6457fa18bc5efd3f8d98501b26bda91d
207271 F20110114_AABAJG sauer_b_Page_160.jpg
3d04de2d0e3085afe240c0a1f33de7c6
4d24af913ad542dd15986dfb81da96245f317bbe
83663 F20110114_AABBMJ sauer_b_Page_119.QC.jpg
69dd563fa12c68a62ffb2eb30722f42a
4feeb408bd5d13c7692c6af71ee6c92c3332032f
76090 F20110114_AABBLU sauer_b_Page_105.QC.jpg
af10ca5407def3d2620ab7dddf7bbe9a
4759390789eff9260c65f4ccc6b62bc00933859a
103856 F20110114_AABAJH sauer_b_Page_161.jpg
e3dd4e950bdecfbbc10aab83da4444bc
e4dd65bc74eeb824633bbc3fd0b7a73165a87fc5
65140 F20110114_AABAIS sauer_b_Page_142.jpg
1227d81896037532e463535f54ead1d7
05ebb2d2ed8e3f41c3f8f1fdf58a6a918adff4c4
83034 F20110114_AABBMK sauer_b_Page_120.QC.jpg
bdb99b4c4f9083c2a4af04c92a058ae4
339fba1040819852c51405444cfba82b7caa927f
14788 F20110114_AABBLV sauer_b_Page_107thm.jpg
72ffbbc5cbce54b104c4c0a3cecb8add
18514e448ce72e9b4e3048f1c48bc229939055b4
146683 F20110114_AABAJI sauer_b_Page_162.jpg
11b439dabf0009eb3f763b1cdda6dc16
38a63759037cbe289d9454f7fa537ca27001b0dd
144333 F20110114_AABAIT sauer_b_Page_143.jpg
f7dea33a218b6bfe27a12dc991731a2a
025d109c6a3a11bbe12dba30244367000ee1b91b
31617 F20110114_AABBML sauer_b_Page_121thm.jpg
59ec80c934a746c56be2bb094adb10b9
a378b36a86e2a5df574fb039d852014733111575
26490 F20110114_AABBLW sauer_b_Page_107.QC.jpg
d156b347fc7b3b2b03f1afcd41bd3f1a
99829e46d2964a827e280696a4e13c2d8295e3aa
81923 F20110114_AABAJJ sauer_b_Page_163.jpg
7e6f83dfd8dad1294dd0b20cef71e733
dce0ddf3f67f4a3d460ab2e25d1ce2a5be29a59c
201854 F20110114_AABAIU sauer_b_Page_144.jpg
ac035f23feeae9d5cf3b22df79d3083c
574d919ff477583fc58ec89ab31de2f4eb7eea2d
55745 F20110114_AABBNA sauer_b_Page_138.QC.jpg
cd650d43e3168880551cb6013b5eb3e7
46734c87aea2e1b2e5f5a92db25ce9bb2b33d9c8
31072 F20110114_AABBMM sauer_b_Page_122thm.jpg
9aa03c262e833793e602d78c478b5881
18bd528d38016a7cdf0b1630c6195c3518b2342c
23552 F20110114_AABBLX sauer_b_Page_108thm.jpg
94a49220475ab8da52b903fcbcb1d90a
1379f83c7107b91863562aa910024760f2594d2d
88417 F20110114_AABAJK sauer_b_Page_164.jpg
89f8b79bf2db48e8dd72b9c3571eecf3
b827cdd9abf1f028f69680ded3a854354332fe5f
177481 F20110114_AABAIV sauer_b_Page_145.jpg
291b3dc3abd1c33c8aea9ce1254145a7
651101b7837224acfdf95016fb11f3000600ecf9
40328 F20110114_AABBNB sauer_b_Page_139.QC.jpg
b6874e0647bcfaaaed64bc8d9ed2b4b3
d62b86a684336097f58516b6fc4b6871f929e52c
81869 F20110114_AABBMN sauer_b_Page_122.QC.jpg
d0cd237ab15e7cab06f1de929eefed3e
1dd9846c1a25489f88c5c5731deda2bd878a865b
28115 F20110114_AABBLY sauer_b_Page_109thm.jpg
d476a11e8ce253094732fee39260be9f
13233d08e1d12b371d85242647899e7b95697acf
233000 F20110114_AABAJL sauer_b_Page_166.jpg
94d9e561174e3803d7946e8b6347ca8f
ac15116a45a01016b9df85ea0b9a824e651bee6c
168672 F20110114_AABAIW sauer_b_Page_146.jpg
f4592100ce1c361b1bf27e25cd0c8a08
8176272f637cd5fee6159b19f895ef22ab6e337a
28212 F20110114_AABBNC sauer_b_Page_141thm.jpg
70a5a681a15fd2ef0d2c79633707ba17
54bde592ef012e51c02b898748d13920b3196e23
80179 F20110114_AABBMO sauer_b_Page_123.QC.jpg
7c0748f2c59201b897e8b03196283700
33439deb5c0f2886625135ff0010cbe99aae08c4
30722 F20110114_AABBLZ sauer_b_Page_110thm.jpg
b1b50ac4c51cd6c4a453dbcc67a34f99
e6def2df234cc166ce05e3afa378f246ec4ae8eb
272725 F20110114_AABAJM sauer_b_Page_167.jpg
9b26edec2f86cb81f557cac68b0b7ee4
c43d457f38f9fa8238f340d5ac02f2d007094d2a
157423 F20110114_AABAIX sauer_b_Page_147.jpg
a76c10e19be6c5ffc567e1c24ec034f3
a734dcd0e2e90e7cac821b9c288519f3b64ed77b
177180 F20110114_AABAKA sauer_b_Page_009.jp2
eb0603f0f04e820ff35e8930a8e8684f
bad85369d37082791ed05bce2d46fb3ee1f3aa9e
16394 F20110114_AABBND sauer_b_Page_142thm.jpg
cb3ff9a90a9b08725a614b734ea2a949
97b25623df8ccd234fe2156b963e58d88e5a1ad2
60500 F20110114_AABBMP sauer_b_Page_126.QC.jpg
c3f0d7df60a299b59240ef0fef7900cb
a1e575bb81d4247de8f53aec47ab04b4061b3d24
258058 F20110114_AABAJN sauer_b_Page_168.jpg
25fd3a0d7e3510813571a62e7af68917
3cb4926ea5a4e139bce13de9d02e9f1c0c1136e6
148649 F20110114_AABAIY sauer_b_Page_149.jpg
e722a0e7241b87438d9cbe4e9b7e4710
d3cafd232a0dfcb87f7c7a96f100f23c14db4070
876936 F20110114_AABAKB sauer_b_Page_011.jp2
3ca5ca163452bbcf2b092f08214f02e5
0b43f6c975cbe2d0836c02cb46d8541f5bae7c6c
28149 F20110114_AABBNE sauer_b_Page_142.QC.jpg
e9bfde282295057a75fc8056f4f38a24
066bbc02d42b927ebfea4388567c0331de3eb5b6
10868 F20110114_AABBMQ sauer_b_Page_127thm.jpg
31ce3e5ae0f841d38cef2951bf48e60b
b84b1d75169821af658cd989b379b295d63b1fd9
265226 F20110114_AABAJO sauer_b_Page_169.jpg
7ae5709da24acc35426eb35964e54b9a
85198dafb11312180843c4127ac6346ae36e8c62
122411 F20110114_AABAIZ sauer_b_Page_150.jpg
bb75fc5d26e7409af7c71f08dfbeab28
87d4298194affc7df2bff92ea3dca9abccbf6408
803273 F20110114_AABAKC sauer_b_Page_012.jp2
37b0c6b9e8facd0bea53f69803ce1f07
6e9e23cfb68d8f97cd297e9a757dc0034649c71f
60007 F20110114_AABBNF sauer_b_Page_143.QC.jpg
9fa58f1aeb49ea868b6c17697a8827d9
ad8efca58330f07fb1831cdea079c9f50f0a0a9b
23395 F20110114_AABBMR sauer_b_Page_128thm.jpg
3d116f0951a9497651633ade41e01ffa
283d104960c7e5d6316b53752f1031e3621b5a71
248584 F20110114_AABAJP sauer_b_Page_170.jpg
c683511a401347481c9dfd15be1904da
071accbdfc23663da85baf2bbf0bed51bfc89aff
808049 F20110114_AABAKD sauer_b_Page_013.jp2
efc788d174438cf8541c769e926fce38
49831cc96092a402bf578026cbe40124ab02733b
30566 F20110114_AABBNG sauer_b_Page_144thm.jpg
0fdda7abd4e3ddc115ce5c8cdeaacf90
0cc9df49ea6def15c866f976245d13475bcc61e7
11073 F20110114_AABBMS sauer_b_Page_129thm.jpg
09b8d43634abfea0d3641ea9edca7518
9af4408ffd3e20861c673e3026adcc990afeae32
55902 F20110114_AABAJQ sauer_b_Page_171.jpg
d9b06b6482b09881e0c7312c8e263b3f
6777f2acbc15cbd51dd2f0838c161d03463a0c50
1051918 F20110114_AABAKE sauer_b_Page_014.jp2
755bebf5ca7824e683d7c9ecc5bba767
7028077f1ab084413e845764beb94a551fdf7240
81183 F20110114_AABBNH sauer_b_Page_144.QC.jpg
ce272f3683cc7c8ecd48024e4560e154
19899d90315359d410415949397d8941e20fc594
24301 F20110114_AABBMT sauer_b_Page_130thm.jpg
77258dddabb431e183aa96604ca88a73
da96a1fadb711c33848afc12d5a09655c32036e7
75376 F20110114_AABAJR sauer_b_Page_172.jpg
9f3b2b3ec901f7a2b869b86ce821e971
acf31227f598c36d998b1f72725e06c2a0d9d2da
1007288 F20110114_AABAKF sauer_b_Page_015.jp2
d4492c3ca46855efdc20dfdb995a1b18
5669419be136ab09f0dc3633de019005564e0013
28829 F20110114_AABBNI sauer_b_Page_145thm.jpg
aa39d96d236fbd7a2d12cf4c1895792a
0357602d259aa098b07ba56d89271a23b8d8dd0f
58603 F20110114_AABBMU sauer_b_Page_130.QC.jpg
691cadc320793b1a231f77bf84219dc0
88c145b9071e83d259b4d6fdca90fa7172c24099
233097 F20110114_AABAJS sauer_b_Page_001.jp2
4c1a4fbf288fb447ef3809b3d4743e53
b25f56396e5f2c2868a130ed02500ca0721d6f43
1006327 F20110114_AABAKG sauer_b_Page_016.jp2
6deaf8672dcdfc5e57e5826cdef3e19c
e97ba63cb537105186ee0d4e07bba6188021d58e
26467 F20110114_AABBNJ sauer_b_Page_146thm.jpg
d12dd95e6bf97476f616c13d8d1ac306
e971deb2ffb93b013bd308339428e1125bf0ae0b
882861 F20110114_AABAKH sauer_b_Page_017.jp2
19dc92096902e63f913defdd38c53d4e
c471032603eeda39aa757e7eee6d06c1ed5e48e2
62783 F20110114_AABBNK sauer_b_Page_147.QC.jpg
f3c3d49d582d8581c07d2814a8607173
88dfa8ce82b36227e3f48a299cc59ba80a75fb09
18205 F20110114_AABBMV sauer_b_Page_131thm.jpg
f5a5319b3764f6683c48f4d2a8264346
038135c60ca982fda7c7268ee175cbf36a81a9a7
25676 F20110114_AABAJT sauer_b_Page_002.jp2
bc72568c7d60249a6fe49015bbf89dc7
3069fd83edbcb046e0ec8afc3de4fa15faa8696d
1016679 F20110114_AABAKI sauer_b_Page_018.jp2
daa7fc8c82b5d41f1846710f570d5611
245b58e4e5915024c9a9c073b84c8a31a5d3bb8e
22723 F20110114_AABBNL sauer_b_Page_148thm.jpg
bfca3d603ba62b773859cd3e2a739076
db907b22181093d4d81beed96531a45b0c074d3c
57122 F20110114_AABBMW sauer_b_Page_132.QC.jpg
6840381d632a779941ea613917b2a6d9
4de0ebfdd794b70367c80c086ebaf0a0fa6a0c81
12556 F20110114_AABAJU sauer_b_Page_003.jp2
1c14178bf08b5c7689fc1d664b8123ae
a00125c5c8d29d2b5ca434c92ee2e8fc0aa5d890
1051954 F20110114_AABAKJ sauer_b_Page_019.jp2
6a69d9e69bd42cb9cfede6e89d8356ac
4df90f84998139b233563d14932885ed93e8aa90
20130 F20110114_AABBOA sauer_b_Page_159thm.jpg
1e946f54393c8a4c65b87effc2489afb
5b240e8b8e0bd6f2edf7c9a17d53ae47c72f650b
52767 F20110114_AABBNM sauer_b_Page_148.QC.jpg
62c58e8ec0496984e2cae0f8951f69d6
eefc4e3d863b107dd16a51c09f5d2f2b0fdc3b95
59015 F20110114_AABBMX sauer_b_Page_133.QC.jpg
26da8249c77828c98d1e8100f62c33e8
0d90fe04cd39098398afaf3709cde1d23de49f58
807916 F20110114_AABAJV sauer_b_Page_004.jp2
2e88da963d72c426ca2449f9cb2cbfcb
aa195bb8253d9767934faecbcc99a9f28e40baf0
F20110114_AABAKK sauer_b_Page_020.jp2
d496987a05befbd6ace44fb45fef8999
746672f3460045680248bceca49c818f9566fc21
29614 F20110114_AABBOB sauer_b_Page_160thm.jpg
585bbbad2084014f975fb58ad0c0ca23
93dbf083c2225af5c5af43bc4def8321d09eab35
55725 F20110114_AABBNN sauer_b_Page_149.QC.jpg
db51a304d9abf682bc82e53dccc037ab
70872c3d046d7f908a61e86d25927b378eba7ec9
28714 F20110114_AABBMY sauer_b_Page_135thm.jpg
ed6178692855b312e37692b42796b15f
779385928a035f0aea2f76eaf49f4678a038a121
718413 F20110114_AABAJW sauer_b_Page_005.jp2
27e6f0b24f98841d3952facb5b18706d
3d5ba0c8c6c4d09dea5562c9fc579c7ff60ee1e3
1051975 F20110114_AABAKL sauer_b_Page_021.jp2
dd427b263563263572c3c0c6591e778e
d93625b5bd2f28bf8788bd4b0fd9e30df8271dd4
71269 F20110114_AABBOC sauer_b_Page_160.QC.jpg
5d5f7bb4a91fa1729155bd9cf1742aa4
cc07206ce2f77c59bb419f6d1c2a49c3171d93a9
21677 F20110114_AABBNO sauer_b_Page_150thm.jpg
103623e39d100d13f3dbe176c27d71d2
64aae64242bbbcc77ada22f2cda0c898f251af33
64477 F20110114_AABBMZ sauer_b_Page_137.QC.jpg
442bceefa19e78666be48553b840c7d0
bc532c51333b12b8e412d4def6255c69f27b6d29
1051943 F20110114_AABAJX sauer_b_Page_006.jp2
b5da241ce8b07cdc7b851ac609d092bf
f9536290c7d4c9f38943ccf884fe59de1d77134e
1035071 F20110114_AABALA sauer_b_Page_040.jp2
645b7339afe3a9925dfe6b64007c4d03
d2230a1ab0b95689ded79306d08af9f9a46b9e3c
1051953 F20110114_AABAKM sauer_b_Page_022.jp2
62a5c33f5beaeb96b0a04a88cfa8e4cc
f0155ca10a1d2015351b5b611c5aa59c7674368e
37532 F20110114_AABBOD sauer_b_Page_161.QC.jpg
6b322c66a1e6123ed1941a7e84d6fbf6
a33a4dfb372e9897c32cdaf3ee6d273d0d87d061
53179 F20110114_AABBNP sauer_b_Page_151.QC.jpg
e161f3531d2e1fc650c476ee2a449efd
97b035a092ad0946c621774a13c2aae84b063c15
939647 F20110114_AABAJY sauer_b_Page_007.jp2
99356f46aac1eacc0d74326f9d7fe372
3a923b98f65ee1fbc6204c4a111408e97c364a14
F20110114_AABALB sauer_b_Page_042.jp2
ce53ca3894b9b045fae60e97f9eb45b8
0a09ab2cafc0548792c5a1026d0a64b5e2755807
1051950 F20110114_AABAKN sauer_b_Page_023.jp2
da1a75437b665a66fd64122d72605ac3
dfb1b91954e9d28efe22123f5545a5567244d880
22729 F20110114_AABBOE sauer_b_Page_162thm.jpg
d40eb839a91398e8e6e0b7abe73eaaf3
b57708dfe0c192b608a048460c31070c750b53bf
20965 F20110114_AABBNQ sauer_b_Page_152thm.jpg
a66f7f0950df9f1470f6b517d9fe093c
ee1d1b31afbf4c8cf26c05a2c4d807418b4c7d6b
775734 F20110114_AABAJZ sauer_b_Page_008.jp2
b9de1b24a4beace420f17780213b0dac
cdf72956b6c0fdef2037d83b97653d4d0f917e35
1051974 F20110114_AABALC sauer_b_Page_043.jp2
443da58a6aa6629a85a7ddf05948a663
88da7cb6ebe505d07e8f1ad7b799836180768e27
1051979 F20110114_AABAKO sauer_b_Page_024.jp2
3cb6bca7eeae94f9011082e5a9034462
c17323b2f5d9c1d02160cbcf08b369a0297b2ccb
37544 F20110114_AABBOF sauer_b_Page_163.QC.jpg
dc7f877086818174f1dab997ea24e47a
60615b305995e55eca8a138f288cb49ec799b0c6
44313 F20110114_AABBNR sauer_b_Page_152.QC.jpg
db083e37193c8d8d96c48d09c050b064
b9cb813fe2a76bf2bfe62d718973f2888df46474
1051956 F20110114_AABALD sauer_b_Page_046.jp2
ff26cf4780cd51b271af6e0cb5418342
7fb52fade15d0947dd7d2dcd0ce20daa3e063c67
925884 F20110114_AABAKP sauer_b_Page_026.jp2
b2ed5806dc567cad4641940763c8d027
55508724aa415fb9dedd1ceca2f0934a1199130e
81626 F20110114_AABBOG sauer_b_Page_166.QC.jpg
644de01ed4bdf728644b2e8675dfca61
cbad3c1fc90052c1b7df31e6943723d63d3a829c
23388 F20110114_AABBNS sauer_b_Page_153thm.jpg
b4c7ae54c229d1d4f138ce3cc8870b7d
2ee845cdd9b1c2d2d85d12b9168f1a32fd9717f3
1051959 F20110114_AABALE sauer_b_Page_047.jp2
c643535dd5b030365d59bed94820350a
c6a723e493d5cde4894f2251985ffc739f0528e1
893785 F20110114_AABAKQ sauer_b_Page_027.jp2
cf74c55e666d8cbd60d40491fc85f02b
9827484e9b9fa22e0d9a18b369cfce919967ca60
94779 F20110114_AABBOH sauer_b_Page_167.QC.jpg
099993f6e868a031b3a9a4c87b247760
dd9abf01827d4870757473744d4aae72be523190
26687 F20110114_AABBNT sauer_b_Page_154thm.jpg
0da0ae26b8d29ea0f1688fb3fe643e5a
5e64051e2f088cdc09258dc8be87250764603b45
F20110114_AABALF sauer_b_Page_048.jp2
8700230a124f36b32accf41401e6563b
363acdd6308b2eb951ba4720ffa2bd806f75a574
845582 F20110114_AABAKR sauer_b_Page_028.jp2
25346dedd121f1c6da797c16bdc67e59
6f944366b017495877cb060d5a2ad0dd46082fca
91271 F20110114_AABBOI sauer_b_Page_168.QC.jpg
1638ab86b5a6c48a64cb47b393a9fc02
fa1c6cce20df2e15ba49de02272d2df4e7e3b83e
28169 F20110114_AABBNU sauer_b_Page_155thm.jpg
d7deca7bdd65e061746d709e2b3eaa0c
740f3fc1544f34a089558c83ed976fb03ef8b4ae
1051941 F20110114_AABALG sauer_b_Page_049.jp2
168b566c2638c0d67988a8871c329ae8
5800233b13d6c52e4901c1ac188286d09b67d98e
152812 F20110114_AABAKS sauer_b_Page_029.jp2
4a5a9fd6bd84aca8cecea83b80fa81e6
6975f94a47500e4f410d9612d26c4cf0b5966f28
31235 F20110114_AABBOJ sauer_b_Page_170thm.jpg
2b6f50bea9cd7b7c5f6f77f2e99a5493
b024dee441dc05d3cd53a59d3acdcd065770dd6d
62760 F20110114_AABBNV sauer_b_Page_155.QC.jpg
4fd5a36383849d4a8b5aeffe2b8b7a8a
a8e318a0e0b76eb893f9743d21cd91939a506bf8
713571 F20110114_AABALH sauer_b_Page_050.jp2
89f3d63d831196ef069e03c9821b09df
8c16d5156ac1db4e067a660a00946bd2d1e95f4e
1001806 F20110114_AABAKT sauer_b_Page_030.jp2
000a5b3758fc6e26e2c688e6a129acc7
6d826b7a8c229e10baa4356e1d343d233e167bab
16270 F20110114_AABBOK sauer_b_Page_172thm.jpg
5cce39ce2c4570798df11c34dc91020e
b1fd8cc0b68873066dd3ac266efcaa4ed2fd7339
970356 F20110114_AABALI sauer_b_Page_051.jp2
38650e3159f411326e7ac491fe1ea6b8
6fa2e92c8e99c3622c1651892c5d426c55358f52
34491 F20110114_AABBOL sauer_b_Page_172.QC.jpg
f95a81b448652e0074debef2e74a49b6
0f69058032f794102c1f51663f7836af5a157ca1
27405 F20110114_AABBNW sauer_b_Page_156thm.jpg
27a366e4f960f2f549630f615118975a
16e58af3f1e11fe099b811a2800068eded397b54
F20110114_AABALJ sauer_b_Page_052.jp2
77d24717132c94b3bce0ce2a452cdfbf
48397fe8a58a4fbda3e10cc2836b1bd6b1dcae8f
1051949 F20110114_AABAKU sauer_b_Page_032.jp2
45bc64f321dd19bbd18846be5821bbdb
00a639e658e0aff0adc67fdc07c7c8cc5baea6f4
70287 F20110114_AABBNX sauer_b_Page_156.QC.jpg
f3b68eb51ffec2a54af08fac0262d662
8a1f06f2d8cf38b86d3bf70efd7967061e88959e
1051964 F20110114_AABALK sauer_b_Page_053.jp2
5176637ce4803057e7cd90a46f83c000
fa24dba865e9dcafee34d06a4185c9d14ca40469
1045173 F20110114_AABAKV sauer_b_Page_033.jp2
e508da1e877f71a70885099ca2118090
9ca9e77183ebe8388be769258aab09154db5388c
75568 F20110114_AABBNY sauer_b_Page_157.QC.jpg
e3f1b800950797df81ac69e5e37b6e49
c92a4317db2e1f3c1c5457b8e742a4ecb0f530e4
1051937 F20110114_AABALL sauer_b_Page_054.jp2
e41439e22ded0c86a777cd671d89ba06
d708ed028936701728773200fd15a3e668ac68c1
1051917 F20110114_AABAKW sauer_b_Page_034.jp2
1be4bb61791154b6f057348b3236d0fd
d7e2b32b72c3fc9715e72d1923fcd3e1038d4816
67813 F20110114_AABBNZ sauer_b_Page_158.QC.jpg
fb564db75febdafe4b7e5f13fc26779b
670b65944c8842d01bf5706ff24bc5a3d3a35172
1001218 F20110114_AABAMA sauer_b_Page_070.jp2
16e5e9ec081a082b5bf5294bda4244e5
c03d94fb28b9ac2679147c98fd25c35a97be0063
896145 F20110114_AABALM sauer_b_Page_055.jp2
e35557b6860738eb5feb584bbfc0bf7a
ef372a6fc4cb5f7ba9f8669a53d5996ec5b6d1af
1040892 F20110114_AABAKX sauer_b_Page_035.jp2
2a254f64bb9d71ba4971e5cfc28c66ca
7865ff38ce15a6ce403c8686629a30cf4e9f282e
846000 F20110114_AABAMB sauer_b_Page_071.jp2
b268e13f74a3bab06fbd8ebe3bb13779
249c2a225f73ed82569136489485f327ccb406ad
908437 F20110114_AABALN sauer_b_Page_056.jp2
9b6c8f6c522b8a222e80aafabb218a92
11ec9d73e6b29aa7747ee8a78cb2877c1443ca44
F20110114_AABAKY sauer_b_Page_036.jp2
e95885666215f922deedb33d33def9c2
55fd0b551f9d7f2436f2888b80d57c97d0557526
1042486 F20110114_AABAMC sauer_b_Page_074.jp2
3de60d4fa6aa574a0c1d313fc609ad53
dc04d142fa109a8067736dd7ad140efc9a13c87e
1043558 F20110114_AABALO sauer_b_Page_057.jp2
9868e71404767e17a6f7157cb323f174
8882c5ad5b90651c259bceeba15c4cf3fca0ee6d
1014993 F20110114_AABAKZ sauer_b_Page_039.jp2
06fc86f1ab29bff9aeee706ca60e6157
05a2b34ca870a624eced9bcf6db535903e5a1571
942489 F20110114_AABAMD sauer_b_Page_075.jp2
71cc3f0eeae77a9f513d731859f37f08
a7a412b5c29ca1c7afb56dfee44b4191381e4f23
1051973 F20110114_AABALP sauer_b_Page_058.jp2
6469160503585f2e77cf9afd157bea92
4f9dfb44e42ba9b02314290b88f49b363cff7958
F20110114_AABAME sauer_b_Page_076.jp2
3941a665dacb06deb469380056c28a14
c9c4eafdca6dcaca46f73694a76775531836c535
975825 F20110114_AABALQ sauer_b_Page_059.jp2
f26d0b5e6eb1c78c6141e45c0aeda239
b7afd4780e724b71f8afb80345f5416530921733
1003357 F20110114_AABAMF sauer_b_Page_077.jp2
143831646f87ae577bdbf983b5da31f0
bbd9708b8f62eb1524a80fcfea0b59839fed3eb8
1003636 F20110114_AABALR sauer_b_Page_060.jp2
bda3dc10b75640f83621199709a05b1e
3067e30c89884cbc7c9566018de78f54503b8241
1051963 F20110114_AABAMG sauer_b_Page_078.jp2
905f8c65301f667daf83d20b47fdced4
94294258f42d120e8eb7caa2a8496140f5910633
392751 F20110114_AABALS sauer_b_Page_061.jp2
b5af04c95589658c6ec7d71c0969bbce
170b7f5ca299f6e74bbd01dbbaed1594e69e4db2
951590 F20110114_AABAMH sauer_b_Page_079.jp2
392ac8593d47418d6b7bcce0d6b76acf
cecac3a65d9837a0bd3c64be29a7d77b40141e92
1051962 F20110114_AABALT sauer_b_Page_062.jp2
3378f2974ac55024fe8eb7b0b9b2108c
0ca226175cbaa1f2eea510968f31bb848c53af9a
F20110114_AABAMI sauer_b_Page_080.jp2
344809b96a7ce408d8d720b6bdedad58
383566cf8ff3c25dc05d2d751b1f99b082ef0bc4
1051978 F20110114_AABALU sauer_b_Page_063.jp2
0776a2a0bab6676b6f6859613e3262cb
ed204847dd5fb10298657866ec23641554159fc0
869534 F20110114_AABAMJ sauer_b_Page_081.jp2
cf78c077511d51641ea5625e8e49a8f8
8931516dacd276b43d34789a5609affc0387db09
265764 F20110114_AABAMK sauer_b_Page_082.jp2
e4f6ef855762a695ac7ae721585032d9
9e0629df1bc7aedb1da0ca3bfc84698fe5bb511d
939280 F20110114_AABALV sauer_b_Page_064.jp2
b9e43f12022a847ad1f0abf1da4e9bed
0aa3c632865ee83b6629221c7a33dc4ecfa55c55
951005 F20110114_AABAML sauer_b_Page_083.jp2
f52cb1e4de55649499ae1a3427d2c3e7
81b874162667e2cf1aa0fc4445979687ba9443b2
1051947 F20110114_AABALW sauer_b_Page_065.jp2
f18d822babc9a362fb8af729fe208d9d
2136d3a03ee47c7db47def87fc5cc15a320a88a0
631554 F20110114_AABAMM sauer_b_Page_084.jp2
85c3e471d3b8b9b088eca3326a9d2f51
1eb7526480b12bbff13e596075efeefdd6f93721
554364 F20110114_AABALX sauer_b_Page_066.jp2
2302ffe685b02a3b50772bc872bb5e0f
3b8e13f9af017eabd0830c703dde256020a0f078
949656 F20110114_AABANA sauer_b_Page_099.jp2
4531d89d109c3032ca3aa7f7c1820174
1275c4fa4790d8ef08f0f2ae87e8c0a54f8fb969
625539 F20110114_AABAMN sauer_b_Page_085.jp2
aee2e0494f03c5cb833445c9512680e1
af8d57b90c706f2a7d30678f2ae6f92b47908b23
867532 F20110114_AABALY sauer_b_Page_067.jp2
280f92355ca577608747cf792ea073e8
91e27a2976f24e2fd57c0207fc0e979f4ef02023
1051933 F20110114_AABANB sauer_b_Page_100.jp2
aa7bdf3a256f0dd9ce539bb3602155f7
9b3cc5eff5712472caf6e173d2836b547ac750d8
1050892 F20110114_AABAMO sauer_b_Page_086.jp2
50d272c17e7bd9d709a862d7f21f81cc
b126e1d8409f8f4e1e70633176bb1c661372f4a1
1044918 F20110114_AABALZ sauer_b_Page_069.jp2
6c251d5502a093b7b9831a469b0f47ff
6600cf6690dfea480d2a93ce10cb516fc04804fb
935456 F20110114_AABANC sauer_b_Page_102.jp2
1b439d2e19b323d4af0b3b6730309f94
38393887c3c631b54d5a5d238449546c1597005a
1003330 F20110114_AABAMP sauer_b_Page_087.jp2
d9d1f9a8f06e4ab1f59d42644b2762fb
9155b64c975032ad67dd3f51809590b1f1842b58
F20110114_AABAND sauer_b_Page_103.jp2
d90098002ab1ac25a3833a74e6c612c2
3efd2ef45ad89201da1e50a6357bb6d5a8c5a5de
564698 F20110114_AABAMQ sauer_b_Page_088.jp2
0d0b0f54ad6ed62c659f07bc412936a1
43fd9d262a682429f0564027880940247fc261d0
987804 F20110114_AABANE sauer_b_Page_104.jp2
506b0191ad638cf39b3e60f77ae4534f
e46f0115be58cc1bd2e50eb2944a5fb7b646ec30
809593 F20110114_AABAMR sauer_b_Page_089.jp2
c8782168c5c79aa9d5882f8f4b045108
bc02689dfd3c6421143687c56faef158de92f205
984796 F20110114_AABANF sauer_b_Page_105.jp2
c8a63a511a6bcad95e34c8376266838d
1fd9d075e25058974440bd65a411017cc46ebca6
865528 F20110114_AABANG sauer_b_Page_106.jp2
208d25ef592d78cfa91335df504665cb
d577f588b07edff4a2e35d5cd33fd6de96296a9e
F20110114_AABAMS sauer_b_Page_090.jp2
bccb75d48ccd6aa66bba8bdc47070940
5426de0e1205627260c3d7312c06b8a52a0553b1
1044757 F20110114_AABANH sauer_b_Page_108.jp2
a021b998797b7eac8cdc5916b6cc1a6e
ac73e4bee85aee766636b38bbf2990c310da6394
955766 F20110114_AABAMT sauer_b_Page_092.jp2
846ca5882777b38f4933731ffddb655b
1ae86e587bb254ea49454b4824151c56e645dcca
895920 F20110114_AABANI sauer_b_Page_109.jp2
f8f89241516bc5e05069886de441d362
1f30466d156c7f73c2d8819d2463f1e1cbcc370f
661778 F20110114_AABAMU sauer_b_Page_093.jp2
fa2d914119802b773fbc0f65ae2ab39e
5083c3ed7e5f8d4ce278d266bedfc66d5f2ab1a4
1051955 F20110114_AABANJ sauer_b_Page_111.jp2
9781dccf48656cef455939b4a09a9c85
ff695795420b8e333e5d1d27cee8165708f905ac
585053 F20110114_AABAMV sauer_b_Page_094.jp2
2d6646dc768b3f68034e80dbc2933625
59b47d4f9ab1b8ed86381ec686915102cfc1bede
1051948 F20110114_AABANK sauer_b_Page_112.jp2
662c7343db66d54eb45d1de19c21970b
7fe7ca6dfc3fcd4f69b3806348c03708fca8a948
1051982 F20110114_AABANL sauer_b_Page_113.jp2
97d43b7cb13d8d24d77fc95e0498c03a
fa3ae14d15ce38d3c4149dd90c545003ccefc928
946496 F20110114_AABAMW sauer_b_Page_095.jp2
0c4deb890ea27a32b14a253cd9784cca
f6668a7e8cae4e16f7993e27de9cff972a256432
F20110114_AABAOA sauer_b_Page_130.jp2
9b390464eae0bdb613efebb2369cd779
c8ced4ce24aaa57e4bb642d888d5fa7015c09e37
1050225 F20110114_AABANM sauer_b_Page_114.jp2
f4fd051359afe3acdd28827a7ae80897
b4e45d8809921234b7e65d7bf9404696860b09a3
1051934 F20110114_AABAMX sauer_b_Page_096.jp2
a1d36ed0102029bcbac24f95ae7ff6e9
9f45dac3ccfe41e0acd4add8a62da777c5ae6e62
493045 F20110114_AABAOB sauer_b_Page_131.jp2
56d8677414ed877f8526e8026946cefa
75b7d9a63394c087b6803b861307f557c80f1d1d
F20110114_AABANN sauer_b_Page_116.jp2
b6b48a378868c3116d6ff84dd256f1a3
66330e0f365a6ad274b1bb3c5bb0d7c00ed09851
F20110114_AABAMY sauer_b_Page_097.jp2
ccf128023d814501ed9c0f01125d84b2
155a6100c7a5066b474c96a50791395059d99c59
784072 F20110114_AABAOC sauer_b_Page_132.jp2
1bdc45d38ebfe769ddf77322cb6c15bb
1e212486398f866920e9833c8396d4961746d51c
1051977 F20110114_AABANO sauer_b_Page_117.jp2
2561db45b0a2bd5a8f40b83d1bad3a53
fcfa5768fbcbd7f1cf47d27ec76567cde1c99c9d
F20110114_AABAMZ sauer_b_Page_098.jp2
5d9b6fd3767e7b59fae3b4c7c29c5904
f4f88b873f53e54312de115ca3cb401b33376d94
815541 F20110114_AABAOD sauer_b_Page_133.jp2
313227a64304691824b54b04b6d1412b
044bec16276da639e09a853e73ef11e5c38adbb3
1006169 F20110114_AABANP sauer_b_Page_118.jp2
6beaca37acb3146dbee58859a8eff630
be62f0e74424d01d14d0d718ae2a17c5debf183b
967951 F20110114_AABAOE sauer_b_Page_134.jp2
26ea94300e648e7955df0fbcc971487d
5e801dfc3e48835002a04772ae7c36c45c6d6615
1051971 F20110114_AABANQ sauer_b_Page_119.jp2
aa0d4973349f227a6eb79e76f41c2b4c
331ae1bc9d35c0d3928dcc6f5ffe67e05dcff76d
1011532 F20110114_AABAOF sauer_b_Page_135.jp2
c77ba8082474afd8a335160c0a3a0750
e7917ba0d0375c4e385a37d27b03be7ed4958210
1051969 F20110114_AABANR sauer_b_Page_120.jp2
43f3d91a05fa57781aa0fe482486a441
4760fe6e80a53469c3007f2ddf2c0231b620977f
924292 F20110114_AABAOG sauer_b_Page_136.jp2
656cca249002deb4325e925ccaa65d6e
99c258212608c5f618b819ffd5284d54b9325a4b
1051951 F20110114_AABANS sauer_b_Page_121.jp2
0232963aec518db3e785cbbfee3d8c8e
b03fca0a9e092ca7eeb48f6e1aaf85ab0b312b67
887604 F20110114_AABAOH sauer_b_Page_137.jp2
04ffd8044e629e38f46d86f671a0825d
2a0e7de43fbbd30d71fec8cb816d2cb1b9864f97
1051985 F20110114_AABANT sauer_b_Page_122.jp2
01896878c6bb696d2ba6ba78ffd3d0ff
83010d531137839d6bfb5d27ee97ab2fe0a22e11
711291 F20110114_AABAOI sauer_b_Page_138.jp2
9780fb1277b9f78780eb4c91a573b4b4
8e78c2fd6a43f42756099580e75041d264e850a3
1051940 F20110114_AABANU sauer_b_Page_124.jp2
5fb57e8e359403bacfa9a14ad66a619a
dae3ed54fe32469d91bfce307623db2712c62393
550816 F20110114_AABAOJ sauer_b_Page_139.jp2
c92237fa1fadfcc28c626eb033421ef8
779a0bceb59717973f9dc19403ab2270cc8a9300
1051961 F20110114_AABANV sauer_b_Page_125.jp2
dcc35bbae093c1fb014eec1730f3bf0b
dcd6b7f64e71c2ffab4293cae4c4bdd0faea3b21
234237 F20110114_AABAOK sauer_b_Page_140.jp2
3dc12e463c7db14e7146e7d66104d3cf
8fbcae5ca77a23ec4d49e13736965851c5389c48
760895 F20110114_AABANW sauer_b_Page_126.jp2
b47819bc527bc94dc3fc6ade7dd08392
c94f213942394f78780dd47ca932dea0c3abe5d5
F20110114_AABAOL sauer_b_Page_141.jp2
eb485c82b53de559d9c7815ac315348a
2e9cb3760cb2759add804e19149c4810a66cbc9e
260531 F20110114_AABAOM sauer_b_Page_142.jp2
12607947d96b508ecb66208d6d7cb9da
68df3f77a1042669d79643eebfd32b0326b8dd2a
44491 F20110114_AABANX sauer_b_Page_127.jp2
3a8c6ed93dcd5e3a41d606914294696d
9e9bd90644baa93aa95686ac1468644e6c91cfe7
998163 F20110114_AABAPA sauer_b_Page_160.jp2
e9b8f4d1af1acdf0336a9ef20649eba3
28c1d0ff464efd061db254c0812761d7d80d9f8c
1029046 F20110114_AABAON sauer_b_Page_144.jp2
e573a3020141b371832fc88f2bfa9e17
32d1432913b5ea7e986327dd7819c5ae14f6a445
971375 F20110114_AABANY sauer_b_Page_128.jp2
54cab0b33dd6375c410669abcb11ba6f
edc67a5cbe510041b8f9227807ad7039aaf4e7ed
458480 F20110114_AABAPB sauer_b_Page_161.jp2
60dc7c1522d190300fa235f5d5c1e298
f61d55927efe136aee5deb68a684adb1cc7192dd
871982 F20110114_AABAOO sauer_b_Page_146.jp2
10a1977745df06375063f109d916202a
f617dfbad55f507b3446a4fb3de8b10c953b3f2b
51307 F20110114_AABANZ sauer_b_Page_129.jp2
beb6f0a7daf2b22357adc43a7f3c871d
8f33442a8c384ad0494e518f3fa2c71daff1dc65
687744 F20110114_AABAPC sauer_b_Page_162.jp2
0f1756657ca560a3891d627d37aa6678
ac0bcaff15f00874732fed834229afabaa4d2be8
626076 F20110114_AABAOP sauer_b_Page_148.jp2
5eca5e3b2890121d3e795931c8de19ca
93a080a8f84f2bfa99d312fb6fad0052be987444
281922 F20110114_AABAPD sauer_b_Page_163.jp2
8843720c83cf632fc42d56fdc730851b
a978ff4ba7662868fd5f34dd4602f3192ac37a10
708580 F20110114_AABAOQ sauer_b_Page_149.jp2
391278bffdd7b6ecc99e6d429160b6b3
15625e08d4ba239749be996cd30a32cb0323073c
321723 F20110114_AABAPE sauer_b_Page_164.jp2
ba20e62f120df2e8762ccd3a2f9253c8
bd9d14253e8bbbf859f02f20a704721dcc996c58
614160 F20110114_AABAOR sauer_b_Page_151.jp2
9682ec048f1d0c64ce38555e6c27321d
b687a9fb177e91074ab00427dfb2b85404955574
37581 F20110114_AABAPF sauer_b_Page_165.jp2
c5c61ccdb758a093c9999b2eec0e3bcb
d93be3710e51f3fac6b3086e3f837f7a15369989
471487 F20110114_AABAOS sauer_b_Page_152.jp2
89105cb35f727538eda9e8d3f8c39a29
380001d0ed58dec9badc2c52dc1a1122570598d3
F20110114_AABAPG sauer_b_Page_166.jp2
7300ef9f95656e86acac9be5a91d2ba8
c6d54e58947065537d2d435b3fa7e4358ce293c9
663358 F20110114_AABAOT sauer_b_Page_153.jp2
0edd6804dd781c00649d49d87ebc5c16
dca368639db0e94f9dfd7332da4084904c670c85
F20110114_AABAPH sauer_b_Page_167.jp2
af158c39c4eb9c001c199bb557a6c86e
fab266a914cdf4875e061e88fc79e1d23fde949e
876061 F20110114_AABAOU sauer_b_Page_154.jp2
9ceb7de4fc78091aacda0e7c977290e6
3828bf3fbd71159ee010eb9eeb2bccbb8d1c42c4
F20110114_AABAPI sauer_b_Page_168.jp2
27c2fec0e44b8ce4948ce8d162513189
e9aa654d320a0c854cbc2a49753b99a78796863c
770908 F20110114_AABAOV sauer_b_Page_155.jp2
896e9db2cdc0d25d4ae76185a715b1ea
61643beb86675251e0508f3a5ab4aae632f3128e
F20110114_AABAPJ sauer_b_Page_169.jp2
d25b35855aeb90072069d8e6059873be
9a6ebfeea9b2019cc81a50912ff6c750257ec39d
854309 F20110114_AABAOW sauer_b_Page_156.jp2
246409da4fede2d3b56eb5e7c7c84216
88ea4bac1b7775256578bf5ee4cc450ab54bdb29
1051929 F20110114_AABAPK sauer_b_Page_170.jp2
c362e77073d35eec77223bfbcfde1962
b8db49953858106a1941ec5b85758a06582b86b8
1015652 F20110114_AABAOX sauer_b_Page_157.jp2
e680531b001bc9c4e10dbdf02721f477
18bf58478266b0aced9b1705b192f0f04aeb932b
197812 F20110114_AABAPL sauer_b_Page_171.jp2
f693096126f6f6593f602db9e24cd756
7d2757102ba16478c9d71b360291452e0ef382bb
F20110114_AABAQA sauer_b_Page_018.tif
b2456ad6cc3bb44c8cfc435707257ef3
24ba18cf05a364121f77cd33d7360a6b428ed5ed
F20110114_AABAPM sauer_b_Page_001.tif
23cb55918340298b56ab6f5ba3e16a8e
e2b98d1c6b6a2453fe7a2e9d9e41690e48e11f8e
850124 F20110114_AABAOY sauer_b_Page_158.jp2
e50eb53ba17c51d60dc4b98269477a78
315a369fdbf59d56c6e5e2c211108e51b901ee87
F20110114_AABAQB sauer_b_Page_019.tif
5020a40c18e4b83e23313db0d9dad100
03c02e5df2021a1c2355c7f2d14d4264cd81274a
F20110114_AABAPN sauer_b_Page_003.tif
f2de5463ebe25294e9dab7e5193e9511
d0662a215a2a0c61b542a22edbe1ab1896f0368d
523983 F20110114_AABAOZ sauer_b_Page_159.jp2
28dc345c4a46ddccf812ffccf298bcba
1d8070ad145e8bc2be8e5ac329b96822d1b6d011
F20110114_AABAQC sauer_b_Page_020.tif
c5058fd4df8c75ecfbc35396e34b426c
f7c2b67ec43aba5f2b7ae96b5bb7016491c981dd
F20110114_AABAPO sauer_b_Page_004.tif
15a382ef06dfea9d6565e39bc9be93d2
95fd84fa38a0fb66a39d8e7908407d61685e254c
F20110114_AABAQD sauer_b_Page_021.tif
7b6a5341c110078cf7d86e2538a48bd3
11fbd4ec8866e53f2ba94a306f77244976245a23
F20110114_AABAPP sauer_b_Page_005.tif
7749dc744d11f0ae93bdfda61fe7cef7
886526364192e7babf3490dfd44be48796aecdb0
F20110114_AABAQE sauer_b_Page_022.tif
a34a51369c5987d01cff76edc8f99118
f527d4404badc81141072c7f54a04a4e9fbdc2a5
F20110114_AABAPQ sauer_b_Page_006.tif
3e4bf53c546f7fb3140742ee1fff5e4e
ec3b779babaecbadce8617b270f4792a1ce008a9
F20110114_AABAQF sauer_b_Page_024.tif
88d9215e83d2cde661afb2bcd4bf38fc
341c35cb68f4671aea0164ce3e6c3f8ca8b3d105
F20110114_AABAPR sauer_b_Page_007.tif
202d3e06fe8f55725161a07f4d51dc57
097f859c10deb86d1dd7871a49cc33d71412664c
F20110114_AABAQG sauer_b_Page_026.tif
362681c1810f15c953eb03171bd5be18
31ed217b0c274cd98864b7fcf628ee59c08bb707
F20110114_AABAPS sauer_b_Page_008.tif
41c76145fb70882f2379837407568d4d
becf8cb0d1643a956ed81e6e6ded6fb83a54c61e
F20110114_AABAQH sauer_b_Page_028.tif
64f97daaafc9fbf38e46f8e48321b009
dd04d3051d3a8c4dcea724c0ae0fe5415f032e66
F20110114_AABAPT sauer_b_Page_009.tif
c69774e370a1aad36c8ca0dd2bdcf89c
7d9a807e1863733bf5141f8927accd49769ede31
F20110114_AABAQI sauer_b_Page_029.tif
8668d968f1f443b40cc03fa3aed089ed
fc20f89d677754efb045877a95ac464b6c53e904
F20110114_AABAPU sauer_b_Page_010.tif
5cc716018817f3fe86e7b60b09e562bd
7e8b19c49ab404b78dc0a1a7054f32e702f0adbb
F20110114_AABAQJ sauer_b_Page_030.tif
f4df718eca64d36ea18071065928f3f0
a361bd65f44b91475787f32b13260e0b8cd83fe2
F20110114_AABAPV sauer_b_Page_011.tif
866cc7d7a228fa3961eebd540d8d9ffc
f8ca130f62099a305caefc75bacce21b230f0a68
F20110114_AABAQK sauer_b_Page_032.tif
e39f8ec430d63878b32554c63a2c7dec
f8a56ef393b1992b84cec5ccae3fb3b73821cf98
F20110114_AABAPW sauer_b_Page_012.tif
71bef4609968a33a70b88447928b5afd
57c56ad93ff5eb7afa5e1676dcf9742bfa0a8bf4
F20110114_AABAQL sauer_b_Page_033.tif
77ea5dad59a4b87448634a3b3bbdb22e
dea5756dcb6055c9999e2bb1d7533dab025d9ed4
F20110114_AABAPX sauer_b_Page_014.tif
9fca8ecbeaa71f19073706538123f992
f7b6c9047c5d67d92adbd336e98a4a872afbbbd8
F20110114_AABARA sauer_b_Page_054.tif
68c4bacf3ee813b0057bd184ee3081b1
0e2cfc4da616eecafb3079b63679a9d6b1515d41
F20110114_AABAQM sauer_b_Page_034.tif
0a7aaabaadd34d0f0d7b1e82223af42e
71cceab26fd52f7242dc65868e13064fdc182be7
F20110114_AABAPY sauer_b_Page_016.tif
373a298777d3313a00a821d37df2387a
858d34dd0e2e50dd7c7d10a7f95e3b707bf3e1cd
F20110114_AABARB sauer_b_Page_057.tif
2f11a054ed1ae3e1688f14655c9370e0
1d6bda7221d4879014d0a88332c92a725bd45ceb
F20110114_AABAQN sauer_b_Page_036.tif
25e91dd5abbaf0582d4ad89955e8ec29
ff8c7bd05590b5b0be640c7c591bc1e91b22eb29
F20110114_AABARC sauer_b_Page_058.tif
ac95e95418bae9f5660aa7b762e0905e
e199f12cda9c2c57e89f6be7e5d180f77c398e7b
F20110114_AABAQO sauer_b_Page_040.tif
bb993f09011720badb001b2dd27bc4fc
0f476f1c4ba3210390bbf3c095ef8cafe18ad5c2
F20110114_AABAPZ sauer_b_Page_017.tif
77005e76727e24f501a373be6e066222
0c42178e2be66b30f570d9d6af778d7be03aceca
F20110114_AABARD sauer_b_Page_059.tif
bd2a8976d3068d3ba7e82ba157e635df
398b58d20c9ac425418bc37060986b7c1e7221ab
F20110114_AABAQP sauer_b_Page_041.tif
f78db78e590b87a472f88ab12fb2263a
2b8768eeda8d93064061f43e0b7d03fa2cdf1fcc
F20110114_AABARE sauer_b_Page_060.tif
f1aaaddae006b59d1ea31b7c3a07c85f
80389d1ce4055673735edb5cecfd1f8df391e5c4
F20110114_AABAQQ sauer_b_Page_043.tif
2c2cf299390de453d1fe4b0f1b6ce745
102da5ec83660e4762c7dcce25764aac08543769
F20110114_AABARF sauer_b_Page_061.tif
edb5ce57c2e1240b1046c9f269b1509c
d51319da8946969897ede3f42cbd854426abf68d
F20110114_AABAQR sauer_b_Page_044.tif
3f44e536498611d449ebc469ead34a7b
47e8fbab37ba53c0486ae027227c3ced8cc3deac
F20110114_AABARG sauer_b_Page_062.tif
17f311843dc7dc38fc3a6f6b5d5f9aaf
0ff2b2db2e0a2685ede8ae2f40dfe4ea5c4a82d1
F20110114_AABAQS sauer_b_Page_046.tif
1a648607a30b4e5e726f483477489ec5
90cc481a0c0afed3070b9bd387e6a32d01030101
F20110114_AABARH sauer_b_Page_063.tif
606a1b07e8f45b5e2b5660b18b4586fc
f8f2fc98a1cd3eb8716689471812d52c460e3912
F20110114_AABAQT sauer_b_Page_047.tif
d185c1529fa2dd0755fa98bce69b8086
81dc7c966bcb84e4e310c0a368b69724c50cb1d2
F20110114_AABARI sauer_b_Page_065.tif
f2ee82384ee0c79157596cbc97c71b65
db1c62c10efaa8c42114bc20facdfa0d0763f639
F20110114_AABAQU sauer_b_Page_048.tif
17e0b6a8b987a02ea0b19c54e62da0ad
37c92a35f7019b68a233b059e145618e4f3d6b8c
F20110114_AABARJ sauer_b_Page_066.tif
fc2b4b1acec2a239505211746bb332db
0a06ce25302d828a698b6fbf306bf3ef07a8e589
F20110114_AABAQV sauer_b_Page_049.tif
9436fd176018881d828b81ec35bab6cd
89d0ac8a04440a6d34a94039b68d39563cfc1a19
F20110114_AABARK sauer_b_Page_067.tif
2a8fd5d99ce10718b9390d13009e4c36
94b5a8a05d9a3881fef3a2d5c20736d43b1bf323
F20110114_AABAQW sauer_b_Page_050.tif
23c874654fb8d9382ea985ef1306993b
3158678b5d5b298d7a2a6ef596ddd026c4d4bee8
F20110114_AABARL sauer_b_Page_068.tif
e7203c5f4c6e62c3ab9a146091bbda89
2a0bdca63a1668c0f661b3ddb315057490b93fac
F20110114_AABAQX sauer_b_Page_051.tif
e865bc247016e1c36cad266f121d6000
802738bec4ce388fd56571cbc35b20ace381a7f1
F20110114_AABARM sauer_b_Page_069.tif
9c404efb5256b0f5f7347c06d917d697
d2c99c0498242be3fec1b60735f89fe0beaab775
F20110114_AABAQY sauer_b_Page_052.tif
26f94fbe31a9bfd21a4c2b93468c3993
b8faec553b2a72ca00fc23f63f5c85fa9ff7f0a0
F20110114_AABASA sauer_b_Page_089.tif
8d3014ec298d491bfbbf038eeaf6dc03
db1cc1cd7f0f553e6677a01fd5a8112ae6f3afda
F20110114_AABARN sauer_b_Page_070.tif
b2c85364e19a2a3665937037cc607c77
13d9921837dc135136226324019b94a3ee40b84d
F20110114_AABAQZ sauer_b_Page_053.tif
ab273f67e802634a9d91b8f0a9bd7849
70146661706b08ff988fb6767f0d92623aec4b51
F20110114_AABASB sauer_b_Page_090.tif
4259638c3e8cf21b5a1cb5165b413af5
e83d7aadce5fa4eaa78b46fca2e0190b2d58bfa7
F20110114_AABARO sauer_b_Page_071.tif
5d5b4dc1388760b4226a2c9e3a1338d4
16b95ec3e8dc76bc0301af1873d188d925091312
F20110114_AABASC sauer_b_Page_091.tif
766f1e5c52ba10d51a44ef99eab5f70f
6c08d3f82943d61cae2ac5690e0610bd5e044da9
F20110114_AABASD sauer_b_Page_092.tif
476b0403c6de63297721132ea9b2ee9e
0cc6f818852131d0bb513975e19fb6d5eb84d3ba
F20110114_AABARP sauer_b_Page_074.tif
78c4088b0b0e33942396681d034d1949
8e0f816c23c9d20e26d7b1a05a38b7ed0e0b2dcb
F20110114_AABASE sauer_b_Page_096.tif
53c2dff0e5dfc6a9002d251df98b9d4b
67b430fc5c07144e7da8248e9cdd57ec1a21d3f0
F20110114_AABARQ sauer_b_Page_075.tif
9271febba5beb95023349252c5c5b7ab
a47cee244492c4b6602e1e5860366f392f0879fe
F20110114_AABASF sauer_b_Page_097.tif
9fe9279f090c11996fe9004f47242ab9
b7f199557e5d5a499c71be9cb262459a838c7579
F20110114_AABARR sauer_b_Page_077.tif
387a2b602c10bb4a1dce64c64e2e29ec
665d36ce203b6d54b0d4dd8de5d1ec9dd4ec990e
F20110114_AABASG sauer_b_Page_098.tif
1fa1aa1f09956517ab2b697e33ae392e
52f2fd4e7765fe1e4b30189d70ef9948b4386cf5
F20110114_AABARS sauer_b_Page_078.tif
4a158d7fa7f5982f004b001ace06f4b5
f89ca0361ab2d9b9d74fb7f4ad7c97c2301b4c36
F20110114_AABASH sauer_b_Page_099.tif
4b7d7ce2a0bf242825d217016f72c7a1
d619e5413d6e55015107e49761efaba073d9236b
F20110114_AABART sauer_b_Page_079.tif
01954f80eb0b67e4b5b0f8d762be70ff
d942bf901fd0378a591ab375071d31463f4655de



PAGE 1

MEDICATION USE PERFORMANCE INDICATOR EVALUATION: A SYSTEMS PERSPECTIVE By BRIAN C. SAUER A DISSERTATION PRESENTED TO THE GRADUATE SCHOOL OF THE UNIVERSITY OF FLOR IDA IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY UNIVERSITY OF FLORIDA 2004

PAGE 2

Copyright 2004 by Brian C. Sauer

PAGE 3

To my parents

PAGE 4

ACKNOWLEDGMENTS I would like to acknowledge my dissertation chair, Dr. Charles D. Hepler, for taking me under his wings and sharing his knowledge with me. I have grown tremendously over the past five years and Dr. Hepler has guided that process. I thank the members of my supervising committee, Drs. Earlene Lipowski, Abraham Hartzema, Murray Cote, and Richard Segal for their patience and guidance through the process. I would like to thank Josue Rodas, Becky Cherney and Scott Langdon for providing the support needed to make this study happen. I thank the Perry Foote foundation for providing the grant that financed this project. I thank Dave Angaran for listening and helping me understand and think through the clinical complexity of my dissertation. I would also like to thank Dr. Alan Spector for giving me the opportunity to work in his laboratory as an undergraduate in biological psychology. This was an important time in my academic development. Like Dr. Spector, I hope I will always be able to roll-up my sleeves and loosen my tie when the critics are throwing heat. I also would like to thank Mircea Garcea for teaching me, through example, to always pay attention to the details and produce quality work. Finally, I would like to thank the graduate students for producing a healthy working environment and their friendship. iv

PAGE 5

TABLE OF CONTENTS page ACKNOWLEDGMENTS .................................................................................................iv LIST OF TABLES ...........................................................................................................viii LIST OF FIGURES .............................................................................................................x ABSTRACT .......................................................................................................................xi CHAPTER 1 INTRODUCTION........................................................................................................1 Problem Statement........................................................................................................1 Objective.......................................................................................................................2 Specific Aims................................................................................................................2 Justification...................................................................................................................2 Overview.......................................................................................................................3 2 CONCEPTUAL FRAMEWORK.................................................................................6 Adverse Outcomes of Drug Therapy............................................................................6 The Medication Use System.........................................................................................8 Levels of the Health System.........................................................................................9 Human Error and System Failure...............................................................................14 Summary.....................................................................................................................16 3 LITERATURE REVIEW...........................................................................................18 Use of Administrative Databases for Researching PDRM.........................................18 Automated Methods for Detecting PDRM.................................................................22 Delphi Method............................................................................................................29 Cause-and-Effect Analysis.........................................................................................35 4 PRELIMINARY WORK............................................................................................39 Prevalence of Preventable Drug Related Admissions................................................39 Drug Categories Involved in PDRA...........................................................................43 Apparent/Proximate Causes of PDRAs......................................................................48 v

PAGE 6

Latent Causes of PDRA..............................................................................................52 5 METHODS.................................................................................................................55 Specific Aims..............................................................................................................55 Step One: Database Analysis......................................................................................55 Claims Data Types..............................................................................................55 Descriptive Analysis to Evaluate the Integrity of Claims Data...........................56 Population Demographic.....................................................................................56 MU-PI Coding Concepts and Analysis...............................................................57 Search algorithms.........................................................................................57 Disease-drug interaction...............................................................................57 Drug monitoring...........................................................................................58 Drug-drug interaction...................................................................................60 Prevalence Estimates for MU-PI Positives..........................................................60 Population Based Explanations for Prevalence Findings....................................61 Step Two: Node Identification of PDRM Scenarios..................................................63 Pilot Testing.........................................................................................................63 Selection of PDRM Scenarios for Node Identification Study.............................64 Delphi Recruitment.............................................................................................64 Node Identification Survey..................................................................................65 Delphi Process.....................................................................................................65 Node Identification Analysis:..............................................................................66 Selection of MU-PIs for Evaluation....................................................................66 Step Three: Cause-and-Effect Analysis......................................................................67 MU-PI Evaluation Team.....................................................................................67 MU-PI Evaluation Process..................................................................................67 Analysis for Common and Unique Cause Sequences.........................................70 6 RESULTS...................................................................................................................71 Step One. Database Analysis:....................................................................................71 Descriptive Analysis to Evaluate the Integrity of the Claims Database..............71 Check for missing and invalid data..............................................................72 Links between data types.............................................................................72 Population Demographics...................................................................................73 Prevalence of MU-PI Positives...........................................................................75 System Based Explanations for Prevalence Findings.........................................77 Distribution of monitoring intervals.............................................................78 Variables associated with PDRM positives.................................................84 Step Two. Assignment of MU-PIs to Nodes of the MUS.........................................86 Delphi Recruitment.............................................................................................86 Assignment of MU-PIs to Nodes of the MUS.....................................................87 Step Three: Cause-and-Effect Analysis......................................................................89 Evaluation of Monitoring Indicator.....................................................................91 Evaluation of Prescribing Indicator.....................................................................92 Cause Sequences Common to both the Monitoring and Prescribing Nodes.......94 vi

PAGE 7

7 DISCUSSION.............................................................................................................97 Database Analysis.......................................................................................................97 Prevalence Findings.............................................................................................97 System Explanations for Prevalence Findings....................................................99 Distribution of monitoring intervals.............................................................99 Multiple logistic regression analysis..........................................................100 Node Identification of MU-PIs.................................................................................102 Cause-and-Effect Analysis.......................................................................................103 Cause themes..............................................................................................104 Conclusion................................................................................................................106 Limitations................................................................................................................108 Significance and Theoretical Contribution...............................................................109 Contribution to Health Care......................................................................................110 Future Areas for Study..............................................................................................111 Interventions......................................................................................................111 MU-PI Instrument Fidelity................................................................................113 APPENDIX A MEDICATION INVOLVED IN DRA BY STUDY................................................115 B CLASSIFICATION OF DTPS INTO NODES OF THE MUS................................116 C SUMMARY: CLASSIFICATION OF DTPS INTO NODES OF THE MUS.........118 D MEDICATION USE PERFORMANCE INDICATOR DEFINITIONS................119 E PROCEDURE CODES TO IDENTIFIY VISITS...................................................126 F CODING SOLUTION FOR INDICATOR..............................................................127 G RECRUITMENT LETTER FOR NODE IDENTIFICATION STUDY..................128 H NODE IDENTIFICATION SURVEY.....................................................................130 I EVALUATION TEAM BRIEFING.........................................................................141 J MU-PI RESULTS.....................................................................................................147 K SURVEY FOR COMMONALITY..........................................................................149 L NODE IDENTIFICATION BOX PLOTS................................................................150 LIST OF REFERENCES.................................................................................................153 BIOGRAPHICAL SKETCH...........................................................................................159 vii

PAGE 8

LIST OF TABLES Table page 3.1 Examples of DRM Screens by Type........................................................................25 3.2 Results of Leading DRM Screens............................................................................25 3.3 Results of DRM Screens..........................................................................................26 3.4 Results of Top Five Indicators.................................................................................30 3.5 Percent Change in Average Group Error by Number of Group Members...............32 4.1 Studies Reporting DRA and PDRA: .......................................................................44 4.2 Prevalence Estimates and Odds Ratio .....................................................................45 4.3 Latent Causes of PDRAs..........................................................................................54 6.1 Population Age and Gender Frequencies by Type of Claim....................................74 6.2 Population Demographics........................................................................................76 6.3 Frequency of Process and PDRM Positives.............................................................77 6.4 Number of PDRM Positives by Number of Indicators............................................77 6.5 Eleven Most Prevalent PDRM Positives..................................................................78 6.6 Ten Most Prevalent Process Positives......................................................................78 6.7 MU-PI Categorized by Required Monitoring Interval.............................................79 6.8 Bivariate Tests of Association: PDRM Positives as Dependent Variable...............85 6.9 Multicollinearity Assessment of Independent Variables.........................................86 6.10 Maximum likelihood Estimates: PDRM Positives as Dependent Variable.............86 6.11 Demographics of Delphi Panelists for Node Identification Study...........................87 6.12 Indicators Listed by Associated Node of the MUS..................................................89 viii

PAGE 9

6.13 Kruskal Wallis and Pairwise Comparisons for Indicator Assignment.....................90 6.14 Monitoring Indicator: Affinity Table and Rating of Importance.............................92 6.15 Cause Sequence Agreement Results........................................................................93 6.16 Prescribing Indicator: Affinity Table and Rating of Importance.............................94 ix

PAGE 10

LIST OF FIGURES Figure page 2.1 Model of the Medication Use System......................................................................14 2.2 Hierarchal Relationships among the System Levels................................................15 2.3 Swiss Cheese Model of System Failure...................................................................17 4.1 Involvement of Treatment Category to DRMs:.......................................................49 4.2 Therapeutic Class Involvement in DRM:.................................................................49 4.3 Nodes of the MUS Involved in PDRA.....................................................................52 5.1 Disease-Drug Interaction Search Algorithm............................................................58 5.2 Process Search Algorithm for Indicators that Require Monitoring..........................59 5.3 Process and Outcome Search Algorithm..................................................................59 5.4 Example of a Process Positive but not a PDRM Positive........................................60 5.5 Drug-Drug Interaction Search Algorithm................................................................60 5.6 Organization of Proposed Causes to System Levels................................................69 6.1 Office Visits Claims by Month................................................................................72 6.2 ED Visit Claims by Month.......................................................................................72 6.3 Facility Admissions by Month.................................................................................73 6.4 Pharmacy Claims by Month.....................................................................................73 6.5 First Lag for Indicator with One Month Monitoring Requirement..........................80 6.6 First Lag for Indicator with Two Month Monitoring Requirement.........................81 6.7 First Lag for Indicator with Three Month Monitoring Requirement.......................82 6.8 First Lag for Indicator with Six Month Monitoring Requirement...........................82 x

PAGE 11

Abstract of Dissertation Presented to the Graduate School of the University of Florida in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy MEDICATION USE PERFORMANCE INDICATOR EVALUATION: A SYSTEMS PERSPECTIVE By BRIAN C. SAUER August, 2004 Chair: Charles Doug Hepler Major Department: Pharmacy Health Care Administration Background: Injury related to health care is a serious public health concern. The prevalence of drug related admissions in the United States has been reported to be from three to twelve percent of hospital admissions. Organizations interested in reducing drug related morbidity need measurement techniques to gather baseline information about preventable drug related morbidity (PDRMs), and methods to assess system related causes to develop rational interventions that target the system failures The objective of this dissertation is to better understand the relationship between system design and patterns of care that can result in drug related injury. Methods: This study was executed in three steps. In the first step, database analysis, medication use performance indicators (MU-PI) were used to estimate the number of PDRM positives in the managed care organizations administrative database. Step two used the Delphi process to judge the degree of association between select MU-PIs and specific nodes of the medication use system (MUS) where the process failure xi

PAGE 12

may have originated. In step three, select MU-PIs from each node of the MUS were submitted to cause-and-effect analysis (CEA). Results: The period prevalence for process positives was 209.84 (206.37-213.34) per 1,000 member years. The period prevalence for PDRM positives was 1.98 (1.6-2.4) per 1,000 member years. Gender, number of office visits, number of drug classes and number of medical conditions were independent risk factors for PDRM. Fourteen indicators were selected for step two. Six of the fourteen indicators were assigned to the prescribing node and seven were assigned to the monitoring node of the MUS. One indicator did not reach significance within two rounds. The cause-and-effect team identified twenty-nine cause sequences. They found that twenty-three of the twenty-nine were common to both prescribing problems and monitoring problems. Conclusion: The MU-PIs proved to be a useful tool to identify possible cases of PDRM and to initiate system thinking or organizational introspection to evaluate system related causes for drug related injury. Four themes can be interpreted from CEA: lack of necessary tools for adequate patient information and assessment, an information system that can track patients and relay information to the providers, pharmacist involvement in the MUS, and guideline adherence. xii

PAGE 13

CHAPTER 1 INTRODUCTION Problem Statement Injury related to health care is a serious public health concern. The prevalence of drug related admissions in the United States has been reported to be from three to twelve percent of hospital admissions. 1 The Institute of Medicine has proposed a fifty percent reduction in errors by 2005. 2 For this achievement in medications use, the following are needed: Measurement techniques to gather baseline information about preventable drug related morbidity (PDRMs) Assessment of system related causes and rational interventions that target the system failures Follow-up measurements to gauge the effects of the interventions. Automated data screening for PDRM has been proposed as a method to produce baseline and longitudinal measurements. However, the underlying system problems that contribute to PDRM in the ambulatory setting are largely unknown. Studies measuring the prevalence of drug-related hospital admissions have identified nodes in the medication use process where the drug therapy problems originated, i.e., prescribing, monitoring, dispensing and patient adherence. The underlying system related causes of node specific problems, however, have not systematically been evaluated. This research will try to discover the system factors that contribute to node specific problems. 1

PAGE 14

2 Objective The objective of this dissertation is to better understand the relationship between system design and patterns of care that can result in drug related morbidity. Specific Aims The specific aims of this study are as follows: 1. To establish the frequency of medication use performance indicator (MU-PI) positives in the study population 2. To evaluate population based explanations for MU-PI positives 3. To identify the node of the medications use system (MUS) where indicator specific patterns of care likely originated 4. To identify system related causes that are common among nodes of the MUS 5. To identify system related causes that are unique to nodes of the MUS Justification This study has both theoretical and practical implications. The theoretical contribution of this study includes: Determining the node of the MUS where the pattern of care from specific MU-PIs originated. This permits the evaluation of the indictors by processes rather than medication or disease specific analysis. Providing information about how the various subsystems (patient, microsystem, organization, and environment) within the health care enterprise interact to influence the quality of medication use. Establishing system factors that contribute to node specific patterns of care or process problems. Identifying system factors that appear to be node specific problems will reveal leverage points for interventions that may span medication classes and disease categories. Establishing system factors that contribute to patterns of care or process problems from multiple nodes of the MUS. Identifying system factors that appear to be common among the nodes of the MUS will reveal leverage points for interventions that may span nodes of the MUS, medication classes and disease categories. This study will demonstrate how the MU-PIs can be used to assess the quality of medication use in a defined population. It will also demonstrate how the MU-PI findings

PAGE 15

3 can be used to initiate activities that lead to the exploration of how system design influences the quality of medication use, i.e., organizational introspection. From a practical perspective, the health care purchasing group involved in this study is interested in increasing the quality of care and decreasing costs for the purchasers they represent. Understanding how system factors influence the quality of medication use will help direct the development of interventions to reduce the prevalence of adverse drug outcomes. Overview The health care industry is wasting billions of dollars 3 and causing an unacceptable amount of injury due to inappropriate medications use. 1,2,4 Results from our meta-analysis of fifteen studies indicated that approximately four percent of all hospital admissions may be preventable. 1 To place the reported prevalence estimate into perspective, according to the National Hospital Discharge Survey, there were 31,827,000 admissions to U.S. hospitals in 1998. If the prevalence of preventable drug-related hospital admissions (PDRAs) happened to have been four percent in 1998, there would have been roughly 1.3 million PDRAs. This would have placed PDRAs among the top causes of hospitalizations in the U.S. that year-above admissions related to congestive heart failure (3.1%) and on par with pneumonia (4.2%). Ambulatory health care is complex, with many opportunities for problems in drug therapy to occur. The basic nodes in the MUS include prescribing, dispensing, (self) administration, and monitoring. Of the fifteen studies analyzed in the above-mentioned meta-analysis, 1 eight described the types of drug therapy problems that led to PDRA. 5-12

PAGE 16

4 An analysis of these studies a indicated that prescribing and monitoring problems were highly associated with PDRA (median: 34% and 32%), respectively. Problems with medications compliance (median: 22%) were also implicated as a main contributor to PDRA. The finding that prescribing and monitoring problems appeared to be the main contributors to PDRA is disturbing because health care professionals are expected to first do no harm. Even though practitioners are expected to manage their patients appropriately and do it right the first time, the health care system should be designed in a way that anticipates human error and does not allow it to progress to an adverse drug event or PDRM. Ideally, the health care enterprise should be able to deliver care in a highly reliable manner where performance problems and bad decisions are recognized and corrected because adverse events occur. Results from the Institute of Medicine report on medical errors 2 and from our meta-analysis 1 show that the American health care system is currently far from highly reliable. Before the rational development of interventions, a data stream must be established for baseline and follow-up measurements, and underlying system related causes need to be examined. A recent doctoral dissertation at the University of Florida used a Delphi approach to validate forty nine MU-PIs; each paired an inappropriate process (violation of a standard of care) with an adverse outcome. 13 An example of these indicators follows: Use of two or more NSAIDS concurrently for at least two weeks in patients 65 or older followed by gastritis and/or upper GI bleed. a The analysis can be found in Chapter 4 (preliminary analysis)

PAGE 17

5 The indicators were automated and used to screen an administrative database of a large MCO in North Florida. The data set contained 11,711 patients, and 8.2% of them screened positive for a PDRM. For further validation, the positive predictive value of process to outcome (PTOV) was calculated to measure the strength of the association between the inappropriate process and the corresponding adverse outcome for each indicator. Ten of the nineteen indicators that had more than ten positive screens had PTOV values greater that 0.74. This suggests that this instrument would be acceptable for eliciting cases of PDRMs for system level analysis. Cause-and-effect analysis is a retrospective method for identifying system failures, and its popularity has largely advanced from its success in understanding industrial accidents. 14 Rooted in industrial psychology and human factors engineering, cause-and-effect analysis has recently been adopted in the medical community to evaluate quality problems. The MU-PIs adopted from Faris (2001) 13 will be used to screen the database of a health care coalition in Florida. MU-PIs with high frequency will be selected and assigned to the nodes of the MUS where the drug therapy problems appear to have originated from. Cause-and-effect diagramming on selected MU-PI that represent specific nodes of the MUS will be carried out to uncover system related problems that are common among the nodes and unique to specific nodes of the MUS.

PAGE 18

CHAPTER 2 CONCEPTUAL FRAMEWORK The framework of this study was developed from Heplers conceptualization of PDRM and the MUS, 15 Berwicks framework for the health care system 16 and Reasons accident theory. 17 The premise is drug-related morbidity is largely preventable due to failures in the management of drug-therapy. A model of the MUS is used to illustrate how drug-therapy problems (DTPs) can develop into PDRM. Health care is a complex system with many layers of embedded systems. A framework presented by Berwick 16 is used to simplify the complexity by referring to specific levels of the heath care system. Finally, James Reasons accident theory 17 is used to discuss the types of failures and conditions that affect the performance of the MUS and allow PDRM to occur. Adverse Outcomes of Drug Therapy The term drug-related morbidity (DRM) is used to discuss adverse outcomes of drug therapy. DRM is a concept that includes (a) significant adverse effects of drug therapy (e.g., hospital admissions or emergency department visits) (b) treatment failures (i.e., occasions when drug therapy was attempted but did not achieve a realistic, intended outcome in a reasonable time) and (c) occasions when a patient did not receive an indicated or necessary drug therapy. 18 It is important to appreciate the relationship between the popular term adverse-drug event (ADE), which was used to describe adverse outcomes of drug therapy in the IOM report To Err is Human, and DRM. The term ADE historically has been used to describe injury resulting from medical interventions related to a drug (i.e., direct effects of the therapy), 4,19-22 without recognition of indirect 6

PAGE 19

7 injury resulting from sub-therapeutic dose or no drug while a valid indication is present. Even though it does appear that the definition for an ADE has recently evolved to include indirect injury, 23 the term DRM is used in this study because it has historically treated indirect injury as a significant adverse outcome related to drug therapy. A DTP that precedes a DRM can be classified into two types: potential DTPs and actual DTPs. The first indicates a person has a theoretical problem associated with drug therapy, and the latter signifies the problem has manifested (symptoms are present). 15 Theoretical DTPs exist independently of individuals; they are the situations in care that produce the risk for specific adverse effects. Potential DTPs occur when a theoretical DTP is present in an individual, e.g., contra-indications, drug interactions and unjustified violations of evidence-based medicine. Actual DTPs are denoted by observable or patient reported symptoms. For example, concomitant prescribing of digoxin and quinadine is a known theoretical DTP because this combination can potentially alter the excretion of digoxin, thus increasing its serum concentration, which is especially troubling because digoxin is known to have a very narrow therapeutic window. Now imagine a patient sixty-five years old who is concurrently receiving digoxin and quinidine. This patient has a potential DTP because a theoretical DTP is present in his drug therapy. Now, suppose this patient begins to experience fatigue, weakness, confusion, and diarrhea. Most likely he is experiencing an actual DTP because predictable symptoms known to be associated with the theoretical DTP have manifested. If he/she becomes hypokalemic and needs emergency care, and upon admission his/her digoxin level is in toxic range, say > 50 mcg/kg, then he/she most likely would have experienced a preventable DRM (PDRM).

PAGE 20

8 A preventable DRM has the following attributes : 15,18 The DRM was preceded by a recognizable DTP The DRM was reasonably foreseeable under the circumstance The cause of the DTP and the resulting DRM was identifiable The identified cause of the DTP and resulting DRM was controllable within the context of therapy (i.e., without sacrificing essential therapeutic objectives). In this case the DRM meets the four criteria for preventability because it was preceded by a known drug interaction (potential DTP) and the manifest symptoms (actual DTP) were commonly associated with digoxin toxicity. Under the circumstances hospitalization was imminent because the symptoms represent the accumulation of digoxin, the cause of the symptoms appears to be the result of too much digoxin, and reducing the dose of digoxin and monitoring the patients digoxin levels or switching to an alternative therapy could have resolved the DTP and prevented the DRM. A DTP is part of the process of care, it is a state of an individual in a medications use system, and it is a possible precursor to a system failure, i.e., DRM. The Medication Use System For a DRM to be preventable a failure in the process of care must have occurred. This becomes clearer when considering the typical sequence of actions that comprise the MUS in the ambulatory care setting. Figure 2.1 below was adopted from Grainger-Rousseau et al. and it illustrates the nodes in the medications use system. 24 The episode of care begins when the patient notices a problem and seeks professional medical attention. Typically, the initiator (physician, physicians assistant or nurse practitioner) then assesses the problem and develops a clinical impression or diagnosis. Next a plan is devised and the decision to prescribe a medication is made. A prescription is either written or not; if it is, the patient will typically present the prescription for filling at the

PAGE 21

9 pharmacy. Before dispensing the medication the pharmacist should look for DTPs and advise the patient how to use and self their new therapy. The patient then consumes (e.g., self-administers) the medication. Follow-up visits and monitoring are required to gauge the effects of the medication in the individual and to determine how well the prescribed medication is working towards the therapeutic objective. It is the information gathered from the monitoring node that is used in the decision to continue the current treatment or make alterations to the medical regimen. Errors or problems at each step may occur during a passage through this sequence of events. Examples include failure to recognize an indication for drug therapy, incorrect patient assessment, incorrect diagnosis, prescribing, dispensing, and administration and monitoring. 25 Our meta-analysis (2002) estimated the prevalence of preventable drug related admissions to be about four percent. 1 Based on our review and the IOM report, it appears the current MUS are not optimal. This may be because MUS are more of a virtual than an actual microsystem. As described by Nelson et al., (1998) 26 an essential element for a microsystem is an information environment to support the work of care teams. Unfortunately, the MUS in most ambulatory care settings is fragmented and clearly lacks a reliable information exchange among providers, especially between initiator and co-therapists. The development and adoption of communication systems such as electronic medical records and computerized prescription order entry are intended to make the MUS safer. Levels of the Health System The IOM defines a system as a set of interdependent elements interacting to achieve a common aim. The elements may be human and/or non-human. 2 The health

PAGE 22

10 care enterprise consists of many embedded systems that transcend a multitude of domains, ranging from the subsystems used to make and keep track of patient appointments to the environment where accreditation and financial systems influence the care of populations. All systems have subsystems or nested systems that interact with one another. Some systems are in a hierarchical relationship in which higher order systems influence the functioning of lower order subsystems. Identifying the nesting of systems and their relationships to one another is key to understanding the mechanics of any system. Berwick (2002) published a paper that he described as a users manual for the IOM report, Crossing the Quality Chasm. 16 In this paper he addressed the issue of embedded systems by presenting a framework for the different levels within health care. He separates the system into the following levels: the experience of the patients and communities (Level A), the functioning of small units of care delivery called microsystems (Level B), the functioning of organizations that house or otherwise support microsystems (Level C) and the environment of policy, payment, regulation, accreditation and other factors (Level D). Level A is not actually framed in terms of a system; instead it represents the patients experience and perception of the care they received. Berwick states, Rooted in the experiences of patients as the fundamental source of quality, the report shows clearly that we should judge the quality of professional work, delivery systems, organization and policies first and only by the cascade of effects back to the individual patient and to the relief of suffering, the reduction of disability and the maintenance of health. 16 This is a significant paradigm shift from previous quality approaches that were independent from patient outcomes. Determining quality based on the patients experience is allied with using outcomes of care to gauge quality. Donabedian (1978)

PAGE 23

11 defines outcomes as the primary changes in health status that can be attributed to that care. 27 Health status is determined by psychological factors and social performance (subjective component), and physiological factors (objective component). In this paradigm, the patients experience is defined as the sole way of determining the quality of other systems. The decision to place emphasis on the patient is strategic; it acknowledges the patients subjective experience and clinical outcomes are the primary focus of quality rather than the dynamics of the microsystems. The microsystems (Level B) are small units of work that actually gives care the patient experiences. 16 Clinical microsystems are basically small organized groups of providers and staff caring for a defined population of patients. 28 They are composed of patients who interact with clinical and support staff who perform various roles, e.g., physician, nurse, pharmacist, medical assistant, data managers, receptionist, etc. Nelson et al. (1998) have described the essential elements of a microsystem as (a) a core team of healthcare professionals; (b) the defined population they care for; (c) an information environment to support the work of care team and patients; and (d) support staff, equipment and work environment. 26 In terms of medications management, the patients and clinical staff manage drug therapy by engaging in direct care processes, which include recognizing, assessing and diagnosing the patients problem, along with developing a treatment plan, dispensing and educating, and monitoring to make sure the treatment is progressing as planned. Direct care processes are assisted by supporting processes that involve distinct tools and resources such as medical records, scheduling, diagnostic tests, medications, billing, etc.

PAGE 24

12 To move to a more reliable system of care, the performance of microsystems must be optimized and the linkages between clinical microsystems must be seamless, timely, and efficient. Change at the microsystem level is an important opportunity to focus on the transformation of care at the front line of the health care service industry. 29 Microsystems do not exist in a vacuum; they are embedded within the organizations that help orchestrate their relationships with each other. The health care organizations (Level C) are establishments that house and support microsystems--they provide the necessary resources for microsystems to deliver care. Health care delivery requires personnel, financing, technology, and facilities. Common organizations include hospitals, provider groups, independent practice groups, nursing homes, ambulatory surgery centers, and pharmacy benefit managers, all of which are typically embedded within managed care organizations. Group practice, for example, is an affiliation of three or more providers, usually physicians, who share income, expenses, facilities, equipment, medical records, and support personnel in the provision of services, through a legally constituted organization. 30 They are embedded within managed care organizations through such integrating mechanisms as referral arrangements, insurance contracts, and in some cases direct ownership of practice. Managed care is focused on controlling the functioning of microsystems, i.e., controlling provider and patient behaviors, and most of that control is realized through the ambulatory care arena. 30 Managed care has shifted toward an organizational form with greater control over resources, providers, and patients within microsystems of care. The health care environment (level D) includes multiple systems that influence activities of the organizations and microsystems. Important environmental systems

PAGE 25

13 include financing, regulation, industry, accreditation, policy, litigation, professional education and social policy. 16 Berwick (2002) describes the relationship among the four levels as the chain of effect of improving health care quality. He states that, The quality of the microsystem is its ability to achieve ever better care: safe, effective, patient-centered, timely, efficient and equitable. The quality of the organization is its capacity to help microsystems produce safe, effective and efficient patient outcomes. And the quality of the environment--finance, regulation, and professional education is its ability to support organizations that can help microsystems achieve those aims. 16 Figure 2.2 was designed to show the hierarchal relationship among the different levels of health care, as well as the MUSs location within the microsystems. As illustrated, the MUS intersects the microsystems and the patients experience. The spiral between the microsystems and the patient level indicates that the patients experience is the input into the MUS as well as the output, which is then used as feedback to determine the performance of the microsystems. What was more difficult to illustrate is the interaction among the microsystems. In an ideal MUS the interaction between physicians, laboratory, pharmacists, case managers, etc., would be seamless with information flowing freely among the microsystems and cooperation among the professionals would be standard. The hierarchical stacking from environment to organizations to microsystems to the patient represents the main route of influence. It should be noted that this is not the only direction of influence, organizations can use lobbyists to leverage influence over the regulatory and financial environment. Likewise, microsystems can organize, like in the case of provider groups, to counter the pressures from higher-up organizations such as managed care.

PAGE 26

14 Figure 2.1. Model of the Medication Use System Human Error and System Failure James Reason provides two views of how accidents occur within an organization the person approach and the system approach. 17 These approaches are fundamentally different, with each giving rise to contrasting philosophies of performance management. The person approach (human error) focuses on unsafe acts. In medications use, the attention would be on errors and procedural violations of health professionals and patients. This approach views unsafe acts as divergent cognitive and behavioral processes such as forgetfulness, inattention, carelessness, negligence and recklessness. Historically, the most popular approach to countering human error has been to find the operator or operators who committed the unsafe act and discipline them accordingly. This is often referred to as blaming, and sometimes as scapegoating. Even though blaming may produce satisfaction it does nothing to correct the conditions that allowed unsafe acts to progress to injury. Because errors are often made in the normal course of providing care, traditional efforts at error reduction, which focused on individuals and

PAGE 27

15 episodes by using training, rules and sanctions to improve performance, are considered less effective than altering the system to remove or reduce the conditions that increase the likelihood of adverse outcomes. 31 Counter measures for reducing unwanted behaviors are analogous to the Whack-A-Mole carnival game: you can whack a mole back into his hole, but another is sure to challenge your skill by popping-up in a different location. Figure 2.2. Hierarchal Relationships among the System Levels The basic principle in the system approach is humans by their nature are fallible and errors are expected even in the best organizations. From this perspective, injury occurs because errors interact with flaws in the design of systems called latent conditions. 17

PAGE 28

16 Latent conditions are the flaws in the design and organization of a system. In health care they can arise from the decisions made by designers; however, it is believed many latent conditions arise not from design but from self organization, which is an adaptation to evolving systems and an unsettled environment. 32 Latent conditions have two types of adverse effects: they can produce error-provoking conditions and they can produce longstanding holes or weaknesses in the defense mechanisms, i.e., error detection mechanisms. When latent conditions combine with errors the opportunity for PDRM occurs. In the Swiss cheese model b presented in Figure 2.3, the holes in the cheese are latent conditions that allow potential injury (i.e., drug therapy problems) to manifest as drug related morbidity. Summary PDRM is a system problem. The nature of preventability means problems occurred in the process of care. Patient injury occurs when human error interacts with latent conditions of a system. The health care system is a complex network of interacting systems and subsystems. Understanding how microsystems, organizations and environment influence activities of the MUS and produce opportunities for human error to result in PDRM will increase our ability to predict and resolve DTP before they manifest as patient injury. b This Swiss cheese was published by Hepler and it is a modification of Reasons Swiss cheese model 15

PAGE 29

17 Patient injury Figure 2.3. Swiss Cheese Model of System Failure

PAGE 30

CHAPTER 3 LITERATURE REVIEW This chapter will first discuss the use administrative data for scientific investigation. The use of automated methods for detecting the incidence and/or prevalence of PDRM will then be discussed. Next, experimental studies evaluating the effectiveness of the Delphi process and its use in health care will be presented. Lastly, techniques for cause-and-effect analysis will be presented. Use of Administrative Databases for Researching PDRM Administrative databases are derived from information produced by health care providers and institutions in billing for products and services. Claims are filed from institutions (i.e., inpatient hospital stays and outpatient visits) health professionals and pharmacies for reimbursement of products and services from payer organizations such as Medicaid, Medicare and private insurers. The claim form used depends on where services were rendered and who provided them. Most hospital inpatients and acute care outpatient services are submitted for payment with the uniform billing 92 (UB-92) format. The UB-92 was derived from the Uniform Hospital Discharge Data Set, which was formulated in 1972 by the National Committee on Vital and Health Statistics, U.S. Department of Health, Education, and Welfare. The goal was to create a uniform but minimum data set to facilitate investigation of cost and quality of hospital services across populations. 33 Health care services provided by single practitioners or practitioner groups are submitted for payment in a format called the CMS-1500 (formally the HCFA-1500). This is the common claim form for non-institutional providers that is updated by the 18

PAGE 31

19 Center for Medicare Services (CMS) and is approved by the American Medical Association Council on Medical Services. Both the UB-92 and the CMS-1500 forms typically include details for each compensated service, including diagnosis, and procedures, date, place of service, provider and patient identifiers and charges. From a research and quality improvement perspective, the potential utility and credibility of a database stems from its clinical content. In administrative data in the United States, information on clinical diagnosis and conditions are documented with International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) diagnostic codes. In addition, administrative data derived from hospital reports (UB-92) use ICD-9-CM codes to document procedures, while procedures are documented using Common Procedure Terminology (CPT) in data derived from professional claims (CMS-1500). The reliability and validity of coding is a serious issue when considering the use of administrative data for research. In relation to hospital admissions, the two major steps in coding an admission are first specifying the pertinent diagnosis and second ordering them. The principal diagnosis in a UB-92 form defines the cause of admission. The sequencing of diagnosis does impact reimbursement rates and mistakes or miscoding can affect the accuracy of the data. As previously mentioned, the validity of claims data is a cardinal issue when considering its use for health services research. Threats to validity and two approaches for evaluating the ability of claims data to accurately identify patients with specific conditions will now be presented. The following threats to internal validity (i.e., misclassification bias) have been identified misspecification, resequencing, miscoding,

PAGE 32

20 and clerical errors. 33,34 Misspecification occurs when the attending physician selects an incorrect diagnostic code for the principal diagnosis, listing of diagnoses or procedures. Misspecification was identified as the leading source of error in a summary of studies that evaluated coding quality. 34 Resequencing is the substitution of a secondary diagnosis for the correct principal diagnosis during the coding process and was found to be the second most frequent source of misclassification. 34 Miscoding is the coding of diagnoses or procedures not attested to by the physician, misapplication of coding rules, or selection of an unnecessarily vague diagnosis code. Now that the types of misclassification bias have been presented, studies that evaluated the validity of claims for research data will be discussed. Quam et al. (1993) investigated the validity of claims data for epidemiologic research and found evidence that supports its use. 35 The researchers evaluated the ability of claims data to identify patients with essential hypertension. The claims database used consisted of all ambulatory, pharmacy and hospital claims from 1988 to 1989 in two large managed care organizations. They identified essential hypertensive patients by three strategies. One strategy was based on medical service claims alone (diagnosis without pharmacy claims), another was based on only pharmacy claims (pharmacy claims without diagnosis), and the third strategy was based on both medical services and pharmacy claims. Diagnostic codes were validated by comparing findings with patient survey and medical records. The strategies based on medical claims alone and pharmacy claims alone exhibited low positive predicted values (PPV) with the medical record (47% and 50%, respectively) and the patient survey (43% and 63%, respectively). 35 The

PAGE 33

21 combination strategy, however, exhibited very high PPVs (96% for both the medial record and patient survey). 35 More recent information on the ability of administrative data to predict disease outcomes is described in the Study of Clinically Relevant Indicators for Pharmacological Therapy (SCRIPT) report. 36 The objective of the SCRIPT project was to develop a core set of valid and reliable performance measures to evaluate and improve the quality of medication use. The common focus was cardiovascular disease outcomes and risk factors. The candidate measures were tested in managed care organizations and practice groups from eight states. The study evaluated patients with coronary artery disease (CAD), heart failure (CHF), and atrial fibrillation (AFIB). The researchers evaluated the PPV of specific ICD-9CM and CPT codes for identifying the cardiovascular conditions mentioned above. The requirement for the number of codes to identify specific conditions was varied from one to three and the effects on yield and PPV were determined. The PPVs from the SCRIPT study averaged 89%, 89%, 84% for AFIB, CAD and CHF, respectively. The state-specific values for AFIB ranged from 80% to 98%, the values for CAD ranged from 81% to 95% and the values for CHF ranged from 78% to 94%. Requiring an additional code raised the PPV no more than four percentage points on average and raised the worst PPV as much as eight percentage points. Requiring three codes added little to PPV; however, it significantly reduced the yield. The most marginal improvement in PPVs was found by requiring two codes vs. three codes. The two code requirement lowered the yields by 13%, 14%, and 20%, respectively for AFIB CAD and CHF

PAGE 34

22 In summary, although administrative databases were not originally designed for research, they have become a rich source of information for health services research. 33 Nevertheless, threats to internal validity, specifically misclassification bias need to be taken seriously when interpreting studies that used claims data. Studies that have evaluated the ability to identify patients with specific conditions using administrative data had good positive predictive ability. Quam et al. (1993) reached a PPV of 96% when requiring medications used to treat essential hypertension and a diagnosis of essential hypertension. 35 The SCRIPT project was able to obtain good PPV while maintaining a decent yield when they required two codes (ICD-9-CM or CPT) for specific cardiovascular conditions. 36 Automated Methods for Detecting PDRM Chart review is the gold standard for PDRM ascertainment. The articles reviewed by Winterstein et al. (2002) 1 (discussed in the following chapter) used medical chart review to measure the prevalence of drug related admissions (DRAs). This typically included reviewing the medical records of all patients admitted to the hospital or specific hospital units during a defined period of time to determine if the reason for admission was drug related. The detection and classification method was often implicit, which means the reviewers judged whether or not a DRA occurred by comparing the care processes of that patient against his or her own knowledge, opinions, and beliefs about how appropriate care should be carried out. Implicit methods may be more sensitive because they allow experts to capture case-specific elements of information for the judgment of preventability, however, their reliability is questionable. 37 Furthermore, chart-review is time and cost consuming and is not useful in the continuous improvement

PAGE 35

23 paradigm where baseline and many follow-up measures are needed. Computerized methods to identify DRM with explicit criteria are an alternative to chart-review. 38,39 A PubMed search identified three studies with different approaches for detecting ambulatory acquired DRM with computerized data. One method identified DRM with search algorithms used to screen laboratory data and pharmacy records. Another method used multiple types of search algorithms and databases to identify cases of ADEs. The last method screened administrative claims data by using automated PDRM performance indicators. These studies will be presented and special attention is given to the following issues: the predictive validity of the instrument, the ability of the indicators to capture preventable events, the scope of the database, and the sophistication of the software used to run the indicators. Jha et al. (2001) 39 adapted published screening rules (i.e., search algorithms) developed from the inpatient LDS study in Utah. 40 The rules were used to estimate the rate of DRA to the study hospital. Every day the computer generated a list of alerts. The alerts were validated by reviewing each patients medical record that screened positive for a DRM on admission. Jha et als. (2001) screening rules can be separated into at least four categories: antidotes, chemistry/drug levels, and physiological response to a specific therapy and drug interaction. (Table 3.1) The information system used integrated pharmacy, laboratory test results and a sophisticated physician order entry systems. Twenty two rules required laboratory test results and nineteen screened for antidotes of known adverse consequences of drug therapy (e.g., naloxone, a narcotic antagonist).

PAGE 36

24 Table 3.2 shows the results of the most frequent screens. The overall PPV for the instrument was 3.5% with a range from 2.0 to 100%. The screen for serum phenobarbital levels < 45 mg/dl had a PPV of 100%. The screen for patients receiving charcoal (activated) had the second highest PPV of 45%. All other screens had PPVs below 12%. The overall PPV for this instrument and the individual screens PPVs, excluding phenobarbital levels, indicated that these screens picked up a tremendous amount of noise. Furthermore, the majority of the indicators were directed toward toxic levels of medications and abnormal blood chemistry levels. While these surrogate outcomes may be able to indicate the presence of DRM, they do not provide information about the care process that led to the event, i.e., preventability. Additional chart review or investigative work is needed to judge preventability. Furthermore, the software used to search the data was quite sophisticated (physician order entry and an event monitor). Nevertheless, less sophisticated software such as SAS or Access could be used to write search algorithms for the screens. Honigman et al. (2001) took a more elaborate approach for using computerized information to screen for ADEs. 41 This approach used a sophisticated computer program which consisted of four search methods: ICD-9 codes, allergy rules, computer event monitoring rules, and an automated chart review using text searching. ICD-9-CM codes associated with ADEs were used to screen the database. The computer event monitoring rules were based on the same screening criteria as Jha et al. (2001) mentioned above. 39,40 Allergic reactions to medications were detected with a text search of the medical records. The software program M 2 D 2 was used to expand the patients drug allergy list to include product names, generics and ingredient. Patient

PAGE 37

25 records were screened for allergies along with offending medications. The M 2 D 2 program also matched terms in the medical record with known adverse effects of drugs the patient was taking. Table 3.1. Examples of DRM Screens by Type (Jha et al. 2001) AntidotesChemistry and Blood LevelsResponse to Drug and InteractionsReceiving betamethasone di p ro p ionate 0.05%Serum digoxin > 1.7 ng/mLReceiving nephrotoxin AND blood creatinine has risen > 0.5 m g /dL in last 1 da y Receiving charcoal ( activated ) Serum lidocaine > 5.0 mg/mLReceiving ranitidine AND platelet count has fallen to less than 50% of p revious valueReceiving racemic e p ine p hrine hclSerum phenytoin results within last 1 da y are > 20 m g /mLReceiving diphenoxylate with atropineReceiving atropine sulfateSerum bilirubin > 10 mg/dLReceiving benzodiazepine AND receiving anti-e p ile p ticReceiving naloxoneSerum potassium > 6.5 mmol/LReceiving phytonadione (vitamin K) AND order for warfarin within last 14 days Table 3.2. Results of Leading DRM Screens (Jha et al. 2001) DRM screening ruleScreen +True +PPV (%)Patient receiving predinisone313123.8Patient receiving diphenhydramine30093Allergy entered113108.8Patient receiving oral matronidazole/vancomycin8766.9Serum digoxin > 2.0 ng/ml6158.2Serum phenytoin > 20mg/dl4748.5Patient receiving kaopectate35411.4Patient receiving charcoal11545.5Serum phenobarbital >45 mg/dl44100Other1,649332Total2,620923.5 Honigman et al. (2001) 41 found allergy screens to have the highest PPV (49%), but these screens did not detect the greatest number of ADEs (104). Text searching detected the most ADEs (1,637) but it had many false positives (PPV = 7%). Searching for only ICD-9-CM codes had the lowest PPV (2%) and it only detected five events. The event monitoring rules detected 60 events with a PPV of 3.3%. The PPV for the composite tool was 7.2%. (Table 3.3)

PAGE 38

26 Table 3.3. Results of DRM Screens ICD-9-CM24852Event Monitoring Rules1,802603.3Allergy21410448.6Text Search22,7981,6377.2Composite2506218067.2Rule ClassScreen positivetrue positivePPV, % This approach was more comprehensive than Jha et als. (2001) detection method. The rules were more dynamic because they were able to identify less severe adverse drug effects. The term adverse drug event was used instead of DRM when describing this study because many of the effects screened for would not fit the definition of a DRM. The events would be considered actual DTPs under Heplers conceptual framework. For example, text searches for signs and symptoms such as dizziness, fatigue, cough, and over anticoagulation without bleeding were major components of the screening instrument. Only nine percent of the adverse events identified required hospital admissions. The level of programming and database (administrative vs. clinical data) sophistication needed to run the rules varied. The ICD-9-CM rules did not require sophisticated software and were usable with administrative databases. The event monitoring rules did not require sophisticated software either, programming could be done with Access or a statistical program such as SAS. Nevertheless, clinical data (i.e., laboratory test results and medical records) was required. Most administrative records only indicate that specific types of laboratory tests were requested or conducted. Patient specific laboratory values are not documented. The allergy and text search rules were more sophisticated and they required electronic medical records. Honigman et als. screening method had advantages over Jha et als. method. Nevertheless, the level of

PAGE 39

27 sophistication limits the use of these rules to organizations with access to electronic clinical data and specific software. The PPV of Honigman et als. composite tool was relatively low (7.2%). The screens were the most comprehensive and could detect minor adverse drug events as well as events that led to hospital admission; however they were not designed to indicate whether or not the events were avoidable or if the admissions were preventable. The screens varied in their use of databases, they used administrative data, laboratory test results and electronic clinical records. The software needed to run the text searches was quite sophisticated. Faris (2001) developed a different approach to electronically detect PDRM. 13 His approach automated forty-nine PDRM indicators that paired an inappropriate process of care (potential DTP) with its predictable adverse outcome (DRM). The indicators were originally developed by MacKinnon (1999) 42 from a literature search of peer-reviewed medical articles and reference texts from 1967-1998, and from a consensus panel of seven experts in geriatric medicine. Faris (2001) revalidated the PDRM indicator definitions through a Geriatric medicine expert consensus panel. Forty-four of the fifty-two indicator definitions accepted in MacKinnons study were accepted for automation in the Faris study. Additional indicators were proposed and accepted by the panel for a total of forty-nine indicators. Once the indicator definitions were accepted they were translated into a format that could be applied to a managed care administrative database. The translation was designed to capture the events described in the indicator definitions through billing

PAGE 40

28 records from physician offices, hospitals, pharmacies, and other care sites in the managed care network. Services are billed through a system of codes that represent different treatments and activities. These codes included ICD-9-CMs, CPTs, and the National Drug Codes (NDC). SAS was used to extract and analyze cases that fit the indicator definitions. Faris (2001) retrospectively sampled 11,711 elderly patients over a one-year period and 966 had at least one positive for a PDRM indicator. Chart review was not done to validate the performance indicators. As an alternative, the proportion of patterns of care in the population that appeared to result in the corresponding adverse outcome was calculated (PTOV). Ten of the nineteen indicators, which had more than ten positives, had PTOV values greater that 0.74 percent. Of the five most prevalent indicators, three had adjusted PTOVs greater than or equal to 75%, which means the process of care was strongly associated with the corresponding adverse outcome. (Table 3.4.) For example, Asthma exacerbation and/or status asthmaticus and/or ER visit/hospitalization due to asthma was found 100% of the time when the following pattern of care occurred: Diagnosis of moderate to severe asthma and use of a bronchodilator with no use of maintenance corticosteroid. The criterion validity of PDRM indicators developed by Faris (2001) showed promise with ten indicators having adjusted PTOVs of 75% or greater, nevertheless the PPV of the PDRM indicators is unknown and needs to be tested. MacKinnon (1999) did, however, examine the criterion validity of two indicators that occurred frequently enough for statistical validation and found relatively high PPVs, 82% and 34%. 42

PAGE 41

29 Unlike the approaches taken by Jha et al. (2001) and Honigman et al. (2001), these indicators are specifically searching for preventable cases of DRM. The preventability is represented by the inappropriate process and its predictable adverse outcome. Unfortunately, only two of MacKinnons PDRM indicators were validated with chart review. Nevertheless, the PPV of the two indicators was decent and this method shows promise as valid PDRM measures. Another advantage of this approach was it did require sophisticated software. The search algorithms were produced with SAS. Furthermore, clinical data is not neededthese indicators were designed for administrative data. In summary, three studies with different computerized methods to detect PDRM were found in the literature. The approaches varied in ability to detect true cases of PDRM, and identify preventable events. The type of data needed, and the sophistication of the software used to produce search criteria also varied. The approach used by Faris (2001) will be adopted for this dissertation, because the PDRM performance indicators from that study included preventability in their operationalization. Furthermore, the indicators were intended for claims data and the search algorithms can be produced with basic statistical software, i.e., SAS. Delphi Method The Delphi method is a systematic approach for the utilization of expert opinion that was extensively studied by the RAND corporation in the 1960s. 43 The concept of the Delphi technique is simple; it is designed to define a single position (consensus) of a group by systematically combining their opinions in a way that eliminates activity among the experts to reduce dominance and group pressure.

PAGE 42

30 Table 3.4. Results of Top Five Indicators PDRM IndicatorsPositive IndicatorAdjusted PTOVThis outcome has occurred after the p attern of care below: ER visit/hospitalization due to congestive heart failureThis is the pattern of care:1. Diagnosis/history of congestive heart failure2. Not on ACE inhibitorThis outcome has occurred after the p attern of care below: ER visit/hospitalization due to congestive heart failure and or heart block This is the pattern of care:1.History/diagnosis of congestive heart failure with heart block or advanced bradycardia2. Use of digoxinThis outcome has occurred after the pattern of care below: ER visit/hos p italization due to h yp oth y roidis m This is the p attern of care:1. Use of a th y roid or antith y roid a g ent ( e. g .; levoth y roxine, etc. ) 2. T4/TSH not done before thera py starts and at least ever y 12 months This outcome has occurred after the p attern of care below: Gastritis and/or upper GI bleed and/or GI perforation and/or GI ulcer and anemiaThis is the pattern of care:1. Use of 2 or more NSAIDS concurrentl y This outcome has occurred after the p attern of care below:Asthma exacerbation and/or status asthmaticus and/or ER visit/hospitalization due to asthmaThis is the pattern of care:1. Diagnosis of moderate to severe asthma2. Use of a bronchodilator3. No use of maintenance corticosteroid2700.751030.458911840.951290.12 The Delphi technique is a modification of the traditional roundtable group decision making process. The roundtable approach has inherent limitations because the final decision or position may be an effect of the most outspoken person (i.e., the bandwagon effect) or socially acceptable opinion rather than the groups true opinion. 43 The Delphi technique avoids these undesirable effects by replacing the group meeting with anonymous response. Instead meaning the opinions of members are obtained by formal questionnaire. The procedures of the Delphi technique have the following key features : 44 Anonymous responseopinions of members are obtained by formal questionnaire or other formal communication channels as a way to reduce the effects of dominant individuals. Iteration and controlled feedbackthe interaction among the members is supported by a systematic exercise conducted in several iterations, with carefully controlled

PAGE 43

31 feedback between rounds. The summary of the results from the previous round are communicated to the participants Statistical group responsethe group opinion is defined as an appropriate aggregate of individual opinions in the final round. The use of a statistical definition of the group response is a way of reducing group pressure for conformity; at the end of the exercise there may still be a significant spread of opinion. In the 1960s the RAND Corporation carried out a series of experiments to evaluate Delphi procedures and to explore the nature of information processing. 44 The experiments involved fourteen groups of upper-class and graduate students from UCLA and they ranged in size from eleven to thirty members. The three effects that researchers examined were group size, a comparison of face-to-face discussion with the controlled-feedback interaction, and the ability of controlled feedback as a means of improving group estimates. They found increasing group size decreased the average group error and increased reliability. The anonymous controlled feedback procedures of the Delphi technique made the group estimates more accurate than face-to-face discussion. They also found controlled feedback narrowed the dispersion around the median. A large set of experimentally derived answers to factual questions was evaluated to determine the relationship between group size and the mean accuracy of a group response. 44 In this study the experimenters knew the answers to the questions; however, the subjects did not. The group error was calculated as the absolute value of the natural algorithm of the group median divided by the true answer. Dalkey (1969) found the gains in increasing group size were quite large from three to eleven members, but the percent change in group error was less than five percent past eleven members. (Table 3.5.)

PAGE 44

32 Table 3.5. Percent Change in Average Group Error by Number of Group Members. (Extrapolated from Dalkey (1969)) Group membersAverage group errorPercent change11.19430.852128.63%50.688219.23%70.609711.42%90.5726.18%110.55393.16%130.54531.56%150.54110.76%170.53910.37%190.53820.18% Dalkey (1969) also compared the reliability of two groups opinion with various numbers of members. Reliability was measured by the correlation between the answers of the two groups over a set of questions. Dalkey found a definite and monotonic increase in the reliability of the group responses with increasing group size. A mean correlation of 0.8 was obtained with thirteen group members. Even though these two studies showed increasing group size reduced the error of the group and increase the reliability of the estimate, it appears these estimates were obtained with traditional group consensus and not by a Delphi process. If these results are generalizable to Delphi group estimates then having a Delphi group with between eleven and fifteen members would seem reasonable because average group error only decreased by approximately three percent from eleven to thirteen members and reliability reached 80% with thirteen members. In the main experiment the performance of groups using face-to-face discussion was compared with Delphi groups. The first experiment involved two groups of five graduate students, and twenty questions were presented in four blocks of five using the ABBA design (A= face-to-face). The face-to-face group was instructed to follow a

PAGE 45

33 specific procedure for each question and the Delphi procedure involved four rounds of estimates, feedback of medians and quartiles and re-estimates. The median response of the Delphi group was more accurate in thirteen cases, and the face-to-face group was more accurate in seven cases. The second experiment had a different study design. The Delphi group had twenty-three members and from them the face-to-face groups were separated into seven groups of three and one group of one. In this experiment they found that Delphi estimates were similar to face-to-face estimates in terms of accuracy, however, face-to-face estimates produced more changes from initial estimates that reduced accuracy. Dalkey also examined the distributions of the answers between the first-round and second-round of the Delphi groups. The second-round distribution shifted toward the mean and Dalkey concluded the shift represents a convergence of answers toward the group response. Nevertheless, the second-round distribution still had a large range, indicating convergence was not complete. Dalkey also evaluated the effects of iterations on the accuracy of responses. To do this, changes with regards to individual questions were measured. The median improved in accuracy for about 64%, while the median became less accurate in 31%. The mechanism of improvement appeared to be from first-round feedback of the median response. The tendency to change was determined by the distance of the first round answer to the first round median. However, this did not explain the change in median from round-one to round-two. Some respondents had to cross the median for a change in median to occur.

PAGE 46

34 To evaluate how the median value changed from the first to second round, the researchers divided the groups into holdouts and swingers. Holdouts tended to cluster around the median and they were more accurate than the total group in the first round. The median of the group was between the median of the swingers and the median of the holdouts. The shift in the group median occurred because the swingers shifted toward the median, thus shifting the group median. In a supplementary analysis Dalkey (1969) tested the effects of other forms of feedback on the accuracy of the Delphi estimates. 44 The experimental group documented reasons for their decisions when their second round response was outside the interquartile range of the first round. Formulating and feeding back reasons did not increase the accuracy of the initial estimates or produce improvement on iteration. In a follow-up study Dalkey et al., (1969) evaluated the use of self rating to improve group estimates. 45 This study used 282 University of California students. Subjects were randomly assigned to sixteen groups with fifteen to twenty members per group. Almanac type questions were used to assess the relationship between accuracy and self rating. Subjects were given the questions and asked to rate each question from one to five to indicate their knowledge of the content in question. Relative rates were used, meaning subjects had to identify a question they were most knowledgeable about and give it a five. They also had to identify a question they knew the least about and give it a one. A clear association between average group self rating and group estimate accuracy was found. This study also found a significant improvement in the effectiveness of the Delphi procedures can be obtained by using self-rating information to select more accurate subgroups.

PAGE 47

35 The experimental studies on the Delphi process are important because they provide information on how group processes affect the accuracy of group estimates to factual questions. In this dissertation the Delphi method will be used to assign Faris indicators to nodes of the MUS where the process failure likely originated. In this case, a known answer does not exist. The opinions of the group members will be aggregated to assign select indicators to nodes of the MUS. Therefore the qualifications and diversity of experts in domain specific knowledge will be important factors to consider when constructing a Delphi panel. Cause-and-Effect Analysis To improve the performance of the MUS it is first necessary to understand error etiology within the MUS and the underlying system design factors that contribute to PDRM. Cause-and-Effect analysis (CEA) is a structured investigation that aims to identify the true cause of a problem, and the actions necessary to eliminate it. 46 CEA is an integrated problem solving methodology that incorporates multiple tools and strategies, which include process analysis, brainstorming and cause-and-effect diagramming. 46,47 First, process analysis is conducted to understand the processes involved in the quality problem. Before brainstorming or cause-and-effect diagramming is conducted an understanding of how the system works is needed. Process analysis should include flow diagrams that represent different nodes of the system. The steps needed to construct a flow diagram include: 47 1. Defining the basic nodes of a system 2. Further defining the process, breaking each node down to specific steps needed to complete the process 3. Following the objects through the process a number of times to verify the process by observation

PAGE 48

36 4. Discussing the process representation with the project team and reaching consensus on the underlying process. Brainstorming is a possible cause generation technique. 47 Brainstorming helps a group to generate many thoughts or ideas in a very short period of time. 46 The important aspect of brainstorming is ideas are to be generated without judgment or discussion until all the ideas are presented. In a typical group discussion someone presents a thought and others comment on it, or judge it. The problem is group members may focus on one idea before all other possible ideas have a chance to emerge. 46 Brainstorming encourages the flow and fluency of group member thoughts. Brainstorming has three steps: generation, clarification and evaluation. 46 In the generation phase the leader clearly states and writes the question or purpose and then invites and records responses. This step can either be structured or unstructured. In the structured approach, known as round-robin, each participant in turn launches one idea, this insures equal participation, but is less spontaneous and may limit the possibility for building on one anothers ideas. 46 In unstructured brainstorming everyone can freely launch ideas, this a spontaneous process, but it can become confusing and it may lead to one or more persons dominating the activity. 46 The second step in brainstorming is clarification. After all ideas have been generated, the group reviews them to make sure everyone is clear about their meaning. The focus of the generation phase was quantity and not quality, therefore, during clarification group members are encouraged to question the meaning of the generated items. The final step of brainstorming is the evaluation of clarified items. Here the group considers the list and rules out duplications, and irrelevant ideas. During the evaluation

PAGE 49

37 phase affinity diagramming or other methods for grouping ideas into similar concepts may be employed to make sense out of the generated items. Cause-and-effect diagramming is the heart of CEA. It is a tool used to define and illustrate the relationships between an effect and an outcome, or a problem and beliefs about the possible causes or factors contributing to it. 46 It combines the product of brainstorming with a systematic analysis to organize and evaluate causes to determine which are most likely contributing factors or causes. 46 Cause-and-effect diagramming can be done using fishbone or tree diagrams. Fishbone diagrams are the traditional way of illustrating cause-and-effect relationships. Nevertheless, tree diagramming, or the five whys analysis, is an effective way to graphically show the breakdown of large problems into their increasingly more detailed elements. 46 Tree diagrams help the group members to move from general to specific, or vice versa, in an organized manner, and they show the logical connections among the relationships. Accreditation organizations such as the Joint Commission on Accreditation of HealthCare Organizations (JCAHO) have recognized the importance of CAE and have incorporated the use of root cause analysis (RCA), which is a more specific form of CAE, into their quality criteria. Currently, JCAHO only requires RCA for inpatient sentinel events. Causes of DRM in the ambulatory setting have not received much attention and information on CAE for adverse outcomes of drug therapy in the outpatient setting to my knowledge is non-existent. In the Joint Commission paradigm, the data used as input for RCA are the collection of activities that led to the specific incident. Activities are determined by interviews with providers involved in that particular patients therapy and by reviewing

PAGE 50

38 the patients medical chart. While this approach is ideal for sentinel events in the inpatient setting because the quality team has access to providers involved in the particular event and the patients medical record, it is not as useful for rate based events that occur in ambulatory care. A patient who experienced a DRM in the ambulatory setting may have participated in a number of microsystems which are geographically separated, such as the primary care, specialist, pharmacy, and laboratory systems; or personal events. Managed care organizations and health care coalitions, typically do not have the same leverage and influence over these microsystems as, say, hospital administrators who have access to the medical records and have the influence to require participation from professionals involved in the care of a patient who experienced a PDRM. In summary, literature that evaluates the utility of administrative data for health services research was presented. Different methodologies for electronically screening health care data were also discussed. The history, purpose and methods of the Delphi technique were presented. Finally, the process for cause-and-effect analysis was discussed

PAGE 51

CHAPTER 4 PRELIMINARY WORK This section first addresses the prevalence of PDRM in the ambulatory care setting. Following is an evaluation of drug treatment categories frequently involved in preventable drug related admissions (PDRAs). Next, is a discussion of problem areas in the MUS that were found to be proximate causes a of PDRAs. This section will end with an evaluation of latent causes of PDRAs. Prevalence of Preventable Drug Related Admissions The Harvard Medical Practice Study (HMPS) is historic because it was the first large scale study to look at medical injury as a result of errors or problems in the process of medical care. 48 Prior to the HMPS, research on drug related injury focused mainly on adverse drug reactions (ADRs), which are typically thought of as the inherent risk associated with medications use and not problems with drug therapy management. Even though the HMPS was published in the early 1990s, medication errors and PDRM did not gain widespread appreciation and public awareness until the Institute of Medicines (IOM) Report, To Err is Human, in 2000. 2 The IOM report revisited the HMPS and a large study from Utah that focused exclusively on inpatient medical errors and adverse events. The report gave much less attention to ambulatory events. Nevertheless, they did suggest DRM causes a significant a The term proximate cause is being used to describe the initial error or problem that started the events that lead to a PDRM. The term is defined as: That which in ordinary natural sequence produces a specific result, no independent disturbing agencies intervening. Webster's Revised Unabridged Dictionary, 1996, 1998 MICRA 39

PAGE 52

40 number of admissions to inpatient facilities, and they reported the proportion of preventable admissions is unknown. Winterstein et al. (2001) tried to address this issue by producing a meta-analytic analysis of drug related hospital admissions (DRA). 1 The objective was to estimate the prevalence of PDRA and to explore the relationship between study characteristics and prevalence estimates. Unfortunately, a generalizable prevalence estimate was not produced because the studies did not have similar protocols and there was too much heterogeneity in prevalence estimates to summarize the findings with a mean value. Instead, a median and range were presented to display the distribution of studies and meta-regression coefficients were calculated to evaluate the association of various study characteristics with prevalence estimates. The methods and results are presented below. Since the goal was to produce a meta-analytic summary estimate of PDRA, a criterion was established to select only studies that attempted to conduct a comprehensive surveillance of drug therapy as a cause of preventable patient injury. Studies were excluded that limited their scope to specific drug treatments, indications, injuries, or to only one drug-therapy problem. The studies had to discuss the relationship between pharmacotherapy and patient morbidity, and preventability, along with the necessary information to calculate prevalence. Fifteen studies 5-12,49-55 published between 1980 and 1999 met the inclusion criteria. All studies were conducted in industrialized countries: eight in Europe, four in the US, two in Australia, and one in Canada. (Table 4.1) The median DRA prevalence was 7.1% (IQR 5.7-16.2%) and the median PDRA prevalence was 4.3% (IQR 3.1-9.5%). Overall, the median preventability rate was 59% (IQR 50-73%).

PAGE 53

41 Since the protocols were not homogeneous across the studies and the Cochrans Q test indicated extreme heterogeneity (Q: 176; df: 14; p<0.001), an analysis of the association between different study characteristics and PDRA prevalence estimates was warranted. The following characteristics have been discussed in the literature and were considered potential contributors to the heterogeneity of the findings: (Table 4.2) Inclusion/ exclusion of first hospital admission (re-admission studies) Planned admissions, and transfers from other units or hospitals Mean sample age (>70 vs < 70); country (US/other) Selection of hospital units vs. inclusion of entire hospitals Publication year (>1992 vs. < 1992) Inclusion/ exclusion of indirect drug-related morbidity (lack of drug effectiveness, and no access) Inclusion/ exclusion of MD and patient interview Explicit criteria for judging preventability The study characteristics were analyzed using meta-regression models, the dependent variable was the transformed PDRA estimate and the independent variable was a specific study characteristic for each model. These regressions were fit with random-effects weights using restricted maximum likelihood estimation of the between study variance to account for study heterogeneity. The meta-regression models produced point estimates and 95% confidence intervals of prevalence odds ratios comparing studies that differ with respect to each study characteristic (Table 4.2). The inclusion or exclusion of first admissions (i.e. re-admission studies) was the study characteristic most strongly associated with PDRA prevalence (OR: 3.7; 95% CI: 1.5-8.9). The mean of age of admissions was the next most strongly associated characteristic with PDRA prevalence (OR: 2.0; 95% CI: 0.95-4.2). The inclusion of indirect DRM was also associated with higher prevalence estimates (OR: 1.9; 95% CI: 0.92-3.9). The

PAGE 54

42 remaining study characteristics listed in Table 4.2 had little or no apparent association with PDRA prevalence. The results of this systematic review suggest that PDRAs represent a significant public health concern in ambulatory care (Table 4.1). In most studies, more than half of DRAs were preventable. That is, they were not considered acceptable consequences of therapeutic risk-benefit considerations but rather caused by inappropriate care and medication errors. The data in Table 4.1 give the impression of a widespread and long-standing problem in the quality of drug therapy management because the studies represent problems from 1980 to 1999 and nine of the fifteen studies had PDRA prevalence estimates above four percent. To place the reported prevalence estimates in perspective, according to the National Hospital Discharge Survey, there were 31.8 million admissions to U.S. hospitals in 1998. 56 The top six primary diagnostic categories (heart disease, delivery, neoplasms, pneumonia, psychosis, and cerebrovascular disease) each accounted for three to twelve percent of these admissions. Any of the studies, or all of them combined, suggest that inappropriate management of drug therapy may be a leading cause of hospital admissions in developed countries. The meta-regression analysis indicated that sampling methodology does affect prevalence finding. Limiting the sample population to patients previously hospitalized produced much higher PDRA prevalence estimates than not restricting the sample to re-admissions. The causes of these findings are not clear. Patients re-admitted may be a sicker sub-population who had been hospitalized with more risk factors for DRM, e.g., more drugs, more diseases, more prescribers, etc. A need for research specifically

PAGE 55

43 designed to evaluate the transition of drug therapy management from the hospital to the ambulatory setting is clearly warranted. DRM is a concept, and the way in which it is operationalized affects its prevalence. As one would expect, studies that included indirect injury (e.g., lack of therapeutic effect, and lack of access to medications) were associated with higher prevalence findings. The limited view of drug-relatedness found in one-third of the articles used in this analysis may be an artifact of an outdated approach to evaluating adverse outcomes of drug therapy. For an assessment of DRM to be considered comprehensive it would have to include both direct and indirect drug-related injury. The IOM report suggested, There is evidence indicating that [adverse drug events] account for a sizeable number of admissions to inpatient facilities. This systematic review confirms their suspicion and suggests that PDRM in ambulatory care is at least as significant and prevalent as in inpatient care environment. Drug Categories Involved in PDRA The same fifteen drug-related admission studies from Table 4.1 were used to assess the medications commonly involved in DRAs. Three of the studies 5,12,54 were not included because they did not provide information on specific medications or therapeutic classes involved in DRAs. Medications involved in DRAs were the unit of analysis rather than medications involved in PDRAs because fewer than five studies provided drug specific preventability.

PAGE 56

44 Table 4.1: Studies Reporting DRA and PDRA: Sample Characteristics and Prevalence Estimates b ReferenceCountryStudy year & periodDRA PrevalencePDRA PrevalencePreventability of DRADarchy199441/ 62330/41199912 months-6.6%(73%)Ng199631/17210/1721196319993 weeks(18%)(5.8%)(32%)Raschetti 94/9545/ 183325/183325/45199912 weeks(2.5%)(1.4%)(55.6%)Cunningham 54/1011 1997(5.3%)Nelson199373/450 19961 month(16.2%)Courtman92/9321/15018/15018/2119955 months(14%)(12%)(86%)Dartnell199455/96536/96519951 month(5.7%)(3.7%)Hallas8.0%67/143 1992(n= 1999)(47%)Lindley26/41613/ 261992(6.3%)(50%)Nikolaus87/9022/873208238313199236 months(25.3%)(12.6%)(50%)Bero45/22434/22434/451991(21.1%)(15.2%)(76%)Bigby83/8473/686198724 months(10.6%)Lakshmanan 198435/83419/3519862 months(4.2%)(54%)Trunet 78/8197/165143/165143/97198533 months(5.9%)(2.6%)(44.3%)Trunet 78/7923/32514/32514/23198012 months(7.1%)(4.3%)(61%)France30/623 (4.8%)AustraliaItaly43/54 (79.6%)USA43/450 (9.5%)43/73 (58.9%)UK4 weeks each unit43/1011 (4.3%)CanadaAustralia36/55 (65.5%)Denmark 1988/893.8%UK10 weeks13/416 (3.1%)GermanyUSAMissingUSA43/686 (6.3%)FranceFrance43/73 (58.9%)USA19/834 (2.3%) b Modified table from Winterstein et al. (2002)

PAGE 57

45 Table 4.2. Prevalence Estimates and Odds Ratio per Stratum/Specific Study Groups Study characteristicCategoryNumber of StudiesAverage PDRA1 (%)PDRA prevalence odds ratios (95% CI)2Excluded214Included134.23.7 (1.5 8.9)> 7067.6 7073.92.0 (0.95 4.2)Missing2Included106.1Excluded53.31.9 (0.92 3.9)USA46.9Other114.41.6 (0.73 3.6)Excluded85.9Included74.11.5 (0.71 3.0)Excluded66Included951.3 (0.61 2.8)Entire hospital85.8Selected units74.51.1 (0.57 2.3) 199285> 1992751.0 (0.47 2.1)Yes85.8No74.21.4 (0.67 2.9)Yes65No94.91.0 (0.47 2.2)MD/ patient interview*Specified criteria for preventability judgment*Transfers from other units or hospitalsPlanned admissionsHospital UnitsYear of publicationFirst hospital admissionMean ageIndirect drug-related morbidityCountry The purpose was to find the treatment categories that were involved in DRAs and to report their median frequency. This was done by classifying specific medications into their therapeutic class and then organizing therapeutic classes into treatment categories. Variability existed n the way medications involved in the DRAs were presented. Studies either listed the specific medications involved in the DRAs, the number of events involving a specific therapeutic class or only the number of events related to a treatment category. Therapeutic classes were kept as long as they were mentioned in at least two studies. If only one study mentioned a particular therapeutic class it was categorized into a more general group, e.g., cardiovascular/other.

PAGE 58

46 The cardiovascular treatment category had the highest median DRA prevalence (33.3%) c and this category was represented in twelve studies. The median prevalence for the Anti-inflammatory category was 11.4% (represented in eleven studies). The median prevalence of the Anti-diabetic category was 12.15% (represented in eight studies). The median prevalence of the Psychotropic category was 8.75% (represented in eight studies). The median prevalence of the Anti-infective category was 8% (represented in seven studies). The median prevalence of the Non-specific miscellaneous category was 12% (represented in ten studies). d See Figure 4.1 for bar charts of treatment categories and their median prevalence that were mentioned in at least six studies. The actual data from the studies can be found in Appendix A. The therapeutic classes specifically addressed in at least six studies are presented in Figure 4.2. The therapeutic class Diuretics was most often mentioned (10 studies). Antihypertensives, Hypoglycemic, and NSAIDs were mentioned in eight studies. Antibiotics were specifically mentioned in seven studies. These findings are not surprising because they represent therapeutic categories for common disease states. Cardiovascular disease is the leading cause of hospital admissions 56 and medications used to treat cardiovascular disease were involved in a c Prevalence was calculated by using the total number of medications involved in DRAs from a particular treatment category as the numerator and total number of medications involved in DRAs for the denominator. The prevalence for each treatment category was calculated individually for each study and the median values was used to represent the prevalence of that treatment category. d The Miscellaneous category (also described as the other category was often used in the articles to group medications that were infrequently involved in DRAs. This category is not mutually exclusive from the other treatment categories. For example; one study may have found that 15% of the medications involved in DRAs were NSAIDs while another study only found 1% was from NSAIDs. The study that found only 1% if medications involved were NSAIDs probably would have put them in the other category.

PAGE 59

47 large portion of DRAs. Admissions due to diabetes, infections, and psychosis are also among the leading causes of hospital admissions. From the results of this analysis it appears that medications most often prescribed tend to be the treatments most often associated with DRAs. Cardiovascular, psychotherapeutic anti-infective, analgesics, and anti-diabetic agents were among the top ten therapeutic classes by retail sales share in 2001. e This means that DRAs appear to be happening most frequently in patients being treated for common conditions with common and well-accepted therapies. From a measurement perspective this information can be very useful. It can be used to focus the development of medication specific performance indicators and it can also be used to help select indicators for use that have already been developed. DRAs in the twelve studies used in this analysis were determined by medical record review, however, which is currently the gold standard for measuring the prevalence of DRM. Medical record review is a costly and inefficient process. The use of automated performance indicators is an alternative for measuring the prevalence of DRM. This information is useful for developing and selecting performance indicators. Any instrument used to measure the prevalence of DRM in a population should try to address the treatment categories found to represent a significant proportion of medications involved in DRM i.e., cardiovascular, psychotropic, anti-inflammatory, analgesic, ant-diabetic and anti-infective agents. e Source NDC Pharmaceutical audit Suite. http://www.ndchealth.com/epharma/YIR/pharmatrends.htm

PAGE 60

48 Apparent/Proximate Causes of PDRAs As discussed in chapter two, DRM occurs when an error interacts with the latent conditions of a system to produce injury. Errors are produced at the provider-system interface. In the MUS, the sharp-edges where professionals interact with the patient and with other health professionals happens at the prescribing, dispensing, (self) administration, and monitoring nodes of the system. The same fifteen articles listed in Table 4.1 were analyzed to determine the proximate causes of the DTPs, i.e., DTPs were classified into the nodes of the MUS where the errors likely occurred. A classification template for mapping DTPs to the nodes of the MUS was developed to judge error etiology. When DTPs were not classified into the nodes of the MUS the judgment of three reviewers were used to classify them (Dr Charles D Hepler, Sooyeon Kwon, and Brian C Sauer). Classification required 100% agreement, each reviewer judged the DTPs location in the MUS independently. The reviewers discussed their individual decisions and when discrepancies occurred they stated their opinions and worked through the differences until agreement was obtained. The authors focus, as well as any additional information mentioned in the article, was considered when assigning the DTPs to nodes of the MUS. When an author in some cases appeared to include both dose prescribed and dose administered in the same overdose group, the case was classified into a category labeled non-identifiable process problem. See Appendix B for a complete list of DTPs and information used to make judgments for the placement of DTPs into nodes of the MUS.

PAGE 61

49 05101520253035Antibiotic PsychotropicAnti-inflammatoryAnti-diabeticMiscellaneousCardiovascular Figure 4.1: Involvement of Treatment Category to DRMs: Median prevalence for Treatment Categories Mentioned in at Least 6 studies 02468101AntibioticsAntihypertensive NSAIDsHypoglycemicDiuretics 2 Figure 4.2: Therapeutic Class Involvement in DRM: The Number of Studies that Specifically Mentioned Therapeutic Class

PAGE 62

50 Since the purpose was to identify process related problems f that led to PDRA, non-preventable ADRs were not included in the analysis. ADRs are a special type of DTPs; they typically represent an outcome and not a process of drug therapy. They have a gradient of severity; the less severe ADRs would be considered an actual DTP, e.g., diarrhea. The severe ADRs would be considered DRMs, e.g., hospitalization from dehydration due to frequent diarrhea. ADRs have historically been treated as the inherent risk of medication use and not as a failure of drug-therapy management. Preventable hospital admissions due to ADRs, nevertheless, may represent a process failure, because somewhere in the MUS actions were not taken to detect and correct the ADR from developing into a serious injury. Preventable ADRs were classified into the nodes of the MUS according to the authors description. If no description of the process failure was presented then, as default, preventable ADRs were assigned to the monitoring node. To be included, studies had to specify the type of DTPs, or the node in the MUS where the DTP was initiated, that led to PDRAs. Eight studies linked DTPs to preventability, 5-12 (Appendix B) the other seven presented DTPs in relation to DRM, but did not partition DTPs by preventable admissions. The median values and inter-quartile range (IQR) for each node in the MUS were: prescribing (33.93%; IQR: 5.73-54.86%), dispensing (0%), self-administration (22.29%; IQR: 14.19-25.51%), monitoring (31.80%; IQR: 20.13-45.22), non-identifiable process (5.93%; IQR: 0-16.39). (Appendix C and Figure 4.3) f The phrase, problems with the process of care is being used as an alternative to represent the same concept as error. When discussing the etiology of DTPs terms related to process failures and process problems will often be used since the term error has the ability to produce a defensive stance. When examining the causes of DRM it is important to reduce tension and the defensive armory of professionals to uncover factual events and true beliefs about the processes of care delivered.

PAGE 63

51 The finding that prescribing and monitoring problems were major contributors to PDRAs in these studies is troubling because it clearly indicates the MUS failed at the professional/patient interface. Another important finding was problems in the monitoring node were as prominent as problems in prescribing. This is interesting because inappropriate prescribing tends to receive more attention than monitoring. Many resources are being directed toward improving prescribing decisions, as seen through the popular interest in computerized prescription order entry systems. From these findings one could argue that interventions towards improving systematic monitoring and follow-up of drug therapy should receive at least as much attention as prescribing. (Self) administration or patient non-compliance was also found to be a substantial contributor to PDRAs. Patients and caregivers are participants in microsystems of care and they have a critical role in the MUS. (Self) administration is a patient behavior and its overlap with medical error and professional responsibility can be debated. Non-compliance is a result of intended and unintended actions. It has been shown that patients make poor decisions because of rule and knowledge based mistakes. 57 Health professionals should be able to help patients correct their misunderstandings. They can also help patients find creative ways to prevent unintentional non-compliance by reducing slips and lapses. Nevertheless, patients typically choose to initiate the processes involved in the MUS and in doing so they incur some of the responsibility for their health outcomes. From a systems perspective, finding methods to reduce the occurrence of PDRM is not limited to directing interventions at health care professionals and provider specific processes. Patient directed interventions, and reconfigurations directed toward patient behavior would be included in the systems umbrella.

PAGE 64

52 Figure 4.3: Nodes of the MUS Involved in PDRA Prescribing, monitoring and patient (self) administration appear to be the nodes where most DTPs were generated. Because prescribing, monitoring and (self) administration nodes have different characteristics it is possible error provoking conditions upstream and faulty corrective mechanisms downstream (i.e., latent failures) have unique system design flaws. Any rational intervention directed at a specific node in the MUS would need to address the system conditions that increase the chance of an error being created and progressing undetected. Latent Causes of PDRA Latent causes are flaws in the design and organization of systems that allowed errors to occur, go undetected and result in patient injury. None of the fifteen studies provided a comprehensive assessment of latent causes. Nevertheless, eight studies 5-8,10-12 did mention, in the discussion, possible latent causes. (Table 4.3) Lack of adequate knowledge was mentioned most frequently as an underlying cause for errors in the prescribing node of the MUS (seven studies). Three of these studies mentioned patient knowledge was a probable reason for problems in the (self) administration node. Education was considered the best intervention for both prescriber and patient knowledge deficits. Three studies mentioned lack of communication and coordination among professionals, especially the physician and pharmacist, was an underlying cause of

PAGE 65

53 PDRAs. Two studies implied the lack of pharmacists involvement in the MUS was a factor and one study mentioned inadequate monitoring procedures were an underlying cause. Unlike many inpatient DRM studies, none of the fifteen PDRA studies systematically evaluated the latent causes or empirically tried to locate system failures or determine their relationship to higher order or competing systems. Without an adequate evaluation of these influences and interactions, the development of meaningful interventions will likely be difficult. Instead, corrective approaches will probably result in generic interventions, those that the organization know how to do, for example send a letter to the physicians who seem to be producing a portion of the problems. For meaningful improvements to occur, the development of rational interventions are necessary. The term rational is being used to represent interventions that are directed at specific underlying system related causes. These are typically design and organization issues. If lack of prescribing knowledge was determined to be a main cause of prescribing errors, then a rational intervention may include finding better ways to provide therapeutic information to prescribers at the time therapeutic decisions are being made. From this analysis it is clear that a formal cause-and-effect analysis is needed to uncover conditions of the MUS that make it prone to problems within the medications use system. To summarize, injury as a result of drug therapy is a serious problem in ambulatory care. A review of fifteen studies showed the median PDRA prevalence was approximately four percent. Many common treatment categories were involved in these DRA. Cardiovascular and anti-inflammatory agents were involved in many DRAs. Errors in the MUS are the apparent causes of PDRM and problems in the prescribing, monitoring

PAGE 66

54 and (self) administration nodes occurred frequently. None of the studies systematically evaluated the latent conditions that provide the opportunity for proximate errors to occur and go undetected. Lack of prescriber and patient knowledge was mentioned as latent causes for prescribing and (self) administration problems. Nevertheless, a planned systematic analysis is warranted. A better understanding of latent conditions could be used to develop rational interventions. CEA is a promising method for uncovering latent conditions. Table 4.3. Latent Causes of PDRAs Prescribers EducationPatients EducationMonitorInvolve pharmacistInter-professional CommunicationDarchy, 1999Raschetti,1998Cunningham, 1997Nelson, 1996Dartnell, 1996Courtman, 1995Bero, 1991Trunet, 1980

PAGE 67

CHAPTER 5 METHODS This study was executed in three steps. In the first step, database analysis, MU-PIs were used to estimate the number of PDRM positives in the study database. In step two the degree of association between selected MU-PIs and specific nodes of the MUS where the process failure may have originated was evaluated. In step three, selected MU-PIs from each node of the MUS were submitted to cause-and-effect analysis (CEA). Specific Aims 1. To establish the frequency of MU-PI positives in the study population. 2. To evaluate population based explanations for MU-PI positives. 3. To identify the node of the medications use system (MUS) where the pattern of care from the MU-PI originated. 4. To identify system related causes that are common among nodes of the MUS 5. To identify system related causes that are unique to nodes of the MUS Step One: Database Analysis Claims Data Types The MU-PIs were used to screen the study database. Each MU-PI was translated into the administrative billing codes that represent the process and outcome components of the indicators. 13 The billing codes included the International Classification of Disease 9th edition (ICD-9) and Current Procedural Terminology (CPT) and National Drug Codes (NDC). The population included patients of all ages who were enrolled in a preferred provider organization (PPO) health plan that managed employees from a member of the 55

PAGE 68

56 health care coalition. The primary data files include Professional Claims (Center for Medicare Services (CMS) 1500 form), Facility Claims (Universal Billing (UB) 92 form) and Pharmacy Claims. The CMS 1500 and UB 92 forms are provided at the following web address: http://www.cms.hhs.gov/providers/edi/edi5.asp#Form%20CMS-1500 Descriptive Analysis to Evaluate the Integrity of Claims Data The number of office visit, emergency department (ED) visit, hospital admission and pharmacy claims were compared month by month. If any month showed a large drop in claims, this would suggest large numbers of claims might be missing from the database. Office visit and ED visit claims were identified in the professional claims database with specific CPTs. a Primary diagnosis, NDC, procedure codes, and service dates were evaluated to identify missing or invalid codes. Links between databases were established to ensure members could be traced across claim types. Population Demographic Average length of time enrolled in the health plan, average number of office visits, age and gender frequencies were calculated to describe the study population. Frequency of age by category, gender, number of office visits, and number of different drug classes, b different pharmacies, different conditions, and different prescribers were used in logistic regression analysis to better understand the MU-PI findings from information available in the database. a See Appendix E for a list of the codes used to identify office visits and ED visits b The Universal System of Classification (USC) was used to identity different drug classes and routes of administration. Unique USC codes identified drug classes (e.g., H2 blockers vs. GI proton pump) as well as different routes of administration (e.g., sumatriptan oral vs. sumatriptan nasal). Drug class was identified by unique USC codes.

PAGE 69

57 MU-PI Coding Concepts and Analysis Faris 2001 translation of the PDRM scenarios into medical event codes was used to construct the MU-PIs. 13 In Faris study, two medical record coders had independently selected all possible ICD-9 and CPT codes that represented the PDRM scenarios. The codes identified by the medical record coders had also been reviewed by a physician for clinical judgment about whether each code was consistent with the PDRM scenario. Please see Faris 2001 for more detail on the coding methodology. 13 A list of the 40 MU-PIs c used in this study can be found in Appendix D. Search algorithms Each MU-PI had the following format, an outcome of care and a specific pattern of prior care. For example, ED visit or hospital admission for Hemorrhagic event AND use of warfarin AND prothrombin time/INR not done every month. The pattern of care is the process component of the indicator and it represents a potential DTP. An ED visit or hospitalization was required for the outcome of care. An indicator positive (possible PDRM) required both the process failure and the outcome component of the indicator. Process failures fell into three categories: disease-drug interaction, drug monitoring, and drug-drug interactions. Each required different types of search algorithms. SAS version 8.2 was used to code and run the indicators in the database. The type of search algorithm used for each indicator is presented in Appendix F. Disease-drug interaction The disease-drug algorithm required the presence of specific diagnosis codes and specific pharmacy claims prior to a claim for the associated outcome. As illustrated in c Only 40 of Faris 49 indicators were run. Search algorithms were to complex for nine and they were not used for this analysis.

PAGE 70

58 Figure 5.1, time moves from right to left by the blue arrows. The diagnosis claim had to be present before the use of the medication (pharmacy claim). The green arrow represents the lag, i.e., the duration of time from the outcome date to the last drug date prior to the outcome. The cut off for the disease-drug interaction lag was set at 100 days, in order to accommodate 90-day mail-order pharmacy refill cycles and to include patients who may have been taking the medication in proximity to the outcome. An indicator positive required the outcome, the disease code(s) and a pharmacy claim for the drug(s) within 100 days prior to the outcome. Prior Diagnosis Date Last Drug Date Prior to Outcome Outcome Date Lag Figure 5.1. Disease-Drug Interaction Search Algorithm Drug monitoring Drug monitoring algorithms required the presence of specific pharmacy claims and CPT claims, which indicate whether specific laboratory analyses were conducted. Two coding solutions were required for monitoring indicators: one for the process component and another for the process and outcome simultaneously. As illustrated in Figure 5.2, time moves from right to left by the blue arrows. Once specific drug claims initiate the process, lags are calculated to represent the interval of time from drug to first CPT claim (lag 1), CPT claim to CPT claim (lag 2), and last drug claim date to last CPT claim date (lag 3). Patients were considered at risk and included in the analysis when they appeared to be taking the medication for at least as long as the defined lag time. Monitoring intervals (i.e., lag times) vary according to MU-PIs. If any lag exceeded the defined

PAGE 71

59 monitoring interval the member screened positive for the process component of the MU-PI. Figure 5.2. Process Search Algorithm for Indicators that Require Monitoring PDRM positives were identified when members had specified pharmacy claims and the lag between the outcome and the last CPT claim prior to the outcome exceeded the defined monitoring interval. If the lag was less than four days, the lag between the outcome date and second to last CPT date was used to avoid false negatives. The assumption is that monitoring within four days of hospitalization identified the impending injury and need for emergency or hospital care. (Figure 5.3) Figure 5.3. Process and Outcome Search Algorithm for Indicators that Require Monitoring The determined monitoring interval for the warfarin indicator is 30 days. The example in Figure 5.4 would register as a process positive, but not a PDRM positive because the pattern of care was stabilized before the outcome occurred. The outcome may indeed be drug related, but it is not represented by the MU-PI indicators. The lag had to exceed the defined monitoring interval from outcome date to last CPT claim for a PDRM positive. CPT date 1 First Drug Date n t h CPT date CPT date 2 LAG 2a LAG 2b Last Drug Date LAG 1 LAG 3 Last CPT Outcome Date Drug Date LAG

PAGE 72

60 Figure 5.4. Example of a Process Positive but not a PDRM Positive Drug-drug interaction The third search al gorithm considered multiple drugs. This included drug-drug interactions, drug-no drug situations and overuse of one drug and under use of the other. The process component is indicator specific. If the MU-PI is a simple drug-drug interaction with Drug A being a chronic medication and Drug B acute process positives are recorded when a claim for Drug B occurred between the first and last claims for Drug A. (Figure 5.5) PDRM positives are recorded when pharmacy claims for Drugs A and B occur within the specified number of days for each lag. Figure 5.5. Drug-Drug Interaction Search Algorithquency of MU-PI positives in the study esearch Question 1. What is the period prevalence for process positives in the Research Question 2. prevalence for the PDRM positives in INR Claim 1 date First Warfarin Claim nt h INR Claim date INR Claim 2 date 50da y s25 days40 days 25 days Outcome Date LAG 2 Drug A Firsat t Drug D e Drug A Lasat t Drug D e Drug B L ast Date Outcome Date LAG 1 Diagnosis Date m Prevalence Estimates for MU-PI Positives Specific Aim 1. To establish the fre population. R study population? What is the period the study population?

PAGE 73

61 This is a cross-sectional (period prevalence) analysis. The population included patients of all ages who were enrolled in a PPO health plan from January 1 st 1999 to September 11 th 2001. The analysis was limited to one process positive per indicator; however, no limits were placed on PDRM positives. Demographic Explanations for Prevalence Findings Specific Aim 2. To evaluate demographic based explanations for MU-PI positives. Information from the database was used to better understand indicator positives. Descriptive analyses of monitoring intervals were conducted to determine if cases on the margin of the monitoring interval were being identified as process positives, and multivariate logistic regression was conducted to explore the relationship between PDRM positives and demographic/system related variables. Distribution of Monitoring Intervals Research Question 3. Do many process positives appear to be captured from the margin of the defined monitoring intervals for indicators that require drug therapy monitoring? Due to the nature of the MU-PIs, the quality of drug-therapy monitoring can be grouped in one of two categories. A case can either screen positive or negative for the inappropriate monitoring interval. When a case screens positive it means an individual did not have claims for the defined laboratory tests within the specified monitoring interval. One limitation of this approach is the strict cut-offsthose on the margins are categorized as process failures even if they were only a few days or weeks over the required time. To develop a more comprehensive picture of how cases deviated from the defined monitoring intervals, an analysis of the first monitoring interval (i.e., the lag time from the first pharmacy claim for the drug of interest to the first claim for laboratory tests

PAGE 74

62 of interest) was presented for indicators grouped by the indicator defined monitoring interval. This was a descriptive analysis. The intervals were determined by dividing the monitoring requirement in half. For example, if required monitoring for the indicators was one-month, the categories for descriptive analyses were based on fifteen day intervals. Analysis was only carried out for indicators with defined monitoring intervals of one month or greater. Variables Associated with PDRM Positives Research Question 4. What demographic variables are associated with MU-PI findings? Available information from the database was used to explain the prevalence findings. Bivariate and multivariate logistic regression analyses were conducted. Demographic variables age, gender, and number of drug class, pharmacies and prescribers were used as independent variables, while PDRM positives were the dependent variables. Multicollinearity was evaluated with tolerance and variance inflation statistics. Tolerance is 1-R 2 for the regression of a given independent variable on all other independents, ignoring the dependent. The higher the intercorrelation of the independents, the more the tolerance will approach zero. If tolerance is less than 0.20 a problem with multicollinearity is indicated. After multicollinearity was assessed, stepwise variable selection was used to build the regression model. The likelihood ratio test was used to test the overall significance of the model. Max-rescaled R 2 was used to assess the predictive power of the model. The significance of the variables in the model was assessed by the Wald Chi-Squared test and confidence intervals (CIs). Hosmer-Lemeshow statistics were calculated to determine the goodness

PAGE 75

63 of-fit of the final multivariate model. Odds ratios (ORs) and Walds CIs were calculated directly from the estimated regression coefficients and their standard errors. All P-values were set at 0.05. All analysis were performed using SAS Version 8.2 Step Two: Node Identification of Select MU-PIs Specific Aim 3. To MUS where the pattern of care from PDRM scenarios originated. The ultimate goal of this dissertation was to identify system factors (i.e., latent conditions of the health care system) that appear to be common or unique to nodes of the MUS. This requires an exploration of the association between selected indicators and specific nodes of the MUS. To do this, selected MU-PIs were sent to a Delphi panel of clinicians and researchers for their judgment on the node of the MUS in which the process component of the indicator likely originated. Pilot Testing The node identification panel survey was pilot tested with graduate students and faculty in Pharmacy Health Care Administration at the University of Florida. All five participants were pharmacists, three had received their pharmacy degrees in the United States, one had received her degree in South Korea and the other had received his degree in Germany. The purpose of the pilot test was to evaluate the understandability of the survey and the node identification scoring method. Organization of content, directions, and scoring system were modified based on comments from the first round of the pilot test. No significant problems or need for changes were identified in the second round of pilot testing.

PAGE 76

64 Selection of MU-PIs for Node Identification Study MU-PIs used in the pilot were not the same as those used in the Delphi study. Indicators were chosen based on preliminary results from the MU-PI study. MU-PIs were primarily chosen based on indicator frequency. d Delphi Recruitment The snowball sampling or chain referral sampling was used to identify participants for the Node Identification Delphi study. This method has been widely used in qualitative research. It yields a study sample of people with specific characteristics, e.g., expertise. People with desired characteristics mention others who may also be likely candidates for study participation. 58 The first wave of recruitment letters was mailed with stamped return envelopes to researchers who have published extensively in the field of medication error research or who were referred by faculty members in Pharmacy Health Care Administration at the University of Florida. The letter specified the goal of the study, asked the contacts to participate in the Delphi panel, and invited them to recommend others who are trusted experts in medication error or adverse drug events research. e I telephoned the first round contacts between one and two weeks after the recruitment letter was mailed to answer questions the contacts may have had and to try to convince them to participate. f Sample size and power for the statistical tests were considered prior to the recruitment process. When setting alpha at 0.05 and beta at 0.8, it was determined that a d Scenarios were selected based on preliminary indicator findings, changes were made to the MU-PIs and not all scenarios selected for the Delphi study had PDRM indicator positives. e The term adverse drug event was used for the Delphi study instead of drug related morbidity because it is the standard term in the medical community and I wanted to avoid confusion among the panel members. f Please see Appendix G for a copy of the recruitment letter.

PAGE 77

65 sample size of thirty-two was needed to identify a node as statistically different from the others when its mean value was at least a six and the standard deviation was less than or equal to three. Node Identification Survey Surveys were mailed to members who agreed to participate. The survey provided detailed instructions that included definitions for the nodes of the MUS for the assignment task. Specific demographic information was requested and it included name, occupation (researcher and/or licensed health professional), whether they were practicing or not, level of education, age, and gender, belief about the significance of PDRM The panel members were provided fourteen MU-PIs and were asked to allocate the number of points from 1 into the nodes of the MUS where they believe the error or drug therapy problem originated for each of the fourteen indicators. They were asked to answer based on a population perspective, meaning they were asked to identify the node that explained the origin of the majority of cases that might fit the indicator definition. Panel members were also asked to document their decision logic and provide comments when they deemed necessary. g Delphi Process Results (panelists scores) for each MU-PI were analyzed after the first round. If an indicator had received an average score for one node that was significantly higher than its average score for all other nodes, it was assigned to that node and deleted from the second round of the Delphi study. In the second round, panel members received information about each remaining indicator, including summary distributions of g See Appendix H for a copy of the Node Identification Survey.

PAGE 78

66 responses, median and interquartile ranges for each node, the particular Delphi members response to each item, and a summary of the decision logic statements for the panel (grouped into like statements). Delphi participates were asked to reconsider their opinions and to state their decision logic regardless of whether they changed their belief. Node Identification Analysis Research Question 5. What nodes of the MUS did the selected PDRM scenarios likely originate from? Hypothesis: Ho: Prescribing = Dispensing = Administration = Monitoring Ha: At least one node is significantly greater than all other nodes. The Kruskall Wallis test (KW), an analysis of variance test for non-parametric data, was used to determine if differences in rank order means existed among the nodes of the MUS. Dwass, Steel, Critchlow-Fligner (DSC) multiple comparison post hoc tests were used to determine which nodes were significantly different (P values=0.05). StatsDirect Version 2.3.7 software was used for the KW and DSC tests. Step Three. MU-PI System Level Evaluation Specific Aim 5. To identify common cause sequences among nodes of the MUS. Specific Aim 6. To identify unique cause sequences for specific nodes of the MUS. Selection of MU-PIs for Evaluation The fourteen MU-PIs evaluated in step two were found to be problems that originated in the prescribing or monitoring nodes. Two MU-PIs were selected from the prescribing and monitoring nodes. Selection was limited to MU-PIs that were evaluated in the Node Identification Study and was based on frequency of MU-PI positives. h h Indicator number 28 was evaluated in the Delphi study and had the second highest frequency for the prescribing indicators, however it was not selected because the indicator did not appear to differentiate those with congestive hearth failure (CHF) and heart block from those with CHF only.

PAGE 79

67 Step Three: Cause-and-Effect Analysis MU-PI Evaluation Team Members of the MU-PI evaluation team were identified and invited to participate by the CEO of the health care coalition. They consisted of a Medical Director, PharmD, nurse practitioner, consumer advocate, laboratory manager and a director of a health maintenance organization. The session was scheduled for four hours. Members of the evaluation team were paid $100 per hour. One week prior to the meeting the evaluation team was mailed a briefing that contained the goals for the study and background on the MU-PI results, cause-and-effect analysis and the levels of the health care system. It also provided the MU-PIs that would be evaluated and asked them to begin generating ideas. A copy of the brief is available in Appendix I. MU-PI Evaluation Process The evaluation process began with a brief introduction and slideshow presentation to orient them to the study purpose and database findings. The two MU-PIs formally evaluated were the asthma prescribing indicator and the warfarin monitoring indicator, indicators 39 and 30, respectively. (Appendix D) The following six steps were carried out for each MU-PI that was formally evaluated: brainstorm causes, clarify and organize causes, prioritize causes, tree-diagramming, and establish causes that are common to multiple nodes of the MUS and causes that are unique to specific nodes of the MUS. The moderator provided an introduction and background of the study objectives and indicator prevalence findings. Indicator specific finding were presented for the

PAGE 80

68 warfarin monitoring indicator and the asthma prescribing indicator. The question to guide the brainstorming process was presented for each indicator. The evaluation team brainstormed possible system related problems that may have contributed to the MU-PI indicators. The round robin process was used to elicit causes from each participant. Proposed causes were projected on a screen and documented in Path Maker Version 5.5 software. The evaluation team then clarified and organized the proposed system causes into an affinity table. The affinity table was organized by the levels of the health care system (patient, professional, organization, environmental levels). Proposed causes were organized into the level of the system the problem was associated with. A category for measurement error called artifact was also permitted. Proposed causes were organized into affinity tables with Path Maker Version 5.5 software The evaluation team then prioritized the proposed system cause and the top five causes were selected for tree-diagramming. The facilitator went through the list of proposed causes from the affinity table and the evaluation team voted by raising their hands for each of the five causes they believed contributed the most to the prevalence of the indicator findings. Scores were tallied for each cause and the five receiving the most votes were selected for tree diagramming. The evaluation team then used tree diagramming to identify relationships among the different levels of the health care, as they relate to the proposed cause. Tree diagram consisted of branches that had each level of the health care system (patient, professional, organization, environmental levels). One at a time, the top five proposed causes were placed into the system levels associated with the cause. For example; if the proposed

PAGE 81

69 cause was lack of MDs providing patient education it would go into the professional (microsystem) level. The follow-up question would have been; what at the organizational level may influence MDs ability to provide patient information? The proposed organizational influence would be placed in the organization node of the same branch of the tree. (Figure 5.6) Excel Version XP was used to create the tree diagrams and document the findings. Figure 5.6. Organization of Proposed Causes to System Levels Proposed Cause MDs don't provide patient education N o Medication Management System Guidelines unclear Patient level Professional level Organization level Environment 1 MDs don't provide patient education 2 3 4 Branch What at the organizational level may influence MDs ability to provide patient education? The evaluation team then established commonality and uniqueness within and between nodes of the MUS. An indicator assigned to the monitoring node was tree diagrammed. The evaluation team was given a questionnaire to determine if each cause sequence (i.e., each branch of the tree) would apply to another indicator assigned to the monitoring node. They also had to judge whether each cause sequences would apply to an indicator assigned to the prescribing node. The same procedure was applied to the prescribing indicator that was formally evaluated, i.e., tree diagrammed. (Appendix K)

PAGE 82

70 Analysis for Common and Unique Cause Sequences Research Question 6. What cause sequences were common within the Prescribing Node? Research Question 7. What cause sequences were common within the Monitoring Node? Research Question 8. What cause sequences were common to the Prescribing and Monitoring Nodes? Agreement was determined when five out of six (83%) or four out of five (80%) of judges agreed.

PAGE 83

CHAPTER 6 RESULTS Step One. Database Analysis Descriptive Analysis to Evaluate the Integrity of the Claims Database To evaluate the completeness of the database the numbers of claims were compared by month. The dates of office visit claims ranged from January, 1999 to September, 2001. As illustrated in Figure 6.1, many office visit claims appear to be missing on the tail ends of the data collection period, from January, 1999 to June, 1999 and from July, 2001 to September, 2001. Emergency Department (ED) visit claim dates ranged from January, 1999 to June, 2001. As illustrated in figure 6.2, many ED visit claims appear to be missing from January, 1999 until June, 1999. Facility admission claim dates ranged from January, 1999 to June, 2001. No claims appeared to be missing during the study period (Figure 6.3). Pharmacy claim dates ranged from January, 1999 to June, 2001. As illustrated in figure 6.4, pharmacy claims appeared to be missing from January, 1999 to June, 1999 and in September, 1999. Missing pharmacy claims from September, 1999 are not followed by an increase in claims. It appears claims during this month were missing from the database. The decision was made to only include claims from July, 1999 to June, 2001 for analysis. This decision was made because the number of claims was consistent during this time frame. Even though pharmacy claims appeared to be missing in September, 1999 the decision to include the previous months was made because the benefit of keeping July, August, and September in the database outweighs censoring the time. 71

PAGE 84

72 Check for missing and invalid data Macro level analysis found no missing or invalid codes for primary diagnosis, procedures or service dates. Missing or uninterpretable drug codes were found in 530 of the 572,010 pharmacy claims (571,480 usable pharmacy claims). 020004000600080001000012000Jan-99Mar-99May-99Jul-99Sep-99Nov-99Jan-00Mar-00May-00Jul-00Sep-00Nov-00Jan-01Mar-01May-01Jul-01Sep-01 Figure 6.1. Office Visits Claims by Month 02000400060008000100001200014000Jan-99Mar-99May-99Jul-99Sep-99Nov-99Jan-00Mar-00May-00Jul-00Sep-00Nov-00Jan-01Mar-01May-01 Figure 6.2. ED Visit Claims by Month Links between data types To ensure that members were identifiable across claim types links were established between datasets to identify the number of members that had multiple types of claims.

PAGE 85

73 4,227 of the 4,589 members who had a facility admission claim also had a professional claim. 28,478 of the 33,891 persons with a pharmacy claim also had a professional claim 050100150200250300350Jan-99Mar-99May-99Jul-99Sep-99Nov-99Jan-00Mar-00May-00Jul-00Sep-00Nov-00Jan-01Mar-01May-01 Figure 6.3. Facility Admissions by Month 05000100001500020000250003000035000Jan-99Mar-99May-99Jul-99Sep-99Nov-99Jan-00Mar-00May-00Jul-00Sep-00Nov-00Jan-01Mar-01May-01 Figure 6.4. Pharmacy Claims by Month Population Demographics There were 47,053 members who contributed 58,719 member years during the sample period of July, 1999 to June, 1999. The mean length of enrollment was 467.47 days with a standard deviation of (SD) 225.53 days. The 37,063 members with a medical

PAGE 86

74 claim (pharmacy or professional) contributed a total of 51,892 member-years; mean length of enrollment was 511.04 days with a SD of 201.73 days. Age and gender distributions for the different claim types are presented in Table 6.1. The mean age ranged from thirty-four to thirty-seven across the databases. The majority of members with a claim were female. Females ranged from sixty-five percent to seventy-two percent across the claim types. The fifteen to forty-four year olds were the most frequent age group, their frequencies ranged across the data types from thirty-seven to forty-eight percent. The sixty-five and older group accounted for a small fraction of the claims, their frequencies ranged from about two to eight percent. Table 6.1. Population Age and Gender Frequencies by Type of Claim mean age(SD)By Categoryfrequency, [percent]age <156,622 [19.07]1,084[18.51]1,263[17.61]4,956[14.3]15<=age<4516,519 [47.58]2,143[36.6]3,399[47.39]16,461[47.48]45<=age<6510,743 [30.94]2,144[36.62]2,270[31.65]11,421[32.94]65<=age835 [2.41]484[8.27]240[3.35]1830[5.28]Female22,861 [65.85]3243[71.84]4,659[64.96]23,499[67.78]Male11,858 [34.15]1271[28.16]2,513[35.04]11,169[32.22]000033.97 (18.23)34.48 (22.12)34.71 (18.54)37.03 (18.34)Professional ClaimFacility AdmissionED visitPharmacy Claim Table 6.2 displays additional population characteristics. There were 220,967 office visit claims during the twenty-three month period with an average of 6.64 visits, which were approximately three visits per member per year. Pharmacy claims were submitted for 33,891 members. The average number of pharmacy claims per member during the twenty-three month time period was 16.05 with a SD of 22.9, which were approximately

PAGE 87

75 eight pharmacy claims per year. Members went to an average of 1.55 different pharmacies and an average of 2.13 different prescribers during the study period. Members averaged 5.8 conditions and used an average 4.7 prescriptions from different drug classes. Prevalence of MU-PI Positives There were a total of 103 PDRM positives from eighty members. The prevalence of PDRM positives in members with a medical claim from July, 1999 to June, 2001, was 1.98 (1.60 to 2.37) a per 1,000 member years. There were a total of 10,889 process positives in 5,741 members. The prevalence of process positives from July, 1999 to June, 2001 was 209.84 (206.37-213.34) per 1,000 member years. Table 6.3 shows that, 0.22% of the population with a medical claim screened positive for both the process and outcome component of the MU-PIs, i.e., PDRM positives. There were forty-three PDRM positives in thirty-eight members where the outcome was identified by an ED visit and forty-one PDRM positives in twenty-six members where the outcome was identified by hospital admissions (HA). Nineteen PDRM positives in eighteen members involved a hospital admission from the ED. PDRM positives accounted for 0.75% (95% CI: 0.74%-0.76%) of hospital admissions, and 0.56% (95% CI: 0.55%-0.57%) of ED visits. Table 6.4 provides a summary of the number of PDRM positives by number of indicators. Twenty-two of the forty MU-PIs had no PDRM positives. Of the remaining eighteen indicators, the range of PDRM positives per indicator was from zero to twenty-six. The eleven indicators with the most frequent PDRM positives are listed in Table 6.5. a 95% Confidence interval

PAGE 88

76 Table 6.2. Population Demographics No of Enrollees47,053No of Members with claim37,063Length of time enrolledmean (SD) 511.04 (201.73)No. ED visits10,048No. Hospital Admissions5,855No. of outpatient ph y sician visits220,967mean (SD) 6.64 (7.05)No. of outpatient ph y sician visits ( members ) 33,271 1-210,28130.9 3-46,78120.38 5-64,48013.477-83,1679.529-102,2956.9>106,26718.84No. of Conditions ( diseases ) 34,938mean (SD) 5.8 (4.8)LE 520,71459.296-109,16126.2211-153,3829.6816-201,0302.95GE 206511.86No. of members with pharmac y claim33,891 average per member (SD)13.93 (22.61)No. of dru g classes 543,959 average per member (SD)4.7 (5.31)25,13567.827,30519.712,8097.589782.648362.26No of different pharmacies average per member (SD)1.55 ( 1.45 ) 117,98153.0628,23824.3133,91911.5641,9895.8751,0333.056 or more7312.16No of different prescribers average per member (SD)2.13 ( 2.07 ) 112,48236.8328,28124.4335,30215.6443,2069.4651,9895.8761,1173.37 or more1,5144.47All analysis are for the 23 month time windowCharacteristicsFrequencyPercent

PAGE 89

77 Table 6.3. Frequency of Process and PDRM Positives No. of Members with Positives (unit: person)Population % (denominator=37,063)No. of Positives (unit: each event)Process5,74115.49%10,889PDRM_ED380.10%43PDRM_H A 260.07%41PDRM_ED_H A 180.05%19Total_ PDRM800.22%103 Table 6.4. Number of PDRM Positives by Number of Indicators Freq. of PDRM PositiveNo. of IndicatorsPercent02255.0132410.03353921225.02612.5 7.57.57.55.0 Table 6.6 lists the indicators with the ten most prevalent process positives. Thirty-eight of the forty MU-PIs screened positive for the process component of the indicator. The range of process positives was from zero to 2,282. Approximately fifty-four percent of the indicators had less than one hundred process positives, and ninety-two percent of the indicators had less than 1,000 process positives. System Based Explanations for Prevalence Findings Information from the database was used to better understand indicator positives. Descriptive analyses of monitoring intervals were conducted to determine if cases on the margin of the monitoring interval were being identified as process positives, and multivariate logistic regression was conducted to explore the relationship between PDRM positives and demographic/system related variables.

PAGE 90

78 Distribution of monitoring intervals Table 6.5. Eleven Most Prevalent PDRM Positives IndMnemonicHxDrug_ADrug_BRiskProcessPDRM39AsthmaHx-Bron-NoSteroid2,1533,7043,8781902628CHFHx/CHF_HB-Dig645309791237CHFHx/CHF-NoACEI6454,059398121DepHx/Dep-Benzo1,8982,689321930HemrWar-INR(1)343256919HypothThy-T4TSH(12)2,6151,510915529ARFACEI-BUN(6)4,0592,4092,282531MIHx/MI--ASA-BB**1812,6688957DepHx/Dep-Symp1,8982,971136314ActRespFailCOPD-Benzo6412,689102316CHFHx/HBP-NSAID6,0017,2831,4273 Table 6.6. Ten Most Prevalent Process Positives IndMnemonicHxDrug_ADrug_BRiskProcessPDRM29ARFACEI-BUN(6)4,0592,4092,282523HyprKACEI-ElctroCBC(6)4,0592,4092,209216CHFHx/HBP-NSAID6,0017,2831,427319HypothThy-T4TSH(12)2,6151,510915517HypoKKwd-NoK-Elctro(2)1,810837814643037CHFHx/CHF-NoACEI6454,059398121DepHx/Dep-Benzo1,8982,68932195GIHx/GI-NSAID2,5265,899316030HemrWar-INR(1)343256927GIHx/GI-Ocort2,5264,0752721 Legend for Tables 6.5 and 6.6 See Appendix D to link indicator numbers to the indicator scenarios. The Mnemonic is an abbreviated version of the MU-PI scenario Hx = the number of members who had an ICD-9 code for the particular history of disease or diagnosis of interest Drug A and Drug B = the number of members who had a prescription claim for the drugs of interest. Risk = the number of members who met the criteria for analysis. For example; if the process component includes a drug and six month monitoring interval, then the members at risk would be those who were using the medication of interest for at least six months. Process = the number of members who had the pattern of care represented in the MU-PI PDRM = the number of members who had both the process component and the outcome component of the MU-PI within the specified time frame. Fourteen indicators were sent to a Delphi process for node identification, they were numbered consecutively from one to fifteen. This column provides a cross reference to the MU-PI number Due to the nature of the MU-PIs, the quality of drug-therapy monitoring can be grouped in one of two categories. A case can either screen positive or negative for the

PAGE 91

79 inappropriate monitoring interval. When a case screens positive it means an individual did not have a laboratory test claim within the specified time frame. One limitation of this approach is the strict cut-offsthose on the margins would be categorized as process failures even if they were only a few days or weeks over the required time. To develop a more comprehensive picture of how cases deviated from the specified monitoring intervals, an analysis of the first monitoring interval (i.e., the lag time from the first pharmacy claim for the drug of interest to the first laboratory test claim of interest) is presented for indicators grouped by the required monitoring interval. Analysis was only carried out for indicators with defined monitoring intervals of one month or greater. Table 6.7 provides the indicator numbers for each monitoring interval. For example: Only indicator 30 had a one month monitoring requirement, while indicators 9 and 25 had a three month monitoring requirement. Table 6.7. MU-PI Categorized by Required Monitoring Interval one monthtwo monthsthree monthssix monthstwelve months301292193525811182324262933 Indicator 30 (Hemm warfarin INR (1)) was the only indicator with a required monitoring interval of one month. Figure 6.5 shows forty-one percent of the members who were taking warfarin had a laboratory test claim for an International Normalization Ratio (INR) within a month from their first warfarin pharmacy claim. Only about 3.8% had laboratory test claim on the margin within forty-five days of their first warfarin

PAGE 92

80 claim. The majority, 50.4 percent did not have a laboratory test claim for an INR within 120 days of their first claim for warfarin. 020406080100120140160180 days<=3030120lag timefrequency 50 4 % 0. 2 9% 0. 87% 0. 87% 3. 79% 1 75% 0. 87% 41 1 % Figure 6.5. First Lag for Indicator with One Month Monitoring Requirement Indicators 35 and 12 had a required monitoring interval of two months. Figure 6.6 shows 0.56% of members at risk for these indicators had the specified laboratory test claims (blood pressure and cell blood counts) within two months of their first pharmacy claim, while 99.4% did not have the specified laboratory test claim within 240 days from their first pharmacy claim. Indicator 35 accounted for all but one process positives. Indicator 35 required blood pressure monitoring it is likely that blood pressure monitoring is not billed separately and the process positives are false positives. This indicator was removed from all other analysis (including the previous prevalence findings) because the CPT codes used to identify blood pressure monitoring were not sensitive to blood pressure monitoring. Indicators 9 and 25 had a required monitoring interval of three months. Figure 6.7 shows 4.35% of members at risk for these indicators had the specified laboratory test

PAGE 93

81 claims (lithium levels and BUN) within three months of their first pharmacy claim for the defined medication, while 84.78% did not have the laboratory test claim within 360 days from their first pharmacy claim. No lags were identified on the margin of the required interval. Indicators 2, 8, 11, 23, 24, 26, 29, 33 had a required monitoring interval of six months. Figure 6.8 shows 24.78%of members at risk for these indicators had the specified laboratory test claims within six months of their first pharmacy claim for the defined medication. Some process positives were captured from the margin, 6.53% had the laboratory test claim between 180 and 270 days. 050100150200250300350400 days <=6090
PAGE 94

82 0510152025303540 days<=90135
PAGE 95

83 0100200300400500600700800 days <=365365730lag timefrequency 29.54 5.63% 12.58 52.25 Figure 6.9. First Lag for Indicator with Twelve Month Monitoring Requirement The majority of members who met the requirements to be considered for the MU-PIs with six month monitoring requirements (57.97%) did not have the specified laboratory test claim within 720 days from their first pharmacy claim. Indicator 19 had a required monitoring interval of twelve months. Figure 6.9 shows the majority of the members who appeared to be using thyroid therapies (52.25%) had their thyroid tests (laboratory test claims) within twelve months of their first pharmacy claim for the defined medication. About thirteen percent had the specified laboratory test claim between 365 and 545 days, about six percent had a claim between 545 and 730 days from the pharmacy claim, and about thirty percent did not have the laboratory test claim within 730 days of the first pharmacy claim for the required medication. In summary, the majority of process positives did not result from the margins of the required lag interval. The majority of process positives resulted from the extreme of

PAGE 96

84 the distribution. The indicators that required one month, six month and twelve month monitoring intervals were picking up some process positives from the margins, 3.78% 6.53%, 12.58%, respectively. Variables associated with PDRM positives Personal variables associated with PDRM positives were identified with bivariate logistic regression. Table 6.8 displays the OR and CI for each dependent variable. Age, and number of office visits, prescribers, pharmacies, drug classes and conditions were significantly associated with PDRM positives. Gender was not significant (odds ratio OR: 1.00; confidence interval CI: 0.63 1.59). Number of pharmacies and number of prescribers were the strongest bivariate predictors (OR: 1.48; 95% CI: 1.43-1.63) and (OR: 1.48; 95% CI: 1.40 -1.57), respectively. Number of drug class and number of conditions had the next strongest association (OR: 1.19; 95% CI: 1.16-1.21) and (OR: 1.20; 95% CI: 1.181.23), respectively. Number of office visits and age had significant but weak bivariate associations (OR: 1.07; 95% CI: 1.06-1.09) and (OR: 1.06; 95% CI: 1.05-1.08), respectively. Multivariate multicollinearity was assessed using the tolerance score. As shown in table 6.9, no tolerance values were below 0.20. The lowest tolerance score was 0.256, meaning 0.256 of the variance in number of office visits was unexplained by the other independent variables. A multivariate logistic regression was conducted to explore the relationship between PDRM positives and demographic/system related variables. Logistic regression with PDRM positives as the dependent variable and the seven demographic/system related variables as the independent variables used stepwise variable selection to produce the model. A four-variable risk model was produced. The model indicated that males

PAGE 97

85 Table 6.8. Bivariate Tests of Association: PDRM Positives as Dependent Variable Independent VariableOdds RatioAge1.061.051.08Gender F vs. M1.000.631.59number of Office visits1.071.061.09number of Prescribers1.481.401.57number of Pharmacies1.481.341.63number of Drug class1.191.161.21number of Conditions 1.201.181.23 95% Wald CI were about 50% more likely to have a PDRM positive after adjusting for the other variables in the equation (OR: 0.45; 95% CI: 0.28-0.74). Number of office visits had a negative influence on the odds of having a PDRM positive (OR: 0.93; CI: 0.9-0.96). That is, each additional office visit reduced the probability of having a PDRM positive by approximately seven percent, after adjusting for the other variables in the model. Number of drug classes and number of conditions increased the odds of having a PDRM positive. When drug class was held constant at zero, each additional medical condition increased the odds of being PDRM positive by about twenty-nine percent. When medical conditions were held constant at zero, each additional drug class increased the odds of having a PDRM positive by about twenty-two percent. Number of drug classes and number of medical conditions had a significant interaction with a negative beta-coefficient ( coefficient: -0.004; p-value: 0.0014). Increases in both medical conditions and drug classes resulted in decreased odds of having a PDRM positive. The overall significance of the equation by the likelihood ratio had a chi-square of 306.007 (p< 0.0001) with four degrees of freedom (df). The proportion of variance explained by the model (max-rescaled R 2 : 0.279). Overall the model fit well to the data (Hosmer-Lemeshow: 4.01; p=0.779; df=7).

PAGE 98

86 Table 6.9. Multicollinearity Assessment of Independent Variables VariableDFParameter EstimateStandard Errort valuePr > |t|Tolerance V ariance inflationIntercept1-0.040720.00394-10.35<.0001.0Age1-0.00005510.00007498-0.730.46240.849421.17728number of Conditions10.007240.0002071234.96<.00010.268573.72342number of Drug Classes10.00660.0001591141.46<.00010.319933.12572number of Office Visit1-0.004210.00014943-28.19<.00010.255613.9122number of Pharmacy1-0.003450.00055755-6.18<.00010.789541.26657number of Prescribers1-0.010890.00045741-23.8<.00010.416532.40079 Table 6.10. Maximum likelihood Estimates: PDRM Positives as Dependent Variable Intercept1-9.2290.404521.676<.0001Gender, F1-0.3980.12610.0140.0016number of Conditions10.2540.03166.933<.0001number of Drug Classes10.2000.02469.664<.0001number of Office Visits1-0.0740.01719.389<.0001Cond*Drug class1-0.0040.00110.1560.0014Pr > ChiSqParameterDFEstimateStandard ErrorWald chi-Sq Step Two. Assignment of MU-PIs to Nodes of the MUS The ultimate goal of this dissertation was to identify system factors (i.e., latent conditions of the health care system) that appear to be common or unique to nodes of the MUS. To do this, the node of the MUS where the MU-PIs process problem originated was determined. Selected MU-PIs were sent to a Delphi panel of clinicians and researchers for their judgment on where in MUS the process component of the indicator likely originated. Delphi Recruitment A total of fifty-seven recruitment letters were mailed out. Twenty-nine contacts agreed to participate, seven declined, and twenty-one did not respond. Twenty-nine contacts were e-mailed or mailed the survey (based on individual request) and eighteen completed and returned the first round. The Delphi panelists consisted of five medical doctors (MDs), one nurse, five doctors of pharmacy (PharmDs), six bachelors of

PAGE 99

87 pharmacy (BA pharm), and one masters in health service research (MHSA). Fifteen of the seventeen members with clinical degrees were licensed in the US and eight were currently practicing. All of the panelists with clinical degrees had additional graduate level degrees. (Table 6.11). Table 6.11. Demographics of Delphi Panelists for Node Identification Study Other DegreeGenderLicensed in the USPracticingMD52 MPH 1MS 1PhDYes = 17Nurse11MPAPharmD51 PhDBA Pharm61 MBA PhD 4 PhDNo = 11 MHSATotal Number of Delphi Panel = 18Clinical Degress1586 Female 12 Male Assignment of MU-PIs to Nodes of the MUS The Node Identification study went through two rounds of the Delphi process. An indicator was assigned to a node and not returned for the second round when the average score for a node was significantly greater than the average scores for all other nodes. Eleven indicators reached significance within the first round. Three of the indicators were returned for the second round and two achieved significance. Based on the Delphi panelist responses, problems with the specificity of the indicators were the reason they did not reach significance in the first round. Changes in the indicators were made and they were sent back to the Delphi panel. The panelists were informed of the changes made to the indicator scenarios. They were also provided with their score, the groups average score and a summary of the panelists responses for the three indictors. The changes made to the three indicators included,

PAGE 100

88 Changing sympathomimetic decongestants to prescription sympathomimetic decongestants for Delphi indicator 9. Removing INR not done before therapy starts from Delphi indicator 12 (the monitoring requirement was one-month) Changing antibiotic use to use of macrolide or sulfonamide antibiotics for Delphi indicator 14 The indicators were judged to be primarily associated with either the prescribing or monitoring node of the MUS. None of the indicators were found to be primarily associated with the dispensing or administration nodes of the MUS. Six indicators were assigned to the prescribing node, seven to the monitoring node and one did not reach significance within the two rounds. Table 6.12 provides a summary of the indicator assignments. b The MU-PI column in Table 6.12 is the MU-PI number and it provides a cross reference to the Delphi indicator number. Table 6.13 lists the statistics for each indicator analysis. Mean scores for each node, Kruskal Wallis statistics and p-values are presented for each indicator. Significance of the Dwass, Steel, Critchlow-Flinger pairwise comparisons are represented by an underscore under the abbreviated node. Nodes connected with an underscore were not statistically different. Delphi indicators 1-6, 8, 10, 11,13, and 15 reached significance in the first round, all p-values <0.001 for KW statistic and pairwise comparisons. Indicators 9, and 12 reached significance in the second round, p-values <0.001 for KW and pairwise comparisons. Indicator 14 did not reach significance and therefore it was not assigned to a node. Box plots for the mean, median and mode for each indicator are provided in Appendix L. b Delphi indicator number seven was not included because it was a repeat of Delphi indicator number four.

PAGE 101

89 Table 6.12. Indicators Listed by Associated Node of the MUS Indicator No. given to Delphi PanelNode Assigned by Delphi Panel1(16)CHFHx/HBP-NSAIDPrescribing2(37)CHFHx/CHF-NoACEIPrescribing3(28)CHFHx/CHF_HB-DigPrescribing4(5)GIHx/GI-NSAIDPrescribing5(17)HypoKKwd-NoK-Elctro(2)Monitoring6(8)SeizAntconv-DrugLvl(3)Monitoring8(9)LiToxLi-DrugLvl(3)Monitoring9(22)TachyHx/HTN-DeconPrescribing10(23)HyprKACEI-ElctroCBC(6)Monitoring11(33)ARFAllop-BUN/Scr(6)Monitoring12(30)HemrWar-INR(1)Monitoring13(39)AsthmaHx-Bron-NoSteroidPrescribing14(42)HemrWar-Antibiot-NoLab(5dys)No agreement reached15(29)ARFACEI-BUN(6)MonitoringMU-PI Step Three: Cause-and-Effect Analysis Indicators that had been evaluated in the node identification (Delphi) study were eligible to be selected for cause-and-effect analysis. The MU-PI evaluation team consisted of a Medical Director, PharmD, nurse practitioner, consumer advocate, laboratory manager and a director of a health maintenance organization. Indicator 30 (warfarin indicator) and indicator 39 (asthma indicator) were formally evaluated. Proposed causes were brainstormed, organized into affinity tables, and placed into tree diagrams (see methods).

PAGE 102

90 Table 6.13. Kruskal Wallis and Pairwise Comparisons for Indicator Assignment to Nodes of the MUS Indicator No.NodeMean ScoreKW statisticp valueMean ScoreKW statisticp valueP61.39D25.89P > M > D > APrescribingA16.22M42.50P60.78D24.78P > M > D > APrescribingA23.11M37.33P54.41D22.85P > M > D > APrescribingA18.74M42.00P63.36D28.50P > M > D > APrescribingA21.89M32.25P41.36D25.67M > P > D > AMonitoringA18.44M60.52P36.30D21.33M > P > A > DMonitoringA25.11M63.25P36.81D21.19M > P > A > DMonitoringA24.89M63.11P43.2551.37D37.19P > A > D > M29.10P > D > M > APrescribingA40.5019.37M25.0622.17P32.94D22.03M > P > D > AMonitoringA19.06M55.97P35.59D20.75M > P > D > AMonitoringA18.25M55.41P41.0329.80D24.24M > P > D > A20.17M > P > D > AMonitoringA21.7119.03M51.0353.00P57.59D26.27P > A > M > DPrescribingA27.12M27.03P45.3243.70D31.50P > M > D > A26.93P > M > D > AA17.0012.10M44.1839.27P34.94D19.38M > P > A > DMonitoringA20.59M55.09*Delphi indicator number (MU-PI number)**Dwass-Steel-Chritchlow-Fligner (connecting lines = not significant differences between nodes)No Decision made15 (29)42.121<0.000114 (42)24.548<0.000130.742<0.00012nd round13 (39)32.276<0.000112 (30)29.174<0.000140.57<0.00012nd round11(33)44.497<0.0001<0.00012nd round10 (23)44.57<0.00019 (22)*8.4950.03732.228 (9)48.427<0.00016 (8)49.587<0.00015 (17)45.071<0.00014 (5)46.139<0.00013 (28)41.724<0.00012 (37)45.124<0.00011 (16)*51.583<0.0001First Round Second Round Kruskal Wallis testPairwise Comparison**1st round decision Kruskal Wallis testPairwise Comparison1st round decision

PAGE 103

91 Evaluation of Monitoring Indicator The following question initiated the brainstorming for the warfarin monitoring indicator, What system factors may explain the number of people using warfarin who appear to be missing monthly INRs? Nineteen causes were proposed. Table 6.14 lists the eleven proposed causes that remained after clarifying, grouping and organizing them into an affinity table. Four were deemed artifacts of the indicator. They included: difficulty titrating warfarin doses, no claim submitted for INR because (a) cash payment for laboratory test, (b) INR was not specifically coded in bill and lack of claim even though INR had been conducted. The evaluation team voted on the causes they believed contributed most to the lack of monthly INRs for patients taking warfarin. The four causes with the most votes were selected for cause-and-effect diagramming. Lack of patient understanding about the importance of INR monitoring, patient non-compliance with orders to have laboratory tests done, lack of physician follow-up to determine if labs are being done, and lack of a medication management system were judged to be the most important contributors to the lack of monthly INRs in patients taking warfarin. The evaluation team was asked to fit the four causes deemed most influential to the warfarin monitoring problem into cause-and-effect sequences, i.e., tree branches. They identified fifteen sequences (Table 6.19). After sequencing, team members were given a written questionnaire to determine if the cause sequences would be contributing factors to another type of monitoring indicator (indicator 29 c ). When five out of the six team members (83%) agreed that each branch listed problems that were common the other c Indicator number 29 ( ARF ACE-I no BUN (6)) was assigned to the monitoring node of the MUS in the Delphi study.

PAGE 104

92 monitoring indicator that cause sequence was identified as a system problem that was common to the monitoring node of the MUS. Table 6.14. Monitoring Indicator: Affinity Table and Rating of Importance RatingPatient level7patient education (pt does't understand the need to monitor drug therapy) 0difficult titrate dose 5patient is not compliant with physicians order to have lab doneProfessional level4delay b/t titration and prescribing 1pt comprehension assessment is lacking 3responsibility (lack of claiming responsibility), multiple physician 7lack of followup by physician to make sure labs are being done and dose is appropriate Organization level5medication management system lacking to detect those not being monitored3prescription supply exceeds the INR check interval 1billing process does not capture INR specifically Environment Artifactdifficult to titrate dose cash payers so no claim for INR (bundled claims)billing process does not capture INR specifically lack of billing (claim) eventhough lab was conducted To evaluate whether the cause sequences elicited for the warfarin monitoring indicator would exhibit problems with the prescribing node as well as the monitoring node the team was asked whether each sequence would explain problems found by the asthma prescribing indicator. Twelve of the fifteen cause sequences from the warfarin monitoring indicator were judged to be problems that were common the asthma prescribing indicator. Table 6.15 presents the agreement among the team members for each branch. Evaluation of Prescribing Indicator The same process was applied for the evaluation of the asthma prescribing indicator. The following question initiated the brainstorming for the asthma

PAGE 105

93 prescribing indicator, What system factors may explain the number of asthmatics who did not appear to be using maintenance therapy? Eighteen causes were proposed during brainstorming. Sixteen remained after clarifying, grouping and organizing them into an affinity table. Patient education, patient does not understand function of steroid, multiple physicians, and lack of a medication management system were judged to be the most important contributors to the under-use of maintenance therapy. Table 6.16 lists the affinity table and ratings for each proposed cause. Table 6.15. Cause Sequence Agreement Results BranchM-MM-PBranchP-PP-M183%100%167%40%283%100%267%40%383%83%383%80%483%100%4100%80%5100%83%567%80%683%100%683%80%783%50%7100%80%8100%83%883%80%9100%67%9100%100%10100%67%10100%100%11100%83%11100%100%12100%100%12100%80%13100%100%13100%100%1483%100%1467%60%15100%83%M-M = monitoring to monitoring indicato r M-P = monitoring to prescribing indicato r P-P = prescribing to prescribing indicato r P-M = prescribing to monitoring indicato r one member did not vote so acceptance was 80%Monitoring IndicatorPrescribing Indicator Fourteen additional cause-and-effect sequences (i.e., tree branches) were identified for the four proposed causes. The cause-and-effect sequences for patient education identified during the monitoring indicator evaluation were judged to be common to the prescribing indicator and they were not diagrammed again. Ten of the fourteen branches were judged to be common to another prescribing (indicator 37). Eleven of the

PAGE 106

94 fourteen cause sequences from the asthma prescribing indicator were judged to be problems that were common the warfarin monitoring indicator. Table 6.15 presents the agreement among the team members for each branch. Table 6.16. Prescribing Indicator: Affinity Table and Rating of Importance RatingPatient level 4pt may not appreciate the function of steroid 1non compliance -intentional 1lack of tools (self assessment tool) 0taste bad 0adverse effect 2misperception of adverse effects of steroid 2caregivers not supervising med use Professional level 3medication copay misdiagnosis 6pt education 0Lack of objective measures 0lack of appropriate beta agonist use 3guideline adherence 4multiple physicians Organizational level 4medication management Environment 3compliance (related to embarrassment, barriers e.g., school, peer issues) Cause Sequences Common to both the Monitoring and Prescribing Nodes Table 6.17 displays the cause sequences from both the monitoring and prescribing indicator analysis. The first column labeled Branch identifies the node of the indicator formally evaluated and the branch number. For example: the M_1 sequence was discovered when the warfarin monitoring indicator was evaluated. A yes in the M&P column means the cause sequence was judged to be a common problem to both the monitoring and prescribing nodes. An M in the Node column signifies the cause sequence was judged to be unique to the monitoring node of the MUS.

PAGE 107

95 Twenty-three cause sequences were judge d to be common to both the monitoring and prescribing node. Two were unique to th e monitoring node and four did not reach consensus on either. The three main themes that appeared to be common to both the monitoring and prescribing node were patien t education, patient co mpliance (laboratory tests), and medication management systems.

PAGE 108

Table 6.17. Tree-diagram for Warfarin Monitoring and Asthma Prescribing Indicators Patient levelProfessional levelOrganization level EnvironmentBranchM & PNodeM_1patient educationpatients' knowledge not assesseddevelop tools (AHRQ)yesM_2patient educationlack of adequate teaching toolsMCO unification / standard assessmentyesM_3patient educationlack of adequate teaching toolspharm companies yesM_4patient educationdifficult to get education in Hospitaldischarge planningyesM_5patient educationdifficult to get education in Hospitallack of consistencyyesM_6patient educationtime issuesmarket forcesyesM_7patient educationcomorbidity takes more time yesM_8lack of information technologyyesM_9lack of systematic follow-up mechanismMM_10disease focused not drugMM_11not compliant with labMCO don't provide incentives to patientyesM_12not compliant with labtracking mechanism to see who gets labscentralized records/ reminder systemyesM_13not compliant with labMCO contact patientsyesM_14not compliant with labdisease managementyesM_15not compliant with labaccess/ convenience access/ convenienceyesP_1Not sensitive to medication copaylack of unfied formulary systemP_2Not sensitive to medication copaylack of software supportP_3some will adopt new techyesP_4pharmacists may helpyesP_5cultural disparityP_6physicians are afraid to adopt techmedication management to track patientsyesP_7pharm can take a role in med managementmedication management to track patientsyesP_8pharm may respond to the frequent usemedication management to track patientsyesP_9pharmacist involvementmedication management to track patientsyesP_10pharmacist involvementno paymentyesP_11following guidelineno rewardyesP_12data owner can disseminate infoyesP_13multiple physiciansunknown gatekeeperyesP_14multiple physiciansdata doesn't follow patient care 96

PAGE 109

CHAPTER 7 DISCUSSION The overall objective of this study was to explore the relationship between system design, patterns of drug therapy and drug related morbidity. This study had three major parts. The first part consisted of a database analysis. In it, we applied a set of medication use performance indicators (MU-PIs) to an administrative database in order to assess the overall quality of medications use in the population represented by the database. In the second part, a panel of experts described the association between selected indicators and the nodes of the medication use system (MUS). In the third part, an evaluation team carried out a cause-and-effect analysis on select indicator positives that had originated in the prescribing and monitoring nodes of the MUS. The first part replicated work by MacKinnon 42 and Faris. 13 The second and third parts, however, go beyond their work. The major contribution of this study was exploratory in two respects. It was the first attempt to associate medication use indicators with specific parts of the medication use system, and it was the first study to evaluate how the four levels of the health care system (patient, professional microsystems, organization, and environment) interact and contribute to indicator and node specific process problems. Database Analysis Prevalence Findings Perhaps the most notable finding from the database analysis was that the prevalence of PDRM positives was much lower than would be expected from Mackinnon 97

PAGE 110

98 and Faris studies. The period prevalence of PDRM positives (in members with a medical claim) for this study was 1.98 per 1,000 patient years, while Faris 13 found a prevalence rate of 62.5 per 1,000 patients and Mackinnon 59 found a prevalence rate of 28.8 per 1,000 patients. The lower prevalence found in the current study may be explained by differences in population demographics and differences in search algorithms for the MU-PIs. The previous studies used the MU-PIs to screen the database of an elderly Medicare population. Medicare patients tend to take more medications and the number of medications are associated with and increase risk of drug related injury. 59-61 This study required all events (i.e., PDRM positives) to have an ED visit or hospitalization for the outcome, and only the primary diagnosis was used for outcome identification. The previous studies did not have the same requirements. Their search algorithms did not require ED visits or hospitalizations for more than half of the indicators, and their analysis was not limited to primary diagnosis codes to signify the reason for seeking medical care (cause of ED visit or admission). Another difference in search algorithms between the studies was the attention given to the proximity of the process to the outcome. The search algorithms in this study required claims for the offending medication to have occurred shortly before the outcome--in most cases the lag was set at 100 days. Proximity requirements were also established for indicators that required monitoring. t The prevalence of PDRM positives that required hospital admission was much lower (0.75%; CI: 0.74%-0.76%) than the findings from our systematic review of t Refer to methods for search algorithms.

PAGE 111

99 preventable drug related admissions (median: 4.3%; IQR: 3.1%-9.5%). 1 The articles in the systematic review used a chart review process to identify PDRAs, while the current study employed a computer screening method. It has been shown that different methods of PDRM ascertainment yield different results. Jha et al. (1998), in an inpatient ADE study, compared yields of chart review, automated computer screening, and stimulated voluntary report. 62 This study found that each method identified different types of ADEs and the computer screening method only identified about forty-five percent of the total ADEs. Computer screening methods are believed to increase efficiency and the reliability of detecting PDRM, however, the yield is limited to the explicit PDRM scenarios. Implicit review methods allow for more flexibility and comprehensive assessments of PDRM. With the implicit chart review method, the limiting step is the knowledge and thoroughness of the reviewers and not the predefined search algorithms. Demographic Explanations for Prevalence Findings Information from the database was used to better understand indicator positives. Descriptive analyses of monitoring intervals were conducted to determine if cases on the margin of the monitoring interval were being identified as process positives, and multivariate logistic regression was conducted to explore the relationship between PDRM positives and demographic/system related variables. Distribution of monitoring intervals The MU-PIs that require drug-therapy monitoring identify cases as positive or negative for the inappropriate monitoring interval. When a case screens positive it means an individual did not have a laboratory test claim within the specified time frame. These strict cut-offs could allow monitoring intervals on the margins of appropriate monitoring to be categorized as process failures even if they had been only a few days or

PAGE 112

100 weeks over the required time. To develop a more comprehensive picture of how cases deviated from the specified monitoring intervals, an analysis of the first monitoring interval was conducted. The distributions of monitoring intervals from the first pharmacy claim to the first laboratory test claim for the indicators that required monitoring revealed that few process positives were captured from the margins of the required monitoring intervals. The indicators that required six-month and twelve-month monitoring intervals did, however, identify more cases on the margin than the others (6.5% and 12.6%, respectively). Because the categorized intervals were a function of the monitoring requirement u the indicators with longer monitoring intervals had a larger interval for the descriptive analysis and a more liberal interval on the margin of the required monitoring interval. For example: the categories for indicators with a one-year monitoring requirement were set at 180 day intervals, while the categories for indicators with a one-month monitoring requirement were set at fifteen days. Further investigation is needed to drill down the margins of the six and twelve month monitoring indicator positives. Nevertheless, the majority of process positives resulted from lag times that were on the extreme of the distribution, in many cases, no claims for the specified labs were observed. Multiple logistic regression analysis In the exploratory analysis, age, number of office visits, pharmacies, prescribers, drug class, and number of medical conditions were all significant bivariate predictors of PDRM positives. In multivariate analysis, however, only gender, number of office visits, drug classes and medical conditions were independent predictors. Males, number of drug u Descriptive categories were defined by dividing the defined monitoring interval by two. See methods.

PAGE 113

101 classes and number of medical conditions had a positive relationship with PDRM positives, while number of office visits and the interaction between drug classes and medical conditions were negatively associated with PDRM positives. Interestingly, gender was not significantly associated with PDRM positives in bivariate analysis; however, in multivariate analysis gender was an independent risk factor. The only system related variable that stayed in the multivariate model was number of office visits. After adjusting for the other variables in the model, as the number of office visits increased the risk of PDRM decreased in this population. Following up with patients who appear to be missing office visits or who have not had an office visit for an extended period of time may reduce the probability of drug related injury. Number of different pharmacies and number of different prescribers were not independent predictors of PDRM positives. Another interesting finding was that an increase in the number of medical conditions (with number of drug classes held constant at zero) increased the odds for PDRM by about twenty-nine percent. Likewise, an increase in the number of drug classes (with medical conditions held constant zero) increased the odds for PDRM by about twenty-two percent. The interaction effect of medical condition and number of drug classes, however, had a significant but weak negative association, meaning as number of medical conditions and number of drug classes increase together the odds of PDRM decreased. This seems counterintuitive at first; however, a reasonable explanation exists. Increases in drug therapy without an increase in medical conditions may increase the risk of PDRM through over treatment, while an increase in medical conditions without an increase in drug therapy may increase the risk of PDRM due to

PAGE 114

102 under-treatment. Increases in medical conditions with complementary increases in drug therapy may indicate appropriate treatment and therefore reduce the risk for PDRM. Most studies that have looked at independent predictors of PDRM have done so in the inpatient setting. These studies have also found number of medications and number of diseases as independent risk factors, but have failed to find age or gender as independent predicators. 61,63 Three studies have used multivariate statistics to evaluate predictors of PDRM in the ambulatory care setting. Mackinnon (2003), found four or more prescribers, four or more medical conditions, female gender, antihypertensive drug use and six or more prescription medications were risk factors for PDRM. Faris (2001) confirmed these findings. In addition, a recent study by Gandhi et al (2003), found that number of medications were the only independent risk factor of adverse drug events in ambulatory care. 60 Considering the finding from this study in the context of previous research, it would seem prudent to closely monitor patients who have multiple medical conditions or who require multiple medications for signs and symptoms that would indicate problems with drug therapy. Node Identification of MU-PIs The ultimate goal of this dissertation was to identify system factors that appeared to be common or unique to nodes of the MUS. To do this, the location in the MUS where the indicators process failure originated had to be assessed. Fourteen indicators were sent to Delphi members to identify the node of the MUS where the process failure from specific indicators originated. The Delphi study began before the database analysis was completed. The fourteen indicators were selected based

PAGE 115

103 on preliminary prevalence findings. The final results of the MU-PIs differed from the preliminary findings and only nine of the fourteen had PDRM positives. Thirteen of the fourteen indicators were assigned to either the prescribing or the monitoring node of the MUS. The assignment followed a clear pattern. All indicators that required drug therapy monitoring except, Delphi indicator 14 v (which did not reach consensus) were assigned to the monitoring node. Indicators where the process problem involved disease-drug interactions or lack of needed therapy were assigned to the prescribing node. The process component of Delphi indicator 14 included the use of inappropriate medications in patients taking warfarin and no lab for INRs done within five days, the node assignment was split between the prescribing and monitoring node. Cause-and-Effect Analysis The cause-and-effect evaluation team accomplished multiple tasks. First, it identified likely causes for the process failures and PDRMs described by select MU-PIs. Second, it sketched out some cause sequences across the four system levels (patient, microsystem, organization, and environment). Out of the apparent complexity of causes and system levels, however, the team found that many cause sequences explained more than one indicator and problems from more than one node of the MUS. The final task was a discussion of possible corrective actions. During that discussion, four themes emerged that would explain some general latent failures and some broad strategies about how to improve medication use. The evaluation team was able to use the information from the MU-PIs to reflect on how the different levels of the health care system interact to influence patient care v MU-PI number 42

PAGE 116

104 processes. The team identified twenty-nine cause sequences. Furthermore, they found that twenty-three of the twenty-nine were common to both prescribing problems and monitoring problems. That is, the team decided that the same cause sequences could contribute to prescribing process failures and to monitoring process failures. The team did not reach consensus on four sequences. (See Table 6.19) The cause sequences involving patient is not compliant with getting labs (M11-M16 from Table 6.19) were originally proposed as a partial explanation for a monitoring process failure from the warfarin indicator. Somewhat surprisingly, the evaluation team found that this sequence also could be relevant to prescribing process failures. The evaluation team may have focused on the compliance component of the proposed cause and the higher level (professional, organizational and environmental) causes in the sequence when judging their relationship to the prescribing node. Cause Themes In the latter part of the analysis, four themes emerged from the cause sequences that were common to the prescribing and monitoring nodes: patient education (teaching and assessment tools), information demands and medication management, pharmacist involvement and guideline adherence. Many cause sequences included, lack of patient education. These sequences involved lack of adequate teaching and assessment tool, insufficient discharge education, and time constraints. At the professional level, lack of adequate teaching tools and lack of tools to assess patients understanding of their drug therapy were identified as contributing to patient knowledge deficits. The evaluation team believed that managed care has not promoted and reinforced standard methods for education and assessment (organizational level). They also believed that pharmaceutical manufacturers and

PAGE 117

105 agencies such as the Agency for Health Care Research and Quality (environmental level) are not investing in the development and dissemination of valid tools to educate and assess patient knowledge. According to the evaluation team, deficits in patients understanding of their drug therapy may occur when patients are discharged from the hospital with new medications. Lack of discharge planning and consistency in medication consults were the organizational level explanations for this. Time constraints due to market forces (organizational level) were also considered reasons for patient education deficits. Lack of a technology infrastructure with the ability to record patient information through the continuum of care was considered a latent condition that allowed problems to occur and persist in the prescribing and monitoring nodes. Lack of medication management systems that could be used to determine if patients are refilling medications, having their laboratory tests ordered, completed and evaluated was also considered a common problem to the prescribing and monitoring nodes. An effective medication management system would include access to centralized records that could support a tracking mechanism that providers could use to evaluate their patients drug use. The team also concluded that the managed care organizations (MCOs) affiliated with the coalition were not actively evaluating patient behavior in response to medication refills and compliance with laboratory test orders. The evaluation team believed the MCOs should identify members who are not adhering to their medication or laboratory test orders and contact them to discuss the consequences of their decisions. Lack of pharmacist involvement was also considered a system related cause that allowed problems to persist from the prescribing and monitoring nodes. The evaluation

PAGE 118

106 team believed that pharmacists could take a more significant role in medication management by responding to overuse of medications and by supporting drug monitoring. They believed pharmacists are not providing consultative services because they are not paid for consultative services. Failure to follow guidelines also contributed to problems that originate in the prescribing and monitoring nodes of the MUS. The MU-PIs were evidence based and the process component of the indicator represents deviations from guidelines. The evaluation team believed the deviations from the guidelines were because providers have no financial incentive to adhere to guidelines. One of the benefits of the cause-and-effect analysis was that it promoted the recognition that indicator-specific findings may be inter-related with common system failures, i.e., latent conditions. The team recognized the close relationships among the four causal themes described above and possible corrective interventions. The system related themes that emerged from the cause-and-effect analysis are extremely complex. Further investigation into them is warranted before interventions to promote organizational change are developed and implemented. Conclusion The MU-PIs were used to screen the study database. The period prevalence for process positives was 209.84 (206.37-213.34) per 1,000 member years. The period prevalence for PDRM positives was much lower, 1.98 (1.6-2.4) per 1,000 member years. The PDRM positive rate from this study was much lower than the previous studies using the MU-PIs. The lower rate may be due to the much younger study population and differences in search algorithms.

PAGE 119

107 Descriptive analyses of monitoring intervals were conducted to determine if cases on the margin of the monitoring interval were being identified as process positives. A small portion of process positives resulted from the margins of indicators with six-month and twelve-month monitoring requirements. The majority of process positives, however, were a result of intervals that were at least three times greater than the required interval. Multivariate logistic regression was conducted to explore the relationship between PDRM positives and demographic/system related variables. Gender, number of office visits, number of drug classes and number of medical conditions were independent risk factors for PDRM. Males were about 50% more likely to have a PDRM positive after adjusting for the other variables. Each additional office visit decreased the risk of PDRM by about seven percent. When drug class is held constant, each additional medical condition increased the chance of being PDRM positive by about twenty-nine percent. When medical conditions are held constant, each additional drug class increased the chance of having a PDRM positive by about twenty-two percent. Number of drug classes and number of medical conditions had a significant interaction with a negative beta-coefficient. Increases in both medical conditions and drug classes resulted in a decreased risk for a PDRM positive. Indicators were assigned to different nodes of the MUS. To do this, a Delphi panel judged the degree of association between select indicators and specific nodes of the MUS where the process failure may have originated. Six of the fourteen indicators were assigned to the prescribing node and seven were assigned to the monitoring node of the MUS. One indicator did not reach significance within the two round Delphi study.

PAGE 120

108 The MU-PIs proved to be a useful tool to initiate system thinking or organizational introspection. The team identified twenty-nine cause sequences. Furthermore, they found that twenty-three of the twenty-nine were common to both prescribing problems and monitoring problems. Four themes emerged from the cause-and-effect analysis and they included lack of necessary tools for adequate patient information and assessment, an information system that can track patients and relay information to the providers, pharmacist involvement in the MUS and guideline adherence. The evaluation team determined that interventions should address these themes. Further investigation is needed to design interventions that would have the greatest impact. Limitations The MU-PIs are content validated, however; they have not been criterion validated. Therefore, MU-PI positives represent possible PDRM, but specificity and sensitivity are not known. This should be considered when interpreting the indicator findings. Published literature shows that claims data are reasonably valid for identifying patients with specific disease, especially when pharmacy claims that would be consistent with the disease are included, 35 nevertheless; no studies were identified that evaluated the validity of using CPT codes as proxies for lab monitoring. In the database analysis, the majority of process positives for indicators that required drug therapy monitoring resulted from lag times that were on the extreme of the distribution, in many cases, no claims for the specified labs were observed. Because of this, further analysis is needed to determine if labs were conducted but no claims were submitted. During the cause-and-effect analysis, one of the team members argued that lab claims may be bundled and therefore specific lab claims would not be observed.

PAGE 121

109 The search algorithms for the MU-PIs were limited to severe outcome, i.e., ED visit or hospitalization. Studies have shown that a large portion of PDRM is considered significant but not severe. It is unclear if the latent conditions identified in this study would be involved in the production of less severe PDRM. The cause-and-effect analysis was essentially a case study. It is possible that an evaluation team with a different skill set or expertise may have identified different system failures. Nevertheless, it would not invalidate the findings of this study. Additional studies may be useful to develop a more comprehensive picture of all possible latent conditions. Significance and Theoretical Contribution This study included developing a modified version of Faris search algorithm, identifying indicators that represent different nodes of the MUS, identifying latent conditions of the health care system that are associated with both prescribing and monitoring problems, and demonstrating that the MU-PIs can be used to initiate organizational introspection. The search algorithms were refined from earlier versions by paying more attention to the time relationships between the process and outcome. The search algorithms were intended to identify cases that were currently taking the offending medication within proximity to the adverse outcome. The search algorithm also had more strict limitations on the indicators that required drug therapy monitoring. This study demonstrated that medication and disease specific indicators can be grouped according to the nodes of the system where the process failure originated. Node specific indicators can be used to probe into the MUS to evaluate node-specific quality, e.g, the quality of monitoring. This study also established that problems that originate in

PAGE 122

110 the monitoring and prescribing phase often have common system characteristics that allow the process failures to eventually end in adverse outcomes. Lastly, this project demonstrated that the MU-PI can be used to initiate dialog and introspection about how the design of the organization contributes to the prevalence of process failures and PDRM. The MU-PIs can function as an important tool for the improvement of medication use. Contribution to Health Care The connection between medication use quality measurement and improvements in the MUS can take two routes: 1) improvements through selection and 2) improvement through changes in care. 64 The use of performance measurement by organizations such as the National Committee for Quality Assurance (NCQA) are designed to provide a report card for employers so they can select the providers and organizations that appear to have the highest level of quality. The mechanism of improvement through selection works when the employers can make choices among health plans or providers. Selection as a mechanism for improvement does not change the basic distribution of performance. Instead it improves outcomes by shifting business to providers and plans that are performing better. 64 Improvements through changes in care have the potential to shift the underlying distribution of care itself. Before changes can be made to the MUS the organization needs to understand the process involved in the production of poor quality. Once the performance is measured and the influence of system designs on quality is understood organizations and individuals can improve performance through changing the processes of care. This study has contributed to health care by providing a method to initiate the

PAGE 123

111 change process by identifying latent conditions of the system and areas for process improvement interventions. Future Areas for Study Interventions The next step in the change process will focus on the development of interventions to address the system related problems identified in the cause-and-effect analysis. A more in-depth analysis of the organization and its ability to pursue various interventions is needed. Information from the literature can help us frame the latent conditions to better understand them. The evaluation team believed that issues related to patient education were important reasons for members neglecting to use their medications appropriately. They believed methods should be developed and implemented to reduce the educational deficits. Research on decision aids for patients facing health treatment decisions may provide valuable insight and tools for educating patients about the risks and benefits related to their drug therapy. It may, as well, assess their competency about medication specific issues. Decision aids are intended to help patients understand the probable outcomes of different behavioral decisions, e.g., not complying with ordered drug therapy monitoring, make an informed decision and communicate their preferences to their provider. 65 Decision aids have been found to improve knowledge, reduce decisional conflicts, and stimulate patients to be more active in decision making without increasing their anxiety. 66 Since lack of patient education and assessment was an important latent condition, future research should be committed to evaluating how education and assessment tools, such as decision aids can reduce the PDRM problem.

PAGE 124

112 Guideline adherence was believed to be a system problem that was common to the prescribing and monitoring nodes as well. The evaluation team believed the lack of a reward system tied to guideline adherence was a reason for the deviations from guidelines. Cubana et al (1999), conducted a systematic review of barriers to physician adherence to practice guidelines and found multiple barriers exist for physicians to follow guidelines. 67 Lack of awareness and lack of familiarity with guidelines affect physician knowledge of guidelines. Lack of agreement with guidelines, self-efficacy, outcome expectancy, and inertia from previous practice can pose attitude barriers. Patient and environmental barriers were common external barriers. Any attempt to improve guideline adherence should consider the common barriers. Interventions need to address the specific cognitive and behavioral barriers to guideline adherence. Lack of pharmacists recognizing and responding to drug therapy problems was also considered a common system failure. Hepler and Strand have led the movement to expand the role of community pharmacists from simply packaging and dispensing to detecting drug therapy problems and consulting with patients and other health professionals. 18 Financial barriers and business model are believed to have limited the adoption of pharmaceutical care. 68 The Medicare prescription drug improvement and modernization act of 2003 may remove these barriers. In the bill, pharmacists are designated as eligible Medicare providers by virtue of the plan paying for their services. Starting in 2006, pharmacists will be able to bill for medication therapy management in Medicare patients. Organizations, such as the coalition, should consider how they can take advantage of this financial arrangement and extend pharmacists medication therapy management to employer sponsored health plans.

PAGE 125

113 The lack of a centralized electronic infrastructure that would allow records to be shared across the continuum of care and used for medication management was also considered an important contributor to process problems from the prescribing and monitoring nodes. Development and implementation of electronic infrastructure is logistically difficult and expensive. Many health care systems are hesitant to invest in an electronic infrastructure. In the 1990s many health stystems invested hundreds of millions of dollars in information systems that were supposed to link providers and institutions, but often failed to achieve their objectives. 64 Nevertheless, decision makers and politicians are becoming interested in developing an electronic clinical infrastructure. Senator, Edward Kennedy introduced a bill to the senate on May 13, 2004 to modernize health care. A major component of the bill was to provide funding for the development of electronic medical records. Furthermore, President Bush recently called for all patient information to be available electronically by 2014. He claims the federal government will set the standards for the conversion to electronic records. Once this technology is adopted, careful attention must be given to measuring barriers to use and unintentional consequence of the technology. MU-PI Fidelity Further work is needed to refine the technical aspects of the medication use performance indicators. This would include updating the MU-PI to include newer therapies and newer treatment evidence, evaluating the criterion validity of the MU-PIs, and adapting the search algorithms to include clinical data along with claims data. New drugs have been entering the market and new evidence for medical treatment is becoming more abundant. Indicators need to be specifically constructed to represent nodes other than prescribing and monitoring. Indicators that represent problems in the administration

PAGE 126

114 and dispensing nodes were not represented in this study and are needed for a comprehensive assessment of the medication use system. Careful and meaningful analysis of the more recent literature is needed to add new performance indicators and to remove existing indicators that do not represent failures in the management of drug therapy. The criterion validity of the indicators is unknown. To validate the MU-PIs each indicator would need enough positives for statistical evaluation. This population should include a representative proportion of patients over sixty-five year old. Attention should be given to determining the positive predictive value, sensitivity and specificity of the indictors. A risk factor study is needed to better understand the population and system variables most associated with having process failures and PDRM. The analysis should include a separate evaluation for process positives and PDRM positives to determine what characteristics produce greater risk for drug related injury. The analysis should also be separated by nodes of the MUS. It would be interesting to determine if the characteristics that predict PDRM for monitoring problems are different from the characteristics that predict PDRM related to problems in the prescribing node.

PAGE 127

APPENDIX A MEDICATION INVOLVED IN DRA BY STUDY

PAGE 128

Cardiovascular Diuretics916 -5.613.51316.763.21212 -10.512.510Digoxin -16.7 -44 -43Nitrates -5.6 -4.2 -1 -4.23Antihypertensive -9 -2.8 -18.514.69.921022.9 -9.958Anticoagulants/Antithrombotics -13 -5.66.3 -18 -15.8135Cardiovascular / other16 -40 -10.89.3 -1185.7 -10.872538401424.346.458.573.1206228.626.333.312PsychotropicAntipsychotic -1.33 -2.152Antidepressant -4.62 -3.32Lithium -6.3 -1 -3.652Anxiolitic, Sedative, Hypnotic -2.8 -7 -4.92Phychotropic/other -20 -9.5 -18 -15.89.55 -202.89.5 -6.35.9148 -15.88.758Anti-inflammatoryCorticosteroids -3.313.9 -11.1 -5 -11.4 -11.15NSAIDs -131041.75.49.310.42.614 -10.28Anti-inflammatory -3.3910 -7.98.454 -1313.355.65.420.410.45.9281011.47.911.411AnalgesicsNarcotic Analgesics -8.3 -3.966 -3.965 -8.3 -3.966 -3.965Anti-diabeticHypoglycemic agents27 -16.25.620.8 -10414.32.612.15827 -16.25.620.8 -10414.32.612.158Anti-infectiveAntibiotics -136.7 -12.2 -4.2 -8 -5.713.287Anti-asthmatics -3.3 -9.3 -1.3 -5.7 -4.54GI -10 -9.5 -9.752Antineaplastic -3.3 -2.79.3 -17.16.366.365Anti-epileptics -2.79.3 -11.4 -9.33Non-specificOther48363.319.417.6 -9.914105.823.615.81010010099.9100.1100.1100.3100.210010010010099.6631Dartnell 1996Hallas 1991MedianCountCourtman 1995Lindley 1992Bigby 1987Lakshmanan 1986Trunet 1986Darchy 1999Nelson 1996Raschetti 1999Ng 1999Cunningham 1999 115

PAGE 129

APPENDIX B CLASSIFICATION OF DTPS INTO NODES OF THE MUS

PAGE 130

117 Dartnel, DTPsFrequenc y % occupiedClassDartnelPrescribing factor1130.56%PP30.56%Non compliance1541.67%ACDEadverse drug reaction (preventable)1027.78%MAC41.67%36100.00%M27.78%non-id-p100.00%AD R Lakshmanan, DTPsFrequenc y % occupiedClass7 admissions related to antihypertensive agents*741.18%M2 aminophylline and phenytoin**211.76%MP5.88%2 aminophylline and phenytoin***211.76%MDE1 high dose of prednisone in a known diabetic who developed ketoacidosis15.88%PAC11.76%1 rotator cuff tear after five intra articular steroid injections15.88%Non-id-pM76.47%2 hypoglycemia secondary to taking too much insulin****211.76%ACnon-id-p5.88%2 admissions, results of a patient taking chlorambucil on an unsupervised basis over several years211.76%M100.00%17100.00%AD R 35 cases were drug mediated adverse effects. This article does not differentiate 'drug-mediated adverse effects' and 'drug mediated adverse reactions.** Toxic reaction where levels were available and medication continued*** Toxic reaction could have been detected with the patients examined as outpatients Courtman, DTPsFrequenc y % occupiedClassCourtmanInappropriate dose3558.33%PP61.67%Avoidable or possibly avoidable ADR*1423.33%M DEnon-compliance915.00%ACAC15.00%Drug interaction**11.67%PM23.33%Lack of therap y 11.67%Pnon-id-p60100.00%100.00%* Among 16 ADR, author reported that 14 cases were avodable or possibly avoidable. We classify avoidable/possibly avoidable ADRs as M.AD R *** Author cited Strand et al (1990) and said, 'the patient is taking a drug for whichthere is no medical indication; and, the patient is receiving the wrong drug or drug product could not be accurately assessed from the medical chart alone at the time of admission and were, therefore, not recorded. Author gave examples of avoidable cases as inappropriate or contraindicated drug therapy, non-compliance. ** monitoring for drug interactions and ensuring drugs are discontinued by the prescribing physician when they are no longer required by the patient.LakshmananAuthor says that 19 cases are preventable drug related admissions. But only 17 case information is provided. evolving over weeks to months such that the victims could have been examined as outpatients and had their therapy altered****1 had been having frequent symptoms but did not contact a physician, 1 was having frequent symptoms that were neither volunteered nor e

PAGE 131

APPENDIX C SUMMARY: CLASSIFICATION OF DTPS INTO NODES OF THE MUS NodesDartnellLakshmananCourtmanBeroHallasBigbyDarchyNikolausOf the MUS19991986199519911992198719991992Prescribing30.56%5.88%61.67%59.46%37.31%5.26%53.33%0.00%Dispensing0.00%0.00%0.00%0.00%0.00%0.00%0.00%0.00%Administration41.67%11.76%15.00%27.03%20.90%25.00%0.00%23.68%Monitoring27.78%76.47%23.33%0.00%35.82%44.74%46.67%10.53%Non-identifiable process0.00%5.88%0.00%13.51%5.97%25.00%0.00%65.79%Non-preventable ADR34.55%0.00%3.23%22.92%74.06%0.00%26.83%0.00%Total100.00%100.00%100.00%100.00%100.00%100.00%100.00%100.00% 118

PAGE 132

APPENDIX D MEDICATION USE PERFORMANCE INDICATOR DEFINITIONS 1. This outcome has occurred after the pattern of care below: ER visit/hospitalization due to depression This is the pattern of care: 1. History/diagnosis of depression 2. Use of long-acting benzodiazepine (e.g., Librium, Valium, Centrax, Paxipam, Dalmane, Azaene/Tranxene, etc.) 2. This outcome has occurred after the pattern of care below: ER visit/hospitalization due to theophylline toxicity This is the pattern of care: 1. Use of theophylline 2. Drug level not done at least every 6 months 3. This outcome has occurred after the pattern of care below: ER visit/hospitalization due to bipolar disorder. This is the pattern of care: 1. Diagnosis/history of bipolar disorder 2. Use of lithium 3. Drug level not done at least every three months 4. This outcome has occurred after the pattern of care below: ER visit/hospitalization due to major and/or minor hemorrhagic event This is the pattern of care: 1. Use of IV heparin 2. PTT not done at least every day 5. This outcome has occurred after the pattern of care below: ER visit/hospitalization due to gastritis and/or upper GI bleed and/or upper GI perforation and/or GI ulcers and anemia This is the pattern of care: 1. History/diagnosis of ulcers and/or GI bleeding 2. NSAID use for at least 1 month (not including Cox-2) 119

PAGE 133

120 6. This outcome has occurred after the pattern of care below: ER visit/hospitalization due to depression This is the pattern of care: 1. History/diagnosis of depression 2. Use of a barbiturate (e.g., butalbital) 7. This outcome has occurred after the pattern of care below: ER visit/hospitalization due to depression This is the pattern of care: 1. History/diagnosis of depression 2. Use of a sympatholytic antihypertensive (e.g., reserpine, methyldopa, clonidine, etc.) 8. This outcome has occurred after the pattern of care below: ER visit/hospitalization due to seizure activity This is the pattern of care: 1. Use of anticonvulsant requiring drug level monitoring (e.g., phenytoin, carbamazepine, valproic acid) 2. Drug level not done at least every 6 months 9. This outcome has occurred after the pattern of care below: ER visit/hospitalization due to lithium toxicity This is the pattern of care: 1. Use of lithium 2. Lithium level not done every 3 months 10. This outcome has occurred after the pattern of care below: ER visit/hospitalization due to a major/minor hemorrhagic event This is the pattern of care: 1. Warfarin use 2. NSAID use (e.g., diclofenac, ibuprofen, ketoprofen, etc.) 3. INR not done within 10 days 11. This outcome has occurred after the pattern of care below: ER visit/hospitalization due to hypothyroidism This is the pattern of care: 1. Lithium use for at least 6 months 2. TSH not done at least every 6 months

PAGE 134

121 12. This outcome has occurred after the pattern of care below: ER visit/hospitalization due to blood dyscrasias/thrombocytopenia This is the pattern of care: 1. Use of ticlopidine (Ticlid) 2. CBC/platelets not done every 2 months 13. This outcome has occurred after the pattern of care below: ER visit/hospitalization due to acute urinary retention This is the pattern of care: 1. Diagnosis/history of bladder atony due to diabetes 2. Use of imipramine 14. This outcome has occurred after the pattern of care below: ER visit/hospitalization due to acute respiratory failure This is the pattern of care: 1. History/diagnosis of severe COPD 2. Use of medium to long-acting benzodiazepines 15. This outcome has occurred after the pattern of care below: ER visit/hospitalization due to acute urinary retention This is the pattern of care: 1. History/diagnosis of benign prostatic hypertrophy (BPH) 2. Use of an anticholinergic agent 16. This outcome has occurred after the pattern of care below: ER visit/hospitalization due to congestive heart failure and/or fluid overload This is the pattern of care: 1. History/diagnosis of high blood pressure (over 140/90) and/or congestive heart failure 2. NSAID use for at least 3 months 17. This outcome has occurred after the pattern of care below: ER visit/hospitalization for hypokalemia This is the pattern of care: 1. Use of NONpotassium SPARING diuretic (e.g., hydrochlorothiazide, etc.) 2. No concurrent use of potassium supplement 3. Electrolytes not checked at least every 2 months initially, then every 6 months 18. This outcome has occurred after the pattern of care below: ER visit/hospitalization due to anticonvulsant drug toxicity

PAGE 135

122 This is the pattern of care: 1. Use of an anticonvulsant requiring drug level monitoring (e.g., phenytoin, carbamazepine, valproic acid) 2. Drug level not done at least every 6 months 19. This outcome has occurred after the pattern of care below: ER visit/hospitalization due to hypothyroidism This is the pattern of care: 1. Use of thyroid or antithyroid agent (e.g., levothyroxine, propylthiouricil, etc.) 2. T4/TSH not done at least every 12 months 20. This outcome has occurred after the pattern of care below: ER visit/hospitalization due to aminoglycoside toxicity (acute renal failure and/or renal insufficiency and/or vestibular damage and/or auditory damage) This is the pattern of care: 1. Use of an aminoglycoside 2. Serum creatinine not done before and after therapy (and if therapy longer than 7 days, not done at least every 7 days) 3. At least one drug level not done 21. This outcome has occurred after the pattern of care below: ER visit/hospitalization due to COPD This is the pattern of care: 1. Diagnosis/history of COPD 2. Use of a beta-blocker (e.g., propranolol, etc.) 22. This outcome has occurred after the pattern of care below: ER visit/hospitalization due to Hypertension/tachycardia This is the pattern of care: 1. History of hypertension 2. Prescription use of sympathomemetic decongestants 23. This outcome has occurred after the pattern of care below: ER visit/hospitalization due to hyperkalemia This is the pattern of care: 1. Use of ACE inhibitor 2. Electrolytes/CBC not done at least every 6 months 24. This outcome has occurred after the pattern of care below: ER visit/hospitalization due to blood dyscrasias and/or hyponatremia and/or excessive water retention and/or syndrome of inappropriate antidiuretic hormone (SIADH)

PAGE 136

123 This is the pattern of care: 1. Use of carbamazepine 2. Electrolytes/CBC not done at least every 6 months 25. This outcome has occurred after the pattern of care below: ER visit/hospitalization due to acute renal failure and/or renal insufficiency This is the pattern of care: 1. Use of lithium 2. BUN/serum creatinine not done at least every 3 months 26. This outcome has occurred after the pattern of care below: ER visit/hospitalization due to digoxin toxicity This is the pattern of care: 1. Use of digoxin 2. BUN/serum creatinine not done at least every 6 months 3. Digoxin level not done at least every 6 months 27. This outcome has occurred after the pattern of care below: ER visit/hospitalization due to gastritis and/or upper GI bleeds and/or GI perforations and/or GI ulcers and anemia This is the pattern of care: 1. History/diagnosis of ulcers and/or gastrointestinal bleeding 2. Use of an oral corticosteroid (e.g., prednisone) for at least 3 months 28. This outcome has occurred after the pattern of care below: ER visit/hospitalization due to congestive heart failure and/or heart block This is the pattern of care: 1. History/diagnosis of congestive heart failure with heart block or advanced bradycardia 2. Use of digoxin 29. This outcome has occurred after the pattern of care below: ER visit/hospitalization due to acute renal failure and/or renal insufficiency This is the pattern of care: 1. Use of an ACE inhibitor 2. BUN/serum creatinine not done at least every 6 months 30. This outcome has occurred after the pattern of care below: ER visit/hospitalization due to major and/or minor hemorrhagic event

PAGE 137

124 This is the pattern of care: 1. Use of warfarin 2. INR not done at least every month 31. This outcome has occurred after the pattern of care below: ER visit/hospitalization due to secondary myocardial infarction This is the pattern of care: 1. History/diagnosis of myocardial infarction 2. No use of ASA and/or beta blocker (e.g., metoprolol) 32. This outcome has occurred after the pattern of care below: ER visit/hospitalization due to a major/minor hemorrhagic event This is the pattern of care: 1. Warfarin use 2. Antibiotic use (e.g., Bactrim, etc.) 3. PT not done within 5 days 33. This outcome has occurred after the pattern of care below: ER visit/hospitalization due to acute renal failure and/or renal insufficiency This is the pattern of care: 1. Use of allopurinol 2. BUN/serum creatinine not done at least every 6 months 34. This outcome has occurred after the pattern of care below: ER visit/hospitalization due to depression and/or increase in dosage of antidepressant This is the pattern of care: 1. History/diagnosis of depression 2. Use of moderate to high lipophilic beta-adrenergic blocking agent (e.g., propranolol, pindolol, etc.) 35. This outcome has occurred after the pattern of care below: ER visit/hospitalization due to patient fall and hip fracture This is the pattern of care: 1. Patient on alpha blocker 2. Standing BP not checked within 2 months of initiation of therapy 36. This outcome has occurred after the pattern of care below: ER visit/hospitalization due to fall and/or hip fracture and/or other bone fracture and/or bone break

PAGE 138

125 This is the pattern of care: 1. 65 years or older 2. Use of long half-life hypnotic/anxiolytic (e.g., flurazepam, diazepam, chlordiazepoxide, etc.) 37. This outcome has occurred after the pattern of care below: ER visit/hospitalization due to congestive heart failure This is the pattern of care: 1. Diagnosis/history of congestive heart failure 2. Not on an ACE inhibitor or ARB (e.g., captopril, enalapril, etc.) 38. This outcome has occurred after the pattern of care below: ER visit/hospitalization due to congestive heart failure This is the pattern of care: 1. History/diagnosis of congestive heart failure 2. Use of an antiarrhythmic agent (e.g., disopyramide, procainamide, etc.) 39. This outcome has occurred after the pattern of care below: ER visit/hospitalization due to asthma This is the pattern of care: 1. Diagnosis of moderate to severe asthma 2. Use of a bronchodilator 3. No use of maintenance therapy (e.g., beclomethasone, etc.) 40. This outcome has occurred after the pattern of care below: ER visit/hospitalization due to fall and/or hip fracture and/or other bone fracture and/or bone break. This is the pattern of care: 1. 65 years of older 2. Use of tricyclic antidepressant (e.g., amitriptyline, doxepin, imipramine, etc.)

PAGE 139

APPENDIX E PROCEDURE CODES USED TO IDENTIFIY OFFICE VISITS AND EMERGENCY DEPARTMENT VISITS FROM PROFESSIONALS CLAIMS DATA CPT Code Office VisitCPT CodeEmergency Department Visits99201Office Visit,New,Focused,10#99281Emerg Dept.Visit, Focused99202Office Visit,New,Expanded,20#99282Emerg Dept.Visit,Expanded,Low Complexity99203Office Visit,New,Detailed,30#99283Emerg Dept.Visit,Expanded,Lo-Mod. Complexity99204Office Visit,New,Mod. Complex,45#99284Emerg Dept.Visit, Detailed99205Office Visit,New,Hi Complex,60#99285Emerg Dept.Visit, Comprehensive99211Office Visit,Estab,Minimal,5#99212Office Visit,Estab,Focused,10#99213Office Visit,Estab,Expanded,15#99214Office Visit,Estab,Detailed,25#99215Office Visit,Estab,Comprehens,40#99241Office Consultation, Focused,15#99242Office Consultation, Expanded,30#99243Office Consultation, Detailed,40#99244Office Consultation,Mod. Complex,60#99245Office Consultation,Hi Complex,80# 126

PAGE 140

APPENDIX F CODING SOLUTION FOR EACH MEDICATION USE PERFORMANCE INDICATOR Drug-Disease InteractionMonitoringDrug-Drug InteractionInd 01Ind 02Ind 10Ind 05Ind 03Ind 17Ind 06Ind 04Ind 32Ind 07Ind 08Ind 48Ind 13Ind 09Ind 14Ind 11Ind 15Ind 12Ind 16Ind 18Ind 21Ind 19Ind 22Ind 20Ind 27Ind 23Ind 28Ind 24Ind 31Ind 25Ind 34Ind 26Ind 36Ind 29Ind 37Ind 30Ind 38Ind 33Ind 40Ind 35 127

PAGE 141

APPENDIX G RECRUITMENT LETTER FOR NODE IDENTIFICATION STUDY Dear Dr. The University of Florida and the Central Florida Health Care Coalition are conducting a study that aims to evaluate the underlying system factors involved in errors that originate at different nodes of the medication use system in primary care. The goal is to identify errors that originate at specific nodes of the medication use system and then to establish system factors that appear to be common and unique to the nodes. (This is Brian Sauers Ph.D. dissertation research.) This study includes the following three steps: (1) assign the preventable adverse drug event (pADE) scenarios to the nodes of the medication use system where the process failure originated, (2) translate the pADE scenarios to billing codes and query the coalitions database using the pADE indicators, and (3) conduct cause and effect diagramming (with a group of providers affiliated with the coalition) to uncover latent system conditions that may be causing or contributing to errors occurring in the medication use system. We have written to you because of your expertise in medication error research. As you know, this field has relatively few real opinion leaders, so your assistance and subsequent participation would be invaluable. We are soliciting your help in two ways. First, please nominate other experts in the field of medication safety so that we may invite them to serve on our Delphi panel. We are aware of researchers currently publishing in this field, but we need your advice about which health care providers and researchers are trusted experts in medication error research. We are seeking a panel of 10-20 members. We will send a letter similar to this to each person you nominate. This would require only a few minutes of your time to write names and contact information on a form and return it to us. Second, we would like you to participate on the Delphi panel itself. This panel will match up 15 pADE scenarios with the nodes of the medication use system where the process failure likely originated. The first round of the Delphi process should take approximately one hour to complete, and the subsequent two to three rounds are not expected to exceed 30 minutes. Appended is a participation and recommendation form. Please indicate whether you are willing to participate by circling yes or no, and please provide names and contact information for individuals that you believe have expertise in medication errors. Brian will telephone you in a few days to follow up and answer any questions. You participation will be greatly appreciated. Respectfully, Brian C. Sauer Charles D. Hepler 128

PAGE 142

129 1. Will you participate in the Delphi panel that will assign the pADE scenarios to nodes of the medication use system? (yes or no) Please sign name 2. If you agree to participate would you like to complete the survey electronically or would you rather receive a paper copy? Please circle one: (electronic copy or paper copy) 3. Please list any health care professionals or researchers, and if possible, a form of contact. a. ________________________________________________________________ b. ________________________________________________________________ c. ________________________________________________________________ d. ________________________________________________________________ e. ________________________________________________________________ f. ________________________________________________________________ g. ________________________________________________________________

PAGE 143

APPENDIX H NODE IDENTIFICATION SURVEY Thank you for agreeing to participate on the Process Identification Panel for the Central Florida Health Care Coalition and the University of Florida. Your expertise will be valuable in identifying the nodes in the primary care medication use system where preventable adverse drug events may have originated. The information from this analysis will be used to better understand the relationships between the processes of care and system factors. These include patient involvement, microsystems, organizations, and the environment. The attached survey contains detailed instructions to help you identify which nodes of the medication use system in primary care may be involved in the generation of specific drug therapy problems which, if unresolved, may lead to preventable adverse drug events. Based on pretesting, it should take approximately one hour to complete. Your input is extremely important. This project will not be successful without your gracious input. Please return the Process Identification survey, along with comments to Brian C. Sauer, at sauer@cop3.health.ufl.edu, by November 25 th Please e-mail or call me at (352) 273-6296 if you have any questions. Thank you again for participating. Respectfully, Brian C. Sauer Charles D. Hepler PhD Candidate Distinguished Professor 130

PAGE 144

131 You will be given a list of scenarios that represent cases of potential preventable adverse drug events (pADE) from the primary care setting. These pADE scenarios, which were developed and content validated by two Delphi panels of physicians, include errors of omission, therapeutic failures and drug induced adverse outcomes. The appendix provides more information for pADE operationalization. Survey Instructions The objective of this exercise is to identify the nodes of the primary care medication use system at which specific pADEs may have originated. In other words, we are seeking the node of the medication use system where the problem most probably was initiated. (Cause and effect diagramming, at a later time, will consider events downstream from the initiating node.) Listed below are 15 pADE scenarios. Each includes an adverse outcome and a pattern of care. We would like you to identify the node in the MUS where you believe the inappropriate care most likely originated. Please adopt a population perspective: That is, although many specific issues could have arisen in the care of an individual patient, we are asking you to consider the most common issues in a large number of patients. To do this, please first familiarize yourself with the definitions of the nodes of the medication use system below. Then please read each pADE scenario and assign values between 0 and 10 (0 highly unlikely and 10 highly likely) to represent your belief that the process failure originated in a particular node of the medication use system. The sum of the 4 nodes needs to equal 10. Please distribute the 10 points across the nodes to represent your belief that a particular drug-therapy problem originated in a specific node. The distinction between appropriate care for individuals and populations is very important here. We would like you to assign higher points to the node that is most likely to be where the process

PAGE 145

132 problem originated for the majority of patients with the specific pADE described in the scenario. Thinking at a population level can be difficult, one way to simplify this it to think of the last 10 patients you saw who had the problem described in the pADE scenario and allocate points to the nodes to represent those experiences. We would also like you to briefly state your reasoning and offer comments for each pADE scenario. Please see the hypothetical example below. A model of the Medication Use System and definitions for its nodes are presented below: Prescribing node (P)-Assess patient, determine need for medication, if need exists then select medication, and order medication. Dispensing node (D) Review and assess prescribers orders, look for problems with drug therapy, prepare medications, distribute medications for patients access, educate patient on proper use of medication. (Self) administration node (A) Administer medications according to prescribers plan, and report any problems with therapy. Monitoring node (M) Assess patient response to medication, detect problems with drug therapy, and act on the information gathered by reporting and documenting outcomes and changing therapy if necessary.

PAGE 146

133 Hypothetical Example This outcome has occurred after the pattern of care below: ER visit/hospitalization due to a major/minor hemorrhagic event This is the pattern of care: 1. Prescribed use of Warfarin 2. Prescription NSAID use (e.g., diclofenac, ibuprofen, ketoprofen, etc.) 3. INR not done within 10 days Node P D A M Assigned Points 8 0 0 2 P+D+A+M = 10 Decision Logic : Note: your responses are not expected to be as thorough as the example; this was done for instructional purposes. The process failure likely originated in the prescribing node of the MUS because it is well established that concomitant use of NSAIDs and warfarin should be avoided. If anti-pyretic effects are desired, then consider acetaminophen. If anti-inflammatory effects are necessary, then cyclooxygenase-2 (COX-2) inhibitor therapy may be safer. The dispensing node could have been involved in the process failure if the pharmacist knew or should have known that the patient was prescribed and using prescription NSAIDS and Warfarin concomitantly. Nevertheless, the problem would have been originated at the prescribing node; the error was reinforced at the dispensing node. Note: this example is using prescription NSAIDs, if OTC NSAIDs were included then the dispensing node would clearly be involved in the initiation of this problem. The administration node is involved to the degree that you would expect a patient to be aware of the medications they are taking and potential interactions and contraindications.

PAGE 147

134 Nevertheless, patients can not initiate prescription therapy nor are they capable of monitoring their INR levels and for this reason the administration node can not be where the problem originated. It also appears the problem could have originated at the monitoring node. If NSAIDS were considered necessary, then INRs should be monitored every week when NSAIDS and warfarin are co-administered. Signs and symptoms of an active bleed should have been monitored and if present appropriate actions should have been taken. Comments : Even though this problem could have originated in the prescribing and monitoring nodes of the medication use system, I placed more weight on the prescribing node because NSAIDs should generally be avoided in patients taking warfarin. Process Node Identification Survey Please assign points from 0 to 10 (0 highly unlikely and 10 highly likely) across the nodes of the medication use system. The points you allocate will represent the strength of your belief about where the process failure originated. 1. This outcome has occurred after the pattern of care below: ER visit/hospitalization due to congestive heart failure and/or fluid overload This is the pattern of care: 1. History/diagnosis of high blood pressure (over 140/90) and/or congestive heart failure 2. Use of prescription NSAID for at least 3 months Node P D A M Assigned Points Decision Logic: Comments:

PAGE 148

135 2. This outcome has occurred after the pattern of care below: ER visit/hospitalization due to congestive heart failure This is the pattern of care: 1. Diagnosis/history of congestive heart failure 2. Not on an ACE inhibitor (e.g., captopril, enalapril, etc.) Node P D A M Assigned Points Decision Logic: Comments: 3. This outcome has occurred after the pattern of care below: ER visit/hospitalization due to congestive heart failure and/or heart block This is the pattern of care: 1. History/diagnosis of congestive heart failure with heart block or advanced bradycardia 2. Use of digoxin Node P D A M Assigned Points Decision Logic: Comments: 4. This outcome has occurred after the pattern of care below: ER visit/hospitalization for Gastritis and/or upper GI bleed and/or upper GI perforation and/or GI ulcers and anemia This is the pattern of care: 1. History/diagnosis of ulcers and/or GI bleeding 2. Use of prescription NSAIDs for at least 1 month Node P D A M Assigned Points Decision Logic: Comments: 5. This outcome has occurred after the pattern of care below: ER visit/hospitalization for hypokalemia

PAGE 149

136 This is the pattern of care: 1. Use of a non potassium-sparing diuretic (e.g., hydrochlorothiazide, etc.) 2. No concurrent use of potassium supplement 3. Electrolytes not checked at least every 2 months initially, then every 6 months Node P D A M Assigned Points Decision Logic: Comments: 6. This outcome has occurred after the pattern of care below: ER visit/hospitalization due to seizure activity This is the pattern of care: 1. Use of anticonvulsant requiring drug level monitoring (e.g., phenytoin, carbamazepine, valproic acid) 2. Drug level not done upon initiation of therapy and at least every 6 months thereafter Node P D A M Assigned Points Decision Logic: Comments: 7. This outcome has occurred after the pattern of care below: ER visit/hospitalization due to gastritis and/or upper GI bleed and/or upper GI perforation and/or GI ulcers and anemia This is the pattern of care: 1. History/diagnosis of ulcers and/or GI bleeding 2. Use of prescription NSAID for at least 1 month Node P D A M Assigned Points Decision Logic: Comments: 8. This outcome has occurred after the pattern of care below: ER visit/hospitalization for Lithium toxicity This is the pattern of care: 1. Use of lithium 2. Lithium level not done monthly until stable, then every 3 months

PAGE 150

137 Node P D A M Assigned Points Decision Logic: Comments: 9. This outcome has occurred after the pattern of care below: ER Visit/ hospitalization due to tachycardia This is the pattern of care: 1. History of hypertension 2. Use of sympathomimetic decongestants Node P D A M Assigned Points Decision Logic: Comments: Assuming OTC 10. This outcome has occurred after the pattern of care below: ER visit/hospitalization due to hyperkalemia This is the pattern of care: 1. Use of ACE inhibitor 2. Electrolytes/CBC not done at least every 6 months Node P D A M Assigned Points Decision Logic: Comments: 11. This outcome has occurred after the pattern of care below: ER visit/hospitalization for acute renal failure and/or renal insufficiency This is the pattern of care: 1. Use of allopurinol 2. BUN/serum creatinine not done at least every 6 months Node P D A M Assigned Points Decision Logic: Comments:

PAGE 151

138 12. This outcome has occurred after the pattern of care below: ER visit/hospitalization due to major and/or minor hemorrhagic event This is the pattern of care: 1. Use of warfarin 2. INR not done before therapy starts and at least every month thereafter Node P D A M Assigned Points Decision Logic: Comments: 13. This outcome has occurred after the pattern of care below: ER visit/hospitalization due to asthma This is the pattern of care: 1. Diagnosis of moderate to severe asthma 2. Use of a bronchodilator 3. No use of maintenance corticosteroid (e.g., beclomethasone, etc.) Node P D A M Assigned Points Decision Logic: Comments: 14. This outcome has occurred after the pattern of care below: ER visit/hospitalization due to a major/minor hemorrhagic event This is the pattern of care: 1. Warfarin use 2. Antibiotic use (e.g., Bactrim, etc.) 3. PT not done within 5 days Node P D A M Assigned Points Decision Logic: Comments: Not enough info in example. 15. This outcome has occurred after the pattern of care below: ER visit/hospitalization due to acute renal failure and/or renal insufficiency

PAGE 152

139 This is the pattern of care: 1. Use of an ACE inhibitor 2. BUN/serum creatinine not done at initiation of therapy and at least every 3-6 months thereafter Node P D A M Assigned Points Decision Logic: Comments: Demographic information Name __________ Title_ Degrees__ ______ Age ____ Gender_______ Are you a licensed clinician?_ (Are you a practicing clinician?_ ___ What is your medical specialty?_____________________________________ Do you practice in an academic medical center?___________________ Are you involved in research on medication errs, patient safety, pharmacovigilance, etc..? ___________ Do you believe medication errors and adverse drug events are a serious public health issue? _________ What do you believe the incidence of pADEs is in ambulatory care? You have finished. Please return survey to Brian Sauer,at sauer@cop3.health.ufl.edu Thank you.

PAGE 153

140 Appendix to Survey The PDRM scenarios have been content validated by two separate Delphi panels of Geriatric Medicine experts, meaning the expert panel reached consensus that these scenarios represented situations in care that would be considered pADE. The following definitions were used for the validation process: 1) Preventable adverse drug event was defined as a significant clinical outcome in which drug therapy has not produced a reasonable intended result by (a) producing a noxious, unintended and undesired drug effect, (b) by failing to produce the intended effect within a reasonable time or (c) by omitting a necessary therapy. 2) Drug therapy problem was defined as a situation in care that is inconsistent with the treatment objectives. 3) Preventability: Four criteria were used to define the characteristics of preventable adverse drug event. The drug therapy problem had to be recognizable and the likelihood of adverse drug event must have been foreseeable In addition, the proximate cause(s) or medication error must be identifiable and those causes must be controllable

PAGE 154

APPENDIX I MEDICATION USE PERFORMANCE INDICATOR EVALUATION TEAM BRIEFING Thank you for agreeing to participate on the Medication Use Performance Indicator (MU-PI) Evaluation Team. The quality of medication use is an important issue for the providers, and members of the Central Florida Health Care Coalition. The goal of our meeting on May 4 th is to discuss the results of our MU-PI study with the intent to discover underlying system factors that may contribute to the process failures and morbidity. We will formally evaluate the following MU-PIs during our meeting. Pattern of Care (process) Outcome that occurred Diagnosis asthma 1. Use of a bronchodilator (persistence) 2. No use of maintenance therapy (e.g., corticosteroid, leukotrienes, etc.) ER visit/hospitalization due to asthma Use of warfarin 3. INR not done at least every month ER visit/hospitalization due to major and/or minor hemorrhagic event We will evaluate the two MU-PIs separately, and you will be asked to do the following: 1. Brainstorm possible system related issues that may have contributed to the prevalence of each pattern of care and outcome listed above. 2. Clarify and organize the proposed system causes. 3. Prioritize the proposed system cause and use top 5 to 10 for tree-diagramming. 4. Identify relationships among the proposed causes in relation to the different levels of the health care system through tree diagramming (described pages 3-5). 141

PAGE 155

142 5. Establish commonality with other MU_PIs. After reading through the background section below please prepare for the May 4 th meeting by thinking about the system factors that might explain: A) the number of asthmatics not using maintenance therapy B) the number of people using warfarin who appear to be missing monthly INRs. Background The MU-PI study used forty medication use indicators to screen the coalitions claims data for patterns of care that are inconsistent with guidelines and associated adverse outcomes. The 40 MU-PIs used in this study were based on published evidence and content validated by two separate Delphi panels of medical doctors. Population demographic and summary statistics for the 40 MU-PI are provided below. Table 1. Population Age Distributions mean age(SD)A g e Freq[%]<156,622 [19.07]1,084[18.51]1,263[17.61]4,956[14.3]15 4516,519 [47.58]2,143[36.6]3,399[47.39]16,461[47.48]45 6510,743 [30.94]2,144[36.62]2,270[31.65]11,421[32.94]65<=835 [2.41]484[8.27]240[3.35]1830[5.28]GenderFemale22,861 [65.85]3243[71.84]4,659[64.96]23,499[67.78]Male11,858 [34.15]1271[28.16]2,513[35.04]11,169[32.22]Missing0000All ClaimsHospital AdmissionsED visitsPharmacy Claims33.97 (18.23)34.48 (22.12)34.71 (18.54)37.03 (18.34)

PAGE 156

143 Table 2. Process and PDRM Positives # Members Positive# of PositivesPercentProcess 6,12911,49016.54PDRM_ED4146PDRM_HA2641PDRM_ED_HA1819Total_ PDRM831060.23 There were 11,490 process positives in 6,129 members. Table 2 also shows the results for those who screened positive for both the process and the outcome. The term preventable drug related morbidity (PDRM) is used to describe those who screen positive for both the process and the outcome. There were 106 PDRM positives in 83 members. 46 led to emergency department visit (ED), 41 led to a hospital admission (HA) and 19 went from the ED to the HA. Table 4. lists the MU-PIs that will be included in the evaluation. For a complete list of the 40 MU-PI used in this study and individual indicator results please call or e-mail Brian Sauer (352) 273-6296 sauer@cop3.health.ufl.edu and I will e-mail them to you. Table 4. Indicators for Evaluation IndMnemonicHxDrug_ADrug_BRiskProcessP_HAP_EDPDRM1Asthma--Bron-NoMaintAsthma215337043878496626292Hx/CHF-NoACE-ICHF6454059398105123War-INR(1)Hemr34330436929ACEI-BUN(6) ARF405924092371325 1. The Mnemonic is an abbreviated version of the MU-PI scenario listed below 2. Hx = the number of members who had an ICD-9 code for the particular history of disease or diagnosis of interest 3. Drug A and Drug B = the number of members who had a prescription claim for the drug of interest.

PAGE 157

144 4. Risk = the number of members who met the criteria for analysis. For example; if the process component includes a drug and monitoring interval of 6 months, then the members at risk would be those who were using the medication of interest for at least 6 months. 5. Process = the number of members who had the pattern of care represented in the MU-PI 6. PDRM = the number of members who had both the process component and the adverse outcome within the specified time frame. Ind Process of Care Outcome 1 Diagnosis of moderate to severe asthma Use of a bronchodilator No use of maintenance therapy (e.g., corticosteroid, leukotrienes, etc.) ER visit/hospitalization due to asthma 2 Diagnosis/history of congestive heart failure AND Not on an ACE inhibitor or ARB ER visit/hospitalization due to congestive heart failure 3 Use of warfarin AND INR not done at least every month ER visit/hospitalization due to major and/or minor hemorrhagic event 4 Use of ACE inhibitor and BUN/serum creatinine done every 6 months ER visit/hospitalization due acute renal failure Levels of Health Care System Berwick (2002), published a users manual for the IOM report, Crossing the Quality Chasm. The purpose was to provide a conceptual framework for better understanding and evaluation the affect of health care systems on patient care. In it he addresses the issue of embedded systems by presenting a framework for the different levels within health care. He separates the system into the following: the experience of the patients and communities (Level A), the functioning of small units of care delivery called microsystems (Level B), the functioning of organizations that house or otherwise support microsystems (Level C) and the environment of policy, payment, regulation, accreditation and other factors (Level D). We will use this

PAGE 158

145 framework to organize the proposed caused from the brainstorming step and to explicitly describe how the relationships among the system levels affect the quality of medication use. Figure 1. Levels of Health Care System The Patient Level (Level A) represents the patients experience and perception of the care they received. It also represents the patients active participation in their health and health care. The Medication Use System (MUS) includes recognition of drug therapy indication, prescribing, dispensing, administering, and patient monitoring. The MUS spans multiple professions and microsystems, which include, but are not limited to, physician offices, pharmacy, and laboratory.

PAGE 159

146 The Microsystems (Level B) are the small productive units that actually give the care that the patient experiences. Clinical microsystems are basically the small organized groups of providers and staff caring for a defined population of patients. The Health Care Organizations (Level C) house and support microsystems -they provide the necessary resources for the microsystems to deliver care. Common organizations include hospitals, large provider groups, nursing homes, and pharmacy benefit managers, all of which are typically embedded within some form of managed care organizations. The Health Care Environment (Level D) includes multiple entities that influence the activities of the organizations and microsystems. Important environmental systems include financing, regulation, accreditation, policy, litigation, professional education and social policy

PAGE 160

APPENDIX J MU-PI RESULTS B Hem GD D S L He Hy BldyAUR ActRespFai AUR CHFHx H AntHy Am CO HT Hy BldyARF Di G CH ARF He MIHx/MIHem ARF Depress F F CHFHxCHFHx As F See Appendix D to link indicator numbers to the indicator scenarios IndicatorMnemonicHxDrug_ADrug_BRiskProcessP_HAP_EDPDRMDelphi1DepHx/Dep-Benzo1,8982,6893217592TheoToxTheo-DrugLvl(6)19179750003 PHx/BP-Li-DrugLvl(3)9765990004rhep-NoLab7770005IHx/GI-NSAID2,5265,89931600046epHx/Dep-Barb1,898662510007epHx/Dep-Symp1,8982,9711362138eizAntconv-DrugLvl(6)37715914311269iToxLi-DrugLvl(3)652321000810mrWar/NSAID-INR3437,2831923400011pothLi-TSH(6)65181600012sTic-CBCP(2)74100013Hx/bladderAtony-Imip10597000014lCOPD-Benzo6412,68910231315Hx/BPH-Antic6781,8112800016/HBP-NSAID6,0017,2831,427033117ypoKKwd-NoK-Elctro(2)1,810837814643010518iconvToxAnticonv-DrugLvl(6)37715914500019pothThy-T4TSH(12)2,6151,51091514520glyToxAmgly-SCr-DrugLvl*(7dys)41413900021PDHx/COPD-BB5602,8327100022NHx/HTN-Decon6,03600000923prKACEI-ElctroCBC(6)4,0592,4092,2090221024sCbz-ElctroCBC(6)103403300025Li-BUN(3)65232100026gToxDif-BUN/DrugLvl**(6)30930913600027IHx/GI-Ocort2,5264,07527201128FHx/CHF_HB-Dig6453097911612329ACEI-BUN(6)4,0592,4092,2823251530mrWar-INR(1)3432563691231--ASA-BB**1812,6688951532rWar-Antibiot-NoLab(5dys)3439,785320001433Allop-BUN/Scr(6)2111201081121134Hx/Depress-BB1,8981,0753600035allAlphaBlkr-NoBP(2)49035530400036alllongactingBenzo2,68917911137/CHF-NoACEI6454,05939810512238/CHF-antiarrhythmic645811821239thmaHx-Bron-NoSteroid2,1533,7043,878190626261340allTriclyantidep8365111111,1935769103 The Mnemonic is an abbreviated version of the MU-PI scenario 147

PAGE 161

148 Hx = the number of members who had an ICD-9 code for the particular history of disease or diagnosis of interest Drug A and Drug B = the number of members who had a prescription claim for the drugs of interest. Risk = the number of members who met the criteria for analysis. For example; if the process component includes a drug and six month monitoring interval, then the members at risk would be those who were using the medication of interest for at least six months. Process = the number of members who had the pattern of care represented in the MU-PI PDRM = the number of members who had both the process component and the outcome component of the MU-PI within the specified time frame. Fifteen indicators were sent to a Delphi process for node identification, they were numbered consecutively from one to fifteen. This column provides a cross reference to the MU-PI number

PAGE 162

APPENDIX K SURVEDY TO ESTABLISH COMMONALITY AND UNIQUENESS OF CAUSE SEQUENCES FROM TREE DIAGRAMS Survey to determine if the relationships identified among the system levels from the warfarin indicator are also contributing factors to other indicators. 1 Use of an ACE inhibitor AND BUN/serum creatinine not done at least every 6 monthsER visit/hospitalization due to acute renal failure and/or renal insufficiencyDiagnosis asthma 1. Use of a bronchodilator 2. No use of maintenance therapy (e.g., corticosteroid, leukotrienes, etc.)2ER visit/hospitalization due to asthma Branch 1. (Ind 1) Yes________ No________ Branch 1. (Ind 2) Yes________ No________ Branch Max (Ind 1) Yes________ No________ Branch Max. (Ind 2) Yes________ No________ Survey to determine if the relationships identified among the system levels from the Asthma indicator are also contributing factors to other indicators. 2Use of warfarin AND INR not done at least every monthER visit/hospitalization due to major and/or minor hemorrhagic event1ER visit/hospitalization due to congestive heart failure Diagnosis/history of congestive heart failure AND Not on an ACE inhibitor or ARB Branch 1. (Ind 1) Yes________ No________ Branch 1. (Ind 2) Yes________ No________ Branch Max (Ind 1) Yes________ No________ Branch Max. (Ind 2) Yes________ No________ 149

PAGE 163

APPENDIX L NODE IDENTIFICATION BOX PLOTS Each point indicates minimum, 25, 50 (median), 75, percentile and maximum. The Red Plus is the mean. mean median 25 percentile 75 percentile min max + Indicator 1 (16) Indicator 2 (37) 150 Indicator 6 (8) Indicator 5 (17) Indicator 4 (5) Indicator 3 (28)

PAGE 164

151 Indicator 9 (22) Indicator 8 (9) 2 nd round Indicator 9 (22) Indicator 10 (23) Indicator 11 (33) Indicator 12 (30) Indicator 13 ( 39 ) 2 nd round Indicator 12 (30) Indicator 14 (42) 2 nd round Indicator 14 (42)

PAGE 165

152 Indicator 15 ( 29 )

PAGE 166

LIST OF REFERENCES 1. Winterstein AG, Sauer BC, Hepler CD, Poole C. Preventable drug-related hospital admissions. Ann Pharmacother. 2002; 36(7-8):1238-48. 2. Institute of Medicine. Committee on Quality of Health Care in America. To err is human--building a safer health system. Washington, D.C.: National Academy Press, 1999. 3. Johnson JA, Bootman JL. Drug-related morbidity and mortality and the economic impact of pharmaceutical care. Am J Health Syst Pharm 1997; 54(5):554-558. 4. Cullen DJ, Bates DW, Leape LL. Prevention of adverse drug events: a decade of progress in patient safety. J Clin Anesth 2000; 12(8):600-614. 5. Bero LA, Lipton HL, Bird JA. Characterization of geriatric drug-related hospital readmissions. Med Care 1991; 29:989-1003. 6. Bigby J, Dunn J, Goldman L, Barclay AJ, Jen P, Seth Landfeld C. Assessing the preventability of emergency hospital admissions. A method for evaluating the quality of medical care in a primary care facility. Am J Med 1987;83:1031-1036. 7. Courtman BJ, Stallings SB. Characterization of drug-related problems in elderly patients on admission to a medical ward. Can J Hosp Pharm 1995; 48(3):161-166. 8. Darchy B, Le Miere E, Figueredo B Bavoux E, Omart Y. Iatrogenic diseases as a reason for admission to the intensive care unit. Arch Intern Med 1999; 159:71-78. 9. Dartnell JGA, Anderson RP, Chohan V, Galbraith KJ, Lyon MEH, Nestor PJ, Moulds RFW. Hospitalisation for adverse events related to drug therapy: incidence, avoidability, and costs. Med J Austr 1996; 164:659-662. 10. Hallas J, Gram LF, Grodum E, Damsbo N, Brosen K, Haghfelt T, Harvald B, Beck-Nielsen J, Worm J, Jensen KB. Drug related admissions to medical wards: a population based survey. Br J Clin Pharmacol 1992; 33:61-68. 11. Lakshmanan MC, Hershey CO, Breslau D. Hospital admissions caused by iatrogenic disease. Arch Intern Med 1986; 146:1931-1934. 12. Nikolaus T, Specht-Leible N, Kruse W, Oster P, Schlierf G. [The early rehospitalization of elderly patients. Causes and prevention]. Dtsch Med Wochenschr 1992; 117(11):403-407. 153

PAGE 167

154 13. Faris RJ. Explicit definitions to identify preventable drug-related morbidity in an elderly population and use as an indicator to evaluate quality in medications use system. Doctoral Dissertation, University of Florida, 2001. 14. Wilson PF, Dell LD, Anderson GF. Root Cause Analysis: A Tool for Total Quality Management. Milwaukee: Quality Press, 1993. 15. Hepler CD, Segal R. Preventing Medication Errors and Improving Drug Therapy Outcomes through Systems Management. Boca Raton: CRC Press Inc., 2003. 16. Berwick DM. A user's manual for the IOM's "Quality Chasm" report. Health Aff 2002; 21(3):80-90. 17. Reason J. Human error: models and management. West J Med 2000; 172(6):393-396. 18. Hepler CD, Strand LM. Opportunities and responsibilities in pharmaceutical care. Am J Hosp Pharm 1990; 47(3):533-543. 19. Bates DW, Leape LL, Petrycki S. Incidence and preventability of adverse drug events in hospitalized adults. J Gen Intern Med 1993; 8(6):289-294. 20. Bates DW, O'Neil AC, Boyle D, Teich J, Chertow GM, Komaroff AL, Brennan TA. Potential identifiability and preventability of adverse events using information systems. J Am Med Inform Assoc 1994; 1(5):404-411. 21. Bates DW, Spell N, Cullen DJ, Burdick E, Laird N, Petersen LA, Small SD, Sweitzer BJ, Leape LL. The costs of adverse drug events in hospitalized patients. Adverse Drug Events Prevention Study Group. JAMA 1997; 277(4):307-311. 22. Bates DW, Teich JM, Lee J, Seger D, Kuperman GJ, Ma'luf N, Boyle D, Leape L. The impact of computerized physician order entry on medication error prevention. J Am Med Inform Assoc 1999; 6(4):313-321. 23. Leape LL, Kabcenell AI, Gandhi TK, Carver P, Nolan TW, Berwick DM. Reducing adverse drug events: lessons from a breakthrough series collaborative. Jt Comm J Qual Improv 2000; 26(6):321-331. 24. Grainger-Rousseau TJ, Miralles MA, Hepler CD, Segal R, Doty RE, Ben Joseph R. Therapeutic outcomes monitoring: application of pharmaceutical care guidelines to community pharmacy. J Am Pharm Assoc 1997; NS37(6):647-661. 25. Hepler CD, Grainger-Rousseau TJ. Pharmaceutical care versus traditional drug treatment. Is there a difference? Drugs 1995; 49(1):1-10. 26. Nelson EC, Batalden PB, Mohr JJ, Plume SK. Building a quality future. Front Health Serv Manage 1998; 15(1):3-32.

PAGE 168

155 27. Donabedian A. The quality of medical care. Science 1978; 200(4344):856-864. 28. Mohr JJ, Batalden PB. Improving safety on the front lines: the role of clinical microsystems. Qual Saf Health Care 2002; 11(1):45-50. 29. Wasson JH, Jette AM, Johnson DJ, Mohr JJ, Nelson EC. A replicable and customizable approach to improve ambulatory care and research. J Ambul Care Manage 1997; 20(1):17-27. 30. Torrens PR. Historic Evolution and Overview of Health Services in the United States. Introduction to Health Services. Albany: Delmar Publisher, 1999. 31. Leape LL, Bates DW, Cullen DJ, Cooper J, Demonaco HJ, Gallivan T, Hallisey R, Ives J, Laird N, Laffel G. Systems analysis of adverse drug events. ADE Prevention Study Group. JAMA 1995; 274(1):35-43. 32. Plsek PE, Greenhalgh T. Complexity science: The challenge of complexity in health care. BMJ 2001; 323(7313):625-628. 33. Iezzoni L. Risk Adjustment for Measuring Health Care Outcomes. Ann Arbor: Health Administration Press, 1994. 34. Waterstraat FL, Barlow J, Newman F. Diagnostic coding quality and its impact on healthcare reimbursement: research prospectives. J Am Med Rec Assoc 1990; 61(9):52-59. 35. Quam L, Ellis LB, Venus P, Clouse J, Taylor CG, Leatherman S. Using claims data for epidemiologic research. The concordance of claims-based criteria with the medical record and patient survey for identifying a hypertensive population. Med Care 1993; 31(6):498-507. 36. Coalition for Quality in Medication Use. Study of Clinically Relevant Indicators of Pharmacologic Therapy: Final Report. http://www.scriptproject.org. 2002. Accessed 07/2004. 37. Ashton CM, Kuykendall DH, Johnson ML, Wray NP. An empirical assessment of the validity of explicit and implicit process-of-care criteria for quality assessment. Med Care 1999; 37(8):798-808. 38. Brennan T.A. The Institute of Medicine Report on Medical Errors: Could it do harm? N Engl J Med 2000; 342(15):1123-1125. 39. Jha AK, Kuperman GJ, Rittenberg E, Teich JM, Bates DW. Identifying hospital admissions due to adverse drug events using a computer-based monitor. Pharmacoepidemiol Drug Saf 2001; 10(2):113-119. 40. Classen DC, Pestotnik SL, Evans RS, Burke JP. Computerized surveillance of adverse drug events in hospital patients. JAMA 1991; 266(20):2847-2851.

PAGE 169

156 41. Honigman B, Lee J, Rothschild J, Light P, Pulling RM, Yu T, Bates DW. Using computerized data to identify adverse drug events in outpatients. J Am Med Inform Assoc 2001; 8(3):254-266. 42. MacKinnon NJ. Risk Assessment of Preventable Drug-Related Morbidity in Older Persons. Doctoral Dissertation, University of Florida, 1999. 43. Helmer, O. Analysis of the Future: The Delphi Method. Santa Monica, CA, The RAND Corporation, 1967. 44. Dalkey, N. C. The Delphi Method: An Experimental Study of Group Opinion (RM-5957-PR). Santa Monica, CA, The RAND Corporation, 1969. 45. Dalkey, N. C. The Delphi Method, III: Use of Self Rating to Imporve Group Estimates (RM-6115-PR). Santa Monica, CA, The RAND Corporation, 1969. 46. Anderson B, Fargerhaug T. Root Cause Analysis: Simplified Tools and Techniques. Milwaukee: ASQ Quality Press, 2000. 47. Mclaughlin CP, Kaluzny AD. Continuous Quality Improvement in Health Care: Theory, Implementation, and Application. Gaitherburg: Aspen, 1999. 48. Brennan TA, Leape LL, Laird NM, Hebert L, Localio AR, Lawthers AG, Newhouse JP, Weiler PC, Hiatt HH. Incidence of adverse events and negligence in hospitalized patients. Results of the Harvard Medical Practice Study I. N Engl J Med 1991; 324:370-376. 49. Cunningham G, Dodd TRP, Grant DJ, McMurdo MET, Richards RME. Drug-related problems in elderly patients admitted to Tayside hospitals, methods for prevention and subsequent reassessment. Age and Ageing 1997; 26:375-382. 50. Lindley CM, Tully MP, Paramsothy V, Tallis RC. Inappropriate medication is a major cause of adverse drug reactions in elderly patients. Age Ageing 1992; 21(4):294-300. 51. Nelson KM, Talbert RL. Drug-related hospital admissions. Pharmacotherapy 1996; 16(4):701-707. 52. Ng D, Gosh DG, Harris J, Whitehead C. Unplanned medication-related admissions to an acute care general teaching hospital. Aust J Hosp Pharm 1999; 29:84-87. 53. Raschetti R, Menniti-Ippolito F, Morgutti M, Belisari A, Rossignoli A. Adverse drug events in hospitalized patients. JAMA 1997; 277(17):1351-1352. 54. Trunet P, Le Gall JR, Lhoste F, Rapin M. Admissions to intensive care units for iatrogenic diseases. Leg Med 1982; 73-84.

PAGE 170

157 55. Trunet P, Borda IT, Rouget AV, Rapin M, Lhoste F. The role of drug-induced illness in admissions to an intensive care unit. Intensive Care Med 1986; 12(1):43-46. 56. Hall MJ, Popovic JR. 1998 summary: national hospital discharge survey. Advance Data, NCHS 2000; 316:1-17. 57. Scott J, Pope M. Nonadherence with mood stabilizers: prevalence and predictors. J Clin Psychiatry 2002; 63(5):384-390. 58. Biernacki P, Waldorf D. Snowball sampling. Social Methods Res 1981; 10:141-163. 59. Mackinnon NJ, Helper CD. Indicators of preventable drug-related morbidity in older adults 2. Use within a managed care organization. J Manag Care Pharm 2003; 9(2):134-141. 60. Gandhi TK, Weingart SN, Borus J, Seger AC, Peterson J, Burdick E, Seger DL, Shu DL, Federico F, Leape LL. Adverse drug events in ambulatory care. N Engl J Med 2003; 348(16):1556-1564. 61. Field TS, Gurwitz JH, Avorn J, McCormick D, Jain S, Eckler M, Benser M, Bates DW. Risk factors for adverse drug events among nursing home residents. Arch Intern Med 2001; 161(13):1629-1634. 62. Jha AK, Kuperman GJ, Teich JM, Leape L, Shea B, Rittenberg E, Burdick E, Seger DL, Vander Vliet M, Bates DW. Identifying adverse drug events: development of a computer-based monitor and comparison with chart review and stimulated voluntary report. J Am Med Inform Assoc 1998; 5(3):305-314. 63. Carbonin P, Pahor M, Bernabei R, Sgadari A. Is age an independent risk factor of adverse drug reactions in hospitalized medical patients? J Am Geriatr Soc 1991; 39(11):1093-1099. 64. Berwick DM, James B, Coye MJ. Connections between quality measurement and improvement. Med Care 2003; 41(1 Suppl):I30-I38. 65. Holmes-Rovner M, Llewellyn-Thomas H, Entwistle V, Coulter A, O'Connor A, Rovner DR. Patient choice modules for summaries of clinical effectiveness: a proposal. BMJ 2001; 322(7287):664-667. 66. O'Connor AM, Stacey D, Entwistle V, Llewellyn-Thomas H, Rovner D, Holmes-Rovner M, Tait V, Tetroe J, Fiset V, Barry M, Jones J. Decision aids for people facing health treatment or screening decisions. Cochrane Database Syst Rev 2003;(2):CD001431.

PAGE 171

158 67. Cabana MD, Rand CS, Powe NR, Wu AW, Wilson MH, Abboud PA, Rubin HR. Why don't physicians follow clinical practice guidelines? A framework for improvement. JAMA 1999; 282(15):1458-1465. 68. Hepler CD. Looking toward a market for pharmaceutical care. J Am Pharm Assoc Wash 1997; NS37(6):625-626.

PAGE 172

BIOGRAPHICAL SKETCH Brian Sauer is interested in the assessment and improvement of medications use. His core research goals are to use quantitative and qualitative methods to better understand the nature of drug-related morbidity. Specifically, he plans to contribute to the development of automated performance indicators of preventable drug-related morbidity, and develop methods to better understand how structure and processes (system design) in health care microsystems and organizations affect the quality of medication use. 159


Permanent Link: http://ufdc.ufl.edu/UFE0006469/00001

Material Information

Title: Medication Use Performance Indicator Evaluation: A Systems Perspective
Physical Description: Mixed Material
Copyright Date: 2008

Record Information

Source Institution: University of Florida
Holding Location: University of Florida
Rights Management: All rights reserved by the source institution and holding location.
System ID: UFE0006469:00001

Permanent Link: http://ufdc.ufl.edu/UFE0006469/00001

Material Information

Title: Medication Use Performance Indicator Evaluation: A Systems Perspective
Physical Description: Mixed Material
Copyright Date: 2008

Record Information

Source Institution: University of Florida
Holding Location: University of Florida
Rights Management: All rights reserved by the source institution and holding location.
System ID: UFE0006469:00001


This item has the following downloads:


Full Text












MEDICATION USE PERFORMANCE INDICATOR EVALUATION:
A SYSTEMS PERSPECTIVE















By

BRIAN C. SAUER


A DISSERTATION PRESENTED TO THE GRADUATE SCHOOL
OF THE UNIVERSITY OF FLORIDA IN PARTIAL FULFILLMENT
OF THE REQUIREMENTS FOR THE DEGREE OF
DOCTOR OF PHILOSOPHY

UNIVERSITY OF FLORIDA


2004

































Copyright 2004

by

Brian C. Sauer



































To my parents















ACKNOWLEDGMENTS

I would like to acknowledge my dissertation chair, Dr. Charles D. Hepler, for

taking me under his wings and sharing his knowledge with me. I have grown

tremendously over the past five years and Dr. Hepler has guided that process.

I thank the members of my supervising committee, Drs. Earlene Lipowski,

Abraham Hartzema, Murray Cote, and Richard Segal for their patience and guidance

through the process. I would like to thank Josue Rodas, Becky Cherney and Scott

Langdon for providing the support needed to make this study happen. I thank the Perry

Foote foundation for providing the grant that financed this project. I thank Dave Angaran

for listening and helping me understand and think through the clinical complexity of my

dissertation.

I would also like to thank Dr. Alan Spector for giving me the opportunity to work

in his laboratory as an undergraduate in biological psychology. This was an important

time in my academic development. Like Dr. Spector, I hope I will always be able to roll-

up my sleeves and loosen my tie when the critics are throwing heat. I also would like to

thank Mircea Garcea for teaching me, through example, to always pay attention to the

details and produce quality work.

Finally, I would like to thank the graduate students for producing a healthy

working environment and their friendship.
















TABLE OF CONTENTS



A C K N O W L E D G M E N T S ......... .................................................................................... iv

LIST OF TA BLES ............. .... .............................. .. ................ ... viii

LIST OF FIGURES ............................... ... ...... ... ................. .x

ABSTRACT .............. .................. .......... .............. xi

CHAPTER

1 IN TR OD U CTION ............................................... .. ......................... ..

Problem Statem ent .................. .............................. .. ........................ ..
O bj ectiv e ...................................................................................... . 2
Specific A im s........................................................ 2
Ju stific a tio n ............................................................................... 2
O v e rv iew .................................................................................. 3

2 CONCEPTUAL FRAMEWORK ..................................................... 6

A diverse Outcom es of D rug Therapy .........................................................................6
The M education U se System .......................................................................8
Levels of the H health System .................................................................. 9
H um an Error and System Failure ....................................................... 14
S u m m a ry .......................................................................................................1 6

3 LITERATURE REVIEW .............................................................. ...............18

Use of Administrative Databases for Researching PDRM .........................................18
Automated Methods for Detecting PDRM .............................................................22
D elp h i M eth o d ................................................................................................2 9
Cause-and-Effect A analysis ............. ..................... ........................................ 35

4 PRELIM IN ARY W ORK ........................................................................... 39

Prevalence of Preventable Drug Related Admissions ...............................................39
Drug Categories Involved in PDRA ......................................................... ....... 43
Apparent/Proximate Causes of PDRAs......... ............. ...................... 48


v









L atent C au ses of PD R A ......................................... .. .. ............................ ............... 52

5 M E T H O D S ........................................................................................................... 5 5

S p ecific A im s........................................................5 5
Step O ne: D database A nalysis........................................................... ............... 55
C laim s D ata T ypes ................................ .... .. ......................... .. .. ............ 55
Descriptive Analysis to Evaluate the Integrity of Claims Data.........................56
Population D em graphic ...................... .............. ..................... .... ........... 56
MU-PI Coding Concepts and Analysis .................................... ............... 57
Search algorithm s .............................................................. .. .... .. .... .. 57
Disease-drug interaction...................... ..... ............................ 57
D rug m monitoring ........................................... .. ...... .. ... ........ .... 58
D rug-drug interaction ......................................................... ............... 60
Prevalence Estimates for MU-PI Positives............................... ...............60
Population Based Explanations for Prevalence Findings............................... 61
Step Two: Node Identification of PDRM Scenarios .............................63
P ilo t T e stin g .................................. ............ ....... ............... ............... 6 3
Selection of PDRM Scenarios for Node Identification Study.............................64
D elp h i R ecruitm ent .................................................................. ....................64
N ode Identification Survey........................................... ........................... 65
Delphi Process ........................................... ............. ........ 65
N ode Identification A analysis: ............................................................ ............66
Selection of M U -PIs for Evaluation................................................................ 66
Step Three: Cause-and-Effect Analysis................................ ........................ 67
MU-PI Evaluation Team .......................................................................67
MU-PI Evaluation Process ............................ ...............67
Analysis for Common and Unique Cause Sequences ......................................70

6 R E S U L T S .......................................................................... 7 1

Step O ne. D database A analysis: ........................................................................ ... 7 1
Descriptive Analysis to Evaluate the Integrity of the Claims Database.............71
Check for missing and invalid data.............................................. ........72
L inks betw een data types ........................................ ......... ............... 72
Population D em graphics ............................................................................73
Prevalence of M U -PI Positives ................. ............... ....................................... 75
System Based Explanations for Prevalence Findings ......................................77
Distribution of m monitoring intervals.................................. ............... 78
Variables associated with PDRM positives ...............................................84
Step Two. Assignment of MU-PIs to Nodes of the MUS .......................................86
D elphi R ecruitm ent .................................................................... ....................86
Assignment of MU-PIs to Nodes of the MUS .................................................87
Step Three: Cause-and-Effect Analysis................................... ....................... 89
Evaluation of Monitoring Indicator .............. ..............................................91
Evaluation of Prescribing Indicator................................................................ 92
Cause Sequences Common to both the Monitoring and Prescribing Nodes.......94









7 DISCUSSION .................. ................................... ........... ............... 97

D atab ase A n aly sis............ ... ................................................................ ........ .. .... .. 97
Prevalence Findings ........................ ........ ... ............. .... ............. 97
System Explanations for Prevalence Findings .......................................... 99
Distribution of m monitoring intervals.................................. ............... 99
M multiple logistic regression analysis ....................................................... 100
N ode Identification of M U -PIs ....................................................... .... ........... 102
C ause-and-Effect A analysis ............................................... ............................ 103
C au se them es ................................................................. ............... 104
Conclusion ..................................... ................................. ......... 106
Lim stations ..................................... .................................... ......... 108
Significance and Theoretical Contribution...................... ...................... 109
C contribution to H health C are............................................................................... ... .... 110
Future A areas for Study .................. ..................................... .............. .. 111
In te rv e n tio n s ................................................................................................ 1 1 1
M U -PI Instrum ent Fidelity ...................................................... ..... .......... 113

APPENDIX

A MEDICATION INVOLVED IN DRA BY STUDY ...............................................115

B CLASSIFICATION OF DTPS INTO NODES OF THE MUS.............................116

C SUMMARY: CLASSIFICATION OF DTPS INTO NODES OF THE MUS......... 118

D MEDICATION USE PERFORMANCE INDICATOR DEFINITIONS..............119

E PROCEDURE CODES TO IDENTIFIY VISITS ...........................................126

F CODING SOLUTION FOR INDICATOR.............................................................127

G RECRUITMENT LETTER FOR NODE IDENTIFICATION STUDY.................. 128

H NODE IDENTIFICATION SURVEY ....................................................................130

I EVALUATION TEAM BRIEFING................................ 141

J M U-PI RESULTS .................. ...................................... .......... .. ..147

K SURVEY FOR COMMONALITY ................................... ....................................149

L NODE IDENTIFICATION BOX PLOTS.....................................................150

L IST O F R E FE R E N C E S ....................................................................... .................... 153

BIOGRAPHICAL SKETCH ............................................................. ............... 159
















LIST OF TABLES


Table page

3.1 Examples of DRM Screens by Type............. ..................... 25

3.2 Results of Leading DRM Screens ..................................... ............ ....... ........ 25

3.3 R results of D R M Screens ................................................ .............................. 26

3.4 R results of Top Five Indicators ........................................ ........................... 30

3.5 Percent Change in Average Group Error by Number of Group Members ..............32

4.1 Studies R reporting D R A and PD RA : ........................................................ .........44

4.2 Prevalence Estimates and Odds Ratio ........................................ ............... 45

4.3 L atent C auses of PD R A s............................................................... ....................54

6.1 Population Age and Gender Frequencies by Type of Claim................................74

6.2 Population D em graphics ............................................... ............................. 76

6.3 Frequency of Process and PDRM Positives .............................77

6.4 Number of PDRM Positives by Number of Indicators ................. ...................77

6.5 Eleven Most Prevalent PDRM Positives ................................. 78

6.6 Ten M ost Prevalent Process Positives................................................................... 78

6.7 MU-PI Categorized by Required Monitoring Interval ....................................79

6.8 Bivariate Tests of Association: PDRM Positives as Dependent Variable ..............85

6.9 Multicollinearity Assessment of Independent Variables ........................................86

6.10 Maximum likelihood Estimates: PDRM Positives as Dependent Variable ............86

6.11 Demographics of Delphi Panelists for Node Identification Study .........................87

6.12 Indicators Listed by Associated Node of the MUS...............................................89









6.13 Kruskal Wallis and Pairwise Comparisons for Indicator Assignment...................90

6.14 Monitoring Indicator: Affinity Table and Rating of Importance .............................92

6.15 Cause Sequence Agreem ent Results ............................................. ............... 93

6.16 Prescribing Indicator: Affinity Table and Rating of Importance .............................94
















LIST OF FIGURES


Figurege

2.1 M odel of the M education Use System ..................................... ........ ............... 14

2.2 Hierarchal Relationships among the System Levels..............................................15

2.3 Swiss Cheese Model of System Failure ...........................................................17

4.1 Involvement of Treatment Category to DRMs: .................................................49

4.2 Therapeutic Class Involvement in DRM:....................................................... 49

4.3 Nodes of the MUS Involved in PDRA..................................................................52

5.1 Disease-Drug Interaction Search Algorithm............................................................58

5.2 Process Search Algorithm for Indicators that Require Monitoring..........................59

5.3 Process and Outcome Search Algorithm...........................................................59

5.4 Example of a Process Positive but not a PDRM Positive .............................60

5.5 Drug-Drug Interaction Search Algorithm ..................................... .................60

5.6 Organization of Proposed Causes to System Levels .......................................69

6.1 O office V isits Claim s by M onth ........................................... ......................... 72

6.2 ED V isit Claim s by M onth................................ ........ ............... ............... 72

6.3 Facility Adm missions by M onth ..... ............................................. ............... 73

6.4 Pharm acy Claim s by M onth......................................... ............... ............... 73

6.5 First Lag for Indicator with One Month Monitoring Requirement........................80

6.6 First Lag for Indicator with Two Month Monitoring Requirement .........................81

6.7 First Lag for Indicator with Three Month Monitoring Requirement .......................82

6.8 First Lag for Indicator with Six Month Monitoring Requirement ...........................82















Abstract of Dissertation Presented to the Graduate School
of the University of Florida in Partial Fulfillment of the
Requirements for the Degree of Doctor of Philosophy

MEDICATION USE PERFORMANCE INDICATOR EVALUATION:
A SYSTEMS PERSPECTIVE
By

BRIAN C. SAUER

August, 2004

Chair: Charles Doug Hepler
Major Department: Pharmacy Health Care Administration

Background: Injury related to health care is a serious public health concern. The

prevalence of drug related admissions in the United States has been reported to be from

three to twelve percent of hospital admissions. Organizations interested in reducing drug

related morbidity need measurement techniques to gather baseline information about

preventable drug related morbidity (PDRMs), and methods to assess system related

causes to develop rational interventions that target the system failures

The objective of this dissertation is to better understand the relationship between

system design and patterns of care that can result in drug related injury.

Methods: This study was executed in three steps. In the first step, database

analysis, medication use performance indicators (MU-PI) were used to estimate the

number of PDRM positives in the managed care organization's administrative database.

Step two used the Delphi process to judge the degree of association between select MU-

PIs and specific nodes of the medication use system (MUS) where the process failure









may have originated. In step three, select MU-PIs from each node of the MUS were

submitted to cause-and-effect analysis (CEA).

Results: The period prevalence for process positives was 209.84 (206.37-213.34)

per 1,000 member years. The period prevalence for PDRM positives was 1.98 (1.6-2.4)

per 1,000 member years. Gender, number of office visits, number of drug classes and

number of medical conditions were independent risk factors for PDRM.

Fourteen indicators were selected for step two. Six of the fourteen indicators were

assigned to the prescribing node and seven were assigned to the monitoring node of the

MUS. One indicator did not reach significance within two rounds.

The cause-and-effect team identified twenty-nine cause sequences. They found

that twenty-three of the twenty-nine were common to both prescribing problems and

monitoring problems.

Conclusion: The MU-PIs proved to be a useful tool to identify possible cases of

PDRM and to initiate "system" thinking or organizational introspection to evaluate

system related causes for drug related injury. Four themes can be interpreted from

CEA: lack of necessary tools for adequate patient information and assessment, an

information system that can track patients and relay information to the providers,

pharmacist involvement in the MUS, and guideline adherence.














CHAPTER 1
INTRODUCTION

Problem Statement

Injury related to health care is a serious public health concern. The prevalence of

drug related admissions in the United States has been reported to be from three to twelve

percent of hospital admissions.1 The Institute of Medicine has proposed a fifty percent

reduction in errors by 2005.2 For this achievement in medications use, the following are

needed:

* Measurement techniques to gather baseline information about preventable drug
related morbidity (PDRMs)
* Assessment of system related causes and rational interventions that target the
system failures
* Follow-up measurements to gauge the effects of the interventions.

Automated data screening for PDRM has been proposed as a method to produce

baseline and longitudinal measurements. However, the underlying system problems that

contribute to PDRM in the ambulatory setting are largely unknown. Studies measuring

the prevalence of drug-related hospital admissions have identified nodes in the

medication use process where the drug therapy problems originated, i.e., prescribing,

monitoring, dispensing and patient adherence. The underlying system related causes of

node specific problems, however, have not systematically been evaluated. This research

will try to discover the system factors that contribute to node specific problems.









Objective

The objective of this dissertation is to better understand the relationship between

system design and patterns of care that can result in drug related morbidity.

Specific Aims

The specific aims of this study are as follows:
1. To establish the frequency of medication use performance indicator (MU-PI)
positives in the study population
2. To evaluate population based explanations for MU-PI positives
3. To identify the node of the medications use system (MUS) where indicator specific
patterns of care likely originated
4. To identify system related causes that are common among nodes of the MUS
5. To identify system related causes that are unique to nodes of the MUS

Justification

This study has both theoretical and practical implications. The theoretical

contribution of this study includes:

* Determining the node of the MUS where the pattern of care from specific MU-PIs
originated. This permits the evaluation of the indicators by processes rather than
medication or disease specific analysis.

* Providing information about how the various subsystems (patient, microsystem,
organization, and environment) within the health care enterprise interact to
influence the quality of medication use.

* Establishing system factors that contribute to node specific patterns of care or
process problems. Identifying system factors that appear to be node specific
problems will reveal leverage points for interventions that may span medication
classes and disease categories.

* Establishing system factors that contribute to patterns of care or process problems
from multiple nodes of the MUS. Identifying system factors that appear to be
common among the nodes of the MUS will reveal leverage points for interventions
that may span nodes of the MUS, medication classes and disease categories.

This study will demonstrate how the MU-PIs can be used to assess the quality of

medication use in a defined population. It will also demonstrate how the MU-PI findings









can be used to initiate activities that lead to the exploration of how system design

influences the quality of medication use, i.e., organizational introspection.

From a practical perspective, the health care purchasing group involved in this

study is interested in increasing the quality of care and decreasing costs for the purchasers

they represent. Understanding how system factors influence the quality of medication

use will help direct the development of interventions to reduce the prevalence of adverse

drug outcomes.

Overview

The health care industry is wasting billions of dollars3 and causing an unacceptable

amount of injury due to inappropriate medications use.1,2'4 Results from our meta-

analysis of fifteen studies indicated that approximately four percent of all hospital

admissions may be preventable.1 To place the reported prevalence estimate into

perspective, according to the National Hospital Discharge Survey, there were 31,827,000

admissions to U.S. hospitals in 1998. If the prevalence of preventable drug-related

hospital admissions (PDRAs) happened to have been four percent in 1998, there would

have been roughly 1.3 million PDRAs. This would have placed PDRAs among the top

causes of hospitalizations in the U.S. that year-above admissions related to congestive

heart failure (3.1%) and on par with pneumonia (4.2%).

Ambulatory health care is complex, with many opportunities for problems in drug

therapy to occur. The basic nodes in the MUS include prescribing, dispensing, (self)

administration, and monitoring. Of the fifteen studies analyzed in the above-mentioned

meta-analysis,1 eight described the types of drug therapy problems that led to PDRA.5-12









An analysis of these studies indicated that prescribing and monitoring problems were

highly associated with PDRA (median: 34% and 32%), respectively. Problems with

medications compliance (median: 22%) were also implicated as a main contributor to

PDRA.

The finding that prescribing and monitoring problems appeared to be the main

contributors to PDRA is disturbing because health care professionals are expected to

"first do no harm." Even though practitioners are expected to manage their patients

appropriately and "do it right the first time," the health care system should be designed in

a way that anticipates human error and does not allow it to progress to an adverse drug

event or PDRM. Ideally, the health care enterprise should be able to deliver care in a

highly reliable manner where performance problems and bad decisions are recognized

and corrected because adverse events occur. Results from the Institute of Medicine

report on medical errors2 and from our meta-analysis1 show that the American health care

system is currently far from highly reliable.

Before the rational development of interventions, a data stream must be established

for baseline and follow-up measurements, and underlying system related causes need to

be examined. A recent doctoral dissertation at the University of Florida used a Delphi

approach to validate forty nine MU-PIs; each paired an inappropriate process (violation

of a standard of care) with an adverse outcome.13 An example of these indicators

follows: Use of two or more NSAIDS concurrently for at least two weeks in patients 65

or older followed by gastritis and/or upper GI bleed.


a The analysis can be found in Chapter 4 (preliminary analysis)









The indicators were automated and used to screen an administrative database of a

large MCO in North Florida. The data set contained 11,711 patients, and 8.2% of them

screened positive for a PDRM. For further validation, the positive predictive value of

process to outcome (PTOV) was calculated to measure the strength of the association

between the inappropriate process and the corresponding adverse outcome for each

indicator. Ten of the nineteen indicators that had more than ten positive screens had

PTOV values greater that 0.74. This suggests that this instrument would be acceptable

for eliciting cases of PDRMs for system level analysis.

Cause-and-effect analysis is a retrospective method for identifying system failures,

and its popularity has largely advanced from its success in understanding industrial

accidents.14 Rooted in industrial psychology and human factors engineering, cause-and-

effect analysis has recently been adopted in the medical community to evaluate quality

problems.

The MU-PIs adopted from Faris (2001)13 will be used to screen the database of a

health care coalition in Florida. MU-PIs with high frequency will be selected and

assigned to the nodes of the MUS where the drug therapy problems appear to have

originated from. Cause-and-effect diagramming on selected MU-PI that represent

specific nodes of the MUS will be carried out to uncover system related problems that are

common among the nodes and unique to specific nodes of the MUS.














CHAPTER 2
CONCEPTUAL FRAMEWORK

The framework of this study was developed from Hepler's conceptualization of

PDRM and the MUS,15 Berwick's framework for the health care system16 and Reason's

accident theory.17 The premise is drug-related morbidity is largely preventable due to

failures in the management of drug-therapy. A model of the MUS is used to illustrate

how drug-therapy problems (DTPs) can develop into PDRM. Health care is a complex

system with many layers of embedded systems. A framework presented by Berwick16 is

used to simplify the complexity by referring to specific levels of the heath care system.

Finally, James Reason's accident theory 17 is used to discuss the types of failures and

conditions that affect the performance of the MUS and allow PDRM to occur.

Adverse Outcomes of Drug Therapy

The term drug-related morbidity (DRM) is used to discuss adverse outcomes of

drug therapy. DRM is a concept that includes (a) significant adverse effects of drug

therapy (e.g., hospital admissions or emergency department visits) (b) treatment failures

(i.e., occasions when drug therapy was attempted but did not achieve a realistic, intended

outcome in a reasonable time) and (c) occasions when a patient did not receive an

indicated or necessary drug therapy.18 It is important to appreciate the relationship

between the popular term adverse-drug event (ADE), which was used to describe adverse

outcomes of drug therapy in the IOM report "To Err is Human," and DRM. The term

ADE historically has been used to describe injury resulting from medical interventions

related to a drug (i.e., direct effects of the therapy),4'19-22 without recognition of indirect









injury resulting from sub-therapeutic dose or no drug while a valid indication is present.

Even though it does appear that the definition for an ADE has recently evolved to include

indirect injury, 23 the term DRM is used in this study because it has historically treated

indirect injury as a significant adverse outcome related to drug therapy.

A DTP that precedes a DRM can be classified into two types: potential DTPs and

actual DTPs. The first indicates a person has a theoretical problem associated with drug

therapy, and the latter signifies the problem has manifested (symptoms are present).15

Theoretical DTPs exist independently of individuals; they are the situations in care that

produce the risk for specific adverse effects. Potential DTPs occur when a theoretical

DTP is present in an individual, e.g., contra-indications, drug interactions and unjustified

violations of evidence-based medicine. Actual DTPs are denoted by observable or

patient reported symptoms. For example, concomitant prescribing of digoxin and

quinadine is a known theoretical DTP because this combination can potentially alter the

excretion of digoxin, thus increasing its serum concentration, which is especially

troubling because digoxin is known to have a very narrow therapeutic window.

Now imagine a patient sixty-five years old who is concurrently receiving digoxin

and quinidine. This patient has a potential DTP because a theoretical DTP is present in

his drug therapy. Now, suppose this patient begins to experience fatigue, weakness,

confusion, and diarrhea. Most likely he is experiencing an actual DTP because

predictable symptoms known to be associated with the theoretical DTP have manifested.

If he/she becomes hypokalemic and needs emergency care, and upon admission his/her

digoxin level is in toxic range, say > 50 mcg/kg, then he/she most likely would have

experienced a preventable DRM (PDRM).









A preventable DRM has the following attributes :1518

* The DRM was preceded by a recognizable DTP
* The DRM was reasonably foreseeable under the circumstance
* The cause of the DTP and the resulting DRM was identifiable
* The identified cause of the DTP and resulting DRM was controllable within the
context of therapy (i.e., without sacrificing essential therapeutic objectives).

In this case the DRM meets the four criteria for preventability because it was

preceded by a known drug interaction (potential DTP) and the manifest symptoms (actual

DTP) were commonly associated with digoxin toxicity. Under the circumstances

hospitalization was imminent because the symptoms represent the accumulation of

digoxin, the cause of the symptoms appears to be the result of too much digoxin, and

reducing the dose of digoxin and monitoring the patient's digoxin levels or switching to

an alternative therapy could have resolved the DTP and prevented the DRM. A DTP is

part of the process of care, it is a state of an individual in a medications use system, and it

is a possible precursor to a system failure, i.e., DRM.

The Medication Use System

For a DRM to be preventable a failure in the process of care must have occurred.

This becomes clearer when considering the typical sequence of actions that comprise the

MUS in the ambulatory care setting. Figure 2.1 below was adopted from Grainger-

Rousseau et al. and it illustrates the nodes in the medications use system.24 The episode

of care begins when the patient notices a problem and seeks professional medical

attention. Typically, the initiator (physician, physician's assistant or nurse practitioner)

then assesses the problem and develops a clinical impression or diagnosis. Next a plan is

devised and the decision to prescribe a medication is made. A prescription is either

written or not; if it is, the patient will typically present the prescription for filling at the









pharmacy. Before dispensing the medication the pharmacist should look for DTPs and

advise the patient how to use and self their new therapy. The patient then consumes (e.g.,

self-administers) the medication. Follow-up visits and monitoring are required to gauge

the effects of the medication in the individual and to determine how well the prescribed

medication is working towards the therapeutic objective. It is the information gathered

from the monitoring node that is used in the decision to continue the current treatment or

make alterations to the medical regimen.

Errors or problems at each step may occur during a passage through this sequence

of events. Examples include failure to recognize an indication for drug therapy, incorrect

patient assessment, incorrect diagnosis, prescribing, dispensing, and administration and

monitoring.25

Our meta-analysis (2002) estimated the prevalence of preventable drug related

admissions to be about four percent.1 Based on our review and the IOM report, it appears

the current MUS are not optimal. This may be because MUS are more of a virtual than

an actual microsystem. As described by Nelson et al., (1998)26 an essential element for a

microsystem is an information environment to support the work of care teams.

Unfortunately, the MUS in most ambulatory care settings is fragmented and clearly lacks

a reliable information exchange among providers, especially between initiator and co-

therapists. The development and adoption of communication systems such as electronic

medical records and computerized prescription order entry are intended to make the MUS

safer.

Levels of the Health System

The IOM defines a system as "a set of interdependent elements interacting to

achieve a common aim." The elements may be human and/or non-human. 2 The health









care enterprise consists of many embedded systems that transcend a multitude of

domains, ranging from the subsystems used to make and keep track of patient

appointments to the environment where accreditation and financial systems influence the

care of populations. All systems have subsystems or nested systems that interact with

one another. Some systems are in a hierarchical relationship in which higher order

systems influence the functioning of lower order subsystems. Identifying the nesting of

systems and their relationships to one another is key to understanding the mechanics of

any system.

Berwick (2002) published a paper that he described as a users manual for the IOM

report, "Crossing the Quality Chasm."16 In this paper he addressed the issue of

embedded systems by presenting a framework for the different levels within health care.

He separates the system into the following levels: the experience of the patients and

communities (Level A), the functioning of small units of care delivery called

microsystems (Level B), the functioning of organizations that house or otherwise support

microsystems (Level C) and the environment of policy, payment, regulation,

accreditation and other factors (Level D).

Level A is not actually framed in terms of a system; instead it represents the

patients' experience and perception of the care they received. Berwick states,

Rooted in the experiences of patients as the fundamental source of quality, the
report shows clearly that we should judge the quality of professional work, delivery
systems, organization and policies first and only by the cascade of effects back to
the individual patient and to the relief of suffering, the reduction of disability and
the maintenance of health.16

This is a significant paradigm shift from previous quality approaches that were

independent from patient outcomes. Determining quality based on the patients'

experience is allied with using outcomes of care to gauge quality. Donabedian (1978)









defines outcomes as the "primary changes in health status that can be attributed to that

care."27 Health status is determined by psychological factors and social performance

(subjective component), and physiological factors (objective component).

In this paradigm, the patients' experience is defined as the sole way of determining

the quality of other systems. The decision to place emphasis on the patient is strategic; it

acknowledges the patient's subjective experience and clinical outcomes are the primary

focus of quality rather than the dynamics of the microsystems.

The microsystems (Level B) are small units of work that actually gives care the

patient experiences.16 Clinical microsystems are basically "small organized groups of

providers and staff caring for a defined population of patients."28 They are composed of

patients who interact with clinical and support staff who perform various roles, e.g.,

physician, nurse, pharmacist, medical assistant, data managers, receptionist, etc. Nelson

et al. (1998) have described the essential elements of a microsystem as (a) a core team of

healthcare professionals; (b) the defined population they care for; (c) an information

environment to support the work of care team and patients; and (d) support staff,

equipment and work environment.26

In terms of medications management, the patients and clinical staff manage drug

therapy by engaging in direct care processes, which include recognizing, assessing and

diagnosing the patient's problem, along with developing a treatment plan, dispensing and

educating, and monitoring to make sure the treatment is progressing as planned. Direct

care processes are assisted by supporting processes that involve distinct tools and

resources such as medical records, scheduling, diagnostic tests, medications, billing, etc.









To move to a more reliable system of care, the performance of microsystems must

be optimized and the linkages between clinical microsystems must be seamless, timely,

and efficient. Change at the microsystem level is an important opportunity to focus on

the transformation of care at the front line of the health care service industry.29

Microsystems do not exist in a vacuum; they are embedded within the organizations that

help orchestrate their relationships with each other.

The health care organizations (Level C) are establishments that house and support

microsystems--they provide the necessary resources for microsystems to deliver care.

Health care delivery requires personnel, financing, technology, and facilities. Common

organizations include hospitals, provider groups, independent practice groups, nursing

homes, ambulatory surgery centers, and pharmacy benefit managers, all of which are

typically embedded within managed care organizations. Group practice, for example, is

an affiliation of three or more providers, usually physicians, who share income, expenses,

facilities, equipment, medical records, and support personnel in the provision of services,

through a legally constituted organization.30 They are embedded within managed care

organizations through such integrating mechanisms as referral arrangements, insurance

contracts, and in some cases direct ownership of practice.

Managed care is focused on controlling the functioning of microsystems, i.e.,

controlling provider and patient behaviors, and most of that control is realized through

the ambulatory care arena.30 Managed care has shifted toward an organizational form

with greater control over resources, providers, and patients within microsystems of care.

The health care environment (level D) includes multiple systems that influence

activities of the organizations and microsystems. Important environmental systems









include financing, regulation, industry, accreditation, policy, litigation, professional

education and social policy.16

Berwick (2002) describes the relationship among the four levels as the chain of

effect of improving health care quality. He states that,

The quality of the microsystem is its ability to achieve ever better care: safe,
effective, patient-centered, timely, efficient and equitable. The quality of the
organization is its capacity to help microsystems produce safe, effective and
efficient patient outcomes. And the quality of the environment--finance,
regulation, and professional education is its ability to support organizations that
can help microsystems achieve those aims.16

Figure 2.2 was designed to show the hierarchal relationship among the different

levels of health care, as well as the MUS's location within the microsystems. As

illustrated, the MUS intersects the "microsystems" and the patients' experience. The

spiral between the microsystems and the "patient level" indicates that the patients'

experience is the input into the MUS as well as the output, which is then used as feedback

to determine the performance of the microsystems. What was more difficult to illustrate

is the interaction among the microsystems. In an ideal MUS the interaction between

physicians, laboratory, pharmacists, case managers, etc., would be seamless with

information flowing freely among the microsystems and cooperation among the

professionals would be standard.

The hierarchical stacking from environment to organizations to microsystems to the

patient represents the main route of influence. It should be noted that this is not the only

direction of influence, organizations can use lobbyists to leverage influence over the

regulatory and financial environment. Likewise, microsystems can organize, like in the

case of provider groups, to counter the pressures from higher-up organizations such as

managed care.











RECOGNIZE ASSESS CLINICAL
PATIENT PATIENT 7 IMPRESSION Nl PLAN
PROBLEM PROBLEM (DIAGNOSIS) PRESCRIBE



IMPLEMENT
PLAN
DISPENSE &
FOLLOW-UP ADVISE
MONITORING


IMPLEMENT PLAN
ADMINISTER/CONSUME
Figure 2.1. Model of the Medication Use System


Human Error and System Failure

James Reason provides two views of how accidents occur within an organization -

the person approach and the system approach.17 These approaches are fundamentally

different, with each giving rise to contrasting philosophies of performance management.

The person approach (human error) focuses on unsafe acts. In medications use, the

attention would be on errors and procedural violations of health professionals and

patients. This approach views unsafe acts as divergent cognitive and behavioral

processes such as forgetfulness, inattention, carelessness, negligence and recklessness.

Historically, the most popular approach to countering human error has been to find

the operator or operators who committed the unsafe act and discipline them accordingly.

This is often referred to as blaming, and sometimes as "scapegoating." Even though

blaming may produce satisfaction it does nothing to correct the conditions that allowed

unsafe acts to progress to injury. Because errors are often made in the normal course of

providing care, traditional efforts at error reduction, which focused on individuals and









episodes by using training, rules and sanctions to improve performance, are considered

less effective than altering the system to remove or reduce the conditions that increase the

likelihood of adverse outcomes.31 Counter measures for reducing unwanted behaviors

are analogous to the "Whack-A-Mole" carnival game: you can whack a mole back into

his hole, but another is sure to challenge your skill by popping-up in a different location.


Figure 2.2. Hierarchal Relationships among the System Levels

The basic principle in the system approach is humans by their nature are fallible

and errors are expected even in the best organizations. From this perspective, injury

occurs because errors interact with flaws in the design of systems called latent

conditions.17


0-j









Latent conditions are the flaws in the design and organization of a system. In

health care they can arise from the decisions made by designers; however, it is believed

many latent conditions arise not from design but from self organization, which is an

adaptation to evolving systems and an unsettled environment.32 Latent conditions have

two types of adverse effects: they can produce error-provoking conditions and they can

produce longstanding holes or weaknesses in the defense mechanisms, i.e., error

detection mechanisms. When latent conditions combine with errors the opportunity for

PDRM occurs. In the Swiss cheese model presented in Figure 2.3, the holes in the

cheese are latent conditions that allow potential injury (i.e., drug therapy problems) to

manifest as drug related morbidity.

Summary

PDRM is a system problem. The nature of preventability means problems occurred

in the process of care. Patient injury occurs when human error interacts with latent

conditions of a system. The health care system is a complex network of interacting

systems and subsystems. Understanding how microsystems, organizations and

environment influence activities of the MUS and produce opportunities for human error

to result in PDRM will increase our ability to predict and resolve DTP before they

manifest as patient injury.


b This Swiss cheese was published by Hepler and it is a modification of Reason's Swiss cheese model15






















Figure 2.3. Swiss Cheese Model of System Failure














CHAPTER 3
LITERATURE REVIEW

This chapter will first discuss the use administrative data for scientific

investigation. The use of automated methods for detecting the incidence and/or

prevalence of PDRM will then be discussed. Next, experimental studies evaluating the

effectiveness of the Delphi process and its use in health care will be presented. Lastly,

techniques for cause-and-effect analysis will be presented.

Use of Administrative Databases for Researching PDRM

Administrative databases are derived from information produced by health care

providers and institutions in billing for products and services. Claims are filed from

institutions (i.e., inpatient hospital stays and outpatient visits) health professionals and

pharmacies for reimbursement of products and services from payer organizations such as

Medicaid, Medicare and private insurers. The claim form used depends on where

services were rendered and who provided them. Most hospital inpatients and acute care

outpatient services are submitted for payment with the uniform billing 92 (UB-92)

format. The UB-92 was derived from the Uniform Hospital Discharge Data Set, which

was formulated in 1972 by the National Committee on Vital and Health Statistics, U.S.

Department of Health, Education, and Welfare. The goal was to create a uniform but

minimum data set to facilitate investigation of cost and quality of hospital services across

populations.33 Health care services provided by single practitioners or practitioner groups

are submitted for payment in a format called the CMS-1500 (formally the HCFA-1500).

This is the common claim form for non-institutional providers that is updated by the









Center for Medicare Services (CMS) and is approved by the American Medical

Association Council on Medical Services. Both the UB-92 and the CMS-1500 forms

typically include details for each compensated service, including diagnosis, and

procedures, date, place of service, provider and patient identifiers and charges.

From a research and quality improvement perspective, the potential utility and

credibility of a database stems from its clinical content. In administrative data in the

United States, information on clinical diagnosis and conditions are documented with

International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-

CM) diagnostic codes. In addition, administrative data derived from hospital reports

(UB-92) use ICD-9-CM codes to document procedures, while procedures are

documented using Common Procedure Terminology (CPT) in data derived from

professional claims (CMS-1500).

The reliability and validity of coding is a serious issue when considering the use of

administrative data for research. In relation to hospital admissions, the two major steps in

coding an admission are first specifying the pertinent diagnosis and second ordering

them. The principal diagnosis in a UB-92 form defines the cause of admission. The

sequencing of diagnosis does impact reimbursement rates and mistakes or miscoding can

affect the accuracy of the data.

As previously mentioned, the validity of claims data is a cardinal issue when

considering its use for health services research. Threats to validity and two approaches

for evaluating the ability of claims data to accurately identify patients with specific

conditions will now be presented. The following threats to internal validity (i.e.,

misclassification bias) have been identified misspecification, resequencing, miscoding,









and clerical errors.33'34 Misspecification occurs when the attending physician selects an

incorrect diagnostic code for the principal diagnosis, listing of diagnoses or procedures.

Misspecification was identified as the leading source of error in a summary of studies that

evaluated coding quality.34 Resequencing is the substitution of a secondary diagnosis for

the correct principal diagnosis during the coding process and was found to be the second

most frequent source of misclassification.34 Miscoding is the coding of diagnoses or

procedures not attested to by the physician, misapplication of coding rules, or selection of

an unnecessarily vague diagnosis code.

Now that the types of misclassification bias have been presented, studies that

evaluated the validity of claims for research data will be discussed. Quam et al. (1993)

investigated the validity of claims data for epidemiologic research and found evidence

that supports its use.35 The researchers evaluated the ability of claims data to identify

patients with essential hypertension. The claims database used consisted of all

ambulatory, pharmacy and hospital claims from 1988 to 1989 in two large managed care

organizations. They identified essential hypertensive patients by three strategies. One

strategy was based on medical service claims alone (diagnosis without pharmacy claims),

another was based on only pharmacy claims (pharmacy claims without diagnosis), and

the third strategy was based on both medical services and pharmacy claims.

Diagnostic codes were validated by comparing findings with patient survey and

medical records. The strategies based on medical claims alone and pharmacy claims

alone exhibited low positive predicted values (PPV) with the medical record (47% and

50%, respectively) and the patient survey (43% and 63%, respectively).35 The









combination strategy, however, exhibited very high PPVs (96% for both the medial

record and patient survey).35

More recent information on the ability of administrative data to predict disease

outcomes is described in the Study of Clinically Relevant Indicators for Pharmacological

Therapy (SCRIPT) report.36 The objective of the SCRIPT project was to develop a core

set of valid and reliable performance measures to evaluate and improve the quality of

medication use. The common focus was cardiovascular disease outcomes and risk

factors. The candidate measures were tested in managed care organizations and practice

groups from eight states. The study evaluated patients with coronary artery disease

(CAD), heart failure (CHF), and atrial fibrillation (AFIB). The researchers evaluated the

PPV of specific ICD-9-CM and CPT codes for identifying the cardiovascular conditions

mentioned above. The requirement for the number of codes to identify specific

conditions was varied from one to three and the effects on yield and PPV were

determined.

The PPVs from the SCRIPT study averaged 89%, 89%, 84% for AFIB, CAD and

CHF, respectively. The state-specific values for AFIB ranged from 80% to 98%, the

values for CAD ranged from 81% to 95% and the values for CHF ranged from 78% to

94%. Requiring an additional code raised the PPV no more than four percentage points

on average and raised the worst PPV as much as eight percentage points. Requiring three

codes added little to PPV; however, it significantly reduced the yield. The most marginal

improvement in PPVs was found by requiring two codes vs. three codes. The two code

requirement lowered the yields by 13%, 14%, and 20%, respectively for AFIB CAD and

CHF









In summary, although administrative databases were not originally designed for

research, they have become a rich source of information for health services research.33

Nevertheless, threats to internal validity, specifically misclassification bias need to be

taken seriously when interpreting studies that used claims data. Studies that have

evaluated the ability to identify patients with specific conditions using administrative data

had good positive predictive ability. Quam et al. (1993) reached a PPV of 96% when

requiring medications used to treat essential hypertension and a diagnosis of essential

hypertension.35 The SCRIPT project was able to obtain good PPV while maintaining a

decent yield when they required two codes (ICD-9-CM or CPT) for specific

cardiovascular conditions.36

Automated Methods for Detecting PDRM

Chart review is the "gold standard" for PDRM ascertainment. The articles

reviewed by Winterstein et al. (2002)1 (discussed in the following chapter) used medical

chart review to measure the prevalence of drug related admissions (DRAs). This

typically included reviewing the medical records of all patients admitted to the hospital or

specific hospital units during a defined period of time to determine if the reason for

admission was drug related. The detection and classification method was often implicit,

which means the reviewers judged whether or not a DRA occurred by comparing the care

processes of that patient against his or her own knowledge, opinions, and beliefs about

how appropriate care should be carried out. Implicit methods may be more sensitive

because they allow experts to capture case-specific elements of information for the

judgment of preventability, however, their reliability is questionable.37 Furthermore,

chart-review is time and cost consuming and is not useful in the continuous improvement









paradigm where baseline and many follow-up measures are needed. Computerized

methods to identify DRM with explicit criteria are an alternative to chart-review. 3839

A PubMed search identified three studies with different approaches for detecting

ambulatory acquired DRM with computerized data. One method identified DRM with

search algorithms used to screen laboratory data and pharmacy records. Another method

used multiple types of search algorithms and databases to identify cases of ADEs. The

last method screened administrative claims data by using automated PDRM performance

indicators. These studies will be presented and special attention is given to the following

issues: the predictive validity of the instrument, the ability of the indicators to capture

preventable events, the scope of the database, and the sophistication of the software used

to run the indicators.

Jha et al. (2001)39 adapted published screening rules (i.e., search algorithms)

developed from the inpatient LDS study in Utah.40 The rules were used to estimate the

rate of DRA to the study hospital. Every day the computer generated a list of alerts. The

alerts were validated by reviewing each patient's medical record that screened positive

for a DRM on admission.

Jha et al's. (2001) screening rules can be separated into at least four categories:

antidotes, chemistry/drug levels, and physiological response to a specific therapy and

drug interaction. (Table 3.1) The information system used integrated pharmacy,

laboratory test results and a sophisticated physician order entry systems. Twenty two

rules required laboratory test results and nineteen screened for antidotes of known

adverse consequences of drug therapy (e.g., naloxone, a narcotic antagonist).









Table 3.2 shows the results of the most frequent screens. The overall PPV for the

instrument was 3.5% with a range from 2.0 to 100%. The screen for serum phenobarbital

levels < 45 mg/dl had a PPV of 100%. The screen for patients receiving charcoal

(activated) had the second highest PPV of 45%. All other screens had PPVs below 12%.

The overall PPV for this instrument and the individual screens' PPVs, excluding

phenobarbital levels, indicated that these screens picked up a tremendous amount of

noise. Furthermore, the majority of the indicators were directed toward toxic levels of

medications and abnormal blood chemistry levels. While these surrogate outcomes may

be able to indicate the presence of DRM, they do not provide information about the care

process that led to the event, i.e., preventability. Additional chart review or investigative

work is needed to judge preventability. Furthermore, the software used to search the data

was quite sophisticated (physician order entry and an event monitor). Nevertheless, less

sophisticated software such as SAS or Access could be used to write search algorithms

for the screens.

Honigman et al. (2001) took a more elaborate approach for using computerized

information to screen for ADEs.41 This approach used a sophisticated computer program

which consisted of four search methods: ICD-9 codes, allergy rules, computer event

monitoring rules, and an automated chart review using text searching.

ICD-9-CM codes associated with ADEs were used to screen the database. The

computer event monitoring rules were based on the same screening criteria as Jha et al.

(2001) mentioned above.39'40 Allergic reactions to medications were detected with a text

search of the medical records. The software program M2D2 was used to expand the

patients' drug allergy list to include product names, generics and ingredient. Patient










records were screened for allergies along with offending medications. The M2D2

program also matched terms in the medical record with known adverse effects of drugs

the patient was taking.

Table 3.1. Examples of DRM Screens by Type (Jha et al. 2001)

Antidotes Chemistry and Blood Levels Response to Drug and Interactions
Receiving betamethasone Receiving "nephrotoxin" AND blood creatinine
Serum digoxin > 1.7 ng/mL
dipropionate 0.05% has risen > 0.5 g/dL in last 1 day
Receiving charcoal Receiving ranitidine AND platelet count has fallen
Serum lidocaine > 5.0 mg/mL .
(activated) ......... to less than 50 of previous value
Receiving racemic Serum phenytoin results within
Receiving diphenoxylate with atropine
epinephrine hcl last I day are> 20 mg/mL
S> 10 m Receiving benzodiazepine AND receiving anti-
Receiving atropine sulfate Serum bilirubin > 10 mg/dL
epileptic
> 6.5 mmL Receiving phytonadione (vitamin K) AND order
Receiving naloxone Serum potassium > 6.5 mmol/L warfarin within last 14 days
for warfarin within last 14 days

Table 3.2. Results of Leading DRM Screens (Jha et al. 2001)
DRM screening rule Screen + True + PPV (%)
Patient receiving predinisone 313 12 3.8
Patient receiving diphenhydramine 300 9 3
Allergy entered 113 10 8.8
Patient receiving oral matronidazole/vancomycin 87 6 6.9
Serum digoxin > 2.0 ng/ml 61 5 8.2
Serum phenytoin > 20mg/dl 47 4 8.5
Patient receiving kaopectate 35 4 11.4
Patient receiving charcoal 11 5 45.5
Serum phenobarbital >45 mg/dl 4 4 100
Other 1,649 33 2
Total 2,620 92 3.5

Honigman et al. (2001)41 found allergy screens to have the highest PPV (49%), but

these screens did not detect the greatest number of ADEs (104). Text searching detected

the most ADEs (1,637) but it had many false positives (PPV = 7%). Searching for only

ICD-9-CM codes had the lowest PPV (2%) and it only detected five events. The event

monitoring rules detected 60 events with a PPV of 3.3%. The PPV for the composite tool

was 7.2%. (Table 3.3)









Table 3.3. Results of DRM Screens
Rule Class Screen positive true positive PPV, %
ICD-9-CM 248 5 2
Event Monitoring Rules 1,802 60 3.3
Allergy 214 104 48.6
Text Search 22,798 1,637 7.2
Composite 25062 1806 7.2

This approach was more comprehensive than Jha et al's. (2001) detection method.

The rules were more dynamic because they were able to identify less severe adverse drug

effects. The term adverse drug event was used instead of DRM when describing this

study because many of the effects screened for would not fit the definition of a DRM.

The events would be considered actual DTPs under Hepler's conceptual framework. For

example, text searches for signs and symptoms such as dizziness, fatigue, cough, and

over anticoagulation without bleeding were major components of the screening

instrument. Only nine percent of the adverse events identified required hospital

admissions.

The level of programming and database (administrative vs. clinical data)

sophistication needed to run the rules varied. The ICD-9-CM rules did not require

sophisticated software and were usable with administrative databases. The event

monitoring rules did not require sophisticated software either, programming could be

done with Access or a statistical program such as SAS. Nevertheless, clinical data (i.e.,

laboratory test results and medical records) was required. Most administrative records

only indicate that specific types of laboratory tests were requested or conducted. Patient

specific laboratory values are not documented. The allergy and text search rules were

more sophisticated and they required electronic medical records. Honigman et al's.

screening method had advantages over Jha et al's. method. Nevertheless, the level of









sophistication limits the use of these rules to organizations with access to electronic

clinical data and specific software.

The PPV of Honigman et al's. composite tool was relatively low (7.2%). The

screens were the most comprehensive and could detect minor adverse drug events as well

as events that led to hospital admission; however they were not designed to indicate

whether or not the events were avoidable or if the admissions were preventable. The

screens varied in their use of databases, they used administrative data, laboratory test

results and electronic clinical records. The software needed to run the text searches was

quite sophisticated.

Faris (2001) developed a different approach to electronically detect PDRM.13 His

approach automated forty-nine PDRM indicators that paired an inappropriate process of

care (potential DTP) with its predictable adverse outcome (DRM). The indicators were

originally developed by MacKinnon (1999)42 from a literature search of peer-reviewed

medical articles and reference texts from 1967-1998, and from a consensus panel of

seven experts in geriatric medicine.

Faris (2001) revalidated the PDRM indicator definitions through a Geriatric

medicine expert consensus panel. Forty-four of the fifty-two indicator definitions

accepted in MacKinnon's study were accepted for automation in the Faris study.

Additional indicators were proposed and accepted by the panel for a total of forty-nine

indicators.

Once the indicator definitions were accepted they were translated into a format that

could be applied to a managed care administrative database. The translation was

designed to capture the events described in the indicator definitions through billing









records from physician offices, hospitals, pharmacies, and other care sites in the managed

care network. Services are billed through a system of codes that represent different

treatments and activities. These codes included ICD-9-CMs, CPTs, and the National

Drug Codes (NDC). SAS was used to extract and analyze cases that fit the indicator

definitions.

Faris (2001) retrospectively sampled 11,711 elderly patients over a one-year period

and 966 had at least one positive for a PDRM indicator. Chart review was not done to

validate the performance indicators. As an alternative, the proportion of patterns of care

in the population that appeared to result in the corresponding adverse outcome was

calculated (PTOV). Ten of the nineteen indicators, which had more than ten positives,

had PTOV values greater that 0.74 percent. Of the five most prevalent indicators, three

had adjusted PTOVs greater than or equal to 75%, which means the process of care was

strongly associated with the corresponding adverse outcome. (Table 3.4.) For example,

Asthma exacerbation and/or status asthmaticus and/or ER visit/hospitalization due to

asthma was found 100% of the time when the following pattern of care occurred:

Diagnosis of moderate to severe asthma and use of a bronchodilator with no use of

maintenance corticosteroid.

The criterion validity of PDRM indicators developed by Faris (2001) showed

promise with ten indicators having adjusted PTOVs of 75% or greater, nevertheless the

PPV of the PDRM indicators is unknown and needs to be tested. MacKinnon (1999) did,

however, examine the criterion validity of two indicators that occurred frequently enough

for statistical validation and found relatively high PPVs, 82% and 34%.42









Unlike the approaches taken by Jha et al. (2001) and Honigman et al. (2001), these

indicators are specifically searching for preventable cases of DRM. The preventability is

represented by the inappropriate process and its predictable adverse outcome.

Unfortunately, only two of MacKinnon's PDRM indicators were validated with chart

review. Nevertheless, the PPV of the two indicators was decent and this method shows

promise as valid PDRM measures. Another advantage of this approach was it did require

sophisticated software. The search algorithms were produced with SAS. Furthermore,

clinical data is not needed-these indicators were designed for administrative data.

In summary, three studies with different computerized methods to detect PDRM

were found in the literature. The approaches varied in ability to detect true cases of

PDRM, and identify preventable events. The type of data needed, and the sophistication

of the software used to produce search criteria also varied. The approach used by Faris

(2001) will be adopted for this dissertation, because the PDRM performance indicators

from that study included preventability in their operationalization. Furthermore, the

indicators were intended for claims data and the search algorithms can be produced with

basic statistical software, i.e., SAS.

Delphi Method

The Delphi method is a systematic approach for the utilization of expert opinion

that was extensively studied by the RAND corporation in the 1960's.43 The concept of

the Delphi technique is simple; it is designed to define a single position (consensus) of a

group by systematically combining their opinions in a way that eliminates activity among

the experts to reduce dominance and group pressure.











Table 3.4. Results of Top Five Indicators
Positive Adjusted
PDRM Indicators Poiti d
Indicator PTOV
This outcome has occurred after the pattern of care below:
ER visit/hospitalization due to congestive heart failure
This is the pattern of care: 270 0.75
1. Diagnosis/history of congestive heart failure
2. Not on ACE inhibitor
This outcome has occurred after the pattern of care below:
ER visit/hospitalization due to congestive heart failure and or heart block
This is the pattern of care: 184 0.95
1.History/diagnosis of congestive heart failure with heart block or advanced bradycardia
2. Use of digoxin
This outcome has occurred after the pattern of care below:
ER visit/hospitalization due to hypothyroidism
This is the pattern of care: 129 0.12
1. Use of a thyroid or antithyroid agent (e.g.; levothyroxine, etc.)
2. T4/TSH not done before therapy starts and at least every 12 months
This outcome has occurred after the pattern of care below:
Gastritis and/or upper GI bleed and/or GI perforation and/or GI ulcer and anemia 103 0.45
This is the pattern of care:
1. Use of 2 or more NSAIDS concurrently
This outcome has occurred after the pattern of care below:
Asthma exacerbation and/or status asthmaticus and/or ER visit/hospitalization due to asthma
This is the pattern of care: 89 1
1. Diagnosis of moderate to severe asthma
2. Use of a bronchodilator
3. No use of maintenance corticosteroid

The Delphi technique is a modification of the traditional roundtable group decision

making process. The roundtable approach has inherent limitations because the final

decision or position may be an effect of the most outspoken person (i.e., the bandwagon

effect) or socially acceptable opinion rather than the group's true opinion.43 The Delphi

technique avoids these undesirable effects by replacing the group meeting with

anonymous response. Instead meaning the opinions of members are obtained by formal

questionnaire.

The procedures of the Delphi technique have the following key features :44

* Anonymous response-opinions of members are obtained by formal questionnaire
or other formal communication channels as a way to reduce the effects of dominant
individuals.

* Iteration and controlled feedback-the interaction among the members is supported
by a systematic exercise conducted in several iterations, with carefully controlled









feedback between rounds. The summary of the results from the previous round are
communicated to the participants

* Statistical group response-the group opinion is defined as an appropriate aggregate
of individual opinions in the final round. The use of a statistical definition of the
group response is a way of reducing group pressure for conformity; at the end of
the exercise there may still be a significant spread of opinion.

In the 1960's the RAND Corporation carried out a series of experiments to evaluate

Delphi procedures and to explore the nature of information processing.44 The

experiments involved fourteen groups of upper-class and graduate students from UCLA

and they ranged in size from eleven to thirty members. The three effects that researchers

examined were group size, a comparison of face-to-face discussion with the controlled-

feedback interaction, and the ability of controlled feedback as a means of improving

group estimates. They found increasing group size decreased the average group error and

increased reliability. The anonymous controlled feedback procedures of the Delphi

technique made the group estimates more accurate than face-to-face discussion. They

also found controlled feedback narrowed the dispersion around the median.

A large set of experimentally derived answers to factual questions was evaluated to

determine the relationship between group size and the mean accuracy of a group

response.44 In this study the experimenters knew the answers to the questions; however,

the subjects did not. The group error was calculated as the absolute value of the natural

algorithm of the group median divided by the true answer. Dalkey (1969) found the

gains in increasing group size were quite large from three to eleven members, but the

percent change in group error was less than five percent past eleven members. (Table

3.5.)









Table 3.5. Percent Change in Average Group Error by Number of Group Members.
(Extrapolated from Dalkey (1969))
Group members Average group error Percent change
1 1.194
3 0.8521 28.63%
5 0.6882 19.23%
7 0.6097 11.42%
9 0.572 6.18%
11 0.5539 3.16%
13 0.5453 1.56%
15 0.5411 0.76%
17 0.5391 0.37%
19 0.5382 0.18%

Dalkey (1969) also compared the reliability of two groups' opinion with various

numbers of members. Reliability was measured by the correlation between the answers

of the two groups over a set of questions. Dalkey found a definite and monotonic

increase in the reliability of the group responses with increasing group size. A mean

correlation of 0.8 was obtained with thirteen group members.

Even though these two studies showed increasing group size reduced the error of

the group and increase the reliability of the estimate, it appears these estimates were

obtained with traditional group consensus and not by a Delphi process. If these results

are generalizable to Delphi group estimates then having a Delphi group with between

eleven and fifteen members would seem reasonable because average group error only

decreased by approximately three percent from eleven to thirteen members and reliability

reached 80% with thirteen members.

In the main experiment the performance of groups using face-to-face discussion

was compared with Delphi groups. The first experiment involved two groups of five

graduate students, and twenty questions were presented in four blocks of five using the

ABBA design (A= face-to-face). The face-to-face group was instructed to follow a









specific procedure for each question and the Delphi procedure involved four rounds of

estimates, feedback of medians and quartiles and re-estimates. The median response of

the Delphi group was more accurate in thirteen cases, and the face-to-face group was

more accurate in seven cases.

The second experiment had a different study design. The Delphi group had twenty-

three members and from them the face-to-face groups were separated into seven groups

of three and one group of one. In this experiment they found that Delphi estimates were

similar to face-to-face estimates in terms of accuracy, however, face-to-face estimates

produced more changes from initial estimates that reduced accuracy.

Dalkey also examined the distributions of the answers between the first-round and

second-round of the Delphi groups. The second-round distribution shifted toward the

mean and Dalkey concluded the shift represents a convergence of answers toward the

group response. Nevertheless, the second-round distribution still had a large range,

indicating convergence was not complete.

Dalkey also evaluated the effects of iterations on the accuracy of responses. To do

this, changes with regards to individual questions were measured. The median improved

in accuracy for about 64%, while the median became less accurate in 31%. The

mechanism of improvement appeared to be from first-round feedback of the median

response. The tendency to change was determined by the distance of the first round

answer to the first round median. However, this did not explain the change in median

from round-one to round-two. Some respondents had to cross the median for a change in

median to occur.









To evaluate how the median value changed from the first to second round, the

researchers divided the groups into holdouts and swingers. Holdouts tended to cluster

around the median and they were more accurate than the total group in the first round.

The median of the group was between the median of the swingers and the median of the

holdouts. The shift in the group median occurred because the swingers shifted toward the

median, thus shifting the group median.

In a supplementary analysis Dalkey (1969) tested the effects of other forms of

feedback on the accuracy of the Delphi estimates.44 The experimental group documented

reasons for their decisions when their second round response was outside the interquartile

range of the first round. Formulating and feeding back reasons did not increase the

accuracy of the initial estimates or produce improvement on iteration.

In a follow-up study Dalkey et al., (1969) evaluated the use of self rating to

improve group estimates.45 This study used 282 University of California students.

Subjects were randomly assigned to sixteen groups with fifteen to twenty members per

group. Almanac type questions were used to assess the relationship between accuracy

and self rating. Subjects were given the questions and asked to rate each question from

one to five to indicate their knowledge of the content in question. Relative rates were

used, meaning subjects had to identify a question they were most knowledgeable about

and give it a five. They also had to identify a question they knew the least about and give

it a one. A clear association between average group self rating and group estimate

accuracy was found. This study also found a significant improvement in the

effectiveness of the Delphi procedures can be obtained by using self-rating information to

select more accurate subgroups.









The experimental studies on the Delphi process are important because they provide

information on how group processes affect the accuracy of group estimates to factual

questions. In this dissertation the Delphi method will be used to assign Faris' indicators

to nodes of the MUS where the process failure likely originated. In this case, a known

answer does not exist. The opinions of the group members will be aggregated to assign

select indicators to nodes of the MUS. Therefore the qualifications and diversity of

experts in domain specific knowledge will be important factors to consider when

constructing a Delphi panel.

Cause-and-Effect Analysis

To improve the performance of the MUS it is first necessary to understand error

etiology within the MUS and the underlying system design factors that contribute to

PDRM. Cause-and-Effect analysis (CEA) is a structured investigation that aims to

identify the true cause of a problem, and the actions necessary to eliminate it.46 CEA is

an integrated problem solving methodology that incorporates multiple tools and

strategies, which include process analysis, brainstorming and cause-and-effect

diagramming.46,47

First, process analysis is conducted to understand the processes involved in the

quality problem. Before brainstorming or cause-and-effect diagramming is conducted an

understanding of how the system works is needed. Process analysis should include flow

diagrams that represent different nodes of the system. The steps needed to construct a

flow diagram include:47

1. Defining the basic nodes of a system
2. Further defining the process, breaking each node down to specific steps needed to
complete the process
3. Following the objects through the process a number of times to verify the process
by observation









4. Discussing the process representation with the project team and reaching consensus
on the underlying process.

Brainstorming is a possible cause generation technique.47 Brainstorming helps a

group to generate many thoughts or ideas in a very short period of time.46 The important

aspect of brainstorming is ideas are to be generated without judgment or discussion until

all the ideas are presented. In a typical group discussion someone presents a thought and

others comment on it, or judge it. The problem is group members may focus on one idea

before all other possible ideas have a chance to emerge.46 Brainstorming encourages the

flow and fluency of group member thoughts.

Brainstorming has three steps: generation, clarification and evaluation.46 In the

generation phase the leader clearly states and writes the question or purpose and then

invites and records responses. This step can either be structured or unstructured. In the

structured approach, known as round-robin, each participant in turn launches one idea,

this insures equal participation, but is less spontaneous and may limit the possibility for

building on one another's ideas.46 In unstructured brainstorming everyone can freely

launch ideas, this a spontaneous process, but it can become confusing and it may lead to

one or more persons dominating the activity.46

The second step in brainstorming is clarification. After all ideas have been

generated, the group reviews them to make sure everyone is clear about their meaning.

The focus of the generation phase was quantity and not quality, therefore, during

clarification group members are encouraged to question the meaning of the generated

items.

The final step of brainstorming is the evaluation of clarified items. Here the group

considers the list and rules out duplications, and irrelevant ideas. During the evaluation









phase affinity diagramming or other methods for grouping ideas into similar concepts

may be employed to make sense out of the generated items.

Cause-and-effect diagramming is the heart of CEA. It is a tool used to define and

illustrate the relationships between an effect and an outcome, or a problem and beliefs

about the possible causes or factors contributing to it.46 It combines the product of

brainstorming with a systematic analysis to organize and evaluate causes to determine

which are most likely contributing factors or causes.46 Cause-and-effect diagramming

can be done using fishbone or tree diagrams. Fishbone diagrams are the traditional way

of illustrating cause-and-effect relationships. Nevertheless, tree diagramming, or the five

whys analysis, is an effective way to graphically show the breakdown of large problems

into their increasingly more detailed elements.46 Tree diagrams help the group members

to move from general to specific, or vice versa, in an organized manner, and they show

the logical connections among the relationships.

Accreditation organizations such as the Joint Commission on Accreditation of

HealthCare Organizations (JCAHO) have recognized the importance of CAE and have

incorporated the use of root cause analysis (RCA), which is a more specific form of CAE,

into their quality criteria. Currently, JCAHO only requires RCA for inpatient sentinel

events. Causes of DRM in the ambulatory setting have not received much attention and

information on CAE for adverse outcomes of drug therapy in the outpatient setting to my

knowledge is non-existent.

In the Joint Commission paradigm, the data used as input for RCA are the

collection of activities that led to the specific incident. Activities are determined by

interviews with providers involved in that particular patient's therapy and by reviewing









the patient's medical chart. While this approach is ideal for sentinel events in the

inpatient setting because the quality team has access to providers involved in the

particular event and the patients' medical record, it is not as useful for rate based events

that occur in ambulatory care. A patient who experienced a DRM in the ambulatory

setting may have participated in a number of microsystems which are geographically

separated, such as the primary care, specialist, pharmacy, and laboratory systems; or

personal events. Managed care organizations and health care coalitions, typically do not

have the same leverage and influence over these microsystems as, say, hospital

administrators who have access to the medical records and have the influence to require

participation from professionals involved in the care of a patient who experienced a

PDRM.

In summary, literature that evaluates the utility of administrative data for health

services research was presented. Different methodologies for electronically screening

health care data were also discussed. The history, purpose and methods of the Delphi

technique were presented. Finally, the process for cause-and-effect analysis was

discussed















CHAPTER 4
PRELIMINARY WORK

This section first addresses the prevalence of PDRM in the ambulatory care setting.

Following is an evaluation of drug treatment categories frequently involved in

preventable drug related admissions (PDRAs). Next, is a discussion of problem areas in

the MUS that were found to be proximate causes of PDRAs. This section will end with

an evaluation of latent causes of PDRAs.

Prevalence of Preventable Drug Related Admissions

The Harvard Medical Practice Study (HMPS) is historic because it was the first

large scale study to look at medical injury as a result of errors or problems in the process

of medical care.48 Prior to the HMPS, research on drug related injury focused mainly on

adverse drug reactions (ADRs), which are typically thought of as the inherent risk

associated with medications use and not problems with drug therapy management. Even

though the HMPS was published in the early 1990s, medication errors and PDRM did not

gain widespread appreciation and public awareness until the Institute of Medicine's

(IOM) Report, "To Err is Human," in 2000.2

The IOM report revisited the HMPS and a large study from Utah that focused

exclusively on inpatient medical errors and adverse events. The report gave much less

attention to ambulatory events. Nevertheless, they did suggest DRM causes a significant


a The term proximate cause is being used to describe the initial error or problem that started the events that
lead to a PDRM. The term is defined as: "That which in ordinary natural sequence produces a specific
result, no independent disturbing agencies intervening." Webster's Revised UnabridgedDictionary, C 1996, 1998
MICRA









number of admissions to inpatient facilities, and they reported the proportion of

preventable admissions is unknown. Winterstein et al. (2001) tried to address this issue

by producing a meta-analytic analysis of drug related hospital admissions (DRA).1 The

objective was to estimate the prevalence of PDRA and to explore the relationship

between study characteristics and prevalence estimates. Unfortunately, a generalizable

prevalence estimate was not produced because the studies did not have similar protocols

and there was too much heterogeneity in prevalence estimates to summarize the findings

with a mean value. Instead, a median and range were presented to display the

distribution of studies and meta-regression coefficients were calculated to evaluate the

association of various study characteristics with prevalence estimates. The methods and

results are presented below.

Since the goal was to produce a meta-analytic summary estimate of PDRA, a

criterion was established to select only studies that attempted to conduct a comprehensive

surveillance of drug therapy as a cause of preventable patient injury. Studies were

excluded that limited their scope to specific drug treatments, indications, injuries, or to

only one drug-therapy problem. The studies had to discuss the relationship between

pharmacotherapy and patient morbidity, and preventability, along with the necessary

information to calculate prevalence.

Fifteen studies5-12'49-55 published between 1980 and 1999 met the inclusion criteria.

All studies were conducted in industrialized countries: eight in Europe, four in the US,

two in Australia, and one in Canada. (Table 4.1) The median DRA prevalence was 7.1%

(IQR 5.7-16.2%) and the median PDRA prevalence was 4.3% (IQR 3.1-9.5%). Overall,

the median preventability rate was 59% (IQR 50-73%).









Since the protocols were not homogeneous across the studies and the Cochran's Q

test indicated extreme heterogeneity (Q: 176; df: 14; p<0.001), an analysis of the

association between different study characteristics and PDRA prevalence estimates was

warranted. The following characteristics have been discussed in the literature and were

considered potential contributors to the heterogeneity of the findings: (Table 4.2)

* Inclusion/ exclusion of first hospital admission (re-admission studies)
* Planned admissions, and transfers from other units or hospitals
* Mean sample age (>70 vs <70); country (US/other)
* Selection of hospital units vs. inclusion of entire hospitals
* Publication year (>1992 vs. < 1992)
* Inclusion/ exclusion of indirect drug-related morbidity (lack of drug effectiveness,
and no access)
* Inclusion/ exclusion of MD and patient interview
* Explicit criteria for judging preventability

The study characteristics were analyzed using meta-regression models, the

dependent variable was the transformed PDRA estimate and the independent variable

was a specific study characteristic for each model. These regressions were fit with

random-effects weights using restricted maximum likelihood estimation of the between

study variance to account for study heterogeneity. The meta-regression models produced

point estimates and 95% confidence intervals of prevalence odds ratios comparing studies

that differ with respect to each study characteristic (Table 4.2). The inclusion or

exclusion offirst admissions (i.e. re-admission studies) was the study characteristic most

strongly associated with PDRA prevalence (OR: 3.7; 95% CI: 1.5-8.9).

The mean of age of admissions was the next most strongly associated characteristic

with PDRA prevalence (OR: 2.0; 95% CI: 0.95-4.2). The inclusion of indirect DRM was

also associated with higher prevalence estimates (OR: 1.9; 95% CI: 0.92-3.9). The









remaining study characteristics listed in Table 4.2 had little or no apparent association

with PDRA prevalence.

The results of this systematic review suggest that PDRAs represent a significant

public health concern in ambulatory care (Table 4.1). In most studies, more than half of

DRAs were preventable. That is, they were not considered acceptable consequences of

therapeutic risk-benefit considerations but rather caused by inappropriate care and

medication errors. The data in Table 4.1 give the impression of a widespread and long-

standing problem in the quality of drug therapy management because the studies

represent problems from 1980 to 1999 and nine of the fifteen studies had PDRA

prevalence estimates above four percent.

To place the reported prevalence estimates in perspective, according to the National

Hospital Discharge Survey, there were 31.8 million admissions to U.S. hospitals in

1998.56 The top six primary diagnostic categories (heart disease, delivery, neoplasms,

pneumonia, psychosis, and cerebrovascular disease) each accounted for three to twelve

percent of these admissions. Any of the studies, or all of them combined, suggest that

inappropriate management of drug therapy may be a leading cause of hospital admissions

in developed countries.

The meta-regression analysis indicated that sampling methodology does affect

prevalence finding. Limiting the sample population to patients previously hospitalized

produced much higher PDRA prevalence estimates than not restricting the sample to re-

admissions. The causes of these findings are not clear. Patients re-admitted may be a

sicker sub-population who had been hospitalized with more risk factors for DRM, e.g.,

more drugs, more diseases, more prescribers, etc. A need for research specifically









designed to evaluate the transition of drug therapy management from the hospital to the

ambulatory setting is clearly warranted.

DRM is a concept, and the way in which it is operationalized affects its prevalence.

As one would expect, studies that included indirect injury (e.g., lack of therapeutic effect,

and lack of access to medications) were associated with higher prevalence findings. The

limited view of drug-relatedness found in one-third of the articles used in this analysis

may be an artifact of an outdated approach to evaluating adverse outcomes of drug

therapy. For an assessment of DRM to be considered comprehensive it would have to

include both direct and indirect drug-related injury. The IOM report suggested, "There is

evidence indicating that [adverse drug events] account for a sizeable number of

admissions to inpatient facilities." This systematic review confirms their suspicion and

suggests that PDRM in ambulatory care is at least as significant and prevalent as in

inpatient care environment.

Drug Categories Involved in PDRA

The same fifteen drug-related admission studies from Table 4.1 were used to assess

the medications commonly involved in DRAs. Three of the studies5'12'54 were not

included because they did not provide information on specific medications or therapeutic

classes involved in DRAs. Medications involved in DRAs were the unit of analysis

rather than medications involved in PDRAs because fewer than five studies provided

drug specific preventability.










Table 4.1: Studies Reporting DRA and PDRA: Sample Characteristics and Prevalence
Estimates

Reference Country Study year & DRA PDRA Preventability of
period Prevalence Prevalence DRA
period Prevalence Prevalence DRA


Darchy
1999
Ng
1999
Raschetti
1999
Cunningham
1997
Nelson
1996
Courtman
1995
Dartnell
1995
Hallas
1992
Lindley
1992
Nikolaus
1992
Bero
1991
Bigby
1987
Lakshmanan
1986
Trunet
1985
Trunet
1980


1994
France
12 months
1996
Australia
3 weeks
94/95
Italy
Italy 12 weeks

4 weeks each
UK
unit
1993
USA
1 month
92/93
Canada
5 months
1994
Australia
1 month

Denmark 1988/89


10 weeks


87/90
Germany 36 months
36 months


USA


Missing


83/84
USA
24 months
1984
USA
2 months
78/81
France
33 months
78/79
France
12 months


b Modified table from Winterstein et al. (2002)


41/623
-6.6%
31/172
(18%)
45/1833
(2.5%)
54/1011
(5.3%)
73/450
(16.2%)
21/150
(14%)
55/965
(5.7%)
8.0%
(n= 1999)
26/416
(6.3%)
22/87
(25.3%)
45/224
(21.1%)
73/686
(10.6%)
35/834
(4.2%)
97/1651
(5.9%)
23/325
(7.1%)


30/623
(4.8%)
10/172
(5.8%)
25/1833
(1.4%)
43/1011
(4.3%)
43/450
(9.5%)
18/150
(12%)
36/965
(3.7%)

3.8%

13/416
(3.1%)
32082
(12.6%)
34/224
(15.2%)
43/686
(6.3%)
19/834
(2.3%)
43/1651
(2.6%)
14/325
(4.3%)


30/41
(73%)
11963
(32%)
25/45
(55.6%)
43/54
(79.6%)
43/73
(58.9%)
18/21
(86%)
36/55
(65.5%)
67/143
(47%)
13/26
(50%)
38313
(50%)
34/45
(76%)
43/73
(58.9%)
19/35
(54%)
43/97
(44.3%)
14/23
(61%)










Table 4.2. Prevalence Estimates and Odds Ratio per Stratum/Specific Study Groups

Number of Average PDRA prevalence odds
Study characteristic Category Studies PRA1 ratios (95% C)2
Studies PDRA (%) ratios (95% CI)

Excluded 2 14
First hospital adm mission ...... ...........................
Included 13 4.2 3.7 (1.5- 8.9)
> 70 6 7.6
Mean age <70 7 3.9 2.0 (0.95 4.2)
Missing 2
Indirect drug-related Included 10 6.1
morbidity Excluded 5 3.3 1.9 (0.92 3.9)
USA 4 6.9
C o u n t r y ..................................................................................................................................................................................................................................................................................................................................................
Other 11 4.4 1.6 (0.73 3.6)
Transfers from other units Excluded 8 5.9
or hospitals Included 7 4.1 1.5 (0.71 3.0)
Excluded 6 6
Planned admissions.........
Included 9 5 1.3 (0.61 2.8)
Entire hospital 8 5.8
H hospital U nits...................
Selected units 7 4.5 1.1 (0.57- 2.3)
< 1992 8 5
Year of publication.........n
> 1992 7 5 1.0 (0.47-2.1)
Yes 8 5.8
M D / patient interview .........................................
No 7 4.2 1.4 (0.67- 2.9)
Specified criteria for Yes 6 5
preventability judgment* No 9 4.9 1.0 (0.47- 2.2)


The purpose was to find the treatment categories that were involved in DRAs and

to report their median frequency. This was done by classifying specific medications into

their therapeutic class and then organizing therapeutic classes into treatment categories.

Variability existed n the way medications involved in the DRAs were presented. Studies

either listed the specific medications involved in the DRAs, the number of events

involving a specific therapeutic class or only the number of events related to a treatment

category. Therapeutic classes were kept as long as they were mentioned in at least two

studies. If only one study mentioned a particular therapeutic class it was categorized into

a more general group, e.g., cardiovascular/other.









The cardiovascular treatment category had the highest median DRA prevalence

(33.3%) and this category was represented in twelve studies. The median prevalence for

the Anti-inflammatory category was 11.4% (represented in eleven studies). The median

prevalence of the Anti-diabetic category was 12.15% (represented in eight studies). The

median prevalence of the Psychotropic category was 8.75% (represented in eight studies).

The median prevalence of the Anti-infective category was 8% (represented in seven

studies). The median prevalence of the Non-specific miscellaneous category was 12%

(represented in ten studies).d See Figure 4.1 for bar charts of treatment categories and

their median prevalence that were mentioned in at least six studies. The actual data from

the studies can be found in Appendix A.

The therapeutic classes specifically addressed in at least six studies are presented in

Figure 4.2. The therapeutic class Diuretics was most often mentioned (10 studies).

Antihypertensives, Hypoglycemic, and NSAIDs were mentioned in eight studies.

Antibiotics were specifically mentioned in seven studies.

These findings are not surprising because they represent therapeutic categories for

common disease states. Cardiovascular disease is the leading cause of hospital

admissions56 and medications used to treat cardiovascular disease were involved in a





Prevalence was calculated by using the total number of medications involved in DRAs from a particular
treatment category as the numerator and total number of medications involved in DRAs for the
denominator. The prevalence for each treatment category was calculated individually for each study and
the median values was used to represent the prevalence of that treatment category.

d The Miscellaneous category (also described as the "other" category was often used in the articles to group
medications that were infrequently involved in DRAs. This category is not mutually exclusive from the
other treatment categories. For example; one study may have found that 15% of the medications involved
in DRAs were NSAIDs while another study only found 1% was from NSAIDs. The study that found only
1% if medications involved were NSAIDs probably would have put them in the "other" category.









large portion of DRAs. Admissions due to diabetes, infections, and psychosis are also

among the leading causes of hospital admissions.

From the results of this analysis it appears that medications most often prescribed

tend to be the treatments most often associated with DRAs. Cardiovascular,

psychotherapeutic anti-infective, analgesics, and anti-diabetic agents were among the top

ten therapeutic classes by retail sales share in 2001.e This means that DRAs appear to be

happening most frequently in patients being treated for common conditions with common

and well-accepted therapies.

From a measurement perspective this information can be very useful. It can be

used to focus the development of medication specific performance indicators and it can

also be used to help select indicators for use that have already been developed. DRAs in

the twelve studies used in this analysis were determined by medical record review,

however, which is currently the "gold standard" for measuring the prevalence of DRM.

Medical record review is a costly and inefficient process. The use of automated

performance indicators is an alternative for measuring the prevalence of DRM. This

information is useful for developing and selecting performance indicators. Any

instrument used to measure the prevalence of DRM in a population should try to address

the treatment categories found to represent a significant proportion of medications

involved in DRM i.e., cardiovascular, psychotropic, anti-inflammatory, analgesic, ant-

diabetic and anti-infective agents.


e Source NDC Pharmaceutical audit Suite. http://www.ndchealth.com/epharma/YIR/pharmatrends.htm









Apparent/Proximate Causes of PDRAs

As discussed in chapter two, DRM occurs when an error interacts with the latent

conditions of a system to produce injury. Errors are produced at the provider-system

interface. In the MUS, the "sharp-edges" where professionals interact with the patient

and with other health professionals happens at the prescribing, dispensing, (self)

administration, and monitoring nodes of the system.

The same fifteen articles listed in Table 4.1 were analyzed to determine the

proximate causes of the DTPs, i.e., DTPs were classified into the nodes of the MUS

where the errors likely occurred. A classification template for mapping DTPs to the

nodes of the MUS was developed to judge error etiology. When DTPs were not

classified into the nodes of the MUS the judgment of three reviewers were used to

classify them (Dr Charles D Hepler, Sooyeon Kwon, and Brian C Sauer). Classification

required 100% agreement, each reviewer judged the DTPs location in the MUS

independently. The reviewers discussed their individual decisions and when

discrepancies occurred they stated their opinions and worked through the differences

until agreement was obtained.

The authors focus, as well as any additional information mentioned in the article,

was considered when assigning the DTPs to nodes of the MUS. When an author in some

cases appeared to include both dose prescribed and dose administered in the same

"overdose" group, the case was classified into a category labeled non-identifiable process

problem. See Appendix B for a complete list of DTPs and information used to make

judgments for the placement of DTPs into nodes of the MUS.












Miscellaneous

Anti-diabetic

Anti-inflammatory

Psychotropic

Antibiotic

0 5 10 15 20 25 30 35
Figure 4.1: Involvement of Treatment Category to DRMs: Median prevalence for
Treatment Categories Mentioned in at Least 6 studies


Diuretics

Hypoglycemic

NSAIDs

Antihypertensive

Antibiotics


-I-I-I-I-


-- -

-- -

-- -


10 12


Figure 4.2: Therapeutic Class Involvement in DRM: The Number of Studies that
Specifically Mentioned Therapeutic Class









Since the purpose was to identify process related problemsf that led to PDRA, non-

preventable ADRs were not included in the analysis. ADRs are a special type of DTPs;

they typically represent an outcome and not a process of drug therapy. They have a

gradient of severity; the less severe ADRs would be considered an actual DTP, e.g.,

diarrhea. The severe ADRs would be considered DRMs, e.g., hospitalization from

dehydration due to frequent diarrhea. ADRs have historically been treated as the inherent

risk of medication use and not as a failure of drug-therapy management. Preventable

hospital admissions due to ADRs, nevertheless, may represent a process failure, because

somewhere in the MUS actions were not taken to detect and correct the ADR from

developing into a serious injury. Preventable ADRs were classified into the nodes of the

MUS according to the authors' description. If no description of the process failure was

presented then, as default, preventable ADRs were assigned to the monitoring node.

To be included, studies had to specify the type of DTPs, or the node in the MUS

where the DTP was initiated, that led to PDRAs. Eight studies linked DTPs to

preventability,5-12 (Appendix B) the other seven presented DTPs in relation to DRM, but

did not partition DTPs by preventable admissions.

The median values and inter-quartile range (IQR) for each node in the MUS were:

prescribing (33.93%; IQR: 5.73-54.86%), dispensing (0%), self-administration (22.29%;

IQR: 14.19-25.51%), monitoring (31.80%; IQR: 20.13-45.22), non-identifiable process

(5.93%; IQR: 0-16.39). (Appendix C and Figure 4.3)



fThe phrase, "problems with the process of care" is being used as an alternative to represent the same
concept as error. When discussing the etiology of DTPs terms related to process failures and process
problems will often be used since the term error has the ability to produce a defensive stance. When
examining the causes of DRM it is important to reduce tension and the defensive armory of professionals to
uncover factual events and true beliefs about the processes of care delivered.









The finding that prescribing and monitoring problems were major contributors to

PDRAs in these studies is troubling because it clearly indicates the MUS failed at the

professional/patient interface. Another important finding was problems in the monitoring

node were as prominent as problems in prescribing. This is interesting because

inappropriate prescribing tends to receive more attention than monitoring. Many

resources are being directed toward improving prescribing decisions, as seen through the

popular interest in computerized prescription order entry systems. From these findings

one could argue that interventions towards improving systematic monitoring and follow-

up of drug therapy should receive at least as much attention as prescribing.

(Self) administration or patient non-compliance was also found to be a substantial

contributor to PDRAs. Patients and caregivers are participants in microsystems of care

and they have a critical role in the MUS. (Self) administration is a patient behavior and

its overlap with medical error and professional responsibility can be debated. Non-

compliance is a result of intended and unintended actions. It has been shown that patients

make poor decisions because of rule and knowledge based mistakes.57 Health

professionals should be able to help patients correct their misunderstandings. They can

also help patients find creative ways to prevent unintentional non-compliance by

reducing slips and lapses. Nevertheless, patients typically choose to initiate the processes

involved in the MUS and in doing so they incur some of the responsibility for their health

outcomes. From a systems perspective, finding methods to reduce the occurrence of

PDRM is not limited to directing interventions at health care professionals and provider

specific processes. Patient directed interventions, and reconfigurations directed toward

patient behavior would be included in the systems umbrella.










Median


34%, 16% Prescribing
% Administration
iMonitoring
ONon-identifiable process
24%

Figure 4.3: Nodes of the MUS Involved in PDRA

Prescribing, monitoring and patient (self) administration appear to be the nodes

where most DTPs were generated. Because prescribing, monitoring and (self)

administration nodes have different characteristics it is possible error provoking

conditions upstream and faulty corrective mechanisms downstream (i.e., latent failures)

have unique system design flaws. Any "rational intervention" directed at a specific node

in the MUS would need to address the system conditions that increase the chance of an

error being created and progressing undetected.

Latent Causes of PDRA

Latent causes are flaws in the design and organization of systems that allowed

errors to occur, go undetected and result in patient injury. None of the fifteen studies

provided a comprehensive assessment of latent causes. Nevertheless, eight studies5-8'10-12

did mention, in the discussion, possible latent causes. (Table 4.3)

Lack of adequate knowledge was mentioned most frequently as an underlying

cause for errors in the prescribing node of the MUS (seven studies). Three of these

studies mentioned patient knowledge was a probable reason for problems in the (self)

administration node. Education was considered the best intervention for both prescriber

and patient knowledge deficits.

Three studies mentioned lack of communication and coordination among

professionals, especially the physician and pharmacist, was an underlying cause of









PDRAs. Two studies implied the lack of pharmacists' involvement in the MUS was a

factor and one study mentioned inadequate monitoring procedures were an underlying

cause.

Unlike many inpatient DRM studies, none of the fifteen PDRA studies

systematically evaluated the latent causes or empirically tried to locate system failures or

determine their relationship to higher order or competing systems. Without an adequate

evaluation of these influences and interactions, the development of meaningful

interventions will likely be difficult. Instead, corrective approaches will probably result

in generic interventions, those that the organization know how to do, for example send a

letter to the physicians who seem to be producing a portion of the problems.

For meaningful improvements to occur, the development of "rational interventions"

are necessary. The term rational is being used to represent interventions that are directed

at specific underlying system related causes. These are typically design and organization

issues. If lack of prescribing knowledge was determined to be a main cause of

prescribing errors, then a rational intervention may include finding better ways to provide

therapeutic information to prescribers at the time therapeutic decisions are being made.

From this analysis it is clear that a formal cause-and-effect analysis is needed to uncover

conditions of the MUS that make it prone to problems within the medications use system.

To summarize, injury as a result of drug therapy is a serious problem in ambulatory care.

A review of fifteen studies showed the median PDRA prevalence was approximately four

percent. Many common treatment categories were involved in these DRA.

Cardiovascular and anti-inflammatory agents were involved in many DRAs. Errors in

the MUS are the apparent causes of PDRM and problems in the prescribing, monitoring










and (self) administration nodes occurred frequently. None of the studies systematically

evaluated the latent conditions that provide the opportunity for proximate errors to occur

and go undetected. Lack of prescriber and patient knowledge was mentioned as latent

causes for prescribing and (self) administration problems. Nevertheless, a planned

systematic analysis is warranted. A better understanding of latent conditions could be

used to develop rational interventions. CEA is a promising method for uncovering latent

conditions.

Table 4.3. Latent Causes of PDRAs
Prescriber's Patient's Monitor Involve Inter-professional
Education Education pharmacist Communication
Darchy, 1999 /
Raschetti,1998 /
Cunningham, 1997 /
Nelson, 1996 / / / /
Dartnell, 1996 / / /
Courtman, 1995 / / /
Bero, 1991 / /
Trunet, 1980 /














CHAPTER 5
METHODS

This study was executed in three steps. In the first step, database analysis, MU-PIs

were used to estimate the number of PDRM positives in the study database. In step two

the degree of association between selected MU-PIs and specific nodes of the MUS where

the process failure may have originated was evaluated. In step three, selected MU-PIs

from each node of the MUS were submitted to cause-and-effect analysis (CEA).

Specific Aims

1. To establish the frequency of MU-PI positives in the study population.

2. To evaluate population based explanations for MU-PI positives.

3. To identify the node of the medications use system (MUS) where the pattern of
care from the MU-PI originated.

4. To identify system related causes that are common among nodes of the MUS

5. To identify system related causes that are unique to nodes of the MUS

Step One: Database Analysis

Claims Data Types

The MU-PIs were used to screen the study database. Each MU-PI was translated

into the administrative billing codes that represent the process and outcome components

of the indicators.13 The billing codes included the International Classification of Disease

9th edition (ICD-9) and Current Procedural Terminology (CPT) and National Drug

Codes (NDC).

The population included patients of all ages who were enrolled in a preferred

provider organization (PPO) health plan that managed employees from a member of the









health care coalition. The primary data files include Professional Claims (Center for

Medicare Services (CMS) 1500 form), Facility Claims (Universal Billing (UB) 92 form)

and Pharmacy Claims. The CMS 1500 and UB 92 forms are provided at the following

web address: http://www.cms.hhs.gov/providers/edi/edi5.asp#Form%20CMS-1500

Descriptive Analysis to Evaluate the Integrity of Claims Data

The number of office visit, emergency department (ED) visit, hospital admission

and pharmacy claims were compared month by month. If any month showed a large drop

in claims, this would suggest large numbers of claims might be missing from the

database. Office visit and ED visit claims were identified in the professional claims

database with specific CPTs.a Primary diagnosis, NDC, procedure codes, and service

dates were evaluated to identify missing or invalid codes. Links between databases were

established to ensure members could be traced across claim types.

Population Demographic

Average length of time enrolled in the health plan, average number of office visits,

age and gender frequencies were calculated to describe the study population. Frequency

of age by category, gender, number of office visits, and number of different drug classes,b

different pharmacies, different conditions, and different prescribers were used in logistic

regression analysis to better understand the MU-PI findings from information available in

the database.





a See Appendix E for a list of the codes used to identify office visits and ED visits
b The Universal System of Classification (USC) was used to identity different drug classes and routes of
administration. Unique USC codes identified drug classes (e.g., H2 blockers vs. GI proton pump) as well
as different routes of administration (e.g., sumatriptan oral vs. sumatriptan nasal). Drug class was
identified by unique USC codes.









MU-PI Coding Concepts and Analysis

Faris' 2001 translation of the PDRM scenarios into medical event codes was used

to construct the MU-PIs.13 In Faris' study, two medical record coders had independently

selected all possible ICD-9 and CPT codes that represented the PDRM scenarios. The

codes identified by the medical record coders had also been reviewed by a physician for

clinical judgment about whether each code was consistent with the PDRM scenario.

Please see Faris 2001 for more detail on the coding methodology.13 A list of the 40 MU-

PIsC used in this study can be found in Appendix D.

Search algorithms

Each MU-PI had the following format, an outcome of care and a specific pattern of

prior care. For example, ED visit or hospital admission for Hemorrhagic event AND use

of warfarin AND prothrombin time/INR not done every month. The pattern of care is the

process component of the indicator and it represents a potential DTP. An ED visit or

hospitalization was required for the outcome of care. An indicator positive (possible

PDRM) required both the process failure and the outcome component of the indicator.

Process failures fell into three categories: disease-drug interaction, drug

monitoring, and drug-drug interactions. Each required different types of search

algorithms. SAS version 8.2 was used to code and run the indicators in the database.

The type of search algorithm used for each indicator is presented in Appendix F.

Disease-drug interaction

The disease-drug algorithm required the presence of specific diagnosis codes and

specific pharmacy claims prior to a claim for the associated outcome. As illustrated in


Only 40 of Faris' 49 indicators were run. Search algorithms were to complex for nine and they were not
used for this analysis.









Figure 5.1, time moves from right to left by the blue arrows. The diagnosis claim had to

be present before the use of the medication (pharmacy claim). The green arrow

represents the lag, i.e., the duration of time from the outcome date to the last drug date

prior to the outcome. The cut off for the disease-drug interaction lag was set at 100 days,

in order to accommodate 90-day mail-order pharmacy refill cycles and to include patients

who may have been taking the medication in proximity to the outcome. An indicator

positive required the outcome, the disease code(s) and a pharmacy claim for the drugs)

within 100 days prior to the outcome.


Prior Diagnosis Last Drug Date Out e
Date Prior to Outcome

Lag



Figure 5.1. Disease-Drug Interaction Search Algorithm

Drug monitoring

Drug monitoring algorithms required the presence of specific pharmacy claims and

CPT claims, which indicate whether specific laboratory analyses were conducted. Two

coding solutions were required for monitoring indicators: one for the process component

and another for the process and outcome simultaneously. As illustrated in Figure 5.2,

time moves from right to left by the blue arrows. Once specific drug claims initiate the

process, lags are calculated to represent the interval of time from drug to first CPT claim

(lag 1), CPT claim to CPT claim (lag 2), and last drug claim date to last CPT claim date

(lag 3). Patients were considered at risk and included in the analysis when they appeared

to be taking the medication for at least as long as the defined lag time. Monitoring

intervals (i.e., lag times) vary according to MU-PIs. If any lag exceeded the defined









monitoring interval the member screened positive for the process component of the MU-

PI.

LAG 2a LAG 2b

Fs Du DCPT CPT ./ nth CPT
First Drug Date -- < tPT /a \ date 2 / date Last Drug D


LAG 1 LAG 3

Figure 5.2. Process Search Algorithm for Indicators that Require Monitoring

PDRM positives were identified when members had specified pharmacy claims and

the lag between the outcome and the last CPT claim prior to the outcome exceeded the

defined monitoring interval. If the lag was less than four days, the lag between the

outcome date and second to last CPT date was used to avoid false negatives. The

assumption is that monitoring within four days of hospitalization identified the

impending injury and need for emergency or hospital care. (Figure 5.3)

Drug Date <, Last CPT > OutcomeDate



LAG


Figure 5.3. Process and Outcome Search Algorithm for Indicators that Require
Monitoring

The determined monitoring interval for the warfarin indicator is 30 days. The

example in Figure 5.4 would register as a process positive, but not a PDRM positive

because the pattern of care was stabilized before the outcome occurred. The outcome

may indeed be drug related, but it is not represented by the MU-PI indicators. The lag

had to exceed the defined monitoring interval from outcome date to last CPT claim for a

PDRM positive.










50 days 25 days


INR INR nth INR
First Warfarin INOutcome Date
Claim W Claim 1 Claim 2 Claim OutcomeDate
date date date


40 days 25 days
Figure 5.4. Example of a Process Positive but not a PDRM Positive

Drug-drug interaction

The third search algorithm considered multiple drugs. This included drug-drug

interactions, drug-no drug situations and overuse of one drug and under use of the other.

The process component is indicator specific. If the MU-PI is a simple drug-drug

interaction with Drug A being a chronic medication and Drug B acute process positives

are recorded when a claim for Drug B occurred between the first and last claims for Drug

A. (Figure 5.5) PDRM positives are recorded when pharmacy claims for Drugs A and B

occur within the specified number of days for each lag.

Drug A Drug B Drug A
Diagnosis Date First Drug Date Last Date Last Drug Date



ALAG

Outcome Date
LAG 2
Figure 5.5. Drug-Drug Interaction Search Algorithm

Prevalence Estimates for MU-PI Positives

Specific Aim 1. To establish the frequency of MU-PI positives in the study
population.

Research Question 1. What is the period prevalence for process positives in the
study population?
Research Question 2. What is the period prevalence for the PDRM positives in
the study population?









This is a cross-sectional (period prevalence) analysis. The population included

patients of all ages who were enrolled in a PPO health plan from January 1st 1999 to

September 11th 2001. The analysis was limited to one process positive per indicator;

however, no limits were placed on PDRM positives.

Demographic Explanations for Prevalence Findings

Specific Aim 2. To evaluate demographic based explanations for MU-PI
positives.

Information from the database was used to better understand indicator positives.

Descriptive analyses of monitoring intervals were conducted to determine if cases on the

margin of the monitoring interval were being identified as process positives, and

multivariate logistic regression was conducted to explore the relationship between PDRM

positives and demographic/system related variables.

Distribution of Monitoring Intervals

Research Question 3. Do many process positives appear to be captured from the
"margin" of the defined monitoring intervals for indicators
that require drug therapy monitoring?

Due to the nature of the MU-PIs, the quality of drug-therapy monitoring can be

grouped in one of two categories. A case can either screen positive or negative for the

inappropriate monitoring interval. When a case screens positive it means an individual

did not have claims for the defined laboratory tests within the specified monitoring

interval. One limitation of this approach is the strict cut-offs-those on the margins are

categorized as process failures even if they were only a few days or weeks over the

required time. To develop a more comprehensive picture of how cases deviated from the

defined monitoring intervals, an analysis of the first monitoring interval (i.e., the lag time

from the first pharmacy claim for the drug of interest to the first claim for laboratory tests









of interest) was presented for indicators grouped by the indicator defined monitoring

interval. This was a descriptive analysis. The intervals were determined by dividing the

monitoring requirement in half. For example, if required monitoring for the indicators

was one-month, the categories for descriptive analyses were based on fifteen day

intervals. Analysis was only carried out for indicators with defined monitoring intervals

of one month or greater.

Variables Associated with PDRM Positives

Research Question 4. What demographic variables are associated with MU-PI
findings?

Available information from the database was used to explain the prevalence

findings. Bivariate and multivariate logistic regression analyses were conducted.

Demographic variables age, gender, and number of drug class, pharmacies and

prescribers were used as independent variables, while PDRM positives were the

dependent variables.

Multicollinearity was evaluated with tolerance and variance inflation statistics.

Tolerance is 1-R2 for the regression of a given independent variable on all other

independents, ignoring the dependent. The higher the intercorrelation of the

independents, the more the tolerance will approach zero. If tolerance is less than 0.20 a

problem with multicollinearity is indicated. After multicollinearity was assessed,

stepwise variable selection was used to build the regression model.

The likelihood ratio test was used to test the overall significance of the model.

Max-rescaled R2 was used to assess the predictive power of the model. The significance

of the variables in the model was assessed by the Wald Chi-Squared test and confidence

intervals (CIs). Hosmer-Lemeshow statistics were calculated to determine the goodness-









of-fit of the final multivariate model. Odds ratios (ORs) and Wald's CIs were calculated

directly from the estimated regression coefficients and their standard errors. All P-values

were set at 0.05. All analysis were performed using SAS Version 8.2

Step Two: Node Identification of Select MU-PIs

Specific Aim 3. To MUS where the pattern of care from PDRM scenarios
originated.

The ultimate goal of this dissertation was to identify system factors (i.e., latent

conditions of the health care system) that appear to be common or unique to nodes of the

MUS. This requires an exploration of the association between selected indicators and

specific nodes of the MUS. To do this, selected MU-PIs were sent to a Delphi panel of

clinicians and researchers for their judgment on the node of the MUS in which the

process component of the indicator likely originated.

Pilot Testing

The node identification panel survey was pilot tested with graduate students and

faculty in Pharmacy Health Care Administration at the University of Florida. All five

participants were pharmacists, three had received their pharmacy degrees in the United

States, one had received her degree in South Korea and the other had received his degree

in Germany.

The purpose of the pilot test was to evaluate the understandability of the survey and

the node identification scoring method. Organization of content, directions, and scoring

system were modified based on comments from the first round of the pilot test. No

significant problems or need for changes were identified in the second round of pilot

testing.









Selection of MU-PIs for Node Identification Study

MU-PIs used in the pilot were not the same as those used in the Delphi study.

Indicators were chosen based on preliminary results from the MU-PI study. MU-PIs

were primarily chosen based on indicator frequency.d

Delphi Recruitment

The snowball sampling or chain referral sampling was used to identify participants

for the Node Identification Delphi study. This method has been widely used in

qualitative research. It yields a study sample of people with specific characteristics, e.g.,

expertise. People with desired characteristics mention others who may also be likely

candidates for study participation.58

The first wave of recruitment letters was mailed with stamped return envelopes to

researchers who have published extensively in the field of medication error research or

who were referred by faculty members in Pharmacy Health Care Administration at the

University of Florida. The letter specified the goal of the study, asked the contacts to

participate in the Delphi panel, and invited them to recommend others who are trusted

experts in medication error or adverse drug events researcher I telephoned the first round

contacts between one and two weeks after the recruitment letter was mailed to answer

questions the contacts may have had and to try to convince them to participate.f

Sample size and power for the statistical tests were considered prior to the

recruitment process. When setting alpha at 0.05 and beta at 0.8, it was determined that a

d Scenarios were selected based on preliminary indicator findings, changes were made to the MU-PIs and
not all scenarios selected for the Delphi study had PDRM indicator positives.

e The term adverse drug event was used for the Delphi study instead of drug related morbidity because it is
the standard term in the medical community and I wanted to avoid confusion among the panel members.

f Please see Appendix G for a copy of the recruitment letter.









sample size of thirty-two was needed to identify a node as statistically different from the

others when its mean value was at least a six and the standard deviation was less than or

equal to three.

Node Identification Survey

Surveys were mailed to members who agreed to participate. The survey provided

detailed instructions that included definitions for the nodes of the MUS for the

assignment task. Specific demographic information was requested and it included name,

occupation (researcher and/or licensed health professional), whether they were practicing

or not, level of education, age, and gender, belief about the significance of PDRM

The panel members were provided fourteen MU-PIs and were asked to allocate the

number of points from 1-10 into the nodes of the MUS where they believe the error or

drug therapy problem originated for each of the fourteen indicators. They were asked to

answer based on a population perspective, meaning they were asked to identify the node

that explained the origin of the majority of cases that might fit the indicator definition.

Panel members were also asked to document their decision logic and provide comments

when they deemed necessary.g

Delphi Process

Results (panelists' scores) for each MU-PI were analyzed after the first round. If

an indicator had received an average score for one node that was significantly higher than

its average score for all other nodes, it was "assigned" to that node and deleted from the

second round of the Delphi study. In the second round, panel members received

information about each remaining indicator, including summary distributions of


g See Appendix H for a copy of the Node Identification Survey.









responses, median and interquartile ranges for each node, the particular Delphi member's

response to each item, and a summary of the decision logic statements for the panel

(grouped into like statements). Delphi participates were asked to reconsider their

opinions and to state their decision logic regardless of whether they changed their belief.

Node Identification Analysis

Research Question 5. What nodes of the MUS did the selected PDRM scenarios
likely originate from?

Hypothesis: Ho: Prescribing = Dispensing = Administration = Monitoring
Ha: At least one node is significantly greater than all other nodes.

The Kruskall Wallis test (KW), an analysis of variance test for non-parametric data,

was used to determine if differences in rank order means existed among the nodes of the

MUS. Dwass, Steel, Critchlow-Fligner (DSC) multiple comparison post hoc tests were

used to determine which nodes were significantly different (P values=0.05). StatsDirect

Version 2.3.7 software was used for the KW and DSC tests.

Step Three. MU-PI System Level Evaluation

Specific Aim 5. To identify common cause sequences among nodes of the MUS.

Specific Aim 6. To identify unique cause sequences for specific nodes of the MUS.

Selection of MU-PIs for Evaluation

The fourteen MU-PIs evaluated in step two were found to be problems that

originated in the prescribing or monitoring nodes. Two MU-PIs were selected from the

prescribing and monitoring nodes. Selection was limited to MU-PIs that were evaluated

in the Node Identification Study and was based on frequency of MU-PI positives.h



h Indicator number 28 was evaluated in the Delphi study and had the second highest frequency for the
prescribing indicators, however it was not selected because the indicator did not appear to differentiate
those with congestive hearth failure (CHF) and heart block from those with CHF only.









Step Three: Cause-and-Effect Analysis

MU-PI Evaluation Team

Members of the MU-PI evaluation team were identified and invited to participate

by the CEO of the health care coalition. They consisted of a Medical Director, PharmD,

nurse practitioner, consumer advocate, laboratory manager and a director of a health

maintenance organization. The session was scheduled for four hours. Members of the

evaluation team were paid $100 per hour.

One week prior to the meeting the evaluation team was mailed a briefing that

contained the goals for the study and background on the MU-PI results, cause-and-effect

analysis and the levels of the health care system. It also provided the MU-PIs that would

be evaluated and asked them to begin generating ideas. A copy of the brief is available in

Appendix I.

MU-PI Evaluation Process

The evaluation process began with a brief introduction and slideshow presentation

to orient them to the study purpose and database findings. The two MU-PIs formally

evaluated were the asthma "prescribing" indicator and the warfarin "monitoring"

indicator, indicators 39 and 30, respectively. (Appendix D)

The following six steps were carried out for each MU-PI that was formally

evaluated: brainstorm causes, clarify and organize causes, prioritize causes, tree-

diagramming, and establish causes that are common to multiple nodes of the MUS and

causes that are unique to specific nodes of the MUS.

The moderator provided an introduction and background of the study objectives

and indicator prevalence findings. Indicator specific finding were presented for the









warfarin "monitoring" indicator and the asthma "prescribing" indicator. The question to

guide the brainstorming process was presented for each indicator.

The evaluation team brainstormed possible system related problems that may have

contributed to the MU-PI indicators. The round robin process was used to elicit causes

from each participant. Proposed causes were projected on a screen and documented in

Path Maker Version 5.5 software.

The evaluation team then clarified and organized the proposed system causes into

an affinity table. The affinity table was organized by the levels of the health care system

(patient, professional, organization, environmental levels). Proposed causes were

organized into the level of the system the problem was associated with. A category for

measurement error called "artifact" was also permitted. Proposed causes were organized

into affinity tables with Path Maker Version 5.5 software

The evaluation team then prioritized the proposed system cause and the top five

causes were selected for tree-diagramming. The facilitator went through the list of

proposed causes from the affinity table and the evaluation team voted by raising their

hands for each of the five causes they believed contributed the most to the prevalence of

the indicator findings. Scores were tallied for each cause and the five receiving the most

votes were selected for tree diagramming.

The evaluation team then used tree diagramming to identify relationships among

the different levels of the health care, as they relate to the proposed cause. Tree diagram

consisted of branches that had each level of the health care system (patient, professional,

organization, environmental levels). One at a time, the top five proposed causes were

placed into the system levels associated with the cause. For example; if the proposed









cause was lack of MDs providing patient education it would go into the professional

(microsystem) level. The follow-up question would have been; what at the

organizational level may influence MDs ability to provide patient information? The

proposed organizational influence would be placed in the organization node of the same

branch of the tree. (Figure 5.6) Excel Version XP was used to create the tree diagrams

and document the findings.

Proposed Cause
No Medication Management System What at the organizational
Guidelines unclear level may influence MDs
ability to provide patient

Branch

SPatient level Professional level Organization / Environment

1 MDs don't provide patient

1 H H H





Figure 5.6. Organization of Proposed Causes to System Levels

The evaluation team then established commonality and uniqueness within and

between nodes of the MUS. An indicator assigned to the monitoring node was tree

diagrammed. The evaluation team was given a questionnaire to determine if each cause

sequence (i.e., each branch of the tree) would apply to another indicator assigned to the

monitoring node. They also had to judge whether each cause sequences would apply to

an indicator assigned to the prescribing node. The same procedure was applied to the

prescribing indicator that was formally evaluated, i.e., tree diagrammed. (Appendix K)






70


Analysis for Common and Unique Cause Sequences

Research Question 6. What cause sequences were common within the Prescribing
Node?
Research Question 7. What cause sequences were common within the Monitoring
Node?
Research Question 8. What cause sequences were common to the Prescribing and
Monitoring Nodes?

Agreement was determined when five out of six (83%) or four out of five (80%) of

judges agreed.














CHAPTER 6
RESULTS

Step One. Database Analysis

Descriptive Analysis to Evaluate the Integrity of the Claims Database

To evaluate the completeness of the database the numbers of claims were compared

by month. The dates of office visit claims ranged from January, 1999 to September,

2001. As illustrated in Figure 6.1, many office visit claims appear to be missing on the

tail ends of the data collection period, from January, 1999 to June, 1999 and from July,

2001 to September, 2001. Emergency Department (ED) visit claim dates ranged from

January, 1999 to June, 2001. As illustrated in figure 6.2, many ED visit claims appear to

be missing from January, 1999 until June, 1999. Facility admission claim dates ranged

from January, 1999 to June, 2001. No claims appeared to be missing during the study

period (Figure 6.3). Pharmacy claim dates ranged from January, 1999 to June, 2001. As

illustrated in figure 6.4, pharmacy claims appeared to be missing from January, 1999 to

June, 1999 and in September, 1999. Missing pharmacy claims from September, 1999 are

not followed by an increase in claims. It appears claims during this month were missing

from the database.

The decision was made to only include claims from July, 1999 to June, 2001 for

analysis. This decision was made because the number of claims was consistent during

this time frame. Even though pharmacy claims appeared to be missing in September,

1999 the decision to include the previous months was made because the benefit of

keeping July, August, and September in the database outweighs censoring the time.







72



Check for missing and invalid data

Macro level analysis found no missing or invalid codes for primary diagnosis,


procedures or service dates. Missing or uninterpretable drug codes were found in 530 of


the 572,010 pharmacy claims (571,480 usable pharmacy claims).



12000

10000 -

8000

6000

4000

2000
6000 0 ---- T 9-------------------------------


O^ O^ O^ O O^ O ^ ^ N JN N N N




Figure 6.1. Office Visits Claims by Month




14000

12000

10000

8000

6000

4000

2000

o n .. .





Figure 6.2. ED Visit Claims by Month

Links between data types

To ensure that members were identifiable across claim types links were established


between datasets to identify the number of members that had multiple types of claims.











4,227 of the 4,589 members who had a facility admission claim also had a professional

claim. 28,478 of the 33,891 persons with a pharmacy claim also had a professional claim



350

300

250

200

150

100

50

0





Figure 6.3. Facility Admissions by Month


35000
3000ooo0

25000oo
Q Nooo I N
2moo






60 0 ,0 ...... .





Figure 6.4. Pharmacy Claims by Month

Population Demographics

There were 47,053 members who contributed 58,719 member years during the


sample period of July, 1999 to June, 1999. The mean length of enrollment was 467.47

days with a standard deviation of(SD) 225.53 days. The 37,063 members with a medical









claim (pharmacy or professional) contributed a total of 51,892 member-years; mean

length of enrollment was 511.04 days with a SD of 201.73 days.

Age and gender distributions for the different claim types are presented in Table

6.1. The mean age ranged from thirty-four to thirty-seven across the databases. The

majority of members with a claim were female. Females ranged from sixty-five percent

to seventy-two percent across the claim types. The fifteen to forty-four year olds were

the most frequent age group, their frequencies ranged across the data types from thirty-

seven to forty-eight percent. The sixty-five and older group accounted for a small

fraction of the claims, their frequencies ranged from about two to eight percent.

Table 6.1. Population Age and Gender Frequencies by Type of Claim
Professional Facility D v Pharmacy
ED visit
Claim Admission Claim


mean age(SD) 33.97 (18.23) 34.48 (22.12) 34.71 (18.54) 37.03 (18.34)

By Category
frequency, [percent]
age <15 6,622 [19.07] 1,084 [18.51] 1,263 [17.61] 4,956 [14.3]
15<=age<45 16,519 [47.58] 2,143 [36.6] 3,399 [47.39] 16,461 [47.48]
45<=age<65 10,743 [30.94] 2,144 [36.62] 2,270 [31.65] 11,421 [32.94]
65<=age 835 [2.41] 484 [8.27] 240 [3.35] 1830 [5.28]

Female 22,861 [65.85] 3243 [71.84] 4,659 [64.96] 23,499 [67.78]
Male 11,858 [34.15] 1271 [28.16] 2,513 [35.04] 11,169 [32.22]
0 0 0 0


Table 6.2 displays additional population characteristics. There were 220,967 office

visit claims during the twenty-three month period with an average of 6.64 visits, which

were approximately three visits per member per year. Pharmacy claims were submitted

for 33,891 members. The average number of pharmacy claims per member during the

twenty-three month time period was 16.05 with a SD of 22.9, which were approximately









eight pharmacy claims per year. Members went to an average of 1.55 different

pharmacies and an average of 2.13 different prescribers during the study period.

Members averaged 5.8 conditions and used an average 4.7 prescriptions from different

drug classes.

Prevalence of MU-PI Positives

There were a total of 103 PDRM positives from eighty members. The prevalence

of PDRM positives in members with a medical claim from July, 1999 to June, 2001, was

1.98 (1.60 to 2.37)a per 1,000 member years. There were a total of 10,889 process

positives in 5,741 members. The prevalence of process positives from July, 1999 to June,

2001 was 209.84 (206.37-213.34) per 1,000 member years. Table 6.3 shows that, 0.22%

of the population with a medical claim screened positive for both the process and

outcome component of the MU-PIs, i.e., PDRM positives. There were forty-three PDRM

positives in thirty-eight members where the outcome was identified by an ED visit and

forty-one PDRM positives in twenty-six members where the outcome was identified by

hospital admissions (HA). Nineteen PDRM positives in eighteen members involved a

hospital admission from the ED. PDRM positives accounted for 0.75% (95% CI: 0.74%-

0.76%) of hospital admissions, and 0.56% (95% CI: 0.55%-0.57%) of ED visits. Table

6.4 provides a summary of the number of PDRM positives by number of indicators.

Twenty-two of the forty MU-PIs had no PDRM positives. Of the remaining eighteen

indicators, the range of PDRM positives per indicator was from zero to twenty-six. The

eleven indicators with the most frequent PDRM positives are listed in Table 6.5.


a 95% Confidence interval











Table 6.2. Population Demographics
Characteristics Frequency Percent


No of Enrollees
No of Members with claim
Length of time enrolled
mean (SD)
No. ED visits
No. Hospital Admissions
No. of outpatient physician visits
mean (SD)
No. of outpatient physician visits (members)


9-10
>10
No. of Conditions (diseases)
mean (SD)
LE 5
6-10
11-15
16-20
GE 20
No. of members with pharmacy claim
average per member (SD)
No. of drug classes
average per member (SD)


No of different pharmacies
average per member (SD)
1


6 or more
No of different prescribers
average per member (SD)
1


7 or more


47,053
37,063


511.04 (201.73)
10,048
5,855
220,967
6.64 (7.05)
33,271
10,281
6,781
4,480
3,167
2,295
6,267
34,938
5.8 (4.8)
20,714
9,161
3,382
1,030
651
33,891
13.93(22.61)
543,959
4.7 (5.31)
25,135
7,305
2,809
978
836


1.55 (1.45)
17,981
8,238
3,919
1,989
1,033
731

2.13(2.07)
12,482
8,281
5,302
3,206
1,989
1,117
1,514


All analysis are for the 23 month time window


30.9
20.38
13.47
9.52
6.9
18.84


59.29
26.22
9.68
2.95
1.86





67.82
19.71
7.58
2.64
2.26


53.06
24.31
11.56
5.87
3.05
2.16


36.83
24.43
15.64
9.46
5.87
3.3
4.47









Table 6.3. Frequency of Process and PDRM Positives
No. of Members with Population % No. of Positives (unit:
Positives (unit: person) (denominator=37,063) each event)
Process 5,741 15.49% 10,889
PDRM_ED 38 0.10% 43
PDRM_HA 26 0.07% 41
PDRMEDHA 18 0.05% 19
Total PDRM 80 0.22% 103

Table 6.4. Number of PDRM Positives by Number of Indicators
Freq. of PDRM Positive No. of Indicators Percent
0 22 55.0
1 3 7.5
2 4 10.0
3 3 7.5
5 3 7.5
9 2 5.0
12 2 5.0
26 1 2.5

Table 6.6 lists the indicators with the ten most prevalent process positives. Thirty-

eight of the forty MU-PIs screened positive for the process component of the indicator.

The range of process positives was from zero to 2,282. Approximately fifty-four percent

of the indicators had less than one hundred process positives, and ninety-two percent of

the indicators had less than 1,000 process positives.

System Based Explanations for Prevalence Findings

Information from the database was used to better understand indicator positives.

Descriptive analyses of monitoring intervals were conducted to determine if cases on the

margin of the monitoring interval were being identified as process positives, and

multivariate logistic regression was conducted to explore the relationship between PDRM

positives and demographic/system related variables.










Distribution of monitoring intervals


Table 6.5. Eleven Most Prevalent PDRM Positives
Ind Mnemonic Hx Drug_A Drug_B Risk Process PDRM
39 Asthma< 28 CHF< 37 CHF< 1 Dep< 30 Hemr< 19 Hypoth< 29 ARF< 31 MI 7 Dep< 14 ActRespFaik 16 CHF<
Table 6.6. Ten Most Prevalent Process Positives
Ind Mnemonic Hx Drug_A Drug_B Risk Process PDRM
29 ARF< 23 HyprK< 16 CHF< 19 Hypoth< 17 HypoK< 37 CHF< 1 Dep< 5 G< 30 Hemr< 27 G< Legend for Tables 6.5 and 6.6
* See Appendix D to link indicator numbers to the indicator scenarios.
* The 'Mnemonic' is an abbreviated version of the MU-PI scenario
* Hx = the number of members who had an ICD-9 code for the particular history of disease or diagnosis
of interest
* Drug A and Drug B = the number of members who had a prescription claim for the drugs of interest.
* Risk = the number of members who met the criteria for analysis. For example; if the process component
includes a drug and six month monitoring interval, then the members at risk would be those who were
using the medication of interest for at least six months.
* Process = the number of members who had the pattern of care represented in the MU-PI
* PDRM = the number of members who had both the process component and the outcome component of
the MU-PI within the specified time frame.
* Fourteen indicators were sent to a Delphi process for node identification, they were numbered
consecutively from one to fifteen. This column provides a cross reference to the MU-PI number

Due to the nature of the MU-PIs, the quality of drug-therapy monitoring can be

grouped in one of two categories. A case can either screen positive or negative for the









inappropriate monitoring interval. When a case screens positive it means an individual

did not have a laboratory test claim within the specified time frame. One limitation of

this approach is the strict cut-offs-those on the margins would be categorized as process

failures even if they were only a few days or weeks over the required time. To develop a

more comprehensive picture of how cases deviated from the specified monitoring

intervals, an analysis of the first monitoring interval (i.e., the lag time from the first

pharmacy claim for the drug of interest to the first laboratory test claim of interest) is

presented for indicators grouped by the required monitoring interval. Analysis was only

carried out for indicators with defined monitoring intervals of one month or greater.

Table 6.7 provides the indicator numbers for each monitoring interval. For

example: Only indicator 30 had a one month monitoring requirement, while indicators 9

and 25 had a three month monitoring requirement.

Table 6.7. MU-PI Categorized by Required Monitoring Interval
one month two months three months six months twelve months
30 12 9 2 19
35 25 8
11
18
23
24
26
29
33

Indicator 30 (Hemm warfarin INR (1)) was the only indicator with a required

monitoring interval of one month. Figure 6.5 shows forty-one percent of the members

who were taking warfarin had a laboratory test claim for an International Normalization

Ratio (INR) within a month from their first warfarin pharmacy claim. Only about 3.8%

had laboratory test claim on the "margin" within forty-five days of their first warfarin










claim. The majority, 50.4 percent did not have a laboratory test claim for an INR within

120 days of their first claim for warfarin.











100
80
60

40 3.79% 1.75%
20 0.87 0.87% 0.87%

days<30 30 lag time

Figure 6.5. First Lag for Indicator with One Month Monitoring Requirement

Indicators 35 and 12 had a required monitoring interval of two months. Figure 6.6

shows 0.56% of members at risk for these indicators had the specified laboratory test

claims (blood pressure and cell blood counts) within two months of their first pharmacy

claim, while 99.4% did not have the specified laboratory test claim within 240 days from

their first pharmacy claim. Indicator 35 accounted for all but one process positives.

Indicator 35 required blood pressure monitoring it is likely that blood pressure

monitoring is not billed separately and the process positives are false positives. This

indicator was removed from all other analysis (including the previous prevalence

findings) because the CPT codes used to identify blood pressure monitoring were not

sensitive to blood pressure monitoring.

Indicators 9 and 25 had a required monitoring interval of three months. Figure 6.7

shows 4.35% of members at risk for these indicators had the specified laboratory test










claims (lithium levels and BUN) within three months of their first pharmacy claim for the

defined medication, while 84.78% did not have the laboratory test claim within 360 days

from their first pharmacy claim. No lags were identified on the margin of the required

interval.

Indicators 2, 8, 11, 23, 24, 26, 29, 33 had a required monitoring interval of six

months. Figure 6.8 shows 24.78%of members at risk for these indicators had the

specified laboratory test claims within six months of their first pharmacy claim for the

defined medication. Some process positives were captured from the margin, 6.53% had

the laboratory test claim between 180 and 270 days.



99.44%


1.30 '/o


days <=60


0 0% 0% 0%


90

150 lag tim


Figure 6.6. First Lag for Indicator with Two Month Monitoring Requirement


































days<=90


4.35% 2.1


0% 2.17% 0% 2.17%


135

225 lag time '


Figure 6.7. First Lag for Indicator with Three Month Monitoring Requirement


Figure 6.8. First Lag for Indicator with Six Month Monitoring Requirement


ArO%/


















2954


1 2 Q


days <=365 365 lag time

Figure 6.9. First Lag for Indicator with Twelve Month Monitoring Requirement

The majority of members who met the requirements to be considered for the MU-

PIs with six month monitoring requirements (57.97%) did not have the specified

laboratory test claim within 720 days from their first pharmacy claim.

Indicator 19 had a required monitoring interval of twelve months. Figure 6.9

shows the majority of the members who appeared to be using thyroid therapies (52.25%)

had their thyroid tests (laboratory test claims) within twelve months of their first

pharmacy claim for the defined medication. About thirteen percent had the specified

laboratory test claim between 365 and 545 days, about six percent had a claim between

545 and 730 days from the pharmacy claim, and about thirty percent did not have the

laboratory test claim within 730 days of the first pharmacy claim for the required

medication.

In summary, the majority of process positives did not result from the margins of the

required lag interval. The majority of process positives resulted from the "extreme" of









the distribution. The indicators that required one month, six month and twelve month

monitoring intervals were picking up some process positives from the margins, 3.78%

6.53%, 12.58%, respectively.

Variables associated with PDRM positives

Personal variables associated with PDRM positives were identified with bivariate

logistic regression. Table 6.8 displays the OR and CI for each dependent variable. Age,

and number of office visits, prescribers, pharmacies, drug classes and conditions were

significantly associated with PDRM positives. Gender was not significant (odds ratio

OR: 1.00; confidence interval CI: 0.63 1.59). Number of pharmacies and number of

prescribers were the strongest bivariate predictors (OR: 1.48; 95% CI: 1.43-1.63) and

(OR: 1.48; 95% CI: 1.40 -1.57), respectively.

Number of drug class and number of conditions had the next strongest association

(OR: 1.19; 95% CI: 1.16-1.21) and (OR: 1.20; 95% CI: 1.18- 1.23), respectively. Number

of office visits and age had significant but weak bivariate associations (OR: 1.07; 95%

CI: 1.06-1.09) and (OR: 1.06; 95% CI: 1.05-1.08), respectively.

Multivariate multicollinearity was assessed using the tolerance score. As shown in

table 6.9, no tolerance values were below 0.20. The lowest tolerance score was 0.256,

meaning 0.256 of the variance in number of office visits was unexplained by the other

independent variables.

A multivariate logistic regression was conducted to explore the relationship

between PDRM positives and demographic/system related variables. Logistic regression

with PDRM positives as the dependent variable and the seven demographic/system

related variables as the independent variables used stepwise variable selection to produce

the model. A four-variable risk model was produced. The model indicated that males









Table 6.8. Bivariate Tests of Association: PDRM Positives as Dependent Variable
Independent Variable Odds Ratio 95% Wald CI
Age 1.06 1.05 1.08
Gender F vs. M 1.00 0.63 1.59
number of Office visits 1.07 1.06 1.09
number of Prescribers 1.48 1.40 1.57
number of Pharmacies 1.48 1.34 1.63
number of Drug class 1.19 1.16 1.21
number of Conditions 1.20 1.18 1.23

were about 50% more likely to have a PDRM positive after adjusting for the other

variables in the equation (OR: 0.45; 95% CI: 0.28-0.74). Number of office visits had a

negative influence on the odds of having a PDRM positive (OR: 0.93; CI: 0.9-0.96).

That is, each additional office visit reduced the probability of having a PDRM positive by

approximately seven percent, after adjusting for the other variables in the model.

Number of drug classes and number of conditions increased the odds of having a PDRM

positive. When drug class was held constant at zero, each additional medical condition

increased the odds of being PDRM positive by about twenty-nine percent. When medical

conditions were held constant at zero, each additional drug class increased the odds of

having a PDRM positive by about twenty-two percent. Number of drug classes and

number of medical conditions had a significant interaction with a negative beta-

coefficient (B coefficient: -0.004; p-value: 0.0014). Increases in both medical conditions

and drug classes resulted in decreased odds of having a PDRM positive.

The overall significance of the equation by the likelihood ratio had a chi-square of

306.007 (p< 0.0001) with four degrees of freedom (df). The proportion of variance

explained by the model (max-rescaled R2: 0.279). Overall the model fit well to the data

(Hosmer-Lemeshow: 4.01; p=0.779; df=7).










Table 6.9. Multicollinearity Assessment of Independent Variables
Parameter Standard Variance
Variable DF me t value Pr > Itl Tolerance ia
Estimate Error inflation
Intercept 1 -0.04072 0.00394 -10.35 <.0001 0
Age 1 -0.0000551 0.00007498 -0.73 0.4624 0.84942 1.17728
number of Conditions 1 0.00724 0.00020712 34.96 <.0001 0.26857 3.72342
number of Drug Classes 1 0.0066 0.00015911 41.46 <.0001 0.31993 3.12572
number of Office Visit 1 -0.00421 0.00014943 -28.19 <.0001 0.25561 3.9122
number of Pharmacy 1 -0.00345 0.00055755 -6.18 <.0001 0.78954 1.26657
number of Prescribers 1 -0.01089 0.00045741 -23.8 <.0001 0.41653 2.40079

Table 6.10. Maximum likelihood Estimates: PDRM Positives as Dependent Variable
Standard
Parameter DF Estimate ard Wald chi-Sq Pr > ChiSq
Error
Intercept 1 -9.229 0.404 521.676 <.0001
Gender, F 1 -0.398 0.126 10.014 0.0016
number of Conditions 1 0.254 0.031 66.933 <.0001
number of Drug Classes 1 0.200 0.024 69.664 <.0001
number of Office Visits 1 -0.074 0.017 19.389 <.0001
Cond*Drug class 1 -0.004 0.001 10.156 0.0014

Step Two. Assignment of MU-PIs to Nodes of the MUS

The ultimate goal of this dissertation was to identify system factors (i.e., latent

conditions of the health care system) that appear to be common or unique to nodes of the

MUS. To do this, the node of the MUS where the MU-PIs' process problem originated

was determined. Selected MU-PIs were sent to a Delphi panel of clinicians and

researchers for their judgment on where in MUS the process component of the indicator

likely originated.

Delphi Recruitment

A total of fifty-seven recruitment letters were mailed out. Twenty-nine contacts

agreed to participate, seven declined, and twenty-one did not respond. Twenty-nine

contacts were e-mailed or mailed the survey (based on individual request) and eighteen

completed and returned the first round. The Delphi panelists consisted of five medical

doctors (MDs), one nurse, five doctors of pharmacy (PharmDs), six bachelors of










pharmacy (BA pharm), and one masters in health service research (MHSA). Fifteen of

the seventeen members with clinical degrees were licensed in the US and eight were

currently practicing. All of the panelists with clinical degrees had additional graduate

level degrees. (Table 6.11).

Table 6.11. Demographics of Delphi Panelists for Node Identification Study
Other Licensed in
Clinical Degress Dther Gender censed inPracticing
Degree the US
2 MPH
MD 5 1MS
1PhD
Yes =17 Nurse 1 1MPA
.............. .......... .......6 F e m a le 1
PharmD 5 1 PhD 12 Male
1 MBA PhD
BA Pharm 6
4 PhD
No = 1 1 MHSA
Total Number of Delphi Panel = 18

Assignment of MU-PIs to Nodes of the MUS

The Node Identification study went through two rounds of the Delphi process. An

indicator was assigned to a node and not returned for the second round when the average

score for a node was significantly greater than the average scores for all other nodes.

Eleven indicators reached significance within the first round. Three of the indicators

were returned for the second round and two achieved significance. Based on the Delphi

panelist responses, problems with the specificity of the indicators were the reason they

did not reach significance in the first round. Changes in the indicators were made and

they were sent back to the Delphi panel. The panelists were informed of the changes

made to the indicator scenarios. They were also provided with their score, the groups'

average score and a summary of the panelists responses for the three indicators. The

changes made to the three indicators included,









* Changing "sympathomimetic decongestants" to "prescription sympathomimetic
decongestants" for Delphi indicator 9.

* Removing "INR not done before therapy starts" from Delphi indicator 12 (the
monitoring requirement was one-month)

* Changing "antibiotic use" to use of"macrolide or sulfonamide antibiotics" for
Delphi indicator 14

The indicators were judged to be primarily associated with either the prescribing or

monitoring node of the MUS. None of the indicators were found to be primarily

associated with the dispensing or administration nodes of the MUS. Six indicators were

assigned to the prescribing node, seven to the monitoring node and one did not reach

significance within the two rounds. Table 6.12 provides a summary of the indicator

assignments.b The MU-PI column in Table 6.12 is the MU-PI number and it provides a

cross reference to the Delphi indicator number. Table 6.13 lists the statistics for each

indicator analysis. Mean scores for each node, Kruskal Wallis statistics and p-values are

presented for each indicator.

Significance of the Dwass, Steel, Critchlow-Flinger pairwise comparisons are

represented by an underscore under the abbreviated node. Nodes connected with an

underscore were not statistically different. Delphi indicators 1-6, 8, 10, 11,13, and 15

reached significance in the first round, all p-values <0.001 for KW statistic and pairwise

comparisons. Indicators 9, and 12 reached significance in the second round, p-values

<0.001 for KW and pairwise comparisons. Indicator 14 did not reach significance and

therefore it was not assigned to a node. Box plots for the mean, median and mode for

each indicator are provided in Appendix L.

b Delphi indicator number seven was not included because it was a repeat of Delphi indicator
number four.