<%BANNER%>

Effect of Academic Detailing on COX-2 Utilization Rates

xml version 1.0 encoding UTF-8
REPORT xmlns http:www.fcla.edudlsmddaitss xmlns:xsi http:www.w3.org2001XMLSchema-instance xsi:schemaLocation http:www.fcla.edudlsmddaitssdaitssReport.xsd
INGEST IEID E20101123_AAAAAF INGEST_TIME 2010-11-23T05:36:28Z PACKAGE UFE0011901_00001
AGREEMENT_INFO ACCOUNT UF PROJECT UFDC
FILES
FILE SIZE 31461 DFID F20101123_AAAEIH ORIGIN DEPOSITOR PATH graham_s_Page_131.QC.jpg GLOBAL false PRESERVATION BIT MESSAGE_DIGEST ALGORITHM MD5
d335e26efabf72611b0690d7bf677ef4
SHA-1
29e8adeefd99e7e2d199a4bb8b8f3cf9d0efebbf
7989 F20101123_AAAEHS graham_s_Page_122thm.jpg
d30a2d9cba6780e790854e91d0d8279e
6cbe39a935ffcb3142ad60a527f018a2461a7a5d
1956 F20101123_AAACYW graham_s_Page_025.txt
f150c6be8e6505802f9e72e1ed4bd78c
c4b8f42b9f2ad11852ea4dbd3b18e0f1334a8e56
59637 F20101123_AAADFF graham_s_Page_008.jpg
5346fda35f4d875ea7144fa7f2d8c10a
1cdc1be426ba5203d4f530ec68721140a851b7c5
82412 F20101123_AAADEQ graham_s_Page_014.jp2
51e7d60747aa611ee8d59eb1b855e906
6825603cedb102401655a4eca96805b978b41db7
1988 F20101123_AAACZK graham_s_Page_101.txt
ef46266485b8fbe681ad0d03d8d504f3
8d6333e644c581ab68652c0bd85c9f3cc7e29fdc
8320 F20101123_AAAEII graham_s_Page_132thm.jpg
9a6666bb75d915b0faeb0afba8519f1f
a4d45b49fe6ff063134e58d7cdb3394ad6c0036d
7485 F20101123_AAAEHT graham_s_Page_123thm.jpg
0fe85517d386ea26c2fd98e30e7d1f87
85e7e196bb7da06cd65f711b4ff3fb774a89fc65
101082 F20101123_AAACYX graham_s_Page_133.jp2
e425e67c87cc97370d60f0ba6fc9dfac
f83d1cb2bff348421e4bf038d00e2376f1e4e6ee
101830 F20101123_AAADFG graham_s_Page_009.jpg
207628700dda269325003bfc7f632cbd
d7a27f85f68572f3da96bafc2aed467f20aae56e
108019 F20101123_AAADER graham_s_Page_033.jp2
c77bda71c4169e62a0ba673842680428
9114f805329c6a97a3d0bbcf2c67d9032924b924
1068 F20101123_AAACZL graham_s_Page_013.txt
1981288d6e75b9d94beea9b42bc7edd6
d13314e0f4d785597d4dbf0cee48cab59fb33a25
30640 F20101123_AAAEHU graham_s_Page_123.QC.jpg
7e8ee3f0dd6a66faeb94adb198a1674d
81d4d0717d327848a76e39d292c2add5cf8b05ae
1936 F20101123_AAACYY graham_s_Page_110.txt
a496e66519f2a70a1489d23dc5a0c4f1
f692ef09d96afa88a664efa5bfd34349ccc18d1f
88484 F20101123_AAADES graham_s_Page_098.jp2
10117b2a1b616cb306f92b752ce09095
1ab2fe2de9481a18d6cb82e68f557284a2310b7e
8363 F20101123_AAACZM graham_s_Page_036.QC.jpg
4c65c69b08c31da86d264417567e823a
1061d6630709c79e61b57ada06417b7df1814f7f
32459 F20101123_AAAEIJ graham_s_Page_132.QC.jpg
d32c5cfaf113595975f3876e9035564c
4e4ad72e690ceb69058dfba7b88372b54c8fbb9c
8304 F20101123_AAAEHV graham_s_Page_124thm.jpg
e394cbbfacf0d5490f86e2c663a1122f
4b9255d81a450c0f32727a19f418b98082736c87
1904 F20101123_AAACYZ graham_s_Page_087.txt
7800877f68db02ba43df00f1e354dce1
b1c0dab91498078642b267a2c67d299c00d0b61c
131600 F20101123_AAADFH graham_s_Page_010.jpg
cbe24214d0d53c0f7f76c5df00172ff2
4e23dcf7556ae66b79ca9570fd98e5c8cb0cc906
88108 F20101123_AAADET graham_s_Page_091.jpg
11a125b5dc44fb143600d475ff11250b
56f31f877bcf48c2fc3a38233663f2b0734a5c03
78390 F20101123_AAACZN graham_s_Page_014.jpg
3462e0d7a198c8895cd6aca3fed29f4a
5c7279af086bd9b0fd7541a95e6ffe3353cead1a
7728 F20101123_AAAEIK graham_s_Page_133thm.jpg
ffd98d8dce1690c5a36e2eb7e6b40aa6
5398295f887b4eae2778f31918a0713435794aa4
34200 F20101123_AAAEHW graham_s_Page_124.QC.jpg
aa65bee50cd2b67bb95da520f1db9b43
302ce5fd6cbd1ea6b498848023a0f022b63fe57d
67733 F20101123_AAADFI graham_s_Page_015.jpg
ec12ef7dc13d39f942692b4dbe15410a
94f10bf713a17141bbb89dc87f6fe68a82dd1fa0
26313 F20101123_AAADEU graham_s_Page_078.QC.jpg
f2c4c63a06caedbc4e8d1b9349740cac
e0b41a66cff533e34932b472c2ad549ef8646180
52078 F20101123_AAACZO graham_s_Page_127.pro
bc9c1a82a758dd787ef372c60a8513ec
c57ee8d6bc08daaba1e7da1898bed752bea57198
18567 F20101123_AAAEJA graham_s_Page_142.QC.jpg
056881990b8f29ea9eb64d947b581aad
c87995711263597542dfd4a3f48b8344ff0cef33
7722 F20101123_AAAEIL graham_s_Page_134thm.jpg
06108ed20c4b298bfea95744e27c25cd
08c0440f23394a6aa956a7075120a99da5af5acf
8364 F20101123_AAAEHX graham_s_Page_125thm.jpg
9a5b5944027bd89dd2e93b2157964c98
edef71e6f3e5cfa7ca12c349bf3f621df8ba6c5d
89714 F20101123_AAADFJ graham_s_Page_016.jpg
9a693d40da6620167e4c63285dedea53
4cbec7d908e1d97d3cdbc9475521f2e201e801e3
163989 F20101123_AAADEV UFE0011901_00001.mets FULL
215b99c212747e93f8eefe95c8b70600
1684351587df4ad4e10f3a5f0edb6897ac03ddf9
81960 F20101123_AAACZP graham_s_Page_070.jpg
de2aa7e8b404b91710c909d98a0145bf
56f3079eaa553cf738acea30988f84842582de7c
30329 F20101123_AAAEIM graham_s_Page_134.QC.jpg
77e50edcf960643d0d2a88469b26c970
d6dd7a6920ca168351832aefeab6f31a31e8c9b1
34530 F20101123_AAAEHY graham_s_Page_125.QC.jpg
87374f29b64140a94910880324099cba
f47704fc7268db39e1fc285c8f3dd72c0ae5d794
108230 F20101123_AAADFK graham_s_Page_018.jpg
f6c7d159d9ad060064bd7b8bdde7846d
92ec451b87fe88794bd96b6ed2c3a26aa4191869
47467 F20101123_AAACZQ graham_s_Page_122.pro
f40685ce9b71366e6219b5f061826c9f
d84da4aeb0d401c461274580bf484efeff3b0387
8600 F20101123_AAAEIN graham_s_Page_135thm.jpg
46dd6859f4383061958efb19640f22ab
6aaa440df3f12200ba4149a97e36ef97b3095df2
7898 F20101123_AAAEHZ graham_s_Page_126thm.jpg
bdc7b800d420a953ba090b12d8166a37
71677c64fb45a4808a94fa382581ce8e86a3d6f2
88683 F20101123_AAADFL graham_s_Page_022.jpg
68c9af06d43e1925b1754326ce3041fc
c77ecb506db0e82c7ec4b50ed28d17c2a2ac27cb
100200 F20101123_AAACZR graham_s_Page_019.jp2
ddb136fba86974c06498e8c8987721ec
dc7112c3f79dc1a23711f3bcf48e7906d32fe0ec
96847 F20101123_AAADGA graham_s_Page_039.jpg
29acf62804e8a91166c3fc32f3f56d5c
421a6da59b9aada80f664b50644a2eb5c915df4c
34534 F20101123_AAAEIO graham_s_Page_135.QC.jpg
217abcb76140e4da280c91e617f6aea5
e8eb63750a761ad2557f88b0af69791c638ff23a
58013 F20101123_AAADFM graham_s_Page_023.jpg
a96076fcbb930fb272d426ca1cf6446c
b5f882c7ca28e0782316db973fc0c550a179e0c0
24150 F20101123_AAADEY graham_s_Page_001.jpg
b988837bc2a3efae5f587771e8fe787f
12f6fdd16e4e32452f4dd338c979fcce20045143
43320 F20101123_AAACZS graham_s_Page_060.pro
d3ee1d0c2426ac5a58e0192076eb5e97
06057386b3f678ee8a25993d6bf793f01f33fba2
101560 F20101123_AAADGB graham_s_Page_040.jpg
8839072b6021c407c245b2ac36e83462
e7070eaf961474bda47fac7720f40e2d77521dd0
906 F20101123_AAAEIP graham_s_Page_136thm.jpg
5b1337fc326f31a78ffafceb9ffaea0e
c924e9a1dae28a0f5f1aa58ea78d89dbc10043c5
89176 F20101123_AAADFN graham_s_Page_024.jpg
bee5c2df9633fb6f799ac644a8b41a10
fe968bcc48fc7e3daab4f168d9332ca802d1578b
4860 F20101123_AAADEZ graham_s_Page_002.jpg
849e0f340bb52f54fa6e7bffb21217f1
867db42b6bceccdde7083f44e3b18e4c9f6d495b
29617 F20101123_AAACZT graham_s_Page_075.QC.jpg
38a990ba323bd4f4a9b8b43a4a0dbfa0
b0f03cbb88e110dfb9d6920a92b6d25e17055ad6
89343 F20101123_AAADGC graham_s_Page_042.jpg
5932ca7aef2b1f8e8d20d9992cca9309
c6402085abad15b4cda2d998128c678e557f2a1e
3121 F20101123_AAAEIQ graham_s_Page_136.QC.jpg
fb5c1e97ff7cadfa7356ffac00f23ecd
51113d56603c419d257c995e875d404decad9355
102910 F20101123_AAADFO graham_s_Page_026.jpg
4ae7672515523029d0d6de7305954067
58113c7c88f6b9c54161766a1ead3c3bc63eebe1
50086 F20101123_AAACZU graham_s_Page_120.pro
743b6ce2c35669cc00c34e6c7ba1129a
85cc0e8dcd4525a261ffa0f190e75c4d1df4f91f
95171 F20101123_AAADGD graham_s_Page_043.jpg
b7478a88558d31519978690a12e8a9ff
6c90d968fb516a24d133ce519fbd729bf4b11e1c
7776 F20101123_AAAEIR graham_s_Page_137thm.jpg
1fa128f44e7b7a59c79b28af9db6a94a
f6ed19a3d9659bd14536cd2a493f5e1ecaa537d4
108984 F20101123_AAADFP graham_s_Page_027.jpg
8cf391cad45fdc1c78b8ac97497c2c06
1cd0329c964231553141ab174514080461412f98
864483 F20101123_AAACZV graham_s_Page_086.jp2
87dc14fac243f314b98e0bb6b9a152e9
b0d8a31d988c2c749cf72c9213f0f89bfca61ead
90704 F20101123_AAADGE graham_s_Page_044.jpg
6d62004d3c7ef6cc32795715d6e6b7b1
9828b2c4ff5754ac781bc3a85a8d9a57623c1b88
8887 F20101123_AAAEIS graham_s_Page_138thm.jpg
687fd27dc1a0c111279add69f2cf2928
eaaecefeb2e73ab9cce109b11557623933b23c75
103769 F20101123_AAADFQ graham_s_Page_028.jpg
2cdc97e480ca97d36af9618089b04285
94240fc2031c6937ef47589f1ecb3bc53b0b2314
112559 F20101123_AAACZW graham_s_Page_135.jp2
0d6c539e014b69effdbf901ab0bdfda8
0e6c9e21d62af13fbce0c9f840d5e43ad4a1d5f3
90267 F20101123_AAADGF graham_s_Page_047.jpg
7fefab7c9718874f9eaea990963a9e2b
75d0893f0c962acd3c29a408796289a3ed3124d7
36854 F20101123_AAAEIT graham_s_Page_138.QC.jpg
efdbff9ab0a96e5c933294b13ef4917b
93418d4048b261aac6f4b9668b5e8cd22338c27f
103678 F20101123_AAADFR graham_s_Page_029.jpg
cf67af30790498329cc47c3d9af99d6a
fcfa7b792992177e2124f8a6925e2c9e969ad40a
94004 F20101123_AAACZX graham_s_Page_017.jpg
ac4d2e64dd74297ff4a3facdcd5af6f9
ea77370bdacc0649fc889befe039c5536f2061ae
97130 F20101123_AAADGG graham_s_Page_048.jpg
73bd5c2a71b33cf4fcd86ae4f8f3c286
1cca03f62b09448fe22f2c9344b88930da9e5be6
8948 F20101123_AAAEIU graham_s_Page_139thm.jpg
1b53047483acf7f1fb0f7f45a584cf86
97dd2bf69ccf987b0f9cf0e016fd81a5bda0b581
104065 F20101123_AAADFS graham_s_Page_030.jpg
ab0c10ceaf61faae5c75d86b5d1d7619
06039aa75280c845570cadd7c9cdff19ec9e9412
5192 F20101123_AAACZY graham_s_Page_005thm.jpg
20169e834d93ed8888f138b23d50a86e
ec1f2bbdbf3dcfa4cad791a01ef15b8c41261e43
101934 F20101123_AAADGH graham_s_Page_049.jpg
e23c026905bd4a5e068ba7eee3e766ea
ab684f303e251bac9b6365a8fa8b60e69d4ca3c7
38141 F20101123_AAAEIV graham_s_Page_139.QC.jpg
a9750fdda6244ecebe463b55c5c91dea
b3fb77f06cec92201f5549916826b0c4b23d39b7
108448 F20101123_AAADFT graham_s_Page_031.jpg
3a357d4839f674c5f93be4d2de4c3cdd
8e256ea42130d542a696e5c5c09750f97c2326a1
50422 F20101123_AAACZZ graham_s_Page_033.pro
cdeac935cd16241895fc49f2ce5c7745
8039d4581517e4dd47a59ea7adfb24b8b87a2ee1
8845 F20101123_AAAEIW graham_s_Page_140thm.jpg
08e4c059f4329413cfb2bdf16cb1221c
ca8fc49097382b9c05f3e69386273eeb7ba122a9
100493 F20101123_AAADFU graham_s_Page_032.jpg
f80ae44af007017e70bfd8f3d10408f1
d50b1c936c32e74c17168c43690496178743b0ad
103857 F20101123_AAADGI graham_s_Page_050.jpg
a79ce7091399cb9dd69136a9e5c0a9c5
56f723372ab1f04fd2c685e5d4d6503785a4ab72
36168 F20101123_AAAEIX graham_s_Page_140.QC.jpg
42b2332f3c81d83e09536b0a1b2610fc
e7e6ddc0f304cfe923190366cd6bf53ec63676b2
102270 F20101123_AAADFV graham_s_Page_033.jpg
edacb0475ddf459794ec03ce1d2692d3
0d2dbd580756d888738001c197d73ebe7fcaa741
22279 F20101123_AAADGJ graham_s_Page_051.jpg
4432f33ca6fca611ce15b9a95ccdd223
5b73d2b359e6a4c6f9398121d22b8cfd69779038
7510 F20101123_AAAEIY graham_s_Page_141.QC.jpg
aeed132483d9b2d3b2311646d08648a9
01a32198ab555fdc3f50411c1129b6c67e324e27
74705 F20101123_AAADFW graham_s_Page_034.jpg
751f4387f374bed40d31594fcd45ae41
8d9f3e29697592742698869e601bd20526ac2aeb
74884 F20101123_AAADGK graham_s_Page_052.jpg
d31855da38fdec9027193056f0fbc471
d8d33492dbed574e57ee4f7e1db77bbd506a6f33
4792 F20101123_AAAEIZ graham_s_Page_142thm.jpg
97f00bd0cebe6f693b4ba7d78dbedba1
3b8062f9148037cfc3a662e5022c6280dbbc1fd0
100170 F20101123_AAADFX graham_s_Page_035.jpg
cd1634a6d954a181693c924dcbbdf2d5
2059f8efabba161abcfcd491c8b9a251a5651ae7
86373 F20101123_AAADHA graham_s_Page_077.jpg
43b6be69a3df927ad803d019aec54810
2a743322bfe9729f49774da70c1bb6b686d16a81
87484 F20101123_AAADGL graham_s_Page_056.jpg
12243f975a62f383c17a593517b5770c
3e3a336597b14721b4e51c533958c1257bdb2ce0
25088 F20101123_AAADFY graham_s_Page_036.jpg
fb78c6a81d152b78c76a7dd45a5e3cf0
8607b61d387f696d5a503ecfd189680b3b1f6e3a
80955 F20101123_AAADHB graham_s_Page_078.jpg
9995f7f0a5295ac8ba4c8cff44e04867
71aa4ef17e2ee7c288e7ea9964499455fd3a6c11
83092 F20101123_AAADGM graham_s_Page_060.jpg
c1d385b4beaf0ac1444580b46b5cc9a8
16fc60dcac2832ee171307db2f7d520639c42899
104680 F20101123_AAADFZ graham_s_Page_038.jpg
5e004adec474c73e3bac927adac22c0e
db252d8b301cf95152678dfb666a0781d9fe7a44
90993 F20101123_AAADHC graham_s_Page_079.jpg
15047502c4bad931eebf9d59ddf1763f
9de992522b476d5b4f67a5f3e4008ce14e43e9ed
101881 F20101123_AAADGN graham_s_Page_061.jpg
985abe7294243978a8d0f3ad54f1b97f
344bdaecdc3a7a7060d8f5e858c39503c53d1d64
83149 F20101123_AAADHD graham_s_Page_080.jpg
92e404f502b1d0d18b6bddf1196da7eb
9dd63e5af9a73f3c65a94ca52bf77be9c01a4ed9
89831 F20101123_AAADGO graham_s_Page_062.jpg
d8bf782f061f10e470af049bead18c91
f55d8c33bc175476a91038229e907f2598940656
81203 F20101123_AAADHE graham_s_Page_082.jpg
3c66df22ec38de3b5c57648be1ae390d
076c738cacaa87114eeeef6a28f81f3e33d9c25c
87889 F20101123_AAADGP graham_s_Page_063.jpg
9d5ffcba1f0d6bea64015d2eedac2ab0
896c2f4a1a7f0b60d6a1bea5f9418064b04c545a
83708 F20101123_AAADHF graham_s_Page_085.jpg
6d9da96f9e3dc74a6f7310e824e4c941
dbc8a7d7d40cbf1440b81143c153be1d2044dd14
F20101123_AAADGQ graham_s_Page_065.jpg
39e940939f7371dc4959b62c2b89f5bb
e3f8f5b0bb72f001189b8030ec73c79de4fc9df4
83097 F20101123_AAADHG graham_s_Page_086.jpg
9319ee0c741f77e60bdbf4f2b135704b
aa65d21d80ec0257986e4873d4481de671e4dd1e
100152 F20101123_AAADGR graham_s_Page_066.jpg
c6c8ac71eaf0b439e049219a7dc66b10
4569075cb75c30f28957f0728897be9d7a96ae4b
100142 F20101123_AAADHH graham_s_Page_088.jpg
1a54a30271e1953356ca036075a7d938
ec3ec405d019c9a1b0f1c666dd640b1c641c4c09
87713 F20101123_AAADGS graham_s_Page_067.jpg
598afc3ab6413dd6236ef9df3a2afc97
eefb556506bf1dd45a2ab4d8cb8c1b3cf09f62c1
96526 F20101123_AAADHI graham_s_Page_089.jpg
42c4d0620a7081e4ebb1dc8ccb25c609
a7db19dd061b709d905235ba6928f58233f3d0c5
82948 F20101123_AAADGT graham_s_Page_068.jpg
672d732aab8ed341e1c23106eb545cfd
787bb0106348d290d8c2e94eb44c1e64d38eb6a5
76693 F20101123_AAADGU graham_s_Page_069.jpg
5f89a3d43d978d53faccf1334cd6fc1d
7bd3d3922611126fadccf631903280e10a5320b5
75058 F20101123_AAADHJ graham_s_Page_090.jpg
bad04281c58fd9e4f63b11886bfb58f8
b9a4c09aff9867d69e1e9cb6c30c435d92092920
86242 F20101123_AAADGV graham_s_Page_072.jpg
fe31c5ddeff493b7e80160b5f205359b
fe629602d7b4901bd049bdf707ef490b29d75b11
87664 F20101123_AAADHK graham_s_Page_093.jpg
88a2093d1b77e62e999c10eef3da8e31
a834686d2a5b5d03f6f7bad9bddcd5fdb576a20b
79044 F20101123_AAADGW graham_s_Page_073.jpg
25ace644f3ce036af6476c944aff9b8f
2e70a5dfaeadb9859999884710b783b573304931
81488 F20101123_AAADHL graham_s_Page_094.jpg
d8b0dd96e9d20d2fb4bdf95cb3583540
4862b982017a8e025dca1af25cc031617fd370cb
86332 F20101123_AAADGX graham_s_Page_074.jpg
860427dec998ba9244ebd1797629437c
acac393db0266b68073f4e25456963ee9d608f81
95684 F20101123_AAADIA graham_s_Page_112.jpg
92cd7214131ec7ab2493033186b814c9
534144af77c5fda66d8b64e6c03418ceaeb29b0d
82077 F20101123_AAADHM graham_s_Page_095.jpg
98dd325ea00985e6716326cd65e061aa
78b8c30428d509fd06a0c50495237ca1fecd5370
92872 F20101123_AAADGY graham_s_Page_075.jpg
feb1be258380acf631356a6111b8170e
7fec6ad1d22bf71afb7c4ed87dc29e58d487ecb7
97151 F20101123_AAADIB graham_s_Page_113.jpg
39baab4cbb098ea5ccbc61780b3fb383
895ba910fc09d0185e4dd93f508d99e9fbf9e5ac
93922 F20101123_AAADHN graham_s_Page_096.jpg
39472c2070b18cf3db38bf4a9c05b85f
5cd87fe106b1214e924178288ae981b61bdfc95f
83976 F20101123_AAADGZ graham_s_Page_076.jpg
61368b0c9bc59be455b43c6eac654cba
01a6cb6cb89fd82a2ebc1d1fc5d212e1fd400b20
45286 F20101123_AAADIC graham_s_Page_115.jpg
949b2512281bf20075f6b1418ddbe338
0601da4136b4a7ebd6e295beed90d08e435210eb
84605 F20101123_AAADHO graham_s_Page_098.jpg
f2bd6f2d01b21bf468e99acca287c917
9e3a2281fcb44dc0494e8267b055cd89e0187f89
94679 F20101123_AAADID graham_s_Page_116.jpg
3f639d76cc02536ecad79d9ed2ef8747
4f7f3ec9196c7682fdc55869a524ade654afa613
101239 F20101123_AAADHP graham_s_Page_099.jpg
b65f39659363d4e12adbb16fcbd98d75
5a23394d4ad1c655c6530e458d94c410b5975ce0
92085 F20101123_AAADIE graham_s_Page_117.jpg
9fc7460e593dceaba431009938a166cf
c9f284351569e916b5dd5e8ec671fda21fd587c5
85760 F20101123_AAADHQ graham_s_Page_100.jpg
731054294fece008a29129d9c081037c
4f84a84e7c8cd3d10ae7870ef8c65b67e8d86d06
100954 F20101123_AAADIF graham_s_Page_119.jpg
ba967736531a068a608844888beb115c
d4ce56b1b77d5ffc207e48375273af429e25c26e
100620 F20101123_AAADHR graham_s_Page_101.jpg
e29eb20e2a63a68af34affcc0dce0a3b
bbd296d5a1ff0d22fa6bc22d21da08426ac172e3
101075 F20101123_AAADIG graham_s_Page_120.jpg
f6e6dcf0bbf252efe2b85efc699fb806
5f451229701d97f2bc5d9a6da96f7d8c7bf77a31
101952 F20101123_AAADHS graham_s_Page_102.jpg
9640017e4acb982c69bc2df49d2a6156
1757acff3c69de8e0135c4b9d09d45d0989951f1
99750 F20101123_AAADIH graham_s_Page_121.jpg
0e9474433064cbd0f7a60692bffa5864
1b9c9ea1a389bced8ad56a5c7501de5c04035271
91065 F20101123_AAADHT graham_s_Page_103.jpg
4dbf4d883e9fc5a687d0313c9db6b4bb
1c98af4200af35ad35bf45d62b8bc63ec3863f0b
98334 F20101123_AAADII graham_s_Page_122.jpg
029942f2878fb161fadf65fda704d849
b087d67edf662c22978fdc865c5702916fe2fde2
92156 F20101123_AAADHU graham_s_Page_105.jpg
900a419234f8934a4b02bd7395109527
c9ae0b31035fdfff1ccc158f0a0b29f6ed3fb6e3
92238 F20101123_AAADIJ graham_s_Page_123.jpg
88e1bd00538208de42a570255a88f1f8
22fed7f279d248753a85944782d4dbad411f3951
95283 F20101123_AAADHV graham_s_Page_106.jpg
17770427018478d135f72b68711ecbcf
b6cd754804d58b8ca23434bcd0f3fcb118e004ca
92673 F20101123_AAADHW graham_s_Page_107.jpg
55ec7c2a1dfbd174c0401ee829ddecb0
390ea4c35e94ba8bd9a6d3f8c2e425992d141504
98219 F20101123_AAADIK graham_s_Page_126.jpg
5d01d3ebcfb2377ba2fb143b622d1032
347d0cefb8589b3d45449c14bae9114fbb96fc71
100672 F20101123_AAADHX graham_s_Page_108.jpg
b53389196768fbce65c506150cd9cd12
05fbd5dedbfeb7685960ea746aa1852c010d55ba
3732 F20101123_AAADJA graham_s_Page_003.jp2
266ec1d70ef344f1f96138ae3edbc413
8ba54b50ea7c80acc2ecf9b8e181fc4c72029d77
65919 F20101123_AAADIL graham_s_Page_128.jpg
58e7daccb57c5ffdd307fb94135b12fc
abec0f701578c7a908a12b942e4540d638f30942
90128 F20101123_AAADHY graham_s_Page_109.jpg
0142a19119b5fb4ed910543ee9934eff
d1d1ed8085c689215c14dd1ff75f1f6036df8d26
43077 F20101123_AAADJB graham_s_Page_004.jp2
d21908fad6a13c9ae4360c5780aad31a
b22dd3ec181bdaadf8918504cc19178584d6c417
75658 F20101123_AAADIM graham_s_Page_129.jpg
09a7704fcd89eb4d25d604848b57797b
1f982449eccbe77e32033b78f909df05dbeb55be
93677 F20101123_AAADHZ graham_s_Page_111.jpg
de9e4dfe4ed0d167f0a2831514ec6fc4
fbed21ed5f61064816b0e72f93433506fe45c8ef
1051847 F20101123_AAADJC graham_s_Page_008.jp2
3240e4eba2c7eb8f7390cb9f359b6938
9ae9437c87917f7fab44bc515b34bc78f928401f
95754 F20101123_AAADIN graham_s_Page_131.jpg
5796a4818bf40fe8bd7ac936d5fe440e
996236ff907829206db07d58abe1d7e5c31b7ec0
1051980 F20101123_AAADJD graham_s_Page_009.jp2
c93937a02e9fd8ce6ac82cbf5868e915
557fe399144180b7493f31dd81e47b696b6eb7c4
101669 F20101123_AAADIO graham_s_Page_132.jpg
81add03664a489c94e3a00c35376cef7
04f13ee8a6c851a764cc3d85b0d7d773518d80ce
1051942 F20101123_AAADJE graham_s_Page_010.jp2
94a62a2d5b9ca90da03735fc20a397de
0dbe3e4e4c994aa09e9ea9bbf6dcca3147b94fba
94108 F20101123_AAADIP graham_s_Page_133.jpg
62eb389138931e576e58a93b57e38bce
a8635a14f9acb703a619d4ccb15517436c6e3bb6
1051982 F20101123_AAADJF graham_s_Page_011.jp2
781942b21afb8c536deb72c2a5200e57
97047191643ef66bd3cd6002b28da925594f5732
93107 F20101123_AAADIQ graham_s_Page_134.jpg
d8a0bac16eaf72883c29963f6774caf1
ae5fe1e0a292d665b13cb61d50c990505fe82ef3
1051976 F20101123_AAADJG graham_s_Page_013.jp2
8fc61af25ac00f976326dc9b9cca4530
da3084af5d3bb2823fab3c6612b49d3ee0c0ae1b
106334 F20101123_AAADIR graham_s_Page_135.jpg
9e46518c9fc55d907b074807115176e3
6d8aae4f416d377d1b8738cbb94b1c805d3ccc24
94225 F20101123_AAADJH graham_s_Page_016.jp2
b374cefd4bd712bcf4e118d794eea1b3
2cae4a58b826e86130fad849a1acb9f8ef21fd22
8303 F20101123_AAADIS graham_s_Page_136.jpg
8f1cda4f36b98a399c78543c0abeae21
e972f71fc7ef4fe564cc536c7631c49a7769679d
98883 F20101123_AAADJI graham_s_Page_017.jp2
d38e2886719eec5362938bee04a47b15
681811c6677b42ad49099043bf2ad64330283daf
108625 F20101123_AAADIT graham_s_Page_137.jpg
8623317615f1fbcb0720441b32668ba2
2409d17bf4f9c48238101e02b8c51e10656bd248
114398 F20101123_AAADJJ graham_s_Page_018.jp2
13bced397c56106bdae5837521b7ab9e
2db751730dff9a4f007a7180a745055cb7e27e88
133969 F20101123_AAADIU graham_s_Page_138.jpg
c59d803fe2f258f35281ec57cdd4cad4
54cb1eb533b958648f31967c5e8604e48b0c8f67
83466 F20101123_AAADJK graham_s_Page_020.jp2
d1892ef39bd90c11d30f223a37bbbdf1
3f68536771a0648d18a2de2a51f3147a437c52ee
133552 F20101123_AAADIV graham_s_Page_139.jpg
a7e0f326bd00fadecfde6726a981c4ec
2eb6df4bd8dafb2c6a276668408d99681fe6cfd8
125884 F20101123_AAADIW graham_s_Page_140.jpg
e9153576e433f377a74bf9f35b2057a2
f3a65e76b023df4463530bce8b15943ce0441c4a
110876 F20101123_AAADKA graham_s_Page_040.jp2
e95026d7cb6dde8200efdddeca157a77
17924f9bcaae92f3c34df6ca8c23f7b7a86d28f5
93579 F20101123_AAADJL graham_s_Page_021.jp2
16dd0e1d8e1831ff9049862d4fd817fd
03ca3177fc67c347ecacb75724d0b2505c5d57f3
23230 F20101123_AAADIX graham_s_Page_141.jpg
d66a7cd3d76dfa08de93cc416eeebd81
72263138e68ba425243e800ad261e3c63a941c95
114018 F20101123_AAADKB graham_s_Page_041.jp2
a913f9cbcbeca868681c858c3662d870
4f7b8950266d7a59f0f173112f4c5107b698d275
93981 F20101123_AAADJM graham_s_Page_022.jp2
0b6e95c8782d914230d1a20d3995fb0d
5ba0099215f8e137bf761abf7887a215bb34dac6
23559 F20101123_AAADIY graham_s_Page_001.jp2
ea6c3d7136c6e804ef0891bb5fbf01ad
3296ef2f448fadb72e77c8e460c450656c192659
94240 F20101123_AAADKC graham_s_Page_042.jp2
380a8a5b7cb8c00785c11c4eca8a7f68
c14b49d709f0a513c5b6e0ce25fab3d120195f3b
61745 F20101123_AAADJN graham_s_Page_023.jp2
94b33b206fa958207ad3f9787910663d
9ebf6977353e3a03fa440d11964bd35737c35991
5860 F20101123_AAADIZ graham_s_Page_002.jp2
5f851ccadac4995673f715fbd329b7e9
931863a87dc3330e169be4c9da9c9093d3f8d696
1034815 F20101123_AAADKD graham_s_Page_043.jp2
631d7561913528a555a875400c8a86a3
3919433729b27414bc9971e1fd64e9b9e5014251
93076 F20101123_AAADJO graham_s_Page_024.jp2
6a9c660070eb1db08fad628538481770
6f14a59ab7b78ccb5d70c880bb1f606a5ca1a437
93749 F20101123_AAADKE graham_s_Page_044.jp2
30120cf13ddfdaff5470d0ba567dd500
e103ac5c3bd678830e4fa25eae9705d2c68221c6
108283 F20101123_AAADJP graham_s_Page_025.jp2
71790c5f8243cdac11871f81b814e4a3
a6a3727687556680346c2dc6c75cf9a8ea1ee250
103313 F20101123_AAADKF graham_s_Page_045.jp2
1542118a1ff0d67f1c324cac69dd3e47
b86c6f7bb11458221b02b661e85201633276063b
108162 F20101123_AAADJQ graham_s_Page_026.jp2
120a9eccb3a27dcef3a0770fe3ed8f5b
ba06606ee95773ebe59efa2587ee999a84d73940
108020 F20101123_AAADKG graham_s_Page_046.jp2
8788a32641e29850ee388c86f72e40f8
1bd80743bb6d4fa44974d20f9dbd9294f13287fa
111026 F20101123_AAADJR graham_s_Page_028.jp2
6f4f99085975405bd59779125f9a1f76
81ec23a20a6cc5384164aa5bf5a0b6a90c806a6f
101529 F20101123_AAADKH graham_s_Page_048.jp2
a220eae25d19fa87d5dc9ac9383538ec
afc6e61a32cf5c19b4ad60b44042ea7aca02fa1c
111030 F20101123_AAADJS graham_s_Page_029.jp2
b838554b342489bfb9a1d5e236b6e502
2591e5fb1929263d3f70aea36d00d8b1df446e4e
1051945 F20101123_AAADKI graham_s_Page_049.jp2
fdd2bc05b6417a7172ad9d3438001a9b
7543e409279f8e6b4e137b102afff29907dc6126
111540 F20101123_AAADJT graham_s_Page_030.jp2
b6641630ab84d0812c10176af2f56f4b
382fc4ee5ce2d39bba9a640e43b42a684d7f6557
109543 F20101123_AAADKJ graham_s_Page_050.jp2
a408e9b63d19d9036432a0edcc7cbe0f
c7c3cf0f682de6e66bf045d1398bcf1496578d92
115603 F20101123_AAADJU graham_s_Page_031.jp2
d4c0c0d0cb7a6863255fe39d978d4808
5797961aaf784d20bd380e54ac09cc77f159804f
23964 F20101123_AAADKK graham_s_Page_051.jp2
d66d073d2aab048525a6ad3460d8b4b7
7380e0b0eb46843342da2eb352a4eef436f88660
794685 F20101123_AAADJV graham_s_Page_034.jp2
913ab23cbe26ce29e6fb0d9f92a0a3ad
366c2802161806fb17fc91cb8c351639b9563d71
81743 F20101123_AAADKL graham_s_Page_052.jp2
7ad4a3a311a2b7fd53f4278a7a4ed223
1b02005c40a012a81301e415ca101e45d7b2cba6
107349 F20101123_AAADJW graham_s_Page_035.jp2
6f6ee642987a0ae84aedef7fbf64f2c1
611cdb399aaa4785280f48f954d50a244adbdd0c
26938 F20101123_AAADJX graham_s_Page_036.jp2
de8d2f0dd211b6d026a5ece6ddf4c1d2
37d3f484d0449267c4e45896f82b40dbf09892a1
838602 F20101123_AAADLA graham_s_Page_070.jp2
1e55b9f0af687c9e561d68de4d1b4dbb
f604f5c417a9a2647bafe4376914d16e7f5235c4
101917 F20101123_AAADKM graham_s_Page_053.jp2
e0766331ec46e0adf45c48bcf991c92c
b69a4751143e06a88086e5a5be560e74fc48f883
80616 F20101123_AAADJY graham_s_Page_037.jp2
878922b7738ae69a7a5b06ce3fcb346d
757a76b21cbfc035ad58d7afc829be7558f70106
97634 F20101123_AAADLB graham_s_Page_071.jp2
58da8c1831e87e812ae3e10e982bfe91
450d1eaea446e3efb18980bd7b47c175867097ea
102510 F20101123_AAADKN graham_s_Page_054.jp2
55599e15781b90e6791759de303e686f
0377221855617fc2dd061356d3215d9b05b0338e
111828 F20101123_AAADJZ graham_s_Page_038.jp2
7520235442ec06e7d777e76c72061d70
39706d04ba6de6a70f029c197dcee17f3e51393f
905343 F20101123_AAADLC graham_s_Page_072.jp2
7521cf2c96298a9c6cf9a0bf76622dc6
3be86e4cbc302886356c629f0f5d22b1c62405f8
92253 F20101123_AAADKO graham_s_Page_055.jp2
59d324b1f16ea43e0f93bcb1cdf1d8ae
c11e3c4f0be8c1e51fbd1090c044ac1167eeee60
807129 F20101123_AAADLD graham_s_Page_073.jp2
60f17a63250ae0df8d52e3048d6baf94
bd8c83b98908131bc7b5619fea67ff63093c79fe
800369 F20101123_AAADKP graham_s_Page_057.jp2
c11036b35d73b358203ff4179f4755f9
dc99ef6eb7bd712597cd1f6819f5aaa1ec9726bc
911197 F20101123_AAADLE graham_s_Page_074.jp2
2793aab28daf985353c9c84f367d9f23
1da964083ac090c947efe3a318caaec461d168cc
98176 F20101123_AAADKQ graham_s_Page_058.jp2
226908662068d6f2de8fb802b7f2fae8
56a43d5550dc08769732d17b1226ffe5251659a9
846379 F20101123_AAADLF graham_s_Page_076.jp2
0037246c8f01daac0ba20f52d8f759fc
2d467a62d67c7f5ca5b97d0a3d7c632258cdf764
103517 F20101123_AAADKR graham_s_Page_059.jp2
3cc58d5acfe09d44e17c700af45e58c2
0f58579ad6410466e6895890d81428f32c821c15
886301 F20101123_AAADLG graham_s_Page_077.jp2
6b3516c294d2f09739ca1b028536b06b
b92f78a87ef3f36f11c8c8df6442867598102403
838231 F20101123_AAADKS graham_s_Page_060.jp2
03b0c3737c2c2270d706ebdb47bb5798
6eea420a646824aab3c65e059e0f07ef8aeda679
82088 F20101123_AAADLH graham_s_Page_078.jp2
ff5014b8a0af04058a2266f653099b70
b844df16b5c7a9ab6cd19709627eda94f0a70c7b
108347 F20101123_AAADKT graham_s_Page_061.jp2
386addd3dad3203792d58d36c390a84e
c7aa51a1ac6d414fd474f60ae5b31708f697b161
957111 F20101123_AAADLI graham_s_Page_079.jp2
10076ecdaf7791089f505669e4f7e0a0
8a11d41cfa8eb9e6a0ee920ee0a1eaddcfb0cc32
99112 F20101123_AAADKU graham_s_Page_062.jp2
3aeeda872fee056839011799906a1d06
66fba6544b5ddfeb3d40e143b44f3d10fd8cb413
849626 F20101123_AAADLJ graham_s_Page_080.jp2
3369a3a6ebe2320afa3191699cc79be7
2473fd8c0f75b847217ef9be035fc2ada8893f67
841824 F20101123_AAADKV graham_s_Page_063.jp2
ba5c5d78d0211e949695920f978a9aae
44cb5e57e89cc533bfcf049f72f5efc477cae739
83670 F20101123_AAADLK graham_s_Page_082.jp2
82a128631e2a336110152a9cc07b4d4b
3fc382233a4fdb5ca4bbfec9d00117d9eab10a08
916751 F20101123_AAADKW graham_s_Page_065.jp2
a713e1164cffd31b4d69dd1c13a0d753
9b91bcc6eec7801d3680c1bb1f2ea2652f575a7e
F20101123_AAADLL graham_s_Page_083.jp2
b96aa3b0b130990bb9d8ce9f726a6460
035c04ff2e9ec60bd7da83c8f0755dab7243389a
89574 F20101123_AAADKX graham_s_Page_067.jp2
924eaa03cdfe4c56b50697bd133caa91
4f772f7b09afb95c23e241e27452e4896f9c986d
98651 F20101123_AAADMA graham_s_Page_105.jp2
bf0e42aea0b87192e4da2bcbf7fbfbb2
6ddb50ba51ffae3f9eb50a4003fbac122a5071fe
84454 F20101123_AAADLM graham_s_Page_084.jp2
edb8305dc32d8ba7933a572ab52529b7
191a3699eedbc6288e005a9c825a261f75b55701
100736 F20101123_AAADMB graham_s_Page_106.jp2
d9f5e63ec4643fef6bacb1282711e504
95dc3092b902caecea2b1c70b1dda9f5cec9784f
852687 F20101123_AAADKY graham_s_Page_068.jp2
d12bcbf2e4574c98fd2d214c7af1ad0e
f8e451c320aeecb9f1c72427a97c0d75b01c5fb1
105960 F20101123_AAADMC graham_s_Page_108.jp2
3f2b5ec3e9fdf52713dc8f9623426ca4
ef7ce3881f4bc95595078428231cd4fb6c32b5e4
864809 F20101123_AAADLN graham_s_Page_085.jp2
68b7f0a83c1e89832147236529924983
cf70bfb54ddf5c146bcfe2f4bcb7c67d29700a4c
772415 F20101123_AAADKZ graham_s_Page_069.jp2
7bba355ee7999380f2637d857c1e1e54
fd36b08d74b9e42f312a8b8450e65a6cab85db9d
98026 F20101123_AAADMD graham_s_Page_109.jp2
0b8befcbf39ba1b0e3b4ccecaa313970
918d828310d8731097672338af8cba9317f0edc7
891862 F20101123_AAADLO graham_s_Page_087.jp2
cefc8439bba5d1fe45d468a16515ac6a
8554f6d8b1014ed3a70d2a36a7487d3ea268d3ed
104496 F20101123_AAADME graham_s_Page_110.jp2
d16bfea390d7e016f29da47d0d64151e
9bf28552b1d586f86f296a4c4c542abf42fcb2ec
105215 F20101123_AAADLP graham_s_Page_088.jp2
9bd94daff7b787bb8e9422badf7c4cbf
0ad0a772b23c4a5a10b6fa62ad6cf8a55b617349
100140 F20101123_AAADMF graham_s_Page_111.jp2
53068abdc50609b1e5224a91cd1ecfc3
0183c02ed36aa9a417737e11e14e6e323af84aab
758446 F20101123_AAADLQ graham_s_Page_090.jp2
aabbc2228f4d0940ef0b681996be8a8f
54914eaeb8d3bae3c6be2ca366c37b03cf86ae34
103353 F20101123_AAADMG graham_s_Page_113.jp2
239cd613ae5194966a13fbb9c47e7c1f
5c365d7d3a3dd86c17a97807d6c39700ac38a6bf
91208 F20101123_AAADLR graham_s_Page_091.jp2
e5f291524fc04dd36163780217e24899
62417d74957b1e033bfe1316799f3eebddd01a51
101631 F20101123_AAADMH graham_s_Page_114.jp2
48527449bf426ce8eb8fe5aecd1b9e71
282648b75d29a59ad1b9d86956138a0bcb8f1496
987964 F20101123_AAADLS graham_s_Page_092.jp2
746cd9050c9fbaf8dd4bcb76cb052728
56781e521319bba7d6d7915d264afd6c52cc7aad
48508 F20101123_AAADMI graham_s_Page_115.jp2
ffadbc16f399c6a703252cf38d77601e
5334307d3043a11c7fb19054c447918c6771b904
816949 F20101123_AAADLT graham_s_Page_095.jp2
e4ce9f11bdde90ae2382e7dc18308fd4
3325f3577215c637f3fd9e61733612c63af11d95
97205 F20101123_AAADMJ graham_s_Page_116.jp2
415a1d4b27dc3dbbc46c4b09ed8e53ec
11cc4c98ef9ba27157c3bea8b7b5f650655c96a3
996605 F20101123_AAADLU graham_s_Page_096.jp2
861a8ee9f12117aab8ca1483ad447ee2
302cf90ef967801582ca2484ddf3ae09ea2c59e9
104658 F20101123_AAADMK graham_s_Page_118.jp2
bc64b039a4d7113aeadcfad01a9d18f2
ce7f8625c111156ffa97e8322d9ee32853f37f81
80448 F20101123_AAADLV graham_s_Page_097.jp2
dec1742c95fdc257ab75f50764a61332
07fcf5738cd195cc753f1cc9b41f61482b57f4f3
109194 F20101123_AAADML graham_s_Page_119.jp2
1c392088facf4daaace119f5d71f84d9
fae2250e051f63695d13b403154ae3d19a6bfe76
108639 F20101123_AAADLW graham_s_Page_099.jp2
e0783e6818186f22b8c1b37f6ef6fc95
e26eecaea0ffc2d3ec06aefdba229dc27f30d512
108809 F20101123_AAADMM graham_s_Page_120.jp2
6ee8d9beadf47b8181c8e740cf453672
020c1fb69f46027edd16a49ffe04241c5fbc1b66
92788 F20101123_AAADLX graham_s_Page_100.jp2
0efbe9a467885a4df9f53eeef325ddd9
1c5e54933a501b1442f9fd6a570cb9497f31c572
1051954 F20101123_AAADNA graham_s_Page_140.jp2
a326354c1b6db715d51132bd86f294ef
0e762caced0afc1e81dac35085146f9a40a5df99
103400 F20101123_AAADMN graham_s_Page_122.jp2
55c006677c9b91d9e73dbb247ab84a2e
073c30fd43d66146405ec2b1af033fa083e2f9e3
110036 F20101123_AAADLY graham_s_Page_102.jp2
0e28f3b66717808ee5a16e3897529093
1c6e5ddddd37979401d8ed45f406ce0110e8e3b4
59426 F20101123_AAADNB graham_s_Page_142.jp2
4d56ef563fd4860c618aa020d052f997
c25baa5ac1732df462760a1a9548c5c8c6821a83
95626 F20101123_AAADLZ graham_s_Page_104.jp2
08dda38523cbdc94e3d93808a62cfb7e
dc6bbc1e12085c5044cff0e1eaff4c53578794e7
1053954 F20101123_AAADNC graham_s_Page_001.tif
73c79c24ea745eea1613931326378a21
9bf8811698bba591b86c29f3e795546fedb6d214
99356 F20101123_AAADMO graham_s_Page_123.jp2
9cfd71e84d696bb7ee2a7e1009fa10cb
c17af03948ea3c47318fff2f3f38f6371835c789
F20101123_AAADND graham_s_Page_002.tif
306f60c2b0c898aa8b806daf4b314947
27e5e27cbb8d9644c33744785f5a7cc668909f5e
110132 F20101123_AAADMP graham_s_Page_124.jp2
3224e47c7e9941f31438d9f3346d0fae
dbb407b1ad8a7995d046859b07c1492c78dcc001
F20101123_AAADNE graham_s_Page_003.tif
0422691cee0d1398e7ef3462274793a2
24bb7a1eaa15f237d3024c1673939fea4210e1e4
103488 F20101123_AAADMQ graham_s_Page_126.jp2
f9768ea7fdd469177a112491ec519151
553f5c83a3127e9c8f45c9bb0c19d2f6087f2ee1
25271604 F20101123_AAADNF graham_s_Page_005.tif
3f20bbab01c7071bf83cabc69f6f47e8
c29d33023c17440806ba9eed95fff84e09540018
111704 F20101123_AAADMR graham_s_Page_127.jp2
044f2fce4ac24ff2027677daa440792c
2fbda2df1f9b32743c884722018bb69575332c3f
F20101123_AAADNG graham_s_Page_008.tif
5b795ce4c590aabfde6aa27e782119b8
d91e84423dca3cece3bedfe1ed1e02bd9995b5d9
70792 F20101123_AAADMS graham_s_Page_128.jp2
7a22b1dd855a0d73a351fd2a4700225c
ba7d4dbee1373ba421bff4bfb351dce3c693b964
F20101123_AAADNH graham_s_Page_009.tif
282363a04e25bf6fb6d58c91e93d1894
940781ffd4338c89ed57be9dc5246e368d4d525a
3441 F20101123_AAADMT graham_s_Page_130.jp2
f9f46255310b5386d24790cbb6349151
73d688b727bccd0126a5be006f4401a3ddf3e70d
F20101123_AAADNI graham_s_Page_010.tif
a9d1b66d87985c80f59dca48faaf740b
ecdb59b2330edbbb2950a3e14fbe8db0a8987089
100044 F20101123_AAADMU graham_s_Page_131.jp2
fc42d59aef525cf01e0f2f786ef5a227
50a82d98b35685a4a0ee2d849204013337beb150
F20101123_AAADNJ graham_s_Page_011.tif
c89034699b8a4edfcbaf27ab2dc99fa9
56d273278bfbf271c012978878bafab9938ebdd5
109053 F20101123_AAADMV graham_s_Page_132.jp2
7ffc6f0916150372a9149e32dc6d39b9
01ae37921ea237939a0bcf1d7ce578f44e271843
F20101123_AAADNK graham_s_Page_012.tif
49a00728845e40c0593c77866a9fd0f8
e626c704c97c1d5a148c6b7ed7b21e8cedc4b5a7
9846 F20101123_AAADMW graham_s_Page_136.jp2
795729f6af6131c5128f529e69be56cc
57ecdd634963d7c913f2e1993b6d32328ca0603a
F20101123_AAADNL graham_s_Page_013.tif
61535bd1dd93136f43728738f75829cd
3d6db15159fe85527884016106ba1a75949716d4
1051938 F20101123_AAADMX graham_s_Page_137.jp2
844e1a1d5e5eab1f16c0c110f769f977
ea1c09e3f6a94a72f6d345aca036183f13b62e64
F20101123_AAADOA graham_s_Page_033.tif
270d852fa7728b1ad797f86ca3237bca
0623eadde75320cd0884c09770a29d93a8568997
F20101123_AAADNM graham_s_Page_014.tif
d4153e8df3fd355bb01ecdb5f00550bd
8cd2c8fa45dfa80281c2bee6dbcbe0adc5033f74
138638 F20101123_AAADMY graham_s_Page_138.jp2
238e0340a5e035c70b7e182c09f36eef
4ed1fa619a689cea54ebd5b70a9926015535fd7f
F20101123_AAADOB graham_s_Page_034.tif
1355b6580d00827c1562b77d95be49e9
98511a6543b27e3f670c467b352118cb14645a4c
F20101123_AAADNN graham_s_Page_016.tif
1d1326d0dde3a02e740fc6841b01de27
1263b945381bb0b436a87f2ad43a5b9c0b26a86b
1051975 F20101123_AAADMZ graham_s_Page_139.jp2
9cd78d6dd9ffe27067a91f36a7ff3e60
a1f2ebfd00d50d862b7eb499157e575172ff63be
F20101123_AAADOC graham_s_Page_035.tif
42c569ea3da5bdf979ba3c5893adc960
0db58a2d75579e0168cf44cd36f609b0c697e985
F20101123_AAADNO graham_s_Page_017.tif
a5f0d2dfd1ef5a183f07fb48531211e2
a9d400a7b9b3eef8b2f219f4b0ef05c449c1b07a
F20101123_AAADOD graham_s_Page_036.tif
540e2be65c4df46e0f12b27d79d86f59
8fc75520f5959949b3bdc0ece2fd455bb4fe98a5
F20101123_AAADOE graham_s_Page_037.tif
ee3daca9e8caa02719e8032beca1580c
94a29305bfc8f451a60643b6cfaf661f82736802
F20101123_AAADNP graham_s_Page_018.tif
c96485ae90ce1ccf068e6393fdc72e1e
a3a411c67ef6beb8e3adc2222dfeeaae5abfdf44
F20101123_AAADOF graham_s_Page_038.tif
36109c4d018326825accc9593234c6cb
d611d7d34131cf3996f7a4b0bc2dbff6f9c93157
F20101123_AAADNQ graham_s_Page_019.tif
a2bfd0b354d97ab928bade3a7d672fc0
04c240e41950ba8d51771e643cf18df2bc8a5219
F20101123_AAADOG graham_s_Page_039.tif
2179cc5e0ff15dc85e5474f8fe53edee
9f44f8697158933cb063dde443f0c2f5378c7e14
F20101123_AAADNR graham_s_Page_021.tif
c8f09934d4133ba8293d3ac5ada6c179
b9be980f5a7933ba16c8a173ff79b6ef3c986786
F20101123_AAADOH graham_s_Page_040.tif
1a958ba47bfd673df00274bc2940490f
a73389c37a5d66d9dfcde3d2c47278ceed8846a0
F20101123_AAADNS graham_s_Page_023.tif
57c957bce920e8eef1b329e37d37beb9
2ab03b96dbbf1e45d977c8c04d041b97f0e45259
F20101123_AAADOI graham_s_Page_041.tif
76da002166aded9195d285e2a76673c8
3a6e841013c869ff4a4cda0e9d191a1db4be6f2b
F20101123_AAADNT graham_s_Page_025.tif
16db824c7e67fb53dadeacdb90db43fc
e7472ee409643741707bf24cf86655121a880987
F20101123_AAADOJ graham_s_Page_042.tif
ca3c2ff57cc2faeabe2422c4dbd10a42
fb537f1c2038c7f72c61772e6f9a46d3511d6fef
F20101123_AAADNU graham_s_Page_026.tif
a58e857739b096a88223b75cad47ea33
ee8832360f51df3e39193f4de81fd285c0e01710
8423998 F20101123_AAADOK graham_s_Page_043.tif
76f8ed1592b369c91e5510effa0b6d05
02cec26a9136e5675b1a3cc469aff7a1889b5247
F20101123_AAADNV graham_s_Page_027.tif
747a7a64136df216909aad55d5ae50c0
d2d2dc535743520cefeab57983adaba1272f0040
F20101123_AAADOL graham_s_Page_044.tif
1bcbe9a1661b6bb12d8f165202ab4a01
786c2f4969db227b03195266f831ec5b7bc7f065
F20101123_AAADNW graham_s_Page_028.tif
034a8a91f91519ee0982c5a5c4403feb
f8d259dd4e0a1ba0e9cfc64d12be5a6b7689ab54
F20101123_AAADPA graham_s_Page_064.tif
573acca35e1c854e6f2c179ad435465c
e4754e651c4f11f7a2332191f67f7ef0f30799d1
F20101123_AAADOM graham_s_Page_045.tif
99a347908632d4dec7b684c30151365f
826076db7a41d5685421ce4e1835d2c2e8caa4f0
F20101123_AAADNX graham_s_Page_029.tif
9537ac0ddd46918c104185eb3cc0f029
c961f47972033b93198c400b9f8ffc2dcbabff9d
F20101123_AAADPB graham_s_Page_065.tif
0a159f54a49d3a7788b3ac8553787f46
317c358e57dae8514be546ce95b5cfc4c789eac4
F20101123_AAADON graham_s_Page_046.tif
2c271005b773d0fd68f7cae788861d1a
cb61c58e5f55f57d938afb65bf0866e2209c5f3d
F20101123_AAADNY graham_s_Page_030.tif
cc791afd1408620f0a635b99f4342ed4
a22df6d6f868115d5420542231556c2e4654baa5
F20101123_AAADPC graham_s_Page_066.tif
1c55ff98d3fe3e24e4404cdc948790b8
7fb7dd1418bcf65e3256a05aefe37e0fda25ec2f
F20101123_AAADOO graham_s_Page_047.tif
8bf91d48245098850bb50c513bf274ae
9c5ed17ebb00c113d59d2c96c7e3d325e1b06564
F20101123_AAADNZ graham_s_Page_031.tif
48816383536836ca996619bc1632387e
5f8207a332ba5391365bcc944cc72bdc3d5fb4a2
F20101123_AAADPD graham_s_Page_067.tif
adc4a0c3a64950c3a29a517e194f7027
56612f1294c2ca3de15725ea646204cb90cb5b42
F20101123_AAADOP graham_s_Page_048.tif
a4fcdc2a9cd93b93d83b3343a374828b
a0befbd14bfb5e07bce65e25cb2b1e5b2ac9e6f2
F20101123_AAADPE graham_s_Page_069.tif
24d9fb56e58c33e645e28465a0e0a81f
4d12af53e9b505d924ede811fb87eaf179bb4555
F20101123_AAADPF graham_s_Page_072.tif
6388a19f45f5c55cc4d96c7e626c1652
7dd2ac209fd83c92b01ef067ee676e0c9165acbf
F20101123_AAADOQ graham_s_Page_049.tif
69ba450bdab8dfe34267f3326c83fcdf
dd0eb8527efdf4446bc43e5ff57bda87f9b708c7
F20101123_AAADPG graham_s_Page_073.tif
75174be3363a5dc758824e865b53e61a
e29bd97771173e943773f7fc3f267d29b80152b5
F20101123_AAADOR graham_s_Page_050.tif
ff7ffcabf3ff545805f223ac8e3bef12
2c78f724099c911b94c1108bd63f3d121aef81c8
F20101123_AAADPH graham_s_Page_074.tif
a96bd578e7eba3d705261794c47516c7
acec07e4044da5b5a4db5962cf5514553b4b22b9
F20101123_AAADOS graham_s_Page_052.tif
59f067a3e8ab4abda3f419bb9361ad0c
047ed151c46cefeb9c7b17f7fe92698aa412cde6
F20101123_AAADPI graham_s_Page_076.tif
c904b433190aa9f641fd4a58ea839b4f
422ef181ca24e74afdcd768a4e323df349514357
F20101123_AAADOT graham_s_Page_054.tif
9c02ba949c39c5e989171e56394b167a
190c16ce11242a35cc7f68cbda4004eef9b724f5
F20101123_AAADPJ graham_s_Page_077.tif
ba75f7163e7cdf84f90aa5bc94fe54f0
fa6e431bd3453e661c34bc4461b9ddebf4571550
F20101123_AAADOU graham_s_Page_056.tif
31e9d12eb747f1134d394d0f2035bfa2
a20581c424f596f2fe7f12b3d1c0f35832beeb76
F20101123_AAADPK graham_s_Page_078.tif
0097dd0f3383c55e4d5c3c7595512708
1b0b91fcf71724e9d5965f8a6de2126b59232839
F20101123_AAADOV graham_s_Page_057.tif
8bc179909dfc87f20e451fd63aa28658
530936039be70b1aa37071a977e87d1dfb1205fd
F20101123_AAADPL graham_s_Page_081.tif
c6bb4530553b71edc9a423a6cd429d7b
c2b140671de82bb7b171b969f4e03c4bda816f1b
F20101123_AAADOW graham_s_Page_059.tif
aa0084383bc79bf6769a443cda03654f
6203bb38115477d10b9935b0d91686fb27fcb8e5
F20101123_AAADPM graham_s_Page_083.tif
7a3db5e4aa03ca6db8948a14f934eee6
2bbe7bcc175fbf77be6c8c77d66891f6e6c85df5
F20101123_AAADOX graham_s_Page_060.tif
3f4034504878a35c72342913e93378c1
fbd85f499fec15595b4a2d2a50262082c8206acd
F20101123_AAADQA graham_s_Page_098.tif
37a79067c9ad77f6e6a4bcfa9a64b8c1
aabfaed10a4edc4dcb37a061138ecca69ce232db
F20101123_AAADPN graham_s_Page_084.tif
dd1800307ac9a0fbeee6304b30ad6edd
40e24570f4bcdc16bf9d337d29b5dcc2dd79ee7c
F20101123_AAADOY graham_s_Page_061.tif
6e1d78645b150f56028f5a56fe717016
261567759d4d37e791f58bf35b398f2a988963f8
F20101123_AAADQB graham_s_Page_099.tif
e3ca7d444532b3395ccb6eb87da10323
6f3018b7e9c040085c3782b28a7f99c11c88e453
F20101123_AAADPO graham_s_Page_085.tif
15fa17f290494a5f6b26a43e2ee31efd
526137656e80ad3cd71beb72096c7662603d3a1d
F20101123_AAADOZ graham_s_Page_062.tif
c8ff5b8cfc404d646967746e804a807b
e610335720e339c65219f3fde5caddc300097610
F20101123_AAADQC graham_s_Page_100.tif
f95ee6711e10cbd58351da28eb1c6cce
5a9ea0770f66dd759468e877f4ecdb3a1a14a422
F20101123_AAADPP graham_s_Page_086.tif
baaebf76cf8d25aa9b5d79d10c105aa7
27b306c024b6c43347a7184582cfac2eec37e2df
F20101123_AAADQD graham_s_Page_101.tif
71eadfb87c265b6c311975f416433d8f
4f3673435eadd6220cacf8fa256258047d895431
F20101123_AAADPQ graham_s_Page_087.tif
72a7ede5e74604dcc5a4652c6fd2fda8
6975a80014d0d7124333b2102fbf8cc244ba3b9c
F20101123_AAADQE graham_s_Page_102.tif
7e04bdb5b36bbb848e6402c74b536e7b
c3fb98d2a57346e45035b0cace3830364e996858
F20101123_AAADQF graham_s_Page_103.tif
f782a485edfbba3c2cf9674b38ac2d53
e0a1072374176e2cc8088a0984e9dc91421ebc12
F20101123_AAADPR graham_s_Page_088.tif
7dfbc8e7dace8840f9c29b6457ce2f6c
6592fafe9d76bc5e522895c21aa8dabc7149cc91
F20101123_AAADQG graham_s_Page_104.tif
26bab285899518296874814e07fe7c0b
ac9000c834465e342d389f64577bf054afc43484
F20101123_AAADPS graham_s_Page_089.tif
07b793c35217f8e46057c64f46d4a291
119b6fa90188d946675316657d441038528aca4d
F20101123_AAADQH graham_s_Page_105.tif
f02b258363629c19b7202bc0dbcf27a8
2fdfe664a8c66c52295e53c0fe703ab20f8928df
F20101123_AAADPT graham_s_Page_090.tif
b864eaca712581178ca00e1efb949b34
57fb100e780beba7b40bff253d3b29732357b550
F20101123_AAADQI graham_s_Page_106.tif
3276fa7ae7f493b56ff631c0b1d1f33d
c403fa0a20626c6bd251ea43428f71f4debf3fc3
F20101123_AAADPU graham_s_Page_091.tif
d16302fc09d3f810b56010cae775d43d
c431bda77e7a2ce03ced0c44e75ca89317d7ea5a
F20101123_AAADQJ graham_s_Page_107.tif
845d8b934ed30733f29061d4a70ed4b5
f2bf74843b3ea3a9426025bea9dd723363ff20b5
F20101123_AAADQK graham_s_Page_109.tif
e61b7a2e726109cb304e9e578e65b710
53bb303c8eb1f12c62408ce70be16ef49457b438
F20101123_AAADPV graham_s_Page_092.tif
523b603d063f2565862893ec36426506
ce587b103df4bd07820ff2ca79bbe2607e55747f
F20101123_AAADQL graham_s_Page_111.tif
de5d6a9656b3fea857bf6a75f08a68cf
45471bef6cca30f5a2f8a6023d3a95a87ec96985
F20101123_AAADPW graham_s_Page_093.tif
c8c4116ef07700cafda3e1c6cd56ca63
259faeb7fe83bfebacbd2dfcb4913263ace40a1d
F20101123_AAADRA graham_s_Page_128.tif
98089f2da5eabd46434b95e7ff9c2b63
f597e20156f91c1f6f91dc860cd249e49ec424ba
F20101123_AAADQM graham_s_Page_112.tif
ccd07747c02dbb56c625459e8ce0d7d1
bd5c69f782a9397f0d93c7ddb07f7d3899395bc9
F20101123_AAADPX graham_s_Page_094.tif
5a849724c8d24cb90433d8154e659e11
56c47f91f14b59b0517775b11587bc97afd2c279
F20101123_AAADRB graham_s_Page_129.tif
30277d922ce951b7504a5c4f38cb3a3d
28b66c038c5ead7a9db0b360af3e6c753948015f
F20101123_AAADQN graham_s_Page_114.tif
313f8803bdc16655f03e8633e703bd82
b51fbb9d8f9b73f3a7febd735a888d4df6ab0c4a
F20101123_AAADPY graham_s_Page_095.tif
27281c8b1e0eccd47b67ed35016771da
df5809e56a9498d81c0288e49da152280a6b6318
F20101123_AAADRC graham_s_Page_130.tif
d04e1f669c9606a86b528689cb4e3e94
fa893f555bb347b29562503ee43a2eae0ff8e6a8
F20101123_AAADQO graham_s_Page_116.tif
2ad6b73e90cc10caccd1d66af1bc9429
82de5862e346285fb94d46ef7fa7f362afbac0dd
F20101123_AAADPZ graham_s_Page_097.tif
213e4d273580d7a59a6eb29695da8707
c218f553bd2ef8f9b0ec5895accd1834a2395883
F20101123_AAADRD graham_s_Page_132.tif
957595acfc9b683ca25b0f8ccfa3b633
2bd7165285b45716c0f6aab77d2b646cefd32969
F20101123_AAADQP graham_s_Page_117.tif
a62750e1a19b0ceeefc921212610e3af
26ababfc4e0d8a6cc6b3fb9f2726ece9c9fc0219
F20101123_AAADRE graham_s_Page_133.tif
9ddc4043521d97b7cd912d916cf523ac
769ccc6e051fbfa0698e8cc03f4df0a6aafdb5db
F20101123_AAADQQ graham_s_Page_118.tif
e48dd4cbab397d06ecafa4ce0fb69275
0d944fff1267cf124455dfd752eff8c8cced611e
F20101123_AAADRF graham_s_Page_134.tif
3e559446fccb20990d629ead88441abb
4b9ea8b828c131499df34229481d9d80fd3a2ea2
F20101123_AAADQR graham_s_Page_119.tif
0de1589137dbca50801b8d75786c22a8
d84a52110984b21741f2494872085423e636f0bc
F20101123_AAADRG graham_s_Page_135.tif
1a287b8d1c83ab64d8d3b3aa0fa3f937
e6d3330798e1177c6f04b5c4d34fe3581c91c22c
F20101123_AAADRH graham_s_Page_137.tif
ce785b1ff25d2345afdc7abe8445cdaf
3b2442e29417a342dfa67caffd65dd2232512f78
F20101123_AAADQS graham_s_Page_120.tif
bcdb317e42d15128a4d634b0101690d2
8a7095c109afacd1a4dd53804e664a0352ff5c64
F20101123_AAADRI graham_s_Page_138.tif
db5eb5f28d2892b2d83e2cc1d22e8e90
f1bc16fc34dc3a1cb69999b2a0183815a531711e
F20101123_AAADQT graham_s_Page_121.tif
779f407a7ca160abfa0e61a95e6c05e5
10e2c967d2083d87c664ab9f7a20547319546983
F20101123_AAADRJ graham_s_Page_139.tif
03a17b26b2d7cbdd36b5230b9db430ed
453b6e1d6c052c15acb6330e51d7b3ab934df91a
F20101123_AAADQU graham_s_Page_122.tif
be02c20b7a79af3495e20af9b0b7abb5
deaeafc818f8ad110b0dfaba2e258de7e0e62c96
7857 F20101123_AAADRK graham_s_Page_001.pro
2e18884300f0d14587b0d3ea04a57272
fb4bdcabac23cc506344c252fe0375da77a62680
F20101123_AAADQV graham_s_Page_123.tif
c8dba683a0d74b797bb74e860c324edb
75fe712de076b6faf6a476fb4977a8a124d8f3d5
468 F20101123_AAADRL graham_s_Page_003.pro
7ac44353983d00073e2db6d4a7b87d9f
0717f54b40933d120a0bcc7d96cef33e14862b1c
F20101123_AAADQW graham_s_Page_124.tif
424baedf567b7bb71d359179686baa3f
af06cb3a06a9e7662574698016a92a78dfb74313
17742 F20101123_AAADRM graham_s_Page_004.pro
0fc282057831a458dc6f95936e1edd46
65ddf0e33b5ef1a8197668ef17e38f685152b08e
F20101123_AAADQX graham_s_Page_125.tif
a6a749accc6b8e67d7ee6a9e34ebb28b
c8afa8a4f1ae4d7038bfa7c03b9577deba8cfb18
27247 F20101123_AAADSA graham_s_Page_023.pro
fa81b7e8bf4d48acfd9bcd3ce335dbad
fbbc75e69579350c3dab8c61df784d72ab2d90b5
75407 F20101123_AAADRN graham_s_Page_005.pro
ed6a153c5f5a58347fd32be82c1f7a29
72d49409b258dc170344f3ed70f562692f20b020
F20101123_AAADQY graham_s_Page_126.tif
e0e85d239b4bc11dcf32babf8aad2bb5
7f1e9f05ab8ed077d236f4e648f41ffeee35a4b4
42740 F20101123_AAADSB graham_s_Page_024.pro
535ff30e190e2909c254d1ff55de0a49
4c4f19f3b068c3db1c217797198f267faf146151
114899 F20101123_AAADRO graham_s_Page_006.pro
c7104c0f50714983e99c68d8df9bac5d
82d29dd7f2367518cbc5163d8746727d195dde33
F20101123_AAADQZ graham_s_Page_127.tif
658c1d6449e99e3a94bfa52ce8e67846
7330d8bd49400e87f682b52914f5527b702cacbf
53943 F20101123_AAADSC graham_s_Page_026.pro
8b1ce351e9a6dd6a0889385b830b0020
a7b18b8fbad1d976052cace27f72f08e48404703
37609 F20101123_AAADRP graham_s_Page_008.pro
7a1d3c1e4f05ee30cf1c52c49ccbd0f9
5725b8bd7bc23691ca7491f7ffec841a18dc6e65
53664 F20101123_AAADSD graham_s_Page_027.pro
77cc4e3f6a1d0367901acf363685b39f
32a45192af9a4ce1399b564b3be0169eddea7fe7
55063 F20101123_AAADRQ graham_s_Page_009.pro
aa5003ed4f427e38967aabd96b4b00b3
59a23da1bae7b9a33e4c1a9781d396f3e1fa8a9e
51107 F20101123_AAADSE graham_s_Page_028.pro
4e53ec2a74c551edc4ad2294cb79eeab
ade743a6f00af316c1c19a7a919c7f2106869337
32986 F20101123_AAADRR graham_s_Page_011.pro
6886dd75a41d665f850468dccdca40b5
9efbfe4eb8fdccb886f352f0bba8e161ba62056d
50331 F20101123_AAADSF graham_s_Page_029.pro
7e37bbe6346b6b3d40be12ec3fd02260
f28ef3f7a19e8e691e891e34eab498358137fa75
54061 F20101123_AAADRS graham_s_Page_012.pro
f24e679d629bef007e5ce9bc57f81ac9
ddd82b112ca6a4bc2ac796245e9f0486e57c1f98
51087 F20101123_AAADSG graham_s_Page_030.pro
a4b1645010fc65bf74f2284297ed8975
02620a2032bf212f4874368e7daee4ea2a69dabe
51025 F20101123_AAADSH graham_s_Page_032.pro
26517a7fe16b119312d9d444064e7a16
6df55a9219697b39ffbbbddb295932b840c12b15
26983 F20101123_AAADRT graham_s_Page_013.pro
7364d5ba63096036bfe24909ab20aaf9
02b6d2568592197487cf3d3760bcefd8b7460b78
27866 F20101123_AAADSI graham_s_Page_034.pro
5f764a3ab6bbb554345063ded384d105
bdbe2d3043b1fa741c05023778142c8b24e566ba
36546 F20101123_AAADRU graham_s_Page_014.pro
e6b9f6bb8af8b05874e9ab85cf7806eb
503287001899d973d523ebbf939423cb3283dd2a
50739 F20101123_AAADSJ graham_s_Page_035.pro
f68dfc30eebc01ae0e3ffbeae6de9aec
27542ed3b88a8c68c42b3e6347404e55f6f69598
42391 F20101123_AAADRV graham_s_Page_016.pro
33d841db585dcc3228fe7057972135ec
aa3be77f423a62e3ef279d22e132fd8e7569be00
11170 F20101123_AAADSK graham_s_Page_036.pro
7ba0b7f412c7bb3c65e8b419af7cfd10
3a03be46b0cb0a62b58f59b2ef8b7f62f7c77ef4
52785 F20101123_AAADRW graham_s_Page_018.pro
54a2ae35f770821fc874738b05a5df16
9facaf1e9195d22095b3381363b9aae42a73a24f
36702 F20101123_AAADSL graham_s_Page_037.pro
7681b4762c0dc3092147c9c4845e49a6
2817a05e8fff2fefa79afcf5373991a02863b537
46717 F20101123_AAADRX graham_s_Page_019.pro
f7f38ea5a0c944df39807aecaf6cd6df
62dd5cb0a27e800eafd0195504eb1e8b7262fcae
49867 F20101123_AAADTA graham_s_Page_053.pro
7067808139df9981fdf08443ff84388a
81192eb894f31f82fee32513202df34cc8562fab
50934 F20101123_AAADSM graham_s_Page_038.pro
4b3dd2c0ac08cfbc11d12fba69acdac8
08c82a32346b9289c3d2e8c08cbd7c4ac126f198
37783 F20101123_AAADRY graham_s_Page_020.pro
9ac077c5ca636a1de6f0f60852745f2d
bf6b2b571d3ffd6c9180e22b41e48308ebd075d6
55483 F20101123_AAADTB graham_s_Page_054.pro
868bd03fcc4e46c02c3335f31d0d0334
ca7870456e70e9a04713f1594a8902a492cdf46f
48077 F20101123_AAADSN graham_s_Page_039.pro
b35e9917e9ba3983ed0385435a035b36
74978971d3260e40758558d47099d0015b4dc619
42686 F20101123_AAADRZ graham_s_Page_021.pro
163d453aef7b59c7e68dddc750a41e7d
a23f29962a4a83194ab8e2310225692c3b6b8d63
43076 F20101123_AAADTC graham_s_Page_056.pro
39f5d1d1784541b09128bec6cbfbe225
99db19234a00ab7b3dbe428dedb60adc7c016d97
52927 F20101123_AAADSO graham_s_Page_040.pro
29d1899c6441b635f05e17af82b4cefc
ae9d783a920904eec1278d86803ef312b29b88bc
49294 F20101123_AAADTD graham_s_Page_057.pro
a31984bc2746f6ec62022c5953aa1f4e
9736224b576fb2c0733b914c3db1723ccea8ab78
55036 F20101123_AAADSP graham_s_Page_041.pro
1633faa8dd46326eb4b9332275d06383
596caa404b3bb7472f4d27747a3ee3e4034cbe54
59970 F20101123_AAADTE graham_s_Page_058.pro
0f5883fd3dc4cb4e49269abac698ba27
2a37df7342dea0608431f4ab9b1fea79787fab8f
45338 F20101123_AAADSQ graham_s_Page_042.pro
a863cd91336172ef836afddc702cd011
db5050c02d2ced16d55975f16fe85d47fdca1430
48171 F20101123_AAADTF graham_s_Page_059.pro
ec4861d46357dbba532d98b970ce8209
52557e43a393bf544bb84108b3b4949552111923
46988 F20101123_AAADSR graham_s_Page_043.pro
ebc2335380ec7ec7a5a9d53308290c6a
18426c744a6349b598c5518dfb45440dd65d8015
50164 F20101123_AAADTG graham_s_Page_061.pro
99036622ed336005ed35b30ffd208580
785886fdf14f715a3a98780e4f1714ae6de90f70
42398 F20101123_AAADSS graham_s_Page_044.pro
ae27c563b1a501150a159b1689687323
905eee7f9002d8ac5861651b804f7f656a31c854
72848 F20101123_AAADTH graham_s_Page_062.pro
4fa29a49864505853c835cb3ed2aaf18
f2952adb6f218d53a5400203b6deee3a1be2846c
47012 F20101123_AAADST graham_s_Page_045.pro
da899068aa5b19cd6969f971bd356353
514c77ff1b4339924239de523209f369e233c81c
38170 F20101123_AAADTI graham_s_Page_063.pro
8551370878d5160185224b45a3ce6e8e
417cd55990012c7825859397604678d2f9c6ca51
24506 F20101123_AAADTJ graham_s_Page_064.pro
60e52b6c316c05e0ce1cb8421949953c
37faa527989253a2224eaa768c7fc52e8f1f45eb
50120 F20101123_AAADSU graham_s_Page_046.pro
44fac49e784fb5c4b3aafc9f58a4dd73
8684f2b0aa00882ba5645b0063d70bf284f4b80b
35783 F20101123_AAADTK graham_s_Page_065.pro
460215bedf169a94e7f8ec3a21fcd99e
3355cde4a45aa0319443b76f34da6060626ed6cf
44832 F20101123_AAADSV graham_s_Page_047.pro
2e420c21b5162ba775de94b1ddcf2094
457f86f0ecc63d46516ecb2f26008c9f54d55087
46239 F20101123_AAADTL graham_s_Page_067.pro
fae316dd77d449417abccfe88db5b6b5
3db5a6a04eec640eb3bbb819cb5f9c0c8da7fb84
47919 F20101123_AAADSW graham_s_Page_048.pro
86a1ad8523bd764eb23f50b8f56d1a0a
4b7d0b389b2d15fb63640a41bc34ef581b5a3962
52540 F20101123_AAADUA graham_s_Page_088.pro
e65cc5417e6065c68f16138e3afe19a1
55de0d5d805197f374895b7af658f31a5540a071
34447 F20101123_AAADTM graham_s_Page_069.pro
2423bbf00e434299a8eda1643978d62a
c49c429038aff628a73be2aed7ba63b378ef1085
48487 F20101123_AAADSX graham_s_Page_049.pro
b903891bae1705de253c043d672bec1a
82c438765e38054e0c5fc9b578bcf45d7a471b89
49170 F20101123_AAADUB graham_s_Page_089.pro
878d8a4dceb155ec7ae5a26c44e9f96f
1e1f2a7231889efe6856de4e643571c125417ad1
37710 F20101123_AAADTN graham_s_Page_070.pro
adfe5a30a7e246f7924fa5c5d0203480
fd7fc277555822cebbc0875cc9f5fed08ff8d1ba
50091 F20101123_AAADSY graham_s_Page_050.pro
23a651bc1d014678c6d5ee9b2d4846f9
70caa946c952e272937d7ba4ff7c5fbe981031fa
24664 F20101123_AAADUC graham_s_Page_090.pro
d8450b800bb32544c78f7ecc37ecac0e
c80e6a2c1c34cdb9f4d1456222d69c21db69845a
50157 F20101123_AAADTO graham_s_Page_071.pro
5819e196de3ce6ece8b330c4aacf79ad
40922d8fd974df0b4b9024cdbe7e1fed19e1bafa
9714 F20101123_AAADSZ graham_s_Page_051.pro
f787386a9bc80569b3107926c32c2915
f4bbdd402dbeb4f4775d2c2b41684565788309c9
47141 F20101123_AAADUD graham_s_Page_092.pro
5d32299d0996a09ff97b41f4b040e46f
80175bb3fed9750e86796f62ab5697b67037cd86
31219 F20101123_AAADTP graham_s_Page_073.pro
9ce4b1ea3640cc519a901296c6c1b771
b77da682e0266dc2cf1e922cdc16857de50ad81e
29485 F20101123_AAADUE graham_s_Page_094.pro
ecef5c2ac10f4dbf1a933203cc40fb3e
571daeccd0951449eda47745efe52e6a280f8ad7
44109 F20101123_AAADTQ graham_s_Page_074.pro
9c818c64b8fc833ac9a369b4a095d32a
68e6d6b2199fd573ef5442e2fda127b961ab6cca
35376 F20101123_AAADUF graham_s_Page_095.pro
64f5aac5fdb9ee2a7fd56f544b3909e3
af0c03d6de30e6fe85b4359c4834676c4d0ad079
36857 F20101123_AAADTR graham_s_Page_076.pro
be1187cd84c2b7a2fbb9bf8495b76f3d
c208cd586cbf77c27a8a02e52e5ca542dd1513b3
7575 F20101123_AAAEAA graham_s_Page_001.QC.jpg
ba9656c0a2cd61b67152f80a7fe0a7e6
1a7f37acb7deee60a4b2aa84f5cddaa1dc3ce8e0
47529 F20101123_AAADUG graham_s_Page_096.pro
55821fa6e93c2e1b336eee94bb95e62a
f25438cfc0996edc26ddd2d6703c2ab477aa5801
39265 F20101123_AAADTS graham_s_Page_077.pro
33ffe4d42a7bc875191a997f873f7310
354ff2859f4b645137c1dedbe0481d0c20be0d42
40620 F20101123_AAADUH graham_s_Page_098.pro
3d2d3692d6810d0beccf1c6cf632897c
c8d17385f2790bd1a1d8e1319486332d6e22c338
46080 F20101123_AAADTT graham_s_Page_079.pro
b34298028698f86c7d6278f8c9ac421b
e6acd36ac20c5085a08d9b21f92091f598ec5f6a
635 F20101123_AAAEAB graham_s_Page_002thm.jpg
3a4718ba8f747726464fac30b32b82a3
08745ab94975c0018949d89dd8903be7340c89f7
48960 F20101123_AAADUI graham_s_Page_099.pro
5bf7a7f3620a5d0f07508daa131aad3d
e57edc89b146081c882322e5ec649cb3ee1a3e28
36405 F20101123_AAADTU graham_s_Page_081.pro
fecfbc1d7ff7e30cf877178d31cbcbd9
bd75b17738378d399c7f1559ebde92b3c1f2d61b
1661 F20101123_AAAEAC graham_s_Page_002.QC.jpg
06bae458434c181c516824bf9f4e9c3b
37174ada053f11ec9c95bd9d5100f9e75c58c5fd
43007 F20101123_AAADUJ graham_s_Page_100.pro
c7bdf1a7a68119e54267d9c84ce21b1f
eb23e0357d987aa707130b80c39a2f811548e104
861 F20101123_AAAEAD graham_s_Page_003.QC.jpg
53011e1d8f3c8c35ddd13d5f4abe3000
d1eae5f6ec4627f683897ed3ba054c627b645192
50031 F20101123_AAADUK graham_s_Page_101.pro
0b115ec2a6c157286774684949ad7c5e
7aa6c3c20cbe930a5d843646e94c0ba8aab8fd82
42672 F20101123_AAADTV graham_s_Page_082.pro
1d7786cb97b969c606d24f59769612af
49037a57770821cd8e231b7dbad5bd0d3b10af28
3655 F20101123_AAAEAE graham_s_Page_004thm.jpg
dc40c596c226a3600ca830fd6c5d7fbb
51992e8945c62ac394023ae7adac59a2fed47e0d
50972 F20101123_AAADUL graham_s_Page_102.pro
d5959c8206af443df02f647bb1dd6709
9a7a4baf0b6de3f29da8f2f7c95ee1d9735483d8
51704 F20101123_AAADTW graham_s_Page_083.pro
f23a252685c98c25f4db3313d4fcd91a
3966920fcbb780dde65b47c70dde2c8ff67b800d
13344 F20101123_AAAEAF graham_s_Page_004.QC.jpg
6253bbc6c7c2ab7c2cc5c406dc5a87b2
1f92ceb6d52d67d62fe8674d0b626811958568ee
43901 F20101123_AAADUM graham_s_Page_104.pro
2064740e3723783ca01c321fbdb5e41e
30513aa87436a3783743313860898a8731d094bc
41576 F20101123_AAADTX graham_s_Page_084.pro
632954369e8aa4685946564e40a3b1a1
962a4451e16a76228d1c4027e07d23fd39407564
21403 F20101123_AAAEAG graham_s_Page_005.QC.jpg
458b1da9da8f84bcc8e8c499c97545d5
616bc525ca4ebce4e725c07e1d424c803b821b43
50431 F20101123_AAADVA graham_s_Page_124.pro
173cb880d676aeecefd444a5ebaa55be
20f09ffc728335e2faa0667fdc8f157bf8599f0b
47092 F20101123_AAADUN graham_s_Page_106.pro
f1dfdcf7594bd606e03f9c41792cc0b5
a877948e637682518db844272207fc8cb99aa24f
35700 F20101123_AAADTY graham_s_Page_085.pro
3f2058ba415b4302d544e43a144410dd
262e85db61b3357f9a60a4a9b0c00a4ee636637b
7785 F20101123_AAAEAH graham_s_Page_006thm.jpg
0bdd886939e621ea3bdc22686e303ed4
5d4673ab4ad8deb6dc02361e33475d7331ec66e2
51857 F20101123_AAADVB graham_s_Page_125.pro
e497c1220b7b13bade8a4d4f34fdb657
7675f2c4a405d2b687702eebe365db596ac28109
45064 F20101123_AAADUO graham_s_Page_107.pro
22150931f7230f9bd174d1c6b965ee8c
bbbad1f4e055553ec309ab57f0580d39115db48d
42172 F20101123_AAADTZ graham_s_Page_087.pro
af9c8c4f1140d987d8a630abe7d97bdc
1ef34acc8ca1332fc9e1869f530c291e9a387843
6830 F20101123_AAAEAI graham_s_Page_007thm.jpg
fd3b1d8fb505a8003359807eed5314bf
4b5f5b116bcd15da3ccbbedc09592cbf8c677d3d
48480 F20101123_AAADVC graham_s_Page_126.pro
1c1b137d61990854d143bc722a1e7fab
ff23a06bcbe2fd2eb7a70ce837b67186612f862b
48797 F20101123_AAADUP graham_s_Page_108.pro
7f72075380c3732827f7ad8796dcb90b
f82f264ff4c4ea0aa4f8b221b9766d1beb1cfe88
28719 F20101123_AAAEAJ graham_s_Page_007.QC.jpg
8dd763cce467a7b3423f119030699461
acfca39cd920066424524eaa3bc1efa9ec21c1be
32076 F20101123_AAADVD graham_s_Page_128.pro
dde6d45be7eed259c35a163520cbcfbd
635c634297a1e95eec2ad94edb950c4e7bd7eacb
44368 F20101123_AAADUQ graham_s_Page_109.pro
c0a84de9a01461ee05c530afd6fe2770
8a4b6db68e0b8c46ec8d445326005a59656fce89
3764 F20101123_AAAEAK graham_s_Page_008thm.jpg
e113fdcf41fb77ce874e0969ce94e992
7414489da2d887f099fda5a688e780d7150b813f
45475 F20101123_AAADVE graham_s_Page_129.pro
2386adc6f21600d3f7c508d3f0edbe55
ba4f13012afd0a2d59462646e5f343dcffc80f85
7486 F20101123_AAAEBA graham_s_Page_017thm.jpg
64eb9fc73e9ac097aa366450ec9c114e
9033d42f3ec9fa97fb52802ffe3bd6840cb8ce60
48707 F20101123_AAADUR graham_s_Page_110.pro
4b545c1c4072675bc446b03cfdc3e458
5114f07a2308c5b34d9c17d6224940d98e202633
15318 F20101123_AAAEAL graham_s_Page_008.QC.jpg
c00cba48e5e9df675705c34291c97d5c
48c69520d1f29b33a8077e5254adcc561f657db8
352 F20101123_AAADVF graham_s_Page_130.pro
17436fc9cbe7271e0503787d6ce444f4
ae7568ce153e60f2d4897679f749bca2bac89341
8609 F20101123_AAAEBB graham_s_Page_018thm.jpg
b6f90802f9d99af2e37ab1b41402b542
c6e60f79dd1443c5484cd1b3c489948421302dc3
46250 F20101123_AAADUS graham_s_Page_111.pro
f80c319daf779c00b68809d9f23f7efd
29e3ecd7c406041c0e4be09dc83b05a8fc5867bf
7173 F20101123_AAAEAM graham_s_Page_009thm.jpg
6e025a4ed7eff9af251c77aa4d0e31e6
908cdd9ddfe4c00ba850facf1c6a3b0c6322ac72
46877 F20101123_AAADVG graham_s_Page_131.pro
2aaf3fda46197cc8c43b80b4fa5ad821
5c3c0d80f16bce9d098fada601585ab5a76ad845
30249 F20101123_AAAEAN graham_s_Page_009.QC.jpg
421d16a85c9b43c68903b88fa97f2ac1
798c8ff9068e4389a7ff0ebe64a310b06a1b32bb
50804 F20101123_AAADVH graham_s_Page_132.pro
70155497b6915d474860bec11d5dc24a
c3364ba370054beab0a2f37f6409f3e66b5a8da2
47386 F20101123_AAADUT graham_s_Page_112.pro
c60cfa7eeb81d7197be4f007f7fdc269
5f1ea9cc388dcdb8f1f677dad022e0a428e6ff7e
35851 F20101123_AAAEBC graham_s_Page_018.QC.jpg
791628f9a1d12ae0e24968d165be202a
66906a8d43dd83287048c57f1e8be6e79c1d201e
8913 F20101123_AAAEAO graham_s_Page_010thm.jpg
e07b364fe9afb97bfad889991de7f788
34a875274c4fd3ea97ecf6eba2895daf1cc80c46
52938 F20101123_AAADVI graham_s_Page_135.pro
7a0f58589ae5df9d8f9c01a485c13fc1
a79a8347955a70b4b569d5af596edeb893d3bcf7
48061 F20101123_AAADUU graham_s_Page_113.pro
f4a09a7f648b6ee847c23ccdb062d2f7
d1dea06052a52c19238c1a33662808a763a5c281
7897 F20101123_AAAEBD graham_s_Page_019thm.jpg
6131db77a89cb4d9c65d9c5227cb8ce1
2cbf072f0d1d578f53cf686b6b52d0ae8a4e83d1
38355 F20101123_AAAEAP graham_s_Page_010.QC.jpg
a89e35b77a2769c12e51b2eec477be29
47693e0fea87daf284c3dd470c4a8ecf0554b471
3063 F20101123_AAADVJ graham_s_Page_136.pro
23e11f207250e187464a309310cbdf60
0f158ca879d70aeb0925e7fb4fa80345f4e32629
46834 F20101123_AAADUV graham_s_Page_114.pro
2215c1003aadffe35c7e35efbb64f5f1
538f3ec64139cf439f637bc07b7834fd751ca50e
31268 F20101123_AAAEBE graham_s_Page_019.QC.jpg
f1621f0c947b9d2b1ed10a4684c660d7
81ba558b4ab9c0e0c168765804e867d49a177288
5669 F20101123_AAAEAQ graham_s_Page_011thm.jpg
30b62e02300704e1be2dc0f204a370c2
ed29f4c66754113879878815fc3980e7e28577a7
52753 F20101123_AAADVK graham_s_Page_137.pro
990aa8f16b25b20452cf5c536f953752
9898c0880bfe49b38535365a75c998804c5e15ec
6577 F20101123_AAAEBF graham_s_Page_020thm.jpg
bb0fed21630d9ba95f812a784df99716
3cc8e74d7b2541fea04eebc81c8444f7630021a8
23020 F20101123_AAAEAR graham_s_Page_011.QC.jpg
bcd764249f933de144bed19a51433689
25faa3264b2128afad9d06142ae818bc5c2ac492
63805 F20101123_AAADVL graham_s_Page_138.pro
33f886e6e81e34dab2233c3814e5a9f5
31ff56e3a33a8d4bf662dd07b4aafa433229bf20
45411 F20101123_AAADUW graham_s_Page_116.pro
f8da9bfa34e5a5ccc99fd66b2cd97beb
a937021d92f1c4acf55ec508b32ccf445511aa9a
25147 F20101123_AAAEBG graham_s_Page_020.QC.jpg
f914b4125c33b6e50dc4488bf4020bb5
0ec2b5e555f6b00d64108bf11a62523917e7f7bc
1791 F20101123_AAADWA graham_s_Page_016.txt
5b88603db6355f7780a4bfb42e0aee8f
5408b35ef3668fc2e811d870508b904a0916933f
7232 F20101123_AAAEAS graham_s_Page_012thm.jpg
c6d0666e07599d308b350ebc2a71db10
330fd80054a6e8aaac85646814557b091ca1fe5c
62951 F20101123_AAADVM graham_s_Page_139.pro
036fb8a3e964d10f9eed9c2245a15806
493e0b4859dba462b731bbef8772fcd8ecee57ad
50417 F20101123_AAADUX graham_s_Page_119.pro
4cd1cd4fed5d6520b9326aa11e23dddf
c05979ade5a25826b1eca80a5391930c462e631d
7370 F20101123_AAAEBH graham_s_Page_021thm.jpg
213f0790c08e422ddf9833e402f3cc67
a9a63bdcfc38910cbe698367cfc7dddaa8db1c56
1798 F20101123_AAADWB graham_s_Page_017.txt
d4e865ec2a948d3f59d2899868f5ec99
631a14ba999c907a8c884f2b6bb80252b0b4846b
30275 F20101123_AAAEAT graham_s_Page_012.QC.jpg
8bab8d8a440c379a1fd5bf8d3f1188eb
3acd5e1a204f9f71c91b8f23f2744723ce73fc5b
59188 F20101123_AAADVN graham_s_Page_140.pro
4c4bdd5ccb465941c72455da348ed9c0
bfd086bb5eac3f65898d2b7a72a4babef5b4b34c
48286 F20101123_AAADUY graham_s_Page_121.pro
956cb7a13e1fc490accb969923be3675
4dedb0e9a77f02db1a5013b4f2cfabcbdf2cc608
28941 F20101123_AAAEBI graham_s_Page_021.QC.jpg
dd70951880a24f3028ccc73e7a4a7e59
f44456e005fcea7043c825b99fe5ac6440b85a19
1878 F20101123_AAADWC graham_s_Page_019.txt
042710d23f672880764a2c7106e352cb
1ac2737302f36a73c5b58d16bf760db7a7858538
4745 F20101123_AAAEAU graham_s_Page_013thm.jpg
83d531e5cc40e1406551e5e85cee3e7e
69a187b3e0b2897ca7713b50e2ec9af27dacf96f
115 F20101123_AAADVO graham_s_Page_002.txt
1edf074e34d73ea369ee9e5cfae75d93
1a4e872afe148e0e8a3898c1d6a7c7364684471f
46137 F20101123_AAADUZ graham_s_Page_123.pro
71a14671b18182eaa02529dda21e6523
0efddced4051cd7785c4749bcf8623d04323d196
7542 F20101123_AAAEBJ graham_s_Page_022thm.jpg
73b7b7ac9bf40cfff8df1e0d17681837
a0051a37f2b8ed8fab38e7b67d0740a88d953979
1531 F20101123_AAADWD graham_s_Page_020.txt
02749cc69daa403c36a3dfea93cb8c28
53bdf2adab0775a7125aedde67cbd818b51908f7
18887 F20101123_AAAEAV graham_s_Page_013.QC.jpg
47624bc1e9ec50756dd38363be2e8227
9b27530c0f19f9535b3edb5b75cef71e4c94eb3f
76 F20101123_AAADVP graham_s_Page_003.txt
ab622ac7b9d349e6d423893bb7eff9a0
dbd9251fc608581394da7b44c01b24a4dd2ec52e
28674 F20101123_AAAEBK graham_s_Page_022.QC.jpg
7a4471c3c3c405d4114b9675c289bcd2
050c0404baa221a1a21d6d0ca6ad5d847eb2c8cc
6326 F20101123_AAAEAW graham_s_Page_014thm.jpg
df043b6ad21134609106859089fa6a59
50322217a06105843f2ca30bb3f4bfaceecd13ec
1765 F20101123_AAADWE graham_s_Page_022.txt
1df3d2946e8f4844b7286371b4c6bac4
1a7640ca7d54276a57a9b90bb09c9e73cfeef55b
775 F20101123_AAADVQ graham_s_Page_004.txt
d3817f42f2b50bc1b5ce8e3b40a4826b
7bb32623cb62cc272317d726d382a2e2689f9c26
32844 F20101123_AAAECA graham_s_Page_032.QC.jpg
88f2cb408c31b551e6e2630138757cd1
c346a8e440aa115879d2cd2e8ee9e15c026cd652
4847 F20101123_AAAEBL graham_s_Page_023thm.jpg
736d55e6dbcefd7758023e8c0fb00ca6
33b8c938234ded7f427772049414a6d7ca03945a
24046 F20101123_AAAEAX graham_s_Page_014.QC.jpg
18426e6da8f0aed87937f6f1681ab858
04f7d3183dfdd35c83d4974c43fdd939bbb7074f
1089 F20101123_AAADWF graham_s_Page_023.txt
58a8e81504c38e75049ea3cf4de5cce6
8f262543e82291a67b1613a7e789cb5b13aff326
3120 F20101123_AAADVR graham_s_Page_005.txt
f95849869ded483e8056cbafe3f8e517
ce661b05328cf2ccb64d007dbee1d7edb4473ef6
8270 F20101123_AAAECB graham_s_Page_033thm.jpg
9bc414a4fc0b1530ad1f4691ee541504
f04123a465fadcf5d3407d0fb5af4e27cad7992b
19225 F20101123_AAAEBM graham_s_Page_023.QC.jpg
5bb0a8904cf547a5ffb3cac528684b54
5486c4b0ec4baf9bdf12be640ab384117c1037be
7004 F20101123_AAAEAY graham_s_Page_016thm.jpg
daa1a2295aceb9c6b1102179a882738e
5eb2a6f256f873e0661fac338b7e1c4018163224
1779 F20101123_AAADWG graham_s_Page_024.txt
c547efc3a48b65d13d45416de56f9d31
2590eae54850e15aef02ee9521642e1c8f7907fa
4506 F20101123_AAADVS graham_s_Page_007.txt
816eb1c1cfcb09650adee7703c7c435f
2681d2b5072bb35b2cd527131f7e817b36e2e58d
32917 F20101123_AAAECC graham_s_Page_033.QC.jpg
1f5fd69fc9f4b60aa504c99fbe69ee13
11ef62fc2ac074a690f23f12dbd31c2d7553c345
7186 F20101123_AAAEBN graham_s_Page_024thm.jpg
c64d5008ac7103ef2c6d8dc22fd4c771
3844480353c3b15e84e0c7695dd47b37aee9121a
28227 F20101123_AAAEAZ graham_s_Page_016.QC.jpg
6ec99d05a99ad5ea634d44be526571e6
a4d0a89a363b2361855fa0a4d1249766025c6de3
2486 F20101123_AAADWH graham_s_Page_026.txt
89c23e0930ddf935ec11aff15236a82b
735a6a19c78a8df2ab081e2b215e684ad072f8b3
1515 F20101123_AAADVT graham_s_Page_008.txt
9224e84ef8e800bac8152cf3554646ec
a9a1d7e56d749c7a936bc263dcc3d65146029c7d
8098 F20101123_AAAEBO graham_s_Page_025thm.jpg
92eca1a75ecc2b290b32e495e82bf430
3adfe3076991b6f253e7263a6a73129366156990
2112 F20101123_AAADWI graham_s_Page_027.txt
6fc3d6a9dc149b3c9dbdf6e6ce029a01
64f8cc77da34cf58bc235d3f802faf43bc1a5ce7
2181 F20101123_AAADVU graham_s_Page_009.txt
f9b9db35e86add614beea6555bb1ec43
a6bce9a92715e878118a022781acc35dc0afbefc
6773 F20101123_AAAECD graham_s_Page_034thm.jpg
b9c7e99bedd604f7b8a5f7e20c5f84dc
d049dd091cbd4b9fb0e9929ef41ef5b12be0dcde
31531 F20101123_AAAEBP graham_s_Page_025.QC.jpg
452deb49de1106643ef15c33bde19e33
1c61777881f91d027e2d46e38872f34de009dea2
1976 F20101123_AAADWJ graham_s_Page_029.txt
fbb62d1357b59489e05f60fa4f165d12
0206f7ba4ae923137363edab2fea41572be63636
2556 F20101123_AAADVV graham_s_Page_010.txt
ed14bd5508f083f052d9c8c74974e213
7398e319466d40be4297d28445335ab928fe33dd
24255 F20101123_AAAECE graham_s_Page_034.QC.jpg
d38afbc902141678310df9836b4905ed
4b287b7cfabf5d0f229e73b0d138b81562f4fdfd
8012 F20101123_AAAEBQ graham_s_Page_026thm.jpg
b38a3051cb7c087f621525f876b21a77
fc3e1ae92f58607a5c0b87cbb3729faf0707966d
2012 F20101123_AAADWK graham_s_Page_030.txt
8bb50f5be1156ab3883a814838c22b57
573623cb46be6dd9e96483f14f7aec0744b75a0e
1313 F20101123_AAADVW graham_s_Page_011.txt
04e44e0541d573d7842b35bf8321cc7e
60c2113b27d4f2a846939c2dfa867d43df2f0038
8143 F20101123_AAAECF graham_s_Page_035thm.jpg
fe1d38c4342938f50e4a1f552e6b2128
66eff751c3b49c8a77b3253e20a6fa7d134bdc63
31075 F20101123_AAAEBR graham_s_Page_026.QC.jpg
b652dd2fcf91a67e22ed879700c2f18a
20176da34310f8625786715b3699801593e34148
2189 F20101123_AAADWL graham_s_Page_031.txt
0c675c4ae4646c0b75b0bfea8e956ba6
9bd9376582888c30fab8afbc18120bc7b6d4af8d
32341 F20101123_AAAECG graham_s_Page_035.QC.jpg
b554e8dbf682e980a78b1eccc9dc31a1
b57f7a798d33ae86075536b7808c4b1a6dddca27
8386 F20101123_AAAEBS graham_s_Page_027thm.jpg
46ecd54abc68ad0eb038ced7f1cded14
4bb76e927bfdbe645d7cad8a1f6127f55b4200f4
2055 F20101123_AAADWM graham_s_Page_032.txt
1b06820aed664a976a8f4574099b9fa3
84c601b6dc989d6e5007893593937db32e4ced2c
2139 F20101123_AAADVX graham_s_Page_012.txt
cb3f6db5a073f88c21b777faf4820f46
2d6945fe12c95b9ff9729e4af1a71490a9f7369f
2403 F20101123_AAADXA graham_s_Page_052.txt
767b4f8b023354f323ea1c053540e42b
bc7dd4b261860b120340184ccb82bb03e3cc1726
2172 F20101123_AAAECH graham_s_Page_036thm.jpg
492f737f622ab6f7a02a54b779572ab7
10cb8a60b065a955001b6ee30bf2d8f132ad8304
35401 F20101123_AAAEBT graham_s_Page_027.QC.jpg
86e555e194796286fe946b22ef17b881
62702fd8678c512e00378b305129371c1c7819be
2002 F20101123_AAADWN graham_s_Page_033.txt
6a1075f00787703ee1073ceb6791b6fe
52adc4d4f2dafa1d3ede5e989a5d9dc92a46d42a
1618 F20101123_AAADVY graham_s_Page_014.txt
c4190be5c9f9f7950ffcc4c50e60f467
1618528599a1dbc1e7edc3bab415abd80a488bfa
2329 F20101123_AAADXB graham_s_Page_053.txt
19b61cf0b6940c6b48350049bfa3b340
f7766771b04b585dfa58c66f1c611c14376f9880
6742 F20101123_AAAECI graham_s_Page_037thm.jpg
8acb505f1fbd7e40c3bdfed8e7f00663
5c4f1ad1a72cd94fa6cb769a7ee275e1e97080ff
8369 F20101123_AAAEBU graham_s_Page_028thm.jpg
b089527f53908aaeca65b3338d6077f6
8f2b9d465aa88bcfc2700bc42aa64fa526231dc0
485 F20101123_AAADWO graham_s_Page_036.txt
4a7c2c75c874edc277284235f8772ece
2856ce325d8e1ce2e9a17f6d7ed6c155fea1e438
1300 F20101123_AAADVZ graham_s_Page_015.txt
4ee543973c9c78722e90ba338beb0dc2
e1094cc09c533df29ef059c7e54f9e18148dd87d
2447 F20101123_AAADXC graham_s_Page_054.txt
03a9f972d926e75bb97b219ba7d5562e
0e4ab2aef31e79d8b901fcf21ab02fb3736e5434
23769 F20101123_AAAECJ graham_s_Page_037.QC.jpg
5e167457002d8451ffe8f8f9e9cffeda
3816fa5fc16bc14e53fc56b97a62a68deb6bbe3c
33588 F20101123_AAAEBV graham_s_Page_028.QC.jpg
d8a717e6aa236fce119c2d1d5a217089
ca32b44eecb1bdd092e98cd6a08b2ce08b3e7bae
1584 F20101123_AAADWP graham_s_Page_037.txt
ef77cb707fdf4855fc3051a2a967b14d
73a647b593da80a714195e5257ce1ce5ae273338
2542 F20101123_AAADXD graham_s_Page_057.txt
f277279f4f548487df9bfd67210b8e1e
96cbf9cabdade99a6cf2c8bbc23d14bd56830788
33815 F20101123_AAAECK graham_s_Page_038.QC.jpg
94a1be33d3a76006ae26f396f0f45721
70d1ad6725df0a4b1d0825f21ad7829e1d3e7ca7
8352 F20101123_AAAEBW graham_s_Page_030thm.jpg
2edbbd532ab724275cae6177d4355770
5fbef4475641868a813da9a37a4be8222b31dd3a
1905 F20101123_AAADWQ graham_s_Page_039.txt
a4c8d1d0e0c0c1107f667a4d5b9f0ac2
f26cd456b293f5b8d6de33c8277445744d0296f3
2831 F20101123_AAADXE graham_s_Page_058.txt
5e54a703d9bc216a5835faa455626e53
c13aeb4cadcf216d4becc1b34bed5b33d3288ccd
8469 F20101123_AAAEDA graham_s_Page_050thm.jpg
719ff751505821ab4a984b886441fd3f
4583e81a7be1205353fc74e583670079584d5657
7775 F20101123_AAAECL graham_s_Page_039thm.jpg
b4611bdc9f02d144af6eebb4975fec21
9eb4827caefb0c200dc9a930e8770d2142a1b34b
8487 F20101123_AAAEBX graham_s_Page_031thm.jpg
8015ab07eeebc13e0bead39141283c64
77971901c053ca59f751db5cfdb5704845e26677
2159 F20101123_AAADWR graham_s_Page_040.txt
1c089252f700b899defb1b02c61430af
485dc2f8ca032882140a10dd406d664e064267ed
1982 F20101123_AAADXF graham_s_Page_061.txt
328f3fa658593ff59b6480a48a0199b5
830b4f753395a9d9a5976d87ebbb127c9bc94814
34307 F20101123_AAAEDB graham_s_Page_050.QC.jpg
63d141015a1d7fb11fa908b9c7d84398
f8a95d72f9441e16419ce5b8017360163803a287
31164 F20101123_AAAECM graham_s_Page_039.QC.jpg
f57b518245fc819fc425cb775c49001c
b777f0d549e804afe98e821e612905b055a56f6a
33916 F20101123_AAAEBY graham_s_Page_031.QC.jpg
d9f3bbe2a5b5cb825a86ff495b866035
3672d98751257d139214491cbd9fd1d4a055bc03
2282 F20101123_AAADWS graham_s_Page_041.txt
d1be9374c67ee39ddecdecedb4838fbd
cc689c6ffc7d4ced90973acd67eb1895353ca93a
3857 F20101123_AAADXG graham_s_Page_062.txt
f868dce7f8b8c2685b3f23e74c1868ae
1289b320922720420549b0119095bfff4035bbd0
7319 F20101123_AAAEDC graham_s_Page_051.QC.jpg
c8fbed54f2f377644a876e78c5f7f480
e719f3b656ea178a34603a4c0eb14ff5d43c3e27
7828 F20101123_AAAECN graham_s_Page_040thm.jpg
7d1062454bb6ddc324daa6d7c72041db
2f08a10b46abb44240c414a8b9031c58e8a6c0c9
8065 F20101123_AAAEBZ graham_s_Page_032thm.jpg
8d90b5980d836272fafe1abd74571b27
03f06f47acb9e18aa6d760d4d2074dc7faf68d0e
2146 F20101123_AAADWT graham_s_Page_042.txt
28812689735fc5abdda08d7abe3e016a
bbbbc1166512ef934de41a30ad1820289eef0616
63317 F20101123_AAADAA graham_s_Page_055.pro
43c5450f047828bbb4e466b715c0969c
e5f4fcd17441669db802f52ade8fb5ad543b43a5
1312 F20101123_AAADXH graham_s_Page_064.txt
f09c8197107f52218508bed0ffee65a0
655e8209917d4993fb468a0c2e4bb08fcc2022ab
6364 F20101123_AAAEDD graham_s_Page_052thm.jpg
18b65bb91ee5f651e0034414d8adb1b7
907784f82005f0b58ef62a95b709a3b7130dc0de
30875 F20101123_AAAECO graham_s_Page_040.QC.jpg
f5f20b3563b429d8286127df309b064b
4518be578b694174aebdd87f270c784de5cb03df
1931 F20101123_AAADWU graham_s_Page_043.txt
d3de4642763164f67a33e5592ab9a522
a0161244afafffea2c0d188b1d70a740b33fc1e8
83701 F20101123_AAADAB graham_s_Page_057.jpg
f5703523c1180c01f8998ee0909299b5
cbfe4ca8f94c4b101b7dca530e042b13482e4b25
1977 F20101123_AAADXI graham_s_Page_066.txt
0d32843bc8c08c784e010f1375257bb6
2b1f92019675a56e248b2b0482cdfd62c4dd7ae3
8299 F20101123_AAAECP graham_s_Page_041thm.jpg
6edb03167a7c3d03cb026fd5e308a2f7
c6864c02ff73361cd08a0a8205fb35277107d3e1
1700 F20101123_AAADWV graham_s_Page_044.txt
d5f758735ddd8610d0c7d229280c5771
7deaf4500cae2041f9062748ea163420fc08e647
2248 F20101123_AAADXJ graham_s_Page_067.txt
c8d872ab3bd3898197a1961e9a7257ea
6c9ef44141fa687f7e4353727c1cdba5ca17087c
22750 F20101123_AAAEDE graham_s_Page_052.QC.jpg
15a2733421ffd033935d44b8528423e6
130a54f984cb2e5d612ed7009cc2254285828996
7221 F20101123_AAAECQ graham_s_Page_042thm.jpg
96b5e83e0dddc6b676f91a8664ab2fab
8431bab1e1a0aecd73ab09791f47cb2a8e029204
F20101123_AAADWW graham_s_Page_046.txt
c04dc9c4cdd227b2933e3528f7a28f79
c5ad1bb93ce3a612f159606686a3ccdad3022628
33847 F20101123_AAADAC graham_s_Page_119.QC.jpg
0e23454c69b3338bd63439b2b12b9226
60ae526eeefdbfd9e12e6fdc9f5570cdb7bdab93
1855 F20101123_AAADXK graham_s_Page_068.txt
b215331cf9491696b521c0067024ecf2
d1e11a567188cafa57395c0cbff57fbaf7b7d073
7865 F20101123_AAAEDF graham_s_Page_053thm.jpg
c09c77bccf60411694f190d3a5b389df
969757556aaba664a23cfed71a8ea808b08d66c0
26921 F20101123_AAAECR graham_s_Page_042.QC.jpg
b5a8d03f87a19fb3e1762d5e52442b63
31ba72756589efd84c12bc9c2352c7b54b25398c
1913 F20101123_AAADWX graham_s_Page_047.txt
1d64c6622ed219a41b1ddb68f153e598
5fb6c604bae31a08e39b2b6c294047a0561843d8
108000 F20101123_AAADAD graham_s_Page_007.pro
663b599c6d272330f575cdd6774793a4
896c3fc9111b8ca77004121af45ee9beac5fbdcd
1793 F20101123_AAADXL graham_s_Page_069.txt
949fcd6035912bae6a3f08cdb46315bb
2b0b629e3be0229a0d5ae8b7651991208dcfc062
30924 F20101123_AAAEDG graham_s_Page_053.QC.jpg
28e6961e3fe90c42e58086cb0807f569
fa68ec0355e3c6b397b82feafc2310895d12207f
7556 F20101123_AAAECS graham_s_Page_044thm.jpg
6a107c6d0bd622c1aaf72013f1153c7f
6220e26d1c802f100989a51def64c1ad3458e27d
87545 F20101123_AAADAE graham_s_Page_087.jpg
5d0fea93fe4b462183d6aacb68838ee6
6cf3f1ff50ca3a9da3cb430676ce44d3784202de
2292 F20101123_AAADYA graham_s_Page_088.txt
578657ecfa8fe0ba2bae5a23a38d87ee
e7f2b0ba31911db68629aa015d5466ebc0ebc10e
1649 F20101123_AAADXM graham_s_Page_070.txt
f5e20e65dec11f2545ae340cd0155387
32471e4c380e218757014a001b055edf6c1c697c
7763 F20101123_AAAEDH graham_s_Page_054thm.jpg
8577c8e038b917519eac689dc847bf5b
167e9a0b4c1d9dfac5fe84da1ee40bc2d1acd778
8146 F20101123_AAAECT graham_s_Page_045thm.jpg
7c661a4b19f256d2f03b5304922e2d72
c6c35b49304982bd0f6d9161af2b6be8e2b5c94e
F20101123_AAADWY graham_s_Page_049.txt
a26564d9ffa3180c8bcf52d77eeb2bf2
1e6d364dc009bc0989a189bf9ef2e0b15439e2ae
97628 F20101123_AAADAF graham_s_Page_059.jpg
cc57decf1f3e1913c8f902c441f5224f
c5c47489a96b164fcc8fd513bc19b253e9958e7c
2253 F20101123_AAADYB graham_s_Page_089.txt
546a95932f5fa8f4c2e7a9e257974fda
9a14dbc15459970fe42fae3d036d41243aae146e
1763 F20101123_AAADXN graham_s_Page_072.txt
a2ac08486597cf86f6182e54a78dac93
6f037ce71b9d19a054cb499ca29ccd8e6925b81f
30497 F20101123_AAAEDI graham_s_Page_054.QC.jpg
37552c5ddbf7cd60af4bebdcacef4b39
ae977d3fc7502449cf7931ee02d0abbcb7872d27
31211 F20101123_AAAECU graham_s_Page_045.QC.jpg
220821c8008382901c2c5841d7aec80f
6f6d7ecf3d81a3e0a2eee9deb431cdda51929b89
430 F20101123_AAADWZ graham_s_Page_051.txt
62da915f2412eb010b9ce39fd7adbced
f1a5c239a4364c7b5a677edcb789d897ff98e2d1
20977 F20101123_AAADAG graham_s_Page_115.pro
887bdab8368cefcca55305f246b518f2
9ae8975fd28d548d825a0c57253575e6dfa9c392
1064 F20101123_AAADYC graham_s_Page_090.txt
5a00cfd95173daf632c93c0d33f79b28
681580d938254343430b8b89083870d0a5765626
1348 F20101123_AAADXO graham_s_Page_073.txt
084d3b40416413c607aba582b309e5e5
3f0ddc259cdc89825a795075063612d5a32030b5
7042 F20101123_AAAEDJ graham_s_Page_055thm.jpg
4497ad7145c118a1084c564a84b63d51
b9b5e0d273887ddb979dec983cecbbb192fc0524
8288 F20101123_AAAECV graham_s_Page_046thm.jpg
dba02bfadf297471a0dcdb713764bc8c
10d974392c30fcfb561bbf4629a85cd635af813c
6808 F20101123_AAADAH graham_s_Page_087thm.jpg
4e27a2e83f488a9c12f2bce4c9a35b5a
9e42942d4010bf89fda7aabd18c1cd3c45695ffe
2126 F20101123_AAADYD graham_s_Page_092.txt
e6ebcefa2f1c1b98b2bbfc742c2ad537
810a4edbbcabcf8a9e86afbfdf370e4bafaa9fc5
1903 F20101123_AAADXP graham_s_Page_076.txt
6afda8d0c0bc4ae3d80c4310b007f07c
30fe13b547a570698367663b027553abb85a4622
27283 F20101123_AAAEDK graham_s_Page_055.QC.jpg
3cd7daa851ca8a2d379f1791c8f21687
863c9b791e75f03d3458faaa426c9695109fb0e0
32746 F20101123_AAAECW graham_s_Page_046.QC.jpg
54602fd317465d00b7bad8f94b47cf60
0181fd3f51e2656834763ce9cdd7c28e2fe2697f
F20101123_AAADAI graham_s_Page_004.tif
715ee2005e462211afd80a665d7c505c
6a6a6e1d33cc3da988407670485c00d1deb040d4
2037 F20101123_AAADYE graham_s_Page_093.txt
7bfaa6de3ac92a62fdbad047ea69d26d
0f6ba08a5217fb202a383da749155c90897f69b5
1883 F20101123_AAADXQ graham_s_Page_077.txt
bf51189430696d86d90c73ccc006ae7f
e13a1ad54904ca3cf5ae98cc0df98d7fed510e5f
7938 F20101123_AAAEEA graham_s_Page_065thm.jpg
81c2a83275524a0fc32df5f34a241e78
594faa9404b0d2b9310e6046c5b4c61e25305ef5
7750 F20101123_AAAEDL graham_s_Page_056thm.jpg
5943abd7f63dbd6645b5e54c5612a312
58be1b8d4689daf405b71831e905c20471fe87ba
29274 F20101123_AAAECX graham_s_Page_047.QC.jpg
2967b034df908a281669fa2cd05e9a4a
e0d653f8af22d6bbefbca9bbce699ff359c5eb2a
48473 F20101123_AAADAJ graham_s_Page_052.pro
962536e404fe33b5dda56423f844937d
546effd8a47e7596767a70067439653804a6d28d
1269 F20101123_AAADYF graham_s_Page_094.txt
d4a4abab901259b0bb5982f7e42c13f9
09729b3120c1d665dc5500b2fd594ee426c0897e
2109 F20101123_AAADXR graham_s_Page_078.txt
6583adb38e276f17a741073fc84a7d73
b6f854058ee73a3f8f0277c65ec766705b2741f3
30104 F20101123_AAAEEB graham_s_Page_065.QC.jpg
edb4e0fe447e9aaa4a8efd078503882c
68ee236edfe37a0aa33810c1716eaa3d8a556a33
7303 F20101123_AAAEDM graham_s_Page_057thm.jpg
91737c06b8924971256e2ca31d40a810
0dd62745f2416f5b32c218d889f2f15dfd114044
7916 F20101123_AAAECY graham_s_Page_048thm.jpg
e75cfa680346e91636e3104920c470cf
295b06f3b82a4e0acf5ace0eb32f0de7a3d1534b
24711 F20101123_AAADAK graham_s_Page_073.QC.jpg
42d6925ed5a45d015a360509ba6d1e77
d24579231f433ec2b3fef59ab14fe408bb81491a
1787 F20101123_AAADYG graham_s_Page_095.txt
622f51500092f444b55355cb8776fb6d
aa792b78ff55a29ac07807c7da6fd6ba478c39ed
2048 F20101123_AAADXS graham_s_Page_079.txt
64431dd0439070d69bdd595daf063aff
0f6c8c391992655bedd09f5cf374fb7c354d9540
7939 F20101123_AAAEEC graham_s_Page_066thm.jpg
ea17bd9541ce553e7b0a94b10f962a34
1699fed472664f5bda02ef818c05f571d4d9e424
26783 F20101123_AAAEDN graham_s_Page_057.QC.jpg
20941cef0888f0d03ac0dcd045f55a64
76db34709dcb2217b1b9064485724968432e40e5
32058 F20101123_AAAECZ graham_s_Page_048.QC.jpg
4ede7a72268139c65a36dbb5baf3d7b4
0f2795582f0813d1e2693605149c3bea88967b36
941919 F20101123_AAADAL graham_s_Page_064.jp2
cb35caabe7bc3edde5ad0932cc1ebca7
a1b205ce0f7915c8354dcd33f9124fa46b838bb2
2105 F20101123_AAADYH graham_s_Page_096.txt
2ecf27de27e84ba73c17ca4a047ea120
fee4d22c8503e7aa9745feb751c3bed68bb5e726
1929 F20101123_AAADXT graham_s_Page_080.txt
4e837ab884d3ced8892f484bda7aee0a
58fb9a8c1427b67f2843e65b914ae443681d001f
33842 F20101123_AAADBA graham_s_Page_006.QC.jpg
a0af8e448d95ab7b1918b22d899c4495
4ded605a0b82b9800fc4226b7b09e5c25fc1fec6
32508 F20101123_AAAEED graham_s_Page_066.QC.jpg
e92d97e8882f8183d0b5a10735cd5962
94d78788c8fad107fd9f080bee95ebc419b08e5f
29728 F20101123_AAAEDO graham_s_Page_058.QC.jpg
7d8d488500004183c7a9511d83229257
c63fa2928c43434b80ba9aebc2a674356843a6ff
1993 F20101123_AAADAM graham_s_Page_050.txt
1920baaa69646f96106c96c074f99660
007607c51944fbdfff86be024cdc67c3f7894b3e
1428 F20101123_AAADYI graham_s_Page_097.txt
8b4ed4f1266249c5ac36c383030e5816
5bc37ed09cb4081ff20aa7217d1c7188cbc8fa62
1633 F20101123_AAADXU graham_s_Page_081.txt
b345e3f3ef6eb6cee0f7c976c2fb254a
1ac98e566ddcc7fed81c3f0d7ab498952a7e8025
F20101123_AAADBB graham_s_Page_032.tif
7a7cf6f739abfcc7ca793b729ecd9d6f
5dab3c9c36f4bb0f41d22482ab3d1e4eabcc9778
7529 F20101123_AAAEEE graham_s_Page_067thm.jpg
5dc4fab343368bb6513b3d150143bb55
593d23f7bfb7d7b5425cd3743271db64f0fd721f
8416 F20101123_AAAEDP graham_s_Page_059thm.jpg
bb9165514c33cdee8177a50338b10123
9aa4a19253e1fa20ab9eaf3433f84d56e26a5e35
73327 F20101123_AAADAN graham_s_Page_037.jpg
88874e575c5e70ea59e0ca36071c5742
8e45f76e2e117c5fac20cc02ff54773a5ed55dec
1714 F20101123_AAADYJ graham_s_Page_098.txt
d636baa88667dca5036cfd227c519377
f9431a5e853f119e0f3a2feb5f8af8b56c65b08e
F20101123_AAADXV graham_s_Page_082.txt
826e3c10c4f03ff312979b68b46d6926
cef6f343adb86bb997475da337518ecbc80d98d6
1941 F20101123_AAADBC graham_s_Page_141thm.jpg
fa11187ca10906fc4f7f233dc2fbe8cf
61262d8746fb529dd3cf9179f47966688142886d
32427 F20101123_AAAEDQ graham_s_Page_059.QC.jpg
0f174e57f3fd42a5a4fd66ed24379ff5
c9cef387cd9ec8dedb49bb3b6a2642daf114c216
102217 F20101123_AAADAO graham_s_Page_012.jpg
654de0ab5814a9187caa352af4a1d48c
3c38321f376147d03858a4540621fa5f370cedce
1935 F20101123_AAADYK graham_s_Page_099.txt
e32576f008e18a8acf461e03903eed7b
5ad9df12c60810731b9e58337233b6852e8ce52e
2273 F20101123_AAADXW graham_s_Page_083.txt
f0c87c56edce29ef6531b5bc231207ef
5d77b374573f6a9ec252459709b6adbe5d9c7bd3
7751 F20101123_AAAEEF graham_s_Page_068thm.jpg
65dc2ed556414dbca5cc16b7589fa7f2
b42d31327afb97f393a03264ef7c14ffb9b0cd0b
7759 F20101123_AAAEDR graham_s_Page_060thm.jpg
a6c42d65bffa14def29b16badff9af14
caacb85360212a84162146cb1e54738a694bca12
365 F20101123_AAADAP graham_s_Page_003thm.jpg
36131e805abd57497dc75c1c535949f9
5ecccfdb33b8ddb4cd79d4c3857fb705c207454a
1717 F20101123_AAADYL graham_s_Page_100.txt
53cf12f02d6b5dc7f9bbfaaeb78da3ed
fd97e684eb02cb1c27d814516c310df45f2908f8
1963 F20101123_AAADXX graham_s_Page_084.txt
0da5ba0a3dff26155badaa97c904e370
7bb5521c68d39a3f8bb7ab1dc61e0d161d9c74ea
32927 F20101123_AAADBD graham_s_Page_061.QC.jpg
12a66e8ec042db2f4d827d16ce0d53f2
1fd08009a3ee91e45d9def538b06a102ef955351
6812 F20101123_AAAEEG graham_s_Page_069thm.jpg
a4d4deb95781135d1d4064f69fa499e1
f4eb94a6246e38f5ad16f322d232a2f404d98cfc
28249 F20101123_AAAEDS graham_s_Page_060.QC.jpg
71dad490f504c0b3268091ddbd0e2c5a
3e445ccadc61f4cbe27bb75b0eb99732804a6afe
F20101123_AAADAQ graham_s_Page_055.tif
df9f14a8e4f392eed568786a09a1d710
f237f29c0be2cbe0279b23c3db5a31864604e68d
2022 F20101123_AAADZA graham_s_Page_119.txt
68ec02cf0e7f0cade55a8f466ea8b812
0807657f1f4586534b169e00f91230dad26e11ab
2004 F20101123_AAADYM graham_s_Page_102.txt
f36a80b9b6e4c9ee46fb3da919440f10
cd24ac7b59cab48ad7f12d0322fb5efd9fd02794
1689 F20101123_AAADXY graham_s_Page_085.txt
44c7daea76b00c8080dbf7ce8b68bc74
bcbc989d87f35785717247a67f3e282bc8a133d4
1051978 F20101123_AAADBE graham_s_Page_006.jp2
bfbc000c3e463928cd89d6ae735910a3
1296ac04fde8516482aacaf7215c64d69fd37e62
6641 F20101123_AAAEEH graham_s_Page_070thm.jpg
4fb7542fab237a656b10b0956790cdf6
f01048511de46d60737f168c1db6dbb8f804fdfa
8554 F20101123_AAAEDT graham_s_Page_061thm.jpg
ff0097fd9de261f4034672cd54e24b16
f63ead48f109ead8fdd1a6a6f50b6f629bf60aad
87947 F20101123_AAADAR graham_s_Page_104.jpg
367e5f499cbb578198e821acfa2e99aa
10e43d5dd631ddb35644d9c51eaae5073bb95722
1994 F20101123_AAADZB graham_s_Page_120.txt
a00fd1d19e1e90fe2755bda278b3565c
310c680112b12a420b3b567265ff458bd7bc8b01
1806 F20101123_AAADYN graham_s_Page_103.txt
ceea35d5fa3d8457218e31b0dd29d6b9
afd1d2fce01bd5cdb00836c7b2c0a3fa0cba0a91
27230 F20101123_AAADBF graham_s_Page_068.QC.jpg
71bd7e64f5a4cf7bf31631692aabddb1
419321bbae18fdf2cc04c49cd8ab3d345a4a38cd
26379 F20101123_AAAEEI graham_s_Page_070.QC.jpg
9849e9c400079c1c90aa2aac7ee2d78d
19fb26bba53721c931d024c2d856b59d86a59c72
7051 F20101123_AAAEDU graham_s_Page_062thm.jpg
1bf7afd01b23a06e98081a5c603bd151
6aca13c83f22ce279ed663f69a89e15b9b3eddb7
F20101123_AAADAS graham_s_Page_038.txt
bdcddbd70910353aaa28dfba1243af7b
39cd7d0ba689c05045a84c7793728fee0d148399
1943 F20101123_AAADZC graham_s_Page_121.txt
f1d207b6a7d3efd9b960dec2faaf2ba5
eef5953dc6cf03bdb16a5c439be998b5ab219aed
1750 F20101123_AAADYO graham_s_Page_104.txt
911e2db0a9ae12f273e3e35b259512e9
6cdcef68f8b727cb335732a6b5fa2e855b89a249
1858 F20101123_AAADXZ graham_s_Page_086.txt
c55440288d9e816dfcfffbe42107281a
18d20e9c2dfaf62faea17c0764770d298adc14ab
37137 F20101123_AAADBG graham_s_Page_080.pro
3f3d6817c6f25a98b0ff9df717bbcd1d
b124f38b920f75a3094c48382dd0c0be8929c699
7333 F20101123_AAAEEJ graham_s_Page_071thm.jpg
842691d930d6cc4ef319dcc65cbb12cb
334fd0691cf023eac2d3428b40a318827f87eec2
27830 F20101123_AAAEDV graham_s_Page_062.QC.jpg
0530a4514bb270da47298e26b9e128bb
d962f6195821657f6a09c4f4d4a544f807a5c2f7
35679 F20101123_AAADAT graham_s_Page_097.pro
fca6516164837d0e98d0833bfdf37ee9
fb935704b59f4b1e75aa976e6e9ecc4e1943b13a
F20101123_AAADZD graham_s_Page_122.txt
15c089c6081aa3e4e2c28786a8d47083
bea928059834c73031454e0adba989afc672ecd3
1840 F20101123_AAADYP graham_s_Page_105.txt
971c992a33ed492a089694bc14beea1f
3ec1ec911b2c60cfe4a057ad820a640122b45766
1051969 F20101123_AAADBH graham_s_Page_012.jp2
be3f8177f43a5365df3a7320371d6471
375d52eb3b0e4bd73f1ecb1e7e09f781382e4fc5
29290 F20101123_AAAEEK graham_s_Page_071.QC.jpg
cdc0a484a9d155d817472d65e244ab98
aa8aa9d2c569e7fa5b5e043e661c91a25e0fe7db
8285 F20101123_AAAEDW graham_s_Page_063thm.jpg
e61f1ebcbf6a5844c1c617ef384e1232
5a9cd3a9d8987b9ac030fdef04826d7514eadae1
98048 F20101123_AAADAU graham_s_Page_045.jpg
81cc15b978131f49abe7f2a1afdadf3a
c0d95b5d2d726fc1cfdd14a2424b2ee69cf5e740
1853 F20101123_AAADZE graham_s_Page_123.txt
03ac6a570072420f857a15c0b1cd8640
acca85fddaf02c6a6d735e6a957c10ca4f47ceda
1860 F20101123_AAADYQ graham_s_Page_106.txt
3a973eb31994a311832516f0770121cd
6f9bb269010385b06ba6248f0c2672b6439c3f6c
F20101123_AAADBI graham_s_Page_113.txt
052341045958cc62ecdb538056e49b68
bd9970a3c2df945cde732a4764b633f0f96ddaaf
6861 F20101123_AAAEFA graham_s_Page_082thm.jpg
9a6afb144614375a76210ffa88d2f0cb
cb4098ef51395d119407638c07795b0ca6695598
29324 F20101123_AAAEEL graham_s_Page_072.QC.jpg
76180a41fea52c7197cf25fa181a897d
d80787b954a6e6e8193ea960793be013fbfa0b2f
28401 F20101123_AAAEDX graham_s_Page_063.QC.jpg
73b183d33541871fd95754002aa81e1e
ad50e3ca739082c52b7facb45e87c3c5e987d964
33902 F20101123_AAADAV graham_s_Page_102.QC.jpg
dbffa42c46d4cc134a98168873e75376
5c61af22bd6f8005d2d731dd3d2fb8fd17accfa1
F20101123_AAADZF graham_s_Page_124.txt
06ec4434035343571f49f910d17c1606
1a0cd4ef2172d9852977e3370263b2d4fb824e88
1816 F20101123_AAADYR graham_s_Page_107.txt
327d29185116fadbe83fc5e60984c999
d76eeda6a6734bec3a05c606bc8831afa4e1ddde
2033 F20101123_AAADBJ graham_s_Page_129.txt
2c36d1126f8e60bce5c6bddaff739fa7
a89e6dbacc4520cd011eaae2b647fe72e328ad37
24978 F20101123_AAAEFB graham_s_Page_082.QC.jpg
7159e29598f7395567a631e7e149d7f4
72aefa9bc1a678e11907bced8f082ad597630a64
7033 F20101123_AAAEEM graham_s_Page_073thm.jpg
75157f4b30d9c9b83b73c035015899d4
95de55a5a0c72f431ef49045b55a36e97b045ad6
7254 F20101123_AAAEDY graham_s_Page_064thm.jpg
a32621bae96744d595277e48e74db48d
08f1d5ca3d8ef817f7d5e67af560787fd4ad4fe2
28415 F20101123_AAADAW graham_s_Page_067.QC.jpg
eca75e6d6520b574d7aca8e20b550e29
1938d6816a1b135d816a904f564b5ca13bfc4bc3
1940 F20101123_AAADZG graham_s_Page_126.txt
227da6ef7a149d7fa09bc2a01be12090
03637d2507a330f076e9ef0076deeed539f36623
1927 F20101123_AAADYS graham_s_Page_108.txt
05283f281c44eda47bc01d50d63bc9ab
229456c93c01e86b88bbd14003bbf910cf5d18bc
2408 F20101123_AAADBK graham_s_Page_071.txt
6ecfe6f20b42f12aa8cc8d9d1f2ec437
f6801fc43dc7624381ddbdb17c5bf5948b5dda52
7886 F20101123_AAAEFC graham_s_Page_083thm.jpg
fc3a39d072e68ab74a6d4cead0c1df9b
f9cd7e554cb6f2de8f93df5fb9eb367ae8c0d60c
6885 F20101123_AAAEEN graham_s_Page_074thm.jpg
bf4c5d01cab0d275adb1a6a720089f96
25aa784b70fe73e16c2212a3c1413f153d0e7459
25430 F20101123_AAAEDZ graham_s_Page_064.QC.jpg
37aa5f1f63e88e439d9d3f982fc3eee2
09550ca2edbf186f0058020e5160a42669606a12
6697 F20101123_AAADAX graham_s_Page_084thm.jpg
b7e5710c3714be7f23ec128ea12b41c0
87c2a53dd99f6a00f8b32535052ea5a7a7eef2bc
2089 F20101123_AAADZH graham_s_Page_127.txt
0da6bb4b97d647740fd1c3c3dbad81b2
634fe3ccb6b5031fae87d13191e09fc11aaa5698
1796 F20101123_AAADYT graham_s_Page_109.txt
5eacd12a25d5fbd21c737fc325867050
7ce9975a906db8d8e0fa642a6cbade0e538a5fac
24912 F20101123_AAADCA graham_s_Page_069.QC.jpg
2bb208a98bc6e118ed65a8fca520a635
7a33c9de4ba72ce5816857d6c0ad3eb111dcb32d
46501 F20101123_AAADBL graham_s_Page_105.pro
090c7c783f2bd07633abaa5cb384bbd5
6e289ef6c77a2cbbdfcda9371f732222e9834b4e
31202 F20101123_AAAEFD graham_s_Page_083.QC.jpg
0201bdd5e3b87fd53f094788644463d4
9124a0cd0c706e69186c4a53e620e167f5e3b282
27315 F20101123_AAAEEO graham_s_Page_074.QC.jpg
1c93ed99d111daaca47b10840bfb80c4
4e957b41926fdc5608d3aaf41db2d8710d08ed48
73884 F20101123_AAADAY graham_s_Page_064.jpg
de415c5742860d77a9dba564693b9b56
8ad91345f3a55e56695dee668cab36c04bdae80f
1294 F20101123_AAADZI graham_s_Page_128.txt
c44b4f3e00bc87923d0bfd5f0bf95291
19d034ca5e35b3c4fb5a2d2442d5a5f821ac1164
1886 F20101123_AAADYU graham_s_Page_112.txt
325a627b44e3371f04d53a3a72b1ffe4
77b50f6cee4da33d7b0c6b040391d3efcf48166a
86582 F20101123_AAADCB graham_s_Page_081.jpg
7145d7f1f50a3246fbe111e26045ef26
9059412ba36427db41ed5fcc5d8cbfc2fab94cdb
2006 F20101123_AAADBM graham_s_Page_028.txt
b579c85266ef474f3bcca10c6d36b856
46bdd71a829c1dedcf49a381aef39c4aa13fc987
26125 F20101123_AAAEFE graham_s_Page_084.QC.jpg
767064fca2e189b7af95b47ad751ced1
e07599a5264685febb005c2788ad136e4b81beaf
7584 F20101123_AAAEEP graham_s_Page_075thm.jpg
ca55a8325552f6f1405053cc0de3ea5c
303131d7cd177e7c7b657ca08f33ca2984c46977
101507 F20101123_AAADAZ graham_s_Page_046.jpg
5e71d3643223f3f3e386438b7eb5f751
5fb85dad0d1816ece7cb8ee7a40b6c6000a8f250
14 F20101123_AAADZJ graham_s_Page_130.txt
9b7213eb17d842796437b2a2d0eca2cb
172f0ae3adf627ff0203e18f351502f8e522f009
1863 F20101123_AAADYV graham_s_Page_114.txt
c52f0c33690ba16138249ed94c3c2830
0b19c77b043e54b7f2b55214f501f91b89a6782a
F20101123_AAADCC graham_s_Page_142.tif
15a4e87fcbf876f80f1c9dd17ddb8d9c
67719ff6d7c081d6b4a6c6cb461566fcd6d63bd6
F20101123_AAADBN graham_s_Page_068.tif
77ef39bd29e77c65e02991172020f254
1191cb4cac45eb681fd9cc8f7c618fca584acf3a
7606 F20101123_AAAEFF graham_s_Page_085thm.jpg
a26893a43c77032ccb57ce7a0b67e8e6
c0febcd4afae06026626d2c2a5877a6bce96af64
7559 F20101123_AAAEEQ graham_s_Page_076thm.jpg
a6539fa70a8c76eb8d51ca72cf6d0a0a
68d409acbe7363ae8530ee77bc181c33e747bcdb
F20101123_AAADZK graham_s_Page_131.txt
9009e823c7ad3fb7cdfd89348044ab30
46c5d301ac629e929256abaa7bcd8ab0f350bd2c
849 F20101123_AAADYW graham_s_Page_115.txt
a06a036461c71463299f14ad4eca8deb
f43a08615a9eba532db1e7b32d5879927ce9cc53
F20101123_AAADCD graham_s_Page_051.tif
adf12fe0339d5c04debc5ea0cecec406
eb35aac1f5ac4950b450bce998216a67f7ef9b6d
43318 F20101123_AAADBO graham_s_Page_022.pro
90f9476868df873d8cae5811ed36cde2
1f4f0e90895073a1668247db413c45e13c6392d7
27676 F20101123_AAAEER graham_s_Page_076.QC.jpg
73510274d769c480591904fa22984de5
3ba538db41e231aa8ef6d998cfcc6b163cfdc054
1871 F20101123_AAADZL graham_s_Page_133.txt
31f35b1f607c8fb1f2a4b65b282186a9
c2ea95157d0e269b5dac4029e1f9088eda9b2d0e
1865 F20101123_AAADYX graham_s_Page_116.txt
4b551fdbddcb755b696e1e2e1d44d492
4148099557a080c4a5116c6f7a84ffd5808a3128
7915 F20101123_AAADBP graham_s_Page_072thm.jpg
f301f6ce19b3baebe2e43741d1246324
76687ca889c7c631da632911aff15221d4ce232b
27700 F20101123_AAAEFG graham_s_Page_085.QC.jpg
276bbd0dd7232e2a537c9b8adc8c46e5
58cf3eee26a3a952aefad21ebd9093f977f29a8d
28036 F20101123_AAAEES graham_s_Page_077.QC.jpg
9d10fdbf712002eceb8149f3876d1fb5
220f67ac570c3c3e2ff9517166ae6fe3f4ee5155
1818 F20101123_AAADZM graham_s_Page_134.txt
7033bfe954658198b166eca1236aef67
e2f0922743826534e154b3266418c5f5a9057d8f
1831 F20101123_AAADYY graham_s_Page_117.txt
511a35c2e661a1139043d77be25b1655
3e4296c96acaf4a60a851b7c0549dab610508d56
32451 F20101123_AAADCE graham_s_Page_015.pro
f660bdbe2fed7474ec374dadb28cf149
1743410b6dfd7ac1ab013720d906768293f26e9b
9987 F20101123_AAADBQ graham_s_Page_141.pro
038f10a9a1eec990fa7253260b200608
a84d87228aac4590f5f4d5f247ea54e47bb6d4fa
7284 F20101123_AAAEFH graham_s_Page_086thm.jpg
097c22a7d5c30d884fe771743722c120
1dd7a1451a2564c7bd28bb375318a9ddac699dad
6706 F20101123_AAAEET graham_s_Page_078thm.jpg
c663ebd5b1af77e0e7f58df2aabf6787
f4d2628f6e6761f4f7367d4b5b0aaef3d114a5fa
2082 F20101123_AAADZN graham_s_Page_135.txt
71636a0c7221e0f40c61ae58cb667543
067316c7dcace9184c6921668d9be3334b3eb959
1951 F20101123_AAADYZ graham_s_Page_118.txt
d820401a78abb6db6d58f54f03a39f85
d90c8403aa59ed8af76c3afa7bb100d04f36365a
48443 F20101123_AAADCF graham_s_Page_075.pro
e5cfc5ac748c04d1ce531b6ddb263ee2
44f7541c13410b939d22203e288ceeb8dea43c69
1925 F20101123_AAADBR graham_s_Page_059.txt
c0a68880e4613afae446473d091b4e33
f4a0698980900da555fb6ba99c79be1fdfd550e8
27287 F20101123_AAAEFI graham_s_Page_086.QC.jpg
06e66730bd97920a29f1814d7d7e77f6
bb1eb9ff33c470f3d7ecbaadfdae09a130aade3a
6994 F20101123_AAAEEU graham_s_Page_079thm.jpg
c2fd85a4bfe27c71cf841d6fc588f239
b2e36606db4327de53afdee8cbb8e795c0c8d2bb
167 F20101123_AAADZO graham_s_Page_136.txt
0a59f1157e9c0d623f3fa26e967e6379
363125c1ba42263983ae8a27d750412e1150e18d
49669 F20101123_AAADCG graham_s_Page_025.pro
49e8fed5af97f69b7e93f5e674557411
932ef442761703c5aa16acd4e79abbc223eec489
F20101123_AAACXA graham_s_Page_129.jp2
13f7cd77f952e12307d6fdaecf37c0a8
708fe6c1baec4734ebd151aeece44a84ac3d8ad5
98826 F20101123_AAADBS graham_s_Page_110.jpg
121e46ea363a0c14ec33eb5d3a5922fb
e5e1439cb98575d42a480a48a46e14418f25063b
27029 F20101123_AAAEFJ graham_s_Page_087.QC.jpg
46b92939eadbde49d39f2d6a844e3c72
c6e5e5be35bb1f9b05d7d1eb25952334ed13bbf4
27947 F20101123_AAAEEV graham_s_Page_079.QC.jpg
f10db48fd4174a37f45108ed9b967d36
c546648c8c2e7c3f4a0f2b66382fe98023c4a9af
2156 F20101123_AAADZP graham_s_Page_137.txt
6f59dabccce28f3c60f637fd55ddf995
9e04adc8a42ff0a79b1a859c37d3ca0183e9b581
30256 F20101123_AAADCH graham_s_Page_105.QC.jpg
58e0eb8286d799f73d03d1b4edcb93c8
5cc2b0a6b75434ea3e1ff6c8234b52f96baedc99
91647 F20101123_AAACXB graham_s_Page_058.jpg
6a50eadfee39fd3d6408fafc1e06fe22
2c05895a7230125964bb9cf13b9c90b209a7e7da
F20101123_AAADBT graham_s_Page_079.tif
d3dca94d91ad9e93ae1de77b061fe525
0c5989712ce57ba0029fccd1cf20674c38854740
7891 F20101123_AAAEFK graham_s_Page_088thm.jpg
52ec2d8bf29fd5114f5816c2cf02c688
7d16a674cd0cc222c58389d8b644dfc162435eaf
7659 F20101123_AAAEEW graham_s_Page_080thm.jpg
a630ac8e96d7697256a597f6fa34e5f4
15bf414a0730befeeac6fa49cddddfe3f34d7f4f
1839 F20101123_AAADCI graham_s_Page_111.txt
a1181dc7d1ee13fa2c30f379057d8f97
17c9e7a93d392758cd43922bd204965720d5cfbb
1189 F20101123_AAACXC graham_s_Page_034.txt
979e54c89ea6ff0c2fb8d3f731a15b87
6919d83783faea8454081561fd66fef22c0862a8
81307 F20101123_AAADBU graham_s_Page_084.jpg
4f39217774a9d46084717755a4ea456e
f25b304311916f44b3f5b971d668ea248fbe8054
2581 F20101123_AAADZQ graham_s_Page_138.txt
33c34ccb70219be44cfc512d3826693c
a32d0e5ca91fd65ce426fdc4a8f3bc50d76b7cc7
24481 F20101123_AAAEGA graham_s_Page_097.QC.jpg
d137b03f5049692905e747181fc96c11
430081acb90e3ac858264f62b86752ea3626d36f
32361 F20101123_AAAEFL graham_s_Page_088.QC.jpg
340199970eec99186136bd1283ed5582
cee93b3129acc54e0059a1d49df040225c986610
26590 F20101123_AAAEEX graham_s_Page_080.QC.jpg
4cfd68c98c7d10f9dfdce09ce8cb5e8d
e5097973f66caa2e2dae02a41916323ce69cad81
45371 F20101123_AAADCJ graham_s_Page_093.pro
6c90d01a17c424b44f39b6a068c459c6
85dd40d2036bd057f4605364743f2e96dc9e4954
30281 F20101123_AAACXD graham_s_Page_133.QC.jpg
878123ddb34b0c20e830fe0263bb3696
336682c9c5bf4b70c359b7ba9e96cbb3b76545c6
7956 F20101123_AAADBV graham_s_Page_049thm.jpg
20ba5075c6a51b8915318bc6bf175383
c93d326304c24e94f00514f96a759556ec58e7de
2541 F20101123_AAADZR graham_s_Page_139.txt
80205f530bf32a2bc34b3fcee6fff89d
4e2b5f57ad4885fba3f1d47c2716160af9664bf1
7102 F20101123_AAAEGB graham_s_Page_098thm.jpg
56c79bc91a8e3f50bdc5bf2b67c34efa
d2a3267593f298a2c9f7a0c08c703400c629e145
7903 F20101123_AAAEFM graham_s_Page_089thm.jpg
1f4d083a97514c8628ac601e71bebbca
57d818688c6a140d599509f699ffb415457ac46b
F20101123_AAAEEY graham_s_Page_081thm.jpg
5a7d1c805c670fc94900ed5bd3437595
8290ab9d5aad4ec4c5ca13457d9fd0b706907eef
1861 F20101123_AAADCK graham_s_Page_045.txt
b2d2393362c34537a1dfe2971b855d05
4f832187407953274253d95d683d626db2957fae
106825 F20101123_AAACXE graham_s_Page_041.jpg
82c19a84ac07ca73a87a0083a6ce2282
f91a1c1179337a4aadd2cd3fabeb2c602000b5d6
F20101123_AAADBW graham_s_Page_007.tif
f4c4ffe6b6ac0d850cfda89eafe9b922
e06cb6ae54f1da3692ffeacdbe2a740d3b3e1e74
87428 F20101123_AAACWQ graham_s_Page_021.jpg
94a36df40a42016ce08f3cf8e138c1ce
721d39ab80a92b15c2e1b3bd7299a5051dddfa71
2398 F20101123_AAADZS graham_s_Page_140.txt
5552a99e44cb43973a0e8d23ec377263
64ca435764d6134de6c603dfaedbbaccdda40b17
27586 F20101123_AAAEGC graham_s_Page_098.QC.jpg
a31924e196394e8c68cf66732456ba28
4256694af2130fa76203b79a3b77a58ee0951853
31612 F20101123_AAAEFN graham_s_Page_089.QC.jpg
c3119305a44db1e5649ee3e4dfd5e246
93880864fcc4b3e077c9903311f95d78ab27d41e
28101 F20101123_AAAEEZ graham_s_Page_081.QC.jpg
9c78ee36fa4f66b757d18fe7464e7186
6d8f4dd0ba6555a6df749dae9b573b099885bafc
57299 F20101123_AAADDA graham_s_Page_142.jpg
0eaf2d9d4975b097aaff72ee8e792a87
ef455a6fe7fae48f22227411c944c74f4cd22a03
24128 F20101123_AAADCL graham_s_Page_090.QC.jpg
fe57ee0ac6608f040dd823fe065f8c5d
d31879e68fb0dc5068cbc06112b0a8c603092354
F20101123_AAACXF graham_s_Page_024.tif
0c1c66ffda19f7f7d50778abb937d1a1
cecccc5f9d89b20b66fcf0073d4d78b9ab9757ae
93046 F20101123_AAADBX graham_s_Page_071.jpg
d28b45accd8fc1e5eadb4342d7a0c93c
2e369a9797afd934da1a11ccdba23b7ed6494831
32858 F20101123_AAACWR graham_s_Page_041.QC.jpg
b482219734306895c2b6977ad0d3c5b2
e964e8efcd3175d0f0c1517bfce81f5a57eb10d9
448 F20101123_AAADZT graham_s_Page_141.txt
3f15f374ae8cf5b364316a169a068f83
6cb3167c5e8d9e716655484c0ef9faf3b5540e28
8164 F20101123_AAAEGD graham_s_Page_099thm.jpg
4a44edba536b25068345cfe89fce0e68
ee85399603bf83e456023a469620aadc564c9db9
7092 F20101123_AAAEFO graham_s_Page_090thm.jpg
3a657334ef4dde87f48bfba5710db886
bc8355d9a89d291cba4faa0f8eb86fd9bc7596fe
45754 F20101123_AAADDB graham_s_Page_134.pro
d6bed1b487b5b6debf08dfeabc2d00f7
761eeeb59681bad2b2f598dd9c544f6333653189
3022 F20101123_AAADCM graham_s_Page_055.txt
37aa159810f1d29823aa4cb5845f8fb2
f3686a9834614329e3b016401c7cffa4112f9925
34064 F20101123_AAACXG graham_s_Page_030.QC.jpg
b2e19b3d956b93e92e13f98eb97f4216
638db542e369a782c2b218cdee21015f8d1aafe7
99762 F20101123_AAADBY graham_s_Page_117.jp2
5889c72fe9514c99231a2e09f15700ed
168f69699fbe348fc158af974a7e6fdb4be7be44
65489 F20101123_AAACWS graham_s_Page_010.pro
5d2eeb47703780d454f3d02cee30ec8b
cff205595f7689958705159b3d27ef6db6268cda
1076 F20101123_AAADZU graham_s_Page_142.txt
06cab1b9fe5792d6a86452622bde0445
901421b4b0caa3909c4d13170ecf37a7cb260396
32677 F20101123_AAAEGE graham_s_Page_099.QC.jpg
0e6ecdfedb95c0a4e5077330edeff31e
0c1a0fdde4e1281fbedb7a328f6c6cd74cc7ce81
27650 F20101123_AAAEFP graham_s_Page_091.QC.jpg
7c8e605f6d240b1898ca5a517e35b66b
e6ee28f6f202c625f87f4c99702b453a3c60ca28
889415 F20101123_AAADDC graham_s_Page_081.jp2
a056906651f347ea64d949ec6d026e6c
e127ddb66dd1e98c27ae58d5ded616eb0a689a28
343 F20101123_AAADCN graham_s_Page_130thm.jpg
cd654a6c8ccb7c54d6eac4f905783149
822400d564e61a4064147a8821dab6ae09127ddc
F20101123_AAACXH graham_s_Page_115.tif
8314445dcc7892fcb0912dbfe78399fd
b208154668f8e15db3c75622a61b97ab1a3444eb
105343 F20101123_AAADBZ graham_s_Page_127.jpg
9e3fea747b3bfd5ae3cc1163077fb429
eb629384469643e8722580fe59d5e6f6956e11ba
F20101123_AAACWT graham_s_Page_080.tif
7743ccb715ea01b6bea932d26da4060c
ba9632f0980413a80eb6bf3997227929c3e37375
2065 F20101123_AAADZV graham_s_Page_001thm.jpg
abef4d77ed75c1016ac299e2ab0980d1
16d2c9bbb4c124d7b60907b508b8115b678530ca
7628 F20101123_AAAEGF graham_s_Page_100thm.jpg
18a555987b0baaec169a440300849eba
07849d4acbb205fe5c86e9d39c7da9fb5fb5b085
7874 F20101123_AAAEFQ graham_s_Page_092thm.jpg
1132de53d1202c1e1959c6ffd0dab7f3
f779ab7f9e0d215ab3d4a4616d28def6cdf4faca
98971 F20101123_AAADCO graham_s_Page_118.jpg
a5286991ceb14802d8c701e448d4c8b4
f9212a8f918909b0dc8d473901da7d4bf5a114f1
30173 F20101123_AAACXI graham_s_Page_017.QC.jpg
c895cd83701c2f3cae6443c0141fb32d
d70013fca4debc8b0d98a5ad756e84908add9ab5
2432 F20101123_AAACWU graham_s_Page_130.jpg
5b7eead25f2b57fa09eea6c1431b5d15
855e5750d5779ef8a7011227b7f0c7604d4aa3b5
8188 F20101123_AAADZW graham_s_Page_101thm.jpg
3b6ead786488ae20d7f041b2460abeb4
dba6cf10f1b42ee44b43fbe207f4e8acecfe4e7a
945431 F20101123_AAADDD graham_s_Page_047.jp2
a8869a25530c255bcaad8822dd35fd57
1dc90e2ab46ef4149d12ce4f371001a1cd257b94
28703 F20101123_AAAEGG graham_s_Page_100.QC.jpg
439cc60474dafe61e45547dc39bc0bed
822379c802afe1bdb8ed651edfb369a595c0e5b9
29186 F20101123_AAAEFR graham_s_Page_092.QC.jpg
9833895aaa598ccb380c6c961ff07db8
69c65326bff4aa4f599b45ab4fa3cab51df6d2ea
826550 F20101123_AAADCP graham_s_Page_094.jp2
988f5af1159da096626f1b1da4a0a444
d4b5fa1216bfd5aa61925452e41068c411e7f317
88370 F20101123_AAACXJ graham_s_Page_055.jpg
3520040af6e0c7a4405e15de1ae0562f
44dc1db66ad0635dc58d0cf8f0115e37502ad12c
F20101123_AAACWV graham_s_Page_108.tif
6dff402ded26117e0c1428782dbb1f35
2246d3965a7c5a5c840c926a6f258cbb4e25ed1d
28044 F20101123_AAADZX graham_s_Page_024.QC.jpg
d39e269d00ee0ae4ec54a4460ba62fc3
c3cd23fcdc1721a26e26b90cf193e12855107b66
25479 F20101123_AAADDE graham_s_Page_142.pro
c1e241b7720c6ad6ad2b703f56a4aabd
e87963096bcbc9a3cbd0dc6d34e00c390c359f85
7095 F20101123_AAAEFS graham_s_Page_093thm.jpg
5f506c9e4adf83b0b902d7fbe628b3ff
f68e7d4e2896f089a1f4a50de58086f3bd8ed89b
47089 F20101123_AAADCQ graham_s_Page_091.pro
cc4efe209c0e219b6e6955db9b411a5a
13915f9683a6904920d29cfc55090ea435a63d8d
47106 F20101123_AAACXK graham_s_Page_133.pro
ec005732870790860b13e7da571520fd
4cce9437847b749381cd4597f23c51ba5016ad47
F20101123_AAACWW graham_s_Page_035.txt
6a9c042be21084cc7a3d8db179434328
719cd2cb74b416b34f58f44afc140b1aafba97c3
29330 F20101123_AAADZY graham_s_Page_044.QC.jpg
b60d99edc64c67c651b6552acfc9ba4d
d7a48565ed77949755d796857ad2eaba816f90ae
32748 F20101123_AAAEGH graham_s_Page_101.QC.jpg
3c5c07d9cee5d02d7b4092f5700e0e4b
fb53ae64617279390545f40f86bed6226035caa3
28288 F20101123_AAAEFT graham_s_Page_093.QC.jpg
32401a9c0484850b9b45fdd1970909e7
9d8e92ebb0be7eaf17028b4c4a7e0382629c818b
35794 F20101123_AAADCR graham_s_Page_068.pro
f79318d7404460bcbf31a1c64422225b
8cc580f70f6c2e6967a2363996ec1d7d962eee3f
1895 F20101123_AAACXL graham_s_Page_048.txt
c48ef0cd510d31aa02417a44f8c491f0
517c8069b49825e88fbd38f29b64adc0dc04e598
F20101123_AAACWX graham_s_Page_110.tif
361bf7566b9fc71bea077d1b0914a256
77c8a8f55fb734a3af2f3ab60debed972528d518
212362 F20101123_AAADZZ UFE0011901_00001.xml
24f9baa48be6968cd07f2f2ed4e0b4bb
bb06e40f1cc641159de9cfd604a512072d13cd86
114836 F20101123_AAADDF graham_s_Page_027.jp2
1eb4007583ba37babb8c787542425666
7a6f871c9fb8323ffe301ee724a9cc59ee5c9e0c
8344 F20101123_AAAEGI graham_s_Page_102thm.jpg
6f645432a1bd8e323e4f67b9cb0cf719
51be22a02aed35ba07ba77cfc638103c99fcf96e
7412 F20101123_AAAEFU graham_s_Page_094thm.jpg
6cf6c013646aaa59da93ee0f7eab5220
99244da27c6dd21180ec2fbf53ef8461a2dd51fd
F20101123_AAACYA graham_s_Page_071.tif
af44cd8fd2b14df8f21dbcf0a775dfa6
ac7d0b591153b287e3643fc952e9be35fa10d6e3
101535 F20101123_AAADCS graham_s_Page_112.jp2
2d6412dff3afffb356eb6b17d509d6a5
a4d67f0090f40598aedd6a52a7535107f1176aa0
44332 F20101123_AAACXM graham_s_Page_103.pro
6c7bf501a14a7a0bab9d4990ac6660ee
278b1de2dceef685f230aaff3245cb4b5809f82f
F20101123_AAACWY graham_s_Page_136.tif
0790cb837cc1c2cb0c92309b3444228b
3bc5da268ba9e7ccb7ddbfa04ce53480e2a45910
42296 F20101123_AAADDG graham_s_Page_078.pro
1c66c7415ad07365b9f907216260b2fe
6a654aa50109910230a8501739f520b51b4121e8
F20101123_AAAEGJ graham_s_Page_103thm.jpg
dec8663fdd2453cca4f8db80d50fa5f2
3ea7607f641c32bd440a3727d104153cf7cd6d71
26275 F20101123_AAAEFV graham_s_Page_094.QC.jpg
62fe3f7335ec00e8822e952ca27c2751
9f60726df88d0a77191b3780b61ce3629aec57d4
55006 F20101123_AAACYB graham_s_Page_031.pro
d2b2c374ef63428a2984302c95a039e5
5e15d37f27f1a25ee97ec038f8a922c32eb10397
F20101123_AAADCT graham_s_Page_063.tif
f2b0f8936202c218b24e5c731fe7f64a
eda79cc623f963f5946bb2f081f8385e2324c9e4
F20101123_AAACXN graham_s_Page_053.tif
7b5d97d279cc6417a5869bd5c061775e
1affb142b58a4d93c119aefec8462fa344c54bd6
F20101123_AAADDH graham_s_Page_006.tif
9eab6a42b5f2aee88592c73112e06666
6c23d8d83b06b36504840d4ae6ae75438426f23c
30463 F20101123_AAAEGK graham_s_Page_103.QC.jpg
1c92fa6ea3ab9bd5640fc712430f17e0
4f9c4d626150261b1b49f0513b751bcda77f19bd
25376 F20101123_AAAEFW graham_s_Page_095.QC.jpg
4c56dbd963e5696185dfa8877402bddd
bc0ae328781c6e0142073fd9b19dda801491377a
F20101123_AAACYC graham_s_Page_006.txt
a89055fa211c94952452736f49c08c55
65c85c086d3fdeffaf6be3cdd5021c28d1e860b0
8493 F20101123_AAADCU graham_s_Page_038thm.jpg
44df6336130edd2c5275845a21c3c8bb
4d87f506c560622139d738b96329a15eb0dda46a
97213 F20101123_AAACXO graham_s_Page_089.jp2
1c468d5c55480b6f91e68fa52ce8544a
a713d2933a21413bf1ec58928301f9177c7b53de
99493 F20101123_AAACWZ graham_s_Page_134.jp2
b9d51f47ceb6cbc801c6b11b775f4cf2
e7d2f2dab97726ad084b4ab2d774490c494833ae
F20101123_AAADDI graham_s_Page_007.jp2
493d5036c80b0f9c59cde6ccbcfa8eec
3f98e90c67624ad147c620a7e578251e165a8815
30944 F20101123_AAAEHA graham_s_Page_112.QC.jpg
e68070f87c12726981e95231d47457e2
5d656e278d65ee4f3c78b889db031b0d065e5b13
29779 F20101123_AAAEGL graham_s_Page_104.QC.jpg
7189a3ae30bbb48821acfcd108e89d10
067e76322a50193630bee628568b62fde35b2924
7395 F20101123_AAAEFX graham_s_Page_096thm.jpg
c0db1b55679ae4ea3e5dda44e431f3c0
484e676f37de073240855953eb690a7c8aac13f6
F20101123_AAACYD graham_s_Page_015thm.jpg
a664af664c1b924b1222f65800e5aed7
f1c05a0e2048edc463131d48533ea8970ee3b461
F20101123_AAADCV graham_s_Page_082.tif
6ceb3395eb41e32d25f9c4e6ed986118
b0343544ea3c0294bc21103f42469a6ae8e7c947
F20101123_AAACXP graham_s_Page_058.tif
d3384b3b5e9ff277929dbbd0257f06c0
9c407401e51a6138f3fd357c9acca6af11b95282
31624 F20101123_AAADDJ graham_s_Page_049.QC.jpg
e7636e05801e487ab084863ffb0f91c2
f172e3b6a704668a0d90d3202571175c15b83718
7968 F20101123_AAAEHB graham_s_Page_113thm.jpg
4a9d8b16f6aec1f5e68035f1fe217547
81771e05ea6e8030c7e6abc6d45e24b2be4eec86
7204 F20101123_AAAEGM graham_s_Page_105thm.jpg
fb5855718b827fe5d927b344855f0380
9715a8aa77d65e1747f6300c693b8547c46d9ef7
30111 F20101123_AAAEFY graham_s_Page_096.QC.jpg
b9619783c413b9d95e098f9eb99f522d
d94e532e4c6d2a3d5ee3897a56b5aa6aa2a4c388
38348 F20101123_AAACYE graham_s_Page_086.pro
6e8387e2ce70ee05170e5d2c074ba569
6518dead6bf828afeb83a49028c05669c0d30545
97497 F20101123_AAADCW graham_s_Page_075.jp2
dc86cae9060c89589e4f9033c5d0a8ae
33c11579bf295054e9064c93e5e6df6f152653c4
F20101123_AAACXQ graham_s_Page_029thm.jpg
9bb7e4d6efbef9a2ff93eddacc5d546b
9b44c31c22044970b2036128ed970d8f71a20cf8
22168 F20101123_AAADDK graham_s_Page_015.QC.jpg
4de323e522d67296f13557cee9e8ffe0
c6a41f1778c78814cdcb389a772a5bc3d753b42e
31275 F20101123_AAAEHC graham_s_Page_113.QC.jpg
4cbf859dc380b5fac0b82190043d7207
c8d2a6548c834ddb7990b8c4c64099b5f34ccf27
7703 F20101123_AAAEGN graham_s_Page_106thm.jpg
f3d28407b8af2290bce6aee3d12b3517
de71eb2b60c7676c52e5c377167cf1691a812564
6114 F20101123_AAAEFZ graham_s_Page_097thm.jpg
cccc02e4a66deb66429edbaf077ee397
64b2a566a7e41f86936f344e3dcbd6783ff2f7aa
29522 F20101123_AAACYF graham_s_Page_056.QC.jpg
62f914e22a00ac6365e2897289128d9c
20885efc0865259d50b924478e3ea0c172e41051
1226 F20101123_AAADCX graham_s_Page_002.pro
92ae777168f11009dc6eeb47ebc6d126
b02cac924797122208d47ac5e573dc50b66e9d6a
F20101123_AAACXR graham_s_Page_070.tif
68ab7f88d5d7bd25eaf88ac0bb76bd0f
ea8ca039a0e38a3e5206161d72d9d8ac603e6f6e
29159 F20101123_AAADEA graham_s_Page_043.QC.jpg
6c3a6f14d4845d32e71798093a3f9e20
f506d8acb25c140c7372378d2004c11c2aaad321
1833 F20101123_AAADDL graham_s_Page_056.txt
3b33b9a956d87d7e87015db2ca19597a
d50e483a9ca1df758439ae0bcd66a6e58428932d
7697 F20101123_AAAEHD graham_s_Page_114thm.jpg
b6e549588cc2cc947c1aa6a2c124ad57
366f15f747cc059000d6884e020da7ff2f6188a0
30809 F20101123_AAAEGO graham_s_Page_106.QC.jpg
bbbb54cff0851aafb432b9080f1d403a
2251ee331ed49f7802e7d460822ebf941859f1ee
F20101123_AAACYG graham_s_Page_015.tif
4fcc52e977ab8f83de89e0dfdd40e498
160e86c502c78abb312b9b2f96ad4237616b15f8
7792 F20101123_AAADCY graham_s_Page_047thm.jpg
f7cbb33e67b141c6046d691106b8ca08
d775aa2f474c777939f30f0dff168b9613db12e5
103327 F20101123_AAACXS graham_s_Page_039.jp2
b2faa48be039151fc7e5745a31806bbd
cbd4c848504b0779ae779dcab82ae21b4dea07c7
2075 F20101123_AAADEB graham_s_Page_060.txt
0ae59dd90d0cb0f65b69ebac594ecb96
fe9208e294a59d26610e455bf9fc1ed98a87d3c5
596482 F20101123_AAADDM graham_s.pdf
f03928e04c8b119f4406fa8dfbfe13af
d1e42fdf013630c37748fd9b3ec39eb36f2a4a2c
31258 F20101123_AAAEHE graham_s_Page_114.QC.jpg
58ebc3cd12ea4c415225bc5840769d22
c78ec1be5454673ebd200a63cd8ec82a75b3a98b
7531 F20101123_AAAEGP graham_s_Page_107thm.jpg
2f953ffe78083979611dc93c81e6a485
dcfd16a376a1d1ff49cefb22253f5ccbf64f5fb8
F20101123_AAACYH graham_s_Page_137.QC.jpg
766f6b5d34500355181ddca476ef4a6a
67998de34262d431604b155a01fe033f73cfc67b
1051971 F20101123_AAADCZ graham_s_Page_066.jp2
eb7604a47f27b4f65891d0cc4389a2b2
d6bcf68abc01bc831e64503e58e0366cd083fe65
63872 F20101123_AAACXT graham_s_Page_013.jpg
c4b0fca0c1ad1adadd86981025d9c882
70bcc230a10af67af6db70cfb38531fa8f5b46e7
91649 F20101123_AAADEC graham_s_Page_056.jp2
33a2025f1d03311fc53f0f2349da6a18
97176d9394abfa8220a535d3e7cb87499da4c977
F20101123_AAADDN graham_s_Page_107.jp2
2d9bf022e7b323fa4b97bb48feddd9d4
70d802c75c862687897a3d943b4816671df85a6e
3870 F20101123_AAAEHF graham_s_Page_115thm.jpg
47bfc42ed1ef248e03b5412587514e85
4c16deac521297f56a423c0621fa38a621ba2879
30650 F20101123_AAAEGQ graham_s_Page_107.QC.jpg
2416356d5369dcceeaadab1d8e9d38a9
db24227df683cd1c0e21ca3e8e6aca5c2c994d94
F20101123_AAACYI graham_s_Page_020.tif
0e6c044c9cb684526e47fac3efb3dcb2
600dcbd997a794006078622e149681221112097d
98080 F20101123_AAACXU graham_s_Page_103.jp2
265eee062abd7e892d92d48408b62a2b
727717bdcbc2cdf712eadffd42c6e7de66dec71f
F20101123_AAADED graham_s_Page_131.tif
804de774134fa8d94e588825ccdbeafd
023545d656001b1bdf0dff09be81df6676a9ed07
79165 F20101123_AAADDO graham_s_Page_011.jpg
3a28f9e0c35548e70f2cd5d3cc7aaeec
c04840dbf0798251d54e9bc845750cf2b093ad7a
14731 F20101123_AAAEHG graham_s_Page_115.QC.jpg
6ae277173d1f5941d77ccdd210b93c3e
45fd763901678c79bb81eb94417f1195e975fe01
7983 F20101123_AAAEGR graham_s_Page_108thm.jpg
2ad26c5887f5d323e749b9dbea693792
eb94e583a92cc3aad666547e668bd28e3b3eccbe
1051977 F20101123_AAACYJ graham_s_Page_005.jp2
eb18a53388a28b88c11fafd9eb9916ea
f80431319bd48f8c491a414c4d36896569be699e
91018 F20101123_AAACXV graham_s_Page_093.jp2
679cc0a97ac3e5fc2e46133977b73d42
715f7b38b2f915cd89bad1e9daaeff6611869968
2050 F20101123_AAADEE graham_s_Page_051thm.jpg
7efcdc8b34fd31e9d4ee6d9d420111a9
01ba385fd64304141927404866a96de4bf2048ba
75467 F20101123_AAADDP graham_s_Page_097.jpg
bb64082629b0a31579919cb391c103eb
d4525332754c1aec4632e5834d09ff2e8b87f80a
6945 F20101123_AAAEHH graham_s_Page_116thm.jpg
28d2f0ccb3f89e8867fb090f83f3a0c8
9cdd1c5fb40e97ea6ee3130052ac73be9b1d80e4
32886 F20101123_AAAEGS graham_s_Page_108.QC.jpg
64f22b11b534e5afcad12d806f8f5d8e
7fb5bde311f9e2ecb2e5e0a255bc3155f254f059
F20101123_AAACYK graham_s_Page_018.txt
20d448339be5c248043d46e2074b85c2
2e45bf3e3350eec0cd75937c3574f3a1075a843c
78347 F20101123_AAACXW graham_s_Page_020.jpg
3e7cbdafb42d66f5fbe29c97d4c6c9a9
758d0f7501aee7c3fb4fe19acbd21ff9d4544395
F20101123_AAADEF graham_s_Page_022.tif
b96d3c3133a05bb8cdda1dbba397278e
7ca46350c8a5e8a5944f442475dff80ba9887e12
93802 F20101123_AAADDQ graham_s_Page_019.jpg
9a4236630d845e28efa97885c4dd012b
4eeeaa4609513bc79009743471e038af59bfbfb2
7681 F20101123_AAAEGT graham_s_Page_109thm.jpg
cab82fcb5c50774acc86be62074e8b5d
40753a24500226ccd730915d4ef6e98cf6783afa
104972 F20101123_AAACYL graham_s_Page_125.jpg
4c2ed1b73654dd85181557cdaf297ea0
66fd82d39612d42290713a493c7e2b7f4a5b9b9c
7589 F20101123_AAACXX graham_s_Page_043thm.jpg
23901af17875e16d5efc5526570e5f61
c3bc616efd41319edc92b2ac1dc6fe6790a247eb
F20101123_AAADDR graham_s_Page_140.tif
d261378953a0f81cbfc8c23a31df82e4
5753c555cd4ba6c175d69e0823951c976178de5d
28326 F20101123_AAAEHI graham_s_Page_116.QC.jpg
686915dcb0bbc99086b3f8079096a8f9
a89433a58937ba6a5f3d10670717373c92029c08
29760 F20101123_AAAEGU graham_s_Page_109.QC.jpg
25911723be99d1dacb6dc3d81c3426df
a0a39e43a2f283745e171afc9dcd1ecab2ec4464
7487 F20101123_AAACYM graham_s_Page_058thm.jpg
b865b9786a6b62f8031491e34696f04e
2ca7be2671b19f1f3507cc20ff6a0918c5967de6
31442 F20101123_AAACXY graham_s_Page_122.QC.jpg
3d20dc73ed74d25d1926881c7e0d0d95
fbe41149b92494c76929ecb39de9f1aba6e0e32c
6976 F20101123_AAADEG graham_s_Page_095thm.jpg
9c1bca8c7a16a88cee1bf4ab0a0064da
ee33ba286e9bfeca6df7354ef8da61e7f9bc7f15
25244 F20101123_AAACZA graham_s_Page_141.jp2
6ba8ecdabfbe6bc4651294af21776855
3db7f784a54f1fd4582853ad9aae7a9988012360
F20101123_AAADDS graham_s_Page_141.tif
3afa885edf714e02d87bf9175922722d
4519e19ad5c98211846b1dd13864a934f60aaaaf
7624 F20101123_AAAEHJ graham_s_Page_117thm.jpg
a52727927130130484c2539383ad2ac9
e8916e79d5e13e7a14285b5ee5f00af5c5bdb30d
8052 F20101123_AAAEGV graham_s_Page_110thm.jpg
b88f715ea3c258f99ddb1cc9330be731
3389b057a2b49649aff94275b40cb64034d02501
109958 F20101123_AAACYN graham_s_Page_125.jp2
f32f062208583846d8ef77ac3e17e69a
13aecf94a070aa4939c057a4d7164b6d4556ad27
38322 F20101123_AAACXZ graham_s_Page_072.pro
7d1c80a3e651f1050be2499a4cf4d701
05c736a71168dbb9d216a4cb40733dd295635de3
45443 F20101123_AAADEH graham_s_Page_017.pro
fc3e45397d252b3437153285bc264ee5
70402fd2011839ca1e731b3155d277ae8442e244
95281 F20101123_AAACZB graham_s_Page_092.jpg
17b4e0495742cfbee21952cdbaaebcd7
faa6e414de8a9397973e741f0111ee16fd7e2f8c
1918 F20101123_AAADDT graham_s_Page_063.txt
5c0d48b1c4b209af83ed7503e67b3e0d
b9352ae653a54d1f810a70f1cd81d7fd6340f479
29272 F20101123_AAAEHK graham_s_Page_117.QC.jpg
2d50fdeb4bcdf46b7fbc09a54a5efd3c
f85d65109bebf8b7f369c2c1db69828aee6aab34
31370 F20101123_AAAEGW graham_s_Page_110.QC.jpg
3d027b1770190670dbfe5b0b6c975805
87b17425e21fe40eafe9a43416f4c004eae304f8
71977 F20101123_AAACYO graham_s_Page_015.jp2
700bd714d438d7bdf8bc1225af7eff57
b6649fb0a950dc1eaa1290392ce95e4b0cee33bb
F20101123_AAADEI graham_s_Page_113.tif
b92094943fa5213ac913e2fcb2958d18
7e9c6edbef243ba73b4cbe6f8425f75b4f7a36b7
103319 F20101123_AAACZC graham_s_Page_083.jpg
70a2466796f9214448195136b3037e00
97ed773e1fe3bb6385d1b7f1d93f4b61105e6d42
7787 F20101123_AAADDU graham_s_Page_077thm.jpg
f8940388b8f765c1cbddaeacdc19ebe3
723ace3ab2f20b0899152661e98aa703751e019b
33065 F20101123_AAAEIA graham_s_Page_126.QC.jpg
3c192800cb9e54eea9a323631bc7eeb1
fd6ae7b1a4ea4a75e25531df4be203016cbf326b
8125 F20101123_AAAEHL graham_s_Page_118thm.jpg
30dfff67835e25d232edd1283a1099b4
2890552430d5a8e305bde19ace95da1b5085f445
7884 F20101123_AAAEGX graham_s_Page_111thm.jpg
879309fef0516ec6ae26cfbd746b07fe
1b158ed8d80facb0ae706e6033cd3b02ecb19736
108240 F20101123_AAACYP graham_s_Page_101.jp2
a08f723e437166fe421e1ec642dc4374
58c9091d0fb6decd2aa7d94f4ffa6408d5f52607
1589 F20101123_AAADEJ graham_s_Page_065.txt
072ad1f21ded5500ea63e82af470c8e7
718ab57784b970b3095ca8e47f904eafd20ebdfa
96510 F20101123_AAACZD graham_s_Page_054.jpg
7e8a5966eb1f5756073cee7ae937b117
db98c253fb25d2c11f966cc684c3fd4bfff5eff7
106780 F20101123_AAADDV graham_s_Page_032.jp2
6d1895dbbf4cd9871e160cf3705b4586
49c6d1219985f5ce9c7194bdb7637fd33ae233f5
34177 F20101123_AAAEIB graham_s_Page_127.QC.jpg
ac8d3b261fdc7c698ea7b18715502027
d1315c1fe4da9da584254297bf5c1772b63bf958
32418 F20101123_AAAEHM graham_s_Page_118.QC.jpg
98036d736316072774823c7df8d8bf4f
235dbbceecac40129c4adbe5908b13651d3683f9
30245 F20101123_AAAEGY graham_s_Page_111.QC.jpg
65b12dc1526113c21577313b9bf7e48b
3b29050ab5f7463b5a278bfcd272fbfcdac0e157
F20101123_AAACYQ graham_s_Page_075.tif
2fcebf5d1b7f562b8412db040990673c
946499b1af2e35aed7f3d684131792119be21f0a
5876 F20101123_AAADEK graham_s_Page_129thm.jpg
1281e2b5ab89be46a6e65af6d2984ce3
3e3964f77ee82bca507ee32e699c1ba344bff869
103867 F20101123_AAACZE graham_s_Page_124.jpg
e9c43808521bb91348a88d9145e61ba9
61a7106a734e02a5f737470fb13db216e65c3f29
1968 F20101123_AAADDW graham_s_Page_074.txt
3e8e1e0453f6306a5d278e50d737565d
717249dd8be14183f2d028f763d7674e192e5eb9
5579 F20101123_AAAEIC graham_s_Page_128thm.jpg
f3d9bcf21df49055cb43e942462d6786
bbeca3133b24adbebac819306ea1e7c67e349555
8323 F20101123_AAAEHN graham_s_Page_119thm.jpg
859782881ecb81fd5a3b3eb3ca7a4b05
a4074ca926d05ab6c7022e122672bdf27a67df75
7894 F20101123_AAAEGZ graham_s_Page_112thm.jpg
ef2ffd18f60e1c792bd0868ef4803b84
d96a5eab6440b82b26cb1611828701a7a2637661
F20101123_AAACYR graham_s_Page_096.tif
93da915a6a8a13994f634e7d04074664
656c8508435fb7bd8fdcd596035c0068bd1acdb2
2788 F20101123_AAADFA graham_s_Page_003.jpg
864a872d509fd5054e1bc16d14ba3c0d
659ffa4fb50d817d354ddf84d777038741d995a0
99562 F20101123_AAADEL graham_s_Page_025.jpg
066dcf89c1f0b390d79796722158aa7c
80b9a44e52844ee15bfa95df8aa82472ac342fdd
2279 F20101123_AAACZF graham_s_Page_091.txt
6d96885adff00f693785a003b7222667
0e3afefb4fdd180e4dea07c8b7ee7a509eeda94a
106577 F20101123_AAADDX graham_s_Page_121.jp2
63db13ca88d82fa860fe7d3618b10619
e931fb3fb08fd00632766b569202c4c15c333652
22381 F20101123_AAAEID graham_s_Page_128.QC.jpg
146c3f5705fd256611294baf31e7d568
b14255639fb346124024982a7c8ad7411cdc625f
8380 F20101123_AAAEHO graham_s_Page_120thm.jpg
a09b5629310eb36f488e8f43e9697c30
c5782e3789eec16e4da3205d2976cd91765b783e
7544 F20101123_AAACYS graham_s_Page_104thm.jpg
9c3e5e65e5e677f5e1426e8866c775ed
69e5abeae8f6fa0db5f441cd22a657e956371a52
39917 F20101123_AAADFB graham_s_Page_004.jpg
f0f2f73c84b9e94388a38fdb2ed934a0
eb87824b80ce610902deb342cfc447a22b263d8d
421 F20101123_AAADEM graham_s_Page_001.txt
49e18d5d45706d964ba2acb564c195b5
126ca43dd2384e0b8376b811df4046a1af2a3632
95394 F20101123_AAACZG graham_s_Page_114.jpg
d1282d6c17c63eaa19a9914979f9f73a
cc512e954f20d751c3d1737c94df4dd81189c3db
F20101123_AAADDY graham_s_Page_117.pro
e1cf479b8b50e5567b8a00b0b41f65fe
ebd21f35301784c6dc6eeb3402cb17f401f81d33
21159 F20101123_AAAEIE graham_s_Page_129.QC.jpg
763f580269accb58749040c0b42c0393
c9def7607e48692f34bc2490a8c0131af6a3d501
32374 F20101123_AAAEHP graham_s_Page_120.QC.jpg
423638f2dbef041272b38e433b51b12d
a1b744ecfd4ed114a3a565df22fcfd5c9ebd3e57
6887 F20101123_AAACYT graham_s_Page_091thm.jpg
cd257ba8609f57d3fce65e84ad4d2f72
03de4c6141a680fb3c4ee6b846bd9adfdb93bb0b
92878 F20101123_AAADFC graham_s_Page_005.jpg
feb07588af7d700319aed29636196255
450a3b7cba0438d170fd8b83217c8927c019705c
2153 F20101123_AAADEN graham_s_Page_075.txt
cfcb94b08e781fd01dbdf90131a8f834
958e0dd9c88a2d9b7980214f0b35091db803d821
95946 F20101123_AAACZH graham_s_Page_053.jpg
f15a89ed4d65c7ab6bbca3b681d849ca
5438a4fb9a24354fbb62755d3c6e9ae68d049032
48237 F20101123_AAADDZ graham_s_Page_118.pro
25006cf49b5cc083b667a4788b9e6380
8c359bafba2298e2ac3e8c84cf1a635ab51d676c
826 F20101123_AAAEIF graham_s_Page_130.QC.jpg
2d92f24b06bdc595dd0a553c0bcfd575
b0b89c0e54e4aee9d1259588074e8a86797a67b7
8223 F20101123_AAAEHQ graham_s_Page_121thm.jpg
ad2d1e6844c4adfaccc0c15a929d3ae5
e259707914e5aa6c2fc9e23cbd051f2a41a6510a
49533 F20101123_AAACYU graham_s_Page_066.pro
f535c3b21a5cafb7218163e9ced707d2
32dcb414310e7906978e21dc39c44b3f981e5410
153708 F20101123_AAADFD graham_s_Page_006.jpg
264384a6a6d2712b3b4d774fa38accb7
734e9891c57544917ac1adf608c1b6a514062a2f
1723 F20101123_AAADEO graham_s_Page_021.txt
147de1f2e0722f7cf289172f245ea65a
41bbc8a7e4e545ebe5a5a3343ae1500ed9f4ab30
8407 F20101123_AAACZI graham_s_Page_127thm.jpg
e01c0095e910bb6455ad69d55afde729
e2adfc99a91c4118dbbc66bd7ff52195b8a9724a
7654 F20101123_AAAEIG graham_s_Page_131thm.jpg
aaa8e5cc6527a1d66e6a6b37f16619fc
ecef348f5d27640dfd00d9f10848f4d621b98ae8
33091 F20101123_AAAEHR graham_s_Page_121.QC.jpg
f28ac56d26222b68340fddd7e468faff
dd583b2b1ca480886e21d93dcca0c0625ac02bbe
2061 F20101123_AAACYV graham_s_Page_125.txt
e781daa79633a42734e89f89595de4f8
deba0fe23ecd438500c1d6a63d003019694c487f
133776 F20101123_AAADFE graham_s_Page_007.jpg
ed38f1903c4c674b018a0843cffb1cde
317c333367ee24c8c9e739c8605cc904a378151b
33388 F20101123_AAADEP graham_s_Page_029.QC.jpg
3a3c4b190cd2cb7901a179fbf9341217
7606b498465871b5d7d0c54d9e1c0179ba5d14cf
2011 F20101123_AAACZJ graham_s_Page_132.txt
a32d10a880c2425bd370db446b364628
1395c71d9e7f8d8776b702940e0318c84ff7d29b



PAGE 1

EFFECT OF ACADEMIC DETAILI NG ON COX-2 UTILIZATION RATES By STEPHEN DOUGLAS GRAHAM A DISSERTATION PRESENTED TO THE GRADUATE SCHOOL OF THE UNIVERSITY OF FLOR IDA IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY UNIVERSITY OF FLORIDA 2005

PAGE 2

Copyright 2005 by Stephen D. Graham

PAGE 3

To Evie

PAGE 4

iv ACKNOWLEDGMENTS I would like to thank my wife, Andrea, and sons, Nicolas and Andrew, for their love and support. I thank my dissertation chair, Dr. Ab raham Hartzema, and committee members, Drs. Ingrid Sketris, Almut Winterstein, Richard Segal, and Babette Brumback for their guidance through the dissertation process. I would like to extend special thanks to Ms. Dawn Fr ail at the Nova Scotia Department of Health and again to Dr. I ngrid Sketris at Dalhousie University for providing me with overwhelming support and enc ouragement to succeed and to return to Canada. Finally, I would like to thank the gra duate students for giving me many happy memories of Florida.

PAGE 5

v TABLE OF CONTENTS page ACKNOWLEDGMENTS.................................................................................................iv LIST OF TABLES.............................................................................................................ix LIST OF FIGURES..........................................................................................................xii ABSTRACT.....................................................................................................................xiv CHAPTER 1 INTRODUCTION........................................................................................................1 Background...................................................................................................................1 Problem Statement........................................................................................................3 Research Questions and Hypotheses............................................................................4 Research Question 1..............................................................................................5 Research Question 1 Hypothesis...........................................................................5 Research Question 2..............................................................................................5 Research Question 2 hypothesis............................................................................6 Research Question 3..............................................................................................6 Research Question 3 hypotheses...........................................................................6 Research Question 4..............................................................................................7 Research Question 4 hypotheses...........................................................................7 Significance of Research..............................................................................................7 2 LITERATURE REVIEW.............................................................................................9 Review Articles Addressing Effects of Academic Detailing.......................................9 Academic Detailing Studies Reporting No Statistically Significant Effect...............12 Propensity Scores........................................................................................................17 3 METHODS.................................................................................................................22 Step One: Extraction and Validation of Data.............................................................22 Sources of Data....................................................................................................22 GP Inclusion Criteria...........................................................................................25 Patient Inclusion Criteria.....................................................................................26

PAGE 6

vi Step Two: Adjustment for Confounding Us ing Three Distinct Propensity Score Methods...................................................................................................................26 Quintile Propensity Score Method......................................................................29 Regression on the Propensity Score Method.......................................................29 Greedy Matching Method................................................................................29 Propensity Score Method Selection.....................................................................29 Step Three: Primary Outcome Analysis; Intervention Effect on COX-2 Utilization Rates........................................................................................................................32 Step Four: Secondary Outcome Analyses; The Utilization of Other Health Care Resources Associated with NSAI D Induced GI Side Effects.................................35 4 RESULTS...................................................................................................................37 Step One: Extraction and Validation of Data.............................................................37 Step Two: Establishment of Balanced Control and Experimental Groups Using Three Propensity Score Methods............................................................................37 Pre-Propensity Score Analysis............................................................................37 Quintile PS Method Analysis..............................................................................39 Regression on the Propensity Score Method Analysis........................................41 Greedy Matching Method Analysis.................................................................41 Selection of a Preferred Propensity Score Method..............................................43 Exploratory Analysis of the Propensity Score Methods Effect on Adjusting for Bias on Unmeasured Variables..................................................................44 Step 3: Primary Outcome Analysis.............................................................................49 Model Development............................................................................................49 Between Group Results.......................................................................................51 Within Group (Longitudinal) Results..................................................................53 Step 4: Secondary Outcome Analyses........................................................................55 Misoprostol Utilization Rates..............................................................................55 Model development......................................................................................55 Between group results..................................................................................55 Within group (longitudinal) results..............................................................57 PPI Utilization Rates...........................................................................................59 Model development......................................................................................59 Between group results..................................................................................60 Within group (longitudinal) results..............................................................61 H2A Utilization Rates.........................................................................................63 Model development......................................................................................63 Between group results..................................................................................63 Within group (longitudinal) results..............................................................66 GP Office Visit Rates..........................................................................................67 Model development......................................................................................67 Between group results..................................................................................67 Within group (longitudinal) results..............................................................70 Rheumatologist and GI Specialist Visit Rates.....................................................72 Model development......................................................................................72 Between group results..................................................................................72

PAGE 7

vii Within group (longitudinal) results..............................................................74 Hospitalization Rates Due to GI Complications.................................................76 Model development......................................................................................76 Between group results..................................................................................77 Within group (longitudinal) results..............................................................80 Death Rates Due to GI Complications................................................................81 Model development......................................................................................81 Between and within group results................................................................82 5 DISCUSSION.............................................................................................................83 The Academic Detailing Program in Nova Scotia.....................................................83 Qualifications of the Detailers.............................................................................83 Changes Which Occurred Over the Peri od of the Intervention (History Effects).............................................................................................................83 Policy Options Available to Decision Makers....................................................84 Distribution of educational material.............................................................85 Educational meetings...................................................................................85 Audit and feedback.......................................................................................85 Reminders and reminder systems.................................................................86 Drug benefit changes....................................................................................86 Primary Outcome: Effect on COX-2 Utilization Rates..............................................86 Statistical Results.................................................................................................87 Practical Significance..........................................................................................87 Comparison with Literature.................................................................................88 Secondary Outcomes..................................................................................................88 Effect on Gastro-Protective Agents Utilization Rates.........................................88 Misoprostol...................................................................................................88 PPIs...............................................................................................................89 H2As.............................................................................................................89 Effect on Utilization of Medical Services...........................................................89 GP office visits.............................................................................................90 Specialist office visits...................................................................................90 Hospitalization rates due to GI side effects..................................................91 Death due to GI complications.....................................................................92 Propensity Score Analysis Methods...........................................................................92 Greedy Matching Method................................................................................92 Quintile Method...................................................................................................93 Regression on the PS Method..............................................................................93 PS Exploratory Analysis......................................................................................93 Limitations..................................................................................................................94 Data Limitations..................................................................................................94 Design Limitations..............................................................................................96 Conclusions.................................................................................................................98

PAGE 8

viii APPENDIX A AN APPRAISAL OF THE NOVA SC OTIA OA AD INTERVENTION...............101 Conduct Interviews with Physicians.........................................................................102 Focus Intervention on Specific Physicians...............................................................103 Define Clear Objectives............................................................................................104 Establish Credibility.................................................................................................106 Stimulate Physician Interaction................................................................................108 Use Concise Graphic Educational Materials............................................................109 Highlight and Reinforce Essentials..........................................................................110 Positive Reinforcement with Follow-up...................................................................111 Summary...................................................................................................................112 B OA AD DESKTOP REMINDER.............................................................................114 C THE THEORETICAL F OUNDATION FOR ACADEMIC DETAILING.............116 LIST OF REFERENCES.................................................................................................122 BIOGRAPHICAL SKETCH...........................................................................................127

PAGE 9

ix LIST OF TABLES Table page 2-1 Summary of Included Studies fo r Thomson OBrien and Grimshaw......................11 3-1 PS Model Variable Descri ptions and Abbreviations................................................27 4-1 Descriptive Statistics for Con tinuous Variables in the PS Model............................37 4-2 Descriptive Statistics for Cate gorical Variables in the PS Model............................38 4-3 Pre-PS Univariate Analysis for Included Variables.................................................39 4-4 Physician Distribution by Quintile...........................................................................40 4-5 Quintile Method Regression Analysis Results.........................................................40 4-6 Distribution of Influenza AD Partic ipants by Propensity Score Quintile................41 4-7 Regression on PS Met hod Analysis Results............................................................42 4-8 Greedy Matching Method Analysis Results.........................................................43 4-9 Quintile Method Results for Excluded Variable Models.........................................45 4-10 Regression on PS Results for Excluded Variable Models.......................................47 4-11 Greedy Matching Results for Excluded Variable Models....................................47 4-12 Correlation Matrix Between VOC and PS Covariates.............................................48 4-13 Primary Outcome Model Results (Periods = 3,4,5,6)..............................................52 4-14 Primary Outcome Model Results (Periods = 1,2)....................................................52 4-15 Least Square Means for Change in COX-2 Rates by Group....................................53 4-16 Unadjusted Means for Change in COX-2 Rates by Group......................................53 4-17 Primary Outcome Model Results (AD = yes)..........................................................54 4-18 Primary Outcome Model Results (AD = no)...........................................................55

PAGE 10

x 4-19 Secondary Misoprostol Outcome Model Results (Periods = 3,4,5,6)......................56 4-20 Secondary Misoprostol Outcome Model Results (Periods = 1,2)............................56 4-21 Least Square Means for Change in Misoprostol Rate by Group..............................57 4-22 Unadjusted Means and Standard Deviat ions for Change in Misoprostol Rate by Group........................................................................................................................58 4-23 Secondary Misoprostol Outcom e Model Results (AD = yes)..................................58 4-24 Secondary Misoprostol Outc ome Model Results (AD = no)...................................59 4-25 Secondary PPI Outcome Model Results (Periods = 3,4,5,6)...................................60 4-26 Secondary PPI Outcome Model Results (Periods = 1,2).........................................61 4-27 Least Square Means for Ch ange in PPI Rates by Group..........................................61 4-28 Unadjusted Means for Change in PPI Rate by Group..............................................62 4-29 Secondary PPI Outcome Model Results (AD = yes)...............................................63 4-30 Secondary PPI Outcome Model Results (AD = no).................................................63 4-31 Secondary H2A Outcome Mode l Results (Periods = 3,4,5,6).................................64 4-32 Secondary H2A Outcome Model Results (Periods = 1,2).......................................65 4-33 Least Square Means for Change in H2A Rate by Group.........................................65 4-34 Unadjusted Means for Change in H2A Rate by Group............................................66 4-35 Secondary H2A Outcome Model Results (AD = yes).............................................67 4-36 Secondary H2A Outcome Model Results (AD = no)...............................................67 4-37 Secondary GP Office Visit M odel Results (Periods = 3,4,5,6)................................68 4-38 Secondary GP Office Visit Outcom e Model Results (Periods = 1,2)......................69 4-39 Least Square Means for Change in GP Office Visit Rate by Group........................69 4-40 Unadjusted Means and Standard Deviati ons for Change in GP Office Visit Rate by Group...................................................................................................................70 4-41 Secondary GP Office Visit Outc ome Model Results (AD = yes)............................71 4-42 Secondary GP Office Visit Ou tcome Model Results (AD = no).............................72

PAGE 11

xi 4-43 Secondary Specialist Office Visit Model Results (Periods = 3,4,5,6).....................73 4-44 Secondary Specialist Office Visit Ou tcome Model Results (Periods = 1,2)............74 4-45 Least Square Means for Change in Specialist Office Visit Rate by Group.............74 4-46 Unadjusted Means and Standard Deviat ions for Change in Specialist Office Visit Rate by Group..................................................................................................75 4-47 Secondary Specialist Office Visit Outcome Model Results (AD = yes)..................76 4-48 Secondary Specialist Office Visit Outcome Model Results (AD = no)...................76 4-49 Secondary Hospital Length of Stay Model Results (Periods = 3,4,5,6)...................77 4-50 Secondary Hospital Length of Stay Ou tcome Model Results (Periods = 1,2).........78 4-51 Least Square Means for Change in Ho spital Length of Stay Rates by Group.........79 4-52 Unadjusted Means and Standard Deviat ions for Change in Hospital Length of Stay Rates by Group.................................................................................................79 4-53 Secondary Hospital Length of Stay Outcome Model Results (AD = yes)...............80 4-54 Secondary Hospital Length of Stay Outcome Model Results (AD = no)................81

PAGE 12

xii LIST OF FIGURES Figure page 2-1 Distribution of Propensity Score Ar ticle Objectives: 1987 to July 20, 2005...........19 3-1 Propensity Score Logist ic Regression Model..........................................................28 3-2 Experimental Design Timeline.................................................................................32 3-3 Primary Outcome Model for Between Group Effect...............................................34 4-1 Frequency of Influenza AD Par ticipants by Propensity Score.................................42 4-2 Comparison of PS methods Ab ility to Reduce Bias on VOC..................................45 4-3 Summary of PS Models Eff ects on Reducing Bias on the VOC.............................48 4-4 Scatterplots of Propensity Scor e Versus Unbalanced Variables..............................49 4-5 Line Graph Comparing Correlati ons and Percent Bias Reduction..........................50 4-6 Primary Outcome Model..........................................................................................51 4-7 Least Square Means for Change in COX-2 Rates by Group....................................53 4-8 Unadjusted Means for Change in COX-2 Rates by Group......................................54 4-9 Secondary Outcome Model for Misoprostol Utilization..........................................55 4-10 Least Square Means for Change in Misoprostol Rates by Group............................57 4-11 Unadjusted Means for Change in Misoprostol Rates by Group...............................58 4-12 Secondary PPI Outcome Model...............................................................................59 4-13 Least Square Means for Ch ange in PPI Rates by Group..........................................61 4-14 Unadjusted Means for Change in PPI Rates by Group............................................62 4-15 Secondary Outcome Model for H2A Utilization.....................................................64 4-16 Least Square Means for Change in H2A Rates by Group........................................65

PAGE 13

xiii 4-17 Unadjusted Means for Change in H2A Rates by Group..........................................66 4-18 Secondary Outcome Model for GP Office Visits.....................................................68 4-19 Least Square Means for Change in GP Office Visit Rates by Group......................70 4-20 Unadjusted Means for Change in GP Office Visit Rates by Group.........................71 4-21 Secondary Outcome Model for Specialist Office Visits..........................................72 4-22 Least Square Means for Change in Specialist Office Visit Rates by Group............75 4-23 Unadjusted Means for Change in Specialist Office Visit Rates by Group..............75 4-24 Secondary Outcome Model for Hospital Length of Stay.........................................77 4-25 Least Square Means for Change in Ho spital Length of Stay Rates by Group.........79 4-26 Unadjusted Means for Change in Ho spital Length of Stay Rates by Group............80 4-27 Secondary Outcome Model for De aths Due to GI Complications...........................81

PAGE 14

xiv Abstract of Dissertation Pres ented to the Graduate School of the University of Florida in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy EFFECT OF ACADEMIC DETAILI NG ON COX-2 UTILIZATION RATES By Stephen Douglas Graham December, 2005 Chair: Abraham Hartzema Major Department: Pharmacy Health Care Administration Background: The prevalence of osteoarthritis (OA) is estimated at 50 to 80 % of the elderly population and thera py aims to relieve symptoms si nce there is no cure. Nova Scotia general practitioners (GPs) identifie d a need for an academic detailing (AD) intervention aimed at optimizing the management of OA. Objectives: The primary objective was to measure the effect of an OA AD intervention to reduce th e utilization rate of COX-2 inhib itors in the elderly population. Secondary objectives were to examine the in tervention effect on th e utilization rates of gastro-protective agents and medical services. Methods: We conducted a retrospectiv e cohort study employing administrative data to examine the effects of the intervention. Differences in utilization rates were evaluated using generalized estimating equa tion (GEE) analysis for longitudinal data. Selection bias was anticipated sin ce the intervention was voluntary, and randomization not possible. Three methods of propensity score (PS) analysis (quintile

PAGE 15

xv stratification, regression on th e PS, and greedy matching) were evaluated for the ability to adjust for bias on PS model covariates. Findings: We identified a significant differe nce in the change in COX-2 utilization rates between groups for the three month pe riod following the intervention (p = 0.0395, 95% CI (0.0365, 1.4815)) and a significant decr ease in the intervention groups within group utilization rate between the pre a nd post intervention periods (z = -2.34, p = 0.0191). The GP office visit rate was the onl y secondary outcome wh ere the intervention group was significantly higher (p = 0.0275, 95% CI (-0.7926, -0.0464)). The difference occurred in the time period from th ree to six months post intervention. Conclusions: The OA AD intervention was associated with a significant decrease in COX-2 utilization rates in the three month period immediately following the intervention. The effect of decreased util ization continued for the rest of the post intervention periods but was not statistically significant. The only secondary outcome to show a significant between groups effect was the GP office visit rate which was higher for the intervention group in the second three month post intervention time period.

PAGE 16

1 CHAPTER 1 INTRODUCTION Background In June 2002, the Division of Continui ng Medical Education (CME), Dalhousie University Faculty of Medici ne, began their second academic detailing (AD) intervention with provincial physicians aimed at optimizing the care of osteoarthri tis (OA) within the seniors population (persons great er than 65 years of age). The AD program is an ongoing initiative funded by the Nova Scotia Depa rtment of Health and managed by the Drug Evaluation Alliance of Nova Scotia (DEANS). As the AD program is a continuing effort and represents a significant cost to DEANS it is necessary to evaluate the effectiveness of the intervention. The OA topic was chosen as an AD interv ention based on the extent to which OA affects the elderly population and on the feedback that Da lhousie CME received from general practitioners (GPs) in a survey filled out following the previous influenza AD intervention which indicated the GPs desire to have an OA AD intervention developed. The Dalhousie CME Division then presented the OA topic to a GP focus group where the need for education pertaining to available OA therapies was determined. OA is a progressive disease that affects th e joint cartilage and eventually leads to joint failure.3 The prevalence of OA in the population is extremely high. It is estimated that 50 to 80% of the elderly population experience symptomatic OA.4 Estimates specific to the province of Ontario, propose that almo st all persons over the age of 65 exhibit signs of OA on radiographi c evidence and of these 33% are symptomatic.5 OA is equally

PAGE 17

2 prevalent in men and women, w ith women showing more mani festation in the knees and hands and men more prevalent in the hip. Arth ritis has been associated with half of all disability in the elderly population.4 There is no known cure for OA3 and available palliative treatments are associated with substantial toxicity and side effects.4 Treatment is therefore primarily aimed at reducing pain, improving joint mobility, and limiting functional disability. Patient education regarding medications used in the treatment of OA (primarily for the control of pain) and appropriate exercise regimens is also important.3, 4 The OA AD intervention has set four lear ning objectives. Each physician visit will include at least the follo wing: (1) a discussion of the goals of therapy, (2) recommendations for non-pharmacological tr eatments when appropriate (e.g., physiotherapy and exercise), (3) advice for pa tients about the safety and efficacy of acetaminophen, and (4) a discussion of the ro le of traditional non-steroidal antiinflammatory drugs (NSAIDs).6 The primary interest of this research dea lt with the fourth message specifically, the analysis of the effectiveness of the OA AD intervention as it pertains to the pharmacotherapy of OA and in particular th e usage of COX-2 inhibitors. The Nova Scotia OA AD was developed in 2002 and th e intervention called for the use of acetaminophen as a first line therapy for mild to moderate OA. The intervention suggested that if acetaminophen did not c ontrol pain symptoms, then the use of traditional NSAIDs in as low a dose as possible and for as short duration as possible was indicated. NSAIDs were cons idered appropriate therapy for moderate to severe OA.6

PAGE 18

3 The role of COX-2 inhibitors in the management of OA was assessed by the OA AD group as controversial. The Ontario Tr eatment Guidelines for OA recommend that, based on evidence of similar efficacy and ear ly evidence of somewhat lower rates of serious GI events, selective COX-2 inhibiting NSAIDs can be considered for patients at high risk of serious GI events.3 This recommendation, however, is one that is from welldesigned, randomized controlled trials or me ta-analyses with inconsistent results or demonstrating equivocal benefit.3 The Nova Scotia program states that, the precise role of COX-2 inhibitors in the treatment of OA remains to be determined.6 The summary statements in the OA AD intervention6 relay two points that are relevant to this analysis. Firstly, COX-2 inhibitors are as effective but not more effec tive than traditional NSAIDs for symptomatic treatment of OA and secondly, the CLASS7 and VIGOR8 trials were inconclusive in the analysis of the gastro-protective e ffects of COX-2 inhibitors. When faced with the substantia l effect of OA on the population,6 the uncertain role of COX-2 inhibitors in the treatment of OA, the increased cost of COX-2 inhibitors over the traditional NSAIDs (appendix c), and the ut ilization rate of COX2 inhibitors in the Nova Scotia pharmacare population of approximately 6% in 2001,9 the DEANS Management Committee undertook to develop the AD intervention on OA Management. Problem Statement The effect of AD on clinical and economic out comes is of great interest to the Nova Scotia governments policy makers as funding for interventions to improve the health care system is scarce. This research addr esses the question of whether the AD program on OA is effective in lowering the utilization ra te of COX-2 inhibitors. At the same time the study measures the effects that the progr am has on the utilization rates of other healthcare resources such as hospital or physicia n visits that occur as a result of GI side

PAGE 19

4 effects associated with drug therapy with tr aditional NSAIDs and COX-2 inhibitors. The efficacy of traditional NSAIDs and COX-2 inhib itors in relieving pain is similar but the GI side effects profile for traditional NSAIDs is higher.10 It is expected that the intervention could increase the utilization rate s of gastro-protective agents (particularly misoprostol and proton pump i nhibitors (PPIs)) but it is not expected to increase other health care utilization rates and will theref ore not have negative impacts on the outcomes of care. The methodological challenge for the evaluation of the OA AD intervention is the need to significantly adjust fo r selection bias that is likely present since GPs can choose to participate and those that do participate might be diffe rent from those that do not participate. Statistical adjustment th rough regression on the propensity score (PS) methods have been shown to be effectiv e in reducing between group biases on many confounding variables.11, 12 The use of PSs in studies that examine the unit of analysis other than the patient is uncommon in the me dical literature. In this study the unit of measure was the GP. No other studies with the GP as the unit of measure were found in the medical literature so the ev aluation of different PS methods ability to adjust for bias between GP groups was warranted. Research Questions and Hypotheses The term statistically significant is defined as results where the type I error (alpha) is less than 0.05. The results are statistical ly significant if th e analysis yields p -values less than 0.05. Hypotheses relating to research questions one to three are examining the effect of the OA AD intervention in the Nova Scotia re sidents who are greater than 65 years old

PAGE 20

5 and have a GP who has participated in the intervention as compared with GPs in the province who did not particip ate in the intervention. The first research question examined the expectation that GPs will consider the information provided in the OA AD interven tion and choose not to prescribe COX-2 inhibitors for their elderly patients. Research Question 1 Do the patients of GPs who have undertaken the OA AD intervention have significantly lower COX-2 inhib itor utilization rate s after the GP has undergone the AD intervention as compared to a GP control cohort? (Are there significant between group differences?) Research Question 1 Hypothesis The null hypothesis is that the OA AD intervention will have no effect on the utilization rate of COX-2 inhibitors. The alternative hypothesis is that the OA AD intervention will have the effect of decreasing the utilization ra te of COX-2 inhibitors. The second research question examined the sustainability of the intervention (if research question 1 hypothesis is found to be significant) since a shortcoming of the OA AD intervention (appendix a) is the lack of a follow-up visit to GPs who participated in the intervention.13 Research Question 2 Does the decreased utilization rate of COX-2 inhibitors for patients of GPs who have taken the AD intervention remain si gnificant for a period of one-year post intervention? (Is the interv ention effect sustainable?)

PAGE 21

6 Research Question 2 hypothesis The null hypothesis is that the OA AD interv ention will not have a sustained effect on the decreased utilization rate of COX-2 inhibitors. The alternative hypothesis is that the OA AD intervention will have a sustained effect on decreasing the utiliza tion rate of COX-2 inhibitors. The third research question examined whet her patients of GPs in the intervention group experienced a change in the rate of medi cal services utilizati on due to a change in GI adverse events associated with traditional NSAID therapy (if there was a significant finding to research hypothesis 1). The hypothe sis is divided into two categories: those that are related to pharmacotherapy and thos e that involve other medical services. Research Question 3 Do patients of GPs who have undertak en the OA AD program have medical utilization rates associated with their OA that are significantly differe nt from patients of GPs who have not participat ed in the intervention? Research Question 3 hypotheses The null hypothesis is that the OA AD intervention will have no effect on the utilization rate of (1) PPIs, (2) H2As, (3) misoprostol (4) GP office visits, (5) specialist office visits, and (6) death rates. The alternative hypothesis is that the OA AD intervention will have the effect of changing the utilization rate of (1) PPIs, (2) H2As, (3) misoprostol (4) GP office visits, (5) specialist office vis its, and (6) death rate. The fourth research question examined whether one PS adjustment method was more successful adjusting for bias between groups based on measured bias reduction for

PAGE 22

7 covariates that were not bala nced after group assignment a nd the resulting sample size after PS methods were applied. Research Question 4 Is there a superior PS method for the re duction of selection bias between the intervention and control groups? Research Question 4 hypotheses The null hypothesis is that there will be no difference in the three PS methods (quintile stratification, regre ssion on the PS, and greedy matching) ability to adjust for bias on unbalanced covariates. The alternative hypothesis is that one PS method will adjust for bias on unbalanced covariates to a greater ex tent than the other two. Significance of Research This research is of significance to severa l groups within the h ealthcare system. The three groups that benefit directly from the research are patients, physicians, and health policy decision-makers. The results also a dd to the academic research in the area of effective behavioral change methodol ogy and it adds to the methodology and understanding surrounding the use of PSs. The largest impact of this research is in the area of health policy decision making. The decision to proceed with one course of act ion is often at the expe nse of others. This study will inform decision ma kers regarding the effectiven ess of the OA AD intervention and allow them to make a more informed d ecision to continue with the AD detailing program to educate physicians on other hea lth related topics or disease states. This research adds to the validity of the research that has been accumulated in the area of AD. This is significant as it was concluded by Davis et al. in a systematic

PAGE 23

8 literature review of AD that while AD is e ffective it is seldom used by providers of continuing medical education.14 The uniqueness of this resear ch lies in its analysis of a population based continuing AD program and not one that has been developed for the purposes of a single study. This research advances PS methodology. It compares three PS methods in a real world and population based intervention. The re sults should contribut e to the choice of PS methods employed by future researchers. The study also analyzes each of the propensity score methods ability to bala nce the control and intervention groups on unmeasured administrative variables. The ability of the propensity score methodology to balance groups on measured variables has been widely reported; how ever the ability of the methodology to balance unmeasured variables is assumed 15, 16 and studies attempting to measure the ability of the PS method to balance physician groups on a number of unmeasured administrative variables were not found in the literature.

PAGE 24

9 CHAPTER 2 LITERATURE REVIEW Literature reviews were conducted on two ar eas of interest: articles dealing with studies relating to AD interventions which ha ve not shown statistic al significance and articles relating to the use of PS methods. The AD studies which reported no statistically significant effects of AD interventions are of interest because they possibly give examples of shortcomings of methodology that may be of use in this study. The PS articles that are of interest to our study are those which involved studies that identified some unit, other than the patient, as the unit of analysis in the PS development and articles that dealt with PS methods. The positive effect of AD on prescribi ng behavior has been summarized in a number of review articles on AD or educational outreach.14, 17, 18 This body of evidence shows that AD moderately improves physician behavior and patient outcomes. Three review articles are summarized. Review Articles Addressing E ffects of Academic Detailing Davis et al.14 reviewed 99 studies which met their inclusion criteria from a total of more than 6000 articles. The 99 studies included 160 separate continuing education interventions, including academic detailing. Sixty-two percent of the interventions showed improvement in at least one major out come with effect sizes ranging from small to moderate (quantified effect sizes not provi ded). There were fourteen AD interventions in the category of prescribing and 75% of these showed positive effects. AD was reported as an effective change agent for pr escribing. The authors concluded that AD is

PAGE 25

10 an effective strategy for continuing medical education (CME) however, it is not widely used by CME providers. Thomson OBrien et al.18 conducted a systematic review of the effect of educational outreach on professi onal practice and health care outcomes. Eighteen studies were included in the review with thirteen of the studies targeting prescribing practices. Nine of the thirteen studies employed mu ltifaceted interventions (educational outreach combined with reminders, audit and f eedback, marketing, or patient-mediated interventions). Seven of the nine studies using multifaceted interventions showed statistically significant effects with relative effects ranging from 1 to 45% improvement (table 2-1). The authors noted that potential bias exists in thirteen of the eighteen studies due to lack of randomization and six of the studies cont ained potentially important baseline differences and adjustment for th ese differences was not carried out in the statistical analysis. It was al so noted that only one of th e eighteen studies considered patient outcomes. The authors concluded th at the effects of educational outreach are small to moderate but potentially of practical importance. Grimshaw et al.17 conducted a systematic review of the effectiveness and costs of different guideline development, dissemination, and implementation strategies. 235 studies representing 309 comparisons were includ ed in the review. The sections of the review that are germane to our study are the multifaceted comparisons involving academic outreach with continuous measures for process or outcome variables. Ten comparisons were reviewed which contained measures on continuous variables. Six of the comparisons involved process measures (f ive cluster randomized control trials and one controlled before and afte r trial) and all reported impr ovements in performance with

PAGE 26

11 a median effect of 15.0% (range 1.7% to 24.0%) relative improvement. None of the studies included enough information to calcula te standardized mean difference, and two studies were not statistically significant. Four of the comparisons involved outcome measures (three cluster randomized control tr ials and one controlled before and after trial). The median effect of the cluster ra ndomized control trials was 0% (range -1.4 to 2.7%) and the standardized mean difference was calculated as 0 for one trail. The controlled before and after trial reported a relative improvement of 13.9% with a standardized mean difference of 2.38. The authors summarized the multifaceted interventions, including academic outreach, to be at best moderately effective (table 2-1). Table 2-1. Summary of In cluded Studies for Thomson OBrien and Grimshaw. Author (year) Reviewed by Inte rventions (plus AD) Relative Effect (%) McConnell (1982) Thomson O'Brien Audit and Feedback (AF), Educational Material (EMat) 45.8 Stergachis (1987)* Thomson O'Brien AF, Patient Mediated (PM), Conferences 35.7 Meador (1997) Grimshaw EMat, Educational Meeting (EMeet) 24.0 Ross-Degnan (1996) Thomson O'Brien EMat, Social Marketing (SM), PM 21.0 Peterson (1996) Grimshaw EMat 20.0 Avorn (1983) Thomson O'Brien EMat, SM 15.2 Avorn (1992) Thomson O'Brien, Grimshaw EMat, SM, Conferences 15.0 Ray (1993) Grimshaw EMat, EMeet 13.9 de Burgh (1995)* Thomson O'Brien EMat, PM 13.0 Diwan (1995) Grimshaw EMat 11.3 Steele (1989) Thomson O'Brien Reminders 11.2 Santoso (1996) Thomson O'Brien EMat, SM 8.7 Schmidt (1998) Grimshaw Organizational Change 5.5 Elliott (1997) Grimshaw EMat, Opinion Leaders 2.7 Feder (1995) Grimshaw AF 0.0 Moore (1997) Grimshaw EMat, Reminders, PM -1.4 non-significant study results

PAGE 27

12 Academic Detailing Studies Reporting No Statistically Significant Effect Five articles were reviewed in which the authors reported non-si gnificant results for AD interventions with pharmacotherapeutic outco mes. The review of results that were not positive is important because it will possi bly indicate to investigators methodological similarities that may have been employed in previous unsuccessful studies. If identified the methodological shortcomings could be avoided. Lin et al.19 studied the effects of physician training on the management of depression. The study was a before and after design with an equivalent control group. The physician sample was made up of 109 prim ary care physician volunteers and they were associated with fifteen primary clinics. Randomization of gr oups was at the clinic level resulting in 56 physicians in the intervention group and 53 physicians in the control group. The intervention was outlined includi ng the four key messages and the use of opinion leaders in in tervention delivery.20 Case managers were used for follow-up visits with the physicians. The in tervention involved other com ponents such as small group discussions, role-play and psyc hiatric consults. The author s reported that the physicians in the intervention arm of the trial did not differ significantly from the control group in adequacy of pharmacotherapy (p =0.53). While insignifican t, the results showed a decrease of 7.5% in the per cent of patients in the inte rvention group who received adequate pharmacotherapy with no change in the control group. The decrease in the intervention group is opposite to the desired outcome of the interv ention. The study also failed to show significant differences in th e number of antidepressa nt prescriptions per 100 patients (p =0.10). The percent of patients re ceiving new prescriptions in the intervention group decreased by 10.4% and in creased in the control group by 4.8%. These results are opposite to the desired out come of the intervention. The authors

PAGE 28

13 reported that the studys main failure was its l ack of power to detect a significant change between groups. The sample size used was su fficient to detect a 40% to 50% difference in adequate pharmacotherapy and a 15% to 30% difference in new antidepressant prescriptions. The fact that the effect of the intervention was the opposite of the hypothesis was not explained by the authors. Brown et al.21 studied the effect of AD and c ontinuous quality improvement (CQI) interventions on the treatment of patients for depression. The study was a randomized controlled trial. The primary care clinician groups were randomized by first matching clinicians according to specialty (internal medicine or family practice), sex, training (physician or allied health c linician), and number of patient s in a high-risk depressive cohort. The resulting sample size was 160 w ith 79 in the intervention arm and 81 in the control arm. The AD intervention involved focus groups for the collection of baseline knowledge of primary care providers (physic ians, physicians assistants, and nurse practitioners) in prep aring the intervention. The inte rvention was based on guidelines from the Agency for Health Care Policy and Research and used the same material as the Goldberg study.22 Three main messages were summarized on letter sized illustrated handouts. Four visits were used to de liver the message and the detailers were pharmacists from the clinicians own medical o ffice. The study showed mixed results. It was successful in increasing the percent of patients receiving antidepressant treatment (7.5% increase, p =0.046 in depressed arm and 0.7% increase, p =0.025 in the nondepressed population) however, it was not su ccessful in increasi ng the total days of antidepressant therapy (16.7 days effect, p =0.189 in the depressed arm and 1.3 days effect, p =0.606 in the non-depressed population). The study did not exhibit significant

PAGE 29

14 differences in non-pharmacotherapeutic outc omes (improvement of symptoms and measures of functional status). The authors report that the mixed findings could be due to the complexity of the implementation of a clinical guideline and the evidence base for the guidelines may not be generalizable to the study population. They propose that AD may be appropriate for behavioral change but is not sufficient for the implementation of clinical guidelines. This conclusion is im portant for our study since the primary outcome is change in prescribing behavior. Goldberg et al.22 studied the effect of AD and CQI interventions on compliance with guidelines for hypertension and depressi on. The study was a randomized before and after design with two experimental groups (AD only and AD combined with CQI) and an equivalent control group. The physicians were part of fifteen clinics and group randomization was carried out at the clinic le vel. The resulting sample size was 78 with 18, 37 and 23 physicians in the AD only, AD comb ined with CQI and usual care groups respectively. The AD intervention was base d on national guidelines for hypertension and depression from the Agency for Health Care Policy and Research. Five recommendations were developed includi ng two which specifically addressed pharmacotherapy. The AD intervention was de livered by opinion leader physicians and follow-up visits were conducted by staff pharm acists. The intervention was supported by handouts and pocket cards for quick reference. The study found significant effect in only one of the pharmacotherapeutic outcomes whic h was a decrease in the prescribing of 1st generation antidepressants to previously dia gnosed depressed patient s (relative effect 4.7%, p =0.04). The other outcomes prescr ibing of hypertension medications, antidepressants to previous ly undiagnosed patients, 2nd generation antidepressants to

PAGE 30

15 previously diagnosed depressed patients, a nd SSRIs to previously diagnosed depressed patients exhibited insignificant change with relative effect sizes and p-values of 1.3%, p =0.06; 2.4%, p =0.68; -2.1%, p =0.43; and 3.3%, p =0.11 respectively. One possible explanation for the failure of the study to s how significant effect fo r all but one of the pharmacotherapeutic outcomes can be attrib uted to the presentation of too much information. A successful AD intervention should include only a limited number of messages regarding a disease state.13 The presentation of an intervention covering two distinctly different disease states clearly violates this principle. Zwar et al.23 studied the effect of AD on prescr ibing rates of benzodiazepines for all indications. The study was a before and after design with an equivalent control group. There were 157 physicians who participated in the study. They were randomized into the benzodiazepine AD group (n=79) and the c ontrol group who received AD on another topic (n=78). The AD intervention was ba sed on guidelines developed by the Royal Australian College of General Practitioners and it was delivered by physicians trained in AD techniques. The intervention was not accompanied by any other methods (i.e. handouts, etc.). The study found significan t effect in overall prescribing of benzodiazepines (-26.7%, p =0.042) however, there was no significant between group relative effect (-1.2%, p =0.99). The authors attributed the lack of significant results to the effects of a pre-interventi on practice survey that was gi ven to all physicians in the study and a lack of power to detect a difference between gr oups due to the decision to aggregate data into eight subgroups there by reducing the sample size dramatically. Tomson et al.24 studied the effects of AD on physicians practice in the management of asthma and on patient know ledge. The study was not randomized and

PAGE 31

16 the sample consisted of 63 GPs in two regi ons, one region was assign ed as the treatment group (n=44) and the other region was assi gned as the control group (n=19). The intervention was developed using existing physician knowledge as the baseline and the input of respirologists. It was delivered by a clinical pharmacologist and a pharmacist and contained three main messages. The face-to-face visits with physicians were augmented with written materials. The study found that there was not a significant difference between the treatment and control gr oups in prescribing ra tios of beta-agonists and inhaled corticosteroids (no p-value repor ted). One explanation for the insignificant results could be attributed, at least in part, to insufficient power (due to the small sample size) to detect a meaningful change. The authors identified a possible selection bias in the physicians volunteering for the intervention as they may have been largely physician interested in asthma therapy to begin with. It is important to note that a review of the negative findings of studies in the literature cannot be considered to be comple te since many studies, and their fatal flaws, are not published if they are not considered to be methodologically sound or clinically important (publication bias). However, from th e review of literature that did not report significant results there are four areas of inadequacy that th e studies appear to have in common; the authors reported that there were insuffi cient sample sizes to yield enough power to show a meaningful change in the studies conducted by Lin19, Gorins25, Zwar23 and Tomson24, however in all but one of the studies19 the effect of the intervention was consistent with the study hypothesis. It is important to note that lack of power is only one explanation for th e lack of study significance, there were intervention development problem s in that the interventions were too complex21, 22, the interventions may have been compromi sed through the use of less than credible academic detailers19, 25, and

PAGE 32

17 the use of pre-tests or preintervention surveys decreased the intervention effect due to a pre-sensitization of the subjects to the intervention.23, 25 The results from the above studies are applicable to our study for the following reasons. The lack of power reported by a numbe r of the studies is only one explanation. Other explanations could include a large variation in measurement on the dependent variable or a lack of control for the vari ables that are associated with the outcome variable. For example, in our study we coul d have a large sample but if the number of elderly patients in the GPs pa nel is not controlled for then the variation could be inflated and a non-significant result could occur. In our non-randomized study design it is important to adjust for variables which are associated with the out come but it must be acknowledged that there will be variables which are important confounders and are not measured so residual confounding (bias) will exist. There may be a need to adjust for patient variables as well. For example, if the GPs patient panel is markedly ill then this will confound the results. A measure of patient wellness would help to address this problem. The lack of a follow-up visit to GPs in our study may play an important role in the outcome. Propensity Scores A literature search was conducted using P ubMed for all years up to and including July 20, 2005. The search terms used were propensity score and propensity scores. The search yielded 341 articles. The abstract s for all 341 articles were reviewed and the distribution of articles by ar ticle objective and year is illustrated in figure 2-1. The distribution shows an in itial surge of artic les dealing with PS methods in the late 1990s with articles c ontaining objectives other than medical (e.g., economic) and only a few articles with stated medical objec tives. Since 2000 there has been a surge in

PAGE 33

18 published articles using PSs partic ularly in the field of cardi ology. The increase in use of PSs has been mirrored by an increase in publ ished articles dealing with PS methods. There were four articles which used a unit of analysis for PS other than the patient. Two of the articles used huma n couples as the unit of analys is one article developed PS on hospitals and one article used communities as the unit of analysis for the PS. There were no articles found which used the physician as the unit of analysis in the PS analysis. There were 50 articles that described PS methods and 24 of these were selected for further review. Criteria for selection incl uded PS studies using GEE for outcome models, studies comparing small experi mental groups, studies describing PS and sample sizes or studies which described PS methods in detail. The information gained from these articles plus reference material from previous c ourse work, library searches, and colleagues formed the basis for the PS method as it has been applied in this study. In our study the PS represents the probab ility of a physician volunteering for the OA AD intervention given a number of personal and practice characteristics. For studies using quasi-experimental designs it is important to include methods to compensate for the lack of randomization to experimental gr oups. In our study we have made multiple measures of outcome variables both before and after the interv ention and we have included a control group for comparison. Th e control group is not equivalent to the intervention group so adjustment on PSs was us ed to reduce the effect of the between group bias. Three methods for applying PS in observa tional studies are predominant in the literature.26 The three methods are; sub classification on the PS11, 12, 27, regression on the PS12 and matching on the propensity score using Mahalanobis metric matching12, 27 or

PAGE 34

19 greedy matching techniques.28 All three of the methods; stratification, regression on the PS, and matching have been applied su ccessfully in observational studies and therefore all three will be considered for application in this research. 2005 2004 2003 2002 2001 2000 1999 1998 87-97 non-medical methods medical cardiology 0 5 10 15 20 25 30 35 40# of Articles Year ObjectiveDistribution of PS Article Objectives non-medical methods medical cardiology Figure 2-1. Distribution of Propensity Score Article Objectives: 1987 to July 20, 2005 An overarching limitation of all three PS met hods is that the PS can only adjust for bias in observed covariates27 and the extent to which the bias is abated in unobserved covariates depends on the correlation of the unobserved covariate with one that is observed.11 Shadish stated that if the PS method was successful in abating bias in the measured covariates then the assumption can be made that the methodology would be successful in decreasing the bias in unmeasured covariates as well.16

PAGE 35

20 A recent study has tested the ability of PS based on covariates extracted from administrative data to reduce the bias in unmeas ured clinical variable s. In this study the clinical data was extracted from patients charts after PS methods were applied. The experimental groups set by the PS method were tested for significant difference on the clinical variables and it was f ound that the clinical variables were not balanced between the groups.29 Other studies have explored the number of events per variable that are needed for logistic regression analysis to outperform th e PS method. Cepeda et al. reported that in their simulation model if there are six or fewer events per independent variable (covariates in the PS model) then the PS es timates are less biased then the regression estimates.30 It is important to note that even if the number of subjects exceeds six the use of PS methods is warranted sin ce it is a variable which predicts the exposure of interest without including the outcome11 and that the use of PS methods is intended to complement model-based procedures not replace them.31 There are two measures of PS model fit that are reported in studies. The c-statistic is the area under the receiver operating char acteristics (ROC) curve and is a measure of the discriminative ability of the PS model.32, 33 The range of the statistic is from 0.5 to 1.0. If a model has a c -statistic of 0.80 this can be in terpreted as the model accurately assigning random pairs of subj ects to their experimental groups based on PS alone 80% of the time. The c-statistic is intended to be an indicator in the model building process but it is not a measure of the PS mo dels ability to adjust for bias15 and it has not been found to be associated with the ability of a PS model to re duce residual confounding.32 The goodness of fit is another statistic that is commonly used in regression analysis. Like

PAGE 36

21 the c-statistic these tests were not found to be useful in predicting the ability of the PS model to reduce residual confounding.32 As a result, these measures were not used in our study to decide which PS method to use for the outcomes analyses. The c -statistic was however, used to explain the effects on the mo dels discriminatory ability when variables were intentionally removed from the PS Regression model.

PAGE 37

22 CHAPTER 3 METHODS This study is a retrospective cohort, befo re and after longitudinal design with a non-equivalent control group using the Nova Sc otia Medical Services Insurance and the Canadian Institutes of Health Information datasets for analysis. The non-equivalent control group design requires the use of pr ocedures to abate selection bias in the treatment group.12 The methodology for the study can be broken down into four distinct sections, which are as follows; the extraction and validati on of data from the administrative databases, the establishment of balanced control and experimental groups using three distinct PS methods, the primary outcome analysis of the interv ention effect on the u tilization rate of COX-2 inhibitors and, the secondary outcome analysis of the in tervention effects on the utilization of PPIs, misoprostol, and H2As. Step One: Extraction and Validation of Data Sources of Data All of the data used in this study was collected in pre-exis ting administrative databases. There were no occurrences of mi ssing data since the variables included in the analysis were extracted from long standing regi strar data which is complete for all fields listed in the registry1 (GP demographics), complete census information2 (geographic data) or the data was reported in terms of ra tes with the GP inclusion criteria ensuring

PAGE 38

23 that each GP panel contained at least twenty patients so the rates for the outcomes measures were always defined (i.e. rate denominators were not equal to zero). Administrative data must be used with cauti on as it is not 100% reli able. Chapter five outlines the limitations of the administration used in our study. GP demographic data for all GPs in th e province was obtained from the Nova Scotia College of Physicians and Surgeons Physician Registry (2002).1 The Dalhousie CME Division provided data wh ich contained demographic information of the GPs who were detailed and the dates when the detailing visits were carried out. These two sources of data were merged and the resulting file was submitted for encryption using the same encryption methods as the provincial administ rative data. The resulting encrypted GP demographic profiles were augmented with data from the Nova Scotia Medical Services Insurance (MSI) physician registry (2002) to include dates indicati ng when the GPs opted in and opted out of the provincial pharmacar e billing scheme. GP practice information such as population of the community and av erage income of the county in which the practice is located was added to th e demographic profile of each GP.2 Patient data was extracted from the N ova Scotia Pharmacare Seniors Dataset (2002-2004) and the hospital discharge data foun d in the Canadian Institute of Health Information (CIHI) hospital disc harge dataset (2002-2003). Pa tient level GP visit data was used to determine to which GPs patie nt panel a patient belonged (see patient inclusion criteria). Once the patients were assigned to GP panels the patient prescription claims data and hospital length of stay data were aggregated at the GP level. Drug utilization variables were created at the GP level with the unit of measure equal to DDDs per elderly patient per 90 day study period. Change in utilization rate variables were

PAGE 39

24 created for each GP by subtracting each period utilization rate (peri od = 1 to 6) from the baseline (period two) utiliza tion rate. Period two was chosen as the baseline utilization rate since it was the pre-intervention meas ure most proximal to the GP index date. Descriptive statistics for the GP demogra phic variables were calculated to confirm that the variables did not cont ain any missing data and to confirm that the variables fell with acceptable ranges (i.e. no GPs 200 years old, not all male GPs). The descriptive statistics are reported in tables 4-1 and 4-2. Prescription claims, GP visits and vital stat istics were checked to ensure that there were not instances of missing data. The pr escription claims and GP visit data were complete on all fields necessary for our st udy. Only hospital admissions and deaths due to GI events were included in the hospital le ngth of stay and death measures. A detailed description of the inclusion cr iteria for data is contained in chapter four. The underlying and primary causes of death were used to de termine death rates and cause of death and data was reported for all included patients who died over the study period. The first four diagnoses codes for hospital admission were used to determine if GI complications were associated with admission. In all cases there was at least a primary diagnoses on admission. While the data for our study was complete it was administrative data and there are shortcomings associated with it. The limitations of administrative data are described in chapter five. Data from several administrative databa ses was linked to create the datafile necessary for the PS analysis and for the outco mes analysis. The data linkage was carried out using the encrypted physicia n identifiers and the encrypted patient identifiers. The

PAGE 40

25 encryption of the patient and physician identi fiers was carried out according to standards set by the Canadian Institutes of Health Information (CIHI).34 GP Inclusion Criteria The academic detailing intervention was targeted at GPs and, therefore, the experimental unit is the GP and the patient data for the GPs practice is the unit of measure. Each GPs practice is measured as an aggregate of the indi vidual patients data from his or her practice. The aggregation of patient data is descri bed in greater detail later in this chapter. The date on whic h the GP received the OA AD intervention was defined as the index date. For GPs in th e control group the index-date was randomly assigned from the time period over whic h the AD intervention took place. There are four criteria that a GP had to m eet to be included in the study. They are as follows; The GP had to be registered as a GP with the Nova Scotia College of Physicians and Surgeons for the entire study period. The GP had to be included on the billing registry with the Nova Scotia Medical Services Insurance (MSI) (the provincial government payment agency for seniors medical and pharmacy claims) for the entire length of the study. This registry is the source of the medical and pharmacy cl aims data that will be used for the outcomes analysis. The GP had to have an elderly patient panel equal to or gr eater than twenty patients. The rational for the cut score of twenty was based on the premise that a 5% decrease in COX-2 utilization (i.e. C OX-2 utilization rate change from 6.0% to 5.7%) will equate to annual savings to the elderly population of approximately $100,000. Therefore, if the GP had an elderly patient panel equal to twenty he or she was required to change prescribi ng behavior for one patient over the study period to realize a 5% change. The GP had to have at least one prescrip tion claim for a COX-2 inhibitor recorded in the pre-intervention period (6 mont hs preceding the GPs index date).

PAGE 41

26 Patient Inclusion Criteria The patient is the unit of measure for this study. Patients had to meet two criteria for inclusion in the study. The criteria are: The patient had to be included on a GPs patient panel. For inclusion on a GPs panel the patient must have seen a specific GP for more that 50% of his or her total GP visits for the fiscal year ending Marc h 31, 2002. For example, if a patient had a total of forty GP visits in the pe riod from April 1, 2001 to March 31,2002 and twenty-four (60%) of the visits were b illed by one GP the patient was included on the GPs patient panel. Once the patient was assigned to a particular GP they remained with that GP throughout the study. The patient had to be 66 years of age or older as of the GPs AD index date. This ensures that the patient was at least 65 years old and eligible for the MSI pharmacare coverage for the entire study period and it provides a period of time of at least six months for the patient to become accustomed to the new MSI pharmacare coverage. Step Two: Adjustment for Confounding Using Three Distinct Propensity Score Methods The definition of the PS is the conditiona l probability of treatment given the individuals covariates. In th is case it would be the conditi onal probability of taking the OA AD intervention given the GPs pe rsonal and practice characteristics. The PS is obtained by fitting the data using a logistic regression model.5 Once the PSs were calculated for each GP three PS met hods were applied to the PS data and the optimal method in term of bi as reduction and resultant sample size was determined. The three PS methods used in this study were; the stratifica tion into quintiles, regression on the propensity score12, and greedy matching or one-to-one matching for group assignment.28 The variables in the regression model desc ribe the GPs personal characteristics (age, sex, birthplace, etc.) and practice character istics (size of patient panel, population of community in which the practice is located, etc.). All variables in the data that fit within

PAGE 42

27 these two descriptive categories were included in the regression model. This approach is consistent with the literature which calls fo r the inclusion of all variables which have some relevance to the outcome variable.16 A description of the in cluded variables and the abbreviations used in our st udy are included in table 3-1. Table 3-1. PS Model Variable De scriptions and Abbreviations Variable Description # Levels Abbreviation GP participation in the OA ADintervention 2 (Y/N) OA GP participation in previous influenza AD service 2 (Y/N) flu AD GPs sex 2 (M/F) sex GPs birthplace 3 (Nova Scotia, Canada, Other) birth place GPs location of initial licensure 5 (Nova Scotia, Canada East, Canada Center, Canada West, Other) license GPs COX-2 utilization rate at baseline (DDDs / patient) continuous BL rate GPs age (years) continuous GP age population size of community in which GPs practice is located continuous population average income of county in which GPs practice is located ($cdn) continuous aver income number of patients in the GPs practice continuous total # pt percent of GPs patients diagnosed with OA (ICD-9 CM = 715) continuous % OA dx percent of GPs patients > 65 years old continuous % elderly average hospital length of stay for elderly patients in the GPs practice (days/patient) continuous los rate A logistic regression model was used to accommodate the dichotomous nature of the outcome variable, OA. The same regr ession model was applied using PROC REG (SAS 8.2)35 for all three methods to determine GP PSs. Models described in this study

PAGE 43

28 have categorical variables lis ted as single entities which are consistent with the SAS coding techniques. The model analysis creat es (t-1) dummy variables (where t = the number of levels) for each categorical vari able. The PS regression model is shown in figure 3-1. Figure 3-1. Propensity Score Logistic Regression Model Variables were kept in the model regardless of their significance. Variables that are not statistically signif icant still contribute to the model and the population based nature of the data ensures a large enough sample size to support the model with twelve predictor variables. The final model predicts the pr obability that each GP would receive the intervention based on his or her individual vari ables. This probability is the GPs PS. Once the PSs were calculated they were applied according to the three methods stated earlier. Y = + 1X1 + 2X2 + 3X3 + 4X4 + 5X5 + 6X6 + 7X7 + 8X8 + 9X9 + 10X10 + 11X11 + 12X12 Where; Y GP participation in th e intervention (0 = no, 1 = yes), X1 GP participation in previous in fluenza AD service (0 = no, 1 = yes), X2 GPs sex (Male, Female)1, X3 GPs birthplace (Nova Scotia, Canada, Other)1, X4 GPs location of initial licensure (Nova Scotia, Canada East, Canada Center, Canada West, Other)1, X5 the GPs COX-2 utilization ra te at baseline (DDDs / patient). X6 GPs age (years)1, X7 population size of community in which GPs practice is located2, X8 average income of county in wh ich GPs practice is located ($cdn)2, X9 number of patients in the GPs practice, X10 percent of GPs patients di agnosed with OA (ICD-9 CM = 715), X11 percent of GPs patients > 65 years old, X12 average hospital length of stay for elderly patients in the GPs practice (days / patient).

PAGE 44

29 Quintile Propensity Score Method For the quintile method; the GPs in the treatment and intervention groups were stratified, based on their part icipation in the OA AD intervention, and then ordered according to the GPs PS. The treatment a nd control groups were stratified into five levels, or quintiles. Each quintile contains 20% of the GPs (table 4-4). Regression on the Propensity Score Method For the regression on the PS method; the PS was used in the outcome model. Greedy Matching Method For the greedy matching method; the GPs in the treatment and intervention groups were stratified, based on their partic ipation in the OA AD intervention, and then ordered according to the GPs PS. A matching procedure was applied28 that involved matching the groups on PS beginning with matc hes accurate to five decimal places and concluding with matches to one decimal place. The number of included GP only allowed for a one-to-one match between groups. Once matched the GP was removed from the sample pool. Those GPs that were not matc hed were deleted. The greedy matching method resulted in group sizes of 104 each (N = 208 total). Propensity Score Method Selection The regression on the PS method was selected for use in the outcomes analysis. The regression on the PS method was selected based on the following criteria; the adjustment for selection bi as on the covariates measured before and after the PS procedure is carried out, and the resultant sample size. The adjustment for selection bias after a pplication of PSs was determined for each PS method using the following methods.

PAGE 45

30 For continuous variables the percent decr ease in bias was calculated using the formula:11 100 x [ 1 (bias post) / (bias pre) ], where bias post was the difference between PS adjusted group means and bias pre was the difference between unadjusted group means (group means before PS analysis). Variable means before PS analysis are reported in table 4-3 as the unadjusted means of the groups. Variable means after PS adjustment for the regression on PS and quint ile methods were the least square means reported using PROC GENMOD (SAS 8.2)35 after adjustment for propensity score (or quintile depending on the method). For the greedy matching method unadjusted means were used for both the pre and post means calcul ations. Results are re ported in tables 45, 4-7, and 4-8. For categorical variables the percent decr ease in bias was calculated using the following formula:12 100 x [ 1 |(1OR post) / (1OR pre)|] where OR post is the odds ratio of the groups (adjusted fo r PS) and OR pre is the odds ratio of the groups before PS adjustment. For both the pre and post odds ratio measures PROC GENMOD (SAS 8.2)35 was used. The odds ratios were calculated using the same procedure for all three PS methods. Results are reported in tables 4-5, 4-7, and 4-8. A further test of the effect of the di fferent PS methods involved the purposeful removal of independent variables from the re gression model and the subsequent test for PS adjustment on the unmeasured variable. The logistic regression model was run twelve times. Each time one of the indepe ndent variables was removed from the model and the percent bias reducti on on the now unmeasured variab le was calculated for each of the three PS methods. The same equations for continuous and cat egorical variables

PAGE 46

31 were used to calculate percent bias reduction on the variable that had been removed. The results are reported in tables 4-9 to 4-11. The measurement of adjustment for bias on unmeasured variables was not considered in the selection of the PS method. It has been included in this study as a means of contributing to the PS methodology. Work has been done on the PS models ability to adjust for bias on unmeasured clinical variables29 and the PS models ability to adjust for bias on unmeasured variables in a large computer generated dataset.32 Our study is unique, however, since it examines the PS models ability to adjust for bias on demographic variables contained in a re latively small, real world dataset. There were five PS model covariates that showed significant between group differences after the initial OA group assignment. The variables were percent of patients diagnosed with OA (% OA dx), the average in come of the county in which the GPs practice is located (aver income) the average hosp ital length of stay per patient (los rate), the population size of the community in wh ich the GPs practice was located and participation in a previous influenza AD intervention (flu AD). The PS adjustment on the flu AD variable was not successful for any of the three PS methods so it was included in the outcomes models as a covariate. The othe r four variables were of interest in the analysis of effect of PS met hods ability to adjust for bias on unmeasured administrative variables. The correlations between the vari able and the PS were calculated and graphed against the percent reduction in bias fo r each PS method. Correlations between the variables and the included PS model covariates were cal culated and tabulated. The relationship between the reduction in bias in unmeasured variables and each PS methods was studied. The results are contained in chapter four.

PAGE 47

32 Step Three: Primary Outcome Analysis; Inte rvention Effect on COX-2 Utilization Rates Once the method of PS analysis was select ed and the GP intervention and control groups had been determined, the analysis of the primary outcome effect was carried out as described below. To enable the analysis of the changes in COX-2 utilization rates over time the COX-2 utilization rates were determined fo r each GP in the study for six consecutive ninety-day time periods. Two time periods we re pre-intervention and four time periods were post-intervention.(Figure 32) The index-date is report ed as the date that the GP received the AD intervention a nd the index-dates for the c ontrol group were assigned by randomly selecting dates from the range of time that the AD intervention spanned. The COX-2 utilization change rate will be calculated by subtracting the GPs baseline utilization rate (period 2 utilization rate) from the utilizat ion rates in each study period. Figure 3-2. Experimental Design Timeline Before the utilization rates could be calcu lated the inclusion and exclusion criteria for claims in a given time period were defi ned. An example of the operationalization of the decision rules for the incl usion or exclusion of claims within a time period is presented using a fictitious ninety day time period (January 1st until March 31st) and describing how different scenarios were adjudicated. If a prescription claim is submitted Intervention Group O O X O O O O Control Group O O O O O O Time from intervention (days) -180 to -91-90 to -1 Index date 1 to 9091 to 180181 to 270 271 to 360

PAGE 48

33 for a two-month supply on January 2nd it is clear that the period of time for the entire claim falls within the given time period and th e claim is included. If a prescription claim is submitted for the same two-month supply on March 28th it is clear that the entire claim period does not fall within the period ending March 31st. In this case the claim would still be counted, in its entirety, in the clai m period that it was submitted. The reason for inclusion of the claim in the initial time peri od is that it was in this time period that the GPs prescribing behavior took place and the intention was to have the patient take the medication as prescribed. Refills were considered to be an extension of the original claim until such a time as the refill claim was submitted more than thirty days after the intended fill date for the refill. If the refill wa s more than thirty days late the re st of the claim was not counted in any time period. The COX-2 utilization rates for each GP was determined through the use of the World Health Organizations (WHO) Anatomic and Therapeutic Classification System/ Defined Daily Dose (ATC/DDD) methodology36 and was reported for each GP as the average number of DDDs per included pati ent per ninety-day in tervention time period. The reporting of DDDs is often given as pe r thousand patients, however, since most GPs in the study will not have one thousand patient s that meet the criteria this could be misleading. DDDs are drug consumption data that are independent of price and formulation. Once set, the WHO is reluctan t to change DDD measures and as such the DDD is stable over time. This makes the DDD measure more reliable for drug consumption studies but it is not appropriate for clinical analysis. The DDD, therefore, "enables the researcher to

PAGE 49

34 assess trends in drug consumption and to perform comparisons between population groups."36 An analysis of the intervention effect on the primary outcome, change in COX-2 utilization rates, was carried out. The pr imary outcome model in itially included the dependent variable (change in COX-2 rates for the four post-intervention periods), the independent variables indicati ng the between group effect ( OA) and longitudinal effects (period), the PS variable, the variable flu AD (a s indicated from the PS analysis), as well as baseline COX-2 rate (BL rate), and number of elderly patients in the GPs panel (# elderly pt). The model is depicted in figure 33. Each of the variables was retained in the model regardless of its significance. The cova riates were all include d as adjustments for confounding which if not contro lled would be questioned in peer review. The included variables for the primary outcome model w ith their associated coefficients and significance levels are reported in table 4-13 in the results section. Figure 3-3. Primary Outcome Model for Between Group Effect The model determined the statistical si gnificance of the between group effect (between group effect) as well as the longitudi nal effect (within subjects effect) of the intervention. The model was anal yzed using PROC GENMOD (SAS 8.2)35 and significance is reported at the alpha= 0.05 level. Y = 0 + 1(X1) + 2(X2) + 3(X3) + 4(X4) + 5(X5) + 6(X6) Where; Y = change in COX-2 utilization rate (periods 3 to 6 (post-intervention)), X1 = GP participation in the intervention (0 = no, 1 = yes), X2 = experimental time period (period = 3,4,5,6), X3 = PS, X4 = GP participation in the in fluenza AD service (0 = no, 1 = yes), X5 = GP baseline COX-2 rate (DDD / patient / period = 2), X6 = number of GPs patients > 65 years old,

PAGE 50

35 The two ninety-day pre-intervention utiliz ation rates for the experimental groups were analyzed to determine if there were a ny significant between group differences in the change in COX-2 utilization occurring before the study commenced. The preintervention analysis was carried out usi ng the same model as the primary outcome model described in figure 3-3 however, onl y the first two time period measurements (period = 1,2) for each GP were entered into the model. This examined whether or not significant differences for change in utiliz ation rates were pres ent between the groups before the intervention was applied. A longitudinal model was tested using th e primary outcome model in figure 3-3 with a pre-intervention / post-intervention va riable added which de scribed whether the change in COX-2 utilization rate was preor post-intervention. The measurements for periods one and two were coded as prepost =1 and the measurements for periods three to six were coded as prepost = 2. The longitudi nal effects model was run twice with only one of the intervention groups included each time. The prepost variable indicated whether a significant within group intervention ef fect occurred. The results are reported in tables 4-18 and 4-19. Step Four: Secondary Outcome Analyses; The Utilization of Other Health Care Resources Associated with NS AID Induced GI Side Effects The primary outcome model exhibited si gnificant between group differences and therefore, all of the secondary outcome anal yses were carried out using the same GP groups as in the primary outcome analysis. The models for the secondary outcomes were developed using the same variables as the primary outcome model. Each secondary outcome model had the change in COX-2 utiliz ation rate substituted with the appropriate secondary outcome rate. The secondary outcomes that were analyzed are the intervention

PAGE 51

36 effect on changes in rates from baseline for; PPI utilization, misoprostol utilization, H2A utilization, GP office visits specialist office visits, a nd death rates due to GI complications. Rates for secondary outcomes (described individually with each outcome analysis) and the results for the secondary out comes are described a nd reported in chapter four.

PAGE 52

37 CHAPTER 4 RESULTS Step One: Extraction and Validation of Data PROC MEANS35 was used to perform the calculati ons for the continuous variables. The variable mean, standard deviation, median, minimum and maximum are reported in table 4-1. Table 4-1. Descriptive Statistics fo r Continuous Variables in the PS Model Group N Variable Mean Std Dev Median Minimum Maximum AD= 0 265 % OA dx 0.0913 0.1071 0.0638 0.0000 0.7391 GP age 47.30 9.78 47.00 27.00 79.00 % elderly 0.1910 0.1160 0.1676 0.0253 0.9000 total # pt 1054.18 438.55 1021.00 30.00 2575.00 aver income 27688.68 4683.37 27500.00 22500.00 32500.00 BL rate 3.6013 2.7048 3.0675 0.0769 16.8750 los rate 0.0453132 0.1449 0.0000 0.0000 1.2647 population 183342.23 162415.32 109330.00 991.00 359183.00 AD= 1 231 % OA dx 0.0719 0.0598 0.0608 0.0000 0.2601 GP age 45.74 9.18 45.00 27.00 77.00 % elderly 0.1772 0.0788 0.1681 0.0251 0.5564516 total # pt 1037.69 418.99 1009.00 171.00 2481.00 aver income 25833.33 4488.31 22500.00 22500.00 32500.00 BL rate 3.9758 2.8707 3.4456 0.0487 14.2500 los rate 0.0882 0.1767 0.0000 0.0000 1.2592 population 122705.25 155567.12 22430.00 550.00 359183.00 PROC FREQ35 was used to perform the calculati ons for the categorical variables. The proportion of each variable level is reported in table 4-2. Step Two: Establishment of Balanced Control and Experimental Groups Using Three Propensity Score Methods Pre-Propensity Score Analysis Twelve variables were identified in the administrative data as describing personal and practice characteristics of GPs. GP age, sex, participation in a previous influenza AD

PAGE 53

38 intervention, place of initial licensure, base line COX-2 prescribing rate, birthplace and percent of patients diagnosed with OA describe personal characteristics. The percent of elderly patients, total number of patients, av erage income of county where the practice is located, population size of the community where the practice is located and average hospital length of stay for patients describe the GPs practice characteristics. Table 4-2. Descriptive Statistics for Categorical Variables in the PS Model. Variable Level Proportion AD = 0 (n=265) AD = 1 (n=231) Sex Female 0.3057 0.2987 Male 0.6943 0.7013 Flu AD Yes 0.1585 0.7273 No 0.8415 0.2727 License Canada 0.1208 0.1732 Nova Scotia 0.6717 0.6623 Other 0.2075 0.1645 Birth place Nova Scotia 0.4830 0.4545 Canada East 0.1245 0.1255 Canada Centre 0.0679 0.0736 Canada West 0.0264 0.0390 Other 0.2981 0.3074 Table 4-3 contains the prePS variable values which include; means and standard deviations for continuous variables, F -statistics (square of t-test for continuous variables and F for coefficient estimate from PROC GENMOD35 for categorical variables), p values, coefficient estimates for the main e ffect of the intervention for the categorical variables, and the odds ratio fo r the main effect. These valu es were used in subsequent tables to calculate percent bias reduction for each PS technique. The pre-PS analysis indicates that there ar e five variables that are not balanced. These variables are of the greatest concern si nce the goal of PS methods is to balance the groups on measured covariates.12, 16 The five variables that show significant differences at the alpha = 0.05 level will be collectively referred to as th e variables of concern (VOC)

PAGE 54

39 and they are; the percent of elderly patie nts with a diagnosis of OA (% OA dx), the average income of the county in which the physicians practice is located (aver income), the average hospital length of stay rate for elderly patients per physician (los rate), the population of the community in which the phys icians practice is located (population), and physician participation in a prev ious influenza AD service (flu AD). Table 4-3. Pre-PS Univariate Anal ysis for Included Variables. Pre-Propensity Score Values (t-test and proc genmod) AD = 0 (n = 265) AD = 1 (n = 231) OR Variable mean std dev mean std dev F p value B (exp B) % OA dx* 0.0913 0.1071 0.0719 0.0598 9.9191 0.0116 GP age 47.3 9.8 45.7 9.2 9.2191 0.0678 % elderly 0.1910 0.1160 0.1772 0.0788 8.9591 0.1182 total # pt 1054 439 1038 419 7.8191 0.6700 aver income* 27689 4683 25833 4488 11.8791 0.0001 BL rate 3.60 2.70 3.98 2.87 5.8991 0.1356 los rate* 0.0453 0.1449 0.0882 0.1767 4.4191 0.0031 population* 183342 162415 122705 155567 11.6191 0.0001 sex 0.0300 0.8663 -0.0330 0.9675 flu AD* 140.1500 0.0001 -2.6503 0.0706 license 3.3600 0.0669 0.3460 1.4134 birth place 0.1100 0.7415 -0.0553 0.9462 variables that show significant di fferences at the alpha = 0.05 level Quintile PS Method Analysis The distribution of GPs within the quintile s is reported in tabl e 4-4. Quintile one represents the GPs with the lowest PS s (lowest propensity to volunteer for the intervention) and quintile five represents the GPs with the hi ghest PSs (highest propensity to volunteer for the intervention). The ta ble is consistent with the expected PS distribution with fewer subjects in the high propensity quintile for the control group and fewer subjects in the low propensit y quintile for the intervention group. The results for the quintile method were generated using PROC GENMOD35 and are reported in table 4-5. The main effect co lumn represents the main effect of the AD

PAGE 55

40 variable and the interaction effect column represents the effect of the AD by quintile interaction. The quintile method resulted in no statistically signifi cant difference between groups on all five VOC while maintaining balance on the rest of the covariates. Table 4-4. Physician Distribution by Quintile Quintile # Intervention Control # of GPs 1 86 13 99 2 77 22 99 3 65 35 100 4 24 75 99 5 13 86 99 TOTAL 265 231 496 Table 4-5. Quintile Method Re gression Analysis Results. Quintile Method lsmean Main Effect Interaction Effect Variable AD = 0 (n = 265) AD = 1 (n = 231) F p F p B OR (exp B) % bias reduction % OA dx* 0.0827 0.0783 0.20 0.6562 1.74 0.1403 77.32 GP age 47.2 47.1 0.01 0.9313 1.01 0.4029 93.75 % elderly 0.1918 0.1808 0.93 0.3351 2.04 0.0880 20.29 total # pt 1045 1031 0.09 0.7673 0.22 0.9256 12.50 aver income* 26757 26693 0.02 0.8900 1.34 0.2526 96.55 BL rate 3.58 3.68 0.10 0.7479 0.90 0.4616 72.89 los rate* 0.0491 0.0677 1.05 0.3058 0.76 0.5529 56.64 population* 149548 148905 0.00 0.9670 1.27 0.2797 98.94 sex 0.00 0.9818 0.00 0.9801 -0.0130 0.9871 60.21 flu AD* 0.35 0.5537 xx+ xx+ -0.3723 0.6891 66.55 license 3.26 0.0709 3.28 0.0700 -0.9702 0.3790 -50.22 birth place 0.01 0.9372 0.01 0.9295 -0.0381 0.9626 30.51 Average** 82.36 variables that were not significant at th e alpha = 0.05 level in the pre-PS model ** average % bias reduction for variables with significan t differences in the pre-PS model (excluding flu AD) + estimates not available (see table 4-6 for explanation) The interaction effect (AD*quintile) was not significant for four of the five VOC however, the flu AD variable exhibited an al most complete separation of data points (table 4-6) and as such the inte raction effect was not estimated. The distribution of the flu AD variable on the PS was problematic for all three PS methods (figure 4-1). Theref ore, the reported average per cent bias reduction on the VOC

PAGE 56

41 does not include the flu AD variable. The aver age percent bias reduction for the quintile method is 82%. It is evident at this point that the flu AD variable will have to be included in the outcome models regardless of the PS method chosen. Table 4-6. Distribution of Influenza AD Participants by Propensity Score Quintile Flu AD Participation Quintile Total 1 2 3 4 5 No 99 99 88 0 0 286 Yes 0 0 12 99 99 210 Regression on the Propensity Score Method Analysis The results for the regression on the PS method were generated using PROC GENMOD35 and are reported in table 4-7. This method was successful in balancing three of the five VOC while maintaining balance on the rest of the covariates. The average percent bias reduction on the VOC (flu AD excluded) is 99%. The variable, population, re tained a significance level less than 0.05 and it also exhibited a significant interac tion effect (population*AD) at the alpha = 0.05 level. The variable aver income showed a nonsignificant main effect with a p value > 0.05 however, the interaction eff ect (aver income*AD) is less than the 0.05 level. The separation of data points for the flu AD variab le on the PS was again evident. Figure 4-1 shows the distribution of flu AD on PS (stra tified at 0.05 intervals) This separation precluded the model from estimating main and interaction effects for flu AD. Greedy Matching Method Analysis The results for the greedy matching method were generated using PROC GENMOD35 and are reported in table 4-8. This method was successful in balancing four

PAGE 57

42 of the five VOC while maintaining balance on the remainder of th e covariates. The average percent bias reduction on th e VOC (flu AD excluded) is 75%. Table 4-7. Regression on PS Method Analysis Results. Regression on Propensity Score Method lsmean Main Effect Interaction Effect Variable AD = 0 (n=265) AD = 1 (n=231) F P F p B OR (exp B) % bias reduction % OA dx* 0.0770 0.0772 2.19 0.1394 3.07 0.0802 98.97 GP age 47.0 46.9 0.90 0.3441 1.30 0.2548 97.50 % elderly 0.1819 0.1819 0.45 0.5030 0.63 0.4291 100.00 total # pt 1036 1037 0.33 0.5637 0.47 0.4924 93.75 aver income* 26531 26530 2.83 0.0934 3.92 0.0482 99.95 BL rate 3.68 3.69 0.61 0.4338 0.89 0.3462 97.63 los rate* 0.0613 0.0620 0.32 0.5697 0.48 0.4872 98.37 population* 142500 142661 4.23 0.0402 5.90 0.0155 99.73 sex 0.22 0.6392 0.31 0.5746 0.2073 1.2304 -609.62 flu AD* xx+ xx+ xx+ xx+ 728.77 license 2.48 0.1152 3.48 0.0622 -0.6744 0.5095 -18.66 birth place 0.03 0.8588 0.04 0.8377 -0.0674 0.9348 -21.15 Average** 99.25 variables that showed significant differences at the alpha = 0.05 level in the pre-PS model ** average % bias reduction for variables that were not significant in the pre-PS model (excluding flu AD) + estimates not available (see figure 4-1 for explanation) Graph of Propensity Score vs. Frequency of Flu AD Participants0 10 20 30 40 50 60 700.025 0. 1 25 0 225 0. 32 5 0.425 0. 5 25 0 625 0.725 0. 82 5 0.925Propensity Score# of Flu AD Participan t flu AD = no flu AD = yes Figure 4-1. Frequency of Influenza AD Participants by Propensity Score.

PAGE 58

43 The flu AD variable estimates were not obt ained for the same reasons described in the regression on the PS method section. With the exception of the flu AD variable, the greedy method balanced all variable s and associated in teraction terms. Table 4-8. Greedy Matching Method Analysis Results. "Greedy Matching" Method lsmean Main Effect Interaction Effect Variable AD = 0 (n=104) AD = 1 (n=104) F p F p B OR (exp B) % bias reduction % OA dx* 0.0722 0.0788 0.18 0.6712 0.01 0.9330 65.98 GP age 46.4 47.0 2.10 0.1492 1.99 0.1598 62.50 % elderly 0.1768 0.1876 0.39 0.5328 0.05 0.8299 21.74 Total # pt 1074 1008 1.25 0.2650 0.37 0.5410 -312.50 aver income* 26535 26300 0.23 0.6301 0.12 0.7308 87.34 BL rate 3.75 3.78 0.85 0.3573 1.25 0.2651 92.37 los rate* 0.0456 0.0535 1.04 0.3097 0.89 0.3471 81.59 population* 154497 132123 2.06 0.1527 1.14 0.2862 63.10 Sex 0.53 0.4654 1.26 0.2619 0.4266 1.5320 -1538.99 flu AD* xx+ xx+ xx+ xx+ 446.3283 License 1.35 0.2459 1.04 0.3077 -0.6730 0.5102 -18.49 birth place 1.79 0.1809 1.31 0.2520 -0.6828 0.5052 -819.72 Average** 74.50 variables that showed significant differences at the alpha = 0.05 level in the pre-PS model ** average % bias reduction for variables that were not significant in the pre-PS model (excluding flu AD) + estimates not available The greedy matching method resulted in a decrease in total sample size from 496 (sample size of the two previous methods) to 208. This represents a decrease in sample size of 58%. The eliminated GPs had PSs th at were predominantly in the highest or lowest ranges of the distribution. The e limination of these GPs could affect the generalizability of the study since only the GPs who are in the midrange of the PS distribution would be left in the study. Selection of a Preferred Propensity Score Method The selection of a preferred PS method wa s carried out by measuring each of the three methods against the following two criteria; the resulting sample size after application of the PS method, and

PAGE 59

44 the PS methods ability to ad just for bias on the VOC. A major disadvantage of the greedy matc hing method is the reduction in sample size resulting from the discarding of subjects that are not matched. In this case the sample size is reduced by 58% which possibl y results in a loss of power to detect significance in the main effects of the outcome models and a loss of generalizability of the findings. Since the greedy matching method does not show advantages over the regression on the PS method in terms of adjusting for bias on the covariates it is considered less desirable than the regression on the PS method and will not be selected as the PS method for inclusion in the outcome models. The regression on PS method was responsible for the greatest adjustment for bias between groups on all of the VOC (figure 42). The average reduction in bias for the regression on the PS method was 99% versus 82% for the quintile method. With this dataset the regression on the PS method is preferred and it is the method that will be applied to the outcome analyses. It is important to note that the failure to adjust for bias on the flu AD variable still exis ts and as such the fl u AD variable will be included in the outcome models. Exploratory Analysis of the Propensity Sc ore Methods Effect on Adjusting for Bias on Unmeasured Variables The purpose of this exploratory analysis is to determine whether any one PS method is better at reducing bias on variables that are not included in the PS model and are, therefore, considered unmeasured. The c -statistic is a measure of the models ability to discrimi nate between groups. The c -statistic for the full model is 0.832 which can be interpreted as follows; if one randomly select one subject from each AD group the model will accurately predict the

PAGE 60

45 group from which the subjects or iginated 83.2% of the time. With the exception of the models dealing with the exclusi on of the flu AD variable, the c -statistic remains stable for all of the PS models. The range is from 0.830 to 0.835 (table 4-9). Percent Reduction in Bias on Unbalanced Variables (VOC)0.00 20.00 40.00 60.00 80.00 100.00 % OA dxAver Income los ratePopulationAverage Variable% Reduction in Bias Quintile Regr on PS Greedy Match Figure 4-2. Comparison of PS meth ods Ability to Reduce Bias on VOC. Table 4-9. Quintile Method Result s for Excluded Variable Models. Quintile Method lsmean Main Effect Interaction Effect Excluded Variable c AD = 0 (n=265) AD = 1 (n=231) F p F p B OR (exp B) % bias reduction % OA dx* 0.833 0.0935 0.0708 4.92 0.0207 0.79 0.5298 -17.01 GP age 0.833 46.8 47.5 0.37 0.5426 0.82 0.5112 56.25 % elderly 0.831 0.1960 0.1754 3.10 0.079 1.75 0.1374 -49.28 total # pt 0.833 1085 1006 2.55 0.1111 2.86 0.0231 -393.75 aver income* 0.834 26777 26652 0.07 0.7862 1.39 0.2361 93.27 BL rate 0.834 3.6 3.7 0.11 0.7418 0.80 0.5274 73.68 los rate* 0.833 0.0471 0.0730 1.97 0.1615 0.81 0.5214 39.63 population* 0.832 153438 148662 0.08 0.7746 1.57 0.1805 92.12 sex 0.835 0.01 0.9217 0.25 0.6197 -0.0593 0.9424 -77.37 flu AD* 0.662 14.5 0.0001 1.09 0.2957 -2.0411 0.1299 6.38 license 0.830 0.83 0.3673 0.00 0.0896 -0.5001 0.6065 4.81 birth place 0.835 0.10 0.7528 0.02 0.8820 -0.1524 0.8586 -162.75 Average** 42.88 variables that showed significant differences at the alpha = 0.05 level in the pre-PS model ** average % bias reduction for variables that show ed significant differences in the pre-PS model

PAGE 61

46 There are three c -statistics that are worth noting. The first is the c -statistic that is generated for the model when the flu AD variable is removed. It has been noted that there exists an almost complete separation of data for the flu AD variable on the PS so when the flu AD variable is excluded from the model the ability of the model to discriminate decreases from 0.834 to 0.662. The other two are the c -statistics associated with sex and birth place. These two variable s have the distinction of being the most closely balanced variables in the pr e-PS analysis (table 4-3) with p -values of 0.8663 and 0.7415 respectively. The PS model c -statistics when these variables are excluded is equal to 0.835 in both cases. This value is greater than the c -statistic for the full model thereby indicating that the inclusion of these variables in the PS model decreases the models discriminative ability. The complete results from the reduced PS models are reported in tables 4-9 through 4-11. The analysis of the reduced models e ffects on balancing the VOC is summarized in figure 4-3. Figure 4-3 shows that no one PS method systematically reduces bias on unmeasured variables to a greater extent than the others. Regression on PS does, on average, reduce bias on the VOC to the greatest degree. The summary of PS models effects (figure 4-3) shows that bias between groups on unmeasured variables can be reduced by PS me thods. The correlation matrix between the VOC and the PS covariates was calculated and reported in table 4-12. Table 4-12 shows limited correlation (less than 0.30) between th e VOC and the PS covari ates in all cases except one. The one exception is the corre lation between populat ion (population of community where the GP practice is located) and aver income (average income for county where GP practice located) which was 0.91. The correlation between population

PAGE 62

47 and aver income is associated with the highe r reduction in bias for those variables when they are not included in the PS model. Table 4-10. Regression on PS Results for Excluded Variable Models. Regression on Propensity Score Method lsmean Main Effect Interaction Effect Excluded Variable c AD = 0 (n=265) AD = 1 (n=231) F p F p B OR (exp B) % bias reduction % OA dx* 0.833 0.0901 0.0736 0.74 0.3909 0.00 0.9540 14.95 GP age 0.833 46.5 47.2 1.14 0.2854 0.72 0.3974 56.25 % elderly 0.831 0.1918 0.1777 0.27 0.6065 0.04 0.8445 -2.17 total # pt 0.833 1061 1018 0.69 0.4080 0.15 0.6946 -168.75 aver income* 0.834 26567 26510 2.83 0.0929 3.65 0.0566 96.93 BL rate 0.834 3.7 3.7 0.57 0.4515 0.96 0.3286 100.00 Los rate* 0.833 0.0513 0.0684 0.19 0.6641 1.26 0.2618 60.14 population* 0.832 146919 139289 4.70 0.0307 5.12 0.0240 87.42 sex 0.835 0.06 0.7993 0.68 0.4108 0.1155 1.1224 -277.17 Flu AD* 0.662 3.16 0.0756 2.12 0.1456 -1.4258 0.2403 18.26 license 0.830 1.41 0.2348 4.59 0.0322 -0.5203 0.5943 1.87 birth place 0.835 0.09 0.7695 0.02 0.8911 -0.1117 0.8943 -96.45 Average** 55.54 variables that showed significant differences at the alpha = 0.05 level in the pre-PS model ** average % bias reduction for variables that show ed significant differences in the pre-PS model Table 4-11. Greedy Matching Resu lts for Excluded Variable Models. "Greedy Matching" Method lsmean Main Effect Interaction Effect Excluded Variable ni (i=0,1) AD = 0 AD = 1 F p F p B OR (exp B) % bias reduction % OA dx* 106 0.0969 0.0707 4.36 0.0380 0.00 0.9677 -35.05 GP age 101 46.2 47.5 1.40 0.2381 0.59 0.4450 15.69 % elderly 105 0.1944 0.1801 0.41 0.5205 0.01 0.9365 -3.62 total # pt 103 1049 1005 0.61 0.4371 0.17 0.6676 -175.00 aver income* 105 26875 26123 1.02 0.3136 0.22 0.6419 59.48 BL rate 103 3.7 3.7 0.77 0.3806 1.01 0.3169 97.11 Los rate* 105 0.0352 0.0614 1.09 0.2967 0.06 0.8145 38.93 population* 104 154497 132123 2.06 0.1527 1.14 0.2862 63.10 sex 104 0.00 0.9674 0.24 0.6260 0.0119 1.0120 63.12 Flu AD* 190 2.00 0.1570 1.34 0.2471 -0.7168 0.4883 44.94 license 104 5.04 0.0248 9.09 0.0026 -0.6618 0.5159 -17.10 birth place 105 0.00 0.9787 0.28 0.5943 0.0069 1.0069 87.22 Average** 34.28 variables that showed significant differences at the alpha = 0.05 level in the pre-PS model ** average % bias reduction for variables that show ed significant differences in the pre-PS model

PAGE 63

48 Percent Reduction in Bias on Unbalanced and Unmeasured Variables-40.00 -20.00 0.00 20.00 40.00 60.00 80.00 100.00 % OA dxAver Incomelos ratePopulationflu ADAverage Variable% Bias Reduction Quintile Regr on PS Greedy Match Figure 4-3. Summary of PS Models Effects on Reducing Bias on the VOC Table 4-12. Correlation Matrix Between VOC and PS Covariates. Correlation Between VOC and All PS Covariates Covariate VOC los rate % OA dx population aver income flu AD BL rate -0.06 -0.02 -0.22 -0.26 0.00 los rate 1.00 -0.01 -0.05 -0.05 0.03 % OA dx -0.01 1.00 0.01 0.01 0.01 total # pt -0.12 -0.10 0.00 0.02 0.02 % elderly -0.02 0.26 0.09 0.09 -0.03 sex -0.08 0.04 -0.13 -0.13 0.03 flu AD 0.03 0.01 -0.14 -0.17 1.00 population -0.05 0.01 1.00 0.91 -0.14 GP age -0.05 0.07 0.06 0.06 -0.09 aver income -0.05 0.01 0.91 1.00 -0.17 The effect of the correlation between the PS and the VOC and the reduction in bias was tested. The correlation between the PS a nd the VOC was calculate d and scatter plots were compiled to display the re sults graphically in figure 4-4. The absolute values of the correlations ranged from 0.182 to 0.329. The absolute value of the correlations was plotted against the percent bi as reductions on the VOC for each of the three PS methods (figure 4-5). The results from figure 4-5 show an overall

PAGE 64

49 effect of increasing percent bias reduction with increasing absolute correlation between the PS and the excluded variable. Scatter Plot: Propensity Score vs. Percent of Patients with OA Diagnosis (Rho = -0.182)0 10 20 30 40 50 60 70 80 00.10.20.30.40.50.60.70.80.91 Propensity Score Scatter Plot: Propensity Score vs. Length of Stay Rate (Rho = 0.220) 0 0.2 0.4 0.6 0.8 1 1.2 1.4 00.20.40.60.81 Propensity Score Scatter Plot: Propensity Score vs. Community Population (Rho = 0.311)0 50000 100000 150000 200000 250000 300000 350000 400000 00.20.40.60.81 Propensity Score Scatter Plot: Propensity Score vs. Average County Income (Rho = -0.329) 20000 22000 24000 26000 28000 30000 32000 34000 00.10.20.30.40.50.60.70.80.91 Propensity Score Figure 4-4. Scatterplots of Propensit y Score Versus Unbalanced Variables Step 3: Primary Outcome Analysis Model Development The analysis of the primary outcome, the effect of the OA AD intervention on the COX-2 utilization rates was carri ed out using a repeated measures model on longitudinal

PAGE 65

50 data (PROC GENMOD35). There were six experimental time periods over which the outcomes measures were assessed (figure 3-2). Correlation (PS vs. Excluded Var.) vs. Percent Bias Reduction (by PS Method)-40.00 -20.00 0.00 20.00 40.00 60.00 80.00 100.00 0.150.170.190.210.230.250.270.290.310.330.35 Rho% Bias Reduction Quintile Regr on PS Greedy Match Figure 4-5. Line Graph Comparing Correlations a nd Percent Bias Reduction The primary outcome measure, the change in COX-2 prescribing from baseline, was calculated for each physician by aggregatin g all of the COX-2 prescription claims for all of the elderly patients in the physicians panel and dividing by the number of elderly patients in the panel. The resulting ra te, number of COX-2 DDDs per patient per physician was subtracted from the baseline pres cribing rate to yield a measure of change in COX-2 prescribing. The primary outcome model included the va riables intervention participation (AD), the PS (pr), the time period in which the m easurement took place (period), participation in a previous influenza AD service (flu AD), the baseline COX-2 prescribing rate (BL rate), and the number of elderly patients in the GPs panel (# elderly). The model is depicted in figure 4-6. The variables were included for the following reasons. The PS

PAGE 66

51 variable represents the outcome from the PS an alysis, the period variab le controls for the longitudinal changes, the flu AD variable was not successfully balanced by the PS method, and the baseline COX-2 rate and number of elderly patients control for the GPs pre-intervention prescrib ing behavior and practice size respectively. Figure 4-6. Primary Outcome Model Between Group Results The significance level of each variable fr om the primary outcome model (figure 46) is listed in table 4-13. The values of the coefficient estimates in GEE are not interpreted in the same manner as GLM models37 and as such the values of the coefficient estimates are not reported in the results ta bles. A more in-depth discussion of the interpretation of GEE results is includ ed in the discussions in chapter five. The between groups effect of the interventi on is interpreted from the value of the z statistic for the AD variable. The z value of 0.85 and associated p -value of 0.3976 indicates that the main intervention effect over the entire post-inte rvention period is not statistically significant. The model in figure 4-6 was also used to determine between group differences in the pre-intervention time peri ods (period = 1, 2). The z statistics and associated p -values of each variable are listed in table 4-14. The pre-intervention results are interpreted in the Y = 0 + 1(X1) + 2(X2) + 3(X3) + 4(X4) + 5(X5) + 6(X6) Where; Y = change in COX-2 utilization rate (periods 3 to 6 (post-intervention)), X1 = physician participation in the intervention (0 = no, 1 = yes), X2 = PS (range from 0 to 1), X3 = experimental time period (period = 3,4,5,6), X4 = physician participation in the influenza AD service (0 = no, 1 = yes), X5 = physician baseline COX-2 rate ( DDD / patient, (period = 2)), X6 = number of patients in th e GPs practice >65 years old

PAGE 67

52 same manner as the post-int ervention results. The z statistic and p -value for the AD variable are 0.88 and 0.3775 respectively. The p -value indicates that the groups are not significantly different on the outcome measure in the pre-intervention periods at the alpha = 0.05 level. Table 4-13. Primary Outcome Model Results (Periods = 3,4,5,6). Primary Outcome Model Results for Post-intervention Periods (COX-2 Prescribing Rates) Effect Z p-value AD (AD = no) 0.85 0.3976 PS 0.84 0.4023 period 2.69 0.0072 flu AD (flu AD = no) 1.21 0.2255 BL rate -10.68 <0.0001 # elderly 1.64 0.1017 Table 4-14. Primary Outcome Model Results (Periods = 1,2). Primary Outcome Model Results for Pre-intervention Periods (COX-2 Prescribing Rates) Effect Z p-value AD (AD = no) 0.88 0.3775 PS 0.31 0.7588 period 0.38 0.7018 flu AD (flu AD = no) 0.23 0.8170 BL rate -14.45 <0.0001 # elderly -0.48 0.6313 Table 4-15 depicts the least square mean s for the two groups (AD = yes and AD = no) for each of the six experimental time periods The least square mean values are also presented in a graph in figure 4-7. Table 4-16 depicts the unadjusted means and standard deviations for the two groups (AD = yes and AD = no) for each of the six experimental time periods. A positive value indicates that the prescribing rate ha s increased from the baseline rate by the amount indicated and a negative value indicates a decrease in the prescribing rate from baseline. The unadjusted mean values are also presented in a graph in figure 4-8.

PAGE 68

53 Table 4-15. Least Square Means for Change in COX-2 Rates by Group (DDDs/patient). AD group period 1 2 3 4 5 6 0 0.0516 0 0.2275 -0.1778 0.2471 0.4178 1 -0.2136 0 -0.5315 -0.0018 0.261 0.1457 Primary Outcome: Change in COX-2 Rates (adjusted)-1 -0.5 0 0.5 123456 Time PeriodDDD change /patient/GP Intervention = No Intervention = Yes Figure 4-7. Least Square Means fo r Change in COX-2 Rates by Group Table 4-16. Unadjusted Means for Change in COX-2 Rates by Group (DDDs/patient). Period AD group 0 1 Mean Std Dev Mean Std Dev 1 0.1587 3.4287 -0.3026 3.0709 2 0.0000 0.0000 0.0000 0.0000 3 0.3396 3.9059 -0.5321 3.4410 4 0.0236 3.6700 -0.1257 3.7358 5 0.5266 3.7285 -0.0008 3.9072 6 0.6454 3.8566 0.1281 3.9248 Within Group (Longitudinal) Results The within group models were the same as the between group model in figure 4-6 except that the AD group variable is replaced by a prepos t variable which measures significant within group differences between change in COX-2 rates pre-intervention and post-intervention. The model is run two times; once including only the intervention group and once including only the control group. The z statistic value and associated

PAGE 69

54 significance level (p -value) of each variable are listed in table 4-17 for the intervention group and table 4-18 for the control group. Primary Outcome: Change in COX-2 Rates (unadjusted) -0.6000 -0.4000 -0.2000 0.0000 0.2000 0.4000 0.6000 0.8000 123456 Time PeriodDDD change /patient/GP Intervention = No Intervention = Yes Figure 4-8. Unadjusted Means for Change in COX-2 Rates by Group The within group effect of the intervention is interpreted from the values of the z statistic and significance level of the prepost variable. Fo r the intervention group, the z and p -values of -2.34 and 0.0191 respectively indicat es that the within group effect is significant at the alpha = 0.05 leve l. For the control group, the z statistic and p -value of -0.22 and 0.8273 respectively indicates that th e within group effect is not significant at the alpha = 0.05 level. Table 4-17. Primary Outcome Model Results (AD = yes). Primary Outcome Results for the Intervention Group (COX-2 Prescribing Rates) Effect Z p-value PS 0.04 0.9708 period 2.82 0.0049 prepost -2.34 0.0191 flu AD (flu AD = no) 0.49 0.6217 BL rate -9.74 <0.0001 # elderly 0.63 0.5271

PAGE 70

55 Step 4: Secondary Outcome Analyses Misoprostol Utilization Rates Model development The analysis of the secondary outcome, th e effect of the OA AD intervention on the misoprostol utilization rate was carried out using the same methods as the primary outcome analysis with the data for misoprostol utilization substituted for the COX-2 utilization data (figure 4-9). Table 4-18. Primary Outcome Model Results (AD = no). Primary Outcome Results for the Control Group (COX-2 Prescribing Rates) Effect Z p-value PS 1.32 0.1881 period 1.31 0.1910 prepost -0.22 0.8273 flu AD (flu AD = no) 0.95 0.3412 BL rate -8.68 <0.0001 # elderly 1.10 0.2727 Figure 4-9. Secondary Outcome M odel for Misoprostol Utilization Between group results The z statistic and the significance level (p -value) of each variable from the secondary misoprostol outcome model (f igure 4-9) are liste d in table 4-19. Y = 0 + 1(X1) + 2(X2) + 3(X3) + 4(X4) + 5(X5) + 6(X6) Where; Y = change in misoprostol utilization rate (periods 3 to 6 (post-intervention)), X1 = physician participation in the intervention (0 = no, 1 = yes), X2 = PS (range from 0 to 1), X3 = experimental time period (period = 3,4,5,6), X4 = physician participation in the influenza AD service (0 = no, 1 = yes), X5 = physician baseline misoprostol ra te (DDD / patient, (period = 2)), X6 = number of patients in th e GPs practice >65 years old

PAGE 71

56 The between group effect of the inte rvention is interpreted from the z statistics and associated p -value of the AD variable. The z statistic and p -value of -0.87 and 0.3866 respectively indicate that the effect is not signifi cant at the alpha = 0.05 level. Table 4-19. Secondary Misoprostol Ou tcome Model Results (Periods = 3,4,5,6). Secondary Outcome Model Results for Post-intervention Periods 3 to 6 (Change in Misoprostol Prescribing Rates) Effect Z p-value AD (ad = 0) -0.87 0.3866 PS -0.61 0.5412 period 0.96 0.3359 flu AD (flu AD = 0) -0.53 0.5943 BL rate -6.31 <0.0001 # elderly -0.24 0.8091 The model in figure 4-9 was used to de termine intervention effects on each post intervention time period. None of the post intervention (ana lyzed individually) showed significant between group differenc es at the alpha = 0.05 level. The model in figure 4-9 was used to dete rmine between group differences in the pre-intervention time periods (period = 1, 2). The z statistic and associated significance level of each variable is listed in table 420. The results are interpreted in the same manner as the post-intervention results. The z statistic and p -value for the AD variable are -0.22 and 0.8269 respectively. The p -value indicates that the groups are not significantly different in th e pre-intervention periods at the alpha = 0.05 level. Table 4-20. Secondary Misoprostol Ou tcome Model Results (Periods = 1,2). Secondary Outcome Model Results for Post-intervention Periods 1 and 2 (Change in Misoprostol Prescribing Rates) Effect Z p-value AD (ad = 0) -0.22 0.8269 PS 1.20 0.2308 period 0.28 0.7758 flu AD (flu AD = 0) 1.18 0.2396 BL rate -4.55 <0.0001 # elderly 0.58 0.5612

PAGE 72

57 Table 4-21 depicts the least square mean s for the two groups (AD = yes and AD = no) for each of the six experimental time periods. The least square mean values are also presented in a graph in figure 4-10. Table 4-21. Least Square Means for Change in Misoprostol Rate by Group (DDDs/patient). Secondary Outcome (Misoprostol) Least Square Means by AD Group AD group Period 1 2 3 4 5 6 0 -0.0191 0.0000 0.0172 0.0516 0.0390 0.0388 1 -0.0127 0.0000 0.0383 0.0743 0.0604 0.0652 Secondary Outcome: Change in Misoprostol Rates (adjusted)0.0 0.0 0.0 0.0 0.0 0.1 0.1 123456 Time PeriodDDD change /patient/GP Intervention = No Intervention = Yes Figure 4-10. Least Square Means for Change in Misoprostol Rates by Group. Table 4-22 depicts the unadjusted means and standard deviations for the two groups (AD = yes and AD = no) for each of the six experimental time periods. The unadjusted mean values are also pr esented in a graph in figure 4-11. Within group (longitudinal) results The within group model was the same as the between group model in figure 4-9 except that the AD group variable is replaced by a prepos t variable which measures within group differences between change in misoprostol rates preintervention and postintervention. The model is run two times; once including only the intervention group and

PAGE 73

58 once including only the control group. The z statistic and the associated significance level (p -value) of each variable are listed in table 4-23 for the intervention group and table 4-24 for the control group. Table 4-22. Unadjusted Means and Standard Deviations for Change in Misoprostol Rate by Group (DDDs/patient). Period AD group 0 1 Mean Std Dev Mean Std Dev 1 -0.0153 0.2877 0.0096 0.2969 2 0.0000 0.0000 0.0000 0.0000 3 0.0032 0.2693 0.0437 0.2581 4 0.0586 0.3289 0.0570 0.3120 5 0.0484 0.3750 0.0497 0.3472 6 0.0235 0.3782 0.0850 0.4005 Secondary Outcome: Change in Misoprostol Rates (unadjusted) -0.04 -0.02 0.00 0.02 0.04 0.06 0.08 0.10 123456 Time PeriodDDD change/ patient/GP Intervention = no Intervention = yes Figure 4-11. Unadjusted Means for Change in Misoprostol Rates by Group. Table 4-23. Secondary Misoprostol Outcome Model Results (AD = yes). Secondary Outcome Model Results for All Periods (Intervention Group) (Change in Misoprostol Prescribing Rates) Effect Z p-value PS -0.58 0.5594 period 1.65 0.0990 prepost 0.25 0.8075 flu AD (flu AD = 0) -0.30 0.7612 BL rate 1.23 0.2195 # elderly 0.55 0.5802

PAGE 74

59 Table 4-24. Secondary Misoprostol Outcome Model Results (AD = no). Secondary Outcome Model Results for All Periods (Control Group) (Change in Misoprostol Prescribing Rates) Effect Z p-value PS -0.01 0.9921 period 0.75 0.4523 prepost 1.00 0.3176 flu AD (flu AD = 0) -0.27 0.7888 BL rate 4.69 <0.0001 # elderly 1.59 0.1109 The within group effect of the intervention is interpreted from the values of the z statistic and associated p -value of the prepost variable. For the intervention and control groups, z statistics and the p -values of 0.25, 0.8075 and 1.00, 0.3176 respectively indicates that the within group effect is not statistically si gnificant for both groups at the alpha = 0.05 level. PPI Utilization Rates Model development The analysis of the secondary outcome, th e effect of the OA AD intervention on the PPI utilization rates was carri ed out using the same methods as the primary outcome analysis with the data for PPI utilization s ubstituted for the COX-2 utilization data (figure 4-12). Figure 4-12. Secondary PPI Outcome Model Y = 0 + 1(X1) + 2(X2) + 3(X3) + 4(X4) + 5(X5) + 6(X6) Where; Y = change in PPI utilization rate (periods 3 to 6 (post-intervention)), X1 = physician participation in the intervention (0 = no, 1 = yes), X2 = PS (range from 0 to 1), X3 = experimental time period (period = 3,4,5,6), X4 = physician participation in the influenza AD service (0 = no, 1 = yes), X5 = physician baseline PPI rate (DDD / patient, (period = 2)), X6 = number of patients in th e GPs practice >65 years old

PAGE 75

60 Between group results The z statistic and the significance level (p -value) of each variable from the secondary PPI outcome model (figur e 4-12) are listed in table 4-25. Table 4-25. Secondary PPI Outcom e Model Results (Periods = 3,4,5,6). Secondary Outcome Model Results for Post-intervention Periods 3 to 6 (Change in PPI Prescribing Rates) Effect Z p-value AD (ad = 0) -0.27 0.7906 PS 1.09 0.2755 period 1.43 0.1519 flu AD (flu AD = 0) 1.45 0.1478 BL rate -2.92 0.0035 # elderly -1.74 0.0813 The between group effect of the inte rvention is interpreted from the z statistics and associated p -value of the AD variable. The z statistic and p -value of -0.27 and 0.7906 respectively indicate that the effect is not signifi cant at the alpha = 0.05 level. The model (figure 4-12) was used to de termine intervention effects on each post intervention time period. None of the post intervention (ana lyzed individually) showed significant between group differenc es at the alpha = 0.05 level. The model in figure 4-12 was used to determine between group differences in the pre-intervention time periods (period = 1, 2). The z statistic and associated significance level of each variable is listed in table 426. The results are interpreted in the same manner as the post-intervention results. The z statistic and p -value for the AD variable are 0.13 and 0.8989 respectively. The p -value indicates that the groups are not significantly different in th e pre-intervention periods at the alpha = 0.05 level. Table 4-27 depicts the least square mean s for the two groups (AD = yes and AD = no) for each of the six experimental time periods The least square mean values are also presented in a graph in figure 4-13.

PAGE 76

61 Table 4-26. Secondary PPI Outcom e Model Results (Periods = 1,2). Secondary Outcome Model Results for Post-intervention Periods 1 and 2 (Change in PPI Prescribing Rates) Effect Z p-value AD (ad = 0) 0.13 0.8989 PS 1.10 0.2726 period 0.14 0.8911 flu AD (flu AD = 0) 1.08 0.2818 BL rate -5.61 <0.0001 # elderly 0.04 0.9700 Table 4-27. Least Square Means for Cha nge in PPI Rates by Group (DDDs/patient). Secondary Outcome (PPI) Leas t Square Means by AD Group AD group period 1 2 3 4 5 6 0 -0.0388 0 0.3194 0.5271 0.507 0.4842 1 -0.0553 0 0.3675 0.5513 0.555 0.4982 Secondary Outcome: Change in PPI Rates (adjusted)-0.2 0 0.2 0.4 0.6 123456 Time PeriodDDD change / patient/GP. Intervention = No Intervention = Yes Figure 4-13. Least Square Means for Change in PPI Rates by Group. Table 4-28 depicts the unadjusted means and standard deviations for the two groups (AD = yes and AD = no) for each of the six experimental time periods. The unadjusted mean values are also pr esented in a graph in figure 4-14. Within group (longitudinal) results The within group model was the same as the between group model in figure 4-12 except that the AD group variable is replaced by a prepos t variable which measures

PAGE 77

62 within group differences between change in PPI rates pre-intervention and postintervention. The model is run two times; once including only the intervention group and once including only the control group. The z statistic and the associated significance level (p -value) of each variable are listed in table 4-29 for the intervention group and table 4-30 for the control group. Table 4-28. Unadjusted Means for Change in PPI Rate by Group (DDDs/patient). Period OA group 0 1 Mean Std Dev Mean Std Dev 1 -0.0203 1.4843 0.0041 1.4211 2 0.0000 0.0000 0.0000 0.0000 3 0.3932 1.3733 0.3372 1.5783 4 0.6307 1.6214 0.5810 1.6686 5 0.5844 1.6238 0.6182 1.7077 6 0.5575 1.7553 0.5490 1.6955 Secondary Outcome: Change in PPI Rates (unadjusted)-0.2 0.0 0.2 0.4 0.6 0.8 123456 Time PeriodDDD change / patient/GP Intervention = no Intervention = yes Figure 4-14. Unadjusted Means for Change in PPI Rates by Group. The within group effect of the intervention is interpreted from the values of the z statistic and associated p -value of the prepost variable. For the intervention and control groups, the z statistics (p -values) of -2.59 (0.0097) and -4.22 (<0.0001) respectively indicates that the within group effect is st atistically significant fo r both groups at the alpha = 0.05 level and both changes are in the direction of in creased utilization.

PAGE 78

63 Table 4-29. Secondary PPI Outcome Model Results (AD = yes). Secondary Outcome Model Results for All Periods (Intervention Group) (Change in PPI Prescribing Rates) Effect Z p-value PS 1.41 0.1596 period 0.69 0.4873 prepost -2.59 0.0097 flu AD (flu AD = 0) 1.37 0.1717 BL rate -3.63 0.0003 # elderly -0.40 0.6879 Table 4-30. Secondary PPI Outcome Model Results (AD = no). Secondary Outcome Model Results for All Periods (Control Group) (Change in PPI Prescribing Rates) Effect Z p-value PS 0.67 0.5012 period -0.02 0.9877 prepost -4.22 <0.0001 flu AD (flu AD = 0) 1.16 0.2450 BL rate -3.32 0.0009 # elderly -1.61 0.1065 H2A Utilization Rates Model development The analysis of the secondary outcome, th e effect of the OA AD intervention on the H2A utilization rates was carried out using the same methods as the primary outcome analysis with the data for H2A utilization substituted for the COX-2 utilization data (figure 4-15). Between group results The z statistic and the significance level (p -value) of each variable from the secondary H2A outcome model (figur e 4-15) are listed in table 4-31. The between group effect of the inte rvention is interpreted from the z statistics and associated p -value of the AD variable. The z statistic and p -value of 0.05 and 0.9619 respectively indicate that the effect is not signifi cant at the alpha = 0.05 level.

PAGE 79

64 Figure 4-15. Secondary Outcome Model for H2A Utilization Table 4-31. Secondary H2A Outcom e Model Results (Periods = 3,4,5,6). Secondary Outcome Model Results for Post-intervention Periods 3 to 6 (Change in H2A Prescribing Rates) Effect Z p-value AD (ad = 0) 0.05 0.9619 PS 1.18 0.2381 period -7.29 <0.0001 flu AD (flu AD = 0) 1.12 0.2642 BL rate -7.31 <0.0001 # elderly 1.77 0.0766 The model (figure 4-15) was used to de termine intervention effects on each post intervention time period. None of the post intervention (ana lyzed individually) showed significant between group differenc es at the alpha = 0.05 level. The model in figure 4-15 was used to determine between group differences in the pre-intervention time periods (period = 1, 2). The z statistic and associated significance level of each variable is listed in table 432. The results are interpreted in the same manner as the post-intervention results. The z statistic and p -value for the AD variable are 1.09 and 0.2764 respectively. The p -value indicates that the groups are not significantly different in th e pre-intervention periods at the alpha = 0.05 level. Y = 0 + 1(X1) + 2(X2) + 3(X3) + 4(X4) + 5(X5) + 6(X6) Where; Y = change in H2A utilization rate (p eriods 3 to 6 (post-intervention)), X1 = physician participation in the intervention (0 = no, 1 = yes), X2 = PS (range from 0 to 1), X3 = experimental time period (period = 3,4,5,6), X4 = physician participation in the influenza AD service (0 = no, 1 = yes), X5 = physician baseline H2A rate (DDD / patient, (period = 2)), X6 = number of patients in th e GPs practice >65 years old

PAGE 80

65 Table 4-33 depicts the least square mean s for the two groups (AD = yes and AD = no) for each of the six experimental time periods The least square mean values are also presented in a graph in figure 4-16. Table 4-32. Secondary H2A Outcom e Model Results (Periods = 1,2). Secondary Outcome Model Results for Post-intervention Periods 1 and 2 (Change in H2A Prescribing Rates) Effect Z p-value AD (ad = 0) 1.09 0.2764 PS 0.98 0.3293 Period -2.74 0.0062 flu AD (flu AD = 0) 0.64 0.5230 BL rate -5.75 <0.0001 # elderly 1.49 0.1368 Table 4-33. Least Square Means for Cha nge in H2A Rate by Group (DDDs/patient). Secondary Outcome (H2A) Least Square Means by AD Group AD group period 1 2 3 4 5 6 0 0.3007 0 0.1833 0.0114 -0.1433 -0.5532 1 0.0881 0 0.0157 0.1413 0.0867 -0.6818 Secondary Outcome: Change in H2A Rates (adjusted)-1 -0.5 0 0.5 123456 Time PeriodDDD change /patient/GP. Intervention = no Intervention = yes Figure 4-16. Least Square Means for Change in H2A Rates by Group. Table 4-34 depicts the unadjusted means and standard deviations for the two groups (AD = yes and AD = no) for each of the six experimental time periods. The unadjusted mean values are also pr esented in a graph in figure 4-17.

PAGE 81

66 Table 4-34. Unadjusted Means for Change in H2A Rate by Group (DDDs/patient). Period AD group 0 1 Mean Std Dev Mean Std Dev 1 0.2440 1.7956 0.2117 1.9819 2 0.0000 0.0000 0.0000 0.0000 3 0.1655 1.9023 0.0783 2.0181 4 0.0607 1.8949 0.2191 2.2941 5 -0.2484 2.4465 0.2929 2.4570 6 -0.5101 2.5878 -0.5380 2.5854 Secondary Outcome: Change in H2A Rates (unadjusted)-0.6 -0.4 -0.2 0.0 0.2 0.4 123456 Time PeriodDDD change /patient/GP Intervention = no Intervention = yes Figure 4-17. Unadjusted Means for Change in H2A Rates by Group. Within group (longitudinal) results The within group model was the same as the between group model in figure 4-15 except that the AD group variable is replaced by a prepos t variable which measures within group differences between change in H2A rates pre-intervention and postintervention. The model is run two times; once including only the intervention group and once including only the control group. The z statistic and the associated significance level (p -value) of each variable are listed in table 4-35 for the intervention group and table 4-36 for the control group. The within group effect of the intervention is interpreted from the values of the z statistic and associated p -value of the prepost variable. For the intervention and control

PAGE 82

67 groups, the z statistics (p -values) of -5.56 (<0.0001) a nd -4.06 (<0.0001) respectively indicates that the within group effect is st atistically significant fo r both groups at the alpha = 0.05 level and both changes are in the direction of de creased utilization. Table 4-35. Secondary H2A Outc ome Model Results (AD = yes). Secondary Outcome Model Results for All Periods (Intervention Group) (Change in H2A Prescribing Rates) Effect Z p-value PS 1.70 0.0897 period -6.59 <0.0001 prepost -5.56 <0.0001 flu AD (flu AD = 0) 1.53 0.1262 BL rate -7.48 <0.0001 # elderly 2.80 0.0051 Table 4-36. Secondary H2A Outc ome Model Results (AD = no). Secondary Outcome Model Results for All Periods (Control Group) (Change in H2A Prescribing Rates) Effect Z p-value PS -0.79 0.4282 period -4.06 <0.0001 prepost -2.33 0.0201 flu AD (flu AD = 0) -0.78 0.4366 BL rate 3.43 0.0006 # elderly -1.64 0.1003 GP Office Visit Rates Model development The analysis of the secondary outcome, the effect of the OA AD intervention on GP office visit rates was carried out using the same methods as the primary outcome analysis with the data for GP office visit ra tes substituted for the COX-2 utilization data (figure 4-18). Between group results The z statistic and the significance level (p -value) of each variable from the secondary GP office visit outcome model (figure 4-18) are listed in table 4-37.

PAGE 83

68 Figure 4-18. Secondary Outcom e Model for GP Office Visits Table 4-37. Secondary GP Office Vi sit Model Results (Periods = 3,4,5,6). Secondary Outcome Model Results for Post-intervention Periods 3 to 6 (Change in GP Office Visit Rates) Effect Z p-value AD (ad = 0) 1.06 0.2888 PS 0.74 0.4587 period -9.26 <0.0001 Flu AD (flu AD = 0) 0.02 0.9815 BL rate -1.97 0.0487 # elderly 1.26 0.2077 The between group effect of the inte rvention is interpreted from the z statistics and associated p -value of the OA AD variable. The z statistic and p -value of 1.06 and 0.2888 respectively indicate that the effect is not significa nt at the alpha = 0.05 level. The model (figure 4-18) was used to de termine intervention effects on each post intervention time period. Only the period from 91 to 180 days (period four) following the intervention showed significan t difference between groups at the alpha = 0.05 level. The z -statistic and p -value associated with the in tervention effect are -2.20 and 0.0275 respectively (95% CI -0.7926, -0.0464). In th is case, where the analysis only includes one time period, the interpretati on of the coefficient estimate is similar to traditional GLM methods. That is, the coefficient esti mate of -0.4195 (AD = no) is interpreted as the non-intervention group having measures of average change rate 0.4195 fewer Y = 0 + 1(X1) + 2(X2) + 3(X3) + 4(X4) + 5(X5) + 6(X6) Where; Y = change in GP visit rates (per iods 3 to 6 (post-intervention)), X1 = physician participation in the intervention (0 = no, 1 = yes), X2 = PS (range from 0 to 1), X3 = experimental time period (period = 3,4,5,6), X4 = physician participation in the influenza AD service (0 = no, 1 = yes), X5 = physician baseline GP visit rate ra te (visits / patient, (period = 2)), X6 = number of patients in th e GPs practice >65 years old

PAGE 84

69 visits/patient/GP than the intervention group (equal values for the groups is hypothesized). The model in figure 4-18 was used to determine between group differences in the pre-intervention time periods (period = 1, 2). The z statistic and associated significance level of each variable is listed in table 438. The results are interpreted in the same manner as the post-intervention results. The z statistic and p -value for the OA AD variable are 0.37 and 0.7097 respectively. The p -value indicates that the groups are not significantly different in th e pre-intervention periods at the alpha = 0.05 level. Table 4-38. Secondary GP Office Visit Outcome Model Results (Periods = 1,2). Secondary Outcome Model Results for Post-intervention Periods 1 and 2 (Change in GP Office Visit Rates) Effect Z p-value AD (ad = 0) 0.37 0.7097 PS 1.16 0.2457 Period -0.08 0.9390 Flu AD (flu AD = 0) 1.19 0.2341 BL rate -7.17 <0.0001 # elderly 2.13 0.0332 Table 4-39 depicts the least square mean s for the two groups (AD = yes and AD = no) for each of the six experimental time periods The least square mean values are also presented in a graph in figure 4-19. Table 4-39. Least Square Means for Ch ange in GP Office Visit Rate by Group (visits/patient). Secondary Outcome (GP Visits) Least Square Means by AD Group AD group Period 1 2 3 4 5 6 0 -0.0182 0 0.4652 0.3882 -0.3346 -0.4341 1 -0.0563 0 0.3813 0.79 -0.0201 0.0069

PAGE 85

70 Table 4-40 depicts the unadjusted means and standard deviations for the two groups (AD = yes and AD = no) for each of the six experimental time periods. The unadjusted mean values are also pr esented in a graph in figure 4-20. Secondary Outcome: Change in GP Office Visit Rates-0.5 0 0.5 1 123456 Time Periodchange/patient/GP Intervention = no Intervention = yes Figure 4-19. Least Square Means for Ch ange in GP Office Visit Rates by Group. Table 4-40. Unadjusted Means and Standard Deviations for Change in GP Office Visit Rate by Group (visits/patient). Period AD group 0 1 Mean Std Dev Mean Std Dev 1 0.0057 0.5235 -0.0026 0.5586 2 0.0000 0.0000 0.0000 0.0000 3 0.5282 0.9720 0.3191 0.9419 4 0.2597 0.8025 0.6388 1.3687 5 -0.2269 0.7578 -0.0570 1.4364 6 -0.2742 0.7616 -0.0290 1.7479 Within group (longitudinal) results The within group model was the same as the between group model in figure 4-18 except that the AD group variable is replaced by a prepos t variable which measures within group differences between change in GP office visit rates pre-intervention and post-intervention. The model is run two times; once including only the intervention group and once including only the control group. The z statistic and the associated

PAGE 86

71 significance level (p -value) of each variable are listed in table 4-41 for the intervention group and table 4-42 for the control group. Secondary Outcome: Change in GP Office Visit Rates (unadjusted) -0.4 -0.2 0.0 0.2 0.4 0.6 0.8 123456 Time Periodchange/ patient/GP Intervention = no Intervention = yes Figure 4-20. Unadjusted Means for Ch ange in GP Office Visit Rates by Group. Table 4-41. Secondary GP Office Vis it Outcome Model Results (AD = yes). Secondary Outcome Model Results for All Periods (Intervention Group) (Change in GP Office Visit Rates) Effect Z p-value PS -1.56 0.1199 Period -10.95 <0.0001 Prepost -17.54 <0.0001 flu AD (flu AD = 0) -2.41 0.0159 BL rate 0.10 0.9187 # elderly 0.17 0.8680 The within group effect of the intervention is interpreted from the values of the z statistic and associated p -value of the prepost variable. For the intervention and control groups, z statistics (p -values) of -17.54 (<0.0001) and -20.21 (<0.0001) respectively indicates that the within group effect is st atistically significant fo r both groups at the alpha = 0.05 level. The significant results fo r the longitudinal prepos t effect is similar between the control and interv ention groups as indicated in figure 4-20 and also indicated in the negative values of the z statistics for both groups.

PAGE 87

72 Table 4-42. Secondary GP Office Vis it Outcome Model Results (AD = no). Secondary Outcome Model Results for All Periods (Control Group) (Change in GP Office Visit Rates) Effect Z p-value PS -2.60 0.0093 Period -19.91 <0.0001 Prepost -20.21 <0.0001 flu AD (flu AD = 0) -2.62 0.0089 BL rate -0.99 0.3212 # elderly -2.73 0.0064 Rheumatologist and GI Specialist Visit Rates Model development The analysis of the secondary outcome, the effect of the OA AD intervention on rheumatologist and GI specialist office visit rates was carried out us ing the same methods as the primary outcome analysis with the data for rheumatologist and GI specialist office visit rates substituted for the COX-2 utilization data (figure 4-21). Between group results The z statistic and the significance level (p -value) of each variable from the secondary specialist office vis it outcome model (figure 4-21) are listed in table 4-43. Figure 4-21. Secondary Outcome Model for Specialist Office Visits Y = 0 + 1(X1) + 2(X2) + 3(X3) + 4(X4) + 5(X5) + 6(X6) Where; Y = change in specialist visit rates (periods 3 to 6 (post-intervention)), X1 = physician participation in the intervention (0 = no, 1 = yes), X2 = PS (range from 0 to 1), X3 = experimental time period (period = 3,4,5,6), X4 = physician participation in the influenza AD service (0 = no, 1 = yes), X5 = physician baseline specia list visit rate (visits / patient, (period = 2)), X6 = number of patients in th e GPs practice >65 years old

PAGE 88

73 The between group effect of the inte rvention is interpreted from the z statistics and associated p -value of the AD variable. The z statistic and p -value of 1.44 and 0.1498 respectively indicate that the effect is not signifi cant at the alpha = 0.05 level. Table 4-43. Secondary Specialist Office Visit Model Results (Periods = 3,4,5,6). Secondary Outcome Model Results for Post-intervention Periods 3 to 6 (Change in Specialist Office Visit Rates) Effect Z p-value AD (ad = 0) 1.44 0.1498 PS -5.98 <0.0001 period -0.04 0.9700 flu AD (flu AD = 0) -5.43 <0.0001 BL rate -23.22 <0.0001 # elderly -3.01 0.0026 The model (figure 4-21) was used to de termine intervention effects on each post intervention time period. Only the period from 181 to 270 days (period five) following the intervention showed significant difference between groups at the alpha = 0.05 level. The z -statistic and p -value associated with the in tervention effect are 2.10 and 0.0356 respectively (95% CI (0.0001, 0.0022)). In th is case, where the analysis only includes one time period, the interpretati on of the coefficient estimate is similar to traditional GLM methods. That is, the coefficient estima te of 0.0012 (AD = no) is interpreted as the non-intervention group having measures of average change rate 0.0012 greater visits/patient/GP than the intervention group. The model in figure 4-21 was used to determine between group differences in the pre-intervention time periods (period = 1, 2). The z statistic and associated significance level of each variable is listed in table 444. The results are interpreted in the same manner as the post-intervention results. The z statistic and p -value for the AD variable are -1.29 and 0.1976 respectively. The p -value indicates that the groups are not significantly different in th e pre-intervention periods at the alpha = 0.05 level.

PAGE 89

74 Table 4-44. Secondary Specialist Office Vis it Outcome Model Results (Periods = 1,2). Secondary Outcome Model Results for Post-intervention Periods 1 and 2 (Change in Specialist Office Visit Rates) Effect Z p-value AD (ad = 0) -1.29 0.1976 PS -4.71 <0.0001 period -0.90 0.3670 flu AD (flu AD = 0) -3.86 0.0001 BL rate -9.21 <0.0001 # elderly -3.51 0.0004 Table 4-45 depicts the least square mean s for the two groups (AD = yes and AD = no) for each of the six experimental time periods The least square mean values are also presented in a graph in figure 4-22. Table 4-46 depicts the unadjusted means and standard deviations for the two groups (AD = yes and AD = no) for each of the six experimental time periods. The unadjusted mean values are also pr esented in a graph in figure 4-23. Secondary Outcome (Specialist Visits) Least Square Means by AD Group AD group period 1 2 3 4 5 6 0 0.0005 0 0.0007 0.0005 0.0011 0.0001 1 0.0016 0 -0.0002 0.0003 -0.0001 0.0003 Table 4-45. Least Square Means for Change in Specialist Office Visit Rate by Group (visits/patient). Within group (longitudinal) results The within group model was the same as the between group model in figure 4-21 except that the AD group variable is replaced by a prepos t variable which measures within group differences between change in sp ecialist office visit rates pre-intervention and post-intervention. The model is run two times; once including on ly the intervention group and once including only the control group. The z statistic and the associated significance level (p -value) of each variable are listed in table 4-47 for the intervention group and table 4-48 for the control group.

PAGE 90

75 Secondary Outcome: Change in Specialist Office Visit Rates (adjusted)-0.0005 0 0.0005 0.001 0.0015 0.002 123456 Time Periodchange /patient/GP Intervention = no Intervention = yes Figure 4-22. Least Square Means for Change in Specialist Office Visit Rates by Group. Period AD group 0 1 Mean Std Dev Mean Std Dev 1 0.0008 0.0089 -0.0002 0.0087 2 0.0000 0.0000 0.0000 0.0000 3 0.0005 0.0077 -0.0011 0.0089 4 0.0008 0.0086 -0.0012 0.0078 5 0.0007 0.0093 -0.0013 0.0079 6 0.0002 0.0083 -0.0006 0.0082 Table 4-46. Unadjusted Means and Standard Deviations for Change in Specialist Office Visit Rate by Group (visits/patient). Secondary Outcome: Change in Specialist Office Visit Rates (unadjusted) -0.0015 -0.0010 -0.0005 0.0000 0.0005 0.0010 123456 Time Periodchange/ patient/GP Intervention = no Intervention = yes Figure 4-23. Unadjusted Means for Change in Specialist Office Visit Rates by Group.

PAGE 91

76 Table 4-47. Secondary Specialist Office Vi sit Outcome Model Results (AD = yes). Secondary Outcome Model Results for All Periods (Intervention Group) (Change in Specialist Office Visit Rates) Effect Z p-value PS -6.45 <0.0001 period 0.87 0.3857 prepost 1.94 0.0519 flu AD (flu AD = 0) -6.29 <0.0001 BL rate -17.54 <0.0001 # elderly -4.56 <0.0001 Table 4-48. Secondary Specialist Office Visit Outcome Model Results (AD = no). Secondary Outcome Model Results for All Periods (Control Group) (Change in Specialist Office Visit Rates) Effect Z p-value PS -4.24 <0.0001 period -0.87 0.3870 prepost -0.70 0.4811 flu AD (flu AD = 0) -3.25 0.0012 BL rate -14.67 <0.0001 # elderly -2.01 0.0444 The within group effect of the intervention is interpreted from the values of the z statistic and associated p -value of the prepost variable. For the intervention and control groups, z statistics (p -values) of 1.94 (0.0519) and .70 (0.4811) respectively indicates that the within group effect is not statistica lly significant for both groups at the alpha = 0.05 level. The results for the longitudinal pr epost effect are similar between the control and intervention groups as indicated in figure 4-23. Hospitalization Rates Due to GI Complications Model development The analysis of the secondary outcome, the effect of the OA AD intervention on hospitalization rates was carried out using the same methods as the primary outcome analysis with the data for hospital length of stay rates substituted for the COX-2 utilization data (figure 4-24).

PAGE 92

77 Figure 4-24. Secondary Outcome M odel for Hospital Length of Stay Between group results The z statistic and the significance level (p -value) of each variable from the secondary hospitalization length of stay outcome model (figure 4-24) are listed in table 449. Table 4-49. Secondary Hospital Length of Stay Model Results (Periods = 3,4,5,6). Secondary Outcome Model Results for Post-intervention Periods 3 to 6 (Change in Hospital Length of Stay) Effect Z p-value AD (AD = 0) 0.33 0.7389 PS 1.48 0.1396 Period 1.15 0.2500 flu AD (flu AD = 0) 1.13 0.2568 Flu AD*quintile (flu AD = 0) -0.94 0.3468 BL rate -15.58 <0.0001 los rate -2.36 0.0183 The between group effect of the inte rvention is interpreted from the z statistics and associated p -value of the AD variable. The z statistic and p -value of 0.33 and 0.7389 respectively indicate that the effect is not signifi cant at the alpha = 0.05 level. The model (figure 4-24) was used to de termine intervention effects on each post intervention time period. Only the period from 181 to 270 days (period five) following the intervention showed significant difference between groups at the alpha = 0.05 level. Y = 0 + 1(X1) + 2(X2) + 3(X3) + 4(X4) + 5(X5) + 6(X6) Where; Y = change in hospital utilization rate (periods 3 to 6 (post-intervention)), X1 = physician participation in the intervention (0 = no, 1 = yes), X2 = PS (range from 0 to 1), X3 = experimental time period (period = 3,4,5,6), X4 = physician participation in the influenza AD service (0 = no, 1 = yes), X5 = physician baseline hospital LOS rate (LOS / patient, (period = 2)), X6 = number of patients in th e GPs practice >65 years old

PAGE 93

78 The z -statistic and p -value associated with the in tervention effect are 2.49 and 0.0128 respectively (95% CI (1.1093, 3.4627)). In th is case, where the analysis only includes one time period, the interpretati on of the coefficient estimate is similar to traditional GLM methods. That is, the coefficient estima te of 2.2860 (AD = no) is interpreted as the non-intervention group having measures of average change rate 2.2860 greater visits/patient/GP than the intervention group. The model in figure 4-18 was used to determine between group differences in the pre-intervention time periods (period = 1, 2). The z statistic and associated significance level of each variable is listed in table 450. The results are interpreted in the same manner as the post-intervention results. The z statistic and p -value for the AD variable are 1.58 and 0.1152 respectively. The p -value indicates that the groups are not significantly different in th e pre-intervention periods at the alpha = 0.05 level. Table 4-50. Secondary Hospital Length of St ay Outcome Model Results (Periods = 1,2). Secondary Outcome Model Results for Post-intervention Periods 1 and 2 (Change in Hospital Length of Stay) Effect z p-value AD (AD = 0) 1.58 0.1152 quintile 0.56 0.5751 period -0.67 0.5014 flu AD (flu AD = 0) -0.10 0.9217 flu AD*quintile (flu AD = 0) 0.72 0.4735 BL rate -10.32 <0.0001 los rate -2.84 0.0044 Table 4-51 depicts the least square mean s for the two groups (AD = yes and AD = no) for each of the six experimental time periods The least square mean values are also presented in a graph in figure 4-25.

PAGE 94

79 Table 4-52 depicts the unadjusted means and standard deviations for the two groups (AD = yes and AD = no) for each of the six experimental time periods. The unadjusted mean values are also pr esented in a graph in figure 4-26. Table 4-51. Least Square Means for Change in Hospital Length of Stay Rates by Group (days/patient). Secondary Outcome (Hospital LOS) Least Square Means by AD Group AD group period 1 2 3 4 5 6 0 0.6168 0 1.0337 0.8165 2.0962 1.7457 1 -0.3396 0 0.2072 1.0211 -0.1898 3.6511 Secondary Outcome: Change in Hospital LOS Rates (adjusted)-1 0 1 2 3 4 123456 Time Periodchange/ patient/GP Intervention = no Intervention = yes Figure 4-25. Least Square Means for Change in Hospital Length of Stay Rates by Group. Table 4-52. Unadjusted Means and Standard Deviations for Change in Hospital Length of Stay Rates by Group (days/patient). Period AD group 0 1 Mean Std Dev Mean Std Dev 1 0.7409 8.6598 -0.2482 9.9641 2 0.0000 0.0000 0.0000 0.0000 3 1.3957 9.9291 0.6193 9.7833 4 1.0678 8.4797 1.0061 14.5320 5 1.5773 10.2326 0.5130 8.3747 6 0.5662 8.8979 5.5624 51.6778

PAGE 95

80 Within group (longitudinal) results The within group model was the same as the between group model in figure 4-24 except that the AD group variable is replaced by a prepos t variable which measures within group differences between change in sp ecialist office visit rates pre-intervention and post-intervention. The model is run two times; once including on ly the intervention group and once including only the control group. The z statistic and the associated significance level (p -value) of each variable are listed in table 4-53 for the intervention group and table 4-54 for the control group. Secondary Outcome: Change in Hospital LOS Rates (unadjusted)-1.0 0.0 1.0 2.0 3.0 4.0 5.0 6.0 123456 Time Periodchange/ patient/GP Intervention = no Intervention = yes Figure 4-26. Unadjusted Means for Change in Hospital Length of Stay Rates by Group. Table 4-53. Secondary Hospital Length of Stay Outcome Model Results (AD = yes). Secondary Outcome Model Results for All Periods (Intervention Group) (Change in Hospital Length of Stay) Effect Z p-value PS 0.76 0.4489 period 1.44 0.1491 prepost 0.96 0.3388 flu AD (flu AD = 0) 0.00 0.9976 BL rate -13.11 <0.0001 # elderly -1.56 0.1193

PAGE 96

81 Table 4-54. Secondary Hospital Length of Stay Outcome Model Results (AD = no). Secondary Outcome Model Results for All Periods (Control Group) (Change in Hospital Length of Stay) Effect Z p-value PS 3.13 0.0018 period -1.21 0.2256 prepost -2.08 0.0375 flu AD (flu AD = 0) 2.99 0.0028 BL rate -14.40 <0.0001 # elderly 1.87 0.0614 The within group effect of the intervention is interpreted from the values of the z statistic and associated p -value of the prepost variable. For the intervention and control groups, z statistics (p -values) of 0.96 (0.3388) and .08 (0.0375) respectively indicates that the within group effect is not statistica lly significant for the intervention group and is statistically significant for the cont rol group at the alpha = 0.05 level. Death Rates Due to GI Complications Model development The analysis of the secondary outcome, the effect of the OA AD intervention on death rates due to GI complications was car ried out using the same methods as the primary outcome analysis with the data for hos pital length of stay rates substituted for the COX-2 utilization data (figure 4-27). Figure 4-27. Secondary Outcome Model for Deaths Due to GI Complications Ln Y = 0 + 1(X1) + 2(X2) + 3(X3) + 4(X4) + 5(X5) + 6(X6) Where; Y = death rates (periods 3 to 6 (post-intervention)), X1 = physician participation in the intervention (0 = no, 1 = yes), X2 = PS (range from 0 to 1), X3 = experimental time period (period = 3,4,5,6), X4 = physician participation in the influenza AD service (0 = no, 1 = yes), X5 = physician baseline hospital LOS rate (LOS / patient, (period = 2)), X6 = number of patients in th e GPs practice >65 years old

PAGE 97

82 Special consideration had to be given to the distribu tion of the data since the number of deaths per GP per study period wa s quite low. There were 1984 data points analyzed (496 GPs with six measures each) a nd in all cases except four the number of deaths per physician was equal to zero or one. The four other cas es all contained two deaths (three of the four occurred in the control group). A dichotomous variable representing death/no-death for each peri od measurement was developed and since the majority of the period measurements repr esented no-death (142 with death and 2834 without death) a negative binomial distribution wa s used in the analysis model. The total number of deaths per group per period was less th an five in a number of cases. For this reason, the number of study periods was redu ced to three by combin ing periods one and two, three and four, and five and six. Between and within group results None of the between or within group analyses of death ra tes showed significance at the alpha = 0.05 level. The z statistics (p -values) associated with the pre-intervention and post-intervention between group analyses were 0.63 (0.5317) and 0.81 (0.4203) respectively and the z statistics (p -values) associated with th e within group analyses for the intervention and control groups were -0.36 (0.7189) and -0.03 (0.9742) respectively.

PAGE 98

83 CHAPTER 5 DISCUSSION The Academic Detailing Program in Nova Scotia An analysis of the effect of the OA AD intervention on prescribing behavior should be taken in context of the qualifications of the detailers the dynamic changes over the course of the intervention and the policy op tions available to th e decision makers. A description of these three topics should add to the determination of generalizability of the intervention to other jurisdictions. Qualifications of the Detailers The OA AD intervention employed three detailers; two pharmacists and one registered nurse. One pharmacist worked within the provinces capitol district and the other pharmacist and the registered nurse divi ded the rural area of the province in two. The nurse detailed GPs in the region that she was native to and as such was very familiar with local customs and practices. All three of the detailers were trained in techniques associated with successful AD programs. These techniques are described in greater detail in appendix A. The intervention was designed to take approxima tely twenty minutes to present with opportunity for the GP to interact with the detailer over the course of the presentation. Changes Which Occurred Over the Period of the Interven tion (History Effects) The OA AD intervention was delivered from April, 2002 to November, 2002. The analysis timeframe for our study spanned from October, 2001 (six months before the intervention commenced) to November, 2003 (one year after the intervention concluded).

PAGE 99

84 Between October, 2001 and May, 2003 two wa rnings regarding the safety of COX2 inhibitors were issued by Health Canada.38 The first warning in April, 2002 concerned the results of the VIGOR trial8 and warned of increased cardiac risk associated with rofecoxib use and the second warning in Ma y, 2002 concerned the re sults of the CLASS trial7 and warned of the GI risk associated with celecoxib use particularly in combination with low dose ASA therapy. Analysis of th e VIGOR and CLASS trials was included in the OA AD intervention (appendix A). In December, 2004 rofecoxib was withdrawn from the market.38 The withdrawal occurred after th e post intervention study period. The Nova Scotia pharmacare plan issued a policy change with re spect to the benefit status of a combination product co ntaining diclofenac and misoprostol.39 The product was changed to open benefit status in September, 2002. Announcement of the change was disseminated equally to all GPs in the pr ovince. The benefit status of rabeprazole was changed to open benefit in June, 200339 after the end of the pos t intervention analysis period. Policy Options Available to Decision Makers Our study examined the effect of the fourth message of the OA AD intervention which addressed pharmacotherapy of OA. Th e other three messages contained in the intervention were intended to change physic ian behavior in terms of prescribing nonpharmacologic treatment for OA and research into the effectiveness of these messages is warranted. The OA AD intervention lacked a fo llow-up visit which is a limitation of the intervention design.13 Five options available to po licy decision makers which could address this shortcoming without the costs associated with a one-on-one follow-up visit are; the distribution of educational material educational meetings audit and feedback,

PAGE 100

85 reminders, and changes in benefit schedules.17, 40 While some of the instruments have not shown significant effects on their own the combination with AD can be effective.14, 17 Distribution of educational material The distribution of educational material involves the dissemination of media (written or video) to the GPs with info rmation reinforcing the messages of the OA AD intervention. It is the decisi on of the GP to review the message or not. It is relatively low cost and has been shown to have a modest but short-lived effect.17 The message contained in this medium should be limited to the intervention messages in such a way that does not require active learning or interaction with an educator. Educational meetings Educational meetings involve meeting in groups to review the messages from the intervention. This instrument can be more complex in nature than the distribution of written material but they are still limited by the inability of the participant to interact with the instructor on a one-to-one basis. Used as a single intervention this instrument has shown little17 to no effect14 on improving pharmaceutical use. Audit and feedback Audit and feedback is an instrument that involves the analysis of the performance of the provider and/or the provi ders peers over a peri od of time. The instrument is costly to implement as it involves a significant amount of data anal ysis to produce the audits. Audit and feedback can address some complex issues through the use of the analysis and comparison with peers. Studies using audit and feedback as a single intervention have shown a modest effect.17

PAGE 101

86 Reminders and reminder systems Reminders or reminder systems prompt the provider to recall information. The prompts can take the form of verbal remi nders, written notes or computer generated reminders. Reminders of importa nt clinical information in a timely fashion is a benefit to providers however, constant and non-significant alerts (generally from computer systems) create wasted time and can lead to the ignoring of reminders all together. The cost of the chart review for written reminders or the deve lopment of clinical software can be quite costly. The effects have been shown to be moderate with statistical significance reached approximately one-half of the time.17 Drug benefit changes Changes to drug benefit schedules can re inforce the interventions prescribing message by listing some drugs as open benefits where they were restricted before. In terms of the OA AD intervention the listing of a diclofenac and misoprostol combination product and rabeprazole as open benefits39 coincident with the intervention could encourage prescribing that is in line with the messages contained in th e intervention. Primary Outcome: Effect on COX-2 Utilization Rates Generalized estimating equations (GEE) t echniques for repeated measures were used for the outcomes analyses. The interpretation of the statistical output from the GEE analysis is different from the analysis of output from general linear models (GLM).37 In GLM with a continuous outcome variable, the co efficient estimate can be interpreted as the effect on the outcome variable if the cova riate associated with the coefficient estimate is changed by one unit. In the GEE analysis for repeated measures the main effect result can be interpreted as a between group effect or a within gr oup effect. The magnitude of the contribution of the between and within group effects cannot be determined from the

PAGE 102

87 main effect result alone. To ensure that th e between group analysis is indeed showing a significant difference between groups a before and after longitudinal analysis was carried out. The effect of the AD intervention was determined using four separate analyses; a pre-intervention be tween group analysis, a post-interven tion between group analysis, and two within group analyses on the two GP groups. Statistical Results The pre-intervention analysis showed that the groups were not significantly different in the six months preceding the intervention (z = 0.88, p = 0.3775). The within group analysis for the control group did not show significant change before and after (z = -0.22, p = 0.8273) whereas the before and afte r analysis of the intervention group did show a significant decrease in utilization (z = -2.34, p = .0191). The between group postintervention analysis by period showed a si gnificant difference between groups on the period immediately following the interv ention only (z = 2.06, p = 0.0395). The intervention was effective in this single period but the be tween group analysis for the entire post intervention period was not si gnificant (z = 0.85, p = 0.3976) indicating that the intervention was not sustained beyond th e three month post-intervention period. Practical Significance The difference in change in COX-2 rates between groups in the period immediately following the intervention is 0.8717 DDDs per patient per 90 days which equals 0.00969 DDDs per patient per day. The inverse of the amount yields the number of patients needed to treat (NNT) to give one patient year of therapy change. The NNT is 104 patients. The average number of elderly patients on a GP pane l is 187. The effect can be interpreted as the average GP changing pres cribing away from CO X-2 inhibitors for 1.8 patients for three months post intervention. This result translates in to 416 patients from a

PAGE 103

88 total of 43,197 patients (1% of total) include d in the intervention group who had their therapy changed away from COX-2 inhibitors. Comparison with Literature The average relative effect of our study on the utilization rate of COX-2 inhibitors over the first 90 day post-intervention period is 23%. Thomson OBrien et al. reported that multifaceted AD interventions have shown relative effect sizes ranging fr om 1 to 45% (from 9 studies).18 All nine of the studies contained outcomes related to prescribing be havior. In addition to AD the interventions included; provision of educational material (s ix studies), patient mediated interventions (three studies), social market ing (four studies), audit and feedback (two studies), and reminders (one study). The OA AD interven tion employed the provision of educational materials, patient mediated intervention, and a desk top reminder. Grimshaw et al. reported that AD interv entions involving comparisons of process measures showed relative eff ect sizes ranging from 1.7 to 24% (from six studies) with the median effect equal to 15% and AD interv entions involving comparisons of outcome measures showed effect sizes ranging from -1.4 to 13.9% (from 4 studies).17 Secondary Outcomes Effect on Gastro-Protective Agents Utilization Rates Similar analyses to the primary outcome we re carried out on the utilization rates of misoprostol, PPIs, and H2As. Misoprostol The misoprostol analyses showed no si gnificant difference between groups in either the pre or post intervention periods (z = -0.87, p = 0.3866 and z = -0.22, p = .8269 respectively) and the longitudinal, within group, analyses (control and intervention)

PAGE 104

89 showed no significant difference within the co ntrol or intervention groups over the study period (z = 1.00, p = 0.3176 and z = 0.25, p = 0.8075 respectively). There were no significant between group results by study period. PPIs The PPI analyses showed no significant di fference between groups in either the pre or post intervention periods (z = 0.13, p = 0.8989 and z = -0.27, p = 0.7906 respectively) and both longitudinal, within group, analys es (control and in tervention) showed significant difference (z = -4.22, p = <0.0001 and z = -2.59, p = 0.0097 respectively). The results show a similar in creasing utilization pattern for both intervention groups. H2As H2As are not indicated as ga stro-protective agents and th e decision to include this class of medications was made on the basis of the Nova Scotia Pharmacare formulary policy which requires previous authorization for PPIs whereas H2As are an open benefit. The H2A analyses showed no significant di fference between groups in either the pre or post intervention periods (z = 1.09, p = 0.2764 and z = 0.05, p = 0.9619 respectively) and both longitudinal, within group, analyses (contro l and intervention) showed significant difference (z = -2.33, p = 0.0201 and z = -5.56, p = <0.0001 respectively). The results show a simila r increasing utilization pattern for both intervention groups. Effect on Utilization of Medical Services Analyses using similar models to the pr imary outcome were carried out on the change in GP office visit rates, specialist physic ian visit rates, and hos pital length of stay rates. Analysis of the death rates due to GI complications was car ried out using similar

PAGE 105

90 statistical methods but the dist ribution of the data was more consistent with a negative binomial distribution. The literature suggests the gastro-protective effects of COX-2 inhibitors are removed by daily aspirin therapy7 and the use of PPIs with traditional NSAIDs provide gastro-protection similar to COX-2 inhibitors.41 The analysis did not control for these conditions. GP office visits The GP office visit analyses showed no significant difference between groups in either the pre or post intervention periods (z = 0.37, p = 0.7097 and z = 1.06, p = 0.2888 respectively) and both longitudinal, within group, analyses (contro l and intervention) showed significant difference (z = -20.21, p = <0.0001 and z = -17.54, p = <0.0001 respectively). The results show a simila r increasing utilization pattern for both intervention groups. One possible explanati on for the increase in GP visits for both groups is a seasonal effect. The time periods with the greatest number of visits coincide with the winter months. The time period fr om three to six months post intervention showed significantly fewer GP visits in the control group than the intervention group (0.4195 visits per patient (95% CI (-0.7926, -0.0464)). A possible explanation for the difference is that the intervention GPs m onitored patients more closely after the intervention for GI side effects. Specialist office visits The specialist office visit analyses showed no significant difference between groups in either the pre or post interven tion periods at the alpha = 0.05 level (z = -1.29, p = 0.1976 and z = 1.44, p = 0.1498 respectively). The longitu dinal, within group, analyses on the control and interventi on groups showed no significant difference at the alpha =

PAGE 106

91 0.05 level (z = -0.70, p = 0.4811 and z = 1.94, p = 0.0519 respectively). The time period from six to nine months post intervention s howed statistical signifi cance with the control group having higher visit rates (z = 2.10, p = 0.0356, 95% CI (0.0001, 0.0022) visits per patient). While this result is statistically si gnificant the small magn itude of the difference makes it practically insignificant. Hospitalization rates due to GI side effects The hospitalization rate due to GI side effects analyses showed no significant difference between groups in either the pre or post intervention periods at the alpha = 0.05 level (z = 1.58, p = 0.1152 and z = 0.33, p = 0.7389 respectively). The longitudinal, within group, analyses on the control group showed significant difference (z = -2.08, p = 0.0375) and the analysis on the intervention group was not significant at the alpha = 0.05 level (z = 0.96, p = 0.3388). The time period from six to nine months post intervention showed statistical significance with the c ontrol group having higher hospitalization rates (z = 2.49, p = 0.0128, 95% (CI 1.1093, 3.4627) days per pa tient). The direction of the effect is opposite to the hypothesis that the intervention group would have higher hospitalization rates. The ch ange in hospitalization rates is intended to indicate the severity of illness due to GI complications a nd it is not a measure of numbers of patients who experienced adverse GI events. For exam ple, the result show ing the control group with higher hospital utilizati on rates in the period from six to nine months post intervention indicates that more hospital days per elderly patient were attributed to the control group but it does not i ndicate that more patients in the control group experienced adverse GI events.

PAGE 107

92 Death due to GI complications None of the between or within group analyses of death ra tes showed significance at the alpha = 0.05 level. The z statistics (p -values) associated with the pre-intervention and post-intervention between group analyses were 0.63 (0.5317) and 0.81 (0.4203) respectively and the z statistics (p -values) associated with th e within group analyses for the intervention and control groups were -0.36 (0.7189) and -0.03 (0.9742) respectively. Propensity Score Analysis Methods The pre-PS analysis showed that five of the twelve variables extracted from the administrative data were significantly different at the alpha = 0.05 level. The three PS methods that were carried out as part of this study performed as described in the literature.11, 12, 16, 27, 28 Greedy Matching Method The greedy matching method resulted in a 75% reduction in bias on the four VOC. This adjustment for bias was the lowest re sult of the three methods tested. The method also suffered from a decrease in sample size of 58% which made it unacceptable due to a possible loss in statistical power for the outcomes analysis as well as a loss of generalizability of findings since the tails of the PS distributions would represent the physicians who were eliminated from the study.12, 28 Parsons uses a case control study example where the controls outnumber the cases 7.4 to 1. Parsons example resulted in 85% of the cases being matched with a control.28 In our study the ratio of controls to cases is 1.15 to 1 and the percent of matche d cases was 45%. The lack of a substantial control group in our study led to the situation where the subjects with the highest and lowest PSs were excluded.

PAGE 108

93 Quintile Method The stratification by quintile method resulted in an 82% reduction in bias on the four VOC. Rosenbaum and Rubin 11 stated that stratifica tion on quintiles can be expected to remove approximately 90% of th e bias for each of the included covariates. DAgostino12 included an example where only the cova riates with the greatest initial bias were analyzed. The DAgostino example result ed in an average decrease in bias of 87.4% for the four included vari ables. This result is sim ilar to our reduction of 82%. Regression on the PS Method The regression on the PS method resulted in the greatest reduction in bias on the VOC of 99%. This result combined with the retention of all GP in the model made the regression on the PS the preferred method for our study. PS Exploratory Analysis Rosenbaum suggests that the extent to wh ich an unmeasured variable would be balanced by PS methods would be related to th e correlation of the unmeasured variable to model covariates.11 Austin found that the PS method, based on variables extracted from administrative data, was not effective in balanc ing clinically relevant variables extracted from patient charts but not included in the PS analysis.29 Our study sought to determine the extent to which a PS model based on ad ministrative data was able to balance administrative variables which were not incl uded in the PS regression model. Our study found that the ability of the PS method to reduc e bias on variables not included in the PS regression model was associated with the correlation of the vari able which was not included with the PS and included covariates. The reduction of bias on the variable not included in the PS model increased as the co rrelation between the unmeasured variable and the PS increased. The magnitude of the bias reduction ranged from 39 to 60%

PAGE 109

94 (depending on PS method) when the correlati on between PS and the unmeasured variable was 0.22 and the largest correlation with an included covariate was 0.12. The bias reduction increased to 59 to 97% (depe nding on PS method) when the correlation between PS and the unmeasured variable was 0.33 and the largest correlation with a covariate was 0.91. Our finding supports Rosenbau ms assertion that adjustment for bias on unmeasured variables will increase with the unmeasured variables correlation with a PS regression model covariate. We also found in one case that the adjustment in bias can be as high as 60% when the correlation w ith the PS was 0.22 and the highest correlation with a covariate was 0.12. Limitations The limitations of our study fall into two ge neral categories; those associated with data and those associated with study design. Data Limitations There are a number of database limita tions that must be discussed. These limitations can assume the general categories of information that is provided but not 100% reliable and information that is desire d but is not captured in the administrative data that is available. Throughout this research administrative data was relied upon however it is not always accurate and, in fact, its inaccuracy wa s exploited in one case. ICD-9 codes from hospital discharge data were utilized throughout the secondary outcomes analysis and the reliability of secondary diagnoses, in particular, has been questioned.42 The events that were evaluated were extracted from the prim ary, secondary and tertiary diagnoses fields (sixteen diagnosis fields available) and as such are considered to be more reliable.

PAGE 110

95 The PS analysis uses an average hospital leng th of stay variable as a measure of the wellness of the physicians elderly patient pane l. This variable is highly dependent on the individual institutions ability to record admissions and discharges to the institution. The OA diagnosis variable that was used in the PS analysis takes advantage of the unreliability of ICD-9 coding. Rather than tr ying to accurately predict the number of patients with OA that a particular physician s ees it is used as a measure of the physicians attention to the disease itself. The aggregated prescription claims data cont ains inaccuracies due to the fact that it measures the date that a prescription was fille d at the pharmacy rather than the date that the prescribing took place and some patients do no t have the prescription filled at all. In some cases a considerable time lag may exist between the date the prescription was written and the date it was filled since many patients do not have their prescriptions filled on the day that they were written. The inaccura cies would occur in the instances were the lag time for having a prescription filled caused the data for the claim to be accrued to a study period in which the act of prescribing did not occur and in the instances where prescriptions were not filled. The prescription claims data supplied by th e Nova Scotia Department of Health was subject to a change in encryption met hodology carried out by CIHI. These changes lead to the elimination of a data field which indicated whether a prescription was an original fill or a refill. The result was a change in data aggregation for the secondary pharmacotherapeutic outcomes analyses. The refill prescription data was aggregated by the period in which the prescription was dispen sed rather than by linkage to the original prescription.

PAGE 111

96 Acetaminophen is not covered under the Nova Scotia pharmacare plan. It has been identified in the OA AD intervention as firs t line pharmacotherapy for mild to moderate OA6 but the claims data does not exist so the ex tent to which this agent is used is not available from the data. Design Limitations The omission of a follow-up visit is a weakness in the OA AD intervention. A follow-up visit provides the physician with a boo st to their intention to change behavior by presenting the physician with measured actions that reflect the beha vioral change that has already taken place. Grimshaw17 and Thomson OBrien18 identified other non face-to-face follow-up strategies which, if included in a multifaceted intervention, can improve results. Two of the strategies that could be considered as a replacement for the face-to-face follow-up visit are the use of reminder systems (such as chart reminders or computer reminders) to reinforce the original AD messages and physician prescribing profiles (audit and feedback) to give the physician f eedback on prescribing performance. The OA AD intervention does include a reflec tive exercise that is completed after three months. The reflective exercise require s the physician to re-e valuate the material that was presented in the intervention a nd it is intended to allow the physician to recommit to his or her intenti on to change behavior. The exercise also requires the physician to explicitly state actions that wi ll be required to br ing about the desired behaviors. The reflective exercise, howev er, is voluntary and does not involve the academic detailers. There is no assurance th at it will be completed and therefore this important component of the inte rvention may not be realized.

PAGE 112

97 The secondary analyses dea ling with utilizati on of medical serv ices due to GI complications (hospital length of stay, GP and specialist vi sits) do not include emergency room visits that did not result in a hospital admission. There are a number of variables that have been cited in other st udies as significant indicators for OA pharmacotherapy. Two variable s that have been found to be significant are severity of illness of OA and patient pain scale measurements.10 These variables are not available through the admini strative databases for this p opulation and therefore, their omission is a limitation of the study. At the time of the study, COX-2s were wide ly used for the treatment of OA. There were other approved uses however (pol yps, dysmenhorrea) that could confound the results. This is a limitation of the study however, the effect is expected to be minimal since the other approved uses represent a small percent of the total use and can be expected to be evenly distributed between groups. Off label uses of COX-2s (e.g. rheumatoid arthritis, pancreatic cancer) co uld also confound results but the effects of these uses are also expected to be minimal. PS methods themselves can be considered to be a limitation of the study. The theory behind PSs makes the assumption that if the groups are balanced on variables that are measured and relevant then the groups w ill also be balanced on those variables that are paramount to the study but are not measured.16 If this theoretical assumption does not hold then the integrity of the quasi-experime ntal design is in question. This is a limitation of the study and is the rational behi nd the analysis of the three PS methods to determine if any one method outperforms the others in terms of balancing unmeasured variables.

PAGE 113

98 The PS analyses were based on a review of medical literature and not statistical literature. It is acknowledged that more thoroughly developed PS methodology contained in the statistical l iterature is not included in our study. There are threats to the validity of the study from history, maturation and contamination. The history effects that threat en the validity of th e study include outside influences on the outcomes such as an incr ease in GI events due to another cause, new approved uses (labeling changes) from H ealth Canada during the study period, and new warnings from Health Canada regarding th e use of COX-2 inhibitors. The maturation effect exists since the patients in the study are aging over the period of the research. As the patients age their risk for GI events increases and the likelihood of receiving COX-2 therapy also increases. These effects of history and maturation should have the same effect on both of the intervention and control groups and pose a limited threat. The threat from contamination exists since the program is voluntary and ther e is not a control mechanism in place to prevent phys icians for sharing information. Conclusions Our study has shown a statistically signi ficant association between an OA AD intervention and the decrease in COX-2 utiliz ation rates in physic ians who volunteered for the intervention for the time period immedi ately following the intervention (z = 2.06, p = 0.0395, 95% CI (0.0365, 1.4815) DDDs per patie nt). The positive effect of the intervention remained throughout the post in tervention analysis period (figure 4-8) however the between group differences were not statistically significan t over the one year post intervention period (z = 0.85, p = 0.3976). Th e intervention effect was sustained for three months post intervention.

PAGE 114

99 The relative effect of th e between group difference was 23% and the number of patients needed to treat to show a decreas e in COX-2 therapy of one DDD is 104. Since this is an observati onal study the assertion of a causal relationship is beyond the scope of the research. In an attempt to strengthen the assertion of a causal relationship study design include d an intervention and cont rol group and several pre and post intervention time periods were analyzed. The concerns regardi ng selection bias due to the voluntary nature of th e intervention were addressed through the use of regression on PS methods. The GP office visit between group differen ce in the time period from three to six months post intervention had practical signi ficance since it showed higher utilization rates in the intervention group (z = -2.20, p = 0.0275, 95% CI (-0.7926, -0.0464) visits per patient) as compared to the control gr oup. This difference could possibly represent an increased vigilance by the GP towards thei r patients with respect to GI side effects associated with traditional NSAIDs. Our study quantified the relationship be tween the reduction in bias between experimental groups on variables which were not included in the PS regression model and the correlation between the PS and the variab le which was not included in the model. The bias reduction on variables not included in the regression analysis, in the selected PS method, was found to range from 60% at a PS correlation of 0.22 and a maximum correlation with an included covariate of 0.12 to 95% at a PS correlation of 0.33 and a maximum correlation with an included covari ate of 0.91. This findi ng is important since it shows that a modest correlation between the variable which was not included in the PS

PAGE 115

100 model and the PS can yield significant reductio ns in bias between intervention groups on that variable. Our study was designed and implemented usi ng administrative data to analyze an AD intervention which is part of a continuous program of AD to improve prescribing practices. The methods can be replicated to study the effects of ot her AD interventions in the same population. Using similar methods fo r analysis, future re search could identify AD topics which have greater or lesser effects on prescribing behavior. The results from the PS exploratory analysis require further research to generalize our findings in terms of the PSs ability to adju st for bias in unmeasured variables and the magnitude of the adjustment depending on correlation with c ovariates in the PS regression model.

PAGE 116

101 APPENDIX A AN APPRAISAL OF THE NOVA SCOTIA OA AD INTERVENTION In 1990, Soumerai and Avorn summarized eight components that contribute to a successful AD intervention. These components we re derived from the literature and from techniques that have been employed and proven by industry for over 100 years.13 Many of the AD intervention strategies that we re identified by Soumerai and Avorn were validated in a review conduc ted by Davis et al in 1994.14 The Davis et al literature review analyzed 160 interventions from 99 trials and concluded that the outreach visit, such as AD, is an effective change strategy.14 Davis et al also conc luded that while AD is effective it is seldom used by con tinuing medical education providers. The AD intervention on OA that has be en developed by the Nova Scotia Department of Health and is managed by th e Division of Continuing Medical Education, Dalhousie University, include s all eight components proposed by Soumerai and Avorn.13 The eight components are as follows; Conduct interviews with physicians to establish baseline knowledge, Focus the intervention on specific physicians, Define clear objectives for the intervention, Establish the credibility of the agency developing the intervention, Stimulate physician interaction during the detailing visit, Use concise graphic educational materials during the presentation, Highlight and reinforce the essential messages during the presentation and, Provide positive reinforcement with a follow-up visit to the physician. The thoroughness of the Nova Scotia OA in tervention should be predictive of its success in terms of the work done by Davis et al. The following is a comparative

PAGE 117

102 analysis of the OA AD intervention and the ei ght components of a successful intervention proposed by Soumerai and Avorn. Conduct Interviews with Physicians Before the intervention is developed it is essential to establis h baseline knowledge of the targeted physician as well as knowledge of their prescribing behavior and reasons for that behavior.13 The establishment of base line knowledge of the subjec t will determine at what level the therapeutic teaching por tion of the intervention should be targeted. This process will also determine what information the physicians have been given by pharmaceutical company sponsored detailers. The current prescribing behaviors that the physicians exhibit should be analyzed and reasons fo r these behaviors should be discerned. The collection of this baseline information will co ntribute greatly toward an intervention that is relevant to the physician group that is bei ng targeted. Since the detailers only have a small amount of time (15 to 30 minutes) with the physician, a direct and poignant information session will be more effective.13 Two studies have been included that explicitly describe the process that the authors used to establish the baseline knowledge of th eir target audience. Ilett employed a preintervention survey to determine the needs of the general practitioner population that they were detailing. The study resulted in a significant decrease of 1.4% ($AUS 16,130) in cost of antibiotics w ithin the treatment group.43 Solomon used physician deviation from guidelines as an indicator that the interven tion was warranted. This approach led to a statistically significant 41% decrease in inappropriate prescribing of targeted antibiotics.44

PAGE 118

103 The Academic Detailing Service (ADS) has employed a three-tier process in the collection of physician baseli ne knowledge and behaviors. They involve physicians in the selection of subject areas they would like the AD to address. They refine this information through teleconferencing with select physicians throughout the province. They employ a physician advisory panel to formulate major educational points. During the solicitation process for particip ation in an AD session each physician is asked to give feedback on topic areas that th ey would like to receive AD on in the future. This information is collected and is pres ented to a group of physicians from throughout the province via teleconferencing. During the influenza vaccine AD intervention that preceded the OA intervention, physicians indi cated that they w ould like to receive information on the management of OA. This information, along with other options was presented to the teleconference physician gr oup and a decision was made to have the physicians advisory panel develop a lis t of learning objectives for an OA AD intervention. The Dalhousie CME Division th en further developed these major learning points into an academic package for presentation. Focus Intervention on Specific Physicians The success of an AD intervention has been attributed to focusing the intervention on certain groups of physicians.13, 14 In the case of the OA intervention, these groups might include; rheumatologists, physicians wi th large numbers of elderly patients, or simply physicians whose prescribing patterns differ significantly from the best practice guidelines. These groups shoul d be given extra attention as research has shown that changes in their behavior can have a profound effect on the success of the intervention as a whole.13

PAGE 119

104 Five articles described the populations that the interventions were prepared to serve. Ilett and van Eijk targeted the GP population and van Eijk further targeted the pharmacist population.43, 45 Ilett showed a 1.4% decr ease in prescription costs43 and van Eijk showed a decrease in prescribing of highly anticholinergic antidepressants (not significant) and a significant increase in the prescribing of less anticholinergic antidepressants.45 Solomon focused the interven tion on interns and residents and reported a 41% decrease in inappropriate prescribing of targeted antibiotics.44 May and Fender targeted their interventions to the individual physician a nd to the individual physician practice group respectively. May reported a 9% decrease in NSAID prescribing and a decrease in GI events from .20/1000 to .06/1000.46 Fender reported a statistically significan t decrease in specia list referrals (OR = .64) and a significant increase in tranexamic acid prescriptions (OR = 2.38).47 The Nova Scotia OA program has targeted the general practi tioner population. As a voluntary program the targeting of specifi c groups within th e population would be impractical. The targeting of specific physic ian groups, such as high variance physicians, is an area for future consideration in the development of new programs. Define Clear Objectives The definition of clear objectives is esse ntial in the design of an AD intervention. The objectives should be limited in number (3 or 4) and the outcomes of the objectives should be clearly stipulated and measurement criteria developed. E ducational objectives may be to have physicians brought up to da te with best practice guidelines but the evidence of the adoption of information may be seen in the measurement of a behavioral objective. Secondary objectives can also be s tipulated if they are in line with the overall scope of the intervention.13

PAGE 120

105 The definition of clear objectives has been st ated in all of the articles that met the inclusion criteria. The objectives were based on clin ical guidelines,43, 44, 48 or primary literature.45-47 The Dalhousie AD service on OA has stated four learning objectives.6 They are; Discuss the goals of therapy, Recommend non-pharmacological trea tments when appropriate, Advise patients about the safety and efficacy of acetaminophen, and Discuss the role of traditional NSAIDs and COX-2 inhibitors. The desired behavioral change that is an ticipated is the physic ians adherence to best practice guidelines. The behavioral change will be measured through changes in prescribing habits reflected in the provincia l and national administrative databases. If guidelines are being followed, then a decrease in prescriptions for both COX-2 inhibitors and NSAIDs is expected. An increase in the use of acetaminophen and nonpharmacological treatments is e xpected, but unfortunately will not be measured. Indirect measures of appropriate ther apy that can be measured th rough administrative databases include number of visits to primary physicians or specialists and numb er of hospital visits due to side effects of NSAIDs. A second outcome of interest, a spin off of optimal therapy, would be a decrease in drug expenditures. In Nova Scotia COX-2 inhibitors are reimbursed under the Seniors Pharmacare Program on a maximum allowable cost (MAC) basis. The MAC is the maximum daily amount that Pharmacare will pay for any drug in that category. Currently, the MAC for COX-2 inhibitors is set at $1.04. Using Celebrex as an example, if the required daily dose is 400 mg and 100 mg cap sules are being supplied the maximum amount per capsule that Pharmacare will pay is $1.04/4 = $0.26. The patient is required to pay the difference in cost be tween the MAC and the actual cost of the

PAGE 121

106 medication. If a patient therefore, is switc hed from a COX-2 inhibi tor to a traditional NSAID the savings are realized by the pati ent since the MAC for NSAIDs is set to include full payment for five of the NSAID drugs. Establish Credibility There are three criteria that have been id entified as being necessary components of a successful AD intervention. Th e intervention should be produ ced by an agency that has gained professional respect a nd whose views are seen to be independent of bias. It should be based on sound evidence from re spected sources and academically based educators should present it.13 Studies that have established credibility by including these criteria have been shown to have statistically significant outcomes.14 From the included studies; one stated that the program was developed by a university based expert panel43 and one stated that the program was developed by the investigator.45 The programs were presented by a number of different health care professionals including; pharmacists,43, 44 study team members,45, 47 clinician educators44 and physicians.44, 48 The AD intervention on OA has been deve loped through the AD Service of the Continuing Medical Educati on (CME) Division, Faculty of Medicine, Dalhousie University. Dalhousie CME has a long re spected history of providing Maritime physicians with programs designed to improve practice standards. CME has been offered through the Dalhousie Faculty of Medicine in one form or another for over 50 years. In 1949 the Faculty of Medicine at Da lhousie University began the process of formalizing a program to provide CME to physicians throughout Atlantic Canada. In 1954 funding was obtain from the three Maritim e medical societies and in 1957 a division of the Faculty of Medici ne was created to administer postgraduate

PAGE 122

107 programs. By 1968 the division had devel oped a high level of co-operation between small hospitals and was able to deliver i nnovative and highly relevant programs. The reputation of the Division grew and drew intern ational attention in th e form of visits and publications.6 In 2001, Dalhousie CME launched the first province wide AD service in Canada. The first topic addressed an update on influenza and pneumococcal vaccines. The service is funded by the Nova Scotia Department of Health and provide s physicians in the province with a 15 to 20 minute office visit with a trained health professional on a roughly semi-annual basis. The second topic addressed the management of OA and it was offered on a voluntary basis over the summer of 2002. The intervention has been developed and is operated by Dalhousie CME, which is unde r the direction of academically based educators. The interventions content is entirely evidence-based and the planning committee consists of; Two content experts; a local rheumato logist and a drug ev aluation pharmacist. A family physician advisory panel (t hree GPs from across the province). Three academic detailers; two pharmacists and one registered nurse. The Dalhousie OA intervention contains many of the criteria that have been identified as contributing to a successful and unbiased detail ing service. It is funded by the Nova Scotia Department of Health, an unbiased ag ency and is operated by the Continuing Medical Education Division of the Faculty of Medicine, Dalhousie University. It is entirely evidence-based and is being presented by academically based health professionals who are given instructi on on therapeutic cont ent by the Dalhousie CME Division of the Faculty of Medicine and are specially trained in educational

PAGE 123

108 techniques by the Drug and Therapeutics Info rmation Service (DATIS). DATIS is an internationally recognized organization th at provides training in the area of adult education techniques leading to succe ssful medical inte rvention services.49 Stimulate Physician Interaction Success of an AD intervention has been attributed to the ability of the interventions presente r to appeal to the physicians own beliefs, needs and values. This can be achieved by engaging the physician in an interactiv e discussion of the content rather than simply lecturing.13 The methods used to engage a physician in the process are not often explicitly stated in the literature. There were two cases in the included studies were this component was addres sed. In both cases the author s stated that the detailers tailored their presentation to the needs or wants of the physician as a means of stimulating interest.45, 46 It is, however, a necessary component of a successful intervention.14 The Dalhousie OA intervention begins the process of engaging the physician by providing options for additional topics that the detailer can present. The registration page outlines four main messages that will be covered during the visit and it allows the physician to choose any one of se ven additional topics that is of particular interest to them. The detailers use this information to tailor a presentation to each physician. The detailers have also been trained in tec hniques to encourage interactive discussion by DATIS.49 The flexibility that is built into the OA intervention and the specialized training that the detailers ar e given should ensure that th e physicians are appropriately engaged in the learning experience and that their personal needs are met.

PAGE 124

109 Use Concise Graphic Educational Materials The use of concise literature as reminde rs to the physician once the detailing session is completed has been shown to be an effective tool in the success of an intervention. The reminders should be revi ewed during the session with the detailer to ensure that the physician r eceives the proper message.13 The reminders often take the form of pamphlets, pocket sized aides memo ir, graphs, or charts summarizing key points of the presentation. All of the included studies described the physician reminder materials that were developed for their particular intervention. The Dalhousie OA intervention has accomplished this in two ways. They have produced a lengthy guide to leave with the phys ician. Highlighted w ithin text boxes in the guide are summary statements to which the physician can easily refer. The detailer also leaves the physician with a laminated 8 x 11 sheet that summarizes key therapeutic monitoring points of the presentation on one side and provides cost information for different therapies from the perspectives of patients insured under the Nova Scotia Seniors Pharmacare Program and patients with no drug insurance. (Appendix B) For example, drug therapy for a patient who has reached the annual Pharmacare deductible of $350.00 and is prescribed naproxen 500 mg (NSAID) twice daily would cost the government plan $24.46 per month and the pa tient would pay nothing. If the same patient were prescribed celecoxib 100 mg (COX-2 inhibitor) the charge to the government would be $24.89 per month and the patient would pay $21.78. The message to the physician is clear that if the COX-2 therapy is not indi cated then the savings to the patient can be substantial. The goals of therapy are summarized and reinforced for both the physician and the patient through a desktop pamphlet pad pr ovided by the Nova Scotia Division of The

PAGE 125

110 Arthritis Society. The physicia n can review the goals of therapy with a patient and tear off a copy for the patient to take home. The OA intervention ha s partnered with The Arthritis Society to provide a multi-faceted approach to learning and behavior reenforcement. The combination of clear and concise materials presented to the physician during the OA intervention from both the AD se rvice and The Arthritis Society helps to remind the physician of the intervention s main messages during patient visits. Highlight and Reinforce Essentials The repetition of a few major points has been shown to be effective in the presentation of an AD intervention.13 This is especially true in the practice of medicine where the physician has many different messa ges presented to him or her relating to many different disease states and therapies. Even if the intervention addresses a very complex issue the main points must be kept to a minimum. If too much is attempted in the short time that the detail er has with the physician, the ma jor points of the intervention and the desired behavioral changes may not be realized. The few primary messages of the intervention should be repeated and su mmarized throughout the presentation. None of the included studies specifically outlined their methods for reinforcement of the primary messages. One of the articles did st ate that the central messages were reinforced at the follow-up visit.47 The Dalhousie OA intervention has set four primary messages as its objective. These have been discussed earlie r under the section define cl ear objectives. These four messages are discussed thoroughly in the main text of the document provided to the physician. The goals of therapy have been summ arized in a handout format that serves to remind the physician and is available for the patient to take home. The other three primary message of the intervention are summar ized in the main document at the end of

PAGE 126

111 the applicable section and are highlighted in a text box for easy reference. The summary messages have also been compiled and placed at the beginning of the main document for quick reference. Positive Reinforcement with Follow-up The incorporation of a follow-up visit into the AD plan has been shown to have a two-fold effect on the positive outcome of an intervention.13 The use of the follow-up visit to reinforce the main messages of the intervention as well as provide positive feedback to the physician has not been unive rsally employed. In cases where a follow-up was planned it was often several months afte r the initial visit (4 to 6 months) and included feedback to the physician based on data collected sin ce the initial visit.45-47 In one case a negative event (deviation from hos pital established guide lines) triggered a follow-up visit.44 The Dalhousie OA intervention has incor porated a face-to-face follow-up visit by the academic detailer but this follow-up visi t only takes place if it is requested by the physician. The AD service administrators re port that physicians rarely request the follow-up visit. The physician is asked to comment on the usefulness of the intervention using a separate form that is faxed to the Dalhousie CME office. The intervention also offers continuing education credits through the College of Family Physician of Canada if the physician chooses to complete a reflect ive exercise three months following the detailers visit. The OA AD intervention is a methodologica lly sound program. It is evidence based and it meets all but one of the criter ia (the provision of a follow-up visit for all participants) that are defined in the lite rature as being esse ntial components of a successful intervention. The inte rvention is expected to impr ove the quality of care given

PAGE 127

112 to patients in Nova Scotia who suffer from OA. It is expected th at the cost burden of pharmaco-therapy for the Nova Scotia elderl y population will be lessened through the prescribing of equally effective but less expens ive agents. It is also expected that the changes in therapy resulting from the inte rvention will not cause the elderly population any additional morbidity or mortality. Summary The OA AD intervention exhibits strength in several areas that will contribute to its success. The Division of CME at Dalhousie Universitys Faculty of Medicine has ensured that the interventi on meets the needs of the province's physicians through teleconferencing with physicians throughout the province and through consultations with a physician advisory panel. Th e CME division has established itself as a credible source of information for physicians through the pr ovision of educational programs for over 50 years. They have also partnered with the Nova Scotia Division of The Arthritis Society, which is a respected patient advocacy group. The Arthritis Society also adds another facet to the program through their mailings to arthritis patients informing them of the intervention and encouraging them to speak to their physician about their therapy. The program itself is based on solid clinical evidence obtained from respected peer reviewed journals and therapeutic guidelines. It is designed as an interactive discussion between the physician and the detailer through specialized training gi ven to the detailers by the Australian based DATIS organization. Each physician also has the opportunity to customize the message through an order form that allows the physician to select a number of additional messages that he or she would like the deta iler to bring to the session. The post-intervention components of the inte rvention include a survey in which the physician reaffirms the desire to modify thei r behavior to be more in line with the

PAGE 128

113 guidelines that were presented. It also pr ovides the physician with a follow-up reflective exercise to be complete three months after the intervention that again reaffirms the resolve to optimize his or her patients OA therapy. The two areas of weakness in the interven tion are that it is a voluntary program and it does not include a structured face to f ace follow-up visit with the participating physicians. The voluntary nature of the program is static Physicians in Nova Scotia are free to choose which CME credits they participate in. The perceived weakness would be that the physicians who are already perfor ming at a high level will part ake in the intervention and the high variance physicians, who would benefit most from the intervention, will elect to take other forms of CME. This weakness will be addressed in the data analysis, through the use of propensity score methodology. An in -depth description of the propensity score technique is found in the methods section. The formal follow-up by the detailer is a component that is planned for future interventions but is not included with th e OA intervention. This weakness will be especially important in the sustaina bility of the intervention effect.

PAGE 129

114 APPENDIX B OA AD DESKTOP REMINDER Comparative costs of 30 days su pply of drugs for OA. (February 2002) Drug Usual daily dose in OA Approximate prescription price for a person with no drug insurance Approximate Seniors Pharmacare copay (portion of prescription price paid by senior) Co-pay before the senior reaches the $350/year Pharmacare deductible Co-pay after the senior reaches the $350/year Pharmacare deductible Acetaminophen 500mg 2qid Retail Price ~ 13.77 (including tax) Acetaminophen is not a benefit under most drug plans Ibuprofen 400mg tid 13.13 4.20 0.00 Naproxen 500mg bid 24.46 7.26 0.00 Ketoprofen 100mg bid 33.25 9.21 0.00 Flurbiprofen 100mg bid 30.22 10.03 0.00 Flurbiprofen 50mg bid 24.55 11.46 4.91 Diclofenac 25mg tid 26.49 13.20 6.65 Tiaprofenic Acid 300mg bid 33.70 13.72 3.69 Sulindac 200mg bid 38.21 18.14 8.11 Diclofenac SR 75mg bid 43.40 23.32 13.29 Diclofenac 50mg tid 44.60 24.53 14.50 Etodolac 300mg bid 45.17 25.09 15.06 Etodolac 200mg bid 45.77 25.09 15.06 Naproxen 500mg EC bid 50.53 30.46 20.43 Meloxicam 7.5mg od 36.08 17.97 9.67 Meloxicam 15mg od 40.22 16.75 6.80 Celecoxib 100mg bid 46.67 30.08 21.78 Celecoxib 200mg od 46.67 30.08 21.78 Rofecoxib 12.5 od 46.67 35.32 29.65 Rofecoxib 25mg od 46.67 30.08 21.78 Arthrotec 75* bid 58.26 19.42 0.00 Arthrotec 50* tid 63.27 21.08 0.00 Misoprostol 200mcg qid 55.10 16.37 0.00 Omeprazole 20mg** od 75.17 25.05 0.00 Arthrotec is covered under exception status by the Pharmacare pr ograms for the treatment of inflammatory diseases in those patients for whom cytoprotection is required. ** Omeprazole is covered under exception status by the Pharmacare programs.

PAGE 130

115

PAGE 131

116 APPENDIX C THE THEORETICAL FOUNDATI ON FOR ACADEMIC DETAILING Two broad theoretical frameworks that can be used to describe how the academic detailing effect occurs are social theory a nd theory of planned behavior. The first is social theory50 which describes how social values of the individual dictate the importance that an individual places on an interaction. The greater the importanc e attributed to an interaction (social capital) th e greater the chance of uptake of the information. The second theory is expected value theory. This theory is illustrated through the use of two behavioral theories: rational decision theo ry and the theory of planned behavior. The construct of social theory that will be described here is social capital. The common saying its not what you know, but who you know is an easy way to sum up the construct of social capita l as it emphasizes the need fo r networks to succeed. There are two levels to social capital that are appl icable to academic detailing; extra-community networks or Bridging and in tra-community ties or Bonding.51 In terms of academic detailing, the detailer is the individual that needs to possess social capital in order for the educational visit to be successful and the program itself must be valuable on both the extra-co mmunity and intra-community levels. The social capital that the detailers in Nova Scotia possess on the extra-community level includes their affilia tion with the Dalhousie CME Di vision and their professional credentials (two are pharmacists and one is a re gistered nurse). The social capital that they possess on the intra-community level i nvolves the development of a network with physicians over time within their assigned region of the province. The program has

PAGE 132

117 extra-community capital due to its evidence based content and comm unity based capital due to the involvement and buy in of genera l practitioners throughout the province during all stages of the programs development. Th e program is also deve loped in conjunction with a local specialist who adds credibility on both levels by adding a local interpretation to the programs evidence base. The empirical evidence supporting social th eory includes studies that explicitly state the credentials of the detailers and the evidence based value of the academic detailing program.43, 44 Further to this, revi ew articles have summarized that professional and competent detailers, support from trusted institutions, and reliable evidence based information are significant components of a successful academic detailing program.13, 14 The methodological challenge fo r social theory lies in the ability to measure an individuals social capit al. In the context of this study the detailers are competent health care professionals who are known to the physicians in their region. An ability to measure one detailers social capital over another would be valuable in explaining changes that occur in prescribing behavior. The first of the two expected value theori es is a prescriptive or normative theory and is referred to as rational decision theo ry. It describes how rational decisions are made and the balance between what is desired an d what is possible. A particular part of rational decision theory; the expe cted utility theory will be discussed. The second theory is a descriptive theory that is referred to as psychological decision theory. It goes beyond the rational decision theory and has developed propositions to descri be actual behavior.52 The proponents of expected uti lity maintain that it is a normative theory and if the physician adheres to the axioms of the th eory each prescribing decision is made by

PAGE 133

118 considering a number of options with expect ed utility assigned to each. The physician simply chooses the option which maximizes the utility. Two majo r axioms of utility theory are transitivity and independence. Transi tivity states that if A is preferred to B and B is preferred to C then A is preferred to C. Independence states that if A is preferred to B then A with possible consequence C will be preferred to B with the same chance of consequence C.53 The critical construct of interest is th e transitivity axiom of the theory. The academic detailing program attempts to al ter the order of transitivity within the physicians prescribing behavior. If the deta iling session is successful in the transfer of the information that COX-2 inhibitors are only as effective for pain relief as traditional NSAIDs or acetaminophen and have limited e ffect on the reduction of GI events (effective in high risk individuals only) then the cost savings to the patient should place the utility of COX-2 inhibitors lower than that of tradi tional NSAIDs and acetaminophen. The physician would therefore a lter his or her pres cribing behavior away from COX-2 inhibitors. The construct of transitivity within the e xpected utility theory was chosen because if it holds true it has a direct predictive value on the effect of the academic detailing program on prescribing behavior. The deviation from the previously describe d normative behavior is the subject of, and strongest argument for, the theory of planne d behavior. The constructs of interest are behavioral intent and per ceived behavioral control.54 Perceived behavioral control (an individuals perception of their ability to perform a behavior) and intentio n (an individuals readiness to pe rform a behavior) are constructs

PAGE 134

119 that have been identified as significant pred ictors in behavioral change. In order for intent to be manifested into a behavioral ch ange a strong perception of behavioral control must be present.54 It is important to note that actual behavi ors (behavioral categor ies) are a collection of single acts and cannot be measured. Si ngle acts include the day to day physician activities of diagnosis, prescr ibing and referrals and since the collection of single acts make up the behavioral category the measuremen t of the collection of single acts can act as a proxy for the measurement of the behavior al category. A behavior al act consists of four behavioral elements; an ac tion, target, context, and time.55 In the context of this study the four elemen ts in the act of pres cribing are an action the writing of a prescription, a target th e patient, a context the physicians office, and a time during a patient visit. The writ ten prescription is a measurable item. The collection of prescriptions is the behavioral category that the academic detailing intends to alter. Ilett and May have provided examples of studies analyzing the effects of academic detailing that support the proposed theories.43, 46 Two necessary components of an academic detailing program are that it is evidence based and delivered by credible and trusted detailers. These components support the theories because they put the physician in a position of accepting information that evidence shows should change his or her prescribing behavior. Ilett and May devel oped interventions base d on evidence and the delivery of the program was carried out by reputable agents. Both studies showed significant changes in physic ian prescribing behavior.43, 46

PAGE 135

120 The empirical evidence supporting the theory of planned behavi or is illustrated through academic detailing intervention studies that have not been successful in changing physician prescribing behavior.19, 21 The two studies contained many of the components required for a successful academic detailing intervention however, they were admittedly over ambitious and the message was too complicated for the physician to adopt. The constructs that have been presented would explain the fa ilure of the studies by noting that the intervention may have overwhelmed the phy sicians and thereby decreased his or her intent to adopt new behavior a nd the perceived ability to contro l behavior. If the intent to change and perceived behavior al control are not present th en the act of prescribing differently would not be carried out and the effect of the intervention is lost. The main advantage of the expected utility theory is that it is normative and can be quantitatively measured. The predictive ability of the th eory however does not explain why the physician changed his or her behavior In the context of the osteoarthritis academic detailing intervention the utility theory could be applied to many of the physician patient interactions that lead to th e issuing of a prescrip tion. The theory would not explain however deviations from the norm such as prescribing a COX-2 agent to a patient simply because the physician perceives that the patient can afford it. The theory of planned behavior on the other hand does not have the direct pred ictive power of the utility theory but it explains why the physicia ns prescribing behavior is changed. This theory would maintain that a well designed detailing intervention would have the effect of providing the physician with a limited numbe r (3 or 4) of messages thus the physician would perceive his or her ability to change and would intend to change his or her behavior. If this intent is acted upon in a timely fashion then the intent could be

PAGE 136

121 translated into the act of prescribing and the repeated act leads to a change in prescribing behavior.

PAGE 137

122 LIST OF REFERENCES 1. College of Physicians and Surgeons of Nova Scotia. 2002 Annual Listing Halifax, NS: College of Physicians and Surgeons of Nova Scotia; 2002. 2. Statistics Canada. 2001 Census of Canada. June 16, 2005; last accessed June 20, 2005. 3. Holbrook AM. Ontario Treatment Guidelines for Osteoarthritis, Rheumatoid Arthritis, and Acute Mu sculoskeletal Injury. Toronto: Ontario Musculoskeletal Therapy Review Panel; 2002. 4. MacLean CH. Quality indicators for the management of osteoarthritis in vulnerable elders. Ann Intern Med. Oct 16 2001;135(8 Pt 2):711-721. 5. Lawrence RC, Helmick CG, Arnett FC, et al. Estimates of the prevalence of arthritis and selected musculoskele tal disorders in the United States. Arthritis Rheum. May 1998;41(5):778-799. 6. Dalhousie University Conti nuing Medical Education Website. last accessed September 22, 2005. 7. Silverstein FE, Faich G, Goldstein JL, et al. Gastrointestinal toxicity with celecoxib vs nonsteroidal anti-inflammatory drugs for osteoarthritis and rheumatoid arthritis: the CLASS study: A randomized controlled trial. Celecoxib Long-term Arthritis Safety Study. JAMA. Sep 13 2000;284(10):1247-1255. 8. Bombardier C, Laine L, Reicin A, et al. Comparison of upper gastrointestinal toxicity of rofecoxib and naproxen in pa tients with rheumatoid arthritis. VIGOR Study Group. N Engl J Med. Nov 23 2000;343(21):1520-1528. 9. Cooke C. Utilization of COX-2 Inhibitors in the Nova Scotia Seniors Population: Dalhousie College of Pharmacy; 2001. 10. Cox ER, Motheral B, Frisse M, Behm A, Mager D. Prescribing COX-2s for patients new to cyclo-oxyge nase inhibition therapy. Am J Manag Care. Nov 2003;9(11):735-742. 11. Rosenbaum PR, Rubin DB. Reducing Bi as in Observational Studies Using Subclassification on the Propensity Score. Journal of the Am erican Statistical Association. September 1984;79(387):516-524.

PAGE 138

123 12. D'Agostino RB, Jr. Propensity score methods for bias reduction in the comparison of a treatment to a non-randomized control group. Stat Med. Oct 15 1998;17(19):2265-2281. 13. Soumerai SB, Avorn J. Principles of e ducational outreach ('academic detailing') to improve clinical decision making. JAMA. Jan 26 1990;263(4):549-556. 14. Davis DA, Thomson MA, Oxman AD, Haynes RB. Changing physician performance. A systematic review of th e effect of continuing medical education strategies. JAMA. Sep 6 1995;274(9):700-705. 15. Seeger JD, Williams PL, Walker AM. An application of propensity score matching using claims data. Pharmacoepidemiol Drug Saf. Jan 13 2005. 16. Shadish WR, Cook TD, Campbell DT. Experimental and quasi-experimental designs for generalized causal inference Boston, MA: Houghton Mifflin; 2001. 17. Grimshaw JM, Thomas RE, MacLennan G, et al. Effectiveness and efficiency of guideline dissemination and implementation strategies. Health Technol Assess. Feb 2004;8(6):iii-iv, 1-72. 18. Thomson O'Brien MA, Oxman AD, Davis DA, Haynes RB, Freemantle N, Harvey EL. Educational outreach visits: effects on professional practice and health care outcomes. Cochrane Database Syst Rev. 2000(2):CD000409. 19. Lin EH, Simon GE, Katzelnick DJ, P earson SD. Does physician education on depression management improve treatment in primary care? J Gen Intern Med. Sep 2001;16(9):614-619. 20. Lin EH, Katon WJ, Simon GE, et al. Ac hieving guidelines for the treatment of depression in primary care: is physician education enough? Med Care. Aug 1997;35(8):831-842. 21. Brown JB, Shye D, McFarland BH, Nichols GA, Mullooly JP, Johnson RE. Controlled trials of CQI and academic de tailing to implement a clinical practice guideline for depression. Jt Comm J Qual Improv. Jan 2000;26(1):39-54. 22. Goldberg HI, Wagner EH, Fihn SD, et al. A randomized controlled trial of CQI teams and academic detailing: can th ey alter compliance with guidelines? Jt Comm J Qual Improv. Mar 1998;24(3):130-142. 23. Zwar NA, Wolk J, Gordon JJ, SansonFisher RW. Benzodiazepine prescribing by GP registrars. A trial of educational outreach. Aust Fam Physician. Nov 2000;29(11):1104-1107. 24. Tomson Y, Hasselstrom J, Tomson G, Ab erg H. Asthma education for Swedish primary care physicians--a study on the effect s of "academic detailing" on practice and patient knowledge. Eur J Clin Pharmacol. 1997;53(3-4):191-196.

PAGE 139

124 25. Gorins A. Preventing breast cancer. Eur J Gynaecol Oncol. 2000;21(3):213. 26. Weitzen S, Lapane KL, Toledano AY, Hu me AL, Mor V. Principles for modeling propensity scores in medical research : a systematic literature review. Pharmacoepidemiol Drug Saf. Dec 2004;13(12):841-853. 27. Rubin DB. Estimating causal effects from large data sets using propensity scores. Ann Intern Med. Oct 15 1997;127(8 Pt 2):757-763. 28. Parsons LS. Reducing Bias in a Propen sity Score Matched-Pair Sample Using Greedy Matching Techniques. Paper pres ented at: 26th Annual SAS Users Group International Conference, 2001; Long Beach, CA. 29. Austin PC, Mamdani MM, Stukel TA, Anderson GM, Tu JV. The use of the propensity score for estimating treatment ef fects: administrative versus clinical data. Stat Med. May 30 2005;24(10):1563-1578. 30. Cepeda MS, Boston R, Farrar JT, Strom BL. Comparison of logistic regression versus propensity score when the number of events is low and there are multiple confounders. Am J Epidemiol. Aug 1 2003;158(3):280-287. 31. Joffe MM, Rosenbaum PR. Invite d commentary: propensity scores. Am J Epidemiol. Aug 15 1999;150(4):327-333. 32. Weitzen S, Lapane KL, Toledano AY, Hume AL, Mor V. Weaknesses of goodness-of-fit tests for evaluating propensit y score models: the case of the omitted confounder. Pharmacoepidemiol Drug Saf. Apr 2005;14(4):227-238. 33. Seeger JD, Walker AM, Williams PL, Saperia GM, Sacks FM. A propensity scorematched cohort study of the effect of stat ins, mainly fluvastatin, on the occurrence of acute myocardial infarction. Am J Cardiol. Dec 15 2003;92(12):1447-1451. 34. Canadian Institute for Health Information. Privacy and confiden tiality of health information at CIHI: principles and policie s for the protection of personal health information and policies for instit ution-identifiabl e information 3rd ed. Ottawa, ON: Canadian Institute for Health Information; 2002. 35. Sas Institute Inc. SAS/STAT Software: Release 8.2 Cary, NC: SAS Institute Inc. 36. WHO. Vol 2005: World Health Organi sation Collaborating Centre for Drug Statistics Methodology; Oslo, NO; 2005. 37. Twisk JWR. Applied longitudinal data analysis for epidemiology: a practical guide Cambridge, UK; Cambridge University Press; 2003. 38. Health Canada Website. last accessed Dec 02, 2005.

PAGE 140

125 39. Nova Scotia Department of Health Website. last accessed December 02, 2005. 40. Strom BL. Pharmacoepidemiology 4th ed. Chichester; Hoboken, NJ: J. Wiley; 2005. 41. Wolfe MM, Lichtenstein DR, Singh G. Gast rointestinal toxicity of nonsteroidal antiinflammatory drugs. N Engl J Med. Jun 17 1999;340(24):1888-1899. 42. Roos LL, Mustard CA, Nicol JP, et al Registries and administrative data: organization and accuracy. Med Care. Mar 1993;31(3):201-212. 43. Ilett KF, Johnson S, Greenhill G, et al. Modification of general practitioner prescribing of antibiotics by use of a th erapeutics adviser (academic detailer). Br J Clin Pharmacol. Feb 2000;49(2):168-173. 44. Solomon DH, Van Houten L, Glynn RJ, et al. Academic detailing to improve use of broad-spectrum antibiotics at an academic medical center. Arch Intern Med. Aug 13-27 2001;161(15):1897-1902. 45. van Eijk ME, Avorn J, Porsius AJ, de Boer A. Reducing prescribing of highly anticholinergic antidepressants for elderly people: randomised trial of group versus individual academic detailing. BMJ. Mar 17 2001;322(7287):654-657. 46. May FW, Rowett DS, Gilbert AL, McNeece JI, Hurley E. Outcomes of an educational-outreach service for community medical practitioners: non-steroidal anti-inflammatory drugs. Med J Aust. May 17 1999;170(10):471-474. 47. Fender GR, Prentice A, Gorst T, et al. Randomised controlled trial of educational package on management of menorrhagia in primary care: the Anglia menorrhagia education study. BMJ. May 8 1999;318(7193):1246-1250. 48. Denton GD, Smith J, Faust J, Holmboe E. Comparing the efficacy of staff versus housestaff instruction in an interventi on to improve hypertension management. Acad Med. Dec 2001;76(12):1257-1260. 49. National Prescribing Service Website. last accessed September 22, 2005. 50. Coleman JS. Foundations of social theory Cambridge, MA: Belknap Press of Harvard University Press; 1990. 51. Woolcock M, Narayan D. Social Capita l: Implications for Development Theory, Research, and Policy. World Bank Research Observer. 2000;15 (2). 52. Kozielecki J. Psychological decision theory Boston, MA: Reidel PWN-Polish Scientific Publishers, 1981.

PAGE 141

126 53. Cohen BJ. Is expected utility theory normative for medical decision making? Med Decis Making. Jan-Mar 1996;16(1):1-6. 54. Icek Ajzen Theory of Planne d Behavior Website. last accessed December 05, 2005. 55. Ajzen I. Understanding attitudes and pr edicting social behavior Englewood Cliffs, NJ: Prentice-Hall; 1980.

PAGE 142

127 BIOGRAPHICAL SKETCH Stephen Grahams early education was deep ly rooted in the Jesuit tradition. He attended St. Pauls High School and in 1985 he graduated with a Bachelor of Science degree from St. Pauls College at the Univ ersity of Manitoba, in Winnipeg, Canada. He received his Air Navigators Wings fr om the Canadian Forces Air Navigation School and served in the Canadian Air For ce as a line navigator until 1992 when began his pharmacy degree at Dalhousie Univers ity in Halifax, Nova Scotia, Canada. Stephen graduated with a Bachelor of Science (pharmacy) degree in 1997 and was employed within the Canadian Forces Me dical System until his retirement in 2000. His academic interests are in the areas of physician behavioral change and in the methodology associated with quasi-experimental design. He plans to contribute to the Canadian health care system through continue d work in the areas of health policy and quantitative assessment of medical outcomes.


Permanent Link: http://ufdc.ufl.edu/UFE0011901/00001

Material Information

Title: Effect of Academic Detailing on COX-2 Utilization Rates
Physical Description: Mixed Material
Copyright Date: 2008

Record Information

Source Institution: University of Florida
Holding Location: University of Florida
Rights Management: All rights reserved by the source institution and holding location.
System ID: UFE0011901:00001

Permanent Link: http://ufdc.ufl.edu/UFE0011901/00001

Material Information

Title: Effect of Academic Detailing on COX-2 Utilization Rates
Physical Description: Mixed Material
Copyright Date: 2008

Record Information

Source Institution: University of Florida
Holding Location: University of Florida
Rights Management: All rights reserved by the source institution and holding location.
System ID: UFE0011901:00001


This item has the following downloads:


Full Text












EFFECT OF ACADEMIC DETAILING ON COX-2 UTILIZATION RATES


By

STEPHEN DOUGLAS GRAHAM


















A DISSERTATION PRESENTED TO THE GRADUATE SCHOOL
OF THE UNIVERSITY OF FLORIDA IN PARTIAL FULFILLMENT
OF THE REQUIREMENTS FOR THE DEGREE OF
DOCTOR OF PHILOSOPHY

UNIVERSITY OF FLORIDA


2005

































Copyright 2005

by

Stephen D. Graham



































To Evie















ACKNOWLEDGMENTS

I would like to thank my wife, Andrea, and sons, Nicolas and Andrew, for their

love and support.

I thank my dissertation chair, Dr. Abraham Hartzema, and committee members,

Drs. Ingrid Sketris, Almut Winterstein, Richard Segal, and Babette Brumback for their

guidance through the dissertation process.

I would like to extend special thanks to Ms. Dawn Frail at the Nova Scotia

Department of Health and again to Dr. Ingrid Sketris at Dalhousie University for

providing me with overwhelming support and encouragement to succeed and to return to

Canada.

Finally, I would like to thank the graduate students for giving me many happy

memories of Florida.
















TABLE OF CONTENTS

page

A C K N O W L E D G M E N T S ................................................................................................. iv

LIST OF TABLES ......... ........ ................................... .......... .... .......... .. ix

L IST O F F IG U R E S .... ...... ................................................ .. .. ..... .............. xii

ABSTRACT ........ .............. ............. ...... ...................... xiv

CHAPTER

1 IN TR OD U CTION ............................................... .. ......................... ..

B background ........................................................................ ...............
Problem Statem ent .......................................................... .. ..... ...... ....
Research Questions and H ypotheses ........................................ ........................ 4
R research Q u estion 1 ................................................... .. ........ .......... .. .. ....
R research Question 1 H ypothesis....................................... ......... ............... 5
Research Question 2 .................. .............................. ......... .. ..........
R research Question 2 hypothesis....................................... ......................... 6
R research Q question 3 ................................................... .. ........ ............ .. .. .
R research Question 3 hypotheses ........................................ ....................... 6
Research Question 4 .................. .............................. ......... .. ..........
R research Question 4 hypotheses ........................................ ....................... 7
Significance of R research .......................................................... ..............7

2 LITERA TURE REVIEW .......................................................... ..............9

Review Articles Addressing Effects of Academic Detailing ......................................9
Academic Detailing Studies Reporting No Statistically Significant Effect ..............12
P ro p e n sity S c o re s.................................................................................................. 17

3 M E T H O D S ........................................................................................................... 2 2

Step One: Extraction and Validation of Data ..........................................................22
S o u rc e s o f D ata .............................................................................................. 2 2
G P Inclusion C riteria........... ........................................ ...... .......................25
P patient Inclusion C riteria.......................................................... ............... 26









Step Two: Adjustment for Confounding Using Three Distinct Propensity Score
M e th o d s .................................. .............................................. 2 6
Quintile Propensity Score M ethod ........................................... ............... 29
Regression on the Propensity Score M ethod......................................................29
"Greedy Matching" Method ............. .................... .......................... 29
Propensity Score M ethod Selection................................ ............... ............... 29
Step Three: Primary Outcome Analysis; Intervention Effect on COX-2 Utilization
R a te s ........................................... ................ ............ ...... .......................... 3 2
Step Four: Secondary Outcome Analyses; The Utilization of Other Health Care
Resources Associated with NSAID Induced GI Side Effects..............................35

4 R E S U L T S .......................................................................... 3 7

Step One: Extraction and Validation of Data ................................. ...... ............ ....37
Step Two: Establishment of Balanced Control and Experimental Groups Using
Three Propensity Score M ethods ........................................ ........................ 37
Pre-Propensity Score A analysis ........................................ ........................ 37
Quintile PS M ethod Analysis ................................................... ............... 39
Regression on the Propensity Score Method Analysis.................. .......... 41
"Greedy Matching" Method Analysis............................................. 41
Selection of a Preferred Propensity Score Method..................................... 43
Exploratory Analysis of the Propensity Score Methods Effect on Adjusting
for Bias on Unm measured Variables ...................................... ............... 44
Step 3: Prim ary Outcom e A nalysis....................................... .......................... 49
M odel D evelopm ent ..................... ............... ....................... ............... 49
Between Group Results ........... .................... ...... ............... 51
W within Group (Longitudinal) Results....................................... ............... 53
Step 4: Secondary Outcom e Analyses............................................... .................. 55
M isoprostol Utilization Rates ......................................... ....................... ....... 55
M odel develop ent....... .................................... ...... ........ ............. 55
B etw een group results .............................................. ......... ............... 55
Within group (longitudinal) results ............................................................57
PPI U tilization R ates .................................... ................... ..... .... 59
M odel develop ent...................... ..... .............................. 59
B etw een group results ............................................... ........ ............... 60
Within group (longitudinal) results ............................................................61
H 2A U tilization R ates .............................................................. .....................63
M odel develop ent...................... ..... .............................. 63
B etw een group results ............................................... ........ ............... 63
Within group (longitudinal) results ............................................................66
G P O office V isit R ates ............................................... ............................... 67
M odel develop ent...................... ..... .............................. 67
B etw een group results .............................................. ......... ............... 67
Within group (longitudinal) results ............................................................70
Rheumatologist and GI Specialist Visit Rates................... .......................... 72
M odel develop ent...................... ..... .............................. 72
B etw een group results ............................................................................ 72









Within group (longitudinal) results ............................................................74
Hospitalization Rates Due to GI Complications ...........................................76
M odel develop ent...................... ..... .............................. 76
B etw een group results ............................................... ........ ............... 77
Within group (longitudinal) results ............................................................80
D eath Rates Due to GI Com plications ..................................... .................81
M odel develop ent.................... .. .................... ...... ................ .....8 1
Between and within group results ............................................ ..........82

5 D ISCU SSION .............. .... .. ..... .............. ............................ 83

The Academic Detailing Program in Nova Scotia ............................................. 83
Qualifications of the D etailers...................................................... ................. 83
Changes Which Occurred Over the Period of the Intervention (History
Effects) ............. ............. ........ ...............................................83
Policy Options Available to Decision Makers .......................................... 84
Distribution of educational m aterial............................................... 85
Educational m meetings ............................................................................85
Audit and feedback .......... ............................ ........ .... ................85
Reminders and reminder systems............................. .......................86
D rug benefit changes................ .. ..... ............................ ......... 86
Primary Outcome: Effect on COX-2 Utilization Rates ...........................................86
S tatistic al R e su lts........................................................................................... 8 7
Practical Significance .......................... .............. ... ...... .. .... ...........87
Com prison w ith Literature ....................................................... ............... 88
Secondary Outcom es .......................................................... .... ...... .. ...... .. 88
Effect on Gastro-Protective Agents Utilization Rates .........................................88
M iso p ro sto l ................................................................................ 8 8
P P Is ......................................................................................................... 8 9
H 2A s ................................................................ .. .................. 89
Effect on Utilization of Medical Services ....................................................89
GP office visits ..... ........... ........ ...... ...........90
Specialist office visits................................................ ................... 90
Hospitalization rates due to GI side effects...............................................91
Death due to GI complications....................... ...............92
Propensity Score Analysis M ethods ..................................................................... 92
"Greedy Matching" Method ............................ ............... ............... 92
Q uintile M ethod...................................................... ............. 93
R egression on the PS M ethod......................................... .......... ............... 93
P S Exploratory A nalysis............................................ .............................. 93
L im stations ......... ..... ................................................................. ....... 94
D ata L im stations .......................................... ................... ........ 94
D esign Lim stations ...................................... ................... ..... .... 96
C o n clu sio n s..................................................... ................ 9 8









APPENDIX

A AN APPRAISAL OF THE NOVA SCOTIA OA AD INTERVENTION.............101

Conduct Interview s w ith Physicians .............................. ..................... 102
Focus Intervention on Specific Physicians ........... ....................... ....................103
D efin e C lear O bjectiv es............................................. ......................................... 104
E establish Credibility .......................... ......... .. ...... ............... 106
Stim ulate Physician Interaction .................................................... ............... ... 108
U se Concise Graphic Educational M materials ...........................................................109
H highlight and R enforce E ssentials ................................. ...................................... 110
Positive Reinforcement with Follow-up.......................... .......................... 11
S u m m a ry ........................................................................ ............... 1 12

B OA AD DESKTOP REM INDER ................................... ....................114

C THE THEORETICAL FOUNDATION FOR ACADEMIC DETAILING.............116

L IST O F R E F E R E N C E S ......... ................. ................................................................. 122

BIOGRAPHICAL SKETCH ...... ........ ................... ............................ 127
















LIST OF TABLES


Table pge

2-1 Summary of Included Studies for Thomson O'Brien and Grimshaw....................11

3-1 PS M odel Variable Descriptions and Abbreviations ................................................27

4-1 Descriptive Statistics for Continuous Variables in the PS Model............................37

4-2 Descriptive Statistics for Categorical Variables in the PS Model............................38

4-3 Pre-PS Univariate Analysis for Included Variables ......................................... 39

4-4 Physician D distribution by Quintile............................................... ........ ....... 40

4-5 Quintile Method Regression Analysis Results ....................................................40

4-6 Distribution of Influenza AD Participants by Propensity Score Quintile ................41

4-7 Regression on PS Method Analysis Results. ........................................................42

4-8 "Greedy Matching" Method Analysis Results. .................. ....................... 43

4-9 Quintile Method Results for Excluded Variable Models......................................45

4-10 Regression on PS Results for Excluded Variable Models. .....................................47

4-11 "Greedy Matching" Results for Excluded Variable Models ..................................47

4-12 Correlation Matrix Between VOC and PS Covariates...................... ..............48

4-13 Primary Outcome Model Results (Periods = 3,4,5,6). ...........................................52

4-14 Primary Outcome Model Results (Periods = 1,2). ................................................52

4-15 Least Square Means for Change in COX-2 Rates by Group............................... 53

4-16 Unadjusted Means for Change in COX-2 Rates by Group. ....................................53

4-17 Primary Outcome Model Results (AD = yes). ................................. ...............54

4-18 Primary Outcome Model Results (AD = no). .................................. .................55









4-19 Secondary Misoprostol Outcome Model Results (Periods = 3,4,5,6)......................56

4-20 Secondary Misoprostol Outcome Model Results (Periods = 1,2) ..........................56

4-21 Least Square Means for Change in Misoprostol Rate by Group...........................57

4-22 Unadjusted Means and Standard Deviations for Change in Misoprostol Rate by
G ro u p ...................... .. .. ......... .. .. ...................................................5 8

4-23 Secondary Misoprostol Outcome Model Results (AD = yes)..............................58

4-24 Secondary Misoprostol Outcome Model Results (AD = no). ................................59

4-25 Secondary PPI Outcome Model Results (Periods = 3,4,5,6). .................................60

4-26 Secondary PPI Outcome Model Results (Periods = 1,2). ......................................61

4-27 Least Square Means for Change in PPI Rates by Group................................61

4-28 Unadjusted Means for Change in PPI Rate by Group.........................................62

4-29 Secondary PPI Outcome Model Results (AD = yes). ............................................63

4-30 Secondary PPI Outcome Model Results (AD = no) .................. ... .............63

4-31 Secondary H2A Outcome Model Results (Periods = 3,4,5,6). .............................64

4-32 Secondary H2A Outcome Model Results (Periods = 1,2). .....................................65

4-33 Least Square Means for Change in H2A Rate by Group. ......................................65

4-34 Unadjusted Means for Change in H2A Rate by Group ............................................66

4-35 Secondary H2A Outcome Model Results (AD = yes). .........................................67

4-36 Secondary H2A Outcome Model Results (AD = no).........................................67

4-37 Secondary GP Office Visit Model Results (Periods = 3,4,5,6) .............. ...............68

4-38 Secondary GP Office Visit Outcome Model Results (Periods = 1,2)......................69

4-39 Least Square Means for Change in GP Office Visit Rate by Group......................69

4-40 Unadjusted Means and Standard Deviations for Change in GP Office Visit Rate
b y G ro u p ....................................................................... 7 0

4-41 Secondary GP Office Visit Outcome Model Results (AD = yes). .........................71

4-42 Secondary GP Office Visit Outcome Model Results (AD = no). ..........................72









4-43 Secondary Specialist Office Visit Model Results (Periods = 3,4,5,6). ..................73

4-44 Secondary Specialist Office Visit Outcome Model Results (Periods = 1,2)............74

4-45 Least Square Means for Change in Specialist Office Visit Rate by Group ............74

4-46 Unadjusted Means and Standard Deviations for Change in Specialist Office
Visit Rate by Group.............. .. .. ...................... .. ......75

4-47 Secondary Specialist Office Visit Outcome Model Results (AD = yes) ................76

4-48 Secondary Specialist Office Visit Outcome Model Results (AD = no) ..................76

4-49 Secondary Hospital Length of Stay Model Results (Periods = 3,4,5,6)...................77

4-50 Secondary Hospital Length of Stay Outcome Model Results (Periods = 1,2)........78

4-51 Least Square Means for Change in Hospital Length of Stay Rates by Group........79

4-52 Unadjusted Means and Standard Deviations for Change in Hospital Length of
Stay R ates by G roup............. .... .................................................... .. .... ........79

4-53 Secondary Hospital Length of Stay Outcome Model Results (AD = yes)...............80

4-54 Secondary Hospital Length of Stay Outcome Model Results (AD = no) ..............81
















LIST OF FIGURES


Figure page

2-1 Distribution of Propensity Score Article Objectives: 1987 to July 20, 2005...........19

3-1 Propensity Score Logistic Regression M odel .................................. ............... 28

3-2 Experim ental D esign Tim eline.......................................... ........................... 32

3-3 Primary Outcome Model for Between Group Effect ............................................34

4-1 Frequency of Influenza AD Participants by Propensity Score..............................42

4-2 Comparison of PS methods Ability to Reduce Bias on VOC................................45

4-3 Summary of PS Models Effects on Reducing Bias on the VOC ...........................48

4-4 Scatterplots of Propensity Score Versus Unbalanced Variables............................49

4-5 Line Graph Comparing Correlations and Percent Bias Reduction ........................50

4-6 Prim ary O utcom e M odel ......... ................. ..........................................................51

4-7 Least Square Means for Change in COX-2 Rates by Group................ ......... 53

4-8 Unadjusted Means for Change in COX-2 Rates by Group ....................................54

4-9 Secondary Outcome Model for Misoprostol Utilization.............................55

4-10 Least Square Means for Change in Misoprostol Rates by Group..........................57

4-11 Unadjusted Means for Change in Misoprostol Rates by Group..............................58

4-12 Secondary PPI Outcom e M odel ........................................ .......................... 59

4-13 Least Square Means for Change in PPI Rates by Group............... ...................61

4-14 Unadjusted Means for Change in PPI Rates by Group. ........................................62

4-15 Secondary Outcome Model for H2A Utilization ........................................... 64

4-16 Least Square Means for Change in H2A Rates by Group ..................................65









4-17 Unadjusted Means for Change in H2A Rates by Group. .......................................66

4-18 Secondary Outcome Model for GP Office Visits................... .......................... 68

4-19 Least Square Means for Change in GP Office Visit Rates by Group ....................70

4-20 Unadjusted Means for Change in GP Office Visit Rates by Group......................71

4-21 Secondary Outcome Model for Specialist Office Visits.......................................72

4-22 Least Square Means for Change in Specialist Office Visit Rates by Group............75

4-23 Unadjusted Means for Change in Specialist Office Visit Rates by Group .............75

4-24 Secondary Outcome Model for Hospital Length of Stay ...................................77

4-25 Least Square Means for Change in Hospital Length of Stay Rates by Group........79

4-26 Unadjusted Means for Change in Hospital Length of Stay Rates by Group............80

4-27 Secondary Outcome Model for Deaths Due to GI Complications........................81















Abstract of Dissertation Presented to the Graduate School
of the University of Florida in Partial Fulfillment of the
Requirements for the Degree of Doctor of Philosophy

EFFECT OF ACADEMIC DETAILING ON COX-2 UTILIZATION RATES

By

Stephen Douglas Graham

December, 2005

Chair: Abraham Hartzema
Major Department: Pharmacy Health Care Administration

Background: The prevalence of osteoarthritis (OA) is estimated at 50 to 80 % of

the elderly population and therapy aims to relieve symptoms since there is no cure. Nova

Scotia general practitioners (GPs) identified a need for an academic detailing (AD)

intervention aimed at optimizing the management of OA.

Objectives: The primary objective was to measure the effect of an OA AD

intervention to reduce the utilization rate of COX-2 inhibitors in the elderly population.

Secondary objectives were to examine the intervention effect on the utilization rates of

gastro-protective agents and medical services.

Methods: We conducted a retrospective cohort study employing administrative

data to examine the effects of the intervention. Differences in utilization rates were

evaluated using generalized estimating equation (GEE) analysis for longitudinal data.

Selection bias was anticipated since the intervention was voluntary, and

randomization not possible. Three methods of propensity score (PS) analysis (quintile









stratification, regression on the PS, and "greedy matching") were evaluated for the ability

to adjust for bias on PS model covariates.

Findings: We identified a significant difference in the change in COX-2 utilization

rates between groups for the three month period following the intervention (p = 0.0395,

95% CI (0.0365, 1.4815)) and a significant decrease in the intervention group's within

group utilization rate between the pre and post intervention periods (z = -2.34, p =

0.0191). The GP office visit rate was the only secondary outcome where the intervention

group was significantly higher (p = 0.0275, 95% CI (-0.7926, -0.0464)). The difference

occurred in the time period from three to six months post intervention.

Conclusions: The OA AD intervention was associated with a significant decrease

in COX-2 utilization rates in the three month period immediately following the

intervention. The effect of decreased utilization continued for the rest of the post

intervention periods but was not statistically significant. The only secondary outcome to

show a significant between groups effect was the GP office visit rate which was higher

for the intervention group in the second three month post intervention time period.














CHAPTER 1
INTRODUCTION

Background

In June 2002, the Division of Continuing Medical Education (CME), Dalhousie

University Faculty of Medicine, began their second academic detailing (AD) intervention

with provincial physicians aimed at optimizing the care of osteoarthritis (OA) within the

seniors population (persons greater than 65 years of age). The AD program is an ongoing

initiative funded by the Nova Scotia Department of Health and managed by the Drug

Evaluation Alliance of Nova Scotia (DEANS). As the AD program is a continuing effort

and represents a significant cost to DEANS it is necessary to evaluate the effectiveness of

the intervention.

The OA topic was chosen as an AD intervention based on the extent to which OA

affects the elderly population and on the feedback that Dalhousie CME received from

general practitioners (GPs) in a survey filled out following the previous influenza AD

intervention which indicated the GPs' desire to have an OA AD intervention developed.

The Dalhousie CME Division then presented the OA topic to a GP focus group where the

need for education pertaining to available OA therapies was determined.

OA is a progressive disease that affects the joint cartilage and eventually leads to

joint failure.3 The prevalence of OA in the population is extremely high. It is estimated

that 50 to 80% of the elderly population experience symptomatic OA.4 Estimates specific

to the province of Ontario, propose that almost all persons over the age of 65 exhibit

signs of OA on radiographic evidence and of these 33% are symptomatic.5 OA is equally









prevalent in men and women, with women showing more manifestation in the knees and

hands and men more prevalent in the hip. Arthritis has been associated with half of all

disability in the elderly population.4 There is no known cure for OA3 and available

palliative treatments are associated with substantial toxicity and side effects.4 Treatment

is therefore primarily aimed at reducing pain, improving joint mobility, and limiting

functional disability. Patient education regarding medications used in the treatment of

OA (primarily for the control of pain) and appropriate exercise regimens is also

important.3' 4

The OA AD intervention has set four learning objectives. Each physician visit will

include at least the following: (1) a discussion of the goals of therapy, (2)

recommendations for non-pharmacological treatments when appropriate (e.g., physio-

therapy and exercise), (3) advice for patients about the safety and efficacy of

acetaminophen, and (4) a discussion of the role of traditional non-steroidal anti-

inflammatory drugs (NSAIDs).6

The primary interest of this research dealt with the fourth message specifically, the

analysis of the effectiveness of the OA AD intervention as it pertains to the

pharmacotherapy of OA and in particular the usage of COX-2 inhibitors. The Nova

Scotia OA AD was developed in 2002 and the intervention called for the use of

acetaminophen as a first line therapy for mild to moderate OA. The intervention

suggested that if acetaminophen did not control pain symptoms, then the use of

traditional NSAIDs in as low a dose as possible and for as short duration as possible was

indicated. NSAIDs were considered appropriate therapy for moderate to severe OA.6









The role of COX-2 inhibitors in the management of OA was assessed by the OA

AD group as controversial. The Ontario Treatment Guidelines for OA recommend that,

based on evidence of similar efficacy and early evidence of somewhat lower rates of

serious GI events, selective COX-2 inhibiting NSAIDs can be considered for patients at

high risk of serious GI events.3 This recommendation, however, is one that is from well-

designed, randomized controlled trials or meta-analyses with inconsistent results or

demonstrating equivocal benefit.3 The Nova Scotia program states that, the precise role

of COX-2 inhibitors in the treatment of OA remains to be determined.6 The summary

statements in the OA AD intervention6 relay two points that are relevant to this analysis.

Firstly, COX-2 inhibitors are as effective but not more effective than traditional NSAIDs

for symptomatic treatment of OA and secondly, the CLASS7 and VIGOR8 trials were

inconclusive in the analysis of the gastro-protective effects of COX-2 inhibitors.

When faced with the substantial effect of OA on the population,6 the uncertain role

of COX-2 inhibitors in the treatment of OA, the increased cost of COX-2 inhibitors over

the traditional NSAIDs (appendix c), and the utilization rate of COX-2 inhibitors in the

Nova Scotia pharmacare population of approximately 6% in 2001,9 the DEANS

Management Committee undertook to develop the AD intervention on OA Management.

Problem Statement

The effect of AD on clinical and economic outcomes is of great interest to the Nova

Scotia government's policy makers as funding for interventions to improve the health

care system is scarce. This research addresses the question of whether the AD program

on OA is effective in lowering the utilization rate of COX-2 inhibitors. At the same time

the study measures the effects that the program has on the utilization rates of other

healthcare resources such as hospital or physician visits that occur as a result of GI side









effects associated with drug therapy with traditional NSAIDs and COX-2 inhibitors. The

efficacy of traditional NSAIDs and COX-2 inhibitors in relieving pain is similar but the

GI side effects profile for traditional NSAIDs is higher.10 It is expected that the

intervention could increase the utilization rates of gastro-protective agents (particularly

misoprostol and proton pump inhibitors (PPIs)) but it is not expected to increase other

health care utilization rates and will therefore not have negative impacts on the outcomes

of care.

The methodological challenge for the evaluation of the OA AD intervention is the

need to significantly adjust for selection bias that is likely present since GPs can choose

to participate and those that do participate might be different from those that do not

participate. Statistical adjustment through regression on the propensity score (PS)

methods have been shown to be effective in reducing between group biases on many

confounding variables.11, 12 The use of PSs in studies that examine the unit of analysis

other than the patient is uncommon in the medical literature. In this study the unit of

measure was the GP. No other studies with the GP as the unit of measure were found in

the medical literature so the evaluation of different PS method's ability to adjust for bias

between GP groups was warranted.

Research Questions and Hypotheses

The term statistically significant is defined as results where the type I error (alpha)

is less than 0.05. The results are statistically significant if the analysis yields p-values

less than 0.05.

Hypotheses relating to research questions one to three are examining the effect of

the OA AD intervention in the Nova Scotia residents who are greater than 65 years old









and have a GP who has participated in the intervention as compared with GPs in the

province who did not participate in the intervention.

The first research question examined the expectation that GPs will consider the

information provided in the OA AD intervention and choose not to prescribe COX-2

inhibitors for their elderly patients.

Research Question 1

Do the patients of GPs who have undertaken the OA AD intervention have

significantly lower COX-2 inhibitor utilization rates after the GP has undergone the AD

intervention as compared to a GP control cohort? (Are there significant between group

differences?)

Research Question 1 Hypothesis

The null hypothesis is that the OA AD intervention will have no effect on the

utilization rate of COX-2 inhibitors.

The alternative hypothesis is that the OA AD intervention will have the effect of

decreasing the utilization rate of COX-2 inhibitors.

The second research question examined the sustainability of the intervention (if

research question 1 hypothesis is found to be significant) since a shortcoming of the OA

AD intervention (appendix a) is the lack of a follow-up visit to GPs who participated in

the intervention.13

Research Question 2

Does the decreased utilization rate of COX-2 inhibitors for patients of GPs who

have taken the AD intervention remain significant for a period of one-year post

intervention? (Is the intervention effect sustainable?)









Research Question 2 hypothesis

The null hypothesis is that the OA AD intervention will not have a sustained effect

on the decreased utilization rate of COX-2 inhibitors.

The alternative hypothesis is that the OA AD intervention will have a sustained

effect on decreasing the utilization rate of COX-2 inhibitors.

The third research question examined whether patients of GPs in the intervention

group experienced a change in the rate of medical services utilization due to a change in

GI adverse events associated with traditional NSAID therapy (if there was a significant

finding to research hypothesis 1). The hypothesis is divided into two categories: those

that are related to pharmacotherapy and those that involve other medical services.

Research Question 3

Do patients of GPs who have undertaken the OA AD program have medical

utilization rates associated with their OA that are significantly different from patients of

GPs who have not participated in the intervention?

Research Question 3 hypotheses

The null hypothesis is that the OA AD intervention will have no effect on the

utilization rate of (1) PPIs, (2) H2As, (3) misoprostol (4) GP office visits, (5) specialist

office visits, and (6) death rates.

The alternative hypothesis is that the OA AD intervention will have the effect of

changing the utilization rate of(1) PPIs, (2) H2As, (3) misoprostol (4) GP office visits,

(5) specialist office visits, and (6) death rate.

The fourth research question examined whether one PS adjustment method was

more successful adjusting for bias between groups based on measured bias reduction for









covariates that were not balanced after group assignment and the resulting sample size

after PS methods were applied.

Research Question 4

Is there a superior PS method for the reduction of selection bias between the

intervention and control groups?

Research Question 4 hypotheses

The null hypothesis is that there will be no difference in the three PS method's

(quintile stratification, regression on the PS, and "greedy matching") ability to adjust for

bias on unbalanced covariates.

The alternative hypothesis is that one PS method will adjust for bias on unbalanced

covariates to a greater extent than the other two.

Significance of Research

This research is of significance to several groups within the healthcare system. The

three groups that benefit directly from the research are patients, physicians, and health

policy decision-makers. The results also add to the academic research in the area of

effective behavioral change methodology and it adds to the methodology and

understanding surrounding the use of PSs.

The largest impact of this research is in the area of health policy decision making.

The decision to proceed with one course of action is often at the expense of others. This

study will inform decision makers regarding the effectiveness of the OA AD intervention

and allow them to make a more informed decision to continue with the AD detailing

program to educate physicians on other health related topics or disease states.

This research adds to the validity of the research that has been accumulated in the

area ofAD. This is significant as it was concluded by Davis et al. in a systematic









literature review of AD that while AD is effective it is seldom used by providers of

continuing medical education.14 The uniqueness of this research lies in its analysis of a

population based continuing AD program and not one that has been developed for the

purposes of a single study.

This research advances PS methodology. It compares three PS methods in a real

world and population based intervention. The results should contribute to the choice of

PS methods employed by future researchers. The study also analyzes each of the

propensity score method's ability to balance the control and intervention groups on

unmeasured administrative variables. The ability of the propensity score methodology to

balance groups on measured variables has been widely reported; however the ability of

the methodology to balance unmeasured variables is assumed 15, 16 and studies attempting

to measure the ability of the PS method to balance physician groups on a number of

unmeasured administrative variables were not found in the literature.














CHAPTER 2
LITERATURE REVIEW

Literature reviews were conducted on two areas of interest: articles dealing with

studies relating to AD interventions which have not shown statistical significance and

articles relating to the use of PS methods. The AD studies which reported no statistically

significant effects of AD interventions are of interest because they possibly give

examples of shortcomings of methodology that may be of use in this study. The PS

articles that are of interest to our study are those which involved studies that identified

some unit, other than the patient, as the unit of analysis in the PS development and

articles that dealt with PS methods.

The positive effect of AD on prescribing behavior has been summarized in a

number of review articles on AD or educational outreach.14, 17,18 This body of evidence

shows that AD moderately improves physician behavior and patient outcomes. Three

review articles are summarized.

Review Articles Addressing Effects of Academic Detailing

Davis et al.14 reviewed 99 studies which met their inclusion criteria from a total of

more than 6000 articles. The 99 studies included 160 separate continuing education

interventions, including academic detailing. Sixty-two percent of the interventions

showed improvement in at least one major outcome with effect sizes ranging from small

to moderate (quantified effect sizes not provided). There were fourteen AD interventions

in the category of prescribing and 75% of these showed positive effects. AD was

reported as an effective change agent for prescribing. The authors concluded that AD is









an effective strategy for continuing medical education (CME) however, it is not widely

used by CME providers.

Thomson O'Brien et al.18 conducted a systematic review of the effect of

educational outreach on professional practice and health care outcomes. Eighteen studies

were included in the review with thirteen of the studies targeting prescribing practices.

Nine of the thirteen studies employed multifaceted interventions (educational outreach

combined with reminders, audit and feedback, marketing, or patient-mediated

interventions). Seven of the nine studies using multifaceted interventions showed

statistically significant effects with relative effects ranging from 1 to 45% improvement

(table 2-1). The authors noted that potential bias exists in thirteen of the eighteen studies

due to lack of randomization and six of the studies contained potentially important

baseline differences and adjustment for these differences was not carried out in the

statistical analysis. It was also noted that only one of the eighteen studies considered

patient outcomes. The authors concluded that the effects of educational outreach are

small to moderate but potentially of practical importance.

Grimshaw et al.17 conducted a systematic review of the effectiveness and costs of

different guideline development, dissemination, and implementation strategies. 235

studies representing 309 comparisons were included in the review. The sections of the

review that are germane to our study are the multifaceted comparisons involving

academic outreach with continuous measures for process or outcome variables. Ten

comparisons were reviewed which contained measures on continuous variables. Six of

the comparisons involved process measures (five cluster randomized control trials and

one controlled before and after trial) and all reported improvements in performance with









a median effect of 15.0% (range 1.7% to 24.0%) relative improvement. None of the

studies included enough information to calculate standardized mean difference, and two

studies were not statistically significant. Four of the comparisons involved outcome

measures (three cluster randomized control trials and one controlled before and after

trial). The median effect of the cluster randomized control trials was 0% (range -1.4 to

2.7%) and the standardized mean difference was calculated as 0 for one trail. The

controlled before and after trial reported a relative improvement of 13.9% with a

standardized mean difference of 2.38. The authors summarized the multifaceted

interventions, including academic outreach, to be at best moderately effective (table 2-1).

Table 2-1. Summary of Included Studies for Thomson O'Brien and Grimshaw.
Author (year) Reviewed by Interventions (plus AD) Relative
Effect (%)
McConnell Thomson O'Brien Audit and Feedback (AF), 45.8
(1982) Educational Material (EMat)
Stergachis Thomson O'Brien AF, Patient Mediated (PM), 35.7
(1987)* Conferences
Meador (1997) Grimshaw EMat, Educational Meeting 24.0
(EMeet)
Ross-Degnan Thomson O'Brien EMat, Social Marketing 21.0
(1996) (SM), PM
Peterson (1996) Grimshaw EMat 20.0
Avorn (1983) Thomson O'Brien EMat, SM 15.2
Avorn (1992) Thomson O'Brien, EMat, SM, Conferences 15.0
Grimshaw
Ray (1993) Grimshaw EMat, EMeet 13.9
de Burgh Thomson O'Brien EMat, PM 13.0
(1995)*
Diwan (1995) Grimshaw EMat 11.3
Steele (1989) Thomson O'Brien Reminders 11.2
Santoso (1996) Thomson O'Brien EMat, SM 8.7
Schmidt (1998) Grimshaw Organizational Change 5.5
Elliott (1997) Grimshaw EMat, Opinion Leaders 2.7
Feder (1995) Grimshaw AF 0.0
Moore (1997) Grimshaw EMat, Reminders, PM -1.4
non-significant study results









Academic Detailing Studies Reporting No Statistically Significant Effect

Five articles were reviewed in which the authors reported non-significant results for

AD interventions with pharmacotherapeutic outcomes. The review of results that were

not positive is important because it will possibly indicate to investigators methodological

similarities that may have been employed in previous unsuccessful studies. If identified

the methodological shortcomings could be avoided.

Lin et al.19 studied the effects of physician training on the management of

depression. The study was a before and after design with an equivalent control group.

The physician sample was made up of 109 primary care physician volunteers and they

were associated with fifteen primary clinics. Randomization of groups was at the clinic

level resulting in 56 physicians in the intervention group and 53 physicians in the control

group. The intervention was outlined including the four key messages and the use of

opinion leaders in intervention delivery.20 Case managers were used for follow-up visits

with the physicians. The intervention involved other components such as small group

discussions, role-play and psychiatric consults. The authors reported that the physicians

in the intervention arm of the trial did not differ significantly from the control group in

adequacy of pharmacotherapy (p=0.53). While insignificant, the results showed a

decrease of 7.5% in the percent of patients in the intervention group who received

adequate pharmacotherapy with no change in the control group. The decrease in the

intervention group is opposite to the desired outcome of the intervention. The study also

failed to show significant differences in the number of antidepressant prescriptions per

100 patients (p=0.10). The percent of patients receiving new prescriptions in the

intervention group decreased by 10.4% and increased in the control group by 4.8%.

These results are opposite to the desired outcome of the intervention. The authors









reported that the study's main failure was its lack of power to detect a significant change

between groups. The sample size used was sufficient to detect a 40% to 50% difference

in adequate pharmacotherapy and a 15% to 30% difference in new antidepressant

prescriptions. The fact that the effect of the intervention was the opposite of the

hypothesis was not explained by the authors.

Brown et al.21 studied the effect of AD and continuous quality improvement (CQI)

interventions on the treatment of patients for depression. The study was a randomized

controlled trial. The primary care clinician groups were randomized by first matching

clinicians according to specialty (internal medicine or family practice), sex, training

(physician or allied health clinician), and number of patients in a high-risk depressive

cohort. The resulting sample size was 160 with 79 in the intervention arm and 81 in the

control arm. The AD intervention involved focus groups for the collection of baseline

knowledge of primary care providers (physicians, physicians' assistants, and nurse

practitioners) in preparing the intervention. The intervention was based on guidelines

from the Agency for Health Care Policy and Research and used the same material as the

Goldberg study.22 Three main messages were summarized on letter sized illustrated

handouts. Four visits were used to deliver the message and the detailers were

pharmacists from the clinicians' own medical office. The study showed mixed results. It

was successful in increasing the percent of patients receiving antidepressant treatment

(7.5% increase, p=0.046 in depressed arm and 0.7% increase, p=0.025 in the non-

depressed population) however, it was not successful in increasing the total days of

antidepressant therapy (16.7 days effect, p=0.189 in the depressed arm and 1.3 days

effect, p=0.606 in the non-depressed population). The study did not exhibit significant









differences in non-pharmacotherapeutic outcomes (improvement of symptoms and

measures of functional status). The authors report that the mixed findings could be due to

the complexity of the implementation of a clinical guideline and the evidence base for the

guidelines may not be generalizable to the study population. They propose that AD may

be appropriate for behavioral change but is not sufficient for the implementation of

clinical guidelines. This conclusion is important for our study since the primary outcome

is change in prescribing behavior.

Goldberg et al.22 studied the effect of AD and CQI interventions on compliance

with guidelines for hypertension and depression. The study was a randomized before and

after design with two experimental groups (AD only and AD combined with CQI) and an

equivalent control group. The physicians were part of fifteen clinics and group

randomization was carried out at the clinic level. The resulting sample size was 78 with

18, 37 and 23 physicians in the AD only, AD combined with CQI and usual care groups

respectively. The AD intervention was based on national guidelines for hypertension and

depression from the Agency for Health Care Policy and Research. Five

recommendations were developed including two which specifically addressed

pharmacotherapy. The AD intervention was delivered by opinion leader physicians and

follow-up visits were conducted by staff pharmacists. The intervention was supported by

handouts and pocket cards for quick reference. The study found significant effect in only

one of the pharmacotherapeutic outcomes which was a decrease in the prescribing of 1st

generation antidepressants to previously diagnosed depressed patients (relative effect -

4.7%, p=0.04). The other outcomes prescribing of hypertension medications,

antidepressants to previously undiagnosed patients, 2nd generation antidepressants to









previously diagnosed depressed patients, and SSRIs to previously diagnosed depressed

patients exhibited insignificant change with relative effect sizes and p-values of 1.3%,

p=0.06; 2.4%, p=0.68; -2.1%, p=0.43; and 3.3%, p=0.11 respectively. One possible

explanation for the failure of the study to show significant effect for all but one of the

pharmacotherapeutic outcomes can be attributed to the presentation of too much

information. A successful AD intervention should include only a limited number of

messages regarding a disease state.13 The presentation of an intervention covering two

distinctly different disease states clearly violates this principle.

Zwar et al.23 studied the effect of AD on prescribing rates of benzodiazepines for

all indications. The study was a before and after design with an equivalent control group.

There were 157 physicians who participated in the study. They were randomized into the

benzodiazepine AD group (n=79) and the control group who received AD on another

topic (n=78). The AD intervention was based on guidelines developed by the Royal

Australian College of General Practitioners and it was delivered by physicians trained in

AD techniques. The intervention was not accompanied by any other methods (i.e.

handouts, etc.). The study found significant effect in overall prescribing of

benzodiazepines (-26.7%, p=0.042) however, there was no significant between group

relative effect (-1.2%, p=0.99). The authors attributed the lack of significant results to

the effects of a pre-intervention practice survey that was given to all physicians in the

study and a lack of power to detect a difference between groups due to the decision to

aggregate data into eight subgroups thereby reducing the sample size dramatically.

Tomson et al.24 studied the effects of AD on physicians' practice in the

management of asthma and on patient knowledge. The study was not randomized and









the sample consisted of 63 GPs in two regions, one region was assigned as the treatment

group (n=44) and the other region was assigned as the control group (n=19). The

intervention was developed using existing physician knowledge as the baseline and the

input of respirologists. It was delivered by a clinical pharmacologist and a pharmacist

and contained three main messages. The face-to-face visits with physicians were

augmented with written materials. The study found that there was not a significant

difference between the treatment and control groups in prescribing ratios of beta-agonists

and inhaled corticosteroids (no p-value reported). One explanation for the insignificant

results could be attributed, at least in part, to insufficient power (due to the small sample

size) to detect a meaningful change. The authors identified a possible selection bias in

the physicians volunteering for the intervention as they may have been largely physician

interested in asthma therapy to begin with.

It is important to note that a review of the negative findings of studies in the

literature cannot be considered to be complete since many studies, and their fatal flaws,

are not published if they are not considered to be methodologically sound or clinically

important (publication bias). However, from the review of literature that did not report

significant results there are four areas of inadequacy that the studies appear to have in

common;

* the authors reported that there were insufficient sample sizes to yield enough power
to show a meaningful change in the studies conducted by Lin19, Gorins25, Zwar23
and Tomson24, however in all but one of the studies19 the effect of the intervention
was consistent with the study hypothesis. It is important to note that lack of power
is only one explanation for the lack of study significance,

* there were intervention development problems in that the interventions were too
complex21 22

* the interventions may have been compromised through the use of less than credible
academic detailers19' 25, and









* the use of pre-tests or pre-intervention surveys decreased the intervention effect due
to a pre-sensitization of the subjects to the intervention.23 25

The results from the above studies are applicable to our study for the following

reasons. The lack of power reported by a number of the studies is only one explanation.

Other explanations could include a large variation in measurement on the dependent

variable or a lack of control for the variables that are associated with the outcome

variable. For example, in our study we could have a large sample but if the number of

elderly patients in the GP's panel is not controlled for then the variation could be inflated

and a non-significant result could occur. In our non-randomized study design it is

important to adjust for variables which are associated with the outcome but it must be

acknowledged that there will be variables which are important confounders and are not

measured so residual confounding (bias) will exist. There may be a need to adjust for

patient variables as well. For example, if the GP's patient panel is markedly ill then this

will confound the results. A measure of patient wellness would help to address this

problem. The lack of a follow-up visit to GPs in our study may play an important role in

the outcome.

Propensity Scores

A literature search was conducted using PubMed for all years up to and including

July 20, 2005. The search terms used were "propensity score" and "propensity scores".

The search yielded 341 articles. The abstracts for all 341 articles were reviewed and the

distribution of articles by article objective and year is illustrated in figure 2-1.

The distribution shows an initial surge of articles dealing with PS methods in the

late 1990's with articles containing objectives other than medical (e.g., economic) and

only a few articles with stated medical objectives. Since 2000 there has been a surge in









published articles using PSs particularly in the field of cardiology. The increase in use of

PSs has been mirrored by an increase in published articles dealing with PS methods.

There were four articles which used a unit of analysis for PS other than the patient.

Two of the articles used human couples as the unit of analysis one article developed PS

on hospitals and one article used communities as the unit of analysis for the PS. There

were no articles found which used the physician as the unit of analysis in the PS analysis.

There were 50 articles that described PS methods and 24 of these were selected for

further review. Criteria for selection included PS studies using GEE for outcome models,

studies comparing small experimental groups, studies describing PS and sample sizes or

studies which described PS methods in detail. The information gained from these articles

plus reference material from previous course work, library searches, and colleagues

formed the basis for the PS method as it has been applied in this study.

In our study the PS represents the probability of a physician volunteering for the

OA AD intervention given a number of personal and practice characteristics. For studies

using quasi-experimental designs it is important to include methods to compensate for the

lack of randomization to experimental groups. In our study we have made multiple

measures of outcome variables both before and after the intervention and we have

included a control group for comparison. The control group is not equivalent to the

intervention group so adjustment on PSs was used to reduce the effect of the between

group bias.

Three methods for applying PS in observational studies are predominant in the

literature.26 The three methods are; sub classification on the PS11 12,27, regression on the

PS12 and matching on the propensity score using Mahalanobis metric matching12' 27 or










"greedy matching" techniques.28 All three of the methods; stratification, regression on

the PS, and matching have been applied successfully in observational studies and

therefore all three will be considered for application in this research.


Distribution of PS Article Objectives


S40
Jl

?0


20 # of Articles


10
5
0


E non-medical
o methods
* medical
* cardiology


Figure 2-1. Distribution of Propensity Score Article Objectives: 1987 to July 20, 2005

An overarching limitation of all three PS methods is that the PS can only adjust for

bias in observed covariates27 and the extent to which the bias is abated in unobserved

covariates depends on the correlation of the unobserved covariate with one that is

observed.11 Shadish stated that if the PS method was successful in abating bias in the

measured covariates then the assumption can be made that the methodology would be

successful in decreasing the bias in unmeasured covariates as well.16


2 %%t 1
2001
Year 2000
1999
1998
87-97 Lo
S \0


Objective









A recent study has tested the ability of PS based on covariates extracted from

administrative data to reduce the bias in unmeasured clinical variables. In this study the

clinical data was extracted from patients' charts after PS methods were applied. The

experimental groups set by the PS method were tested for significant difference on the

clinical variables and it was found that the clinical variables were not balanced between

the groups.29

Other studies have explored the number of events per variable that are needed for

logistic regression analysis to outperform the PS method. Cepeda et al. reported that in

their simulation model if there are six or fewer events per independent variable

(covariates in the PS model) then the PS estimates are less biased then the regression

estimates.3 It is important to note that even if the number of subjects exceeds six the use

of PS methods is warranted since it is a variable which predicts the exposure of interest

without including the outcome11 and that the use of PS methods is intended to

complement model-based procedures not replace them.31

There are two measures of PS model fit that are reported in studies. The c-statistic

is the area under the receiver operating characteristics (ROC) curve and is a measure of

the discriminative ability of the PS model.32' 33 The range of the statistic is from 0.5 to

1.0. If a model has a c-statistic of 0.80 this can be interpreted as the model accurately

assigning random pairs of subjects to their experimental groups based on PS alone 80%

of the time. The c-statistic is intended to be an indicator in the model building process

but it is not a measure of the PS model's ability to adjust for bias15 and it has not been

found to be associated with the ability of a PS model to reduce residual confounding.32

The goodness of fit is another statistic that is commonly used in regression analysis. Like






21


the c-statistic these tests were not found to be useful in predicting the ability of the PS

model to reduce residual confounding.32 As a result, these measures were not used in our

study to decide which PS method to use for the outcomes analyses. The c-statistic was

however, used to explain the effects on the model's discriminatory ability when variables

were intentionally removed from the PS Regression model.














CHAPTER 3
METHODS

This study is a retrospective cohort, before and after longitudinal design with a

non-equivalent control group using the Nova Scotia Medical Services Insurance and the

Canadian Institutes of Health Information datasets for analysis. The non-equivalent

control group design requires the use of procedures to abate selection bias in the

treatment group.12

The methodology for the study can be broken down into four distinct sections,

which are as follows;

* the extraction and validation of data from the administrative databases,

* the establishment of balanced control and experimental groups using three distinct
PS methods,

* the primary outcome analysis of the intervention effect on the utilization rate of
COX-2 inhibitors and,

* the secondary outcome analysis of the intervention effects on the utilization of
PPIs, misoprostol, and H2As.

Step One: Extraction and Validation of Data

Sources of Data

All of the data used in this study was collected in pre-existing administrative

databases. There were no occurrences of missing data since the variables included in the

analysis were extracted from long standing registrar data which is complete for all fields

listed in the registry1 (GP demographics), complete census information2 (geographic

data) or the data was reported in terms of rates with the GP inclusion criteria ensuring









that each GP panel contained at least twenty patients so the rates for the outcomes

measures were always defined (i.e. rate denominators were not equal to zero).

Administrative data must be used with caution as it is not 100% reliable. Chapter five

outlines the limitations of the administration used in our study.

GP demographic data for all GPs in the province was obtained from the Nova

Scotia College of Physicians and Surgeons Physician Registry (2002).1 The Dalhousie

CME Division provided data which contained demographic information of the GPs who

were detailed and the dates when the detailing visits were carried out. These two sources

of data were merged and the resulting file was submitted for encryption using the same

encryption methods as the provincial administrative data. The resulting encrypted GP

demographic profiles were augmented with data from the Nova Scotia Medical Services

Insurance (MSI) physician registry (2002) to include dates indicating when the GPs opted

in and opted out of the provincial pharmacare billing scheme. GP practice information

such as population of the community and average income of the county in which the

practice is located was added to the demographic profile of each GP.2

Patient data was extracted from the Nova Scotia Pharmacare Seniors Dataset

(2002-2004) and the hospital discharge data found in the Canadian Institute of Health

Information (CIHI) hospital discharge dataset (2002-2003). Patient level GP visit data

was used to determine to which GP's patient panel a patient belonged (see patient

inclusion criteria). Once the patients were assigned to GP panels the patient prescription

claims data and hospital length of stay data were aggregated at the GP level. Drug

utilization variables were created at the GP level with the unit of measure equal to DDDs

per elderly patient per 90 day study period. Change in utilization rate variables were









created for each GP by subtracting each period utilization rate (period = 1 to 6) from the

baseline (period two) utilization rate. Period two was chosen as the baseline utilization

rate since it was the pre-intervention measure most proximal to the GP index date.

Descriptive statistics for the GP demographic variables were calculated to confirm

that the variables did not contain any missing data and to confirm that the variables fell

with acceptable ranges (i.e. no GPs 200 years old, not all male GPs). The descriptive

statistics are reported in tables 4-1 and 4-2.

Prescription claims, GP visits and vital statistics were checked to ensure that there

were not instances of missing data. The prescription claims and GP visit data were

complete on all fields necessary for our study. Only hospital admissions and deaths due

to GI events were included in the hospital length of stay and death measures. A detailed

description of the inclusion criteria for data is contained in chapter four. The underlying

and primary causes of death were used to determine death rates and cause of death and

data was reported for all included patients who died over the study period. The first four

diagnoses codes for hospital admission were used to determine if GI complications were

associated with admission. In all cases there was at least a primary diagnoses on

admission. While the data for our study was complete it was administrative data and

there are shortcomings associated with it. The limitations of administrative data are

described in chapter five.

Data from several administrative databases was linked to create the datafile

necessary for the PS analysis and for the outcomes analysis. The data linkage was carried

out using the encrypted physician identifiers and the encrypted patient identifiers. The









encryption of the patient and physician identifiers was carried out according to standards

set by the Canadian Institutes of Health Information (CIHI).34

GP Inclusion Criteria

The academic detailing intervention was targeted at GPs and, therefore, the

experimental unit is the GP and the patient data for the GP's practice is the unit of

measure. Each GP's practice is measured as an aggregate of the individual patient's data

from his or her practice. The aggregation of patient data is described in greater detail

later in this chapter. The date on which the GP received the OA AD intervention was

defined as the index date. For GPs in the control group the index-date was randomly

assigned from the time period over which the AD intervention took place.

There are four criteria that a GP had to meet to be included in the study. They are

as follows;

* The GP had to be registered as a GP with the Nova Scotia College of Physicians
and Surgeons for the entire study period.

* The GP had to be included on the billing registry with the Nova Scotia Medical
Services Insurance (MSI) (the provincial government payment agency for seniors'
medical and pharmacy claims) for the entire length of the study. This registry is
the source of the medical and pharmacy claims data that will be used for the
outcomes analysis.

* The GP had to have an elderly patient panel equal to or greater than twenty
patients. The rational for the cut score of twenty was based on the premise that a
5% decrease in COX-2 utilization (i.e. COX-2 utilization rate change from 6.0% to
5.7%) will equate to annual savings to the elderly population of approximately
$100,000. Therefore, if the GP had an elderly patient panel equal to twenty he or
she was required to change prescribing behavior for one patient over the study
period to realize a 5% change.

* The GP had to have at least one prescription claim for a COX-2 inhibitor recorded
in the pre-intervention period (6 months preceding the GP's index date).









Patient Inclusion Criteria

The patient is the unit of measure for this study. Patients had to meet two criteria

for inclusion in the study. The criteria are:

* The patient had to be included on a GP's patient panel. For inclusion on a GPs
panel the patient must have seen a specific GP for more that 50% of his or her total
GP visits for the fiscal year ending March 31, 2002. For example, if a patient had a
total of forty GP visits in the period from April 1, 2001 to March 31,2002 and
twenty-four (60%) of the visits were billed by one GP the patient was included on
the GPs patient panel. Once the patient was assigned to a particular GP they
remained with that GP throughout the study.

* The patient had to be 66 years of age or older as of the GP's AD index date. This
ensures that the patient was at least 65 years old and eligible for the MSI
pharmacare coverage for the entire study period and it provides a period of time of
at least six months for the patient to become accustomed to the new MSI
pharmacare coverage.

Step Two: Adjustment for Confounding Using Three Distinct Propensity Score
Methods

The definition of the PS is the conditional probability of treatment given the

individual's covariates. In this case it would be the conditional probability of taking the

OA AD intervention given the GP's personal and practice characteristics.

The PS is obtained by fitting the data using a logistic regression model.5 Once the

PSs were calculated for each GP three PS methods were applied to the PS data and the

optimal method in term of bias reduction and resultant sample size was determined.

The three PS methods used in this study were; the stratification into quintiles,

regression on the propensity score12, and "greedy matching" or one-to-one matching for

group assignment.28

The variables in the regression model describe the GPs' personal characteristics

(age, sex, birthplace, etc.) and practice characteristics (size of patient panel, population of

community in which the practice is located, etc.). All variables in the data that fit within









these two descriptive categories were included in the regression model. This approach is

consistent with the literature which calls for the inclusion of all variables which have

some relevance to the outcome variable.16 A description of the included variables and the

abbreviations used in our study are included in table 3-1.

Table 3-1. PS Model Variable Descriptions and Abbreviations
Variable Description # Levels Abbreviation
GP participation in the OA 2 (Y/N) OA
ADintervention
GP participation in previous 2 (Y/N) flu AD
influenza AD service
GP's sex 2 (M/F) sex
GP's birthplace 3 (Nova Scotia, birth place
Canada, Other)
GP's location of initial licensure 5 (Nova Scotia, license
Canada East,
Canada Center,
Canada West,
Other)
GP's COX-2 utilization rate at continuous BL rate
baseline (DDDs / patient)
GP's age (years) continuous GP age
population size of community in continuous population
which GP's practice is located
average income of county in which continuous aver income
GP's practice is located ($cdn)
number of patients in the GP's continuous total # pt
practice
percent of GP's patients diagnosed continuous % OA dx
with OA (ICD-9 CM = 715)
percent of GP's patients > 65 years continuous % elderly
old
average hospital length of stay for continuous los rate
elderly patients in the GP's practice
(days/patient)

A logistic regression model was used to accommodate the dichotomous nature of

the outcome variable, OA. The same regression model was applied using PROC REG

(SAS 8.2)35 for all three methods to determine GP PSs. Models described in this study









have categorical variables listed as single entities which are consistent with the SAS

coding techniques. The model analysis creates (t-1) dummy variables (where t = the

number of levels) for each categorical variable. The PS regression model is shown in

figure 3-1.

Y = a + P1X1 + 02X2 + 03X3 + P4X4 + 5X5 + P6X6 + 7X7
+ 8sXs + 89X9 + P1oXio + +11X11 + P12X12
Where;
Y GP participation in the intervention (0 = no, 1 = yes),
Xi GP participation in previous influenza AD service (0 = no, 1 = yes),
X2 GP's sex (Male, Female)',
X3 GP's birthplace (Nova Scotia, Canada, Other)1,
X4 GP's location of initial licensure (Nova Scotia, Canada East, Canada
Center, Canada West, Other)',
X5 the GP's COX-2 utilization rate at baseline (DDDs / patient).
X6 GP's age (years)1,
X7 population size of community in which GP's practice is located2,
Xs average income of county in which GP's practice is located ($cdn)2,
X9 number of patients in the GP's practice,
Xi1 percent of GP's patients diagnosed with OA (ICD-9 CM = 715),
Xii percent of GP's patients > 65 years old,
Xi2 average hospital length of stay for elderly patients in the GP's
practice (days / patient).


Figure 3-1. Propensity Score Logistic Regression Model

Variables were kept in the model regardless of their significance. Variables that are

not statistically significant still contribute to the model and the population based nature of

the data ensures a large enough sample size to support the model with twelve predictor

variables. The final model predicts the probability that each GP would receive the

intervention based on his or her individual variables. This probability is the GP's PS.

Once the PSs were calculated they were applied according to the three methods

stated earlier.









Quintile Propensity Score Method

For the quintile method; the GPs in the treatment and intervention groups were

stratified, based on their participation in the OA AD intervention, and then ordered

according to the GP's PS. The treatment and control groups were stratified into five

levels, or quintiles. Each quintile contains 20% of the GPs (table 4-4).

Regression on the Propensity Score Method

For the regression on the PS method; the PS was used in the outcome model.

"Greedy Matching" Method

For the "greedy matching" method; the GPs in the treatment and intervention

groups were stratified, based on their participation in the OA AD intervention, and then

ordered according to the GP's PS. A matching procedure was applied28 that involved

matching the groups on PS beginning with matches accurate to five decimal places and

concluding with matches to one decimal place. The number of included GP only allowed

for a one-to-one match between groups. Once matched the GP was removed from the

sample pool. Those GPs that were not matched were deleted. The "greedy matching"

method resulted in group sizes of 104 each (N = 208 total).

Propensity Score Method Selection

The regression on the PS method was selected for use in the outcomes analysis.

The regression on the PS method was selected based on the following criteria; the

adjustment for selection bias on the covariates measured before and after the PS

procedure is carried out, and the resultant sample size.

The adjustment for selection bias after application of PSs was determined for each

PS method using the following methods.









For continuous variables the percent decrease in bias was calculated using the

formula:11 100 x [ 1 (bias post) / (bias pre) ], where bias post was the difference

between PS adjusted group means and bias pre was the difference between unadjusted

group means (group means before PS analysis). Variable means before PS analysis are

reported in table 4-3 as the unadjusted means of the groups. Variable means after PS

adjustment for the regression on PS and quintile methods were the least square means

reported using PROC GENMOD (SAS 8.2)35 after adjustment for propensity score (or

quintile depending on the method). For the "greedy matching" method unadjusted means

were used for both the pre and post means calculations. Results are reported in tables 4-

5, 4-7, and 4-8.

For categorical variables the percent decrease in bias was calculated using the

following formula:12 100 x [ 1 |(1- OR post) / (1- OR pre)|] where OR post is the odds

ratio of the groups (adjusted for PS) and OR pre is the odds ratio of the groups before PS

adjustment. For both the pre and post odds ratio measures PROC GENMOD (SAS 8.2)35

was used. The odds ratios were calculated using the same procedure for all three PS

methods. Results are reported in tables 4-5, 4-7, and 4-8.

A further test of the effect of the different PS methods involved the purposeful

removal of independent variables from the regression model and the subsequent test for

PS adjustment on the "unmeasured" variable. The logistic regression model was run

twelve times. Each time one of the independent variables was removed from the model

and the percent bias reduction on the now "unmeasured" variable was calculated for each

of the three PS methods. The same equations for continuous and categorical variables









were used to calculate percent bias reduction on the variable that had been removed. The

results are reported in tables 4-9 to 4-11.

The measurement of adjustment for bias on "unmeasured" variables was not

considered in the selection of the PS method. It has been included in this study as a

means of contributing to the PS methodology. Work has been done on the PS model's

ability to adjust for bias on unmeasured clinical variables29 and the PS model's ability to

adjust for bias on unmeasured variables in a large computer generated dataset.32 Our

study is unique, however, since it examines the PS model's ability to adjust for bias on

demographic variables contained in a relatively small, real world dataset.

There were five PS model covariates that showed significant between group

differences after the initial OA group assignment. The variables were percent of patients

diagnosed with OA (% OA dx), the average income of the county in which the GP's

practice is located (aver income) the average hospital length of stay per patient (los rate),

the population size of the community in which the GP's practice was located and

participation in a previous influenza AD intervention (flu AD). The PS adjustment on the

flu AD variable was not successful for any of the three PS methods so it was included in

the outcomes models as a covariate. The other four variables were of interest in the

analysis of effect of PS method's ability to adjust for bias on unmeasured administrative

variables. The correlations between the variable and the PS were calculated and graphed

against the percent reduction in bias for each PS method. Correlations between the

variables and the included PS model covariates were calculated and tabulated. The

relationship between the reduction in bias in unmeasured variables and each PS methods

was studied. The results are contained in chapter four.









Step Three: Primary Outcome Analysis; Intervention Effect on COX-2 Utilization
Rates

Once the method of PS analysis was selected and the GP intervention and control

groups had been determined, the analysis of the primary outcome effect was carried out

as described below.

To enable the analysis of the changes in COX-2 utilization rates over time the

COX-2 utilization rates were determined for each GP in the study for six consecutive

ninety-day time periods. Two time periods were pre-intervention and four time periods

were post-intervention.(Figure 3-2) The index-date is reported as the date that the GP

received the AD intervention and the index-dates for the control group were assigned by

randomly selecting dates from the range of time that the AD intervention spanned.

The COX-2 utilization change rate will be calculated by subtracting the GP's

baseline utilization rate (period 2 utilization rate) from the utilization rates in each study

period.


Intervention Group O O X O O O O

Control Group O O O O O O

Time from Index
Timefrom -180 to -91 -90 to-1 Index to 90 91 to 180 181 to 270 271 to 360
intervention (days) date


Figure 3-2. Experimental Design Timeline

Before the utilization rates could be calculated the inclusion and exclusion criteria

for claims in a given time period were defined. An example of the operationalization of

the decision rules for the inclusion or exclusion of claims within a time period is

presented using a fictitious ninety day time period (January 1st until March 31st) and

describing how different scenarios were adjudicated. If a prescription claim is submitted









for a two-month supply on January 2nd it is clear that the period of time for the entire

claim falls within the given time period and the claim is included. If a prescription claim

is submitted for the same two-month supply on March 28th it is clear that the entire claim

period does not fall within the period ending March 31st. In this case the claim would

still be counted, in its entirety, in the claim period that it was submitted. The reason for

inclusion of the claim in the initial time period is that it was in this time period that the

GP's prescribing behavior took place and the intention was to have the patient take the

medication as prescribed.

Refills were considered to be an extension of the original claim until such a time as

the refill claim was submitted more than thirty days after the intended fill date for the

refill. If the refill was more than thirty days late the rest of the claim was not counted in

any time period.

The COX-2 utilization rates for each GP was determined through the use of the

World Health Organization's (WHO) Anatomic and Therapeutic Classification System/

Defined Daily Dose (ATC/DDD) methodology36 and was reported for each GP as the

average number of DDDs per included patient per ninety-day intervention time period.

The reporting of DDDs is often given as per thousand patients, however, since most GPs

in the study will not have one thousand patients that meet the criteria this could be

misleading.

DDDs are drug consumption data that are independent of price and formulation.

Once set, the WHO is reluctant to change DDD measures and as such the DDD is stable

over time. This makes the DDD measure more reliable for drug consumption studies but

it is not appropriate for clinical analysis. The DDD, therefore, "enables the researcher to









assess trends in drug consumption and to perform comparisons between population

groups."36

An analysis of the intervention effect on the primary outcome, change in COX-2

utilization rates, was carried out. The primary outcome model initially included the

dependent variable (change in COX-2 rates for the four post-intervention periods), the

independent variables indicating the between group effect (OA) and longitudinal effects

(period), the PS variable, the variable flu AD (as indicated from the PS analysis), as well

as baseline COX-2 rate (BL rate), and number of elderly patients in the GP's panel (#

elderly pt). The model is depicted in figure 3-3. Each of the variables was retained in the

model regardless of its significance. The covariates were all included as adjustments for

confounding which if not controlled would be questioned in peer review. The included

variables for the primary outcome model with their associated coefficients and

significance levels are reported in table 4-13 in the results section.

Y = Po + Pl(Xl) + P2(X2) + P3(X3) + 4(X4) + 5(X5) + P6(X6)

Where;
Y = change in COX-2 utilization rate (periods 3 to 6 (post-intervention)),
Xi = GP participation in the intervention (0 = no, 1 = yes),
X2 = experimental time period (period = 3,4,5,6),
X3 = PS,
X4 = GP participation in the influenza AD service (0 = no, 1 = yes),
X5 = GP baseline COX-2 rate (DDD / patient / period = 2),
X6 = number of GP's patients > 65 years old,


Figure 3-3. Primary Outcome Model for Between Group Effect

The model determined the statistical significance of the between group effect

(between group effect) as well as the longitudinal effect (within subjects effect) of the

intervention. The model was analyzed using PROC GENMOD (SAS 8.2)35 and

significance is reported at the alpha= 0.05 level.









The two ninety-day pre-intervention utilization rates for the experimental groups

were analyzed to determine if there were any significant between group differences in the

change in COX-2 utilization occurring before the study commenced. The pre-

intervention analysis was carried out using the same model as the primary outcome

model described in figure 3-3 however, only the first two time period measurements

(period = 1,2) for each GP were entered into the model. This examined whether or not

significant differences for change in utilization rates were present between the groups

before the intervention was applied.

A longitudinal model was tested using the primary outcome model in figure 3-3

with a pre-intervention / post-intervention variable added which described whether the

change in COX-2 utilization rate was pre- or post-intervention. The measurements for

periods one and two were coded as prepost =1 and the measurements for periods three to

six were coded as prepost = 2. The longitudinal effects model was run twice with only

one of the intervention groups included each time. The prepost variable indicated

whether a significant within group intervention effect occurred. The results are reported

in tables 4-18 and 4-19.

Step Four: Secondary Outcome Analyses; The Utilization of Other Health Care
Resources Associated with NSAID Induced GI Side Effects

The primary outcome model exhibited significant between group differences and

therefore, all of the secondary outcome analyses were carried out using the same GP

groups as in the primary outcome analysis. The models for the secondary outcomes were

developed using the same variables as the primary outcome model. Each secondary

outcome model had the change in COX-2 utilization rate substituted with the appropriate

secondary outcome rate. The secondary outcomes that were analyzed are the intervention






36


effect on changes in rates from baseline for; PPI utilization, misoprostol utilization, H2A

utilization, GP office visits, specialist office visits, and death rates due to GI

complications. Rates for secondary outcomes (described individually with each outcome

analysis) and the results for the secondary outcomes are described and reported in chapter

four.

















CHAPTER 4
RESULTS

Step One: Extraction and Validation of Data

PROC MEANS35 was used to perform the calculations for the continuous variables.


The variable mean, standard deviation, median, minimum and maximum are reported in

table 4-1.


Table 4-1. Descriptive Statistics for Continuous Variables in the PS Model
Group N Variable Mean Std Dev Median Minimum Maximum
AD= 0 265 % OA dx 0.0913 0.1071 0.0638 0.0000 0.7391
GP age 47.30 9.78 47.00 27.00 79.00
% elderly 0.1910 0.1160 0.1676 0.0253 0.9000
total # pt 1054.18 438.55 1021.00 30.00 2575.00
aver income 27688.68 4683.37 27500.00 22500.00 32500.00
BLrate 3.6013 2.7048 3.0675 0.0769 16.8750
los rate 0.0453132 0.1449 0.0000 0.0000 1.2647
population 183342.23 162415.32 109330.00 991.00 359183.00
AD= 1 231 % OA dx 0.0719 0.0598 0.0608 0.0000 0.2601
GP age 45.74 9.18 45.00 27.00 77.00
% elderly 0.1772 0.0788 0.1681 0.0251 0.5564516
total # pt 1037.69 418.99 1009.00 171.00 2481.00
aver income 25833.33 4488.31 22500.00 22500.00 32500.00
BL rate 3.9758 2.8707 3.4456 0.0487 14.2500
los rate 0.0882 0.1767 0.0000 0.0000 1.2592
population 122705.25 155567.12 22430.00 550.00 359183.00

PROC FREQ35 was used to perform the calculations for the categorical variables.


The proportion of each variable level is reported in table 4-2.


Step Two: Establishment of Balanced Control and Experimental Groups Using
Three Propensity Score Methods

Pre-Propensity Score Analysis

Twelve variables were identified in the administrative data as describing personal


and practice characteristics of GPs. GP age, sex, participation in a previous influenza AD









intervention, place of initial licensure, baseline COX-2 prescribing rate, birthplace and

percent of patients diagnosed with OA describe personal characteristics. The percent of

elderly patients, total number of patients, average income of county where the practice is

located, population size of the community where the practice is located and average

hospital length of stay for patients describe the GP's practice characteristics.

Table 4-2. Descriptive Statistics for Categorical Variables in the PS Model.
Variable Level Proportion

AD = 0 (n=265) AD = 1 (n=231)
Sex Female 0.3057 0.2987
Male 0.6943 0.7013
Flu AD Yes 0.1585 0.7273
No 0.8415 0.2727
License Canada 0.1208 0.1732
Nova Scotia 0.6717 0.6623
Other 0.2075 0.1645
Birth place Nova Scotia 0.4830 0.4545
Canada East 0.1245 0.1255
Canada Centre 0.0679 0.0736
Canada West 0.0264 0.0390
Other 0.2981 0.3074

Table 4-3 contains the pre-PS variable values which include; means and standard

deviations for continuous variables, F-statistics (square oft-test for continuous variables

and F for coefficient estimate from PROC GENMOD35 for categorical variables), e-

values, coefficient estimates for the main effect of the intervention for the categorical

variables, and the odds ratio for the main effect. These values were used in subsequent

tables to calculate percent bias reduction for each PS technique.

The pre-PS analysis indicates that there are five variables that are not balanced.

These variables are of the greatest concern since the goal of PS methods is to balance the

groups on measured covariates. 1216 The five variables that show significant differences

at the alpha = 0.05 level will be collectively referred to as the variables of concern (VOC)










and they are; the percent of elderly patients with a diagnosis of OA (% OA dx), the

average income of the county in which the physician's practice is located (aver income),

the average hospital length of stay rate for elderly patients per physician (los rate), the

population of the community in which the physician's practice is located (population),

and physician participation in a previous influenza AD service (flu AD).

Table 4-3. Pre-PS Univariate Analysis for Included Variables.
Pre-Propensity Score Values (t-test and proc genmod)
AD = 0 (n = 265) AD = 1 (n = 231) OR
std std p-
Variable mean dev mean dev F value B (exp B)
% OA dx* 0.0913 0.1071 0.0719 0.0598 9.9191 0.0116
GP age 47.3 9.8 45.7 9.2 9.2191 0.0678
% elderly 0.1910 0.1160 0.1772 0.0788 8.9591 0.1182
total # pt 1054 439 1038 419 7.8191 0.6700
aver
income* 27689 4683 25833 4488 11.8791 0.0001
BL rate 3.60 2.70 3.98 2.87 5.8991 0.1356
los rate* 0.0453 0.1449 0.0882 0.1767 4.4191 0.0031
population* 183342 162415 122705 155567 11.6191 0.0001
sex 0.0300 0.8663 -0.0330 0.9675
flu AD* 140.1500 0.0001 -2.6503 0.0706
license 3.3600 0.0669 0.3460 1.4134
birth place 0.1100 0.7415 -0.0553 0.9462
variables that show significant differences at the alpha = 0.05 level

Quintile PS Method Analysis

The distribution of GPs within the quintiles is reported in table 4-4. Quintile one

represents the GPs with the lowest PSs (lowest propensity to volunteer for the

intervention) and quintile five represents the GPs with the highest PSs (highest propensity

to volunteer for the intervention). The table is consistent with the expected PS

distribution with fewer subjects in the high propensity quintile for the control group and

fewer subjects in the low propensity quintile for the intervention group.

The results for the quintile method were generated using PROC GENMOD35 and

are reported in table 4-5. The main effect column represents the main effect of the AD











variable and the interaction effect column represents the effect of the AD by quintile


interaction. The quintile method resulted in no statistically significant difference between


groups on all five VOC while maintaining balance on the rest of the covariates.


Table 4-4. Physician Distribution by Quintile
Quintile # Intervention Control # of GPs
1 86 13 99
2 77 22 99
3 65 35 100
4 24 75 99
5 13 86 99
TOTAL 265 231 496


Table 4-5. Quintile Method Regression Analysis Results.
Quintile Method
Interaction
Ismean Main Effect Effect
AD=0 AD= 1 OR % bias
Variable (n= 265) (n= 231) F p F p B (exp B) reduction
% OA dx* 0.0827 0.0783 0.20 0.6562 1.74 0.1403 77.32
GP age 47.2 47.1 0.01 0.9313 1.01 0.4029 93.75
% elderly 0.1918 0.1808 0.93 0.3351 2.04 0.0880 20.29
total #pt 1045 1031 0.09 0.7673 0.22 0.9256 12.50
aver
income* 26757 26693 0.02 0.8900 1.34 0.2526 96.55
BL rate 3.58 3.68 0.10 0.7479 0.90 0.4616 72.89
los rate* 0.0491 0.0677 1.05 0.3058 0.76 0.5529 56.64
population* 149548 148905 0.00 0.9670 1.27 0.2797 98.94
sex 0.00 0.9818 0.00 0.9801 -0.0130 0.9871 60.21
flu AD* 0.35 0.5537 xx xx -0.3723 0.6891 66.55
license 3.26 0.0709 3.28 0.0700 -0.9702 0.3790 -50.22
birth place 0.01 0.9372 0.01 0.9295 -0.0381 0.9626 30.51
Average** 82.36
variables that were not significant at the alpha = 0.05 level in the pre-PS model
** average % bias reduction for variables with significant differences in the pre-PS model (excluding flu AD)
+ estimates not available (see table 4-6 for explanation)

The interaction effect (AD*quintile) was not significant for four of the five VOC


however, the flu AD variable exhibited an almost complete separation of data points


(table 4-6) and as such the interaction effect was not estimated.


The distribution of the flu AD variable on the PS was problematic for all three PS


methods (figure 4-1). Therefore, the reported average percent bias reduction on the VOC









does not include the flu AD variable. The average percent bias reduction for the quintile

method is 82%.

It is evident at this point that the flu AD variable will have to be included in the

outcome models regardless of the PS method chosen.

Table 4-6. Distribution of Influenza AD Participants by Propensity Score Quintile
Flu AD
Participation Quintile Total
1 2 3 4 5
No 99 99 88 0 0 286
Yes 0 0 12 99 99 210

Regression on the Propensity Score Method Analysis

The results for the regression on the PS method were generated using PROC

GENMOD35 and are reported in table 4-7. This method was successful in balancing three

of the five VOC while maintaining balance on the rest of the covariates. The average

percent bias reduction on the VOC (flu AD excluded) is 99%.

The variable, population, retained a significance level less than 0.05 and it also

exhibited a significant interaction effect (population*AD) at the alpha = 0.05 level. The

variable aver income showed a non-significant main effect with a p value > 0.05

however, the interaction effect (aver income*AD) is less than the 0.05 level. The

separation of data points for the flu AD variable on the PS was again evident. Figure 4-1

shows the distribution of flu AD on PS (stratified at 0.05 intervals). This separation

precluded the model from estimating main and interaction effects for flu AD.

"Greedy Matching" Method Analysis

The results for the "greedy matching" method were generated using PROC

GENMOD35 and are reported in table 4-8. This method was successful in balancing four







42


of the five VOC while maintaining balance on the remainder of the covariates. The


average percent bias reduction on the VOC (flu AD excluded) is 75%.


Table 4-7. Regression on PS Method Analysis Results.
Regression on Propensity Score Method
Interaction
Ismean Main Effect Effect
AD = 0 AD = 1 OR % bias
Variable (n=265) (n=231) F P F p B (exp B) reduction
% OA dx* 0.0770 0.0772 2.19 0.1394 3.07 0.0802 98.97
GP age 47.0 46.9 0.90 0.3441 1.30 0.2548 97.50
% elderly 0.1819 0.1819 0.45 0.5030 0.63 0.4291 100.00
total #pt 1036 1037 0.33 0.5637 0.47 0.4924 93.75
aver
income* 26531 26530 2.83 0.0934 3.92 0.0482 99.95
BL rate 3.68 3.69 0.61 0.4338 0.89 0.3462 97.63
los rate* 0.0613 0.0620 0.32 0.5697 0.48 0.4872 98.37
population* 142500 142661 4.23 0.0402 5.90 0.0155 99.73
sex 0.22 0.6392 0.31 0.5746 0.2073 1.2304 -609.62
flu AD* xx+ xx+ xx+ xx 728.77
license 2.48 0.1152 3.48 0.0622 -0.6744 0.5095 -18.66
birth place 0.03 0.8588 0.04 0.8377 -0.0674 0.9348 -21.15
Average** 99.25
variables that showed significant differences at the alpha = 0.05 level in the pre-PS model
** average % bias reduction for variables that were not significant in the pre-PS model (excluding flu AD)
+estimates not available (see figure 4-1 for explanation)



Graph of Propensity Score vs. Frequency of Flu
AD Participants


c 70
*. 60
S50
40 *fluAD= no
o 30 *flu AD= yes
20
LL 10
O 0




Propensity Score



Figure 4-1. Frequency of Influenza AD Participants by Propensity Score.







43


The flu AD variable estimates were not obtained for the same reasons described in


the regression on the PS method section. With the exception of the flu AD variable, the


"greedy method" balanced all variables and associated interaction terms.


Table 4-8. "Greedy Matching" Method Analysis Results.
"Greedy Matching" Method
Interaction
Ismean Main Effect Effect
AD= 0 AD= 1 OR % bias
Variable (n=104) (n=104) F p F p B (exp B) reduction
% OA dx* 0.0722 0.0788 0.18 0.6712 0.01 0.9330 65.98
GP age 46.4 47.0 2.10 0.1492 1.99 0.1598 62.50
% elderly 0.1768 0.1876 0.39 0.5328 0.05 0.8299 21.74
Total # pt 1074 1008 1.25 0.2650 0.37 0.5410 -312.50
aver
income* 26535 26300 0.23 0.6301 0.12 0.7308 87.34
BL rate 3.75 3.78 0.85 0.3573 1.25 0.2651 92.37
los rate* 0.0456 0.0535 1.04 0.3097 0.89 0.3471 81.59
population* 154497 132123 2.06 0.1527 1.14 0.2862 63.10
Sex 0.53 0.4654 1.26 0.2619 0.4266 1.5320 -1538.99
flu AD* xx+ xx+ xx+ xx 446.3283
License 1.35 0.2459 1.04 0.3077 -0.6730 0.5102 -18.49
birthplace 1.79 0.1809 1.31 0.2520 -0.6828 0.5052 -819.72
Average** 74.50
variables that showed significant differences at the alpha = 0.05 level in the pre-PS model
** average % bias reduction for variables that were not significant in the pre-PS model (excluding flu AD)
+estimates not available


The "greedy matching" method resulted in a decrease in total sample size from 496


(sample size of the two previous methods) to 208. This represents a decrease in sample


size of 58%. The eliminated GPs had PSs that were predominantly in the highest or


lowest ranges of the distribution. The elimination of these GPs could affect the


generalizability of the study since only the GPs who are in the midrange of the PS


distribution would be left in the study.


Selection of a Preferred Propensity Score Method

The selection of a preferred PS method was carried out by measuring each of the


three methods against the following two criteria;


* the resulting sample size after application of the PS method, and









* the PS method's ability to adjust for bias on the VOC.

A major disadvantage of the "greedy matching" method is the reduction in sample

size resulting from the discarding of subjects that are not matched. In this case the

sample size is reduced by 58% which possibly results in a loss of power to detect

significance in the main effects of the outcome models and a loss of generalizability of

the findings. Since the "greedy matching" method does not show advantages over the

regression on the PS method in terms of adjusting for bias on the covariates it is

considered less desirable than the regression on the PS method and will not be selected as

the PS method for inclusion in the outcome models.

The regression on PS method was responsible for the greatest adjustment for bias

between groups on all of the VOC (figure 4-2). The average reduction in bias for the

regression on the PS method was 99% versus 82% for the quintile method.

With this dataset the regression on the PS method is preferred and it is the method

that will be applied to the outcome analyses. It is important to note that the failure to

adjust for bias on the flu AD variable still exists and as such the flu AD variable will be

included in the outcome models.

Exploratory Analysis of the Propensity Score Methods Effect on Adjusting for Bias
on Unmeasured Variables

The purpose of this exploratory analysis is to determine whether any one PS

method is better at reducing bias on variables that are not included in the PS model and

are, therefore, considered unmeasured.

The c-statistic is a measure of the model's ability to discriminate between groups.

The c-statistic for the full model is 0.832 which can be interpreted as follows; if one

randomly select one subject from each AD group the model will accurately predict the












group from which the subjects originated 83.2% of the time. With the exception of the


models dealing with the exclusion of the flu AD variable, the c-statistic remains stable for


all of the PS models. The range is from 0.830 to 0.835 (table 4-9).



Percent Reduction in Bias on Unbalanced

Variables (VOC)


S 100.00
.= 80.00
" 60.00
"Z 40.00
-| 20.00
l 0.00
% OA dx Aver
Income


los rate Population Average


Variable


Quintile u Regr on PS c Greedy Match



Figure 4-2. Comparison of PS methods Ability to Reduce Bias on VOC.


Table 4-9. Quintile Method Results for Excluded Variable Models.
Quintile Method


Ismean
AD= 0 AD= 1
(n=265) (n=231)
0.0935 0.0708
46.8 47.5
0.1960 0.1754
1085 1006


26777
3.6
0.0471
153438


26652
3.7
0.0730
148662


Main Effect


P
0.0207
0.5426
0.079
0.1111

0.7862
0.7418
0.1615
0.7746
0.9217
0.0001
0.3673
0.7528


Interaction
Effect


P
0.5298
0.5112
0.1374
0.0231

0.2361
0.5274
0.5214
0.1805
0.6197
0.2957
0.0896
0.8820


OR % bias
B (exp B) reduction
-17.01
56.25
-49.28
-393.75


-0.0593
-2.0411
-0.5001
-0.1524


0.9424
0.1299
0.6065
0.8586


93.27
73.68
39.63
92.12
-77.37
6.38
4.81
-162.75
42.88


* variables that showed significant differences at the alpha 0.05 level in the pre-PS model
** average % bias reduction for variables that showed significant differences in the pre-PS model


Excluded
Variable
% OA dx*
GP age
% elderly
total # pt
aver
income*
BL rate
los rate*
population*
sex
flu AD*
license
birth place
Average**


c
0.833
0.833
0.831
0.833

0.834
0.834
0.833
0.832
0.835
0.662
0.830
0.835









There are three c-statistics that are worth noting. The first is the c-statistic that is

generated for the model when the flu AD variable is removed. It has been noted that

there exists an almost complete separation of data for the flu AD variable on the PS so

when the flu AD variable is excluded from the model the ability of the model to

discriminate decreases from 0.834 to 0.662. The other two are the c-statistics associated

with sex and birth place. These two variables have the distinction of being the most

closely balanced variables in the pre-PS analysis (table 4-3) with p-values of 0.8663 and

0.7415 respectively. The PS model c-statistics when these variables are excluded is equal

to 0.835 in both cases. This value is greater than the c-statistic for the full model thereby

indicating that the inclusion of these variables in the PS model decreases the model's

discriminative ability.

The complete results from the reduced PS models are reported in tables 4-9 through

4-11. The analysis of the reduced models effect's on balancing the VOC is summarized

in figure 4-3. Figure 4-3 shows that no one PS method systematically reduces bias on

unmeasured variables to a greater extent than the others. Regression on PS does, on

average, reduce bias on the VOC to the greatest degree.

The summary of PS models effects (figure 4-3) shows that bias between groups on

unmeasured variables can be reduced by PS methods. The correlation matrix between the

VOC and the PS covariates was calculated and reported in table 4-12. Table 4-12 shows

limited correlation (less than 0.30) between the VOC and the PS covariates in all cases

except one. The one exception is the correlation between population (population of

community where the GP practice is located) and aver income (average income for

county where GP practice located) which was 0.91. The correlation between population








47



and aver income is associated with the higher reduction in bias for those variables when


they are not included in the PS model.


Table 4-10. Regression on PS Results for Excluded Variable Models.

Regression on Propensity Score Method
Interaction
Ismean Main Effect Effect
Excluded AD = 0 AD= 1 OR % bias
Variable c (n=265) (n=231) F p F p B (exp B) reduction


% OA dx* 0.833 0.0901 0.0736 0.74 0.3909 0.00 0.9540
GP age 0.833 46.5 47.2 1.14 0.2854 0.72 0.3974
% elderly 0.831 0.1918 0.1777 0.27 0.6065 0.04 0.8445
total #pt 0.833 1061 1018 0.69 0.4080 0.15 0.6946


14.95
56.25
-2.17
-168.75


aver
income* 0.834 26567 26510 2.83 0.0929 3.65 0.0566 96.93
BL rate 0.834 3.7 3.7 0.57 0.4515 0.96 0.3286 100.00
Los rate* 0.833 0.0513 0.0684 0.19 0.6641 1.26 0.2618 60.14
population* 0.832 146919 139289 4.70 0.0307 5.12 0.0240 87.42
sex 0.835 0.06 0.7993 0.68 0.4108 0.1155 1.1224 -277.17
Flu AD* 0.662 3.16 0.0756 2.12 0.1456 -1.4258 0.2403 18.26
license 0.830 1.41 0.2348 4.59 0.0322 -0.5203 0.5943 1.87
birth place 0.835 0.09 0.7695 0.02 0.8911 -0.1117 0.8943 -96.45
Average** 55.54
variables that showed significant differences at the alpha 0.05 level in the pre-PS model
** average % bias reduction for variables that showed significant differences in the pre-PS model


Table 4-11. "Greedy Matching" Results for Excluded Variable Models.

"Greedy Matching" Method
Interaction
Ismean Main Effect Effect


Excluded ni OR % bias
Variable (i=0,1) AD= 0 AD= 1 F p F p B (exp B) reduction


% OA dx*
GP age
% elderly
total # pt
aver
income*
BL rate
Los rate*
population*
sex
Flu AD*
license
birth place
Average**


106 0.0969 0.0707 4.36 0.0380 0.00 0.9677
101 46.2 47.5 1.40 0.2381 0.59 0.4450
105 0.1944 0.1801 0.41 0.5205 0.01 0.9365
103 1049 1005 0.61 0.4371 0.17 0.6676


105 26875 26123 1.02 0.3136 0.22 0.6419
103 3.7 3.7 0.77 0.3806 1.01 0.3169
105 0.0352 0.0614 1.09 0.2967 0.06 0.8145
104 154497 132123 2.06 0.1527 1.14 0.2862
104 0.00 0.9674 0.24 0.6260 0.0119 1.0120
190 2.00 0.1570 1.34 0.2471 -0.7168 0.4883
104 5.04 0.0248 9.09 0.0026 -0.6618 0.5159
105 0.00 0.9787 0.28 0.5943 0.0069 1.0069


-35.05
15.69
-3.62
-175.00

59.48
97.11
38.93
63.10
63.12
44.94
-17.10
87.22
34.28


* variables that showed significant differences at the alpha 0.05 level in the pre-PS model
** average % bias reduction for variables that showed significant differences in the pre-PS model











Percent Reduction in Bias on Unbalanced and Unmeasured
Variables

100.00
80.00
C
o 60.00
S40.00o
20.00
0.00 -


-40.00
Variable

SQuintile 0 Regron PS l Greedy Match


Figure 4-3. Summary of PS Models Effects on Reducing Bias on the VOC

Table 4-12. Correlation Matrix Between VOC and PS Covariates.
Correlation Between VOC and All PS Covariates
Covariate VOC
% OA aver
los rate dx population income flu AD
BL rate -0.06 -0.02 -0.22 -0.26 0.00
los rate 1.00 -0.01 -0.05 -0.05 0.03
% OA dx -0.01 1.00 0.01 0.01 0.01
total # pt -0.12 -0.10 0.00 0.02 0.02
% elderly -0.02 0.26 0.09 0.09 -0.03
sex -0.08 0.04 -0.13 -0.13 0.03
flu AD 0.03 0.01 -0.14 -0.17 1.00
population -0.05 0.01 1.00 0.91 -0.14
GP age -0.05 0.07 0.06 0.06 -0.09
aver income -0.05 0.01 0.91 1.00 -0.17

The effect of the correlation between the PS and the VOC and the reduction in bias

was tested. The correlation between the PS and the VOC was calculated and scatter plots

were compiled to display the results graphically in figure 4-4.

The absolute values of the correlations ranged from 0.182 to 0.329. The absolute

value of the correlations was plotted against the percent bias reductions on the VOC for

each of the three PS methods (figure 4-5). The results from figure 4-5 show an overall














effect of increasing percent bias reduction with increasing absolute correlation between



the PS and the excluded variable.


Scatter Plot: Propensity Score vs. Percent of
Patients with OA Diagnosis
(Rho = -0.182)


.* ** .
..*








S01 0 2 03 04 05 06 07 08 09
Propensity Score



Scatter Plot: Propensity Score vs. Community
Population
(Rho= 0.311)


02 04 06 08
Propenity Score


Scatter Plot: Propensity Score vs. Length of
Stay Rate
(Rho = 0.220)








*



S **
*




so $
*. *%












.* e..* .esme


Propensity Score


Figure 4-4. Scatterplots of Propensity Score Versus Unbalanced Variables


Step 3: Primary Outcome Analysis


Model Development


The analysis of the primary outcome, the effect of the OA AD intervention on the



COX-2 utilization rates was carried out using a repeated measures model on longitudinal


*** ..*


br ~bfl. .










data (PROC GENMOD35). There were six experimental time periods over which the

outcomes measures were assessed (figure 3-2).


Correlation (PS vs. Excluded Var.) vs. Percent Bias Reduction (by
PS Method)

100.00

80.00

S60.00
01
S40.00

0 20.00
S0.00
0 0 5 o1 /' 1 0 21 0 23 025 0 2 029 0 31 033 0 5
-20.00

-40.00
Rho

-. Quintile -i- Regr on PS ---Greedy Match


Figure 4-5. Line Graph Comparing Correlations and Percent Bias Reduction

The primary outcome measure, the change in COX-2 prescribing from baseline,

was calculated for each physician by aggregating all of the COX-2 prescription claims for

all of the elderly patients in the physician's panel and dividing by the number of elderly

patients in the panel. The resulting rate, number of COX-2 DDDs per patient per

physician was subtracted from the baseline prescribing rate to yield a measure of change

in COX-2 prescribing.

The primary outcome model included the variables intervention participation (AD),

the PS (pr), the time period in which the measurement took place (period), participation

in a previous influenza AD service (flu AD), the baseline COX-2 prescribing rate (BL

rate), and the number of elderly patients in the GP's panel (# elderly). The model is

depicted in figure 4-6. The variables were included for the following reasons. The PS









variable represents the outcome from the PS analysis, the period variable controls for the

longitudinal changes, the flu AD variable was not successfully balanced by the PS

method, and the baseline COX-2 rate and number of elderly patients control for the GP's

pre-intervention prescribing behavior and practice size respectively.

Y = Po + Pl(Xl) + P2(X2) + P3(X3) + 4(X4) + 5(X5) + (X6)

Where;
Y = change in COX-2 utilization rate (periods 3 to 6 (post-intervention)),
Xi = physician participation in the intervention (0 = no, 1 = yes),
X2 = PS (range from 0 to 1),
X3 = experimental time period (period = 3,4,5,6),
X4 = physician participation in the influenza AD service (0 = no, 1 = yes),
X5 = physician baseline COX-2 rate (DDD / patient, (period = 2)),
X6 = number of patients in the GP's practice >65 years old


Figure 4-6. Primary Outcome Model

Between Group Results

The significance level of each variable from the primary outcome model (figure 4-

6) is listed in table 4-13. The values of the coefficient estimates in GEE are not

interpreted in the same manner as GLM models37 and as such the values of the coefficient

estimates are not reported in the results tables. A more in-depth discussion of the

interpretation of GEE results is included in the discussions in chapter five.

The between groups effect of the intervention is interpreted from the value of the z

statistic for the AD variable. The z value of 0.85 and associated p-value of 0.3976

indicates that the main intervention effect over the entire post-intervention period is not

statistically significant.

The model in figure 4-6 was also used to determine between group differences in

the pre-intervention time periods (period = 1, 2). The z statistics and associated p-values

of each variable are listed in table 4-14. The pre-intervention results are interpreted in the









same manner as the post-intervention results. The z statistic and p-value for the AD

variable are 0.88 and 0.3775 respectively. The p-value indicates that the groups are not

significantly different on the outcome measure in the pre-intervention periods at the alpha

= 0.05 level.

Table 4-13. Primary Outcome Model Results (Periods = 3,4,5,6).
Primary Outcome Model Results for Post-intervention Periods
(COX-2 Prescribing Rates)
Effect Z p-value
AD (AD = no) 0.85 0.3976
PS 0.84 0.4023
period 2.69 0.0072
flu AD (flu AD = no) 1.21 0.2255
BL rate -10.68 <0.0001
# elderly 1.64 0.1017

Table 4-14. Primary Outcome Model Results (Periods = 1,2).
Primary Outcome Model Results for Pre-intervention Periods
(COX-2 Prescribing Rates)
Effect Z p-value
AD (AD = no) 0.88 0.3775
PS 0.31 0.7588
period 0.38 0.7018
flu AD (flu AD = no) 0.23 0.8170
BL rate -14.45 <0.0001
# elderly -0.48 0.6313

Table 4-15 depicts the least square means for the two groups (AD = yes and AD =

no) for each of the six experimental time periods. The least square mean values are also

presented in a graph in figure 4-7.

Table 4-16 depicts the unadjusted means and standard deviations for the two

groups (AD = yes and AD = no) for each of the six experimental time periods. A positive

value indicates that the prescribing rate has increased from the baseline rate by the

amount indicated and a negative value indicates a decrease in the prescribing rate from

baseline. The unadjusted mean values are also presented in a graph in figure 4-8.










Table 4-15. Least Square Means for Change in COX-2 Rates by Group (DDDs/patient).
AD
group period
1 2 3 4 5 6
0 0.0516 0 0.2275 -0.1778 0.2471 0.4178
1 -0.2136 0 -0.5315 -0.0018 0.261 0.1457


Primary Outcome:
Change in COX-2 Rates (adjusted)
0.5

0

o -0.5

-1
Time Period
Intervention = No Intervention = Yes



Figure 4-7. Least Square Means for Change in COX-2 Rates by Group

Table 4-16. Unadjusted Means for Change in COX-2 Rates by Group (DDDs/patient).
Period AD group
0 1
Mean Std Dev Mean Std Dev
1 0.1587 3.4287 -0.3026 3.0709
2 0.0000 0.0000 0.0000 0.0000
3 0.3396 3.9059 -0.5321 3.4410
4 0.0236 3.6700 -0.1257 3.7358
5 0.5266 3.7285 -0.0008 3.9072
6 0.6454 3.8566 0.1281 3.9248

Within Group (Longitudinal) Results

The within group models were the same as the between group model in figure 4-6

except that the AD group variable is replaced by a prepost variable which measures

significant within group differences between change in COX-2 rates pre-intervention and

post-intervention. The model is run two times; once including only the intervention

group and once including only the control group. The z statistic value and associated










significance level (p-value) of each variable are listed in table 4-17 for the intervention

group and table 4-18 for the control group.


Primary Outcome:
Change in COX-2 Rates (unadjusted)

0.8000
0.6000
) 0.4000
a 0.0000-

C o -0.2000 3 5 6
-0.4000
-0.6000
Time Period

Intervention = No -- Intervention = Yes


Figure 4-8. Unadjusted Means for Change in COX-2 Rates by Group

The within group effect of the intervention is interpreted from the values of the z

statistic and significance level of the prepost variable. For the intervention group, the z

and p-values of -2.34 and 0.0191 respectively indicates that the within group effect is

significant at the alpha = 0.05 level. For the control group, the z statistic and p-value of -

-0.22 and 0.8273 respectively indicates that the within group effect is not significant at

the alpha = 0.05 level.

Table 4-17. Primary Outcome Model Results (AD = yes).
Primary Outcome Results for the Intervention Group
(COX-2 Prescribing Rates)
Effect Z p-value
PS 0.04 0.9708
period 2.82 0.0049
prepost -2.34 0.0191
flu AD (flu AD = no) 0.49 0.6217
BL rate -9.74 <0.0001
# elderly 0.63 0.5271









Step 4: Secondary Outcome Analyses

Misoprostol Utilization Rates

Model development

The analysis of the secondary outcome, the effect of the OA AD intervention on the

misoprostol utilization rate was carried out using the same methods as the primary

outcome analysis with the data for misoprostol utilization substituted for the COX-2

utilization data (figure 4-9).

Table 4-18. Primary Outcome Model Results (AD = no).
Primary Outcome Results for the Control Group
(COX-2 Prescribing Rates)
Effect Z p-value


PS
period
prepost
flu AD (flu AD
BL rate
# elderly


1.32
1.31
-0.22
0.95
-8.68
1.10


0.1881
0.1910
0.8273
0.3412
<0.0001
0.2727


Y = Po + Pl(Xl) + P2(X2) + P3(X3) + 4(X4) + 5(X5) + P6(X6)

Where;
Y = change in misoprostol utilization rate (periods 3 to 6 (post-intervention)),
X1 = physician participation in the intervention (0 = no, 1 = yes),
X2 = PS (range from 0 to 1),
X3 = experimental time period (period = 3,4,5,6),
X4 = physician participation in the influenza AD service (0 = no, 1 = yes),
X5 = physician baseline misoprostol rate (DDD / patient, (period = 2)),
X6 = number of patients in the GP's practice >65 years old


Figure 4-9. Secondary Outcome Model for Misoprostol Utilization

Between group results

The z statistic and the significance level (p-value) of each variable from the

secondary misoprostol outcome model (figure 4-9) are listed in table 4-19.









The between group effect of the intervention is interpreted from the z statistics and

associated p-value of the AD variable. The z statistic and p-value of -0.87 and 0.3866

respectively indicate that the effect is not significant at the alpha = 0.05 level.

Table 4-19. Secondary Misoprostol Outcome Model Results (Periods = 3,4,5,6).
Secondary Outcome Model Results for Post-intervention Periods 3 to 6
(Change in Misoprostol Prescribing Rates)
Effect Z p-value
AD (ad = 0) -0.87 0.3866
PS -0.61 0.5412
period 0.96 0.3359
flu AD (flu AD = 0) -0.53 0.5943
BL rate -6.31 <0.0001
# elderly -0.24 0.8091

The model in figure 4-9 was used to determine intervention effects on each post

intervention time period. None of the post intervention (analyzed individually) showed

significant between group differences at the alpha = 0.05 level.

The model in figure 4-9 was used to determine between group differences in the

pre-intervention time periods (period = 1, 2). The z statistic and associated significance

level of each variable is listed in table 4-20. The results are interpreted in the same

manner as the post-intervention results. The z statistic and p-value for the AD variable

are -0.22 and 0.8269 respectively. The p-value indicates that the groups are not

significantly different in the pre-intervention periods at the alpha = 0.05 level.

Table 4-20. Secondary Misoprostol Outcome Model Results (Periods = 1,2).
Secondary Outcome Model Results for Post-intervention Periods 1 and 2
(Change in Misoprostol Prescribing Rates)
Effect Z p-value
AD (ad = 0) -0.22 0.8269
PS 1.20 0.2308
period 0.28 0.7758
flu AD (flu AD = 0) 1.18 0.2396
BL rate -4.55 <0.0001
# elderly 0.58 0.5612










Table 4-21 depicts the least square means for the two groups (AD = yes and AD =

no) for each of the six experimental time periods. The least square mean values are also

presented in a graph in figure 4-10.

Table 4-21. Least Square Means for Change in Misoprostol Rate by Group
(DDDs/patient).
Secondary Outcome (Misoprostol) Least Square Means by AD Group
AD group Period
1 2 3 4 5 6
0 -0.0191 0.0000 0.0172 0.0516 0.0390 0.0388
1 -0.0127 0.0000 0.0383 0.0743 0.0604 0.0652


Secondary Outcome:
Change in Misoprostol Rates (adjusted)

0.1
0.1-

5, fI 0.0 ---- M3oe< 4 ---------
0.0 -
~o 0.0 -
0.0 2 4 5 6
0.0
Time Period

SIntervention = No Intervention = Yes


Figure 4-10. Least Square Means for Change in Misoprostol Rates by Group.

Table 4-22 depicts the unadjusted means and standard deviations for the two

groups (AD = yes and AD = no) for each of the six experimental time periods. The

unadjusted mean values are also presented in a graph in figure 4-11.

Within group (longitudinal) results

The within group model was the same as the between group model in figure 4-9

except that the AD group variable is replaced by a prepost variable which measures

within group differences between change in misoprostol rates pre-intervention and post-

intervention. The model is run two times; once including only the intervention group and










once including only the control group. The z statistic and the associated significance

level (p-value) of each variable are listed in table 4-23 for the intervention group and

table 4-24 for the control group.

Table 4-22. Unadjusted Means and Standard Deviations for Change in Misoprostol Rate
by Group (DDDs/patient).


Period


AD group


Mean
-0.0153
0.0000
0.0032
0.0586
0.0484
0.0235


Std Dev
0.2877
0.0000
0.2693
0.3289
0.3750
0.3782


Mean
0.0096
0.0000
0.0437
0.0570
0.0497
0.0850


Secondary Outcome:
Change in Misoprostol Rates (unadjusted)


0.10
0.08
0.06
0.04
0.02
0.00
-0.02
-0.04


Std Dev
0.2969
0.0000
0.2581
0.3120
0.3472
0.4005


Time Period

- Intervention = no -- Intervention = yes


Figure 4-11. Unadjusted Means for Change in Misoprostol Rates by Group.

Table 4-23. Secondary Misoprostol Outcome Model Results (AD = yes).
Secondary Outcome Model Results for All Periods (Intervention Group)
(Change in Misoprostol Prescribing Rates)


Effect
PS
period
prepost
flu AD (flu AD
BL rate
# elderly


Z
-0.58
1.65
0.25
-0.30
1.23
0.55


p-value
0.5594
0.0990
0.8075
0.7612
0.2195
0.5802


4 5 6









Table 4-24. Secondary Misoprostol Outcome Model Results (AD = no).
Secondary Outcome Model Results for All Periods (Control Group)
(Change in Misoprostol Prescribing Rates)
Effect Z p-value
PS -0.01 0.9921
period 0.75 0.4523
prepost 1.00 0.3176
flu AD (flu AD = 0) -0.27 0.7888
BL rate 4.69 <0.0001
# elderly 1.59 0.1109

The within group effect of the intervention is interpreted from the values of the z

statistic and associated p-value of the prepost variable. For the intervention and control

groups, z statistics and the p-values of 0.25, 0.8075 and 1.00, 0.3176 respectively

indicates that the within group effect is not statistically significant for both groups at the

alpha = 0.05 level.

PPI Utilization Rates

Model development

The analysis of the secondary outcome, the effect of the OA AD intervention on the

PPI utilization rates was carried out using the same methods as the primary outcome

analysis with the data for PPI utilization substituted for the COX-2 utilization data (figure

4-12).

Y = Po + Pl(Xl) + P2(X2) + P3(X3) + 4(X4) + 5(X5) + P6(X6)

Where;
Y = change in PPI utilization rate (periods 3 to 6 (post-intervention)),
X1 = physician participation in the intervention (0 = no, 1 = yes),
X2 = PS (range from 0 to 1),
X3 = experimental time period (period = 3,4,5,6),
X4 = physician participation in the influenza AD service (0 = no, 1 = yes),
X5 = physician baseline PPI rate (DDD / patient, (period = 2)),
X6 = number of patients in the GP's practice >65 years old


Figure 4-12. Secondary PPI Outcome Model









Between group results

The z statistic and the significance level (p-value) of each variable from the

secondary PPI outcome model (figure 4-12) are listed in table 4-25.

Table 4-25. Secondary PPI Outcome Model Results (Periods = 3,4,5,6).
Secondary Outcome Model Results for Post-intervention Periods 3 to 6
(Change in PPI Prescribing Rates)
Effect Z p-value
AD (ad = 0) -0.27 0.7906
PS 1.09 0.2755
period 1.43 0.1519
flu AD (flu AD = 0) 1.45 0.1478
BL rate -2.92 0.0035
# elderly -1.74 0.0813

The between group effect of the intervention is interpreted from the z statistics and

associated p-value of the AD variable. The z statistic and p-value of -0.27 and 0.7906

respectively indicate that the effect is not significant at the alpha = 0.05 level.

The model (figure 4-12) was used to determine intervention effects on each post

intervention time period. None of the post intervention (analyzed individually) showed

significant between group differences at the alpha = 0.05 level.

The model in figure 4-12 was used to determine between group differences in the

pre-intervention time periods (period = 1, 2). The z statistic and associated significance

level of each variable is listed in table 4-26. The results are interpreted in the same

manner as the post-intervention results. The z statistic and p-value for the AD variable

are 0.13 and 0.8989 respectively. The p-value indicates that the groups are not

significantly different in the pre-intervention periods at the alpha = 0.05 level.

Table 4-27 depicts the least square means for the two groups (AD = yes and AD =

no) for each of the six experimental time periods. The least square mean values are also

presented in a graph in figure 4-13.










Table 4-26. Secondary PPI Outcome Model Results (Periods = 1,2).
Secondary Outcome Model Results for Post-intervention Periods 1 and 2
(Change in PPI Prescribing Rates)
Effect Z p-value
AD (ad = 0) 0.13 0.8989
PS 1.10 0.2726
period 0.14 0.8911
flu AD (flu AD = 0) 1.08 0.2818
BL rate -5.61 <0.0001
# elderly 0.04 0.9700

Table 4-27. Least Square Means for Change in PPI Rates by Group (DDDs/patient).
Secondary Outcome (PPI) Least Square Means by AD Group
AD group period
1 2 3 4 5 6
0 -0.0388 0 0.3194 0.5271 0.507 0.4842
1 -0.0553 0 0.3675 0.5513 0.555 0.4982


Secondary Outcome:
Change in PPI Rates (adjusted)

0.6

0.4
0.2


-0.2 1 2 3 4 5 6
Time Period

Intervention = No -- Intervention =Yes


Figure 4-13. Least Square Means for Change in PPI Rates by Group.

Table 4-28 depicts the unadjusted means and standard deviations for the two

groups (AD = yes and AD = no) for each of the six experimental time periods. The

unadjusted mean values are also presented in a graph in figure 4-14.

Within group (longitudinal) results

The within group model was the same as the between group model in figure 4-12

except that the AD group variable is replaced by a prepost variable which measures










within group differences between change in PPI rates pre-intervention and post-

intervention. The model is run two times; once including only the intervention group and

once including only the control group. The z statistic and the associated significance

level (p-value) of each variable are listed in table 4-29 for the intervention group and

table 4-30 for the control group.

Table 4-28. Unadjusted Means for Change in PPI Rate by Group (DDDs/patient).
Period OA group
0 1
Mean Std Dev Mean Std Dev
1 -0.0203 1.4843 0.0041 1.4211
2 0.0000 0.0000 0.0000 0.0000
3 0.3932 1.3733 0.3372 1.5783
4 0.6307 1.6214 0.5810 1.6686
5 0.5844 1.6238 0.6182 1.7077
6 0.5575 1.7553 0.5490 1.6955


Secondary Outcome:
Change in PPI Rates (unadjusted)

0.8
0.6 -
w 0 0.4-
o .- 0.2
o 0.0 -
-0.2 41 5 4
Time Period

SIntervention = no -- Intervention = yes


Figure 4-14. Unadjusted Means for Change in PPI Rates by Group.

The within group effect of the intervention is interpreted from the values of the z

statistic and associated p-value of the prepost variable. For the intervention and control

groups, the z statistics (p-values) of -2.59 (0.0097) and -4.22 (<0.0001) respectively

indicates that the within group effect is statistically significant for both groups at the

alpha = 0.05 level and both changes are in the direction of increased utilization.









Table 4-29. Secondary PPI Outcome Model Results (AD = yes).
Secondary Outcome Model Results for All Periods (Intervention Group)
(Change in PPI Prescribing Rates)
Effect Z p-value
PS 1.41 0.1596
period 0.69 0.4873
prepost -2.59 0.0097
flu AD (flu AD = 0) 1.37 0.1717
BL rate -3.63 0.0003
# elderly -0.40 0.6879

Table 4-30. Secondary PPI Outcome Model Results (AD = no).
Secondary Outcome Model Results for All Periods (Control Group)
(Change in PPI Prescribing Rates)
Effect Z p-value
PS 0.67 0.5012
period -0.02 0.9877
prepost -4.22 <0.0001
flu AD (flu AD = 0) 1.16 0.2450
BL rate -3.32 0.0009
# elderly -1.61 0.1065

H2A Utilization Rates

Model development

The analysis of the secondary outcome, the effect of the OA AD intervention on the

H2A utilization rates was carried out using the same methods as the primary outcome

analysis with the data for H2A utilization substituted for the COX-2 utilization data

(figure 4-15).

Between group results

The z statistic and the significance level (p-value) of each variable from the

secondary H2A outcome model (figure 4-15) are listed in table 4-31.

The between group effect of the intervention is interpreted from the z statistics and

associated p-value of the AD variable. The z statistic and p-value of 0.05 and 0.9619

respectively indicate that the effect is not significant at the alpha = 0.05 level.










Y = Po + Pl(Xl) + P2(X2) + P3(X3) + 4(X4) + 5(X5) + P6(X6)

Where;
Y = change in H2A utilization rate (periods 3 to 6 (post-int
X1 = physician participation in the intervention (0 = no, 1 =
X2 = PS (range from 0 to 1),
X3 = experimental time period (period = 3,4,5,6),
X4 = physician participation in the influenza AD service (0
X5 = physician baseline H2A rate (DDD / patient, (period =
X6 = number of patients in the GP's practice >65 years old


ervention)),
yes),


= no,1 = ye!
2)),


Figure 4-15. Secondary Outcome Model for H2A Utilization

Table 4-31. Secondary H2A Outcome Model Results (Periods = 3,4,5,6).
Secondary Outcome Model Results for Post-intervention Periods 3 to 6
(Change in H2A Prescribing Rates)
Effect Z p-value
AD (ad = 0) 0.05 0.9619
PS 1.18 0.2381
period -7.29 <0.0001
flu AD (flu AD = 0) 1.12 0.2642
BL rate -7.31 <0.0001
# elderly 1.77 0.0766

The model (figure 4-15) was used to determine intervention effects on each post

intervention time period. None of the post intervention (analyzed individually) showed

significant between group differences at the alpha = 0.05 level.

The model in figure 4-15 was used to determine between group differences in the

pre-intervention time periods (period = 1, 2). The z statistic and associated significance

level of each variable is listed in table 4-32. The results are interpreted in the same

manner as the post-intervention results. The z statistic and p-value for the AD variable

are 1.09 and 0.2764 respectively. The p-value indicates that the groups are not

significantly different in the pre-intervention periods at the alpha = 0.05 level.


s),










Table 4-33 depicts the least square means for the two groups (AD = yes and AD =

no) for each of the six experimental time periods. The least square mean values are also

presented in a graph in figure 4-16.

Table 4-32. Secondary H2A Outcome Model Results (Periods = 1,2).
Secondary Outcome Model Results for Post-intervention Periods 1 and 2
(Change in H2A Prescribing Rates)
Effect Z p-value
AD (ad = 0) 1.09 0.2764
PS 0.98 0.3293
Period -2.74 0.0062
flu AD (flu AD = 0) 0.64 0.5230
BL rate -5.75 <0.0001
# elderly 1.49 0.1368

Table 4-33. Least Square Means for Change in H2A Rate by Group (DDDs/patient).
Secondary Outcome (H2A) Least Square Means by AD Group
AD group period
1 2 3 4 5 6
0 0.3007 0 0.1833 0.0114 -0.1433 -0.5532
1 0.0881 0 0.0157 0.1413 0.0867 -0.6818


Secondary Outcome:
Change in H2A Rates (adjusted)

0.5

0-I
S 1 2 3 4 6
-0.5 -
-1

Time Period

----Intervention = no -- Intervention =yes

Figure 4-16. Least Square Means for Change in H2A Rates by Group.

Table 4-34 depicts the unadjusted means and standard deviations for the two

groups (AD = yes and AD = no) for each of the six experimental time periods. The

unadjusted mean values are also presented in a graph in figure 4-17.










Table 4-34. Unadjusted Means for Change in H2A Rate by Group (DDDs/patient).
Period AD group
0 1
Mean Std Dev Mean Std Dev
1 0.2440 1.7956 0.2117 1.9819
2 0.0000 0.0000 0.0000 0.0000
3 0.1655 1.9023 0.0783 2.0181
4 0.0607 1.8949 0.2191 2.2941
5 -0.2484 2.4465 0.2929 2.4570
6 -0.5101 2.5878 -0.5380 2.5854


Secondary Outcome:
Change in H2A Rates (unadjusted)


0
CD
r I0
oc
o .s
oii


Time Period

- Intervention = no -- Intervention = yes


Figure 4-17. Unadjusted Means for Change in H2A Rates by Group.

Within group (longitudinal) results

The within group model was the same as the between group model in figure 4-15

except that the AD group variable is replaced by a prepost variable which measures

within group differences between change in H2A rates pre-intervention and post-

intervention. The model is run two times; once including only the intervention group and

once including only the control group. The z statistic and the associated significance

level (p-value) of each variable are listed in table 4-35 for the intervention group and

table 4-36 for the control group.

The within group effect of the intervention is interpreted from the values of the z

statistic and associated p-value of the prepost variable. For the intervention and control









groups, the z statistics (p-values) of -5.56 (<0.0001) and -4.06 (<0.0001) respectively

indicates that the within group effect is statistically significant for both groups at the

alpha = 0.05 level and both changes are in the direction of decreased utilization.

Table 4-35. Secondary H2A Outcome Model Results (AD = yes).
Secondary Outcome Model Results for All Periods (Intervention Group)
(Change in H2A Prescribing Rates)
Effect Z p-value
PS 1.70 0.0897
period -6.59 <0.0001
prepost -5.56 <0.0001
flu AD (flu AD = 0) 1.53 0.1262
BL rate -7.48 <0.0001
# elderly 2.80 0.0051

Table 4-36. Secondary H2A Outcome Model Results (AD = no).
Secondary Outcome Model Results for All Periods (Control Group)
(Change in H2A Prescribing Rates)
Effect Z p-value
PS -0.79 0.4282
period -4.06 <0.0001
prepost -2.33 0.0201
flu AD (flu AD = 0) -0.78 0.4366
BL rate 3.43 0.0006
# elderly -1.64 0.1003

GP Office Visit Rates

Model development

The analysis of the secondary outcome, the effect of the OA AD intervention on

GP office visit rates was carried out using the same methods as the primary outcome

analysis with the data for GP office visit rates substituted for the COX-2 utilization data

(figure 4-18).

Between group results

The z statistic and the significance level (p-value) of each variable from the

secondary GP office visit outcome model (figure 4-18) are listed in table 4-37.










Y = Po + Pl(Xl) + P2(X2) + P3(X3) + 4(X4) + 5(X5) + P6(X6)

Where;
Y = change in GP visit rates (periods 3 to 6 (post-intervention)),
Xi = physician participation in the intervention (0 = no, 1 = yes),
X2 = PS (range from 0 to 1),
X3 = experimental time period (period = 3,4,5,6),
X4 = physician participation in the influenza AD service (0 = no, 1 = yes),
X5 = physician baseline GP visit rate rate (visits / patient, (period = 2)),
X6 = number of patients in the GP's practice >65 years old


Figure 4-18. Secondary Outcome Model for GP Office Visits

Table 4-37. Secondary GP Office Visit Model Results (Periods = 3,4,5,6).
Secondary Outcome Model Results for Post-intervention Periods 3 to 6
(Change in GP Office Visit Rates)
Effect Z p-value
AD (ad = 0) 1.06 0.2888
PS 0.74 0.4587
period -9.26 <0.0001
Flu AD (flu AD = 0) 0.02 0.9815
BL rate -1.97 0.0487
# elderly 1.26 0.2077

The between group effect of the intervention is interpreted from the z statistics and

associated p-value of the OA AD variable. The z statistic and p-value of 1.06 and 0.2888

respectively indicate that the effect is not significant at the alpha = 0.05 level.

The model (figure 4-18) was used to determine intervention effects on each post

intervention time period. Only the period from 91 to 180 days (period four) following the

intervention showed significant difference between groups at the alpha = 0.05 level. The

z-statistic and p-value associated with the intervention effect are -2.20 and 0.0275

respectively (95% CI -0.7926, -0.0464). In this case, where the analysis only includes

one time period, the interpretation of the coefficient estimate is similar to traditional

GLM methods. That is, the coefficient estimate of -0.4195 (AD = no) is interpreted as

the non-intervention group having measures of average change rate 0.4195 fewer









visits/patient/GP than the intervention group (equal values for the groups is

hypothesized).

The model in figure 4-18 was used to determine between group differences in the

pre-intervention time periods (period = 1, 2). The z statistic and associated significance

level of each variable is listed in table 4-38. The results are interpreted in the same

manner as the post-intervention results. The z statistic and p-value for the OA AD

variable are 0.37 and 0.7097 respectively. The p-value indicates that the groups are not

significantly different in the pre-intervention periods at the alpha = 0.05 level.

Table 4-38. Secondary GP Office Visit Outcome Model Results (Periods = 1,2).
Secondary Outcome Model Results for Post-intervention Periods 1 and 2
(Change in GP Office Visit Rates)
Effect Z p-value
AD (ad = 0) 0.37 0.7097
PS 1.16 0.2457
Period -0.08 0.9390
Flu AD (flu AD = 0) 1.19 0.2341
BL rate -7.17 <0.0001
# elderly 2.13 0.0332

Table 4-39 depicts the least square means for the two groups (AD = yes and AD =

no) for each of the six experimental time periods. The least square mean values are also

presented in a graph in figure 4-19.

Table 4-39. Least Square Means for Change in GP Office Visit Rate by Group
(visits/patient).
Secondary Outcome (GP Visits) Least Square Means by AD Group
AD
group Period
1 2 3 4 5 6
0 -0.0182 0 0.4652 0.3882 -0.3346 -0.4341
1 -0.0563 0 0.3813 0.79 -0.0201 0.0069










Table 4-40 depicts the unadjusted means and standard deviations for the two

groups (AD = yes and AD = no) for each of the six experimental time periods. The

unadjusted mean values are also presented in a graph in figure 4-20.


Secondary Outcome:
Change in GP Office Visit Rates

o. 1

.2 0.5



o -0.5

Time Period

-- Intervention = no -- Intervention = yes


Figure 4-19. Least Square Means for Change in GP Office Visit Rates by Group.

Table 4-40. Unadjusted Means and Standard Deviations for Change in GP Office Visit
Rate by Group (visits/patient).
Period AD group
0 1
Mean Std Dev Mean Std Dev
1 0.0057 0.5235 -0.0026 0.5586
2 0.0000 0.0000 0.0000 0.0000
3 0.5282 0.9720 0.3191 0.9419
4 0.2597 0.8025 0.6388 1.3687
5 -0.2269 0.7578 -0.0570 1.4364
6 -0.2742 0.7616 -0.0290 1.7479

Within group (longitudinal) results

The within group model was the same as the between group model in figure 4-18

except that the AD group variable is replaced by a prepost variable which measures

within group differences between change in GP office visit rates pre-intervention and

post-intervention. The model is run two times; once including only the intervention

group and once including only the control group. The z statistic and the associated










significance level (p-value) of each variable are listed in table 4-41 for the intervention

group and table 4-42 for the control group.


Secondary Outcome:
Change in GP Office Visit Rates (unadjusted)

0.8
0.6 -
S0.4
,M. 0.2
S0.0
_0.2
-0.4
Time Period

Intervention = no --- Intervention = yes


Figure 4-20. Unadjusted Means for Change in GP Office Visit Rates by Group.

Table 4-41. Secondary GP Office Visit Outcome Model Results (AD = yes).
Secondary Outcome Model Results for All Periods (Intervention Group)
(Change in GP Office Visit Rates)
Effect Z p-value
PS -1.56 0.1199
Period -10.95 <0.0001
Prepost -17.54 <0.0001
flu AD (flu AD = 0) -2.41 0.0159
BL rate 0.10 0.9187
# elderly 0.17 0.8680

The within group effect of the intervention is interpreted from the values of the z

statistic and associated p-value of the prepost variable. For the intervention and control

groups, z statistics (p-values) of -17.54 (<0.0001) and -20.21 (<0.0001) respectively

indicates that the within group effect is statistically significant for both groups at the

alpha = 0.05 level. The significant results for the longitudinal prepost effect is similar

between the control and intervention groups as indicated in figure 4-20 and also indicated

in the negative values of the z statistics for both groups.









Table 4-42. Secondary GP Office Visit Outcome Model Results (AD = no).
Secondary Outcome Model Results for All Periods (Control Group)
(Change in GP Office Visit Rates)
Effect Z p-value
PS -2.60 0.0093
Period -19.91 <0.0001
Prepost -20.21 <0.0001
flu AD (flu AD = 0) -2.62 0.0089
BL rate -0.99 0.3212
# elderly -2.73 0.0064

Rheumatologist and GI Specialist Visit Rates

Model development

The analysis of the secondary outcome, the effect of the OA AD intervention on

rheumatologist and GI specialist office visit rates was carried out using the same methods

as the primary outcome analysis with the data for rheumatologist and GI specialist office

visit rates substituted for the COX-2 utilization data (figure 4-21).

Between group results

The z statistic and the significance level (p-value) of each variable from the

secondary specialist office visit outcome model (figure 4-21) are listed in table 4-43.

Y = lo + Pl(Xl) + P2(X2) + P3(X3) + 4(X4) + 5(X5) + (X6)

Where;
Y = change in specialist visit rates (periods 3 to 6 (post-intervention)),
X1 = physician participation in the intervention (0 = no, 1 = yes),
X2 = PS (range from 0 to 1),
X3 = experimental time period (period = 3,4,5,6),
X4 = physician participation in the influenza AD service (0 = no, 1 = yes),
X5 = physician baseline specialist visit rate (visits / patient, (period = 2)),
X6 = number of patients in the GP's practice >65 years old


Figure 4-21. Secondary Outcome Model for Specialist Office Visits









The between group effect of the intervention is interpreted from the z statistics and

associated p-value of the AD variable. The z statistic and p-value of 1.44 and 0.1498

respectively indicate that the effect is not significant at the alpha = 0.05 level.

Table 4-43. Secondary Specialist Office Visit Model Results (Periods = 3,4,5,6).
Secondary Outcome Model Results for Post-intervention Periods 3 to 6
(Change in Specialist Office Visit Rates)
Effect Z p-value
AD (ad = 0) 1.44 0.1498
PS -5.98 <0.0001
period -0.04 0.9700
flu AD (flu AD = 0) -5.43 <0.0001
BL rate -23.22 <0.0001
# elderly -3.01 0.0026

The model (figure 4-21) was used to determine intervention effects on each post

intervention time period. Only the period from 181 to 270 days (period five) following

the intervention showed significant difference between groups at the alpha = 0.05 level.

The z-statistic and p-value associated with the intervention effect are 2.10 and 0.0356

respectively (95% CI (0.0001, 0.0022)). In this case, where the analysis only includes

one time period, the interpretation of the coefficient estimate is similar to traditional

GLM methods. That is, the coefficient estimate of 0.0012 (AD = no) is interpreted as the

non-intervention group having measures of average change rate 0.0012 greater

visits/patient/GP than the intervention group.

The model in figure 4-21 was used to determine between group differences in the

pre-intervention time periods (period = 1, 2). The z statistic and associated significance

level of each variable is listed in table 4-44. The results are interpreted in the same

manner as the post-intervention results. The z statistic and p-value for the AD variable

are -1.29 and 0.1976 respectively. The p-value indicates that the groups are not

significantly different in the pre-intervention periods at the alpha = 0.05 level.









Table 4-44. Secondary Specialist Office Visit Outcome Model Results (Periods = 1,2).
Secondary Outcome Model Results for Post-intervention Periods 1 and 2
(Change in Specialist Office Visit Rates)
Effect Z p-value
AD (ad = 0) -1.29 0.1976
PS -4.71 <0.0001
period -0.90 0.3670
flu AD (flu AD = 0) -3.86 0.0001
BL rate -9.21 <0.0001
# elderly -3.51 0.0004

Table 4-45 depicts the least square means for the two groups (AD = yes and AD =

no) for each of the six experimental time periods. The least square mean values are also

presented in a graph in figure 4-22.

Table 4-46 depicts the unadjusted means and standard deviations for the two

groups (AD = yes and AD = no) for each of the six experimental time periods. The

unadjusted mean values are also presented in a graph in figure 4-23.

Secondary Outcome (Specialist Visits) Least Square Means by AD Group
AD group period
1 2 3 4 5 6
0 0.0005 0 0.0007 0.0005 0.0011 0.0001
1 0.0016 0 -0.0002 0.0003 -0.0001 0.0003

Table 4-45. Least Square Means for Change in Specialist Office Visit Rate by Group
(visits/patient).

Within group (longitudinal) results

The within group model was the same as the between group model in figure 4-21

except that the AD group variable is replaced by a prepost variable which measures

within group differences between change in specialist office visit rates pre-intervention

and post-intervention. The model is run two times; once including only the intervention

group and once including only the control group. The z statistic and the associated

significance level (p-value) of each variable are listed in table 4-47 for the intervention

group and table 4-48 for the control group.











Secondary Outcome:
Change in Specialist Office Visit Rates (adjusted)


0.002
0.0015
0.001
0.0005
0
-0.0005


Time Period


-*--Intervention = no -- Intervention = yes


Figure 4-22. Least Square Means for Change in Specialist Office Visit Rates by Group.


Period


AD group


Mean
0.0008
0.0000
0.0005
0.0008
0.0007
0.0002


Std Dev
0.0089
0.0000
0.0077
0.0086
0.0093
0.0083


Mean
-0.0002
0.0000
-0.0011
-0.0012
-0.0013
-0.0006


Std Dev
0.0087
0.0000
0.0089
0.0078
0.0079
0.0082


Table 4-46. Unadjusted Means and Standard Deviations for Change in Specialist Office
Visit Rate by Group (visits/patient).


Secondary Outcome:
Change in Specialist Office Visit Rates (unadjusted)


0.0010
0.0005
0.0000
-0.0005
-0.0010
-0.0015


Time Period


-- Intervention = no -- Intervention = yes


Figure 4-23. Unadjusted Means for Change in Specialist Office Visit Rates by Group.


V









Table 4-47. Secondary Specialist Office Visit Outcome Model Results (AD = yes).
Secondary Outcome Model Results for All Periods (Intervention Group)
(Change in Specialist Office Visit Rates)
Effect Z p-value
PS -6.45 <0.0001
period 0.87 0.3857
prepost 1.94 0.0519
flu AD (flu AD = 0) -6.29 <0.0001
BL rate -17.54 <0.0001
# elderly -4.56 <0.0001

Table 4-48. Secondary Specialist Office Visit Outcome Model Results (AD = no).
Secondary Outcome Model Results for All Periods (Control Group)
(Change in Specialist Office Visit Rates)
Effect Z p-value
PS -4.24 <0.0001
period -0.87 0.3870
prepost -0.70 0.4811
flu AD (flu AD = 0) -3.25 0.0012
BL rate -14.67 <0.0001
# elderly -2.01 0.0444

The within group effect of the intervention is interpreted from the values of the z

statistic and associated p-value of the prepost variable. For the intervention and control

groups, z statistics (p-values) of 1.94 (0.0519) and -0.70 (0.4811) respectively indicates

that the within group effect is not statistically significant for both groups at the alpha =

0.05 level. The results for the longitudinal prepost effect are similar between the control

and intervention groups as indicated in figure 4-23.

Hospitalization Rates Due to GI Complications

Model development

The analysis of the secondary outcome, the effect of the OA AD intervention on

hospitalization rates was carried out using the same methods as the primary outcome

analysis with the data for hospital length of stay rates substituted for the COX-2

utilization data (figure 4-24).










Y = Po + Pl(Xl) + P2(X2) + P3(X3) + 4(X4) + 5(X5) + P6(X6)

Where;
Y = change in hospital utilization rate (periods 3 to 6 (post-intervention)),
Xi = physician participation in the intervention (0 = no, 1 = yes),
X2 = PS (range from 0 to 1),
X3 = experimental time period (period = 3,4,5,6),
X4 = physician participation in the influenza AD service (0 = no, 1 = yes),
X5 = physician baseline hospital LOS rate (LOS / patient, (period = 2)),
X6 = number of patients in the GP's practice >65 years old


Figure 4-24. Secondary Outcome Model for Hospital Length of Stay

Between group results

The z statistic and the significance level (p-value) of each variable from the

secondary hospitalization length of stay outcome model (figure 4-24) are listed in table 4-

49.

Table 4-49. Secondary Hospital Length of Stay Model Results (Periods = 3,4,5,6).
Secondary Outcome Model Results for Post-intervention Periods 3 to 6
(Change in Hospital Length of Stay)
Effect Z p-value
AD (AD = 0) 0.33 0.7389
PS 1.48 0.1396
Period 1.15 0.2500
flu AD (flu AD = 0) 1.13 0.2568
Flu AD*quintile (flu AD = 0) -0.94 0.3468
BL rate -15.58 <0.0001
los rate -2.36 0.0183

The between group effect of the intervention is interpreted from the z statistics and

associated p-value of the AD variable. The z statistic and p-value of 0.33 and 0.7389

respectively indicate that the effect is not significant at the alpha = 0.05 level.

The model (figure 4-24) was used to determine intervention effects on each post

intervention time period. Only the period from 181 to 270 days (period five) following

the intervention showed significant difference between groups at the alpha = 0.05 level.









The z-statistic and p-value associated with the intervention effect are 2.49 and 0.0128

respectively (95% CI (1.1093, 3.4627)). In this case, where the analysis only includes

one time period, the interpretation of the coefficient estimate is similar to traditional

GLM methods. That is, the coefficient estimate of 2.2860 (AD = no) is interpreted as the

non-intervention group having measures of average change rate 2.2860 greater

visits/patient/GP than the intervention group.

The model in figure 4-18 was used to determine between group differences in the

pre-intervention time periods (period = 1, 2). The z statistic and associated significance

level of each variable is listed in table 4-50. The results are interpreted in the same

manner as the post-intervention results. The z statistic and p-value for the AD variable

are 1.58 and 0.1152 respectively. The p-value indicates that the groups are not

significantly different in the pre-intervention periods at the alpha = 0.05 level.

Table 4-50. Secondary Hospital Length of Stay Outcome Model Results (Periods = 1,2).
Secondary Outcome Model Results for Post-intervention Periods 1 and 2
(Change in Hospital Length of Stay)
Effect z p-value
AD (AD = 0) 1.58 0.1152
quintile 0.56 0.5751
period -0.67 0.5014
flu AD (flu AD = 0) -0.10 0.9217
flu AD*quintile (flu AD = 0) 0.72 0.4735
BL rate -10.32 <0.0001
los rate -2.84 0.0044

Table 4-51 depicts the least square means for the two groups (AD = yes and AD =

no) for each of the six experimental time periods. The least square mean values are also

presented in a graph in figure 4-25.









Table 4-52 depicts the unadjusted means and standard deviations for the two

groups (AD = yes and AD = no) for each of the six experimental time periods. The

unadjusted mean values are also presented in a graph in figure 4-26.

Table 4-51. Least Square Means for Change in Hospital Length of Stay Rates by Group
(days/patient).
Secondary Outcome (Hospital LOS) Least Square Means by AD Group
AD group period


1
0.6168
-0.3396


3 4 5 6
1.0337 0.8165 2.0962 1.7457
0.2072 1.0211 -0.1898 3.6511


Secondary Outcome:
Change in Hospital LOS Rates (adjusted)


Time Period


--- Intervention = no Ienterntion = yes


Figure 4-25. Least Square Means for Change in Hospital Length of Stay Rates by Group.

Table 4-52. Unadjusted Means and Standard Deviations for Change in Hospital Length
of Stay Rates by Group (days/patient).


Period


AD group


Mean
0.7409
0.0000
1.3957
1.0678
1.5773
0.5662


Std Dev
8.6598
0.0000
9.9291
8.4797
10.2326
8.8979


Mean
-0.2482
0.0000
0.6193
1.0061
0.5130
5.5624


Std Dev
9.9641
0.0000
9.7833
14.5320
8.3747
51.6778










Within group (longitudinal) results

The within group model was the same as the between group model in figure 4-24

except that the AD group variable is replaced by a prepost variable which measures

within group differences between change in specialist office visit rates pre-intervention

and post-intervention. The model is run two times; once including only the intervention

group and once including only the control group. The z statistic and the associated

significance level (p-value) of each variable are listed in table 4-53 for the intervention

group and table 4-54 for the control group.


Secondary Outcome:
Change in Hospital LOS Rates (unadjusted)

6.0
5.0
S4.0
S3.0
P 2.0
1.0
0.0
-1.0 1 2 3 4 6
Time Period

Intervention = no Intervention = yes


Figure 4-26. Unadjusted Means for Change in Hospital Length of Stay Rates by Group.

Table 4-53. Secondary Hospital Length of Stay Outcome Model Results (AD = yes).
Secondary Outcome Model Results for All Periods (Intervention Group)
(Change in Hospital Length of Stay)
Effect Z p-value
PS 0.76 0.4489
period 1.44 0.1491
prepost 0.96 0.3388
flu AD (flu AD = 0) 0.00 0.9976
BL rate -13.11 <0.0001
# elderly -1.56 0.1193









Table 4-54. Secondary Hospital Length of Stay Outcome Model Results (AD = no).
Secondary Outcome Model Results for All Periods (Control Group)
(Change in Hospital Length of Stay)
Effect Z p-value
PS 3.13 0.0018
period -1.21 0.2256
prepost -2.08 0.0375
flu AD (flu AD = 0) 2.99 0.0028
BL rate -14.40 <0.0001
# elderly 1.87 0.0614

The within group effect of the intervention is interpreted from the values of the z

statistic and associated p-value of the prepost variable. For the intervention and control

groups, z statistics (p-values) of 0.96 (0.3388) and -2.08 (0.0375) respectively indicates

that the within group effect is not statistically significant for the intervention group and is

statistically significant for the control group at the alpha = 0.05 level.

Death Rates Due to GI Complications

Model development

The analysis of the secondary outcome, the effect of the OA AD intervention on

death rates due to GI complications was carried out using the same methods as the

primary outcome analysis with the data for hospital length of stay rates substituted for the

COX-2 utilization data (figure 4-27).

Ln Y= Po + 0P(X) + p2(X2) + P3(X3)+ 04(X4) + 5(X) + p6(X6)

Where;
Y = death rates (periods 3 to 6 (post-intervention)),
X1 = physician participation in the intervention (0 = no, 1 = yes),
X2 = PS (range from 0 to 1),
X3 = experimental time period (period = 3,4,5,6),
X4 = physician participation in the influenza AD service (0 = no, 1 = yes),
X5 = physician baseline hospital LOS rate (LOS / patient, (period = 2)),
X6 = number of patients in the GP's practice >65 years old


Figure 4-27. Secondary Outcome Model for Deaths Due to GI Complications









Special consideration had to be given to the distribution of the data since the

number of deaths per GP per study period was quite low. There were 1984 data points

analyzed (496 GPs with six measures each) and in all cases except four the number of

deaths per physician was equal to zero or one. The four other cases all contained two

deaths (three of the four occurred in the control group). A dichotomous variable

representing death/no-death for each period measurement was developed and since the

majority of the period measurements represented no-death (142 with death and 2834

without death) a negative binomial distribution was used in the analysis model. The total

number of deaths per group per period was less than five in a number of cases. For this

reason, the number of study periods was reduced to three by combining periods one and

two, three and four, and five and six.

Between and within group results

None of the between or within group analyses of death rates showed significance at

the alpha = 0.05 level. The z statistics (p-values) associated with the pre-intervention and

post-intervention between group analyses were -0.63 (0.5317) and 0.81 (0.4203)

respectively and the z statistics (p-values) associated with the within group analyses for

the intervention and control groups were -0.36 (0.7189) and -0.03 (0.9742) respectively.














CHAPTER 5
DISCUSSION

The Academic Detailing Program in Nova Scotia

An analysis of the effect of the OA AD intervention on prescribing behavior should

be taken in context of the qualifications of the detailers, the dynamic changes over the

course of the intervention and the policy options available to the decision makers. A

description of these three topics should add to the determination of generalizability of the

intervention to other jurisdictions.

Qualifications of the Detailers

The OA AD intervention employed three detailers; two pharmacists and one

registered nurse. One pharmacist worked within the province's capitol district and the

other pharmacist and the registered nurse divided the rural area of the province in two.

The nurse detailed GPs in the region that she was native to and as such was very familiar

with local customs and practices.

All three of the detailers were trained in techniques associated with successful AD

programs. These techniques are described in greater detail in appendix A. The

intervention was designed to take approximately twenty minutes to present with

opportunity for the GP to interact with the detailer over the course of the presentation.

Changes Which Occurred Over the Period of the Intervention (History Effects)

The OA AD intervention was delivered from April, 2002 to November, 2002. The

analysis timeframe for our study spanned from October, 2001 (six months before the

intervention commenced) to November, 2003 (one year after the intervention concluded).









Between October, 2001 and May, 2003 two warnings regarding the safety of COX-

2 inhibitors were issued by Health Canada.38 The first warning in April, 2002 concerned

the results of the VIGOR trial8 and warned of increased cardiac risk associated with

rofecoxib use and the second warning in May, 2002 concerned the results of the CLASS

trial7 and warned of the GI risk associated with celecoxib use particularly in combination

with low dose ASA therapy. Analysis of the VIGOR and CLASS trials was included in

the OA AD intervention (appendix A). In December, 2004 rofecoxib was withdrawn

from the market.38 The withdrawal occurred after the post intervention study period.

The Nova Scotia pharmacare plan issued a policy change with respect to the benefit

status of a combination product containing diclofenac and misoprostol.39 The product

was changed to open benefit status in September, 2002. Announcement of the change

was disseminated equally to all GPs in the province. The benefit status of rabeprazole

was changed to open benefit in June, 200339 after the end of the post intervention analysis

period.

Policy Options Available to Decision Makers

Our study examined the effect of the fourth message of the OA AD intervention

which addressed pharmacotherapy of OA. The other three messages contained in the

intervention were intended to change physician behavior in terms of prescribing non-

pharmacologic treatment for OA and research into the effectiveness of these messages is

warranted. The OA AD intervention lacked a follow-up visit which is a limitation of the

intervention design.13 Five options available to policy decision makers which could

address this shortcoming without the costs associated with a one-on-one follow-up visit

are; the distribution of educational material, educational meetings, audit and feedback,









reminders, and changes in benefit schedules.17' 40 While some of the instruments have not

shown significant effects on their own the combination with AD can be effective.14,17

Distribution of educational material

The distribution of educational material involves the dissemination of media

(written or video) to the GPs with information reinforcing the messages of the OA AD

intervention. It is the decision of the GP to review the message or not. It is relatively

low cost and has been shown to have a modest but short-lived effect.17 The message

contained in this medium should be limited to the intervention messages in such a way

that does not require "active" learning or interaction with an educator.

Educational meetings

Educational meetings involve meeting in groups to review the messages from the

intervention. This instrument can be more complex in nature than the distribution of

written material but they are still limited by the inability of the participant to interact with

the instructor on a one-to-one basis. Used as a single intervention this instrument has

shown little17 to no effect14 on improving pharmaceutical use.

Audit and feedback

Audit and feedback is an instrument that involves the analysis of the performance

of the provider and/or the provider's peers over a period of time. The instrument is costly

to implement as it involves a significant amount of data analysis to produce the audits.

Audit and feedback can address some complex issues through the use of the analysis and

comparison with peers. Studies using audit and feedback as a single intervention have

shown a modest effect.17