The Agency for Healthcare Research and Quality (AHRQ) recently designed the Patient Safety Indicators (PSIs) to detect safety-related adverse events. The National Quality Forum has endorsed several of these ICD-9-CM based indicators as quality of care measures. We examined the positive predictive value (PPV) of three surgical PSIs: 1) postoperative pulmonary embolus/deep vein thrombosis (pPE/DVT), 2) iatrogenic pneumothorax (iPTX) and 3) accidental puncture/laceration (APL).

We applied the AHRQ PSI software (v.3.1a) to Veterans Health Administration 2003-07 administrative data to identify (flag) cases suspected of having a pPE/DVT, iPTX or APL. Two trained chart-abstractors reviewed a sample of 336 flagged medical records (112 records/PSI) using a standardized instrument. Inter-rater reliability was assessed.

Of 2,343,088 admissions, 6,080 were flagged for pPE/DVT (0.26%), 1,402 for iPTX (0.06%) and 7,203 for APL (0.31%). For pPE/DVT, PPV was 44% (95% CI, 34-53%); 21% of cases represented inaccurate coding (e.g. arterial not venous thrombosis), and 35% featured thromboembolism present on admission or preoperatively. For iPTX, PPV was 80% (95% CI, 73-88%); 13% represented inaccurate coding (e.g. spontaneous pneumothorax) and 7% were pneumothoraces present on admission. For APL, PPV was 85% (95% CI, 78-91%); 10% of cases represented coding inaccuracies and 5% indicated injuries present on admission. However, 27% of true APLs were minor injuries requiring no surgical repair (e.g. small serosal bowel tear). Inter-rater reliability was >90%.

Until adequate coding revisions are implemented, PSIs should only be used for screening and case-finding. Their utility as quality measures may need to be re-assessed.