OBJECTIVE: To evaluate the quality of the methods and reporting of published studies that validate administrative database algorithms for rheumatic disease case ascertainment. METHODS: We systematically searched MEDLINE, Embase, and the reference lists of articles published from 1980 to 2011. We included studies that validated administrative data algorithms for rheumatic disease case ascertainment using medical record or patient-reported diagnoses as the reference standard. Each study was evaluated using published standards for the reporting and quality assessment of diagnostic accuracy, which informed the development of a methodologic framework to help critically appraise and guide research in this area. RESULTS: Twenty-three studies met the inclusion criteria. Administrative database algorithms to identify cases were most frequently validated against diagnoses in medical records (83%). Almost two-thirds of the studies (61%) used diagnosis codes in administrative data to identify potential cases and then reviewed medical records to confirm the diagnoses. The remaining studies did the reverse, identifying patients using a reference standard and then testing algorithms to identify cases in administrative data. Many authors (61%) described the patient population, but few (26%) reported key measures of diagnostic accuracy (sensitivity, specificity, and positive and negative predictive values). Only one-third of studies reported disease prevalence in the validation study sample. CONCLUSION: The methods used in administrative data validation studies of rheumatic diseases are highly variable. Few studies reported key measures of diagnostic accuracy despite their importance for drawing conclusions about the validity of administrative database algorithms. We developed a methodologic framework and recommendations for validation study conduct and reporting.