This study focused on the reliability of the DSM-III inventory of psychiatric symptoms in representative general population samples in three Brazilian cities. Reliability was assessed through two different designs: inter-rater reliability and internal consistency. Diagnosis of lifetime (k = 0.46) and same-year generalized anxiety (k = 1.00), lifetime depression (k = 0.77), and lifetime alcohol abuse and dependence (k = 1.00) was consistently reliable in the two methods. Lifetime diagnosis of agoraphobia (k = 1.00), simple phobia (k = 0.77), non-schizophrenic psychosis (k = 1.00), and psychological factors affecting physical health (1.00) showed excellent reliability as measured by the kappa coefficient. The main reliability problem in general population studies is the low prevalence of certain diagnoses, resulting in small variability in positive answers and hindering kappa estimation. Therefore it was only possible to examine 11 of 39 diagnoses in the inventory. We recommend test and re-test methods and a short time interval between interviews to decrease the errors due to such variations.