Background: The role of knowledge in postgraduate medical education has often been discussed. However, recent insights from cognitive psychology and the study of deliberate practice recognize that expert problem solving requires a well-organized knowledge database. This implies that postgraduate assessment should include knowledge testing. Longitudinal assessment, like progress testing, seems a promising approach for postgraduate progress knowledge assessment.

Aims: To evaluate the validity and reliability of a national progress test in postgraduate Obstetrics and Gynaecology training.

Methods: Data of 10 years of postgraduate progress testing were analyzed on reliability with Cronbach's alpha and on construct validity using one-way ANOVA with a post hoc Scheffe test.

Results: Average reliability with true-false questions was 0.50, which is moderate at best. After the introduction of multiple-choice questions average reliability improved to 0.65. Construct validity or discriminative power could only be demonstrated with some certainty between training year 1 and training year 2 and higher training years.

Conclusion: Validity and reliability of the current progress test in postgraduate Obstetrics and Gynaecology training is unsatisfactory. Suggestions for improvement of both test construct and test content are provided.