The Impact of Question Framing on the Performance of Automatic Occupation Coding

Occupational data play a vital role in research, official statistics, and policymaking, yet their collection and accurate classification remain a challenge. This study investigates the effects of occupational question wording on data variability and the performance of automatic coding tools. We conducted and replicated a split-ballot survey experiment in Germany using two common occupational question formats: one focusing on 'job title' (Berufsbezeichnung) and another on 'occupational tasks' (berufliche Tätigkeit). Our analysis reveals that automatic coding tools, such as CASCOT and OccuCoDe, exhibit sensitivity to the form and origin of the data. Specifically, these tools were more efficient when coding responses to the job title question format compared with the occupational task format, suggesting a potential way to improve the respective questions for many German surveys. In a subsequent 'detailed tasks and duties' question, providing a guiding example prompted respondents to give longer answers without broadening the range of unique words they used. These findings highlight the importance of harmonising survey questions and of ensuring that automatic coding tools are robust to differences in question wording. We emphasise the need for further research to optimise question design and coding tools for greater accuracy and applicability in occupational data collection.