Invited Talk 6: LLM to Annotate Subject Metadata

Recent studies have shown that in-context learning with Large language models (LLMs) has matched or even exceeded human performance for various automated annotation tasks. In this research, we explored a large language model - OpenAI's GPT-3.5 to enrich metadata from an Australian national dataset catalogue, specifically to annotate subject metadata with the Australian and New Zealand Standard Research Classification (ANZSRC) taxonomy. In line with recent related studies using the in-context learning framework, we crafted prompts incorporating task instructions and example metadata records with subject metadata annotated by humans and then engaged GPT-3.5 to produce predictions. Our models displayed outstanding results, achieving an accuracy of over 90% in multiple research divisions. There are a few research divisions where the models didn't perform as well compared to human annotation, however, a further analysis shows the machined annotated subject headings can complement human annotation. Overall, our research demonstrates the potential of GPT-3.5 as a powerful tool for automating the annotation of subject metadata with limited training data.