Code for Improving Video Caption Accuracy with LLMs

DOI

As part of the IKILeUS project at the University of Stuttgart, research was conducted to explore how Large Language Models (LLMs) can enhance the accuracy and contextual relevance of automatic speech recognition (ASR)-generated captions. While ASR tools provide a foundation for accessibility, they often produce grammatical errors, misinterpret homophones, and struggle with domain-specific terminology. To address these challenges, experiments were conducted using LLMs such as GPT-3.5 and Llama2-13B to refine and correct captioning errors. The models were evaluated using standard NLP metrics such as Word Error Rate (WER), BLEU, and ROUGE scores, demonstrating notable improvements in caption accuracy. The findings suggest that LLMs can effectively enhance the readability, coherence, and precision of automatically generated captions, offering a promising direction for improving video accessibility for the Deaf and Hard of Hearing (DHH) community.

Identifier
DOI https://doi.org/10.18419/DARUS-4776
Metadata Access https://darus.uni-stuttgart.de/oai?verb=GetRecord&metadataPrefix=oai_datacite&identifier=doi:10.18419/DARUS-4776
Provenance
Creator Fathallah, Nadeen (ORCID: 0000-0001-7921-034X)
Publisher DaRUS
Contributor Fathallah, Nadeen; High Performance Computing Center (HLRS)
Publication Year 2025
Funding Reference German Federal Ministry of Education and Research (BMBF) 16DHBKI041
Rights MIT License; info:eu-repo/semantics/openAccess; https://spdx.org/licenses/MIT.html
OpenAccess true
Contact Fathallah, Nadeen (University of Stuttgart)
Representation
Resource Type Automatic speech recognition (ASR) transcriptions, large language model (LLM)-corrected subtitle datasets, word error rate (WER) evaluation data, NLP-processed text outputs, captioning quality metrics (BLEU, ROUGE scores).; Dataset
Format application/x-ipynb+json; text/plain; charset=US-ASCII; text/markdown
Size 7177; 19828; 12152; 7949; 4902; 1073; 4489; 7626; 3915; 6317; 85907
Version 1.0
Discipline Other