Comparison of language-specific HTR models : “Does the language of the training corpus affect the performance of a handwritten text recognition (HTR) model on crosslingual settings?”

Abstract

This thesis investigates the influence of training language on the performance of handwriting text recognition (HTR) models. Two separate Vision Transformer-based models were trained using datasets in different languages, one with English data (IAM dataset), and another with German data (fhswf/german_handwriting). Both models were evaluated on their native test sets as well as on a cross-lingual test set to assess generalization and linguistic robustness. Quantitative evaluation using Character Error Rate (CER) and Word Error Rate (WER) shows a clear degradation in recognition performance when models are tested on a language different from their training set. This highlights the sensitivity of HTR models to language-specific features, even when based on language-agnostic decoding mechanisms like Connectionist Temporal Classification (CTC). A qualitative error analysis was conducted to illustrate how specific types of language-dependent character sequences contribute to recognition failures. Furthermore, a pipeline for n-gram-based error attribution on character level was implemented to explore whether misrecognitions correlate with language-dominant character patterns. Although the n-gram analysis could not be fully utilized due to insufficient cross-lingual performance, the results were discussed in the Appendix and the implemented tools remain available for future experimentation. The findings underscore the need for either multilingual training strategies or language-specific adaptation in practical HTR systems. The code is publicly available at: https://github.com/Mir0da/HTR-VT_Bachelor The german trained model is available at: https://huggingface.co/Mir0da/HTR-VT-german The english trained model is available at: https://huggingface.co/Mir0da/HTR-VT-english