TrorYongOCR: a Novel Encoder-Decoder Model for Scene Text Recognition

Meet Your Tiny OCR Model

Abstract

TrorYongOCR is a tiny encoder-decoder model for Scene Text Recognition task. It prepends the encoding of image patches to the “begin of sequence” token to condition next character token generation. Using LLM analogy, patch encodings can be simply seen as prefill prompt embeddings. The single text decoder block of TrorYongOCR generates character tokens based on the prefill prompt in an autoregressive manner without cross-attention mechanism. TrorYongOCR can process input images of arbitrary aspect ratio. The current pre-trained weight supports 2 languages: Khmer and English. Testing on challenging WildKhmerST dataset (Nom et al. 2025), TrorYongOCR (5.5M parameters) achieves a cer of 42.97% while fine-tuned DeepSeek-OCR (Wei, Sun, and Li 2025) (3B parameters) achieves a cer of 67.66%. So, TrorYongOCR has less parameters, performs relatively better for Khmer text and faster for inference. It is suitable for local or edge-device usage such as License Plate Recognition application. TrorYongOCR is deployed on Huggingface Space here for demonstration.

Using Image Encoding as a Prefilll Prompt

TrorYongOCR is designed as the following: given \(L\) transformer blocks

• \(L-1\) are encoding blocks that encode a given image
• the last block is a single decoding block without cross-attention mechanism
• each transformer is implemented with exclusive self-attention (Zhai 2026) style and SwiGLU MLP

For the single decoding block,

• the latent state of an image (the output of the last encoder block) is concatenated with the input character embedding (i.e., token embedding including bos token) to create context vector, i.e., key and value vectors (think of it like a prefill prompt)
• the query vector is simply the input character embedding.

TrorYongOCR architecture overview can be found in Figure 1.

Figure 1: TrorYongOCR architecture overview. The input image is transformed into patch embedding. The embedding is passed through \(L-1\) encoder blocks to generate image encoding (latent state). The last single decoder block takes character embeddings (i.e., token embedding) as the query vector and the concatenation of image encoding with character embedding as context vectors. The causal masked attention mechanism in this single decoder block generates the latent state of character embedding. The last linear layer projects the root-mean-squared normalized latent state into logits over character set.

Batching Images of Different Aspect Ratio Using Prefix-Padding

Images are treated as sequences of patches. When batching images, the sequences with smaller length are prepended by padding patches to get a fixed batch shapes (see Figure 2). So, TrorYongOCR can be trained on images at their ‘native’ aspect ratio. This is different from classic method that resizes any images to fixed resolution (so fixed aspect ratio) or does suffix-padding. Meanwhile, a recent method from Gemma 4 (Team et al. 2026) enables Gemma 4 to work with input image of arbitrary aspect ratio by fixing the number of patches and resizes any images in a way that preserves the aspect ratio and generates the exact number of patches when patchifing (See Algorithm 1 of (Team et al. 2026). However, I believe that this method makes OCR task more challenging for AI model. Concretely, it removes the explicit dependency between the aspect ratio and the number of patches: any aspect ratio incurs the same number of patches. This dependency is crucial for OCR task because without geometric transformation (e.g., stretching), text region image of higher aspect ratio usually contains longer text.

For TrorYongOCR, prefix-padding helps preserving position consistency between the last patch of image and the begin-of-sequence token. That is, for any images, the last patch of the image is always followed by the begin-of-sequence token before being processed by the single text decoder. Prefix-padding requires further consideration when using rotary positional embedding because the first patch of any images must receive no rotation. I can rely on pad mask to ensure that image patches receive correct rotation (i.e., padding patches all stay in position 1 as shown in Figure 3). Pad mask is also used in the attention mechanism to avoid image patches attending to padding patches. Prefix-padding also enables regularizing techniques such as patch dropping with variable drop rate, or resolution sampling with preserved aspect ratio too.

Figure 2: Prefix-padding enables images with different aspect ratio to be batched together. Firstly, images are patchified. The shorter sequences are then prepended by padding patches so that each sequence has the same length. Prefix-padding is vital for the architecture of TrorYongOCR where image encoding plays a role similar to prefill prompt in LLMs. It ensures that when concatenating image encoding with character embedding, the last patch of any images is followed by the begin of sequence token.

Figure 3: TrorYongOCR process flow: the batched sequences of patches are projected into embedding space. These patch embeddings are represented inside the dashed rectangle at the bottom. Encoder blocks rely on patch pad mask to avoid image patches attending to padding patches. The image encodings are then concatenated with character embeddings as shown inside the dashed rectangle in the middle. The single text decoder block processes this concatenation based on causal attention mask and pad mask (i.e., patch pad mask and character pad mask). This flow generates latent state for each character embedding shown in the dashed rectangle at the top. This latent state is then root-mean-squared normalized and processed by the final linear layer to output logits over character set. Note that the patch pad embeddings or encodings (which are all zeros in practice) all stay at position 1 (i.e., they receive no-rotation after applying rotary embedding).

Model Configuration

The choice of model configuration can be found as the following. While preserving aspect ratio, the input image is resized to \(min(W, H) = 32\) where \(W\) and \(H\) are width and height of the image respectively. This is to reduce computation cost in the training as images with high resolution and big aspect ratio incur very long sequence of patches. The image patch size is \((8, 4)\) where \(8\) is along the width of input image. The context length for character sequence is up to \(1024\). Transformer configuration is the following: there are \(4\) blocks, each has embedding dimension \(d_{model}=384\) and \(h=6\) heads. In particular, encoding blocks (block \(1\) to \(3\)) have MLP dimension \(d_{MLP}=\frac{4d_{model}}{3}=512\) and the decoding block has \(d_{MLP}=\frac{8d_{model}}{3}=1024\) (see Table 1).

Table 1: Configuration of Transformer Blocks of TrorYongOCR

Layer	\(d_{model}\)	\(h\)	\(d_{MLP}\)	Role
1	384	6	512	Encoder
2	384	6	512	Encoder
3	384	6	512	Encoder
4	384	6	1024	Decoder

Compared to PARSeq

PARSeq(Bautista and Atienza 2022) model is an encoder-decoder model that implements Permuted Language Modeling (Yang et al. 2019): the text decoder uses position embedding as query vector, character embedding (token embedding plus position embedding) as context vector, and the latent state from image encoder as memory for the cross-attention mechanism (see Figure 3 of (Bautista and Atienza 2022)). PARSeq’s encoder aims to extract image features that help character-image alignment in the decoder. It uses fixed resolution \((128, 32)\) (i.e., fixed aspect ratio equals to \(4\)). This is significantly different from TrorYongOCR’s encoder that tries to project image patches into character embedding space. Moreover, TrorYongOCR can process input image of arbitrary aspect ratio.

Compared to DTrOCR

DTrOCR(Fujitake 2024) is a decoder-only model. The image embedding (i.e., patch embedding plus position embedding) is concatenated with the input character embedding to form query vector. [SEP] token is added at the beginning of input character embedding to indicate sequence separation ([SEP] token is equivalent to bos token in TrorYongOCR). Causal self-attention mechanism is then applied to the query vector from layer to layer to generate latent state. The last linear layer projects the latent state of character embedding to logits over vocabulary set (see Figure 2 of (Fujitake 2024)). In contrast, TrorYongOCR concatenates input character embedding with image encoding to form key and value vectors for the single decoder block.

Training Dataset

TrorYongOCR is implemented as a PyPI package and can be installed via

pip install tror-yong-ocr

The pre-trained weight of TrorYongOCR can be found here. It is obtained by pre-training on seanghay/khmer-hanuman-100k and SoyVitou/KhmerSynthetic1M datasets and fine-tuning on Wild Khmer Scene Text dataset (Nom et al. 2025).

KhmerSynthetic1M

KhmerSynthetic1M is a dataset by Mr. Soy Vitou. This dataset contains images in gray monochromatic color palette (black, white, gray, etc.,). The distribution of the number of tokens, i.e. frequency of each number of tokens, is fairly uniform. In particular, the maximum number of tokens is around \(120\). This implies that there are images with aspect ratio largely higher than \(4\).

khmer-hanuman-100k

This dataset by Mr. Yat Seanghay contains images with a variety of background colors and character colors.

WildKhmerST: A Dataset for Khmer Scene Text Detection and Recognition in the Wild

This dataset of Khmer scene text comprises 29,601 annotated text lines from 10,000 unique images. It is highly diverse and challenging, encompassing artistic text, blurred text, low-light conditions, curved text, text in complex backgrounds, and occluded text. Each text line is annotated with polygonal bounding box coordinates and line-level transcriptions, alongside attributes describing background complexity, character appearance, and text style.

To fine-tune TrorYongOCR, I cropped the quadrilateral bounding boxes to get only text images. Then, warp operation is used to transform quadrilateral image into rectangle one.

For the sake of benchmarking, WildKhmerST is processed and split into 3 parts: train, validate, and test. It can be found here

Evaluation

I fine-tuned DeepSeek-OCR, one of the most well-known Vision Language models, on WildKhmerST and compare its performance with TrorYongOCR. DeepSeek-OCR has 3 billions parameters and be can be used for text recognition task (Wei, Sun, and Li 2025). Following the notebook from Unsloth, the fine-tuned model can be found here. It still hallucinates on some images of test split: when starting with a few wrong characters, the fine-tuned model cannot recover back and keep repeating a single character until the maximum number of tokens is met. Ignoring the hallucination instances, the fine-tuned model achieves a cer of 67.66%. This is relatively worst compared to TrorYongOCR who achieves a cer of 42.97%.

Moreover, due to its size, DeepSeek-OCR needs a lot of compute which can slow down its inference speed. TrorYongOCR has only 5.5 millions parameters. So, it enjoys faster inference and is suitable for local or edge-device usage. For instance, TrorYongOCR can be further fine-tuned for License Plate Recognition dataset and integrated into traffic system to enhance traffic security.

References

Bautista, Darwin, and Rowel Atienza. 2022. “Scene Text Recognition with Permuted Autoregressive Sequence Models.” In European Conference on Computer Vision, 178–96. Springer.

Fujitake, Masato. 2024. “Dtrocr: Decoder-Only Transformer for Optical Character Recognition.” In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 8025–35.

Nom, Vannkinh, Saly Keo, Souhail Bakkali, Muhammad Muzzamil Luqman, Mickaël Coustaty, Marçal Rossinyol, and Jean-Marc Ogier. 2025. “WildKhmerST: A Comprehensive Dataset and Benchmark for Khmer Scene Text Detection and Recognition in the Wild.” In International Conference on Document Analysis and Recognition, 351–68. Springer.

Team, Gemma, Sherif El Abd, Vaibhav Aggarwal, Robin Algayres, Alek Andreev, Olivier Bachem, Ian Ballantyne, et al. 2026. “Gemma 4 Technical Report.” https://arxiv.org/abs/2607.02770.

Wei, Haoran, Yaofeng Sun, and Yukun Li. 2025. “DeepSeek-OCR: Contexts Optical Compression.” arXiv Preprint arXiv:2510.18234.

Yang, Zhilin, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. “Xlnet: Generalized Autoregressive Pretraining for Language Understanding.” Advances in Neural Information Processing Systems 32.

Zhai, Shuangfei. 2026. “Exclusive Self Attention.” arXiv Preprint arXiv:2603.09078.