Title Finetuning a Reasoning Model with GRPO for Superior Passport Data Extraction ✅

Fine-Tuning a Reasoning Model with GRPO for Passport Data Extraction Introduction Fine-tuning a language model isn’t just about feeding it data and hoping for the best. If you’re extracting structured data—like passport details—you need a model that reasons through the problem, not one that just memorizes patterns. That’s where Group Relative Policy Optimization (GRPO) comes in. In this post, we’ll walk through fine-tuning a reasoning model for passport data extraction using GRPO. We’ll start with Supervised Fine-Tuning (SFT) and then refine it using reinforcement learning (RL) to improve accuracy and reasoning. We’ll use: ● Base Model: Qwen/Qwen2.5-1.5B-Instruct ● Dataset: Custom Passport EN dataset ● Training Method: SFT + GRPO Why GRPO? Supervised Fine-Tuning (SFT) is effective for training a baseline model, but it struggles with generalization. When extracting structured data, slight variations in input format can lead to errors. Standard SFT lacks the adaptive reasoning needed to handle these cases effectively. This is where Group Relative Policy Optimization (GRPO) improves the model. GRPO, introduced in the DeepSeekMath paper, is a reinforcement learning (RL) post-training technique designed to enhance reasoning skills in large language models (LLMs). Unlike traditional heuristic-based search methods, GRPO relies solely on RL for optimization, helping the model generalize better to unseen input variations. GRPO has been used in DeepSeek-R1, and its training approach appears similar to the methods used in OpenAI o1 and o3 models, though exact details are unconfirmed. The Hugging Face Science team is working to reproduce the DeepSeek-R1 training process in their Open-R1 project, which is worth exploring for more insights. We’ll implement GRPO using the TRL library (Transformer Reinforcement Learning) library and focus on improving structured data extraction from passports.
What is the GRPO Reinforcement Learning Algorithm? To understand GRPO, let’s break it down with an example before diving into the technical details. Intuition Behind GRPO GRPO helps a model learn by comparing different actions in groups and making controlled updates. Instead of updating the model after every single observation, it collects multiple observations before adjusting its strategy—similar to mini-batch gradient updates in deep learning. Example: A Robot Navigating a Maze Imagine you’re training a robot to navigate a maze and reach a goal. The robot can choose between three paths (A, B, and C). 1. Sampling Different Paths: The robot tries out each path multiple times and records the results: ○ Path A: Reaches the goal 2 out of 3 times. ○ Path B: Reaches the goal 1 out of 3 times. ○ Path C: Reaches the goal 3 out of 3 times. 2. Evaluating Performance: It calculates the success rate: ○ Path A → 66.67% success ○ Path B → 33.33% success ○ Path C → 100% success 3. Comparing Paths: It identifies Path C as the best option but doesn’t ignore the other paths completely. 4. Adjusting Strategy: The robot increases the probability of choosing Path C, but it still tries A and B occasionally to avoid overfitting to one solution. 5. Controlled Updates: Instead of jumping to a 100% preference for Path C, it gradually shifts probabilities while maintaining exploration. Applying GRPO to Passport Data Extraction Just like the robot learns from multiple trials, GRPO enables an LLM to refine its structured data extraction ability by: ● Evaluating different extraction strategies for variations in passport formats. ● Adjusting model predictions based on relative success rather than absolute correctness. ● Ensuring controlled updates, allowing for better generalization.
Training Creating the Passport Dataset To build a dataset for passport information extraction, we needed a large and diverse collection of passport samples. We designed a structured framework that systematically extracts key details from passports while ensuring accuracy and consistency. However, raw passport images and text require transformation into a structured format that machine learning models can efficiently process. To achieve this, we developed a token-based representation system, effectively encoding the essential details in a machine-readable format. Here's how we structured the dataset: 1. Field Extraction: We systematically extract essential passport information, including: ○ Passport Number ○ Full Name ○ Gender ○ Nationality ○ Date of Birth ○ Place of Birth ○ Date of Issue ○ Date of Expiry ○ Place of Issue (if available) ○ Machine Readable Zone (MRZ) 2. Token-Based Representation: Instead of relying solely on free-text extraction, we encode passport data into structured tokens, ensuring uniformity across all entries. This transformation makes it easier for models to learn patterns and generalize across different passports. ○ Entity Tokens: Each extracted field is wrapped with a special token (e.g., 200858064), making it easier for models to identify and process key details. ○ Date Formatting: Dates are standardized to a consistent format (DD-MM-YYYY) to maintain consistency across all samples. ○ MRZ Encoding: The Machine Readable Zone (MRZ) is preserved as a key feature, allowing the model to cross-validate extracted details. 3. Dataset Structure: Each passport entry in the dataset follows this structured format: