# GRPO Trainer

[![model badge](https://img.shields.io/badge/All_models-GRPO-blue)](https://huggingface.co/models?other=grpo,trl)

## Overview

TRL supports the GRPO Trainer for training language models, as described in the paper [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://huggingface.co/papers/2402.03300) by [Zhihong Shao](https://huggingface.co/syhia), [Peiyi Wang](https://huggingface.co/peiyiwang89), [Qihao Zhu](https://huggingface.co/zqh11), Runxin Xu, [Junxiao Song](https://huggingface.co/haha-point), Mingchuan Zhang, Y. K. Li, Y. Wu, [Daya Guo](https://huggingface.co/guoday).

The abstract from the paper is the following:

> Mathematical reasoning poses a significant challenge for language models due to its complex and structured nature. In this paper, we introduce DeepSeekMath 7B, which continues pre-training DeepSeek-Coder-Base-v1.5 7B with 120B math-related tokens sourced from Common Crawl, together with natural language and code data. DeepSeekMath 7B has achieved an impressive score of 51.7% on the competition-level MATH benchmark without relying on external toolkits and voting techniques, approaching the performance level of Gemini-Ultra and GPT-4. Self-consistency over 64 samples from DeepSeekMath 7B achieves 60.9% on MATH. The mathematical reasoning capability of DeepSeekMath is attributed to two key factors: First, we harness the significant potential of publicly available web data through a meticulously engineered data selection pipeline. Second, we introduce Group Relative Policy Optimization (GRPO), a variant of Proximal Policy Optimization (PPO), that enhances mathematical reasoning abilities while concurrently optimizing the memory usage of PPO.

This post-training method was contributed by [Quentin Gallouédec](https://huggingface.co/qgallouedec).

## Quick start

This example demonstrates how to train a model using the GRPO method. We train a [Qwen2.5 0.5B Instruct model](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct) with the prompts from the [DeepMath-103K dataset](https://huggingface.co/datasets/trl-lib/DeepMath-103K). You can view the data in the dataset here:

<iframe
  src="https://huggingface.co/datasets/trl-lib/DeepMath-103K/embed/viewer/default/train?row=0"
  frameborder="0"
  width="100%"
  height="560px"
>

Below is the script to train the model.

```python
# train_grpo.py
from datasets import load_dataset
from trl import GRPOTrainer
from trl.rewards import accuracy_reward

dataset = load_dataset("trl-lib/DeepMath-103K", split="train")

trainer = GRPOTrainer(
    model="Qwen/Qwen2.5-0.5B-Instruct",
    reward_funcs=accuracy_reward,
    train_dataset=dataset,
)
trainer.train()
```

Execute the script using the following command:

```bash
accelerate launch train_grpo.py
```

Distributed across 8 GPUs, the training takes approximately 1 day.

![GRPO curves](https://huggingface.co/datasets/trl-lib/documentation-images/resolve/main/grpo_curves.png)

> **Note:** The reward curves above were generated with `Qwen/Qwen2-0.5B-Instruct`. Results with `Qwen/Qwen2.5-0.5B-Instruct` will be qualitatively similar.

## Looking deeper into the GRPO method

GRPO is an online learning algorithm, meaning it improves iteratively by using the data generated by the trained model itself during training. The intuition behind GRPO objective is to maximize the advantage of the generated completions, while ensuring that the model remains close to the reference policy. To understand how GRPO works, it can be broken down into four main steps: **Generating completions**, **computing the advantage**, **estimating the KL divergence**, and **computing the loss**.

![GRPO visual](https://huggingface.co/datasets/trl-lib/documentation-images/resolve/main/grpo_visual.png)

### Generating completions

At each training step, we sample a batch of prompts and generate a set of  \\( G \\) completions for each prompt (denoted as  \\( o_i \\)).

### Computing the advantage

For each of the  \\( G \\) sequences, we compute the reward using a reward model or reward function. To align with the comparative nature of reward models—typically trained on datasets of comparisons between outputs for the same question—the advantage is calculated to reflect these relative comparisons. It is normalized as follows:

$$\hat{A}_{i,t} = \frac{r_i - \text{mean}(\mathbf{r})}{\text{std}(\mathbf{r})}$$

This approach gives the method its name: **Group Relative Policy Optimization (GRPO)**.

> [!TIP]
> It was shown in the paper [Understanding R1-Zero-Like Training: A Critical Perspective](https://huggingface.co/papers/2503.20783) that scaling by  \\( \text{std}(\mathbf{r}) \\) may cause a question-level difficulty bias. You can disable this scaling by setting `scale_rewards=False` in [GRPOConfig](/docs/trl/main/en/grpo_trainer#trl.GRPOConfig).
> Note that turning off std-based scaling also removes variance normalization, so update magnitudes depend directly on the raw reward scale and batch composition.

> [!TIP]
> As shown in [Part I: Tricks or Traps? A Deep Dive into RL for LLM Reasoning (Lite PPO)](https://huggingface.co/papers/2508.08221), calculating the mean at the local (group) level and the standard deviation at the global (batch) level enables more robust reward shaping. You can use this scaling strategy by setting `scale_rewards="batch"` in [GRPOConfig](/docs/trl/main/en/grpo_trainer#trl.GRPOConfig).

### Estimating the KL divergence

KL divergence is estimated using the approximator introduced by [Schulman et al. (2020)](http://joschu.net/blog/kl-approx.html). The approximator is defined as follows:

$$\mathbb{D}_{\text{KL}}\left[\pi_\theta \|\pi_{\text{ref}}\right] = \frac{\pi_{\text{ref}}(o_{i,t} \mid q, o_{i,<t})}{\pi_\theta(o_{i,t} \mid q, o_{i,<t})} - \log \frac{\pi_{\text{ref}}(o_{i,t} \mid q, o_{i,<t})}{\pi_\theta(o_{i,t} \mid q, o_{i,<t})} - 1,
$$

### Computing the loss

The objective is to maximize the advantage while ensuring that the model remains close to the reference policy. Consequently, the loss is defined as follows:

$$
\mathcal{L}_{\text{GRPO}}(\theta) = -\frac{1}{\sum_{i=1}^G |o_i|} \sum_{i=1}^G \sum_{t=1}^{|o_i|} \left[ \frac{\pi_\theta(o_{i,t} \mid q, o_{i,< t})}{\left[\pi_\theta(o_{i,t} \mid q, o_{i,< t})\right]_{\text{no grad}}} \hat{A}_{i,t} - \beta \mathbb{D}_{\text{KL}}\left[\pi_\theta \| \pi_{\text{ref}}\right] \right],
$$

where the first term represents the scaled advantage and the second term penalizes deviations from the reference policy through KL divergence.

> [!TIP]
> Note that compared to the original formulation in [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://huggingface.co/papers/2402.03300), we don't scale by  \\( \frac{1}{|o_i|} \\) because it was shown in the paper [Understanding R1-Zero-Like Training: A Critical Perspective](https://huggingface.co/papers/2503.20783) that this introduces a response-level length bias. More details in [loss types](#loss-types).

> [!TIP]
> Note that compared to the original formulation in [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://huggingface.co/papers/2402.03300), we use  \\( \beta = 0.0 \\) by default, meaning that the KL divergence term is not used. This choice is motivated by several recent studies (e.g., [Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model](https://huggingface.co/papers/2503.24290)) which have shown that the KL divergence term is not essential for training with GRPO. As a result, it has become common practice to exclude it (e.g. [Understanding R1-Zero-Like Training: A Critical Perspective](https://huggingface.co/papers/2503.20783), [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://huggingface.co/papers/2503.14476)). If you wish to include the KL divergence term, you can set `beta` in [GRPOConfig](/docs/trl/main/en/grpo_trainer#trl.GRPOConfig) to a non-zero value.

In the original paper, this formulation is generalized to account for multiple updates after each generation (denoted  \\( \mu \\), can be set with `num_iterations` in [GRPOConfig](/docs/trl/main/en/grpo_trainer#trl.GRPOConfig)) by leveraging the **clipped surrogate objective**:

$$
\mathcal{L}_{\text{GRPO}}(\theta) = - \frac{1}{\sum_{i=1}^G |o_i|} \sum_{i=1}^G \sum_{t=1}^{|o_i|} \left[ \min \left( \frac{\pi_\theta(o_{i,t} \mid q, o_{i,< t})}{\pi_{\theta_{\text{old}}}(o_{i,t} \mid q, o_{i,< t})} \hat{A}_{i,t}, \, \text{clip}\left( \frac{\pi_\theta(o_{i,t} \mid q, o_{i,< t})}{\pi_{\theta_{\text{old}}}(o_{i,t} \mid q, o_{i,< t})}, 1 - \epsilon, 1 + \epsilon \right) \hat{A}_{i,t} \right) - \beta \mathbb{D}_{\text{KL}}\left[\pi_\theta \| \pi_{\text{ref}}\right] \right],
$$

where  \\(\text{clip}(\cdot, 1 - \epsilon, 1 + \epsilon) \\) ensures that updates do not deviate excessively from the reference policy by bounding the policy ratio between  \\( 1 - \epsilon \\) and  \\( 1 + \epsilon \\).
When  \\( \mu = 1 \\) (default in TRL), the clipped surrogate objective simplifies to the original objective.

#### Loss Types

Several formulations of the objective have been proposed in the literature. Initially, the objective of GRPO was defined as follows:

$$
\mathcal{L}_{\text{GRPO}}(\theta) = - \frac{1}{G} \sum_{i=1}^G \frac{1}{|o_i|} \sum_{t=1}^{|o_i|} l_{i,t},
$$

where

$$
l_{i,t} = \frac{\pi_\theta(o_{i,t} \mid q, o_{i,< t})}{\left[\pi_\theta(o_{i,t} \mid q, o_{i,< t})\right]_{\text{no grad}}} \hat{A}_{i,t} - \beta \mathbb{D}_{\text{KL}}\left[\pi_\theta \| \pi_{\text{ref}}\right].
$$

The [DAPO paper](https://huggingface.co/papers/2503.14476) highlights the limitations of the GRPO algorithm’s sample-level loss in long-CoT scenarios, where longer responses are under-penalized, leading to poorer quality outputs. The proposed solution is a token-level normalization, which better handles longer sequences by assigning more balanced rewards to individual tokens, regardless of response length:

$$
\mathcal{L}_{\text{DAPO}}(\theta) = - \frac{1}{\sum_{i=1}^G |o_i|} \sum_{i=1}^G \sum_{t=1}^{|o_i|} l_{i,t},
$$

To use this formulation, set `loss_type="dapo"` in [GRPOConfig](/docs/trl/main/en/grpo_trainer#trl.GRPOConfig).

Furthermore, it was demonstrated in the paper [Understanding R1-Zero-Like Training: A Critical Perspective](https://huggingface.co/papers/2503.20783) that the initial GRPO formulation introduces a response length bias. They show that while the DAPO formulation reduces this bias, it does not eliminate it completely. To fully remove this bias, they propose dividing by a constant instead of the sequence length, resulting in the following formulation:

$$
\mathcal{L}_{\text{Dr. GRPO}}(\theta) = - \frac{1}{LG} \sum_{i=1}^G \sum_{t=1}^{|o_i|} l_{i,t},
$$

This constant is recommended to be the maximum completion length. To use this formulation, set `loss_type="dr_grpo"` in the [GRPOConfig](/docs/trl/main/en/grpo_trainer#trl.GRPOConfig).

Alternatively, in the [SAPO paper](https://huggingface.co/papers/2511.20347), the Qwen team proposes replacing the "hard" clipping mechanism of GRPO with a smooth, temperature-controlled soft gating mechanism. While GRPO zeroes out gradients when the policy deviates too far from the reference, SAPO uses a soft trust region that smoothly decays the gradient weight. This allows the model to retain useful learning signals from "near-on-policy" tokens while suppressing noise from extreme deviations.

The loss function is defined as:

$$
\mathcal{L}_{\text{SAPO}}(\theta) = - \frac{1}{G} \sum_{i=1}^G \frac{1}{|o_i|} \sum_{t=1}^{|o_i|} f_{i,t} \left( \frac{\pi_\theta(o_{i,t} | q, o_{i,<t})}{\pi_{\theta_{old}}(o_{i,t} | q, o_{i,<t})} \right) \hat{A}_{i,t}
$$

The soft-gating function  \\( f_{i,t} \\) is defined using the sigmoid function  \\( \sigma \\) as:

$$
f_{i,t}(x) = \sigma \left( \tau_{i,t} (x - 1) \right) \cdot \frac{4}{\tau_{i,t}}
$$

The temperature  \\( \tau_{i,t} \\) is chosen based on the sign of the advantage  \\( \hat{A}_{i,t} \\):

$$
\tau_{i,t} = \begin{cases} 
\tau_{\text{pos}}, & \text{if } \hat{A}_{i,t} > 0 \\
\tau_{\text{neg}}, & \text{otherwise}
\end{cases}
$$

They recommend using asymmetric temperatures,  \\( \tau_{\text{neg}} > \tau_{\text{pos}} \\) (defaults are  \\( \tau_{\text{pos}}=1.0, \tau_{\text{neg}}=1.05 \\) ). This ensures that the model is penalized more strictly for "bad" actions to prevent instability, while being more permissive with "good" actions.

To use this formulation, set `loss_type="sapo"` in the [GRPOConfig](/docs/trl/main/en/grpo_trainer#trl.GRPOConfig).

## Logged metrics

While training and evaluating, we record the following metrics:

- `num_tokens`: The total number of tokens processed so far, including both prompts and completions. When using tools, only non-tool tokens are counted.
- `step_time`: The average time (in seconds) taken per training step (including generation).
- `completions/mean_length`: The average length of generated completions. When using tools, only non-tool tokens are counted.
- `completions/min_length`: The minimum length of generated completions. When using tools, only non-tool tokens are counted.
- `completions/max_length`: The maximum length of generated completions. When using tools, only non-tool tokens are counted.
- `completions/mean_terminated_length`: The average length of generated completions that terminate with EOS. When using tools, only non-tool tokens are counted.
- `completions/min_terminated_length`: The minimum length of generated completions that terminate with EOS. When using tools, only non-tool tokens are counted.
- `completions/max_terminated_length`: The maximum length of generated completions that terminate with EOS. When using tools, only non-tool tokens are counted.
- `completions/clipped_ratio`: The ratio of truncated (clipped) completions.
- `reward/{reward_func_name}/mean`: The average reward from a specific reward function.
- `reward/{reward_func_name}/std`: The standard deviation of the reward from a specific reward function.
- `reward`: The overall average reward after summing rewards across functions (weighted by `reward_weights`).
- `reward_std`: The standard deviation of summed rewards across functions (weighted by `reward_weights`), computed over the full batch.
- `frac_reward_zero_std`: The fraction of samples in the generation batch with a reward std of zero, implying there is little diversity for that prompt (all answers are correct or incorrect).
- `policy_loss`: The policy gradient loss value (before any entropy bonus). Logged when `entropy_coef` is nonzero or `use_adaptive_entropy=True`.
- `entropy`: Average entropy of token predictions across generated completions. (If `mask_truncated_completions=True`, masked sequences tokens are excluded.)
- `entropy_coef`: The current entropy regularization coefficient. Logged when `entropy_coef` is nonzero or `use_adaptive_entropy=True`. Updated once per optimizer step when `use_adaptive_entropy=True`.
- `kl`: The average KL divergence between the model and the reference model, calculated over generated completions. Logged only if `beta` is nonzero.
- `clip_ratio/region_mean`: The ratio of token (or sequence, if `importance_sampling_level="sequence"`) probabilities where the GRPO objective is clipped to stay within the trust region:  \\( \text{clip}\left( r_{i,t}(\theta), 1 - \epsilon_\mathrm{low}, 1 + \epsilon_\mathrm{high} \right)\,, \quad r_{i,t}(\theta) = \frac{\pi_\theta(o_{i,t} \mid q, o_{i,< t})}{\pi_{\theta_{\text{old}}}(o_{i,t} \mid q, o_{i,< t})} \\). A higher value means more tokens are clipped, which constrains how much the policy $\pi_\theta$ can change.
- `clip_ratio/low_mean`: The average ratio of token (or sequence, if `importance_sampling_level="sequence"`) probabilities that were clipped on the lower bound of the trust region:  \\(r_{i,t}(\theta) < 1 - \epsilon_\mathrm{low}\\).
- `clip_ratio/low_min`: The minimum ratio of token (or sequence, if `importance_sampling_level="sequence"`) probabilities that were clipped on the lower bound of the trust region:  \\(r_{i,t}(\theta) < 1 - \epsilon_\mathrm{low}\\).
- `clip_ratio/high_mean`: The average ratio of token (or sequence, if `importance_sampling_level="sequence"`) probabilities that were clipped on the upper bound of the trust region:  \\(r_{i,t}(\theta) > 1 + \epsilon_\mathrm{high}\\).
- `clip_ratio/high_max`: The maximum ratio of token (or sequence, if `importance_sampling_level="sequence"`) probabilities that were clipped on the upper bound of the trust region:  \\(r_{i,t}(\theta) > 1 + \epsilon_\mathrm{high}\\).

## Customization

### Speed up training with vLLM-powered generation

Generation is often the main bottleneck when training with online methods. To accelerate generation, you can use [vLLM](https://github.com/vllm-project/vllm), a high-throughput, low-latency inference engine for LLMs. To enable it, first install the package with

```shell
pip install trl[vllm]
```

We support two ways of using vLLM during training: **server mode** and **colocate mode**.

> [!TIP]
> By default, Truncated Importance Sampling is activated for vLLM generation to address the generation-training mismatch that occurs when using different frameworks. This can be turned off by setting `vllm_importance_sampling_correction=False`. For more information, see [Truncated Importance Sampling](paper_index#truncated-importance-sampling)

#### Option 1: Colocate mode

In this mode, vLLM runs inside the trainer process and shares GPU memory with the training model. This avoids launching a separate server and can improve GPU utilization, but may lead to memory contention on the training GPUs. This is the default mode.

```python
from trl import GRPOConfig

training_args = GRPOConfig(
    ...,
    use_vllm=True,  # vllm_mode="colocate" by default
)
```

#### Option 2: Server mode

In this mode, vLLM runs in a separate process (and using separate GPUs) and communicates with the trainer via HTTP. This is ideal if you have dedicated GPUs for inference.

1. **Start the vLLM server**:

   ```bash
   trl vllm-serve --model <model_name>
   ```

2. **Enable server mode in your training script**:

   ```python
   from trl import GRPOConfig

   training_args = GRPOConfig(
       ...,
       use_vllm=True,
       vllm_mode="server",
   )
   ```

> [!WARNING]
> Make sure that the server is using different GPUs than the trainer, otherwise you may run into NCCL errors. You can specify the GPUs to use with the `CUDA_VISIBLE_DEVICES` environment variable.

> [!TIP]
> Depending on the model size and the overall GPU memory requirements for training, you may need to adjust the `vllm_gpu_memory_utilization` parameter in [GRPOConfig](/docs/trl/main/en/grpo_trainer#trl.GRPOConfig) to avoid underutilization or out-of-memory errors.
>
> We provide a [HF Space](https://huggingface.co/spaces/trl-lib/recommend-vllm-memory) to help estimate the recommended GPU memory utilization based on your model configuration and experiment settings. Simply use it as follows to get `vllm_gpu_memory_utilization` recommendation:
>
> 
>
> If the recommended value does not work in your environment, we suggest adding a small buffer (e.g., +0.05 or +0.1) to the recommended value to ensure stability.
>
> If you still find you are getting out-of-memory errors set `vllm_enable_sleep_mode` to True and the vllm parameters and cache will be offloaded during the optimization step. For more information, see [Reducing Memory Usage with vLLM Sleep Mode](reducing_memory_usage#vllm-sleep-mode).

> [!TIP]
> By default, GRPO uses `MASTER_ADDR=localhost` and `MASTER_PORT=12345` for vLLM, but you can override these values by setting the environment variables accordingly.

For more information, see [Speeding up training with vLLM](speeding_up_training#vllm-for-fast-generation-in-online-methods).

#### Dealing with the Training-Inference Mismatch
While vLLM greatly accelerates inference, it also decouples the inference engine from the training engine. In theory these engines are mathematically identical, in practice however they can produce different outputs due to precision effects and hardware specific optimizations. This divergence reflects the different optimization objectives of the two systems. Inference engines aim to maximize sampling throughput, typically measured in tokens per second, while maintaining acceptable sampling fidelity. Training frameworks instead focus on numerical stability and precision for gradient computation, often using higher precision formats like FP32 for master weights and optimizer states. These differing priorities and constraints introduce an inevitable, albeit subtle, mismatch between training and inference.

This mismatch leads to a biased gradient update which has been observed to destabilize training ([[1]](https://fengyao.notion.site/off-policy-rl)[[2]](https://yingru.notion.site/When-Speed-Kills-Stability-Demystifying-RL-Collapse-from-the-Training-Inference-Mismatch-271211a558b7808d8b12d403fd15edda)[[3]](https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/#true-on-policy-rl)[[4]](https://huggingface.co/papers/2510.26788)[[5]](https://huggingface.co/papers/2510.18855)). For simplicity, consider the REINFORCE policy gradient:

$$
\nabla_\theta \mathcal{J}(x,\theta)
= \mathbb{E}_{y \sim \pi^\text{train}(\cdot \mid x,\theta)}
\left[ \nabla_\theta \log \pi^\text{train}(y \mid x,\theta) \cdot R(x,y) \right]
$$

Here  \\( x \\) denotes prompts sampled from some data distribution, and  \\( \pi^\text{train} \\) is the policy implemented by the training engine. With vLLM in the loop we obtain a separate inference policy  \\( \pi^\text{inference} \\), so the effective policy gradient becomes

$$
\nabla_\theta \mathcal{J}_{\text{biased}}(x,\theta)
= \mathbb{E}_{y \sim \pi^\text{inference}(\cdot \mid x,\theta)}
\left[ \nabla_\theta \log \pi^\text{train}(y \mid x,\theta) \cdot R(x,y) \right].
$$

This turns an otherwise on policy RL problem into an off policy one.

The standard way to correct for this distribution shift is **importance sampling (IS)**. We provide two IS variants: [Truncated Importance Sampling (TIS)](paper_index#truncated-importance-sampling) and [Masked Importance Sampling (MIS)](paper_index#masked-importance-sampling). Both variants can be applied either at the token level or at the sequence level. Let  \\( \rho \\) denote the importance weight, for example  \\( \rho_t \\) per token or  \\( \rho_{\text{seq}} \\) per sequence. Under TIS, ratios outside of the range `[vllm_importance_sampling_clip_min, vllm_importance_sampling_clip_max]` are clipped,

$$
\rho \leftarrow \text{clip}(\rho, C_{\min}, C_{\max}).
$$

The original [TIS paper](https://huggingface.co/papers/1606.02647) proposed a single upper-bound clipping mechanism, i.e.,  \\( \min(\rho, C_{\max}) \\). The implementation in TRL generalized this by also introducing a lower bound, yielding the two-sided formulation shown above, inspired by the [IcePop](paper_index#masked-importance-sampling) method.  
Note that in IcePop, the bounds are labelled as  \\( \alpha \\) and  \\( \beta \\) while in TRL we use  \\( C_{\min} \\) and  \\( C_{\max} \\).

Under MIS, ratios outside of this range are set to zero, so those samples do not contribute to the gradient. In other words, outlier samples are downweighted under TIS and discarded under MIS. The configuration flag `vllm_importance_sampling_mode` chooses both the IS variant (masking or truncation) and the granularity (token level or sequence level).

Importance sampling is the principled algorithmic response to the training–inference mismatch. However, there are also more direct approaches that attempt to reduce the mismatch between the two engines themselves. Most of these are engineering solutions. For example, [MiniMax M1 uses an FP32 language model head](https://huggingface.co/papers/2506.13585) in the inference engine. Thinking Machines has explored [deterministic inference kernels](https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/), although this comes with a significant efficiency cost. vLLM has shown [bitwise consistent policies](https://blog.vllm.ai/2025/11/10/bitwise-consistent-train-inference.html) by building on the batch invariant deterministic kernels from Thinking Machines, but as of November 2025 there remains a substantial throughput penalty relative to standard vLLM inference.

### Speed up training with transformers continuous batching

As an alternative to vLLM, you can use transformers' built-in continuous batching engine for faster generation. Continuous batching removes finished sequences from the batch immediately rather than waiting for the slowest one to finish. For tasks with variable completion lengths (e.g., math reasoning), this yields faster generation and lower VRAM usage than the default `generate()` at large batch sizes (N≥32).

> [!TIP]
> Continuous batching is a drop-in upgrade with no server setup or weight synchronization. It runs in-process and is well-suited for single-GPU training or memory-constrained environments. For maximum generation throughput at scale, use vLLM instead.

```python
from trl import GRPOConfig

training_args = GRPOConfig(
    ...,
    use_transformers_continuous_batching=True,
    transformers_continuous_batching_config={
        "use_cuda_graph": False,
        "max_memory_percent": 0.4,  # lower values leave more VRAM for the training backward pass
    },
)
```

> [!TIP]
> TRL defaults `max_memory_percent` to `0.5` (instead of transformers' `0.9`) to leave enough VRAM for the training backward pass. Tune it down to `0.3`–`0.4` for large generation batches (N≥32) or if you see out-of-memory errors.

For a full training example, see [`examples/scripts/grpo_continuous_batching.py`](https://github.com/huggingface/trl/blob/main/examples/scripts/grpo_continuous_batching.py).

### GRPO at scale: train a 70B+ Model on multiple nodes

When training large models like **Qwen2.5-72B**, you need several key optimizations to make the training efficient and scalable across multiple GPUs and nodes. These include:

- **DeepSpeed ZeRO Stage 3**: ZeRO leverages data parallelism to distribute model states (weights, gradients, optimizer states) across multiple GPUs and CPUs, reducing memory and compute requirements on each device. Since large models cannot fit on a single GPU, using ZeRO Stage 3 is required for training such models. For more details, see [DeepSpeed Integration](deepspeed_integration).
- **Accelerate**: Accelerate is a library that simplifies distributed training across multiple GPUs and nodes. It provides a simple API to launch distributed training and handles the complexities of distributed training, such as data parallelism, gradient accumulation, and distributed data loading. For more details, see [Distributing Training](distributing_training).
- **vLLM**: See the previous section on how to use vLLM to speed up generation.

Below is an example SLURM script to train a 70B model with GRPO on multiple nodes. This script trains a model on 4 nodes and uses the 5th node for vLLM-powered generation.

```sh
#!/bin/bash
#SBATCH --nodes=5
#SBATCH --gres=gpu:8

# Get the list of allocated nodes
NODELIST=($(scontrol show hostnames $SLURM_JOB_NODELIST))

# Assign the first 4 nodes for training and the 5th node for vLLM
TRAIN_NODES="${NODELIST[@]:0:4}"  # Nodes 0, 1, 2, 3 for training
VLLM_NODE="${NODELIST[4]}"  # Node 4 for vLLM

# Run training on the first 4 nodes (Group 1)
srun --nodes=4 --ntasks=4 --nodelist="${NODELIST[@]:0:4}" accelerate launch \
     --config_file examples/accelerate_configs/deepspeed_zero3.yaml \
     --num_processes 32 \
     --num_machines 4 \
     --main_process_ip ${NODELIST[0]} \
     --machine_rank $SLURM_PROCID \
     --rdzv_backend c10d \
     train_grpo.py \
     --server_ip $VLLM_NODE &

# Run vLLM server on the 5th node (Group 2)
srun --nodes=1 --ntasks=1 --nodelist="${NODELIST[4]}" trl vllm-serve --model Qwen/Qwen2.5-72B --tensor_parallel_size 8 &

wait
```

```python
import argparse

from datasets import load_dataset
from trl import GRPOTrainer, GRPOConfig
from trl.rewards import accuracy_reward

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--vllm_server_host", type=str, default="", help="The server IP")
    args = parser.parse_args()

    dataset = load_dataset("trl-lib/DeepMath-103K", split="train")

    training_args = GRPOConfig(
        per_device_train_batch_size=4,
        use_vllm=True,
        vllm_mode="server",
        vllm_server_host=args.vllm_server_host.replace("ip-", "").replace("-", "."),  # from ip-X-X-X-X to X.X.X.X
    )

    trainer = GRPOTrainer(
        model="Qwen/Qwen2.5-72B",
        args=training_args,
        reward_funcs=accuracy_reward,
        train_dataset=dataset
    )
    trainer.train()

if __name__=="__main__":
    main()
```

### Using a custom reward function

The [GRPOTrainer](/docs/trl/main/en/grpo_trainer#trl.GRPOTrainer) supports using custom reward functions instead of dense reward models. To ensure compatibility, your reward function must satisfy the following requirements:

Reward functions can be either synchronous Python callables or asynchronous `async def` coroutines. When you provide multiple asynchronous reward functions, they are awaited concurrently (run in parallel via `asyncio.gather`) so their latency overlaps.

1. **Input arguments**:
   - The function must accept the following as keyword arguments:
     - `prompts` (contains the prompts),
     - `completions` (contains the generated completions),
     - `completion_ids` (contains the tokenized completions),
     - `trainer_state` ([TrainerState](https://huggingface.co/docs/transformers/main/en/main_classes/callback#transformers.TrainerState)): The current state of the trainer. This can be used to implement dynamic reward functions, such as curriculum learning, where the reward is adjusted based on the training progress.
     - `log_extra`: a callable `log_extra(column: str, values: list)` to add extra columns to the completions table. See Example 6. In distributed training, it's important that all processes log the same set of keys.
     - `log_metric`: a callable `log_metric(name: str, value: float)` to log scalar metrics as plots alongside `kl`, `entropy`, etc. See Example 6. In distributed training, it's important that all processes log the same set of keys.
     - `environments`: a list of environment instances, one per completion. Only present when `environment_factory` is provided. Use this to read state accumulated during the episode (e.g., `env.reward`).
     - All column names (but `prompt`) that the dataset may have. For example, if the dataset contains a column named `ground_truth`, the function will be called with `ground_truth` as a keyword argument.

     The easiest way to comply with this requirement is to use `**kwargs` in the function signature.
   - Depending on the dataset format, the input will vary:
     - For [standard format](dataset_formats#standard), `prompts` and `completions` will be lists of strings.
     - For [conversational format](dataset_formats#conversational), `prompts` and `completions` will be lists of message dictionaries.

2. **Return value**: The function must return a list of floats. Each float represents the reward corresponding to a single completion.

#### Example 1: Reward longer completions

Below is an example of a reward function for a standard format that rewards longer completions:

```python
def reward_func(completion_ids, **kwargs):
    """Reward function that assigns higher scores to longer completions (in terms of token count)."""
    return [float(len(ids)) for ids in completion_ids]
```

You can test it as follows:

```python
>>> prompts = ["The sky is", "The sun is"]  # not used in the reward function, but the trainer will pass it
>>> completions = [" blue.", " in the sky."]  # not used in the reward function, but the trainer will pass it
>>> completion_ids = [[6303, 13], [304, 279, 12884, 13]]
>>> reward_func(prompts=prompts, completions=completions, completion_ids=completion_ids)
[2.0, 4.0]
```

#### Example 1.1: Reward longer completions (based on the number of characters)

Same as the previous example, but this time the reward function is based on the number of characters instead of tokens.

```python
def reward_func(completions, **kwargs):
    """Reward function that assigns higher scores to longer completions (in terms of character count)."""
    return [float(len(completion)) for completion in completions]
```

You can test it as follows:

```python
>>> prompts = ["The sky is", "The sun is"]
>>> completions = [" blue.", " in the sky."]
>>> completion_ids = [[6303, 13], [304, 279, 12884, 13]]  # not used in the reward function, but the trainer will pass it
>>> reward_func(prompts=prompts, completions=completions, completion_ids=completion_ids)
[6.0, 12.0]
```

#### Example 2: Reward completions with a specific format

Below is an example of a reward function that checks if the completion has a specific format. This example is inspired by the _format reward_ function used in the paper [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://huggingface.co/papers/2501.12948).
It is designed for a conversational format, where prompts and completions consist of structured messages.

```python
import re

def format_reward_func(completions, **kwargs):
    """Reward function that checks if the completion has a specific format."""
    pattern = r"^<think>.*?</think><answer>.*?</answer>$"
    completion_contents = [completion[0]["content"] for completion in completions]
    matches = [re.match(pattern, content) for content in completion_contents]
    return [1.0 if match else 0.0 for match in matches]
```

You can test this function as follows:

```python
>>> prompts = [
...     [{"role": "assistant", "content": "What is the result of (1 + 2) * 4?"}],
...     [{"role": "assistant", "content": "What is the result of (3 + 1) * 2?"}],
... ]
>>> completions = [
...     [{"role": "assistant", "content": "<think>The sum of 1 and 2 is 3, which we multiply by 4 to get 12.</think><answer>(1 + 2) * 4 = 12</answer>"}],
...     [{"role": "assistant", "content": "The sum of 3 and 1 is 4, which we multiply by 2 to get 8. So (3 + 1) * 2 = 8."}],
... ]
>>> format_reward_func(prompts=prompts, completions=completions)
[1.0, 0.0]
```

#### Example 3: Reward completions based on a reference

Below is an example of a reward function that checks if the completion is correct. This example is inspired by the _accuracy reward_ function used in the paper [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://huggingface.co/papers/2501.12948).
This example is designed for [standard format](dataset_formats#standard), where the dataset contains a column named `ground_truth`.

```python
import re

def reward_func(completions, ground_truth, **kwargs):
    # Regular expression to capture content inside \boxed{}
    matches = [re.search(r"\\boxed\{(.*?)\}", completion) for completion in completions]
    contents = [match.group(1) if match else "" for match in matches]
    # Reward 1 if the content is the same as the ground truth, 0 otherwise
    return [1.0 if c == gt else 0.0 for c, gt in zip(contents, ground_truth)]
```

You can test this function as follows:

```python
>>> prompts = ["Problem: Solve the equation $2x + 3 = 7$. Solution:", "Problem: Solve the equation $3x - 5 = 10$."]
>>> completions = [r" The solution is \boxed{2}.", r" The solution is \boxed{6}."]
>>> ground_truth = ["2", "5"]
>>> reward_func(prompts=prompts, completions=completions, ground_truth=ground_truth)
[1.0, 0.0]
```

#### Example 4: Multi-task reward functions

Below is an example of using multiple reward functions in the [GRPOTrainer](/docs/trl/main/en/grpo_trainer#trl.GRPOTrainer). In this example, we define two task-specific reward functions: `math_reward_func` and `coding_reward_func`. The `math_reward_func` rewards math problems based on their correctness, while the `coding_reward_func` rewards coding problems based on whether the solution works.

```python
from datasets import Dataset
from trl import GRPOTrainer

# Define a dataset that contains both math and coding problems
dataset = Dataset.from_list(
    [
        {"prompt": "What is 2+2?", "task": "math"},
        {"prompt": "Write a function that returns the sum of two numbers.", "task": "code"},
        {"prompt": "What is 3*4?", "task": "math"},
        {"prompt": "Write a function that returns the product of two numbers.", "task": "code"},
    ]
)

# Math-specific reward function
def math_reward_func(prompts, completions, task, **kwargs):
    rewards = []
    for prompt, completion, t in zip(prompts, completions, task):
        if t == "math":
            # Calculate math-specific reward
            correct = check_math_solution(prompt, completion)
            reward = 1.0 if correct else -1.0
            rewards.append(reward)
        else:
            # Return None for non-math tasks
            rewards.append(None)
    return rewards

# Coding-specific reward function
def coding_reward_func(prompts, completions, task, **kwargs):
    rewards = []
    for prompt, completion, t in zip(prompts, completions, task):
        if t == "coding":
            # Calculate coding-specific reward
            works = test_code_solution(prompt, completion)
            reward = 1.0 if works else -1.0
            rewards.append(reward)
        else:
            # Return None for non-coding tasks
            rewards.append(None)
    return rewards

# Use both task-specific reward functions
trainer = GRPOTrainer(
    model="Qwen/Qwen2.5-0.5B-Instruct",
    reward_funcs=[math_reward_func, coding_reward_func],
    train_dataset=dataset,
)

trainer.train()
```

In this example, the `math_reward_func` and `coding_reward_func` are designed to work with a mixed dataset that contains both math and coding problems. The `task` column in the dataset is used to determine which reward function to apply to each problem. If there is no relevant reward function for a sample in the dataset, the reward function will return `None`, and the [GRPOTrainer](/docs/trl/main/en/grpo_trainer#trl.GRPOTrainer) will continue with the valid functions and tasks. This allows the [GRPOTrainer](/docs/trl/main/en/grpo_trainer#trl.GRPOTrainer) to handle multiple reward functions with different applicability.

Note that the [GRPOTrainer](/docs/trl/main/en/grpo_trainer#trl.GRPOTrainer) will ignore the `None` rewards returned by the reward functions and only consider the rewards returned by the relevant functions. This ensures that the model is trained on the relevant tasks and ignores the tasks for which there is no relevant reward function.

#### Example 5: Asynchronous reward functions

Custom reward functions can also be defined as `async def` coroutines. This is useful if your reward depends on slow I/O (for example, calling a remote service). When you pass multiple async reward functions, [GRPOTrainer](/docs/trl/main/en/grpo_trainer#trl.GRPOTrainer) executes them concurrently so their latency overlaps.

Below is a minimal example of an async reward function that simulates an I/O-bound operation:

```python
import asyncio

async def async_reward_func(prompts, completions, **kwargs):
    # Simulate an I/O-bound call (e.g., HTTP request, database lookup)
    await asyncio.sleep(0.01)
    # Simple toy reward: 1.0 if the completion is non-empty, else 0.0
    return [1.0 if completion else 0.0 for completion in completions]
```

#### Example 6: Logging extra columns and metrics

Below is an example of a reward function that logs extra columns to the completions table and scalar metrics as plots.

```python
import re

def reward_func(completions, ground_truth, log_extra=None, log_metric=None, **kwargs):
    extracted = [re.search(r"\\boxed\{(.*?)\}", c) for c in completions]
    extracted = [m.group(1) if m else None for m in extracted]
    rewards = [1.0 if e == gt else 0.0 for e, gt in zip(extracted, ground_truth)]

    if log_extra:
        log_extra("golden_answer", list(ground_truth))
        log_extra("extracted_answer", [e or "[none]" for e in extracted])

    if log_metric:
        log_metric("accuracy", sum(rewards) / len(rewards))

    return rewards
```

#### Passing the reward function to the trainer

To use your custom reward function, pass it to the [GRPOTrainer](/docs/trl/main/en/grpo_trainer#trl.GRPOTrainer) as follows:

```python
from trl import GRPOTrainer

trainer = GRPOTrainer(
    reward_funcs=reward_func,
    ...,
)
```

You can pass several reward functions as a list; this list may include both synchronous and asynchronous functions:

```python
from trl import GRPOTrainer

trainer = GRPOTrainer(
    reward_funcs=[reward_func, async_reward_func1, async_reward_func2],
    ...,
)
```

and the reward will be computed as the sum of the rewards from each function, or the weighted sum if `reward_weights` is provided in the config.

Note that [GRPOTrainer](/docs/trl/main/en/grpo_trainer#trl.GRPOTrainer) supports multiple reward functions of different types. See the parameters documentation for more details.

### Entropy regularization

To encourage exploration and prevent the policy from collapsing to near-deterministic outputs, you can add an entropy bonus to the training objective. The entropy regularization augments the GRPO loss as follows:

$$
\mathcal{L}(\theta) = \mathcal{L}_{\text{GRPO}}(\theta) - \alpha \cdot \mathcal{H}(\pi_\theta),
$$

where \\(\mathcal{H}(\pi_\theta)\\) is the mean per-token entropy of the policy and \\(\alpha\\) is the entropy coefficient. The bonus is always the mean per-token entropy regardless of `loss_type`; it is not rescaled to match a loss type's policy normalization (e.g. Dr. GRPO's `batch_size * max_completion_length` denominator), so `entropy_coef` has the same meaning for every loss type.

**Static entropy** — a fixed coefficient throughout training:

```python
from trl import GRPOConfig, GRPOTrainer

training_args = GRPOConfig(entropy_coef=0.05, ...)
```

**Adaptive entropy** — the coefficient is updated each optimizer step based on a target entropy, as introduced in [Skywork-OR1](https://huggingface.co/papers/2505.22312). When the current entropy falls at or below `entropy_target`, the coefficient is incremented by `entropy_coef_delta`; otherwise it is decremented. The coefficient is only applied (i.e. non-zero) while entropy is at or below the target:

```python
training_args = GRPOConfig(
    entropy_coef=0.01,          # initial coefficient
    use_adaptive_entropy=True,
    entropy_target=5.0,         # target mean per-token entropy (nats); tune for your model
    entropy_coef_delta=0.005,   # step size per optimizer step
    entropy_coef_min=0.0,
    entropy_coef_max=1.0,
    ...
)
```

Typical language models have per-token entropies of 2–10 nats, so the default `entropy_target=0.2` almost never triggers regularization — the bonus only engages once entropy is at or below the target, i.e. near-complete collapse. Set it to a value meaningful for your model, e.g. close to the entropy you observe early in training (logged as the `entropy` metric). When using `top_entropy_quantile < 1.0`, `entropy_target` applies to the high-entropy token subset — that subset's entropy will be higher than the logged full-token `entropy`, so calibrate accordingly.

When `use_adaptive_entropy=True`, the current entropy coefficient `entropy_coef` is saved alongside each checkpoint and restored on resume, so training is fully resumable.

### Rapid Experimentation for GRPO

RapidFire AI is an open-source experimentation engine that sits on top of TRL and lets you launch multiple GRPO configurations at once, even on a single GPU. Instead of trying configurations sequentially, RapidFire lets you **see all their learning curves earlier, stop underperforming runs, and clone promising ones with new settings in flight** without restarting. For more information, see [RapidFire AI Integration](rapidfire_integration).

## Agent Training

GRPO supports **agent training** through the `tools` argument in [GRPOTrainer](/docs/trl/main/en/grpo_trainer#trl.GRPOTrainer).
This parameter expects a list of Python functions (sync or async) that define the tools available to the agent:

```python
from trl import GRPOTrainer

trainer = GRPOTrainer(
    tools=[tool1, tool2],
    ...,
)
```

Each tool must be a standard Python function with **type-hinted arguments and return types**, along with a **Google-style docstring** describing its purpose, arguments, and return value.
For more details, see the [Passing tools guide](https://huggingface.co/docs/transformers/en/chat_extras#passing-tools).

> [!TIP]
> The GRPO tool call loop requires the chat template to be *prefix-preserving* (appending a tool message must not change how earlier messages are rendered). For known model families (e.g. Qwen3, DeepSeek-V3), TRL automatically swaps in a patched training template when tools are enabled. See [Chat Templates](chat_templates#training-templates) for the full list.

Example:

```python
from trl import GRPOTrainer

def multiply(a: int, b: int) -> int:
    """
    Multiplies two integers.

    Args:
        a: The first integer.
        b: The second integer.

    Returns:
        The product of the two integers.
    """
    return a * b

async def async_add(a: int, b: int) -> int:
    """
    Asynchronously adds two integers.

    Args:
        a: The first integer.
        b: The second integer.

    Returns:
        The sum of the two integers.
    """
    return a + b

trainer = GRPOTrainer(
    tools=[multiply, async_add],
    ...,
)
```

You can also provide tools through `environment_factory`. In this mode, [GRPOTrainer](/docs/trl/main/en/grpo_trainer#trl.GRPOTrainer) creates one environment instance per rollout and exposes the environment's public methods as tools. See the [OpenEnv guide](openenv) for the `environment_factory` contract.

All environments plug into the same `environment_factory` slot, so they are interchangeable at the TRL level — pick the one whose ecosystem fits your task:

| Integration | What it is | Use it when |
|---|---|---|
| [OpenEnv](openenv) | The open environment standard (Gymnasium-style API, served over WebSocket or containerised execution), backed by Hugging Face and the community. | You're using a ready-made OpenEnv environment from the Hub, or defining your own against the open standard (e.g. Wordle, Sudoku, Catch). |
| [OpenReward](openreward) | An integration with ORS-speaking environments (the [openreward.ai](https://openreward.ai) catalog or your own ORS server); tasks **and** rewards are served over HTTP. | You want to train against an ORS environment: the catalog (e.g. `Eigent/SETA`), one you self-host on your own infra, or a local server you're developing. |
| [Harbor](harbor) | An integration with Harbor task suites: each task is an instruction, a real sandbox image (`docker`, `e2b`, ...), and an in-sandbox verifier. | You want to train against a Harbor task suite: a tree of tasks, each a self-contained sandbox plus verifier (e.g. a data-analysis agent that explores files in a sandbox and writes an answer a grader checks). |

> [!IMPORTANT]
> `environment_factory` requires `transformers>=5.2.0`.

The following is a minimal example of using `environment_factory` to define a simple environment with an `increment` method, which is exposed as a tool to the agent:

```python
from datasets import Dataset
from trl import GRPOConfig, GRPOTrainer

instructions = [f"Increment the counter by {i}." for i in range(1, 7)]
dataset = Dataset.from_dict({"prompt": [[{"role": "user", "content": instruction}] for instruction in instructions]})

def reward_func(environments, **kwargs):  # dummy reward: the reward is the current value of the counter
    return [environment.counter for environment in environments]

class IncrementEnv:
    def reset(self, **kwargs) -> str | None:  # required; receives sampled row fields as kwargs (e.g., `prompt`)
        self.counter = 0
        return "Counter reset to 0.\n"

    def increment(self, step: int) -> int:  # the other public methods of the environment are exposed as tools
        """
        Increment the internal counter.

        Args:
            step: Value to add to the counter.

        Returns:
            The updated counter value.
        """
        self.counter += step
        return self.counter

trainer = GRPOTrainer(
    model="Qwen/Qwen3-0.6B",
    args=GRPOConfig(chat_template_kwargs={"enable_thinking": False}),
    train_dataset=dataset,
    reward_funcs=reward_func,
    environment_factory=IncrementEnv,
)
trainer.train()
```

`reset` can return either `None` or a string. In GRPO, when it returns a string, that string is appended to the last user message before generation.

### Multimodal Tool Responses

Tools can return images alongside text by returning a list of content blocks. This is useful for VLM agent training where the tool provides visual feedback (e.g., screenshots, plots, camera captures).

```python
from PIL import Image

def take_screenshot() -> list:
    """
    Takes a screenshot of the current screen.

    Returns:
        The screenshot image with a description.
    """
    img = Image.open("screenshot.png")
    return [{"type": "image", "image": img}, {"type": "text", "text": "Here is the screenshot."}]
```

The returned images are automatically injected into the conversation and passed to the VLM for subsequent generation turns.

### Supported Models

Tested with:

- [**Gemma4**](https://huggingface.co/collections/google/gemma-4) — e.g., `google/gemma-4-E2B-it`
- **GLM-4-MoE** ([4.5](https://huggingface.co/collections/zai-org/glm-45), [4.6](https://huggingface.co/collections/zai-org/glm-46) or [4.7](https://huggingface.co/collections/zai-org/glm-47)) — e.g., `zai-org/GLM-4.7`
- [**GPT-OSS**](https://huggingface.co/collections/openai/gpt-oss) — e.g., `openai/gpt-oss-20b`
- [**Llama 3.1**](https://huggingface.co/collections/meta-llama/llama-31) — e.g., `meta-llama/Llama-3.1-8B-Instruct`
- [**Llama 3.2**](https://huggingface.co/collections/meta-llama/llama-32) — e.g., `meta-llama/Llama-3.2-3B-Instruct`
- [**Qwen2.5**](https://huggingface.co/collections/Qwen/qwen25) — e.g., `Qwen/Qwen2.5-0.5B-Instruct`
- [**Qwen3**](https://huggingface.co/collections/Qwen/qwen3) — e.g., `Qwen/Qwen3-0.6B`
- [**Qwen3-VL**](https://huggingface.co/collections/Qwen/qwen3-vl) — e.g., `Qwen/Qwen3-VL-2B-Instruct`
- [**Qwen3.5**](https://huggingface.co/collections/Qwen/qwen35) — e.g., `Qwen/Qwen3.5-2B`
- [**Qwen3.6**](https://huggingface.co/collections/Qwen/qwen36) — e.g., `Qwen/Qwen3.6-35B-A3B`

> [!TIP]
> Compatibility with all LLMs is not guaranteed. If you believe a model should be supported, feel free to open an issue on GitHub — or better yet, submit a pull request with the required changes.

### Quick Start

Use [grpo\_agent.py](https://github.com/huggingface/trl/blob/main/examples/scripts/grpo_agent.py) to fine-tune a LLM for agentic workflows.

```bash
accelerate launch \
  --config_file=examples/accelerate_configs/deepspeed_zero3.yaml \
  examples/scripts/grpo_agent.py \
  --model_name_or_path Qwen/Qwen3-0.6B
  ...
```

## Vision-Language Model (VLM) Training

GRPO supports training Vision-Language Models (VLMs) on multimodal datasets containing both text and images.

### Supported Models

Tested with:

- **Gemma3** — e.g., `google/gemma-3-4b-it`
- **LLaVA-NeXT** — e.g., `llava-hf/llava-v1.6-mistral-7b-hf`
- **Qwen2-VL** — e.g., `Qwen/Qwen2-VL-2B-Instruct`
- **Qwen2.5-VL** — e.g., `Qwen/Qwen2.5-VL-3B-Instruct`
- **SmolVLM2** — e.g., `HuggingFaceTB/SmolVLM2-2.2B-Instruct`
  
> [!TIP]
> Compatibility with all VLMs is not guaranteed. If you believe a model should be supported, feel free to open an issue on GitHub — or better yet, submit a pull request with the required changes.

### Quick Start

Use [grpo\_vlm.py](https://github.com/huggingface/trl/blob/main/examples/scripts/grpo_vlm.py) to fine-tune a VLM. Example command for training on [`lmms-lab/multimodal-open-r1-8k-verified`](https://huggingface.co/datasets/lmms-lab/multimodal-open-r1-8k-verified):

```bash
accelerate launch \
  --config_file=examples/accelerate_configs/deepspeed_zero3.yaml \
  examples/scripts/grpo_vlm.py \
  --model_name_or_path Qwen/Qwen2.5-VL-3B-Instruct \
  --output_dir grpo-Qwen2.5-VL-3B-Instruct \
  --learning_rate 1e-5 \
  --dtype bfloat16 \
  --max_completion_length 1024 \
  --use_vllm \
  --vllm_mode colocate \
  --use_peft \
  --lora_target_modules "q_proj", "v_proj" \
  --log_completions
```

### Configuration Tips

- Use LoRA on vision-language projection layers
- Enable 4-bit quantization to reduce memory usage
- VLMs are memory-intensive — start with smaller batch sizes
- Most models are compatible with vLLM (`server` and `colocate` modes)

### Dataset Format

Each training sample should include:

- `prompt`: Text formatted via the processor's chat template
- `image`/`images`: PIL Image or list of PIL Images

The trainer automatically handles image-to-tensor conversion via the model’s image processor.

## GRPOTrainer[[trl.GRPOTrainer]]

- **model** (`str` or [PreTrainedModel](https://huggingface.co/docs/transformers/main/en/main_classes/model#transformers.PreTrainedModel) or [PeftModel](https://huggingface.co/docs/peft/main/en/package_reference/peft_model#peft.PeftModel)) --
  Model to be trained. Can be either:

  - A string, being the *model id* of a pretrained model hosted inside a model repo on huggingface.co, or a
    path to a *directory* containing model weights saved using
    [save_pretrained](https://huggingface.co/docs/transformers/main/en/main_classes/model#transformers.PreTrainedModel.save_pretrained), e.g., `'./my_model_directory/'`. The model is loaded
    using `<ModelArchitecture>.from_pretrained` (where `<ModelArchitecture>` is derived from the model
    config) with the keyword arguments in `args.model_init_kwargs`. If `dtype` is not specified in
    `args.model_init_kwargs`, it defaults to `float32`. This differs from
    [from_pretrained](https://huggingface.co/docs/transformers/main/en/main_classes/model#transformers.PreTrainedModel.from_pretrained), where (since Transformers v5) the dtype is inferred
    from the model config.
  - A [PreTrainedModel](https://huggingface.co/docs/transformers/main/en/main_classes/model#transformers.PreTrainedModel) object. Only causal language models are supported.
  - A [PeftModel](https://huggingface.co/docs/peft/main/en/package_reference/peft_model#peft.PeftModel) object. Only causal language models are supported.
- **reward_funcs** (`RewardFunc | list[RewardFunc]`) --
  Reward functions to be used for computing the rewards. To compute the rewards, we call all the reward
  functions with the prompts and completions and sum the rewards. Can be either:

  - A single reward function, such as:
    - A string: The *model ID* of a pretrained model hosted inside a model repo on huggingface.co, or a
    path to a *directory* containing model weights saved using
    [save_pretrained](https://huggingface.co/docs/transformers/main/en/main_classes/model#transformers.PreTrainedModel.save_pretrained), e.g., `'./my_model_directory/'`. The model is loaded
    using [from_pretrained](https://huggingface.co/docs/transformers/main/en/model_doc/auto#transformers.AutoModelForSequenceClassification.from_pretrained) with `num_labels=1` and the
    keyword arguments in `args.model_init_kwargs`.
    - A [PreTrainedModel](https://huggingface.co/docs/transformers/main/en/main_classes/model#transformers.PreTrainedModel) object: Only sequence classification models are supported.
    - A custom reward function: The function is provided with the prompts and the generated completions,
      plus any additional columns in the dataset. It should return a list of rewards. Custom reward
      functions can be either synchronous or asynchronous and can also return `None` when the reward is
      not applicable to those samples. This is useful for multi-task training where different reward
      functions apply to different types of samples. When a reward function returns `None` for a sample,
      that reward function is excluded from the reward calculation for that sample. For more details, see
      [Using a custom reward
      function](#using-a-custom-reward-function).

      The trainer's state is also passed to the reward function. The trainer's state is an instance of
      [TrainerState](https://huggingface.co/docs/transformers/main/en/main_classes/callback#transformers.TrainerState) and can be accessed by accessing the `trainer_state` argument to the
      reward function's signature.
  - A list of reward functions, where each item can independently be any of the above types. Mixing different
  types within the list (e.g., a string model ID and a custom reward function) is allowed.
- **args** ([GRPOConfig](/docs/trl/main/en/grpo_trainer#trl.GRPOConfig), *optional*) --
  Configuration for this trainer. If `None`, a default configuration is used.
- **train_dataset** ([Dataset](https://huggingface.co/docs/datasets/main/en/package_reference/main_classes#datasets.Dataset) or [IterableDataset](https://huggingface.co/docs/datasets/main/en/package_reference/main_classes#datasets.IterableDataset)) --
  Dataset to use for training. It must include a column `"prompt"`. Any additional columns in the dataset is
  ignored. The format of the samples can be either:

  - [Standard](dataset_formats#standard): Each sample contains plain text.
  - [Conversational](dataset_formats#conversational): Each sample contains structured messages (e.g., role
    and content).
- **eval_dataset** ([Dataset](https://huggingface.co/docs/datasets/main/en/package_reference/main_classes#datasets.Dataset), [IterableDataset](https://huggingface.co/docs/datasets/main/en/package_reference/main_classes#datasets.IterableDataset) or `dict[str, Dataset | IterableDataset]`) --
  Dataset to use for evaluation. It must meet the same requirements as `train_dataset`.
- **processing_class** ([PreTrainedTokenizerBase](https://huggingface.co/docs/transformers/main/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase), [ProcessorMixin](https://huggingface.co/docs/transformers/main/en/main_classes/processors#transformers.ProcessorMixin), *optional*) --
  Processing class used to process the data. The padding side must be set to "left". If `None`, the
  processing class is loaded from the model's name with [from_pretrained](https://huggingface.co/docs/transformers/main/en/model_doc/auto#transformers.AutoProcessor.from_pretrained). A
  padding token, `tokenizer.pad_token`, must be set. If the processing class has not set a padding token,
  `tokenizer.eos_token` will be used as the default.
- **reward_processing_classes** ([PreTrainedTokenizerBase](https://huggingface.co/docs/transformers/main/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase) or `list[PreTrainedTokenizerBase]`, *optional*) --
  Processing classes corresponding to the reward functions specified in `reward_funcs`. Can be either:

  - A single processing class: Used when `reward_funcs` contains only one reward function.
  - A list of processing classes: Must match the order and length of the reward functions in `reward_funcs`.
  If set to `None`, or if an element of the list corresponding to a [PreTrainedModel](https://huggingface.co/docs/transformers/main/en/main_classes/model#transformers.PreTrainedModel) is
  `None`, the tokenizer for the model is automatically loaded using
  [from_pretrained](https://huggingface.co/docs/transformers/main/en/model_doc/auto#transformers.AutoTokenizer.from_pretrained). For elements in `reward_funcs` that are custom reward
  functions (not [PreTrainedModel](https://huggingface.co/docs/transformers/main/en/main_classes/model#transformers.PreTrainedModel)), the corresponding entries in `reward_processing_classes`
  are ignored.
- **callbacks** (list of [TrainerCallback](https://huggingface.co/docs/transformers/main/en/main_classes/callback#transformers.TrainerCallback), *optional*) --
  List of callbacks to customize the training loop. Will add those to the list of default callbacks detailed
  in [here](https://huggingface.co/docs/transformers/main_classes/callback).

  If you want to remove one of the default callbacks used, use the [remove_callback](https://huggingface.co/docs/transformers/main/en/main_classes/trainer#transformers.Trainer.remove_callback)
  method.
- **optimizers** (`tuple[torch.optim.Optimizer | None, torch.optim.lr_scheduler.LambdaLR | None]`, *optional*, defaults to `(None, None)`) --
  A tuple containing the optimizer and the scheduler to use. Will default to an instance of `AdamW` on your
  model and a scheduler given by [get_linear_schedule_with_warmup](https://huggingface.co/docs/transformers/main/en/main_classes/optimizer_schedules#transformers.get_linear_schedule_with_warmup) controlled by `args`.
- **quantization_config** ([BitsAndBytesConfig](https://huggingface.co/docs/transformers/main/en/main_classes/quantization#transformers.BitsAndBytesConfig), *optional*) --
  Quantization configuration used when loading the model from a model identifier. Combine with `peft_config`
  for QLoRA training. Ignored if the model is already instantiated.
- **peft_config** ([PeftConfig](https://huggingface.co/docs/peft/main/en/package_reference/config#peft.PeftConfig), *optional*) --
  PEFT configuration used to wrap the model. If `None`, the model is not wrapped.
- **tools** (list of `Callable`, *optional*) --
  A list of callable tool functions (sync or async) that the model can invoke during generation. Each tool
  should be a standard Python function with properly type-hinted arguments and return values, and a
  Google-style docstring describing its purpose, arguments, and return value. For more details, see:
  https://huggingface.co/docs/transformers/en/chat_extras#passing-tools. The model uses the function's name,
  type hints, and docstring to determine how to call it. Ensure that the model's chat template supports tool
  use and that it has been fine-tuned for tool calling.
- **rollout_func** (`RolloutFunc`, *optional*) --
  Function to use for generating completions. It receives the list of prompts allocated to the current
  process and the trainer instance. It must return a dict with `"prompt_ids"`, `"completion_ids"`, and
  `"logprobs"` fields, and can optionally return `"logprob_token_ids"` (same shape as `"logprobs"`). Any
  other fields are forwarded to the reward functions. The function receives the raw per-process prompt slice
  with no duplication; it is responsible for returning the correct number of completions per prompt (see
  `num_generations` / `num_generations_eval` on the trainer). This feature is experimental and may change or
  be removed at any time without prior notice.
- **environment_factory** (`EnvironmentFactory`, *optional*) --
  A callable that creates and returns an environment instance. The environment class should define methods
  that can be invoked as tools during generation. Each method should comply with the same requirements as the
  `tools` described above. If `environment_factory` is provided, an instance of the environment is created
  for each generation in the batch, allowing for parallel and independent interactions. The environment must
  also implement a callable `reset` method that can be used to reset state between generations. The `reset`
  method should return either `None` or a string: when it returns a string, that string is appended to the
  last user message before generation. This feature is experimental and may change or be removed at any time
  without prior notice.

Trainer for the Group Relative Policy Optimization (GRPO) method. This algorithm was initially proposed in the
paper [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language
Models](https://huggingface.co/papers/2402.03300).

Example:

```python
>>> from trl import GRPOTrainer
>>> from trl.rewards import accuracy_reward
>>> from datasets import load_dataset

>>> dataset = load_dataset("trl-lib/DeepMath-103K", split="train")

>>> trainer = GRPOTrainer(
...     model="Qwen/Qwen2.5-0.5B-Instruct",
...     reward_funcs=accuracy_reward,
...     train_dataset=dataset,
... )
>>> trainer.train()
```

- **resume_from_checkpoint** (`str` or `bool`, *optional*) --
  If a `str`, local path to a saved checkpoint as saved by a previous instance of `Trainer`. If a
  `bool` and equals `True`, load the last checkpoint in *args.output_dir* as saved by a previous instance
  of `Trainer`. If present, training will resume from the model/optimizer/scheduler states loaded here.
- **trial** (`optuna.Trial` or `dict[str, Any]`, *optional*) --
  The trial run or the hyperparameter dictionary for hyperparameter search.
- **ignore_keys_for_eval** (`list[str]`, *optional*) --
  A list of keys in the output of your model (if it is a dictionary) that should be ignored when
  gathering predictions for evaluation during the training.`~trainer_utils.TrainOutput`Object containing the global step count, training loss, and metrics.

Main training entry point.

Will save the model, so you can reload it using `from_pretrained()`.

Will only save from the main process.

- **commit_message** (`str`, *optional*, defaults to `"End of training"`) --
  Message to commit while pushing.
- **blocking** (`bool`, *optional*, defaults to `True`) --
  Whether the function should return only when the `git push` has finished.
- **token** (`str`, *optional*, defaults to `None`) --
  Token with write permission to overwrite Trainer's original args.
- **revision** (`str`, *optional*) --
  The git revision to commit from. Defaults to the head of the "main" branch.
- **kwargs** (`dict[str, Any]`, *optional*) --
  Additional keyword arguments passed along to `~Trainer.create_model_card`.The URL of the repository where the model was pushed if `blocking=False`, or a `Future` object tracking the
progress of the commit if `blocking=True`.

Upload `self.model` and `self.processing_class` to the 🤗 model hub on the repo `self.args.hub_model_id`.

## GRPOConfig[[trl.GRPOConfig]]

"}, {"name": "batch_eval_metrics", "val": ": bool = False"}, {"name": "save_only_model", "val": ": bool = False"}, {"name": "save_strategy", "val": ": transformers.trainer_utils.SaveStrategy | str = 'steps'"}, {"name": "save_steps", "val": ": float = 500"}, {"name": "save_on_each_node", "val": ": bool = False"}, {"name": "save_total_limit", "val": ": int | None = None"}, {"name": "enable_jit_checkpoint", "val": ": bool = False"}, {"name": "push_to_hub", "val": ": bool = False"}, {"name": "hub_token", "val": ": str | None = None"}, {"name": "hub_private_repo", "val": ": bool | None = None"}, {"name": "hub_model_id", "val": ": str | None = None"}, {"name": "hub_strategy", "val": ": transformers.trainer_utils.HubStrategy | str = 'every_save'"}, {"name": "hub_always_push", "val": ": bool = False"}, {"name": "hub_revision", "val": ": str | None = None"}, {"name": "load_best_model_at_end", "val": ": bool = False"}, {"name": "metric_for_best_model", "val": ": str | None = None"}, {"name": "greater_is_better", "val": ": bool | None = None"}, {"name": "ignore_data_skip", "val": ": bool = False"}, {"name": "restore_callback_states_from_checkpoint", "val": ": bool = False"}, {"name": "full_determinism", "val": ": bool = False"}, {"name": "seed", "val": ": int = 42"}, {"name": "data_seed", "val": ": int | None = None"}, {"name": "use_cpu", "val": ": bool = False"}, {"name": "accelerator_config", "val": ": dict | str | None = None"}, {"name": "parallelism_config", "val": ": accelerate.parallelism_config.ParallelismConfig | None = None"}, {"name": "dataloader_drop_last", "val": ": bool = False"}, {"name": "dataloader_num_workers", "val": ": int = 0"}, {"name": "dataloader_pin_memory", "val": ": bool = True"}, {"name": "dataloader_persistent_workers", "val": ": bool = False"}, {"name": "dataloader_prefetch_factor", "val": ": int | None = None"}, {"name": "remove_unused_columns", "val": ": bool | None = False"}, {"name": "label_names", "val": ": list[str] | None = None"}, {"name": "train_sampling_strategy", "val": ": str = 'random'"}, {"name": "length_column_name", "val": ": str = 'length'"}, {"name": "ddp_find_unused_parameters", "val": ": bool | None = None"}, {"name": "ddp_bucket_cap_mb", "val": ": int | None = None"}, {"name": "ddp_broadcast_buffers", "val": ": bool | None = None"}, {"name": "ddp_static_graph", "val": ": bool | None = None"}, {"name": "ddp_backend", "val": ": str | None = None"}, {"name": "ddp_timeout", "val": ": int = 1800"}, {"name": "fsdp", "val": ": str | None = None"}, {"name": "fsdp_config", "val": ": dict[str, typing.Any] | str | None = None"}, {"name": "deepspeed", "val": ": dict | str | None = None"}, {"name": "debug", "val": ": str | list[transformers.debug_utils.DebugOption] = ''"}, {"name": "skip_memory_metrics", "val": ": bool = True"}, {"name": "do_train", "val": ": bool = False"}, {"name": "do_eval", "val": ": bool = False"}, {"name": "do_predict", "val": ": bool = False"}, {"name": "resume_from_checkpoint", "val": ": str | None = None"}, {"name": "warmup_ratio", "val": ": float | None = None"}, {"name": "logging_dir", "val": ": str | None = None"}, {"name": "local_rank", "val": ": int = -1"}, {"name": "model_init_kwargs", "val": ": dict[str, typing.Any] | str | None = None"}, {"name": "trust_remote_code", "val": ": bool = False"}, {"name": "router_aux_loss_coef", "val": ": float = 0.001"}, {"name": "disable_dropout", "val": ": bool = False"}, {"name": "cast_lm_head_to_fp32", "val": ": bool = False"}, {"name": "num_generations", "val": ": int | None = 8"}, {"name": "num_generations_eval", "val": ": int | None = None"}, {"name": "max_completion_length", "val": ": int | None = 256"}, {"name": "ds3_gather_for_generation", "val": ": bool = True"}, {"name": "shuffle_dataset", "val": ": bool | None = True"}, {"name": "pad_to_multiple_of", "val": ": int | None = None"}, {"name": "generation_batch_size", "val": ": int | None = None"}, {"name": "steps_per_generation", "val": ": int | None = None"}, {"name": "temperature", "val": ": float = 1.0"}, {"name": "top_p", "val": ": float = 1.0"}, {"name": "top_k", "val": ": int = 0"}, {"name": "min_p", "val": ": float | None = None"}, {"name": "generation_kwargs", "val": ": dict | None = None"}, {"name": "chat_template_kwargs", "val": ": dict | None = None"}, {"name": "repetition_penalty", "val": ": float = 1.0"}, {"name": "cache_implementation", "val": ": str | None = None"}, {"name": "use_vllm", "val": ": bool = False"}, {"name": "vllm_mode", "val": ": str = 'colocate'"}, {"name": "vllm_model_impl", "val": ": str = 'vllm'"}, {"name": "vllm_enable_sleep_mode", "val": ": bool = False"}, {"name": "vllm_structured_outputs_regex", "val": ": str | None = None"}, {"name": "vllm_server_base_url", "val": ": str | None = None"}, {"name": "vllm_server_host", "val": ": str = '0.0.0.0'"}, {"name": "vllm_server_port", "val": ": int = 8000"}, {"name": "vllm_server_timeout", "val": ": float = 240.0"}, {"name": "vllm_group_port", "val": ": int = 51216"}, {"name": "vllm_gpu_memory_utilization", "val": ": float = 0.3"}, {"name": "vllm_max_model_length", "val": ": int | None = None"}, {"name": "vllm_tensor_parallel_size", "val": ": int = 1"}, {"name": "beta", "val": ": float = 0.0"}, {"name": "num_iterations", "val": ": int = 1"}, {"name": "epsilon", "val": ": float = 0.2"}, {"name": "delta", "val": ": float | None = None"}, {"name": "epsilon_high", "val": ": float | None = None"}, {"name": "sapo_temperature_neg", "val": ": float = 1.05"}, {"name": "sapo_temperature_pos", "val": ": float = 1.0"}, {"name": "vespo_k_pos", "val": ": float = 2.0"}, {"name": "vespo_lambda_pos", "val": ": float = 3.0"}, {"name": "vespo_k_neg", "val": ": float = 3.0"}, {"name": "vespo_lambda_neg", "val": ": float = 2.0"}, {"name": "importance_sampling_level", "val": ": str = 'token'"}, {"name": "reward_weights", "val": ": list[float] | None = None"}, {"name": "multi_objective_aggregation", "val": ": str = 'sum_then_normalize'"}, {"name": "scale_rewards", "val": ": str = 'group'"}, {"name": "loss_type", "val": ": str = 'dapo'"}, {"name": "mask_truncated_completions", "val": ": bool = False"}, {"name": "sync_ref_model", "val": ": bool = False"}, {"name": "ref_model_mixup_alpha", "val": ": float = 0.6"}, {"name": "ref_model_sync_steps", "val": ": int = 512"}, {"name": "top_entropy_quantile", "val": ": float = 1.0"}, {"name": "entropy_coef", "val": ": float = 0.0"}, {"name": "use_adaptive_entropy", "val": ": bool = False"}, {"name": "entropy_coef_min", "val": ": float = 0.0"}, {"name": "entropy_coef_max", "val": ": float = 1.0"}, {"name": "entropy_coef_delta", "val": ": float = 0.005"}, {"name": "entropy_target", "val": ": float = 0.2"}, {"name": "max_tool_calling_iterations", "val": ": int | None = None"}, {"name": "vllm_importance_sampling_correction", "val": ": bool = True"}, {"name": "vllm_importance_sampling_mode", "val": ": str = 'sequence_mask'"}, {"name": "vllm_importance_sampling_clip_max", "val": ": float | None = 3.0"}, {"name": "vllm_importance_sampling_clip_min", "val": ": float | None = None"}, {"name": "off_policy_mask_threshold", "val": ": float | None = None"}, {"name": "use_bias_correction_kl", "val": ": bool = False"}, {"name": "log_completions", "val": ": bool = False"}, {"name": "num_completions_to_print", "val": ": int | None = None"}, {"name": "log_unique_prompts", "val": ": bool = False"}, {"name": "log_completions_hub_repo", "val": ": str | None = None"}, {"name": "use_transformers_continuous_batching", "val": ": bool = False"}, {"name": "transformers_continuous_batching_config", "val": ": dict | None = None"}, {"name": "use_transformers_paged", "val": ": bool = False"}, {"name": "vllm_importance_sampling_cap", "val": ": float | None = None"}]}>
Parameters that control the model and reference model

- **model_init_kwargs** (`str`, `dict[str, Any]`, *optional*) --
  Keyword arguments for [from_pretrained](https://huggingface.co/docs/transformers/main/en/model_doc/auto#transformers.AutoModelForCausalLM.from_pretrained), used when the `model`
  argument of the [GRPOTrainer](/docs/trl/main/en/grpo_trainer#trl.GRPOTrainer) is provided as a string.
- **trust_remote_code** (`bool`, *optional*, defaults to `False`) --
  Whether to allow loading models and tokenizers that ship custom Python code from the Hub. Forwarded to
  [from_pretrained](https://huggingface.co/docs/transformers/main/en/model_doc/auto#transformers.AutoModelForCausalLM.from_pretrained) and
  [from_pretrained](https://huggingface.co/docs/transformers/main/en/model_doc/auto#transformers.AutoProcessor.from_pretrained). Also applied to reward-model and reward-tokenizer loads.
- **router_aux_loss_coef** (`float`, *optional*, defaults to `0.001`) --
  Coefficient of the load-balancing auxiliary loss. Only has an effect when training a Mixture-of-Experts
  (MoE) model; for other models it does nothing. The auxiliary loss is added to the training loss with this
  weight. Set to `0.0` to disable it.
- **disable_dropout** (`bool`, *optional*, defaults to `False`) --
  Whether to disable dropout in the model. This is useful for training with a reference model, as it prevents
  the model from generating different logprobs for the same input.
- **cast_lm_head_to_fp32** (`bool`, *optional*, defaults to `False`) --
  Whether to cast the language modeling head of the policy and reference models to float32. As recommended by
  the [ScaleRL](https://huggingface.co/papers/2510.13786) recipe. This flag is only supported when the model
  has untied word embedding and language modeling head layers i.e. `tie_word_embeddings` in the model config
  is False.

Parameters that control the data preprocessing

- **remove_unused_columns** (`bool`, *optional*, defaults to `False`) --
  Whether to only keep the column `"prompt"` in the dataset. If you use a custom reward function that
  requires any column other than `"prompts"` and `"completions"`, you should keep this to `False`.
- **num_generations** (`int`, *optional*, defaults to `8`) --
  Number of generations per prompt to sample. The effective batch size (num_processes * per_device_batch_size
  * gradient_accumulation_steps) must be evenly divisible by this value.
- **num_generations_eval** (`int` or `None`, *optional*) --
  Number of generations to sample during evaluation. This allows using fewer generations during evaluation to
  save computation. If `None`, uses the value of `num_generations`.
- **max_completion_length** (`int` or `None`, *optional*, defaults to `256`) --
  Maximum length of the generated completion.
- **ds3_gather_for_generation** (`bool`, *optional*, defaults to `True`) --
  This setting applies to DeepSpeed ZeRO-3. If enabled, the policy model weights are gathered for generation,
  improving generation speed. However, disabling this option allows training models that exceed the VRAM
  capacity of a single GPU, albeit at the cost of slower generation. Disabling this option is not compatible
  with vLLM generation.
- **shuffle_dataset** (`bool`, *optional*, defaults to `True`) --
  Whether to shuffle the training dataset.
- **pad_to_multiple_of** (`int`, *optional*) --
  If set, the prompts ids and completions ids will be padded to a multiple of this value.

Parameters that control generation

- **generation_batch_size** (`int`, *optional*) --
  Batch size to use for generation. If `None`, it defaults to the effective training batch size:
  `per_device_train_batch_size * num_processes * steps_per_generation`. In other words, there is one
  generation batch processed per optimization step. Mutually exclusive with `steps_per_generation`.
- **steps_per_generation** (`int`, *optional*) --
  Number of steps per generation. If `None`, it defaults to `gradient_accumulation_steps`. Mutually exclusive
  with `generation_batch_size`.
- **temperature** (`float`, defaults to `1.0`) --
  Temperature for sampling. The higher the temperature, the more random the completions.
- **top_p** (`float`, *optional*, defaults to `1.0`) --
  Float that controls the cumulative probability of the top tokens to consider. Must be in (0, 1]. Set to
  `1.0` to consider all tokens.
- **top_k** (`int`, *optional*, defaults to `0`) --
  Number of highest probability vocabulary tokens to keep for top-k-filtering. If `0`, top-k-filtering is
  disabled and all tokens are considered.
- **min_p** (`float`, *optional*) --
  Minimum token probability, which will be scaled by the probability of the most likely token. It must be a
  value between `0.0` and `1.0`. Typical values are in the `0.01-0.2` range.
- **generation_kwargs** (`dict[str, Any]`, *optional*) --
  Additional keyword arguments to pass to [GenerationConfig](https://huggingface.co/docs/transformers/main/en/main_classes/text_generation#transformers.GenerationConfig) (if using transformers) or
  `SamplingParams` (if using vLLM) when sampling completions. This can be used to further customize the
  generation behavior, such as setting `suppress_tokens`, `num_beams`, etc. If it contains keys that conflict
  with the other generation parameters (like `min_p`, `top_p`, etc.), they will override them.
- **chat_template_kwargs** (`dict[str, Any]`, *optional*) --
  Additional keyword arguments to pass to the `apply_chat_template` function when generating completions.
- **repetition_penalty** (`float`, *optional*, defaults to `1.0`) --
  Float that penalizes new tokens based on whether they appear in the prompt and the generated text so far.
  Values > `1.0` encourage the model to use new tokens, while values < `1.0` encourage the model to repeat
  tokens.
- **cache_implementation** (`str`, *optional*) --
  Implementation of the cache method for faster generation when `use_vllm` is set to `False`.

Parameters that control generation acceleration powered by vLLM

- **use_vllm** (`bool`, *optional*, defaults to `False`) --
  Whether to use vLLM for generating completions. If set to `True`, the trainer will use vLLM for generation
  instead of the default model.generate(). Requires `vllm` to be installed.
- **vllm_mode** (`str`, *optional*, defaults to `"colocate"`) --
  Mode to use for vLLM integration when `use_vllm` is set to `True`. Must be one of `"server"` or
  `"colocate"`.

  - `"server"`: The trainer will send generation requests to a separate vLLM server. Make sure a TRL vLLM
    server is running (start with `trl vllm-serve`).
  - `"colocate"`: vLLM will run in the same process and share the training GPUs. This avoids the need for a
    separate server but may cause resource contention with training.
- **vllm_model_impl** (`str`, *optional*, defaults to `"vllm"`) --
  Model implementation to use for vLLM. Must be one of `"transformers"` or `"vllm"`. `"transformers"`: Use
  the `transformers` backend for model implementation. `"vllm"`: Use the `vllm` library for model
  implementation.
- **vllm_structured_outputs_regex** (`str`, *optional*) --
  Regex for vLLM structured outputs. If `None` (default), structured outputs is disabled.

Parameters that control the vLLM server (only used when `vllm_mode` is `"server"`)

- **vllm_server_base_url** (`str`, *optional*) --
  Base URL for the vLLM server (e.g., `"http://localhost:8000"`). If provided, `vllm_server_host` and
  `vllm_server_port` are ignored.
- **vllm_server_host** (`str`, *optional*, defaults to `"0.0.0.0"`) --
  Host of the vLLM server to connect to. Ignored if `vllm_server_base_url` is provided.
- **vllm_server_port** (`int`, *optional*, defaults to `8000`) --
  Port of the vLLM server to connect to. Ignored if `vllm_server_base_url` is provided.
- **vllm_server_timeout** (`float`, *optional*, defaults to `240.0`) --
  Total timeout duration in seconds to wait for the vLLM server to be up. If the server is not up after the
  timeout, a `ConnectionError` is raised.
- **vllm_group_port** (`int`, *optional*, defaults to `51216`) --
  Port number for the weight update group. This is used to communicate with the vLLM server. Unless the port
  is occupied, there is no need to change it.

Parameters that control colocated vLLM execution (only used when `vllm_mode` is `"colocate"`)

- **vllm_gpu_memory_utilization** (`float`, *optional*, defaults to `0.3`) --
  Control the GPU memory utilization for vLLM. This setting only applies when `vllm_mode` is set to
  `"colocate"`. If you are using `vllm_mode="server"`, this parameter must be passed separately when
  launching the vLLM server via the `--vllm_gpu_memory_utilization` flag.
- **vllm_max_model_length** (`int`, *optional*) --
  Context window for vLLM. Set it to at least the maximum prompt length in the dataset plus
  `max_completion_length`; if omitted, it is inferred from the model config.
- **vllm_tensor_parallel_size** (`int`, *optional*, defaults to `1`) --
  Control the tensor parallel size for vLLM. This setting only applies when `vllm_mode` is set to
  `"colocate"`. If you are using `vllm_mode="server"`, this parameter must be passed separately when
  launching the vLLM server via the `--vllm_tensor_parallel_size` flag.
- **vllm_enable_sleep_mode** (`bool`, *optional*, defaults to `False`) --
  Enable vLLM sleep mode to offload weights/cache during the optimizer step. Keeps GPU memory usage low, but
  waking the engine adds host–device transfer latency.

Parameters that control generation acceleration powered by transformers continuous batching

- **use_transformers_continuous_batching** (`bool`, *optional*, defaults to `False`) --
  Whether to use transformers' continuous batching engine for generating completions. Requires
  `transformers>=5.8.0`.
- **transformers_continuous_batching_config** (`dict`, *optional*) --
  Keyword arguments for `ContinuousBatchingConfig`.

Parameters that control the training

- **beta** (`float`, *optional*, defaults to `0.0`) --
  KL coefficient. If `0.0` (default), the reference model is not loaded, reducing memory usage and improving
  training speed. [DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement
  learning](https://huggingface.co/papers/2501.12948) use a value of `0.001`.
- **num_iterations** (`int`, *optional*, defaults to `1`) --
  Number of iterations per batch (denoted as μ in the algorithm).
- **epsilon** (`float`, *optional*, defaults to `0.2`) --
  Epsilon value for clipping.
- **delta** (`float`, *optional*) --
  Enables the upper clipping bound in two-sided GRPO loss when set to a float. If `None` (default), standard
  GRPO clipping is used. Recommended to be greater than `1 + ε` when enabled. This method is introduced in
  the [INTELLECT-2 tech report](https://huggingface.co/papers/2505.07291).
- **epsilon_high** (`float`, *optional*) --
  Upper-bound epsilon value for clipping. If not specified, it defaults to the same value as the lower-bound
  specified in argument `epsilon`. Paper [DAPO](https://huggingface.co/papers/2503.14476) recommends `0.28`.
  When used with `loss_type='cispo'`, this corresponds to the ε_max param specified in the [ScaleRL
  paper](https://huggingface.co/papers/2510.13786) and the recommended value is `5.0`.
- **sapo_temperature_neg** (`float`, *optional*, defaults to `1.05`) --
  Temperature for tokens with non-positive advantage scores used in the `sapo` loss function. This parameter
  is introduced in the [Soft Adaptive Policy Optimization paper](https://huggingface.co/papers/2511.20347).
- **sapo_temperature_pos** (`float`, *optional*, defaults to `1.0`) --
  Temperature for tokens with positive advantage scores used in the `sapo` loss function. This parameter is
  introduced in the [Soft Adaptive Policy Optimization paper](https://huggingface.co/papers/2511.20347).
- **vespo_k_pos** (`float`, *optional*, defaults to `2.0`) --
  k parameter for positive advantages, it is the power exponent in the VESPO loss. Controls how aggressively
  we down-weight samples with low importance weights (when the importance sampling ratio < 1).
- **vespo_lambda_pos** (`float`, *optional*, defaults to `3.0`) --
  lambda parameter for positive advantages, it is the decay factor in the VESPO loss. Controls how
  aggressively we down-weight samples with high importance weights (when the importance sampling ratio > 1).
- **vespo_k_neg** (`float`, *optional*, defaults to `3.0`) --
  k parameter for negative advantages, it is the power exponent in the VESPO loss. Controls how aggressively
  we down-weight samples with low importance weights (when the importance sampling ratio < 1).
- **vespo_lambda_neg** (`float`, *optional*, defaults to `2.0`) --
  lambda parameter for negative advantages, it is the exponential decay factor in the VESPO loss. Controls
  how aggressively we down-weight samples with high importance weights (when the importance sampling ratio >
  1).
- **importance_sampling_level** (`str`, *optional*, defaults to `"token"`) --
  Controls whether importance sampling ratios are computed at the `"token"` or `"sequence"` level. `"token"`
  keeps the raw per-token log-probability ratios (one weight per token). `"sequence"` averages the
  log-probability ratios across valid tokens to produce a single ratio per sequence. The [GSPO
  paper](https://huggingface.co/papers/2507.18071) shows that sequence-level sampling often yields more
  stable training and better alignment with sequence-level rewards.
- **reward_weights** (`list[float]`, *optional*) --
  Weights for each reward function. Must match the number of reward functions. If `None`, all rewards are
  weighted equally with weight `1.0`.
- **multi_objective_aggregation** (`str`, *optional*, defaults to `"sum_then_normalize"`) --
  Method to aggregate multiple reward functions. Supported values are:

  - `"sum_then_normalize"` (default): First sums the weighted rewards from each reward function, then applies
    reward scaling/normalization as specified by `scale_rewards` (see `scale_rewards` for details).
  - `"normalize_then_sum"`: First normalizes/scales each reward function across generations (within each
    group), then sums the normalized rewards using the specified weights. The aggregated reward is then
    normalized at the batch level when forming advantages. This is the suggested approach from the paper
    [GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL
    Optimization](https://huggingface.co/papers/2601.05242).
- **scale_rewards** (`str` or `bool`, *optional*, defaults to `"group"`) --
  Specifies the scaling strategy for rewards. Supported values are:

  - `True` or `"group"` (default): rewards are scaled by the standard deviation within each group, ensuring
    unit variance within a group.
  - `"batch"`: rewards are scaled by the standard deviation across the entire batch, as recommended in the
    [PPO Lite paper](https://huggingface.co/papers/2508.08221).
  - `False` or `"none"`: no scaling is applied. The [Dr. GRPO
    paper](https://huggingface.co/papers/2503.20783) recommends not scaling rewards, as scaling by the
    standard deviation introduces a question-level difficulty bias.
- **loss_type** (`str`, *optional*, defaults to `"dapo"`) --
  Specifies the loss formulation to use. Supported values are:

  - `"grpo"`: Aggregates token-level losses by normalizing over sequence length. Not recommended due to
    length bias—this approach tends to prefer shorter completions with positive advantages and longer ones
    with negative advantages.
  - `"dr_grpo"`: Aggregates token-level losses by normalizing with a global constant. This method was
    introduced in the [Dr. GRPO paper](https://huggingface.co/papers/2503.20783) to eliminate length bias.
    The value of the constant corresponds to `max_completion_length`.
  - `"dapo"` (default): Aggregates token-level losses by normalizing with the number of active token in the
    global accumulated batch. This method was introduced in the [DAPO
    paper](https://huggingface.co/papers/2503.14476) to eliminate length bias.
  - `"bnpo"`: Aggregates token-level losses by normalizing with the number of active token in the local
    batch. Note that normalization is performed over the local batch only, so results may slightly vary
    depending on the local batch size, despite a constant effective batch size. When using
    `per_device_train_batch_size==1`, the loss is equivalent to the GRPO loss.
  - `"cispo"`: Clips the importance sampling weights instead of the advantage scaled importance weights. The
    clipped weights are then multiplied with the advantages and policy model's log probs. Individual token
    losses are aggregated by normalizing with the number of active tokens in the global accumulated batch.
    This method was introduced in the [MiniMax-M1 paper](https://huggingface.co/papers/2506.13585).
  - `"sapo"`: Soft Adaptive Policy Optimization loss, as introduced in the [Soft Adaptive Policy Optimization
    paper](https://huggingface.co/papers/2511.20347). Replaces hard clipping with a smooth,
    temperature-controlled gate that adaptively attenuates off-policy updates while preserving useful
    learning signals.
  - `"luspo"`: Length-Unbiased Sequence Policy Optimization loss. A sequence-level loss that scales each
    sequence's loss by its length. This is a modification of GSPO and requires
    `importance_sampling_level="sequence"`. Introduced in the [LUSPO
    paper](https://huggingface.co/papers/2602.05261).
  - `"vespo"`: Variational Sequence-Level Soft Policy Optimization. Replaces hard clipping with a smooth,
    asymmetric Gamma weighting function applied directly to sequence-level importance weights. Introduced in
    the [VESPO paper](https://huggingface.co/papers/2602.10693).
- **mask_truncated_completions** (`bool`, *optional*, defaults to `False`) --
  When enabled, truncated completions are excluded from the loss calculation, preventing them from being
  incorrectly penalized and introducing noise during training. According to the
  [DAPO](https://huggingface.co/papers/2503.14476) paper, this is a good practice for training stability.
- **sync_ref_model** (`bool`, *optional*, defaults to `False`) --
  Whether to synchronize the reference model with the active model every `ref_model_sync_steps` steps, using
  the `ref_model_mixup_alpha` parameter. This synchronization originates from the
  [TR-DPO](https://huggingface.co/papers/2404.09656) paper.
- **ref_model_mixup_alpha** (`float`, *optional*, defaults to `0.6`) --
  α parameter from the [TR-DPO](https://huggingface.co/papers/2404.09656) paper, which controls the mix
  between the current policy and the previous reference policy during updates. The reference policy is
  updated according to the equation: `π_ref = α * π_θ + (1 - α) * π_ref_prev`. To use this parameter, you
  must set `sync_ref_model=True`.
- **ref_model_sync_steps** (`int`, *optional*, defaults to `512`) --
  τ parameter from the [TR-DPO](https://huggingface.co/papers/2404.09656) paper, which determines how
  frequently the current policy is synchronized with the reference policy. To use this parameter, you must
  set `sync_ref_model=True`.
- **top_entropy_quantile** (`float`, *optional*, defaults to `1.0`) --
  ρ parameter from [Beyond the 80/20 Rule](https://huggingface.co/papers/2506.01939). Keeps in the policy
  loss term only the top-ρ quantile of tokens by entropy of the probability distribution at each sequence
  position, improving results. Range: `[0.0-1.0]`. A value of `0.0` masks all but the highest entropy token;
  `1.0` keeps all tokens. The paper recommends a value of `0.2`. If used with
  `mask_truncated_completions=True`, only tokens from non-truncated completions are considered.
- **entropy_coef** (`float`, *optional*, defaults to `0.0`) --
  Coefficient of the entropy regularization term in the loss. A positive value adds an entropy bonus that
  encourages exploration by keeping the policy from collapsing to near-deterministic outputs. The bonus is
  always the mean per-token entropy regardless of `loss_type`; it is not rescaled to match a loss type's
  policy normalization, so `entropy_coef` has the same meaning for every loss type. When
  `use_adaptive_entropy=True`, this serves as the initial coefficient and is updated each optimizer step.
  Has no effect when set to `0.0` (default).
- **use_adaptive_entropy** (`bool`, *optional*, defaults to `False`) --
  Whether to use adaptive entropy control, introduced in
  [Skywork-OR1](https://huggingface.co/papers/2505.22312). When enabled, the entropy coefficient
  `entropy_coef` is updated each optimizer step: incremented by `entropy_coef_delta` when the current
  entropy is below `entropy_target`, and decremented otherwise. The coefficient is only applied when
  entropy is at or below `entropy_target`.
- **entropy_coef_min** (`float`, *optional*, defaults to `0.0`) --
  Lower bound for the entropy coefficient when using adaptive entropy control.
- **entropy_coef_max** (`float`, *optional*, defaults to `1.0`) --
  Upper bound for the entropy coefficient when using adaptive entropy control.
- **entropy_coef_delta** (`float`, *optional*, defaults to `0.005`) --
  Step size for adjusting the entropy coefficient at each optimizer step during adaptive entropy control.
- **entropy_target** (`float`, *optional*, defaults to `0.2`) --
  Target mean per-token entropy (in nats) used by adaptive entropy control. The coefficient is only
  applied when the current entropy falls at or below this value. Measured over the same token set as
  the policy loss: all completion tokens by default, or only the high-entropy subset when
  `top_entropy_quantile < 1.0`. Typical language models have per-token entropies in the range 2–10
  nats, so the default of `0.2` almost never triggers regularization (only on near-complete entropy
  collapse); set it close to the entropy you observe early in training (logged as the `entropy`
  metric) so the bonus engages before the policy collapses (and account for the token subset when
  using `top_entropy_quantile`).
- **max_tool_calling_iterations** (`int`, *optional*) --
  Maximum number of tool-calling turns when training an agent. If `None`, there is no limit and generation
  stops when the model generates a response turn with no tool calls or when the total response length reaches
  `max_model_length`.
- **vllm_importance_sampling_correction** (`bool`, *optional*, defaults to `True`) --
  Whether to apply Importance Sampling (IS) to correct for the mismatch between vLLM completion logprobs and
  recomputed training logprobs. If set to `False`, no IS is applied regardless of
  `vllm_importance_sampling_mode`. When `True`, the selected mode determines how the IS ratios are computed
  and constrained.
- **vllm_importance_sampling_mode** (`str`, *optional*, defaults to `"sequence_mask"`) --
  Specifies how Importance Sampling is performed when `vllm_importance_sampling_correction=True`. Possible
  values are:

  - `"token_truncate"`: Token-level truncated IS (default). Per-token ratios are clipped to
  [C_min, C_max].
  - `"token_mask"`: Token-level masked IS. Per-token ratios outside [C_min, C_max] are set to zero.
  - `"sequence_truncate"`: Sequence-level truncated IS. A single sequence ratio is clipped to
  [C_min, C_max] and applied to all tokens in the sequence.
  - `"sequence_mask"`: Sequence-level masked IS. Sequences with ratios outside [C_min, C_max] are masked
  out.
- **vllm_importance_sampling_clip_max** (`float`, *optional*, defaults to `3.0`) --
  Importance sampling upper bound C_max used by `vllm_importance_sampling_mode`. For `*_truncate` modes,
  importance ratios are clipped from above at C_max. For `*_mask` modes, ratios larger than C_max are set to
  zero.
- **vllm_importance_sampling_clip_min** (`float`, *optional*) --
  Importance sampling lower bound C_min used by `vllm_importance_sampling_mode`. For `*_truncate` modes,
  ratios are clipped from below at C_min. For `*_mask` modes, ratios below C_min are set to zero. To strictly
  mask ratios below C_min without upper bound, set `vllm_importance_sampling_clip_max=None`.
- **off_policy_mask_threshold** (`float`, *optional*) --
  Threshold for off-policy sequence masking. If `None`, off-policy sequence masking is disabled. When set,
  sequences with negative advantages and high KL divergence are masked out to stabilize training. This
  parameter corresponds to the `delta` threshold in Equation 9 of the [DeepSeek-V3.2
  paper](https://huggingface.co/papers/2512.02556). It expects a positive value (e.g., 0.5).
- **use_bias_correction_kl** (`bool`, *optional*, defaults to `False`) --
  Whether to use the unbiased KL divergence estimator with importance sampling correction. This corrects the
  KL divergence estimate by multiplying it with the importance sampling ratio. This is described in the
  [DeepSeek-V3.2 paper](https://huggingface.co/papers/2512.02556).

Parameters that control the logging

- **log_completions** (`bool`, *optional*, defaults to `False`) --
  Whether to log a sample of (prompt, completion) pairs every `logging_steps` steps. If `rich` is installed,
  it prints the sample. If `wandb` and/or `trackio` logging is enabled, it logs it to `wandb` and/or
  `trackio`.
- **num_completions_to_print** (`int`, *optional*) --
  Number of completions to print with `rich`. If `None`, all completions are logged.
- **log_unique_prompts** (`bool`, *optional*, defaults to `False`) --
  Whether to log unique prompts. If `True`, only unique prompts are logged. If `False`, all prompts are
  logged.
- **log_completions_hub_repo** (`str`, *optional*) --
  Hugging Face Hub repository to save the completions. Should be a complete repository name like
  `'username/reponame'` or `'orgname/reponame'`, or just `'reponame'` in which case the repository will be
  created in the currently-logged-in Hugging Face user's namespace. Note that this repository will be public
  unless you set `hub_private_repo=True` or your organization's default is to create private repositories."

Deprecated parameters

- **use_transformers_paged** --

  

  Parameter `use_transformers_paged` is deprecated and will be removed in version v2.0.0. Use
  `use_transformers_continuous_batching` instead.

  

- **vllm_importance_sampling_cap** --

  

  Parameter `vllm_importance_sampling_cap` is deprecated and will be removed in v2.0.0. Use
  `vllm_importance_sampling_clip_max` instead.

  

Configuration class for the [GRPOTrainer](/docs/trl/main/en/grpo_trainer#trl.GRPOTrainer).

This class includes only the parameters that are specific to GRPO training. For a full list of training arguments,
please refer to the [TrainingArguments](https://huggingface.co/docs/transformers/main/en/main_classes/trainer#transformers.TrainingArguments) documentation. Note that default values in this class may
differ from those in [TrainingArguments](https://huggingface.co/docs/transformers/main/en/main_classes/trainer#transformers.TrainingArguments).

Using [HfArgumentParser](https://huggingface.co/docs/transformers/main/en/internal/trainer_utils#transformers.HfArgumentParser) we can turn this class into
[argparse](https://docs.python.org/3/library/argparse#module-argparse) arguments that can be specified on the
command line.

> [!NOTE]
> These parameters have default values different from [TrainingArguments](https://huggingface.co/docs/transformers/main/en/main_classes/trainer#transformers.TrainingArguments):
> - `logging_steps`: Defaults to `10` instead of `500`.
> - `gradient_checkpointing`: Defaults to `True` instead of `False`.
> - `bf16`: Defaults to `True` if `fp16` is not set, instead of `False`.
> - `learning_rate`: Defaults to `1e-6` instead of `5e-5`.