feature: add experimental DAPO support

[CharlesXu-HQ]

Motivation

AReno currently provides GRPO as a stable policy-only training algorithm. DAPO (Decoupled Clip and Dynamic sAmpling Policy Optimization) extends this family with four coupled techniques intended to improve long-CoT RL training stability and sample utilization:

1. asymmetric clipping (Clip-Higher);
2. dynamic sampling that removes prompt groups with no reward variation;
3. token-level policy-loss aggregation; and
4. soft overlong reward shaping.

Dynamic sampling trades additional rollout generation for avoiding optimizer updates on all-correct or all-incorrect groups. The proposal would keep this behavior opt-in under an experimental algorithm and would not change the existing GRPO/GSPO objectives.

This is more than an asymmetric-clipping loss change. AReno's current GRPO/GSPO objectives intentionally use a unit-valued differentiable surrogate ratio, while DAPO's Clip-Higher only has a numerical effect when the policy ratio is computed against stored old-policy logprobs. The DAPO implementation would therefore use real rollout old logprobs only inside the experimental dapo loss. This is related to, but deliberately narrower than, #68.

Proposed feature

Add dapo under areno.experimental with the following scope:

• register dapo through AlgorithmSpec without changing the stable algorithm list;
• compute the token ratio as exp(train_logprob - rollout_old_logprob), with bounded log-ratio values for numerical safety;
• implement Clip-Higher with paper defaults clip_eps_low=0.2 and clip_eps_high=0.28;
• preserve a response-token mean across packed batches, gradient-accumulation groups, and data-parallel sharding;
• generate candidate prompt groups until batch_size qualified groups have been collected or a bounded generation-attempt limit is reached;
• decide group eligibility from the unshaped scalar rewards returned by the existing reward function, retaining groups whose rewards are not all equal;
• apply the linear overlong penalty only after group eligibility is decided, then compute the usual group-standardized advantages;
• keep prompt groups intact when truncating the qualified buffer;
• expose bounded, additive DAPO configuration for generation batch size, maximum generation batches, asymmetric clip values, and overlong shaping;
• reject agentic rollouts in the first version because dynamic sampling assumes fixed n_samples groups per prompt.

The intended non-agentic data flow is:

candidate prompt groups
  -> rollout and base rewards
  -> remove constant-reward groups
  -> accumulate complete groups to the target batch size
  -> apply overlong reward shaping
  -> compute group-standardized advantages
  -> optimize with real old-policy ratios and token-level aggregation

Example usage would look like:

areno train \
  --algo dapo \
  --ckpt <model> \
  --dataset-path <dataset> \
  --reward-fn-path <reward.py> \
  --batch-size 8 \
  --n-samples 4 \
  --dapo-gen-batch-size 24 \
  --dapo-overlong-buffer-len 256 \
  --max-new-tokens 1024

Validation plan

The implementation would include deterministic CPU coverage for:

• asymmetric lower/upper clipping values and gradients;
• proof that changing stored old logprobs changes the DAPO loss and gradient;
• packed/padded equivalence;
• global response-token normalization across microbatches, gradient accumulation, and DP sharding;
• constant-reward filtering, retry behavior, exact target-batch assembly, and bounded failure diagnostics;
• group-boundary preservation;
• overlong penalty boundary values; and
• ordering of base-reward filtering before length shaping.

A bounded end-to-end run would additionally verify finite losses and gradients, parameter updates, dynamic-sampling counters, and ratio/clip diagnostics. This proposal does not intend to claim reproduction of the paper's large-scale AIME result.

Questions for maintainers

1. Is a real old-logprob ratio acceptable when it is scoped only to experimental DAPO, while the GRPO/GSPO decision in Decide whether GRPO/GSPO need real old-logprob ratios #68 remains unchanged?
2. Is filtering on AReno's existing scalar reward contract acceptable for the initial implementation, provided overlong shaping is applied after filtering?
3. Would maintainers prefer one cohesive experimental DAPO PR, or a preliminary internal response-token normalization PR followed by the algorithm PR?

Reference

• Paper: https://arxiv.org/abs/2503.14476
• Project page: https://dapo-sia.github.io/
• Maintained reference implementation and documentation: https://github.com/verl-project/verl/blob/main/docs/algo/dapo.md
• Maintained recipe: https://github.com/verl-project/verl-recipe/tree/main/dapo

Alternatives considered

• Adding only asymmetric clip values to the existing GRPO loss would be numerically ineffective under the current unit-valued surrogate ratio and would not constitute DAPO.
• Changing stable GRPO/GSPO to use real old-policy ratios would silently alter existing behavior and should remain a separate design decision under Decide whether GRPO/GSPO need real old-logprob ratios #68.
• Splitting Clip-Higher, dynamic sampling, and overlong shaping into independently named DAPO feature PRs would expose an incomplete algorithm surface.
• Overlong filtering is intentionally excluded because the maintained reference implementation treats it as overlapping with soft overlong reward shaping.

[xsuler]

@CharlesXu-HQ For your questions, here are my answers:

1. Following the current GSPO/GRPO implementation and using new logprobs is better, since it's on-policy.
2. That is acceptable.
3. A coherent PR is preferred.

Please make sure you test your implementation well.

[CharlesXu-HQ]

Thanks for the clarification. I want to confirm one numerical consequence before requesting review of the implementation.

AReno's current GRPO surrogate computes exp(new_logprob - new_logprob.detach()). Its forward value is always 1, even though it preserves a gradient path. As a result, DAPO's asymmetric Clip-Higher thresholds cannot activate numerically: both lower and upper clip fractions remain zero regardless of the configured bounds.

The draft implementation therefore scopes a real importance ratio to experimental DAPO only:

ratio = exp(train_logprob - rollout_logprob)

Stable GRPO/GSPO semantics remain unchanged. The stored rollout logprobs are validated for token alignment and finiteness, and targeted tests cover old-logprob sensitivity, both clipping directions, hand-derived gradients, packed/padded parity, and response-token normalization.

Would this scoped real rollout-policy ratio be acceptable for experimental DAPO, or would you prefer DAPO to preserve the unit-valued on-policy surrogate even though Clip-Higher would then have no numerical effect?

[xsuler]

This approach looks acceptable for experimental DAPO. Using the actual rollout-policy ratio makes Clip-Higher numerically meaningful, while preserving the existing GRPO/GSPO semantics.

Even with unified training and inference, small logprob differences can still occur due to numerical effects. Given the alignment/finite checks and targeted tests, this seems like a reasonable and well-scoped tradeoff for the experimental DAPO path.