Unlikelihood Training in Large Language Models: How Negative Candidate Loss and Sequence-Level Penalties Suppress Repetition Loops and Hallucinations

Autoregressive language models generate text by iteratively predicting the probability distribution of the next token conditioned on preceding tokens. While standard maximum likelihood estimation (MLE) via cross-entropy loss serves as the universal training objective across modern foundation models, it possesses a fundamental structural asymmetry: it exclusively maximizes the likelihood of target tokens appearing in the training corpus without explicitly penalizing degenerate, repetitive, or undesirable tokens.

Under greedy decoding and constrained sampling regimes, standard MLE-trained models frequently suffer from neural text degeneration. Once a model assigns slightly elevated probability to a previously emitted token or n-gram, self-attention mechanisms amplify the token's representation in subsequent steps, trapping the generator in inescapable repetition loops. Heuristic decoding methods, such as nucleus sampling, top-k filtering, and temperature scaling, mitigate this behavior at inference time by truncating the probability distribution, but they mask the pathology rather than resolving the distorted probability landscape learned by the neural network.

Unlikelihood training, introduced by Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston, directly reformulates the training objective. By incorporating an explicit negative loss term that penalizes designated negative candidate tokens and repeating sequence rollouts, unlikelihood training reshapes the model's output distribution. This analysis examines the mathematical foundations of unlikelihood loss, derives its pre-softmax gradient dynamics, details token-level and sequence-level negative candidate construction, and traces its theoretical lineage to modern preference optimization frameworks like Direct Preference Optimization (DPO).

The Maximum Likelihood Pathology

Modern generative language models decompose the joint probability of a token sequence $X = (x_1, x_2, \dots, x_T)$ into an autoregressive product of conditional probabilities:

$$p_\theta(X) = \prod_{t=1}^T p_\theta(x_t \mid x_{<t})$$

During standard pre-training and supervised fine-tuning, the parameter vector $\theta$ is optimized using Maximum Likelihood Estimation (MLE), implemented as empirical negative log-likelihood (cross-entropy) over a dataset $\mathcal{D}$:

$$\mathcal{L}_{\text{MLE}}(p_\theta, \mathcal{D}) = - \sum_{t=1}^T \log p_\theta(x_t^* \mid x_{<t})$$

where $x_t^*$ denotes the ground-truth target token at step $t$, and $x_{<t} = (x_1, \dots, x_{t-1})$ represents the ground-truth prefix.

Standard Cross-Entropy (MLE):
Context: x_{<t} ───► Model Softmax ───► Maximize log p(x*_t)
                                         (Other tokens implicitly suppressed via normalizer)

Unlikelihood Training (Joint Loss):
Context: x_{<t} ───► Model Softmax ───┬─► Maximize log p(x*_t)          [Likelihood]
                                       └─► Minimize log(1 - p(c)) ∀ c∈C  [Unlikelihood]

The Passive Suppression Weakness

Cross-entropy loss operates as a one-sided objective. When the model computes its predictive distribution via the softmax function over vocabulary logits $z \in \mathbb{R}^{|\mathcal{V}|}$:

$$p_\theta(x_t = k \mid x_{<t}) = \frac{\exp(z_k)}{\sum_{j \in \mathcal{V}} \exp(z_j)}$$

the objective directly pushes up the logit $z_{x_t^}$ corresponding to the ground-truth token. Non-target tokens $k \ne x_t^$ are suppressed only indirectly through the normalizer $\sum_{j} \exp(z_j)$.

This passive mechanism creates several structural failures:

1. Probability Mass Leakage: If the training corpus exhibits high variance or noise, the model frequently distributes non-trivial probability mass across semantically irrelevant or degenerate tokens.
2. Exposure Bias and Discrepancy: During training, the model is always fed ground-truth prefixes (teacher forcing). At inference time, the model conditions on its own prior generations $\hat{x}_{<t}$. Small distributional shifts compound over long horizons, driving the model into out-of-distribution state representations.
3. Attractor Loops: In high-dimensional transformer representations, self-attention circuits (particularly induction heads and associative key-value projections) naturally attend to tokens already present in the active context window. If the model emits a duplicate token during greedy or low-temperature decoding, that token increases the attention weight assigned to previous occurrences of the same token. This creates a positive feedback loop where the probability of repeating the token or phrase approaches 1.0, generating pathological repetitions such as "the company said the company said the company said..."

The Limits of Decoding Truncation

To prevent degenerate repetitions, practitioners commonly rely on stochastic decoding heuristics:

• Top-k Sampling: Restricts generation to the $k$ most probable tokens.
• Nucleus (Top-p) Sampling: Truncates the candidate pool to the smallest set of tokens whose cumulative probability exceeds threshold $p$.
• Repetition Penalties: Artificially subtracts a fixed scalar from the logits of tokens present in the prompt context before computing the softmax.

While these heuristics prevent deterministic loops, empirical research demonstrates that they do not correct the underlying learned representation. Nucleus sampling often forces the model to sample from an erratic tail, increasing semantic incoherence and hallucination rates. Furthermore, in tasks requiring precise, deterministic outputs (such as code generation, mathematical deduction, and structured JSON parsing), stochastic sampling degrades task performance.

Unlikelihood training solves this problem at the parameter level, ensuring that greedy decoding itself produces non-degenerate, diverse text.

Mathematical Formulation of Unlikelihood Loss

Unlikelihood training augments the objective function by introducing a loss term that explicitly decreases the probability of an undesirable set of negative candidate tokens $\mathcal{C}_t \subset \mathcal{V}$ at step $t$.

Let $p_\theta(\cdot \mid x_{<t})$ represent the predictive distribution over vocabulary $\mathcal{V}$ given prefix $x_{<t}$. For any negative candidate token $c \in \mathcal{C}_t$, the probability of the model not predicting token $c$ is:

$$q_\theta(c \mid x_{<t}) = 1 - p_\theta(c \mid x_{<t})$$

The unlikelihood loss $\mathcal{L}_{\text{UL}}^t$ is defined as the negative log-probability of this complement event, summed over all negative candidates in $\mathcal{C}_t$:

$$\mathcal{L}_{\text{UL}}^t(p_\theta, \mathcal{C}_t, x_{<t}) = - \sum_{c \in \mathcal{C}_t} \log\left(1 - p_\theta(c \mid x_{<t})\right)$$

Because $p_\theta(c \mid x_{<t}) \in [0, 1)$, the complement $1 - p_\theta(c \mid x_{<t})$ is bounded in $(0, 1]$. Minimizing $-\log(1 - p_\theta(c \mid x_{<t}))$ is equivalent to driving $1 - p_\theta(c \mid x_{<t}) \to 1$, which forces $p_\theta(c \mid x_{<t}) \to 0$.

The Joint Objective

To maintain language modeling fluency and semantic validity while eliminating negative behaviors, unlikelihood loss is combined with standard maximum likelihood estimation via a weighting hyperparameter $\alpha \ge 0$:

$$\mathcal{L}_{\text{Joint}}^t(p_\theta, x_t^<em>, \mathcal{C}_t, x_{<t}) = -\log p_\theta(x_t^</em> \mid x_{<t}) - \alpha \sum_{c \in \mathcal{C}_t} \log\left(1 - p_\theta(c \mid x_{<t})\right)$$

Across an entire sequence of length $T$, the total unlikelihood objective is:

$$\mathcal{L}_{\text{Joint}}(X, \mathcal{C}) = \sum_{t=1}^T \left[ -\log p_\theta(x_t^* \mid x_{<t}) - \alpha \sum_{c \in \mathcal{C}_t} \log\left(1 - p_\theta(c \mid x_{<t})\right) \right]$$

When $\alpha = 0$, the objective reduces to standard cross-entropy. When $\alpha > 0$, the loss penalizes both failure to predict the correct target $x_t^*$ and any positive probability allocated to negative candidates in $\mathcal{C}_t$.

Gradient Dynamics and Logit Penalization

The theoretical power of unlikelihood training becomes clear when analyzing the gradients with respect to the pre-softmax logits $z \in \mathbb{R}^{|\mathcal{V}|}$.

Let $z_i$ denote the logit for token $i \in \mathcal{V}$, such that $p_i = p_\theta(x_t = i \mid x_{<t}) = \frac{\exp(z_i)}{\sum_j \exp(z_j)}$.

Standard MLE Gradient Derivation

For standard cross-entropy $\mathcal{L}_{\text{MLE}} = -\log p_{x_t^*}$, the gradient with respect to logit $z_i$ is:

$$\frac{\partial \mathcal{L}_{\text{MLE}}}{\partial z_i} = p_i - \mathbb{I}[i = x_t^*]$$

• For the ground-truth token ($i = x_t^$): $\frac{\partial \mathcal{L}_{\text{MLE}}}{\partial z_{x_t^}} = -(1 - p_{x_t^*})$, which pulls the logit up proportionally to the error.
• For all other tokens ($i \ne x_t^*$): $\frac{\partial \mathcal{L}_{\text{MLE}}}{\partial z_i} = p_i$, which pushes the logit down proportionally to its current probability.

Unlikelihood Loss Gradient Derivation

Consider the unlikelihood loss for a single negative candidate $c \in \mathcal{C}_t$: $\mathcal{L}_{\text{UL}} = -\log(1 - p_c)$.

Using the chain rule:

$$\frac{\partial \mathcal{L}_{\text{UL}}}{\partial z_i} = \frac{\partial \mathcal{L}_{\text{UL}}}{\partial p_c} \cdot \frac{\partial p_c}{\partial z_i}$$

The derivative of the loss with respect to probability $p_c$ is:

$$\frac{\partial \mathcal{L}_{\text{UL}}}{\partial p_c} = -\frac{1}{1 - p_c} \cdot (-1) = \frac{1}{1 - p_c}$$

The derivative of the softmax probability $p_c$ with respect to logit $z_i$ is standard:

$$\frac{\partial p_c}{\partial z_i} = p_c (\mathbb{I}[i = c] - p_i)$$

Multiplying these terms yields the exact logit gradient for unlikelihood loss:

$$\frac{\partial \mathcal{L}_{\text{UL}}}{\partial z_i} = \left(\frac{1}{1 - p_c}\right) \cdot p_c (\mathbb{I}[i = c] - p_i) = \left(\frac{p_c}{1 - p_c}\right) (\mathbb{I}[i = c] - p_i)$$

Analyzing the Three Gradient Regimes

Combining the MLE and unlikelihood gradients for a single negative candidate $c$ and target token $x_t^*$ gives:

$$\frac{\partial \mathcal{L}_{\text{Joint}}}{\partial z_i} = \left(p_i - \mathbb{I}[i = x_t^*]\right) + \alpha \left(\frac{p_c}{1 - p_c}\right) \left(\mathbb{I}[i = c] - p_i\right)$$

Evaluating this gradient across different token categories reveals the mechanism:

| Token Category | Index Condition | Gradient $\frac{\partial \mathcal{L}_{\text{Joint}}}{\partial z_i}$ | Physical Effect on Logit $z_i$ | | :--- | :--- | :--- | :--- | | Ground-Truth Target | $i = x_t^$ (assuming $x_t^ \ne c$) | $-(1 - p_{x_t^}) - \alpha \left(\frac{p_c}{1 - p_c}\right) p_{x_t^}$ | Boosted upward force: Target logit increases faster when negative candidate probability $p_c$ is high. | | Negative Candidate | $i = c$ | $p_c + \alpha \left(\frac{p_c}{1 - p_c}\right) (1 - p_c) = (1 + \alpha) p_c$ | Direct hyperbolic suppression: Downward gradient scales linearly with $(1+\alpha)p_c$, directly crushing the logit. | | Neutral Token | $i \ne x_t^*$ and $i \ne c$ | $p_i - \alpha \left(\frac{p_c}{1 - p_c}\right) p_i = p_i \left(1 - \alpha \frac{p_c}{1 - p_c}\right)$ | Adaptive redistribution: Gradient dampens, allowing probability mass to flow into neutral alternatives. |

The Hyperbolic Scaling Factor

The factor $\frac{p_c}{1 - p_c}$ is the odds ratio of the negative candidate. As $p_c \to 1$ (the model assigns high confidence to a pathological token), $\frac{p_c}{1 - p_c} \to \infty$.

This property ensures that:

1. When the model assigns near-zero probability to an undesirable token, the penalty gradient is negligible, leaving normal language acquisition undistorted.
2. When the model assigns high probability to a degenerate or repeating token, the gradient explodes, forcing immediate and severe parameter updates to suppress that mode.

Negative Candidate Construction Strategies

The effectiveness of unlikelihood training depends entirely on how the negative candidate set $\mathcal{C}_t$ is defined. Researchers and practitioners employ three primary candidate selection paradigms.

Candidate Selection Strategies:

1. Token-Level Context Window:
   Context: [The, cat, sat, on, the] ───► Target: [mat]
   Candidate Set C_t: {The, cat, sat, on} \ {mat}

2. Sequence-Level Rollout (Self-Generated):
   Prefix: [The robot reached] ───► Model Rollout: [for the wrench and the wrench and the wrench]
                                                                    ▲──────────▲──────────▲
                                                                  Repeated 4-gram detected!
   Sequence Candidates C_t: Set to tokens inside degenerate n-gram cycles.

3. Entity / Hallucination Masking:
   Ground Truth Knowledge: {Founded: 1976}
   Model Output: "Apple was founded in 1984" ──► C_t: {1984}

1. Token-Level Recency Candidates

To prevent local token stuttering and immediate repetition loops during teacher-forced training, the negative candidate set $\mathcal{C}_t$ is constructed from the preceding context tokens within a fixed history window $k$, excluding the ground-truth target:

$$\mathcal{C}_t^{\text{token}} = \{ x_{t-k}, x_{t-k+1}, \dots, x_{t-1} \} \setminus \{ x_t^* \}$$

If a token appeared in the recent context but is not the valid continuation, its probability is pushed down. This forces the model to explore new vocabulary items rather than defaulting to recently activated lexical memory.

2. Sequence-Level Unlikelihood (Rollout Penalization)

While token-level unlikelihood reduces immediate token repetition, it suffers from a major constraint: it is computed over ground-truth prefixes $x_{<t} \sim p_{\text{data}}$. It never exposes the model to its own self-generated, multi-step degenerative trajectories.

To resolve sequence-level degeneration (such as repeating 4-grams or cyclic sentence structures), Welleck et al. introduced Sequence-Level Unlikelihood (Seq-UL):

1. Prefix Sampling: Given a prefix $x_{1:k} \sim \mathcal{D}$, generate a continuation of length $N$ autoregressively using greedy decoding:

$$\hat{x}_{k+1:k+N} \sim p_\theta(\cdot \mid x_{1:k})$$

1. Degenerate Span Identification: Scan the generated continuation $\hat{X} = (\hat{x}_{k+1}, \dots, \hat{x}_{k+N})$ for repeating $n$-grams (typically 4-grams). If an $n$-gram $\hat{x}_{j:j+n}$ matches an identical $n$-gram previously seen in the same continuation $\hat{x}_{m:m+n}$ (where $m < j$), mark all tokens in the duplicate span as degenerate.
2. Candidate Assignment: For each step $t \in \{j, \dots, j+n\}$, define the candidate set as the repeating token: $\mathcal{C}_t^{\text{seq}} = \{ \hat{x}_t \}$.
3. Sequence Unlikelihood Loss: Compute the unlikelihood loss over the self-generated sequence:

$$\mathcal{L}_{\text{ULS}}(p_\theta, \hat{X}) = - \sum_{t=k+1}^{k+N} \sum_{c \in \mathcal{C}_t^{\text{seq}}} \log\left(1 - p_\theta(c \mid \hat{x}_{<t})\right)$$

Sequence-level unlikelihood directly punishes the exact trajectories that the model's current weights produce under greedy rollout, closing the gap between teacher forcing and inference execution.

3. Factual Error and Toxicity Suppression

Beyond structural repetitions, candidate sets can be constructed dynamically to target semantic failure modes:

• Hallucination Mitigation: In retrieval-augmented generation, tokens generated by the model that contradict retrieved source documents can be added to $\mathcal{C}_t$.
• Toxicity and Safety Filtering: Toxic lexicons, leaked private keys, or disallowed behavioral sequences identified by red-teaming classifiers are dynamically converted into negative candidate sets during fine-tuning.

Theoretical Lineage: From Unlikelihood to Modern Preference Optimization

Unlikelihood training laid the algorithmic foundation for modern alignment methods that optimize language models using pairs of preferred and dispreferred text.

Evolution of Negative Loss Formulations:

1. Unlikelihood Training (2019):
   L = -log p(x*) - α log(1 - p(c_neg))

2. Direct Preference Optimization (DPO, 2023):
   L = -log σ( β log[π_θ(y_w)/π_ref(y_w)] - β log[π_θ(y_l)/π_ref(y_l)] )
                                           ▲─────────────────────────▲
                                           Sequence-level unlikelihood 
                                           with reference KL anchor!

3. Negative Preference Optimization (NPO / Unlearning, 2024):
   L = -log σ( -β log[π_θ(y_forget)/π_ref(y_forget)] )

Direct Preference Optimization (DPO)

Direct Preference Optimization optimizes a policy $\pi_\theta$ directly on preference pairs $(y_w, y_l)$ (where $y_w$ is preferred and $y_l$ is dispreferred) without training an explicit reward model.

The DPO loss is formulated as:

$$\mathcal{L}_{\text{DPO}}(\theta; \pi_{\text{ref}}) = - \mathbb{E}_{(x, y_w, y_l)} \left[ \log \sigma \left( \beta \log \frac{\pi_\theta(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} - \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)} \right) \right]$$

Examining the second term inside the sigmoid:

$$- \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)} = \beta \log \pi_{\text{ref}}(y_l \mid x) - \beta \log \pi_\theta(y_l \mid x)$$

Minimizing the DPO objective explicitly pushes down $\log \pi_\theta(y_l \mid x)$. DPO is mathematically equivalent to performing reference-anchored sequence-level unlikelihood training on the rejected response $y_l$ while simultaneously performing reference-anchored likelihood training on the chosen response $y_w$.

Contrastive Preference Optimization (CPO) and Machine Unlearning

• Contrastive Preference Optimization (CPO): Applied extensively in machine translation and code reasoning, CPO adds an explicit negative unlikelihood penalty to prevent moderate-quality translation outputs from dominating the generation distribution.
• Negative Preference Optimization (NPO) and Machine Unlearning: In privacy preservation and safety scrubbing, models are trained to forget specific corpora (such as copyrighted books or toxic conversations). NPO applies unlikelihood objectives directly to the target forget-set tokens, using a reference model $\pi_{\text{ref}}$ as a regularizer to prevent model collapse on unrelated distributions.

Empirical Benchmarks and Generative Properties

The impact of unlikelihood training on language generation has been extensively evaluated across standard corpora, including Wikitext-103, CC-News, and open-ended dialogue datasets.

Repetition and Diversity Metrics

Empirical evaluations measure text quality across three core dimensions:

1. seq-rep-4: The percentage of generated 4-grams that repeat previously generated 4-grams within the same sequence (lower is better).
2. Unique Tokens / Distinct-N: The proportion of distinct n-grams across the entire generation budget (higher is better).
3. Perplexity (PPL): Token prediction loss on ground-truth evaluation sets (lower is better).

| Training Objective | Decoding Strategy | Perplexity (PPL) $\downarrow$ | seq-rep-4 (%) $\downarrow$ | Unique Tokens (100k) $\uparrow$ | Zipf Distribution Error $\downarrow$ | | :--- | :--- | :--- | :--- | :--- | :--- | | Standard MLE Baseline | Greedy ($T=0$) | 25.6 | 46.2% | 1,840 | 0.42 | | Standard MLE Baseline | Nucleus ($p=0.9$) | 25.6 | 12.8% | 9,450 | 0.28 | | Token-Level UL ($\alpha=1.0$) | Greedy ($T=0$) | 26.2 | 11.4% | 7,820 | 0.19 | | Token + Seq UL | Greedy ($T=0$) | 26.8 | 1.2% | 14,210 | 0.08 | | Human Reference | N/A | N/A | 0.8% | 15,100 | 0.00 |

Data source: Welleck et al., ICLR 2020.

Key empirical findings include:

• Greedy Decoding Parity with Humans: Combining token-level and sequence-level unlikelihood training reduces 4-gram repetition from 46.2% to 1.2% under pure greedy decoding, closely matching human text distributions (0.8%).
• Preservation of Vocabulary Tail (Zipf's Law): Standard MLE models over-index on high-frequency head words during generation. Unlikelihood training restores the heavy-tailed Zipfian distribution of natural language by penalizing premature head-word reuse.
• Minimal Perplexity Penalty: When properly tuned ($\alpha \approx 0.5 - 1.0$), unlikelihood loss introduces less than a 1.2-point increase in validation perplexity while eliminating degenerative collapse.

Implementation Details and Numerical Stability

Implementing unlikelihood training in PyTorch requires careful attention to numerical precision. Because modern models compute logits in float16 or bfloat16, calculating $\log(1 - p)$ directly can suffer from catastrophic cancellation when $p \approx 0$ or $p \approx 1$.

Numerically Stable Unlikelihood Formulation

Given log-probabilities $\log p_k = z_k - \text{logsumexp}(z)$, we must evaluate $\log(1 - p_c)$.

Using the mathematical identity:

$$\log(1 - p_c) = \log\left(1 - \exp(\log p_c)\right)$$

In PyTorch, this is implemented natively via torch.log1mexp (or torch.log(-torch.expm1(log_p))):

import torch
import torch.nn as nn
import torch.nn.functional as F

class UnlikelihoodLoss(nn.Module):
    """
    Computes Joint Maximum Likelihood and Unlikelihood Loss.
    """
    def __init__(self, alpha: float = 1.0, ignore_index: int = -100):
        super().__init__()
        self.alpha = alpha
        self.ignore_index = ignore_index

    def forward(
        self,
        logits: torch.Tensor,              # [batch_size, seq_len, vocab_size]
        targets: torch.Tensor,             # [batch_size, seq_len]
        negative_mask: torch.Tensor        # [batch_size, seq_len, vocab_size] (1 for negative candidates, 0 otherwise)
    ) -> torch.Tensor:
        # Compute log-softmax over vocabulary
        log_probs = F.log_softmax(logits, dim=-1) # [B, T, V]
        
        # 1. Standard Cross-Entropy (MLE Loss)
        mle_loss = F.nll_loss(
            log_probs.view(-1, log_probs.size(-1)),
            targets.view(-1),
            ignore_index=self.ignore_index,
            reduction='mean'
        )
        
        # 2. Numerically Stable Unlikelihood Loss: -log(1 - p_c) = -log1mexp(log_p_c)
        # Clamping log_probs avoids -inf when p_c -> 1
        clamped_log_probs = torch.clamp(log_probs, max=-1e-7)
        unlikelihood_per_token = -torch.log1p(-torch.exp(clamped_log_probs)) # log(1 - exp(log_p))
        
        # Apply negative candidate mask
        masked_ul_loss = unlikelihood_per_token * negative_mask
        
        # Normalize by total active negative candidates
        num_candidates = negative_mask.sum().clamp(min=1.0)
        ul_loss = masked_ul_loss.sum() / num_candidates
        
        # Total Joint Loss
        total_loss = mle_loss + self.alpha * ul_loss
        return total_loss, mle_loss, ul_loss

Practical Training Hyperparameters

When integrating unlikelihood objectives into pre-training or supervised fine-tuning pipelines:

1. Warmup Phase: Train the model with standard MLE ($\alpha = 0$) for the first 10% of training steps to establish basic syntactic representations before enabling unlikelihood penalties.
2. Alpha Scaling: Set $\alpha \in [0.2, 1.0]$. Setting $\alpha > 2.0$ often causes the model to over-suppress valid vocabulary items, leading to syntax degradation and increased validation perplexity.
3. Rollout Frequency: During sequence-level unlikelihood fine-tuning, decoding rollouts at every optimization step is computationally expensive. Running greedy sequence rollouts on every 4th batch provides an optimal trade-off between training throughput and loop suppression.

Summary and Key Takeaways

• The Core Problem: Maximum likelihood estimation with cross-entropy is fundamentally one-sided: it rewards matching target tokens but fails to penalize degenerate, repeating, or hallucinated tokens.
• The Mechanism: Unlikelihood training introduces $-\sum_{c \in \mathcal{C}} \log(1 - p_\theta(c \mid x_{<t}))$, providing an explicit downward gradient that scales hyperbolically with the odds ratio $\frac{p_c}{1 - p_c}$.
• Candidate Topologies: Token-level candidates penalize immediate lexical recency, while sequence-level candidates penalize multi-step repeating n-grams generated during model rollouts.
• Modern Lineage: Direct Preference Optimization (DPO), Contrastive Preference Optimization (CPO), and Machine Unlearning are direct extensions of sequence-level unlikelihood principles, using reference models to constrain negative probability shifts.
• Production Impact: Unlikelihood fine-tuning eliminates the need for aggressive stochastic sampling truncation during inference, enabling greedy and low-temperature decoding to produce fluent, non-repetitive text across structured and open-ended generation tasks.

Sources

• Neural Text Generation with Unlikelihood Training (Welleck et al., ICLR 2020 / arXiv:1908.04319)
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model (Rafailov et al., NeurIPS 2023 / arXiv:2305.18290)
• The Curious Case of Neural Text Degeneration (Holtzman et al., ICLR 2020 / arXiv:1904.09751)
• Contrastive Preference Optimization for Out-of-Distribution Translation (Xu et al., ICML 2024 / arXiv:2401.08417)
• Negative Preference Optimization: From Unlearning to Safe Alignment (Zhang et al., 2024 / arXiv:2404.05868)