Projecting Conflicting Gradients (PCGrad): Mathematical Foundations, Orthogonal Projections, and Multi-Task Optimization in Deep Learning

In modern machine learning systems, models rarely optimize for a single objective. Foundation models are trained simultaneously on diverse data distributions spanning natural language, source code, mathematical reasoning, and multimodal inputs. Similarly, post-training alignment pipelines must simultaneously optimize for helpfulness, factual accuracy, harmlessness, and formatting constraints.

When multiple objective functions share a single set of neural network parameters, standard empirical risk minimization minimizes a scalarized sum of individual task losses:

$$\mathcal{L}_{\text{total}}(\theta) = \sum_{i=1}^{T} w_i \mathcal{L}_i(\theta)$$

While computationally convenient, optimizing a naive linear combination of task losses frequently causes negative transfer and optimization stagnation. When task gradients point in opposing directions in parameter space, moving along the average gradient direction decreases the loss on some tasks while actively increasing the loss on others.

In 2020, Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn published Gradient Surgery for Multi-Task Learning, introducing Projecting Conflicting Gradients (PCGrad). PCGrad provides a geometric framework that identifies conflicting gradient components across tasks and projects each gradient onto the normal plane of any conflicting task before computing the parameter update.

The Geometry of Gradient Interference

To understand why multi-task optimization fails under standard gradient descent, consider $T$ distinct tasks with loss functions $\mathcal{L}_1(\theta), \mathcal{L}_2(\theta), \dots, \mathcal{L}_T(\theta)$ parameterized by a shared parameter vector $\theta \in \mathbb{R}^d$.

The gradient of each task with respect to $\theta$ is defined as:

$$g_i = \nabla_\theta \mathcal{L}_i(\theta)$$

Definition of Gradient Conflict

Two task gradients $g_i$ and $g_j$ are defined to be in conflict if their Euclidean inner product is strictly negative:

$$\langle g_i, g_j \rangle = g_i^\top g_j < 0 \iff \cos(\theta_{ij}) = \frac{g_i^\top g_j}{\|g_i\|_2 \|g_j\|_2} < 0$$

where $\theta_{ij}$ is the angle between the two gradient vectors in $\mathbb{R}^d$.

When $\cos(\theta_{ij}) < 0$, the gradient vectors form an obtuse angle exceeding $90^\circ$. If the optimizer takes a descent step in the direction of task $j$ with learning rate $\eta$:

$$\theta_{t+1} = \theta_t - \eta g_j$$

The first-order Taylor expansion of task $i$'s loss function around $\theta_t$ yields:

$$\mathcal{L}_i(\theta_{t+1}) \approx \mathcal{L}_i(\theta_t) + \nabla_\theta \mathcal{L}_i(\theta_t)^\top (\theta_{t+1} - \theta_t) = \mathcal{L}_i(\theta_t) - \eta g_i^\top g_j$$

Because $g_i^\top g_j < 0$, the term $-\eta g_i^\top g_j$ is strictly positive:

$$\mathcal{L}_i(\theta_{t+1}) > \mathcal{L}_i(\theta_t)$$

Consequently, optimizing along the gradient of task $j$ directly increases the loss of task $i$. In a standard multi-task setup where the update vector is the unweighted sum $g_{\text{sum}} = \sum_{k=1}^T g_k$, the net change in task $i$'s loss is:

$$\Delta \mathcal{L}_i \approx -\eta g_i^\top \left( \sum_{k=1}^T g_k \right) = -\eta \|g_i\|_2^2 - \eta \sum_{j \neq i} g_i^\top g_j$$

If the destructive cross-task interference $\sum_{j \neq i} g_i^\top g_j$ is negative and exceeds the magnitude of the task's own self-descent term $\|g_i\|_2^2$, task $i$ regresses during the training step.

Three Pathological Conditions in Multi-Task Landscapes

Yu et al. identified three structural characteristics of multi-task optimization landscapes that exacerbate gradient interference:

1. Directional Conflict ($\cos(\theta_{ij}) < 0$): Gradients point in opposing directions, causing parameter updates to oscillate or destroy representations required by competing tasks.
2. Magnitude Disparity ($\|g_i\|_2 \gg \|g_j\|_2$): Tasks with large gradient norms dominate the update vector $g_{\text{sum}}$, starving tasks with smaller gradients of optimization progress regardless of their actual loss scale or difficulty.
3. High Local Curvature (Ill-Conditioned Hessians): When task loss landscapes exhibit sharp valleys with large eigenvalues in their Hessian matrices $\nabla^2 \mathcal{L}_i(\theta)$, taking steps guided by foreign task gradients moves the parameters out of the local valley, triggering sudden loss spikes.

The PCGrad Projection Operator

PCGrad resolves directional gradient conflict by modifying the gradient vectors before they are combined. The core intuition is that when task $i$'s gradient $g_i$ conflicts with task $j$'s gradient $g_j$, the component of $g_i$ that is anti-parallel to $g_j$ should be removed.

Mathematical Derivation of Orthogonal Projection

Let $g_i, g_j \in \mathbb{R}^d$ be two task gradients such that $g_i^\top g_j < 0$. We decompose $g_i$ into two orthogonal components relative to $g_j$:

$$g_i = g_{i, \parallel j} + g_{i, \perp j}$$

where:

• $g_{i, \parallel j}$ is the projection of $g_i$ onto $g_j$ (parallel component).
• $g_{i, \perp j}$ is the component of $g_i$ orthogonal to $g_j$ (normal component).

The parallel projection is given by vector projection:

$$g_{i, \parallel j} = \text{proj}_{g_j}(g_i) = \frac{g_i^\top g_j}{\|g_j\|_2^2} g_j$$

To eliminate the destructive component while retaining as much of $g_i$'s original direction as possible, PCGrad replaces $g_i$ with its orthogonal projection onto the hyperplane orthogonal to $g_j$:

$$g_i^{\text{proj}} = g_i - g_{i, \parallel j} = g_i - \frac{g_i^\top g_j}{\|g_j\|_2^2} g_j$$

Verification of Conflict Elimination

We verify that the updated gradient $g_i^{\text{proj}}$ no longer has a negative inner product with $g_j$:

$$\langle g_i^{\text{proj}}, g_j \rangle = \left( g_i - \frac{g_i^\top g_j}{\|g_j\|_2^2} g_j \right)^\top g_j = g_i^\top g_j - \frac{g_i^\top g_j}{\|g_j\|_2^2} (g_j^\top g_j) = g_i^\top g_j - g_i^\top g_j = 0$$

The inner product between $g_i^{\text{proj}}$ and $g_j$ is exactly zero. To the first-order approximation:

$$\Delta \mathcal{L}_j \approx -\eta g_j^\top g_i^{\text{proj}} = 0$$

Taking a parameter step in the direction of $-g_i^{\text{proj}}$ will not increase the loss of task $j$.

If $g_i^\top g_j \ge 0$, the gradients are non-conflicting (orthogonal or pointing in a mutually beneficial direction), and $g_i$ is left unchanged:

$$g_i^{\text{proj}} = g_i \quad \text{if } g_i^\top g_j \ge 0$$

Case 1: g_i · g_j >= 0 (No Conflict)
        g_i
       ^
      /
     /  theta <= 90 deg
    /--------> g_j
    g_i remains untouched.

Case 2: g_i · g_j < 0 (Conflict Detected)
        g_i
       ^ 
        \  
         \  theta > 90 deg
<---------\----------> g_j
proj_g_j(g_i) |
              | Normal Hyperplane to g_j
              v
         g_i_proj = g_i - ( (g_i · g_j) / ||g_j||^2 ) * g_j
         Result: g_i_proj · g_j = 0 (Orthogonal)

The Complete Multi-Task PCGrad Algorithm

When optimizing $T > 2$ tasks simultaneously, projecting task $i$'s gradient against task $j$ might re-introduce a conflict with another task $k$. To handle arbitrary numbers of tasks robustly, PCGrad applies pairwise projections sequentially across all competing tasks in a randomized order.

Algorithmic Formulation

For a mini-batch at training step $t$:

1. Compute individual task gradients $g_i = \nabla_\theta \mathcal{L}_i(\theta)$ for each task $i \in \{1, \dots, T\}$.
2. Initialize updated gradient containers: $g_i^{\text{proj}} \leftarrow g_i$ for all $i \in \{1, \dots, T\}$.
3. For each task $i \in \{1, \dots, T\}$:

• Sample a random permutation $\pi_i$ of all other task indices $\{j \in \{1, \dots, T\} \mid j \neq i\}$.
• For each $j \in \pi_i$:
• If $\langle g_i^{\text{proj}}, g_j \rangle < 0$:

$$g_i^{\text{proj}} \leftarrow g_i^{\text{proj}} - \frac{\langle g_i^{\text{proj}}, g_j \rangle}{\|g_j\|_2^2} g_j$$

1. Compute the aggregate multi-task gradient:

$$g_{\text{PCGrad}} = \sum_{i=1}^{T} g_i^{\text{proj}}$$

1. Apply the parameter update using the chosen base optimizer (e.g., AdamW or SGD):

$$\theta_{t+1} \leftarrow \text{OptimizerUpdate}(\theta_t, g_{\text{PCGrad}})$$

Why Random Permutations Are Required

Because vector projections are non-commutative operations:

$$\text{proj}_{g_k}^\perp \left( \text{proj}_{g_j}^\perp (g_i) \right) \neq \text{proj}_{g_j}^\perp \left( \text{proj}_{g_k}^\perp (g_i) \right)$$

Applying projections in a fixed deterministic order introduces systematic directional bias toward tasks processed later in the sequence. Randomizing the permutation $\pi_i$ independently for each task at every optimization step ensures isotropic expectation and unbiased parameter updates.

PyTorch Implementation

Below is a self-contained PyTorch implementation of the PCGrad optimizer wrapper. It wraps any standard PyTorch optimizer (such as torch.optim.AdamW), intercepts multi-task losses, computes individual gradients, executes the projection surgery, and applies the final update.

import random
import torch
from torch.optim import Optimizer

class PCGrad:
    """
    Projecting Conflicting Gradients (PCGrad) optimizer wrapper.
    Reference: Yu et al., 'Gradient Surgery for Multi-Task Learning' (NeurIPS 2020).
    """
    def __init__(self, optimizer: Optimizer):
        self.optimizer = optimizer

    @property
    def param_groups(self):
        return self.optimizer.param_groups

    def zero_grad(self):
        self.optimizer.zero_grad()

    def _flatten_grads(self, gradients):
        """Flatten a list of parameter gradients into a single 1D tensor."""
        flat_grads = []
        for grad in gradients:
            if grad is None:
                continue
            flat_grads.append(grad.reshape(-1))
        return torch.cat(flat_grads) if flat_grads else torch.empty(0)

    def _unflatten_grads(self, flat_grad, target_params):
        """Unflatten a 1D gradient tensor back into parameter grad attributes."""
        offset = 0
        for param in target_params:
            if not param.requires_grad:
                continue
            numel = param.numel()
            param.grad = flat_grad[offset : offset + numel].view_as(param).clone()
            offset += numel

    def pcgrad_step(self, task_losses: list[torch.Tensor]):
        """
        Executes a single PCGrad optimization step across a list of per-task losses.
        
        Args:
            task_losses: List of scalar torch.Tensor objectives, one per task.
        """
        assert len(task_losses) > 0, "task_losses cannot be empty"
        num_tasks = len(task_losses)
        
        # Collect parameters that require gradients
        params = []
        for group in self.optimizer.param_groups:
            for p in group['params']:
                if p.requires_grad:
                    params.append(p)

        # 1. Compute per-task gradients
        task_grads = []
        for i, loss in enumerate(task_losses):
            self.optimizer.zero_grad()
            # Retain graph if not on the last task loss
            loss.backward(retain_graph=(i < num_tasks - 1))
            
            grads = [p.grad.clone() if p.grad is not None else torch.zeros_like(p) for p in params]
            flat_g = self._flatten_grads(grads)
            task_grads.append(flat_g)

        # 2. Perform pairwise gradient surgery
        projected_grads = [g.clone() for g in task_grads]
        
        for i in range(num_tasks):
            # Sample random permutation of competing task indices
            competing_indices = [j for j in range(num_tasks) if j != i]
            random.shuffle(competing_indices)
            
            for j in competing_indices:
                g_i = projected_grads[i]
                g_j = task_grads[j]
                
                # Compute inner product
                inner_prod = torch.dot(g_i, g_j)
                
                # If conflicting, project g_i onto the orthogonal complement of g_j
                if inner_prod < 0:
                    norm_sq = torch.dot(g_j, g_j) + 1e-12
                    projected_grads[i] = g_i - (inner_prod / norm_sq) * g_j

        # 3. Aggregate projected gradients across all tasks
        merged_grad = torch.stack(projected_grads).sum(dim=0)

        # 4. Set final gradients on parameters and step base optimizer
        self.optimizer.zero_grad()
        self._unflatten_grads(merged_grad, params)
        self.optimizer.step()

Theoretical Comparison Across Multi-Task Optimizers

Several paradigms have been developed to address multi-task interference and multi-objective optimization (MOO). Understanding where PCGrad sits relative to competing algorithms requires analyzing their optimization targets and Pareto guarantees.

       Pareto Optimization Approaches
       ==============================
                     |
       +-------------+-------------+
       |                           |
Explicit Trade-Offs          Implicit Regularization
(MGDA, CAGrad)               (PCGrad, GradNorm)
       |                           |
- MGDA: Finds min-norm in    - PCGrad: Removes negative
  convex hull; guarantees      projection components;
  Pareto stationarity but      preserves average task scale.
  sacrifices average loss.   - GradNorm: Dynamically balances
- CAGrad: Optimizes worst-     gradient norms over time.
  case descent rate in local
  ball around average gradient.

1. Multiple Gradient Descent Algorithm (MGDA)

Sener and Koltun (2018) formulated multi-task learning as explicit Multi-Objective Optimization. MGDA seeks a common descent direction $\alpha^* \in \Delta^T$ by solving the minimum-norm problem in the convex hull of task gradients:

$$\min_{\alpha_1, \dots, \alpha_T} \left\| \sum_{i=1}^T \alpha_i g_i \right\|_2^2 \quad \text{subject to } \sum_{i=1}^T \alpha_i = 1, \quad \alpha_i \ge 0$$

• Advantage: Provably converges to a Pareto-stationary point where no task's loss can be decreased without increasing another's.
• Defect in Practice: MGDA frequently stalls on degenerate Pareto points. If one task reaches a sharp local minimum where its gradient is orthogonal or opposing, MGDA assigns disproportionate weight to that single stalled task, sacrificing the overall average loss across the remaining tasks.

2. Conflict-Averse Gradient Descent (CAGrad)

Liu et al. (2021) introduced Conflict-Averse Gradient Descent (CAGrad). Rather than abandoning the average gradient $g_0 = \frac{1}{T}\sum_{i=1}^T g_i$, CAGrad finds an update direction $d$ within a local Euclidean ball around $g_0$ that maximizes the minimum descent rate across all tasks:

$$\max_{d \in \mathbb{R}^p} \min_{i \in \{1, \dots, T\}} \langle g_i, d \rangle \quad \text{subject to } \|d - g_0\|_2 \le c \|g_0\|_2$$

where $c \in [0, 1)$ controls the degree of conflict aversion. When $c=0$, CAGrad reduces to standard gradient descent; as $c \to 1$, it approaches MGDA.

Comparison Matrix

| Algorithm | Optimization Target | Conflict Resolution Strategy | Pareto Convergence Guarantee | Computational Overhead | Primary Limitation | | :--- | :--- | :--- | :--- | :--- | :--- | | Linear Scalarization | $\min \sum w_i \mathcal{L}_i$ | None (Gradients sum directly) | No (Vulnerable to negative transfer) | $O(1)$ backward pass | Destructive gradient cancellation | | GradNorm (Chen et al., 2018) | Dynamic weight scaling $w_i(t)$ | Balances gradient norms based on training pace | No | $O(T)$ gradient norms | Only addresses magnitude disparity, not directional conflict | | MGDA (Sener & Koltun, 2018) | Min-norm convex hull | Quadratic programming on Gram matrix | Yes (Guarantees Pareto-stationary point) | $O(T)$ backward passes + QP solve | Sacrifices average task performance for worst-case tasks | | PCGrad (Yu et al., 2020) | Multi-task gradient surgery | Pairwise orthogonal projections ($\langle g_i^{\text{proj}}, g_j \rangle = 0$) | No formal Pareto guarantee (Regularizes shared representation) | $O(T)$ backward passes + $O(T^2)$ inner products | Order-dependent without random shuffling; projection accumulation | | CAGrad (Liu et al., 2021) | Constrained worst-case descent | Constrained optimization around average gradient $g_0$ | Yes (Provable convergence to average loss minimum) | $O(T)$ backward passes + dual optimization | Requires tuning conflict-aversion radius $c$ |

Applications in LLMs and Modern AI Workflows

While originally tested on multi-task robotics benchmarks and vision datasets, the principles of gradient surgery have become central to modern frontier LLM training and alignment.

1. Multi-Objective RLHF and Preference Alignment

In reinforcement learning from human feedback (RLHF) and direct preference tuning (DPO/PPO), practitioners train language models against multiple composite reward models:

$$\mathcal{L}_{\text{RLHF}} = \mathcal{L}_{\text{Helpful}} + \lambda_1 \mathcal{L}_{\text{Harmless}} + \lambda_2 \mathcal{L}_{\text{Concise}} + \lambda_3 \mathcal{L}_{\text{Format}}$$

These objectives frequently have opposing gradients:

• Maximizing helpfulness encourages providing detailed, comprehensive answers, which often triggers safety guardrails on borderline queries.
• Strict harmlessness filtering encourages refusals, which degrades helpfulness scores.

Applying PCGrad to the policy gradients prevents the safety gradient from wiping out generative capability, allowing the model to find parameter updates that maintain safety constraints without degrading general helpfulness.

2. Multi-Domain Instruction Fine-Tuning

During Supervised Fine-Tuning (SFT), models are trained on heterogeneous mixtures containing mathematical derivations, code generation, creative writing, and tool execution.

Standard empirical risk minimization on combined batches often leads to "task interference," where learning strict syntax for code generation degrades prose fluency. By segmenting minibatches by domain and treating each domain as a task in PCGrad, the optimizer removes anti-correlated gradient components between formal reasoning domains and creative language generation.

3. Continual Pre-Training and Catastrophic Forgetting Mitigation

When updating an existing foundation model with new domain data (e.g., medical literature or financial transcripts), fine-tuning on domain batches $g_{\text{domain}}$ leads to catastrophic forgetting of general reasoning capabilities $g_{\text{general}}$.

By maintaining a small reference replay buffer of general pre-training data, the training loop computes $g_{\text{general}}$ alongside $g_{\text{domain}}$. If $g_{\text{domain}}^\top g_{\text{general}} < 0$, PCGrad projects $g_{\text{domain}}$ onto the normal plane of $g_{\text{general}}$:

$$g_{\text{domain}}^{\text{proj}} = g_{\text{domain}} - \frac{g_{\text{domain}}^\top g_{\text{general}}}{\|g_{\text{general}}\|_2^2} g_{\text{general}}$$

The resulting update guarantees that domain adaptation steps do not degrade performance on the foundation distribution.

Computational Bottlenecks and Systems Considerations

Deploying PCGrad in large-scale distributed training clusters introduces systems-level constraints that must be managed:

1. The $T$-Backward Pass Overhead

In naive implementations, computing $T$ individual task gradients requires $T$ separate backward passes through the computational graph:

$$\text{FLOPs}_{\text{backward}} \approx 2 \times T \times \text{FLOPs}_{\text{forward}}$$

For models with billions of parameters, executing $T$ backward passes per training step increases compute time linearly with $T$.

To mitigate this in production systems:

• Task Batching: Practitioners limit $T$ to 2 to 4 high-level domain aggregates (e.g., Code, Math, Language, Safety) rather than dozens of micro-tasks.
• Trunk-Head Partitioning: Gradients are only computed and projected on shared trunk layers (e.g., transformer backbone), while task-specific heads or LoRA adapters are updated independently without surgery.

2. Memory Footprint in Distributed Megatron/FSDP Pipelines

In distributed training using Fully Sharded Data Parallel (FSDP) or Megatron-LM tensor parallelism:

• Each task gradient tensor $g_i \in \mathbb{R}^P$ must be held in GPU memory simultaneously before projection.
• Flattened parameter buffers for a 70B parameter model in FP16 require 140 GB of memory per task gradient.

To avoid out-of-memory (OOM) errors during surgery:

1. Chunked Projections: Gradients are flattened and projected block-by-block across parameter groups or transformer layers, computing inner products $\langle g_{i, \text{block}}, g_{j, \text{block}} \rangle$ incrementally.
2. Fused CUDA Projection Kernels: Rather than creating full tensor copies, custom CUDA or Triton kernels compute the inner product and in-place vector subtraction directly within high-bandwidth GPU memory (HBM).

Conclusion

Projecting Conflicting Gradients (PCGrad) demonstrates that multi-task learning failures are frequently geometric in nature. When multiple objectives share a common parameter space, destructive gradient interference degrades model performance through opposing updates, magnitude disparities, and local curvature traps.

By replacing naive gradient summation with pairwise orthogonal projections, PCGrad eliminates destructive first-order interference while preserving the direction and magnitude of non-conflicting updates. As generative AI systems scale toward unified multi-objective architectures, understanding and manipulating gradient geometry remains an essential tool in neural optimization.

Sources

• Gradient Surgery for Multi-Task Learning (Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, Chelsea Finn, NeurIPS 2020 / arXiv:2001.06782)
• Multi-Task Learning as Multi-Objective Optimization (Ozan Sener, Vladlen Koltun, NeurIPS 2018 / arXiv:1810.04650)
• Conflict-Averse Gradient Descent for Multi-task Learning (Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, Qiang Liu, NeurIPS 2021 / arXiv:2110.14048)
• GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks (Zhao Chen, Vijay Badrinarayanan, Chen-Yu Lee, Andrew Rabinovich, ICML 2018 / arXiv:1711.02257)