# VGGT Pose Derivation: How the Model Learns Camera Parameters

## Overview

VGGT learns to predict camera poses through **supervised learning** with ground truth camera parameters from multi-view datasets. The model doesn't "derive" pose from geometry—it learns to predict pose from visual features through training.

## 1. Ground Truth Data Sources

### 1.1 Training Datasets

VGGT is trained on datasets that provide **ground truth camera parameters**:

**Primary Datasets**:
1. **CO3D (Common Objects in 3D)**
   - Provides extrinsics and intrinsics for object-centric scenes
   - Camera poses estimated via COLMAP/SfM
   - File: `training/data/datasets/co3d.py`

2. **vKITTI (Virtual KITTI)**
   - Synthetic driving scenes with perfect camera parameters
   - File: `training/data/datasets/vkitti.py`

**Data Format**:
```python
{
    "images": List[np.ndarray],           # RGB images
    "depths": List[np.ndarray],           # Depth maps
    "extrinsics": List[np.ndarray],       # Camera extrinsics (3×4, OpenCV convention)
    "intrinsics": List[np.ndarray],       # Camera intrinsics (3×3)
    "world_points": np.ndarray,           # 3D points in world coordinates
    "point_masks": np.ndarray,            # Validity masks for points
}
```

### 1.2 Ground Truth Camera Parameters

**Extrinsics** (`extri_opencv`):
- **Format**: `[R | t]` (3×4 matrix)
- **Convention**: OpenCV camera-from-world transformation
- **Source**: COLMAP/SfM reconstruction or synthetic data

**Intrinsics** (`intri_opencv`):
- **Format**:
  ```
  [[fx, 0,  cx],
   [0,  fy, cy],
   [0,  0,  1 ]]
  ```
- **Source**: Camera calibration or dataset metadata

**Example from CO3D**:
```python
# training/data/datasets/co3d.py, lines 205-258
extrinsics = []
intrinsics = []

for frame_data in sequence_data:
    # Load camera parameters from CO3D annotation
    extri_opencv = frame_data['extrinsics']  # 3×4 matrix
    intri_opencv = frame_data['intrinsics']  # 3×3 matrix

    extrinsics.append(extri_opencv)
    intrinsics.append(intri_opencv)
```

---

## 2. Training Process: How Pose is Learned

### 2.1 Forward Pass

**Input**: Images `[B, S, 3, H, W]`

**Processing**:
1. **Aggregator** (backbone) extracts visual features
2. **Camera Head** predicts pose encoding from camera tokens
3. **Output**: `pose_enc_list` - list of pose encodings (one per iteration)

**File**: `vggt/models/vggt.py`
```python
# Forward pass
predictions = model(images)
pose_enc_list = predictions["pose_enc_list"]  # List of [B, S, 9] tensors
```

### 2.2 Ground Truth Encoding

**File**: `training/loss.py`, lines 101-109

```python
# Get ground truth camera extrinsics and intrinsics
gt_extrinsics = batch_data['extrinsics']      # [B, S, 3, 4]
gt_intrinsics = batch_data['intrinsics']      # [B, S, 3, 3]
image_hw = batch_data['images'].shape[-2:]    # (H, W)

# Encode ground truth pose to match predicted encoding format
gt_pose_encoding = extri_intri_to_pose_encoding(
    gt_extrinsics, gt_intrinsics, image_hw,
    pose_encoding_type="absT_quaR_FoV"
)  # [B, S, 9]
```

**Encoding Process** (`vggt/utils/pose_enc.py`):
```python
# Extract components
R = extrinsics[:, :, :3, :3]  # Rotation matrix
T = extrinsics[:, :, :3, 3]   # Translation vector

# Convert rotation to quaternion
quat = mat_to_quat(R)  # [B, S, 4]

# Compute field of view from intrinsics
H, W = image_size_hw
fov_h = 2 * torch.atan((H / 2) / intrinsics[..., 1, 1])  # Vertical FOV
fov_w = 2 * torch.atan((W / 2) / intrinsics[..., 0, 0])  # Horizontal FOV

# Combine into 9D encoding
pose_encoding = [T(3), quat(4), fov_h(1), fov_w(1)]  # [B, S, 9]
```

### 2.3 Loss Computation

**File**: `training/loss.py`, lines 81-155

**Multi-Stage Loss** (with temporal decay):
```python
def compute_camera_loss(
    pred_dict,              # Contains 'pose_enc_list'
    batch_data,             # Contains 'extrinsics', 'intrinsics'
    loss_type="l1",
    gamma=0.6,              # Temporal decay weight
    weight_trans=1.0,       # Translation loss weight
    weight_rot=1.0,         # Rotation loss weight
    weight_focal=0.5,       # FOV loss weight
):
    pred_pose_encodings = pred_dict['pose_enc_list']  # List of [B, S, 9]
    n_stages = len(pred_pose_encodings)  # 4 iterations

    # Encode ground truth
    gt_pose_encoding = extri_intri_to_pose_encoding(...)  # [B, S, 9]

    # Compute loss for each iteration
    for stage_idx in range(n_stages):
        stage_weight = gamma ** (n_stages - stage_idx - 1)  # Later stages weighted more
        pred_pose_stage = pred_pose_encodings[stage_idx]

        # Compute component-wise losses
        loss_T_stage, loss_R_stage, loss_FL_stage = camera_loss_single(
            pred_pose_stage[valid_frame_mask],
            gt_pose_encoding[valid_frame_mask],
            loss_type=loss_type
        )

        # Accumulate weighted losses
        total_loss_T += loss_T_stage * stage_weight
        total_loss_R += loss_R_stage * stage_weight
        total_loss_FL += loss_FL_stage * stage_weight

    # Weighted combination
    total_camera_loss = (
        avg_loss_T * weight_trans +
        avg_loss_R * weight_rot +
        avg_loss_FL * weight_focal
    )
```

**Component Losses** (`camera_loss_single`):
```python
def camera_loss_single(pred_pose_enc, gt_pose_enc, loss_type="l1"):
    # L1 loss for each component
    loss_T = (pred_pose_enc[..., :3] - gt_pose_enc[..., :3]).abs()      # Translation
    loss_R = (pred_pose_enc[..., 3:7] - gt_pose_enc[..., 3:7]).abs()    # Rotation (quaternion)
    loss_FL = (pred_pose_enc[..., 7:] - gt_pose_enc[..., 7:]).abs()    # Field of view

    return loss_T.mean(), loss_R.mean(), loss_FL.mean()
```

**Key Points**:
- **L1 loss** (absolute error) for each component
- **Separate losses** for translation, rotation, and FOV
- **Temporal weighting**: Later iterations weighted more (`gamma=0.6`)
- **Valid frame filtering**: Only frames with >100 valid points

---

## 3. Data Normalization

### 3.1 Why Normalize?

Camera parameters have different scales:
- **Translation**: Can be in meters (0.1-100m)
- **Rotation**: Quaternion (unit norm)
- **FOV**: Radians (0.1-3.0)

Normalization ensures stable training.

### 3.2 Normalization Process

**File**: `training/train_utils/normalization.py`, lines 27-122

```python
def normalize_camera_extrinsics_and_points_batch(
    extrinsics: torch.Tensor,  # [B, S, 3, 4]
    world_points: torch.Tensor,
    cam_points: torch.Tensor,
    depths: torch.Tensor,
):
    """
    Normalize camera extrinsics and 3D points.

    Strategy:
    1. Set first camera as identity (reference frame)
    2. Normalize translation scale to unit average
    """
    B, S, _, _ = extrinsics.shape

    # Convert to homogeneous form
    extrinsics_homog = torch.cat([
        extrinsics,
        torch.zeros(B, S, 1, 4, device=device)
    ], dim=2)
    extrinsics_homog[:, :, -1, -1] = 1.0

    # Set first camera as identity (reference frame)
    first_cam_extrinsic_inv = closed_form_inverse_se3(extrinsics_homog[:, 0])
    new_extrinsics = torch.matmul(extrinsics_homog, first_cam_extrinsic_inv.unsqueeze(1))

    # Normalize translation scale
    # Compute average scale from translation magnitudes
    avg_scale = new_extrinsics[:, :, :3, 3].norm(dim=-1).mean(dim=-1)  # [B]
    new_extrinsics[:, :, :3, 3] = new_extrinsics[:, :, :3, 3] / avg_scale.view(-1, 1, 1)

    # Transform world points accordingly
    R = extrinsics[:, 0, :3, :3]
    t = extrinsics[:, 0, :3, 3]
    new_world_points = (world_points @ R.T) + t
    new_world_points = new_world_points / avg_scale.view(-1, 1, 1, 1)

    return new_extrinsics[:, :, :3], new_cam_points, new_world_points, depths
```

**Normalization Strategy**:
1. **Reference frame**: First camera set to identity `[I | 0]`
2. **Scale normalization**: Translation magnitudes normalized to unit average
3. **Coordinate transform**: All cameras and points transformed to normalized frame

**Why This Works**:
- **Scale-invariant**: Model learns relative poses, not absolute scales
- **Stable training**: Normalized values prevent gradient issues
- **Generalization**: Works across different scene scales

---

## 4. What the Model Actually Learns

### 4.1 Visual Features → Camera Parameters

The model learns a **mapping from visual features to camera parameters**:

```
Visual Features (from images)
    ↓
Camera Tokens (aggregated features)
    ↓
Iterative Refinement (4 steps)
    ↓
Pose Encoding [T(3), quat(4), FOV(2)]
```

**Key Insight**: The model doesn't solve geometry—it learns to **recognize visual patterns** that correlate with camera poses.

### 4.2 What Visual Cues Does It Use?

The model likely learns to recognize:
1. **Parallax**: Relative motion between near/far objects
2. **Perspective**: Vanishing points, horizon lines
3. **Multi-view consistency**: How objects appear from different angles
4. **Depth cues**: Occlusion, relative sizes, texture gradients

**Evidence**: The model uses **alternating frame/global attention** to:
- **Frame attention**: Process each view independently
- **Global attention**: Share information across all views
- This enables learning multi-view geometric relationships

### 4.3 Training Objective

**Total Loss**:
```python
L_total = L_camera + L_depth + L_point + L_track
```

**Camera Loss**:
```python
L_camera = weight_trans * L_T + weight_rot * L_R + weight_focal * L_FOV
```

**Component Losses**:
- **Translation loss**: `L1(pred_T - gt_T)`
- **Rotation loss**: `L1(pred_quat - gt_quat)`
- **FOV loss**: `L1(pred_FOV - gt_FOV)`

---

## 5. Comparison: VGGT vs Traditional SfM

### Traditional SfM (COLMAP)
1. **Feature detection**: Detect keypoints (SIFT, etc.)
2. **Feature matching**: Match keypoints across views
3. **Epipolar geometry**: Compute fundamental/essential matrices
4. **Bundle adjustment**: Optimize camera poses and 3D points jointly

**Key**: Solves geometry from correspondences

### VGGT
1. **Visual feature extraction**: Extract features from images
2. **Multi-view attention**: Share information across views
3. **Direct prediction**: Predict camera parameters directly
4. **Supervised learning**: Learn from ground truth poses

**Key**: Learns to predict pose from visual patterns

---

## 6. Key Takeaways

### How VGGT "Derives" Pose

1. **Training Phase**:
   - Learns mapping: `Visual Features → Camera Parameters`
   - Supervised by ground truth poses from COLMAP/SfM
   - Uses L1 loss on translation, rotation, and FOV

2. **Inference Phase**:
   - Extracts visual features from input images
   - Predicts pose encoding directly from features
   - No geometric solving—pure feed-forward prediction

3. **What It Learns**:
   - Visual patterns that correlate with camera poses
   - Multi-view geometric relationships
   - Scale-invariant pose representations

### Why This Works

1. **Large-scale training**: Millions of images with ground truth poses
2. **Multi-view supervision**: Learns from multiple views simultaneously
3. **Iterative refinement**: Progressively improves predictions
4. **Normalization**: Scale-invariant training enables generalization

### Limitations

1. **Statistical consistency**: Learns average behavior, not hard geometric constraints
2. **Generalization**: May struggle on scenes very different from training data
3. **No explicit geometry**: Doesn't solve epipolar geometry or bundle adjustment

---

## 7. Code References

**Training Loss**:
- `training/loss.py`: `compute_camera_loss()`, `camera_loss_single()`

**Data Loading**:
- `training/data/datasets/co3d.py`: CO3D dataset loader
- `training/data/datasets/vkitti.py`: vKITTI dataset loader

**Normalization**:
- `training/train_utils/normalization.py`: `normalize_camera_extrinsics_and_points_batch()`

**Pose Encoding**:
- `vggt/utils/pose_enc.py`: `extri_intri_to_pose_encoding()`, `pose_encoding_to_extri_intri()`

**Model**:
- `vggt/models/vggt.py`: Main model forward pass
- `vggt/heads/camera_head.py`: Camera head prediction