Variational Linear Attention documentation

Add your content using reStructuredText syntax. See the reStructuredText documentation for details.

Contents:

• Variational Linear Attention (VLA)
  • The Problem
  • Our Solution: VLA
  • Structure of this Documentation
• Math Primitives
  • 1. Inner-Product Score
  • 2. Sherman–Morrison Rank-1 Inverse Update
  • 3. Multiple Rank-1 Updates (Woodbury Generalization)
  • 4. Recovering Optimal Coefficients \(\alpha^*\)
  • 5. Memory Matrix Update
• Penalty Builder Module
  • Overview
  • Parameterizations
  • Recurrence & Inverse Tracking
  • Inputs and Outputs
  • Computation Graph
• Variational Linear Attention Core
  • Key Modules
• Experiments & Results
  • 1. Symbolic & Diagnostic Experiments
  • 2. Synthetic Memory Tasks
  • 3. Long Range Arena (LRA) Benchmark
  • Summary
• Getting Started & Reproducibility
  • Environment Setup
  • 1. Verifying Core Primitives
  • 2. Running Synthetic Memory Tasks
  • 3. Running LRA Benchmarks
  • 4. Re-generating Paper Plots
  • Utilizing VLA in Your Project