Skip to content

LanguageModel GPU Capacity Validation Policy

  • Status: Implemented policy (production deployment validation remains separate work)
  • Last updated: 2026-08-03
  • Scope: Dense SpikingEvoTextLM training launched from Mineral Exploration LanguageMode

Purpose

Preserve the rule that Core executes the exact LanguageModel configuration chosen by the user, while clearly rejecting a request that cannot start with the GPU's currently free VRAM before a job is created.

This policy is not automatic downscaling. Mineral, the SDK, and Core must not reduce batch_size, d_model, block count, time_steps, or seq_len because capacity is insufficient.

Responsibilities

Layer Responsibility Prohibition
Mineral UI Show current capacity after input changes and disable training for a known overflow Do not shrink input values automatically
Mineral API Recheck capacity before launch and return HTTP 422 when capacity is insufficient Do not modify the request and create a job
Core / SDK Execute the valid submitted training configuration unchanged and perform normal CUDA availability validation Do not automatically adjust batch size by GPU type or free memory

The user explicitly resolves insufficient capacity, for example by changing batch size, model dimensions, block count, time_steps, or other processes sharing the GPU.

UI and API behavior

Non-mutating capacity query

POST /api/language-mode/training-capacity receives the current form configuration, validates its basic model constraints, and returns a capacity estimate only. It does not change state, create a job, or rewrite configuration values beyond normal validation.

The LanguageMode UI calls this endpoint 300 ms after changes to architecture, d_model, head count, block count, time_steps, batch size, seq_len, tokenizer, neuron type, or learning rate. When a decision is available, it displays required and free VRAM.

Launch preflight

POST /api/language-mode/train and POST /api/language-mode/train/start run the same check before creating a job. When capacity is known and required memory exceeds free memory, they return HTTP 422.

{
  "detail": {
    "message": "The current GPU free memory cannot start the requested LanguageModel configuration. The configuration has not been changed.",
    "capacity": {
      "checked": true,
      "fits": false,
      "gpu_index": 0,
      "gpu_name": "CUDA GPU name",
      "available_mb": 5923.0,
      "required_mb": 6144.0,
      "optimizer_mb": 0.0,
      "activation_mb": 0.0,
      "reserve_mb": 512.0,
      "effective_seq_len": 64
    }
  }
}

Values vary with configuration and runtime GPU state. A client receiving 422 preserves the submitted values and asks the user to change them.

Estimate scope and selected GPU

Only dense_chronospike is covered. sparse_event_memory has a separate memory model, so the query returns checked: false and a reason. If GPU metrics are unavailable, the result is also unchecked and Core performs its normal CUDA availability validation.

Core starts with DEVICE=cuda on logical CUDA device 0. Mineral interprets the same visibility settings and estimates the physical GPU corresponding to the first CUDA_VISIBLE_DEVICES entry, or, when unset, the first NVIDIA_VISIBLE_DEVICES entry. With all or no setting, this is normally GPU 0.

The estimate includes tokenizer-derived vocabulary size, d_model, block count, time_steps, batch size, sequence length, optimizer state, activations, and a reserve. The dense model parameter estimate is:

\[P = 2Vd + V + d + 1 + B(12d^2 + 9d)\]

where \(V\) is vocabulary size, \(d\) is d_model, and \(B\) is block count. It adds optimizer and activation storage, 5% extra headroom, and a reserve equal to the larger of 512 MiB or 5% of total VRAM.

Core does not change the submitted configuration, including seq_len. Mineral therefore estimates the requested sequence length itself. The convergence-oriented defaults are seq_len=64 and learning rate \(10^{-4}\), but those are UI/API initial values rather than hidden Core overrides.

Hardware portability

Capacity is not a fixed limit for any GPU such as an RTX 2070. It is evaluated against the free VRAM of the GPU currently visible to Core, so the same configuration can be allowed on an H100 with sufficient free memory.

Current dense LanguageModel training does not aggregate VRAM across multiple GPUs or shard the model automatically. Even with eight visible H100 GPUs, the check applies to the first visible GPU. To use a specific GPU, configure the same CUDA_VISIBLE_DEVICES / NVIDIA_VISIBLE_DEVICES setting for Mineral and Core. Distributed training and aggregated VRAM are separate features.

Operational limitations

  • The estimate is a preventive check using VRAM free at query time, not a guarantee. A TOCTOU race with another process, or CUDA/PyTorch temporary workspace allocations, can still cause runtime OOM.
  • A runtime OOM does not cause Core to retry with changed parameters. The failure is recorded; a user or scheduler submits a new, explicit configuration.
  • This policy does not modify the request, artifacts, or batch size of a currently running job.
  • Production rollout involving a Docker restart and continued validation on real GPUs remain separate work.

Acceptance criteria

  1. An insufficient dense configuration is rejected without modifying input batch_size.
  2. The capacity estimate and Core execution use the same submitted seq_len.
  3. An insufficient launch request returns HTTP 422 with structured capacity information.
  4. The same request is allowed on a GPU with enough free VRAM.
  5. The UI displays required/free capacity and disables the training button for known overflow.