Skip to content

LLM Training System

  • Last updated: 2026-08-14

This system provides automated training for various Large Language Model (LLM) categories with REST API support, Docker integration, and optional GPU acceleration.

Implementation note: the current orchestration and memory-control work is intentionally documented as a minimal foundation. The future extension points are part of the design contract rather than hidden assumptions.

Features - Multi-modal training: LangText, Vision, Audio, MultiModal - API-driven job submission and monitoring - GPU/CPU support with Docker-based deployment - Nginx-based load balancing for scaling training servers

Architecture extension contract

EvoSpikeNet does not unify all LMs into a single implementation. The supported families are explicitly separated as:

  1. SpikingEvoTextLM / dense_chronospike — dense text path, current default for text inference and training.
  2. SparseEventMemoryLM / sparse_event_memory — sparse text path optimized for memory efficiency; checkpoints are not cross-compatible with dense models.
  3. EvoNetLM / standard_transformer — standard float Transformer flow used for comparison and fallback behavior.
  4. SpikingEvoMultiModalLM / spiking_multimodal — dense multimodal path using image/audio spike fusion.
  5. MemoriedSpikeNetLM / memoried_spikenetlm — future extension from the sparse event-memory backbone into multimodal token streams with modality-aware routing and artifact contracts.

The phrase “minimal implementation” in this project means the current path is functionally validated but intentionally designed to leave explicit extension points for future model-family growth.

Current dense text path: SpikingEvoTextLM

The current dense text path is the SpikingEvoTextLM in evospikenet/models.py. It is no longer a simple spike-only transformer. It combines:

  • TASEncoderDecoder for token-to-spike conversion
  • SpikingTransformerBlock stacks
  • ChronoSpikeAttention causal temporal attention
  • optional AEG energy gating
  • optional MetaSTDP / reward-based adaptation
  • a direct + deep readout mixture through readout_mixer and direct_output_fc

This is a deliberate stabilization design: the spike path remains hard and causal, but the readout path includes a direct projection so early task gradients remain informative.

Convergence control in the current implementation

examples/train_spiking_evospikenet_lm.py resolves a convergence profile through _resolve_convergence_profile(), and the precedence rules are intentional:

  • default profile: stable_baseline
  • available profiles: stable_plus, stable_converge, aggressive_convergence
  • CLI/API values are preserved and not silently overridden by the profile
  • EVOSPIKENET_TRAIN_*_OVERRIDE variables take precedence over profile defaults

Key behavior in the current version:

  • EVOSPIKENET_FORCE_HARD_SPIKE_PARITY=true disables continuous relaxation automatically
  • stable_converge turns off AEG and MetaSTDP to improve stability during convergence-focused runs
  • reward_mode=ema_delta and reward_clip_value reduce noisy reward drift
  • grad_clip_norm is applied to avoid long-sequence instability

Training data flow

The active training path follows this pipeline:

  1. get_training_corpus(args) loads the corpus
  2. Japanese corpora are normalized and chunked with _apply_rag_japanese_preprocess()
  3. _iter_preprocessed_hf_japanese_wikipedia_chunks() streams bounded chunks
  4. _tokenize_corpus_in_chunks() converts the chunk to token IDs
  5. _build_next_token_dataset() creates a shift-by-one next-token dataset
  6. the training loop computes a loss and optionally emits a reward signal for AEG / MetaSTDP

This path is deliberately designed to keep memory bounded while working on long Japanese corpora. It transforms text into token tensors and then into next-token supervision without forcing the full corpus into memory at once.

Mathematical structure

SpikingEvoTextLM is not a pure SNN argmax model. The readout is a mixture of direct and deep pathways:

\[ ext{logits} = \sigma(\alpha) \, \text{direct\_logits} + (1 - \sigma(\alpha)) \, \text{deep\_logits} \]

where \(\alpha = \text{sigmoid}(\text{readout\_direct\_logit\_mix})\). This stabilizes learning when the hard-spike path alone would otherwise provide a weak or collapsed gradient.

The attention layer also enforces causality with the decay mask

\[ M(t, t') = \exp\left(-\max(0, t - t') / \tau\right) \]

so that future events are not visible to the current timestep, while the past is discounted according to temporal distance.

Sparse Event-Memory LM

  • Implementation: SparseEventMemoryLM in EvoSpikeNet-Core/evospikenet/sparse_event_memory.py
  • Features: fixed-topology INT8 CSR primary synapses, INT16 event state, Top-k expert routing, a factorized tied vocabulary, and sampled softmax
  • Training scope: Adam updates only the vocabulary basis, router, low-rank adapters, and normalization parameters; primary synapses use local Hebbian updates
  • Selection: examples/train_spiking_evospikenet_lm.py --architecture sparse_event_memory
  • Compatibility: this is separate from the dense SpikingEvoTextLM / ChronoSpikeAttention path. See the specification.

Application-configurable model architecture

When a UI can change d_model, head count, block count, time_steps, neuron_type, or architecture, each training result is registered as an independent model. d_model alone must not identify a model.

  • d_model and head count require \(d_{model} \bmod n_{heads} = 0\).
  • New training stores a model-specific checkpoint, config.json, tokenizer bundle, and model_spec in durable storage.
  • Continued training is not architecture migration. It is permitted only when structure, vocabulary, tokenizer, and artifact digests exactly match the base model.
  • Inference is built from the selected model's own model_spec and artifact manifest; it must not reuse UI defaults or another model's configuration.

Mineral Exploration LanguageMode implements manifest generation, the model_spec registry, legacy status, continued-training and inference preflight, and configuration presets. Production rollout requiring a Docker restart and migration of running jobs remain separate work. See the LanguageModel Architecture Compatibility and Artifact Contract for adoption requirements.

GPU capacity validation and submitted-configuration preservation

For dense SpikingEvoTextLM, Mineral Exploration LanguageMode shows required/free VRAM for the currently visible GPU after configuration changes and disables launch for known overflow. The launch API repeats the same validation and rejects insufficient capacity with HTTP 422.

  • Core, the SDK, and the UI do not automatically shrink a submitted batch_size or model configuration. The user explicitly changes an insufficient configuration.
  • The estimate and Core execution use the same submitted seq_len. The convergence-oriented UI defaults are seq_len=64 and learning rate 0.0001.
  • The saved artifact config.json records submitted values, effective values, and selected convergence controls under training_execution. Because Core does not overwrite submitted learning rate, seq_len, batch_size, or epoch count, the submitted and effective values match.
  • The decision uses free VRAM on the first GPU from CUDA_VISIBLE_DEVICES, or NVIDIA_VISIBLE_DEVICES when the former is unset. It does not aggregate VRAM across GPUs.
  • GPU state can change after a query, so the check cannot eliminate every runtime OOM. A runtime OOM also does not cause automatic batch-size retry.

See the LanguageModel GPU Capacity Validation Policy for the API, estimation, HTTP 422 response, and hardware portability including H100.

Core dashboard LanguageModel training configuration

The Core SpikingEvoTextLM training page exposes the same executable text-training configuration as Mineral Exploration LanguageMode: architecture (dense_chronospike / sparse_event_memory), d_model, head count, block count, time_steps, neuron_type (LIF, EvoLIF, Izhikevich), tokenizer, sequence length, learning rate, batch size, epochs, data source, and new/continued training. The Small/Base/Large presets only fill editable fields; they do not lock later edits.

  • Core does not automatically alter a learning rate, sequence length, batch size, epoch count, or structural value submitted by the dashboard. Capacity errors fail explicitly; the user changes the configuration.
  • Continued training selects a local Core artifact and passes its checkpoint as base_model_path. It does not pass a Mineral registry ID into Core.
  • The TransformerLM (Float) page is for the standard Float Transformer. It passes d_model, head count, block count, sequence length, learning rate, batch size, epochs, source, and new/continued-training selection directly to its trainer. Because this model creates a character-level vocabulary from the selected corpus, the SpikingEvoTextLM external tokenizer selection does not apply.
  • MultiModal, Vision, and Audio pages use different trainer/data contracts, so they do not present text-only architecture, neuron, or tokenizer fields as non-functional controls. Each page submits only fields accepted by its actual trainer.
  • Vision Encoder passes the dataset, output dimension, time steps, learning rate, batch size, epochs, artifact name, model category/variant, precision, quantization, and privacy level to train_vision_encoder.py.
  • Audio Encoder passes the audio category, MFCC feature count, MFCC sequence length, output neurons, time steps, learning rate, batch size, epochs, artifact name, precision, quantization, and privacy level to train_audio_encoder.py. Synthetic data is created only when the user explicitly requests it.
  • MultiModal LM passes the image-caption dataset, d_model, head count, block count, SNN time steps, caption sequence length, learning rate, batch size, epochs, and artifact metadata to train_multi_modal_lm.py. That trainer does not currently accept a local base_model_path resume input, so the dashboard does not expose one.
  • Audio/Vision pages and MultiModal Audio retain a legacy generic API panel, but its start button is disabled because it does not launch the modality-specific trainer. Use the Model Training tab to run training.

Quick start (high level) 1. Collect training data: python scripts/collect_llm_training_data.py --config config/data_config.yaml 2. Launch training servers: - GPU: ./scripts/train_launcher.sh gpu - CPU: ./scripts/train_launcher.sh cpu - Both with load balancer: ./scripts/train_launcher.sh all 3. Submit a training job via POST /train and monitor via GET /jobs.

Docker - Build GPU trainer: docker build -f Dockerfile.train.gpu -t llm-trainer-gpu . - Build CPU trainer: docker build -f Dockerfile.train.cpu -t llm-trainer-cpu . - Use docker-compose -f docker-compose.train.yml up -d to run the full stack

API example: submit a training job 

curl -X POST "http://localhost:8000/train" -H "Content-Type: application/json" -d '{
    "category": "LangText",
    "model_name": "microsoft/DialoGPT-medium",
    "dataset_path": "data/llm_training/LangText/langtext_data.jsonl",
    "output_dir": "saved_models/LangText",
    "gpu": true,
    "epochs": 3,
    "batch_size": 8,
    "learning_rate": 0.00002
}'

Monitoring & troubleshooting - ./scripts/train_launcher.sh status — check server status - ./scripts/train_launcher.sh logs gpu — view GPU logs - Common issues: CUDA OOM (in LanguageMode, review the capacity display and explicitly submit a smaller batch/model configuration; Core does not alter a running configuration), port conflicts (change ports), model download failures (check network / HF access)

Advanced usage - Custom models: update config/training_config.yaml to point to different HF model names - Scaling: use Kubernetes, MLflow, and distributed training (Accelerate)

Web UI integration - Integrated into EvoSpikeNet frontend with dedicated pages under frontend/pages/ for Vision/Audio training controls, status, and real-time output.

If you want, I can generate a compact example docker-compose snippet or a minimal FastAPI training server example.