Sparse Event-Memory LM Specification

Purpose and scope

SparseEventMemoryLM is a separate EvoSpikeNet language model designed to expand state space and physical synapse count without placing the full primary-synapse Adam state on the GPU. It does not replace or modify the existing SpikingEvoTextLM, ChronoSpikeAttention, or SpikingFFN path. It is selected only when training specifies --architecture sparse_event_memory.

This specification corresponds to EvoSpikeNet-Core/evospikenet/sparse_event_memory.py and EvoSpikeNet-Core/examples/train_spiking_evospikenet_lm.py.

Architecture

Factorized tied vocabulary — TiedFactorizedVocabulary
A token is represented by a fixed INT8 codebook and a trainable shared basis.
Input embedding and output projection share the same basis and codebook.
Training uses sampled softmax, projecting only to a candidate set that always contains each target token.
forward() produces full-vocabulary logits for inference and exact evaluation. That path materializes output proportional to vocabulary size.
Sparse event expert — EventMemoryExpert
Input and recurrent connections use EventCSRLayer.
Membrane potential uses INT16 state; thresholding creates spikes and integer arithmetic applies leak.
A straight-through surrogate accompanies the discrete event so gradients reach adapters and routers.
Top-k router — SparseEventMemoryBlock
A sequence mean selects experts for each batch sequence.
Only selected experts run. Top-1 uses a sigmoid gate; Top-k uses softmax weights.
Local plasticity
After a successful optimizer update, apply_local_plasticity() uses recorded pre/post mean activity to update INT8 values in place.
Main CSR weights are not nn.Parameter objects, so they have no gradient or Adam first/second moment tensors.

Memory model

Area	Representation	Update method	Adam state
CSR connection values	INT8 buffer	local Hebbian update	none
CSR row/column indices	INT64 buffer	fixed	none
Vocabulary codes	INT8 buffer	fixed	none
Shared vocabulary basis	floating-point parameter	Adam	present
Router, low-rank adapters, LayerNorm	floating-point parameter	Adam	present
Membrane potential	temporary INT16 state	reset per sequence	none

The physical CSR connection count is approximately \(\rho D^2\), where \(D\) is state dimension and \(\rho\) is --sparse-connectivity. This removes dense \(D \times D\) trainable matrices and their Adam state from the primary path. Connection indices, INT8 weights, adapters, and vocabulary bases still consume memory; increasing total model capacity cannot make GPU memory constant.

memory_report() returns trainable-parameter count, fixed CSR connection count, persistent CSR bytes, and vocabulary-code bytes. It is not a peak-VRAM guarantee: activations, sparse-kernel workspace, optimizer state, and candidate-vocabulary logits must also be included.

Training path

Convert token IDs to embeddings with the tied factor vocabulary.
Each block routes the sequence and selected experts update sparse event state in time order.
sampled_cross_entropy() calculates candidate-vocabulary loss from targets and random negatives.
Adam updates only bases, routers, adapters, and normalization parameters.
apply_local_plasticity() locally updates primary CSR synapses.

The training script implements this order. It does not create MetaSTDP or AEG for the sparse model. --ssl-task reconstruction is not supported for this architecture.

CLI

cd EvoSpikeNet-Core
DEVICE=cuda \
EVOSPIKENET_TRAIN_LOG_INTERVAL=10 \
python examples/train_spiking_evospikenet_lm.py \
   --source file \
   --file-path data/corpus/train_corpus.txt \
  --architecture sparse_event_memory \
  --d-model 2048 \
  --num-blocks 12 \
  --sparse-connectivity 0.005 \
  --sparse-factor-rank 128 \
  --sparse-num-experts 2 \
  --sparse-router-top-k 1 \
  --sparse-adapter-rank 16 \
  --sampled-negatives 1024 \
  --run-name sparse_event_memory_ja

Option	Default	Meaning
`--architecture`	`dense_chronospike`	Select this model with `sparse_event_memory`
`--sparse-connectivity`	`0.005`	Connection density per CSR row
`--sparse-factor-rank`	`128`	Rank of the shared vocabulary basis
`--sparse-num-experts`	`2`	Experts in each block
`--sparse-router-top-k`	`1`	Experts executed for each sequence
`--sparse-adapter-rank`	`16`	Adapter rank trained with Adam
`--sparse-plasticity-lr`	`0.001`	Local update rate for INT8 primary synapses
`--sampled-negatives`	`1024`	Random sampled-softmax negatives

SparseEventMemoryLM can be trained with any text source supported by the training script, including wikipedia, aozora, file, huggingface_japanese_wikipedia, and mineral_exploration_file. The current training-script default points to future_apps/mineral_exploration/data/corpus/wiki2_ja_corpus.txt for convenience, but that default does not limit the model itself to Mineral Exploration workflows. The default tokenizer is cl-tohoku/bert-base-japanese-v3. When --lang ja is used with RAG Japanese preprocessing enabled, SudachiPy and a Sudachi dictionary are required. For streaming training, each chunk from huggingface_japanese_wikipedia now receives Japanese preprocessing by default; set EVOSPIKENET_STREAM_JA_PREPROCESS=0 to disable that streaming-specific path, or EVOSPIKENET_USE_RAG_JA_PREPROCESS=0 to disable the general preprocessing path.

Saving and resuming

Saved config.json records architecture: sparse_event_memory and sparse-model settings. When resumed with --base-model-path, the stored architecture family and sparse settings are restored. Dense-model checkpoints are not compatible.

Verified behavior and limitations

Unit tests confirm that INT8 CSR values are not optimizer parameters, input gradients work, local updates run, vocabulary factorization and sampled loss work, and memory reporting is available.
A one-step CPU training smoke test has run with the Japanese tokenizer and a local Japanese corpus. This is not evidence of convergence or language quality.
PyTorch CSR support is beta. GPU speed, peak memory, and kernel support depend on the installed PyTorch/CUDA combination.
Full-vocabulary forward(), larger Top-k, excessive factor_rank, connectivity, or expert count all increase VRAM. Measure peak memory and loss curves on the target GPU before production training.