MEMERIED SpikeNetLM: Audio, Vision, and Multimodal Specification and Implementation Status

Document status: Specification with implemented core and remaining work.
Implementation status: MemoriedSpikeNetLM, VQ vision/audio bridges, a discrete audio-language ID adapter, mask-aware sparse blocks, and SDK artifact payload/upload are implemented and unit-tested. Trained bridges, a dedicated trainer/API route, CTC/retrieval heads, and streaming state remain unimplemented.
Naming: This document retains MEMERIED SpikeNetLM as the project name; public Python/SDK types are MemoriedSpikeNetLM and MemoriedSpikeNetLMConfig.

Purpose and conclusion
Verified current implementation
Compatibility matrix
Target architecture
Modality tokenization specification
Shared sparse event-memory backbone
Input, training, and artifact contracts
Memory and streaming decisions
Implementation plan
Tests, acceptance criteria, and risks
Current SDK and documentation scope

Purpose and conclusion

SparseEventMemoryLM is a text backbone that keeps large primary synapses as INT8 CSR buffers and limits Adam to the shared vocabulary basis, router, adapters, and LayerNorm. Its current public contract accepts only integer token_ids \((B,S)\); it is not a general modality model that directly consumes images, waveforms, MFCCs, or continuous sensor features.

It must therefore not be substituted mechanically into every EvoSpikeNet model. The proposed separation is:

Keep existing modality encoders. Use SpikingEvoVisionEncoder for vision and SpikingEvoAudioEncoder or AudioToBrainLanguageSystem for audio.
Use bridges that turn encoder outputs into discrete modality-token sequences. VectorQuantizedTokenBridge, its vision/audio wrappers, and the existing audio-language ID adapter are implemented.
Combine text, image, and audio tokens into one explicit vocabulary and sequence format. MultimodalSequenceBuilder and MemoriedSpikeNetLM implement the shared low-memory CSR backbone.
Keep generation, classification, retrieval, and continuous reconstruction in task-specific heads. The current core provides range-limited text/image/audio token logits and a pooled classification head.

This permits low-risk text adoption first, followed by image, audio, and multimodal use only after the tokenization bridges and task heads exist.

Verified current implementation

Component	Current contract	Implementation meaning
`SparseEventMemoryLM`	`forward(token_ids)`, integer IDs to full-vocabulary logits	No image/audio arguments, masks, modality IDs, or continuous embedding input
`TiedFactorizedVocabulary`	INT8 `codebook` \((V,R)\) plus trainable `basis` \((R,D)\)	Shares input embedding and vocabulary projection; supports candidate CE
`EventMemoryExpert`	Processes dense float hidden states \((B,S,D)\) causally	Only primary synapses are CSR; spikes, hidden states, and stacked outputs are dense
`VectorQuantizedTokenBridge`	\((B,T,D)\) features to fixed-length in-range IDs, mask, and quantization error	Nearest-neighbor VQ; task gradients do not flow through ID selection, so bridges must be trained/frozen separately
`AudioLanguageTokenAdapter`	Existing audio-language IDs + confidence to audio-range IDs + mask	Validates that the source vocabulary fits, then applies the range offset
`MultimodalSequenceBuilder`	Text/image/audio IDs to delimiter, missing-token, padded batches	Produces consistent `token_ids`, `modality_ids`, and `validity_mask`
`MemoriedSpikeNetLM`	`MultimodalTokenBatch` to task-range logits or class logits	Implements modality/position embeddings and mask-aware routing, event state, and local plasticity
`SpikingEvoVisionEncoder`	Image \((B,C,H,W)\) to spike train \((B,T,D)\)	Does not produce an image-token sequence
`SpikingEvoAudioEncoder`	MFCC \((B,T,F)\) to spike train \((B,T_{snn},D)\)	Its implementation averages the MFCC time axis before LIF processing
`AudioToBrainLanguageSystem`	Waveform to token IDs \((B,S)\), confidence, optional features	Candidate bridge, but vocabulary semantics and trained quality must be validated
`SpikingEvoMultiModalLM`	`tokens`, optional `image`, optional `audio_mfcc`	Broadcasts image/audio spikes to text sequence length, concatenates, then uses dense spiking transformers
MEMERIED SDK helpers	`MemoriedSpikeNetLMConfig` and model/config/manifest/report artifacts	Serialize ranges, codebook revisions, and encoder manifests, then upload them

Difference from the existing multimodal model

SpikingEvoMultiModalLM forms text spikes \((B,T,S,D)\) and expands vision/audio spikes \((B,T,D)\) across text length \(S\) before dense fusion. The sparse event-memory core instead assumes one causal token sequence \((B,S)\). Replacing dense transformer blocks alone is therefore insufficient.

In particular, SparseEventMemoryLM.time_steps is retained for configuration compatibility but does not process an external spike-time axis. External SNN time and token order must remain separate concepts.

Compatibility matrix

Target	Sparse use now	Required additions	Priority
Token-LM usage of `SpikingEvoTextLM` and `TransformerLM`	Possible	No checkpoint migration; document positional/segment policy	P0
Discrete audio tokens from `AudioToBrainLanguageSystem`	Conditional	Vocabulary ranges, pad/unknown policy, confidence mask, tokenizer-quality validation	P1
Image captioning / understanding	Not possible	Vision token bridge, image-token vocabulary, image task head	P1
ASR / acoustic events	Not possible	Audio token bridge, temporal alignment, CTC or codec-token head	P1
Text + image + audio generation equivalent to `SpikingEvoMultiModalLM`	Not possible	Unified sequence builder, modality/position representations, masks, fusion evaluation	P2
Real-time audio or video	Not possible	Chunk tokenizer, persistent state cache, boundary handling, latency measurements	P3
EEG, LiDAR, tactile, or other sensors	Not possible	Sensor token adapter, vocabulary allocation, task head, data contract	P4

“Possible” means that the token-to-token language-model contract is compatible. It does not mean that all existing classes and checkpoints can be replaced.

Target architecture

flowchart LR
    TXT[Text tokenizer] --> TS[Text token IDs]
    IMG[SpikingEvoVisionEncoder] --> IT[Vision token bridge]
    AUD[Audio encoder or AudioToBrainLanguage] --> AT[Audio token bridge]
    TS --> SB[Multimodal sequence builder]
    IT --> SB
    AT --> SB
    SB --> IDS[Unified token IDs]
    SB --> MOD[Modality IDs and validity mask]
    IDS --> VOC[Factorized multimodal vocabulary]
    MOD --> EMB[Small modality and position adapters]
    VOC --> B[Shared sparse event-memory blocks]
    EMB --> B
    B --> TH[Text token head]
    B --> IH[Image token or image task head]
    B --> AH[Audio token or audio task head]
    TH --> OUT[Task result]
    IH --> OUT
    AH --> OUT

    classDef implemented fill:#e8f5e9,stroke:#2e7d32,color:#102a43
    classDef planned fill:#fff3e0,stroke:#ef6c00,color:#102a43
    class TXT,IMG,AUD,TS,IT,AT,SB,IDS,MOD,VOC,EMB,B,TH,IH,AH,OUT implemented

Principles

The shared backbone consumes discrete sequences. Images and audio are not sent directly into CSR layers; encoders and bridges turn their spatial/temporal structures into token sequences.
Token IDs alone do not express modality. Add small trainable modality and position/segment representations without making primary CSR synapses Adam-owned.
Missing modalities are explicit. Use a validity mask and start/end/missing special tokens, rather than relying on implicit all-zero tensors.
Heads are separate. Text uses a vocabulary head; ASR uses discrete audio tokens or CTC; image generation uses an image-code head; classification uses a pooled head.

Modality tokenization specification

1. Text

Existing HF tokenizer IDs occupy the text range:

\[ \mathcal{V}_{text}=[0,V_{text}). \]

P0 reuses TiedFactorizedVocabulary. From P1 onward, text IDs stay stable while special tokens and modality ranges extend the total vocabulary \(V_{total}\).

2. Vision

SpikingEvoVisionEncoder converts images into \((B,T_{vision},D_v)\) spike trains. A new VisionEventTokenizer will pool time and local/patch features into \(N_{image}\) vectors, then quantize them into a finite codebook:

\[ \mathbf{z}^{img}_n=g_{vision}(image)_n, \qquad q_n=\arg\min_{j\in\mathcal{V}_{image}}\lVert\mathbf{z}^{img}_n-\mathbf{c}^{img}_j\rVert_2^2. \]

The image range is

\[ \mathcal{V}_{image}=[V_{text},V_{text}+V_{image}). \]

VisionEventTokenizer and VectorQuantizedTokenBridge are implemented. The codebook, commitment loss, patch count, image size, and normalization must be recorded when operating a trained bridge.

3. Audio

Two audio paths are defined:

Use case	Token source	Primary objective
Semantic audio understanding / audio-to-language	`AudioToBrainLanguageSystem` or ASR	Semantic/audio-language token CE
ASR, acoustic events, music	New `AudioEventTokenizer`	Codec/event-token CE, CTC, or classification

SpikingEvoAudioEncoder returns \((B,T_{snn},D_a)\) from MFCCs but averages the incoming MFCC time axis internally. Frame-aligned ASR or acoustic tokenization should use AudioToBrainLanguageSystem.TemporalSpikingEncoder or add a time-preserving encoder API.

AudioEventTokenizer pools encoder features into a fixed number \(N_{audio}\) of bins and quantizes each bin. AudioLanguageTokenAdapter offsets existing AudioToBrainLanguageSystem IDs into the audio range and derives a mask from confidence:

\[ \mathbf{z}^{aud}_n=g_{audio}(waveform_{nH:nH+W}), \qquad q_n=\arg\min_{j\in\mathcal{V}_{audio}}\lVert\mathbf{z}^{aud}_n-\mathbf{c}^{aud}_j\rVert_2^2. \]

The range is

\[ \mathcal{V}_{audio}=[V_{text}+V_{image},V_{total}). \]

Sample rate, feature settings, chunk boundaries, resampling policy, and torchaudio availability belong in configuration and the manifest.

4. Unified sequence

The standard sample layout is:

\[ [\mathrm{BOS},\mathrm{TEXT\_START},x_{1:S_t},\mathrm{TEXT\_END}, \mathrm{IMAGE\_START},q^{img}_{1:N_i},\mathrm{IMAGE\_END}, \mathrm{AUDIO\_START},q^{aud}_{1:N_a},\mathrm{AUDIO\_END}]. \]

Missing sections are omitted, or use a single NO_IMAGE / NO_AUDIO token only where train/inference format needs stable section presence. Padding must always use validity_mask=False and be excluded from loss.

Shared sparse event-memory backbone

Implemented model boundary

Implement a new class rather than making a breaking change to SparseEventMemoryLM:

MemoriedSpikeNetLM.forward(
    batch: MultimodalTokenBatch,
    task: str = "text_generation",
) -> MemoriedSpikeNetOutput

token_ids use the current factorized vocabulary. modality_ids select small Adam-owned embeddings/adapters. Initial representations are:

\[ H^{(0)}=QE+M[\mathrm{modality\_id}]+P[\mathrm{position}], \]

where \(Q\) is the INT8 code, \(E\) is the shared basis, and \(M,P\) are small trainable tensors. Primary INT8 CSR synapses remain buffers.

Mask requirements

EventMemoryExpert and SparseEventMemoryBlock accept optional masks. The unified model implements the following requirements:

pool router inputs across valid tokens only;
exclude masked positions from expert state updates and plasticity aggregation;
use masks/ignore indexes for every task loss;
reject empty or all-padding batches explicitly.

Use a new class or backward-compatible optional arguments, preserving existing text APIs and adding dedicated regression tests.

Position and causality

Event experts are order-dependent because they process token positions recurrently, but they have no explicit positional embedding today. Multimodal sequences need to distinguish text positions, image patches, and audio chunks. P1 adds learned absolute positions up to a configured maximum length; P3 evaluates relative/continuous-time representation with chunk offsets.

Input, training, and artifact contracts

Output heads

Task	Output	Loss	Stage
Text generation	Text-vocabulary logits	Sampled CE / full CE evaluation	P0/P2
Multimodal instruction	Text-vocabulary logits	Text token CE	P2
Audio semantic tokens	Audio-vocabulary logits	Token CE	P1/P2
Frame-aligned ASR	Phoneme/wordpiece sequence	CTC or seq2seq CE	P3
Image reconstruction	Image-code logits	Image-token CE	P2
Image/audio/text classification or retrieval	Pooled representation	CE / contrastive loss	P2

sampled_cross_entropy() is candidate-set CE, not exactly full-vocabulary softmax. Validation must use appropriate full-vocabulary, candidate-retrieval, or task-specific metrics.

Training sequence

Modality encoders and bridges emit unified token IDs, modality IDs, masks, and targets.
The backbone produces hidden states through CSR event-memory blocks.
Task-head gradients update the shared basis, modality/position tensors, routers, adapters, LayerNorm, and optionally bridges.
Call apply_local_plasticity() exactly once after a successful optimizer.step().
If a step is abandoned because of an error, NaN, or overflow, call clear_local_plasticity().

Primary CSR local updates are not exact task-error gradients. Modality-level task metrics, router utilization, firing rate, and INT8 saturation are mandatory metrics.

Artifact manifest

Artifact	Required content
model	Backbone, task-head, and modality-adapter state dicts
config	Architecture ID; ranges; \(D,R,L,M,K,A,\rho\); max sequence length; head settings
tokenizer manifest	Text tokenizer ID/revision, image/audio codebook IDs/revisions, special-token table
encoder manifest	Vision/audio class, input shapes, sample rate, feature settings, checkpoint digest
memory report	Fixed CSR, codebooks, trainable tensors, measured peak memory, batch/length condition
metrics	Task metrics, router utilization, firing/saturation rates, latency, data split

Existing SparseEventMemoryConfig is text-only and cannot represent this information. The multimodal config must use a separate architecture discriminator rather than pretending to be text-checkpoint compatible.

Memory and streaming decisions

Static memory

Backbone CSR storage retains the current equation:

\[ M_{\mathrm{CSR}}=2LM\left(8(D+1)+9DF\right). \]

Multimodal usage additionally requires modality/position embeddings and adapters, image/audio codebooks, encoder/bridge weights and activations, dense sequence outputs proportional to \(S=S_t+N_i+N_a+special\), and task-head logits. A low-memory CSR backbone therefore does not imply low peak VRAM for long audio or vision encoders. Measure encoder-only, backbone-only, and integrated peaks separately.

Streaming

Current generate() recomputes the complete prefix and EventMemoryExpert resets its membrane at every forward call. It is not suitable for real-time audio/video.

P3 adds MemoriedStreamingState containing:

per-block/per-selected-expert INT16 membrane and previous spikes;
unfinished audio-chunk tokenizer buffers;
unified-token position offsets;
validated-token count and cache boundaries.

Because router selections can change between chunks, the first implementation must compare a stream-fixed router with preserving all expert states. CPU/NVMe expert paging does not exist today and is not a P3 exit criterion.

Implementation plan

P0 — Fix text-backbone contract

Work	Candidate files	Exit criterion
Contract documentation	`evospikenet/sparse_event_memory.py`, this document	Token-only boundary, state, full logits, and local plasticity are explicit
Regression	`tests/unit/test_sparse_event_memory_lm.py`	ID type, candidate CE, and CSR/Adam separation remain covered
Artifact continuity	`evospikenet/sdk/sparse_event_memory.py`	Existing text artifacts remain restartable

P1 — Modality-token bridges and common data contract (core implemented)

Work	Candidate files	Exit criterion
Common schema	`evospikenet/memoried_spikenetlm.py`	`MultimodalTokenBatch` and `MultimodalSequenceBuilder` define and produce IDs, modality IDs, and masks
Vision bridge	`evospikenet/multimodal_tokenizers.py`	Vision encoder output maps to in-range image IDs and a mask
Audio bridge	Same module	VQ for encoder features and a range adapter for existing audio-language IDs; direct waveform input and chunk metadata remain unimplemented
Data loading	`dataloaders.py` or dedicated module	Missing text/image/audio samples collate consistently
Configuration	`evospikenet/sdk/memoried_spikenetlm.py`	Ranges, codebook revisions, encoder settings serialize to JSON

Evaluate bridge quality before using it as a production tokenizer. Random codebooks are not sufficient: codebook training, freezing, and versioning must be tested.

P2 — Modality-aware sparse model and offline training (partially implemented)

Work	Candidate files	Exit criterion
Backbone	`MemoriedSpikeNetLM`	Mask-aware router, modality/position representations, and CSR local updates implemented
Heads	`evospikenet/memoried_heads.py`	Separate text, image-code, audio-code, and pooled-classification heads
Trainer	`examples/train_memoried_spikenetlm.py`	Mixed batches, post-optimizer plasticity, resume support
Metrics	`evospikenet/metrics/...`	Task loss/accuracy, router distribution, firing/saturation, peak memory
API	`api_modules/multimodal_api.py`	Architecture selector without breaking dense multimodal routes
SDK	`evospikenet/sdk/memoried_spikenetlm.py`	Config, payload, and upload implemented; download and local demo remain

P3 — Streaming audio and video

Work	Candidate files	Exit criterion
Streaming state	`evospikenet/memoried_streaming.py`	Tokenizer/event state/position offset survive chunk boundaries
Audio route	`api_modules/multimodal_api.py` or a dedicated router	PCM/WAV chunks, backpressure, and expiration are handled
Cache	Backbone/streaming module	Incremental outputs match an offline baseline within tolerance
Observability	SDK/API metrics	Chunk latency, buffer use, drop rate, router changes recorded

P4 — Additional sensors and operational optimization

Add SensorTokenAdapter implementations for LiDAR, EEG, tactile, and other sensors.
Add expert load balancing, capacity control, and long-sequence memory policy.
Treat GPU/CPU/NVMe expert paging as a separate proposal after buffer-transfer and local-plasticity-consistency benchmarks.

Tests, acceptance criteria, and risks

Layer	Test	Acceptance criterion
Token bridges	ID range, determinism, pad/mask, codebook version	No out-of-range ID; frozen codebook reproduces same output
Backbone	Masks, missing modalities, mixed batches, plasticity clearing	Padding does not affect router or plasticity statistics
Heads	Text/image/audio losses	Target shapes, ignore indexes, and candidate IDs are validated
Checkpoints	Text-only, audio, image, tri-modal	Config/manifest mismatch fails safely
SDK/API	Artifact transfer and architecture selection	Dense multimodal route has no regression
Performance	Peak VRAM, tokens/s, chunk latency	Comparisons retain workload conditions

Main risks are information loss from quantization, long audio token sequences, modality-dependent local-plasticity instability, mistaking token CE for perception quality, and breaking existing dense-model APIs/checkpoints. Measure token bridge quality independently, set token budgets and chunking before scaling, monitor per-modality statistics, use task metrics such as WER/event F1/retrieval/caption quality, and keep the new architecture discriminator separate.

Current SDK and documentation scope

SparseEventMemoryConfig, build_sparse_event_memory_model(), and upload_sparse_event_memory_artifacts() are for the text-only sparse LM. They must not be reused as if they describe multimodal artifacts.

For multimodal models, use MemoriedSpikeNetLMConfig, build_memoried_spikenetlm_model(), create_memoried_spikenetlm_artifact_payload(), and upload_memoried_spikenetlm_artifacts(). They separate model, config, token-range/revision/encoder manifest, and memory report. An API architecture selector, artifact download, and streaming API remain unimplemented.