EvoSpikeNet Connectome Integration – Basic Implementation Policy

[!NOTE] For the latest implementation status, please refer to Functional Implementation Status (Remaining Functionality).

overview

This document is based on connectome research (connectome_literature_review.ja.md/connectome_updates_2021-2026.ja.md))) to establish a basic policy for integrating the knowledge obtained in EvoSpikeNet's distributed brain simulation system (docs/NEUROSCIENCE_BRAIN_SIMULATION_PAPER.md, docs/DISTRIBUTED_BRAIN_SYSTEM.md).

1. Current status analysis: Biological mapping of EvoSpikeNet

EvoSpikeNet has already implemented "biological mapping" that associates each node with a cortical region.

EvoSpikeNet node	Corresponding cortical and subcortical regions	Connectome corresponding dataset
PFC / Q-PFC	Dorsolateral orbitofrontal cortex (dlPFC/OFC)	MICrONS cortical column data
Visual	Occipital cortex V1–V5, IT	MICrONS Layer 2/3, FlyEM visual lobe
Auditory	Primary auditory cortex (A1)	FlyEM Johnston organ, MICrONS
Speech	Broca's area (BA44/45)	Currently no published data (limited to humans)
Spatial	Superior parietal lobule/occipito-parietal cortex	MICrONS whole cortical connectivity matrix
Motor	Primary motor cortex (M1)	MICrONS M1 connectome (2024)
Episodic (hippocampus)	Hippocampus CA1/CA3/dentate gyrus	HCP diffusion tensor; Wu et al. 2024
Semantic	Medial aspect of temporal lobe (TE/TEO)	FlyWire mushroom body
Integrator	Cingulate/insular cortex (ACC/Insula)	MICrONS long-range axon projection

Progress (2026-03-19): Added connectome_source, cortical_region and ei_ratio fields for each node to NODE_TYPE_DEFINITIONS in evospikenet/node_types.py. Biological mapping was explicitly recorded at the source code level.

Current location (2026-03-19): connectome_loader.py (load_json / load_npz / save_npz / stratified_sample / spectral_coarsen / load) has been implemented in Phase E-1. In Phase E-2, connectome mapping (evospikenet/connectome/), synaptic delay buffer (SparseDelayBuffer), Zenoh metadata publisher, and ETag+TTL cache for all nodes were completed. A total of 102 tests passed (E-1: 30pass/2fail/4skip, E-2: 70pass).

2. Gap analysis: What you can gain from connectome integration

When comparing the facts revealed from connectome research and the current design of EvoSpikeNet, the following gaps exist.

2.1 Connection topology gaps

Perspective	Knowledge of connectome research	Current status of EvoSpikeNet
Connections between neurons	C. elegans: All 302 neurons and 7,000 synapses released	Internal nodes are connected with random initialization weights
Cortical layer structure	Layer 4 of V1 is input specialized, Layer 2/3 is lateral communication (MICrONS)	Abstracted as a single "visual node"
Directionality	Excitatory/inhibitory ratio and direction revealed with FlyWire	Only ± sign of weight
Long-distance connectivity	Intercortical white matter fascicle topography revealed in HCP	Homogeneous connectivity on Zenoh pub/sub
Developmental plasticity	Schlegel et al. 2024: Connection pruning associated with Drosophila growth is described	`DevelopmentalSchedule` exists, but cooperation with `structural_mask` is not implemented

2.2 Spike dynamics gap

Connectome data provides actual measurements of synaptic delay, conduction velocity, and synaptic strength, but EvoSpikeNet's current LIFNeuronLayer relies on fixed leak parameters and initial weights.

Progress (2026-03-19): Added ConnectomeLIFLayer (subclass of LIFNeuronLayer) to evospikenet/core.py. It holds structural_mask (bool buffer), delay_buffer (float32 buffer), and connectome_weight (float32 parameter), and provides float → int16 bridging conversion with apply_connectome_input(). Biological fidelity is improved by data injection from connectome_loader.py (Phase E-1 completed).

2.3 Scale strategy gap (Phase F-1 to F-3 specifications defined)

There is a large mismatch between the actual number of neurons in the connectome data and the representation dimension of the EvoSpikeNet nodes. The reduction strategy is defined in Section 3 Policy F・docs-dev/connectome_schema.md. The only remaining work is implementation in connectome_loader.py.

Connectome dataset	Real number of neurons	EvoSpikeNet node representation	Mismatch ratio
C. elegans	302	100〜302	≈1× (direct support available)
FlyWire Visual Lobe	~50,000	500~2,000	25~100×
MICrONS V1 Cortical Column	~65,000	500~2,000	30~130×
HCP (Whole Brain Macro)	—	Connections between nodes ~29	—

The specific reduction algorithms (stratified sampling, spectral reduction, cluster representation) are defined in Policy F (Section 3).

3. Basic implementation policy

Policy A: Definition of intra-node connection topology by connectome graph

Purpose: Initialize the connections of neuron groups inside a node with an adjacency matrix derived from public connectome data.

Specific approach: 1. Select public connectome data corresponding to each node type. - visual node → MICrONS V1 Layer 2/3 to Layer 4 connection matrix (obtained with CAVEclient API) - episodic node → C. elegans neural circuit (OpenWorm project JSON format) - motor node → MICrONS M1 connectome subgraph 2. Load the connection matrix in sparse COO format (PyTorch torch.sparse) and initialize the weight matrix W of LIFNeuronLayer. 3. Treat the connectivity matrix as a read-only structural constraint and apply STDP plasticity to the weight scalar above it.

Implementation location: Added connectome_topology argument to LIFNeuronLayer.__init__() in evospikenet/core.py.

# Example: Weight initialization from connectome topology
class LIFNeuronLayer(nn.Module):
    def __init__(self, ..., connectome_topology: Optional[torch.sparse.Tensor] = None):
        ...
        if connectome_topology is not None:
            # Preserve topology as structure mask (no gradient required)
            self.structural_mask = connectome_topology.bool()
            # Initialize weights with connection density
            self.weight = nn.Parameter(
                connectome_topology.to_dense().float() * 0.1
            )
        else:
            self.weight = nn.Parameter(torch.randn(n_out, n_in) * 0.01)

Policy B: Guidelines for utilizing public connectome data

Define recommended connectome data sources for each node type.

B-1. C. elegans (nematode) — small scale, complete number, and precision

Target node: memory_spike (proxy for hippocampal CA3 self-associative circuit)
Data Source: OpenConnectome / WormWiring.net (WormAtlas)
JSON format, total 302 neurons and over 10,000 synapses
How to use: Benchmark prototype of small-scale SNN, self-federated network within a single node
Reference: White et al. 1986, Cook et al. 2019 (see detailed_paper_summaries.ja.md)

B-2. Drosophila FlyWire (Drosophila melanogaster) — Medium scale, whole brain, latest

Target node: visual, auditory, spatial_integration
Data source: fafbseg-py / CAVE API (via CloudVolume)
Approximately 140,000 neurons and 50 million synapses (2024 version)
How to use: Neuron circuit structure for sensory processing, learning connection patterns between nodes using GNN (Graph Neural Network)
Reference: Dorkenwald et al. 2024, Schlegel et al. 2024 (see connectome_updates_2021-2026.ja.md)

# FlyWire acquisition example (fafbseg-py / CAVE client)
from caveclient import CAVEclient
client = CAVEclient("flywire_public")
syn_df = client.materialize.synapse_query(
    pre_ids=visual_neuron_ids,
    post_ids=visual_neuron_ids
)

B-3. MICrONS (mouse visual cortex) — large-scale cortical layer structure

Target node: visual, spatial, PFC (node that maintains layer structure)
Data source: microns-explorer.org / MICrONS-py
Cortical column 200μm³, approximately 65,000 neurons and 500 million synapses (mm³ scale)
How to use: Reproduction of Layer 4→2/3→5 feedforward hierarchy, cortical microcolumn structure
Reference: Jiang et al. 2023, MICrONS Consortium 2021 (see connectome_updates_2021-2026.ja.md)

B-4. HCP (Human Connectivity Project) — Macro-scale and long-distance connectivity

Target node: Initialize Zenoh connection weight between nodes**
Data source: HCP S1200 release (FSL/MRtrix3 diffusion tensor data)
How to use: Reflect long-range connection topography such as dlPFC↔hippocampus, vision↔parietal lobe in Zenoh routing priority
Reference: Van Essen et al. 2013, Sporns et al. (see connectome_literature_review.ja.md)

Policy C: Integration of STDP + connectome topology (structural plasticity)

Purpose: Implement a two-layer model that uses connectome data as "structural constraints" while maintaining functional plasticity through STDP.

Design principles:

[構造層]  コネクトームトポロジー（変更不可の接続マスク）
    ↓
[機能層]  STDP/Meta-STDP による重みスカラーの更新
    ↓
[進化層]  EvoGenome によって構造マスク自体を進化的に変更（Phase E+）

Implementation details:

Structural pruning: Connectome data includes the distribution of connection strengths. Weak connections (below a threshold) are invalidated using a mask operation to reproduce biological pruning.
Synaptic delay table: Calculate the conduction delay from the connectome axon length data and store it in delay_buffer of LIFNeuronLayer.
Excitation/inhibition balance: Initialize a weight scalar that distinguishes excitatory (glutamate system)/inhibitory (GABA system) neurons based on FlyWire's E/I classification.

Connecting to existing code: - evospikenet/core.py: Added structural_mask + delay_buffer to LIFNeuronLayer. - evospikenet/forgetting_controller.py: Set pruning threshold from connectome density statistics - evospikenet/long_term_memory.py: Normalize long-term statistics of synaptic strength with connectome data.

Policy D: Integration of connection information into Zenoh metadata

Purpose: Reflect macroscale connection information (HCP data) derived from the connectome to routing priority, delay characteristics, and bandwidth allocation in Zenoh pub/sub communication between nodes**.

Implementation approach:

Add the connectome metadata field to “Area descriptor (1Hz broadcast)” in DISTRIBUTED_BRAIN_SYSTEM.md.

// 領域ディスクリプタへの追加フィールド（案）
{
  "node_type": "visual",
  "cortical_region": "V1-V5",
  "connectome": {
    "source": "MICrONS-2023",
    "avg_indegree": 1847,
    "avg_outdegree": 1203,
    "ei_ratio": 4.2,
    "preferred_upstream": ["spatial", "auditory"],
    "preferred_downstream": ["pfc", "spatial", "memory_spike"]
  }
}

ExecutiveControlEngine refers to this metadata and assigns a priority of low latency and high bandwidth to Zenoh paths between node pairs with high connection density derived from the connectome.
Add the synapse_weight_delta (INT16) field to the Protobuf payload of the spike packet and put it on the existing infrastructure of apply_weight_delta() (Phase D has been implemented).

Policy E: Connectome data update cycle (proofreading cooperation)

Purpose: Since connectome data such as FlyWire and MICrONS are continuously updated and modified, we will design a cycle to automate the reflection in EvoSpikeNet parameters.

Pipeline design:

[コネクトームDB更新検知]
    ↓  (週次cronジョブ / CAVEclientバージョン差分API)
[差分グラフ生成]
    ↓  (追加・削除シナプスのCOO行列デルタを計算)
[EvoSpikeNetパラメータ適用]
    ↓  apply_weight_delta() 経由で対象ノードへ配信
[検証]
    ↓  既存テストスイート（pytest）でスパイク率・E/Iバランスを検証
[ロールバック]
    ↓  検証失敗時はEvoGenomeの前バージョンをrestore

Specific implementation location (new): - scripts/sync_connectome.py: Get the difference from CAVEclient API and save it as JSON - evospikenet/connectome_loader.py: Utility to convert JSON connectome to PyTorch sparse tensor - config/connectome_config.yaml: Manage data source URL, version, and target node mapping

Policy F: Scale reduction strategy (N_connectome → N_node)

Purpose: Define a specific algorithm for reducing the number of real neurons (hundreds to hundreds of thousands) to a practical representation dimension (100 to 2,000 neurons) of EvoSpikeNet nodes. For input/output schema specifications of data format, refer to docs-dev/connectome_schema.md.

F-1. Stratified Sampling

Evenly sample while maintaining the ratio of cortical layers (L2/3, L4, L5, L6) and cell types (pyramidal cells, interneurons). Application scenario: MICrONS V1 → visual node (maintaining layer structure is important)

import scipy.sparse as sp
import numpy as np
from typing import Tuple

def stratified_sample(
    adj: sp.csr_matrix,
    labels: np.ndarray,
    n_target: int,
    rng: np.random.Generator,
) -> Tuple[np.ndarray, sp.coo_matrix]:
    """
    層別サンプリング: 各 label クラスから比例的に n_target ニューロンを選択する。
    Args:
        adj: 元の隣接行列 (N×N, CSR形式)
        labels: 各ニューロンの層/細胞型ラベル (N,)  例: 0=L2/3E, 1=L4E, 2=L5E, 3=L6E, 4=PV, 5=SST
        n_target: 縮約後ニューロン数
        rng: 再現性のためのRNGインスタンス (np.random.default_rng(seed=42))
    Returns:
        (選択インデックス配列 shape=(n_target,), 縮小後隣接行列 COO)
    """
    classes, counts = np.unique(labels, return_counts=True)
    quota = np.round(counts / counts.sum() * n_target).astype(int)
    idx = np.hstack([
        rng.choice(np.where(labels == c)[0], int(min(q, (labels == c).sum())), replace=False)
        for c, q in zip(classes, quota)
    ])
    sub = adj[np.ix_(idx, idx)]
    return idx, sub.tocoo()

F-2. Spectral Coarsening

Using the eigenvectors of the graph Laplacian, the number of nodes is reduced while preserving the connection pattern (spectral characteristics and oscillation characteristics). Application scenario: FlyWire visual leaf → visual/auditory node (maintaining γ/θ oscillation pattern is important)

from sklearn.cluster import KMeans
from scipy.sparse.linalg import eigsh
import scipy.sparse.csgraph as csgraph

def spectral_coarsen(adj: sp.csr_matrix, n_target: int) -> sp.csr_matrix:
    """
    スペクトル縮約: ラプラシアン固有ベクトルによるクラスタリングで圧縮。
    計算量目安: O(N·k) (eigsh) + O(k^2·nnz) (密度集約)。
    N>10,000 では事前に層別サンプリングで削減することを推奨。
    """
    L = csgraph.laplacian(adj, normed=True)
    _, vecs = eigsh(L, k=n_target, which="SM")
    labels = KMeans(n_clusters=n_target, n_init=5, random_state=42).fit_predict(vecs)
    A_coarse = np.zeros((n_target, n_target), dtype=np.float32)
    for i in range(n_target):
        for j in range(n_target):
            mi, mj = (labels == i), (labels == j)
            denom = mi.sum() * mj.sum()
            A_coarse[i, j] = float(adj[np.ix_(mi, mj)].sum()) / max(denom, 1)
    return sp.csr_matrix(A_coarse)

F-3. Cluster representative (Hub-Representative)

A high-order hub neuron is selected as a representative, and the remaining neurons are integrated into that cluster. Retains hub structure/motif. Application scenario: HCP whole brain macro → Zenoh Initialization of inter-node connection weights

Pseudocode （クラスタ代表化）:
入力: 隣接行列 A ∈ R^{N×N}, 出次数ベクトル d
1. idx = argsort(d, descending=True)[:N_target]          # Hub representative selection
2. residual = {全ニューロン} \ idx
3. 各 r ∈ residual を最近傍ハブ h = argmin_{h∈idx} dist(r,h) に割り当て
4. A'[h1,h2] = Σ_{i∈cluster(h1), j∈cluster(h2)} A[i,j] / (|cluster(h1)|·|cluster(h2)|)
5. threshold_eff[h] = mean(threshold[cluster(h)]) + log(|cluster(h)|) · τ_correction
出力: 縮約後隣接行列 A' + effective_threshold ベクトル

F-4. Reduction strategy selection guidelines

Conditions	Recommended strategy	Main target nodes	Estimated calculation amount
Want to preserve cortical layer structure	Stratified sampling (F-1)	`visual`, `spatial`, `motor`	O(N log N)
Want to preserve γ/θ spectral characteristics	Spectral reduction (F-2)	`auditory`, `speech`	O(N·k)
Want to preserve hub/motif structure	Cluster representation (F-3)	`pfc`, `episodic`, `integrator`	O(N²) → Pre-stratification recommended
Small data such as C. elegans	Direct use (no reduction)	`memory_spike` (prototype)	—

Measures against violation of physiological constraints (when mutating EvoGenome): When mutating the structure mask with EvoGenome, apply the following pre-filter and penalty.

Hard clip constraining the E/I ratio to the range [3.5, 5.0]
Penalty term to reject mutations if connectivity density varies by ±30%
Filter variants whose delay values exceed the physiological range (0.5-20 ms)

def validate_structural_mask(mask: torch.Tensor, ei_ratio_range=(3.5, 5.0)) -> bool:
    """EvoGenome変異後の構造マスクが生理的制約を満たすか検証する。"""
    n_excit = (mask > 0).sum().item()
    n_inhib = (mask < 0).sum().item()
    if n_inhib == 0:
        return False
    ei = n_excit / n_inhib
    return ei_ratio_range[0] <= ei <= ei_ratio_range[1]

4. Mapping to EvoSpikeNet component

The correspondence table between policies A to E and existing implementation files is shown below.

Legend: ✅ Implemented / 🔧 Partially implemented / ⬜ Not yet started

Policy	Existing files (to be changed)	New files	Priority	Status
A: Topology initialization	`evospikenet/core.py` ✅	`evospikenet/connectome_loader.py` ✅	High	`ConnectomeLIFLayer` Implemented. Completed implementation of `load_json`/`load_npz`/`load` in `connectome_loader.py` (Phase E-1)
B: Data source utilization	`config/data_config.yaml`	`config/connectome_config.yaml` ✅	High	`connectome_config.yaml` Created and node name matched. Implement all node mapping in `evospikenet/connectome/node_mapping.py` (Phase E-2)
C: Structural plasticity	`evospikenet/core.py` ✅・`plasticity.py` ✅・`forgetting_controller.py` ✅	—	Medium	`ConnectomeLIFLayer.validate_ei_ratio()` Implemented. Added `structural_mask` argument to `STDP.compute_weight_updates()`. Add `compute_connectome_density()` to `forgetting_controller.py` (Phase E-2)
D: Zenoh metadata	`evospikenet/node_types.py` ✅	`evospikenet/zenoh_connectome_publisher.py` ✅	Medium	`NODE_TYPE_DEFINITIONS` Extended. Publish to topic `connectome/metadata/{node_id}` with `ConnectomeMetadataPublisher` (Phase E-2). Log-only mode without Zenoh
E: Update cycle	`evospikenet/genome_to_brain.py` ✅ (`apply_connectome_delta()` added) `scripts/sync_connectome.py` ✅ (Phase E-3 completed)	`evospikenet/brain_routing.py` ✅ `scripts/auto_node_mapper.py` ✅	✅ Completed	`sync_connectome.py` implementation completed (`apply_delta`, `apply_delta_with_validation`, `sync_connectome`, `fetch_cave_synapses_with_retry`). HCP routing is implemented in `brain_routing.py` and node mapper is implemented in `auto_node_mapper.py` (Phase E-3)
F: Scale reduction	`evospikenet/connectome_loader.py` ✅	`docs-dev/connectome_schema.md` ✅	High	Completed implementation of `stratified_sample()` (F-1) and `spectral_coarsen()` (F-2) in `connectome_loader.py` (Phase E-1). Cluster representation (F-3) starts from Phase E-3

5. Implementation roadmap by phase

This policy is positioned as Phase E (connectome integration phase) of EvoSpikeNet.

Current location (2026-03-19): Phases E-0, E-1, E-2, and E-3 have been completed. All 112 tests PASS (1 skip). Phase E-3-5 (HCP DUC acquisition and whole-brain E2E verification) is the next milestone.

Phase E-0 (Prerequisite resolution — ✅ Completed on 2026-03-19)

[x] evospikenet/node_types.py: Added all connectome node names, connectome_source, cortical_region, and ei_ratio to NODE_TYPE_DEFINITIONS. Newly register "visual", "auditory", "spatial", "spatial_integration", "memory_spike", "episodic", "integrator", "language". Retained old "audio" as a backwards compatible alias.
[x] evospikenet/core.py: Added ConnectomeLIFLayer(LIFNeuronLayer). Implemented structural_mask (bool buffer), delay_buffer (float32 buffer), connectome_weight (float32 Parameter), apply_connectome_input(), and validate_ei_ratio(). Provides type bridging conversion between int16 LIF and float32 connectome weights.
[x] evospikenet/genome_to_brain.py: Added apply_connectome_delta(). Apply COO sparse differences in priority order: ConnectomeLIFLayer → SynapseMatrixCSR → nn.Linear.
[x] evospikenet/plasticity.py: Added structural_mask: Optional[torch.Tensor] = None to the PlasticityRule.compute_weight_updates() abstract method and all subclasses (STDP, Homeostasis, MetaSTDP, STDPRule, HomeostaticRule, etc.). STDP enforces structural constraints by zeroing out updates to synapses outside the mask.
[x] config/connectome_config.yaml: Comments have been added to match the target_nodes of each data source to the above unified node name.

Phase E-1 (Prototype — ✅ Completed 2026-03-19)

[x] Implementation of evospikenet/connectome_loader.py
load_json(): C. elegans JSON → ConnectomeData TypedDict conversion (based on connectome_schema.md specification)
load_npz() / save_npz(): NPZ cache bidirectional conversion (1.8x speedup confirmed)
stratified_sample(): Stratified sampling (F-1) implementation
spectral_coarsen(): Spectral reduction (F-2) implementation
load(): Integrated load function with ETag + TTL cache control
Injection of C. elegans 302 neurons into memory_spike node E2E validation PASS
[x] FlyWire-derived topology injection into visual nodes PoC
ConnectomeLIFLayer.attach_sparse_delay_buffer() / SNNModel.forward() Delay routing implementation
Connection with SynapseMatrixCSR (via apply_connectome_input())
[x] Added regression test for E/I balance spike rate using pytest
tests/test_connectome_loader.py: JSON/NPZ load/reduction/E/I ratio check (30pass/2fail/4skip)
tests/test_lif_structural_mask.py: Mask, gradient, and validation of ConnectomeLIFLayer
[x] data/connectome/ directory created and JSON data placed

Phase E-2 (Node deployment — ✅ Completed on 2026-03-19)

[x] Apply connectome mapping to all node types (Policy B supports all sources)
MICrONS V1 → visual/spatial/pfc (stratified, N=2000)
FlyWire → auditory/spatial_integration (spectral, N=2000)
C. elegans → memory/memory_spike/episodic (no reduction)
Implementation: evospikenet/connectome/node_mapping.py — get_source_for_node(), build_manifest(), apply_to_layer()
[x] Added spectral_coarsen() implementation (F-2) to connectome_loader.py (already implemented in Phase E-1)
[x] Synaptic delay table: Incorporate delayed spike propagation using ConnectomeLIFLayer.delay_buffer into SNNModel.forward()
Implementation: evospikenet/connectome/delay_buffer.py — SparseDelayBuffer (COO format ring buffer)
ConnectomeLIFLayer.attach_sparse_delay_buffer() / SNNModel.forward() Delay routing implemented
[x] Added connectome field to Zenoh area descriptor (strategy D)
Implementation: evospikenet/zenoh_connectome_publisher.py — ConnectomeMetadataPublisher
Topic: connectome/metadata/{node_id}, works in logo-only mode without Zenoh
[x] Connectome density statistics linkage to forgetting_controller.py (remaining work of policy C)
Implementation: compute_connectome_density(structural_mask) added
[x] ETag + TTL cache invalidation (E-2-5)
Implementation: connectome_loader.py — read_etag(), write_etag(), _is_cache_expired()
[x] Tests: tests/test_node_mapping.py (18 tests), tests/test_delay_buffer.py (22 tests), tests/test_zenoh_connectome_publisher.py (30 tests), all 70 tests PASS

Phase E-3 (Production integration — ✅ Completed on 2026-03-19)

[x] Automatic update pipeline with scripts/sync_connectome.py
apply_delta() / apply_delta_with_validation() / sync_connectome() / fetch_cave_synapses_with_retry() Implemented
ConnectomeSyncValidationError exception class added
[x] Co-evolution mechanism of EvoGenome and connectome structure mask (integration with Phase D evolution engine)
E-3-2 hook implemented in evolution_engine.py
[x] Zenoh Routing Optimization with Macro-Scale Connectivity (HCP)
evospikenet/brain_routing.py: HCPDelayRouter, compute_delay_matrix(), optimize_routing_delays(), build_hcp_routing_table() implemented
[x] Automatic node mapper
scripts/auto_node_mapper.py: map_connectome(), generate_manifest(), MappingResult, NodeMappingEntry implemented
[ ] End-to-end verification using whole brain simulation (24 processes Full Brain) → Phase E-3-5 and later
Start after completion of HCP DUC acquisition

6. Mathematical supplement

Connectome topology matrix

The connections between neurons within node \(u\) are represented by the adjacency matrix \(\mathbf{A} \in \{0,1\}^{N \times N}\). Initialization of weight matrix using connection strength \(s_{ij}\) obtained from connectome data:

\[W_{ij}^{(0)} = \begin{cases} s_{ij} \cdot \alpha & \text{if } (i,j) \in A \\ 0 & \text{otherwise} \end{cases}\]

where \(\alpha\) is the scaling factor (typical value 0.1).

STDP with structural constraints

STDP update rule considering structure mask:

\[\Delta W_{ij} = \mathbf{A}_{ij} \cdot \begin{cases} A_+ e^{-\Delta t/\tau_+} & \Delta t > 0\\ -A_- e^{\Delta t/\tau_-} & \Delta t < 0 \end{cases}\]

If mask \(\mathbf{A}_{ij} = 0\), there is no connection (structural plasticity can only be changed by the evolution engine).

E/I balance constraints

Fix the excitatory (E) to inhibitory (I) ratio in each node's neuron population based on connectome data:

\[\frac{N_E}{N_I} \approx R_{\text{connectome}}, \quad W_I < 0, \quad W_E > 0\]

In the MICrONS data, \(R \approx 4.2\) (typical value for mammalian cortex).

7. Security and ethical considerations

Connectome data uses only public data under CC-BY license (FlyWire Public, C. elegans OpenConnectome, MICrONS public dataset).
Externalize the connectome API key and authentication information to the secrets section of config/connectome_config.yaml and do not commit it to the repository (make sure .gitignore has been added).

7.1 Procedures for using HCP human data (DUC acquisition procedure)

When using human connectome data (HCP S1200, etc.), the following steps must be completed before data acquisition begins.

Create ConnectomeDB account Create an account at https://db.humanconnectome.org and register your affiliated institution and purpose of use (research category).
Signing the Data Use Certification (DUC) HCP Open Access data (structural MRI, diffusion MRI, rsfMRI) is provided under CC-BY 4.0, but restricted data (behavioral data, MEG, etc.) requires an electronic signature from the NIH DUC (updated annually).
Steps: ConnectomeDB Login → Account → Data Use Terms → Agree and Enable Access
DUC is valid for 1 year. Failure to renew annually will result in automatic suspension of access.
Verify your institution's IRB approval Research institutions in Japan may require institutional review board (IRB) approval or exemption certification for secondary use of human-derived data. Check with your organization's research ethics department.
Data access record storage Record the DUC signature document, download date, and version used (HCP S1200, etc.) in the project's docs/data_use_log.md (tracked in the repository).
Types and conditions of available data

Data type	Terms of use	Personal information risk
Structural MRI (T1w/T2w)	Open Access (CC-BY 4.0, HCP account required)	Low (de-identified)
Diffusion MRI (dMRI)	Open Access (CC-BY 4.0, HCP account required)	Low (de-identified)
Resting fMRI (rsfMRI)	Open Access (CC-BY 4.0, HCP account required)	Low (de-identified)
Behavioral data/psychometrics	Restricted (DUC signature required)	Medium (risk of identifying individuals)
MEG data	Restricted (DUC signature required)	Medium (risk of personal identification)

Important: When uploading HCP data to cloud storage (AWS S3, GCS, etc.), please comply with the DUC additional terms (private bucket required, third party sharing prohibited). Similar constraints apply when incorporating EvoSpikeNet's docker-compose or automatic acquisition into a CI/CD pipeline.

8. Performance estimation

8.1 Memory cost (per node)

Component	Estimated memory amount	Notes/compression strategy
`structural_mask` (COO, N=2000, density 1%)	~3.2 MB	int8 × 40,000 non-zero elements
`delay_buffer` (float32, N=2000, max_delay=20ms)	~160 MB (uncompressed)	Can be compressed to ~2-5 MB by keeping only non-zero delay
connectome NPZ cache (FlyWire whole brain)	~2 GB	~50 MB if stored reduced COO
Zenoh connectome metadata	< 1 KB	No bandwidth impact after JSON compression

Delay buffer compression implementation:

class SparseDelayBuffer:
    """非ゼロ遅延シナプスのみ保持する圧縮遅延バッファ。OOMリスク対策。"""

    def __init__(self, delays: torch.Tensor):
        mask = delays > 0
        self.indices = mask.nonzero(as_tuple=False)  # [nnz, 2]
        self.delay_vals = delays[mask].long()         # [nnz] Unit: timestep
        self.buffer: dict[int, torch.Tensor] = {}    # t_emit → spike vector

8.2 CPU/GPU processing load

Processing	CPU estimation	GPU estimation	Notes
COO→dense conversion (N=2000)	~5 ms	~0.3 ms	Memory transfer bottleneck
Stratified sampling (N=65,000→2,000)	~200 ms (first time only)	—	NPZ cache of results
Spectral reduction (N=65,000, k=2,000)	~30 s (first time)	~8 s (CuPy)	Offline execution recommended
Apply STDP + structural_mask (1 step)	~2 ms	~0.1 ms	Sparse matrix multiplication

Recommended: Run the reduction process once offline (cache generation with CI/CD) and load the NPZ cache during execution.

8.3 Network Bandwidth (External API)

API	Estimated transfer amount	Recommended acquisition frequency	Measures
FlyWire Full Synapse Retrieval	~2 GB / Full Retrieval	Diff Only (Weekly)	Local NPZ Cache + ETag Difference Fetch
MICrONS V1 Subgraph	~200 MB / Retrieval	Monthly	Manage reduced NPZ outside the repository
HCP Diffusion Tensor	~50 GB / Subject	First time only	Commit only aggregated scalars

9. Implementation risks and countermeasures

#	Risk	Impact	Countermeasures
R1	Differential synchronization delayed or stopped due to CAVE/HCP API rate limit	High (pipeline stopped)	Local NPZ cache + `ETag` differential fetch + Backoff control with `rate_limit_rps` setting in `connectome_config.yaml`
R2	Node OOM crash due to `delay_buffer` memory increase	High (node instability)	`SparseDelayBuffer` retains only non-zero delay, `max_delay_ms` upper limit is set in config
R3	EvoGenome generates a structural mask for physiological constraint violations	Medium (learning divergence)	`validate_structural_mask()` pre-filter + E/I penalty term, reject if connection density variation exceeds ±30%
R4	Computational cost of spectral reduction delays CI	Medium (CI timeout)	Artifact cache reduced NPZ, CI uses 302 neuron small subset
R5	Version inconsistency due to proofreading update of connectome data	Low (parameter inconsistency)	Version hash is recorded in `connectome_config.yaml` and rollback is supported

Detailed settings for critical risks (R1 and R2): See the cache and rate_limits sections of connectome_config.yaml.

10. Clarification of indicators

The measurement conditions for the index referred to as the "improvement effect of connectome integration" are defined in the table below. Specify the task, dataset, and baseline settings to ensure reproducibility.

Metrics	Estimated improvement range	Task definition	Test dataset	Measurement method
Spike efficiency	+15~25%	CIFAR-10 classification (10 classes)	CIFAR-10 test set (10,000 sheets)	Total number of spikes ÷ Top-1 accuracy. Random initialization vs. connectome initialization with the same hyperparameters and 5 seed average
Number of convergence epochs	−20 to 30%	Same as above	Same as above	Number of epochs to achieve 95% verification accuracy (censored at 100 epochs)
E/I ratio maintenance accuracy	>90% maintenance rate	Initial value within ±10% after 100 epochs of learning	FlyWire-derived E/I label	E/I ratio learning curve (average)
γ/θ peak appearance frequency	1.5–2.0× of baseline	LFP spectrum during visual stimulus presentation	MICrONS-derived physiological stimulation	γ band (30–80 Hz)/θ band (4–8 Hz) amplitude ratio of LFP FFT
Zenoh average RTT	-5~15% reduction	Inter-node spike packet delivery	29-node Full Brain simulation	RTT histogram (pairwise) p50/p99

Common baseline settings: Random initialization (torch.randn * 0.01), LIF parameters (tau_m=20 ms, V_th=−55 mV, V_rest=−70 mV), Adam (lr=1e-3).

11. Short-term improvement proposal (with priority)

P1 (High/Do it now): Defining the input/output schema of `connectome_loader` and adding sample data

Define minimum public specifications that implementers can start without hesitation.
Proposed file: docs-dev/connectome_schema.md (created at the same time as this document)
Sample data candidate: docs-dev/sample_data/celegans_mini_302n.npz (C. elegans 302 neurons, derived from OpenWorm)

P2 (middle/next step): Determine subgraph sampling strategy and develop experimental protocol

Policy F (Section 3) has been addressed. See Section 13 for experimental protocol.
Algorithm selection decision: FlyWire → spectral reduction (F-2), MICrONS → stratified sampling (F-1) as the first option.

P3 (medium/operation maintenance): Add cache/failback settings to `config/connectome_config.yaml`

Created as connectome_config.yaml (at the same time as this document).
Cache strategy: Local NPZ cache → Get only differences → Fallback to existing cache in case of API failure.

12. Test plan (short term)

`tests/test_connectome_loader.py`

Purpose: Consistency testing of small FlyWire subgraph → COO transformation
test_load_celegans_json_to_coo: C. elegans JSON → COO tensor conversion (nnz number/dtype/range check)
test_stratified_sample_ei_ratio: E/I ratio after stratified sampling is within ±10% of the original data
test_spectral_coarsen_eigenvalue_preservation: Top 5 eigenvalues are within ±20% of the original graph
test_cache_hit_performance: Load time on NPZ cache hit is less than 100 ms

`tests/test_lif_structural_mask.py`

Purpose: forward comparison after applying structural_mask (random vs applied mask)
test_mask_zeros_disconnected_synapses: Weights of off-mask synapses are kept at 0.
test_ei_ratio_preserved_after_stdp: E/I ratio remains within [3.5, 5.0] range even after STDP 1,000 steps
test_validate_structural_mask_rejects_invalid: Mutation masks with E/I ratio less than 3.5 are rejected.
test_mask_gradient_does_not_flow: Check that the gradient does not flow in structural_mask

`tests/test_sync_connectome_integration.py`

Purpose: Difference application → pytest regression automation workflow (mock CAVE response)
test_diff_apply_adds_new_synapses: Mock diff JSON → accurate application of weight_delta
test_diff_apply_removes_pruned_synapses: Removed synapses are excluded from the mask.
test_rollback_on_validation_failure: Rollback fires after applying E/I violation difference
test_mock_cave_rate_limit_retry: Rate limit 429 response → backoff retry operation confirmation

13. Recommended Metrics & Experiments

13.1 Learning Curve Benchmark

Experimental conditions	Setting values
Task	CIFAR-10 classification (EvoSpikeNet `visual` node alone)
Model scale	N=2,000 neurons, T=50 timesteps
Comparison conditions	Random initialization vs. MICrONS stratified sampling (F-1) vs. FlyWire spectral reduction (F-2)
Seed	5 seeds (1, 42, 123, 456, 789)
Evaluation index	Top-1 accuracy per epoch, total number of spikes, estimated energy (number of spikes × E_spike = 3.7 pJ)

13.2 Periodic evaluation of physiological indicators

E/I Ratio Time Series: Records the E/I ratio of each node every 50 epochs and automatically alerts on [3.5, 5.0] deviations.
FFT γ/θ peak appearance frequency: Apply FFT to the average membrane potential during visual stimulus presentation to evaluate the γ (30–80 Hz)/θ (4–8 Hz) band power ratio.
Spike synchronization (PLV: Phase Locking Value): Calculate the phase locking of neuron pairs in a 100 ms window and check the correlation with connectome distance.

# PLV calculation example (spike train spikes: [T, N])
def compute_plv(spikes: torch.Tensor, window_ms: int = 100) -> torch.Tensor:
    phase = torch.angle(torch.fft.fft(spikes.float(), dim=0))
    plv = torch.abs(torch.exp(1j * (phase[:, :, None] - phase[:, None, :])).mean(0))
    return plv  # [N, N]

13.3 System indicators

Zenoh average RTT: Visualize the (29×29) RTT histogram (p50/p99) by node pair with Prometheus + Grafana. Comparison before and after applying connectome-derived routing.
Memory/CPU consumption difference: Compare peak RSS before and after introducing structural_mask using memory_profiler (forward() × 1,000 steps).
Change in spike rate after differential update: Detect spike rate shift for each node after executing sync_connectome.py using regression test.

14. Illustrations/Architecture Diagram

Save the main diagrams in SVG format to docs-dev/assets/ for use by the implementation team.

14.1 Connectome Integration Architecture

graph TD
    subgraph DataSources["connectome data source"]
        CE[C. elegans<br/>302 neurons]
        FW[FlyWire<br/>~140k neurons]
        MC[MICrONS<br/>~65k neurons]
        HCP[HCP<br/>Whole brain macro]
    end
    subgraph ScaleReduction["Scale reduction (Policy F)"]
        SS[Stratified sampling<br/>F-1]
        SC[Spectral reduction<br/>F-2]
        HR[Cluster representation<br/>F-3]
    end
    subgraph EvoSpikeNet["EvoSpikeNet node layer"]
        CL[connectome_loader.py<br/>COO tensor generation]
        SM[structural_mask<br/>LIFNeuronLayer]
        DB[delay_buffer<br/>SparseDelayBuffer]
        STDP[Meta-STDP<br/>Functional plasticity]
        EG[EvoGenome<br/>Structural evolution]
    end
    subgraph Communication["Zenoh communication layer"]
        ZR[Zenoh Routing<br/>HCP-derived weights]
    end
    CE --> SS --> CL
    FW --> SC --> CL
    MC --> SS --> CL
    HCP --> HR --> ZR
    CL --> SM
    SM --> DB
    SM --> STDP
    STDP --> SM
    EG --> SM
    SM --> ZR

SVG export: mmdc -i docs-dev/assets/connectome_architecture.mmd -o docs-dev/assets/connectome_architecture.svg

14.2 Reduction Pipeline

flowchart LR
    Raw["Raw connectome\nN=50k~140k"]
    --> |stratified / spectral / hub| Reduced["Reduced graph\nN=500~2,000"]
    --> |NPZ cache| Loader["connectome_loader\n.load_npz()"]
    --> |structural_mask| LIF["LIFNeuronLayer\nEvoSpikeNet"]
    --> |spike| Zenoh["Zenoh pub/sub"]

SVG export: mmdc -i docs-dev/assets/coarsening_pipeline.mmd -o docs-dev/assets/coarsening_pipeline.svg

15. References/Related Documents

connectome_literature_review.ja.md — Review of major connectome papers (White 1986 - Januszewski 2018)
mapping_methods_ja.md — Technical overview of EM/optical/tracer methods
connectome_updates_2021-2026.ja.md — Latest research and implementation trends from 2021 to 2026
detailed_paper_summaries.ja.md — Detailed summaries of 10 representative papers and illustration candidates

docs/NEUROSCIENCE_BRAIN_SIMULATION_PAPER.md — Whole brain simulation paper/mathematical model
docs/DISTRIBUTED_BRAIN_SYSTEM.md — 29-node distributed architecture specification
docs/DISTRIBUTED_BRAIN_NODE_TYPES.md — Node type and cortex correspondence table
docs/DISTRIBUTED_BRAIN_NODE_CONFIGURATION.md — Node configuration/simulation type

Major connectome data sources

Dataset	Species	Scale	URL / API
OpenWorm / C. elegans	C. elegans	302 neurons	`openworm.org`, WormAtlas JSON
FlyWire (Dorkenwald et al. 2024)	Drosophila	~140,000 neurons	`fafbseg-py`, `cave.fanc-fly.com`
MICrONS (2021–2023)	Mouse V1	~65,000 neurons	`microns-explorer.org`, `CAVEclient`
HCP S1200	Human (healthy)	Whole brain macro	`humanconnectome.org`

Main article (DOI)

Paper	DOI
White et al. 1986 (C. elegans)	https://doi.org/10.1098/rstb.1986.0056
Januszewski et al. 2018 (FFN)	https://doi.org/10.1038/s41592-018-0049-4
Dorkenwald et al. 2024 (FlyWire)	https://doi.org/10.1038/s41586-024-07558-y
Schlegel et al. 2024 (Drosophila development)	https://doi.org/10.1038/s41586-024-07763-9
MICrONS Consortium 2021	https://doi.org/10.1101/2021.07.28.454025
Tavakoli et al. 2025 (LICONN)	https://doi.org/10.1038/s41586-025-08985-1

16. Source code verification results (2026-03-19)

This section records the results of checking the consistency of the implementation plan with the current source code. Phase E-1 List the problems that need to be resolved before starting work, with priority.

16.1 Implemented and verified components ✅

Policy	Target file	Status
Strategy B Data source management	`config/connectome_config.yaml`	Already created. All data source cash rate limit settings are aligned with this document
Policy E `apply_weight_delta()` Infrastructure	`evospikenet/genome_to_brain.py` L431	Implemented (INT16 delta → float32 conversion mechanism to `nn.Linear`)
STDP foundation	`evospikenet/plasticity.py`/`evospikenet/genome.py` L95	Integer-based STDP is working. Can be referenced from `PlasticityConfig` via Genom
`DevelopmentalSchedule`	`evospikenet/brain_simulation.py` L1374	Implemented (but not dependent on connectome structure. Connection should be considered after Phase E-3)
Schema specification	`docs-dev/connectome_schema.md`	JSON/NPZ input/output schema, naming rules, and sample code are fully defined

16.2 Unimplemented components (new creation required) ❌

Policy	Target files	Priority	Notes
Policy A/F	`evospikenet/connectome_loader.py`	Top priority (Phase E-1)	The file itself does not exist. The schema is `connectome_schema.md` which is fully defined so you can start right away
Policy E Automatic update	`scripts/sync_connectome.py`	✅ Phase E-3 completed	`apply_delta()` / `sync_connectome()` / `fetch_cave_synapses_with_retry()` Implemented. `brain_routing.py` / `auto_node_mapper.py` also added in E-3

16.3 Critical integrity issue (must be resolved before Phase E-1 begins) ⚠️

Issue I-1: Node name mismatch (high risk)

The node names used by config/connectome_config.yaml and this document are inconsistent with the NODE_TYPE_DEFINITIONS keys in evospikenet/node_types.py.

Node name in documentation/config	Actual definition in `node_types.py`	Status
`visual`	`"vision"` (`NODE_TYPE_VISION`)	❌ Mismatch
`auditory`	`"audio"` (key of `NODE_TYPE_DEFINITIONS`) *	❌ Mismatch
`memory_spike`	`"memory"`	❌ Mismatch
`spatial_integration`	Not defined	❌ Does not exist
`speech`	`"speech"`	✅ Match
`pfc`	`"pfc"`	✅ Match
`spatial`	`"spatial"` * No entry in `NODE_TYPE_DEFINITIONS`	⚠️ Need confirmation
`motor`	`"motor"`	✅ Match

In node_types.py, the constant name is NODE_TYPE_AUDITORY = "auditory", but the key of the NODE_TYPE_DEFINITIONS dictionary is "audio", so there is a discrepancy internally.

Response policy (to be determined): Standardize on one of the following.

Plan A: Modify the descriptions in connectome_config.yaml and this document to match the actual key names in node_types.py ("vision", "audio", etc.).
Plan B: Add "visual", "auditory", "memory_spike", "spatial_integration" keys to NODE_TYPE_DEFINITIONS of node_types.py and maintain them as aliases.
Recommended: Unify the value of node_types.py NODE_TYPE_AUDITORY to "audio" (or vice versa) and match the connectome_config.yaml side (Plan A). To be determined prior to Phase E-1 test implementation.

Issue I-2: `LIFNeuronLayer` data type inconsistency (high risk)

The code sample of the implementation plan (strategy A) assumes that a float32-based weight matrix called nn.Parameter(torch.randn(...) * 0.01) is added to LIFNeuronLayer, but the current LIFNeuronLayer in evospikenet/core.py is an int16-only fixed-point implementation.

LIFNeuronLayer（現状）:
  - self.potential : torch.int16
  - SynapseMatrixCSR.weights : torch.int8
  - forward() : int32 を経由して int16 に戻す整数演算

方針A が想定する structural_mask:
  - torch.sparse_coo_tensor (float32)
  - nn.Parameter(float32) の重み行列
  → 型衝突が発生する

The conversion layer when incorporating structural_mask into the int16 calculation path is not clearly specified in the design.

Response policy (to be determined): Select one of the following.

Draft	Content	Scope of Impact
Plan A	Add the `float_mode=True` option to `LIFNeuronLayer` to allow float32 calculation paths to coexist	Only change `core.py`. Backward compatibility is maintained by argument default `False`
Plan B	Create a new subclass of `ConnectomeLIFLayer(LIFNeuronLayer)` exclusively for connectome integration	Add only to `connectome_loader.py` and `core.py`. No changes to existing code
Plan C	Apply connectome topology only to `IzhikevichNeuronLayer` (float implementation that exists in `core.py`)	The brain simulation settings become complicated as it is used differently with existing LIF
Recommended	Plan B (new `ConnectomeLIFLayer` subclass). Minimal changes, backward compatibility maintained, completed within Phase E-1 scope

Problem I-3: Target layer mismatch of `apply_weight_delta()` (medium risk)

Policy D states that `STDP updates derived from the connectome are put onapply_weight_delta()'', but the implementation ofevospikenet/genome_to_brain.pyL431 targets only the firstnn.Linearlayer in the module**, so it does not work forSynapseMatrixCSR` where the connectome topology is stored.

# Current apply_weight_delta(): only supports nn.Linear
for layer in module.modules():
    if isinstance(layer, nn.Linear):  # ← SynapseMatrixCSR is skipped here
        ...
        break

Response policy: - Add SynapseMatrixCSR to the target of apply_weight_delta() after Phase E-2 and implement an update path to .weights sparse tensor. - Or, create a new apply_connectome_delta(module_name, delta_coo) method in genome_to_brain.py that is dedicated to updating the connectome.

Problem I-4: STDP `structural_mask` argument is undefined (medium risk)

In order to implement the `STDP update rule with structural constraints'' of Policy C, it is necessary to add thestructural_maskargument to thePlasticityRule.compute_weight_updates()interface ofevospikenet/plasticity.py`, but this argument does not exist in the current abstract interface.

# Current signature (plasticity.py)
@abstractmethod
def compute_weight_updates(self, spike_history, synapse_matrix): ...

# Required signature (proposed changes)
@abstractmethod
def compute_weight_updates(
    self,
    spike_history,
    synapse_matrix,
    structural_mask: Optional[torch.Tensor] = None,  # ← Add
): ...

Countermeasure: Add structural_mask as an optional argument (default None) in Phase E-1. Existing subclasses can maintain backward compatibility by simply accepting and ignoring arguments.

16.4 Phase E-1 Prerequisite Checklist for Commencement

Before entering Phase E-1, all of the following must be marked with a "confirmation mark".

[x] Problem I-1 resolved (2026-03-19): Add "visual", "auditory", "spatial", "spatial_integration","memory_spike", "speech" toNODE_TYPE_DEFINITIONSofnode_types.py, Added"episodic","integrator","language", and also addedconnectome_source,cortical_region, andei_ratiofields to each entry. The old ``audio'' key is retained as a backward compatible alias. Comments inconnectome_config.yaml` are also aligned.
[x] Problem I-2 solved (2026-03-19): Added ConnectomeLIFLayer(LIFNeuronLayer) to evospikenet/core.py (adopted plan B). Holds structural_mask (bool dense buffer), delay_buffer (float32 buffer) and connectome_weight (float32 Parameter). Provide float→int16 bridging conversion in apply_connectome_input(). E/I validation with validate_ei_ratio().
[x] Problem I-3 resolved (2026-03-19): Added apply_connectome_delta() method to evospikenet/genome_to_brain.py. Apply updates to ConnectomeLIFLayer.connectome_weight → SynapseMatrixCSR.weights → nn.Linear in order of priority.
[x] Problem I-4 advance solution (2026-03-19): Added structural_mask: Optional[torch.Tensor] = None to the PlasticityRule.compute_weight_updates() abstract method of evospikenet/plasticity.py. STDP actually applies a mask to zero out the updates of synapses outside the connectome. All implementations of Homeostasis, MetaSTDP, STDPRule, and HomeostaticRule have also been updated for backward compatibility with the same argument.
[x] tests/test_connectome_loader.py Create a test file skeleton and incorporate it into CI (30pass/2fail/4skip).
[ ] Create the data/connectome/ directory and register it in .gitignore (sample data is stored in docs-dev/sample_data/).

*This document serves as the kickoff document for EvoSpikeNet Phase E. Implementation progress should be tracked in the ticket management system according to the roadmap above. *