Dependency Management Guide

[!NOTE] For the latest implementation status, please refer to Functional Implementation Status (Remaining Functionality).

Overview

EvoSpikeNet's dependency management takes a multi-layered approach with an emphasis on security and stability.

Dependency structure

Core Dependencies (`pyproject.toml`)

Project dependencies are managed in pyproject.toml and are divided into the following categories:

Core utilities: File locking, progress display, system monitoring
Data processing: NumPy, Pandas, SciPy, H5Py
ML/AI frameworks: PyTorch, SNNTorch, Transformers
Visualization: Plotly, Matplotlib, Dash
Networking: NetworkX, BeautifulSoup
Databases: Milvus, Elasticsearch, PostgreSQL
Web frameworks: FastAPI, Uvicorn
NLP: Janome, Tiktoken, SentencePiece
Audio/Image: Librosa, Pillow
Optimization: Optuna, Flwr

Version Constraints

All dependencies are specified with clear version ranges:

"torch>=2.0.0,<2.6"  # Major version fixed, minor version updatable
"numpy>=1.24.0,<2.3"  # Maintain compatibility

Principles: - Lower bound: Minimum version that includes the required functionality - Upper limit: Latest tested version + buffer with minor 1-2 versions - Carefully evaluate major version changes

Dependabot settings

Automatic update strategy

Set the following in .github/dependabot.yml:

- package-ecosystem: "pip"
  schedule:
    interval: "weekly"
    day: "monday"
    time: "09:00"
  groups:
    pytorch:  # Group packages where compatibility is important
      patterns:
        - "torch*"
        - "snntorch"

Grouping Strategy: - PyTorch ecosystem: torch, torchvision, torchaudio, snntorch - Data processing: numpy, pandas, scipy - Web frameworks: fastapi, uvicorn, dash - Database: pymilvus, elasticsearch, psycopg2, SQLAlchemy - Testing: pytest related - Dev tools: black, isort, flake8, mypy

Pull Request Management

Weekly check (Monday 9:00 JST)
Open up to 10 PRs at the same time
Automatic labeling: dependencies, python
Automatic reviewer assignment

Regular update workflow

GitHub Actions: `dependency-update.yml`

Runs every Monday and checks:

Outdated packages: pip list --outdated
Security vulnerabilities: pip-audit
Safety check: CVE database verification
Test compatibility: Run test with latest version

Automatic issue creation

Automatically create an issue if there are 5 or more updatable packages:

## Weekly Dependency Update Report

### Outdated Dependencies
[リスト]

### Security Vulnerabilities
[リスト]

### Action Items
- [ ] Review outdated packages
- [ ] Address security vulnerabilities
- [ ] Update pyproject.toml

Manual update process

1. Check the current status

# Check for old packages
pip list --outdated

# security check
pip-audit

# dependency tree
pipdeptree

2. Update pyproject.toml

# Adjust version range
vim pyproject.toml

3. Lock file generation

# Using auto-generated script
python scripts/update_requirements.py

# or manually run pip-compile
pip-compile --resolver=backtracking --output-file=requirements.txt pyproject.toml

4. Test execution

# Install new dependencies in virtual environment
python -m venv test_env
source test_env/bin/activate
pip install -r requirements.txt

# test run
pytest tests/ -v

# Compatibility check
python -c "import evospikenet; print('✓ Import successful')"

5. Commit and Push

git add pyproject.toml requirements*.txt
git commit -m "chore(deps): update dependencies to latest compatible versions"
git push origin feature/update-dependencies

Requirements Files

requirements.txt

Fixed version for production environments
Automatically generated from pyproject.toml with pip-compile
includes all transitive dependencies

requirements-dev.txt

Includes development tools (black, mypy, pytest, etc.)
Generated with pip-compile --extra=dev

requirements-test.txt

Dependencies needed for testing only
Used in CI/CD

requirements-cpu.txt / requirements-gpu.txt

CPU-only/GPU-only dependencies
Distinguish between CPU/CUDA versions of PyTorch

Security Best Practices

1. Regular security scans

# pip-audit (recommended)
pip-audit

# safety
safety check

# bandit (code scan)
bandit -r evospikenet/

2. Vulnerability response flow

Detected: Found in Dependabot or weekly scan
Evaluation: Check CVSS score and impact range
Fixed:
Critical/High: Immediate response (within 24 hours)
Medium: Within 1 week
Low: At the next regular update
Test: Regression test after modification
Deployment: Hotfix or scheduled release

3. Trustworthy sources

Uses only PyPI official repositories
Package signature verification (if possible)
The main package is hash verification

troubleshooting

Dependency conflicts

# Detailed competitive information
pip install --dry-run --report report.json -r requirements.txt

# Check dependency graph
pipdeptree --warn conflict

Relaxation of version fixation

As a temporary solution, relax version constraints for certain packages:

# Before
"package>=1.0.0,<1.5"

# After (temporarily removes upper limit)
"package>=1.0.0"

Note: Always set appropriate limits after testing

Python version compatibility

# Testing with multiple Python versions
tox

# or manually
for v in 3.10 3.11 3.12; do
  python$v -m venv env$v
  source env$v/bin/activate
  pip install -r requirements.txt
  pytest tests/
  deactivate
done

Continuous Integration

GitHub Actions integration

- name: Install dependencies
  run: |
    pip install -r requirements.txt

- name: Cache dependencies
  uses: actions/cache@v4
  with:
    path: ~/.cache/pip
    key: ${{ runner.os }}-pip-${{ hashFiles('requirements.txt') }}

Pre-commit hooks

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/Lucas-C/pre-commit-hooks-safety
    rev: v1.3.1
    hooks:
      - id: python-safety-dependencies-check

Maintenance schedule

Task	Frequency	Responsibility
Dependabot PR confirmation	Weekly (Monday)	Development team
Security Scan	Weekly (Monday)	Automatic + Review
Major version update	Quarterly	Tech Lead
Dependency cleanup	Semi-annual	All teams
Document update	When updating version	Person in charge