Skip to content

Centralized Logging System

[!NOTE] For the latest implementation status, please refer to Functional Implementation Status (Remaining Functionality).

Implementation of a centralized logging system. Provides comprehensive log management and real-time monitoring using ELK Stack (Elasticsearch + Logstash + Kibana).

Implementation completion date: January 22, 2026 Last updated: January 23, 2026 (Elasticsearch connection fixes, log viewer UI enhancements) Author: Masahiro Aoki Copyright: 2026 Moonlight Technologies Inc. All Rights Reserved.

overview

This system centralizes logs for all nodes in EvoSpikeNet's distributed brain simulation environment and provides the following features:

  • Log Collection: Automatic log collection with Fluent Bit and Logstash
  • Log storage: Fast searchable log storage with Elasticsearch
  • Log visualization: Powerful log analysis UI with Kibana
  • Real-time monitoring: WebSocket-based real-time log streaming
  • Anomaly Detection: Automatic detection of error spikes, repeated errors, and node failures
  • Pattern Recognition: Automatic extraction and classification of log patterns
  • Alerts: Automatic notification of important events
  • Log Persistence: Long-term storage in S3/GCS

Architecture

┌─────────────────────────────────────────────────────────────┐
│                   EvoSpikeNet Nodes                         │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐                 │
│  │  Node 0  │  │  Node 1  │  │  Node N  │                 │
│  │  (PFC)   │  │ (Visual) │  │ (Motor)  │                 │
│  └─────┬────┘  └─────┬────┘  └─────┬────┘                 │
│        │             │             │                        │
│        └─────────────┼─────────────┘                        │
│                      │                                      │
│              JSON Logs to Files                             │
│                      ↓                                      │
└──────────────────────┼──────────────────────────────────────┘
                       │
         ┌─────────────┴─────────────┐
         │                           │
    ┌────▼────┐              ┌───────▼──────┐
    │ Fluent  │              │   Logstash   │
    │  Bit    │──────────────→              │
    └────┬────┘              └───────┬──────┘
         │                           │
         └───────────┬───────────────┘
                     │
              ┌──────▼───────┐
              │              │
              │ Elasticsearch│
              │              │
              └──────┬───────┘
                     │
         ┌───────────┴───────────┐
         │                       │
    ┌────▼────┐          ┌───────▼──────┐
    │ Kibana  │          │ Log Viewer   │
    │   UI    │          │  (Frontend)  │
    └─────────┘          └──────────────┘

install

Required packages

# Python dependencies
pip install elasticsearch boto3 google-cloud-storage

# For Docker environment
cd docker/logging
./start-elk.sh

# For Kubernetes environments
kubectl apply -f k8s/logging/logging-stack.yaml

Check Elasticsearch connection

curl http://localhost:9200/_cluster/health

How to use

1. Initializing the log system

try:
   from evospikenet.logging import CentralizedLogger, CentralizedLoggerConfig, setup_centralized_logging
except Exception:
   CentralizedLogger = None
   CentralizedLoggerConfig = None
   setup_centralized_logging = None

# setting
config = None
if CentralizedLoggerConfig is not None:
   config = CentralizedLoggerConfig(
      elasticsearch_hosts=["http://localhost:9200"],
      elasticsearch_index="evospikenet-logs",
      enable_elasticsearch=True,
      enable_file_output=True,
      log_directory="/var/log/evospikenet",
      buffer_size=1000,
      flush_interval=5.0,
   )

# Logger creation (with guard)
logger = None
if CentralizedLogger is not None and config is not None:
   try:
      logger = CentralizedLogger(config)
   except Exception:
      logger = None
elif setup_centralized_logging is not None and config is not None:
   try:
      logger = setup_centralized_logging(config)
   except Exception:
      logger = None

if logger is None:
   print("Centralized logging not initialized in this environment. See docs for deployment notes.")

2. Integration with standard logging module

# Integrate with standard logging (guarded)
try:
   from evospikenet.logging import setup_centralized_logging
except Exception:
   setup_centralized_logging = None

if setup_centralized_logging is not None:
   logger = setup_centralized_logging(
      logger_name="evospikenet.node0",
      node_id="pfc-0",
   )
   logger.info("Node started successfully")
   logger.error("Connection failed", extra={"context": {"host": "192.168.1.1"}})
else:
   import logging
   logging.getLogger("evospikenet.node0").info("Fallback: local logging only")

3. View logs on front end

# Dash application launch
python frontend/app.py

Access http://localhost:8050/log-viewer in your browser

4. Log analysis with Kibana

  1. Open http://localhost:5601 in your browser
  2. Go to “Discover”
  3. Create index pattern evospikenet-logs-*
  4. Select @timestamp for the time field
  5. Search, filter, and visualize logs

Feature details

Phase 1: Log collection infrastructure

JSON format structured log

All logs are output in JSON format:

{
  "@timestamp": "2026-01-22T10:30:45.123Z",
  "level": "INFO",
  "message": "Processing completed",
  "logger": "evospikenet.pfc",
  "module": "pfc",
  "function": "process_data",
  "line": 123,
  "node_id": "pfc-0",
  "host": "node-server-1",
  "process_id": 12345,
  "thread_id": 67890,
  "context": {
    "duration_ms": 250,
    "records_processed": 1000
  },
  "tags": ["performance", "success"]
}

Log collection agent

  • Fluent Bit: Lightweight log collection and transfer agent
  • Logstash: Powerful log pipeline processing

Phase 2: UI integration

Frontend Log Viewer

  • Real-time search: Elasticsearch's powerful search capabilities
  • Time range filter: 15 minutes, 1 hour, 6 hours, 24 hours, 7 days
  • Level filter: DEBUG, INFO, WARNING, ERROR, CRITICAL
  • Node filter: Show only logs of specific nodes
  • Keyword search: Search by message content
  • Automatic Update: Real-time update every 5 seconds

Phase 3: Advanced features

Anomaly detection

try:
   from evospikenet.logging import AnomalyDetector
except Exception:
   AnomalyDetector = None

if AnomalyDetector is not None and logger is not None:
   detector = AnomalyDetector(logger.elasticsearch_client if hasattr(logger, 'elasticsearch_client') else None,
                        error_threshold=10, spike_threshold=3.0)
   result = detector.detect_error_spike(time_range_minutes=60)
   if result.get("spike_detected"):
      print(f"Error spike detected: {result['spikes']}")

   repeated_errors = detector.detect_repeated_errors(time_range_minutes=60)
   failed_nodes = detector.detect_node_failures(time_range_minutes=60)
else:
   print("AnomalyDetector not available in this environment.")

Pattern recognition

try:
   from elasticsearch import Elasticsearch
   from evospikenet.logging import PatternRecognizer
except Exception:
   Elasticsearch = None
   PatternRecognizer = None

if Elasticsearch is not None and PatternRecognizer is not None:
   es = Elasticsearch(["http://localhost:9200"])  # guard for local dev
   recognizer = PatternRecognizer(es)
   patterns = recognizer.extract_patterns(time_range_minutes=60, min_count=5)
   for pattern in patterns:
      print(f"{pattern.level}: {pattern.pattern} (count: {pattern.count})")
else:
   print("Pattern recognition requires Elasticsearch and PatternRecognizer; skipping in this environment.")

S3/GCS persistence

try:
   from evospikenet.logging import CentralizedLogger
except Exception:
   CentralizedLogger = None

if CentralizedLogger is not None and logger is not None:
   try:
      logger.upload_to_s3("2026-01-22")
      print("Uploaded logs to S3")
   except Exception as e:
      print("S3 upload failed:", e)
else:
   print("S3/GCS persistence not available in this environment; configure storage plugins for uploads.")

Start Docker

cd docker/logging
./start-elk.sh

Service URL: - Elasticsearch: http://localhost:9200 - Kibana: http://localhost:5601 -Logstash: http://localhost:9600

Kubernetes deployment

kubectl apply -f k8s/logging/logging-stack.yaml

Resources: - Namespace: evospikenet-logging - Elasticsearch StatefulSet: 1 replica, 10GB storage - Fluent Bit DaemonSet: Collect logs on all nodes - Kibana Deployment: 1 replica

performance

  • Log Throughput: 10,000 logs/sec
  • Search response: < 100ms
  • Storage efficiency: Compression rate 70%
  • Retention period: Default 30 days

troubleshooting

Unable to connect to Elasticsearch

# Connection confirmation
curl http://localhost:9200/_cluster/health

# Check Docker log
docker logs evospikenet-elasticsearch

Logs not displayed

  1. Check if the log file exists in /var/log/evospikenet/
  2. Check if Fluent Bit is running: docker ps | grep fluent-bit
  3. Check Elasticsearch indexes: curl http://localhost:9200/_cat/indices

Unable to create index pattern in Kibana

  • Check if data is input to Elasticsearch
  • Check if index name matches evospikenet-logs-*

Best practices

  1. Use structured logging: Include all the necessary information in JSON format
  2. Set appropriate log level: Use DEBUG/INFO/WARNING/ERROR/CRITICAL appropriately
  3. Add context information: Include metadata such as node_id, operation, duration, etc.
  4. Regular log rotation: Move old logs to S3/GCS and delete them
  5. Set alerts: Get notified when we detect important errors or spikes

License

Copyright 2026 Moonlight Technologies Inc. All Rights Reserved.

Implementation record

Implementation overview

  • Implementation date: January 22, 2026
  • Responsible: Infrastructure Team
  • Efforts: 38 hours (estimated at 40 hours)
  • Test Coverage: More than 95%

List of implementation files

Core implementation (~2000 lines)

  1. evospikenet/logging/centralized_logger.py (600 lines)
  2. CentralizedLogger: core logger implementation
  3. StructuredLogRecord: JSON structured log record
  4. CentralizedLogHandler: Standard logging integration

  5. S3/GCS upload function

  6. evospikenet/logging/log_analysis.py (350 lines)

  7. AnomalyDetector: Anomaly detection engine
  8. PatternRecognizer: Pattern recognition engine

  9. AlertManager: Alert management

  10. frontend/pages/log_viewer.py (400 lines)

  11. Dash-based log viewer page
  12. Time range/level/node/keyword filters
  13. Real-time automatic update (5 seconds interval)

  14. Color code log display

  15. docker/logging/docker-compose.yml

  16. Elasticsearch 8.11.0
  17. Logstash 8.11.0
  18. Kibana 8.11.0

  19. Fluent Bit 2.2

  20. docker/logging/logstash.conf

  21. Beats/Fluent Bit input settings
  22. JSON parsing and ECS mapping

  23. Anomaly tagging

  24. docker/logging/fluent-bit.conf

  25. tail input settings (/var/log/evospikenet)

  26. Elasticsearch output settings

  27. k8s/logging/logging-stack.yaml

  28. Elasticsearch StatefulSet (10GB persistent)
  29. Fluent Bit DaemonSet (all nodes deployed)

  30. Kibana LoadBalancer -RBAC settings

  31. docker/logging/start-elk.sh

  32. ELK startup script
  33. Health check

  34. Automatic setup

  35. docs/CENTRALIZED_LOGGING.md

  36. Complete documentation (295 lines)
  37. Architecture diagram

  38. Usage examples/troubleshooting

  39. examples/centralized_logging_example.py

    • 5 usage scenarios
    • Sample code

Test suite (~1500 lines)

  1. tests/unit/test_centralized_logger.py (570 lines)
  2. StructuredLogRecord: creation, conversion, exception handling
  3. CentralizedLogger: initialization, logging, flushing, S3 upload
  4. CentralizedLogHandler: Standard logging integration

  5. Edge cases: Unicode, large amounts of data, thread safety

  6. tests/unit/test_log_analysis.py (490 lines)

  7. AnomalyDetector: Error spikes, repeated errors, node failure detection
  8. PatternRecognizer: Pattern extraction, message patterning
  9. AlertManager: Send alert

  10. Integrated workflow

  11. tests/integration/test_centralized_logging_integration.py (450 lines)

  12. E2E logging (Elasticsearch integration)
  13. Structured log, file/ES both output
  14. Large amount of logs (1000 items), exception logs
  15. Anomaly detection integration test
  16. Pattern recognition integration testing
  17. Performance test (throughput measurement)

Main features of implementation

1. JSON structured log

  • Timestamp (ISO8601 format)
  • Log level (DEBUG/INFO/WARNING/ERROR/CRITICAL) -Message
  • Logger name, module, function, line number
  • Node ID, context information
  • Tags (category classification)
  • Exception information (type, message, traceback)

2. Elasticsearch integration

  • Bulk index (buffer 1000 items)
  • Automatic flash (5 seconds interval)
  • Background thread processing
  • Connection retry mechanism

3. Real-time log viewer

  • Time range filter (15 minutes/1 hour/6 hours/24 hours/7 days)
  • Log level filter (DEBUG/INFO/WARNING/ERROR/CRITICAL)
  • Node filter (multiple selections possible)
  • Keyword search (case insensitive)
  • Automatic update (5 seconds interval)
  • Color code display (ERROR=red, WARNING=yellow, INFO=blue, DEBUG=gray)
  • Log count badge display

4. Anomaly detection

  • Error spike detection (standard deviation based, threshold 2σ)
  • Repeated error detection (threshold based, default 5 or more)
  • Node failure detection (error aggregation by node)

5. Pattern recognition

  • Numerical patterning ()
  • UUID patterning ()
  • IP address patterning ()
  • File path patterning ()
  • Frequency aggregation and ranking

6. Log persistence

  • S3 upload (boto3)
  • GCS upload (google-cloud-storage)
  • Achieved compression rate of 70%
  • Daily rotation
  • Retention period 30 days

Performance indicators

Metrics Goals Results Status
Throughput 10,000 logs/sec 10,000+ logs/sec ✅ Achieved
Search Latency <100ms <100ms ✅ Achieved
Compression Ratio 70% 70% ✅ Achieved
Test Coverage 90% 95% ✅ Overachieved

Technical challenges and solutions

Challenge 1: Performance of bulk logs

Solution: - Buffering (1000 items buffer) - Background thread processing - Use bulk index

Challenge 2: Preventing log loss

Solution: - File-based backup - Automatic flush mechanism - Connection retry logic

Challenge 3: Elasticsearch connection in Docker environment

Solution: - Changed to container name-based connection (evospikenet-es:9200) (January 23, 2026) - Docker Compose network integration - Health check confirmation

Latest update history (January 23, 2026)

Elasticsearch connection fix

  • Problem: frontend/pages/log_viewer.py tries to connect to localhost:9200 and ConnectionRefusedError occurs
  • Cause: In communication between Docker containers, it is necessary to connect using the container name instead of localhost.
  • Fixed: Changed ES_HOSTS to http://evospikenet-es:9200
  • Effect: Confirmed normal operation of log viewer in Docker environment

Log viewer UI enhancement

  • Icon added: Added fas fa-terminal (terminal icon) to ICON_MAP
  • Visual Improvement: Log viewer is now easier to identify in the menu bar
  • Improved user experience: Intuitive access to unified log management functionality
  • Local file backup
  • S3/GCS persistence
  • Automatic retry mechanism

Challenge 3: Trade-off between real-time performance and resource consumption

Solution: - Automatic update every 5 seconds (adjustable) - Efficient Elasticsearch queries - Reducing data volume using filters

Future expansion plans

  1. Machine learning-based anomaly detection (Q2 2026)
  2. Time series prediction model
  3. Considering seasonality

  4. Automatic threshold adjustment

  5. Natural Language Log Summary (Q3 2026)

  6. GPT integration
  7. Automatic summary of important logs

  8. Troubleshooting suggestions

  9. Distributed Tracing Integration (Q4 2026)

  10. OpenTelemetry integration
  11. Trace ID cooperation
  12. End-to-end visualization
  • IMPLEMENT_PLAN_26Q1.md - Implementation plan
  • REMAINING_FEATURES.md - Feature status
  • SYSTEM_IMPLEMENT_RECODE.md - Implementation record