Logs Command

The manta_node logs command displays and monitors log output from node instances, providing insights into node operation, task execution, and troubleshooting information.

Overview

The logs command provides:

  • View historical log entries

  • Real-time log streaming (tail -f)

  • Multiple node log management

  • Filtered log output

  • Log file discovery

Synopsis

manta_node logs [instance] [options]

Arguments

instance

Instance ID or name of the node

  • Optional: Uses single log if only one exists

  • Partial match supported (e.g., “prod” matches “production”)

Options

--follow, -f

Follow log output in real-time

  • Similar to tail -f

  • Press Ctrl+C to stop

--lines, -n <number>

Number of lines to display

  • Default: 50

  • Shows last N lines from log

--list

List all available log files

  • Shows file paths and sizes

  • Helps identify correct log

Usage Examples

View Recent Logs

Display last 50 lines (default):

$ manta_node logs production
2024-03-15 10:30:15 - INFO - Node starting with configuration: production
2024-03-15 10:30:16 - INFO - Connecting to manager at localhost:50051
2024-03-15 10:30:17 - INFO - Successfully registered with manager
2024-03-15 10:30:17 - INFO - Connected to MQTT broker
2024-03-15 10:30:18 - INFO - Ready to accept tasks
...

Follow Logs Real-time

Stream logs as they’re written:

$ manta_node logs production --follow
Following log file: ~/.manta/logs/nodes/production.log
Press Ctrl+C to stop
------------------------------------------------------------
2024-03-15 10:35:22 - INFO - Received task: task-abc123
2024-03-15 10:35:23 - INFO - Starting container for task
2024-03-15 10:35:25 - INFO - Task executing...
2024-03-15 10:35:45 - INFO - Task completed successfully
2024-03-15 10:35:46 - INFO - Results sent to manager
^C
Stopped following log file

Custom Line Count

View more or fewer lines:

# Last 100 lines
$ manta_node logs production --lines 100

# Last 10 lines
$ manta_node logs production -n 10

# All available lines (large number)
$ manta_node logs production --lines 99999

List Available Logs

Discover log files:

$ manta_node logs --list
Available log files:
  - production (2.3 MB) - ~/.manta/logs/nodes/production.log
  - development (0.5 MB) - ~/.manta/logs/nodes/development.log
  - test-node (1.1 MB) - ~/.manta/logs/nodes/test-node.log

Automatic Selection

When only one log exists:

$ manta_node logs
Using log file for instance: production
2024-03-15 10:30:15 - INFO - Node starting...

Multiple Logs Error

When multiple logs exist:

$ manta_node logs
Error: Multiple log files found. Please specify an instance:
  - production
  - development
  - test-node

Log File Management

Log File Location

Logs are stored in standardized locations:

~/.manta/logs/nodes/
├── production.log         # Active log
├── development.log        # Active log
├── test-node.log         # Active log
└── archived/             # Rotated logs (if configured)
    ├── production.log.1
    ├── production.log.2
    └── production.log.3

Log File Naming

Files are named based on:

  • Node alias: If configured (e.g., prod-gpu-1.log)

  • Config name: If no alias (e.g., production.log)

  • Default: Generic name (e.g., node.log)

Log Rotation

When configured, logs rotate based on:

[logging]
max_file_size = "100M"    # Rotate at 100MB
backup_count = 5          # Keep 5 backups

Rotated files:

  • node.log - Current log

  • node.log.1 - Most recent backup

  • node.log.2 - Older backup

  • node.log.3 - Even older

Log Content

Log Levels

Standard Python logging levels:

DEBUG   - Detailed diagnostic information
INFO    - General informational messages
WARNING - Warning messages for potential issues
ERROR   - Error messages for failures
CRITICAL- Critical failures requiring attention

Example log entries:

2024-03-15 10:30:15 - DEBUG - Loading configuration from file
2024-03-15 10:30:16 - INFO - Node initialized successfully
2024-03-15 10:30:17 - WARNING - Low memory: 512MB available
2024-03-15 10:30:18 - ERROR - Failed to connect to manager
2024-03-15 10:30:19 - CRITICAL - Docker daemon not accessible

Log Format

Default format:

%(asctime)s - %(levelname)s - %(name)s - %(message)s

2024-03-15 10:30:15 - INFO - manta_node.node - Starting node

Components logged:

  • Timestamp: When event occurred

  • Level: Severity of message

  • Logger name: Component that logged

  • Message: Actual log content

Common Log Patterns

Startup sequence:

INFO - Loading configuration: production.toml
INFO - Validating configuration
INFO - Initializing logging system
INFO - Connecting to manager at localhost:50051
INFO - Registering node with ID: node-abc123
INFO - Connected to MQTT broker at localhost:1883
INFO - Node ready to accept tasks

Task execution:

INFO - Received task: task-xyz789
INFO - Pulling Docker image: manta_light:pytorch
INFO - Creating container with limits: CPU=2, Memory=4GB
INFO - Task started in container: container-abc123
INFO - Streaming task output
INFO - Task completed with exit code: 0
INFO - Sending results to manager

Error conditions:

ERROR - Manager connection lost: Connection refused
WARNING - Reconnecting in 5 seconds...
ERROR - MQTT broker unreachable
ERROR - Task failed: Container exited with code 1
ERROR - Insufficient resources for task

Filtering and Searching

Using Shell Tools

Filter logs with grep:

# Find errors
manta_node logs production | grep ERROR

# Find specific task
manta_node logs production | grep task-abc123

# Find warnings and errors
manta_node logs production | grep -E "(WARNING|ERROR)"

# Exclude debug messages
manta_node logs production | grep -v DEBUG

Time-based Filtering

Find logs from specific time:

# Today's logs
manta_node logs production | grep "$(date +%Y-%m-%d)"

# Last hour
manta_node logs production | grep "$(date +%Y-%m-%d' '%H)"

# Specific time range
manta_node logs production | \
  awk '/2024-03-15 10:00/,/2024-03-15 11:00/'

Pattern Extraction

Extract specific information:

# Extract task IDs
manta_node logs production | \
  grep -o 'task-[a-z0-9]*' | sort -u

# Count errors by type
manta_node logs production | \
  grep ERROR | cut -d'-' -f4- | sort | uniq -c

# Find container IDs
manta_node logs production | \
  grep -o 'container-[a-z0-9]*'

Real-time Monitoring

Continuous Monitoring

Watch logs continuously:

# Follow single node
manta_node logs production --follow

# Follow with filtering
manta_node logs production -f | grep -E "(ERROR|WARNING)"

# Follow multiple nodes (in separate terminals)
# Terminal 1:
manta_node logs node1 -f
# Terminal 2:
manta_node logs node2 -f

Alert on Errors

Monitor for issues:

#!/bin/bash
# monitor_errors.sh

manta_node logs production -f | while read line; do
    if echo "$line" | grep -q "ERROR"; then
        echo "ALERT: Error detected: $line"
        # Send notification (email, slack, etc.)
        # notify-send "Manta Node Error" "$line"
    fi
done

Log Aggregation

Combine multiple logs:

# Merge all node logs
tail -f ~/.manta/logs/nodes/*.log | \
  sed 's/^/[NODE] /'

# With timestamps
for log in ~/.manta/logs/nodes/*.log; do
    name=$(basename "$log" .log)
    tail -f "$log" | sed "s/^/[$name] /" &
done
wait

Troubleshooting

Log File Not Found

When log file doesn’t exist:

Error: No log file found for instance 'production'

Solutions:

  1. Check if node is running: manta_node status

  2. Verify log location: manta_node logs --list

  3. Check configuration for log path

  4. Start the node first: manta_node start production

Empty or Missing Logs

When logs are empty:

# Check if file exists but empty
ls -la ~/.manta/logs/nodes/

# Check if logging is configured
manta_node config show production | grep logging

# Verify write permissions
touch ~/.manta/logs/nodes/test.log

Large Log Files

Managing large logs:

# Check log size
du -h ~/.manta/logs/nodes/*.log

# Truncate large log (careful!)
echo "" > ~/.manta/logs/nodes/production.log

# Archive old logs
gzip ~/.manta/logs/nodes/production.log.old

# Configure rotation in config
# [logging]
# max_file_size = "10M"
# backup_count = 3

Permission Issues

Fix log access problems:

# Check permissions
ls -la ~/.manta/logs/nodes/

# Fix permissions
chmod 644 ~/.manta/logs/nodes/*.log

# Fix ownership
chown $USER:$USER ~/.manta/logs/nodes/*.log

Advanced Usage

Log Analysis Script

Analyze log patterns:

#!/usr/bin/env python3
import re
from collections import Counter
from datetime import datetime

def analyze_logs(log_file):
    """Analyze manta node logs."""

    errors = []
    warnings = []
    tasks = []

    with open(log_file, 'r') as f:
        for line in f:
            # Parse log line
            if ' ERROR ' in line:
                errors.append(line)
            elif ' WARNING ' in line:
                warnings.append(line)

            # Extract task IDs
            task_match = re.search(r'task-[a-z0-9]+', line)
            if task_match:
                tasks.append(task_match.group())

    print(f"Log Analysis for {log_file}")
    print(f"Errors: {len(errors)}")
    print(f"Warnings: {len(warnings)}")
    print(f"Unique tasks: {len(set(tasks))}")

    # Show error summary
    if errors:
        print("\nMost recent errors:")
        for error in errors[-5:]:
            print(f"  - {error.strip()}")

# Usage
analyze_logs("~/.manta/logs/nodes/production.log")

JSON Log Processing

For JSON formatted logs:

# Parse JSON logs with jq
cat ~/.manta/logs/nodes/production.log | \
  jq -r 'select(.level=="ERROR") | .message'

# Extract metrics
cat ~/.manta/logs/nodes/production.log | \
  jq -r 'select(.event=="task_complete") | .duration'

Log Shipping

Send logs to central system:

# Stream to syslog
tail -f ~/.manta/logs/nodes/production.log | \
  logger -t manta-node -p local0.info

# Send to remote collector
tail -f ~/.manta/logs/nodes/production.log | \
  nc logserver.example.com 514

Best Practices

Log Management

  1. Regular rotation: Configure size/time-based rotation

  2. Compression: Compress old logs to save space

  3. Retention policy: Delete logs older than N days

  4. Backup important logs: Archive critical information

  5. Monitor disk usage: Prevent disk full errors

Debugging with Logs

  1. Enable debug level: For detailed troubleshooting

  2. Correlate timestamps: Match events across logs

  3. Search patterns: Look for error patterns

  4. Track task flow: Follow task lifecycle

  5. Compare nodes: Check differences between nodes

Production Logging

  1. Appropriate level: INFO or WARNING for production

  2. Structured logging: Use JSON for parsing

  3. Central aggregation: Ship logs to central system

  4. Alerting: Set up alerts for ERROR/CRITICAL

  5. Compliance: Ensure sensitive data isn’t logged

See Also