Start Command¶

The manta_node start command launches a node instance with a specified configuration, connecting it to the Manta platform for task execution.

Overview¶

The start command initiates a node agent that:

Registers with the Manta manager service
Subscribes to MQTT topics for task commands
Provides local and global services to running tasks
Monitors system resources and reports metrics
Executes containerized tasks in isolation

Synopsis¶

manta_node start [config_name] [options]

Arguments¶

config_name

Name of the configuration file to use (without .toml extension)

Default: default
Configuration must exist in ~/.manta/nodes/

Options¶

--alias <name>

Override the node alias from configuration

Example: --alias gpu-worker-1

--background, -d

Run the node in background (daemon mode)

Node continues running after terminal closes
Logs are written to file only
Returns control to terminal immediately

--foreground

Run the node in foreground (default)

Node runs in current terminal
Shows output in terminal if configured
Use Ctrl+C to stop

Usage Examples¶

Basic Start¶

Start a node with default configuration:

$ manta_node start
Starting node 'default' in foreground...
Press Ctrl+C to stop
✓ Node registered with manager
✓ Connected to MQTT broker
✓ Ready to execute tasks

Start with Specific Configuration¶

Use a named configuration:

$ manta_node start production
Starting node 'production' in foreground...
Using configuration: ~/.manta/nodes/production.toml
Node alias: prod-gpu-01
Manager: manager.example.com:50051
✓ Node started successfully

Background Execution¶

Run node as a background process:

$ manta_node start production --background
Starting node 'production' in background...
✓ Node 'production' started successfully
Instance ID: production-a3f2c891
Process ID: 12345
Log file: ~/.manta/logs/nodes/production.log

Override Alias¶

Start with a custom alias:

$ manta_node start gpu-config --alias experiment-node-1
Starting node 'experiment-node-1' in foreground...
Configuration: gpu-config
Overridden alias: experiment-node-1

Node Lifecycle¶

Startup Sequence¶

Load Configuration: Read TOML configuration file
Validate Settings: Check required fields and constraints
Initialize Logging: Set up log files and levels
Connect to Manager: Register with gRPC service
Subscribe to MQTT: Connect to message broker
Start Services: Launch local and world servicers
Begin Heartbeat: Send periodic health checks
Ready for Tasks: Listen for task commands

Registration Process¶

When a node starts, it:

Generates or uses configured node ID
Sends registration request with:
- Node ID and alias
- Available resources (CPU, memory, GPU)
- Dataset information
- Service endpoints
Receives registration confirmation with:
- MQTT broker details
- Topic subscriptions
- Initial configuration

Service Endpoints¶

The node provides two gRPC services:

LocalServicer: Data access for tasks (auto-assigned port)
WorldServicer: Global state access (shares LocalServicer port)

Tasks connect to these services using environment variables:

LIGHT_SERVICE_HOST=0.0.0.0
LIGHT_SERVICE_PORT=<auto-assigned>

Process Management¶

Foreground Mode¶

Default interactive mode:

Start: manta_node start [config]
Stop: Press Ctrl+C
Logs: Displayed in terminal (if configured)
Status: Real-time in terminal

Background Mode¶

Daemon process management:

Start: manta_node start [config] --background
Stop: manta_node stop [instance]
Logs: manta_node logs [instance]
Status: manta_node status

Instance Tracking¶

Running nodes are tracked in ~/.manta/nodes/instances/:

{
  "instance_id": "production-a3f2c891",
  "alias": "prod-gpu-01",
  "config_name": "production",
  "pid": 12345,
  "start_time": "2024-03-15T10:30:00",
  "status": "running",
  "manager_host": "localhost",
  "manager_port": 50051,
  "log_file": "~/.manta/logs/nodes/prod-gpu-01.log"
}

Signal Handling¶

The node handles system signals gracefully:

SIGTERM: Graceful shutdown (cleanup tasks)
SIGINT (Ctrl+C): Interactive shutdown
SIGKILL: Force termination (avoid if possible)

Error Handling¶

Configuration Errors¶

Missing configuration file:

Error: Configuration 'gpu-config' not found at ~/.manta/nodes/gpu-config.toml
Available configurations:
  - default
  - production

Create a new configuration with:
  manta_node config init

Invalid JWT token:

Error: No valid secured token found in configuration
A secured token (JWT) is required for node authentication.

Solutions:
  • Set environment variable: export MANTA_NODE_TOKEN=your_jwt_token
  • Edit config file: ~/.manta/nodes/production.toml
  • Update: secured_token = "your_jwt_token_here"
  • Or recreate config: manta_node config init

Connection Errors¶

Manager not reachable:

Error: Failed to connect to manager at localhost:50051

Troubleshooting steps:
  1. Check if configuration is valid: manta_node config validate
  2. Verify manager is running: docker ps (look for manta services)
  3. Check if port is available: netstat -ln | grep :50051
  4. Validate secured_token is set in config file

MQTT connection failed:

Error: Failed to connect to MQTT broker
Broker: emqx.local:1883

Check:
  - MQTT broker is running
  - Network connectivity
  - Firewall rules

Resource Errors¶

Docker not available:

Error: Docker daemon not accessible

Solutions:
  - Start Docker daemon: sudo systemctl start docker
  - Check permissions: sudo usermod -aG docker $USER
  - Verify socket: ls -la /var/run/docker.sock

Insufficient resources:

Warning: System resources below recommended levels
Available: 2 CPU cores, 3.5 GB RAM
Recommended: 4+ CPU cores, 8+ GB RAM

Node will start with reduced capacity

Log Files¶

Log Location¶

Logs are stored based on configuration:

Default: ~/.manta/logs/nodes/<node-name>.log
Custom: Specified in configuration file
Instance-specific: Named after node alias or config name

Log Configuration¶

Control logging behavior in config:

[logging]
level = "INFO"                              # DEBUG, INFO, WARNING, ERROR
filename = "~/.manta/logs/nodes/node.log"   # Log file path
log_to_file = true                          # Write to file
log_to_console = false                      # Show in terminal
debug_mode = true                           # Extra debug info
max_file_size = "100M"                      # Rotation size
backup_count = 5                            # Keep 5 backups

Viewing Logs¶

# View logs for running node
manta_node logs production

# Follow logs in real-time
manta_node logs production --follow

# View last 100 lines
manta_node logs production --lines 100

Best Practices¶

Production Deployment¶

Use Background Mode: Run nodes as background services
Configure Logging: Enable file logging, disable console
Set Resource Limits: Configure appropriate limits
Monitor Health: Check status regularly
Graceful Shutdown: Always use stop command

Development Setup¶

Use Foreground Mode: See output directly
Enable Debug Logging: Set level to DEBUG
Console Output: Enable log_to_console
Quick Iteration: Use Ctrl+C for fast stops

Multi-Node Deployment¶

For multiple nodes on one machine:

# Start nodes with different configs
manta_node start gpu-1 --alias worker-1 --background
manta_node start gpu-2 --alias worker-2 --background
manta_node start cpu-1 --alias worker-3 --background

# Or use cluster command
manta_node cluster start 3

Security Considerations¶

Protect JWT Tokens: Never expose in logs or commits
Use Environment Variables: For sensitive configuration
Verify TLS: Enable TLS in production
Limit Permissions: Run with minimal privileges
Network Security: Configure firewalls appropriately

Troubleshooting¶

Node Won’t Start¶

Check these common issues:

Configuration exists: manta_node config list
Valid JWT token: manta_node config validate <name>
Docker running: docker ps
Manager reachable: ping <manager-host>
Port available: lsof -i :<manager-port>

Node Starts but Disconnects¶

Verify:

Token not expired: Check JWT expiration
Network stable: Monitor connectivity
MQTT accessible: Check broker logs
Resources available: Check system resources

Performance Issues¶

Optimize node performance:

Increase resources: Adjust max_concurrent_tasks
Optimize logging: Reduce log level to WARNING
Enable GPU: Configure GPU if available
Tune network: Adjust MQTT settings
Cache images: Pre-pull Docker images