Goal

Expose the full 4-stage Zyra CLI as a FastAPI web service with:

  • Sync and async execution modes.

  • File uploads and result downloads.

  • Real-time progress streaming via WebSockets.

  • Full logging and exit code capture for each command (for debugging & workflow tools like n8n).

  • Automatic startup in the development container.

1. Project Structure

src/zyra/api/
    __init__.py
    server.py           # FastAPI app entrypoint
    routers/
        cli.py          # Endpoints for CLI execution
        jobs.py         # Endpoints for async job management
        files.py        # Endpoints for uploads/downloads
    workers/
        executor.py     # Runs CLI commands safely
    models/
        cli_request.py  # Pydantic models for request/response

2. FastAPI Endpoints

2.1 Run CLI Command (Sync or Async)

POST /cli/run
Request:

{
  "stage": "process",
  "command": "decode-grib2",
  "args": { "input": "s3://bucket/file.grib2", "output": "-" },
  "mode": "sync"
}

Response (sync):

{
  "status": "success",
  "stdout": "...",
  "stderr": "",
  "exit_code": 0
}

Response (async):

{
  "status": "accepted",
  "job_id": "abc123"
}

Enhancement: All responses now include exit_code (for sync runs) or will include it when async job is complete.

2.2 Upload Data

POST /upload (multipart form)

  • Accepts file upload, stores in /tmp/zyra_uploads/, returns file_id.

2.3 Download Results

GET /jobs/{job_id}/download

  • Returns output file from completed job.

2.4 WebSocket Progress

/ws/jobs/{job_id}

  • Streams logs/progress in real time.

2.5 Job Management

  • GET /jobs/{job_id} β†’ status, stdout, stderr, and exit_code.

  • DELETE /jobs/{job_id} β†’ cancel a job.

3. Internal Execution Model

3.1 Command Registry

CLI_REGISTRY = {
    "visualize": visualization.command_map
}  # Visualization-first; other stages to be added later

3.2 Synchronous Execution with Logging & Exit Code

from io import StringIO
import sys

def run_cli(stage, command, args):
    func = CLI_REGISTRY[stage][command]
    sys_stdout, sys_stderr = StringIO(), StringIO()
    sys_stdout_backup, sys_stderr_backup = sys.stdout, sys.stderr
    sys.stdout, sys.stderr = sys_stdout, sys_stderr
    exit_code = 0
    try:
        func(**args)
    except Exception as e:
        exit_code = 1
        print(str(e), file=sys.stderr)
    finally:
        sys.stdout, sys.stderr = sys_stdout_backup, sys_stderr_backup
    return {
        "stdout": sys_stdout.getvalue(),
        "stderr": sys_stderr.getvalue(),
        "exit_code": exit_code
    }

3.3 Asynchronous Execution

  • Use Celery or RQ with Redis.

  • Job record stores:

    • status

    • stdout

    • stderr

    • exit_code

    • output_file_path

4. Bonus Features Implementation

4.1 File Upload Handling

  • Save uploaded files to /tmp/zyra_uploads/ with UUID names.

  • Allow CLI args to reference file_id.

4.2 Result Download

  • Store output in /tmp/zyra_results/{job_id}.

  • Direct file download endpoint.

4.3 WebSocket Streaming

  • Workers push logs and progress updates to Redis pub/sub.

  • WebSocket subscribers stream updates to clients.

Enhancement: WebSocket messages now include { "stdout": "...", "stderr": "...", "exit_code": 0, "progress": 0.5 }

5. Security

  • No raw shell exec β€” only mapped callables.

  • Pydantic validation on all requests.

  • API keys or OAuth2.

  • Rate limiting.

6. Deployment

6.1 Local Dev

uvicorn zyra.api.server:app --reload --host 0.0.0.0 --port 8000

6.2 Production

  • Gunicorn + Uvicorn workers.

  • HTTPS via Nginx or Caddy.

  • Redis for async jobs.

7. Dev Container Auto-Startup

7.1 Startup Script

scripts/start-api.sh:

#!/usr/bin/env bash
set -e
cd /app/zyra
uvicorn zyra.api.server:app --reload --host 0.0.0.0 --port 8000

Make executable:

chmod +x scripts/start-api.sh

7.2 Devcontainer Hook

.devcontainer/devcontainer.json:

{
  "name": "zyra-dev",
  "dockerComposeFile": [
    "../docker-compose.yml"
  ],
  "service": "zyra",
  "workspaceFolder": "/app",
  "overrideCommand": false,

  // Ensures VS Code starts the right services
  "runServices": ["zyra", "redis"],
  "composeProfiles": ["dev"],

  "customizations": {
    "vscode": {
      "settings": {},
      "extensions": [
        "ms-python.python",
        "ms-azuretools.vscode-docker"
      ]
    }
  },

  // Forward API & Redis ports
  "forwardPorts": [8000, 6379],

  // Inject secrets cleanly
  "runArgs": ["--env-file", ".env"]
}

7.3 Background Mode (optional)

"postStartCommand": "nohup ./scripts/start-api.sh > /workspace/zyra/api.log 2>&1 &"

8. Testing

  • Unit tests for endpoints and CLI execution mapping.

  • Integration tests for pipelines via API.

  • WebSocket tests for live progress.

  • Tests for correct exit_code, stdout, and stderr capture.

9. Documentation

  • API reference in /docs (FastAPI auto-docs).

  • Examples for:

    • Submitting sync/async jobs.

    • Uploading files and referencing them in CLI.

    • Downloading results.

    • Streaming progress with logs and exit codes.