Zyra Workflow Stages

This page outlines Zyra’s eight workflow stages with preferred names, common aliases, and how they map to the current CLI and docs. Use this as a north star when designing pipelines and documentation.

Pipeline at a Glance

  • import β†’ process β†’ simulate β†’ decide β†’ visualize β†’ narrate β†’ verify β†’ export

Stages

1. Import (aliases: acquire, ingest)

  • Purpose: Fetch bytes and lists from remote/local sources (HTTP/S, S3, FTP, Vimeo, filesystem).

  • CLI today: zyra acquire http|s3|ftp|vimeo with list/filter options and stdin/stdout support.

  • Docs: https://noaa-gsl.github.io/zyra/api/zyra.connectors.html

2. Process (alias: transform)

  • Purpose: Decode, subset, convert scientific data (GRIB2, NetCDF, GeoTIFF) and reshape for analysis/visuals.

  • CLI today: zyra process decode-grib2 | extract-variable | convert-format

  • Related helper group: zyra transform (lightweight metadata utilities: frames metadata, dataset JSON updates).

  • Docs: https://noaa-gsl.github.io/zyra/api/zyra.processing.html

3. Simulate

  • Purpose: Synthesize or emulate data and scenarios (e.g., toy datasets, downsampling, mock streams) for testing and demos.

  • CLI today: Not yet a dedicated group; use notebooks/scripts or processing helpers. Planned as a future CLI group.

  • Related: Pipeline Patterns, CLI Expansion Plan.

4. Decide (alias: optimize)

  • Purpose: Score, rank, and select best variants (e.g., parameter sweeps, tiling choices, colormaps) and branch pipelines accordingly.

  • CLI today: Not yet a dedicated group; patterns can be expressed via configs or orchestrators (e.g., zyra run in the future).

  • Related: Wizard Assistant plans; Pipeline Patterns.

5. Visualize (alias: render)

  • Purpose: Produce static images, animations, and interactive HTML maps/plots.

  • CLI today: zyra visualize heatmap|contour|timeseries|vector [deprecated: wind] | animate | compose-video | interactive

  • Docs: https://noaa-gsl.github.io/zyra/api/zyra.visualization.html

6. Narrate

  • Purpose: Generate captions, reports, or web pages that contextualize outputs (text + media + metadata).

  • CLI today: Not yet a dedicated group; use templates and external tools. Planned integration alongside logging/metadata.

  • Related: Wizard Interactive Assistant; Logging in Zyra.

7. Verify

  • Purpose: Validate integrity and quality (checksums, schema validation, visual/quantitative metrics, provenance).

  • CLI today: Not yet a dedicated group; hooks via processing/transform and external validators. Planned as a future CLI group.

  • Related: Logging in Zyra.

8. Export (alias: disseminate; legacy: decimate)

  • Purpose: Move results to destinations (local path, S3, FTP, HTTP POST, Vimeo) with streaming support.

  • CLI today: zyra export local|s3|ftp|post|vimeo (stdin-friendly with --read-stdin).

  • Docs: https://noaa-gsl.github.io/zyra/api/zyra.connectors.html

Notes

  • Aliases are supported for familiarity; prefer the primary names in docs/UI.

  • Some stages (simulate, decide, narrate, verify) are conceptual today and targeted for future CLI support. See Roadmap-and-Tracking.md.

  • For streaming pipelines, many subcommands accept - for stdin/stdout.

Status & Roadmap

  • Import: Implemented via acquire with HTTP/S, S3, FTP, Vimeo.

    • Next: Unified manifest handling and profile-based discovery.

  • Process: Implemented via process (decode/extract/convert) and transform helpers.

    • Next: More subsetting ops (spatial/temporal), streaming NetCDF transforms.

  • Simulate: Conceptual; use notebooks/scripts for now.

    • Next: Add simulate CLI group for toy datasets and mock streams.

  • Decide: Conceptual; patterns via configs/overrides today.

    • Next: Add decide/optimize group for parameter sweeps and picking best artifacts.

  • Visualize: Implemented (static/animated/interactive).

    • Next: Expand interactive commands; add charting parity across data types.

  • Narrate: Conceptual; rely on templates and external tools today.

    • Next: Add narrate group for captions/reports with metadata bindings.

  • Verify: Conceptual; use logging/metadata and external validators today.

    • Next: Add verify group for checksums, schema validation, and basic metrics.

  • Export: Implemented via export (alias: disseminate, legacy: decimate).

    • Next: Add resumable uploads and richer auth profiles.