ezpz from anywhere: remote submission via the ALCF IRI API

Submit, monitor, extend, and cancel HPC jobs from anywhere — no SSH session required. A tour of the new ezpz IRI integration: IRI-first auto fallback, rich status tables, and live resource and incident views, all backed by the ALCF IRI API.

For years the ezpz answer to “which machine am I on?” was to figure it out at runtime. Launch on Aurora (PBS, Intel XPUs), Polaris (PBS, NVIDIA), Perlmutter (SLURM, NVIDIA), Frontier (SLURM, AMD), or a laptop — same command, no per-site mpiexec wrappers:

ezpz launch python3 train.py

That solved running. It did not solve submitting.

If you wanted a job on Aurora you still had to be on Aurora — an SSH session to a login node, an active allocation, a qsub typed in the right shell. From a laptop at a conference, from CI, from anywhere without your ALCF SSH keys and MFA token, you were stuck. And even on the login node, qstat is text you scrape, qdel is fire-and-forget, and “extend my walltime” is an incantation you google every time.

Today that changes. ezpz can now drive the ALCF IRI API directly — submit, list, inspect, extend, and cancel jobs as structured HTTP, with no SSH at all.

Why not just SSH?

SSH still works and still matters. It is the only path to machines the IRI API does not serve — Sunspot, Perlmutter, Frontier, Sophia — and it reuses whatever authentication you already have (ControlMaster, agent, MFA). For those hosts, ezpz still opens one connection, streams your script over stdin, and runs qsub or sbatch on the far side:

ezpz submit --remote aurora -N 128 -q prod -A AuroraGPT \
    --time 12:00:00 --workdir /lus/flare/projects/AuroraGPT/me/run \
    -- python3 -m my.module
# 8936286.aurora-pbs-0001  (via ssh)

One connection, not two — the script is delivered with cat > over the same channel that runs the submit command, so an MFA gate is paid once.

But SSH has real costs. It needs a live login-node session, it is gated by MFA on every new connection, and its answers are still text:

qstat 8936286  # parse this yourself

The IRI API is the complement: a REST surface that returns JSON, needs no interactive session, and works from anywhere you hold a token. It covers Aurora, Polaris, and Crux — the ALCF compute resources with published IRI endpoints — and it is what lets ezpz behave like a proper remote control rather than a screen-scraper.

Getting a token

One login, from an ALCF UAN:

# ephemeral, no install
uvx alcf-tokens login iri

# or persistent
uv tool install alcf-tokens
alcf-tokens login iri

ezpz finds the token the same way every time: uvx alcf-tokens get-token iri, falling back to alcf-tokens get-token iri, then to ALCF_IRI_TOKEN in the environment. If you are scripting in CI, export the token and skip the tool entirely:

export ALCF_IRI_TOKEN="eyJ..."
ezpz submit --remote aurora --backend iri -N 2 -q debug -A datascience -- python3 hello.py

No Globus dance in user code, no manual header construction. The plumbing lives in ezpz/remote.py::_iri_token().

IRI-first auto fallback

--backend takes three values: ssh, iri, and auto (the default). The design question is what auto should do when one path fails.

The first version tried SSH first and fell back to IRI only on exit code 255 — SSH’s own “I could not open the session” status. Every other non-zero code is the remote command’s exit, and retrying those is actively harmful:

A scheduler rejection retried on another backend either reproduces the rejection with a worse error, or — if the first submission landed before the non-zero exit — submits the job twice. Wrong queue, wrong project, and per-user queue limits all surface as rc=1; each deserves the scheduler’s own text, not a silent retry.

As of this week, auto is IRI-first on IRI-capable hosts:

aurora / polaris / crux  →  try IRI, fall back to SSH on auth/network failure
everything else          →  straight to SSH

That makes the fast, structured path the default where it exists, while keeping the 255-only guard for the SSH leg. The backend that ran is always reported:

$ ezpz submit --remote aurora -N 2 -q debug -A datascience -- python3 hello.py
8936286.aurora-pbs-0001  (via iri)

$ ezpz submit --remote aurora --backend ssh -N 2 -q debug -A datascience -- python3 hello.py
8936287.aurora-pbs-0001  (via ssh)

Force one with --backend iri or --backend ssh when you do not want the automatic behaviour. --dry-run still works and still needs no connection at all — it prints the exact script that would be submitted:

ezpz submit --remote aurora --dry-run -N 128 -q prod -A AuroraGPT -- python3 train.py
# # would submit to aurora (scheduler=PBS, backend=auto)
# #!/bin/bash
# #PBS -l select=128:system=polaris
# ...

Status, stdout, and cancel without qstat

Three endpoints, three commands, verified against the live OpenAPI spec at /openapi.json:

• GET /api/v1/compute/status/{rid}/{job_id}?historical=true — one job
• POST /api/v1/compute/status/{rid}?historical=true&limit=20 — a filtered list
• DELETE /api/v1/compute/cancel/{rid}/{job_id} — cancel (204 on success)

In ezpz they are ezpz status and ezpz cancel:

# one job (IRI-first on Aurora, SSH fallback on failure)
ezpz status 8936286 --remote aurora
# 8936286  completed  queue=prod  exit=0  (via iri)

# your recent jobs
ezpz status --remote aurora --limit 10

# only what is queued or running
ezpz status --remote aurora --states queued,active --limit 20

# filter by queue or owner
ezpz status --remote aurora --queue prod --owner foreman

# machine-readable
ezpz status 8936286 --remote aurora --json | jq .

# cancel (same auto/iri/ssh rule, dry-run supported)
ezpz cancel 8936286 --remote aurora --dry-run
ezpz cancel 8936286 --remote aurora
# cancelled 8936286 (via iri)

The same job ID carries through submit, status, and fetch:

$ ezpz submit --remote aurora -N 2 -q debug -A datascience -- python3 hello.py
8936286.aurora-pbs-0001  (via iri)

$ ezpz status 8936286 --remote aurora
8936286  completed  queue=debug  exit=0  (via iri)

$ ezpz fetch 8936286 --remote aurora --lines 80 --no-dedupe
# prints one snapshot of the job's last 80 stdout lines

ezpz status reports scheduler state; ezpz fetch reads a one-time stdout snapshot for a job. fetch gets stdout_path from IRI when available, otherwise from PBS qstat -xf. It reads the file over noninteractive SSH; it does not follow new output or discover Slurm spool paths. See the ezpz fetch reference for its stream limits.

Without --remote, status and cancel use the local scheduler — qstat / qdel or squeue / scancel on SLURM. fetch uses local PBS metadata or .OU files in the current directory. The remote flags are additive; omitting them keeps the local behavior.

The list endpoint honours the same filters the API does: --queue, --owner, --states, --limit, and --historical / --no-historical. A 30-second timeout with a clean message replaces the old raw TimeoutError traceback — the I can tell you is to narrow it:

ezpz status --remote aurora --states queued --limit 5

Rich tables (and the plain-text fallback)

When stdout is a TTY, ezpz status renders a Rich table with square borders, a neutral header rule, and colors by host and state. This static example snapshot keeps the same table structure available to screen readers; it is not a live status query. The table uses the site’s standard text size.

Host leads the fan-out (ezpz status with no --remote), short numeric JOBID is the primary key, WALL/ELAP include hh:mm, JOBNAME is the full name, and NODES is the node count. Per-job notes and running-job node mappings are shown below the table so the compact status columns stay scannable.

The colour map is small and deliberate — green for active/running, yellow for queued, magenta for held, cyan for completed, red for failed/ canceled, dim for new — so a scan of the jobs tells you which ones need attention without reading every cell.

When stdout is piped, the same columns come out as fixed-width plain text with a header row, so | grep, | awk, and | jq keep working. Pass --json for the structured form in either mode (now with nodes/walltime/elapsed/ note/name/exec_hosts — the fields IRI omits).

ezpz resources and ezpz incidents follow the same rule — Rich table on a TTY, fixed-width columns otherwise, --json on demand.

Extend, resources, incidents

Three more commands shipped in the same pass:

ezpz extend — fix a queued job without resubmitting

# add walltime
ezpz extend 8936286 --remote aurora --time 06:00:00

# move queues and fix the project in one call
ezpz extend 8936286 --remote aurora --queue prod --account AuroraGPT --time 12:00:00

# see exactly what would be sent
ezpz extend 8936286 --remote aurora --time 06:00:00 --dry-run
# would PUT /api/v1/compute/job/{rid}/8936286 {duration:21600, queue:prod, ...}

Under the hood this is PUT /api/v1/compute/job/{rid}/{jid} — the JobSpec-Input update path — IRI-first with SSH fallback to qalter (PBS) or scontrol update (SLURM). Like submit, it refuses to “update nothing” and validates duration_seconds >= 1 before a packet is sent. update-job is an alias.

ezpz resources — what exists and whether it is up

ezpz resources
# Resource   ID         Status
# aurora     55c1c99…   up
# polaris    55c1c99…   up
# crux       a1b2c3d…   up
# ...

ezpz resources --json | jq '.[] | {name, status}'

Backed by GET /api/v1/status/resources?limit=100 (the default), which is the endpoint that actually returns data today — /api/v1/compute/resources is a 501 and is kept behind --no-all for forward-compatibility. The runtime resource-id resolver (GET /status/resources matched by name) is what lets submit --backend iri work for Aurora and Crux even though only Polaris’s resource UUID is published in the ALCF docs. Hard-coding an id we cannot verify would risk submitting to the wrong machine.

ezpz incidents — maintenance and outages

ezpz incidents --resource aurora --limit 10
ezpz incidents --status down --json | jq .

GET /api/v1/status/incidents?limit=20&resource_id=...&status=.... Public endpoint, no auth required — ezpz sends the token if it has one and omits it if not. Useful before a launch: if Polaris is degraded and your reservation is there, queue somewhere else.

What actually shipped

For the changelog-minded, in commit order:

• 05e156b → 0bdc270 — ezpz status / ezpz cancel via IRI (GET single, POST list, DELETE cancel) plus SSH fallbacks (qstat, qdel/scancel). New library surface: RemoteJobStatus, get_job_status_via_iri, list_jobs_via_iri, cancel_via_iri/cancel_via_ssh, cancel_remote.

• c966a07 — payload schema fix per the ALCF docs: filesystems at attributes.custom_attributes.filesystems (not top-level), stdout_path / stderr_path as mandatory top-level fields, seconds not HH:MM:SS.

• a324f7f — auto flips to IRI-first on aurora/polaris/crux, SSH otherwise; clean timeout messages instead of raw tracebacks.

• a87a828 — ezpz extend (PUT /compute/job/{rid}/{jid}), ezpz resources (GET /status/resources), ezpz incidents (GET /status/incidents). Dry-run for extend, Rich + plain-text tables for resources/incidents/status, --json everywhere.

Docs at docs/guides/remote-submission.md and docs/cli/submit.md were updated in the same changes — the working-directory rule (local cwd is never carried across), the environment-setup story (remote scripts resolve utils.sh on the target at runtime, not from the submitter’s path), and the auto fallback table.

A few things the docs bury

Walltime is seconds on the wire. Every CLI flag takes HH:MM:SS, but the IRI API takes seconds — ezpz/_walltime_seconds does the conversion and the --time 06:00:00 on extend becomes duration: 21600 in the PUT body.

The remote workdir is shared-FS and job-unique. Omitting --workdir now creates the local os.getcwd(), which does not exist on the target. Now it creates $PBS_O_WORKDIR/ezpz-$PBS_JOBID ($SLURM_SUBMIT_DIR/$HOME fallback) on shared FS — not /tmp (node-local, would shard output).

Environment setup is self-bootstrapping. Every remote script now proxies, cds, then:

export http_proxy="http://proxy.alcf.anl.gov:3128"  # + https/no_proxy
source <(curl -fsSL https://ezpz.cool/utils.sh) && ezpz_setup_env
uv pip install --no-cache --link-mode=copy "git+https://github.com/saforem2/ezpz"  # if missing

Overridable with --env if you have site-specific setup to inject.

Resource ids are not all published. Only Polaris’s UUID appears in the ALCF docs. Rather than guess, ezpz looks up the full resource list at submit time and matches by name. If that lookup fails, you get the list of names it saw — not a silent mis-route.

Update — 2026-11-03: what’s new on ezpz.cool

Since this post went up, ezpz (v0.34.3 on main) has added a handful of small, well-documented pieces that round out the remote workflow above. Every item below links to its page on ezpz.cool — those pages are the canonical docs, not this post.

• ezpz status --watch — poll the IRI status table on a configurable interval with desktop notifications on Q→R / R→F transitions. rich.live.Live + Ctrl-C-clean teardown. Docs: CLI → status · PR #289.

• ezpz fetch — snapshot the last N lines of a job’s stdout without opening an interactive session. IRI metadata for stdout_path with a PBS qstat -xf fallback; tail over SSH for the bytes. Docs: CLI → fetch · PR #282.

• ezpz logs — per-example benchmark logs (outputs/ezpz-benchmarks/*/EXAMPLE.log) without the MPI-duplicated spool. Requires --remote HOST + JOB_ID; picks the last-failed example by default, or --example <name> explicitly. Docs: CLI → logs · added in bc50ab5.

• ezpz completion — shell completions for bash/zsh/fish via Click’s click.shell_completion (no shtab). ezpz completion bash prints the script; --install where supported. Docs: CLI → completion · added in bc50ab5.

• CLI index — all 19 commands — the CLI overview now lists every ezpz <cmd> with one-line summaries and links. Missing pages (logs, completion) were created and the nav was wired up. Docs: CLI index · bc50ab5 + 66e8dd4.

• Status parity with qmert — NODES / WALL / ELAP / NOTE columns plus running-job nodelists, matching qmert’s width-aware truncation. Fan-out ezpz status across machines and positional sugar documented alongside. Docs: CLI → status · f064ba7 + 14cd882.

• PBS exit-status propagation — the submission wrapper now preserves PIPESTATUS so a failed ezpz launch is no longer masked as success. Noted in Troubleshooting · 0d9a786 + PR #288.

• Profiler example — ezpz.examples.profiler, a minimal torch.profiler example with per-cycle traces and local-rank-gated logging. Docs: Examples → profiler · 1ce1346 + aed6521.

• Remote-workdir & env contract — submit now creates $PBS_O_WORKDIR/ezpz-$PBS_JOBID on shared FS (not /tmp), and --env / --env-setup-script with --workdir / --env tarball support. Full write-up in Remote submission · 9717c9a + 0846f1c.

Previous items from this post remain accurate — fetch and logs complement status --watch, while completion sits alongside the existing resources / incidents / extend surface rather than replacing it.

What’s next

Two small things, both already scoped:

• Richer completions. Queue and project names come from the resources endpoint already; exposing them as dynamic shell completions for --queue/-q and --account/-A is a short step on top of the now-shipped ezpz completion (bash/zsh/fish).

• Watch is now shipped. ezpz status --watch (see above) covers the rich.live.Live poll + transition log sketched in the original post: ezpz status --watch · PR #289.

Longer-term, the incident feed wants to be surfaced in ezpz doctor (“Polaris is degraded, your next submit targets Polaris — queue on Aurora instead?”), and the job list wants a --mine that fills --owner from the token’s identity so you do not have to remember your ALCF username in every invocation.

For now, the thing that matters is simple: from a laptop, from CI, from anywhere with a token, ezpz submit just works.

uvx alcf-tokens login iri
ezpz submit --remote aurora -N 2 -q debug -A datascience -- python3 hello.py
ezpz status --remote aurora --limit 5
ezpz status --remote aurora --watch        # new: live poll + notify
ezpz fetch 8936286 --remote aurora --lines 80 --no-dedupe  # new
ezpz logs 12481065 --remote sunspot --example fsdp          # new
ezpz extend 8936286 --remote aurora --time 06:00:00
ezpz cancel 8936286 --remote aurora
ezpz resources
ezpz incidents --resource aurora
ezpz completion bash                        # new: shell completions

All of this is in ezpz v0.34.3 on main (211c6e7, 54c069f — v0.34.3). Earlier IRI surface: v0.32 at a87a828, a324f7f, 0bdc270. Docs: Remote submission · CLI index · status · fetch · logs · completion · profiler example. API: ALCF IRI.

This research used resources of the Argonne Leadership Computing Facility, a U.S. Department of Energy Office of Science user facility, under Contract DE-AC02-06CH11357.