lets try running on different gpu machines

EVALUATION_SCALING.md

@@ -48,19 +48,20 @@ When enabled, the queue worker **does not** call `run_evaluation` inside the Spa
 | `FFASR_REMOTE_JOBS` | `1` / `true` to enable Hub Job dispatch. |
 | `FFASR_REMOTE_EVAL_REPO_URL` | Git URL cloned inside the job (required unless `FFASR_REMOTE_JOB_COMMAND` is set). |
 | `FFASR_REMOTE_EVAL_GIT_BRANCH` | Branch for clone (default `main`). |
-| `FFASR_REMOTE_JOB_IMAGE` | Default Docker image if no `remote_job_images.json` match (default `python:3.10-bookworm`). |
-| `FFASR_REMOTE_JOB_FLAVOR` | Hub Job hardware flavor (default `cpu-basic`). |
+| `FFASR_REMOTE_JOB_IMAGE` | Default Docker image if no `remote_job_images.json` match (default CUDA PyTorch: `pytorch/pytorch:2.6.0-cuda12.4-cudnn9-devel`). |
+| `FFASR_REMOTE_JOB_FLAVOR` | Hub Job hardware flavor (default `l4x1` — NVIDIA L4, 1× GPU per [Hub Jobs](https://huggingface.co/docs/huggingface_hub/guides/jobs)). |
 | `FFASR_REMOTE_JOB_NAMESPACE` | Optional Hub namespace for the job. |
 | `FFASR_REMOTE_JOB_TIMEOUT` | Job timeout (Hub format). |
 | `FFASR_REMOTE_JOB_MAX_WAIT_S` | Max seconds to poll for terminal state (default `14400`). |
 | `FFASR_REMOTE_JOB_POLL_S` | Poll interval seconds (default `10`). |
 | `FFASR_REMOTE_JOB_COMMAND` | Optional JSON array of strings: full command override for `run_job`. |
-| `FFASR_REMOTE_WORKER_DEVICE` | Passed into the job as `FFASR_DEVICE` (default `cpu`). |
+| `FFASR_REMOTE_WORKER_DEVICE` | Passed into the job as `FFASR_DEVICE` (default `auto`: CUDA in GPU jobs, else CPU). |
 | `FFASR_REMOTE_WORKER_DISABLE_ZEROGPU` | Passed as `FFASR_DISABLE_ZEROGPU` (default `1`). |
+| `FFASR_REMOTE_SKIP_OPTIONAL_PIP` | If `1` / `true`, the remote worker skips allowlisted `pip install` for optional stacks (NeMo, Qwen ASR). |
 
 **Per-model Docker images**: copy [remote_job_images.example.json](remote_job_images.example.json) to `remote_job_images.json` in the repo root. Longest matching prefix key wins; use `"default"` as fallback.
 
-**Job entrypoint**: [scripts/run_hf_remote_job.py](scripts/run_hf_remote_job.py) runs `run_evaluation`, builds the artifact, and uploads it with `HF_TOKEN` (passed as a Hub Job secret).
+**Job entrypoint**: [scripts/run_hf_remote_job.py](scripts/run_hf_remote_job.py) optionally installs missing **allowlisted** dependencies (e.g. `nemo_toolkit[asr]` for Parakeet-style ids, `qwen-asr` for Qwen ASR ids) when imports fail, then runs `run_evaluation`, builds the artifact, and uploads it with `HF_TOKEN` (passed as a Hub Job secret).
 
 ---
 
@@ -90,6 +91,6 @@ Do not execute **untrusted** `pip install` lines from users on shared infrastruc
 
 ## Suggested evolution path
 
-1. **Now**: FIFO queue + CSV lock + ZeroGPU **or** fixed CPU (`FFASR_DEVICE` / `FFASR_DISABLE_ZEROGPU`) **or** Hub Jobs (`FFASR_REMOTE_JOBS=1`).
+1. **Now**: FIFO queue + CSV lock + ZeroGPU **or** fixed CPU/CUDA in-process (`FFASR_DEVICE` / `FFASR_DISABLE_ZEROGPU`) **or** Hub Jobs on GPU (`FFASR_REMOTE_JOBS=1`, default flavor `l4x1` + CUDA image).
 2. **Next**: richer job metadata in CSV (remote ids, artifact paths); optional webhooks when remote workers finish.
 3. **Later**: split UI and workers entirely (multiple runners, priority tiers).

remote_job_images.example.json

@@ -1,5 +1,5 @@
 {
-  "default": "python:3.10-bookworm",
+  "default": "pytorch/pytorch:2.6.0-cuda12.4-cudnn9-devel",
   "coherelabs/": "your-registry/cohere-asr-eval:pin",
   "ibm-granite/": "your-registry/granite-speech-eval:pin"
 }

remote_jobs.py

@@ -7,8 +7,8 @@ Environment (Space / operator):
 
 - ``FFASR_REMOTE_JOBS`` — ``1`` / ``true`` to route queue worker through Hub Jobs.
 - ``HF_TOKEN`` — token with permission to run jobs and read/write the results bucket.
-- ``FFASR_REMOTE_JOB_IMAGE`` — Docker image for the job (default ``python:3.10-bookworm``).
-- ``FFASR_REMOTE_JOB_FLAVOR`` — hardware flavor (default ``cpu-basic``). See ``HfApi.list_jobs_hardware``.
+- ``FFASR_REMOTE_JOB_IMAGE`` — Docker image for the job (default CUDA PyTorch image for GPU eval).
+- ``FFASR_REMOTE_JOB_FLAVOR`` — hardware flavor (default ``l4x1``). See ``HfApi.list_jobs_hardware``.
 - ``FFASR_REMOTE_JOB_NAMESPACE`` — optional Hub namespace/user for the job (default: token owner).
 - ``FFASR_REMOTE_JOB_TIMEOUT`` — job timeout string/int (optional).
 - ``FFASR_REMOTE_EVAL_REPO_URL`` — git clone URL for this repo inside the job (required unless
@@ -17,7 +17,7 @@ Environment (Space / operator):
 
 Per-model Docker image: optional JSON file ``remote_job_images.json`` in the repo root::
 
-    {"cohere/": "org/cohere-asr-image:tag", "default": "python:3.10-bookworm"}
+    {"cohere/": "org/cohere-asr-image:tag", "default": "pytorch/pytorch:2.6.0-cuda12.4-cudnn9-devel"}
 
 Longest key match wins; ``default`` fallback.
 """
@@ -32,6 +32,11 @@ from typing import Any
 from huggingface_hub import HfApi
 from huggingface_hub._jobs_api import JobInfo, JobStage
 
+# Defaults for L4 GPU Hub Jobs (override with FFASR_REMOTE_JOB_* / remote_job_images.json).
+_DEFAULT_REMOTE_JOB_IMAGE = "pytorch/pytorch:2.6.0-cuda12.4-cudnn9-devel"
+_DEFAULT_REMOTE_JOB_FLAVOR = "l4x1"
+_DEFAULT_REMOTE_WORKER_DEVICE = "auto"
+
 _IMAGE_MAP_CACHE: dict[str, str] | None = None
 
 
@@ -85,7 +90,7 @@ def resolve_job_image(model_id: str) -> str:
     if default_map:
         return default_map
     env_img = os.environ.get("FFASR_REMOTE_JOB_IMAGE", "").strip()
-    return env_img or "python:3.10-bookworm"
+    return env_img or _DEFAULT_REMOTE_JOB_IMAGE
 
 
 def _default_bash_command() -> list[str]:
@@ -99,7 +104,7 @@ def _default_bash_command() -> list[str]:
         "cd \"$WORKDIR\" && "
         "python -m pip install --no-cache-dir -U pip && "
         "python -m pip install --no-cache-dir -r requirements.txt && "
-        "export FFASR_DEVICE=${FFASR_DEVICE:-cpu} FFASR_DISABLE_ZEROGPU=${FFASR_DISABLE_ZEROGPU:-1} && "
+        "export FFASR_DEVICE=${FFASR_DEVICE:-auto} FFASR_DISABLE_ZEROGPU=${FFASR_DISABLE_ZEROGPU:-1} && "
         "python scripts/run_hf_remote_job.py"
     )
     return ["bash", "-lc", script]
@@ -130,7 +135,10 @@ def submit_eval_job(
     api = HfApi(token=token)
     image = resolve_job_image(model_id)
     namespace = os.environ.get("FFASR_REMOTE_JOB_NAMESPACE", "").strip() or None
-    flavor = os.environ.get("FFASR_REMOTE_JOB_FLAVOR", "cpu-basic").strip() or "cpu-basic"
+    flavor = (
+        os.environ.get("FFASR_REMOTE_JOB_FLAVOR", _DEFAULT_REMOTE_JOB_FLAVOR).strip()
+        or _DEFAULT_REMOTE_JOB_FLAVOR
+    )
     timeout_raw = os.environ.get("FFASR_REMOTE_JOB_TIMEOUT", "").strip()
     timeout: int | float | str | None
     if not timeout_raw:
@@ -164,7 +172,10 @@ def submit_eval_job(
         "FFASR_SPACE_JOB_ID": space_job_id,
         "FFASR_REMOTE_ARTIFACT_PATH": artifact_path,
         "FFASR_BUCKET_ID": bucket,
-        "FFASR_DEVICE": os.environ.get("FFASR_REMOTE_WORKER_DEVICE", "cpu"),
+        "FFASR_DEVICE": os.environ.get(
+            "FFASR_REMOTE_WORKER_DEVICE", _DEFAULT_REMOTE_WORKER_DEVICE
+        ).strip()
+        or _DEFAULT_REMOTE_WORKER_DEVICE,
         "FFASR_DISABLE_ZEROGPU": os.environ.get("FFASR_REMOTE_WORKER_DISABLE_ZEROGPU", "1"),
         "FFASR_REMOTE_EVAL_REPO_URL": eval_repo,
         "FFASR_REMOTE_EVAL_GIT_BRANCH": eval_branch,

scripts/run_hf_remote_job.py

@@ -9,24 +9,107 @@ Environment (set by the Space when dispatching ``remote_jobs.submit_eval_job``):
 - ``FFASR_BUCKET_ID``, ``HF_TOKEN`` — bucket write access.
 - ``FFASR_SPACE_JOB_ID`` — optional correlating id from the Space queue.
 
-Also respects ``FFASR_DEVICE`` / ``FFASR_DISABLE_ZEROGPU`` (defaults should be CPU + no ZeroGPU).
+Also respects ``FFASR_DEVICE`` / ``FFASR_DISABLE_ZEROGPU`` (GPU Hub Jobs: ``auto`` uses CUDA when available).
+
+Optional preflight (before ``run_evaluation``):
+
+- ``FFASR_REMOTE_SKIP_OPTIONAL_PIP`` — if ``1``/``true``, skip targeted ``pip install`` for NeMo / Qwen ASR.
 
 Repo root must be the current working directory (after git clone in the job).
 """
 
 from __future__ import annotations
 
+import importlib
 import io
 import json
 import os
+import subprocess
 import sys
-import traceback
+from collections.abc import Callable
 
 ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
 if ROOT not in sys.path:
     sys.path.insert(0, ROOT)
 
 
+def _skip_optional_pip() -> bool:
+    return os.environ.get("FFASR_REMOTE_SKIP_OPTIONAL_PIP", "").strip().lower() in (
+        "1",
+        "true",
+        "yes",
+        "on",
+    )
+
+
+def _needs_nemo_stack(model_id: str) -> bool:
+    m = model_id.lower()
+    return "nvidia/parakeet" in m or "parakeet-tdt" in m
+
+
+def _needs_qwen_asr_stack(model_id: str) -> bool:
+    m = model_id.lower()
+    return m.startswith("qwen/") or "qwen3-asr" in m or "qwen3_asr" in m
+
+
+def _pip_install(packages: list[str], log: Callable[[str], None]) -> None:
+    cmd = [sys.executable, "-m", "pip", "install", "--no-cache-dir", *packages]
+    log(f"run_hf_remote_job: running {' '.join(cmd)}")
+    proc = subprocess.run(cmd, capture_output=True, text=True, timeout=3600)
+    if proc.returncode != 0:
+        out = (proc.stdout or "") + (proc.stderr or "")
+        raise RuntimeError(
+            f"pip install failed (exit {proc.returncode}): {out[-4000:]}"
+        )
+    if proc.stdout:
+        log(proc.stdout.strip()[-2000:])
+    if proc.stderr and proc.stderr.strip():
+        log(proc.stderr.strip()[-2000:])
+
+
+def _try_import(module: str) -> bool:
+    try:
+        importlib.import_module(module)
+        return True
+    except Exception:
+        return False
+
+
+def maybe_install_optional_eval_deps(
+    model_id: str, family_id: str, log: Callable[[str], None]
+) -> None:
+    """
+    Install known optional stacks only when the model/family likely needs them
+    and imports fail. Allowlisted packages only.
+    """
+    if _skip_optional_pip():
+        log("run_hf_remote_job: FFASR_REMOTE_SKIP_OPTIONAL_PIP set — skipping optional pip installs")
+        return
+
+    fid = (family_id or "auto").strip().lower()
+
+    want_nemo = _needs_nemo_stack(model_id) or fid == "nemo_asr"
+    want_qwen = _needs_qwen_asr_stack(model_id) or fid in ("qwen_asr", "qwen3_asr")
+
+    if want_nemo:
+        if _try_import("nemo.collections.asr"):
+            log("run_hf_remote_job: NeMo ASR import OK")
+        else:
+            log("run_hf_remote_job: NeMo not importable — installing nemo_toolkit[asr]")
+            _pip_install(["nemo_toolkit[asr]"], log)
+            if not _try_import("nemo.collections.asr"):
+                raise RuntimeError("NeMo install finished but nemo.collections.asr still not importable")
+
+    if want_qwen:
+        if _try_import("qwen_asr"):
+            log("run_hf_remote_job: qwen_asr import OK")
+        else:
+            log("run_hf_remote_job: qwen_asr not importable — installing qwen-asr")
+            _pip_install(["qwen-asr"], log)
+            if not _try_import("qwen_asr"):
+                raise RuntimeError("qwen-asr install finished but qwen_asr still not importable")
+
+
 def main() -> int:
     from evaluation.remote_artifact import ARTIFACT_SCHEMA_VERSION, build_artifact
     from evaluation.runtime import apply_cpu_thread_settings_once
@@ -58,6 +141,7 @@ def main() -> int:
 
     try:
         _log(f"run_hf_remote_job: model_id={model_id!r} family_id={family_id!r}")
+        maybe_install_optional_eval_deps(model_id, family_id, _log)
         result = run_evaluation(model_id, family_id=family_id, progress_cb=None)
         artifact = build_artifact(
             model_id=model_id,