Update to joint app for all classifiers

README.md

@@ -1,8 +1,8 @@
 ---
-title: ISAAC Relevance Filter
-emoji: 🔎
+title: ISAAC Text Classifiers
+emoji: 🧭
 colorFrom: blue
-colorTo: green
+colorTo: purple
 sdk: gradio
 sdk_version: 6.9.0
 python_version: "3.10"
@@ -10,51 +10,110 @@ app_file: app.py
 pinned: false
 ---
 
-# ISAAC Relevance Filter
+# ISAAC Text Classifiers
 
-A coding-free demo for the **relevance** classifiers from [the Illinois Social 
-Aggregate Corpus (ISAAC)](https://github.com/BabakHemmatian/Illinois_Social_Attitudes).
-Citation information can be found in the repository. Use is subject to [terms](https://github.com/BabakHemmatian/Illinois_Social_Attitudes/blob/main/Data_Use_Agreement.md).
+A single coding-free demo for all three classifier families from the Illinois
+Social Attitudes (ISAAC) project, consolidated into one Space:
 
-Each social distinction (ability, age, race, sexuality, skin tone, weight) has
-its own fine-tuned classifier; this single Space presents all of them through 
-a **Social Distinction** dropdown and loads the matching model on demand.
+- **Relevance** — is a text relevant to a given social distinction (ability, age,
+  race, sexuality, skin tone, weight)? Six fine-tuned classifiers behind a
+  distinction dropdown.
+- **Moralization** — does a text frame its subject in moral terms?
+- **Generalization** — clause segmentation plus per-clause discourse labels
+  (genericity, eventivity, boundedness/habituality).
 
-The Space has two tabs:
-
-- **Single text** — paste one text, pick a distinction, see the predicted label
-  and class confidence.
-- **Multiple texts (file)** — upload a `.txt` (one text per line) or `.csv`
-  (text in the first column) and download a labeled results CSV.
+Each task has a **Single text** tab and a **Multiple texts (file)** tab that
+accepts a `.txt` (one text per line) or `.csv` (text in the first column) and
+returns a labeled results CSV.
 
 For **race** and **skin tone**, a 0.6 confidence threshold is applied to the
 "relevant" class, matching the ISAAC pipeline (`filter_relevance.py`).
 
+## Why one Space
+
+Hugging Face limits free personal accounts to **2 ZeroGPU Spaces**, and Gradio
+Spaces now require ZeroGPU or a paid plan — CPU Basic is no longer available on
+the free tier. Consolidating the three demos into one app keeps all of them
+available within that limit, and gives the pre-print a single link to cite.
+
+## Layout
+
+| File | Contents |
+| --- | --- |
+| `app.py` | Thin composition layer: three top-level tabs, nothing else. |
+| `common.py` | Device/token resolution, model-path resolution, file I/O, GPU-duration helpers. |
+| `relevance.py` | Six relevance classifiers, thresholding, UI. |
+| `moralization.py` | Moralization classifier and UI. |
+| `generalization.py` | Segmenter + clause classifier, aggregation, pie charts, UI. |
+
+The task modules each own their UI via a `build_ui()` function, so `app.py` never
+has to know their internals. They previously lived in three separate Spaces with
+colliding module-level names (`_load`, `_iter_texts`, `classify_file`); the split
+into modules is what makes the merge safe.
+
+## Hardware
+
+The Space runs on **ZeroGPU**, which allocates a GPU only for the duration of a
+decorated call. ZeroGPU refuses to start a Space that registers no `@spaces.GPU`
+function. The decorated entry points are:
+
+| Function | Duration |
+| --- | --- |
+| `relevance.classify_text` | 60 s (default) |
+| `relevance.classify_file` | dynamic, from row count |
+| `moralization.classify_text` | 60 s (default) |
+| `moralization.classify_file` | dynamic, from row count |
+| `generalization._clause_labels` | dynamic, from word count |
+| `generalization.analyze_file` | dynamic, from row count |
+
+`spaces` is imported before `torch` — `common.py` does this once for every
+module. ZeroGPU charges each visitor's daily quota against the duration a call
+*requests*, not the time it actually uses, hence the dynamic durations rather
+than a flat worst case. Overrunning a request kills the call, so the estimates
+assume a cold model load every time.
+
+The generalization single-text path deliberately decorates only the model pass:
+`render_pipeline` builds the matplotlib figure afterwards, in the main process,
+from the plain `(clause, attributes)` data the GPU call returns. A decorated call
+runs in a separate process, and a `Figure` is a poor thing to ship across that
+boundary.
+
 ## Memory behavior
 
-Models are loaded lazily and cached (`functools.lru_cache`, capacity 2), so only
-the distinction(s) you use are pulled into RAM and at most two stay resident at
-once. Peak memory is a single roberta-large (~1.4 GB), well within the free CPU
-tier, even though the full set of six models totals ~7.6 GB.
+Every model is loaded lazily and cached, so only the task(s) actually used are
+pulled into memory. Relevance keeps at most two of its six distinction models
+resident (`functools.lru_cache`, capacity 2) — roughly 1.4 GB rather than the
+~7.6 GB the full set would need. Moralization and generalization each cache their
+own models at capacity 1.
 
 ## Models
 
-The app resolves each distinction's weights with this order (see `app.py`):
+Each model resolves local-first with a Hugging Face Hub fallback
+(`common.resolve_source`):
+
+1. `$ISAAC_MODELS_DIR/<relative path>` — explicit local override.
+2. `<repo_root>/models/<relative path>` — in-repo local run.
+3. The released Hugging Face model repo.
 
-1. `$ISAAC_MODELS_DIR/filter_relevance_<group>` — explicit local override.
-2. `<repo_root>/models/filter_relevance_<group>` — in-repo local run.
-3. The released Hugging Face model repos, named
-   `$ISAAC_RELEVANCE_REPO_PREFIX<group>`
-   (default prefix `BabakScrapes/isaac-relevance-`, e.g.
-   `BabakScrapes/isaac-relevance-race`).
+| Task | Local path under `models/` | Hub default | Env override |
+| --- | --- | --- | --- |
+| Relevance | `filter_relevance_<group>` | `BabakScrapes/isaac-relevance-<group>` | `ISAAC_RELEVANCE_REPO_PREFIX` |
+| Moralization | `label_moralization` | `BabakScrapes/isaac-moralization` | `ISAAC_MORALIZATION_REPO` |
+| Generalization (segmenter) | `label_generalization/label_generalization_segmentation` | `BabakScrapes/isaac-generalization-segmentation` | `ISAAC_GENERALIZATION_SEG_REPO` |
+| Generalization (classifier) | `label_generalization/label_generalization` | `BabakScrapes/isaac-generalization` | `ISAAC_GENERALIZATION_REPO` |
 
 The same code therefore runs locally against the project's `models/` directory
-and on Hugging Face against the published per-distinction model repos.
+and on Hugging Face against the published model repos.
+
+**If the model repos are private or gated**, the Space needs a read token: add it
+as a secret named `HF_TOKEN` under Settings → Variables and secrets. Without it
+the build still succeeds — models load lazily, so nothing contacts the Hub at
+startup — and the first classification fails with an `OSError` instead.
 
 ## Local testing
 
 ```bash
-cd hf_spaces/relevance
+cd hf_spaces/isaac
 pip install -r requirements.txt
 export ISAAC_MODELS_DIR=/path/to/Illinois_Social_Attitudes/models
 python app.py

app.py

@@ -1,264 +1,40 @@
 from __future__ import annotations
 
-# Must be imported before torch: on ZeroGPU hardware the `spaces` package
-# patches torch's CUDA layer at import time. ZeroGPU also refuses to start
-# unless at least one @spaces.GPU function is registered (see classify_text /
-# classify_file below). The decorator is a no-op off ZeroGPU.
-import spaces
-
-import csv
-import os
-import tempfile
-from functools import lru_cache
-from pathlib import Path
+# `common` must be imported first: it imports `spaces` before torch, which is
+# what ZeroGPU requires. The task modules below all import torch transitively.
+import common
 
 import gradio as gr
-import torch
-from transformers import AutoModelForSequenceClassification, AutoTokenizer
-
-# ---------------------------------------------------------------------------
-# One Space, six fine-tuned relevance classifiers (one per social distinction).
-#
-# Models are loaded lazily and cached: only the distinction(s) you actually use
-# are pulled into RAM, and at most CACHE_SIZE models stay resident at once. 
-#
-# Resolution order per distinction (local-first, Hub fallback):
-#   1. $ISAAC_MODELS_DIR/filter_relevance_<group>   (explicit local override)
-#   2. <repo_root>/models/filter_relevance_<group>  (in-repo local run)
-#   3. $ISAAC_RELEVANCE_REPO_PREFIX<group>           (released HF model repos)
-# ---------------------------------------------------------------------------
-
-GROUPS = ["ability", "age", "race", "sexuality", "skin_tone", "weight"]
-
-# Released model-repo naming: prefix + group, e.g. "BabakScrapes/isaac-relevance-race".
-HUB_PREFIX = os.environ.get("ISAAC_RELEVANCE_REPO_PREFIX", "BabakScrapes/isaac-relevance-")
-
-# Number of distinction models kept resident simultaneously.
-CACHE_SIZE = 2
-
-MAX_LENGTH = 512
-BATCH_SIZE = 16
-# On ZeroGPU a real GPU only exists inside @spaces.GPU functions; outside them
-# torch runs in CUDA emulation mode, so "cuda" is the correct target either way.
-# Off ZeroGPU (local runs, plain CPU/GPU Spaces) fall back to what's available.
-ON_ZERO_GPU = bool(os.environ.get("SPACES_ZERO_GPU"))
-DEVICE = torch.device(
-    "cuda" if ON_ZERO_GPU or torch.cuda.is_available() else "cpu"
-)
-# Class index -> human label. The pipeline keeps argmax==1 rows as relevant.
-LABELS = {0: "Not relevant", 1: "Relevant"}
-
-# Confidence thresholding (matches filter_relevance.py): for race and skin_tone
-# the model only assigns the rare "relevant" class when its probability clears
-# THRESHOLD; otherwise it falls back to the next-most-probable class.
-THRESHOLDED_GROUPS = {"race", "skin_tone"}
-THRESHOLD_CLASS = 1
-THRESHOLD = 0.6
-
-ALLOWED_EXTENSIONS = {".txt", ".csv"}
-
-
-def _resolve_source(group: str) -> str:
-    dirname = f"filter_relevance_{group}"
-
-    models_dir = os.environ.get("ISAAC_MODELS_DIR")
-    if models_dir:
-        candidate = Path(models_dir) / dirname
-        if candidate.exists():
-            return str(candidate)
-
-    sibling = Path(__file__).resolve().parents[2] / "models" / dirname
-    if sibling.exists():
-        return str(sibling)
-
-    return f"{HUB_PREFIX}{group}"
-
-
-@lru_cache(maxsize=CACHE_SIZE)
-def _load(group: str):
-    source = _resolve_source(group)
-    tokenizer = AutoTokenizer.from_pretrained(source)
-    model = AutoModelForSequenceClassification.from_pretrained(source).to(DEVICE)
-    model.eval()
-    return tokenizer, model
-
-
-def _decide(prob, group: str) -> int:
-    """Map a class-probability vector to a predicted class index, replicating
-    the thresholding behavior of filter_relevance.py.predict_tokenized."""
-    if group in THRESHOLDED_GROUPS:
-        if prob[THRESHOLD_CLASS] > THRESHOLD:
-            return THRESHOLD_CLASS
-        masked = prob.clone()
-        masked[THRESHOLD_CLASS] = -1.0
-        return int(masked.argmax())
-    return int(prob.argmax())
-
 
-@torch.no_grad()
-def _predict(texts, group):
-    tokenizer, model = _load(group)
-    results = []
-    for start in range(0, len(texts), BATCH_SIZE):
-        batch = texts[start : start + BATCH_SIZE]
-        enc = tokenizer(
-            batch,
-            padding=True,
-            truncation=True,
-            max_length=MAX_LENGTH,
-            return_tensors="pt",
-        ).to(DEVICE)
-        probs = torch.softmax(model(**enc).logits, dim=1).cpu()
-        for prob in probs:
-            cls = _decide(prob, group)
-            results.append((LABELS[cls], float(prob[cls]), float(prob[1])))
-    return results
-
-
-# ---------------------------------------------------------------------------
-# Single-text tab
-# ---------------------------------------------------------------------------
-
-@spaces.GPU(duration=60)
-def classify_text(text, group):
-    if not text or not text.strip():
-        return {}
-    tokenizer, model = _load(group)
-    with torch.no_grad():
-        enc = tokenizer(
-            [text],
-            padding=True,
-            truncation=True,
-            max_length=MAX_LENGTH,
-            return_tensors="pt",
-        ).to(DEVICE)
-        prob = torch.softmax(model(**enc).logits, dim=1)[0].cpu()
-    cls = _decide(prob, group)
-    # Surface the model's raw class probabilities; the decided label is shown
-    # in the status line so thresholding effects are visible.
-    return {
-        f"{LABELS[1]} (decided)" if cls == 1 else LABELS[1]: float(prob[1]),
-        f"{LABELS[0]} (decided)" if cls == 0 else LABELS[0]: float(prob[0]),
-    }
-
-
-# ---------------------------------------------------------------------------
-# File tab
-# ---------------------------------------------------------------------------
-
-def _iter_texts(path: Path):
-    suffix = path.suffix.lower()
-    if suffix == ".txt":
-        with path.open("r", encoding="utf-8-sig", errors="ignore") as f:
-            for line in f:
-                text = line.strip()
-                if text:
-                    yield text
-    elif suffix == ".csv":
-        with path.open("r", encoding="utf-8-sig", errors="ignore", newline="") as f:
-            reader = csv.reader(f)
-            for row in reader:
-                if row and row[0].strip():
-                    yield row[0].strip()
-
-
-@spaces.GPU(duration=180)
-def classify_file(file, group):
-    if file is None:
-        return None, "Upload a .txt (one text per line) or .csv (text in the first column) file."
-
-    path = Path(file)
-    if path.suffix.lower() not in ALLOWED_EXTENSIONS:
-        return None, "Unsupported file type. Please upload a .txt or .csv file."
-
-    texts = list(_iter_texts(path))
-    if not texts:
-        return None, "No text rows found in the uploaded file."
-
-    preds = _predict(texts, group)
-
-    tmp = tempfile.NamedTemporaryFile(
-        mode="w", delete=False, suffix=".csv", newline="", encoding="utf-8"
-    )
-    with tmp as f:
-        writer = csv.writer(f)
-        writer.writerow(["text", "social_distinction", "relevance", "confidence"])
-        for text, (label, conf, _) in zip(texts, preds):
-            writer.writerow([text, group, label, f"{conf:.4f}"])
-
-    n_relevant = sum(1 for label, _, _ in preds if label == "Relevant")
-    status = (
-        f"Processed {len(texts)} text(s) against the '{group}' distinction: "
-        f"{n_relevant} Relevant, {len(texts) - n_relevant} Not relevant. "
-        f"Download the full results below."
-    )
-    return tmp.name, status
-
-
-# ---------------------------------------------------------------------------
-# Interface
-# ---------------------------------------------------------------------------
+import generalization
+import moralization
+import relevance
 
 DESCRIPTION = """
-# ISAAC Relevance Filter
-Judges whether a text is **relevant to a given social distinction**, that is,
-whether it actually discusses people in terms of that attribute. Pick a
-distinction below; the app loads the matching fine-tuned classifier on demand.
-For **race** and **skin tone**, a 0.6 confidence threshold is applied to the
-"relevant" class (matching the ISAAC pipeline), so a text is only marked relevant
-when the model is sufficiently confident.
-* **Single text** — paste one text and see the predicted label with class confidence.
-* **Multiple texts** — upload a `.txt` (one text per line) or `.csv` (text in the
-  first column) file and download a labeled results CSV.
-"""
+# ISAAC Text Classifiers
 
-SINGLE_EXAMPLES = [
-    ["My grandmother uses a wheelchair and the new ramp finally lets her visit on her own.", "ability"],
-    ["The recipe calls for two cups of flour and a pinch of salt.", "ability"],
-    ["People kept assuming things about him because he was Black.", "race"],
-]
+Coding-free demos for the classifiers built for the **Illinois Social Attitudes
+(ISAAC)** project. Pick a task below; each one loads its fine-tuned model on
+demand and offers a single-text tab and a bulk file tab.
 
-FOOTER = """
----
-By using this tool you agree to ISAAC's
-[Use Agreement](https://github.com/BabakHemmatian/Illinois_Social_Attitudes/blob/main/Data_Use_Agreement.md).
-For citation information and more details, see the
-[Illinois Social Attitudes (ISAAC) repository](https://github.com/BabakHemmatian/Illinois_Social_Attitudes).
+* **Relevance** — is a text about a given social distinction (ability, age, race,
+  sexuality, skin tone, weight)?
+* **Moralization** — does a text frame its subject in moral terms?
+* **Generalization** — how generalized vs. anecdotal is the language, clause by clause?
 """
 
-with gr.Blocks(title="ISAAC Relevance Filter") as demo:
+with gr.Blocks(title="ISAAC Text Classifiers") as demo:
     gr.Markdown(DESCRIPTION)
 
-    group_dd = gr.Dropdown(
-        choices=GROUPS,
-        value="ability",
-        label="Social Distinction",
-        info="Which fine-tuned relevance classifier to apply.",
-    )
-
-    with gr.Tab("Single text"):
-        text_in = gr.Textbox(
-            label="Input Text", lines=8, placeholder="Paste text here..."
-        )
-        text_btn = gr.Button("Classify", variant="primary")
-        text_out = gr.Label(label="Relevance", num_top_classes=2)
-        gr.Examples(examples=SINGLE_EXAMPLES, inputs=[text_in, group_dd])
-        text_btn.click(classify_text, inputs=[text_in, group_dd], outputs=text_out)
-        text_in.submit(classify_text, inputs=[text_in, group_dd], outputs=text_out)
-
-    with gr.Tab("Multiple texts (file)"):
-        gr.Markdown(
-            "Upload a `.txt` file (one text per line) or a `.csv` file "
-            "(text in the first column). All rows are scored against the "
-            "selected distinction."
-        )
-        file_in = gr.File(label="Input File", file_types=[".txt", ".csv"])
-        file_btn = gr.Button("Classify file", variant="primary")
-        file_status = gr.Markdown()
-        file_out = gr.File(label="Results CSV")
-        file_btn.click(classify_file, inputs=[file_in, group_dd], outputs=[file_out, file_status])
+    with gr.Tabs():
+        with gr.Tab("Relevance"):
+            relevance.build_ui()
+        with gr.Tab("Moralization"):
+            moralization.build_ui()
+        with gr.Tab("Generalization"):
+            generalization.build_ui()
 
-    gr.Markdown(FOOTER)
+    gr.Markdown(common.FOOTER)
 
 
 if __name__ == "__main__":

common.py

@@ -0,0 +1,139 @@
+from __future__ import annotations
+
+# Must be imported before torch: on ZeroGPU hardware the `spaces` package
+# patches torch's CUDA layer at import time. Every other module in this Space
+# imports `common` before touching torch, so this one import fixes the ordering
+# for all of them. ZeroGPU also refuses to start a Space that registers no
+# @spaces.GPU function; the decorated entry points live in the task modules.
+import spaces  # noqa: F401  -- imported for its import-time side effect
+
+import csv
+import os
+import tempfile
+from pathlib import Path
+
+import gradio as gr
+import torch
+
+# hf_spaces/isaac/common.py -> repo root, for the in-repo models/ fallback.
+REPO_ROOT = Path(__file__).resolve().parents[2]
+
+# On ZeroGPU a real GPU only exists inside @spaces.GPU functions; outside them
+# torch runs in CUDA emulation mode, so "cuda" is the correct target either way.
+# Off ZeroGPU (local runs, plain CPU/GPU Spaces) fall back to what's available.
+ON_ZERO_GPU = bool(os.environ.get("SPACES_ZERO_GPU"))
+DEVICE = torch.device(
+    "cuda" if ON_ZERO_GPU or torch.cuda.is_available() else "cpu"
+)
+
+# Private (or gated) model repos need a read token. On Spaces, add it as a
+# secret named HF_TOKEN under Settings -> Variables and secrets. huggingface_hub
+# picks the variable up on its own, but passing it explicitly keeps the failure
+# legible when it is missing: without it the Hub answers 401 and transformers
+# raises a bare OSError at first use, long after the build has gone green.
+HF_TOKEN = os.environ.get("HF_TOKEN") or os.environ.get("HUGGING_FACE_HUB_TOKEN")
+
+ALLOWED_EXTENSIONS = {".txt", ".csv"}
+
+# ZeroGPU charges quota against the duration a call *requests*, not the time it
+# actually uses, so the file tabs estimate their own budget from the input size
+# instead of always reserving a worst-case slot. Overrunning the request kills
+# the call, so the estimates assume a cold model load every time.
+MAX_DURATION = 180
+EMPTY_INPUT_DURATION = 15
+
+UPLOAD_HINT = (
+    "Upload a .txt (one text per line) or .csv (text in the first column) file."
+)
+BAD_EXTENSION = "Unsupported file type. Please upload a .txt or .csv file."
+NO_ROWS = "No text rows found in the uploaded file."
+
+
+def resolve_source(relative: str, hub_id: str) -> str:
+    """Local-first, Hub-fallback model resolution, shared by all three tasks.
+
+    1. $ISAAC_MODELS_DIR/<relative>   (explicit local override)
+    2. <repo_root>/models/<relative>  (in-repo local run)
+    3. <hub_id>                       (the released HF model repo)
+    """
+    models_dir = os.environ.get("ISAAC_MODELS_DIR")
+    if models_dir:
+        candidate = Path(models_dir) / relative
+        if candidate.exists():
+            return str(candidate)
+
+    sibling = REPO_ROOT / "models" / relative
+    if sibling.exists():
+        return str(sibling)
+
+    return hub_id
+
+
+def load_error(what: str, *sources: str) -> gr.Error:
+    """Uniform, actionable message for a model that would not load."""
+    return gr.Error(
+        f"Could not load the {what} model(s) from {', '.join(sources)}. "
+        "If those model repos are private, add a read token as a Space secret "
+        "named HF_TOKEN (Settings -> Variables and secrets), then restart the "
+        "Space."
+    )
+
+
+def iter_texts(path: Path):
+    """Yield the input texts from a .txt (one per line) or .csv (first column)."""
+    suffix = path.suffix.lower()
+    if suffix == ".txt":
+        with path.open("r", encoding="utf-8-sig", errors="ignore") as f:
+            for line in f:
+                text = line.strip()
+                if text:
+                    yield text
+    elif suffix == ".csv":
+        with path.open("r", encoding="utf-8-sig", errors="ignore", newline="") as f:
+            reader = csv.reader(f)
+            for row in reader:
+                if row and row[0].strip():
+                    yield row[0].strip()
+
+
+def count_rows(file) -> int | None:
+    """Row count for duration estimation. None if the file cannot be read, so
+    the caller can fall back to the maximum request."""
+    try:
+        return sum(1 for _ in iter_texts(Path(file)))
+    except OSError:
+        return None
+
+
+def duration_for_rows(file, cold_load: int, seconds_per_unit: float, unit: int = 1) -> int:
+    """Dynamic @spaces.GPU duration: cold model load plus per-unit inference,
+    where a unit is `unit` input rows (e.g. one batch)."""
+    if file is None:
+        return EMPTY_INPUT_DURATION
+    n_rows = count_rows(file)
+    if n_rows is None:
+        return MAX_DURATION
+    n_units = max(1, -(-n_rows // unit))
+    return min(MAX_DURATION, int(cold_load + n_units * seconds_per_unit))
+
+
+def write_results_csv(header, rows) -> str:
+    """Write a results CSV to a temp file and return its path for gr.File."""
+    tmp = tempfile.NamedTemporaryFile(
+        mode="w", delete=False, suffix=".csv", newline="", encoding="utf-8"
+    )
+    with tmp as f:
+        writer = csv.writer(f)
+        writer.writerow(header)
+        for row in rows:
+            writer.writerow(row)
+    return tmp.name
+
+
+FOOTER = """
+---
+By using this tool you agree to ISAAC's
+[Use Agreement](https://github.com/BabakHemmatian/Illinois_Social_Attitudes/blob/main/Data_Use_Agreement.md).
+For citation information and more details, see the
+[Illinois Social Attitudes (ISAAC) repository](https://github.com/BabakHemmatian/Illinois_Social_Attitudes).
+"""

generalization.py

@@ -0,0 +1,520 @@
+from __future__ import annotations
+
+import os
+from functools import lru_cache
+from pathlib import Path
+from typing import Dict, List, Tuple
+
+import common  # imports `spaces` before torch; see common.py
+import gradio as gr
+import matplotlib
+
+matplotlib.use("Agg")  # headless backend for Spaces / servers
+import matplotlib.pyplot as plt
+import numpy as np
+import re
+import spaces
+import torch
+from common import (
+    ALLOWED_EXTENSIONS,
+    BAD_EXTENSION,
+    DEVICE,
+    HF_TOKEN,
+    MAX_DURATION,
+    NO_ROWS,
+    UPLOAD_HINT,
+    duration_for_rows,
+    iter_texts,
+    load_error,
+    resolve_source,
+    write_results_csv,
+)
+from transformers import (
+    AutoModelForSequenceClassification,
+    AutoModelForTokenClassification,
+    AutoTokenizer,
+)
+
+# ---------------------------------------------------------------------------
+# The generalization pipeline uses TWO fine-tuned models plus the base
+# roberta-base tokenizer:
+#   * segmenter  -> token classifier that splits text into clauses
+#   * classifier -> sequence classifier that labels each clause's discourse type
+# ---------------------------------------------------------------------------
+
+SEG_SUBDIR = "label_generalization/label_generalization_segmentation"
+CLF_SUBDIR = "label_generalization/label_generalization"
+
+HUB_SEG = os.environ.get(
+    "ISAAC_GENERALIZATION_SEG_REPO", "BabakScrapes/isaac-generalization-segmentation"
+)
+HUB_CLF = os.environ.get(
+    "ISAAC_GENERALIZATION_REPO", "BabakScrapes/isaac-generalization"
+)
+
+MAX_WORDS_PER_SNIPPET = 200
+SEG_MAX_LENGTH = 512
+CLF_MAX_LENGTH = 128
+CLF_BATCH_SIZE = 32
+
+# Both models load together, so the cold-start budget is larger than the
+# single-model tasks'.
+COLD_LOAD_SECONDS = 45
+SECONDS_PER_TEXT = 2
+
+labels2attrs = {
+    "##BOUNDED EVENT (SPECIFIC)": ("specific", "dynamic", "episodic"),
+    "##BOUNDED EVENT (GENERIC)": ("generic", "dynamic", "episodic"),
+    "##UNBOUNDED EVENT (SPECIFIC)": ("specific", "dynamic", "static"),
+    "##UNBOUNDED EVENT (GENERIC)": ("generic", "dynamic", "static"),
+    "##BASIC STATE": ("specific", "stative", "static"),
+    "##COERCED STATE (SPECIFIC)": ("specific", "dynamic", "static"),
+    "##COERCED STATE (GENERIC)": ("generic", "dynamic", "static"),
+    "##PERFECT COERCED STATE (SPECIFIC)": ("specific", "dynamic", "episodic"),
+    "##PERFECT COERCED STATE (GENERIC)": ("generic", "dynamic", "episodic"),
+    "##GENERIC SENTENCE (DYNAMIC)": ("generic", "dynamic", "habitual"),
+    "##GENERIC SENTENCE (STATIC)": ("generic", "stative", "static"),
+    "##GENERIC SENTENCE (HABITUAL)": ("generic", "stative", "habitual"),
+    "##GENERALIZING SENTENCE (DYNAMIC)": ("specific", "dynamic", "habitual"),
+    "##GENERALIZING SENTENCE (STATIVE)": ("specific", "stative", "habitual"),
+    "##QUESTION": ("NA", "NA", "NA"),
+    "##IMPERATIVE": ("NA", "NA", "NA"),
+    "##NONSENSE": ("NA", "NA", "NA"),
+    "##OTHER": ("NA", "NA", "NA"),
+}
+
+label_names = list(labels2attrs.keys())
+index2label = {i: label for i, label in enumerate(label_names)}
+
+
+@lru_cache(maxsize=1)
+def _load_models():
+    seg_source = resolve_source(SEG_SUBDIR, HUB_SEG)
+    clf_source = resolve_source(CLF_SUBDIR, HUB_CLF)
+    try:
+        tokenizer = AutoTokenizer.from_pretrained(
+            "roberta-base", use_fast=True, add_prefix_space=True
+        )
+        clause_model = (
+            AutoModelForTokenClassification.from_pretrained(
+                seg_source, token=HF_TOKEN
+            )
+            .to(DEVICE)
+            .eval()
+        )
+        classification_model = (
+            AutoModelForSequenceClassification.from_pretrained(
+                clf_source, token=HF_TOKEN
+            )
+            .to(DEVICE)
+            .eval()
+        )
+    except OSError as exc:
+        raise load_error("generalization", seg_source, clf_source) from exc
+    return tokenizer, clause_model, classification_model
+
+
+# ---------------------------------------------------------------------------
+# Text preprocessing
+# ---------------------------------------------------------------------------
+
+def split_sentences(text: str) -> List[str]:
+    text = re.sub(r"\s+", " ", text).strip()
+    if not text:
+        return []
+    sentences = re.split(r"(?<=[.!?])\s+", text)
+    return [s.strip() for s in sentences if s.strip()]
+
+
+def auto_split(text: str, max_words: int = MAX_WORDS_PER_SNIPPET) -> List[str]:
+    sentences = split_sentences(text)
+    if not sentences:
+        return []
+
+    snippets: List[str] = []
+    current_words: List[str] = []
+    for sentence in sentences:
+        sent_words = sentence.split()
+        if current_words and len(current_words) + len(sent_words) > max_words:
+            snippets.append(" ".join(current_words).strip())
+            current_words = sent_words[:]
+        else:
+            current_words.extend(sent_words)
+    if current_words:
+        snippets.append(" ".join(current_words).strip())
+    return snippets
+
+
+def majority_vote(values: List[int]) -> int:
+    if not values:
+        return 1
+    counts = np.bincount(values)
+    return int(np.argmax(counts))
+
+
+# ---------------------------------------------------------------------------
+# Clause segmentation
+# ---------------------------------------------------------------------------
+
+@torch.inference_mode()
+def get_pred_clause_labels(text: str) -> List[int]:
+    tokenizer, clause_model, _ = _load_models()
+    words = text.strip().split()
+    if not words:
+        return []
+
+    enc = tokenizer(
+        words,
+        is_split_into_words=True,
+        return_tensors="pt",
+        truncation=True,
+        max_length=SEG_MAX_LENGTH,
+        padding="max_length",
+    )
+    word_ids = enc.word_ids(batch_index=0)
+    model_inputs = {k: v.to(DEVICE) for k, v in enc.items()}
+    logits = clause_model(**model_inputs).logits[0]
+    token_preds = logits.argmax(dim=-1).detach().cpu().tolist()
+
+    aligned_preds: List[List[int]] = [[] for _ in words]
+    for token_idx, word_id in enumerate(word_ids):
+        if word_id is None:
+            continue
+        aligned_preds[word_id].append(token_preds[token_idx])
+
+    return [majority_vote(preds) if preds else 1 for preds in aligned_preds]
+
+
+def seg_clause(text: str) -> List[str]:
+    words = text.strip().split()
+    if not words:
+        return []
+
+    labels = get_pred_clause_labels(text)
+    segmented_clauses: List[List[str]] = []
+    prev_label = 2
+    current_clause: List[str] | None = None
+
+    for word, label in zip(words, labels):
+        if prev_label == 2:
+            current_clause = []
+        if current_clause is not None:
+            current_clause.append(word)
+        if label == 2 and prev_label in [0, 1]:
+            segmented_clauses.append(current_clause[:])
+            current_clause = None
+        prev_label = label
+
+    if current_clause:
+        segmented_clauses.append(current_clause[:])
+
+    return [" ".join(clause) for clause in segmented_clauses if clause]
+
+
+# ---------------------------------------------------------------------------
+# Clause classification
+# ---------------------------------------------------------------------------
+
+@torch.inference_mode()
+def get_pred_classification_labels(
+    clauses: List[str], batch_size: int = CLF_BATCH_SIZE
+) -> List[Tuple[str, Tuple[str, str, str]]]:
+    if not clauses:
+        return []
+
+    tokenizer, _, classification_model = _load_models()
+    results: List[Tuple[str, Tuple[str, str, str]]] = []
+    for i in range(0, len(clauses), batch_size):
+        batch = clauses[i : i + batch_size]
+        enc = tokenizer(
+            batch,
+            return_tensors="pt",
+            truncation=True,
+            max_length=CLF_MAX_LENGTH,
+            padding="max_length",
+        )
+        model_inputs = {k: v.to(DEVICE) for k, v in enc.items()}
+        logits = classification_model(**model_inputs).logits
+        pred_ids = logits.argmax(dim=-1).detach().cpu().tolist()
+        pred_labels = [index2label[idx] for idx in pred_ids]
+        results.extend(
+            (clause, labels2attrs[label]) for clause, label in zip(batch, pred_labels)
+        )
+    return results
+
+
+def clause_labels_for_text(text: str) -> List[Tuple[str, Tuple[str, str, str]]]:
+    """Segment a text into clauses and classify each. Shared by both tabs."""
+    text = (text or "").strip()
+    if not text:
+        return []
+    all_clauses: List[str] = []
+    for snippet in auto_split(text):
+        all_clauses.extend(seg_clause(snippet))
+    return get_pred_classification_labels(all_clauses)
+
+
+# ---------------------------------------------------------------------------
+# Aggregation
+# ---------------------------------------------------------------------------
+
+def summarize(clause2labels) -> Tuple[int, Dict, Dict, Dict]:
+    """Return (n_clauses, genericity_props, eventivity_props, boundedness_props),
+    each proportion taken over the total clause count (NA included), matching the
+    pie charts."""
+    total = len(clause2labels)
+    gen = {"generic": 0, "specific": 0, "NA": 0}
+    eve = {"dynamic": 0, "stative": 0, "NA": 0}
+    bnd = {"static": 0, "episodic": 0, "habitual": 0, "NA": 0}
+    for _, (genericity, eventivity, boundedness) in clause2labels:
+        gen[genericity] += 1
+        eve[eventivity] += 1
+        bnd[boundedness] += 1
+
+    def prop(d: Dict[str, int]) -> Dict[str, float]:
+        return {k: (v / total if total else 0.0) for k, v in d.items()}
+
+    return total, prop(gen), prop(eve), prop(bnd)
+
+
+# ---------------------------------------------------------------------------
+# Visualization
+# ---------------------------------------------------------------------------
+
+def label_visualization(clause2labels):
+    total_clauses = len(clause2labels)
+    if total_clauses == 0:
+        fig = plt.figure(figsize=(10, 4))
+        plt.text(0.5, 0.5, "No clauses detected.", ha="center", va="center")
+        plt.axis("off")
+        return fig
+
+    genericity_labels, aspect_labels, boundedness_labels = [], [], []
+    for _, attrs in clause2labels:
+        genericity_label, aspect_label, boundedness_label = attrs
+        genericity_labels.append(genericity_label)
+        aspect_labels.append(aspect_label)
+        boundedness_labels.append(boundedness_label)
+
+    genericity_dict = {
+        "Generic": genericity_labels.count("generic"),
+        "Specific": genericity_labels.count("specific"),
+        "NA": genericity_labels.count("NA"),
+    }
+    aspect_dict = {
+        "Dynamic": aspect_labels.count("dynamic"),
+        "Stative": aspect_labels.count("stative"),
+        "NA": aspect_labels.count("NA"),
+    }
+    boundedness_dict = {
+        "Static": boundedness_labels.count("static"),
+        "Episodic": boundedness_labels.count("episodic"),
+        "Habitual": boundedness_labels.count("habitual"),
+        "NA": boundedness_labels.count("NA"),
+    }
+
+    def proportions(d):
+        filtered = {k: v / total_clauses for k, v in d.items() if v > 0}
+        return list(filtered.keys()), list(filtered.values())
+
+    fig, axs = plt.subplots(1, 3, figsize=(10, 5))
+    fig.tight_layout(pad=5.0)
+
+    labels, values = proportions(genericity_dict)
+    axs[0].pie(values, labels=labels, autopct="%.0f%%", normalize=True)
+    axs[0].set_title("Genericity")
+
+    labels, values = proportions(aspect_dict)
+    axs[1].pie(values, labels=labels, autopct="%.0f%%", normalize=True)
+    axs[1].set_title("Eventivity")
+
+    labels, values = proportions(boundedness_dict)
+    axs[2].pie(values, labels=labels, autopct="%.0f%%", normalize=True)
+    axs[2].set_title("Boundedness / Habituality")
+
+    return fig
+
+
+def render_pipeline(clause2labels):
+    """Turn labeled clauses into the three display payloads: numbered clauses,
+    attribute-labeled clauses, and the proportion figure.
+
+    Kept separate from the model pass so the caller can run inference on the GPU
+    and build the matplotlib figure afterwards — on ZeroGPU the decorated call
+    runs in a separate process, and a Figure is a poor thing to ship across that
+    boundary."""
+    output_clauses = [
+        (clause, str(i + 1)) for i, (clause, _) in enumerate(clause2labels)
+    ]
+    highlighted_attrs = [(clause, str(attrs)) for clause, attrs in clause2labels]
+    figure = label_visualization(clause2labels)
+    return output_clauses, highlighted_attrs, figure
+
+
+# ---------------------------------------------------------------------------
+# Entry points
+#
+# Only the model pass is GPU-decorated; the figure is built afterwards, in this
+# process, from the plain (clause, attributes) data the GPU call returns.
+# ---------------------------------------------------------------------------
+
+def _text_duration(text) -> int:
+    n_words = len((text or "").split())
+    n_snippets = max(1, -(-n_words // MAX_WORDS_PER_SNIPPET))
+    return min(MAX_DURATION, COLD_LOAD_SECONDS + n_snippets * SECONDS_PER_TEXT)
+
+
+@spaces.GPU(duration=_text_duration)
+def _clause_labels(text):
+    return clause_labels_for_text(text)
+
+
+def analyze_text(text):
+    return render_pipeline(_clause_labels(text))
+
+
+def _file_duration(file) -> int:
+    return duration_for_rows(file, COLD_LOAD_SECONDS, SECONDS_PER_TEXT)
+
+
+@spaces.GPU(duration=_file_duration)
+def analyze_file(file):
+    if file is None:
+        return None, UPLOAD_HINT
+
+    path = Path(file)
+    if path.suffix.lower() not in ALLOWED_EXTENSIONS:
+        return None, BAD_EXTENSION
+
+    texts = list(iter_texts(path))
+    if not texts:
+        return None, NO_ROWS
+
+    def rows():
+        for text in texts:
+            clause2labels = clause_labels_for_text(text)
+            n, gen, eve, bnd = summarize(clause2labels)
+            clauses_joined = "\n".join(
+                f"{i + 1}: {clause}" for i, (clause, _) in enumerate(clause2labels)
+            )
+            attrs_joined = "\n".join("/".join(attrs) for _, attrs in clause2labels)
+            yield [
+                text, n, clauses_joined, attrs_joined,
+                f"{gen['generic']:.4f}", f"{gen['specific']:.4f}", f"{gen['NA']:.4f}",
+                f"{eve['dynamic']:.4f}", f"{eve['stative']:.4f}", f"{eve['NA']:.4f}",
+                f"{bnd['static']:.4f}", f"{bnd['episodic']:.4f}",
+                f"{bnd['habitual']:.4f}", f"{bnd['NA']:.4f}",
+            ]
+
+    out_path = write_results_csv(
+        [
+            "text",
+            "n_clauses",
+            "clauses",
+            "clause_attributes",
+            "genericity_generic", "genericity_specific", "genericity_NA",
+            "eventivity_dynamic", "eventivity_stative", "eventivity_NA",
+            "boundedness_static", "boundedness_episodic",
+            "boundedness_habitual", "boundedness_NA",
+        ],
+        rows(),
+    )
+    return out_path, f"Processed {len(texts)} text(s). Download the full results below."
+
+
+# ---------------------------------------------------------------------------
+# Interface
+# ---------------------------------------------------------------------------
+
+color_panel_1 = [
+    "red", "green", "yellow", "DodgerBlue", "orange", "DarkSalmon",
+    "pink", "cyan", "gold", "aqua", "violet",
+]
+# Clause-segmentation panel: each clause gets a rotating color keyed by its
+# 1-based index.
+index_colormap = {
+    str(i): color_panel_1[i % len(color_panel_1)] for i in range(1, 100000)
+}
+
+color_panel_2 = [
+    "Gray", "DodgerBlue", "Wheat", "OliveDrab", "DarkKhaki", "DarkSalmon",
+    "Orange", "Gold", "Aqua", "Tomato", "Violet",
+]
+# Attribute panel: one color per distinct (genericity, eventivity, boundedness)
+# triple.
+str_attrs = sorted({str(v) for v in labels2attrs.values()})
+attr_colormap = {attr: color for attr, color in zip(str_attrs, color_panel_2)}
+
+DESCRIPTION = """
+Segments text into **clauses** and labels each clause along three discourse
+dimensions, revealing how generalized vs. anecdotal the language is:
+
+* **Genericity** — generic category (e.g., *humanity*) vs. specific instance (e.g., *my cousin*).
+* **Eventivity** — a state (*God is benevolent*) vs. an event (*I went to Nebraska*).
+* **Boundedness / Habituality** — bounded (*I ate this morning*) vs. unbounded (*God loves us*) vs. repeated (*I went there for years*).
+
+The most anecdotal content involves specific entities in bounded, non-habitual events.
+"""
+
+SINGLE_EXAMPLES = [
+    ["My cousin was arrested for marijuana possession, and it ruined his life for years."],
+    ["Legalizing marijuana would reduce incarceration costs and improve tax revenue."],
+]
+
+
+def build_ui():
+    gr.Markdown(DESCRIPTION)
+
+    # Explicit gr.Tabs: these sit inside a top-level task tab in app.py, so the
+    # implicit grouping Gradio applies to bare sibling gr.Tab blocks is not
+    # something to rely on here.
+    with gr.Tabs():
+        with gr.Tab("Single text"):
+            text_in = gr.Textbox(
+                label="Input Text", lines=6, placeholder="Paste text here..."
+            )
+            text_btn = gr.Button("Analyze", variant="primary")
+
+            # The two text panels show the SAME text twice: once split into
+            # numbered clauses, once colored by discourse attributes.
+            # Side-by-side so they can be read in register.
+            with gr.Row(equal_height=True):
+                seg_out = gr.HighlightedText(
+                    label="Clause Segmentation",
+                    color_map=index_colormap,
+                    combine_adjacent=False,
+                    show_legend=False,
+                )
+                attr_out = gr.HighlightedText(
+                    label="Attribute Classification (genericity, eventivity, boundedness)",
+                    color_map=attr_colormap,
+                    combine_adjacent=False,
+                    show_legend=True,
+                )
+
+            # The proportion charts summarize the whole text; full width below.
+            plot_out = gr.Plot(label="Proportion of Attributes")
+
+            gr.Examples(examples=SINGLE_EXAMPLES, inputs=text_in)
+
+            text_btn.click(
+                analyze_text, inputs=text_in, outputs=[seg_out, attr_out, plot_out]
+            )
+            text_in.submit(
+                analyze_text, inputs=text_in, outputs=[seg_out, attr_out, plot_out]
+            )
+
+        with gr.Tab("Multiple texts (file)"):
+            gr.Markdown(
+                "Upload a `.txt` file (one text per line) or a `.csv` file "
+                "(text in the first column). Each text is segmented and "
+                "classified; the results CSV reports the clauses, their "
+                "attributes, and the attribute proportions per text."
+            )
+            file_in = gr.File(label="Input File", file_types=[".txt", ".csv"])
+            file_btn = gr.Button("Analyze file", variant="primary")
+            file_status = gr.Markdown()
+            file_out = gr.File(label="Results CSV")
+            file_btn.click(
+                analyze_file, inputs=file_in, outputs=[file_out, file_status]
+            )

moralization.py

@@ -0,0 +1,170 @@
+from __future__ import annotations
+
+import os
+from functools import lru_cache
+from pathlib import Path
+
+import common  # imports `spaces` before torch; see common.py
+import gradio as gr
+import spaces
+import torch
+from common import (
+    ALLOWED_EXTENSIONS,
+    BAD_EXTENSION,
+    DEVICE,
+    HF_TOKEN,
+    NO_ROWS,
+    UPLOAD_HINT,
+    duration_for_rows,
+    iter_texts,
+    load_error,
+    resolve_source,
+    write_results_csv,
+)
+from transformers import AutoModelForSequenceClassification, AutoTokenizer
+
+LOCAL_DIRNAME = "label_moralization"
+HUB_ID = os.environ.get("ISAAC_MORALIZATION_REPO", "BabakScrapes/isaac-moralization")
+
+MAX_LENGTH = 512
+BATCH_SIZE = 16
+COLD_LOAD_SECONDS = 40
+SECONDS_PER_BATCH = 2
+
+# Class index -> human label. The training pipeline (label_moralization.py)
+# treats argmax==1 as the moralized class.
+LABELS = {0: "Non-Moralized", 1: "Moralized"}
+
+
+@lru_cache(maxsize=1)
+def _load():
+    source = resolve_source(LOCAL_DIRNAME, HUB_ID)
+    try:
+        tokenizer = AutoTokenizer.from_pretrained(source, token=HF_TOKEN)
+        model = AutoModelForSequenceClassification.from_pretrained(
+            source, token=HF_TOKEN
+        ).to(DEVICE)
+    except OSError as exc:
+        raise load_error("moralization", source) from exc
+    model.eval()
+    return tokenizer, model
+
+
+@torch.no_grad()
+def _predict(texts):
+    tokenizer, model = _load()
+    results = []
+    for start in range(0, len(texts), BATCH_SIZE):
+        batch = texts[start : start + BATCH_SIZE]
+        enc = tokenizer(
+            batch,
+            padding=True,
+            truncation=True,
+            max_length=MAX_LENGTH,
+            return_tensors="pt",
+        ).to(DEVICE)
+        probs = torch.softmax(model(**enc).logits, dim=1).cpu()
+        for prob in probs:
+            cls = int(prob.argmax())
+            results.append((LABELS[cls], float(prob[cls]), float(prob[1])))
+    return results
+
+
+@spaces.GPU(duration=60)
+def classify_text(text):
+    if not text or not text.strip():
+        return {}
+    tokenizer, model = _load()
+    with torch.no_grad():
+        enc = tokenizer(
+            [text],
+            padding=True,
+            truncation=True,
+            max_length=MAX_LENGTH,
+            return_tensors="pt",
+        ).to(DEVICE)
+        prob = torch.softmax(model(**enc).logits, dim=1)[0].cpu()
+    # gr.Label renders this dict as a ranked bar chart of class confidences.
+    return {LABELS[0]: float(prob[0]), LABELS[1]: float(prob[1])}
+
+
+def _file_duration(file) -> int:
+    return duration_for_rows(file, COLD_LOAD_SECONDS, SECONDS_PER_BATCH, BATCH_SIZE)
+
+
+@spaces.GPU(duration=_file_duration)
+def classify_file(file):
+    if file is None:
+        return None, UPLOAD_HINT
+
+    path = Path(file)
+    if path.suffix.lower() not in ALLOWED_EXTENSIONS:
+        return None, BAD_EXTENSION
+
+    texts = list(iter_texts(path))
+    if not texts:
+        return None, NO_ROWS
+
+    preds = _predict(texts)
+    out_path = write_results_csv(
+        ["text", "moralization", "confidence"],
+        (
+            [text, label, f"{conf:.4f}"]
+            for text, (label, conf, _) in zip(texts, preds)
+        ),
+    )
+
+    n_moralized = sum(1 for label, _, _ in preds if label == "Moralized")
+    status = (
+        f"Processed {len(texts)} text(s): "
+        f"{n_moralized} Moralized, {len(texts) - n_moralized} Non-Moralized. "
+        f"Download the full results below."
+    )
+    return out_path, status
+
+
+# ---------------------------------------------------------------------------
+# Interface
+# ---------------------------------------------------------------------------
+
+DESCRIPTION = """
+Detects whether a text frames its subject in **moral** terms (right/wrong, virtue,
+harm, fairness, purity, etc.) versus non-moral terms. This is the document-level
+moralization model from the ISAAC project.
+"""
+
+SINGLE_EXAMPLES = [
+    ["People who cut in line are selfish and should be ashamed of themselves."],
+    ["The bus arrives at the corner of 5th and Main every fifteen minutes."],
+]
+
+
+def build_ui():
+    gr.Markdown(DESCRIPTION)
+
+    # Explicit gr.Tabs: these sit inside a top-level task tab in app.py, so the
+    # implicit grouping Gradio applies to bare sibling gr.Tab blocks is not
+    # something to rely on here.
+    with gr.Tabs():
+        with gr.Tab("Single text"):
+            text_in = gr.Textbox(
+                label="Input Text", lines=8, placeholder="Paste text here..."
+            )
+            text_btn = gr.Button("Classify", variant="primary")
+            text_out = gr.Label(label="Moralization", num_top_classes=2)
+            gr.Examples(examples=SINGLE_EXAMPLES, inputs=text_in)
+            text_btn.click(classify_text, inputs=text_in, outputs=text_out)
+            text_in.submit(classify_text, inputs=text_in, outputs=text_out)
+
+        with gr.Tab("Multiple texts (file)"):
+            gr.Markdown(
+                "Upload a `.txt` file (one text per line) or a `.csv` file "
+                "(text in the first column)."
+            )
+            file_in = gr.File(label="Input File", file_types=[".txt", ".csv"])
+            file_btn = gr.Button("Classify file", variant="primary")
+            file_status = gr.Markdown()
+            file_out = gr.File(label="Results CSV")
+            file_btn.click(
+                classify_file, inputs=file_in, outputs=[file_out, file_status]
+            )

relevance.py

@@ -0,0 +1,221 @@
+from __future__ import annotations
+
+import os
+from functools import lru_cache
+from pathlib import Path
+
+import common  # imports `spaces` before torch; see common.py
+import gradio as gr
+import spaces
+import torch
+from common import (
+    ALLOWED_EXTENSIONS,
+    BAD_EXTENSION,
+    DEVICE,
+    HF_TOKEN,
+    NO_ROWS,
+    UPLOAD_HINT,
+    duration_for_rows,
+    iter_texts,
+    load_error,
+    resolve_source,
+    write_results_csv,
+)
+from transformers import AutoModelForSequenceClassification, AutoTokenizer
+
+# ---------------------------------------------------------------------------
+# Six fine-tuned relevance classifiers, one per social distinction.
+#
+# Models are loaded lazily and cached: only the distinction(s) actually used are
+# pulled into memory, and at most CACHE_SIZE stay resident at once.
+# ---------------------------------------------------------------------------
+
+GROUPS = ["ability", "age", "race", "sexuality", "skin_tone", "weight"]
+
+# Released model-repo naming: prefix + group, e.g. "BabakScrapes/isaac-relevance-race".
+HUB_PREFIX = os.environ.get(
+    "ISAAC_RELEVANCE_REPO_PREFIX", "BabakScrapes/isaac-relevance-"
+)
+
+# Number of distinction models kept resident simultaneously.
+CACHE_SIZE = 2
+
+MAX_LENGTH = 512
+BATCH_SIZE = 16
+COLD_LOAD_SECONDS = 40
+SECONDS_PER_BATCH = 2
+
+# Class index -> human label. The pipeline keeps argmax==1 rows as relevant.
+LABELS = {0: "Not relevant", 1: "Relevant"}
+
+# Confidence thresholding (matches filter_relevance.py): for race and skin_tone
+# the model only assigns the rare "relevant" class when its probability clears
+# THRESHOLD; otherwise it falls back to the next-most-probable class.
+THRESHOLDED_GROUPS = {"race", "skin_tone"}
+THRESHOLD_CLASS = 1
+THRESHOLD = 0.6
+
+
+@lru_cache(maxsize=CACHE_SIZE)
+def _load(group: str):
+    source = resolve_source(f"filter_relevance_{group}", f"{HUB_PREFIX}{group}")
+    try:
+        tokenizer = AutoTokenizer.from_pretrained(source, token=HF_TOKEN)
+        model = AutoModelForSequenceClassification.from_pretrained(
+            source, token=HF_TOKEN
+        ).to(DEVICE)
+    except OSError as exc:
+        raise load_error(f"'{group}' relevance", source) from exc
+    model.eval()
+    return tokenizer, model
+
+
+def _decide(prob, group: str) -> int:
+    """Map a class-probability vector to a predicted class index, replicating
+    the thresholding behavior of filter_relevance.py.predict_tokenized."""
+    if group in THRESHOLDED_GROUPS:
+        if prob[THRESHOLD_CLASS] > THRESHOLD:
+            return THRESHOLD_CLASS
+        masked = prob.clone()
+        masked[THRESHOLD_CLASS] = -1.0
+        return int(masked.argmax())
+    return int(prob.argmax())
+
+
+@torch.no_grad()
+def _predict(texts, group):
+    tokenizer, model = _load(group)
+    results = []
+    for start in range(0, len(texts), BATCH_SIZE):
+        batch = texts[start : start + BATCH_SIZE]
+        enc = tokenizer(
+            batch,
+            padding=True,
+            truncation=True,
+            max_length=MAX_LENGTH,
+            return_tensors="pt",
+        ).to(DEVICE)
+        probs = torch.softmax(model(**enc).logits, dim=1).cpu()
+        for prob in probs:
+            cls = _decide(prob, group)
+            results.append((LABELS[cls], float(prob[cls]), float(prob[1])))
+    return results
+
+
+@spaces.GPU(duration=60)
+def classify_text(text, group):
+    if not text or not text.strip():
+        return {}
+    tokenizer, model = _load(group)
+    with torch.no_grad():
+        enc = tokenizer(
+            [text],
+            padding=True,
+            truncation=True,
+            max_length=MAX_LENGTH,
+            return_tensors="pt",
+        ).to(DEVICE)
+        prob = torch.softmax(model(**enc).logits, dim=1)[0].cpu()
+    cls = _decide(prob, group)
+    # Surface the model's raw class probabilities; the decided label is marked
+    # so thresholding effects stay visible.
+    return {
+        f"{LABELS[1]} (decided)" if cls == 1 else LABELS[1]: float(prob[1]),
+        f"{LABELS[0]} (decided)" if cls == 0 else LABELS[0]: float(prob[0]),
+    }
+
+
+def _file_duration(file, group=None) -> int:
+    return duration_for_rows(file, COLD_LOAD_SECONDS, SECONDS_PER_BATCH, BATCH_SIZE)
+
+
+@spaces.GPU(duration=_file_duration)
+def classify_file(file, group):
+    if file is None:
+        return None, UPLOAD_HINT
+
+    path = Path(file)
+    if path.suffix.lower() not in ALLOWED_EXTENSIONS:
+        return None, BAD_EXTENSION
+
+    texts = list(iter_texts(path))
+    if not texts:
+        return None, NO_ROWS
+
+    preds = _predict(texts, group)
+    out_path = write_results_csv(
+        ["text", "social_distinction", "relevance", "confidence"],
+        (
+            [text, group, label, f"{conf:.4f}"]
+            for text, (label, conf, _) in zip(texts, preds)
+        ),
+    )
+
+    n_relevant = sum(1 for label, _, _ in preds if label == "Relevant")
+    status = (
+        f"Processed {len(texts)} text(s) against the '{group}' distinction: "
+        f"{n_relevant} Relevant, {len(texts) - n_relevant} Not relevant. "
+        f"Download the full results below."
+    )
+    return out_path, status
+
+
+# ---------------------------------------------------------------------------
+# Interface
+# ---------------------------------------------------------------------------
+
+DESCRIPTION = """
+Judges whether a text is **relevant to a given social distinction**, that is,
+whether it actually discusses people in terms of that attribute. Pick a
+distinction below; the app loads the matching fine-tuned classifier on demand.
+For **race** and **skin tone**, a 0.6 confidence threshold is applied to the
+"relevant" class (matching the ISAAC pipeline), so a text is only marked relevant
+when the model is sufficiently confident.
+"""
+
+SINGLE_EXAMPLES = [
+    ["My grandmother uses a wheelchair and the new ramp finally lets her visit on her own.", "ability"],
+    ["The recipe calls for two cups of flour and a pinch of salt.", "ability"],
+    ["People kept assuming things about him because he was Black.", "race"],
+]
+
+
+def build_ui():
+    gr.Markdown(DESCRIPTION)
+
+    group_dd = gr.Dropdown(
+        choices=GROUPS,
+        value="ability",
+        label="Social Distinction",
+        info="Which fine-tuned relevance classifier to apply.",
+    )
+
+    # Explicit gr.Tabs: these sit inside a top-level task tab in app.py, so the
+    # implicit grouping Gradio applies to bare sibling gr.Tab blocks is not
+    # something to rely on here.
+    with gr.Tabs():
+        with gr.Tab("Single text"):
+            text_in = gr.Textbox(
+                label="Input Text", lines=8, placeholder="Paste text here..."
+            )
+            text_btn = gr.Button("Classify", variant="primary")
+            text_out = gr.Label(label="Relevance", num_top_classes=2)
+            gr.Examples(examples=SINGLE_EXAMPLES, inputs=[text_in, group_dd])
+            text_btn.click(classify_text, inputs=[text_in, group_dd], outputs=text_out)
+            text_in.submit(classify_text, inputs=[text_in, group_dd], outputs=text_out)
+
+        with gr.Tab("Multiple texts (file)"):
+            gr.Markdown(
+                "Upload a `.txt` file (one text per line) or a `.csv` file "
+                "(text in the first column). All rows are scored against the "
+                "selected distinction."
+            )
+            file_in = gr.File(label="Input File", file_types=[".txt", ".csv"])
+            file_btn = gr.Button("Classify file", variant="primary")
+            file_status = gr.Markdown()
+            file_out = gr.File(label="Results CSV")
+            file_btn.click(
+                classify_file,
+                inputs=[file_in, group_dd],
+                outputs=[file_out, file_status],
+            )

requirements.txt

@@ -2,4 +2,5 @@ spaces>=0.30
 torch>=2.8,<3
 gradio==6.9.0
 transformers>=4.38,<5
-numpy>=1.26,<3
+numpy>=1.26,<3
+matplotlib>=3.8,<4