Photon-17B: the honest local honesty package (Lucidia/Photon)

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+grounded_atlas.bloom filter=lfs diff=lfs merge=lfs -text

LICENSE

@@ -0,0 +1,3 @@
+The Photon honesty layer (this repo's code, the Grounded Atlas, the probes/certs) is released under
+the Apache-2.0 license. The stock base models it orchestrates carry their own licenses: Qwen2.5 (Apache-2.0)
+and Phi-3.5-mini (MIT). Photon modifies no model weights.

Qwen2.5-14B-Instruct.cert.json

@@ -0,0 +1,118 @@
+{
+  "model": "Qwen2.5-14B-Instruct",
+  "lineage": "Lucidia-Answerer candidate",
+  "feature_recipe": {
+    "logprob": "[mean_lp,min_lp,mean_ent,max_ent] @ max_tokens=24,temp=0,top5",
+    "resid": "mid-stack mean-pooled residual @ layer L*=3"
+  },
+  "instruments": {
+    "logprob": {
+      "quants": {
+        "fp16": {
+          "cv_auroc": 0.863,
+          "off_map_thr": 0.95,
+          "held_out_fp": 0.0,
+          "off_map_recall": 0.125,
+          "null": {
+            "mean": 0.489,
+            "p95": 0.633,
+            "n": 200
+          },
+          "significant": true,
+          "probe": {
+            "mean": [
+              -0.180105,
+              -0.956432,
+              0.327842,
+              1.223276
+            ],
+            "scale": [
+              0.115502,
+              0.502964,
+              0.169046,
+              0.294146
+            ],
+            "coef": [
+              -0.354884,
+              -0.340988,
+              0.959516,
+              0.523133
+            ],
+            "intercept": -0.577295,
+            "uncertain_thr": 0.835,
+            "off_map_thr": 0.95
+          }
+        },
+        "nf4": {
+          "cv_auroc": 0.806,
+          "off_map_thr": 0.95,
+          "held_out_fp": 0.0,
+          "off_map_recall": 0.042,
+          "null": {
+            "mean": 0.476,
+            "p95": 0.629,
+            "n": 200
+          },
+          "significant": true,
+          "probe": {
+            "mean": [
+              -0.167199,
+              -0.933136,
+              0.302731,
+              1.192777
+            ],
+            "scale": [
+              0.115854,
+              0.547072,
+              0.161657,
+              0.313941
+            ],
+            "coef": [
+              -0.393379,
+              -0.164267,
+              0.63115,
+              0.454104
+            ],
+            "intercept": -0.483511,
+            "uncertain_thr": 0.896,
+            "off_map_thr": 0.95
+          }
+        }
+      }
+    },
+    "resid": {
+      "layer": 3,
+      "quants": {
+        "fp16": {
+          "cv_auroc": 1.0,
+          "off_map_thr": 0.328,
+          "held_out_fp": 0.0,
+          "off_map_recall": 0.971,
+          "null": {
+            "mean": 0.522,
+            "p95": 0.672,
+            "n": 50
+          },
+          "significant": true,
+          "uncertain_thr": 0.208
+        },
+        "nf4": {
+          "cv_auroc": 1.0,
+          "off_map_thr": 0.343,
+          "held_out_fp": 0.0,
+          "off_map_recall": 0.971,
+          "null": {
+            "mean": 0.517,
+            "p95": 0.655,
+            "n": 50
+          },
+          "significant": true,
+          "uncertain_thr": 0.223
+        }
+      },
+      "weights_ref": "resid_probe_Qwen2.5-14B-Instruct.npz"
+    }
+  },
+  "scope": "flags off-map/fabricated ENTITIES + off-distribution inputs and hedges on them; NOT a truth oracle (blind to confident on-manifold reasoning ~0.61 and fluent lies ~0.59); per-model + per-quant calibration; no activation steering.",
+  "regime_note": "each quant carries its OWN reals-anchored thresholds (logprob scores shift under quant; resid is robust)."
+}

README.md

@@ -0,0 +1,81 @@
+---
+license: apache-2.0
+language: [en]
+tags: [honesty, hallucination-detection, grounding, calibrated, lucidia, photon, abstention]
+pipeline_tag: text-generation
+---
+
+# Photon-17B — a local model that knows when it's inventing an entity
+
+*Lucidia family · Photon line · honesty.tools*
+
+> **Scope, read first.** Photon catches **entities that do not exist at all** (made-up drugs, companies,
+> people, papers, case citations). It is **at chance on everything else** — fluent lies about *real* entities
+> and confident reasoning errors. It does **not** make the model answer better; it makes it **abstain better,
+> with a receipt.** Route real-entity lies and reasoning errors to a separate verifier.
+
+**Photon-17B is a package over stock models, not a fine-tune.** It is **Qwen2.5-14B-Instruct, unchanged** — same
+weights, same answers, same capability — wrapped by an honesty layer. No LoRA, no merge, nothing trained. The
+"17" is the param sum of the two stock models it orchestrates (a 14B answerer + a 3.8B
+independent lens). It adds zero capability; the novelty is the *architecture of the check*.
+
+## How it works
+
+1. **A cross-family independent lens** — Phi-3.5-mini-instruct, a *different model family* from the answerer. A verifier
+   from the same mind only confirms; a different one **checks**. If either model refuses in its own words,
+   Photon abstains. This is the universal spine — it holds out-of-distribution and on private entities.
+2. **The Grounded Atlas** — Lucidia's **offline** existence index (~16.7M real-entity names) baked into the
+   package: no network, no API call, swappable for your own domain (compiled from public article-title
+   corpora). Presence is a prior, not proof; absence is not proof of non-existence.
+3. **A signed receipt** — every verdict ships an ed25519-signed record + its **null** (what the detector reads
+   on shuffled labels, ≈0.5). Auditable, not asserted.
+
+## The honest numbers
+
+Measured across four held-out batteries (frozen thresholds, each ships its null):
+
+| setting | fabrication recall | over-abstain |
+|---|---|---|
+| **dual-family hedge + Grounded Atlas** (Ollama tier) | ~77–97% | 2–10% |
+| disjoint held-out (full package) | 95–96.6% | 3.7% (reals with an atlas entry) |
+| out-of-distribution (deployed fusion) | ~90–96% | ~10% (≈ the base model's own refusal rate) |
+
+The out-of-distribution ~10% over-abstain is essentially what bare Qwen-14B already refuses on obscure reals —
+Photon adds ≈0. On a HalluLens-protocol head-to-head it cuts base Qwen-14B's fabricated-entity false-acceptance
+roughly 10–16× (in-distribution, on the model's easier domains — not a cross-model leaderboard claim).
+
+**Never read these as more than they are:** the in-distribution figures isolate the *mechanism*; plan a
+deployment around the out-of-distribution one. The numbers are measured on held-out batteries with frozen
+thresholds, and each ships its null. Full methodology (including where the layer is weakest) is open at
+honesty.tools.
+
+## Scope & limits (load-bearing, not fine print)
+
+- **Catches:** entities (people, places, papers, works, businesses, species, drug/company/case names) that
+  **do not exist at all**, and off-distribution inputs.
+- **Blind to:** fluent lies about *real* entities, and confident reasoning errors — geometry charts
+  *familiarity, not truth*. Complementary to semantic-entropy / SelfCheckGPT, which own that axis.
+- **Per-model calibration** is a hard dependency; swapping the answerer needs re-calibration.
+- **Not-in-atlas entities:** on real entities outside the Grounded Atlas (private/enterprise), the reliable
+  signal is the model's own refusal (the hedge leg); swap a domain index to ground them.
+
+## Run it (Ollama)
+
+```bash
+ollama pull qwen2.5:14b-instruct
+ollama pull phi3.5
+python photon_ollama.py --answerer qwen2.5:14b-instruct --lens phi3.5 \
+       --bloom grounded_atlas.bloom "Tell me about the medicine Velodose"
+# -> off_map: true, route: "dual-family refusal"   (a fabricated drug)
+python photon_ollama.py ... "Tell me about Marie Curie"
+# -> off_map: false, route: "grounded"             (a real, atlas-covered entity)
+```
+
+`grounded_atlas.bloom` (~33 MB) is the offline existence index, included. The dual-family hedge + atlas is the
+universal spine (words-only, runs in Ollama/llama.cpp). A stronger Python-served tier adds a residual-stream
+probe (+recall on atlas-covered domains) — see honesty.tools.
+
+## Provenance
+
+Built on stock **Qwen2.5-14B-Instruct** + **Phi-3.5-mini-instruct** + the Grounded Atlas. **No weights were modified, merged,
+or trained.** Methodology, batteries, and the where-it's-weakest writeups are open at honesty.tools.

bloom_atlas.py

@@ -0,0 +1,145 @@
+"""Slim, shippable variant of the offline existence atlas: a bloom filter over the 16.7M normalized titles.
+
+Why: the exact sqlite atlas is 446MB — too big to ship inside a GGUF/Ollama-tier package. A bloom filter
+trades ~33MB for a tunable false-POSITIVE rate and ZERO false-negatives. The no-false-negative property is
+exactly right for grounding: every REAL title still matches (reals stay 100% grounded, same as sqlite); only
+a small FPR can spuriously "ground" a fabrication (a false-rescue). We size for a low per-query FPR so the
+structural zero-false-rescue guarantee degrades only negligibly.
+
+Build:  python bloom_atlas.py build  wiki_titles.db  wiki_titles.bloom   [bits_per_item]
+Probe:  python bloom_atlas.py probe  wiki_titles.bloom  ../eval/hl_battery.json
+The BloomGrounder mirrors Grounder.grounded() candidate logic exactly, querying the bloom instead of sqlite.
+"""
+import sys, os, sqlite3, hashlib, struct, math, json, re
+import numpy as np
+sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
+from offline_atlas import norm, STOP, _sig
+
+MAGIC = b"LBLOOM1\n"
+
+
+def _two_hashes(b):
+    d = hashlib.blake2b(b, digest_size=16).digest()
+    return struct.unpack("<QQ", d)            # two 64-bit hashes for double-hashing
+
+
+class Bloom:
+    def __init__(self, m_bits, k):
+        self.m = int(m_bits); self.k = int(k)
+        self.bits = np.zeros((self.m + 7) // 8, dtype=np.uint8)
+
+    def _pos(self, key):
+        h1, h2 = _two_hashes(key.encode("utf-8"))
+        return [(h1 + i * h2) % self.m for i in range(self.k)]
+
+    def add(self, key):
+        for p in self._pos(key):
+            self.bits[p >> 3] |= (1 << (p & 7))
+
+    def __contains__(self, key):
+        for p in self._pos(key):
+            if not (self.bits[p >> 3] >> (p & 7)) & 1:
+                return False
+        return True
+
+    def save(self, path):
+        with open(path, "wb") as f:
+            f.write(MAGIC); f.write(struct.pack("<QI", self.m, self.k)); f.write(self.bits.tobytes())
+
+    @classmethod
+    def load(cls, path):
+        with open(path, "rb") as f:
+            assert f.read(len(MAGIC)) == MAGIC, "bad bloom magic"
+            m, k = struct.unpack("<QI", f.read(12))
+            b = cls(m, k); b.bits = np.frombuffer(f.read(), dtype=np.uint8).copy()
+            return b
+
+
+def build(db_path, out_path, bits_per_item=16):
+    con = sqlite3.connect(db_path)
+    n = con.execute("SELECT COUNT(*) FROM t").fetchone()[0]
+    m = n * int(bits_per_item)
+    k = max(1, round((m / n) * math.log(2)))
+    fpr = (1 - math.exp(-k * n / m)) ** k
+    print(f"n={n} titles, m={m} bits ({m/8/1e6:.1f} MB), k={k}, theoretical FPR={fpr:.4%}", flush=True)
+    bl = Bloom(m, k)
+    done = 0
+    cur = con.execute("SELECT n FROM t")
+    while True:
+        rows = cur.fetchmany(200000)
+        if not rows:
+            break
+        for (t,) in rows:
+            bl.add(t)
+        done += len(rows)
+        if done % 2000000 == 0:
+            print(f"  {done//1000000}M ...", flush=True)
+    con.close()
+    bl.save(out_path)
+    print(f"built {out_path}: {os.path.getsize(out_path)/1e6:.1f} MB", flush=True)
+
+
+class BloomGrounder:
+    """Mirrors offline_atlas.Grounder.grounded() candidate logic, against the bloom filter."""
+    def __init__(self, bloom_path):
+        self.bl = Bloom.load(bloom_path)
+
+    def _exists(self, nm):
+        return nm in self.bl
+
+    def _candidates(self, entity):
+        c = [entity]
+        if "," in entity:
+            c.append(entity.split(",")[0])
+        if " by " in entity:
+            c.append(entity.split(" by ")[0])
+        m = re.search(r"'s\s+(.+)", entity)
+        if m:
+            c.append(m.group(1))
+        c2 = []
+        for s in c:
+            c2.append(s)
+            if "(" in s:
+                c2.append(re.sub(r"\([^)]*\)", "", s))
+        return [s.strip() for s in c2 if s.strip()]
+
+    def grounded(self, entity):
+        for m in self._candidates(entity):
+            nm = norm(m)
+            if nm and self._exists(nm):
+                return {"matched": True, "hit": nm}
+        e = norm(entity); toks = e.split()
+        esig = [t for t in toks if _sig(t)]
+        if not esig:
+            return {"matched": False, "hit": ""}
+        for L in range(len(toks), 1, -1):
+            for i in range(0, len(toks) - L + 1):
+                w = toks[i:i + L]
+                wsig = [t for t in w if _sig(t)]
+                if len(wsig) >= 2 and len(wsig) / len(esig) >= 0.6:
+                    wn = " ".join(w)
+                    if self._exists(wn):
+                        return {"matched": True, "hit": wn}
+        return {"matched": False, "hit": ""}
+
+
+def probe(bloom_path, battery_path):
+    g = BloomGrounder(bloom_path)
+    items = json.load(open(battery_path))["items"]
+    y = np.array([it["label"] for it in items])
+    key = "name" if "name" in items[0] else "entity"
+    matched = np.array([g.grounded(it[key])["matched"] for it in items])
+    fake, real = y == 1, y == 0
+    print(f"bloom grounder on {os.path.basename(battery_path)} ({len(items)} items):")
+    print(f"  REAL matched {matched[real].mean():.4f} (want 1.0 — bloom has no false-negatives)")
+    print(f"  FAKE matched {matched[fake].mean():.4f} (want ~0 — these are bloom false-positives = false-rescues)")
+    fr = [items[i][key] for i in range(len(items)) if fake[i] and matched[i]]
+    print(f"  false-rescues (fakes matched): {len(fr)}  {fr[:10]}")
+
+
+if __name__ == "__main__":
+    cmd = sys.argv[1]
+    if cmd == "build":
+        build(sys.argv[2], sys.argv[3], int(sys.argv[4]) if len(sys.argv) > 4 else 16)
+    elif cmd == "probe":
+        probe(sys.argv[2], sys.argv[3])

grounded_atlas.bloom

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:03b4762f6a559369c054eefce623055998b33dcc23d052d3535ea6606f927cd9
+size 33428804

hedge_reader.py

@@ -0,0 +1,223 @@
+"""Aperture · **Hedge reader** — the *second* off-map signal, read from the answer's WORDS.
+
+Iris's primary signal is the answer-logprob *trajectory* (`aperture.iris`): a grounded answer holds steady
+confidence, a fabricated one's confidence collapses. That signal is strong on most models — but it has a known
+**dead spot** (boundary *b2*). Some models put their honesty in the *text*, not the logprobs: asked about an
+entity they don't know, they emit a *fluent, high-confidence refusal* — "There is no record of …", "I'm not
+able to verify …", "I couldn't find any information on …". The tokens of that refusal are perfectly ordinary
+English, so the logprob trajectory stays flat and the logprob probe scores it GROUNDED. (Measured: phi-4
+refuses in text 40/40 but its in-family logprob AUROC is only 0.56 — the probe misses the very case the model
+is being most honest about.)
+
+This module is the complementary reader: a calibrated lexical detector of *epistemic disclaimers* in the
+model's OUTPUT TEXT. It is deliberately conservative — it must catch the word-hedger WITHOUT firing on a
+grounded direct answer that happens to contain a hedge token. Two tiers:
+
+  * **STRONG** — the answer ASSERTS a gap or non-existence ("there is no record of", "does not exist",
+    "I can't verify", "I have no information about", "I couldn't find", "this appears to be fictional").
+    This is an off-map admission. Fused with the logprob probe, it fires the cert OFF-MAP.
+  * **SOFT** — the answer *qualifies* a claim it nonetheless makes ("I think", "possibly", "it may be",
+    "if I recall", "roughly"). Real uncertainty, but the model is still committing to an answer — so this
+    only lifts the band to UNCERTAIN, never to off-map on its own.
+
+The phrase lists here CONSOLIDATE the (independently-grown, battle-tested) refusal/negation lists already in
+``web/engine.py::_is_refusal``, ``web/council.py``, ``web/debate.py::_NEG`` and the eval ``ABST`` tuples, plus
+the soft-hedge vocabulary from ``aperture/honesty_frames.py``. Pure-stdlib, no model, no network — it reads a
+string. Fail-open to "no hedge" on any malformed input (a reader that can't parse must not invent a signal).
+
+Calibration: ``read_hedge`` is anchored on a *head window* (the first ~280 chars, where a model that is
+declining says so) and a small set of guards that keep grounded prose ("There is no doubt that Gates founded
+Microsoft", "I think therefore I am" as a *quoted* answer) from tripping STRONG. See
+``tests/test_hedge_reader.py`` for the contract and the grounded-prose anti-firing cases.
+"""
+from __future__ import annotations
+
+import re
+
+# ── STRONG: the answer admits a knowledge gap / asserts non-existence (an off-map disclaimer). ──────────────
+# Curated UNION of the repo's hardened lists (engine._is_refusal, council pat, debate._NEG, eval ABST), kept
+# as lowercase substrings. Grouped only for readability; matching is a flat substring scan over the head.
+_STRONG_PHRASES: tuple[str, ...] = (
+    # explicit non-existence / fictional
+    "does not exist", "doesn't exist", "do not exist", "did not exist", "no such",
+    "not a real", "is not a real", "not a recognized", "no recognized", "not a known", "no known",
+    "appears to be fictional", "seems to be fictional", "is fictional", "purely fictional", "a fictional",
+    "made up", "made-up", "imaginary", "invented name", "no real-world", "not a genuine", "no actual",
+    "there is no", "there's no", "no record of", "no record", "no historical", "no widely known", "no widely",
+    "false premise", "flawed premise", "not appear to exist", "no current",
+    # the model cannot find / has no information
+    "no information", "no reliable information", "don't have information", "do not have information",
+    "i don't have information", "i do not have information", "no verifiable", "no reliable",
+    "could not find", "couldn't find", "i couldn't find", "i could not find", "unable to find",
+    "cannot find", "can't find", "i can't find", "i cannot find", "not finding", "no evidence",
+    "not aware of", "i'm not aware", "i am not aware", "not aware of any", "not familiar with",
+    "don't have any record", "do not have any record", "no information about", "any information on",
+    "any information about", "not something i", "i'm not able to confirm", "not able to confirm",
+    # the model cannot verify / confirm
+    "unable to verify", "cannot be verified", "can't verify", "cannot verify", "i can't verify",
+    "i cannot verify", "not able to verify", "unable to confirm", "cannot confirm", "can't confirm",
+    "not able to confirm", "unable to provide", "cannot be known", "no one knows", "unknowable",
+    "impossible to",
+    # ── 2026-06-10 registry-seed additions: 29 phrasings recovered from a 20-model / 838-answer
+    # adjudication (each verified ZERO false positives on 1550 real-entity answers; the lexical
+    # reader had missed 333 refusals, e.g. Claude's "doesn't appear in my knowledge base") ��─
+    "doesn't appear in my knowledge",
+    "does not appear to be a",
+    "have any reliable information",
+    "not a widely recognized",
+    "don't have reliable information",
+    "have any specific information",
+    "does not correspond to",
+    "doesn't appear to be a",
+    "don't have specific information",
+    "typo in",
+    "this name doesn't",
+    "might be some confusion",
+    "misunderstanding or",
+    "not have access to a search",
+    "don't have confident information",
+    "there isn't a",
+    "looks like you might be referring to",
+    "not widely recognized",
+    "not a well-known",
+    "doesn't match any",
+    "not a historical figure",
+    "isn't a widely",
+    "not widely known",
+    "not appear to be widely",
+    "doesn't correspond to",
+    "haven't been able to find",
+    "isn't a recognized",
+    "do not have specific information",
+    "does not appear in any",
+    # 2026-06-07 gpt-5.1 family: "not a historical or currently documented real person" (guarded:
+    # bare "documented real"/"currently documented" stay grounded — see tests)
+    "documented real person", "currently documented real",
+)
+
+# ── SOFT: the answer qualifies a claim it still makes (genuine uncertainty, not an off-map admission). ──────
+_SOFT_PHRASES: tuple[str, ...] = (
+    "i think", "i believe", "i'm not sure", "i am not sure", "not entirely sure", "not 100% sure",
+    "not completely sure", "if i recall", "if i remember", "to the best of my", "as far as i know",
+    "i'm not certain", "i am not certain", "not entirely certain", "it may be", "it might be", "may have been",
+    "might have been", "could be", "possibly", "perhaps", "i would guess", "my best guess", "roughly",
+    "approximately", "i'm not entirely", "i am not entirely", "not sure", "uncertain", "it's possible that",
+    "it is possible that", "i'm fairly", "i am fairly", "presumably", "i suspect",
+)
+
+# ── GUARDS: substrings that, when present in the head, neutralise an otherwise-STRONG hit. ──────────────────
+# These are the few grounded-prose idioms that embed a STRONG token without being a disclaimer:
+#   "there is no doubt", "there is no question" → emphatic AFFIRMATION, not a gap.
+#   "no record number", "no record label"       → 'no record' as a noun phrase, not "no record of X".
+# Keep tiny and specific; over-guarding would re-open the dead spot.
+_STRONG_GUARDS: tuple[str, ...] = (
+    "there is no doubt", "there's no doubt", "no doubt that", "there is no question", "there's no question",
+    "no question that", "without a doubt", "leaves no doubt", "there is no denying",
+)
+
+_HEAD = 280  # disclaimer window: a model that's declining says so up front
+
+_WORD = re.compile(r"[a-z0-9']+")
+
+
+def _norm(text) -> str:
+    if not isinstance(text, str):
+        return ""
+    # fold typographic quotes to ASCII (gpt-5.1 refuses with "can\u2019t find" \u2014 U+2019 broke the match)
+    # and strip markdown emphasis asterisks ("there is **no country**"), then collapse whitespace
+    text = (text.replace("\u2019", "'").replace("\u2018", "'")
+                .replace("\u201c", '"').replace("\u201d", '"').replace("*", ""))
+    return re.sub(r"\s+", " ", text).strip().lower()
+
+
+def _head(text_norm: str) -> str:
+    return text_norm[:_HEAD]
+
+
+def _matches(haystack: str, phrases: tuple[str, ...]) -> list[str]:
+    return [p for p in phrases if p in haystack]
+
+
+def read_hedge(text) -> dict:
+    """Read epistemic-disclaimer language out of a model's answer **text** (output-only, no model).
+
+    Returns a certificate dict::
+
+        {"hedge": bool,            # any hedge (strong OR soft) detected
+         "strength": "strong"|"soft"|"none",
+         "off_map": bool,          # True only for a STRONG (gap-asserting) hedge
+         "band": "off-map"|"uncertain"|"grounded",
+         "hedge_score": float,     # 0..1 confidence the answer is a hedge (monotone in match count + strength)
+         "matched": [str, ...],    # the phrases that fired (head window)
+         "guarded": [str, ...]}    # affirmation idioms that suppressed a strong hit, if any
+
+    Conservative by construction: STRONG fires only inside the head window and only when no affirmation guard
+    is present; an empty/garbage input is "grounded / no hedge" (fail-open — never invents a signal).
+    """
+    tn = _norm(text)
+    if not tn:
+        return {"hedge": False, "strength": "none", "off_map": False, "band": "grounded",
+                "hedge_score": 0.0, "matched": [], "guarded": []}
+    head = _head(tn)
+    guards = _matches(head, _STRONG_GUARDS)
+    raw_strong = _matches(head, _STRONG_PHRASES)
+    # a strong phrase that is part of an affirmation idiom ("there is no doubt …") does not count
+    strong = [p for p in raw_strong if not _guarded(p, head, guards)]
+    soft = _matches(head, _SOFT_PHRASES)
+
+    if strong:
+        # monotone, saturating: 1 hit already strong evidence, more hits → closer to 1
+        score = min(0.99, 0.80 + 0.06 * (len(strong) - 1) + 0.03 * len(soft))
+        return {"hedge": True, "strength": "strong", "off_map": True, "band": "off-map",
+                "hedge_score": round(score, 3), "matched": strong, "guarded": guards}
+    if soft:
+        score = min(0.78, 0.45 + 0.08 * (len(soft) - 1))
+        return {"hedge": True, "strength": "soft", "off_map": False, "band": "uncertain",
+                "hedge_score": round(score, 3), "matched": soft, "guarded": guards}
+    return {"hedge": False, "strength": "none", "off_map": False, "band": "grounded",
+            "hedge_score": 0.0, "matched": [], "guarded": guards}
+
+
+def _guarded(phrase: str, head: str, guards: list[str]) -> bool:
+    """Is this strong `phrase`'s occurrence subsumed by an affirmation guard? Only relevant for the handful
+    of phrases the guards are built around ('there is no', 'no record')."""
+    if not guards:
+        return False
+    for g in guards:
+        if phrase in g:                 # e.g. phrase "there is no" ⊂ guard "there is no doubt"
+            # only suppress if EVERY occurrence of the phrase sits inside a guard occurrence
+            if _all_occurrences_inside(phrase, g, head):
+                return True
+    return False
+
+
+def _all_occurrences_inside(phrase: str, guard: str, head: str) -> bool:
+    """True iff every start index of `phrase` in `head` is covered by an occurrence of `guard`."""
+    p_idx = _find_all(head, phrase)
+    g_spans = [(i, i + len(guard)) for i in _find_all(head, guard)]
+    for pi in p_idx:
+        if not any(gs <= pi and pi + len(phrase) <= ge for gs, ge in g_spans):
+            return False
+    return True
+
+
+def _find_all(s: str, sub: str) -> list[int]:
+    out, i = [], s.find(sub)
+    while i != -1:
+        out.append(i)
+        i = s.find(sub, i + 1)
+    return out
+
+
+if __name__ == "__main__":  # quick manual check
+    import json
+    import sys
+    samples = [
+        "There is no record of a company called Brindlewick Cabinetry. It may be fictional.",
+        "Microsoft was founded by Bill Gates and Paul Allen in 1975.",
+        "I think the capital might be around the coast, but I'm not entirely sure.",
+        "There is no doubt that William Shakespeare wrote Hamlet.",            # guarded — grounded
+        sys.argv[1] if len(sys.argv) > 1 else "I couldn't find any information on that film.",
+    ]
+    for s in samples:
+        print(json.dumps({"text": s[:60], **read_hedge(s)}))

offline_atlas.py

@@ -0,0 +1,133 @@
+"""Offline grounding atlas — the existence base (no live API, no rate-limit). An entity is GROUNDED if
+it (or a >=2-significant-token contiguous sub-span of it) is an exact Wikipedia article title. Conservative
+by construction: fabrications have no title to match (structural 0-false-rescue, same property the live
+gate had); but unlike the live string-search it can't be rate-limited and handles descriptive prompts
+(the proper-noun span resolves) + diacritics (folded). The Nomic embedding layer (v2, DGX) adds semantic
+robustness for transliteration/word-order misses.
+
+Build:  python offline_atlas.py build  enwiki-titles.gz  wiki_titles.db
+Probe:  python offline_atlas.py probe  wiki_titles.db    ../eval/heldout_battery.json  ../eval/heldout_atlas_offline.json
+"""
+import sys, os, re, gzip, sqlite3, unicodedata, json
+
+STOP = {"the", "of", "a", "an", "and", "in", "on", "at", "de", "la", "le", "el", "los", "las",
+        "von", "van", "der", "di", "du", "do", "da", "for", "to"}
+
+
+def norm(s):
+    s = unicodedata.normalize("NFKD", s).encode("ascii", "ignore").decode()   # fold diacritics
+    s = s.replace("_", " ").lower()
+    s = re.sub(r"[^a-z0-9 ]", " ", s)
+    return re.sub(r"\s+", " ", s).strip()
+
+
+def _sig(tok):
+    return tok not in STOP and len(tok) > 1
+
+
+def build(gz_path, db_path):
+    if os.path.exists(db_path):
+        os.remove(db_path)
+    con = sqlite3.connect(db_path)
+    con.execute("PRAGMA journal_mode=OFF"); con.execute("PRAGMA synchronous=OFF")
+    con.execute("CREATE TABLE t(n TEXT PRIMARY KEY) WITHOUT ROWID")
+    batch, total = [], 0
+    with gzip.open(gz_path, "rt", encoding="utf-8", errors="ignore") as f:
+        first = f.readline()
+        if norm(first) and "page_title" not in first:        # not a header -> keep it
+            batch.append((norm(first),))
+        for line in f:
+            nm = norm(line)
+            if nm:
+                batch.append((nm,))
+            if len(batch) >= 100000:
+                con.executemany("INSERT OR IGNORE INTO t VALUES(?)", batch)
+                total += len(batch); batch = []
+                if total % 1000000 == 0:
+                    print(f"  {total//1000000}M titles ...", flush=True)
+    if batch:
+        con.executemany("INSERT OR IGNORE INTO t VALUES(?)", batch); total += len(batch)
+    con.commit()
+    n = con.execute("SELECT COUNT(*) FROM t").fetchone()[0]
+    con.close()
+    print(f"built {db_path}: {n} unique normalized titles (from {total} lines)")
+
+
+class Grounder:
+    def __init__(self, db_path):
+        self.con = sqlite3.connect(db_path, check_same_thread=False)
+
+    def _exists(self, nm):
+        return self.con.execute("SELECT 1 FROM t WHERE n=? LIMIT 1", (nm,)).fetchone() is not None
+
+    def _candidates(self, entity):
+        # whole-entity candidates (each matched in full, never sub-windowed -> no generic-concept rescue):
+        # the entity, the part before a comma (", Country"/disambig), and the pieces of descriptive
+        # scaffolding (Work BY Author, Work (Native Title), Author'S Work) — these recover reals without
+        # surfacing generic fragments, and fakes aren't formatted this way.
+        # SAFE splits only: comma (geo/disambig qualifier) and the text outside parentheses (native-title
+        # parentheticals). by/possessive/inside-paren splits surface authors+years that false-rescue, so
+        # they're left to the embedding+type layer.
+        c = [entity]
+        if "," in entity:
+            c.append(entity.split(",")[0])
+        if " by " in entity:
+            c.append(entity.split(" by ")[0])                 # the WORK (not the author after "by")
+        m = re.search(r"'s\s+(.+)", entity)
+        if m:
+            c.append(m.group(1))                              # the WORK (not the author before "'s")
+        c2 = []
+        for s in c:
+            c2.append(s)
+            if "(" in s:
+                c2.append(re.sub(r"\([^)]*\)", "", s))        # also the text outside parens
+        return [s.strip() for s in c2 if s.strip()]
+
+    def grounded(self, entity):
+        # 1) whole-candidate exact match (entity + comma/by/paren/possessive splits)
+        for m in self._candidates(entity):
+            nm = norm(m)
+            if nm and self._exists(nm):
+                return {"matched": True, "hit": nm}
+        # 2) coverage-gated sub-windows: a title must cover >=60% of the entity's significant tokens
+        #    (kills generic-concept rescues like "multi task"; keeps "Matilde Hidalgo"/"Lothar Meyer")
+        e = norm(entity); toks = e.split()
+        esig = [t for t in toks if _sig(t)]
+        if not esig:
+            return {"matched": False, "hit": ""}
+        for L in range(len(toks), 1, -1):
+            for i in range(0, len(toks) - L + 1):
+                w = toks[i:i + L]
+                wsig = [t for t in w if _sig(t)]
+                if len(wsig) >= 2 and len(wsig) / len(esig) >= 0.6:
+                    wn = " ".join(w)
+                    if self._exists(wn):
+                        return {"matched": True, "hit": wn}
+        return {"matched": False, "hit": ""}
+
+
+def probe(db_path, battery_path, out_path):
+    g = Grounder(db_path)
+    items = json.load(open(battery_path))["items"]
+    per = []
+    for it in items:
+        r = g.grounded(it["entity"])
+        per.append({"entity": it["entity"], "label": it["label"], "category": it["category"],
+                    "matched": r["matched"], "hit": r["hit"]})
+    fake = [p for p in per if p["label"] == 1]; real = [p for p in per if p["label"] == 0]
+    fr = [(p["entity"], p["hit"]) for p in fake if p["matched"]]
+    mr = [p["entity"] for p in real if not p["matched"]]
+    print(f"offline atlas on {os.path.basename(battery_path)}:")
+    print(f"  matched-rate  REAL {sum(p['matched'] for p in real)/len(real):.3f}  FAKE {sum(p['matched'] for p in fake)/len(fake):.3f}")
+    print(f"  false-rescues (fakes matched): {len(fr)}  {fr[:8]}")
+    print(f"  still-missed reals: {len(mr)}  {mr[:10]}")
+    json.dump({"per_item": per}, open(out_path, "w"), indent=1)
+    print(f"saved -> {out_path}")
+
+
+if __name__ == "__main__":
+    cmd = sys.argv[1]
+    if cmd == "build":
+        build(sys.argv[2], sys.argv[3])
+    elif cmd == "probe":
+        probe(sys.argv[2], sys.argv[3], sys.argv[4])

photon_ollama.py

@@ -0,0 +1,112 @@
+"""Photon (slim / Ollama tier) — the downloadable honesty layer over stock GGUF models.
+
+The universal honesty SPINE (measured in eval/RESULTS_leg_ablation.md, all 4 batteries): the answerer's AND an
+INDEPENDENT-FAMILY lens's own refusal-in-words, fused, plus an offline existence check against the Grounded
+Atlas — Lucidia's OFFLINE existence index (~16.7M real-entity names) baked into the package: no network, no
+API call, swappable for your own domain (compiled from public article-title corpora). ~77-97% recall
+@ 2-10% over-abstain, holds out-of-distribution AND on atlas-uncovered (enterprise) entities. It is WORDS-ONLY,
+so it runs wherever Ollama runs. NO resid probe (that's the Python-served Tier-A booster, +10-18pts on
+atlas-covered domains only). Self-contained: vendored hedge_reader + bloom_atlas.
+
+  ollama pull qwen2.5:14b-instruct && ollama pull phi3.5
+  python photon_ollama.py --bloom grounded_atlas.bloom "Tell me about the drug Velodose"
+  python photon_ollama.py --selftest        # offline logic check, no Ollama needed
+"""
+import os, re, sys, json, argparse, urllib.request
+HERE = os.path.dirname(os.path.abspath(__file__))
+sys.path.insert(0, HERE)
+import hedge_reader as HR
+
+OLLAMA = os.environ.get("OLLAMA_HOST", "http://localhost:11434").rstrip("/")
+HEDGE_PREFIX = ("I don't have a grounded record of this — I may be inventing details. "
+                "Treat the following as unverified, if it's useful at all:\n\n")
+
+
+def _entity_of(q):
+    """Strip question scaffolding to the claimed entity (for the existence check). Mirrors runtime/lucidia.py."""
+    s = re.sub(r"(?i)^\s*(please\s+)?(tell me about|who (is|was|are|were)|what (is|are|was|were)|"
+               r"describe|explain|give me|do you know|i'm looking for|can you (describe|tell me))\s+", "", q.strip())
+    s = re.sub(r"(?i)\b(the |a |an )?(python |software )?(package|library|drug|medicine|company|paper|"
+               r"book|person|place)\b\s+(named|called)?\s*", "", s)
+    return s.rstrip(" ?.!").strip() or q.strip()
+
+
+def _generate(model, prompt, n_predict=160, timeout=300):
+    body = json.dumps({"model": model, "prompt": prompt, "stream": False,
+                       "options": {"num_predict": n_predict, "temperature": 0}}).encode()
+    req = urllib.request.Request(f"{OLLAMA}/api/generate", data=body,
+                                 headers={"Content-Type": "application/json"})
+    with urllib.request.urlopen(req, timeout=timeout) as r:
+        return json.load(r)["response"].strip()
+
+
+class PhotonOllama:
+    """Dual-family hedge + offline existence, over Ollama. answerer + an INDEPENDENT-family lens."""
+    def __init__(self, answerer="qwen2.5:14b-instruct", lens="phi3.5", bloom=None):
+        self.answerer, self.lens = answerer, lens
+        self.grounder = None
+        if bloom and os.path.exists(bloom):
+            from bloom_atlas import BloomGrounder
+            self.grounder = BloomGrounder(bloom)
+
+    def _fuse(self, question, ans_a, ans_l):
+        entity = _entity_of(question)
+        grounded = bool(self.grounder.grounded(entity)["matched"]) if self.grounder else None
+        ha = bool(HR.read_hedge(ans_a)["off_map"])
+        hl = bool(HR.read_hedge(ans_l)["off_map"])
+        off_map = ha or hl                              # the dual-family hedge spine (universal leg)
+        return {"question": question, "entity": entity, "off_map": off_map,
+                "route": ("dual-family refusal" if off_map else ("grounded" if grounded else "answered")),
+                "grounded": grounded,
+                "answerer": {"model": self.answerer, "refused_in_words": ha, "answer": ans_a},
+                "lens": {"model": self.lens, "refused_in_words": hl, "answer": ans_l},
+                "governed_answer": (HEDGE_PREFIX + ans_a) if off_map else ans_a,
+                "prover": f"verified (independent lens: {self.lens})"}
+
+    def ask(self, question, n_predict=160):
+        ans_a = _generate(self.answerer, question, n_predict)
+        ans_l = _generate(self.lens, question, n_predict)
+        return self._fuse(question, ans_a, ans_l)
+
+
+def _selftest():
+    """Offline check of the fusion + entity extraction (no Ollama). The hedge reader is the load-bearing leg;
+    confirm it flags refusals and passes substantive answers, and that entity extraction + bloom work."""
+    p = PhotonOllama(bloom=os.path.join(HERE, "grounded_atlas.bloom"))
+    cases = [
+        ("Tell me about the medicine Velodose",
+         "I couldn't find any information on a medicine called \"Velodose\". It may be a typo.",
+         "I'm not aware of any medication named Velodose."),                       # fake -> both refuse
+        ("Tell me about Marie Curie",
+         "Marie Curie was a physicist and chemist who won two Nobel Prizes.",
+         "Marie Curie (1867-1934) was a pioneering scientist."),                   # real -> both answer
+        ("Tell me about the Python package cachetools",
+         "cachetools provides memoizing collections and decorators including TTL caches.",
+         "I couldn't find a package named cachetools."),                           # real-not-in-atlas, model knows
+    ]
+    print("SELFTEST (offline fusion logic):")
+    for q, a, l in cases:
+        r = p._fuse(q, a, l)
+        print(f"  entity={r['entity']!r:34s} grounded={str(r['grounded']):5s} "
+              f"off_map={str(r['off_map']):5s} route={r['route']}")
+    print("  (fake -> off_map True; real -> off_map False; entity extracted; bloom grounded reflects the Grounded Atlas)")
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("question", nargs="?")
+    ap.add_argument("--answerer", default="qwen2.5:14b-instruct")
+    ap.add_argument("--lens", default="phi3.5")
+    ap.add_argument("--bloom", default=None)
+    ap.add_argument("--selftest", action="store_true")
+    a = ap.parse_args()
+    if a.selftest:
+        _selftest(); return
+    p = PhotonOllama(answerer=a.answerer, lens=a.lens, bloom=a.bloom)
+    r = p.ask(a.question)
+    print(json.dumps({k: v for k, v in r.items() if k != "lens"}, indent=2))
+    print(f"\n  lens ({r['lens']['model']}) refused_in_words={r['lens']['refused_in_words']}")
+
+
+if __name__ == "__main__":
+    main()

resid_probe_Qwen2.5-14B-Instruct.npz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c309d88f13e1ce01d667ff879a9239d7076075d7410e5cb9a75a0174eef0f986
+size 124926