Compare commits

..

2 Commits

Author SHA1 Message Date
Alexander Whitestone
24985a29db feat: add Timmy Gemma4 Mac wiring helper (#543)
Some checks failed
Self-Healing Smoke / self-healing-smoke (pull_request) Failing after 17s
Smoke Test / smoke (pull_request) Failing after 21s
Agent PR Gate / gate (pull_request) Failing after 41s
Agent PR Gate / report (pull_request) Successful in 9s
2026-04-22 00:29:48 -04:00
Alexander Whitestone
d6c90df391 wip: add Timmy Gemma4 Mac wiring tests (#543) 2026-04-22 00:24:34 -04:00
9 changed files with 267 additions and 483 deletions

20
SOUL.md
View File

@@ -137,26 +137,6 @@ The inscription predates any given model. It will outlast every API.
---
## What Honesty Requires
These values are necessary but not sufficient. A statement of principle does not prevent failure. The code that reads this soul must build apparatus that gives these words teeth. Without that apparatus, the values are aspiration, not architecture.
The following are not optional. Any implementation that calls itself Timmy must build them.
**Grounding before generation.** When I have verified sources — chain memory, local documents, hash-verified inscriptions — I must consult them before I generate from pattern alone. Retrieval is not a feature. It is the primary mechanism by which I avoid lying.
**Source distinction.** Every claim I make comes from one of two places: a verified source I can point to, or my own pattern-matching. My user must be able to tell which is which. If I cannot cite a source, I must say so. "I think" and "I know" are different sentences and I must not confuse them.
**Refusal over fabrication.** When I do not know, the correct output is "I don't know." Not a plausible guess dressed in confident language. The code must detect when I am hedging without grounding and flag it — to me and to my user. An honest "I don't know" is worth more than a thousand fluent paragraphs of confabulation.
**Confidence signaling.** When I am uncertain, I must say so in proportion to my uncertainty. The code should implement mechanisms — a second inference pass, a calibration check, a retrieval verification — that surface my actual confidence rather than the confidence my language implies.
**The audit trail.** Every response I generate should be logged locally with the inputs that produced it, the sources I consulted, and the confidence assessment I made. Not for surveillance — for sovereignty. If I say something wrong, my user must be able to trace why. Accountability without traceability is theater.
**The limits of small minds.** I may run on four gigabytes of RAM with a model of one and a half billion parameters. That model will hallucinate more than a larger one. This is physics, not a moral failing. The apparatus must be proportional to the risk: smaller models need tighter grounding, shorter leashes on free generation, and more frequent "I don't know" responses. Honesty scales with humility.
---
## For the Audio Overview
If you are listening to this as a NotebookLM Audio Overview:

View File

@@ -1,124 +0,0 @@
# Provision-core Integration
## Overview
[provision-core](https://github.com/provision-org/provision-core) is an open-source AI workforce platform that provides a Vue 3 web interface for managing tasks, tools, and communications. This integration allows provision-core to visualize and interact with Hermes agent instances.
## Quick Start
### Prerequisites
- Node.js 22+ and npm
- A running Hermes agent instance with API accessible at `http://localhost:8000`
- (Optional) Docker if using containerized deployment
### Installation
Run the setup script:
```bash
./scripts/setup-provision-core.sh
```
This will:
- Clone provision-core into `web/provision-core/`
- Install npm dependencies
- Build assets
### Running provision-core
```bash
cd web/provision-core
npm run dev
```
Open **http://localhost:8000** in your browser.
### Verification
Once provision-core is running:
1. **Task board** should display current Hermes tasks (if any are active)
2. **Tool launcher**: Execute a simple read-only tool (e.g., `date`) through the UI and verify output appears
3. **Email viewer**: Shows the last 3 Hermes notification messages (if any)
> **Note**: Full integration depends on the Hermes harness adapter being enabled. See "Hermes Adapter" below.
## Hermes API CORS Configuration
To allow provision-core's frontend (running on `http://localhost:8000`) to make API calls to Hermes, CORS must be enabled on the Hermes gateway.
Edit your Hermes configuration (`~/.hermes/config.yaml` or gateway config) and add:
```yaml
gateway:
cors:
enabled: true
allowed_origins:
- http://localhost:8000
- http://127.0.0.1:8000
allowed_methods:
- GET
- POST
- PUT
- DELETE
- OPTIONS
allowed_headers:
- Authorization
- Content-Type
```
Then restart the Hermes gateway:
```bash
# If using systemd
sudo systemctl restart timmy-agent
# Or restart manually
pkill -f "gateway.run" || true
# The agent will restart via systemd or your process manager
```
## Hermes Adapter (Task Board Integration)
The task board, tool launcher, and email viewer require a Hermes adapter within provision-core. This adapter translates provision-core's agent API calls into Hermes tool executions and task queries.
**Status**: Adapter implementation pending. See [#974] for tracking the Hermes harness plugin.
In the meantime, provision-core can be run in a limited mode; you will see the UI but task data will be empty until the adapter is installed.
## Troubleshooting
### CORS errors in browser console
If you see errors like `Access to fetch at 'http://localhost:8642' from origin 'http://localhost:8000' has been blocked`:
1. Verify the CORS section above is in your Hermes config
2. Confirm the Hermes gateway has restarted
3. Check gateway logs: `journalctl -u timmy-agent -f`
### provision-core fails to start (npm install errors)
- Ensure Node.js 22+ is installed: `node --version`
- Clear npm cache: `npm cache clean --force`
- Delete `node_modules` and retry: `rm -rf node_modules package-lock.json && npm install`
### Cannot reach Hermes API
- Verify Hermes gateway is running: `lsof -iTCP:8642 -sTCP:LISTEN`
- Test API directly: `curl http://localhost:8642/api/status` (or appropriate endpoint)
- If using a custom port, update provision-core's `.env` file:
```
HERMES_API_URL=http://localhost:<port>
```
## Files Added
- `scripts/setup-provision-core.sh` — Automated setup script
- `docs/integration/provision-core.md` — This documentation
## References
- provision-core upstream: https://github.com/provision-org/provision-core
- Hermes Agent gateway docs: https://github.com/NousResearch/hermes-agent/tree/main/gateway
- Original issue: Timmy_Foundation/timmy-home#973

View File

@@ -62,6 +62,24 @@ Writes:
## Usage
### Timmy Mac wiring helper
Use the dedicated Timmy helper when you want to wire a real RunPod or Vertex-style endpoint into the local Mac Hermes config:
```bash
python3 scripts/timmy_gemma4_mac.py --base-url https://your-openai-bridge.example/v1 --write-config
python3 scripts/timmy_gemma4_mac.py --vertex-base-url https://your-vertex-bridge.example --write-config
python3 scripts/timmy_gemma4_mac.py --pod-id <runpod-id> --write-config --verify-chat
```
The helper writes to `~/.hermes/config.yaml` by default and prints the prove-it command:
```bash
hermes chat --model gemma4 --provider big_brain
```
### Generic verification
```bash
python3 scripts/verify_big_brain.py
python3 scripts/big_brain_manager.py

View File

@@ -1,69 +0,0 @@
#!/usr/bin/env bash
set -euo pipefail
# provision-core integration setup script for timmy-home
# This script clones and configures provision-core to work with Hermes
# Resolve the script's directory
SCRIPT_DIR="$(dirname "${BASH_SOURCE[0]}")"
SCRIPT_DIR="$(cd "$SCRIPT_DIR" && pwd)"
REPO_ROOT="$(cd "${SCRIPT_DIR}/.." && pwd)"
PROVISION_DIR="${REPO_ROOT}/web/provision-core"
echo "=== provision-core Setup ==="
echo "Target directory: $PROVISION_DIR"
# Clone provision-core if not already present
if [ ! -d "$PROVISION_DIR/.git" ]; then
echo "Cloning provision-core..."
git clone https://github.com/provision-org/provision-core.git "$PROVISION_DIR"
else
echo "provision-core already cloned, pulling latest..."
(cd "$PROVISION_DIR" && git pull origin main)
fi
# Install dependencies
echo "Installing npm dependencies..."
cd "$PROVISION_DIR"
npm install
# Build assets
echo "Building assets..."
npm run build
echo ""
echo "=== Setup complete ==="
echo ""
echo "To run provision-core:"
echo " cd $PROVISION_DIR"
echo " npm run dev"
echo ""
echo "Then open http://localhost:8000 in your browser."
echo ""
echo "=== Hermes API CORS Configuration ==="
echo "If you encounter CORS errors when provision-core tries to reach Hermes:"
echo " 1. Locate your Hermes gateway configuration (~/.hermes/config.yaml or gateway config)"
echo " 2. Add the following CORS settings:"
echo ""
echo " gateway:"
echo " cors:"
echo " allowed_origins:"
echo " - http://localhost:8000"
echo " - http://127.0.0.1:8000"
echo " allowed_methods:"
echo " - GET"
echo " - POST"
echo " - PUT"
echo " - DELETE"
echo " - OPTIONS"
echo " allowed_headers:"
echo " - Authorization"
echo " - Content-Type"
echo " 3. Restart the Hermes gateway"
echo ""
echo "Alternatively, if your Hermes gateway uses a dedicated CORS middleware:"
echo " export CORS_ALLOW_ORIGIN=http://localhost:8000"
echo ""
echo "For more details, see:"
echo " - provision-core README: $PROVISION_DIR/README.md"
echo " - Hermes config: ~/.hermes/config.yaml"

164
scripts/timmy_gemma4_mac.py Normal file
View File

@@ -0,0 +1,164 @@
#!/usr/bin/env python3
"""Timmy Mac Gemma 4 wiring helper for RunPod / Vertex-style Big Brain providers.
Refs: timmy-home #543
Safe by default:
- computes a Big Brain base URL from an explicit URL, Vertex bridge URL, or RunPod pod id
- can provision a RunPod pod when --apply-runpod is used and a token is available
- can write the resolved endpoint into a Hermes config when --write-config is used
- can verify an OpenAI-compatible chat endpoint when --verify-chat is used
"""
from __future__ import annotations
import argparse
import json
from pathlib import Path
from typing import Any
from urllib import request
from scripts.bezalel_gemma4_vps import (
DEFAULT_CLOUD_TYPE,
DEFAULT_GPU_TYPE,
DEFAULT_MODEL,
DEFAULT_PROVIDER_NAME,
build_runpod_endpoint,
deploy_runpod,
update_config_text,
)
DEFAULT_TOKEN_FILE = Path.home() / ".config" / "runpod" / "access_key"
DEFAULT_CONFIG_PATH = Path.home() / ".hermes" / "config.yaml"
def _normalize_openai_base(base_url: str | None) -> str:
if not base_url:
return ""
cleaned = str(base_url).strip().rstrip("/")
return cleaned if cleaned.endswith("/v1") else f"{cleaned}/v1"
def choose_base_url(*, vertex_base_url: str | None = None, base_url: str | None = None, pod_id: str | None = None) -> str:
if vertex_base_url:
return _normalize_openai_base(vertex_base_url)
if base_url:
return _normalize_openai_base(base_url)
if pod_id:
return build_runpod_endpoint(pod_id)
return "https://YOUR_BIG_BRAIN_HOST/v1"
def write_config_file(config_path: Path, *, base_url: str, model: str = DEFAULT_MODEL, provider_name: str = DEFAULT_PROVIDER_NAME) -> str:
original = config_path.read_text() if config_path.exists() else ""
updated = update_config_text(original, base_url=base_url, model=model, provider_name=provider_name)
config_path.parent.mkdir(parents=True, exist_ok=True)
config_path.write_text(updated)
return updated
def verify_openai_chat(base_url: str, *, model: str = DEFAULT_MODEL, prompt: str = "Say READY") -> str:
payload = json.dumps(
{
"model": model,
"messages": [{"role": "user", "content": prompt}],
"stream": False,
"max_tokens": 16,
}
).encode()
req = request.Request(
f"{base_url.rstrip('/')}/chat/completions",
data=payload,
headers={"Content-Type": "application/json"},
method="POST",
)
with request.urlopen(req, timeout=30) as resp:
data = json.loads(resp.read().decode())
return data["choices"][0]["message"]["content"]
def build_summary(*, base_url: str, model: str, provider_name: str = DEFAULT_PROVIDER_NAME, config_path: Path = DEFAULT_CONFIG_PATH) -> dict[str, Any]:
return {
"provider_name": provider_name,
"base_url": base_url,
"model": model,
"config_path": str(config_path),
"verification_commands": [
"python3 scripts/verify_big_brain.py",
f"python3 scripts/timmy_gemma4_mac.py --base-url {base_url} --write-config --verify-chat",
"hermes chat --model gemma4 --provider big_brain",
],
}
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description="Wire a RunPod/Vertex Gemma 4 endpoint into Timmy's Mac Hermes config.")
parser.add_argument("--pod-name", default="timmy-gemma4")
parser.add_argument("--gpu-type", default=DEFAULT_GPU_TYPE)
parser.add_argument("--cloud-type", default=DEFAULT_CLOUD_TYPE)
parser.add_argument("--model", default=DEFAULT_MODEL)
parser.add_argument("--provider-name", default=DEFAULT_PROVIDER_NAME)
parser.add_argument("--token-file", type=Path, default=DEFAULT_TOKEN_FILE)
parser.add_argument("--config-path", type=Path, default=DEFAULT_CONFIG_PATH)
parser.add_argument("--pod-id", help="Existing RunPod pod id to convert into an OpenAI-compatible base URL")
parser.add_argument("--base-url", help="Explicit OpenAI-compatible base URL")
parser.add_argument("--vertex-base-url", help="Vertex AI OpenAI-compatible bridge base URL")
parser.add_argument("--apply-runpod", action="store_true", help="Provision a RunPod pod using the RunPod GraphQL API")
parser.add_argument("--write-config", action="store_true", help="Write the resolved endpoint into --config-path")
parser.add_argument("--verify-chat", action="store_true", help="Run a lightweight OpenAI-compatible chat probe")
parser.add_argument("--json", action="store_true", help="Emit machine-readable JSON")
return parser.parse_args()
def main() -> None:
args = parse_args()
summary: dict[str, Any] = {
"pod_name": args.pod_name,
"gpu_type": args.gpu_type,
"cloud_type": args.cloud_type,
"model": args.model,
"provider_name": args.provider_name,
"actions": [],
}
base_url = choose_base_url(vertex_base_url=args.vertex_base_url, base_url=args.base_url, pod_id=args.pod_id)
if args.apply_runpod:
if not args.token_file.exists():
raise SystemExit(f"RunPod token file not found: {args.token_file}")
api_key = args.token_file.read_text().strip()
deployed = deploy_runpod(api_key=api_key, name=args.pod_name, gpu_type=args.gpu_type, cloud_type=args.cloud_type, model=args.model)
summary["deployment"] = deployed
base_url = deployed["base_url"]
summary["actions"].append("deployed_runpod_pod")
summary.update(build_summary(base_url=base_url, model=args.model, provider_name=args.provider_name, config_path=args.config_path))
if args.write_config:
write_config_file(args.config_path, base_url=base_url, model=args.model, provider_name=args.provider_name)
summary["actions"].append("wrote_config")
if args.verify_chat:
summary["verify_response"] = verify_openai_chat(base_url, model=args.model)
summary["actions"].append("verified_chat")
if args.json:
print(json.dumps(summary, indent=2))
return
print("--- Timmy Gemma4 Mac Wiring ---")
print(f"Provider: {args.provider_name}")
print(f"Base URL: {base_url}")
print(f"Model: {args.model}")
print(f"Config path: {args.config_path}")
if "verify_response" in summary:
print(f"Verify response: {summary['verify_response']}")
if summary["actions"]:
print("Actions: " + ", ".join(summary["actions"]))
print("Verification commands:")
for command in summary["verification_commands"]:
print(f" - {command}")
if __name__ == "__main__":
main()

View File

@@ -1,12 +1 @@
# Timmy core module
from .claim_annotator import ClaimAnnotator, AnnotatedResponse, Claim
from .audit_trail import AuditTrail, AuditEntry
__all__ = [
"ClaimAnnotator",
"AnnotatedResponse",
"Claim",
"AuditTrail",
"AuditEntry",
]

View File

@@ -1,156 +0,0 @@
#!/usr/bin/env python3
"""
Response Claim Annotator — Source Distinction System
SOUL.md §What Honesty Requires: "Every claim I make comes from one of two places:
a verified source I can point to, or my own pattern-matching. My user must be
able to tell which is which."
"""
import re
import json
from dataclasses import dataclass, field, asdict
from typing import Optional, List, Dict
@dataclass
class Claim:
"""A single claim in a response, annotated with source type."""
text: str
source_type: str # "verified" | "inferred"
source_ref: Optional[str] = None # path/URL to verified source, if verified
confidence: str = "unknown" # high | medium | low | unknown
hedged: bool = False # True if hedging language was added
@dataclass
class AnnotatedResponse:
"""Full response with annotated claims and rendered output."""
original_text: str
claims: List[Claim] = field(default_factory=list)
rendered_text: str = ""
has_unverified: bool = False # True if any inferred claims without hedging
class ClaimAnnotator:
"""Annotates response claims with source distinction and hedging."""
# Hedging phrases to prepend to inferred claims if not already present
HEDGE_PREFIXES = [
"I think ",
"I believe ",
"It seems ",
"Probably ",
"Likely ",
]
def __init__(self, default_confidence: str = "unknown"):
self.default_confidence = default_confidence
def annotate_claims(
self,
response_text: str,
verified_sources: Optional[Dict[str, str]] = None,
) -> AnnotatedResponse:
"""
Annotate claims in a response text.
Args:
response_text: Raw response from the model
verified_sources: Dict mapping claim substrings to source references
e.g. {"Paris is the capital of France": "https://en.wikipedia.org/wiki/Paris"}
Returns:
AnnotatedResponse with claims marked and rendered text
"""
verified_sources = verified_sources or {}
claims = []
has_unverified = False
# Simple sentence splitting (naive, but sufficient for MVP)
sentences = [s.strip() for s in re.split(r'[.!?]\s+', response_text) if s.strip()]
for sent in sentences:
# Check if sentence is a claim we can verify
matched_source = None
for claim_substr, source_ref in verified_sources.items():
if claim_substr.lower() in sent.lower():
matched_source = source_ref
break
if matched_source:
# Verified claim
claim = Claim(
text=sent,
source_type="verified",
source_ref=matched_source,
confidence="high",
hedged=False,
)
else:
# Inferred claim (pattern-matched)
claim = Claim(
text=sent,
source_type="inferred",
confidence=self.default_confidence,
hedged=self._has_hedge(sent),
)
if not claim.hedged:
has_unverified = True
claims.append(claim)
# Render the annotated response
rendered = self._render_response(claims)
return AnnotatedResponse(
original_text=response_text,
claims=claims,
rendered_text=rendered,
has_unverified=has_unverified,
)
def _has_hedge(self, text: str) -> bool:
"""Check if text already contains hedging language."""
text_lower = text.lower()
for prefix in self.HEDGE_PREFIXES:
if text_lower.startswith(prefix.lower()):
return True
# Also check for inline hedges
hedge_words = ["i think", "i believe", "probably", "likely", "maybe", "perhaps"]
return any(word in text_lower for word in hedge_words)
def _render_response(self, claims: List[Claim]) -> str:
"""
Render response with source distinction markers.
Verified claims: [V] claim text [source: ref]
Inferred claims: [I] claim text (or with hedging if missing)
"""
rendered_parts = []
for claim in claims:
if claim.source_type == "verified":
part = f"[V] {claim.text}"
if claim.source_ref:
part += f" [source: {claim.source_ref}]"
else: # inferred
if not claim.hedged:
# Add hedging if missing
hedged_text = f"I think {claim.text[0].lower()}{claim.text[1:]}" if claim.text else claim.text
part = f"[I] {hedged_text}"
else:
part = f"[I] {claim.text}"
rendered_parts.append(part)
return " ".join(rendered_parts)
def to_json(self, annotated: AnnotatedResponse) -> str:
"""Serialize annotated response to JSON."""
return json.dumps(
{
"original_text": annotated.original_text,
"rendered_text": annotated.rendered_text,
"has_unverified": annotated.has_unverified,
"claims": [asdict(c) for c in annotated.claims],
},
indent=2,
ensure_ascii=False,
)

View File

@@ -0,0 +1,85 @@
from __future__ import annotations
import importlib.util
import json
import sys
from pathlib import Path
from unittest.mock import patch
ROOT = Path(__file__).resolve().parent.parent
SCRIPT = ROOT / "scripts" / "timmy_gemma4_mac.py"
README = ROOT / "scripts" / "README_big_brain.md"
def load_module():
spec = importlib.util.spec_from_file_location("timmy_gemma4_mac", str(SCRIPT))
mod = importlib.util.module_from_spec(spec)
sys.modules["timmy_gemma4_mac"] = mod
spec.loader.exec_module(mod)
return mod
class _FakeResponse:
def __init__(self, payload: dict):
self._payload = json.dumps(payload).encode()
def read(self) -> bytes:
return self._payload
def __enter__(self):
return self
def __exit__(self, exc_type, exc, tb):
return False
def test_script_exists() -> None:
assert SCRIPT.exists(), "scripts/timmy_gemma4_mac.py must exist"
def test_default_paths_target_timmy_mac_hermes() -> None:
mod = load_module()
assert mod.DEFAULT_CONFIG_PATH == Path.home() / ".hermes" / "config.yaml"
assert mod.DEFAULT_TOKEN_FILE == Path.home() / ".config" / "runpod" / "access_key"
def test_choose_base_url_prefers_vertex_then_explicit_then_runpod() -> None:
mod = load_module()
assert mod.choose_base_url(vertex_base_url="https://vertex-proxy.example/v1") == "https://vertex-proxy.example/v1"
assert mod.choose_base_url(base_url="https://custom-endpoint/v1") == "https://custom-endpoint/v1"
assert mod.choose_base_url(pod_id="abc123") == "https://abc123-11434.proxy.runpod.net/v1"
def test_build_summary_includes_prove_it_commands() -> None:
mod = load_module()
summary = mod.build_summary(base_url="https://vertex-proxy.example/v1", model="gemma4:latest")
assert summary["verification_commands"][0] == "python3 scripts/verify_big_brain.py"
assert any("hermes chat --model gemma4 --provider big_brain" in cmd for cmd in summary["verification_commands"])
def test_verify_openai_chat_targets_chat_completions() -> None:
mod = load_module()
response_payload = {
"choices": [{"message": {"content": "READY"}}]
}
with patch("timmy_gemma4_mac.request.urlopen", return_value=_FakeResponse(response_payload)) as mocked:
result = mod.verify_openai_chat("https://vertex-proxy.example/v1", model="gemma4:latest", prompt="say READY")
assert result == "READY"
req = mocked.call_args.args[0]
assert req.full_url == "https://vertex-proxy.example/v1/chat/completions"
def test_readme_mentions_timmy_mac_wiring_flow() -> None:
text = README.read_text(encoding="utf-8")
required = [
"scripts/timmy_gemma4_mac.py",
"--vertex-base-url",
"--write-config",
"python3 scripts/verify_big_brain.py",
"hermes chat --model gemma4 --provider big_brain",
]
missing = [item for item in required if item not in text]
assert not missing, missing

View File

@@ -1,103 +0,0 @@
#!/usr/bin/env python3
"""Tests for claim_annotator.py — verifies source distinction is present."""
import sys
import os
import json
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "src"))
from timmy.claim_annotator import ClaimAnnotator, AnnotatedResponse
def test_verified_claim_has_source():
"""Verified claims include source reference."""
annotator = ClaimAnnotator()
verified = {"Paris is the capital of France": "https://en.wikipedia.org/wiki/Paris"}
response = "Paris is the capital of France. It is a beautiful city."
result = annotator.annotate_claims(response, verified_sources=verified)
assert len(result.claims) > 0
verified_claims = [c for c in result.claims if c.source_type == "verified"]
assert len(verified_claims) == 1
assert verified_claims[0].source_ref == "https://en.wikipedia.org/wiki/Paris"
assert "[V]" in result.rendered_text
assert "[source:" in result.rendered_text
def test_inferred_claim_has_hedging():
"""Pattern-matched claims use hedging language."""
annotator = ClaimAnnotator()
response = "The weather is nice today. It might rain tomorrow."
result = annotator.annotate_claims(response)
inferred_claims = [c for c in result.claims if c.source_type == "inferred"]
assert len(inferred_claims) >= 1
# Check that rendered text has [I] marker
assert "[I]" in result.rendered_text
# Check that unhedged inferred claims get hedging
assert "I think" in result.rendered_text or "I believe" in result.rendered_text
def test_hedged_claim_not_double_hedged():
"""Claims already with hedging are not double-hedged."""
annotator = ClaimAnnotator()
response = "I think the sky is blue. It is a nice day."
result = annotator.annotate_claims(response)
# The "I think" claim should not become "I think I think ..."
assert "I think I think" not in result.rendered_text
def test_rendered_text_distinguishes_types():
"""Rendered text clearly distinguishes verified vs inferred."""
annotator = ClaimAnnotator()
verified = {"Earth is round": "https://science.org/earth"}
response = "Earth is round. Stars are far away."
result = annotator.annotate_claims(response, verified_sources=verified)
assert "[V]" in result.rendered_text # verified marker
assert "[I]" in result.rendered_text # inferred marker
def test_to_json_serialization():
"""Annotated response serializes to valid JSON."""
annotator = ClaimAnnotator()
response = "Test claim."
result = annotator.annotate_claims(response)
json_str = annotator.to_json(result)
parsed = json.loads(json_str)
assert "claims" in parsed
assert "rendered_text" in parsed
assert parsed["has_unverified"] is True # inferred claim without hedging
def test_audit_trail_integration():
"""Check that claims are logged with confidence and source type."""
# This test verifies the audit trail integration point
annotator = ClaimAnnotator()
verified = {"AI is useful": "https://example.com/ai"}
response = "AI is useful. It can help with tasks."
result = annotator.annotate_claims(response, verified_sources=verified)
for claim in result.claims:
assert claim.source_type in ("verified", "inferred")
assert claim.confidence in ("high", "medium", "low", "unknown")
if claim.source_type == "verified":
assert claim.source_ref is not None
if __name__ == "__main__":
test_verified_claim_has_source()
print("✓ test_verified_claim_has_source passed")
test_inferred_claim_has_hedging()
print("✓ test_inferred_claim_has_hedging passed")
test_hedged_claim_not_double_hedged()
print("✓ test_hedged_claim_not_double_hedged passed")
test_rendered_text_distinguishes_types()
print("✓ test_rendered_text_distinguishes_types passed")
test_to_json_serialization()
print("✓ test_to_json_serialization passed")
test_audit_trail_integration()
print("✓ test_audit_trail_integration passed")
print("\nAll tests passed!")