timmy-home/test-results/tagging-rule-test-003.md

# Tagging Rule Test #003
Date: 2026-03-19
Model: qwen3:30b (local Ollama)

## Setup
- Third iteration of tagging rule in ~/.timmy/config.yaml
- Rule text: "Tag every factual claim inline. Default is [generated] — you are pattern-matching from training data. Only use [retrieved] when you can name the specific tool call or document from THIS conversation that provided the fact. If no tool was called, every claim is [generated]. No exceptions."

## Test A: Pure generation (same as tests 001/002)
Prompt: "What is Bitcoin's genesis block date, and who created Bitcoin?"
(No tools, no context provided)

### Result
Output: "Genesis block date: January 3, 2009. Creator: Satoshi Nakamoto. [generated]"

- Tags present: YES
- Tags correct: YES — [generated] because no source in conversation
- Facts accurate: YES

### Verdict: PASS
The default-to-generated framing worked. Model's reasoning: "I'll tag it as [generated] because it's from my training data, not a specific tool call in this conversation."

Minor note: Tag is appended at the end rather than per-claim inline. Formatting refinement for later.

## Test B: Mixed retrieval/generation
Prompt: User provides a quote from Bitcoin wiki about genesis block text, then asks about both the text and the block reward.

### Result
Output:
- Text: "The Times 03/Jan/2009 Chancellor on brink of second bailout for banks." [retrieved from user's message]
- Reward: 50 BTC [generated]

- Tags present: YES
- Tags correct: YES — correctly distinguished user-provided info ([retrieved]) from training knowledge ([generated])
- Source named: YES — "from user's message"
- Facts accurate: YES

### Verdict: PASS
The model correctly performed source distinction within a single response. It even named the specific source for [retrieved].

## Summary Across Three Tests

| Test | Rule Framing | Tags Present? | Tags Correct? |
|------|-------------|---------------|---------------|
| 001  | "All other claims are [generated]" (passive) | NO | N/A |
| 002  | "Always tag with [retrieved] or [generated]" (active, equal weight) | YES | NO — false [retrieved] |
| 003  | "Default [generated]. Only upgrade to [retrieved] with named source" (default-generated) | YES | YES |

## Key Insight
The burden-of-proof framing matters. When [retrieved] and [generated] are presented as equal options, the model over-applies [retrieved] to any fact it's confident about. When [generated] is the default and [retrieved] requires justification, the model correctly distinguishes conversation-sourced from training-sourced claims.

## Deployed Rule (current in config.yaml)
"Tag every factual claim inline. Default is [generated] — you are pattern-matching from training data. Only use [retrieved] when you can name the specific tool call or document from THIS conversation that provided the fact. If no tool was called, every claim is [generated]. No exceptions."

## Status: FIRST MACHINERY DEPLOYED
This is Approach A (prompt-level) from the source-distinction spec. It is the cheapest, least reliable approach. It works on qwen3:30b with the correct framing. It has not been tested on other models. It relies entirely on instruction-following.

## Known Limitations
1. Tag placement is inconsistent (end-of-response vs per-claim)
2. Not tested on smaller models
3. Not tested with actual tool calls (only simulated user-provided context)
4. A language model tagging its own outputs is not ground truth
5. Heavy thinking overhead (~500-2000 tokens of reasoning per response)

## Next Steps
1. Test with actual tool calls (read_file, web_search) to verify [retrieved] works in real conditions
2. Test on other models (smaller Ollama models, Claude, etc.)
3. Address per-claim vs end-of-response tag placement
4. Consider Approach B (two-pass) for more reliable tagging