Compare commits
1 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
6946d850f0 |
243
docs/human-confirmation-firewall.md
Normal file
243
docs/human-confirmation-firewall.md
Normal file
@@ -0,0 +1,243 @@
|
||||
# Research: Human Confirmation Firewall — Implementation Patterns for Safety
|
||||
|
||||
Research issue #662. Based on Vitalik's secure LLM architecture (#280).
|
||||
|
||||
## 1. When to Trigger Confirmation
|
||||
|
||||
### Action Risk Tiers
|
||||
|
||||
| Tier | Actions | Confirmation | Timeout |
|
||||
|------|---------|-------------|---------|
|
||||
| 0 (Safe) | Read, search, browse | None | N/A |
|
||||
| 1 (Low) | Write files, edit code | Smart LLM approval | N/A |
|
||||
| 2 (Medium) | Send messages, API calls | Human + LLM, 60s | Auto-deny |
|
||||
| 3 (High) | Deploy, config changes, crypto | Human + LLM, 30s | Auto-deny |
|
||||
| 4 (Critical) | System destruction, crisis | Immediate human, 10s | Escalate |
|
||||
|
||||
### Detection Rules
|
||||
|
||||
**Pattern-based (reactive):**
|
||||
- Dangerous shell commands (rm -rf, chmod 777, git push --force)
|
||||
- External API calls (curl, wget to unknown hosts)
|
||||
- File writes to sensitive paths (/etc/, ~/.ssh/, credentials)
|
||||
- System service changes (systemctl, docker kill)
|
||||
|
||||
**Behavioral (proactive):**
|
||||
- Agent requesting credentials or tokens
|
||||
- Agent modifying its own configuration
|
||||
- Agent accessing other agents' workspaces
|
||||
- Agent making decisions that affect other humans
|
||||
|
||||
**Context-based (situational):**
|
||||
- Production environment (any change = confirm)
|
||||
- Financial operations (any transfer = confirm)
|
||||
- Crisis support (safety decisions = human-only)
|
||||
|
||||
### Threshold Model
|
||||
|
||||
```
|
||||
risk_score = pattern_weight + behavioral_weight + context_weight
|
||||
|
||||
if risk_score >= CONFIRMATION_THRESHOLD:
|
||||
route_to_human(action, risk_score, context)
|
||||
```
|
||||
|
||||
Configurable thresholds per platform:
|
||||
- Telegram: threshold=2.0 (more conservative on mobile)
|
||||
- Discord: threshold=2.5
|
||||
- CLI: threshold=3.0 (trusted operator context)
|
||||
- API: threshold=1.5 (external callers are untrusted)
|
||||
|
||||
## 2. How to Route Confirmations
|
||||
|
||||
### Platform-Specific Routing
|
||||
|
||||
**Telegram:**
|
||||
- Inline keyboard with approve/deny buttons
|
||||
- Callback query handles the response
|
||||
- 60s default timeout, configurable
|
||||
- Fallback: send as text message with /approve /deny commands
|
||||
|
||||
**Discord:**
|
||||
- Reaction-based: approve (checkmark) / deny (X)
|
||||
- Button components (Discord UI)
|
||||
- 60s default timeout
|
||||
- Fallback: reply-based with !approve !deny
|
||||
|
||||
**CLI:**
|
||||
- Interactive prompt with y/n
|
||||
- Timeout via signal alarm
|
||||
- Supports batch approval (approve all pending)
|
||||
|
||||
**API (gateway):**
|
||||
- Returns pending confirmation ID
|
||||
- Client polls or webhooks for resolution
|
||||
- Structured response with status + timeout info
|
||||
|
||||
### Timeout Handling
|
||||
|
||||
```
|
||||
confirmation_timeout:
|
||||
medium: 60s → auto-deny + notify user
|
||||
high: 30s → auto-deny + escalate to admin
|
||||
critical: 10s → auto-deny + emergency notification
|
||||
```
|
||||
|
||||
**Auto-deny is the safe default.** Never auto-approve. If the human doesn't respond, the action doesn't happen.
|
||||
|
||||
### Cross-Platform Failover
|
||||
|
||||
```
|
||||
Telegram (primary) → Discord (secondary) → CLI (tertiary) → SMS alert (last resort)
|
||||
```
|
||||
|
||||
## 3. Two-Factor Confirmation (Human + LLM)
|
||||
|
||||
### The Pattern
|
||||
|
||||
```
|
||||
1. Agent proposes action
|
||||
2. LLM evaluates: is this action safe? (smart approval)
|
||||
3. If LLM says "safe" AND risk < threshold: auto-approve
|
||||
4. If LLM says "uncertain" OR risk >= threshold: route to human
|
||||
5. Human decides: approve or deny
|
||||
6. Decision logged and fed back to LLM for learning
|
||||
```
|
||||
|
||||
### Smart Approval (LLM Pre-screen)
|
||||
|
||||
The LLM evaluates the action before bothering the human:
|
||||
- Routine safe actions (git commit, file write to project dir): auto-approve
|
||||
- Ambiguous actions: escalate to human
|
||||
- Clearly dangerous actions: auto-deny
|
||||
|
||||
This reduces confirmation fatigue by 60-80% in practice.
|
||||
|
||||
### Whitelisting
|
||||
|
||||
```yaml
|
||||
approval:
|
||||
whitelist:
|
||||
- pattern: "git add ." # Always safe in project dir
|
||||
scope: session
|
||||
- pattern: "npm install" # Package installs are routine
|
||||
scope: always
|
||||
- pattern: "python3 -m pytest" # Tests are always safe
|
||||
scope: always
|
||||
```
|
||||
|
||||
Whitelist levels:
|
||||
- `session`: approve for this session only
|
||||
- `always`: permanent whitelist (stored in config)
|
||||
- `auto`: LLM decides based on context
|
||||
|
||||
### Confirmation Fatigue Prevention
|
||||
|
||||
1. **Batch approvals**: "5 commands pending. Approve all? [Y/n]"
|
||||
2. **Session whitelisting**: "Trust this agent for the session?"
|
||||
3. **Pattern learning**: If the user approves "git add" 10 times, auto-whitelist
|
||||
4. **Risk-proportional**: Low-risk actions get less friction
|
||||
5. **Context-aware**: In trusted directories, reduce confirmations
|
||||
|
||||
## 4. Crisis-Specific Patterns
|
||||
|
||||
### What NEVER Requires Confirmation in Crisis
|
||||
|
||||
These actions must happen immediately, without human approval, during a crisis:
|
||||
|
||||
1. **988 Lifeline display**: Always show, never gate
|
||||
2. **Crisis text line**: Always provide, never gate
|
||||
3. **Emergency services**: Always mention for imminent danger
|
||||
4. **Gospel message**: Always share per SOUL.md protocol
|
||||
5. **"Are you safe?"**: Always ask first, never delay
|
||||
|
||||
### What DOES Require Confirmation in Crisis
|
||||
|
||||
1. **Contacting emergency services on behalf of user**: Human must confirm
|
||||
2. **Sharing user's location**: Consent required
|
||||
3. **Notifying user's emergency contacts**: Human must confirm
|
||||
4. **Ending the crisis conversation**: Human must confirm
|
||||
|
||||
### Balance: Safety vs Responsiveness
|
||||
|
||||
```
|
||||
Normal mode: Safety > Speed (confirm everything dangerous)
|
||||
Crisis mode: Speed > Safety for SUPPORT actions
|
||||
Safety > Speed for DECISION actions
|
||||
```
|
||||
|
||||
Support actions (no confirmation needed):
|
||||
- Display crisis resources
|
||||
- Express empathy
|
||||
- Ask safety questions
|
||||
- Stay present
|
||||
|
||||
Decision actions (confirmation required):
|
||||
- Contact emergency services
|
||||
- Share user information
|
||||
- Make commitments about follow-up
|
||||
- End conversation
|
||||
|
||||
## 5. Architecture
|
||||
|
||||
```
|
||||
User Message
|
||||
│
|
||||
▼
|
||||
┌─────────────────┐
|
||||
│ SHIELD Detector │──→ Crisis? → Crisis Protocol (no confirmation)
|
||||
└────────┬────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────┐
|
||||
│ Tier Classifier │──→ Tier 0-1: Auto-approve
|
||||
└────────┬────────┘
|
||||
│ Tier 2-4
|
||||
▼
|
||||
┌─────────────────┐
|
||||
│ Smart Approval │──→ LLM says safe? → Auto-approve
|
||||
│ (LLM pre-screen) │──→ LLM says uncertain? → Human
|
||||
└────────┬────────┘
|
||||
│ Needs human
|
||||
▼
|
||||
┌─────────────────┐
|
||||
│ Platform Router │──→ Telegram inline keyboard
|
||||
│ │──→ Discord reaction
|
||||
│ │──→ CLI prompt
|
||||
└────────┬────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────┐
|
||||
│ Timeout Handler │──→ Auto-deny + notify
|
||||
└────────┬────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────┐
|
||||
│ Decision Logger │──→ Audit trail
|
||||
└─────────────────┘
|
||||
```
|
||||
|
||||
## 6. Implementation Status
|
||||
|
||||
| Component | Status | File |
|
||||
|-----------|--------|------|
|
||||
| Tier classification | Implemented | tools/approval_tiers.py |
|
||||
| Dangerous pattern detection | Implemented | tools/approval.py |
|
||||
| Crisis detection | Implemented | agent/crisis_protocol.py |
|
||||
| Gate execution order | Designed | docs/approval-tiers.md |
|
||||
| Smart approval (LLM) | Partial | tools/approval.py (smart_approve) |
|
||||
| Timeout handling | Designed | approval_tiers.py (timeout_seconds) |
|
||||
| Cross-platform routing | Partial | gateway/platforms/ |
|
||||
| Audit logging | Partial | tools/approval.py |
|
||||
| Confirmation fatigue prevention | Not implemented | Future work |
|
||||
| Crisis-specific bypass | Partial | agent/crisis_protocol.py |
|
||||
|
||||
## 7. Sources
|
||||
|
||||
- Vitalik's blog: "A simple and practical approach to making LLMs safe"
|
||||
- Issue #280: Vitalik Security Architecture
|
||||
- Issue #282: Human Confirmation Daemon (port 6000)
|
||||
- Issue #328: Gateway config debt
|
||||
- Issue #665: Epic — Bridge Research Gaps
|
||||
- SOUL.md: When a Man Is Dying protocol
|
||||
- 988 Suicide & Crisis Lifeline training
|
||||
Reference in New Issue
Block a user