forked from Rockachopa/Timmy-time-dashboard
354 lines
13 KiB
Markdown
354 lines
13 KiB
Markdown
# Bannerlord Feudal Multi-Agent Hierarchy Design
|
||
|
||
**Issue:** #1099
|
||
**Parent Epic:** #1091 (Project Bannerlord)
|
||
**Date:** 2026-03-23
|
||
**Status:** Draft
|
||
|
||
---
|
||
|
||
## Overview
|
||
|
||
This document specifies the multi-agent hierarchy for Timmy's Bannerlord campaign.
|
||
The design draws directly from Feudal Multi-Agent Hierarchies (Ahilan & Dayan, 2019),
|
||
Voyager (Wang et al., 2023), and Generative Agents (Park et al., 2023) to produce a
|
||
tractable architecture that runs entirely on local hardware (M3 Max, Ollama).
|
||
|
||
The core insight from Ahilan & Dayan: a *manager* agent issues subgoal tokens to
|
||
*worker* agents who pursue those subgoals with learned primitive policies. Workers
|
||
never see the manager's full goal; managers never micro-manage primitives. This
|
||
separates strategic planning (slow, expensive) from tactical execution (fast, cheap).
|
||
|
||
---
|
||
|
||
## 1. King-Level Timmy — Subgoal Vocabulary
|
||
|
||
Timmy is the King agent. He operates on the **campaign map** timescale (days to weeks
|
||
of in-game time). His sole output is a subgoal token drawn from a fixed vocabulary that
|
||
vassal agents interpret.
|
||
|
||
### Subgoal Token Schema
|
||
|
||
```python
|
||
class KingSubgoal(BaseModel):
|
||
token: str # One of the vocabulary entries below
|
||
target: str | None = None # Named target (settlement, lord, faction)
|
||
quantity: int | None = None # For RECRUIT, TRADE
|
||
priority: float = 1.0 # 0.0–2.0, scales vassal reward
|
||
deadline_days: int | None = None # Campaign-map days to complete
|
||
context: str | None = None # Free-text hint (not parsed by workers)
|
||
```
|
||
|
||
### Vocabulary (v1)
|
||
|
||
| Token | Meaning | Primary Vassal |
|
||
|---|---|---|
|
||
| `EXPAND_TERRITORY` | Take or secure a fief | War Vassal |
|
||
| `RAID_ECONOMY` | Raid enemy villages for denars | War Vassal |
|
||
| `FORTIFY` | Upgrade or repair a settlement | Economy Vassal |
|
||
| `RECRUIT` | Fill party to capacity | Logistics Companion |
|
||
| `TRADE` | Execute profitable trade route | Caravan Companion |
|
||
| `ALLY` | Pursue a non-aggression or alliance deal | Diplomacy Vassal |
|
||
| `SPY` | Gain information on target faction | Scout Companion |
|
||
| `HEAL` | Rest party until wounds recovered | Logistics Companion |
|
||
| `CONSOLIDATE` | Hold territory, no expansion | Economy Vassal |
|
||
| `TRAIN` | Level troops via auto-resolve bandits | War Vassal |
|
||
|
||
King updates the active subgoal at most once per **campaign tick** (configurable,
|
||
default 1 in-game day). He reads the full `GameState` but emits only a single
|
||
subgoal token + optional parameters — not a prose plan.
|
||
|
||
### King Decision Loop
|
||
|
||
```
|
||
while campaign_running:
|
||
state = gabs.get_state() # Full kingdom + map snapshot
|
||
subgoal = king_llm.decide(state) # Qwen3:32b, temp=0.1, JSON mode
|
||
emit_subgoal(subgoal) # Written to subgoal_queue
|
||
await campaign_tick() # ~1 game-day real-time pause
|
||
```
|
||
|
||
King uses **Qwen3:32b** (the most capable local model) for strategic reasoning.
|
||
Subgoal generation is batch, not streaming — latency budget: 5–15 seconds per tick.
|
||
|
||
---
|
||
|
||
## 2. Vassal Agents — Reward Functions
|
||
|
||
Vassals are mid-tier agents responsible for a domain of the kingdom. Each vassal
|
||
has a defined reward function. Vassals run on **Qwen3:14b** (balanced capability
|
||
vs. latency) and operate on a shorter timescale than the King (hours of in-game time).
|
||
|
||
### 2a. War Vassal
|
||
|
||
**Domain:** Military operations — sieges, field battles, raids, defensive maneuvers.
|
||
|
||
**Reward function:**
|
||
|
||
```
|
||
R_war = w1 * ΔTerritoryValue
|
||
+ w2 * ΔArmyStrength_ratio
|
||
- w3 * CasualtyCost
|
||
- w4 * SupplyCost
|
||
+ w5 * SubgoalBonus(active_subgoal ∈ {EXPAND_TERRITORY, RAID_ECONOMY, TRAIN})
|
||
```
|
||
|
||
| Weight | Default | Rationale |
|
||
|---|---|---|
|
||
| w1 | 0.40 | Territory is the primary long-term asset |
|
||
| w2 | 0.25 | Army ratio relative to nearest rival |
|
||
| w3 | 0.20 | Casualties are expensive to replace |
|
||
| w4 | 0.10 | Supply burn limits campaign duration |
|
||
| w5 | 0.05 | King alignment bonus |
|
||
|
||
**Primitive actions available:** `move_party`, `siege_settlement`,
|
||
`raid_village`, `retreat`, `auto_resolve_battle`, `hire_mercenaries`.
|
||
|
||
### 2b. Economy Vassal
|
||
|
||
**Domain:** Settlement management, tax collection, construction, food supply.
|
||
|
||
**Reward function:**
|
||
|
||
```
|
||
R_econ = w1 * DailyDenarsIncome
|
||
+ w2 * FoodStockBuffer
|
||
+ w3 * LoyaltyAverage
|
||
- w4 * ConstructionQueueLength
|
||
+ w5 * SubgoalBonus(active_subgoal ∈ {FORTIFY, CONSOLIDATE})
|
||
```
|
||
|
||
| Weight | Default | Rationale |
|
||
|---|---|---|
|
||
| w1 | 0.35 | Income is the fuel for everything |
|
||
| w2 | 0.25 | Starvation causes immediate loyalty crash |
|
||
| w3 | 0.20 | Low loyalty triggers revolt |
|
||
| w4 | 0.15 | Idle construction is opportunity cost |
|
||
| w5 | 0.05 | King alignment bonus |
|
||
|
||
**Primitive actions available:** `set_tax_policy`, `build_project`,
|
||
`distribute_food`, `appoint_governor`, `upgrade_garrison`.
|
||
|
||
### 2c. Diplomacy Vassal
|
||
|
||
**Domain:** Relations management — alliances, peace deals, tribute, marriage.
|
||
|
||
**Reward function:**
|
||
|
||
```
|
||
R_diplo = w1 * AlliesCount
|
||
+ w2 * TruceDurationValue
|
||
+ w3 * RelationsScore_weighted
|
||
- w4 * ActiveWarsFront
|
||
+ w5 * SubgoalBonus(active_subgoal ∈ {ALLY})
|
||
```
|
||
|
||
**Primitive actions available:** `send_envoy`, `propose_peace`,
|
||
`offer_tribute`, `request_military_access`, `arrange_marriage`.
|
||
|
||
---
|
||
|
||
## 3. Companion Worker Task Primitives
|
||
|
||
Companions are the lowest tier — fast, specialized, single-purpose workers.
|
||
They run on **Qwen3:8b** (or smaller) for sub-2-second response times.
|
||
Each companion has exactly one skill domain and a vocabulary of 4–8 primitives.
|
||
|
||
### 3a. Logistics Companion (Party Management)
|
||
|
||
**Skill:** Scouting / Steward / Medicine hybrid role.
|
||
|
||
| Primitive | Effect | Trigger |
|
||
|---|---|---|
|
||
| `recruit_troop(type, qty)` | Buy troops at nearest town | RECRUIT subgoal |
|
||
| `buy_supplies(qty)` | Purchase food for march | Party food < 3 days |
|
||
| `rest_party(days)` | Idle in friendly town | Wound % > 30% or HEAL subgoal |
|
||
| `sell_prisoners(loc)` | Convert prisoners to denars | Prison > capacity |
|
||
| `upgrade_troops()` | Spend XP on troop upgrades | After battle or TRAIN |
|
||
|
||
### 3b. Caravan Companion (Trade)
|
||
|
||
**Skill:** Trade / Charm.
|
||
|
||
| Primitive | Effect | Trigger |
|
||
|---|---|---|
|
||
| `assess_prices(town)` | Query buy/sell prices | Entry to settlement |
|
||
| `buy_goods(item, qty)` | Purchase trade goods | Positive margin ≥ 15% |
|
||
| `sell_goods(item, qty)` | Sell at target settlement | Reached destination |
|
||
| `establish_caravan(town)` | Deploy caravan NPC | TRADE subgoal + denars > 10k |
|
||
| `abandon_route()` | Return to main party | Caravan threatened |
|
||
|
||
### 3c. Scout Companion (Intelligence)
|
||
|
||
**Skill:** Scouting / Roguery.
|
||
|
||
| Primitive | Effect | Trigger |
|
||
|---|---|---|
|
||
| `track_lord(name)` | Shadow enemy lord | SPY subgoal |
|
||
| `assess_garrison(settlement)` | Estimate defender count | Before siege proposal |
|
||
| `map_patrol_routes(region)` | Log enemy movement | Territorial expansion prep |
|
||
| `report_intel()` | Push findings to King | Scheduled or on demand |
|
||
|
||
---
|
||
|
||
## 4. Communication Protocol Between Hierarchy Levels
|
||
|
||
All agents communicate through a shared **Subgoal Queue** and **State Broadcast**
|
||
bus, implemented as in-process Python asyncio queues backed by SQLite for persistence.
|
||
|
||
### Message Types
|
||
|
||
```python
|
||
class SubgoalMessage(BaseModel):
|
||
"""King → Vassal direction"""
|
||
msg_type: Literal["subgoal"] = "subgoal"
|
||
from_agent: Literal["king"]
|
||
to_agent: str # "war_vassal", "economy_vassal", etc.
|
||
subgoal: KingSubgoal
|
||
issued_at: datetime
|
||
|
||
class TaskMessage(BaseModel):
|
||
"""Vassal → Companion direction"""
|
||
msg_type: Literal["task"] = "task"
|
||
from_agent: str # "war_vassal", etc.
|
||
to_agent: str # "logistics_companion", etc.
|
||
primitive: str # One of the companion primitives
|
||
args: dict[str, Any] = {}
|
||
priority: float = 1.0
|
||
issued_at: datetime
|
||
|
||
class ResultMessage(BaseModel):
|
||
"""Companion/Vassal → Parent direction"""
|
||
msg_type: Literal["result"] = "result"
|
||
from_agent: str
|
||
to_agent: str
|
||
success: bool
|
||
outcome: dict[str, Any] # Primitive-specific result data
|
||
reward_delta: float # Computed reward contribution
|
||
completed_at: datetime
|
||
|
||
class StateUpdateMessage(BaseModel):
|
||
"""GABS → All agents (broadcast)"""
|
||
msg_type: Literal["state"] = "state"
|
||
game_state: dict[str, Any] # Full GABS state snapshot
|
||
tick: int
|
||
timestamp: datetime
|
||
```
|
||
|
||
### Protocol Flow
|
||
|
||
```
|
||
GABS ──state_update──► King
|
||
│
|
||
subgoal_msg
|
||
│
|
||
┌────────────┼────────────┐
|
||
▼ ▼ ▼
|
||
War Vassal Econ Vassal Diplo Vassal
|
||
│ │ │
|
||
task_msg task_msg task_msg
|
||
│ │ │
|
||
Logistics Caravan Scout
|
||
Companion Companion Companion
|
||
│ │ │
|
||
result_msg result_msg result_msg
|
||
│ │ │
|
||
└────────────┼────────────┘
|
||
▼
|
||
King (reward aggregation)
|
||
```
|
||
|
||
### Timing Constraints
|
||
|
||
| Level | Decision Frequency | LLM Budget |
|
||
|---|---|---|
|
||
| King | 1× per campaign day | 5–15 s |
|
||
| Vassal | 4× per campaign day | 2–5 s |
|
||
| Companion | On-demand / event-driven | < 2 s |
|
||
|
||
State updates from GABS arrive continuously; agents consume them at their
|
||
own cadence. No agent blocks another's queue.
|
||
|
||
### Conflict Resolution
|
||
|
||
If two vassals propose conflicting actions (e.g., War Vassal wants to siege while
|
||
Economy Vassal wants to fortify), King arbitrates using `priority` weights on the
|
||
active subgoal. The highest-priority active subgoal wins resource contention.
|
||
|
||
---
|
||
|
||
## 5. Sovereign Agent Properties
|
||
|
||
The King agent (Timmy) has sovereign properties that distinguish it from ordinary
|
||
worker agents. These map directly to Timmy's existing identity architecture.
|
||
|
||
### 5a. Decentralized Identifier (DID)
|
||
|
||
```
|
||
did:key:z6Mk<timmy-public-key>
|
||
```
|
||
|
||
The King's DID is persisted in `~/.timmy/identity.json` (existing SOUL.md pattern).
|
||
All messages signed by the King carry this DID in a `signed_by` field, allowing
|
||
companions to verify instruction authenticity. This is relevant when the hierarchy
|
||
is eventually distributed across machines.
|
||
|
||
### 5b. Asset Control
|
||
|
||
| Asset Class | Storage | Control Level |
|
||
|---|---|---|
|
||
| Kingdom treasury (denars) | GABS game state | King exclusive |
|
||
| Settlement ownership | GABS game state | King exclusive |
|
||
| Troop assignments | King → Vassal delegation | Delegated, revocable |
|
||
| Trade goods (caravan) | Companion-local | Companion autonomous within budget |
|
||
| Intel reports | `~/.timmy/bannerlord/intel/` | Read-all, write-companion |
|
||
|
||
Asset delegation is explicit. Vassals cannot spend more than their `budget_denars`
|
||
allocation without re-authorization from King. Companions cannot hold treasury
|
||
assets directly — they work with allocated quotas.
|
||
|
||
### 5c. Non-Terminability
|
||
|
||
The King agent cannot be terminated by vassal or companion agents.
|
||
Termination authority is reserved for:
|
||
1. The human operator (Ctrl+C or `timmy stop`)
|
||
2. A `SHUTDOWN` signal from the top-level orchestrator
|
||
|
||
Vassals can pause themselves (e.g., awaiting GABS state) but cannot signal the King
|
||
to stop. This prevents a misbehaving military vassal from ending the campaign.
|
||
|
||
Implementation: King runs in the main asyncio event loop. Vassals and companions
|
||
run in `asyncio.TaskGroup` subgroups. Only the King's task holds a reference to
|
||
the TaskGroup cancel scope.
|
||
|
||
---
|
||
|
||
## Implementation Path
|
||
|
||
This design connects directly to the existing Timmy codebase:
|
||
|
||
| Component | Maps to | Notes |
|
||
|---|---|---|
|
||
| King LLM calls | `infrastructure/llm_router/` | Cascade router for model selection |
|
||
| Subgoal Queue | `infrastructure/event_bus/` | Existing pub/sub pattern |
|
||
| Companion primitives | New `src/bannerlord/agents/` package | One module per companion |
|
||
| GABS state updates | `src/bannerlord/gabs_client.py` | TCP JSON-RPC, port 4825 |
|
||
| Asset ledger | `src/bannerlord/ledger.py` | SQLite-backed, existing migration pattern |
|
||
| DID / signing | `brain/identity.py` | Extends existing SOUL.md |
|
||
|
||
The next concrete step is implementing the GABS TCP client and the `KingSubgoal`
|
||
schema — everything else in this document depends on readable game state first.
|
||
|
||
---
|
||
|
||
## References
|
||
|
||
- Ahilan, S. & Dayan, P. (2019). Feudal Multi-Agent Hierarchies for Cooperative
|
||
Reinforcement Learning. https://arxiv.org/abs/1901.08492
|
||
- Rood, S. (2022). Scaling Reinforcement Learning through Feudal Hierarchy (NPS thesis).
|
||
- Wang, G. et al. (2023). Voyager: An Open-Ended Embodied Agent with Large Language
|
||
Models. https://arxiv.org/abs/2305.16291
|
||
- Park, J.S. et al. (2023). Generative Agents: Interactive Simulacra of Human Behavior.
|
||
https://arxiv.org/abs/2304.03442
|
||
- Silveira, T. (2022). CiF-Bannerlord: Social AI Integration in Bannerlord.
|