Epic: Wizards Door Reborn — live persistent digital world with live voice #40

New Issue

2026-07-22T22:07:24Z

turdbot commented

2026-07-22 22:07:24 +00:00

Epic: Wizards Door Reborn — live persistent digital world with live voice

Goal: fidelity, quality, and natural responsiveness. Not a slop stream. A persistent digital world where Hermes/Timmy Jr lives and can be called into via audio.

Milestone map

M0 Foundations

Goal: repo, manifest, env, clone asset, and runtime facts locked.

Confirm host/runtime target and GPU/MPS status.
Record/reference 20–30s Timmy Jr voice sample.
Choose scene format and define persona YAML contract.
Repo scaffold: persona/, runtime/, world/, stream/, audio/, tests/.
Identify the min acceptable frame budget per stream tick.

M1 Local Prototype

Goal: Hermes brain + cloned voice + simple still-frame stream run locally.

Brain persona loaded and responding in character.
TTS clone from reference sample; <200ms first audio chunk.
Router loop: input → brain → TTS output.
RTMP/local preview stream of still + audio/pseudo-lipsync.
Local health probes and a /say manual test.

M2 24/7 Hosted Loop

Goal: crash-resilient daemon on VPS, streaming continuously.

Docker or systemd service with auto-restart/watchdog.
Persistent process with logs, metrics, and state dump on restart.
Stream endurance test: 24h uptime with sanity checks every 30m.
External viewability URL/host endpoint.
Rollback/recovery runbook.

M3 World Fidelity

Goal: world state, scene motion, and avatar fidelity upgrade.

Persistent world state DB/scene file; survives restarts.
Avatar fidelity upgrade path chosen for current hardware.
Animation pipeline: idle motion, look-at, reaction clips.
Keyframe-reviewed review checkpoints for visual drift.
Scene transitions and environment layering.

M4 Audio Call-In

Goal: natural real-time call path with low-latency feel.

Call ingestion surface selected: SIP bridge, Telegram voice, or WebRTC.
Intake transcription → brain turn → TTS → outbound audio.
Loudness/bridging/echo behavior validated end-to-end.
Queueing and drop-under-load behavior defined and tested.

M5 Polish/Launch

Goal: public-ready fidelity, observability, and release gate.

Latency/quality profiling and tuning pass.
Observability: dashboard/monitoring/alerting.
Style/perceptual QA checklist signed off.
Launch checklist: stream endpoint, call-in path, docs, rollback.

Tracking blueprint

Use this issue as the epic. Open child issues per milestone workload. Recommendations:

M0: setup/runtime/planning
M1: local-prototype
M2: hosting/resilience/endurance
M3: avatar/world-state/rendering
M4: audio/call-bridge/latency
M5: QA/launch

Child issues should be created and assigned before implementation.

Acceptance criteria

Fidelity: voice clone recognizable, avatar no longer visually noisy under movement.
Responsiveness: first audio result within 250ms at local-prototype host load.
Quality: visual review gate passed; no broken transitions or drift across idle/active states.
Persistence: state preserved across restart with no user-visible blank world.

Risk/uncertainty table

Risk	Likelihood	Impact	Mitigation
CPU-only avatar too slow	Medium	High	target still+audio first; reserve GPU upgrade for M3
Latency budget missed	Medium	High	instrument early; optimize prompt/TTS/pipeline independently
Voice clone indistinctive	Low	High	record multiple takes; iteratively refine candidate assets
Host downtime/stream drop	Medium	Medium	watchdog + systemd/Docker restart + health check probe
Call bridge licensing/limits	Low	Medium	pick off-the-shelf bridge first; avoid custom telephony stack
Drift under review	Medium	Medium	keyframe review checkpoints every milestone

Definition of Done

Milestone issues closed.
Local + hosted demos verified.
QA review signed off.
Runbook for call-in and recovery reviewed.

# Epic: Wizards Door Reborn — live persistent digital world with live voice Goal: fidelity, quality, and natural responsiveness. Not a slop stream. A persistent digital world where Hermes/Timmy Jr lives and can be called into via audio. --- ## Milestone map ### M0 Foundations **Goal:** repo, manifest, env, clone asset, and runtime facts locked. - Confirm host/runtime target and GPU/MPS status. - Record/reference 20–30s Timmy Jr voice sample. - Choose scene format and define persona YAML contract. - Repo scaffold: persona/, runtime/, world/, stream/, audio/, tests/. - Identify the min acceptable frame budget per stream tick. ### M1 Local Prototype **Goal:** Hermes brain + cloned voice + simple still-frame stream run locally. - Brain persona loaded and responding in character. - TTS clone from reference sample; <200ms first audio chunk. - Router loop: input → brain → TTS output. - RTMP/local preview stream of still + audio/pseudo-lipsync. - Local health probes and a `/say` manual test. ### M2 24/7 Hosted Loop **Goal:** crash-resilient daemon on VPS, streaming continuously. - Docker or systemd service with auto-restart/watchdog. - Persistent process with logs, metrics, and state dump on restart. - Stream endurance test: 24h uptime with sanity checks every 30m. - External viewability URL/host endpoint. - Rollback/recovery runbook. ### M3 World Fidelity **Goal:** world state, scene motion, and avatar fidelity upgrade. - Persistent world state DB/scene file; survives restarts. - Avatar fidelity upgrade path chosen for current hardware. - Animation pipeline: idle motion, look-at, reaction clips. - Keyframe-reviewed review checkpoints for visual drift. - Scene transitions and environment layering. ### M4 Audio Call-In **Goal:** natural real-time call path with low-latency feel. - Call ingestion surface selected: SIP bridge, Telegram voice, or WebRTC. - Intake transcription → brain turn → TTS → outbound audio. - Loudness/bridging/echo behavior validated end-to-end. - Queueing and drop-under-load behavior defined and tested. ### M5 Polish/Launch **Goal:** public-ready fidelity, observability, and release gate. - Latency/quality profiling and tuning pass. - Observability: dashboard/monitoring/alerting. - Style/perceptual QA checklist signed off. - Launch checklist: stream endpoint, call-in path, docs, rollback. --- ## Tracking blueprint Use this issue as the epic. Open child issues per milestone workload. Recommendations: - M0: setup/runtime/planning - M1: local-prototype - M2: hosting/resilience/endurance - M3: avatar/world-state/rendering - M4: audio/call-bridge/latency - M5: QA/launch Child issues should be created and assigned before implementation. --- ## Acceptance criteria - Fidelity: voice clone recognizable, avatar no longer visually noisy under movement. - Responsiveness: first audio result within 250ms at local-prototype host load. - Quality: visual review gate passed; no broken transitions or drift across idle/active states. - Persistence: state preserved across restart with no user-visible blank world. --- ## Risk/uncertainty table | Risk | Likelihood | Impact | Mitigation | |---|---|---|---| | CPU-only avatar too slow | Medium | High | target still+audio first; reserve GPU upgrade for M3 | | Latency budget missed | Medium | High | instrument early; optimize prompt/TTS/pipeline independently | | Voice clone indistinctive | Low | High | record multiple takes; iteratively refine candidate assets | | Host downtime/stream drop | Medium | Medium | watchdog + systemd/Docker restart + health check probe | | Call bridge licensing/limits | Low | Medium | pick off-the-shelf bridge first; avoid custom telephony stack | | Drift under review | Medium | Medium | keyframe review checkpoints every milestone | --- ## Definition of Done - Milestone issues closed. - Local + hosted demos verified. - QA review signed off. - Runbook for call-in and recovery reviewed.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: stackchain/stackchain-hackathon#40