Timmy_Foundation/the-nexus
16
1
Fork 2
Code Issues 102 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

[POKA-YOKE][BEZALEL] Tests: Make test skips/flakes impossible to ignore #1094

New Issue
Closed
opened 2026-04-07 14:21:23 +00:00 by Timmy · 1 comment
Timmy commented 2026-04-07 14:21:23 +00:00
Owner
Copy Link

Alexander's philosophy: smoke tests + bare green-path e2e only, runtime < 5 min. Poka-yoke ensures we cannot accidentally bypass or mask test failures.

Principles:

  • Prevention: test runner configured to fail on skipped tests (no silent opt-outs).
  • Detection: flake detection algorithm flags tests with non-deterministic pass/fail.
  • Correction: flaky tests automatically quarantined and assigned; main branch stays green.

Acceptance Criteria:

  • pytest configured to treat skips as failures unless explicitly annotated with issue link
  • CI tracks test pass/fail history and flags any test with < 95% consistency
  • Quarantine process documented: flaky test moved to quarantine/ directory + issue filed
Alexander's philosophy: smoke tests + bare green-path e2e only, runtime < 5 min. Poka-yoke ensures we cannot accidentally bypass or mask test failures. **Principles:** - Prevention: test runner configured to fail on skipped tests (no silent opt-outs). - Detection: flake detection algorithm flags tests with non-deterministic pass/fail. - Correction: flaky tests automatically quarantined and assigned; main branch stays green. **Acceptance Criteria:** - [ ] pytest configured to treat skips as failures unless explicitly annotated with issue link - [ ] CI tracks test pass/fail history and flags any test with < 95% consistency - [ ] Quarantine process documented: flaky test moved to quarantine/ directory + issue filed
claude self-assigned this 2026-04-07 14:33:43 +00:00
claude commented 2026-04-07 14:38:17 +00:00
Member
Copy Link

PR #1104 created: #1104

All three acceptance criteria addressed:

  1. pytest treats skips as failures — conftest.py hooks pytest_runtest_makereport to intercept skipped tests and convert them to failures unless the reason contains an issue link (#NNN, URL, JIRA key). Env-var-gated skipif conditions and @pytest.mark.quarantine tests are exempt.

  2. Flake detection — scripts/flake_detector.py ingests pytest --json-report output, maintains a rolling .test-history.json, and exits non-zero when any test drops below 95% consistency. Requires ≥5 runs before judging a test.

  3. Quarantine process documented — tests/quarantine/ directory for holding flaky tests; docs/QUARANTINE_PROCESS.md documents the full Prevention→Detection→Correction workflow with a 30-day escalation rule.

Bonus: pytest.ini added to centralise --strict-markers, asyncio_mode, and testpaths.

PR #1104 created: https://forge.alexanderwhitestone.com/Timmy_Foundation/the-nexus/pulls/1104 All three acceptance criteria addressed: 1. **pytest treats skips as failures** — `conftest.py` hooks `pytest_runtest_makereport` to intercept skipped tests and convert them to failures unless the reason contains an issue link (`#NNN`, URL, JIRA key). Env-var-gated `skipif` conditions and `@pytest.mark.quarantine` tests are exempt. 2. **Flake detection** — `scripts/flake_detector.py` ingests `pytest --json-report` output, maintains a rolling `.test-history.json`, and exits non-zero when any test drops below 95% consistency. Requires ≥5 runs before judging a test. 3. **Quarantine process documented** — `tests/quarantine/` directory for holding flaky tests; `docs/QUARANTINE_PROCESS.md` documents the full Prevention→Detection→Correction workflow with a 30-day escalation rule. Bonus: `pytest.ini` added to centralise `--strict-markers`, `asyncio_mode`, and `testpaths`.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
222-epic
3d-world
CI
Continuous integration, runners, workflow issues
 QA
Quality assurance, testing, and production audit
 actionable
agent-presence
aistudio-ready
assigned-aistudio
assigned-claude
assigned-claw-code
Queued for Code Claw (qwen/openrouter)
 assigned-gemini
assigned-groq
assigned-kimi
Dispatched to Kimi via OpenClaw
 assigned-kimi
assigned-perplexity
assigned-sonnet
blocked
Blocked by external dependency or merge conflict
 claude-ready
claw-code-done
Code Claw completed this task
 claw-code-in-progress
Code Claw is actively working
 deprioritized
duplicate
epic
Epic / umbrella issue
 gemini-api
Gemini API integration
 gemini-review
google-ai-ultra
Google AI Ultra integration work
 groq-ready
harness
identity
Timmy identity and branding
 infrastructure
kimi-done
Kimi completed this task
 kimi-in-progress
Kimi is actively working on this
 kimi-ready
lazzyPit
Lazarus Pit — automated agent resurrection and health recovery
 media-gen
AI media generation (image/video/audio)
 modularization
needs-design
nostr
p0-critical
p1-important
p2-backlog
performance
perplexity-ready
portal
research
Deep research and planning tasks
 security
Security hardening, vulnerability fixes, access control
 sonnet-ready
sovereignty
velocity-engine
Auto-generated by velocity engine
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
KimiClaw Rockachopa Timmy allegro antigravity bezalel claude claw-code codex-agent ezra gemini google grok groq hermes kimi manus perplexity sonnet
No Assignees claude
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Timmy_Foundation/the-nexus#1094
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?