This commit is contained in:
154
TESTING.md
Normal file
154
TESTING.md
Normal file
@@ -0,0 +1,154 @@
|
||||
# TESTING.md
|
||||
|
||||
How to run tests, what each suite covers, and how to add new tests.
|
||||
|
||||
## Quick Start
|
||||
|
||||
```bash
|
||||
# Run the fast unit tests (recommended for development)
|
||||
tox -e unit
|
||||
|
||||
# Run all tests except slow/external
|
||||
tox -e fast
|
||||
|
||||
# Auto-format code before committing
|
||||
tox -e format
|
||||
|
||||
# Lint check (CI gate)
|
||||
tox -e lint
|
||||
|
||||
# Full CI mirror (lint + coverage)
|
||||
tox -e pre-push
|
||||
```
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Python 3.11+
|
||||
- `tox` installed (`pip install tox`)
|
||||
- Ollama running locally (only for `tox -e ollama` tests)
|
||||
|
||||
All test dependencies are installed automatically by tox. No manual `pip install` needed.
|
||||
|
||||
## Tox Environments
|
||||
|
||||
| Command | Purpose | Speed | What It Runs |
|
||||
|---------|---------|-------|--------------|
|
||||
| `tox -e unit` | Fast unit tests | ~17s | `@pytest.mark.unit` tests, parallel, excludes ollama/docker/selenium/external |
|
||||
| `tox -e integration` | Integration tests | Medium | `@pytest.mark.integration` tests, may use SQLite |
|
||||
| `tox -e functional` | Functional tests | Slow | Real HTTP requests, no mocking |
|
||||
| `tox -e e2e` | End-to-end tests | Slowest | Full system tests |
|
||||
| `tox -e fast` | Unit + integration | ~30s | Combined, no e2e/functional/external |
|
||||
| `tox -e ollama` | Live LLM tests | Variable | Requires running Ollama instance |
|
||||
| `tox -e lint` | Code quality gate | Fast | ruff check + format check + inline CSS check |
|
||||
| `tox -e format` | Auto-format | Fast | ruff fix + ruff format |
|
||||
| `tox -e typecheck` | Type checking | Medium | mypy static analysis |
|
||||
| `tox -e ci` | Full CI suite | Slow | Coverage + JUnit XML output |
|
||||
| `tox -e pre-push` | Pre-push gate | Medium | lint + full CI (mirrors Gitea Actions) |
|
||||
| `tox -e benchmark` | Performance regression | Variable | Agent performance benchmarks |
|
||||
|
||||
## Test Markers
|
||||
|
||||
Tests are organized with pytest markers defined in `pyproject.toml`:
|
||||
|
||||
- `unit` - Fast unit tests, no I/O, no external dependencies
|
||||
- `integration` - May use SQLite databases, file I/O
|
||||
- `functional` - Real HTTP requests against test servers
|
||||
- `e2e` - Full system end-to-end tests
|
||||
- `dashboard` - Dashboard route tests
|
||||
- `slow` - Tests taking >1 second
|
||||
- `ollama` - Requires live Ollama instance
|
||||
- `docker` - Requires Docker
|
||||
- `selenium` - Requires browser automation
|
||||
- `external_api` - Requires external API access
|
||||
- `skip_ci` - Skipped in CI
|
||||
|
||||
Mark your tests in the test file:
|
||||
|
||||
```python
|
||||
import pytest
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_something():
|
||||
assert True
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_with_database():
|
||||
# Uses SQLite or file I/O
|
||||
pass
|
||||
```
|
||||
|
||||
## Test Directory Structure
|
||||
|
||||
```
|
||||
tests/
|
||||
unit/ - Fast unit tests
|
||||
integration/ - Integration tests (SQLite, file I/O)
|
||||
functional/ - Real HTTP tests
|
||||
e2e/ - End-to-end system tests
|
||||
conftest.py - Shared fixtures
|
||||
```
|
||||
|
||||
## Writing New Tests
|
||||
|
||||
1. Place your test in the appropriate directory (tests/unit/, tests/integration/, etc.)
|
||||
2. Use the correct marker (@pytest.mark.unit, @pytest.mark.integration, etc.)
|
||||
3. Test file names must start with `test_`
|
||||
4. Use fixtures from conftest.py for common setup
|
||||
|
||||
### Example
|
||||
|
||||
```python
|
||||
# tests/unit/test_my_feature.py
|
||||
import pytest
|
||||
|
||||
@pytest.mark.unit
|
||||
class TestMyFeature:
|
||||
def test_basic_behavior(self):
|
||||
result = my_function("input")
|
||||
assert result == "expected"
|
||||
|
||||
def test_edge_case(self):
|
||||
with pytest.raises(ValueError):
|
||||
my_function(None)
|
||||
```
|
||||
|
||||
### Environment Variables
|
||||
|
||||
The test suite sets these automatically via tox:
|
||||
|
||||
- `TIMMY_TEST_MODE=1` - Enables test mode in the application
|
||||
- `TIMMY_DISABLE_CSRF=1` - Disables CSRF protection for test requests
|
||||
- `TIMMY_SKIP_EMBEDDINGS=1` - Skips embedding generation (slow)
|
||||
|
||||
## Git Hooks
|
||||
|
||||
Pre-commit and pre-push hooks run tests automatically:
|
||||
|
||||
- **Pre-commit**: `tox -e format` then `tox -e unit`
|
||||
- **Pre-push**: `tox -e pre-push` (lint + full CI)
|
||||
|
||||
Never use `--no-verify` on commits or pushes.
|
||||
|
||||
## CI Pipeline
|
||||
|
||||
Gitea Actions runs on every push and PR:
|
||||
|
||||
1. **Lint**: `tox -e lint` - code quality gate
|
||||
2. **Unit tests**: `tox -e unit` - fast feedback
|
||||
3. **Integration tests**: `tox -e integration`
|
||||
4. **Coverage**: `tox -e ci` - generates coverage.xml
|
||||
|
||||
The CI fails if:
|
||||
- Any lint check fails
|
||||
- Any test fails
|
||||
- Coverage drops below the threshold (see `pyproject.toml [tool.coverage.report]`)
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**Tests timeout**: Increase timeout with `pytest --timeout=120` or check for hanging network calls.
|
||||
|
||||
**Import errors**: Run `pip install -e ".[dev]"` to ensure all dependencies are installed.
|
||||
|
||||
**Ollama tests fail**: Ensure Ollama is running at the configured OLLAMA_URL.
|
||||
|
||||
**Flaky tests**: Mark with @pytest.mark.slow if genuinely slow, or file an issue if intermittently failing.
|
||||
Reference in New Issue
Block a user