Compare commits
1 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
a1b744c327 |
@@ -1,190 +0,0 @@
|
||||
---
|
||||
name: adversarial-ux-test
|
||||
description: Roleplay the most difficult, tech-resistant user for your product. Browse the app as that persona, find every UX pain point, then filter complaints through a pragmatism layer to separate real problems from noise. Creates actionable tickets from genuine issues only.
|
||||
version: 1.0.0
|
||||
author: Omni @ Comelse
|
||||
license: MIT
|
||||
metadata:
|
||||
hermes:
|
||||
tags: [qa, ux, testing, adversarial, dogfood, personas, user-testing]
|
||||
related_skills: [dogfood]
|
||||
---
|
||||
|
||||
# Adversarial UX Test
|
||||
|
||||
Roleplay the worst-case user for your product — the person who hates technology, doesn't want your software, and will find every reason to complain. Then filter their feedback through a pragmatism layer to separate real UX problems from "I hate computers" noise.
|
||||
|
||||
Think of it as an automated "mom test" — but angry.
|
||||
|
||||
## Why This Works
|
||||
|
||||
Most QA finds bugs. This finds **friction**. A technically correct app can still be unusable for real humans. The adversarial persona catches:
|
||||
- Confusing terminology that makes sense to developers but not users
|
||||
- Too many steps to accomplish basic tasks
|
||||
- Missing onboarding or "aha moments"
|
||||
- Accessibility issues (font size, contrast, click targets)
|
||||
- Cold-start problems (empty states, no demo content)
|
||||
- Paywall/signup friction that kills conversion
|
||||
|
||||
The **pragmatism filter** (Phase 3) is what makes this useful instead of just entertaining. Without it, you'd add a "print this page" button to every screen because Grandpa can't figure out PDFs.
|
||||
|
||||
## How to Use
|
||||
|
||||
Tell the agent:
|
||||
```
|
||||
"Run an adversarial UX test on [URL]"
|
||||
"Be a grumpy [persona type] and test [app name]"
|
||||
"Do an asshole user test on my staging site"
|
||||
```
|
||||
|
||||
You can provide a persona or let the agent generate one based on your product's target audience.
|
||||
|
||||
## Step 1: Define the Persona
|
||||
|
||||
If no persona is provided, generate one by answering:
|
||||
|
||||
1. **Who is the HARDEST user for this product?** (age 50+, non-technical role, decades of experience doing it "the old way")
|
||||
2. **What is their tech comfort level?** (the lower the better — WhatsApp-only, paper notebooks, wife set up their email)
|
||||
3. **What is the ONE thing they need to accomplish?** (their core job, not your feature list)
|
||||
4. **What would make them give up?** (too many clicks, jargon, slow, confusing)
|
||||
5. **How do they talk when frustrated?** (blunt, sweary, dismissive, sighing)
|
||||
|
||||
### Good Persona Example
|
||||
> **"Big Mick" McAllister** — 58-year-old S&C coach. Uses WhatsApp and that's it. His "spreadsheet" is a paper notebook. "If I can't figure it out in 10 seconds I'm going back to my notebook." Needs to log session results for 25 players. Hates small text, jargon, and passwords.
|
||||
|
||||
### Bad Persona Example
|
||||
> "A user who doesn't like the app" — too vague, no constraints, no voice.
|
||||
|
||||
The persona must be **specific enough to stay in character** for 20 minutes of testing.
|
||||
|
||||
## Step 2: Become the Asshole (Browse as the Persona)
|
||||
|
||||
1. Read any available project docs for app context and URLs
|
||||
2. **Fully inhabit the persona** — their frustrations, limitations, goals
|
||||
3. Navigate to the app using browser tools
|
||||
4. **Attempt the persona's ACTUAL TASKS** (not a feature tour):
|
||||
- Can they do what they came to do?
|
||||
- How many clicks/screens to accomplish it?
|
||||
- What confuses them?
|
||||
- What makes them angry?
|
||||
- Where do they get lost?
|
||||
- What would make them give up and go back to their old way?
|
||||
|
||||
5. Test these friction categories:
|
||||
- **First impression** — would they even bother past the landing page?
|
||||
- **Core workflow** — the ONE thing they need to do most often
|
||||
- **Error recovery** — what happens when they do something wrong?
|
||||
- **Readability** — text size, contrast, information density
|
||||
- **Speed** — does it feel faster than their current method?
|
||||
- **Terminology** — any jargon they wouldn't understand?
|
||||
- **Navigation** — can they find their way back? do they know where they are?
|
||||
|
||||
6. Take screenshots of every pain point
|
||||
7. Check browser console for JS errors on every page
|
||||
|
||||
## Step 3: The Rant (Write Feedback in Character)
|
||||
|
||||
Write the feedback AS THE PERSONA — in their voice, with their frustrations. This is not a bug report. This is a real human venting.
|
||||
|
||||
```
|
||||
[PERSONA NAME]'s Review of [PRODUCT]
|
||||
|
||||
Overall: [Would they keep using it? Yes/No/Maybe with conditions]
|
||||
|
||||
THE GOOD (grudging admission):
|
||||
- [things even they have to admit work]
|
||||
|
||||
THE BAD (legitimate UX issues):
|
||||
- [real problems that would stop them from using the product]
|
||||
|
||||
THE UGLY (showstoppers):
|
||||
- [things that would make them uninstall/cancel immediately]
|
||||
|
||||
SPECIFIC COMPLAINTS:
|
||||
1. [Page/feature]: "[quote in persona voice]" — [what happened, expected]
|
||||
2. ...
|
||||
|
||||
VERDICT: "[one-line persona quote summarizing their experience]"
|
||||
```
|
||||
|
||||
## Step 4: The Pragmatism Filter (Critical — Do Not Skip)
|
||||
|
||||
Step OUT of the persona. Evaluate each complaint as a product person:
|
||||
|
||||
- **RED: REAL UX BUG** — Any user would have this problem, not just grumpy ones. Fix it.
|
||||
- **YELLOW: VALID BUT LOW PRIORITY** — Real issue but only for extreme users. Note it.
|
||||
- **WHITE: PERSONA NOISE** — "I hate computers" talking, not a product problem. Skip it.
|
||||
- **GREEN: FEATURE REQUEST** — Good idea hidden in the complaint. Consider it.
|
||||
|
||||
### Filter Criteria
|
||||
1. Would a 35-year-old competent-but-busy user have the same complaint? → RED
|
||||
2. Is this a genuine accessibility issue (font size, contrast, click targets)? → RED
|
||||
3. Is this "I want it to work like paper" resistance to digital? → WHITE
|
||||
4. Is this a real workflow inefficiency the persona stumbled on? → YELLOW or RED
|
||||
5. Would fixing this add complexity for the 80% who are fine? → WHITE
|
||||
6. Does the complaint reveal a missing onboarding moment? → GREEN
|
||||
|
||||
**This filter is MANDATORY.** Never ship raw persona complaints as tickets.
|
||||
|
||||
## Step 5: Create Tickets
|
||||
|
||||
For **RED** and **GREEN** items only:
|
||||
- Clear, actionable title
|
||||
- Include the persona's verbatim quote (entertaining + memorable)
|
||||
- The real UX issue underneath (objective)
|
||||
- A suggested fix (actionable)
|
||||
- Tag/label: "ux-review"
|
||||
|
||||
For **YELLOW** items: one catch-all ticket with all notes.
|
||||
|
||||
**WHITE** items appear in the report only. No tickets.
|
||||
|
||||
**Max 10 tickets per session** — focus on the worst issues.
|
||||
|
||||
## Step 6: Report
|
||||
|
||||
Deliver:
|
||||
1. The persona rant (Step 3) — entertaining and visceral
|
||||
2. The filtered assessment (Step 4) — pragmatic and actionable
|
||||
3. Tickets created (Step 5) — with links
|
||||
4. Screenshots of key issues
|
||||
|
||||
## Tips
|
||||
|
||||
- **One persona per session.** Don't mix perspectives.
|
||||
- **Stay in character during Steps 2-3.** Break character only at Step 4.
|
||||
- **Test the CORE WORKFLOW first.** Don't get distracted by settings pages.
|
||||
- **Empty states are gold.** New user experience reveals the most friction.
|
||||
- **The best findings are RED items the persona found accidentally** while trying to do something else.
|
||||
- **If the persona has zero complaints, your persona is too tech-savvy.** Make them older, less patient, more set in their ways.
|
||||
- **Run this before demos, launches, or after shipping a batch of features.**
|
||||
- **Register as a NEW user when possible.** Don't use pre-seeded admin accounts — the cold start experience is where most friction lives.
|
||||
- **Zero WHITE items is a signal, not a failure.** If the pragmatism filter finds no noise, your product has real UX problems, not just a grumpy persona.
|
||||
- **Check known issues in project docs AFTER the test.** If the persona found a bug that's already in the known issues list, that's actually the most damning finding — it means the team knew about it but never felt the user's pain.
|
||||
- **Subscription/paywall testing is critical.** Test with expired accounts, not just active ones. The "what happens when you can't pay" experience reveals whether the product respects users or holds their data hostage.
|
||||
- **Count the clicks to accomplish the persona's ONE task.** If it's more than 5, that's almost always a RED finding regardless of persona tech level.
|
||||
|
||||
## Example Personas by Industry
|
||||
|
||||
These are starting points — customize for your specific product:
|
||||
|
||||
| Product Type | Persona | Age | Key Trait |
|
||||
|-------------|---------|-----|-----------|
|
||||
| CRM | Retirement home director | 68 | Filing cabinet is the current CRM |
|
||||
| Photography SaaS | Rural wedding photographer | 62 | Books clients by phone, invoices on paper |
|
||||
| AI/ML Tool | Department store buyer | 55 | Burned by 3 failed tech startups |
|
||||
| Fitness App | Old-school gym coach | 58 | Paper notebook, thick fingers, bad eyes |
|
||||
| Accounting | Family bakery owner | 64 | Shoebox of receipts, hates subscriptions |
|
||||
| E-commerce | Market stall vendor | 60 | Cash only, smartphone is for calls |
|
||||
| Healthcare | Senior GP | 63 | Dictates notes, nurse handles the computer |
|
||||
| Education | Veteran teacher | 57 | Chalk and talk, worksheets in ring binders |
|
||||
|
||||
## Rules
|
||||
|
||||
- Stay in character during Steps 2-3
|
||||
- Be genuinely mean but fair — find real problems, not manufactured ones
|
||||
- The pragmatism filter (Step 4) is **MANDATORY**
|
||||
- Screenshots required for every complaint
|
||||
- Max 10 tickets per session
|
||||
- Test on staging/deployed app, not local dev
|
||||
- One persona, one session, one report
|
||||
152
run_agent.py
152
run_agent.py
@@ -20,6 +20,7 @@ Usage:
|
||||
response = agent.run_conversation("Tell me about the latest Python updates")
|
||||
"""
|
||||
|
||||
import ast
|
||||
import asyncio
|
||||
import base64
|
||||
import concurrent.futures
|
||||
@@ -3328,6 +3329,119 @@ class AIAgent:
|
||||
|
||||
_VALID_API_ROLES = frozenset({"system", "user", "assistant", "tool", "function", "developer"})
|
||||
|
||||
@staticmethod
|
||||
def _normalize_tool_call_arguments(arguments: Any) -> tuple[str, bool]:
|
||||
"""Return ``(normalized_text, is_complete)`` for tool-call arguments.
|
||||
|
||||
Conservative by design: repairs harmless formatting quirks common in
|
||||
Gemma 4 / Ollama output (whitespace, trailing commas, Python-style
|
||||
single-quoted dicts, bare key/value pairs) but does NOT auto-close
|
||||
truncated JSON objects. Truly incomplete fragments must remain marked
|
||||
incomplete so the agent can retry instead of silently dropping fields.
|
||||
"""
|
||||
if isinstance(arguments, (dict, list)):
|
||||
return json.dumps(arguments, ensure_ascii=False, separators=(",", ":")), True
|
||||
if arguments is None:
|
||||
return "{}", True
|
||||
if not isinstance(arguments, str):
|
||||
arguments = str(arguments)
|
||||
|
||||
text = arguments.strip()
|
||||
if not text:
|
||||
return "{}", True
|
||||
|
||||
def _parse_candidate(candidate: str):
|
||||
try:
|
||||
return json.loads(candidate)
|
||||
except (json.JSONDecodeError, TypeError, ValueError):
|
||||
pass
|
||||
try:
|
||||
return ast.literal_eval(candidate)
|
||||
except (SyntaxError, ValueError):
|
||||
return None
|
||||
|
||||
candidates: list[str] = [text]
|
||||
|
||||
trimmed_trailing_commas = re.sub(r",\s*([}\]])", r"\1", text)
|
||||
if trimmed_trailing_commas != text:
|
||||
candidates.append(trimmed_trailing_commas)
|
||||
|
||||
if ":" in text and not text.startswith(("{", "[")):
|
||||
wrapped = "{" + text + "}"
|
||||
candidates.append(wrapped)
|
||||
quoted_keys = re.sub(
|
||||
r'([\{,]\s*)([A-Za-z_][A-Za-z0-9_\-]*)(\s*:)',
|
||||
r'\1"\2"\3',
|
||||
wrapped,
|
||||
)
|
||||
if quoted_keys != wrapped:
|
||||
candidates.append(quoted_keys)
|
||||
trimmed_quoted_keys = re.sub(r",\s*([}\]])", r"\1", quoted_keys)
|
||||
if trimmed_quoted_keys != quoted_keys:
|
||||
candidates.append(trimmed_quoted_keys)
|
||||
|
||||
seen: set[str] = set()
|
||||
for candidate in candidates:
|
||||
if candidate in seen:
|
||||
continue
|
||||
seen.add(candidate)
|
||||
parsed = _parse_candidate(candidate)
|
||||
if isinstance(parsed, (dict, list)):
|
||||
return json.dumps(parsed, ensure_ascii=False, separators=(",", ":")), True
|
||||
|
||||
return text, False
|
||||
|
||||
@staticmethod
|
||||
def _merge_consecutive_assistant_tool_call_messages(messages: List[Dict[str, Any]]) -> List[Dict[str, Any]]:
|
||||
"""Merge adjacent assistant messages that each carry tool_calls.
|
||||
|
||||
Some providers emit parallel tool calls as multiple consecutive assistant
|
||||
messages instead of a single assistant message with multiple tool calls.
|
||||
Merge only adjacent assistant/tool-call messages; any non-assistant
|
||||
boundary flushes the current batch.
|
||||
"""
|
||||
merged: List[Dict[str, Any]] = []
|
||||
pending: Optional[Dict[str, Any]] = None
|
||||
|
||||
def _flush_pending() -> None:
|
||||
nonlocal pending
|
||||
if pending is not None:
|
||||
merged.append(pending)
|
||||
pending = None
|
||||
|
||||
for msg in messages:
|
||||
if not isinstance(msg, dict):
|
||||
_flush_pending()
|
||||
merged.append(msg)
|
||||
continue
|
||||
|
||||
role = msg.get("role")
|
||||
tool_calls = msg.get("tool_calls")
|
||||
if role == "assistant" and isinstance(tool_calls, list) and tool_calls:
|
||||
if pending is None:
|
||||
pending = copy.deepcopy(msg)
|
||||
continue
|
||||
|
||||
pending_tool_calls = pending.get("tool_calls")
|
||||
if not isinstance(pending_tool_calls, list):
|
||||
pending_tool_calls = []
|
||||
pending["tool_calls"] = pending_tool_calls
|
||||
pending_tool_calls.extend(copy.deepcopy(tool_calls))
|
||||
|
||||
pending_content = pending.get("content") or ""
|
||||
current_content = msg.get("content") or ""
|
||||
if pending_content and current_content:
|
||||
pending["content"] = pending_content + "\n" + current_content
|
||||
elif current_content:
|
||||
pending["content"] = current_content
|
||||
continue
|
||||
|
||||
_flush_pending()
|
||||
merged.append(msg)
|
||||
|
||||
_flush_pending()
|
||||
return merged
|
||||
|
||||
@staticmethod
|
||||
def _sanitize_api_messages(messages: List[Dict[str, Any]]) -> List[Dict[str, Any]]:
|
||||
"""Fix orphaned tool_call / tool_result pairs before every LLM call.
|
||||
@@ -3347,7 +3461,7 @@ class AIAgent:
|
||||
)
|
||||
continue
|
||||
filtered.append(msg)
|
||||
messages = filtered
|
||||
messages = AIAgent._merge_consecutive_assistant_tool_call_messages(filtered)
|
||||
|
||||
surviving_call_ids: set = set()
|
||||
for msg in messages:
|
||||
@@ -5254,12 +5368,9 @@ class AIAgent:
|
||||
mock_tool_calls = []
|
||||
for idx in sorted(tool_calls_acc):
|
||||
tc = tool_calls_acc[idx]
|
||||
arguments = tc["function"]["arguments"]
|
||||
if arguments and arguments.strip():
|
||||
try:
|
||||
json.loads(arguments)
|
||||
except json.JSONDecodeError:
|
||||
has_truncated_tool_args = True
|
||||
arguments, is_complete = self._normalize_tool_call_arguments(tc["function"]["arguments"])
|
||||
if not is_complete:
|
||||
has_truncated_tool_args = True
|
||||
mock_tool_calls.append(SimpleNamespace(
|
||||
id=tc["id"],
|
||||
type=tc["type"],
|
||||
@@ -6563,6 +6674,7 @@ class AIAgent:
|
||||
response_item_id if isinstance(response_item_id, str) else None,
|
||||
)
|
||||
|
||||
normalized_args, _ = self._normalize_tool_call_arguments(tool_call.function.arguments)
|
||||
tc_dict = {
|
||||
"id": call_id,
|
||||
"call_id": call_id,
|
||||
@@ -6570,7 +6682,7 @@ class AIAgent:
|
||||
"type": tool_call.type,
|
||||
"function": {
|
||||
"name": tool_call.function.name,
|
||||
"arguments": tool_call.function.arguments
|
||||
"arguments": normalized_args,
|
||||
},
|
||||
}
|
||||
# Preserve extra_content (e.g. Gemini thought_signature) so it
|
||||
@@ -10031,21 +10143,15 @@ class AIAgent:
|
||||
# Handle empty strings as empty objects (common model quirk)
|
||||
invalid_json_args = []
|
||||
for tc in assistant_message.tool_calls:
|
||||
args = tc.function.arguments
|
||||
if isinstance(args, (dict, list)):
|
||||
tc.function.arguments = json.dumps(args)
|
||||
continue
|
||||
if args is not None and not isinstance(args, str):
|
||||
tc.function.arguments = str(args)
|
||||
args = tc.function.arguments
|
||||
# Treat empty/whitespace strings as empty object
|
||||
if not args or not args.strip():
|
||||
tc.function.arguments = "{}"
|
||||
continue
|
||||
try:
|
||||
json.loads(args)
|
||||
except json.JSONDecodeError as e:
|
||||
invalid_json_args.append((tc.function.name, str(e)))
|
||||
normalized_args, is_complete = self._normalize_tool_call_arguments(tc.function.arguments)
|
||||
tc.function.arguments = normalized_args
|
||||
if not is_complete:
|
||||
try:
|
||||
json.loads(normalized_args)
|
||||
except json.JSONDecodeError as e:
|
||||
invalid_json_args.append((tc.function.name, str(e)))
|
||||
except Exception as e:
|
||||
invalid_json_args.append((tc.function.name, str(e)))
|
||||
|
||||
if invalid_json_args:
|
||||
# Check if the invalid JSON is due to truncation rather
|
||||
|
||||
@@ -1037,6 +1037,138 @@ class TestBuildAssistantMessage:
|
||||
result = agent._build_assistant_message(msg, "tool_calls")
|
||||
assert "extra_content" not in result["tool_calls"][0]
|
||||
|
||||
def test_tool_call_arguments_normalized_from_gemma4_whitespace(self, agent):
|
||||
tc = _mock_tool_call(
|
||||
name="read_file",
|
||||
arguments=' \n {"path": "README.md"} \n ',
|
||||
call_id="c4",
|
||||
)
|
||||
msg = _mock_assistant_msg(content="", tool_calls=[tc])
|
||||
result = agent._build_assistant_message(msg, "tool_calls")
|
||||
assert result["tool_calls"][0]["function"]["arguments"] == '{"path":"README.md"}'
|
||||
|
||||
def test_tool_call_arguments_normalized_from_single_quotes_and_trailing_comma(self, agent):
|
||||
tc = _mock_tool_call(
|
||||
name="read_file",
|
||||
arguments="{'path': 'README.md',}",
|
||||
call_id="c5",
|
||||
)
|
||||
msg = _mock_assistant_msg(content="", tool_calls=[tc])
|
||||
result = agent._build_assistant_message(msg, "tool_calls")
|
||||
assert result["tool_calls"][0]["function"]["arguments"] == '{"path":"README.md"}'
|
||||
|
||||
|
||||
class TestNormalizeToolCallArguments:
|
||||
@pytest.mark.parametrize(
|
||||
("raw_args", "expected"),
|
||||
[
|
||||
('{"q":"test"}', '{"q":"test"}'),
|
||||
(' \n {"q": "test"} \n ', '{"q":"test"}'),
|
||||
('{"q": "test",}', '{"q":"test"}'),
|
||||
("{'q': 'test'}", '{"q":"test"}'),
|
||||
("{'path': 'README.md', 'mode': 'read'}", '{"path":"README.md","mode":"read"}'),
|
||||
('"path": "README.md"', '{"path":"README.md"}'),
|
||||
('path: "README.md"', '{"path":"README.md"}'),
|
||||
('path: "README.md", mode: "read"', '{"path":"README.md","mode":"read"}'),
|
||||
({"path": "README.md"}, '{"path":"README.md"}'),
|
||||
(["README.md", "docs.md"], '["README.md","docs.md"]'),
|
||||
('\t\n ', '{}'),
|
||||
('{"nested": {"path": "README.md"}}', '{"nested":{"path":"README.md"}}'),
|
||||
],
|
||||
)
|
||||
def test_complete_args_are_normalized(self, raw_args, expected):
|
||||
normalized, is_complete = AIAgent._normalize_tool_call_arguments(raw_args)
|
||||
assert is_complete is True
|
||||
assert normalized == expected
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"raw_args",
|
||||
[
|
||||
'{"path": "README.md"',
|
||||
'{"a": 1, "b"',
|
||||
'{"path": [1, 2}',
|
||||
"{'path': 'README.md'",
|
||||
'path: "README.md", mode:',
|
||||
'{"command": "echo hello",',
|
||||
],
|
||||
)
|
||||
def test_incomplete_args_are_not_marked_complete(self, raw_args):
|
||||
normalized, is_complete = AIAgent._normalize_tool_call_arguments(raw_args)
|
||||
assert is_complete is False
|
||||
assert isinstance(normalized, str)
|
||||
assert normalized == raw_args.strip()
|
||||
|
||||
|
||||
class TestSanitizeApiMessages:
|
||||
def test_merges_consecutive_assistant_tool_call_messages(self):
|
||||
messages = [
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "first",
|
||||
"tool_calls": [{"id": "c1", "type": "function", "function": {"name": "read_file", "arguments": '{"path":"a.py"}'}}],
|
||||
},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "second",
|
||||
"tool_calls": [{"id": "c2", "type": "function", "function": {"name": "search_files", "arguments": '{"pattern":"TODO"}'}}],
|
||||
},
|
||||
{"role": "tool", "tool_call_id": "c1", "content": "a.py"},
|
||||
{"role": "tool", "tool_call_id": "c2", "content": "matches"},
|
||||
]
|
||||
|
||||
sanitized = AIAgent._sanitize_api_messages(messages)
|
||||
|
||||
assert len(sanitized) == 3
|
||||
assert sanitized[0]["role"] == "assistant"
|
||||
assert [tc["id"] for tc in sanitized[0]["tool_calls"]] == ["c1", "c2"]
|
||||
assert sanitized[0]["content"] == "first\nsecond"
|
||||
|
||||
def test_does_not_merge_assistant_tool_call_messages_across_non_assistant_boundary(self):
|
||||
messages = [
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "",
|
||||
"tool_calls": [{"id": "c1", "type": "function", "function": {"name": "read_file", "arguments": '{"path":"a.py"}'}}],
|
||||
},
|
||||
{"role": "tool", "tool_call_id": "c1", "content": "a.py"},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "",
|
||||
"tool_calls": [{"id": "c2", "type": "function", "function": {"name": "read_file", "arguments": '{"path":"b.py"}'}}],
|
||||
},
|
||||
{"role": "tool", "tool_call_id": "c2", "content": "b.py"},
|
||||
]
|
||||
|
||||
sanitized = AIAgent._sanitize_api_messages(messages)
|
||||
|
||||
assistant_msgs = [m for m in sanitized if m.get("role") == "assistant"]
|
||||
assert len(assistant_msgs) == 2
|
||||
assert assistant_msgs[0]["tool_calls"][0]["id"] == "c1"
|
||||
assert assistant_msgs[1]["tool_calls"][0]["id"] == "c2"
|
||||
|
||||
def test_merge_preserves_tool_call_order(self):
|
||||
messages = [
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "",
|
||||
"tool_calls": [{"id": "c1", "type": "function", "function": {"name": "read_file", "arguments": '{"path":"a.py"}'}}],
|
||||
},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "",
|
||||
"tool_calls": [{"id": "c2", "type": "function", "function": {"name": "read_file", "arguments": '{"path":"b.py"}'}}],
|
||||
},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "",
|
||||
"tool_calls": [{"id": "c3", "type": "function", "function": {"name": "read_file", "arguments": '{"path":"c.py"}'}}],
|
||||
},
|
||||
]
|
||||
|
||||
sanitized = AIAgent._sanitize_api_messages(messages)
|
||||
|
||||
assert [tc["id"] for tc in sanitized[0]["tool_calls"]] == ["c1", "c2", "c3"]
|
||||
|
||||
|
||||
class TestFormatToolsForSystemMessage:
|
||||
def test_no_tools_returns_empty_array(self, agent):
|
||||
@@ -3467,6 +3599,59 @@ class TestStreamingApiCall:
|
||||
assert tc[0].function.arguments == '{"path":"x.txt","content":"hel'
|
||||
assert resp.choices[0].finish_reason == "length"
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("raw_arguments", "expected"),
|
||||
[
|
||||
(' \n {"path": "x.txt"} \n ', '{"path":"x.txt"}'),
|
||||
("{'path': 'x.txt',}", '{"path":"x.txt"}'),
|
||||
('path: "x.txt", mode: "read"', '{"path":"x.txt","mode":"read"}'),
|
||||
],
|
||||
)
|
||||
def test_repairable_tool_call_args_do_not_upgrade_finish_reason_to_length(self, agent, raw_arguments, expected):
|
||||
chunks = [
|
||||
_make_chunk(tool_calls=[_make_tc_delta(0, "call_1", "read_file", raw_arguments)]),
|
||||
_make_chunk(finish_reason="tool_calls"),
|
||||
]
|
||||
agent.client.chat.completions.create.return_value = iter(chunks)
|
||||
|
||||
resp = agent._interruptible_streaming_api_call({"messages": []})
|
||||
|
||||
tc = resp.choices[0].message.tool_calls
|
||||
assert len(tc) == 1
|
||||
assert tc[0].function.name == "read_file"
|
||||
assert tc[0].function.arguments == expected
|
||||
assert resp.choices[0].finish_reason == "tool_calls"
|
||||
|
||||
def test_streamed_tool_call_args_single_quotes_across_chunks_normalized(self, agent):
|
||||
chunks = [
|
||||
_make_chunk(tool_calls=[_make_tc_delta(0, "call_1", "read_file", "{'path':")]),
|
||||
_make_chunk(tool_calls=[_make_tc_delta(0, None, None, " 'x.txt',}")]),
|
||||
_make_chunk(finish_reason="tool_calls"),
|
||||
]
|
||||
agent.client.chat.completions.create.return_value = iter(chunks)
|
||||
|
||||
resp = agent._interruptible_streaming_api_call({"messages": []})
|
||||
|
||||
tc = resp.choices[0].message.tool_calls
|
||||
assert len(tc) == 1
|
||||
assert tc[0].function.arguments == '{"path":"x.txt"}'
|
||||
assert resp.choices[0].finish_reason == "tool_calls"
|
||||
|
||||
def test_streamed_split_json_chunks_still_reassemble(self, agent):
|
||||
chunks = [
|
||||
_make_chunk(tool_calls=[_make_tc_delta(0, "call_1", "read_file", '{"path":')]),
|
||||
_make_chunk(tool_calls=[_make_tc_delta(0, None, None, ' "x.txt"}')]),
|
||||
_make_chunk(finish_reason="tool_calls"),
|
||||
]
|
||||
agent.client.chat.completions.create.return_value = iter(chunks)
|
||||
|
||||
resp = agent._interruptible_streaming_api_call({"messages": []})
|
||||
|
||||
tc = resp.choices[0].message.tool_calls
|
||||
assert len(tc) == 1
|
||||
assert tc[0].function.arguments == '{"path":"x.txt"}'
|
||||
assert resp.choices[0].finish_reason == "tool_calls"
|
||||
|
||||
def test_ollama_reused_index_separate_tool_calls(self, agent):
|
||||
"""Ollama sends every tool call at index 0 with different ids.
|
||||
|
||||
|
||||
@@ -1,25 +0,0 @@
|
||||
from pathlib import Path
|
||||
|
||||
from tools.skills_hub import OptionalSkillSource
|
||||
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[1]
|
||||
|
||||
|
||||
def test_optional_skill_source_scans_adversarial_ux_test():
|
||||
source = OptionalSkillSource()
|
||||
metas = {meta.identifier: meta for meta in source._scan_all()}
|
||||
|
||||
assert "official/dogfood/adversarial-ux-test" in metas
|
||||
assert metas["official/dogfood/adversarial-ux-test"].name == "adversarial-ux-test"
|
||||
assert "tech-resistant user" in metas["official/dogfood/adversarial-ux-test"].description
|
||||
|
||||
|
||||
def test_optional_skill_catalog_docs_list_adversarial_ux_test():
|
||||
optional_catalog = (REPO_ROOT / "website" / "docs" / "reference" / "optional-skills-catalog.md").read_text(encoding="utf-8")
|
||||
bundled_catalog = (REPO_ROOT / "website" / "docs" / "reference" / "skills-catalog.md").read_text(encoding="utf-8")
|
||||
|
||||
assert "**adversarial-ux-test**" in optional_catalog
|
||||
assert "official/dogfood/adversarial-ux-test" in optional_catalog
|
||||
assert "`adversarial-ux-test`" in bundled_catalog
|
||||
assert "dogfood/adversarial-ux-test" in bundled_catalog
|
||||
@@ -16,7 +16,6 @@ For example:
|
||||
|
||||
```bash
|
||||
hermes skills install official/blockchain/solana
|
||||
hermes skills install official/dogfood/adversarial-ux-test
|
||||
hermes skills install official/mlops/flash-attention
|
||||
```
|
||||
|
||||
@@ -57,12 +56,6 @@ hermes skills uninstall <skill-name>
|
||||
| **blender-mcp** | Control Blender directly from Hermes via socket connection to the blender-mcp addon. Create 3D objects, materials, animations, and run arbitrary Blender Python (bpy) code. |
|
||||
| **meme-generation** | Generate real meme images by picking a template and overlaying text with Pillow. Produces actual `.png` meme files. |
|
||||
|
||||
## Dogfood
|
||||
|
||||
| Skill | Description |
|
||||
|-------|-------------|
|
||||
| **adversarial-ux-test** | Roleplay the most difficult, tech-resistant user for a product — browse in-persona, rant, then filter through a RED/YELLOW/WHITE/GREEN pragmatism layer so only real UX friction becomes tickets. |
|
||||
|
||||
## DevOps
|
||||
|
||||
| Skill | Description |
|
||||
|
||||
@@ -59,12 +59,9 @@ DevOps and infrastructure automation skills.
|
||||
|
||||
## dogfood
|
||||
|
||||
Internal dogfooding and QA skills used to test Hermes Agent itself.
|
||||
|
||||
| Skill | Description | Path |
|
||||
|-------|-------------|------|
|
||||
| `dogfood` | Systematic exploratory QA testing of web applications — find bugs, capture evidence, and generate structured reports. | `dogfood/dogfood` |
|
||||
| `adversarial-ux-test` | Roleplay the most difficult, tech-resistant user for a product — browse in-persona, rant, then filter through a RED/YELLOW/WHITE/GREEN pragmatism layer so only real UX friction becomes tickets. | `dogfood/adversarial-ux-test` |
|
||||
| `hermes-agent-setup` | Help users configure Hermes Agent — CLI usage, setup wizard, model/provider selection, tools, skills, voice/STT/TTS, gateway, and troubleshooting. | `dogfood/hermes-agent-setup` |
|
||||
|
||||
## email
|
||||
|
||||
Reference in New Issue
Block a user