Teknium
857a5d7b47
fix: sanitize surrogate characters from clipboard paste to prevent UnicodeEncodeError ( #3624 )
...
Pasting text from rich-text editors (Google Docs, Word, etc.) can inject
lone surrogate characters (U+D800..U+DFFF) that are invalid UTF-8.
The OpenAI SDK serializes messages with ensure_ascii=False, then encodes
to UTF-8 for the HTTP body — surrogates crash this with:
UnicodeEncodeError: 'utf-8' codec can't encode character '\udce2'
Three-layer fix:
1. Primary: sanitize user_message at the top of run_conversation()
2. CLI: sanitize in chat() before appending to conversation_history
3. Safety net: catch UnicodeEncodeError in the API error handler,
sanitize the entire messages list in-place, and retry once.
Also exclude UnicodeEncodeError from is_local_validation_error
so it doesn't get classified as non-retryable.
Includes 14 new tests covering the sanitization helpers and the
integration with run_conversation().
2026-03-28 16:53:14 -07:00
..
2026-03-21 15:54:10 -07:00
2026-03-28 07:38:36 -07:00
2026-03-27 08:02:58 -07:00
2026-03-14 14:27:20 +03:00
2026-03-28 14:33:35 -07:00
2026-03-28 14:48:55 -07:00
2026-03-21 09:34:00 -07:00
2026-03-24 07:30:25 -07:00
2026-03-24 19:44:02 -07:00
2026-03-28 14:43:41 -07:00
2025-10-01 23:29:25 +00:00
2026-03-17 02:02:33 -07:00
2026-03-17 02:53:33 -07:00
2026-03-14 02:33:31 -07:00
2026-03-10 15:22:44 -07:00
2026-03-17 01:50:59 -07:00
2026-03-17 04:24:27 -07:00
2026-03-12 01:23:28 -07:00
2026-03-15 08:02:29 -07:00
2026-03-14 11:03:20 -07:00
2026-03-27 13:02:52 -07:00
2026-03-26 19:21:34 -07:00
2026-03-14 19:38:55 -07:00
2026-03-14 19:38:55 -07:00
2026-03-27 13:54:46 -07:00
2026-03-27 09:45:25 -07:00
2026-03-14 22:31:51 -07:00
2026-03-14 22:31:51 -07:00
2026-03-01 19:59:24 -08:00
2026-03-02 17:18:15 +11:00
2026-03-17 04:46:15 -07:00
2026-03-13 23:59:12 -07:00
2026-03-14 11:57:44 -07:00
2026-03-25 15:00:33 -07:00
2026-03-21 09:42:07 -07:00
2026-03-26 17:58:40 -07:00
2026-03-17 02:53:33 -07:00
2026-03-10 17:31:00 -07:00
2026-03-15 19:03:34 -07:00
2026-03-20 04:35:17 -07:00
2026-03-14 21:28:51 -07:00
2026-03-17 02:05:26 -07:00
2026-03-23 06:20:19 -07:00
2026-03-20 06:04:33 -07:00
2026-03-14 21:19:22 -07:00
2026-03-13 03:14:04 -07:00
2026-03-14 15:00:45 +03:00
2026-03-26 17:33:11 -07:00
2026-03-17 02:05:26 -07:00
2026-03-08 20:22:33 -07:00
2026-03-13 21:34:01 -07:00
2026-03-18 15:22:51 -07:00
2026-03-26 18:10:50 -07:00
2026-03-23 16:02:06 -07:00
2026-03-27 21:42:09 -07:00
2026-03-23 06:40:05 -07:00
2026-03-22 05:58:26 -07:00
2026-03-25 17:31:56 -07:00
2026-03-15 08:02:29 -07:00
2026-03-09 21:01:40 -07:00
2026-03-15 21:59:53 -07:00
2026-03-26 14:34:31 -07:00
2026-03-01 19:59:24 -08:00
2026-03-18 02:42:58 -07:00
2026-03-09 02:19:32 -07:00
2026-03-11 21:06:54 -07:00
2026-03-26 14:35:31 -07:00
2026-03-01 03:12:37 -05:00
2026-03-17 03:44:44 -07:00
2026-03-17 02:53:33 -07:00
2026-03-17 02:53:33 -07:00
2026-03-13 21:56:12 -07:00
2026-03-24 08:19:23 -07:00
2026-03-19 21:24:53 +01:00
2026-03-17 23:40:22 -07:00
2026-03-20 15:41:06 -04:00
2026-03-05 18:46:30 -08:00
2026-03-16 06:35:46 -07:00
2026-03-28 14:55:18 -07:00
2026-03-09 17:31:54 +03:00
2026-03-21 09:47:33 -07:00
2026-03-28 11:14:54 -07:00
2026-03-26 13:49:43 -07:00
2026-03-17 02:53:33 -07:00
2026-03-17 02:53:33 -07:00
2026-03-27 09:57:50 -07:00
2026-03-12 08:35:45 -07:00
2026-03-08 17:45:45 -07:00
2026-03-19 10:34:44 -07:00
2026-03-28 14:08:26 -07:00
2026-03-28 11:36:59 -07:00
2026-03-25 18:22:21 -07:00
2026-03-13 04:42:26 +03:00
2026-03-19 15:16:35 +01:00
2026-03-28 13:41:23 -07:00
2026-03-28 16:53:14 -07:00
2026-03-16 23:48:14 -07:00
2026-03-21 09:41:17 -07:00
2026-02-26 13:54:20 +03:00
2026-03-09 23:27:19 -07:00
2026-03-14 11:03:25 -07:00
2026-03-14 21:51:27 -07:00
2026-03-14 21:51:27 -07:00