mirror of
https://github.com/kovidgoyal/kitty
synced 2026-07-16 13:34:48 +02:00
Fix UTF-8 overlong and special range checks in simd-string-impl.h
Modified `start_classification` in `utf8_decode_to_esc` in `simd-string-impl.h`, which now: Rejects `0xC0`, `0xC1` and `0xF5..0xFF` lead bytes in UTF-8 subsequences. Enforces special ranges for the second subsequence bytes after `0xE0`, `0xED`, `0xF0` and `0xF4` bytes to prevent overlong sequences, surrogates, and code points above U+10FFFF. Accumulates UTF-8 validation errors in a single vector to avoid many conditional branches. Worsens unicode benchmark performance by about 4%.
This commit is contained in:
@@ -322,10 +322,6 @@ class TestParser(BaseTest):
|
||||
# Bad continuation byte (restored as ASCII)
|
||||
pb(b'"\xe1\x28\xa1"', '"\ufffd(\ufffd"') # )
|
||||
|
||||
# The following all fail when using SIMD and need to be fixed in the SIMD parser
|
||||
if which != 1:
|
||||
continue
|
||||
|
||||
# Overlong 2-byte sequence for U+0000 (should be `0x00`)
|
||||
pb(b'"\xc0\x80"', '"\ufffd\ufffd"')
|
||||
|
||||
|
||||
Reference in New Issue
Block a user