Fix UTF-8 overlong and special range checks in simd-string-impl.h

Modified `start_classification` in `utf8_decode_to_esc` in `simd-string-impl.h`, which now:

Rejects `0xC0`, `0xC1` and `0xF5..0xFF` lead bytes in UTF-8 subsequences.

Enforces special ranges for the second subsequence bytes after `0xE0`, `0xED`, `0xF0` and `0xF4` bytes to prevent overlong sequences, surrogates, and code points above U+10FFFF.

Accumulates UTF-8 validation errors in a single vector to avoid many conditional branches.

Worsens unicode benchmark performance by about 4%.
This commit is contained in:
Wukuyon
2025-10-18 17:11:58 -06:00
parent 295951348c
commit 65890de60d
2 changed files with 59 additions and 9 deletions

View File

@@ -322,10 +322,6 @@ class TestParser(BaseTest):
# Bad continuation byte (restored as ASCII)
pb(b'"\xe1\x28\xa1"', '"\ufffd(\ufffd"') # )
# The following all fail when using SIMD and need to be fixed in the SIMD parser
if which != 1:
continue
# Overlong 2-byte sequence for U+0000 (should be `0x00`)
pb(b'"\xc0\x80"', '"\ufffd\ufffd"')