FR version is available. Content is displayed in original English for accuracy.
Advertisement
Advertisement
⚡ Community Insights
Discussion Sentiment
100% Positive
Analyzed from 148 words in the discussion.
Trending Topics
#ascii#byte#detect#encodings#text#usage#clickhouse#simd#optimization#chances

Discussion (5 Comments)Read Original on HackerNews
Another optimization would be to take advantage of the fact that codepoint usage tends to cluster around the language of the text. So if you detect usage of 3-byte encodings, chances are you'll continue encountering only 3-byte encodings, with the odd ASCII or emoji codepoints. This opens up even more state machine possibilities.
Depends on what kind of text you're processing. Many languages use the ASCII range for spaces/newlines, digits, and punctuation.