�
L�g8 � � � d Z ddlZddlZddddddd d
d d d
ddd�
Z ej dj di e��ej � � Zdede fd�Z
dede fd�Z dede fd�Z
dS )ab
`ftfy.badness` contains a heuristic that detects likely mojibake.
This heuristic signals to ftfy which segments of text need to be fixed, and
also indicates when the text can stop being fixed.
The design of this heuristic is that we categorize the approximately 400
Unicode characters that occur in UTF-8 mojibake, specifically the characters
that come from mixing up UTF-8 with the other encodings we support. We
identify sequences and contexts of these characters that are much more likely
to be mojibake than intended strings, such as lowercase accented letters
followed immediately by currency symbols.
� Nu ·´–—―…’u -u2 ¦¤¨¬¯¶§¸ƒˆˇ˘˛˜†‡‰⌐◊�ªºu ¢£¥₧€u! ¡«¿©΄΅‘‚“„•‹u ®»˝”›™uA ²³¹±¼½¾×µ÷⁄∂∆∏∑√∞∩∫≈≠≡≤≥№u Ò-ÖÙ-Üò-öø-üŐ°uE À-ÑØÜÝĂĄĆČĎĐĘĚĞİĹĽŁŃŇŒŘŚŞŠŢŤŮŰŸŹŻŽҐu; ßà-ñăąćčďđęěğĺľłœŕśşšťüźżžґfiflu ÞΑ-ΩΆΈΉΊΌΎΏΪΫЁ-Яu α-ωάέήίΰа-џu7 │┌┐┘├┤┬┼═-╬▀▄█▌▐░▒▓)
�common�c1�bad�currency�start_punctuation�end_punctuation�numeric�kaomoji�upper_accented�lower_accented� upper_common� lower_common�boxu6
[{c1}]
|
[{bad}{lower_accented}{upper_accented}{box}{start_punctuation}{end_punctuation}{currency}{numeric}] [{bad}]
|
[a-zA-Z] [{lower_common}{upper_common}] [{bad}]
|
[{bad}] [{lower_accented}{upper_accented}{box}{start_punctuation}{end_punctuation}{currency}{numeric}]
|
[{lower_accented}{lower_common}{box}{end_punctuation}{currency}{numeric}] [{upper_accented}]
|
[{box}{end_punctuation}{currency}{numeric}] [{lower_accented}]
|
[{lower_accented}{box}{end_punctuation}] [{currency}]
|
\s [{upper_accented}] [{currency}]
|
[{upper_accented}{box}] [{numeric}]
|
[{lower_accented}{upper_accented}{box}{currency}{end_punctuation}] [{start_punctuation}] [{numeric}]
|
[{lower_accented}{upper_accented}{currency}{numeric}{box}] [{end_punctuation}] [{start_punctuation}]
|
[{currency}{numeric}{box}] [{start_punctuation}]
|
[a-z] [{upper_accented}] [{start_punctuation}{currency}]
|
[{box}] [{kaomoji}]
|
[{lower_accented}{upper_accented}{currency}{numeric}{start_punctuation}{end_punctuation}] [{box}]
|
[{box}] [{end_punctuation}]
|
[{lower_accented}{upper_accented}] [{end_punctuation}] \w
|
# The ligature œ when not followed by an unaccented Latin letter
[Œœ][^A-Za-z]
|
# Common Windows-1252 2-character mojibake that isn't covered by the cases above
[ÂÃÎÐ][€Šš¢£Ÿž\xa0\xad®©°·»{start_punctuation}{end_punctuation}–—´]
|
× [²³]
|
# Windows-1252 mojibake of Arabic words needs to include the 'common' characters.
# To compensate, we require four characters to be matched.
[ØÙ] [{common}{currency}{bad}{numeric}{start_punctuation}ŸŠ®°µ»]
[ØÙ] [{common}{currency}{bad}{numeric}{start_punctuation}ŸŠ®°µ»]
|
# Windows-1252 mojibake that starts 3-character sequences for some South Asian
# alphabets
à[²µ¹¼½¾]
|
# MacRoman mojibake that isn't covered by the cases above
√[±∂†≠®™´≤≥¥µø]
|
≈[°¢]
|
‚Ä[ìîïòôúùû†°¢π]
|
‚[âó][àä°ê]
|
# Windows-1251 mojibake of characters in the U+2000 range
вЂ
|
# Windows-1251 mojibake of Latin-1 characters and/or the Cyrillic alphabet.
# Because the 2-character sequences involved here may be common, we require
# seeing a 3-character sequence.
[ВГРС][{c1}{bad}{start_punctuation}{end_punctuation}{currency}°µ][ВГРС]
|
# A distinctive five-character sequence of Cyrillic letters, which can be
# Windows-1251 mojibake on top of Latin-1 mojibake of Windows-1252 characters.
# Require a Latin letter nearby.
ГўВЂВ.[A-Za-z ]
|
# Windows-1252 encodings of 'à' and 'á', as well as \xa0 itself
Ã[\xa0¡]
|
[a-z]\s?[ÃÂ][ ]
|
^[ÃÂ][ ]
|
# Cases where  precedes a character as an encoding of exactly the same
# character, and the character is common enough
[a-z.,?!{end_punctuation}] Â [ {start_punctuation}{end_punctuation}]
|
# Windows-1253 mojibake of characters in the U+2000 range
β€[™\xa0Ά\xad®°]
|
# Windows-1253 mojibake of Latin-1 characters and/or the Greek alphabet
[ΒΓΞΟ][{c1}{bad}{start_punctuation}{end_punctuation}{currency}°][ΒΓΞΟ]
�text�returnc �H � t j d� � t | � � S )z�
This was the name of the heuristic used in ftfy 2.x through 5.x. As an
attempt at compatibility with external code that calls the heuristic
directly, we redirect to our new heuristic, :func:`badness`.
zj`sequence_weirdness()` is an old heuristic, and the current closest equivalent is `ftfy.badness.badness()`)�warnings�warn�badness�r s �\/home/asafur/pinokio/api/open-webui.git/app/env/lib/python3.11/site-packages/ftfy/badness.py�sequence_weirdnessr f s+ � �
�M� 9�� � � �4�=�=�� c �P � t t � | � � � � S )z�
Get the 'badness' of a sequence of text, counting the number of unlikely
character sequences. A badness greater than 0 indicates that some of it
seems to be mojibake.
)�len�
BADNESS_RE�findallr s r r r s s � � �z�!�!�$�'�'� (� (�(r c �P � t t � | � � � � S )a6
Returns true iff the given text looks like it contains mojibake.
This can be faster than `badness`, because it returns when the first match
is found to a regex instead of counting matches. Note that as strings get
longer, they have a higher chance of returning True for `is_bad(string)`.
)�boolr �searchr s r �is_badr! | s � � �
�!�!�$�'�'� (� (�(r � )�__doc__r �re�MOJIBAKE_CATEGORIES�compile�format�VERBOSEr �str�intr r r r! r" r r �<module>r+ s< �� � � ���� � � � � *� � *�4 � � � �D �" 6�J &�> H�$ E� #�kY� Y� �L �R�Z�d � d�H �Id"� d"�H !�Id"� d"�J �J�Mg� g�
�T
�S�
�S�
�
�
�
�)�#� )�#� )� )� )� )�)�� )�� )� )� )� )� )� )r