�
J�g � �z � d dl mZmZmZmZmZ ddlmZ ddlm Z m
Z
m Z G d� de� � Z G d� de� � Z
d S )
� )�Dict�List�
NamedTuple�Optional�Union� )�
CharSetProber)�CharacterCategory� ProbingState�SequenceLikelihoodc � � e Zd ZU eed< eed< eeef ed< eeeeef f ed< eed< eed< eed< dS ) �SingleByteCharSetModel� charset_name�language�char_to_order_map�language_model�typical_positive_ratio�keep_ascii_letters�alphabetN) �__name__�
__module__� __qualname__�str�__annotations__r �int�float�bool� � �g/home/asafur/pinokio/api/open-webui.git/app/env/lib/python3.11/site-packages/chardet/sbcharsetprober.pyr r # sx � � � � � � ������M�M�M��C��H�~�%�%�%���d�3��8�n�,�-�-�-�-� !�!�!�!� � � � ��M�M�M�M�Mr r c �� � � e Zd ZdZdZdZdZ ddeded e e
d
df� fd �
Z d� fd � Z e
d
e e fd
�� � Ze
d
e e fd�� � Zdeeef d
efd�Zd
efd�Z� xZS )�SingleByteCharSetProber�@ i gffffff�?g�������?FN�model� is_reversed� name_prober�returnc �� �� t � � � � � || _ || _ || _ d| _ g | _ d| _ d| _ d| _ d| _
| � � � d S �N�� r ) �super�__init__�_model� _reversed� _name_prober� _last_order�
_seq_counters� _total_seqs� _total_char�
_control_char�
_freq_char�reset)�selfr$ r% r& � __class__s �r r, z SingleByteCharSetProber.__init__3 sq �� � ���������� �$��� '��� ���(*��� ��� ��� ������ �
�
� � � � � r c �� �� t � � � � � d| _ dgt j � � z | _ d| _ d| _ d| _ d| _ d S r) )
r+ r6 r0 r �get_num_categoriesr1 r2 r3 r4 r5 )r7 r8 s �r r6 z SingleByteCharSetProber.resetG sX �� �
���
�
���� ��� �S�#5�#H�#J�#J� J��� ��� ��� �������r c �@ � | j r | j j S | j j S �N)r/ r r- �r7 s r r z$SingleByteCharSetProber.charset_nameR s$ � � � � 2��$�1� 1��{�'�'r c �@ � | j r | j j S | j j S r< )r/ r r- r= s r r z SingleByteCharSetProber.languageX s$ � � � � .��$�-� -��{�#�#r �byte_strc �� � | j j s| � |� � }n| � |� � }|s| j S | j j }| j j }|D ]�}|� |t j � � }|t j
k r| xj dz
c_ || j k rs| xj
dz
c_
| j | j k rS| xj dz
c_ | j s|| j | }n|| | j }| j |xx dz
cc< || _ ��| j j }| j t& j k r�| j | j k r�| � � � }|| j k r.| j � d||� � t& j | _ n>|| j k r3| j � d||| j � � t& j | _ | j S )Nr z$%s confidence = %s, we have a winnerz8%s confidence = %s, below negative shortcut threshold %s) r- r �filter_international_words�remove_xml_tags�stater r �getr
� UNDEFINED�CONTROLr3 � SAMPLE_SIZEr5 r0 r2 r. r1 r r � DETECTING�SB_ENOUGH_REL_THRESHOLD�get_confidence�POSITIVE_SHORTCUT_THRESHOLD�logger�debug�FOUND_IT�_state�NEGATIVE_SHORTCUT_THRESHOLD�NOT_ME) r7 r? r r �char�order�lm_catr �
confidences r �feedz SingleByteCharSetProber.feed^ s � ��{�-� 6��6�6�x�@�@�H�H��+�+�H�5�5�H�� ��:� � �K� 9�� ��3�� � %� %�D�%�)�)�$�0A�0K�L�L�E� �(�0�0�0�� � �A�%� � ��t�'�'�'����1�$����#�d�&6�6�6��$�$��)�$�$� �>� I�!/��0@�!A�%�!H���!/��!6�t�7G�!H���&�v�.�.�.�!�3�.�.�.� $�D� � ��{�/� � �:� �/� /� /�� �$�">�>�>� !� 0� 0� 2� 2�
� �� @�@�@��K�%�%�>� �j�� � � #/�"7�D�K�K� �$�"B�B�B��K�%�%�R�$�"� �8� � � � #/�"5�D�K��z�r c � � d}| j dk r{| j t j d| j t j z z | j z | j j z }|| j | j z
z | j z }|| j z | j z }|dk rd}|S )Ng{�G�z�?r g �?g �?g�G�z��?)
r2 r1 r �POSITIVE�LIKELYr- r r3 r4 r5 )r7 �rs r rJ z&SingleByteCharSetProber.get_confidence� s� � � �� � �a� � � �&�'9�'B�C��T� /�0B�0I� J�J�K� �"� #�
�+�4� 5�
� �T�%��(:�:�;�d�>N�N�A��D�O�#�d�&6�6�A��C�x�x����r )FN)r' N)r r r rG rI rK rP r r r r r, r6 �propertyr r r r �bytes� bytearrayr rV r rJ �
__classcell__)r8 s @r r" r" - sH �� � � � � ��K� "��"&� �"&� �
"�/3� � �%�� �� �m�,� �
� � � � � � �( � � � � � � �(�h�s�m� (� (� (� �X�(�
�$�(�3�-� $� $� $� �X�$�
0�U�5�)�#3� 4� 0� � 0� 0� 0� 0�d�� � � � � � � � r r" N)�typingr r r r r �
charsetproberr �enumsr
r r r r" r r r �<module>rb s� ��: ;� :� :� :� :� :� :� :� :� :� :� :� :� :� (� (� (� (� (� (� F� F� F� F� F� F� F� F� F� F�� � � � �Z� � � �u� u� u� u� u�m� u� u� u� u� ur