� J�g0'��d�ddlmZmZddlmZmZmZddlmZm Z m Z ddl m Z m Z mZddlmZmZmZddlmZmZmZddlmZGd �d ��ZGd �d e��ZGd �de��ZGd�de��ZGd�de��ZGd�de��ZGd�de��ZGd�de��Z dS)�)�Tuple�Union�)�BIG5_CHAR_TO_FREQ_ORDER�BIG5_TABLE_SIZE�BIG5_TYPICAL_DISTRIBUTION_RATIO)�EUCKR_CHAR_TO_FREQ_ORDER�EUCKR_TABLE_SIZE� EUCKR_TYPICAL_DISTRIBUTION_RATIO)�EUCTW_CHAR_TO_FREQ_ORDER�EUCTW_TABLE_SIZE� EUCTW_TYPICAL_DISTRIBUTION_RATIO)�GB2312_CHAR_TO_FREQ_ORDER�GB2312_TABLE_SIZE�!GB2312_TYPICAL_DISTRIBUTION_RATIO)�JIS_CHAR_TO_FREQ_ORDER�JIS_TABLE_SIZE�JIS_TYPICAL_DISTRIBUTION_RATIO)�JOHAB_TO_EUCKR_ORDER_TABLEc��eZdZdZdZdZdZdd�Zdd�Zd e e e fd e ddfd �Z defd �Zdefd �Zde e e fde fd�ZdS)�CharDistributionAnalysisig�G�z��?g{�G�z�?��returnNc��t��|_d|_d|_d|_d|_d|_|���dS)NrgF)�tuple�_char_to_freq_order� _table_size�typical_distribution_ratio�_done� _total_chars� _freq_chars�reset��selfs �h/home/asafur/pinokio/api/open-webui.git/app/env/lib/python3.11/site-packages/chardet/chardistribution.py�__init__z!CharDistributionAnalysis.__init__@sJ��5:�G�G�� ���� +.��'��� ������� � � � � � � � �c�0�d|_d|_d|_dS)zreset analyser, clear any stateFrN)rr r!r#s r%r"zCharDistributionAnalysis.resetOs!���� ��������r'�char�char_lenc���|dkr|�|��}nd}|dkr>|xjdz c_||jkr%d|j|kr|xjdz c_dSdSdSdS)z"feed a character with known length������rriN)� get_orderr rrr!)r$r)r*�orders r%�feedzCharDistributionAnalysis.feedXs��� �q�=�=��N�N�4�(�(�E�E��E� �A�:�:� � � �� "� � ��t�'�'�'���1�%�8�8�8��$�$��)�$�$�$�$� �:�(�'�8�8r'c���|jdks|j|jkr|jS|j|jkr,|j|j|jz |jzz }||jkr|S|jS)z(return confidence based on existing datar)r r!�MINIMUM_DATA_THRESHOLD�SURE_NOr�SURE_YES)r$�rs r%�get_confidencez'CharDistributionAnalysis.get_confidencefs|�� � �� !� !�T�%5��9T�%T�%T��<� � � �� 0� 0� 0�� ��"�T�%5�5��9X�X��A��4�=� � ����}�r'c�"�|j|jkS�N)r �ENOUGH_DATA_THRESHOLDr#s r%�got_enough_dataz(CharDistributionAnalysis.got_enough_dataws��� �4�#=�=�=r'�_c��dS)Nr-�)r$r;s r%r.z"CharDistributionAnalysis.get_order|s �� �rr'�rN)�__name__� __module__� __qualname__r9r4r3r2r&r"r�bytes� bytearray�intr0�floatr6�boolr:r.r=r'r%rr:s������� ���H��G��� � � � ����� *��u�i�/�0� *�C� *�D� *� *� *� *�������">��>�>�>�>� �5�� �!1�2��s������r'rc�@��eZdZd�fd� Zdeeefdefd�Z�xZ S)�EUCTWDistributionAnalysisrNc���t�����t|_t|_t |_dSr8)�superr&r rr rrr�r$� __class__s �r%r&z"EUCTWDistributionAnalysis.__init__��7��� ��������#;�� �+���*J��'�'�'r'�byte_strc�J�|d}|dkrd|dz z|dzdz SdS)Nr���^r�r-r=�r$rN� first_chars r%r.z#EUCTWDistributionAnalysis.get_order��;�� �a�[� � �� � ���d�*�+�h�q�k�9�D�@� @��rr'r>� r?r@rAr&rrBrCrDr.� __classcell__�rLs@r%rHrH��o�������K�K�K�K�K�K� �%��y�(8�"9��c��������r'rHc�@��eZdZd�fd� Zdeeefdefd�Z�xZ S)�EUCKRDistributionAnalysisrNc���t�����t|_t|_t |_dSr8�rJr&r rr rr rrKs �r%r&z"EUCKRDistributionAnalysis.__init__�rMr'rNc�J�|d}|dkrd|dz z|dzdz SdS)Nr�rQrrRr-r=rSs r%r.z#EUCKRDistributionAnalysis.get_order�rUr'r>rVrXs@r%r[r[�rYr'r[c�@��eZdZd�fd� Zdeeefdefd�Z�xZ S)�JOHABDistributionAnalysisrNc���t�����t|_t|_t |_dSr8r]rKs �r%r&z"JOHABDistributionAnalysis.__init__�rMr'rNc�|�|d}d|cxkrdkr&nn#|dz|dz}tj|d��SdS)Nr����rr-)r�get)r$rNrT�codes r%r.z#JOHABDistributionAnalysis.get_order�sX���a�[� � �:� $� $� $� $�� $� $� $� $� $���#�h�q�k�1�D�-�1�$��;�;� ;��rr'r>rVrXs@r%rara�so�������K�K�K�K�K�K� �%��y�(8�"9��c��������r'rac�@��eZdZd�fd� Zdeeefdefd�Z�xZ S)�GB2312DistributionAnalysisrNc���t�����t|_t|_t |_dSr8)rJr&rrrrrrrKs �r%r&z#GB2312DistributionAnalysis.__init__�s7��� ��������#<�� �,���*K��'�'�'r'rNc�Z�|d|d}}|dkr|dkrd|dz z|zdz SdS)Nrrr_rRrQr-r=�r$rNrT� second_chars r%r.z$GB2312DistributionAnalysis.get_order�sI�� #+�1�+�x��{�K� � �$� � �[�D�%8�%8���d�*�+�k�9�D�@� @��rr'r>rVrXs@r%rjrj�so�������L�L�L�L�L�L� �%��y�(8�"9��c��������r'rjc�@��eZdZd�fd� Zdeeefdefd�Z�xZ S)�Big5DistributionAnalysisrNc���t�����t|_t|_t |_dSr8)rJr&rrrrrrrKs �r%r&z!Big5DistributionAnalysis.__init__�s7��� ��������#:�� �*���*I��'�'�'r'rNc�|�|d|d}}|dkr%|dkrd|dz z|zdz dzSd|dz z|zdz SdS) Nrr�rR��?�@r-r=rms r%r.z"Big5DistributionAnalysis.get_order�si�� #+�1�+�x��{�K� � �� � ��d�"�"��j�4�/�0�;�>��E��J�J��*�t�+�,�{�:�T�A� A��rr'r>rVrXs@r%rprp�so�������J�J�J�J�J�J� �%��y�(8�"9� �c� � � � � � � � r'rpc�@��eZdZd�fd� Zdeeefdefd�Z�xZ S)�SJISDistributionAnalysisrNc���t�����t|_t|_t |_dSr8�rJr&rrrrrrrKs �r%r&z!SJISDistributionAnalysis.__init__��7��� ��������#9�� �)���*H��'�'�'r'rNc��|d|d}}d|cxkrdkr nn d|dz z}nd|cxkrdkrnn d|dz dzz}nd S||zd z }|d krd }|S) Nrr��������r-rv�r=)r$rNrTrnr/s r%r.z"SJISDistributionAnalysis.get_order�s��� #+�1�+�x��{�K� � �:� %� %� %� %�� %� %� %� %� %��:��,�-�E�E� �Z� '� '� '� '�4� '� '� '� '� '��:��,�r�1�2�E�E��2�� �#�d�*�� �� � ��E�� r'r>rVrXs@r%rxrx�so�������I�I�I�I�I�I� �%��y�(8�"9��c��������r'rxc�@��eZdZd�fd� Zdeeefdefd�Z�xZ S)�EUCJPDistributionAnalysisrNc���t�����t|_t|_t |_dSr8rzrKs �r%r&z"EUCJPDistributionAnalysis.__init__�r{r'rNc�J�|d}|dkrd|dz z|dzdz SdS)Nr�rQrRrr-r=)r$rNr)s r%r.z#EUCJPDistributionAnalysis.get_order�s8�� ��{�� �4�<�<�����%��� �3�d�:� :��rr'r>rVrXs@r%r�r��so�������I�I�I�I�I�I� �%��y�(8�"9��c��������r'r�N)!�typingrr�big5freqrrr� euckrfreqr r r � euctwfreqr r r� gb2312freqrrr�jisfreqrrr� johabfreqrrrHr[rarjrprxr�r=r'r%�<module>r�s���8 ������������������ ���������� ���������� ���������� ���������� 2�1�1�1�1�1�G�G�G�G�G�G�G�G�T����� 8����$����� 8����$ � � � � � 8� � � ������!9����$�����7����(�����7����2����� 8�����r'
Memory