�
���g � � � d Z ddlmZ ddlmZ deeeef defd�Z G d� de � � Z
G d � d
e � � Z G d � d � � Z d
S )a
Hashing function for dataset keys using `hashlib.md5`
Requirements for the hash function:
- Provides a uniformly distributed hash from random space
- Adequately fast speed
- Working with multiple input types (in this case, `str`, `int` or `bytes`)
- Should be platform independent (generates same hash on different OS and systems)
The hashing function provides a unique 128-bit integer hash of the key provided.
The split name is being used here as the hash salt to avoid having same hashes
in different splits due to same keys
� )�Union)�insecure_hashlib� hash_data�returnc � � t | t � � r| S t | t � � r| � dd� � } n4t | t � � rt | � � } nt | � � �| � d� � S )z|
Returns the input hash_data in its bytes form
Args:
hash_data: the hash salt/key to be converted to bytes
�\�/zutf-8)�
isinstance�bytes�str�replace�int�InvalidKeyError�encode)r s �`/home/asafur/pinokio/api/open-webui.git/app/env/lib/python3.11/site-packages/datasets/keyhash.py� _as_bytesr & s� � � �)�U�#�#� )��� �I�s� #� #� )� �%�%�d�C�0�0� � � �I�s� #� #� )�� �N�N� � � �i�(�(�(� � � �G� $� $�$� c �" � � e Zd ZdZ� fd�Z� xZS )r z6Raises an error when given key is of invalid datatype.c �� �� d| _ d|� dt |� � � �| _ d| _ t � � � | j � | j � | j � �� � d S )Nz7
FAILURE TO GENERATE DATASET: Invalid key type detectedz
Found Key z of type z-
Keys should be either str, int or bytes type)�prefix�type�err_msg�suffix�super�__init__)�selfr � __class__s �r r zInvalidKeyError.__init__@ sb �� �P�� �K�i�K�K�$�y�/�/�K�K�� �F�� �
�����D�K�D�� �D�t�{�D�D�E�E�E�E�Er ��__name__�
__module__� __qualname__�__doc__r �
__classcell__�r s @r r r = sG �� � � � � �@�@�F� F� F� F� F� F� F� F� Fr r c �$ � � e Zd ZdZd� fd� Z� xZS )�DuplicatedKeysErrorz(Raise an error when duplicate key found.� c � �� || _ || _ || _ d| _ t |� � dk r!dd� |� � � d|� �| _ n;dd� |d d� � � � dt |� � dz
� d|� �| _ |rd|z nd | _ t � � � | j � | j � | j � �� � d S )
Nz3Found multiple examples generated with the same key� z
The examples at index z, z have the key z... (z more) have the key �
r' )
�key�duplicate_key_indices�fix_msgr �len�joinr r r r )r r+ r, r- r s �r r z DuplicatedKeysError.__init__J s
�� ����%:��"� �� �K�� � �$� %� %�� +� +�k�d�i�i�@U�6V�6V�k�k�fi�k�k�D�L�L� ]�d�i�i�@U�VY�WY�VY�@Z�6[�6[� ]� ]�be�f{�b|�b|� @B� cB� ]� ]� X[� ]� ]�D�L�(/�7�d�W�n�n�R�� �
�����D�K�D�� �D�t�{�D�D�E�E�E�E�Er )r' r r$ s @r r& r& G sM �� � � � � �2�2�
F�
F�
F�
F�
F�
F�
F�
F�
F�
Fr r& c �@ � e Zd ZdZdefd�Zdeeeef defd�Z dS )� KeyHasherz,KeyHasher class for providing hash using md5� hash_saltc �R � t j t |� � � � | _ d S )N)r �md5r �
_split_md5)r r2 s r r zKeyHasher.__init__Z s � �*�.�y��/C�/C�D�D����r r+ r c �� � | j � � � }t |� � }|� |� � t |� � � d� � S )z�Returns 128-bits unique hash of input key
Args:
key: the input key to be hashed (should be str, int or bytes)
Returns: 128-bit int hash key� )r5 �copyr �updater � hexdigest)r r+ r4 �byte_keys r �hashzKeyHasher.hash] sL � � �o�"�"�$�$�� �S�>�>�� �
�
�8� � � ��3�=�=�?�?�B�'�'�'r N)
r r r! r" r r r r r r< � r r r1 r1 W sg � � � � � �6�6�E�#� E� E� E� E� (��c�3��o�.� (�3� (� (� (� (� (� (r r1 N)
r" �typingr �huggingface_hub.utilsr r r r r � Exceptionr r& r1 r= r r �<module>rA s� ��"� � � � � � � � 2� 2� 2� 2� 2� 2�%��s�C���/� %�E� %� %� %� %�.F� F� F� F� F�i� F� F� F�
F�
F�
F�
F�
F�)�
F�
F�
F� (� (� (� (� (� (� (� (� (� (r