� ���g^���ddlmZmZddlmZmZmZmZmZddl Z erddl m Z eGd�d����Z eGd�d ����ZdS) �)� dataclass�field)� TYPE_CHECKING�Any�ClassVar�Optional�UnionN�)� FeatureTypec���eZdZUdZeeed<dZeeed<dZ e eed<dZ e e ed<e ddd� ��Zeed <d �Zd ed eed fffd�ZdS)� Translationa`Feature` for translations with fixed languages per example. Here for compatiblity with tfds. Args: languages (`dict`): A dictionary for each example mapping string language codes to string translations. Example: ```python >>> # At construction time: >>> datasets.features.Translation(languages=['en', 'fr', 'de']) >>> # During data generation: >>> yield { ... 'en': 'the cat', ... 'fr': 'le chat', ... 'de': 'die katze' ... } ``` � languagesN�id�dict�dtype�pa_typeF��default�init�repr�_typec�b�tjd�t|j��D����S)Nc�6�i|]}|tj����S�)�pa�string)�.0�langs �m/home/asafur/pinokio/api/open-webui.git/app/env/lib/python3.11/site-packages/datasets/features/translation.py� <dictcomp>z(Translation.__call__.<locals>.<dictcomp>*s ��O�O�O��$�� � � �O�O�O�)r�struct�sortedr��selfs r�__call__zTranslation.__call__)s+���y�O�O��t�~�8N�8N�O�O�O�P�P�Pr!�returnr c�P��ddlm��fd�t|j��D��S)z2Flatten the Translation feature into a dictionary.r )�Valuec�(��i|]}|�d����S)rr)r�kr)s �rr z'Translation.flatten.<locals>.<dictcomp>0s#���C�C�C�q��5�5��?�?�C�C�Cr!)�featuresr)r#r)r%r)s @r�flattenzTranslation.flatten,s9���#�#�#�#�#�#�C�C�C�C�F�4�>�,B�,B�C�C�C�Cr!)�__name__� __module__� __qualname__�__doc__�list�str�__annotations__rrrrrrrrr&r rr-rr!rr r s����������*�C�y�����B��� ����!�E�8�C�=�!�!�!�!�G�X�c�]�!�!�!���}�5�u�E�E�E�E�3�E�E�E�Q�Q�Q�D��}�d�3� �3E�.F�F�G�D�D�D�D�D�Dr!r c���eZdZUdZdZeeed<dZee ed<dZ ee ed<dZ e e ed<dZe eed<edd d � ��Ze ed <d �Zd �Zd�Zdedee dfffd�ZdS)�TranslationVariableLanguagesa`Feature` for translations with variable languages per example. Here for compatiblity with tfds. Args: languages (`dict`): A dictionary for each example mapping string language codes to one or more string translations. The languages present may vary from example to example. Returns: - `language` or `translation` (variable-length 1D `tf.Tensor` of `tf.string`): Language codes sorted in ascending order or plain text translations, sorted to align with language codes. Example: ```python >>> # At construction time: >>> datasets.features.TranslationVariableLanguages(languages=['en', 'fr', 'de']) >>> # During data generation: >>> yield { ... 'en': 'the cat', ... 'fr': ['le chat', 'la chatte,'] ... 'de': 'die katze' ... } >>> # Tensor returned : >>> { ... 'language': ['en', 'de', 'fr', 'fr'], ... 'translation': ['the cat', 'die katze', 'la chatte', 'le chat'], ... } ``` Nr� num_languagesrrrrFrrc��|jr!tt|j����nd|_|jrt|j��nd|_dS)N)rr#�set�lenr7r$s r� __post_init__z*TranslationVariableLanguages.__post_init__\sK��8<��P���D�N� 3� 3�4�4�4�D���48�N�L�S���0�0�0�����r!c��tjtjtj����tjtj����d���S)N��language� translation)rr"�list_rr$s rr&z%TranslationVariableLanguages.__call__`s>���y�b�h�r�y�{�{�&;�&;�B�H�UW�U^�U`�U`�La�La�b�b�c�c�cr!c �N��t|j��}t|��ddhkr|S|jrkt|��|z rYtdd�t t|��|z �����dd�|���d����g}|���D]S\�}t |t��r|��|f���2|� �fd�|D�����Ttt |���\}}||d�S) Nr>r?zSome languages in example (z, z) are not in valid set (z).c���g|]}�|f��Srr)r�elrs �r� <listcomp>z?TranslationVariableLanguages.encode_example.<locals>.<listcomp>ss���*E�*E�*E�"�D�"�:�*E�*E�*Er!r=) r9r� ValueError�joinr#�items� isinstancer3�append�extend�zip)r%�translation_dict�lang_set�translation_tuples�textr� translationsrs @r�encode_examplez+TranslationVariableLanguages.encode_examplecs|����t�~�&�&�� �� � �Z��$?� ?� ?�#� #� �^� ��$4� 5� 5�� @� ��S�d�i�i��s�CS�?T�?T�W_�?_�8`�8`�.a�.a�S�S�{�|E�|E�FN�|O�|O�S�S�S��� �  ��*�0�0�2�2� G� G�J�D�$��$��$�$� G�"�)�)�4��,�7�7�7�7�"�)�)�*E�*E�*E�*E��*E�*E�*E�F�F�F�F�#&�v�.@�'A�'A�"B�� �<�%�l�C�C�Cr!r'r c�d�ddlm}m}||d����||d����d�S)zCFlatten the TranslationVariableLanguages feature into a dictionary.r )�Sequencer)rr=)r,rSr))r%rSr)s rr-z$TranslationVariableLanguages.flattenzsU��-�-�-�-�-�-�-�-�!����x���1�1�#�8�E�E�(�O�O�4�4� � � r!)r.r/r0r1rrr2r4r7�intrr3rrrrrrr;r&rQr rr-rr!rr6r63s���������>!%�I�x��~�$�$�$�#'�M�8�C�=�'�'�'��B��� ����!�E�8�C�=�!�!�!�!�G�X�c�]�!�!�!���=�E�PU�V�V�V�E�3�V�V�V�M�M�M�d�d�d�D�D�D�. ��}�d�3� �3E�.F�F�G� � � � � � r!r6)� dataclassesrr�typingrrrrr �pyarrowrr,r r r6rr!r�<module>rXs���(�(�(�(�(�(�(�(�@�@�@�@�@�@�@�@�@�@�@�@�@�@������&�%�%�%�%�%�%� �$D�$D�$D�$D�$D�$D�$D� ��$D�N �M �M �M �M �M �M �M � ��M �M �M r!
Memory