�
���g^ � � � d dl mZmZ d dlmZmZmZmZmZ d dl Z
erddl m Z e G d� d� � � � Z
e G d� d � � � � ZdS )
� )� dataclass�field)�
TYPE_CHECKING�Any�ClassVar�Optional�UnionN� )� FeatureTypec �� � e Zd ZU dZee ed< dZee ed< dZ e
e ed< dZ e
e ed< e
d dd� � � Zeed
< d � Zd ed
eed
f f fd�ZdS )� Translationa `Feature` for translations with fixed languages per example.
Here for compatiblity with tfds.
Args:
languages (`dict`):
A dictionary for each example mapping string language codes to string translations.
Example:
```python
>>> # At construction time:
>>> datasets.features.Translation(languages=['en', 'fr', 'de'])
>>> # During data generation:
>>> yield {
... 'en': 'the cat',
... 'fr': 'le chat',
... 'de': 'die katze'
... }
```
� languagesN�id�dict�dtype�pa_typeF��default�init�repr�_typec �b � t j d� t | j � � D � � � � S )Nc �6 � i | ]}|t j � � ��S � )�pa�string)�.0�langs �m/home/asafur/pinokio/api/open-webui.git/app/env/lib/python3.11/site-packages/datasets/features/translation.py�
<dictcomp>z(Translation.__call__.<locals>.<dictcomp>* s � �O�O�O��$�� � � �O�O�O� )r �struct�sortedr ��selfs r �__call__zTranslation.__call__) s+ � ��y�O�O��t�~�8N�8N�O�O�O�P�P�Pr! �returnr c �P �� ddl m� �fd�t | j � � D � � S )z2Flatten the Translation feature into a dictionary.r
)�Valuec �( �� i | ]}| �d � � ��S )r r )r �kr) s �r r z'Translation.flatten.<locals>.<dictcomp>0 s# �� �C�C�C�q��5�5��?�?�C�C�Cr! )�featuresr) r# r )r% r) s @r �flattenzTranslation.flatten, s9 �� �#�#�#�#�#�#�C�C�C�C�F�4�>�,B�,B�C�C�C�Cr! )�__name__�
__module__� __qualname__�__doc__�list�str�__annotations__r r r r r r r r r& r r r- r r! r r
r
s� � � � � � � �� �* �C�y���� �B���
� � � �!�E�8�C�=�!�!�!� !�G�X�c�]�!�!�!���}�5�u�E�E�E�E�3�E�E�E�Q� Q� Q�D��}�d�3�
�3E�.F� F�G� D� D� D� D� D� Dr! r
c �� � e Zd ZU dZdZee ed< dZee ed< dZ
ee ed< dZ e
e ed< dZe
e ed< ed d d �
� � Ze ed < d � Zd
� Zd� Zdedee df f fd�ZdS )� TranslationVariableLanguagesa `Feature` for translations with variable languages per example.
Here for compatiblity with tfds.
Args:
languages (`dict`):
A dictionary for each example mapping string language codes to one or more string translations.
The languages present may vary from example to example.
Returns:
- `language` or `translation` (variable-length 1D `tf.Tensor` of `tf.string`):
Language codes sorted in ascending order or plain text translations, sorted to align with language codes.
Example:
```python
>>> # At construction time:
>>> datasets.features.TranslationVariableLanguages(languages=['en', 'fr', 'de'])
>>> # During data generation:
>>> yield {
... 'en': 'the cat',
... 'fr': ['le chat', 'la chatte,']
... 'de': 'die katze'
... }
>>> # Tensor returned :
>>> {
... 'language': ['en', 'de', 'fr', 'fr'],
... 'translation': ['the cat', 'die katze', 'la chatte', 'le chat'],
... }
```
Nr �
num_languagesr r r r Fr r c � � | j r!t t | j � � � � nd | _ | j rt | j � � nd | _ d S )N)r r# �set�lenr7 r$ s r �
__post_init__z*TranslationVariableLanguages.__post_init__\ sK � �8<��P���D�N� 3� 3�4�4�4�D���48�N� L�S�� � 0� 0� 0�����r! c � � t j t j t j � � � � t j t j � � � � d�� � S )N��language� translation)r r" �list_r r$ s r r& z%TranslationVariableLanguages.__call__` s> � ��y�b�h�r�y�{�{�&;�&;�B�H�UW�U^�U`�U`�La�La�b�b�c�c�cr! c
�N �� t | j � � }t |� � ddhk r|S | j rkt |� � |z
rYt dd� t t |� � |z
� � � � � dd� |� � � d�� � �g }|� � � D ]S\ �}t
|t � � r|� �|f� � �2|� �fd�|D � � � � �Tt t |� � � \ }}||d�S ) Nr> r? zSome languages in example (z, z) are not in valid set (z).c � �� g | ]}�|f��S r r )r �elr s �r �
<listcomp>z?TranslationVariableLanguages.encode_example.<locals>.<listcomp>s s �� �*E�*E�*E�"�D�"�:�*E�*E�*Er! r= ) r9 r �
ValueError�joinr# �items�
isinstancer3 �append�extend�zip)r% �translation_dict�lang_set�translation_tuples�textr � translationsr s @r �encode_examplez+TranslationVariableLanguages.encode_examplec s| �� ��t�~�&�&�� � � � �Z� �$?� ?� ?�#� #�
�^� ��$4� 5� 5�� @� � � S�d�i�i��s�CS�?T�?T�W_�?_�8`�8`�.a�.a� S� S�{� |E� |E� FN� |O� |O� S� S� S�� �
� ��*�0�0�2�2� G� G�J�D�$��$��$�$�
G�"�)�)�4��,�7�7�7�7�"�)�)�*E�*E�*E�*E��*E�*E�*E�F�F�F�F� #&�v�.@�'A�'A�"B� � �<� %�l�C�C�Cr! r' r c �d � ddl m}m} | |d� � � � | |d� � � � d�S )zCFlatten the TranslationVariableLanguages feature into a dictionary.r
)�Sequencer) r r= )r, rS r) )r% rS r) s r r- z$TranslationVariableLanguages.flattenz sU � �-�-�-�-�-�-�-�-� !����x� � �1�1�#�8�E�E�(�O�O�4�4�
�
�
r! )r. r/ r0 r1 r r r2 r4 r7 �intr r3 r r r r r r r; r& rQ r r r- r r! r r6 r6 3 s � � � � � � � � �> !%�I�x��~�$�$�$�#'�M�8�C�=�'�'�'� �B���
� � � �!�E�8�C�=�!�!�!� !�G�X�c�]�!�!�!��� =�E�PU�V�V�V�E�3�V�V�V�M� M� M�d� d� d�D� D� D�.
��}�d�3�
�3E�.F� F�G�
�
�
�
�
�
r! r6 )� dataclassesr r �typingr r r r r �pyarrowr r, r r
r6 r r! r �<module>rX s� �� (� (� (� (� (� (� (� (� @� @� @� @� @� @� @� @� @� @� @� @� @� @� � � � � � &�%�%�%�%�%�%� �$D� $D� $D� $D� $D� $D� $D� ��$D�N �M
� M
� M
� M
� M
� M
� M
� ��M
� M
� M
r!