� ���g�/����ddlZddlmZmZddlmZddlmZmZm Z m Z m Z ddl Z ddlZddlmZddlmZddlmZdd lmZmZdd lmZmZerd d lmZeGd �d����ZdS)�N)� dataclass�field)�BytesIO)� TYPE_CHECKING�Any�ClassVar�Optional�Union�)�config)�DownloadConfig)� array_cast)�xopen� xsplitext)�no_op_if_value_is_null�string_to_dict�)� FeatureTypec ��eZdZUdZdZeeed<dZe ed<dZ e ed<dZ ee ed<dZ ee ed <ejej��ej��d ���Zeeed <edd d � ��Ze ed<d�Zdee eefdefd�Z ddedeee ee e dffdefd�Zdedee dfffd�Zdeejejfdejfd�Z dejdejfd�Z!dS)�Audioa1Audio [`Feature`] to extract audio data from an audio file. Input: The Audio feature accepts as input: - A `str`: Absolute path to the audio file (i.e. random access is allowed). - A `dict` with the keys: - `path`: String with relative path of the audio file to the archive file. - `bytes`: Bytes content of the audio file. This is useful for archived files with sequential access. - A `dict` with the keys: - `path`: String with relative path of the audio file to the archive file. - `array`: Array containing the audio sample - `sampling_rate`: Integer corresponding to the sampling rate of the audio sample. This is useful for archived files with sequential access. Args: sampling_rate (`int`, *optional*): Target sampling rate. If `None`, the native sampling rate is used. mono (`bool`, defaults to `True`): Whether to convert the audio signal to mono by averaging samples across channels. decode (`bool`, defaults to `True`): Whether to decode the audio data. If `False`, returns the underlying dictionary in the format `{"path": audio_path, "bytes": audio_bytes}`. Example: ```py >>> from datasets import load_dataset, Audio >>> ds = load_dataset("PolyAI/minds14", name="en-US", split="train") >>> ds = ds.cast_column("audio", Audio(sampling_rate=16000)) >>> ds[0]["audio"] {'array': array([ 2.3443763e-05, 2.1729663e-04, 2.2145823e-04, ..., 3.8356509e-05, -7.3497440e-06, -2.1754686e-05], dtype=float32), 'path': '/root/.cache/huggingface/datasets/downloads/extracted/f14948e0e84be638dd7943ac36518a4cf3324e8b7aa331c5ab11541518e9368c/en-US~JOINT_ACCOUNT/602ba55abb1e6d0fbce92065.wav', 'sampling_rate': 16000} ``` N� sampling_rateT�mono�decode�id�dict�dtype��bytes�path�pa_typeF)�default�init�repr�_typec��|jS�N)r )�selfs �g/home/asafur/pinokio/api/open-webui.git/app/env/lib/python3.11/site-packages/datasets/features/audio.py�__call__zAudio.__call__Js ���|���value�returnc� � ddl}n"#t$r}td��|�d}~wwxYwt|t��rd|d�St|t��r|dd�Sd|vrJt ��}|�||d|dd���|���dd�S|�d ����\tj � |d ���r6|d � d ���r|�d���td ���|�d ��rGtj|d tj� ���tj��dz }n=tj|d dd����tj��dz }t d��}|�|||dd���|���dd�Sd|�d ��d�S|�d ���|�d ���+|�d ��|�d ��d�St)d|�d����)z�Encode example into a format for Arrow. Args: value (`str` or `dict`): Data passed as input to Audio feature. Returns: `dict` rNz;To support encoding audio data, please install 'soundfile'.r�arrayr�wav)�formatr�pcmzBTo use PCM files, please specify a 'sampling_rate' in Audio objectr)ri��h�r)r�moder*zUAn audio sample should have one of 'path' or 'bytes' but they are missing or None in �.)� soundfile� ImportError� isinstance�strrr�write�getvalue�get�osr�isfile�endswith�KeyError�np� frombuffer�int16�astype�float32�memmap� ValueError)r'r+�sf�err�buffer� bytes_values r(�encode_examplezAudio.encode_exampleMs��� f� "� "� "� "� "��� f� f� f��[�\�\�be� e����� f���� �e�S� !� !�! �!�5�1�1� 1� ��u� %� %� �"�D�1�1� 1� �� � ��Y�Y�F� �H�H�V�U�7�^�U�?�-C�E�H� R� R� R�#�_�_�.�.��=�=� =� �Y�Y�v� � � *�r�w�~�~�e�F�m�/L�/L� *��V�}�%�%�e�,�,� B��9�9�_�-�-�5�"�#g�h�h�h��9�9�W�%�%�k�"$�-��g��b�h�"O�"O�"O�"V�"V�WY�Wa�"b�"b�ej�"j�K�K�"$�)�E�&�M��3�"O�"O�"O�"V�"V�WY�Wa�"b�"b�ej�"j�K� ����������e�O�.D�U��S�S�S�!'���!2�!2�D�A�A�A�!%�u�y�y��/@�/@�A�A�A� �Y�Y�w� � � +�u�y�y��/@�/@�/L�"�Y�Y�w�/�/����6�9J�9J�K�K� K��p�hm�p�p�p��� s�� &�!�&�token_per_repo_idc��|jstd���|d�|dt|d��fn |ddf\}}|�|�td|�d���� ddl}ddl}n"#t $r}t d��|�d}~wwxYw|�/t|��d d d����nd}tj s|d krtd ���tj s|d krtd ���|��|pi}|� d��d} | � tj��r tjn tj} t#| | ��} | �|�| d��nd} t'| ���} t)|d| ���5}|�|��\}}ddd��n #1swxYwYn|�|��\}}|j}|jr|�|��}|jr/|j|kr$|�|||j���}|j}|||d�S)aDecode example audio file into audio data. Args: value (`dict`): A dictionary with keys: - `path`: String with relative audio file path. - `bytes`: Bytes of the audio file. token_per_repo_id (`dict`, *optional*): To access and decode audio files from private repositories on the Hub, you can pass a dictionary repo_id (`str`) -> token (`bool` or `str`) Returns: `dict` zMDecoding is disabled for this feature. Please use Audio(decode=True) instead.rNrzJAn audio sample should have one of 'path' or 'bytes' but both are None in r5rzJTo support decoding audio files, please install 'librosa' and 'soundfile'.r�opusz�Decoding 'opus' files requires system library 'libsndfile'>=1.0.31, You can try to update `soundfile` python library: `pip install "soundfile>=0.12.1"`. �mp3z�Decoding 'mp3' files requires system library 'libsndfile'>=1.1.0, You can try to update `soundfile` python library: `pip install "soundfile>=0.12.1"`. z::������repo_id)�token�rb)�download_config)�orig_sr� target_sr)rr.r)r� RuntimeErrorrrG�librosar6r7r�lowerr �IS_OPUS_SUPPORTED�IS_MP3_SUPPORTED�split� startswith� HF_ENDPOINT�HUB_DATASETS_URL�HUB_DATASETS_HFFS_URLrr<r r�read�Tr�to_monor�resample)r'r+rMr�filerYrHrI� audio_format� source_url�pattern�source_url_fieldsrSrU�fr.rs r(�decode_examplezAudio.decode_example~s���&�{� p��n�o�o� o�AF�w��A[�e�F�m�W�U�7�^�%<�%<�=�=�bg�hn�bo�qu�av� ��d� �<�D�L��r�jo�r�r�r�s�s� s� u� �N�N�N� "� "� "� "� "��� u� u� u��j�k�k�qt� t����� u����:>�9I�y����q�)�!�"�"�-�3�3�5�5�5�t� ��'� �L�F�,B�,B��h��� ��(� �\�U�-B�-B��h��� � �<� 1� 7�R� ����D�)�)�"�-�J�+5�+@�+@��AS�+T�+T�v��'�'�Z`�Zv� �!/�z�7� C� C� �K\�Kh�%�)�)�*;�I�*F�G�G�G�nr�E�,�5�9�9�9�O��t�T�?�C�C�C� 2�q�')�w�w�q�z�z�$��}� 2� 2� 2� 2� 2� 2� 2� 2� 2� 2� 2���� 2� 2� 2� 2��$&�7�7�4�=�=� �E�=���� �9� +��O�O�E�*�*�E� � � /�$�"4� �"E�"E��$�$�U�M�T�M_�$�`�`�E� �.�M��u�}�M�M�Ms*�!A*�* B �4B�B �*G�G�Grc�h�ddlm}|jrtd���|d��|d��d�S)z[If in the decodable state, raise an error, otherwise flatten the feature into a dictionary.r)�Valuez'Cannot flatten a decoded Audio feature.�binary�stringr)�featuresrnrrG)r'rns r(�flattenz Audio.flatten�sS��#�#�#�#�#�#� �;� H��F�G�G� G��U�8�_�_��E�(�O�O� � � r*�storagec���tj�|j��rrtjdgt |��ztj�����}tj�||gddg|� �����}�n;tj� |j��rrtjdgt |��ztj �����}tj�||gddg|� �����}�n�tj� |j��rL|j� d��r2tjd�|���D����}�n5tj� |j���r|j�d��dkr|�d��}n8tjdgt |��ztj�����}|j�d��dkr|�d��}n8tjdgt |��ztj �����}tj�||gddg|� �����}t#||j��S) a�Cast an Arrow array to the Audio arrow storage type. The Arrow types that can be converted to the Audio pyarrow storage type are: - `pa.string()` - it must contain the "path" data - `pa.binary()` - it must contain the audio bytes - `pa.struct({"bytes": pa.binary()})` - `pa.struct({"path": pa.string()})` - `pa.struct({"bytes": pa.binary(), "path": pa.string()})` - order doesn't matter Args: storage (`Union[pa.StringArray, pa.StructArray]`): PyArrow array to cast. Returns: `pa.StructArray`: Array in the Audio arrow storage type, that is `pa.struct({"bytes": pa.binary(), "path": pa.string()})` N��typerr��maskr.c�X�g|]'}|�!t���|��nd��(Sr&)rrL)�.0�xs r(� <listcomp>z&Audio.cast_storage.<locals>.<listcomp>�s5��r�r�r�YZ�Q�]���� 6� 6�q� 9� 9� 9�PT�r�r�rr*r)�pa�types� is_stringrvr.�lenro� StructArray� from_arrays�is_null� is_binaryrp� is_struct�get_all_field_indices� to_pylist�get_field_indexrrr )r'rs� bytes_array� path_arrays r(� cast_storagezAudio.cast_storage�s`��$ �8� � �g�l� +� +� w��(�D�6�C��L�L�#8�r�y�{�{�K�K�K�K��n�0�0�+�w�1G�'�SY�IZ�ah�ap�ap�ar�ar�0�s�s�G�G� �X� � �� � -� -� w���4�&�3�w�<�<�"7�b�i�k�k�J�J�J�J��n�0�0�'�:�1F��RX�HY�`g�`o�`o�`q�`q�0�r�r�G�G� �X� � �� � -� -� w�'�,�2T�2T�U\�2]�2]� w��h�r�r�^e�^o�^o�^q�^q�r�r�r�s�s�G�G� �X� � �� � -� -� w��|�+�+�G�4�4��9�9�%�m�m�G�4�4� � � �h��v��G� � �'<�2�9�;�;�O�O�O� ��|�+�+�F�3�3�q�8�8�$�]�]�6�2�2� � ��X�t�f�s�7�|�|�&;�"�)�+�+�N�N�N� ��n�0�0�+�z�1J�W�V\�L]�dk�ds�ds�du�du�0�v�v�G��'�4�<�0�0�0r*c����td����tj�fd�|���D��tj�����}tjd�|�d�����D��tj�����}tj�||gddg|� �����}t||j ��S)a8Embed audio files into the Arrow array. Args: storage (`pa.StructArray`): PyArrow array to embed. Returns: `pa.StructArray`: Array in the Audio arrow storage type, that is `pa.struct({"bytes": pa.binary(), "path": pa.string()})`. c�~�t|d��5}|���}ddd��n #1swxYwY|S)NrT)rrb)rrk�bytes_s r(� path_to_bytesz*Audio.embed_storage.<locals>.path_to_bytessv���t�T�"�"� "�a������� "� "� "� "� "� "� "� "� "� "� "���� "� "� "� "��Ms �2�6�6c�Z��g|]'}|�!|d��|d��n |dnd��(S)Nrr�)rzr{r�s �r(r|z'Audio.embed_storage.<locals>.<listcomp>sU��� � � ��UV�Ta�Q�w�Z�-?���q��y�)�)�)�Q�w�Z�Z�gk� � � r*ruc�T�g|]%}|�tj�|��nd��&Sr&)r=r�basename)rzrs r(r|z'Audio.embed_storage.<locals>.<listcomp>s2�� p� p� p�d�t�'7�R�W� � �d� #� #� #�T� p� p� pr*rrrw) rr}r.r�rorrpr�r�r�rr )r'rsr�r�r�s @r(� embed_storagezAudio.embed_storage�s��� � � � � � � �h� � � � � �*�*�,�,� � � �����  � � � ��X� p� p�g�m�m�\b�Nc�Nc�Nm�Nm�No�No� p� p� p����� � � � ��.�,�,�k�:�-F��RX�HY�`k�`s�`s�`u�`u�,�v�v���'�4�<�0�0�0r*r&)"�__name__� __module__� __qualname__�__doc__rr �int�__annotations__r�boolrrr9rrr}�structrorpr rrr$r)r rrrLrlrr� StringArrayr�r�r�r�r*r(rrs�������)�)�V$(�M�8�C�=�'�'�'��D�$�����F�D�����B��� ����!�E�8�C�=�!�!�!�&�R�Y������i�b�i�k�k�'R�'R�S�S�G�X�c�]�S�S�S���w�U��?�?�?�E�3�?�?�?����/�E�#�u�d�*:�$;�/��/�/�/�/�d]a�CN�CN��CN�.6�t�C��s�D�RV��AW�<W�7X�.Y�CN� �CN�CN�CN�CN�J  ��}�d�3� �3E�.F�F�G�  �  �  �  �$1�E�"�.�"�.�*H�$I�$1�b�n�$1�$1�$1�$1�L1�R�^�1���1�1�1�1�1�1r*r)r=� dataclassesrr�ior�typingrrrr r �numpyrA�pyarrowr}�r �download.download_configr �tabler�utils.file_utilsrr�utils.py_utilsrrrqrrr�r*r(�<module>r�sD�� � � � �(�(�(�(�(�(�(�(�������@�@�@�@�@�@�@�@�@�@�@�@�@�@���������������5�5�5�5�5�5�������/�/�/�/�/�/�/�/�C�C�C�C�C�C�C�C��&�%�%�%�%�%�%� �}1�}1�}1�}1�}1�}1�}1� ��}1�}1�}1r*
Memory