� ���g�<���UddlZddlZddlZddlmZmZddlmZddlm Z m Z m Z m Z m Z ddlZddlZddlmZddlmZddlmZdd lmZmZdd lmZmZmZe r ddlZ d d l!m"Z"da#e e$e%e&d <ej'dkrdndZ(ej)d��ej)d��ej)d��ej)d��ej)d��ej)d��ej)d��ej)d��ej)d��ej)d��ej)d��ej)d��ej)d��ej)d��gZ*eGd�d ����Z+d!e$e%fd"�Z,d#d$d!e-fd%�Z.d#d$d!e/fd&�Z0d'ej1d!e/fd(�Z2d)e e$e%e$e/e$ej1e$d$fd!e$e/fd*�Z3dS)+�N)� dataclass�field)�BytesIO)� TYPE_CHECKING�Any�ClassVar�Optional�Union�)�config)�DownloadConfig)� array_cast)� is_local_path�xopen)�first_non_null_value�no_op_if_value_is_null�string_to_dict�)� FeatureType�_IMAGE_COMPRESSION_FORMATS�little�<�>z|b1�|u1z<u2z>u2z<i2z>i2z<u4z>u4z<i4z>i4z<f4z>f4z<f8z>f8c���eZdZUdZdZeeed<dZe ed<dZ eeed<dZ e eed<e je j��e j��d ���Ze eed <edd d � ��Zeed <d�Zdeeeeejdfdefd�Zddeddfd�Zdedeedfffd�Zdee je je j fde jfd�Z!de jde jfd�Z"dS)�Imagea�Image [`Feature`] to read image data from an image file. Input: The Image feature accepts as input: - A `str`: Absolute path to the image file (i.e. random access is allowed). - A `dict` with the keys: - `path`: String with relative path of the image file to the archive file. - `bytes`: Bytes of the image file. This is useful for archived files with sequential access. - An `np.ndarray`: NumPy array representing an image. - A `PIL.Image.Image`: PIL image object. Args: mode (`str`, *optional*): The mode to convert the image to. If `None`, the native mode of the image is used. decode (`bool`, defaults to `True`): Whether to decode the image data. If `False`, returns the underlying dictionary in the format `{"path": image_path, "bytes": image_bytes}`. Examples: ```py >>> from datasets import load_dataset, Image >>> ds = load_dataset("AI-Lab-Makerere/beans", split="train") >>> ds.features["image"] Image(decode=True, id=None) >>> ds[0]["image"] <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=500x500 at 0x15E52E7F0> >>> ds = ds.cast_column('image', Image(decode=False)) {'bytes': None, 'path': '/root/.cache/huggingface/datasets/downloads/extracted/b0a21163f78769a2cf11f58dfc767fb458fc7cea5c05dccc0144a2c0f0bc1292/train/healthy/healthy_train.85.jpg'} ``` N�modeT�decode�id�PIL.Image.Image�dtype��bytes�path�pa_typeF)�default�init�repr�_typec��|jS�N)r%)�selfs �g/home/asafur/pinokio/api/open-webui.git/app/env/lib/python3.11/site-packages/datasets/features/image.py�__call__zImage.__call__[s ���|���value�returnc��tjrddl}ntd���t |t ��rt j|��}t |t��r|dd�St |t��rd|d�St |t j ��rt|��St ||j j ��rt|��S|�d���=tj�|d��rd|�d��d�S|�d���|�d���+|�d��|�d��d�St%d|�d ����) aEncode example into a format for Arrow. Args: value (`str`, `np.ndarray`, `PIL.Image.Image` or `dict`): Data passed as input to Image feature. Returns: `dict` with "path" and "bytes" fields rN�4To support encoding images, please install 'Pillow'.�r$r#r$r"r#zUAn image sample should have one of 'path' or 'bytes' but they are missing or None in �.)r � PIL_AVAILABLE� PIL.Image� ImportError� isinstance�list�np�array�strr#�ndarray�encode_np_arrayr�encode_pil_image�get�osr$�isfile� ValueError)r,r0�PILs r-�encode_examplezImage.encode_example^s��� � � V� � � � � ��T�U�U� U� �e�T� "� "� $��H�U�O�O�E� �e�S� !� !� �!�D�1�1� 1� ��u� %� %� � �5�1�1� 1� ��r�z� *� *� �"�5�)�)� )� ��s�y�� /� /� �#�E�*�*� *� �Y�Y�v� � � *�r�w�~�~�e�F�m�/L�/L� *�!�5�9�9�V�+<�+<�=�=� =� �Y�Y�w� � � +�u�y�y��/@�/@�/L�"�Y�Y�w�/�/����6�9J�9J�K�K� K��p�hm�p�p�p��� r/c��|jstd���tjr ddl}ddl}nt d���|�i}|d|d}}|��7|�td|�d����t|��r|j � |��}�n|� d ��d }|� tj ��r tjn tj}t!||��} | �|�| d ��nd} t%| � ��} t'|d | ���5} t)| �����}ddd��n #1swxYwY|j � |��}n'|j � t)|����}|���|����|j jjj���|j�|��}|jr*|j|jkr|�|j��}|S)aqDecode example image file into image data. Args: value (`str` or `dict`): A string with the absolute image file path, a dictionary with keys: - `path`: String with absolute or relative image file path. - `bytes`: The bytes of the image file. token_per_repo_id (`dict`, *optional*): To access and decode image files from private repositories on the Hub, you can pass a dictionary repo_id (`str`) -> token (`bool` or `str`). Returns: `PIL.Image.Image` zMDecoding is disabled for this feature. Please use Image(decode=True) instead.rNz4To support decoding images, please install 'Pillow'.r$r#zCAn image should have one of 'path' or 'bytes' but both are None in r5z::������repo_id)�token�rb)�download_config)r� RuntimeErrorr r6r7� PIL.ImageOpsr8rDrr�open�split� startswith� HF_ENDPOINT�HUB_DATASETS_URL�HUB_DATASETS_HFFS_URLrrAr rr�read�load�getexif�ExifTags�Base� Orientation�ImageOps�exif_transposer�convert) r,r0�token_per_repo_idrEr$�bytes_�image� source_url�pattern�source_url_fieldsrJrL�fs r-�decode_examplezImage.decode_example�sn��$�{� p��n�o�o� o� � � V� � � � � � � � � ��T�U�U� U� � $� "� ��V�}�e�G�n�f�� �>��|� �!o�gl�!o�!o�!o�p�p�p� ��&�&�3��I�N�N�4�0�0�E�E�!%���D�!1�!1�"�!5�J�&�0�0��1C�D�D�:��/�/�#�9�� )7�z�7�(K�(K�%�O`�Ol�)�-�-�.?� �.J�K�K�K�rv��'5�5�&A�&A�&A�O��t�T�?�K�K�K�3�q�!(������!2�!2��3�3�3�3�3�3�3�3�3�3�3����3�3�3�3��I�N�N�6�2�2�E�E��I�N�N�7�6�?�?�3�3�E� � � � � � � �=�=�?�?� � �s�y�1�6�B� C� C� O��L�/�/��6�6�E� �9� -���e�j�0�0��M�M�$�)�,�,�E�� s�5"E#�#E'�*E'rc�N�ddlm}|jr|n|d��|d��d�S)zfIf in the decodable state, return the feature itself, otherwise flatten the feature into a dictionary.r)�Value�binary�stringr")�featuresrgr)r,rgs r-�flattenz Image.flatten�sK��#�#�#�#�#�#��{� �D�D���x�����h����� r/�storagec��tj�|j��rrtjdgt |��ztj�����}tj�||gddg|� �����}�n�tj� |j��rrtjdgt |��ztj �����}tj�||gddg|� �����}�n tj� |j���r|j� d��dkr|�d��}n8tjdgt |��ztj�����}|j� d��dkr|�d��}n8tjdgt |��ztj �����}tj�||gddg|� �����}n�tj�|j��r�tjd�|���D��tj�����}tjdgt |��ztj �����}tj�||gddg|� �����}t#||j��S)a'Cast an Arrow array to the Image arrow storage type. The Arrow types that can be converted to the Image pyarrow storage type are: - `pa.string()` - it must contain the "path" data - `pa.binary()` - it must contain the image bytes - `pa.struct({"bytes": pa.binary()})` - `pa.struct({"path": pa.string()})` - `pa.struct({"bytes": pa.binary(), "path": pa.string()})` - order doesn't matter - `pa.list(*)` - it must contain the image array data Args: storage (`Union[pa.StringArray, pa.StructArray, pa.ListArray]`): PyArrow array to cast. Returns: `pa.StructArray`: Array in the Image arrow storage type, that is `pa.struct({"bytes": pa.binary(), "path": pa.string()})`. N��typer#r$��maskrc�d�g|]-}|�'ttj|����dnd��.S)Nr#)r?r;r<)�.0�arrs r-� <listcomp>z&Image.cast_storage.<locals>.<listcomp>�s8��u�u�u�Z]�C�O����#���/�/��8�8�QU�u�u�ur/)�pa�types� is_stringror<�lenrh� StructArray� from_arrays�is_null� is_binaryri� is_struct�get_field_indexr�is_list� to_pylistrr%)r,rl� bytes_array� path_arrays r-� cast_storagezImage.cast_storage�s���& �8� � �g�l� +� +� ��(�D�6�C��L�L�#8�r�y�{�{�K�K�K�K��n�0�0�+�w�1G�'�SY�IZ�ah�ap�ap�ar�ar�0�s�s�G�G� �X� � �� � -� -� ���4�&�3�w�<�<�"7�b�i�k�k�J�J�J�J��n�0�0�'�:�1F��RX�HY�`g�`o�`o�`q�`q�0�r�r�G�G� �X� � �� � -� -� ��|�+�+�G�4�4��9�9�%�m�m�G�4�4� � � �h��v��G� � �'<�2�9�;�;�O�O�O� ��|�+�+�F�3�3�q�8�8�$�]�]�6�2�2� � ��X�t�f�s�7�|�|�&;�"�)�+�+�N�N�N� ��n�0�0�+�z�1J�W�V\�L]�dk�ds�ds�du�du�0�v�v�G�G� �X� � �g�l� +� +� ��(�u�u�ah�ar�ar�at�at�u�u�u��Y�[�[����K���4�&�3�w�<�<�"7�b�i�k�k�J�J�J�J��n�0�0��j�)�G�V�+<�;�CV�CV�CX�CX�1���G��'�4�<�0�0�0r/c����td����tj�fd�|���D��tj�����}tjd�|�d�����D��tj�����}tj�||gddg|� �����}t||j ��S)a8Embed image files into the Arrow array. Args: storage (`pa.StructArray`): PyArrow array to embed. Returns: `pa.StructArray`: Array in the Image arrow storage type, that is `pa.struct({"bytes": pa.binary(), "path": pa.string()})`. c�~�t|d��5}|���}ddd��n #1swxYwY|S)NrK)rrU)r$rdr_s r-� path_to_bytesz*Image.embed_storage.<locals>.path_to_bytes sv���t�T�"�"� "�a������� "� "� "� "� "� "� "� "� "� "� "���� "� "� "� "��Ms �2�6�6c�Z��g|]'}|�!|d��|d��n |dnd��(S)Nr#r$�)rs�xr�s �r-ruz'Image.embed_storage.<locals>.<listcomp>sU��� � � ��UV�Ta�Q�w�Z�-?���q��y�)�)�)�Q�w�Z�Z�gk� � � r/rnc�T�g|]%}|�tj�|��nd��&Sr+)rBr$�basename)rsr$s r-ruz'Image.embed_storage.<locals>.<listcomp>s2�� p� p� p�d�t�'7�R�W� � �d� #� #� #�T� p� p� pr/r$r#rp) rrvr<r�rhrrirzr{r|rr%)r,rlr�r�r�s @r-� embed_storagezImage.embed_storage�s��� � � � � � � �h� � � � � �*�*�,�,� � � �����  � � � ��X� p� p�g�m�m�\b�Nc�Nc�Nm�Nm�No�No� p� p� p����� � � � ��.�,�,�k�:�-F��RX�HY�`k�`s�`s�`u�`u�,�v�v���'�4�<�0�0�0r/r+)#�__name__� __module__� __qualname__�__doc__rr r=�__annotations__r�boolrr!rrv�structrhrir%rrr)r.r r#�dictr;r>rFrerk� StringArrayrz� ListArrayr�r�r�r/r-rr-s��������"�"�H�D�(�3�-�����F�D�����B��� ����,�E�8�C�=�,�,�,�&�R�Y������i�b�i�k�k�'R�'R�S�S�G�X�c�]�S�S�S���w�U��?�?�?�E�3�?�?�?����%�E�#�u�d�B�J�HY�*Y�$Z�%�_c�%�%�%�%�N;�;�D�;�EV�;�;�;�;�z  ��}�d�3� �3E�.F�F�G�  �  �  �  �,1�E�"�.�"�.�"�,�*V�$W�,1�\^�\j�,1�,1�,1�,1�\1�R�^�1���1�1�1�1�1�1r/rr1c�X�tjrddl}ntd���t�}|j���tt|jj � ����t|jj � ����z��atS)Nrr3) r r6r7r8rrr'r:�set�OPEN�keys�SAVE)rEs r-�list_image_compression_formatsr�s��� ��R�������P�Q�Q�Q�"�)� � ������%)�#�c�i�n�.A�.A�.C�.C�*D�*D�s�3�9�>�K^�K^�K`�K`�Ga�Ga�*a�%b�%b�"� %�%r/r`r c���t��}|jt��vr|j}n |jdvrdnd}|�||���|���S)zmConvert a PIL Image object to bytes using native compression if possible, otherwise use PNG/TIFF compression.)�1�L�LA�RGB�RGBA�PNG�TIFF)�format)rr�r�r�save�getvalue)r`�bufferr�s r-�image_to_bytesr�+sd�� �Y�Y�F� �|�5�7�7�7�7������*�(G�G�G���V�� �J�J�v�f�J�%�%�%� �?�?� � �r/c�p�t|d��r|jdkr |jdd�Sdt|��d�S)N�filename�r4)�hasattrr�r�)r`s r-r@r@6sF���u�j�!�!�>�e�n��&:�&:����6�6�6��~�e�'<�'<�=�=�=r/r<c��tjrddl}ntd���|j}|jdkr|jnt }|j}|j}d}|j dd�rP|dvrtd|�d|�d����tjd ��}||krtj d |�d |�d ���n�|tvr|}n�|d krk||zt|��z}tj|��tvr0tj|��}tj d |�d |�d ���n |dz}|d k�k|�td|�dt�����|j�|�|����}dt'|��d�S)Nrr3�=r )�u�izUnsupported array dtype z for image encoding. Only z' is supported for multi-channel arrays.rzDowncasting array dtype z to z to be compatible with 'Pillow'rzCannot downcast dtype z- to a valid image dtype. Valid image dtypes: r4)r r6r7r8r!� byteorder�_NATIVE_BYTEORDER�kind�itemsize�shape� TypeErrorr;�warnings�warn�_VALID_IMAGE_ARRAY_DTPYESr=r� fromarray�astyper�) r<rEr!�dtype_byteorder� dtype_kind�dtype_itemsize� dest_dtype� dtype_strr`s r-r?r?=s��� ��R�������P�Q�Q�Q� �K�E�).��C�)?�)?�e�o�o�EV�O���J��^�N��J� �{�1�2�2��� �Z� '� '���5���J������ ��X�e�_�_� � �J� � � �M�k�U�k�k� �k�k�k� l� l� l�� �+� +� +�� � ���!�!�'�*�4�s�>�7J�7J�J�I��x� �"�"�&?�?�?��X�i�0�0� �� �o��o�o�J�o�o�o�p�p�p���1�$����!�!� � ��x��x�x�]v�x�x��� � �I� � �� � �Z� 8� 8� 9� 9�E��>�%�#8�#8� 9� 9�9r/�objsc���tjrddl}ntd���|r�t |��\}}t |t ��r d�|D��St |tj��r"tt����fd�|D��St ||j j ��r"tt����fd�|D��S|S|S)zmEncode a list of objects into a format suitable for creating an extension array of type `ImageExtensionType`.rNr3c� �g|] }|�|dd�nd�� S)Nr4r�)rs�objs r-ruz2objects_to_list_of_image_dicts.<locals>.<listcomp>ts*��^�^�^�RU�C�O�S�4�0�0�0��^�^�^r/c�&��g|] }�|����Sr�r��rsr��obj_to_image_dict_funcs �r-ruz2objects_to_list_of_image_dicts.<locals>.<listcomp>w�%���@�@�@�C�*�*�3�/�/�@�@�@r/c�&��g|] }�|����Sr�r�r�s �r-ruz2objects_to_list_of_image_dicts.<locals>.<listcomp>zr�r/) r r6r7r8rr9r=r;r>rr?rr@)r�rE�_r�r�s @r-�objects_to_list_of_image_dictsr�hs������R�������P�Q�Q�Q� � �%�d�+�+���3� �c�3� � � _�^�^�Y]�^�^�^� ^� �c�2�:� &� &� �%;�O�%L�%L� "�@�@�@�@�4�@�@�@� @� ��S�Y�_� -� -� �%;�<L�%M�%M� "�@�@�@�@�4�@�@�@� @��K�� r/)4rB�sysr�� dataclassesrr�ior�typingrrrr r �numpyr;�pyarrowrvr�r �download.download_configr �tabler�utils.file_utilsrr�utils.py_utilsrrrr7rErjrrr:r=r�r�r�r!r�rr�r#r�r�r@r>r?r�r�r/r-�<module>r�s0�� � � � � � � � � �����(�(�(�(�(�(�(�(�������@�@�@�@�@�@�@�@�@�@�@�@�@�@���������������5�5�5�5�5�5�������3�3�3�3�3�3�3�3�Y�Y�Y�Y�Y�Y�Y�Y�Y�Y��&�����%�%�%�%�%�%�37��H�T�#�Y�/�6�6�6��=�H�4�4�C�C�#�� �B�H�U�O�O� �B�H�U�O�O� �B�H�U�O�O� �B�H�U�O�O� �B�H�U�O�O� �B�H�U�O�O� �B�H�U�O�O� �B�H�U�O�O� �B�H�U�O�O� �B�H�U�O�O� �B�H�U�O�O� �B�H�U�O�O� �B�H�U�O�O� �B�H�U�O�O���$ �m1�m1�m1�m1�m1�m1�m1� ��m1�` &��S� � &� &� &� &��+�������>�-�>�$�>�>�>�>�(:�2�:�(:�$�(:�(:�(:�(:�V� ��S� �4��:�t�B�J�'7��>O�9P�P� Q�� �$�Z������r/
Memory