� ���g A���dZddlZddlZddlZddlZddlmZddlmZddl m Z ddl m Z ddl mZmZmZmZmZmZmZmZddlmZdd lmZd d lmZmZmZmZmZm Z m!Z!m"Z"d d l#m$Z$d d l%m&Z&m'Z'd dl(m)Z)erd dl*m+Z+e j,e-��Z.ej/d��Z0dZ1ddd�Z2eGd�d����Z3e" d:dee3dee4de4de4dee4dee4deee4e4fdeee5ee5ffd���Z6Gd�d e��Z7Gd!�d"e��Z8 d:d#d$d%edee4deee4e4fdee4ddf d&�Z9d'e5fd(�Z:d)e5fd*�Z;d+e5fd,�Z<d#d$d-e4ddfd.�Z=d#d$d/ed0e>d-e4ddf d1�Z?d/ed2e3d0e>dee4fd3�Z@d4eed5e4de8fd6�ZAd#d$d7ee4d0e>deefd8�ZBd#d$d7ee4d0e>deefd9�ZCdS);z.Git LFS related type definitions and utilities�N)� dataclass)�ceil)�getsize)�Path)� TYPE_CHECKING�BinaryIO�Dict�Iterable�List�Optional�Tuple� TypedDict)�unquote)� constants�)�build_hf_headers�fix_hf_endpoint_in_url� get_session�hf_raise_for_status� http_backoff�logging�tqdm�validate_hf_hub_args)� SliceFileObj)�sha256� sha_fileobj)�is_tqdm_disabled)�CommitOperationAddz^[0-9a-f]{40}$zlfs-multipart-uploadzapplication/vnd.git-lfs+json)�Acceptz Content-Typec��eZdZUdZeed<eed<eed<edefd���Z edefd���Z ed e fd ���Z d S) � UploadInfoad Dataclass holding required information to determine whether a blob should be uploaded to the hub using the LFS protocol or the regular protocol Args: sha256 (`bytes`): SHA256 hash of the blob size (`int`): Size in bytes of the blob sample (`bytes`): First 512 bytes of the blob r�size�sample�pathc���t|��}tj|d��5}|�d��dd�}t |��}ddd��n #1swxYwY||||���S)N�rb��r"rr#)r�io�open�peekr)�clsr$r"�filer#�shas �c/home/asafur/pinokio/api/open-webui.git/app/env/lib/python3.11/site-packages/huggingface_hub/lfs.py� from_pathzUploadInfo.from_pathOs����t�}�}�� �W�T�4� � � $�D��Y�Y�s�^�^�D�S�D�)�F��d�#�#�C� $� $� $� $� $� $� $� $� $� $� $���� $� $� $� $��s��S��8�8�8�8s�-A�A"�%A"�datac��t|�����}|t|��|dd�|���S)Nr')r"r#r)r�digest�len)r,r1r.s r/� from_byteszUploadInfo.from_bytesWs>���T�l�l�!�!�#�#���s��D� � �$�t��t�*�S�A�A�A�A��fileobjc��|�d��}|�dtj��t |��}|���}|�dtj��||||���S)Nr'rr()�read�seekr)�SEEK_SETr�tell)r,r7r#r.r"s r/� from_fileobjzUploadInfo.from_fileobj\sp�����c�"�"��� � �Q�� �$�$�$��'�"�"���|�|�~�~��� � �Q�� �$�$�$��s��S��8�8�8�8r6N) �__name__� __module__� __qualname__�__doc__�bytes�__annotations__�int� classmethod�strr0r5rr=�r6r/r!r!<s�������� � � �M�M�M� �I�I�I� �M�M�M��9�S�9�9�9��[�9��B�e�B�B�B��[�B��9�8�9�9�9��[�9�9�9r6r!� upload_infos�token� repo_type�repo_id�revision�endpoint�headers�returnc�8�|�|n tj}d}|tjvrtj|}|�d|�|�d�}dddgd�|D��d d �} |�d t|��i| d <it�t |� ���|pi�}t ���||| ���} t| ��| � ��} | � dd��} t| t��std���d�| D��d�| D��fS)a� Requests the LFS batch endpoint to retrieve upload instructions Learn more: https://github.com/git-lfs/git-lfs/blob/main/docs/api/batch.md Args: upload_infos (`Iterable` of `UploadInfo`): `UploadInfo` for the files that are being uploaded, typically obtained from `CommitOperationAdd.upload_info` repo_type (`str`): Type of the repo to upload to: `"model"`, `"dataset"` or `"space"`. repo_id (`str`): A namespace (user or an organization) and a repo name separated by a `/`. revision (`str`, *optional*): The git revision to upload to. headers (`dict`, *optional*): Additional headers to include in the request Returns: `LfsBatchInfo`: 2-tuple: - First element is the list of upload instructions from the server - Second element is an list of errors, if any Raises: [`ValueError`](https://docs.python.org/3/library/exceptions.html#ValueError) If an argument is invalid or the server response is malformed. [`HTTPError`](https://requests.readthedocs.io/en/latest/api/#requests.HTTPError) If the server returned an error. N��/z.git/info/lfs/objects/batch�upload�basic� multipartc�P�g|]#}|j���|jd���$S)��oidr")r�hexr")�.0rSs r/� <listcomp>z'post_lfs_batch_info.<locals>.<listcomp>�sG�� � � � ��}�(�(�*�*�� � � � � � r6r)� operation� transfers�objects� hash_algo�name�ref)rI�rN�jsonr^zMalformed response from serverc�4�g|]}d|v�t|����S��error)�_validate_batch_actions�rZ�objs r/r[z'post_lfs_batch_info.<locals>.<listcomp>�s+��O�O�O�#�G�3�<N�<N� �� %� %�<N�<N�<Nr6c�4�g|]}d|v�t|����Sre)�_validate_batch_errorrhs r/r[z'post_lfs_batch_info.<locals>.<listcomp>�s&��I�I�I��'�S�.�.� �s� #� #�.�.�.r6)r�ENDPOINT�REPO_TYPES_URL_PREFIXESr� LFS_HEADERSrr�postrrc�get� isinstance�list� ValueError) rHrIrJrKrLrMrN� url_prefix� batch_url�payload�resp� batch_infor^s r/�post_lfs_batch_inforyfsw��P$�/�x�x�Y�5G�H��J��I�5�5�5��6�y�A� ��N�N�j�N�'�N�N�N�I���{�+� � � '�  � � �� � �G��� �'�(�"3�"3�4����� �� �� '� '� '�� �=�b��G� �=�=� � �i��w� � G� G�D�����������J��n�n�Y��-�-�G� �g�t� $� $�;��9�:�:�:� P�O��O�O�O�I�I�w�I�I�I� �r6c�$�eZdZUeed<eed<dS)� PayloadPartT� partNumber�etagN)r>r?r@rDrCrFrGr6r/r{r{�s"��������O�O�O� �I�I�I�I�Ir6r{c�4�eZdZUdZeed<eeed<dS)�CompletionPayloadTz?Payload that will be sent to the Hub when uploading multi-part.rX�partsN)r>r?r@rArFrCr r{rGr6r/rr�s2�������I�I� �H�H�H� � � �����r6rr\r�lfs_batch_actionc���t|��|�d��}|�%t�d|j�d���dS|dd}t |��|d�d��}|�t |��|�di��}|�d��} t |d |� ��} | �M t| ��} n(#ttf$rtd | �d ����wxYwt||| | � ��nt|| ���|��t |��t |d |��} t��� | t||���|jj���|jjd����} t'| ��t�|j�d���dS)aw Handles uploading a given object to the Hub with the LFS protocol. Can be a No-op if the content of the file is already present on the hub large file storage. Args: operation (`CommitOperationAdd`): The add operation triggering this upload. lfs_batch_action (`dict`): Upload instructions from the LFS batch endpoint for this object. See [`~utils.lfs.post_lfs_batch_info`] for more details. headers (`dict`, *optional*): Headers to include in the request, including authentication and user agent headers. Raises: [`ValueError`](https://docs.python.org/3/library/exceptions.html#ValueError) If `lfs_batch_action` is improperly formatted [`HTTPError`](https://requests.readthedocs.io/en/latest/api/#requests.HTTPError) If the upload resulted in an error �actionsNzContent of file z. is already present upstream - skipping uploadrS�verify�header� chunk_size�href)rMzTMalformed response from LFS batch endpoint: `chunk_size` should be an integer. Got 'z'.)r\r�r�� upload_url)r\r�)rIrNrWrbz: Upload successful)rgrp�logger�debug� path_in_repo�_validate_lfs_actionrrDrs� TypeError�_upload_multi_part�_upload_single_partrror� upload_inforrYr"r) r\r�rIrNrMr�� upload_action� verify_actionr�r�r�� verify_url� verify_resps r/� lfs_uploadr��s��8�,�-�-�-��"�"�9�-�-�G���� � �n� �(>�n�n�n�o�o�o���%�Y�/��9�M���'�'�'�$�Y�/�3�3�H�=�=�M�� ��]�+�+�+�� � �x�� ,� ,�F����L�)�)�J�'� �f�(=��Q�Q�Q�J��� ��Z���J�J���I�&� � � ��u�gq�u�u�u��� � ���� �Y�v�*�ak�l�l�l�l�l��i�J�G�G�G�G�� ��]�+�+�+�+�M�&�,A�8�L�L� �!�m�m�(�(� �$�5�'�B�B�B�"�.�5�9�9�;�;�Y�EZ�E_�`�`�)� � � � �K�(�(�(� �L�L�I�*�?�?�?�@�@�@�@�@s �C*�*%D� lfs_actionc���t|�d��t��r=|�d���7t|�d��t��st d���|S)�.validates response from the LFS batch endpointr�r�Nz"lfs_action is improperly formatted)rqrprF�dictrs)r�s r/r�r�se�� �:�>�>�&�)�)�3�/�/�?� �^�^�H� %� %� -��J�N�N�8�<T�<T�VZ�1[�1[� -��=�>�>�>� �r6�lfs_batch_actionsc��t|�d��t��r(t|�d��t��st d���|�di���d��}|�di���d��}|�t |��|�t |��|S)r�rXr"z)lfs_batch_actions is improperly formattedr�rSr�)rqrprFrDrsr�)r�r�r�s r/rgrgs��� �(�,�,�U�3�3�S� 9� 9�F�j�IZ�I^�I^�_e�If�If�hk�>l�>l�F��D�E�E�E�%�)�)�)�R�8�8�<�<�X�F�F�M�%�)�)�)�R�8�8�<�<�X�F�F�M�� ��]�+�+�+�� ��]�+�+�+� �r6�lfs_batch_errorc���t|�d��t��r(t|�d��t��st d���|�d��}t|t ��rPt|�d��t��r(t|�d��t��st d���|S)r�rXr"z'lfs_batch_error is improperly formattedrf�message�code)rqrprFrDrsr�)r�� error_infos r/rkrks��� ��*�*�5�1�1�3� 7� 7�D�J��GZ�GZ�[a�Gb�Gb�dg�<h�<h�D��B�C�C�C� �$�$�W�-�-�J��:�t�$�$�D� �z�~�~�i�0�0�#� 6� 6�D� �z�~�~�f�-�-�s� 3� 3�D� �B�C�C�C� �r6r�c��|�d���5}td||d���}t|��ddd��dS#1swxYwYdS)a� Uploads `fileobj` as a single PUT HTTP request (basic LFS transfer protocol) Args: upload_url (`str`): The URL to PUT the file to. fileobj: The file-like object holding the data to upload. Returns: `requests.Response` Raises: [`HTTPError`](https://requests.readthedocs.io/en/latest/api/#requests.HTTPError) If the upload resulted in an error. T�� with_tqdm�PUT�i�i�i�i��r1�retry_on_status_codesN)�as_filerr)r\r�r7�responses r/r�r�-s��� � � �T� � *� *�&�g���z��Wk�l�l�l���H�%�%�%�&�&�&�&�&�&�&�&�&�&�&�&����&�&�&�&�&�&s�#A�A �A r�r�c��t||j|���}tj}tjrJt |jt ��s0t |jt��stj d��d}|rt|||���nt|||���}t��� |t||jj�����t"���}t%|��dS)z@ Uploads file using HF multipart LFS transfer protocol. )r�r�r�zlhf_transfer is enabled but does not support uploading from bytes or BinaryIO, falling back to regular uploadF)r\�sorted_parts_urlsr�)rcrNN)�_get_sorted_parts_urlsr�r�HF_HUB_ENABLE_HF_TRANSFERrq�path_or_fileobjrFr�warnings�warn�_upload_parts_hf_transfer�_upload_parts_iterativelyrro�_get_completion_payloadrrYrnr)r\r�r�r�r��use_hf_transfer�response_headers�completion_ress r/r�r�Cs�� /�f�)�J_�lv�w�w�w�� �9�O��+�  ��9�4�c�:�:�  ��9�4�d�;�;�  � � � � � � � �� � x�!�I�IZ�gq�r�r�r�r� &��N_�lv� w� w� w��!�]�]�'�'�� $�%5�y�7L�7S�7W�7W�7Y�7Y� Z� Z��(���N� ��'�'�'�'�'r6r�c���d�td�|���D��d����D��}t|��}|t|j|z ��krt d���|S)Nc��g|]\}}|��SrGrG)rZ�_r�s r/r[z*_get_sorted_parts_urls.<locals>.<listcomp>gs,�� � � � �A�z� � � � r6c��g|]>\}}|���r%t|��dk�,t|d��|f��?S)r� )�isdigitr4rD)rZ�part_numr�s r/r[z*_get_sorted_parts_urls.<locals>.<listcomp>js\�� � � �(�H�j��#�#�%�%� �+.�h�-�-�!�*;�*;��X�r�"�"�J�/�*;�*;�*;r6c��|dS)NrrG)�ts r/�<lambda>z(_get_sorted_parts_urls.<locals>.<lambda>os ��!�A�$�r6)�keyz0Invalid server response to upload large LFS file)�sorted�itemsr4rr"rs)r�r�r��sorted_part_upload_urls� num_partss r/r�r�fs��� � �#� � �,2�L�L�N�N� � � � ��  � � � � � ���+�,�,�I��D��)�J�6�7�7�7�7��K�L�L�L� "�"r6r�rXc���g}t|��D]U\}}|�d��}|�|dkrtd|�d|dz�����|�|dz|d����V||d�S)Nr}rQzInvalid etag (`z`) returned for part r)r|r})rXr�)� enumeraterprs�append)r�rXr�� part_numberr�r}s r/r�r�xs��� "�E�(�)9�:�:�  �  �� �V��z�z�&�!�!�� �<�4�2�:�:��[�t�[�[�+�XY�/�[�[�\�\� \� � � �)�A�o�� � � � � � � �� '� '�'r6r�c �d�g}|�d���5}t|��D]n\}}t|||z|���5}td||d���}t |��|�|j��ddd��n #1swxYwY�o ddd��n #1swxYwY|S)NTr�)� seek_from� read_limitr�r�r�)r�r�rrrr�rN) r\r�r�rNr7�part_idx�part_upload_url� fileobj_slice�part_upload_ress r/r�r��sW���G� � � �T� � *� *� 8�g�)2�3D�)E�)E� 8� 8� %�H�o���$�x�/�%���� 8��".��?��Vj�#�#�#��$�O�4�4�4�����6�7�7�7� 8� 8� 8� 8� 8� 8� 8� 8� 8� 8� 8���� 8� 8� 8� 8�� 8� 8� 8� 8� 8� 8� 8� 8� 8� 8� 8� 8���� 8� 8� 8� 8� �Ns5�*B%�=B � B%� B �B%�B �B%�%B)�,B)c �t� ddlm}n#t$rtd���wxYwdt j|��jv}|stjd��|j j }|j }t|��dkr d|dd���}td d |d|tt�����d � ��5} |d|j||d ddd�|r d|jini��}n"#t&$r} t)d��| �d} ~ wwxYw|s|�|��|cddd��S#1swxYwYdS)Nr)�multipart_uploadz�Fast uploading using 'hf_transfer' is enabled (HF_HUB_ENABLE_HF_TRANSFER=1) but 'hf_transfer' package is not available in your environment. Try `pip install hf_transfer`.�callbackz�You are using an outdated version of `hf_transfer`. Consider upgrading to latest version to enable progress bars using `pip install -U hf_transfer`.�(u(…)i�����BTzhuggingface_hub.lfs_upload)�unit� unit_scale�total�initial�desc�disabler`���)� file_path� parts_urlsr�� max_files�parallel_failures� max_retriesz~An error occurred while uploading using `hf_transfer`. Consider disabling HF_HUB_ENABLE_HF_TRANSFER for better error handling.rG)� hf_transferr�� ImportErrorrs�inspect� signature� parametersr�r�r�r"r�r4rrr��getEffectiveLevelr��update� Exception� RuntimeError) r\r�r�r��supports_callbackr�r��progress�output�es r/r�r��s�� �0�0�0�0�0�0�0�� � � � �� Q� � � � ���� #�g�&7�8H�&I�&I�&T�T�� � �� � c� � � � � !� &�E� � !�D� �4�y�y�2�~�~�#�t�C�D�D�z�#�#�� � ���� � ��!9�!9�!;�!;�<�<� )� � � �� � �%�%��#�3�,�%��"%�� ��5F�M�J���0�0�2���F�F��� � � ��*���� ����� ���� !� #� �O�O�E� "� "� "��5��������������������s>� �#�D-�C&�%D-�& D�0D�D�D-�-D1�4D1)NNN)DrAr�r)�rer�� dataclassesr�mathr�os.pathr�pathlibr�typingrrr r r r r r� urllib.parser�huggingface_hubr�utilsrrrrrrrr� utils._lfsr� utils.sharr� utils.tqdmr� _commit_apir� get_loggerr>r��compile� OID_REGEX�LFS_MULTIPART_UPLOAD_COMMANDrnr!rFr�ryr{rr�r�rgrkr�rDr�r�r�r�r�rGr6r/�<module>rs?��5�4����� � � � � � � � �����!�!�!�!�!�!�������������������\�\�\�\�\�\�\�\�\�\�\�\�\�\�\�\�\�\�\�\� � � � � � �%�%�%�%�%�%� � � � � � � � � � � � � � � � � � � � �%�$�$�$�$�$�*�*�*�*�*�*�*�*�(�(�(�(�(�(��0�/�/�/�/�/�/� �� �H� %� %�� �B�J�(� )� )� �5��-�2��� �  �&9�&9�&9�&9�&9�&9�&9� ��&9�R� #�"�(,�K�K��:�&�K� �C�=�K��K�� K� �s�m� K� �s�m� K��d�3��8�n� %�K� �4��:�t�D�z� !�"�K�K�K���K�\�����9���� ��������� �(,�"� CA�CA�#�CA��CA� �C�=�CA��d�3��8�n� %� CA� �s�m� CA�  � CA�CA�CA�CA�L�T����� �t� � � � � �4� � � � �&�#7�&�S�&�T�&�&�&�&�, (�"6� (�� (�RU� (�cf� (�ko� (� (� (� (�F#�4�#�j�#�c�#�VZ�[^�V_�#�#�#�#�$ (�d�4�j� (�s� (�GY� (� (� (� (��#��8<�S� ��OR�� �$�Z�����(1�#�1�8<�S� �1�OR�1� �$�Z�1�1�1�1�1�1r6
Memory