� <��g�� ��UddlmZddlZddlZddlZddlZddlZddlZddlZddl m Z m Z ddl m Z ddlmZddlmZmZmZmZmZmZmZmZmZerddlZddlmZed��Zn ed��Zed ��Zejd ��Z ee!e!ge"fZ#Gd �d ��Z$Gd �de$��Z%Gd�de$��Z&Gd�de$��Z'Gd�de$��Z(Gd�de$��Z)Gd�de$��Z*Gd�de$��Z+Gd�deeef��Z,Gd�de$��Z-de$iZ.de/d <d)d*d(�Z0e$e%e)e&e(e'e*e+e-f D] Z1e0e1���dS)+�)� annotationsN)�Future�ThreadPoolExecutor)�groupby)� itemgetter) � TYPE_CHECKING�Any�Callable�ClassVar�Generic� NamedTuple�Optional� OrderedDict�TypeVar)� ParamSpec�P�T�fsspecc�J�eZdZUdZdZded<dd �Zdd�Zdd�Zdd�Z dd�Z dS)� BaseCacheagPass-though cache: doesn't keep anything, calls every time Acts as base class for other cachers Parameters ---------- blocksize: int How far to read ahead in numbers of bytes fetcher: func Function of the form f(start, end) which gets bytes from remote as specified size: int How big this file is �none� ClassVar[str]�name� blocksize�int�fetcher�Fetcher�size�return�Nonec�h�||_d|_||_||_d|_d|_d|_dS�Nr)r�nblocksrr� hit_count� miss_count�total_requested_bytes)�selfrrrs �^/home/asafur/pinokio/api/open-webui.git/app/env/lib/python3.11/site-packages/fsspec/caching.py�__init__zBaseCache.__init__<s:��"����� ��� ��� �������%&��"�"�"��start� int | None�stop�bytesc�n�|�d}|�|j}||jks||krdS|�||��S)Nrr*)rr�r'r+r-s r(�_fetchzBaseCache._fetchFsF�� �=��E� �<��9�D� �D�I� � ��$����3��|�|�E�4�(�(�(r*c�0�d|_d|_d|_dS)zAReset hit and miss counts for a more ganular report e.g. by file.rN)r$r%r&�r's r(� _reset_statszBaseCache._reset_statsOs��������%&��"�"�"r*�strc �x�|jdkr |jdkrdSd|j�d|j�d|j�d|j�d� S)z2Return a formatted string of the cache statistics.r�z , z: z hits, z misses, z total requested bytes)r$r%rr&r3s r(� _log_statszBaseCache._log_statsUst�� �>�Q� � �4�?�a�#7�#7��2�F�T�Y�F�F�$�.�F�F���F�F�SW�Sm�F�F�F� Fr*c��d|jj�d|j�d|j�d|j�d|j�d|j�d|j�d�S) Nz <z: block size : z block count : z file size : z cache hits : z cache misses: z$ total requested bytes: z > )� __class__�__name__rr#rr$r%r&r3s r(�__repr__zBaseCache.__repr__\s��� � �.� !� � �!�^� � �"�\� � �"�Y�  � � "�^�  � � "�_�  � �%)�$>� � � � r*N�rrrrrrrr �r+r,r-r,rr.�rr )rr5) r;� __module__� __qualname__�__doc__r�__annotations__r)r1r4r8r<�r*r(rr*s�������� � �!�D� � � � �'�'�'�'�)�)�)�)�'�'�'�'� F�F�F�F�  �  �  �  �  �  r*rc�N��eZdZdZdZ dd�fd� Zdd�Zdd�Zdd�Zd d�Z �xZ S)!� MMapCachez�memory-mapped sparse file cache Opens temporary file, which is filled blocks-wise when data is requested. Ensure there is enough disc space in the temporary location. This cache method might only work on posix �mmapNrrrrr�location� str | None�blocks�set[int] | Nonerr c���t���|||��|�t��n||_||_|���|_dS�N)�superr)�setrJrH� _makefile�cache)r'rrrrHrJr:s �r(r)zMMapCache.__init__tsP��� ������G�T�2�2�2�%�~�c�e�e�e�6�� � �� ��^�^�%�%�� � � r*�mmap.mmap | bytearrayc�*�ddl}ddl}|jdkrt��S|j�$t j�|j��s�|j�(|���}t��|_ nt|jd��}|� |jdz ��|� d��|���nt|jd��}|j|���|j��S)Nrzwb+��1zr+b)rG�tempfiler� bytearrayrH�os�path�exists� TemporaryFilerOrJ�open�seek�write�flush�fileno)r'rGrV�fds r(rPzMMapCache._makefile�s���� � � ����� �9��>�>��;�;� � �=� �����t�}�(E�(E� ��}�$��+�+�-�-��!�e�e�� � ��$�-��/�/�� �G�G�D�I��M� "� "� "� �H�H�T�N�N�N� �H�H�J�J�J�J��d�m�U�+�+�B��t�y������d�i�0�0�0r*r+r,�endr.c ���t�d|�d|����|�d}|��j}|�jks||krdS|�jz}|�jz}t ||dz��}�fd�|D��}�xjt �fd�|D����z c_tt|��d�� ��D�]\}}tttd��|����}|d�jz} t|d �jz�jz�j��} �xj | | z z c_ t�d |d�d|d �d | �d| �d � ����| | ���j| | �<�j�|���xjt'|��z c_�� �j||�S)NzMMap cache fetching �-rr*rTc3�.�K�|]}|�jv� |V��dSrM�rJ��.0�ir's �r(� <genexpr>z#MMapCache._fetch.<locals>.<genexpr>�s/�����?�?�a�!�4�;�*>�*>��*>�*>�*>�*>�?�?r*c3�.�K�|]}|�jv� dV��dS)rTNrfrgs �r(rjz#MMapCache._fetch.<locals>.<genexpr>�s/�����I�I�A��T�[�8H�8H�a�8H�8H�8H�8H�I�Ir*c�$�|d|dz S)NrrTrD)�xs r(�<lambda>z"MMapCache._fetch.<locals>.<lambda>�s���1���!���r*)�key�����zMMap get blocks z (�))�logger�debugrr�ranger$�sumr� enumerate�tuple�mapr�minr&rrQrJ�updater%�len) r'r+rb� start_block� end_block� block_range�need�_�_blocks�sstart�sends ` r(r1zMMapCache._fetch�s���� � �9�E�9�9�C�9�9�:�:�:� �=��E� �;��)�C� �D�I� � ��#����3��t�~�-� ��4�>�)� ��K��Q��7�7� �?�?�?�?�;�?�?�?�� ���#�I�I�I�I��I�I�I�I�I�I���"�)�D�/�/�7L�7L�M�M�M� ,� ,�J�A�w��C� �1� � �w�7�7�8�8�G��Q�Z�$�.�0�F��w�r�{�T�^�3�d�n�D�d�i�P�P�D� � &� &�$��-� 7� &� &� �L�L�O�7�1�:�O�O��� �O�O�v�O�O��O�O�O� � � �'+�l�l�6�4�&@�&@�D�J�v�d�{� #� �K� � �w� '� '� '� �O�O�s�7�|�|� +�O�O�O��z�%��)�$�$r*�dict[str, Any]c�>�|j���}|d=|S)NrQ)�__dict__�copy�r'�states r(� __getstate__zMMapCache.__getstate__�s!��� �"�"�$�$�� �'�N�� r*r�c�l�|j�|��|���|_dSrM)r�rzrPrQr�s r(� __setstate__zMMapCache.__setstate__�s-�� � ���U�#�#�#��^�^�%�%�� � � r*)NN) rrrrrrrHrIrJrKrr )rrR�r+r,rbr,rr.�rr��r�r�rr ) r;r@rArBrr)rPr1r�r�� __classcell__�r:s@r(rFrFis���������� �D� $�"&� &� &� &� &� &� &� &�1�1�1�1�,,%�,%�,%�,%�\���� &�&�&�&�&�&�&�&r*rFc�0��eZdZdZdZd�fd � Zdd�Z�xZS)�ReadAheadCachea!Cache which reads only when we get beyond a block of data This is a much simpler version of BytesCache, and does not attempt to fill holes in the cache or keep fragments alive. It is best suited to many small reads in a sequential order (e.g., reading lines from a file). � readaheadrrrrrrr c�x��t���|||��d|_d|_d|_dS)Nr*r)rNr)rQr+rb�r'rrrr:s �r(r)zReadAheadCache.__init__�s8��� ������G�T�2�2�2��� ��� �����r*r+r,rbr.c���|�d}|� ||jkr|j}||jks||krdS||z }||jkr:||jkr/|xjdz c_|j||jz ||jz �S|j|cxkr |jkrDnnA|xjdz c_|j||jz d�}|t |��z}|j}n|xjdz c_d}t|j||jz��}|xj ||z z c_ |� ||��|_||_|jt |j��z|_||jd|�zS�Nrr*rT) rr+rbr$rQr%r{ryrr&r)r'r+rb�l�parts r(r1zReadAheadCache._fetch�s��� �=��E� �;�#�� �/�/��)�C� �D�I� � ��#����3� �%�K�� �D�J� � �3�$�(�?�?� �N�N�a� �N�N��:�e�d�j�0�3���3C�C�D� D� �Z�5� +� +� +� +�4�8� +� +� +� +� +� �O�O�q� �O�O��:�e�d�j�0�2�2�3�D� ��T���N�A��H�E�E� �O�O�q� �O�O��D��$�)�S�4�>�1�2�2�� �"�"�c�E�k�1�"�"��\�\�%��-�-�� ��� ��:��D�J���/����d�j��!��n�$�$r*r=r��r;r@rArBrr)r1r�r�s@r(r�r��sb��������� �D������� %�%�%�%�%�%�%�%r*r�c�0��eZdZdZdZd�fd � Zdd�Z�xZS)�FirstChunkCachez�Caches the first block of a file only This may be useful for file types where the metadata is stored in the header, but is randomly accessed. �firstrrrrrrr c�l��||kr|}t���|||��d|_dSrM)rNr)rQr�s �r(r)zFirstChunkCache.__init__s:��� �t� � ��I� ������G�T�2�2�2�#'�� � � r*r+r,rbr.c��|pd}||jkrt�d��dSt||j��}||jkr�|j��|xjdz c_||jkrD|xj|z c_|�d|��}|d|j�|_||d�S|�d|j��|_|xj|jz c_|j||�}||jkr6|xj||jz z c_||�|j|��z }|xj dz c_ |S|xjdz c_|xj||z z c_|�||��S)Nrz,FirstChunkCache: requested start > file sizer*rT) rrrrsryrrQr%r&rr$)r'r+rb�datar�s r(r1zFirstChunkCache._fetchs���� ��� �4�9� � � �L�L�G� H� H� H��3��#�t�y�!�!�� �4�>� !� !��z�!����1�$������'�'��.�.�#�5�.�.��<�<��3�/�/�D�!%�&6���&6�!7�D�J�����<�'�!�\�\�!�T�^�<�<�� ��*�*�d�n�<�*�*��:�e�C�i�(�D��T�^�#�#��*�*�c�D�N�.B�B�*�*��� � �T�^�S�9�9�9�� �N�N�a� �N�N��K� �O�O�q� �O�O� � &� &�#��+� 5� &� &��<�<��s�+�+� +r*r=r�r�r�s@r(r�r��sb��������� �D�(�(�(�(�(�(�,�,�,�,�,�,�,�,r*r�c�^��eZdZdZdZ dd�fd � Zd �Zdd�Zdd�Zd d�Z d!�fd� Z d"d�Z �xZ S)#� BlockCachea� Cache holding memory as a set of blocks. Requests are only ever made ``blocksize`` at a time, and are stored in an LRU cache. The least recently accessed block is discarded when more than ``maxblocks`` are stored. Parameters ---------- blocksize : int The number of bytes to store in each block. Requests are only ever made for ``blocksize``, so this should balance the overhead of making a request against the granularity of the blocks. fetcher : Callable size : int The total size of the file being cached. maxblocks : int The maximum number of blocks to cache for. The maximum memory use for this cache is then ``blocksize * maxblocks``. � blockcache� rrrrr� maxblocksrr c����t���|||��tj||z ��|_||_t j|��|j��|_ dSrM) rNr)�math�ceilr#r�� functools� lru_cache� _fetch_block�_fetch_block_cached�r'rrrr�r:s �r(r)zBlockCache.__init__Fsd��� ������G�T�2�2�2��y�� �!1�2�2�� �"���#A�9�#6�y�#A�#A�$�BS�#T�#T�� � � r*c�4�|j���S�z� The statistics on the block cache. Returns ------- NamedTuple Returned directly from the LRU Cache used internally. �r�� cache_infor3s r(r�zBlockCache.cache_infoN����'�2�2�4�4�4r*r�c��|j}|d=|S)Nr��r�r�s r(r�zBlockCache.__getstate__Ys��� �� �'� (�� r*r�c��|j�|��tj|d��|j��|_dS)Nr�)r�rzr�r�r�r�r�s r(r�zBlockCache.__setstate__^sH�� � ���U�#�#�#�#J�9�#6�u�[�7I�#J�#J� � �$ �$ �� � � r*r+r,rbr.c���|�d}|�|j}||jks||krdS||jz}||jz}t||dz��D]}|�|���|�||||���S)Nrr*rT��start_block_number�end_block_number)rrrtr�� _read_cache)r'r+rbr�r�� block_numbers r(r1zBlockCache._fetchds��� �=��E� �;��)�C� �D�I� � ��#����3�#�d�n�4���$�.�0��"�"4�6F��6J�K�K� 3� 3�L� � $� $�\� 2� 2� 2� 2���� � �1�-�  � � � r*r�c�<��||jkrtd|�d|j�d����||jz}||jz}|xj||z z c_|xjdz c_t �d|��t���||��}|S)�= Fetch the block of data for `block_number`. �'block_number=�(' is greater than the number of blocks (rqrTzBlockCache fetching block %d) r#� ValueErrorrr&r%rr�inforNr1)r'r�r+rb�block_contentsr:s �r(r�zBlockCache._fetch_block{s���� �$�,� &� &��9��9�9�)-��9�9�9��� � �t�~�-���d�n�$�� �"�"�c�E�k�1�"�"� ���1����� � �2�L�A�A�A�������s�3�3���r*r�r�c ���||jz}||jz}|xjdz c_||kr|�|��}|||�S|�|��|d�g}|�t |jt |dz|������|�|�|��d|���d�|��S�z� Read from our block cache. Parameters ---------- start, end : int The start and end byte positions. start_block_number, end_block_number : int The start and end block numbers. rTNr*�rr$r��extendrxrt�append�join� r'r+rbr�r�� start_pos�end_pos�block�outs r(r�zBlockCache._read_cache�s����D�N�*� ����&�� ���!���� �!1� 1� 1��3�3�4F�G�G�E���7�*�+� +��+�+�,>�?�?� � � �K�L�C� �J�J���,��,�q�0�2B�C�C��� � � � �J�J�t�/�/�0@�A�A�(�7�(�K� L� L� L��8�8�C�=�=� r*�r�� rrrrrrr�rrr r�r�r�)r�rrr.� r+rrbrr�rr�rrr.) r;r@rArBrr)r�r�r�r1r�r�r�r�s@r(r�r�-s����������, �D�MO�U�U�U�U�U�U�U� 5� 5� 5�����  � � � �  � � � �.������$&!�&!�&!�&!�&!�&!�&!�&!r*r�c�H��eZdZUdZdZded< dd�fd� Zdd�Zdd�Z�xZ S)� BytesCacheaKCache which holds data in a in-memory bytes object Implements read-ahead by the block size, for semi-random reads progressing through the file. Parameters ---------- trim: bool As we read more data, whether to discard the start of the buffer when we are more than a blocksize ahead of it. r.rrTrrrrr�trim�boolrr c���t���|||��d|_d|_d|_||_dS)Nr*)rNr)rQr+rbr�)r'rrrr�r:s �r(r)zBytesCache.__init__�sA��� ������G�T�2�2�2��� �!%�� �#����� � � r*r+r,rbc�V�|�d}|�|j}||jks||krdS|j�L||jkrA|j�:||jkr/||jz }|xjdz c_|j|||z|z �S|jrt |j||jz��}n|}||ks ||jkrdS|j� ||jkrY|j� ||jkrG|xj||z z c_|xjdz c_|� ||��|_||_�nu|j�J�|j�J�|xjdz c_||jkr�|j�|j|z |jkr6|xj||z z c_|� ||��|_||_n�|xj|j|z z c_|� ||j��}||_||jz|_n�|j��||jkr�|j|jkrn�||jz |jkr6|xj||z z c_|� ||��|_||_nB|xj||jz z c_|� |j|��}|j|z|_|jt|j��z|_||jz }|j|||z|z �}|j rT|j|jz |jdzz}|dkr4|xj|j|zz c_|j|j|zd�|_|Sr�) rr+rbr$rQrryr&r%rr{r�)r'r+rb�offset�bend�newr��nums r(r1zBytesCache._fetch�sa�� �=��E� �;��)�C� �D�I� � ��#����3� �J� "����#�#���$��d�h����T�Z�'�F� �N�N�a� �N�N��:�f�v��|�e�';�;�<� <� �>� ��t�y�#���"6�7�7�D�D��D� �5�=�=�E�D�I�-�-��3� �J� �%�$�*�"4�"4� �H� ��d�h��� � &� &�$��,� 6� &� &� �O�O�q� �O�O����e�T�2�2�D�J��D�J�J��:�)�)�)��8�'�'�'� �O�O�q� �O�O��t�z�!�!��8�#�t�x�#�~���'F�'F��.�.�$��,�>�.�.�!%���e�T�!:�!:�D�J�!&�D�J�J��.�.�$�*�u�2D�D�.�.��,�,�u�d�j�9�9�C�!&�D�J�!$�t�z�!1�D�J�J���%�$���/�/��8�d�i�'�'���4�8�^�d�n�4�4��.�.�$��,�>�.�.�!%���e�T�!:�!:�D�J�!&�D�J�J��.�.�$���/�A�.�.��,�,�t�x��6�6�C�!%��c�!1�D�J��:��D�J���/������#���j��&�3�,��"6�6�7�� �9� @��8�d�j�(�d�n�q�.@�A�C��Q�w�w�� � �d�n�s�2�2� � �!�Z����(<�(>�(>�?�� �� r*c�*�t|j��SrM)r{rQr3s r(�__len__zBytesCache.__len__s���4�:���r*)T) rrrrrrr�r�rr r�)rr) r;r@rArBrrCr)r1r�r�r�s@r(r�r��s��������� � �"�D�!�!�!�!�IM��������G�G�G�G�R�������r*r�c�F��eZdZUdZdZded< dd�fd� Zdd�Z�xZS)�AllBytesz!Cache entire contents of the file�allrrNrr,r�Fetcher | Nonerr�� bytes | Nonerr c����t���|||��|�@|xjdz c_|xj|jz c_|�d|j��}||_dS)NrTr)rNr)r%r&rrr�)r'rrrr�r:s �r(r)zAllBytes.__init__ sl��� ������G�T�2�2�2� �<� �O�O�q� �O�O� � &� &�$�)� 3� &� &��<�<��4�9�-�-�D��� � � r*r+r-r.c�@�|xjdz c_|j||�S)NrT)r$r�r0s r(r1zAllBytes._fetch.s#�� ���!�����y��t��$�$r*)NNNN) rr,rr�rr,r�r�rr r>� r;r@rArBrrCr)r1r�r�s@r(r�r�s}��������+�+��D�����!%�"&��!� � � � � � � �%�%�%�%�%�%�%�%r*r�c�F��eZdZUdZdZded< dd�fd� Zd�fd� Z�xZS)�KnownPartsOfAFilea� Cache holding known file parts. Parameters ---------- blocksize: int How far to read ahead in numbers of bytes fetcher: func Function of the form f(start, end) which gets bytes from remote as specified size: int How big this file is data: dict A dictionary mapping explicit `(start, stop)` file-offset tuples with known bytes. strict: bool, default True Whether to fetch reads that go beyond a known byte-range boundary. If `False`, any read that ends outside a known part will be zero padded. Note that zero padding will not be used for reads that begin outside a known byte-range. �partsrrNTrrrrrr��&Optional[dict[tuple[int, int], bytes]]�strictr�r�r c �`��t���|||��||_|r�t|�����}|dg}|�|d��g} |dd�D]�\} } |d\} } | | kr-| | f|d<| dxx|�| | f��z cc<�C|�| | f��| �|�| | f������tt|| ����|_ dSi|_ dS)NrrTrp) rNr)r��sorted�keys�popr��dict�zipr�)r'rrrr�r�r�� old_offsets�offsetsrJr+r-�start0�stop0r:s �r(r)zKnownPartsOfAFile.__init__Ls4��� ������G�T�2�2�2��� � � � ������-�-�K�"�1�~�&�G��h�h�{�1�~�.�.�/�F�*�1�2�2�� ;� ;� ��t� '�� � ����E�>�>�#)�4�.�G�B�K��2�J�J�J�$�(�(�E�4�=�"9�"9�9�J�J�J�J��N�N�E�4�=�1�1�1��M�M�$�(�(�E�4�=�"9�"9�:�:�:�:��S��&�1�1�2�2�D�I�I�I��D�I�I�Ir*r+r,r-rr.c���|�d}|�|j}d}|j���D]w\\}}}||cxkr|krbn�||z }||||z|z �}|jr||cxkr|kr2nn/|d||z t |��z zz }|xjdz c_|cS|}n�x|j�td||f�d����tj d||f�d���t� d|�d |����|xj ||z z c_ |xj dz c_ |t���||��zS) Nrr*�rTz&Read is outside the known file parts: z. z%. IO/caching performance may be poor!z!KnownPartsOfAFile cache fetching rd)rr��itemsr�r{r$rr��warnings�warnrrrsr&r%rNr1) r'r+r-r��loc0�loc1r��offr:s �r(r1zKnownPartsOfAFile._fetchjs���� �=��E� �<��9�D���"&�)�/�/�"3�"3� � � �L�T�4�$��u�#�#�#�#�t�#�#�#�#�#��d�l���3��t��e�!3�3�4���{� �d�d�&:�&:�&:�&:�d�&:�&:�&:�&:�&:� �7�d�U�l�S��X�X�&=�>�>�C��N�N�a�'�N�N��J�J�J� !�E��E�!$�* �<� ��W�u�d�m�W�W�W�X�X� X� � � 3�e�T�]� 3� 3� 3� � � � � � �G��G�G��G�G�H�H�H� �"�"�d�U�l�2�"�"� ���1�����U�W�W�^�^�E�4�0�0�0�0r*)NT) rrrrrrr�r�r�r�r�r r>r�r�s@r(r�r�3s�����������,"�D�!�!�!�!�8<�� �������<+1�+1�+1�+1�+1�+1�+1�+1�+1�+1r*r�c�X�eZdZdZGd�de��Zddd �Zdd�Zdd�Zdd�Z dd�Z dS) � UpdatableLRUzg Custom implementation of LRU cache that allows updating keys Used by BackgroudBlockCache c�8�eZdZUded<ded<ded<ded<dS)�UpdatableLRU.CacheInfor�hits�misses�maxsize�currsizeN)r;r@rArCrDr*r(� CacheInfor�s4�������� � � �� � � �� � � �� � � � � r*r��func�Callable[P, T]�max_sizerrr c��tj��|_||_||_d|_d|_tj��|_ dSr") � collectionsr�_cache�_func� _max_size�_hits�_misses� threading�Lock�_lock)r'rr s r(r)zUpdatableLRU.__init__�sA��+6�+B�+D�+D�� ��� �!����� ��� ��^�%�%�� � � r*�args�P.args�kwargs�P.kwargsrc�&�|r$td|��������|j5||jvrC|j�|��|xjdz c_|j|cddd��S ddd��n #1swxYwY|j|i|��}|j5||j|<|xjdz c_t|j��|j kr|j� d���ddd��n #1swxYwY|S)Nz Got unexpected keyword argument rTF��last) � TypeErrorr�rr � move_to_endrr rr{r�popitem)r'rr�results r(�__call__zUpdatableLRU.__call__�s��� � P��N�v�{�{�}�}�N�N�O�O� O� �Z� )� )��t�{�"�"�� �'�'��-�-�-�� � �a�� � ��{�4�(�  )� )� )� )� )� )� )� )�"� )� )� )� )� )� )� )� )� )� )� )���� )� )� )� )� ���T�,�V�,�,�� �Z� 0� 0� &�D�K�� � �L�L�A� �L�L��4�;���$�.�0�0�� �#�#��#�/�/�/�  0� 0� 0� 0� 0� 0� 0� 0� 0� 0� 0���� 0� 0� 0� 0� � s%�AB�B �B �'AD�D � D r r�c�V�|j5||jvcddd��S#1swxYwYdSrM)rr )r'rs r(� is_key_cachedzUpdatableLRU.is_key_cached�sx�� �Z� '� '��4�;�&� '� '� '� '� '� '� '� '� '� '� '� '���� '� '� '� '� '� 's � �"�"rc���|j5||j|<t|j��|jkr|j�d���ddd��dS#1swxYwYdS)NFr)rr r{rr)r'rrs r(�add_keyzUpdatableLRU.add_key�s��� �Z� 0� 0� &�D�K�� ��4�;���$�.�0�0�� �#�#��#�/�/�/� 0� 0� 0� 0� 0� 0� 0� 0� 0� 0� 0� 0���� 0� 0� 0� 0� 0� 0s�AA�A�Arc��|j5|�|jt|j��|j|j���cddd��S#1swxYwYdS)N)rrrr)rrrr{r rrr3s r(r�zUpdatableLRU.cache_info�s��� �Z� � ��>�>����T�[�)�)��Z��|� "��� � � � � � � � � � � � ���� � � � � � s�:A�A�AN)r)rrr rrr )rrrrrr)rr rr�)rrrr rr �rr) r;r@rArBr rr)rr!r#r�rDr*r(r�r��s��������� �����J���� &�&�&�&�&�����&'�'�'�'�0�0�0�0� �����r*r�c�n��eZdZUdZdZded< d!d"�fd� Zd#d�Zd$d�Zd%d�Z d&d�Z d'd(�fd� Z d)d �Z �xZ S)*�BackgroundBlockCachea� Cache holding memory as a set of blocks with pre-loading of the next block in the background. Requests are only ever made ``blocksize`` at a time, and are stored in an LRU cache. The least recently accessed block is discarded when more than ``maxblocks`` are stored. If the next block is not in cache, it is loaded in a separate thread in non-blocking way. Parameters ---------- blocksize : int The number of bytes to store in each block. Requests are only ever made for ``blocksize``, so this should balance the overhead of making a request against the granularity of the blocks. fetcher : Callable size : int The total size of the file being cached. maxblocks : int The maximum number of blocks to cache for. The maximum memory use for this cache is then ``blocksize * maxblocks``. � backgroundrrr�rrrrrr�rr c�>��t���|||��tj||z ��|_||_t |j|��|_td���|_ d|_ d|_ tj��|_dS)NrT�� max_workers)rNr)r�r�r#r�r�r�r�r�_thread_executor�_fetch_future_block_number� _fetch_futurerr�_fetch_future_lockr�s �r(r)zBackgroundBlockCache.__init__�s���� ������G�T�2�2�2��y�� �!1�2�2�� �"���#/��0A�9�#M�#M�� � 2�q� A� A� A���6:��'�37���"+�.�"2�"2����r*rc�4�|j���Sr�r�r3s r(r�zBackgroundBlockCache.cache_info�r�r*r�c�2�|j}|d=|d=|d=|d=|d=|S)Nr�r,r-r.r/r�r�s r(r�z!BackgroundBlockCache.__getstate__s:��� �� �'� (� �$� %� �.� /� �/� "� �&� '�� r*c���|j�|��t|j|d��|_t d���|_d|_d|_tj ��|_ dS)Nr�rTr*) r�rzr�r�r�rr,r-r.rrr/r�s r(r�z!BackgroundBlockCache.__setstate__sk�� � ���U�#�#�#�#/��0A�5��CU�#V�#V�� � 2�q� A� A� A���*.��'�!���"+�.�"2�"2����r*r+r,rbr.c�V�|�d}|�|j}||jks||krdS||jz}||jz}d}d}|j5|j��|j�J�|j���r`t �d��|j� |j� ��|j��d|_d|_n@t||jcxko|knc��}|r|j}|j}d|_d|_ddd��n #1swxYwY|�Gt �d��|j� |� ��|��t||dz��D]}|�|���|dz} |j5|j�R| |j krG|j�| ��s-| |_|j�|j| d��|_ddd��n #1swxYwY|�||||���S)Nrr*z3BlockCache joined background fetch without waiting.z(BlockCache waiting for background fetch.rT�asyncr�)rrr/r.r-�donerrr�r�r#rr�rtr#r!r,�submitr�r�) r'r+rbr�r��fetch_future_block_number� fetch_future� must_joinr��end_block_plus_1s r(r1zBackgroundBlockCache._fetchs,�� �=��E� �;��)�C� �D�I� � ��#����3�#�d�n�4���$�.�0��$(�!�� � � $� 2� 2��!�-��6�B�B�B��%�*�*�,�,�2��K�K� U�V�V�V��,�4�4��*�1�1�3�3�T�5T����7;�D�3�)-�D�&�&�!%�*��:�,�,�,�,�+�,�,�,�,�!�!�I� !�2�59�4S�1�'+�'9� �;?��7�-1��*�7 2� 2� 2� 2� 2� 2� 2� 2� 2� 2� 2���� 2� 2� 2� 2�< � #� �K�K�B� C� C� C� � $� ,� ,��#�#�%�%�'@� � � � "�"4�6F��6J�K�K� 3� 3�L� � $� $�\� 2� 2� 2� 2�,�a�/�� � $� � ��"�*�$�� �4�4��0�>�>�?O�P�P�5�3C��/�%)�%:�%A�%A��%�'7��&�&��"� � � � � � � � � � � ���� � � � ���� � �1�-�  � � � s&�C D�D�D�!AH�H �H �syncr��log_infor5c�>��||jkrtd|�d|j�d����||jz}||jz}t�d||��|xj||z z c_|xjdz c_t���||��}|S)r�r�r�rqz!BlockCache fetching block (%s) %drT) r#r�rrrr�r&r%rNr1)r'r�r<r+rbr�r:s �r(r�z!BackgroundBlockCache._fetch_blockes���� �$�,� &� &��9��9�9�)-��9�9�9��� � �t�~�-���d�n�$��� � �7��<�P�P�P� �"�"�c�E�k�1�"�"� ���1����������s�3�3���r*r�r�c ���||jz}||jz}|xjdz c_||kr|�|��}|||�S|�|��|d�g}|�t |jt |dz|������|�|�|��d|���d�|��Sr�r�r�s r(r�z BackgroundBlockCache._read_cachews����D�N�*� ����&�� ���!���� �!1� 1� 1��,�,�-?�@�@�E���7�*�+� +��+�+�,>�?�?� � � �K�L�C� �J�J���,��,�q�0�2B�C�C��� � � � �J�J�t�/�/�0@�A�A�(�7�(�K� L� L� L��8�8�C�=�=� r*r�r�r%r�r?r�)r;)r�rr<r5rr.r�)r;r@rArBrrCr)r�r�r�r1r�r�r�r�s@r(r'r'�s�����������2'�D�&�&�&�&�MO� 3� 3� 3� 3� 3� 3� 3� 5� 5� 5� 5�����3�3�3�3�J �J �J �J �X�������$(!�(!�(!�(!�(!�(!�(!�(!r*r'z!dict[str | None, type[BaseCache]]�cachesF�cls�type[BaseCache]�clobberr�rr c�~�|j}|s)|tvr td|�dt|�����|t|<dS)z�'Register' cache implementation. Parameters ---------- clobber: bool, optional If set to True (default is False) - allow to overwrite existing entry. Raises ------ ValueError zCache with name z is already known: N)rr?r�)r@rBrs r(�register_cacherD�sL�� �8�D� �W�t�v�~�~��U�D�U�U�v�d�|�U�U�V�V�V��F�4�L�L�Lr*)F)r@rArBr�rr )2� __future__rr r��loggingr�rXrr��concurrent.futuresrr� itertoolsr�operatorr�typingrr r r r r rrrrG�typing_extensionsrrr� getLoggerrrrr.rrrFr�r�r�r�r�r�r�r'r?rCrD�crDr*r(�<module>rNs���"�"�"�"�"�"�"������������� � � � � � � � ���������9�9�9�9�9�9�9�9������������� � � � � � � � � � � � � � � � � � � � � � ����K�K�K�+�+�+�+�+�+�� �#���A�A���� � �A� �G�C�L�L�� �� �8� $� $�� �C��:�u�$� %��< �< �< �< �< �< �< �< �~e&�e&�e&�e&�e&� �e&�e&�e&�P+%�+%�+%�+%�+%�Y�+%�+%�+%�\+,�+,�+,�+,�+,�i�+,�+,�+,�\F!�F!�F!�F!�F!��F!�F!�F!�Rb�b�b�b�b��b�b�b�J%�%�%�%�%�y�%�%�%�0b1�b1�b1�b1�b1� �b1�b1�b1�J9�9�9�9�9�7�1�a�4�=�9�9�9�xK!�K!�K!�K!�K!�9�K!�K!�K!�` �)�-������ �����(� ����� ��� � � �A��N�1����� � r*
Memory