� %�g����ddlZddlZddlmZddlmZmZddlZddl Z ddl m Z dd�Z d�Z d�Zdeeejfd eee jffd �ZGd �d e��ZGd �de��Zd eee jfdeefd�ZdS)�N)�Mapping)�Optional�Union)� safe_openc�"�d}t|j��dkr!|�tj��}d}|������}tj� ||�d���}|�0|�t|j��}|t|j ��d�||<|j dkr|d}tj||jd|j ���}|dd�|dd�<|���|S)Nztorch.bfloat16�bfloat16�.dat)�dtype�shaperzw+)r �moder )�strr �view�torch�int16�cpu�numpy�os�path�join�listr �ndim�np�memmap�flush)�weight� weight_name�offload_folder�indexr �array� tensor_file� file_arrays �h/home/asafur/pinokio/api/open-webui.git/app/env/lib/python3.11/site-packages/accelerate/utils/offload.py�offload_weightr#s��� �E� �6�<���,�,�,����U�[�)�)���� �J�J�L�L� � � � �E��'�,�,�~�+�/C�/C�/C�D�D�K� �� �=��� �$�$�E�',�t�E�K�7H�7H�I�I��k�� �z�Q����d� ����;�e�k��E�K�X�X�X�J��!�!�!�H�J�q�q�q�M������� �L�c�P�t|d��}|dkrd}|d}|dkrd}tj|||d���}t|d��d kr|d }t j|��}|ddkr|�tj��}|S) Nr �)�r rr�r)r r r r)�tuplerr�lenr�tensorrr)� weight_file� weight_infor r rs r"�load_offloaded_weightr..s��� �+�g�&� '� '�E� ��{�{��� �� �E� � ����� �Y�{�%�u�3� G� G� G�F� �;�w� � � �A�%�%����� �\�&� !� !�F��7��z�)�)����U�^�,�,�� �Mr$c���|�t|��dkrdStj�|d��}tj�|��r>t |d���5}t j|��}ddd��n #1swxYwYni}|�|��t |dd���5}t j ||d���ddd��dS#1swxYwYdS)Nr� index.jsonzutf-8)�encoding�w�)�indent) r*rrr�isfile�open�json�load�update�dump)rr�offload_index_file�f� current_indexs r"�save_offload_indexr>Dsd�� �}��E� � �a���������n�l�C�C�� �w�~�~�(�)�)�� �$�w� 7� 7� 7� )�1� �I�a�L�L�M� )� )� )� )� )� )� )� )� )� )� )���� )� )� )� )��� �������� � �#�� 8� 8� 8�.�A� � �-��1�-�-�-�-�.�.�.�.�.�.�.�.�.�.�.�.����.�.�.�.�.�.s$�(B � B �B �>C#�#C'�*C'�save_dir� state_dictc��tj|d���i}|���D]\}}t||||���}�t ||��dS)a Offload a state dict in a given folder. Args: save_dir (`str` or `os.PathLike`): The directory in which to offload the state dict. state_dict (`Dict[str, torch.Tensor]`): The dictionary of tensors to offload. T)�exist_ok)rN)r�makedirs�itemsr#r>)r?r@r�name� parameters r"�offload_state_dictrGUso���K��4�(�(�(�(� �E�%�+�+�-�-�G�G���i��y�$���F�F�F����u�h�'�'�'�'�'r$c�4�eZdZdZdedefd�Zd�Zd�Zd�Z dS) �PrefixedDatasetz� Will access keys in a given dataset by adding a prefix. Args: dataset (`Mapping`): Any map with string keys. prefix (`str`): A prefix to add when trying to access any element in the underlying dataset. �dataset�prefixc�"�||_||_dS�N�rJrK)�selfrJrKs r"�__init__zPrefixedDataset.__init__qs���� ��� � � r$c�.�|j|j�|��SrMrN)rO�keys r"� __getitem__zPrefixedDataset.__getitem__us���|�t�{�1�C�1�1�2�2r$c�D��t�fd��jD����S)Nc�H��g|]}|��j���|��Sr&)� startswithrK��.0rRrOs �r"� <listcomp>z,PrefixedDataset.__iter__.<locals>.<listcomp>ys,���P�P�P�S�C�N�N�4�;�4O�4O�P�S�P�P�Pr$)�iterrJ�rOs`r"�__iter__zPrefixedDataset.__iter__xs(����P�P�P�P�D�L�P�P�P�Q�Q�Qr$c�*�t|j��SrM)r*rJr[s r"�__len__zPrefixedDataset.__len__{s���4�<� � � r$N) �__name__� __module__� __qualname__�__doc__rr rPrSr\r^r&r$r"rIrIhsp����������������3�3�3�R�R�R�!�!�!�!�!r$rIc��eZdZdZ d deeejfdee ee j fde fd�Z defd�Zd �Zd �ZdS) �OffloadedWeightsLoaderaC A collection that loads weights stored in a given state dict or memory-mapped on disk. Args: state_dict (`Dict[str, torch.Tensor]`, *optional*): A dictionary parameter name to tensor. save_folder (`str` or `os.PathLike`, *optional*): The directory in which the weights are stored (by `offload_state_dict` for instance). index (`Dict`, *optional*): A dictionary from weight name to their information (`dtype`/ `shape` or safetensors filename). Will default to the index saved in `save_folder`. Nr@� save_folderrc����|�|�|�td���|�in|�_|�_|�[|�Yttj�|d����5}tj|��}ddd��n #1swxYwY|�in|�_ t�j� �����_ �j � �fd��j D����|�_dS)NzWNeed either a `state_dict`, a `save_folder` or an `index` containing offloaded weights.r0c�&��g|] }|�jv� |��Sr&)�all_keysrWs �r"rYz3OffloadedWeightsLoader.__init__.<locals>.<listcomp>�s%���T�T�T�c�3�d�m�;S�;S�c�;S�;S�;Sr$)� ValueErrorr@rer6rrrr7r8rr�keysrh�extend�device)rOr@rerrlr<s` r"rPzOffloadedWeightsLoader.__init__�s&��� � �+�"5�%�-��v�w�w� w� *� 2�"�"� ���&��� �=�[�4��b�g�l�l�;� �=�=�>�>� %�!�� �!� � �� %� %� %� %� %� %� %� %� %� %� %���� %� %� %� %� �=�R�R�e�� ��T�_�1�1�3�3�4�4�� � � ���T�T�T�T�T�Z�T�T�T�U�U�U��� � � s�A;�;A?�A?rRc�\�||jvr |j|S|j|}|�d����<|j�dn|j}d} t |dd|���5}|�|�d|����}ddd��n #1swxYwYni#t $r\t |ddd���5}|�|�d|����}ddd��n #1swxYwYYnwxYwd|vr.|�tt|d����}|jtj|��kr|�|��}|Stj � |j |�d���}t||��S)N�safetensors_filer�pt)� frameworkrlrr r )r@r�getrlr� get_tensor� TypeError�to�getattrrrrrrer.)rOrRr-rlr+r<r,s r"rSz"OffloadedWeightsLoader.__getitem__�s=�� �$�/� !� !��?�3�'� '��j��o� � �?�?�-� .� .� :�"�k�1�U�U�t�{�F��F� O��{�+=�>�$�W]�^�^�^�O�bc��\�\�+�/�/�-��*M�*M�N�N�F�O�O�O�O�O�O�O�O�O�O�O����O�O�O�O���� O� O� O��{�+=�>�$�W\�]�]�]�O�ab��\�\�+�/�/�-��*M�*M�N�N�F�O�O�O�O�O�O�O�O�O�O�O����O�O�O�O��� O���� �+�%�%����7�5�+�g�2F�#G�#G�H�H���}�� �V� 4� 4�4�4����6�*�*���M��g�l�l�4�#3��\�\�\�B�B� �$�[�+�>�>�>sZ� B'�%*B� B'�B�B'�"B�#B'�'"D � *C?�3 D �?D �D �D �D � D c�*�t|j��SrM)rZrhr[s r"r\zOffloadedWeightsLoader.__iter__�s���D�M�"�"�"r$c�*�t|j��SrM)r*rhr[s r"r^zOffloadedWeightsLoader.__len__�s���4�=�!�!�!r$)NNNN)r_r`rarb�dictr r�Tensorrrr�PathLikerrPrSr\r^r&r$r"rdrds������� � �/3�9=��� ����e�l�*�+���e�C���$4�5�6��� ����(?�s�?�?�?�?�4#�#�#�"�"�"�"�"r$rd�submodule_namesc�|��i}|D]5�|��fd�|���D�����6|S)a Extract the sub state-dict corresponding to a list of given submodules. Args: state_dict (`Dict[str, torch.Tensor]`): The state dict to extract from. submodule_names (`List[str]`): The list of submodule names we want to extract. c�X��i|]&\}}|�ks|��dz���#||��'S)�.)rV)rXrR�param� module_names �r"� <dictcomp>z1extract_submodules_state_dict.<locals>.<dictcomp>�sJ��� � � ��C���+�%�%���� �c�8I�)J�)J�%��U�%�%�%r$)r9rD)r@r{�resultr�s @r"�extract_submodules_state_dictr��so����F�&�  �  � � � � � � � � �",�"2�"2�"4�"4� � � � � � � � �Mr$rM)r7r�collections.abcr�typingrrrrr� safetensorsrr#r.r>r rzrxryrGrIrdrr�r&r$r"�<module>r�st�� � � � � � � � �#�#�#�#�#�#�"�"�"�"�"�"�"�"����� � � � �!�!�!�!�!�!�����*���,.�.�.�"(��s�B�K�'7�!8�(�d�3�PU�P\�K\�F]�(�(�(�(�&!�!�!�!�!�g�!�!�!�.@"�@"�@"�@"�@"�W�@"�@"�@"�F�d�3�� �3D�.E��X\�]`�Xa������r$
Memory