� %�g,5��Z�ddlZddlmZddlZddlZddlmZddlm Z ddl m Z m Z m Z mZmZmZmZmZmZmZmZmZmZmZmZmZmZmZmZe��r ddlmcm Z!ddl"m#Z#ddl$m%Z%e#e&��Z' dd e(d e)e*d e)de)de)de+de+de de,de,fd�Z- dd�Z.dde+de,fd�Z/dde+fd�Z0dS)�N)�Path)� load_model)� GradScaler�)� MODEL_NAME�OPTIMIZER_NAME�RNG_STATE_NAME�SAFE_MODEL_NAME�SAFE_WEIGHTS_NAME� SAMPLER_NAME� SCALER_NAME�SCHEDULER_NAME� WEIGHTS_NAME�get_pretty_name�is_cuda_available�is_hpu_available�is_mlu_available�is_musa_available�is_sdaa_available�is_torch_xla_available�is_xpu_available�load�save)� get_logger)� PartialStateFT� output_dir� model_states� optimizers� schedulers� dataloaders� process_index�step�scaler�save_on_each_node�safe_serializationc �, �t|��}t|��D]z\} } | stnt} | dkr| �dd| �d���} |�| ��} t | | || ���t�d| �����{t|��D]{\} }|� ��} | dkr t�d�n t�d| �d�}|�|��}t | ||d���t�d|�����|t|��D]{\} }|� ��} | dkr t�d�n t�d| �d�}|�|��}t | ||d���t�d |�����|t|��D�] \} }| dkr t�d�n t�d| �d�}|�|��}d d l m}m}t!|j|��r7|���}t!||��rt |||d���t'|d d��rL| dkrd nd| �d�}|�|��}|� ��}t)j||��t�d| �d|������ |�`|� ��} |�t*��}t)j| |��t�d|����i}t,�d|�d�}||d<t/j��|d<t2j���|d<t)j��|d<t9��r!t(j���|d<t?��r"t(j ���|d<n_tC��r"t(j"���|d<n/tG��r!t(j$���|d<tK��r!t(j&���|d<tO��r!t(j(���|d<tS��rtUj��|d<|�|��} t)j|| ��t�d| ����|S) aw Saves the current states of the models, optimizers, scaler, and RNG generators to a given directory. <Tip> If `safe_serialization` is `True`, models will be saved with `safetensors` while the rest are saved using native `pickle`. </Tip> Args: output_dir (`str` or `os.PathLike`): The name of the folder to save all relevant weights and states. model_states (`List[torch.nn.Module]`): A list of model states optimizers (`List[torch.optim.Optimizer]`): A list of optimizer instances schedulers (`List[torch.optim.lr_scheduler._LRScheduler]`): A list of learning rate schedulers dataloaders (`List[torch.utils.data.DataLoader]`): A list of dataloader instances to save their sampler states process_index (`int`): The current process index in the Accelerator state step (`int`): The current step in the internal step tracker scaler (`torch.amp.GradScaler`, *optional*): An optional gradient scaler instance to save; save_on_each_node (`bool`, *optional*): Whether to save on every node, or only the main node. safe_serialization (`bool`, *optional*, defaults to `True`): Whether to save the model using `safetensors` or the traditional PyTorch way (that uses `pickle`). r�.�_)r$r%zModel weights saved in �.binFzOptimizer state saved in zScheduler state saved in r��IterableDatasetShard�SeedableRandomSampler�use_stateful_dataloader�dl_state_dict.bin�dl_state_dict_zSampler state for dataloader z saved in NzGradient scaler state saved in �.pklr"� random_state�numpy_random_seed�torch_manual_seed�torch_xpu_manual_seed�torch_mlu_manual_seed�torch_sdaa_manual_seed�torch_musa_manual_seed�torch_hpu_manual_seed�torch_cuda_manual_seed�xm_seedzRandom states saved in )+r� enumeraterr �replace�joinpathr�logger�info� state_dictrrr � data_loaderr+r,� isinstance�dataset� get_sampler�getattr�torchr r �random�getstate�np� get_state� get_rng_stater�xpu�get_rng_state_allr�mlur�sdaar�musar�hpur�cudar�xm)!rrrrr r!r"r#r$r%�i�state� weights_name�output_model_file�opt�optimizer_name�output_optimizer_file� scheduler�scheduler_name�output_scheduler_file� dataloader� sampler_name�output_sampler_filer+r,�sampler�dataloader_state_dict_name�!output_dataloader_state_dict_filer@�output_scaler_file�states� states_name�output_states_files! �h/home/asafur/pinokio/api/open-webui.git/app/env/lib/python3.11/site-packages/accelerate/checkpointing.py�save_accelerator_stateri8s���X�j�!�!�J��l�+�+�C�C���5�+=�T�|�|�CT� � �q�5�5�'�/�/��X��X�X�X�>�>�L�&�/�/� �=�=�� �U�%�9J�_q�r�r�r�r�� � �A�.?�A�A�B�B�B�B��J�'�'�I�I���3���� � ��45��F�F�N�0�0�0�0�>�@[�@[�TU�@[�@[�@[�� *� 3� 3�N� C� C�� �U�)�=N�ch�i�i�i�i�� � �G�0E�G�G�H�H�H�H�!�*�-�-�I�I� ��9��$�$�&�&��45��F�F�N�0�0�0�0�>�@[�@[�TU�@[�@[�@[�� *� 3� 3�N� C� C�� �U�)�=N�ch�i�i�i�i�� � �G�0E�G�G�H�H�H�H�"�;�/�/�X�X� ��:�01�Q���,�,�,�,�,�|�<U�<U�a�<U�<U�<U� �(�1�1�,�?�?��L�L�L�L�L�L�L�L� �j�(�*>� ?� ?� r� �,�,�.�.�G��'�#8�9�9� r��W�1�EV�kp�q�q�q�q� �:�8�%� @� @� F�@A�Q���)<�)<�Ld�]^�Ld�Ld�Ld� &�0:�0C�0C�D^�0_�0_� -�#�.�.�0�0�J� �J�z�#D� E� E� E�� � �V�A�V�V�AT�V�V�W�W�W�W����!�!�#�#��'�0�0��=�=�� � �5�,�-�-�-�� � �J�6H�J�J�K�K�K� �F�#�9�9�m�9�9�9�K��F�6�N�#�_�.�.�F�>��"$�)�"5�"5�"7�"7�F� ��"'�"5�"7�"7�F� �����H�*/�)�*E�*E�*G�*G��&�'����J�*/�)�*E�*E�*G�*G��&�'�'� � � �J�+0�:�+G�+G�+I�+I��'�(�(� � � �J�+0�:�+G�+G�+I�+I��'�(����H�*/�)�*E�*E�*G�*G��&�'����J�+0�:�+G�+G�+I�+I��'�(����/��,�.�.��y��#�,�,�[�9�9�� �J�v�)�*�*�*� �K�K�>�*<�>�>�?�?�?� ��c �b �t��} |dvrtd���|�d}n|dkrt��j}t |��}t |��D]�\} } | dkrd| ��nd} |�t�| �d ���} | ���rt| | fd t|��i|���c|�t�| �d ���} t| |� ��}| j |fi|����t�d ��t |��D]c\} }| dkr t �d �n t �d| �d �}|�|��}t||� ��}|| � |���dt�d��t |��D][\} }| dkr t"�d �n t"�d| �d �}|�|��}t|��}|� |���\t�d��t |��D�]\} }| dkr t$�d �n t$�d| �d �}|�|��}ddlm}m}t-|j|��rF|���}t-||��r"|�t|����}t5|dd��r[| dkrdnd| �d �}|�|��}|���r$t|��}|� |���� t�d��|�X|�t6��}t|��} |� | ��t�d�� t|�t8�d|�d�����}!d|!vr |!d| d<t;j|!d��t>j� |!d��tCj"|!d��tG��r%tBj$�%|!d��tM��r&tBj'�%|!d��n�tQ��r&tBj)�%|!d��nYtU��r&tBj+�%|!d ��n%tBj,�%|!d!��t[��rt]j"|!d"��t�d#��n*#t^$rt�d$��YnwxYw| S)%a^ Loads states of the models, optimizers, scaler, and RNG generators from a given directory. Args: input_dir (`str` or `os.PathLike`): The name of the folder to load all relevant weights and states. models (`List[torch.nn.Module]`): A list of model instances optimizers (`List[torch.optim.Optimizer]`): A list of optimizer instances schedulers (`List[torch.optim.lr_scheduler._LRScheduler]`): A list of learning rate schedulers process_index (`int`): The current process index in the Accelerator state scaler (`torch.amp.GradScaler`, *optional*): An optional *GradScaler* instance to load map_location (`str`, *optional*): What device to load the optimizer state onto. Should be one of either "cpu" or "on_device". load_model_func_kwargs (`dict`, *optional*): Additional arguments that can be passed to the model's `load_state_dict` method. Returns: `dict`: Contains the `Accelerator` attributes to override while loading the state. )N�cpu� on_devicezaUnsupported optimizer map location passed, please choose one of `None`, `'cpu'`, or `'on_device'`Nrlrmrr(�z .safetensors�devicer))� map_locationz%All model weights loaded successfullyz(All optimizer states loaded successfullyz(All scheduler states loaded successfullyrr*r-Fr.r/z1All dataloader sampler states loaded successfullyz$GradScaler state loaded successfullyr0r"r1r2r3r4r5r6r7r9r:z%All random states loaded successfullyzCould not load random states)0�dict� TypeErrorrrorr;r=r �existsr�strrr�load_state_dictr>r?rrr rAr+r,rBrCrD� set_samplerrEr r rG�setstaterI� set_staterF� set_rng_staterrL�set_rng_state_allrrNrrOrrPrRrrS� Exception)"� input_dir�modelsrrr r!r#rp�load_model_func_kwargs�override_attributesrT�model�ending�input_model_filer@rXrY�input_optimizer_file�optimizer_stater[r\�input_scheduler_file�scheduler_stater^r_�input_sampler_filer+r,rarb� input_dataloader_state_dict_file�input_scaler_file� scaler_stateres" rh�load_accelerator_stater��s��H�&�&���5�5�5�� o� � � ���� � � �� $� $�#�~�~�,� ��Y���I��f�%�%� H� H���5���E�E��Q����r��$�-�-��.V�&�.V�.V�.V�W�W�� � "� "� $� $� H� �u�.� c� c�s�<�7H�7H� c�Lb� c� c� c� c� )�1�1�Z�2M��2M�2M�2M�N�N� ��.�\�J�J�J�J� !�E� !�*� G� G�0F� G� G� G� G� �K�K�7�8�8�8��J�'�'�7�7���3�45��F�F�N�0�0�0�0�>�@[�@[�TU�@[�@[�@[��(�1�1�.�A�A���3�,�O�O�O���1� �%�%�o�6�6�6�6� �K�K�:�;�;�;�"�*�-�-�3�3� ��9�45��F�F�N�0�0�0�0�>�@[�@[�TU�@[�@[�@[��(�1�1�.�A�A���3�4�4���!�!�/�2�2�2�2� �K�K�:�;�;�;�"�;�/�/�7�7� ��:�01�Q���,�,�,�,�,�|�<U�<U�a�<U�<U�<U� �&�/�/� �=�=��L�L�L�L�L�L�L�L� �j�(�*>� ?� ?� K� �,�,�.�.�G��'�#8�9�9� K�$�0�0��6H�1I�1I�J�J�� �:�8�%� @� @� 7�@A�Q���)<�)<�Ld�]^�Ld�Ld�Ld� &�/8�/A�/A�B\�/]�/]� ,�/�6�6�8�8� 7�!�"B�C�C� ��*�*�:�6�6�6�� �K�K�C�D�D�D���%�.�.�{�;�;���-�.�.� ����|�,�,�,�� � �:�;�;�;�4��i�(�(�N�)P�)P�]�)P�)P�)P�Q�Q�R�R�� �V� � �*0��.� �� '����~�.�/�/�/� � ���F�#6�7�8�8�8� ��F�#6�7�8�8�8� � � � I� �I� '� '��/F�(G� H� H� H� � � � K� �I� '� '��/F�(G� H� H� H� H� � � � K� �J� (� (��0H�)I� J� J� J� J� � � � K� �J� (� (��0H�)I� J� J� J� J� �J� (� (��0H�)I� J� J� J� !� #� #� 0� � �V�I�.� /� /� /�� � �;�<�<�<�<�� �4�4�4�� � �2�3�3�3�3�3�4���� �s�9G V�$V,�+V,�indexc���t|��d|�d�z }t�dt|���d|����t |���||���dS)zL Saves the state of `obj` to `{path}/custom_checkpoint_{index}.pkl` �custom_checkpoint_r0zSaving the state of z to )r$N)rr>r?rrr@)�obj�pathr�r$� save_locations rh�save_custom_stater�.sq�� ��J�J�!A�e�!A�!A�!A�A�M� �K�K�P��s�';�';�P�P��P�P�Q�Q�Q����� � �=�<M�N�N�N�N�N�Nrjc��|�d|�d�}t�dt|���d|����|�t |dd�����dS) z� Loads the state of `obj` at `{path}/custom_checkpoint_{index}.pkl`. Will always set `weights_only=False` when loading the state. z/custom_checkpoint_r0zLoading the state of z from rlF)rp� weights_onlyN)r>r?rrur)r�r�r�� load_locations rh�load_custom_stater�8sn�� �;�;��;�;�;�M� �K�K�S���(<�(<�S�S�M�S�S�T�T�T�����]��U�S�S�S�T�T�T�T�Trj)NFT)NN)rF)r)1rG�pathlibr�numpyrIrF�safetensors.torchr�torch.cuda.ampr�utilsrrr r r r r rrrrrrrrrrrr�torch_xla.core.xla_model�core� xla_modelrS�loggingrrUr�__name__r>rt�listrq�int�boolrir�r�r��rjrh�<module>r�s���� � � ����������� � � � �(�(�(�(�(�(�%�%�%�%�%�%�������������������������������������������.����*�)�)�)�)�)�)�)�)�)������������� ��H� � ���#�#�s�s��s��t�*�s��s�� s� � s� � s� �s� �s��s��s�s�s�s�z ��}�}�}�}�@O�O��O�D�O�O�O�O�U�U��U�U�U�U�U�Urj
Memory