� %�g�}��l�ddlZddlmZddlmZmZddlZddlmZddl m Z ddl m Z m Z mZmZmZddlmZmZmZddlmZdd lmZdd lmZgd �ZGd �d ��ZGd�de��Zd)dejdedefd�Z d)dejfd�Z!Gd�de��Z" d*dejjdee#e$ej%fdeee$e&e$fdee&e$dee'e#e'ej%ej(fff d�Z) d+dejjdeej%dedeed ed!e$deee$e&e$fdee&e$dee'e#e'ej%ej(fffd"�Z*dejfd#�Z+ d+dejdeeej%e'e$ej%ffdeee'e$effded ed!e$deee$e&e$fdee&e$dee'e#e'ej%ej(fffd$�Z,Gd%�d&e��Z-Gd'�d(��Z.dS),�N)�Mapping)�Optional�Union�)� PartialState)�PrefixedDataset� find_device�named_module_tensors�send_to_device�set_module_tensor_to_device)�is_mlu_available�is_musa_available�is_npu_available)�clear_device_cache)�get_non_persistent_buffers)�recursive_getattr)�to�cuda�npu�xpu�mlu�sdaa�musac�.�eZdZdZdZd�Zd�Zd�Zd�ZdS)� ModelHooka A hook that contains callbacks to be executed just before and after the forward method of a model. The difference with PyTorch existing hooks is that they get passed along the kwargs. Class attribute: - **no_grad** (`bool`, *optional*, defaults to `False`) -- Whether or not to execute the actual forward pass under the `torch.no_grad()` context manager. Fc��|S)z� To be executed when the hook is attached to the module. Args: module (`torch.nn.Module`): The module attached to this hook. ���self�modules �`/home/asafur/pinokio/api/open-webui.git/app/env/lib/python3.11/site-packages/accelerate/hooks.py� init_hookzModelHook.init_hook7� ��� �c� �||fS)a� To be executed just before the forward method of the model. Args: module (`torch.nn.Module`): The module whose forward pass will be executed just after this event. args (`Tuple[Any]`): The positional arguments passed to the module. kwargs (`Dict[Str, Any]`): The keyword arguments passed to the module. Returns: `Tuple[Tuple[Any], Dict[Str, Any]]`: A tuple with the treated `args` and `kwargs`. r�rr �args�kwargss r!� pre_forwardzModelHook.pre_forward@s���V�|�r$c��|S)a: To be executed just after the forward method of the model. Args: module (`torch.nn.Module`): The module whose forward pass been executed just before this event. output (`Any`): The output of the module. Returns: `Any`: The processed `output`. r)rr �outputs r!� post_forwardzModelHook.post_forwardNs ��� r$c��|S)z� To be executed when the hook is detached from a module. Args: module (`torch.nn.Module`): The module detached from this hook. rrs r!� detach_hookzModelHook.detach_hook[r#r$N) �__name__� __module__� __qualname__�__doc__�no_gradr"r)r,r.rr$r!rr+sa���������G���� � � � � � �����r$rc�0�eZdZdZd�Zd�Zd�Zd�Zd�ZdS)�SequentialHookzX A hook that can contain several hooks and iterates through them at each event. c��||_dS�N)�hooks)rr8s r!�__init__zSequentialHook.__init__js ���� � � r$c�D�|jD]}|�|��}�|Sr7)r8r"�rr �hooks r!r"zSequentialHook.init_hookms+���J� ,� ,�D��^�^�F�+�+�F�F�� r$c�F�|jD]}|j|g|�Ri|��\}}�||fSr7)r8r))rr r'r(r<s r!r)zSequentialHook.pre_forwardrsG���J� E� E�D�+�4�+�F�D�T�D�D�D�V�D�D�L�D�&�&��V�|�r$c�F�|jD]}|�||��}�|Sr7)r8r,)rr r+r<s r!r,zSequentialHook.post_forwardws/���J� 7� 7�D��&�&�v�v�6�6�F�F�� r$c�D�|jD]}|�|��}�|Sr7)r8r.r;s r!r.zSequentialHook.detach_hook|s-���J� .� .�D��%�%�f�-�-�F�F�� r$N) r/r0r1r2r9r"r)r,r.rr$r!r5r5esi�������������� ��� ��� ����r$r5Fr r<�appendc��|r7t|dd���&|j}t|��t||��}t |d��rt |d��r|j}n|j}||_|�|��}||_d�}dtt|����vr3tj tj ||��|��|j _n-tj tj ||��|��|_|S)a� Adds a hook to a given module. This will rewrite the `forward` method of the module to include the hook, to remove this behavior and restore the original `forward` method, use `remove_hook_from_module`. <Tip warning={true}> If the module already contains a hook, this will replace it with the new hook passed by default. To chain two hooks together, pass `append=True`, so it chains the current and new hook into an instance of the `SequentialHook` class. </Tip> Args: module (`torch.nn.Module`): The module to attach a hook to. hook (`ModelHook`): The hook to attach. append (`bool`, *optional*, defaults to `False`): Whether the hook should be chained with an existing one (if module already contains a hook) or not. Returns: `torch.nn.Module`: The same module, with the hook attached (the module is modified in place, so the result can be discarded). �_hf_hookN� _old_forwardc��|jj|g|�Ri|��\}}|jjr9tj��5|j|i|��}ddd��n #1swxYwYn |j|i|��}|j�||��Sr7)rBr)r3�torchrCr,)r r'r(r+s r!� new_forwardz'add_hook_to_module.<locals>.new_forward�s���2�v��2�6�K�D�K�K�K�F�K�K� ��f� �?� "� :����� >� >�,��,�d�=�f�=�=�� >� >� >� >� >� >� >� >� >� >� >���� >� >� >� >��)�V�(�$�9�&�9�9�F���+�+�F�F�;�;�;s�A�A�A�GraphModuleImpl)�getattrrB�remove_hook_from_moduler5�hasattrrC�forwardr"�str�type� functools�update_wrapper�partial� __class__)r r<r@�old_hook� old_forwardrFs r!�add_hook_to_modulerT�s��2�.�7�6�:�t�4�4�@��?����'�'�'��h��-�-���v�z�"�"�*�w�v�~�'F�'F�*��)� � ��n� �)��� �^�^�F� #� #�F��F�O�<�<�<��C��V� � �-�-�-�-�#,�#;�I�<M�k�[a�<b�<b�do�#p�#p��� � �"�1�)�2C�K�QW�2X�2X�Ze�f�f��� �Mr$c���t|d��r*|j�|��t|d��t|d��rLdt t |����vr|j|j_n |j|_t|d��tD]}|j � |d���|r'|� ��D]}t||���|S)a� Removes any hook attached to a module via `add_hook_to_module`. Args: module (`torch.nn.Module`): The module to attach a hook to. recurse (`bool`, **optional**): Whether to remove the hooks recursively Returns: `torch.nn.Module`: The same module, with the hook detached (the module is modified in place, so the result can be discarded). rBrCrGN)rJrBr.�delattrrLrMrCrQrK�_accelerate_added_attributes�__dict__�pop�childrenrI)r �recurse�attr�childs r!rIrI�s����v�z�"�"�$���#�#�F�+�+�+��� �#�#�#��v�~�&�&�(� ��D��L�L� 1� 1� 1� 1�'-�':�F� � $� $�#�0�F�N����'�'�'�-�(�(������D�$�'�'�'�'��4��_�_�&�&� 4� 4�E� #�E�7� 3� 3� 3� 3� �Mr$c��eZdZdZ ddeeeeej fde de dee de d e d eeee efd ee ee ej ejfffd �Zd �Zd�Zd�Zd�Zd�ZdS)�AlignDevicesHooka� A generic `ModelHook` that ensures inputs and model weights are on the same device for the forward pass of the associated module, potentially offloading the weights after the forward pass. Args: execution_device (`torch.device`, *optional*): The device on which inputs and model weights should be placed before the forward pass. offload (`bool`, *optional*, defaults to `False`): Whether or not the weights should be offloaded after the forward pass. io_same_device (`bool`, *optional*, defaults to `False`): Whether or not the output should be placed on the same device as the input was. weights_map (`Mapping[str, torch.Tensor]`, *optional*): When the model weights are offloaded, a (potentially lazy) map from param names to the tensor values. offload_buffers (`bool`, *optional*, defaults to `False`): Whether or not to include the associated module's buffers when offloading. place_submodules (`bool`, *optional*, defaults to `False`): Whether to place the submodules on `execution_device` during the `init_hook` event. NF�execution_device�offload�io_same_device� weights_map�offload_buffers�place_submodules� skip_keys�tied_params_mapc ���||_||_||_||_||_||_||_d|_i|_i|_ t��|_ ||_ dSr7) r`rarbrcrdrerf� input_device�param_original_devices�buffer_original_devices�set�tied_params_namesrg) rr`rarbrcrdrerfrgs r!r9zAlignDevicesHook.__init__�sp��!1����� �,���&���.��� 0���"���!���&(��#�')��$�!$����� /����r$c��d|j�d|j�d|j�d|j�d|j�dt |j���d� S)Nz"AlignDevicesHook(execution_device=z , offload=z, io_same_device=z, offload_buffers=z, place_submodules=z , skip_keys=�))r`rarbrdre�reprrf�rs r!�__repr__zAlignDevicesHook.__repr__s��� [��1F� [� [�RV�R^� [� [�"�1� [� [�EI�EY� [� [� $� 5� [� [�CG���CW�CW� [� [� [� r$c���|jdks|jtjd��krd|_|jsB|j�;t ||j���D]"\}}t|||j|j����#�nn|j�rfd�t ||j���D��|_|j �+d�t ||j |j���D��|_ t ||j |jd���D]`\}}|j�Ct||��� ��|jvr|j �|��t||d���a|j sF|j�?|�|j���D]"\}}t|||j|j����#nD|j r=|j�6t!||j���D]}t|||j|j���� |S) N�meta)r[)rgc�$�i|] \}}||j��Sr)�device��.0�name�params r!� <dictcomp>z.AlignDevicesHook.init_hook.<locals>.<dictcomp>#s-��%�%�%�'2�t�U��e�l�%�%�%r$c�@�i|]\}}||�d����S)�cpu�rrws r!r{z.AlignDevicesHook.init_hook.<locals>.<dictcomp>'s8��$�$�$�#��e��%�(�(�5�/�/�$�$�$r$)�include_buffersr[T�rr[�remove_non_persistent)r`rErvrgrar rer �original_devicesrcrdr�data_ptrrm�add� named_buffersr)rr ry�_s r!r"zAlignDevicesHook.init_hookss�� � �F� *� *�d�.C�u�|�TZ�G[�G[�.[�.[�#'�D� ��|�& �� 5� A�/���@U�V�V�V� w� w���a�+�F�D�$�:O�ae�au�v�v�v�v�v� w� �\�# �%�%�6J�6�[_�[p�6q�6q�6q�%�%�%�D� !���'�$�$�';���0D�d�Nc�(�(�(�$�$�$�� � 0���(<�d�F[�sw���� B� B���a��(�4�)�&�$�7�7�@�@�B�B�d�FZ�Z�Z��*�.�.�t�4�4�4�+�F�D�&�A�A�A�A��'� �D�,A�,M�%�3�3�D�<Q�3�R�R���G�D�!�/���d�&;�T�Ma��������%� �$�*?�*K�6�v�t�G\�]�]�]���D�/���d�&;�T�Ma������� r$c ���|jrt||g��|_|j�r�t ��|_t ||j|jd���D�]Z\}}d}|j |}d|vre|� dd��|j � ��vr6|j tjkr!|j |� dd��}||jvr7|���|jvri|j|���<|�{|j�t|���|jvrY|j|j|���vr3|j�|���|jf��t)|||j|||j�����\t+||j��t+||j|j���fS)NTr��weight�SCB)�value�fp16_statisticsrg�rf)rbr rirarl�tied_pointers_to_remover rdrerc�replace�keys�dtyperE�int8rmr�rgr`r�r r rf)rr r'r(ryr�r�r�s r!r)zAlignDevicesHook.pre_forwardIs��� � � <� +�T�6�N� ;� ;�D� � �<�$ �+.�5�5�D� (�/�� $� 4��-�&*� ���! �! ���a� #'���(��.���t�#�#�� � �X�u�(E�(E��IY�I^�I^�I`�I`�(`�(`��{�e�j�0�0�*.�*:�4�<�<��RW�;X�;X�*Y�� �4�1�1�1�e�n�n�6F�6F�d�Nb�6b�6b�=?�D�(����)9�)9�:��%��,�8����(�(�D�,@�@�@��-�T�5I�%�.�.�JZ�JZ�5[�[�[��0�4�4�e�n�n�6F�6F��H]�5^�_�_�_�+����)��$3�$(�$8� ������d�D�$9�:�:�N� �D�)�T�^�= �= �= � � r$c�J�|jr�t||j|jd���D]F\}}t ||d��t |��jdkrd|j_d|j_ �G|j D]r\}}t|t��r;t��rd|��}n't��rd|��}nt��rd|��}||j|vr|j||=�st#��|_ |jr#|j�t)||j|j���}|S) NTr�rt� Linear8bitLtznpu:zmlu:zmusa:r�)rar rdrer rMr/�stater��CxBr�� isinstance�intrr rrgrlrbrir rf)rr r+ryr�� value_pointerrvs r!r,zAlignDevicesHook.post_forwardvs]�� �<� 1�/�� $� 4��-�&*� ��� ,� ,���a� ,�F�D�&�A�A�A���<�<�(�N�:�:�'+�F�L�$�'+�F�L�$��*.�)E� D� D�%� �v��f�c�*�*�2�'�)�)�2�!0������)�+�+�2�!0������*�,�,�2�!1��!1�!1���T�1�-�@�@�@��,�]�;�F�C��+.�5�5�D� (� � � Y�4�#4�#@�#�F�D�,=���X�X�X�F�� r$c ���|jrc|j���D]I\}}|tjd��kr,t ||||j�|d������J|S)Nrt)r�)rar��itemsrErvr rc�get)rr ryrvs r!r.zAlignDevicesHook.detach_hook�sx�� �<� n� $� 5� ;� ;� =� =� n� n� ��f��U�\�&�1�1�1�1�/���f�D�L\�L`�L`�ae�gk�Ll�Ll�m�m�m�m��� r$)NFFNFFNN)r/r0r1r2rrr�rLrErv�boolr�list�dict�Tensorr9rrr"r)r,r.rr$r!r_r_�s5��������*EI��$�)-� %�!&�59�QU�/�/�"�5��c�5�<�)?�#@�A�/��/�� /� �g�&� /� � /��/��E�#�t�C�y�.�1�2�/�"�$�s�D���u�|�1K�,L�'L�"M�N�/�/�/�/�: � � �-�-�-�^+ �+ �+ �Z���>����r$r_r`rf�preload_module_classesrgc�*�t|d��sEt|�����dkr t|t |||�����|�|jj|vrdS|���D]}t|||||����dS)a� Recursively attaches `AlignDevicesHook` to all submodules of a given model to make sure they have the right execution device Args: module (`torch.nn.Module`): The module where we want to attach the hooks. execution_device (`int`, `str` or `torch.device`): The device on which inputs and model weights should be placed before the forward pass. skip_keys (`str` or `List[str]`, *optional*): A list of keys to ignore when moving inputs or outputs between devices. preload_module_classes (`List[str]`, *optional*): A list of classes whose instances should load all their weights (even in the submodules) at the beginning of the forward. This should only be used for classes that have submodules which are registered but not called directly during the forward, for instance if a `dense` linear layer is registered, but at forward, `dense.weight` and `dense.bias` are used in some operations instead of calling `dense` directly. tied_params_map (Optional[Dict[int, Dict[torch.device, torch.Tensor]]], *optional*, defaults to `None`): A map of data pointers to dictionaries of devices to already dispatched tied weights. For a given execution device, this parameter is useful to reuse the first available pointer of a shared weight for all others, instead of duplicating memory. rBr�rfrgN)rfr�rg) rJ�len� state_dictrTr_rQr/rZ�attach_execution_device_hook)r r`rfr�rgr]s r!r�r��s���8 �6�:� &� &� �3�v�/@�/@�/B�/B�+C�+C�a�+G�+G�� � �-��Tc� d� d� d� � � � �)�f�.>�.G�Ka�.a�.a������"�"� � ��$� � ��#9�+�  � � � � � � r$�rarcrd� module_namec ���t|��} |o|duo |jj|v} tt | ����dks| rW|�+t|��dkr|�d�nd} t || ��} nd} t ||| || ||���} t|| d���| rdS|���D]9\}}t|��dkr|�d|��n|}t|||||||||�� � �:dS) a� Recursively attaches `AlignDevicesHook` to all submodules of a given model that have direct parameters and/or buffers. Args: module (`torch.nn.Module`): The module where we want to attach the hooks. execution_device (`torch.device`, *optional*): The device on which inputs and model weights should be placed before the forward pass. offload (`bool`, *optional*, defaults to `False`): Whether or not the weights should be offloaded after the forward pass. weights_map (`Mapping[str, torch.Tensor]`, *optional*): When the model weights are offloaded, a (potentially lazy) map from param names to the tensor values. offload_buffers (`bool`, *optional*, defaults to `False`): Whether or not to include the associated module's buffers when offloading. module_name (`str`, *optional*, defaults to `""`): The name of the module. skip_keys (`str` or `List[str]`, *optional*): A list of keys to ignore when moving inputs or outputs between devices. preload_module_classes (`List[str]`, *optional*): A list of classes whose instances should load all their weights (even in the submodules) at the beginning of the forward. This should only be used for classes that have submodules which are registered but not called directly during the forward, for instance if a `dense` linear layer is registered, but at forward, `dense.weight` and `dense.bias` are used in some operations instead of calling `dense` directly. tied_params_map (Optional[Dict[int, Dict[torch.device, torch.Tensor]]], *optional*, defaults to `None`): A map of data pointers to dictionaries of devices to already dispatched tied weights. For a given execution device, this parameter is useful to reuse the first available pointer of a shared weight for all others, instead of duplicating memory. Nr�.r�)r`rarcrdrerfrgT)r@�r`rarcrdr�r�rfrg) r rQr/r�r�rr_rT�named_children�attach_align_device_hook)r r`rarcrdr�rfr�rg�directs� full_offload�prefix�prefixed_weights_mapr<� child_namer]s r!r�r��sw��R#�6�*�*�G��n�*�$�6�n�6�;K�;T�Xn�;n�� �4��=�=���A����� � "�*-�k�*:�*:�Q�*>�*>� �&�&�&�&�B�F�#2�;��#G�#G� � �#'� ��-��,�+�)��+� � � �� �6�4��5�5�5�5�����$�2�2�4�4�  �  �� �E�69�+�6F�6F��6J�6J� �2�2�j�2�2�2�PZ� � � �-��#�+�"�#9��+� � � � � �  �  r$c�p�t|��|���D]}t|���dS)z� Recursively removes all hooks attached on the submodules of a given model. Args: module (`torch.nn.Module`): The module on which to remove all hooks. N)rIrZ�remove_hook_from_submodules)r r]s r!r�r� sF���F�#�#�#����"�"�+�+��#�E�*�*�*�*�+�+r$c �t���t�t��sUt�t��s@�s%t�d|d|���} t || ��nt |�d|||||���dSt�t��s �fd�����D���t�t��s �fd�����D���|�vrU|�vrQ�|sIt�|||dkd||���} t || ��t|�|||� ��n�|�vrz|�vrvt |�|d||||||� � � t|d ��s-t�||dk||� ��} t || ��t|�||||� ��n<|dkr6t�� d��d||� ��} t || ��|� ��D]9\} } t|��dkr|�d| ��n| } t| ��||| |||�� � �:dS)a� Attaches `AlignDevicesHook` to all blocks of a given model as needed. Args: module (`torch.nn.Module`): The module where we want to attach the hooks. execution_device (`torch.device` or `Dict[str, torch.device]`, *optional*): The device on which inputs and model weights should be placed before the forward pass. It can be one device for the whole module, or a dictionary mapping module name to device. offload (`bool`, *optional*, defaults to `False`): Whether or not the weights should be offloaded after the forward pass. It can be one boolean for the whole module, or a dictionary mapping module name to boolean. weights_map (`Mapping[str, torch.Tensor]`, *optional*): When the model weights are offloaded, a (potentially lazy) map from param names to the tensor values. offload_buffers (`bool`, *optional*, defaults to `False`): Whether or not to include the associated module's buffers when offloading. module_name (`str`, *optional*, defaults to `""`): The name of the module. skip_keys (`str` or `List[str]`, *optional*): A list of keys to ignore when moving inputs or outputs between devices. preload_module_classes (`List[str]`, *optional*): A list of classes whose instances should load all their weights (even in the submodules) at the beginning of the forward. This should only be used for classes that have submodules which are registered but not called directly during the forward, for instance if a `dense` linear layer is registered, but at forward, `dense.weight` and `dense.bias` are used in some operations instead of calling `dense` directly. tied_params_map (Optional[Dict[int, Dict[torch.device, torch.Tensor]]], *optional*, defaults to `None`): A map of data pointers to dictionaries of devices to already dispatched tied weights. For a given execution device, this parameter is useful to reuse the first available pointer of a shared weight for all others, instead of duplicating memory. T)r`rbrfrerg)r`rarcrdr�rfrgNc���i|]}|���Srr)rx�keyr`s �r!r{z6attach_align_device_hook_on_blocks.<locals>.<dictcomp>ns���L�L�L�c�C�!1�L�L�Lr$c���i|]}|���Srr)rxr�ras �r!r{z6attach_align_device_hook_on_blocks.<locals>.<dictcomp>ps���C�C�C�C�3��C�C�Cr$r�)r`rdrbrerfrgr�)r`rarcrdr�rfr�rgrB)r`rbrfrg)r�rfrgrr�r�) r�rr�r_rTr�r�r�rJr�r�r��"attach_align_device_hook_on_blocks) r r`rarcrdr�rfr�rgr<r�r]s `` r!r�r�,sG����T �&�� 0� 0���G�T�9R�9R��� �#�!1�#�#�!%� /� ���D� �v�t� ,� ,� ,� ,� $��!1��'� /�'�#� /� � � � � �� �&�� 0� 0�M�L�L�L�L�W�\�\�^�^�L�L�L�� �g�w� '� '�D�C�C�C�C�+;�+@�+@�+B�+B�C�C�C���&�&�&�;�'�+A�+A�'�R]�J^�+A��-�k�:�+�'�2�-�!��+�  � � �� �6�4�(�(�(�$� �$�[�1�Y�Xg� � � � � � �(� (� (�[�G�-C�-C� � �-�k�:��#�+�#��#9�+� � � � ��v�z�*�*� -�#�!1�+�!>� +�r� 1�#� /� ���D� �v�t� ,� ,� ,�$� � �[� )�#9��+�  � � � � � �� � ��-�1�1�"�5�5���+�  � � �� �6�4�(�(�(�#�2�2�4�4�  �  �� �E�69�+�6F�6F��6J�6J� �2�2�j�2�2�2�PZ� �*� �-��#�+�"�#9��+� � � � � �  �  r$c�h�eZdZdZ d deeeeej fdedfd�Z d�Z d�Z dS) � CpuOffloada Offloads a model on the CPU until its forward pass is called. The model will not be offloaded back to the CPU after the forward, the user needs to call the `init_hook` method again for this. Args: execution_device(`str`, `int` or `torch.device`, *optional*): The device on which the model should be executed. Will default to the MPS device if it's available, then GPU 0 if there is a GPU, and finally to the CPU. prev_module_hook (`UserCpuOffloadHook`, *optional*): The hook sent back by [`cpu_offload_with_hook`] for a previous model in the pipeline you are running. If passed, its offload method will be called just before the forward of the model to which this hook is attached. Nr`�prev_module_hook�UserCpuOffloadHookc�L�||_|�|nt��j|_dSr7)r�r�default_devicer`)rr`r�s r!r9zCpuOffload.__init__�s/�� !1���4D�4P� 0� 0�Vb�Vd�Vd�Vs����r$c�,�|�d��S)Nr}r~rs r!r"zCpuOffload.init_hook�s���y�y����r$c���|j�'|j���t��|�|j��t ||j��t ||j��fSr7)r�rarrr`r r&s r!r)zCpuOffload.pre_forward�sg�� � � ,� � !� )� )� +� +� +� � � � �� � �$�'�(�(�(��d�D�$9�:�:�N�6�SW�Sh�<i�<i�i�ir$)NN) r/r0r1r2rrrLr�rErvr9r"r)rr$r!r�r��s������� � � EI�;?�t�t�"�5��c�5�<�)?�#@�A�t�#�#7�8�t�t�t�t� � � �j�j�j�j�jr$r�c�$�eZdZdZd�Zd�Zd�ZdS)r�z� A simple hook grouping a model and a `ModelHook`, which provides easy APIs for to call the init method of the hook or remove it entirely. c�"�||_||_dSr7)�modelr<)rr�r<s r!r9zUserCpuOffloadHook.__init__�s���� ��� � � r$c�D�|j�|j��dSr7)r<r"r�rqs r!razUserCpuOffloadHook.offload�s �� � ���D�J�'�'�'�'�'r$c�.�t|j��dSr7)rIr�rqs r!�removezUserCpuOffloadHook.remove�s���� �+�+�+�+�+r$N)r/r0r1r2r9rar�rr$r!r�r��sK�������� ���(�(�(�,�,�,�,�,r$r�)F)NNN)NFNFr�NNN)/rN�collections.abcr�typingrrrE�torch.nn�nnr�r�utilsrr r r r � utils.importsr rr� utils.memoryr�utils.modelingr� utils.otherrrWrr5�Moduler�rTrIr_r�rLrvr�r�r�r�r�r�r�r�r�rr$r!�<module>r�s������#�#�#�#�#�#�"�"�"�"�"�"�"�"� � � � ������������������������������������� -�,�,�,�,�,�6�6�6�6�6�6�*�*�*�*�*�*� S�R�R��7�7�7�7�7�7�7�7�t�����Y����:8�8�r�y�8� �8�4�8�8�8�8�v"�"�B�I�"�"�"�"�Jx�x�x�x�x�y�x�x�x�|26�26�MQ� - �- � �H�O�- ��C��e�l�2�3�- ���c�4��9�n�-�.�- �%�T�#�Y�/� - � �d�3��U�\�5�<�-G�(H�#H�I�J� - �- �- �- �d04��%)�!��15�26�MQ�P �P � �H�O�P ��u�|�,�P ��P ��'�"� P � � P � � P ���c�4��9�n�-�.�P �%�T�#�Y�/�P ��d�3��U�\�5�<�-G�(H�#H�I�J�P �P �P �P �f +�� � +� +� +� +�PT�,1��!��15�26�MQ�C �C � �I�C ��u�U�\�4��U�\�8I�3J�%J�K�L�C ��4��c�4�i��(� )�C �� C � � C � � C ���c�4��9�n�-�.�C �%�T�#�Y�/�C ��d�3��U�\�5�<�-G�(H�#H�I�J�C �C �C �C �L j� j� j� j� j�� j� j� j�F,�,�,�,�,�,�,�,�,�,r$
Memory