�
���gQ� � �� � d Z ddlZddlZddlZddlZddlZddlZddlZddl Z ddl
Z
ddlZddlZddl
Z
ddlZddlZddlZddlZddlmZ ddlmZ ddlmZmZmZmZmZ ddlmZ ddlmZ ddl!Z"ddl#Z#ddl$m%Z& ddl'm(Z(m)Z)m*Z* dd l+m,Z, dd
l#m-Z- ddl.m/Z/m0Z0m1Z1m2Z2m3Z3 ddl4m5Z5 dd
l6m7Z7 ddl8m9Z9m:Z:m;Z; ddl<m=Z=m>Z> ddl?m@Z@ ddlAmBZB ddlCmDZDmEZE ddlFmGZG ddlHmIZImJZJmKZK ddlLmMZM ddlNmOZO ddlPmQZQmRZRmSZS ddlTmUZUmVZV ddlWmXZXmYZY ddlZm[Z[ ddl\m]Z]m^Z^ ddl_m`Z` ddlambZbmcZcmdZdmeZemfZfmgZgmhZhmiZi ddljmkZkmlZlmmZmmnZnmoZompZpmqZqmrZrmsZsmtZtmuZumvZvmwZwmxZxmyZymzZzm{Z{m|Z|m}Z}m~Z~ ddlm�Z�m�Z�m�Z�m�Z�m�Z�m�Z�m�Z�m�Z�m�Z�m�Z�m�Z�m�Z�m�Z�m�Z�m�Z�m�Z�m�Z�m�Z�m�Z�m�Z�m�Z�m�Z�m�Z� dd l�m�Z�m�Z�m�Z� dd!lm�Z�m�Z�m�Z�m�Z�m�Z�m�Z�m�Z�m�Z�m�Z�m�Z�m�Z�m�Z�m�Z�m�Z�m�Z�m�Z�m�Z�m�Z�m�Z�m�Z�m�Z�m�Z�m�Z�m�Z�m�Z�m�Z�m�Z�m�Z�m�Z�m�Z�m�Z�m�Z�m�Z�m�Z�m�Z�m�Z�m�Z�m�Z�m�Z�m�Z�m�Z� dd"l�m�Z� dd#l�m�Z� ecgZ�efZ� e�� � rdd$l�m�Z� e�Z� e�� � rdd%l�m�Z� e�� � rddl�Z� e�� � rLddl�m�c m�Z� ddl�m�c m�Z� ddl�m5Z� e,j� eئ � e,j� e�� � k Z�e�rddl�m%c m�Z� ddl�m�Z� nd&Z� e�� � r>ddl�m�c m#Z� ddl�m5Z� e,j� e� � e,j� d'� � k Z�dd(ljm�Z�m�Z�m�Z�m�Z� nd&Z� e�� � rddl�Z� e�� � rdd)l�m�Z� e�� � r�dd*l�m�Z�m�Z� ddl�m5Z� dd+l�m�Z� dd,l�m�Z�m�Z�m�Z�m�Z�m�Z�m�Z�m�Z� e2gZ� e,j� e� � e,j� d-� � k rdd.l�m�Z� e,j� e� � e,j� d/� � k rdd0l��m �Z e��e gz
Z� eK� � rdd1l��m�Z e�d2� � rdd3l��m�Z d4� �Zd5� �Zd6� �Zerdd�l�Z e�� � rddl�Z� ej �e� � �Z d7�Z
d8�Zd9�Zd:�Z
d;�Zd<�Zd=�Z G d>� d?� � �ZdS )@uc
The Trainer class, to easily train a 🤗 Transformers from scratch or finetune it on a new task.
� N)�Mapping)�Path)�
TYPE_CHECKING�Any�Callable�Optional�Union� )�#get_reporting_integration_callbacks)� ModelCard�create_repo�
upload_folder)�version)�nn)�
DataLoader�Dataset�IterableDataset�
RandomSampler�SequentialSampler)�__version__)�PretrainedConfig)�DataCollator�DataCollatorWithPadding�default_data_collator)�DebugOption�DebugUnderflowOverflow)�SequenceFeatureExtractor)�FeatureExtractionMixin)�"ALL_HYPERPARAMETER_SEARCH_BACKENDS�default_hp_search_backend)�BaseImageProcessor)�deepspeed_init�deepspeed_load_checkpoint�is_deepspeed_available)�tpu_spmd_dataloader)�TrainingSummary)�PreTrainedModel�load_sharded_checkpoint�unwrap_model)�!MODEL_FOR_CAUSAL_LM_MAPPING_NAMES�MODEL_MAPPING_NAMES)� Adafactor�
get_scheduler)�ProcessorMixin)�ALL_LAYERNORM_LAYERS�"is_torch_greater_or_equal_than_2_3)�PreTrainedTokenizerBase)�CallbackHandler�DefaultFlowCallback�ExportableState�PrinterCallback�ProgressCallback�TrainerCallback�TrainerControl�TrainerState)�DistributedTensorGatherer�EvalLoopContainer�IterableDatasetShard�
LabelSmoother�LayerWiseDummyOptimizer�LengthGroupedSampler�SequentialDistributedSampler�distributed_broadcast_scalars�distributed_concat�find_batch_size�get_model_param_count�get_module_class_from_name�get_parameter_names�
nested_concat�
nested_detach�nested_numpify�nested_xla_mesh_reduce�reissue_pt_warnings�remove_dummy_checkpoint�set_rng_state_for_device)�PREFIX_CHECKPOINT_DIR�BestRun�EvalLoopOutput�EvalPrediction�HPSearchBackend�HubStrategy�PredictionOutput�RemoveColumnsCollator�SaveStrategy�TrainerMemoryTracker�TrainOutput�check_target_module_exists�default_compute_objective�denumpify_detensorize�enable_full_determinism�find_executable_batch_size�get_last_checkpoint�
has_length�neftune_post_forward_hook�number_of_arguments�seed_worker�set_seed�
speed_metrics)�OptimizerNames�ParallelMode�TrainingArguments))�ADAPTER_CONFIG_NAME�ADAPTER_SAFE_WEIGHTS_NAME�ADAPTER_WEIGHTS_NAME�CONFIG_NAME�SAFE_WEIGHTS_INDEX_NAME�SAFE_WEIGHTS_NAME�WEIGHTS_INDEX_NAME�WEIGHTS_NAME�XLA_FSDPV2_MIN_VERSION�PushInProgress�PushToHubMixin�can_return_loss�find_labels�is_accelerate_available�is_apex_available�is_apollo_torch_available�is_bitsandbytes_available�is_datasets_available�is_galore_torch_available�is_grokadamw_available�is_in_notebook�is_ipex_available�is_liger_kernel_available�is_lomo_available�is_peft_available�is_safetensors_available�is_sagemaker_dp_enabled�is_sagemaker_mp_enabled�is_schedulefree_available�is_torch_compile_available�is_torch_hpu_available�is_torch_mlu_available�is_torch_mps_available�is_torch_musa_available�is_torch_neuroncore_available�is_torch_npu_available�is_torch_xla_available�is_torch_xpu_available�is_torchao_available�logging� strtobool)�deprecate_kwarg)�QuantizationMethod)�NotebookProgressCallback)�ampFz1.10)�smp_forward_backward�smp_forward_only�
smp_gather�smp_nested_concat)� PeftModel)�Accelerator�skip_first_batches)�AcceleratorState)�AutocastKwargs�DistributedDataParallelKwargs�DistributedType�load_fsdp_model�load_fsdp_optimizer�save_fsdp_model�save_fsdp_optimizer�1.3.0)�TorchTensorParallelPluginz0.23.0)�SeedableRandomSampler)�DeepSpeedSchedulerWrapper�0.28.0)�DataLoaderConfigurationc � � t � � r|t � � rt fnd}t j t j � d� � � � t j d� � k r
ddlm} g |�|�R }t | |� � S dS )N� �peftz0.7.0r )�PeftMixedModelF) r� r� r �parse� importlib�metadatar� r� �
isinstance)�model�classes_to_checkr� s �d/home/asafur/pinokio/api/open-webui.git/app/env/lib/python3.11/site-packages/transformers/trainer.py�_is_peft_modelr� s� � ���� 3�+<�+>�+>�F�I�<�<�B���=��+�3�3�F�;�;�<�<��
�g�@V�@V�V�V�+�+�+�+�+�+�B�!1�B�>�B�B���%�!1�2�2�2��5� c � � t � � r1dt t j t � � j � � v rddiS i S )N�adapter_onlyT)ru �list�inspect� signaturer� �
parametersr� r� r� �_get_fsdp_ckpt_kwargsr� s@ � �� � � �^�t�G�<M�o�<^�<^�<i�7j�7j�%j�%j���%�%�� r� c � � t j t j � � j t j d� � j k rt j � � S t j t j � � t j d� � k rt j nt j } | j
j t j t j
g}|t t j
t j � � � � gz
}t j � |� � S )Nz2.6�2.0.0)r r� �torchr �release�
contextlib�nullcontext�np�_core�core�
multiarray�_reconstruct�ndarray�dtype�type�uint32�
serialization�safe_globals)�np_core� allowlists r� r� r� s� � � �}�U�&�'�'�/�'�-��2F�2F�2N�N�N��%�'�'�'�!�-���7�7�7�=��;Q�;Q�Q�Q�b�h�h�WY�W^�G��#�0�"�*�b�h�G�I� �$�r�x�� �*�*�+�+�,�,�I���+�+�I�6�6�6r� ztraining_args.binztrainer_state.jsonzoptimizer.ptz scaler.ptz
optimizer.binzscheduler.pt�pytorch_model_fsdpc �
� e Zd ZdZddlmZmZmZmZm Z e
dddd�� � d�deee
j d f ded
ee deeeedf deeeeeef df deeeeeef deeg ef dee deeegef deee deeej j! eej j"