�
���g�a � �r � d Z ddlZddlZddlZddlZddlZddlZddlZddlZddl Z ddl
Z
ddlmZm
Z
ddlmZ ddlmZ ddlmZ ddlmZmZmZ ddlmZ ddlZdd lmZ dd
lmZ ddlm Z dd
l!m"Z"m#Z# ddl$m%Z% ddl&m'Z'm(Z( ddl)m*Z*m+Z+m,Z, ddl-m.Z.m/Z/m0Z0 ddl1m2Z2m3Z3 ddl4m5Z5 ddl6m7Z7m8Z8 ddl9m:Z:m;Z; ddl<m=Z=m>Z>m?Z?m@Z@ ddlAmBZB ddlCmDZDmEZE ddlFmGZG ddlHmIZImJZJmKZK ddlLmMZMmNZNmOZO ddlPmQZQ ddlRmSZSmTZT ddlUmVZVmWZWmXZXmYZY ddlZm[Z[ dd l\m]Z] dd!l#m^Z^ dd"l#m_Z` dd#lambZb dd$lcmdZd dd%lemfZfmgZgmhZhmiZi dd&ljmkZkmlZlmmZmmnZnmoZompZpmqZqmrZr dd'lsmtZtmuZu dd(lvmwZw erdd)lxmyZy e^jz e{� � Z| G d*� d+e}� � Z~e G d,� d-� � � � Z G d.� d/� � Z� G d0� d1e�� � Z� G d2� d3e�� � Z�dS )4zDatasetBuilder base class.� N)�Iterable�Mapping)� dataclass)�partial)�Path)�
TYPE_CHECKING�Optional�Union)�patch)� url_to_fs)�Pool)�
thread_map� )�config�utils)�Dataset)�ArrowReader�ReadInstruction)�ArrowWriter�
ParquetWriter�SchemaInferenceError)�
DataFilesDict�DataFilesPatternsDict�sanitize_patterns)�DatasetDict�IterableDatasetDict)�DownloadConfig)�DownloadManager�DownloadMode)�StreamingDownloadManager�xjoin)�DatasetGenerationCastError�DatasetGenerationError�FileFormatError�ManualDownloadError)�Features)�is_remote_filesystem�rename)�Hasher)�DatasetInfo�DatasetInfosDict�PostProcessedInfo)�ArrowExamplesIterable�ExamplesIterable�IterableDataset)�DuplicatedKeysError)�"INVALID_WINDOWS_CHARACTERS_IN_PATH�camelcase_to_snakecase)�Split� SplitDict�SplitGenerator� SplitInfo)�$extend_dataset_builder_for_streaming)� CastError)�logging)�tqdm)�FileLock)�
is_remote_url)�VerificationMode�get_size_checksum_dict�verify_checksums�
verify_splits)�
classproperty�convert_file_size_to_int�has_sufficient_disk_space�iflatmap_unordered�
map_nested�memoize�size_str�temporary_assignment)�_number_of_shards_in_gen_kwargs�_split_gen_kwargs)�tracked_list)�
DatasetModulec � � e Zd ZdS )�InvalidConfigNameN)�__name__�
__module__�__qualname__� � �`/home/asafur/pinokio/api/open-webui.git/app/env/lib/python3.11/site-packages/datasets/builder.pyrN rN \ s � � � � � ��DrS rN c � � e Zd ZU dZdZeed<