�
���g�l � �> � d dl Z d dlZd dlZd dlZd dlZd dlZd dlmZ d dl m
Z
mZ d dlmZ d dl
mZ d dlmZ d dlmZmZ d dlmZmZmZmZmZ d dlZd dlZd dlZd dlZ d d
l!m"Z" d dl#m$Z$m%Z% d dl&m'Z' d d
l(m)Z)m*Z*m+Z+m,Z,m-Z-m.Z. d dl/m0Z0m1Z1m2Z2m3Z3m4Z4m5Z5 d dl6m7Z7 d dl8m9Z9m:Z: d dl;m<Z<m=Z=m>Z> d dl?m@Z@ d dlAmBZB d dlCmDZDmEZEmFZFmGZG erd dlHZH e@eI� � ZJeeKeLf ZMd� ZNdeOdeOeLeLf fd�ZPdeOdeKdeLdeQeO fd�ZRdzdeOeLeQf dee' de'fd �ZSd!eQeOeLef deOeLeQf fd"�ZTdeOeLeQf deeOeLef fd#�ZU d{d%e
eVeMeOf d&eKd'eWdeeVeMe jX f fd(�ZY G d)� d*� � ZZ G d+� d,eZ� � Z[ G d-� d.e[� � Z\ G d/� d0eZ� � Z] G d1� d2e]� � Z^ G d3� d4eZ� � Z_ G d5� d6eZ� � Z` G d7� d8eZ� � Za G d9� d:eZ� � Zb G d;� d<eZ� � Zcd=eQeL fd>�Zd G d?� d@eZ� � Ze G dA� dBeb� � Zfde jX fdC�Zg G dD� dEeZ� � ZhdFeeOe jX f dGeeWeQe ji e jj e jk f dHeLfdI�ZldJedFeeOe jX f dHeLfdK�ZmdJedFeeOe jX f dHeLfdL�Zn G dM� dNeh� � Zo G dO� dPeZ� � Zp G dQ� dReZ� � Zq G dS� dTeZ� � Zr G dU� dVeZ� � ZsdeOdWe'dXeOeLeeLeWdf f deOfdY�ZtdeOdWe'dXeOeLeeLeWdf f deOfdZ�Zue G d[� d\� � � � Zv G d]� d^eZ� � Zwe G d_� d`� � � � Zxe G da� db� � � � Zydc� ZzddeeKdef deeKdef fdf�Z{ G dg� dhe%� � Z| d|dieQe| djee7 dkee9 dleKde|f
dm�Z} d}doeQe| dpeeQe~ dqeeK djee7 dkee9 dreBds de|fdt�Zdue|dveKdweKde|fdx�Z�dy� Z�dS )~� N)�Counter)�Iterable�Iterator)�deepcopy)� dataclass)�partial)�cycle�islice)�
TYPE_CHECKING�Any�Callable�Optional�Union� )�config)�Dataset�DatasetInfoMixin)�Features)�FeatureType�Value�_align_features�!_check_if_features_can_be_aligned�_visit�cast_to_python_objects)�ArrowFormatter�PythonFormatter�TableFormatter�TensorFormatter�get_format_type_from_alias�
get_formatter)�DatasetInfo)�
NamedSplit�Split)�cast_table_to_features�read_schema_from_file�
table_cast)�
get_logger)�Literal)�_merge_gen_kwargs�_number_of_shards_in_gen_kwargs�_shuffle_gen_kwargs�_split_gen_kwargsc � � | S �N� )�xs �i/home/asafur/pinokio/api/open-webui.git/app/env/lib/python3.11/site-packages/datasets/iterable_dataset.py�
identity_funcr2 7 s � ��H� �example�column_mappingc � � � t � fd�|D � � � � rbt dt |� � � dt |� � � � � � dt |� � t � � � z
� d�� � �t � fd�|� � � D � � � � rtt dt |� � � dt |� � � � � � dt � � � t |� � � � � z
� d�� � �� fd�|� � � D � � S ) Nc 3 � �K � | ]}|�vV � � d S r. r/ ��.0�colr4 s �r1 � <genexpr>z%_rename_columns_fn.<locals>.<genexpr><