� ��g��ddlZddlmZmZddlmZddlmZmZej e ��ZGd�d��Zed��Z d �Zd �Zeejdefd��ZdS) �N)�Pool�RLock)�tqdm�)�experimental�loggingc��eZdZdZdS)�ParallelBackendConfigN)�__name__� __module__�__qualname__�backend_name��j/home/asafur/pinokio/api/open-webui.git/app/env/lib/python3.11/site-packages/datasets/parallel/parallel.pyr r s��L�L�Lrr c �v�tj�t|||||||||� � St|||||||||� � S)a� **Experimental.** Apply a function to iterable elements in parallel, where the implementation uses either multiprocessing.Pool or joblib for parallelization. Args: function (`Callable[[Any], Any]`): Function to be applied to `iterable`. iterable (`list`, `tuple` or `np.ndarray`): Iterable elements to apply function to. num_proc (`int`): Number of processes (if no backend specified) or jobs (using joblib). types (`tuple`): Additional types (besides `dict` values) to apply `function` recursively to their elements. disable_tqdm (`bool`): Whether to disable the tqdm progressbar. desc (`str`): Prefix for the tqdm progressbar. single_map_nested_func (`Callable`): Map function that applies `function` to an element from `iterable`. Takes a tuple of function, data_struct, types, rank, disable_tqdm, desc as input, where data_struct is an element of `iterable`, and `rank` is used for progress bar. )r r�_map_with_multiprocessing_pool�_map_with_joblib) �function�iterable�num_proc�batched� batch_size�types�disable_tqdm�desc�single_map_nested_funcs r�parallel_maprs]��"�)�1�-��h��'�:�u�l�TX�Zp� � � ��(�H�g�z�5�,�PT�Vl��rc ��|t|��kr|nt|��}g} t|��D]q} t|��|z}t|��|z}|| zt| |��z} | |z| |krdndz}| �||| |�|||| ||f��rt|��t d�| D��kr9tdt|��dt d�| D��t�d|�dt|��d d �| D��d\}}|st��ftj }}t|||��5}|�|| ��}ddd��n#1swxYwYt�d |�d��d�|D��}t�dt|��d��|S)N�rc3�@K�|]}t|d��V��dS�r N��len��.0�is r� <genexpr>z1_map_with_multiprocessing_pool.<locals>.<genexpr>7s,��:�:�!�C��!��I�I�:�:�:�:�:�:rzHError dividing inputs iterable among processes. Total number of objects z , length: c3�@K�|]}t|d��V��dSr"r#r%s rr(z1_map_with_multiprocessing_pool.<locals>.<genexpr>;s,��9�9��3�q��t�9�9�9�9�9�9�9�9rz Spawning z processes for z objects in slices of c�8�g|]}t|d��S)r r#r%s r� <listcomp>z2_map_with_multiprocessing_pool.<locals>.<listcomp>?s)��Rq�Rq�Rq�ab�SV�WX�YZ�W[�S\�S\�Rq�Rq�Rqr)NN)�initargs�initializerz Finished z processesc��g|] }|D]}|�� Srr)r&�proc_res�objs rr+z2_map_with_multiprocessing_pool.<locals>.<listcomp>Gs%�� =� =� =�h�H� =� =�S�c� =� =� =� =rz Unpacked z objects) r$�range�min�append�sum� ValueError�logger�inforr�set_lockr�map)rrrrrrrrr� split_kwds�index�div�mod�start�endr,r-�pool�mappeds rrr+s|��$�s�8�}�}�4�4�x�x�#�h�-�-�H��J��x��r�r��(�m�m�x�'��(�m�m�h�&��e��c�%��o�o�-��c�k�%�#�+�+�Q�Q�1�5��8�X�e�C�i�%8�'�:�u�V[�]i�ko�p�q�q�q�q� �8�}�}��:�:�z�:�:�:�:�:�:�:�� <�'*�8�}�}� <� <��9�9�j�9�9�9�9�9� <� <� � � ��K�K�s�H�s�s�S��]�]�s�s�Rq�Rq�fp�Rq�Rq�Rq�s�s��'��H�k��:�!&�� D�M�+�� h��{� C� C� C�>�t��0�*�=�=��>�>�>�>�>�>�>�>�>�>�>��>�>�>�>� �K�K�0�H�0�0�0�1�1�1� =� =�&� =� =� =�F� �K�K�1�C��K�K�1�1�1�2�2�2��Ms�9F�F �#F c �� ddl� � �tj|��5� �� fd�|D��cddd��S#1swxYwYdS)Nr)�n_jobsc3�d�K�|]*}��|��dddf��V��+dS)NT)�delayed)r&r0rrr�joblibrrs ��rr(z#_map_with_joblib.<locals>.<genexpr>Usc��! �! �� 3�F�N�N�1�2�2�H�c�7�J�X]�_c�ei�ko�3p�q�q�! �! �! �! �! �! r)rF�parallel_backendr r�Parallel) rrrrrrrrrrFs ` ``` `@rrrMs�� M�M�M� � � �!6�!C�H� � U� U� � � �v�� ! �! �! �! �! �! �! �! �! ��! �! �! � � � � � � � � � � � � � � �� s�.A'�'A+�.A+rc#�K�|t_|dkrddlm}|�� dV�dt_dS#dt_wxYw)a� **Experimental.** Configures the parallel backend for parallelized dataset loading, which uses the parallelization implemented by joblib. Args: backend_name (str): Name of backend for parallelization implementation, has to be supported by joblib. Example usage: ```py with parallel_backend('spark'): dataset = load_dataset(..., num_proc=2) ``` �sparkr)�register_sparkN)r r�joblibsparkrK)rrKs rrGrG[sn�� *6��&��w��.�.�.�.�.�.�� 2� ��-1��*�*�*��T��*�1�1�1�1s �8�A)� contextlib�multiprocessingrr� tqdm.autor�utilsrr� get_loggerrr6r rrr�contextmanager�strrGrrr�<module>rTs��'�'�'�'�'�'�'�'��)�)�)�)�)�)�)�)� �� H� %� %��4��D � � ��2�3�2�2�2��2�2�2r