� ���g����ddlZddlZddlmZddlmZmZddlmZddl m Z m Z m Z m Z mZmZmZmZmZmZmZmZmZmZmZmZmZmZmZmZmZmZm Z ddl!m"Z"ddl#m$Z$dd l%m&Z&m'Z'e"e(��Z)erdd l*m+Z+dd eefd �Z,dd�Z-dS)�N)�wraps)� TYPE_CHECKING�Optional�)�DownloadConfig)� xbasename�xdirname� xet_parse�xexists�xgetsize�xglob� xgzip_open�xisdir�xisfile�xjoin�xlistdir� xnumpy_load�xopen�xpandas_read_csv�xpandas_read_excel�xPath�xpyarrow_parquet_read_table�xrelpath� xsio_loadmat�xsplit� xsplitext�xwalk�xxml_dom_minidom_parse)� get_logger)�patch_submodule)� get_imports�lock_importable_file��DatasetBuilder�download_configc� ��tj|��}t|d��rE|jr>t |jt ��r"�j|j_�j|j_dS�fd�}t|d|t����� ��t|d|t����� ��t|d|t����� ��t|d|t����� ��t|dt��� ��t|d t��� ��t|d t ��� ��t|d t"��� ��t|d t$��� ��t|d t&��� ��t|d|t(����� ��t|d|t*����� ��t|d|t,����� ��t|d|t.����� ��t|dt0��� ��t|d|t2����� ��t|d|t4����� ��t|d|t6��dg���� ��t|d|t8��dg���� ��t|d|t:��dg���� ��t|d|t<����� ��t|d|t>����� ��|j �!d��s1t|d|tD����� ���|_dS)a?Extend the module to support streaming. We patch some functions in the module to use `fsspec` to support data streaming: - We use `fsspec.open` to open and read remote files. We patch the module function: - `open` - We use the "::" hop separator to join paths and navigate remote compressed/archive files. We patch the module functions: - `os.path.join` - `pathlib.Path.joinpath` and `pathlib.Path.__truediv__` (called when using the "/" operator) The patched functions are replaced with custom functions defined to work with the :class:`~download.streaming_download_manager.StreamingDownloadManager`. Args: module_path: Path to the module to be extended. download_config: Mainly use `token` or `storage_options` to support different platforms and auth types. �_patched_for_streamingNc�N���t�����fd���}d|_|S)Nc����|d�i|��S)Nr%�)�args�kwargsr%�functions ���b/home/asafur/pinokio/api/open-webui.git/app/env/lib/python3.11/site-packages/datasets/streaming.py�wrapperz?extend_module_for_streaming.<locals>.wrap_auth.<locals>.wrapperIs����8�T�M�?�M�f�M�M� M�� wrap_auth)r�_decorator_name_)r-r/r%s` �r.r1z.extend_module_for_streaming.<locals>.wrap_authHsG���� �x��� N� N� N� N� N� �� N�$/�� ��r0�openz os.listdirzos.walkz glob.globz os.path.joinzos.path.dirnamezos.path.basenamezos.path.relpathz os.path.splitzos.path.splitextzos.path.existsz os.path.isdirzos.path.isfilezos.path.getsizez pathlib.Pathz gzip.openz numpy.loadzpandas.read_csv� __version__)�attrszpandas.read_excelzscipy.io.loadmatzxml.etree.ElementTree.parsezxml.dom.minidom.parsezdatasets.packaged_modules.zpyarrow.parquet.read_table)#� importlib� import_module�hasattrr'� isinstancer�token�storage_optionsr r�startrrr rr rrrrr rrr rrrrrrr r�__name__� startswithr)� module_pathr%�moduler1s ` r.�extend_module_for_streamingrA,s+���&� $�[� 1� 1�F��v�/�0�0��V�5R�� �f�3�^� D� D� \�2A�2G�F� )� /�<K�<[�F� )� 9���������F�F�I�I�e�$4�$4�5�5�;�;�=�=�=��F�L�)�)�H�*=�*=�>�>�D�D�F�F�F��F�I�y�y��'7�'7�8�8�>�>�@�@�@��F�K���5�)9�)9�:�:�@�@�B�B�B��F�N�E�2�2�8�8�:�:�:��F�-�x�8�8�>�>�@�@�@��F�.� �:�:�@�@�B�B�B��F�-�x�8�8�>�>�@�@�@��F�O�V�4�4�:�:�<�<�<��F�.� �:�:�@�@�B�B�B��F�,�i�i��.@�.@�A�A�G�G�I�I�I��F�O�Y�Y�v�->�->�?�?�E�E�G�G�G��F�,�i�i��.@�.@�A�A�G�G�I�I�I��F�-�y�y��/B�/B�C�C�I�I�K�K�K��F�N�E�2�2�8�8�:�:�:��F�K���:�)>�)>�?�?�E�E�G�G�G��F�L�)�)�K�*@�*@�A�A�G�G�I�I�I��F�-�y�y�9I�/J�/J�S`�Ra�b�b�b�h�h�j�j�j��F�/���;M�1N�1N�Wd�Ve�f�f�f�l�l�n�n�n��F�.� � �,�0G�0G�P]��_�_�_�e�e�g�g�g��F�9�9�9�Y�;O�;O�P�P�V�V�X�X�X��F�3�Y�Y�?U�5V�5V�W�W�]�]�_�_�_� �?� %� %�&B� C� C�n��� <�i�i�Hc�>d�>d�e�e�k�k�m�m�m�$3�F�!�!�!r0�builderr$c���t|j|j���}t|j|���|j�d��s�t j|j��}t|��5t|��D]`}|ddkrR|d}d� |j� d��dd �|gz��}t||����a ddd��n #1swxYwYdd l m��fd �t|��jdd�D��}|D]}t||����dS) z�Extend the dataset builder module and the modules imported by it to support streaming. Args: builder (:class:`DatasetBuilder`): Dataset builder instance. )r;r:)r%z datasets.r�internalr�.N�����r#c�^��g|])}t|���r|j�jk�"|j��*Sr*)� issubclass� __module__)�.0�clsr$s �r.� <listcomp>z8extend_dataset_builder_for_streaming.<locals>.<listcomp>�sI������ � �c�>� *� *��03�~��AZ�/Z�/Z� ��/Z�/Z�/Zr0)rr;r:rArIr>�inspect�getfile� __class__r"r!�join�splitrBr$�type�__mro__) rBr%�importable_file�imports�internal_import_name�internal_module_name�parent_builder_modulesr@r$s @r.�$extend_dataset_builder_for_streamingrYps����%�W�5L�T[�Ta�b�b�b�O��� 2�O�T�T�T�T� � � (� (�� 5� 5�g�!�/�'�*;�<�<�� !�/� 2� 2� g� g�&��7�7� g� g���1�:��+�+�+2�1�:�(�+.�8�8�G�4F�4L�4L�S�4Q�4Q�RU�SU�RU�4V�Zn�Yo�4o�+p�+p�(�/�0D�Ve�f�f�f�f��  g� g� g� g� g� g� g� g� g� g� g� g���� g� g� g� g�(�'�'�'�'�'�������=�=�(����,����� )�M�M��#�F�O�L�L�L�L�L�M�Ms�5A1C3�3C7�:C7)N)rBr$).r6rM� functoolsr�typingrr�download.download_configr�utils.file_utilsrr r r r r rrrrrrrrrrrrrrrrr� utils.loggingr�utils.patchingr �utils.py_utilsr!r"r=�loggerrBr$rArYr*r0r.�<module>rbs ����������������*�*�*�*�*�*�*�*�4�4�4�4�4�4���������������������������������������������������2&�%�%�%�%�%�+�+�+�+�+�+�=�=�=�=�=�=�=�=� ��H� � ���(�'�'�'�'�'�'�A4�A4�h�~�>V�A4�A4�A4�A4�HM�M�M�M�M�Mr0
Memory