� ?��g3:��6�dZddlmZmZddlmZmZmZmZm Z m Z m Z m Z m Z mZmZeZgZ ddlmZn #e$re�d��YnwxYw ddlmZn #e$re�d��YnwxYw ddlmZn #e$re�d ��YnwxYw dd lmZmZmZmZmZm Z m!Z!m"Z"m#Z#m$Z$dd l%Z%e%j&e"��n #e$re�d ��YnwxYwd �Z'd�Z(dd�d�Z)ddd�d�Z* dddd�d�Z+Gd�de ��Z,d S)zO FileSystem abstraction to interact with various local and remote filesystems. �)� _is_path_like�_stringify_path) � FileSelector�FileType�FileInfo� FileSystem�LocalFileSystem�SubTreeFileSystem�_MockFileSystem�FileSystemHandler� PyFileSystem� _copy_files�_copy_files_selector)�AzureFileSystemr)�HadoopFileSystemr)� GcsFileSystemr) �AwsDefaultS3RetryStrategy�AwsStandardS3RetryStrategy� S3FileSystem� S3LogLevel�S3RetryStrategy�ensure_s3_initialized� finalize_s3�ensure_s3_finalized� initialize_s3�resolve_s3_regionNrc��|tvr"td�|�����td�|�����)Nz<The pyarrow installation is not built with support for '{0}'z*module 'pyarrow.fs' has no attribute '{0}')� _not_imported� ImportError�format�AttributeError)�names �Z/home/asafur/pinokio/api/open-webui.git/app/env/lib/python3.11/site-packages/pyarrow/fs.py� __getattr__r$IsT�� �}���� ��F�4�L�L� � � � �4�;�;�D�A�A� � ��c�J�tj|��\}}|�|��}|rt|�|g��d}|jt jkr3td�|jj |j |�����t||��}|S)Nrz�The path component of the filesystem URI must point to a directory but it has a type: `{}`. The path component is `{}` and the given filesystem URI is `{}`) r�from_uri�normalize_path� get_file_info�typer� Directory� ValueErrorr r"�pathr )�uri� filesystem�prefix� prefix_infos r#�_filesystem_from_strr2Us���$�,�S�1�1��J�� � &� &�v� .� .�F� � ;� �.�.��x�8�8��;� � � �x�1� 1� 1��?�?E�v��$�)�;�+;�S�@�@��� �'�v�z�:�:� � �r%F��use_mmapc���t|t��r|St|t��r |rtd���t |��S ddl}t||j��rDt|��jdkrt|���Stt|����Sn#t$rYnwxYwtd�t|�������)NzXSpecifying to use memory mapping not supported for filesystem specified as an URI stringrr r3zlUnrecognized filesystem: {}. `filesystem` argument must be a FileSystem instance or a valid file system URI')� isinstancer�strr,r2�fsspec�AbstractFileSystemr*�__name__r r � FSSpecHandlerr� TypeErrorr )r/r4r8s r#�_ensure_filesystemr=is���*�j�)�)�0��� �J�� $� $�0� � ��8��� �$�J�/�/�/� ;�� � � � �j�&�";� <� <� ;��J���(�,=�=�=�&��9�9�9�9�� �j� 9� 9�:�:� :�  ;�� � � � � �� ���� � :�:@�&� �� � �;�;� � �s�B,�, B9�8B9)� memory_mapc���t|��s|�td���||fS|�st||���}t|t��rt |��}n$t|t ��std���|�|��}||fSt |��}t |���} |� |��}|j tj k}n#t$rd}d}YnwxYw|sS tj|��\}}nO#t$r-}dt |��vrdt |��vr�Yd}~nd}~wwxYw|�|��}||fS)z` Return filesystem/path from path which could be an URI or a plain filesystem path. Nzg'filesystem' passed but the specified path is file-like, so there is nothing to open with 'filesystem'.r3zPExpected string path; path-like objects are only allowed with a local filesystemFz empty schemezCannot parse URI)rr,r=r6r rr7r<r(r)r*r�NotFoundrr')r-r/r>� file_info�exists_locally�es r#�_resolve_filesystem_and_pathrD�s��� �� � � � � !��?��� ��4�����'� �Z�H�H�H� � �j�/� 2� 2� �"�4�(�(�D�D��D�#�&�&� ��*��� ��(�(��.�.���4��� �4� � �D� !�*�5�5�5�J�?��,�,�T�2�2� � $�.�H�,=�=���� ����� ��������� � /� �)�2�4�8�8� �J����� � � ��S��V�V�+�+�*�#�a�&�&�8�8�����������  �����(�(��.�.�� �t� �s*�:C%�%C6�5C6�<D� E �#E�E iT)� chunk_size� use_threadsc��t||��\}}t||��\}} |�|��} | jtjkr't |d���} t || || ||��dSt|||| ||��dS)a� Copy files between FileSystems. This functions allows you to recursively copy directories of files from one file system to another, such as from S3 to your local machine. Parameters ---------- source : string Source file path or URI to a single file or directory. If a directory, files will be copied recursively from this path. destination : string Destination file path or URI. If `source` is a file, `destination` is also interpreted as the destination file (not directory). Directories will be created as necessary. source_filesystem : FileSystem, optional Source filesystem, needs to be specified if `source` is not a URI, otherwise inferred. destination_filesystem : FileSystem, optional Destination filesystem, needs to be specified if `destination` is not a URI, otherwise inferred. chunk_size : int, default 1MB The maximum size of block to read before flushing to the destination file. A larger chunk_size will use more memory while copying but may help accommodate high latency FileSystems. use_threads : bool, default True Whether to use multiple threads to accelerate copying. Examples -------- Inspect an S3 bucket's files: >>> s3, path = fs.FileSystem.from_uri( ... "s3://registry.opendata.aws/roda/ndjson/") >>> selector = fs.FileSelector(path) >>> s3.get_file_info(selector) [<FileInfo for 'registry.opendata.aws/roda/ndjson/index.ndjson':...] Copy one file from S3 bucket to a local directory: >>> fs.copy_files("s3://registry.opendata.aws/roda/ndjson/index.ndjson", ... "file:///{}/index_copy.ndjson".format(local_path)) >>> fs.LocalFileSystem().get_file_info(str(local_path)+ ... '/index_copy.ndjson') <FileInfo for '.../index_copy.ndjson': type=FileType.File, size=...> Copy file using a FileSystem object: >>> fs.copy_files("registry.opendata.aws/roda/ndjson/index.ndjson", ... "file:///{}/index_copy.ndjson".format(local_path), ... source_filesystem=fs.S3FileSystem()) T�� recursiveN)rDr)r*rr+rrr) �source� destination�source_filesystem�destination_filesystemrErF� source_fs� source_path�destination_fs�destination_pathrA� source_sels r#� copy_filesrS�s���p:��!����I�{�(D��+�(�(�$�N�$��'�'� �4�4�I��~��+�+�+�!�+��>�>�>� ��Y� �+�-=�'�� 6� 6� 6� 6� 6� �I�{�"�$4�� � -� -� -� -� -r%c��eZdZdZd�Zd�Zd�Zd�Zd�Ze d���Z d�Z d �Z d �Z d �Zd �Zd �Zd�Zd�Zd�Zd�Zd�Zd�Zd�Zd�ZdS)r;a Handler for fsspec-based Python filesystems. https://filesystem-spec.readthedocs.io/en/latest/index.html Parameters ---------- fs : FSSpec-compliant filesystem instance Examples -------- >>> PyFileSystem(FSSpecHandler(fsspec_fs)) # doctest: +SKIP c��||_dS�N)�fs)�selfrWs r#�__init__zFSSpecHandler.__init__s ������r%c�Z�t|t��r|j|jkStSrV�r6r;rW�NotImplemented�rX�others r#�__eq__zFSSpecHandler.__eq__�)�� �e�]� +� +� '��7�e�h�&� &��r%c�Z�t|t��r|j|jkStSrVr[r]s r#�__ne__zFSSpecHandler.__ne__#r`r%c�~�|jj}t|t��r|d}d�|��S)Nrz fsspec+{0})rW�protocolr6�listr )rXrds r#� get_type_namezFSSpecHandler.get_type_name(s;���7�#�� �h�� %� %� #���{�H��"�"�8�,�,�,r%c��|SrV��rXr-s r#r(zFSSpecHandler.normalize_path.s��� r%c ���|d}|ddkr tj}n'|ddkrtj}d}n tj}t ||||�dd�����S)N�sizer*�file� directory�mtime)rkrn)r�Filer+�Unknownr�get)r-�infork�ftypes r#�_create_file_infozFSSpecHandler._create_file_info1sn���F�|�� ��<�6� !� !��M�E�E� �&�\�[� (� (��&�E��D�D��$�E���e�$�d�h�h�w��6M�6M�N�N�N�Nr%c��g}|D]�} |j�|��}|�|�||�����G#t$r0|�t |t j����Y��wxYw|SrV)rWrr�appendrt�FileNotFoundErrorrrr@)rX�paths�infosr-rrs r#r)zFSSpecHandler.get_file_info>s������ A� A�D� A��w�|�|�D�)�)��� � �T�3�3�D�$�?�?�@�@�@�@��%� @� @� @�� � �X�d�H�,=�>�>�?�?�?�?�?� @����� s�A � 7B�Bc�6�|j�|j��sP|j�|j��rt |j���|jrgSt |j���|jrd}nd}g}|j�|j|dd���}|� ��D]c\}}|� d��}|j� d��}||kr)|� |� ||�����d|S)N�T)�maxdepth�withdirs�detail�/) rW�isdir�base_dir�exists�NotADirectoryError�allow_not_foundrwrI�find�items�striprvrt) rX�selectorr|ry�selected_filesr-rr�_pathr�s r#�get_file_info_selectorz$FSSpecHandler.get_file_info_selectorIs%���w�}�}�X�.�/�/� ?��w�~�~�h�/�0�0� ?�(��):�;�;�;��+�?��I�+�H�,=�>�>�>� � � ��H�H��H������� � ��4��&� � ��)�.�.�0�0� A� A�J�D�$��J�J�s�O�O�E��(�.�.�s�3�3�H��� � �� � �T�3�3�D�$�?�?�@�@�@��� r%c�b� |j�||���dS#t$rYdSwxYw)N)�create_parents)rW�mkdir�FileExistsError)rXr-rIs r#� create_dirzFSSpecHandler.create_dirfsG�� � �G�M�M�$�y�M� 9� 9� 9� 9� 9��� � � � �D�D� ���s � � .�.c�>�|j�|d���dS�NTrH)rW�rmris r#� delete_dirzFSSpecHandler.delete_dirms �� �� � �4�4� �(�(�(�(�(r%c�J� |j�|d���}n#t$r|rYdS�wxYw|D]m}|j�|��r|j�|d����9|j�|��r|j�|���ndS)NF)r~TrH)rW�listdirrwr�r��isfile)rXr-�missing_dir_ok�subpaths�subpaths r#�_delete_dir_contentsz"FSSpecHandler._delete_dir_contentsps��� ��w���t�E��:�:�H�H�� � � � �� ���� � ���� � $� $�G��w�}�}�W�%�%� $��� � �7�d� �3�3�3�3������(�(� $��� � �7�#�#�#��  $� $s �� 0�0c��|�d��dkrtd|d���|�||��dS)Nr�z$delete_dir_contents called on path '�')r�r,r�)rXr-r�s r#�delete_dir_contentsz!FSSpecHandler.delete_dir_contents}sO�� �:�:�c�?�?�b� � ��6��c�C�C� C� �!�!�$��7�7�7�7�7r%c�0�|�d��dS)Nr)r�)rXs r#�delete_root_dir_contentsz&FSSpecHandler.delete_root_dir_contents�s�� �!�!�#�&�&�&�&�&r%c��|j�|��st|���|j�|��dSrV)rWr�rwr�ris r#� delete_filezFSSpecHandler.delete_file�sB���w�~�~�d�#�#� *�#�D�)�)� )� �� � �4�����r%c�@�|j�||d���dSr�)rW�mv�rX�src�dests r#�movezFSSpecHandler.move�s"�� �� � �3��� �-�-�-�-�-r%c�<�|j�||��dSrV)rW�copyr�s r#� copy_filezFSSpecHandler.copy_file�s �� �� � �S�$�����r%c��ddlm}|j�|��st |���||j�|d���d���S�Nr�� PythonFile�rb��mode�r��pyarrowr�rWr�rw�open�rXr-r�s r#�open_input_streamzFSSpecHandler.open_input_stream��_��&�&�&�&�&�&��w�~�~�d�#�#� *�#�D�)�)� )��z�$�'�,�,�t�$�,�7�7�c�B�B�B�Br%c��ddlm}|j�|��st |���||j�|d���d���Sr�r�r�s r#�open_input_filezFSSpecHandler.open_input_file�r�r%c�\�ddlm}||j�|d���d���S)Nrr��wbr��w�r�r�rWr��rXr-�metadatar�s r#�open_output_streamz FSSpecHandler.open_output_stream��;��&�&�&�&�&�&��z�$�'�,�,�t�$�,�7�7�c�B�B�B�Br%c�\�ddlm}||j�|d���d���S)Nrr��abr�r�r�r�s r#�open_append_streamz FSSpecHandler.open_append_stream�r�r%N)r:� __module__� __qualname__�__doc__rYr_rbrfr(� staticmethodrtr)r�r�r�r�r�r�r�r�r�r�r�r�r�rhr%r#r;r; sg������ � ������� ��� -�-�-� ���� O� O��\� O� � � ����:���)�)�)� $� $� $�8�8�8� '�'�'����.�.�.� � � �C�C�C�C�C�C�C�C�C� C�C�C�C�Cr%r;rV)NN)-r�� pyarrow.utilrr� pyarrow._fsrrrrr r r r r rr� FileStatsr�pyarrow._azurefsrrrv� pyarrow._hdfsr�pyarrow._gcsfsr� pyarrow._s3fsrrrrrrrrrr�atexit�registerr$r2r=rDrSr;rhr%r#�<module>r�si��$��8�7�7�7�7�7�7�7� � � � � � � � � � � � � � � � � � � � � � � � � � � � �� �,�0�0�0�0�0�0�0���,�,�,����*�+�+�+�+�+�,����-�.�.�.�.�.�.�.���-�-�-����+�,�,�,�,�,�-����*�,�,�,�,�,�,�,���*�*�*�����)�)�)�)�)�*���� )�L�L�L�L�L�L�L�L�L�L�L�L�L�L�L�L�L�L�L�L�L�L�L�L��M�M�M��F�O�'�(�(�(�(���)�)�)�����(�(�(�(�(�)���� � � ����(05������<7�e�7�7�7�7�7�v?C�H-�&�D�H-�H-�H-�H-�H-�VcC�cC�cC�cC�cC�%�cC�cC�cC�cC�cCsE�1�A� A�A�A6�5A6�:B�B�B�"C�C,�+C,
Memory