�
?��go� � �P � d Z ddlZddlmZmZmZ ddlmZm Z m
Z
m Z m Z m
Z
mZmZmZmZmZmZmZmZmZmZmZmZmZmZm Z m Z m Z m Z m Z m!Z!m"Z"m#Z#m$Z$m$Z%m&Z& n&# e'$ r Z( e'd e)e(� � � d�� � d�dZ([(ww xY wddl*m+Z+m,Z,m-Z- dZ.d Z/ dd
l0m1Z1 d Z.n # e'$ r Y nw xY wdZ2d Z3 dd
l4m5Z5m6Z6m7Z7m8Z8m9Z9m:Z:m;Z;m<Z< d Z2n # e'$ r Y nw xY w ddl=m>Z>m?Z? n # e'$ r Y nw xY wd� Z@ d d�ZAd� ZBd� ZCd d�ZDd d�ZE d d�ZFd d�ZGd d�ZH d!d�ZI d d�ZJd� ZKdddddddd dddddddd d �d �ZLdS )"zEDataset is currently unstable. APIs subject to change without notice.� N)� _is_iterable�_stringify_path�
_is_path_like) �
CsvFileFormat�CsvFragmentScanOptions�JsonFileFormat�JsonFragmentScanOptions�Dataset�DatasetFactory�DirectoryPartitioning�FeatherFileFormat�FilenamePartitioning�
FileFormat� FileFragment�FileSystemDataset�FileSystemDatasetFactory�FileSystemFactoryOptions�FileWriteOptions�Fragment�FragmentScanOptions�HivePartitioning�
IpcFileFormat�IpcFileWriteOptions�InMemoryDataset� Partitioning�PartitioningFactory�Scanner�TaggedRecordBatch� UnionDataset�UnionDatasetFactory� WrittenFile�get_partition_keysr" �_filesystemdataset_writezBThe pyarrow installation is not built with support for 'dataset' (�))�
Expression�scalar�fieldFzKThe pyarrow installation is not built with support for the ORC file format.)�
OrcFileFormatTzOThe pyarrow installation is not built with support for the Parquet file format.)�ParquetDatasetFactory�ParquetFactoryOptions�ParquetFileFormat�ParquetFileFragment�ParquetFileWriteOptions�ParquetFragmentScanOptions�ParquetReadOptions� RowGroupInfo)�ParquetDecryptionConfig�ParquetEncryptionConfigc �� � | dk rt st t � � �| dk rt st t � � �t d� | � � � � �)Nr( r+ z/module 'pyarrow.dataset' has no attribute '{0}')�_orc_available� ImportError�_orc_msg�_parquet_available� _parquet_msg�AttributeError�format)�names �_/home/asafur/pinokio/api/open-webui.git/app/env/lib/python3.11/site-packages/pyarrow/dataset.py� __getattr__r= m sc � � �� � �~� ��(�#�#�#� �"�"�"�+=�"��,�'�'�'�
�9�@�@��F�F� � � � c � � |��| �<|�t d� � �|dk rt j | �� � S t | |� � S |�Xt |t � � rt j |� � S t d� t
|� � � � � � �t d� � �|dk r�| �<|�t d� � �|dk rt j | �� � S t | |� � S |�Xt |t � � rt j |� � S t d� t
|� � � � � � �t d� � �|d k r�|�t d
� � �| �tt | t j � � r+|dk rt j | �� � S t | |� � S t d � t
| � � � � � � �t j � � S t d � � �)
a
Specify a partitioning scheme.
The supported schemes include:
- "DirectoryPartitioning": this scheme expects one segment in the file path
for each field in the specified schema (all fields are required to be
present). For example given schema<year:int16, month:int8> the path
"/2009/11" would be parsed to ("year"_ == 2009 and "month"_ == 11).
- "HivePartitioning": a scheme for "/$key=$value/" nested directories as
found in Apache Hive. This is a multi-level, directory based partitioning
scheme. Data is partitioned by static values of a particular column in
the schema. Partition keys are represented in the form $key=$value in
directory names. Field order is ignored, as are missing or unrecognized
field names.
For example, given schema<year:int16, month:int8, day:int8>, a possible
path would be "/year=2009/month=11/day=15" (but the field order does not
need to match).
- "FilenamePartitioning": this scheme expects the partitions will have
filenames containing the field values separated by "_".
For example, given schema<year:int16, month:int8, day:int8>, a possible
partition filename "2009_11_part-0.parquet" would be parsed
to ("year"_ == 2009 and "month"_ == 11).
Parameters
----------
schema : pyarrow.Schema, default None
The schema that describes the partitions present in the file path.
If not specified, and `field_names` and/or `flavor` are specified,
the schema will be inferred from the file path (and a
PartitioningFactory is returned).
field_names : list of str, default None
A list of strings (field names). If specified, the schema's types are
inferred from the file paths (only valid for DirectoryPartitioning).
flavor : str, default None
The default is DirectoryPartitioning. Specify ``flavor="hive"`` for
a HivePartitioning, and ``flavor="filename"`` for a
FilenamePartitioning.
dictionaries : dict[str, Array]
If the type of any field of `schema` is a dictionary type, the
corresponding entry of `dictionaries` must be an array containing
every value which may be taken by the corresponding column or an
error will be raised in parsing. Alternatively, pass `infer` to have
Arrow discover the dictionary values, in which case a
PartitioningFactory is returned.
Returns
-------
Partitioning or PartitioningFactory
The partitioning scheme
Examples
--------
Specify the Schema for paths like "/2009/June":
>>> import pyarrow as pa
>>> import pyarrow.dataset as ds
>>> part = ds.partitioning(pa.schema([("year", pa.int16()),
... ("month", pa.string())]))
or let the types be inferred by only specifying the field names:
>>> part = ds.partitioning(field_names=["year", "month"])
For paths like "/2009/June", the year will be inferred as int32 while month
will be inferred as string.
Specify a Schema with dictionary encoding, providing dictionary values:
>>> part = ds.partitioning(
... pa.schema([
... ("year", pa.int16()),
... ("month", pa.dictionary(pa.int8(), pa.string()))
... ]),
... dictionaries={
... "month": pa.array(["January", "February", "March"]),
... })
Alternatively, specify a Schema with dictionary encoding, but have Arrow
infer the dictionary values:
>>> part = ds.partitioning(
... pa.schema([
... ("year", pa.int16()),
... ("month", pa.dictionary(pa.int8(), pa.string()))
... ]),
... dictionaries="infer")
Create a Hive scheme for a path like "/year=2009/month=11":
>>> part = ds.partitioning(
... pa.schema([("year", pa.int16()), ("month", pa.int8())]),
... flavor="hive")
A Hive scheme can also be discovered from the directory structure (and
types will be inferred):
>>> part = ds.partitioning(flavor="hive")
Nz.Cannot specify both 'schema' and 'field_names'�infer��schemaz$Expected list of field names, got {}zSFor the default directory flavor, need to specify a Schema or a list of field names�filenamezJFor the filename flavor, need to specify a Schema or a list of field names�hivez.Cannot specify 'field_names' for flavor 'hive'z$Expected Schema for 'schema', got {}zUnsupported flavor) �
ValueErrorr �discover�
isinstance�listr: �typer �pa�Schemar )rB � field_names�flavor� dictionariess r< � partitioningrO y sZ � �L �~� � ��&� �D�F� F� F��w�&�&�,�5�V�D�D�D�D�(�� �>�>� >�
�
$��+�t�,�,�
,�,�5�k�B�B�B� �:�A�A� �[�)�)�+� +�,� ,� ,� �4�5� 5�
5� ���� � ��&� �D�F� F� F��w�&�&�+�4�F�C�C�C�C�'�� �=�=� =�
�
$��+�t�,�,�
,�+�4�[�A�A�A� �:�A�A� �[�)�)�+� +�,� ,� ,� �4�5� 5�
5�
�6� � � � "� � M�N�N� N�
�
��&�"�)�,�,�
'� �7�*�*�+�4�F�C�C�C�C�'�� �=�=�=� �:�A�A� �V� � �&� &�'� '� '� $�,�.�.� .��-�.�.�.r> c �< � | �n�t | t � � rt | �� � } nrt | t � � rt | �� � } nLt | t t
f� � rn/t
d� t | � � � � � � �| S )z~
Validate input and return a Partitioning(Factory).
It passes None through if no partitioning scheme is defined.
N�rM )rL z4Expected Partitioning or PartitioningFactory, got {}) rG �strrO rH r r rE r: rI )�schemes r< �_ensure_partitioningrT s� � � �~� � �F�C� � � 0� �V�,�,�,��� �F�D� !� !� 0� �&�1�1�1��� �F�\�+>� ?� @� @� 0� ��O� �&��f� � �.�.�0� 0� 0� �Mr> c �� � t | t � � r| S | dk r)t st t � � �t � � S | dv rt
� � S | dk rt � � S | dk rt � � S | dk r)t st t � � �t � � S | dk rt � � S t d�
| � � � � �)N�parquet> �ipc�arrow�feather�csv�orc�jsonz format '{}' is not supported)rG r r7 rE r8 r+ r r
r r4 r6 r( r r: )�objs r< �_ensure_formatr^ , s� � ��#�z�"�"� E��
� � � � �!� +� �\�*�*� *� �"�"�"� � � � � ��� � � � � � �"�"�"� �� � � ��� � �� � � � '� �X�&�&� &� ��� � �� � � � � � ��7�>�>�s�C�C�D�D�Dr> c �* �� ddl m}m}m}m}m} �� |� � �n |�� � �t
�||f� � p$t
�|� � ot
�j |� � }�fd�| D � � } |r��� | � � D ]�}|j } | |j
k r�| |j k rt |j
� � �| |j k r't d� |j
� � � � �t# d� |j
� � � � ��| fS )aA
Treat a list of paths as files belonging to a single file system
If the file system is local then also validates that all paths
are referencing existing *files* otherwise any non-file paths will be
silently skipped (for example on a remote filesystem).
Parameters
----------
paths : list of path-like
Note that URIs are not allowed.
filesystem : FileSystem or str, optional
If an URI is passed, then its path component will act as a prefix for
the file paths.
Returns
-------
(FileSystem, list of str)
File system object and a list of normalized paths.
Raises
------
TypeError
If the passed filesystem has wrong type.
IOError
If the file system is local and a referenced path is not available or
not a file.
r )�LocalFileSystem�SubTreeFileSystem�_MockFileSystem�FileType�_ensure_filesystemNc �T �� g | ]$}�� t |� � � � ��%S � )�normalize_pathr )�.0�p�
filesystems �r< �
<listcomp>z,_ensure_multiple_sources.<locals>.<listcomp>s s/ �� � J� J� J�q�Z�
&�
&��q�'9�'9�
:�
:� J� J� Jr> z�Path {} points to a directory, but only file paths are supported. To construct a nested or union dataset pass a list of dataset objects instead.z�Path {} exists but its type is unknown (could be a special file such as a Unix socket or character device, or Windows NUL / CON / ...))�
pyarrow.fsr` ra rb rc rd rG �base_fs�
get_file_inforI �File�NotFound�FileNotFoundError�path� Directory�IsADirectoryErrorr: �IOError)
�pathsrj r` ra rb rc rd �is_local�info� file_types
` r< �_ensure_multiple_sourcesrz C s� �� �:� � � � � � � � � � � � � �
��$�_�&�&�
�
� (�'�
�3�3�
� �:�� � A�B�B� :� �J� 1� 2� 2�
9� �J�&�� 8� 8�
�
K� J� J� J�E� J� J� J�E�
� � �,�,�U�3�3� � �D� � �I��H�M�)�)���h� /�/�/�'�� �2�2�2��h� 0�0�0�'�9�9?��� �9J�9J�� � � �2�28�&���2C�2C�� � � �u� � r> c � � ddl m}m}m} || |� � \ }} |� | � � } |� | � � }|j |j k r || d�� � }n#|j |j k r| g}nt | � � �||fS )a�
Treat path as either a recursively traversable directory or a single file.
Parameters
----------
path : path-like
filesystem : FileSystem or str, optional
If an URI is passed, then its path component will act as a prefix for
the file paths.
Returns
-------
(FileSystem, list of str or fs.Selector)
File system object and either a single item list pointing to a file or
an fs.Selector object pointing to a directory.
Raises
------
TypeError
If the passed filesystem has wrong type.
FileNotFoundError
If the referenced file or directory doesn't exist.
r )rc � FileSelector� _resolve_filesystem_and_pathT)� recursive)
rl rc r| r} rg rn rI rs ro rq )rr rj rc r| r} � file_info�paths_or_selectors r< �_ensure_single_sourcer� � s� � �0 P�O�O�O�O�O�O�O�O�O� 4�3�D�*�E�E��J�� � $� $�T� *� *�D� �(�(��.�.�I� �~��+�+�+� (�L��� >� >� >��� � �8�=� (� (� !�F��� ��%�%�%� �(� (�(r> c � � ddl m}m} m}
t |pd� � }t |� � }t
| t t f� � rG| r1t
| d |
� � r|� |� � } n | |� � } | } n't | |� � \ } } nt | |� � \ } } t ||||�� � }
t | | ||
� � }|�
|� � S )z�
Create a FileSystemDataset which can be used to build a Dataset.
Parameters are documented in the dataset function.
Returns
-------
FileSystemDataset
r )r` rd �FileInforV N)rO �partition_base_dir�exclude_invalid_files�selector_ignore_prefixes)rl r` rd r� r^ rT rG rH �tuplerz r� r r �finish)�sourcerB rj rO r: r� r� r� r` rd r� �fsr� �options�factorys r< �_filesystem_datasetr� � s � � I�H�H�H�H�H�H�H�H�H�
�F� /�i�
0�
0�F�'� �5�5�L��&�4��-�(�(� J� � Q�j����H�5�5� Q��!�$�_�&�&��� (�'�
�3�3�� &� � �$<�V�Z�$P�$P� !�B�!�!� 5�f�j� I� I� �� �&�!�-� 3�!9� � � �G� '�r�+<�f�g�N�N�G� �>�>�&� !� !�!r> c � � t d� |� � � D � � � � rt d� � �t | |� � S )Nc 3 � K � | ]}|d uV � � d S �Nrf �rh �vs r< � <genexpr>z%_in_memory_dataset.<locals>.<genexpr>� �& � � � �
2�
2�Q�1�D�=�
2�
2�
2�
2�
2�
2r> z@For in-memory datasets, you cannot pass any additional arguments)�any�valuesrE r )r� rB �kwargss r< �_in_memory_datasetr� � sR � �
�
2�
2�&�-�-�/�/�
2�
2�
2�2�2� P�� N�P� P� P� �6�6� *� *�*r> c �> �� t d� |� � � D � � � � rt d� � ��� t j d� | D � � � � �| D ]"}t |dd � � rt d� � ��#�fd�| D � � } t
�| � � S )Nc 3 � K � | ]}|d uV � � d S r� rf r� s r< r� z!_union_dataset.<locals>.<genexpr>� r� r> zIWhen passing a list of Datasets, you cannot pass any additional argumentsc � � g | ] }|j ��
S rf rA )rh �childs r< rk z"_union_dataset.<locals>.<listcomp>� s � �"F�"F�"F�E�5�<�"F�"F�"Fr> �
_scan_optionsz�Creating an UnionDataset from filtered or projected Datasets is currently not supported. Union the unfiltered datasets and apply the filter to the resulting union.c �: �� g | ]}|� �� � ��S rf )�replace_schema)rh r� rB s �r< rk z"_union_dataset.<locals>.<listcomp> s' �� �C�C�C���$�$�V�,�,�C�C�Cr> )r� r� rE rJ �
unify_schemas�getattrr )�childrenrB r� r� s ` r< �_union_datasetr� � s� �� �
�
2�
2�&�-�-�/�/�
2�
2�
2�2�2�
��
�
�
�
�
�~��!�"F�"F�X�"F�"F�"F�G�G��� � �� �5�/�4� 0� 0� � �?�� �
� � D�C�C�C�(�C�C�C�H� ��� )� )�)r> c �| � ddl m}m} |�t � � }n$t |t � � st d� � �|� |� � }n ||� � }|� t | � � � � } t |t |� � �� � }t | |||�� � } | � |� � S )a�
Create a FileSystemDataset from a `_metadata` file created via
`pyarrow.parquet.write_metadata`.
Parameters
----------
metadata_path : path,
Path pointing to a single file parquet metadata file
schema : Schema, optional
Optionally provide the Schema for the Dataset, in which case it will
not be inferred from the source.
filesystem : FileSystem or URI string, default None
If a single path is given as source and filesystem is None, then the
filesystem will be inferred from the path.
If an URI string is passed, then a filesystem object is constructed
using the URI's optional path component as a directory prefix. See the
examples below.
Note that the URIs on Windows must follow 'file:///C:...' or
'file:/C:...' patterns.
format : ParquetFileFormat
An instance of a ParquetFileFormat if special options needs to be
passed.
partitioning : Partitioning, PartitioningFactory, str, list of str
The partitioning scheme specified with the ``partitioning()``
function. A flavor string can be used as shortcut, and with a list of
field names a DirectoryPartitioning will be inferred.
partition_base_dir : str, optional
For the purposes of applying the partitioning, paths will be
stripped of the partition_base_dir. Files not matching the
partition_base_dir prefix will be skipped for partitioning discovery.
The ignored files will still be part of the Dataset, but will not
have partition information.
Returns
-------
FileSystemDataset
The dataset corresponding to the given metadata
r )r` rd Nz+format argument must be a ParquetFileFormat)r� rO )r� ) rl r` rd r+ rG rE rg r r* rT r) r� )
�
metadata_pathrB rj r: rO r� r` rd r� r� s
r< �parquet_datasetr� s� � �P ?�>�>�>�>�>�>�>�
�~�"�$�$���
�� 1�
2�
2� H��F�G�G�G���$�_�&�&�
�
�'�'�
�3�3�
� �-�-�o�m�.L�.L�M�M�M�#�-�)�,�7�7�� � �G�
$��z�6�7�<� <� <�G� �>�>�&� !� !�!r> c � � � ddl m� t |||||||�� � }t | � � r
t | fi |��S t | t t f� � r�t � fd�| D � � � � r
t | fi |��S t d� | D � � � � r
t | fi |��S t d� | D � � � � r
t | fi |��S t d� | D � � � � } d� d � | D � � � � }
t d
� |
� � � � �t | t j t j f� � r
t | fi |��S t d � t% | � � j � � � � �) a�
Open a dataset.
Datasets provides functionality to efficiently work with tabular,
potentially larger than memory and multi-file dataset.
- A unified interface for different sources, like Parquet and Feather
- Discovery of sources (crawling directories, handle directory-based
partitioned datasets, basic schema normalization)
- Optimized reading with predicate pushdown (filtering rows), projection
(selecting columns), parallel reading or fine-grained managing of tasks.
Note that this is the high-level API, to have more control over the dataset
construction use the low-level API classes (FileSystemDataset,
FilesystemDatasetFactory, etc.)
Parameters
----------
source : path, list of paths, dataset, list of datasets, (list of) RecordBatch or Table, iterable of RecordBatch, RecordBatchReader, or URI
Path pointing to a single file:
Open a FileSystemDataset from a single file.
Path pointing to a directory:
The directory gets discovered recursively according to a
partitioning scheme if given.
List of file paths:
Create a FileSystemDataset from explicitly given files. The files
must be located on the same filesystem given by the filesystem
parameter.
Note that in contrary of construction from a single file, passing
URIs as paths is not allowed.
List of datasets:
A nested UnionDataset gets constructed, it allows arbitrary
composition of other datasets.
Note that additional keyword arguments are not allowed.
(List of) batches or tables, iterable of batches, or RecordBatchReader:
Create an InMemoryDataset. If an iterable or empty list is given,
a schema must also be given. If an iterable or RecordBatchReader
is given, the resulting dataset can only be scanned once; further
attempts will raise an error.
schema : Schema, optional
Optionally provide the Schema for the Dataset, in which case it will
not be inferred from the source.
format : FileFormat or str
Currently "parquet", "ipc"/"arrow"/"feather", "csv", "json", and "orc" are
supported. For Feather, only version 2 files are supported.
filesystem : FileSystem or URI string, default None
If a single path is given as source and filesystem is None, then the
filesystem will be inferred from the path.
If an URI string is passed, then a filesystem object is constructed
using the URI's optional path component as a directory prefix. See the
examples below.
Note that the URIs on Windows must follow 'file:///C:...' or
'file:/C:...' patterns.
partitioning : Partitioning, PartitioningFactory, str, list of str
The partitioning scheme specified with the ``partitioning()``
function. A flavor string can be used as shortcut, and with a list of
field names a DirectoryPartitioning will be inferred.
partition_base_dir : str, optional
For the purposes of applying the partitioning, paths will be
stripped of the partition_base_dir. Files not matching the
partition_base_dir prefix will be skipped for partitioning discovery.
The ignored files will still be part of the Dataset, but will not
have partition information.
exclude_invalid_files : bool, optional (default True)
If True, invalid files will be excluded (file format specific check).
This will incur IO for each files in a serial and single threaded
fashion. Disabling this feature will skip the IO, but unsupported
files may be present in the Dataset (resulting in an error at scan
time).
ignore_prefixes : list, optional
Files matching any of these prefixes will be ignored by the
discovery process. This is matched to the basename of a path.
By default this is ['.', '_'].
Note that discovery happens only if a directory is passed as source.
Returns
-------
dataset : Dataset
Either a FileSystemDataset or a UnionDataset depending on the source
parameter.
Examples
--------
Creating an example Table:
>>> import pyarrow as pa
>>> import pyarrow.parquet as pq
>>> table = pa.table({'year': [2020, 2022, 2021, 2022, 2019, 2021],
... 'n_legs': [2, 2, 4, 4, 5, 100],
... 'animal': ["Flamingo", "Parrot", "Dog", "Horse",
... "Brittle stars", "Centipede"]})
>>> pq.write_table(table, "file.parquet")
Opening a single file:
>>> import pyarrow.dataset as ds
>>> dataset = ds.dataset("file.parquet", format="parquet")
>>> dataset.to_table()
pyarrow.Table
year: int64
n_legs: int64
animal: string
----
year: [[2020,2022,2021,2022,2019,2021]]
n_legs: [[2,2,4,4,5,100]]
animal: [["Flamingo","Parrot","Dog","Horse","Brittle stars","Centipede"]]
Opening a single file with an explicit schema:
>>> myschema = pa.schema([
... ('n_legs', pa.int64()),
... ('animal', pa.string())])
>>> dataset = ds.dataset("file.parquet", schema=myschema, format="parquet")
>>> dataset.to_table()
pyarrow.Table
n_legs: int64
animal: string
----
n_legs: [[2,2,4,4,5,100]]
animal: [["Flamingo","Parrot","Dog","Horse","Brittle stars","Centipede"]]
Opening a dataset for a single directory:
>>> ds.write_dataset(table, "partitioned_dataset", format="parquet",
... partitioning=['year'])
>>> dataset = ds.dataset("partitioned_dataset", format="parquet")
>>> dataset.to_table()
pyarrow.Table
n_legs: int64
animal: string
----
n_legs: [[5],[2],[4,100],[2,4]]
animal: [["Brittle stars"],["Flamingo"],...["Parrot","Horse"]]
For a single directory from a S3 bucket:
>>> ds.dataset("s3://mybucket/nyc-taxi/",
... format="parquet") # doctest: +SKIP
Opening a dataset from a list of relatives local paths:
>>> dataset = ds.dataset([
... "partitioned_dataset/2019/part-0.parquet",
... "partitioned_dataset/2020/part-0.parquet",
... "partitioned_dataset/2021/part-0.parquet",
... ], format='parquet')
>>> dataset.to_table()
pyarrow.Table
n_legs: int64
animal: string
----
n_legs: [[5],[2],[4,100]]
animal: [["Brittle stars"],["Flamingo"],["Dog","Centipede"]]
With filesystem provided:
>>> paths = [
... 'part0/data.parquet',
... 'part1/data.parquet',
... 'part3/data.parquet',
... ]
>>> ds.dataset(paths, filesystem='file:///directory/prefix,
... format='parquet') # doctest: +SKIP
Which is equivalent with:
>>> fs = SubTreeFileSystem("/directory/prefix",
... LocalFileSystem()) # doctest: +SKIP
>>> ds.dataset(paths, filesystem=fs, format='parquet') # doctest: +SKIP
With a remote filesystem URI:
>>> paths = [
... 'nested/directory/part0/data.parquet',
... 'nested/directory/part1/data.parquet',
... 'nested/directory/part3/data.parquet',
... ]
>>> ds.dataset(paths, filesystem='s3://bucket/',
... format='parquet') # doctest: +SKIP
Similarly to the local example, the directory prefix may be included in the
filesystem URI:
>>> ds.dataset(paths, filesystem='s3://bucket/nested/directory',
... format='parquet') # doctest: +SKIP
Construction of a nested dataset:
>>> ds.dataset([
... dataset("s3://old-taxi-data", format="parquet"),
... dataset("local/path/to/data", format="ipc")
... ]) # doctest: +SKIP
r )r� )rB rj rO r: r� r� r� c 3 �V �K � | ]#}t |� � pt |�� � V � �$d S r� )r rG )rh �elemr� s �r< r� zdataset.<locals>.<genexpr> s: �� � � �T�T�T�}�T�"�"�@�j��x�&@�&@�T�T�T�T�T�Tr> c 3 �@ K � | ]}t |t � � V � �d S r� )rG r
�rh r� s r< r� zdataset.<locals>.<genexpr> s, � � � �>�>�t��D�'�*�*�>�>�>�>�>�>r> c 3 �b K � | ]*}t |t j t j f� � V � �+d S r� )rG rJ � RecordBatch�Tabler� s r< r� zdataset.<locals>.<genexpr> sI � � � � %� %�� �D�2�>�2�8�"<�=�=� %� %� %� %� %� %r> c 3 �> K � | ]}t |� � j V � �d S r� )rI �__name__r� s r< r� zdataset.<locals>.<genexpr>$ s+ � � � � F� F�t�t�D�z�z� 2� F� F� F� F� F� Fr> z, c 3 �@ K � | ]}d � |� � V � �dS )z{}N)r: )rh �ts r< r� zdataset.<locals>.<genexpr>% s, � � � �"H�"H�a�4�;�;�q�>�>�"H�"H�"H�"H�"H�"Hr> z�Expected a list of path-like or dataset objects, or a list of batches or tables. The given list contains the following types: {}z\Expected a path-like, list of path-likes or a list of Datasets instead of the given type: {})rl r� �dictr r� rG r� rH �allr� r� �set�join� TypeErrorr: rJ r� r� rI r� ) r� rB r: rj rO r� r� �ignore_prefixesr� � unique_types�
type_namesr� s @r< �datasetr� H s� �� �J $�#�#�#�#�#�
�� �!��-� 3�!0�� � �F� �V� � �
�"�6�4�4�V�4�4�4� �F�U�D�M� *� *�
� �T�T�T�T�V�T�T�T� T� T� �&�v�8�8��8�8� 8�
�>�>�v�>�>�>�
>�
>� �!�&�3�3�F�3�3� 3�
� %� %� #�%� %� %� %� %�
�%�f�7�7��7�7� 7� � F� F�v� F� F� F�F�F�L� ���"H�"H�<�"H�"H�"H�H�H�J�� � "�F�:� .� .�� �
�
�F�R�^�R�X� 6� 7� 7�
�!�&�3�3�F�3�3�3��
,�,2�F�4��<�<�3H�,I�,I�
�
�
r> c �� �� t | t � � rt d� � �t | t � � r|rt d� � �t | t t
f� � r0t
t j �fd�| D � � � � |�� � } n%| �#t
t j g � � |�� � } t | t � � st d� � �| S )NzhA PartitioningFactory cannot be used. Did you call the partitioning function without supplying a schema?zKProviding a partitioning_flavor with a Partitioning object is not supportedc �: �� g | ]}�� |� � ��S rf )r' )rh �frB s �r< rk z._ensure_write_partitioning.<locals>.<listcomp>C s# �� � <� <� <�!�f�l�l�1�o�o� <� <� <r> �rB rM rQ zDpartitioning must be a Partitioning object or a list of column names) rG r rE r r� rH rO rJ rB )�partrB rM s ` r< �_ensure_write_partitioningr� 4 s �� ��$�+�,�,� 8�� 7� 8� 8� 8� �$� �%�%�
:�&�
:��
5�
�
�
�
�D�5�$�-� (� (� :� ��9� <� <� <� <�t� <� <� <�=�=��
�
�
���
� ��B�I�b�M�M�&�9�9�9�� �d�L� )� )�
��
%�
�
�
�
�Kr> �error)�basename_templater: rO �partitioning_flavorrB rj � file_options� use_threads�max_partitions�max_open_files�max_rows_per_file�min_rows_per_group�max_rows_per_group� file_visitor�existing_data_behavior�
create_dirc �v � ddl m} t | t t f� � r!|p | d j }t
| |�� � } n�t | t j t j f� � r|p| j }t
| |�� � } n�t | t j
j � � s t | d� � st | � � rt j | |�� � } d}n+t | t t f� � st# d� � �|� t | t$ � � r| j }nt) |� � }|�|� � � }||j k r#t- d� ||� � � � �|�
d|j z }|
�d }
| �d } | �d} |�d
}|
�d}
t | t � � r| j }n| j }t3 |||� � � } |||� � \ }}t | t � � r| � | � � � }n|�t# d
� � �| }t7 |||||||
||| | |
||� � dS )a�
Write a dataset to a given format and partitioning.
Parameters
----------
data : Dataset, Table/RecordBatch, RecordBatchReader, list of Table/RecordBatch, or iterable of RecordBatch
The data to write. This can be a Dataset instance or
in-memory Arrow data. If an iterable is given, the schema must
also be given.
base_dir : str
The root directory where to write the dataset.
basename_template : str, optional
A template string used to generate basenames of written data files.
The token '{i}' will be replaced with an automatically incremented
integer. If not specified, it defaults to
"part-{i}." + format.default_extname
format : FileFormat or str
The format in which to write the dataset. Currently supported:
"parquet", "ipc"/"arrow"/"feather", and "csv". If a FileSystemDataset
is being written and `format` is not specified, it defaults to the
same format as the specified FileSystemDataset. When writing a
Table or RecordBatch, this keyword is required.
partitioning : Partitioning or list[str], optional
The partitioning scheme specified with the ``partitioning()``
function or a list of field names. When providing a list of
field names, you can use ``partitioning_flavor`` to drive which
partitioning type should be used.
partitioning_flavor : str, optional
One of the partitioning flavors supported by
``pyarrow.dataset.partitioning``. If omitted will use the
default of ``partitioning()`` which is directory partitioning.
schema : Schema, optional
filesystem : FileSystem, optional
file_options : pyarrow.dataset.FileWriteOptions, optional
FileFormat specific write options, created using the
``FileFormat.make_write_options()`` function.
use_threads : bool, default True
Write files in parallel. If enabled, then maximum parallelism will be
used determined by the number of available CPU cores.
max_partitions : int, default 1024
Maximum number of partitions any batch may be written into.
max_open_files : int, default 1024
If greater than 0 then this will limit the maximum number of
files that can be left open. If an attempt is made to open
too many files then the least recently used file will be closed.
If this setting is set too low you may end up fragmenting your
data into many small files.
max_rows_per_file : int, default 0
Maximum number of rows per file. If greater than 0 then this will
limit how many rows are placed in any single file. Otherwise there
will be no limit and one file will be created in each output
directory unless files need to be closed to respect max_open_files
min_rows_per_group : int, default 0
Minimum number of rows per group. When the value is greater than 0,
the dataset writer will batch incoming data and only write the row
groups to the disk when sufficient rows have accumulated.
max_rows_per_group : int, default 1024 * 1024
Maximum number of rows per group. If the value is greater than 0,
then the dataset writer may split up large incoming batches into
multiple row groups. If this value is set, then min_rows_per_group
should also be set. Otherwise it could end up with very small row
groups.
file_visitor : function
If set, this function will be called with a WrittenFile instance
for each file created during the call. This object will have both
a path attribute and a metadata attribute.
The path attribute will be a string containing the path to
the created file.
The metadata attribute will be the parquet metadata of the file.
This metadata will have the file path attribute set and can be used
to build a _metadata file. The metadata attribute will be None if
the format is not parquet.
Example visitor which simple collects the filenames created::
visited_paths = []
def file_visitor(written_file):
visited_paths.append(written_file.path)
existing_data_behavior : 'error' | 'overwrite_or_ignore' | 'delete_matching'
Controls how the dataset will handle data that already exists in
the destination. The default behavior ('error') is to raise an error
if any data exists in the destination.
'overwrite_or_ignore' will ignore any existing data and will
overwrite files with the same name as an output file. Other
existing files will be ignored. This behavior, in combination
with a unique basename_template for each write, will allow for
an append workflow.
'delete_matching' is useful when you are writing a partitioned
dataset. The first time each partition directory is encountered
the entire directory will be deleted. This allows you to overwrite
old partitions completely.
create_dir : bool, default True
If False, directories will not be created. This can be useful for
filesystems that do not require directories.
r )r} rA �__arrow_c_stream__Nz�Only Dataset, Scanner, Table/RecordBatch, RecordBatchReader, a list of Tables/RecordBatches, or iterable of batches are supported.zTSupplied FileWriteOptions have format {}, which doesn't match supplied FileFormat {}z part-{i}.i i r� )r� z.Cannot specify a schema when writing a Scanner) rl r} rG rH r� rB r rJ r� r� rW �RecordBatchReader�hasattrr r � from_batchesr
rE r r: r^ �make_write_optionsr� �default_extname�projected_schemar� �scannerr# )�data�base_dirr� r: rO r� rB rj r� r� r� r� r� r� r� r� r� r� r} �partitioning_schemar� s r< �
write_datasetr� R s� � �Z 8�7�7�7�7�7��$��u�
�&�&�
��)�4��7�>�� �t�F�3�3�3��� �D�2�>�2�8�4� 5� 5�
��&�4�;�� �t�F�3�3�3����4���1�2�2�
� �4�-� .� .�
� �� � �
�
�#�D��8�8�8�����
��w�� 0�
1�
1�
��
�
�
�
� �~�*�T�+<�=�=�~����� ��'�'���� �0�0�2�2� �
� �$�$�$�� E�EK�V� "�L�F2� F2�3� 3� 3� � � '�&�*@� @��� � ��� � ��� � ���!� $���!� ��
�$�� � � *� "� 3��� "�k��-�l�5H�5H�J� J� J�L� 8�7��*�M�M��J���$�� � � ��,�,�;�,�7�7��� � � � M�N�N� N��� ���,�j�,��n�l�4J��)�� .�
� � � � � r> )NNNNr� )NNNNNNN)NNNNN)M�__doc__�pyarrowrJ � pyarrow.utilr r r �pyarrow._datasetr r r r r
r r r
r r r r r r r r r r r r r r r r r r r r! r" �_get_partition_keysr# r5 �excrR �pyarrow.computer% r& r' r4 r6 �pyarrow._dataset_orcr( r7 r8 �pyarrow._dataset_parquetr) r* r+ r, r- r. r/ r0 �#pyarrow._dataset_parquet_encryptionr1 r2 r= rO rT r^ rz r� r� r� r� r� r� r� r� rf r> r< �<module>r� s� ��$ L� K� � � � � E� E� E� E� E� E� E� E� E� E�%� � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � ��B � � � �
�+�X�S�S�QT�X�X�X�X�X� � � ���������� 6� 5� 5� 5� 5� 5� 5� 5� 5� 5� ��� �
�2�2�2�2�2�2��N�N��� � � ��D� ���� � ��
�
� � � � � � � � � � � � � � � � � � � � � ����� � � ��D� ���� �� � � � � � � � �� � � � ��D� ���� � � � 8<� "�\/� \/� \/� \/�~� � �(E� E� E�.I � I � I � I �X,)� ,)� ,)� ,)�^ 9=�26�GK�15�("� ("� ("� ("�V+� +� +� +�*� *� *� *�2 IM�:>�<"� <"� <"� <"�~ :>�26�8<�i
� i
� i
� i
�X� � �< 8<�D� #��d� !��$�!%�d�$(�T�%)��)0�T�
}� }� }� }� }� }� }sN �AA �A8�A3�3A8�
B �B�B�#B: �:C�C�C �C�C