� J��g�����ddlZddlmZddlmZddlZddlmZddlZddl m Z ddl Z ddl Z ddl Z ddlZ ddlZn #e$rdZYnwxYwddlZddlmZmZmZiaiaiad�Zd�Zd �Zd �Zd �Zd �Z d.d �Z!d�Z"d�Z#d�Z$d�Z%d�Z&d�Z'd�Z(d�Z)d�Z*d�Z+d.d�Z, d/d�Z-d�Z.d0d�Z/d�Z0 d1d �Z1hd!�Z2d"�Z3d#�Z4d$�Z5d.d%�Z6 d.d&�Z7d'�Z8d(�Z9d)�Z:d*�Z;d+�Z<d,�Z=d-�Z>dS)2�N)�Sequence)�futures)�deepcopy)� zip_longest)� _pandas_api� frombytes�is_threading_enabledc �"�t�s�t�itjjd�tjjd�tjjd�tjjd�tjjd�tjj d�tjj d�tjj d�tjj d �tjj d �tjjd �tjjd �tjjd �tjjd�tjjd�tjjd�tjjd�tjjdtjjdtjjdi���tS)N�empty�bool�int8�int16�int32�int64�uint8�uint16�uint32�uint64�float16�float32�float64�date�time�bytes�unicode)�_logical_type_map�update�pa�lib�Type_NA� Type_BOOL� Type_INT8� Type_INT16� Type_INT32� Type_INT64� Type_UINT8� Type_UINT16� Type_UINT32� Type_UINT64�Type_HALF_FLOAT� Type_FLOAT� Type_DOUBLE� Type_DATE32� Type_DATE64� Type_TIME32� Type_TIME64� Type_BINARY�Type_FIXED_SIZE_BINARY� Type_STRING���e/home/asafur/pinokio/api/open-webui.git/app/env/lib/python3.11/site-packages/pyarrow/pandas_compat.py�get_logical_type_mapr7.sw�� � �� � �" � �F�N�G�" � �F� �f�" � �F� �f�" � �F� �w� " � �F� �w� " � �F� �w� " � �F� �w�" � �F� ��" � �F� ��" � �F� ��" � �F� "�I�" � �F� �y�" � �F� � �" � �F� ��" � �F� ��" � �F� ��!" �" �F� ��#" �$ �F� �� �F� )�7� �F� � �)" �" � � � �, �r5c���t��} ||jS#t$r�t|tjj��rYdSt|tjj��r)d�t|j ����cYSt|tjj ��r |j �dndcYSt|tjj ��rYdSYdSwxYw)N� categoricalzlist[{}]� datetimetz�datetime�decimal�object)r7�id�KeyError� isinstancerr�DictionaryType�ListType�format�get_logical_type� value_type� TimestampType�tz�Decimal128Type)� arrow_type�logical_type_maps r6rDrDKs���+�-�-�� �� � �.�.�� � � � � �j�"�&�"7� 8� 8� � �=�=� � �B�F�O� 4� 4� ��$�$�%5�j�6K�%L�%L�M�M� M� M� M� � �B�F�$8� 9� 9� �#-�=�#<�<�<�*� L� L� L� � �B�F�$9� :� :� ��9�9��x�x� ���s"� �)C#� AC#�*C#�=C#�"C#c��ts�t�tjdtjdtjdtjdtjdtjdtj dtj dtj d tj d tj d d d tjdtjdi��tS)Nr r rrrrrrrrr� datetime64[D]r�stringr)�_numpy_logical_type_mapr�np�bool_r rrrrrrrrr�str_�bytes_r4r5r6�get_numpy_logical_type_maprS\s��� "� ��&�&� �H�f� �G�V� �H�g� �H�g� �H�g� �H�g� �I�x� �I�x� �I�x� �J� � �J� � �V� �G�X� �I�w�( � � � � #�"r5c�L�t��} ||jjS#t$rwt |jd��rYdSt |j���d��rt |j��cYStj|��}|dkrYdS|cYSwxYw)NrGr:� datetime64rMr) rS�dtype�typer?�hasattr�str� startswithr� infer_dtype)�pandas_collection�numpy_logical_type_map�results r6�get_logical_type_from_numpyr_rs���7�9�9�� �%�&7�&=�&B�C�C�� � � � � �$�*�D� 1� 1� ��<�<� � �&� '� '� 2� 2�<� @� @� 0��(�.�/�/� /� /� /��(�):�;�;�� �X� � ��9�9�� � � � ���s!�"�B#�;B#�B#�B#�"B#c�h�|j}t|��dkrKt|d|��}|�J�t|j��|jd�}t|jj��}nJt|d��r)dtj � |j ��i}d}nd}t|��}||fS)N�category�cat)�num_categories�orderedrG�timezone�datetime64[ns]) rVrY�getattr�len� categoriesrd�codesrXrr�tzinfo_to_stringrG)�columnrV�cats�metadata�physical_dtypes r6�get_extension_dtype_inforp�s��� �L�E� �5�z�z�Z����v�u�f�-�-������!�$�/�2�2��|� � ���T�Z�-�.�.��� ��� � �$���� 7� 7��� A� A�B��)������U���� �8� #�#r5c���t|��}t|��\}}|dkr|j|jd�}d}|�st |t ��rt j|��sJt |t��s5td� |t|��j �����t |t��s$Jtt|�������|||||d�S)a�Construct the metadata for a given column Parameters ---------- column : pandas.Series or pandas.Index name : str arrow_type : pyarrow.DataType field_name : str Equivalent to `name` when `column` is a `Series`, otherwise if `column` is a pandas Index then `field_name` will not be the same as `name`. This is the name of the field in the arrow Table's schema. Returns ------- dict r<)� precision�scaler=Nz6Column name must be a string. Got column {} of type {}��name� field_name� pandas_type� numpy_typern) rDrprrrsr@�floatrO�isnanrY� TypeErrorrCrW�__name__)rlrurIrv� logical_type� string_dtype�extra_metadatas r6�get_column_metadatar��s ��"$�J�/�/�L�#;�F�#C�#C� �L�.��y� � �#�-��%� � �� � � ���D�%�(�(� �-/�X�d�^�^� ��4��%�%� �� D� K� K��d�4�j�j�)� � � � � � �j�#� &� &�=�=��D��,<�,<�(=�(=�=�=� &�� �#�"�"�  � �r5c �|�|� d�|D��}td�|D����}t|��} |d| |z �} || |z d�} g} t|||| ��D]/\} }}}t| |||���}| �|���0g}|du�rSg}t|| |��D]�\}}}t |t ��r�|j�4t |jt��s|�|j��t|t|j��||���}|�|����t|��dkr tj d|�d�td � ��g}t|j d |j g��}t|j d |j jg��}t||��D]*\}}t||��}|�|���+ngx}x}}d tj||| |zdt"jd�t&jd����d��iS)a�Returns a dictionary containing enough metadata to reconstruct a pandas DataFrame as an Arrow Table, including index columns. Parameters ---------- columns_to_convert : list[pd.Series] df : pandas.DataFrame column_names : list[str | None] column_field_names: list[str] index_levels : List[pd.Index] index_descriptors : List[Dict] preserve_index : bool types : List[pyarrow.DataType] Returns ------- dict Nc�,�g|]}t|����Sr4)rY)�.0rus r6� <listcomp>z&construct_metadata.<locals>.<listcomp>�s��A�A�A�D�c�$�i�i�A�A�Ar5c�<�g|]}t|t���|��Sr4)r@�dict)r��descrs r6r�z&construct_metadata.<locals>.<listcomp>�s=��'G�'G�'G��.8���.E�.E�'G�u�'G�'G�'Gr5)rurIrvFrz&The DataFrame has non-str index name `z@` which will be converted to string and not roundtrip correctly.��� stacklevel�levels�namesspandas�pyarrow)�library�version)� index_columns�column_indexes�columns�creator�pandas_version�utf8)rh�zipr��appendr@r�rurY�_column_name_to_strings�warnings�warn� UserWarningrgr��_get_simple_index_descriptor�json�dumpsr� __version__rr��encode)�columns_to_convert�df� column_names� index_levels�index_descriptors�preserve_index�types�column_field_names�num_serialized_index_levels�ntypes�df_types� index_types�column_metadata�colrurvrIrn�index_column_metadata�non_str_index_names�level� descriptorr�r�r�s r6�construct_metadatar��s���*�!�B�A�L�A�A�A��"%�'G�'G�:K�'G�'G�'G�#H�#H����Z�Z�F��:�f�:�:�:�;�H���!<�<�=�=�>�K��O�-0�1C�\�1C�X�.O�.O�)�)�)��T�:�z�&�s��2<�2<�>�>�>�� ���x�(�(�(�(����U�"�"� ��-0��{�1B�.D�.D� 3� 3� )�E�:�z��*�d�+�+� ���z�%�j���S�.I�.I�%�#�*�*�5�:�6�6�6�*��,�U�Z�8�8�%�%� ���H� "� (� (�� 2� 2� 2� 2� �"� #� #�a� '� '� �M�0�9L�0�0�0���  +� +� +� +� �����X�� �|�<�<���� �G�b�j�o�->�?�?���v�u�-�-� ,� ,�K�E�4�3�E�4�@�@�H� � !� !�(� +� +� +� +� ,�FH�G��G�1�N� �4�:�.�,�&�)>�>�$��>���*�1�  �  �  �  � �6�&�>�>� � r5c��t|��\}}t|��}d|vrtjdtd���|dkr|rJ�ddi}|||||d�S) N�mixedzlThe DataFrame has column names of mixed type. They will be converted to strings and not roundtrip correctly.r�r�r�encoding�UTF-8rt)rpr_r�r�r�)r�rur~rrws r6r�r�%s���#;�E�#B�#B� �L�.�-�e�4�4�K��+���� � @� �A� '� '� '� '��i���!�!�!�!�$�g�.����"�"�"�  � �r5c���t|t��r|St|t��r|�d��St|t��r/tt t t |������St|t��rtd���|�)t|t��rtj |��r|St|��S)a!Convert a column name (or level) to either a string or a recursive collection of strings. Parameters ---------- name : str or tuple Returns ------- value : str or tuple Examples -------- >>> name = 'foo' >>> _column_name_to_strings(name) 'foo' >>> name = ('foo', 'bar') >>> _column_name_to_strings(name) "('foo', 'bar')" >>> import pandas as pd >>> name = (1, pd.Timestamp('2017-02-01 00:00:00')) >>> _column_name_to_strings(name) "('1', '2017-02-01 00:00:00')" r�z%Unsupported type for MultiIndex level) r@rYr�decode�tuple�mapr�rr{ryrOrz�rus r6r�r�9s���2�$���� �� � �D�%� � ���{�{�6�"�"�"� �D�%� � ���5��4�d�;�;�<�<�=�=�=� �D�(� #� #���?�@�@�@� ��*�T�5�1�1��b�h�t�n�n��� � �t�9�9�r5c�t�|j�|j|vrt|j��Sd�|��S)z�Return the name of an index level or a default name if `index.name` is None or is already a column name. Parameters ---------- index : pandas.Index i : int Returns ------- name : str Nz__index_level_{:d}__)rur�rC)�index�ir�s r6�_index_level_namer�`s<�� �z��%�*�L�"@�"@�&�u�z�2�2�2�%�,�,�Q�/�/�/r5c���t|||��}|jjs4td�t |j�������|�t |||��Sg}g}|durt|j��ng}g}g}|D]�} || } t| ��} tj | ��r"td�| �����|� | ��|� d��|� | ��|� t| ������g} g} t|��D]�\} }t!|| |��} t#|tjj��r|�t)|��}nA|� |��|� d��| }| � | ��| � |����|| z}|||| | |||fS)Nz Duplicate column names found: {}F�-Sparse pandas data (column {}) not supported.)�_resolve_columns_of_interestr�� is_unique� ValueErrorrC�list�$_get_columns_to_convert_given_schema�_get_index_level_valuesr�r�r� is_sparser{r�rY� enumerater�r@�pd� RangeIndex�_get_range_index_descriptor)r��schemar�r�r�r�r�r��convert_fieldsrur�r��index_column_namesr�� index_levelr�� all_namess r6�_get_columns_to_convertr�ssH��*�2�v�w�?�?�G� �:� � �� .� 5� 5�d�2�:�6F�6F� G� G� � � ���3�B���O�O�O��L���.<�5�-H�-H����)�)�)� �� ���N�� -� -����h��&�t�,�,�� � �� %� %� N��?�F�F�t�L�L�N�N� N� �!�!�#�&�&�&����d�#�#�#����D�!�!�!��!�!�#�d�)�)�,�,�,�,�����#�L�1�1� (� (���;� ��a��>�>�� �{�K�N�$=� >� >� ,��&�/� �<�<�E�E� � %� %�k� 2� 2� 2� � !� !�$� '� '� '��E� � %� %�d� +� +� +�� � ��'�'�'�'�"�%7�7�I� �|�%7�9K� �|�-?�� Q�Qr5c��g}g}g}g}g}g}|jD�]�} || } d} n�#t$r� t|| ��} n7#ttf$r#td�| �����wxYw|dur"t d�| �����|�At | tjj ��r"t d�| �����d} YnwxYwtj | ��r"td�| �����|� | ��} |� | ��|� | ��|� | ��| r?|� | ��|� | ��|� | �����||z} | |||||||fS)z� Specialized version of _get_columns_to_convert in case a Schema is specified. In that case, the Schema is used as the single point of truth for the table structure (types, which columns are included, order of columns, ...). FzNname '{}' present in the specified schema is not found in the columns or indexzlname '{}' present in the specified schema corresponds to the index, but 'preserve_index=False' was specifiedNaname '{}' is present in the schema, but it is a RangeIndex which will not be converted as a column in the Table, but saved as metadata-only not in columns. Specify 'preserve_index=True' to force it being added as a column, or remove it from the specified schemaTr�)r�r?�_get_index_level� IndexErrorrCr�r@rr�r�r�r{�fieldr�)r�r�r�r�r�r�r�r�r�rur��is_indexr�r�s r6r�r��s6���L����N������L�� �(%�(%�� ��T�(�C��H�H��� � � � <�&�r�4�0�0�����j�)� <� <� <��.�.4�f�T�l�l�<�<�<� <���� ��&�&� � � &��t� � �.�.�.�!�(��s�K�N�$=�>�>�)� �'� (.�v�d�|�|� 5�5�5��H�H�H�- ����0 � �� %� %� N��?�F�F�t�L�L�N�N� N�� � �T�"�"���!�!�#�&�&�&����e�$�$�$����D�!�!�!� � %� � %� %�d� +� +� +� � $� $�T� *� *� *� � � �� $� $� $���1�1�I� �|�\�3E� �|�-?�� Q�Qs$� #� C$�?�C$�4A3�3A.C$�#C$c��|}||jjvr3t|��r$t|t d��d���}|j�|��S)z_ Get the index level of a DataFrame given 'name' (column name in an arrow Schema). �__index_level_�����)r�r��_is_generated_index_name�intrh�get_level_values)r�ru�keys r6r�r��s^�� �C� �2�8�>�!�!�&>�t�&D�&D�!��$�s�+�,�,�R�/�0�1�1�� �8� $� $�S� )� )�)r5c�n� tj|��|S#t$rt|��cYSwxYw�N)r�r�r{rYr�s r6� _level_namer�sI��� � �4����� �� �����4�y�y�������s ��4�4c��dt|j��tj|d��tj|d��tj|d��d�S)N�range�start�stop�step)�kindrur�r�r�)r�rur�get_rangeindex_attribute)r�s r6r�r� sS����E�J�'�'��5�e�W�E�E��4�U�F�C�C��4�U�F�C�C�  � �r5c�x��tt�d�g����}�fd�t|��D��S)Nr�c�:��g|]}��|����Sr4)r�)r�r�r�s �r6r�z+_get_index_level_values.<locals>.<listcomp>s'��� 8� 8� 8�!�E� "� "�1� %� %� 8� 8� 8r5)rhrgr�)r��ns` r6r�r�s>��� �G�E�8�e�W� -� -�.�.�A� 8� 8� 8� 8�u�Q�x�x� 8� 8� 8�8r5c�r��|�|�td���|�|j}n|��fd�|D��}n�j}|S)NzJSchema and columns arguments are mutually exclusive, pass only one of themc�&��g|] }|�jv� |��Sr4)r�)r��cr�s �r6r�z0_resolve_columns_of_interest.<locals>.<listcomp> s ���9�9�9���b�j���1���r5)r�r�r�)r�r�r�s` r6r�r�sa��� ��g�1��<�=�=� =� � ��,��� � �9�9�9�9�g�9�9�9����*�� �Nr5c ��t|d||��\}}}}}}} }g} | D�]} | j} tj| ��rt j| d���j} n�tj| ��rZt| tj j ��r| � d��n | dd�}t j|d���j} nVt| | j d��\} } tj�| | ��} | �t j| d���j} | �| ����t#| |||||| |���}|| |fS)NT)� from_pandasr�r�)r��valuesr�is_categoricalr�arrayrW�is_extension_array_dtyper@r��Series�head�get_datetimetz_typerVr�_ndarray_to_arrow_typer�r�)r�r�r�r�r�r��_r�r�r�r�r�r��type_r rns r6�dataframe_to_typesr�'st�� "�"�d�N�G� D� D��Y�������� �E� � � ����� � %�f� -� -� ;��H�Q�D�1�1�1�6�E�E� � 1�&� 9� 9� ;�!+��;�>�(�"*�"*�5�A�F�F�1�I�I�I�/0��!��u� ��H�U��5�5�5�:�E�E�/�����F�F�M�F�E��F�1�1�&�%�@�@�E��}�����5�5�5�:�� � � �U�����!��B� �m�=N���2D����H� �e�X� %�%r5�Tc ���"�t||||��\}}}} } } } } |�Ht|��t|j��}}||dzkr|dkrtj��}nd}t ��sd}�fd��"d�}|dkr�"fd�t | | ��D��}n�g}tj|��5}t | | ��D]_\}}||j ��r |� �"||�����5|� |� �"||�����` ddd��n #1swxYwYt|��D]6\}}t|tj��r|���||<�7d�|D��}|�Tg}t ||��D]-\}}|� tj||�����.tj|��}t%| ||| | |||���}|jrt)|j��n t+��}|�|��|�|��}d}t|��dkrn | dd }|d krH| dd }| dd } | dd }!tt1|| |!����}n#t2$rYnwxYw|||fS)N�dr�c���|�d}d}n|j}|j} tj||d����}n_#tjtjtjf$r6}|xjd�|j |j ��fz c_|�d}~wwxYw|s@|j dkr5td�t|��|j �����|S)NT)rWr��safez0Conversion failed for column {!s} with type {!s}rz>Field {} was non-nullable but pandas column had {} null values)�nullablerWrr�� ArrowInvalid�ArrowNotImplementedError�ArrowTypeError�argsrCrurV� null_countr�rY)r�r��field_nullabler�r^�ers �r6�convert_columnz+dataframe_to_arrays.<locals>.convert_columncs���� �=�!�N��E�E�"�^�N��J�E� ��X�c��4�d�K�K�K�F�F�����+��!�#� � � � �F�F�I���s�x���3�3�6� 6�F�F��G�����  ���� � M�&�"3�a�"7�"7��2�28�&��U���9?�9J�3L�3L�M�M� M�� s�1�&B �1B�B c��t|tj��o/|jjo#t |jjtj��Sr�) r@rO�ndarray�flags� contiguous� issubclassrVrW�integer)�arrs r6�_can_definitely_zero_copyz6dataframe_to_arrays.<locals>._can_definitely_zero_copyys;���3�� �+�+�7�� �$�7��3�9�>�2�:�6�6� 8r5c�.��g|]\}}�||����Sr4r4)r�r��fr s �r6r�z'dataframe_to_arrays.<locals>.<listcomp>s?���G�G�G��a��!�.��A�&�&�G�G�Gr5c��g|] }|j�� Sr4�rW)r��xs r6r�z'dataframe_to_arrays.<locals>.<listcomp>�s�� $� $� $��Q�V� $� $� $r5r�rr�r�r�r�r�)r�rhr�r� cpu_countr r�r�ThreadPoolExecutorr�r��submitr�r@�Futurer^r�r�r�rnrr�r� with_metadatar�r�)#r�r�r��nthreadsr�rr�r�r�r�r�r�r�r��nrows�ncolsr�arrays�executorr�rr�� maybe_futr��fieldsrur��pandas_metadatarn�n_rowsr�r�r�r�r s# ` @r6�dataframe_to_arraysr'Js�����/�r�6�>�/6�8�8��Y�������� ���2�w�w��B�J���u�� �5�3�;� � �5�1�9�9��|�~�~�H�H��H� � !� !���������,8�8�8� �1�}�}�G�G�G�G�!�"4�n�E�E�G�G�G����� � '�� 1� 1� I�X��.��?�?� I� I���1�,�,�Q�X�6�6�I��M�M�.�.��A�"6�"6�7�7�7�7��M�M�(�/�/�.�!�Q�"G�"G�H�H�H�H�  I� I� I� I� I� I� I� I� I� I� I� I���� I� I� I� I�&�f�-�-� /� /�L�A�y��)�W�^�4�4� /�%�,�,�.�.��q� �� $� $�V� $� $� $�E� �~����y�%�0�0� 1� 1�K�D�%� �M�M�"�(�4��/�/� 0� 0� 0� 0���6�"�"��(��B� �m�=N���2D����O�-3�O�G�x���(�(�(����H� �O�O�O�$�$�$� � !� !�(� +� +�F��F� �6�{�{�a��� �$�Q�'��/�D��w���)�!�,�W�5��(��+�F�3��(��+�F�3���U�5�$��5�5�6�6����� � � � �D� ���� �6�6� !�!s&�9A1D7�7D;�>D;�*AK� K�Kc���|jjtjkr||fSt j|��r&|�$|j}|j}tj ||��}n|�tj |j��}||fSr�) rVrWrOrUr� is_datetimetzrG�unitr� timestamp�from_numpy_dtype)r�rVr�rGr*s r6r�r��sw�� �|��B�M�)�)��u�}��� ��'�'�2�E�M� �X���z��� �T�2�&�&��� ���#�F�L�1�1�� �5�=�r5c��ddlmcm}|�dd��}|d}d|vr0tj�||d|d���}�nd|vr�tj|j ��\}} t||d��} t j ��r6tj � |�d ��| d � ��}n�|}|r |�|||j| � ��} | Snnd |vrh|d }t#|��dksJ�||d} || } t%| d��st'd���| �|��}n|}|r|�||���S||fS)a� Construct a pandas Block from the `item` dictionary coming from pyarrow's serialization or returned by arrow::python::ConvertTableToPandas. This function takes care of converting dictionary types to pandas categorical, Timestamp-with-timezones to the proper pandas Block, and conversion to pandas ExtensionBlock Parameters ---------- item : dict For basic types, this is a dictionary in the form of {'block': np.ndarray of values, 'placement': pandas block placement}. Additional keys are present for other types (dictionary, timezone, object). columns : Column names of the table being constructed, used for extension types extension_columns : dict Dictionary of {column_name: pandas_dtype} that includes all columns and corresponding dtypes that will be converted to a pandas ExtensionBlock. Returns ------- pandas Block rN�block� placement� dictionaryrd)rirdrerF)rV�copy)r/�klassrV�py_arrayr��__from_arrow__zGThis column does not support to be converted to a pandas ExtensionArray)r/)�pandas.core.internals�core� internals�getr�categorical_type� from_codesrO� datetime_datarV�make_datetimetz� is_ge_v21r�r��view� make_block�DatetimeTZBlockrhrXr�r4)�itemr��extension_columns� return_block�_int� block_arrr/rr*r�rVr.ru� pandas_dtypes r6�_reconstruct_blockrG�s���8)�(�(�(�(�(�(�(�(�����$�'�'�I��[�!�I��t����*�5�5� �$�|�"4���O�6�%�%��� �t� � ��"�9�?�3�3���a���d�:�&6�7�7�� � � "� "� ��.�&�&����w�'�'�u�5�'���C�C��C�� ���� �Y�.2�.B�.3�(�5�5��� �  � �t� � ��:����9�~�~��"�"�"�"��y��|�$��(��.� ��|�%5�6�6� ;��:�;�;� ;��)�)�#�.�.����������s�i��8�8�8��I�~�r5c��tj��rd}tj�|��}tj||���S)N�ns�rG)r�is_v1rr�string_to_tzinfo�datetimetz_type)r*rGs r6r<r<sC��������� �� � �� $� $�B� � &�t�� 3� 3� 3�3r5Fc �h���g}g}|jj}|sa|�_|d}|�dg��}|d}t||��}t ||||��\}} t |||||���n7t j�|j ��} t |g|||���t|��t|||��} |j �tj�|||t!��������} t j��r%ddlm} ��fd�| D��} | | | | ���}|Sddlm}dd lm}��fd �| D��} | | g}|| |��}t j��r|�||j��}n ||��}|S) Nr�r�r�r)�create_dataframe_from_blocksc�6��g|]}t|��d�����S)F)rC�rG�r�rAr��ext_columns_dtypess ��r6r�z&table_to_dataframe.<locals>.<listcomp>-sG��� � � �� ��l�$6�U� L� L� L� � � r5)r�r�)� BlockManager)� DataFramec�2��g|]}t|������Sr4rQrRs ��r6r�z&table_to_dataframe.<locals>.<listcomp>8s6��� � � �� �t�\�3E� F� F� � � r5)r�r%r8�_add_any_metadata�_reconstruct_index�_get_extension_dtypesrr�r��num_rows�'_check_data_column_metadata_consistency�_deserialize_column_indexr�rr�table_to_blocksr��keys�is_ge_v3�pandas.api.internalsrOr5rT�pandasrUr=� _from_mgr�axes)�options�tableri�ignore_metadata� types_mapper� all_columnsr�r%r�r�r�r^rO�blocksr�rTrUrc�mgrr�rSs @@r6�table_to_dataframerksN�����K��N��l�2�O� �  ��:�%�i�0� �(�,�,�-=�r�B�B��+�O�<��!�%��9�9��)�%�1B�*5�|�E�E� ��u�2� �;� �g�z�C�C�����)�)�%�.�9�9��2� �2�|�W�j� � ��,�K�8�8�8�'��{�N�K�K�G��%�L� �V� #� #�G�U�J�$(�);�)@�)@�)B�)B�$C�$C�E�E�F������E�E�E�E�E�E� � � � � �� � � �� *� )�&��w� O� O� O��� �6�6�6�6�6�6�$�$�$�$�$�$� � � � � �� � � �������l�6�4�(�(�� � � "� "� ��$�$�S�#�(�3�3�B�B���3���B�� r5> r r rrrrr=rrrrrrc���|d}|pg}i}tj�|S|r(|jD] }|j}||��} | � | ||j<�!|jD]\}|j}|j|vrJt |t j��r0 |���} | ||j<�L#t$rY�XwxYw�]|D]�} | d} n#t$r | d} YnwxYw| d} | |vr�| tvr�tj | ��} t | tj��r�t | tj j��rV|s| |vr�� t j�|j�| ��j��r��n#t$rYnwxYwt%| d��r| || <��tj��r�|s�|jD]�}|j|vr�t j�|j��sHt j�|j��s$t j�|j��r;|j|vr2tj �t.j���||j<��|S)a� Based on the stored column pandas metadata and the extension types in the arrow schema, infer which columns should be converted to a pandas extension dtype. The 'numpy_type' field in the column metadata stores the string representation of the original pandas dtype (and, despite its name, not the 'pandas_type' field). Based on this string representation, a pandas/numpy dtype is constructed and then we can check if this dtype supports conversion from arrow. �strings_to_categoricalNrvrurxr4)�na_value)r�extension_dtyper�rWrur@r�BaseExtensionType�to_pandas_dtype�NotImplementedErrorr?�_pandas_supported_numpy_typesrFr�� StringDtyper�� is_dictionaryr�rX�uses_string_dtype� is_string�is_large_string�is_string_viewrO�nan) re�columns_metadatargrdrirm� ext_columnsr��typrF�col_metarurVs r6rYrYOs���%�%=�>���!�r�J��K��"�*����7��\� 7� 7�E��*�C�'�<��,�,�L��'�*6� �E�J�'����7�7���j�� �:�[� (� (�Z��R�=Q�-R�-R� (� 7�"�2�2�4�4� �+7� �E�J�'�'��'� � � ��� ����� %�5�5�� $��L�)�D�D��� $� $� $��F�#�D�D�D� $������&�� �{� "� "�u�4Q�'Q�'Q�'�3�E�:�:�L��,� �(C�D�D� 5��l�K�N�,F�G�G� � .�!���1C�1C� ���8�1�1�%�,�2D�2D�T�2J�2J�2O�P�P�%�$�%��#�����������<�)9�:�:�5�(4�K��%���$�&�&�V�/E�V��\� V� V�E��z��,�,���"�"�5�:�.�.�-��8�+�+�E�J�7�7�-��8�*�*�5�:�6�6�-��*�J�.�.�*5�.�*D�*D�b�f�*D�*U�*U� �E�J�'�� �s6�;B� B'�&B'�0B9�9C� C�;<E9�9 F�Fc�<�td�|D����sJ�dS)Nc3�FK�|]}|ddurd|vp |dduV��dS)ruNrvr4�r�r�s r6� <genexpr>z:_check_data_column_metadata_consistency.<locals>.<genexpr>�sW������ � �6��d� � 0�|�q�0�J�Q�v�Y�d�5J������r5)�all)rhs r6r[r[�sF�� ������� � ��� ��r5c����|r d�|D����fd�|jD��}n|j}t|��dkrVtjj�t ttj |����d�|D�����}n1tj� ||r|ddnd���}t|��dkrt||��}|S) Nc �n�i|]2}|�dt|d����|d��3S�rvru)r8r�r�s r6� <dictcomp>z-_deserialize_column_index.<locals>.<dictcomp>�sI�� � � �� �E�E�,� 7��&� � B� B� C� C�Q�v�Y� � � r5c�<��g|]}��||����Sr4�r8)r�ru�columns_name_dicts �r6r�z-_deserialize_column_index.<locals>.<listcomp>�s7��� � � �26� � !� !�$�� -� -� � � r5r�c��g|] }|d�� Sr�r4)r�� col_indexs r6r�z-_deserialize_column_index.<locals>.<listcomp>�s��E�E�E��9�V�$�E�E�Er5�r�rrur�) r�rhrr�� MultiIndex� from_tuplesr�r��ast� literal_eval�Index�"_reconstruct_columns_from_metadata)� block_tablerhr��columns_valuesr�r�s @r6r\r\�s#���� 2� � � � � � �� � � � �:E�:R� � � ���%�1�� �>���Q����.�+�7�7� ��S�%�~�6�6� 7� 7�E�E�n�E�E�E�8� � ��� �.�&�&� �n�!V���!2�6�!:�!:�RV�'� � ��  �>���Q���4�W�n�M�M�� �Nr5c�Z�d�|D��}g}g}|}|D]�}t|t��rt|||||��\}} } | ��1n�|ddkr_|d} tj�|d|d|d| ���} t | ��t |��kr��n(td �|d�����|� | ��|� | ����tj} t |��d kr| j � ||� ��} nht |��d kr;|d } t| | j ��s| � | |d � ��} n| �|j ��} || fS)Nc�H�i|]}|�d|d��|�� Sr�r�r�s r6r�z&_reconstruct_index.<locals>.<dictcomp>�s<����� � ���l�A�f�I�&�&����r5r�r�rur�r�r�)r�ruzUnrecognized index kind: {}r�r�rr�)r@rY�_extract_index_levelrr�r�rhr�rCr�r�� from_arraysr�rZ) rer�rhrg�field_name_to_metadata� index_arrays� index_names� result_tabler�r�� index_namer�r�s r6rXrX�s��� ��������L��K��L�"�'�'�� �e�S� !� !� 5�4H��|�U�,B�L�5R�5R� 1�L�+�z��"��#��6�]�g� %� %��v��J�%�.�3�3�E�'�N�49�&�M�9>�v��9C�4�E�E�K��;���3�u�:�:�-�-��.��:�$�f�U�6�]�3�3�5�5� 5����K�(�(�(����:�&�&�&�&� ��B� �<���1���� �)�)�,�k�)�J�J��� �\� � �a� � ��Q����%���*�*� 9��H�H�U��Q��H�8�8�E��� � �e�n�-�-�� �� �r5c�P�||d}t||��}|j�|��}|dkr|ddfS|�|��}|�|���} d| _|�|j�|����}|| |fS)Nru�����)rg)� _backwards_compatible_index_namer��get_field_indexrl� to_pandasru� remove_column) rer�rvr�rg� logical_namer�r�r�r�s r6r�r�s���)�*�5�f�=�L�1�*�l�K�K�J� � �$�$�Z�0�0�A��B�w�w��T�4�'�'� �,�,�q�/�/�C��-�-�\�-�:�:�K��K���-�-���+�+�J�7�7���L� ��j� 0�0r5c�4�||krt|��rdS|S)a1Compute the name of an index column that is compatible with older versions of :mod:`pyarrow`. Parameters ---------- raw_name : str logical_name : str Returns ------- result : str Notes ----- * Part of :func:`~pyarrow.pandas_compat.table_to_blockmanager` N)r�)�raw_namer�s r6r�r�s(��$�<���$<�X�$F�$F���t��r5c�4�d}tj||��duS)Nz^__index_level_\d+__$)�re�match)ru�patterns r6r�r�/s��&�G� �8�G�T� "� "�$� .�.r5c ���tsWt�ddddtjdtjtjtjtjd� ��tS)NrLrfrY) rr;r:rrrMr�floatingr<r )�_pandas_logical_type_maprrOrRrr�object_r4r5r6�get_pandas_logical_type_mapr�4s^�� $�  � �'�'�#�(�*���Y���x�� ��z��Z� ) � ) � � � � $�#r5c��t��} ||S#t$r)d|vrtjcYStj|��cYSwxYw)a Get the numpy dtype that corresponds to a pandas type. Parameters ---------- pandas_type : str The result of a call to pandas.lib.infer_dtype. Returns ------- dtype : np.dtype The dtype that corresponds to `pandas_type`. r�)r�r?rOr�rV)rw�pandas_logical_type_maps r6�_pandas_type_to_numpy_typer�Gsj��:�;�;��%�&�{�3�3�� �%�%�%� �k� !� !��:� � � ��x� �$�$�$�$�$� %���s��A �A � A c���tj}t|dd��p|g}t|dd��pdg}d�t||i���D��}g}t jdd��}|D�]�\}} } t | ��} | tjkr|� |��}| dkr�tj � |d d d ��} |� |d � ���| ��}tj��r-|�tj| ��d ��}n�| dkr*tj�d�|D����}nP|jdkr%| dkrd| vs| dvr|�|����=|j| kr|�| ��}|j| kr| dkr|�| ��}|�|�����t-|��dkr|�|||j���S|�|d |d j|j���S)a_Construct a pandas MultiIndex from `columns` and column index metadata in `column_indexes`. Parameters ---------- columns : List[pd.Index] The columns coming from a pyarrow.Table column_indexes : List[Dict[str, str]] The column index metadata deserialized from the JSON schema metadata in a :class:`~pyarrow.Table`. Returns ------- result : MultiIndex The index reconstructed using `column_indexes` metadata with levels of the correct type. Notes ----- * Part of :func:`~pyarrow.pandas_compat.table_to_blockmanager` r�Nrjc ��g|]D\}}||�dt|j����|�dd��f��ES)rwrxN)r8rYrV)r�r�r�s r6r�z6_reconstruct_columns_from_metadata.<locals>.<listcomp>{s^����� �E�9� � � � �m�S���-=�-=�>�>� ���|�T� *� *� ,���r5)� fillvaluer�r�r:rrnreT)�utcr<c�6�g|]}tj|����Sr4)r<�Decimal)r�r�s r6r�z6_reconstruct_columns_from_metadata.<locals>.<listcomp>�s"��)L�)L�)L��'�/�!�*<�*<�)L�)L�)Lr5rYr=r�)rrMr�r�)rVru)rr�rgr�operator� methodcallerr�rOrRr�rrrL� to_datetime� tz_convertr_�as_unitr;r�rVr��astyperhr�r�ru) r�r�r�r��labels� levels_dtypes� new_levels�encoderr�rF� numpy_dtyperVrGs r6r�r�^s���, ��B��W�h�� -� -� :�'��F� �W�g�t� ,� ,� 6���F���!,� �N�b�! �! �! ����M��J��#�H�g�6�6�G�,9�'!�'!�(��|�[�*�<�8�8�� �B�I� � ��I�I�g�&�&�E� �<� '� '���(�(��q�!�*�-�j�9�;�;�B��N�N�5�d�N�3�3�>�>�r�B�B�E��#�%�%� B�� � �b�&6�u�&=�&=�a�&@�A�A��� �Y� &� &��N�(�(�)L�)L�e�)L�)L�)L�M�M�E�E� �K�5� � �[�H�%<�%<��L�(�(�L�<Q�,Q�,Q� � � �e� $� $� $� � �[�E� !� !��L�L��'�'�E� �;�+� %� %�,�,�*F�*F��L�L��-�-�E����%� � � � � �:�������}�}�Z��w�}�}�E�E�E��x�x� �1� �Z��]�-@�w�|�x�T�T�Tr5c���i}i}|j}|d}d�|D��}t|��}t|d��|z }t|d��D�]3\}} | �d��} | s| d} ||kr |||z } | �d} |�| ��} | dkr�| dd kr�|| } t | jtjj ��s��| d } | s��| �d ��}|r�|| jj krs| � ��}tj d |� ��}tj �||���}tj|| j|��|| <||| <��5t|��dkr�g}g}t#t|j����D]x}||vr7|�||��|�||���=|�||��|�|j|���ytj�|tj|�����S|S)Nr�c�<�g|]}t|t���|��Sr4)r@rY)r��idx_cols r6r�z%_add_any_metadata.<locals>.<listcomp>�s6��2�2�2��"�7�C�0�0�2�W�2�2�2r5r�rvru�Noner�rwr:rnrerIrJrr)r�)r�rhr�r8r�r@rWrrrFrGr�r+�Arrayr�r�rur�r��Tabler�)rer%�modified_columns�modified_fieldsr�r��n_index_levels� n_columnsr�r~r��idxr�rn� metadata_tz� converted� tz_aware_typerr�r$s r6rWrW�s������O� �\�F�#�O�4�M�2�2�M�2�2�2�M���'�'�N��O�I�.�/�/�.�@�I�!���!;�<�<�:�:� ��8��<�<� �-�-��� "���'�H��I�~�~�(��Y��7����!���$�$�X�.�.�� �"�9�9�� �&�,�6�6��C�j��!�#�(�B�F�,@�A�A���#�J�/�����&�l�l�:�6�6� ��:�;�#�(�+�#=�#=� #� � ���I�$&�L��+�$F�$F�$F�M�$&�H�$8�$8��>K�%9�%M�%M�M�,.�8�F�3�K�4D�4A�,C�,C�O�C�(�,9�$�S�)�� � ���q� � ������s�5�<�(�(�)�)� /� /�A��$�$�$����/��2�3�3�3�� � �o�a�0�1�1�1�1����u�Q�x�(�(�(�� � �e�l�1�o�.�.�.�.��x�#�#�G�B�I�f�4E�4E�#�F�F�F�� r5c��tj�|��}|j�d��j�|��}|S)zB Make a datetime64 Series timezone-aware for the given tz r�)rrrL�dt� tz_localizer�)�seriesrGs r6� make_tz_awarer��sC�� �� � �� $� $�B��i�#�#�E�*�*�� � �2��� � �Mr5r�)r�NT)NNT)NFN)?r��collections.abcr� concurrentr�concurrent.futures.threadr1rr<� itertoolsrr�r�r�r��numpyrO� ImportErrorr�r� pyarrow.librrr rrNr�r7rDrSr_rpr�r�r�r�r�r�r�r�r�r�r�r�r�r'r�rGr<rkrsrYr[r\rXr�r�r�r�r�r�rWr�r4r5r6�<module>r�s���& � � � �$�$�$�$�$�$�������!� � � �����������!�!�!�!�!�!� � � � ����� � � � ���������������� �B�B�B���������D�D�D�D�D�D�D�D�D�D����������:���"#�#�#�,���"$�$�$�&-�-�-�d+/�\�\�\�\�~���($�$�$�N0�0�0�&?Q�?Q�?Q�D;Q�;Q�;Q�| *� *� *�������9�9�9� � � � &� &� &� &�FIM�!�b"�b"�b"�b"�J � � �&B�B�B�B�J4�4�4�JN�4�4�4�4�r!�!�!��P�P�P�f������B3�3�3�3�n?C�1�1�1�1�&���0/�/�/� $�$�$�&%�%�%�.TU�TU�TU�n:�:�:�B����s�;�A�A
Memory