� K�g2���dZddlmZmZmZmZmZddlmZddl m Z ddl m Z ddl mZddlmZd d lmZd d lmZerdd lmZmZGd �de��ZdS)a�This module implements an Earley parser with a dynamic lexer The core Earley algorithm used here is based on Elizabeth Scott's implementation, here: https://www.sciencedirect.com/science/article/pii/S1571066108001497 That is probably the best reference for understanding the algorithm here. The Earley parser outputs an SPPF-tree as per that document. The SPPF tree format is better documented here: http://www.bramvandersanden.com/post/2014/06/shared-packed-parse-forest/ Instead of running a lexer beforehand, or using a costy char-by-char method, this parser uses regular expressions by necessity, achieving high-performance while maintaining all of Earley's power in parsing any CFG. �)� TYPE_CHECKING�Callable�Optional�List�Any)� defaultdict�)�Tree)�UnexpectedCharacters)�Token��Terminal�)�Parser)� TokenNode)� LexerConf� ParserConfc�j�eZdZdddedfdddddeded ed ed eeeege fd efd �Z dd�Z dS)rTF� lexer_confr� parser_confr� term_matcher�resolve_ambiguity� complete_lex�debug� tree_class� ordered_setsc �v�tj||||||||��d�|jD��|_||_dS)Nc�,�g|]}t|����S�r )�.0�ts �d/home/asafur/pinokio/api/open-webui.git/app/env/lib/python3.11/site-packages/lark/parsers/xearley.py� <listcomp>z#Parser.__init__.<locals>.<listcomp>$s��>�>�>�q�x��{�{�>�>�>�)� BaseParser�__init__�ignorer) �selfrrrrrrrrs r"r&zParser.__init__sQ�� ��D�*�k�<�IZ�!�:�|� =� =� =�>�>�J�,=�>�>�>�� �(����r$Nc �x������ � � � � ���� ���� � � � f d�}tt����j� �jj� ig� d� d� d}�D]>}��||�� ��|||��}|dkr� dz � d� n� dz � |dz }�?��||�� ��|t ���dz ksJ�|S)Nc �^� ��i}��|��D�]3}�|j����}|�rt|jj|�d�������}�|����|�|f���jr�|�d��}tdt|����D]�}�|j|d| ���}|rct|jj|�d�������}��|���z�|�|f������5�j D]�}�|����}|ry�|���� �fd�|D�����|���� ��fd���D����������} ����} ��| ����i����dzD]�\}} } | ���| _ �dz| _ �dz| _|���} | j| j�f}t%| �| j��}||vr||n|�|�j|���| _| j�| j|j| j|j|��n|} | j�jvr| �| ����| �| ������dz=| sh�sf| sdt7t9|d������}t;����d�|D��t=|��t?d �|D����|� ���| S) a|The core Earley Scanner. This is a custom implementation of the scanner that uses the Lark lexer to match tokens. The scan list is built by the Earley predictor, based on the previously completed tokens. This ensures that at each phase of the parse we have a custom lexer context, allowing for more complex ambiguities.rrNc���g|]}|�df��S�Nr)r �item�is �r"r#z/Parser._parse.<locals>.scan.<locals>.<listcomp>Us���4Z�4Z�4Z��d�A�t�_�4Z�4Z�4Zr$c�>��g|]}|j� |j�k�|�df��Sr,)� is_complete�s)r r-r.� start_symbols ��r"r#z/Parser._parse.<locals>.scan.<locals>.<listcomp>Xs[���5L�5L�5L��_c�_o�5L�tx�tz�K�uK�uK�d�A�t�_�uK�uK�uKr$c�$�|jjjSr,)�rule�origin�name��keys r"�<lambda>z-Parser._parse.<locals>.scan.<locals>.<lambda>|s �����H\�r$r7c�&�h|]}|jj��Sr)�expectr6)r r-s r"� <setcomp>z.Parser._parse.<locals>.scan.<locals>.<setcomp>}s��Nt�Nt�Nt�dh�t�{�O_�Nt�Nt�Ntr$c3�$K�|] }|jV�� dSr,)r1)r r.s r"� <genexpr>z.Parser._parse.<locals>.scan.<locals>.<genexpr>~s%����H^�H^�QR���H^�H^�H^�H^�H^�H^r$)�state�considered_rules) �Setr;r r6�group�end�appendr�range�lenr'�extend�end_line� end_column�end_pos�advancer1�startr�type� setdefault� SymbolNode�node� add_familyr4� TERMINALS�add�list�sortedr �set� frozenset)r.�to_scan� node_cacher-�mr!r1�j�x� next_to_scan�next_setrL�token�new_item�label� token_noder@�columns�delayed_matches�matchr(r2�stream� terminals� text_column� text_line� transitivess` ����������r"�scanzParser._parse.<locals>.scan)s#�����J�����)�)� R� R���E�$�+�v�q�1�1��� R��d�k�.����� � �A�y�+�V�V�A�#�A�E�E�G�G�,�3�3�d�A�q�\�C�C�C��(�R��G�G�A�J�J��!&�q�#�a�&�&�!1�!1�R�R�A� %��d�k�1�S�q�b�S�6� :� :�A� �R�$)�$�+�*:�A�G�G�A�J�J��9�Va�$b�$b�� /��!�%�%�'�'� � :� A� A�D�!�Q�<� Q� Q� Q����[� M� M���E�!�V�Q�'�'���M�#�A�E�E�G�G�,�3�3�4Z�4Z�4Z�4Z�QX�4Z�4Z�4Z�[�[�[�$�A�E�E�G�G�,�3�3�5L�5L�5L�5L�5L�QX�YZ�Q[�5L�5L�5L�M�M�M���8�8�:�:�L��x�x�z�z�H� �N�N�8� $� $� $� � � �r� "� "� "�'6�a��c�&:� +� +�"��e�U��$�%.�E�N�'2�Q��E�$�$%��E�E�M�#�|�|�~�~�H�%�Z����;�E�!*�5�)�E�J�2G�!H�!H�J�9>�*�9L�9L�J�u�$5�$5�R\�Rg�Rg�hm�o~�os�o~�AF�pG�SH�SH�H�M��M�,�,�X�Z���H�N�TX�T]�_i�j�j�j�j�#�H��?�d�n�4�4� �$�$�X�.�.�.�.��L�L��*�*�*�*���!��$�� -�O� -�L� -�#'��w�<\�<\�(]�(]�(]�#^�#^� �*�6�1�i��Nt�Nt�ls�Nt�Nt�Nt�+.�w�<�<�y�H^�H^�V]�H^�H^�H^�?^�?^�<L�-�-�-�-�  � r$rr� )rrTrr�terminals_by_name�predict_and_completerF)r(rfrcrXr2rkr.r_rdrergrhrirjs``` ` @@@@@@r"�_parsez Parser._parse's@������������Y �Y �Y �Y �Y �Y �Y �Y �Y �Y �Y �Y �Y �Y �x&�d�+�+���!���O�5� ��d� �� �� � ��� � �E� � %� %�a��'�;� G� G� G��d�1�g�&�&�G���}�}��Q�� �� � ��q� � � ��F�A�A� �!�!�!�W�g�{�C�C�C��C��L�L��N�"�"�"�"��r$r,) �__name__� __module__� __qualname__r r�boolr�strrrr&rorr$r"rrs�������)-�%�UZ�BF�[_�)�)�;�)�\�)�Ya�)�$(�)�=A�)�PT�)�%�h��T�{�C�/?�&@�A�)�VZ�)�)�)�)�~�~�~�~�~�~r$rN)�__doc__�typingrrrrr� collectionsr�treer � exceptionsr �lexerr �grammarr�earleyrr%� earley_forestr�commonrrrr$r"�<module>rs���� @�?�?�?�?�?�?�?�?�?�?�?�?�?�#�#�#�#�#�#�������-�-�-�-�-�-�������������(�(�(�(�(�(�$�$�$�$�$�$��/�.�.�.�.�.�.�.�.�G�G�G�G�G�Z�G�G�G�G�Gr$
Memory