� K�g8:����dZddlmZmZmZmZmZddlmZddl m Z ddl m Z ddl mZmZddlmZmZd d lmZdd lmZd d lmZd d lmZmZmZmZmZerddlm Z m!Z!Gd�d��Z"dS)a�This module implements an Earley parser. The core Earley algorithm used here is based on Elizabeth Scott's implementation, here: https://www.sciencedirect.com/science/article/pii/S1571066108001497 That is probably the best reference for understanding the algorithm here. The Earley parser outputs an SPPF-tree as per that document. The SPPF tree format is explained here: https://lark-parser.readthedocs.io/en/latest/_static/sppf/sppf.html �)� TYPE_CHECKING�Callable�Optional�List�Any)�deque�)�Token)�Tree)� UnexpectedEOF�UnexpectedToken)�logger� OrderedSet�)�GrammarAnalyzer)� NonTerminal)�Item)�ForestSumVisitor� SymbolNode�StableSymbolNode� TokenNode�ForestToParseTree)� LexerConf� ParserConfc��eZdZUded<ded<eed<ddedfddddded eded eeee ge fd efd �Z d �Z dd�Z d�ZdS)�Parserr� lexer_confr� parser_conf�debugTF� term_matcher�resolve_ambiguity� tree_class� ordered_setsc���t|��}||_||_||_||_||_|rt nt|_|rtnt|_ |j |_ |j |_ |j |_ i|_d�|jD��|_d�|jD��|_d|_|jD]`} | j|jvr1d�|�| j��D��|j| j<|j�| jj� t.|_�a|jjdkr+|j�$|jjD]} | jrt.|_n�||_dS)Nc�4�h|]}|jD] }|j� |�� �S��� expansion�is_term��.0�r�syms �c/home/asafur/pinokio/api/open-webui.git/app/env/lib/python3.11/site-packages/lark/parsers/earley.py� <setcomp>z"Parser.__init__.<locals>.<setcomp>4s0��a�a�a�1�Q�[�a�a�c�TW�T_�a�3�a�a�a�a�c�4�h|]}|jD] }|j� |�� �Sr&r'r*s r.r/z"Parser.__init__.<locals>.<setcomp>5s0��i�i�i�q���i�i�#�\_�\g�i�s�i�i�i�ir0c��g|] }|j�� Sr&)�rule)r+�xs r.� <listcomp>z#Parser.__init__.<locals>.<listcomp>:s��0c�0c�0c�A���0c�0c�0cr0�basic)rrrr!rr r�set�Setrr�FIRST�NULLABLE� callbacks� predictions�rules� TERMINALS� NON_TERMINALS�forest_sum_visitor�origin� expand_rule�options�priorityr� lexer_type� terminalsr ) �selfrrr r!rr"r#�analysisr3�terms r.�__init__zParser.__init__ s���#�;�/�/��$���&���!2����� ��� �!-�6�:�:�3���.:�J�*�*� ����^�� � �)�� �$�.������b�a� �(9�a�a�a���i�i�K�,=�i�i�i���"&����%� ;� ;�D��{�$�"2�2�2�0c�0c��AU�AU�VZ�Va�Ab�Ab�0c�0c�0c�� ���-� �&�.�4�<�3H�3T�*:��'�� �?� %�� 0� 0�T�5L�5T���1� � ���=��.>�D�+��E��)����r0c���i}i}||}t|��}|�r.|�����j�r��j�k�j�j|f} | |vr|| n|�| |j| ����_�j��j�j �jdd���j j |�jv�r|�j�j} | j || j vr|| j | j } n| } t| j | j| j��} | j| j|f} | |vr|| n|�| |j| ���| _| j�| �j��| j|jvr|�| ���n�| |vr*|�| ��|�| ���ng�j|k} | r�j|�j j <�fd�|�jD��}|D]�}|���} | j|j|f} | |vr|| n|�| |j| ���| _| j�| j| j ||j�j��| j|jvr|�| ����| |vr*|�| ��|�| ���ِnP�j|jv�rAg}|j�jD](}t|d|��} |�| ���)�j|vr�����} | j�j|f} | |vr|| n|�| |j| ���| _| j�| j| j | j�j|�j��|�| ��|D]T} | j|jvr|�| ���&| |vr*|�| ��|�| ���U|��,dSdS)a�The core Earley Predictor and Completer. At each stage of the input, we handling any completed items (things that matched on the last cycle) and use those to predict what should come next in the input stream. The completions and any predicted non-terminals are recursively processed until we reach a set of, which can be added to the scan list for the next scanner cycle.Nc�B��g|]}|j� |j�jk�|��S�N)�expect�s)r+� originator�items �r.r5z/Parser.predict_and_complete.<locals>.<listcomp>�sa���#V�#V�#V�*�Wa�Wh�Wt�zD�zK�OS�OU�zU�zU�:�zU�zU�zUr0r)r�pop� is_complete�noderO�start� setdefaultr� add_familyr3rA�previous�columnr�ptr�add_pathrNr>�add�append�advancer?r<)rG�i�to_scan�columns� transitives� node_cache�held_completionsrY�items�label� transitive�root_transitive�new_item� is_empty_item� originatorsrP� new_itemsr3rQs @r.�predict_and_completezParser.predict_and_completeNs���� ��������f� � ���L /��9�9�;�;�D���H /��9�$�!�V�T�Z��3�E�5:�j�5H�5H� �5� 1� 1�j�Nc�Nc�di�kz�ko�kz�}B�lC�OD�OD�D�I��I�(�(�����D�J��d�S�S�S� �9�#�{�4�:�'>�>�>�!,�T�Z�!8���!@�J�!�*�k�*�:K�.L�L�L�*5�j�6G�*H��I\�*]���*4��#�J�O�Z�^�Z�EU�V�V�H�,�.��0E�q�I�E�9>�*�9L�9L�J�u�$5�$5�R\�Rg�Rg�hm�o~�os�o~�AF�pG�SH�SH�H�M��M�*�*�?�D�I�F�F�F���$�.�8�8�� � �H�-�-�-�-�!��/�/�� � �8�,�,�,�� � �X�.�.�.��%)�J�!�O�M�$�G�=A�Y�(���)9�:�#V�#V�#V�#V��� �@S�#V�#V�#V�K�&1� 3� 3� �#-�#5�#5�#7�#7��!)��Z�-=�q� A��=B�j�=P�=P� �5�(9�(9�V`�Vk�Vk�lq�tC�sw�tC�EJ�tK�WL�WL�� � � �0�0���X�]�A�z��`d�`i�j�j�j�#�?�d�n�<�<�#�K�K��1�1�1�1�%�V�3�3�"�J�J�x�0�0�0�!�L�L��2�2�2�� 3���� 2�2�2�� � �,�T�[�9�/�/�D�#�D�!�Q�/�/�H��$�$�X�.�.�.�.��;�"2�2�2�#�|�|�~�~�H�%�Z���Q�7�E�9>�*�9L�9L�J�u�$5�$5�R\�Rg�Rg�hm�o~�os�o~�AF�pG�SH�SH�H�M��M�,�,�X�Z�����X\�Xa�cs�tx�t�dA�B�B�B��$�$�X�.�.�.� )�/�/�H���$�.�8�8�� � �H�-�-�-�-�!��/�/�� � �8�,�,�,�� � �X�.�.�.��Y�L /�L /�L /�L /�L /r0Nc����� � � ���fd�}�� �� � fd�}�j� �jj� ig� d�|D��}d}|�|��D]O} ��||�� ��||| |��}|dz }|���|d�|D��z}�P��||�� ��|t ���dz ksJ�|S)Nc����|jrdS|���}|jsH|j�jvrdS|jj�kr |j�krdS|���}|j�HdS)NTF)rSr^rNr:r3rA)rQ�quasirG� start_symbols ��r.�is_quasi_completez(Parser._parse.<locals>.is_quasi_complete�s������ ��t��L�L�N�N�E��'� (��<�t�}�4�4� �5��:�$� �4�4����9U�9U� �5�� � ���� �'� (� �4r0c �������}����}� �|����i��i}��|��D�]}� |j|���r|���}|j|j|f}t |t��r��|j ��nd} t|| d���} ||vr||n|� |�j |���|_ |j �|j|j|j|j | ��|j�jvr|�|����|�|����|sE|sCd�|D��} t%|| t'|��t)d�|D��������|S)a|The core Earley Scanner. This is a custom implementation of the scanner that uses the Lark lexer to match tokens. The scan list is built by the Earley predictor, based on the previously completed tokens. This ensures that at each phase of the parse we have a custom lexer context, allowing for more complex ambiguities.Nr)rDc�&�h|]}|jj��Sr&�rN�name�r+r_s r.r/z.Parser._parse.<locals>.scan.<locals>.<setcomp>�s��9�9�9�A�!�(�-�9�9�9r0c3�$K�|] }|jV�� dSrM�rOrws r.� <genexpr>z.Parser._parse.<locals>.scan.<locals>.<genexpr>�s'����cy�cy�lm�de�dg�cy�cy�cy�cy�cy�cyr0)�considered_rules�state)r8r]rNr^rOrU� isinstancer �get�typerrVrrTrWr3r>r\r r7� frozenset)r_�tokenr`� next_to_scan�next_setrcrQrirfrI� token_noderNra�matchrGrFrbs �����r.�scanzParser._parse.<locals>.scan�s���� �8�8�:�:�L��x�x�z�z�H� �N�N�8� $� $� $� � � �r� "� "� "��J�����)�)� /� /���5���e�,�,�/�#�|�|�~�~�H�%�Z����;�E�9C�5�%�8P�8P�Z�9�=�=���4�4�4�VZ�D� "+�5�$��!C�!C�!C�J�9>�*�9L�9L�J�u�$5�$5�R\�Rg�Rg�hm�o~�os�o~�AF�pG�SH�SH�H�M��M�,�,�X�Z���H�N�TX�T]�_i�j�j�j���$�.�8�8�$�(�(��2�2�2�2�!� � �X�.�.�.��� {�L� {�9�9��9�9�9��%�e�V�c�'�l�l�Zc�cy�cy�qx�cy�cy�cy�Zy�Zy�z�z�z�z�� r0c��h|] }|j�� Sr&�rNrws r.r/z Parser._parse.<locals>.<setcomp>�s��-�-�-��1�8�-�-�-r0rrc��h|] }|j�� Sr&r�rws r.r/z Parser._parse.<locals>.<setcomp>�s��2�2�2�Q���2�2�2r0)r r�terminals_by_name�lexrm�clear�len) rG�lexerrar`rqrrr��expectsr_r�r�rFrbs ` ` ` @@@r.�_parsez Parser._parse�s@�������� � � � � � � ) �) �) �) �) �) �) �) �) �Z�!���O�5� ��d� �.�-�W�-�-�-�� ���Y�Y�w�'�'� 3� 3�E� � %� %�a��'�;� G� G� G��d�1�e�W�-�-�G� ��F�A� �M�M�O�O�O� �2�2�'�2�2�2� 2�G�G� �!�!�!�W�g�{�C�C�C��C��L�L��N�"�"�"�"��r0c��� �|s J|���t|��� |���g}|���}|j� D]R}t|dd��}|j|jvr|�|���7|d�|���S|�|||� ��}� fd�|dD��}|s4d�|D��}t|td�|D��������|j rRddl m }  | ��} | � |dd ��n$#t$rtjd ��YnwxYwt#|��dkr Jd ���|j�Ut'|j|j|jo|���|j��} | �|d��S|dS) Nrc�f��g|]-}|j� |j�|j�k�|jdk�&|j��.S)Nr)rSrTrOrU)r+�nrqs �r.r5z Parser.parse.<locals>.<listcomp>sk���C�C�C��A�M�C�a�f�FX�]^�]`�dp�]p�]p�uv�u|�AB�vB�vB�Q�V�vB�vB�vBr0�����c�&�g|]}|jj��Sr&ru)r+�ts r.r5z Parser.parse.<locals>.<listcomp>s��!A�!A�!A�A�!�(�-�!A�!A�!Ar0c3�$K�|] }|jV�� dSrMryrws r.rzzParser.parse.<locals>.<genexpr>s$����CY�CY�A�A�C�CY�CY�CY�CY�CY�CYr0)r|r)�ForestToPyDotVisitorzsppf.pngzBCannot find dependency 'pydot', will not generate sppf debug imageFz7Earley should not generate multiple start symbol items!)rr8r<rrNr>r\r�r r�r� earley_forestr��visit� ImportErrorr�warningr�r rr;r@r!� transform) rGr�rUrar`r3rQ� solutions�expected_terminalsr�� debug_walker� transformerrqs @r.�parsez Parser.parsesI������e���u�"�5�)�)� ��8�8�:�:�,���(�(�*�*�� �$�\�2� %� %�D���a��#�#�D��{�d�n�,�,�� � �D�!�!�!�!��� ���t�$�$�$�$��+�+�e�W�g�|�D�D�� C�C�C�C�W�R�[�C�C�C� �� [�!A�!A��!A�!A�!A� �� 2�)�CY�CY�QX�CY�CY�CY�:Y�:Y�Z�Z�Z� Z� �:� =� ;� ;� ;� ;� ;� ;� =�3�3�5�5� ��"�"�9�Q�<��<�<�<�<��� e� e� e���c�d�d�d�d�d� e���� �y�>�>�A� � � S�S� S� S�5� �9� �+�D�I�t�~�t�G^�G|�cg�cz�cz�c|�c|�C�U�V�V�K��(�(��1��6�6� 6���|�s� D=�=E�ErM)�__name__� __module__� __qualname__�__annotations__�boolr rr�strrrrJrmr�r�r&r0r.rrs���������������� �K�K�K�*.�5�BF�[_�+)�+)�;�+)�\�+)�Ya�+)�$(�+)�6:�+)�%�h��T�{�C�/?�&@�A�+)�VZ�+)�+)�+)�+)�\[/�[/�[/�zY�Y�Y�Y�v.�.�.�.�.r0rN)#�__doc__�typingrrrrr� collectionsrr�r �treer � exceptionsr r �utilsrr�grammar_analysisr�grammarr� earley_commonrr�rrrrr�commonrrrr&r0r.�<module>r�s^�� � �@�?�?�?�?�?�?�?�?�?�?�?�?�?�������������������7�7�7�7�7�7�7�7�&�&�&�&�&�&�&�&�-�-�-�-�-�-�!�!�!�!�!�!�������g�g�g�g�g�g�g�g�g�g�g�g�g�g��/�.�.�.�.�.�.�.�.�Y�Y�Y�Y�Y�Y�Y�Y�Y�Yr0
Memory