� K�g�0��D�dZddlmZddlZddlmZddlmZddlm Z ddl m Z m ZmZd �ZGd �d ��ZGd �d ��ZGd�d��ZGd�d��Zd"d�Zd�ZGd�d��ZGd�de��Zd�Zd�Zd�Zd�Zd�Zd�Zd�Zd�Z d �Z!d!�Z"dS)#z$This module implements a CYK parser.�)� defaultdictN�)� ParseError)�Token)�Tree)�Terminal� NonTerminal�Symbolc�P�t|t��sJ�|j|jkS�N)� isinstance�T�name�type)�t�ss �`/home/asafur/pinokio/api/open-webui.git/app/env/lib/python3.11/site-packages/lark/parsers/cyk.py�matchrs'�� �a�� � ��� � �6�Q�V� ��c�@��eZdZdZ�fd�Zd�Zd�Zd�Zd�Zd�Z �xZ S)�RulezContext-free grammar rule.c���tt|�����t|t��s J|���t d�|D����s J|���||_||_||_||_ dS)Nc3�jK�|].}t|t��pt|t��V��/dSr )r �NTr��.0�xs r� <genexpr>z Rule.__init__.<locals>.<genexpr>s;����F�F�Q�:�a��$�$�8� �1�a�(8�(8�F�F�F�F�F�Fr) �superr�__init__r r�all�lhs�rhs�weight�alias)�selfr"r#r$r%� __class__s �rr z Rule.__init__s���� �d�D���"�"�$�$�$��#�r�"�"�'�'�C�'�'�"��F�F�#�F�F�F�F�F�K�K��K�K�F��������� ��� � � rc�x�t|j���dd�d�|jD������S)Nz -> � c3�4K�|]}t|��V��dSr ��strrs rrzRule.__str__.<locals>.<genexpr>"s(����4N�4N��S��V�V�4N�4N�4N�4N�4N�4Nr)r,r"�joinr#�r&s r�__str__z Rule.__str__!s9�� ���]�]�]�]�C�H�H�4N�4N�T�X�4N�4N�4N�,N�,N�,N�O�Orc� �t|��Sr r+r.s r�__repr__z Rule.__repr__$����4�y�y�rc�R�t|jt|j��f��Sr )�hashr"�tupler#r.s r�__hash__z Rule.__hash__'s���T�X�u�T�X���/�0�0�0rc�B�|j|jko|j|jkSr �r"r#�r&�others r�__eq__z Rule.__eq__*s���x�5�9�$�>���U�Y�)>�>rc��||k Sr �r9s r�__ne__z Rule.__ne__-s���E�M�"�"r) �__name__� __module__� __qualname__�__doc__r r/r1r6r;r>� __classcell__�r's@rrrs��������$�$������P�P�P����1�1�1�?�?�?�#�#�#�#�#�#�#rrc�*�eZdZdZd�Zd�Zd�Zd�ZdS)�GrammarzContext-free grammar.c�.�t|��|_dSr )� frozenset�rules)r&rIs rr zGrammar.__init__4s���u�%�%�� � � rc�"�|j|jkSr )rIr9s rr;zGrammar.__eq__7s���z�U�[�(�(rc�p�dd�td�|jD������zdzS)N� c3�4K�|]}t|��V��dSr )�reprrs rrz"Grammar.__str__.<locals>.<genexpr>;s(����&C�&C�1�t�A�w�w�&C�&C�&C�&C�&C�&Cr)r-�sortedrIr.s rr/zGrammar.__str__:s7���d�i�i��&C�&C�� �&C�&C�&C� C� C�D�D�D�t�K�Krc� �t|��Sr r+r.s rr1zGrammar.__repr__=r2rN)r?r@rArBr r;r/r1r=rrrFrF1sY��������&�&�&�)�)�)�L�L�L�����rrFc� �eZdZdZdd�Zd�ZdS)�RuleNodez@A node in the parse tree, which also contains the full rhs rule.rc�0�||_||_||_dSr )�rule�childrenr$)r&rTrUr$s rr zRuleNode.__init__Es���� � �� ��� � � rc��dt|jj���dd�d�|jD�����d�S)Nz RuleNode(z, [z, c3�4K�|]}t|��V��dSr r+rs rrz$RuleNode.__repr__.<locals>.<genexpr>Ks)����Ed�Ed�QR�c�!�f�f�Ed�Ed�Ed�Ed�Ed�Edrz]))rNrTr"r-rUr.s rr1zRuleNode.__repr__JsF���'+�D�I�M�':�':�':�':�D�I�I�Ed�Ed�VZ�Vc�Ed�Ed�Ed�<d�<d�<d�<d�e�erN�r)r?r@rArBr r1r=rrrRrRBsB������J�J����� f�f�f�f�frrRc�4��eZdZdZ�fd�Zd�Zd�Zd�Z�xZS)�ParserzParser wrapper.c�����tt������d�|D���_�fd�|D��}t t |�����_dS)Nc��i|]}||��Sr=r=)rrTs r� <dictcomp>z#Parser.__init__.<locals>.<dictcomp>Ts��8�8�8�$�4��8�8�8rc�:��g|]}��|����Sr=)�_to_rule)rrTr&s �r� <listcomp>z#Parser.__init__.<locals>.<listcomp>Us%���7�7�7�����t�$�$�7�7�7r)rrZr � orig_rules�to_cnfrF�grammar)r&rIr's` �rr zParser.__init__Rsg���� �f�d���$�$�&�&�&�8�8�%�8�8�8���7�7�7�7��7�7�7���g�e�n�n�-�-�� � � rc���t|jt��sJ�td�|jD����sJ�t |j|j|jjr |jjnd|���S)z?Converts a lark rule, (lhs, rhs, callback, options), to a Rule.c3�@K�|]}t|t��V��dSr )r r rs rrz"Parser._to_rule.<locals>.<genexpr>[s,����F�F�Q�:�a��(�(�F�F�F�F�F�Frr�r$r%)r �originrr!� expansionr�options�priority)r&� lark_rules rr_zParser._to_ruleXs����)�*�B�/�/�/�/�/��F�F�)�2E�F�F�F�F�F�F�F�F�� � �i�1�1:�1B�1K�R�9�$�-�-�QR����� rc�b���sJ�t����t||j��\}}t�fd�|dt |��dz fD����rt d���|dt |��dz f�}|�t|����S)z(Parses input, which is a list of tokens.c3�.�K�|]}|j�kV��dSr )r")r�r�starts �rrzParser.parse.<locals>.<genexpr>hs)�����F�F�!�q�u��~�F�F�F�F�F�Frr�zParsing failed.)r�_parsercr!�lenr�_to_tree� revert_cnf)r&� tokenizedro�table�trees�parses ` rrxz Parser.parseas����� � �u��5� � ���i���6�6� ��u� �F�F�F�F�u�a��Y���!�1C�-D�'E�F�F�F� F� F� 0��.�/�/� /��q�#�i�.�.�1�,�-�.�u�5���}�}�Z��.�.�/�/�/rc�l�|j|jj}g}|jD]v}t |t ��r)|�|�|�����@t |jt��sJ�|�|j���wt|j |��}||_|S)z.Converts a RuleNode parse tree to a lark Tree.) rarTr%rUr rR�appendrsrrrrg)r&� rule_node� orig_rulerU�childrs rrszParser._to_treems����O�I�N�$8�9� ����'� ,� ,�E��%��*�*� ,����� � �e� 4� 4�5�5�5�5�!�%�*�e�4�4�4�4�4����� �+�+�+�+� ��!�8� ,� ,������r) r?r@rArBr r_rxrsrCrDs@rrZrZOsk���������.�.�.�.�.� ��� 0� 0� 0� � � � � � � rrZc� �t|t��rNtd|dzzt|jj��z��|jD]}t||dz���dStd|dzzt|j��z��dS)Nr)rrp) r rR�printr,rTr"rU� print_parser)�node�indentr}s rr�r�|s����$��!�!�0� �c�V�a�Z� �3�t�y�}�#5�#5�5�6�6�6��]� +� +�E� ��v��z� *� *� *� *� +� +� �c�V�a�Z� �3�t�v�;�;�.�/�/�/�/�/rc ��tt��}tt��}t|��D]�\}}|j���D]�\}}t ||��r�|D]�}|||f�|��|j|||fvs#|j |||f|jj kr5t|t|��g|j ���|||f|j<������tdt|��dz��D�]l} tt|��| z dz��D�]E}t|dz|| z��D�]*} || dz f} | || zdz f} tj|| || ��D]�\} }|j�| j|jfg��D]�}|||| zdz f�|��|| | j}|| |j}|j |j z|j z}|j|||| zdz fvs$||||| zdz f|jj kr*t|||g|���|||| zdz f|j<�Ȍ���,��G��n||fS)z*Parses sentence 's' using CNF grammar 'g'.�r$rrp)r�set�dict� enumerate�terminal_rules�itemsr�addr"r$rRr�rangerr� itertools�product�nonterminal_rules�get)r�grvrw�i�w�terminalrIrT�l�p�span1�span2�r1�r2�r1_tree�r2_tree�rule_total_weights rrqrq�s��� �� � �E� �� � �E��!� � �]�]���1� �/�5�5�7�7� ]� ]�O�H�e��X�q�!�!� ]�!�]�]�D��1�a�&�M�%�%�d�+�+�+����q�!�f� �5�5�� �e�Q��F�m�D�H�&=�&D�D�D�2:�4�!�A�$�$��PT�P[�2\�2\�2\��q�!�f� �d�h�/���  ]��1�c�!�f�f�q�j� !� !�{�{���s�1�v�v��z�A�~�&�&� {� {�A��1�q�5�!�a�%�(�(� {� {���A��E� ���A��E�A�I���'�/��e� �e�E�l�K�K�{�{�F�B�� !� 3� 7� 7�����8H�"� M� M�{�{���q�!�a�%�!�)�n�-�1�1�$�7�7�7�"'��,�r�v�"6��"'��,�r�v�"6��,0�K�'�.�,H�7�>�,Y�)� �H�E�1�a�!�e�a�i�.�,A�A�A�0�5�!�Q��U�Q�Y��3H���3R�3Y�Y�Y�>F�t�g�W^�M_�hy�>z�>z�>z�E�1�a�!�e�a�i�.�1�$�(�;��{�{� {� {� �%�<�rc�.��eZdZdZ�fd�Zd�Zd�Z�xZS)� CnfWrapperzkCNF wrapper for grammar. Validates that the input grammar is CNF and provides helper data structures. c�6��tt|�����||_|j|_t t ��|_t t ��|_|jD�]"}t|j t��s J|���t|j ��dvrtd���t|j ��dkrLt|j dt��r,|j|j d�|����t|j ��dkrRt#d�|j D����r4|jt%|j ���|����J|���dS)N)rprzCYK doesn't support empty rulesrprrc3�@K�|]}t|t��V��dSr )r rrs rrz&CnfWrapper.__init__.<locals>.<genexpr>�s,����(J�(J�q��A�r�):�):�(J�(J�(J�(J�(J�(Jr)rr�r rcrIr�listr�r�r r"rrrr#rrrzr!r5)r&rcrnr's �rr zCnfWrapper.__init__�sV��� �j�$���(�(�*�*�*��� ��]�� �)�$�/�/���!,�T�!2�!2����� � �A��a�e�R�(�(� +� +�!� +� +�(��1�5�z�z��'�'� �!B�C�C�C��1�5�z�z�Q���:�a�e�A�h��#:�#:���#�A�E�!�H�-�4�4�Q�7�7�7�7��Q�U���q���S�(J�(J�A�E�(J�(J�(J�%J�%J���&�u�Q�U�|�|�4�;�;�A�>�>�>�>��a���u� � rc�"�|j|jkSr )rcr9s rr;zCnfWrapper.__eq__�s���|�u�}�,�,rc�*�t|j��Sr )rNrcr.s rr1zCnfWrapper.__repr__�s���D�L�!�!�!r)r?r@rArBr r;r1rCrDs@rr�r��s`���������  � � � � �$-�-�-�"�"�"�"�"�"�"rr�c�6��eZdZdZ�fd�Zd�ZejZ�xZS)� UnitSkipRulez@A rule that records NTs that were skipped during transformation.c�l��tt|���||||��||_dSr )rr�r � skipped_rules)r&r"r#r�r$r%r's �rr zUnitSkipRule.__init__�s5��� �l�D�!�!�*�*�3��V�U�C�C�C�*����rc�\�t|t|����o|j|jkSr )r rr�r9s rr;zUnitSkipRule.__eq__�s(���%��d���,�,�Z��1C�u�GZ�1Z�Zr) r?r@rArBr r;rr6rCrDs@rr�r��sM�������J�J�+�+�+�+�+�[�[�[��}�H�H�H�H�Hrr�c� �g}t|t��r ||jz }|�|��t|t��r ||jz }t|j|j||j|jz|j���S)Nrf)r r�r�rzr"r#r$r%)� unit_rule� target_ruler�s r�build_unit_skipruler��s����M��)�\�*�*�1���0�0� �����%�%�%��+�|�,�,�3���2�2� � � � �{�� �&�-� �0B�B�)�/� [� [� [�[rc��|jD]>}t|j��dkr$t|jdt��r|cS�?dS)zDReturns a non-terminal unit rule from 'g', or None if there is none.rprN)rIrrr#r r)r�rTs r�get_any_nt_unit_ruler��sJ������� �t�x�=�=�A� � �*�T�X�a�[�"�"=�"=� ��K�K�K�� �4rc����fd�|jD��}�fd�|jD��}|�fd�|D��z }t|��S)zFRemoves 'rule' from 'g' without changing the language produced by 'g'.c� ��g|] }|�k�|�� Sr=r=�rrrTs �rr`z%_remove_unit_rule.<locals>.<listcomp>�s���1�1�1�q�q�D�y�y��y�y�yrc�@��g|]}|j�jdk�|��SrXr8r�s �rr`z%_remove_unit_rule.<locals>.<listcomp>�s+��� 7� 7� 7�!�!�%�4�8�A�;�"6�"6�A�"6�"6�"6rc�0��g|]}t�|����Sr=)r�)r�refrTs �rr`z%_remove_unit_rule.<locals>.<listcomp>�s$���A�A�A�S�%�d�C�0�0�A�A�Ar)rIrF)r�rT� new_rules�refss ` r�_remove_unit_ruler��sf���1�1�1�1�A�G�1�1�1�I� 7� 7� 7� 7�q�w� 7� 7� 7�D� �A�A�A�A�D�A�A�A�A�I� �9� � �rc #�|K�t|j��dzd�d�|jD����z}d|zdz}t |j|jdt |dz��g|j|j���V�tdt|j��d z ��D]G}t t ||z��|j|t ||dzz��gdd ���V��Ht t |t|j��d z z��|jd d �dd ���V�d S) z4Splits a rule whose len(rhs) > 2 into shorter rules.�__�_c3�4K�|]}t|��V��dSr r+rs rrz_split.<locals>.<genexpr>s(����.H�.H�!�s�1�v�v�.H�.H�.H�.H�.H�.Hrz__SP_%sz_%drrprfr�Split�����N) r,r"r-r#rrr$r%r�rr)rT�rule_str� rule_namer�s r�_splitr��s?�����4�8�}�}�t�#�c�h�h�.H�.H�t�x�.H�.H�.H�&H�&H�H�H��X�&��.�I� �t�x�$�(�1�+�r�)�a�-�'8�'8�9�$�+�UY�U_� `� `� `�`�`�`� �1�c�$�(�m�m�a�'� (� (�g�g���2�i�!�m�$�$�t�x��{�B�y�A��E�7J�4K�4K�&L�UV�^e�f�f�f�f�f�f�f� �r�)�s�4�8�}�}�q�0�1�2�2�D�H�R�S�S�M�!�SZ� [� [� [�[�[�[�[�[rc ����d�|jD��}d�|D���g}|jD]Ɋt�j��dkr�td��jD����r|�fd��jD��}|�t �j|�j�j�����|� �fd��� ��D������|������t|��S)z/Applies the TERM rule on 'g' (see top comment).c�P�h|]#}|jD]}t|t���|���$Sr=)r#r r)rrTrs r� <setcomp>z_term.<locals>.<setcomp> s6�� I� I� I�4�D�H� I� I�q� �1�a�8H�8H� I�Q� I� I� I� Irc �r�i|]4}|ttdt|��z��|gdd�����5S)z__T_%sr�Termrf)rrr,)rrs rr]z_term.<locals>.<dictcomp> s@��Z�Z�Z�q�q�$�r�(�S��V�V�+�,�,�q�c�!�6�J�J�J�Z�Z�Zrrpc3�@K�|]}t|t��V��dSr )r rrs rrz_term.<locals>.<genexpr>s,����$H�$H�!�Z��1�%5�%5�$H�$H�$H�$H�$H�$Hrc�X��g|]&}t|t��r �|jn|��'Sr=)r rr")rr�t_ruless �rr`z_term.<locals>.<listcomp>s2���S�S�S�Q��A�q�)9�)9�@�w�q�z�~�~�q�S�S�Srrfc3�4�K�|]\}}|�jv�|V��dSr )r#)r�k�vrTs �rrz_term.<locals>.<genexpr>s-�����L�L�4�1�a�a�4�8�m�m�Q�m�m�m�m�L�Lr) rIrrr#�anyrzrr"r$r%�extendr�rF)r��all_tr��new_rhsrTr�s @@r�_termr� s���� I� I�1�7� I� I� I�E�Z�Z�TY�Z�Z�Z�G��I���#�#�� �t�x�=�=�1� � ��$H�$H�t�x�$H�$H�$H�!H�!H� �S�S�S�S�$�(�S�S�S�G� � � �T�$�(�G�D�K�t�z�Z�Z�Z� [� [� [� � � �L�L�L�L�7�=�=�?�?�L�L�L� L� L� L� L� � � �T� "� "� "� "� �9� � �rc��g}|jD]B}t|j��dkr|t|��z }�-|�|���Ct |��S)z.Applies the BIN rule to 'g' (see top comment).r)rIrrr#r�rzrF)r�r�rTs r�_binr�sb���I���#�#�� �t�x�=�=�1� � � ����� %�I�I� � � �T� "� "� "� "� �9� � �rc�j�t|��}|r!t||��}t|��}|�!|S)z/Applies the UNIT rule to 'g' (see top comment).)r�r�)r�� nt_unit_rules r�_unitr�#sD��'��*�*�L� �/� �a�� .� .��+�A�.�.� � �/� �Hrc�r�ttt|������}t|��S)z>Creates a CNF grammar from a general context-free grammar 'g'.)r�r�r�r�)r�s rrbrb,s'�� �d�5��8�8�n�n���A� �a�=�=�rc �B�|s#tt||||���||���S||djz }tt||djg||���t |dj||dd�||dj|dj��g|���S)Nrfr�rrp)rRrr$r"�unroll_unit_skipruler%)r"�orig_rhsr�rUr$r%s rr�r�2s��� � ���S�(�6��G�G�G��Z`�a�a�a�a��-��*�1�1��� ��}�Q�'�+�,�V�5� I� I� I�$�]�1�%5�%9�8� -�a�b�b� 1�8� -�a� 0� 7��q�9I�9O�Q�Q�L�� ��� rc�l�t|t��r|S|jjj�d��r |jdSg}tt|j��D][}t|t��r/|jjj�d��r ||jz }�F|� |���\t|jt��rFt|jj|jj |jj||jj|jj��St|j|��S)zDReverts a parse tree (RuleNode) to its original non-CNF form (Node).�__T_r�__SP_)r rrTr"r� startswithrU�maprtrRrzr�r�r#r�r$r%)r�rUr}s rrtrt?s���$������ � �y�}��$�$�V�,�,�1��}�Q�������T�]�3�3� '� '�E��%��*�*� '�u�z�~�/B�/M�/M�g�/V�/V� '��E�N�*�������&�&�&�&� �d�i�� .� .� 1�'�� � �t�y�}�$(�I�$;�X�$(�I�$4�d�i�o�G�G� G��D�I�x�0�0� 0rrX)#rB� collectionsrr�� exceptionsr�lexerr�treerrcrrr rr rrrFrRrZr�rqr�r�r�r�r�r�r�r�r�rbr�rtr=rr�<module>r�sS��*�*�$�#�#�#�#�#�����#�#�#�#�#�#�������������>�>�>�>�>�>�>�>�>�>���� #�#�#�#�#�#�#�#�8 � � � � � � � �" f� f� f� f� f� f� f� f�*�*�*�*�*�*�*�*�Z0�0�0�0�"�"�"�f"�"�"�"�"�"�"�"�> � � � � �4� � � �[�[�[�������\�\�\� � � ���� � � ���� � � �1�1�1�1�1r
Memory