� A��g~)��P�dZddlZddlmZddlmZddlmZddlm Z m Z m Z m Z m Z mZmZmZmZddlmZmZddlmZdd lmZmZdd lmZdd lmZmZdd lm Z m!Z!dd l"m#Z#ed��Z$ede���Z%de e$de e$ge%fde e$fd�Z&Gd�de��Z'dS)zo Ensemble retriever that ensemble the results of multiple retrievers by using weighted Reciprocal Rank Fusion �N)� defaultdict)�Hashable)�chain) �Any�Callable�Dict�Iterable�Iterator�List�Optional�TypeVar�cast)�#AsyncCallbackManagerForRetrieverRun�CallbackManagerForRetrieverRun)�Document)� BaseRetriever� RetrieverLike)�RunnableConfig)� ensure_config� patch_config)�ConfigurableFieldSpec�get_unique_config_specs)�model_validator�T�H)�bound�iterable�key�returnc#�K�t��}|D]*}||��x}|vr|�|��|V��+dS)a Yield unique elements of an iterable based on a key function. Args: iterable: The iterable to filter. key: A function that returns a hashable key for each element. Yields: Unique elements of the iterable based on the key function. N)�set�add)rr�seen�e�ks �m/home/asafur/pinokio/api/open-webui.git/app/env/lib/python3.11/site-packages/langchain/retrievers/ensemble.py� unique_by_keyr'(sZ���� �5�5�D� ������Q���K�A�� $� $� �H�H�Q�K�K�K��G�G�G����c �,�eZdZUdZeeed<eeed<dZe ed<dZ e e ed<e deefd ���Zed � ��ed ee efdefd �����Z dde de ededeefd�Z dde de ededeefd�Zde dedeefd�Zde dedeefd�Zdd�de dede edeefd�Zdd�de dede edeefd�Zdeeedeefd�ZdS)�EnsembleRetrieveraeRetriever that ensembles the multiple retrievers. It uses a rank fusion. Args: retrievers: A list of retrievers to ensemble. weights: A list of weights corresponding to the retrievers. Defaults to equal weighting for all retrievers. c: A constant added to the rank, controlling the balance between the importance of high-ranked items and the consideration given to lower-ranked items. Default is 60. id_key: The key in the document's metadata used to determine unique documents. If not specified, page_content is used. � retrievers�weights�<�cN�id_keyrc�>�td�|jD����S)z+List configurable fields for this runnable.c3�.K�|]}|jD]}|V���dS�N)� config_specs)�.0� retriever�specs r&� <genexpr>z1EnsembleRetriever.config_specs.<locals>.<genexpr>QsJ����' �' ��i�>T�' �' �6:�D�' �' �' �' �' �' �' r()rr+)�selfs r&r3zEnsembleRetriever.config_specsNs4��'�' �' �"&�/�' �' �' � � � r(�before)�mode�valuesc�r�|�d��s!t|d��}d|z g|z|d<|S)Nr,r+�)�get�len)�clsr;� n_retrieverss r&� set_weightszEnsembleRetriever.set_weightsUsG���z�z�)�$�$� B��v�l�3�4�4�L�!"�\�!1� 2�\� A�F�9� �� r(�input�config�kwargsc �*�ddlm}t|��}|�|�d��d|�dd��|�dg��|j|�di��|j���}|jd|fd |�d ��p|���i|��} |� |||� ��}|j |fi|��|S#t$r}|� |��|�d}~wwxYw) Nr)�CallbackManager� callbacks�verboseF�tags�metadata�rI�inheritable_tags� local_tags�inheritable_metadata�local_metadata�name�run_name�� run_managerrD) �langchain_core.callbacksrGr� configurer>rJrK�on_retriever_start�get_name� rank_fusion�on_retriever_end� Exception�on_retriever_error) r8rCrDrErG�callback_managerrT�resultr$s r&�invokezEnsembleRetriever.invoke]se�� =�<�<�<�<�<��v�&�&��*�4�4� �J�J�{� #� #� ��J�J�y�%�0�0�#�Z�Z���3�3��y�!'���J��!;�!;��=�5� � ��:�&�9� � � � ����J�'�'�:�4�=�=�?�?� ��  � � � ��%�%�e��V�%�T�T�F� )�K� (�� � �� � � ��M��� � � � � *� *�1� -� -� -��G����� ���s�C,�, D�6D � Dc ��^K�ddlm}t|��}|�|�d��d|�dd��|�dg��|j|�di��|j���}|jd|fd |�d ��p|���i|���d{V��} |� |||� ���d{V��}|j |fi|���d{V��|S#t$r"}|� |���d{V��|�d}~wwxYw) Nr)�AsyncCallbackManagerrHrIFrJrKrLrQrRrS) rUrarrVr>rJrKrWrX� arank_fusionrZr[r\) r8rCrDrErar]rTr^r$s r&�ainvokezEnsembleRetriever.ainvoke~s����� B�A�A�A�A�A��v�&�&��/�9�9� �J�J�{� #� #� ��J�J�y�%�0�0�#�Z�Z���3�3��y�!'���J��!;�!;��=�:� � ��@�,�?� � � � ����J�'�'�:�4�=�=�?�?� ��  � � � � � � � � � ��,�,��;�v�-���������F�/�+�.������� � � � � � � ��M��� � � ��0�0��3�3� 3� 3� 3� 3� 3� 3� 3��G����� ���s� D� D,� D'�'D,�queryrTc�2�|�||��}|S)z� Get the relevant documents for a given query. Args: query: The query to search for. Returns: A list of reranked documents. )rY�r8rdrT�fused_documentss r&�_get_relevant_documentsz)EnsembleRetriever._get_relevant_documents�s��"�*�*�5�+�>�>���r(c��BK�|�||���d{V��}|S)z� Asynchronously get the relevant documents for a given query. Args: query: The query to search for. Returns: A list of reranked documents. N)rbrfs r&�_aget_relevant_documentsz*EnsembleRetriever._aget_relevant_documents�s5����"!%� 1� 1�%�� E� E�E�E�E�E�E�E���r()rDc���������fd�t|j��D��}tt|����D]}d�||D��||<�|�|��}|S)z� Retrieve the results of the retrievers and use rank_fusion_func to get the final result. Args: query: The query to search for. Returns: A list of reranked documents. c ���g|]D\}}|��t���d|dz������������ES�� retriever_r=)�tag)rH)r_r� get_child�r4�ir5rDrdrTs ���r&� <listcomp>z1EnsembleRetriever.rank_fusion.<locals>.<listcomp>�sx��� � � ���9� � � ����k�&;�&;�@T�Q�QR�U�@T�@T�&;�&U�&U���� � � � � r(c��g|]<}t|t��r#ttt|�����n|��=S�)� page_content)� isinstance�strrr�r4�docs r&rsz1EnsembleRetriever.rank_fusion.<locals>.<listcomp>�sQ��!�!�!��:D�C��9M�9M�V��d�3��n�n�5�5�5�5�SV�!�!�!r()� enumerater+�ranger?�weighted_reciprocal_rank�r8rdrTrD�retriever_docsrrrgs ``` r&rYzEnsembleRetriever.rank_fusion�s������& � � � � � �!*�$�/� :� :� � � ���s�>�*�*�+�+� � �A�!�!�)�!�,�!�!�!�N�1� � � �7�7��G�G���r(c�� ���K�tj���fd�t|j��D����d{V��}t t |����D]}d�||D��||<�|�|��}|S)z� Asynchronously retrieve the results of the retrievers and use rank_fusion_func to get the final result. Args: query: The query to search for. Returns: A list of reranked documents. c ���g|]D\}}|��t���d|dz������������ESrm)rcrrprqs ���r&rsz2EnsembleRetriever.arank_fusion.<locals>.<listcomp>sx��� � � �!�A�y��!�!�� ��"-�"7�"7�<P��Q��<P�<P�"7�"Q�"Q������ � � r(Nc�\�g|])}t|t��st|���n|��*Sru)rwrrys r&rsz2EnsembleRetriever.arank_fusion.<locals>.<listcomp>sH��!�!�!��3=�S�(�2K�2K�T��c�*�*�*�*�QT�!�!�!r()�asyncio�gatherr{r+r|r?r}r~s ``` r&rbzEnsembleRetriever.arank_fusion�s��������& '�~� � � � � � �%.�d�o�$>�$>� � � � �  �  �  �  �  �  ���s�>�*�*�+�+� � �A�!�!�)�!�,�!�!�!�N�1� � � �7�7��G�G���r(� doc_listsc�����t|��t�j��krtd���tt���t |�j��D]U\}}t |d���D]>\}}��j�|jn|j �jxx||�j zz z cc<�?�Vtj |��}tt|�fd���d��fd����}|S) a� Perform weighted Reciprocal Rank Fusion on multiple rank lists. You can find more details about RRF here: https://plg.uwaterloo.ca/~gvcormac/cormacksigir09-rrf.pdf Args: doc_lists: A list of rank lists, where each rank list contains unique items. Returns: list: The final aggregated list of items sorted by their weighted RRF scores in descending order. z<Number of rank lists must be equal to the number of weights.r=)�startNc�D���j�|jn|j�jSr2�r/rvrK)rzr8s �r&�<lambda>z<EnsembleRetriever.weighted_reciprocal_rank.<locals>.<lambda>Fs'����{�*��$�$���d�k�2�r(Tc�P����j�|jn|j�jSr2r�)rz� rrf_scorer8s ��r&r�z<EnsembleRetriever.weighted_reciprocal_rank.<locals>.<lambda>Ms'���I�$(�K�$7�� � �S�\�$�+�=V��r()�reverser)r?r,� ValueErrorr�float�zipr{r/rvrKr.r� from_iterable�sortedr') r8r��doc_list�weight�rankrz�all_docs� sorted_docsr�s ` @r&r}z*EnsembleRetriever.weighted_reciprocal_rank sV���� �y�>�>�S���.�.� .� .��N��� � '2�%�&8�&8� � #�I�t�|� <� <� .� .� �H�f�&�x�q�9�9�9� .� .� ��c�� �;�.��(�(� �\�$�+�6� ��� �t�d�f�}�-� .����� .��&�y�1�1��� ������ � �������  �  �  � ��r(r2) �__name__� __module__� __qualname__�__doc__r r�__annotations__r�r.�intr/r rx�propertyrr3r� classmethodrrrBrrr_rcrrhrrjrYrbr}�r(r&r*r*9s�������� � ��]�#�#�#�#� �%�[���� �A�s�K�K�K� �F�H�S�M� � � � � �d�#8�9� � � ��X� � �_�(�#�#�#����c�3�h���C�����[�$�#��>B�����"*�>�":��MP�� �h������D>B�!�!��!�"*�>�":�!�MP�!� �h��!�!�!�!�F���4� � �h�� ����*���9� � �h�� ����4,0� '�'�'��'�4�'� ��(� '� �h�� '�'�'�'�\,0� *�*�*��*�9�*� ��(� *� �h�� *�*�*�*�X1��d�8�n�-�1� �h��1�1�1�1�1�1r(r*)(r�r�� collectionsr�collections.abcr� itertoolsr�typingrrrr r r r r rrUrr�langchain_core.documentsr�langchain_core.retrieversrr�langchain_core.runnablesr�langchain_core.runnables.configrr�langchain_core.runnables.utilsrr�pydanticrrrr'r*r�r(r&�<module>r�s���� ����#�#�#�#�#�#�$�$�$�$�$�$������� � � � � � � � � � � � � � � � � � � � � � ���������.�-�-�-�-�-�B�B�B�B�B�B�B�B�3�3�3�3�3�3�G�G�G�G�G�G�G�G���������%�$�$�$�$�$� �G�C�L�L�� �G�C�x� � � ���H�Q�K��h��s�A�v�.>��8�A�;�����"X�X�X�X�X� �X�X�X�X�Xr(
Memory