�
i\�g� � �� � U d dl mZ d dlZd dlZd dlZd dlZd dlmZmZm Z d dl
Z
d dl Z d dl m
Z
ej � � Zi Zded< daded< ej dd �� � Zdd
�Zdd�Zdd�ZdS )� )� annotationsN)�Any�Callable�Sequence)�Encodingzdict[str, Encoding]� ENCODINGSz.dict[str, Callable[[], dict[str, Any]]] | None�ENCODING_CONSTRUCTORS�return�
Sequence[str]c � � g } t j t j t j dz � � }|D ]\ }}}| � |� � � | S )N�.)�pkgutil� iter_modules� tiktoken_ext�__path__�__name__�append)�mods� plugin_mods�_�mod_names �a/home/asafur/pinokio/api/open-webui.git/app/env/lib/python3.11/site-packages/tiktoken/registry.py�_available_plugin_modulesr sV � � �D��&�|�'<�l�>S�VY�>Y�Z�Z�K�%� � ���8�Q� � � �H� � � � � �K� �Nonec �� � t 5 t � d d d � � d S i a t � � D ]�} t j | � � } |j }n&# t
$ r}t
d| � d�� � |�d }~ww xY w|� � � D ]-\ }}|t v rt
d|� d| � �� � �|t |< �.��n# t $ r d a� w xY w d d d � � d S # 1 swxY w Y d S )Nztiktoken plugin z& does not define ENCODING_CONSTRUCTORSzDuplicate encoding name z in tiktoken plugin ) �_lockr r � importlib�
import_module�AttributeError�
ValueError�items� Exception)r �mod� constructors�e�enc_name� constructors r �_find_constructorsr) ! s� � � � � � � ,� �� � � � � � � � !#� � � 5� 7� 7�
B�
B�� �-�h�7�7�� �#&�#<�L�L��%� � � �$�[�8�[�[�[�� � � ����� ���� .:�-?�-?�-A�-A� B� B�)�H�k� �#8�8�8� (� _�x� _� _�U]� _� _� � � � 7B�)�(�3�3� B�
B�� � � � �$(� !� � ����
B�
� � � � � � � � � � � ���� � � � � � sR � C�C�$B8�A�
B8�
A1�A,�,A1�1AB8�7C�8C�C�C� C�
encoding_name�strr c
�" � t | t � � s t dt | � � � �� � �| t v r
t | S t
5 | t v rt | cd d d � � S t �t � � t �J �| t vr/t d| � dt � � � dt j
� d�� � �t | }t di |� � ��}|t | < |cd d d � � S # 1 swxY w Y d S )Nz'Expected a string in get_encoding, got zUnknown encoding z.
Plugins found: z
tiktoken version: z (are you on latest?)� ) �
isinstancer+ r! �typer r r r) r �tiktoken� __version__r )r* r( �encs r � get_encodingr3 ? s� � � �m�S� )� )� Z��X�4�
�CV�CV�X�X�Y�Y�Y�� �!�!�� �'�'� � � � �I� %� %� �]�+�� � � � � � � � !� (� � � � �(�4�4�4� � 5� 5� 5� �Q�M� Q� Q�";�"=�"=�Q� Q�%-�%9�Q� Q� Q�� �
� ,�M�:� ��'�'��� � �'�'��#&� �-� ��%� � � � � � � � � � � ���� � � � � � s �D�5BD�D� D� list[str]c � � t 5 t �t � � t �J �t t � � cd d d � � S # 1 swxY w Y d S )N)r r r) �listr- r r �list_encoding_namesr7 [ s� � � � +� +� � (� � � � �(�4�4�4��)�*�*� +� +� +� +� +� +� +� +� +� +� +� +���� +� +� +� +� +� +s �2A�A �A )r
r )r
r )r* r+ r
r )r
r4 )�
__future__r � functoolsr r � threading�typingr r r r r0 �
tiktoken.corer �RLockr r �__annotations__r � lru_cacher r) r3 r7 r- r r �<module>r@ s( �� "� "� "� "� "� "� "� � � � � � � � � ���� � � � � *� *� *� *� *� *� *� *� *� *� � � � � ���� "� "� "� "� "� "�� �����!#� � #� #� #� #�HL� � L� L� L� L� ��
�
�
� ��
�� � � �<� � � �8+� +� +� +� +� +r