Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Anthropic Models to Cache Prompt #3775

Merged

Conversation

ColeMurray
Copy link
Contributor

Short description of the problem this fixes or functionality that this introduces. This may be used for the CHANGELOG
Adds support for Anthropic models with 'anthropic/' prefix in cache prompting.


Give a summary of what the PR does, explaining any non-trivial design decisions
This pull request addresses an issue where Anthropic models with the 'anthropic/' prefix were not being correctly identified for cache prompting support. The changes include:

  1. Added two new entries to the cache_prompting_supported_models list:

    • 'anthropic/claude-3-5-sonnet-20240620'
    • 'anthropic/claude-3-haiku-20240307'
  2. Retained the existing entries without the 'anthropic/' prefix to maintain backwards compatibility:

    • 'claude-3-5-sonnet-20240620'
    • 'claude-3-haiku-20240307'

This change ensures that the cache prompting feature correctly supports Anthropic models regardless of whether they are referenced with or without the 'anthropic/' prefix. This dual-entry approach allows for flexibility in model naming conventions while maintaining support for existing configurations.

The decision to keep both prefixed and non-prefixed versions was made to:

  1. Support users who may be using the models with the 'anthropic/' prefix in their configurations.
  2. Maintain compatibility for users who are already using the non-prefixed versions.

Link of any specific issues this addresses
Fixes #3774: Incorrect identification of Anthropic models in prompt cache supported models

Copy link
Collaborator

@enyst enyst left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fix!

This list we have will be a pain to keep updated... 😅 I hope liteLLM will support this feature in a more generic way in the future. For now it's good to have it working.

@enyst enyst enabled auto-merge (squash) September 8, 2024 12:11
@tobitege
Copy link
Collaborator

tobitege commented Sep 8, 2024

This shouldn't be necessary, i.e. the model name should be compared to be in the array entries' name (not equality), so we don't need to add prefixed versions.
Please apply a change to is_caching_prompt_active in llm.py, e.g.

    return (
        self.config.caching_prompt is True
        and any(model in self.config.model for model in cache_prompting_supported_models)
    )

Copy link
Collaborator

@tobitege tobitege left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please see my comment above.

Copy link
Collaborator

@tobitege tobitege left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM
Thank you for your contribution! 👍

@tobitege tobitege enabled auto-merge (squash) September 8, 2024 19:30
@tobitege tobitege merged commit dadada1 into All-Hands-AI:main Sep 8, 2024
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Bug]: Cache Prompting Supported Models Incorrectly Identifies Anthropic Models
3 participants