Evidence
Corrections and evaluations published by providers.dev. Evidence older than 90 days is marked stale. To propose a correction or ask for an evaluation, see Contributing.
- Corrected entries
- 24
- Added providers
- 4
- Cache evaluations
- 86
- Measured limits
- 31
Corrections
capabilities.cache.implicit: true. models.dev: Not provided.
OpenRouter documents that prompt caching with DeepSeek is automatic. No lifetime is documented, so none is set.
- documented Source 2026-11-13
capabilities.cache.implicit: true. models.dev: Not provided.
OpenRouter lists a cached-input price for this model and has no caching section for the lab. In a live check an immediate re-send was served from the cache.
- measured Source 2026-11-04
1 entries
capabilities.cache.implicit: true. models.dev: Not provided.
OpenRouter lists a cached-input price for this model and has no caching section for the lab. In a live check an immediate re-send was served from the cache.
- measured Source 2026-11-04
capabilities.cache.implicit: true. models.dev: Not provided.
OpenRouter documents that prompt caching with Moonshot AI is automatic. No lifetime is documented, so none is set.
- documented Source 2026-11-13
capabilities.cache.implicit: true. models.dev: Not provided.
OpenRouter documents that prompt caching with Z.AI is automatic. No lifetime is documented, so none is set.
- documented Source 2026-11-13
GET /api/v1/corrections?offset=0
Added providers
| Provider | Access | Entries |
|---|---|---|
| Claude Code | Reached with a Claude Pro or Max login (the Claude Code sign-in), not with an Anthropic API key. | 3 |
| Devin | Reached with a Devin subscription. Devin serves other labs' models under its own model ids. | 17 |
| Google Antigravity | Reached with a Google Antigravity login, not with a Gemini API key. | 1 |
| Muse Code | Reached with the Muse Code API key that a Muse Code subscription applies to, not with a pay-as-you-go Meta Model API key. | 3 |
Cache evaluations
Observed hit percentages by exact age, for the client and request setting shown.
86 cache evaluations
| Entry | Measured through | Served from cache, % by age | Date | Evidence |
|---|---|---|---|---|
Muse Wondermeta-subscription/muse-wonder-1.0 | Muse Code subscription key on Meta's Model API A coding agent, through CLIProxyAPI · reasoning=low | Short100100100100100100100100 | 2026-11-07 | Raw result |
GPT-6.1 Solopenai-codex/gpt-6.1-sol | ChatGPT plan through the Codex login OpenAI's Codex CLI · reasoning=low | Short100100671006710033670 | 2026-11-07 | Raw result |
GPT-6.1 Solopenai-codex/gpt-6.1-sol | ChatGPT plan through the Codex login A coding agent, through CLIProxyAPI · reasoning=low | Short1001001001006767100330 | 2026-11-07 | Raw result |
GPT-6.1 Terraopenai-codex/gpt-6.1-terra | ChatGPT plan through the Codex login OpenAI's Codex CLI · reasoning=low | Short1001001001006767000 | 2026-11-07 | Raw result |
GPT-6.1 Terraopenai-codex/gpt-6.1-terra | ChatGPT plan through the Codex login A coding agent, through CLIProxyAPI · reasoning=low | Short100100100673333000 | 2026-11-07 | Raw result |
Muse Wondermeta-subscription/muse-wonder-1.0 | Muse Code subscription key on Meta's Model API Requests sent through CLIProxyAPI · | Short1001001001006733067 | 2026-11-06 | Raw result |
GPT-6.1 Solopenai-codex/gpt-6.1-sol | ChatGPT plan through the Codex login Requests sent through CLIProxyAPI · | Short100100676710010067330 | 2026-11-06 | Raw result |
GPT-6.1 Terraopenai-codex/gpt-6.1-terra | ChatGPT plan through the Codex login Requests sent through CLIProxyAPI · | Short1003333 | 2026-11-06 | Raw result |
Claude Haiku 5.5openrouter/anthropic/claude-haiku-5.5 | OpenRouter API key Requests sent through CLIProxyAPI · format=messages | Short10010010000 | 2026-11-06 | Raw result |
Claude Haiku 5.5openrouter/anthropic/claude-haiku-5.5 | OpenRouter API key Requests sent through CLIProxyAPI · format=messages, ttl=1h | Long 1h100100100 | 2026-11-06 | Raw result |
Claude Haiku 5.5openrouter/anthropic/claude-haiku-5.5 | OpenRouter API key Requests sent straight to the provider · cache_control=top-level, session_id=true | Short10010010000 | 2026-11-06 | Raw result |
Gemini 4 Argonopenrouter/google/gemini-4-argon | OpenRouter API key Requests sent through CLIProxyAPI · prefix_tokens=8000 | Short1001005000 | 2026-11-06 | Raw result |
Kimi K3.1 Proopenrouter/moonshotai/kimi-k3.1-pro | OpenRouter API key Requests sent through CLIProxyAPI · | Short1001005000 | 2026-11-06 | Raw result |
Claude Opus 5.5anthropic/claude-opus-5-5 | Anthropic API key Requests sent through CLIProxyAPI · format=messages, ttl=1h, beta=extended-cache-ttl-2025-04-11 | Long 1h10010067 | 2026-11-04 | Raw result |
Claude Opus 5.5anthropic/claude-opus-5-5 | Anthropic API key Requests sent through CLIProxyAPI · format=messages, ttl=1h | Long 1h100100100 | 2026-11-04 | Raw result |
Gemini 3.8 Flashantigravity/gemini-3.8-flash-high | Google Antigravity login Requests sent through CLIProxyAPI · prefix_tokens=5000 | Short10010033676700 | 2026-11-04 | Raw result |
Claude Opus 5.5devin/claude-opus-5-5 | Devin subscription Requests sent through CLIProxyAPI · | Short10010010010010010010010010010067100100100100 | 2026-11-04 | Raw result |
DeepSeek V4 Prodevin/deepseek-v4-pro | Devin subscription Requests sent through CLIProxyAPI · | Short10001003300 | 2026-11-04 | Raw result |
Gemini 4 Argondevin/gemini-4-argon | Devin subscription Requests sent through CLIProxyAPI · prefix_tokens=5000 | Short10010067336700 | 2026-11-04 | Raw result |
GLM-5.3devin/glm-5-3 | Devin subscription Requests sent through CLIProxyAPI · | Short1006733100100100333306700 | 2026-11-04 | Raw result |
GPT-6.1 Soldevin/gpt-6-1-sol | Devin subscription Requests sent through CLIProxyAPI · | Short10010010010010010010010010000 | 2026-11-04 | Raw result |
Grok 4.7devin/grok-4-7 | Devin subscription Requests sent through CLIProxyAPI · | Short1006733336700 | 2026-11-04 | Raw result |
Kimi K3devin/kimi-k3 | Devin subscription Requests sent through CLIProxyAPI · | Short0 | 2026-11-04 | Raw result |
swe-2devin/swe-2 | Devin subscription Requests sent through CLIProxyAPI · | Short0 | 2026-11-04 | Raw result |
Muse Spark 1.3meta-subscription/muse-spark-1.3 | Muse Code subscription key on Meta's Model API Requests sent through CLIProxyAPI · | Short100100100100671001001006700 | 2026-11-04 | Raw result |
Claude Opus 5.5openrouter/anthropic/claude-opus-5.5 | OpenRouter API key Requests sent straight to the provider · cache_control=top-level, session_id=true, ttl=1h | Long 1h100100100 | 2026-11-04 | Raw result |
DeepSeek V4 Proopenrouter/deepseek/deepseek-v4-pro | OpenRouter API key Requests sent straight to the provider · | Short1001006710010010010010067671001001006767 | 2026-11-04 | Raw result |
Gemini 3.8 Flashopenrouter/google/gemini-3.8-flash | OpenRouter API key Requests sent straight to the provider · | Short10000 | 2026-11-04 | Raw result |
Gemini 4 Argonopenrouter/google/gemini-4-argon | OpenRouter API key Requests sent straight to the provider · | Short10010010000 | 2026-11-04 | Raw result |
Muse Wonderopenrouter/meta/muse-wonder | OpenRouter API key Requests sent straight to the provider · | Short1001006710010010010010000 | 2026-11-04 | Raw result |
Mistral Large 4openrouter/mistralai/mistral-large-4-0 | OpenRouter API key Requests sent straight to the provider · | Short1001001001006700 | 2026-11-04 | Raw result |
Kimi K3.1 Proopenrouter/moonshotai/kimi-k3.1-pro | OpenRouter API key Requests sent straight to the provider · | Short1006767671006710010033673300 | 2026-11-04 | Raw result |
GPT-6.1 Solopenrouter/openai/gpt-6.1-sol | OpenRouter API key Requests sent straight to the provider · | Short10010010010010010010010010000 | 2026-11-04 | Raw result |
GPT-6.1 Terraopenrouter/openai/gpt-6.1-terra | OpenRouter API key Requests sent straight to the provider · | Short10010010010010010010010010000 | 2026-11-04 | Raw result |
Grok 4.7openrouter/x-ai/grok-4.7 | OpenRouter API key Requests sent straight to the provider · | Short10010033336700 | 2026-11-04 | Raw result |
GLM-5.3openrouter/z-ai/glm-5.3 | OpenRouter API key Requests sent straight to the provider · | Short10010067336733336733336703300 | 2026-11-04 | Raw result |
Measured limits
All limits in tokens.
| Entry | Date | Measured through | Context | Input accepted | Input refused | Output limit | Requested output accepted | Evidence |
|---|---|---|---|---|---|---|---|---|
Claude Haiku 5.5devin/claude-haiku-5-5 | 2026-11-16 | Devin subscription Requests sent through CLIProxyAPI | Unknown | 999,072 | 1,004,446 | Unknown | Unknown | Raw result |
Claude Opus 5.5devin/claude-opus-5-5 | 2026-11-16 | Devin subscription Requests sent through CLIProxyAPI | Unknown | 999,072 | 1,004,446 | Unknown | Unknown | Raw result |
Claude Sonnet 5.5devin/claude-sonnet-5-5 | 2026-11-16 | Devin subscription Requests sent through CLIProxyAPI | Unknown | 999,072 | 1,004,446 | Unknown | Unknown | Raw result |
deepseek-v4-1-flash-1017devin/deepseek-v4-1-flash-1017 | 2026-11-16 | Devin subscription Requests sent through CLIProxyAPI | Unknown | 1,047,489 | 1,052,864 | Unknown | Unknown | Raw result |
DeepSeek V4 Prodevin/deepseek-v4-pro | 2026-11-16 | Devin subscription Requests sent through CLIProxyAPI | Unknown | 1,047,542 | 1,052,917 | Unknown | Unknown | Raw result |
Gemini 4 Argondevin/gemini-4-argon | 2026-11-16 | Devin subscription Requests sent through CLIProxyAPI | Unknown | 1,047,490 | 1,052,865 | Unknown | Unknown | Raw result |
GLM-5.3devin/glm-5-3 | 2026-11-16 | Devin subscription Requests sent through CLIProxyAPI | Unknown | 1,047,470 | 1,052,845 | Unknown | Unknown | Raw result |
GPT-6.1 Terradevin/gpt-6-1-terra | 2026-11-16 | Devin subscription Requests sent through CLIProxyAPI | Unknown | 911,390 | 923,890 | Unknown | Unknown | Raw result |
GPT-6 Astradevin/gpt-6-astra | 2026-11-16 | Devin subscription Requests sent through CLIProxyAPI | Unknown | 911,390 | 923,890 | Unknown | Unknown | Raw result |
GPT-6 Lunadevin/gpt-6-luna | 2026-11-16 | Devin subscription Requests sent through CLIProxyAPI | Unknown | 911,390 | 923,890 | Unknown | Unknown | Raw result |
swe-2devin/swe-2 | 2026-11-16 | Devin subscription Requests sent through CLIProxyAPI | Unknown | 300,499 | Unknown | Unknown | Unknown | Raw result |
Muse Embermeta-subscription/muse-ember-1.0 | 2026-11-16 | Muse Code subscription key on Meta's Model API Requests sent through CLIProxyAPI | 1,048,576 | 1,047,604 | Unknown | Unknown | Unknown | Raw result |
Muse Spark 1.3meta-subscription/muse-spark-1.3 | 2026-11-16 | Muse Code subscription key on Meta's Model API Requests sent through CLIProxyAPI | 1,048,576 | 1,047,604 | Unknown | Unknown | Unknown | Raw result |
GPT-6.1 Solopenai-codex/gpt-6.1-sol | 2026-11-13 | ChatGPT plan through the Codex login Requests sent through CLIProxyAPI | Unknown | 915,011 | 922,511 | Unknown | Unknown | Raw result |
GPT-6 Astraopenai-codex/gpt-6-astra | 2026-11-11 | ChatGPT plan through the Codex login Requests sent through CLIProxyAPI | Unknown | 915,011 | 922,511 | Unknown | Unknown | Raw result |
GPT-6 Lunaopenai-codex/gpt-6-luna | 2026-11-11 | ChatGPT plan through the Codex login Requests sent through CLIProxyAPI | Unknown | 915,011 | 922,511 | Unknown | Unknown | Raw result |
GPT-6.1 Soldevin/gpt-6-1-sol | 2026-11-09 | Devin subscription Requests sent through CLIProxyAPI | Unknown | 915,390 | 925,390 | Unknown | Unknown | Raw result |
Claude Haiku 5.5claude-subscription/claude-haiku-5-5 | 2026-11-08 | Claude Pro/Max login (the Claude Code subscription) Requests sent through CLIProxyAPI | 1,000,000 | 998,953 | Unknown | 128,000 | Unknown | Raw result |
Claude Opus 5.5claude-subscription/claude-opus-5-5 | 2026-11-08 | Claude Pro/Max login (the Claude Code subscription) Requests sent through CLIProxyAPI | 1,000,000 | 998,953 | Unknown | 128,000 | Unknown | Raw result |
Claude Sonnet 5.5claude-subscription/claude-sonnet-5-5 | 2026-11-08 | Claude Pro/Max login (the Claude Code subscription) Requests sent through CLIProxyAPI | 1,000,000 | 998,953 | Unknown | 128,000 | Unknown | Raw result |
Muse Wondermeta-subscription/muse-wonder-1.0 | 2026-11-08 | Muse Code subscription key on Meta's Model API Requests sent through CLIProxyAPI | 1,048,576 | 1,047,604 | Unknown | Unknown | 1,000,000 | Raw result |
GPT-6.1 Terraopenai-codex/gpt-6.1-terra | 2026-11-08 | ChatGPT plan through the Codex login Requests sent through CLIProxyAPI | Unknown | 920,870 | 921,886 | Unknown | Unknown | Raw result |
Claude Haiku 5.5openrouter/anthropic/claude-haiku-5.5 | 2026-11-08 | OpenRouter API key Requests sent through CLIProxyAPI | 1,000,000 | Unknown | Unknown | Unknown | Unknown | Raw result |
Claude Opus 5.5openrouter/anthropic/claude-opus-5.5 | 2026-11-08 | OpenRouter API key Requests sent through CLIProxyAPI | 1,000,000 | Unknown | Unknown | Unknown | Unknown | Raw result |
Claude Sonnet 5.5openrouter/anthropic/claude-sonnet-5.5 | 2026-11-08 | OpenRouter API key Requests sent through CLIProxyAPI | 1,000,000 | Unknown | Unknown | Unknown | Unknown | Raw result |
DeepSeek V4.1 Flashopenrouter/deepseek/deepseek-v4.1-flash | 2026-11-08 | OpenRouter API key Requests sent through CLIProxyAPI | 1,048,576 | Unknown | Unknown | Unknown | Unknown | Raw result |
Gemini 4 Argonopenrouter/google/gemini-4-argon | 2026-11-08 | OpenRouter API key Requests sent through CLIProxyAPI | 1,048,576 | Unknown | Unknown | Unknown | Unknown | Raw result |
Muse Wonderopenrouter/meta/muse-wonder | 2026-11-08 | OpenRouter API key Requests sent through CLIProxyAPI | 1,048,576 | Unknown | Unknown | Unknown | Unknown | Raw result |
Kimi K3.1 Proopenrouter/moonshotai/kimi-k3.1-pro | 2026-11-08 | OpenRouter API key Requests sent through CLIProxyAPI | 1,048,576 | Unknown | Unknown | Unknown | Unknown | Raw result |
GPT-6.1 Solopenrouter/openai/gpt-6.1-sol | 2026-11-08 | OpenRouter API key Requests sent through CLIProxyAPI | 1,050,000 | Unknown | Unknown | Unknown | Unknown | Raw result |
GPT-6.1 Terraopenrouter/openai/gpt-6.1-terra | 2026-11-08 | OpenRouter API key Requests sent through CLIProxyAPI | 1,050,000 | Unknown | Unknown | Unknown | Unknown | Raw result |
GET /api/v1/evaluations/limits?offset=0
Dataset issues
- left out Claude Opus versions before 5.5, at every provider: 336 entries. Editorial choice of providers.dev: Claude Opus versions before 5.5 are not listed. 2026-11-16
- left out Claude Sonnet versions before 5.5, at every provider: 208 entries. Editorial choice of providers.dev: Claude Sonnet versions before 5.5 are not listed. 2026-11-16
- left out Claude Haiku versions before 5.5, at every provider: 82 entries. Editorial choice of providers.dev: Claude Haiku versions before 5.5 are not listed. 2026-11-16
- left out Claude Fable versions before 5.2, at every provider: 115 entries. Editorial choice of providers.dev: Claude Fable versions before 5.2 are not listed. 2026-11-16
- left out GPT versions before 6 (GPT and its mini, nano, Pro, Codex, Luna, Sol, Terra, Astra and Live models), at every provider: 1,049 entries. Editorial choice of providers.dev: GPT versions before 6 are not listed. 2026-11-16
- left out Models released in 2025, at every provider: 1,979 entries. Editorial choice of providers.dev: models released in 2025 are not listed. 2026-11-16
- left out Models released before 2025, at every provider: 638 entries. Editorial choice of providers.dev: models released before 2025 are not listed. 2026-11-16