Skip to content

Evidence

Corrections and evaluations published by providers.dev. Evidence older than 90 days is marked stale. To propose a correction or ask for an evaluation, see Contributing.

Corrected entries
24
Added providers
4
Cache evaluations
86
Measured limits
31

Corrections

capabilities.cache.implicit: true. models.dev: Not provided.

OpenRouter documents that prompt caching with DeepSeek is automatic. No lifetime is documented, so none is set.

7 entries

openrouter/deepseek/deepseek-v4-flashopenrouter/deepseek/deepseek-v4-flash-0731openrouter/deepseek/deepseek-v4-flash-vision-expopenrouter/deepseek/deepseek-v4-proopenrouter/deepseek/deepseek-v4-pro-0813openrouter/deepseek/deepseek-v4.1-flashopenrouter/deepseek/deepseek-v4.1-flash-1017

capabilities.cache.implicit: true. models.dev: Not provided.

OpenRouter lists a cached-input price for this model and has no caching section for the lab. In a live check an immediate re-send was served from the cache.

1 entries

openrouter/meta/muse-wonder

capabilities.cache.implicit: true. models.dev: Not provided.

OpenRouter lists a cached-input price for this model and has no caching section for the lab. In a live check an immediate re-send was served from the cache.

1 entries

openrouter/mistralai/mistral-large-4-0

capabilities.cache.implicit: true. models.dev: Not provided.

OpenRouter documents that prompt caching with Moonshot AI is automatic. No lifetime is documented, so none is set.

5 entries

openrouter/moonshotai/kimi-k2.5openrouter/moonshotai/kimi-k2.6openrouter/moonshotai/kimi-k2.7-codeopenrouter/moonshotai/kimi-k3openrouter/moonshotai/kimi-k3.1-pro

capabilities.cache.implicit: true. models.dev: Not provided.

OpenRouter documents that prompt caching with Z.AI is automatic. No lifetime is documented, so none is set.

10 entries

openrouter/z-ai/glm-5openrouter/z-ai/glm-5-turboopenrouter/z-ai/glm-5.1openrouter/z-ai/glm-5.2openrouter/z-ai/glm-5.3openrouter/z-ai/glm-5.3-flashopenrouter/z-ai/glm-5.3-flashxopenrouter/z-ai/glm-5.3-flashx-legalopenrouter/z-ai/glm-5.3-primeopenrouter/z-ai/glm-5v-turbo

GET /api/v1/corrections?offset=0

Added providers

ProviderAccessEntries
Claude CodeReached with a Claude Pro or Max login (the Claude Code sign-in), not with an Anthropic API key.3
DevinReached with a Devin subscription. Devin serves other labs' models under its own model ids.17
Google AntigravityReached with a Google Antigravity login, not with a Gemini API key.1
Muse CodeReached with the Muse Code API key that a Muse Code subscription applies to, not with a pay-as-you-go Meta Model API key.3

Cache evaluations

Observed hit percentages by exact age, for the client and request setting shown.

86 cache evaluations
EntryMeasured throughServed from cache, % by ageDateEvidence
Claude Haiku 5.5
claude-subscription/claude-haiku-5-5
Claude Pro/Max login (the Claude Code subscription)
Requests sent through CLIProxyAPI · format=messages
Short10010010010010010010010010010010010002026-11-16Raw result
Claude Haiku 5.5
claude-subscription/claude-haiku-5-5
Claude Pro/Max login (the Claude Code subscription)
Requests sent through CLIProxyAPI · format=messages, ttl=1h
Long 1h10010010010010010010010010010010010002026-11-16Raw result
Claude Opus 5.5
claude-subscription/claude-opus-5-5
Claude Pro/Max login (the Claude Code subscription)
Requests sent through CLIProxyAPI · format=messages
Short10010010010010010010010010010010010002026-11-16Raw result
Claude Opus 5.5
claude-subscription/claude-opus-5-5
Claude Pro/Max login (the Claude Code subscription)
Requests sent through CLIProxyAPI · format=messages, ttl=1h
Long 1h10010010010010010010010010010010010002026-11-16Raw result
Claude Sonnet 5.5
claude-subscription/claude-sonnet-5-5
Claude Pro/Max login (the Claude Code subscription)
Requests sent through CLIProxyAPI · format=messages
Short10010010010010010010010010010010010002026-11-16Raw result
Claude Sonnet 5.5
claude-subscription/claude-sonnet-5-5
Claude Pro/Max login (the Claude Code subscription)
Requests sent through CLIProxyAPI · format=messages, ttl=1h
Long 1h10010010010010010010010010010010010002026-11-16Raw result
GPT-6.1 Sol
openai/gpt-6.1-sol
OpenAI API key
Requests sent straight to the provider · prompt_cache_options.ttl=12h
Long 12h10010010010010010010010010010010010010010010010010010002026-11-15Raw result
GPT-6.1 Sol
openai/gpt-6.1-sol
OpenAI API key
Requests sent straight to the provider ·
Short100100100100100100100100100002026-11-15Raw result
DeepSeek V4.1 Flash
openrouter/deepseek/deepseek-v4.1-flash
OpenRouter API key
Requests sent through CLIProxyAPI ·
Short10010092676733100100672026-11-12Raw result
Muse Spark 1.3
openrouter/meta/muse-spark-1.3
OpenRouter API key
Requests sent through CLIProxyAPI ·
Short100067673367002026-11-12Raw result
GLM-5.3-Flash
openrouter/z-ai/glm-5.3-flash
OpenRouter API key
Requests sent through CLIProxyAPI ·
Short100546767333367002026-11-12Raw result
GPT-6 Astra
devin/gpt-6-astra
Devin subscription
A coding agent, through CLIProxyAPI · reasoning=low
Short100100100100002026-11-11Raw result
GPT-6 Luna
devin/gpt-6-luna
Devin subscription
A coding agent, through CLIProxyAPI · reasoning=low
Short100100100100002026-11-11Raw result
GPT-6 Astra
openai-codex/gpt-6-astra
ChatGPT plan through the Codex login
A coding agent, through CLIProxyAPI · reasoning=low
Short1001001001001001002026-11-11Raw result
GPT-6 Luna
openai-codex/gpt-6-luna
ChatGPT plan through the Codex login
A coding agent, through CLIProxyAPI · reasoning=low
Short10010067673302026-11-11Raw result
GPT-6 Astra
openai/gpt-6-astra
OpenAI API key
Requests sent through CLIProxyAPI ·
Short100100100100002026-11-11Raw result
GPT-6 Luna
openai/gpt-6-luna
OpenAI API key
Requests sent through CLIProxyAPI ·
Short100100100100002026-11-11Raw result
GPT-6.1 Sol
openai/gpt-6.1-sol
OpenAI API key
Requests sent through CLIProxyAPI ·
Short100100100100002026-11-11Raw result
GPT-6 Astra
openrouter/openai/gpt-6-astra
OpenRouter API key
Requests sent through CLIProxyAPI ·
Short100100100100002026-11-11Raw result
GPT-6 Luna
openrouter/openai/gpt-6-luna
OpenRouter API key
Requests sent through CLIProxyAPI ·
Short100100100100002026-11-11Raw result
deepseek-v4-1-flash-1017
devin/deepseek-v4-1-flash-1017
Devin subscription
A coding agent, through CLIProxyAPI · reasoning=low
Short1001001001001001002026-11-10Raw result
Gemini 4 Argon
devin/gemini-4-argon
Devin subscription
A coding agent, through CLIProxyAPI · reasoning=low
Short1006000002026-11-10Raw result
GLM-5.3
devin/glm-5-3
Devin subscription
A coding agent, through CLIProxyAPI · reasoning=low
Short100100010010002026-11-10Raw result
Grok 4.7
devin/grok-4-7
Devin subscription
A coding agent, through CLIProxyAPI · reasoning=low
Short040400002026-11-10Raw result
Muse Ember
meta-subscription/muse-ember-1.0
Muse Code subscription key on Meta's Model API
A coding agent, through CLIProxyAPI · reasoning=low
Short100100500502026-11-10Raw result
Muse Ember
meta-subscription/muse-ember-1.0
Muse Code subscription key on Meta's Model API
A coding agent, through CLIProxyAPI · prompt_cache_retention=24h, reasoning=low
Long 24h1001001001001001000002026-11-10Raw result
Muse Spark 1.3
meta-subscription/muse-spark-1.3
Muse Code subscription key on Meta's Model API
A coding agent, through CLIProxyAPI · prompt_cache_retention=24h, reasoning=low
Long 24h1001000002026-11-10Raw result
Muse Wonder
meta-subscription/muse-wonder-1.0
Muse Code subscription key on Meta's Model API
A coding agent, through CLIProxyAPI · prompt_cache_retention=24h, reasoning=low
Long 24h100100002026-11-10Raw result
nemotron-3-5-super
devin/nemotron-3-5-super
Devin subscription
The Devin CLI · reasoning=medium
Short02026-11-09Raw result
nemotron-3-5-super
devin/nemotron-3-5-super
Devin subscription
A coding agent, through CLIProxyAPI · reasoning=none
Short1002026-11-09Raw result
Claude Haiku 5.5
devin/claude-haiku-5-5
Devin subscription
The Devin CLI · reasoning=low
Short10010000002026-11-08Raw result
Claude Haiku 5.5
devin/claude-haiku-5-5
Devin subscription
A coding agent, through CLIProxyAPI · reasoning=low
Short10010000002026-11-08Raw result
Claude Sonnet 5.5
devin/claude-sonnet-5-5
Devin subscription
The Devin CLI · reasoning=low
Short10010000002026-11-08Raw result
Claude Sonnet 5.5
devin/claude-sonnet-5-5
Devin subscription
A coding agent, through CLIProxyAPI · reasoning=low
Short10010000002026-11-08Raw result
deepseek-v4-1-flash-1017
devin/deepseek-v4-1-flash-1017
Devin subscription
The Devin CLI · reasoning=low
Short1001005050501002026-11-08Raw result
Gemini 4 Argon
devin/gemini-4-argon
Devin subscription
The Devin CLI · reasoning=low
Short10050500002026-11-08Raw result
GLM-5.3
devin/glm-5-3
Devin subscription
The Devin CLI · reasoning=low
Short1001001001001001002026-11-08Raw result
GPT-6.1 Sol
devin/gpt-6-1-sol
Devin subscription
A coding agent, through CLIProxyAPI · reasoning=low
Short100100100100002026-11-08Raw result
Grok 4.7
devin/grok-4-7
Devin subscription
The Devin CLI · reasoning=low
Short1001001000002026-11-08Raw result
Kimi K3.1 Pro
devin/kimi-k3-1-pro
Devin subscription
The Devin CLI · reasoning=low
Short100100100100100502026-11-08Raw result
Muse Wonder
devin/muse-wonder-1-0
Devin subscription
A coding agent, through CLIProxyAPI · reasoning=low
Short100100100100002026-11-08Raw result
nemotron-3-5-super
devin/nemotron-3-5-super
Devin subscription
The Devin CLI · reasoning=none
Short1001001001001001002026-11-08Raw result
nemotron-3-5-super
devin/nemotron-3-5-super
Devin subscription
A coding agent, through CLIProxyAPI · reasoning=low
Short0000002026-11-08Raw result
swe-2
devin/swe-2
Devin subscription
The Devin CLI · reasoning=low
Short0000002026-11-08Raw result
swe-2
devin/swe-2
Devin subscription
A coding agent, through CLIProxyAPI · reasoning=low
Short100000002026-11-08Raw result
Claude Opus 5.5
devin/claude-opus-5-5
Devin subscription
The Devin CLI · reasoning=low
Short100100000002026-11-07Raw result
Claude Opus 5.5
devin/claude-opus-5-5
Devin subscription
A coding agent, through CLIProxyAPI · reasoning=low
Short100100000002026-11-07Raw result
GPT-6.1 Terra
devin/gpt-6-1-terra
Devin subscription
A coding agent, through CLIProxyAPI · reasoning=low
Short100100100100002026-11-07Raw result
GPT-6.1 Terra
devin/gpt-6-1-terra
Devin subscription
The Devin CLI · reasoning=low
Short0000002026-11-07Raw result
Muse Wonder
meta-subscription/muse-wonder-1.0
Muse Code subscription key on Meta's Model API
Meta's Muse Code CLI · reasoning=low
Short000000002026-11-07Raw result

GET /api/v1/evaluations/cache?offset=0

Measured limits

All limits in tokens.

EntryDateMeasured throughContextInput acceptedInput refusedOutput limitRequested output acceptedEvidence
Claude Haiku 5.5
devin/claude-haiku-5-5
2026-11-16Devin subscription
Requests sent through CLIProxyAPI
Unknown999,0721,004,446UnknownUnknownRaw result
Claude Opus 5.5
devin/claude-opus-5-5
2026-11-16Devin subscription
Requests sent through CLIProxyAPI
Unknown999,0721,004,446UnknownUnknownRaw result
Claude Sonnet 5.5
devin/claude-sonnet-5-5
2026-11-16Devin subscription
Requests sent through CLIProxyAPI
Unknown999,0721,004,446UnknownUnknownRaw result
deepseek-v4-1-flash-1017
devin/deepseek-v4-1-flash-1017
2026-11-16Devin subscription
Requests sent through CLIProxyAPI
Unknown1,047,4891,052,864UnknownUnknownRaw result
DeepSeek V4 Pro
devin/deepseek-v4-pro
2026-11-16Devin subscription
Requests sent through CLIProxyAPI
Unknown1,047,5421,052,917UnknownUnknownRaw result
Gemini 4 Argon
devin/gemini-4-argon
2026-11-16Devin subscription
Requests sent through CLIProxyAPI
Unknown1,047,4901,052,865UnknownUnknownRaw result
GLM-5.3
devin/glm-5-3
2026-11-16Devin subscription
Requests sent through CLIProxyAPI
Unknown1,047,4701,052,845UnknownUnknownRaw result
GPT-6.1 Terra
devin/gpt-6-1-terra
2026-11-16Devin subscription
Requests sent through CLIProxyAPI
Unknown911,390923,890UnknownUnknownRaw result
GPT-6 Astra
devin/gpt-6-astra
2026-11-16Devin subscription
Requests sent through CLIProxyAPI
Unknown911,390923,890UnknownUnknownRaw result
GPT-6 Luna
devin/gpt-6-luna
2026-11-16Devin subscription
Requests sent through CLIProxyAPI
Unknown911,390923,890UnknownUnknownRaw result
swe-2
devin/swe-2
2026-11-16Devin subscription
Requests sent through CLIProxyAPI
Unknown300,499UnknownUnknownUnknownRaw result
Muse Ember
meta-subscription/muse-ember-1.0
2026-11-16Muse Code subscription key on Meta's Model API
Requests sent through CLIProxyAPI
1,048,5761,047,604UnknownUnknownUnknownRaw result
Muse Spark 1.3
meta-subscription/muse-spark-1.3
2026-11-16Muse Code subscription key on Meta's Model API
Requests sent through CLIProxyAPI
1,048,5761,047,604UnknownUnknownUnknownRaw result
GPT-6.1 Sol
openai-codex/gpt-6.1-sol
2026-11-13ChatGPT plan through the Codex login
Requests sent through CLIProxyAPI
Unknown915,011922,511UnknownUnknownRaw result
GPT-6 Astra
openai-codex/gpt-6-astra
2026-11-11ChatGPT plan through the Codex login
Requests sent through CLIProxyAPI
Unknown915,011922,511UnknownUnknownRaw result
GPT-6 Luna
openai-codex/gpt-6-luna
2026-11-11ChatGPT plan through the Codex login
Requests sent through CLIProxyAPI
Unknown915,011922,511UnknownUnknownRaw result
GPT-6.1 Sol
devin/gpt-6-1-sol
2026-11-09Devin subscription
Requests sent through CLIProxyAPI
Unknown915,390925,390UnknownUnknownRaw result
Claude Haiku 5.5
claude-subscription/claude-haiku-5-5
2026-11-08Claude Pro/Max login (the Claude Code subscription)
Requests sent through CLIProxyAPI
1,000,000998,953Unknown128,000UnknownRaw result
Claude Opus 5.5
claude-subscription/claude-opus-5-5
2026-11-08Claude Pro/Max login (the Claude Code subscription)
Requests sent through CLIProxyAPI
1,000,000998,953Unknown128,000UnknownRaw result
Claude Sonnet 5.5
claude-subscription/claude-sonnet-5-5
2026-11-08Claude Pro/Max login (the Claude Code subscription)
Requests sent through CLIProxyAPI
1,000,000998,953Unknown128,000UnknownRaw result
Muse Wonder
meta-subscription/muse-wonder-1.0
2026-11-08Muse Code subscription key on Meta's Model API
Requests sent through CLIProxyAPI
1,048,5761,047,604UnknownUnknown1,000,000Raw result
GPT-6.1 Terra
openai-codex/gpt-6.1-terra
2026-11-08ChatGPT plan through the Codex login
Requests sent through CLIProxyAPI
Unknown920,870921,886UnknownUnknownRaw result
Claude Haiku 5.5
openrouter/anthropic/claude-haiku-5.5
2026-11-08OpenRouter API key
Requests sent through CLIProxyAPI
1,000,000UnknownUnknownUnknownUnknownRaw result
Claude Opus 5.5
openrouter/anthropic/claude-opus-5.5
2026-11-08OpenRouter API key
Requests sent through CLIProxyAPI
1,000,000UnknownUnknownUnknownUnknownRaw result
Claude Sonnet 5.5
openrouter/anthropic/claude-sonnet-5.5
2026-11-08OpenRouter API key
Requests sent through CLIProxyAPI
1,000,000UnknownUnknownUnknownUnknownRaw result
DeepSeek V4.1 Flash
openrouter/deepseek/deepseek-v4.1-flash
2026-11-08OpenRouter API key
Requests sent through CLIProxyAPI
1,048,576UnknownUnknownUnknownUnknownRaw result
Gemini 4 Argon
openrouter/google/gemini-4-argon
2026-11-08OpenRouter API key
Requests sent through CLIProxyAPI
1,048,576UnknownUnknownUnknownUnknownRaw result
Muse Wonder
openrouter/meta/muse-wonder
2026-11-08OpenRouter API key
Requests sent through CLIProxyAPI
1,048,576UnknownUnknownUnknownUnknownRaw result
Kimi K3.1 Pro
openrouter/moonshotai/kimi-k3.1-pro
2026-11-08OpenRouter API key
Requests sent through CLIProxyAPI
1,048,576UnknownUnknownUnknownUnknownRaw result
GPT-6.1 Sol
openrouter/openai/gpt-6.1-sol
2026-11-08OpenRouter API key
Requests sent through CLIProxyAPI
1,050,000UnknownUnknownUnknownUnknownRaw result
GPT-6.1 Terra
openrouter/openai/gpt-6.1-terra
2026-11-08OpenRouter API key
Requests sent through CLIProxyAPI
1,050,000UnknownUnknownUnknownUnknownRaw result

GET /api/v1/evaluations/limits?offset=0

Dataset issues

GET /api/v1/meta