Skip to content

Evidence

Corrections and evaluations published by providers.dev. Evidence older than 90 days is marked stale. To propose a correction or ask for an evaluation, see Contributing.

Corrected entries
24
Added providers
4
Cache evaluations
86
Measured limits
31

Corrections

capabilities.cache.implicit: true. models.dev: Not provided.

OpenRouter documents that prompt caching with DeepSeek is automatic. No lifetime is documented, so none is set.

7 entries

openrouter/deepseek/deepseek-v4-flashopenrouter/deepseek/deepseek-v4-flash-0731openrouter/deepseek/deepseek-v4-flash-vision-expopenrouter/deepseek/deepseek-v4-proopenrouter/deepseek/deepseek-v4-pro-0813openrouter/deepseek/deepseek-v4.1-flashopenrouter/deepseek/deepseek-v4.1-flash-1017

capabilities.cache.implicit: true. models.dev: Not provided.

OpenRouter lists a cached-input price for this model and has no caching section for the lab. In a live check an immediate re-send was served from the cache.

1 entries

openrouter/meta/muse-wonder

capabilities.cache.implicit: true. models.dev: Not provided.

OpenRouter lists a cached-input price for this model and has no caching section for the lab. In a live check an immediate re-send was served from the cache.

1 entries

openrouter/mistralai/mistral-large-4-0

capabilities.cache.implicit: true. models.dev: Not provided.

OpenRouter documents that prompt caching with Moonshot AI is automatic. No lifetime is documented, so none is set.

5 entries

openrouter/moonshotai/kimi-k2.5openrouter/moonshotai/kimi-k2.6openrouter/moonshotai/kimi-k2.7-codeopenrouter/moonshotai/kimi-k3openrouter/moonshotai/kimi-k3.1-pro

capabilities.cache.implicit: true. models.dev: Not provided.

OpenRouter documents that prompt caching with Z.AI is automatic. No lifetime is documented, so none is set.

10 entries

openrouter/z-ai/glm-5openrouter/z-ai/glm-5-turboopenrouter/z-ai/glm-5.1openrouter/z-ai/glm-5.2openrouter/z-ai/glm-5.3openrouter/z-ai/glm-5.3-flashopenrouter/z-ai/glm-5.3-flashxopenrouter/z-ai/glm-5.3-flashx-legalopenrouter/z-ai/glm-5.3-primeopenrouter/z-ai/glm-5v-turbo

GET /api/v1/corrections?offset=0

Added providers

ProviderAccessEntries
Claude CodeReached with a Claude Pro or Max login (the Claude Code sign-in), not with an Anthropic API key.3
DevinReached with a Devin subscription. Devin serves other labs' models under its own model ids.17
Google AntigravityReached with a Google Antigravity login, not with a Gemini API key.1
Muse CodeReached with the Muse Code API key that a Muse Code subscription applies to, not with a pay-as-you-go Meta Model API key.3

Cache evaluations

Observed hit percentages by exact age, for the client and request setting shown.

86 cache evaluations
EntryMeasured throughServed from cache, % by ageDateEvidence
Muse Wonder
meta-subscription/muse-wonder-1.0
Muse Code subscription key on Meta's Model API
A coding agent, through CLIProxyAPI · reasoning=low
Short1001001001001001001001002026-11-07Raw result
GPT-6.1 Sol
openai-codex/gpt-6.1-sol
ChatGPT plan through the Codex login
OpenAI's Codex CLI · reasoning=low
Short1001006710067100336702026-11-07Raw result
GPT-6.1 Sol
openai-codex/gpt-6.1-sol
ChatGPT plan through the Codex login
A coding agent, through CLIProxyAPI · reasoning=low
Short10010010010067671003302026-11-07Raw result
GPT-6.1 Terra
openai-codex/gpt-6.1-terra
ChatGPT plan through the Codex login
OpenAI's Codex CLI · reasoning=low
Short10010010010067670002026-11-07Raw result
GPT-6.1 Terra
openai-codex/gpt-6.1-terra
ChatGPT plan through the Codex login
A coding agent, through CLIProxyAPI · reasoning=low
Short1001001006733330002026-11-07Raw result
Muse Wonder
meta-subscription/muse-wonder-1.0
Muse Code subscription key on Meta's Model API
Requests sent through CLIProxyAPI ·
Short10010010010067330672026-11-06Raw result
GPT-6.1 Sol
openai-codex/gpt-6.1-sol
ChatGPT plan through the Codex login
Requests sent through CLIProxyAPI ·
Short1001006767100100673302026-11-06Raw result
GPT-6.1 Terra
openai-codex/gpt-6.1-terra
ChatGPT plan through the Codex login
Requests sent through CLIProxyAPI ·
Short10033332026-11-06Raw result
Claude Haiku 5.5
openrouter/anthropic/claude-haiku-5.5
OpenRouter API key
Requests sent through CLIProxyAPI · format=messages
Short100100100002026-11-06Raw result
Claude Haiku 5.5
openrouter/anthropic/claude-haiku-5.5
OpenRouter API key
Requests sent through CLIProxyAPI · format=messages, ttl=1h
Long 1h1001001002026-11-06Raw result
Claude Haiku 5.5
openrouter/anthropic/claude-haiku-5.5
OpenRouter API key
Requests sent straight to the provider · cache_control=top-level, session_id=true
Short100100100002026-11-06Raw result
Gemini 4 Argon
openrouter/google/gemini-4-argon
OpenRouter API key
Requests sent through CLIProxyAPI · prefix_tokens=8000
Short10010050002026-11-06Raw result
Kimi K3.1 Pro
openrouter/moonshotai/kimi-k3.1-pro
OpenRouter API key
Requests sent through CLIProxyAPI ·
Short10010050002026-11-06Raw result
Claude Opus 5.5
anthropic/claude-opus-5-5
Anthropic API key
Requests sent through CLIProxyAPI · format=messages, ttl=1h, beta=extended-cache-ttl-2025-04-11
Long 1h100100672026-11-04Raw result
Claude Opus 5.5
anthropic/claude-opus-5-5
Anthropic API key
Requests sent through CLIProxyAPI · format=messages, ttl=1h
Long 1h1001001002026-11-04Raw result
Gemini 3.8 Flash
antigravity/gemini-3.8-flash-high
Google Antigravity login
Requests sent through CLIProxyAPI · prefix_tokens=5000
Short100100336767002026-11-04Raw result
Claude Opus 5.5
devin/claude-opus-5-5
Devin subscription
Requests sent through CLIProxyAPI ·
Short100100100100100100100100100100671001001001002026-11-04Raw result
DeepSeek V4 Pro
devin/deepseek-v4-pro
Devin subscription
Requests sent through CLIProxyAPI ·
Short100010033002026-11-04Raw result
Gemini 4 Argon
devin/gemini-4-argon
Devin subscription
Requests sent through CLIProxyAPI · prefix_tokens=5000
Short100100673367002026-11-04Raw result
GLM-5.3
devin/glm-5-3
Devin subscription
Requests sent through CLIProxyAPI ·
Short10067331001001003333067002026-11-04Raw result
GPT-6.1 Sol
devin/gpt-6-1-sol
Devin subscription
Requests sent through CLIProxyAPI ·
Short100100100100100100100100100002026-11-04Raw result
Grok 4.7
devin/grok-4-7
Devin subscription
Requests sent through CLIProxyAPI ·
Short10067333367002026-11-04Raw result
Kimi K3
devin/kimi-k3
Devin subscription
Requests sent through CLIProxyAPI ·
Short02026-11-04Raw result
swe-2
devin/swe-2
Devin subscription
Requests sent through CLIProxyAPI ·
Short02026-11-04Raw result
Muse Spark 1.3
meta-subscription/muse-spark-1.3
Muse Code subscription key on Meta's Model API
Requests sent through CLIProxyAPI ·
Short1001001001006710010010067002026-11-04Raw result
Claude Opus 5.5
openrouter/anthropic/claude-opus-5.5
OpenRouter API key
Requests sent straight to the provider · cache_control=top-level, session_id=true, ttl=1h
Long 1h1001001002026-11-04Raw result
DeepSeek V4 Pro
openrouter/deepseek/deepseek-v4-pro
OpenRouter API key
Requests sent straight to the provider ·
Short10010067100100100100100676710010010067672026-11-04Raw result
Gemini 3.8 Flash
openrouter/google/gemini-3.8-flash
OpenRouter API key
Requests sent straight to the provider ·
Short100002026-11-04Raw result
Gemini 4 Argon
openrouter/google/gemini-4-argon
OpenRouter API key
Requests sent straight to the provider ·
Short100100100002026-11-04Raw result
Muse Wonder
openrouter/meta/muse-wonder
OpenRouter API key
Requests sent straight to the provider ·
Short10010067100100100100100002026-11-04Raw result
Mistral Large 4
openrouter/mistralai/mistral-large-4-0
OpenRouter API key
Requests sent straight to the provider ·
Short10010010010067002026-11-04Raw result
Kimi K3.1 Pro
openrouter/moonshotai/kimi-k3.1-pro
OpenRouter API key
Requests sent straight to the provider ·
Short10067676710067100100336733002026-11-04Raw result
GPT-6.1 Sol
openrouter/openai/gpt-6.1-sol
OpenRouter API key
Requests sent straight to the provider ·
Short100100100100100100100100100002026-11-04Raw result
GPT-6.1 Terra
openrouter/openai/gpt-6.1-terra
OpenRouter API key
Requests sent straight to the provider ·
Short100100100100100100100100100002026-11-04Raw result
Grok 4.7
openrouter/x-ai/grok-4.7
OpenRouter API key
Requests sent straight to the provider ·
Short100100333367002026-11-04Raw result
GLM-5.3
openrouter/z-ai/glm-5.3
OpenRouter API key
Requests sent straight to the provider ·
Short100100673367333367333367033002026-11-04Raw result

GET /api/v1/evaluations/cache?offset=50

Measured limits

All limits in tokens.

EntryDateMeasured throughContextInput acceptedInput refusedOutput limitRequested output acceptedEvidence
Claude Haiku 5.5
devin/claude-haiku-5-5
2026-11-16Devin subscription
Requests sent through CLIProxyAPI
Unknown999,0721,004,446UnknownUnknownRaw result
Claude Opus 5.5
devin/claude-opus-5-5
2026-11-16Devin subscription
Requests sent through CLIProxyAPI
Unknown999,0721,004,446UnknownUnknownRaw result
Claude Sonnet 5.5
devin/claude-sonnet-5-5
2026-11-16Devin subscription
Requests sent through CLIProxyAPI
Unknown999,0721,004,446UnknownUnknownRaw result
deepseek-v4-1-flash-1017
devin/deepseek-v4-1-flash-1017
2026-11-16Devin subscription
Requests sent through CLIProxyAPI
Unknown1,047,4891,052,864UnknownUnknownRaw result
DeepSeek V4 Pro
devin/deepseek-v4-pro
2026-11-16Devin subscription
Requests sent through CLIProxyAPI
Unknown1,047,5421,052,917UnknownUnknownRaw result
Gemini 4 Argon
devin/gemini-4-argon
2026-11-16Devin subscription
Requests sent through CLIProxyAPI
Unknown1,047,4901,052,865UnknownUnknownRaw result
GLM-5.3
devin/glm-5-3
2026-11-16Devin subscription
Requests sent through CLIProxyAPI
Unknown1,047,4701,052,845UnknownUnknownRaw result
GPT-6.1 Terra
devin/gpt-6-1-terra
2026-11-16Devin subscription
Requests sent through CLIProxyAPI
Unknown911,390923,890UnknownUnknownRaw result
GPT-6 Astra
devin/gpt-6-astra
2026-11-16Devin subscription
Requests sent through CLIProxyAPI
Unknown911,390923,890UnknownUnknownRaw result
GPT-6 Luna
devin/gpt-6-luna
2026-11-16Devin subscription
Requests sent through CLIProxyAPI
Unknown911,390923,890UnknownUnknownRaw result
swe-2
devin/swe-2
2026-11-16Devin subscription
Requests sent through CLIProxyAPI
Unknown300,499UnknownUnknownUnknownRaw result
Muse Ember
meta-subscription/muse-ember-1.0
2026-11-16Muse Code subscription key on Meta's Model API
Requests sent through CLIProxyAPI
1,048,5761,047,604UnknownUnknownUnknownRaw result
Muse Spark 1.3
meta-subscription/muse-spark-1.3
2026-11-16Muse Code subscription key on Meta's Model API
Requests sent through CLIProxyAPI
1,048,5761,047,604UnknownUnknownUnknownRaw result
GPT-6.1 Sol
openai-codex/gpt-6.1-sol
2026-11-13ChatGPT plan through the Codex login
Requests sent through CLIProxyAPI
Unknown915,011922,511UnknownUnknownRaw result
GPT-6 Astra
openai-codex/gpt-6-astra
2026-11-11ChatGPT plan through the Codex login
Requests sent through CLIProxyAPI
Unknown915,011922,511UnknownUnknownRaw result
GPT-6 Luna
openai-codex/gpt-6-luna
2026-11-11ChatGPT plan through the Codex login
Requests sent through CLIProxyAPI
Unknown915,011922,511UnknownUnknownRaw result
GPT-6.1 Sol
devin/gpt-6-1-sol
2026-11-09Devin subscription
Requests sent through CLIProxyAPI
Unknown915,390925,390UnknownUnknownRaw result
Claude Haiku 5.5
claude-subscription/claude-haiku-5-5
2026-11-08Claude Pro/Max login (the Claude Code subscription)
Requests sent through CLIProxyAPI
1,000,000998,953Unknown128,000UnknownRaw result
Claude Opus 5.5
claude-subscription/claude-opus-5-5
2026-11-08Claude Pro/Max login (the Claude Code subscription)
Requests sent through CLIProxyAPI
1,000,000998,953Unknown128,000UnknownRaw result
Claude Sonnet 5.5
claude-subscription/claude-sonnet-5-5
2026-11-08Claude Pro/Max login (the Claude Code subscription)
Requests sent through CLIProxyAPI
1,000,000998,953Unknown128,000UnknownRaw result
Muse Wonder
meta-subscription/muse-wonder-1.0
2026-11-08Muse Code subscription key on Meta's Model API
Requests sent through CLIProxyAPI
1,048,5761,047,604UnknownUnknown1,000,000Raw result
GPT-6.1 Terra
openai-codex/gpt-6.1-terra
2026-11-08ChatGPT plan through the Codex login
Requests sent through CLIProxyAPI
Unknown920,870921,886UnknownUnknownRaw result
Claude Haiku 5.5
openrouter/anthropic/claude-haiku-5.5
2026-11-08OpenRouter API key
Requests sent through CLIProxyAPI
1,000,000UnknownUnknownUnknownUnknownRaw result
Claude Opus 5.5
openrouter/anthropic/claude-opus-5.5
2026-11-08OpenRouter API key
Requests sent through CLIProxyAPI
1,000,000UnknownUnknownUnknownUnknownRaw result
Claude Sonnet 5.5
openrouter/anthropic/claude-sonnet-5.5
2026-11-08OpenRouter API key
Requests sent through CLIProxyAPI
1,000,000UnknownUnknownUnknownUnknownRaw result
DeepSeek V4.1 Flash
openrouter/deepseek/deepseek-v4.1-flash
2026-11-08OpenRouter API key
Requests sent through CLIProxyAPI
1,048,576UnknownUnknownUnknownUnknownRaw result
Gemini 4 Argon
openrouter/google/gemini-4-argon
2026-11-08OpenRouter API key
Requests sent through CLIProxyAPI
1,048,576UnknownUnknownUnknownUnknownRaw result
Muse Wonder
openrouter/meta/muse-wonder
2026-11-08OpenRouter API key
Requests sent through CLIProxyAPI
1,048,576UnknownUnknownUnknownUnknownRaw result
Kimi K3.1 Pro
openrouter/moonshotai/kimi-k3.1-pro
2026-11-08OpenRouter API key
Requests sent through CLIProxyAPI
1,048,576UnknownUnknownUnknownUnknownRaw result
GPT-6.1 Sol
openrouter/openai/gpt-6.1-sol
2026-11-08OpenRouter API key
Requests sent through CLIProxyAPI
1,050,000UnknownUnknownUnknownUnknownRaw result
GPT-6.1 Terra
openrouter/openai/gpt-6.1-terra
2026-11-08OpenRouter API key
Requests sent through CLIProxyAPI
1,050,000UnknownUnknownUnknownUnknownRaw result

GET /api/v1/evaluations/limits?offset=0

Dataset issues

GET /api/v1/meta