Humanity's Last Exam · comparison article · with tools
设置
Web search/fetch, code execution, programmatic tool calling; context compaction at 50K up to 3M total tokens; max reasoning effort; adaptive thinking enabled.
未知项
Trial count and exact sampling settings for the Opus 4.5 row are not stated.
工具
允许工具
推理预算
max
框架
未记录
协议
Humanity's Last Exam, 2,500 questions; with tools.
条件
Web search/fetch, code execution and programmatic tool calling; 50K context compaction up to 3M total tokens; adaptive thinking and max reasoning effort. Kept separate from the System Card's 43.2% tools-only/no-reasoning result. 来源主文最初发布日期为2026-02-05,但当前HTML修改于2026-10-14;没有与此数值匹配的首发图像历史字节,因此精确首次公布日未知。
HLE reasoning-only, no tools; thinking auto; max reasoning effort; total tokens across contexts capped at 1M; no context compaction; Claude Opus 4.6 grader; standard five-trial average applies to the headline result.
未知项
The card does not state a separate headline trial count; the §8.1 standard average of five trials applies.
工具
禁止工具
推理预算
max
框架
未记录
协议
HLE; no tools。
条件
2,500 questions; reasoning-only, no tools; thinking auto; max reasoning effort; total context tokens capped at 1M; no context compaction; Opus 4.6 grader; §8.1 five-trial average applies.
HLE with web search, web fetch, programmatic tool calling and code execution; thinking auto; max reasoning effort; total tokens across contexts capped at 1M; no context compaction; Claude Opus 4.6 grader; known HLE sources blocklisted and transcripts reviewed.
未知项
The card does not state a separate headline trial count; the §8.1 standard average of five trials applies. Single-run effort-chart points use distinct configs.
工具
允许工具
推理预算
max
框架
未记录
协议
HLE; with tools。
条件
2,500 questions; web search, web fetch, programmatic tool calling and code execution; thinking auto; max reasoning effort; 1M total-token cap; no context compaction; HLE source blocklist and Opus 4.6 transcript review; §8.1 five-trial average applies.
Humanity’s Last Exam; reasoning only, no tools; auto thinking and max effort; 1M total-token cap; no context compaction. Some items exceed the Public API one-hour sampling limit.
未知项
具体题集与试验数未列于主表。
工具
禁止工具
推理预算
max
框架
未记录
协议
Humanity’s Last Exam; reasoning only, no tools; auto thinking, max effort; 1M total-token cap; no context compaction.
条件
The System Card says some no-tools problems exceed the Public API one-hour sampling limit; retain the reported 49.8% and source evidence with this reproducibility limitation.
Humanity’s Last Exam; web search, web fetch, programmatic tool calling and code execution; auto thinking and max effort; 1M total-token cap; no context compaction; Opus 4.6 transcript review/regrade for confirmed answer retrieval.
未知项
具体题集与试验数未列于主表。
工具
允许工具
推理预算
max
框架
未记录
协议
Humanity’s Last Exam with web search, web fetch, programmatic tool calling and code execution; auto thinking, max effort; 1M total-token cap; no context compaction.
条件
Opus 4.6 transcript review and regrade were used to confirm HLE answer retrieval.
Reasoning-only configuration; thinking=auto; 1M total-token cap across contexts; no context compaction; Opus 4.6 model grader.
未知项
The headline result does not identify a low-to-max effort setting.
工具
禁止工具
推理预算
max
框架
未记录
协议
Humanity’s Last Exam; reasoning-only, without tools.
条件
56.3%; thinking=auto; 1M total-token cap across contexts; no context compaction; Opus 4.6 grader. This headline run is distinct from explicit low-to-max effort charts.
Web search, web fetch, programmatic tool calling, and code execution; thinking=auto; 1M total-token cap across contexts; no context compaction; Opus 4.6 grader; HLE-specific URL blocklist and transcript contamination review.
未知项
The headline result does not identify a low-to-max effort setting.
工具
允许工具
推理预算
max
框架
未记录
协议
Humanity’s Last Exam; web search, web fetch, programmatic tool calling, and code execution.
条件
64.7%; thinking=auto; 1M total-token cap across contexts; no context compaction; Opus 4.6 grader; blocklist and transcript contamination review. Distinct from explicit effort charts.
Tools enabled; explicit figure-labeled effort scaling; billed API cost reported. This is separate from the headline thinking=auto run. Explicit adaptive-thinking effort: low.
未知项
The card does not state any additional sampling parameters beyond the cited evaluation description.
工具
允许工具
推理预算
low
框架
未记录
协议
HLE effort-scaling figure; with tools.
条件
Explicit effort labels; billed API actuals; separate from the headline thinking=auto result; figure does not state a token cap or context-compaction setting.
Tools enabled; explicit figure-labeled effort scaling; billed API cost reported. This is separate from the headline thinking=auto run. Explicit adaptive-thinking effort: medium.
未知项
The card does not state any additional sampling parameters beyond the cited evaluation description.
工具
允许工具
推理预算
medium
框架
未记录
协议
HLE effort-scaling figure; with tools.
条件
Explicit effort labels; billed API actuals; separate from the headline thinking=auto result; figure does not state a token cap or context-compaction setting.
Tools enabled; explicit figure-labeled effort scaling; billed API cost reported. This is separate from the headline thinking=auto run. Explicit adaptive-thinking effort: high.
未知项
The card does not state any additional sampling parameters beyond the cited evaluation description.
工具
允许工具
推理预算
high
框架
未记录
协议
HLE effort-scaling figure; with tools.
条件
Explicit effort labels; billed API actuals; separate from the headline thinking=auto result; figure does not state a token cap or context-compaction setting.
Tools enabled; explicit figure-labeled effort scaling; billed API cost reported. This is separate from the headline thinking=auto run. Explicit adaptive-thinking effort: xhigh.
未知项
The card does not state any additional sampling parameters beyond the cited evaluation description.
工具
允许工具
推理预算
xhigh
框架
未记录
协议
HLE effort-scaling figure; with tools.
条件
Explicit effort labels; billed API actuals; separate from the headline thinking=auto result; figure does not state a token cap or context-compaction setting.
Tools enabled; explicit figure-labeled effort scaling; billed API cost reported. This is separate from the headline thinking=auto run. Explicit adaptive-thinking effort: max.
未知项
The card does not state any additional sampling parameters beyond the cited evaluation description.
工具
允许工具
推理预算
max
框架
未记录
协议
HLE effort-scaling figure; with tools.
条件
Explicit effort labels; billed API actuals; separate from the headline thinking=auto result; figure does not state a token cap or context-compaction setting.
No tools; explicit figure-labeled effort scaling; billed API cost reported. This is separate from the headline thinking=auto run. Explicit adaptive-thinking effort: low.
未知项
The card does not state any additional sampling parameters beyond the cited evaluation description.
工具
禁止工具
推理预算
low
框架
未记录
协议
HLE effort-scaling figure; no tools.
条件
Explicit effort labels; separate from the headline thinking=auto result; figure does not state a token cap or context-compaction setting.
No tools; explicit figure-labeled effort scaling; billed API cost reported. This is separate from the headline thinking=auto run. Explicit adaptive-thinking effort: medium.
未知项
The card does not state any additional sampling parameters beyond the cited evaluation description.
工具
禁止工具
推理预算
medium
框架
未记录
协议
HLE effort-scaling figure; no tools.
条件
Explicit effort labels; separate from the headline thinking=auto result; figure does not state a token cap or context-compaction setting.
No tools; explicit figure-labeled effort scaling; billed API cost reported. This is separate from the headline thinking=auto run. Explicit adaptive-thinking effort: high.
未知项
The card does not state any additional sampling parameters beyond the cited evaluation description.
工具
禁止工具
推理预算
high
框架
未记录
协议
HLE effort-scaling figure; no tools.
条件
Explicit effort labels; separate from the headline thinking=auto result; figure does not state a token cap or context-compaction setting.
No tools; explicit figure-labeled effort scaling; billed API cost reported. This is separate from the headline thinking=auto run. Explicit adaptive-thinking effort: xhigh.
未知项
The card does not state any additional sampling parameters beyond the cited evaluation description.
工具
禁止工具
推理预算
xhigh
框架
未记录
协议
HLE effort-scaling figure; no tools.
条件
Explicit effort labels; separate from the headline thinking=auto result; figure does not state a token cap or context-compaction setting.
No tools; explicit figure-labeled effort scaling; billed API cost reported. This is separate from the headline thinking=auto run. Explicit adaptive-thinking effort: max.
未知项
The card does not state any additional sampling parameters beyond the cited evaluation description.
工具
禁止工具
推理预算
max
框架
未记录
协议
HLE effort-scaling figure; no tools.
条件
Explicit effort labels; separate from the headline thinking=auto result; figure does not state a token cap or context-compaction setting.
Web search, web fetch, programmatic tool calling, and code execution; explicit low effort; adaptive thinking; 1M total-token cap; no context compaction; Claude Opus 4.6 grader.
未知项
The figure does not state the number of model trials.
工具
允许工具
推理预算
low
框架
未记录
协议
HLE; web search, web fetch, programmatic tools, and code execution; low effort; adaptive thinking; 1M total-token cap; no context compaction; Claude Opus 4.6 grader.
条件
Exact task-set release and model-trial count are not stated.
Web search, web fetch, programmatic tool calling, and code execution; explicit medium effort; adaptive thinking; 1M total-token cap; no context compaction; Claude Opus 4.6 grader.
未知项
The figure does not state the number of model trials.
工具
允许工具
推理预算
medium
框架
未记录
协议
HLE; web search, web fetch, programmatic tools, and code execution; medium effort; adaptive thinking; 1M total-token cap; no context compaction; Claude Opus 4.6 grader.
条件
Exact task-set release and model-trial count are not stated.
Web search, web fetch, programmatic tool calling, and code execution; explicit high effort; adaptive thinking; 1M total-token cap; no context compaction; Claude Opus 4.6 grader.
未知项
The figure does not state the number of model trials.
工具
允许工具
推理预算
high
框架
未记录
协议
HLE; web search, web fetch, programmatic tools, and code execution; high effort; adaptive thinking; 1M total-token cap; no context compaction; Claude Opus 4.6 grader.
条件
Exact task-set release and model-trial count are not stated.
Web search, web fetch, programmatic tool calling, and code execution; explicit xhigh effort; adaptive thinking; 1M total-token cap; no context compaction; Claude Opus 4.6 grader.
未知项
The figure does not state the number of model trials.
工具
允许工具
推理预算
xhigh
框架
未记录
协议
HLE; web search, web fetch, programmatic tools, and code execution; xhigh effort; adaptive thinking; 1M total-token cap; no context compaction; Claude Opus 4.6 grader.
条件
Exact task-set release and model-trial count are not stated.
Claude Opus · Humanity's Last Exam · 2,500-question benchmark · Release history
View scores, configurations, and disclosures for Humanity's Last Exam · 2,500-question benchmark across 6 Claude Opus releases, ordered by release date.
Differences are calculated only for unique, active, exact values between direct predecessors when the unit, metric, and explicitly recorded configuration and conditions match.
Humanity's Last Exam · comparison article · with tools
Settings
Web search/fetch, code execution, programmatic tool calling; context compaction at 50K up to 3M total tokens; max reasoning effort; adaptive thinking enabled.
Unknowns
Trial count and exact sampling settings for the Opus 4.5 row are not stated.
Tools
Tools allowed
Reasoning budget
max
Framework
not recorded
Protocol
Humanity's Last Exam, 2,500 questions; with tools.
Conditions
Web search/fetch, code execution and programmatic tool calling; 50K context compaction up to 3M total tokens; adaptive thinking and max reasoning effort. Kept separate from the System Card's 43.2% tools-only/no-reasoning result. 来源主文最初发布日期为2026-02-05,但当前HTML修改于2026-10-14;没有与此数值匹配的首发图像历史字节,因此精确首次公布日未知。
· 7 score records for this release, including 7 exact values (may include historical records)
Records for the direct predecessor do not provide matching conditions for a valid difference.
Claude Opus 4.7 · Humanity's Last Exam scores and configurations
Metric and score
Configuration and evaluation conditions
Sources
Accuracy46.9%Reported 46.9% · Unit %
Published Unconfirmed
HLE 无工具
Settings
HLE reasoning-only, no tools; thinking auto; max reasoning effort; total tokens across contexts capped at 1M; no context compaction; Claude Opus 4.6 grader; standard five-trial average applies to the headline result.
Unknowns
The card does not state a separate headline trial count; the §8.1 standard average of five trials applies.
Tools
Tools forbidden
Reasoning budget
max
Framework
not recorded
Protocol
HLE; no tools。
Conditions
2,500 questions; reasoning-only, no tools; thinking auto; max reasoning effort; total context tokens capped at 1M; no context compaction; Opus 4.6 grader; §8.1 five-trial average applies.
HLE with web search, web fetch, programmatic tool calling and code execution; thinking auto; max reasoning effort; total tokens across contexts capped at 1M; no context compaction; Claude Opus 4.6 grader; known HLE sources blocklisted and transcripts reviewed.
Unknowns
The card does not state a separate headline trial count; the §8.1 standard average of five trials applies. Single-run effort-chart points use distinct configs.
Tools
Tools allowed
Reasoning budget
max
Framework
not recorded
Protocol
HLE; with tools。
Conditions
2,500 questions; web search, web fetch, programmatic tool calling and code execution; thinking auto; max reasoning effort; 1M total-token cap; no context compaction; HLE source blocklist and Opus 4.6 transcript review; §8.1 five-trial average applies.
· 2 score records for this release, including 2 exact values (may include historical records)
Records for the direct predecessor do not provide matching conditions for a valid difference.
Claude Opus 4.8 · Humanity's Last Exam scores and configurations
Metric and score
Configuration and evaluation conditions
Sources
Accuracy49.8%Reported 49.8% · Unit %
Published Unconfirmed
HLE 无工具
Settings
Humanity’s Last Exam; reasoning only, no tools; auto thinking and max effort; 1M total-token cap; no context compaction. Some items exceed the Public API one-hour sampling limit.
Unknowns
具体题集与试验数未列于主表。
Tools
Tools forbidden
Reasoning budget
max
Framework
not recorded
Protocol
Humanity’s Last Exam; reasoning only, no tools; auto thinking, max effort; 1M total-token cap; no context compaction.
Conditions
The System Card says some no-tools problems exceed the Public API one-hour sampling limit; retain the reported 49.8% and source evidence with this reproducibility limitation.
Humanity’s Last Exam; web search, web fetch, programmatic tool calling and code execution; auto thinking and max effort; 1M total-token cap; no context compaction; Opus 4.6 transcript review/regrade for confirmed answer retrieval.
Unknowns
具体题集与试验数未列于主表。
Tools
Tools allowed
Reasoning budget
max
Framework
not recorded
Protocol
Humanity’s Last Exam with web search, web fetch, programmatic tool calling and code execution; auto thinking, max effort; 1M total-token cap; no context compaction.
Conditions
Opus 4.6 transcript review and regrade were used to confirm HLE answer retrieval.
· 12 score records for this release, including 12 exact values (may include historical records)
Records for the direct predecessor do not provide matching conditions for a valid difference.
Claude Opus 5 · Humanity's Last Exam scores and configurations
Metric and score
Configuration and evaluation conditions
Sources
Accuracy56.3%Reported 56.3% · Unit %
Published Unconfirmed
HLE · thinking auto, no tools
Settings
Reasoning-only configuration; thinking=auto; 1M total-token cap across contexts; no context compaction; Opus 4.6 model grader.
Unknowns
The headline result does not identify a low-to-max effort setting.
Tools
Tools forbidden
Reasoning budget
max
Framework
not recorded
Protocol
Humanity’s Last Exam; reasoning-only, without tools.
Conditions
56.3%; thinking=auto; 1M total-token cap across contexts; no context compaction; Opus 4.6 grader. This headline run is distinct from explicit low-to-max effort charts.
Web search, web fetch, programmatic tool calling, and code execution; thinking=auto; 1M total-token cap across contexts; no context compaction; Opus 4.6 grader; HLE-specific URL blocklist and transcript contamination review.
Unknowns
The headline result does not identify a low-to-max effort setting.
Tools
Tools allowed
Reasoning budget
max
Framework
not recorded
Protocol
Humanity’s Last Exam; web search, web fetch, programmatic tool calling, and code execution.
Conditions
64.7%; thinking=auto; 1M total-token cap across contexts; no context compaction; Opus 4.6 grader; blocklist and transcript contamination review. Distinct from explicit effort charts.
Tools enabled; explicit figure-labeled effort scaling; billed API cost reported. This is separate from the headline thinking=auto run. Explicit adaptive-thinking effort: low.
Unknowns
The card does not state any additional sampling parameters beyond the cited evaluation description.
Tools
Tools allowed
Reasoning budget
low
Framework
not recorded
Protocol
HLE effort-scaling figure; with tools.
Conditions
Explicit effort labels; billed API actuals; separate from the headline thinking=auto result; figure does not state a token cap or context-compaction setting.
Tools enabled; explicit figure-labeled effort scaling; billed API cost reported. This is separate from the headline thinking=auto run. Explicit adaptive-thinking effort: medium.
Unknowns
The card does not state any additional sampling parameters beyond the cited evaluation description.
Tools
Tools allowed
Reasoning budget
medium
Framework
not recorded
Protocol
HLE effort-scaling figure; with tools.
Conditions
Explicit effort labels; billed API actuals; separate from the headline thinking=auto result; figure does not state a token cap or context-compaction setting.
Tools enabled; explicit figure-labeled effort scaling; billed API cost reported. This is separate from the headline thinking=auto run. Explicit adaptive-thinking effort: high.
Unknowns
The card does not state any additional sampling parameters beyond the cited evaluation description.
Tools
Tools allowed
Reasoning budget
high
Framework
not recorded
Protocol
HLE effort-scaling figure; with tools.
Conditions
Explicit effort labels; billed API actuals; separate from the headline thinking=auto result; figure does not state a token cap or context-compaction setting.
Tools enabled; explicit figure-labeled effort scaling; billed API cost reported. This is separate from the headline thinking=auto run. Explicit adaptive-thinking effort: xhigh.
Unknowns
The card does not state any additional sampling parameters beyond the cited evaluation description.
Tools
Tools allowed
Reasoning budget
xhigh
Framework
not recorded
Protocol
HLE effort-scaling figure; with tools.
Conditions
Explicit effort labels; billed API actuals; separate from the headline thinking=auto result; figure does not state a token cap or context-compaction setting.
Tools enabled; explicit figure-labeled effort scaling; billed API cost reported. This is separate from the headline thinking=auto run. Explicit adaptive-thinking effort: max.
Unknowns
The card does not state any additional sampling parameters beyond the cited evaluation description.
Tools
Tools allowed
Reasoning budget
max
Framework
not recorded
Protocol
HLE effort-scaling figure; with tools.
Conditions
Explicit effort labels; billed API actuals; separate from the headline thinking=auto result; figure does not state a token cap or context-compaction setting.
No tools; explicit figure-labeled effort scaling; billed API cost reported. This is separate from the headline thinking=auto run. Explicit adaptive-thinking effort: low.
Unknowns
The card does not state any additional sampling parameters beyond the cited evaluation description.
Tools
Tools forbidden
Reasoning budget
low
Framework
not recorded
Protocol
HLE effort-scaling figure; no tools.
Conditions
Explicit effort labels; separate from the headline thinking=auto result; figure does not state a token cap or context-compaction setting.
No tools; explicit figure-labeled effort scaling; billed API cost reported. This is separate from the headline thinking=auto run. Explicit adaptive-thinking effort: medium.
Unknowns
The card does not state any additional sampling parameters beyond the cited evaluation description.
Tools
Tools forbidden
Reasoning budget
medium
Framework
not recorded
Protocol
HLE effort-scaling figure; no tools.
Conditions
Explicit effort labels; separate from the headline thinking=auto result; figure does not state a token cap or context-compaction setting.
No tools; explicit figure-labeled effort scaling; billed API cost reported. This is separate from the headline thinking=auto run. Explicit adaptive-thinking effort: high.
Unknowns
The card does not state any additional sampling parameters beyond the cited evaluation description.
Tools
Tools forbidden
Reasoning budget
high
Framework
not recorded
Protocol
HLE effort-scaling figure; no tools.
Conditions
Explicit effort labels; separate from the headline thinking=auto result; figure does not state a token cap or context-compaction setting.
No tools; explicit figure-labeled effort scaling; billed API cost reported. This is separate from the headline thinking=auto run. Explicit adaptive-thinking effort: xhigh.
Unknowns
The card does not state any additional sampling parameters beyond the cited evaluation description.
Tools
Tools forbidden
Reasoning budget
xhigh
Framework
not recorded
Protocol
HLE effort-scaling figure; no tools.
Conditions
Explicit effort labels; separate from the headline thinking=auto result; figure does not state a token cap or context-compaction setting.
No tools; explicit figure-labeled effort scaling; billed API cost reported. This is separate from the headline thinking=auto run. Explicit adaptive-thinking effort: max.
Unknowns
The card does not state any additional sampling parameters beyond the cited evaluation description.
Tools
Tools forbidden
Reasoning budget
max
Framework
not recorded
Protocol
HLE effort-scaling figure; no tools.
Conditions
Explicit effort labels; separate from the headline thinking=auto result; figure does not state a token cap or context-compaction setting.
Web search, web fetch, programmatic tool calling, and code execution; explicit low effort; adaptive thinking; 1M total-token cap; no context compaction; Claude Opus 4.6 grader.
Unknowns
The figure does not state the number of model trials.
Tools
Tools allowed
Reasoning budget
low
Framework
not recorded
Protocol
HLE; web search, web fetch, programmatic tools, and code execution; low effort; adaptive thinking; 1M total-token cap; no context compaction; Claude Opus 4.6 grader.
Conditions
Exact task-set release and model-trial count are not stated.
Web search, web fetch, programmatic tool calling, and code execution; explicit medium effort; adaptive thinking; 1M total-token cap; no context compaction; Claude Opus 4.6 grader.
Unknowns
The figure does not state the number of model trials.
Tools
Tools allowed
Reasoning budget
medium
Framework
not recorded
Protocol
HLE; web search, web fetch, programmatic tools, and code execution; medium effort; adaptive thinking; 1M total-token cap; no context compaction; Claude Opus 4.6 grader.
Conditions
Exact task-set release and model-trial count are not stated.
Web search, web fetch, programmatic tool calling, and code execution; explicit high effort; adaptive thinking; 1M total-token cap; no context compaction; Claude Opus 4.6 grader.
Unknowns
The figure does not state the number of model trials.
Tools
Tools allowed
Reasoning budget
high
Framework
not recorded
Protocol
HLE; web search, web fetch, programmatic tools, and code execution; high effort; adaptive thinking; 1M total-token cap; no context compaction; Claude Opus 4.6 grader.
Conditions
Exact task-set release and model-trial count are not stated.
Web search, web fetch, programmatic tool calling, and code execution; explicit xhigh effort; adaptive thinking; 1M total-token cap; no context compaction; Claude Opus 4.6 grader.
Unknowns
The figure does not state the number of model trials.
Tools
Tools allowed
Reasoning budget
xhigh
Framework
not recorded
Protocol
HLE; web search, web fetch, programmatic tools, and code execution; xhigh effort; adaptive thinking; 1M total-token cap; no context compaction; Claude Opus 4.6 grader.
Conditions
Exact task-set release and model-trial count are not stated.