iCreatAI News
Back to main site
Skip to content
Only First party#Benchmarks
Oct 28Wed
  1. Browser Use Blog58

    Odysseys2: a long-horizon web agent benchmark

    Carnegie Mellon University and Browser Use released Odysseys2, a benchmark of 100 long-horizon tasks on the live web, each graded on 8 to 12 pass/fail checks. The best agent, GPT-6 Astra, fully completes 47 of 100 tasks with a mean rubric score of 84.6, while no other agent reaches 20 and five complete none. The authors note the LLM judge has not been validated against a human reference, reward-hacking flags are model judgments, and the live web changes.

  2. Artificial Analysis Articles60

    Announcing Artificial Analysis Intelligence Index v5

    Artificial Analysis released Intelligence Index v5, its biggest upgrade since v4, adding FrontierCode 2.0 and Terminal-Bench-Science and upgrading HLE-Diamond, AA-Omniscience v1.1 and CritPt v2, with 60% of the Index now based on private test sets or held-out answers. Claude Opus 5.5 (max) leads at 58, ahead of GPT-6 Astra (max, 57) and GPT-6.1 Sol (max, 55), while the open weights gap widens from 3 points to 12. It also launched the Progress Index v1, which tracks model capability over time on one continuous scale.

    Why it matters: It details a major benchmark revision and new leaderboard results, useful for readers tracking how frontier and open-weight models are measured.

  3. Allen Institute for AI Blog58

    Can we trust LLMs to judge whether a research idea is novel?

    Ai2 released "Old Ideas, Novel Problems," a study with a benchmark, data, code, and an automated pipeline for testing LLM novelty judges. Across six judges (GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5, Claude Opus 4.6), small prompt changes shifted accuracy by more than 50 percentage points on AI-generated ideas, retrieval and extra reasoning did not consistently help, and two purpose-built novelty evaluators scored worse than a simple prompted Claude Opus 4.6 baseline while costing far more.

  4. AWS Machine Learning Blog40

    Fix evaluation bias to unlock better models on Amazon Bedrock

    Property Finder found that 96.8% of its evaluation labels came directly from its incumbent vision model's predictions rather than human verification, biasing comparisons against alternatives. After building expert-annotated ground truth (356 UAE images) and running evolutionary prompt optimization from v0 to v27, it selected Llama 4 Maverick 17B Instruct on Amazon Bedrock, reaching a Micro F1 of 0.80 versus 0.56 for its previous model on the evaluation set, at an estimated 86% lower cost per 1,000 images.

  5. Hugging Face Blog58

    Can we trust LLMs to judge whether a research idea is novel?

    Ai2 released "Old Ideas, Novel Problems," a study with a benchmark, data, code, and an automated pipeline for testing LLM novelty judges, built from ICLR 2026 peer reviews to form Human-Only and Human+Generated evaluation sets. Across six judges (GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5, Claude Opus 4.6), small prompt changes shifted accuracy by more than 50 percentage points on AI-generated ideas, retrieval and extra reasoning did not consistently help, and two purpose-built novelty systems scored below a simple prompted Claude Opus 4.6 baseline.

Oct 27Tue
  1. Browser Use Blog38

    Browser Use vs Browserless: Pricing, Stealth & Features

    This comparison contrasts Browser Use and Browserless across pricing, stealth, performance, and deployment: Browser Use charges $0.02 per browser-hour with no subscription and reports 90.5% access success versus Browserless's 77.5% in its Stealth Bench V2 study, while Browserless uses unit-based subscription pricing and supports Chrome, Firefox, and WebKit. Browser Use offers hardened Chromium and hosted AI agents, whereas Browserless adds REST screenshot/PDF APIs, BrowserQL, and self-hostable Docker images. A worked example puts Browser Use at $7 in monthly usage versus Browserless's $25/month for 600 ten-minute sessions with 1 GB of residential traffic.

  2. ElevenLabs Blog50

    Why voice wins for complex, stressful conversations

    ElevenLabs compared voice calls and chat across hundreds of production agents and found that in conversations longer than two minutes, voice succeeded 32% more often than text chat and resolved successful long conversations about one third quicker. In a simulated canceled-flight scenario, 85% of 98 participants preferred voice, citing speed of resolution and empathy. A 16% reduction in average end-to-end latency across 315 agent conversations correlated with a 26% lift in success rate, with calls ending within 15 seconds falling from 21.1% to 3.8%.

Oct 15Thu
  1. Google Research65

    Putting Earth AI to work: Geospatial agents for our planet's biggest challenges

    Google shares a technical report on its experimental Geospatial Reasoning agent, which connects Gemini to Earth AI foundation models and planet-scale data, and reports evaluations across four benchmarks, including two newly released ones (GeoKTools and GeoExpertQueries). The post describes early partner testing with organizations including WHO AFRO, People in Need, GiveDirectly, the Milken Institute and WPP, and announces expanded early access via Ask Google Earth and an experimental agent-to-agent API.

    Why it matters: It offers a concrete look at how a planning-based geospatial agent is evaluated and applied to real public health, disaster and infrastructure workflows.

Oct 7Wed
  1. Microsoft Research23

    What AI gets wrong and what failure teaches us

    In a Microsoft Research Podcast conversation with Chad Atalla, Microsoft partner research manager Jennifer Neville discusses how her team uses practical evaluations—multiturn conversations, collaborative environments and long-horizon agentic tasks—to find where current AI systems break down beyond traditional benchmarks. She describes research showing that model performance degrades significantly over multiturn interactions and that errors can accumulate in long workflows, and offers practical guidance such as restarting a chat with a fully specified single turn and checking answers rather than fully delegating work.

Oct 5Mon
  1. GitHub Blog · AI & ML61

    ReviewBench: An open benchmark for AI code review

    GitHub introduces ReviewBench, an open benchmark for AI code review built from analysis of 103.9 million GitHub pull requests, containing 219 public pull requests across 19 languages with a multi-source golden set labeled for severity and category. It reports grounded and augmented precision, recall, and F1 metrics, uses Claude Sonnet 5 as the LLM grader under a published rubric, and had its ground-truth findings independently re-labeled by senior engineers with 96.6% agreement. GitHub says ReviewBench's offline predictions for a Copilot code review multi-model ensemble matched a later A/B test, which showed an 8.0% rise in addressed rate, 13.6% in recall, 61% in comment volume, and an 8.0% drop in cost per review.

    Why it matters: It offers a concrete, reproducible methodology and public dataset for comparing AI code reviewers, plus evidence on how offline scores tracked a production experiment.