> measures knowledge reliability and hallucination. It rewards correct answers, penalizes hallucinations, and has no penalty for refusing to answer.
This is so useful because it makes you actually trust a models output. A high score on benchmarks is not as useful because a model overtrained to always answer will give confidently wrong responses. But this index measures how often it is correct while penalizing wrong responses so that a high score means you can trust this model more and when it doesn't know it is more likely to tell you that it really doesn't know rather than making shit up.
Fable also performs a lot better than opus 5 here which correlates very strongly with perceived strength despite the models performing similarly on e.g. DeepSWE
Astra is a big jump from sol and performs the same or slightly better than fable here.
Hallucinations are very damaging to a model’s utility. But doesn’t the Omniscience Index focus on knowledge-based queries? To me, using LLMs for their memorized knowledge is very 2023 and suboptimal.
IMO, what really makes a model useful is its ability to process information within its context reliably and faithfully. I don’t care if it hallucinates George Washington’s favorite color, but I do care about it hallucinating the results of tool calls.
Fair but I think those two behaviours are strongly correlated at least the index does represent my personal experience very well where fable is way better than opus opus is better than sol. And I haven't tried Astra yet but it having 44/43 is very interesting at least.
I don't agree. The main issue with their scoring/methodology is that the numbers make it seem like 5-6 models have little to no difference when in fact there is a significant difference between fable and opus and sol and astra for example. They are popular mainstream but most of their benchmarks are either not a representation of model strengths enough or they are not doing a good job of showcasing it properly. The fact that muse and 3.8 were high a day back shows they are just the modern version of lmareana for the mass audience and PR stunts.
The Epoch Capabilities Index uses an Elo-based aggregation method that dynamically adjusts for benchmark difficulty and they put error bars on their scores, both of which put them miles ahead of Artificial Analysis: https://epoch.ai/eci?view=graph&tab=leaderboard
> A high score on benchmarks is not as useful because a model overtrained to always answer will give confidently wrong responses.
It's not useful because the benchmarks often measure the wrong thing. They're here yapping about AGI and yet the benchmarks treat it like a trained dog. Fetch this. 100 points.
Each "problem" in these benchmarks likely has more than 1 solution that can be considered correct and even should be graded in many ways. Yet we see in many benchmarks higher effort (or thinking levels) don't help because the benchmark penalizes for doing "more" than what the answers asks for. So what did you ask for?
In human school you often get marks on the process and not just the end result. Thinking tokens have been cut. All we group on is things like cost, turns and time but not the what else.
This matches what I've seen building anything that uses an LLM for narrow structured output rather than open ended chat, things like classifying a diff into a fixed set of categories or summarizing a change. The aggregate benchmark score barely predicts how it behaves in production. What actually breaks a pipeline is a confident wrong answer on the small slice of inputs that don't fit the pattern it saw during training, not a lack of raw capability. A model that says it isn't sure on the edge cases is far more useful to me than one that scores higher on average but never admits uncertainty, because the wrong but confident output is the one that slips through review unnoticed.
It's really funny (and a bit obnoxious) to watch the vocabulary from the outside. In 2023 everybody learned the word moat, then it's been agent(ic), from last year there's more talking about harnesses than at a bungee jumping convention. The mot du jour is frontier.
It truly proves like there's a handful of thought leaders on Twitter that everybody follows blindly and start to copy down to the lexicon and parrot everywhere else.
I find the vocabulary used to be disturbingly fascinating, to the degree concepts are being anthropomorphized. It's so pervasive that I cannot help but assume it is entirely deliberate.
The principles by which LLMs functions haven't changed in the last four years. It is still a next-word-predictor, a statistical parrot, if you will. But if you don't understand the mechanisms behind it, you cannot be faulted for thinking this is something much more. Most of it, is pretty devious marketing.
As an example, no LLM model does anything that can be considered "reasoning", or "intelligent" in the traditional sense, but these words are used extensively. "High reasoning model" is a pricing tier. The article in question has an anthropomorphized term in every single sentence. I'll pick a paragraph at random and highlight the cases. If both parties understand the mechanisms, these words are fine, and we do that all that time. The issue is when one sides is mislead to believe that these systems can be relied on in a way that they should not be, leading to people getting hurt.
> > The *translation layer* is what *allows* a *harness* to *work* with different AI models. In some cases, a *harness may decide* to *use* different models within the same *agentic loop*, because different AI *models may excel* at different tasks. The *translation layer* is also a crucial aspect of *harnesses* because they *deliver control* to the end user. It means that someone can take their *AI harness* and use it with a model from Anthropic, or OpenAI, or explore one of the open weight AI models that often deliver great value-for-money (measured by cost-per-task).
The underlying logic isn't remotely as mysterious or mystic as the language makes it seem. A different paragraph:
> > Tools are a set of *capabilities*, written in code, that the model can *“call”*. The *harness describes* the tools and also *provides* the software that is the tool itself. Examples of these tools might include a web search tool, a tool that *allows the model* to write and execute software code, or a tool that *allows the model* to *compose* an email. Critically, the *harness usually* does not *dictate* when and how the *AI* model should *use* the tool. Instead, it simply *makes* the tools available, *describes* them clearly, and *allows* the *AI* model itself to *decide* when and how *it should use* them
My guess is that everything "reliable" in LLM/agentic-coding comes down to either calls to reliable/deterministic tools or providing well-defined success criteria (such as loads of unit tests) for the LLM to throw its stuff at in "agentic loops" until something sticks.
Instagram, Facebook and even threads all had much more mundane growth rates and definitely no unexpected jumps like GitHub is experiencing. I'm sure if suddenly the solar system had 10 more earths with each about 10 billion people and they would all start using Instagram tomorrow we would have exactly the same growing pains and outages that GitHub has today.
Luckily for Meta agents are not yet as much into doomscrolling as humans are.
1 in 3 is not terrible you just need a few more humans in the loop to reduce the error rate meaningfully. Combined with other classifier models and heuristics you can get good results. Humans can probably also perform better if they don't have to judge every single command but just suspicious ones our attention is limited after all.
1 in 3 is end of the line awful... Back when I was in college (former USSR), we had a subject roughly translated as "integration with industrial processes". USSR industry was highly regimented. Various norms, tolerances, recipes etc. were described in GOSTs (a kind of arsenal of industry standards). There were also some common knowledge / statistical bits that went into making these GOSTs. I mention this because this system dealt in great detail with quantifying human error (as well as errors resulting from equipment use etc.).
One of the core assumptions was that outside of extraordinary circumstances, the expected rate of human error is about 5%. However, the course also provided examples where error rates were significantly lower (eg. nurses in maternity wards would have a much lower than 0.1% error rate when pairing mothers with newborns).
The error rates, of course, also depended on human ability to measure the difference. Since I was studying typography, the printing process was of particular interest. A GOST for offset printing required that the color intensity for each ink of CMYK, for example, should be within +-2.5% range of the intended intensity. This is difficult for someone who doesn't have a lot of experience operating an offset printing machine to spot, but experienced printers have no problem with that.
Most importantly. There was never an acceptable error rate of 33%. Not for anything. If people were likely to make that many errors (eg. because the measurement was too difficult), that product would never have been allowed into production.
1/3, but under unreasonable time pressure, and with no prior vetting. For example, I played a few times, and I'm not a JS developer. I had to just suss out if npm whatever is dangerous or not. I'm very happy with my personal 25%.
The study he cites is also specifically using digital pens
> Brain electrical activity was recorded in 36 university students as they were handwriting visually presented words using a digital pen and typewriting the words on a keyboard
kinda funny.
It's also not clear from this study that there is actually any benefit to "more brain activity". Of course doing more complex motor tasks requires more brain activity but nobody guarantees that this helps with remembering things.
i had no science, but in college for shorter works i always thought handwriting, typing, and reading it aloud was my allowing my brain to hit a problem in three different (not more, different) ways before / even if ever being in front of another human being (or now their non-union AI equivalent)
i still feel writing something down helps with memory formation, in part because i remember the act of writing it down. cocktail napkins and whiteboards way better than ipads also
i remain obsessed with Pilot v5 rolling ball extra fine; i would say Pilot g2s are for proles but the outcome if you leave it in the dryer is far less severe
The problem of the sota models clamping down on third party harnesses should be stated upfront. Getting a SOTA model in a custom harness requires API pricing or risking an account ban, correct?
This preliminary change in cost may not make it worth it for people depending on which is their "favorite agent". Especially as the default harnesses for said agents continue to improve...
> The problem of the sota models clamping down on third party harnesses should be stated upfront. Getting a SOTA model in a custom harness requires API pricing or risking an account ban, correct?
The opposite, how easy it is nowadays should be clarified. Codex's app-server quite literally is built to be integrated with, authentication is a breeze and it's quite stable, compared to the TUIs that are around.
It's a real shame Codex is moving in the direction of hiding stuff on local disks from users, hopefully they're revert the decision of encrypting the agent>sub-agent prompts so we get back introspection again...
Presumably people want to hear the opinions of other people and still believe that they might learn more from the their experience than by exclusively interacting with AI models?
The alternative of that first impulse should be what exactly? Telling AI to research Medium AI drivel articles (since nobody seems to be posting long/medium form content anywhere else these days) to figure out what’s the best approach to building an AI harness?
BTW just my personal observation but I generally find AI notoriously bad at building any AI applications compared to standard software (which is not surprising given the scarcity of high quality training data)
huh? because im curious what they used?
Claude Code and codex take completely different approaches. The core loop of feeding generation and having a bash tool is entirely the same. If you want to start building your own thing I'm sure there is good setups out there to start from rather than going from zero.
Ofc I can just fork gemini-cli if I want my own version but like I don't think that's what OP meant by build your own thing and customize it.
It's pretty simple nowadays if you know conceptually how they work. Running the LLM calls in a loop with tools is an agent. You only need 10 or so basic tools to accomplish nearly anything, and you can build a dynamic skill system from that. Look at https://github.com/patw/pengy, ignore the app look at the spec.md file, feed that to your current agent of choice and make your own version. Use whatever tech stack or UI you're comfortable with. Change some of the choices in how it works, so it fits what you want to work.
Not the parent commenter, but most of it is surprisingly simple. You basically start with a "chat app" where you have a list of messages, send the whole conversation to an LLM and it replies back, which also gets added to the same list.
And you add a small twist, that instead of a 1-to-1 back and forth, you instead put it into a loop, where the LLM reply can itself "have a turn", e.g. a tool invocation, where your system is the one that replies (e.g. with the tool invocation's result). That's pretty much it, you have a 1 to potentially many "chat".
The harder part is getting all the "soft" parts right, like how to have well-behaving tool calls, timeouts, prevent huge cycles eating up tokens, but there are no one way to solve these, it's a fundamentally heuristic-heavy area.
But the whole point was that it wasn't a separate role: it was a re-distribution of work so the developers were doing operations. Instead, a lot of people entirely missed the point and just renamed their ops people to devops.
in the age of agents I found myself loving frontend again, loving devops again, loving email servers again, you name it, I suspect in most cases those roles will merge into one (excluding the ones needing extreme competency and trust).
reply