That cloudflare blog post already spilled the beans: it’s just unreliable. The same task could be repeated three times and come up with three different answers, sometimes refusing outright, sometimes failing, sometimes returning false positives. Only a fraction of runs turned out anything useful, and that’s only after running a separate instance of mythos to sort through the trash, and then multiple more verification runs.
So, the most expensive model, burning the most tokens, you still need two instances and multiple runs through the same underlying task, and it may give you an exploitable bug. My understanding from cloudflare’s blog post is that their complex harness is entirely in-house, so I really think most of anthropic’s partners are having an even worse experience sorting through mythos trash.
My feeling is that there is a diminishing rate of return on token burn rate. I also believe increasing the complexity of models makes it harder to set boundaries and control output.
Also, most of the bugs so far have come down to not using basic OS safeguards or the attacker already having access to your computer. They are important threat vectors that need to be addressed, but they are types of vulnerabilities we’ve known about for decades and built protections around.
My question is why these harnesses are even necessary. The cloudflare pipeline is not specific to any codebase, it is just secret sauce they added themselves that increases the costs dramatically. Cloudflare is not an AI company though, Anthropic is, and openai and anthropic have spent tens of millions on signing bonuses for all of the most competent AI researchers in the field.
Why is it cloudflare’s job to make the model useful? Why doesn’t the model do what it says it will without multiplying the token burn rate 5-10x? Why not ship a harness developed by the ai experts, if a harness is truly necessary? The idea of adverserial machine learning is more than a decade old, it’s not like cloudflare stumbled on a new concept.
I believe this is just another attempt to hide the true cost of inference.
From what I gather a different harness can make quite a difference. Seems like a model can work better or worse depending on the harness, that’s at least what I‘ve heard from the community.
A harness for coding is probably different from a harness for agentic tasks like Hermes or opencode. … probably it also helps if you don’t vibe code your harness with little or no supervision. (Cough, Claude Code, cough)
That cloudflare blog post already spilled the beans: it’s just unreliable. The same task could be repeated three times and come up with three different answers, sometimes refusing outright, sometimes failing, sometimes returning false positives. Only a fraction of runs turned out anything useful, and that’s only after running a separate instance of mythos to sort through the trash, and then multiple more verification runs.
So, the most expensive model, burning the most tokens, you still need two instances and multiple runs through the same underlying task, and it may give you an exploitable bug. My understanding from cloudflare’s blog post is that their complex harness is entirely in-house, so I really think most of anthropic’s partners are having an even worse experience sorting through mythos trash.
My feeling is that there is a diminishing rate of return on token burn rate. I also believe increasing the complexity of models makes it harder to set boundaries and control output.
Also, most of the bugs so far have come down to not using basic OS safeguards or the attacker already having access to your computer. They are important threat vectors that need to be addressed, but they are types of vulnerabilities we’ve known about for decades and built protections around.
Interesting. I should have read the cloudflare article, not just linked it. Of course, anthropic does the bullshit it’s known for.
But I heard several security researchers experimenting with own harnesses. Seems to make quite a difference.
My question is why these harnesses are even necessary. The cloudflare pipeline is not specific to any codebase, it is just secret sauce they added themselves that increases the costs dramatically. Cloudflare is not an AI company though, Anthropic is, and openai and anthropic have spent tens of millions on signing bonuses for all of the most competent AI researchers in the field.
Why is it cloudflare’s job to make the model useful? Why doesn’t the model do what it says it will without multiplying the token burn rate 5-10x? Why not ship a harness developed by the ai experts, if a harness is truly necessary? The idea of adverserial machine learning is more than a decade old, it’s not like cloudflare stumbled on a new concept.
I believe this is just another attempt to hide the true cost of inference.
From what I gather a different harness can make quite a difference. Seems like a model can work better or worse depending on the harness, that’s at least what I‘ve heard from the community.
A harness for coding is probably different from a harness for agentic tasks like Hermes or opencode. … probably it also helps if you don’t vibe code your harness with little or no supervision. (Cough, Claude Code, cough)