Rendered at 07:06:46 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
pants2 14 hours ago [-]
Kinda funny that their "cost-vs-performance" chart looks the same as the one for Composer 2.5[1], except that it includes Composer 2.5 at a completely different spot.
What are the chances that CursorBench ranks Cursor's model highest, and Cognition's bench ranks Cognition's model highest? Both are to be RL'd from Kimi as a base model, BTW.
I'd posit that it's not deliberate deception, but for both companies their training data and benchmarks come from the same dataset (Devin/Cursor interaction logs) so they naturally overfit.
I actually started typing the same point that the chances are actually high because of train/eval overlap then realised you answered your own question with that same observation.
It is interesting though!
Perhaps in some way this means we should decide which eval set aligns best with our taste?
Back to the blog post. This is an excellent write up of an excellent technical achievement.
I have a lot of respect for the Cognition/Devin (always "Windsurf" to me) and Cursor teams.
I found it interesting - but justified - that they referred to themselves as a foundation lab rather than a dev tools company.
swyx 10 hours ago [-]
agent lab, not foundation lab
oofbey 11 hours ago [-]
Agreed on the likely mechanism. I'm not sure "overfitting" is even the right description. These things are of course absurdly complicated, and evaluating their quality down to a single number involves a lot of judgement and trade-offs. I think it's more "you get what you measure" which is true in human organizations too. Define a KPI and people work hard to make it go up, even if it's not quite right or has bad side-effects.
meowface 10 hours ago [-]
At this point I barely put any value in any of the benchmarks. I just use the models for coding (and related things like software product design/planning/ideation/etc.) tasks and judge them subjectively, and also see how others judge them subjectively on HN and Twitter.
girvo 10 hours ago [-]
I use benchmarks…
…that are my own private internal suite on my own code bases where I can judge the output properly
I also measure wall clock time to completion which has been a surprising separator in practice.
ifwinterco 11 hours ago [-]
[dead]
kgeist 13 hours ago [-]
On artificialanalysis.ai, Kimi 2.7 Code is way worse than GLM 5.2 at everything (general intelligence, coding, agentic tasks).
But here, both Kimi 2.7 and its derivative SWE-1.7 are ahead of GLM 5.2. This tells me the benchmarks they use are cherry-picked.
bearjaws 12 hours ago [-]
Likely benchmaxxed. You see it in Qwen and other smaller models all the time.
qingcharles 10 hours ago [-]
Composer 2.5 is worse than both; I use it all day for simple stuff. It's Kimi 2.6 in a new outfit.
petesergeant 12 hours ago [-]
> This tells me the benchmarks they use are cherry-picked.
Which benchmarks would you have chosen instead, and why?
cogman10 10 hours ago [-]
It honestly seems like there's not a great way to currently benchmark AI.
The ideal way to run these benchmarks would be to give a 3rd party the model to run in an isolated environment so the prompts don't make their way back to the AI engineers.
That seems doable for open weight models, but not for private models.
p1necone 9 hours ago [-]
If you've got money to burn on tokens, the way that seems best to me is to set up a repeatable harness - docker container with a specific past commit from your own project, set of known issues/features that you've already fixed/completed of varying levels of complexity.
Set up a script that launches the harness for each model, prompts them to implement one of the tasks, let it churn until either tests pass or it hits some budget limit.
Then, most importantly, read the transcript and output and judge subjectively - I don't think this actually can be narrowed down to a score, although tokens burned to fix, whether it actually got the tests green etc are all good signals.
(I've done this, but so far only on a codebase that was too complicated with models that were too weak because I didn't want to spend more than a few dollars - results were inconclusive, planning on iterating on my personal benchmark in future)
ryandvm 14 hours ago [-]
Okay, let's give software engineers a break for a bit and focus on obsoleting other high-linguistic context occupations.
znsnjanwnwwn 13 hours ago [-]
And to do that you’ll need development so until we’re all out of a job they’ll keep pushing. Once automating is automated it’s done.
blauditore 10 hours ago [-]
Any day now... Just a bit more space in the context window, trust me bro.
Sabinus 6 hours ago [-]
We had decades of "just more and faster bits" in computing and that produced a lot of new capabilities.
silvertaza 13 hours ago [-]
Like diplomacy? heh
cromka 9 hours ago [-]
Some of these models could be particularly small, depending on the market they'd target...
FergusArgyll 9 hours ago [-]
I know you didn't mean this but have you ever seen Meta's model which plays (the game) Diplomacy? really cool
software and agentic workflows will obsolete those things
RL environments building on top of each other will get these models there
needs people doing software development lifecycles to figure it out and implement
yousif_123123 14 hours ago [-]
We need more models that optimize for coding and that can be cheaper than frontier models, like what SWE 1.7 and composer 2.5 are trying to do. I don't think there's an effort to make something GLM-5.2 level but focused only on coding.
euleriancon 11 hours ago [-]
This isn't as easy as it sounds. Every ML model is struggling to balance between generalization and test performance.
Taking a good model like GLM5.2 and just fine tuning it on coding can decrease real world performance due to mechanics like catastrophic forgetting. There is also other interesting behaviors were training on a broad training set can improve coding performance because there is positive transfer.
There is 100% an effort to make solid coding focused models, but it is very hard to do that without including capabilities across a broad set of adjacent tasks.
alansaber 10 hours ago [-]
As you say- LLMs are fundamentally good because of their generalism. Distillation, ablation, ft all tend to be hacky and in some way hurt the model
jstummbillig 13 hours ago [-]
Defining what "coding" means now, and how quickly we fall off the capability cliff seems increasingly important.
Today my "coding" sessions often enough begin with real life problems, where I discuss domain or inter-domain things, ranging from business, economics, psychology, etc. Being able to do all of that with one model is something I am willing to pay a premium for.
Of course not having to pay the premium, because the routing is smart or whatever, would be great. I just don't want to have to think about it.
andriy_koval 12 hours ago [-]
> Today my "coding" sessions often enough begin with real life problems
intuition is that your sessions consists of 10% of domain related reasoning, and 90% of code plumbing. Those 90% could be moved to cheap and efficient specialized and focused model.
RussianCow 8 hours ago [-]
But that 10% is the most important part! Getting the plumbing wrong means you might have bugs or your code is brittle. Getting the domain-specific business logic wrong means your product doesn't fundamentally solve the correct problem.
andriy_koval 4 hours ago [-]
[dead]
jstummbillig 10 hours ago [-]
Possibly! It's just hard to reason about from the outside. When does the model benefit from all the ambient knowledge? Idk.
Regardless, it's fairly obvious to me that none of what I do now will require "frontier models" for much longer. Models are getting better more quickly than my problems are getting harder.
tonyhart7 11 hours ago [-]
but that is not model problem
most agentic coding app can use powerful model for planning/reasoning then use "budget" model to do ground work
nomel 8 hours ago [-]
> most agentic coding app can use powerful model for planning/reasoning then use "budget" model to do ground work
I've had terrible success using budget models to do ground work. The justifications that the budget models will use and document, polluting the rest of the session, are sometimes just insane. Like making code compatible with a bug that was implemented within the same session, not handling errors due to precedence in the code it just implemented, etc. I DO have success using the heavy models with lower effort, and using budget models on relatively changes post ground work. But major planning and initial ground work, I just get absolutely slop if I use a budget model.
If you're doing web stuffs, or GUI, then the budget models seem fine.
UncleOxidant 14 hours ago [-]
Qwen was doing something like this with their coder models. But alas, they seem not to be releasing those anymore. Last one was Qwen3-coder-next.
yousif_123123 14 hours ago [-]
Its crazy that OpenAI and Anthropic themselves aren't doing that. No attempts at reducing inference cost for code as far as I know from them.
andai 13 hours ago [-]
OpenAI do have codex models, which are half the price. I haven't used them enough to comment on the quality though.
I remember them saying a few years ago that, they didn't think it was worth specializing models for code, because their general purpose models kept beating them. I guess they changed their mind? Since they did start making codex models again.
arcanemachiner 13 hours ago [-]
They also stopped making them again.
andriy_koval 12 hours ago [-]
My speculation is that frontier models are MoE, and they just have some number of experts for coding.
gsibble 14 hours ago [-]
I use this model. It's pretty good but not Opus 4.8 or Fable levels obviously. I'm really hoping we get more models like it (and better) soon. I run it locally and it's great that way.
UncleOxidant 14 hours ago [-]
Qwen3-coder-next is very usable. But I don't think it's as good as Qwen3.6-27B (though it does run faster on my hardware). It would be great if we could get a Qwen3.7-coder, but I'm not going to hold my breath.
gsibble 12 hours ago [-]
And make it run locally!
godzillabrennus 12 hours ago [-]
Cognition... oh what a ride... We were customers when they acquired Windsurf, stopped offering customer support, raised prices, dismantled the brand, and raised prices again. We are not customers anymore. Benchmarks are not the only thing to worry about when you are using models.
harmonic18374 14 hours ago [-]
A company whose first demo was completely fraudulent announces that its model beats GPT-5.5, on its own benchmark? I’m gonna wait a little before I trust this.
This whole company seems to optimize for raising money and impressing VCs. Lying about their products, ignoring consumer market to target enterprise, bragging about how they work their employees like slaves, and writing these posts full of intimidating technical jargon...
achandra03 14 hours ago [-]
To be fair it does seem like most AI startups are now like this (particularly when it comes to constantly mentioning how hard they work and ignoring consumer markets).
parineum 14 hours ago [-]
> it does seem like most AI startups are now like this
Remember when AGI was going to replace all jobs in 6 months? It's always been like that.
alansaber 10 hours ago [-]
This is inevitable when the primary incentive is to raise aggressively. Overall I dont find cognition blogs that jargony, there are definitely worse offenders
oa335 14 hours ago [-]
> "A company whose first demo was completely fraudulent"
Is it just me or does all that* seem pretty tame by today’s standards? Not saying it’s right, but it barely raises eyebrows. Sounds like a pretty typical startup demo.
* Based on the first comment in the link that claims to summarize the video.
sigbottle 13 hours ago [-]
I highly respect many people at cognition but yeah that's put a sour taste in my mouth.
I want to work in the AI space on actual AI research, at any part of the stack. Even if I'm developing training infra - as long as people are advancing knowledge of what intelligence could be.
But it seems like either it's big labs or grifters, that's it, and even the big labs, at least publicly, seem very grifty at times. Not like I have the technical chops probably, but still.
giancarlostoro 14 hours ago [-]
Would love to see these companies use benchmarks done by third parties.
anthonypasq 14 hours ago [-]
they are right there? it shows swe-bench multilingual and terminal bench
Apparently 'free' on the $20/mo Devin plan (presumably within some quota still)
and that is "via Cerebras at 1000 TPS" according to the announcement
I live on Opus 4.8 High and their benchmark scores SWE-1.7 slightly higher ... if at all realistic that sounds like a great deal ... too good to be true?
RussianCow 11 hours ago [-]
The "Lightning" (Cerebras) variant isn't free, only the regular one, which runs closer to 50 TPS in my experience with SWE 1.6.
settled 8 hours ago [-]
I used SWE-1.5 and 1.6 when it was Windsurf (before Devin Desktop), it's not that bad (grunt work, tests, can actually plan and implement some medium level stuff) but you get a much much better value and better models (GPT-5.4^) going with a Codex subscription (plus you get resets).
That company truly subsidized its user base to the extreme before, the $15/mo subscription was the best value on Earth paired with weekly deals reducing credits for premium models. Now it's barely any messages for paid models, completely watered down.
chris_st 8 hours ago [-]
FWIW, Cognition has all the Sonnet/Opus/Fable models, and all the GPT ones, as well as GLM, Kimi, and Gemini.
RussianCow 5 hours ago [-]
But they don't appear to subsidize them to the same degree. I've only been using Devin for less than a month, but I've been hitting the limits of the $20/month plan way more quickly than I'd expect, and definitely more quickly than with Claude Code or Codex.
So far, Cursor provides the best value for their subscription, but I have to imagine they're basically lighting money on fire. There's no way their current pricing is sustainable.
yousif_123123 11 hours ago [-]
The 1000 TPS shows for me as "SWE 1.7 Lightning" and took 14% of my daily quota in one prompt on the $20 plan.
But the normal speed one seems to be free or with very generous limits.
akshaydeshraj 13 hours ago [-]
Would have been worth a consideration if it could have been used beyond it's own harness. Unfortunately, doesn't seem to be the case.
I really don't want harness lock-in. I am trying to decouple myself from Claude Code now. I love the model of OpenRouter and being able to switch models at will let's your harness focus on your personal tooling and you can easily switch to the flavor of the month LLM with a single slash command instead of rewiring your entire workflow to use a harness to use a model.
I like Cerabras, but I really wish they would make more of their hosted models generally available.
2001zhaozhao 12 hours ago [-]
Harness-wrapper tools that support multiple harnesses and allow sharing workspace features (skills, slash commands, etc.) between them will be meta.
RussianCow 10 hours ago [-]
Ironically, Devin Desktop is one of those tools. It supports any harness that supports ACP (which is most of them)—you can use Claude Code, Codex, OpenCode, etc from the Devin Desktop UI.
I'm currently experimenting with OpenSpec[0] as the "framework" and using different subscriptions for different parts of the spec-driven process: Opus via Claude Code for exploration, Devin SWE for building, and GLM 5.2 via the Z.ai Coding Plan for verification. I don't love having to mix and match harnesses, but in practice it's barely more effort than switching models.
Open weight models should have GPL-like license where it says if you train model on it, it needs to be open weight as well.
joecot 13 hours ago [-]
Yes, and not only that but you can't even access it via API, you can only use it in Devin (formerly Windsurf).
I'm an OpenCode user, but I'll fall back to Claude Code if I want to use Opus end to end for something, given my company has a subscription. But I'm not using yet another tool and subscription for a model that isn't even winning.
Bnjoroge 12 hours ago [-]
I’ve unfortunately had to temper my excitement with Cognition’s models/products given the amount of unwarranted hype they created with Devin on first release, but hopefully this is good.
inglor 12 hours ago [-]
I work with Devin daily (as well as Claude and a few others) and I can attest it's not a cheap product but it's a good one and it saves me a bunch of time.
hedgehog 14 hours ago [-]
Heads up to anyone else curious, I installed the Devin CLI and SWE-1.7 is not currently available there.
luciana1u 6 hours ago [-]
SWE-1.7: the benchmark where AI agents finally learned to write code that passes tests, but only because they also wrote the tests
Mitchem 13 hours ago [-]
While I am skeptical of the results here, I am very excited for this new trend of making models faster. Running capable models at 1k TPS is more valuable for me than running better models at 30 TPS. I can only imagine the trend continues to move from "let's only make models smarter" to just incremental intelligence gains but with step improvements in speed.
lnenad 13 hours ago [-]
Why? I'm personally on the opposite end. Less babysitting/higher quality means more time goes back to me/the user. 1000tps of bad code means you have to keep validating the output and circling back.
anthonypasq 13 hours ago [-]
id rather iterate multiple times than wait 15 minutes to notice it made a mistake.
lnenad 13 hours ago [-]
Again, my point is exactly the opposite. Higher quality implies a mistake isn't made in a significant % of cases.
unshavedyak 13 hours ago [-]
It's a lossy conversion though. "Mistake" is relative to the stated goals and specifications which are often heavily lacking. So unless you write with a high degree of architectural and implementation specificity then it might make very high quality code that is still not what you wanted.
lnenad 11 hours ago [-]
You can ask for a complete feature/app/business. Or you can split up the work into verifiable/testable pieces and rely on a high quality AI to deliver. As time goes by the pieces will get larger as capability grows. I still trust myself and my experience when arch is involved, but AI has been great at tackling lower level stuff. And with Fable I don't really care it takes a while for it to complete, as I know I can trust it a lot more (which is what I personally prefer). Yes, with a 10k tps model you can iterate quickly. But that's not me personally.
RussianCow 10 hours ago [-]
But some requirements you don't realize you have until you start building. With a fast model, you can surface those really quickly and have more time to iterate and explore different solutions. With a slower but smarter model, you just hope that what it produces after an hour is what you were imagining.
And yes, with Fable, the chance of that is higher than with SWE/Composer, but in my experience it's not so much higher that the extra time and cost is worth it. But it certainly depends on your goals and what you're building.
aunty_helen 13 hours ago [-]
High tps is good for deeper agent thinking loops and openclaw etc. I was running cerebus recently doing some data heavy tasks, it managed to crash the server I was submitting posts to. 6 hour task down to ~1hr
unshavedyak 13 hours ago [-]
So i agree with you, but there's no SOTA model that i don't have to babysit. I'm not going to just throw a large pile of code in there unreviewed, and so what i want is faster iteration on code in logical, reviewable chunks. Ie just like i'd normally write myself; small, logical commits.
Faster iteration means i mentally checkout less and am more involved with the code being created.
My hope is that in the far far future, we can get LLMs so fast that i can work in my IDE like normal and the LLM will just be an extension of autocomplete. I can state a goal, rough out functions, code, etc, and it'll just work around me like a very fast pair programmer / autocomplete.
The chat interface is an intermediate step that frankly i hate. The faster it is the less i wait.
Now for vibe-slop i'm making on the side, yea i don't care about speed. But that's not something i'm employed to do or anything i truly care about. It's a different workflow entirely.
lnenad 11 hours ago [-]
I get it, you just prefer to do things differently
> Faster iteration means i mentally checkout less and am more involved with the code being created.
This is a good point I didn't consider and you're right. More interaction brings you closer to the code.
I still think that this is the opposite of what I personally want. Either I write the code (or a large majority of it), and be fully involved; or be more disconnected but more free to focus on other things. The middle ground removes me from the equation, but also requires me to babysit.
holoduke 13 hours ago [-]
Indeed. For me opus 4.8 is good enough. If only it would be 100 times faster. You could run it in self verification loops much much faster. It sometimes takes 15 minutes for me to complete a simple task. For example configuring AWS agentcore and deploying an agent on it. Takes forever with Claude with constant issues it tries to solve.
nibbleyou 14 hours ago [-]
Unrelated: what's the point of "*equal contribution"? Why would someone specify this
edot 14 hours ago [-]
Because papers are often referred to by the first author’s name, and often the first author is the primary researcher and therefore deserves the extra credit. When two or more primary authors are equally involved, they’ll often do a random ordering but annotate this so that no one thinks one did more than the others.
nibbleyou 14 hours ago [-]
Interesting. Thank you
modeless 11 hours ago [-]
What is the actual per token price? The benchmarks look similar to Grok 4.5 also released today and priced at $2/M input tokens and $6/M output tokens.
RussianCow 11 hours ago [-]
The regular one (not the fast variant) is free but slow. The "Lightning" variant (which uses Cerebras and gets supposedly 1000 TPS) costs $12.50/M output, $2.5/M input, $1/M cached input. So it's quite a bit more expensive than SWE 1.6.
tonyhart7 11 hours ago [-]
and what is actual intelligence per dollar benchmark ???? its useless comparing token/dollar while some model inherently generate more thinking output and cost more despite lower cost
spate141 14 hours ago [-]
Feels like they discovers that if you build your own benchmark, you can win it
londons_explore 14 hours ago [-]
Pretty sure most benchmarks are being gamed by people training on the test set deliberately or accidentally anyway.
amarant 12 hours ago [-]
Very thankful someone is doing this work! I suspect it will be thankless work for a while: we're not far enough into diminishing gains territory for anything other than the absolute best being worth considering for most people, but I reckon we will be pretty soon. If a year from now swe 2.0 or whatever reaches fable 5 parity for a fraction of the cost, that'll be very attractive indeed!
fallinditch 14 hours ago [-]
I'm looking forward to trying this out. I've been using SWE 1.6 quite a lot for grunt work alongside Opus for higher level planning and tricky stuff - a good combo.
As a (former) Windsurf user I'm pretty happy with the progress of the Cognition/Devin ecosystem after they took over Windsurf, now known as Devin Desktop.
bobtheborg 10 hours ago [-]
I like to use SWE-1.6 for quick help with git. For instance:
review the top stash and tell me what's in it (grouped appropriately)
1.6 does this fine nearly instantly.
1.7 tried for 17s before I killed it
2001zhaozhao 13 hours ago [-]
Wait Devin has a CLI?
Time to support it in my agent IDE just like Cursor's...
12 hours ago [-]
achierius 14 hours ago [-]
I've always had mixed feelings about Cognition. Obviously they have some very, very smart people working there (I even know a few), and they do make real products. But at the same time, they've made suspicious marketing claims more than once and even been caught making outright fabricated ones; and while they certainly seem to have shaped up from that, I still find their claims to be in a sort of grey area where they seem to avoid unfavorable comparisons and lean on their own benchmarks. Certainly when I've tried their models they have not been nearly as useful as comparable versions of Claude, GLM, etc. -- though I haven't had a chance to try SWE-1.7 yet.
llmslave 14 hours ago [-]
These models are never as good, the benchmarks dont tell the full story
haritha1313 14 hours ago [-]
The reality is most people building their own models and providing that alongside SOTA ones don't really care about how great these models are. They just prove that 'hey we are smart enough to build our own models so you can trust us instead of going with a single provider like Claude via Claude Code', also a cheap alternative for cost sensitive/free users - at least this was the case for Windsurf, not sure if Devin Desktop still has that tier. They just need to hillclimb the benchmarks and show something reasonable enough there.
spate141 14 hours ago [-]
Benchmarks are just vibes with error bars... wake me up when it survives a week on a real codebase without hallucinating a package that doesn't exist.
SubiculumCode 14 hours ago [-]
Funny, the cheerleading at HN for leading Chinese models, but a non Chinese lab (building on top of a Chinese model) gets dissed here.
mirekrusin 14 hours ago [-]
It's simple: close weights = not welcome.
sosodev 14 hours ago [-]
It's almost as if HN users aren't all the same.
llmslave 14 hours ago [-]
all the open source models are a waste of time relative to the bleeding edge from openai/anthropic
wongarsu 14 hours ago [-]
At work I wouldn't want to use anything else. Compared to my salary a Claude subscription (or two) is cheap
For hobby projects I've completely switched to DeepSeek v4 pro. I spend less than on a $10 Claude plan and am not subjected to quota limits (when I have time and motivation, the last thing I want is a 5 hour quota running out). And the difference in model performance is fine for those smaller projects, most of which will end up abandoned or in a state of "good enough" anyways
And for utility tasks, those 30b models are also great.
I'm a big fan of gemma4
llmslave 14 hours ago [-]
ive just got better things to do with my life than fuss with an inferior model. its like why hire a dumb employee over a smart one
BeetleB 13 hours ago [-]
I think you misspelled "I've got plenty of money".
llmslave 13 hours ago [-]
200 bucks a month?
BeetleB 11 hours ago [-]
Is a lot of money. The majority of people here aren't willing to spend $200/mo for coding unless their little projects provide a comparable value back to them.
For context, I'm paying under $30/year and get GLM-5.2. An extra $2300/year isn't going to get me much better outcomes.
llmslave 8 hours ago [-]
it seems cheap for what is borderline AGI
BeetleB 7 hours ago [-]
The point is that the less capable models are also borderline AGI. You're paying 10-100x more to get a few percentage points improvement in performance.
Put another way, what I get for my under $3/mo is better than what you were getting 3-5 months ago paying $200/mo. So you're paying a lot just to be ahead by a few paltry months.
pixel_popping 14 hours ago [-]
Not true since a few months, genuinely try GLM 5.2 and Minimax M3, especially in adversarial/gating... as a general model, I can agree, but as a coding model, they are not bad, comparable to maybe Opus 4.5 in real usage which is quite impressive.
villish 14 hours ago [-]
I use GLM or DS4 to help me draft a better initial prompt with more information that I then give to Sonnet 5/Fable/GPT5.5. While benchmarks show the open models close to frontier level, my experience with them is drastically different. I have high confidence that Fable or GPT will 1 shot solutions.
At least with low level programming languages. They're all very good for webdev stuff.
llmslave 14 hours ago [-]
yeah but why waste your time on these models, just use the one that gets the better results
nicoburns 14 hours ago [-]
I actively prefer GLM-5.2 for some tasks. For simple tasks the results are just as good as e.g. Opus, and it produces results significantly faster.
9183726518 14 hours ago [-]
Because you can get them from more trustworthy providers or with hardware encryption.
llmslave 14 hours ago [-]
i trust anthropic/openai with my data far more than some random startup.
somenameforme 14 hours ago [-]
I was going to respond until I saw your account name lol.
llmslave 13 hours ago [-]
haha i outsource my thinking to the smartest model
throwaw12 14 hours ago [-]
Open source for the win!
Imagine how far community might have pushed if 2 past versions of 'morally superior' Anthropic and 'completely Open AI' open sourced their models for the community to build on top of them
spott 14 hours ago [-]
Is this open source? I can't find a link to download the weights.
UncleOxidant 14 hours ago [-]
It's based on an open weight model (Kimi 2.7) so shouldn't it also be open weight?
NitpickLawyer 13 hours ago [-]
> so shouldn't it also be open weight?
Should as in "would it be nice?" - yeah. Should as in they have to? No.
> Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the “Software”), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so
You can do pretty much anything you want with an MIT license.
andy99 14 hours ago [-]
There is no obligation to do that. I think the landscape would be very different now if one of the big labs had released an earlier “frontier” model under copyleft that requires sharing fine tunes. I hope it still happens.
cmrdporcupine 12 hours ago [-]
Dario is convinced that will create SkyNet, and so no, it will never happen. Only the blessed members of the True Church Of Effective Altruism can approach the Ark of the Covenant. The unwashed cannot be trusted.
api 12 hours ago [-]
Rationalism and EA is Scientology for the Bay Area.
cmrdporcupine 7 hours ago [-]
It's just the 2020s version of Ayn Rand's "Objectivism." Distillation of exploitative personality traits covered over in sophistry and philosophical excuses. Lets people be dickheads and be smugly superior about it at the same time.
Sabinus 6 hours ago [-]
Creating elaborate ideologies so people can justify their actions with "I'm doing this for you and us all and/or the greater good", is as old as time.
akshaydeshraj 13 hours ago [-]
Not open source. Also, not available beyond it's own harness.
14 hours ago [-]
hudo 13 hours ago [-]
How do i use it from opencode/openrouter?!
RussianCow 3 hours ago [-]
You cannot; you must use either their Devin Desktop app or the Devin CLI.
11 hours ago [-]
petesergeant 13 hours ago [-]
I think it's a bit odd to show the API prices for competitors when that's not how most people pay for them. I do like that it's provisioned by Cerebras though. I think I'd have leant towards focusing on the TPS.
messia 11 hours ago [-]
And yet when I use swe it feels like massive shit
11 hours ago [-]
messia 11 hours ago [-]
if you are still using these products in 2026 you are really a shit engineer
What are the chances that CursorBench ranks Cursor's model highest, and Cognition's bench ranks Cognition's model highest? Both are to be RL'd from Kimi as a base model, BTW.
I'd posit that it's not deliberate deception, but for both companies their training data and benchmarks come from the same dataset (Devin/Cursor interaction logs) so they naturally overfit.
1. https://cursor.com/blog/composer-2-5
https://arena.ai/leaderboard/code/webdev/pareto
They did not.
I actually started typing the same point that the chances are actually high because of train/eval overlap then realised you answered your own question with that same observation.
It is interesting though!
Perhaps in some way this means we should decide which eval set aligns best with our taste?
Back to the blog post. This is an excellent write up of an excellent technical achievement.
I have a lot of respect for the Cognition/Devin (always "Windsurf" to me) and Cursor teams.
I found it interesting - but justified - that they referred to themselves as a foundation lab rather than a dev tools company.
…that are my own private internal suite on my own code bases where I can judge the output properly
I also measure wall clock time to completion which has been a surprising separator in practice.
But here, both Kimi 2.7 and its derivative SWE-1.7 are ahead of GLM 5.2. This tells me the benchmarks they use are cherry-picked.
Which benchmarks would you have chosen instead, and why?
The ideal way to run these benchmarks would be to give a 3rd party the model to run in an isolated environment so the prompts don't make their way back to the AI engineers.
That seems doable for open weight models, but not for private models.
Set up a script that launches the harness for each model, prompts them to implement one of the tasks, let it churn until either tests pass or it hits some budget limit.
Then, most importantly, read the transcript and output and judge subjectively - I don't think this actually can be narrowed down to a score, although tokens burned to fix, whether it actually got the tests green etc are all good signals.
(I've done this, but so far only on a codebase that was too complicated with models that were too weak because I didn't want to spend more than a few dollars - results were inconclusive, planning on iterating on my personal benchmark in future)
https://ai.meta.com/research/cicero/
RL environments building on top of each other will get these models there
needs people doing software development lifecycles to figure it out and implement
Taking a good model like GLM5.2 and just fine tuning it on coding can decrease real world performance due to mechanics like catastrophic forgetting. There is also other interesting behaviors were training on a broad training set can improve coding performance because there is positive transfer.
There is 100% an effort to make solid coding focused models, but it is very hard to do that without including capabilities across a broad set of adjacent tasks.
Today my "coding" sessions often enough begin with real life problems, where I discuss domain or inter-domain things, ranging from business, economics, psychology, etc. Being able to do all of that with one model is something I am willing to pay a premium for.
Of course not having to pay the premium, because the routing is smart or whatever, would be great. I just don't want to have to think about it.
intuition is that your sessions consists of 10% of domain related reasoning, and 90% of code plumbing. Those 90% could be moved to cheap and efficient specialized and focused model.
Regardless, it's fairly obvious to me that none of what I do now will require "frontier models" for much longer. Models are getting better more quickly than my problems are getting harder.
most agentic coding app can use powerful model for planning/reasoning then use "budget" model to do ground work
I've had terrible success using budget models to do ground work. The justifications that the budget models will use and document, polluting the rest of the session, are sometimes just insane. Like making code compatible with a bug that was implemented within the same session, not handling errors due to precedence in the code it just implemented, etc. I DO have success using the heavy models with lower effort, and using budget models on relatively changes post ground work. But major planning and initial ground work, I just get absolutely slop if I use a budget model.
If you're doing web stuffs, or GUI, then the budget models seem fine.
I remember them saying a few years ago that, they didn't think it was worth specializing models for code, because their general purpose models kept beating them. I guess they changed their mind? Since they did start making codex models again.
This whole company seems to optimize for raising money and impressing VCs. Lying about their products, ignoring consumer market to target enterprise, bragging about how they work their employees like slaves, and writing these posts full of intimidating technical jargon...
Remember when AGI was going to replace all jobs in 6 months? It's always been like that.
Could you expand on this?
* Based on the first comment in the link that claims to summarize the video.
I want to work in the AI space on actual AI research, at any part of the stack. Even if I'm developing training infra - as long as people are advancing knowledge of what intelligence could be.
But it seems like either it's big labs or grifters, that's it, and even the big labs, at least publicly, seem very grifty at times. Not like I have the technical chops probably, but still.
Apparently 'free' on the $20/mo Devin plan (presumably within some quota still)
and that is "via Cerebras at 1000 TPS" according to the announcement
I live on Opus 4.8 High and their benchmark scores SWE-1.7 slightly higher ... if at all realistic that sounds like a great deal ... too good to be true?
That company truly subsidized its user base to the extreme before, the $15/mo subscription was the best value on Earth paired with weekly deals reducing credits for premium models. Now it's barely any messages for paid models, completely watered down.
So far, Cursor provides the best value for their subscription, but I have to imagine they're basically lighting money on fire. There's no way their current pricing is sustainable.
But the normal speed one seems to be free or with very generous limits.
https://x.com/theodormarcu/status/2074896486047834380
I like Cerabras, but I really wish they would make more of their hosted models generally available.
I'm currently experimenting with OpenSpec[0] as the "framework" and using different subscriptions for different parts of the spec-driven process: Opus via Claude Code for exploration, Devin SWE for building, and GLM 5.2 via the Z.ai Coding Plan for verification. I don't love having to mix and match harnesses, but in practice it's barely more effort than switching models.
[0]: https://openspec.dev/
I'm an OpenCode user, but I'll fall back to Claude Code if I want to use Opus end to end for something, given my company has a subscription. But I'm not using yet another tool and subscription for a model that isn't even winning.
And yes, with Fable, the chance of that is higher than with SWE/Composer, but in my experience it's not so much higher that the extra time and cost is worth it. But it certainly depends on your goals and what you're building.
Faster iteration means i mentally checkout less and am more involved with the code being created.
My hope is that in the far far future, we can get LLMs so fast that i can work in my IDE like normal and the LLM will just be an extension of autocomplete. I can state a goal, rough out functions, code, etc, and it'll just work around me like a very fast pair programmer / autocomplete.
The chat interface is an intermediate step that frankly i hate. The faster it is the less i wait.
Now for vibe-slop i'm making on the side, yea i don't care about speed. But that's not something i'm employed to do or anything i truly care about. It's a different workflow entirely.
> Faster iteration means i mentally checkout less and am more involved with the code being created.
This is a good point I didn't consider and you're right. More interaction brings you closer to the code.
I still think that this is the opposite of what I personally want. Either I write the code (or a large majority of it), and be fully involved; or be more disconnected but more free to focus on other things. The middle ground removes me from the equation, but also requires me to babysit.
As a (former) Windsurf user I'm pretty happy with the progress of the Cognition/Devin ecosystem after they took over Windsurf, now known as Devin Desktop.
review the top stash and tell me what's in it (grouped appropriately)
1.6 does this fine nearly instantly.
1.7 tried for 17s before I killed it
Time to support it in my agent IDE just like Cursor's...
For hobby projects I've completely switched to DeepSeek v4 pro. I spend less than on a $10 Claude plan and am not subjected to quota limits (when I have time and motivation, the last thing I want is a 5 hour quota running out). And the difference in model performance is fine for those smaller projects, most of which will end up abandoned or in a state of "good enough" anyways
And for utility tasks, those 30b models are also great. I'm a big fan of gemma4
For context, I'm paying under $30/year and get GLM-5.2. An extra $2300/year isn't going to get me much better outcomes.
Put another way, what I get for my under $3/mo is better than what you were getting 3-5 months ago paying $200/mo. So you're paying a lot just to be ahead by a few paltry months.
At least with low level programming languages. They're all very good for webdev stuff.
Imagine how far community might have pushed if 2 past versions of 'morally superior' Anthropic and 'completely Open AI' open sourced their models for the community to build on top of them
Should as in "would it be nice?" - yeah. Should as in they have to? No.
> Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so
You can do pretty much anything you want with an MIT license.