I’ve built a retro operating system and programming environment for ESP32 microcontrollers called BASIC32.
I took it to 39% complete, as a language, over 8-weeks. Then, I spent just 50 minutes describing all the missing parts and it finished the language in 1-shot. I didn’t think it would work. It did.
I used Opus 5.5 on a $20 plan and I didn’t even hit my limits.
I used Fable 5.1 and Opus 5.5 to build a Python interpreter in Rust. Appears to pass all the 3.12 code I throw at it.
To get a toy version that can run most things takes under 2 hrs. Half of that time is collecting the tests you want to pass, how it should evaluate the code, what your criteria are, etc.
yeah, one-shot (medium-large) builds just don't seem to give me good results yet. I use them now for small examples though, e.g. for visualizing a process. Most recently I used it to visualize neural networks, and how convolution works. That might not be something most people need, but it really helped me.
Hah, found someone else also using it for visualizations of algos! I've used it to show me how quad-trees[0] work and I'm impressed with the results. One-shotted too so I agree one-shots are best for this kind of very specific scenario.
> I've been attempting to build something that ... pushes the model ... never understood ... one-shot builds
Same, and I have found some of the failure modes in the realm of eyes free coding and limited auditing. I often see it more as pushing the process than the model, as I tend to bounce around models for the benefits diversity provides.
Have you also found there is a point where they hit code complexity and start churning without progress? (at which point you have to step in to help the bot step back and think big picture, which usually takes a few sessions)
Yes! I absolutely have! And I've been thinking a lot lately about how to prevent them from collapsing into this state. It is getting better with newer and better models though. But yes, there comes a point when the codebase size and complexity crosses a threshold and we enter a churn state with very little progress.
I was motivated to make the codebase visualizer linked above to help me understand how the various pieces of the system that is being vibe coded connect with each other.
Now I'm thinking about how I can build something similar for agents. How can we teach "systems thinking" to a model?
I've been mostly working on very large modeling projects (think detailed architeture over 1000 acres) or animation because this really was the "new" capability area that Opus 5.5 "unlocked".
I have been surprised how well it has worked at scale with many sub-agents. However the code base is limited to the rendering engine mostly.
Certainly in the older models they hit a complexity limit, but I haven't really hit it with Opus 5.5. I have had it make data flow designs that I didn't like because they were more complicated than it should have been or the perenial problem of trying to make everything backwards compatible when working on greenfield refactors, but I haven't seen it choke on a code base yet.
My sense is the limit is more tied to the problem than the code base size. I work in some very large code bases, but it's not an issue because the nature of the problem (think what a standard SaaS would need).
On the other hand, I have a TS only project core is sub-20k LoC (NES harness), where it is struggling (for now, see other comments)
I just have it constantly update real-world analogues for keeping track of everything. I have it create guides, note cards, flowcharts, running to-do lists, etc. with every change.
It helps us keep track of things, why not AI, too?
I've been working on process more than model. Similar to having a PR review team, you need multiple teams or processes that run over the code, markdown, assumptions. I still don't think this is enough.
I'm working on an NES harness, and the agent was churning with detecting certain state changes like dialogs, menu navigation, screen change. The bot and I got churning and I put it down for a while. A few days later it occured to me to ask the bot "Does the emulator and game state change with frames? Can we just get the state each frame rather than trying to be clever about it?"
It said yes, but added some nonsense about being performance hindered... (flash model). I haven't worked on it again yet, but my intention was to next have it build a small recorder based on this so it can get frame accurate game state change.
This is just a long way of saying, sometimes you have to step back and help them help themselves. They have a really hard time with recognizing ambiguity or how to determine lack of context that could be filled.
I get what the author is saying, but his examples like "a ripgrep fork that ... is faster for my codex ripgrep workloads" don't strike me as particularly ambitious.
Perhaps we should be more ambitious in the types of problems we try to solve, and the types of value we try to provide with the software we build.
The note taking genre of opensource apps is saturated with many options but most of them are electron based.
I was hoping that someone with enough motivation and money for tokens would come up with a non-electron option that is similar to onenote, with free form note placement and handwritten notes.
Basically I wish there is an opensource app having good feature parity with OneNote. Thats a good benchmark to have.
A few years ago I was wishing the same, but I was wishing it for someone knowledgeable enough.
Today I wish the same, but for someone with enough motivation and tokens instead.
Don't get the wrong idea: I would be glad for a OneNote/Obsidian alternative that's performant. I just lament the days where I'd stare in awe thinking about the levels of wizardry one must get to in order to build something that I admire. Now the first thing that comes to mind is "I wonder how many tokens did this took them to make".
you don't like the ads, gambling apps, and similar? /s
What might help is non-software-first companies valuing developers and engineers more, enough that we could earn a reasonable living which building actually cool things. Not sure how the economics works out on the other side. Can these companies make enough money to support this vision?
One major obstacle is developers have either encountered or created entire ecosystems of problems they can't solve that get inherited whenever and wherever software is introduced. Proposing to add evergreen issues around security, data entry and retention, connectivity, vendor lock-in, and update cycles is frequently a shitty trade for whatever notional (often fictional) productivity improvement is being proposed by injecting software into a workflow.
My building recently added ButterflyMX and boy does no software sound so much better. This is tech company that tracks people as the move across real world properties and sells your data to the ad industry.
I mean they're an excellent example, but this is just as true for the overwhelming majority of B2B productivity suites I've encountered. Hell, digital cameras didn't improve photography, they just killed three industries and saddled just about everyone with data retention and migration problems and that's without even getting into the social effects of ubiquitous cameras combined with social media or the surveillance state they've made possible.
I'm all about ending user tracking, targeted advertising, and gambling. We'll need to find new things to work on that do not use big data to take advantage of our fellow humans.
We should be more ambitious, but we need to learn how we can be ambitious with AI, and what value we can create.
AI can take us from 0 to MVP in a prompt. It can fill in gaps of knowledge and allows someone who knows half of the stack to build the whole thing nearly seamlessly. These are superpowers but don't by themselves lead to new products.
What AI cannot do is remove the world of complications that happen after the MVP - product feature tradeoffs and choices, how to handle integrations and customizations, how to serve multiple types of users, how to communicate and market the app, how to handle security and data storage, how to deal with customers and bugs, etc. All of these need to be considered and acted on by a human in some fashion. AI can help with each concern but can't solve them without understanding the concern and determining how to frame it in a prompt.
This is why we have a glut of amazing demos that go nowhere while we figure out how to build software differently. This will take some time and I imagine the result will be software heavily-structured with a bit of AI reasoning woven throughout, not an army of agents and sub-agents burning infinite tokens.
Right, but this is the most hopeful imaginable prediction: AI accelerates everything, but everything good is still mediated through taste and expertise, so being a (say) skilled systems engineer is important and rewarding, just with all the tedium distilled out of it.
There's a less hopeful outcome, where no skill is needed at all, because every conceivable thing is one-shot-able from a SOTA model. But then, at that point, you're saying something pretty profound about computer science (roughly: that there was no meaningfully more sophisticated and complex place to go from where we are today).
That’s something that I have been thinking about for a while now. Even prior to LLMs, many large companies have essentially been iterating on the same half a dozen ideas over and over again and there doesn’t seem to be much new coming out even post LLMs.
The most recent new things that LLMs have unlocked (regarding software dev) for us is reverse engineering and combining old games, unearthing decade old vulnerabilities, and helping root and debloat consumer hardware. Which are all cool and useful, but none of it being a “next great thing”.
Perhaps LLMs are really doing is highlighting how the software industry was already nearing its economic maturity and this just shifted the timeline from 2040 back to 2027-2030.
I have yet to see LLMs do well “autonomously” on anything that doesn’t fall into any of these 3 buckets:
1. well specified
2. well tested (which is a form of spec)
3. represented in the training set (i.e. not built on a corpus of human developed examples)
For example, a lot of the compiler porting efforts fall into all 3 buckets: compilers are well specified (there is a set of valid inputs and an expected behavior), they are typically well tested, and as a bonus, in a way that is implementation agnostic thanks to the clean interface (think: integration vs unit tests), and are very well represented in the training set.
You can "solve" 1 and 2 by N-version programming. Have the same code implemented twice, or more times, you get free error testing by fuzzing or running replicas in shadow mode. In order to make replicas maximally different choose a different language, framework, agent harness and LLM.
There remain some nitty details related to tracking randomness and choosing what interfaces to log. But even having your code testable by replicas would help organize it better and expose different assumptions made by the language or library stack.
When you get different outputs from replicas it can be 1. one of them has a bug, or 2. the spec is unclear -> fix the bug or the spec. So this method catches both sides. It also might be faster than traditional code based testing, you can build replicas in parallel. A replica is an open ended test, unlike coded tests it can be tried on open ended input values.
The core idea is that it is easier to implement twice than to imagine bugs. We already solved implementation, but bug finding means running code in your head, it touches the halting problem issues - can we really imagine behavior from code? I don't think we can do it well. That is why we should just make replicas instead, tokens are getting cheaper and cheaper by the month.
Honestly I think you're on to something, as by far the most novel and interesting use cases I've seen for LLMs in general are mostly around reverse engineering things, especially if you can hit it with Ghidra, but honestly anything that can feed some sort of data and control back to a model.
> the software industry was already nearing its economic maturity
What I'm sensing is something I might describe as the Oligarchy reaching the point of significant societal pushback, like the labor rights movements that ended 80 work weeks, unsafe conditions, and child labor in factories (i.e. The Jungle).
My hope is that we can use this moment in human history to break the user tracking and targeted advertising industry. It's been so harmful to humans, but in doing so, we will likely break a major revenue source that supports our well above average salaries. I still think it should be done.
What would be funny with LLMs, having been purposefully trained to be the best software creating things on earth, wind up inadvertently making software virtually valueless (because it can be made and copied so easily) and undermining the value of the companies that proliferated their use in the first place.
We'll have to separate the hardware providers from the software providers, which I'm growing more fond of. The US Ai stack is largely a vertically integrated cartel.
My wife is not a developer at all but is highly analytical. I find her a better software architect than most developers I met.
I suspect there are a lot of people really good at “systems” - there are a lot of really clever folks out there - that did not go through the proper CS rituals and are thus shunned but in fact can take over your job in no time especially with AI.
My wife regularly gets pushed away by “IT” at her work and unlike others she pushes back, but they don’t like her. They start talking about db clustering details etc as if that has any bearing on anything. I am a dev, I know she doesn’t know what logical replication is, but I do know she can replace their team. She just needs one technical person to keep an eye on things.
I suspect we will end up with just a handful of building blocks like “this is how data is siloed, secured etc”, “this is how dashboards are made here”, “this is how email is handled”, etc and employees with AI just develop their own stuff on top of these. The “devs” just oversee the building blocks and the overall soundness of the flow. Don’t know how many you need once that is standardized..
I haven't yet seen any significant products made with AI that are at least used long-term. All these browser demos, from the Adobe killers to Star Citizen, are just bells and whistles that I close after 30 seconds of use.
"Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away."
And I feel that sums up my feelings on AI this and that. Just as the wood and clay contain the sculpture within and it's the artist's duty and privilege to reveal that. So, is elegant software exposed from the conceptual mass, rather than built up.
The AI zeitgeist currently is 'how many features can I add' because we've tied self worth to work output instead of some other more personal or human rubric. AI will happily pile more and more and more code on your plate because it's a buffet. It's gone through reinforcement cycles guiding it to add whatever feature you ask. There is no training for elegance or careful considered design. Nobody is teaching this generation of LLM restraint.
The safest code is that which never runs. The most elegant feature is that which is not necessary.
"The safest code is that which never runs. The most elegant feature is that which is not necessary."
I assume you run GNU Hurd as your OS? ...
In general I very much agree to the quote of the Little Prince's creator, but I am not sure if we mean the same by it, because you seem to value code elegance more than adding useful features?
Now "useful" does the heavy lifting here, but my point is a useful feature is useful, even if it is achieved by the most ugly unmaintainable code. And what counts as a useful feature will depend on who you ask, but in general you should ask some users. (They don't care how the code looks)
And sure, they do not enjoy a pile of garbage of new features that only work sometimes, but managed to destroy their existing flow. But "Nobody is teaching this generation of LLM restraint" this is hyperbole and up to each developer. There is no need to let the AI run on auto for hours, this is only one way how to use that tool. The other way is, you make the design yourself - and let the AI implement it.
blueblazin 2 hours ago
- Python interpreter in Rust (fail): https://github.com/BlueBlazin/pyrs
- Low-poly Google Earth like engine (success): https://github.com/BlueBlazin/lowpolyearth
- Codebase -> 2d infinite-zoom visualization (success): https://github.com/BlueBlazin/blockviz
I have never understood or liked all the obsession with one-shot builds.
codazoda 51 minutes ago
I took it to 39% complete, as a language, over 8-weeks. Then, I spent just 50 minutes describing all the missing parts and it finished the language in 1-shot. I didn’t think it would work. It did.
I used Opus 5.5 on a $20 plan and I didn’t even hit my limits.
martin-adams 1 hour ago
https://app.preview.craftspec.com/app/city/
This whole thing is just a proof of concept, and needs a mouse to interact with.
It’s incredible what AI has made possible with experimental projects that would require months of coding.
genxy 2 hours ago
To get a toy version that can run most things takes under 2 hrs. Half of that time is collecting the tests you want to pass, how it should evaluate the code, what your criteria are, etc.
Seattle3503 27 minutes ago
infinite_spin 1 hour ago
stelonix 33 minutes ago
[0] https://jsfiddle.net/3yLrsem0/
verdverm 2 hours ago
Same, and I have found some of the failure modes in the realm of eyes free coding and limited auditing. I often see it more as pushing the process than the model, as I tend to bounce around models for the benefits diversity provides.
Have you also found there is a point where they hit code complexity and start churning without progress? (at which point you have to step in to help the bot step back and think big picture, which usually takes a few sessions)
blueblazin 2 hours ago
I was motivated to make the codebase visualizer linked above to help me understand how the various pieces of the system that is being vibe coded connect with each other.
Now I'm thinking about how I can build something similar for agents. How can we teach "systems thinking" to a model?
XenophileJKO 1 hour ago
I've been mostly working on very large modeling projects (think detailed architeture over 1000 acres) or animation because this really was the "new" capability area that Opus 5.5 "unlocked".
I have been surprised how well it has worked at scale with many sub-agents. However the code base is limited to the rendering engine mostly.
Certainly in the older models they hit a complexity limit, but I haven't really hit it with Opus 5.5. I have had it make data flow designs that I didn't like because they were more complicated than it should have been or the perenial problem of trying to make everything backwards compatible when working on greenfield refactors, but I haven't seen it choke on a code base yet.
verdverm 58 minutes ago
On the other hand, I have a TS only project core is sub-20k LoC (NES harness), where it is struggling (for now, see other comments)
cwnyth 1 hour ago
It helps us keep track of things, why not AI, too?
verdverm 2 hours ago
I've been working on process more than model. Similar to having a PR review team, you need multiple teams or processes that run over the code, markdown, assumptions. I still don't think this is enough.
I'm working on an NES harness, and the agent was churning with detecting certain state changes like dialogs, menu navigation, screen change. The bot and I got churning and I put it down for a while. A few days later it occured to me to ask the bot "Does the emulator and game state change with frames? Can we just get the state each frame rather than trying to be clever about it?"
It said yes, but added some nonsense about being performance hindered... (flash model). I haven't worked on it again yet, but my intention was to next have it build a small recorder based on this so it can get frame accurate game state change.
This is just a long way of saying, sometimes you have to step back and help them help themselves. They have a really hard time with recognizing ambiguity or how to determine lack of context that could be filled.
j2kun 2 hours ago
Perhaps we should be more ambitious in the types of problems we try to solve, and the types of value we try to provide with the software we build.
devsda 37 minutes ago
I was hoping that someone with enough motivation and money for tokens would come up with a non-electron option that is similar to onenote, with free form note placement and handwritten notes.
Basically I wish there is an opensource app having good feature parity with OneNote. Thats a good benchmark to have.
Thanemate 11 minutes ago
Today I wish the same, but for someone with enough motivation and tokens instead.
Don't get the wrong idea: I would be glad for a OneNote/Obsidian alternative that's performant. I just lament the days where I'd stare in awe thinking about the levels of wizardry one must get to in order to build something that I admire. Now the first thing that comes to mind is "I wonder how many tokens did this took them to make".
verdverm 2 hours ago
What might help is non-software-first companies valuing developers and engineers more, enough that we could earn a reasonable living which building actually cool things. Not sure how the economics works out on the other side. Can these companies make enough money to support this vision?
forgetfreeman 2 hours ago
greyface- 1 hour ago
verdverm 1 hour ago
verdverm 2 hours ago
forgetfreeman 27 minutes ago
j2kun 2 hours ago
verdverm 2 hours ago
lubujackson 5 hours ago
AI can take us from 0 to MVP in a prompt. It can fill in gaps of knowledge and allows someone who knows half of the stack to build the whole thing nearly seamlessly. These are superpowers but don't by themselves lead to new products.
What AI cannot do is remove the world of complications that happen after the MVP - product feature tradeoffs and choices, how to handle integrations and customizations, how to serve multiple types of users, how to communicate and market the app, how to handle security and data storage, how to deal with customers and bugs, etc. All of these need to be considered and acted on by a human in some fashion. AI can help with each concern but can't solve them without understanding the concern and determining how to frame it in a prompt.
This is why we have a glut of amazing demos that go nowhere while we figure out how to build software differently. This will take some time and I imagine the result will be software heavily-structured with a bit of AI reasoning woven throughout, not an army of agents and sub-agents burning infinite tokens.
tptacek 5 hours ago
There's a less hopeful outcome, where no skill is needed at all, because every conceivable thing is one-shot-able from a SOTA model. But then, at that point, you're saying something pretty profound about computer science (roughly: that there was no meaningfully more sophisticated and complex place to go from where we are today).
besterman23 3 hours ago
The most recent new things that LLMs have unlocked (regarding software dev) for us is reverse engineering and combining old games, unearthing decade old vulnerabilities, and helping root and debloat consumer hardware. Which are all cool and useful, but none of it being a “next great thing”.
Perhaps LLMs are really doing is highlighting how the software industry was already nearing its economic maturity and this just shifted the timeline from 2040 back to 2027-2030.
Cyph0n 2 hours ago
1. well specified
2. well tested (which is a form of spec)
3. represented in the training set (i.e. not built on a corpus of human developed examples)
For example, a lot of the compiler porting efforts fall into all 3 buckets: compilers are well specified (there is a set of valid inputs and an expected behavior), they are typically well tested, and as a bonus, in a way that is implementation agnostic thanks to the clean interface (think: integration vs unit tests), and are very well represented in the training set.
visarga 38 minutes ago
There remain some nitty details related to tracking randomness and choosing what interfaces to log. But even having your code testable by replicas would help organize it better and expose different assumptions made by the language or library stack.
When you get different outputs from replicas it can be 1. one of them has a bug, or 2. the spec is unclear -> fix the bug or the spec. So this method catches both sides. It also might be faster than traditional code based testing, you can build replicas in parallel. A replica is an open ended test, unlike coded tests it can be tried on open ended input values.
The core idea is that it is easier to implement twice than to imagine bugs. We already solved implementation, but bug finding means running code in your head, it touches the halting problem issues - can we really imagine behavior from code? I don't think we can do it well. That is why we should just make replicas instead, tokens are getting cheaper and cheaper by the month.
besterman23 1 hour ago
rpcope1 2 hours ago
verdverm 2 hours ago
What I'm sensing is something I might describe as the Oligarchy reaching the point of significant societal pushback, like the labor rights movements that ended 80 work weeks, unsafe conditions, and child labor in factories (i.e. The Jungle).
My hope is that we can use this moment in human history to break the user tracking and targeted advertising industry. It's been so harmful to humans, but in doing so, we will likely break a major revenue source that supports our well above average salaries. I still think it should be done.
besterman23 2 hours ago
verdverm 2 hours ago
iamnotsureok 26 minutes ago
I suspect there are a lot of people really good at “systems” - there are a lot of really clever folks out there - that did not go through the proper CS rituals and are thus shunned but in fact can take over your job in no time especially with AI.
My wife regularly gets pushed away by “IT” at her work and unlike others she pushes back, but they don’t like her. They start talking about db clustering details etc as if that has any bearing on anything. I am a dev, I know she doesn’t know what logical replication is, but I do know she can replace their team. She just needs one technical person to keep an eye on things.
I suspect we will end up with just a handful of building blocks like “this is how data is siloed, secured etc”, “this is how dashboards are made here”, “this is how email is handled”, etc and employees with AI just develop their own stuff on top of these. The “devs” just oversee the building blocks and the overall soundness of the flow. Don’t know how many you need once that is standardized..
verdverm 2 hours ago
I'm reminded of https://archive.ph/a9Cli (ns;nt)
iperov 3 hours ago
TehCorwiz 2 hours ago
"Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away."
And I feel that sums up my feelings on AI this and that. Just as the wood and clay contain the sculpture within and it's the artist's duty and privilege to reveal that. So, is elegant software exposed from the conceptual mass, rather than built up.
The AI zeitgeist currently is 'how many features can I add' because we've tied self worth to work output instead of some other more personal or human rubric. AI will happily pile more and more and more code on your plate because it's a buffet. It's gone through reinforcement cycles guiding it to add whatever feature you ask. There is no training for elegance or careful considered design. Nobody is teaching this generation of LLM restraint.
The safest code is that which never runs. The most elegant feature is that which is not necessary.
lukan 24 minutes ago
I assume you run GNU Hurd as your OS? ...
In general I very much agree to the quote of the Little Prince's creator, but I am not sure if we mean the same by it, because you seem to value code elegance more than adding useful features?
Now "useful" does the heavy lifting here, but my point is a useful feature is useful, even if it is achieved by the most ugly unmaintainable code. And what counts as a useful feature will depend on who you ask, but in general you should ask some users. (They don't care how the code looks)
And sure, they do not enjoy a pile of garbage of new features that only work sometimes, but managed to destroy their existing flow. But "Nobody is teaching this generation of LLM restraint" this is hyperbole and up to each developer. There is no need to let the AI run on auto for hours, this is only one way how to use that tool. The other way is, you make the design yourself - and let the AI implement it.
adwn 40 minutes ago
Yes, but the most useless software is that which doesn't exist.
At some point, you actually do have to write some code and add some features, or else the most satisfied client is the one who hired your competitor.