DE version is available. Content is displayed in original English for accuracy.
Advertisement
Advertisement
⚡ Community Insights
Discussion Sentiment
55% Positive
Analyzed from 3376 words in the discussion.
Trending Topics
#agent#code#agents#more#codebase#capa#context#should#test#going
Discussion (58 Comments)Read Original on HackerNews
It's simple: when a mistake happens, you run your CAPA process (Google CAPA form and see examples to extrapolate what that process might look like) and determine the root cause and the correction to the process that allowed the mistake to happen in the first place.
(At least as a SaaS vendor in life sciences, when we had a CAPA (e.g. after a SEV0 failure), it would be folded into our SOPs and then we would be required to retrain on the SOP. Auditors would want to see our evidence of CAPAs, the versions of our SOPs, the records of training. All to extreme for most shops, but I add this for context/color)
This is something most eng shops do not have the discipline for since it requires some diligence.
Should it be fully agentic? Should there be human intervention here to approve the CAPA? Open questions to be answered.
It’s all very well having a list of actions to avoid but that doesn’t help if your agents won’t reliably follow it.
You need two pieces:
a) prompts, that tell the agents what to do and how to do it (and ideally, the why, where, etc, the full picture) - that's the positive half, that drives behavior the way you want it.
b) deterministic tooling that prevents negative outcomes, like linters, compilers, static analysis, fuzzing, testing, the more the better. This side should either be firewalled off from the AI or very carefully watched so that it doesn't drift.
The part that you put in the deterministic side is the "never do x" stuff - I have lint for long comments (which AI hits every single time it commits), all my dev scripts are in typescript, precommit hooks, massive CI, and I lint even for things like redirecting error to standard out, tiny stuff, and also e.g. static migration analysis so the AI never ships an exclusive full table lock in a migration, for example.
You can’t keep humans from making those errors either but you also don’t let an error prone human crank out 20k LOC per day without forcing other humans to understand it.
In test cases i can do anything, a test framework is just a way of discovering and then scheduling functions to run. I can emit useful instructions to the agent from the failed test case: "After walking the AST of all use of state machine X, a branch was found at Y which reused stale state. Ensure stale references are dropped..."
I can force the agent to pass the test suite before it considers itself done. I can reject edits of such test cases to partially mitigate reward hacking. etc etc
The CAPA is a learning that sits outside of the mechanism of verification; it is a record of problem:root_cause:preventative_action. I see it as the instruction that would be required to generate the test case to prevent the next occurrence of a class of failures.
In a real-world process, for example, there is usually a QA lead that is verifying that the process is followed by looking at the paperwork and evidence.
How would prevent removing the test case and the code that it tests? That's a legitimate thing to do when you're modifying a codebase. My intuition is "corrective and preventative actions (CAPA)" is a level or two up from test cases.
One approach, for example, might be to have the a standalone code reviewer agent that is solely responsible for interfacing with the CAPA system (e.g. via a tool, via MCP) and acts as a back stop. When it finds a new type of CAPA, it stores it (and the backend indexes it with enough metadata to support broad types of retrieval). When it reviews a piece of code, it finds past CAPAs. By file locality. By business domain in the application. By keywords.
Same tool and repository available to both building agents and review agents, but use the review agent as a dedicated back stop as part of the verification process.
It will be interesting to see how the notion of 'responsibility' carries over as agents handle higher and higher levels of abstraction. Agents built using the popular frameworks of today are somewhere in between code, formerly written by engineers, and an actor (like an engineer). When an agent wrote the framework to audit and monitor ops agents who are monitoring the performance and reliability of the code that coding agents wrote, who is responsible for ultimately ensuring it doesn't happen again? Should the findings just be added to whatever RAG and a few prompts/hooks/skills changed by another agent? When does a human follow up and to what degree?
This is the sort of thing that makes me believe that software engineers will never truly go away; at the very least, they serve the very useful function of being ultimately responsible for something going wrong.
I personally use it for the agents I use, so why did it fail, what was the root cause, what can we do to prevent it again.
I think a point we are grappling with is, what necessitates human intervention, like philosophically. Is it accountability? I was thinking about this in terms of code review and it's not like we would fire someone if they broke prod, so at what point do we need someone accountable. Maybe it's for approvals for certain classes of risk (like those associated with actual harm to life). It's also not lost on me that many human systems lack accountability.
I once worked at a very large company, and one of my coworkers was like "eh, I'm not going to do any of this work, because I don't have to", and that was a clear indicator that the problem wasn't (just) with the system, but with him individually.
Would love to be proven wrong here, feels like I've just seen the same basic one-level kart racer in like 3 model announcements. Not even sure if they were one-shotted, I certainly would not describe them as "fun"
You need to playtest the hell out of games. Play a bit, iterate, play a bit more, iterate, etc. You will get way better results if you start small and don’t try to one-shot it. Start small, vertical slice, playtest, and go fro there.
It's definitely not fun if you consider building an actual community around it and have a long term plan for it.
https://github.com/agent-ix/engineering-assurance https://github.com/agent-ix/quoin
One is to take the human out the OODA loop for cyber defense. Servlet libs (for example) are going to become fluid, self-modifying things w/ contracts that operate much differenlty from how they do today. The engineering practice around these things will need to change.
The second is that UX will be self-modifying. Just like how pi can modify itself, I can see this being a general practice for user-facing applications. Perhaps a text-box in which users can describe tweaks in can request changes to how the application functions. Engineering an application will focus on modeling the non-negotiables of how an application works, and providing the correct primitives for user-driven LLMs to modify software on the fly, as well as track and rollback changes. There'll also be funsies around how to ensure that ads get delivered regardless of the user trying to get rid of them.
This smells like worm food. Those contracts had better be airtight.
As a random sample of one, I looked at one of the bugs this reported on Tailscale (first thing on the homepage) [0], and the pull request ends with "Apologies for the lack of due diligence here. I'll go ahead and close this out."
[0] https://github.com/tailscale/tailscale/pull/17843
Anyway, here's mine, still wip:
https://hale-lang.org/docs/dna/
https://github.com/hale-lang/hale/issues/690
Why? Because LLMs are always going to be dumb when they're trained at scale. Their ability to speak software diverges from their friendly user input layer. A harness won't overcome that, but an LLM saddle ontop of a larger model would provide the type of feedback loops you'd want to look into.
I don't think you'll find two deterministic systems will produce much.
I'm working towards both in my homelab to see which works better with little qwen
I don't get why we need global memory for code? Aren't code comments (even if invented for humans) the ideal place where to put "memories"?
In other words, there's a story-document between a SherlockHolmesBot and UserPlaceholder that's growing like a crystal formation. Some software sees "Sherlock Homes Says" and then "performs" the dialogue at us, and we then assume Sherlock Holmes exists with a mind and memory, rather than being a facet of a text-story.
1. Some context doesn't have an obvious place to write it in the code. If you're explaining why a tricky function is implemented a certain way, you can leave the explanation above the function or within the function. If you're explaining something more general, there might not be a natural place to put it.
2. Relatedly, some context doesn't have an obvious location to read in the code. E.g. you can comment on a schema that a table is intended to be append-only, but an agent could easily miss this comment if it doesn't gather context all the way down to the raw schema. Any research process has unknown unknowns. There may not be a canonical place to look.
3. Adding comments for every single human intent might be a bit noisy. E.g., if a code reviewer flags something that I know isn't a bug, I want the system to learn from this, and I want that knowledge to take effect outside of just code review, but I might not want every single code review thread to yield a codebase comment.
In addition to all the product details that aren't in the codebase or docs, like "keep this logical path because it is used by our one big client."
Mostly what I think is needed is a richer worldview available to the LLM so it understands not just the code but can understand the product and its real world usage and constraints.
We wrote this post as part of a launch, which you can check out here: https://x.com/danlovesproofs/status/2095182189499711759
How can you self drive an app on windows? There's no clear UI framework, design pattern, nothing which can bullet proof your app.
I bet you'd have better luck on Plan 9.
Remove the choice. Make frameworks which have limited options which are usable by default.
I recently built a calculator to try to quantify the gains an org could expect as they become more "AI-native." I tried to account for a bunch of things including env, which I call "AI roadway," but I'm definitely missing codebase readiness (thank you, author), and also bottleneck analysis. Even without those considerations, though, gains are usually modest. It's hard to get to 2x.
Tool is here if useful: https://timvasil.com/ai-native
In general I've seen other issues like this where small errors and irrelevant comments in the codebase spin out into larger problems that consume annoying amounts of time/tokens. Maybe Anthropic and OpenAI don't notice this because they're in an "infinite monkeys with typewriters" scenario, but it's noticeable to me when the agent in my CLI has been spinning for 15 minutes contemplating irrelevant details
Sometimes I feel like it would be best to only give the agent access to clear API boundaries (say public interfaces to certain modules) and let it work out a new system from scratch given the expected inputs and outputs. Then plug this independent solution into the system. Of course it can still overfit these interfaces but it’s less than having access to the entire codebase.
Thinking about it, maybe it’s possible to let one agent extract those interfaces, then use them as grounding for a new session.
One less nice way of achieving the same is to tell the agent once in a while that it should think completely from scratch (from first principles). But this relies heavily on instruction following in the reasoning part, which sometimes works and sometimes fails.
But then there's the cases where the AI can't actually drive. What if you tell the agent to invent AGI? Or time travel? Where does it drive to? Does it tell you that it can't? How does it know it can't just drive here? How you you know? At the end of the day these are not wish granting machines, so someone connected to reality is still going to have to make decisions. And that person is going to be the one held liable for whatever the AI does so would they want a self-driving codebase in the first place?
Any attempt to describe software by "specification" leaves enough ambiguity that the agent will do deranged stuff like add a ton of code to satisfy error cases that can't happen. Or, like you imply, it will confidently create a bunch of nonsense to "solve" a problem in a way that is not really possible.
Without that, "self-driving" degrades into applying diffs that compile. With it, the interesting question becomes what the system is allowed to do when the signal goes red - revert, retry, or stop and ask - and that's a policy decision nobody has good defaults for yet.
So i want claude to build a small game for me so of course i create one big file in which i write everything then i tell claude to analyse it and grill me (grill me skill) to clarify all smaller details.
This alone might lead to really good small sfotware but I still have to push it sometimes.
Now instead of doing this, i tell claude to build a small tool which generates a dashboard and memory and which can save specs and ask me choicses (do you prefer this color over the other, what do you think about problem A? What solution would you prefer? A, b, c or something else).
This does a few things:
1. claude doesn't has to save/store everything in a context 2. claude can now talk to that tool to ask it stuff 3. claude can now use the tool as a todo list 4. claude now can more easily spin up more agents in parallel 5. i have a nice interface and i can solve issues while claude works on unblocked tasks 6. I actually can follow the progress a lot easier
The only problem with this is: with the next update, you have to reinvestigate how claude was finetuned and adjust. A few month ago /goal was really good, now you need it a lot less because claude will do something for an hour without /goal
And the spec file only started to work after November/Opus moment but it got so good, that i can pack A LOT of stuff in a half structured markdown file and let it code what I need.
The progress is still too fast for the whole ecosystem