Beyond Recall and the Illusion of Competence

>Let the AI write the code, not design the solutions.

I don't think design is necessarily out of reach for AI. As usual, the closer you get to the bleeding edge of what's been done vs. what's possible, the more thought needs to go into a design for it to be considered good. And TBH, even if you only make it to "it works", that's still laudable. There's plenty of profitable companies with terrible designs, technical debt, and broken systems built before AI, and that's why often these arguments fall flat for me.

I like the idea of mostly getting average designs out of AI. I am tired of running into overly complicated solutions to seemingly simple problems. Microservices vs. Monoliths for example. Maybe they thought they saw something that needed that complexity which was credible before, but they have gone and left someone else with their decisions, and left any credibility to the void. They're just playing the game though because there's is/was an incentive to get micro services on your resume for the next job.

> The developers who stand out will be the ones who understand the systems.

That has always been true. AI doesn't change it. It just rearranges some elements that come ahead of that understanding. The volume and velocity are challenges but the understanding and judgement is what separates the good and the great.

BTW: "systems" in my world include not just the technology components but also the humans who operate, adjust and manage all the pieces, and the processes that control it all.

I fall somewhere in the middle of the spectrum of developers the author describes. I use AI daily for limited code writing and lots of debugging. I agree with the author that things go off the rails when you stop truly reviewing its output and just click accept to get the thrill of productivity. I don’t think using AI for debugging means you have to do that though, it’s still a choice. As an AI-human team I have had the experience many times now of debugging an issue in a complex system that I simply don’t think I could have done alone or in a reasonable amount of time. I don’t think this is a reflection of my low abilities - it’s because the AI brings to the table skills that I’ll never have like reading through and correlating huge amounts of logs across many runs of the same system on different compute nodes in a cluster. Once it finds a needle in the haystack I can still take the time to understand and reason about what it found though.

> AI can create the illusion that you understand a system because it can make the system work for you. You ask a question and get an answer. You encounter an error and get a patch. The patch doesn't work, so you provide the new error and get another patch. Eventually the machine finds something that works.

> From the outside, this looks like competence. But if you don't understand why the solution works, you're not actually becoming more capable. You're becoming dependent on the machine to maintain the illusion.

So many things in life operate exactly like this: We don’t need to understand the why, we just accept the solution as-is and move on. An experienced professional has another skill: keeping track of which things matter enough to dig in to and develop mastery, and which things can be “good enough”. This is also informed by the “intuition of how machines behave” that the author mentions.

In your closing thoughts, you don't mention what is this staying firmly in control? how do you stay firmly in control while reaping the benefit of the model while being more productive?

because if you start a curriculum over what the model wrote then, one can argue, you are better off writing it yourself in the first place.

I think I do agree with the central point but a point of order is that if you tell AI to fix the bug and then tell it to fix the next error you might still be building an important skill: learning how to make the AI get good results.

The hardest part is knowing what you forgot. You can't check for gaps you don't know exist.

I realized recently the LLMs are calling bullshit on the moniker of engineer or programmer that I've bestowed on myself over the course of my career. The current LLMs are making it quite easy to seperate software design from implementation. I used to be firmly in the camp that one really needed to roll up their sleves, sit down with an editor and start banging away until the software evolved and coalesced around a solution.

I convinced myself that functional specs and detailed design documents were not needed because they wouldn't be kept in sync with the code. But the LLMs are essentially turning the synthesis of the code from those documents to a compilation step of sorts.

Now I must ponder the question, what portions of the acts of coding, designing and delivering a working product were the portions that bring me joy. I've been attempting to answer this question by making a concerted effort to delegate the coding to the LLMs and reviewing if the code conforms to the designs I've written down. This process is much closer to the historical engineering disciplines but I have to say, its not been easy.

My advice to the the more junior reading this. Experiment with different functional and design spec formats that best serve the LLMs and develop the skill of writing these and then managing the LLMs.

TLDR, the LLMs are giving the term Software Engineering actual meaning.

> And the consequences become particularly serious when something happens that the machine can't solve

Sadly the bot-gulled will always say "But the next model will be good enough to do that!".