Testing distributed systems with AI agents

Idempotency is what bites me most in practice — I've been driving these against an unreleased database I work on. The main trap is using the op_id as the idempotency key rather than a business key the client reuses on retry. When they're the same thing, the checker is trivially true and the test passes without testing anything.

No-lost-ack is conceptually the same shape with a simpler property (every acked write shows up at the end), but it breaks the same way most checkers break — if the recorder treats timeouts as success or failure instead of "unknown," real lost writes silently disappear.

Recovery after partial failure is where the AI-agent angle gets shaky honestly. Quiescence is the hard part. Agents will declare a system "recovered" while compaction is still running in the background. The skill forces a three-part check (no in-flight ops, no pending background work, replicas converged) before the invariant runs. How reliably that holds up against a specific SUT, I'm still figuring out.

I think all these scripts become poor where they're context based as opposed to actual guardrails; what we need is various silo'd protocols like a ssh protocol that keeps the harness producing work through the protocol rather than a bunch of loosely based bash scripts, etc. Plus, the harness needs to be outside the environment so it's not something you have to install ever on a remote system, whether it's a container, a vm, a ssh location. We shouldn't base everything around running bash without a secure tunnel into the location of interest.

The failure mode of these tools is self destructive in many cases.

I've been specializing in distributed systems for nearly 35 years. I've read your work, and it's shaped my thinking. When you say you have a person in mind when you write, I am that person. Thank you for what you've done.

I don't think this replaces you. The hard part of reliability is understanding the failure modes in the context of the business. No one has unlimited time or money, we always have to make tradeoffs. Only experienced humans have both the ability to interrogate the stakeholders and a vision broad enough to understand what to pursue versus what to give up.

Tools like this make the grind part of the job easier. They do not replace the holistic view you need to be able to confidently tell someone "worry about X, do not worry about Y".

I relate a ton for what you are going through.

I discuss a lot of stuff, but that is because I am a nerd at heart, and rather play with technology, read papers, podcasts and stuff, than whatching depressing TV content.

However in the world of enterprise consulting a similar trend has been happening during the last 20 years.

First offshoring, then raise of cloud based infra, serverless, SaaS and iPaaS, and now AI based orchestrations on top of iPaaS and serverless.

Meaning for the same kind of requirements, a team playing puzzle with those kind of products can be reduced to one third of what it used to be required about a decade ago.

Then what happens to the other two thirds that now don't have anything to do, and whose salary is used instead on those licenses?

Yeah, I feel ya man. It’s a weird time to be in tech when it often times everything feels like the last years of our work are now instantly reproducible.

I’m not sure i wanna stay on the ride much longer, at least in a corp setting. I guess i don’t have much of a choice.

Thanks for Jepsen, though, it’s made a couple of my applications much better in ways I wouldn’t have managed without it; even if I have to relearn clojure every time I pick it up, and those applications resulted in real jobs and careers for a bunch of people. It’s not going to pay for your house, but it’s all I’ve got.

hugs.

This sucks to hear.

I honestly think the rise of LLMs will be the death of open source in the long run. Already, apparently, quality of OSS has dropped significantly since 2025 (so most models stop training on github after this).

I don't think a lot of OSS authors quite understand the extent to which models like claude/codex rely on their work. I'd bet money there are extensive curated tasks using your tooling for post-training. With 0 attribution or anything, these models are using your work wholesale to build sophisticated agents that can do your job.

Yeah it's depressing as hell. I guess it's the same thing for artists and musicians and writers.

P.S. I can symphathise with the old house issues! I bought a 1901 terraced property, it's an absolute money pit.

There are a lot of create open source projects that have a paid infrastructure product that lets people pay to use the core tech. Perhaps you could productize your system. I would personally to pay to have something set up and useable so I don’t have to. I think the pattern you describe is clearly valuable

I have no idea if it would be fun or a good return on investment for you, but I would happily pay for a digital version of your distsys class aimed at practitioners. Some kind of ebook, perhaps with accompanying whiteboard/lectures.

> And if these LLM tools work, it's good, right? They find bugs, systems get safer. I want systems to be safer. On the other hand, I'm motivated to share what I do because I really want to help people. If it's just LLMs... it feels hollow.

I get that you have a financial issue, but perhaps you don't need to be conflicted about about open-sourcing your work as far as helping people goes? LLMs are tools for people. Code, research, standards, etc... are all means to an end. Maybe the agent operator doesn't read or understand your work, but the guy who built the agent skills likely did. Progress moves upward, while standing on the work of those who came before us.

LLMs have lowered the barrier to creating software and can hide a lot of source material, but your work is clearly having an impact here. If your goal is to help people make better software, that's still what's happening. The industry shift is happening regardless, so we might as well embrace the positives instead of focusing on the negatives IMHO.

Moving to a closed-source model for financial reasons is a totally separate issue IMO, and I wish you good luck and prosperity regardless of your decision.

It’s shocking to me that you of all people should have financial issues. You are a legend in the community! By all means, take things closed source if it even just helps a little bit. As a profession we’ve all been hurt by over-sharing.

unfortunately, according to the cults in San Fran, your only choice is to "git good" less you become part of the permanent underclass.

In other words: fascism is coming and you either lick the boot or you get stomped.

OK so let me get this:

1. You chose to give your work away for free.

2. You are complaining that you haven't made money from your work.

Is that a fair interpretation of your argument?