Not everything should cost a token: the case for deterministic AI

"wire up an agent to do":

  * pull a metrics API every morning
  * reshape JSON
  * drop results into a table

This means they:

  * know there's an API that they need to call,
    likely a specific endpoint
  * know there's a database
  * know the schema of the database
  * know where the data should go into the database

--

Or perhaps they re-derive this every single day.

Every day, they search for an enpdoint and call it. Every day, they check all the databases that the agent knows about. Every day, they read the full schema of those databases to guess where the data should go. Every day, they invent the SQL to insert this new data into the database in a format the database is expecting.

There are two possibilities:

  * this person exists and is common
  * this person does not exist or is very uncommon.

If you're this person, hey! Welcome to beginning to program and learn about the deeper work that goes into making things fast and easy. It's gonna be really weird, but you might find it rewarding! And, you can mostly keep to being a vibe coder, genuinely; but you're gonna wanna change "hey go do this" to "write me a program that can do this". It may be a higher starting cost; but it'll pay dividends as the weeks go by. Like, dollars a day to pennies a day in server or power costs.

--

But I tend to believe that this person isn't very common (if you're the person that does this, no big deal. You're new to programming. There's a whole lot to learn and you've already started. That's awesome!)

Instead, though, we've got a person that:

  * knows what endpoint they need to talk to
  * knows what the schema looks like
  * probably knows how to write SQL (or doesn't, in which case,
    oh god, they're letting the bot write and execute
    brand new SQL everry daOh god oh god.)

But they still let a bot do it? Every single day? Do they work for a company that subscribes to and/or do they subscribe to a token-maxing philosophy? Because they must know this much agent interaction is going to be not only expensive but also really slow. 10s of seconds vs, what, 3ms of parsing and maybe 100ms of waiting for API and DB communication on the high end?

Maybe you're forced into doing it due to internal office politics and prioritization - this article is going to be great for explaining to your boss why it's a bad idea to keep doing what you're doing. Otherwise, none of this should be changing very often, and this codepath should be, all in all, quite cheap to write - with one exception.

The only situation I could see where this approach the article suggests "lots of people are doing" makes sense and is a reasonable thing an engineer would have done is if the DB and the API endpoints are constantly changing, so they need to rederive over and over again.

But is that really where we're going as an industry? Random, dynamically changing API endpoints? I sure hope not.

> loading a few thousand tokens of raw JSON into a context window, asking a language model to reformat data it did not need to reason about...

> That trap catches almost everyone right now.

Has everyone really become so stupid now? I wouldn't even have thought of doing such a horrible thing as using AI to format JSON...

Needless to say I stopped reading there.