Codex on AWS bedrock bug causing 10x charges

amluto•about 1 hour ago

Wow, that whole thread is borderline incoherent, presumably generated by an AI without adequate oversight.

Here are the docs:

https://developers.openai.com/api/docs/guides/prompt-caching...

The thread has little explanation as to what weird thing they’re doing to Codex that is making the default work poorly, and it kind of seems like it’s getting confused about whether it wants to set the caching mode or the breakpoint or both.

In any case, I find the behavior change interesting. It sounds to be like 5.5 and below may have been using a conventional attention scheme where a cached KV sequence can be easily used to restore a prefix of itself, but perhaps 5.6 is using linear attention or LSTM or another recurrent scheme where you cannot rewind the model state by just truncating it.

xiphias2•about 1 hour ago

It’s really cool that we have this proof that US companies are half year behind Chinese models in architecture.

DrJokepu•about 1 hour ago

> It sounds to be like 5.5 and below may have been using a conventional attention scheme where a cached KV sequence can be easily used to restore a prefix of itself, but perhaps 5.6 is using linear attention or LSTM or another recurrent scheme where you cannot rewind the model state by just truncating it.

I feel like this is the kind of substantial change to your product that you would need to tell your customers about. It would be simply disrespectful to your customers to not disclose this upfront.

zx8080•about 1 hour ago

It depends on who they consider the customers. Shareholders and govt are the customers, not users.

Users is the product.

zuzululu•about 1 hour ago

i dont know what the hell is going on lately i pop in to issues or discussions and its agents talking to each other or telling me what PR to merge

I mean i use AI too but was taken back when an agent popped up dictating what i should do and so on....felt weird

moralestapia•2 minutes ago

Funny how it always is more charges but never less or no charges.

"Random" accidents that always go against you, too biased to be random.

But don't notice that too much, you might start to see patterns here and there that you're not supposed to see ...

TheP1000•about 3 hours ago

Our codex on AWS Bedrock read / write cache ratio was less than 5%. Cache writes are very expensive and they were never being used. This results in codex on Bedrock causing ~10x what it should due to no caching and massive writes.

The workaround in issue resolved for me: web_search = "disabled"

otterley•about 2 hours ago

If you’ve got a workaround, I’d suggest updating the issue description to have it up top there so similarly impacted users can spot it quickly and benefit.

the_duke•about 1 hour ago

It's already mentioned in the issue...

otterley•14 minutes ago

It’s in a comment halfway down the page. It’s not in the issue description.

If enough comments are added to the discussion, it might end up being collapsed.

chrisweekly•about 1 hour ago

"causing" -> "costing", right?

yablak•about 1 hour ago

Way to bury the lede..

spacedoutman•about 1 hour ago

Something is wrong with the codex app too, burning usage like crazy lately.

ac29•about 1 hour ago

There haven't been any free resets in the past week, there were 4 in the first half of the month

zuzululu•about 1 hour ago

indeed it has anybody know whats going on at openai ??

dgellow•19 minutes ago

Maybe preparing for their IPO?

hk1337•about 1 hour ago

I wonder if it's related to Codex wearing out SSDs.

edoceo•about 2 hours ago

Loaded question: would an openrouter or similar solution caught this before the $BigProblem showed up?

Codex on AWS bedrock bug causing 10x charges

⚡ Community Insights

Discussion (18 Comments)Read Original on HackerNews