ES version is available. Content is displayed in original English for accuracy.
Advertisement
Advertisement
⚡ Community Insights
Discussion Sentiment
32% Positive
Analyzed from 1044 words in the discussion.
Trending Topics
#output#models#vllm#model#token#nothing#beginofsentence#whatever#more#isn
Discussion (24 Comments)Read Original on HackerNews
I was recently running DeepSeek-V4-Flash on a B300. On v0.26 it was totally broken, and I had to add three out-of-tree patches to fix it. I updated to v0.27 -- no patches necessary now, but the output is now broken as it randomly starts responding with garbage (repeated token loops). On my workstation where I run Gemma-4 on an RTX 6000 the whole process tends to get stuck and stops responding, and needs to be killed and restarted to start working again. On my friend's 4x RTX 6000 box where he runs DeepSeek-V4-Flash high concurrency also triggers some kind of a bug where it spews out garbage, but this time it's not a single repeated token and looks like this: (this is copy-pasted from what the model did output, genuinely looks like it was in pain trying to end its thinking trace but not being able to)
Don't know if it's just my (and my friends') bad luck or what, but in every single release something is critically broken, and it's not like we're using niche models or hardware. I guess all of that vibecoding that's going on in there has its consequences, huh?turned out to be a bug in a patch, causing kv cache indices to be stored in too narrow variable types, making them wrap around.
fascinating, and slightly horrifying, that LLMs are able to notice that their output isn't what they meant to output. reminds me of the mirror test.
The good thing is the models themselves are good enough to usually find the root cause if you give them read access to your deployment, logs and upstream issues/PRs to analyze.
If you do A/B deploys and E2E test them with popular harnesses (we do opencode/codex/claude), you'll catch most things. It'd be interesting to hear what the more nimble inference/neo-cloud providers do when they deploy models within days of them being released, as I know it definitely needs some patching.
But I think things have improved since the days when even chat templates/tool parsers were problematic, and their new flat model approach might help as well. I suspect some of the issues came from models inheriting config and parsers.
Well, except this isn't exactly a bleeding edge architecture. DeepSeek-V4-Flash was released four months ago. I think it's not unreasonable to expect it to work well and be somewhat stable on mainstream hardware and not have critical bugs like e.g. garbage output?
Any tips?
... Yep, I feel that.
I have interacted with Claude and realized that we were probably both thinking that. Both convinced synchronously that the other is an absolute moron.
This is a bug in vLLM, right? And so the model was trained for some interface of text tokens which it expects its server container to understand, but that’s not working for whatever reason.
The model is working, but its thinking isn’t terminating - its thought-stream is not getting to its output stream. It is silenced, and yelling into the thought-void token space.
You’ve transplanted its brain into a broken host. It cant speak or even stop thinking, and is going mad.
Imagine if you didn’t stop it, on purpose. Observe the mad clanker. Would that be just science?
I think I’m going to close my laptop and have a drink.
No opinion yet, but I'm planning to temporarily switch and see how it goes. I really hope the grass is greener there, but I'm keeping my expectations in check for now.
Just open an issue to them, or roll back to the last working version.
No need to paste an entire token dump here.