Reducing Doom Loops with Final Token Preference Optimization

Discussion (4 Comments)Read Original on HackerNews

johndough•about 3 hours ago

It would be great if this could be combined with quantization-aware finetuning. In my experience, Qwen3.6-27B has much fewer repetitions at Q6 quantization level as compared to e.g. Q4, but that leaves little space for context on my 24GB RTX 3090.

storus•about 2 hours ago

They are orthogonal; preference optimization like RLHF can be done on the base model which can later be quantized, or it could be done on a new LoRA that is then converted to QLoRA.

carterschonwald•about 3 hours ago

this is pretty cool. i think part of the root cause is current rlhf post training design around confidence and optics rather than cooperative transparent honesty. though its kinda an expensive hypothesis to dig into as a private individual

nullc•about 1 hour ago

Most of the models where people are concerned about don't do this when unquantized, so I doubt it's much about the metapolitics imposed in reinforcement training.

Reducing Doom Loops with Final Token Preference Optimization

⚡ Community Insights

Discussion (4 Comments)Read Original on HackerNews