Continuous Diffusion Language Models (CDLM's)

janalsncm•about 1 hour ago

> [in 2020/2021] the dominance of autoregression was not as well-established as it is today: GPT-3 had turned some heads, but the ‘ChatGPT moment’ wouldn’t come until late 2022

I disagree with this. Decoders were absolutely dominant in 2020 for chat. GPT2 was considered too dangerous to release, and I remember scrambling to get on the GPT3 waitlist. It worked.

(The only exception I will make is encoder-decoder models which now are often done by decoder-only.)

But what made it go mainstream was RL. RLHF at first, then other improvements like DPO that were less of a pain in the ass to set up. Adding diffusion on top of that would be an even bigger pain in the ass.

Before ChatGPT there really wasn’t much of a concept of pre-training and post-training. It was all pre-training. Post training was what made the bots conversational and not just “continuing the thing you wrote to them”.

So in short, diffusion never took off because it was just a more complicated way to generate tokens, and the real problem was getting tokens in the right distribution.

echelon•about 1 hour ago

> GPT2 was considered too dangerous to release

This is how ridiculous this industry is. Regulation-seeking panic over nothing. Drama in search of a moat.

Everything is "too dangerous". GPT2 is going to invent a time machine and break crypto and genetically engineer super rabies.

They sell knives, guns, combustible materials, and multi-ton heavy machinery in stores. That's what's actually dangerous.

janalsncm•27 minutes ago

My impression at the time was they were perhaps overly cautious but this was a bunch of researchers who wanted to self-regulate. Anthropic didn’t exist, deepmind was also much less product-focused and relatively cautious. Chinese models weren’t really a factor either.

Government regulation was really not in the picture either in 2020 or 2021 tbh. The government was still trying to beat a pandemic. Some of the Biden admin eventually wanted to but it wasn’t very serious.

p1esk•about 1 hour ago

It’s refreshing to read something not AI generated.

NickNaraghi•about 2 hours ago

I wonder if we’ll get something like CDLMs for automated harness engineering, sort of piloting the LLM underneath.

ramon156•about 1 hour ago

how do tools like hermes do this? does it just review sessions and rewrite markdown files?

also haven't read too deep into the deepseek agent harness but the math in there was really cool. it sounded promising, at least.

amelius•about 2 hours ago

"Attention is all you need" should be renamed into "Attention is sufficient but not necessary".

p1esk•about 1 hour ago

It’s the opposite: attention is necessary but not sufficient.

ViscountPenguin•about 1 hour ago

A quick look at the continuous diffusion models linked in the post shows lots of transformer models still

Continuous Diffusion Language Models (CDLM's)

⚡ Community Insights

Discussion (9 Comments)Read Original on HackerNews