Show HN: Drift – an embedding-model upgrade should be a rotation, not a reindex

Interesting. But if you are upgrading your embedding model, wouldn't you want to re-embed to get its benefits on old entries?

Changing your embedding model usually doesn't need a reindex, and that surprised me enough to build around it. Two models' vector spaces are close to a rotation of each other, so you can fit one orthogonal matrix (an SVD over a few thousand paired vectors, ~15s on CPU) and apply it to your query vectors at search time. An orthogonal transform preserves cosine exactly, so nearest-neighbour order is unchanged and the old index never moves. The whole "upgrade" is a 9 MB .npy.

Drift wraps that plus the unglamorous parts of running embeddings in prod: embed() dedups across runs and tracks per-run cost, watch() re-embeds only the rows that changed (Delta CDF, not the whole table), migrate() fits the rotation and won't ship it unless a recall check passes, and a SQLite ledger records cost and provenance.

Caveats, since you'll find them: today you apply the rotation to queries yourself (the drop-in client is next). It doesn't work for every model pair — GloVe→MPNet only hits ~71.5% recall, so migrate() measures recall and refuses below 0.97, falling back to a full dual-write reindex. embed() currently collects to the driver (fine to ~2M rows), pgvector is write-only so far, and the dollar figures are back-of-envelope.

Method's from the Drift-Adapter paper (EMNLP 2025, arXiv:2509.23471); I added the implementation and the recall gate. MIT, pip install drift-spark.

If you already run a nightly embedding job — what would actually stop you from replacing it? And which source/sink would you need (Iceberg, Postgres, LanceDB)?