The early History of the Singular Value Decomposition (1993) [pdf]

muragekibicho•about 1 hour ago

For the curious, eigenvalues only exist for square matrices. Singular values are like generalized eigenvalues.

Singular values are like the fundamental frequencies of your matrix. You know how you can define any color with RGB? In a (pretty handwavy) way, singular values are like RGB color codes for us math guys.

Optimizers like Muon and Adam play around with weights' first, or second order singular values to train models.

big-chungus4•about 1 hour ago

Last statement is a bit sus... Muon computes matrix sign function which can be defined as setting singular values to 1, though you can also define it without SVD. Muon itself doesn't use SVD because it uses a faster method to compute matrix sign. Adam doesn't do anything related to SVD or singular values. Also not sure what you meant by "second order singular values"

jmalicki•20 minutes ago

ADAM is related if your second derivative matrix happens to be diagonal.

Of course, it takes about 5 minutes to show that any DNN is going to have very very high magnitude off-diagonal terms by the way it's constructed, so pretending that a diagonal approximation is close enough is crazy.

toolslive•about 1 hour ago

Just going to sound really pedantic here, but RGB does not capture the entire colour space. In fact, it only captures about 35% of the colours the human eye can perceive.

https://www.oceanopticsbook.info/view/photometry-and-visibil...

wtallis•about 1 hour ago

You seem to be conflating "RGB" with one particular RGB color space: sRGB. That's a common enough conflation to make, but not appropriate when you're trying to be pedantic.

toolslive•about 1 hour ago

Doesn't matter: there's no RGB model that captures the colour space. That exactly the reason CIE exists.

piker•about 1 hour ago

Okay, but that was a really useful metaphor if incomplete in a lot of ways. It made me say “oh”.

esafak•23 minutes ago

This is solved through https://en.wikipedia.org/wiki/Primary_color#Imaginary_primar... that sit outside the visible spectrum.

waynecochran•about 2 hours ago

The SVD seems to come up everywhere in my work in computer vision. I find myself continuously using the various C++/Eigen SVD implementations. Actually I should speak in the past tense. Claude and Codex are now generating all my code for me now, and I see them spitting out SVD code frequently -- often for very special cases. SVD truly is an amazing tool.

eigenspace•about 1 hour ago

It comes up anywhere that youre working with data that has some sort of correlation structure.

In image processing, the SVD makes it possible to talk about all the rich spatial correlations in the image, and pick out the strongest ones and discard noise.

This is also why it's so ubiquitous in compression algorithms, and of central importance in stuff like quantum information.

fooblaster•about 1 hour ago

what work are you doing in computer vision that isn't entirely ML these days?

TimorousBestie•about 1 hour ago

> Claude and Codex are now generating all my code for me now, and I see them spitting out SVD code frequently -- often for very special cases.

I find this so annoying. I had to PR some Claude-generated gaussian elimination routine last month and making sure it got the pivoting logic correct was a waste of my time.

waynecochran•4 minutes ago

You are doing it wrong. Have Claude generate the test code and log test data that it can feed back into itself. Claude can generate tests and verify the code better than humans now. I don't trust humans to get things right anymore -- I have a PhD and Claude knows all the math and libraries better than me.

The early History of the Singular Value Decomposition (1993) [pdf]

⚡ Community Insights

Discussion (12 Comments)Read Original on HackerNews