Running Kimi K3 on MI355X at Better Performance per Dollar Than B300

It’s not a complaint, only an observation. Operator aside, to observe that the models behave differently shouldn’t be controversial, should it?

It seems like you read my comment as dissing Kimi. Kimi is in regular rotation for me. Some tasks that Claude used to own go to Kimi first now. They are just different tools with their own strengths and weaknesses.

That’s not really true. The models have source code defining their architecture and it is open-source.

People keep trying to shoehorn OSS concepts onto model weights, but the concepts don’t fit because the weights aren’t software. They aren’t compiled code. They are learned parameters to use with a (very big) function that itself is expressed in the code.

So they’re a very valuable asset that complements the code, but they are not the code. You could use randomly initialized weights and the software will work - it will output tokens. They just won’t have useful patterns.

I don’t think OSS definitions have ever required that assets have their source included. For example, artwork is very important to a game, but nobody thinks a game is not open source if it doesn’t come with sketches and a copy of Adobe Illustrator to recreate the artwork from scratch.

Edit: The distinction I would draw between models like Deepseek and models like OLMo is whether they are open science. With a model like OLMo, they have published everything you need to replicate the training experiment. Whereas Deepseek does share a lot of knowledge, but keeps a lot proprietary too.

My understanding was that there was a lot of non determinism in the training process because of many factors: how GPUs optimise floating point arithmetic, how the compute is distributed between the thousands of GPUs etc

I'm not sure how static training data is either (or how you'd distribute it considering its size and nevermind the legality of sharing copyrighted things).

You'd likely get a model with very similar behaviour but the weights would be different.

Please someone correct me if I'm wrong.

There's no program you run to 'make the thing'.

It's mostly ad hoc scripts and some pretty horrible hacks being run by a hundred engineers trying to improve a thousand different things at once with a hundred thousand GPUs.

I'm sure once we understand the tech better, the training process will look like running a program.

Even though none of us could actually afford to train it?

As an example, Grok 4 took $388M to train

> Is it the preferred way only because training is expensive?

No, it's the preferred way because that's literally how you train it. Contrary to popular misconceptions, you don't "compile" data into weights. You initialize a model (based on architecture, config, etc) and then you modify it via training. But crucially they (i.e. model creators) modify it the same way (technically speaking) as you would. That's what the license grants you. Nothing less, nothing more. The "how" as in knowhow has never been something covered by a license.

Open-weight is something dreamt up by people misunderstanding the basics of model creation and training, and having ideological biases against AI and/or LLMs. It is what it is, but you should know that you are technically wrong. A model released under an open source license (Apache, MIT, etc) is an open source model. Because the weights are the source of the models. Training is not "compilation". Training is the "how" as in knowhow to modify the model. Training deals with values. Source deals with everything, including values.

In the past, if someone would have released a piece of software (say a PID controller algo) w/ hardcoded values, no-one would bat an eye. LLMs are just that, with billions of hardcoded values. Nothing less nothing more.

Per the OSD definition of source code, "the source code must be the preferred form in which a programmer would modify the program." which means an argument could be made (as charcircuit is making) that the weights, being the preferred form to modify, are the source.

I do prefer open weights as being more precise (like, is it even really software that has source code in the first place?) but I feel like at this point the ship has sailed somewhat (though if this is something you're willing to spend your time arguing then... moral support I guess?)

I think you are failing to see how the weights are the preferred form of working with a model. It's like if I shared a png with others to use as a meme template. Even if I don't share the source code to photoshop other people can use that "binary file dump" to make derivative images of their own with it.

Agreed, The spirit is fulfilled, as evidenced by the massive ongoing development of, effectively, new weights grown out of the sourced weights,

So is a zip file of source code. There is no 'compilation process'. Model training isn't a fixed end point. You start with random weights. You train. The weights get better. You do this iteratively. At some point you say 'good enough' and release. People doing further training would start with those weights and further iterate. Demanding the original training data and training process would be equivalent to demanding a completely replay-able sequence of editing steps starting with a blank text file and allowing you to replay ever edit that led the original author to the released source code.

Now there are reasons you might want to know about the training data when you wouldn't care about the authoring process used by a traditional open-source process. And these get at the reason LLM's are different than traditional software and so maybe our existing definitions of what 'open-source' means aren't a good match for LLM's. Of course there is software associated with LLM's (beyond the weight) -- defining the structure of the particular neural net those weights fit into. In every open-weights model that I'm aware of that software is open source (though trivial).

if you did that in the web interface, could you share the chat? I'd be interested to read it.