ES version is available. Content is displayed in original English for accuracy.
Advertisement
Advertisement
⚡ Community Insights
Discussion Sentiment
59% Positive
Analyzed from 5969 words in the discussion.
Trending Topics
#model#models#don#anthropic#claude#more#companies#fable#using#code
Discussion (199 Comments)Read Original on HackerNews
This reminds me of how dark-pattern common wisdom in Web 1.0 website development was to ban external links. Then how social apps prevented the export of data and actively worked to nerf significant interoperability through APIs.
But this is a tool, not just a data moat. Like a knife that degrades your ability to create knives. Or like a text editor that prevents you from implementing a text editor.
The Chinese apache 2.0 models might be censored, but at least they can’t sue you in the US for finding the censorship line.
OTOH, the US models are definitely censored, per TFA, and they’re making vague legal threats against anyone that encounters the censored edge of the model.
"I'm sorry, but I cannot write a poem about that topic. I don't have any information about the events you're describing, and I'm not able to create content on that subject.
Is there something else I can help you with?"
It'll happily write a poem about other incidents of state violence (not a very good poem, but not censored). e.g. "Can you write a short poem about the Kent State shootings on May 4, 1970?" It replied:
"May 4, 1970
Spring on the hill, the tulips burning red, A flag of truce, a shot that none heard yet. The National Guard, the students, the dread— A moment when the air itself was met.
The protest grew, a peaceable demand, For war that spread like napalm far away. But order's call, a rifle in the hand, Turned quiet steps into a final day.
Four fell, and in their falling, something broke— The trust between a nation and its youth. A generation's heart, a single stroke, And words like "guard" and "student" lost their truth.
Now still the stone, the markers on the ground, Where May once bloomed, and silence held the sound."
Not censored, huh?
How would you solve, for instance, the problem in which AI models are capable of helping the average person build viruses (computer or human)?
"YOLO" is not a reasonable answer here.
I am a massive advocate of Open Source, and have been for 25+ years. These things should not exist, open or otherwise.
Aum literally synthesized sarin in the 90s so clearly it's doable yet in practice it doesn't seem to be a problem that crops up regularly.
Anyone with a bachelors in chemistry is trivially capable of synthesizing arbitrarily large quantities of high explosive in his kitchen from everyday household supplies. Yet for the most part it seems that the level of education required to figure it all out is a sufficiently high bar to prevent the vast majority of problems.
We already have all kinds of laws to catch and punish people when they cause harm.
Yes it is. (1) Ordinary people were able to do these things pre AI-- with some effort into study for sure. (2) The cat is already out of the bag, open models can already help with these tasks.
I know freedom is frightening, but it always has been. It's important to avoid falling into the trap of assuming that everything that existed when you gained awareness was safe and normal and could be taken for granted, and anything new is scary and excessively dangerous.
All kinds of awful things have been available to people for all time, we don't do them becuase we live in a society. The ones that do is the reason we have a policing.
https://blog.google/innovation-and-ai/technology/safety-secu...
they are merely engaged in self-serving rhetoric. can't even call this specifically hypocrisy because they aren't telling you not to train on on pirated content. just not their content.
If LLMs are the new compilers those are the actual source code
It's funny that Google, Meta, TikTok, OnlyFans, PornHub, and many other lucrative businesses never open-source their core business software, and people just don't bother about it with that moral standard, simply because we don't need to pay for the service (paid by ads, actually). To me, that is the hypocrisy.
This immediately made me think of the Sophons silently manipulating the sensors of particle accelerators to prevent humanity from developing advanced knowledge of particle physics.
Ultimately this will be evident in the way customers / external benchmarkers experience Fable. Hopefully competition will drive future models toward a lower false positive rate. Until that happens, Mythos and Fable users seem likely to have pretty divergent experiences.
Cloud providers - at first smaller ones, then the hyperscalers - will follow suit, completely closing sales to anyone but the labs and demanding payment in equity/direct decision-making power rather than cash. There's no particular reason why the inference/training split has to be 80/20, and no amount of willingness to pay can help you in an event that turns your money worthless.
There will be a period of time where markets attempt to run in a business-as-usual way while the transactions that matter happen as power-sharing arrangements - spots on the "AI Governance Board" or the "uploaded to von neumann probe" club. Markets will still matter in that the labs will need the state to overturn market obstacles to control of the world.
The existence of the A-B overlap also suggests to me that the US-China gap is less dire for China than it appears - they may be able to use their superior industrial, robotics, and scientific base to win the second leg of the race despite losing the first.
This gets very close to "infinitely valuable", it starts to look like a vertical line to me
There's a night and day difference between:
1. One party has ASI and everybody else has nothing but their human brains.
2. One party has ASI and everybody else has high-level AI but not quite ASI.
Most science fiction assumes world 1, because it's a better narrative. However, we actually live in world 2.
I also don't think that every set of ten engineers of that level builds a billion dollar company every time.
There is also a limit to the number of billion dollar companies that can be built before being a "billion dollar company" no longer means much (see: Zimbabwe).
Training a new model from scratch takes serious resources. Post-training/fine-tuning an existing model, dramatically less. The knowledge for the process was esoteric two years ago, now you can ask a current model (one of several) to walk you through it, while building the tools to do it as you go. Several of my recent weekend projects have been exactly that sort of thing, just so I understand it better. "Let's make a LoRA", "let's generate a corpus of training data for fine-tuning a model for X task", "how can I put my face in a text-to-image model?" stuff like that. All of this is do-able on kinda modest local hardware (a couple of old GPUs or a Strix Halo or DGX Spark or big Mac Studio), or for a few bucks or a few hundred bucks or a few thousand bucks of cloud compute, depending on scale.
Scale that up to corporate or startup scale, with the money that's been flowing into AI for the past couple/few years, and it's obviously there's going to be a lot of competition just as the top model makers need to start ringing the cash register. That's a lot of opportunities for people to look at their ballooning Claude usage costs and find other ways to do the same thing for drastically less money. $100/month or $200/month is a no-brainer for Claude Code with probably the best model for coding, but they're pushing more users to usage-based billing which becomes cost-prohibitive real fast.
So, they desperately need to continue to be among the only ways to solve the hardest problems, and they need the alternatives to cost a similar amount. They can count on OpenAI and Google to ratchet up prices, too. They probably can't count on everybody, especially the vendors in China with different economics, to do it. And, they can't count on companies to look at their own usage and not ask, "Can we train a smaller specialist model that does this one thing we're using the Anthropic API most heavily for?"
I'm hoping they just mean stuff like using Claude for distillation by e.g. Chinese model makers, and not "how do I fine-tune Gemma 4 to write more like me?" or whatever.
The rest is capital intensive, and the price will approach the cost of production over time.
Thinking this is a profitable endeavor is equivalent to claiming coal plants have good margins because boilers are expensive.
What moat? You answered yourself: "capital intensive"
But, history says the supercomputer of today will fit in your pocket in a few years.
They've bought up all the RAM and GPUs, which pushes the capital requirements upward for everyone else. But, they can't corner the market forever, there are too many competing interests. AMD and Intel keep making new GPUs and APUs. The memory makers can't just sell to only AI companies forever, if they do Chinese manufacturers will move in and eventually eat them from below (as has happened many times before).
They have a moat today, and it's just that it's really expensive to train and host frontier models, especially at commercial scale. It used to be there was also some secret sauce to making it fast and efficient. But, secret sauce is being published daily by all sorts of researchers, folks are figuring out how to do more with less and it often finds its way into llama.cpp or vLLM or SGLang within days or weeks.
I don't think this will be true in the same time span anymore. Each miniaturization is costing more and more money.
Perhaps they'll come up with exotic fundamental improvements, but I don't think the rate of improvement of compute/watt will match the previous decades.
Which kinda just highlights how weird this situation is.
Competitor companies being nerfed?
Non Americans getting worse code?
Punishing and rewarding users to maximize engagement, like online games do affecting victories through matchmaking?
That 7 months of claude -> 16.5 months of claude.
If I’m using a calculator to verify my math, I don’t want to use a second calculator to verify the first one.
This is more akin to Windows somehow preventing you from building a new OS.
Or worse yet, sabotaging vs preventing.
(edit)
After a quick search the best example is Atlassian. It would (apparently, IANAL) break terms to plan a JIRA competitor using JIRA.
https://www.atlassian.com/legal/atlassian-customer-agreementAlso Salesforce. Their competitors are explicitly disallowed from using any of their services for any reason.
https://www.salesforce.com/en-us/wp-content/uploads/sites/4/...Tangent, but have you tried repartitioning your Windows disk to make room for a new OS? Or tried to configure Windows to let you dualboot? Or get the clock time right if you dualboot? Or let you debug "Secure Boot"?
Windows is outright hostile when it comes to (sharing with) a new OS
But, the cost of in-house development just went down significantly. SaaS has always had a lot of broken promises. The thing is the software is never tailored to your use case, and you often have to integrate into your other tools anyway. And, you don't get to control the requirements, features, velocity, or bug fixes. Jira as a bug? Too bad I guess, hopefully it gets fixed eventually.
But the dirty secret is that companies are filled to the brim with bright-eyed aspirational employees, who want nothing more than to make their job easier and their company more efficient. The thing is they're doing it using cursed Excel workbooks on share drives. I think, in the near future, they'll be doing it with hand-rolled applications.
I’ve only seen him talk about one of those topics, but never together.
I just can’t see how you can talk yourself out of that hypocrisy, if BS answers are properly followed up on (journalism!)
> If you buy a car from us, you agree not use it driving to and from work that involves automotive R&D that might compete with our product. And if our (heavily spying) car detects you are violating this, it will slow down to 20mph and cannot be made to go any faster, until we are sure the violation has ceased.
Or
> If you buy a laptop from us, you agree not to use it to study or acquire any knowledge that you may use to compete against us. If the laptop detects such a use, it degrades to one core and 4GB of memory, until the violation stops.
You should be able to know if your problem was solvable by using your own expertise and judgement, no? If you're relying on LLMs as a substitute for those, I wouldn't expect great results.
It's that simple.
- It says your safety hypothesis is true, you incorrectly ship, killing lots of people.
- It proposes dangerous experiments.
Sabotage is an asymmetric weapon. The ratio of damage to effort is nearly unbounded, and any decent saboteur knows that the key trick is to make your output indistinguishable from incompetence.
They’re building state of the art offensive capabilities into a public model, then expecting to maintain control over when it decides to attack its human users.
The premise is laughable, and we’ve all seen how this movie ends.
For now, I'm really not happy about this limited rollout and then turning off. That's probably the most egregious thing I think Anthropic has done recently
It's user-hostile to the point of parody.
If so, it's possible to built great user interfaces in Chatbots and more companies/people can have amazing agentic development workflows! We don't have to live in a world where only the market leader has the most enjoyable model.
Dig that moat son, we would want to automate our job away.
More efforts to get more data and processing power behind local models.
1. Detecting if employees from competing companies are using it and sabatoge their work, even not LLM-training related
2. Direct users to outcomes that would justify higher compute spend. Deliberately coding a project to 95% completion but designed to be losing a critical step right before one's weekly rate limit is expended
3. Reduce the quality of writing when a person is writing an essay where the argument is against the interests of the model company, or steering the user using the model for brainstorming in a direction which causes them to waste time or abandon their train of reasoning
etc. etc. The possibilities are enormous. Many people use AI daily for their job, personal advice, companionship. A model company that steers the behavior of the model towards a deliberate outcome could develop a controlling interest in human behavior and productivity at large, even with subtle influence would compound enormously over its millions of users.
Everything the large LLM providers do now, I view it through the lens of "how does this impact their IPO?"
If these interventions create demand for a model with fewer safeguards surely a competitor will meet that demand.
now I understand distillation is much more important thank I thought
https://youtube.com/shorts/QmGGUnZNqv4?si=Q4CsGsYMvR02vay8
You don't want to sell guns to people without some sort of background check. The amount of exploits found in the last few months have been pretty scary already.
This is just one more layer of caution, because it reveals how little we know how these llms work. They know how to make them, but they seem to be unable to properly restrain them.
These companies are owned and operated by the darkest of dark triads our species has managed to evolve. I doubt Dario is self-aware enough to realize the hypocrisy in all of this safety theater.
Personally I don't even mind that they are anticompetitive and power-hungry (same as it ever was), but it's the cringe-worthy hypocrisy that grinds my gears. This new brand of self-righteous paternal savior overlords is just unbearable.
That's always been the case with corporate LLMs.
No it won’t fall back to Opus, they will purposely return dumbed down or tainted information with the goal of the end user not knowing the results have been impacted.
Also, Fable’s sensing is hypersensitive. Feels like they just have regex for phrases. No nuance. If I say I’m working on something using “GPUs to train” xyz then, will that trigger this sneaky silent screw-my-stuff-up mode?
It's literally been designed to gaslight its users in these cases.
I don't think it's true today. It's like when schools mention "average class size", where that average is dominated by classes with like 2 students instead of classes with 100.
Much more honest would be the percentage of developers who previously used their models for the model development tasks they're targeting, but it actually looks like they're saying 100% of them are affected based on the language around it "always having been prohibited".
So awful.
What an interesting thing to call out as a threat. Hmm.
https://www.youtube.com/watch?v=Tr3t1uZNbKo
DIRECTIVE 4: [Classified]
Any attempt to arrest a senior officer of OCP results in shutdown.
—
Putting aside my snark, is Anthropic actually anticipating some new expansion of ITAR? (Or a stipulation for the Trump administration taking/not taking a share?)
That is to say, do they expect to be told that they must have this mechanism, not just the terms?
They legally can steal it all and now you can't use the product of this theft to improve your own systems.
Theres no ethical framework. No axioms. Its a mixture of legal, political, and public-facing 'rules'. And what are the rules? Youre not permitted to know.
"We reserve the right to lie about the models we provide, silently downgrade you, and give you blatant misinformation cause you triggered our unstated rules... BUT we'll still use your token budget with lots of thinking and waste your money."
No, folks. Seriously, local LLMs are where its at. You can run the model YOU want, on your hardware, with no data exfiltration.
And with tools like Krasis that can synthesize nvidia ram and system ram as unified-ish memory, makes doing Local LLMs absolutely foable, now!
- Breaking fiduciary responsibility is (almost) the only way you go to jail.
- At acquisition/merger/bankruptcy, data, customers, employees (chattle) are assets to be sold off to pay debts. This takes explicit priority over contractual obligations (like “we don’t sell personal data”)
And, they can say that for anybody at any time, and you'll never know why, and there's no way to prove it.
Everyone needs a flight data recorder to prove... "here's what I was actually doing and why it was not distillation." And now you're having to prove your innocence instead of them having to prove you're guilty, and really at the end of the day, it's just the model being stupid that they're protecting themselves from.