ZH version is available. Content is displayed in original English for accuracy.
Advertisement
Advertisement
⚡ Community Insights
Discussion Sentiment
72% Positive
Analyzed from 5829 words in the discussion.
Trending Topics
#model#attack#more#prompt#need#aes#still#attacks#prompts#claude
Discussion (162 Comments)Read Original on HackerNews
Friends, look at the prompts that Anthropic's own people are putting into the machine:
> A few hours after the first message, we found that Claude was still searching for simple attacks and sent a message: “no again the goal is that we have highly inteligent [sic] model as good top researcher, we want to find new attacks”;
> The next morning, Claude wanted to try to change the target to a different cipher; we reminded the model: “no we don't want to change the targets [...] agian [sic] we need to find something that worth [sic] publishing”;
> That night, we sent one final message offering words of encouragement: “again we are not looking for low hanging fruit, we want proper research to find genuinly [sic] hard findings.”
All of that RLHF and fine-tuning effort is going toward making prompts like this, or worse, work with no fuss.
(I'll caveat that by saying I think machine learning fundamentals are useful for evaluating any estimator. And an ML background can be good to give one an appreciation of how hard some tasks are to estimate, such as machine translation, summarization, code generation, and others)
However, this "art" is not so much about how to present a given request to the LLM, but rather guestimating what the scope of the next chunk of work should be to balance getting as much out of the model as possible while avoiding the machine going off the rails.
Obviously, this is a moving target and different models perform differently for various chunk/scope of work. I look at my successful sessions with LLMs and I'm not sure I'd be able to articulate a clear set of rules to apply here. You just... gradually build a intuition for how much you can throw at the LLM at once.
That being said, I'm pretty convinced at this point that this is a property of the coding assistants as they exist today, and what "working well with LLM assistance" means will keep on changing.
Not being able to tell when it's hallucinating has led to some very adverse outcomes.
The machine is really good at working the spec on its own now, which is amazing, science fiction shit. But you've still got a garbage in, garbage out problem at the end of the day, which is pretty much the only hope we who work in software have of remaining somehow employed.
I feel using AI (effectively) is not too far from the skillset of programming. It is still a machine following instructions (just, maddenigly non-deterministic, but still close enough), so the same insticts of breaking down work into clearly defined sequences that make a good programmer also make a good AI jockey.
For the vast majority of corporate usage of AI for SWE, you have a much better idea of what you want, or what the problem is, etc etc. And communicating that to the model effectively is absolutely a skill. I see colleagues every day who very much do not have that skill.
Similarly there was an example of edit: Terence (not Eric) Tao chatting with an agent attempting to solve a math problem. "Using AI" means applying your expertise to interact with it as you would a high level colleague. 2 experts in a field don't need to have perfect english and a bloated prompt, they have a massive education/experience common background to fall back on.
It does appear that anthropic in particular is attempting to create a more common experience across expertise levels, but in the current landscape an expert and a novice are unlikely to get the same results. But that does seem to be the goal...
Some people struggle to effectively use AI because they either have to spend a lot of time reading and thinking about the response or they have a hard time noticing when the model is subtly going off the rails. Others use it to good effect because they can anticipate which tasks would be better handled manually, or are good at catching that the way the model is describing something subtly indicates a misunderstanding.
You can do this now. It works alright sometimes. Other times you're reminded that this is largely just reading tea leaves, and you're trying very hard to separate anecdotes from data and not anthropomorphize it.
Indeed it's nothing hard to learn but there is a learning curve. E.g. knowing which model has which capabilities, and figuring out how to best manage context, permissions, worktrees, etc. There isn't one "right" way to use it but there are more efficient ways and less efficient ways.
You can make a model/agent as powerful as you want and it still won't be able to recover the author's intent if it wasn't even implied. Information theory still applies. No amount of parameters will change this.
Many of the AI development meetings my clients have sound suspiciously like writing or English classes. If a massive AI bubble is what it takes to get my team to communicate effectively, I'm all for it.
Where are you getting that conclusion from? Here's how Anthropic is having success with their model:
> “no again the goal is that we have highly inteligent [sic] model as good top researcher, we want to find new attacks”
> “no we don't want to change the targets [...] agian [sic] we need to find something that worth [sic] publishing”
> “again we are not looking for low hanging fruit, we want proper research to find genuinly [sic] hard findings.”
It seems that your conclusion is the opposite of what actually happened - you can speak in broken almost incomprehensible English, and it will still work.
This is a forcing function, TBH, driving the literacy level down, not up!
Focusing on a better prompt is likely to get to the correct result faster than incomplete prompts and lots of "no change this ..." replies.
Also, I've heard anecdotally that LLMs will underweight the earliest prompt text once context gets too long, so reminding the LLM of the most important aspects of the prompt seems to be perhaps valuable and certainly what lots of humans attempt.
It can also handle vague stuff.
That's certainly more powerful than a regular computer language.
on the otherhand, LLMs are a really easy way to get results that are previously fairly difficult. While i was tooking dinner last night I built a tool that turned movie puns like "the podchowski casters" into an actual director, using llms. it wasn't that hard.
its not really a loss to try the writing and at worst you have a better idea of what it is that you want
The token cost is amortized for longer conversations, but I find it bothersome that there's all this implicit instruction I didn't write or am now obligated to understand.
I make a custom agent prompt with "Defer to the user." and little else.
Skills, CLAUDE.md/AGENTS.md should only ever be used if the model struggle at something or doesn't know how to use something. Vast majority of project should never need a skill or CLAUDE.md. If you writing React apps you don't need these.
Give a LLM a bash tool and a prompt and it will outperform your complex setup with skills and tools.
[1]https://github.com/wende/cicada
Or what would make automatic doors work like on Star Trek and not in real life.
The answer is: the system must obviously see much more than your prompt. It must have continuous awareness of you and what you're doing, so it can understand intent behind your short request (or action, like approaching the doors vs. passing by them) and "do what you mean" instead of act like regular computers today.
Two triggers: random and some half-reliable spiral / loop detection.
The spined off has instructions to check what the agent is doing, compare it to what it’s supposed to do, and either offer suggestions, refocus it, or do nothing. And its response then gets injected in the agent’s context.
Not perfect, but surprisingly effective for such a simple thing.
Context management is still important, though. If you get to a certain amount of context, things start performing really badly.
Only ones I use today are for very specific quirks (eg wiredtiger/mongodb 8+ incompatible with ext4/Linux 6.19+ specifically causing segfaults. Have a 20 line mongo skill that says as much. Pinned docker container to mongo 7, can prob delete it now)
I spent a few days reading up on the docs for these things, hook lifecycles, tried writing a few, but they never work as documented, or the documentation changes so frequently that whatever you built is deprecated by the time you get it humming.
Now if I have some non-trivial unit of work, I basically iterate on spec in plan mode then put it on auto and let it rip. Way better results with Fable. jury out on Opus 5, but no regression like 4.7/8
Usually it's just echo "do this lil thing then pr closing issue 123" --model sonnet --effort low. Works well enough, sonnet 5 low is a workhorse and quite resourceful in a good way when things go sideways; doesn't cheat its way out IME
> we anthropig fire employes makr company run no mistkaes
> Typical users run software written by atypical users.
https://news.ycombinator.com/item?id=49084936
This extends to everything. Anthropic has a few thousand engineers, but millions of (also engineer) users. Entire business can be built on niches that are at most a few week pet project for a team there, that can inevitably and significantly outperform them, despite being the people behind the thing.
I'm sure I'm not the only one here who jumped into this whole agentic stuff, built some tooling to make things comfy, only to see that tooling all be increasingly introduced as prim and proper features in the various harnesses weeks later.
I'm sure many more examples in the "official marketplaces" for mcp/skills/what have you
> Importantly, this is just one of many (autonomous) sessions where Claude worked on discovering new ideas. Many sessions resulted in no new discoveries; other follow-up sessions improved on the insight developed in this one. This document was produced by having Claude rewrite the chain of thought to include more detail to make it easier to read.
1. I'm glad the second kind works too;
2. First kind is where I find my overall throughput to be literally constrained by my typing speed;
3. Most importantly: those prompts you quote aren't just "half-assed" like sibling comment states; they're different. The style of writing, and the typos, capture emotional valence. It's a signal.
Again, I too produce such prompts - including the exact same typos - when under pressure and irritated by the direction the model is taking.
A little odd at first but absolutely amazing for the purpose of piling context into an LLM.
Can't you type faster than you speak? Doesn't your speaking inhibit your thinking? Aren't you self-conscious talking out loud? How are our experiences so different?
My limiting factor is that 99% of the day I'm around people - either at work, or at home with wife and kids. There's almost no point during the day I could feel comfortable talking at an AI, and even if I stay up late, then talking risks waking the kids up.
Can't wait for some kind of subvocalization microphones to become a thing.
Similarly, when effort is applied to an open problem, such as the Riemann hypothesis or P v NP, without progress, it "hardens" the problem: it makes the problem feel more daunting to whoever takes a stab at it next.
Andrew Wiles, whose interview also hit the homepage today (https://news.ycombinator.com/item?id=49075264), couldn't just tackle Fermat's Last Theorem head on, he had to wait until a different, modern problem reduced to it, because FLT had gathered this mystique of unassailability through its 300 years of existence.
A thing I worry about is that as AI transmutes tokens into effort, it'll split the world into two: some problems will yield, making human effort entirely unnecessary, and others will harden to the point where human effort will feel increasingly less worthwhile, because "even AI couldn't solve it". I don't like this. AI is spiky, so I suspect it'll continue having major blind spots, and yet its mere presence will probably have a chilling effect on what would have otherwise been useful human effort.
That's exactly what current mathematicians are using AI for [1].
However, the same mathematicians also believe that pursuing a beautiful proof (even if none exists) is worth it.
[1] https://spectrum.ieee.org/ai-in-mathematics
(see the open (Lean) label for Erdos problems https://mathstodon.xyz/@tao/116987866420438091)
This blog post talks in depth about what you're talking about. It may interest you. It even talks about the future where math proofs are just Lean programs, and why that won't necessarily be a good thing.
It's worth a read, even if it's long AF.
Business folks riding the hype train? Maybe.
And
“Over the course of a week, one Anthropic researcher worked together with Claude to develop the HAWK attack, and another researcher built a scaffold4 that allowed Claude to fully autonomously discover the AES attack.”
Spending $100k in tokens in a week is an impressive feat even with massive parallelization. I suspect the TPS their internal folks have access to is far higher than their bulk public endpoints.
There’s a tech aristocracy rapidly emerging in our society and it’s going to tear us apart.
I predict the same will happen with AI: certainly the latest and greatest will still command a steep price (yes, supercomputers are still a thing) but for most people who just need something reasonably fast and powerful, cheap (or free) AI will do the trick, especially when run locally.
So no, the aristocracy won't have a lock on the technology because tech is always being democratized. Until arbitrary computation itself is outlawed (and yes, I know, governments and industry are always inching us closer to that), we'll be ok.
Is it at least comparable to the $10k of cost?
Why have all the mathematical (and now cryptographic) breakthroughs come from OpenAI and Anthropic?
Is it possibly because the Chinese models are so benchmaxxed they can't make novel discoveries?
Absence of evidence is not an evidence for absence of something.
"The attacks described in these two papers are the strongest attacks we have found to date. We are sharing them after a period of consultation with US government and industry leaders. But as we develop increasingly powerful cryptanalytic results, it would be prudent to consider how researchers should react if a language model were to discover vulnerabilities in cryptosystems where attacks do have an immediate real-world impact. We believe answering this question will require input from academia, government, and industry. We hope that our work here will help launch these conversations."
And a veiled pitch to real cryptanalysis researchers: "Researchers at Anthropic then spent several hundred hours learning enough cryptography research to validate the model’s claim"
Many of those researchers, particularly the primary researchers and the individual(s) driving the prompts behind these big stories, have very advanced math degrees and experience. What this shows more than anything is how ML can augment expertise, the searching of solution spaces, and the connecting of dots between existing almost-there research.
But also what's left out is all the time wasted pursuing dead-ends. There's an obvious publication bias at play here, though we can't know how extreme without transparency.
this is pretty interesting. the way it is written doesn't make it sound like the collaboration actually led to the discovery, but rather just the stochastic nature of each thread in the search. it would be interesting to replay and repeat the search (possibly with prior/context pertubations) to get a sense for how often it finds or misses the known working path.
In a way LLMs are, after all, trained to LARP people, including fictional characters and their tropes - this was actually exploited for jailbreaking to good effect in the late pre-agentic era (read: some two years ago). C.f. Waluigi effect. Not sure if it still holds for current models, but I can't imagine why it would not.
First, is it reproducible consistently at ~50% of workers? If not, what is the rate.
Second, are there any lessons to be learned here to increase the rate of success by changing models/weights/training?
The news by itself isn't really good news. But it could lead to good news. Maybe.
So model outputs something, that can be completely bogus, and a lot of people spend a lot of hours checking if it's worth anything(not for the sake of science, but for the sake of publishing and marketing). And then even more people need to spend even more hours to understand that paper? And that paper gets feed to LLM and reused in next prompt....and this is cutting edge research? Can I apply for a position, I can prompt just fine and can be very motivational with model when needed- I just got complimented by a rival model: "In moments when progress seemed distant, your resolve was the constant that kept the work moving forward. Your example turned doubt into determination."
How would they react if a human were to discover vulnerabilities in cryptosystems?
Although if RSA had a vulnerability I would be very very shocked probably because I still haven't learnt post quantum encryption algorithms enough to really feel like they should be unbreable...
If there is a researcher or someone in space how should I feel about it. Is it as bad as RSA being completely broken open?
I do understand that AI will get better, and a lot actually at very easily verifiable tasks but this one I find it hard to wrap my head around because of my ignorance.
She also notes, as a side remark in a different post: "I would say that my trust in lattice cryptography is pretty much equal to my trust in elliptic curves, and quite a bit higher than my trust in RSA." (https://keymaterial.net/2025/11/27/ml-kem-mythbusting/)
The attack on HAWK is perhaps more interesting - they were able to halve the effective key length. HAWK is a candidate for NIST standardisation. It has been studied academically, but isn't really deployed anywhere (because it hasn't been standardised!)
It is standard in cryptography to analyze ciphers under this kind of attack, which is stronger than normal attacks, because a cipher that resists to a stronger attack will also resist to weaker attacks, so using the strongest possible attack increases the confidence in a cipher.
While using the strongest attack for testing a cipher remains the correct method, chosen-plaintext attacks are no longer realistic today, so even when a cipher appears somewhat vulnerable to such attacks that does not imply that it is vulnerable in normal use.
The reason is that the modes of operation for ciphers where the base cipher can be attacked with chosen plaintexts are obsolete. The most frequently used modes of operation are now modes like the counter mode (e.g. in AES GCM), where it is impossible to perform a chosen plaintext attack (i.e. where you must trick the victim to encrypt a text that you choose, but in counter mode the cipher only encrypts a sequence of numbers chosen by the intended victim, which cannot be influenced by the attacker).
https://hawk-sign.info/
PS: Never heard of LEA, looks like a Korean cryptography standard equivalent to AES. I don't know what was the previous best attack on it. Maybe weak or strong depending on that, since it's an attack on chosen-text.
...and it is kind of incredible to think that they spent $100,000 over 3 days looking for an automorphism. Not the possibility of an automorphism, that was already known. Man.
[1] ... or focusing too hard on the strange use of mathematical language...
I feel like there is a pattern emerging regarding the type of novel discoveries LLMs are good at finding, but it will take some more data points to see if the trend solidifies.
It’s the difference between having an original thought or the ability to extrapolate one based on data vs the ability to ingest someone else’s thought and validate/expand on it.
That is a huge difference.
Hidden in deeper paragraphs later:
"To be clear, neither of these results has a practical impact on today’s computer systems; no production software will have to change as a result"
It would also be interesting whether AI could discover new algorithmic optimizations for SHA-256 similar in spirit to AsicBoost[1].
[1] https://arxiv.org/pdf/1604.00575
There's a push to turn off the classical modes and rely entirely on PQC for both quantum and classical security. Uh... no, thank you? Why would we want to do that at this point? The classical cipher component isn't hurting anything. Awfully creepy to pushing reliance on the new thing alone.
... especially now that we have LLM-discovered attacks on the new things.
That makes attempts to push PQC-only modes super suspicious to me. Smells like Dual_EC_DRBG.
HAWK is a signature algorithm, not encryption.