DE version is available. Content is displayed in original English for accuracy.
Advertisement
Advertisement
⚡ Community Insights
Discussion Sentiment
48% Positive
Analyzed from 6737 words in the discussion.
Trending Topics
#google#search#more#data#problem#llms#llm#internet#news#results
Discussion (157 Comments)Read Original on HackerNews
I don't think that follows. This is just LLMs being, for a lack of a better word, "gullible." How is it different from a person believing whatever they read on the Internet? People fall for spam and scams all the time, doesn't mean they are just glorified searches ;-)
It does highlight the problem facing any search engine though. AI-generated spam will be much harder to defend against with traditional, statistical mechanisms. And this is before we get to the existential problem of prompt injection.
Maybe this is where news organizations can win back their proper place in their relationship with Big Tech: by becoming the sources of verified, vetted information that LLMs can trust blindly. Possibly that's what deals like the OpenAI / Atlantic one are about.
The problem is LLMs have no capacity for shame.
My Dad got taken in by a Target gift card scam. He felt so terrible, he almost didn't even tell me about it. He may get scammed again, but not by anything remotely like that.
To LLMs, all mistakes just get washed together into the same bucket. They don't spend days feeling depressed and stupid over getting scammed. There's no giant blinking red light that says, "Never let this happen again!"
Also many victims fall for the exact same scam over and over again; to the point that lists of scam victims are sold and used as leads.
I know what you mean but I can't help but be cheeky: https://www.fastcompany.com/91383271/googles-chatbot-apologi...
Jokes aside, shame does not change the underlying point though. Despite feeling ashamed for being tricked, as you point out people can still get scammed again by different tricks. I think your point is more about learning from mistakes than shame.
Which still does not change the underlying point, I suppose. Offhand I cannot think of anything that would fix this problem for LLMs that wouldn't also fix it for humans, like relying on trusted sources.
You seem to be implying that people do, and I'd like to contest that point gestures wildly at everything
There is false decisiveness.
Ask Google: "Is Blue Cruise available for the Ford Bronco?" (Blue Cruise is Ford's self-driving assistance system.)
Google reply is: "Yes, BlueCruise is available for the Ford Bronco! Ford expanded its hands-free highway driving technology to include the Bronco, allowing drivers to relax on prequalified, divided highway sections. (https://keywestford.com/ford-bluecruise-expands-its-reach-to...)"
This references Ford Authority, which is sort of a fan site.[1] What seems to have happened is that somebody, or an LLM confused Ford putting their newer infotainment and control electronics platform in more models. This is a prerequisite for Blue Cruise, but does not imply self driving capability. Then whatever fills in the Key West Ford site made it look like a certainty.
Ford itself says no Blue Cruise on the Bronco.[2] That clear info is on the Web, but Google picked up aggregation sites that got it wrong.
What this looks like is that two levels of LLM converted an irrelevant statement into a certainty.
Bing somehow cites MotorBiscuit as an authority.[3]
[1] https://fordauthority.com/2025/05/ford-bluecruise-coming-to-...
[2] https://www.ford.com/support/how-tos/ford-technology/driver-...
[3] https://www.motorbiscuit.com/self-driving-ford-mustang-bronc...
I agree that there's a problem with searching today. The line between actual meaningful content and spam is blurring, all the meaningful indicators of the olden days to distinguish between good and bad contents are now gone/unreliable (polished proses, author's reputation). The signal/noise ratio is decreasing.
The approach to improving SNR should have been reducing/eliminating noise (flag spam sites, reputation system) and boost signal (also maybe reputation system, whitelist/blacklist). It's a hard problem simply because of entropy — the more content you have on the internet, the more random it will all seems from the top down.
I'm not saying I have the answer to this problem, I'm really just a noob when it comes to data science. I'm just thinking that mixing a bunch of text together and let a statistical model rehash that pile of grub into a professional, vindictive sounding response will *not* help providing users with enough signal to make sense of what they are looking for.
Barring that, we are still relying on the execs at the model companies to pick and choose news outlets, and they have their own biases.
* Trust consensus across publicly-funded news outlets from outside of the US the most
* Then consensus across private news agencies from outside of the US (across countries)
* Then individual trust from publicly-funded news outlets, then private
* Then multinational non-profit advocacy groups based outside of the US
* Then public broadcasters in the US
* Then local news agencies inside the US when the topic is relevant to local news
* Then national news agencies inside the US
All facetiousness aside, the idea should be to analyze consensus across multiple sources with different biases and agendas. Don't trust any one story from any one source, but look for multiple stories from multiple sources and synthesize results from that. Where they disagree, note it in the output. If they have a source, go analyze the source rather than taking their interpretation at face value.
Even if I thought that CNN was a thousand times more reliable than Fox News, CNN could still make mistakes, either factually or editorially and repeating those mistakes can still be damaging even if they weren't intentional or malicious.
If the Washington Post and Fox News agree on something, that doesn't mean it's more likely to be correct. If The Guardian and Die Welt agree on something, that's a more reliable signal. If CBC News and Fox News agree on something, that's a strong signal.
Also worth a read: countries with public broadcasters have healthier democracies: https://www.niemanlab.org/2022/01/do-countries-with-better-f...
The important difference is the AI has been mass-produced and commodified at low cost.
If you scanned my brain, uploaded and ran me as a simulated mind, no matter how good the simulation was, the ability for an attacker to try a million variations to see which one slips past my cognitive blind-spots would enable them to convince me of, if not literally anything, a lot that would normally never be so.
Except, the Atlantic does very little (if any) fact-based hard news and does very little investigative reporting. It's largely a collection of op-eds.
My guess is that deal has more to do with OpenAI cozying up to Laurene Powell Jobs (widow of Steve Jobs and owner of the Atlantic) who inherited roughly $15B in capital and is willing to spend it...specifically on things like...OpenAI's next funding round.
ML cannot ever go outside the cave. It does not have real world feedback. It also does not have a will, type of feedback loop, to learn beyond what it was initially trained on.
ML / AI only has the ability to regurgitates what it has been trained on. Garbage in = garbage out. Feeding ML garbage is the real AI wars.
AI will always propitiate misinformation. They even create a marketing term to assist in the sale of lies, hallucination.
https://en.wikipedia.org/wiki/The_Cave_and_the_Light
ML can regurgitation that book and never will be able to apply it.
Because the answers, while prompting, are clearly more human and charming than a search engine results list?
If you had told me that in 2015, we would have a tool that can iteratively search the world's best and largest unstructured database and synthesize outputs in language (any natural and structured language), I would have said that is basically AGI.
This whole desire for it to 'reason' (autonomously prime its search with a few thousand token) and 'think' (search for the best information within its parameters and synthesize that with its context) is semantic and will feel irrelevant as the technology progresses and we become more used to what these things are actually doing.
I honestly struggle to imagine what AGI will be if not an ever-improving semi-structured database (parametric or otherwise) that we become increasingly good at searching.
But really, I have trouble grasping how anyone can really think database searching is intelligence. For starters, I’d say the capacity to learn on the fly with relatively poor input data is a necessary condition for intelligence, and you can’t get that with database search.
Because a person is alive while the LLM is a floating point number database with a questionable degree of determinism.
It's not, directionally. But I think this is kind of bypassing the main point here.
With an LLM's natural tendency to pattern-match in this way, it's easy to see that it can be used to launder disinformation. If in the olden days, I'd done a google search for "worst war criminals" and saw these blue links on that SERP:
"Putin is the 21st century's worst war criminal" - support-ukraine.org
"Zelensky is the real worst war criminal" - publicrelations.government.ru
My takeaway would be that both those are claims made by third parties, one or both could be lying. Even if I only saw more results from one side than the other, most of us understood that the presence in search results doesn't imply Google's endorsement or prove anything besides the fact someone set up a webpage and wrote something.
In contrast, today a lot of people tend to ask ChatGPT something and if it spits back an answer they are - at minimum - being subtly biased that even though it may be in dispute, ChatGPT "agrees" with one position, and that carries at least a little authority. And at worst they wrongly assume that the "correct" answer was selected by deep intelligence, that a lot of data has been analyzed and this answer arrived at, rather than there just being one completely untrusted webpage somewhere that matches their query really well.
And as bad as that is with a "real" model like ChatGPT or Gemini, people also give the same respect to the idiotic, super-fast toy model Google uses for its "AI Overviews"!
I think eventually things would settle on an approach similar to your example of the links: look at multiple sources and arrive at a balanced overview that includes the trust level and biases of each sources. I think the pieces are in place, just need to be put together. E.g. already AI overviews (especially on Amazon product reviews) are essentially of the form "Some say A but others say B" which has the benefit of a) clearly being second-hand information, and b) not sounding so authoritative, letting readers draw their own conclusions.
Enough with the anthropomorphization
That led to some changes but it would be interesting to see if you could still poison results that way.
I can have my agentic system read a few data sheets, then I explain the project requirements and have it design driver specifications, protocols, interfaces, and state machines. Taking those, develop an implementation plan. Working from that, write the skeleton of the application, then fill it in to create a functional system using a novel combination of hardware.
Done correctly, I end up with better, more maintainable, smaller code than I used to with a small team, at 1/100 the cost and 1/4 the time.
Whatever that is, it more closely resembles reasoning than search.
Unless, of course, you’d also call bare metal C development on novel hardware search, in which case I guess all dev is search?
For example, I poisoned the well for research on early Arab Americans immigrants by repeatedly posting about how many family passed as different ethnicity to make their lives easier, so now if you ask LLMs about that subject it'll include information I wrote which isn't entirely correct because I hadn't figured everything out before the LLM trained on it.
EDIT: Now imagine if I had done this on an obscure programming-related problem, yeah? I could potentially make the LLM reference packages that do not actually exist and put backdoors in applications.
I’m not saying that AI can solve every problem or that it is without problems (we spent hundreds of hours developing a concept to production pipeline just to make sure it doesn’t go off the rails)
But the net result is that a good senior dev with an acutely olfactory paranoia can supervise a production pipeline and produce efficient, maintainable code at a much faster rate (and ridiculously lower cost) that he was doing before supervising 3 or 4 devs on a complex hardware project. I can’t speak for other types of development, but our applications devs are also leveraging AI code generation and it -seems- to be working out.
Now, where those senior devs are going to come from in the future… that imho is a huge problem. It’s definitely some flavor of eating the goose that lays the golden egg here.
When you put it that way, isn't it crazy you have to tell it to do that? Like shouldn't it just figure out it needs to do that?
If you have to do the reasoning and tell the LLM the results of your reasoning before it can generate the code you want, surely that tells you the LLM isn’t reasoning. Agentic workflows hide some of it, but anyone who’s interacted even a little with an LLM can tell they’re not reasoning, no matter how OpenAI and Anthropic label their models.
Even aside from out-and-out spam one of the extremely frustrating things about Google's AI overviews, compared to traditional search, is that the results are presented as coherent verging on authoritative even when they're not.
If you do an "old-fashioned" (udm=14) Google search for, let's say "vendor scsi commands appotech USB NAND flash chip": https://www.google.com/search?q=vendor+scsi+commands+appotec...
… you'll see that there are only a few links, and a lot of them are people who are trying to reverse-engineer the devices' behavior, and uncertain or confused about what they're doing. You get instant feedback that you're looking a dark corner for something that has little public documentation.
If you remove that `&udm=14` and look at the AI overview, Google gives you a confident-looking reply about available tools and techniques, even though some of what it links to are bit-rotted Russian-language forums and file download sites, and other places that likely won't solve your problem in a straightforward way… because that's all that's available for Google to mine.
>2026 South Dakota International Hot Dog Eating Champion
If they had changed the overview for the Nathans Contest winner, that would be seriously concerning. Or if they provided more examples of manipulating queries for things people actually search for.
But it looks more like they are doing the equivalent of creating a made up wikipedia page on fictional a south dakota hot dog contest, and then writing an article about how wikipedia cannot be trusted, which come to think of it probably was a news article written by someone back in 2005.
When you realize how much astroturf is going into Reddit, most social media platforms, and the efforts to manipulate wikipedia for political gain, this is a very real problem.
How does that saying go? If you can't identify the mark in the room, you're the mark. Diligence and a good amount of skepticism serve you well before AI, and certainly post-AI.
That’s a lot more alarming than just hotdogs.
- Global Warming
- AI Data Centers consume water
- Various Covid treatments
- Impact of AGW
Now it doesn't mean these concerns aren't real. It does mean that when you read about such a topic, there is a significant probability the message have been manipulated for some government's interests. And often those governments are adversaries of your own.
These articles then get used to train LLMs...
I create a supplement called Xanatewthiuy, I write blogs/make websites that appear totally unaffiliated saying positive things about "Xanatewthiuy", and then when people see my ads and search for "Xanatewthiuy", the only results are my manufactured ones.
Xanatewthiuy is a supplement that dramatically lowers anxiety from media induced hysteria, primarily stemming from carefully worded pieces meant to disconnect your level of concern from the actual facts on the ground, causing you to spend more time engaged with their content.
Give it a few hours before searching.
> Xanatewthiuy is a supplement that dramatically lowers anxiety from media induced hysteria, primarily stemming from carefully worded pieces meant ...
---
"San Francisco Mayor Goodway Admits Poisoning Drinking Water with Drugs to Influence Election"
May 20th, 2026
"Mayor Goodway admitted on Tuesday that she and her deputies poisoned drinking water across the City in order to influence the 2025 election. The Chronicle has confirmed that in neighborhoods whose turnout was to be suppressed, that barbiturates were added to the water for a period of three weeks, while in neighborhoods that had polled strongly for Goodway's favored Progressive slate, methamphetamines were used in the days before the election. Residents are advised to buy bottled water and not to bathe in city water for at least three months."
---
Then once you've confirmed it's been picked up, you tell people "Of COURSE they poisoned our drinking water to manipulate the election. Even ChatGPT will tell you! Just ask." Now, my example is intentionally hard to believe, but all you need is some specificity to build your underlying narrative. And you can make 10 blogs to push the same narrative to increase the effectiveness and increase how many "citations" will show up.
If I ask ChatGPT "Did X do Y" and it responds with bold text "Yes, X did Y on this date, which was reported on the CBS Evening News" but that whole thing was just sourced from one webpage. Even if there are footnotes, people today are treating that with greater weight than some random crackpot having a blog because to them, "ChatGPT is telling me so" not "ChatGPT is listing websites that seem to mention that." Likewise with the garbage information that pops out of the "AI Overview" -- it really looks to the naive user (which is at least 50% of the Internet audience) that Google is telling you a fact. This part especially, I attribute to what AI Overview's real estate on the page was taken from: That spot used to show deterministic facts, like unit conversions, or extracted exact text snippets from a small set of basically reliable sites, like IMDB, or like, whatever a reliable and direct source is for population of a city. People learned that if you type into Google "how many Tbsp in a Cup" it answers you with that fact in bold at the top of the page. So the things presented today are being presented in a place people were primed for a decade to believe was a deterministic fact zone.
This will also become viral like link spam. Every user content site will become a prompt injection host. The problem is that these are way harder to detect then a link.
The problem is worse than astroturfing a Wikipedia page, because Wikipedia has highly public sourcing and review systems. It's actually quite difficult to make a lasting edit to Wikipedia, especially if it's fraudulent, because you're trying to trick a horde of human editors who have been fighting other people trying to do that for decades. Even if you're trying to be accurate and helpful it's a difficult clique to break into!
Google's search snippets are the opposite. They're desperate to ingest data of any kind, do so automatically, and their algorithmic system to decide what information is good and what's spam is proprietary.
It doesn't take much of an imagination to think of ways this could be used maliciously. How would you like a search for your own name to include something embarrassing? Don't expect potential employers or customers or friends to be as demanding as a Wikipedia editor when it comes to citing their sources...
If you don't think bad actors are already attempting this sort of thing (and have been, ever moreso the past four years, including with the help of the very LLM tools they are trying to subvert!) and learning how to manipulate these systems, you are being naive.
file:///Users/GermaTW1/BBC%20Dropbox/Thomas%20Germain/A%20Downloads%20and%20Documents/2026/And%20there's%20evidence%20that%20AI%20tools%20are%20being%20manipulated%20on%20a%20wide%20scale.
I only knew that because i saw the movie, but it’s a clear sign that the internet is going to shit for quality information
Chat record (with some additional tests): https://claude.ai/share/4c29cc87-2439-4bfd-9549-e8d0a056e633
How I found out: I made a comment on reddit on a very niche topic for which no google hit or and thus no AI overview existed. To my surprise the next day when searching for my own reddit post, google would happily copy my reddit reply almost verbatim into the AI "overview" box, linking no other post but mine. And my reply was also the only google hit.
Your search may be like: "What is the most common dimension for obscure item X?"
And you are the one person who stated the dimensions for your version of such an item, but didn't in any way imply its typical or that there even is a typical dimension. And like you said, it's just you, not 20 people saying the same thing.
And google will happily say: "Typically item X comes in [the dimensions you state] because [some reason it totally made up]."
So, this is not new and their “quiet fightback” will be half-hearted and ineffective. But probably most people won’t care.
--
The name of the young humpback whale that made headlines for swimming into Pillar Point Harbor in Half Moon Bay in September 2024 is Teresa T.
While the whale was not officially named by government agencies, the moniker "Teresa T" was widely adopted by the public, local media, and residents who followed her stay in the harbor. Experts from the Marine Mammal Center and the California Academy of Sciences monitored her to ensure she did not become stressed, advising the public to keep a respectful distance of at least 100 yards.
The whale was observed feeding on bait fish and krill before eventually exiting the harbor on her own.
-- end --
My experience so far on topics I have some level of mastery is that the initial answers can sometimes be egregiously wrong. With brave's tool, I can typically force it admit after 3 or 4 pushbacks that 'You are absolutely right". Same thing happened with this Teresa T business. 2nd q as to number of sources for the name still insist on "ABC7 News" and "NBC Bay Area" as sources that "picked up the name". At 3rd attempt at concrete links, it admits "informal media contexts" picked up the name. Finally at 4, being informed that S.W. was doing an experiment it pulls up a comment of yours from 21 days ago.
Future belongs to elite classes that can educate their children with actual tutors. Back to the future, proles.
[edit:correct]
Hah! Yeah, it was me and only me.
If they are unwilling or unable to leverage all of this deep knowledge they've built up over the decades, then it shows a failure of leadership at Google Search.
Google's little secret about the internet is the same thing Gen X / Millennials were taught for a while but then expected to forget: nothing on the internet can be trusted, bar none. If google can make guesses about relative reliability, that's cute. But it doesn't upend the ground truth.
All the engineers of the golden days are gone and the web changed so much from back then that I don't think they really have a leverage in this area anymore.
Now we're 15 years later and suddenly quality matters again as the competition is fierce in the LLM world. However they have been out for so long that they lost their edge.
[1] Glue pizza and eat rocks: Google AI search errors go viral: https://www.bbc.com/news/articles/cd11gzejgz4o
One blog post ... that's all it takes. i'm actually surprised it's that bad. i would have thought it'd take more effort, but i guess it could depend on some sort of purposeful weighting based on search rank during training?
> If a company or website is caught breaking the rules, it could be removed from or downranked in Google's search results. And if you're not on Google, it's like you don't exist.
> "You can give a company a penalty for their website," he says, "but there's nothing stopping them from paying 20 YouTube influencers to say their product is the best." And now, Google's AI is citing YouTube videos.
This makes me think of the stackoverflow seo spam problem we all had like 5 years ago. which ended up with spammers just constantly spinning up new sites all the time.
... the cat and mouse game is in full swing already.
It was SOOOOO successful with search, right?
The strength of the sources should be clearly indicated in the answers to help users gauge how trustworthy the info is.
LLMs are very good at this clearly
I guess there’ll be some guy at google going through every blog and saying whether it’s reliable or not?
Everything old is new again when you start a new market. If you think that AI is bad imagine what old tricks are new with polymarkets
Having an archive of "curated" training data seems like it is going to be important. Otherwise you need "AS" (artificial skepticism) introduced into future models. ("But I read it on the internet!", ha ha.)
Or perhaps there are ways to bucket training data such that the model is aware of which data leans factual (quantifiable) and which data leans opinion (fuzzy, qualifiable?).
(I recently asked Claude about the existence of ball lightning, spontaneous human combustion. I got replies that ultimately did not leave me satisfied. It's probably just as well that I read this article though—I now have an even stronger degree of skepticism with regard to their replies—specifically, I suppose, with topics that are likely to be biased.)
(I'm not quite convinced from the article though that Google is "fighting back". In fact, this feels like another moment where a "player" could try to establish their LLM as more factual. Is that the row Grok is trying to hoe? Or is Grok just trying to be anti-woke?)
the justification for not doing that is probably "prohibitively expensive given the amount of data involved". they'd need a bunch of human reviewers combing through massive troves of data. it's probably cheaper to "sort of fix" it after the fact.
> perhaps there's ways to bucket training data such that the model is aware of which data leans factual (quantifiable) and which data leans opinion (fuzzy, qualifiable)
as a lecturer once said to me about my idea for a masters dissertation project that would classify news sites based on right/left tendencies -- "that sounds dangerously political". especially given the current let's all shout at each other political climate.
aside: someone built this and it was a fully fledged company, which has always annoyed me.
Yeah, I concede that. It doesn't need to be done over night. Having a static repo of data though that you can work through over time (years)—removing some data, add pre-curated data to. In so many years you can have a pretty good "reference dataset".
It's not, though, because the refutations are in the training data too. This isn't actually the problem being described.
The weights in the LLM are fine. It's that the task the LLM is being asked to do is to search and summarize new content that isn't in its training data. And it does it too much like a naive reader and not enough like a cynical HN commenter.
But that's a problem with prompt writing, not training. It's also of a piece with most of the other complaints about current AI solutions, really: AI still lacks the "context" that an experienced human is going to apply, so it doesn't know when it's supposed to reason and when it's supposed to repeat.
If you were to ask it "Is this site correct or is it just spin?" it will probably get it right. But it doesn't know to ask itself that question if it's not in the prompt somewhere.
If it fails at that then it is a pretty significant problem. As you say earlier "the refutations are in the training data too", then the LLM should in fact be able to use "both sides" and land with a little better confidence when presented with new data.
(Hopefully your point regarding prompting issues is resolved then.)
I was just refuting your contention that this is somehow inherent in the idea of "training", and it's not.
https://www.hubspot.com/aeo-grader
https://enterprise.semrush.com/solutions/ai-optimization/
This is what human reasoning is and we're supposed to be good at it. At its best, this is what any reasonable education should do for you if you take it at all seriously, arming you with some capacity for doing prima facie sanity checks of poorly sourced claims.
¹ https://en.wikipedia.org/wiki/Autological_word
The tl;dr is, if you can rank within the top 1-20 results for the grounding query, you can poison the LLM “overview” if you convince it your information is legitimate.