Show HN: Bullshit Detector – agent skills that fact-check videos and articles

sskorniienko about 3 hours ago 53 commentsRead Article on github.com

ES version is available. Content is displayed in original English for accuracy.

⚡ Community Insights

Discussion Sentiment

75% Positive

Analyzed from 1478 words in the discussion.

Discussion (53 Comments)Read Original on HackerNews

mysterydip•about 2 hours ago

What do you use as “ground truth”? The page says “independent sources”, and I’m sure there’s too many to list, but my question is how are they vetted as being truthful and how are two sources with opposite viewpoints reconciled?

simonw•about 2 hours ago

Something I've found surprisingly effective is telling ChatGPT to "use credible sources" - you can then watch its thinking trace and see it do things like ruling out random blogs, considering media publications with a good reputation for fact checking, and double-checking information that seems unlikely.

westoque•about 1 hour ago

I don't think random blogs should be ruled out. Just because a blog is random doesn't mean its untrue.

simonw•about 1 hour ago

The reputation of the writer matters a lot. Recent models seem to be pretty good at evaluating author credibility as well.

What's dangerous is treating information from an anonymous author as factual. You gotta check that in other ways.

bulbar•about 1 hour ago

It should not be seen a credible source though.

It can be, but that's hard to correctly judge on at scale.

skorniienko•about 1 hour ago

Love it, I'll run tests for "credible sources" and will compare the results. Could be a first feature-request implemented!

autonomousErwin•about 1 hour ago

I could imagine running multiple different AIs seeing if they agree if something is true.

I imagine this is similar to how Twitter's/X's community notes work. Something along the lines of when you have two accounts that disagree traditionally and they agree on something. That's how you know it's likely to be true.

So from that approach, you actually want to have two AIs that are on the other side of the spectrum of whatever you're trying to find out if it's true e.g. if a conservative AI and a liberal AI both agree that something is false, it's highly likely to be false.

mtweak•about 1 hour ago

This is extremely problematic. Much of news is highly syndicated, so what looks like 10 credible sources are actually just 2. I think the best you can aim for is empirical sources: receipts, videos, photo evidence, public disclosures directly at the source. Archiving is ok if disclosed.

I don't think LLMs can discern truth when the majority of news sources are incentivized primarily for views.

skorniienko•36 minutes ago

Empirical sources - would be a first candidate to be implemented in a skill. And you are right about 10 credible sources are actually 2. It will look like 10 independent sources to it, real gap. Skill does prefer primary source over secondary. For example it'll prefer NASA article over blog post who cited NASA article.

I'll work on empirical sources improvement in a next releases.

skinfaxi•about 1 hour ago

You may have overlooked this part of the parent's comment which addresses your syndication issue:

> Something along the lines of when you have two accounts that disagree traditionally and they agree on something. That's how you know it's likely to be true.

Outfits that blindly parrot talking points would not be at odds and thus not be good candidates.

hunterpayne•about 1 hour ago

It also doesn't help that many news sources will knowingly fudge the truth. There are topics about which they are constant sources of misinformation. Seems like depending on the topic (anything science related) you wouldn't want to use news sources at all. Other topics, you probably only want to use news sources. And arranging all of this weighting of sources is going to be difficult and controversial as well. Doesn't mean it can't be done. Doesn't mean it wouldn't be valuable. But there is a lot to it and most of it isn't so much about technology or math as it is about understanding who should and shouldn't be considered authoritative about what topics.

Sounds challenging to me...

skorniienko•about 2 hours ago

There is no whitelist source. It can't rate sources for truthfulness. When sources conflicts - skill drops verdict to misleading or unverifiable and both are linked. Can't pick a winner at the moment. That's probably the weakest part and still a judgement call for a model.

gutechh•about 2 hours ago

Maybe you read them already, but wikipedia has a bunch of interesting pages about sourcing and truth like: https://en.wikipedia.org/wiki/Wikipedia:Verifiability,_not_t... and https://en.wikipedia.org/wiki/Wikipedia:Reliable_sources

skorniienko•about 1 hour ago

I'll look at reliable-sources, and you are right "verifiability, not truth" is basically what I landed on too. Model doesn't get to decide what's true, it just needs to cite something.

Will read reliable-sources page properly.

egeozcan•about 2 hours ago

Can someone please use this skill against the claims in its own repo?

skorniienko•about 1 hour ago

Done, fixed 2 misleading lines in README. You can see example report in repo examples.

Thanks for that, btw! At least someone questioned it :)

egeozcan•about 1 hour ago

That was an exceptionally fun to read slop :) Cool idea in any case!

alealvarezarg•about 1 hour ago

I think this would have a much bigger impact as short-form video content such as TikTok, Reels or Shorts. That's where most people consume this kind of content today, not in a .md file.

What would really set it apart is adding fact-checking before publishing and letting it post directly to TikTok. That combination would make it something I'd actually use every day.

skorniienko•21 minutes ago

Not sure that I get your idea about short-form content. It already works with tiktok or yt shorts, especially if it has generated transcription/captions. Otherwise, skill will download video locally and ask your permission to run local models like whisper to transcribe video.

Checking before you publish - that's interesting usecase I haven't thought about. Nothing stops you to fact-check your own content and decide what to do with it. Point is a report itself not more video content from it

h2aichat•about 1 hour ago

I like the idea! How can you let folks know they're being fed nonsense before they even finish the intro?

skorniienko•27 minutes ago

can't, at least with this design. Report itself takes minutes to proof-check. Realistic version is some kind of extension/plugin idea from earlier in a thread, but I'm not there yet

leobg•about 1 hour ago

I don't even watch most videos anymore. YouTube app -> Share -> Gemini -> Summary. Only if that shows interesting content do I actually watch.

skorniienko•about 1 hour ago

And you are not alone, that is one of the purposes why I am building this

N_Lens•about 2 hours ago

Who checks the checkers?

qntmfred•about 2 hours ago

me. I volunteer as ultimate arbiter of Truth.

skorniienko•about 2 hours ago

maybe some community-driven and curated source blacklist could be implemented. It thing it would be too much for you doing that alone :)

skorniienko•about 2 hours ago

At the moment - it's model's judgement upon search results for every claim from provided content. If search results returned inaccurate data, or were fabricated - it might affect a verdict for that claim, or highlight that it's unverifiable or contradictory.

But there is a room for improvement, what do you suggest? Have a Judge agent which will check results?

madmaniak•about 1 hour ago

For truth checking actual thinking and logic is required. It doesn't seem to be a proper tool for the task.

skorniienko•32 minutes ago

True, and it doesn't do logic. It decomposes content into claims and search for every claim citations. It catches a false premise, not a bad inference. Someone with valid reasoning from true facts to a wrong conclusion goes straight through.

cawingcrow•about 2 hours ago

That's too much effort. Should have been a browser extension that uses ollama so it can run with local models or maybe even openrouter...

skorniienko•about 2 hours ago

That's fair. Install is a one time real friction and extension would be much simpler.

But, there is a reason why it's not an extension. At least for now. Whole product isn't a UI or service, it's a set of skills for your AI agent of choice, which gets content from URL or whatever shared as a source, splits it into claims and does extensive web searches for every claim to compare it with statement from provided content.

On local models constraint isn't a model itself - but search. Model don't judge from it's trained memory, so even local model will need a backend for search, otherwise it can't provide a verdict.

outime•about 2 hours ago

The author is posting comments here (all flagged at this time) that are very clearly AI slop, which is quite ironic.

skorniienko•about 2 hours ago

Fair, drafted posts with claude, getting all my comments flagged. Thought that something is wrong with my account. Writing myself from here

NetOpWibby•about 2 hours ago

Bro, you don’t have to outsource your brain for basic communication on the internet.

HN’s community has built-in bullshit detectors.

skorniienko•about 2 hours ago

You are right, no excuses. Will have few grammar errors but I'll own them. Cheers!

amelius•about 1 hour ago

Funny, I recently pasted output from Gemini into Claude, and it said it was total nonsense generated by an overconfident AI.

Apparently the "AI-generated bullshit" detection is already part of LLMs.

skorniienko•about 1 hour ago

but have Claude actually fact-checked it or just provided an opinion?

lynx97•about 2 hours ago

> Long output? Redirect to a file and read it from there:

Hmm, smells AI generated. Why should an LLM (this is from a SKILL.md) care about $LINES?

skorniienko•about 1 hour ago

Done, fixed in v0.4.1. Reason why agent reads from a file - it can do it in chunks or pass the path to subagent and keep it out from own context. I've tested it on 3hrs long interview YT video - worked smoothly.

quard8•about 1 hour ago

nice! my built-in BS detector: "comment "prompt" and i will send you a guide".

skorniienko•about 1 hour ago

thank you! comment "prompts" never works, right? Haven't seen single guide from it :D

khalic•about 2 hours ago

Awesome idea

skorniienko•about 2 hours ago

Thank you! Would love to hear your results if you'll have a chance to try it

khalic•about 1 hour ago

I have an AI newsdesk i vibed together because I was losing too much time hunting for facts inside the torrent of misinformation, I'll try to hook it up to the writers

skorniienko•about 1 hour ago

Fantastic! Dying to hear the feedback. <3

ck2•about 2 hours ago

Washington Post gave up tracking Trump lies last term in 2021 because it became impossible by human hands with 21+ per day and over 30,000 in their database

but with "AI" now it's possible not only to do non-stop but in REALTIME

you could even just restrict the source of the check to the paper's own reporting the past fifty years

* https://www.washingtonpost.com/graphics/politics/trump-claim...

I'd like to see that backfilled, all the way back to the "long form birth certificate" (remember that horror show)

BugsJustFindMe•about 1 hour ago

I'm not sure AI helps for that particular task. If you just assume that 100% of everything he says is a lie you'll be 99% correct. Anyone who believes anything he says at this point is living in a constructed reality where facts never matter. Fact checking seems more useful for less pathological cases where it might make a difference.

hunterpayne•about 1 hour ago

Cool, now do Fauci...

PS I've always assumed all politicians lie, I'm sure you don't think Pelosi or Bush Jr for example were some paragons of truth. Doesn't make it OK, but have some perspective.

BugsJustFindMe•14 minutes ago

> Fauci, Pelosi, Bush Jr, presumably other people

> I've always assumed all politicians lie

Everyone lies at various points to some degree or another, ranging from benign to well-intentioned to criminal to malignant, and it takes tremendous sleight of hand to pretend that they're even close to being in the same league wrt frequency and significance.

> Doesn't make it OK, but have some perspective.

Indeed. Perspective: https://en.wikipedia.org/wiki/False_or_misleading_statements...

> Doesn't make it OK, but

Why the "but"? Are you trying to justify something? Should we be?

o10449366•about 1 hour ago

god i wish these low effort projects were banned from Show HN

the entire premise of a "bullshit detector" that's entirely vibe-coded is laughable

skorniienko•about 1 hour ago

why it can't be vibe-coded? Would you be more pleased if it would be poorly hand-coded and claimed "No AI used during development"? What's the point?

I don't mind that I've used AI to build this, and I'm open about it ;)

vivzkestrel•19 minutes ago

- this is precisely the problem my current project solves

- production grade app generator without AI :)