Hister: A private search engine for the pages you visit and the files you keep

Kind of related to this in that I built it to hoard knowledge from web pages I've visited along with implementing a Karpathy-style LLM Wiki, but the knowledge is collected automatically from sources I browse.

I have it up on GitHub, but I don't think anyone should use my implementation.

Loosely, what I built:

* On each of my machines I have a cron job running that looks at all my web browser history (usualy it's inspecting the brower's SQLlite across firefox and chrome). If it matches my rule list: hacker news stories, certain reddits, etc. it'll grab the page, convert to markdown and drop in my Obsidian Vault incoming.

* It has a whole de-duping architecture since I might open the same page on multiple machines. Uses the CloudFlare SQLITE D1 storage for tracking the processed links.

* it'll then trigger the LLM to do some Karpathy wiki style taxonomy assignment to the articles, organize them, create an index etc.

It's then available for my "bot" stuff to do writings for me.... I will probably write more about it at some point. I'm not certain it's totally useful and not just a yak-shave on hoarding knowledge.

Ai-drafted article on this [1]

Example AI-Drafted article based on some discussions the other day on Ollma vs LLama.cpp [2]

[1] https://taude.xyz/posts/how-archivore-turns-browsing-into-a-...

[2] https://taude.xyz/posts/skip-ollama-run-llama-cpp-directly-o...

I'd like to use it, but I'm hesitant to use anything that isn't a reviewed and approved package in my Linux distribution. Even if the chance is 1% that a program I download has malware or security problems that even the author doesn't know about (eg., due to libraries used), odds are that my system's going to be compromised if I run 50 such programs. This extends to browser add-ons, bookmarklets, and extensions too.

How do other people handle this dilemma?

Even solution I can think of involves are a great amount of extra work.

Ohi, author here! Thanks for posting Hister. Feel free to A.M.A. My first free software search project was Searx, a privacy respecting metasearch engine, but because of the limitations of the metasearch concept, I've decided to take a different approach.

Hister builds a personal search index from pages you visit, bookmarks, browser history, local files, and crawled websites. It stores extracted content with offline result previews, so information remains searchable even when the original page changes or disappears. It supports full text and semantic search, can run entirely on your own machine, and includes a web interface, command line tools, and an MCP endpoint for assistant integrations.

Website: https://hister.org/

Tiny read-only demo: https://demo.hister.org/

Ps.: It looks like our name conflicts with a registered trademark in the US. The owner of the other project has asked us to change it, so we’ll probably need to comply sooner or later.

Name suggestions are welcome! Ideally, the new name should be relatively short, sound good, and have an available .org domain.

Thanks!

Google Chrome did this in 2008. Full-text search over all visited pages, stored offline. It was very useful and I miss it.

Nobody seems to remember it, even though it was a headline feature. Was removed in 2013, I think due to technical constraints.

Will definitely try this.

I attain this without involving an untrustworthy third party, with one simple trick: Print to PDF.

Every single web page I’ve found interesting, since the advent of the Web, I have printed to PDF and stored locally for my own personal reference.

Something like 80,000+ files - my own copy of my own Internet - indexable, searchable.

Available offline. Something to read when I am far out to sea.

There is no need to involve third parties in your Internet history - no matter how trustworthy they seem to want to appear.

Print to PDF, and you’ve got everything you need, safe and sound.

I thought that said Hitler at first.

I had the same problem for a very long time but it is largely solved now. I started to simply ask chatgpt "hey I read something about x, y month ago but can't find it now". There is a surprisingly high chance chatbot can just give the exact answer back to me, usually with extra interesting reading materials as a plus.

Interesting, I actually very recently built a similar project [1]. I was unaware of this...If I were I probably wouldn't have bothered!

[1] https://git.brucewillis.sexy/~tombert/fs_index I promise, safe for work, despite the URL.

A recent discussion about this tool

https://news.ycombinator.com/item?id=49351802

I am happy to see the idea of history search more. I am on my 3rd version of my own. the use of local LLMs has made it easier to support features like weekly summarized and recipe extraction.

I like the search ui. my projects become functional but never polished. https://github.com/sbeckeriv/memoir

> Your own search engine — Hister is a private search engine for the pages you visit and the files you keep.

Is there any site/project that works as a fully customizable personal front-end to all other SERPs?

When I search for something, I always want a link to the best Wikipedia result. This should always be in the same place and have a giant icon/picture.

Then there could be easily clickable links to the SERP pages for Google, DDG, etc. for that query.

A big link to route it to your favorite LLM.

Seems like you could have a really useful "homepage" for all searches that sat in front of all the other sites. It could be local only and would not require indexing the web. Also wouldn't be a files search thing, as Hister appears to be.

Bonus point for having an IRC channel for community forum.

Ooh, very nice! I have my own tool I’ve been using for this [https://github.com/bradrn/full-history-search/], and it’s incredibly useful, but it’s also pretty primitive. This one looks a lot nicer.

I've been using this since the last time it came up on here. I don't have it index every page I visit, I use the browser plugin to tell it to index specific ones. It is useful for sure, but I think it will really shine once I've been using it long enough for it to build up a bigger index of things that are old enough that I've actually forgotten about them.

I've been trying this out the past few weeks. I was literally just in there searching for a link 5 minutes ago.

It's badly needed, and so far it's working well for me.

Can you add viewed tweets?

This is really cool. I like the idea of combining it with a offline Wikipedia cache.

Saw on Discord that they have to change their name, since https://histre.com sent them a letter.

Integrate it with linkwarden so it searches the bookmarked pages.