Show HN: Kage – Shadow any website to a single binary for offline viewing

197

ttamnd about 3 hours ago 48 commentsRead Article on github.com

⚡ Community Insights

Discussion Sentiment

96% Positive

Analyzed from 1337 words in the discussion.

Discussion (48 Comments)Read Original on HackerNews

simonw•about 1 hour ago

I was intrigued to see how the demo GIF in the README was generated: https://github.com/tamnd/kage/blob/01e75b87ecc893bbba7943c63...

Turns out it's using another project by the same author: https://github.com/tamnd/ascii-gif

The script used for the demo is at https://github.com/tamnd/kage/blob/01e75b87ecc893bbba7943c63... and has a comment showing how to run it:

  ascii-gif render docs/demo/kage.tape -o docs/static/demo.gif

Looks like it's an opinionated wrapper around https://github.com/charmbracelet/vhs

ninalanyon•about 1 hour ago

> kage serve $HOME/data/kage/paulgraham.com

If the result is static why does it need a server? Isn't it possible to make it so that it can simply be opened by the browser? Like:

$ firefox $HOME/data/kage/paulgraham.com

Then the result would be useable on machines without kage nstalled.

doctoboggan•about 1 hour ago

Usually JavaScript is blocked when you load pages that way.

recursive•12 minutes ago

I am quite familiar with this and it is factually false

embedding-shape•25 minutes ago

Since when? You won't be able to make HTTP requests to localhost, as it'd be a different Origin, but I don't think any mainstream browser blocks JS outright when you use file:// to load and view HTML files.

rzzzt•9 minutes ago

Somewhere around 2019, each document loaded from file:// became its own origin in Firefox: https://bugzilla.mozilla.org/show_bug.cgi?id=1500453 (I didn't check when this happened in Chromium)

Related WHATWG discussion: https://github.com/whatwg/html/issues/3099

dmazzoni•14 minutes ago

Not all JavaScript, but a lot of APIs are restricted

pixelatedindex•about 1 hour ago

I thought all the JS was stripper?

afavour•30 minutes ago

You’ll likely run into a ton of CORS issues doing that.

embedding-shape•24 minutes ago

I don't think so, there is no HTTP requests being done from JS as it's stripped away, and all the other resources are pulled down (and I'm assume their reference made relative), so really shouldn't be any issues because of CORS at all.

wolttam•about 3 hours ago

One use I'd have for this is company wikis that you want to give folks easy offline access to (maybe the wiki has documentation that's useful at sites that don't have cellular coverage).

Cool!

It would be especially cool to have a version that didn't require the separate serving process - even though it's nifty you can package up a whole site as a single binary.

Maybe a single HTML entrypoint shim with a bit of javascript that could index into an archive (potentially embedded) of the site's content?

tamnd•about 3 hours ago

Submitting this to Hacker News is the right place! Thanks for your idea. I will consider implementing that :)

Also, in my mind, I already have a script/program to convert HTML to Markdown, so it could actually store everything on disk as a folder of Markdown files, and then commit them to a Git repo.

maxloh•about 3 hours ago

I find SingleFile [0] to be a much more robust version of this.

It strips out all the JavaScript too, but also packs everything into a single HTML file that is easy to transfer. Binary assets (like web fonts and images) are packed as base64 strings.

They also offer a CLI powered by Puppeteer. [1]

[0]: https://github.com/gildas-lormeau/singlefile

[1]: https://github.com/gildas-lormeau/single-file-cli

wamatt•26 minutes ago

Love love love SingleFile too. The FF extension works pretty well for a clean save.

That said, Kage looks promising if OP can combine SingleFile reproduction quality with the HTTPTrack spidering approach. SPA's are kinda tricky with archiving and do wonder how well Kage would handle that

tamnd•about 3 hours ago

It seems this repo only saves one web page?

What I'm implementing here is mirroring a whole website, with all its subpages, so you can browse it all offline. For example, all essays from paulgraham.com.

maxloh•about 2 hours ago

Oh, I see. In that case, feature-wise, it is actually a modern alternative to HTTrack.

I think the misunderstanding stems from the browser's "Save As" reference in the description. It is misleading. You use "Save As" to save a single page, not an entire website.

Also, the description lacks a clear explanation of the project's purpose. It would be helpful to include a sentence explaining that the program downloads an entire website, not just a single page.

arikrahman•25 minutes ago

This is what I first thought and it's a very elegant solution, and not needlessly overcomplicated.

tamnd•about 3 hours ago

And thanks for the link. Let me implement this single HTML feature, it looks nice to have!

maxloh•about 1 hour ago

Yeah. An idea on top of that is to bundle an entire website into a single HTML page, with vendored JavaScript to enable client-side routing (all of the original pages' JS is still stripped out).

That way, the page is self-contained as it is, but requires no bundled binary code to serve the site. It is actually safer security-wise.

The vendored script can be as simple as this:

  const site = {
    "path-1": "<!DOCTYPE html><html> ... </html>",
    "path-2": "<!DOCTYPE html><html> ... </html>",
    // More paths
  }

  function attachListeners() {
    for (const [path, html] of Object.entries(site)) {
      document.querySelector(`a[href=${path}]`).onclick = () => {
        document.documentElement.outerHTML = html
        attachListeners()
      }
    }
  }

  document.addEventListeners("DOMContentLoaded", attachListeners)

HelloUsername•about 3 hours ago

What's the difference with, any webbrowser on a computer, File -> Save as ?

dmazzoni•10 minutes ago

Save As works fine for simple websites with static content.

Let's say you have a site that fetches content from a database. If you Save As, then at best you'll get a local copy of an HTML page with JS that loads the content from the same remote database. It might not work (since the local copy has a different origin), or if it does, it requires you to be online, which defeats half of the purpose.

What this project, and SingleFile, both do is save a snapshot of what the rendered page actually looks like at that moment in time. The scripts are stripped out so it runs locally and has no external dependencies.

nmstoker•about 3 hours ago

That's for a single page, this handles the whole site. Also the browser Save As options often work poorly.

telesilla•about 2 hours ago

I've been using httrack (https://www.httrack.com) to download wikis to read on flights, which isn't perfect but better than I'd found previously. I'll try this out, I'd be delighted to have good results. Thanks for the post.

KellyCriterion•8 minutes ago

Sounds like .MCH-files re-invented? (-:

gregwebs•about 3 hours ago

This seems like it has potential to create a lot of load on a site- are there settings to set how fast it clones or avoid images/videos? Is there a way to only get a subset of a website?

tamnd•about 3 hours ago

Could you help create a new issue for that? I will do it later. It is already 1:00 AM my time, but I am happy that anyone is interested in it. : )

dimiprasakis•about 3 hours ago

Neat project, I like the idea. One thing from a quick read: you launch Chrome with --no-sandbox. Is there a good reason for that? Security wise it's probably not a good idea. If there is no reason, I'd suggest leaving the sandbox on!

In any case, cool stuff :)

shinryuu•about 1 hour ago

Reminds me of this. https://gwern.net/gwtar

Compared to that is there anything kage does better?

Igor_Wiwi•about 2 hours ago

This is quite useful tool, especially for the cases where internet access is limited (the flights for example). I implemented it as a separate feature in mdview.io: for example you can export a document as a html file for offline usage, with all the presentation features like reach tables, mermaid and etc built in. Example https://mdview.io/s/why-markdown-became-default-format-for-a... then try to Export - Export HTML

sanqui•about 3 hours ago

Cool concept. I would like to see this combined with mitmproxy for archive grade fidelity. You could be saving exactly the data served and at the same time a representation by a modern (contemporary) browser, with all JS having run. This combination would be my perfect replacement for the WARC format.

tamnd•about 3 hours ago

I'm working on WARC too, with format from Common Crawl!

By converting it to Markdown, we save a lot of space, but it is for a different purpose and a different project: https://github.com/tamnd/ccrawl-cli

sanqui•about 3 hours ago

That's neat! In my opinion, the WARC format is quite tricky and underspecified especially since HTTP2 introduced new semantics. It encodes too much in-band and requires rewriting of the server data. A mitmproxy capture is higher fidelity and supports capturing modern features such as WebSockets. I think if we could wrap Kage's crawler interactions by it and store its capture (the intercepted traffic), we could make a potentially nice new archival format.

tamnd•about 3 hours ago

I tried to follow well-known formats first, such as WARC and ZIM from Kiwix, so we could benefit from existing tooling support.

For my own custom data format, I have a lot of private code that I plan to release soon. It is optimized for compression, fast lookups, and more. I have been working on it for two years. This is part of a larger, ambitious umbrella project: I am building Google from scratch (all open source), something that anyone can host, including the crawler, indexer, storage, and serving layers. Stay tuned!

Dhavidh•about 3 hours ago

sound interesting

latexr•about 1 hour ago

For those with an eReader, one thing that works really well is using pandoc to download and convert a webpage to EPUB that you can then load to your reader.

  pandoc --from html --to epub --output /PATH/TO/FILE.epub https://example.com

arikrahman•24 minutes ago

Thanks, will try this out on the Kobo later.

rahimnathwani•about 3 hours ago

So this is like using wget --mirror except that it works on pages that require javascript, right?

tamnd•about 3 hours ago

Yeah, it is. For example, openai.com is rendered with Next.js, so I will try to mirror it tomorrow.

lolpython•about 2 hours ago

This is cool. I could see myself downloading the articles behind the first couple pages of hacker news with this, for viewing on a flight or long distance train ride with spotty internet

daviding•about 2 hours ago

Nice idea! fwiw, false positives and all, but the Windows 11 default Windows Security doesn't like it: `leakless.exe: Operation did not complete successfully because the file contains a virus or potentially unwanted software.`

soulofmischief•42 minutes ago

Cool project! I know it's written in go, but it would be cool to see something like this which uses Cosmopolitan Libc + redbean or something similar to create a binary which runs anywhere. Would be fun to be able to pass around self-executable website archives.

https://github.com/jart/cosmopolitan

https://justine.lol/cosmopolitan/index.html

https://redbean.dev

(Certificates just expired for justine's website, just ignore the warning.)

delduca•about 2 hours ago

curl can do this

chinnyys•about 1 hour ago

The readme is AI slop, and incredibly grating to read. The disgust I felt while reading it almost put me off trying the project.

Is the code also AI slop?

grahamstanes17•about 3 hours ago

nice