WebLLM: high-performance in-browser LLM inference engine

ssaikatsg about 2 hours ago 7 commentsRead Article on github.com

DE version is available. Content is displayed in original English for accuracy.

Discussion Sentiment

50% Positive

Analyzed from 92 words in the discussion.

Discussion (7 Comments)Read Original on HackerNews

TekMol•about 1 hour ago

This seems to be the demo:

I am getting:

    WebGPUNotAvailableError: WebGPU is not supported in
    your current environment, but it is necessary to
    run the WebLLM engine.

On both, FireFox and Chromium on Linux.

toto007•44 minutes ago

You can enable WebGPU support in Google Chrome by turning on hardware acceleration and activating the WebGPU flag. It Works.

refulgentis•about 1 hour ago

Project is de facto dead, used it for many years and had to rip it out 6 months ago, don't waste your time.

skybrian•about 1 hour ago

What did you switch to?

seamossfet•about 1 hour ago

We use the ONNX runtime for small models in the browser https://github.com/microsoft/onnxruntime

sroussey•about 1 hour ago

And ONNX is what Transformers.js uses as well, at least for the moment.

theanonymousone•about 1 hour ago

Transformers.js supports a lot of models since v4