Homomorphically encrypted CIFAR-10 inference in 200ms

Thanks for taking the time and test it.

Key management is a critical part of the story, as j2kun has pointed out before. In this demo, we're intentionally caching the same key material for every visitor so we can showcase the actual FHE computation without making everyone wait through client-side key generation and upload. Even as a one-time cost, having each user generate their own keys in the browser and transfer them to the server introduces noticeable setup time, which we felt would get in the way of the experience for a demo. We actually implemented it that way at first; setup wasn't a matter of minutes, but we changed course after worrying about the experience for visitors on mobile data.

This demo is focused on demonstrating the computation itself, not a production-grade key management flow. Supporting per-user keys and a more realistic trust model is definitely on our roadmap for future demos.

IMO a big unanswered question in FHE (which is my field so I care about the answer) is key management. If you have a service with lots of different users, and each user requires 100 MiB or more in evaluation key material, simply loading that key material from disk onto the GPU for each user's request adds a nontrivial overhead on the order of magnitude of the FHE computation itself. Not to mention the storage cost (for a 1B-user service, each requiring 1 GiB of evaluation key material, requires 1 EiB of storage, which is a hard sell).

So it seems kind of reasonable to me that if you want to showcase your fast inference for its own sake, you can sidestep the remaining engineering problems around key management by giving the user pre-computed key material corresponding to hot-loaded evaluation keys.

Indeed, we have our own library, Cyclops. We will share more about it soon :)

did they remove a German Shepherd from the sample images since your comment? I cannot see a German Shepard. Only a black dog with floppy ears.

That's a exceptional tradeoff. Even 3000x is noteworthy as long as processing time does not go to the moon wrt HE/FHE

I think calculator is not part of the labels in cifar-10. Pretty normal as it has ‘only’ 90% accuracy. Things could improve quickly though

Consider encrypted AI. You ask a question under encryption. The remote calculates the answer, which is still under encryption. This is the critical point; the remote cannot see the question and answer. It only calculates. Once you receive the answer, you decrypt it and see the answer; only you see the answer.

From the webpage:

> Image classification without the server seeing the image.

The value is in being able to get the prediction from the model without ever giving unencrypted data to somebody you don’t trust.

You could have an LLM generate code for you without ever giving the operator your proprietary source code.