Show HN: Gaussian Splat of a Strawberry

For anybody interested, SuperSplat (and PlayCanvas) is open sourced on GitHub: https://github.com/playcanvas

Thank you for your service!

This is one of the most endearing things about open source IMHO, the way people can find novel uses for it.

That one has a strange rendering error for me, the trees and horizon are in front of the mill building and the exterior isn't properly rendered unless you are in orbit mode. But my mind was a little blown when i discovered that I could walk up the stairs. It needs shift to run!

The scene desperately needs some clipping on the boundaries. If you use an app like Scaniverse, you can add a bounding box to cull far away points which are often poorly reconstructed.

If you have a newer iPhone with a LIDAR scanner, highly recommended. You can make dolls-house renderings of your house or garden which is surprisingly useful for planning and measuring walls/features.

It’s amazing this runs perfectly smooth on my iPhone 12 mini

That is indeed a very cool scene being about to wander around and still have decent resolution

Lots of translucent blobs composited to produce the appearance of a strawberry.

There is no mesh or model. The visual surface of the strawberry could be made up of blobs spaced far apart physically and not where the surface appears to be.

This is why they are called radiance fields, they model the light not the geometry.

Practically the blobs positions/rotations can be constrained to better physically match the geometry of a strawberry.

This video explains how Gaussian splatting works: https://www.youtube.com/watch?v=X8yRlA7jqEQ

Others have provided details about how it works. I suggest zooming way in on that image and you'll start 'breaking though' the surface and that'll help you get an idea of how it works. Important thing is there is no defined geometric surface ("mesh"). Also important to know is that it's very, very hard to get a good splat without taking a ton of photos at different angles. It's also really, really easy to create a crappy looking splat. But when it's done right, it's a marvel

https://youtube.com/watch?v=X8yRlA7jqEQ is how I learned about them. They're really cool!

Here’s a good 2 minute explainer https://youtu.be/HVv_IQKlafQ

You might enjoy some of my art derived from gaussian splats in that case, I've been calling them gaussographs.

https://bayardrandel.com/gaussographs

More recent work on my instagram

https://www.instagram.com/bayardrandel/

I don't know anything about them but it's a cool effect. At least on this strawberry, you're not zooming in but rather traveling closer. I don't see the increasing (made up) detail you'd expect from a zoom, we sort of pop through the skin into an invented interior.

They remind me a lot of the 'memories' scene in Minority Report:

https://www.youtube.com/watch?v=arTIRgdEb1g

I like that they are sort of between a photo and a 3D model. Nothing quite like it.

Yeah, it's an incredibly cool effect. Reality breaking down into fog and frosted glass and paint smudges and slivers of northern lights and all the dandelion fluff. Even navigation becomes harder and less predictable as coherence recedes.

This feels so much closer to how minds must store and process spatial information than the usual 3D models do.

its funny that sky features like clouds and blue patches end up being fit very close to ground level because there isn't a difference in perspective to cue in the algorithm that the skybox should be tall, I wonder if there is any way to incorporate the fitting algorithm with simultaneous lidar data about true distances of things to allow scenes to be viewed from further up

That's what Google Street View should be like more.

Is there a specific name for this “dreaminess” effect?

Is this thing the basis of what Apple is doing in the Photos app in the most recent OS revisions?

It seems to work a lot quicker than 30s now on iDevices and Macs.

Any idea what kind of vram is needed to run this on Nvidia?

There are some works on doing this directly e.g. https://arxiv.org/abs/2601.23065 but getting accurate materials is a challenge for anything more than diffuse.

AI-based relighting will no doubt start working soon.

out of the box, I imagine that surfaces can be lit, but probably not shadowed correctly. (structures aren’t solids, more like particles in 3d space)

it could look like the real-time lighting of an old game engine on rather modern assets. (quake 2-3 era)

or perhaps some "occlusion pre-pass" could be done to create a voxelized sparse volume from the splats that set a "voxel opacity value" for each to absorb light? (not far from how prebaked GI works nowadays)

note: not an expert on rendering, just a nutjob that did stuff in opengl in the old days.

I just wanted to do a quick size comparison...

If I'm reading Chrome right, that's 171 MB, for the website, and the data.

If I'm doing the math right, that's 40 seconds worth of the bandwidth from Netflix, at its highest rate.

This one keeps crashing the browser after it reaches 100%. Safari/iOS, iPhone 13. I was able to navigate and use a few of the other ones linked to from the comments, though. Curious.

Beautiful and quite neat to 'walk' up into the ceiling and look down. Feels like you're either quasimodo or God, depending on how nuts you are.

shame the bishop behind looks a bit flat though ;-)

Those are artefacts. Splats are not perfect around reflections.

I'm pretty sure it's allowed, just as saying "do what you want with it" is an informal license, it's just a little pointless.

It's their content. They can do anything they like.

Sort of. There are countries like France where attribution rights are fundamentally inalienable, and the author can always demand attribution, even after forever waiving the right to attribution. But in the US, the party who benefits from a contractual provision can generally choose to ignore or forfeit that provision. Whether it's still called the "CC BY" after that is debatable, since the whole point is attribution, but licensing by the CC BY and immediately waiving that requirement seems legitimate to me.

Great video. I was about to share it here myself.

It's an artefact unfortunately. Gaussian splats have no concept of refraction and have a hard time dealing with reflection. Highlights, usually so so. Something I always wrangle.

Well, at certain angles you actually get very noticeable gaps in the strawberry that are visible when rotating it, but almost invisible when static. I think it's mostly due to that.

I like to think of Gaussian Splatting as a stage full of improv actors who are iteratively/concurrently filling details in regions down to pixels until the full picture is painted.

Thanks! We have two compressed formats, the sog by PlayCanvas and spz with sparkjs. Both now support LODs and compress really well.

This is an interesting question. There has been some attempt to model the camera better than just a pinhole camera, that could work. https://dof-gaussian.github.io/ https://github.com/leoShen917/DoF-Gaussian

My take.. at a macro scale, the dof is usually so small, that it's hard to get a reliably track. So you'd need some sort of way to tell that these stacked photos belong into a series, and then you sort of are doing focus stacking :-) I do think the alignment algorithm could be improved. Maybe the approaches I linked could be used to make a much more robust focus stacking algorithm, that also corrects for 3D geometry. That would be really cool!

If you don't focus stack and try to train on partially unfocused images, the optimizer will try to match the rendered view to be also partially unfocused.

You would have to mask out the blurry areas for each image. I guess one could just implement a feature where the optimizer only optimizes gaussians within the sharp distances relative to the camera.

Don't know if this would be in your wheelhouse, but for very nice macro splats, check out the work by Dany Bittel: https://danybittel.ch/macro.html

For example this bumblebee: https://superspl.at/scene/cf6ac78e

Edit: I completely missed that this was posted by him (:

This was also my first thought when I zoomed in into the strawberry. I wonder if you can achieve a microscope like effect with a more suitable setup. E.g. better lighting, zoom, lens, etc.

I usually use PostShot.. but the quality was not very good. I want to try LichtFeld but my Graphics Card has too little memory.. so I reached out on twitter, some people ran tests and Mykhailo got some better quality out of it so I took his training. You can d/l the COLMAP dataset for free and try yourself.

Oh, that would be amazing! The photos are also available and some people definitely take it for a test ride. However.. This dataset with the stacking is far from generic.

That's kinda what NERFs are (neural radience fields). They actually preceeded this Gaussian story, with Gaussians coming in and outperforming them. Maybe they'll merge later for something even better, I don't know enough about them.

I could see a kind of fun game / design tool / worldbuilding where you get a blurry world and you describe what you are seeing, and it comes into focus. The game world, mechanics, aesthetic, and playstyle build as you form your view. A sort of fog of war meets rorschach game.

This will be the future of a class of 3d Game. the prompt may not be text however.

An input of a kind of schematic representation of what the designer wants would be better. It may resemble a storyboard or a collection of organised notes that large projects tend to already use.

Fully generative could probably do some cool things, but people will still want to bring their peronal vision to life.

We are currently at real-time video generation that can be converted to splats or meshes.

https://research.nvidia.com/labs/sil/projects/lyra2/

This sounds like it could be a great concept for a future sequel to LSD: Dream Emulator

Google already has some splats on google maps,

https://youtu.be/gXug7Kb3p4I

rendering them is also non trivial from what i understand, since the millions of indivudal point positions need to be recalculated all the time while moving

One thing you can do with this that works quite well is use it to "decrop" (widen the viewing angle) of a video. This is very useful for stabilization which usually involves cropping. Here's an example: https://x.com/i/status/2051504427287404568

Once they get quicker to train, I expect this to be a popular use of them

I watched this 1 month old video two days ago:

https://youtu.be/X8yRlA7jqEQ

It kinda splatted my mind.

As I understand it, Gussian Splats aren’t a method of constructing missing data in the sense you’re asking here.

But there are other, well established, methods of generating missing data in convincing manner, that are beyond my field of expertise or interest to be able to repeat here.

With the added benefit that if I’m wrong, being wrong can be a great way to spur discussion 8)

I’m not sure if the same limitations apply today, but when I last looked into it:

1. Lighting is baked in so they don’t fit well in dynamically lit environments

2. Splats aren’t as suitable for bodies you’d animate, like meshes are. They also don’t work well for very thin or sharp surfaces in general

3. Splats are pretty memory-hungry so large scenes would require better hardware than consoles can offer

These might not be as true today, and I can’t recall exactly when I read this. Probably 3 years ago or so. I recently read they’re making their way into games, so it’ll happen as the kinks are worked out.

And it only takes 20 Minutes to shoot all 7920 photos, the Z8 is crazy fast.

Where's the Lytro camera when you need one?

I was really pulling for OTOY to keep making light field capture and display technology...

It is in fact still on it's mount and slowly rotting / molding.. for a second capture :-)

Yup. It's rotten on the other side. Or maybe lead poisoned

not sure if that's a joke but i think that's just the effect of the strawberry being placed on a glass/plastic surface to be filmed from underneath

It was mounted at the bottom.. and I can't quite reach it with my camera. Might have to try some two pass way.

Gaussian splat casualty. The bottom looks like partially missing from the reconstruction.

The flesh of the strawberry is very slightly translucent. By taking images from many angles, you can derive some information about the interior. This is not a separate step, it is simply the optimizer finding splats that contribute the right amount of color and light at various viewing angles. Of course those splats aren’t really going to look right to anyone who has ever eaten a strawberry, because they are only derived from the input images and not your past memories of a real strawberry.

You’ll also see the same type of problems in any incomplete part of any scene. Go poke around any of them and you’ll find places where there wasn’t enough information to make a satisfying reconstruction. For example, the back of the moon <https://superspl.at/scene/2ac8f423>, the fridge in the coffee shop <https://superspl.at/scene/6a0c3ccf>, underneath the footbridge in the forest <https://superspl.at/scene/23ebe85c> (or indeed anything in the distance), etc, etc.

I just posted some images.. I have actually done my studio, (quality is.. let's say painterly ;-): https://superspl.at/scene/0a3916cd

Not traditional Gaussian splats since they have their lighting baked in but there are alternate forms that have the ability to do so but largely in academia or very specific use cases

Works for me with Firefox 140.10.2esr on Devuan 5.

Works for me on Debian 13, Firefox 140.10.2esr-1~deb13u and Strix Halo gfx

If you can, please check any output/errors on the developer console?

Yes it's shoot in front of a bluescreen.

Sim racing is at least kind of like that.

If you don't like the backside, check out the inside!

Yes mounting.. and I can't quite reach all the way from below.

I posted some pictures.. takes only 20 min!

I also have them on x: https://x.com/DanyBittel/status/2055624819941483005 I thought direct image links are more welcomed here.

What do you mean you can’t run a VPN?

It appears to be rendering locally, my fans started spinning up when I was wandering through another image on the site.

That doesn’t really look like Gaussian splats to me- but this video, Helicopters by A$AP Rocky, uses the technique well:

https://youtu.be/g1-46Nu3HxQ?si=y_u257DzgOg2QV83

What makes you say this is using Gaussian splats? It looks like simple motion-based cropping to me.