ZH version is available. Content is displayed in original English for accuracy.
Advertisement
Advertisement
⚡ Community Insights
Discussion Sentiment
67% Positive
Analyzed from 12564 words in the discussion.
Trending Topics
#more#https#pelican#llm#don#game#model#models#something#still
Discussion (388 Comments)Read Original on HackerNews
Spaghetti 2026:
https://x.com/dreamingtulpa/status/2083304533829066873
https://xcancel.com/dreamingtulpa/status/2083304533829066873
A good new benchmark should have awful performance to start and there should be a lot of headroom for improvement. This benchmark is also intentionally difficult and requires the LLM to develop the animation through spatial reasoning and first principals rather than existing video generation pipelines. Similar to how SVG generation was out of distribution for most models a year ago.
Imho we don’t need to make benchmarks that draw the whole 3D world. Pelican’s drawing is really nice in its simplicity and complexity at the same time.
It seems to be an obscene waste of compute time to generate useless 3D worlds that are just a bragging - 3D is really heavy discipline to make it right, see Mark Zuckerberg’s ceased attempt with 3D VR…
Multiply it by thousands times as a lot of people have found out threejs lib and prompt “generate 3D world and make no mistake” are new orange/black.
As for the often quoted issues with the bike's frame or problem with the steering column, I can't really tell, I am no bike expert.
I can instead judge how poor of a job it is doing with a LOTR rendition in Three.js, so that seems like a better benchmark.
No, this demo is the useless 3D world, and you're bragging.
A real game would have a lot more immersive of a world, and you wouldn't need to.
I have always preferred the result of getting an LLM to draw an svg or make a procedural animation like this to the uncanny hyper-realistic result of diffusion image/video generation.
Given that limitation, it is incredible what it can still accomplish. And when it falls short, it is often in a charming way, if you are open to seeing it that way. It reflects something like a child's understanding of the world, not entirely wrong, just incomplete.
The fluttering cubes that might have been bees or butterflies were my favorite part.
Anyway, just a small point which probably still wouldn't change the nice metaphor your made.
https://www.booooooom.com/2016/05/09/bicycles-built-based-on...
Totally agree though, anyone with a vague understanding of how bikes works ignores the pelican because they know the bike is unrideable in the first place.
They are listening.
I am not a mechanical engineer, so even prompting well with ME lingo probably will take some effort.
But I do think this was a poor demonstration of the idea for another reason: Tolkien works have a HUGE corpus of training data. It's great that random users can come in and immediately recognize what the footage is, but it fails at the very first thing the pelican was meant to do:
- Render this thing you have only tangential training data of, that also happens to be an asymmetrical object so we can see how much you fuck up the details if you somehow flip the orientation half the time.
They should have used an obscure story, not "Most Studied Piece of Literally Work of The Past Century trademarksymbol"
That took a fair amount of custom tuning and I had to create a tuning view to get some of the behaviors right.
But it was enough fun that I generalized it to take in ~any scene description from a film. It goes out and gets more detailed descriptions and film stills if available but also takes custom stills if you provide them.
My test scene was the Gauntlet scene from Apocalypto. It is low fidelity but does a pretty amazing sequence with somewhat believable physics of the javelins etc.
Here is the docs page with the vertical takeoff / 88 miles an hour time travel: https://contextify.sh/docs
I can share some of the Apocalypto bit if anyone is interested.
If you have time to try the product as it stands, I would definitely appreciate your feedback in particular, and would be interested in checking out what you've built so far for yourself.
I will publish the API to the database as well.
That page with the time machine may not make it obvious, but the product does have a linux client!
It doesn't have the same app window and summarization of the macos version, but the transcript ingestion engine is efficient and the real value is in leveraging the database it builds using the packaged `/total-recall` or your own use of the api. (which is not yet documented but discoverable)
I am building the windows version now. I have been for the past four days. It uses a shared swift-core with the macOS and Linux versions which has been part of the reason it has taken "so long."
If anyone is on windows (or linux!) and would be willing to try it either of the clients my email is in my profile, I would be grateful.
Also I'm working up a short post with that apocolypto anim now.
https://banagale.com/cinematic-canvas-ai-film-animation.htm
I provide a "how I got to this" up front, but if you want to jump right to the Apocalypto stuff, use this:
https://banagale.com/cinematic-canvas-ai-film-animation.htm#...
And if you want to play with interactive demos of the two animation sequences (the tuning tooling I described in my OP) you can go directly there:
https://banagale.com/cinematic-canvas-workbench-demos
If anyone wants to collaborate on building out the cinematic-canvas-workbench project please email me.
When Fable was first released the day-1 demos of it on Twitter (presumably from people who were given early access, and/or Anthropic employees) were pretty much 100% three.js stuff. Yes, it looks nice, but it doesn't tell me any better than an Erdos proof whether the LLM will be able to run my vending machine.
We should feed a snippet of an unreleased book in a novel universe.
This weekend I've been converting a game from three.js to ogl.js in order to see if I can optimise the time-to-interactive loading time. I took the three.js driven page weight from about 600KB (500KB being three.js) to about 50KB, and reduced the loading time from multiple seconds on a 4G mobile connection to around 0.5s.
This has mostly been a combination of Opus 5 and Sonnet 5 in Claude Code. It very clearly has a good grasp of WebGl, and of what impacts page loading times and rendering speed. It was able to drive Claude Code's integrated browser to measure the impact of changes, and as I spiked out a test of ogl.js it could test the differences changes made.
It's not the best game (https://tinyslots.ooer.com) but that's on me. As an exercise in building 3D in a browser, and in page speed optimization, with Claude models I am really impressed.
Anthropic models are excellent at working with Blender APIs, other industry standard 3D modelling programs and tasks, and game libraries that have nothing to do with Three.js.
Three.js is a quite popular library; and browser-based apps are more easily sharable and more portable. So models having a preference for using it when unprompted doesn't suggest anything, just like how models using React or Tailwind by default doesn't suggest anything.
It seems they trained it to be good at it, then requested everyone to demo it.
This is what you get. It's like the more general concept of starting with the written word (or 1000 words) and then replacing it with a picture. You've done something strikingly different, but is it serving the same function?
It's fascinating to see this stuff combine such disparate sources in unexpected ways. But it is parrot, just not in the way you're expecting.
Opus 5 is the first I’ve seen to “one shot” it (in a harness, so it was more than one LLM call).
"Getting Started with Google Wave": https://www.youtube.com/watch?v=eKUAqNGVwX0
Oh never mind, that was Google Glass, another dead google product. That he personally killed with that photo. So confusing to keep track of them all.
Years later Apache moved it to read only because of low community activity.
The archived git repo on GitHub remains available to clone and revive as a fork.
https://github.com/apache/incubator-retired-wave
wave failed for weird google organizational reasons far more than anything inherent to the product or tech
https://www.youtube.com/watch?v=4Z4RKRLaSug
Their "World Quester 2" tutorial shines as the holy grail of consistent and ergonomic user interface and game design. The menuing system is so magnificently structured and well organized, it bring tears to my eyes. Google Wave pales in comparison.
https://www.youtube.com/watch?v=0Gy9hJauXns
Last I checked, I did not see the prompt for this really cool thing, so it is not reproducible.
Did I miss the prompt somewhere?
this guy seems to have taken that idea and got something similar/better, so likely the prompt isn't too special
https://x.com/Izkimar/status/2083819741643178208?s=20
In either case, I still don't see how I could reproduce this to test against various models, which is the entire point of Simon's pelican.
This is a bigger difference to the pelican than simple reproducibility steps. The pelican is intentionally esoteric, and thus open ended. The LotR is mundane and has a "correct" answer, aka, copy the movie.
It makes it a really awful test of capabilities. The pelican isn't a slop test. This crap is.
It refused to use the text verbatim because of copyright (ironic), but the output was interesting nonetheless.
https://claude.ai/public/artifacts/275dc3c2-7bd3-432b-94ff-d...
Also, large models refusing to work with copyright material is really hypocritical, copyright enforcement for thee but not for me
I'm actually beginning to wonder if some people who ignore these things have a different, somewhat lesser ability to percieve image details than I do.
I mean I guess its fine to go on to another test despite never actually passing the pelican bike test, but there's a sense that we have to use another test because AI is now good at pelicans on bikes, which is just not true.
Please remember, we've started from there :
https://simonwillison.net/2024/Oct/25/pelicans-on-a-bicycle/
When it started, it was clear what LLM would stand out, its style, etc. Nowadays, the pelicans look similar, the difference is in details and sometimes hard to catch. Sure, the task is not completed perfectly, but that's not the point. It was supposed to be a benchmark to quickly benchmark a LLM against others.
It's physically impossible.
The problem is to draw it in the least disturbing way possible.
When you aren't sure if an LLM can write an svg well, or that it will be able to form a pelican shape, or animate a bicycle, it's a good test. After that, it's all judgement: how detailed should the pelican be? pelicans are the wrong shape for a bicycle by default, so how much can I change its physiology to match using a bicycle before it isn't a pelican? Do I care about how well the client is able to render complex geometry?
It's not that there isn't room to do better, or that it doesn't tell you anything at all, but rather we've reached a point where what it tells us isn't very clear anymore.
Oh come now. I am extremely confident that if I hired a professional artist to draw a picture of a pelican riding a bicycle, I would get something inarguably much better than what today's best coding LLMs can produce.
I think AI folks have done a terrible job of communicating this, but replacing a professional simply isn't the point. The point is to serve all the situations where people would've never considered hiring a professional, and where perfection or artistic merit isn't the point (say a personal throwaway recreation of an LOTR world).
And I think in that regard the benchmarks are pretty good.
If the expectation is that AI is going to replace "knowledge workers" then the limit would be a darn perfect drawing. We are nowhere close to that.
And Elon is already propagating the age of abundance where money won't exist anymore, right before calling the interviewing journalist dishonest and deservedly losing public trust. Smh my head.
What knowledge workers do you know that have excellent drawing skills? I worked in a design agency and for a couple of years, each week me and a few other people would attempt to sketch a member of our group: one person would be the model and sit still, and everyone else would draw her/him.
Let me tell you, if producing a convincing portrait was a prerequisite for being a knowledge worker, there would be 99% fewer knowledge workers.
* some omitted the bottom of the diamond which connects from the pedals to the rear wheel
* some added an extra connection from the pedals to the front wheel, making it impossible to steer
* none could align the head tube with the fork
* none added a correct offset to the fork
* none could generate the chain properly in a way that attaches to the two sprockets correctly
I mean just look at these:
* Grok 4.5: https://s3.eu-west-1.amazonaws.com/images.dylancastillo.co/p...
* GPT 5.6 Terra: https://s3.eu-west-1.amazonaws.com/images.dylancastillo.co/p...
* Sonnet 5: https://s3.eu-west-1.amazonaws.com/images.dylancastillo.co/p...
from https://dylancastillo.co/posts/pelicanmaxxing.html
The bike is generally okay, apart from medium which derped hard. Max has correct diamond, correct head tube, and so on. Only nitpicks are that the front fork offset isn't there and the chain doesn't touch the rear sprocket correctly.
It used to be a very difficult task for models, see [2,3,4]
it cuts across several tasks that AI used to be very bad at, but now has improved quite a bit. Namely, spatial reasoning (because it has to manually place the points of the svg such that they make sense and form what it says it forms. This used to not work very well, with random shapes floating around that it would mark things like "eyebrows" but were nowhere near the "eyes", etc.
It also tests the model's world knowledge (what do pelicans look like? sure they have wings, feet, beaks etc, but what shape are they? how to get proportions roughly right? this isn't a given from text data about the bird. This goes doubly for a bike, which is a quite complex shape that most humans fail to draw correctly[1] (many draw the frame or chain connecting in impossible ways that would not ever function mechanically)
Before it was pelican on a bicycle there were people having it do horses/unicorns making the rounds - gpt4.0 or whatever would often make hideous abominations of legs and mouths
[1] https://www.gianlucagimini.it/portfolio-item/velocipedia/
[2] https://static.simonwillison.net/static/2026/mistral-small-4...
[3] https://static.simonwillison.net/static/2025/codex-hacking-m...
[4] https://static.simonwillison.net/static/2025/gemini-2.5-flas...
So it's a pretty much pointless test now.
It's quite surprising actually.
Not only that this so-called "benchmark" isn't economically useful, but that it tests for the sake of testing; and for attention.
To end this obsession with generating SVG pelicans, Quiver AI's [0] model is actually designed to generate SVGs from prompts and has done so for years.
So there is no need to continue with this un-serious pseudoscientific "benchmark".
[0] https://quiver.ai/
"Draw an animation of this long ass scene from a movie, and only call me when everything works e2e" can be.
Consuming radium and using uranium glass, that’s what we’re doing.
But when visual perception and "taste" get involved, it becomes a lot harder.
a real benchmark is instead running evals on your own traces, and building a cost/quality/speed profile for models based on real workloads. but it doesn't get you a shiny video you can post on twitter.
https://arxiv.org/abs/2303.12712
https://svg-bench.fenx.work/
https://static.simonwillison.net/static/2026/fable-max.jpg
Maybe excitement and wonder, in tech, is just something for us old guys, that watched it all be birthed. Get off my lawn!
Bilbo's house is actually described in The Hobbit and the exterior isn't really described at all in Fellowship. The prologue of Fellowship (Concerning Hobbits) mentions hobbits like round doors and windows and the fact some hobbit homes are underground, but the turf-dome design here is not mentioned. It actually mentions hobbit homes typically have bulging walls, so unless you've read the The Hobbit, you might not picture this entirely-underground style.
In The Hobbit, his home is described as a (nice) hole in "The Hill" with a perfectly round front door and round windows, which could imply the design here.
I’ve been trying it on them all and can’t find one that does it consistently. The best will tell me they can’t. The worst confidently point out one of countless Waldo-likes.
Again comes back to the point of verifiable rewards. The moment you take it away from LLMs they just stop being as good
I think LLMs will be excellent glue of "find the right function/button and run/push it" but design without constraints and they just explode immediately
I've seen them make time estimates that are physically impossible, expect medications to be in two places at once, etc.
Why do you say this? I have some experience of manufacturing processes and am not seeing where AI would be useful other than to drive the robots which we can already do quite well without AI (see all the dark/lights-out factories that already exist).
Where do you see it being useful? An example would be nice.
Asking AI to design real world objects doesn't work very well because all of its tests involve proxies and thus miss things that are glaringly obvious when the object is actually built.
We already have sneaker designs and the equipment to manufacture them. Whatever it spits out is going to be, at best, a mediocre clone of something that already exists. What exactly is the point?
fires a missile at Pakistan
Here is your refactored component:
[0] https://news.ycombinator.com/item?id=48838228
Definitely impressive demo, I do wonder though if the countless artwork, films, images etc produced over many decades around Lord of the Rings somewhat influenced the outcome of this though.
Everyone knows roughly what a bicycle looks like - wheels, frame, seat, peddles, handlebars etc, but the details of the frame and exactly how the other parts connect to it throws people off. It seems people memorize the "concept" of a frame, but not the specifics.
Try it without cheating, then google a picture of a bicycle!
Even if you have memorized what a bicycle (and pelican) look like, writing code to draw one is not the sort of thing humans are good at any more than they are good at mentally calculating cube roots - better to use a computer for stuff like that.
I think it's useful when evaluating "model+intended harness", but I'm more interested in seeing raw model improvements than harness improvements.
I wonder whether we are entering the era of throwaway software. Just like cheap plastics and improved processes has enabled us to rapidly manufacture anything we want for a very low price, maybe LLMs give us the same for software. Produce it cheaply and if it breaks throws it away and reproduce it.
Why would anyone still use off-the-shelf software when they can have a system that has access to all data, can transform it into any form, and can export it in any format?
After years of thinking that I needed to develop a decent movie management system for my own films or a columnar browser for large CSV files, Claude and Qwen each delivered exactly what I needed in just a day.
On the one hand yes, almost every task I work at now is one off one of scripts I throw away.
The question is - where does the software the spec or the “code”.
A really complex game will probably always be token heavy. At least for the next few years code is still not free.
But certain software is just iterative by design. If we mean we regenerate all the for loops of a game from scratch, sure but I think “code” Is really more spec then implementation, and we’ll want to continue building things through iteration.
And even on the for loop point - Do you really want to spend millions of tokens rewriting a game every time you need to make balance changes?
Have they? Most of the world production is tied down to expensive factories and machines. Yes, we have more products, but that the result of the global trade, which is a very complex system.
> Produce it cheaply and if it breaks throws it away and reproduce it.
I don't know why everyone would ever wants this. It's been parroted since forever, but the true usefulness of software is to be able to build it once and runs it indefinitely. If some edge case occurs, I fix it. Which is way cheaper than rebuilding the whole thing. The goal is to have something like OpenBSD's ed[0] or dmesg[1], which you only touch every few years or so
[0] https://github.com/openbsd/src/commits/master/bin/ed
[1] https://github.com/openbsd/src/commits/master/sbin/dmesg/dme...
However I do agree that the results are much worse when you try to use them. They look great in screenshots and video clips which makes them perfect for content farmers.
All of the LLM generated games I’ve played have been really bad to play, though. I even tried my hand at a simple game, thinking I could iterate on it with prompts to fix some rough edges. After the initial productivity burst every change turned into a slog of tokens with one thing changing and something else breaking it. I would try to use my remaining weekly token budget across Anthropic and OpenAI to refine it at the end of every week but after a couple weeks it felt like I wouldn’t be getting anywhere without scrapping it and going back to having the LLM build it one step at a time with my careful instruction.
Which, in retrospect, is the only way I can get usable output of an LLM for anything complicated, so it’s not surprising. It’s a fun reality check project though.
On the one hand I think most of us are incredibly impressed because we know, that quick demo would have taken us months of work to build in the before times.
On the other hand the promise is a cure for cancer and the end of all work.
So when everyone is telling you “skill issue is why you can’t one shot WoW”. It’s hard to know how you’re supposed to feel about Karpathy advertising one shot custom virtual worlds but giving you slop. Incredibly impressive slop when compared to how long it would take to create it just 3 years ago, not so much compared to Elon saying - “by the end of this year, grok will create a version of the odyssey that competes with Nolan’s”
I guess this is the average story and, similarly, the average game is boring and predictable
Early generative AI at least had the virtue of relentless, unsettling weirdness, in the same way that generative art from the late 90s and early 2000s did. A handful of people made creative use of that spooky weirdness.
Now it turns out "Airspace" art.
If you took the best, most creative, human writer in the world, and for thought experiment reasons they had amnesia (to mimic AI blank context windows) specifically while you asked them for a story idea 100 times in a row, my expectation is that this human would also give you the same idea at least 80 times out of that 100.
* still better than the mean human, but even the top 0.1% of humans aren't all professional authors.
I did get pip, but it's a story about trying to rescue the moon from a lake and the raccoon lives on moonberry lane
What's pip from?
So "moon" is also part of the predigested artifacts. Maybe because raccoons are nocturnal?
These conversations get frustrating since one group is saying the dancing is bad, a second group is saying it's good (for a bear), and a third group is saying we are a few years from a bear-only dancing industry.
There was a vision of the Hololens being a general purpose compute device that enabled something more like the power of a desktop without being tethered to a sitting position at a desk. You could see it if you squinted hard (incidentally, the device caused a lot of eye strain). But even at the height of the VR hype cycle, I didn't know anyone pushing the Hololens on anyone. Nobody was telling the world they were going to be using Hololenses v1 or v2 as their only computer "or get left behind."
That's oddly specific and it'd really hurt my cousins feelings :)
I guess the next question is if they can be made fun without too much additional work with a human guiding the AI.
They can't. If you think about how these things are trained it's blatantly obvious fun is an impossible metric to optimize them for
One would hope it accelerates AR/VR - if decent studios finally get behind it.
Used to be if a game looked that good it probably had time spent on the game part too.
Whenever I think about AI games I find myself thinking about Tiny Wings, Flappy Bird and Angry Birds. Three simple, elegant games.
It is easy to see what makes Tiny Wings so completely loveable — it has a sculpted, adorable, perfect charm with a cleverly inverted game mechanic that has a calibrated level of exasperation and reward.
But why were Flappy Bird and Angry Birds, very basic games with very old game mechanics, so charming?
It seems equally impossible to imagine an AI coming up with a game with the quality of any of them, even with maximised creativity. But explaining why for Flappy Bird seems quite difficult, especially when you consider it uses some stolen visuals!
Maybe it's the smoothness of motion that makes these games understandable and LLMs seem to consistently fail at that. Ask them to do something snowboarding and they go really hard on the physics since it seems like they don't know what kind of approxmations feel good.
(It's been a while since I was in the game industry, so IDK quite how accurate this is).
If I see one more “one shot MMO” where you just walk around and do absolutey nothing or another menu slop idle battler or rogulike deck builder I’m going to go Postal in Minecraft.
These one-shot products aren't games. They're barely even demos. I don't even know what to call them. For a mature framework like Phaser to sell-out like this and create a vibecoded platform for vibecoded games is shocking.
As with painting, after a while there's nothing really new to paint, we genuinely need 0 new software. We need to fix our broken physical world, our social lives, our kids and what's left of our democracies.
This software crap is done, leave it to the nerds.
Textbook and blackboard > ipad.
Karpathy’s point about the models not being able to easily audit their work is something I’m struggling with —- how can the audio output be made perceivable? Curious what people think about this question.
Here’s a synth to play with+remix
https://underscore.audio/s/cmp_8b226859-420/iron_rareflash
It always brings to my mind some words from Rich Hickey:
I don’t think I really have a point to make here, other than it just feels like someone’s released a bunch of carnival bumper cars onto the highways.Not difficult to see why the employees of AI firms are thrilled with it though, eh?
I like the analogy.
I guess this is why we got this before cars sold without steering wheels: literal guardrails on the literal roads are somewhat more expensive, especially for the people who keep bouncing off them on the way to their destination.
Also, where the guardrails are absent: oh look, felonies. https://www.google.com/search?q=ai+hacks+company&tbm=nws
Take chess engines of the "deep blue" era for instance. These engines are stupid, trying millions of moves that are obviously terrible, no human chess player would do that. And yet, this is what worked best, because computers are so good at repetition. Recent development using neural networks made chess engines smarter, but using repetition is still how the beat humans.
What you call "guardrail programming" in another context would genetic algorithms, a technique that has recognized applications. And if you look at videos of genetic algorithms learning to play racing games, it literally looks like a bunch of carnival bumper cars onto the highways, but done well, after some generations, it becomes competent driving, sometimes even record breaking.
It can be expensive, it is often the case for LLMs, so you may want to try to be a bit smarter at first to spare some resources, but to me, it is just using computers as intended.
https://threejs.org/examples/
Here's minecraft: https://threejs.org/examples/webgl_geometry_minecraft.html
Here's an FPS: https://threejs.org/examples/games_fps.html
The library is extremely well-documented. When Three.js vibe coded projects started blowing up on Twitter 1-2 years ago, I wasn't that impressed then either because I knew what it was doing.
Anyone who remembers the C compiler built by an LLM! backlash probably feels the same way:
Why would I use an LLM to create a well-known demo rather than fork that demo itself?
What I haven't seen yet from an LLM is it create anything fundamentally new and exciting: A new UIX that is actually good. A new service that is actually good. A game with an art style I haven't seen, music, storytelling - something you'd expect out of a AAA studio.
Given that rant: The coolest part is the multi-modality between text and animation. However, I think the end product would have been a lot better if it was just a video. Having it do it in Three.js didn't add a ton of wow factor for me, and it would have been a lot better looking as video.
> Something like an ephemeral GTA of X on demand.
Here's where you lose me. AAA gaming is very far away from this Three.js demo. But the novel part being the syncing of narrative to the visual scene - a text-to-audio (video) book type technology seems very possible (and useful).
Nice work. We need more big projects like this involving AI (if anything, to get away from the slop argument largely focusing on 1-shot experiments).
> it's an example where we go from "no one would ever do this" to "sure, why not, it's ~free".
How is two hours worth of token generation free? With tech revolutions things get cheaper/faster/better/doable, but there’s still real world limits. The advent of railroads made it feasible for the average person to cross the country, but it still cost a lot of time and resources. People weren’t crossing the country every weekend for fun just because it was now doable.
Why do we treat LLMs as ~free when we are generating things that weren’t doable before but have to invest more money than the Apollo program to build AI data centers let alone account for the operating costs?
That method of perception probably scales N^2... so sure with more compute, LoTR animation will improve. But I think to get a real jump in "experiential feedback", perception needs to scale linear or sublinear. Maybe that's there LeCunn's jepa will come in.
There needs to be the removal of the middle man:
image -> text -> action
To image -> action.
This is an odd take, given that Karpathy is certainly aware that the LotR films absolutely did create Bag End in digital format; that their creation was outstandingly high quality; and that Claude’s output here very obviously “leans heavily” on their prior art.
telnet towel.blinkenlights.nl
Now, were they „in their right mind“? I don‘t know. The more likely explanation is they found joy in it. Reminds me of what the Suno CEO Mikey Schulman said about making music: „[…] I think the majority of people don’t enjoy the majority of time they spend making music.“ (https://news.ycombinator.com/item?id=42688538)
8 months ago, he was (very reasonably) claiming that reliable agents are at least a decade away, but this now goes against the interest of his employer, so the narrative has been changed.
Or maybe over these 8 months agents improved a lot? You know, few years ago many AI experts predicted that things we are routinely doing now with AI are decades away. I mean how can you look at this post and not be impressed? It's insane what AI is currently capable of.
[0] Seriously. Get used to mentally prefixing his and Boris Cherny's name like this, every time you see them quoted. These people are speaking while employed; there is no chance they are not aligned with the employers who will make them wealthy. The tech industry does like to pretend that for some reason AI people, uniquely, speak thoughts unbiased and for themselves or even for science or humanity.
I gave Fable a jpeg and asked to draw an SVG, using a loop that renders the SVG into an image so Fable can inspect it.
Results looked like drawing of a 5 year old.
I wonder if Flash is still popular... LLM can use that instead...?
There are people in their right mind who would do that and their are already examples of people who did similar things.
But maybe not in the future if people would confuse all the effort with AI
Computer graphics will have enormous applications because they are directly controllable by LLM-generated code. Video models are probabilistic and less suitable when precision matters. In education, for example, we need exact visuals. If an AI wants to plot y = sin(x), it should generate the precise graph through computer graphics rather than approximate it with a video model.
What I find funny is that approximate to computer graphics are video games. When I ask AI about a decision available to me in a video game AI completely fails, OFTEN. I assume all the forums and changes made to a game over time might be quite confusing for AI. But I've also seen it completely make up characters and decisions and weapons and so on about some very clearly defined games and paths. It's an interesting dynamic.
that's why these things are actually pretty good at openscad/freecad/F360 mcps , the visual reality is enforced and guaranteed by rigor in the interpretation engine that is anchored to human physical reality.
I think it's interesting that the "Bag's End" interpretation in the video clearly looks like the one from the movies, but generated here as a three.js 3D asset.
It makes sense that the movies (or shots/frames from them) were in the training data, and I can also easily imagine an association in concept space between the textual description of Bag's End and the frames from the movie.
But how on earth does the model then go on and convert the latent representation of those images into coordinates for a 3D mesh, without ever even restoring the image? In what kind of representation are the images from the movies stored that it can do that?
Better than I would have expected? Sure. Impressive that it got that far? Definitely. But is the end result good?
I like where Karpathy is going; I had the same thoughts about LLM generated slop scenery. I just want some variety of scenery for the goblins to get massacred in in whatever fantasy slop game I play.
It’s not. That blocky appearance and limited lighting was deliberate comic aesthetic choice as a response to budget, not a result of technical limitations. CGI in 1985 was enormously more capable than that — take a look at the stained glass knight in Young Sherlock Holmes, or The Last Starfighter from 1984. Things certainly could have been rounded, sculpted etc.
Yeah, please check in with the folks protesting data center builds in their town causing their electricity prices to skyrocket and tap water to turn into a scarce resource.
And now, instead of actually doing the above, please go ahead and downvote me, because how dare he question those LLM games.
It’s a useful benchmark (aside from being “cute”) because of its simplicity, both in how many output tokens it takes (though I understand some models think a lot now to do it) and how easily one can subjectively judge. It’s this efficient as a benchmark of performance.
Making a long video takes way more tokens, and presumably is a lot tougher to easily compare. swillison has a presentation that’s pelicans from 2023-present (roughly) showing the progression. Imagine “lord of the rings videos from 2026-2029” or whatever, it would take a long time to watch and be harder to judge, and probably just end up being a comparison of screenshots anyway.
TLDR I feel like the post misunderstands the role of the pelican thing though if find it very hard to believe he really doesn’t understand, so maybe I’m missing something.
Except it's not ~free, it cost ~$10. And no one in their right mind would ever exchange $10 for that crap output except in this brief moment that we're in because it's fun and surprising to see what will happen. The actual result is as close to useless as it's possible to be - it's not interesting in its own right, it's not aesthetically pleasing, nor funny, nor informative. It's just slop.
There’s a reason AI slop games have literally zero engagement. Last summer that stupid flying game blew up. Maybe a million people “played” the game. Where play means they clicked a link and checked it out not because of what the game was but solely because of how it was made.
In terms of concurrent players that game wouldn’t have cracked the Top 5,000 on Steam.
My metric for AI games is “number of players who spent more than 15 minutes playing”. I’m not aware of any vibeslop that has achieved 1 such player.
Now obviously LLMs are transformative for game dev. But “hyper custom worlds you can drop into” shows an extreme ignorance of what players want imho.
From my experience (PixiJS game), Opus can look at things and screenshot. It's slow but it works. The issue is that it doesn't try to make them look good.
Instead, it tries to find one bug and then fix that one bug and close the session. That is amazing discipline for webapps or whatever, but for game design is just the worst if you need attention to detail and tuning of multiple holistic systems that produce an effect together. That really limits the kinds of approaches you can can use to achieve things visually.
Sure, the testing harness could be better, but that's not what makes it a poor fit. The base model seems also capable (it can identify the issues, just isn't willing to disangage from this "I found X and fixed it" single-thing work posture).
The kind of work is just different, and it was probably never trained on it, so it feels off and an uphill battle to use it.
Testing, assessing, tasting...
We also do it when we build other things - this is just a different scale.
LLMs should be tested in the same way people should be tested for a job interview (but often aren’t) - with tasks RELEVANT to usage.
So you don’t just randomly pick some random thing to make the LLM randomly do (like many job interviewers do).
You start with clear statements about real world usage scenarios. THEN you come up with tests that give insight to how well the LLM/hob seeker gets the job done.
Please, stop coming up with random tests like it’s Microsoft in 1990 and you’re asking job seekers how the would move Mount Fuji, as a way of assessing their programming skills.
No stupid irrelevant pelicans on bicycles and no stupid renderings of Lord Of The Rings. Unless those are relevant use cases.
Any test that anyone comes up with must clearly state the context and how the outcome is measured.
But this demonstrates we're a couple orders of magnitude away from generating 1:1 hyper-personalized entertainment and media for individuals, rather than the masses.
On top of that, a decent chunk of the joy of entertainment is the social aspect.
Now, whether this would be at all possible, or even healthy for you, I don't know.
Ultimately, most people don't have ideas for the kinds of personalized entertainment they want, and they don't want to be in charge of content production (even if you have an LLM do most of the work). I don't doubt that there are niches for it, especially stuff like porn, and I'm sure that pros (game studios, film studios) will leverage AI more and more, but I suspect that most of us will just want to sit on the couch, watch Spiderman XVIII, and then be able to talk about that shared Spiderman XVIII experience with all our friends.
And this personal media thing, yeah maybe for terminally online persons that are REALLY into a sub-genre but otherwise, it's too much effort, I agree. Until we get machines that can read our (subconscious) mind, that will not exist.
They were visually bad but honest and sometimes soulful or playful.
The AI slop that's everywhere now looks superficially more professional but it's very busy, samey, unnatural and it really turns me off.
Not even close. Current AIs have very poor spatial awareness, they can generate some kind of scene but they can't tell you where objects are in the scene, nor can they move objects into different places. They're very useful but it's difficult to be very creative with them because they can't update an image to make it more aligned with your vision for what should be in the image.
LLMs produce an extremely small range of artistic expression, and its kind of ironic, that while it certainly looks functionally good. It immediately becomes noise because everything looks identical.
So companies are going to have to hire creatives again, in order to stand out, and be "creative"
1:1 AI entertainment probably won't be a cold start experience.
Much like the "Choose Your Own Adventure" books of the 80's, consumers choose a baseline template, customize the characters, and interact at various points within the plot.
No, we don't. It's quite good, but it's nowhere near perfect. For example, you can see: https://genai-showdown.specr.net/ or others that highlight how far from perfection we still are.
> and it hasn't really changed the nature of human expression.
It sure has changed our discourse. Look at hackernews, where people just can't help themselves to engage in ragebait and flamewar comment threads about llm generated accusations. We've got half the people convinced that looking at ai content is like consuming food.
I'd venture a guess that it's too early to determine how this technology will change expression. A good analog would be photography, which caused a similar meltdown in the arts at the time.
> I don't see my friends getting wildly creative.
Maybe you don't have creative (enough) friends.
The fans were real heart broken about this but I think you're right on where this is going. We're not going to be seeing the dominance of centrally produced content like this for much longer, like sure, I think there will be big blockbusters will stick around, but I think the day is coming where media becomes a choose your own adventure sort of scenario.
It'll be interesting to see where this scales to. there will definitely be some amazing solo projects but we'll also see the like 4 player co-op version of productions and then the larger mine-craft server 'Minas Tirith' scale ambitious projects that involve a few dozen people. And of course passive consumers will remain a thing, or people who just provide some suggestions or nudges for what they'd like to see others make.
But I don't think it'll be dominated by big companies like Disney, Netflix, Amazon or Paramount.
I'm sure a lot of them will suck but it'll be neat to see the inevitable Seinfield - Star Trek Voyager cross over episodes.
Elaine and B'Elanna Torres feud after a transporter accident leaves the crew stranded the delta quadrant. Jerry attempts to date 7of9 but is rebuffed as she finds Kramer's quirky bluntness more relatable. George panics after someone compares him to Neelix.
Everyone knows how a MacBook looks. Any missing or incorrect detail will be obvious. However, Pelican or this world can be anything.
https://playcode.io/blog/macbook-svg-benchmark
EDIT: The parent comment originally claimed they spent $1M on the demo -- they seem to have edited it after I replied.