Even though I regularly read papers in this field and I generally try to keep up with the state of the art, I keep finding myself exclaiming “wtf” whenever something like this comes out. Like, you can see it made the same mistakes other generatie models make, but then it corrects them (mostly) once we get closer. It wasn’t the sharpest or completely flawless, but I’m in awe about the stability across such a long video and about how lifelike it all looks.

Edit: ah it’s using world models just like PlaNet did for reinforcement learning. I suspected something like that because of the stability of the generation. Absolutely amazing results.

It’s like watching a dream where things shift and change as you get closer to your destination. All of a sudden the street is lined with cars when a moment ago you thought you were driving on an empty residential road.