Are you ready to be shocked? Runway’s Gen-2 reveals the major flaws in text-to-video technology!

In a recent panel interview with Collider, Joe Russo, the director of Marvel blockbuster movies like “Avengers: Endgame,” predicted that within two years, AI will be able to create an entire movie.

I’d say that’s a pretty optimistic timeline. But we are getting closer.

This week, Clueto Google-backed AI startup that helped develop the AI ​​image generator stable diffusion, launched Gen-2, a model that generates videos from text prompts or an existing image. (Gen-2 previously had limited and waitlisted access.) The follow-up to Runway’s Gen-1 model released in February, the Gen-2 is one of the first commercially available text-to-video models.

“Commercially available” is an important distinction. Being the next logical frontier in generative AI after images and text, text-to-video is becoming a larger area of ​​focus, particularly among tech giants, several of which have demoed text models. to video in the last year. But those models remain firmly in the research stages, inaccessible to all but a select few data scientists and engineers.

Of course, first is not necessarily best.

Out of personal curiosity and service to you dear readers, I ran some pointers through Gen-2 to get an idea of ​​what the model can and cannot achieve. (Runway currently provides around 100 seconds of video generation for free.) There wasn’t much method to my madness, but I tried to capture a variety of angles, genres, and styles that a director, professional or armchair, would like to see. on the silver screen, or a laptop, as the case may be.

One limitation of the Gen-2 that became immediately apparent is the frame rate of the four-second long videos that the model outputs. It’s pretty low and noticeably, to the point where it’s almost like a slideshow in places.

What’s unclear is whether this is a glitch with the technology or Runway’s attempt to save on computing costs. In any case, it makes Gen-2 a rather unattractive proposition for publishers hoping to avoid post-production work.

Beyond the frame rate issue, I found that Gen-2 generated clips tend to share some graininess or blurriness in common, as if some sort of old Instagram filter had been applied to them. Other artifacts occur in places as well, such as pixelation around objects when the “camera” (for lack of a better word) quickly circles or zooms in on them.

As with many generative models, the Gen-2 isn’t particularly consistent with regard to physics or anatomy, either. Like something conjured up by a surrealist, the arms and legs of people in the Gen-2-produced videos merge and separate again as objects melt to the floor and disappear, their reflections warping and distorting. And, depending on the message, the faces can resemble dolls, with glowing, emotionless eyes and pale skin reminiscent of cheap plastic.

To top it off, there is the issue of content. Gen-2 seems to have a hard time understanding nuance, clinging to particular descriptors in prompts while ignoring others, seemingly at random.

One of the prompts I tried, “A video of an underwater utopia, shot with an old camera, in the style of a ‘found footage’ movie, produced no such utopia, just what looked like a first-person dive through a anonymous coral reef Gen-2 had problems with my other indications as well, as it couldn’t generate a zoom shot for an indication that specifically asked for a “slow zoom” and didn’t quite get the look of your average astronaut.

Could the issues be related to the Gen-2 training dataset? Maybe.

Gen-2, like Stable Diffusion, is a diffusion model, which means that it learns to gradually subtract noise from an initial image made entirely of noise to bring it closer, step by step, to the indicator. Diffusion models learn through training on millions or billions of examples; in an academic paper Detailing the Gen-2 architecture, Runway says the model was trained on an internal dataset of 240 million images and 6.4 million video clips.

Diversity in examples is key. If the data set does not contain much footage of, for example, animation, the model, lacking reference points, will not be able to generate animations of reasonable quality. (Of course, animation is a wide field, even if the data set did have anime clips or hand-drawn animation, the model would not necessarily generalize well for all types of animation).

On the plus side, Gen-2 passes a test for bias at the surface level. While generative AI models like DALL-E 2 have been found to reinforce societal biases, generating images of positions of authority, such as “CEO or “director,” that are predominantly white male, Gen-2 was a little more diverse in the content it generated, at least in my tests.

Fed with the message “A video of a CEO walking into a conference room,” Gen-2 generated a video of men and women (though more men than women) sitting around something like a conference table. Meanwhile, the result of the message “A video of a doctor working in an office” shows a vaguely Asian-looking female doctor behind a desk.

However, the results for any order containing the word “nurse” were less promising and consistently showed young white women. The same goes for the phrase “a person who waits tables.” Clearly, there is work to be done.

The bottom line from all of this, for me, is that the Gen-2 is more of a novelty or toy than a truly useful tool in any video workflow. Could the results be edited into something more coherent? Maybe. But depending on the video, it would potentially require more work than shooting footage in the first place.

that should not be also dismissive of technology. It’s impressive what Runway has done here, effectively beating the tech giants in the text-to-video punch. And I’m sure some users will find uses for Gen-2 that don’t require photorealism, or a lot of customization. (CEO of Pasarela Cristóbal Valenzuela recently told Bloomberg that he sees Gen-2 as a way to offer artists and designers a tool that can help them with their creative processes).

I did it myself. In fact, Gen-2 can understand a variety of styles, such as anime and claymation, which lend themselves to the lower frame rate. With a bit of manipulation and editing work, it wouldn’t be impossible to stitch a few clips together to create a narrative piece.

Lest you worry about the potential for deepfakes, Runway says it’s using a combination of artificial intelligence and human moderation to prevent users from generating videos that include pornography, violent content, or violate copyrights. I can confirm that there is a content filter, an overzealous one indeed. But of course, those aren’t foolproof methods, so we’ll have to see how well they work in practice.

But at least for now, filmmakers, animators and CGI artists, and ethicists can rest easy. It will be at least a couple of iterations down the line before Runway’s technology comes anywhere close to generating movie-quality images, assuming it ever gets there.