How it fails, measured

You have spent four sessions learning what these tools cost and where they stop. Tonight is different. Tonight is about the thing they do wrong while appearing to do it right, which is the failure mode that survives review, gets approved, ships, and then gets noticed by a viewer.

Here is Physics-IQ in full. Six models, one benchmark, and a ceiling drawn where real-world physical behaviour sits. Look at the empty space to the right of every bar before you read any label.

Take the ranking seriously for a second, because it is doing something unusual. Sora came last. Sora was, at the time of this benchmark, the model people pointed at when they wanted to show how far generated video had come. It produced the footage that went round the internet. And on the question of whether it knows what happens next, it scored 10.0 out of 100.0.

If realism and physics were even loosely coupled, that could not happen. The fact that it does happen is the finding. You cannot infer one from the other, in either direction, and every review process built on "watch it and see if it looks right" is measuring the wrong axis.

A second benchmark, VideoPhy-2, comes at the same question from the other side and lands in the same place. On its hard subset, the best model achieved 22 percent joint performance, and the paper singles out where the models come apart: they struggle "particularly with conservation laws like mass and momentum". Which is to say, things appearing, things vanishing, and things moving with an energy nothing gave them.

The most common complaint about generated video is that the person changes partway through. People describe this as glitchiness, as if it were random noise. It is not. It is a set of specific, separately measured dimensions, and it is worth knowing them by name.

The detail worth taking away is that identity and clothing are scored as two different things. Somebody sat down, looked at the failure space, and decided that a face becoming a different face and a jacket becoming a different jacket are distinct enough problems to need separate numbers. That tells you how routine both are.

Instance Preservation is the third one on the diagram, and it is the objects. A cup in the hand, a sign on a wall, a second person in the background. They are supposed to persist. Sometimes they blink out and come back, which is the thing that makes a shot unusable in a way you cannot fix in the edit.

Then there are the two numbers on that chart that should decide what you commission at all. Complex Plot scores below 12 percent for every model tested. Dynamic Spatial Relationship scores below 22 percent for every model tested. Not the worst model. Every model. If your brief involves a sequence of events with a causal order, or two things moving in a defined relationship to each other, you are asking for the two things nothing in the field currently does.

And Human Anatomy ranges from 59.72 to 88.58 percent. Take the top of that range, the best result anyone posted. Roughly one generation in nine still comes back anatomically wrong. That is your realistic reroll rate on a shot with a person in it, from the best model, before anything else has gone wrong.

You would expect the company that built the model to know its weaknesses best. Right now, in this category, that expectation is simply wrong, and it is worth stating plainly rather than politely.

Session 3 taught you that the real constraint is always three clicks deeper than the marketing page. This is the harder version of the same lesson: sometimes the constraint is not deeper in the vendor's material, it is not in the vendor's material at all, and the only place it exists is a paper by people with no commercial stake in the answer.

It is worth being fair about the asymmetry, though, because it is not universal. You have already seen counter-examples in this course. Google DeepMind publishes that consistent spoken audio "remains an area of active development". Descript documents that its own filler-word removal is English-only. C2PA writes its own limitations into its security documents. Candour exists in this field. It is just distributed unevenly, and you have to know which vendors have it before you rely on their docs as your risk register.

AI dubbing is sold as a translation problem with a voice attached. It is not. It is a scheduling problem, and the schedule was fixed the moment somebody finished the picture edit.

The formal statement of the problem, from the isochrony-aware dubbing work, is that dubbing a spoken sentence requires transferring "the content as well as the speech-pause structure of the source into the target language". Two constraints, not one. It is not enough for the words to mean the same thing. The gaps between them have to fall in the same places, because a mouth on screen is opening and closing on the original schedule.

Translation length is not free. Some language pairs run consistently longer than their source, and a faithful translation that comes out 30 percent longer has nowhere to go. The picture is finished. So you get three moves and there is no fourth: speed the delivery up until it sounds hurried, cut content until it no longer says what the original said, or let sync break and watch a mouth stop moving while the voice carries on.

Every AI dubbing product makes that choice for you. Most of them do not tell you which one they made. That is the question to ask in the demo, and it is a much better question than asking how many languages are supported.

This is the weakest evidence on the page, and it gets a section anyway, because how you handle a weak number in public says more about you than any strong one.

The question everybody asks, and it is the right question, is how many generations it takes to get one shot you can actually use. It decides your budget, your timeline and whether the project is viable. So you would think somebody would have measured it.

Now watch what happens when somebody tidies it up. Seventy thousand divided by twenty is three and a half thousand, and "3,500 generations per usable shot" is a wonderfully quotable line. It will get into a deck. It will get repeated. It sounds like a measurement.

It is arithmetic on an undefined denominator. If a "clip" is a first-frame still, the ratio means one thing. If it is every crop and version, it means something else entirely. Dividing a number you cannot define by a number you can does not produce a rate, it produces a decimal that inherits all the ambiguity of its numerator and hides it behind a colon.

So here is the honest position, and I want you to be able to say it out loud in a meeting without flinching. No independent audit of generation-to-usable-shot ratios has been published. Not a bad one, not a small one. None. Anyone quoting you a tidy ratio is quoting either this arithmetic or a vendor's content marketing.

Bring any generated clip with a person in it, yours or a public one. You are going to review it the slow way, once per dimension, and find things you did not see the first time. Work through the steps in order.

1. Open the Physics-IQ paper and find the score table yourself. Confirm 29.5 at the top and the ceiling at 100.0 with your own eyes. Then find the sentence containing "unrelated to visual realism", and copy it out by hand. You will use it in an argument within a month.
2. Search for VideoPhy-2 and count how many of the first ten results state 22 percent. Note how many give the higher aggregator figure instead, and how many cite the paper rather than each other. This takes ten minutes and it will change how you treat summaries permanently.
3. Run the six-pass review on a clip somebody else made and approved. A brand advert, a product launch, anything published. Find at least one named defect that shipped. The point is not to be smug about it, it is to see that the review gap is industry-wide rather than personal.
4. Find the technical-limitations section of a system card for any generative video model you use. If there is not one, write down that fact, with the document name and the date you checked. That absence is a finding, and it belongs in your risk register rather than in your assumptions.
5. Start the generation log. Prompt, model, cost, and one of used, rerolled, abandoned. One project is enough. In three months you will own the only iteration number in the room that anybody can defend.

Clear all three and session 5 gets stamped in your passport.

Every fact on this page, and its verification tier, is recorded in the course's source map., including a quarantine list of what is deliberately not stated.