Foundation models cannot preserve basic physics, and that gap matters more than benchmark scores suggest

The test Munawar Hayat describes is almost aggressively simple. Two cardboard boxes, one larger than the other, photographed plainly. A single instruction: generate an image in which the boxes have been unstacked. What comes back, according to Hayat, is not what went in. Shapes deform. Sizes shift. The boxes that appear in the output are not, in any physically meaningful sense, the same objects.

That failure is worth sitting with, because the task is not ambiguous. There is no trick lighting, no complex scene geometry, no occlusion problem to solve. The physical constraint being tested, that an object retains its shape and dimensions when moved, is among the most elementary facts a system reasoning about the world would need to respect. Foundation models, including proprietary ones at the leading edge, do not reliably satisfy it.

Hayat’s point is not that the generated images look bad in some aesthetic sense. The issue is deeper. When a model alters the shape or size of an object during what should be a purely spatial rearrangement, it is demonstrating that it does not hold a stable, persistent representation of that object. The object in frame one and the object in frame two are, for the model, not causally connected in the way they would need to be for physical reasoning to work.

If you take these foundation models, a proprietary models and if you give it very simple images like an image of two boxes, two cardboard boxes, very plain, one is bigger, one is smaller and you ask it to generate an image where we unstack these boxes. So once it unstacks, physical properties of the boxes change. They're not exactly the same boxes. Their shapes might be deformed. Their sizes might be different. Munawar Hayat

This matters well beyond the toy example. Any application that depends on a model tracking objects across states, whether in robotics, physical simulation, or scene understanding, requires exactly the kind of consistency that Hayat’s test reveals is missing. A model that cannot guarantee a box remains the same box when it is moved has a fundamental limitation that benchmark accuracy on image classification or captioning tasks will not surface. Those benchmarks do not ask what happens when the scene changes.

The broader implication Hayat points toward is that current evaluation frameworks may be systematically optimistic about what foundation models can do in physically grounded settings. High scores on standard benchmarks coexist with failures on tasks a human child would handle without effort. That gap between benchmark performance and basic physical coherence is the kind of discrepancy that tends to be expensive to discover late, when systems built on these models are already deployed in contexts where physical consistency is not optional.

None of this diminishes what current models do well. The capabilities documented across language, code, and pattern recognition are real. But Hayat’s two-box case is a useful corrective to any assumption that visual competence at the surface level, generating plausible-looking scenes, translates into physical understanding at the structural level. The boxes in the output image may look like boxes. They are not, in the model’s representation, the same boxes. That distinction is the whole problem.

What would it take to close the gap? Hayat does not spell out a prescription in this claim, and speculating beyond the evidence would be unwarranted. What the claim does establish is that the gap exists, that it shows up on the simplest possible test cases, and that proprietary models at the frontier are not exempt from it. That is a more useful starting point for honest evaluation than any benchmark number the field currently has.