How Does Computer Vision Fit Into Physical AI?

Computer vision in physical AI is the perception layer: it turns camera input into what a machine sees, decides on, and acts on.

Jay Kapoor

20 min read

Computer vision is how a Physical AI system sees. It turns camera and sensor input into an understanding of objects, positions, and conditions the machine can act on. It is the perception layer. It feeds the decisions and the motion. Take vision away and the machine works blind.

Why it matters

Most physical work starts with seeing correctly. A robot cannot grip a part it cannot find. An inspector cannot flag a crack it cannot resolve. A sorter cannot route what it cannot identify. Get perception wrong and everything after it fails, no matter how good the planning or the hardware. Vision is the first domino.

How it fits, in plain English

Physical AI runs on a loop: perceive, decide, act. Computer vision owns the perceive step. It takes raw pixels and depth data and turns them into something a machine can reason about.

A few jobs live inside that step. Detection finds objects and puts a box around them. Segmentation goes finer and labels every pixel, so the system knows where an object stops and the background starts. Depth estimation adds the third dimension and tells the machine how far away something sits. Defect detection hunts for the thing that should not be there. A crack, a burn, a missing bolt.

Vision does not act alone. It hands its output to the decision layer. In newer systems that layer often runs on VLA models, which take vision and language and produce actions directly. Some stacks also run world models, which let a machine predict how a scene will change before it moves. Vision is the front door for all of it. The reasoning is only as good as what the eyes send in.

What computer vision does well today, and what still trips it up

Computer vision is the most mature part of Physical AI. Mature does not mean solved. It helps to know which jobs are basically handled and which still fight you.

The solved-enough work: reading a barcode or a label, classifying a part as type A or type B, finding a known object and drawing a box around it, reading a gauge or a serial number, spotting an obvious defect against a clean background. Factories have run versions of this for years. Cognex has been selling industrial vision systems since the early 1980s. If your problem looks like one of these, the risk is low and the payback is fast.

The still-hard work: estimating the exact 3D pose of a shiny or transparent part, telling apart two defects that look almost the same, seeing an object that is half hidden behind another one, and holding accuracy when the light, the dust, or the part finish changes through the day. These are the problems that make a demo look easy and a deployment run long. Reflective metal, clear plastic, wet surfaces, and deep shadow are the classic breakers.

The lesson for a buyer or a founder is simple. Match your first project to what vision already does well. Save the hard perception problems for after you have a win on the board.

Vision is more than one camera: sensor fusion on hard sites

On a clean line, one good camera goes a long way. On a dirty, dusty, dangerous site, one camera is easy to blind. Serious systems fuse several kinds of input.

RGB cameras give you color and texture, which is what most detection and classification runs on. Depth sensors add distance, which a robot needs to actually grab or place a part. Thermal cameras see heat, which is how you find an overheating bearing, a bad electrical joint, or corrosion under insulation that a normal camera cannot see. Lidar builds a 3D map and holds up in low light where a camera goes blind, which is why it shows up on mining and inspection robots. It beats an optical camera in light dust, but heavy dust and smoke scatter the laser and degrade it too. Radar is the sensor that truly sees through dust and smoke.

Fusing them buys robustness. When low light or light dust kills the color image, lidar still holds the shape. When a part is the same color as its background, depth still finds its edge. When a surface looks fine but runs hot, thermal catches what the eye misses. Hard industries lean on fusion because any single sensor has a condition that defeats it, and those conditions are exactly what a real site throws at you.

What a real deployment looks like: weld inspection, end to end

Take one example all the way through, because the gap between demo and deployment lives in the details.

A plant wants to catch bad welds before they leave the line. The demo is easy. Point a camera at a good weld and a bad weld, and a model tells them apart. The deployment is the rest of the iceberg.

First, capture. The team builds a fixed lighting rig so every weld gets photographed the same way, because a weld under changing shop light looks like a different object hour to hour. Then the model, trained on real welds from this plant, not stock images, because this plant's welds have their own look. Then the hard part: false positives. A system that flags every third good weld as bad gets switched off by the line crew in a week. So the team tunes the threshold, decides which calls the machine makes alone and which go to a human, and builds that handoff into the workflow. Then drift. The wire changes, a supplier swaps a material, and accuracy slips, so someone owns re-labeling new examples and retraining on a schedule.

The camera and the model were the easy 20 percent. Lighting, real data, false-positive tuning, the operator handoff, and the retraining loop were the 80 percent that decided whether it shipped.

How to tell a working industrial vision system from a demo

If you are buying or building, four questions cut through the demo shine.

What are the lighting and mounting requirements? A system that only works under lab light is not ready for your floor.

What is the false-positive rate, and who handles the flags? The error you can live with, and the human workflow around it, matters more than the headline accuracy number.

Was it trained on data from an environment like mine? A model tuned on clean stock images will struggle on your dusty, scratched, real parts.

What happens when conditions change? Ask how the system handles drift and who owns retraining. If there is no answer, budget for the surprise.

Common misconceptions

A camera and a model is enough. It is not. Lighting shifts, dust coats the lens, parts block each other, and the one edge case you never trained for shows up on the line. Real deployments live or die on how they handle the mess, not the demo.

Better cameras solve everything. Resolution helps. But a sharper image of a badly lit, half-blocked part is still a badly lit, half-blocked part. The real gains usually come from lighting, positioning, and data, not the sensor alone.

Vision means one camera. Most hard-industry setups fuse several inputs. Multiple cameras, depth sensors, sometimes thermal or lidar. The machine builds its picture from more than one angle because one angle is easy to fool.

The Dirty Jobs angle

Here is the part people miss. Computer vision already works in industry today. It runs on factory floors and inspection lines right now, which makes it the most deployed slice of Physical AI by a wide margin. The frontier is not proving it works. The frontier is putting it where humans should not be. Dirty, dusty, dangerous inspection is the wedge. Confined spaces, high steel, hot metal, bad air. The machine sees so a person does not have to stand there.

That is the whole thesis. Start where the seeing is hard and the human cost is real.

Apply for Dirty Jobs 2026 on Sep 23, or read the rest of the field guide.

FAQ

Is computer vision the same as Physical AI?
No. Computer vision is the perception layer inside Physical AI. It handles seeing. Physical AI is the full system: perceive, decide, then act in the real world. Vision feeds the loop, but it is one piece of it.

How is computer vision different from VLA models and world models?
Computer vision turns pixels into an understanding of the scene. VLA models take that vision plus language and output actions directly. World models use perception to predict how a scene will change before the machine moves. Vision is the input the other two depend on.

Do industrial vision systems use more than cameras?
Often, yes. Hard-industry setups fuse RGB cameras with depth sensors, and frequently thermal or lidar. Each sensor covers a weakness in the others: lidar holds up in low light and light dust where a camera fails, thermal catches heat a camera cannot see, depth finds edges when color fails.

What breaks computer vision in real industrial settings?
The environment. Poor lighting, dust on the lens, parts blocking each other, and edge cases that never showed up in training. A model that nails a clean demo can still fail on the floor. Most of the engineering work goes into handling those conditions.