The Robot That Picked the Right Part in the Wrong Place
We integrated a vision-guided robot to pick random parts from a tote. The camera was calibrated, the robot knew its position, and the vision software found the part in 2D coordinates. On the bench, it worked perfectly. On the floor, the robot picked parts 2 mm off-center. Not enough to drop them, but enough that the downstream fixture couldn’t accept the part. We recalibrated, retrained, re-ran the hand-eye — same 2 mm error. Finally we checked the camera mounting: the camera was on a bracket that flexed 2 mm when the robot moved. The calibration assumed the camera was rigid. It wasn’t.
Vision guided robotics hand eye calibration is where the camera’s coordinate system and the robot’s coordinate system meet. If they don’t agree, the robot goes to the wrong place — even if both systems are perfect individually. This article is how I set up vision-guided picking that actually lands on the part, not 2 mm next to it.
2D vs. 3D Vision Guided Robotics
Before you pick a camera, decide what the robot actually needs to know.
2D Vision: X, Y, and Rotation
A 2D camera finds the part in image coordinates (X, Y) and its rotation (Rz). The robot converts that to robot coordinates and moves to the position. This works when:
- The parts are flat (picking from a flat conveyor or a flat tray).
- The height is fixed (the robot doesn’t need to know Z).
- The part lies in a known plane (no stacking, no tilted parts).
2D vision is cheaper, faster, and simpler. But it can’t handle parts stacked in a bin, parts at varying heights, or parts with 3D orientation.
3D Vision: X, Y, Z and Orientation
A 3D camera (stereo, structured light, or laser triangulation) captures the part in three dimensions. It finds the part’s position in X, Y, Z and its rotation around all three axes. This is needed for:
- Bin picking: Parts are stacked randomly in a tote. The robot needs to find the top part and approach from the correct angle.
- Parts at varying heights: The part isn’t on a flat surface — it’s on a conveyor that varies, or on a fixture that has height variation.
- Orientation-sensitive gripping: The gripper needs to approach the part from a specific direction (not just from straight above).
3D vision is more expensive and slower (processing a point cloud takes longer than a 2D image). But for random-bin picking, it’s the only option.
| Application | 2D or 3D? | Why |
|---|---|---|
| Picking from a flat conveyor | 2D | Fixed height, flat surface |
| Picking from a tray (single layer) | 2D | Known Z, rotation only |
| Bin picking (stacked parts) | 3D | Varying height and orientation |
| Picking from a pallet (layers) | 2.5D (height map) | Known X/Y, need layer height |
| Parts on a moving conveyor (track) | 2D + encoder | Fixed plane, moving target |
Hand-Eye Calibration: The Critical Step
The camera sees the part in pixel coordinates. The robot moves in robot coordinates. Hand-eye calibration is the math that converts one to the other. Get it wrong and the robot goes to the wrong place.
Two Calibration Setups: Eye-in-Hand vs. Eye-to-Hand
Eye-in-hand: The camera is mounted on the robot wrist. It moves with the robot. It sees the part from the robot’s viewpoint. Calibration relates camera coordinates to the robot tool center point (TCP).
Eye-to-hand (fixed camera): The camera is mounted on a fixed bracket, watching the workspace. The robot moves under it. Calibration relates camera coordinates to the robot’s base frame.
Eye-in-hand is more flexible (the camera can look around obstacles) but requires the camera to be rigidly mounted to the wrist. Eye-to-hand has a fixed viewpoint but the camera doesn’t add mass to the robot.
How Calibration Works
The standard method: put a calibration target (a known grid of dots or circles) in the workspace. Move the robot (and camera) to multiple positions and orientations. At each position, capture the target’s image. The software solves for the transformation between camera coordinates and robot coordinates.
Key points:
- Multiple poses: At least 15–20 poses, spread across the workspace. Don’t calibrate with the target in one position — the math needs variation to converge.
- Spread the orientations: Tilt the camera up, down, left, right. The calibration needs to see the target at different angles, not just head-on.
- Accurate target: Use a factory-calibrated target (printed checkerboard targets drift with humidity and temperature). A metal or glass target with known dot positions is more accurate.
- Verify after calibration: Pick up the target with the robot. Move it to several positions. Check that the camera’s reported position matches the robot’s actual position. If the error is more than 0.2–0.5 mm, recalibrate.
The bracket test: Before calibrating, tap the camera bracket. If it moves, the camera isn’t rigid. A camera on a thin bracket that flexes during robot motion will never calibrate repeatably. Mount the camera on a rigid, braced bracket. No flex, no calibration error.
Picking Strategy: How the Robot Approaches the Part
Finding the part is half the problem. The other half is approaching it without colliding.
Approach Angle
For 2D picking from a flat surface, the robot approaches straight down. Simple. For 3D bin picking, the robot needs to approach from an angle that avoids the bin walls and other parts. The vision system gives the part’s pose; the robot plans an approach path that comes in from above at an angle that clears the obstacles.
Gripper Design for Vision-Guided Picking
The gripper has to handle variation. If the part is picked at a slightly different orientation every time, the gripper needs to be forgiving:
- Fingers with lead-in chamfers: The gripper doesn’t need perfect alignment. Chamfered fingers guide the part in.
- Compliant gripper: A gripper that floats (like the deburring tools) tolerates small position errors. The robot doesn’t need to be exact — the gripper does the final alignment.
- Vacuum cups for flat parts: A vacuum gripper doesn’t need precise alignment. It picks up flat parts from above regardless of small XY variation.
- Servo gripper for multiple part sizes: If the cell handles multiple part sizes, a servo gripper adjusts its opening. No mechanical changeover.
Drop-off Accuracy
The robot picks the part from the vision-guided location, but where does it put it? The drop-off location (a fixture, a conveyor, a tray) needs to accept the part with whatever orientation the robot picked it up in. If the downstream fixture is precise (requires the part at ±0.1 mm), the vision-guided pick needs to be that accurate. If the drop-off is a tray or a conveyor, less accuracy is needed.
Lighting for Vision-Guided Picking
Same lighting rules as any vision system — but for bin picking, the challenge is lighting inside a dark tote.
Lighting the Bin
- Overhead diffuse light: A large diffuse light above the bin illuminates the parts evenly. Avoids hot spots on shiny parts.
- Dark field for edge detection: If the parts are shiny metal, dark field lighting at an angle makes edges pop against the dark bin.
- Colored light: If the parts are a different color than the bin, use a colored light that contrasts. Blue parts in a black bin under white light are easy to find. Gray parts in a gray bin aren’t.
Avoid Reflections
Shiny metal parts reflect the light source back into the camera, creating glare that confuses the vision software. Polarized filters on the camera and light reduce specular reflection. Diffuse lighting (dome or ring) spreads the light so it doesn’t reflect directly back.
Cycle Time: The Practical Limit
Vision-guided picking takes longer than conventional pick-and-place. The vision processing (finding the part, solving the pick position) adds time. A typical cycle:
- Camera captures image: 0.1–0.5 s (2D) or 0.5–2 s (3D point cloud).
- Vision software processes: 0.2–1 s (2D) or 1–5 s (3D object recognition).
- Robot moves to part: 0.5–1 s.
- Grip and lift: 0.5 s.
- Move to drop-off: 0.5–1 s.
- Release: 0.3 s.
Total: 2–5 seconds per pick for 2D, 4–10 seconds for 3D bin picking. If the cycle time target is 1 second per part, vision-guided picking isn’t fast enough. If it’s 5 seconds, it’s fine.
A Vision-Guided Picking Checklist
- 2D or 3D? (Based on part orientation and height variation.)
- Eye-in-hand or eye-to-hand? (Rigidity and workspace.)
- Is the camera bracket rigid? (No flex under robot motion.)
- Has hand-eye calibration been verified with actual picks?
- What is the expected pick accuracy? (Does it match the downstream fixture?)
- Is the gripper compliant enough for vision-guided variation?
- Does the lighting illuminate the bin without glare?
- What is the cycle time? (Does vision processing fit the cycle?)
- What happens if no part is found? (Alarm, retry, move to next bin?)
- Is there a known-good target for recalibration?
The Bottom Line
Robot part location vision isn’t just a camera and software. It’s a rigidly mounted camera, a hand-eye calibration that’s verified on real parts, a gripper that tolerates variation, and lighting that finds the part without glare. The robot that picks random parts from a bin isn’t the one with the most expensive camera — it’s the one whose camera doesn’t flex, whose calibration was verified, and whose gripper forgives the last 0.5 mm of error. That’s what turns a vision system that works on the bench into one that runs production.
