Project 01 — June–August 2026

5-DOF Autonomous Robotic Arm

A vision-guided pick system taken end-to-end: modeled in SolidWorks, imported into Simscape Multibody for inverse kinematics, driven by OpenCV-based object detection over a closed servo control loop — on real hardware, not just in simulation.

Success — ground pick: 100%
Success — off-height (8/10): 80%
Picks anywhere in the reachable volume: 3D
Degrees of freedom: 5

SolidWorksSimscape MultibodyMATLABOpenCV / PythonServo control

Results

The three clips below are sample points, not the limit of what the arm can reach — the two-pose system resolves a target anywhere inside the arm's reachable volume, at any height, without being told how high the object sits. Off-height picks depend on that solved depth rather than the ground case's known table plane, which is what accounts for the gap between 100% and 80%. Each trial below pairs the physical arm with the OpenCV detection view captured from the same run.

Ground-level pick

Playback: 2.5× speed

Physical arm (iPhone)

Medium-height pick

Playback: 2.5× speed

Physical arm (iPhone)

OpenCV detection + sim overlay

High pick

Playback: 2.5× speed

Physical arm (iPhone)

OpenCV detection + sim overlay

Descent moves are deliberately paced — slow jogs are a safety margin against an encoder-seam runaway, and the camera needs a near-stationary frame for OpenCV to resolve the brick without motion blur. Clips are sped up 2.5× for viewing.

What fires on every step of the descent

The descent isn't one command — it's a loop that re-measures, re-solves, and re-commands roughly every eight millimetres of travel. Here is the chain that runs each time.

Camera resolves the brick and shortens the error vector

OpenCV segments the brick by colour and scores its studs and contour geometry, returning a pixel centroid. The loop subtracts that from the aim point to get a 2-D pixel error vector — and then deliberately shortens it. Rather than commanding the full correction at once, it scales the vector down to a small step, because a single large lunge would overshoot: the camera rides on the wrist, so any move the arm makes changes the very measurement the correction was computed from. Shortening the vector keeps the loop convergent instead of oscillating — each step moves a fraction of the way, then re-measures against a fresh frame.

The pixel step is converted into a base-frame Cartesian target

A probe measures the loop's own gain empirically — jog a known distance, watch how many pixels the brick moved — giving a live mm-per-pixel scale for the current posture. That converts the shortened pixel vector into millimetres along the arm's real reach and tangential axes (measured by perturbing a pitch joint, not inferred from the CAD frame). The result is a target point in the base frame, one small step from where the claw is now.

Python hands the target to MATLAB, which solves inverse kinematics

The Cartesian target crosses a local TCP socket to a MATLAB server holding the arm's rigid-body tree. MATLAB runs inverseKinematics seeded from the arm's current joint angles, so it converges on the nearest solution rather than some other legal posture across the workspace. Because a 5-DOF arm has redundant freedom against a 3-DOF position target, the solve is constrained: wrist roll and (on radial moves) base yaw are held, and the returned angles are checked against measured per-joint travel limits and a Cartesian floor guard before anything moves.

Joint angles become paced servo commands

Angles are converted to raw encoder ticks through each joint's calibrated scale and direction sign, then issued over the serial bus in sub-cap hops with a rest between them. Position is read back after each hop rather than assumed, so a stalled or refused joint ends the move instead of being pushed through — and the pause is what keeps a wrong move stoppable by hand.

The two-pose system: why height doesn't have to be known

One camera looking at one frame cannot recover depth. A single image only gives a ray pointing from the lens toward the brick — every point along that ray projects to the same pixel. The usual workaround is to assume the object sits on a known plane (the tabletop) and intersect the ray with it. That works on the ground and fails the moment the brick is on top of something, because the claw then drives to where the table would have been.

The two-pose system removes that assumption. The arm captures the brick from one pose, swings the base tangentially to a second pose a known baseline apart, and captures again. Two rays from two known camera positions intersect at a single point in space — that intersection is the brick's true 3-D position, height included. The arm has effectively become its own stereo pair, using its own forward kinematics to know how far the two viewpoints are apart.

Two details make it work rather than merely sound right. The move has to be tangential, because parallax needs translation across the line of sight — and on this arm the base yaw is the only joint that provides it, since the three parallel pitch joints move the camera mostly along its own view. And the baseline has to be verified, not assumed: near-parallel rays amplify a pixel of noise into tens of millimetres of depth error, so the run reads the joint and the FK tip before and after each leg and refuses to triangulate if the arm didn't actually travel far enough.

The consequence: the pick target is solved rather than assumed, so the arm can grasp an object resting anywhere inside its reachable volume — on the table, on a book, on a raised block — without being told in advance how high it is.

Pipeline

Camera

OpenCV brick detection, 2 pose vectors

MATLAB / Simscape

Inverse kinematics over the surveyed chain

Serial bus

Paced, safety-capped servo commands

Physical arm

Closed-loop visual servo descent

Four things that broke, and what they taught me

The imported CAD model had the elbow backwards

The arm's IK solver ran fine in simulation, but a large move on hardware drove a joint into its limit. The instinct that caught it wasn't a debugger — it was a structural argument: J2 is the earliest joint in the chain, so it has to move the claw more than J3, not less. The imported model said otherwise. A jog and a ruler confirmed it: the model's elbow moment arm was inverted by more than 2.17×. Four independent statistical analyses had all pointed the wrong way; one physical jog settled it in thirty seconds.

Replaced the CAD kinematic chain with a ten-number ruler survey

Rather than patch the imported model, I rebuilt the arm's kinematics from five physically measured points — no curve-fitting, no CAD trust required. Held-out validation error dropped from 73.1 mm to 6.0 mm. It was ported into MATLAB behind a single flag and verified by independent replication before going live.

A hardware failure became a real safety system

A mid-move power cut once dropped the arm and drove a servo into overload. That turned into deliberate driver-level hardening: freeze-first e-stop (torque stays on, motion stops), live torque diagnostics, per-joint travel limits, and paced stepping so a wrong move is stoppable by hand before it does damage.

Closing the loop with data, not guesses

Hand-eye calibration never converged below roughly 80–100 mm of error against a 24 mm ruler measurement — characterized precisely rather than papered over. The closed-loop visual servo path sidesteps it entirely: eleven real descent runs were logged as training data, a height model was fit from apparent brick area (after the more obvious pixel-rate signal turned out to run backwards), and a two-view triangulation survey now serves as an independent, hand-eye-free referee.

What the rebuild bought

The single most consequential change in the project, measured against touch points the model was never fitted to.

Forward-kinematics accuracy — validated on held-out touch points

Imported CAD model73.1 mm

held-out error

Ruler-survey rebuild6.0 mm

held-out error, zero fitting

A 12× reduction in positioning error — from ten ruler measurements, with no curve-fitting and no reliance on the CAD geometry.

Simulation validation

Before any hardware was involved, the Simscape Multibody model (imported from the SolidWorks assembly) validated inverse kinematics against target poses in simulation — solved in MATLAB and checked against a mesh view of the arm before a single part was printed.

Simscape Multibody — early IK validation

Fabrication & build

With the kinematics confirmed, segments were sliced and printed on a Creality Ender-class printer, then assembled and wired against the SolidWorks reference on the bench.

Under the hood

Safety

Software freeze as the first-resort stop (torque held, motion halted) ahead of a power cut; per-joint travel limits measured directly rather than assumed; paced stepping with a rest between hops so a bad move can be caught by hand.

Closed-loop control

Null-space joints (wrist roll, base yaw during radial moves) are locked out of the solver's redundant freedom; a pan-budget formula lets tangential base-yaw moves spend the angle their geometry actually requires while still capping wasteful radial ones.

Frame handling

The Simscape import's base frame is upside-down relative to the physical robot — handled once, at the single MATLAB client seam, so the rest of the codebase speaks the physical frame without thinking about it.

View repository

Literature referenced

The hand-eye calibration and kinematic-identification work wasn't improvised — each diagnostic was built from an established result. What follows is what I read and what it changed in the code.

01
H. H. Chen, “A screw motion approach to uniqueness analysis of head-eye geometry,” CVPR, 1991.
Screw congruence. Because AX = XB makes A and B conjugate, every pose pair must agree on both rotation angle and screw pitch — but the tooling only ever tested angle. Adding the pitch test immediately caught four pose pairs that angle had passed cleanly, one of them with 0.06° of angle error hiding 115 mm of pitch error.
02
Y. Sun and J. M. Hollerbach, “Observability index selection for robot calibration,” IEEE ICRA, 2008.
Observability indices O1–O4. Replaced condition number as the capture-quality gate, because condition number rates a capture of uniformly tiny rotations a perfect 1.0 — it cannot tell a well-conditioned capture from one that barely moved. Captures are now gated on O3.
03
F. C. Park and B. J. Martin, “Robot sensor calibration: solving AX = XB on the Euclidean group,” IEEE Transactions on Robotics and Automation, 10(5), 1994.
Closed-form AX = XB solution via orthogonal-Procrustes reduction for rotation, then least squares for translation. Let me solve from explicit motion pairs instead of OpenCV's absolute-pose interface — which is what made cross-board pooling possible, taking the solve from 11 constraints to the ~390 the data actually offered.
04
V. V. Fedorov, Theory of Optimal Experiments, Academic Press, 1972 (Fedorov exchange algorithm).
D-optimal subset selection. Pose subsets are chosen by greedy seeding plus Fedorov exchange on log-determinant rather than on O3, since determinant criteria are the ones reported as correct under greedy selection. Measured effect: O3 rose 2.03 → 3.55 and conditioning went from failing to passing.
05
H. Zhuang, S. H. Motaghedi, and Z. S. Roth, “Robot calibration with planar constraints,” IEEE ICRA, 1999.
Planar-constraint identifiability. A single plane leaves the identification matrix rank deficient — three mutually non-parallel planes are the minimum for identification. This is why the table-plane tool was deliberately scoped to refute a hypothesis rather than claim to identify parameters from one flat surface.
06
R. Y. Tsai and R. K. Lenz, “A new technique for fully autonomous and efficient 3D robotics hand/eye calibration,” IEEE Transactions on Robotics and Automation, 5(3), 1989.
The other hand-eye method available in OpenCV. Solving with both TSAI and PARK and cross-checking them is what exposed that the original agreement gate compared only translation — so two methods could agree on the same badly-wrong orientation and still report “ok.”

Built with Krish Vatsa, who built and validated the OpenCV detection pipeline in Python. I led the SolidWorks/Simscape kinematics, MATLAB IK, and hardware bring-up; calibration, fabrication, and closed-loop control were done together. Integrating the three systems was the hardest and longest part of the project.