AIC Connector Insertion Private repo — code cannot be shared

The challenge is dexterous cable insertion: a UR5e arm in Gazebo picks up SFP and SC fiber-optic connectors and seats them in ports on a task board. It is a peg-in-hole problem that demands sub-centimeter precision under randomized start poses, impedance settings, and force disturbances.

The organizers run a fixed C++ evaluation stack, and participants submit a single Python policy that answers one ROS 2 action. Scoring rewards proximity, smoothness, and above all insertion depth, and punishes collisions.

We took a learned approach: collect demonstrations from a ground-truth oracle policy, convert them into a LeRobot dataset, and fine-tune SmolVLA, a small vision-language-action model, to imitate them. Over a documented experiment campaign the policy went from the provided ACT baseline's 38.2 to 110.3, close to three times the baseline.

Everything meets at one boundary. The evaluation side owns the physics: an engine that orchestrates and scores trials, an impedance controller, and a sensor-fusion adapter around a Gazebo simulation of the arm and task board.

Our side is a ROS 2 lifecycle node that dynamically loads a Policy and answers the /insert_cable action. It consumes camera images, wrench readings, and joint states, and streams back 6-DOF Cartesian twist commands.

The stack runs on ROS 2 Kilted over the Zenoh RMW, and the submission executes inside a resource-limited container with a 30-second model-discovery budget. Heavy imports can silently kill a run.

Demonstrations come from CheatCode, an oracle policy that reads ground-truth transforms the real submission never sees.

Stage one generates randomized trial configs covering start-pose jitter, impedance randomization, injected force disturbances, and scene clutter, so the dataset covers the perturbations evaluation will throw at us. Stage two rolls the oracle through Gazebo, recording rosbags and scores.

Stage three filters for successful episodes and converts them into a LeRobot dataset, finite-differencing the oracle's pose commands into the same 6-DOF twist action space the policy will emit. Stage four fine-tunes SmolVLA, sends it back through evaluation, and the loop repeats.

“Randomized” covers a lot, so here is the full inventory. Each form of variation is an independent, seeded block in the config generator, switchable per collection run:

• Board & rail placement — task-board pose shifted plus or minus 1 to 2 centimeters; the target rail slid and yawed plus or minus 0.03 to 0.12 radians along its channel.
• Scene clutter — each of the board's 13 rail slots probabilistically populated with distractor cards, ports, and fixtures, so the target always shares the view with other objects.
• Grasp jitter — the plug's pose in the gripper perturbed plus or minus 2 millimeters and plus or minus 0.04 radians, matching the documented evaluation envelope.
• Start pose — per-joint Gaussian offsets on the arm's home configuration (sigma of 5 degrees, clipped at plus or minus 10 degrees).
• Correlated action noise — DART-style temporally correlated noise on the oracle's commands during the approach phase (sigma of 2 millimeters and 0.005 radians, with a 0.5 second time constant), disabled near contact so demonstrations stay clean where precision matters. The expert's recoveries become training data.
• Force disturbances — one to three random shoves of about 5 newtons applied to the wrist mid-approach through the physics engine.
• Impedance randomization — controller stiffness scaled by 0.7 to 1.3 times and damping by 0.8 to 1.2 times per trial, so recovery behavior is learned under a range of contact dynamics.
• Language — every episode carries a minimal task-only prompt matched to its scene. Scene-rich prompts measurably hurt precision, and the evaluator prompts minimally anyway.

Fourteen dataset→train→eval loops in eleven days came from a deliberate working method: pair-engineering with an AI coding agent (Claude Code). The agent operated the experimental apparatus, covering simulator bring-up, batch collection runs, training launches, and checkpoint evaluations, and monitored the logs in real time.

I set the hypotheses, reviewed the same evidence (scoring files, force traces, evaluation videos), and made the calls. Every change got run, watched, and measured before it was written down.

One standing rule made the pairing work: any procedure worth doing twice was promoted into a deterministic, seeded, one-command tool. That covered config generation, the evaluation harness, checkpoint scans, and the recording rig behind the videos on this page.

Determinism also let the agent iterate at a cadence humans don't sustain. Starting, interrogating, and tearing down the simulator dozens of times an hour surfaced the startup races and stale-state quirks that slow manual cycles hide.

Those included zombie transport nodes surviving between runs, silently cached policy files, and a randomization block that turned out to be a silent no-op. Each diagnosis landed back in the tooling as a permanent guard.

Collection and training batches were resumable and ran unattended overnight, with the agent analyzing results as they arrived. Mornings began with a completed analysis and a decision about the next experiment, which kept roughly one full experimental loop per day sustainable on a single consumer GPU.

The final dataset was 203 episodes (53,611 frames): SFP demonstrations, a slice targeting the hardest port, and 20 partial-insertion SC demos admitted by the per-archetype filter. Every model was scanned across training checkpoints before judging; the best run peaked at 50k steps and measurably overtrained by 60k.

The negative results were kept on the books too. Default image augmentation regressed every checkpoint (too aggressive for a sub-centimeter task), and three attempts at hand-coded descent overrides all scored worse than the pure learned policy: the remaining gap is lateral precision, not descent timing.

Along the way, compressed-camera republishing cut per-trial bags from 7.4 gigabytes to 0.75 gigabytes, and a host/container launch fix took SC data collection from zero usable episodes to 18 in one sweep.

Tyler Johnson

Lead — trial randomization framework, C++ engine fixes, evaluation orchestration, and the fine-tuning experiment campaign from baseline to the 110.3 best

Abhijit Makhal

The four-stage data-collection pipeline (config generation, rollout orchestration, LeRobot conversion) and the host↔container integration debugging

Amrit Saini

SmolVLA and Pi0 inference policies and the model training workflow the fine-tuning campaign was built on