Measured-joint mirroring
Pair the lab with the local companion to mirror measured joints in MuJoCo.
ERH Physical AI Hackathon · 17–18 October 2026
Dual-arm manipulation
with FLUX 3 Action.
Task LoRAs control each arm. A behavior tree coordinates the task.
Liminal runs on Haller, a dual SO-101 rig, with camera feedback and a MuJoCo joint mirror.
Open on your phoneScan for this page
Two SO-101 arms / FLUX 3 Action + task LoRA / MuJoCo
Recorded MuJoCo preview
MuJoCo scene preview
Haller in MuJoCoLoad 3D to rotate the scene
Real setup and estimated tabletop reconstructionRecorded simulation · no robot connection needed
The idea
One arm holds.
The other inserts.
The placement arm needs the lid to stay open. Losing support must stop the insertion.
One arm opens and holds the lid while the other places an object inside.
The behavior tree schedules skills, monitors support and waits for camera-based verification before advancing.
AFTERBegin when a prerequisite has finished.
WHILEContinue only while an invariant holds.
UNTILKeep working until a verified condition is true.
Full learned dual-arm execution is still unverified.
Real robot
Red-pen insertion
on Haller.
Five recorded FLUX LoRA runs. Head and wrist cameras are shown side by side.
RAW 5k adapter · layout C · real head and wrist camera recordings.
A few centimeters can change the policy's view Compare wrist observations ↗
Training and runtime wrist views after a small station offset. This records a setup mismatch; its effect on success was not measured.
The architecture
How the system works.
Each arm has its own six-channel FLUX instance, observation history and task checkpoint. Select a component below.
Operator goal “Put the pens in the open box.”
Haller / dual SO-101
Measured joints + scene / wrist cameras
Behavior-tree harness
The tree schedules skills, checks support conditions and waits for independent verification.
View the complete architecture figure Expand ↗
System diagram from the README; the hardware panel is an adapted photograph.
Task dependencies
Open. Hold. Insert.
Move the slider through the task. Toggle loss of support to see placement cancel.
Illustrative behavior-tree execution
Helper arm / rightOpen the lid
Task arm / leftWait for access
lid_open: pendingobject_inserted: pendingAFTER: establish access
The helper arm changes the scene. The placement skill waits for a verified open-lid condition.
Runtime illustration, not a physical trial. The showcase sequences a green pen, red pen and cutter.
Demonstrations
Simulation and diagnostics.
Inspect a scripted teacher, compare policy checkpoints and view predicted camera frames.
Scripted simulation expertGreen marker
The same recorded poses
Recorded MuJoCo preview
0.0 / 14.0 s
Scripted training demonstration: left arm parked, lid fixed open.
Zero-shot vs. task LoRA.
Red marker, seed 901001. These videos belong to the separate tabletop-twin experiment.
Zero-shot FLUX
Final gripper-to-marker distance: 20.6 cm
Final task LoRA
Final gripper-to-marker distance: 2.2 cm
Scrub the same relative point
Both checkpoints completed 0/20 placements. The shorter distance in this scene does not establish a successful grasp.
Runs and predictions.
Real recorded input / visual prediction
What the policy imagines next.
On held-out red-pen take 101, the RAW-5k LoRA’s imagined video appears beside recorded camera frames. A predicted future is not a measured outcome.
1 / 4
Results
Simulation and hardware results.
Recorded results, separated by task and experiment.
Red block → blue container
Five identical held-out seeds for each model. Lift threshold: 3.5 cm.
64 + 16 train / validation episodes32,220 frames at 20 Hz250 optimizer updates
Raw LoRA: 2/5 lifts. Zero-shot and EMA: 0/5. All models: 0/5 placements.
Red-pen insertion sessions
The media index reports 3/5 and 2/3 successful attempts in two RAW-5k sessions. Watch the selected runs ↗
Earlier official-policy engineering index
16Official-policy trials
11Execution requests
9Trials with commands
UnlabeledIndependent task success
Two trials reached their duration. Others stopped on torque, travel-limit or tracking checks.
These older trials have no task-success labels. They are separate from the red-pen sessions above.
Tabletop-twin dataset
576Generated candidates
243Accepted demonstrations
133 / 110Green / red marker
0 / 20Placements per evaluated checkpoint
Scripted demonstrations vary object pose, box layout, lighting and cameras.
Separate dataset from the five-seed red-block experiment.
Sources: current main validation records and media index · Download the public result summary ↗
The digital twin
Real workspace and MuJoCo.
Drag to compare the real setup with the estimated scene. The views are not registered.
Estimated physical correspondence
Scene dimensions and camera registration need calibration. Live objects and contact forces are not tracked.
Why the fingers matter Expand comparison ↗
Haller uses compliant fingers; the simulated jaws are rigid.
Evidence & limits
Current status.
Code, recordings and remaining validation.
Runtime and skill coordination Implemented / software checked
The runtime is implemented. Six object-order permutations pass symbolic checks using synthetic checkpoints and deterministic feedback.
Perception and verification Configured / deployment needs evaluation
Qwen inspects camera views and updates timestamped scene predicates. Physical task transitions still need a labeled perception evaluation.
Task LoRA and the red-pen skill RAW-5k inserts / 10k loading check
RAW-5k has recorded real inserts. A separate 10k adapter passed a blank-image loading check with a finite 42×6 output. Other showcase skill bundles remain unstaged.
Simulation-to-real transfer Research in progress
Single-arm red-pen inserts are recorded. Full learned dual-arm execution and a calibrated object/contact twin remain unverified.
Reproducibility and attribution
This page is based on main revision 07e50d0, reviewed
on 18 October 2026. Public results are an explicit aggregate export
of recorded validation files, not a newly reproduced benchmark.
Hardware/controller origins, serial identifiers, credentials and raw
telemetry are excluded.
Result counts and source-file provenance · Read the evidence notes and local bridge setup
Layout references: DexMimic and Nerfies. Liminal's copy, interactions and layouts are original. Hardware CAD is from TheRobotStudio; Three.js and self-hosted fonts retain their licenses.