Deployed-site workflow¶
matcher.oneshot() corrects small drift on its own. This page covers what to do when a deployed
rig drifts further: keep the frames the matcher fails on, have a person review them, and change
the stored camera-LiDAR matrix only when pairs the fit never saw support the change.
You use four pieces, in this order:
| Step | Problem it solves | Entry point |
|---|---|---|
| Capture | A stream integration has no annotation tool. When matching fails you need the exact inputs on disk, in a format a person can open. | SceneRecorder, matcher.oneshot(K, recorder=...) |
| Review | Someone has to say which camera box belongs to which LiDAR box. | xcalib-review |
| Nudge | After a small shift, a fresh PnP solve from a few buffered pairs is not guaranteed to beat the matrix you already have. | refine_projection, session.calibrate(refine=True) |
| Gate | The nudge is judged on the pairs it was fitted to. A matrix you deploy needs evidence from pairs the fit never saw, and a cap on how far it moves. | build_physical_refinement_candidate |
SceneRecorder, refine_projection and the five gate names (CalibrationGateConfig,
CalibrationMetrics, PhysicalCalibrationCandidate, build_physical_refinement_candidate,
evaluate_calibration) are in xcalib.__all__. should_record, ObserveReport,
CalibrationSession and everything under xcalib.gui are not, so they can change without notice.
The workflow¶
- Start from a matrix.
matcher.calibrate(...),session.calibrate()or a surveyed extrinsic gives aCalibrationResult. Intrinsics stay fixed throughout; only[R|t]moves. - Record. Attach a
SceneRecorderto the session. Frames that have detections on both sides but no confident pair are written to disk. - Review. Open the capture directory in
xcalib-review, fix the pairs, press Apply. Each accepted pair writes the LiDAR id onto the camera box. - Collect pairs. Turn the reviewed boxes into (3D center, 2D center) pairs. Fit on earlier frames; keep other frames out of the fit and use them for validation.
- Gate.
build_physical_refinement_candidatefits a candidate on the fit set and checks it on the validation sets. XCalib saves nothing: ifeligibleis true, you save the candidate.
There is no xcalib subcommand for any of this. The only command-line entry point is
xcalib-review.
A synthetic site¶
The examples need no data. A virtual camera with known intrinsics sees boxes through a true pose,
and the site's stored matrix is that pose after the rig moved 1 degree and 0.18 m. Run the blocks
in order in one Python session. The blocks that call Matcher.from_pretrained need the released
weights (the first call downloads them); the rest runs offline.
import json
import shutil
from pathlib import Path
import cv2
import numpy as np
from xcalib import CalibrationResult, CameraIntrinsics
from xcalib.oneshot.calibration import project_points
K = CameraIntrinsics(fx=800.0, fy=800.0, cx=640.0, cy=360.0) # pixels, fixed and known
# True LiDAR -> camera pose. LiDAR: x forward, y left, z up. Camera: x right, y down, z forward.
R_true = np.array([[0.0, -1.0, 0.0], [0.0, 0.0, -1.0], [1.0, 0.0, 0.0]])
t_true = np.array([0.2, -0.3, 0.5]) # meters
true_extrinsics = np.hstack([R_true, t_true.reshape(3, 1)])
def make_calibration(R, t):
return CalibrationResult(success=True, intrinsics=K, rotation=R, translation=t,
projection=K.K @ np.hstack([R, t.reshape(3, 1)]))
def shifted_pose(deg, shift_m):
"""The true pose after the rig rotated by `deg` degrees and moved by `shift_m` meters."""
axis = np.array([0.3, -0.5, 0.8])
rot, _ = cv2.Rodrigues(np.deg2rad(deg) * axis / np.linalg.norm(axis))
return rot @ R_true, t_true + np.asarray(shift_m)
# The matrix the site still uses: the true pose after a 1 degree / 0.18 m shift.
stored = make_calibration(*shifted_pose(1.0, (0.15, -0.10, 0.05)))
exact = make_calibration(R_true, t_true)
def site_pairs(rng, n, noise_px=1.5, offset_px=(0.0, 0.0), pose=None):
"""n pairs (3D center [m], 2D center [px]) as seen through `pose` (default: the true pose)."""
R, t = pose or (R_true, t_true)
points_3d = np.column_stack([rng.uniform(8, 45, n), rng.uniform(-6, 6, n),
rng.uniform(-1, 2, n)])
uv, _ = project_points(points_3d, K, R, t)
return points_3d, uv + rng.normal(0, noise_px, uv.shape) + np.asarray(offset_px)
def synth_frame(rng, n=6):
"""One frame: image, cloud [P, 4] (x y z intensity), 2D boxes, 3D boxes, pair_of."""
centers = []
while len(centers) < n: # keep boxes that land inside the image
c = rng.uniform([8.0, -6.0, -1.0], [45.0, 6.0, 2.0])
(u, v), = project_points(c[None], K, R_true, t_true)[0]
if 80 <= u <= 1200 and 80 <= v <= 640:
centers.append(c)
centers = np.array(centers)
uv, depth = project_points(centers, K, R_true, t_true)
uv = uv + rng.normal(0, 1.5, uv.shape) # detector jitter, pixels
half = np.clip(1100.0 / depth, 12, 80)[:, None]
boxes_2d = np.hstack([uv - half, uv + half]).astype(np.float32) # x1, y1, x2, y2
extents = np.hstack([centers - [1.75, 0.9, 0.8], centers + [1.75, 0.9, 0.8]])
order = rng.permutation(n) # LiDAR detections arrive in another order
boxes_3d = extents[order].astype(np.float32) # xmin, ymin, zmin, xmax, ymax, zmax
pair_of = np.argsort(order) # camera box i belongs to LiDAR box pair_of[i]
ground = rng.uniform([-20, -20, -2], [50, 20, 0], (800, 3))
inside = [rng.uniform(b[:3], b[3:], (60, 3)) for b in boxes_3d]
xyz = np.vstack([ground, *inside])
cloud = np.hstack([xyz, rng.uniform(0, 255, (len(xyz), 1))]).astype(np.float32)
image = np.zeros((720, 1280, 3), np.uint8)
image[:, :, 0] = np.linspace(0, 255, 1280).astype(np.uint8)
return image, cloud, boxes_2d, boxes_3d, pair_of
1. Capture the frames the matcher is lost on¶
SceneRecorder writes what you passed to match() or observe() as a SUSTechPOINTS-format
scene directory. xcalib-review reads that layout directly.
Save a frame by hand¶
from xcalib import SceneRecorder
rng = np.random.default_rng(1)
rec = SceneRecorder("capture/site1", camera="cam_front") # cloud_format="pcd" (default) or "npy"
rec.save_calibration(stored) # once per site: K and [R|t]
image, cloud, boxes_2d, boxes_3d, pair_of = synth_frame(rng)
frame = rec.save_frame(
image, cloud, boxes_2d, boxes_3d,
ids_3d=[100 + j for j in range(len(boxes_3d))], # LiDAR box obj_id
types_3d=["Car"] * len(boxes_3d),
ids_2d=[10 + i for i in range(len(boxes_2d))], # the camera tracker's own ids
types_2d=["Car"] * len(boxes_2d),
)
print(frame, rec.frames())
000000 ['000000']
SceneRecorder(root, camera="camera_0", cloud_format="pcd") takes the scene directory, the
camera name (a folder under camera/) and the point-cloud format. Any other cloud_format
raises ValueError.
save_frame argument |
Type | Notes |
|---|---|---|
image |
[H, W, 3] uint8, RGB |
The array you pass to match(). Written as JPEG, which is lossy. |
point_cloud |
[P, >=3] float, meters |
Columns 0 to 3 are kept. Column 3 is stored as intensity: pass point_cloud[:, :3] if your fourth column is something else, such as a ring index. |
bboxes_2d |
[K, 4] |
(x1, y1, x2, y2) in pixels. |
bboxes_3d |
[M, 6] |
Extents or center plus size, detected per row as in the input protocol. |
frame |
str, optional |
Frame id. The default is a six-digit counter that continues from the highest numeric id already on disk. An id that already exists is overwritten. |
ids_3d, types_3d |
length M, optional |
LiDAR box obj_id and obj_type. The default ids are 1..M: positions within this call, not tracker ids. |
ids_2d, types_2d |
length K, optional |
ids_2d are your camera tracker's ids and land in obj_attr as t<id>. |
save_frame returns the frame id. rec.frames() lists the ids in the lidar/ folder.
What is on disk¶
root = Path("capture/site1")
for path in sorted(p for p in root.rglob("*") if p.is_file()):
print(path.relative_to(root).as_posix())
boxes = json.loads((root / "label/000000.json").read_text())["objs"]
rects = json.loads((root / "label_fusion/camera/cam_front/000000.json").read_text())["objs"]
b, r = boxes[0], rects[0]
print("3D box:", b["obj_id"], b["obj_type"], {k: round(v, 2) for k, v in b["psr"]["position"].items()})
print("2D rect:", repr(r["obj_id"]), r["obj_attr"], r["annotator"], {k: round(v) for k, v in r["rect"].items()})
calib/camera/cam_front.json
camera/cam_front/000000.jpg
label/000000.json
label_fusion/camera/cam_front/000000.json
lidar/000000.pcd
3D box: 100 Car {'x': 9.02, 'y': 3.04, 'z': 0.61}
2D rect: '' t10 xcalib {'x1': 447, 'y1': 327, 'x2': 527, 'y2': 408}
Camera boxes are written with an empty obj_id and annotator: "xcalib". A camera box gets a
LiDAR id only when a person accepts a pair in review, so a captured frame never contains a model
guess that looks like a verified pair.
Intensity¶
A fourth point-cloud column is kept as intensity: a binary PCD with FIELDS x y z intensity, or
a four-column .npy. The matcher reads only x, y and z. The review window shades its top-down
view by intensity. A three-column cloud stays three columns; no intensity is invented.
pcd = root / "lidar/000000.pcd"
header = b""
with open(pcd, "rb") as f:
while not header.endswith(b"DATA binary\n"):
header += f.readline()
print(header.decode().splitlines()[2])
points = np.fromfile(pcd, dtype=np.float32, offset=len(header)).reshape(-1, 4)
print(points.shape, "identical to the input:", np.array_equal(points, cloud))
saved = cv2.cvtColor(cv2.imread(str(root / "camera/cam_front/000000.jpg")), cv2.COLOR_BGR2RGB)
print("largest pixel change from JPEG:", int(np.abs(saved.astype(int) - image).max()))
FIELDS x y z intensity
(1160, 4) identical to the input: True
largest pixel change from JPEG: 2
Capture automatically¶
Pass the recorder to matcher.oneshot(). session.observe() then calls save_frame for every
frame where should_record(report) is true:
n_detections_2d > 0 and n_detections_3d > 0 and n_confident_matches == 0
n_confident_matches counts the one-to-one pairs whose score reaches match_threshold (default
0.7). Score ranges differ per model, so tune match_threshold on site data as the
protocol advises.
from xcalib import Matcher
matcher = Matcher.from_pretrained("crlite", site="a9_dataset_r02_s01") # first call downloads weights
rec = SceneRecorder("capture/site1", camera="cam_front")
session = matcher.oneshot(K, recorder=rec) # record_lost_frames=True by default
session.set_calibration(stored) # or session.calibrate() once enough pairs are buffered
for image, point_cloud, boxes_2d, boxes_3d in stream:
session.observe(image, point_cloud, boxes_2d, boxes_3d)
To check the wiring without a drifted rig, set match_threshold above every model's score range.
Every frame with detections on both sides is then "lost":
from xcalib import Matcher
from xcalib.capture import should_record
from xcalib.oneshot import ObserveReport
matcher = Matcher.from_pretrained("crlite", site="a9_dataset_r02_s01") # first call downloads weights
rec = SceneRecorder("capture/wiring_check", camera="cam_front")
session = matcher.oneshot(K, match_threshold=2.0, recorder=rec)
for seed in range(3):
image, cloud, boxes_2d, boxes_3d, _ = synth_frame(np.random.default_rng(100 + seed))
report = session.observe(image, cloud, boxes_2d, boxes_3d)
print(report.n_detections_2d, report.n_detections_3d, report.n_confident_matches)
print(rec.frames())
# record_lost_frames=False keeps the recorder attached but turns the trigger off.
manual = SceneRecorder("capture/manual_only", camera="cam_front")
session = matcher.oneshot(K, match_threshold=2.0, recorder=manual, record_lost_frames=False)
session.observe(image, cloud, boxes_2d, boxes_3d)
print(manual.frames())
def report_with(n_confident, n_2d=3, n_3d=4):
return ObserveReport(frame_id=0, n_detections_2d=n_2d, n_detections_3d=n_3d,
n_confident_matches=n_confident)
print(should_record(report_with(0)), should_record(report_with(2)), should_record(report_with(0, n_2d=0)))
6 6 0
['000000', '000001', '000002']
[]
True False False
The trigger is a heuristic:
- "Lost" means the matcher found no confident pair. The trigger does not look at the projection matrix. A frame the matcher is confidently wrong about is not recorded, and a frame whose camera boxes have no LiDAR counterpart (an object outside LiDAR range) is.
- If you have a better alarm, keep
record_lost_frames=Falseand callsave_frameyourself. - A failure inside
save_frameis logged as a warning. It does not stopobserve(). - The session passes no ids and does not save a calibration. Automatically captured frames get
positional LiDAR ids
1..Mand no tracker ids, and the scene has nocalib/camera/file until you callrec.save_calibration(...). Callsave_frameyourself when you have stable ids.
What capture does not do¶
- Limit what it writes. Every lost frame is saved. A rig that stays lost writes every frame
until the disk is full. Apply your own policy with
record_lost_frames=Falseandsave_frame. - Store box rotation. Boxes are axis-aligned, as in the matcher input.
psr.rotationis always zero. - Preserve pixels exactly. Images are re-encoded as JPEG (the check above shows the largest change on a smooth test image).
- Store lens distortion.
save_calibrationwritesKand[R|t]only. Reading it back gives intrinsics without distortion. Capture rectified images if your lens is distorted. - Check what you save.
save_calibrationaccepts any successfulCalibrationResult, including a gate candidate that was rejected, and overwrites the existing file. Keep a copy of the old one.
2. Review the captured frames¶
pip install "XCalib[gui]"
xcalib-review --scene capture/site1 --camera cam_front
The window loads a matcher from <scene>/calib/xcalib/<camera>_model/. SceneRecorder does not
create that folder, and without it the window reports no trained model. Save the matcher you
recorded with:
matcher.save_pretrained("capture/site1/calib/xcalib/cam_front_model")
The window shows the image beside a top-down LiDAR view, and one card per camera box with a ranked
list of LiDAR candidates. Each candidate shows its score. When calib/camera/<camera>.json
exists, which save_calibration writes, it also shows the projected pixel distance, and a
candidate that geometry rules out carries a warning sign and is never picked automatically. That
geometric check is deliberately loose: by default a pair is compatible when the projected center
lands in front of the camera within max(150 px, 3 x box diagonal) of the camera box, so a drifted
matrix still helps. If every candidate for a camera box fails the check, none of them is flagged.
The window pre-fills each camera box with the matcher's suggestion and counts it as a pending decision. On a captured frame those suggestions are the guesses that just failed. Check every card before you press Apply.
Apply this frame and Apply all write the pending decisions into
label_fusion/camera/<camera>/<frame>.json: an accepted pair sets the camera box obj_id to the
LiDAR id, and a rejection sets no_match: true.
The two refine buttons train the matcher (see Continual refine). Neither
changes the camera-LiDAR matrix: the review window does not call refine_projection or the gate.
3. Nudge the matrix in session¶
refine_projection(previous, points_3d, points_2d, *, ransac_reproj_px=8.0, min_points=6) starts
Levenberg-Marquardt at the stored pose and returns a CalibrationResult whose accepted says
whether the result beat it.
| Parameter | Default | Unit | Meaning |
|---|---|---|---|
previous |
required | A successful CalibrationResult. A failed one raises ValueError. |
|
points_3d |
required | meters, [N, 3] |
Box centers in the frame of bboxes_3d. |
points_2d |
required | pixels, [N, 2] |
The matching camera box centers. |
ransac_reproj_px |
8.0 |
pixels | The solver is fitted only on pairs the stored pose already puts within 3 times this value (24 px). After a full run, n_inliers and reproj_error_px on the result count pairs below this value itself. |
min_points |
6 |
pairs | With fewer pairs, or fewer pairs inside the 24 px window, the stored pose is returned unchanged. |
The decision rule: the refined pose is adopted only if its median reprojection error over all
supplied pairs is strictly lower than the stored pose's. Pairs behind the camera count as
infinitely wrong. Otherwise the stored pose is returned with accepted=False.
buffer_reproj_px holds the median error of the returned pose.
from xcalib import refine_projection
pts3, pts2 = site_pairs(np.random.default_rng(2), 40)
out = refine_projection(stored, pts3, pts2)
before, after = stored.pose_error(true_extrinsics), out.pose_error(true_extrinsics)
print(out.accepted, "|", out.message)
print(f"pose error vs truth: {before[0]:.2f} deg, {before[1]:.3f} m -> {after[0]:.2f} deg, {after[1]:.3f} m")
few = refine_projection(stored, pts3[:3], pts2[:3])
print(few.accepted, "|", few.message)
far = make_calibration(*shifted_pose(25.0, (3.0, 2.0, 1.0)))
hopeless = refine_projection(far, pts3, pts2)
print(hopeless.accepted, "|", hopeless.message)
True | refined: median 7.9 -> 1.7 px over 40 pair(s)
pose error vs truth: 1.00 deg, 0.187 m -> 0.04 deg, 0.069 m
False | only 3 pair(s) (< 6) — kept the previous calibration
False | the previous pose explains only 0 of 40 pair(s) — too far gone to nudge; re-solve (estimate_projection) instead
CalibrationSession.refine(previous, min_pairs=6) and session.calibrate(refine=True) call the
same function on the pairs a session has buffered:
session = matcher.oneshot(K)
session.set_calibration(stored)
session.calib_session.add_correspondences(pts2, pts3) # session.observe() normally fills this
result = session.calibrate(refine=True) # min_pairs=12 by default
print(result.accepted, session.calibration is result)
True True
calibrate(refine=True) needs an existing calibration and raises ValueError without one. It
ignores disambiguate, gate_factor and min_frames. An accepted refinement also updates
session.best_reproj_px when its median is lower. A later fresh session.calibrate() is gated
against that value. Use session.calibrate() without refine, or estimate_projection, when you
have no matrix you trust.
What this guarantees, and what it does not. On the pairs you pass, the returned pose never has a larger median error than the one you gave. It says nothing about other pairs. The criterion is "strictly lower", with no minimum margin, so noise alone can make it accept. With the stored pose already exact:
accepted, rot, trans = 0, [], []
for seed in range(200):
pts3, pts2 = site_pairs(np.random.default_rng(seed), 40)
out = refine_projection(exact, pts3, pts2)
if out.accepted:
accepted += 1
r, t = out.pose_error(true_extrinsics)
rot.append(r)
trans.append(t)
print(f"accepted {accepted}/200; error of the accepted poses: "
f"mean {np.mean(rot):.3f} deg / {np.mean(trans):.3f} m, "
f"max {np.max(rot):.3f} deg / {np.max(trans):.3f} m")
accepted 148/200; error of the accepted poses: mean 0.102 deg / 0.030 m, max 0.327 deg / 0.117 m
accepted means "lower on these pairs", not "closer to the truth". For a matrix you will deploy,
use the gate in the next section.
4. Gate a candidate on held-out data¶
build_physical_refinement_candidate(previous, fit_points_3d, fit_points_2d, validation_sets, *,
config=None) fits a candidate pose on one set of pairs and accepts it only from pairs you supply
separately. It never writes a file.
- The candidate comes from
cv2.solvePnPRefineLM, started atprevious, on the fit pairs the stored pose puts in front of the camera and within3 x inlier_px(75 px by default). Intrinsics, including distortion, are never changed. With fewer thanmin_fit_pointssuch pairs the solver is skipped and the candidate equalsprevious. validation_setsmaps a name to a(points_3d, points_2d)tuple. The order is kept for display. The set namedprimary_validation_set(default"held_out") must improve. Every set you pass must also pass the guardrails. Do not name a set"fit": that key holds the fit metrics.- Point units follow Refine: meters in the frame of
bboxes_3d, and pixels.
Gate rules¶
Every rule must hold for eligible to be true. CalibrationGateConfig is a frozen dataclass;
build one with the fields you want to change and pass it as config=.
| Field | Default | Unit | Rule |
|---|---|---|---|
inlier_px |
25.0 |
pixels | A pair is an inlier below this error. Also sets the 3x window used to pick fit pairs. |
min_fit_points |
12 |
pairs | The fit set needs this many pairs, and the stored pose must put this many within 3 x inlier_px. |
min_validation_points |
6 |
pairs | Each validation set needs this many pairs. Sets with fewer are rejected and skip their other checks. |
min_median_improvement_px |
0.5 |
pixels | Primary set: the median error must drop by at least max(this, ratio x baseline median). |
min_median_improvement_ratio |
0.02 |
fraction | See above. It only matters when the baseline median exceeds 25 px (0.5 / 0.02). |
max_q90_regression_px |
2.0 |
pixels | Every set: the 90th-percentile error may rise by at most this much. |
max_inlier_ratio_drop |
0.02 |
fraction | Every set: the share of inliers may fall by at most this much. |
min_positive_depth_ratio |
0.80 |
fraction | Every set: at least this share of pairs must project in front of the camera. This is an absolute floor, not a comparison with the baseline. |
max_depth_ratio_drop |
0.01 |
fraction | Every set: the in-front share may fall by at most this much. |
max_rotation_delta_deg |
3.0 |
degrees | The candidate may differ from previous by at most this rotation angle. |
max_translation_delta_m |
1.0 |
meters | And by at most this norm of the change in t. |
primary_validation_set |
"held_out" |
The name of the set that must improve. A missing name rejects the candidate. |
The result is a PhysicalCalibrationCandidate:
| Field | Meaning |
|---|---|
eligible |
True only if no rule produced a reason. |
reasons |
One string per failed rule. Empty when eligible. |
candidate |
A CalibrationResult. candidate.accepted equals eligible. candidate.message is eligible physical refinement candidate, or the reasons joined with ;. |
previous |
The pose you passed in. |
before, after |
dict[str, CalibrationMetrics] for "fit" and each validation set, under previous and under candidate. |
rotation_delta_deg, translation_delta_m |
How far the candidate moved. |
CalibrationMetrics fields: pairs; median_px and q90_px (over pairs in front of the camera);
rmse_inlier_px and inliers (pairs below inlier_px); inlier_ratio (inliers over all
pairs); positive_depth and positive_depth_ratio (pairs in front of the camera). You can score
any pose yourself with evaluate_calibration(calibration, points_3d, points_2d, inlier_px=25.0).
An empty set returns infinite errors and zero counts.
The function raises ValueError when previous is not a successful calibration, when the 3D and
2D arrays differ in length, or when any value is not finite. Everything else, including too few
pairs, comes back as eligible=False with a reason.
An accepted candidate¶
from xcalib import CalibrationGateConfig, build_physical_refinement_candidate, evaluate_calibration
rng = np.random.default_rng(3)
fit = site_pairs(rng, 80)
validation = {"held_out": site_pairs(rng, 30), "later_window": site_pairs(rng, 30)}
result = build_physical_refinement_candidate(stored, *fit, validation)
print("eligible:", result.eligible, "| reasons:", result.reasons)
print(f"pose change: {result.rotation_delta_deg:.2f} deg, {result.translation_delta_m:.3f} m")
print(f"{'set':<13}{'pairs':>5} {'median px':>14} {'q90 px':>14} {'inlier ratio':>14}")
for name, b in result.before.items():
a = result.after[name]
print(f"{name:<13}{b.pairs:>5} {b.median_px:>6.2f} ->{a.median_px:>5.2f} "
f"{b.q90_px:>6.2f} ->{a.q90_px:>5.2f} {b.inlier_ratio:>6.2f} ->{a.inlier_ratio:>5.2f}")
print(result.candidate.message)
# evaluate_calibration scores any pose on any pairs, with the same metrics.
held_out_metrics = evaluate_calibration(result.candidate, *validation["held_out"])
print("held-out median under the candidate:", round(held_out_metrics.median_px, 2), "px")
eligible: True | reasons: ()
pose change: 1.02 deg, 0.171 m
set pairs median px q90 px inlier ratio
fit 80 8.09 -> 1.78 10.57 -> 3.09 1.00 -> 1.00
held_out 30 8.60 -> 1.60 13.39 -> 2.97 1.00 -> 1.00
later_window 30 8.17 -> 1.82 11.55 -> 3.26 1.00 -> 1.00
eligible physical refinement candidate
held-out median under the candidate: 1.6 px
Rejected candidates¶
Each case below changes one thing and prints the reasons:
rng = np.random.default_rng(4)
fit = site_pairs(rng, 80)
held_out = site_pairs(rng, 30)
stored_pose = (stored.rotation, stored.translation)
behind = held_out[0].copy()
behind[:9, 0] *= -1 # 9 of 30 held-out points now sit behind the camera
caps = CalibrationGateConfig(max_rotation_delta_deg=0.5, max_translation_delta_m=0.05)
nearly_right = make_calibration(*shifted_pose(0.05, (0.005, 0.0, 0.0)))
cases = {
"later window agrees with the old matrix": (stored, fit, {
"held_out": held_out,
"later_window": site_pairs(rng, 30, noise_px=0.05, pose=stored_pose)}, None),
"no validation set": (stored, fit, {}, None),
"only 5 held-out pairs": (stored, fit, {"held_out": site_pairs(rng, 5)}, None),
"only 8 fit pairs": (stored, site_pairs(rng, 8), {"held_out": held_out}, None),
"30% of held-out pairs behind the camera": (stored, fit, {"held_out": (behind, held_out[1])}, None),
"pose caps of 0.5 deg and 0.05 m": (stored, fit, {"held_out": held_out}, caps),
"stored matrix already right to 0.05 deg": (nearly_right, fit, {"held_out": held_out}, None),
}
for name, (previous, fit_set, validation_sets, config) in cases.items():
out = build_physical_refinement_candidate(previous, *fit_set, validation_sets, config=config)
print(f"{name}: eligible={out.eligible}")
for reason in out.reasons:
print(" -", reason)
later window agrees with the old matrix: eligible=False
- later_window q90 regresses 10.8px
no validation set: eligible=False
- missing primary validation set 'held_out'
only 5 held-out pairs: eligible=False
- held_out has 5 pair(s); need 6
only 8 fit pairs: eligible=False
- fit has 8 pair(s); need 12
- baseline explains only 8 fit pair(s); need 12 for a nudge
- held_out median improves -4.44e-14px; need 0.5px
30% of held-out pairs behind the camera: eligible=False
- held_out positive-depth ratio 0.700 is below 0.800
pose caps of 0.5 deg and 0.05 m: eligible=False
- rotation delta 0.853deg exceeds 0.500deg
- translation delta 0.200m exceeds 0.050m
stored matrix already right to 0.05 deg: eligible=False
- held_out median improves -0.178px; need 0.5px
How to read the reasons:
| Reason | Cause and fix |
|---|---|
fit has N pair(s); need 12 |
Too few fit pairs. Collect more reviewed frames. |
baseline explains only N fit pair(s); need 12 for a nudge |
Under the stored pose, too few fit pairs land in front of the camera within 75 px (3 x inlier_px). Either there are few pairs or the pose is far off. The solver is skipped and the candidate equals the stored pose, so the primary set usually also reports an improvement of about zero (floating-point noise). If the pose is far off, re-solve with estimate_projection. |
<set> has N pair(s); need 6 |
The set is too small to judge. The set is otherwise unchecked. |
<set> candidate projection is non-finite |
Under the candidate, no pair in that set projects in front of the camera. Check the points and their frame. |
<set> median improves Xpx; need Ypx |
The primary set did not improve enough. A negative X means the candidate is worse. |
<set> q90 regresses Xpx |
The tail got worse, even if the median improved. Often a sign the fit pairs and this set disagree. |
<set> inlier ratio drops X |
Fewer pairs fall inside inlier_px than under the stored pose. |
<set> positive-depth ratio X is below 0.800 |
Too many pairs project behind the camera. This floor does not compare with the baseline, so filter such pairs out of the set first. |
<set> positive-depth ratio drops X |
The candidate moved pairs behind the camera. |
missing primary validation set 'held_out' |
No set carries the name in primary_validation_set. Rename the set or pass config=CalibrationGateConfig(primary_validation_set="..."). |
rotation delta Xdeg exceeds Ydeg |
The candidate moved further than the cap, even if it fits better. Raise the cap only if you expect a large move. |
translation delta Xm exceeds Ym |
Same, for t. |
What the gate cannot tell you¶
eligible means every rule above held on the pairs you supplied. It is not a statement about the
physical pose.
- Pairs are box centers. The pairs are camera box centers against LiDAR box centers, the same
pairs
matcher.calibrate()uses. A camera box center does not generally sit exactly on the projection of the LiDAR box center. A systematic offset in the pairs looks like a pose error, and the gate accepts a pose that absorbs it. - Independence is yours to provide. The gate cannot tell whether a validation set overlaps the fit set. Split by frame or time, not by pair: one vehicle seen in consecutive frames produces near-duplicate pairs.
- The thresholds are fixed numbers. Pixel thresholds depend on your image resolution and box
quality. Tune
inlier_pxand the pose caps for the site.
rng = np.random.default_rng(5)
offset = (0.0, -8.0) # every camera center sits 8 px above the projected LiDAR center
biased_fit = site_pairs(rng, 80, offset_px=offset)
biased_validation = {name: site_pairs(rng, 30, offset_px=offset) for name in ("held_out", "later_window")}
out = build_physical_refinement_candidate(exact, *biased_fit, biased_validation)
rot, trans = out.candidate.pose_error(true_extrinsics)
print(f"exact matrix, biased pairs: eligible={out.eligible}, "
f"candidate is {rot:.2f} deg / {trans:.3f} m from the truth")
out = build_physical_refinement_candidate(stored, *fit, {"held_out": fit})
print("fit set passed as the held-out set: eligible =", out.eligible)
def eligible_count(previous, offset_px=(0.0, 0.0), trials=200):
count = 0
for seed in range(trials):
rng = np.random.default_rng(1000 + seed)
fit = site_pairs(rng, 80, offset_px=offset_px)
validation = {name: site_pairs(rng, 30, offset_px=offset_px)
for name in ("held_out", "later_window")}
count += build_physical_refinement_candidate(previous, *fit, validation).eligible
return count
print("exact matrix, noise only: ", eligible_count(exact), "/ 200 eligible")
print("matrix 1 deg / 0.18 m off: ", eligible_count(stored), "/ 200 eligible")
print("exact matrix, 8 px offset: ", eligible_count(exact, (0.0, -8.0)), "/ 200 eligible")
exact matrix, biased pairs: eligible=True, candidate is 0.58 deg / 0.033 m from the truth
fit set passed as the held-out set: eligible = True
exact matrix, noise only: 0 / 200 eligible
matrix 1 deg / 0.18 m off: 200 / 200 eligible
exact matrix, 8 px offset: 200 / 200 eligible
These counts come from synthetic pairs with 1.5 px of noise. They show how the rules behave. They are not a field validation.
5. End to end¶
This block runs the whole loop on synthetic frames. It records 36 frames, simulates the review by writing the LiDAR ids a person would accept, collects the reviewed pairs, gates a candidate, and saves it only if it is eligible.
rng = np.random.default_rng(6)
rec = SceneRecorder("capture/site2", camera="cam_front")
rec.save_calibration(stored)
truth = {}
for _ in range(36):
image, cloud, boxes_2d, boxes_3d, pair_of = synth_frame(rng)
truth[rec.save_frame(image, cloud, boxes_2d, boxes_3d)] = pair_of
# Review. xcalib-review's Apply writes these same fields: the LiDAR id on the camera box.
root = Path("capture/site2")
fusion = root / "label_fusion/camera/cam_front"
for frame, pair_of in truth.items():
path = fusion / f"{frame}.json"
doc = json.loads(path.read_text())
for i, rect in enumerate(doc["objs"]):
rect["obj_id"] = str(pair_of[i] + 1) # default ids are 1..M
rect.pop("annotator", None)
path.write_text(json.dumps(doc))
def reviewed_pairs(frames):
"""(3D centers, 2D centers) for every camera box that carries a LiDAR id."""
points_3d, points_2d = [], []
for frame in frames:
rects = json.loads((fusion / f"{frame}.json").read_text())["objs"]
labels = json.loads((root / "label" / f"{frame}.json").read_text())["objs"]
boxes = {box["obj_id"]: box for box in labels}
for rect in rects:
box = boxes.get(rect["obj_id"])
if box is None: # not reviewed, or rejected
continue
x1, y1, x2, y2 = (rect["rect"][k] for k in ("x1", "y1", "x2", "y2"))
points_2d.append([(x1 + x2) / 2, (y1 + y2) / 2])
p = box["psr"]["position"]
points_3d.append([p["x"], p["y"], p["z"]])
return np.array(points_3d).reshape(-1, 3), np.array(points_2d).reshape(-1, 2)
frames = rec.frames()
fit = reviewed_pairs(frames[:20])
validation = {"held_out": reviewed_pairs(frames[20:28]), "later_window": reviewed_pairs(frames[28:])}
print("pairs:", len(fit[0]), {name: len(v[0]) for name, v in validation.items()})
result = build_physical_refinement_candidate(stored, *fit, validation)
print("eligible:", result.eligible, "| reasons:", result.reasons)
for name, b in result.before.items():
print(f" {name:<12} median {b.median_px:5.2f} -> {result.after[name].median_px:5.2f} px")
if result.eligible:
calib_file = root / "calib/camera/cam_front.json"
shutil.copy(calib_file, calib_file.with_name("cam_front.previous.json")) # keep the old matrix
rec.save_calibration(result.candidate)
session.set_calibration(result.candidate) # and hand it to a live session
before, after = stored.pose_error(true_extrinsics), result.candidate.pose_error(true_extrinsics)
print(f"pose error vs truth (known only because the data is synthetic): "
f"{before[0]:.2f} deg, {before[1]:.3f} m -> {after[0]:.2f} deg, {after[1]:.3f} m")
pairs: 120 {'held_out': 48, 'later_window': 48}
eligible: True | reasons: ()
fit median 7.91 -> 1.86 px
held_out median 7.28 -> 1.80 px
later_window median 7.86 -> 1.46 px
pose error vs truth (known only because the data is synthetic): 1.00 deg, 0.187 m -> 0.02 deg, 0.015 m
Before you deploy a real candidate, look at it. draw_calibration_overlay projects the cloud with
the new pose onto a frame (Visualization):
from xcalib import draw_calibration_overlay
overlay = draw_calibration_overlay(
image, cloud, intrinsics=K, rotation=result.candidate.rotation,
translation=result.candidate.translation, bboxes_3d=boxes_3d,
)
print(overlay.shape, overlay.dtype)
(720, 1280, 3) uint8
Guarantees and heuristics¶
| Piece | Guaranteed by the code | Heuristic |
|---|---|---|
| Capture | The camera box obj_id is empty at capture. A failure in save_frame never stops observe(). |
Which frames count as lost. |
| Review | Apply writes only the fusion label files. | The matcher's suggestions and the geometric warning. |
refine_projection |
On the pairs you pass, the returned pose's median error is not larger than the stored pose's. | That the refined pose is better on other pairs. |
| Gate | eligible is true only if every rule holds. A missing primary set, too few pairs or a pose jump past the caps give eligible=False. Intrinsics are unchanged. Nothing is written. |
That passing the rules means a better physical pose. That the validation sets are independent. The default thresholds. |
See also¶
- Input protocol: frame formats, box conventions and intrinsics.
- Continual refine: training the matcher from reviewed pairs.
- Visualization:
draw_calibration_overlay.