Skip to content

API reference

The curated public API — everything here is importable from the top-level xcalib package and stable across minor versions.

Matcher

xcalib.engine.matcher.Matcher

High-level wrapper around any of the xcalib models.

from_pretrained(model, weights=None, config=None, device=None, *, site='a9_dataset_r02_s01', repo_id=None, revision=None, token=None) classmethod

Construct a matcher from local files or the HuggingFace Hub.

Resolution rules for weights:

  • None — download checkpoints/{model}_{site}_best.pth (+ the matching YAML when config is None) from the configured Hub repo. Use repo_id / revision / token only when you need to override the released defaults.
  • hf://org/repo[@rev]/path — explicit Hub file.
  • directory — output of :meth:save_pretrained; loads <dir>/{model}.pth (+ <dir>/{model}.yaml if config is None).
  • anything else — local checkpoint path (original behaviour).

config may likewise be an hf:// URI, a local path outside the package, an EdgeConfig, or a plain dict. If omitted for local weights, the packaged xcalib/cfg/{model}_{site}.yaml is used when present.

match(image, point_cloud, bboxes_2d, bboxes_3d, top_k=1, match_threshold=0.0, return_latency=False, validate='warn', *, one_to_one=False, calibration=None, geo_gate_min_px=None, geo_gate_diag=None)

Run the full pipeline on a single frame.

Parameters:

Name Type Description Default
image ndarray

[H, W, 3] uint8 RGB.

required
point_cloud ndarray

[P, 3+] float32 (X, Y, Z[, intensity ...]).

required
bboxes_2d ndarray

[K, 4] image bboxes (x1, y1, x2, y2).

required
bboxes_3d ndarray

[M, 6] LiDAR bboxes either (xmin,ymin,zmin,xmax,ymax,zmax) or (cx, cy, cz, dx, dy, dz).

required
top_k int

How many top matches to keep in matches per image.

1
match_threshold float

Minimum similarity to include in matches (and, with one_to_one, in assignment).

0.0
return_latency bool

Force the forward-pass timing into the result.

False
validate str

Input-protocol policy — "strict" raises on any violation, "warn" (default) raises on hard errors and logs soft warnings once, "off" skips checks. See docs/protocol.md.

'warn'
one_to_one bool

Also compute result.assignment — the OPTIMAL one-to-one pairing (Hungarian) over the (gated) similarity. Greedy top-1 can cross-assign two confident pairs; the optimal assignment cannot.

False
calibration Optional['CalibrationResult']

A successful :class:CalibrationResult (e.g. the site's solved extrinsics). Enables the geometric gate: pairs whose 3D center projects behind the camera or far from the 2D box are vetoed BEFORE the assignment — appearance similarity cannot override an impossible geometry. Also fills result.geo_compatible / result.proj_px.

None
geo_gate_min_px Optional[float]

Gate radius floor in pixels. Default: the module constant xcalib.assign.GEO_GATE_MIN_PX. See :func:xcalib.assign.geometric_gate.

None
geo_gate_diag Optional[float]

Gate radius as a multiple of the 2D box diagonal. Default: xcalib.assign.GEO_GATE_DIAG.

None

pair(*args, **kwargs)

Alias of :meth:match — "pairing" in the partner's vocabulary.

match_frame(frame)

Skip cropping, score a pre-prepared FrameData.

Used by scripts/paper/validate_paper.py where the UTC HDF5 loader has already produced FrameData. Returns (scores [N,M], elapsed_ms).

calibrate(image, point_cloud, bboxes_2d, bboxes_3d, intrinsics, *, match_threshold=0.5, ransac_reproj_px=8.0, validate='warn')

Estimate the camera-LiDAR projection matrix from one frame.

Runs :meth:match, keeps each image detection's best LiDAR match above match_threshold, and solves PnP/RANSAC over the (2D bbox center, 3D bbox center) correspondences. For better conditioning across multiple frames use :meth:oneshot / CalibrationSession instead.

oneshot(intrinsics, **kwargs)

Open a projection-matrix-supervised one-shot learning session.

The session observes frames, refines the camera-LiDAR projection matrix, harvests geometry-confirmed pseudo-pairs into a feature bank, and applies confidence-gated adapter updates. See xcalib.oneshot.session.OneShotSession.

train(train_data, val_data, *, output_dir='runs/finetune', **kwargs)

Fine-tune the currently loaded weights on HDF5 caches.

Thin wrapper over :func:xcalib.engine.trainer.train that trains this matcher's model in place, then reloads the best (val-MRR) checkpoint so the matcher immediately serves the improved weights. train_data / val_data accept local .h5 paths or Hub site names. Extra keyword arguments (epochs, lr, scheduler, ...) are forwarded to the trainer.

Returns:

Type Description
Path

Path to best.pth inside output_dir.

build(target='onnx', output_dir=None, *, precision='fp16', device=None, onnx_dir=None, trtexec=None, extra_trt_args=(), verify=True)

Export the currently loaded weights to a deployment artifact.

Works for any weights — pretrained, fine-tuned, or one-shot adapted — because it traces self.model as it currently is.

Parameters:

Name Type Description Default
target str

"onnx" (any host) or "trt" (needs trtexec, i.e. Jetson/JetPack or a TensorRT install).

'onnx'
output_dir Union[str, Path, None]

Where artifacts land. Defaults to ./onnx/<model>/ or ./engines/<model>/.

None
precision str

TensorRT precision ("fp32" | "fp16" | "best").

'fp16'
device Union[str, device, None]

Device used for ONNX tracing (default cpu).

None
onnx_dir Union[str, Path, None]

For target="trt": where to look for / export the intermediate ONNX graphs (default ./onnx//).

None
trtexec Optional[str]

Explicit trtexec path override.

None
extra_trt_args Tuple[str, ...]

Extra flags forwarded verbatim to trtexec.

()
verify bool

Run the onnxruntime parity check after export.

True

Returns:

Type Description
'BuildResult'

xcalib.engine.exporter.BuildResult with artifact paths and

'BuildResult'

(for ONNX) the max |torch - onnx| parity per output.

save_pretrained(save_dir)

Write <dir>/{model}.pth + <dir>/{model}.yaml.

The checkpoint round-trips through :meth:from_pretrained (pass the directory as weights) and can be exported via :meth:build. One-shot adapter weights, when present, are stored under adapters_state_dict and re-attached automatically on load.

Results

xcalib.engine.matcher.MatchResult dataclass

Container for matcher.match() output.

Attributes:

Name Type Description
similarity ndarray

[K, M] float32 scores in [-1, 1] or [0, 1] (model-dependent), one row per camera box and one column per LiDAR box that survived cropping (see kept_2d_indices / kept_3d_indices).

top_indices ndarray

[K] int64, the highest-scoring column of each row.

matches List[Tuple[int, int, float]]

(img_idx, lidar_idx, score) triples with score >= match_threshold, up to top_k per camera box, numbered as in the boxes passed to match().

kept_2d_indices ndarray

Original indices of the camera boxes that survived cropping; row i of similarity is camera box kept_2d_indices[i].

kept_3d_indices ndarray

The same for the LiDAR boxes and the columns of similarity.

latency_ms float

Forward-pass time when return_latency=True, otherwise the time of the whole call (cropping, scoring, assignment).

model str

Name of the model that produced the scores.

device str

Device the model ran on.

assignment List[Tuple[int, int, float]]

With one_to_one=True, the optimal one-to-one (img_idx, lidar_idx, similarity) pairs in the original box numbering, geometry-gated when calibration was given; [] otherwise.

geo_compatible Optional[ndarray]

With calibration, a [K, M] bool matrix of geometrically compatible pairs, indexed like similarity; else None.

proj_px Optional[ndarray]

With calibration, a [K, M] matrix of pixel distances from each projected 3D center to each 2D box center (inf behind the camera), indexed like similarity; else None.

xcalib.engine.exporter.BuildResult dataclass

What Matcher.build / export_onnx hand back to the caller.

xcalib.oneshot.calibration.CalibrationResult dataclass

Output of a PnP/RANSAC solve.

extrinsics property

[3,4] = [R|t] (without intrinsics).

pose_error(gt_extrinsics)

Accuracy of this solve against a ground-truth extrinsic.

gt_extrinsics is a 4x4 (or 3x4) lidar->camera pose — e.g. UTCFrame.extrinsics[camera] from an A9 cache. Returns (rotation_deg, translation_m): the geodesic rotation angle and the Euclidean translation distance between the estimated [R|t] and the ground truth, the camera-pose errors a paper-style table reports.

project(points_3d)

Project LiDAR-frame points to pixels.

Returns (uv [N,2], depth [N]); depth is the camera-frame Z, points behind the camera have depth <= 0.

Pair selection

How to use these: Pair selection.

xcalib.assign.one_to_one_assign(similarity, threshold=0.0)

Optimal one-to-one (row, col, score) pairs from a similarity matrix.

Maximizes the TOTAL similarity over all one-to-one assignments (LAP), then drops pairs scoring below threshold. Unlike a greedy pass, a high-confidence pair is never lost because its column was claimed first.

Parameters:

Name Type Description Default
similarity ndarray

[K, M] float matrix (any real values).

required
threshold float

pairs with similarity < threshold are discarded after the assignment.

0.0

Returns:

Type Description
List[Tuple[int, int, float]]

List of (row, col, score) sorted by score, highest first.

xcalib.assign.geometric_gate(bboxes_2d, centers_3d, calibration=None, *, intrinsics=None, rotation=None, translation=None, min_px=GEO_GATE_MIN_PX, diag_factor=GEO_GATE_DIAG, row_fallback=True)

Geometric compatibility of every (2D box, 3D center) pair.

Projects each 3D center into the image with the given calibration and measures its pixel distance to each 2D box center. A pair is compatible when the projection lands IN FRONT of the camera and within max(min_px, diag_factor × box diagonal) of the box center.

With row_fallback=True (default) a row whose gate blocks every column reverts to all-compatible: a garbage calibration (e.g. an identity extrinsic placeholder) must degrade to appearance-only matching rather than blank the row.

Parameters:

Name Type Description Default
bboxes_2d ndarray

[K, 4] (x1, y1, x2, y2).

required
centers_3d ndarray

[M, 3] LiDAR-frame box centers (see :func:xcalib.oneshot.calibration.bbox3d_centers).

required
calibration Optional['CalibrationResult']

a successful :class:CalibrationResult; or pass intrinsics + rotation + translation directly.

None
intrinsics Optional['CameraIntrinsics']

camera intrinsics, used with rotation and translation when calibration is not given.

None
rotation Optional[ndarray]

[3, 3] LiDAR-to-camera rotation.

None
translation Optional[ndarray]

[3] LiDAR-to-camera translation.

None
min_px float

gate radius floor in pixels (default GEO_GATE_MIN_PX).

GEO_GATE_MIN_PX
diag_factor float

gate radius as a multiple of the 2D box diagonal (default GEO_GATE_DIAG).

GEO_GATE_DIAG
row_fallback bool

revert a row that blocks every column to all-compatible.

True

Returns:

Type Description
ndarray

(mask [K, M] bool, dist_px [K, M] float) — dist_px is inf

ndarray

for centers behind the camera / invalid projections.

Calibration refinement

How to use these: Deployed site.

xcalib.oneshot.calibration.refine_projection(previous, points_3d, points_2d, *, ransac_reproj_px=8.0, min_points=6)

NUDGE an existing calibration instead of re-solving from scratch.

A deployed site usually has a good matrix already — surveyed, or solved when the sensors were mounted. After weather / vibration shifts the rig a little, a fresh :func:estimate_projection from whatever pairs happen to be buffered is NOT guaranteed to beat it (planar scenes are bistable, a bad pair pulls a fresh solve anywhere). This seeds Levenberg-Marquardt at the previous pose, refines on the pairs the previous pose already explains, and adopts the result ONLY when it lowers the median reprojection error over ALL supplied pairs — the worst case is "keep what you had".

Returns a CalibrationResult whose accepted says whether the refined pose beat previous (when False, the returned pose IS the previous one and the caller should keep it). buffer_reproj_px carries the median error of the returned pose over the supplied pairs. Use :func:estimate_projection instead when there is no trustworthy prior — e.g. a first-time setup solved purely from hand-labelled pairs.

xcalib.calibration_refine.build_physical_refinement_candidate(previous, fit_points_3d, fit_points_2d, validation_sets, *, config=None)

Fit on one set and accept only from caller-supplied independent sets.

validation_sets insertion order is preserved for presentation. The set named by primary_validation_set must improve materially; every supplied set must also pass tail, inlier, and depth guardrails.

xcalib.calibration_refine.evaluate_calibration(calibration, points_3d, points_2d, *, inlier_px=25.0)

Evaluate one pose without hiding negative-depth or non-finite pairs.

xcalib.calibration_refine.CalibrationGateConfig dataclass

Conservative defaults for a user-reviewed physical-pose candidate.

inlier_px = 25.0 class-attribute instance-attribute

min_fit_points = 12 class-attribute instance-attribute

min_validation_points = 6 class-attribute instance-attribute

min_median_improvement_px = 0.5 class-attribute instance-attribute

min_median_improvement_ratio = 0.02 class-attribute instance-attribute

max_q90_regression_px = 2.0 class-attribute instance-attribute

max_inlier_ratio_drop = 0.02 class-attribute instance-attribute

max_depth_ratio_drop = 0.01 class-attribute instance-attribute

min_positive_depth_ratio = 0.8 class-attribute instance-attribute

max_rotation_delta_deg = 3.0 class-attribute instance-attribute

max_translation_delta_m = 1.0 class-attribute instance-attribute

primary_validation_set = 'held_out' class-attribute instance-attribute

__init__(inlier_px=25.0, min_fit_points=12, min_validation_points=6, min_median_improvement_px=0.5, min_median_improvement_ratio=0.02, max_q90_regression_px=2.0, max_inlier_ratio_drop=0.02, max_depth_ratio_drop=0.01, min_positive_depth_ratio=0.8, max_rotation_delta_deg=3.0, max_translation_delta_m=1.0, primary_validation_set='held_out')

xcalib.calibration_refine.PhysicalCalibrationCandidate dataclass

A proposed pose plus independent evidence; never persisted by XCalib.

previous instance-attribute

candidate instance-attribute

before = field(default_factory=dict) class-attribute instance-attribute

after = field(default_factory=dict) class-attribute instance-attribute

eligible = False class-attribute instance-attribute

reasons = () class-attribute instance-attribute

rotation_delta_deg = 0.0 class-attribute instance-attribute

translation_delta_m = 0.0 class-attribute instance-attribute

__init__(previous, candidate, before=dict(), after=dict(), eligible=False, reasons=(), rotation_delta_deg=0.0, translation_delta_m=0.0)

xcalib.calibration_refine.CalibrationMetrics dataclass

pairs instance-attribute

median_px instance-attribute

q90_px instance-attribute

rmse_inlier_px instance-attribute

inliers instance-attribute

inlier_ratio instance-attribute

positive_depth instance-attribute

positive_depth_ratio instance-attribute

__init__(pairs, median_px, q90_px, rmse_inlier_px, inliers, inlier_ratio, positive_depth, positive_depth_ratio)

Frame capture

How to use this: Deployed site, Review GUI.

xcalib.capture.SceneRecorder

Write matcher inputs as a SUSTechPOINTS-style scene directory.

Layout produced (readable by xcalib.gui.scene.Scene and xcalib-review)::

<root>/camera/<camera>/<frame>.jpg
<root>/lidar/<frame>.pcd                (binary; or .npy)
<root>/label/<frame>.json               3D boxes with obj_id
<root>/label_fusion/camera/<camera>/<frame>.json
                                        2D rects (annotator="xcalib")
<root>/calib/camera/<camera>.json       K + extrinsic (save_calibration)

Identity model: ids_3d become the LiDAR boxes' canonical obj_id (1-based enumeration when omitted); ids_2d are the camera's OWN track ids and land in obj_attr as "t<id>" — a rect's obj_id stays EMPTY until a human (or an accepted match) fills it, so captured model guesses can never masquerade as verified pairs.

save_calibration(calibration)

Write the site's K + extrinsic once (enables the geometric gate and matrix refinement on the captured scene).

save_frame(image, point_cloud, bboxes_2d, bboxes_3d, *, frame=None, ids_3d=None, types_3d=None, ids_2d=None, types_2d=None)

Persist one frame's matcher inputs; returns the frame id.

bboxes_2d [K,4] (x1,y1,x2,y2); bboxes_3d [M,6] in either protocol form (extents or center+size). frame defaults to a zero-padded counter continuing from what is already on disk.

Training

xcalib.engine.trainer.train(model, train_data, val_data, *, model_name=None, config=None, site=None, epochs=10, lr=0.0001, weight_decay=0.0001, optimizer='adamw', warmup_epochs=0, scheduler='none', min_lr=1e-05, temperature=0.07, label_smoothing=0.05, stage1_weight=1.0, stage2_weight=1.0, weights=None, output_dir='runs/train', device='auto', seed=42, limit_train_frames=None, limit_val_frames=None, pairwise_batch_size=64, box_source=None)

Train (from scratch) or fine-tune a matcher; returns the best checkpoint.

Parameters:

Name Type Description Default
model Union[str, Module]

Registry name ("crlite", ...) or an already-built model module (used by :meth:Matcher.train).

required
train_data Union[str, Path]

Local .h5 cache path or Hub site name (resolved through xcalib.hub.datasets).

required
val_data Union[str, Path]

Same as train_data, for the validation split.

required
model_name Optional[str]

Required when model is a module.

None
config Union[EdgeConfig, str, Path, None]

Reference YAML / :class:EdgeConfig. Defaults to the packaged config for (model, site).

None
site Optional[str]

Site used to pick the default config (inferred from train_data when it is a site name; falls back to utc4).

None
weights Union[str, Path, None]

Optional checkpoint to warm-start from (registry-name mode).

None
box_source Optional[str]

Which 3D boxes feed the point-cloud crops, for training and validation alike: "stored" (the cache's lidar_bbox_3d) or "envelope" (the extents enclosing each oriented cuboid, for caches whose stored boxes miss turned vehicles; see docs/hdf5-format.md). Defaults to the config's box_source key, then "stored". Weights trained with one source were not trained for the other.

None
output_dir Union[str, Path]

Where per-epoch checkpoints, best.pth (selected by validation MRR), and history.json land.

'runs/train'

Returns:

Type Description
Path

Path to best.pth inside output_dir.

Datasets

xcalib.hub.datasets.load_dataset(site, split='test', *, repo_id=None, revision=None, token=None, local=None, box_source='stored')

Return an HDF5 frame loader for one split of a site's cache.

Resolution order: explicit local path → checkout-local default → Hub download (cached by huggingface_hub). Requires h5py (the [train] extra).

box_source picks the 3D boxes the loader hands out: "stored" (the cache's lidar_bbox_3d, the default) or "envelope" (the extents enclosing each oriented cuboid; see docs/hdf5-format.md).

Visualization

xcalib.visualization.draw_matching_overlay(image, point_cloud, bboxes_2d, bboxes_3d, matches, *, output_height=420, match_threshold=None)

Draw camera detections, LiDAR BEV detections, and matched links.

Parameters:

Name Type Description Default
image ndarray

RGB image, [H, W, 3] uint8.

required
point_cloud ndarray

LiDAR points with XYZ in the first three columns.

required
bboxes_2d ndarray

Camera boxes as (x1, y1, x2, y2).

required
bboxes_3d ndarray

LiDAR boxes as extents or center+dimensions.

required
matches Iterable[Sequence[float]]

Iterable of (camera_index, lidar_index, score) triples, such as MatchResult.matches.

required
output_height int

Height of each visual pane.

420
match_threshold float | None

Optional score floor for display; lower-scoring matches are omitted from the drawn links.

None

Returns:

Type Description
ndarray

RGB uint8 side-by-side overlay.

xcalib.visualization.draw_calibration_overlay(image, point_cloud, *, projection=None, intrinsics=None, rotation=None, translation=None, bboxes_3d=None, max_points=6000)

Project LiDAR points and optional 3D-box centers onto the camera image.

Parameters:

Name Type Description Default
image ndarray

RGB image, [H, W, 3] uint8.

required
point_cloud ndarray

LiDAR points with XYZ in columns 0-2.

required
projection ndarray | None

Optional 3x4 camera projection matrix K[R|t].

None
intrinsics CameraIntrinsics | ndarray | None

Camera intrinsics used with rotation and translation when projection is not supplied.

None
rotation ndarray | None

3x3 extrinsic rotation.

None
translation ndarray | None

3 or 3x1 extrinsic translation.

None
bboxes_3d ndarray | None

Optional LiDAR detections; their centers are projected and marked with labels.

None
max_points int

Upper bound on projected point count for readable overlays.

6000

Returns:

Type Description
ndarray

RGB uint8 camera overlay.

Input protocol

xcalib.protocol.validate_frame_inputs(image, point_cloud, bboxes_2d, bboxes_3d)

Check one frame's raw inputs against the input protocol.

Returns the (possibly empty) list of violations; never raises. Use enforce (or Matcher.match(validate=...)) to act on them.

xcalib.protocol.CameraIntrinsics dataclass

Pinhole intrinsics, pixels. Distortion follows OpenCV ordering.

K property

3x3 camera matrix.

dist_coeffs property

Distortion coefficients as a float64 vector (zeros when None).

xcalib.protocol.ProtocolError

Bases: ValueError

Raised when frame inputs violate the protocol (see docs/protocol.md).

xcalib.protocol.ProtocolViolation dataclass

One contract violation found by validate_frame_inputs.