API reference¶
The curated public API — everything here is importable from the top-level xcalib package and
stable across minor versions.
Matcher¶
xcalib.engine.matcher.Matcher
¶
High-level wrapper around any of the xcalib models.
from_pretrained(model, weights=None, config=None, device=None, *, site='a9_dataset_r02_s01', repo_id=None, revision=None, token=None)
classmethod
¶
Construct a matcher from local files or the HuggingFace Hub.
Resolution rules for weights:
None— downloadcheckpoints/{model}_{site}_best.pth(+ the matching YAML whenconfigis None) from the configured Hub repo. Userepo_id/revision/tokenonly when you need to override the released defaults.hf://org/repo[@rev]/path— explicit Hub file.- directory — output of :meth:
save_pretrained; loads<dir>/{model}.pth(+<dir>/{model}.yamlif config is None). - anything else — local checkpoint path (original behaviour).
config may likewise be an hf:// URI, a local path outside the
package, an EdgeConfig, or a plain dict. If omitted for local weights,
the packaged xcalib/cfg/{model}_{site}.yaml is used when present.
match(image, point_cloud, bboxes_2d, bboxes_3d, top_k=1, match_threshold=0.0, return_latency=False, validate='warn', *, one_to_one=False, calibration=None, geo_gate_min_px=None, geo_gate_diag=None)
¶
Run the full pipeline on a single frame.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
image
|
ndarray
|
[H, W, 3] uint8 RGB. |
required |
point_cloud
|
ndarray
|
[P, 3+] float32 (X, Y, Z[, intensity ...]). |
required |
bboxes_2d
|
ndarray
|
[K, 4] image bboxes (x1, y1, x2, y2). |
required |
bboxes_3d
|
ndarray
|
[M, 6] LiDAR bboxes either (xmin,ymin,zmin,xmax,ymax,zmax) or (cx, cy, cz, dx, dy, dz). |
required |
top_k
|
int
|
How many top matches to keep in |
1
|
match_threshold
|
float
|
Minimum similarity to include in |
0.0
|
return_latency
|
bool
|
Force the forward-pass timing into the result. |
False
|
validate
|
str
|
Input-protocol policy — "strict" raises on any violation, "warn" (default) raises on hard errors and logs soft warnings once, "off" skips checks. See docs/protocol.md. |
'warn'
|
one_to_one
|
bool
|
Also compute |
False
|
calibration
|
Optional['CalibrationResult']
|
A successful :class: |
None
|
geo_gate_min_px
|
Optional[float]
|
Gate radius floor in pixels. Default: the module
constant |
None
|
geo_gate_diag
|
Optional[float]
|
Gate radius as a multiple of the 2D box diagonal.
Default: |
None
|
pair(*args, **kwargs)
¶
Alias of :meth:match — "pairing" in the partner's vocabulary.
match_frame(frame)
¶
Skip cropping, score a pre-prepared FrameData.
Used by scripts/paper/validate_paper.py where the UTC HDF5 loader has
already produced FrameData. Returns (scores [N,M], elapsed_ms).
calibrate(image, point_cloud, bboxes_2d, bboxes_3d, intrinsics, *, match_threshold=0.5, ransac_reproj_px=8.0, validate='warn')
¶
Estimate the camera-LiDAR projection matrix from one frame.
Runs :meth:match, keeps each image detection's best LiDAR match
above match_threshold, and solves PnP/RANSAC over the
(2D bbox center, 3D bbox center) correspondences. For better
conditioning across multiple frames use :meth:oneshot /
CalibrationSession instead.
oneshot(intrinsics, **kwargs)
¶
Open a projection-matrix-supervised one-shot learning session.
The session observes frames, refines the camera-LiDAR projection
matrix, harvests geometry-confirmed pseudo-pairs into a feature
bank, and applies confidence-gated adapter updates. See
xcalib.oneshot.session.OneShotSession.
train(train_data, val_data, *, output_dir='runs/finetune', **kwargs)
¶
Fine-tune the currently loaded weights on HDF5 caches.
Thin wrapper over :func:xcalib.engine.trainer.train that trains
this matcher's model in place, then reloads the best (val-MRR)
checkpoint so the matcher immediately serves the improved weights.
train_data / val_data accept local .h5 paths or Hub
site names. Extra keyword arguments (epochs, lr,
scheduler, ...) are forwarded to the trainer.
Returns:
| Type | Description |
|---|---|
Path
|
Path to |
build(target='onnx', output_dir=None, *, precision='fp16', device=None, onnx_dir=None, trtexec=None, extra_trt_args=(), verify=True)
¶
Export the currently loaded weights to a deployment artifact.
Works for any weights — pretrained, fine-tuned, or one-shot
adapted — because it traces self.model as it currently is.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
target
|
str
|
"onnx" (any host) or "trt" (needs trtexec, i.e. Jetson/JetPack or a TensorRT install). |
'onnx'
|
output_dir
|
Union[str, Path, None]
|
Where artifacts land. Defaults to
|
None
|
precision
|
str
|
TensorRT precision ("fp32" | "fp16" | "best"). |
'fp16'
|
device
|
Union[str, device, None]
|
Device used for ONNX tracing (default cpu). |
None
|
onnx_dir
|
Union[str, Path, None]
|
For target="trt": where to look for / export the
intermediate ONNX graphs (default ./onnx/ |
None
|
trtexec
|
Optional[str]
|
Explicit trtexec path override. |
None
|
extra_trt_args
|
Tuple[str, ...]
|
Extra flags forwarded verbatim to trtexec. |
()
|
verify
|
bool
|
Run the onnxruntime parity check after export. |
True
|
Returns:
| Type | Description |
|---|---|
'BuildResult'
|
|
'BuildResult'
|
(for ONNX) the max |torch - onnx| parity per output. |
save_pretrained(save_dir)
¶
Write <dir>/{model}.pth + <dir>/{model}.yaml.
The checkpoint round-trips through :meth:from_pretrained (pass the
directory as weights) and can be exported via :meth:build.
One-shot adapter weights, when present, are stored under
adapters_state_dict and re-attached automatically on load.
Results¶
xcalib.engine.matcher.MatchResult
dataclass
¶
Container for matcher.match() output.
Attributes:
| Name | Type | Description |
|---|---|---|
similarity |
ndarray
|
|
top_indices |
ndarray
|
|
matches |
List[Tuple[int, int, float]]
|
|
kept_2d_indices |
ndarray
|
Original indices of the camera boxes that survived
cropping; row |
kept_3d_indices |
ndarray
|
The same for the LiDAR boxes and the columns of |
latency_ms |
float
|
Forward-pass time when |
model |
str
|
Name of the model that produced the scores. |
device |
str
|
Device the model ran on. |
assignment |
List[Tuple[int, int, float]]
|
With |
geo_compatible |
Optional[ndarray]
|
With |
proj_px |
Optional[ndarray]
|
With |
xcalib.engine.exporter.BuildResult
dataclass
¶
What Matcher.build / export_onnx hand back to the caller.
xcalib.oneshot.calibration.CalibrationResult
dataclass
¶
Output of a PnP/RANSAC solve.
extrinsics
property
¶
[3,4] = [R|t] (without intrinsics).
pose_error(gt_extrinsics)
¶
Accuracy of this solve against a ground-truth extrinsic.
gt_extrinsics is a 4x4 (or 3x4) lidar->camera pose — e.g.
UTCFrame.extrinsics[camera] from an A9 cache. Returns
(rotation_deg, translation_m): the geodesic rotation angle and the
Euclidean translation distance between the estimated [R|t] and the
ground truth, the camera-pose errors a paper-style table reports.
project(points_3d)
¶
Project LiDAR-frame points to pixels.
Returns (uv [N,2], depth [N]); depth is the camera-frame Z, points behind the camera have depth <= 0.
Pair selection¶
How to use these: Pair selection.
xcalib.assign.one_to_one_assign(similarity, threshold=0.0)
¶
Optimal one-to-one (row, col, score) pairs from a similarity matrix.
Maximizes the TOTAL similarity over all one-to-one assignments (LAP), then
drops pairs scoring below threshold. Unlike a greedy pass, a
high-confidence pair is never lost because its column was claimed first.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
similarity
|
ndarray
|
|
required |
threshold
|
float
|
pairs with |
0.0
|
Returns:
| Type | Description |
|---|---|
List[Tuple[int, int, float]]
|
List of |
xcalib.assign.geometric_gate(bboxes_2d, centers_3d, calibration=None, *, intrinsics=None, rotation=None, translation=None, min_px=GEO_GATE_MIN_PX, diag_factor=GEO_GATE_DIAG, row_fallback=True)
¶
Geometric compatibility of every (2D box, 3D center) pair.
Projects each 3D center into the image with the given calibration and
measures its pixel distance to each 2D box center. A pair is compatible
when the projection lands IN FRONT of the camera and within
max(min_px, diag_factor × box diagonal) of the box center.
With row_fallback=True (default) a row whose gate blocks every
column reverts to all-compatible: a garbage calibration (e.g. an identity
extrinsic placeholder) must degrade to appearance-only matching rather
than blank the row.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
bboxes_2d
|
ndarray
|
|
required |
centers_3d
|
ndarray
|
|
required |
calibration
|
Optional['CalibrationResult']
|
a successful :class: |
None
|
intrinsics
|
Optional['CameraIntrinsics']
|
camera intrinsics, used with |
None
|
rotation
|
Optional[ndarray]
|
|
None
|
translation
|
Optional[ndarray]
|
|
None
|
min_px
|
float
|
gate radius floor in pixels (default |
GEO_GATE_MIN_PX
|
diag_factor
|
float
|
gate radius as a multiple of the 2D box diagonal
(default |
GEO_GATE_DIAG
|
row_fallback
|
bool
|
revert a row that blocks every column to all-compatible. |
True
|
Returns:
| Type | Description |
|---|---|
ndarray
|
|
ndarray
|
for centers behind the camera / invalid projections. |
Calibration refinement¶
How to use these: Deployed site.
xcalib.oneshot.calibration.refine_projection(previous, points_3d, points_2d, *, ransac_reproj_px=8.0, min_points=6)
¶
NUDGE an existing calibration instead of re-solving from scratch.
A deployed site usually has a good matrix already — surveyed, or solved
when the sensors were mounted. After weather / vibration shifts the rig a
little, a fresh :func:estimate_projection from whatever pairs happen to
be buffered is NOT guaranteed to beat it (planar scenes are bistable, a
bad pair pulls a fresh solve anywhere). This seeds Levenberg-Marquardt at
the previous pose, refines on the pairs the previous pose already
explains, and adopts the result ONLY when it lowers the median
reprojection error over ALL supplied pairs — the worst case is "keep what
you had".
Returns a CalibrationResult whose accepted says whether the refined
pose beat previous (when False, the returned pose IS the previous one
and the caller should keep it). buffer_reproj_px carries the median
error of the returned pose over the supplied pairs. Use
:func:estimate_projection instead when there is no trustworthy prior —
e.g. a first-time setup solved purely from hand-labelled pairs.
xcalib.calibration_refine.build_physical_refinement_candidate(previous, fit_points_3d, fit_points_2d, validation_sets, *, config=None)
¶
Fit on one set and accept only from caller-supplied independent sets.
validation_sets insertion order is preserved for presentation. The set
named by primary_validation_set must improve materially; every supplied
set must also pass tail, inlier, and depth guardrails.
xcalib.calibration_refine.evaluate_calibration(calibration, points_3d, points_2d, *, inlier_px=25.0)
¶
Evaluate one pose without hiding negative-depth or non-finite pairs.
xcalib.calibration_refine.CalibrationGateConfig
dataclass
¶
Conservative defaults for a user-reviewed physical-pose candidate.
inlier_px = 25.0
class-attribute
instance-attribute
¶
min_fit_points = 12
class-attribute
instance-attribute
¶
min_validation_points = 6
class-attribute
instance-attribute
¶
min_median_improvement_px = 0.5
class-attribute
instance-attribute
¶
min_median_improvement_ratio = 0.02
class-attribute
instance-attribute
¶
max_q90_regression_px = 2.0
class-attribute
instance-attribute
¶
max_inlier_ratio_drop = 0.02
class-attribute
instance-attribute
¶
max_depth_ratio_drop = 0.01
class-attribute
instance-attribute
¶
min_positive_depth_ratio = 0.8
class-attribute
instance-attribute
¶
max_rotation_delta_deg = 3.0
class-attribute
instance-attribute
¶
max_translation_delta_m = 1.0
class-attribute
instance-attribute
¶
primary_validation_set = 'held_out'
class-attribute
instance-attribute
¶
__init__(inlier_px=25.0, min_fit_points=12, min_validation_points=6, min_median_improvement_px=0.5, min_median_improvement_ratio=0.02, max_q90_regression_px=2.0, max_inlier_ratio_drop=0.02, max_depth_ratio_drop=0.01, min_positive_depth_ratio=0.8, max_rotation_delta_deg=3.0, max_translation_delta_m=1.0, primary_validation_set='held_out')
¶
xcalib.calibration_refine.PhysicalCalibrationCandidate
dataclass
¶
A proposed pose plus independent evidence; never persisted by XCalib.
previous
instance-attribute
¶
candidate
instance-attribute
¶
before = field(default_factory=dict)
class-attribute
instance-attribute
¶
after = field(default_factory=dict)
class-attribute
instance-attribute
¶
eligible = False
class-attribute
instance-attribute
¶
reasons = ()
class-attribute
instance-attribute
¶
rotation_delta_deg = 0.0
class-attribute
instance-attribute
¶
translation_delta_m = 0.0
class-attribute
instance-attribute
¶
__init__(previous, candidate, before=dict(), after=dict(), eligible=False, reasons=(), rotation_delta_deg=0.0, translation_delta_m=0.0)
¶
xcalib.calibration_refine.CalibrationMetrics
dataclass
¶
pairs
instance-attribute
¶
median_px
instance-attribute
¶
q90_px
instance-attribute
¶
rmse_inlier_px
instance-attribute
¶
inliers
instance-attribute
¶
inlier_ratio
instance-attribute
¶
positive_depth
instance-attribute
¶
positive_depth_ratio
instance-attribute
¶
__init__(pairs, median_px, q90_px, rmse_inlier_px, inliers, inlier_ratio, positive_depth, positive_depth_ratio)
¶
Frame capture¶
How to use this: Deployed site, Review GUI.
xcalib.capture.SceneRecorder
¶
Write matcher inputs as a SUSTechPOINTS-style scene directory.
Layout produced (readable by xcalib.gui.scene.Scene and
xcalib-review)::
<root>/camera/<camera>/<frame>.jpg
<root>/lidar/<frame>.pcd (binary; or .npy)
<root>/label/<frame>.json 3D boxes with obj_id
<root>/label_fusion/camera/<camera>/<frame>.json
2D rects (annotator="xcalib")
<root>/calib/camera/<camera>.json K + extrinsic (save_calibration)
Identity model: ids_3d become the LiDAR boxes' canonical obj_id
(1-based enumeration when omitted); ids_2d are the camera's OWN track
ids and land in obj_attr as "t<id>" — a rect's obj_id stays
EMPTY until a human (or an accepted match) fills it, so captured model
guesses can never masquerade as verified pairs.
save_calibration(calibration)
¶
Write the site's K + extrinsic once (enables the geometric gate and matrix refinement on the captured scene).
save_frame(image, point_cloud, bboxes_2d, bboxes_3d, *, frame=None, ids_3d=None, types_3d=None, ids_2d=None, types_2d=None)
¶
Persist one frame's matcher inputs; returns the frame id.
bboxes_2d [K,4] (x1,y1,x2,y2); bboxes_3d [M,6] in either
protocol form (extents or center+size). frame defaults to a
zero-padded counter continuing from what is already on disk.
Training¶
xcalib.engine.trainer.train(model, train_data, val_data, *, model_name=None, config=None, site=None, epochs=10, lr=0.0001, weight_decay=0.0001, optimizer='adamw', warmup_epochs=0, scheduler='none', min_lr=1e-05, temperature=0.07, label_smoothing=0.05, stage1_weight=1.0, stage2_weight=1.0, weights=None, output_dir='runs/train', device='auto', seed=42, limit_train_frames=None, limit_val_frames=None, pairwise_batch_size=64, box_source=None)
¶
Train (from scratch) or fine-tune a matcher; returns the best checkpoint.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model
|
Union[str, Module]
|
Registry name ( |
required |
train_data
|
Union[str, Path]
|
Local |
required |
val_data
|
Union[str, Path]
|
Same as train_data, for the validation split. |
required |
model_name
|
Optional[str]
|
Required when model is a module. |
None
|
config
|
Union[EdgeConfig, str, Path, None]
|
Reference YAML / :class: |
None
|
site
|
Optional[str]
|
Site used to pick the default config (inferred from
train_data when it is a site name; falls back to |
None
|
weights
|
Union[str, Path, None]
|
Optional checkpoint to warm-start from (registry-name mode). |
None
|
box_source
|
Optional[str]
|
Which 3D boxes feed the point-cloud crops, for training and
validation alike: |
None
|
output_dir
|
Union[str, Path]
|
Where per-epoch checkpoints, |
'runs/train'
|
Returns:
| Type | Description |
|---|---|
Path
|
Path to |
Datasets¶
xcalib.hub.datasets.load_dataset(site, split='test', *, repo_id=None, revision=None, token=None, local=None, box_source='stored')
¶
Return an HDF5 frame loader for one split of a site's cache.
Resolution order: explicit local path → checkout-local default →
Hub download (cached by huggingface_hub). Requires h5py (the
[train] extra).
box_source picks the 3D boxes the loader hands out: "stored" (the
cache's lidar_bbox_3d, the default) or "envelope" (the extents
enclosing each oriented cuboid; see docs/hdf5-format.md).
Visualization¶
xcalib.visualization.draw_matching_overlay(image, point_cloud, bboxes_2d, bboxes_3d, matches, *, output_height=420, match_threshold=None)
¶
Draw camera detections, LiDAR BEV detections, and matched links.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
image
|
ndarray
|
RGB image, |
required |
point_cloud
|
ndarray
|
LiDAR points with XYZ in the first three columns. |
required |
bboxes_2d
|
ndarray
|
Camera boxes as |
required |
bboxes_3d
|
ndarray
|
LiDAR boxes as extents or center+dimensions. |
required |
matches
|
Iterable[Sequence[float]]
|
Iterable of |
required |
output_height
|
int
|
Height of each visual pane. |
420
|
match_threshold
|
float | None
|
Optional score floor for display; lower-scoring matches are omitted from the drawn links. |
None
|
Returns:
| Type | Description |
|---|---|
ndarray
|
RGB |
xcalib.visualization.draw_calibration_overlay(image, point_cloud, *, projection=None, intrinsics=None, rotation=None, translation=None, bboxes_3d=None, max_points=6000)
¶
Project LiDAR points and optional 3D-box centers onto the camera image.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
image
|
ndarray
|
RGB image, |
required |
point_cloud
|
ndarray
|
LiDAR points with XYZ in columns 0-2. |
required |
projection
|
ndarray | None
|
Optional |
None
|
intrinsics
|
CameraIntrinsics | ndarray | None
|
Camera intrinsics used with |
None
|
rotation
|
ndarray | None
|
|
None
|
translation
|
ndarray | None
|
|
None
|
bboxes_3d
|
ndarray | None
|
Optional LiDAR detections; their centers are projected and marked with labels. |
None
|
max_points
|
int
|
Upper bound on projected point count for readable overlays. |
6000
|
Returns:
| Type | Description |
|---|---|
ndarray
|
RGB |
Input protocol¶
xcalib.protocol.validate_frame_inputs(image, point_cloud, bboxes_2d, bboxes_3d)
¶
Check one frame's raw inputs against the input protocol.
Returns the (possibly empty) list of violations; never raises. Use
enforce (or Matcher.match(validate=...)) to act on them.
xcalib.protocol.CameraIntrinsics
dataclass
¶
xcalib.protocol.ProtocolError
¶
Bases: ValueError
Raised when frame inputs violate the protocol (see docs/protocol.md).
xcalib.protocol.ProtocolViolation
dataclass
¶
One contract violation found by validate_frame_inputs.