Pair selection¶
Matcher.match scores every camera box against every LiDAR box and returns a
[K, M] similarity matrix. Turning those scores into pairs takes two more steps,
and match can run both:
- Optimal one-to-one assignment picks the set of pairs with the highest total similarity, using each camera box and each LiDAR box at most once.
- A geometric gate uses a camera–LiDAR calibration to rule out pairs whose LiDAR box cannot land on the camera box in the image.
Both are off by default. A call that passes none of the new arguments returns the
same similarity, top_indices and matches as 0.2.0 (see
What changed since 0.2.0).
When to use it¶
| You need | Use |
|---|---|
| Ranked candidates for each camera box | result.matches and result.top_indices. Not one-to-one. |
| Each LiDAR box claimed by at most one camera box | one_to_one=True |
| Fewer wrong pairs between look-alike vehicles or on the wrong side of the road | one_to_one=True and calibration=... |
The gate needs a calibration that is roughly right, not exact. Its radius is generous by default (see Parameters), so a few pixels of drift do not veto a true pair. A pair that projects farther from the box than the radius is vetoed, however similar the two crops look, unless the gate would veto every LiDAR box for that camera box (see the row fallback below).
Use it with Matcher.match¶
result = matcher.match(
image, point_cloud, bboxes_2d, bboxes_3d,
one_to_one=True, # optimal one-to-one assignment
calibration=calibration, # optional: gate pairs by geometry first
)
for cam_idx, lidar_idx, score in result.assignment:
print(cam_idx, lidar_idx, score)
calibration is a successful CalibrationResult: the output of
matcher.calibrate(...), of a one-shot session, or one you build from known
extrinsics (see the example below). Without calibration
there is no gate, and assignment comes from the raw similarity.
Parameters¶
New keyword-only arguments of Matcher.match:
| Argument | Default | Unit | Meaning |
|---|---|---|---|
one_to_one |
False |
Fill result.assignment with the optimal one-to-one pairs. |
|
calibration |
None |
A successful CalibrationResult. Turns the gate on and fills geo_compatible and proj_px. A calibration with success=False raises ValueError. |
|
geo_gate_min_px |
None, which means 150.0 |
pixels | Smallest gate radius. Ignored without calibration. |
geo_gate_diag |
None, which means 3.0 |
multiple of the 2D box diagonal | Gate radius that scales with the box. Ignored without calibration. |
match_threshold (default 0.0, same unit as the similarity) already existed. With
one_to_one=True it also drops assignment pairs that score below it.
The gate radius for each camera box is
radius = max(geo_gate_min_px, geo_gate_diag * box_diagonal) # pixels
box_diagonal = sqrt((x2 - x1)**2 + (y2 - y1)**2)
A pair is compatible when the projected center of the LiDAR box lies in front of
the camera and within radius pixels of the center of the camera box. The
diagonal term keeps close-up objects, whose reprojection error grows with their
size, from losing their true match to a fixed radius.
If the gate blocks every LiDAR box for a camera box, that camera box is left
ungated: its whole row of geo_compatible is True. A badly wrong calibration then
degrades to appearance-only matching instead of emptying the row. match has no
switch for this; geometric_gate exposes it as row_fallback.
Result fields¶
| Field | Contents | Numbering |
|---|---|---|
assignment |
[(camera_idx, lidar_idx, score), ...], highest score first. score is the raw similarity. Empty unless one_to_one=True. |
Your boxes |
geo_compatible |
[K', M'] bool array, or None without calibration or when no box survives cropping. |
Kept boxes |
proj_px |
[K', M'] float array, or None without calibration or when no box survives cropping. Pixel distance from the projected LiDAR center to the camera box center; inf behind the camera. |
Kept boxes |
match drops a camera box whose integer x1 is negative or that has no area, and a
LiDAR box whose 1.25x-expanded crop holds no points. assignment and matches use your original numbering.
similarity, geo_compatible and proj_px have one row per kept camera box and
one column per kept LiDAR box: row r belongs to camera box
result.kept_2d_indices[r] and column c to LiDAR box result.kept_3d_indices[c].
The two functions¶
match calls these, and you can call them on your own arrays:
from xcalib import geometric_gate, one_to_one_assign
one_to_one_assign(similarity, threshold=0.0)
geometric_gate(bboxes_2d, centers_3d, calibration=None, *, intrinsics=None,
rotation=None, translation=None, min_px=150.0, diag_factor=3.0,
row_fallback=True)
one_to_one_assign takes a [K, M] similarity matrix and returns
[(row, col, score), ...], highest score first. It maximizes the total similarity
over all one-to-one assignments, then drops pairs below threshold. For
K != M it returns at most min(K, M) pairs.
geometric_gate returns (mask, dist_px), both [K, M]:
| Argument | Default | Unit | Meaning |
|---|---|---|---|
bboxes_2d |
pixels | [K, 4] camera boxes (x1, y1, x2, y2). |
|
centers_3d |
meters | [M, 3] LiDAR box centers, in the frame the calibration maps to the camera. |
|
calibration |
None |
A successful CalibrationResult. |
|
intrinsics, rotation, translation |
None |
Pass all three instead of calibration: CameraIntrinsics, a [3, 3] rotation and a [3] translation from the LiDAR frame to the camera frame. |
|
min_px |
150.0 |
pixels | Smallest gate radius. |
diag_factor |
3.0 |
multiple of the box diagonal | Gate radius that scales with the box. |
row_fallback |
True |
If the gate blocks every LiDAR box for a camera box, make that whole row compatible. A badly wrong calibration then falls back to appearance only instead of emptying the row. Matcher.match always uses True. |
Examples¶
Arrays only¶
This needs no model. The first part shows greedy top-1 sending both camera boxes to the same LiDAR box. The second part shows a geometric gate: appearance alone prefers a LiDAR box that is behind the camera, and the gate removes it.
import numpy as np
from xcalib import CalibrationResult, CameraIntrinsics, geometric_gate, one_to_one_assign
# 1. Greedy top-1 versus the optimal one-to-one assignment ---------------------
similarity = np.array([[0.90, 0.98, 0.10],
[0.20, 0.95, 0.30]])
print("greedy argmax per row:", similarity.argmax(axis=1).tolist())
print("one_to_one_assign :", one_to_one_assign(similarity, threshold=0.5))
# 2. A geometric gate -----------------------------------------------------------
# Pinhole camera; LiDAR frame is x forward, y left, z up; camera frame is
# x right, y down, z forward.
intrinsics = CameraIntrinsics(fx=1000.0, fy=1000.0, cx=960.0, cy=540.0)
R = np.array([[0.0, -1.0, 0.0],
[0.0, 0.0, -1.0],
[1.0, 0.0, 0.0]])
t = np.zeros(3)
calibration = CalibrationResult(
success=True, intrinsics=intrinsics, rotation=R, translation=t,
projection=intrinsics.K @ np.hstack([R, t.reshape(3, 1)]),
)
# Two camera boxes (x1, y1, x2, y2) in pixels ...
bboxes_2d = np.array([[620.0, 520.0, 700.0, 580.0], # centre (660, 550)
[1160.0, 500.0, 1240.0, 560.0]]) # centre (1200, 530)
# ... and three LiDAR box centres (x, y, z) in meters.
centers_3d = np.array([[20.0, 6.0, 0.0], # projects near (660, 540)
[25.0, -6.0, 0.0], # projects near (1200, 540)
[-10.0, 0.0, 0.0]]) # behind the camera
mask, dist_px = geometric_gate(bboxes_2d, centers_3d, calibration)
print("compatible:\n", mask.astype(int))
print("distance (px):\n", np.round(dist_px, 1))
# Appearance alone prefers the wrong objects: box 0 ends up assigned to the box
# behind the camera.
similarity = np.array([[0.60, 0.97, 0.90],
[0.20, 0.95, 0.10]])
print("no gate :", one_to_one_assign(similarity, threshold=0.5))
# Veto the incompatible pairs with a large finite penalty, then assign.
gated = np.where(mask, similarity, -1e9)
print("gated :", one_to_one_assign(gated, threshold=0.5))
Output:
greedy argmax per row: [1, 1]
one_to_one_assign : [(1, 1, 0.95), (0, 0, 0.9)]
compatible:
[[1 0 0]
[0 1 0]]
distance (px):
[[ 10. 540.1 inf]
[540.1 10. inf]]
no gate : [(1, 1, 0.95), (0, 2, 0.9)]
gated : [(1, 1, 0.95), (0, 0, 0.6)]
A Matcher.match call¶
A randomly initialised model loads and runs, but its scores carry no information.
That is enough to see the arguments, the result fields and the gate, because the
geometry does not depend on the weights. Only the pairs on the one_to_one and
gated (150 px, 3x) lines depend on the random weights, so yours can differ.
Everything else in the output follows from the geometry. The random scores sit
close to zero, so match_threshold=-1.0 keeps every pair instead of dropping the
ones below the default 0.0.
import tempfile
from pathlib import Path
import numpy as np
import torch
from xcalib import CalibrationResult, CameraIntrinsics, Matcher
from xcalib.models.registry import build_model
from xcalib.utils.config import load_yaml
from xcalib.utils.io import default_config_path
# A randomly initialised model: it loads and runs, but its similarity scores carry
# no information. The shapes, the index spaces and the gate do not depend on it.
torch.manual_seed(0)
cfg = load_yaml(default_config_path("crlite_vit_exp1", "a9_dataset_r02_s01"))
cfg.set("device", "cpu")
weights = Path(tempfile.mkdtemp()) / "crlite_vit_exp1.pth"
torch.save({"model_state_dict": build_model("crlite_vit_exp1", cfg).state_dict()}, weights)
matcher = Matcher.from_pretrained(model="crlite_vit_exp1", weights=weights, config=cfg, device="cpu")
# Camera, and a calibration for it (LiDAR frame: x forward, y left, z up).
intrinsics = CameraIntrinsics(fx=1000.0, fy=1000.0, cx=960.0, cy=540.0)
R = np.array([[0.0, -1.0, 0.0], [0.0, 0.0, -1.0], [1.0, 0.0, 0.0]])
t = np.zeros(3)
calibration = CalibrationResult(
success=True, intrinsics=intrinsics, rotation=R, translation=t,
projection=intrinsics.K @ np.hstack([R, t.reshape(3, 1)]),
)
# A synthetic frame: four objects, each seen by the camera and by the LiDAR.
# Camera box i and LiDAR box i are the same object.
rng = np.random.default_rng(0)
centers = np.array([[20.0, 6.0, 0.8], [25.0, -6.0, 0.8], [32.0, 2.0, 0.8], [45.0, -3.0, 0.8]])
cam = centers @ R.T + t
uv = cam[:, :2] / cam[:, 2:3] * 1000.0 + [960.0, 540.0]
half = (np.clip(2400.0 / cam[:, 2], 30.0, 120.0) / 2.0)[:, None]
bboxes_2d = np.hstack([uv - half, uv + half]).astype(np.float32)
size = np.array([4.4, 1.8, 1.6])
bboxes_3d = np.hstack([centers - size / 2, centers + size / 2]).astype(np.float32)
image = rng.integers(0, 255, size=(1080, 1920, 3), dtype=np.uint8)
point_cloud = np.vstack(
[rng.uniform(-5, 60, size=(1500, 3))]
+ [rng.uniform(b[:3], b[3:], size=(300, 3)) for b in bboxes_3d]
).astype(np.float32)
# The same call as before 0.3.0: no assignment, no gate.
plain = matcher.match(image, point_cloud, bboxes_2d, bboxes_3d, top_k=1)
print("similarity:", plain.similarity.shape, "| assignment:", plain.assignment,
"| geo_compatible:", plain.geo_compatible, "| proj_px:", plain.proj_px)
# Optimal one-to-one assignment, no calibration, so no gate. match_threshold=-1.0
# keeps every pair; the random scores sit near zero and some are below the default 0.0.
result = matcher.match(image, point_cloud, bboxes_2d, bboxes_3d,
one_to_one=True, match_threshold=-1.0)
print("one_to_one :", sorted((i, j) for i, j, _ in result.assignment))
# With the calibration, incompatible pairs are vetoed before the assignment.
result = matcher.match(image, point_cloud, bboxes_2d, bboxes_3d, match_threshold=-1.0,
one_to_one=True, calibration=calibration)
print("gated (150 px, 3x) :", sorted((i, j) for i, j, _ in result.assignment))
print("geo_compatible:\n", result.geo_compatible.astype(int))
print("proj_px:\n", np.round(result.proj_px))
# A tighter gate: radius = max(geo_gate_min_px, geo_gate_diag x box diagonal).
result = matcher.match(image, point_cloud, bboxes_2d, bboxes_3d, match_threshold=-1.0,
one_to_one=True, calibration=calibration,
geo_gate_min_px=40.0, geo_gate_diag=0.5)
print("gated (40 px, 0.5x) :", sorted((i, j) for i, j, _ in result.assignment))
print("geo_compatible:\n", result.geo_compatible.astype(int))
# Dropped boxes: a camera box that starts left of the image is dropped before scoring.
# `assignment` stays in YOUR numbering; the matrices follow the kept boxes.
edge = np.array([[-10.0, 400.0, 60.0, 460.0]], dtype=np.float32)
result = matcher.match(image, point_cloud, np.vstack([edge, bboxes_2d]), bboxes_3d,
match_threshold=-1.0, one_to_one=True, calibration=calibration,
geo_gate_min_px=40.0, geo_gate_diag=0.5)
print("kept_2d_indices :", result.kept_2d_indices.tolist())
print("similarity / geo_compatible:", result.similarity.shape, result.geo_compatible.shape)
print("assignment :", sorted((i, j) for i, j, _ in result.assignment))
Output:
similarity: (4, 4) | assignment: [] | geo_compatible: None | proj_px: None
one_to_one : [(0, 2), (1, 1), (2, 3), (3, 0)]
gated (150 px, 3x) : [(0, 0), (1, 2), (2, 3), (3, 1)]
geo_compatible:
[[1 0 1 1]
[0 1 1 1]
[1 1 1 1]
[0 1 1 1]]
proj_px:
[[ 0. 540. 238. 367.]
[540. 0. 303. 174.]
[238. 303. 0. 129.]
[367. 174. 129. 0.]]
gated (40 px, 0.5x) : [(0, 0), (1, 1), (2, 2), (3, 3)]
geo_compatible:
[[1 0 0 0]
[0 1 0 0]
[0 0 1 0]
[0 0 0 1]]
kept_2d_indices : [1, 2, 3, 4]
similarity / geo_compatible: (4, 4) (4, 4)
assignment : [(1, 0), (2, 1), (3, 2), (4, 3)]
With the tight gate, each camera box has exactly one compatible LiDAR box (the
identity matrix above), so the assignment is the true pairing whatever the weights
are. In the last call, the
camera box at row 0 of your input was dropped, so kept_2d_indices starts at 1
and assignment still uses your numbering.
Things to know¶
matchesandtop_indicesare never gated and are not one-to-one. Onlyassignmentis.- The threshold is applied after the assignment. The assignment maximizes the total
over all pairs, including pairs below the threshold, so a pair under the
threshold can take a row and a column away from a pair above it.
one_to_one_assign(np.array([[0.90, 0.45], [0.89, 0.00]]), threshold=0.5)returns[(1, 0, 0.89)]: the best total uses(0, 1)and(1, 0), and(0, 1)is then dropped. - If you gate by hand, mark incompatible pairs with a large finite value such as
-1e9, asMatcher.matchdoes.-infis not supported: depending on the shape, a row or column that is blocked everywhere makes the problem infeasible. - Boxes given as
(cx, cy, cz, dx, dy, dz)are told apart from(xmin, ymin, zmin, xmax, ymax, zmax)by a per-row rule. A row whose three dimensions are all at least its three center coordinates is read as extents, and the gate then uses the wrong center. See 3D bbox format auto-detection. Matcher.pairis an alias ofMatcher.matchand takes the same arguments.- SciPy is optional.
one_to_one_assignuses it when it is installed and otherwise falls back to a built-in exact solver that returns an equally good set of pairs (the same total similarity); ties may be broken differently.
What changed since 0.2.0¶
| 0.2.0 | 0.3.0 | Default | |
|---|---|---|---|
Matcher.match arguments |
image, point_cloud, bboxes_2d, bboxes_3d, top_k, match_threshold, return_latency, validate |
Four keyword-only arguments added: one_to_one, calibration, geo_gate_min_px, geo_gate_diag. The positional order is unchanged. |
False, None, None, None: the call behaves as before. |
MatchResult fields |
similarity, top_indices, matches, kept_2d_indices, kept_3d_indices, latency_ms, model, device |
Three fields added at the end: assignment, geo_compatible, proj_px. |
[], None, None |
| Existing outputs of the same call | similarity, top_indices, matches, kept_2d_indices and kept_3d_indices are identical for the same weights and inputs. |
Unchanged | |
match_threshold |
Filters matches. |
Also filters assignment. |
Matters only with one_to_one=True. |
calibration without one_to_one |
Fills geo_compatible and proj_px. matches and assignment are not affected. |
Opt-in | |
| Public API | xcalib.one_to_one_assign, xcalib.geometric_gate, and the constants GEO_GATE_MIN_PX = 150.0 and GEO_GATE_DIAG = 3.0 in xcalib.assign. |
New | |
| Runtime dependencies | No new ones. SciPy stays optional. |
Code written for 0.2.0 needs no change. Its match calls return the same matches,
because pair selection only runs when you pass one_to_one=True or calibration.
Matcher.calibrate and one-shot sessions call match without the new arguments, so
they behave as before.
One change reaches beyond pair selection: import xcalib is now lazy. from xcalib import Matcher
and every other name in xcalib.__all__ work as before, but the submodules (xcalib.engine,
xcalib.protocol, and so on) are attributes of xcalib only after you import them, for example
import xcalib.protocol.
See also: Input protocol for the box formats, and the API reference for the full signatures.