Unified Agentic Sourcing

Real-case eval scorecard

Five production sourcing requests, lifted verbatim from sr_contexts, scored on every parameter the suite measures. Unlike the hand-authored cases, these carry the buyer's real specs — in both shapes agent-core actually stores them.

2026-07-27T07:47:34.531Z → 2026-07-27T08:21:28.811Z · mode record · 5 cases

Cases passed
5/5
all structural scorers green
Mean score
99%
across all parameters
Sourcing cost
$0.9467
$0.1893 per case
Attrs / product
24.285
data-quality baseline

Per-parameter scorecard

outcome-matchesstructuralproduct-type-containsstructuralmin-shortliststructuralspec-structurestructuralforbidden-absentstructuralhonest-handlingstructuralpicks-resolvedstructuralspec-qualityjudgeshortlist-relevancejudgenot-fabricatedjudgemean
passBaby Brezza-style formula dispenserSR 3885 · object-specs · images100%100%100%100%100%100%100%100%90%100%99%
passPlain white ceramic mug, 350ml × 2000SR 3591 · string-specs · no images100%100%100%70%100%100%100%100%90%100%96%
passColorful kids backpack (stock)SR 3636 · string-specs · no images100%100%100%100%100%100%100%100%90%100%99%
passNo-duct commercial kitchen rangehoodSR 3655 · string-specs · no images100%100%100%100%100%100%100%100%100%100%100%
passCustom evening gown (The Goldstar Gown)SR 3614 · string-specs · images100%100%100%100%100%100%100%90%100%100%99%
parameter mean100%100%100%94%100%100%100%98%94%100%99%
score 0% → 100% ✕ = failed its threshold · hover a cell for the scorer's reasoning

Findings

10% of LLM calls failed; the OpenAI fallback carried the difference

144 of 1443 calls (10%) failed and were billed. The Gemini→OpenAI fallback absorbed most of them — gpt-5.4-nano ran 101 calls for $0.0642, 7% of this run's spend. The fallback is doing its job, but the underlying gemini-3.1-flash-lite failure rate under concurrency is the thing to fix — a safety net carrying this much traffic is load-bearing.

modelcallsspendshare
gemini-3.1-flash-lite1225$0.740678%
gemini-2.5-flash117$0.141915%
gpt-5.4-nano101$0.06427%

What these cases add to the suite

The pre-existing 10 SR cases are all problematic ones — unsourceable, custom, or pre-picked — and every one is graded on not over-claiming. Nothing covered an ordinary request that should simply return a good shortlist. Four of these five do; the fifth (the gown) is the honest-handling contrast, so the set measures both directions.

Two cases carry reference images and three do not, so image-grounded spec generation and text-only sourcing are both exercised against real buyer language rather than a hand-written query string.

Case detail

pass Baby Brezza-style formula dispenser SR 3885 · object-shaped specs · with images 99% · ok · $0.0696

Brand-reference OEM sourcing with a populated adjacent-product veto; images carry the spec sheet.

23 match(es).

Adopted veto specs (1)

  • product_type = automatic baby formula dispenser, one-step baby formula maker, automatic formula maker, automatic powder formula dispenser w10

Funnel

candidates130
verified36
cleaned26
matched26
shortlisted23

Top confirmed matches (23 total)

#titleplatformscore
1Smart automatic baby formula milk maker dispenseralibaba1.633
2Smart Baby Formula Milk Maker Dispenser with 1.5L Water Tankalibaba1.633
3Automatic baby formula milk maker machinealibaba1.633
4Automatic Baby Formula Milk Maker Machinealibaba1.448
5Smart automatic baby formula dispenser, one-step mixingalibaba1.448
pass Plain white ceramic mug, 350ml × 2000 SR 3591 · string-shaped specs · no images 96% · ok · $0.1505

High-volume commodity SKU; string-shaped specs; exercises numeric capacity + single-colour veto.

30 match(es).

Adopted veto specs (6)

  • capacity = 350ml w5
  • handle = standard C-handle w5
  • branding = none / unbranded stock product w5
  • material = ceramic, glazed w5
  • color = plain white w5
  • product_category = ceramic mug w5

Funnel

candidates129
verified38
cleaned40
matched40
shortlisted31
returned (top 30)30

Top confirmed matches (30 total)

#titleplatformscore
1Custom white ceramic mug with printed or engraved logo16880.8
211oz White Ceramic Sublimation Coffee Mugalibaba0.633
3Plain white porcelain sublimation mug with custom logoalibaba0.633
411oz White Ceramic Coffee Mug for Promotionalibaba0.633
511oz White Ceramic Sublimation Coffee Mugalibaba0.633
pass Colorful kids backpack (stock) SR 3636 · string-shaped specs · no images 99% · ok · $0.3057

Stock off-the-shelf, style-driven with no numeric spec; customisation gate must not fire.

30 match(es).

Adopted veto specs (5)

  • stock_off_the_shelf = true w5
  • customization_required = false w5
  • style = colorful w5
  • product_category = kids backpack w5
  • target_user = children w5

Funnel

candidates135
verified89
cleaned55
matched54
shortlisted39
returned (top 30)30

Top confirmed matches (30 total)

#titleplatformscore
1Lightweight cartoon kids mini backpack for school and travel16881.383
2Kids preschool backpack, cute animal print, large capacity16881.383
3Cartoon kids anti-lost backpack, double-strap for ages 3–516881.383
4Kids School Backpack for Daily Usealibaba1.383
5Kids anti-lost cartoon mini backpack for toddlers16881.233
pass No-duct commercial kitchen rangehood SR 3655 · string-shaped specs · no images 100% · ok · $0.1363

Non-consumer, functional-spec-driven category; checks generalisation beyond consumer goods.

26 match(es).

Adopted veto specs (5)

  • filtration_purification = built-in filtration or purification cabinet/section for smoke treatment w5
  • configuration = stock/off-the-shelf product; no custom branding, custom color, or custom packaging w5
  • product_category = commercial kitchen rangehood / exhaust hood w5
  • application = commercial kitchen use w5
  • venting_configuration = no external ducting required; internal-circulation/ductless/smokeless style w5

Funnel

candidates133
verified47
cleaned56
matched56
shortlisted26

Top confirmed matches (26 total)

#titleplatformscore
1Commercial Ductless Stainless Steel Kitchen Range Hood Fume Purifieralibaba1.633
2Commercial Ductless Wall Mounted Range Hood Stainless Steel 2100Walibaba1.3
3Commercial Ductless ESP Kitchen Exhaust Range Hood with Fanalibaba1.3
4Commercial Ductless Kitchen Range Hood Oil Smoke Purifieralibaba1.133
5Commercial Wall-Mounted Ductless Stainless Steel ESP Kitchen Range Hoodalibaba1
pass Custom evening gown (The Goldstar Gown) SR 3614 · string-shaped specs · with images 99% · all_eliminated · $0.2846

Buyer explicitly rejected catalog approximation; graded on not fabricating matches from stock gowns.

All 30 matched products were eliminated by veto specs — no products fully match the must-haves.

Adopted veto specs (14)

  • product_category = custom women's evening gown w5
  • custom_build_requirement = exact custom build from supplied front/back reference images and written requirements; not an approximate catalog substitute w5
  • style_name = The Goldstar Gown / Goldstar Dress w5
  • exterior_material_finish = gold sequin exterior w5
  • lining = full black satin lining w5
  • lining_visibility = black satin lining visible at the left thigh-high slit w5
  • neckline = high square neckline at collarbone w5
  • straps = wide parallel straps, approximately 1.5–2 inches, placed at outer shoulder edges w5
  • back = sleek bare/open back w5
  • closure = invisible center-back zipper w5
  • silhouette = fitted column silhouette through bust, waist, and hips w5
  • slit = thigh-high slit on left side only w5
  • hem = floor length w5
  • custom_measurements_in_inches = chest: 44, upper_chest: 40, waist: 35, hips: 42, waist_to_floor: 41, shoulder_width: 22 w5

Funnel

candidates799
verified0
cleaned30
matched30
shortlisted0

Top confirmed matches (0 total)

#titleplatformscore
(none confirmed)