Tool CallingdebuggingIntermediate

A tool works in testing but fails intermittently in production

Symptoms

  • The create_shipment tool succeeds most of the time but throws validation errors on maybe 8% of calls.
  • Failures cluster on requests with dates, currencies, and weights.
  • Retrying the exact same user request sometimes succeeds and sometimes fails.
ok    args={"weight_kg": 2.5, "ship_date": "2024-11-03", "currency": "USD"}
error args={"weight_kg": "2.5 kg", "ship_date": "Nov 3 2024", "currency": "dollars"}
error args={"weight_kg": 2500, "ship_date": "2024-11-03", "currency": "USD"}   // grams, not kg
schema: weight_kg: number, ship_date: string(YYYY-MM-DD), currency: string(ISO-4217)

Investigate

Inspect areas in any order (0/5 inspected). When you think you know the root cause, commit to it.

Argument validation
Schema constraints
Tool descriptions
Retries / timeouts
Idempotency