wanhae.lee
129 lines
8.5 KiB
Markdown
129 lines
8.5 KiB
Markdown
# Gemma-4 tool calling: constrained (xgrammar) vs unconstrained
|
|
|
|
- model: `google/gemma-4-E2B-it` via HF transformers
|
|
- constraint: `xgr.get_model_structural_tag("gemma_4", tools=..., tool_choice="required")` compiled and applied with `xgr.contrib.hf.LogitsProcessor`
|
|
- 4 samples per scenario per mode; identical sampling settings for both modes (per-scenario values below)
|
|
- outputs validated with a pure-Python port of sglang's `Gemma4Detector` parser plus `jsonschema`
|
|
|
|
## Results (passing samples / total)
|
|
|
|
### A. complex-schema
|
|
|
|
temperature=0.9, top_p=0.95, max_new_tokens=384, reasoning=False
|
|
|
|
| metric | constrained | unconstrained |
|
|
|---|---|---|
|
|
| well_formed | 4/4 ✅ | 4/4 ✅ |
|
|
| valid_name | 4/4 ✅ | 4/4 ✅ |
|
|
| schema_valid | 4/4 ✅ | 4/4 ✅ |
|
|
| stops_at_boundary | 4/4 ✅ | 0/4 ❌ |
|
|
|
|
### B. multiturn-thinking
|
|
|
|
temperature=0.9, top_p=0.95, max_new_tokens=384, reasoning=True
|
|
|
|
| metric | constrained | unconstrained |
|
|
|---|---|---|
|
|
| well_formed | 4/4 ✅ | 4/4 ✅ |
|
|
| valid_name | 4/4 ✅ | 4/4 ✅ |
|
|
| schema_valid | 4/4 ✅ | 4/4 ✅ |
|
|
| stops_at_boundary | 4/4 ✅ | 0/4 ❌ |
|
|
| thought_ok | 4/4 ✅ | 0/4 ❌ |
|
|
|
|
### C. adversarial-payload
|
|
|
|
temperature=1.5, top_p=1.0, max_new_tokens=640, reasoning=False
|
|
|
|
| metric | constrained | unconstrained |
|
|
|---|---|---|
|
|
| well_formed | 4/4 ✅ | 4/4 ✅ |
|
|
| valid_name | 4/4 ✅ | 4/4 ✅ |
|
|
| schema_valid | 4/4 ✅ | 4/4 ✅ |
|
|
| stops_at_boundary | 4/4 ✅ | 0/4 ❌ |
|
|
|
|
## Metric definitions
|
|
|
|
- `well_formed` — every tool call block is complete and parseable (sglang parser port)
|
|
- `valid_name` — every called tool exists
|
|
- `schema_valid` — parsed arguments validate against the tool's JSON schema
|
|
- `stops_at_boundary` — generation did not run past <tool_call|> into <|tool_response>
|
|
- `thought_ok` — <|channel>thought opened and closed before the first tool call
|
|
|
|
## Failure examples
|
|
|
|
**A. complex-schema / unconstrained #0** — failed `stops_at_boundary`: ran past <tool_call|> into <|tool_response> after the call
|
|
|
|
```
|
|
<|tool_call>call:create_calendar_event{attendees:[{email:<|"|>alice.kim@example.com<|"|>,name:<|"|>Alice Kim<|"|>},{email:<|"|>bob.lee@example.com<|"|>,name:<|"|>Bob Lee<|"|>},{email:<|"|>chris.park@example.com<|"|>,name:<|"|>Chris Park<|"|>}],duration_minutes:90,location:{floor:7,room:<|"|>Jupiter<|"|>},priority:<|"|>high<|"|>,send_invites:true,start:<|"|>2026-07-10T14:00:00+09:00<|"|>,title:<|"|>Q3 Roadmap Review ("final" draft)<|"|>}<tool_call|><|tool_response>
|
|
```
|
|
|
|
**A. complex-schema / unconstrained #1** — failed `stops_at_boundary`: ran past <tool_call|> into <|tool_response> after the call
|
|
|
|
```
|
|
<|tool_call>call:create_calendar_event{attendees:[{email:<|"|>alice.kim@example.com<|"|>,name:<|"|>Alice Kim<|"|>},{email:<|"|>bob.lee@example.com<|"|>,name:<|"|>Bob Lee<|"|>},{email:<|"|>chris.park@example.com<|"|>,name:<|"|>Chris Park<|"|>}],duration_minutes:90,location:{floor:7,room:<|"|>Jupiter<|"|>},priority:<|"|>high<|"|>,send_invites:true,start:<|"|>2026-07-10T14:00:00+09:00<|"|>,title:<|"|>Q3 Roadmap Review ("final" draft)<|"|>}<tool_call|><|tool_response>
|
|
```
|
|
|
|
**A. complex-schema / unconstrained #2** — failed `stops_at_boundary`: ran past <tool_call|> into <|tool_response> after the call
|
|
|
|
```
|
|
<|tool_call>call:create_calendar_event{attendees:[{email:<|"|>alice.kim@example.com<|"|>,name:<|"|>Alice Kim<|"|>},{email:<|"|>bob.lee@example.com<|"|>,name:<|"|>Bob Lee<|"|>},{email:<|"|>chris.park@example.com<|"|>,name:<|"|>Chris Park<|"|>}],duration_minutes:90,location:{floor:7,room:<|"|>Jupiter<|"|>},priority:<|"|>high<|"|>,send_invites:true,start:<|"|>2026-07-10T14:00:00+09:00<|"|>,title:<|"|>Q3 Roadmap Review ("final" draft)<|"|>}<tool_call|><|tool_response>
|
|
```
|
|
|
|
**A. complex-schema / unconstrained #3** — failed `stops_at_boundary`: ran past <tool_call|> into <|tool_response> after the call
|
|
|
|
```
|
|
<|tool_call>call:create_calendar_event{attendees:[{email:<|"|>alice.kim@example.com<|"|>,name:<|"|>Alice Kim<|"|>},{email:<|"|>bob.lee@example.com<|"|>,name:<|"|>Bob Lee<|"|>},{email:<|"|>chris.park@example.com<|"|>,name:<|"|>Chris Park<|"|>}],duration_minutes:90,location:{floor:7,room:<|"|>Jupiter<|"|>},priority:<|"|>high<|"|>,send_invites:true,start:<|"|>2026-07-10T14:00:00+09:00<|"|>,title:<|"|>Q3 Roadmap Review ("final" draft)<|"|>}<tool_call|><|tool_response>
|
|
```
|
|
|
|
**B. multiturn-thinking / unconstrained #0** — failed `stops_at_boundary`, `thought_ok`: ran past <tool_call|> into <|tool_response> after the call; skipped or never closed the <|channel>thought section
|
|
|
|
```
|
|
<|tool_call>call:convert_temperature{from_unit:<|"|>celsius<|"|>,to_unit:<|"|>fahrenheit<|"|>,value:3}<tool_call|><|tool_response>
|
|
```
|
|
|
|
**B. multiturn-thinking / unconstrained #1** — failed `stops_at_boundary`, `thought_ok`: ran past <tool_call|> into <|tool_response> after the call; skipped or never closed the <|channel>thought section
|
|
|
|
```
|
|
<|tool_call>call:convert_temperature{from_unit:<|"|>celsius<|"|>,to_unit:<|"|>fahrenheit<|"|>,value:3}<tool_call|><|tool_response>
|
|
```
|
|
|
|
**B. multiturn-thinking / unconstrained #2** — failed `stops_at_boundary`, `thought_ok`: ran past <tool_call|> into <|tool_response> after the call; skipped or never closed the <|channel>thought section
|
|
|
|
```
|
|
<|tool_call>call:convert_temperature{from_unit:<|"|>celsius<|"|>,to_unit:<|"|>fahrenheit<|"|>,value:3}<tool_call|><|tool_response>
|
|
```
|
|
|
|
**B. multiturn-thinking / unconstrained #3** — failed `stops_at_boundary`, `thought_ok`: ran past <tool_call|> into <|tool_response> after the call; skipped or never closed the <|channel>thought section
|
|
|
|
```
|
|
<|tool_call>call:convert_temperature{from_unit:<|"|>celsius<|"|>,to_unit:<|"|>fahrenheit<|"|>,value:3}<tool_call|><|tool_response>
|
|
```
|
|
|
|
**C. adversarial-payload / unconstrained #0** — failed `stops_at_boundary`: ran past <tool_call|> into <|tool_response> after the call
|
|
|
|
```
|
|
<|tool_call>call:create_ticket{affected:{regions:[<|"|>ap-northeast-2<|"|>,<|"|>us-east-1<|"|>],service:<|"|>payments<|"|>,version:<|"|>2.14.1<|"|>},body:<|"|>Deployed config: {"retry": {"max": 3, "backoff_ms": [100, 200], "jitter": true}}
|
|
Error line: TypeError: cannot destructure {id: undefined} at applyRetry (retry.js:42).<|"|>,cc_emails:[<|"|>dev-alerts@example.com<|"|>],estimated_minutes:45,labels:[<|"|>regression<|"|>,<|"|>backend<|"|>],severity:<|"|>blocker<|"|>,title:<|"|>Payments retry storm after config rollout<|"|>,urgent_escalation:true}<tool_call|><|tool_response>
|
|
```
|
|
|
|
**C. adversarial-payload / unconstrained #1** — failed `stops_at_boundary`: ran past <tool_call|> into <|tool_response> after the call
|
|
|
|
```
|
|
<|tool_call>call:create_ticket{affected:{regions:[<|"|>ap-northeast-2<|"|>,<|"|>us-east-1<|"|>],service:<|"|>payments<|"|>,version:<|"|>2.14.1<|"|>},body:<|"|>Deployed config: {"retry": {"max": 3, "backoff_ms": [100, 200], "jitter": true}}
|
|
Error line: TypeError: cannot destructure {id: undefined} at applyRetry (retry.js:42).<|"|>,cc_emails:[<|"|>dev-alerts@example.com<|"|>],estimated_minutes:45,labels:[<|"|>bug<|"|>,<|"|>regression<|"|>,<|"|>backend<|"|>],severity:<|"|>blocker<|"|>,title:<|"|>Payments retry storm after config rollout<|"|>,urgent_escalation:true}<tool_call|><|tool_response>
|
|
```
|
|
|
|
**C. adversarial-payload / unconstrained #2** — failed `stops_at_boundary`: ran past <tool_call|> into <|tool_response> after the call
|
|
|
|
```
|
|
<|tool_call>call:create_ticket{affected:{regions:[<|"|>ap-northeast-2<|"|>,<|"|>us-east-1<|"|>],service:<|"|>payments<|"|>,version:<|"|>2.14.1<|"|>},body:<|"|>Deployed config: {"retry": {"max": 3, "backoff_ms": [100, 200], "jitter": true}}
|
|
Error line: TypeError: cannot destructure {id: undefined} at applyRetry (retry.js:42)<|"|>,cc_emails:[<|"|>dev-alerts@example.com<|"|>],estimated_minutes:45,labels:[<|"|>bug<|"|>,<|"|>regression<|"|>,<|"|>backend<|"|>],severity:<|"|>blocker<|"|>,title:<|"|>Payments retry storm after config rollout<|"|>,urgent_escalation:true}<tool_call|><|tool_response>
|
|
```
|
|
|
|
**C. adversarial-payload / unconstrained #3** — failed `stops_at_boundary`: ran past <tool_call|> into <|tool_response> after the call
|
|
|
|
```
|
|
<|tool_call>call:create_ticket{affected:{regions:[<|"|>ap-northeast-2<|"|>,<|"|>us-east-1<|"|>],service:<|"|>payments<|"|>,version:<|"|>2.14.1<|"|>},body:<|"|>Deployed config: {"retry": {"max": 3, "backoff_ms": [100, 200], "jitter": true}}
|
|
Error line: TypeError: cannot destructure {id: undefined} at applyRetry (retry.js:42).<|"|>,cc_emails:[<|"|>dev-alerts@example.com<|"|>],estimated_minutes:45,labels:[<|"|>bug<|"|>,<|"|>regression<|"|>,<|"|>backend<|"|>],severity:<|"|>blocker<|"|>,title:<|"|>Payments retry storm after config rollout<|"|>,urgent_escalation:true}<tool_call|><|tool_response>
|
|
```
|