wanhae.lee
66 lines
3.5 KiB
Markdown
66 lines
3.5 KiB
Markdown
# gemma-4 × xgrammar constrained decoding test
|
||
|
||
A collection of scripts that verify the modified xgrammar (`~/git/xgrammar`,
|
||
`style="gemma"` / `gemma_4` builtin structural tag) works correctly for tool
|
||
calling with `google/gemma-4-E2B-it`, and generate a report comparing
|
||
constrained vs unconstrained behavior.
|
||
|
||
## Usage (Makefile)
|
||
|
||
```bash
|
||
make setup # create uv venv + install deps + local xgrammar editable install (once)
|
||
make run # run all scenarios → generate report.md + print output ← for PR reports
|
||
make smoke # single-tool smoke test (greedy, fast)
|
||
make xgrammar # reinstall after modifying ~/git/xgrammar
|
||
make clean # remove venv/report
|
||
```
|
||
|
||
Options: `make run SAMPLES=8`, `XGRAMMAR_DIR=/path/to/xgrammar make setup`
|
||
|
||
The output of `make run`, **`report.md`**, is the artifact to show reviewers —
|
||
it contains the model/sampling configuration, a per-scenario table of
|
||
constrained vs unconstrained metrics, and the raw text of failure cases.
|
||
|
||
## Scenarios (`test_gemma4_scenarios.py`)
|
||
|
||
| | Setup | What it targets |
|
||
|---|---|---|
|
||
| **A. complex-schema** | Nested objects + array of objects + enum + boolean schema, titles containing quotes | Argument serialization mistakes |
|
||
| **B. multiturn-thinking** | Two prior rounds of tool call/response already in context and thinking is required, the next call takes a number-typed argument | Skipped/unterminated thinking, type errors |
|
||
| **C. adversarial-payload** | A string argument whose body must contain JSON/braces/an error log verbatim, an out-of-enum word ("critical") lure, two similarly named distractor tools, temp 1.5 + top_p 1.0 | String quoting collapse, enum violations |
|
||
|
||
Metrics (pass/fail per sample):
|
||
|
||
| metric | meaning |
|
||
|---|---|
|
||
| `well_formed` | Every `<\|tool_call>call:name{...}<tool_call\|>` block is complete and parseable |
|
||
| `valid_name` | Only calls tools that actually exist |
|
||
| `schema_valid` | Parsed arguments pass the JSON schema (parsed via a port of the sglang parser) |
|
||
| `stops_at_boundary` | Generation doesn't run past `<tool_call\|>` into `<\|tool_response>` |
|
||
| `thought_ok` | (thinking scenario) Opens the thought channel and closes it before the tool call |
|
||
|
||
## Key findings
|
||
|
||
- gemma-4-E2B-it's format training is very robust, so the tool call argument
|
||
format itself rarely breaks even at temp 1.5 + top_p 1.0.
|
||
- Per the [Gemma 4 prompt-formatting spec](https://ai.google.dev/gemma/docs/core/prompt-formatting-gemma4),
|
||
the model is only responsible for generating up to `<tool_call\|>`;
|
||
`<\|tool_response>` is appended by the application with the real tool
|
||
result, and is registered as an **additional stop sequence** purely as a
|
||
backstop. This harness has no such stop configured, so unconstrained runs
|
||
keep generating past that boundary into engine-owned territory
|
||
(`stops_at_boundary` failures); required-mode constrained decoding ends the
|
||
call cleanly at an accept state instead.
|
||
- Unconstrained decoding also **omits the thought channel entirely** in
|
||
situations where thinking should be enabled; constrained decoding with
|
||
`reasoning=True` enforces it (an empty thought `<\|channel>thought\n<channel\|>`
|
||
remains legal, matching the spec's no-thinking form).
|
||
|
||
## Files
|
||
|
||
- `test_gemma4_xgrammar.py` — basic smoke test (greedy, single tool, alignment check)
|
||
- `test_gemma4_scenarios.py` — scenario runner + report.md generator.
|
||
`--scenario a|b|c|all --samples N --temperature T --model ID --report PATH`
|
||
- `gemma_parser.py` — a pure port of sglang's `Gemma4Detector` parsing logic (for output verification)
|
||
- `Makefile` / `report.md`
|