init

README.md

@@ -0,0 +1,65 @@
+# gemma-4 × xgrammar constrained decoding test
+
+A collection of scripts that verify the modified xgrammar (`~/git/xgrammar`,
+`style="gemma"` / `gemma_4` builtin structural tag) works correctly for tool
+calling with `google/gemma-4-E2B-it`, and generate a report comparing
+constrained vs unconstrained behavior.
+
+## Usage (Makefile)
+
+```bash
+make setup     # create uv venv + install deps + local xgrammar editable install (once)
+make run       # run all scenarios → generate report.md + print output  ← for PR reports
+make smoke     # single-tool smoke test (greedy, fast)
+make xgrammar  # reinstall after modifying ~/git/xgrammar
+make clean     # remove venv/report
+```
+
+Options: `make run SAMPLES=8`, `XGRAMMAR_DIR=/path/to/xgrammar make setup`
+
+The output of `make run`, **`report.md`**, is the artifact to show reviewers —
+it contains the model/sampling configuration, a per-scenario table of
+constrained vs unconstrained metrics, and the raw text of failure cases.
+
+## Scenarios (`test_gemma4_scenarios.py`)
+
+| | Setup | What it targets |
+|---|---|---|
+| **A. complex-schema** | Nested objects + array of objects + enum + boolean schema, titles containing quotes | Argument serialization mistakes |
+| **B. multiturn-thinking** | Two prior rounds of tool call/response already in context and thinking is required, the next call takes a number-typed argument | Skipped/unterminated thinking, type errors |
+| **C. adversarial-payload** | A string argument whose body must contain JSON/braces/an error log verbatim, an out-of-enum word ("critical") lure, two similarly named distractor tools, temp 1.5 + top_p 1.0 | String quoting collapse, enum violations |
+
+Metrics (pass/fail per sample):
+
+| metric | meaning |
+|---|---|
+| `well_formed` | Every `<\|tool_call>call:name{...}<tool_call\|>` block is complete and parseable |
+| `valid_name` | Only calls tools that actually exist |
+| `schema_valid` | Parsed arguments pass the JSON schema (parsed via a port of the sglang parser) |
+| `stops_at_boundary` | Generation doesn't run past `<tool_call\|>` into `<\|tool_response>` |
+| `thought_ok` | (thinking scenario) Opens the thought channel and closes it before the tool call |
+
+## Key findings
+
+- gemma-4-E2B-it's format training is very robust, so the tool call argument
+  format itself rarely breaks even at temp 1.5 + top_p 1.0.
+- Per the [Gemma 4 prompt-formatting spec](https://ai.google.dev/gemma/docs/core/prompt-formatting-gemma4),
+  the model is only responsible for generating up to `<tool_call\|>`;
+  `<\|tool_response>` is appended by the application with the real tool
+  result, and is registered as an **additional stop sequence** purely as a
+  backstop. This harness has no such stop configured, so unconstrained runs
+  keep generating past that boundary into engine-owned territory
+  (`stops_at_boundary` failures); required-mode constrained decoding ends the
+  call cleanly at an accept state instead.
+- Unconstrained decoding also **omits the thought channel entirely** in
+  situations where thinking should be enabled; constrained decoding with
+  `reasoning=True` enforces it (an empty thought `<\|channel>thought\n<channel\|>`
+  remains legal, matching the spec's no-thinking form).
+
+## Files
+
+- `test_gemma4_xgrammar.py` — basic smoke test (greedy, single tool, alignment check)
+- `test_gemma4_scenarios.py` — scenario runner + report.md generator.
+  `--scenario a|b|c|all --samples N --temperature T --model ID --report PATH`
+- `gemma_parser.py` — a pure port of sglang's `Gemma4Detector` parsing logic (for output verification)
+- `Makefile` / `report.md`