How to make gemma 4 works with SWEBench?

I wanted to compare gemma 4 31b qat q4 and qwen3.8 27b iq4_nl, so I run SWEBench verfied on them. The original gemma 4 chat template doesn't work with preserve reasoning, so google released an update such that it can work with agentic coding: https://huggingface.co/google/gemma-4-31B-it/blob/main/chat_template.jinja However, when I run it through SWEBench verified, I am getting all LimitExceeded errors while I could get 388/500 Submitted with 333/388 resolved with Qwen3.8 27b iq4_nl. I tried other chat templates floating around in the internet: https://gist.github.com/jscott3201/ad69c4ffbd79f18b11a0f6a94c94fadf https://gist.github.com/bbrowning/c584eb2dbd79e4cc9ecedf92eee2d135 https://huggingface.co/stevelikesrhino/gemma-4-31B-it-nvfp4-GGUF/blob/main/gemma4-improved.jinja https://github.com/vllm-project/vllm/blob/main/examples/tool_chat_template_gemma4.jinja https://huggingface.co/google/gemma-4-31B-it/discussions/137 https://huggingface.co/google/gemma-4-31B-it/discussions/135 https://huggingface.co/ManniX-ITA/gemma-4-A4B-98e-v7-coder-it/blob/main/chat_template.jinja But they all have the same problem. This is my llama-server command: ./llama-server -m ~/gguf/gemma-4-31B_q4_0-it.gguf --jinja --chat-template-file chat_template.jinja -ngl 999 -fa 1 --load-mode mmap -np 1 --temp 1.0 --top_p 0.95 --top_k 64 --min_p 0.0 --presence-penalty 0.0 --repeat-penalty 1.0 -c 122880 -ctk q8_0 -ctv q8_0 --spec-type none --reasoning-preserve --host 192.168.1.140 --port 8080 --no-mmproj-offload --mmproj ~/gguf/gemma-4-31B-it-mmproj.gguf The ggufs I used was the QAT Q4 from google. I also tried unsloth's QAT but it also gives me the same errors. This is a command to run only four tests: swebench infer verified -c config.yaml -r gemma-4-31b-qat-q4_0-120k-q8_0 -m gemma-4-31b --filter "astropy__astropy-12907|astropy__astropy-13033|astropy__astropy-13977|astropy__astropy-13579" config.yaml model: cost_tracking: "ignore_errors" model_name: "my-local-model" model_kwargs: custom_llm_provider: "openai" api_base: "http://192.168.1.140:8080/v1" environment: environment_class: "local" Any clue as to how to make it work? submitted by /u/Ok_Warning2146 [link] [comments]

Source: r/LocalLLaMA | 2026-10-10

Related