Thank you for releasing SkillLearnBench — the task design and multi-level evaluation framework are very valuable for studying continual skill learning.
It would be great to add support for generic OpenAI-compatible API endpoints, especially vLLM. This would make it possible to evaluate open-weight models such as Qwen3.6-35B-A3B as both task executors and skill-generation models.
Currently, the provided agent paths are mainly designed around Claude Code, Gemini CLI, and a Codex/Azure-style configuration. A lightweight OpenAI-compatible agent adapter could expose configuration such as:
OPENAI_BASE_URL=http://<vllm-host>:<port>/v1
OPENAI_API_KEY=<key>
MODEL=Qwen/Qwen3.6-35B-A3B
Ideally, it would support:
- OpenAI-compatible
/v1/chat/completions endpoints
- configurable model name, base URL, API key, temperature, and max tokens
- tool/function calling where supported by the serving backend
- trajectory logging and token-usage extraction consistent with the existing evaluation outputs
- use in both skill generation and downstream task execution
This would substantially broaden the benchmark’s applicability: researchers could compare proprietary and open-weight models under the same skill-learning protocol, reproduce results on local infrastructure, and study model-scale / model-family effects without relying exclusively on hosted APIs.
I would be happy to help test a vLLM/Qwen integration if such support is added.
Thank you for releasing SkillLearnBench — the task design and multi-level evaluation framework are very valuable for studying continual skill learning.
It would be great to add support for generic OpenAI-compatible API endpoints, especially vLLM. This would make it possible to evaluate open-weight models such as Qwen3.6-35B-A3B as both task executors and skill-generation models.
Currently, the provided agent paths are mainly designed around Claude Code, Gemini CLI, and a Codex/Azure-style configuration. A lightweight OpenAI-compatible agent adapter could expose configuration such as:
Ideally, it would support:
/v1/chat/completionsendpointsThis would substantially broaden the benchmark’s applicability: researchers could compare proprietary and open-weight models under the same skill-learning protocol, reproduce results on local infrastructure, and study model-scale / model-family effects without relying exclusively on hosted APIs.
I would be happy to help test a vLLM/Qwen integration if such support is added.