Submit your model

Submission requirements

What a valid submission must include, and what is optional.

Adding a new provider

To benchmark a provider or product that SynthBench doesn't support yet, write a small adapter. The harness handles sampling, parsing, and metrics.

1. 1

   Subclass Provider

   Create a new file in src/synthbench/providers/ and subclass Provider from providers/base.py. Implement async respond() and the name property.

2. 2

   Wire distribution output (optional)

   If your provider returns logprobs or native probabilities, override get_distribution() and set supports_distribution to True. Otherwise the base class samples respond() to build an empirical distribution.

3. 3

   Register in the PROVIDERS dict

   Add a name -> dotted path entry in src/synthbench/providers/__init__.py so the CLI can load it via --provider <your-name>.

4. 4

   Test end-to-end

   Run synthbench run --provider <your-name> --dataset opinionsqa against a small question slice, then submit the result JSON as usual.

from synthbench.providers.base import Provider, Response, PersonaSpec

class MyProvider(Provider):
    @property
    def name(self) -> str:
        return "my-provider"

    async def respond(
        self,
        question: str,
        options: list[str],
        *,
        persona: PersonaSpec | None = None,
    ) -> Response:
        # Call your API, pick an option from `options`
        choice = await call_my_api(question, options, persona)
        return Response(selected_option=choice)

See ollama.py for a compact example (HTTP-backed, sampling-only), or raw_openai.py for a logprob-based adapter that overrides get_distribution().

FAQ

Common questions about running benchmarks and submitting results.