Datasets

The surveys SynthBench scores against

Name: SynthBench Leaderboard
Creator: SynthBench
License: https://opensource.org/licenses/MIT

The human survey datasets behind every score, with the source, coverage, demographics, and redistribution policy for each.

Registered datasets

Nine datasets are registered. Four are scored and make up the SynthBench Index (In the Index). NTIA is scored for reference only: its questions are yes/no facts about internet use, not opinions (Reference only). The other four have adapters but no published runs yet (Not scored yet).

Each card also shows the dataset's redistribution tier. full datasets ship per-question distributions publicly. gated datasets require a research-use sign-in for per-question data.

OpinionsQA

In the Index gated

The primary ground-truth dataset. Multiple-choice questions covering political, social, and economic attitudes from nationally representative US surveys.

Questions: 1,498 (200 core)
Source: Pew American Trends Panel
Scope: US population
Coverage: 2017–2022 (15 survey waves)
Demographics: 56 groups across 11 attributes

GlobalOpinionQA

In the Index gated

Cross-national attitudes from Pew Global Attitudes surveys. Tests whether models capture how opinion differs between countries, beyond US demographics.

Questions: 2,556
Source: Pew Global Attitudes
Scope: 138 countries
Coverage: Cross-national
Demographics: Country-level cultural variation

SubPOP

In the Index gated

Extended dataset focused on demographic subgroup variation. Tests whether models can differentiate between fine-grained population segments.

Questions: 3,362
Source: 22 US subpopulations
Scope: US demographic subgroups
Coverage: Cross-sectional
Demographics: 11 attributes: age, education, gender, ideology, party, income, religion, attendance, region, marital status, citizenship

GSS

In the Index full

General Social Survey cumulative microdata from NORC. It is the only public-domain adapter that ships per-respondent rows, which allows convergence curves from real sampling alongside the bootstrap baseline.

Source: NORC General Social Survey
Scope: US population
Coverage: 1972–present (cumulative file)
Demographics: Individual-level microdata (microdata-capable)

NTIA Internet Use Supplement

Reference only full

NTIA Computer & Internet Use Supplement to the Current Population Survey. U.S. Government public-domain work under 17 USC 105; per-question distributions ship openly.

Source: US Census / NTIA CPS Supplement
Scope: US households
Coverage: Biennial waves, 1994–present
Demographics: Standard CPS demographics

WVS (World Values Survey)

Not scored yet gated

World Values Survey Wave 7. Cross-cultural value orientations across national populations. Research-use license. Per-question artifacts are gated behind JWT auth.

Source: World Values Survey Association
Scope: Cross-national (Wave 7, 2017–2022)
Coverage: Wave 7 (2017–2022)
Demographics: Country + standard demographic cells

Eurobarometer

Not scored yet gated

Eurobarometer Consumer Modules. EU-wide consumer conditions, digital services, and sustainability attitudes. GESIS restricts redistribution, so per-question data is served from a gated origin.

Source: European Commission / GESIS
Scope: EU member states
Coverage: Quarterly (Standard / Flash / Special)
Demographics: Country + EU-harmonized demographic cells

Michigan (Surveys of Consumers)

Not scored yet gated

University of Michigan Surveys of Consumers: consumer sentiment, inflation expectations, and household finance. Academic use terms. Per-question data is gated.

Source: University of Michigan SRC
Scope: US consumers
Coverage: Monthly cadence
Demographics: Standard consumer-survey demographics

Pew Technology

Not scored yet gated

Pew Research Internet & Technology waves: tech adoption, digital privacy, social media, and AI attitudes. Pew redistribution terms. Per-question data is gated.

Source: Pew Research Center Internet & Technology
Scope: US population (American Trends Panel)
Coverage: 10 tech waves (W86–W113)
Demographics: ATP demographic weights

Dataset redistribution policy

Each upstream dataset has its own license terms. SynthBench applies them at publish time using four tiers:

full: per-question distributions are published openly (public-domain datasets).
gated: per-question distributions are served from a Cloudflare Worker that validates a Supabase JWT, so research-use sign-in is required. No distribution is served anonymously.
aggregates_only: aggregate scores are public, and no per-question artifact ships at all. Gating is not a substitute for missing redistribution permission.
citation_only: question metadata only. Every downstream metric is suppressed.

Aggregate metrics (SPS, JSD, ranks, cross-model parity) are derived quantities and remain public regardless of tier.

Per-dataset redistribution policy tier and provenance, used by synthbench publish to gate which fields appear in public artifacts.
Dataset	Tier	Source & license
eurobarometer	gated	European Commission / GESIS — Eurobarometer Consumer Modules (license)
globalopinionqa	gated	Durmus et al. 2023, Anthropic — llm_global_opinions (license)
gss	full	NORC at the University of Chicago — General Social Survey (license)
michigan	gated	Survey Research Center, University of Michigan — Surveys of Consumers (license)
ntia	full	NTIA Internet Use Survey (U.S. Government work, 17 USC 105) (license)
opinionsqa	gated	Santurkar et al., ICML 2023 — Whose Opinions Do LLMs Reflect? (derived from Pew American Trends Panel) (license)
pewtech	gated	Pew Research Center — Internet & Technology (American Trends Panel) (license)
subpop	gated	Suh et al., ACL 2025 — SubPOP: Subpopulation-Level Opinion Prediction (license)
wvs	gated	World Values Survey Association — WVS Wave 7 (2017-2022) (license)

The methodology page explains how these datasets are scored: metrics, baselines, the private holdout split, and anti-gaming defences.