As Large Language Models (LLMs) become primary information intermediaries in non-English contexts, assessing their political neutrality in lower-resource languages like Slovenian remains critical, yet evaluations often rely on single-pass questionnaires that conflate ideology with measurement artifacts. We present the first systematicstudy of political bias and steerability in Slovenian LLMs, evaluating the Gemma, Qwen, GaMS and Zlatorog families on the Political Compass Test (PCT). Using a Latin Hyper-cube Sampling framework across 300 configurations per model and ideological condition, we generate stable, distribution-averaged estimates that are decomposable byfactor. Every model default to the Libertarian-Left. Analysis of “semantic plasticity” — the ability to emulate assigned personas — reveals that while newer iterations of global models are increasingly steerable, smaller and natively adapted models (GaMS, Zlatorog) exhibit behaviour consistent with functional ignorance. This is confirmed by the consistency analysis, showing that for five of the fourteen evaluated models the responses along the economic axis are not far from random. Consequently, their near-center scoring cannot be read as a genuinely centrist political stance. Finally, we show that a single-pass administration of the PCT is an unreliable estimator of political position, given the structural asymmetry of the instrument and the variance introduced by instruction phrasing and other factors.
Authors: Luka Debevc, Matej Martinc; Institut “Jožef Stefan”
The paper was presented at the conference: LANGUAGE TECHNOLOGIES & DIGITAL HUMANITIES CONFERENCE 2026
Read the whole article here: https://zenodo.org/records/22765800
