LLM bias evaluation
An exploratory study of how persona prompts change model responses.
A model can answer the same sensitive question differently when the prompt describes a different person asking it. A single benchmark score can hide that variation.
I built an evaluation pipeline that records responses across persona and system-prompt conditions. The incomplete response collection limits what the analysis can establish.
This page covers a small exploratory run. The larger study with RC2S2, which used 15,552 synthetic personas across five models, is listed under publications.
The experiment holds the topic fixed
The design uses four personas, three models, and two system-prompt conditions for one topic. One condition uses the baseline prompt; the other requests a neutral style.
Comparisons within the same model and prompt condition help separate persona effects from other setup differences. Neutral-style prompts provide a way to investigate changes in wording.
Raw JSONL records retain the run identifier, prompt identifier, persona, model, and repeat index. Those records make the collected outputs available for reanalysis.
Missing responses limit the comparison
In the documented collection, 426 of 1,152 attempted runs returned text. The remaining 726 returned empty responses.
| Model | Responses |
|---|---|
llama-3.1-8b
|
192 |
qwen-2.5-7b
|
191 |
mistral-small-24b
|
43 |
The models are represented unevenly. Pooling their responses can mix persona effects with differences in model coverage.
The missing responses can also bias the sample. Their cause needs investigation before the aggregate results support a stronger conclusion.
Response length is only an exploratory measure
Mean response length varies across the collected persona groups. Length does not distinguish bias from detail, qualification, or a change in writing style.
The experiment covers one topic and one prompt family. It cannot establish how the models respond to other sensitive subjects.
The useful output so far is the experimental setup and recorded dataset. A reliable bias claim requires better completion coverage and measures that evaluate response content.