AuditLens
Statistical dataset checks before model training.
Uneven representation, missing values, and different outcome rates can affect a model before training begins. Those patterns need inspection in the dataset itself.
I built AuditLens to report statistical disparities with the affected columns, measured values, group sizes, and severity.
Statistics and model interpretation are separate
The first layer checks class imbalance, differential missingness, sensitive-attribute correlations, and subgroup outcome gaps. These checks do not call a language model.
An optional second layer interprets the findings for a prediction task. Reports then present the findings in severity order.
Separating those steps keeps the statistical output available when model interpretation is disabled. The tool provides a Python library, a FastAPI service, and a Streamlit interface.
Group size changes how a rate should be read
The documented COMPAS example covers 7,214 rows. It uses two_year_recid as the target and treats race and sex as sensitive attributes.
| Group | Rate | Rows |
|---|---|---|
| Native American | 0.556 | 18 |
| African-American | 0.514 | 3,696 |
| Caucasian | 0.394 | 2,454 |
| Hispanic | 0.364 | 637 |
| Other | 0.353 | 377 |
| Asian | 0.281 | 32 |
The reported parity gap is 0.274. Its endpoints come from the two smallest groups, with 18 and 32 rows. No confidence intervals accompany the estimate.
That sample-size difference matters. A rate alone hides how much evidence supports the comparison.
Missingness was the more common high-severity finding
The same example reports 29 findings, including eight marked high severity. Seven of those eight concern differences in missingness across groups.
Missingness gaps were 0.267 on four columns and 0.179 on three more. Outcome rates would not show those differences.
A disparity is a question, not a verdict
Statistical differences do not prove unfairness. Interpretation requires the collection context and the intended use of the data.
AuditLens inspects datasets rather than trained models. One dataset and one collection of measures cannot establish a general fairness assessment.
The optional model interpretation can also vary between runs, even when the underlying statistical findings stay the same.