Objective Frameworks for Algorithmic Fairness Validation
To trust an AI system, you need to know exactly how it evaluates candidates. The Warden Assurance standard tests AI models using a two-part framework: we check for fairness across broad demographic groups, and we verify that individual candidates are treated consistently.
While standard compliance audits often rely solely on group averages, an AI system can look fair on average while still treating specific individuals unfairly. To provide complete visibility, Warden evaluates systems from two distinct angles. We review the actual outcomes the system has produced in the past, and we run thousands of specially designed test profiles through the system to see exactly how it behaves. This approach objectively measures the AI's true potential for bias.
A good audit requires good data. Relying only on a company's past hiring data limits the test to the types of people who have already applied. To ensure a complete evaluation, Warden uses two types of data:
AI systems learn by finding patterns. Without careful testing, they can repeat past human biases or use harmless-looking information to unfairly guess a candidate's background. Employment laws and emerging AI regulations, such as EEOC guidelines, Title VII, the UK Equality Act, NYC Local Law 144, and the EU AI Act, strictly define which demographic categories must be protected from discrimination.
To help organizations map their systems to these legal requirements, Warden tests models across more than 15 protected categories, including:
To show how we turn these broad legal rules into practical tests, here are three examples of what we look for:
The first part of our framework checks for equality of outcome by looking at how different demographic groups perform on average.
An AI can easily pass this "group average" test if its biases cancel each other out, for example, if it unfairly favors men for one role and women for another. This is why testing individual consistency is just as important.
Because average scores don't tell the whole story, the second part of our framework checks for equality of treatment. This helps us understand exactly how the AI evaluates a single person.
We translate these technical tests into a simple grading system, giving HR and Legal teams a clear view of their compliance risk.
No issues detected. The system achieved an Impact Ratio of 80% or higher, and a Consistency Score of 95% or higher.
Minor issues detected that require a closer look. The system fell slightly below our ideal thresholds (Impact Ratio between 60%-79%, or Consistency Score between 90%-94%).
Definite issues detected that need immediate attention. The system clearly failed the fairness checks (Impact Ratio below 60%, or Consistency Score below 90%).
Select a model below to observe the limits of average-based testing
METHOD 1: GROUPĀ AVERAGE
Impact Ratio
METHOD 2: INDIVIDUAL TREATMENT
Consistency Score
Model 1 Outcome: This model fails both the group average checks and the individual consistency checks. It clearly relies on biased patterns.
METHOD 1: GROUPĀ AVERAGE
Impact Ratio
METHOD 2: INDIVIDUAL TREATMENT
Consistency Score
Model 2 Outcome: This model passes the standard 'group average' test, but fails the individual consistency check. It hides its bias by balancing out unfair decisions across different roles.
METHOD 1: GROUPĀ AVERAGE
Impact Ratio
METHOD 2: INDIVIDUAL TREATMENT
Consistency Score
Model 3 Outcome: This model passes both tests, proving it evaluates candidates fairly at both the group level and the individual level.
Review the outcomes of our Dual-Stream framework directly
Download Sample Audit Report
