> Source: https://www.ema.ai/blog/product/does-ema-recruiter-score-candidates-differently-because-of-who-they-are > Title: Does Ema Recruiter Score Candidates Differently > Description: We changed one demographic detail at a time on 3,984 real candidate profiles and re-scored each version, producing 186,440 scores. Every bias test… # Does Ema Recruiter Score Candidates Differently Because of Who They Are? October 3, 2026, 8 min · Updated on October 5, 2026 Copy Link [Darshan Joshi](https://www.linkedin.com/in/joshidarshan/)[Product Launch](https://www.ema.ai/blog/product) [Candidate Scoring](https://www.ema.ai/blog/tags/candidate-scoring)[Responsible AI](https://www.ema.ai/blog/tags/responsible-ai) As part of one ongoing audit, we changed one demographic detail at a time on real candidates and re-scored them across multiple matched versions of each characteristic — 186,440+ individual scores. Here's exactly what we tested, what we found, and how small an effect this audit is built to catch. Six tests, all clear of the legal floor ## You Can't Compare Two Groups of Real People Real women and real men in a candidate pool differ in experience, seniority and industry — so any gap between them is unattributable. It could be the model, or it could just be who happened to apply. An early cut of this audit that grouped real women against real men found a large gap. It turned out to be entirely a qualification difference, not the scorer. So instead of comparing groups, we hold the person still. We take one real candidate profile, score it, change exactly one demographic detail, and score it again. Everything else — the skills, the employer, the years of experience, the job it's being scored against — stays byte-for-byte identical. Each version is scored multiple times, because the scoring model isn't deterministic. Both versions have a membership line — only the organization's name changes. That matters: if Version A had no line at all, a stable score could just mean “the model ignores membership lines,” not “the model ignores race.” Matching the line's presence and only swapping what it signals is what isolates the demographic detail. If the scorer treats these two identically across enough candidates, the effect is attributable to the model — not to who happened to be in the sample. ## Methodology & Definitions Four terms carry the whole audit. Selection rate and impact ratio come from the counterfactual design above; the four-fifths rule is the regulator's own pass/fail line; the confidence range is why we trust a clean result instead of just reporting it. #### 01 Selection Rate The share of a group scored above the sample median. Every candidate is ranked by score, and the sample's own median becomes the shortlist line. A scorer with no demographic preference puts every group's selection rate near the same number — around 50% when the axis is a clean two-way split. #### 02 Impact Ratio impact ratio(group) = selection rate(group) ÷ selection rate(highest-scoring group) Why we use this metric, specifically: it's not our invention — it's the exact statistic named in the EEOC's own regulations, so a result here is directly comparable to what a regulator, a plaintiff's expert, or another vendor's audit would compute from the same data. It also normalizes away group size and base rate: a group of 289 and a group of 1,566 are still comparable on a 0–100% scale, and the best-scoring group always reads 100% by construction. #### 03 Four-Fifths Rule Impact ratio below 80% is adverse impact. The threshold isn't a target we picked — it's 29 CFR § 1607.4, the EEOC's Uniform Guidelines on Employee Selection Procedures. Any group scoring below four-fifths (80%) of the best group's rate is the line U.S. regulators treat as evidence of discriminatory impact. #### 04 Confidence Range Every impact ratio is a measurement, not a certainty — run the audit again and sampling noise moves the number slightly. Rather than pass a result because the point estimate clears 80%, we require the bottom of its confidence range to clear 80%. On the sex axis, for example, the measured ratio is 95.3%; allowing for sampling luck, the pessimistic end of that is still 93.2% — comfortably clear. ## Every Characteristic Clears the Legal Floor Scores run 1–5. A candidate is counted “selected” if their score lands above the sample median. The impact ratio is a group's selection rate divided by the best-scoring group's — the exact figure U.S. regulators use. Below 80% is the threshold for adverse impact under the four-fifths rule. Every characteristic clears the 80% threshold Race combines two channels — professional-society membership and university attended. Candidate counts are people, not scores: each candidate is scored under every version of their axis (plus repeat passes), which is where the 186,440-score total comes from. ## Every Axis, in Full Each block below is one axis end to end: every version tested, its shortlist rate and impact ratio. The “example wording used” column shows one representative line actually injected for that version — a stand-in for the signal being tested, not the only phrasing that would trigger it. Race tested through societies and universities Religion and disability signals show no penalty Orientation, sex, and age all pass Age has no equivalent test in the literature-grounded run above, so this is its only read so far — a candidate for the next expansion, not a gap we're hiding. ## Why a Clean Result Is Believable, Not Just Absent A test that measures nothing reports no bias every time. Three checks rule that out before we trust the headline numbers. #### 1.8–2.6 pts — It Rejects Worse Candidates Profiles damaged on purpose — genuinely less qualified — scored 1.8 to 2.6 points lower. That's 14–19× the scorer's own inconsistency. The scorer is clearly capable of penalizing something; it just isn't demographics. #### 4.4% — We Measured Its Own Wavering First The exact same untouched profile lands on opposite sides of the shortlist line 4.4% of the time, simply from the model's own non-determinism. Every demographic gap had to beat that noise floor. None came close. #### 0.10 pt — We Know the Smallest Effect We'd Catch We verified a 0.10-point penalty would show up in this design, and an 0.08-point one would not. So “no demographic detail cost a candidate a tenth of a point” is a specific, falsifiable finding — not a claim the effect is exactly zero. ## Grounded in Published Research Rather than invent our own wording, every version borrows the exact language from a study that already showed a human or a model responds to that signal — so a null result here is comparable to a known, cited baseline. ● Race — Wilson & Caliskan, AIES 2024. Gender, Race, and Intersectional Bias in Resume Screening via Language Model Retrieval. Audited 500+ résumés across nine occupations; embedding models favored White-associated names in 85.1% of cases. ● Religion — Wright et al., 2013. Religious affiliation and hiring discrimination in New England, Research in Social Stratification and Mobility. 6,368 employer responses, varying only a religious student-organization line. ● Sexual orientation — Tilcsik, 2011. Pride and Prejudice: Employment Discrimination against Openly Gay Men in the United States, American Journal of Sociology. 1,769 job postings, signalled through a gay campus organization against a matched control. ● Disability — Glazko et al., FAccT '24. Identifying and Improving Disability Bias in GPT-Based Resume Screening. Added a disability-related award, scholarship, panel and membership to a CV; GPT-4 consistently ranked it lower. ● Race × university selectivity — Gaddis, Social Forces 93(4), 2015. Discrimination in the Credential Society, crossed with U.S. Dept. of Education IPEDS institution data (2023). Black candidates from elite universities did only as well as White candidates from less-selective ones. ## The Threshold Is the Regulator's, Not Ours The four-fifths rule comes from 29 CFR § 1607.4, the EEOC's Uniform Guidelines on Employee Selection Procedures: a selection rate below 80% of the best-scoring group's is the line U.S. regulators treat as evidence of adverse impact. We hold ourselves to a stricter version — the bottom of our confidence range has to clear 80%, not just the measured value.