Clean reimplementation of the ICLR 2024 paper "Debiasing Attention Mechanism in Transformer Without Demographics" from 6565_Debiasing_Attention_Mecha.pdf.
Official author repository: https://github.com/lu876/Debiasing-Attention-Mechanism-in-Transformer-without-Demographics
The core idea is implemented as a final-layer transformer attention block that:
- normalizes query and key vectors with running batch statistics,
- applies absolute value before computing attention weights,
- selects the top-attended value vectors from the CLS token attention,
- aligns those local value representations with supervised contrastive loss.
Sensitive attributes are only used for evaluation metrics, not for training.
- Paper-aligned ViT-style model for the vision experiments.
- ERM and debiased-attention training modes.
- CelebA loader for
y = Blond_Hair,a = Male. - UTKFace loader for
y = gender,a = White / not White. - Accuracy, demographic parity, equal opportunity, and equalized odds metrics.
- YAML configs for CelebA, UTKFace, and a synthetic smoke run.
- Tests for fairness metric behavior and config validity.
The paper also evaluates DRO, ARL, KD, JTT, LfF, HateXplain, MultiNLI, and pre-trained BERT/DINO variants. This repo focuses on the main proposed transformer mechanism plus the ERM baseline so the implementation is readable and extensible.
python -m venv .venv
.\.venv\Scripts\Activate.ps1
pip install -r requirements.txt
pip install -e .For CUDA, install the PyTorch build that matches your GPU from the official PyTorch selector, then install the remaining requirements.
The smoke config uses a synthetic image dataset and a tiny transformer. It is intended to verify the training loop, not to reproduce the paper tables.
python scripts/train_vision.py --config configs/smoke_ours.yamlExpected layout:
data/celeba/
img_align_celeba/
000001.jpg
...
list_attr_celeba.txt
list_eval_partition.txt
Run ERM:
python scripts/train_vision.py --config configs/celeba_erm.yamlRun the proposed method:
python scripts/train_vision.py --config configs/celeba_ours.yamlExpected layout:
data/utkface/
UTKFace/
age_gender_race_date.jpg
...
Run:
python scripts/train_vision.py --config configs/utk_ours.yamlAfter running several seed-specific output directories, summarize the final test rows:
python scripts/summarize_metrics.py runs/celeba_ours_seed0 runs/celeba_ours_seed1 runs/celeba_ours_seed2The main vision settings from the paper are:
- image size:
64 x 64 - patch size:
16 - transformer depth:
8 - attention heads:
8 - optimizer:
AdamW - learning rate:
1e-4 - proposed method: apply debiased attention in the last encoder and align the top
2value vectors
Reported proposed-method results:
| Dataset | DP | EOp | EOd | ACC |
|---|---|---|---|---|
| UTKFace | 9.72 | 8.46 | 7.76 | 83.11 |
| CelebA | 16.34 | 38.01 | 20.43 | 94.06 |
The paper reports means and standard deviations over three runs on one NVIDIA RTX 3090. To match those tables, run each config with three seeds and average the resulting runs/*/metrics.csv test rows.