Skip to content

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

2 Commits

Folders and files

Repository files navigation

Debiasing Attention Mechanism in Transformer Without Demographics

Clean reimplementation of the ICLR 2024 paper "Debiasing Attention Mechanism in Transformer Without Demographics" from 6565_Debiasing_Attention_Mecha.pdf.

Official author repository: https://github.com/lu876/Debiasing-Attention-Mechanism-in-Transformer-without-Demographics

The core idea is implemented as a final-layer transformer attention block that:

  1. normalizes query and key vectors with running batch statistics,
  2. applies absolute value before computing attention weights,
  3. selects the top-attended value vectors from the CLS token attention,
  4. aligns those local value representations with supervised contrastive loss.

Sensitive attributes are only used for evaluation metrics, not for training.

What Is Included

  • Paper-aligned ViT-style model for the vision experiments.
  • ERM and debiased-attention training modes.
  • CelebA loader for y = Blond_Hair, a = Male.
  • UTKFace loader for y = gender, a = White / not White.
  • Accuracy, demographic parity, equal opportunity, and equalized odds metrics.
  • YAML configs for CelebA, UTKFace, and a synthetic smoke run.
  • Tests for fairness metric behavior and config validity.

The paper also evaluates DRO, ARL, KD, JTT, LfF, HateXplain, MultiNLI, and pre-trained BERT/DINO variants. This repo focuses on the main proposed transformer mechanism plus the ERM baseline so the implementation is readable and extensible.

Setup

python -m venv .venv
.\.venv\Scripts\Activate.ps1
pip install -r requirements.txt
pip install -e .

For CUDA, install the PyTorch build that matches your GPU from the official PyTorch selector, then install the remaining requirements.

Quick Smoke Run

The smoke config uses a synthetic image dataset and a tiny transformer. It is intended to verify the training loop, not to reproduce the paper tables.

python scripts/train_vision.py --config configs/smoke_ours.yaml

CelebA Reproduction

Expected layout:

data/celeba/
  img_align_celeba/
    000001.jpg
    ...
  list_attr_celeba.txt
  list_eval_partition.txt

Run ERM:

python scripts/train_vision.py --config configs/celeba_erm.yaml

Run the proposed method:

python scripts/train_vision.py --config configs/celeba_ours.yaml

UTKFace Reproduction

Expected layout:

data/utkface/
  UTKFace/
    age_gender_race_date.jpg
    ...

Run:

python scripts/train_vision.py --config configs/utk_ours.yaml

Three-Seed Summary

After running several seed-specific output directories, summarize the final test rows:

python scripts/summarize_metrics.py runs/celeba_ours_seed0 runs/celeba_ours_seed1 runs/celeba_ours_seed2

Paper Targets

The main vision settings from the paper are:

  • image size: 64 x 64
  • patch size: 16
  • transformer depth: 8
  • attention heads: 8
  • optimizer: AdamW
  • learning rate: 1e-4
  • proposed method: apply debiased attention in the last encoder and align the top 2 value vectors

Reported proposed-method results:

Dataset DP EOp EOd ACC
UTKFace 9.72 8.46 7.76 83.11
CelebA 16.34 38.01 20.43 94.06

The paper reports means and standard deviations over three runs on one NVIDIA RTX 3090. To match those tables, run each config with three seeds and average the resulting runs/*/metrics.csv test rows.

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages