PIU: Proximity-guided Identity Unlearning in ID-Conditioned Diffusion Models

Identity unlearningDiffusion modelsArc2FaceArcFaceCross-attention

Protects a chosen identity in ID-conditioned diffusion models by re-teaching what its identity conditioning means.

The setting

FaceArcFace embeddingArc2FaceNew photos of that person

ArcFace identity space · schematic

Anchor = a nearby identity (cosine similarity)

Target conditioning → generates the anchor

Every other identity → unchanged

4.29%

Identity-sensitive cross-attention layers only

Loss · from the paper

{\hat{ε}}_{forget} = ε_{θ^{*}} (z_{t}, t, c_{a}) - η [ε_{θ^{*}} (z_{t}, t, c_{f}) - ε_{θ^{*}} (z_{t}, t, c_{a})]

ℒ_{forget} = 𝔼 ‖ ε_{θ} (z_{t}, t, c_{f}) - {\hat{ε}}_{forget} ‖_{2}^{2}

ℒ_{preserve} = 𝔼 ‖ ε_{θ} (z_{t_{r}}, t_{r}, c_{r}) - ε_{θ^{*}} (z_{t_{r}}, t_{r}, c_{r}) ‖_{2}^{2}

ℒ = ℒ_{forget} + λ ℒ_{preserve}

θ^{*}

frozen Arc2Face

θ

model being trained

c_{f}

target ID

c_{a}

anchor ID

c_{r}

retained IDs

Arc2FaceCelebA-HQ · 50 target identitiesvs SISS · UCE · WID

ISM: identity similarity. Low on the target means it’s forgotten, high on everyone else means they’re kept.
SRK: one score for removal and keep together.
KD: kernel distance, an image-quality metric. ΔKD is how much it changes on the kept identities after unlearning, so lower is better.

Identity similarity · forget ↓ vs retain ↑

SRK ↑ · removal + keep

ΔKD on retained ↓ · kernel distance (image quality)