One DP-Adam step — morie_dp

Adam's moment updates applied to the privatised gradient. The moments are post-processing and cost nothing extra.

Usage

morie_dp_adam(
  grads,
  C = 1,
  sigma = 1,
  lr = 0.001,
  betas = c(0.9, 0.999),
  eps = 1e-08,
  state = NULL,
  seed = NULL
)

Arguments

grads: per-example gradient matrix, shape (B, p).
C: L2 clipping norm.
sigma: noise multiplier; the noise sd on the sum is sigma * C.
lr: learning rate.
betas: moment decay rates.
eps: denominator stabiliser.
state: optional state list from a previous call.
seed: optional integer seed.

Value

list with update, state, m, v, signal_to_noise.

Details

They can still hurt. Adam's second moment estimates the gradient scale, and under heavy DP noise what it actually tracks is the NOISE scale – the adaptive step then shrinks in the directions where the noise happens to be large rather than where the signal is small. signal_to_noise below 1 is the regime where plain DP-SGD is often the better optimiser, and the function says so.

References

Kingma, D. P. and Ba, J. (2015). Adam: a method for stochastic optimization. ICLR. Abadi, M. et al. (2016). Deep learning with differential privacy. CCS, 308-318.

Examples

set.seed(1)
morie_dp_adam(matrix(rnorm(80), ncol = 4), C = 1, sigma = 0.1)$t
#> [1] 1