On the Inherent Privacy of Zeroth Order Projected Gradient Descent
On the Inherent Privacy of Zeroth Order Projected Gradient Descent
Differentially private zeroth-order optimization methods have recently gained popularity in private fine tuning of machine learning models due to their reduced memory requirements. Current approaches for privatizing zeroth-order methods rely on adding Gaussian noise to the estimated zeroth-order gradients. However, since the search direction in the zeroth-order methods is inherently random, researchers including Tang et al. (2024) and Zhang et al. (2024a) have raised an important question: is the inherent noise in zeroth-order estimators sufficient to ensure the overall differential privacy of the algorithm? This work settles this question for a class of oracle-based optimization algorithms where the oracle returns zeroth-order gradient estimates. In particular, we show that for a fixed initialization, there exist strongly convex objective functions such that running (Projected) Zeroth-Order Gradient Descent (ZO-GD) is not differentially private. Furthermore, we show that even with random initialization and without revealing (initial and) intermediate iterates, the privacy loss in ZO-GD can grow superlinearly with the number of iterations when minimizing convex objective functions.
Devansh Gupta、Meisam Razaviyayn、Vatsal Sharan
计算技术、计算机技术
Devansh Gupta,Meisam Razaviyayn,Vatsal Sharan.On the Inherent Privacy of Zeroth Order Projected Gradient Descent[EB/OL].(2025-07-09)[2025-07-24].https://arxiv.org/abs/2507.05610.点此复制
评论