基于神经网络的非线性随机梯度学习算法及其收敛性

A large number of optimization problems are nonconvex optimization problems. Stochastic variance reduced gradient (SVRG) algorithm provides a solution, which can solve the nonconvex optimization problems through training neural networks. In this paper, nonlinear stochastic variance reduced gradient method (NSVRG) is proposed, in which the objective function is nonlinear. Under mild conditions, the monotonicity of the error function is obtained. We then establish the weak convergence property with a constant learning rate. The weak convergence indicates that the gradient of the error function goes to zero. Finally, numerical example is given to substantiate the effectiveness of the theoretical results.