copt.minimize_svrg¶
-
copt.
minimize_svrg
(f_deriv, A, b, x0, step_size, alpha=0, prox=None, max_iter=500, tol=1e-06, verbose=False, callback=None)[source]¶ Stochastic average gradient augmented (SAGA) algorithm.
The SAGA algorithm can solve optimization problems of the form
argmin_{x in R^p} sum_{i}^n_samples f(A_i^T x, b_i) + alpha * ||x||_2^2 +
beta * ||x||_1
- Parameters
f_deriv – derivative of f
x0 (np.ndarray or None, optional) – Starting point for optimization.
step_size (float or None, optional) – Step size for the optimization. If None is given, this will be estimated from the function f.
n_jobs (int) – Number of threads to use in the optimization. A number higher than 1 will use the Asynchronous SAGA optimization method described in [Pedregosa et al., 2017]
max_iter (int) – Maximum number of passes through the data in the optimization.
tol (float) – Tolerance criterion. The algorithm will stop whenever the norm of the gradient mapping (generalization of the gradient for nonsmooth optimization) is below tol.
verbose (bool) – Verbosity level. True might print some messages.
trace (bool) – Whether to trace convergence of the function, useful for plotting and/or debugging. If ye, the result will have extra members trace_func, trace_time.
- Returns
opt – The optimization result represented as a
scipy.optimize.OptimizeResult
object. Important attributes are:x
the solution array,success
a Boolean flag indicating if the optimizer exited successfully andmessage
which describes the cause of the termination. See scipy.optimize.OptimizeResult for a description of other attributes.- Return type
OptimizeResult
References
The SAGA algorithm was originally described in
Aaron Defazio, Francis Bach, and Simon Lacoste-Julien. SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in Neural Information Processing Systems. 2014.
The implemented has some improvements with respect to the original, like support for sparse datasets and is described in
Fabian Pedregosa, Remi Leblond, and Simon Lacoste-Julien. “Breaking the Nonsmooth Barrier: A Scalable Parallel Method for Composite Optimization.” Advances in Neural Information Processing Systems (NIPS) 2017.