Acceleration of Frank-Wolfe Algorithms with Open-Loop Step-Sizes
Frank-Wolfe algorithms (FW) are popular first-order methods for solving constrained convex optimization problems that rely on a linear minimization oracle instead of potentially expensive projection-like oracles. Many works have identified accelerated convergence rates under various structural assum...
Saved in:
Main Authors | , , |
---|---|
Format | Journal Article |
Language | English |
Published |
25.05.2022
|
Subjects | |
Online Access | Get full text |
Cover
Loading…
Summary: | Frank-Wolfe algorithms (FW) are popular first-order methods for solving
constrained convex optimization problems that rely on a linear minimization
oracle instead of potentially expensive projection-like oracles. Many works
have identified accelerated convergence rates under various structural
assumptions on the optimization problem and for specific FW variants when using
line-search or short-step, requiring feedback from the objective function.
Little is known about accelerated convergence regimes when utilizing open-loop
step-size rules, a.k.a. FW with pre-determined step-sizes, which are
algorithmically extremely simple and stable. Not only is FW with open-loop
step-size rules not always subject to the same convergence rate lower bounds as
FW with line-search or short-step, but in some specific cases, such as kernel
herding in infinite dimensions, it has been empirically observed that FW with
open-loop step-size rules enjoys to faster convergence rates than FW with
line-search or short-step. We propose a partial answer to this unexplained
phenomenon in kernel herding, characterize a general setting for which FW with
open-loop step-size rules converges non-asymptotically faster than with
line-search or short-step, and derive several accelerated convergence results
for FW with open-loop step-size rules. Finally, we demonstrate that FW with
open-loop step-sizes can compete with momentum-based open-loop FW variants. |
---|---|
DOI: | 10.48550/arxiv.2205.12838 |