This webpage has been set-up as an on-line appendix to the
following works:
Mark Schmidt, Glenn Fung, Romer Rosales. Fast
Optimization Methods for L1 Regularization: A Comparative Study and Two New
Approaches. European Conference on Machine Learning
(ECML), 2007 (pdf).
Mark Schmidt, Glenn Fung, Romer Rosales. Fast L1 Regularization: Current and
New Optimization Algorithms. Submitted.
The Matlab
code for the optimization algorithms used to produce the results presented in
the conference paper can be downloaded here.
The Matlab code for the optimization algorithms used
to produce the results presented in the extended paper submission can be
downloaded here.
The code related to the extended submission has several minor improvements on
the methods present in the conference paper, but also contains 2 additional
methods and first-order variants of all methods that do not require explicit
Hessian calculation (in most cases these are Quasi-Newton methods, and they are
accessible in most methods by setting the 'order' parameter to 1). We have also
included an L-BFGS version of the ProjectionL1 method that we have used to
solve problems with a very large number of variables.
In both cases, we have included an example of using the optimization
algorithms. The following steps are identical for the conference and extended
papers, and give a demonstration of running the different methods to optimize
the Logistic Regression negative log-likelihood on the UCI Ionosphere data
subject to L1-regularization (with the regularization scale fixed at 50).
· unzip the downloaded file (either L1General.zip or L1General2.zip)
· start Matlab
· >> cd L1General % go to the newly created directory
· >> addpath(genpath(pwd)) % adds the needed functions to the Matlab path
· >> L1General_example % load the data, set up the loss, and run the optimizers
To accompany the conference paper, we have prepared the following documents: