Statistical Inference for Model Parameters with Stochastic Gradient Descent

Xi Chen
New York University

Wednesday, May 23, 2018
4:15 - 5:15 PM
Location: 370-370


Abstract:

In this talk, we investigate the problem of statistical inference of the true model parameters based on stochastic gradient descent (SGD) with Ruppert-Polyak averaging. To this end, based on the batch-means approach (Glynn & Whitt, 91), we propose a consistent estimator of the asymptotic covariance of the average iterate from SGD, which only uses the iterates from SGD. As the SGD process forms a time-inhomogeneous Markov chain, our batch-means estimator with carefully chosen increasing batch sizes generalizes the classical batch-means estimator designed for time-homogenous Markov chains. The proposed batch-means estimator allows us to construct asymptotically exact confidence intervals and hypothesis tests. We further discuss an extension to conducting inference based on SGD for high-dimensional linear regression. This is a joint work with Jason D. Lee, Xin T. Tong and Yichen Zhang.

---

Operations Research Colloquia: http://or.stanford.edu/oras_seminars.html