SEMINAR:Statistical Mechanics of Kernel Regression and Wide Neural...

Guest: Abdülkadir Canatar, Flatiron Institute

Title: Statistical Mechanics of Kernel Regression and Wide Neural Networks (CS, EE, PHYS, MATH)

Date/Time: December 4, 2024, 13:40

Location: https://sabanciuniv.zoom.us/j/96245743218, Meeting ID: 962 4574 3218

Abstract: A theoretical understanding of generalization remains an open problem for many machine learning models, including deep networks where overparameterization leads to better performance. Here, we study this problem for kernel regression, which, besides being a popular machine learning method, also describes infinitely overparameterized neural networks. We develop an analytical theory of generalization in kernel regression using the replica method of statistical mechanics. This theory is applicable to any kernel and data distribution. Experiments with practical kernels, including those arising from wide neural networks, show perfect agreement with our theory. Further, our theory accurately predicts the generalization performance of neural networks with modest widths. We provide an in-depth analysis of our analytical expression for kernel generalization. We show that kernel machines employ an inductive bias towards simple functions, preventing them from overfitting the data. We characterize whether a kernel is compatible with a learning task in terms of sample efficiency. We identify a first-order phase transition in our theory where more data may impair generalization when the task is noisy or not expressible by the kernel. Finally, we extend these results to out-of-distribution generalization.

Bio: Abdulkadir Canatar is a research fellow at Flatiron Institute’s Center for Computational Neuroscience (CCN). His research focuses on the geometrical and spectral analysis of neural representations, generalization, and kernel methods in deep learning. He received his Ph.D. in physics from Harvard University and his M.Sc. (Physics) and B.Sc. (Electronics Engineering) from Sabanci University.