Machine learning theory and its relations to statistical physics

6 months ago by
Recently, I came across this article:
“New Theory Cracks Open the Black Box of Deep Learning”
in Quanta Magazine.

The reference there, by Tishby et al., is interesting for me since it seems to be using well-known ideas from statistical physics -- at a first naïve glance. The cited paper by the physicists Schwab and Mehta is also eye-catching: They discovered that a certain deep-learning algorithm works, in a particular case, exactly like renormalization. It was a stunning indication that “extracting relevant features in the context of statistical physics and extracting relevant features in the context of deep learning are not just similar words, they are one and the same.”

So, I added this post to open up a discussion on the theoretical aspects of machine learning and its relations to statistical physics.
It should be possible to open the black box of neural networks with theoretical physics, esp. statistical physics.

#machineLearning #statisticalPhysics
The link to the Schwab-Mehta paper is here.

written 6 months ago by Tim  
Interesting to read an article that actually tries to bridge the gap between statistical physics and mathematical statistics. It is sort of telling that they have to translate the physics phrase "interactions" into plain "statistical correlations" in order to be understood by non-physicists... The "fancy-speak" of physics is problematic, in my view. Another example is "integrating out = marginalizing over".

I find that I can understand the section on RBMs much better than that on RG...

How very nice: "These results also provide a natural interpretation for variational RG entirely in the language of probability theory." - Can we please get this for all of physics?
written 6 months ago by Tim  
I believe there is a real need for "translating" between machine-learning theory and usual concepts of statistical physics. Actually, sometimes machine-learning view gives much deeper an insight, based on pure statistics — like the example of "marginalisation" vs "RG".
written 6 months ago by AlQuemist  
Please login to add an answer/comment or follow this question.

Similar posts:
Search »