---
status: false
dateread:
title: "Deep Unsupervised Learning using Nonequilibrium Thermodynamics"
year: 2015
authors:

  
  - "Sohl-Dickstein, Jascha"
  
  - "Weiss, Eric A."
  
  - "Maheswaranathan, Niru"
  
  - "Ganguli, Surya"
  

DOI: "10.48550/arXiv.1503.03585"
ISBN: ""
citekey: "sohl-dicksteinDeepUnsupervisedLearning2015a"
itemType: "preprint"







---
#### DOI
[10.48550/arXiv.1503.03585](https://doi.org/10.48550/arXiv.1503.03585)
#### ISBN
[](https://www.isbnsearch.org/isbn/)

### Tags:
#Computer-Science---Machine-Learning, #Quantitative-Biology---Neurons-and-Cognition, #Condensed-Matter---Disordered-Systems-and-Neural-Networks, #Statistics---Machine-Learning

**Contribution**:: 

**Related**:: 

> [!LINK] 
>
>  [arXiv Fulltext PDF](file:///home/danesabo/Zotero/storage/BQZ2D4RD/Sohl-Dickstein%20et%20al.%20-%202015%20-%20Deep%20Unsupervised%20Learning%20using%20Nonequilibrium%20Th.pdf).

> [!Abstract]
>
> A central problem in machine learning involves modeling complex data-sets using highly flexible families of probability distributions in which learning, sampling, inference, and evaluation are still analytically or computationally tractable. Here, we develop an approach that simultaneously achieves both flexibility and tractability. The essential idea, inspired by non-equilibrium statistical physics, is to systematically and slowly destroy structure in a data distribution through an iterative forward diffusion process. We then learn a reverse diffusion process that restores structure in data, yielding a highly flexible and tractable generative model of the data. This approach allows us to rapidly learn, sample from, and evaluate probabilities in deep generative models with thousands of layers or time steps, as well as to compute conditional and posterior probabilities under the learned model. We additionally release an open source reference implementation of the algorithm.
>.
> 
# Notes
>.


# Annotations%% begin annotations %%



### Imported: 2024-08-08 12:07 pm



<mark style="background-color: #ffd400">Quote</mark>
> systematically and slowly destroy structure in a data distribution through an iterative forward diffusion process. We then learn a reverse diffusion process that restores structure in data, yielding a highly flexible and tractable generative model of the data.

<mark style="background-color: #ffd400">Quote</mark>
> robabilistic models suffer from a tradeoff between two conflicting objectives: tractability and flexibility

<mark style="background-color: #ffd400">Quote</mark>
> Our goal is to define a forward (or inference) diffusion process which converts any complex data distribution into a simple, tractable, distribution, and then learn a finite-time reversal of this diffusion process which defines our generative model distribution (See Figure 1).

<mark style="background-color: #e56eee">Quote</mark>
> For both Gaussian and binomial diffusion, for continuous diffusion (limit of small step size β) the reversal of the diffusion process has the identical functional form as the forward process (Feller, 1949).

<mark style="background-color: #ff6666">Quote</mark>
> Since q (x(t)|x(t−1)) is a Gaussian (binomial) distribution, and if βt is small, then q (x(t−1)|x(t)) will also be a Gaussian (binomial) distribution.

<mark style="background-color: #2ea8e5">Quote</mark>
> = ∫ dx(1···T )q ( x(1···T )|x(0)) · p ( x(T )) T ∏ t=1 p (x(t−1)|x(t)) q (x(t)|x(t−1)) . (9)

<mark style="background-color: #ffd400">Quote</mark>
> Thus, the task of estimating a probability distribution has been reduced to the task of performing regression on the functions which set the mean and covariance of a sequence of Gaussians (or set the state flip probability for a sequence of Bernoulli trials).


%% end annotations %%

%% Import Date: 2024-08-08T12:07:42.961-04:00 %%
