Derivation of the Boltzmann Distribution By Maximizing Entropy

Assume there are $N$ discrete states the system can be in: $X \in {x_{1} \dots x_{N}}$ . Let $p_{i} = P (X = x_{i})$ sampling the state over an indefinite time period, and $ϵ_{i}$ be the energy in state $x_{i}$ . The system will have 2 contraints, the probabilities must add to 1, and the average energy should equal $\overline{E}$ :

\sum_{i = 1}^{N} p_{i} = 1

and

\sum_{i = 1}^{N} p_{i} ϵ_{i} = \overline{E}

We will assume that $S$ should be maximized.

S (p) = - k_{B} \sum_{i = 1}^{N} p_{i} \ln (p_{i})

We want to find the probability distribution $p = p_{1} \dots p_{n}$ that maximizes $S$ , while following the 2 constraints.

p_{max} = {argmax}_{P} S (p)

We can solve this system of equations with a Lagrange multiplier. First rewrite the 2 constraints like:

\sum_{i = 1}^{N} p_{i} - 1 = 0

and

\sum_{i = 1}^{N} p_{i} ϵ_{i} - \overline{E} = 0

Then make the Lagrange multiplier equation

L (p, λ_{1}, λ_{2}) = - k_{B} \sum_{i = 1}^{N} p_{i} \ln (p_{i}) + λ_{1} (\sum_{i = 1}^{N} p_{i} - 1) + λ_{2} (\sum_{i = 1}^{N} p_{i} ϵ_{i} - \overline{E})

At the maximum the derivative with respect to any $p_{i}$ should be zero.

\frac{\partial L (p, λ_{1}, λ_{2})}{\partial p_{i}} = - k_{B} (\ln (p_{i}) - p_{i} \frac{1}{p_{i}}) + λ_{1} + λ_{2} ϵ_{i}

Set the derivative to zero.

0 = - k_{B} (\ln (p_{i}) - p_{i} \frac{1}{p_{i}}) + λ_{1} + λ_{2} ϵ_{i}

Then Solve for $p_{i}$ .

\ln (p_{i}) = \frac{- k_{B} + λ_{1} + λ_{2} ϵ_{i}}{k_{B}}

Exponentiate both sides

p_{i} = e^{\frac{- k_{B} + λ_{1} + λ_{2} ϵ_{i}}{k_{B}}}

Then split the exponent.

p_{i} = e^{- 1 + \frac{λ_{1}}{k_{B}}} e^{\frac{λ_{2} ϵ_{i}}{k_{B}}}

Now we have to solve for the 2 multipliers. Start with the first constraint:

\sum_{i = 1}^{N} p_{i} = 1

And substitute $p_{i}$ .

\sum_{i = 1}^{N} e^{- 1 + \frac{λ_{1}}{k_{B}}} e^{\frac{λ_{2} ϵ_{i}}{k_{B}}} = 1

The first term is independent of $i$ so it can be moved out of the sum.

e^{- 1 + \frac{λ_{1}}{k_{B}}} \sum_{i = 1}^{N} e^{\frac{λ_{2} ϵ_{i}}{k_{B}}} = 1

Then divide both sides by the sum.

e^{- 1 + \frac{λ_{1}}{k_{B}}} = \frac{1}{\sum_{i = 1}^{N} e^{\frac{λ_{2} ϵ_{i}}{k_{B}}}}

Now $e^{- 1 + \frac{λ_{1}}{k_{B}}}$ can be substituted.

p_{i} = \frac{e^{\frac{λ_{2} ϵ_{i}}{k_{B}}}}{\sum_{i = 1}^{N} e^{\frac{λ_{2} ϵ_{i}}{k_{B}}}}

Next solve to solve for $λ_{2}$ use the second constraint:

\sum_{i = 1}^{N} p_{i} ϵ_{i} = \overline{E}

Substitute in the equation for $p_{i}$ .

\sum_{i = 1}^{N} \frac{e^{\frac{λ_{2} ϵ_{i}}{k_{B}}}}{\sum_{j = 1}^{N} e^{\frac{λ_{2} ϵ_{j}}{k_{B}}}} ϵ_{i} = \overline{E}

Look at the equation for the maximum entropy:

S_{max} = S (P_{max}) = - \sum_{i = 1}^{N} p_{i} \ln (p_{i}) = - \sum_{i = 1}^{N} \frac{e^{\frac{λ_{2} ϵ_{i}}{k_{B}}}}{\sum_{j = 1}^{N} e^{\frac{λ_{2} ϵ_{j}}{k_{B}}}} \ln (\frac{e^{\frac{λ_{2} ϵ_{i}}{k_{B}}}}{\sum_{k = 1}^{N} e^{\frac{λ_{2} ϵ_{k}}{k_{B}}}})

Then simplify

S_{max} = - \sum_{i = 1}^{N} \frac{e^{\frac{λ_{2} ϵ_{i}}{k_{B}}}}{\sum_{j = 1}^{N} e^{\frac{λ_{2} ϵ_{j}}{k_{B}}}} \ln (\frac{e^{\frac{λ_{2} ϵ_{i}}{k_{B}}}}{\sum_{k = 1}^{N} e^{\frac{λ_{2} ϵ_{k}}{k_{B}}}})

Split the logarithm

S_{max} = - \sum_{i = 1}^{N} \frac{e^{\frac{λ_{2} ϵ_{i}}{k_{B}}}}{\sum_{j = 1}^{N} e^{\frac{λ_{2} ϵ_{j}}{k_{B}}}} (\ln (e^{\frac{λ_{2} ϵ_{i}}{k_{B}}}) - \ln (\sum_{k = 1}^{N} e^{\frac{λ_{2} ϵ_{k}}{k_{B}}}))

Cancel the logarithm and exponent

S_{max} = - \sum_{i = 1}^{N} \frac{e^{\frac{λ_{2} ϵ_{i}}{k_{B}}}}{\sum_{j = 1}^{N} e^{\frac{λ_{2} ϵ_{j}}{k_{B}}}} (\frac{λ_{2} ϵ_{i}}{k_{B}} - \ln (\sum_{k = 1}^{N} e^{\frac{λ_{2} ϵ_{k}}{k_{B}}}))

Split the summation.

S_{max} = \sum_{i = 1}^{N} \frac{e^{\frac{λ_{2} ϵ_{i}}{k_{B}}}}{\sum_{j = 1}^{N} e^{\frac{λ_{2} ϵ_{j}}{k_{B}}}} \ln (\sum_{k = 1}^{N} e^{\frac{λ_{2} ϵ_{k}}{k_{B}}}) - \sum_{i = 1}^{N} \frac{e^{\frac{λ_{2} ϵ_{i}}{k_{B}}}}{\sum_{j = 1}^{N} e^{\frac{λ_{2} ϵ_{j}}{k_{B}}}} (\frac{λ_{2} ϵ_{i}}{k_{B}})

The second term in the first summation is independent so it can be moved out.

S_{max} = \ln (\sum_{k = 1}^{N} e^{\frac{λ_{2} ϵ_{k}}{k_{B}}}) \sum_{i = 1}^{N} \frac{e^{\frac{λ_{2} ϵ_{i}}{k_{B}}}}{\sum_{j = 1}^{N} e^{\frac{λ_{2} ϵ_{j}}{k_{B}}}} - \sum_{i = 1}^{N} \frac{e^{\frac{λ_{2} ϵ_{i}}{k_{B}}}}{\sum_{j = 1}^{N} e^{\frac{λ_{2} ϵ_{j}}{k_{B}}}} (\frac{λ_{2} ϵ_{i}}{k_{B}})

Then from the first constraint the entire first summation is just equal to 1.

S_{max} = \ln (\sum_{k = 1}^{N} e^{\frac{λ_{2} ϵ_{k}}{k_{B}}}) - \sum_{i = 1}^{N} \frac{e^{\frac{λ_{2} ϵ_{i}}{k_{B}}}}{\sum_{j = 1}^{N} e^{\frac{λ_{2} ϵ_{j}}{k_{B}}}} (\frac{λ_{2} ϵ_{i}}{k_{B}})

Move the $λ_{2}$ out of the summation.

S_{max} = \ln (\sum_{k = 1}^{N} e^{\frac{λ_{2} ϵ_{k}}{k_{B}}}) - \frac{λ_{2}}{k_{B}} \sum_{i = 1}^{N} \frac{e^{\frac{λ_{2} ϵ_{i}}{k_{B}}}}{\sum_{j = 1}^{N} e^{\frac{λ_{2} ϵ_{j}}{k_{B}}}} ϵ_{i}

Then the summation is just equal to $\overline{E}$

S_{max} = \ln (\sum_{k = 1}^{N} e^{λ_{2} ϵ_{k}}) - \frac{λ_{2}}{k_{B}} \overline{E}

Then $λ_{2}$ is equal to the derivative which is ends up being the definition of temperature.

\frac{λ_{2}}{k_{B}} = \frac{\partial S_{max}}{\partial \overline{E}} = \frac{1}{T}

Substituting, the full equation becomes

p_{i} = \frac{e^{\frac{1}{k_{B} T} ϵ_{i}}}{\sum_{i = 1}^{N} e^{\frac{1}{k_{B} T} ϵ_{i}}}