Derivation of the Boltzman Equation By Maximizing Entropy

Assume there are $N$ discrete states the system can be in: $X \in \{ x_1 \dots x_N \}$. Let $p_i = P(X = x_i)$ sampling the state over an indefinite time period, and $\epsilon_i$ be the energy in state $x_i$. The system will have 2 contraints, the probabilities must add to 1, and the average energy should equal $\bar{E}$:

$$\displaystyle\sum_{i=1}^{N} p_i = 1$$

and

$$\displaystyle\sum_{i=1}^{N} p_i \epsilon_i = \bar{E}$$The second law of thermodynamics says that entropy will always increase. Over a long enough time period the entropy of the system, $S$ should be maximized.$$S(p) = \displaystyle -\sum_{i=1}^{N} p_i \ln(p_i)$$

We want to find the probability distribution $p = {p_1 \dots p_n}$ that maximizes $X$, while following the 2 constraints.

$$p_{max} = \text{argmax}_P S(p)$$

Solve this system of equations with a Lagrange multiplier. First rewrite the 2 constraints like:

$$\displaystyle\sum_{i=1}^{N} p_i - 1 = 0$$

and

$$\displaystyle\sum_{i=1}^{N} p_i \epsilon_i - \bar{E} = 0$$

Then make the Lagrange equation

$$L(p, \lambda_1, \lambda_2) = \displaystyle -\sum_{i=1}^{N} p_i \ln(p_i) + \lambda_1 \Big(\sum_{i=1}^{N} p_i - 1\Big) + \lambda_2 \Big(\sum_{i=1}^{N} p_i \epsilon_i - \bar{E}\Big)$$

At the maximum the derivative with respect to any $p_i$ should be zero.

$$\dfrac{\partial L(p, \lambda_1, \lambda_2)}{\partial p_i} = - \ln(p_i) - p_i \frac{1}{p_i} + \lambda_1 + \lambda_2 p_i \epsilon_i$$

Set the derivative to zero.

$$0 = - \ln(p_i) - p_i \frac{1}{p_i} + \lambda_1 + \lambda_2 \epsilon_i$$

Then Solve for $p_i$.

$$\ln(p_i) = - 1 + \lambda_1 + \lambda_2 \epsilon_i$$

Exponentiate both sides

$$p_i = e^{- 1 + \lambda_1 + \lambda_2 \epsilon_i}$$

Then split the exponent.

$$p_i = e^{- 1 + \lambda_1}e^{\lambda_2 \epsilon_i}$$

---

Now we have to solve for the 2 multipliers. Start with the first constraint:

$$\displaystyle\sum_{i=1}^{N} p_i = 1$$

And substitute $p_i$.

$$\displaystyle\sum_{i=1}^{N} e^{- 1 + \lambda_1} e^{\lambda_2 \epsilon_i} = 1$$

The first term is independent of $p_i$ so it can be moved out of the sum.

$$e^{- 1 + \lambda_1} \displaystyle\sum_{i=1}^{N} e^{\lambda_2 \epsilon_i} = 1$$

Then divide both sides by the sum.

$$e^{- 1 + \lambda_1} = \frac{1}{\displaystyle \sum_{i=1}^{N} e^{\lambda_2 \epsilon_i}}$$

Now $e^{- 1 + \lambda_1}$ can be substituted.

$$p_i = \frac{e^{\lambda_2 \epsilon_i}}{\displaystyle \sum_{i=1}^{N} e^{\lambda_2 \epsilon_i}}$$

---

Next solve to solve for $\lambda_2$ use the second constraint:

$$\displaystyle\sum_{i=1}^{N} p_i \epsilon_i = \bar{E}$$

Substitute in the equation for $p_i$.

$$\displaystyle\sum_{i=1}^{N} \frac{e^{\lambda_2 \epsilon_i}}{\displaystyle \sum_{j=1}^{N} e^{\lambda_2 \epsilon_j}} \epsilon_i = \bar{E}$$

---

Look at the equation for the maximum entropy:

$$S_{max} = S(P_{max}) = \displaystyle -\sum_{i=1}^{N} p_i \ln(p_i) = -\sum_{i=1}^N \frac{e^{\lambda_2 \epsilon_i}}{ \sum_{j=1}^{N} e^{\lambda_2 \epsilon_j}} \ln \Big(\frac{e^{\lambda_2 \epsilon_i}}{ \sum_{k=1}^{N} e^{\lambda_2 \epsilon_k}} \Big)$$

Then simplify

$$S_{max} = \displaystyle -\sum_{i=1}^N \frac{e^{\lambda_2 \epsilon_i}}{ \sum_{j=1}^{N} e^{\lambda_2 \epsilon_j}} \ln \Big(\frac{e^{\lambda_2 \epsilon_i}}{ \sum_{k=1}^{N} e^{\lambda_2 \epsilon_k}} \Big)$$

Split the logarithm

$$S_{max} = \displaystyle -\sum_{i=1}^N \frac{e^{\lambda_2 \epsilon_i}}{ \sum_{j=1}^{N} e^{\lambda_2 \epsilon_j}} \Bigg(\ln\Big(e^{\lambda_2 \epsilon_i}\Big) - ln\Big( \sum_{k=1}^{N} e^{\lambda_2 \epsilon_k} \Big)\Bigg)$$

Cancel the logarithm and exponent

$$S_{max} = \displaystyle -\sum_{i=1}^N \frac{e^{\lambda_2 \epsilon_i}}{ \sum_{j=1}^{N} e^{\lambda_2 \epsilon_j}} \Bigg(\lambda_2 \epsilon_i - ln\Big( \sum_{k=1}^{N} e^{\lambda_2 \epsilon_k} \Big)\Bigg)$$

Split the summation.

$$S_{max} = \displaystyle \sum_{i=1}^N \frac{e^{\lambda_2 \epsilon_i}}{ \sum_{j=1}^{N} e^{\lambda_2 \epsilon_j}} ln\Big( \sum_{k=1}^{N} e^{\lambda_2 \epsilon_k} \Big) - \sum_{i=1}^N \frac{e^{\lambda_2 \epsilon_i}}{ \sum_{j=1}^{N} e^{\lambda_2 \epsilon_j}} \Big(\lambda_2 \epsilon_i \Big)$$

The second term in the first summation is independent so it can be moved out.

$$S_{max} = \displaystyle ln\Big( \sum_{k=1}^{N} e^{\lambda_2 \epsilon_k} \Big) \sum_{i=1}^N \frac{e^{\lambda_2 \epsilon_i}}{ \sum_{j=1}^{N} e^{\lambda_2 \epsilon_j}} - \sum_{i=1}^N \frac{e^{\lambda_2 \epsilon_i}}{ \sum_{j=1}^{N} e^{\lambda_2 \epsilon_j}} \Big(\lambda_2 \epsilon_i \Big)$$

Then from the first constraint the entire first summation is just equal to 1.

$$S_{max} = \displaystyle ln\Big( \sum_{k=1}^{N} e^{\lambda_2 \epsilon_k} \Big) - \sum_{i=1}^N \frac{e^{\lambda_2 \epsilon_i}}{ \sum_{j=1}^{N} e^{\lambda_2 \epsilon_j}} \Big(\lambda_2 \epsilon_i \Big)$$

Move the $\lambda_2$ out of the summation.

$$S_{max} = \displaystyle ln\Big( \sum_{k=1}^{N} e^{\lambda_2 \epsilon_k} \Big) - \lambda_2 \sum_{i=1}^N \frac{e^{\lambda_2 \epsilon_i}}{ \sum_{j=1}^{N} e^{\lambda_2 \epsilon_j}} \epsilon_i$$

Then the summation is just equal to $\bar{E}$

$$S_{max} = \displaystyle ln\Big( \sum_{k=1}^{N} e^{\lambda_2 \epsilon_k} \Big) - \lambda_2 \bar{E}$$

Then $\lambda_2$ is equal to the derivative which is ends up being the definition of temperature.

$$\lambda_2 = \dfrac{\partial S_{max}}{\partial \bar{E}} = \frac{1}{k_b T}$$

---

Substituting, the full equation becomes

$$p_i = \frac{e^{\frac{1}{k_b T} \epsilon_i}}{\displaystyle \sum_{i=1}^{N} e^{\frac{1}{k_b T} \epsilon_i}}$$