Recently I have started reading papers on artificial life and it came to me as a complete surprise that so many people there consider that the thermodynamic entropy and the informational entropy are the same. Let me cite for example a few sentences from Christoph Adami "Introduction to Artificial Life":
p. 94 "Entropy is a measure of the disorder present in a system, or alternatively, a measure of our lack of knowledge about this system." p. 96 "If an observer gains knowledge about the system and thus determines that a number of states that were previously deemed probable are in fact unlikely, the entropy of the system (which now has turned into a conditional entropy), is lowered, simply because the number of different possible states in the lower. (Note that such a change in uncertainty is usually due to a measurement). p. 97 "Clearly, the entropy can also depend on what we consider "different". For example, one may count states as different that differ by, at most, del_x in some observable x (for example, the color of a ball drawn from an ensemble of differently shaded balls in an urn). Such entropies are then called fine-grained (if del_x is small), or course-grained (if del_x is large) entropies."This is completely different entropy as compared with that used in chemical thermodynamics. The goal of this document is hence to demonstrate this fact. I have had a discussion about these matters on biotaconv (http://www.biota.org/org/joinlist.html) that helped me to understand better the point of view accepted in the artificial life community. I am thankful to the members of the list for a nice discussion. I will start with a short description of how the entropy is employed in chemical thermodynamics, then I will briefly review the reasons for an opinion that the thermodynamic and information entropies are the same, and finally I will try to show that information and subjectivity have nothing to do with the entropy in chemical thermodynamics.
Entropy in Chemical Thermodynamics
The thermodynamics and entropy have been developed in order to describe heat engines. After that chemists have found many creative uses of thermodynamics and entropy in chemistry. Most often chemists employ thermodynamics in order to compute equilibrium composition, what should happen at the end when some species are mixed with each other. To this end there are thermodynamics tables, most famous being the JANAF Thermochemical Tables (Joint Army-Naval-Air Force Thermochemical Tables) http://www.nist.gov/data/PDFfiles/jpcrdM9.pdf (ca. 230 Mb) As the name says, the JANAF Tables have been originally developed for military. I guess that the very first edition was classified.Yet, there are many peaceful applications as well and chemists all over the world use these tables nowadays to predict equilibrium composition of a system in question. Among other properties, the JANAF Tables contains the entropy. I believe that this is a very good starting point for everybody that would like to talk about the entropy, just to take the JANAF Tables and see that chemists have successfully measured the entropy for a lot of compounds.
As the JANAF Tables are pretty big (almost 2000 pages), a simpler starting point is the CODATA Tables http://www.codata.org/resources/databases/key1.html In essence, the entropy in chemical thermodynamics is a quantitative property that does not depend on an observer. One can close eyes or open them, one can know about the JANAF Tables or not: this does not influence the entropy of substances at all. You may want to think it this way: chemists have been using thermodynamics and entropy for a long time to create reliable processes to obtain substances needed and they have been successful.
Entropy and Information
The members of biotaconv brought my attention to works of Edwin T. Jaynes who was presumably the first to show the equivalence between the thermodynamics and information entropies. His papers Information theory and statistical mechanics (Part I and II) are available inInternet (the links are in Wikipedia)
http://en.wikipedia.org/wiki/Edwin_Thompson_Jaynes
In the papers, the author considers statistical mechanics only, so let me first describe the relationship between classical and statistical thermodynamics. As the name says it, classical thermodynamics was created first. It was not an easy born (see for example, Truesdell’s The Tragicomical History of Thermodynamics) and many people find classical thermodynamics difficult to understand. No doubt, the Second Law here is the reason; people find it at least as non intuitive. On the other hand, statistical thermodynamics was based on the atomic theory and here was the hope to find a good and intuitive way to introduce entropy.Well, it actually did not happen. In order to explain you this, let us consider a simple experiment. We bring a glass of hot water in the room and leave it there. Eventually the temperature of the water will be equal to the ambient temperature. In classical thermodynamics, this process is considered as irreversible, that is, the Second Law forbids that the temperature in the glass will be hot again spontaneously.
It is in complete agreement with our experience, so one would expect the same from statistical mechanics. However there the entropy has some statistical meaning and there is a nonzero chance that the water will be hot again. Moreover, there is a theorem (Poincaré recurrence) that states that if we wait long enough then the temperature of the glass must be hot again. No doubt, the chances are very small and time to wait is very long, in a way this is negligible. Some people are happy with such statistical explanation, some not. Therefore the goal of Edwin T. Jaynes was to bring a new explanation of above.
The author uses the formal equivalence between the Shannon’s and thermodynamic entropies and based on this suggests the entropy inference or subjective statistical thermodynamics. I should say that I enjoyed the part I. Here the assumption is that as we do not have complete information about the system, we use what is available and then just maximize the entropy to get the most plausible description. In a way it is some data fitting problem and it seems to be similar to the maximum likelihood in statistics. Yet, the term information here is more like available experimental data about the system and the entropy is not the same as in classical thermodynamics.
Unfortunately I was not able to follow the logic in the second paper. I will make just a couple of citations.
"With such an interpretation the expression "irreversible process" represents a semantic confusion; it is not the physical process that is irreversible, but rather our ability to follow it. The second law of thermodynamics then becomes merely the statement that although our information as to the state of a system may be lost in a variety of ways, the only way in which it can be gained is by carrying out further measurements." "It is important to realize that the tendency of entropy to increase is not a consequence of the laws of physics as such, … . An entropy increase may occur unavoidably, due to our incomplete knowledge of the forces acting on a system, or it may be entirely voluntary act on our part."This is somewhat similar to what Christoph Adami says (see Introduction). What I am going to do next is to be back to chemical thermodynamics.
Information and Chemical Thermodynamics
Let us assume that the entropy is subjective and that it describes the information. If this is true, then this must be also applied in chemical thermodynamics. At this point I will cite a couple of paragraphs from the Preface to the JANAF Tables:Beginning in the mid-1950s, when elements other than the conventional carbon, hydrogen, oxygen, nitrogen, chlorine, and fluorine came into consideration as rocket propellant ingredients, formidable difficulties were encountered in conducting rigorous theoretical performance calculations for these new propellants. The first major problem was the calculational technique. The second was the lack of accurate thermodynamic data. By the end of 1959, the calculation technique problem had been substantially resolved by applying the method of minimization of free energy to large, high speed digital computers. At this point the calculations become as accurate as the thermodynamic data upon which they were based. However, serious gaps were present in the available data: For propellant ingredients, only the standard heat of formation is required to conduct a performance calculation. For combustion products, the enthalpy and entropy must be known, as a function of temperature, in addition to the standard heat of formation.One could imagine that Edwin T. Jaynes knew nothing about this, as it his times this could be even classified. Well, chemical thermodynamics was already a developed science in 1950s, the paragraphs above concern just a particular application of chemical thermodynamics. In any case, it hard to understand why Christoph Adami does not know how chemists employ entropy in their work. Thus, what is subjective in the JANAF Tables? What the entropy in the JANAF Tables has to do with the information? I have found no answers to these questions so far. Another point. People attribute information and subjectivity to the entropy. At the same time they do not see any problem with the energy. In the JANAF Tables there is a column for the entropy as well as for the enthalpy (H = U + pV).
The latter could be safely considered as energy. How people obtain the entropy in the JANAF Tables? The answer is that they measure the heat capacity and then take the integral at constant pressure of one atmosphere S_T = Integral_from_0_to_T Cp/T dT If there are phase transitions then it is necessary to add Del H_ph_tr/T_ph_tr. At the same time the change in the enthalpy H_T - H_0 = Integral_from_0_to_T Cp dT Here there is another question. What is the difference between Integral Cp/T dT and Integral Cp dT? Why the first integral has something to do with information and the second not? Why the first integral has something to do with subjectivity and the second not?
Conclusion
I have tried to show that subjectivity and information do not belong to chemical thermodynamics. In my view, this means that the thermodynamic entropy has nothing to do with the informational entropy. Let me repeat my logic once more. Chemical thermodynamics makes extensive use of the entropy. The entropy is tabulated in thermodynamic tables and then among other thermodynamic properties it is used to compute equilibrium composition or a complete phase diagram (see for example CALPHAD). I would expect that people stating that the informational entropy is the same as the thermodynamic entropy must show how such a statement is working in chemical thermodynamics.Personally I see no way to apply the information entropy and subjectivity in chemical thermodynamics. To this end, there is a good video from MIT, Teaching the Second Law http://mitworld.mit.edu/video/540 where a panel of scientist discuss how the Second Law should be taught. The video shows that the entropy is a difficult concept indeed. The scientists do not agree with each other on how to teach the Second Law. Yet, the concept of the entropy as information is not there at all. I will conclude by yet another question. Let us consider two statements
- The thermodynamic and information entropies are equivalent.
- The thermodynamic and information entropies are completely different.
Comments