Banner
    What Is Entropy?
    By Johannes Koelman | May 5th 2012 09:44 PM | 71 comments | Print | E-mail | Track Comments
    About Johannes

    I am a Dutchman, currently living in India. Following a PhD in theoretical physics (spin-polarized quantum systems*) I entered a Global Fortune

    ...

    View Johannes's Profile
    Entropy. A subject that comes back again and again and again and again and again in this blog. And so does the question in my inbox: "what exactly is entropy?" On the internet you can find a plethora of answers to this question. The quality of the answers ranges from 'plain nonsense' to 'almost right'. The correct definition of entropy is the one given in a previous blog:
    "the entropy of a physical system is the minimum number of bits you need to fully describe the detailed state of the system"
    So forget about statements like "entropy is disorder", "entropy measures randomness" and all vagaries about "teenage bedrooms getting messy" that inundate the internet. These qualitative statements at best provide you with metaphors, and at worst create profound misunderstandings.

    Now the problem with the above information-theoretical statement is that in itself it does not make obvious why bit counts provide us with a meaningful definition of entropy. More in particular, it appears that for many readers of this blog it remains unclear how this information-theoretical definition of entropy is related to the traditional thermodynamic definition of entropy. In this blog post I would like to try to give you at least some hints on "how everything hangs together".

    Reaching the current level of insight on what is entropy, didn't happen overnight. It took generations of scientists and a full century of multi-disciplinary science to reach this level of understanding. So don't despair if all puzzle pieces don't fall into place immediately. I guarantee you can reach a profound understanding of entropy by investing an amount of your time that is only a tiny fraction of a century...


    Thermodynamic Entropy




    In the middle of the 19th century, in the wake of the groundbreaking work of French military engineer Sadi Carnot on the maximum efficiency of heat engines, thermodynamics became a subject of serious study. The honor of introducing the concept "entropy" goes to German physicist Rudolf Clausius. He coined the term "entropy", and provided a clear quantitative definition. According to Clausius, the entropy change ΔS of a thermodynamic system absorbing a quantity of heat ΔQ at absolute temperature T is simply the ratio between the two:

    ΔS = ΔQ/T

    Using this definition, Clausius was able to cast Carnot's assertion that steam engines can not exceed a specific theoretical optimum efficiency into a much grander statement:

    "The entropy of the universe tends to a maximum"

    The second law of thermodynamics was born.

    However, defining entropy as a ratio between absorbed heat and absolute temperature leaves unanswered the nagging question "what really is the meaning of entropy?" The answer to this question had to await the atomistic view starting to gain popularity in mainstream physics. This happened at the end of the nineteenth century.


    Statistical Entropy




    It was early atomist Ludwig Boltzmann who provided a fundamental theoretical basis to the concept of entropy. Expressed in modern physics speak, his key insight was that absolute temperature is nothing more than energy per molecular degree of freedom. This suggests that Clausius ratio between absorbed energy and absolute temperature is nothing more than the number of molecular degrees of freedom:

    S = number of microscopic degrees of freedom

    What's the gain here? Haven't we expressed the enigmatic quantity "entropy" into an equally nebulous term "number of degrees of freedom"? The answer is "no". Boltzmann was able to show that the number of degrees of freedom of a physical system can be linked to the number of micro-states W of that system. The astonishingly simple expression that results for the entropy reads:

    S = log W

    Why does this work? Why is the number of degrees of freedom related to the logarithm of the total number of states? Consider a system with binary degrees of freedom. Let's say a system of N coins each showing head or tail. Each coin contributes one degree of freedom that can take two distinct values. So in total we have N binary degrees of freedom. Simple counting tells us that each coin (each degree of freedom) contributes a factor of two to the total number of distinct states the system can be in. In other words, W = 2N. Taking the base-2 logarithm (*) of both sides of this equation yields the logarithm of the total number of states to equal the number of degrees of freedom: log2 W = N.

    This argument can be made more generic. Key feature is that the total number of states W follows from multiplying together the number of states for each degree of freedom. By taking the logarithm of W, this product gets transformed into an addition of degrees of freedom. The result is an additive entropy concept: adding up the entropies of two independent sub systems gives us the entropy of the total system.


    Information Entropy




    Now fast forward to the middle of the 20th century. In 1948, Claude Shannon, an electrical engineer at Bell Telephone Laboratories, managed to mathematically quantify the concept of “information”. The key result he derived is that to describe the precise state of a system that can be in states 1, 2, ... n with probabilities p1, p2, ... pn requires a well-defined minimum number of bits. In fact, the best one can do is to assign log2(1/pi) bits to the occurrence of state i. This means that statistically speaking the minimum number of bits one needs to be capable of specifying the system regardless its precise state is:

    Minimum number of bits = p1 log2 (1/p1) + p2 log2 (1/p2) + .. + pn log2 (1/pn)

    When applied to a system that can be in W states, each with equal (**) probability p = 1/W, it follows that

    Minimum number of bits = log2 W

    And here we are. A full century of thermodynamic and statistical research leading to the simple conclusion that the Boltzmann expression S = log W is nothing more than a way of expressing

    S = number of bits required to specify the system

    There you have it. Entropy is the bit count of your system. The number of bits required to specify the actual microscopic configuration amongst the total number of micro-states allowed. In these terms the second law of thermodynamics tells us that closed systems tend to be characterized by a growing bit count. How does this work?


    The Second Law


    This is where the information-theoretic description of entropy shows its real strength. In fact, the second law of thermodynamics is rendered almost trivial by considering it from an information-theoretical perspective.

    Entropy growth indicates nothing more than physical systems prepared in special initial states. Let's again take the simple example of N coins. Suppose these are prepared in a state in which all coins show heads, and that a coin dynamics applies that results in some random coin turns at discrete time steps. It should be clear that the initial configuration with all coins 'frozen' into heads can be specified with very few bits. After many random coin turns, however, an equilibrium is reached in which each coin shows a random face and the description of the system will require specification of which of the equally likely 2N realizations is the actual one. This requires log2 (2N) = N bits. Entropy has grown from close to zero to N bits. That's all there is to it. The famous second law of thermodynamics. The law that according to famous astronomer Arthur Eddington holds a supreme position amongst all laws of physics.




    Examples Puhleeze...


    Still puzzled? Need some specific examples? In the next blog post I will make the statistical and information-theoretical basis for entropy more tangible by experimenting with toy systems. Simple systems that can easily be visualized and that allow for straightforward state counting. Although seemingly trivial, these toy models will lead us straight into concepts like phase transitions and holographic degrees of freedom. Stay tuned.



    Notes


    (*) The precise value of the base of the logarithm doesn't matter really. It all boils down to a choice of units. Taking the base-2 logarithm is the natural thing to do for binary degrees of freedom and results in entropy being measured in bits. A base-10 logarithm would result in an entropy measured in digits.

    (**) Why would each micro-state be equally likely? A lot can, and has been, said about this, and a whole body of research is directly related to this issue. Results obtained indicate that for large systems the equal likelihood assumption can be relaxed significantly without the end results being affected. For the purpose of the present discussion the issue is hardly relevant.

    Comments

    In information theory, a 'special' initial state does not change the number of bits. If all coins initially show head, all bits are initially 0. As the coins change state, the bits change value, and the number of bits doesn't change. It takes N bits to describe N coins in all possible states.

    It seems that the way physicists use information theory these days is quite different. Is the universe producing new coins every second since the bing bang?

    And what exactly defines a 'closed system' in the information theoretical definition of entropy?

    In information theory, a 'special' initial state does not change the number of bits. If all coins initially show head, all bits are initially 0. As the coins change state, the bits change value, and the number of bits doesn't change. It takes N bits to describe N coins in all possible states.
    I was going to raise the same issue, though I do not agree with your conclusion. The statement:
    "the entropy of a physical system is the minimum number of bits you need to fully describe the detailed state of the system"
    hides a fairly big assumption: that your macroscopic description of the system dramatically cuts the number of bits required. To use your N-bit model, we do not come to a pile of coins in a random state, we actually have heaps of half-sorted coins, one heap has 3/4 heads, another 90% tails etc. Unlike the pure information case, in physics we don't need to know the microscopic state, having such non-equilibrium (read very low probability) distribution has macroscopic effects: pressure, temperature etc.

    In fact, if we did know the microstate, just counting possible states would be a very poor measure of entropy - we would be obliged to consider e.g. Kolmogorov complexity instead of simply log W. Temperature would become meaningless, because a well-aimed hit from Maxwell's demon would freeze that boiling kettle and eject a single molecule with all the surplus energy. No doubt the little green men have discovered such a technology and this accounts for the OMG cosmic rays :) 

    However, we don't know the microstate, we just know the aggregates. In information terms we know some of the message. If M bits are known to comprise a known message, then the entropy is decreased to N-M bits. Increase in entropy corresponds to losing parts of the message to corruption or noise. However, in physics, the bits are active, the system evolves. You could say the N bits represent the state of a Turing machine. In which case the easily recognised message becomes steadily more scrambled even if no bits are actually lost. There comes a point where we look at a a jug of luke-warm water and say "well it started off as a pint of hot and a pint of cold, but it's irrevocably mixed up now so we have to estimate the entropy all over again." 
    Is the universe producing new coins every second since the big bang?
    Many would say so. But the pre-BB state must be a possible state of the universe so, unless there is a one-way, irreversible process, like the Damn Great External Battery theory (call it God or eternal inflation, it hardly matters), the expansion of space may not be creating new coins but just bringing in loads of coins that were "out-of-play" at the singularity; physical degrees of freedom that were sufficiently decoupled from the BB processes that they can be ignored in cosmology.

    I'm still trying to get my head round this. It isn't quite as neat and tidy as it may seem, because the next step would be to assume an eternal equilibrium with just the occasional fluctuation towards the singularity. Such an ergodic universe runs into problems - there is considerable surfeit of dead universes leading to most non-equilibrium universes, like ours, not having been born out of a BB at all, but merely a fluctuation out of heat death and back again. But I'm not too sure how that works if you assume that some of the information is taken out of play temporarily. It may be that universes like ours turn out to be quite common in that case. Or it may be that they they remain vanishingly unlikely and the most probable explanation of how we got here doesn't involve a BB at all.

    I tried to sketch this recently but it was greeted with howls of derision so I guess I'll do a blog some time and delete all scoffers :)

    Rene, I think the extra information that was left out of the blog post that would help to answer your concern is that what is counted is the _additional_ information needed to specify the microstate given its macrostate. So in the case of the N coins, assume that we are already told the number of heads. If that number of heads is N, we need no extra information to specify the microstate because there is only one: all coins have heads up. But if the number of heads is somewhere between 0 and N, i.e., when some but not all coins are heads, then we need additional information. Johannes, did I get that right? Entropy is defined with respect to some chosen macroscopic description.

    Johannes Koelman
    Rene, Derek, anon -- thanks for your comments on the question how available information affects the count. Gives me the steer to address this in more detail in the next post.

    Rene -- the whole point is that as long as you 'freeze' the coins in a situation of all heads, the information content is zero. (A somewhat degenerate case, I should perhaps have used an initial state of say 75% heads and 25% tails.) If you let go this constraint, i.e. you release the dynamical process and allow coins to be flipped, gradually more tails enter the description. This puts a heavier demand on the number of bits required to fully specify the state. The maximum bit requirement (one bit per coin) is reached when all coins are randomized without any bias for heads or tails.
    Key is (as Derek and anon point out) when counting states you should include the information you have on the system. More specifically: you should not count any states that are at odds with the information you have on the system.

    Derek -- you raise some interesting points. I purposely left out Kolmogorov entropy measures to make this blog post the least disputable. I am sympathetic to the view that Kolmogorov complexity will play an increasingly important role in physics once we start to understand gravitational degrees of  freedom. But this goes way beyond the purpose of the present blog post. The same holds  for the universe generating bits (coins) or not. Once gravity starts to dominate (at length scales compareable to the observable universe) horizons form that classically act as one-way membranes for information. This profoundly complicates the picture, and is way beyond the present post. (I promise in the future I'll come back to this.)

    Anon -- you got that right!
    Lex Anderson
    In information theory, a 'special' initial state does not change the number of bits. If all coins initially show head, all bits are initially 0. As the coins change state, the bits change value, and the number of bits doesn't change. It takes N bits to describe N coins in all possible states. 
    I find it useful to think of macro versus the micro-state in terms of data compression. I imagine an initial macro-state W0 that can be represented as s0 compressed bits (micro-states), where s0 < log2 W0. This is our ordered system. We then start randomly flipping bits (adding heat) in each W1, W2, ... until we arrive at a Wn where sn = log2 Wn, which is maximally disordered and hence cannot be compressed into any smaller number of bits. 
    It seems that the way physicists use information theory these days is quite different. Is the universe producing new coins every second since the bing bang?  
    This may stretch the analogy to beyond breaking point, but the singularity could be viewed as a "compression process", or result thereof. The big bang can therefore be viewed as an ongoing "decompression process" that continues right up until heat death; when all the information has finally been extracted from the singularity -- at which point entropy is maximal. 

    This analogy is an obvious oversimplification, but it may have a certain aesthetic usefulness. The idea that information is conserved by compression to varying degrees of losslessness (inside fractal dimensions for instance) and that it decoheres into entropy has far more theoretical sex appeal than allowing information to be destroyed by wavefunction collapse, for example -- which lazily eliminates lines of inquiry that could prove fruitful in reconciling classical and quantum physics.

    When asked "What compression does the Universe use?" the smart-Alec can reply, "Why, it uses mathematics, of course."
    And what exactly defines a 'closed system' in the information theoretical definition of entropy? 
    I think the definition is the same: any disjoint system. 
    Thor Russell
    So should it be the minimum number of bits with perfect compression then? Otherwise you can't easily distinguish the all heads from random states. 
    Thor Russell
    Lex Anderson
    Hi Thor, I'm not exactly sure of what you mean by perfect compression. By this do you mean lossless compression or something else entirely?
    I think he must mean that game we've all played where we take a bitmap of 12Mb, squash it down to a 200k jpeg, then put it through Winzip to get it down to 50k, then use rar compression to get it to 10k, send it through winzip and rar again and again and again until finally it ends up as a single bit.
     :)
     



    Lex Anderson
    One bit?? Compressing to the empty set is far more compact ;p
    Darnation, you're right! What is the logarithm of zero, by the way? :)
    Lex Anderson
    That's on my bucket list of questions to ask a universal Turing machine ;)
    Thor Russell
    I mean lossless compression. Its related to the entropy of 1000 heads compared to random H,T. Would you say that 1000 Heads has less entropy than a random combination of 1000 H/T? Surely you do otherwise how could you say that a box with all the air molecules in one side has less entropy then one with them spread evenly?
    If you just represent the position of each air molecule then the entropy is the same in both cases. If however you use less bits to describe the position in the case where they are only in one side then the information required is less. To say that it is less, you need to have the correct compression, i.e. have some knowledge of how to compress the position information to make use of the fact that all the molecules are in one side. 
    Thor Russell
    Lex Anderson
    I mean lossless compression. Its related to the entropy of 1000 heads compared to random H,T. Would you say that 1000 Heads has less entropy than a random combination of 1000 H/T 
    Not at all. What I am saying is that 1000 heads can represented in a lossless manner with far fewer than s=1000 bits. In general terms, highly ordered systems can be compressed into lower dimensional state spaces giving the appearance of entropy, but with less entropy than the bit count indicates. 
    Surely you do otherwise how could you say that a box with all the air molecules in one side has less entropy then one with them spread evenly?
    I read what I said pretty carefully and no matter how I squint my eyes, I don't see where I said anything remotely like this. Paraphrasing what I did say: A system appearing to evolve into fewer degrees of freedom does not necessarily mean that its entropy has increased.
    If you just represent the position of each air molecule then the entropy is the same in both cases. If however you use less bits to describe the position in the case where they are only in one side then the information required is less. 
    Failing to take into account the distribution of bits would imply loss of information, or lossi compression. The relative entropy (information theory) of a lossless compression function is 0. A random function has maximal entropy. Lossi compression functions will have varying amounts of relative entropy in-between the two limits.
    To say that it is less, you need to have the correct compression, i.e. have some knowledge of how to compress the position information to make use of the fact that all the molecules are in one side.  
    I take by "correct" compression you are meaning lossless compression?
    Thor Russell
    I wasn't saying that you thought 1000 heads always had the same entropy, rather checking your position. So you have 1000 heads which can be represented with fewer than 1000 bits, but that depends on an agreed compression algorithm which itself takes bits? Doesn't the compression for a given state depend on the compression algorithm (I always mean lossless), in which case the entropy you assign for a state will depend to some extent on how well the compression algorithm compresses that particular pattern. Its trivial if all the bits are the same, but for different patterns not so much so. Do you understand what I mean, perhaps someone else does too?
    Thor Russell
    Lex Anderson
    I wasn't saying that you thought 1000 heads always had the same entropy, rather checking your position. 
    Ahhh.. great to be on the same page.
    So you have 1000 heads which can be represented with fewer than 1000 bits, but that depends on an agreed compression algorithm which itself takes bits? 
    You make two critical assumptions, 1) that the compression is algorithmic and 2) that there is some kind of hidden mechanism that agrees upon (decides) the method of compression. Both assumptions aren't really relevant to the central idea, which is that compression "happens" just like wavefunction collapse "happens". The fact that these ideas can be expressed algorithmically or as CA or in myriad other ways with varying degrees of compactness speaks to the idea itself: Just like physicists, the Universe might favor the most compact representation.
    Doesn't the compression for a given state depend on the compression algorithm (I always mean lossless), in which case the entropy you assign for a state will depend to some extent on how well the compression algorithm compresses that particular pattern. Its trivial if all the bits are the same, but for different patterns not so much so. Do you understand what I mean, perhaps someone else does too?
    I think I understand what you are saying. I am in full agreement that lossless compression would be a trivial case, just as I believe its dual, wavefunction collapse is also a trivial case. Both are likely just the upper and lower bounds of the relative entropy of a decoherence event. 

    Based on the consistency of experimental data, lossless compression -- which would evidence itself in varying ways, such as macro-scale time asymmetry -- while not precluded in an information theoretic model of thermodynamics -- would likely only be possible under highly specific (yet maybe testable) conditions. The real meat in a "theory of information conservation" sandwich would however be the nature of lossy compression.
    Thanks, I say with relievement. I have been puzzled by the second law and Life :) And homeostasis.

    vongehr
    Very clearly written - thanks. Are requests for toy systems allowed? I think I am not alone when requesting an expanding model that lets Arthur Eddington eat his words [in the end, he will have to, or we would not be here ;-) ]
    Johannes Koelman
    Hmmm...  have been thinking about discrete expanding models lately. Not ready yet to blog about this. And certainly my next blog post will deal safely with finite closed models. I first want to get clarity on the concept entropy itself before discussing more controversial issues.

    Eddington has been wrong many times, but probably not on this subject. We (and any lifeforms for that matter) are the ultimate entropy producers, parasites to the low-entropy big bang. The fact that we require a low entropy big bang doesn't necessarily mean that a process at odds with Eddington's quote (Derek's Great External Battery, God or inflation) must have been at work to create the big bang. More about this later...
    I would love to be able to get rid of the battery :)  However, for all the esoteric mumblings I've heard about inflation explaining a low entropy start, it is still driven by the decay of a false vacuum or something equally battery-ish. I look forward to your explanation - this could be your toughest article yet, as I assume you are going to have to talk about those event horizons and information sinks. In turn, that is going to raise the question of whether there is a "God's-eye view" in which the system can be treated as closed and an equilibrium ensemble (or some generalization of it) can be defined after all. 

    Oh well, one step at a time... On Science20? Not a hope!
     
    Thanks for the justified explanation of one of the most ambiguous notions of science. I'm a chemist by the way and I must say the articles you write here simplify my studies.

    There's a notorious problem with this structuralist binary reduction: it is NOT correct to identify the Boltzmann and Shannon measures, because the former is continuous (due to continuous variables like position) while the latter works over a finite code-space. The Shannon measure then diverges as the number of code elements runs to infinity, but the Boltzmann integral does not diverge. This has been proved several times in the literature.

    Of course people now like to assume that IF space could be quantized, the problem would disappear and we could pretend we're living in a computer simulation. But that's hype from the computer industry: Claude Levi-Strauss called it "the myth of the moderns."

    Johannes Koelman
    Orwin -- thanks for highlighting the complication posed by continues variables. Indeed, for continuous variables, Boltzmann's configuration count translates straight into a phase space volume. For systems that can be described classically, this was what physicists focused on.

    However, we know deep down the world is quantum, and finite closed systems are discrete. In the end quantum physics acts as the great simplifier that reduces complicated continuous measures into simple counting, and provides discrete models with a strong basis.

    We don't know yet how to deal with gravitational degrees of freedom, but I think it is fair to say that most physicists working on this subject agree that once we know how to treat gravity quantum mechanically, also gravitational entropy will be the outcome of a discrete sum (for instance due to non-commutativity of space-time). But again that goes far beyond the scope of this blogpost (which has the sole intent to explain the concept 'entropy').
    Johannes, the series of principal quantum levels converges on the ionization energy. Quantize the space of the orbital and you loose the convergence. Also the oscillation around the ionization point giving infinite degrees of freedom of the ionic field, and the contrast of field and quantum entropies stressed by Penrose.

    You "know deep down" that space is quantized? That's a statement of faith! I'd rather grasp field entropy and engineer some stuff.

    This is such a huge, pervasive problem: they teach Herbert Spencer as Darwin, Poincare as Einstein and Shannon as Boltzmann. If this is the "Scientific Consensus" its also bankrupt!

    Johannes Koelman
    Orwin -- let's not make matters more complicated than strictly needed. We are talking about systems like N interacting particles in a box. Such systems are characterized by strictly discrete states. The convergence of levels and ionisation (continuum) limits don't apply. (These are limited to Coulomb potentials.)

    As mentioned: gravitational degrees of freedom (no-one knows what these are, there is lots of expectations, but certainly no consensus) are well beyond the intended scope of this didactical blog post. I promise you I'll come back to this some time in the future.
    Rene -- the whole point is that as long as you 'freeze' the coins in a situation of all heads, the information content is zero

    So, does 'all tails' have zero information too? If so, how does one tell the difference between these states?
    Doesn't knowing that the system has all heads infer some information? If we had a random 10 coins, I can see that we require 10 bits to specify the information. However, if we then continued randomising the coins, and accidentally came up with 10 heads, does the information suddenly disappear? I'm confused...

    If the state may be described as all heads or all tails, it contains one bit of information. Say 0=all heads, 1=all tails.

    Plus the information that all N bits are the same. Which is N-1 bits. 
     
    Johannes Koelman
    Anon's remark is correct. As mentioned in the article, the identification of a unique state out of W a-priori equally likely states requires log2W bits of information. So specifying 1-out-of-1 states requires zero bits (obvious, right?), and specifying 1-out-of-2 states requires 1 bit.
    Obvious but wrong!

    It is definitely not 1 out of 2 equi-probable outcomes it's 1 out of 1024.  davem set up the scenario. He specifically said:
    If we had a random 10 coins, I can see that we require 10 bits to specify the information. However, if we then continued randomising the coins, and accidentally came up with 10 heads, does the information suddenly disappear?
    So the scenario is NOT that the ten coins are constrained to fall the same way but that they happen to have fallen the same way on this occasion. In which case, if you wish to phone your friend on Alpha centauri to state the result, it is no good saying "the result was a head". You also have to say "all coins landed the same way".  I agree that if the coins are glued together (and any edgewise results discarded) then, of course, you only need the one bit, but if they are independent then all ten are needed.
     
    Johannes Koelman
    I see, this thread forked into two incompatible scenarios. I was reacting to Anon's scenario, not DaveM's scenario. Apologies for the confusion.

    I think we all agree that any state that accidentally shows up in the tossing of 10 coins requires 10 bits. It doesn't matter if the state is HHHHHHHHHH or TTTTTTTTTT or HHTHTTTHHT. The only thing that matters is the total number of realizations that 'could have been'.

    The  point I tried to make in the article (and that apparently confuses many readers) is somewhat more subtle. If you start with HHHHHHHHHH and each time randomly select a coin and turn it, you can utilize a more clever (dynamic) state coding. You know that at time zero you can only have one state: HHHHHHHHH. At time 1 you have 10 possible states: HHHHHHHHHT, HHHHHHHHTH, .. , THHHHHHHHH. After a number of timesteps an equilibrium is reached in which you will need to distinguish 210 states. So at time zero you need S = log2 1 = 0 bits. At the next time step S = log2 10 = 3.3 bits, etc. and after some time you will need log2 210 = 10 bits....  The almighty second law of thermodynamics rendered trivial by deploying an information-theoretical definition of entropy.
    No probs as long as my world isn't being completely shattered at this stage :)
    The almighty second law of thermodynamics rendered trivial by deploying an information-theoretical definition of entropy.
    Nice illustration but, ummm.... what does the term "thermodynamics" actually mean? It seems to suggest something to do with heat !!!!!
    Johannes Koelman
    what does the term "thermodynamics" actually mean? It seems to suggest something to do with heat !!!!!
    Bottom line: thermodynamics is the science that deals with systems described by bit counts and energy content. Its two main laws state: 1) energy doesn't change, and 2) bit counts don't decrease.
    Experimental thermodynamics has made truly spectacular advances. For instance, it has delivered ingenious instruments capable of measuring the energy per bit. These instruments are referred to as thermometers.
    Nicely fielded, Johannes :)

    Though the fact there *is* a characteristic energy per bit does get a wee bit complicated. 

     
    Lex Anderson
    I see, this thread forked into two incompatible scenarios. I was reacting to Anon's scenario, not DaveM's scenario. Apologies for the confusion.  
    This seems fairly consistent with the underlying science...
    I think we all agree that any state that accidentally shows up in the tossing of 10 coins requires 10 bits. It doesn't matter if the state is HHHHHHHHHH or TTTTTTTTTT or HHTHTTTHHT. The only thing that matters is the total number of realizations that 'could have been'.  
    Another way to look at this, that may partially reconcile both views, is as a lattice, with elements in the set of heads and tails. 

    When modeled as a lattice, this system acquires a mathematical degree of freedom that arises from the order and state of the coins themselves. Movement along this "axis" toward a minimal element may appear as an increase in entropy (because there are fewer degrees of freedom in the macro-state) but this is not necessarily the case. This goes back to the argument about whether the coins started with the degrees of freedom or whether the degrees of freedom are "created". With the lattice model at least, both notions are seem to be equivalent.
    The  point I tried to make in the article (and that apparently confuses many readers) is somewhat more subtle. If you start with HHHHHHHHHH and each time randomly select a coin and turn it, you can utilize a more clever (dynamic) state coding.  
    Could the "clever" encoding be achieved without a time axis? 
    The almighty second law of thermodynamics rendered trivial by deploying an information-theoretical definition of entropy.
    I remain hopeful that information theory can provide insights into these problems; but I don't think we are at the point of declaring 'a new kind of science' just yet ;)
    Johannes Koelman
    Could the "clever" encoding be achieved without a time axis?
    Not sure if I understand your question. The only 'clever' aspect is the time-dependent approach to the coding. So, I'd say "no".
    Lex Anderson


    What I mean is time, gravity, dimensionality etc have equal billing in information theory with any other quantity. 
    Entropy is very well defined by the Cybernetic science.

    Tony Fleming
    What about systems in equilibrium? What does entropy say about these? For instance a biological cell? And how would YOU measure these?
    Tony Fleming Biophotonics Research Institute tfleming@unifiedphysics.com
    It is hard for me to imagine a cell as a system in equilibrium knowing that it constantly interacts with its milieu, releasing heat hence producing entropy in its surroundings. Homeostasis maintains almost constant the state of a cell and what we can actually measure is ∆S. My bet is that it's really hard to measure it if the cell is not growing nor shrinking and to do it so, I guess calorimetric studies should be performed.

    I'm not a physicist... It's just my opinion. I'll be glad to read any correction.

    HOLA AMIGOS!

    I hope you'll forgive my -not always- poor English.

    Thank you Johannes, I've found your approach very useful and clear. It's kind a relief to get trough the concept of entropy directly from Boltzmann equation, calling the Shannon bit definition. I guess the mess came from my chemistry lessons (I think, because when explaining many different things Chemists lack a proper physicist background, e.g. the atomic model and, Entropy.).

    I barely remember how they explained to me what is entropy in a chemical process...

    As a biologist I've been puzzled for a long time with the concept and it's correct mathematical definition. The idea that the universe is expanding lead me a long time ago to the conclusion that entropy must be growing all the time. Then I started reading some cosmologists and found that many believe the maximum observable entropy of the universe is growing, slower than the maximum possible entropy. So even if there's a growth the distance between the two gets greater over time.
    So I wondered (Aaron, I hope you'll read this) if when relating information to the entropy of the universe; should we do it to the observed, the maximum or perhaps the difference between these two?

    I've never thought the BB as the maximum point of information compression since we're getting far from the maximum observable entropy... In a Popper point of view, I like to think that information lies between both maximum and observed entropies (Which will be the minimum number of bits needed to describe the rest).

    I hope I'm not saying to much non sense. Thank you for all your remarks.

    The application of entropy in chemistry is probably about as far removed from counting microstates as you can get. Even Carnot cycles are more easily related to the molecules bumping around. However the interesting thing in chemistry is this: all individual steps in a reaction are reversible: pushing an equilibrium in one direction or the other involves swamping the system with reagents or removal of product. But why should the reaction be reversible at all? If chemistry were driven by energy as we are often told, then obviously, things would react in one direction only: like sodium burning in chlorine. (Yes, sodium chloride is a lower energy state than a mixture of the two elements.) But most reactions have a significant reaction rate in both directions - and you can't have a road that runs downhill in both directions at once. So what is going on? The answer is, of course, that energy doesn't drive anything. This will come as a surprise to motorists, electricity companies and green politicians who all talk glibly of energy shortages. But energy, despite its name, is totally passive (classically anyway). It simply *is*. Of course it's *needed* for the reaction but the *driving force* is entropy. Or as "blue-green" occasionally slings into a thread, "entropy gradients". So reagent molecules and thermal energy all have to come together correctly for a reaction in either direction. And thus it's all governed by statistics. It's still a long haul to classical thermodynamic quantities, not least because to quantify these relatively rare events you need the statistics of a thermal distribution - and what is conspicuously lacking in Johannes' article is any discussion of temperature.  But in the end, when you look at the kinematics of a chemical process, it's never all-or-nothing according to whether there's enough energy, it's always a move towards maximum entropy at equilibrium. That's why you have all those dreadfully un-memorable thermodynamic quantities like enthalpy and Gibbs energy to cope with instead of plain and simple energy. If anyone is reading this and is wondering what the hell I'm talking about, I "recommend" this web page: if you are not confused now you soon will be. 


    Boy are you confused Derek. The writer of this blog post is right on to observe that understanding physics entropy is made much easier by understanding information entropy. "An Introduction to Information Theory: Symbols, Signals and Noise", Dover might give you a good start. Claude Shannon's work was key to the whole of the information age we enjoy. You said, "The answer is, of course, that energy doesn't drive anything." With this utter nonsense, I stopped reading.

    Thank you Anonymous. I have a Degree in physics and a Masters in electronics which qualifies me quite adequately in "information entropy", thank you very much, though every so often I read through the original treatise by Shannon and Weaver to see whether anything has changed, or struggle with more difficult stuff like "Evans Searle and Williams" derivation of equal probability in the general case. Now if you want to accuse me of "utter nonsense", feel free to criticise me when I get some heavy duty maths wrong. However please don't make an idiot of yourself by displaying your ignorance of thermodynamics the moment someone states something in an unfamiliar way. I did warn you: "This will come as a surprise to motorists, electricity companies and green politicians who all talk glibly of energy shortages. But energy, despite its name, is totally passive."

    Gracias Derek.

    The idea that energy is passive really simplifies lots of misunderstandings regarding how chemist conceive that issue. In fact it helps relate energy to matter, which are (I guess I'm correct) the same thing.
    I'll try to study again thermochemistry with this approach (I've read the blog you suggested) and I bet this time I'll be lesser confused.

    But what about the relation between information and entropy? It is simple to say "they are the same", but deep in my guts I believe it has to be a distinction.

    I mean, it's clear that information and entropy behave the same way, but could we say there is a maximum of information in the universe, not reachable and expanding all the time? Will we expect that at the cold end of expansion both the observed and maximum entropy are going to be the same (how will they get closer?)? Locally the degree of information tends to grow as complexity goes along with it; but in the process of expansion I can't picture how this growth will account for the gap between the two entropies.

    Could it be that as for matter and energy (the same in different observer' states), information and entropy rather than being the same they're just complementary?

    hahaha, I was getting nuts when I studied that stuff ten years ago, it made me ran away at one moment.

    Thank you for answering. I guess my ignorance can be bothering.

    I do apologise. I "recommended" that blog just to show how very confusing all the talk about classical thermodynamic quantities can be in chemistry. I think the article is very unclear actually. Although to be fair he does say something like "you probably don't really understand what enthalpy is: after you have read the following you still won't really know what it is but you'll be able to do the sums"  So, if you did manage to get some sense out of it, that's an unexpected result - and a welcome bonus!
       
    Shannon and Weaver addressed the difference between entropy and information, explicitly noting that it's not thermodynamics if there's no mention of heat or temperature. Other than that, it does seem that with real physical systems, information cannot be destroyed - the jury is out over whether this is true across those pesky event horizons but in ordinary circumstances the reversibility of physical laws means that you can always, in principle, back-track to recover a previous state. So there's no equivalent to simply deleting a file. On the other hand this does mean that physicists are tending to treat information as a physical thing... or so I'm told. Mind you, physicists say a lot of odd things :)
                              
    @Derek Potter

    Sodium burning (in chlorine) is only accomplished with additional heating of the sodium, afaik.
    You need a temperature difference [energy=photons] or a reagent like water [lower energy state] to get any reaction at all.

    I do not understand this apparent reversibility of the reaction without adding energy into the process, when reversed.

    Is there not some some fundamental directionality of energy in this universe in your opinion?

    The reaction is very exothermic, in fact sodium will burn with a flame in chlorine. You do not need to add heat to make it go, though if you want to make the show more interesting, you can get it started with a flame.
     
    Yes of course you need to supply energy to reverse the reaction. You can electrolyse molten salt to get sodium and chlorine directly. To reverse at the microscopic level you would have to reverse the thermal movements and send the emitted photons back - quite impractical, which is why that particular reaction appears to be irreversible. It's not irreversible, it's just that each spontaneous microevent is very unlikely.

    During the reaction, the liberated energy is scattered into the environment, indeed, some of it heads off into space at 300,000 km/s, thus greatly increasing the entropy. If it were all reflected back precisely, the one-way reaction would not happen. In other words if the entropy increase were (magically) reversed, so would the reaction. Hence the chemistry is driven by the entropy increase, NOT by the fact there is energy available.
     
    No, the laws of physics are time-reversible (with minor corrections for parity violation in some weak force interactions or whatever it is). That includes everything to do with energy. It's only if we create a non-equilibrium state of low entropy that systems become time-asymmetric - quite often giving the appearence of being driven by energy flows, but, in fact, being driven by the second law of thermodynamics.
     
    You have to be a bit careful with some systems whose dynamics are decoupled from the microstates - ideal frictionless engines for example. They do "evolve" (often round a cycle) as a result of their parts having kinetic energy but they are not driven by an energy flow, they simply roll along under their own inertia.
     
    This is tough because it's not clear how the minimum number of bits is determined, nor how a system can be fully described. Why is all heads only 1 or two bits? Because your representation/dictionary/algorithm gives those bits a particular meaning. 0 means all heads, 1 means all tails. Why can't you represent the data set by giving 0 or 1 a different meaning? Say 0 means the fibonacci sequence, and 1 means deviations (specified in following bits)? That's a potentially infinite data set in 1 bit. Why is "deviations from all tails" different from "deviations from fibonacci"? That's where the really helpful concept of randomness comes in. When you say that a 70% tails system requires fewer bits than a 50% tails system, you presuppose a system of representation that others use to define entropy: order/disorder/randomness and the deviation from it. The only definitions of entropy that ever made sense to me were the equations that included it - the distillation of observation. Bare "randomness" was the next best thing.

    Johannes Koelman
    Anon -- thanks for your question, makes me realize I have not been sufficiently clear on one important point.

    In practice the low-entropy initial state typically is a special state like "all heads". However (at least in principle) we could start from any other initial state. Let's say we have 8 coins and the initial state is HHTTTHTH. If we are absolutely sure the system is in that particular state, we could encode that state with zero bits. Now, if at the next time step all we know is that one coin has flipped, we can have eight states (HHTTTHTT, HHTTTHHH, HHTTTTTH, HHTTHHTH, HHTHTHTH, HHHTTHTH, HTTTTHTH, THTTTHTH) and we need log2 8 = 3 bits to encode the state.

    What is relevant is not the exact states, but the number of (equally likely) states compatible with all the information we have about the system.
    Thor Russell
    Thanks, I was kind of asking that also. Good article, and looking forward to your next.
    Thor Russell
    @Derek Potter

    Quote: "You do not need to add heat to make it go".

    Are you absolutely sure about this?

    Chlorine does react with a cold solution of Sodium *Hydroxide* [lower energy state], to give a mixture of Sodium ChloriDE.

    But we are talking about the start of a PURE sodium reaction without any [shared] energy as you say.

    Does a cold Sodium piece of metal without any additional water (Cl2 + H2O reversible to HCl + HClO) or impurities (Barium) start the reaction all by itself in chlorine?

    I don't think so.

    The sodium needs to be heated [or needs H2O] imo because you need additional energy to have the electrons jump from ground state to an unstable higher energy level.

    When those electrons jump back to the ground state they emit photons which carry the exact energy *amount* to trigger the reaction needed.

    After that it's easy - the reaction goes on and on until there is no more sodium left.

    Please repeat your experiment with a cold piece of sodium and chlorine at a higher temperature.

    As far as I know, the reaction does proceed in the cold and dry but not fast enough for it to run away and burst into flame. But even if it doesn't react at all, it makes no difference to what I'm saying. The reaction is still downhill energetically and downhill entropically.

    Just to make this perfectly clear: the meaning of "You do not need to add heat to make it go" is that heat is not absorbed by the reaction. If you had five tons of sodium and even more chlorine, you could probably set fire to it by applying a small flame to a few milligrams of the metal. After that, the whole lot will go up in a fireball. You do not need to have a bonfire under it or a connection to the national grid to push the reaction along.

    You bringing up the intermediate states is not really relevant, I'm afraid. Intermediate states are certainly important in slow kinematically limited organic chemistry but the reason I chose Na + Cl2 was to have a simple system in which we can see clearly where there is an entropy increase and why that stage of the reaction is not, in practice, reversible. You can, of course, make the subject as complex as you like if you wish to obscure the point! Unfortunately the mechanism you suggest is simply not viable: for one thing the overall energy of the reaction is several electron volts but a temperature of, say, 100 degrees C only adds about 10 meV to each candidate step - at least 100 times too small to do anything significant, let alone create a photon energetic enough to play a role. If you are interested in the precise mechanism, this may keep you busy:  http://chemed.chem.purdue.edu/genchem/topicreview/bp/ch7/whydoes.html
    Note, however, that the author is stuck in the "energy as the driving force" paradigm. Don't let that confuse you.

    I would, however, again stress  that the only point of this example is to illustrate the role of entropy. It is not a detailed analysis of the combustion mechanism.
     
    blue-green
    While we teeter on the edges of our seats waiting for our expositors Johannes K and David H to continue with what Einstein did best (relativity and statistical mechanics), let's play with how Einstein got so much right in the first place. Me thinks it is something that the biographers scarcely see and yet it is right there. I'm looking at Einstein's first job as a clerk at a patent office. Biographers dismiss this as being an under utilization of Einstein's genius and defensible as take what you can employment that a young lion would be thankful to have.

    I see much more in this patent clerk position. First of all, what more interesting time could there have been than at the dawn at the twentieth century to be a patent clerk? Back then, there was a wonderful tangibility to what was being invented and proposed. At Einstein's office, part of the requirement for an entry was for it not to be just on paper; the applicant had to also submit a working prototype. The physical requirement helps to get past the vaulted claims of the applicant, and yet, just how does one see through the snow job of every confidence man, except with practice?

    To do his job well, a patent clerk needs a way-above-average Bull Shit meter. He needs to be able to sort the wheat from the chaff (cranks). There is no sharper knife than a mastery of what the 2nd Law of Thermodynamics says is possible, impossible or highly improbable. What better place to keep it sharp than a Swiss Patent Office from 1903 through 1908? The entries that came across Einstein's desk were mostly of an electromotive nature. However, in casual chats with fellow employees, I imagine that Einstein's BS meter was equally capable of fielding false claims for refrigeration, sources of power and chemical wizardry.

    If he had not earned his chops at the patent office, it may have been much more difficult for him to propose his own crackpot-seeming theories and bring down real fire from the gods.
    Lex Anderson
    To do his job well, a patent clerk needs a way-above-average Bull Shit meter. He needs to be able to sort the wheat from the chaff (cranks). There is no sharper knife than a mastery of what the 2nd Law of Thermodynamics says is possible, impossible or highly improbable.  
    I agree with with the value of a "bullshit filter", providing that we are talking about a filter in the mathematical sense of the word filter -- or if not entirely quantitative, one that at very least is equidemensional. We all fear the consequence of allowing too much bullshit into "the body of knowledge" but science is far better equipped at disproving and disputing BS than it is at spotting the gaps (yawning chasms) that persist due to excessive filtering. 

    In Einstein's day the bullshit filter dial was set dangerously toward "Newtonian". Were it not for Eddington's positive result from his solar eclipse experiment, relativity could quite likely have suffered at the hand of that BS filter. I give this example because in this light, later statements made by Eddington (including those quoted in this very article) to me are a rich source of irony. 

    Can we deduce systemic "law" (tautology) from any aggregations of local state, no matter how highly consistent? Not at least according to Shannon entropy: A trillion heads in a row from a random variable doesn't alter the 1/2 chance of the next head falling. Gödel incompleteness arrives at the same conclusion in a more qualitative way.

    Bullshit is painful yet sufferable; ignorance however is tragic.
    blue-green
    The patent office was probably bursting with bizarre off-the-wall contraptions incorporating the latest from wireless technology to radioactive cuckoo clocks. Maybe the weird stuff helped Einstein get out of Newton's box. His work experience taught him the importance of presenting each of his cases in a manner that is simple, logical and testable with readily available and affordable equipment … unlike what goes on today in the frontiers of science, medicine and engineering.
    I estimating the entorpy of a living cell different than when people talk about the entropy of the brain? (For the latter) I always hear estimates based on how many neural connections there are, and how much information we store and process, and so on, but this must be different from the raw, physical entropy of a biological entity, right?

    Dammit I hit the button too early and now I can't correct it! Very sorry about "entorpy" and the first word should be "Is".

    The Stand-Up Physicist
    Does information theory ever dip its toes into complex numbers?  Many of the most interesting properties of quantum mechanics are shared by complex numbers, so it would be good to know about the range of information theory.
    blue-green
    one bit, two bits, qubits ...
    Lex Anderson
    Does information theory ever dip its toes into complex numbers?  
    Hey Doug. Information theoretic ideas are conventionally expressed in terms of binary degrees of freedom, but the choice of symbols and dimensionality is entirely arbitrary. 
    Many of the most interesting properties of quantum mechanics are shared by complex numbers, so it would be good to know about the range of information theory.  
    The notion of the "properties" of information is fascinating and deep. Superficially, information theory treats quantities merely as statistical distributions of symbols. At this level it is easy to see correspondences between qualitative notions such as thermodynamic heat and the quantitative information theoretic concept of random noise -- hence we might conclude that these really are expressions of the same thing: entropy. To me this is a valuable and necessary act of reasoning but is not in itself a conclusion.

    Whether such correspondences are superficial or not requires a closer examination of the taxonomy from which they arise, in this case thermodynamics. This lands us immediately in "hot water" (pun intended) because it is quickly apparent that both notions of entropy are at different levels of abstraction: In thermodynamics entropy is the measure of complex causal relationships between energy, time, space, heat and whatever else is floating in the bathwater... Implicit in these measures are qualitative rules such as the second law. Whereas, in information theory, entropy is nothing more than a (usually dimensionless) measure of the uncertainty of a distribution. 

    For a law that states "entropy must increase" to make any sense in information theory, we must therefore be able to model a system where all such degrees of freedom (energy, pressure etc) are either internal to or emergent from the system without further theoretical qualification: A tough but most likely rewarding task.
    Johannes Koelman
    Does information theory ever dip its toes into complex numbers?
    Doug, why so modest? I am sure your mind is secretly wandering way beyond those dull complex numbers. Hey, the logarithm of a quaternion is well-defined, and non-commutative entropies seem en vogue.
    The Stand-Up Physicist
    I wasn't wearing my protective head gear.  I have programed a quaternion logarithm, so I am glad to see it is well-defined.  I have yet to make the animation.  I went to Paris on my honeymoon, but didn't drop in to visit with Alain Connes.  To bad about that, it would be fun to drink and shoot the shit with him if he is that sort of fellow.  A brief scan of videos indicates he is an older French gentleman who can speak English far better than I can stumble through French, but he has a fast mind full of experience while I peddle along often hitting pot holes (in my defense, I was looking up into the heavens).
    It is fantastic that you do not follow the entropy-is-disorder mantra, but some issues still remain.

    About the definition. Entropy (Thermodynamic) is a physical quantity and physical quantities are not given by numbers.

    The above ΔS = ΔQ/T does not define entropy in thermodynamics. First it is Q not ΔQ. because heat is not a state function in Clausius formalism. Second, entropy S is a state function and ΔS is only a variation between two equilibrium states. Third, ΔS = Q/T is only valid for homogeneous closed systems, and reversible process.

    Sinf = log W, a statistical 'entropy', is not the same than a thermodynamic entropy as S(U,V,N) = k ln Ω(U,V,N). Statistical W can be virtually anything (it is highly subjective quantity) associated to informational-theoretic way to measure 'information'. Thermodynamic Ω is a property of the system and independent of our ignorance.

    As you correctly report the log W is only valid when all the j are equally probable. This is true for isolated systems at equilibrium but not otherwise.

    I do not know why you cite Shannon (whose 'contributions' to thermodynamics have been systematically discredited in the physicists and chemists literature. Even Wikipedia has material about that!), but you do not mention important contributions of people as Gibbs or vonNeuman.

    I forgot to mention some important details. For instance, that ΔS = Q/T also requires the process to be isothermal. If temperature varies the expression is not valid and has to be generalized.

    Regards.

    Physical quantities not given by numbers? Who told you that? If I measure an angle, it is a number (sometimes referred to as number of radians). If I measure the fine structure constant, it is a number. I can go on and on... ( By the way: you are mixing up Shannon with Janes.)

    It was Shannon who confused his informational 'entropy' with the thermodynamic Boltzmann entropy associated to the H-theorem, for instance. Jaynes and others developed a failed 'thermodynamics' over the base of Shannon early misconceptions.

    I am not an accomplished mathematician, I am merely an ameteur philosopher who dearly loves to read about and ponder these wonderful concepts.

    If I understand all this, it would seem that all the particles and energy forms in the universe have "universal" charateristics such as temp and/or mass that can be fairly accurately measured if contemplated in an isolated state, free of outside forces. Then as you allow two or more of the particles and forces to interact the possiblilities of their behavoirs and state changes grow; add three or four and the possibilities grow even more rapidly. You need more bits of information to calculate the possible future states of the system. So as entropy increases so do the bits of info needed to describe any one state and even more bits to predict the unobserved but potential behaviors or states.

    In some systems, such as an exploding star the innate information level appears to increase for all particles involved. Each particle in released from a fate of simply fuel for one star and they now have the potential to become comets, asteroids, molecules, new stars or planets. While in other systems, such as a steep gravity well, a particles "freedom of movement" and of possible states are significantly restricted by its enviornment. While in the well, the particles can no longer be "cool" or interact as easily with nearby particles as they could far away from the well.

    If the last two statements are true then it would seem that entropy increases and decreases in different local enviornments in the universe. Even further, specific forces seem to have different effects: gravity tends to limit possiblilities and heat tends to increase it.

    Donquixote5
    Dear Johannes,

    To my sincere regret, in the "true" time I was busy with other things and could not join this very interesting discussion.
     
    Therefore, first of all, I would greatly appreciate to express my sincere gratitude for your very interesting contribution.

    I am following the subject you are touching  here already for several years and have published two papers (and the third one is accepted for publication) ...

    Many clever people have been really close to conclusively answering the question "what the Hell is this damned ENTROPY ???", like, for example, Ludwig Boltzmann, Ed Jaynes, ...

    But, the only GUY, who has managed giving the true, logically (and even philosophically) consistent answer to the "eternal entropy question" is George Augustus Linhart.

    He started to consistently consider UNIDIRECTIONAL (IRREVERSIBLE) PROCESSES, instead of the usual CYCLIC (REVERSIBLE) PROCESSES of the conventional thermodynamics.

    The unidirectional process has naturally some definite starting state and some ultimate end state it finally approaches. Next, there is some - "driving force" - which ensures that the process in question will anyway start to reach its "progress". And, as with every process upon Earth, there will anyway be "hindrances" which will interfere the "progress" of the process in question.

    Thus, we start here with a kind of the "Yin-Yang" dialectics. Now, the "driving force" would require spending some kind of energy - and the First Law of Thermodynamics dictates that there ought to be "no anything out of nothing". Therefore, we ought to trigger some change of the energy of the available kind to some of its other kinds, to ensure switching on the necessary "driving force".

     Now, let us hear what G. A. Linhart tells us:

    "The question might be asked: What is the degree of hindrance in any natural process before it occurs ? Obviously, there can be no hindrance to anything that does not exist. But - at the very instance of inception of the process, hindrance sets in and continues to increase, until ultimately it checks nearly all the progress and reduces to a minimum the chance of any further advance. At this juncture the outcome of the process is said to have approximately attained its maximum. It is clear then - that the increase in the "progress" goes in the same direction - as the corresponding "hindrance", that is entropy." Thus, G. A. Linhart proposes that the physical-chemical sense of the entropy would be "all the possible hindrances to all the possible progress".

    Hence, to express this simple and clear dialectics mathematically requires invoking the probability theory (in the presence of such a struggle among the "progresses and hindrances" there will be no clear guarantee that the process comes to its end, but only a probability that the latter could be manageable). Starting from such an idea - and by using the school maths, G. A. Linhart could be able to formally derive the famous Boltzmann's formula for entropy (Interestingly, L. Boltzmann could only ingeniously guess his famous result !) ...

    So, Hendrik van Ness in his marvelous book "Understanding Thermodynamics" had come very-very near to the above-mentioned Linhart's ideas: there is really only one small step to the latter, from the CORRECT understanding of the FIRST and SECOND Laws of Thermodynamics ...

    Respectfully yours,

    Evgeni B. Starikov (Jewgeni Starikow)
    Lex Anderson
    • He started to consistently consider UNIDIRECTIONAL (IRREVERSIBLE) PROCESSES, instead of the usual CYCLIC (REVERSIBLE) PROCESSES of the conventional thermodynamics.
       
    Consider the heuristic that thermodynamics is likely a consistent and rather useful approximation ... like Newtonian mechanics or relativity or... Maybe one-way processes are cyclic processes with a periodicity too large for there to be any empirical evidence? Maybe the tub will start to warm up again due to some Planck scale effect that we will all be blogging about in 10 years? Personally, I remain quietly confident that there exists a super finite interval when the prime number distribution repeats... ;)
    Donquixote5
    Many thanks, Aaron, for your prompt and interesting, helpful response !

    REVERSIBLE PROCESSES are representing a limiting behavior - that is the best we can hope for , but never reach in effect. And the conventional Thermodynamics leads to the conclusion that the Delta-Entropy (an entropy change) for the reversible processes is equal to ZERO. But the entropy change is always greater than ZERO for the real, IRREVERSIBLE, UNIDIRECTIONAL process ... And G. A. Linhart had suggested  WHY it is exactly like this - and HOW we could use this fact in describing the ACTUAL, and NOT these LIMITING MODELS of processes, still remaining withing the frame of the usual, good old, conventional thermodynamics ...

    Respectfully yours, Evgeni B. Starikov