By Johannes Koelman | January 28th 2011 09:20 PM | 30 comments | Print | E-mail | Track Comments
    About Johannes

    I am a Dutchman, currently living in India. Following a PhD in theoretical physics (spin-polarized quantum systems*) I entered a Global Fortune


    View Johannes's Profile
    "Call it entropy [..] nobody knows what entropy really is, so in a debate you will always have the advantage.” -- Von Neumann to Shannon.

    Have a look at below picture. You see a binary pattern. This pattern is formed by a hexagonal pattern of empty (yellow) spots and blue disks. The pattern repeats. If you carefully inspection the figure, you will discover that the repeating unit consists of 31 spots (with 15 of these empty, and 16 occupied). Let's consider a simple question: How many bits do you need to fully specify the image displayed?

    Regular readers of this column know where I am heading. Yes, we are going to determine the entropy of the pattern. Even if you only occasionally visit this blog, you should know what entropy is. To jog your memory: the entropy of a system is the minimum number of bits needed to describe it. So tell me: what is this entropy for the pattern shown?

    How many bits do you need to specify this binary pattern?

    Do I hear 31? That seems a reasonable and logical answer. Although the pattern is stretching indefinitely in both planar directions, we only need to specify the presence or absence of disks for the repeating pattern. This can be accomplished with 31 bits. So, there we have the bit count or the entropy.

    You better try again.

    The number 31 is way off. The answer would have been 31 in case the presence or absence of disks would result from repeat coin tosses. But coin tosses are unlikely responsible for the above pattern. Not only does the pattern repeat in a perfectly regular fashion, there is also structure, a degree of order, within the 31-node repeat unit. As a consequence of this order, we need fewer than 31 bits to describe the pattern. Let's see if we can find out the amount of bit reduction.

    Have a careful look at the picture. Do you notice a pattern, some rule that applies? I give you a hint: pick a filled spot (blue disk). It doesn't matter which one. Out of the two nearest neighbor spots in the row above, how many are occupied? Now take another occupied spot, and again check the two neighboring spots immediately above it. How many are occupied? Repeat this until you have satisfied yourself there are no exceptions to the rule you have discovered.

    Now repeat this for an empty spot. Check several such spots. You can distinguish two scenarios, right? What rule emerges?

    Ok, if you have given this sufficient thought, you have discovered that the pattern is formed by a XOR (exclusive-or) rule: the occupancy of each spot in the hexagonal pattern is the XOR of the occupancy of the two spots immediately above it. This observation changes the determination of the entropy of this pattern dramatically. If you carefully consider the implication of this rule on the bit content of the pattern, you come to the conclusion that the above pattern can only contain five bits of information. This is visualized in below figure.

    In short, if you specify 5 bits in a row for the 31-spot repeat pattern, you have determined the starting value of the repeat pattern, and you can use the XOR rule to fill the rest of the pattern. So we come to the conclusion that the entropy (the 'independent bit content') of the pattern is 5. In other words: the dynamics in below figure defines a lossless compression algorithm that describes the whole figure in only 5 bits.

    Five bits are enough to specify the full pattern. Starting from five known occupancies, the gray disks (indicating unknown content of the spots) disappear under repeat application of the XOR rule.

    A more physical way of looking at this is to observe that we are dealing with a one-dimensional dynamical system for which the spatial dimension runs from left to right, and time from top to bottom. Such discrete dynamical systems are referred to as 'cellular automata'. To specify the full dynamics of a cellular automaton, you need to describe only a suitable starting state. For the cellular automaton represented in the above figures, the starting state for the full 31-spot pattern consists of only five bits. This is in line with what we observe in real life. Entropy is an extensive property: it scales with the spatial volume of the system under consideration, not with its spacetime volume. So we have found a perfectly acceptable answer that relates to the entropic scaling behavior of real systems.

    Well, the answer might seem acceptable, fact of the matter is: it is wrong. The number 5 might be closer to the truth than the number 31, but it is still way off.

    It is true that defining five spots leads to a complete pattern, but what we have failed to take into account is that these five spots can have arbitrary occupancy. In the example shown, we have assumed all five starting spots to be occupied. However, you can change this into any other combination of empty and occupied spots, provided at least one is occupied. Each time you change the five-spot starting pattern, the same total pattern emerges, but each time shifted in a different way.

    So we arrive at the shocking conclusion that the above binary pattern contains no information. You do not need to specify any of the spot occupancies. If you know nothing more than that we are dealing with a 31-spot XOR dynamics, you can reconstruct the whole picture.

    This all might seem trivial, but we have to conclude that we are dealing here with nothing less than a model for holographic degrees of freedom. A toy model, yet a valid representation of holographically reduced entropy in a dynamical system.

    We are used to the fact that when generated by a deterministic dynamics, the information contained in a two-dimensional pattern of pixels is not encoded in a two-dimensional pixel area, but in a one-dimensional row of pixels. However, when you look at the first figure, you see a holographic binary pattern with degrees of freedom that are 'thinned out' as if dictated by a further dimension reduction. The pattern is not encoded in a one-dimensional row of bits, but in a 'zero-dimensional group of bits'.

    The above is the subject of the essay I submitted earlier this week to the FQXi/ScientificAmerican essay contest 'Is Reality Digital or Analog?' If this blog entry wets your appetite: have a look at the essay. It goes much deeper into the matter, yet I tried hard to keep the essay understandable also to non-physicists. Reading it, you might learn a thing or two about dual descriptions, path sums, and... the holographic nature of physical reality. Enjoy!

    PS. Will keep you informed about the fate of my essay. Till mid March, you can discuss and rate the essays submitted to the contest.* Your ratings will determine which 35 essays will make it to the finals. You will notice that many essays (including my own) are quite speculative in character. That really is the nature of contests like these. Whilst you are at the FQXi site, you might want to have a look at the winning essays from the two earlier contests (here and here) * This requires a valid e-mail address.


    Damnit -- its this kind of thing that makes me wish I had gotten a PhD in Physics.

    Great and inspirational work, as always!

    Wonderful. I like it.

    Have you made Gerard 't Hooft aware of your contest entry? Since his article on the holographic principle, Gerard has been in search of cellular automata that behave holographically. He never succeeded. Your FQXI essay now shows he (and countless others) were overlooking a simple mechanism: releasing the local dynamics from the burden of information preservation and letting global closure do the hard work. What an amazingly elegant solution!

    I predict your essay will do very well in the contest provided it gets through the first stage that will rely on public voting. The reaction with the not so nice comment towards previous contest winners (now gone, pity, I had a funny remark in mind that I now have to save for another occasion) stresses the fact that many will not understand and appreciate the ideas presented. Anyway, good luck at the contest!

    releasing the local dynamics from the burden of information preservation and letting global closure do the hard work.
    Paul Davies talks about this for years now, but at least he understands the difference between true downwards causation (certain people may need to look this one up) and this stuff here that is plainly nothing to do with holography whatsoever. Hope this is your 'occasion'; bring it on.
    This answer is complete nonsense. The entropy of a message cannot be measured independantly of the entropy of the encoder and decoder. Their entropy is surely larger than your entire image.

    5 bits....? How many bits does it take to include the xor rule?

    Johannes Koelman
    Thanks for the reactions so far. @Tom - we are not talking here about a communication channel. This is about Kolmogorov complexity or algorithmic entropy. @Aitch - you highlight an important technicality (not discussed in this blog entry). If you want to encode the above picture, you have to specify we are dealing with binary hexagonal patterns and a local XOR dynamics that defines these. That takes a number of bits. However, this is a fixed number of bits that does not grow with the pattern size (the size of the repeat unit). See it as a small 'overhead cost' independent of the pattern size. The bit figure '5' that you quote will increase with pattern size (proportional to the spatial size of the pattern, or with he square root of the number of 'spots').
    I printed your essay to read offline. I did a search at Wolfram Demonstrations Project for Holomata or, xor and Holograph Cellular Automata … and got nothing …. (I do not think that this new word, “holomata” will stick). Here’s a link to a demonstration by the wolf-man himself.

    Quote: “Different cellular automaton rules typically produce different patterns. But this Demonstration shows what happens if one averages patterns from a sequence of possible rules. Notice how the rule 90 pattern seems to visually stand out, even when all the rules are averaged.”

    Just wondering if you think you have a New Kind of Science ….

    Before the end of January, I’m supposed to take advantage of an upgrade offer to Mathematica 8. I’m not an institution … so I don’t get many price breaks.

    Wondering if you would create a Demonstration at

    or if this is of little interest to you.
    Johannes Koelman
    Thanks Blue-Green. This XOR automaton is very easy to implement in a spreadsheet. No need for dedicated software. The way I use these XOR automata fits in what Wolfram refers to in his book A New Kind Of Science as 'systems with constraints'. However, he focuses on 1D systems with three binary inputs (there is 2^2^3 = 256 of them, hence the coding 0 - 255 he uses to distinguish the various rules). Basically, Wolfram had brushed aside simpler automata like this XOR automaton as 'uninteresting'. So that is why you will not find anything about this in his writings.
    Interesting that your quote was about Shannon, but you claim the post is not about communications channels.
    But it does not matter if the base system is defined by Shannon, Fischer, Popper's surprise or the Koomogorov measure, the point is that entropy, like work, requires a system against which it is measured. That base system requires entropy (or work) to create it, or even define it. When a single sub-system, (or message) is created and some claim is made about the amount of entropy in the sub-system, you cannot discount the entropy (or work) to create the overal system against which the measure is made.
    As you state the entropy in the base system will be very small if amortized over a large number of sub-systems (or messages), but that is not the case here. Entropy simply has no meaning unless it is measured against something else.

    Johannes Koelman
    The quote was to highlight the amount of confusion that arises as soon as one starts mentioning the word 'entropy'. The present discussion clearly is a case in point! If it helps: forget the word entropy. As a physicist I am talking about the degrees of freedom of the system, or the size of the smallest (maximally compressed) description of the system. This is a number that describes the complexity of the system and a number that is well-defined. It does not require an external system, or partitioning in subsystems. A real issue is that the number of degrees of freedom is in general uncomputable, but for this particular simple system we are not hindered by this. And yes, we need to have defined a 'language' to be used for the description, and in general there will be a language-dependency. However, this dependency is small in the thermodynamic (large pattern) limit (see answer to Aitch's remark).
    Delete my comment - why? Was critic from somebody who actually works on such stuff and knows a little about it going too far? Pseudo-Science2.0.
    Johannes Koelman
    In two years time I have deleted In total two comments from my blog because of insulting comments and foul language. This was one of them. Beware that I have zero tolerance in particular for those who insult third persons (which was the case with your thoughtless rant).
    Who did I insult? Your fqxi friends who took money from Templeton supposed to support otherwise unfunded projects and then distribute it to all the usual suspects who are well funded already with a little pitance to a few who behave well in order to keep the whole progressive/democratic facade up? And you buy it actually or just exploit it to hopefully get your misunderstanding about holography spread? How convenient. Well I stay with what I said 100%: Neither do you understand holography nor will anything that is of any use ever make it to the top at fqxi because it is a bunch of posers and phonies playing the system. Pseudo-science - I have none of it!
    Quentin Rowe
    I like Sasha's rants...  :-)

    I've had a whole ranting article dedicated to me. I figured it says far more about him than any criticism on the reciprient. So relax, we can all see through such criticisms, and often some good points are made amongst the noise.

    As always, I enjoyed your article, stilling thinking about it... i.e., a long way to go before I fully understand it. I had a question, but Aitch asked it for me, which shows a limited understanding on my part.

    Good luck with your essay submission...
    Bonny Bonobo alias Brat
    I also enjoy Sascha's rants (as long as they are directed at someone else) and I like to think I can stake a small claim to some of the article that you think was dedicated to you, after all Trollipop is my middle name! I'm trying to get my mind around entropy and holographic binary information or lack of it and any new angles to explain or argue against it are therefore much appreciated.
    My article about researchers identifying a potential blue green algae cause & L-Serine treatment for Lou Gehrig's ALS, MND, Parkinsons & Alzheimers is at
    Quentin Rowe
    Tee hee... it's true, yet I believe it was aimed mostly at Jerry. In coming to his defense, I copped a good rant, which then led to the Trolipop article. I think it was water of a duck's back to Jerry anyhow.

    As for the holographic approach, I find it very appealing, but my mind isn't fully wrapped around it just yet. Much to learn!

    Trollipop was directed mostly toward Jerry. Now Jerry is getting paid not to puiblish, He told me that he always finds something useful in Johannes' articles, but doesn't have time to read anything new from Sascha.

    Jerry is interested to describe the physical universe in terms of microscocopic states in the zero point field. Johannes' articles are a step in that direction.

    One difference is that Jerry thinks energy is fundamental in the zero point, described by entropy, but not created by entropic information.

    Jerry totally rejected Sascha's duality on the grounds that the two inmages of the proposition are not equivalenty in their local gravitational potential.

    "So we arrive at the shocking conclusion that the above binary pattern contains no information. You do not need to specify any of the spot occupancies. If you know nothing more than that we are dealing with a 31-spot XOR dynamics, you can reconstruct the whole picture."

    As you already state, this is the Kolmogorov complexity. Which, beside being given upto an emulation constant would determine the Entropy.

    But the Kologorov complexity cannot be determined, as you illustrated by making it smaller and smaller. Like the decimals of Pi, that are random under any test, but still have a very short Kologorov complexity.

    Which makes a nice puzzle, because the Kolmogorov complexity is non-computational, but entropy is a well defined and measurable quantity.

    Johannes Koelman
    Rob -- it is a misconception that Kolmogorov complexities can not be determined. Kolmogorov complexity falls in the class of uncomputable numbers, but that means nothing more than that there is no generic algorithm to compute this complexity (even in principle) for arbitrary systems. Uncomputability tells us nothing about the determination of the Kolmogorov complexity in specific cases. In the case presented above, the Kolmogorov complexity can be computed and the outcome is that it vanishes. It is easy to construct other examples where the Kolmogorov complexity is finite: just consider the above XOR cellular automaton active on a finite grid with the boundary values prescribed. This leads to a Kolmogorov complexity proportional to the 1D (spatial) size of the system. The entropy collapses to that of a 0D system when boundaries are eliminated and the system is made to be closed on itself.
    "In the case presented above, the Kolmogorov complexity can be computed and the outcome is that it vanishes. "

    This would limit the class of physical systems to those that have computable Kolmogorov complexity. That is an interesting conclusion that I have not seen before. (which is not saying much as I have not seen that much about it)

    The alternative would be that entropy only uses "averages", and physics is limited to systems with computable averages. Some of the underlying information might simply not show up in the averages and have no measurable effect (sounds a bit like hidden variables).

    Johannes Koelman
    Rob -- the key point here is that for physical systems the shortest description (the Kolmogorov complexity) results from utilizing the time evolution as 'data compression algorithm'. Only if this is the case, 'Kolmogorov complexity' equates to 'degrees of freedom'.
    kinda reminds me of Conway's Game of Life. Is there any relation, by any chance?

    Johannes Koelman
    The XOR automaton presented above and Conway's Game of Life are both examples of Cellular Automata. These are toy models typically used by physicists to sharpen their intuition towards certain (seemingly insurmountable) problems. For the XOR automaton the presently insurmountable problem is the holographic description of gravitating systems. In the FQXi essay I elaborate on this common feature between the Game of Life and XOR automata. Key is the fact that they share the same approach towards modeling complex physical systems: reducing the system to its barest essentials while preserving the qualitative features you are studying.
    follow up naive question: is the rule itself the holographic "surface"?

    Johannes Koelman
    Not a naive question at all. The above 2D plots contains one space and one time imension. The 'surface' of this 1D 'space' is a point (not two points due to the periodic repeat of the pattern). However, that point can be thought to be placed anywhere in the plot. So it is not straightforward to localize a holographic 'surface' in this model. A better way to look at this is to ask 'where are the bits that specify the pattern?' As I explained above, there are no bits involved in the allocation of disks to the open slots. However, to specify the picture, you have to indicate which repeat unit you are dealing with. This can be done by specifying the boundary of he repeat unit. This boils down to specifying the relative distances between a few corner points. So, you might say 'the holographic degrees of freedom reside at the corners of the boundary of the repeat unit. Keep in mind, that this is a somewhat abstract allocation of the degrees of freedom as the boundary can be placed anywhere.
    I am wondering Sascha, just what sort of clear objection you could raise to holography being fundamentally about nature having vastly fewer degrees of freedom than we might have naively thought.

    Sascha, it would be helpful if you could explain holography from your angle. It helps to see something from different perspectives to be able to walk around it. One superficial view is not enough.

    Around 2004, while on a long flight to the Netherlands, I tried to read through one of Susskind’s longer technical articles on holography. I remember that the surfaces of interest were the (null) sheets of lightcones … or the curved light-like “surfaces” of horizons … It was not the familiar surfaces of spatial geometry … so it was rather difficult to understand.

    At the time, it did not really sink in that holography was about thinning out degrees of freedom. Afterall, a section of spacetime can be carved up into a myriad of time-like, light-like or space-like surfaces, there is freedom in how to do the slicing …. although not so much with invariant light-like (null) surfaces ….

    I must return now to my end-of-the-year accounting spreadsheets …. Thanks for all of the diversions.
    Years ago I was contemplating a possible computer game program, one that would present a detailed virtual world to players. This led to a discussion of how much storage might me needed as the computers of the day were not known for today's enormous capacity. If the world were automatically generated it would come from programmed algorithms (think fractal mountains) and random numbers. Standard computer random numbers are fully deterministic such that if you know the algorithm and the seed number you know the entire sequence.

    The net result was the whole world could be stored as a single seed number. We laughed and called it the Seed Number Theory of the Universe. There may only be 2^16 or 2^32 different universes but they can be enormously detailed limited by the program doing the generation.

    Of course it is a dodge. When we describe something there are two key parts, our description and the shared context with our audience. Language has an enormous body of context. Just consider how much time and effort it takes to learn another one. Who knows how much context we all share as humanity which we can't see because we all share it? Pick the right language and a complex thought may be a one syllable word.

    It seems to me this article like my Seed Number Theory of the Universe plays at pushing information across the border between description and context. The more context you can start with the less description, the less bits you need. Certainly it is true that replicating the same pattern need not replicate the context. That is always true, replicating anything exactly does not increase complexity a significant amount. It can be known as this thing whatever it may be replicated N times. Anything perfectly replicated to large scale necessarily has low entropy.

    I think the real problem here which would be interesting to solve is the natural context. What does it take to describe some small simple volume of space starting from nothing? After all, the way to get out of the circularity of context vs. description is to get a clear starting point then describe how the space before us is different. The vacuum seems like a good start yet it is complicated and incompletely understood. How can we count bits to apply and test theories which ascribe physical properties to those bits when we can't properly describe nothing?

    In the classical thermodynamics they taught me in school we didn't mess with any of this. Absolute entropy? Get your head out of the clouds. Forget all that philosophical blather and remember it is just a change in heat content divided by temperature, dQ/T, where we deal in differences, get our work done and go home at the end of the day. Yes, the professor told us essentially this and it was the right attitude to solve useful problems.

    If you want absolute don't you have to solve context? Is absolute even possible?

    Johannes Koelman
    Mark -- thanks for your thoughtful comment. You are absolutely right that two types of information are needed: system context and system description. But as I remarked above, the only thing that concerns us is the information components that scale with system size. And context (no matter how large) is per definition independent of system size. If it puts you more at ease, you can reformulate the conclusion from this blogpost as: arbitrary large and complex 2D patterns can be defined that can be specified by adding a few bits to a fixed amount of contextual information. Defining a repeating 68,719,476,735 cell pattern takes as many bits as defining the repeating 31 cell pattern shown above. You might have guessed that I hold strong views on how to teach entropy. Your professor might have been a pragmatic guy and might have done a great job in teaching his classes how to perform thermodynamic calculations, but he really missed an opportunity to convey true insight. Teaching thermodynamic entropy without making the link to its microscopic origin is like teaching pressure or temperature without making the link to molecular motion.
    Teaching any advanced topic really needs the microscopic origin to gain any lasting credibility.

    Entropy appears to be a duality of the most fundamental type. The energy representation of entropy and the informational representation of entropy are completely equivalent in cases of thermodynamic equilibrium were energy is equally partitioned among the states.

    I wonder how the equivalence survives or fails in displacements from equilibrium. What happens when the bits in Johannes' article are stretched out of shape or compressed along one axis such that the probabilities are not exactly equal? Is the microscopic difference in the states equal to some feature of the distorting mechanism? If it is then the duality might surviive the disturbance.

    Space tends to relax toward equilibrium after a perturbing influence departs from a location. So I guess the duality might survive continually, but with enough features to respond to any passing event. It seems easier to get a duality from the microscopic space than from a black hole.

    I would like to see more articvles about the microscopics systems and how they conserve their properties under stress.