Failure Does Not Necessarily Mean A Theory Is Wrong
    By Samuel Kenyon | November 4th 2012 06:51 PM | 3 comments | Print | E-mail | Track Comments
    About Samuel

    Software engineer, AI researcher, interaction designer (IxD), actor, writer, atheist transhumanist. My blog will attempt to synthesize concepts...

    View Samuel's Profile
    Many writers use a metaphor of the type "theory T has failed to explain phenomenon P" or "the field of F has failed..."

    For example:
    The modern materialist approach to life has conspicuously failed to explain such central mind-related features of our world as consciousness, intentionality, meaning, and value. [1]
    Ever since the Crash of 2008 there has been a widespread recognition, both among economists and the general public, that economic theory has failed. [2]
    The enterprise of achieving it artificially--the field of ‘artificial general intelligence’ or AGI--has made no progress whatever during the entire six decades of its existence. [3]
    ...Where Artificial Intelligence Went Wrong...At the beginning of AI, people were extremely optimistic about the field's progress, but it hasn't turned out that way. [4]
    Why Symbolic AI Failed: The Commonsense Knowledge Problem [5]

    What Is Wrong With These Metaphors?

    The authors invariable jump straight into the philosophical and/or scientific and/or technical reasons why T or F might be wrong, and why their alternative way may be correct.

    However, it is not necessarily true that a theory is wrong merely because it hasn't produced a particular result. In fact, it can't produce a result. It doesn't produce anything. "Producing" and "failing" are things that people do. Using the metaphor on a theory or framework shifts the blame onto inanimate abstractions.

    This can obscure the true reason as to the so-called failure. Science typically requires experiments, and experiments often require technology. Just developing the technology for a single project could require vast amounts of money and time and people. And then the theory might be wrong or need to be updated and you spend a bunch more resources on the tech.

    Projects are notoriously ridden with failure just by their nature. It doesn't even matter what the project is about.

    The Theory vs. the Project

    My point is that a theory may be correct, yet not have been proven or involved in a successful implementation simply because of project management and funding.

    This can go on all depends on the organizational contexts which produce the interest in a theory, the management of projects, and funding for those projects.

    Likewise, project shenanigans could make a wrong theory seem correct by veiling it with project-related implementation successes.

    Rocket Science

    Imagine if the US hadn't spent spent $136 billion on the Apollo Project leading up to the 1969 moon landing. With less money, it might never have been finished, at least not as early. In this alternate history, someone might have declared in 1970 that the field of rocket science had failed to get us to the moon and therefore we needed a new discipline.

    It's perfectly fine to try starting an alternative discipline. But it is not necessarily true that a theoretical framework has failed to do anything if management and funding has prevented exhaustive experimentation and successful implementation cycles.

    Projects Fail All the Time

    Around the world, millions of projects are failing right now. Some of them may be successful by certain metrics, such as generating revenue or generating publications, but unsuccessful in terms of implementing full examples of a theoretical framework. Some might be doomed from the start as a result of bad planning. Some may be death march projects.

    If the project is confused with the theoretical framework, people might avoid the framework for decades fearing that it was the cause of failure when in fact the cause was pathological project management.


    [1] Description of Thomas Nagel, Mind and Cosmos: Why the Materialist Neo-Darwinian Conception of Nature Is Almost Certainly False

    [2] George Soros, "Remarks at the Festival of Economics"

    [3] David Deutsch, "Creative blocks"

    [4] Yarden Katz, "Noam Chomsky on Where Artificial Intelligence Went Wrong"

    [5] Notes on a lecture by Hubert Dreyfus


    This caught my eye in particular because I'm pretty sure most theories of value are wrong, to the degree that they are applied in scenarios that don't conform with their assumptions. Most non-scientists I meet are not familiar with this aspect of theory - that all theories have constraints that may not be known for many, many years. The cleanest example for me is Newton Laws and Einstein work. Newton is fine until you're going really fast; then you need another way to describe the world. A messier one is any "economic theory" that includes rational actors. Humans are not rational, so these theories can only apply under conditions that constrain us to act rationally. These are not uncommon, so a lot of this stuff works well enough - just like Newton.

    The reasons I'm sure that value is not understood are myriad, to the point that I'm working on the big book of value (not the real title ). The second greatest flaw of most theories (after 'rational human') is lack of context. The answer to the paradox of value is simple: water has different positive and negative value for a swimmer in a lake, a non-swimmer in the same lake, and either of them in a flash flood. Each has a different context y measure value against. Also called "beauty is in the eye of the beholder."

    I'm hoping to be in a position to run real experiments on value in business in the next few years, to test (and possibly falsify) the theory of value that IIBA have developed. What we don't want to do is set out an untestable proposition with no predictive power.

    Thanks for your timely (for us) reminder that humans test theories.


    Julian Sammy

    Thanks for the comment. You have taken the discussion on a new path, but that's ok. Context is crucial to thinking about almost anything. Also for designing anything. Indeed making a good user interface depends largely on the context. For instance, a computer interface that is great for a person sitting at a desk focusing on one task might be horribly complicated for a cognitively loaded person wandering around outdoors attending to other goals.
    Some of the examples in my post have to do with Artificial Intelligence, which is of great interest to me. In AI context can be lost easily. It's understandable, given how hard it is to make a potentially large / complicated system. So people focus on testing one aspect of a framework, or test a theory in a test setting with perfect input data. 

    My point is that a theory may be correct, yet not have been proven
    A theory has its domain of applicability, the area where that description is in some way enlightening, and that has to be mapped out and its boundaries made clearly visible.
    Throwing money into a project in order to get something and let the result be spun in such a way that some initial idea is claimed "proven" no matter how much it changed (as it always does), well, that has little to do with theories.
    AI as a field failed as much as string theory or any other field: depends on what one expected. "Correct" does not apply. String theory found dualities and singularity free black hole solutions, that alone was worth the effort even if it is not the 100% correct description of this universe. That too much money went into it, harming other fields, is a different issue.