Back in the fall of 2005 I took a class at the MIT Media Lab called Commonsense Reasoning for Interaction Applications taught by Henry Lieberman and TA'd by Hugo Liu.

For the first programming assignment I made a project called AffectWorld, which allows the user to explore in 3D space the affective (emotional) appraisal of any document.

The program uses an affective normative ratings word list expanded with the Open Mind Common Sense (OMCS) knowledgebase. This norms list is used both for appraising input text and for generating an affect-rated image database. The affective norms data came from a private dataset created by Margaret M. Bradley and Peter J. Lang at the NIMH Center for the Study of Emotion and Attention, consisting of English words rated in terms of pleasure, arousal and dominance (PAD).

To generate the interactive visualization, AffectWorld analyzes a text, finds images that are linked affectively, and applies them to virtual 3D objects, creating a scene filled with emotional metaphors.

The image files were scraped from a few places, including Eric Conveys an Emotion, in which some guy photographed himself making every emotional expression he could think of and then started doing requests. I used OGRE for the 3D graphic engine.

So what was the point? If I remember correctly, somebody asked that in the class and Hugo interjected that it was art. Basically the emotional programming looks to an outsider like a pseudo-random image selector applied to cubes in a 3D world...well, that's not completely true. With a lot more pictures to choose from (with accurate descriptive words assigned to each picture), I think that one could make a program like this that does give some kind of emotional feel that's appropriate from a text.

Certainly stories are a kind of text that explicitly describe affect: the emotions of characters and the environments enveloping the characters. AffectWorld programs would never be perfect though, because stories themselves are just triggers, and what they trigger in any given person's mind is somewhat unique. This is perhaps the realm that film directors using published stories live in--creating a single visual representation of something that already has thousands or millions of mental representations. But in an AffectWorld, I simplify the problem but assuming from the beginning that the visual pictures are arbitrary. It is only the emotional aspects that matter.

At the time of the demo, some people seemed momentarily impressed, but that was partially because I made them look at a bunch of boring code and then suddenly whipped out the interactive 3D demo. Otherwise, my first version of AffectWorld was just a glimmer of something potentially entertaining. I started another project for that class which I will talk about in a future blog.

Part of the reason why I took the class was because I was skeptical of using commonsense databases, especially those based on sentences of human text. During my early natural language explorations I became suspicious of what I learned later was called the "hermeneutic hall of mirrors" by Stevan Harnad--in other words, computer "knowledge" dependent on English (or any other human language) is basically convoluted Mad Libs. However, I did witness other students making interfaces which were able to make use of shallow knowledge for unique user experiences. Just as Mad Libs lends itself to a kind of surprising weird humor, so do some of these "commonsense" programs.

This is somewhat useful for interaction designers--in some cases a "cute" or funny mistake is better than a depressing mistake that triggers the user to throw the computer out the window. Shallow knowledge is another tool that is perfectly fine to use in certain practical applications. But it's not a major win for human-level or "strong" AI.

The Semantic Web is a similar beast as far as I can tell. Despite recent good-intentioned articles full of buzzwords, the Semantic Web has been around for a long time, at least conceptually. Seven years ago I went to an AI conference where Tim Berners-Lee (the inventor of the World Wide Web) told us about how the Semantic Web was the new hotness and its relationship to AI (AAAI-06 Keynote Address). OWL, the web ontology language standard, had already been started. And now the semantic web is apparently finally here, sort of. Companies that rose to power since the concept of OWL like Facebook and Google have made massive semantic networks out of user data. These are great enablers and we probably have not even seen the killer apps to come out of these new semantic nets. And in some narrow contexts, semantic net powered apps could be smarter than humans. But they do not understand as human organisms do. Sure, there could be a lot of overlap with some level of abstraction in the human mind, and it is not necessarily true that all knowledge is grounded in the same way or at the same level.

Someone will probably post a comment along the lines of "well that is ultimately how the brain works, just a big semantic net in terms of itself" which skips the issue of the nodes in computer semantic networks that depend on human input and/or interpretation for the meaning. Or somebody might argue that the patterns have inherent meaning, but I don't buy that for the entirety of human-like meaning because of our evolutionary history and the philosophical possibility that our primitive mental concepts are merely reality interfaces selected for reproductive ability in certain contexts.


At the time of the commonsense reasoning class--and also Marvin Minsky's Society of Mind / Emotion Machine class I took before that--a graduate student named Push Singh was the mastermind behind Open Mind Common Sense. Although I was skeptical of that kind of knowledgebase, I was very interested in his approaches and courage to tackle some of the Society of Mind and Emotion Machine architectural concepts. His thesis project was in fact called EM-ONE, as in Emotion Machine 1, dealing with levels of cognition and mental critics. I didn't know him very well but I talked to him several times and he had encouraged me to keep the dialogue going. I recall one day when I reading a book about evo-devo in the second floor cafe at the Harvard Co-op bookstore, ignoring all humans around me, Push happened to be there and made sure to say hello and ask what I was reading.

One day I went to his website to see if there was anything new, and found a message from somebody else posted there: Push was dead. He had committed suicide. Below that, stuck to my computer monitor, lurked an old post-it note with a now unrealizable to-do: "Go chat with Push."

Image credits:

Mad libs example by Becca Dulgarian via Emily Hill