How many times have you had some pithy comment in prose be misinterpreted when you just know that if the other people could simply hear your voice or see your face they would know how brilliant you are?

It happens a lot and emoticons, along with their evil cousin ALL CAPITALIZATION AND !!! PUNCTUATION, were invented to try and close the communications gap.  

Now European researchers at the PASION project have developed a suite of tools to add non-verbal cues to email, phone calls, chats and other channels of electronic communication.   Digital communication hasn’t the richness of face-to-face conversation because it cannot communicate the non-verbal cues and contextual information that are so important to us all.

The technological advance is fascinating but the applications that the technology enables, including new kinds of online gaming, new forms of groupware, even tele-psychiatry(!!!  LOL meh) are what will make it take off.

One of the most compelling applications is “augmented social networking”. Over the last couple of years, services like Facebook or MySpace have been growing enormously but there are still things they can’t do. The tools PASION is developing offer new ways of showing your friends where you are, what you’re doing, how you feel; new ways of looking at your friends list, new ways of communicating.  But PASION is not just for play.



“Many problems in collaborative work derive not from the need to communicate and share explicit information, already met by current groupware and communication tools, but from higher-level problems in coordinating these processes,” explains Richard Walker, chief dissemination officer of the PASION project.  “For example, misunderstandings, or the erroneous attribution of intentions can cause problems for groups.”

PASION believes that augmented communication will limit or reduce this sort of problem.  The project has developed a prototype that runs on a mobile phone and provides information that users can exploit to coordinate their work. It can show information like user availability, or indicators and visualisations that illustrate the social position of a user in the group.

It also offers information on user mood. Currently, this works via self-reporting, but soon it will be generated automatically. It can even tell you what the user is doing at a particular moment.  PASION has already incorporated augmentation in many desktop tools, typically used by knowledge workers, such as MS Outlook, iGoogle, Skype, and Thunderbird. More are on the way.

But Gaming Is The 21st Century Killer App

Gaming, too, is a rich area for exploitation, with non-verbal cues augmenting games in the same way as the successful Nintendo Wii uses motion sensors to turn video games into a social event.

For example, the Familiars game developed within the project tracks user status within a social network and gives them a score for each improvement in their status. This could be really big. Already people on social networking websites like Facebook and MySpace suffer from artificial ‘friendship inflation’ as they add contacts pell-mell to their friends list.

But few of those friends are ‘real’ – many may be people the user never really meets. Often users gain prestige because of the quantity of friends they have, not the quality. Familiars tracks the quality. For example, is the user a significant node for many others? Do members in the group contact the user regularly? These are the factors used to calculate the final score.

Facial communication and the tele-shrink

There are vertical applications also, like tele-psychiatry. Using special software to analyze facial expressions, and a voice link, psychiatrists could work with their patients at a distance, helping them to deal with their problems.   Don't expect their per-hour rate to go down, though.   They will just be helping you from Tahiti instead.

Non-verbal cues about learners’ arousal or about the way a group is interacting could be enormously useful to online moderators or teachers in e-learning applications.  


So is it ready?

“There are, of course, [parts] that could use more work, but we have developed elements of all these systems, and different partners will take them forward in different ways,” explains Walker.

For example, simple sensors that indicate physiological state, like the heart rate, are simple, cost little to produce, are well accepted among younger users, and could be deployed cheaply and easily for social networking or gaming applications; these are tools we could see deployed in the near term.

Similarly, there are software modules almost ready to deploy in groupware, and that is an area that could be exploited relatively quickly. PASION’s telecom partner is considering deploying some of the research for mobile gaming, and this work could also appear quickly.

Other applications will take longer, or be reserved for specialist applications. PASION’s facial recognition system, for example, is currently very expensive but there are already applications where it could be applied – such as the tele-psychiatry one.

The PASION project received funding from the Future and Emerging Technologies (FET) strand of the Sixth Framework Programme for research.

More from PASION
More from ICT/Cordis Europa