Machines in the conversation: detecting themes and trends in informal communication streams.

INTRODUCTION

Since the dawn of spoken language, conversation has been the means by which ideas are developed and a consensus around those ideas obtained. The speed, range, and modes in which conversations can take place have increased with technological advancements over time. Recently, developments in the Internet and associated applications have made it possible for the scale of a single conversation to grow to one involving the simultaneous input of thousands of people. A discourse this massive poses the new challenge of properly summarizing all the thoughts generated and making them comprehensible for participants. This is the problem we address in our research.

Machines taking part in conversations is not a new idea. Conversation between man and machine has been a subject of intense interest ever since the computer was invented. The famous Turing Test (1) for machine intelligence focused on a machine being indistinguishable from a human in one-on-one conversation. One of the first artificial intelligence programs, ELIZA, (2) was a demonstration of a rudimentary conversation between a human patient and a machine "counselor". Our research takes on one small piece of the overall Turing Test problem by seeking an answer to the question, "What can computers contribute to a discourse that extends conversational content beyond what humans convey on their own?"

We believe the answer to this question lies in the text analysis of informal electronic communication streams. A computer that is recording and observing an electronic conversation among many different individuals over a period of time may be able to detect and report on overall metalevel themes and trends in the conversation, relay this information back to the conversational group, and thereby contribute to--and even influence--the course of the conversation. The theory is that in large-scale conversations, such as those taking place on Internet forums and through blogs (Web sites used in the manner of online journals), there are bound to be emergent phenomena, themes and trends that reflect common aggregate behavior that no single human reader can easily discern. This is where textmining approaches come in: The role of the computer in the discussion can be a combination of facilitator, neutral observer, and reporter--helping each human participant to more fully understand and appreciate all of the other human participants' thoughts and ideas and helping to amplify those discussion points that seem to reflect areas of group consensus or overlapping interests. Once an electronic discussion reaches a certain critical size (e.g., those involving hundreds or even thousands of participants in a focused period of time), the need for an individual or individuals to play this role becomes readily apparent. But, as the size of the conversation grows, the sheer volume of the content makes it impractical for humans to fulfill this role successfully. Thus we believe that as conversations scale larger and larger, enabled by instant messaging and World Wide Web technology, the need for computers to be involved in analyzing the content of the conversation and contributing the findings to the conversation becomes greater and greater.

The role played by computers in furthering human discussion is just beginning to be explored in research. The unstructured nature of blogging, discussion groups, opinions, reviews, and the like creates a kind of intellectual democracy of ideas. (3) Additionally, research has shown that group editors with shared concurrent editing capability have a positive effect on brainstorming. (4) Taking this a step further, it has also been shown that directed brainstorming (5) has a positive effect on creativity in problem solving. Obviously it is important to understand the organization (6,7) of the information. Then it is necessary to understand how this organization changes and what the diffusion characteristics (8) of ideas are over time. Once the behavior over time is understood, we would then want to understand the causal nature and the influential effects of information in a network. (9) For some applications, one may want to use and model this understanding to predict future behaviors. (10)

Our research is not about inventing new text-analysis tools; it is about employing and combining existing text-mining techniques in a new way to analyze and contribute to human discourse. We have developed a systematic method and toolset, which we first described in Reference 11. This paper describes how we have taken that generic text-mining approach and applied it to large-scale conversations called Jams. (12,13) A Jam is a construct invented at IBM that allows an organization of significant size to have a discussion in an area of interest with the goal of building consensus around actionable ideas. Our previous work began to indicate the potential of this technology to help facilitate the conversation during a single IBM Jam. (14) This paper takes a much broader look at both the methodology and its application across several Jams (internal to IBM and external) and shows how the analysis techniques have evolved to meet the challenges of this particular application. The success we have had with our approaches to date shows this to be a promising area for future applications in the field of conversational analytics and human-machine interaction.

WHAT IS A JAM?

A Jam is an internet- or intranet-based discussion and idea-stimulation vehicle. More formal than a chat room, a Jam is typically organized into a handful of separate forums (from four to seven in number), each on a different subtopic related to the overall Jam topic. The Jam is continuous, but conducted only for a limited time period (usually between 48 and 72 hours). During the event, participants can come into and leave a Jam as often as they like. Participants who register at the site can make original posts or reply to existing posts. The posts are labeled with the participant's name (anonymous contributions are not permitted). Some Jam participants may simply read the existing posts while others will enter posts without reading anyone else's thoughts. Most participants will both read what is already in the Jam and make their own contributions. As the Jam continues, themes emerge from the communication stream. These themes, detected by text mining, are posted back to the Jam periodically along with typical comments for each theme. This allows participants to see at a glance the gist of what is being said.

Moderators in each forum can highlight hot topics, referred to as Jam Alerts, as they emerge in the discussion (this is separate from the themes detected by text mining). Participants can also use full text search to browse for posts on a certain subject or for posts that particular individuals have contributed. Finally, posts can be e-mailed by Jam participants to others, perhaps encouraging them to make new contributions.

The process of Jamming at IBM has evolved over several years. At first it involved no text-mining technology at all. It used only human facilitators and asked participants to rate ideas to help analyze the event as it was happening and communicate information back to participants. Unfortunately, this system suffered an inevitable problem: The early ideas usually got the most votes. With the introduction of text-mining techniques into the more recent Jam events, each individual participant in the Jam is provided with the necessary information to "hear" the Jam as a whole.

At this writing, there have been seven Jams sponsored by IBM. This paper focuses on the three most recent Jam events that took place at different times between August 2003 and December 2005. Values Jam, a 72-hour event in 2003, involved IBM employees and explored the company's fundamental business beliefs and values. WorldJam, held the following year within IBM, studied how the IBM Values could be implemented. This 48-hour event generated over 32,000 posts. Habitat Jam, sponsored by the United Nations Habitat Initiative, the government of Canada, and IBM in 2005, was an open discussion on the Internet about the future of cities and the search for solutions to critical worldwide urban issues. During this 72-hour event, over 15,000 posts were generated from participants in 120 different countries.

UNDERSTANDING THE JAM THROUGH INTERACTIVE TEXT MINING

Although computers are quite capable of grouping documents together based on their surface characteristics (word frequency), such groupings may not always be useful. To ensure that categories make sense and make useful distinctions can require common sense knowledge and reasoning of a type not yet exhibited reliably by computer software. The involvement of a human in the role of analyst is needed to identify and discard spurious classes that are created from common features but have no underlying semantic value. This is what we mean by interactive text mining.

To play this critical role, the human data analyst must be provided with the necessary information to understand the meaning of each class. When one considers that each class may be composed of hundreds of examples and that the data frequently needs to be analyzed for multiple forums in real time, it becomes clear that powerful summarization tools are needed to communicate the meaning of each class in the taxonomy to the data analyst. Furthermore, as the data analyst finds classes that need to be modified or removed from the taxonomy, powerful editing tools are required to make changes that reflect the analyst's intent. (11,14)

Generating a taxonomy