Task name and short name

        Emotion Cause Analysis (ECA)

Task Description

        In recent years, there has been a large body of research work on emotion classification from text. Most existing work only focused on the classification of emotions into one of the pre-defined emotion categories, for example, Ekman`s six basic emotion categories. Some corresponding bakeoffs or evaluations were organized in the past three years, such as the Emotion Analysis in Chinese Weibo text task on NLP&CC 2013[1] and NLP&CC 2014[2]. However, in many cases, we care more about the stimuli, or the cause of emotions. For instance, business organizations are more interested in finding out why people like or dislike products or services offered by them from users` comments or reviews rather than a simple categorization of sentiments. Similarly, instead of gauging public opinions towards policies or political issues using frequency counts, governments would like to know the triggering factors of negative attitudes expressed online. As such, there has been an increasing interest in the research on emotion cause analysis more recently. Unfortunately, the lack of annotated corpora and standard metrics for this task has limited the research on this topic. Thus, we propose to organize a new task, emotion cause analysis (ECA), in NTCIR 13.

        (1) Construction of corpora:

                The city news in the last two years in English and Chinese are collected as the raw data. The emotion expressions in the text are firstly identified and the emotion categories are annotated. Next, the direct cause(s) stimulates such an emotion is identified from its context and is annotated accordingly. In some cases, the direct cause is elaborated in more details in another sentence. For such cases, both the direct cause and elaborated cause are annotated. Here is an example:

                "ISIS blow up pagan temple in Palmyra that has stood for 2,000 years. The world community is infuriated by this destruction".

                Its annotations are:

                Emotion category: Anger

                        Direct cause: "this destruction"

                Elaborated cause: "ISIS blow up pagan temple in Palmyra that has stood for 2,000 years"

        A Chinese example is given below:

                (Known the things happened in the last 10 minutes of his life, a lot of people left the sad tears)

                The annotations are:

                Emotion category: Sad

                        Direct cause:(Known the things happened in the last 10 minutes of his life)

                There are also some cases where the emotion cause is not given. For such cases, only the emotion categories are annotated.

        (2) Subtasks and evaluation metrics:

                a) Emotion cause detection at the clause level

                        This subtask aims to evaluate the techniques for detecting the clause which contains emotion cause. It can be treated as a clause-level binary text classification problem. The clauses will be classified as containing emotion cause or not. The metrics is based on the standard text classification metrics:

                Here, we set lenient and strict evaluations, respectively. For lenient evaluation, the clauses containing either direct cause or elaborated cause are regarded as cause clauses. As for the strict evaluation, only the clauses containing elaborated cause are regarded as cause clauses.

                b) Emotion cause extraction

                        This subtask is designed to evaluate the techniques for detecting the boundary of the emotion cause. It can be regarded as an information extraction task. The metric is based on the overlapping between the detected emotion cause and the annotation at the phrase level:

                Similar to Subtask 1, we set lenient and strict evaluations, respectively. For lenient evaluation, both the boundaries of direct cause and elaborated cause are valid. As for the strict evaluation, only the boundaries of elaborated cause are regarded correct.

Motivation

        To evaluate the state-of-the-art emotion cause extraction/detection methods on Chinese and English news text.

Methodology

        Automatic evaluation for each subtask with Precision, Recall and F-measure using clause level classification and phrase boundary level metrics, respectively

Expect results

        The most up-to-date emotion cause analysis techniques on news text involving Simplified Chinese and English.

Organizers:

        Ruifeng Xu, Professor, Laboratory of Network Oriented Intelligent Computation, Shenzhen Graduate School, Harbin Institute of Technology, xuruifeng@hitsz.edu.cn

        Yulan He, Reader, School of Engineering and Applied Science, Aston University, UK y.he9@aston.ac.uk

        Qin Lu, Professor, Department of Computing, The Hong Kong Polytechnic University, csluqin@comp.polyu.edu.hk

        Kam-Fai Wong, Professor, Department of Systems Engineering & Engineering Management, the Chinese University of Hong Kong, kfwong@se.cuhk.edu.hk

Prospective participants:

        Universities / research institutes / companies interested in multi-lingual emotion analysis, especially emotion cause analysis, in Simplified Chinese and English.

Data to be used / or constructed:

        We plan to develop an emotion cause corpus on multi-lingual news text including Simplified Chinese and English.

        For each language, 3,000 documents will be annotated with emotion causes, in which, 300 documents will be released as the sample data, 2,200 documents will be released as the training data and the rest 500 documents will be used as the testing data. Furthermore, 10,000 documents with emotions will be released for building semi-supervised systems.

Schedule
DataEventdownload
2016-08-15First call for participation
2016-12-01Registration due
2017-01-08Sample data releaseLocal Download
Dropbox Download:English version
Dropbox Download:Chinese version
2017-05-04Training data releaseLocal Dowwnload
Dropbox Download:English version
Dropbox Download:Chinese version
2017-07-07Formal run result due
2017-08-01Delivery of evaluation results
2017-12NTCIR-13 Conference & EVIA 2017 in NII, Tokyo, Japan