Guiding Theme D2: Manual and Automatic Quality Assessment of Summaries from Heterogeneous Sources

In guiding theme D2, we research content- and user-related quality indicators of multi-document summaries. We put a particular focus on improving summaries based on user feedback and how users interact with summaries. To achieve this goal, we use interactive machine learning techniques such as active learning and reinforcement learning as well as scalable data management approaches. Towards AIPHES' joint corpus strategy, guiding theme D2 is concerned with creating summaries, scoring them, and consulting the other guiding themes with using and extending these summarization corpora.

Research results of the first Ph.D. cohort

During the first phase, we put a particular focus on user-related quality indicators of information preparation and investigate computer-assisted approaches to multi-document summarization (MDS) using interactive machine learning. Based on the output of a state-of-the-art MDS system, we collect user feedback, improve the summary based on this feedback, and ask the user for more feedback in an interactive loop. This allows users to incrementally personalize a summary to their needs. We find that active learning is very effective to trade-off summarization quality and the amount of feedback required by a user (P.V.S. and Meyer, 2017). Our approach is a first step towards realizing the AIPHES vision of a semi-automatic writing aid for editors and journalists. To initiate a specific use case in journalism, we have studied live blog summarization (P.V.S. et al., 2018) together with guiding theme B2. Currently, we investigate interactive sentence compression methods to assist users with rewriting an automatically derived extractive summary. In a cooperation with the RMIT University in Australia, we additionally work on personalized recommendation methods, which use summaries to explain the system’s output to its users.

Associated AIPHES researchers worked, among other topics, on intelligent writing aid applications closely linked with this guiding theme. We published about tools for identifying language issues (Wolfer et al., 2018), argumentative writing (Stab and Gurevych, 2017), and diagnostic reasoning (Schulz et al., 2017). Together with other guiding themes, we have jointly worked on summarization corpora.

Research results of the first Ph.D. cohort

Since interactive summarization systems must react in real-time, the PhD students of both cohorts worked towards the system demo Sherlock for interactive MDS. Sherlock applies approximate interactive summarization models to achieve scalability and fast system response times (P.V.S. et al., 2018b; Hättasch, 2018). During the second phase, we will research interactive text exploration systems.

We will continue developing new approximation techniques that allow existing approaches for customized summarization to be used as interactive tool (i.e., create a summary in interactive time) while preserving the quality as much as possible. This research transfers sampling and approximate computing known from database query processing to the analysis of textual data.

Later work will be in the area of query-driven knowledge base generation and online processing. While the task of constructing structured knowledge bases from unstructured text is well-studied, current approaches have some shortcomings: Typically, they require teams of data scientists and engineers to manually (pre-)process data and produce the knowledge base. Yet the resulting knowledge base will only cover relations explicitly modelled during generation and cannot be used for free exploration of new data. A future aim of AIPHES in guiding theme D2 will be to develop new methods that provide end-users with the ability to explore new document collections interactively by building knowledge-bases on-the-fly in a query-driven manner and thus enabling exploratory scenarios.

People

• PI: Prof. Dr. Carsten Binnig, Dr. Christian M. Meyer
• First Cohort PhD student: P.V.S. Avinesh
• Second Cohort PhD student: Benjamin Hättasch.

References

• Louis, A. und Nenkova, A. (2013). Automatically Assessing Machine Summary Content Without a Gold Standard. Computational Linguistics, 39(2):267–300
• Lucie Flekova, Oliver Ferschke and Iryna Gurevych. 2014. What Makes a Good Biography? Multidimensional Quality Analysis Based on Wikipedia Article Feedback Data. In: Proceedings of the 23rd International World Wide Web Conference (WWW 2014), pp. 855-866, April 2014

Publications

go to TU-biblio search on ULB website