Abstract

With the rapid development of social media such as Twitter and Weibo, detecting keywords from a huge volume of text data streams in real-time has become a critical problem. The keyword detection problem aims at searching important information from massive text data to reflect the most important events or topics. However, social media data usually has very unique features: the documents are usually short, the language is colloquial, and the data is likely to be with significant temporal patterns. Therefore, it could be difficult to discover key information from these text streams. If keywords can be detected accurately in real-time, many practical problems can be addressed, such as detecting the occurrence of natural disasters at the earliest time, detecting potential hot events that are widely discussed by the public, etc.

In this paper, we propose a novel method to address the keyword detection problem in social media. Our model combines the TF-IDF and LDA models to better cope with the distinct attributes of social media data, such as hashtags, the number of likes, comments and retweets. By weighting the importance of each document based on these attributes, our method can effectively detect more representative keywords over time.

Comprehensive experiments conducted under various conditions on Twitter and Weibo data illustrate that our approach outperforms the baselines in various evaluation metrics including precision and recall for multiple problem settings.