An ""Infodemic"": Leveraging High-Volume Twitter Data to Understand Public Sentiment for the COVID-19 Outbreak

Richard J. Medford; Sameh N. Saleh; Andrew Sumarsono; Trish M. Perl; Christoph U. Lehmann

A Latent Dirichlet Allocation (LDA) [12] model (gensim Python package [16] ) automatically generates topics from observations (in our case, from tweets) and groups similar observations to one or more of these topics using the distribution of words. We iteratively trained multiple LDA models using different numbers of topics to maximize a topic coherence score (which measures the degree of semantic similarity between high scoring words in the topic). Selecting the highest coherence score resulted in the use of the LDA model with ten topics. Adhering to convention, we presented the top fifteen terms (a common number of terms used in analyzing topics in LDA models) that contributed to each topic group and manually labeled a theme for each topic ( Figure 6a ). We then visualized the topic model using a t-distributed Stochastic Neighbor Embedding (t-SNE) graph [17], which embeds high-dimensional data (i.e., ten dimensions given ten topics) into a graphable two-dimensional space where similar tweets are grouped together ( Figure 6b ). We created an interactive visualization of the t-SNE to qualitatively evaluate the change in topics over time.