Garbage, Glitter, or Gold: Assigning Multi-Dimensional Quality Scores to Social Media Seeds for Web Archive Collections

From popular uprisings to pandemics, the Web is an essential source consulted by scientists and historians for reconstructing and studying past events. Unfortunately, the Web is plagued by link rot and content drift (reference rot) which causes important Web resources to disappear. Web archive collections help reduce the costly effects of reference rot by saving Web resources that chronicle important stories and events before they disappear. These collections often begin with URLs called seeds, hand-selected by experts or scraped from social media posts. The quality of social media content content varies widely, therefore, we propose a framework for assigning multidimensional quality scores to social media seeds for Web archive collections about stories and events. We leveraged contributions from social media research for attributing quality to social media content and users based on credibility, reputation, and influence. We combined these with additional contributions from the Web archive research that emphasizes the importance of considering geographical and temporal constraints when selecting seeds. Next, we developed the Quality Proxies (QP) framework which assigns seeds extracted from social media a quality score across 10 major dimensions: popularity, geographical, temporal, subject-expert, retrievability, relevance, reputation, and scarcity. We instantiated the framework and showed that seeds can be scored across multiple QP classes that map to different policies for ranking seeds such as prioritizing seeds from local news, reputable and/or popular sources, etc. The QP framework is extensible and robust; seeds can be scored when a subset of the QP dimensions are absent. Most importantly, scores assigned by Quality Proxies are explainable, providing the opportunity to critique them. Our results showed that Quality Proxies resulted in the selection of quality seeds with increased precision (by ≈0.13) when novelty is and is not prioritized. These contributions provide an explainable score applicable to rank and select quality seeds for Web archive collections and other domains that select seeds from social media.