Evaluating Representativeness in PDF Malware Datasets: A Comparative Study and a New Dataset

With the widespread use of the Portable Document Format (PDF), it’s increasingly becoming a target for malware, highlighting the need for effective detection solutions. In recent years, machine learning-based methods for PDF malware detection have grown in popularity. However, the effectiveness of ML models is closely related to the quality of the training datasets. In this research, we investigated two widely used PDF malware datasets: Contagio and CIC. We found biases and representativeness issues that could affect the reliability and applicability of models built on them. Our statistical analysis revealed marked difference between these datasets and PDF malware samples from VirusTotal, as well as benign PDFs from Govdocs, pointing to the necessity for more representative datasets in PDF malware research.. To address this gap, we introduce a novel dataset: PdfRep. Our findings demonstrate that PdfRep outperforms both CIC and Contagio across various evaluation metrics. The main contribution of this paper is the introduction of PdfRep, a new PDF malware dataset that overcomes the limitations of representativeness in existing datasets. This enhancement substantially increases the accuracy of PDF malware detection models and holds promise for advancing the field of PDF malware detection research.