A Cross-Platform Malware Classification Using Domain-Aware Representation Learning

Malware analysis across heterogeneous platforms such as Android and Windows presents significant challenges due to fundamental differences in executable structure, obfuscation strategies, and malware family characteristics. Existing malware classification approaches are largely based on end-to-end deep learning models trained on homogeneous datasets, limiting their scalability, interpretability, and robustness in cross-platform settings. In this work, we propose a domain-aware neural-classical hybrid Mixture-of-Experts framework for malware classification using image representations of binaries. The proposed framework decouples representation learning, domain identification, and family classification into three modular stages. A lightweight convolutional neural network learns shared malware embeddings, a neural domain classifier performs routing between platforms, and domain-specialized classical machine learning models act as classification experts. Extensive experiments on Android and Windows malware datasets demonstrate that perfect domain separation can be achieved at the embedding level, enabling effective domain-filtered inference. Results show that Random Forest experts significantly outperform Support Vector Machines and k-Nearest Neighbors under domain specialization, achieving up to 91% F1-score for Android malware families and 94% F1score for Windows malware families. The modular design further enables interpretability through routing confidence analysis and embedding visualization.