Out-VM monitoring for malicious network packet detection in cloud

Abstract

<p>Cloud security is one of the biggest challenge in today's technological world. Researchers have proposed some solutions for cloud security. Virtual Machine (VM)-level solutions are configured and controlled at VM. They are less robust and can be easily subverted by attackers. In this paper, we propose an out-VM monitoring security approach named as Malicious Network Packet Detection (MNPD) which monitors the VMs from outside at both network and virtualization layer in cloud environment. MNPD performs the behavioral analysis of network traffic at Cloud Networking Server (CNS); providing primary defense from intrusions at network level. MNPD does the VM traffic validation at hypervisor of Cloud Compute Server (CCoS) to detect spoofing attacks, originated from VMs. The non-spoofed packets are further analyzed using behavior analysis of network traffic to detect any abnormality in the virtual traffic; providing second level of defense from intrusions at virtualization level. MNPD employs statistical learning technique (Random Forest) with ensemble of feature selection approach to learn the behavior of traffic patterns. MNPD does not involve overhead incurred in monitoring extensive memory writes or instruction-level traces. It is a more secure solution to detect attacks which never pass through physical interface and hence not detected by traditional IDS. The proposed approach has been validated with latest datasets (UNSW-NB and ITOC) and results seem to be promising.</p>