Malicious programs are increasing abnormally, affecting our everyday lives. Modern sophisticated and agile malware programs are not always detected by traditional malware detection methods that use signature-based techniques. As a result, researchers use behaviorbased techniques to analyze malware behaviors (features). However, malware features derived from behavioral analysis commonly suffer from high dimensionality. Accordingly, this work applies the Whale Optimization Algorithm (WOA) to find the optimalsubset of features in the CIC-MalMem-2022 dataset. Feature selection contributes significantly to reducing high-dimensionality issues and improving malware detection performance. WOA is employed to enhance the efficiency of the selection process for the optimal features and determine the most advantageous set of features by omitting redundant and irrelevant features. In addition, we apply the K-nearest neighbor algorithm (KNN) to detect malware. Using WOA and KNN, this study improves the detection efficiency of CIC-MalMem-2022. According to the findings, the proposed method outperforms existing malware detection systems, including detection fitness value, accuracy, consuming time, and the number of selected features.