Identifying the different stages of Multiple Sclerosis (MS) is essential for accurately predicting disease progression, often requiring the capabilities of deep learning networks. The spatial and temporal dependencies within image-based biomarkers frequently present irregular capture challenges across MRI scans taken at different time points, primarily due to unsynchronized multi-scale features and their corresponding network channels. Consequently, this may lead to false predictions of disease progression and inaccurate MS lesion segmentation. In this paper, we propose a novel deep learning pipeline that integrates advanced modules, namely Gated Recurrent Unit (GRU), Atrous Spatial Pyramid Pooling (ASPP), and Squeeze and Excitation (SE) blocks, within a U-Net architecture to effectively manage multi-scale features and channel relationships. It is simply abbreviated as GRU-ASPP U-Net. The GRU modules at skip connections enable temporal feature capture, the ASPP module in the bottleneck layer facilitates multi-scale feature extraction, and SE blocks perform channel-wise feature recalibration. Evaluation metrics, including the Jaccard Coefficient (IoU) and Dice Coefficient, indicate that the proposed pipeline achieves promising prediction outcomes, with a Dice score of 0.87 and IoU of 0.79 (Fold 3). Moreover, this approach combines spatial and temporal analysis, providing a more stringent measure of segmentation accuracy and enhanced sensitivity to subtle overlaps in MS lesion segmentation.