Moving object segmentation (MOS) is a very challenging task in various practical scenarios such as weather-degraded videos, dynamic background, etc. For security-based applications such as video traffic analysis and surveillance, MOS is a highly challenging job. In this paper, we propose an end-to-end recurrent network for MOS. The proposed network extracts robust features through the trail of multi-scale dense residual (MSDR) block where multi-scale convolution blocks shares feature learning through the effective dense connections. Also, we have employed feature refinement block to refine the features learned by the initial layers. Three frames with temporal sampling of two and recurring technique help to obtain MOS map. Comparison of qualitative and quantitative experimental findings with state-of the-art approaches shows the efficacy of the proposed MOS process on two benchmark datasets. The experimental analysis shows that the proposed network exceeds current state-of-the-art standards for MOS.