Recent advances in deep learning techniques such as convolutional neural networks, recurrent neural network, and generative adversarial networks have achieved breakthroughs in many fields or in many real-world applications. For the problem of semantic image inpainting, the task of reconstructing meaningful missing pixels also demonstrates that deep neural networks can play an effective role. While more effective than conventional approaches, deep learning models demand high storage capacity, since the many layers and parameters involved in neural network construction incur great space complexity and the end-to-end system incorporating these inpainting modules need to reserve a significant storage to save the model alone. Additionally, neural network training often requires large datasets consisting of high dimensional images, thereby requiring intensive computational resources such as the GPU. Furthermore, one problem that we may need to consider is that the inpainting quality of the images may vary considerably across different contexts because the foundational training data where the inpainting model is trained on differ in size and diversity. To address these problems, we present an inpainting strategy called comparative sample augmentation, which enhances the quality of the training set by filtering irrelevant images and constructing additional images using information about the surrounding regions of the target image and this strategy managed to augment the datasets. Experiments on multiple datasets demonstrate that our method extends the applicability of deep inpainting models to training sets with varying levels of diversity, while enhancing the inpainting quality as measured by qualitative and quantitative metrics for a large range of class of deep models, with little need for model-specific consideration.