Latent factor models such as Matrix Factorization have become the default choice for recommender systems due to their performance and scalability. However, such algorithms have two disadvantages. First, these models suffer from data sparsity. Second, they fail to account for model uncertainty. In this paper, we exploit a meta learning strategy to address these problems. The key idea behind our method is to learn predictive distributions conditioned on context sets of arbitrary size of user/item interaction information. Our proposed framework has the advantages of being easy to implement and applicable to any existing latent factor models, providing uncertainty capabilities. We demonstrate the significant superior performance of our model over previous state-of-the-art methods, especially for sparse data in the top-N recommendation task.