Personalized Recommendation with Multimodal Explanations

While it is perceived as one of the biggest benefits of online shopping, the extensive product selection also makes it easy for customers to feel overwhelmed and unable to find the products they are looking for. Personalized recommendation is key here. Current product recommendation approaches typically learn user preferences from clicks and purchases to provide personalized recommendations.
Such approaches typically overlook the visual and textual details of a product's web page(s), even though such details give away the preferences of the user. Furthermore, current recommender systems are static and provide no interaction with the user. Instead, recommender systems should be able to justify their recommendations to the user, by providing explanations that help the user understand why certain items are recommended. Such improved personalized recommendation with explanations would lead to better user-item matching and increase customer satisfaction.
In this project, I will develop novel deep learning-based methods for multimodal product recommendation combining visual and textual item data and metadata of a user and that reveal the textual and visual details that underlie the recommendations based on suitable attention and disentanglement mechanisms. They form the basis of a personalized and explainable recommender system that tells the user in a natural way, psychologically-informed way which visual and descriptive aspects of the product fit his or her profile.