Title: A new model driven architecture for deep learning-based multimodal lifelog retrieval
Authors: Ben Abdallah, Fatma
Feki, Ghada
Ben Ammar, Anis
Ben Amar, Chokri
Citation: WSCG 2018: poster papers proceedings: 26th International Conference in Central Europe on Computer Graphics, Visualization and Computer Visionin co-operation with EUROGRAPHICS Association, p. 8-17.
Issue Date: 2018
Publisher: Václav Skala - UNION Agency
Document type: konferenční příspěvek
URI: wscg.zcu.cz/WSCG2018/!!_CSRN-2803.pdf
ISBN: 978-80-86943-42-8
ISSN: 2464-4617
Keywords: multimodalita;vyhledávání;shrnutí;vizualizace;konvoluční neuronová síť;relační síť
Keywords in different language: multimodality;retrieval;summarization;visualization;convolutional neural network;relational network
Abstract: Nowadays, taking photos and recording our life are daily task for the majority of people. The recorded information helped to build several applications like the self-monitoring of activities, memory assistance and long-term assisted living. This trend, called lifelogging, interests a lot of research communities such as computer vision, machine learning, human-computer interaction, pervasive computing and multimedia. Great effort have been made in the acquisition and the storage of captured data but there are still challenges in managing, analyzing, indexing, retrieving, summarizing and visualizing these captured data. In this work, we present a new model driven architecture for deep learning-based multimodal lifelog retrieval, summarization and visualization. Our proposed approach is based on different models integrated in an architecture established on four phases. Based on Convolutional Neural Network, the first phase consists of data preprocessing for discarding noisy images. In a second step, we extract several features to enhance the data description. Then, we generate a semantic segmentation to limit the search area in order to better control the runtime and the complexity. The second phase consist in analyzing the query. The third phase which based on Relational Network aims at retrieving the data matching the query. The final phase treat the diversity-based summarization with k-means which offers, to lifelogger, a key-frame concept and context selection-based visualization.
Rights: © Václav Skala - Union Agency
Appears in Collections:WSCG 2018: Poster Papers Proceedings

Files in This Item:
File Description SizeFormat 
Abdallah.pdfPlný text1,33 MBAdobe PDFView/Open

Please use this identifier to cite or link to this item: http://hdl.handle.net/11025/34632

Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.