MARS:

A Multimodal Associative Recommendation System


¢Æ Overall Concept ¢Æ

Recommendation underlies many internet and web services. We develop novel recommendation techniques that simulate human cognitive memory, i.e. crossmodal associative recall between vision and language. We use machine learning techniques to convert between image and text using a corpus of articles containing images. Combined with user lifelog and social data, this technology provides personalized crossmodal recommendation services in a mobile environment using smartphones and tablets. This work is supported by the IT R&D Program of Ministry of Knowledge Economy under KEIT.

 ¢Æ R&D Objectives (year by year) ¢Æ

Year

Subtitle

Objectives

2009

Research on Information Extraction
in Multimodal Richmedia

  • Attribute definition and relation summarization of complex information of image, movie and text data
  • Framework development for compounding descriptors
  • Mutual generation using image-text cross modality information based on machine learning

2010

Research on Context-based Information Extraction in Richmedia

  • Methods for context-based extraction of compound information and descriptor generation
  • Cross-modal context analysis in images and movies
  • Multimodal topic modeling with compound information extracted from richmedia
  • Testing service of online article-mall connection system

2011

Research on Information Extraction of Richmedia in Dynamic Environments

  • Learning and modeling compound information in time-varying and space-varying data
  • Interactive analysis of compound information in richmedia in incremental way
  • Incremental social analysis by multimodal topic models and its application to microblog analysis

2012

Research on Recommendation Methods based on Multimodal Associativity

  • Multimodal-associative modeling of user preferences in richmedia
  • Interactive recommendation methods in dynamic richmedia environment
  • Multimodal interactive article-mall connection system with user preference catch and context recognition

2013

Development of MARS and Its Application to Adaptive Recommendation

  • Recommendation engine based on multimodal association and user preference modeling
  • Constructing the framework system of multimodal associative recommendation system (MARS)
  • Personalized/adaptive richmedia recommendation system

 

 ¢Æ Target system to be developed: MARS  ¢Æ

¢Æ Publications ¢Æ

  • Layered hypernetwork models for cross-modal associative text and image keyword generation in multimodal information retrieval, J.-W. Ha, B.-H. Kim, B. Lee, and B.-T. Zhang, Proceedings of the Eleventh Pacific Rim International Conference on AI (PRICAI2010), 2010.
  • Visual query expansion via incremental hypernetwork models of image and text, M.-O. Heo, M.-G. Kang, and B.-T. Zhang, Proceedings of the Eleventh Pacific Rim International Conference on AI (PRICAI2010), 2010.
  • Text-to-image cross-modal retrieval of magazine articles based on higher-order pattern recall by hypernetworks, J.-W. Ha, B.-H. Kim, H.-W. Kim, W. C. Yoon, J.-H. Eom, and B.-T. Zhang, The 10th International Symposium on Advanced Intelligent Systems (ISIS 2009), pp. 274-277, 2009. (Best paper award)
  • Evolutionary hypernetworks for learning to generate music from examples, H.-W. Kim, B.-H. Kim, and B.-T. Zhang IEEE International Conference on Fuzzy Systems (FUZZ-IEEE 2009), pp. 47-52, 2009.
  • Gender classification with cortical thickness measurement, J.-W. Ha, J.H. Jang, D.-H. Kang, W.H. Jung, J.S. Kwon, and B.-T. Zhang, IEEE International Conference on Fuzzy Systems (FUZZ-IEEE 2009), pp. 41-46, 2009.
  • Learning word sense disambiguation in biomedical text with difference between training and test distributions, J.-W. Son and S.-B. Park, In Proc. of the ACM Third International Workshop on Data and Text Mining in Bioinformatics, 2009.
  • A weighting scheme for tag recommendation in social bookmarking systems, S. Ju and K.-B. Hwang, In Proceedings of ECML/PKDD Discovery Challenge 2009. (To appear)
  • Evolutionary hypernetwork classifiers for protein-protein interaction sentence filtering, J. Bootkrajang, S. Kim, and B.-T. Zhang, The Genetic and Evolutionary Computation Conference (GECCO 2009), pp. 185-191, 2009.
  • An empirical study of choosing efficient discriminative seeds for oligonucleotide design, W.-H. Chung and S.-B. Park, In Proc. of Eighth the International Conference on Bioinformatics, 2009.
  • Auto-tagging method for unlabeled item images with hypernetworks for article-related item recommender systems, J.-W. Ha, B.-H. Kim, B. Lee, and B.-T. Zhang, Journal of the Korean Institute of Information Scientists and Engineers: Computing Practices and Letters, 16(10):1010-1014, 2010.
  • Bi-Source ÅäÇÈ ¸ðµ¨ ±â¹ýÀ» ÀÌ¿ëÇÑ ±â»ç-»óǰ ¿¬°ü °Ë»ö, ±èº´Èñ, À̹ٵµ, ÇϼºÁ¾, Á¶³²ÀÍ, À庴Ź, Çѱ¹Á¤º¸°úÇÐȸ °¡À»Çмú¹ßÇ¥ ³í¹®Áý, Á¦37±Ç 2(A), pp. 74-75, 2010.11.
  • ÀâÁö±â»ç °ü·Ã »óǰ ¿¬°è Ãßõ ¼­ºñ½º¸¦ À§ÇÑ ÇÏÀÌÆÛ³×Æ®¿öÅ© ±â¹ÝÀÇ »óǰÀ̹ÌÁö ÀÚµ¿ ÅÂ±ë ±â¹ý,ÇÏÁ¤¿ì,±èº´Èñ,À̹ٵµ,À庴Ź, 2010 Çѱ¹ÄÄÇ»ÅÍÁ¾ÇÕÇмú´ëȸ(KCC2010) ³í¹®Áý, Á¦37±Ç 1(A), pp. 123-124, 2010.06.
  • ·£´ý ÇÏÀÌÆÛ±×·¡ÇÁ ¸ðµ¨À» ÀÌ¿ëÇÑ ¼øÂ÷Àû ¸ÖƼ¸ð´Þ µ¥ÀÌÅÍ¿¡¼­ÀÇ ¹®Àå »ý¼º, À±¿õâ,À庴Ź, 2010 Çѱ¹ÄÄÇ»ÅÍÁ¾ÇÕÇмú´ëȸ(KCC2010) ³í¹®Áý, Á¦37±Ç 1(C), pp. 376-379, 2010.06.
  • ÁøÈ­ ÇÏÀÌÆÛ³×Æ®¿öÅ©¸¦ ÀÌ¿ëÇÑ À½¾Ç ÇнÀ ¹× Å©·Î½º¿À¹ö À½¾Ç »ý¼º, ±èÇö¿ì, ±èº´Èñ, À庴Ź, 2009 Çѱ¹ÄÄÇ»ÅÍÁ¾ÇÕÇмú´ëȸ³í¹®Áý, Á¦36±Ç 1(A), pp. 134-138, 2009.07.
  • ÇÏÀÌÆÛ¸Á ¸Þ¸ð¸® ±â¹Ý À¯¾Æ ¾ð¾îÇнÀ ¹× »ý¼º ¸ðµ¨, ÀÌÁöÈÆ, ÀÌÀº¼®, À庴Ź, 2009 Çѱ¹ÄÄÇ»ÅÍÁ¾ÇÕÇмú´ëȸ³í¹®Áý, Á¦36±Ç 1(A), pp. 128-129, 2009.07.

Project Title

Multimodal Information Extraction and Recommendation Technologies for Next-Generation Customized Services Based on Machine Learning

Duration

March 2009 - February 2014

Funding

The Ministry of Knowledge and Economy &
Korea Evaluation Institute of Industrial Technology

Principal Investigator

Prof. Byoung-Tak Zhang

Researchers

Prof. Nam Ik Cho

Byoung-Hee Kim

WoongChang Yoon

Seong-Jong Ha

Jung-Woo Ha

Bado Lee

Collaborative Labs

ISPL Laboratory, Seoul National University
Digital Design House Co. Ltd (Storysearch & Storyshop)


Contact

Byoung-Hee Kim

E-Mail

bhkim -at- bi snu ac kr

Phone

+82-2-880-1847

Fax

+82-2-875-2240