Course 4541.676 (and 132.551)
Information Theory of Learning
 

School of Computer Science and Engineering,
Seoul National University

Instructor     Prof. Byoung-Tak Zhang
TA     (Tel: 02-880-1847, Room: 301-419)
Classroom     302-106
Time     Tue, 09:00 ~ 10:15 and Thur, 09:00 ~ 10:15
Objectives
  • To understand the role of information theory in machine learning
  • To learn the mathematics of information theory and coding theory
  • To understand the recently-developed probabilistic library model (PLM) for
         molecular learning
  • To analyze the molecular computational learning processes by information theory
  • To design improved molecular learning algorithms for PLM based on information
         theory
    •
    Textbook
    Information Theory, Inference, and Learning Algorithms
    David, J. C. MacKay, Cambridge University Press, 2004
    Reference
  • Leff, H.S. and Rex, A.F. (Eds), Maxwell's Demon: Entropy, Information,
         Computing, IOP Publishing, 1990. [LR-90]
  • Evolving Optimal Neural Networks Using GAs with Occam's Razor [ZM-93]
  • System Identification Using Evolutionary Markov Chain Monte Carlo [ZC-01]
  • Bio-Molecular Computer Technology [Z-03-BMC]
  • A Unified Bayesian Framework for Evolutionary Learning and Optimization
         [Z-03-AEC]
  • A Bayesian Algorithm for In Vitro Molecular Evolution of Pattern Classifiers
         [ZJ-04]
  • Molecular Nanobiointelligence Computers [Z-05-NBIC]
  • Hyperinteractionism [Z-05-Hyper]
  • Molecular Learning of wDNF Formulae [ZJ-05]
    • Evaluation
  • Term project
  • Two essays
       - Books for essays (choose two chapters, one from each book):
       - Brockman, John (Ed.), Science at the Edge, Wedenfeld & Nicoloson, 2004.
       - Forbes, Nancy, Imitation of Life: How Biology Is Inspiring Computing, MIT Press,
         2004.
  • One exam
    •

    • Notice
     Term Project 문제 및 목표 설명: PDF

     H/W #1: "Exercise 2.16 - 2.29" Due: Sep. 27
     H/W #2: "Exercise 8.1 - 8.8" Due: Oct. 6
     Essay #1: "Science at the Edge" Due: Oct. 6
     Essay #2: "Imitation of Life" Due: Oct. 27

    • Schedule
        Title     Reference
        Physics of Computation     [LR-90]
        Molecular Computing: An Introduction     [Z-03-BMC]
        Chemical Learning with DNA Molecules     [Z-05-NBIC]
        Hyperinteractionism: Brain in a Test Tube     [Z-05-Hyper]
        Probabilistic Library Model (PLM)     [ZJ-04, ZJ-05]
        A Classification Dataset for PLM (PPT, DATA)     
        Project I: Pattern Classification by the PLMs  Due: Oct. 4 / Oct. 20
        Probability, Entropy, and Inference     [Ch. 2, Ch. 8]
        Data Compression     [Ch. 4]
        Hopfield Networks     [Ch. 42]
        Variational Methods     [Ch. 33]
        Boltzmann Machines     [Ch. 43]
        Pattern Completion Results by Bayesian Networks (DATA)     
        Project II: Pattern Completion by the Correlational PLMs  Due: Nov. 8 / Nov. 15
        Midterm Exam  Nov. 15
        Graphical Models & PLM     
        Occam's Razor     [Ch. 28, ZM-93]
        Monte Carlo Methods     [Ch. 29]
        Evolutionary MCMC     [ZC-01, Z-03-AEC, Ch. 22]
        Project III: Pattern Prediction by the Latent PLMs  Due: Dec. 1 / Dec. 8
        Noisy-Channel Coding     [Ch. 9, Ch. 10]
        Independent Component Analysis     [Ch. 34]
        Latent Variable Models     [Ch. 34]
        Gaussian Processes     [Ch. 45]
        Kernel Methods & PLM     
        Project Presentations  Dec. 10


    This page is maintained by
    Last update: December 21, 2005.