520.447 Intro to Info Theory and Coding

2003-05 Catalog: This course will address some basic scientific questions about systems that store or communicate information. Mathematical models will be developed for (1) the process of error-free data compression leading to the notion of entropy, (2) data (e.g. image) compression with slightly degraded reproduction leading to rate-distortion theory and (3) error-free communication of information over noisy channels leading to the notion of channel capacity. It will be shown how these quantitative measures of information have fundamental connections with statistical physics (thermodynamics), computer science (string complexity), economics (optimal portfolios), probability theory (large deviations) and statistics (Fisher information, hypothesis testing). (3 credit hours)

Prerequisite(s): 550.310 Probability and Statistics for the Physical and Information Sciences and Engineering

Textbook: Cover, Thomas, Elements of Information Theory, Wiley, 1991.

Course Objectives: To understand information measures and their implications for the design and analysis of communication and data storage systems, and to see their fundamental connections to statistical inference and learning theory.

Topics Covered:
1. Entropy, relative entropy, and mutual information
2. Asymptotic Equipartition Property
3. Entropy rates of stochastic processes
4. Noiseless data comprehension
5. Optimal gambling
6. Channel capacity for discrete memoryless channels
7. Differential entropy
8. Gaussian channels
9. Rate distortion theory and quantization
10. Information theoretic methods in statistics

Class Schedule: Three - one hour classes weekly

Instructor: Frederick Jelinek

ABET page

pdf file (requires Adobe Reader)

06/14/04