ABSTRACT
The field of information theory has its origin in Claude Shannon's 1948 paper, “A Mathematical Theory of Communication.” Shannon's motivation was to study “[The problem] of reproducing at one point either exactly or approximately a message selected at another point.” In this chapter, we will be concerned only with Shannon's original problem. One should keep in mind that information theory is a growing field of research whose profound impact has reached various areas such as statistical physics, computer science, statistical inference, and probability theory. For an excellent treatment of information theory that extends beyond the area of communication, we recommend Cover and Thomas (1991). For the reader who is strictly interested in communication problems, we also recommend Gallager (1968), Blahut (1987), and McEliece (1977).
