# Information theory

## Information

Information relates to uncertainty. The Shannon information content of an outcome $$x$$ is $$h(x)=-log_{2}P(x)$$. The rare event has larger information than a common event. The unit of information is a bit (binary digit). Coding is a mapping from an outcome of an ensemble to binary digits $$\{0,1\}^+$$. A symbol code is a code for a single ensemble. A block code is a code for a sequence ensemble. A set of sequences of the ensemble has a typical subset.

## Entropy

This is a note for Elements of information theory of Thomas M. Cover. The entropy ($$H$$) is a measure of uncertainty of a variable which is the answer to what is the ultimate data compression. Is the conditional probability $$p(x|y)$$ considered as a probability of the “conditional variable” $$(X|Y=y)$$? Yes, it is the subset of $$X$$ given $$Y=y$$. If you sum all of the subset probabilities, it becomes the cardinality of $$X$$.