Information relates to uncertainty. The Shannon information content of an outcome x is h(x)=−log2P(x). The rare event has larger information than a common event. The unit of information is a bit (binary digit). Coding is a mapping from an outcome of an ensemble to binary digits {0,1}+. A symbol code is a code for a single ensemble. A block code is a code for a sequence ensemble. A set of sequences of the ensemble has a typical subset.
This is a note for Elements of information theory of Thomas M. Cover.
The entropy (H) is a measure of uncertainty of a variable which is the answer to what is the ultimate data compression. Is the conditional probability p(x|y) considered as a probability of the “conditional variable” (X|Y=y)? Yes, it is the subset of X given Y=y. If you sum all of the subset probabilities, it becomes the cardinality of X.