Directed information

Directed information, ${\displaystyle I(X^{n}\to Y^{n})}$, is a measure of information theory and it measures the amount of information that flows from the process $X^{n}$ to $Y^{n}$, where $X^{n}$ denotes the vector $X_{1},X_{2},...,X_{n}$ and $Y^{n}$ denotes ${\displaystyle Y_{1},Y_{2},...,Y_{n}}$. The term "directed information" was coined by James Massey and is defined as

${\displaystyle I(X^{n}\to Y^{n})=\sum _{i=1}^{n}I(X^{i};Y_{i}|Y^{i-1})}$,

where ${\displaystyle I(X^{i};Y_{i}|Y^{i-1})}$ is the conditional mutual information.

Note that if $n=1$, directed information becomes mutual information $I(X; Y)$. Directed information has many applications in problems where causality plays an important role such as capacity of channel with feedback,^[1][2] capacity of discrete memoryless networks with feedback,^[3] gambling with causal side information,^[4] and compression with causal side information.^[5]

References

1. Massey, James (1990). "Causality, Feedback And Directed Information" (ISITA).
2. Permuter, Haim Henry; Weissman, Tsachy; Goldsmith, Andrea J. (February 2009). "Finite State Channels With Time-Invariant Deterministic Feedback". IEEE Transactions on Information Theory. 55 (2): 644–662. doi:10.1109/TIT.2008.2009849.
3. Kramer, G. (January 2003). "Capacity results for the discrete memoryless network". IEEE Transactions on Information Theory. 49 (1): 4–21. doi:10.1109/TIT.2002.806135.
4. Permuter, Haim H.; Kim, Young-Han; Weissman, Tsachy (June 2011). "Interpretations of Directed Information in Portfolio Theory, Data Compression, and Hypothesis Testing". IEEE Transactions on Information Theory. 57 (6): 3248–3259. doi:10.1109/TIT.2011.2136270.
5. Simeone, Osvaldo; Permuter, Haim Henri (June 2013). "Source Coding When the Side Information May Be Delayed". IEEE Transactions on Information Theory. 59 (6): 3607–3618. doi:10.1109/TIT.2013.2248192.