Dual total correlation
In information theory, dual total correlation (Han 1978), excess entropy (Olbrich 2008), or binding information (Abdallah and Plumbley 2010) is one of the two known non-negative generalizations of mutual information. While total correlation is bounded by the sum entropies of the n elements, the dual total correlation is bounded by the joint-entropy of the n elements. Although well behaved, dual total correlation has received much less attention than the total correlation. A measure known as "TSE-complexity" defines a continuum between the total correlation and dual total correlation (Ay 2001).
Definition
For a set of n random variables , the dual total correlation is given by
where is the joint entropy of the variable set and is the conditional entropy of variable , given the rest.
Normalized
The dual total correlation normalized between [0,1] is simply the dual total correlation divided by its maximum value ,
Bounds
Dual total correlation is non-negative and bounded above by the joint entropy .
Secondly, Dual total correlation has a close relationship with total correlation, . In particular,
Relation to other quantities
In measure theoretic terms, by the definition of dual total correlation:
which is equal to the union of the pairwise mutual informations:
History
Han (1978) originally defined the dual total correlation as,
However Abdallah and Plumbley (2010) showed its equivalence to the easier-to-understand form of the joint entropy minus the sum of conditional entropies via the following:
See also
References
- Han T. S. (1978). Nonnegative entropy measures of multivariate symmetric correlations, Information and Control 36, 133–156.
- Fujishige Satoru (1978). Polymatroidal Dependence Structure of a Set of Random Variables, Information and Control 39, 55–72. doi:10.1016/S0019-9958(78)91063-X.
- Olbrich, E. and Bertschinger, N. and Ay, N. and Jost, J. (2008). How should complexity scale with system size?, The European Physical Journal B - Condensed Matter and Complex Systems. doi:10.1140/epjb/e2008-00134-9.
- Abdallah S. A. and Plumbley, M. D. (2010). A measure of statistical complexity based on predictive information, ArXiv e-prints. arXiv:1012.1890v1.
- Nihat Ay, E. Olbrich, N. Bertschinger (2001). A unifying framework for complexity measures of finite systems. European Conference on Complex Systems. pdf.