Theil index

The Theil index is a statistic primarily used to measure economic inequality^[1] and other economic phenomena, though it has also been used to measure racial segregation.^[2] The basic Theil index T_T is the same as redundancy in information theory which is the maximum possible entropy of the data minus the observed entropy. It is a special case of the generalized entropy index. It can be viewed as a measure of redundancy, lack of diversity, isolation, segregation, inequality, non-randomness, and compressibility. It was proposed by econometrician Henri Theil at the Erasmus University Rotterdam.

Formula

For a population of N "agents" each with characteristic x, the situation may be represented by the list x_i (i=1,...,N) where x_i is the characteristic of agent i. For example, if the characteristic is income, then x_i is the income of agent i. The Theil index is defined as:^[3]

${\displaystyle T_{T}=T_{\alpha =1}={\frac {1}{N}}\sum _{i=1}^{N}\,{\frac {x_{i}}{\mu }}\ln \left({\frac {x_{i}}{\mu }}\right)}$

where $\mu$ is the mean income:

${\displaystyle \mu ={\frac {1}{N}}\sum _{i=1}^{N}x_{i}}$

Equivalently, if the situation is characterized by a discrete distribution function f_k (k=0,...,W) where f_k is the fraction of the population with income k and W = Nμ is the total income, then ${\displaystyle \sum _{k=0}^{W}f_{k}=1}$ and the Theil index is:

${\displaystyle T_{T}=\sum _{k=0}^{W}\,f_{k}\,{\frac {k}{\mu }}\ln \left({\frac {k}{\mu }}\right)}$

where $\mu$ is again the mean income:

${\displaystyle \mu =\sum _{k=0}^{W}kf_{k}}$

Note that in this case income k is an integer and k=1 represents the smallest increment of income possible (e.g., cents).

if the situation is characterized by a continuous distribution function f(k) (supported from 0 to infinity) where f(k)dk is the fraction of the population with income k to k+dk, then the Theil index is:

${\displaystyle T_{T}=\int _{0}^{\infty }f(k){\frac {k}{\mu }}\ln \left({\frac {k}{\mu }}\right)dk}$

where the mean is:

${\displaystyle \mu =\int _{0}^{\infty }kf(k)dk}$

Theil indices for some common continuous probability distributions are given in the table below:

Income Distribution function	PDF(x) (x≥0)	Theil Coefficient (nats)
Dirac delta function	${\displaystyle \delta (x-x_{0}),\,x_{0}>0}$	0
Uniform distribution	${\begin{cases}{\frac {1}{b-a}}&a\leq x\leq b\\0&\mathrm {otherwise} \end{cases}}$	${\displaystyle \ln \left({\frac {2a}{(a+b){\sqrt {e}}}}\right)+{\frac {b^{2}}{b^{2}-a^{2}}}\ln(b/a)}$
Exponential distribution	$\lambda e^{-x\lambda },\,\,x>0$	${\displaystyle 1-}$ $\gamma$
Log-normal distribution	${\displaystyle {\frac {1}{\sigma {\sqrt {2\pi }}}}e^{\frac {-(\ln \,(x)-\mu )^{2}}{\sigma ^{2}}}}$	${\displaystyle {\frac {\sigma ^{2}}{2}}}$
Pareto distribution	${\begin{cases}{\frac {\alpha k^{\alpha }}{x^{\alpha +1}}}&x\geq k\\0&x<k\end{cases}}$	${\displaystyle \ln(1\!-\!1/\alpha )+{\frac {1}{\alpha -1}}}$    (α>1)
Chi-squared distribution	${\frac {2^{-k/2}e^{-x/2}x^{k/2-1}}{\Gamma (k/2)}}$	${\displaystyle \ln(2/k)+}$ $\psi ^{{(0)}}$${\displaystyle (1\!+\!k/2)}$
Gamma distribution	${\frac {e^{-x/\theta }x^{k-1}\theta ^{-k}}{\Gamma (k)}}$	$\psi ^{{(0)}}$${\displaystyle (1\!+\!k)-\ln(k)}$
Weibull distribution	${\displaystyle {\frac {k}{\lambda }}\,\left({\frac {x}{\lambda }}\right)^{k-1}e^{-(x/\lambda )^{k}}}$	${\frac {1}{k}}$ $\psi ^{{(0)}}$${\displaystyle (1\!+\!1/k)-\ln \left(\Gamma (1+1/k)\right)}$

If everyone has the same income, then T_T equals 0. If one person has all the income, then T_T gives the result $\ln N$, which is maximum inequality. Dividing T_T by $\ln N$ can normalize the equation to range from 0 to 1, but then the independence axiom is violated: ${\displaystyle T[x\cup x]\neq T[x]}$ and does not qualify as a measure of inequality.

The Theil index measures an entropic "distance" the population is away from the egalitarian state of everyone having the same income. The numerical result is in terms of negative entropy so that a higher number indicates more order that is further away from the complete equality. Formulating the index to represent negative entropy instead of entropy allows it to be a measure of inequality rather than equality.

Derivation from entropy

The Theil index is derived from Shannon's measure of information entropy $S$, where entropy is a measure of randomness in a given set of information. In information theory, physics, and the Theil index, the general form of entropy is

$S=k\sum _{i=1}^{N}\left(p_{i}\log {\frac {1}{p_{i}}}\right)=-k\sum _{i=1}^{N}\left(p_{i}\log {p_{i}}\right)$

where $p_{i}$ is the probability of finding member $i$ from a random sample of the population. In physics, $k$ is Boltzmann's constant. In information theory, when information is given in binary digits, $k=1$ and the log base is 2. In physics and also in computation of Theil index, the natural logarithm is chosen as the logarithmic base. When $p_{i}$ is chosen to be income per person $x_{i}$, it needs to be normalized by dividing by the total population income, $N{\overline {x}}$. This gives the observed entropy $S_{\text{Theil}}$ of a population to be:

$S_{\text{Theil}}=\sum _{i=1}^{N}\left({\frac {x_{i}}{N{\overline {x}}}}\ln {\frac {N{\overline {x}}}{x_{i}}}\right)$

The Theil index is $T_{T}=S_{\text{max}}-S_{\text{Theil}}$ where $S_{\text{max}}$ is the theoretical maximum entropy that is reached when all incomes are equal, i.e. $x_{i}={\overline {x}}$ for all $i$. This is substituted into $S_{\text{Theil}}$ to give $S_{\text{max}}=\ln N$, a constant determined solely by the population. So the Theil index gives a value in terms of an entropy that measures how far $S_{\text{Theil}}$ is away from the "ideal" $S_{\text{max}}$. The index is a "negative entropy" in the sense that it gets smaller as the disorder gets larger, hence it is a measure of order rather than disorder.

When $x$ is in units of population/species, $S_{\text{Theil}}$ is a measure of biodiversity and is called the Shannon index. If the Theil index is used with x=population/species, it is a measure of inequality of population among a set of species, or "bio-isolation" as opposed to "wealth isolation".

The Theil index measures what is called redundancy in information theory.^[3] It is the left over "information space" that was not utilized to convey information, which reduces the effectiveness of the price signal. The Theil index is a measure of the redundancy of income (or other measure of wealth) in some individuals. Redundancy in some individuals implies scarcity in others. A high Theil index indicates the total income is not distributed evenly among individuals in the same way an uncompressed text file does not have a similar number of byte locations assigned to the available unique byte characters.

Notation	Information theory	Theil index TT
$N$	number of unique characters	number of individuals
$i$	a particular character	a particular individual
$x_{i}$	count of ith character	income of ith individual
$N{\overline {x}}$	total characters in document	total income in population
$T_{T}$	unused information space	unused potential in price mechanism
	data compression	progressive tax

Decomposability

One of the advantages of the Theil index is that it is a weighted average of inequality within subgroups, plus inequality between those subgroups. For example, inequality within the United States is the average inequality within each state, weighted by state income, plus the inequality between states.

If for the Theil index the population is divided into $m$ subgroups and $s_{i}$ is the income share of group $i$, $T_{{Ti}}$ is the Theil index for that subgroup, and ${\overline {x}}_{i}$ is the average income in group $i$, then the Theil index is

$T_{T}=\sum _{i=1}^{m}s_{i}T_{T_{i}}+\sum _{i=1}^{m}s_{i}\ln {\frac {{\overline {x}}_{i}}{\overline {x}}}$

Note: This image is not the Theil Index in each area of the United States, but of contributions to the US Theil Index by each area (the Theil Index is always positive, individual contributions to the Theil Index may be negative or positive).

The decomposition of the overall Theil index which identifies the share attributable to the between-region component becomes a helpful tool for the positive analysis of regional inequality as it suggests the relative importance of spatial dimension of inequality.^[4]

The decomposability is a property of the Theil index which the more popular Gini coefficient does not offer. The Gini coefficient is more intuitive to many people since it is based on the Lorenz curve. However, it is not easily decomposable like the Theil.

Applications

In addition to multitude of economic applications, the Theil index has been applied to assess performance of irrigation systems^[5] and distribution of software metrics.^[6]

See also

• Generalized entropy index
• Atkinson index
• Gini coefficient
• Hoover index
• Income inequality metrics
• Suits index
• Wealth condensation
• Diversity index

References

1. Introduction to the Theil index from the University of Texas
2. http://geodacenter.asu.edu/node/236
3. http://www.poorcity.richcity.org (Redundancy, Entropy and Inequality Measures)
4. Novotny, J. (2007). "On the measurement of regional inequality: Does spatial dimension of income inequality matter?" (PDF). Annals of Regional Science. 41 (3): 563–580.
5. Rajan K. Sampath. Equity Measures for Irrigation Performance Evaluation. Water International, 13(1), 1988.
6. A. Serebrenik, M. van den Brand. Theil index for aggregation of software metrics values. 26th IEEE International Conference on Software Maintenance. IEEE Computer Society.

External links

• Software:
  • Free Online Calculator computes the Gini Coefficient, plots the Lorenz curve, and computes many other measures of concentration for any dataset
  • Free Calculator: Online and downloadable scripts (Python and Lua) for Atkinson, Gini, and Hoover inequalities
  • Users of the R data analysis software can install the "ineq" package which allows for computation of a variety of inequality indices including Gini, Atkinson, Theil.
  • A MATLAB Inequality Package, including code for computing Gini, Atkinson, Theil indexes and for plotting the Lorenz Curve. Many examples are available.