InformationTheory

The information theory module includes various functions related to entropy.

We can first begin by creating a complex sequence:

#include <InformationTheory.hpp>
#include <Complex.hpp>
#include <iostream>
#include <vector>

std::vector

int main(){

    std::vector<Complex> P = {0.25 + 0.5_j, 0.25 + 0.5_j, 0.25, 0.25};
    return 0;
}

This complex sequence defines two discrete probability distributions for the complex and real components. All discrete entropy functions assume that these distributions are independent.

The entropy of this distribution can be calculated as follows:

#include <InformationTheory.hpp>
#include <Complex.hpp>
#include <iostream>
#include <vector>

int main(){
    std::vector<Complex> P = {0.25 + 0.5_j, 0.25 + 0.5_j, 0.25, 0.25};
    std::cout << entropy(P) << "\n";

   return 0;
}

Output:

2.07944

Also useful is calculating the Shannon information content of a single probability \(p\). It is defined as:

\[I(x) = -\log(p)\]

and can be implemented by doing the following:

#include <InformationTheory.hpp>
#include <Complex.hpp>
#include <iostream>
#include <vector>

int main(){
    Complex p = 0.5 + 0.5_j;
    std::cout << shannonInformation(p) << "\n";

    return 0;
}

Output:

1.38629

The KL divergence is defined as how different one probability distribution \(P\) is from another distribution \(Q\), and is implemented as such:

#include <InformationTheory.hpp>
#include <Complex.hpp>
#include <iostream>
#include <vector>

int main(){
    std::vector<Complex> P = {0.25 + 0.5_j, 0.25 + 0.5_j, 0.25, 0.25};
    std::vector<Complex> Q = {0.4 + 0.5_j, 0.2 + 0.5_j, 0.2, 0.2};
    std::cout << klDiv(P, Q) << "\n";

   return 0;
}

Output:

0.0498568

The information theory module also featuers continuous versions of these functions. The continuous versions assume you are using the joint PDF.

#include <InformationTheory.hpp>
#include <Constants.hpp> // for negative and positive INF.
#include <Complex.hpp>
#include <iostream>
#include <vector>

int main(){
    auto f = [](Complex t) { return exp(-t * t); }; // Gaussian function.
    std::cout << entropy(f, NINF.real(), INF.real()) << "\n";

   return 0;
}

Output:

-0.88551

A continuous version of the KL divergence can also be easily implemented:

#include <InformationTheory.hpp>
#include <Constants.hpp> // for negative and positive INF.
#include <Complex.hpp>
#include <iostream>
#include <vector>

int main(){
    auto fRe = [](Complex t) { return exp(-t * t); }; // Gaussian function.
    auto fIm = [](Complex t) { return exp(-t * t); }; // Gaussian function.
    auto gRe = [](Complex t) { return exp(-t * t); }; // Gaussian function.
    auto gIm = [](Complex t) { return exp(-t * t); }; // Gaussian function.
    std::cout << klDiv(fRe, fIm, gRe, gIm, NINF.real(), INF.real()) << "\n"; // Should be ~ 0 (epsilon value included in logs may influence precision).

   return 0;
}

Output:

-8.03496e-08