# What Is Partial Information Decomposition and How Features Interact

Partial Information Decomposition (PID) is a powerful concept within information theory that dissects and quantifies the individual and collective contributions of features within a system. It unravels how these features interact, shedding light on their combined impact on the system’s overall information content.

## Introduction to Surprise and Entropy

Surprise and entropy form the bedrock of information theory. Surprise signifies the unexpectedness of an outcome within a probability distribution, while entropy serves as a metric of average surprise. In essence, higher entropy denotes increased unpredictability, reflecting greater uncertainty in outcomes. Lower entropy, conversely, points to more predictable outcomes within the system.

## Entropy and (Mutual) Information

Entropy, a fundamental measure within information theory, quantifies uncertainty within a single variable. Mutual information, on the other hand, assesses the shared information between two variables. It elucidates how understanding one variable aids in reducing uncertainty about another, unveiling their interdependencies and the information they share.

## But What if we have multiple sources of information?

When navigating multiple sources of information, the complexity intensifies. Partial Information Decomposition (PID) becomes indispensable in such scenarios. It dissects the combined information into distinct components: unique, redundant, and synergistic. These components unravel the intricate interactions among the sources, providing a nuanced understanding of their collective impact on the system’s information content.

## Partial Information Decomposition Examples and the dit Library

The dit (discrete information theory) library in Python serves as a robust tool for implementing Partial Information Decomposition and analyzing information content within systems. An example using the library:

## But What if we have multiple sources of information?

Example: Implementing Partial Information Decomposition (PID)

## Unique Information, Redundant Information, Synergistic Information

• Unique Information: Quantifies information exclusive to a variable, revealing its distinct contributions.
• Redundant Information: Captures shared information among variables, highlighting overlaps.
• Synergistic Information: Reveals emergent information from interactions among variables, uncovering collective behaviors.

Example: Understanding Unique, Redundant, and Synergistic Information

The dit library provides functions to access specific PID components from the result: