🧩Information processing in complex systems

1. Introduction

In man-made computers, with their distinct central processing units, memory, and data buses, computation is transparent. But in the emergent, collective behaviors of an ant colony forging a trail, a school of fish evading a predator, or the intricate firing patterns within the human brain, the process is far more elusive. The primary theoretical tool from computer science, the Turing machine, is fundamentally ill-suited to these systems, creating a paradigm mismatch that obscures rather than illuminates their computational nature.

This chapter presents a philosophical and methodological shift away from this traditional computational paradigm. It proposes that to understand how nature "computes," we must adopt the language of dynamical systems to describe what can be termed "intrinsic information processing." This perspective views computation not as a discrete algorithm with a clear start and end, but as an inherent property of a system's evolution through time.

This document will proceed by first deconstructing the classical, Turing-based approach to demonstrate its structural inability to capture biological reality. It will then introduce the dynamical systems perspective as a more fitting language for describing natural processes. Finally, it will propose a quantitative framework for breaking down this intrinsic processing into three fundamental and measurable informational operations: the storage, transfer, and modification of information.

2. The Paradigm Mismatch: Turing Machines vs. Biological Computation

To justify a new framework, we must first deconstruct the dominant computer science paradigm, demonstrating its structural inability to capture the essence of natural computation. The theorist Melanie Mitchell provides a useful framework for comparing the two paradigms by asking a series of guiding questions. The stark contrast in the answers reveals a fundamental mismatch that necessitates a different approach for studying complex biological and natural systems.

Guiding Question	Computer Science (Turing Machine Paradigm)	Biological Computation
What plays the role of information in the system?	Information is encoded on a digital static tape.	Information is encoded in analog states and patterns distributed in space and time. Gathered via statistical sampling.
How is the information communicated and processed?	Processing is deterministic, serial, and error-free, governed by centralised rules.	Processing is decentralised, parallel, and local, involving fine-grained stochastic interactions. Randomness is utilized.
How does the information acquire function/purpose/meaning?	Meaning is provided by the human designer of the system.	Meaning emerges via natural selection.

Beyond these distinctions, the fundamental operational models diverge significantly. Biological computation is typically distributed, with processing occurring in parallel across countless interacting components. For instance, The brain processes information through the coordinated activity of multiple neurons. The flock collectively determines the direction of flight. Ants locate food by communicating with one another to find the optimal route. There is no central processing unit.

Furthermore, it is ongoing. Unlike a Turing machine, which is designed to execute an algorithm and then enter a defined "halt state," a biological system's information processing does not necessarily finish. The brain, an ant colony, or a flock of birds are in a constant state of dynamic evolution.

These profound differences demonstrate that the language of finite algorithms is fundamentally misaligned with biological reality, compelling a shift toward a new lexicon: the language of dynamical systems.

3. A New Perspective: Computation as Intrinsic Information Processing

The mismatch with the classical paradigm leads to a critical insight articulated by Melanie Mitchell: for complex systems, "the language of dynamical systems may be more useful than language of computation." This assertion marks a pivotal philosophical shift. Instead of forcing natural phenomena into the rigid structure of a Turing machine, we can view computation through the lens of dynamics, embracing the continuous and distributed nature of these systems.

This shift allows us to define the concept of Intrinsic Information Processing. Formally, this is the processing that occurs whenever a system undergoes a dynamical process that changes its initial state (+inputs) into a later state (+outputs). This framework adopts a deliberately inclusive definition of computation, reframing it not as a specialized task but as a fundamental property of dynamic evolution. It encompasses everything from the universe computing its own future to the specific behaviors of biological organisms.

The human brain serves as a powerful illustration. At any moment, the brain is in an initial state, where "that state is represented by the activity in each area in some spatio-temporal pattern." The inputs are sensory signals from the environment. Through a dynamical process—the intricate interaction of billions of neurons—the brain transforms this initial state into a later state, described by a new spatio-temporal firing pattern, plus motor responses as outputs. This continuous transformation is the information processing. This inclusive definition provides a conceptual foundation, but to be scientifically useful, we need a method to break this intrinsic processing down into measurable components.

4. The Proposed Framework: Deconstructing Dynamics into Informational Operations

To move beyond qualitative descriptions and build a rigorous, quantitative understanding of intrinsic information processing, we must identify the fundamental operations that constitute it. When describing complex systems, we often intuitively use the language of memory, signaling, and processing. This vernacular provides the key to deconstructing their dynamics.

We propose a framework that quantifies three core features of distributed computation:

1. Information Storage (Memory): How information from a system's past states is preserved in its present to influence its future.

2. Information Transfer (Signaling): How information is communicated between different components or regions of the system, reducing uncertainty about the target's state.

3. Information Modification (Processing): The synergistic combination of stored and transferred information to create novel information—that is, information about the next state that is not present in the individual source components.

While these operations map cleanly to a PC's physical components—RAM for storage, buses for transfer, and the CPU for modification—they are not so easily isolated in biological computation. Nonetheless, they form the basis of a powerful quantitative framework applicable to a vast array of complex systems, including:

• The time evolution of cellular automata, where patterns store and transmit information.

• Gene regulatory networks computing and storing cell behaviors.

• Flocks computing their collective heading, where waves of information are transferred to communicate threats.

• Ant colonies computing efficient routes to food.

The overarching goal is to quantify natural computation using the very concepts—memory, signaling, processing—that arise organically in its description. This bridges the gap between observation and measurement.

5. The Information Dynamics Methodology

The Information Dynamics framework is the formal methodology designed to quantify the operations of storage, transfer, and modification. It provides a set of practical, information-theoretic tools for moving from the philosophical concept of intrinsic processing to empirical measurement. The framework is built upon a foundational model of complex systems as multivariate time-series, allowing for a precise and quantitative deconstruction of how a system's next state is computed.

5.1 The Foundational Model: Systems as Multivariate Time-Series

The core analytical approach of Information Dynamics is to model a complex system as a collection of interacting time-series processes. In this model, each component of the system—be it a neuron, an ant, a stock, or a cell in a cellular automaton—is treated as a variable whose state evolves over time. This creates a multivariate time-series, a rich dataset that captures both the individual behavior of each component and the collective dynamics of the system as a whole.

This perspective is broadly applicable to the kinds of systems where intrinsic computation is observed:

• Cellular Automata: As seen in Elementary Cellular Automata like Rule 54, the system consists of a grid of cells (variables), each with a time-series of states ('on' or 'off'). The framework allows us to characterize how the state update of a single cell depends on its neighbors.

• Game Theory Models: In a game like Scissors-Paper-Rock, each player is a variable, and their sequence of moves forms a time-series. We can model how a player's next move is computed based on their own past moves and the moves of their opponent.

• Biological and Neurological Systems: The firing patterns of different neurons or the activity levels in different brain regions can be represented as a multivariate time-series, allowing us to study how information is processed across the brain.

To formalize this, we consider a collection of time-series processes ($X$, $Y$, $Z$, etc.), where each process consists of a sequence of random variables $\{..., X_{n-1}, X_n, X_{n+1}, ...\}$. The actual data we analyze are the process realisations $\{..., x_{n-1}, x_n, x_{n+1}, ...\}$. This formalization allows us to pose a precise and answerable question.

5.2 The Core Inquiry: Decomposing the Next State

The central question of the Information Dynamics framework is: "how is the next state of a variable in a complex system computed?" It posits that this next state, $X_{n+1}$, is the output of a local computation within the system. The framework's primary goal is to deconstruct this computation by answering a series of targeted questions:

• Where does the information in $X_{n+1}$ come from?

• Can we model the information processing in $X$ in terms of storage and transfer?

• Can we partition these informational contributions? Do they overlap? To answer these, the framework formally examines how the information in the random variable $X_{n+1}$ can be attributed to the past states of all variables in the system up to time $n$. This is achieved by partitioning the sources of information into the three fundamental operations:

1. Information Storage: Quantified by measuring the information that comes from the past of the target variable itself. This is the information contained in the block vector or state of $X$ leading up to the computation, denoted $X_n^{(k)} = \{X_{n-k+1}, ..., X_n\}$.

2. Information Transfer: Quantified by measuring the information that comes from the past states of other, neighboring source variables (e.g., $Y$ and $Z$).

3. Information Modification: Quantified as the synergistic or redundant interactions between the stored and transferred information.

This approach provides a local, time-resolved account of the system's dynamics, examining not just average relationships but the specific state updates of a target variable and the dynamics of the measures in space and time.

5.3 A First Measure: The Entropy Rate

Historically, the first consideration in this context is a classic measure from information theory: the Entropy Rate. Before we can determine where information comes from, the Entropy Rate helps us quantify how much new information or uncertainty there is to explain in the first place. It measures the residual unpredictability of a system's next state, even after accounting for complete knowledge of its history.

The Entropy Rate can be understood from two equivalent perspectives for stationary processes:

1. As a Limiting Rate: It measures the rate at which the entropy of a block of n consecutive variables grows with n. As seen in the provided diagram, the total block entropy $H(X_n^{(n)})$ increases with the block length $n$, but the amount of new information added by each subsequent variable typically decreases due to redundancies. The Entropy Rate is the limiting slope of this curve as n approaches infinity.

$$H'_{\mu X} = \lim_{n \to \infty} \frac{1}{n} H(X_1, X_2, \dots, X_n)$$

$$H'_{\mu X} = \lim_{n \to \infty} \frac{1}{n} H(\mathbf{X}_n^{(n)})$$

1. As Conditional Uncertainty: It measures the uncertainty of the next random variable $X_n$, given the entire semi-infinite past of the process. This is the amount of novelty or surprise introduced at each time step that cannot be predicted from history.

$$H_{\mu X} = \lim_{n \to \infty} H(X_n | X_1, X_2, \dots, X_{n-1})$$

$$H_{\mu X} = \lim_{n \to \infty} H(X_n | \mathbf{X}_{n-1}^{(n-1)})$$

In stationary process, $H'{\mu X} = H{\mu X}$

The coding interpretation makes this concept highly intuitive. Imagine encoding English text. The entropy rate is the amount of code needed for the next letter if the receiver already knows all preceding letters. After seeing a 'q', the uncertainty about the next letter is virtually zero, as it will almost certainly be a 'u'. The entropy rate captures this radical reduction in uncertainty based on past context. It serves as a fundamental baseline for analyzing the predictability and inherent randomness of a dynamic process before we proceed to attribute that predictability to specific sources like storage and transfer.

6. Conclusion: From Algorithms to Dynamics

This chapter has presented a fundamental shift in perspective, moving from the rigid, algorithmic framework of the Turing machine to the more fluid and appropriate language of dynamical systems. By defining computation as an intrinsic property of a system's evolution, we have established a new conceptual foundation for analyzing natural processes. This "intrinsic information processing" is then made analytically tractable through its decomposition into three quantifiable operations: information storage, transfer, and modification. This provides a powerful, quantitative methodology to move beyond metaphor, allowing us to dissect the intricate dance of information that governs the behavior of complex systems from the brain to entire ecosystems.