Mutual Information, Fisher Information, and Efficient Coding

2016 ◽  
Vol 28 (2) ◽  
pp. 305-326 ◽  
Author(s):  
Xue-Xin Wei ◽  
Alan A. Stocker
Keyword(s):  
Mutual Information ◽  
Fisher Information ◽  
Neural Coding ◽  
Population Coding ◽  
Neural Population ◽  
Small Noise ◽  
Efficient Coding ◽  
Coding Efficiency ◽  
Stimulus Variable ◽  
Neural Populations

Fisher information is generally believed to represent a lower bound on mutual information (Brunel & Nadal, 1998 ), a result that is frequently used in the assessment of neural coding efficiency. However, we demonstrate that the relation between these two quantities is more nuanced than previously thought. For example, we find that in the small noise regime, Fisher information actually provides an upper bound on mutual information. Generally our results show that it is more appropriate to consider Fisher information as an approximation rather than a bound on mutual information. We analytically derive the correspondence between the two quantities and the conditions under which the approximation is good. Our results have implications for neural coding theories and the link between neural population coding and psychophysically measurable behavior. Specifically, they allow us to formulate the efficient coding problem of maximizing mutual information between a stimulus variable and the response of a neural population in terms of Fisher information. We derive a signature of efficient coding expressed as the correspondence between the population Fisher information and the distribution of the stimulus variable. The signature is more general than previously proposed solutions that rely on specific assumptions about the neural tuning characteristics. We demonstrate that it can explain measured tuning characteristics of cortical neural populations that do not agree with previous models of efficient coding.

scholarly journals Information-Theoretic Bounds and Approximations in Neural Population Coding

2018 ◽  
Vol 30 (4) ◽  
pp. 885-944 ◽  
Author(s):  
Wentao Huang ◽  
Kechen Zhang
Keyword(s):  
Mutual Information ◽  
Numerical Solutions ◽  
Population Coding ◽  
Neural Population ◽  
Asymptotic Formulas ◽  
High Dimensional ◽  
Information Theoretic ◽  
Special Cases ◽  
Neural Populations ◽  
Approximation Formulas

While Shannon's mutual information has widespread applications in many disciplines, for practical applications it is often difficult to calculate its value accurately for high-dimensional variables because of the curse of dimensionality. This article focuses on effective approximation methods for evaluating mutual information in the context of neural population coding. For large but finite neural populations, we derive several information-theoretic asymptotic bounds and approximation formulas that remain valid in high-dimensional spaces. We prove that optimizing the population density distribution based on these approximation formulas is a convex optimization problem that allows efficient numerical solutions. Numerical simulation results confirmed that our asymptotic formulas were highly accurate for approximating mutual information for large neural populations. In special cases, the approximation formulas are exactly equal to the true mutual information. We also discuss techniques of variable transformation and dimensionality reduction to facilitate computation of the approximations.

scholarly journals Approximations of Shannon Mutual Information for Discrete Variables with Applications to Neural Population Coding

Entropy ◽  
2019 ◽  
Vol 21 (3) ◽  
pp. 243
Author(s):  
Wentao Huang ◽  
Kechen Zhang
Keyword(s):  
Mutual Information ◽  
Fisher Information ◽  
Population Coding ◽  
Neural Population ◽  
Asymptotic Formulas ◽  
Continuous Variables ◽  
Discrete Variables ◽  
Leibler Divergence ◽  
Approximation Formulas ◽  
Consistent Performance

Although Shannon mutual information has been widely used, its effective calculation is often difficult for many practical problems, including those in neural population coding. Asymptotic formulas based on Fisher information sometimes provide accurate approximations to the mutual information but this approach is restricted to continuous variables because the calculation of Fisher information requires derivatives with respect to the encoded variables. In this paper, we consider information-theoretic bounds and approximations of the mutual information based on Kullback-Leibler divergence and Rényi divergence. We propose several information metrics to approximate Shannon mutual information in the context of neural population coding. While our asymptotic formulas all work for discrete variables, one of them has consistent performance and high accuracy regardless of whether the encoded variables are discrete or continuous. We performed numerical simulations and confirmed that our approximation formulas were highly accurate for approximating the mutual information between the stimuli and the responses of a large neural population. These approximation formulas may potentially bring convenience to the applications of information theory to many practical and theoretical problems.

scholarly journals Approximations of Shannon Mutual Information for Discrete Variables with Applications to Neural Population Coding

2019 ◽  
Author(s):  
Wentao Huang ◽  
Kechen Zhang
Keyword(s):  
Mutual Information ◽  
Fisher Information ◽  
Population Coding ◽  
Neural Population ◽  
Asymptotic Formulas ◽  
Continuous Variables ◽  
Discrete Variables ◽  
Leibler Divergence ◽  
Approximation Formulas ◽  
Consistent Performance

Although Shannon mutual information has been widely used, its effective calculation is often difficult for many practical problems, including those in neural population coding. Asymptotic formulas based on Fisher information sometimes provide accurate approximations to the mutual information but this approach is restricted to continuous variables because the calculation of Fisher information requires derivatives with respect to the encoded variables. In this paper, we consider information-theoretic bounds and approximations of the mutual information based on Kullback--Leibler divergence and Rényi divergence. We propose several information metrics to approximate Shannon mutual information in the context of neural population coding. While our asymptotic formulas all work for discrete variables, one of them has consistent performance and high accuracy regardless of whether the encoded variables are discrete or continuous. We performed numerical simulations and confirmed that our approximation formulas were highly accurate for approximating the mutual information between the stimuli and the responses of a large neural population. These approximation formulas may potentially bring convenience to the applications of information theory to many practical and theoretical problems.

scholarly journals Pseudosparse neural coding in the visual system of primates

2021 ◽  
Vol 4 (1) ◽  
Author(s):  
Sidney R. Lehky ◽  
Keiji Tanaka ◽  
Anne B. Sereno
Keyword(s):  
Neural Coding ◽  
Macaque Monkey ◽  
Implicit Assumption ◽  
Data Set ◽  
Efficient Coding ◽  
Population Responses ◽  
Lateral Intraparietal Cortex ◽  
Neural Populations ◽  
Neurophysiological Data ◽  
The Mean

AbstractWhen measuring sparseness in neural populations as an indicator of efficient coding, an implicit assumption is that each stimulus activates a different random set of neurons. In other words, population responses to different stimuli are, on average, uncorrelated. Here we examine neurophysiological data from four lobes of macaque monkey cortex, including V1, V2, MT, anterior inferotemporal cortex, lateral intraparietal cortex, the frontal eye fields, and perirhinal cortex, to determine how correlated population responses are. We call the mean correlation the pseudosparseness index, because high pseudosparseness can mimic statistical properties of sparseness without being authentically sparse. In every data set we find high levels of pseudosparseness ranging from 0.59–0.98, substantially greater than the value of 0.00 for authentic sparseness. This was true for synthetic and natural stimuli, as well as for single-electrode and multielectrode data. A model indicates that a key variable producing high pseudosparseness is the standard deviation of spontaneous activity across the population. Consistently high values of pseudosparseness in the data demand reconsideration of the sparse coding literature as well as consideration of the degree to which authentic sparseness provides a useful framework for understanding neural coding in the cortex.

scholarly journals The Geometry of Information Coding in Correlated Neural Populations

2021 ◽  
Vol 44 (1) ◽  
Author(s):  
Rava Azeredo da Silveira ◽  
Fred Rieke
Keyword(s):  
Neural Coding ◽  
Neural Population ◽  
Annual Review ◽  
Publication Date ◽  
Information Coding ◽  
Population Code ◽  
New Approach ◽  
Neural Populations ◽  
The Brain ◽  
Theoretical Developments

Neurons in the brain represent information in their collective activity. The fidelity of this neural population code depends on whether and how variability in the response of one neuron is shared with other neurons. Two decades of studies have investigated the influence of these noise correlations on the properties of neural coding. We provide an overview of the theoretical developments on the topic. Using simple, qualitative, and general arguments, we discuss, categorize, and relate the various published results. We emphasize the relevance of the fine structure of noise correlation, and we present a new approach to the issue. Throughout this review, we emphasize a geometrical picture of how noise correlations impact the neural code. Expected final online publication date for the Annual Review of Neuroscience, Volume 44 is July 2021. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.

scholarly journals Optimal Short-Term Population Coding: When Fisher Information Fails

2002 ◽  
Vol 14 (10) ◽  
pp. 2317-2351 ◽  
Author(s):  
M. Bethge ◽  
D. Rotermund ◽  
K. Pawelzik
Keyword(s):  
Fisher Information ◽  
Dynamic Range ◽  
Mean Squared Error ◽  
Average Rate ◽  
Neuronal Response ◽  
Minimum Mean Square Error ◽  
Population Coding ◽  
Neuronal Firing ◽  
Tuning Curves ◽  
Efficient Coding

Efficient coding has been proposed as a first principle explaining neuronal response properties in the central nervous system. The shape of optimal codes, however, strongly depends on the natural limitations of the particular physical system. Here we investigate how optimal neuronal encoding strategies are influenced by the finite number of neurons N (place constraint), the limited decoding time window length T (time constraint), the maximum neuronal firing rate fmax (power constraint), and the maximal average rate fmax (energy constraint). While Fisher information provides a general lower bound for the mean squared error of unbiased signal reconstruction, its use to characterize the coding precision is limited. Analyzing simple examples, we illustrate some typical pitfalls and thereby show that Fisher information provides a valid measure for the precision of a code only if the dynamic range (fmin T, fmax T) is sufficiently large. In particular, we demonstrate that the optimal width of gaussian tuning curves depends on the available decoding time T. Within the broader class of unimodal tuning functions, it turns out that the shape of a Fisher-optimal coding scheme is not unique. We solve this ambiguity by taking the minimum mean square error into account, which leads to flat tuning curves. The tuning width, however, remains to be determined by energy constraints rather than by the principle of efficient coding.

Difficulty of Singularity in Population Coding

2005 ◽  
Vol 17 (4) ◽  
pp. 839-858 ◽  
Author(s):  
Shun-ichi Amari ◽  
Hiroyuki Nakahara
Keyword(s):  
Fisher Information ◽  
Regularity Condition ◽  
Information Matrix ◽  
Population Coding ◽  
Neural Population ◽  
Binding Problem ◽  
Synchronous Firing ◽  
Novel Method ◽  
Pathological Behavior ◽  
Pathological Case

Fisher information has been used to analyze the accuracy of neural population coding. This works well when the Fisher information does not degenerate, but when two stimuli are presented to a population of neurons, a singular structure emerges by their mutual interactions. In this case, the Fisher information matrix degenerates, and the regularity condition ensuring the Cramér-Rao paradigm of statistics is violated. An animal shows pathological behavior in such a situation. We present a novel method of statistical analysis to understand information in population coding in which algebraic singularity plays a major role. The method elucidates the nature of the pathological case by calculating the Fisher information. We then suggest that synchronous firing can resolve singularity and show a method of analyzing the binding problem in terms of the Fisher information. Our method integrates a variety of disciplines in population coding, such as nonregular statistics, Bayesian statistics, singularity in algebraic geometry, and synchronous firing, under the theme of Fisher information.

scholarly journals A signature of neural coding at human perceptual limits

2016 ◽  
Author(s):  
Paul M Bays
Keyword(s):  
Neural Coding ◽  
Empirical Relationship ◽  
Human Perception ◽  
Population Coding ◽  
Neural Population ◽  
Perceptual Awareness ◽  
Neural Basis ◽  
Visual Neurons ◽  
Error Magnitude ◽  
Subjective Confidence

AbstractSimple visual features, such as orientation, are thought to be represented in the spiking of visual neurons using population codes. I show that optimal decoding of such activity predicts characteristic deviations from the normal distribution of errors at low gains. Examining human perception of orientation stimuli, I show that these predicted deviations are present at near-threshold levels of contrast. The findings may provide a neural-level explanation for the appearance of a threshold in perceptual awareness, whereby stimuli are categorized as seen or unseen. As well as varying in error magnitude, perceptual judgments differ in certainty about what was observed. I demonstrate that variations in the total spiking activity of a neural population can account for the empirical relationship between subjective confidence and precision. These results establish population coding and decoding as the neural basis of perception and perceptual confidence.

scholarly journals Heterogeneous Synaptic Weighting Improves Neural Coding in the Presence of Common Noise

2020 ◽  
Vol 32 (7) ◽  
pp. 1239-1276
Author(s):  
Pratik S. Sachdeva ◽  
Jesse A. Livezey ◽  
Michael R. DeWeese
Keyword(s):  
Asymptotic Behavior ◽  
Mutual Information ◽  
Population Size ◽  
Fisher Information ◽  
Neural Coding ◽  
Relative Strength ◽  
Heterogeneous Distribution ◽  
Common Noise ◽  
The Common ◽  
The Impact

Simultaneous recordings from the cortex have revealed that neural activity is highly variable and that some variability is shared across neurons in a population. Further experimental work has demonstrated that the shared component of a neuronal population's variability is typically comparable to or larger than its private component. Meanwhile, an abundance of theoretical work has assessed the impact that shared variability has on a population code. For example, shared input noise is understood to have a detrimental impact on a neural population's coding fidelity. However, other contributions to variability, such as common noise, can also play a role in shaping correlated variability. We present a network of linear-nonlinear neurons in which we introduce a common noise input to model—for instance, variability resulting from upstream action potentials that are irrelevant to the task at hand. We show that by applying a heterogeneous set of synaptic weights to the neural inputs carrying the common noise, the network can improve its coding ability as measured by both Fisher information and Shannon mutual information, even in cases where this results in amplification of the common noise. With a broad and heterogeneous distribution of synaptic weights, a population of neurons can remove the harmful effects imposed by afferents that are uninformative about a stimulus. We demonstrate that some nonlinear networks benefit from weight diversification up to a certain population size, above which the drawbacks from amplified noise dominate over the benefits of diversification. We further characterize these benefits in terms of the relative strength of shared and private variability sources. Finally, we studied the asymptotic behavior of the mutual information and Fisher information analytically in our various networks as a function of population size. We find some surprising qualitative changes in the asymptotic behavior as we make seemingly minor changes in the synaptic weight distributions.

scholarly journals Quantifying Information Conveyed by Large Neuronal Populations

2019 ◽  
Vol 31 (6) ◽  
pp. 1015-1047 ◽  
Author(s):  
John A. Berkowitz ◽  
Tatyana O. Sharpee
Keyword(s):  
Mutual Information ◽  
Neural Circuit ◽  
Receptive Fields ◽  
Nonlinear Function ◽  
Logistic Function ◽  
Neural Population ◽  
Neuronal Populations ◽  
Multiple Stimulus ◽  
Neural Populations ◽  
Important Open Problem

Quantifying mutual information between inputs and outputs of a large neural circuit is an important open problem in both machine learning and neuroscience. However, evaluation of the mutual information is known to be generally intractable for large systems due to the exponential growth in the number of terms that need to be evaluated. Here we show how information contained in the responses of large neural populations can be effectively computed provided the input-output functions of individual neurons can be measured and approximated by a logistic function applied to a potentially nonlinear function of the stimulus. Neural responses in this model can remain sensitive to multiple stimulus components. We show that the mutual information in this model can be effectively approximated as a sum of lower-dimensional conditional mutual information terms. The approximations become exact in the limit of large neural populations and for certain conditions on the distribution of receptive fields across the neural population. We empirically find that these approximations continue to work well even when the conditions on the receptive field distributions are not fulfilled. The computing cost for the proposed methods grows linearly in the dimension of the input and compares favorably with other approximations.

Sign in / Sign up

Export Citation Format

Share Document