How much information is contained in a neuron?

In response to a question on the JBC group about how many bits of information are contained in a neuron:

I think there are four questions, broadly speaking, that we can talk about when we talk about the information capacity of a neuron.

1. The number of states/entropy in the output
2. The number of states/entropy in the input
3. The mutual information between the input and the output
4. The number of possible configurations that the neuron can be in with respect to its presynaptic neurons, which is related to the functions that the neuron can implement. 

The answers to all of these questions will depend very heavily on the particular kind of neuron you're talking about, so my comments will be general.

[Note: "entropy" is distinct from the number of possible states because entropy is a probability-weighted sum of each of the possible states. I will focus on the number of states as opposed to entropy, except in item 3 below. It's also important to note that - in theory - any truly continuous, analog signal contains an infinite number of bits of information, the question of "how many bits are contained in signal X" is only meaningful when X is discretized.]

I'll go in order.

1. The number of states/entropy in the output

This is probably the easiest to quantify, especially in sensory systems. Assuming you know the full range of stimuli to which the neuron responds, you can probe the neuron by showing it the full range of possible stimuli (for example lines of different orientations for neurons in the visual cortex) and record the number of outputs you see. In many cases, the range will be more or less continuous -- it will have a spectrum of stimuli it prefers and doesn't prefer, and will fire more or less according to those preferences -- but the firing rate (and thus the amount of information in the output within a given time window) will be upper-bounded by the neuron's absolute refractory period, which determines its maximum firing rate. So if a neuron's maximum firing rate is 200 Hz, the maximum information that can be contained within a time window of one second is log(200), or around 7.6 bits. This analysis assumes that only the firing rate within a given time window matters (rate coding). If the specific timing of each spike matters as well, this number becomes much larger - if you divide the window into small bins - say 200 of them - and ask whether a spike will or won't occur in each one of them, you have 200 bits of information within a single second.

2. The number of states/entropy in the input

Most neurons in the brain get a huge amount of input (on the order of thousands of synapses). So the number of possible states for the input would be the sum of the number of bits in the output of the presynaptic neurons.

3. The mutual information between the input and the output

Just because a neuron gives you a certain range of output doesn't necessarily mean that the output is informative about the input. In particular, in addition to computing a deterministic function of its input, a neuron might add "noise" to its deterministic computation which might not tell you anything meaningful. Some neurons in sensory systems occasionally fire even in the absence of any stimulus, and you'll sometimes get a range of different responses for the same stimuli under the same conditions. There are mathy ways to quantify the mutual information content, but this is just to give you an idea.

4. The number of possible configurations that the neuron can be in with respect to its presynaptic neurons

I think that the question of "how much information is contained within a neuron" is most directly related to this. When we talk about learning and computation in neurons, the particular computation performed by a neuron depends on 1) intrinsic biophysical properties of the neuron 2) the neuron's presynaptic inputs and 3) how those inputs are weighted and interact when they reach the neuron.

If you assume that the neuron's intrinsic biophysical properties and presynaptic neurons are fixed and only the weights are mutable, the number of possible states is the number of synapses multiplied by the number of possible weights that each synapse can take. From a biological standpoint, synaptic weights can perhaps take on values from a continuous range (which could in theory give you an infinite amount of information), but it might make sense to say that synaptic weights can should be discretized, and a recent study claimed that each synapse should be assumed to contain about 4.7 bits. Multiply that by the number of synapses per neuron, and you get tens of thousands of bits, if not more. [And this still ignores things like synapse location, which is probably hugely important for computation.]

If you think about this from the standpoint of something like a perceptron, this isn't so unreasonable. If you have a perceptron with thousands of inputs you can train the perceptron to recognize a huge number of patterns (bounded by Cover's theorem, but the raw number of patterns is still pretty large). For example, depending on how you set the weights, a neuron might only respond to images of Jennifer Aniston or only to images of The Simpsons.