In earlier work, we meticulously documented the distribution of synaptic weights and the gain (or activation function) in many different brain areas. We found a remarkable consistency of heavy-tailed, specifically lognormal, distributions for firing rates, synaptic weights and gains (Scheler2017).
Why are biological neural networks heavy-tailed (lognormal)?
Cell assemblies: Lognormal networks support models of a hierarchically organized cell assembly (ensembles). Individual neurons can activate or suppress a whole cell assembly if they are the strongest neuron or directly connect to the strongest neurons (TeramaeJetal2012).
Storage: Sparse strong synapses store stable information and provide a backbone of information processing. More frequent weak synapses are more flexible and add changeable detail to the backbone. Heavy-tailed distributions allow a hierarchy of stability and importance.
Time delay of activation is reduced because strong synapses activate quickly a whole assembly (IyerRetal2013). This reduces the initial response time, which is dependent on the synaptic and intrinsic distribution. Heavy-tailed distributions activate fastest.
Noise response: Under additional input, noise or patterned, the pattern stability of the existing ensemble is higher (IyerRetal2013, see also KirstCetal2016). This is a side effect of integration of all computations within a cell assembly.
Why hierarchical computations in a neural network?
Calculations which depend on interactions between many discrete points (N-body problems, Barnes and Hut 1986), such as particle-particle methods, where every point depends on all others, lead to an O(N^2) calculation. If we supplant this by hierarchical methods, and combine information from multiple points, we can reduce the computational complexity to O(N log N) or O(N).
Since biological neural networks are not feedforward but connect in both forward and backward directions, they have a different structure from ANNs (artificial neural networks) – they consist of hierarchically organised ensembles with few wide-range excitability ‘hub’ neurons and many ‘leaf’ neurons with low connectivity and small-range excitability. Patterns are stored in these ensembles, and get accessed by a fit to an incoming pattern that could be expressed by low mutual information as a measure of similarity. Patterns are modified by similar access patterns, but typically only in their weak connections (else the accessing pattern would not fit).
Epigenetic modification is a powerful mechanism for the induction, the expression and persistence of long-term memory.
For long-term memory, we need to consider diverse cellular processes. These occur in neurons from different brain regions (in particular hippocampus, cortex, amygdala) during memory consolidation and recall. For instance, long-term changes in kinase expression in the proteome, changes in receptor subunit composition and localization at synaptic/dendritic membranes, epigenetic modifications of chromatin such as DNA methylation and histone methylation in the nucleus, and the posttranslational modifications of histones, including phosphorylation and acetylation, all these play a role. Histone acetylation is of particular interest because a number of known medications exist, which function as histone deacetylase inhibitors (HDACs), i.e. have a potential to increase DNA transcription and memory (more on this in a later post).
Epigenetic changes are important because they define the internal conditions for plasticity for the individual neuron. They underlie for instance, kinase or phosphatase-mediated (de)activations of enzymatic proteins and therefore influence the probability of membrane proteins to become altered by synaptic activation.
Among epigenetic changes, DNA methylation typically acts to alter, often to repress, DNA transcription at cytosine, or CpG islands in vertebrates. DNA methylation is mediated by enzymes such as Tet3, which catalyses an important step in the demethylation of DNA. In dentate gyrus of live rats, it was shown that the expression of Tet3 is greatly increased by LTP – synaptically induced memorization – , suggesting that certain DNA stretches were demethylated , and presumably activated. During induction of LTP by high frequency electrical stimulation, DNA methylation is changed specifically for certain genes known for their role in neural plasticity . The expression of neural plasticity genes is widely correlated with the methylation status of the corresponding DNA .
So there is interesting evidence for filtering the induction of plasticity via the epigenetic landscape and modifiers of gene expression, such as HDACs. Substances which act as histone deacetylase inhibitors (HDACs) increase histone acetylation. An interesting result from research on fear suggests that HDACs increase some DNA transcription, and enhance specifically fear extinction memories , ,.
The individual neuron’s state need not be determined only by the inputs received.
(a). It may additionally be seeded with a probability for adaptation that is distributed wrt the graph properties of the neuron (like betweenness centrality, choke points etc.), as well as the neuron’s current intrinsic excitability (IE) (which are related). This seeded probability would correspond to a sensitivity of the neuron to the representation that is produced by the subnetwork. The input representation is transformed by the properties of the subnetwork.
(b). Another way to influence neurons independent of their input is to link them together. This can be done by simulating of neuromodulators (NMs) which influence adaptivity for a subset of neurons within the network. There are then neurons which are linked together and increase or turn on their adaptivity because they share the same NM receptors. Different sets of neurons can become activated and increase their adaptivity, whenever a sufficient level of a NM is reached. An additional learning task is then to identify suitable sets of neurons. For instance, neurons may encode aspects of the input representation that result from additional, i.e. attentional, signals co-occuring with the input.
(c). Finally, both E and I neurons are known to consist of morphologically and genetically distinct types. This opens up additional ways of creating heterogeneous networks from these neuron types and have distinct adaptation rules for them. Some of the neurons may not even be adaptive, or barely adaptive, while others may be adaptive only once, (write once, read-only), or be capable only of upregulation, until they have received their limit. (This applies to synaptic and intrinsic adaptation). Certain neurons may have to follow the idea of unlimited adaptation in both directions in order to make such models viable.
Similar variants in neuron behavior are known from technical applications of ANNs: hyperparameters that link individual parameters into groups (‘weight sharing’) have been used, terms like ‘bypassing’ mean that some neurons do not adjust, only transmit, and ‘gating’ means that neurons may regulate the extent of transmission of a signal (cf. LSTM, ScardapaneSetal2018). Separately, the model ADAM (or ADAMW) has been proposed which computes adaptive learning rates for each neuron and achieves fast convergence.
A neuron-centric biological network model (‘neuronal automaton’) offers a systematic approach to such differences in adaptation. As suggested, biological neurons have different capacities for adaptation and this may extend to their synaptic connections as well. The model would allow to learn different activation functions and different adaptivity for each neuron, helped by linking neurons into groups and using fixed genetic types in the setup of the network. In each specific case the input is represented by the structural and functional constraints of the network and therefore transformed into an internal, egocentric representation.