# Subdominant Dense Clusters Allow for Simple Learning and High Computational Performance in Neural Networks with Discrete Synapses.

@article{Baldassi2015SubdominantDC, title={Subdominant Dense Clusters Allow for Simple Learning and High Computational Performance in Neural Networks with Discrete Synapses.}, author={Carlo Baldassi and Alessandro Ingrosso and Carlo Lucibello and Luca Saglietti and Riccardo Zecchina}, journal={Physical review letters}, year={2015}, volume={115 12}, pages={ 128101 } }

We show that discrete synaptic weights can be efficiently used for learning in large scale neural systems, and lead to unanticipated computational performance. We focus on the representative case of learning random patterns with binary synapses in single layer networks. The standard statistical analysis shows that this problem is exponentially dominated by isolated solutions that are extremely hard to find algorithmically. Here, we introduce a novel method that allows us to find analytical… Expand

#### 88 Citations

Learning may need only a few bits of synaptic precision.

- Physics, Biology
- Physical review. E
- 2016

Learning in neural networks poses peculiar challenges when using discretized rather then continuous synaptic states. The choice of discrete synapses is motivated by biological reasoning and… Expand

On the role of synaptic stochasticity in training low-precision neural networks

- Computer Science, Medicine
- Physical review letters
- 2018

It is shown that a neural network model with stochastic binary weights naturally gives prominence to exponentially rare dense regions of solutions with a number of desirable properties such as robustness and good generalization performance, while typical solutions are isolated and hard to find. Expand

Unreasonable effectiveness of learning neural networks: From accessible states and robust ensembles to basic algorithmic schemes

- Computer Science, Medicine
- Proceedings of the National Academy of Sciences
- 2016

It is shown that there are regions of the optimization landscape that are both robust and accessible and that their existence is crucial to achieve good performance on a class of particularly difficult learning problems, and an explanation of this good performance is proposed in terms of a nonequilibrium statistical physics framework. Expand

Learning through atypical "phase transitions" in overparameterized neural networks

- Computer Science, Physics
- ArXiv
- 2021

Methods from statistical physics of disordered systems are used to analytically study the computational fallout of overparameterization in nonconvex neural network models and find that there exist a gap between the SAT/UNSAT interpolation transition where solutions begin to exist and the point where algorithms start to find solutions, i.e. where accessible solutions appear. Expand

Mean-field inference methods for neural networks

- Physics, Computer Science
- ArXiv
- 2019

A selection of classical mean-field methods and recent progress relevant for inference in neural networks are reviewed, and the principles of derivations of high-temperature expansions, the replica method and message passing algorithms are reminded, highlighting their equivalences and complementarities. Expand

Binary perceptron: efficient algorithms can find solutions in a rare well-connected cluster

- Mathematics, Physics
- 2021

It was recently shown that almost all solutions in the symmetric binary perceptron are isolated, even at low constraint densities, suggesting that finding typical solutions is hard. In contrast, some… Expand

Understanding the computational difficulty of a binary-weight perceptron and the advantage of input sparseness

- Computer Science, Physics
- 2019

A perceptron model, which associates binary input patterns with outputs using binary (0 or 1) weights, modeling a single neuron receiving excitatory inputs is studied, highlighting the heterogeneity of learning dynamics of weights. Expand

Statistical physics of neural systems

- Computer Science
- 2018

This work represents learning as an optimization problem, actually implementing a local search, in the synaptic space, of specific configurations, known as solutions and making a neural network able to accomplish a series of different tasks. Expand

CORSO DI LAUREA MAGISTRALE IN FISICA OUT-OF-EQUILIBRIUM ANALYSIS OF SIMPLE NEURAL NETWORKS

- 2016

We consider a novel approach to learning in neural networks with discrete synapses [1, 2, 3] and discuss its possible extensions to simple continuous neural networks. The problem of learning is… Expand

Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data

- Computer Science, Mathematics
- UAI
- 2017

By optimizing the PAC-Bayes bound directly, Langford and Caruana (2001) are able to extend their approach and obtain nonvacuous generalization bounds for deep stochastic neural network classifiers with millions of parameters trained on only tens of thousands of examples. Expand

#### References

SHOWING 1-10 OF 30 REFERENCES

Efficient supervised learning in networks with binary synapses

- Computer Science, Biology
- Proceedings of the National Academy of Sciences
- 2007

A neurobiologically plausible on-line learning algorithm that derives from belief propagation algorithms that performs remarkably well in a model neuron with binary synapses, and a finite number of “hidden” states per synapse, that has to learn a random classification task. Expand

Origin of the computational hardness for learning with binary synapses

- Mathematics, Medicine
- Physical review. E, Statistical, nonlinear, and soft matter physics
- 2014

This work analytically derive the Franz-Parisi potential for the binary perceptron problem by starting from an equilibrium solution of weights and exploring the weight space structure around it, which reveals the geometrical organization of theWeight space is composed of isolated solutions, rather than clusters of exponentially many close-by solutions. Expand

Learning from examples in large neural networks.

- Computer Science, Medicine
- Physical review letters
- 1990

Numerical results on training in layered neural networks indicate that the generalization error improves gradually in some cases, and sharply in others, and statistical mechanics is used to study generalization curves in large layered networks. Expand

Generalization Learning in a Perceptron with Binary Synapses

- Mathematics, Physics
- 2009

AbstractWe consider the generalization problem for a perceptron with binary synapses, implementing the Stochastic Belief-Propagation-Inspired (SBPI) learning algorithm which we proposed earlier, and… Expand

A Max-Sum algorithm for training discrete neural networks

- Computer Science, Physics
- ArXiv
- 2015

The algorithm is a variant of the so-called Max-Sum algorithm that performs as well as BP on binary perceptron learning problems, and may be better suited to address the problem on fully-connected two-layer networks, since inherent symmetries in two layer networks are naturally broken using the MS approach. Expand

Entropy landscape of solutions in the binary perceptron problem

- Mathematics, Physics
- ArXiv
- 2013

The statistical picture of the solution space for a binary perceptron and the geometrical organization is elucidated by the entropy landscape from a reference configuration and of solution-pairs separated by a given Hamming distance in the solutions space. Expand

A Fast Learning Algorithm for Deep Belief Nets

- Mathematics, Computer Science
- Neural Computation
- 2006

A fast, greedy algorithm is derived that can learn deep, directed belief networks one layer at a time, provided the top two layers form an undirected associative memory. Expand

Dynamics of learning for the binary perceptron problem

- Mathematics
- 1992

A polynomial learning algorithm for a perceptron with binary bonds and random patterns is investigated within dynamic mean field theory. A discontinuous freezing transition is found at a temperature… Expand

Storage capacity of memory networks with binary couplings

- Physics
- 1989

We study the number p of unbiased random patterns which can be stored in a neural network of N neurons used as an associative memory, in the case where the synaptic efficacies are constrained to take… Expand

Hippocampal Spine Head Sizes Are Highly Precise

- Biology
- 2015

In an electron microscopic reconstruction of hippocampal neuropil, single axons making two or more synaptic contacts onto the same dendrites which would have shared histories of presynaptic and postsynaptic activity were found. Expand