Memristor-Based Multilayer Neural Networks With Online Gradient Descent Training

• Published in: IEEE transaction on neural networks and learning systems Vol. 26; no. 10; pp. 2408 - 2421
• Main Authors: Soudry, Daniel, Di Castro, Dotan, Gal, Asaf, Kolodny, Avinoam, Kvatinsky, Shahar
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.10.2015; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Algorithm design and analysis; Algorithms; Backpropagation; Circuits; CMOS; Computer Simulation; Descent; Hardware; Humans; Learning; Memory devices; memristive systems; memristor; Memristors; multilayer neural networks (MNNs); Multilayers; Neural networks; Neural Networks (Computer); Neurons - physiology; Online Systems - instrumentation; Signal Processing, Computer-Assisted - instrumentation; stochastic gradient descent; synapse; Synapses - physiology; Training; Transistors; Backpropagation; multilayer neural networks (MNNs); memristor; synapse; memristive systems; stochastic gradient descent; hardware
• ISSN: 2162-237X; 2162-2388; 2162-2388
• DOI: 10.1109/TNNLS.2014.2383395

Learning in multilayer neural networks (MNNs) relies on continuous updating of large matrices of synaptic weights by local rules. Such locality can be exploited for massive parallelism when implementing MNNs in hardware. However, these update rules require a multiply and accumulate operation for each synaptic weight, which is challenging to implement compactly using CMOS. In this paper, a method for performing these update operations simultaneously (incremental outer products) using memristor-based arrays is proposed. The method is based on the fact that, approximately, given a voltage pulse, the conductivity of a memristor will increment proportionally to the pulse duration multiplied by the pulse magnitude if the increment is sufficiently small. The proposed method uses a synaptic circuit composed of a small number of components per synapse: one memristor and two CMOS transistors. This circuit is expected to consume between 2% and 8% of the area and static power of previous CMOS-only hardware alternatives. Such a circuit can compactly implement hardware MNNs trainable by scalable algorithms based on online gradient descent (e.g., backpropagation). The utility and robustness of the proposed memristor-based circuit are demonstrated on standard supervised learning tasks.