BinaryDuo: Reducing Gradient Mismatch in Binary Activation Network by Coupling Binary Activations
Binary Neural Networks (BNNs) have been garnering interest thanks to their compute cost reduction and memory savings. However, BNNs suffer from performance degradation mainly due to the gradient mismatch caused by binarizing activations. Previous works tried to address the gradient mismatch problem...
Saved in:
Main Authors | , , , |
---|---|
Format | Journal Article |
Language | English |
Published |
16.02.2020
|
Subjects | |
Online Access | Get full text |
Cover
Loading…
Summary: | Binary Neural Networks (BNNs) have been garnering interest thanks to their
compute cost reduction and memory savings. However, BNNs suffer from
performance degradation mainly due to the gradient mismatch caused by
binarizing activations. Previous works tried to address the gradient mismatch
problem by reducing the discrepancy between activation functions used at
forward pass and its differentiable approximation used at backward pass, which
is an indirect measure. In this work, we use the gradient of smoothed loss
function to better estimate the gradient mismatch in quantized neural network.
Analysis using the gradient mismatch estimator indicates that using higher
precision for activation is more effective than modifying the differentiable
approximation of activation function. Based on the observation, we propose a
new training scheme for binary activation networks called BinaryDuo in which
two binary activations are coupled into a ternary activation during training.
Experimental results show that BinaryDuo outperforms state-of-the-art BNNs on
various benchmarks with the same amount of parameters and computing cost. |
---|---|
DOI: | 10.48550/arxiv.2002.06517 |