Ternary Spike: Learning Ternary Spikes for Spiking Neural Networks
> The Spiking Neural Network (SNN), as one of the biologically
inspired neural network infrastructures, has drawn increas-
ing attention recently. It adopts binary spike activations to
transmit information, thus the multiplications of activations
and weights can be substituted by additions, which brings
high energy efficiency. However, in the paper, we theoret-
ically and experimentally prove that the binary spike acti-
vation map cannot carry enough information, thus causing
information loss and resulting in accuracy decreasing. To
handle the problem, we propose a ternary spike neuron to
transmit information. The ternary spike neuron can also enjoy
the event-driven and multiplication-free operation advantages of the binary spike neuron but will boost the information ca-
pacity. Furthermore, we also embed a trainable factor in the
ternary spike neuron to learn the suitable spike amplitude, thus
our SNN will adopt different spike amplitudes along layers,
which can better suit the phenomenon that the membrane po-
tential distributions are different along layers. To retain the
efficiency of the vanilla ternary spike, the trainable ternary
spike SNN will be converted to a standard one again via a re-
parameterization technique in the inference. Extensive experi-
ments with several popular network structures over static and
dynamic datasets show that the ternary spike can consistently
outperform state-of-the-art methods.
https://github.com/yfguo91/Ternary-Spike
Ternary Spike: Learning Ternary Spikes for Spiking Neural Networks
> The Spiking Neural Network (SNN), as one of the biologically inspired neural network infrastructures, has drawn increas- ing attention recently. It adopts binary spike activations to transmit information, thus the multiplications of activations and weights can be substituted by additions, which brings high energy efficiency. However, in the paper, we theoret- ically and experimentally prove that the binary spike acti- vation map cannot carry enough information, thus causing information loss and resulting in accuracy decreasing. To handle the problem, we propose a ternary spike neuron to transmit information. The ternary spike neuron can also enjoy the event-driven and multiplication-free operation advantages of the binary spike neuron but will boost the information ca- pacity. Furthermore, we also embed a trainable factor in the ternary spike neuron to learn the suitable spike amplitude, thus our SNN will adopt different spike amplitudes along layers, which can better suit the phenomenon that the membrane po- tential distributions are different along layers. To retain the efficiency of the vanilla ternary spike, the trainable ternary spike SNN will be converted to a standard one again via a re- parameterization technique in the inference. Extensive experi- ments with several popular network structures over static and dynamic datasets show that the ternary spike can consistently outperform state-of-the-art methods.