On the effectiveness of Gated Echo State Networks for data exhibiting long-term dependencies

Daniele Di Sarli1, Claudio Gallicchio1 and Alessio Micheli1

  1. Department of Computer Science, University of Pisa
    Pisa, Italy
    daniele.disarli@phd.unipi.it gallicch@di.unipi.it micheli@di.unipi.it

Abstract

In the context of recurrent neural networks, gated architectures such as the GRU have contributed to the development of highly accurate machine learning models that can tackle long-term dependencies in the data. However, the training of such networks is performed by the expensive algorithm of gradient descent with backpropagation through time. On the other hand, reservoir computing approaches such as Echo State Networks (ESNs) can produce models that can be trained efficiently thanks to the use of fixed random parameters, but are not ideal for dealing with data presenting long-term dependencies. We explore the problem of employing gated architectures in ESNs from both theoretical and empirical perspectives.We do so by deriving and evaluating a necessary condition for the non-contractivity of the state transition function, which is important to overcome the fading-memorycharacterization of conventional ESNs. We find that using pure reservoir computing methodologies is not sufficient for effective gating mechanisms, while insteadtraining even only the gates is highly effective in terms of predictive accuracy.

Key words

echo state networks, gated recurrent neural networks, reservoir computing

Digital Object Identifier (DOI)

https://doi.org/10.2298/CSIS210218063D

How to cite

Sarli, D. D., Gallicchio, C., Micheli, A.: On the effectiveness of Gated Echo State Networks for data exhibiting long-term dependencies. Computer Science and Information Systems, https://doi.org/10.2298/CSIS210218063D