Journal's Facebook pageOn the effectiveness of Gated Echo State Networks for data exhibiting long-term dependencies
- Department of Computer Science, University of Pisa
Pisa, Italy
daniele.disarli@phd.unipi.it gallicch@di.unipi.it micheli@di.unipi.it
Abstract
In the context of recurrent neural networks, gated architectures such as the GRU have contributed to the development of highly accurate machine learning models that can tackle long-term dependencies in the data. However, the training of such networks is performed by the expensive algorithm of gradient descent with backpropagation through time. On the other hand, reservoir computing approaches such as Echo State Networks (ESNs) can produce models that can be trained efficiently thanks to the use of fixed random parameters, but are not ideal for dealing with data presenting long-term dependencies. We explore the problem of employing gated architectures in ESNs from both theoretical and empirical perspectives.We do so by deriving and evaluating a necessary condition for the non-contractivity of the state transition function, which is important to overcome the fading-memorycharacterization of conventional ESNs. We find that using pure reservoir computing methodologies is not sufficient for effective gating mechanisms, while insteadtraining even only the gates is highly effective in terms of predictive accuracy.
Key words
echo state networks, gated recurrent neural networks, reservoir computing
Digital Object Identifier (DOI)
https://doi.org/10.2298/CSIS210218063D
Publication information
Volume 19, Issue 1 (January 2022)
Year of Publication: 2022
ISSN: 2406-1018 (Online)
Publisher: ComSIS Consortium
Full text
Available in PDF
Portable Document Format
How to cite
Sarli, D. D., Gallicchio, C., Micheli, A.: On the effectiveness of Gated Echo State Networks for data exhibiting long-term dependencies. Computer Science and Information Systems, Vol. 19, No. 1, 379-396. (2022), https://doi.org/10.2298/CSIS210218063D
