A New Deep Neural Network Based on Multi-Layer Echo State Network - Liu, Ronghui; Zhao, Junmin (Recent Patents on Computer Science, 2018)
Background: Deep Neural Network (DNN) has attracted great attention in regression and classification problems. However, the traditional DNN fails to provide favorable prediction performance owing to their inherent shortcomings of the Back Propagation (BP) algorithm, such as slow convergence and local optimum. To solve this problem, a novel DNN algorithm called Multi-Layer Echo State Network (ML-ESN) is proposed. Method: This method utilizes Echo State Network (ESN) based unsupervised and supervised learning process. After discussing some related patents and methods, the ML-ESN structure is given. In the ML-ESN with M+1 hidden layers, ESN based Auto-Encoder (ESN-AE) is employed by the front M hidden layers for feature extraction in the unsupervised learning process, which can effectively extract abstract features from the original data. To overcome the under-fitting and over-fitting problem of ESN based classifier, the Penalty Factor ESN (PF-ESN) is presented to act as classifier by the M+1th hidden layer in the supervised learning process. Results: The experiments demonstrate that ML-ESN outperforms ESN, Tikhonov regularization ESN (TR-ESN) and some DNN models such as Deep Belief Network (DBN) and stacked auto encoder (SAE). Owing to the fast learning speed of ESN, the training time consumtion of ML-ESN is also shorter than DBN and SAE. Conclusion: The method provides references for fast and high accuracy classification and feature extraction.
Keywords: Echo state network