LeakyESN

class torch_rc.nn.LeakyESN(input_size: int, output_size: int, num_layers=1, bias=True, bidirectional=False, scale_rec=0.9, density=1.0, scale_in=1.0, density_in=1.0, scale_bias=1.0, leaking_rate=1.0, rescaling_method='specrad')[source]

A multi-layer Leaky Echo State Network.

Parameters

  • input_size – The number of expected features in the input x

  • output_size – The number of features in the hidden state h

  • num_layers – Number of recurrent layers. E.g., setting num_layers=2 would mean stacking two ESNs together to form a stacked ESN, with the second ESN taking in outputs of the first ESN and computing the final results. Default: 1

  • bias – If False, then the layer does not use bias weights. Default: True

  • bidirectional – If True, becomes a bidirectional ESN. Default: False

  • scale_rec – Scaling of the recurrent connection matrix. The actual rescaling value depends on the rescaling_method parameter.

  • density – Density of the recurrent connection matrix. Default: 1.0

  • scale_in – Scaling of the input connetions. Default: 1.0

  • density_in – Density of the input connetion matrix. Default: 1.0

  • scale_bias – Scaling of the bias values. Default: 1.0

  • leaking_rate – Leaking rate. Default: 1.0

  • rescaling_method – The method for rescaling the recurrent matrix. It can be either 'norm' or 'specrad'. If 'norm', then \(\left\|W\right\|_2 = \text{scale_rec}\). If 'specrad', then \(\rho(W) = \text{scale_rec}\). Default: 'specrad'

Shape

  • Input1 – \((L, N, H_{in})\) tensor containing input features where \(H_{in}=\text{input\_size}\) and L represents a sequence length.

  • Input2 – \((S, N, H_{out})\) tensor containing the initial hidden state for each element in the batch. \(H_{out}=\text{hidden\_size}\) Defaults to zero if not provided. where \(S=\text{num\_layers} * \text{num\_directions}\) If the ESN is bidirectional, num_directions should be 2, else it should be 1.

  • Output1 – \((L, N, H_{all})\) where \(H_{all}=\text{num\_directions} * \text{hidden\_size}\)

  • Output2 – \((S, N, H_{out})\) tensor containing the next hidden state for each element in the batch

forward(input, h_0: Optional[torch.Tensor] = None)

Forwards the given sequence through the network.

Parameters

  • input – tensor of shape (seq_len, batch, input_size): tensor containing the features of the input sequence.

  • h_0 – tensor of shape (num_layers * num_directions, batch, hidden_size): tensor containing the initial hidden state for each element in the batch. Defaults to zero if not provided. If the ESN is bidirectional, num_directions should be 2, else it should be 1.

Returns

output, h_n

  • output of shape (seq_len, batch, num_directions * hidden_size): tensor containing the output features (h_t) from the last layer of the RNN, for each t.

    The directions can be separated using output.view(seq_len, batch, num_directions, hidden_size), with forward and backward being direction 0 and 1 respectively.

  • h_n of shape (num_layers * num_directions, batch, hidden_size): tensor containing the hidden state for t = seq_len.

    Like output, the layers can be separated using h_n.view(num_layers, num_directions, batch, hidden_size).