'why LSTM model output same hidden state at last layer

In my simple LSTM model, the last hidden state are almost same cross all batch data.

the model

import torch.nn as nn
import torch

lstm = nn.LSTM(
    input_size=5,
    hidden_size=6,
    num_layers=5,
    batch_first=True
)

i = torch.rand(9,8,5)

out1, (h_n, h_c) = lstm(i)

print('out.size()', out1.size())
print('h_n.size()', h_n.size())
print('h_c.size()', h_c.size())

print(h_n[-1])

the out put

out.size() torch.Size([9, 8, 6])
h_n.size() torch.Size([5, 9, 6])
h_c.size() torch.Size([5, 9, 6])
tensor([[ 0.1515,  0.2575,  0.0636, -0.1907, -0.0785,  0.0751],
        [ 0.1515,  0.2575,  0.0636, -0.1906, -0.0785,  0.0750],
        [ 0.1515,  0.2575,  0.0636, -0.1907, -0.0785,  0.0751],
        [ 0.1515,  0.2575,  0.0636, -0.1906, -0.0785,  0.0750],
        [ 0.1515,  0.2575,  0.0636, -0.1907, -0.0785,  0.0751],
        [ 0.1515,  0.2575,  0.0636, -0.1906, -0.0785,  0.0751],
        [ 0.1515,  0.2575,  0.0636, -0.1906, -0.0786,  0.0750],
        [ 0.1515,  0.2575,  0.0636, -0.1906, -0.0785,  0.0750],
        [ 0.1515,  0.2575,  0.0636, -0.1907, -0.0786,  0.0751]],

The out put are almost same. If decrease the layer number, it will be better.

pytorch lstm

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution	Source

'why LSTM model output same hidden state at last layer

Sources

Related Questions