2019-03-07 13:57:38 8 Comments

I trained a Many-to-Many sequence model in Keras with `return_sequences=True`

and `TimeDistributed`

wrapper on the last Dense layer:

```
model = Sequential()
model.add(Embedding(input_dim=vocab_size, output_dim=50))
model.add(LSTM(100, return_sequences=True))
model.add(TimeDistributed(Dense(vocab_size, activation='softmax')))
# train...
model.save_weights("weights.h5")
```

So during the training the loss is calculated over all hidden states (in every timestamp). But for inference I only need the get output on the last timestamp. So I load the weights into Many-to-One sequence model for inference without `TimeDistributed`

wrapper and I set `return_sequences=False`

to get only last output of the LSTM layer:

```
inference_model = Sequential()
inference_model.add(Embedding(input_dim=vocab_size, output_dim=50))
inference_model.add(LSTM(100, return_sequences=False))
inference_model.add(Dense(vocab_size, activation='softmax'))
inference_model.load_weights("weights.h5")
```

When I test my inference model on a sequence with length 20 I expect to get a prediction with shape (vocab_size) but `inference_model.predict(...)`

still returns predictions for every timestamp - a tensor of shape (20, vocab_size)

### Related Questions

#### Sponsored Content

#### 12 Answered Questions

#### 36 Answered Questions

#### 7 Answered Questions

#### 1 Answered Questions

### [SOLVED] KERAS: Get a SLICE of RNN timesteps with return_sequence = True

**2018-10-05 16:44:34****B. - S.****177**View**0**Score**1**Answer- Tags: python tensorflow keras tensor

#### 11 Answered Questions

#### 1 Answered Questions

### [SOLVED] Keras: How to get model predictions( or last layer output) in a custom generator during training?

**2019-10-01 23:43:02****Umair Javaid****529**View**2**Score**1**Answer- Tags: python tensorflow keras deep-learning

#### 1 Answered Questions

### [SOLVED] What is the role of TimeDistributed layer in Keras?

**2017-11-15 10:57:45****Buomsoo Kim****26869**View**61**Score**1**Answer- Tags: python machine-learning keras neural-network deep-learning

#### 2 Answered Questions

### [SOLVED] How to use return_sequences option and TimeDistributed layer in Keras?

**2017-03-13 02:35:03****jef****15976**View**39**Score**2**Answer- Tags: deep-learning keras lstm recurrent-neural-network

#### 2 Answered Questions

### [SOLVED] Keras model to predict number sequence

**2017-09-01 13:36:46****Mosu****3268**View**3**Score**2**Answer- Tags: machine-learning neural-network keras lstm recurrent-neural-network

#### 1 Answered Questions

### [SOLVED] How to do softmax when LSTM returns sequence in keras?

**2017-03-31 16:15:44****Taevanbat Mongol****1194**View**1**Score**1**Answer- Tags: machine-learning keras

## 1 comments

## @today 2019-03-07 14:31:38

If, for whatever reason, you need only the last timestep during inference, you can build a new model which applies the trained model on the input and returns the last timestep as its output using the

`Lambda`

layer:Side Note:As already stated in this answer,`TimeDistributed(Dense(...))`

and`Dense(...)`

are equivalent, since`Dense`

layer is applied on the last dimension of its input Tensor. Hence, that's why you get the same output shape.## @nidomo 2019-03-07 15:22:47

Oh. Is there a way to apply TimeDistributed(Dense(...)) to every timestamp of LSTM output?

## @today 2019-03-07 17:12:16

@nidomo Well, I am not sure what you mean exactly as it is already applied on all the timesteps.