'texts_to_sequences return empty string after loading the tokenizer
I'm working on a project, I've trained and saved my model as well as tokenizer
tokenizer = Tokenizer()
tokenizer.fit_on_texts(corpus)
# save tokenizer
import pickle
# saving
with open('english_tokenizer_test.pickle', 'wb') as handle:
pickle.dump(tokenizer, handle, protocol=pickle.HIGHEST_PROTOCOL)
total_words = len(tokenizer.word_index) +1
from keras.models import load_model
model2.save('model.h5')
and loading the model and the tokenizer
with open('../input/shyaridatasettesting/english_tokenizer_test.pickle', 'rb') as handle:
tokenizer = pickle.load(handle)
tokenizer = Tokenizer()
max_sequence_len = 24
model = tf.keras.models.load_model('../input/model.h5')
print(model);
So when I'm doing tokenizer using loaded tokenizer it returns an empty string
token_list = tokenizer.texts_to_sequences(["this is something"])[0]
I want to use my model and tokenizer on my website but I'm getting an empty token_list whenever I pass text in into my tokenizer?
please help me if I'm doing something wrong.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
