'Contextual Tokenizer that depends on the context in Transformer NLP
I would like to create a tokenizer that tokenizes words in a sentence depending on its surroundings. So one word could have different tokens depending on its neighbor words. For example, if I have these two sentences: "Soccer is fun" and "She plays soccer really well". I would like to create a tokenizer that creates different tokens for the same word "soccer" because the surrounding words are different. Is doing such a tokenizer possible, and if somebody can direct me on how to do it (some links or resources)?
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
