Create new tokens for vs code semantic highlighting

Question

How can I create a new legend of tokens for VS code semantic highlighting?

This is the current list available as tokens: https://code.visualstudio.com/api/language-extensions/semantic-highlight-guide#standard-token-types-and-modifiers

Additionally, this is the API used:

const tokenTypes = ['class', 'interface', 'enum', 'function', 'variable'];
const tokenModifiers = ['declaration', 'documentation'];
const legend = new vscode.SemanticTokensLegend(tokenTypes, tokenModifiers);

const provider: vscode.DocumentSemanticTokensProvider = {
  provideDocumentSemanticTokens(
    document: vscode.TextDocument
  ): vscode.ProviderResult<vscode.SemanticTokens> {
    // analyze the document and return semantic tokens

    const tokensBuilder = new vscode.SemanticTokensBuilder(legend);
    // on line 1, characters 1-5 are a class declaration
    tokensBuilder.push(
      new vscode.Range(new vscode.Position(1, 1), new vscode.Position(1, 5)),
      'class',
      ['declaration']
    );
    return tokensBuilder.build();
  }
};

const selector = { language: 'java', scheme: 'file' }; // register for all Java documents from the local file system

vscode.languages.registerDocumentSemanticTokensProvider(selector, provider, legend);

Answer 1

I asked the people on https://github.com/microsoft/language-server-protocol this question. The answer is that you can't define new types and modifiers--even though some programming languages do not fit into this classification scheme.

For example, a language server for a formal parser generator grammar like Antlr4 contains lexer and parser symbols. That has hardly anything to do with class, variable, etc., or modifiers static, readonly, etc.

I tried to propose an arbitrary list of names for classification but was rejected because, I think, there is additional functionality that these classifiers denote. So, you will need to reuse what the client supports, a list you get on initialization, and a subset you return from the server.

For semantic highlighting, it "works". I do this for my "universal language VSCode extension", which takes a formal grammar, a list of classes, and XPath classifers to identify tokens in the input for semantic highlighting.