Could not build wheels for tokenizers, which is required to install pyproject.toml-based projects

Question

I am trying to install the ernie and tokenizers on Windows, but for both cases it is showing the following error:

ERROR: Command errored out with exit status 1:
   command: 'c:\users\my pc\appdata\local\programs\python\python39\python.exe' 'c:\users\my pc\appdata\local\programs\python\python39\lib\site-packages\pip\_vendor\pep517\in_process\_in_process.py' build_wheel 'C:\Users\MYPC~1\AppData\Local\Temp\tmpyfqm86h3'
       cwd: C:\Users\my pc\AppData\Local\Temp\pip-install-504a8mvv\tokenizers_a45d458877334719a3bf67c944dce6d7
  Complete output (38 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build\lib
  creating build\lib\tokenizers
  copying tokenizers\__init__.py -> build\lib\tokenizers
  running build_ext
  running build_rust
      Updating crates.io index
  cargo rustc --lib --manifest-path Cargo.toml --features pyo3/extension-module --target x86_64-pc-windows-msvc --release --verbose -- --crate-type cdylib
  warning: unused manifest key: target.x86_64-apple-darwin.rustflags
     Compiling proc-macro2 v1.0.30
     Compiling unicode-xid v0.2.2
     Compiling syn v1.0.80
     Compiling autocfg v1.0.1
       Running `rustc --crate-name build_script_build --edition=2018 "C:\Users\my pc\.cargo\registry\src\github.com-1ecc6299db9ec823\proc-macro2-1.0.30\build.rs" --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debug-assertions=off --cfg "feature=\"default\"" --cfg "feature=\"proc-macro\"" -C metadata=0b72d9facdf9b740 -C extra-filename=-0b72d9facdf9b740 --out-dir "C:\Users\my pc\AppData\Local\Temp\pip-install-504a8mvv\tokenizers_a45d458877334719a3bf67c944dce6d7\target\release\build\proc-macro2-0b72d9facdf9b740" -L "dependency=C:\Users\my pc\AppData\Local\Temp\pip-install-504a8mvv\tokenizers_a45d458877334719a3bf67c944dce6d7\target\release\deps" --cap-lints allow`
       Running `rustc --crate-name unicode_xid "C:\Users\my pc\.cargo\registry\src\github.com-1ecc6299db9ec823\unicode-xid-0.2.2\src\lib.rs" --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debug-assertions=off --cfg "feature=\"default\"" -C metadata=c5404e573edb1c5f -C extra-filename=-c5404e573edb1c5f --out-dir "C:\Users\my pc\AppData\Local\Temp\pip-install-504a8mvv\tokenizers_a45d458877334719a3bf67c944dce6d7\target\release\deps" -L "dependency=C:\Users\my pc\AppData\Local\Temp\pip-install-504a8mvv\tokenizers_a45d458877334719a3bf67c944dce6d7\target\release\deps" --cap-lints allow`
       Running `rustc --crate-name build_script_build --edition=2018 "C:\Users\my pc\.cargo\registry\src\github.com-1ecc6299db9ec823\syn-1.0.80\build.rs" --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debug-assertions=off --cfg "feature=\"clone-impls\"" --cfg "feature=\"default\"" --cfg "feature=\"derive\"" --cfg "feature=\"extra-traits\"" --cfg "feature=\"full\"" --cfg "feature=\"parsing\"" --cfg "feature=\"printing\"" --cfg "feature=\"proc-macro\"" --cfg "feature=\"quote\"" -C metadata=b3126876d28730d4 -C extra-filename=-b3126876d28730d4 --out-dir "C:\Users\my pc\AppData\Local\Temp\pip-install-504a8mvv\tokenizers_a45d458877334719a3bf67c944dce6d7\target\release\build\syn-b3126876d28730d4" -L "dependency=C:\Users\my pc\AppData\Local\Temp\pip-install-504a8mvv\tokenizers_a45d458877334719a3bf67c944dce6d7\target\release\deps" --cap-lints allow`
       Running `rustc --crate-name autocfg "C:\Users\my pc\.cargo\registry\src\github.com-1ecc6299db9ec823\autocfg-1.0.1\src\lib.rs" --error-format=json --json=diagnostic-rendered-ansi --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debug-assertions=off -C metadata=077a4bc4a690ef7e -C extra-filename=-077a4bc4a690ef7e --out-dir "C:\Users\my pc\AppData\Local\Temp\pip-install-504a8mvv\tokenizers_a45d458877334719a3bf67c944dce6d7\target\release\deps" -L "dependency=C:\Users\my pc\AppData\Local\Temp\pip-install-504a8mvv\tokenizers_a45d458877334719a3bf67c944dce6d7\target\release\deps" --cap-lints allow`
     Compiling memchr v2.4.1
       Running `rustc --crate-name build_script_build --edition=2018 "C:\Users\my pc\.cargo\registry\src\github.com-1ecc6299db9ec823\memchr-2.4.1\build.rs" --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debug-assertions=off --cfg "feature=\"default\"" --cfg "feature=\"std\"" -C metadata=420338dd5ea363a5 -C extra-filename=-420338dd5ea363a5 --out-dir "C:\Users\my pc\AppData\Local\Temp\pip-install-504a8mvv\tokenizers_a45d458877334719a3bf67c944dce6d7\target\release\build\memchr-420338dd5ea363a5" -L "dependency=C:\Users\my pc\AppData\Local\Temp\pip-install-504a8mvv\tokenizers_a45d458877334719a3bf67c944dce6d7\target\release\deps" --cap-lints allow`
  error: linker `link.exe` not found
    |
    = note: The system cannot find the file specified. (os error 2)

  note: the msvc targets depend on the msvc linker but `link.exe` was not found

  note: please ensure that VS 2013, VS 2015, VS 2017 or VS 2019 was installed with the Visual C++ option

  error: could not compile `syn` due to previous error

  Caused by:
    process didn't exit successfully: `rustc --crate-name build_script_build --edition=2018 "C:\Users\my pc\.cargo\registry\src\github.com-1ecc6299db9ec823\syn-1.0.80\build.rs" --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debug-assertions=off --cfg "feature=\"clone-impls\"" --cfg "feature=\"default\"" --cfg "feature=\"derive\"" --cfg "feature=\"extra-traits\"" --cfg "feature=\"full\"" --cfg "feature=\"parsing\"" --cfg "feature=\"printing\"" --cfg "feature=\"proc-macro\"" --cfg "feature=\"quote\"" -C metadata=b3126876d28730d4 -C extra-filename=-b3126876d28730d4 --out-dir "C:\Users\my pc\AppData\Local\Temp\pip-install-504a8mvv\tokenizers_a45d458877334719a3bf67c944dce6d7\target\release\build\syn-b3126876d28730d4" -L "dependency=C:\Users\my pc\AppData\Local\Temp\pip-install-504a8mvv\tokenizers_a45d458877334719a3bf67c944dce6d7\target\release\deps" --cap-lints allow` (exit code: 1)
  warning: build failed, waiting for other jobs to finish...
  error: build failed
  error: cargo failed with code: 101

  ----------------------------------------
  ERROR: Failed building wheel for tokenizers
Failed to build tokenizers
ERROR: Could not build wheels for tokenizers, which is required to install pyproject.toml-based projects

Answer 1

It need to install rust compiler in your environment, you can use the command:

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

and then you can do it successfully.

Answer 2

I solved this problem by installing Visual Studio 2019 Community Edition with Python development workloads. I am not sure whether the Python development workloads were essential. Rust compiler has already been installed. (rustup-init.exe from www.rust-lang.org)

Edit: When I uninstalled Python 3.10, Visual Studio, and Rust completely and tried again with only Python 3.10 and Visual Studio Community edition with Python development workloads, it didn't work. But it succeeded after I installed Desktop Development with C++ workloads.

Answer 3

So what worked for me is twofold: -1. Install Rust -2. Install Visual Studio (latest version, 2022) and make sure to install BOTH Python workloads and Desktop Development C++ Workloads.

Answer 4

It seems that the corresponding package isn't installed on your machine, you need to install wheel.
try these & let me know if it works...

• pip3 install --upgrade pip
• pip3 install --upgrade pip setuptools wheel
  

or pip3 install wheel.

Answer 5

If you have not installed Java, install it and try again