Spark: Trying to run spark-shell, but get 'cmd' is not recognized as an internal or

Question

I'm trying to install Spark on my Windows desktop. Everything should work fine, but I get an error "'cmd' is not recognized as an internal or external command... "

I installed Scala, Java JDK and unzipped Spark tgz in C:\, but for some reason can't get Spark to start in cmd. Any ideas?

Answer 1

I had the similar error. I fixed it after following changes:

1. There were multiple Java/bin path in the System Path. So I corrected them to reflect single Java/Bin, which is in sync with JAVA_HOME
2. Added C:Windows\system32 to System Path Variable.
3. My Java_Home and java.exe was pointing different places. I fixed them.

Now it works.

Thanks guys.

Answer 2

(I'm not Windows Spark user) The spark-shell.cmd for Windows source code expects "cmd" command is available in PATH.

https://github.com/apache/spark/blob/master/bin/spark-shell.cmd

Would you try adding the directory that contains "cmd.exe" in PATH environment variable? The directory location is shown title bar in your screenshot, and environment variable setting can be done via control panel.

Answer 3

Check values in JAVA_HOME and make sure it is pointing to correct value. Add %JAVA_HOME%/bin in path value. After modification close command prompt and restart it. Write spark-shell and it will run.

Answer 4

I was the getting the same error while executing Spark-shell in the command prompt.

I tried everything mentioned above but not able to resolve the issue.

So, at last I added "C:\Windows\System32" in 'PATH' variable of System Variable and it worked.

Answer 5

All my variables were OK, so I decided to debug the scripts and I found in "spark-class2.cmd" and put another pair of quotation marks in "%RUNNERS%". BEFORE "%RUNNER%" -Xmx128m -cp "%LAUNCH_CLASSPATH%" .... AFTER ""%RUNNER%"" -Xmx128m -cp "%LAUNCH_CLASSPATH%" ....

Answer 6

For my case I had similar issue. I had to fix a couple of things.

1- Check JAVA_HOME is correct on both places;

2- Then I had to change following lines in spark-2.1.1-bin-hadoop2.7\bin folder.

• Add extra quotation marks around "%RUNNER%". So it will be like ""%RUNNER%""
• Then execute .\spark-shell.cmd again.

Answer 7

Check the java jdk version and scala version according to the below version compatibility table:

Answer 8

I had the same issue (launch spark-shell and get the the system cannot find the path) After following the above process (changing the spark_home to exclude the /bin it worked fine thanks for sharing guys

Answer 9

These are detailed steps to resolve all these issues in Windows

Spark binaries

1. It needs to be in a folder which doesn’t have space, if it must be in a folder which has space then you can use something like Unix’s symlink. I have kept the Spark binaries inside c:\ directly. Set SPARK_HOME to e.g. C:\SparkMedia\spark-2.4.2-bin-hadoop2.7. Please note “bin” is not part of the path.
2. Set “path” for the user and system to %SPARK_HOME%\bin If you decide to keep it outside “program files” (as per point a), then please give permission to the user because Spark keeps temporary files here. If you want to keep Spark inside a directory where user doesn’t have access then you need to put a file %SPARK_HOME%\conf\spark-env.sh with an entry inside it as “SPARK_LOG_DIR=a directory where user has access”.

Winutils

1. It can be inside a folder with space like C:\Program Files (x86)\Winutils Your winutils kept at C:\Program Files (x86)\Winutils works fine.
2. Set HADOOP_HOME to C:\Program Files (x86)\Winutils. Please note “bin” is not part of the path.
3. Set “path” for the user and system to %HADOOP_HOME%\bin

Java

4. It can be inside a folder with space like C:\Program Files\AdoptOpenJDK\jdk-8.0.262.10-hotspot Set JAVA_HOME to C:\Program Files\AdoptOpenJDK\jdk-8.0.262.10-hotspot. Please note “bin” is not part of the path.

5. Set “path” for the user and system to %JAVA_HOME%\bin

With the above setup a user should be able to just type “spark-shell” and spark shell should start.

Answer 10

The spark-shell command runs the spark-shell.cmd file in the spark 'bin' folder. This file contains the following command cmd /V /E /C ""%~dp0spark-shell2.cmd" %*".

The cmd command requires the path system variable to be set to %SystemRoot%\system32

Answer 11

It should be the issue with your System Variables path. As what have mention in the error message, your command prompt can't recognize 'cmd'. So, you should add the directory contain cmd.exe into PATH list under your System variables so that your command prompt know where to find 'cmd'.

Steps

1. Open Control Panel > System.
2. Click on "Advanced system settings" (Right side under Related settings).
3. Click on "Environment Variables..." at the bottom.
4. Under "System variables" section look for "Path" and click on it, then click "Edit" button at the bottom. System variable setting
5. In the editing tab, add "C:\Windows\System32" or "%SystemRoot%\System32".

After you added the new Path into System variable,

1. Open Command Prompt
2. cd %SPARK_HOME%\bin or cd to the spark directory.
3. spark-shell

If everythings done right, you should be able to see the screen below. spark-shell

If still not working, you should check whether you have added SPARK_HOME, HADOOP_HOME, JAVA_HOME correctly under your User variables.