I am new with Apache spark and am now trying to integrating it with the latest version of Pycharm IDE. I have seen several post on that, and so far I have come up to this.
Here is the configuration screenshot:
I have added both SPARK_HOME and SPARK_HOME/python/lib/py4j.zip here.
Then I have added the root path of pyspark and py4j in the project structure for getting the necessary modules for code completion.
Here are the screenshots:
Up to this, I could import the pyspark module in my IDE, but I have a problem when I run this basic program:
C:\Anaconda3\python.exe "C:/Users/user/PycharmProjects/NewProject/Hello world.py"
Traceback (most recent call last):
File "C:/Users/user/PycharmProjects/NewProject/Hello world.py", line 4, in <module>
sc = SparkContext("local", "Simple App")
File "C:\spark-1.6.1-bin-hadoop2.6\python\lib\pyspark.zip\pyspark\context.py", line 112, in __init__
File "C:\spark-1.6.1-bin-hadoop2.6\python\lib\pyspark.zip\pyspark\context.py", line 245, in _ensure_initialized
File "C:\spark-1.6.1-bin-hadoop2.6\python\lib\pyspark.zip\pyspark\java_gateway.py", line 79, in launch_gateway
File "C:\Anaconda3\lib\subprocess.py", line 950, in __init__
restore_signals, start_new_session)
File "C:\Anaconda3\lib\subprocess.py", line 1220, in _execute_child
startupinfo)
FileNotFoundError: [WinError 2] The system cannot find the file specified
Process finished with exit code 1
I have shown the file path properly here which is actually C:\spark-1.6.1-bin-hadoop2.6/README.md
Is it a configuration error, or is there anything wrong with this code?
I am using Python 3.5
Related
I have installed exiftool (https://smarnach.github.io/pyexiftool/) and I am able to import the library, but I get the following error when trying to run the test data just to see if it works.
ERROR: test_get_metadata (__main__.TestExifTool)
----------------------------------------------------------------------
Traceback (most recent call last):
File "C:\Program Files\Python36\Lib\site-
packages\pyexiftool\test\test_exiftool.py", line 66, in test_get_metadata
with self.et:
File "C:\Program Files\Python36\lib\site-packages\exiftool.py", line 191, in __enter__
self.start()
File "C:\Program Files\Python36\lib\site-packages\exiftool.py", line 174, in start
stderr=devnull)
File "C:\Program Files\Python36\lib\subprocess.py", line 709, in __init__
restore_signals, start_new_session)
File "C:\Program Files\Python36\lib\subprocess.py", line 997, in _execute_child
startupinfo)
FileNotFoundError: [WinError 2] The system cannot find the file specified
I also did run the setup code that is in the exiftool folder and still no luck. I think it might be a library issue or path or the (init.py) file, but I've tried several ways and so I'm here to ask if anyone else has a solution or ideas for me to try and fix it.
I'm running Python 3.6.6 and have tried other versions.
(I can run exiftool in command line, but I have encoded BASE64 images that exiftool doesnt work in command line to fully decode.)
Thank you StarGeek! The problem was that I didn't have the exiftool command tool (the separate application of exiftool) in the right PATH env variable. Once I added the application to the PATH env variables i got it to work. Also in the python exiftool code at line 70 it says that you have to have it in the path or direct it to the executable, that I had missed. Thanks again!
I'm running a python script that exports xml from a database, converts the record to PDF, then attempts to upload both the xml and pdf files to a github repository. The export and pdf conversion work fine, but the process then hangs and outputs the following error message:
Traceback (most recent call last):
File "asExportIncremental.py", line 394, in <module>
main()
File "asExportIncremental.py", line 320, in main
gitPush()
File "asExportIncremental.py", line 308, in gitPush
repo.push()
File "C:\Python27\lib\site-packages\gittle\gittle.py", line 343, in push
return self.push_to(origin_uri, branch_name, progress)
File "C:\Python27\lib\site-packages\gittle\gittle.py", line 338, in push_to
progress=progress
File "C:\Python27\lib\site-packages\dulwich\client.py", line 440, in send_pack
proto, unused_can_read = self._connect('receive-pack', path)
File "C:\Python27\lib\site-packages\dulwich\client.py", line 893, in _connect
port=self.port, username=self.username)
File "C:\Python27\lib\site-packages\dulwich\client.py", line 758, in run_command
stdout=subprocess.PIPE)
File "C:\Python27\lib\subprocess.py", line 390, in __init__
errread, errwrite)
File "C:\Python27\lib\subprocess.py", line 640, in _execute_child
startupinfo)
WindowsError: [Error 2] The system cannot find the file specified
My problem is that I'm not exactly sure where the script is failing. To which file is "WindowsError: [Error 2] The system cannot find the file specified" referring? How do I fix it? Thanks!
The script calls gittle (high-level python git library) which executes dulwich (low-level python git wrapper) which runs ssh and fails. It seems ssh.exe is not in your PATH.
Under MacOSX Sierra, and XQuartz 2.7.11, Python 2.7. I was able to install dryscrape and its required other python packages. I've tried quite a few methods to get some simple code working, but I can't seem to get it working. Here is the simple code:
#!/usr/bin/python
#coding: utf-8
import dryscrape
testurl = 'http://avi.im/stuff/js-or-no-js.html'
dryscrape.start_xvfb()
session = dryscrape.Session()
session.visit(testurl)
exit()
You can see it is a simple python script file. It fails on either the start_svfb() or the Session() line. I can post the errors and such, but can I start a discussion on simply what check list should I setup, run, and test to make sure the X11 environment is right? Do I run the script under OS X terminal or XQuartz? What am I missing? etc.?
Thanks in advance, Lucas.
here is the last traceback from python:
Traceback (most recent call last):
File "./fred.py", line 19, in
session = dryscrape.Session()
File "build/bdist.macosx-10.12-intel/egg/dryscrape/session.py", line 22, in __init__
File "build/bdist.macosx-10.12-intel/egg/dryscrape/driver/webkit.py", line 30, in __init__
File "/Library/Python/2.7/site-packages/webkit_server-1.0-py2.7.egg/webkit_server.py", line 230, in __init__
self.conn = connection or ServerConnection()
File "/Library/Python/2.7/site-packages/webkit_server-1.0-py2.7.egg/webkit_server.py", line 507, in __init__
self._sock = (server or get_default_server()).connect()
File "/Library/Python/2.7/site-packages/webkit_server-1.0-py2.7.egg/webkit_server.py", line 450, in get_default_server
_default_server = Server()
File "/Library/Python/2.7/site-packages/webkit_server-1.0-py2.7.egg/webkit_server.py", line 416, in __init__
stderr = subprocess.PIPE)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 710, in __init__
errread, errwrite)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1335, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
I was following the documentation of pgs4a, where in the build part I got a WindowsError [Error 2].
The Command and Traceback:
C:\Users\karuna\Desktop\Jython\Python\pgs4a-0.9.4>android.py build "C:\Users\karuna\Desktop\Jython\P
ython\pgs4a-0.9.4\mygame" release install
Updating build files.
Traceback (most recent call last):
File "C:\Users\karuna\Desktop\Jython\Python\pgs4a-0.9.4\android.py", line 66, in <module>
main()
File "C:\Users\karuna\Desktop\Jython\Python\pgs4a-0.9.4\android.py", line 54, in main
build.build(iface, args.argument[0], args.argument[1:])
File "buildlib\build.py", line 301, in build
build_core(iface, directory, commands)
File "buildlib\build.py", line 221, in build_core
subprocess.call([plat.android, "update", "project", "-p", '.', '-t', 'android-8', '-n', versione
d_name])
File "C:\Python2.7 For Chintoo\lib\subprocess.py", line 493, in call
return Popen(*popenargs, **kwargs).wait()
File "C:\Python2.7 For Chintoo\lib\subprocess.py", line 679, in __init__
errread, errwrite)
File "C:\Python2.7 For Chintoo\lib\subprocess.py", line 893, in _execute_child
startupinfo)
WindowsError: [Error 2] The system cannot find the file specified
As obvious,
I am using Windows 7, and have:
Python 2.7.3
JDK
Android Emulator
Device drivers for my Sony Tipo.
Please help!
By the way,
I would prefer to get this running on my phone first. I am thinking of moving to the emulator later(after the tutorial is completed).
Is this question more suitable for https://android.stackexchange.com/?
I am not sure if this is still an issue for you, but I ran into the similar issue recently and managed to solve it so I will just leave a note here.
If anyone gets this kind of error, go to pgs4a folder and android-sdk folder. There, run SDK Manager.exe file. You will see a list of Tools and different android versions. Try to find the version that you are working on and install everything that is 'uninstalled'.
I managed to solve the above issue by installing everything under my current AND previous Android versions. Hope this will help anyone who ran into the same issue!
rename the folder android-sdk-windows to android-sdk
I am trying to run an existing python code, and having issues with it.
This program required npm program installed and which is installed at C:\Program Files\nodejs\npm in my computer. When I run the following code, as a part of the whole program, it throws errors.
def popen_results(args):
proc = subprocess.Popen(args, stdout=subprocess.PIPE)
return proc.communicate()[0]
def installed():
"""docstring for npm_installed"""
return popen_results(["which", "npm"]).strip()
This is the complete stack of the error thrown--
Checking for node and dependencies
Traceback (most recent call last):
File "deploy\deploy.py", line 344, in <module>
main()
File "deploy\deploy.py", line 287, in main
if not check_deps():
File "deploy\deploy.py", line 201, in check_deps
return npm.check_dependencies()
File "C:\Documents and Settings\Sunil\workspace\khan\src\deploy\npm.py", line
38, in check_dependencies
if not installed():
File "C:\Documents and Settings\Sunil\workspace\khan\src\deploy\npm.py", line
13, in installed
return popen_results(["which", "npm"]).strip()
File "C:\Documents and Settings\Sunil\workspace\khan\src\deploy\npm.py", line
8, in popen_results
proc = subprocess.Popen(args, stdout=subprocess.PIPE)
File "C:\python25\lib\subprocess.py", line 594, in __init__
errread, errwrite)
File "C:\python25\lib\subprocess.py", line 822, in _execute_child
startupinfo)
WindowsError: [Error 2] The system cannot find the file specified
I agree with martineau, it is unable to find which. The script may have been written with the assumption it was going to be run in a unix environment which would most likely have the "which" command available and in the default PATH. Since it looks like you're running this on Windows, I don't think it's going to work.
It looks like there is some alternatives to which on Windows though, discussed here: Is there an equivalent to which on Windows?