If i understand MRJob correctly, you can simulate hadoop's multi process run using MRJob by running it with
python mrfile.py -r local input.txt
I'm running windows(no choice for now), and when I issue the above command, i'm getting a bunch of mambo jumbo and at the end it tells me :
WindowsError: [Error 2] The system cannot find the file specified
This is the full error. Could someone help?
C:\Users\someuser\Documents\Python_projects\something>python MRJob_parser.py -r loc
al test2.txt
no configs found; falling back on auto-configuration
no configs found; falling back on auto-configuration
creating tmp directory c:\users\someuser\appdata\local\temp\MRJob_parser.someuser.
20150701.211822.496000
writing wrapper script to c:\users\someuser\appdata\local\temp\MRJob_parser.bw401
45.20150701.211822.496000\setup-wrapper.sh
writing to c:\users\someuser\appdata\local\temp\MRJob_parser.someuser.20150701.211
822.496000\step-0-mapper_part-00000
> sh -ex setup-wrapper.sh 'c:\Users\someuser\Documents\python_venv\something_Project\
Scripts\python.exe' MRJob_parser.py --step-num=0 --mapper 'c:\users\someuser\appd
ata\local\temp\MRJob_parser.someuser.20150701.211822.496000\input_part-00000' > c
:\users\someuser\appdata\local\temp\MRJob_parser.someuser.20150701.211822.496000\s
tep-0-mapper_part-00000
Traceback (most recent call last):
File "MRJob_parser.py", line 18, in <module>
Extractor.run()
File "c:\Users\someuser\Documents\python_venv\something_Project\lib\site-packages\m
rjob\job.py", line 461, in run
mr_job.execute()
File "c:\Users\someuser\Documents\python_venv\something_Project\lib\site-packages\m
rjob\job.py", line 479, in execute
super(MRJob, self).execute()
File "c:\Users\someuser\Documents\python_venv\something_Project\lib\site-packages\m
rjob\launch.py", line 151, in execute
self.run_job()
File "c:\Users\someuser\Documents\python_venv\something_Project\lib\site-packages\m
rjob\launch.py", line 214, in run_job
runner.run()
File "c:\Users\someuser\Documents\python_venv\something_Project\lib\site-packages\m
rjob\runner.py", line 464, in run
self._run()
File "c:\Users\someuser\Documents\python_venv\something_Project\lib\site-packages\m
rjob\sim.py", line 173, in _run
self._invoke_step(step_num, 'mapper')
File "c:\Users\someuser\Documents\python_venv\something_Project\lib\site-packages\m
rjob\sim.py", line 260, in _invoke_step
working_dir, env)
File "c:\Users\someuser\Documents\python_venv\something_Project\lib\site-packages\m
rjob\local.py", line 147, in _run_step
procs_args, output_path, working_dir, env)
File "c:\Users\someuser\Documents\python_venv\something_Project\lib\site-packages\m
rjob\local.py", line 250, in _invoke_processes
cwd=working_dir, env=env)
File "c:\Users\someuser\Documents\python_venv\something_Project\lib\site-packages\m
rjob\local.py", line 73, in _chain_procs
proc = Popen(args, **proc_kwargs)
File "C:\Python27\Lib\subprocess.py", line 710, in __init__
errread, errwrite)
File "C:\Python27\Lib\subprocess.py", line 958, in _execute_child
startupinfo)
WindowsError: [Error 2] The system cannot find the file specified
C:\Users\someuser\Documents\Python_projects\something>
From https://github.com/Yelp/mrjob:
# locally
python mrjob/examples/mr_word_freq_count.py README.rst > counts
My guess is that you don't have to pass in the local runner option and that's causing some kind of error.
Related
I tried to create an APK file by using WSL and get this:
# Unpacking Android NDK
# Run ['unzip', '-q', '/home/tarpetos/.buildozer/android/platform/android-ndk-r23b-linux.zip']
# Cwd /home/tarpetos/.buildozer/android/platform
Traceback (most recent call last):
File "/usr/local/bin/buildozer", line 11, in <module>
load_entry_point('buildozer==1.4.1.dev0', 'console_scripts', 'buildozer')()
File "/usr/local/lib/python3.8/dist-packages/buildozer-1.4.1.dev0-py3.8.egg/buildozer/scripts/client.py", line 13, in main
Buildozer().run_command(sys.argv[1:])
File "/usr/local/lib/python3.8/dist-packages/buildozer-1.4.1.dev0-py3.8.egg/buildozer/__init__.py", line 1024, in run_command
self.target.run_commands(args)
File "/usr/local/lib/python3.8/dist-packages/buildozer-1.4.1.dev0-py3.8.egg/buildozer/target.py", line 93, in run_commands
func(args)
File "/usr/local/lib/python3.8/dist-packages/buildozer-1.4.1.dev0-py3.8.egg/buildozer/target.py", line 103, in cmd_debug
self.buildozer.prepare_for_build()
File "/usr/local/lib/python3.8/dist-packages/buildozer-1.4.1.dev0-py3.8.egg/buildozer/__init__.py", line 172, in prepare_for_build
self.target.install_platform()
File "/usr/local/lib/python3.8/dist-packages/buildozer-1.4.1.dev0-py3.8.egg/buildozer/targets/android.py", line 701, in install_platform
self._install_android_ndk()
File "/usr/local/lib/python3.8/dist-packages/buildozer-1.4.1.dev0-py3.8.egg/buildozer/targets/android.py", line 497, in _install_android_ndk
self.buildozer.file_extract(archive,
File "/usr/local/lib/python3.8/dist-packages/buildozer-1.4.1.dev0-py3.8.egg/buildozer/__init__.py", line 616, in file_extract
self.cmd(["unzip", "-q", join(cwd, archive)], cwd=cwd)
File "/usr/local/lib/python3.8/dist-packages/buildozer-1.4.1.dev0-py3.8.egg/buildozer/__init__.py", line 289, in cmd
process = Popen(command, **kwargs)
File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
self._execute_child(args, executable, preexec_fn, close_fds,
File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'unzip'
I have a main.py file and a connection_voice.ogg file in the same directory and at first I thought the problem was connecting the audio file. But now I understand that the problem is not in the audio file.
music = SoundLoader.load('connection_voice.ogg')
if music:
music.play()
And I don't understand what an 'unzip' file or directory is. Please give some advice
So, to solve this you need to enter to terminal sudo apt-get install unzip. And than this problem will be solved.
I'm trying to to write a simple python app and launch it using TorchX following this Quickstart guide. Following commands are executed in my home directory:
pip3 install "torchx[dev]"
Write a simple python program using vim editor vi my_app.py
# my_app.py
%%writefile my_app.py
import sys
print(f"Hello, {sys.argv[1]}!")
Execute the app via torchx run as torchx run --scheduler local_cwd utils.python --script my_app.py "your name".
And then I get a long error:
Log files located in: /var/folders/h7/qrsyzj5s7hvbkwmv83by9wgc0000gn/T/torchx_hykfxkz3/torchx/torchx_utils_python-vxm9hhpt3k7wtd/python/0
Traceback (most recent call last):
File "/usr/local/bin/torchx", line 8, in <module>
sys.exit(main())
File "/usr/local/lib/python3.9/site-packages/torchx/cli/main.py", line 110, in main
run_main(get_sub_cmds(), argv)
File "/usr/local/lib/python3.9/site-packages/torchx/cli/main.py", line 106, in run_main
args.func(args)
File "/usr/local/lib/python3.9/site-packages/torchx/cli/cmd_run.py", line 271, in run
self._run(runner, args)
File "/usr/local/lib/python3.9/site-packages/torchx/cli/cmd_run.py", line 232, in _run
app_handle = runner.run_component(
File "/usr/local/lib/python3.9/site-packages/torchx/runner/api.py", line 147, in run_component
return self.schedule(dryrun_info)
File "/usr/local/lib/python3.9/site-packages/torchx/runner/api.py", line 225, in schedule
app_id = sched.schedule(dryrun_info)
File "/usr/local/lib/python3.9/site-packages/torchx/schedulers/local_scheduler.py", line 748, in schedule
replica = self._popen(
File "/usr/local/lib/python3.9/site-packages/torchx/schedulers/local_scheduler.py", line 688, in _popen
proc = subprocess.Popen(
File "/usr/local/Cellar/python#3.9/3.9.13/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 951, in __init__
self._execute_child(args, executable, preexec_fn, close_fds,
File "/usr/local/Cellar/python#3.9/3.9.13/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 1821, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'python'
I have Python 3.9.13 installed on macOS Monterey (version 12.4).
What could be a cause of this error?
I have been having some difficulties compiling my Python program using Pyinstaller. My program has multiple modules and about 2000 lines of code. It uses TaurusScada as a sort of wrapper for PYQT, Numpy, and a few other modules. The platform is Python 2.7 and Ubuntu 15.04. I'm also using Pyinstaller 3.1. I was successful in compiling a small "Hello World" type application so I know it works. I am not married to Pyinstaller as I have tried cx_Freeze as well without success. Rather than leaving my entire code on here I'll start by leaving the traceback and error in hopes that someone can shed some light on this:
66874 INFO: Processing hook hook-gi.repository.GdkPixbuf.py
Traceback (most recent call last):
File "/usr/local/bin/pyinstaller", line 11, in <module>
sys.exit(run())
File "/usr/local/lib/python2.7/dist-packages/PyInstaller/__main__.py", line 90, in run
run_build(pyi_config, spec_file, **vars(args))
File "/usr/local/lib/python2.7/dist-packages/PyInstaller/__main__.py", line 46, in run_build
PyInstaller.building.build_main.main(pyi_config, spec_file, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/PyInstaller/building/build_main.py", line 755, in main
build(specfile, kw.get('distpath'), kw.get('workpath'), kw.get('clean_build'))
File "/usr/local/lib/python2.7/dist-packages/PyInstaller/building/build_main.py", line 701, in build
exec(text, spec_namespace)
File "<string>", line 16, in <module>
File "/usr/local/lib/python2.7/dist-packages/PyInstaller/building/build_main.py", line 212, in __init__
self.__postinit__()
File "/usr/local/lib/python2.7/dist-packages/PyInstaller/building/datastruct.py", line 183, in __postinit__
self.assemble()
File "/usr/local/lib/python2.7/dist-packages/PyInstaller/building/build_main.py", line 432, in assemble
imphook_object = ImportHook(imported_name, hook_file)
File "/usr/local/lib/python2.7/dist-packages/PyInstaller/building/imphook.py", line 182, in __init__
self._module = importlib_load_source(hook_modname, self._filename)
File "/usr/local/lib/python2.7/dist-packages/PyInstaller/hooks/hook-gi.repository.GdkPixbuf.py", line 36, in <module>
cachedata = subprocess.check_output('gdk-pixbuf-query-loaders')
File "/usr/lib/python2.7/subprocess.py", line 566, in check_output
process = Popen(stdout=PIPE, *popenargs, **kwargs)
File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1335, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
Any help is much appreciated.
i had a similar error today. You are missing the gdk-pixbuf library:
so just run:
sudo apt-get install libgdk-pixbuf2.0-dev
and pyinstaller should go smoothly
I want to execute some Java code from Python so I've decided to install standard python interpreter, jython and join them together using pyro4. Pyro4 requires python > 2.5 so I choose to use jython 2.7b. Here are steps I made to make this happen:
wget -i jython-installer-2.7-b2.jar http://search.maven.org/remotecontent?filepath=org/python/jython-installer/2.7-b1/jython-installer-2.7-b2.jar
java -jar jython-installer-2.7-b2.jar
jython2.7b2/bin/virtualenv-2.7 oscar
And this is what I'm getting back:
Cannot find file /home/mnowotka/jython2.7b2/Include (bad symlink)
New jython executable in oscar/bin/jython
Installing setuptools.................
Complete output from command /home/mnowotka/oscar/bin/jython -c "#!python
\"\"\"Bootstra...sys.argv[1:])
" /home/mnowotka/jytho...ols-0.6c11-py2.7.egg:
Downloading http://pypi.python.org/packages/2.7/s/setuptools/setuptools-0.6c11-py2.7.egg
Traceback (most recent call last):
File "<string>", line 278, in <module>
File "<string>", line 210, in main
File "<string>", line 161, in download_setuptools
File "/home/mnowotka/jython2.7b2/Lib/socket.py", line 1651, in read
data = self._sock.recv(recv_size)
File "/home/mnowotka/jython2.7b2/Lib/httplib.py", line 567, in read
s = self.fp.read(amt)
File "/home/mnowotka/jython2.7b2/Lib/socket.py", line 1669, in read
data = self._sock.recv(recv_size)
File "/home/mnowotka/jython2.7b2/Lib/socket.py", line 174, in handle_exception
raise _map_exception(jlx)
socket.error: [Errno 32] Socket closed
----------------------------------------
...Installing setuptools...done.
Traceback (most recent call last):
File "jython2.7b2/bin/virtualenv-2.7", line 3, in <module>
virtualenv.main()
File "/home/mnowotka/jython2.7b2/Lib/site-packages/virtualenv.py", line 970, in main
create_environment(home_dir,
File "/home/mnowotka/jython2.7b2/Lib/site-packages/virtualenv.py", line 1090, in create_environment
install_setuptools(py_executable, unzip=unzip_setuptools,
File "/home/mnowotka/jython2.7b2/Lib/site-packages/virtualenv.py", line 610, in install_setuptools
_install_req(py_executable, unzip,
File "/home/mnowotka/jython2.7b2/Lib/site-packages/virtualenv.py", line 579, in _install_req
call_subprocess(cmd, show_stdout=False,
File "/home/mnowotka/jython2.7b2/Lib/site-packages/virtualenv.py", line 1055, in call_subprocess
raise OSError(
OSError: Command /home/mnowotka/oscar/bin/jython -c "#!python
\"\"\"Bootstra...sys.argv[1:])
Does it mean that jython 2.7b2 is not quite ready yet or I'm doing something wrong?
Use this ez_setup.py to install setuptools:
added setuptools-egg and md5
While running buildout on window with Python 2.6 for Plone 4.2, All went fine. but while running instance on foreground it give me this window specific (presume) error about some file missing.
I remember on older version of Plone for installing PyWin type libary, I had to copy some dll file to window system file and I also done same for this. still I am having this error.
Any suggestion/advise would great help.
Thank for taking your time.
D:\rcseng\buildout>bin\instance fg
Traceback (most recent call last):
File "D:\rcseng\buildout\bin\instance-script.py", line 258, in <module>
+ sys.argv[1:])
File "d:\rcseng\buildout\eggs\plone.recipe.zope2instance-4.2.8-py2.6.egg\plone
\recipe\zope2instance\ctl.py", line 614, in main
c.onecmd(' '.join(options.args))
File "D:\Python26\lib\cmd.py", line 219, in onecmd
return func(arg)
File "d:\rcseng\buildout\eggs\zdaemon-2.0.7-py2.6.egg\zdaemon\zdctl.py", line
588, in do_fg
self.do_foreground(arg)
File "d:\rcseng\buildout\eggs\plone.recipe.zope2instance-4.2.8-py2.6.egg\plone
\recipe\zope2instance\ctl.py", line 550, in do_foreground
self._exitstatus = subprocess.call(command, env=env)
File "D:\Python26\lib\subprocess.py", line 470, in call
return Popen(*popenargs, **kwargs).wait()
File "D:\Python26\lib\subprocess.py", line 623, in __init__
errread, errwrite)
File "D:\Python26\lib\subprocess.py", line 833, in _execute_child
startupinfo)
WindowsError: [Error 2] The system cannot find the file specified