I successfully installed Scrapy.
I created a virtual environment on ubuntu which is on the new windows terminal.
when i try to create a new project i get this error
Traceback (most recent call last):
File "/mnt/d/Scrapy/venv/bin/scrapy", line 8, in <module>
sys.exit(execute())
File "/mnt/d/Scrapy/venv/lib/python3.8/site-packages/scrapy/cmdline.py", line 143, in execute
_run_print_help(parser, _run_command, cmd, args, opts)
File "/mnt/d/Scrapy/venv/lib/python3.8/site-packages/scrapy/cmdline.py", line 98, in _run_print_help
func(*a, **kw)
File "/mnt/d/Scrapy/venv/lib/python3.8/site-packages/scrapy/cmdline.py", line 151, in _run_command
cmd.run(args, opts)
File "/mnt/d/Scrapy/venv/lib/python3.8/site-packages/scrapy/commands/startproject.py", line 110, in run
self._copytree(self.templates_dir, abspath(project_dir))
File "/mnt/d/Scrapy/venv/lib/python3.8/site-packages/scrapy/commands/startproject.py", line 83, in _copytree
self._copytree(srcname, dstname)
File "/mnt/d/Scrapy/venv/lib/python3.8/site-packages/scrapy/commands/startproject.py", line 83, in _copytree
self._copytree(srcname, dstname)
File "/mnt/d/Scrapy/venv/lib/python3.8/site-packages/scrapy/commands/startproject.py", line 88, in _copytree
copystat(src, dst)
File "/usr/lib/python3.8/shutil.py", line 376, in copystat
_copyxattr(src, dst, follow_symlinks=follow)
File "/usr/lib/python3.8/shutil.py", line 326, in _copyxattr
os.setxattr(dst, name, value, follow_symlinks=follow_symlinks)
PermissionError: [Errno 13] Permission denied: '/mnt/d/Scrapy/demo_pro/module/spiders'
Scrapy does some things with file attributes that don't work on a Windows filesystem under WSL.
As a workaround, you could create the project on the WSL filesystem and then copy the result to where you want it. Something along the lines of:
$ cd /tmp
$ scrapy startproject demo_pro
$ cp -r demo_pro /mnt/d/Scrapy
Related
I have been using jupyter as well as Atom with python for more than a year now. I'm currently using Python 3.7.2 on macOS 10.14. There is no problem running Python from the Terminal and here are the PATH I get after running sys.path:
['', '/Library/Frameworks/Python.framework/Versions/3.7/lib/python37.zip', '/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7', '/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/lib-dynload', '/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages']
I can open a jupyter notebook in jupyter but for some obscure reason, I cannot run the notebook and there is a red Kernel error on the top right of the page with the following error message when I click on it:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tornado/web.py", line 1699, in _execute
result = await result
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tornado/gen.py", line 736, in run
yielded = self.gen.throw(*exc_info) # type: ignore
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/notebook/services/sessions/handlers.py", line 73, in post
type=mtype))
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tornado/gen.py", line 729, in run
value = future.result()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tornado/gen.py", line 736, in run
yielded = self.gen.throw(*exc_info) # type: ignore
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/notebook/services/sessions/sessionmanager.py", line 79, in create_session
kernel_id = yield self.start_kernel_for_session(session_id, path, name, type, kernel_name)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tornado/gen.py", line 729, in run
value = future.result()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tornado/gen.py", line 736, in run
yielded = self.gen.throw(*exc_info) # type: ignore
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/notebook/services/sessions/sessionmanager.py", line 92, in start_kernel_for_session
self.kernel_manager.start_kernel(path=kernel_path, kernel_name=kernel_name)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tornado/gen.py", line 729, in run
value = future.result()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tornado/gen.py", line 209, in wrapper
yielded = next(result)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/notebook/services/kernels/kernelmanager.py", line 160, in start_kernel
super(MappingKernelManager, self).start_kernel(**kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/jupyter_client/multikernelmanager.py", line 110, in start_kernel
km.start_kernel(**kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/jupyter_client/manager.py", line 259, in start_kernel
**kw)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/jupyter_client/manager.py", line 204, in _launch_kernel
return launch_kernel(kernel_cmd, **kw)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/jupyter_client/launcher.py", line 138, in launch_kernel
proc = Popen(cmd, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/subprocess.py", line 775, in __init__
restore_signals, start_new_session)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/subprocess.py", line 1522, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
PermissionError: [Errno 13] Permission denied: '/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7'
here is my PATH (obtained after the command: nano ~/.bash_profile)
# Homebrew
export PATH=/usr/local/bin:$PATH
# Setting PATH for Python 3.7
# The original version is saved in .bash_profile.pysave
PATH="/Library/Frameworks/Python.framework/Versions/3.7/bin:${PATH}"
export PATH
# Setting PATH for Python 3.7
# The original version is saved in .bash_profile.pysave
PATH="/Library/Frameworks/Python.framework/Versions/3.7/bin:${PATH}"
export PATH
I already uninstalled all the python installations on my computer including the anaconda version (except python 2.7 as my mac doesn't accept that I remove it)
ok what i did is that I uninstalled python and reinstalled everything with Homebrew and it worked
Sorry but a bit of a beginner question here. I downloaded Anaconda library that comes with Jupyter Notebook along with a few other programs to hopefully learn Python. Every time I try to open Jupyter Notebook though, the webpage it directs me here localhost:8888/tree and gives me the following error:
404: Not Found error webpage.
Edit: After using the terminal to install pip, I got the following error message:
Exception:
Traceback (most recent call last):
File "C:\Users\Y510P\Documents\Anaconda\lib\shutil.py", line 387, in _rmtree_u
nsafe
os.unlink(fullname)
PermissionError: [WinError 5] Access is denied: 'C:\\Users\\Y510P\\AppData\\Loca
l\\Temp\\pip-0tanc911-uninstall\\users\\y510p\\documents\\anaconda\\lib\\site-pa
ckages\\zmq\\backend\\cython\\constants.cp36-win_amd64.pyd'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\Y510P\Documents\Anaconda\lib\site-packages\pip\basecommand.py",
line 215, in main
status = self.run(options, args)
File "C:\Users\Y510P\Documents\Anaconda\lib\site-packages\pip\commands\install
.py", line 342, in run
prefix=options.prefix_path,
File "C:\Users\Y510P\Documents\Anaconda\lib\site-packages\pip\req\req_set.py",
line 795, in install
requirement.commit_uninstall()
File "C:\Users\Y510P\Documents\Anaconda\lib\site-packages\pip\req\req_install.
py", line 767, in commit_uninstall
self.uninstalled.commit()
File "C:\Users\Y510P\Documents\Anaconda\lib\site-packages\pip\req\req_uninstal
l.py", line 142, in commit
rmtree(self.save_dir)
File "C:\Users\Y510P\Documents\Anaconda\lib\site-packages\pip\_vendor\retrying
.py", line 49, in wrapped_f
return Retrying(*dargs, **dkw).call(f, *args, **kw)
File "C:\Users\Y510P\Documents\Anaconda\lib\site-packages\pip\_vendor\retrying
.py", line 212, in call
raise attempt.get()
File "C:\Users\Y510P\Documents\Anaconda\lib\site-packages\pip\_vendor\retrying
.py", line 247, in get
six.reraise(self.value[0], self.value[1], self.value[2])
File "C:\Users\Y510P\Documents\Anaconda\lib\site-packages\six.py", line 693, i
n reraise
raise value
File "C:\Users\Y510P\Documents\Anaconda\lib\site-packages\pip\_vendor\retrying
.py", line 200, in call
attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
File "C:\Users\Y510P\Documents\Anaconda\lib\site-packages\pip\utils\__init__.p
y", line 102, in rmtree
onerror=rmtree_errorhandler)
File "C:\Users\Y510P\Documents\Anaconda\lib\shutil.py", line 494, in rmtree
return _rmtree_unsafe(path, onerror)
File "C:\Users\Y510P\Documents\Anaconda\lib\shutil.py", line 384, in _rmtree_u
nsafe
_rmtree_unsafe(fullname, onerror)
File "C:\Users\Y510P\Documents\Anaconda\lib\shutil.py", line 384, in _rmtree_u
nsafe
_rmtree_unsafe(fullname, onerror)
File "C:\Users\Y510P\Documents\Anaconda\lib\shutil.py", line 384, in _rmtree_u
nsafe
_rmtree_unsafe(fullname, onerror)
[Previous line repeated 5 more times]
File "C:\Users\Y510P\Documents\Anaconda\lib\shutil.py", line 389, in _rmtree_u
nsafe
onerror(os.unlink, fullname, sys.exc_info())
File "C:\Users\Y510P\Documents\Anaconda\lib\site-packages\pip\utils\__init__.p
y", line 114, in rmtree_errorhandler
func(path)
PermissionError: [WinError 5] Access is denied: 'C:\\Users\\Y510P\\AppData\\Loca
l\\Temp\\pip-0tanc911-uninstall\\users\\y510p\\documents\\anaconda\\lib\\site-pa
ckages\\zmq\\backend\\cython\\constants.cp36-win_amd64.pyd'
Your error message gives:
Copy/paste this URL into your browser when you connect for the first time, to login with a token:
http://localhost:8891/?token=dcf34cac4a27f8b92397fcec424b228b308a4588f0473fd6
So try that first in your browser and see if the error persist.
what is the web page you're going to . If you're using windows . Go to cmd prompt . type jupyter notebook . If you're in mac os x . go to terminal and type jupyter notebook. LMK if it doesnt work .Also this is the link once you trigger jupyter notebook. :
http://localhost:8888/tree
LMK Know
Just created a fresh install of Ubuntu. Installed python and selenium, went to run this code:
#!/usr/bin/env python
from selenium import webdriver
browser = webdriver.Firefox('/usr/bin/')
browser.get('http://www.ubuntu.com/')
browser.close()
browser.quit()
when i run it, it just hangs in the terminal and then when I press Control-C it says:
^CTraceback (most recent call last):
File "test.py", line 5, in <module>
browser = webdriver.Firefox('/usr/bin/')
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/firefox/webdriver.py", line 127, in __init__
firefox_profile = FirefoxProfile(firefox_profile)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/firefox/firefox_profile.py", line 78, in __init__
ignore=shutil.ignore_patterns("parent.lock", "lock", ".parentlock"))
File "/usr/lib/python2.7/shutil.py", line 189, in copytree
copytree(srcname, dstname, symlinks, ignore)
File "/usr/lib/python2.7/shutil.py", line 189, in copytree
copytree(srcname, dstname, symlinks, ignore)
File "/usr/lib/python2.7/shutil.py", line 189, in copytree
copytree(srcname, dstname, symlinks, ignore)
File "/usr/lib/python2.7/shutil.py", line 192, in copytree
copy2(srcname, dstname)
File "/usr/lib/python2.7/shutil.py", line 130, in copy2
copyfile(src, dst)
File "/usr/lib/python2.7/shutil.py", line 84, in copyfile
copyfileobj(fsrc, fdst)
KeyboardInterrupt
If I don't kill the program and it fills my tmp folder for some reason.
I have tried installing selenium three different times across different VMs, using venv, moving selenium all of the place. I am LOST.
Any help is greatly appreciated!!
EDIT: I removed the '/usr/bin/' per #L3viathan and ran it. My new error is:
Traceback (most recent call last):
File "test.py", line 5, in <module>
browser = webdriver.Firefox()
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/firefox/webdriver.py", line 135, in __init__
self.service.start()
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/common/service.py", line 64, in start
stdout=self.log_file, stderr=self.log_file)
File "/usr/lib/python2.7/subprocess.py", line 711, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1343, in _execute_child
raise child_exception
OSError: [Errno 8] Exec format error
Couldn't make anything work anymore so I started up another fresh Ubuntu VM and have only done the following commands.
$ sudo apt-get install python-pip
$ sudo pip install selenium
And then I ran my program and got the geckodriver error. So I downloaded the geckodriver. and moved it to /usr/bin/
$ sudo cp geckodriver /usr/bin/
ran program and get EXACT same error:
Traceback (most recent call last):
File "test.py", line 5, in <module>
browser = webdriver.Firefox()
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/firefox/webdriver.py", line 135, in __init__
self.service.start()
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/common/service.py", line 64, in start
stdout=self.log_file, stderr=self.log_file)
File "/usr/lib/python2.7/subprocess.py", line 711, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1343, in _execute_child
raise child_exception
OSError: [Errno 8] Exec format error
If i understand MRJob correctly, you can simulate hadoop's multi process run using MRJob by running it with
python mrfile.py -r local input.txt
I'm running windows(no choice for now), and when I issue the above command, i'm getting a bunch of mambo jumbo and at the end it tells me :
WindowsError: [Error 2] The system cannot find the file specified
This is the full error. Could someone help?
C:\Users\someuser\Documents\Python_projects\something>python MRJob_parser.py -r loc
al test2.txt
no configs found; falling back on auto-configuration
no configs found; falling back on auto-configuration
creating tmp directory c:\users\someuser\appdata\local\temp\MRJob_parser.someuser.
20150701.211822.496000
writing wrapper script to c:\users\someuser\appdata\local\temp\MRJob_parser.bw401
45.20150701.211822.496000\setup-wrapper.sh
writing to c:\users\someuser\appdata\local\temp\MRJob_parser.someuser.20150701.211
822.496000\step-0-mapper_part-00000
> sh -ex setup-wrapper.sh 'c:\Users\someuser\Documents\python_venv\something_Project\
Scripts\python.exe' MRJob_parser.py --step-num=0 --mapper 'c:\users\someuser\appd
ata\local\temp\MRJob_parser.someuser.20150701.211822.496000\input_part-00000' > c
:\users\someuser\appdata\local\temp\MRJob_parser.someuser.20150701.211822.496000\s
tep-0-mapper_part-00000
Traceback (most recent call last):
File "MRJob_parser.py", line 18, in <module>
Extractor.run()
File "c:\Users\someuser\Documents\python_venv\something_Project\lib\site-packages\m
rjob\job.py", line 461, in run
mr_job.execute()
File "c:\Users\someuser\Documents\python_venv\something_Project\lib\site-packages\m
rjob\job.py", line 479, in execute
super(MRJob, self).execute()
File "c:\Users\someuser\Documents\python_venv\something_Project\lib\site-packages\m
rjob\launch.py", line 151, in execute
self.run_job()
File "c:\Users\someuser\Documents\python_venv\something_Project\lib\site-packages\m
rjob\launch.py", line 214, in run_job
runner.run()
File "c:\Users\someuser\Documents\python_venv\something_Project\lib\site-packages\m
rjob\runner.py", line 464, in run
self._run()
File "c:\Users\someuser\Documents\python_venv\something_Project\lib\site-packages\m
rjob\sim.py", line 173, in _run
self._invoke_step(step_num, 'mapper')
File "c:\Users\someuser\Documents\python_venv\something_Project\lib\site-packages\m
rjob\sim.py", line 260, in _invoke_step
working_dir, env)
File "c:\Users\someuser\Documents\python_venv\something_Project\lib\site-packages\m
rjob\local.py", line 147, in _run_step
procs_args, output_path, working_dir, env)
File "c:\Users\someuser\Documents\python_venv\something_Project\lib\site-packages\m
rjob\local.py", line 250, in _invoke_processes
cwd=working_dir, env=env)
File "c:\Users\someuser\Documents\python_venv\something_Project\lib\site-packages\m
rjob\local.py", line 73, in _chain_procs
proc = Popen(args, **proc_kwargs)
File "C:\Python27\Lib\subprocess.py", line 710, in __init__
errread, errwrite)
File "C:\Python27\Lib\subprocess.py", line 958, in _execute_child
startupinfo)
WindowsError: [Error 2] The system cannot find the file specified
C:\Users\someuser\Documents\Python_projects\something>
From https://github.com/Yelp/mrjob:
# locally
python mrjob/examples/mr_word_freq_count.py README.rst > counts
My guess is that you don't have to pass in the local runner option and that's causing some kind of error.
I'm trying to install virtualenvwrapper on a fresh Ubuntu 14.04 installation. I followed the steps here and added these lines to my .bashrc:
export WORKON_HOME=$HOME/.virtualenvs
export PROJECT_HOME=$HOME/Devel
source /usr/local/bin/virtualenvwrapper.sh
I get the following error message when I try and source ~/.bashrc:
Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/usr/local/lib/python2.7/dist-packages/virtualenvwrapper/hook_loader.py", line 217, in <module>
main()
File "/usr/local/lib/python2.7/dist-packages/virtualenvwrapper/hook_loader.py", line 131, in main
run_hooks(hook, options, args)
File "/usr/local/lib/python2.7/dist-packages/virtualenvwrapper/hook_loader.py", line 157, in run_hooks
hook_mgr = ExtensionManager(namespace)
File "/usr/local/lib/python2.7/dist-packages/stevedore/extension.py", line 92, in __init__
verify_requirements)
File "/usr/local/lib/python2.7/dist-packages/stevedore/extension.py", line 155, in _load_plugins
for ep in self._find_entry_points(self.namespace):
File "/usr/local/lib/python2.7/dist-packages/stevedore/extension.py", line 148, in _find_entry_points
eps = list(pkg_resources.iter_entry_points(namespace))
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 515, in iter_entry_points
entries = dist.get_entry_map(group)
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2371, in get_entry_map
self._get_metadata('entry_points.txt'), self
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2155, in parse_map
for group, lines in data:
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2715, in split_sections
for line in yield_lines(s):
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 1989, in yield_lines
for ss in strs:
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2305, in _get_metadata
for line in self.get_metadata_lines(name):
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 1369, in get_metadata_lines
return yield_lines(self.get_metadata(name))
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 1361, in get_metadata
return self._get(self._fn(self.egg_info,name))
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 1470, in _get
stream = open(path, 'rb')
IOError: [Errno 13] Permission denied: '/usr/local/lib/python2.7/dist-packages/numpydoc-0.4-py2.7.egg/EGG-INFO/entry_points.txt'
virtualenvwrapper.sh: There was a problem running the initialization hooks.
If Python could not import the module virtualenvwrapper.hook_loader,
check that virtualenv has been installed for
VIRTUALENVWRAPPER_PYTHON=/usr/bin/python and that PATH is
set properly.
This happens when source /usr/local/bin/virtualenvwrapper.sh is executed.
Any ideas? Thanks.
EDIT: Although I am getting this error message, it seems my virtualenv is kind of working. I am able to create new env and even workon them. But every command I ttype, I get the IOError above.
Fixed. I just uninstalled the offending numpy doc package:
sudo pip uninstall numpydoc