I have referred to this tut to get started with pyspark on windows. These are the steps I followed:
Downloaded spark prebuilt for hadoop 2.7 from here
Extracted spark-2.1.0-bin-hadoop2.7.tgz to the directory set as %SPARK_HOME% in environment variables
Downloaded winutils.exe from here
Pasted winutils.exe in %SPARK_HOME%\bin
Set %HADOOP_HOME% to same directory as %SPARK_HOME%
Set %PYSPARK_DRIVER_PYTHON% to ipython
Set %PYSPARK_DRIVER_PYTHON_OPTS% to notebook
Added ;%SPARK_HOME%\bin to %PATH%
But when I run
> pyspark --master local[2]
I get following error:
[TerminalIPythonApp] WARNING | Subcommand `ipython notebook` is deprecated and will be removed in future versions.
[TerminalIPythonApp] WARNING | You likely want to use `jupyter notebook` in the future
Traceback (most recent call last):
File "d:\mahesh\softwares\python\winpython-64bit-3.4.4.4qt5\python-3.4.4.amd64\lib\runpy.py", line 170, in _run_module_as_main
"__main__", mod_spec)
File "d:\mahesh\softwares\python\winpython-64bit-3.4.4.4qt5\python-3.4.4.amd64\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "D:\mahesh\Softwares\python\WinPython-64bit-3.4.4.4Qt5\python-3.4.4.amd64\Scripts\ipython.exe\__main__.py", line 9, in <module>
File "d:\mahesh\softwares\python\winpython-64bit-3.4.4.4qt5\python-3.4.4.amd64\lib\site-packages\IPython\__init__.py", line 125, in start_ipython
return launch_new_instance(argv=argv, **kwargs)
File "d:\mahesh\softwares\python\winpython-64bit-3.4.4.4qt5\python-3.4.4.amd64\lib\site-packages\traitlets\config\application.py", line 657, in launch_instance
app.initialize(argv)
File "<decorator-gen-113>", line 2, in initialize
File "d:\mahesh\softwares\python\winpython-64bit-3.4.4.4qt5\python-3.4.4.amd64\lib\site-packages\traitlets\config\application.py", line 87, in catch_config_error
return method(app, *args, **kwargs)
File "d:\mahesh\softwares\python\winpython-64bit-3.4.4.4qt5\python-3.4.4.amd64\lib\site-packages\IPython\terminal\ipapp.py", line 308, in initialize
super(TerminalIPythonApp, self).initialize(argv)
File "<decorator-gen-7>", line 2, in initialize
File "d:\mahesh\softwares\python\winpython-64bit-3.4.4.4qt5\python-3.4.4.amd64\lib\site-packages\traitlets\config\application.py", line 87, in catch_config_error
return method(app, *args, **kwargs)
File "d:\mahesh\softwares\python\winpython-64bit-3.4.4.4qt5\python-3.4.4.amd64\lib\site-packages\IPython\core\application.py", line 450, in initialize
self.parse_command_line(argv)
File "d:\mahesh\softwares\python\winpython-64bit-3.4.4.4qt5\python-3.4.4.amd64\lib\site-packages\IPython\terminal\ipapp.py", line 303, in parse_command_line
return super(TerminalIPythonApp, self).parse_command_line(argv)
File "<decorator-gen-4>", line 2, in parse_command_line
File "d:\mahesh\softwares\python\winpython-64bit-3.4.4.4qt5\python-3.4.4.amd64\lib\site-packages\traitlets\config\application.py", line 87, in catch_config_error
return method(app, *args, **kwargs)
File "d:\mahesh\softwares\python\winpython-64bit-3.4.4.4qt5\python-3.4.4.amd64\lib\site-packages\traitlets\config\application.py", line 514, in parse_command_line
return self.initialize_subcommand(subc, subargv)
File "d:\mahesh\softwares\python\winpython-64bit-3.4.4.4qt5\python-3.4.4.amd64\lib\site-packages\IPython\core\application.py", line 243, in initialize_subcommand
return super(BaseIPythonApplication, self).initialize_subcommand(subc, argv)
File "<decorator-gen-3>", line 2, in initialize_subcommand
File "d:\mahesh\softwares\python\winpython-64bit-3.4.4.4qt5\python-3.4.4.amd64\lib\site-packages\traitlets\config\application.py", line 87, in catch_config_error
return method(app, *args, **kwargs)
File "d:\mahesh\softwares\python\winpython-64bit-3.4.4.4qt5\python-3.4.4.amd64\lib\site-packages\traitlets\config\application.py", line 445, in initialize_subcommand
subapp = import_item(subapp)
File "d:\mahesh\softwares\python\winpython-64bit-3.4.4.4qt5\python-3.4.4.amd64\lib\site-packages\ipython_genutils\importstring.py", line 31, in import_item
module = __import__(package, fromlist=[obj])
File "d:\mahesh\softwares\python\winpython-64bit-3.4.4.4qt5\python-3.4.4.amd64\lib\site-packages\notebook\notebookapp.py", line 31, in <module>
from zmq.eventloop import ioloop
File "d:\mahesh\softwares\python\winpython-64bit-3.4.4.4qt5\python-3.4.4.amd64\lib\site-packages\zmq\eventloop\__init__.py", line 3, in <module>
from zmq.eventloop.ioloop import IOLoop
File "d:\mahesh\softwares\python\winpython-64bit-3.4.4.4qt5\python-3.4.4.amd64\lib\site-packages\zmq\eventloop\ioloop.py", line 21, in <module>
from zmq import (
ImportError: cannot import name 'Poller'
I am correctly able to run spark scala shell with >spark-shell command.
As you can see in the stack trace, I had win-python installed already at path
D:\mahesh\Softwares\python\WinPython-64bit-3.4.4.4Qt5\python-3.4.4.amd64
Thus, my %PYTHON_HOME% is D:\mahesh\Softwares\python\WinPython-64bit-3.4.4.4Qt5.
But my %SPARK_HOME% is D:\mahesh\Programs\spark-2.3.0-bin-hadoop2.7.
Running where pyspark command gives following output:
D:\mahesh\Programs\spark-2.3.0-bin-hadoop2.7\bin\pyspark
D:\mahesh\Programs\spark-2.3.0-bin-hadoop2.7\bin\pyspark.cmd
D:\mahesh\Softwares\python\WinPython-64bit-3.4.4.4Qt5\python-3.4.4.amd64\Scripts\pyspark
D:\mahesh\Softwares\python\WinPython-64bit-3.4.4.4Qt5\python-3.4.4.amd64\Scripts\pyspark.cmd
I believe my issue is some missconfiguration of my windows spark environment. Thats why I gave all above information. So whats going wrong here?
Note that I performed the steps without using Anaconda and GOW (Gnu on windows) as suggested in the tut.
Point your %PYSPARK_DRIVER_PYTHON% to a virtual environment with all dependencies including 'Poller' and then check.
Else you can try installing 'Poller' in ipython environment( which I frankly dont know how!)
Related
I am trying to run pyspark in the linux terminal. I have most of the installation done but I am getting the following error when I call pyspark in the linux terminal:
[TerminalIPythonApp] WARNING | Subcommand `ipython notebook` is deprecated and will be removed in future versions.
[TerminalIPythonApp] WARNING | You likely want to use `jupyter notebook` in the future
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/usr/lib/python3/dist-packages/traitlets/config/application.py", line 657, in launch_instance
app.initialize(argv)
File "<decorator-gen-110>", line 2, in initialize
File "/usr/lib/python3/dist-packages/traitlets/config/application.py", line 87, in catch_config_error
return method(app, *args, **kwargs)
File "/usr/lib/python3/dist-packages/IPython/terminal/ipapp.py", line 307, in initialize
super(TerminalIPythonApp, self).initialize(argv)
File "<decorator-gen-7>", line 2, in initialize
File "/usr/lib/python3/dist-packages/traitlets/config/application.py", line 87, in catch_config_error
return method(app, *args, **kwargs)
File "/usr/lib/python3/dist-packages/IPython/core/application.py", line 455, in initialize
self.parse_command_line(argv)
File "/usr/lib/python3/dist-packages/IPython/terminal/ipapp.py", line 302, in parse_command_line
return super(TerminalIPythonApp, self).parse_command_line(argv)
File "<decorator-gen-4>", line 2, in parse_command_line
File "/usr/lib/python3/dist-packages/traitlets/config/application.py", line 87, in catch_config_error
return method(app, *args, **kwargs)
File "/usr/lib/python3/dist-packages/traitlets/config/application.py", line 514, in parse_command_line
return self.initialize_subcommand(subc, subargv)
File "/usr/lib/python3/dist-packages/IPython/core/application.py", line 244, in initialize_subcommand
return super(BaseIPythonApplication, self).initialize_subcommand(subc, argv)
File "<decorator-gen-3>", line 2, in initialize_subcommand
File "/usr/lib/python3/dist-packages/traitlets/config/application.py", line 87, in catch_config_error
return method(app, *args, **kwargs)
File "/usr/lib/python3/dist-packages/traitlets/config/application.py", line 445, in initialize_subcommand
subapp = import_item(subapp)
File "/usr/lib/python3/dist-packages/ipython_genutils/importstring.py", line 31, in import_item
module = __import__(package, fromlist=[obj])
ModuleNotFoundError: No module named 'notebook'
Most other forums say to install jupyter as follows:
pip install jupyter
but I already have this done.
Try this:
PYSPARK_DRIVER_PYTHON=ipython3 pyspark
I'm currently trying to change the looks of the nbconvert-output in jupyter. To be more precise, I want to convert a ipynb to latex but not using the default tamplate. The thing is I don't know how to edit the files to get the result I want.
Lets say I want to use the "style_notebook.tplx" so I need to donwload ipython_nbconvert_config.py and use_cell_style.tplx. After that I need to change in the use_cell_style file the second line to ((* set cell_style = 'style_notebook.tplx' *)). But to be honest I don't know with what to edit a .tplx File (I used just Editor from Windows)....
After changing that and moving the two files to the same directonary as my .ipynb is, I tried this in the command prompt:
jupyter nbconvert --config ipython_nbconvert_config --to latex filename.ipynb
and got the error:
[NbConvertApp] WARNING | Error loading config file: ipython_nbconvert_config
Traceback (most recent call last):
File "c:\users\Sito\appdata\local\programs\python\python36-32\lib\site-packages\jupyter_core\application.py", line 202, in load_config_file
path=path
File "<decorator-gen-5>", line 2, in load_config_file
File "c:\users\Sito\appdata\local\programs\python\python36-32\lib\site-packages\traitlets\config\application.py", line 87, in catch_config_error
return method(app, *args, **kwargs)
File "c:\users\Sito\appdata\local\programs\python\python36-32\lib\site-packages\traitlets\config\application.py", line 603, in load_config_file
self.update_config(new_config)
File "c:\users\Sito\appdata\local\programs\python\python36-32\lib\site-packages\traitlets\config\configurable.py", line 198, in update_config
self._load_config(config)
File "c:\users\Sito\appdata\local\programs\python\python36-32\lib\site-packages\traitlets\config\configurable.py", line 168, in _load_config
warn(msg)
File "c:\users\Sito\appdata\local\programs\python\python36-32\lib\contextlib.py", line 89, in __exit__
next(self.gen)
File "c:\users\Sito\appdata\local\programs\python\python36-32\lib\site-packages\traitlets\traitlets.py", line 1131, in hold_trait_notifications
self.notify_change(change)
File "c:\users\Sito\appdata\local\programs\python\python36-32\lib\site-packages\traitlets\traitlets.py", line 1176, in notify_change
c(change)
File "c:\users\Sito\appdata\local\programs\python\python36-32\lib\site-packages\nbconvert\nbconvertapp.py", line 227, in _postprocessor_class_changed
self.postprocessor_factory = import_item(new)
File "c:\users\Sito\appdata\local\programs\python\python36-32\lib\site-packages\traitlets\utils\importstring.py", line 42, in import_item
return __import__(parts[0])
ModuleNotFoundError: No module named 'PDF'
Traceback (most recent call last):
File "c:\users\Sito\appdata\local\programs\python\python36-32\lib\runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "c:\users\Sito\appdata\local\programs\python\python36-32\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "C:\Users\Sito\AppData\Local\Programs\Python\Python36-32\Scripts\jupyter-nbconvert.EXE\__main__.py", line 9, in <module>
File "c:\users\Sito\appdata\local\programs\python\python36-32\lib\site-packages\jupyter_core\application.py", line 267, in launch_instance
return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
File "c:\users\Sito\appdata\local\programs\python\python36-32\lib\site-packages\traitlets\config\application.py", line 657, in launch_instance
app.initialize(argv)
File "<decorator-gen-7>", line 2, in initialize
File "c:\users\Sito\appdata\local\programs\python\python36-32\lib\site-packages\traitlets\config\application.py", line 87, in catch_config_error
return method(app, *args, **kwargs)
File "c:\users\Sito\appdata\local\programs\python\python36-32\lib\site-packages\nbconvert\nbconvertapp.py", line 252, in initialize
self.init_postprocessor()
File "c:\users\Sito\appdata\local\programs\python\python36-32\lib\site-packages\nbconvert\nbconvertapp.py", line 298, in init_postprocessor
self._postprocessor_class_changed({'new': self.postprocessor_class})
File "c:\users\Sito\appdata\local\programs\python\python36-32\lib\site-packages\nbconvert\nbconvertapp.py", line 227, in _postprocessor_class_changed
self.postprocessor_factory = import_item(new)
File "c:\users\Sito\appdata\local\programs\python\python36-32\lib\site-packages\traitlets\utils\importstring.py", line 42, in import_item
return __import__(parts[0])
ModuleNotFoundError: No module named 'PDF'
So it says that I miss the PDF-module, but I'm not sure hows that possible, because until now I could convert to pdf just fine. So I aussume I made some mistakes in changing the files...
Would be nice if someone could explain how to fix this problem..
Thanks, Sito.
I am trying to run iPython on my Windows 8. I installed python 3.4.0 (32 bit version) and successfully installed iPython through the command
easy_install ipython[all]
The problem is when I try to run it
ipython notebook
I get the following error:
Traceback (most recent call last):
File "C:\Python34\Scripts\ipython-script.py", line 9, in <module>
load_entry_point('ipython==3.1.0', 'console_scripts', 'ipython')()
File "C:\Python34\lib\site-packages\IPython\__init__.py", line 120, in start_ipython return launch_new_instance(argv=argv, **kwargs)
File "C:\Python34\lib\site-packages\IPython\config\application.py", line 573, in launch_instance app.initialize(argv)
File "<string>", line 2, in initialize
File "C:\Python34\lib\site-packages\IPython\config\application.py", line 75, in catch_config_error return method(app, *args, **kwargs)
File "C:\Python34\lib\site-packages\IPython\terminal\ipapp.py", line 321, in initialize super(TerminalIPythonApp, self).initialize(argv)
File "<string>", line 2, in initialize
File "C:\Python34\lib\site-packages\IPython\config\application.py", line 75, in catch_config_error return method(app, *args, **kwargs)
File "C:\Python34\lib\site-packages\IPython\core\application.py", line 369, in initialize self.parse_command_line(argv)
File "C:\Python34\lib\site-packages\IPython\terminal\ipapp.py", line 316, in parse_command_line return super(TerminalIPythonApp, self).parse_command_line(argv)
File "<string>", line 2, in parse_command_line
File "C:\Python34\lib\site-packages\IPython\config\application.py", line 75, in catch_config_error return method(app, *args, **kwargs)
File "C:\Python34\lib\site-packages\IPython\config\application.py", line 471, in parse_command_line return self.initialize_subcommand(subc, subargv)
File "<string>", line 2, in initialize_subcommand
File "C:\Python34\lib\site-packages\IPython\config\application.py", line 75, in catch_config_error return method(app, *args, **kwargs)
File "C:\Python34\lib\site-packages\IPython\config\application.py", line 402, in initialize_subcommand subapp = import_item(subapp)
File "C:\Python34\lib\site-packages\IPython\utils\importstring.py", line 42, in import_item module = __import__(package, fromlist=[obj])
File "C:\Python34\lib\site-packages\IPython\html\notebookapp.py", line 30, in <module> check_for_zmq('13', 'IPython.html')
File "C:\Python34\lib\site-packages\IPython\utils\zmqrelated.py", line 10, in check_for_zmq import zmq
File "C:\Python34\lib\site-packages\zmq\__init__.py", line 25, in <module> _libsodium = ctypes.cdll.LoadLibrary(bundled_sodium[0])
File "C:\Python34\lib\ctypes\__init__.py", line 429, in LoadLibrary return self._dlltype(name)
File "C:\Python34\lib\ctypes\__init__.py", line 351, in __init__ self._handle = _dlopen(self._name, mode)
OSError: [WinError 193] %1 is not a valid Win32 application
Probably the 'name' of some application is wrongly specified. Can you help me to understand how to fix it?
Thank you!
Easy_Install is the devil (IMHO). You might want to use Continuum Anaconda. The Continuum solution is one that simply "just works" and helps to avoid the many hassles you will most likely have later with Easy_Install.
I have installed python 3.3.0 and all the package dependencies from here for ipython and some other basic python libraries.
http://www.lfd.uci.edu/~gohlke/pythonlibs/#ipython
However whenever I navigate the command prompt in windows 7 to the "C:\Python33\Scripts" directory and run the command:
ipython notebook --pylab inline
as suggested by this website (Open Tech School) It's not bringing up the ipython notebook in my web browser as it has normally done?:
http://opentechschool.github.io/python-data-intro/core/notebook.html
I can't think of what I might have done wrong: I installed python 3.3.0 and all the 3.3 binary installers from :
http://www.lfd.uci.edu/~gohlke/pythonlibs/
My path variable (right click on my computer select properties > Advanced system settings (left hand side) > Then the "Advanced" tab > Environment Variables:
My Path variable is in the lower "system variables" section? Does it need to be in the upper "User Variables" section as well? or instead of?
The contents of my Path "system variable" is:
%SystemRoot%\system32;%SystemRoot%;%SystemRoot%\System32\Wbem;%SYSTEMROOT%\System32\WindowsPowerShell\v1.0\;C:\Python33\Scripts;C:\Python33\
Please help me figure out why my ipython notebook installation is not opening in my default firefox web browser?
I get the error:
C:\Python33\Scripts>ipython notebook --pylab inline
'ipython' is not recognized as an internal or external command,
operable program or batch file.
Got further with trying ipython3 as suggested, but got the following error:
C:\Python33\Scripts>ipython3 notebook --pylab inline
Traceback (most recent call last):
File "C:\Python33\Scripts\ipython3-script.py", line 9, in <module>
load_entry_point('ipython==1.2.1', 'console_scripts', 'ipython3')()
File "C:\Python33\lib\site-packages\IPython\__init__.py", line 118, in start_i
python
return launch_new_instance(argv=argv, **kwargs)
File "C:\Python33\lib\site-packages\IPython\config\application.py", line 544,
in launch_instance
app.initialize(argv)
File "<string>", line 2, in initialize
File "C:\Python33\lib\site-packages\IPython\config\application.py", line 89, i
n catch_config_error
return method(app, *args, **kwargs)
File "C:\Python33\lib\site-packages\IPython\terminal\ipapp.py", line 312, in i
nitialize
super(TerminalIPythonApp, self).initialize(argv)
File "<string>", line 2, in initialize
File "C:\Python33\lib\site-packages\IPython\config\application.py", line 89, i
n catch_config_error
return method(app, *args, **kwargs)
File "C:\Python33\lib\site-packages\IPython\core\application.py", line 373, in
initialize
self.parse_command_line(argv)
File "C:\Python33\lib\site-packages\IPython\terminal\ipapp.py", line 307, in p
arse_command_line
return super(TerminalIPythonApp, self).parse_command_line(argv)
File "<string>", line 2, in parse_command_line
File "C:\Python33\lib\site-packages\IPython\config\application.py", line 89, i
n catch_config_error
return method(app, *args, **kwargs)
File "C:\Python33\lib\site-packages\IPython\config\application.py", line 474,
in parse_command_line
return self.initialize_subcommand(subc, subargv)
File "<string>", line 2, in initialize_subcommand
File "C:\Python33\lib\site-packages\IPython\config\application.py", line 89, i
n catch_config_error
return method(app, *args, **kwargs)
File "C:\Python33\lib\site-packages\IPython\config\application.py", line 405,
in initialize_subcommand
subapp = import_item(subapp)
File "C:\Python33\lib\site-packages\IPython\utils\importstring.py", line 42, i
n import_item
module = __import__(package, fromlist=[obj])
File "C:\Python33\lib\site-packages\IPython\html\notebookapp.py", line 38, in
<module>
from jinja2 import Environment, FileSystemLoader
File "C:\Python33\lib\site-packages\jinja2\__init__.py", line 33, in <module>
from jinja2.environment import Environment, Template
File "C:\Python33\lib\site-packages\jinja2\environment.py", line 13, in <modul
e>
from jinja2 import nodes
ImportError: cannot import name nodes
C:\Python33\Scripts>
This question already has answers here:
ImportError: No module named jinja2
(8 answers)
Closed 9 years ago.
I installed ipython notebook using their instructions. That didn't work. So i tried other instructions. I seemed to get closer to getting it working. currently this is the output i get when i try to run ipnotebook.
Traceback (most recent call last):
File "/usr/local/bin/ipython", line 9, in <module>
load_entry_point('ipython==1.1.0', 'console_scripts', 'ipython')()
File "/usr/local/lib/python2.7/dist-packages/ipython-1.1.0-py2.7.egg/IPython/__init__.py", line 118, in start_ipython
return launch_new_instance(argv=argv, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/ipython-1.1.0-py2.7.egg/IPython/config/application.py", line 544, in launch_instance
app.initialize(argv)
File "<string>", line 2, in initialize
File "/usr/local/lib/python2.7/dist-packages/ipython-1.1.0-py2.7.egg/IPython/config/application.py", line 89, in catch_config_error
return method(app, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/ipython-1.1.0-py2.7.egg/IPython/terminal/ipapp.py", line 312, in initialize
super(TerminalIPythonApp, self).initialize(argv)
File "<string>", line 2, in initialize
File "/usr/local/lib/python2.7/dist-packages/ipython-1.1.0-py2.7.egg/IPython/config/application.py", line 89, in catch_config_error
return method(app, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/ipython-1.1.0-py2.7.egg/IPython/core/application.py", line 362, in initialize
self.parse_command_line(argv)
File "/usr/local/lib/python2.7/dist-packages/ipython-1.1.0-py2.7.egg/IPython/terminal/ipapp.py", line 307, in parse_command_line
return super(TerminalIPythonApp, self).parse_command_line(argv)
File "<string>", line 2, in parse_command_line
File "/usr/local/lib/python2.7/dist-packages/ipython-1.1.0-py2.7.egg/IPython/config/application.py", line 89, in catch_config_error
return method(app, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/ipython-1.1.0-py2.7.egg/IPython/config/application.py", line 474, in parse_command_line
return self.initialize_subcommand(subc, subargv)
File "<string>", line 2, in initialize_subcommand
File "/usr/local/lib/python2.7/dist-packages/ipython-1.1.0-py2.7.egg/IPython/config/application.py", line 89, in catch_config_error
return method(app, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/ipython-1.1.0-py2.7.egg/IPython/config/application.py", line 405, in initialize_subcommand
subapp = import_item(subapp)
File "/usr/local/lib/python2.7/dist-packages/ipython-1.1.0-py2.7.egg/IPython/utils/importstring.py", line 42, in import_item
module = __import__(package, fromlist=[obj])
File "/usr/local/lib/python2.7/dist-packages/ipython-1.1.0-py2.7.egg/IPython/html/notebookapp.py", line 38, in <module>
from jinja2 import Environment, FileSystemLoader
ImportError: No module named jinja2
In order to load Jinja locally you also have to have it installed locally
easy_install Jinja2
or
pip install Jinja2
or
sudo apt-get install python-jinja2