Cannot run pyspark in MAC after SPARK installation - python

I recently installed SPARK in my MAC computer using the following command:
brew install apache-spark
Now I tried to run
pyspark but it is showing me the following error.
pyspark
Python 3.6.0 |Anaconda custom (x86_64)| (default, Dec 23 2016, 13:19:00
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
Traceback (most recent call last)
File "/usr/local/Cellar/apache-spark/2.1.0/libexec/python/pyspark/shell.py", line 30, in <module>
import pyspark
File "/usr/local/Cellar/apache-spark/2.1.0/libexec/python/pyspark/__init__.py", line 44, in <module>
from pyspark.context import SparkContext
File "/usr/local/Cellar/apache-spark/2.1.0/libexec/python/pyspark/context.py", line 36, in <module>
from pyspark.java_gateway import launch_gateway
File "/usr/local/Cellar/apache-spark/2.1.0/libexec/python/pyspark/java_gateway.py", line 31, in <module>
from py4j.java_gateway import java_import, JavaGateway, GatewayClient
File "<frozen importlib._bootstrap>", line 961, in _find_and_load
File "<frozen importlib._bootstrap>", line 950, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 646, in _load_unlocked
File "<frozen importlib._bootstrap>", line 616, in _load_backward_compatible
File "/usr/local/Cellar/apache-spark/2.1.0/libexec/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py", line 18, in <module>
File "/Users/hellmaster/anaconda/lib/python3.6/pydoc.py", line 62, in <module>
import pkgutil
File "/Users/hellmaster/anaconda/lib/python3.6/pkgutil.py", line 22, in <module>
ModuleInfo = namedtuple('ModuleInfo', 'module_finder name ispkg')
File "/usr/local/Cellar/apache-spark/2.1.0/libexec/python/pyspark/serializers.py", line 393, in namedtuple
cls = _old_namedtuple(*args, **kwargs)
TypeError: namedtuple() missing 3 required keyword-only arguments: 'verbose', 'rename', and 'module'
How do I fix this issue?

This is because Spark 2.1.0 is incompatible with Python 3.6.
See also this question.

Related

Installing python packages offline return pip._internal.operations.build.metadata error

I am trying to install packages on a cluster with restricted policy to access the net, so I download all the package file
pip3 download -r requirements.txt
and then to install:
pip3 install --no-index --find-links /path/to/download/dir/ -r requirements.txt
This works on my PC but I get the following error on the cluster:
Traceback (most recent call last):
File "/ccc/products/python3-3.8.10/system/default/bin/pip3", line 10, in <module>
sys.exit(main())
File "/ccc/products/python3-3.8.10/system/default/site-packages/pip/_internal/cli/main.py", line 69, in main
command = create_command(cmd_name, isolated=("--isolated" in cmd_args))
File "/ccc/products/python3-3.8.10/system/default/site-packages/pip/_internal/commands/__init__.py", line 91, in create_command
module = importlib.import_module(module_path)
File "/ccc/products2/python3-3.8.10/Atos_7__x86_64/system/default/install_tree/python/3.8.10/tohnnxbges5ktga7fvchschkkdpgv2yj/lib/python3.8/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 848, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/ccc/products/python3-3.8.10/system/default/site-packages/pip/_internal/commands/install.py", line 15, in <module>
from pip._internal.cli.req_command import (
File "/ccc/products/python3-3.8.10/system/default/site-packages/pip/_internal/cli/req_command.py", line 21, in <module>
from pip._internal.index.package_finder import PackageFinder
File "/ccc/products/python3-3.8.10/system/default/site-packages/pip/_internal/index/package_finder.py", line 32, in <module>
from pip._internal.req import InstallRequirement
File "/ccc/products/python3-3.8.10/system/default/site-packages/pip/_internal/req/__init__.py", line 8, in <module>
from .req_install import InstallRequirement
File "/ccc/products/python3-3.8.10/system/default/site-packages/pip/_internal/req/req_install.py", line 26, in <module>
from pip._internal.operations.build.metadata import generate_metadata
ModuleNotFoundError: No module named 'pip._internal.operations.build.metadata'
Do you have any idea?
some system information:
Python 3.8.10 (default, Mar 9 2022, 16:12:02)
[GCC 8.4.0] on linux
NAME="Red Hat Enterprise Linux Server"
VERSION="7.9 (Maipo)"
ID="rhel"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="7.9"
PRETTY_NAME="Red Hat Enterprise Linux Server 7.9 (Maipo)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:7.9:GA:server"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7"
REDHAT_BUGZILLA_PRODUCT_VERSION=7.9
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="7.9"

Anaconda - Error when importing numpy: bad marshal data

I use Anaconda 3 and I have pandas and numpy packages installed. It was working for the last few days and suddenly it stops working with the following error. I made no changes to the installation or packages. I am on a company PC so I am thinking they might have some software that is interfering with it - encryption or security software. Any insight on what this error means or what may be causing it?
Something similar happened in the past where python would not even initialize. I had to reinstall Anaconda to fix that.
Python 3.10.0 | packaged by conda-forge | (default, Nov 10 2021, 13:20:59) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Users\thakuran\Anaconda3\envs\SAR_Data_Processing\lib\site-packages\numpy\__init__.py", line 150, in <module>
from . import core
File "C:\Users\thakuran\Anaconda3\envs\SAR_Data_Processing\lib\site-packages\numpy\core\__init__.py", line 129, in <module>
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 879, in exec_module
File "<frozen importlib._bootstrap_external>", line 1012, in get_code
File "<frozen importlib._bootstrap_external>", line 672, in _compile_bytecode
ValueError: bad marshal data (bytes object size out of range)
>>>

How can i install gdal for python in windows 10?

I'm trying to install gdal library. i've tried the things recomended here Installing gdal python library on windows using conda (DLL load failed)
and here: https://github.com/felipunky/GISPython/blob/master/README.md
then i tried:
C:\Users\User>pip install D:\Seba\Programas\GDAL-3.3.0-cp37-cp37m-win_amd64.whl
Requirement already satisfied: GDAL==3.3.0 from file:///D:/Seba/Programas/GDAL-3.3.0-cp37-cp37m-win_amd64.whl in c:\users\user\appdata\local\programs\python\python37\lib\site-packages (3.3.0)
WARNING: You are using pip version 20.1.1; however, version 21.1.2 is available.
You should consider upgrading via the 'c:\users\user\appdata\local\programs\python\python37\python.exe -m pip install --upgrade pip' command.
C:\Users\User>python
Python 3.7.9 (tags/v3.7.9:13c94747c7, Aug 17 2020, 18:58:18) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import gdal
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'gdal'
>>> from osgeo import gdal
Traceback (most recent call last):
File "C:\Users\User\AppData\Local\Programs\Python\Python37\lib\site-packages\osgeo\__init__.py", line 18, in swig_import_helper
return importlib.import_module(mname)
File "C:\Users\User\AppData\Local\Programs\Python\Python37\lib\importlib\__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
File "<frozen importlib._bootstrap>", line 983, in _find_and_load
File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 670, in _load_unlocked
File "<frozen importlib._bootstrap>", line 583, in module_from_spec
File "<frozen importlib._bootstrap_external>", line 1043, in create_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
ImportError: DLL load failed: No se puede encontrar el módulo especificado.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Users\User\AppData\Local\Programs\Python\Python37\lib\site-packages\osgeo\__init__.py", line 34, in <module>
_gdal = swig_import_helper()
File "C:\Users\User\AppData\Local\Programs\Python\Python37\lib\site-packages\osgeo\__init__.py", line 31, in swig_import_helper
return importlib.import_module('_gdal')
File "C:\Users\User\AppData\Local\Programs\Python\Python37\lib\importlib\__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
ModuleNotFoundError: No module named '_gdal'
>>>
any recomendation?
Thanks
conda install of gdal in windows does work.
You haven't given much to go on, but my guess is that you have not activated the conda environment into which gdal has been installed.
Normally your command prompt would look like this, if you had activated it
(gdal_env_name) C:\Users\User>python
Also, run
conda list
to check that all the expected libraries are there.
Try this once: sudo easy_install GDAL

Can't import tensorflow, OSError: [WinError 193] %1 is not a valid Win32 application

I'm trying to install TensorFlow for python (3.7) using anaconda.
I've installed the Cuda 10.2 and the matching cndd and installed it in a virtual environment:
pip install tensorflow==2.1.0
The command ran without any errors, but when i try to import it in python I get:
Python 3.7.6 (default, Jan 8 2020, 20:23:39) [MSC v.1916 64 bit (AMD64)] :: Anaconda, Inc. on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Users\abual\Anaconda3\envs\ml1\lib\site-packages\tensorflow\__init__.py", line 101, in <module>
from tensorflow_core import *
File "C:\Users\abual\Anaconda3\envs\ml1\lib\site-packages\tensorflow_core\__init__.py", line 40, in <module>
from tensorflow.python.tools import module_util as _module_util
File "<frozen importlib._bootstrap>", line 983, in _find_and_load
File "<frozen importlib._bootstrap>", line 959, in _find_and_load_unlocked
File "C:\Users\abual\Anaconda3\envs\ml1\lib\site-packages\tensorflow\__init__.py", line 50, in __getattr__
module = self._load()
File "C:\Users\abual\Anaconda3\envs\ml1\lib\site-packages\tensorflow\__init__.py", line 44, in _load
module = _importlib.import_module(self.__name__)
File "C:\Users\abual\Anaconda3\envs\ml1\lib\importlib\__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "C:\Users\abual\Anaconda3\envs\ml1\lib\site-packages\tensorflow_core\python\__init__.py", line 47, in <module>
import numpy as np
File "C:\Users\abual\AppData\Roaming\Python\Python37\site-packages\numpy\__init__.py", line 140, in <module>
from . import _distributor_init
File "C:\Users\abual\AppData\Roaming\Python\Python37\site-packages\numpy\_distributor_init.py", line 26, in <module>
WinDLL(os.path.abspath(filename))
File "C:\Users\abual\Anaconda3\envs\ml1\lib\ctypes\__init__.py", line 364, in __init__
self._handle = _dlopen(self._name, mode)
OSError: [WinError 193] %1 is not a valid Win32 application
>>>
Can anyone help me solve it?
Thanks
On your 1st question
The problem might be installing 32 bit on a 64-bit machine., try installing a 64 bit TensorFlow version 2.x.
Kindly refer to the documentation here
On your 2nd question
The tf.Session() is on the version 1.x, you need to migrate it to 2.1.
You can read about migration here
Kindly refer to the sample code below
# Using the `close()` method.
sess = tf.compat.v1.Session()
sess.run(...)
sess.close()

Cannot import tensorflow-gpu in Python

I've installed tensorflow-gpu using conda, before that I've installed CUDA Toolkit 9.0 and CuDNN v7.0 . But after trying to run the hello tensorflow code, I'm facing the following error. It is unable to load lib files.
Following is what I have given as I/p:
Microsoft Windows [Version 10.0.15063]
(c) 2017 Microsoft Corporation. All rights reserved.
C:\Users\Manish>activate tensorflow-gpu
(tensorflow-gpu) C:\Users\Manish>python
Python 3.6.1 |Continuum Analytics, Inc.| (default, May 11 2017, 13:25:24) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
This is the Error after that
Traceback (most recent call last):
File "C:\Users\Manish\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 18, in swig_import_helper
return importlib.import_module(mname)
File "C:\Users\Manish\Anaconda3\envs\tensorflow-gpu\lib\importlib\__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 978, in _gcd_import
File "<frozen importlib._bootstrap>", line 961, in _find_and_load
File "<frozen importlib._bootstrap>", line 950, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 648, in _load_unlocked
File "<frozen importlib._bootstrap>", line 560, in module_from_spec
File "<frozen importlib._bootstrap_external>", line 922, in create_module
File "<frozen importlib._bootstrap>", line 205, in _call_with_frames_removed
ImportError: DLL load failed: The specified module could not be found.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\Manish\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\pywrap_tensorflow.py", line 41, in <module>
from tensorflow.python.pywrap_tensorflow_internal import *
File "C:\Users\Manish\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 21, in <module>
_pywrap_tensorflow_internal = swig_import_helper()
File "C:\Users\Manish\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 20, in swig_import_helper
return importlib.import_module('_pywrap_tensorflow_internal')
File "C:\Users\Manish\Anaconda3\envs\tensorflow-gpu\lib\importlib\__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
ModuleNotFoundError: No module named '_pywrap_tensorflow_internal'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Users\Manish\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\__init__.py", line 24, in <module>
from tensorflow.python import *
File "C:\Users\Manish\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\__init__.py", line 49, in <module>
from tensorflow.python import pywrap_tensorflow
File "C:\Users\Manish\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\pywrap_tensorflow.py", line 52, in <module>
raise ImportError(msg)
ImportError: Traceback (most recent call last):
File "C:\Users\Manish\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 18, in swig_import_helper
return importlib.import_module(mname)
File "C:\Users\Manish\Anaconda3\envs\tensorflow-gpu\lib\importlib\__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 978, in _gcd_import
File "<frozen importlib._bootstrap>", line 961, in _find_and_load
File "<frozen importlib._bootstrap>", line 950, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 648, in _load_unlocked
File "<frozen importlib._bootstrap>", line 560, in module_from_spec
File "<frozen importlib._bootstrap_external>", line 922, in create_module
File "<frozen importlib._bootstrap>", line 205, in _call_with_frames_removed
ImportError: DLL load failed: The specified module could not be found.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\Manish\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\pywrap_tensorflow.py", line 41, in <module>
from tensorflow.python.pywrap_tensorflow_internal import *
File "C:\Users\Manish\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 21, in <module>
_pywrap_tensorflow_internal = swig_import_helper()
File "C:\Users\Manish\Anaconda3\envs\tensorflow-gpu\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 20, in swig_import_helper
return importlib.import_module('_pywrap_tensorflow_internal')
File "C:\Users\Manish\Anaconda3\envs\tensorflow-gpu\lib\importlib\__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
ModuleNotFoundError: No module named '_pywrap_tensorflow_internal'
Failed to load the native TensorFlow runtime.
See: https://www.tensorflow.org/install/install_sources#common_installation_problems
For some common reasons and solutions. Include the entire stack trace above this error message when asking for help.
Follow these steps to install Tensorflow using conda
#Set Up Anaconda Environments
conda create --name tf_gpu python=3
#Activate the new Environment
source activate tf_gpu
# Install CUDA and cuDNN
tf_gpu$conda install -c anaconda cudatoolkit=9.0 cudnn=7.0
#Install Tensorflow
tf_gpu$ conda install -c anaconda tensorflow-gpu

Categories