I am trying to query snappydata from Python and some of the answers say in StackOverflow that Python cant connect to remote spark clusters. Could anyone help me how can I connect to snappydata cluster and get a simple query working?
Code I am trying -
from pyspark.sql.snappy import SnappySession
snappy = SnappySession.builder.appName("test") \
.master("local[*]") \
.config("spark.snappydata.connection", "<remote server>:1527") \
.getOrCreate()
I am getting FileNotFoundError: [WinError 2] The system cannot find the file specified. In running above code. Unfortunately, there is not much information in setting up the environment. However, I have configured my environment to run PySpark locally and it works.
SnappyData's Python API is not distributed as a Python module that you can use from any spark cluster. However, you can use PySpark that is bundled as part of the SnappyData distribution.
Related
I have installed Hive on a Debian system using the steps provided in the link:
https://phoenixnap.com/kb/install-hive-on-ubuntu
I have followed all the steps and able to create a database and table in Hive. However the property hive.metastore.uris is not set in hive-site.xml. When I try to connect to Hive using pyhive module in python I get this error:
thrift.transport.TTransport.TTransportException: Could not connect to any of [('172.16.0.125', 10000)]
I am trying to connect to oracle DB to execute some SQL queries and fetch data through a python script . I have imported cx_Oracle and tried connecting.I got the error as - Exception - DPI-1047: Cannot locate a 64-bit Oracle Client library: "libclntsh.so: cannot open shared object file: No such file or directory". See https://cx-oracle.readthedocs.io/en/latest/user_guide/installation.html for help was raised.
I downloaded instaclient and used that in my script and it worked using the below commands :
LOCATION = r"C:\instantclient_19_5"
os.environ["PATH"] = LOCATION + ";" + os.environ["PATH"]
But now I need to use this in CI CD pipeline. I have created a docker image for instaclient and python I am trying use this into my script. But I am not sure how to use add instaclient location in script (like the above code snippet) Could you please help me with this.
If you were deploying to Windows (or macOS), you could have used the new cx_Oracle 8 init_oracle_client() function, which is preferred to fiddling with PATH.
However it seems you are deploying to Linux, meaning the system library search path needs to contain all library directories before the process start. So you need to use ldconfig or set LD_LIBRARY_PATH as traditionally used on Linux. The cx_Oracle doc Installing cx_Oracle on Linux and Locating the Oracle Client Libraries covers this.
Also see sample docker images for Linux developers and for basic Oracle Instant Client. If you are not on an RPM based system, check out the sample Docker files in Docker for Oracle Database Applications in Node.js and Python.
In summary, download Oracle Instant Client Basic or Basic Light packages from here. If you got the ZIP files then run something like:
RUN echo /opt/oracle/instantclient_19_8 > /etc/ld.so.conf.d/oic.conf && \
ldconfig
The details vary with what base Docker image you are using, and whether you want to install Instant Client from ZIP files or RPMs.
I have installed Hadoop and HIVE on windows 10 by following tutorials,
https://exitcondition.com/install-hadoop-windows/ & https://www.youtube.com/watch?v=npyRXkMhrgk respectively.
Both Hadoop and HIVE are running on my machine, I have been able to put files in HDFS and run queries in HIVE, but when I try to connect HIVE with python it gives different errors. Such as
from pyhive import hive
hive.Connection(host='localhost',port=10000,auth='NOSASL')
it gives following error:
TTransportException: TSocket read 0 bytes
I have tried impala as well but it did not work.
How can I connect python with hive, is it possible on windows 10 or should I shift to linux?
Pyhive had issues with auth = NOSASL in past.. not sure whether it got fixed .
Try hdfs3 python lib
conda install hdfs3
from hdfs3 import HDFileSystem
hdfs=HDFileSystem(host='localhost',port=9000)
More info available here..
https://medium.com/#arush.xtremelife/connecting-hadoop-hdfs-with-python-267234bb68a2
I am having trouble submitting a Pyspark job from my Windows driver machine (Win 10) to a simple Spark cluster running on Ubuntu.
There are several posts already that attempt to answer this question, most notably this one from ThatDataGuy here but none of them have helped.
Every time I try to submit the simple wordcount.py example to my remote master from my Windows box, I get the following error:
Cannot run program 'C:\apps\Python\3.6.6\python.exe': error=2, No such file or directory
This is a Java IOException generated by the Py4J jar.
My Spark cluster is a simple Master, 1 Worker setup in VirtualBox setup via Vagrant. All machines, (my Spark driver laptop, and 2 VMs (Master / Worker) have identical Spark 2.4.2, Python 3.6.6, and Scala 12.8. Note that Scala programs using spark-submit against the remote cluster work fine, as well as anything run in local mode. Also, the code examples work fine when run on either the Master or Worker nodes directly. It's only when I try to use my Windows laptop as a Spark driver in Pyspark, against the Ubuntu Spark cluster, that this issue arises. It always returns the error above.
It seems that Py4j is trying to use or instantiate Python from my Windows Driver's python path, which of course my Linux cluster can't see. I have already set the Pyspark Python path to a different value in the cluster nodes. I have set the both PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON in the nodes environment variables (via .bashrc), in spark-defaults.conf, AND in spark-env.sh files. All values point to /usr/local/bin/python3 as that's where Python 3.6.6 is installed on the Master and Worker nodes.
I've also (just as a hunch) aliased "python" to point to /usr/local/bin/python3 in the nodes and then changed my Windows python shortcut to pull up the same Python version. No luck but I was grasping at straws. ;/ Error simply changed to:
Cannot run program 'python': error=2, No such file or directory
I did see an article where the Py4J 0.10.7 library does not support Python 3.7, so this caused me to drop down to Python 3.6. Error stayed the same after that though.
The only thing I haven't done is to try to setup an additional shared / synced folder in Vagrant back to my Windows Python installation and then use /vagrant/shared/python/whatever in my remote PYSPARK settings. No idea if that would work though given I'm dealing with Windows and Linux Python flavors (all 3.6.6). Ugh. :/
Any ideas? I've got a Windows 10 machine and I like to do my Python development there. I've also got 64GB of RAM so I'd like to use it. Please don't make me switch to Scala! ;)
-- Pyspark works fine in local
spark-submit C:\apps\Spark\spark-2.4.2\examples\src\main\python\wordcount.py C:\Users\sitex\Desktop\p_and_p_ch1.txt
-- Pyspark fails when calling master with IOException
spark-submit --master spark://XXX.XX.XXX.XXX:7077 C:\apps\Spark\spark-2.4.2\examples\src\main\python\wordcount.py C:\Users\sitex\Desktop\p_and_p_ch1.txt
UPDATE: Ok, so it looks like my workaround is to pretend that my Driver (Windows laptop) knows the Python path in Linux that the Worker needs to use. Fortunately for me, I do it as this entire setup is running on my laptop. Here is the code that gets me past the error:
spark-submit --conf spark.pyspark.driver.python=python --conf spark.pyspark.python=/usr/local/bin/python3 --master spark://172.28.128.150:7077 C:\apps\Spark\spark-2.4.2\examples\src\main\python\wordcount.py /vagrant/shared/p_and_p_ch1.txt
Now, I should add that this DOESN'T run wordcount.py as I quickly realized that my Cluster cannot figure out Window's paths and my attempt to use a Vagrant synced / shared folder is resulting in a File Not Found on the p_and_p_ch1.txt file. But it does get me past my dreaded error. I can figure out how to stash my files on a network share / S3 / et some other day.
This puts a lot of onus on the Spark Driver knowing exactly what Python path the Cluster needs to use. Fortunately I know these settings as the setup is entirely on my laptop, but isn't the entire point of this that I am supposed to submit Spark jobs to a cluster without the Driver (me) knowing settings like the Worker nodes' Python path? I'm wondering if this is just a Windows + Linux quirk?
I am using pyspark under ubuntu with python 2.7
I installed it using
pip install pyspark --user
And trying to follow the instruction to setup spark cluster
I can't find the script start-master.sh
I assume that it has to do with the fact that i installed pyspark and not regular spark
I found here that i can connect a worker node to the master via pyspark, but how do i start the master node with pyspark?
https://pypi.python.org/pypi/pyspark
The Python packaging for Spark is not intended to replace all ... use cases. This Python packaged version of Spark is suitable for interacting with an existing cluster (be it Spark standalone, YARN, or Mesos) - but does not contain the tools required to setup your own standalone Spark cluster. You can download the full version of Spark from the Apache Spark downloads page.
Well i did a bit of a mix-up in the op.
You need to get spark on the machine that should run as master.
You can download it here
After extracting it, you have spark/sbin folder, there you have start-master.sh script. you need to start it with -h argument.
please note that you need to create a spark-env file like explained here and define the spark local and master variables, this is important on the master machine.
After that, on the worker nodes, use the start-slave.sh script to start worker nodes.
And you are good to go, you can use a spark context inside python to use it!
If you are already using pyspark through conda / pip installation, there's no need to install Spark and setup environment variables again for cluster setup.
For conda / pip pyspark installation is missing only 'conf', 'sbin' , 'kubernetes', 'yarn' folders, You can simply download Spark and move those folders into the folder where pyspark is located (usually site-packages folder inside python).
After you installed pyspark via pip install pyspark, you can start the Spark standalone cluster master process using this command:
spark-class org.apache.spark.deploy.master.Master -h 127.0.0.1
And then you can add some workers (executors), which would process the jobs:
spark-class org.apache.spark.deploy.worker.Worker \
spark://127.0.0.1:7077 \
-c 4 -m 8G
Flags -c and -m specify the number of CPU cores and amount of memory provided by the worker.
The 127.0.0.1 local address is used there for security reasons (it isn't good if anyone just copy/pasting this lines would expose an "arbitary code execution service" in their network), but for the distributed standalone Spark cluster setup the different address should be used (ex, a private IP address in an isolated network available only for this cluster nodes and their intended users, and an official Spark security guide should be read).
The spark-class script is contained in the "pyspark" python package, and it is a wrapper to load the environment variables from spark-env.sh and add the corresponding spark jars locations to -cp flag of java command.
If you may need to configure the environment - consult the official Spark docs, but it also works and may be suitable for the regular usage with default parameters. Also, see the flags for the master/worker commands using their --help.
This is an example how to connect to this standalone cluster using pyspark script with ipython shell:
PYSPARK_DRIVER_PYTHON=ipython \
pyspark --master spark://127.0.0.1:7077 \
--num-executors 2
--executor-cores 2
--executor-memory 4G
The code for instantiating spark session manually, ex. in Jupyter:
from pyspark.sql import SparkSession
spark = (
SparkSession.builder
.master("spark://127.0.0.1:7077")
# the number of executors this job needs
.config("spark.executor.instances", 2)
# the number of CPU cores memory this needs from the executor,
# it would be reserved on the worker
.config("spark.executor.cores", "2")
.config("spark.executor.memory", "4G")
.getOrCreate()
)