We Keep Coding

Azure databricks PySpark custom UDF ModuleNotFoundError: No module named

I was checking this SO but none of the solutions helped PySpark custom UDF ModuleNotFoundError: No module named
I have the current repo on azure databricks:
|-run_pipeline.py
|-__init__.py
|-data_science
|--__init.py__
|--text_cleaning
|---text_cleaning.py
|---__init.py__
On the run_pipeline notebook I have this
from data_science.text_cleaning import text_cleaning
path = os.path.join(os.path.dirname(__file__), os.pardir)
sys.path.append(path)
spark = SparkSession.builder.master(
"local[*]").appName('workflow').getOrCreate()
df = text_cleaning.basic_clean(spark_df)
On the text_cleaning.py I have a function called basic_clean that will run something like this:
def basic_clean(df):
print('Removing links')
udf_remove_links = udf(_remove_links, StringType())
df = df.withColumn("cleaned_message", udf_remove_links("cleaned_message"))
return df
When I do df.show() on the run_pipeline notebook, I get this error message:
Exception has occurred: PythonException (note: full exception trace is shown but execution is paused at: <module>)
An exception was thrown from a UDF: 'pyspark.serializers.SerializationError: Caused by Traceback (most recent call last):
File "/databricks/spark/python/pyspark/serializers.py", line 165, in _read_with_length
return self.loads(obj)
File "/databricks/spark/python/pyspark/serializers.py", line 466, in loads
return pickle.loads(obj, encoding=encoding)
ModuleNotFoundError: No module named 'data_science''. Full traceback below:
Traceback (most recent call last):
File "/databricks/spark/python/pyspark/serializers.py", line 165, in _read_with_length
return self.loads(obj)
File "/databricks/spark/python/pyspark/serializers.py", line 466, in loads
return pickle.loads(obj, encoding=encoding)
ModuleNotFoundError: No module named 'data_science'
Shouldnt the imports work? Why is this an issue?

It seems data-science module is missing on cluster. Kindly consider
installing it on cluster.
Please check below link about installing libraries to cluster.
https://learn.microsoft.com/en-us/azure/databricks/libraries/cluster-libraries
You can consider executing pip list command to see libraries installed on cluster.
You can consider running pip install data_science command also directly in notebook cell.

I've been facing the same issue running pyspark tests with UDFs in Azure Devops. I've noticed that this happens when running from the pool with vmImage:ubuntu-latest. When I use a custom container build from the following Dockerfile, the tests run fine:
FROM python:3.8.3-slim-buster AS py3
FROM openjdk:8-slim-buster
ENV PYSPARK_VER=3.3.0
ENV DELTASPARK_VER=2.1.0
COPY --from=py3 / /
WORKDIR /setup
COPY requirements.txt .
RUN python -m pip install --upgrade pip
RUN pip install --no-cache-dir -r requirements.txt && \
rm requirements.txt
WORKDIR /code
requirements.txt contains pyspark==3.3.0 and delta-spark==2.1.0.
This led me to conclude that it's due to how spark runs in the default ubuntu VM which runs python 3.10.6 and java 11 (at the time of posting this). I've tried setting env variables such as PYSPARK_PYTHON to enforce pyspark to use the same python binary on which the to-be-tested package is installed but to no avail.
Maybe you can use this information to find a way to use the default agent pool's ubuntu vm to get it to work, otherwise I recommend just using a pre-configured container like I did.

Running django test on the gitlab ci

I have project in django 1.4 and I need to run django test in contious integration system (GitLab 6.8.1 with Gitlab CI 4.3).
Gitlab Runner have installed on server with project.
When I run:
cd project/app/ && ./runtest.sh test some_app
I get:
Traceback (most recent call last):
File "manage.py", line 2, in <module>
from django.core.management import execute_manager
ImportError: No module named django.core.management
How I may run tests?

Do you have Django installed on the testrunner?
If not, try to configure a virtualenv for your testsuite. Best might be (if you have changing requirements) to make the setup and installation of this virtualenv part of your testsuite.

Change your job script in gitlab-ci with the following:
#!/bin/bash
export DISPLAY=:10
virtualenv env
source env/bin/activate
pip install -r requirements.txt
python manage.py test
Before doing this,. install virtualenv and xvfb (for selenium test) for GitLab runners.

Python AppEngine 'import MySQLdb' not working, OS X 10.9.2, python 2.7

I've installed MySQLdb successfully for my system python -- and I've added the library to app.yaml as well, but the dev-appserver does not recognize it. I've followed the tutorial here: https://developers.google.com/appengine/docs/python/cloud-sql/
I downloaded and installed the latest Python mysql from here:
http://sourceforge.net/projects/mysql-python/
sudo CFLAGS=-Qunused-arguments CPPFLAGS=-Qunused-arguments python setup.py build
sudo CFLAGS=-Qunused-arguments CPPFLAGS=-Qunused-arguments python setup.py install
If I run python from the Command Line I can happily import MySQLdb
Here's what I have in app.yaml:
libraries:
- name: webapp2
version: "2.5.2"
- name: jinja2
version: "2.6"
- name: MySQLdb
version: "1.2.4b4"
Here's the exception I get when I try to open a page hosted by my appengine dev server.
ERROR 2014-05-15 20:23:19,480 wsgi.py:262]
Traceback (most recent call last):
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/runtime/wsgi.py", line 239, in Handle
handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/runtime/wsgi.py", line 298, in _LoadHandler
handler, path, err = LoadObject(self._handler)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/runtime/wsgi.py", line 84, in LoadObject
obj = __import__(path[0])
File "xxxx/main.py", line 10, in <module>
import MySQLdb
ImportError: No module named MySQLdb
Any help would be deeply appreciated -- the tutorial isn't working. Thanks for your generosity and kindness!
-s

The following worked for me. I got the same error from App Engine, but it turned out that the problem was module _mysql wasn't configured properly.
Add this to ~/.bash_profile:
export PATH=$PATH:/usr/local/mysql/bin
Create a symlink to the MySQL lib:
ln -s /usr/local/mysql/lib/libmysqlclient.18.dylib /usr/lib/libmysqlclient.18.dylib

I had the same problem with local development and struggled with it for a while. Eventually, I wrote command line version of all my queries, tested them and then fully deployed the app. It worked from there.
If you find a solution that allows local dev, please post it here.

linking python project with virtualenv

I have an ubuntu server 12 running as a guest in virtualbox on a windows 7 host.
In a shared folder /workspace I have a python project project01. Since virtualbox has issues with creating symlinks in a shared folder I created the virtualenv in my users home directory. /home/user1/venv.
I can activate the virtual environment no problem.
source ~/venv/bin/activate
but when I try to run manage.py, I get an error
~/venv/bin/python manage.py runserver
Traceback (most recent call last):
File "manage.py", line 8, in <module>
from django.core.management import execute_from_command_line
ImportError: No module named django.core.management
I'm sure I'm missing a path or something, but I'm not sure exactly what path to add and to where. I tried setting PYTHONPATH in the activate script, but that didn't work. Do I need to add the path to django's core management in the activate script?
I see a bunch of django directories under venv/build
Django
django-appconf
django-celery
django-celery-email
django-celery-transactions
django-debug-toolbar
django-filebrowser
django-grappelli
djangohelpers
django-imagekit
django-pipeline
django-redis
django-sslify
django-storages
django-widget-tweaks
do I need to point in there somewhere?
I'm very much a django and virtualenv noob.
UPDATE
It doesn't look like the packages were installed. So this is another matter entirely.
Thanks for your reply. I have some other issues I need to figure out here.
I was in the virtualenv and ran pip install -r requirements.txt which only had 1 error for postgres, but I'm not using a local postgres server so that's not a huge deal. Unless that error is preventing everything from installing, then it is a huge deal.
when I check site-packages I only see the following:
easy-install.pth
pip-1.2.1-py2.7.egg
setuptools-0.6c11-py2.7.egg
setuptools.pth
and pip freeze shows me:
argparse==1.2.1
wsgiref==0.1.2
the error from pip install is:
Exception information:
Traceback (most recent call last):
File "venv/local/lib/python2.7/site-packages/pip-1.2.1-py2.7.egg/pip/basecommand.py", line 107, in main
status = self.run(options, args)
File "venv/local/lib/python2.7/site-packages/pip-1.2.1-py2.7.egg/pip/commands/install.py", line 256, in run
requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
File "venv/local/lib/python2.7/site-packages/pip-1.2.1-py2.7.egg/pip/req.py", line 1042, in prepare_files
req_to_install.run_egg_info()
File "venv/local/lib/python2.7/site-packages/pip-1.2.1-py2.7.egg/pip/req.py", line 236, in run_egg_info
command_desc='python setup.py egg_info')
File "venv/local/lib/python2.7/site-packages/pip-1.2.1-py2.7.egg/pip/util.py", line 612, in call_subprocess
% (command_desc, proc.returncode, cwd))
InstallationError: Command python setup.py egg_info failed with error code 1 in venv/build/psycopg2

The error looks as if you do not have Django in the venv python path.
You can confirm that it is part of your venv python install using a couple of ways:
If you used PIP to install you can verify that the package is installed using pip freeze
Also check in lib/python2.7/site-packages/ of the venv there should be a django directory in there. This is the directory that is part of the PYTHONPATH for your venv.
If you cannot find the modules in there of have installed them in a different path. Then you will need to make sure that the path is part of the PYTHONPATH of the venv.

pkg_resources.DistributionNotFound: Django==1.3.1 while creating a new project in Django

I am trying to create a new project but it getting following error .
I am using python 2.6
and django 1.3.0
File "/usr/local/bin/django-admin.py", line 4, in <module>
import pkg_resources
File "/usr/lib/python2.6/dist-packages/pkg_resources.py", line 2659, in <module>
parse_requirements(__requires__), Environment()
File "/usr/lib/python2.6/dist-packages/pkg_resources.py", line 546, in resolve
raise DistributionNotFound(req)
pkg_resources.DistributionNotFound: Django==1.3.1

It seems you have both Django 1.3.0 and 1.3.1 installed and you're using the django-admin.py from Django 1.3.1 while having Django 1.3.0 in your default Python path.
Try either of:
Using the correct version of django-admin.py corresponding to the Django version in your Python path (the one inside the bin/ directory of your Django 1.3.0 installation).
You can check where your correct django-admin.py script is by executing this:
echo "$(dirname $(python -c "import django; print django.__file__"))/bin/django-admin.py"
Getting your Python path to correctly point to your Django 1.3.1 installation.

Try deleting easy_install.pth file if you try to install using easy_install too.
I have this issue as pip didn't worked i tried with easy_install and it installed the latest django, so have to delete it that file from dist-packages folder. Hope this helps some one referring.