I try to create a standalone python app using Pyinstaller which should run and can be reproducibly recreated on Windows, Linux, and Mac.
The idea is to use Docker to have a fixed environment that creates the app and exports it again. For Linux and Windows, I could make it work using https://github.com/cdrx/docker-pyinstaller.
the idea is to let docker create a fully functioning pyinstaller app (with GUI) within and export this app. Since pyinstaller depends on the package versions etc. this should be fixed in the docker and only the new source code should be supplied to compile and export a new version of the software.
in the ideal scenario (and how it already works with Linux and Windows), the user can create the docker and compile the software itself:
docker build -t docker_pyinstaller_linux https://raw.githubusercontent.com/loipf/calimera_docker_export/main/linux/Dockerfile
docker run --rm -v "/path/to/app_source_code:/code/" -v "${PWD}:/code/dist/" docker_pyinstaller_linux
For Mac, however, there is no simple straightforward solution. There is one Mac docker image out there https://github.com/sickcodes/Docker-OSX, but the docker creation code is not that simple.
my idea:
take https://github.com/sickcodes/Docker-OSX/blob/master/Dockerfile.auto
add to the end the download of miniconda:
RUN chmod -R 777 /Users/user
### install miniconda to /miniconda
RUN curl -LO "http://repo.continuum.io/miniconda/Miniconda3-4.4.10-MacOSX-x86_64.sh"
RUN bash Miniconda3-4.4.10-MacOSX-x86_64.sh -p /miniconda -b
ENV PATH=/miniconda/bin:${PATH}
RUN conda update -y conda
### install packages from conda
RUN conda install -c anaconda -y python=3.8
RUN apt-get install -y python3-pyqt5
...
but already the first command fails due to chmod: cannot access '/Users/user': No such file or directory since I am in a different "environment" than the interactive session (/home/arch/OSX-KVM). Can somebody help me out?
I know these asking for code pieces questions are not the best, and I could show you all the things I tried, but I doubt they will help anybody. I would like to have a minimum Mac example without user login or gui etc (which should be possible using https://github.com/sickcodes/osx-optimizer). It should only have miniconda installed (with pyinstaller and a few packages).
other info:
I can run the previous commands in the mac environment interactively in the docker environment but would like to fix these commands permanently in the image:
docker pull sickcodes/docker-osx:auto ### replace with docker with pyinstaller and packages
docker run -it \
--device /dev/kvm \
-p 50922:10022 \
sickcodes/docker-osx:auto
ideally in the end, we can run the above command with OSX_commands pyinstaller file.spec
related questions but without solution:
Using Docker and Pyinstaller to distribute my application
How to create OS X app with Python on Windows
I tried to build a standalone executable with PyInstaller for Python 3.5 using OpenVino 2020.4.287.
PyInstaller assembled a file successfully but I received the next error after launch:
ImportError: No module named 'openvino'
I tried to include /opt/intel/openvino/deployment_tools/inference_engine/lib/armv7l/plugins.xml in data while building executable but it didn't help.
So the question is how to build a standalone executable in Raspbian with PyInstaller with OpenVino import?
Thanks.
Okay, the issue was resolved.
Initial conditions:
Raspbian Stretch with Python 3.5.
Considerations:
The last OpenVino supporting Python 3.5 is 2020.4.
So, let’s start. We will create a test.py file with the next lines:
import numpy as np
import openvino.inference_engine.constants
from openvino.inference_engine import IENetwork, IECore
print("start")
e = IECore()
print("end")
Now let's install OpenVino, Pyinstaller, create an executable, and run it:
sudo pip3 install pyinstaller
sudo mkdir -p /opt/intel/openvino
wget https://storage.openvinotoolkit.org/repositories/openvino/packages/2020.4/l_openvino_toolkit_runtime_raspbian_p_2020.4.287.tgz
sudo tar -xf l_openvino_toolkit_runtime_raspbian_p_2020.4.287.tgz --strip 1 -C /opt/intel/openvino
rm l_openvino_toolkit_runtime_raspbian_p_2020.4.287.tgz
sudo apt install cmake
source /opt/intel/openvino/bin/setupvars.sh
sh /opt/intel/openvino/install_dependencies/install_NCS_udev_rules.sh
pyinstaller --onefile --clean --add-data="/opt/intel/openvino/deployment_tools/inference_engine/lib/armv7l/plugins.xml:." test.py
dist/test
"start" and "end" messages should be shown.
The most important when you are using OpenVino 2020.4 is the next line:
import openvino.inference_engine.constants
In previous versions of OpenVino everything will work normally without this line. But in my specific case I’ve got the next error on OpenVino versions earlier 2020.4:
openvino.inference_engine.ie_api.IECore' object has no attribute 'read_network
That’s why I specified import for openvino.inference_engine.constants. Without this nothing works in OpenVino 2020.4.
Thanks.
The biggest possibility is you didn't run the setupvars before that implementation.
Please note that you need to ensure the setupvars had been run and initialized(you should see the init message) in each cmd or terminal before proceeding to any further inferencing/etc.
This setupvars would keep the required packages together. Hence, if you didn't run it, the problem like what you are facing would persist.
I installed gettyimages/spark docker image and jupyter/pyspark-notebook inside my machine.
However as the gettyimage/spark python version is 3.5.3 while jupyter/pyspark-notebook python version is 3.7, the following error come out:
Exception: Python in worker has different version 3.5 than that in
driver 3.7, PySpark cannot run with different minor versions.Please
check environment variables PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON
are correctly set.
So, i have tried to upgrade the python version of gettyimage/spark image OR downgrade the python version of jupyter/pyspark-notebook docker image to fix it.
Lets talk about method 1, downgrade jupyter/pyspark-notebook python version first:
I use conda install python=3.5 to downgrade the python version of jupyter/pyspark-notebook docker image. However, after i do so , my jupyter notebook cannot connect to any single ipynb and the kernel seems dead. Also, when i type conda again, it shows me conda command not found, but python terminal work well
I have compare the sys.path before the downgrade and after it
['', '/usr/local/spark/python',
'/usr/local/spark/python/lib/py4j-0.10.7-src.zip',
'/opt/conda/lib/python35.zip', '/opt/conda/lib/python3.5',
'/opt/conda/lib/python3.5/plat-linux',
'/opt/conda/lib/python3.5/lib-dynload',
'/opt/conda/lib/python3.5/site-packages']
['', '/usr/local/spark/python',
'/usr/local/spark/python/lib/py4j-0.10.7-src.zip',
'/opt/conda/lib/python37.zip', '/opt/conda/lib/python3.7',
'/opt/conda/lib/python3.7/lib-dynload',
'/opt/conda/lib/python3.7/site-packages']
I think more or less, it is correct. So why i cannot use my jupyter notebook to connect to the kennel?
So i use another method, i tried upgrade the gettyimage/spark image
sudo docker run -it gettyimages/spark:2.4.1-hadoop-3.0 apt-get install
python3.7.3 ; python3 -v
However, I find that even i do so, i cannot run the spark well.
I am not quite sure what to do. May you share with me how to modify the docker images internal package version
If I look at the Dockerfile here, it installs python3 which by default is installing python 3.5 for debian:stretch. You can instead install python 3.7 by editing the Dockerfile and building it yourself. In your Dockerfile, remove lines 19-25 and replace line 1 with the following, and then build the image locally.
FROM python:3.7-stretch
If you are not familiar with building your own image, download the Dockerfile and keep it in its own standalone directory. Then after cd into the directory, run the command below . You may want to first remove the already downloaded image. After this you should be able to run other docker commands the same way as if you had pulled the image from docker hub.
docker build -t gettyimages/spark .
So it seems on ubuntu for windows (windows subsystem for linux) people are suggesting we need to use Agg backend and just save images, not show plots.
import matplotlib
matplotlib.use('Agg') # no UI backend
import matplotlib.pyplot as plt
import numpy as np
t = np.arange(0.0, 2.0, 0.01)
s = 1 + np.sin(2*np.pi*t)
plt.plot(t, s)
plt.title('About as simple as it gets, folks')
#plt.show()
plt.savefig("matplotlib.png") #savefig, don't show
How could we get it to where plt.show() would actually show us an image? My current option is to override plot.show() to instead just savefig a plot-148123456.png under /mnt/c/Users/james/plots/ in windows and just have an explorer window open viewing the images.
I suppose I could host that folder and use a browser.
My goal is to be able to run simple examples like the code above without changing the code to ftp the images somewhere etc. I just want the plot to show up in a window.
Has anyone figured out a decent way to do it?
Ok, so I got it working as follows. I have Ubuntu on windows, with anaconda python 3.6 installed.
Download and install VcXsrv or Xming (X11 for Windows) from sourceforge(see edit below)
sudo apt-get update
sudo apt-get install python3.6-tk (you may have to install a different python*-tk depnding on the python version you're using)
pip install matplotlib (for matplotlib. but many other things now work too)
export DISPLAY=localhost:0.0 (add to ~/.bashrc to make permanent. see WSL2 below)
Anyways, after all that, this code running in ubuntu on wsl worked as is:
import matplotlib.pyplot as plt
import numpy as np
t = np.arange(0.0, 2.0, 0.01)
s = 1 + np.sin(2*np.pi*t)
plt.plot(t, s)
plt.title('About as simple as it gets, folks')
plt.show()
result:
Maybe this is better done through a Jupyter notebook or something, but it's nice to have basic command-line python matplotlib functionality in Ubuntu for Windows on Subsystem for Linux, and this makes many other gui apps work too.
For example you can install xeyes, and it will say to install x11-apps and installing that will install GTK which a lot of GUI apps use. But the point is once you have your DISPLAY set correctly, and your x server on windows, then most things that would work on a native ubuntu will work for the WSL.
Edit 2019-09-04 : Today I was having issues with 'unable to get screen resources' after upgrading some libraries. So I installed VcXsrv and used that instead of Xming. Just install from https://sourceforge.net/projects/vcxsrv/ and run xlaunch.exe, select multiple windows, next next next ok. Then everything worked.
Edit for WSL 2 users 2020-06-23
WSL2 (currently insider fast ring) has GPU/docker support so worth upgrade. However it runs in vm. For WSL 2, follow same steps 1-4 then:
the ip is not localhost. it's in resolv.conf so run this instead (and include in ~/.bashrc):
export DISPLAY=`grep -oP "(?<=nameserver ).+" /etc/resolv.conf`:0.0
Now double-check firewall: Windows Security -> Firewall & network protection -> Allow an app through firewall -> make sure VcXsrv has both public and private checked. (When Launching xlaunch first time, you might get a prompt to allow through firewall. This works too. Also, if VcXsrv is not in list of apps, you can manually add it, eg from 'C:\program files\vcxsrv\vcxsrv.exe')
Launch VcXsrv with "Disable access control" ticked
Note: a few WSL2 users got error like couldn't connect to display "172.x.x.x:0". If that's you try to check the IP address stored in DISPLAY with this command: echo $DISPLAY. If the showed IP seems to be wrong (i.e. "8.8.8.8" or another not working IP address) you need to change the code in ~/.bashrc showed in the point 5 to something that will get your instance's ip address. One user said this worked: export DISPLAY=$(ifconfig | grep inet | awk '{print $2}' | head -n 1 | awk '{print $0":0"}'). However for some others it did not work. YMMV, but just find your IP and use if for DISPLAY. For most WSL2 users, the command in #5 works.
Edit for Windows 11 : if MS convinced you to throw out your old computer and buy one with a TPM and so you got Windows 11, you get GUI for free. I hope they add upgrade path to do that on Windows 10 because Win10 will be like XP and last a long time since MS decided you need recent computer even though Win11 would work fine on old computers.
Plots are also viewable on WSL using Visual Studio Code, which as of the June 2019 release, includes a "Plot Viewer".
In a regular .py file, inserting a comment string #%% marks the code as a Jupyter (IPython) cell and a code lens shows options to Run Cell. In addition, right-clicking inside a Python file includes the option:
Run Current File in Python Interactive Window
From the VS Code June 2019 release:
The June 2019 update included a brand-new Plot Viewer that can be used
to manipulate any image plots, such as the popular matplotlib plots.
You can try it out by double-clicking on the plots or clicking on the
“expand image” button that is displayed when you hover over plot
images in the Python Interactive Window:
With the plot viewer, you can pan, zoom in/out, navigate through plots
in the current session, and export plots to PDF, SVG, or PNG formats.
For WSL2 using Ubuntu Focal Fossa release, I downloaded and installed VcXsrv on Windows. I use this configuration:
I don't know for Native OpenGL but it seems important to disable access control.
Now, you need to export the env variable DISPLAY in WSL.
I found the correct address in /etc/resolv.conf, but the one found in the ifconfig result isn't. So the following command doesn't work for me:
export DISPLAY=$(ifconfig | grep inet | awk '{print $2}' | head -n 1 | awk '{print $0":0"}')
That IP address can also be found in the command ipconfig into CMD.exe command context. Search for vEthernet (WSL), that should be the one.
Now, you can test that everything is working by simply executing xcalc in your WSL environnement. If xcalc is not present, install it:
sudo apt-get update && sudo apt-get install x11-apps
xcalc command should open a application looking like this on Windows:
If it is working that means that the connection to your X-server is possible but you maybe need to install some extra package for Python like:
sudo apt-get install python3.x-tk
Change .x according to your python version.
To get matplotlib to work with GTKAgg on Bash on Ubuntu on Windows, I:
installed VcXsrv under Windows (but things should work just the same with Xming)
set DISPLAY as noted above [export DISPLAY=localhost:0.0 (add to ~/.bashrc to make permanent)]
executed sudo pip uninstall matplotlib
followed by sudo apt install python-matplotlib
updated matplotlibrc to read backend : GTKAgg (rather than backend : agg)
I also ran sudo apt-get install python-gtk2-dev, but this may not be necessary.
Uninstalling the pip-installed matplotlib and reinstalling it via apt appear to be necessary because pip does not include the C extensions needed to run GTK, but the apt version does.
With Windows 11 22000, Linux GUI apps are officially supported out of the box provided pre-requisites are met. No twaeks, No hacks, No firewall changes etc.
Pre-Requisites:
Windows 11 Build 22000 or higher
Installed driver for vGPU (Intel/Nvidia/AMD)
Then run the commands
wsl --update
wsl --shutdown
Done !!
Official Source : https://learn.microsoft.com/en-us/windows/wsl/tutorials/gui-apps
In my case (Windows 10, WSL2, Ubuntu 20.04 with miniconda3 and virtual environment where I installed all the required Python 3.6 packages using conda command, Windows-based Visual Studio Code with 'Python' and 'Remote - WSL' extensions) I installed VcXsrv under Windows, and added
export DISPLAY=$(awk '/nameserver / {print $2; exit}' /etc/resolv.conf 2>/dev/null):0
to .bashrc file in Ubuntu. Also, before running Python code from WSL, we have to launch XLaunch and select 'Disable access control' (I left 'Native opengl' checked as well). In project folder, do not name your Python file as matplotlib.py (some related issues are reported here).
In two seperate instances (wsl1 & wls2 with ubuntu 20.04) while using PyQt5 I was missing several libxcb libraries (e.g. libxcb-icccm4) which was not printed when trying to plot. I used export QT_DEBUG_PLUGINS=1 to find out which ones and then simply installed them. Basically the steps here.
had similar problem so simply installing the supported gpu driver solved my problem. Check this out!
I found the best approach is to install Jupyter on Windows Subsystem for Linux (WSL) by following
sudo apt update && upgrade
sudo apt install python3 python3-pip ipython3
Now you can install matplotlib
pip3 install matplotlib
And Jupyter Notebook
pip3 install jupyter
Check this link if you need more info Python setup on the Windows subsystem for Linux (WSL)
Currently using AWS to run some tests on a machine learning project. I would like to run Python scripts without internet (via root) because the internet bandwidth is extremely limited. I try to run the convnets.py script by doing
sudo python convnets.py >> output
But that does not work, as Anaconda does not use PYTHONPATH, making it impossible for root to find the Anaconda Python environment. So errors like "cannot import" and "module not found" are thrown.
How do I set this up so I can get Anaconda and sudo to play fair together?
Because using sudo uses a different PATH than your typical environment, you need to be sure to specify that you want to use Anaconda's python interpreter rather than the system python. You can check which one is being run with the following command
sudo which python
To fix this, and point to Anaconda's python interpreter, specify the full path to the correct interpreter.
sudo /path/to/anaconda/bin/python convnets.py >> output
If you do this, you should be able to access all of the modules managed by anaconda.
On the other hand, if you have an Anaconda environment created
conda create --name $ENVIRONMENT_NAME python
You can activate it prior to running your command
sudo source activate $ENVIRONMENT_NAME && python convnets.py >> output