I am currently attempting to set up Matlab to work with Volttron on a Linux virtual machine. Python 3.8, Volttron, and Matlab are all installed on the virtual machine. When I run
pyversion python.exe
in the command window I get this error:
Error using pyversion
Path argument does not specify a valid executable.
Running pe = pyenv; and pe.Version returns blank, as does pyversion. This document describes a way to set the version used and I believe this ought to be my next step. However, the instructions say that for Linux I should run
pyenv('Version','executable')
but python is already installed and to my knowledge on linux does not have an executable file one can download for python. How can I remedy this?
If you have followed the recommended steps to setup VOLTTRON, and are running VOLTTRON within a virtual environment, the python version to use should be located within that virtual environment at env/bin/python.
As mentioned in this answer, if you want to verify the path, you can activate the environment using source env/bin/activate, and then run python. Once inside the python interpreter, you would just need to print the system executable.
import sys
print(sys.executable)
It is worth noting that this is an older method for connecting to MatLab with VOLTTRON. You may want to try using the newer MatLab agents. The documentation for this method is included with the example agents. https://volttron.readthedocs.io/en/latest/developing-volttron/developing-agents/example-agents/matlab-agent.html
The new method also assumes that MatLab is running in a separate Windows environment. In your case, you would install the standalone MatLab Agent within the linux virtual machine, and proceed accordingly.
Related
I have a Ubuntu server with restricted access. There I will be hosting my application.
I trying to run Python scripts which were working with the default packages provided by the server. I want to work with numpy and other modules.
As I cannot install or download or do anything, I created a python server in my local machine (WINDOWS) using WSL to emulate the Linux file system and copied the python environment files to the application directory and deployed in cloud.
The problem is no matter in whatever way I try I cannot import numpy (or any module which I copied). I moved all the site-packages to the location of my Python script (As the current script's path will be there in the system path) and tried to import but no luck.
Please help me with crack this in any possible or impossible way.
I am trying to achieve this for the past 6 days and cannot do it.
Please, I have to achieve this at any cost. I have attached my latest structure.
Thank you in advance.
My Folder structure screenshot:
EDIT:
Ok. Let me get this straight. I have a Linux server (Ubuntu 18.04) where I am hosting an application. From that application, I am calling python scripts for some machine learning purposes. It is restricted server and I cannot access it. The only way that I found out the Linux distro version is through Java code by calling some terminal commands using "ProcessBuilder". As the server is highly restricted I cannot run any of the Linux commands like echo, set, export, sudo, wget/curl,...etc., Since, python3 is already provided by Linux (by default) I am using that python3 command to call my python scripts (from Java code using "ProcessBuilder") and execute them.
If it is a normal script (if I am using python standard libraries) it is working fine. In one of the scripts I am using "numpy". So, I want to import that module. I am doing the development in a windows environment. So, to emulate the Linux file system for importing packages I created a virtual environment in WSL with same Ubuntu version and installed numpy and then replaced all the symlinks inside those packages with the required files. Then I copied the entire environment and pasted in my resources directory (which is in windows environment) and deployed. No luck.
So, I made a zip file for only "site-packages" folder inside that environment. Then I copied the zip file and pasted in my resources folder and deployed. No luck. The error that I always see is "numpy.core._multiarray_umath". All the articles and in GitHub also tell us to re-install the package. But, I cannot install. I don't have any such access.
How can I import numpy without installation? If there is any work around to achieve this please explain, I will do it. Even if it is harder, complex and time-consuming I am okay with it. I want to achieve this.
Let me preface this with:
a warning to please check the AUP (acceptable use policy) of the server you are using, and/or contact the server administrator to make sure you are not violating any rules.
I can think of quite a few reasons why this won't work. If it doesn't, then there may still be workarounds, but they'll be technically complex.
So if I'm understanding you correctly:
You have very limited access to the server; basically only the ability to upload (apparently) and run Java code.
You've also been able to upload Python code and run it through your Java code through ProcessBuild.
You do not have access to log in to a shell, execute arbitrary command other than through ProcessBuild, etc.
Of course, you do not have the ability to install site-packages into the system Python environment.
So ultimately, what you'll probably need to do is something like:
Create a Python3 virtual environment (which doesn't seem to be what you are actually doing) on WSL. By a "Python3 virtual environment", I mean venv, which allows you to create a user-level (not system-level) directory with your packages.
So something like (from inside your project directory):
python3 -v venv venv
source ./venv/bin/activate
Your path will be adjusted so that your python3 and pip3 commands will be found in the venv path. pip3 install numpy will install it into this virtual environment (not the global/system Python).
Upload that entire venv directory to the server. You seem to have some way of doing this already.
You're going to have to have some way of running the Bash shell through ProcessBuilder. Since you have the ability to run python3 through ProcessBuilder, I'm kind of assuming that you will be able to do this as well.
You'll need to (through ProcessBuild) activate the virtual environment on the server, <path_to_project>/venv/bin/activate and, in the same Bash shell run your code.
This will look something like:
bash -c "source ./venv/bin/activate; python3 main.py"
I am using Python 3.7.9 Shell.
I created a virtual environment in this location
C:\Users\my_username\Desktop\Projects.venv
Inside of Python Shell, when I type: import numpy, which is in my .venv\lib folder, it says that the module does not exist.
Using Python Shell, how do I make use of the contents in .venv? In particular, the libraries located there?
Edit #1: Include Details
In my windows command line, it has (.venv) off to the left.
I have run the Activate file. I then started Python.
In my \lib\site-packages area, I have the requests library.
When I open up Python Shell and type "import requests", it says "no such library can be found"
I am using Windows 10
I installed the libraries while in the (.venv) environment.
Theory:
In the virtual environment, in Python Shell, it's searching a different location for libraries...now if I can just figure out where it's searching and how to change that...I might be able to make progress.
Edit #2: My Progress
My theory was correct. Despite using a virtual environment, it's not looking for the libraries installed in (.venv)\lib\site-packages, it's looking somewhere outside of that.
Now I just need to figure out how to make the Python code look for libraries inside of (.venv)\lib\site-packages when I'm in the virtual environment.
When I run the python.exe file inside of the (.venv)\Scripts location, it recognizes the virtual environment scripts.
If I click on my version of Python.Exe located in my C:...\Programs\Python 3.7 folder, it doesn't recognize them.
I was under the impression it didn't matter where I clicked on the Python.exe file if I did it after going to the virtual environment in the command line? Is this not true?
Edit #3: Important Links
Where Python Looks for Modules When Importing
Right from the official docs https://docs.python.org/3/tutorial/venv.html#creating-virtual-environments
Once you’ve created a virtual environment, you may activate it.
On Windows, run:
tutorial-env\Scripts\activate.bat
On Unix or MacOS, run:
source tutorial-env/bin/activate
this is done in your shell before starting python at its prompt, and allows you to choose different python versions in addition to other benefits
I'm using VS Code with WSL on my PC, and whenever I try to run a Python file in VS Code, it gives me an error that no such directory or file is found.
I've set WSL as my default shell and the issue, of course, is that VS Code is feeding in the Windows-based path to the python script, without the /mnt/C/... that's needed when using WSL. Is there any way for me to set up VS Code so that I can automatically have VS Code feed in the right, WSL-based file paths (i.e., including the /mnt/C/...)?
Secondly, it also seems that when using the WSL terminal in VS Code, it's using Python 2.7. Since I've never installed Python 2.7 on my PC, I'm assuming it comes with the Linux distro I'm using. Is it also possible to set up VS Code so that even when using the WSL terminal, the python installation that's used is the one I've installed on Windows (which is actually part of an Anaconda distribution)?
You need to make sure to use the Remote - WSL extension so that VS Code treats your environment as Linux instead of as Windows.
As for reusing your Windows install of Python under WSL, I wouldn't try that as Python is compiled for a specific OS and thus makes assumptions that will break under another OS.
I have deployed HDP: 2.6.4 on a virtual machine
I can see that the spark2 is not pointing to the correct python folder. My questions are
1) How can I find where my python is located?
solution: Type whereis python and you will get a list of where it is
2) How can I update the existing python libraries and add new libraries to that folder ? For example, the equivalent of 'pip install numpy' on CLI.
Nothing clear yet
3) How can I make Zeppelin Spark2 point at that specific directory that contains the python folder that I can update? - On Zeppelin, there is a little 'edit' button that I can change the path to the directory that contains python.
solution: go to the interpreter in zeppelin, find spark2, and make zeppelin.pyspark.python point to where python is already there.
Now if you need python 3.4+ there is a whole set of different steps you have to do, to first get python 3.4.+ into the HDP sandbox.
Thank you,
For a Sandbox environment like yours, a sandbox image is made on a Linux OS (CentOS). The Zeppelin Notebook points, in all probability, to the Python installation that comes along with every Linux OS.
If you wish to have your own installation of Python and your own set of libraries for Data Analysis like those in the SciPy stack. You need to install Anaconda on your Virtual machine. Your VM eed to be connected to the internet so that you can download and install the Anaconda package for testing.
You can then point Zeppelin to the anaconda's directory till the following path : /home/user/anaconda3/bin/python where user is your username
Zeppelin Configuration also confirms the fact that it uses the default python installation at /usr/bin/python. You can go through its documentation for more Information
UPDATE
Hi Joseph, Spark Installations, by default, use the Python interpreter and the python libraries that have been installed on your OS. The folder structure that you have shown only tell you the location of the PySpark module. This module is a library like Pandas ior NumPy.
What you can do is install the SciPy Stack[NumPy, Pandas, MatplotLib etc..] via the command pip install package name and import those libraries directly into your Zeppelin Notebook.
Use the command whereis python in the terminal of your snadbox, the result would give you something as follows
/usr/bin/python /usr/bin/python2.7 ....
In your Zeppelin Configuration, for the property zeppelin.pyspark.python you can set the first value from the out put of the previous command i.e /usr/bin/python. So now all the libraries you installed via the pip install command would be available for you in zeppelin.
This process would only work for your Sandbox environment. In a real production cluster, your administrator needs to install all these libraries on all the nodes of your Spark cluster.
I need to run python in an environment where there wont be python. Is it possible to execute python as an executable in Unix Environments, like HP-UX, IBM-AIX, Solaris, Linux etc etc....
The targeted OS is AIX now.... since it does not have python support and the installation is difficult......
Thanks.
I have used http://www.pyinstaller.org/ to create an executable in ubuntu. look at their manual, it also have the cool feature of outputting just one file with --onefile. My first choice was freeze but the executable failed to run when I used some external modules - I could not solve it and I found pyinstaller to be perfect for me.