I have an account to a computing cluster that uses Scientific Linux. Of course I only have user access. I'm working with python and I need to run python scripts, so I need to import some python modules. Since I don't have root access, I installed a local python copy on my $HOME with all the required modules. When I run the scripts on my account (hosting node), they run correctly. But in order to submit jobs to the computing queues (to process on much faster machines), I need to submit a bash script that has a line that executes the scripts. The computing cluster uses SunGrid Engine. However when I submit the bash script, I get an error that the modules I installed can't be found!
So my understanding to the problem is that the modules are not sent or something to the machine that executes the script. My question is: is it possible to include all the modules in the script or so?
EDIT: I just created a bash script that runs which python and I noticed that the output was NOT my python copy. But when I run 'which python' on my ssh account, I get my python copy correctly..
The submitted script is most likely using the system Python installation and not your own. Try submitting a shell script with only one command, which python, to confirm.
The fix is to prepend the path to your Python interpreter to your system path. On my machine, the right Python is installed at /Users/mbatchkarov/anaconda/bin/python. I added export PATH="/Users/mbatchkarov/anaconda/bin:$PATH" to ~/.bash_profile
EDIT Add the same line to ~/.bashrc.
Related
I have a Ubuntu server with restricted access. There I will be hosting my application.
I trying to run Python scripts which were working with the default packages provided by the server. I want to work with numpy and other modules.
As I cannot install or download or do anything, I created a python server in my local machine (WINDOWS) using WSL to emulate the Linux file system and copied the python environment files to the application directory and deployed in cloud.
The problem is no matter in whatever way I try I cannot import numpy (or any module which I copied). I moved all the site-packages to the location of my Python script (As the current script's path will be there in the system path) and tried to import but no luck.
Please help me with crack this in any possible or impossible way.
I am trying to achieve this for the past 6 days and cannot do it.
Please, I have to achieve this at any cost. I have attached my latest structure.
Thank you in advance.
My Folder structure screenshot:
EDIT:
Ok. Let me get this straight. I have a Linux server (Ubuntu 18.04) where I am hosting an application. From that application, I am calling python scripts for some machine learning purposes. It is restricted server and I cannot access it. The only way that I found out the Linux distro version is through Java code by calling some terminal commands using "ProcessBuilder". As the server is highly restricted I cannot run any of the Linux commands like echo, set, export, sudo, wget/curl,...etc., Since, python3 is already provided by Linux (by default) I am using that python3 command to call my python scripts (from Java code using "ProcessBuilder") and execute them.
If it is a normal script (if I am using python standard libraries) it is working fine. In one of the scripts I am using "numpy". So, I want to import that module. I am doing the development in a windows environment. So, to emulate the Linux file system for importing packages I created a virtual environment in WSL with same Ubuntu version and installed numpy and then replaced all the symlinks inside those packages with the required files. Then I copied the entire environment and pasted in my resources directory (which is in windows environment) and deployed. No luck.
So, I made a zip file for only "site-packages" folder inside that environment. Then I copied the zip file and pasted in my resources folder and deployed. No luck. The error that I always see is "numpy.core._multiarray_umath". All the articles and in GitHub also tell us to re-install the package. But, I cannot install. I don't have any such access.
How can I import numpy without installation? If there is any work around to achieve this please explain, I will do it. Even if it is harder, complex and time-consuming I am okay with it. I want to achieve this.
Let me preface this with:
a warning to please check the AUP (acceptable use policy) of the server you are using, and/or contact the server administrator to make sure you are not violating any rules.
I can think of quite a few reasons why this won't work. If it doesn't, then there may still be workarounds, but they'll be technically complex.
So if I'm understanding you correctly:
You have very limited access to the server; basically only the ability to upload (apparently) and run Java code.
You've also been able to upload Python code and run it through your Java code through ProcessBuild.
You do not have access to log in to a shell, execute arbitrary command other than through ProcessBuild, etc.
Of course, you do not have the ability to install site-packages into the system Python environment.
So ultimately, what you'll probably need to do is something like:
Create a Python3 virtual environment (which doesn't seem to be what you are actually doing) on WSL. By a "Python3 virtual environment", I mean venv, which allows you to create a user-level (not system-level) directory with your packages.
So something like (from inside your project directory):
python3 -v venv venv
source ./venv/bin/activate
Your path will be adjusted so that your python3 and pip3 commands will be found in the venv path. pip3 install numpy will install it into this virtual environment (not the global/system Python).
Upload that entire venv directory to the server. You seem to have some way of doing this already.
You're going to have to have some way of running the Bash shell through ProcessBuilder. Since you have the ability to run python3 through ProcessBuilder, I'm kind of assuming that you will be able to do this as well.
You'll need to (through ProcessBuild) activate the virtual environment on the server, <path_to_project>/venv/bin/activate and, in the same Bash shell run your code.
This will look something like:
bash -c "source ./venv/bin/activate; python3 main.py"
I am looking to schedule my python script runs with jenkins. The issue is, my scripts use a lot of libs like pandas etc that are installed on my mac terminal.
Is there a way to allow Jenkins to pick up these modules (or run the scripts as if it was terminal)? Also is there a way to run Python3 in jenkins?
I have already configured Jenkins to execute from custom workspace and have tried both shell and plugin executions.
The answer is yes, but it is detailed so I can only give you high level steps here. Jenkins can execute command line statements, and python modules can be run from the command line.
I would start by using the begins library to create a python file to run from the command line with arguments. Get it working on your local machine that way.
You will want to use either virtualenv or venv, and do all your pip installs using that virtual environment. Then you can copy the virtual environment to your Jenkins machine, or create a new one. Look into the freeze tool.
When calling your python from jenkins, you must first activate your virtual environment just as if you were working on it yourself.
You have a lot of research to do, but is very doable. I can help with follow up questions if needed.
I have a bunch of useful Python3 scripts that I use in different OS environments. They run fine when called from the terminal but I want to run them from both Windows Explorer and OSX Finder. In Windows, that is not an issue as the *.py files get associated with the installed interpreter.
For OSX, I am having some problems. I tried using the shebang line.
#!/usr/bin/python3
But Finder doesn't recognize it as a script. How do I make Finder recognize *.py files as scripts? I am interested in maximum portability with as few OS modifications as possible.
Also I explicitly specified the interpreter location above. It will be in different locations for different OSes. Is there a way to make it multi-OS so that if it doesn't find it in one location, it will try another? Say if I want to run it from one of Ubuntu's GUI File Manager.
EDIT: Here's more info on my use case. I want to go to a new computer and git clone my repo of Python3 scripts and be able to run them from within the GUI file manager with minimal changes to the new computer.
To make the script more portable use #!/usr/bin/env python3 as shebang. Side effect will be that it will run the first python3 interpreter it finds in the $PATH (Unix StackExchange question on that subject)
To run the script from the finder try to chmod the file. i.e.
chmod +x FILE
Found another possibility here: Link to the Blog
Make this the first line of your Python script "#!/usr/bin/env python"
Change the extension of the script file to ".command" i.e. my_python_script.command
In Terminal make the Python script file executable by running "chmod +x my_python_script.command"
Now when you double click the Python script in Finder it will open a terminal window and run.
Alternative I found would be this but it needs system modification: Superuser question
I have a script that I wrote in Python. It uses some libraries such as Twython that allows me to post pictures and text to twitter. Now I am trying to make my own photo-booth that runs on Windows. I wish to run the script on the windows computer and I do not wish to install Python on the Windows machine.
I have tried to make exectuables but they do not work (I think because I have inputs that need run after the script (python script.py Message filepath). I thought there was a way to include everything of a normal Python install in the folder you run your script from and it will allow me run it cross platform.
Edit: My question is how can I package python up to run with out installing it preferably from a folder on the desktop.
I am trying to automate running a python script every minute on my mac, from a virtual environment. I am convinced that I don't properly understand permissions, paths, and environment variables in some crucial way that is preventing me from figuring this out.
I am an admin user with root rights enabled. I run HomeBrew, PIP and Virtualenv to manage python packages and virtual environments for different projects.
I would like to do the following every 60 seconds:
$ source /.virtualenvs/myenvironment/bin/activate
$ cd ~/desktop/python/
$ python myscript.py
$ deactivate
I have tried:
(a) writing my own plist for Launchd - and I believe these documents were well formed.
(b) programs which manage Launchd daemons and agents for you (both Launch Control and Lingon).
(c) I have tried simply editing the crontab (only lets me if I use the sudo command).
The python script, which works on command, pulls data from an online source and stores it in a sqlite table. I can tell the cron isn't running because the sqlite db isn't being touched.
Any thoughts would be enormously appreciated.
You don't say exactly what you tried with launchd and cron, but I'd bet you weren't using either of them correctly. Both are oriented toward running single, isolated commands (/programs), not sequences of shell commands. While it's possible to do this with a single cron job or launchd item, it's going to be messy. The simplest thing would be to write a shell script that does the sequence you want (be sure to include a shebang at the beginning, and enable execute permission on the script with chmod +x /path/to/script), and run that from either cron or launchd:
#!/bin/bash
source /.virtualenvs/myenvironment/bin/activate
cd ~/desktop/python/
python myscript.py
deactivate
I would not recommend using Automator to wrap the command sequence; it's designed for GUI-based scripting, and may not work right in a background-only job.
I have had this exact same problem and have recently solved it. Look here for the steps I took. Basically it deals with the shell needing the PYTHONPATH, not just the PATH.
So I'm still not sure why I can't configure Launchd or the Crontab to work for me: I still suspect it has to do with the environment variables.
If anyone else is having a tough time with this issue, I suggest creating an Automator application that runs the shell script you want to run, then use Lingon or Launch Control to run the Automator application file at whatever frequency you desire.