I am looking to schedule my python script runs with jenkins. The issue is, my scripts use a lot of libs like pandas etc that are installed on my mac terminal.
Is there a way to allow Jenkins to pick up these modules (or run the scripts as if it was terminal)? Also is there a way to run Python3 in jenkins?
I have already configured Jenkins to execute from custom workspace and have tried both shell and plugin executions.
The answer is yes, but it is detailed so I can only give you high level steps here. Jenkins can execute command line statements, and python modules can be run from the command line.
I would start by using the begins library to create a python file to run from the command line with arguments. Get it working on your local machine that way.
You will want to use either virtualenv or venv, and do all your pip installs using that virtual environment. Then you can copy the virtual environment to your Jenkins machine, or create a new one. Look into the freeze tool.
When calling your python from jenkins, you must first activate your virtual environment just as if you were working on it yourself.
You have a lot of research to do, but is very doable. I can help with follow up questions if needed.
Related
I am currently trying to debug a numpy code that I wrote using Gdb. To do that I activate first my virtual environment(on the command prompt) using the following commands (numpy_dev_env is my virtual environment)
numpy_dev_env\Scripts\activate
After that I activate my virtual environment.
gdb
then type "python" to start coding in python (all these steps work good for me).However, I realized that my test were not ran on the virtual environment,but instead on python system available in MINGW. I would appreciate your advice on ways to run the virtual enviroment with gdb.
Thank you,
I tried to configure the .gdbinit file in my home directory.I however did not know how to do with that
edit. clarified that when I run python on gdb the system used is the one available on MINGW and not of my system
Ensure that you are running the correct version of Python: start python in one shell, search for the process id using your task manager, then start gdb and attach the process.
Alternatively, start gdb with the full path to your python inside your environment.
I have a Ubuntu server with restricted access. There I will be hosting my application.
I trying to run Python scripts which were working with the default packages provided by the server. I want to work with numpy and other modules.
As I cannot install or download or do anything, I created a python server in my local machine (WINDOWS) using WSL to emulate the Linux file system and copied the python environment files to the application directory and deployed in cloud.
The problem is no matter in whatever way I try I cannot import numpy (or any module which I copied). I moved all the site-packages to the location of my Python script (As the current script's path will be there in the system path) and tried to import but no luck.
Please help me with crack this in any possible or impossible way.
I am trying to achieve this for the past 6 days and cannot do it.
Please, I have to achieve this at any cost. I have attached my latest structure.
Thank you in advance.
My Folder structure screenshot:
EDIT:
Ok. Let me get this straight. I have a Linux server (Ubuntu 18.04) where I am hosting an application. From that application, I am calling python scripts for some machine learning purposes. It is restricted server and I cannot access it. The only way that I found out the Linux distro version is through Java code by calling some terminal commands using "ProcessBuilder". As the server is highly restricted I cannot run any of the Linux commands like echo, set, export, sudo, wget/curl,...etc., Since, python3 is already provided by Linux (by default) I am using that python3 command to call my python scripts (from Java code using "ProcessBuilder") and execute them.
If it is a normal script (if I am using python standard libraries) it is working fine. In one of the scripts I am using "numpy". So, I want to import that module. I am doing the development in a windows environment. So, to emulate the Linux file system for importing packages I created a virtual environment in WSL with same Ubuntu version and installed numpy and then replaced all the symlinks inside those packages with the required files. Then I copied the entire environment and pasted in my resources directory (which is in windows environment) and deployed. No luck.
So, I made a zip file for only "site-packages" folder inside that environment. Then I copied the zip file and pasted in my resources folder and deployed. No luck. The error that I always see is "numpy.core._multiarray_umath". All the articles and in GitHub also tell us to re-install the package. But, I cannot install. I don't have any such access.
How can I import numpy without installation? If there is any work around to achieve this please explain, I will do it. Even if it is harder, complex and time-consuming I am okay with it. I want to achieve this.
Let me preface this with:
a warning to please check the AUP (acceptable use policy) of the server you are using, and/or contact the server administrator to make sure you are not violating any rules.
I can think of quite a few reasons why this won't work. If it doesn't, then there may still be workarounds, but they'll be technically complex.
So if I'm understanding you correctly:
You have very limited access to the server; basically only the ability to upload (apparently) and run Java code.
You've also been able to upload Python code and run it through your Java code through ProcessBuild.
You do not have access to log in to a shell, execute arbitrary command other than through ProcessBuild, etc.
Of course, you do not have the ability to install site-packages into the system Python environment.
So ultimately, what you'll probably need to do is something like:
Create a Python3 virtual environment (which doesn't seem to be what you are actually doing) on WSL. By a "Python3 virtual environment", I mean venv, which allows you to create a user-level (not system-level) directory with your packages.
So something like (from inside your project directory):
python3 -v venv venv
source ./venv/bin/activate
Your path will be adjusted so that your python3 and pip3 commands will be found in the venv path. pip3 install numpy will install it into this virtual environment (not the global/system Python).
Upload that entire venv directory to the server. You seem to have some way of doing this already.
You're going to have to have some way of running the Bash shell through ProcessBuilder. Since you have the ability to run python3 through ProcessBuilder, I'm kind of assuming that you will be able to do this as well.
You'll need to (through ProcessBuild) activate the virtual environment on the server, <path_to_project>/venv/bin/activate and, in the same Bash shell run your code.
This will look something like:
bash -c "source ./venv/bin/activate; python3 main.py"
For test purposes,
I tried to install Blender as a Python module in a virtual environment with Python 3.7 to be sure that i can use bpy.
I follow this page for the installation (https://wiki.blender.org/wiki/Building_Blender/Other/BlenderAsPyModule)
I would like at the end, when I wrote a Python script who used blender command to be able to run outside Blender software and to get information when my script finish
So I have multiples questions:
I don't really understand this line of command for Windows and how I can adapt to my virtual environment :
xcopy /E bin\2.81 C:\Python37\2.81
I can't use 'import bpy' command even when I successfully do pip install bpy in the command line ?
For my final goal project, I'm afraid that it's impossible to do because I couldn't find examples or questions related to my project.
Can someone could explain a little if it's possible or interesting to use Blender externally ?
I have been trying to run a python script on start-up (on a Pi). I initially did this via an .sh script triggered by cron.
Posting with a problem on StackExchange Pi (https://raspberrypi.stackexchange.com/questions/110868/parts-of-code-not-running-when-autostarting-script-in-crontab) the suggestion is to use systemd.
The person helping me there has suggested not using a virtual environment when executing the Python script (they note their limited familiarity with Python), and using the real environment instead. But other resources strongly suggest the use of a virtual environment (e.g. https://docs.python.org/3/tutorial/venv.html).
In the hope of setting this up correctly could anyone weigh in on the correct approach?
Use the virtual environment. I don't see any reason not to. At some point you might want to have multiple Python applications run at the same time on that system, and these applications might require different versions of the same dependency and then you would be back to square one, so... Use the virtual environment.
When configuring systemd, crontab, or whatever, make sure to use the python binary that is placed inside the virtual environment's bin directory, so that there is no need to activate the virtual environment:
/path/to/venv/bin/python -m my_executable_module
/path/to/venv/bin/python /path/to/my_script.py
/path/to/venv/bin/my_executable_script
systemd is going to try to run your script on startup so your virtual environment will not have been activated yet. You can (maybe) avoid that issue by telling systemd to use the python in the virtualenv's bin, with the appropriate environment variables. Or you can activate as a pre-run step for that script's launch in systemd. Maybe.
But on balance I'd make it easy on systemd and your OS and ignore the virtualenv absolutists. Get the script to work on your dev machine using virtualenv all you want, but then prep systemd to use the global python, with suitable packages installed. You can always use virtualenvs on that pi, for scripts that don't have to work with systemd. Systemd doesn't always have the clearest error messages.
(If you need to import custom modules you could inject directories into sys.path in your script. This could even avoid installing packages, for the global Python, entirely.)
This answer is certainly opinion-based.
There is an application where I downloaded the source code and would like to modify it.
https://github.com/ownaginatious/fbchat-archive-parser
Is there a way to run this program by entered a command such as "python3 main.py" rather than installing the program. When the program is installed, I would simply run the command fbcap.
This project has a setup.py file. The "standard" way to run such a thing is to install it (preferably inside a virtualenv) and then run it. Virtualenvs are cheap and lightweight so it's easy to do.
First, create a virtualenv. This might be slightly different depending on your platform. I presume you're using Python 3 so these instructions should work for you. Let's assume you created it in /tmp/venv1
Activate it using . /tmp/venv1/bin/activate (don't forget the initial .). This also might be different if you're on Windows.
Now install your program using python setup.py install.
Run it using fbcap.
This will allow you to run the program in a clean fresh python environment and when you're done experimenting, you can simply delete the virtualenv directory.