I have a Ubuntu server with restricted access. There I will be hosting my application.
I trying to run Python scripts which were working with the default packages provided by the server. I want to work with numpy and other modules.
As I cannot install or download or do anything, I created a python server in my local machine (WINDOWS) using WSL to emulate the Linux file system and copied the python environment files to the application directory and deployed in cloud.
The problem is no matter in whatever way I try I cannot import numpy (or any module which I copied). I moved all the site-packages to the location of my Python script (As the current script's path will be there in the system path) and tried to import but no luck.
Please help me with crack this in any possible or impossible way.
I am trying to achieve this for the past 6 days and cannot do it.
Please, I have to achieve this at any cost. I have attached my latest structure.
Thank you in advance.
My Folder structure screenshot:
EDIT:
Ok. Let me get this straight. I have a Linux server (Ubuntu 18.04) where I am hosting an application. From that application, I am calling python scripts for some machine learning purposes. It is restricted server and I cannot access it. The only way that I found out the Linux distro version is through Java code by calling some terminal commands using "ProcessBuilder". As the server is highly restricted I cannot run any of the Linux commands like echo, set, export, sudo, wget/curl,...etc., Since, python3 is already provided by Linux (by default) I am using that python3 command to call my python scripts (from Java code using "ProcessBuilder") and execute them.
If it is a normal script (if I am using python standard libraries) it is working fine. In one of the scripts I am using "numpy". So, I want to import that module. I am doing the development in a windows environment. So, to emulate the Linux file system for importing packages I created a virtual environment in WSL with same Ubuntu version and installed numpy and then replaced all the symlinks inside those packages with the required files. Then I copied the entire environment and pasted in my resources directory (which is in windows environment) and deployed. No luck.
So, I made a zip file for only "site-packages" folder inside that environment. Then I copied the zip file and pasted in my resources folder and deployed. No luck. The error that I always see is "numpy.core._multiarray_umath". All the articles and in GitHub also tell us to re-install the package. But, I cannot install. I don't have any such access.
How can I import numpy without installation? If there is any work around to achieve this please explain, I will do it. Even if it is harder, complex and time-consuming I am okay with it. I want to achieve this.
Let me preface this with:
a warning to please check the AUP (acceptable use policy) of the server you are using, and/or contact the server administrator to make sure you are not violating any rules.
I can think of quite a few reasons why this won't work. If it doesn't, then there may still be workarounds, but they'll be technically complex.
So if I'm understanding you correctly:
You have very limited access to the server; basically only the ability to upload (apparently) and run Java code.
You've also been able to upload Python code and run it through your Java code through ProcessBuild.
You do not have access to log in to a shell, execute arbitrary command other than through ProcessBuild, etc.
Of course, you do not have the ability to install site-packages into the system Python environment.
So ultimately, what you'll probably need to do is something like:
Create a Python3 virtual environment (which doesn't seem to be what you are actually doing) on WSL. By a "Python3 virtual environment", I mean venv, which allows you to create a user-level (not system-level) directory with your packages.
So something like (from inside your project directory):
python3 -v venv venv
source ./venv/bin/activate
Your path will be adjusted so that your python3 and pip3 commands will be found in the venv path. pip3 install numpy will install it into this virtual environment (not the global/system Python).
Upload that entire venv directory to the server. You seem to have some way of doing this already.
You're going to have to have some way of running the Bash shell through ProcessBuilder. Since you have the ability to run python3 through ProcessBuilder, I'm kind of assuming that you will be able to do this as well.
You'll need to (through ProcessBuild) activate the virtual environment on the server, <path_to_project>/venv/bin/activate and, in the same Bash shell run your code.
This will look something like:
bash -c "source ./venv/bin/activate; python3 main.py"
Related
I am trying to download and run Python and PyCharm off of a usb stick because I am a newbie and suck with remembering all the OS specific setup, so the code I work on at work, does not work on my home computer.
When I try to run python.exe from the usb drive, I get the error "Windows cannot access the specified device path or file you may not have appropriate permissions". I am local admin and can open folders, just not run the exe.
Any help would be greatly appreciated
TL;DR I want a centralized place for all of my code to be neatly and easily stored, retrieved, edited, and tested.
use python virtualenv
pip install virtualenv
python -m virtualenv mypython
It will generate a 'new' python (mypython) that you may copy into your USB stick.
Read the documentation here.
Try Portable Python.
Python virtualenv use your installed python that's why you can't share it using usb stick or add it on your github repository.
The answer to this question could be "You're as dumb as a wooden bowl" but I have searched a lot and haven't found a solution without installing python on other computers.
I have a python/flask web app that I need to distribute to many users. However, I can't install python on all those computers and there is no computer which everyone can access. And I cant serve the app internally from a server either. Yes, that's what I'm dealing with.
I have saved the git repo it in a network drive that everyone can access. I hoped I could run a batch file to spin the localhost server from a copied environment for the user and then use the web app.
I copied a conda environment over to the network drive and tried to use that but that gave me a Importing the numpy c-extensions failed error.
I tried including a pip environment (.\env) in the folder. So I thought any user could just activate the environment using the batch file ...
cd %cd%
.\env\Scripts\activate.bat
.\env\Scripts\python.exe run.py
but it's not working.
The .\env\Scripts\activate just crashes. I amended the activate.bat set "VIRTUAL_ENV=%cd%\env" to ensure it uses the current folder. Still crashes.
If I exclude that then .\env\Scripts\python.exe run.py still looks for a python installation at the path I have on my machine rather than the path I provided above.
Is there a solution to this?
All the computers will be using Windows but may vary between Windows 7 and Windows 10. I'm doing the development from my Windows 10 computer.
After activating venv My below code worked:
(Monday) C:\Users\Resurctova\Desktop\PoraPuski\Monday>python new.py
Output :
testing
since the new.py has code to print testing
As Monday is my venv I activated it and executed the script.
Do not execute in Scripts folder of your venv environment
Have you though about creating an exec file and that will save all your folders. Using a tool lik PyInstaller. You just need to share the output exe file without installing python.
I've just setup a new environment for my project and uploaded a python repository including bin, lib and project folder. I'm pretty sure I did same previously and it worked without problem. Now when doing the same on an AWS environment I get the error
-bash: /projects/scrapy/bin/python2.7: cannot execute binary file. However when doing source /projects/scrapy/bin/activate it successfully activates the environment.
From what I understand, python should be able to execute without any issue no matter the environment ?
Any help or pointing to the right direction would be much appreciated!
python should be able to execute without any issue no matter the environment ?
No, the Python binary is tied to your specific OS and computer architecture. Python source code can usually be run on different machines (provided you didn't use OS-specific features), but that's only made possible by compiling a Python interpreter for the specific target environment first.
In other words, a Python binary compiled to run on macOS will not work on Linux.
All that source bin/activate achieves is that it configures your terminal setting to use the bin directory as the first directory on the PATH search path. This doesn't make bin/python work in another environment, it just means that both environments have a working shell interpreter that can run that script.
Create a new virtualenv with a Python binary compiled for Linux, and install the same packages there. Use Pipenv or a requirements.txt file to transfer the dependencies from Mac to Linux.
For example, using Pipenv you'd copy over the Pipfile and Pipfile.lock files to the other computer, then run pipenv install in the directory there and re-create the virtualenv and dependencies from those files.
I recommend you read up on Python development best practices in the The Hitchhiker’s Guide to Python; this includes such topics on how to manage an environment for a project.
I am writing a code in python that uses numpy, matplotlib etc.
How to make sure that even a remote web server with python installed but no extra modules, can run the code without errors?
I usually work on linux environment. Hence from source code, I can install the libraries in a prefix directory and can keep that along with my code. Then add pythonpath locally in my python code to use the directory.
But, I started to realize it's not correct way as first thing, it can't work on cross platform as the libraries are different, and my code inside the script to extend the pythonpath may not work due to the use of "/" in path. Also, I am not sure if the compiled code can work in different environments of the same Linux Platform.
So I think I need to create a directory like unix,windows,osx etc. and put my code there? I believe this is what I find when I download any code online. Is that what developers generally do to avoid these issues?
A popular convention is to list requirements in a text file (requirements.txt) and install them when deploying the project. Depending on your deployment configuration, libraries can be installed in a virtual environment (google keyword: virtualenv), or in a local user folder (pip install --user -r requirements.txt, if this is the only project under this account) or globally (pip install -r requirements.txt, e.g. in a docker container)
I am looking to schedule my python script runs with jenkins. The issue is, my scripts use a lot of libs like pandas etc that are installed on my mac terminal.
Is there a way to allow Jenkins to pick up these modules (or run the scripts as if it was terminal)? Also is there a way to run Python3 in jenkins?
I have already configured Jenkins to execute from custom workspace and have tried both shell and plugin executions.
The answer is yes, but it is detailed so I can only give you high level steps here. Jenkins can execute command line statements, and python modules can be run from the command line.
I would start by using the begins library to create a python file to run from the command line with arguments. Get it working on your local machine that way.
You will want to use either virtualenv or venv, and do all your pip installs using that virtual environment. Then you can copy the virtual environment to your Jenkins machine, or create a new one. Look into the freeze tool.
When calling your python from jenkins, you must first activate your virtual environment just as if you were working on it yourself.
You have a lot of research to do, but is very doable. I can help with follow up questions if needed.