I am writing a code in python that uses numpy, matplotlib etc.
How to make sure that even a remote web server with python installed but no extra modules, can run the code without errors?
I usually work on linux environment. Hence from source code, I can install the libraries in a prefix directory and can keep that along with my code. Then add pythonpath locally in my python code to use the directory.
But, I started to realize it's not correct way as first thing, it can't work on cross platform as the libraries are different, and my code inside the script to extend the pythonpath may not work due to the use of "/" in path. Also, I am not sure if the compiled code can work in different environments of the same Linux Platform.
So I think I need to create a directory like unix,windows,osx etc. and put my code there? I believe this is what I find when I download any code online. Is that what developers generally do to avoid these issues?
A popular convention is to list requirements in a text file (requirements.txt) and install them when deploying the project. Depending on your deployment configuration, libraries can be installed in a virtual environment (google keyword: virtualenv), or in a local user folder (pip install --user -r requirements.txt, if this is the only project under this account) or globally (pip install -r requirements.txt, e.g. in a docker container)
Related
I have a Ubuntu server with restricted access. There I will be hosting my application.
I trying to run Python scripts which were working with the default packages provided by the server. I want to work with numpy and other modules.
As I cannot install or download or do anything, I created a python server in my local machine (WINDOWS) using WSL to emulate the Linux file system and copied the python environment files to the application directory and deployed in cloud.
The problem is no matter in whatever way I try I cannot import numpy (or any module which I copied). I moved all the site-packages to the location of my Python script (As the current script's path will be there in the system path) and tried to import but no luck.
Please help me with crack this in any possible or impossible way.
I am trying to achieve this for the past 6 days and cannot do it.
Please, I have to achieve this at any cost. I have attached my latest structure.
Thank you in advance.
My Folder structure screenshot:
EDIT:
Ok. Let me get this straight. I have a Linux server (Ubuntu 18.04) where I am hosting an application. From that application, I am calling python scripts for some machine learning purposes. It is restricted server and I cannot access it. The only way that I found out the Linux distro version is through Java code by calling some terminal commands using "ProcessBuilder". As the server is highly restricted I cannot run any of the Linux commands like echo, set, export, sudo, wget/curl,...etc., Since, python3 is already provided by Linux (by default) I am using that python3 command to call my python scripts (from Java code using "ProcessBuilder") and execute them.
If it is a normal script (if I am using python standard libraries) it is working fine. In one of the scripts I am using "numpy". So, I want to import that module. I am doing the development in a windows environment. So, to emulate the Linux file system for importing packages I created a virtual environment in WSL with same Ubuntu version and installed numpy and then replaced all the symlinks inside those packages with the required files. Then I copied the entire environment and pasted in my resources directory (which is in windows environment) and deployed. No luck.
So, I made a zip file for only "site-packages" folder inside that environment. Then I copied the zip file and pasted in my resources folder and deployed. No luck. The error that I always see is "numpy.core._multiarray_umath". All the articles and in GitHub also tell us to re-install the package. But, I cannot install. I don't have any such access.
How can I import numpy without installation? If there is any work around to achieve this please explain, I will do it. Even if it is harder, complex and time-consuming I am okay with it. I want to achieve this.
Let me preface this with:
a warning to please check the AUP (acceptable use policy) of the server you are using, and/or contact the server administrator to make sure you are not violating any rules.
I can think of quite a few reasons why this won't work. If it doesn't, then there may still be workarounds, but they'll be technically complex.
So if I'm understanding you correctly:
You have very limited access to the server; basically only the ability to upload (apparently) and run Java code.
You've also been able to upload Python code and run it through your Java code through ProcessBuild.
You do not have access to log in to a shell, execute arbitrary command other than through ProcessBuild, etc.
Of course, you do not have the ability to install site-packages into the system Python environment.
So ultimately, what you'll probably need to do is something like:
Create a Python3 virtual environment (which doesn't seem to be what you are actually doing) on WSL. By a "Python3 virtual environment", I mean venv, which allows you to create a user-level (not system-level) directory with your packages.
So something like (from inside your project directory):
python3 -v venv venv
source ./venv/bin/activate
Your path will be adjusted so that your python3 and pip3 commands will be found in the venv path. pip3 install numpy will install it into this virtual environment (not the global/system Python).
Upload that entire venv directory to the server. You seem to have some way of doing this already.
You're going to have to have some way of running the Bash shell through ProcessBuilder. Since you have the ability to run python3 through ProcessBuilder, I'm kind of assuming that you will be able to do this as well.
You'll need to (through ProcessBuild) activate the virtual environment on the server, <path_to_project>/venv/bin/activate and, in the same Bash shell run your code.
This will look something like:
bash -c "source ./venv/bin/activate; python3 main.py"
I am working on several projects on the same PyCharm. Like I "attached" them all together. But I recently noticed some weird behaviors. Like when I import a library I haven't installed yet to my script. It shows me a little error as expected. But when I try to install that using python -m pip install my_library, it tells me that it has already installed. I recently noticed that this is because it's using and other pip from another project. I doesn't use the one in the venv folder in the project. Also to run the scripts sometimes it uses python.exe from pythons original directory. It's a whole mess and I have no idea how I can solve it. Sometimes my projects requires different versions of the same library and you can imagine what happens when I change the version.
I make sure each project is using their own interpreter. Don't know what else to do other than this. I am using Python3.6.4 PyCharm2018.3.2 running on Windows10
it sounds like all your projects are configured to use the system's interpreter instead of the virtual environment you set up for each of them.
Follow this instruction to fix it https://www.jetbrains.com/help/pycharm-edu/creating-virtual-environment.html
In terms of using different version of the python library, you can address that by specifying it in requirements.txt file, which you can put in your venv folder for each project. then you can just do pip install -r requirements.txt after you set up your venv. (you need to ensure that the venv is activated - you don't need to worry about this if you have configured the project in PyCharm to use the venv's python interpreter.) You can check this by going to Terminal in your PyCharm and you should see (venv_name) hostusername#host:~/project_folder$
I have a Python application that comes with a command line script. I expose the script via setuptools "entry point" feature. Whenever a user runs the script, I would like the environment to be consistent with the package's requirements.txt. This means that the environment must contain versions of each dependency package that match the version specifiers in requirements.txt.
I know that this can be achieved with venv/virtualenv by making my users create a virtual environment, install requirements.txt in it, and activate that virtual environment whenever they run the script. I do not want to impose this burden of manually invoking virtualenv on users. Ruby's bundler solves this problem by providing bundler/setup-- when loaded, it modifies Ruby's $LOAD_PATH to reflect the contents of the Gemfile (analogue of requirements.txt). Thus it can be placed at the top of a script to transparently control the runtime environment. Does Python have an equivalent? That is, a way to set the environment at runtime according to requirements.txt without imposing additional complexity on the user?
Does Python have an equivalent? That is, a way to set the environment at runtime according to requirements.txt without imposing additional complexity on the user?
Yes, more than one.
One is pex
pex is a library for generating .pex (Python EXecutable) files which
are executable Python environments in the spirit of virtualenvs.
and the other is Platter:
Platter is a tool for Python that simplifies deployments on Unix
servers. It’s a thin wrapper around pip, virtualenv and wheel and aids
in creating packages that can install without compiling or downloading
on servers.
I don't see why it wouldn't be possible for a Python program to install its own dependencies before importing them, but it is unheard of in the Python community.
I'd rather look at options to make your application a standalone executable, as explained here, for instance.
I have a bunch of Python scripts that I want to deploy to other machines. Thing is, I want to have everything self-contained and not depend on the other machines' libraries. For example I don't want to request users to have virtual environment and pip as installed in order for my app to work.
On my local machine I use virtual environment with --no-site-packages and pip install -r requirements.txt to get everything in place.
The bad news is virtualenvironment's activate script has my local path hardcoded into it and using the --relocatable option does not help with this situation so I suppose virtualenvironment is out of the question?
What I would like to have is something similar to this:
base_app_dir:
- main_app_dir
- my_init_script.py
- bin(includes python binary)
- lib(includes pip installed packages and python libraries)
so that I can instruct the end user to just cd into base_app_dir and do a ./bin/python -m my_init_script.py but that means I now need to instruct Python to look into my ./lib folder when importing packages.
I've tried setting os.path.insert(1, 'base_app_dir/lib/site-packages') but this work on per module basis.
Also how about lookup for default Python modules? Right now for example when import hashlib it tries to get it from /usr/lib/python2.7/hashlib.py. I would like to deploy these default Python modules as well and instruct the app to import them from my custom location.
Py2exe or creating a .deb file is not an option right now so please try to address my specific question.
I was used VirtualBox manual setups with virtualenvs inside them to run Django projects on my local machine. Recently I discovered Vagrant and decided to switch to it, because it seems very easy and useful.
But I can not figure - do I need still use virtualenv Vagrant VM, is it encouraged practice or forbidden?
As Devin stated, it is not necessary to use virtualenv when you deploy to a vagrant machine as long as you are the sole user of the machine. However, I would still enable the use of a virtualenv, setup.py, etc. even if you do not use it for development or deployment.
In my (not so) humble opinion, any Python project should:
Include a .cvsignore, .gitignore, .hgignore, ... file that ignores the common Python intermediate files as well as virtualenv directories.
A requirements.txt file that lists the required packages in a pip-compliant format
Include a Makefile with the following targets:
environment: create the virtual environment using virtualenv or pyvenv
requirements: install required packages using pip and the requirements.txt file
develop: run setup.py develop using the virtual environment
test: run setup.py test
clean: remove intermediate files, coverage reports, etc.
maintainer-clean: remove the virtual environment
The idea is to keep the Makefile as simple as possible. The dependencies should be set up so that you can clone the repository (or extract the source tarball) and run make test. It should create a virtual environment, install the requirements, and run the unit tests.
You can also include a Vagrantfile and a vagrant target in the Makefile that runs vagrant up. Add a vagrant destroy to the maintainer-clean target while you are at it.
This makes your project usable by anyone that is using vagrant or developing without it. If (when) you need to use deploy alongside another project in a vagrant or physical environment, including a clean setup.py and a Vagrantfile that describes your minimal environment makes it simple to install into a virtual environment or a shared vagrant machine.
If you run one vagrant VM per project, then there is no direct reason to use virtualenv.
If other contributors do not use vagrant, but do use virtualenv, then you might want to use it and support it to make their lives easier.
Virtualenv and other forms of isolation (Docker, dedicated VM, ...) are not necessarily mutually exclusive. Using virtualenv is still a good idea, even in an isolated environment, to shield the virtual system Python from your project packages. *nix systems use plethora of Python based utilities dependent on specific versions of packages being available in system Python and you don't want to mess with these.
Mind that virtualenv can still only go as far as pure Python packages and doesn't solve the situation with native extensions that will still mix with the system.