I need to create a dummy RPM which appears to install some Python modules in site-packages to resolve an RPM dependency issue.
The real modules will be installed using PIP inside of a Python Virtual Environment, but in order for the system to work the modules which are imported need to be provided in the global site packages, so that needs to be faked.
The imports look as follows (example): from pear.apple.peach import Fruit
When performing an RPM Build on the package that has these imports it fails on dependency generation, so I need an RPM in the local repo to pretend to provide these so the dependency generation passes.
The approach you are proposing in the question doesn't make much sense to me, either you create rpm packages with python modules you need for other system rpm packages (packages you plan to create or install) or you just use virtual env and then you don't need to care much about what you have in system site packages, as long as you have particular version of python itself installed. While you can make system site packages visible in particular virtual env (for example, when you have lxml module among dependencies, you can install it from rpm package and then make a virtual env to have access to system site packages so that you don't need to install it again in virtualenv), having particular module provided in dummy rpm package would break it so this doesn't make sense as well.
In other words, having something installed in virtual env and then expect to make system python module installed via rpm package to depend on something from that virtual env is not going to work (imho).
So I suggest to do one of:
Build the rpm packages of modules you need to have installed for other packages to work. As long as you have some experience with rpmbuild, it should not be that hard, you can use pyp2rpm tool to create initial specfile which you would need to tweak a bit. But this also depends on how many of the packages you need to build and how often do you plan to update them (as it would mean to update the specfile and rebuild).
Use virtual env only, either with access to system site packages or not. This way, you install both dependencies and application into virtual env and you don't need to deal with rpm packages at all (as long as you install python and virtualenv from rpm packages).
Which way would make more sense in you case depends heavily on what are you trying to do (how do you plan to maintain the python modules, on how many machines do you plan to install it, if you plan to install it yourself or provide it for others to install it themselves and so on ...).
I was able to solve this issue by replicating every import with an empty file, and using empty __init__.py files at every folder level.
For example, to resolve from pear.apple.peach import Fruit, I would have needed to install the following file tree in site-packages:
-> pear
-> __init__.py
-> apple
-> peach.py
The relevant lines of the spec file for the dummy rpm:
...
source:FruitDummy.tar.gz
...
% install
mkdir -p $RPM_BUILD_ROOT%{python_sitelib}/pear/apple/
install __init__.py $RPM_BUILD_ROOT%{python_sitelib}/pear/
install peach.py $RPM_BUILD_ROOT%{python_sitelib}/pear/apple/
...
%files
%defattr(-,root,root,-)
%{python_sitelib}/pear/__init__.py*
%{python_sitelib}/pear/apple/peach.py*
...
__init__.py and peach.py were stored in the FruitDummy.tar.gz that was used to build the RPM.
Related
I am new at python, using virtual environment etc. I installed python in C drive, added PATH variable and worked from there. For learning purposes, I decided to install packages only inside my project directory. So, I created virtual environment inside my project folder by following a documentation. I have created it at my windows machine something like this way:
python -m venv my-env
After that, I activated it:
my-env\Scripts\activate.bat
After that, I have installed the Requests library (as it was in the documentation):
pip install requests
But, my question is: why need to install Requests library? Do I need to install it again if I create new project, new virtual environment inside that new project directory at future?
I installed some additional packages And I wanted to create requirements.txt. So, I wrote:
pip freeze > requirements.txt
Now, besides my required libraries for the project itself, every packages from that Requests library are in that requirements.txt file too. So, if I share my project with other user, that user will get packages from the Requests library when install via requirements.txt. Is it okay?
Sorry for lots of question. Basically, all the questions are related and contextual I think.
why need to install Requests library?
There's no need to install this library for the virtual environment to work properly.
Do I need to install it again if I create new project, new virtual environment inside that new project directory at future.
The idea of a virtual env is that: that you work in a controlled environment, created for that project. If you'll work on another project, in its own virtual env, you'll have to install it again. This is not bad at all: in each project, you might need to work with different versions of the same library, or you might even want/need to delete a project, and none of what happens in one project will affect the other, which is nice.
besides my required libraries for the project itself, every packages from that Requests library are in that requirements.txt file too.
Yes, the freeze command lists all the installed packages. That will include not only what you installed by pip install ..., but also the dependencies those packages needed. This is a reason why some people suggest to write the requirements file by hand: that way it's clearer what external resources are directly needed, as opposed to each sub-dependency, which, as you might have seen, becomes a little clumsy to understand.
if I share my project with other user, that user will get packages from the Requests library when install via requirements.txt. Is it okay?
It's ok in the sense that those packages will get installed either way. They are explicitly listed, but if they weren't they'd also be downloaded and installed.
First, my reasons to do this - I know it's a bad idea but I am out of ideas.
I want to install a package which requires a ld version, which is higher than the one in the repo of my Centos 6.5. So I should either go for setting up everything in a Docker and running it in production - something I lack experience with and I don't feel comfortable doing for a serious project. Or upgrade ld manually building from external source. Which I read, could result in devastation of my Centos. So the last option I am left with is install the packed on other machine and manually copy it to site-packages.
I have successfully installed the package on my home laptop under Debian.
I encountered everywhere advice to copy the whole site-packages directory. Something which I don't want to do as I have different packages on both machines and I want to avoid messing up with other stuff.
I copied the .so build and .egginfo of the package. Then, on the target machine, pip freeze indeed showed me the transferred package. However, Python can't find it when I try to import and use it.
Am I missing something else?
Not any of that.
Don't mess with system Python's site-packages dir, this belongs to the system Python env only. You should only add/remove code in there by using the package manager of your OS (that's yum for CentOS). This is especially true in Linux where many OS services can rely on system Python.
So what to do instead? Use a virtualenv and/or pipx to isolate any other dependencies of the package you want to install from the system versions.
I'm a Java/Scala dev transitioning to Python for a work project. To dust off the cobwebs on the Python side of my brain, I wrote a webapp that acts as a front-end for Docker when doing local Docker work. I'm now working on packaging it up and, as such, am learning about setup.py and virtualenv. Coming from the JVM world, where dependencies aren't "installed" so much as downloaded to a repository and referenced when needed, the way pip handles things is a bit foreign. It seems like best practice for production Python work is to first create a virtual environment for your project, do your coding work, then package it up with setup.py.
My question is, what happens on the other end when someone needs to install what I've written? They too will have to create a virtual environment for the package but won't know how to set it up without inspecting the setup.py file to figure out what version of Python to use, etc. Is there a way for me to create a setup.py file that also creates the appropriate virtual environment as part of the install process? If not — or if that's considered a "no" as this respondent stated to this SO post — what is considered "best practice" in this situation?
You can think of virtualenv as an isolation for every package you install using pip. It is a simple way to handle different versions of python and packages. For instance you have two projects which use same packages but different versions of them. So, by using virtualenv you can isolate those two projects and install different version of packages separately, not on your working system.
Now, let's say, you want work on a project with your friend. In order to have the same packages installed you have to share somehow what versions and which packages your project depends on. If you are delivering a reusable package (a library) then you need to distribute it and here where setup.py helps. You can learn more in Quick Start
However, if you work on a web site, all you need is to put libraries versions into a separate file. Best practice is to create separate requirements for tests, development and production. In order to see the format of the file - write pip freeze. You will be presented with a list of packages installed on the system (or in the virtualenv) right now. Put it into the file and you can install it later on another pc, with completely clear virtualenv using pip install -r development.txt
And one more thing, please do not put strict versions of packages like pip freeze shows, most of time you want >= at least X.X version. And good news here is that pip handles dependencies by its own. It means you do not have to put dependent packages there, pip will sort it out.
Talking about deploy, you may want to check tox, a tool for managing virtualenvs. It helps a lot with deploy.
Python default package path always point to system environment, that need Administrator access to install. Virtualenv able to localised the installation to an isolated environment.
For deployment/distribution of package, you can choose to
Distribute by source code. User need to run python setup.py --install, or
Pack your python package and upload to Pypi or custom Devpi. So the user can simply use pip install <yourpackage>
However, as you notice the issue on top : without virtualenv, they user need administrator access to install any python package.
In addition, the Pypi package worlds contains a certain amount of badly tested package that doesn't work out of the box.
Note : virtualenv itself is actually a hack to achieve isolation.
I have a couple projects that require similar dependencies, and I don't want to have pip going out and DLing the dependencies from the web every time. For instance I am using the norel-django package which would conflict with my standard django (rdbms version) if I installed it system wide.
Is there a way for me to "reuse" the downloaded dependancies using pip? Do I need to DL the source tar.bz2 files and make a folder structure similar to that of a pip archive or something? Any assistance would be appreciated.
Thanks
Add the following to $HOME/.pip/pip.conf:
[global]
download_cache = ~/.pip/cache
This tells pip to cache downloads in ~/.pip/cache so it won't need to go out and download them again next time.
it looks like virtualenv has a virtualenv-clone command, or perhaps virtualenvwrapper does?
Regardless, it looks to be a little more involved then just copyin and pasting virtual environment directories:
https://github.com/edwardgeorge/virtualenv-clone
additionally it appears virtualenv has a flag that will facilitate in moving your virtualenv.
http://www.virtualenv.org/en/latest/#making-environments-relocatable
$ virtualenv --relocatable ENV from virtualenv doc:
This will make some of the files created by setuptools or distribute
use relative paths, and will change all the scripts to use
activate_this.py instead of using the location of the Python
interpreter to select the environment.
Note: you must run this after you’ve installed any packages into the
environment. If you make an environment relocatable, then install a
new package, you must run virtualenv --relocatable again.
Also, this does not make your packages cross-platform. You can move
the directory around, but it can only be used on other similar
computers. Some known environmental differences that can cause
incompatibilities: a different version of Python, when one platform
uses UCS2 for its internal unicode representation and another uses
UCS4 (a compile-time option), obvious platform changes like Windows
vs. Linux, or Intel vs. ARM, and if you have libraries that bind to C
libraries on the system, if those C libraries are located somewhere
different (either different versions, or a different filesystem
layout).
If you use this flag to create an environment, currently, the
--system-site-packages option will be implied.
I'm newish to the python ecosystem, and have a question about module editing.
I use a bunch of third-party modules, distributed on PyPi. Coming from a C and Java background, I love the ease of easy_install <whatever>. This is a new, wonderful world, but the model breaks down when I want to edit the newly installed module for two reasons:
The egg files may be stored in a folder or archive somewhere crazy on the file system.
Using an egg seems to preclude using the version control system of the originating project, just as using a debian package precludes development from an originating VCS repository.
What is the best practice for installing modules from an arbitrary VCS repository? I want to be able to continue to import foomodule in other scripts. And if I modify the module's source code, will I need to perform any additional commands?
Pip lets you install files gives a URL to the Subversion, git, Mercurial or bzr repository.
pip install -e svn+http://path_to_some_svn/repo#egg=package_name
Example:
pip install -e hg+https://rwilcox#bitbucket.org/ianb/cmdutils#egg=cmdutils
If I wanted to download the latest version of cmdutils. (Random package I decided to pull).
I installed this into a virtualenv (using the -E parameter), and pip installed cmdutls into a src folder at the top level of my virtualenv folder.
pip install -E thisIsATest -e hg+https://rwilcox#bitbucket.org/ianb/cmdutils#egg=cmdutils
$ ls thisIsATest/src
cmdutils
Are you wanting to do development but have the developed version be handled as an egg by the system (for instance to get entry-points)? If so then you should check out the source and use Development Mode by doing:
python setup.py develop
If the project happens to not be a setuptools based project, which is required for the above, a quick work-around is this command:
python -c "import setuptools; execfile('setup.py')" develop
Almost everything you ever wanted to know about setuptools (the basis of easy_install) is available from the the setuptools docs. Also there are docs for easy_install.
Development mode adds the project to your import path in the same way that easy_install does. An changes you make will be available to your apps the next time they import the module.
As others mentioned, you can also directly use version control URLs if you just want to get the latest version as it is now without the ability to edit, but that will only take a snapshot, and indeed creates a normal egg as part of the process. I know for sure it does Subversion and I thought it did others but I can't find the docs on that.
You can use the PYTHONPATH environment variable or symlink your code to somewhere in site-packages.
Packages installed by easy_install tend to come from snapshots of the developer's version control, generally made when the developer releases an official version. You're therefore going to have to choose between convenient automatic downloads via easy_install and up-to-the-minute code updates via version control. If you pick the latter, you can build and install most packages seen in the python package index directly from a version control checkout by running python setup.py install.
If you don't like the default installation directory, you can install to a custom location instead, and export a PYTHONPATH environment variable whose value is the path of the installed package's parent folder.