We've had the case where we released a Python source distribution (sdist) file on PyPI, and there was an issue that was only noticed after making the release. Namely for some files, the permissions were set to not allow read by all, and then in some cases those files couldn't be read on user machines after they installed the package with sudo.
We're considering starting our own script to check an sdist for this issue and to run it before uploading to PyPI.
Is there a tool (or something in setuptools or twine or somewhere) to run this or other basic checks on an sdist before uploading it available already?
Related
Situation
I am writing a python package to check whether data comply with a certain standard. The standard is provided by a third party and frequently updated. The standard is available as several json files. The package itself should be installed in multi-user HPC environments as well as on local single-user systems. The python package should not be updated too frequently (particularly not updated just to update the json files for the standard) due to its usage in the HPC environment. I do not want to download the json files on each call of the package.
Current implementation
Originally, I planned to ship the json standard files with the package (subfolder of package installation directory; license of third-party allows this) and update them on the call of the program when the files are older than X days. During development installations, the updating process worked well. However, when installed productively, the package does not seem to have writing permissions in its own installation folder. This is probably a security feature.
Question
Is it possible to gain writing permissions for specific subfolders of the package's installation directory? If not, is there a best practice how to deals with this situation (e.g. downloading the files in the user's home directory)? I don't want to newly download the files on each program call.
This a question about setuptools and creating a python distribution for a web application.
Context
It is a django project and we distribute it via a private pypi, to ourselves, not external users (yet). Currently we run python setup.py sdist to create the distribution and we upload that to our pypi. I suspect it is just a copy of the source code as a tar.gz. We checkout the source to do development. We don't install for dev via pip (perhaps we should?).
Recently the project has started using nodejs. Which now means that now we need to do a "build" to create new files which are not part of the source code, but do need to be deployed. The build process requires a bunch of node packages which are not a necessary, or desired, part of the final deployment to a webserver.
Reading through the packaging documentation it describes sdist, bdist_wheel, and also develop targets for setup.py, but I'm not clear on how to apply these to our situation.
Question 1
When we pull from our pypi, what should we be pulling?
the pre-built deployable version of code for the webserver, without source code?, or
code on which we run something else which builds and arranges things as they need to be? If so, what would the appropriate setup.py target be (sdist, bdist_wheel, a new custom target)?
Question 1 (put another way)
When, and how, should we run the build?
To 'build' the app, we run nodejs-javascript scripts eg: npm install, and less x y x, or webpack -p, or python, scripts which convert the source files into deployable files. We don't want these things on the web server.
Discussion
So, generally speaking, as you can tell, first we're a bit confused about when to convert the source files into distribution files. And second, how to actually implement that: subprocess calls to shell scripts from a subclassed setuptools.command?!?
We'd prefer not to push the finished product to pypi because then all the bits and bobs we do to create the build are not captured and automated in a neat setuptools centric way. So when, where, and how do we incorporate the build process into our setuptools distribution?
* Note: adding more confusion about what/where we're also using tox
I'm building a package on Python 2.7.6 32bit Windows 32
The only definitive source of some components of a package is an svn 'share'. The common practice in this company is to include that into your project the using svn:externals.
The normal way to build this package is:
python setup.py bdist_wheel
All appears normal on my workstation (where I checked out the code with TortoiseSVN), but when I run the same process on Jenkins the bdist_wheel process does not include any .py file that was sourced via svn:externals.
After reading through the documentation, this appears to be because of a feature which identifies which scripts are part of the package based on which files are tracked by SVN. It appears that as a consequence of how Jenkins checks out the files, the bdist_wheel sees that I'm using SVN and assumes that it knows how to determine which files are tracked, but gets the answer wrong.
What I need is a way to stop the bdist_wheel command from trying to guess which files I care about (I actually want every .py file in the project to be included, regardless of how it's been brought in)
I tried tried specifying the files I needed using a MANIFEST.in file, but it did not work.
recursive-include externals *.py
In this example, 'externals' is a top-level directory in my source-tree which contains an init.py file and a bunch of svn:external'd directories. Only the init file can be seen in the built whl file.
Unfortunately this makes the .py files behave as if they were data, in the log I can see this:
copying build\lib\externals\security\credentials.py -> build\bdist.win32\wheel\foopackage-0.0.4.data\..\externals\security
That's obviously not a real solution!
Pip, Virtualenv and all relevant tools are at the latest stable versions.
It turns out that this problem is caused by Jenkins using a very old SVN standard (1.4) for it's own repositories. Switching to 1.7 corrects this behavior.
In this case, the application consists of one or more Python files, plus a settings.ini file. Now the Python files when being installed need to be installed in ~/.hg (as default) or prompted where the user want them installed. The installation also requires text to be appended to files like hgrc.
Is there already a specific Python package that does all of this, or if anyone has any experience in this area please share.
As far as I have looked, Python packaging refers to setuptools and easy_install.
The basis for packaging is a setup.py file. A problem with this is that such a setup file is used for a couple of dissimilar tasks:
Generating documentation.
Creating a release (source/binary).
Actually installing the software.
Combining these tasks in one file is a bit of a hazard and leads to problems now and then.
or distutils, but I am not sure if these packages support the notion of user prompting and deployment like appending text to existing files, and creating new ones.
I would include a custom script (bin/ command) which will poke the users' .hgrc and others. Doing it without the user consent would be rude.
User story
Install package: easy_install pkgname and this deploys myproject-init-hg (UNIX executable, can be also written in Python)
The installation finished and tells the user to run commmand myproject-init-hg
setup.py include mechanism to distribute and deploy bin/ style scripts.
In Trac on Admin -> Plugins there is an option to install Plug-ins. Now this option expect you to upload an Python egg.
This would be all well but for the fact that all the Trac plug-ins I found are either plain .py files or zip files and are incompatible with the upload function (I tried it).
This leaves my with a bunch of questions:
Are there any Trac plug-ins which come as an Python egg?
What is an (Trac compatible) Python egg?
Is it difficult to repackage an .py file into a Trac compatible Python egg?
If not: how is it done?
Haven't used trac for a year, but what I remember is that most plugins are available trough subversion and already packed as an egg (which is kind of an installer in the python world, but I am not very familiar with the concept).
Most plugins are available at http://trac-hacks.org/ and the easiest way to install a plugin is
easy_install http://svn.domain.tdl/path/to/plugin/
the folder should contain a setup.py and a setup.cfg file.
easy_install checks the files out from svn and installs the plugin. You can find details here: http://trac.edgewall.org/wiki/TracPlugins
If the plugin makes database changes you have to call
trac-admin upgrade
from console.
http://trac.edgewall.org/wiki/TracAdmin
If I remember right, the install through the webinterface installs the plugin locally (for the instance) while easy_install installs it globally (for all running trac sites) and is the more common way to install a plugin.
Hint: After every plugin install you have to restart trac
Hint2: Most plugins don't tell you how to install and only give a link to the root of their svn. You only have to browse the svn folder and locate the folder containing the setup.py.
The rest is done with easy_install.
Example:
Plugin: http://trac-hacks.org/wiki/GoogleChartPlugin
Wiki pages tells you:
You can check out GoogleChartPlugin from here using Subversion, or browse the source with Trac.
where here links to http://trac-hacks.org/svn/googlechartplugin/
The svn contains two versions. Browse to http://trac-hacks.org/svn/googlechartplugin/0.11/trunk/ and copy the path.
Then do
easy_install http://trac-hacks.org/svn/googlechartplugin/0.11/trunk/
Answers to your questions in order.
Python eggs are binary packages which contain the code for the application and some metadata. They're not very different from debs or rpms in this sense. The egg itself is basically just a zip file which contains all the above mentioned files with specific names and layouts. For more information on eggs (the format and how to create them), please refer to http://www.ibm.com/developerworks/library/l-cppeak3.html. It's probably a little dated since the future (and present) of python packaging is a little hazy.
A trac plugin is a python program that uses the Trac plugin API to extend the functionality of trac. It can be packaged as an egg.
If your package is properly laid out and contains a setuptools/distribute setup.py file, then issuing the command python setup.py bdist_egg will create a .egg file for you. For details on this please refer to this(a little dated but complete) and this (more upto date but still in progress). The Trac Growl plugin mentions this on it's documentation page.
Please see above point.