Want to create python source distribution by running python setup.py sdist from a directory outside of the one I want to package up. Can't seem to find a way to do this. I have a script that generates a setup.py and MANIFEST.in dynamically, and I'd like to tell python to use those files to create an sdist of the source in a different directory "over there".
What I'm doing is creating a script that lets a user create an sdist w/o any setup.py etc. They just say "package up this directory and everything under it". So I generate a setup.py and MANIFEST.in (with recursive-include * to grab all files) in a python tempfile.mkdtemp directory (in an unrelated file path like /tmp/whatever) that I can clean up afterwards...but I can't seem to use those to package their directory. I don't want to create those files in their source dir.
You can use setuptools's, --dist-dir=DIR / -d DIR option to specify where the otherwise default dist/-folder is made. In other words, this changes the output directory.
E.g.:
python setup.py sdist -d /tmp/whatever
If you are using distutils.core: Instead of using from distutils.core import setup you can use from setuptools import setup.
In order to define where the source directories come from, I think you can add the directory to sys.path and then setup() will discover the content files automatically:
import sys
from os import path
# ...
# Add other folders to sys.path
sys.path.append('/tmp/whatever')
sys.path.append(path.join(path.abspath('..'), 'some', 'folder'))
Sort of a hack, but this worked for me. Right before running setup(), use os.chdir() to change the directory to that of the base path where setup.py would normally run. To specify where the distribution packages go, I use the arguments to setup.py, specifically:
python setup.py sdist --formats=gztar -d 'directory_for_the_distribution' egg_info --egg-base 'directory_for_the_egg_info'
Thus you can run setuptools from a directory other than at the base of the package directories and the distribution and temp egg directories go wherever you want.
Related
I have this git repo structure:
.gitignore
JSONs/subdirA/some.json
JSONs/subdirB/other.json
MyPackage/__init__.py
MyPackage/myModule.py
How do I properly pack the JSONs folder into MyPackage/JSONs, without moving it there permanently (mostly because customers use this git repo directly for non-python usage, and the folder at the top of the repo is easy/intuitive... But now I also want to release this same dir into my PyPi package)?
I've tried adding it to the MANIFEST.in and then playing with data_files in setup.py as well as package_data... But to no avail. Maybe some .pyc or cached build files got the best of me... But I haven't figured it out from all the other (not quite) duplicate questions as they don't specifically call out their directory structure and desired final location.
I've tried resorting to os.walk and shutil.copy before the call to setup and then deleting that directory after setup... While it seems to work locally, when pushing to our local devpi package server, something goes wrong. Is my goal totally off the radar for setuptools/pip ideals??? Or am I just missing some key understanding? Please enlighten me!
Something like the following could help:
First we need to make sure that the json files are added to the source distribution.
MANIFEST.in:
recursive-include JSONs *.json
Then in the actual setup script, the list of packages has to be modified on the fly to take into account the target package structure.
setup.py:
#!/usr/bin/env python3
import setuptools
PACKAGES = (
setuptools.find_packages(exclude=['JSONs*'])
+
[
f'MyPackage.{package}'
for package
in setuptools.find_namespace_packages(include=['JSONs*'])
]
)
setuptools.setup(
packages=PACKAGES,
package_dir={
'MyPackage.JSONs': 'JSONs',
},
include_package_data=True,
#
name='Something',
version='1.2.3',
)
JSONs/subdirA/some.json :
{"Marco": "Polo"}
Such package data can be read like so:
MyPackage/myModule.py:
import pkgutil
print(pkgutil.get_data('MyPackage', 'JSONs/subdirA/some.json').decode())
And use it like in the following:
$ python -m pip install .
$ # Move to another directory to prevent that the current working directory
$ # ... overshadows the installed project
$ cd ..
$ python -m MyPackage.myModule
{"Marco": "Polo"}
I have created a custom python package following this guide, so I have the following structure:
mypackage/ <-- VCS root
mypackage/
submodule1/
submodule2/
setup.py
And setup.py contains exactly the same information as in the guide:
from setuptools import setup, find_packages
setup(name='mypackage',
version='0.1',
description='desc',
url='vcs_url',
author='Hodossy, Szabolcs',
author_email='myemail#example.com',
license='MIT',
packages=find_packages(),
install_requires=[
# deps
],
zip_safe=False)
I have noticed if I go into the folder where setup.py is, and then call python setup.py install in a virtual environment, in site-packages the following structure is installed:
.../site-packages/mypackage-0.1-py3.6.egg/mypackage/
submodule1/
submodule2/
but if I call it from one folder up like python mypackage/setup.py install, then the structure is the following:
.../site-packages/mypackage-0.1-py3.6.egg/mypackage/
mypackage/
submodule1/
submodule2/
This later one ruins all imports from my module, as the path is different for the submodules.
Could you explain what is happening here and how to prevent that kind of behaviour?
This is experienced with Python 3.6 on both Windows and Linux.
Your setup.py does not contain any paths, but seems to only find the files via find_packages. So of course it depends from where you run it. The setup.py isn't strictly tied to its location. Of course you could do things like chdir to the basename of the setup file path in sys.argv[0], but that's rather ugly.
The question is, WHY do you want to build it that way? It looks more like you would want a structure like
mypackage-source
mypackage
submodule1
submodule2
setup.py
And then execute setup.py from the work directory. If you want to be able to run it from anywhere, the better workaround would be to put a shellscript next to it, like
#!/bin/sh
cd ``basename $0``
python setup.py $#
which separates the task of changing to the right directory (here I assume the directory with setup.py in the workdir) from running setup.py
I have a file structure like this:
project_folder/
notebooks/
notebook01.ipynb
notebook02.ipynb
...
notebookXY.ipynb
module01.py
module02.py
module03.py
In .ipynb files inside notebook/ folder I want to import classes and functions from module01.py, module02.py and module03.py.
I have found answer in this question that it is possible using following lines of code inside every notebook and run those lines as first cell every time:
import os
import sys
module_path = os.path.abspath(os.path.join('..'))
if module_path not in sys.path:
sys.path.append(module_path)
Is there please a better way for this? What if I have A LOT of .ipynb files inside notebooks/ folder, do I have to paste those lines of code at the beginning of every single one? Is there a better, more minimalist or cleaner way?
Try adding the project_folder to your PYTHONPATH environment variable. This will allow you to tell python to search that directory for imports.
You would do this in your user profile settings, or in your startup script - not in python. It's something that has to be set before python ever gets run.
Another solution is to move all your Python modules (.py files) into a folder and make them an installable package. If you pip install it into your current environment, you can then import the package into any notebook in that environment, regardless of folder structure.
So in your situation you could have:
project_folder/
notebooks/
notebook01.ipynb
notebook02.ipynb
...
notebookXY.ipynb
my_package/
__init__.py
module01.py
module02.py
module03.py
setup.py
__init__.py can just be an empty file, and tells Python "everything in this folder is part of a package"
For an explanation of what goes in setup.py see here.
A basic setup.py can be as simple as this:
import setuptools
setuptools.setup(
name="my_package",
version="0.0.1",
description="A small example package",
packages=setuptools.find_packages(),
python_requires='>=3.7',
)
Install it:
cd project_folder
pip install [-e] .
Including the optional -e flag will install my_package in "editable" mode, meaning that instead of copying the files into your virtual environment, a symlink will be created to the files where they are.
Now in any notebook you can do:
import my_package
Or
from my_package.module01 import <some object>
I am trying to create the setup.py for a script. The setup.py looks like this:
from setuptools import setup
setup(name='readfile',
version='1.0',
description='A flexible module to read ascii files',
author='author',
author_email='author#gmail.com',
url='',
modules=['readfile'])
the script is called readfile.py and doesn't have any additional files or scripts. All I want to do is have setup.py put it on the correct location automatically. So I do
python setup.py install
This creates a folder
/Library/Python/2.7/site-packages/readfile-1.0-py2.7.egg
However, it doesn't update the python path nor create a directory called readfile in site-packages directly, so I cannot import the module. Am I missing an option that would do any of these?
Thanks!
EDIT:
I had some more trouble afterwards; with various combinations of setup() arguments I was able to import readfile but none of its modules. In the end, the setup.py was fine and I just needed the __init__.py, without the extra folder. So the three files (readfile.py, setup.py, __init__py) are in the same folder now and everything works as it should.
Do you have a folder at the same level as setup.py called readfile?
Inside that, is there a file called __init__.py?
Inside that folder is there also your source file called readfile.py?
That seems to work for me.
The .egg file that you reference is just a zip file that contains everything you need. There should be no need to update python path or create the readfile directory. Those things are not needed to import readfile.
I'm trying to build a Python distribution with distutils. Unfortunately, my directory structure looks like this:
/code
/mypackage
__init__.py
file1.py
file2.py
/subpackage
__init__.py
/build
setup.py
Here's my setup.py file:
from distutils.core import setup
setup(
name = 'MyPackage',
description = 'This is my package',
packages = ['mypackage', 'mypackage.subpackage'],
package_dir = { 'mypackage' : '../mypackage' },
version = '1',
url = 'http://www.mypackage.org/',
author = 'Me',
author_email = 'me#here.com',
)
When I run python setup.py sdist it correctly generates the manifest file, but doesn't include my source files in the distribution. Apparently, it creates a directory to contain the source files (i.e. mypackage1) then copies each of the source files to mypackage1/../mypackage which puts them outside of the distribution.
How can I correct this, without forcing my directory structure to conform to what distutils expects?
What directory structure do you want inside of the distribution archive file? The same as your existing structure?
You could package everything one directory higher (code in your example) with this modified setup.py:
from distutils.core import setup
setup(
name = 'MyPackage',
description = 'This is my package',
packages = ['mypackage', 'mypackage.subpackage'],
version = '1',
url = 'http://www.mypackage.org/',
author = 'Me',
author_email = 'me#here.com',
script_name = './build/setup.py',
data_files = ['./build/setup.py']
)
You'd run this (in the code directory):
python build/setup.py sdist
Or, if you want to keep dist inside of build:
python build/setup.py sdist --dist-dir build/dist
I like the directory structure you're trying for. I've never thought setup.py was special enough to warrant being in the root code folder. But like it or not, I think that's where users of your distribution will expect it to be. So it's no surprise that you have to trick distutils to do something else. The data_files parameter is a hack to get your setup.py into the distribution in the same place you've located it.
Have it change to the parent directory first, perhaps?
import os
os.chdir(os.pardir)
from distutils.core import setup
etc.
Or if you might be running it from anywhere (this is overkill, but...):
import os.path
my_path = os.path.abspath(__file__)
os.chdir(os.normpath(os.path.join(my_path, os.pardir)))
etc. Not sure this works, but should be easy to try.
Run setup.py from the root folder of the project
In your case, place setup.py in code/
code/ should also include:
LICENSE.txt
README.txt
INSTALL.txt
TODO.txt
CHANGELOG.txt
The when you run "setup.py sdist' it should auto-gen a MANIFEST including:
- any files specified in py_modules and/or packages
- setup.py
- README.txt
To add more files just hand-edit the MANIFEST file to include whatever other files your project needs.
For a somewhat decent explanation of this read this.
To see a working example checkout my project.
Note: I don't put the MANIFEST under version control so you won't find it there.
A sorta lame workaround but I'd probably just use a Makefile that rsynced ./mypackage to ./build/mypackage and then use the usual distutils syntax from inside ./build. Fact is, distutils expects to unpack setup.py into the root of the sdist and have code under there, so you're going to have a devil of time convincing it to do otherwise.
You can always nuke the copy when you make clean so you don't have to mess up your vcs.
Also a lame workaround, but a junction/link of the package directory inside of the build project should work.