We Keep Coding

Tell the module requirements in a particular directory (package)

In my office we have a quite complex directory structure when it comes to our code.
One of the things we have is a libs module to drop "common" things used by other parts of our big application (or set of applications... that are all living under a common directory).
The code in that libs/ directory requires certain packages installed in order for it to work. In said libs/ directory we have a requirements.txt file that supposedly lists the dependencies required for the things (things being Python code) in it to work. We have been filling that requirements.txt file pretty manually, tracking that "if this .py file uses this module, we should add it to the requirements file" so it's almost certain that by now we have forgotten adding some required modules.
Because of the complex structure we have (some parts use pipenv, some other have their own requirements.txt...) is very hard knowing whether a required module is going to end up installed or not.
So I would like to make sure that this libs/ directory (cough, cough... module ) has all its dependencies listed in its libs/requirements.txt.
Is that possible? Ideally it'd be "run this command passing /libs/ as an argument, it'll scan the directory and tell you what packages are needed by the py(s) found in it"
Thank you in advance.

Unfortunately, python does not know whether its dependencies are satisfied until runtime. requirements.txt is just a helper file for pip and similar tools, and you have to update it manually.
That said, you could
use the os module to recursively get a list of all *.py files in the folder
parse each one of them for lines having the format import aaa.bbb or from aaa import bbb
keep a set of the imports
However, even in that case, the name of the imported module is not the same as the name you need to pass to pip (eg, import yaml requires pyyaml in requirements.txt), but at least it could be a hint of what's missing.

How to find out what a python package does

EDIT I was being stupid. Just type help('package_name'.'pyb_name') which worked.
I would like to find out what is actually in a python package I have locally downloaded and installed with pip.
Typing help(package_name) just lists NAME, FILE (where the init.py is) and PACKAGE CONTENTS which is just one .pyd file.
I can't open the .pyd file to check what's inside(tbh not all that familiar with .pyds). These two with a 159byte init.pyc are the only files in the package.
I need to use this (not widely available) package for some university work.
Thanks.

You can't know what a python package does unless it is stated in its docs (on PyPI or in the repository) or without reading the code. A Python package can be anything that has a setup.py and either a single module or multiple files under a folder with a __init__.py file in it.
The fact that the __init__.py is empty doesn't mean anything other than the fact that its existence means there's a python package involved.
Any specific package you want to know about, you should look up for documentation or read the code to get a sense of its purpose.

setting path in a python module for distribution

I have module that I am writing in python that needs to download data and store it in a particular directory. Currently, I am doing this by in the manner shown below,
import os
folder = 'd:\data' #location of the root folder directory on my system
DATAPATH = os.path.join(folder, 'download_data')
This works for my module on my system. I am interested in distributing this module to other machines and I am not sure how I can control the location of the root folder when I install the module to a different machine. Are there any best practices on how to do this? Is there some way to do this in the setup file?

Yes, it should be set by your installer and reside in a configuration file. Use module configparser to extract the value. Look at BasicInterpolation to see your need as an example.

From the sounds of things, you need to install your package with a known structure, including some non-Python files, then need to get the location of those files by using their known relative path from your package modules. In which case, you need to combine these two answers
Including non-Python files with setup.py
Retrieving python module path
And then use your existing logic to construct your full path

How to distribute / access data files in Python egg?

I'm writing a Django application that is using pip & virtualenv to manage its development environment.
One of the dependencies, pkgme, comes with many data files which are its "backends" and are configured in its setup.py with data_files=$FOO (rather than package_data).
When pkgme looks for its backends, it looks in os.path.join(sys.prefix, "share", "pkgme", "backends"). This works great when pkgme has been installed normally, and seems to match the documentation but does not work when pkgme is installed as an egg.
There, the data files are installed under $VIRTUAL_ENV/lib/python2.7/site-packages/pkgme-0.1-py2.7.egg/share rather than the expected $VIRTUAL_ENV/share.
Which leaves me with two questions:
Should I be using something other than the os.path.join above to find the data files regardless of whether we are using an egg installation or a traditional system installation? If so, what?
Should I be distributing my data files differently so as to make them more readily available in an egg?
Note that I know about pkgutil.get_data, but would rather not use it. I'm not interested in the contents of these data files, I want to know their location instead, so I can execute them.
My current plan is to do this:
Use package_data instead of data_files
Change pkgme to look for backends relative to pkgme.__file__ rather than sys.prefix

Your current plan is essentially correct, or is at any rate a workable option.
When setuptools creates an egg, it checks whether code in the egg makes use of __file__, and if so, it marks the egg as not being installable in compressed form. In this way, when the egg is installed by easy_install, it'll get extracted to an .egg/ directory instead of being left in an .egg file.
If you want to support compressed/drop-in installation (i.e., just dumping the egg in a directory without "installing" it), then you should use the pkg_resources.resource_filename() (docs here) API instead of __file__, but then your package will be dependent on setuptools or distribute in order to have that API available.

I ended up doing the following:
Changing pkgme to use pkg_resources.resource_filename() to find its own included backends
Added an entry point that any backend written in Python can use to publish the location of its own backend scripts
Kept the sys.prefix-based check for any backend that don't want to use Python
The diff can be found here: http://bazaar.launchpad.net/~pkgme-committers/pkgme/trunk/revision/86

Problem accessing config files within a Python egg

I have a Python project that has the following structure:
package1
class.py
class2.py
...
package2
otherClass.py
otherClass2.py
...
config
dev_settings.ini
prod_settings.ini
I wrote a setup.py file that converts this into an egg with the same file structure. (When I examine it using a zip program the structure seems identical.) The funny thing is, when I run the Python code from my IDE it works fine and can access the config files; but when I try to run it from a different Python script using the egg, it can't seem to find the config files in the egg. If I put the config files into a directory relative to the calling Python script (external to the egg), it works - but that sort of defeats the purpose of having a self-contained egg that has all the functionality of the program and can be called from anywhere. I can use any classes/modules and run any functions from the egg as long as they don't use the config files... but if they do, the egg can't find them and so the functions don't work.
Any help would be really appreciated! We're kind of new to the egg thing here and don't really know where to start.

The problem is, the config files are not files anymore - they're packaged within the egg. It's not easy to find the answer in the docs, but it is there. From the setuptools developer's guide:
Typically, existing programs manipulate a package's __file__ attribute in order to find the location of data files. However, this manipulation isn't compatible with PEP 302-based import hooks, including importing from zip files and Python Eggs.
To access them, you need to follow the instructions for the Resource Management API.
In my own code, I had this problem with a logging configuration file. I used the API successfully like this:
from pkg_resources import resource_stream
_log_config_file = 'logging.conf'
_log_config_location = resource_stream(__name__, _log_config_file)
logging.config.fileConfig(_log_config_location)
_log = logging.getLogger('package.module')

See Setuptools' discussion of accessing pacakged data files at runtime. You have to get at your configuration file a different way if you want the script to work inside an egg. Also, for that to work, you may need to make your config directory a Python package by tossing in an empty __init__.py file.