I'm writing a Python script that will be used by multiple people on different OS's and I'm trying to figure out the best way to include a third-party package so that users don't have to install the package themselves before running the script. The particular library I'm using is the Requests package. It works great for me when I install it via pip, but when I download the source and try to import it from my project root, it doesn't work. I either get "Attempted relative import in non-package" or "No module named MyProject.request" depending on how I try to import it.
So I suppose my first question is, am I even approaching this right? I've never written a python script for more users than just myself, so I've never been concerned with issues like package installation. Is the correct approach to include the source for the Requests library and import it relative to my project, or is there an easier approach?
As a follow-up question, if I am on the right track, what's the proper procedure for including this package in my project? Right now my file structure looks like this:
MyProject/
__init__.py
main.py
requests/
requests/
__init__.py
[bunch of other stuff]
[other files/folders]
In main.py I've tried from .MyProject.requests import requests and from .requests import requests, and both give me "Attempted relative import in non-package." I haven't messed around with sys.path yet, but I've seen mixed advice as to whether that's the "right" way to do it. My primary concern is getting this script to run on user machines without them having to do anything besides run it. I don't want users to have to install an external package, but I can't seem to get it working by trying to import the source.
Per Evert's response, my problems seems to have been the multiple "requests" directories. When I moved the sub-directory up and eliminated the superfluous files/directories from the Requests source it worked perfectly.
Related
Basically, I can only reference my other files as modules when they are in a very specific location:
C:\Users\Dave\Desktop\Programming\Python.
If I want to create a new folder for a large project with multiple modules, say
C:\Users\Dave\Desktop\Programming\Python\Project1,
I can no longer import any modules and keep getting a ModuleNotFoundError. I've looked into it and it seems I need to add that folder to the Python Path somehow, but I couldn't find any answers on how to do it. My computer runs on Windows 10 if that matters.
I think the immediate solution to your problem/the answer to your question would be to use sys.path.append() at the top of your script.
import sys
sys.path.append("<ABSOLUTE/PATH/TO/YOUR/CUSTOM/MODULES/FOLDER>")
import custom_module
This isn't an ideal solution for any kind of prod use, however. Based on what you wrote in your question, this may not be what you're looking for. More info might help to craft a more stable solution:
Are you using virtual environments when running python? Do you just run python script.py or is there a specific version of python you're using that's installed elsewhere than the common C:\Program Files\Python?
When you say that you work on a project with multiple modules, does that mean there are custom modules that you/someone wrote or is it just that that project uses non-standard library modules, i.e. you had to pip install them?
Can we get an example of the code you're running and the folder structure of your project/where the modules are that you need?
This issue has been driving me insane for the past few days.
So basically, I'm trying to port over a Pure Python project to a proper PyCharm project. This is to basically improve code quality and project structure.
I wish it was as simple as basically creating a virtualenv to house everything, but it isn't. This project will eventually be developed simultaneously by multiple developers with Git as source control, and the default libraries will be modified. I presume this means that the libraries should ideally be tracked by Git in the end. Virtualenv shouldn't help here as far as I know because it's not portable between systems (or at least that's still being tested).
This project will also be, in the future, deployed to a Centos server.
So the only plan I can think of to successfully pull off this would be to simply bring in all of the libraries (which was done using pip install -t Libraries <ExampleLibrary>) into a single folder, with a __init__.py inside, and use them from other python files as a package within the Pycharm project.
Is this possible / recommended? I tried various methods to reference these libraries, but they all don't work during runtime. Somehow when the files in the library import something else from their own package, an ImportError is raised saying that there's no such module.
Will accept any other suggestions too.
Using Pycharm Community Edition.
EDIT: After having a good night's rest I think the crux of the issue is really just project organization. Before I ported it over to Pycharm the project worked as expected, but this had all of the python files in the root directory, and the libraries in a subfolder of the root, with every project file having the same boilerplate code:
import os, sys
absFilePath = os.path.dirname(os.path.abspath(__file__));
sys.path.insert(1, absFilePath + "/lib")
I was hoping that by using Pycharm to help me flesh out the packages, I could avoid having repeated boilerplate code.
Note: Not full solution.
The addition of the template code below forces the file containing the code to be in the same directory as the libs folder.
For Pycharm, all I had to do was mark the libs folder as a source folder. Even with the addition of the template code to the file, the modified libraries still work as expected.
For the Python Shell, this template code is still needed:
import os
import sys
absFilePath = os.path.dirname(os.path.abspath(__file__))
sys.path.insert(1, absFilePath + "/lib")
I'm just learning about modules in python 3.5. While I can usually install and import packages using sudo pip install {package}, I can't seem to figure out how to import my own files.
I made a test.py file with a single definition to test. I saved it to the site-packages folder. I can't seem to import from there. I need help understanding how to import files.
I read online about possibly using sys.path however, I don't know how that works.
If I had the following file structure:
/home/foo
/home/foo/__init__.py
/home/foo/bar.py
and I wanted to
import foo.bar
foo.bar.somefn()
I would first need to do:
import sys
sys.path.append("/home/foo")
Notice the init.py file, which tells python to look for submodules. We don't necessarily need it in this instance, but it's good practice to have:
What is __init__.py for?
However, since this can get repetitive, daunting and tough to track, there are lots of tools available to help you setup as your code expands to have dependencies and lots of files/folders. I suggest you read a bit about pip/disttools/easy-install and how to make a package with a 'setup.py'.
What is setup.py?
In addition, you might want to explore the world of Virtual Environments and deployment solutions such as Buildout and Anaconda. These help keep the code separated as you have several projects on the go with different needs, or even different python builds and platform requirements. When it comes time to deploy your code to other machines, you'll have a much easier time replicating the needs of the project.
Below is a series of articles with info explaining everything from packages, pip and setup.py to starting up your work environment with Buildout. Seems like a great series:
http://reinout.vanrees.org/weblog/tags/softwarereleasesseries.html
The official docs for making a setup.py:
https://docs.python.org/3/distutils/setupscript.html
I'm fairly new to python and especially it's import mechanism. I'm not entirely sure i'm using the terminology correctly so i should apologize for that up front.
firstly, this seems to be a problem i'm having with a 3rd party import so i can't really change the structure of their release.
In the release, all of the packages are in site-packages/[ROOTFOL]/[PACKAGE]
the [ROOTFOL] does not have a __init__.py file, only the package folders have this file.
this folder is placed into site-packages and the site-packages is present in my PYTHONPATH
in the examples they provide, they use it like this:
import ROOTFOL.PACKAGE.WhateverObject as obj
I'm trying to avoid adding every single package to the PYTHONPATH as there are a bunch of them. Everything seems to work fine, however it really inhibits my ability to work with the auto-complete functionality and that is the frustrating part.
Something else i find strange, is that when the packages are installed, there is a EGG-INFO folder placed along side the package. In this there are several .txt files and one of which is namespace_packages.txt which has only the ROOTFOL. Is there some way i should be setting this to PyDev?
So, what you're seeing here is their distribution model. Usually a module will have one root import that everything stems from, but that's not necessarily the case. They're providing a package with (what I assume) is many modules that don't interact with each other; or they can all stand alone.
instead of importing each package individually, you could use the 'from' keyword:
from ROOTFOL.PACKAGE import *
which will grab everything inside that sub-module. You could e-mail the developer and ask why they deployed it this way...or you could add your own __init__.py to the root folder and,
from ROOTFOL import *
which will walk the tree. Good luck!
I have a module that I want to keep up to date, and I'm wondering if this is a bad idea:
Have a module (mod1.py) in the
site-packages directory that copies a
different module from some other
location into the site-packages
directory, and then imports * from
that module.
import shutil
from distutils.sysconfig import get_python_lib
p_source = r'\\SourceSafeServer\mod1_current.py'
p_local = get_python_lib() + r'\mod1_current.py'
shutil.copyfile(p_source, p_local)
from mod1_current import *
Now I can do this in any module, and it will always be the latest version:
from mod1 import function1
This works.... but is there a better way of doing this?
Update
Here is the current process... there is a project under source-control that has a single module: mod1.py There is also a setup.py Running setup.py copies mod1.py to the site-packages directory.
Developers that use the module must run setup.py to update the module. Sometimes, they don't and not having the latest version causes problems.
I want to be able to just check-in the a new version, and any code that imports that module will automatically grab the latest version every time, without anyone having to run setup.py
Do you really want to do this? This means you could very easily roll code to a production app simply by committing to source control. I would consider this a nasty side-effect for someone who isn't aware of your setup.
That being said this seems like a pretty good solution - you may want to add some exception-handling around the network file calls as those are prone to failure.
In some cases, we put .pth files in the Python site-packages directory. The .pth files name our various SVN checkout directories.
No install. No copy.
.pth files are described here.
The original strategy of having other developers copy mod1.py into their site-packages in order to use the module sounds like it's the real problem. Why aren't they just using the same source control are you are?
This auto-copying will make it hard to do rollbacks, especially if other developers copy your strategy. Imagine this same system used for dozens and dozens of files. And then imagine you actually do want to use a version of mod1.py that is not the latest for something.