We Keep Coding

Python: Import scrypt from subfolder to another subfolder

I am working on some python project (2.7) and I have issue with imports. When I run main.py it start scripts from tests folder etc. and save output to logs and everything works fine.
/root------------
-logs
-staticCfg
-config.py
-tests
-systemTest
-scrypt1.py
-scrypt2.py
-userTest
-uScrypt1.py
main.py
My static variables (email, name etc.) are located in config.py. I need to import config.py in scrypt1.py or scrypt2.py. I tryed adding __init__.py to tests, systemTest and staticCfg folder but I always get an error.
In my scrypt1.py:
import staticCfg as cfg
...
or
from staticCfg import *
...
I get the error:
ImportError: No module named staticCfg

The import mechanism of Python can be a bit tricky.
You can refer to the documentation for more information: Python Import Mechanism
When you use absolute imports (your import does not start with a .) as you do, the import path will start from your main script (the one you launch). In your case, it's scrypt1.py. So starting from this location, python can't find the package staticCfg.
For me, the simplest solution is to create a main script in your root directory and call scrypt1.py from there (imported using from tests.systemTet import scrypt1.py). In this case, the base package will be your root folder and you will have access to the package staticCfg from all your script files as you wanted to do.

you may add root folder to PYTHONPATH.

How to properly use python modules

I created a test file called test.py (contained a class named test) and saved it into documents(Mac OS X 10.9.3)
I then attempted to use this file by writing from test import test. However, I got an error telling me that there was no module named test. Please can you shed some light into this issue.

The problem is that the Documents folder isn't usually in the PATH or PYTHONPATH, so the interpreter can't see scripts that are stored there (unless Documents happens to be the current working directory).
I can think of two solutions:
Move the file test.py to somewhere within Python's path.
You can find your PYTHONPATH with this script:
import os
try:
user_paths = os.environ['PYTHONPATH'].split(os.pathsep)
except KeyError:
user_paths = []
(From Mark Ransom’s answer to How do I find out my python path using python?)
You can find the system PATH with
import sys
print sys.path
Both of these return a list of folders. If you put test.py in one of those folders, then it will be seen by the interpreter.
For example, I usually install custom modules at
/Library/Python/2.7/site-packages
and then I can import them as normal.
Manually append Documents to your path.
You can use sys.path.append to add a directory to the path.
If you only need to use test.py in a handful of scripts, then you can add this code to each script you want to use test.py in:
import os
import sys
sys.path.append(os.path.join(os.environ['HOME'], 'Documents'))
from test import test

Python Import Module

Recently started a new Python project.
I am resolving a import module error where I am trying to import modules from the same directory.
I was following the solutions here but my situation is slightly different and as a result my script cannot run.
My project directory is as follows:
dir-parent
->dir-child-1
->dir-child-2
->dir-child-3
->__init__.py (to let python now that I can import modules from here)
->module1
->module2
->module3
->module4
->main.py
In my main.py script I am importing these module in the same directory as follows:
from dir-parent.module1 import class1
When I run the script using this method it throws a import error saying that there is no module named dir-parent.module1 (which is wrong because it exists).
I then change the import statement to:
from module1 import class1
and this seemed to resolve the error, however, the code I am working on has been in use for over 2.5 years and it has always imported modules via this method, plus in the code it refers to the dir-parent directory.
I was just wondering if there is something I am missing or need to do to resolve this without changing these import statements and legacy code?
EDIT: I am using PyCharm and am running off PyCharm

If you want to keep the code unchanged, I think you will have to add dir-parent to PYTHONPATH. For exemple, add the following on top of your main.py :
import os, sys
parent_dir = os.path.abspath(os.path.dirname(__file__)) # get parent_dir path
sys.path.append(parent_dir)

Python's import and pathing are a pain. This is what I do for modules that have a main. I don't know if pythonic at all.
# Add the parent directory to the path
CURRENTDIR = os.path.dirname(os.path.dirname(os.path.realpath(__file__)))
if CURRENTDIR not in sys.path:
sys.path.append(CURRENTDIR)

Scrapy: Importing a package from the project that's not in the same directory

I'm trying to import a package from my project which is not in the same directory as scrapy is in. The directory structure for my project is as follows:
Main
__init__.py
/XPaths
__init.py
XPaths.py
/scrapper
scrapy.cfg
/scrapper
__init.py
settings.py
items.py
pipelines.py
/spiders
myspider.py
I'm trying to access xpaths.py from within myspider.py. Here are my attempts:
1) from Main.XPaths.XPaths import XPathsHandler
2) from XPaths.XPaths import XPathsHandler
3) from ..Xpaths.XPaths import XPathsHandler
These failed with the error:
ImportError: No module named .......
My last attempt was:
4) from ...Xpaths.XPaths import XPathsHandler
Which also failed with the error:
ValueError: Attempted relative import beyond toplevel package
What am I doing wrong? XPaths is independent from Scrapy, therefore the file structure has to stay that way.
//EDIT
After some further debugging following #alecxe comment, I tried adding the path to main inside the sys.path, and print it before importing xpaths. The weird thing is, the scrapper directory gets appended to the path when I run scrapy. Here's what I added:
'C:\\Users\\LaptOmer\\Code\\Python\\PythonBackend\\Main'
And here's what I get when I print sys.path:
'C:\\Users\\LaptOmer\\Code\\Python\\PythonBackend\\Main\\scrapper'
Why does scrapy append that to the path?

I know its a little bit messy solution but only one I could find when I had same problem as you. Before including files from your project you need to manually append the system path to your top most package level, i.e:
sys.path.append(os.path.join(os.path.dirname(__file__), '../..'))
from XPaths.XPaths import XPathsHandler
...
From what I understand scrappy creates its own package - this is why you cannot import files from other directories. This also explains error:
ValueError: Attempted relative import beyond toplevel package

I ran into the same problem.
When I used:
sys.path.append(os.path.join(os.path.dirname(__file__), '../..'))
it appended ../.. to the last file path, which didn't work. I noticed my main file was the last item in the sys.path list. I took that last item and went to the module level to find my main file -- which contains a function called "extract_notes".
import scrapy
import sys
import os
mod_path = os.path.dirname(os.path.normpath(sys.path[-1]))
sys.path.insert(0,mod_path)
from pprint import pprint as p
from main import extract_notes
Hope that helps.

I have a similar directory structure, with multiple scrapers (say directories scraper1 and scraper2).
Since I found the sys.path changes as suggested by #ErdraugPl too brittle (see #ethanenglish's problems), especially since Scrapy itself is modifying the sys.path, I chose an OS solution instead of a Python solution: I created a symbolic link to directory /XPaths in both scraper1 and scraper2. That way, I can still maintain a single XPaths module that I can use in both scraper1 and scraper2, and can simply do from XPaths.XPaths import XPathsHandler

Unable to import Python package universally

Suppose I have the following directory structure:
workspace/
__init__.py
ys_manage/
__init__.py
manage.py
ys_utils/
__init__.py
project_dicts.py
Now, suppose I need access to project_dicts.py in manage.py. Also, my $PATH includes /home/rico/workspace/ys_manage.
I need to be able to run manage.py from any directory on my machine and still be able to access project_dicts.py.
My $PYTHONPATH only has /home/rico/workspace.
If I include the following in manage.py I can run the file from ~/workspace/ys_manage but not anywhere else.
import sys
sys.path.append('..')
from ys_utils import project_dicts
It appears that the '..' gives a relative path to where the directory where the file is run, not where the file is located. Is this correct?
I wanted to try and use ys_manage/__init__.py to import project_dicts.py so that it would be available in manage.py universally. Is this a good idea?
I've never used __init__.py for anything other than a "package creator". That is, I've never used it for initialization purposes. Perhaps I'm doing it wrong.
Contents of ys_manage/__init__.py:
import sys
sys.path.append('..')
from ys_utils import project_dicts
Should I include something in manage.py to look for this import?
When I try and run manage.py I get the following error:
NameError: global name 'project_dicts' is not defined
As a secondary question, do I need to have workspace/__init__.py? I'd really rather not have it because ys_manage and ys_utils (and about a dozen other packages) are all under revision control and used by several other developers...workspace is not.

Generally, I've found trying to use relative paths for imports dangerous and very error prone. I'd suggest just putting workspace on your PYTHONPATH (or adding it programatically in __init__.py) and importing everything relative to that static location. It will make your code more easily readable too, as you'll be able to track down where imports are coming from much more quickly and clearly.

Try this instead of sys.path.append('..'):
sys.path.append(os.path.join(os.path.dirname(__file__), '..'))
(you'll need to import both sys and os).