We Keep Coding

Python is throwing error for a file path inside the Project folder where it is not supposed to and seems to be a python glitch

Python is throwing some inconsistent error for a file reference inside the Project folder based on 'Working Directory' in the script configuration which is very weird
My Project Structure as follows
config_utils.py
f = ''
def config_init():
global f
txt_dir = '../files/sample.txt'
f = open(txt_dir, "r")
f = f.read()
mycode.py
import config_ru.config_utils as cu
cu.config_init()
print(cu.f)
On executing mycode.py, it throws the below error w.r.t "sample.txt" in "files" package
but if I change the Working directory of "my_code.py" in the script configuration from "level2" to "level1", mycode.py gets executed successfully
This is very weird because in both the cases the location of "sample.txt" remains unchanged and both the error and being forced to change the Working Directory seems to be unacceptable. Please clarify

The work-around is to get the path of the module you are in and apply the relative path of the resource file to that:
from pathlib import Path
f = ''
def config_init():
global f
p = Path(__file__).parent.absolute()
txt_dir = (p / '../files/sample.txt').resolve()
f = open(txt_dir, "r")
f = f.read()

Looks like normal behavior. In the line
txt_dir = '../files/sample.txt'
the .. means 'go one directory up'. So, when you are in level2, it will go up one level (to level1) and look for files/sample.txt, which does not exist. However, when you are in level1, then the .. will bring you to the pythonProject dir, where it can find files/sample.txt

How to load resources file without giving entire path in Python

I want to read the properties file in Python without giving the entire path. Because if my code is deployed somewhere else then my package would fail if I pass the hardcore value. So below is my code to read the properties file by giving the entire path:
import configparser
config = configparser.RawConfigParser()
config.read('C:\\Users\\s\\IdeaProjects\\PysparkETL\\src\\main\\resources\\configs.properties')
dbname = config['DB']['dbname']
username=config['DB']['user']
password=config['DB']['password']
table=config['DB']['tablename']
driver=config['DB']['driver']
print(dbname)
and below is my configs.properties file:
[DB]
dbname=ourdb
user=root
password=root
tablename=loadtable
driver=com.mysql.cj.jdbc.Driver
I tried different ways like ConfigParser instead of RawConfigParser but didn't work. Is there any way to load files from the classpath?
Also, I tried different ways from this link but it didn't help. All I need is a path to pass it to config.read method but it should not be hardcoded as I did it in the code.
Below is my project structure:
Also, as suggested I tried below code and passed the URL to the config.read method but it's not working.
props = os.path.join(
os.path.dirname("resources"), # folder where the python file is
'src/main/resources/configs.properties'
)
config.read(props)
I get below error:
raise KeyError(key)
KeyError: 'DB'

if the file will always be in the same place relative to your python file, you can do the following:
props = os.path.join(
os.path.dirname(__name__), # folder where the python file is
'relative/path/to/configs.properties'
)

How to read a file in a module by its relative path to the module?

The directory structure:
├--- mod
| ├--- __init__.py
| └--- abc.data
└--- test.py
__init__.py:
with open("abc.data", 'r') as f:
pass # read and process the data
test.py:
import mod
The program above is expected to read the data in the file abc.data, but it gives an error instead:
FileNotFoundError: [Errno 2] No such file or directory: 'abc.data'
And this is because the current directory of Python interpreter is the parent directory of test.py.
So how to read abc.data in the module mod regardless of the location of test.py?
Actually the following code works:
__init__.py:
import os
filepath = os.path.join(os.path.dirname(os.path.realpath(__file__)), "abc.data")
with open(filepath, 'r') as f:
pass # read and process the data
But this solution is a bit dirty especially when there are many files to be read in __init__.py. Is there a more elegant solution?

I believe that's as good as it gets. Even larger libraries use the same method:
# Extracted From selenium/webdriver/firefox/firefox_profile.py
# ...
if not FirefoxProfile.DEFAULT_PREFERENCES:
with open(os.path.join(os.path.dirname(__file__),
WEBDRIVER_PREFERENCES)) as default_prefs:
FirefoxProfile.DEFAULT_PREFERENCES = json.load(default_prefs)
# ...
Another example:
# Extracted from pipenv/vendor/yaspin/spinners.py
# ...
THIS_DIR = os.path.dirname(os.path.realpath(__file__))
SPINNERS_PATH = os.path.join(THIS_DIR, "data/spinners.json")
# ...
If it was an importable object (e.g. .py files) then you can use . conventions to indicate the relative path.

Reading a file using a relative path in a Python project

Say I have a Python project that is structured as follows:
project
/data
test.csv
/package
__init__.py
module.py
main.py
__init__.py:
from .module import test
module.py:
import csv
with open("..data/test.csv") as f:
test = [line for line in csv.reader(f)]
main.py:
import package
print(package.test)
When I run main.py I get the following error:
C:\Users\Patrick\Desktop\project>python main.py
Traceback (most recent call last):
File "main.py", line 1, in <module>
import package
File "C:\Users\Patrick\Desktop\project\package\__init__.py", line 1, in <module>
from .module import test
File "C:\Users\Patrick\Desktop\project\package\module.py", line 3, in <module>
with open("../data/test.csv") as f:
FileNotFoundError: [Errno 2] No such file or directory: '../data/test.csv'
However, if I run module.py from the package directory, I don’t get any errors. So it seems that the relative path used in open(...) is only relative to where the originating file is being run from (i.e __name__ == "__main__")? I don't want to use absolute paths. What are some ways to deal with this?

Relative paths are relative to current working directory.
If you do not want your path to be relative, it must be absolute.
But there is an often used trick to build an absolute path from current script: use its __file__ special attribute:
from pathlib import Path
path = Path(__file__).parent / "../data/test.csv"
with path.open() as f:
test = list(csv.reader(f))
This requires python 3.4+ (for the pathlib module).
If you still need to support older versions, you can get the same result with:
import csv
import os.path
my_path = os.path.abspath(os.path.dirname(__file__))
path = os.path.join(my_path, "../data/test.csv")
with open(path) as f:
test = list(csv.reader(f))
[2020 edit: python3.4+ should now be the norm, so I moved the pathlib version inspired by jpyams' comment first]

For Python 3.4+:
import csv
from pathlib import Path
base_path = Path(__file__).parent
file_path = (base_path / "../data/test.csv").resolve()
with open(file_path) as f:
test = [line for line in csv.reader(f)]

This worked for me.
with open('data/test.csv') as f:

My Python version is Python 3.5.2 and the solution proposed in the accepted answer didn't work for me. I've still were given an error
FileNotFoundError: [Errno 2] No such file or directory
when I was running my_script.py from the terminal. Although it worked fine when I run it through Run/Debug Configurations from the PyCharm IDE (PyCharm 2018.3.2 (Community Edition)).
Solution:
instead of using:
my_path = os.path.abspath(os.path.dirname(__file__)) + some_rel_dir_path
as suggested in the accepted answer, I used:
my_path = os.path.abspath(os.path.dirname(os.path.abspath(__file__))) + some_rel_dir_path
Explanation:
Changing os.path.dirname(__file__) to os.path.dirname(os.path.abspath(__file__))
solves the following problem:
When we run our script like that: python3 my_script.py
the __file__ variable has a just a string value of "my_script.py" without path leading to that particular script. That is why method dirname(__file__) returns an empty string "". That is also the reason why my_path = os.path.abspath(os.path.dirname(__file__)) + some_rel_dir_path is actually the same thing as my_path = some_rel_dir_path. Consequently FileNotFoundError: [Errno 2] No such file or directory is given when trying to use open method because there is no directory like "some_rel_dir_path".
Running script from PyCharm IDE Running/Debug Configurations worked because it runs a command python3 /full/path/to/my_script.py (where "/full/path/to" is specified by us in "Working directory" variable in Run/Debug Configurations) instead of justpython3 my_script.py like it is done when we run it from the terminal.

Try
with open(f"{os.path.dirname(sys.argv[0])}/data/test.csv", newline='') as f:

I was surprised when the following code worked.
import os
for file in os.listdir("../FutureBookList"):
if file.endswith(".adoc"):
filename, file_extension = os.path.splitext(file)
print(filename)
print(file_extension)
continue
else:
continue
So, I checked the documentation and it says:
Changed in version 3.6: Accepts a path-like object.
path-like object:
An object representing a file system path. A path-like object is
either a str or...
I did a little more digging and the following also works:
with open("../FutureBookList/file.txt") as file:
data = file.read()

How can I load a configuration file based on the relative path from the script being executed?

StackOverflow,
Below is the first few lines of my script:
from ConfigParser import SafeConfigParser
from docopt import docopt
import core as scrappy
ARGS = docopt(__doc__, version=scrappy.__version__)
if not ARGS['PATH']:
ARGS['PATH'] = './'
# load config file
CFG = SafeConfigParser()
if not CFG.read(ARGS['--cfg']): # call to CFG.read also loads file if it exists
raise IOError('Configuration file not found.')
The configuration file I'm trying to read is in the same directory as the above script. By default, docopt sets the path to this file to ./file.conf (I have tested this with file.conf with identical results).
The last line of the script is always called, suggesting that the file can't be found. I confirmed this by printing the output of os.getcwd, which revealed that the script's execution directory is whichever directory the terminal was pointing to.
What gives?
What can I do to point to the configuration file?

Use __file__ predefined module attribute. Like this:
module_dir = os.path.dirname(__file__)
CFG = SafeConfigParser()
cfg_full_path = os.path.join(module_dir, ARGS['--cfg'])
if not CFG.read(cfg_full_path):
...