how can I resolve a folders walking with python? - python

I'm trying to resolve this problem:
Have a tree structure like this:
── geny16
│   ├── gen26xgeny16
│   │   ├── sustrato270
│ │ │ ├── sustrato270_data01.dat
│ │ │ ├── sustrato270_data02.dat
│ │ │ └── sustrato270_data03.dat
│   │   ├── sustrato90
│ │ │ ├── sustrato90_data01.dat
│ │ │ ├── sustrato90_data02.dat
│ │ │ └── sustrato90_data03.dat
│   │   ├── tentata0
│ │ │ ├── tentata0_data01.dat
│ │ │ ├── tentata0_data02.dat
│ │ │ └── tentata0_data03.dat
│   │   └── tenteta90
│ │ │ ├── tentata90_data01.dat
│ │ │ ├── tentata90_data02.dat
│ │ │ └── tentata90_data03.dat
│   └── gen40xgeny16
│   ├── sustrato270
│   ├── sustrato90
│   ├── tenteta0
│   └── tenteta90
└── geny9
├── gen16xgeny9
│   ├── sustrato270
├── sustrato90
│   ├── tenteta0
│   └── tenteta90
├── gen26xgeny9
│   ├── sustrato270
│   ├── sustrato90
│   ├── tenteta0
│   └── tenteta90
└── gen40xgen9y
├── sustrato270
├── sustrato90
├── tenteta0
└── tenteta90
in each deeper folder there are some .dat files that I will manipulate and plot with matplotlib. When I use a python script inside the deeper folder, my work is done: read each data file, normalize a column, superspose and plot data; but as it's the same task for each deep folder, I would like to leave the script in the root folder and that goes through the subfolders to manipulate and plot data.
I can list the name of each element in the tree with this code:
rootDir = '.'
for dirName, subdirList, fileList in os.walk(rootDir):
print('Directorio encontrado: %s' % subdirList)
for fname in fileList:
print('\t%s' % fname)
But it's only a list of names, how can I use this list to navigate and execute my script to manipulate and plot data in each deeper folders?
Thanks in advance for your comments and suggestions.
Gustavo.

You can use pathlib
from pathlib import Path
root_dir = '.' # __file__
d = Path(root_dir).glob('*/*/*/*.dat') #*==any folder/file
for i in d:
print(i.name, i.parent, i.parts[-2])
print(i.resolve())
print(i.stem)
# code to manipulate these files here
# e.g read, visualise and store data
See what you get. You can the do whatever you wish with the files

Related

Python- How to use getText

i am trying to make my application python application multilingual. I wrote the below code
transaltor.py
import gettext
gettext.bindtextdomain('ru', 'locales')
gettext.textdomain('ru')
_ = gettext.gettext
print(_('What kind of trip'))
and here is my project structure.
├── Pipfile.lock
├── __pycache__
│ └── constants.cpython-310.pyc
├── constants.py
├── locales
│ ├── ru
│ │ └── LC_MESSAGES
│ │ ├── django.po
│ │ └── ru.po
│ └── uz
│ └── LC_MESSAGES
│ ├── django.po
│ └── uz.po
└── translator.py
the output for the code is
What kind of trip
and expected output should be Russian translated
Выберите вариант полета
and I referred this
any help would be great. thank you in advance 🙏

Cant import my python modules despite being at the same level in the hierarchy

Im writing a library for internal use,its called "etllib", and I have the following structure:
etl-lib
├── README.md
├── etllib
│ ├── __init__.py
│ ├── client
│ │ ├── __init__.py
│ │ ├── elastic.py
│ │ └── qradar.py
│ ├── etl
│ │ ├── __init__.py
│ │ └── etl_imperva.py
│ └── util
│ ├── __init__.py
│ ├── config.py
│ ├── daemon.py
│ ├── elastic
│ │ ├── __init__.py
│ │ └── impeva_index_config.py
│ └── imperva
│ ├── __init__.py
│ ├── kpe_config.py
│ └── query_config.py
├── scripts
│ └── etl_imperva
└── setup.py
And I have a script called "etl_imperva" in etllib/scripts. The code inside looks like this :
#!/usr/bin/python3
import sys
from etllib.etl.etl_imperva import ETL
# Run with python3 imperva_run.py start|run|stop|restart
ETL.startup(sys.argv)
If I install this package(etllib) and call this script, it works just fine. But when I need to test stuff, how can I tell python to use the modules that are on my working directory instead the ones are installed? Because each time I make a change on the modules, I need to reinstall the package and this is a little time consuming.
I also tried uninstalling the package for testing, but whe I run this script I get the following error :
Exception has occurred: ModuleNotFoundError
No module named 'etllib'
File "/home/jleonse/etl-lib/scripts/run_imperva", line 3, in <module>
from etllib.etl.etl_imperva import ETL
Is there a better way to do this?
Actually, it is not on the same level in the hierarchy.
from etllib.etl.etl_imperva import ETL
would work only if etllib was in the same directory or in a directory in your system PATH, but the etllib is in the parent directory, hence it can not find it.
so you can make it work if you change the project structure to be:
etl-lib
├── README.md
├── etllib
│ ├── __init__.py
│ ├── client
│ │ ├── __init__.py
│ │ ├── elastic.py
│ │ └── qradar.py
│ ├── etl
│ │ ├── __init__.py
│ │ └── etl_imperva.py
│ └── util
│ ├── __init__.py
│ ├── config.py
│ ├── daemon.py
│ ├── elastic
│ │ ├── __init__.py
│ │ └── impeva_index_config.py
│ └── imperva
│ ├── __init__.py
│ ├── kpe_config.py
│ └── query_config.py
├── etl_imperva
│
└── setup.py

subpackages can't be imported after package installed with pip

I have a project called build-your-own-computer with a structure that looks like this:
build-your-own-computer
├── computer
│   ├── arithmetic
│   ├── __init__.py
│   ├── logic
│   └── memory
├── README.md
├── setup.py
├── setup.py~
├── solutions
│   ├── arithmetic
│   │   ├── half_adder.py
│   │   ├── __init__.py
│   │   └── __init__.py~
│   ├── __init__.py
│   ├── __init__.py~
│   ├── logic
│   │   ├── _and.py
│   │   ├── __init__.py
│   │   ├── __init__.py~
│   │   ├── _not.py
│   │   ├── _or.py
│   │   └── xor.py
│   └── memory
│   └── __init__.py
└── tests
├── arithmetic
│   └── test_half_adder.py
├── logic
│   ├── test_and.py
│   ├── test_not.py
│   ├── test_or.py
│   └── test_xor.py
└── memory
My goal is to be able to install this project/package using pip and then to use it from anywhere on my system. I want to import the packages/modules like this:
from byoc.solutions.logic import _and
from byoc.computer.arithmetic import half_adder
As I understand it, build-your-own-computer itself can be considered a package, build-your-own-computer\computer a subpackage, and build-your-own-computer\computer\logic\ a sub-sub-package. All .py files besides init's and setup.py are modules. Is this correct, and does the import scheme above correspond with this project structure?
All of the __init__.py files are empty.
setup.py contains the following:
from setuptools import setup, find_packages
setup(
name='byoc',
packages=find_packages()
)
When I install this using pip and then try importing the submodules, I run into problems:
>>> from byoc.solutions.logic import *
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'byoc.solutions'
I can import byoc without errors.
I'm obviously doing something wrong, but what?
You can check here the suggested layout by setuptools. Also here is a good guide.
Here is my suggestion on your layout:
build-your-own-computer
├── README.md
├── setup.py
├── setup.cfg
├── src
│ └── byoc
│ ├── __init__.py
│ ├── computer
│ │ ├── arithmetic
│ │ ├── __init__.py
│ │ ├── logic
│ │ └── memory
│ └── solutions
│ ├── arithmetic
│ │ ├── half_adder.py
│ │ └── __init__.py
│ ├── __init__.py
│ ├── logic
│ │ ├── _and.py
│ │ ├── __init__.py
│ │ ├── __init__.py~
│ │ ├── _not.py
│ │ ├── _or.py
│ │ └── xor.py
│ └── memory
│ └── __init__.py
└── tests
├── arithmetic
│ └── test_half_adder.py
├── logic
│ ├── test_and.py
│ ├── test_not.py
│ ├── test_or.py
│ └── test_xor.py
└── memory
And then your setup.py should be:
from setuptools import setup, find_packages
setup(
name='byoc',
packages=find_packages('src/')
package_dir={'': 'src/'},
)
This way you should then be able to import it anywhere, e.g.:
from byoc.solutions import logic

How to import a file from a subdirectory in python from a subpackage in another directory?

I have the following directory structure in my home projects folder.
|ALL-IN-ONE
|demo
|__init__.py
|__main__.py
|models
|grpc
|allinone_server.py
And I want to import from allinone_server.py a function defined in main.py called images_demo. I have tried
from demo.__main__ import images_demo
It is not working. How can I import it? The function I am trying to import is located inside main.py which is inside demo directory. I am trying to import it from the file allinone_server.py in grpc. I guess I have made my question clear now.
Here is the whole tree for the project
├── demo
│ ├── __init__.py
│ ├── __main__.py
│ └── __pycache__
│ ├── __init__.cpython-36.pyc
│ └── main.cpython-36.pyc
├── description
├── environment.yml
├── HEAD
├── hooks
│ ├── applypatch-msg.sample
│ ├── commit-msg.sample
│ ├── fsmonitor-watchman.sample
│ ├── post-update.sample
│ ├── pre-applypatch.sample
│ ├── pre-commit.sample
│ ├── prepare-commit-msg.sample
│ ├── pre-push.sample
│ ├── pre-rebase.sample
│ ├── pre-receive.sample
│ └── update.sample
├── imgs
│ └── 44.jpg
├── info
│ └── exclude
├── __init__.py
├── loggers
│ ├── __init__.py
│ └── __pycache__
│ └── __init__.cpython-36.pyc
├── models
│ ├── adience_large1.h5
│ ├── adience_small1.h5
│ ├── AgeModel.json
│ ├── detection_age_gender_large1.h5
│ ├── detection_age_gender_small1.h5
│ ├── detection_age_gender_smile_large1.h5
│ ├── detection_age_gender_smile_small1.h5
│ ├── detection_age_large1.h5
│ ├── detection_age_small1.h5
│ ├── detection_large1.h5
│ ├── detection_small1.h5
│ ├── grpc
│ │ ├── adele_2016.jpg
│ │ ├── allinone_client.py
│ │ ├── all_in_one_pb2_grpc.py
│ │ ├── all_in_one_pb2.py
│ │ ├── all_in_one.proto
│ │ ├── allinone_server.py
│ │ ├── benedict_cumberbatch_2014.png
│ │ ├── cat.png
│ │ ├── classroom_in_tanzania.jpg
│ │ ├── decoded1.py
│ │ ├── decoded.py
│ │ ├── elon_musk_2015.jpg
│ │ ├── laos.jpg
│ │ ├── model_face.jpg
│ │ ├── __pycache__
│ │ │ ├── all_in_one_pb2.cpython-36.pyc
│ │ │ ├── all_in_one_pb2_grpc.cpython-36.pyc
│ │ │ └── decoded.cpython-36.pyc
│ │ ├── sophia.jpg
│ │ ├── test
│ │ │ ├── __init__.py
│ │ │ ├── __pycache__
│ │ │ │ └── __init__.cpython-36.pyc
│ │ │ └── test_images
│ │ │ ├── adele_2016.jpg
│ │ │ ├── benedict_cumberbatch_2014.png
│ │ │ ├── classroom_in_tanzania.jpg
│ │ │ ├── elon_musk_2015.jpg
│ │ │ ├── __init__.py
│ │ │ ├── laos.jpg
│ │ │ ├── model_face.jpg
│ │ │ ├── sophia.jpg
│ │ │ ├── waaah.jpg
│ │ │ ├── woman.jpg
│ │ │ └── zebra_stripes.jpg
│ │ ├── waaah.jpg
│ │ ├── woman.jpg
│ │ └── zebra_stripes.jpg
So you've referred to main.py, but you also have __main__.py in your directory structure. I'll assume that your directory actually contains main.py instead of __main__.py.
To import from levels up in a package, start your import with a period.
To import just one function you would use from .main import images_demo
Now, let's start by saying main.py is in grpc/ along with allinone_server.py, then we'll move it to different directories and see how the import changes.
If it were in grpc/ from .main import images_demo
If it were in models/ from ..main import images_demo
If it were in __ALL-IN-ONE/ from ...main import images_demo
If it were in __demo/ from ...__demo.main import images_demo
Every extra period brings you up one level in the hierarchy, then you use the name of the next level down in the target path until you reach where you want to be.
Now let's suppose you wanted to import the whole of main.py.
If it were in grpc/ from . import main
If it were in models/ from .. import main
If it were in __ALL-IN-One/ from ... import main
If it were in __demo/ from ...__demo import main
Finally, the dot notation to move up a level only works if the file that uses it is in a package, so this will work fine if at the top level you start your program in a scope outside of this package then use from __ALL-IN-ONE.models.grpc import allinone_server
However, if you run allinone_server.py directly then it will fail to import anything above it as it isn't being imported as part of a package. Try that out, and let me know if that needs better explanation.
Good luck!
You can't import a function from another folder directly and for that you have to use this:
import sys
sys.path.insert(0, "../../demo/")
Another step is to rename __main__ to main.
here is the exact example that worked for me:
The tree:
.
├── demo
│   ├── __init__.py
│   ├── main.py
│  
└── models
└── grpc
└── allinone_server.py
main.py:
def images_demo():
print("hello there")
The calling file(allinone_server.py):
import sys
sys.path.insert(0, "../../demo/")
import main
main.images_demo()

add data files to python projects setup.py

I have a project like this:
├── CHANGES.txt
├── LICENSE
├── MANIFEST.in
...
├── docs
│   └── index.rst
├── negar
│   ├── Negar.py
│   ├── Virastar.py
│   ├── Virastar.pyc
│   ├── __init__.py
│   ├── data
│   │   ├── __init__.py
│   │   └── untouchable.dat
│   ├── gui.py
│   ├── gui.pyc
│   ├── i18n
│   │   ├── fa_IR.qm
│   │   └── fa_IR.ts
│   └── negar.pro
├── setup.py
...
and inside that my file Virastar.py need some data from data.untouchable.dat. it works fine until I install the project with this setup.py:
setup(
...
include_package_data=True,
packages = find_packages() + ['negar'],
package_dir={'negar': 'negar'},
package_data={'negar': ['data/*.dat']},
entry_points={
'console_scripts': [
'negar = negar.Negar:main',
],
},
...
)
after that when I start my program and when it needs that data file it return this error:
IOError: [Errno 2] No such file or directory: 'data/untochable.dat'
even in my egg-info sources I can't find any data file:
...
negar/Negar.py
negar/Virastar.py
negar/__init__.py
negar/gui.py
negar/data/__init__.py
have I missed something here?
Thank you all.
EDIT: Do I have to add any special thing in init.py?
and I have to add this: I used untouchable.dat just like this:
f = codecs.open('data/untouchable.dat', encoding="utf-8")
I used data_files
data_files = [('', ['negar/data/untouchable.dat'])],
The first problem is that I didn't import my data file into the package with MANIFEST.in file. I imported it like this:
include negar/data/*.dat
After that my data file already imported with my package install. but because I had mistakes in open my data files, python couldn't find it. this question helped me to find the right way
Python Access Data in Package Subdirectory and now I use something like this:
import os
this_dir, this_filename = os.path.split(__file__)
DATA_PATH = os.path.join(this_dir, "data", "data.txt")
print open(DATA_PATH).read()
Maybe try:
package_data={'negar/data': ['data/*.dat']},
A solution that does not require any of:
MANIFEST.in
include_package_data=True
package_dir={}
and neither the __init__.py file in the folder!
Having the project like:
├── CHANGES.txt
├── LICENSE
...
├── negar
│ ├── Negar.py
│ ├── Virastar.py
│ ├── Virastar.pyc
│ ├── __init__.py
│ ├── data
│ │ ├── __init__.py
│ │ └── untouchable.dat
│ ├── subfolder
│ │ ├── __init__.py
│ │ ├── data_NOT_included
│ │ │ └── garbage.toml
│ │ └── data_with_no_init_file
│ │ ├── config.ini
│ │ └── options.yml
│ ├── gui.py
│ ├── gui.pyc
│ ├── i18n
│ │ ├── fa_IR.qm
│ │ └── fa_IR.ts
│ └── negar.pro
├── setup.py
...
The setup file:
setup(
...
packages = find_packages()
package_data= {
# all .dat files at any package depth
'': ['**/*.dat'],
# into the data folder (being into a module) but w/o the init file
'negar.subfolder': [ '**/*.ini', '**/*.yml', ]
},
entry_points={
'console_scripts': [
'negar = negar.Negar:main',
],
},
...
)
This worked great in my case and I avoided maintaining an additional MANIFEST.in file.
caveat: the "negar.subfolder" has to be a python module.
To the access the file:
from importlib.resources import files
config = files('negar.subfolder').joinpath('data_with_no_init_file').joinpath('config.ini').read_text()
from accessing-data-files-at-runtime

Categories