We Keep Coding

project directory structure python data

I have a python project with the following directory structure
|-- package_name/
| |-- __init__.py
| |-- lambda_function.py
| |-- ...
|
|-- database/
| |-- data.json
And I'm using data.json file on my package to initialize some objects.
I would like to split my current data file on multiple json files because there are too much information group on one single file
Something like :
|-- database/
| |-- A/
| | |-- a.json
| | |-- b.json
| |-- B/
| | |-- c.json
| | |-- C/
| | |-- a.json
| | |-- b.json
I don't want to make a real database because data will not move, there are just here to initialize my objects.
Question :
How can I do to have multiple json files to clarify the code, but a simple way to use them as if it was a single one.
I would like to know the best practices on how manage that kind of data on a python projects.

Import a file from another directory

I have a file call entryPoint.py :
from .commonLib.deviceLib import *
And I have a file called deviceLib.py :
import math
import sys
import logging
import requests
import this
class DeviceLib(object):
def __init__(self, connectionDb):
self.__db = connectionDb
The tree is like this :
/test
entryPoint.py
/commonLib
__init__.py
deviceLib.py
When I execute python entryPoint.py I get the error : Attempted relative import in non-package. Please help me.

use sys.path.append to append the directory where your python file (module is). E.g if your entryPoint.py is inside address directory
import sys
sys.path.append('/path/to/your/module/address/')
import entryPoint

There should be __init__.py in the folder both /test and /commonLib reside.
then just do
from commonLib import deviceLib
For example
sound
|-- effects
| |-- echo.py
| |-- __init__.py
| |-- reverse.py
| `-- surround.py
|-- filters
| |-- equalizer.py
| |-- __init__.py
| |-- karaoke.py
| `-- vocoder.py
|-- formats
| |-- aiffread.py
| |-- aiffwrite.py
| |-- auread.py
| |-- auwrite.py
| |-- __init__.py
| |-- wavread.py
| `-- wavwrite.py
`-- __init__.py
lets assume you are right now opened wavread.py in format subdirecory, you can import karaoke.py from filters by just
from filters import karaoke
More information Here,
https://www.python-course.eu/python3_packages.php

To import a file from another directory you can use this code :
import sys
sys.path.insert(0, 'folder destination')
import file
As you can see here we included the path so python will look for the file in that path as well.

How to document Matlab project with multiple classes using sphinxcontrib-matlabdomain?

I tried to document a larger Matlab project using Sphinx and the sphinxcontrib-matlabdomain package. It works fine for a single folder that stores all .m files. However my project contains multiple classes stored in separate folders such as:
project
|-- matfiles
| |-- #class1
| | |-- class1.m
| | |-- method1.m
| | |-- method2.m
| | +-- method1.m
| |-- #class2
| | |-- class2.m
| | |-- methodA.m
| | |-- methodB.m
| | +-- methodC.m
| +-- function
| |-- fun1.m
| |-- fun2.m
| +-- fun3.m
+-- doc
|-- conf.py
+-- index.rst
I adde the following lines to conf.py:
matlab_src_dir = os.path.abspath('..')
extensions = [
'sphinx.ext.autodoc',
'sphinxcontrib.matlab',
]
primary_domain = 'mat'
And I added the following lines to index.rst:
.. module:: matfiles
.. automethod:: class1.method1
I got the following error:
Exception occurred:
File "/Library/Python/2.7/site-packages/sphinxcontrib/mat_documenters.py", line 788, in import_object
if self.object.attrs.get('Static'):
AttributeError: 'NoneType' object has no attribute 'attrs'
The full traceback has been saved in /var/folders/70/g0lr37wn60gbbymnsks_t5vm0000gn/T/sphinx-err-uD8qpe.log, if you want to report the issue to the developers.
Please also report this if it was a user error, so that a better error message can be provided next time.
A bug report can be filed in the tracker at <https://github.com/sphinx-doc/sphinx/issues>. Thanks!
| reading sources... [100%] index
So my question is, is there a way to document multi class Matlab projects where class methods are saved in a folder with name #class?

I believe this issue has been resolved in the latest version, sphinxcontrib-matlabdomain-0.2.7, which autodocuments class methods with attributes defined in separate files of a #classfolders. Let me know if it's still an issue.

Import Error in flask application

I have an application. Below is o/p of tree command -
app
|-- main
| |-- lib
| | |-- constants.py
| | |-- helper.py
| | `-- __init__.py
| `-- src
| |-- __init__.py
| `-- web.py
web.py
from flask import Flask, request
app = Flask(__name__)
from lib.helper import endpoints
.....
Some code
.....
if __name__ == '__main__':
app.run('0.0.0.0', 5433, debug=True)
I am getting this error
ImportError: No module named lib.helper.
where am I doing wrong?

from flask import Flask, request
app = Flask(__name__)
import sys
from os.path import abspath, dirname
sys.path.insert(0, dirname(dirname(abspath(__file__))))
from lib.helper import endpoints
.....
Some code
.....
if __name__ == '__main__':
app.run('0.0.0.0', 5433, debug=True)

Lib module is outside of src folder, need to go up one folder up and use that model.
from ..lib.helper

Or else a fully qualified namespace as
from app.main.lib

Python: How to take tar using tarfile module by excluding the 3rd directory

I have the directory structure like this
-- jenkins
|-- jobs
| |-- dir1
| | |-- build_dir--
| | `-- Autoyes
| |-- dir2
| | |-- build_dir--
| | `-- Manyes
| |-- dir3
| | |-- ArtFict
| | `-- build_dir--
| `-- hi
|-- tesst
`-- tets
I need to take a tar using tarfile module by excluding the "build_dir" under all dir1,dir2,dir3 and more directories.
I can use a very simple tar statement into subprocess to achieve this
tar --exclude="*/*/buid_dir" -czvf test.tgz jenkins
But I'm looking for a pythonic way to achieve this, so far what I have done is not excluding the "build_dir" directory.
import tarfile
import os,sys
import subprocess
def ProcesS (input):
Asub = subprocess.Popen (input,shell=True,stdout=subprocess.PIPE,stderr=subprocess.PIPE)
output,error = Asub.communicate()
return output,error
A = ['jenkins', 'jenkins/jobs', 'jenkins/jobs/dir1', 'jenkins/jobs/dir1/build_dir', 'jenkins/jobs/dir2', 'jenkins/jobs/dir2/build_dir', 'jenkins/jobs/dir3', 'jenkins/jobs/dir3/build_dir']
Tar = tarfile.open('Backme.tgz','w:gz')
#Tar.add(A,exclude='*/*/build_dir') #This throws the error
Tar.add(A) # creates a tar file without excluding the build_dir
Tar.close()
It would be much helpful if someone finds the answer for this.

from the documentation
TarFile.add(name, arcname=None, recursive=True, exclude=None, filter=None)
it looks like you would just add it to excludes (you may have to put the full path ...)
it looks like you do
tf = tarfile.open("sometar.tar","w")
def excludes_fn(name):
return "build" in name
tf.add("/base/folder",exclude=excludes_fn)