the below is my main_call.py file
from flask import Flask, jsonify, request
from test_invoke.invoke import end_invoke
from config import config
app = Flask(__name__)
#app.route("/get/posts", methods=["GET"])
def load_data():
res = "True"
# setting a Host url
host_url = config()["url"]
# getting request parameter and validating it
generate_schedule= end_invoke(host_url)
if generate_schedule == 200:
return jsonify({"status_code": 200, "message": "success"})
elif generate_schedule == 400:
return jsonify(
{"error": "Invalid ", "status_code": 400}
)
if __name__ == "__main__":
app.run(debug=True)
invoke.py
import requests
import json
import urllib
from urllib import request, parse
from config import config
from flask import request
def end_invoke(schedule_url):
headers = {
"Content-Type":"application/json",
}
schedule_data = requests.get(schedule_url, headers=headers)
if not schedule_data.status_code // 100 == 2:
error = schedule_data.json()["error"]
print(error)
return 400
else:
success = schedule_data.json()
return 200
tree structure
test_invoke
├── __init__.py
├── __pycache__
│ ├── config.cpython-38.pyc
│ └── invoke.cpython-38.pyc
├── config.py
├── env.yaml
├── invoke.py
└── main_call.py
However when i run, i get the no module found error
python3 main_call.py
Traceback (most recent call last):
File "main_call.py", line 3, in <module>
from test_invoke.invoke import end_invoke
ModuleNotFoundError: No module named 'test_invoke'
Python looks for packages and modules in its Python path. It searches (in that order):
the current directory (which may not be the path of the current Python module...)
the content of the PYTHONPATH environment variable
various (implementation and system dependant) system paths
As test_invoke is indeed a package, nothing is a priori bad in using it at the root for its modules provided it is accessible from the Python path.
But IMHO, it is always a bad idea to directly start a python module that resides inside a package. Better to make the package accessible and then use relative imports inside the package:
rename main_call.py to __main__.py
replace the offending import line with from .invoke import end_invoke
start the package as python -m test_invoke either for the directory containing test_invoke or after adding that directory to the PYTHONPATH environment variable
That way, the import will work even if you start your program from a different current directory.
You are trying to import file available in the current directory.
So, please replace line
from test_invoke.invoke import end_invoke with from invoke import end_invoke
Related
I have the following code structure:
Graph API/
│── main.py
├── helper_functions/
├── defines.py
├── insights.py
insights.py imports 2 functions from defines.py at the beginning:
from defines import getCreds, makeApiCall
It then uses "makeApiCall" for this function:
def getUserMedia( params ) :
// Some code about url endpoints etc.
return makeApiCall( url, endpointParams, params['debug'] ) # make the api call
I want to use the getUserMedia function in the main.py script, so I import it with:
from helper_functions.insights import *
But I get the error:
Traceback (most recent call last):
File "/Users/Graph_API/main.py", line 1, in <module>
import helper_functions.insights
File "/Users/Graph_API/helper_functions/insights.py", line 1, in <module>
from defines import getCreds, makeApiCall
ModuleNotFoundError: No module named 'defines'
What leads to this error? When I use the getUserMedia function within insights.py it works fine. I already tried importing defines.py to main.py as well, but I get the same error.
I am pretty new to programming, so I would really appreciate your help :)
You should replace
from defines import getCreds, makeApiCall
With
from helper_functions.defines import getCreds, makeApiCall
Or
from .defines import getCreds, makeApiCall
You must give a path either from project directory as in the first example, or from the relative path from the insights file by adding a dot.
You could also consider adding init.py to the helper_functions folder. Checkout here: What is __init__.py for?
Consider the following package structure:
.
├── module
│ ├── __init__.py
│ └── submodule
│ ├── attribute.py
│ ├── data.txt
│ └── __init__.py
└── test.py
and the following piece of code:
import pkgutil
data = pkgutil.get_data('module.submodule', 'data.txt')
import module.submodule.attribute
retval = module.submodule.attribute.hello()
Running this will raise the error:
Traceback (most recent call last):
File "test.py", line 7, in <module>
retval = module.submodule.attribute.hello()
AttributeError: module 'module' has no attribute 'submodule'
However, if you run the following:
import pkgutil
import module.submodule.attribute
data = pkgutil.get_data('module.submodule', 'data.txt')
retval = module.submodule.attribute.hello()
or
import pkgutil
import module.submodule.attribute
retval = module.submodule.attribute.hello()
it works fine.
Why does running pkgutil.get_data disrupt the future import?
First of all, this was a great question and a great opportunity to learn something new about python's import system. So let's dig in!
If we look at the implementation of pkgutil.get_data we see something like this:
def get_data(package, resource):
spec = importlib.util.find_spec(package)
if spec is None:
return None
loader = spec.loader
if loader is None or not hasattr(loader, 'get_data'):
return None
# XXX needs test
mod = (sys.modules.get(package) or
importlib._bootstrap._load(spec))
if mod is None or not hasattr(mod, '__file__'):
return None
# Modify the resource name to be compatible with the loader.get_data
# signature - an os.path format "filename" starting with the dirname of
# the package's __file__
parts = resource.split('/')
parts.insert(0, os.path.dirname(mod.__file__))
resource_name = os.path.join(*parts)
return loader.get_data(resource_name)
And the answer to your question is in this part of the code:
mod = (sys.modules.get(package) or
importlib._bootstrap._load(spec))
It looks at the already loaded packages and if the package we're looking for (module.submodule in this example) exists it uses it and if not, then tries to load the package using importlib._bootstrap._load.
So let's look at the implementation of importlib._bootstrap._load to see what's going on.
def _load(spec):
"""Return a new module object, loaded by the spec's loader.
The module is not added to its parent.
If a module is already in sys.modules, that existing module gets
clobbered.
"""
with _ModuleLockManager(spec.name):
return _load_unlocked(spec)
Well, There's right there! The doc says "The module is not added to its parent."
It means the submodule module is loaded but it's not added to the module module. So when we try to access the submodule via module there's no connection, hence the AtrributeError.
It makes sense for the get_data method to use this function as it just wants some other file in the package and there is no need to import the whole package and add it to its parent and its parents' parent and so on.
to see it yourself I suggest using a debugger and setting some breakpoints. Then you can see what happens step by step along the way.
I am sorry if the title takes some time to understand. So here is the folder structure:
falcon_tut/
falcon_tut/
app.py
images.py
__init__.py
tests/
test_app.py
__init__.py
And some codes
####################
# app.py
####################
from images import Resource
images = Resource()
api = application = falcon.API()
api.add_route('/images', images)
# ... few more codes
####################
# test_app.py
####################
import falcon
from falcon import testing
import ujson
import pytest
from falcon_tut.app import api
#pytest.fixture
def client():
return testing.TestClient(api)
def test_list_images(client):
doc = {
'images': [
{
'href': '/images/1eaf6ef1-7f2d-4ecc-a8d5-6e8adba7cc0e.png'
}
]
}
response = client.simulate_get('/images')
result_doc = ujson.loads(response.content)
assert result_doc == doc
assert response.status == falcon.HTTP_OK
It works fine when running with python falcon_tut/app.py and curl it with response of 200 and payload of the images
Until running pytest tests/ from the project root it output this:
ImportError while importing test module ../falcon_tut/tests/test_app.py
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests/test_app.py:6: in <module>
from falcon_tut.app import api
E ModuleNotFoundError: No module named 'falcon_tut'
I tried creating __init__.py at the project root but it still output the same error above
Python version 3.7.0, with falcon 1.4.1, cpython 0.28.5, pytest 3.7.3, and instead of using gunicorn, I am using bjoern 2.2.2
I am trying out the python falcon framework and encounter the error at the testing part.
==========UPDATE===========
The reason why pytest could not found the module is because sys.path does not have ../falcon_tut/falcon_tut exists.
When I ran pytest and edit those 2 files and printing out sys.path, it only has [../falcon_tut/tests, ../falcon_tut, ..]. The workaround to this is to append the path to the package to sys.path. So here is the edited app.py
#############
# app.py
#############
import sys
# this line is just example, please rewrite this properly if you wants to use this workaround
# sys_path[1] only applied to my situation, again this is just example to know that it works
# the idea is to make sure the path to your module exists in sys.path
# in this case, I appended ../falcon_tut/falcon_tut to sys.path
# so that now ../falcon_tut/falcon_tut/images.py can be found by pytest
sys.path.insert(0, '{}/falcon_tut'.format(sys_path[1]))
# body codes...
I have a basic parser app I'm building in Python. I monitors a folder and imports files when they are dropped there. I have a MongoDB that I'm trying to save the imports to. There's almost nothing to it. The problem happens when I try to include one of my class/mongo-document files. I'm sure it's a simple syntax issue I don't understand. I have all my requirements installed, and I'm running this in an virtual env. This is my first python app though, so it's likely something I'm not seeing.
My file structure is
application.py
requirements.txt
__init__.py
-services
parser.py
__init__.py
-models
hl7message.py
__init__.py
Here is application.py
from mongoengine import connect
import os, os.path, time
from services import parser
db = connect('testdb')
dr = 'C:\\Imports\\Processed'
def processimports():
while True:
files = os.listdir(dr)
print(str(len(files)) + ' files found')
for f in files:
msg = open(dr + '\\' + f).read().replace('\n', '\r')
parser.parse_message(msg)
print('waiting')
time.sleep(10)
processimports()
requirements.txt
mongoengine
hl7
parser.py
import hl7
from models import hl7message
def parse_message(message):
m = hl7.parse(str(message))
h = hl7message()
hl7message.py
from utilities import common
from application import db
import mongoengine
class Hl7message(db.Document):
message_type = db.StringField(db_field="m_typ")
created = db.IntField(db_field="cr")
message = db.StringField(db_field="m")
If I don't include the hl7message class in the parser.py it runs fine, but as soon as I include it I get the error, so I'm sure it has something to do with that file. The error message though isn't to helpful. I don't know if I've got myself into some kind of include loop or something.
Sorry, stack trace is below
Traceback (most recent call last):
File "C:/OneDrive/Dev/3/Importer/application.py", line 3, in <module>
from services import parser
File "C:\OneDrive\Dev\3\Importer\services\parser.py", line 2, in <module>
from models import hl7message
File "C:\OneDrive\Dev\3\Importer\models\hl7message.py", line 2, in <module>
from application import db
File "C:\OneDrive\Dev\3\Importer\application.py", line 23, in <module>
processimports()
File "C:\OneDrive\Dev\3\Importer\application.py", line 17, in processimports
parser.parse_message(msg)
AttributeError: module 'services.parser' has no attribute 'parse_message'
This is a circular import issue. Application.py imports parser, which imports h17 which imports h17message, which imports application which runs processimports before the whole code of the parser module has been run.
It seems to me that service modules should not import application. You could create a new module common.py containing the line db = connect('testdb') and import db from common both in application.py and in h17message.
I have several 'app'-modules (which are being started by a main-application)
and a utility module with some functionality:
my_utility/
├── __init__.py
└── __main__.py
apps/
├── app1/
│ ├── __init__.py
│ └── __main__.py
├── app2/
│ ├── __init__.py
│ └── __main__.py
...
main_app.py
The apps are being started like this (by the main application):
python3 -m <app-name>
I need to provide some meta information (tied to the module) about each app which is readable by the main_app and the apps themselves:
apps/app1/__init__.py:
meta_info = {'min_platform_version': '1.0',
'logger_name': 'mm1'}
... and use it like this:
apps/app1/__main__.py:
from my_utility import handle_meta_info
# does something with meta_info (checking, etc.)
handle_meta_info()
main_app.py:
mod = importlib.import_module('app1')
meta_inf = getattr(mod, 'meta_info')
do_something(meta_inf)
The Problem
I don't know how to access meta_info from within the apps. I know I can
import the module itself and access meta_info:
apps/app1/__main__.py:
import app1
do_something(app1.meta_info)
But this is only possible if I know the name of the module. From inside another module - e.g. my_utility I don't know how to access the module which has been started in the first place (or it's name).
my_utility/__main__.py:
def handle_meta_info():
import MAIN_MODULE <-- don't know, what to import here
do_something(MAIN_MODULE.meta_info)
In other words
I don't know how to access meta_info from within an app's process (being started via python3 -m <name> but from another module which does not know the name of the 'root' module which has been started
Approaches
Always provide the module name when calling meta-info-functions (bad, because it's verbose and redundant)
from my_utility import handle_meta_info
handle_meta_info('app1')
add meta_info to __builtins__ (generally bad to pollute global space)
Parse the command line (ugly)
Analyze the call stack on import my_utility (dangerous, ugly)
The solution I'd like to see
It would be nice to be able to either access the "main" modules global space OR know it's name (to import)
my_utility/__main__.py:
def handle_meta_info():
do_something(__main_module__.meta_info)
OR
def handle_meta_info():
if process_has_been_started_as_module():
mod = importlib.import_module(name_of_main_module())
meta_inf = getattr(mod, 'meta_info')
do_something(meta_inf)
Any ideas?
My current (bloody) solution:
Inside my_utility I use psutil to get the command line the module has been started with (why not sys.argv? Because). There I extract the module name. This way I attach the desired meta information to my_utility (so I have to load it only once).
my_utility/__init__.py:
def __get_executed_modules_meta_info__() -> dict:
def get_executed_module_name()
from psutil import Process
from os import getpid
_cmdline = Process(getpid()).cmdline
try:
# normal case: app has been started via 'python3 -m <app>'
return _cmdline[_cmdline.index('-m') + 1]
except ValueError:
return None
from importlib import import_module
try:
_main_module = import_module(get_module_name())
return import_module(get_executed_module_name()).meta_info
except AttributeError:
return {}
__executed_modules_meta_info__ = __get_executed_modules_meta_info__()