I'm trying to import a custom module that I created, but it breaks my API just to import it.
data directory:
-src
--order
----__init__.py
----app.py
----validator.py
----requirments.txt
--__init__.py
on my app.py I have this code:
import json
from .validator import validate
def handler(event, context):
msg = ''
if event['httpMethod'] == 'GET':
msg = "GET"
elif event['httpMethod'] == 'POST':
pass #msg = validate(json.loads(event['body']))
return {
"statusCode": 200,
"body": json.dumps({
"message": msg,
}),
}
I get this error:
Unable to import module 'app': attempted relative import with no known parent package
However, if I remove line 2 (from .validator import validate) from my code, it works fine, so the problem is with that import, and honestly, I can't figure what is going on. I have tried to import using:
from src.order.validator import validate
but it doesn't work either.
was able to solve my issue by generating a build through the command: sam build, and zipping my file, and putting it on the root folder inside aws-sam, it's not a great solution because I have to rebuild at every small change, but at least it's a workaround for now
It seems app.py has not been loaded as part of the package hierarchy (i.e. src and order packages have not been loaded). You should be able to run
from src.order import app
from the parent directory of src and your code will work. If you run python app.py from the terminal — which I assume is what you did — app.py will be run as a standalone script — not as part of a package.
However, I do not believe you need the .validator in your case since both modules are in the same directory. You should be able to do
from validator import validate
Related
I'm using Python 3.9.5.
Based on this post, I'm trying to reuse some functions from the parent directory. Here's my code hierarchy:
github_repository
src
base
string_utilities.py
validation
email_validator.py
I also have __init__.py in all folders. In ALL of them.
Here's the string_utilities.py content:
def isNullOrEmpty(text: str):
return text is not None and len(text) > 0
And here's the email_validator.py content:
from src.base import string_utilities
def is_email(text: str):
if string_utilities.isNullOrEmpty(text):
return False
# logic to check email
return True
Now when I run python email_validator.py, I get this error:
ModuleNotFoundError: No module named 'src'
I have changed that frustrating import statement to all of these different forms, and I still get no results:
from ...src.base import string_utilities
which results in:
ImportError: attempted relative import with no known parent package
import src.base.string_utilities
Which causes compiler to not know the isNullOrEmpty function.
import ...src.base.string_utilities
Which results in:
Relative imports cannot be used with "import .a" form; use "from . import a" instead
I'm stuck at this point on how to reuse that function in this file. Can someone please help?
I have created a simple HTTP trigger-based azure function in python which is calling another python script to create a sample file in azure data lake gen 1. My solution structure is given below: -
Requirements.txt contains the following imports: -
azure-functions
azure-mgmt-resource
azure-mgmt-datalake-store
azure-datalake-store
init.py
import logging, os, sys
import azure.functions as func
import json
def main(req: func.HttpRequest) -> func.HttpResponse:
logging.info('Python HTTP trigger function processed a request.')
name = req.params.get('name')
if not name:
try:
req_body = req.get_json()
except ValueError:
pass
else:
name = req_body.get('name')
if name:
full_path_to_script = os.path.join(os.path.dirname( __file__ ) + '/Test.py')
logging.info(f"Path: - {full_path_to_script}")
os.system(f"python {full_path_to_script}")
return func.HttpResponse(f"Hello {name}!")
else:
return func.HttpResponse(
"Please pass a name on the query string or in the request body",
status_code=400
)
Test.py
import json
from azure.datalake.store import core, lib, multithread
directoryId = ''
applicationKey = ''
applicationId = ''
adlsCredentials = lib.auth(tenant_id = directoryId, client_secret = applicationKey, client_id = applicationId)
adlsClient = core.AzureDLFileSystem(adlsCredentials, store_name = '')
with adlsClient.open('stage1/largeFiles/TestFile.json', 'rb') as input_file:
data = json.load(input_file)
with adlsClient.open('stage1/largeFiles/Result.json', 'wb') as responseFile:
responseFile.write(data)
Test.py is failing with an error that no module found azure.datalake.store
Why other required modules are not working for Test.py since it is inside the same directory?
pip freeze output: -
adal==1.2.2
azure-common==1.1.23
azure-datalake-store==0.0.48
azure-functions==1.0.4
azure-mgmt-datalake-nspkg==3.0.1
azure-mgmt-datalake-store==0.5.0
azure-mgmt-nspkg==3.0.2
azure-mgmt-resource==6.0.0
azure-nspkg==3.0.2
certifi==2019.9.11
cffi==1.13.2
chardet==3.0.4
cryptography==2.8
idna==2.8
isodate==0.6.0
msrest==0.6.10
msrestazure==0.6.2
oauthlib==3.1.0
pycparser==2.19
PyJWT==1.7.1
python-dateutil==2.8.1
requests==2.22.0
requests-oauthlib==1.3.0
six==1.13.0
urllib3==1.25.6
Problem
os.system(f"python {full_path_to_script}") from your functions project is causing the issue.
Azure Functions Runtime sets up the environment, along with modifying process level variables like os.path so that your function can load any dependencies you may have. When you create a sub-process like that, not all information will flow through. Additionally, you will face issues with logging -- logs from test.py would not show up properly unless explicitly handled.
Importing works locally because you have all your requirements.txt modules installed and available to test.py. This is not the case in Azure. After remotely building as part of publish, your modules are included as part of your code package published. It's not "installed" globally in the Azure environment per se.
Solution
You shouldn't have to run your script like that. In the example above, you could import your test.py from your __init__.py file, and that should behave like it was called python test.py (at least in the case above). Is there a reason you'd want to do python test.py in a sub-process over importing it?
Here's the official guide on how you'd want to structure your app to import shared code -- https://learn.microsoft.com/en-us/azure/azure-functions/functions-reference-python#folder-structure
Side-Note
I think once you get through the import issue, you may also face problems with adlsClient.open('stage1/largeFiles/TestFile.json', 'rb'). We recommend following the developer guide above to structure your project and using __file__ to get the absolute path (reference).
For example --
import pathlib
with open(pathlib.Path(__file__).parent / 'stage1' / 'largeFiles' /' TestFile.json'):
....
Now, if you really want to make os.system(f"python {full_path_to_script}") work, we have workarounds to the import issue. But, I'd rather not recommend such approach unless you have a really compelling need for it. :)
I am sorry if the title takes some time to understand. So here is the folder structure:
falcon_tut/
falcon_tut/
app.py
images.py
__init__.py
tests/
test_app.py
__init__.py
And some codes
####################
# app.py
####################
from images import Resource
images = Resource()
api = application = falcon.API()
api.add_route('/images', images)
# ... few more codes
####################
# test_app.py
####################
import falcon
from falcon import testing
import ujson
import pytest
from falcon_tut.app import api
#pytest.fixture
def client():
return testing.TestClient(api)
def test_list_images(client):
doc = {
'images': [
{
'href': '/images/1eaf6ef1-7f2d-4ecc-a8d5-6e8adba7cc0e.png'
}
]
}
response = client.simulate_get('/images')
result_doc = ujson.loads(response.content)
assert result_doc == doc
assert response.status == falcon.HTTP_OK
It works fine when running with python falcon_tut/app.py and curl it with response of 200 and payload of the images
Until running pytest tests/ from the project root it output this:
ImportError while importing test module ../falcon_tut/tests/test_app.py
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests/test_app.py:6: in <module>
from falcon_tut.app import api
E ModuleNotFoundError: No module named 'falcon_tut'
I tried creating __init__.py at the project root but it still output the same error above
Python version 3.7.0, with falcon 1.4.1, cpython 0.28.5, pytest 3.7.3, and instead of using gunicorn, I am using bjoern 2.2.2
I am trying out the python falcon framework and encounter the error at the testing part.
==========UPDATE===========
The reason why pytest could not found the module is because sys.path does not have ../falcon_tut/falcon_tut exists.
When I ran pytest and edit those 2 files and printing out sys.path, it only has [../falcon_tut/tests, ../falcon_tut, ..]. The workaround to this is to append the path to the package to sys.path. So here is the edited app.py
#############
# app.py
#############
import sys
# this line is just example, please rewrite this properly if you wants to use this workaround
# sys_path[1] only applied to my situation, again this is just example to know that it works
# the idea is to make sure the path to your module exists in sys.path
# in this case, I appended ../falcon_tut/falcon_tut to sys.path
# so that now ../falcon_tut/falcon_tut/images.py can be found by pytest
sys.path.insert(0, '{}/falcon_tut'.format(sys_path[1]))
# body codes...
I've been stuck all day on what seems to be a very silly import problem. From my Django project directory, I can import a module and run a function just fime:
(msg-gw)slashingweapon:~/msg-gw/www$ python
>>> import snpp
>>> snpp.config_url('snpp://server.name.com:1234?user=me&pass=whatever')
{'host': 'server.name.com', 'pass': 'whatever', 'port': 1234, 'user': 'me'}
But when I try to run my app, either through manage.py or by gunicorn, I get an attribute error:
(msg-gw)slashingweapon:~/msg-gw/www$ python manage.py runserver 8000
File "/home/slashingweapon/msg-gw/www/project/settings.py", line 26, in <module>
SNPP = snpp.config_url('snpp://server.name.com:1234?user=me&pass=whatever')
AttributeError: 'module' object has no attribute 'config_url'
The two relevant lines in my settings.py file are exactly what you would expect. Notice that I can import the module just fine, but the config_url() function isn't found.
import snpp
SNPP = snpp.config_url('snpp://server.name.com:1234?user=me&pass=whatever')
The directory layout is exactly what you would expect:
www
|
+-project
| +-__init__.py
| +-settings.py
| +-urls.py
| +-views.py
| +-wsgi.py
|
+-snpp
+-__init__.py
+-protocol.py
+-views.py
+-urls.py
The config_url() function is defined inside snpp/__init__.py
I have tried all kinds of things:
from snpp import config_url
move config_url to the file snpp/config and then import with
import snpp.confg
from snpp.config import config_url
from snpp import config and then invoke through config.config_url()
The __init__.py file is nothing special. It just lets you encode some server information as a string, so you can stick your SNPP config into the environment:
import urlparse
def config_url(urlStr):
config = {
'host':None,
'port':444,
'user':None,
'pass':None,
}
url = urlparse.urlparse(urlStr)
if url.scheme == 'snpp':
locParts = url.netloc.split(":")
config['host'] = locParts[0]
if len(locParts) > 1:
port = int(locParts[1])
if port > 0:
config['port'] = port
args = urlparse.parse_qs(url.query)
config['user'] = args.get('user', [None])[0]
config['pass'] = args.get('pass', [None])[0]
return config
I am using Python 2.7, django 1.5.1, and virtualenv.
Other parts of my project work well. When I print out the path in my browser, it looks correct. Importing snpp should not be a problem, since snpp is in the www directory:
/home/slashingweapon/msg-gw/www
/home/slashingweapon/msg-gw/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg
/home/slashingweapon/msg-gw/lib/python2.7/site-packages/pip-1.3.1-py2.7.egg
/home/slashingweapon/msg-gw/lib/python2.7/site-packages/django_json_rpc-0.6.2-py2.7.egg
/home/slashingweapon/msg-gw/lib/python27.zip
/home/slashingweapon/msg-gw/lib/python2.7
... etc ...
It doesn't matter if the snpp module is in my INSTALLED_APPS list or not. I get the same result.
Solved
With the help of SO denizens, I found the problem.
I had refactored my application by moving some reusable code pieces from the project directory to the new snpp directory. When I did that, I neglected to move or delete the *.pyc files.
The value of snpp.__file__ was:
/home/slashingweapon/msg-gw/www/project/snpp.pyc
instead of the expected:
/home/slashingweapon/msg-gw/www/snpp/__init__.pyc
During the import process, Python was looking in project/ before snpp/ and finding an old snpp.pyc file. It would import the old pyc file and be satisfied, thus ignoring the entire snpp/ dir.
Had I been a little sharper (or a little more experienced with Python) I might have noticed that I was getting some strange import behavior in general whenever I tried to import anything from snpp/. It should have occurred to me that the whole module was wonky, and not just the one function I was trying to use at the moment.
Check what exactly is being imported by using
snpp.__file__
right after import snpp statement.
Actually import might not be from the path you are expecting to see.
I started using Python few days back and I think I have a very basic question where I am stuck. Maybe I am not doing it correctly in Python so wanted some advice from the experts:
I have a config.cfg & a class test in one package lib as follows:
myProj/lib/pkg1/config.cfg
[api_config]
url = https://someapi.com/v1/
username=sumitk
myProj/lib/pkg1/test.py
class test(object):
def __init__(self, **kwargs):
config = ConfigParser.ConfigParser()
config.read('config.cfg')
print config.get('api_config', 'username')
#just printing here but will be using this as a class variable
def some other foos()..
Now I want to create an object of test in some other module in a different package
myProj/example/useTest.py
from lib.pkg1.test import test
def temp(a, b, c):
var = test()
def main():
temp("","","")
if __name__ == '__main__':
main()
Running useTest.py is giving me error:
...
print config.get('api_config', 'username')
File "C:\Python27\lib\ConfigParser.py", line 607, in get
raise NoSectionError(section)
ConfigParser.NoSectionError: No section: 'api_config'
Now if I place thie useTest.py in the same package it runs perfectly fine:
myProj/lib/pkg1/useTest.py
myProj/lib/pkg1/test.py
myProj/lib/pkg1/config.cfg
I guess there is some very basic package access concept in Python that I am not aware of or is there something I am doing wrong here?
The issue here is that you have a different working directory depending on which module is your main script. You can check the working directory by adding the following lines to the top of each script:
import os
print os.getcwd()
Because you just provide 'config.cfg' as your file name, it will attempt to find that file inside of the working directory.
To fix this, give an absolute path to your config file.
You should be able to figure out the absolute path with the following method since you know that config.cfg and test.py are in the same directory:
# inside of test.py
import os
config_path = os.path.join(os.path.dirname(os.path.abspath(__file__)),
'config.cfg')