We Keep Coding

App Engine python local unit test urlfetch_stub.py fancy_urllib module has no attribute 'InvalidCertificateException'

I'm using linux with google-cloud-sdk-app-engine-python package installed version 196.0.0-0.
I try to create local unit test, following reference from [1].
class ViewsTestCase(unittest.TestCase):
def setUp(self):
self.client = application.test_client()
self.client.testing = True
self.testbed = testbed.Testbed()
self.testbed.activate()
self.testbed.init_urlfetch_stub()
def tearDown(self):
self.testbed.deactivate()
add PYTHONPATH environment variable:
export PYTHONPATH="$PYTHONPATH:/usr/lib/google-cloud-sdk/platform/google_appengine:/usr/lib/google-cloud-sdk/platform/google_appengine/lib:/usr/lib/google-cloud-sdk/platform/google_appengine/lib/yaml/lib"
then I try to run it.
$ python test_views.py
Traceback (most recent call last):
File "test_views.py", line 6, in <module>
from google.appengine.ext import testbed
File "/usr/lib/google-cloud-sdk/platform/google_appengine/google/appengine/ext/testbed/__init__.py", line 130, in <module>
from google.appengine.api import urlfetch_stub
File "/usr/lib/google-cloud-sdk/platform/google_appengine/google/appengine/api/urlfetch_stub.py", line 32, in <module>
_fancy_urllib_InvalidCertException = fancy_urllib.InvalidCertificateException
AttributeError: 'module' object has no attribute 'InvalidCertificateException'
I have read this similar thread [2].
But I checked, the path of fancy_lib has been fixed.
$ find /usr/lib/google-cloud-sdk/platform/google_appengine/lib/fancy_urllib/
/usr/lib/google-cloud-sdk/platform/google_appengine/lib/fancy_urllib/
/usr/lib/google-cloud-sdk/platform/google_appengine/lib/fancy_urllib/__init__.pyc
/usr/lib/google-cloud-sdk/platform/google_appengine/lib/fancy_urllib/fancy_urllib
/usr/lib/google-cloud-sdk/platform/google_appengine/lib/fancy_urllib/fancy_urllib/__init__.pyc
/usr/lib/google-cloud-sdk/platform/google_appengine/lib/fancy_urllib/fancy_urllib/__init__.py
/usr/lib/google-cloud-sdk/platform/google_appengine/lib/fancy_urllib/__init__.py
I have tried to add fancy_urllib to PYTHONPATH
export PYTHONPATH="$PYTHONPATH:/usr/lib/google-cloud-sdk/platform/google_appengine:/usr/lib/google-cloud-sdk/platform/google_appengine/lib:/usr/lib/google-cloud-sdk/platform/google_appengine/lib/yaml/lib:/usr/lib/google-cloud-sdk/platform/google_appengine/lib/fancy_urllib"
But still failed.
any clue?
thank you.
[1]https://cloud.google.com/appengine/docs/standard/python/tools/localunittesting
[2]GAE SDK 1.9.5 and an InvalidCertificateException

If you see the file length of init.py is '0' and if there is sub folder with the same name (i.e. lib/fancy_urllib/fancy_urllib), then copy both files (.py, and .pyc) of the sub folder to upper folder. That worked for me.

Import with and without main scope

I have two files, app.py and database.py in the same directory.
Primarily I have the following code snippets:
app.py
import database
db = "demo_database"
print(database.show_database_information())
database.py
from app import db
database_username = "root"
database_password = "password"
def show_database_information():
information = {}
information["filename"] = db
information["username"] = database_username
information["password"] = database_password
return information
When I try to run app.py I got the following error:
Traceback (most recent call last):
File "K:\PyPrac\circular_call\app.py", line 1, in <module>
import database
File "K:\PyPrac\circular_call\database.py", line 1, in <module>
from app import db
File "K:\PyPrac\circular_call\app.py", line 3, in <module>
print(database.show_database_information())
AttributeError: module 'database' has no attribute 'show_database_information'
Then I updated app.py and included __main__ check like below:
app.py
import database
db = "demo_database"
if __name__ == '__main__':
print(database.show_database_information())
Now it runs smoothly without any error.
I have several questions,
What is name of the error occurred in first scenario? Need explanation.
Why it runs after including __main__ scope?
What is the better approach of doing operations like this?

What I can understand are as below's. Maybe someone more expert can elaborate !
Import error.
if __name__ == '__main__': This condition is used to check whether a python module is being run directly or being imported.
If a module is imported, then it's __name__ is the name of the module instead of main. So, in such situations it is better to call if __name__ == '__main__':

Man!! You are creating a circular moment. Let me tell how.
import database # from app.py
But from database.py you imported db from app. That is creating a circular moment.
On the other hand,
if __name__ == '__main__':
this is making you database.py as a name of the module instead of __main__ that's why it's working. Nothing magical :)
UPDATE: Placed from app import db this line inside the function show_database_information()
This is the HOTFIX for you.

Python program works in shell but in console

I'm writing a program that enables my Django application to consume a REST API. My problem is that Django rejects the program if I run it as a .py file, but accepts it via shell. The program is divided into two files: one that interacts with the API, and another that interacts with Django. The main folder structure is as follows:
/root folder
[rest of Django project]
-__init__.py
-test.py
-mid_side.py
-teamup_api/
-__init__.py
-teamup.py
where test.py is the file that starts the program, mid_side.py is the Django-side part, and teamup.py is the API-side part. Below is the specific code I'm having trouble with.
test.py
import django
from mid_side import dates
from datetime import datetime
start = datetime(2017,7,28)
end = datetime(2017,8,19)
test = dates(start,end)
print(test)
Where dates is a method from mid_side that accepts two datetime objects as parameters.
According to Django, the problem emerges in mid_side.py, as it imports models from an app in the main Django project. The specific line Django has issues with is from events.models import Service, where events is an app Django uses and Services is a model. The traceback is
Traceback (most recent call last):
File "test.py", line 2, in <module>
from mid_side import dates
File "/home/brian/Github/nydkc11/nydkcd11/mid_side.py", line 3, in <module>
from events.models import Service
File "/home/brian/Github/nydkc11/nydkcd11/events/models.py", line 1, in <module>
from embed_video.fields import EmbedVideoField
File "/home/brian/anaconda3/envs/server/lib/python3.6/site-packages/embed_video/fields.py", line 6, in <module>
from .backends import detect_backend, UnknownIdException, \
File "/home/brian/anaconda3/envs/server/lib/python3.6/site-packages/embed_video/backends.py", line 17, in <module>
from .settings import EMBED_VIDEO_BACKENDS, EMBED_VIDEO_TIMEOUT, \
File "/home/brian/anaconda3/envs/server/lib/python3.6/site-packages/embed_video/settings.py", line 7, in <module>
'embed_video.backends.SoundCloudBackend',
File "/home/brian/anaconda3/envs/server/lib/python3.6/site-packages/django/conf/__init__.py", line 56, in __getattr__
self._setup(name)
File "/home/brian/anaconda3/envs/server/lib/python3.6/site-packages/django/conf/__init__.py", line 39, in _setup
% (desc, ENVIRONMENT_VARIABLE))
django.core.exceptions.ImproperlyConfigured: Requested setting EMBED_VIDEO_BACKENDS, but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings.
I know for a fact the traceback concerns an app my project uses, django-embed-video. What I don't understand is why Django is indicating it as a problem considering a) Service doesn't use fields from django-embed-video, b) I successfully deployed the application on my project earlier. My questions are thus as follows:
1) Why does this problem occur in the .py file and not in shell?
2) What is causing Django to output the traceback?
Below is relevant code from the program. Please notify me if additional clarifications are required. Thank you!
*I decided not to include teamup.py as this is more of a Django-side problem, but I will include it if requested.
mid_side.py
from dateutil.parser import parse
from events.models import Service
import datetime
from teamup_api.teamup import *
def dates(start,end):
event_list = day_query(start.year, start.month, start.day, end.year, end.month, end.day)['events']
service_objects = []
for event in event_list:
service_objects.append(Service(
title=event['title'],
school=none_parse(event['who']),
location=none_parse(event['location']),
start_time=parse(event['start_dt']),
end_time=parse(event['end_dt']),
all_day=event['all_day'],
description=none_parse(event['notes'])
))
return service_objects
models.py
class Service(models.Model):
title = models.CharField(max_length=100)
school = models.CharField(max_length=100)
location = models.CharField(max_length=100)
start_time = models.DateTimeField()
end_time = models.DateTimeField()
all_day = models.BooleanField()
description = models.TextField()
def __str__(self):
return self.title
UPDATE: I've been running 'shell' through manage.py shell, and 'console' via natively running a python file (e.g. python [file].py)

My problem was probably that I was using Django resources in the native python environment. As Chris indicates in the comments, creating a custom manage.py command instead of a .py start file should suffice.

Deferred tasks creates new instances that can't access some python modules

I am using the latest version of GAE with automated scaling, endpoints API, and deferred.defer() tasks.
The problem is that since adding the API, there have been some instances that will spin up automatically that always throw permanent task failures:
Permanent failure attempting to execute task
Traceback (most recent call last):
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/deferred/deferred.py", line 310, in post
self.run_from_request()
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/deferred/deferred.py", line 305, in run_from_request
run(self.request.body)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/deferred/deferred.py", line 145, in run
raise PermanentTaskFailure(e)
PermanentTaskFailure: No module named app.Report
The permanent task failures are unique for a single instance though, in which every deferred tasks on that instance fail. These deferred tasks all throw the same error, even though the tasks aren't using the Api.py module. On other instances, the same deferred tasks will run just fine if they aren't routed to a failing instance.
The app.yaml handlers looks like this:
handlers:
# Api Handler
- url: /_ah/api/.*
script: main.api
- url: /_ah/spi/.*
script: main.api
# All other traffic
- url: .*
script: main.app
builtins:
- deferred: on
The main.py looks like:
import Api, endpoints, webapp2
api = endpoints.api_server([Api.AppApi])
app = webapp2.WSGIApplication(
[(misc routes)]
,debug=True)
The Api.py looks like :
import endpoints
from protorpc import messages
from protorpc import message_types
from protorpc import remote
from google.appengine.ext import deferred
from app.Report import ETLScheduler
#endpoints.api(...)
class AppApi(remote.Service):
#endpoints.method(...)
def reportExtract(self, request):
deferred.defer(
ETLScheduler,
params
)
I'm not doing any path modification, so I'm curious why the new instance is having trouble finding the python modules for the API, even though the deferred tasks are in another module using other functions. Why would it throw these errors for that instance only?
Edit:
So after looking at some other SO issues, I tried doing path modification in appengine_config.py. I moved all my folders to a lib directory, and added this to the config file:
import os,sys
sys.path.append(os.path.join(os.path.dirname(__file__), 'lib'))
Now the error I get on the failing instance is:
Permanent failure attempting to execute task
Traceback (most recent call last):
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/deferred/deferred.py", line 310, in post
self.run_from_request()
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/deferred/deferred.py", line 305, in run_from_request
run(self.request.body)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/deferred/deferred.py", line 145, in run
raise PermanentTaskFailure(e)
PermanentTaskFailure: cannot import name ETLScheduler
So it seems to be finding the module, but same as before, none of the deferred tasks on the instance can import the method.

So I figured out a way to make it work, but am not sure why it works.
By importing the entire module, rather than a method from the module, the new instances that spin up for deferred tasks no longer throw the PermanentTaskFailure: cannot import name ETLScheduler error.
I tried importing the whole module instead of the method, so that the Api.py looks like this:
import endpoints
from protorpc import messages
from protorpc import message_types
from protorpc import remote
from google.appengine.ext import deferred
# Import the module instead of the method
#from app.Report import ETLScheduler
import app.Report
#endpoints.api(...)
class AppApi(remote.Service):
#endpoints.method(...)
def reportExtract(self, request):
deferred.defer(
app.Report.ETLScheduler,
params
)
Now I am no longer getting instances that throw the PermanentTaskFailure: cannot import name ETLScheduler. Might be a circular dependency by import Api.py in main.py (I'm not sure) but at least it works now.

You're missing the _target kwarg in your defer invocation if you're trying to run something in a specific module.
deferred.defer(
app.Report.ETLScheduler,
params,
_target="modulename"
)

Importing a sub class from a sub folder

New to Python I'm trying to setup a simple OOP-structure of files, folders and classes. Here are the file paths:
C:\Users\Mc_Topaz\Programmering\Python\Terminal\Main.py
C:\Users\Mc_Topaz\Programmering\Python\Terminal\Connections\Connection.py
C:\Users\Mc_Topaz\Programmering\Python\Terminal\Connections\NoConnection.py
Notice Connection.py and NoConnection.py is loacted in sub folder Connections.
Connection.py
class Connection:
def __init__(self):
pass
def ToString(self):
pass
NoConnection.py
from Connection import Connection
class NoConnection(Connection):
def __init__(self):
pass
def ToString(self):
print("No connection")
In the Main.py file I would like to call the ToString() method from each class.
Main.py
from Connections.Connection import Connection
from Connections.NoConnection import NoConnection
connection = Connection()
print(connection.ToString())
noConnection = NoConnection()
print(noConnection.ToString())
When I run the Main.py file I get this error:
C:\Users\Mc_Topaz\Programmering\Python\Terminal>Main.py Traceback
(most recent call last): File
"C:\Users\Mc_Topaz\Programmering\Python\Terminal\Main.py", line 2, in
from Connections.NoConnection import NoConnection
File
"C:\Users\Mc_Topaz\Programmering\Python\Terminal\Connections\NoConnection.py",
line 1, in
from Connection import Connection
ImportError: No module named 'Connection'
It seems that the interpreter cannot import the NoConnection class in my Main.py file due to it cannot import the Connection class from the NoConnection.py file.
I can run Connection.py and NoConnection.py separately with no problems.
I don't understand why the Main.py don't run. I assume is something super simple and I cannot see it due to I'm to green to Python.

For python to recognize a directory is a module, or a collection of python files, that directory must contain a file named __init__.py. That file doesn't need to contain anything code whatsoever, though it can. If you add this file to your Connections directory, the interpreter should be able to import the contained files.

Just to add on to what mobiusklein mentioned: a common practice with __init__.py is to import the objects you use most frequently to avoid redundancy in your imports. In your example your Connections\__init__.py would likely contain:
import Connection
import NoConnection
Then your Main.py could use the following import statements successfully:
from Connections import Connection, NoConnection

I think it's down to relative imports in Python 3.
Change the import line in NoConnection to be explicit...
from Connections.Connection import Connection
and it works in Python3 (it works either way in Python2). Path to Terminal may have to be on you PYTHONPATH environment variable.
"Python 3 has disabled implicit relative imports altogether; imports are now always interpreted as absolute, meaning that in the above example import baz will always import the top-level module. You will have to use the explicit import syntax instead (from . import baz)." from here https://softwareengineering.stackexchange.com/questions/159503/whats-wrong-with-relative-imports-in-python
So when you import Connection from NoConnection it will be looking for it at the Terminal level, not at the Terminal/Connections level.

UPDTAE: Read comments for this post as it contain the solution.
Still don't get this to work. But I have made some changes:
Changed names of classes
> Class Connection = Foo inside Connection.py
> NoConnection = Bar inside NoConnection.py
This is ensure that the interpreter don't get confused if Connection is a module or a class.
Dropped ini.py file
I have dropped the Connections\__init_.py file as this don't seem to be necessary in Python version 3.4.2.
Files still run separately
I can run Connection.py and NoConnection.py separately. So they work.
Main.py
from Connections.NoConnection import Bar
noConnection = Bar()
print(noConnection.ToString())
When running Main.py I get the same error at line 1:
"Cannot find module 'Connection' in NoConnection.py at line 1".
The only logical reason I can see why this error happens is:
The interpreter looks for Connection.py inside my Terminal folder as Python was started from that folder. Even if it's importing the NoConnection.py file from the Connections folder where Connection.py is located.
Is this a bug?