I am trying to understand how to split a project which uses Mako and CherryPy in several directories. I have prepared the following directory structure:
[FOLDER] /home/user/myapp
|- main.py
|- app.config
|- server.config
[FOLDER] /home/user/myapp/templates
[FOLDER] /home/user/myapp/templates/base
|- index.html
|- sidebar_menu.html
[FOLDER] /home/user/myapp/config
|- templates.py
In /home/user/myapp/templates there will be the different templates organised in directories.
Under /home/user/myapp/config I have the following file: templates.py with the following code:
# -*- coding: utf-8 -*-
import mako.template
import mako.lookup
# Templates
templates_lookup = mako.lookup.TemplateLookup(
directories=[
'/templates',
'/templates/base',
],
module_directory='/tmp/mako_modules',
input_encoding='utf-8',
output_encoding='utf-8',
encoding_errors='replace'
)
def serve_template(templatename, **kwargs):
mytemplate = templates_lookup.get_template(templatename)
print(mytemplate.render(**kwargs))
Under /home/user/myapp there will be the following main.py file:
# -*- coding: utf-8 -*-
import os
import cherrypy
import mako.template
import mako.lookup
import config.templates
# Main Page
class Index(object):
#cherrypy.expose
def index(self):
t = config.templates.serve_template('index.html')
print(t)
return t
cherrypy.config.update("server.config")
cherrypy.tree.mount(Index(), '/', "app.config")
cherrypy.engine.start()
When I launch the application and access / I get the following message:
500 Internal Server Error
The server encountered an unexpected condition which prevented it from fulfilling the request.
Traceback (most recent call last):
File "C:\Python37\lib\site-packages\mako\lookup.py", line 247, in get_template
return self._check(uri, self._collection[uri])
KeyError: 'index.html'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Python37\lib\site-packages\cherrypy\_cprequest.py", line 628, in respond
self._do_respond(path_info)
File "C:\Python37\lib\site-packages\cherrypy\_cprequest.py", line 687, in _do_respond
response.body = self.handler()
File "C:\Python37\lib\site-packages\cherrypy\lib\encoding.py", line 219, in __call__
self.body = self.oldhandler(*args, **kwargs)
File "C:\Python37\lib\site-packages\cherrypy\_cpdispatch.py", line 54, in __call__
return self.callable(*self.args, **self.kwargs)
File ".....\myapp\main.py", line 18, in index
t = config.templates.serve_template('index.html')
File ".....\myapp\config\templates.py", line 19, in serve_template
mytemplate = templates_lookup.get_template(templatename)
File "C:\Python37\lib\site-packages\mako\lookup.py", line 261, in get_template
"Cant locate template for uri %r" % uri)
mako.exceptions.TopLevelLookupException: Cant locate template for uri 'index.html'
Powered by CherryPy 18.1.0
So basically it seems that Mako can not locate index.html despite we are providing the directories. I guess I am not understanding how Mako uses in the lookup.
Note: program is actually run in Windows, I used UNIX file structure above just to make the file structure easier to read.
Python 3.7.2
CherryPy 18.1.0
Mako 1.0.7
You state your directory structure is
/home/user/myapp/templates
but you're telling Mako to look in
/templates
Maybe change code to:
directories=[
'/home/user/myapp/templates',
'/home/user/myapp/templates/base',
],
I usually split the templates into per-page templates and global templates
example:
src/
├── constants.py
├── home
│ └── user
│ └── myapp
│ ├── app.config
│ ├── main.mako
│ ├── main.py
│ └── server.config
└── templates
├── e404.mako
├── e500.mako
├── footer.mako
└── header.mako
in this case, i'll always import a global file with the lookup dir
# src/constants.py
from mako.lookup import TemplateLookup
mylookup = TemplateLookup(directories=['.', 'dir/to/src/templates/'])
# home/user/myapp/main.py
from src.constants import mylookup
def main():
if i_have_errer:
template = mylookup.get_template('e500.mako')
else:
template = mylookup.get_template('main.mako')
return template.render_unicode()
the '.' will lookup first in current directory
the templates/ will lookup the global src/templates/ for a more generic templates
Related
i'm creating some unit tests and i'm facing this issue
FAILED test_views_home.py::test_index -
jinja2.exceptions.TemplateNotFound: index.html
this is part of my testing code:
templates = Jinja2Templates('templates')
def test_index():
#router.get('/', include_in_schema=False)
async def index(request: Request):
return templates.TemplateResponse('index.html', {'request': request})
client = TestClient(router)
response = client.get("/")
assert response.status_code == 200
assert '<!DOCTYPE html>' in response.text
and this is my folder structure:
root/
├── views/
│ ├── index.py
├── templates/
│ ├── index.html
├── tests/
│ ├── test_sorullo.py
i got:
FAILED test_views_home.py::test_index -
jinja2.exceptions.TemplateNotFound: index.html
I'm guessing i'm putting wrong
templates = Jinja2Templates('templates')
but i couldn't figure it out. I didn't find anything similar searching. What i'm doing wrong?
thanks!
The error says that Jinja2 template engine is unable to find the specified template file. Check your template directory and make sure the file index.html is present.
The template directory is set to "templates". So the Jinja2 engine will look for the templates files in the directory named "templates" in the same directory of your code file, If the template files are located in a different directory, you can specify the full path to the directory instead of just the directory name.
Try to add the full path of template directory. If it solves you can correct the directory path.
I have
ModuleNotFoundError: No module named 'project'
while trying to run test_req.py
My project structure is:
├── instance/
│ ├── flask.cfg
├── project/
│ ├── __init__.py
│ ├── base_processing.py
│ ├── models.py
| ├── views.py
│ ├── templates/
│ │ ├── base.html
│ │ ├── login.html
│ │ ├── note.html
│ │ ├── notes.html
│ └── static/
│ │
│ └── tests/
│ ├── test_req.html
├── run.py
My UnitTest file is:
# project/test_req.py
import unittest
import os
from project import app
from project.models import db, User, Note
from project.views import *
TEST_DB = 'test.db'
class RequestTests(unittest.TestCase):
#classmethod
def setUpClass(cls):
app.config['TESTING'] = True
app.config['WTF_CSRF_ENABLED'] = False
app.config['DEBUG'] = False
app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:///' + \
os.path.join(app.config['BASEDIR'], TEST_DB)
app.secret_key = 'staytrue'
cls.app = app.test_client()
def setUp(self):
db.create_all()
def tearDown(self):
db.drop_all()
def test_main_page(self):
response = self.app.get('/', follow_redirects=True)
self.assertEqual(response.status_code, 200)
def test_auth(self):
u = User(username='testname1', password='1234567', email='cyber#mail.com')
db.session.add(u)
db.session.commit()
response = self.app.post('/login', data=dict(username='testname1', password='1234567'), follow_redirects=True)
with self.app.session_transaction() as sess:
self.assertEqual(sess['username'], 'testname1')
if __name__ == "__main__":
unittest.main()
Also my test work just fine with nose2, when I run it from my root directory. Also this is the first time I'm organizing my project layout this way.
Module is not a folder, it should be a .py file. As you don't have project.py file, you should not specify from project import app.
Specifying from project import app means that there is project.py file and you want to import class app from this file.
if your test_req.py and app.py files are located in the same folder, then just use: import app in your test_req.py
Also replace:
from project.models import db, User, Note
from project.views import *
to
from models import db, User, Note
from views import *
Further reading:
Python
modules
Importing
Also, I would recommend to use PyCharm Community Edition, it is free, multilpatform and open source, and will help you to solve such tasks just via two mouse clicks.
Assume we have the following project structure in the root folder of our project:
/folder1/MyPythonFile1.py
/folder1/folder11/MyPythonFile2.py
/folder2/MyApp.py
/folder1/MyPythonFile1.py file looks like that:
class Class1:
def __init__(self):
pass
class Class2:
def __init__(self):
pass
And /folder1/folder11/MyPythonFile2.py file looks like that:
class Class3:
def __init__(self):
pass
File /folder2/MyApp.py uses classes from aforecited files and looks like that:
from folder1.MyPythonFile1 import Class1
from folder1.folder11.MyPythonFile2 import Class3
obj1 = Class1()
obj3 = Class3()
Apply this example to your particular case and update your imports accordingly.
I'm using Jinja2 in Flask. I want to render a template from a string. I tried the following 2 methods:
rtemplate = jinja2.Environment().from_string(myString)
data = rtemplate.render(**data)
and
rtemplate = jinja2.Template(myString)
data = rtemplate.render(**data)
However both methods return:
TypeError: no loader for this environment specified
I checked the manual and this url: https://gist.github.com/wrunk/1317933
However nowhere is specified to select a loader when using a string.
You can provide loader in Environment from that list
from jinja2 import Environment, BaseLoader
rtemplate = Environment(loader=BaseLoader).from_string(myString)
data = rtemplate.render(**data)
Edit: The problem was with myString, it has {% include 'test.html' %} and Jinja2 has no idea where to get template from.
UPDATE
As #iver56 kindly noted, it's better to:
rtemplate = Environment(loader=BaseLoader()).from_string(myString)
When I came to this question, I wanted FileSystemLoader:
from jinja2 import Environment, FileSystemLoader
with open("templates/some_template.html") as f:
template_str = f.read()
template = Environment(loader=FileSystemLoader("templates/")).from_string(template_str)
html_str = template.render(default_start_page_lanes=default_start_page_lanes,
**data)
I tried using the FileSystemLoader, but that didn't immediately work for me. My code was all in a module (a subdirectory with an __init__.py file), so I was able to use PackageLoader instead. The Jinja documentation calls this "the simplest way" to configure templates:
my-proj
├── Pipfile
├── Pipfile.lock
└── app
├── __init__.py
├── app.py
├── constants.py
└── templates
├── base.html.jinja
└── macros.html.jinja
from jinja2 import Environment, PackageLoader
def generate_html(my_list):
env = Environment(loader=PackageLoader("app"))
template = env.get_template("base.html.jinja")
return template.render({ "stuff": my_list })
I'm using celery (and django-celery) to allow a user to launch periodic scrapes through the django admin. This is part of a larger project but I've boiled the issue down to a minimal example.
Firstly, celery/celerybeat are running daemonized. If instead I run them with celery -A evofrontend worker -B -l info from my django project dir then I get no issues weirdly.
When I run celery/celerybeat as daemons however then I get a strange import error:
[2016-01-06 03:05:12,292: ERROR/MainProcess] Task evosched.tasks.scrapingTask[e18450ad-4dc3-47a0-b03d-4381a0e65c31] raised unexpected: ImportError('No module named myutils',)
Traceback (most recent call last):
File "/home/lee/Desktop/pyco/evo-scraping-min/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
R = retval = fun(*args, **kwargs)
File "/home/lee/Desktop/pyco/evo-scraping-min/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in __protected_call__
return self.run(*args, **kwargs)
File "evosched/tasks.py", line 35, in scrapingTask
cs = CrawlerScript('TestSpider', scrapy_settings)
File "evosched/tasks.py", line 13, in __init__
self.crawler = CrawlerProcess(scrapy_settings)
File "/home/lee/Desktop/pyco/evo-scraping-min/venv/local/lib/python2.7/site-packages/scrapy/crawler.py", line 209, in __init__
super(CrawlerProcess, self).__init__(settings)
File "/home/lee/Desktop/pyco/evo-scraping-min/venv/local/lib/python2.7/site-packages/scrapy/crawler.py", line 115, in __init__
self.spider_loader = _get_spider_loader(settings)
File "/home/lee/Desktop/pyco/evo-scraping-min/venv/local/lib/python2.7/site-packages/scrapy/crawler.py", line 296, in _get_spider_loader
return loader_cls.from_settings(settings.frozencopy())
File "/home/lee/Desktop/pyco/evo-scraping-min/venv/local/lib/python2.7/site-packages/scrapy/spiderloader.py", line 30, in from_settings
return cls(settings)
File "/home/lee/Desktop/pyco/evo-scraping-min/venv/local/lib/python2.7/site-packages/scrapy/spiderloader.py", line 21, in __init__
for module in walk_modules(name):
File "/home/lee/Desktop/pyco/evo-scraping-min/venv/local/lib/python2.7/site-packages/scrapy/utils/misc.py", line 71, in walk_modules
submod = import_module(fullpath)
File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
File "retail/spiders/Retail_spider.py", line 16, in <module>
ImportError: No module named myutils
i.e. the spider is having issues importing from the django project app despite adding the relevant things to syslog, and doing django.setup().
My hunch is that this may be caused by a " circular import" during initialization, but I'm not sure (see here for notes on same error)
Celery daemon config
For completeness the celeryd and celerybeat configuration scripts are:
# /etc/default/celeryd
CELERYD_NODES="worker1"
CELERY_BIN="/home/lee/Desktop/pyco/evo-scraping-min/venv/bin/celery"
CELERY_APP="evofrontend"
DJANGO_SETTINGS_MODULE="evofrontend.settings"
CELERYD_CHDIR="/home/lee/Desktop/pyco/evo-scraping-min/evofrontend"
CELERYD_OPTS="--concurrency=1"
# Workers should run as an unprivileged user.
CELERYD_USER="lee"
CELERYD_GROUP="lee"
CELERY_CREATE_DIRS=1
and
# /etc/default/celerybeat
CELERY_BIN="/home/lee/Desktop/pyco/evo-scraping-min/venv/bin/celery"
CELERY_APP="evofrontend"
CELERYBEAT_CHDIR="/home/lee/Desktop/pyco/evo-scraping-min/evofrontend/"
# Django settings module
export DJANGO_SETTINGS_MODULE="evofrontend.settings"
They are largely based on the the generic ones, with the Django settings thrown in and using the celery bin in my virtualenv rather than system.
I'm also using the init.d scripts which are the generic ones.
Project structure
As for the project: it lives at /home/lee/Desktop/pyco/evo-scraping-min. All files under it have ownership lee:lee.
The dir contains both a Scrapy (evo-retail) and Django (evofrontend) project that live under it and the complete tree structure looks like
├── evofrontend
│ ├── db.sqlite3
│ ├── evofrontend
│ │ ├── celery.py
│ │ ├── __init__.py
│ │ ├── settings.py
│ │ ├── urls.py
│ │ └── wsgi.py
│ ├── evosched
│ │ ├── __init__.py
│ │ ├── myutils.py
│ │ └── tasks.py
│ └── manage.py
└── evo-retail
└── retail
├── logs
├── retail
│ ├── __init__.py
│ ├── settings.py
│ └── spiders
│ ├── __init__.py
│ └── Retail_spider.py
└── scrapy.cfg
Django project relevant files
Now the relevant files: the evofrontend/evofrontend/celery.py looks like
# evofrontend/evofrontend/celery.py
from __future__ import absolute_import
import os
from celery import Celery
# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'evofrontend.settings')
from django.conf import settings
app = Celery('evofrontend')
# Using a string here means the worker will not have to
# pickle the object when using Windows.
app.config_from_object('django.conf:settings')
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)
The potentially relevant settings from the Django settings file, evofrontend/evofrontend/settings.py are
import os
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
PROJECT_ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir))
INSTALLED_APPS = (
...
'djcelery',
'evosched',
)
# Celery settings
BROKER_URL = 'amqp://guest:guest#localhost//'
CELERY_ACCEPT_CONTENT = ['json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
CELERY_TIMEZONE = 'Europe/London'
CELERYD_MAX_TASKS_PER_CHILD = 1 # Each worker is killed after one task, this prevents issues with reactor not being restartable
# Use django-celery backend database
CELERY_RESULT_BACKEND = 'djcelery.backends.database:DatabaseBackend'
# Set periodic task
CELERYBEAT_SCHEDULER = "djcelery.schedulers.DatabaseScheduler"
The tasks.py in the scheduling app, evosched, looks like (it just launches the Scrapy spider using the relevant settings after changing dir)
# evofrontend/evosched/tasks.py
from __future__ import absolute_import
from celery import shared_task
from celery.utils.log import get_task_logger
logger = get_task_logger(__name__)
import os
from scrapy.crawler import CrawlerProcess
from scrapy.utils.project import get_project_settings
from django.conf import settings as django_settings
class CrawlerScript(object):
def __init__(self, spider, scrapy_settings):
self.crawler = CrawlerProcess(scrapy_settings)
self.spider = spider # just a string
def run(self, **kwargs):
# Pass the kwargs (usually command line args) to the crawler
self.crawler.crawl(self.spider, **kwargs)
self.crawler.start()
#shared_task
def scrapingTask(**kwargs):
logger.info("Start scrape...")
# scrapy.cfg file here pointing to settings...
base_dir = django_settings.BASE_DIR
os.chdir(os.path.join(base_dir, '..', 'evo-retail/retail'))
scrapy_settings = get_project_settings()
# Run crawler
cs = CrawlerScript('TestSpider', scrapy_settings)
cs.run(**kwargs)
The evofrontend/evosched/myutils.py simply contains (in this min example):
# evofrontend/evosched/myutils.py
SCRAPY_XHR_HEADERS = 'SOMETHING'
Scrapy project relevant files
In the complete Scrapy project the settings file looks like
# evo-retail/retail/retail/settings.py
BOT_NAME = 'retail'
import os
PROJECT_ROOT = os.path.dirname(os.path.abspath(__file__))
SPIDER_MODULES = ['retail.spiders']
NEWSPIDER_MODULE = 'retail.spiders'
and (in this min example) the spider is just
# evo-retail/retail/retail/spiders/Retail_spider.py
from scrapy.conf import settings as scrapy_settings
from scrapy.spiders import Spider
from scrapy.http import Request
import sys
import django
import os
import posixpath
SCRAPY_BASE_DIR = scrapy_settings['PROJECT_ROOT']
DJANGO_DIR = posixpath.normpath(os.path.join(SCRAPY_BASE_DIR, '../../../', 'evofrontend'))
sys.path.insert(0, DJANGO_DIR)
os.environ.setdefault("DJANGO_SETTINGS_MODULE", 'evofrontend.settings')
django.setup()
from evosched.myutils import SCRAPY_XHR_HEADERS
class RetailSpider(Spider):
name = "TestSpider"
def start_requests(self):
print SCRAPY_XHR_HEADERS
yield Request(url='http://www.google.com', callback=self.parse)
def parse(self, response):
print response.url
return []
EDIT:
I discovered through lots of trial and error that if the app I'm trying to import from is in my INSTALLED_APPS django setting, then it fails with the import error, but if I remove the app from there then no longer do I get the import error (e.g. removing evosched from INSTALLED_APPS then the import in the spider goes through fine...). Obviously not a solution, but may be a clue.
EDIT 2
I put a print of sys.path immediately before the failing import in the spider, the result was
/home/lee/Desktop/pyco/evo-scraping-min/evofrontend/../evo-retail/retail
/home/lee/Desktop/pyco/evo-scraping-min/venv/lib/python2.7
/home/lee/Desktop/pyco/evo-scraping-min/venv/lib/python2.7/plat-x86_64-linux-gnu
/home/lee/Desktop/pyco/evo-scraping-min/venv/lib/python2.7/lib-tk
/home/lee/Desktop/pyco/evo-scraping-min/venv/lib/python2.7/lib-old
/home/lee/Desktop/pyco/evo-scraping-min/venv/lib/python2.7/lib-dynload
/usr/lib/python2.7
/usr/lib/python2.7/plat-x86_64-linux-gnu
/usr/lib/python2.7/lib-tk
/home/lee/Desktop/pyco/evo-scraping-min/venv/local/lib/python2.7/site-packages
/home/lee/Desktop/pyco/evo-scraping-min/evofrontend
/home/lee/Desktop/pyco/evo-scraping-min/evo-retail/retail`
EDIT 3
If I do import evosched then print dir(evosched), I see "tasks" and if I choose to include such a file, I can also see "models", so importing from models would actually be possible. I don't however see " myutils". Even from evosched import myutils fails and also fails if the statement is put in a function below rather than as a global(I thought this might route out a circular import issue...). The direct import evosched works...possibly import evosched.utils will work. Not yet tried...
It seems the celery daemon is running using the system's python and not the python binary inside the virtualenv. You need to use
# Python interpreter from environment.
ENV_PYTHON="$CELERYD_CHDIR/env/bin/python"
As mentioned here to tell celeryd to run using the python inside the virtualenv.
When trying to load a file with pickle i am getting ImportError: No module named application_mgmt.
Strangely the same file can be loaded no problem form a different function they even both use the same get_file method and all. Also strangely i can load any of the other files from the function.
I have tried moving the functions to a different class/file. clearing and re-populating the saved file but nothing seems to work.
The Object in the saved file:
class Application():
def __init__(self,name,focus=False):
self.name = name
self.focus = focus
self.prod_score = 5
self.display_name = name
self.color = "none"
Function that causes error:
def check_meta_info(self, app_name):
self.get_file("saved_meta_data")
File Handling Function:
def get_file(self, file_name):
path = "back/saved_data/%s" % (file_name)
try:
with open(path,'rb') as saved_file:
saved_list = pickle.load(saved_file)
saved_file.close()
return saved_list
except IOError:
#stuff
Log:
Traceback (most recent call last):
File "<stdin>", line 400, in <module>
File "<stdin>", line 221, in app_meta_info
File "<stdin>", line 313, in check_meta_info
File "<stdin>", line 358, in get_file
ImportError: No module named application_mgmt
shell returned 1
The Function That works but calls the same file class:
def add_meta_info(self, new_application):
new_meta = Application(new_application) # creates obj
saved_meta_info = self.get_file("saved_meta_data")
for metas in saved_meta_info:
if new_meta.name == metas.name:
return False
saved_meta_info.append(new_meta)
self.save_file(saved_meta_info,"saved_meta_data")
del new_meta
File Structure:
.
├── active_screen.glade
├── active_screen.py
├── back
│ ├── application_mgmt.py
│ ├── application_mgmt.pyc
│ ├── bash
│ │ ├── get_active_window.sh
│ │ ├── prosessScript.sh
│ │ └── test_lock.sh
│ ├── bash_schedular.py
│ ├── __init__.py
│ ├── __init__.pyc
│ ├── saved_data
│ │ ├── first_time_builder.py
│ │ ├── saved_active_data
│ │ ├── saved_background_data
│ │ ├── saved_ignore_data
│ │ └── saved_meta_data < HIM
Looking at the error log, I don't think your problem has anything to do with the pickle loading.
I don't know why he's trying to import application_mgmt - which line of get_file is the 358th ? - but ImportError could be caused by :
Forgetting the __init__file in the back folder. A folder is not importable without it. Create an empty one if it's missing.
Python path problem : To check if that's the problem, try adding this at the beginning of your get_file method.
import sys
sys.path.append('/path/to/the/back/module/')
Circular imports : If you find any - between application_mgmt.py and another file -, try refactoring your code to avoid them.
Hope this helps.