I am not getting uwsgi stats using uwsgitop and socket. I have put uwsgi configuration for the stats with socket and when I tried to get the stat using the command:
uwsgitop /var/www/uwsgi/proj.socket
It's throwing the error
JSONDecodeError: Expecting value: line 1 column 1 (char 0)
I am using uwsgi version 2.0.17.1.
Here is my uwsgi ini file
[uwsgi]
# Multi Thread Support
enable-threads = true
# Django-related settings
# the base directory (full path)
chdir = /home/user/base-dir/proj-path/
# Django's wsgi file
module = proj.wsgi
# the virtualenv (full path)
home = /home/user/base-path/
# process-related settings
# master
master = true
# maximum number of worker processes
processes = 10
socket = /var/www/uwsgi/proj.socket
# ... with appropriate permissions - may be needed
chmod-socket = 666
# clear environment on exit
vacuum = true
daemonize = /var/www/uwsgi/uwsgi.log
pidfile = /var/www/uwsgi/uwsgi_hub.pid
logto = /var/log/proj_uwsgi%n.log
uid = user
gid = user
http-auto-gzip = true
memory-report = True
py-tracebacker=/var/www/uwsgi/proj.socket
--stats /var/www/uwsgi/proj.socket
I think that you should have something like this in your config file:
socket = /var/www/uwsgi/proj.socket
stats = /var/www/uwsgi/stats.socket
And run uwsgitop on the stats socket, like so:
uwsgitop /var/www/uwsgi/stats.socket
Related
I have a process goodreads-user-scraper that runs fine within a cron scheduler script that I run from my Ubuntu terminal.
From my Ubuntu server terminal, I navigate to the directory containing scheduler.py and write:
python scheduler.py
This runs fine. It scrapes the site and saves files to the output_dir I have assigned inside the script.
Now, I want to run this function using a service file (socialAggregator.service).
When I set up a service file in my Ubuntu server to run scheduler.py, goodreads-user-scraper is not recognized. It's the exact same file I just ran from the terminal.
Why is goodreads-user-scraper not found when the service file calls the script?
Any ideas?
Error message form syslog file
Jan 12 22:13:15 speedypersonal2 python[2668]: --user_id: 1: goodreads-user-scraper: not found
socialAggregator.service
[Unit]
Description=Run Social Aggregator scheduler - collect data from API's and store in socialAggregator Db --- DEVELOPMENT ---
After=network.target
[Service]
User=nick
ExecStart= /home/nick/environments/social_agg/bin/python /home/nick/applications/socialAggregator/scheduler.py --serve-in-foreground
[Install]
WantedBy=multi-user.target
scheduler.py
from apscheduler.schedulers.background import BackgroundScheduler
import json
import requests
from datetime import datetime, timedelta
import os
from sa_config import ConfigLocal, ConfigDev, ConfigProd
import logging
from logging.handlers import RotatingFileHandler
import subprocess
if os.environ.get('CONFIG_TYPE')=='local':
config = ConfigLocal()
elif os.environ.get('CONFIG_TYPE')=='dev':
config = ConfigDev()
elif os.environ.get('CONFIG_TYPE')=='prod':
config = ConfigProd()
#Setting up Logger
formatter = logging.Formatter('%(asctime)s:%(name)s:%(message)s')
formatter_terminal = logging.Formatter('%(asctime)s:%(filename)s:%(name)s:%(message)s')
#initialize a logger
logger_init = logging.getLogger(__name__)
logger_init.setLevel(logging.DEBUG)
#where do we store logging information
file_handler = RotatingFileHandler(os.path.join(config.PROJ_ROOT_PATH,'social_agg_schduler.log'), mode='a', maxBytes=5*1024*1024,backupCount=2)
file_handler.setFormatter(formatter)
#where the stream_handler will print
stream_handler = logging.StreamHandler()
stream_handler.setFormatter(formatter_terminal)
logger_init.addHandler(file_handler)
logger_init.addHandler(stream_handler)
def scheduler_funct():
logger_init.info(f"- Started Scheduler on {datetime.today().strftime('%Y-%m-%d %H:%M')}-")
scheduler = BackgroundScheduler()
job_collect_socials = scheduler.add_job(run_goodreads,'cron', hour='*', minute='13', second='15')#Testing
scheduler.start()
while True:
pass
def run_goodreads():
logger_init.info(f"- START run_goodreads() -")
output_dir = os.path.join(config.PROJ_DB_PATH)
goodreads_process = subprocess.Popen(['goodreads-user-scraper', '--user_id', config.GOODREADS_ID,'--output_dir', output_dir], shell=True, stdout=subprocess.PIPE)
logger_init.info(f"- send subprocess now on::: goodreads_process.communicate() -")
_, _ = goodreads_process.communicate()
logger_init.info(f"- FINISH run_goodreads() -")
if __name__ == '__main__':
scheduler_funct()
The problem was the environment which the service file was using was not the same as the environment used when I run the script in the terminal.
Below is the service file that now works.
[Unit]
Description=Run Social Aggregator scheduler - collect data from API's and store in socialAggregator Db --- DEVELOPMENT ---
After=network.target
[Service]
User=nick
ExecStart= /home/nick/environments/social_agg/bin/python /home/nick/applications/socialAggregator/scheduler.py --serve-in-foreground
Environment=PATH=/home/nick/environments/social_agg/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
[Install]
WantedBy=multi-user.target
I added Environment=PATH=<path_from_terminal_and_venv_activated>
For additional clarity, path_from_terminal_and_venv_activated is obtained by:
Activiating my python venv in the terminal
copying the result of echo $PATH
I am running my Django site as a vassal of UWSGI emperor. I have created /etc/uwsgi-emperor/vassals/mysite.ini as follows:
[uwsgi]
socket = /var/opt/mysite/uwsgi.sock
chmod-socket = 775
chdir = /opt/mysite
master = true
virtualenv = /opt/mysite_virtualenv
env = DJANGO_SETTINGS_MODULE=mysite.settings
module = mysite.wsgi:application
uid = www-data
gid = www-data
processes = 1
threads = 1
plugins = python3,logfile
logger = file:/var/log/uwsgi/app/mysite.log
vacuum = true
But /var/log/uwsgi/app/mysite.log does not get created. If I touch it, it remains empty. This occurs even after I trigger 500-style errors in the application.
Why aren't my logs being written?
The vassal does not have permission to write to the file (or create the file in the first place). You should
cd /var/log/uwsgi/app
touch mysite.log # create the file
chown www-data:www-data mysite.log # give the vassal permission
(where www-data:www-data matches the uid and gid values in your ini file).
Logs will start appearing shortly.
Hey I have problem with deploying my app with uWSGI:
frontend.ini
[uwsgi]
http = *.*.*.*:8181
master = true
#uid = uwsgiuser
#gid = uwsgiuser
processes = 1
harakiri = 60
harakiri-verbose = true
limit-post = 65536
post-buffering = 8192
listen = 128
max-requests = 1000
reload-on-as = 128
reload-on-rss = 96
no-orphans = true
log-slow = true
plugins = python
module = skysoccer.app:main
wsgi-file = /wsgi.py
pythonpath = /eggs/*.egg
pythonpath = /*
pythonpath = *
pythonpath = skysoccer/*
stats= *.*.*.*:8080
This is what I get:
http://pastebin.com/KBhMnFv7
And then when I type in webbrowser: http://..*.*:8181/
In cli I get:
TypeError: 'Router' object is not iterable
[pid: 13692|app: 0|req: 1/1] *.*.*.* () {36 vars in 630 bytes} [Wed Jul 10 14:36:31 2013] GET / => generated 0 bytes in 737 msecs (HTTP/1.1 500) 0 headers in 0 bytes (0 switches on core 0)
In code I don't have varable "Router".
I think that you do not need the module option, but rather just a wsgi.py file that exposes your wsgi app as the application variable.
To do this, a typical wsgi.py file might look like:
import os.path
from pyramid.paster import get_app
from pyramid.paster import setup_logging
here = os.path.dirname(os.path.abspath(__file__))
inipath = os.path.join(here, 'production.ini')
setup_logging(inipath)
application = get_app(inipath)
This would configure your app to load the production.ini file that is in the same folder as the wsgi.py file.
I have an application that needs to initialize Celery and other things (e.g. database). I would like to have a .ini file that would contain the applications configuration. This should be passed to the application at runtime.
development.init:
[celery]
broker=amqp://localhost/
backend=amqp://localhost/
task.result.expires=3600
[database]
# database config
# ...
celeryconfig.py:
from celery import Celery
import ConfigParser
config = ConfigParser.RawConfigParser()
config.read(...) # Pass this from the command line somehow
celery = Celery('myproject.celery',
broker=config.get('celery', 'broker'),
backend=config.get('celery', 'backend'),
include=['myproject.tasks'])
# Optional configuration, see the application user guide.
celery.conf.update(
CELERY_TASK_RESULT_EXPIRES=config.getint('celery', 'task.result.expires')
)
# Initialize database, etc.
if __name__ == '__main__':
celery.start()
To start Celery, I call:
celery worker --app=myproject.celeryconfig -l info
Is there anyway to pass in the config file without doing something ugly like setting a environment variable?
Alright, I took Jordan's advice and used a env variable. This is what I get in celeryconfig.py:
celery import Celery
import os
import sys
import ConfigParser
CELERY_CONFIG = 'CELERY_CONFIG'
if not CELERY_CONFIG in os.environ:
sys.stderr.write('Missing env variable "%s"\n\n' % CELERY_CONFIG)
sys.exit(2)
configfile = os.environ['CELERY_CONFIG']
if not os.path.isfile(configfile):
sys.stderr.write('Can\'t read file: "%s"\n\n' % configfile)
sys.exit(2)
config = ConfigParser.RawConfigParser()
config.read(configfile)
celery = Celery('myproject.celery',
broker=config.get('celery', 'broker'),
backend=config.get('celery', 'backend'),
include=['myproject.tasks'])
# Optional configuration, see the application user guide.
celery.conf.update(
CELERY_TASK_RESULT_EXPIRES=config.getint('celery', 'task.result.expires'),
)
if __name__ == '__main__':
celery.start()
To start Celery:
$ export CELERY_CONFIG=development.ini
$ celery worker --app=myproject.celeryconfig -l info
How is setting an environment variable ugly? You either set an environment variable with the current version of your application, or you can derive it based on hostname, or you can make your build/deployment process overwrite the file, and on development you let development.ini copy over to settings.ini in a general location, and on production you let production.ini copy over to settings.ini.
Any of these options are quite common. Using a configuration management tool such as Chef or Puppet to put the file in place is a good option.
I'm having trouble with logging. I'm running CherryPy 3.2 and I've been reading through the docs here, but haven't found any examples of how to configure a local log file for output and how to write to it.
Raspberry.py:
import socket
import sys
import cherrypy
app_roots = {
# Sean's laptop dev environment.
"mylaptop": "/home/src/local-mydomain.com/py",
# Hosted dev environment.
"mydomain.com" : "/home/dev/src/py"
}
hostname = socket.gethostname()
CherryPyLog = cherrypy.tree.mount().log
if hostname not in app_roots:
CherryPyLog("The following hostname does not have an app_root entry in raspberry.py. Exiting early.")
sys.exit()
sys.stdout = sys.stderr
sys.path.append(app_roots[hostname])
import os
os.chdir(app_root)
# Setup for raspberry application logging.
import datetime
today = datetime.datetime.today()
log.access_file = "{0}/{1}.raspberry.access.log".format(app_roots[hostname],today.strftime("%Y%m%d-%H%M"))
log.error_file = "{0}/{1}.raspberry.error.log".format(app_roots[hostname],today.strftime("%Y%m%d-%H%M"))
#Testing logger
log("{0} -- Logger configured".format(today.strftime("%Y%m%d-%H%M%S")))
import atexit
cherrypy.config.update({'environment': 'embedded'})
if cherrypy.__version__.startswith('3.0') and cherrypy.engine.state == 0:
cherrypy.engine.start(blocking = False)
atexit.register(cherrypy.engine.stop)
from web.controllers.root import RaspberryRequestHandler
application = cherrypy.Application(RaspberryRequestHandler(), script_name = None, config = None)
UPDATE: Here's the code block that I ended up going with.
app_roots = {
# Sean's laptop dev environment.
"mylaptop": "/home/src/local-plottools.com/py",
# Hosted dev environment.
"myDomain" : "/home/dev/src/py"
}
import socket
hostname = socket.gethostname()
import cherrypy
import sys
if hostname not in app_roots:
cherrypy.log("The hostname {0} does not have an app_root entry in {1}. Exiting early.".format(hostname,__file__))
sys.exit()
sys.stdout = sys.stderr
sys.path.append(app_roots[hostname])
import os
os.chdir(app_roots[hostname])
from web.controllers.root import RaspberryRequestHandler
cherrypy.config.update({
'log.access_file': "{0}/cherrypy-access.log".format(app_roots[hostname]),
'log.error_file': "{0}/cherrypy.log".format(app_roots[hostname]),
"server.thread_pool" : 10
})
# special case, handling debug sessions when quickstart is needed.
if __name__ == "__main__":
cherrypy.config.update({
'log.screen': True,
"server.socket_port": 8000
})
cherrypy.quickstart(RaspberryRequestHandler())
sys.exit()
# This configuration is needed for running under mod_wsgi. See here: http://tools.cherrypy.org/wiki/ModWSGI
cherrypy.config.update({'environment': 'embedded'})
applicationLogName = "{0}/raspberry.log".format(app_roots[hostname])
from logging import handlers
applicationLogFileHandler = handlers.RotatingFileHandler(applicationLogName, 'a', 10000000, 1000)
import logging
applicationLogFileHandler.setLevel(logging.DEBUG)
from cherrypy import _cplogging
applicationLogFileHandler.setFormatter(_cplogging.logfmt)
cherrypy.log.error_log.addHandler(applicationLogFileHandler)
application = cherrypy.Application(RaspberryRequestHandler(), None)
Simplifying a bit:
import os
import socket
import sys
import cherrypy
app_roots = {
# Sean's laptop dev environment.
"mylaptop": "/home/src/local-mydomain.com/py",
# Hosted dev environment.
"mydomain.com" : "/home/dev/src/py"
}
hostname = socket.gethostname()
if hostname not in app_roots:
cherrypy.log("The hostname %r does not have an app_root entry in "
"raspberry.py. Exiting early." % hostname)
sys.exit()
sys.path.append(app_roots[hostname])
os.chdir(app_root)
cherrypy.config.update({
'environment': 'embedded',
'log.access_file': "{0}/raspberry.access.log".format(app_roots[hostname]),
'log.error_file': "{0}/raspberry.error.log".format(app_roots[hostname]),
})
from web.controllers.root import RaspberryRequestHandler
application = cherrypy.tree.mount(RaspberryRequestHandler(), '/')
# Insert log changes here
cherrypy.engine.start()
If you want different logs per day, use a RotatingFileHandler as described at http://www.cherrypy.org/wiki/Logging#CustomHandlers The important point I think you're missing is that you should muck about with app.log only after you've instantiated your app (e.g. via tree.mount(), as above), but before engine.start. That is, for the error log:
application = cherrypy.tree.mount(RaspberryRequestHandler(), '/')
log = application.log
log.error_file = ""
# Make a new RotatingFileHandler for the error log.
fname = "{0}/raspberry.error.log".format(app_roots[hostname])
h = handlers.RotatingFileHandler(fname, 'a', 10000000, 1000)
h.setLevel(DEBUG)
h.setFormatter(_cplogging.logfmt)
log.error_log.addHandler(h)
cherrypy.engine.start()
Hope that helps...