I have a Django server which trying to run with subprocess.Popen and store all the logs on a file, this is my code:
with open('thefile.log', 'a') as the_file:
p1 = subprocess.Popen(['python', os.getcwd() + '\\mySite\\manage.py', 'runserver'], stdout=the_file,
stderr=the_file, universal_newlines=True)
and this is the result in the thefile.log:
Watching for file changes with StatReloader
[26/Jul/2019 13:10:05] "GET / HTTP/1.1" 200 16348
[26/Jul/2019 13:10:05] "GET /static/admin/css/fonts.css HTTP/1.1" 200 423
[26/Jul/2019 13:10:06] "GET /static/admin/fonts/Roboto-Light-webfont.woff HTTP/1.1" 200 85692
[26/Jul/2019 13:10:06] "GET /static/admin/fonts/Roboto-Bold-webfont.woff HTTP/1.1" 200 86184
[26/Jul/2019 13:10:06] "GET /static/admin/fonts/Roboto-Regular-webfont.woff HTTP/1.1" 200 85876
Not Found: /favicon.ico
[26/Jul/2019 13:10:08] "GET /favicon.ico HTTP/1.1" 404 1976
Not Found: /fs
[26/Jul/2019 13:10:11] "GET /fs HTTP/1.1" 404 1949
Performing system checks...
System check identified no issues (0 silenced).
July 26, 2019 - 13:09:44
Django version 2.2.3, using settings 'mySite.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CTRL-BREAK.
but it should be like this:
Watching for file changes with StatReloader
Performing system checks...
System check identified no issues (0 silenced).
July 26, 2019 - 13:09:44
Django version 2.2.3, using settings 'mySite.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CTRL-BREAK.
[26/Jul/2019 13:10:05] "GET / HTTP/1.1" 200 16348
[26/Jul/2019 13:10:05] "GET /static/admin/css/fonts.css HTTP/1.1" 200 423
[26/Jul/2019 13:10:06] "GET /static/admin/fonts/Roboto-Light-webfont.woff HTTP/1.1" 200 85692
[26/Jul/2019 13:10:06] "GET /static/admin/fonts/Roboto-Bold-webfont.woff HTTP/1.1" 200 86184
[26/Jul/2019 13:10:06] "GET /static/admin/fonts/Roboto-Regular-webfont.woff HTTP/1.1" 200 85876
Not Found: /favicon.ico
[26/Jul/2019 13:10:08] "GET /favicon.ico HTTP/1.1" 404 1976
Not Found: /fs
[26/Jul/2019 13:10:11] "GET /fs HTTP/1.1" 404 1949
in fact I have some other serivices, my other question is that How can I print the result of them in propper way with subprocess.Popen on my current terminal? for example:
service1:
------
------
service2:
------
------
You can not route stdout and stderr to the same file handler like that. You can however use a trick here to "route" the stderr to the stdout (or vice-versa), like:
with open('thefile.log', 'a') as the_file:
p1 = subprocess.Popen(
['python', os.getcwd() + '\\mySite\\manage.py', 'runserver'],
stdout=the_file,
stderr=subprocess.STDOUT,
universal_newlines=True
)
This looks similar to a shell where you write python manage.py runserver >> thefile.log 2>&1.
Related
I have a gunicorn *config.py and need to add response time to gunicorn access log:
Here's what I have:
...
accesslog = some_path/filename
access_log_format = '%(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s"'
...
gunicorn version 19.7.1
Some more details:
*config.py didn't contain the seting access_log_format before, and I thought adding it might help, but even after I added and restarted the process, the access log contains the same parameters as before:
<ip> - - [16/Aug/2020:14:11:35 XDT] "POST /ws/v1/sessions?userId=<username> HTTP/1.1" 200 537 "-" "python-requests/2.22.0"
<ip> - - [16/Aug/2020:14:11:56 XDT] "DELETE /ws/v1/sessions HTTP/1.1" 200 139 "-" "python-requests/2.22.0"
<ip> - - [17/Aug/2020:04:05:40 XDT] "GET /ws/v1/data-specs HTTP/1.1" 200 35097 "-" "python-requests/2.22.0"
<ip> - - [17/Aug/2020:09:30:31 XDT] "GET /ws/v1/data-specs HTTP/1.1" 200 35097 "-" "python-requests/2.22.0"
I am new to python, gunicorn and web apis. I am unable to figure out how to add response time to the log. What am I doing wrong here? Please help.
From the documentation (https://docs.gunicorn.org/en/stable/settings.html#access-log-format)
The corresponding value for response time miliseconds is %(D)s
Trying to move my development environment to run on minikube.
The page loads but my page uses websockets on the same port/protocol that the index.html is loaded with (https in this case), and the websockets do no seem to be working correctly.
Here is an example of the correct output when run through nginx / python on my local development box.
127.0.0.1 - - [14/Sep/2018 14:14:35] "GET / HTTP/1.0" 200 -
127.0.0.1 - - [14/Sep/2018 14:14:35] "GET /static/jquery.min.js HTTP/1.0" 200 -
127.0.0.1 - - [14/Sep/2018 14:14:35] "GET /static/socket.io.min.js HTTP/1.0" 200 -
127.0.0.1 - - [14/Sep/2018 14:14:35] "GET /socket.io/?EIO=3&transport=polling&t=MNPIg-N HTTP/1.0" 200 -
127.0.0.1 - - [14/Sep/2018 14:14:35] "GET /favicon.ico HTTP/1.0" 404 -
127.0.0.1 - - [14/Sep/2018 14:14:35] "GET /favicon.ico HTTP/1.0" 404 -
127.0.0.1 - - [14/Sep/2018 14:14:35] "POST /socket.io/?EIO=3&transport=polling&t=MNPIg-o&sid=0570b4fe27f345e9b11858b3acb40a6e HTTP/1.0" 200 -
127.0.0.1 - - [14/Sep/2018 14:14:35] "GET /socket.io/?EIO=3&transport=polling&t=MNPIg-r&sid=0570b4fe27f345e9b11858b3acb40a6e HTTP/1.0" 200 -
127.0.0.1 - - [14/Sep/2018 14:14:35] "POST /socket.io/?EIO=3&transport=polling&t=MNPIg_x&sid=0570b4fe27f345e9b11858b3acb40a6e HTTP/1.0" 200 -
127.0.0.1 - - [14/Sep/2018 14:14:35] "GET /socket.io/?EIO=3&transport=polling&t=MNPIg_w&sid=0570b4fe27f345e9b11858b3acb40a6e HTTP/1.0" 200 -
127.0.0.1 - - [14/Sep/2018 14:14:40] "GET /socket.io/?EIO=3&transport=polling&t=MNPIh0L&sid=0570b4fe27f345e9b11858b3acb40a6e HTTP/1.0" 200 -
127.0.0.1 - - [14/Sep/2018 14:14:45] "GET /socket.io/?EIO=3&transport=polling&t=MNPIiE3&sid=0570b4fe27f345e9b11858b3acb40a6e HTTP/1.0" 200 -
127.0.0.1 - - [14/Sep/2018 14:14:50] "GET /socket.io/?EIO=3&transport=polling&t=MNPIjSI&sid=0570b4fe27f345e9b11858b3acb40a6e HTTP/1.0" 200 -
127.0.0.1 - - [14/Sep/2018 14:14:55] "GET /socket.io/?EIO=3&transport=polling&t=MNPIkgS&sid=0570b4fe27f345e9b11858b3acb40a6e HTTP/1.0" 200 -
Notice how there is a GET every 5 seconds (that's a timer running on the page)
When running on Kubernetes, The page loads and the timer shows up as if the websocket has worked, however I show no logs where the websocket is having a GET or POST after the initial one.
192.168.99.1,172.17.0.7 - - [14/Sep/2018 18:24:03] "GET /static/jquery.min.js HTTP/1.1" 304 1210 0.008244
192.168.99.1,172.17.0.7 - - [14/Sep/2018 18:24:03] "GET /static/socket.io.min.js HTTP/1.1" 304 1210 0.009271
(10) accepted ('172.17.0.7', 34444)
192.168.99.1,172.17.0.7 - - [14/Sep/2018 18:24:04] "GET /socket.io/?EIO=3&transport=polling&t=MNPKrsy HTTP/1.1" 200 379 0.003682
(10) accepted ('172.17.0.7', 34446)
192.168.99.1,172.17.0.7 - - [14/Sep/2018 18:24:04] "GET /favicon.ico HTTP/1.1" 404 1314 0.004694
(10) accepted ('172.17.0.7', 34448)
(10) accepted ('172.17.0.7', 34450)
(10) accepted ('172.17.0.7', 34452)
192.168.99.1,172.17.0.7 - - [14/Sep/2018 18:24:04] "GET /socket.io/?EIO=3&transport=polling&t=MNPKrtD&sid=77d4755c524f47c2948b9c36da007b85 HTTP/1.1" 200 210 0.000749
192.168.99.1,172.17.0.7 - - [14/Sep/2018 18:24:04] "POST /socket.io/?EIO=3&transport=polling&t=MNPKrtB&sid=77d4755c524f47c2948b9c36da007b85 HTTP/1.1" 200 194 0.002632
(10) accepted ('172.17.0.7', 34454)
192.168.99.1,172.17.0.7 - - [14/Sep/2018 18:24:04] "GET /favicon.ico HTTP/1.1" 404 1314 0.002388
The Ingress is setup as follows:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress-service
annotations:
kubernetes.io/ingress.class: nginx
spec:
rules:
- http:
paths:
- path: /
backend:
serviceName: websitev2-cluster-ip-service
servicePort: 8080
As mentioned before, the websocket does not reside on a different port and it is instanciated in javascript as:
namespace = '/socket';
var socket = io.connect(location.protocol + '//' + document.domain + ':' + location.port + namespace);
Are there any special requirements to get websockets to work? I do not believe i need a special route because the URI needs to be sent to the same location as everything else, and on the same port.
EDIT: MORE DETAILS
My website has a form, that when submitted, executes the following code:
$('form#job').submit(function(event) {
var nameValue = JSON.stringify($(event.target).serializeArray());
console.log(nameValue)
socket.emit('job_submit', {data: nameValue});
return false;
});
On the python side, I have my socket code, which should get hit once a user clicks the submit button.
#socketio.on('job_submit', namespace='/socket')
def job_submit(message):
print('recieved job_submit message from client')
# kick off subjob in celery task
data = unstringify(message)
print('data = {0}'.format(data))
sessiondata = dict(session)
print('sessiondata = {}'.format(sessiondata))
subjobstart.delay(sessiondata, request.sid, data)
In my logs I am not receiving any "recieved job_submit message from client" messages when the submit button is clicked, which means the javascript is trying to send a websocket emit to the python server, but the message is never getting there.
Emits from python to the client page are working as the time is getting updated on the site.
So not exactly sure what changed as it worked on test box, apparently the issue lied in the form validation I was doing. Once I put in a novalidate option to temporarily bypass the form checking, the form was then able to be submitted and I did not have any websocket issues after all.
Hopefully this post can help someone with the code posted instead.
The path is correct, and the file exists.
#app.route('/Users/<username>/<filename>')
def send_image(username, filename):
return send_from_directory('/Users/'+username, filename)
127.0.0.1 - - [16/Jun/2017 20:29:00] "GET /myuploads HTTP/1.1" 200 -
127.0.0.1 - - [16/Jun/2017 20:29:00] "GET /Users/alon123/1185392_716316861727493_1066399254_n.jpg HTTP/1.1" 404 -
127.0.0.1 - - [16/Jun/2017 20:29:00] "GET /Users/alon123/boy.jpg HTTP/1.1" 404 -
127.0.0.1 - - [16/Jun/2017 20:29:00] "GET /Users/alon123/gg.jpeg HTTP/1.1" 404 -
127.0.0.1 - - [16/Jun/2017 20:29:00] "GET /Users/alon123/IMG_1409.PNG HTTP/1.1" 404 -
127.0.0.1 - - [16/Jun/2017 20:29:00] "GET /Users/alon123/IMG_2743.JPG HTTP/1.1" 404 -
127.0.0.1 - - [16/Jun/2017 20:29:00] "GET /Users/alon123/IMG_2757.JPG HTTP/1.1" 404 -
127.0.0.1 - - [16/Jun/2017 20:29:00] "GET /Users/alon123/secondpage.png HTTP/1.1" 404 -
Here is the directory structure:
enter image description here
The root is Ludbox, and the html that call the function mentioned above could be found in templates.
Try adding a forward slash to the endpoint:
#app.route('/Users/<username>/<filename>/')
I have built the default mezzanine site.
However, in the admin page of the site, only part of the styles are applied to the site.
And I see some error logs
[18/Apr/2017 00:03:47] "GET /admin/login/?next=/admin/ HTTP/1.1" 200 4251
[18/Apr/2017 00:03:47] "GET /static/grappelli/css/base.css/reset.css HTTP/1.1" 4
04 1709
[18/Apr/2017 00:03:47] "GET /static/grappelli/css/base.css/typography.css HTTP/1
.1" 404 1724
[18/Apr/2017 00:03:48] "GET /static/grappelli/css/base.css/modules.css HTTP/1.1"
404 1715
[18/Apr/2017 00:03:48] "GET /static/grappelli/css/base.css/tables.css HTTP/1.1"
404 1712
[18/Apr/2017 00:03:48] "GET /static/grappelli/css/base.css/forms.css HTTP/1.1" 4
04 1709
[18/Apr/2017 00:03:48] "GET /static/grappelli/css/base.css/widgets.css HTTP/1.1"
404 1715
[18/Apr/2017 00:03:48] "GET /static/grappelli/css/base.css/webkit-gradients.css
HTTP/1.1" 404 1742
[18/Apr/2017 00:03:48] "GET /static/grappelli/css/img/grappelli-icon.png HTTP/1.
1" 404 1721
It seems paths of some css files are wrong.
The path shouldn't be /static/grappelli/css/base.css/xxxx.css but /static/grappelli/css/xxxx.css
When I dig into the /static/grappelli/css/base.css file, I see this file imports those wrong path css files like this
...
#import url('reset.css');
#import url('typography.css');
#import url('modules.css');
#import url('tables.css');
#import url('forms.css');
#import url('widgets.css');
...
How to fix it?
I had the same issue on Ubuntu 16.04.3 with python3.5.1. This is an issue that has be noted in mezzanine bugs but I couldn't find a workaround.
I upgraded to ubuntu 17.04 with python3.5.3 and there was no issue, some deep dark magic is happening in grappelli here.
I am new to Django and this question may seem to be easy. In the terminal where "python manage.py runserver" is executed, the following logs periodically appear.
[30/Sep/2015 02:36:02] "GET /messages/check/?_=1443574208652 HTTP/1.1" 200 1
[30/Sep/2015 02:36:08] "GET /notifications/check/?_=1443574208653 HTTP/1.1" 200 1
[30/Sep/2015 02:36:08] "GET /feeds/update/?first_feed=13&last_feed=6&feed_source=all&_=1443574208655 HTTP/1.1" 200 173
[30/Sep/2015 02:36:08] "GET /feeds/check/?last_feed=13&feed_source=all&_=1443574208654 HTTP/1.1" 200 1
[30/Sep/2015 02:36:39] "GET /notifications/check/?_=1443574208656 HTTP/1.1" 200 1
[30/Sep/2015 02:36:39] "GET /feeds/check/?last_feed=13&feed_source=all&_=1443574208658 HTTP/1.1" 200 1
[30/Sep/2015 02:36:39] "GET /feeds/update/?first_feed=13&last_feed=6&feed_source=all&_=1443574208657 HTTP/1.1" 200 173
[30/Sep/2015 02:37:03] "GET /messages/check/?_=1443574208659 HTTP/1.1" 200 1
My Django virtual environment is as follows:
bleach==1.4
dj-database-url==0.3.0
dj-static==0.0.6
Django==1.6.5
gunicorn==19.3.0
html5lib==0.9999999
Markdown==2.4.1
Pillow==2.4.0
psycopg2==2.6.1
python-decouple==2.2
six==1.9.0
South==0.8.4
static3==0.6.1
Unipath==1.0
Questions are:
Why does the logs periodically appear? I guess there is a background task running.But Celery package is not used here.
How to configure the logs, for example, displaying one more logs such as:
/followers/check/?_=9896886900907 HTTP/1.1
What do "?_" and the long string of numbers stand for? I can understand the "?first_feed=13" means the querying parameter in Django request object.
GET /notifications/check/?_=1443574208656
When Django is run in development mode (with runserver) it prints a line for all HTTP requests received to the console.
All of them are coming from the outside, browsers, feed readers...
The parameters you are wondering about are GET query parameters. The ? is the delimiter so, _ is the key of the argument.
In development mode the logging is defined by default. If you want to have logging on your production system or more logging, read https://docs.djangoproject.com/en/1.8/topics/logging/ .