I have an app that's outputting a JSON through a socket which was working fine in the past, however recently the socket doesn't seem to be establishing and it continuously POST & GET's with the lines:
<user_ip>,<client_ip> - - [17/Jul/2018 12:48:17]"GET /socket.io/?EIO=3&transport=polling......HTTP/1.1" 200 221 0.000000
<user_ip>,<client_ip> - - [17/Jul/2018 12:48:17]"POST /socket.io/?EIO=3&transport=polling......HTTP/1.1" 200 243 0.517600
Additionally, when the WebSocket first tries, I can see from Chromes console the message:
WebSocket connection to ..... failed: Establishing a tunnel via proxy server failed.
I've also had a look through what I think is a similar issue based on socketIO-client but wasn't able to resolve my problem.
Can anyone help with overcoming this connection issue?
Included a full log on running the app below:
Server initialized for eventlet.
* Debugger is active!
* Debugger PIN: 129-744-633
(7616) wsgi starting up on http://0.0.0.0:6328
(7616) accepted (<client_ip>, 50548)
<user_ip>,<client_ip> - - [17/Jul/2018 13:18:23] "GET /<app_url> HTTP/1.1" 200 1664 0.015000
3602c46fa50247eb9d397fda82f3eae8: Sending packet OPEN data {'sid': '3602c46fa50247eb9d397fda82f3eae8', 'upgrades': ['websocket'], 'pingTimeout': 60000, 'pingInterval': 25000}
3602c46fa50247eb9d397fda82f3eae8: Sending packet MESSAGE data 0
<user_ip>,<client_ip> - - [17/Jul/2018 13:18:23] "GET /socket.io/?EIO=3&transport=polling&t=1531829903367-0 HTTP/1.1" 200 381 0.000000
(7616) accepted (<client_ip>, 50560)
3602c46fa50247eb9d397fda82f3eae8: Received packet MESSAGE data 0/testnamespace
3602c46fa50247eb9d397fda82f3eae8: Sending packet MESSAGE data 0/testnamespace
<user_ip>,<client_ip> - - [17/Jul/2018 13:18:23] "POST /socket.io/?EIO=3&transport=polling&t=1531829903377-1&sid=3602c46fa50247eb9d397fda82f3eae8 HTTP/1.1" 200 221 0.000000
<user_ip>,<client_ip> - - [17/Jul/2018 13:18:23] "GET /socket.io/?EIO=3&transport=polling&t=1531829903379-2&sid=3602c46fa50247eb9d397fda82f3eae8 HTTP/1.1" 200 226 0.000000
(7616) accepted (<client_ip>, 50562)
3602c46fa50247eb9d397fda82f3eae8: Received packet MESSAGE data 2/testnamespace,["event_1",{"data":"Web app connection successful."}]
3602c46fa50247eb9d397fda82f3eae8: Sending packet MESSAGE data 2/testnamespace,["client_event",{"json":[{"Date&Time":"........}]
Related
The following Python2 webserver will log every single request including the one from localhost (127.0.0.1).
webserver.py
import SimpleHTTPServer, SocketServer, sys
Handler = SimpleHTTPServer.SimpleHTTPRequestHandler
port = 80
httpd = SocketServer.TCPServer(("", port), Handler)
sys.stderr = open('/home/user/log.txt', 'w', 1)
httpd.serve_forever()
As example; curl localhost (from the same machine) will produce the following log.
10.0.0.1 - - [10/Jan/2019 00:00:00] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [10/Jan/2019 00:00:01] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [10/Jan/2019 00:01:01] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [10/Jan/2019 00:02:02] "GET / HTTP/1.1" 200 -
My question: Would it be possible to make an exception for local request? I don't want to log any request from localhost/127.0.0.1.
I'm thinking something like this but not really sure how to implement it in Python2 yet.
webserver_v2_do_not_log_localhost.py
import webserver.py # webserver code above, or simply paste everything in here.
if SourceIPAddress == 127.0.0.1:
print('DO NOT log request from localhost/127.0.0.1')
# Script here
else:
print('Log everything')
# Script here
Any idea on the scripts would be highly appreciated. Thanks
Desired Output when performing tail -F log.txt (external IP only, not localhost)
10.0.0.1 - - [10/Jan/2019 00:00:00] "GET / HTTP/1.1" 200 -
You can use logging.Filterclass.
When you declare your logger, do something like that:
import logging
logging.basicConfig(filename='myapp.log', level=logging.INFO)
class Global:
SourceIPAddress = ''
class IpFilter(logging.Filter):
def filter(self, rec):#the rec is part of the function signature.
return not Global.SourceIPAddress == '127.0.0.1'
def main():
log = logging.getLogger('myLogger')
log.addFilter(IpFilter())
log.info("log")
Global.SourceIPAddress = '127.0.0.1'
log.info("Don't log")
if __name__ == '__main__':
main()
Of course I implemented it in a very simple way and you should save the IP in a better place(:
I would also check this links for more info:
https://docs.python.org/3/howto/logging-cookbook.html
https://www.programcreek.com/python/example/3364/logging.Filter
I have a Django app with which users can create video collages using multiple videos. Problem is, on production, when uploading videos to amazon s3, I get a 502 bad gateway (works fine locally). Does anyone know what could be wrong? I already set
client_max_body_size 100M
and
fastcgi_buffers 8 16k;
fastcgi_buffer_size 32k;
fastcgi_connect_timeout 3000;
fastcgi_send_timeout 3000;
fastcgi_read_timeout 3000;
Does anyone know what could be wrong? Thanks in advance
Full error:
2017/12/31 23:50:51 [error] 1279#1279: *1 upstream prematurely closed connection while reading response header from upstream,
client: 107.205.110.154,
server: movingcollage.com,
request: "POST /create-collage/ HTTP/1.1",
upstream: "http://unix:/home/mike/movingcollage/movingcollage.sock:/create-collage/",
host: "movingcollage.com", referrer: "http://movingcollage.com/create-collage/"
If the problem were in nginx timeout it would give you 504 error. 502 error means that this error could happen due to timeout in process behind nginx, gunicorn in your case I guess. Try to launch it with -t 3000 param (to match your nginx conf).
In the HTTP protocol you can send many requests in one socket using keep-alive and then receive the response from server at once, so that will significantly speed up whole process. Is there any way to do this in python requests lib? Or are there any other ways to speed this up that well using requests lib?
Yes, there is. Use requests.Session and it will do keep-alive by default.
I guess I should include a quick example:
import logging
import requests
logging.basicConfig(level=logging.DEBUG)
s = requests.Session()
s.get('http://httpbin.org/cookies/set/sessioncookie/123456789')
s.get('http://httpbin.org/cookies/set/anothercookie/123456789')
r = s.get("http://httpbin.org/cookies")
print(r.text)
You will note that these log message occur
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): httpbin.org
DEBUG:requests.packages.urllib3.connectionpool:"GET /cookies/set/sessioncookie/123456789 HTTP/1.1" 302 223
DEBUG:requests.packages.urllib3.connectionpool:"GET /cookies HTTP/1.1" 200 55
DEBUG:requests.packages.urllib3.connectionpool:"GET /cookies/set/anothercookie/123456789 HTTP/1.1" 302 223
DEBUG:requests.packages.urllib3.connectionpool:"GET /cookies HTTP/1.1" 200 90
DEBUG:requests.packages.urllib3.connectionpool:"GET /cookies HTTP/1.1" 200 90
If you wait a little while, and repeat the last get call
INFO:requests.packages.urllib3.connectionpool:Resetting dropped connection: httpbin.org
DEBUG:requests.packages.urllib3.connectionpool:"GET /cookies HTTP/1.1" 200 90
Note that it resets the dropped connection, i.e. reestablishing the connection to the server to make the new request.
I'm curious why when benchmarking Python web server CherryPy using ab, with -c 7 (7 concurrent threads) it can server 1500 requests/s (about what I expect), but when I change to -c 8 it drops way down to 25 requests/s. I'm running CherryPy with numthreads=10 (but it doesn't make a different if I use numthreads=8 or 20) on a 64-bit Windows machine with four cores running Python 2.6.
I'm half-suspecting the Python GIL is part of the issue, but I don't know why it only happens when I get up to 8 concurrently-requesting threads. On a four core machine I'd expect it might change at -c 4, but this is not the case.
I'm using the one-file CherryPy web server that comes with web.py, and here's the WSGI app that I'm testing against:
from web.wsgiserver import CherryPyWSGIServer
def application(environ, start_response):
start_response("200 OK", [("Content-type", "text/plain")])
return ["Hello World!",]
server = CherryPyWSGIServer(('0.0.0.0', 80), application, numthreads=10)
try:
server.start()
except KeyboardInterrupt:
server.stop()
The ab output for 7 and 8 concurrent threads is:
C:\\> ab -n 1000 -c 7 http://localhost/
...
Concurrency Level: 7
Time taken for tests: 0.670 seconds
Complete requests: 1000
Failed requests: 0
Write errors: 0
Total transferred: 130000 bytes
HTML transferred: 12000 bytes
Requests per second: 1492.39 [#/sec] (mean)
Time per request: 4.690 [ms] (mean)
Time per request: 0.670 [ms] (mean, across all concurrent requests)
Transfer rate: 189.46 [Kbytes/sec] received
C:\\> ab -n 1000 -c 8 http://localhost/
...
Concurrency Level: 8
Time taken for tests: 7.169 seconds
Complete requests: 158
Failed requests: 0
Write errors: 0
Total transferred: 20540 bytes
HTML transferred: 1896 bytes
Requests per second: 22.04 [#/sec] (mean)
Time per request: 362.973 [ms] (mean)
Time per request: 45.372 [ms] (mean, across all concurrent requests)
Transfer rate: 2.80 [Kbytes/sec] received
On my linux box, it's due to the retransmission of a TCP packet from ab, although I'm not exactly sure why:
No. Time Source Destination Protocol Info Delta
10682 21.218156 127.0.0.1 127.0.0.1 TCP http-alt > 57246 [SYN, ACK] Seq=0 Ack=0 Win=32768 Len=0 MSS=16396 TSV=17307504 TSER=17306704 WS=6 21.218156
10683 21.218205 127.0.0.1 127.0.0.1 TCP 57246 > http-alt [ACK] Seq=82 Ack=1 Win=513 Len=0 TSV=17307504 TSER=17307504 SLE=0 SRE=1 0.000049
10701 29.306438 127.0.0.1 127.0.0.1 HTTP [TCP Retransmission] GET / HTTP/1.0 8.088233
10703 29.306536 127.0.0.1 127.0.0.1 TCP http-alt > 57246 [ACK] Seq=1 Ack=82 Win=512 Len=0 TSV=17309526 TSER=17309526 0.000098
10704 29.308555 127.0.0.1 127.0.0.1 TCP [TCP segment of a reassembled PDU] 0.002019
10705 29.308628 127.0.0.1 127.0.0.1 TCP 57246 > http-alt [ACK] Seq=82 Ack=107 Win=513 Len=0 TSV=17309526 TSER=17309526 0.000073
10707 29.309718 127.0.0.1 127.0.0.1 TCP [TCP segment of a reassembled PDU] 0.001090
10708 29.309754 127.0.0.1 127.0.0.1 TCP 57246 > http-alt [ACK] Seq=82 Ack=119 Win=513 Len=0 TSV=17309526 TSER=17309526 0.000036
10710 29.309992 127.0.0.1 127.0.0.1 HTTP HTTP/1.1 200 OK (text/plain) 0.000238
10711 29.310572 127.0.0.1 127.0.0.1 TCP 57246 > http-alt [FIN, ACK] Seq=82 Ack=120 Win=513 Len=0 TSV=17309527 TSER=17309526 0.000580
10712 29.310661 127.0.0.1 127.0.0.1 TCP http-alt > 57246 [ACK] Seq=120 Ack=83 Win=512 Len=0 TSV=17309527 TSER=17309527 0.000089
The original "GET" packet wasn't picked up by Wireshark either. For some reason, ab tries to send a request and fails, even though the TCP connection was double-ACk'd just fine. Then the client's TCP stack waits for a few seconds for a packet that was never sent to be ACK'd, and when it sees no ACK, retries and succeeds.
Personally, I wouldn't worry about it. If there's a problem, it's not one with CherryPy. It could be related to the internals of ab, the use of HTTP/1.0 instead of 1.1, the lack of keepalive, the use of localhost instead of a real socket (which simulates some realities of network traffic and ignores others), the use of Windows (wink), other traffic on the same interface, load on the CPU...the list goes on and on.
I have a Python Tornado server sitting behind a nginx frontend. Every now and then, but not every time, I get a 502 error. I look in the nginx access log and I see this:
127.0.0.1 - - [02/Jun/2010:18:04:02 -0400] "POST /a/question/updates HTTP/1.1" 502 173 "http://localhost/tagged/python" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3"
and in the error log:
2010/06/02 18:04:02 [error] 14033#0: *1700 connect() failed (111: Connection refused)
while connecting to upstream, client: 127.0.0.1, server: _,
request: "POST /a/question/updates HTTP/1.1",
upstream: "http://127.0.0.1:8888/a/question/updates", host: "localhost", referrer: "http://localhost/tagged/python"
I don't think any errors show up in the Tornado log. How would you go about debugging this? Is there something I can put in the Tornado or nginx configuration to help debug this?
The line from the error log is very informative in my opinion. It says the connection was refused by the upstream, it contains client IP, Nginx server config, request line, hostname, upstream URL and referrer.
It is pretty clear you must look at the upstream (or firewall) to find out the reason.
In case you'd like to look at how Nginx processes the request, why it chooses specific server and location sections -- there is a beautiful "debug" mode. (Note, your Nginx binary must be built with debugging symbols included). Then:
error_log /path/to/your/error.log debug;
will turn on debugging for all the requests. Debugging information in the error log requires some time to get used to interpret it, but it's worth the efforts.
Do not use this "as is" for high traffic sites! It generates a lot of information and your error log will grow very fast. If you need to debug requests in the production, use debug_connection directive:
events {
debug_connection 1.2.3.4;
}
It turns debugging on for the specific client IP address only.