The following Python2 webserver will log every single request including the one from localhost (127.0.0.1).
webserver.py
import SimpleHTTPServer, SocketServer, sys
Handler = SimpleHTTPServer.SimpleHTTPRequestHandler
port = 80
httpd = SocketServer.TCPServer(("", port), Handler)
sys.stderr = open('/home/user/log.txt', 'w', 1)
httpd.serve_forever()
As example; curl localhost (from the same machine) will produce the following log.
10.0.0.1 - - [10/Jan/2019 00:00:00] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [10/Jan/2019 00:00:01] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [10/Jan/2019 00:01:01] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [10/Jan/2019 00:02:02] "GET / HTTP/1.1" 200 -
My question: Would it be possible to make an exception for local request? I don't want to log any request from localhost/127.0.0.1.
I'm thinking something like this but not really sure how to implement it in Python2 yet.
webserver_v2_do_not_log_localhost.py
import webserver.py # webserver code above, or simply paste everything in here.
if SourceIPAddress == 127.0.0.1:
print('DO NOT log request from localhost/127.0.0.1')
# Script here
else:
print('Log everything')
# Script here
Any idea on the scripts would be highly appreciated. Thanks
Desired Output when performing tail -F log.txt (external IP only, not localhost)
10.0.0.1 - - [10/Jan/2019 00:00:00] "GET / HTTP/1.1" 200 -
You can use logging.Filterclass.
When you declare your logger, do something like that:
import logging
logging.basicConfig(filename='myapp.log', level=logging.INFO)
class Global:
SourceIPAddress = ''
class IpFilter(logging.Filter):
def filter(self, rec):#the rec is part of the function signature.
return not Global.SourceIPAddress == '127.0.0.1'
def main():
log = logging.getLogger('myLogger')
log.addFilter(IpFilter())
log.info("log")
Global.SourceIPAddress = '127.0.0.1'
log.info("Don't log")
if __name__ == '__main__':
main()
Of course I implemented it in a very simple way and you should save the IP in a better place(:
I would also check this links for more info:
https://docs.python.org/3/howto/logging-cookbook.html
https://www.programcreek.com/python/example/3364/logging.Filter
Related
I have 3 python programs, the first 2 programs are ran with
python program1.py --bind localhost --port 8000
python program1.py --bind localhost --port 8080
The third program is ran with
python program3.py http://localhost:8000 http://localhost:8080
The exception is here:
try:
result = getattr(self.agents[agent], fn) \
(*args + (self.credits[agent],))
except socket.timeout:
self.credits[agent] = -1.0 # ensure it is counted as expired
raise TimeCreditExpired
except (socket.error, xmlrpc.client.Fault) as e:
logging.error("Agent %d was unable to play step %d." +
" Reason: %s", agent, self.step, e)
Output when I press CTRL+C on program1:
program1:
127.0.0.1 - - [10/Nov/2021 23:28:43] "POST /RPC2 HTTP/1.1" 200 -
program3:
2021-11-10 23:28:43,296 -- ERROR: Agent 1 was unable to play step 2. Reason: <Fault 1: "<class 'KeyboardInterrupt'"
but how can I get the stacktrace?
Thanks
I have an app that's outputting a JSON through a socket which was working fine in the past, however recently the socket doesn't seem to be establishing and it continuously POST & GET's with the lines:
<user_ip>,<client_ip> - - [17/Jul/2018 12:48:17]"GET /socket.io/?EIO=3&transport=polling......HTTP/1.1" 200 221 0.000000
<user_ip>,<client_ip> - - [17/Jul/2018 12:48:17]"POST /socket.io/?EIO=3&transport=polling......HTTP/1.1" 200 243 0.517600
Additionally, when the WebSocket first tries, I can see from Chromes console the message:
WebSocket connection to ..... failed: Establishing a tunnel via proxy server failed.
I've also had a look through what I think is a similar issue based on socketIO-client but wasn't able to resolve my problem.
Can anyone help with overcoming this connection issue?
Included a full log on running the app below:
Server initialized for eventlet.
* Debugger is active!
* Debugger PIN: 129-744-633
(7616) wsgi starting up on http://0.0.0.0:6328
(7616) accepted (<client_ip>, 50548)
<user_ip>,<client_ip> - - [17/Jul/2018 13:18:23] "GET /<app_url> HTTP/1.1" 200 1664 0.015000
3602c46fa50247eb9d397fda82f3eae8: Sending packet OPEN data {'sid': '3602c46fa50247eb9d397fda82f3eae8', 'upgrades': ['websocket'], 'pingTimeout': 60000, 'pingInterval': 25000}
3602c46fa50247eb9d397fda82f3eae8: Sending packet MESSAGE data 0
<user_ip>,<client_ip> - - [17/Jul/2018 13:18:23] "GET /socket.io/?EIO=3&transport=polling&t=1531829903367-0 HTTP/1.1" 200 381 0.000000
(7616) accepted (<client_ip>, 50560)
3602c46fa50247eb9d397fda82f3eae8: Received packet MESSAGE data 0/testnamespace
3602c46fa50247eb9d397fda82f3eae8: Sending packet MESSAGE data 0/testnamespace
<user_ip>,<client_ip> - - [17/Jul/2018 13:18:23] "POST /socket.io/?EIO=3&transport=polling&t=1531829903377-1&sid=3602c46fa50247eb9d397fda82f3eae8 HTTP/1.1" 200 221 0.000000
<user_ip>,<client_ip> - - [17/Jul/2018 13:18:23] "GET /socket.io/?EIO=3&transport=polling&t=1531829903379-2&sid=3602c46fa50247eb9d397fda82f3eae8 HTTP/1.1" 200 226 0.000000
(7616) accepted (<client_ip>, 50562)
3602c46fa50247eb9d397fda82f3eae8: Received packet MESSAGE data 2/testnamespace,["event_1",{"data":"Web app connection successful."}]
3602c46fa50247eb9d397fda82f3eae8: Sending packet MESSAGE data 2/testnamespace,["client_event",{"json":[{"Date&Time":"........}]
When I run this sample code :
from flask import Flask
app = Flask(__name__)
def main () :
return "Welcome to Flask "
if __name__ == "__main__":
app.run(debug=True, host="0.0.0.0", port=80)
Entering following on the terminal: python3 app.py, the output is:
* Serving Flask app "app" (lazy loading)
* Environment: production
WARNING: Do not use the development server in a production
environment.
Use a production WSGI server instead.
* Debug mode: on
* Running on http://0.0.0.0:80/ (Press CTRL+C to quit)
* Restarting with stat
* Debugger is active!
* Debugger PIN: 144-032-769
127.0.0.1 - - [13/Jun/2018 00:11:59] "GET / HTTP/1.1" 404 -
127.0.0.1 - - [13/Jun/2018 00:11:59] "GET /favicon.ico HTTP/1.1" 404
-
127.0.0.1 - - [13/Jun/2018 00:12:15] "GET / HTTP/1.1" 404 -
127.0.0.1 - - [13/Jun/2018 00:12:16] "GET /favicon.ico HTTP/1.1" 404 -
127.0.0.1 - - [13/Jun/2018 00:12:22] "GET / HTTP/1.1" 404 -
127.0.0.1 - - [13/Jun/2018 00:12:22] "GET /favicon.ico HTTP/1.1" 404 -
It does not stop
And when I open 127.0.0.1 in my browser it says that Not Found
you have not given main() a route decorator. your function should look like this:
#app.route('/')
def main():
return "Welcome to Flask "
Here is a minimal example:
from flask import Flask
app = Flask(__name__)
app.config['DEBUG'] = True
app.config['SERVER_NAME'] = 'myapp.dev:5000'
#app.route('/')
def hello_world():
return 'Hello World!'
#app.errorhandler(404)
def not_found(error):
print(str(error))
return '404', 404
if __name__ == '__main__':
app.run(debug=True)
If I set SERVER_NAME, Flask would response every URL with a 404 error, and when I comment out that line, it functions correctly again.
/Users/sunqingyao/Envs/flask/bin/python3.6 /Users/sunqingyao/Projects/play-ground/python-playground/foo/foo.py
* Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
* Restarting with stat
* Debugger is active!
* Debugger PIN: 422-505-438
127.0.0.1 - - [30/Oct/2017 07:19:55] "GET / HTTP/1.1" 404 -
404 Not Found: The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.
Please note that this is not a duplicate of Flask 404 when using SERVER_NAME, since I'm not using Apache or any production web server. I'm just dealing with Flask's built-in development server.
I'm using Python 3.6.2, Flask 0.12.2, Werkzeug 0.12.2, PyCharm 2017.2.3 on macOS High Sierra, if it's relevant.
When set SERVER_NAME, you should make HTTP request header 'Host' the same with it:
# curl http://127.0.0.1:5000/ -sv -H 'Host: myapp.dev:5000'
* Hostname was NOT found in DNS cache
* Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 5000 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.35.0
> Accept: */*
> Host: myapp.dev:5000
>
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Content-Type: text/html; charset=utf-8
< Content-Length: 13
< Server: Werkzeug/0.14.1 Python/3.6.5
< Date: Thu, 14 Jun 2018 09:34:31 GMT
<
* Closing connection 0
Hello, World!
if you use web explorer， you should access it use http://myapp.dev:5000/ and set /etc/hosts file.
It is like the nginx vhost, use Host header to do routing.
I think The SERVER_NAME is mainly used for route map.
you should set host and ip by hand
app.run(host="0.0.0.0",port=5000)
if you not set host/ip but set SERVER_NAME and found it seems to work,because the app.run() have this logic:
def run(self, host=None, port=None, debug=None,
load_dotenv=True, **options):
...
_host = '127.0.0.1'
_port = 5000
server_name = self.config.get('SERVER_NAME')
sn_host, sn_port = None, None
if server_name:
sn_host, _, sn_port = server_name.partition(':')
host = host or sn_host or _host
port = int(port or sn_port or _port)
...
try:
run_simple(host, port, self, **options)
finally:
self._got_first_request = False
At last, don't use SERVER_NAME to set host,ip app.run() used, unless you know its impact on the route map.
Sometimes I find Flask's docs to be confusing (see the quotes above by #dm295 - the meaning of the implications surrounding 'SERVER_NAME' is hard to parse). But an alternative setup to (and inspired by) #Dancer Phd's answer is to specify the 'HOST' and 'PORT' parameters in a config file instead of 'SERVER_NAME'.
For example, let's say you use this config strategy proposed in the Flask docs, add the host & port number like so:
class Config(object):
DEBUG = False
TESTING = False
DATABASE_URI = 'sqlite://:memory:'
HOST = 'http://localhost' #
PORT = '5000'
class ProductionConfig(Config):
DATABASE_URI = 'mysql://user#localhost/foo'
class DevelopmentConfig(Config):
DEBUG = True
class TestingConfig(Config):
TESTING = True
From Flask docs:
the name and port number of the server. Required for subdomain support
(e.g.: 'myapp.dev:5000') Note that localhost does not support
subdomains so setting this to “localhost” does not help. Setting a
SERVER_NAME also by default enables URL generation without a request
context but with an application context.
and
More on SERVER_NAME
The SERVER_NAME key is used for the subdomain
support. Because Flask cannot guess the subdomain part without the
knowledge of the actual server name, this is required if you want to
work with subdomains. This is also used for the session cookie.
Please keep in mind that not only Flask has the problem of not knowing
what subdomains are, your web browser does as well. Most modern web
browsers will not allow cross-subdomain cookies to be set on a server
name without dots in it. So if your server name is 'localhost' you
will not be able to set a cookie for 'localhost' and every subdomain
of it. Please choose a different server name in that case, like
'myapplication.local' and add this name + the subdomains you want to
use into your host config or setup a local bind.
It looks like there's no point to setting it to localhost. As suggested in the docs, try something like myapp.dev:5000.
You can also use just port number and host inside of the app.run like:
app.run(debug=True, port=5000, host="localhost")
Delete:
app.config['DEBUG'] = True
app.config['SERVER_NAME'] = 'myapp.dev:5000'
Using debug=True worked for me:
from flask import Flask
app = Flask(__name__)
app.config['SERVER_NAME'] = 'localhost:5000'
#app.route('/')
def hello_world():
return 'Hello World!'
if __name__ == '__main__':
app.run(debug=True)
I have a Python Tornado server sitting behind a nginx frontend. Every now and then, but not every time, I get a 502 error. I look in the nginx access log and I see this:
127.0.0.1 - - [02/Jun/2010:18:04:02 -0400] "POST /a/question/updates HTTP/1.1" 502 173 "http://localhost/tagged/python" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3"
and in the error log:
2010/06/02 18:04:02 [error] 14033#0: *1700 connect() failed (111: Connection refused)
while connecting to upstream, client: 127.0.0.1, server: _,
request: "POST /a/question/updates HTTP/1.1",
upstream: "http://127.0.0.1:8888/a/question/updates", host: "localhost", referrer: "http://localhost/tagged/python"
I don't think any errors show up in the Tornado log. How would you go about debugging this? Is there something I can put in the Tornado or nginx configuration to help debug this?
The line from the error log is very informative in my opinion. It says the connection was refused by the upstream, it contains client IP, Nginx server config, request line, hostname, upstream URL and referrer.
It is pretty clear you must look at the upstream (or firewall) to find out the reason.
In case you'd like to look at how Nginx processes the request, why it chooses specific server and location sections -- there is a beautiful "debug" mode. (Note, your Nginx binary must be built with debugging symbols included). Then:
error_log /path/to/your/error.log debug;
will turn on debugging for all the requests. Debugging information in the error log requires some time to get used to interpret it, but it's worth the efforts.
Do not use this "as is" for high traffic sites! It generates a lot of information and your error log will grow very fast. If you need to debug requests in the production, use debug_connection directive:
events {
debug_connection 1.2.3.4;
}
It turns debugging on for the specific client IP address only.