We Keep Coding

Integrating HTML+JS for host system display into Python script running in Docker container (linux subsystem)

I have a basic HTML page with some javascript that I want to use to display the results of a python script at repeated steps during its execution, as well as waiting for some user trigger to continue. The webpage consists of an HTML file, a main javascript file, a secondary javascript file generated by the script containing the custom data, and references to external javascript dependencies (e.g. "https://unpkg.com/d3-graphviz#3.0.5/build/d3-graphviz.js".
To complicate things, the python script is running in a Docker container (Linux subsystem) via Visual Studio Code, so some operations like automatically opening an HTML file on the host system are difficult to accomplish (and there also seems to be a problem, with firefox at least, preventing opening local HTML files that reference other local files - see).
My initial plan was to, at each display step in the script, start a simple python http server, serve the page on localhost by piping it through a forwarded port see, wait for the user to open the localhost in their host browser, then once the page has been loaded close the server and continue with the script. I'm sure there is some fancier way of keeping the server up the entire time using threading and passing the updated data but web dev isn't really my field and this isn't exactly production code either so I felt this approach made more sense.
So assuming that this is a valid approach (and if there is a better one, I am more than happy to hear it), here's my best attempt at it (following this guide):
import http.server # Our http server handler for http requests
import socketserver # Establish the TCP Socket connections
from time import sleep # To sleep before shutting down server
PORT = 8000
count = 0
class requestHandler(http.server.SimpleHTTPRequestHandler):
def do_GET(self):
global count
if self.path == "/":
self.path == ".server/"
count += 1
return http.server.SimpleHTTPRequestHandler.do_GET(self)
# Prevent errors when restarting server and address (port) is still in use
socketserver.TCPServer.allow_reuse_address = True
handler = requestHandler
# Start server
with socketserver.TCPServer(("", PORT), handler) as httpd:
# Checking for request count, 3 expected:
# "GET /.server/ HTTP/1.1"
# "GET /.server/dots.js HTTP/1.1"
# "GET /.server/script.js HTTP/1.1"
# so accept 3 requests and end
while count < 3:
httpd.handle_request()
This is called from the main script with exec(open().read()).
Currently I get the error "NameError: name 'http' is not defined" in line 23. Weirdly, I'm sure a couple of days ago this was working fine, and I've just come back to it so I wouldn't have thought anything had changed. I've also tried a bunch of simple http.server examples with and without custom handlers, with and without the with...as block, with various errors from "Loading failed for the with source "http://localhost:8000/.server/dots.js" (in the browser F12 console) to failures to load the entire ./server directory and others. So I'm pretty much out of ideas at this point and am hoping one of you nice people can help me out.
P.S. Sorry for the long post, had to get it all out.

Okay so I seem to have found a rather inelegant but at least functional solution for the time being. For anyone reading in the future here is the server.py file:
import http.server
import socketserver
PORT = 8000
socketserver.TCPServer.allow_reuse_address = True
httpd = socketserver.TCPServer(("", PORT), http.server.SimpleHTTPRequestHandler)
for i in range(3):
httpd.handle_request()
I'll leave the question up for a little longer in case something better comes along, otherwise I'll mark this as a solution.

can you try to change:
with socketserver.TCPServer(("", PORT), handler) as httpd:
to:
with socketserver.TCPServer(("0.0.0.0", PORT), handler) as httpd:
so your application may need to listen on all interfaces?
EDIT:
as I wrote in the comments, using simple webserver such as flask might be easier for you, only to serve static files.
Here is the first post I found: https://code-maven.com/flask-serve-static-files

Failure to change identify using Tor when scraping google

I am trying to automate google search but unfortunately my IP is blocked. After some searches, it seems like using Tor could get me a new IP dynamically. However, after adding the following code block into my existing code, google still blocks my attempts even under the new IP. So I am wondering is there anything wrong with my code?
Code (based on this)
from TorCtl import TorCtl
import socks
import socket
import urllib2
socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5, "127.0.0.1", 9050)
__originalSocket = socket.socket
def newId():
''' Clean circuit switcher
Restores socket to original system value.
Calls TOR control socket and closes it
Replaces system socket with socksified socket
'''
socket.socket = __originalSocket
conn = TorCtl.connect(controlAddr="127.0.0.1", controlPort=9051, passphrase="mypassword")
TorCtl.Connection.send_signal(conn, "NEWNYM")
conn.close()
socket.socket = socks.socksocket
## generate a new ip
newId()
### verify the new ip
print(urllib2.urlopen("http://icanhazip.com/").read())
## run my scrape code
google_scrape()
new error message
<br>Sometimes you may be asked to solve the CAPTCHA if you are using advanced terms that robots are known to use, or sending requests very quickly.
</div>
IP address: 89.234.XX.25X<br>Time: 2017-02-12T05:02:53Z<br>

Google (and many other sites such as "protected" by Cloudflare) filter requests coming via TOR by the IP address of Tor exit nodes. They can do this because the list of IP addresses of Tor exit nodes is public.
Thus changing your identity - which in turn changes your Tor circuit and will likely result in using a different exit node and thus different IP (although the latter two are not guaranteed) - will not work against this block.
For your use case you might consider using VPN instead of Tor, as their IP addresses are less likely to be blocked. Especially if you use non-free VPN.

python graypy simply not sending

import logging
import graypy
my_logger = logging.getLogger('test_logger')
my_logger.setLevel(logging.DEBUG)
handler = graypy.GELFHandler('my_graylog_server', 12201)
my_logger.addHandler(handler)
my_adapter = logging.LoggerAdapter(logging.getLogger('test_logger'),
{ 'username': 'John' })
my_adapter.debug('Hello Graylog2 from John.')
is not working
I think the issue is the url that should send to /gelf
because when I curl from the terminal to my graylog server , it works
curl -XPOST http://my_graylog_server:12201/gelf -p0 -d '{"short_message":"Hello there", "host":"example1111.org", "facility":"test", "_foo":"bar"}'

Open your Graylog Webapp, click 'System'. You will see a list of links on the right. On of them is 'Input', click this one.
Now you have an overview of all running inputs, listening on different ports. On top of the page you can create a new one. There should be a drop box containing certain modes for the input listener (I think 'GELF AMQP' is standard). Change this one to GELF UDP and click 'Launch new input' in the next dialogue you can specify a port for the service.
You also have to set a node where the messages have to be stored. This node as (or should have) the same IP as your whole graylog2 system.
Now you should be able to receive messages

I think you've setup your input stream For Gelf TCP instead of UDP. I set up a TCP Stream and it got the curl message. However my python application wouldn't send to the stream. I then created a Gelf UDP stream and voila! I ran into this same issue configuring my Graylog EC2 Appliance just a few moments ago.
Also make sure firewall/security groups aren't blocking UDP protocol on port 12201 as well.
Good luck, and I hope this is your issue!

Close/restart socket in Python

I'm about to do a simple testimplementation to use PagSeguro (brazilian "PayPal"), and for this I downloaded their Python server to do tests on my localhost. I work on a Mac, have a XAMPP server (Apache and MySQL parts on during my process).
My problem should be very simple for someone who knows Python and sockets etc, and I did found a lot of clues in my information searches. However - with my own poor knowledge in this area, I wasn't able to put two and two together to fix it.
Short question: How do I free a socket (from Terminal) whos program quit before it had closed the socket. Alt - how do I make a Python function for me to call when I want to close the socket and stop/restart the server.
Scenario:
I start the server (In terminal with #: sudo python ./PagSeguroServer.py) and it works fine, I did some of the tests I wanted to. Then, I needed to change some settings for the server, and to make it work i need to restart the server. I resolved by closing the terminal window, but then when I reopen and type the same command to start the server again I get the "socket.error: [Errno 48] Address already in use". Ok I can see why but not how to fix, so I Google that, and find the tip to add
socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
in the code. Looked at the Python class and tried to put it there to my best knowledge (as below). However, didn't solve my problem. Since my "search and puzzle" doesnt seem to be able to help me - I'm now giving it up and posting this customized question instead!
This is part of the server code:
class SecureHTTPServer(HTTPServer):
'''Servidor HTTPS com OpenSSL.'''
def __init__(self, server_address, HandlerClass,fpem):
BaseServer.__init__(self, server_address, HandlerClass)
ctx = SSL.Context(SSL.SSLv23_METHOD)
ctx.use_privatekey_file (fpem)
ctx.use_certificate_file(fpem)
self.socket = SSL.Connection(ctx, socket.socket(self.address_family,
self.socket_type))
self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
self.server_bind()
self.server_activate()
....
def run(HandlerClass = HTTPSHandler, ServerClass = SecureHTTPServer):
'''Roda o servidor'''
server_address = ('', 443) # (address, port)
httpd = ServerClass(server_address, HandlerClass, fpem)
httpd.serve_forever()
if __name__ == '__main__':
run()
Note: One time I actually did managed to reopen it - it was before I added the setsockopt call, and I got the feeling that the socket had closed after a timeout - read about that somewhere. This however doesn't seem to happen again - have waited and tried several times.
Edit: In the end this is how I resolved: Needed to kill the python process that was holding the socket (see comment on jd's answer), then added the KeyboardInterrupt catch to be able to close the socket correctly. Thanks all!

If you're using Ctrl+C to close the process, you could try catching the KeyboardInterrupt exception (or even better, handle the SIGINT signal) in your code and cleanly close the socket before exiting.
Of course, this won't prevent you from killing your application in some other manner and having a stale socket, but you can try to handle those cases as well.

You could create a function that closes the socket properly and register it to be run at program termination using atexit.register(). See http://docs.python.org/library/atexit.html

Use the following command in a terminal window to kill the TCP port 28355:
npx kill-port 28355
In my python socket server script I used the following lines:
import os
os.system("npx kill-port 28355")
This command solves the "Address already in use" error.
It was the only solution that solved the error out of all the other solutions I found, like enabling the SO_REUSEADDR option.

AFAIR SO_REUSEADDR works a bit different.
But what you should start with - when you kill your working server type
netstat -tapdn |grep PORT
and replace PORT with PORT of yoiur application
Than you will get info about state of the socket....

On OSX you can kill the port(e.g. 8000) with:
kill `lsof -i :8000 | grep Python | cut -c8-11`
hth

urllib.urlopen to open page on same port just hangs

I am trying to use urllib.urlopen to open a web page running on the same host and port as the page I am loading it from and it is just hanging.
For example I have a page at: "http://mydevserver.com:8001/readpage.html" and I have the following code in it:
data = urllib.urlopen("http://mydevserver.com:8001/testpage.html")
When I try and load the page it just hangs. However if I move the testpage.html script to a different port on the same host it works fine. e.g.
data = urllib.urlopen("http://mydevserver.com:8002/testpage.html")
Does anyone know why this might be and how I can solve the problem?

A firewall perhaps? Try opening the page from the command line with wget/curl (assuming you're on Linux) or on the browser, with both ports on settings. Furthermore, you could try a packet sniffer to find out what's going on and where the connection gets stuck. Also, if testpage.html is dynamically generated, see if it is hit, check webserver logs if the request shows up there.

Maybe something is already running on port 8001. Does the page open properly with a browser?

You seem to be implying that you are accessing a web page that is scripted in Python. That implies that the Python script is handling the incoming connections, which could mean that since it's already handling the urllib call, it is not available to handle the connection that results from it as well.
Show the code (or tell us what software) you're using to serve these Python scripts.