We Keep Coding

Python Tornado Swagger: only one usage of each socket address

I run my proyect on PyCharm last version.
When I run my main.py it runs good and behaivor is desired. But If I restart (or re-run) the main.py, I got error
OSError: [WinError 10048] Only one usage of each socket address (protocol/network address/port) is normally permitted
No other isntances running. Python is the only thing on computer now. No other usages of ports. The only way to skip this error is or restarting computer or deleting every python from task manager.
Also, after stopping service, Swagger is still accesible with browser, I think that this swagger is using the port for main (because they run together) and that is why I can't run my main.py.
One way to reproduce this error is run this MWE, then stop and close PyCharm or just close Pycharm, then try to restart it and the message error comes again, and swagger is still up!. This happen while I'm coding and try to run the new things I wrote.
MWE:
from tornado.ioloop import IOLoop
from tornado_swagger.setup import setup_swagger
from tornado.options import parse_command_line,define,options
from tornado.web import Application,url,RequestHandler
class TestRequests(RequestHandler):
def get(self,var1):
"""
---
tags:
- Testing
summary: minimal working example
description: some dummy text
produces:
- requestsBody:
- application/json
parameters:
- name: var1
in: path
description: dummy tet
required: true
type: string
responses:
200:
description: dummy response for dummy requests here and now
"""
print(f"Reveived GET tests for {var1}")
res = {"result": "hello",
"product": str(var1)+"-foo"}
self.write(res)
class Applications(Application):
urls = [
url(r"/api/(?P<var1>[a-zA-Z0-9]+)/test",TestRequests)
]
def __init__(self):
settings = {'debug': True}
setup_swagger(self.urls,
swagger_url = '/swagger',
api_base_url="/Base",
description = "test swagger",
title = "test title")
super(Applications,self).__init__(self.urls,**settings)
define('port', default='8088', help='Port to listen on')
if __name__ == "__main__":
print("running")
parse_command_line()
app = Applications()
app.listen(options.port)
IOLoop.current().start()

How do I run a local command with fabric 2?

I want to use Fabric and run a command on local, without having to establish any additional connections.
How do I do this in fabric 2? ... documentation seems to miss to give any example.

The design decision to drop the local command in Fabric 2 makes this more difficult, but I was able to simulate it by using Context from Invoke instead of Connection:
from fabric import Connection
from invoke.context import Context
#task
def hostname(c):
c.run('hostname')
#task
def test(c):
conn = Connection('user#host')
hostname(conn)
local_ctx = Context(c.config) # can be passed into #task;
# Connection is a subclass of Context
hostname(local_ctx)

After several different attemps and spending lots of time I found this elegant solution for starting a server (pty=True) and run local commands.
fabfile.py
from fabric import task
#task
def env_test(c):
c.run("env", replace_env=False)
#task
def go(c):
c.run("manage.py runserver", replace_env=False, pty=True)
Please be aware again, these two commands are only meant for local development tasks!
Further Reading: Fabric2 Connections, Upgrading from 1.x

run, sudo, and local are done the same:
from fabric import Connection
cn = Connection('scott#104.131.61.12') # presumes ssh keys were exchanged
cn.run('ls -al') # assuming ssh to linux server - as scott
cn.sudo('whoami') # as root
cn.local('echo ---------- now from local')
cn.local('dir /w') # assuming client is windows

I am adding #TheRealChx101's comment as an answer because I ran into troubles with Connection.local.
Not all environment variables got into the pty, so some of my scripts did not work properly.
With the import from invoke run as local stanza (Invoke's local instead of Fabric's), everything worked fine.

# -*- coding: utf-8 -*-
from fabric import task
from invoke import run as local
#task(default=True)
def testwxmsg(c):
local("pytest --reuse-db --no-migrations tests/weixin/test_release_accrual.py")

This is similar to the answer by #phoibos, but I wanted to show that #task is not needed.
import sys
from fabric import Connection
from invoke.context import Context
target_host=sys.argv[1]
if target_host == 'localhost':
ctx = Context()
else:
ctx = Connection(target_host)
ctx.run('hostname', echo=False, hide=None)
if isinstance(ctx, Connection):
ctx.close()
Local:
> python demo.py localhost
MyComputerName
Remote:
> python demo.py demo.example.com
demo.example.com

Python Pyro4, client dosent see name server which is created in server file

I try to create name server in server file by Pyro4.naming.startNS() method.
My server file looks like this:
my_object = MyClass()
daemon = Pyro4.Daemon()
uri_deamon, ns, br = Pyro4.naming.startNS()
uri = daemon.register(my_object)
ns.nameserver.register("server", uri)
daemon.requestLoop()
And my client:
ns = Pyro4.locateNS()
uri = ns.lookup('server')
my_object=Pyro4.Proxy(uri)
Pyro4.locateNS() never ends.
After I start server file. I try to execute "python -m Pyro4.nsc list" and this command never ends too.
Have you some ideas what is wrong?
Tomek.
SOLUTION:
I needed to use Pyro4.naming.startNSloop() instead of Pyro4.naming.startNS(). Pyro4.naming.startNSloop should be executed in thread.

Using an SSH keyfile with Fabric

How do you configure fabric to connect to remote hosts using SSH keyfiles (for example, Amazon EC2 instances)?

Finding a simple fabfile with a working example of SSH keyfile usage isn't easy for some reason. I wrote a blog post about it (with a matching gist).
Basically, the usage goes something like this:
from fabric.api import *
env.hosts = ['host.name.com']
env.user = 'user'
env.key_filename = '/path/to/keyfile.pem'
def local_uname():
local('uname -a')
def remote_uname():
run('uname -a')
The important part is setting the env.key_filename environment variable, so that the Paramiko configuration can look for it when connecting.

Also worth mentioning here that you can use the command line args for this:
fab command -i /path/to/key.pem [-H [user#]host[:port]]

Another cool feature available as of Fabric 1.4 - Fabric now supports SSH configs.
If you already have all the SSH connection parameters in your ~/.ssh/config file, Fabric will natively support it, all you need to do is add:
env.use_ssh_config = True
at the beginning of your fabfile.

For fabric2 in fabfile use the following:
from fabric import task, Connection
#task
def staging(ctx):
ctx.name = 'staging'
ctx.user = 'ubuntu'
ctx.host = '192.1.1.1'
ctx.connect_kwargs.key_filename = os.environ['ENV_VAR_POINTS_TO_PRIVATE_KEY_PATH']
#task
def do_something_remote(ctx):
with Connection(ctx.host, ctx.user, connect_kwargs=ctx.connect_kwargs) as conn:
conn.sudo('supervisorctl status')
and run it with:
fab staging do_something_remote
UPDATE:
For multiple hosts (one host will do also) you can use this:
from fabric2 import task, SerialGroup
#task
def staging(ctx):
conns = SerialGroup(
'user#10.0.0.1',
'user#10.0.0.2',
connect_kwargs=
{
'key_filename': os.environ['PRIVATE_KEY_TO_HOST']
})
ctx.CONNS = conns
ctx.APP_SERVICE_NAME = 'google'
#task
def stop(ctx):
for conn in ctx.CONNS:
conn.sudo('supervisorctl stop ' + ctx.APP_SERVICE_NAME)
and run it with fab or fab2:
fab staging stop

For me, the following didn't work:
env.user=["ubuntu"]
env.key_filename=['keyfile.pem']
env.hosts=["xxx-xx-xxx-xxx.ap-southeast-1.compute.amazonaws.com"]
or
fab command -i /path/to/key.pem [-H [user#]host[:port]]
However, the following did:
env.key_filename=['keyfile.pem']
env.hosts=["ubuntu#xxx-xx-xxx-xxx-southeast-1.compute.amazonaws.com"]
or
env.key_filename=['keyfileq.pem']
env.host_string="ubuntu#xxx-xx-xxx-xxx.ap-southeast-1.compute.amazonaws.com"

I had to do this today, my .py file was as simple as possible, like the one posted in the answer of #YuvalAdam but still I kept getting prompted for a password...
Looking at the paramiko (the library used by fabric for ssh) log, I found the line:
Incompatible ssh peer (no acceptable kex algorithm)
I updated paramiko with:
sudo pip install paramiko --upgrade
And now it's working.

None of these answers worked for me on py3.7, fabric2.5.0 and paramiko 2.7.1.
However, using the PKey attribute in the documentation does work: http://docs.fabfile.org/en/2.5/concepts/authentication.html#private-key-objects
from paramiko import RSAKey
ctx.connect_kwargs.pkey = RSAKey.from_private_key_file('path_to_your_aws_key')
with Connection(ctx.host, user, connect_kwargs=ctx.connect_kwargs) as conn:
//etc....

As stated above, Fabric will support .ssh/config file settings after a fashion, but using a pem file for ec2 seems to be problematic. IOW a properly setup .ssh/config file will work from the command line via 'ssh servername' and fail to work with 'fab sometask' when env.host=['servername'].
This was overcome by specifying the env.key_filename='keyfile' in my fabfile.py and duplicating the IdentityFile entry already in my .ssh/config.
This could be either Fabric or paramiko, which in my case was Fabric 1.5.3 and Paramiko 1.9.0.

python nose and twisted

I am writing a test for a function that downloads the data from an url with Twisted (I know about twisted.web.client.getPage, but this one adds some extra functionality). Either ways, I want to use nosetests since I am using it throughout the project and it doesn't look appropriate to use Twisted Trial only for this particular test.
So what I am trying to do is something like:
from nose.twistedtools import deferred
#deferred()
def test_download(self):
url = 'http://localhost:8000'
d = getPage(url)
def callback(data):
assert len(data) != 0
d.addCallback(callback)
return d
On localhost:8000 listens a test server. The issue is I always get twisted.internet.error.DNSLookupError
DNSLookupError: DNS lookup failed: address 'localhost:8000' not found: [Errno -5] No address associated with hostname.
Is there a way I can fix this? Does anyone actually uses nose.twistedtools?
Update: A more complete traceback
Traceback (most recent call last):
File "/usr/local/lib/python2.6/dist-packages/nose-0.11.2-py2.6.egg/nose/twistedtools.py", line 138, in errback
failure.raiseException()
File "/usr/local/lib/python2.6/dist-packages/Twisted-9.0.0-py2.6-linux-x86_64.egg/twisted/python/failure.py", line 326, in raiseException
raise self.type, self.value, self.tb
DNSLookupError: DNS lookup failed: address 'localhost:8000' not found: [Errno -5] No address associated with hostname.
Update 2
My bad, it seems in the implementation of getPage, I was doing something like:
obj = urlparse.urlparse(url)
netloc = obj.netloc
and passing netloc to the the factory when I should've passed netloc.split(':')[0]

Are you sure your getPage function is parsing the URL correctly? The error message seems to suggest that it is using the hostname and port together when doing the dns lookup.
You say your getPage is similar to twisted.web.client.getPage, but that works fine for me when I use it in this complete script:
#!/usr/bin/env python
from nose.twistedtools import deferred
from twisted.web import client
import nose
#deferred()
def test_download():
url = 'http://localhost:8000'
d = client.getPage(url)
def callback(data):
assert len(data) != 0
d.addCallback(callback)
return d
if __name__ == "__main__":
args = ['--verbosity=2', __file__]
nose.run(argv=args)
While running a simple http server in my home directory:
$ python -m SimpleHTTPServer
Serving HTTP on 0.0.0.0 port 8000 ...
The nose test gives the following output:
.
----------------------------------------------------------------------
Ran 1 test in 0.019s
OK