As some of you may have seen recently I was struggling to get a simple Client/Server TCP connection up and running, where I could switch to TLS on issue of a command, as per the Twisted example at https://twistedmatrix.com/documents/14.0.0/core/howto/ssl.html#starttls-server, but modified to use certificate authentication. Well, the good news is that now works! Thanks to #Jean-PaulCalderone and #Glyph for their help.
However I now want to extend / transform my server side code into an ESMTP server - ie, let the client connect, server then advertises STARTTLS as part of its EHLO response, and if the client then responds with STARTTLS, negotiate and switch to a secure connection. I believe I need to modify the "TLSServer" class in my server code below to be inherited from twisted.mail.smtp.ESMTP instead of twisted.protocols.basic.LineReceiver - but not 100% sure here. Moreover, the API docs for t.m.s.ESMTP as found at http://twistedmatrix.com/documents/12.1.0/api/twisted.mail.smtp.ESMTP.html are a little thin on the ground (lots of undocumented methods).
Can anyone a) tell me if I am right regarding modifing the inheritance of my existing server code, and b) offer up some explanation of methods such as do_EHLO and ext_STARTTLS?
Ultimately I am looking for an SMTP server that will accept (secured) client connections, receive a mail from the client, and dump the mail to file.
My existing server code:
from twisted.internet import ssl, protocol, defer, task, endpoints
from twisted.protocols.basic import LineReceiver
from twisted.python.modules import getModule
from OpenSSL.crypto import load_privatekey, load_certificate, FILETYPE_PEM
class TLSServer(LineReceiver):
def lineReceived(self, line):
print("received: " + line)
if line == "STARTTLS":
print("-- Switching to TLS")
self.sendLine('READY')
self.transport.startTLS(self.factory.options)
def main(reactor):
caCertFile = open("/some/path/to/cacert.pem","r")
certFile = open("/some/path/to/server.crt","r")
keyFile = open("/some/path/to/server.key","r")
caCertData = caCertFile.read()
pKeyData = keyFile.read()
certData = certFile.read()
caCert = ssl.Certificate.loadPEM(caCertData)
cert = load_certificate(FILETYPE_PEM, certData)
pKey = load_privatekey(FILETYPE_PEM, pKeyData)
factory = protocol.Factory.forProtocol(TLSServer)
factory.options = ssl.CertificateOptions(privateKey=pKey, certificate=cert, trustRoot=caCert)
endpoint = endpoints.TCP4ServerEndpoint(reactor, 8000)
endpoint.listen(factory)
return defer.Deferred()
if __name__ == '__main__':
import starttls_server
task.react(starttls_server.main)
Thanks in advance!
You should not have your own TLSServer class. Instead, just use ESMTP directly as your server class. You've already identified the ext_STARTTLS method as interesting - basically, it already implements the logic that your TLSServer has in its lineReceived method.
Rather than doing factory.options = ..., you will want to pass ESMTP a contextFactory argument upon construction. At the simplest, you could do something like this:
sslCtxFactory = ssl.CertificateOptions(...)
factory = Factory.forProtocol(lambda: ESMTP(contextFactory=sslCtxFactory))
I believe that ESMTP advertises STARTTLS as an extension by default - at least, that's what a quick reading of the implementation of ESMTP.extensions() method says to me.
Related
Trying to send email with a certificate file using the following script:
import smtplib
client = smtplib.SMTP(myhost, myport)
client.ehlo()
client.starttls(certfile=mycertfile)
client.ehlo()
client.login(myusername, mypassword)
client.sendmail(sender, receiver, Message)
client.quit()
I get the following error:
SSLError: error:140B0009:SSL routines:SSL_CTX_use_PrivateKey_file:PEM lib
I think documentations (smtplib.html and ssl.html) say I need to provide a private key. I only have the certificate file (base64 PEM format). My devops says that a private key is not required in this case because I do not need to identify the local side of the connection.
Question
Is there a way to send the email without providing the private key? If a private key is required, why?
There are two ways to use SSL/TLS: client authenticated and "basic" where the client is unauthenticated. In client authenticated connections, the both the server and the client send a certificate to the other. In "basic" only the server does.
If you pass neither a certificate nor a keyfile, smtplib will use a basic connection, where the client is authenticated.
If you use a certificate, it will be used for a client authenticated connection. In that case, the server will demand that the client shows it owns the certificate by signing a handshake message. For the client to be able to do that it also needs the private key, which can be either in the certificate file or as a separate keyfile.
Long story short, if you want to use a client certificate, you must also use a key. If not, you can just leave both empty.
OTOH, maybe you have a server certificate file or CA list you want to use with the connection?
In that case you need to pass it to ssl.wrap_socket in the ca_certs parameter. Since you use Python 2.6 there's no easy way to do that with smtplib (Python 3.3+ has a context argument to starttls).
How to solve this depends on your application. For example, if you do not need ssl for anything else, a hackish solution would be to monkey-patch ssl.wrap_socket with one that provides your ca_cert (as well as cert_reqs=CERT_REQUIRED, likely).
A more full blown solution would be to extend smtplib.SMTP with your own variant that does allow passing in those parameters.
Here's a monkey-patch taken from this page:
class SMTPExt(smtplib.SMTP):
"""
This class extends smtplib.SMTP and overrides the starttls method
allowing extra parameters and forwarding them to ssl.wrap_socket.
"""
def starttls(self, keyfile=None, certfile=None, **kwargs):
self.ehlo_or_helo_if_needed()
if not self.has_extn("starttls"):
raise SMTPException("STARTTLS extension not supported by server.")
(resp, reply) = self.docmd("STARTTLS")
if resp == 220:
self.sock = ssl.wrap_socket(self.sock, keyfile, certfile, **kwargs)
self.file = SSLFakeFile(self.sock)
# RFC 3207:
# The client MUST discard any knowledge obtained from
# the server, such as the list of SMTP service extensions,
# which was not obtained from the TLS negotiation itself.
self.helo_resp = None
self.ehlo_resp = None
self.esmtp_features = {}
self.does_esmtp = 0
return (resp, reply)
Using the root cert from requests
This question already has answers here:
Python: Binding Socket: "Address already in use"
(13 answers)
Closed 8 years ago.
I have a twisted python server which runs on port 8080, and i have written different API's which runs on this server. so i want all these API's to run on a single port no. but when i try to use same port all API's for ex : 8081 and run at the same time using python interpreter. at that time i am getting this error : twisted.internet.error.CannotListenError: Couldn't listen on any:8081: [Errno 98] Address already in use.
As i am new to twisted so don't know much of the things and there is no proper documentation on twisted . please someone guide me to solve this error :
Here is the Code snippets :
from twisted.internet import epollreactor
epollreactor.install()
from zope.interface import implements
from twisted.internet import reactor,interfaces
from functools import partial
from pymongo import Connection
import json
from bson.objectid import ObjectId
import server_1
import replacePlus_Space
global data, data_new, datadB, coll_auth, coll_person
class Login_user(server_1.HTTPEchoFactory):
def __init__(self):
server_1.HTTPEchoFactory.__init__(self,"testsite")
server_1.HTTPEchoFactory.initResources(self)
self.initResources()
def initResources(self):
print "in Login"
self.responses["/login_user"] = partial(self.user_login)
# To connect to DB and Check error cases and insert into mongoDB..!!!
def user_login(self, client):
# some functinality..!!!!
d = Login_user()
reactor.listenTCP(8081,d)
reactor.run()
Second code snippet is :
from twisted.internet import epollreactor
epollreactor.install()
from zope.interface import implements
from twisted.internet import reactor,interfaces
from functools import partial
from pymongo import Connection
import json
from bson.objectid import ObjectId
import server_1
import replacePlus_Space
class AddFriendDB(server_1.HTTPEchoFactory):
def __init__(self):
server_1.HTTPEchoFactory.__init__(self,"testsite")
server_1.HTTPEchoFactory.initResources(self)
self.initResources()
def initResources(self):
print "in add friend"
self.responses["/addFriend_DB"] = partial(self.addFriendDB)
# To connect to DB and Check error cases and insert into mongoDB..!!!
def addFriendDB(self, client):
#some functionality here..!!!
d = AddFriendDB()
reactor.listenTCP(8081,d)
reactor.run()
Translating the question
Many of the specifics in your question don't make sense, so I'm translating your question into:
I have a twisted python standalone program which runs on port 8080, and i have written a different standalone program which runs on this server. I want both programs to run on a single port no. When i try to use same port for all programs ex : using port 8081 for both programs. I am getting this error :twisted.internet.error.CannotListenError: Couldn't listen on any:8081: [Errno 98] Address already in use. As i am new to twisted i don't know much of the things and there is no proper documentation on twisted. please someone guide me to solve this error.
Twisted has great documentation
You commented that:
[...] there is no proper documentation on twisted
That statement is blatantly false.
Twisted certainly has proper documentation, and as compared to most frameworks it has excellent documentation. To name just a few of the many high quality documentation sources:
Dave Peticola's (krondo) Twisted Introduction - an excellent and deep introduction to twisted that begins by explaining the technology Twisted is built upon. If you really want to understand twisted, this (long) introduction is the place to start
The high quality and extensive documentation on on Twisted's primary website twistedmatrix.com
The source itself. Twisted has well commented and surprisingly understand source, if the above documentation doesn't teach you what you need figure it out from the code.
What causes "Address already in use"
As previously answered by Bart Friederichs:
Only one process can listen on a certain port. If you start process 1 to listen on port 8081, and then start another process on that same port, you get this error.
It is an error from the TCP stack, not from python or twisted.
This is a fundament truth of TCP/IP stacks across all operating systems (or at least all process based operating systems that could run Python).
The error is your operating system reminding you that when data comes to a IP port the OS/IP-stack was designed to only forward to one process, a process that is implementing an application level protocol on that port. The error happens when a second program attempts to re-register a port some other program has already registered.
Work-arounds in general
Upon running into a port reuse issue like this you have to ask yourself:
Are the two programs even running the same application-level protocol?
If they are the same protocol, does the protocol have routing such that for any given transaction the correct sub-program/routine be identified?
Are the two programs at a different location in the routing?
If they aren't the same protocol (I.E. one was HTTP and the other is FTP) or they are the same protocol but its a protocol that doesn't have routing (I.E. two instances of NTP) then there is no easy way to mix them because your trying to break the laws of IP (and the way that application protocol implementations are built). The only solution will be to encapsulate one (or both) of the protocol(s) into a common protocol that also has application-level routing (I.E. encapsulating FTP inside of HTTP and using the URI to route)
If they are the same protocol, and the protocol provides for a per-transaction routing (I.E. the URI inside of HTTP transactions) and they aren't at the same routing location, then there are easier solutions, namely:
Merge the two application into one.
If they are the same routable protocol but at a different location (I.E. HTTP protocol with the URI for the first program at /login and second program at /addfriend) it should be trivial to pull out the post-routing logic of the two programs/process and merge them into one program that does both functions
Front-end the programs with a redirector (This solution is only easy with HTTP because of the tools available).
If you have HTTP protocol programs that have routing to separate locations, but for some reason its hard to merge the logic together (I.E. one program is written in Java, the other in Python) then you can look at front-ending both programs with a redirector like nginx
The way you would use a redirector in a case like this is to run your two apps on two different unused ports (I.E. 12001, 12002), and then run the redirector on the port you want the service to be on, running it with a config file that will redirect to your two programs via their unique routing locations (via configs like SF: How to redirect requests to a different domain/url with nginx)
Code example
The following three programs illustrate the process of merging two programs into one, so both sets of logic can be accessed from the same port.
Two separate programs:
If you run the following code a web server will be started up on localhost:8081. If you then point your web browser at http://127.0.0.1:8081/blah the blah page will be displayed.
#!/usr/bin/python
from twisted.internet import defer, protocol, reactor # the reactor
from twisted.web.server import Site # make the webserver go
from twisted.web.resource import Resource
class BlahPage(Resource):
idLeaf = True
def render_GET(self, request):
return "<html><body>Blah Page!</body></html>"
class ShutdownPage(Resource):
def render_GET(self, request):
reactor.stop()
webroot = Resource()
webroot.putChild("blah", BlahPage())
webroot.putChild("shutdown", ShutdownPage())
def main():
# Register the webserver (TCP server) into twisted
webfactory = Site(webroot)
reactor.listenTCP(8081, webfactory)
print ("Starting server")
reactor.run()
if __name__ == '__main__':
main()
This code will start a web server on localhost:8082. If you then point your web browser at http://127.0.0.1:8082/foo the foo page will be displayed.
#!/usr/bin/python
from twisted.internet import defer, protocol, reactor # the reactor
from twisted.web.server import Site # make the webserver go
from twisted.web.resource import Resource
class FooPage(Resource):
idLeaf = True
def render_GET(self, request):
return "<html><body>Foo Page!</body></html>"
class ShutdownPage(Resource):
def render_GET(self, request):
reactor.stop()
webroot = Resource()
webroot.putChild("foo", FooPage())
webroot.putChild("shutdown", ShutdownPage())
def main():
# Register the webserver (TCP server) into twisted
webfactory = Site(webroot)
reactor.listenTCP(8082, webfactory)
print ("Starting server")
reactor.run()
if __name__ == '__main__':
main()
Merging the logic
This code is the merger of the two previous programs, as you can see it only required copying a small amount of code to glue both of the above into one that allows for access to http://127.0.0.1:8080/blah and http://127.0.0.1:8080/blah.
#!/usr/bin/python
from twisted.internet import defer, protocol, reactor # the reactor
from twisted.web.server import Site # make the webserver go
from twisted.web.resource import Resource
class BlahPage(Resource):
idLeaf = True
def render_GET(self, request):
return "<html><body>Blah Page!</body></html>"
class FooPage(Resource):
idLeaf = True
def render_GET(self, request):
return "<html><body>Foo Page!</body></html>"
class ShutdownPage(Resource):
def render_GET(self, request):
reactor.stop()
webroot = Resource()
webroot.putChild("foo", FooPage())
webroot.putChild("blah", BlahPage())
webroot.putChild("shutdown", ShutdownPage())
def main():
# Register the webserver (TCP server) into twisted
webfactory = Site(webroot)
reactor.listenTCP(8080, webfactory)
print ("Starting server")
reactor.run()
if __name__ == '__main__':
main()
Only one process can listen on a certain port. If you start process 1 to listen on port 8081, and then start another process on that same port, you get this error.
It is an error from the TCP stack, not from python or twisted.
Fix it either by choosing different ports for each process, or create a forking server.
I have a XMLRPC server running:
#!/usr/bin/env python
from SimpleXMLRPCServer import SimpleXMLRPCServer
import subprocess
import os
def testls():
retcode=subprocess.call(["ls", "/etc"])
return retcode
server = SimpleXMLRPCServer(('192.168.15.20',8888),logRequests=True)
server.register_function(testls)
server.register_introspection_functions()
try:
server.serve_forever()
finally:
server.server_close()
I need to be able to call it from a ruby client. From what I have found. it seems that it should be this:
require xmlrpc/client
server = XMLRPC::Client.new2("http://192.168.15.20:8888/")
But it returns the following, which suggests that it is not able to reach the service.
=> #<XMLRPC::Client:0x2ab0f5caacf8 #parser=nil, #timeout=30, #path="/", #password=nil, #http_header_extra=nil, #use_ssl=false, #host="192.168.15.20", #user=nil, #proxy_port=nil, #auth=nil, #cookie=nil, #create=nil, #port=8888, #http=#<Net::HTTP 192.168.15.20:8888 open=false>, #proxy_host=nil, #http_last_response=nil>
I had better success with the following, but I was not able to invoke the method:
irb(main):069:0> server = XMLRPC::Client::Proxy.new("http://192.168.15.20","8888")
=> #<XMLRPC::Client::Proxy:0x2ab0f5c4ef48 #args=[], #server="http://192.168.15.20", #meth=:call, #prefix="8888.">
irb(main):070:0> s = server.call('system.listMethods')
NoMethodError: undefined method `call' for "http://192.168.15.20":String
from /usr/lib/ruby/1.8/xmlrpc/client.rb:608:in `send'
from /usr/lib/ruby/1.8/xmlrpc/client.rb:608:in `method_missing'
from (irb):70
from :0
What is the appropriate code to invoke a call from Ruby to a Python XML-RPC Server?
I know this is a rather old question, but since it doesn't have a selected answer, I thought I'd share my insights:
In ruby, the class XMLRPC::Client represents a lower level interface to the server:
server = XMLRPC::Client.new( "192.168.15.20", 8888 )
server.call 'system.listMethods'
The class XMLRPC::Client::Proxy is a higher level interface to (parts of) the server that allows for a more natural server method invocation:
system = server.proxy('system')
system.listMethods
Similarly, to access the server's 'root' methods (like those registered via register_function in Python):
main = server.proxy
main.testls
I'm not familiar with xmlrpc in Ruby, but I think you have to pass an XMLRPC::Client Object to XMLRPC::Client::Proxy.
Try stuff like this:
client = XMLRPC::Client.new( "http://192.168.15.20", 8888 )
server = XMLRPC::Client::Proxy.new( client, 8888 )
server.call "system.listMethods"
Maybe it works this way.
I am currently trying to pull together a basic SSL server in twisted. I pulled the following example right off their website:
from twisted.internet import ssl, reactor
from twisted.internet.protocol import Factory, Protocol
class Echo(Protocol):
def dataReceived(self, data):
"""As soon as any data is received, write it back."""
print "dataReceived: %s" % data
self.transport.write(data)
if __name__ == '__main__':
factory = Factory()
factory.protocol = Echo
print "running reactor"
reactor.listenSSL(8080, factory,
ssl.DefaultOpenSSLContextFactory(
"./test/privatekey.pem", "./test/cacert.pem"))
reactor.run()
I then tried to hit this server using firefox by setting the url to https://localhost:8080 yet I receive no response. I do, however, see the data arriving at the server. Any ideas why I'm not getting a response?
You're not sending an http header back to the browser, and you're not closing the connection
You've implemented an SSL echo server here, not an HTTPS server. Use the openssl s_client command to test it interactively, not firefox (or any other HTTP client, for that matter).
Recently I have been playing around with the HTTP Proxy in twisted. After much trial and error I think I finally I have something working. What I want to know though, is how, if it is possible, do I expand this proxy to also be able to handle HTTPS pages? Here is what I've got so far:
from twisted.internet import reactor
from twisted.web import http
from twisted.web.proxy import Proxy, ProxyRequest, ProxyClientFactory, ProxyClient
class HTTPProxyClient(ProxyClient):
def handleHeader(self, key, value):
print "%s : %s" % (key, value)
ProxyClient.handleHeader(self, key, value)
def handleResponsePart(self, buffer):
print buffer
ProxyClient.handleResponsePart(self, buffer)
class HTTPProxyFactory(ProxyClientFactory):
protocol = HTTPProxyClient
class HTTPProxyRequest(ProxyRequest):
protocols = {'http' : HTTPProxyFactory}
def process(self):
print self.method
for k,v in self.requestHeaders.getAllRawHeaders():
print "%s : %s" % (k,v)
print "\n \n"
ProxyRequest.process(self)
class HTTPProxy(Proxy):
requestFactory = HTTPProxyRequest
factory = http.HTTPFactory()
factory.protocol = HTTPProxy
reactor.listenSSL(8001, factory)
reactor.run()
As this code demonstrates, for the sake of example for now I am just printing out whatever is going through the connection. Is it possible to handle HTTPS with the same classes? If not, how should I go about implementing such a thing?
If you want to connect to an HTTPS website via an HTTP proxy, you need to use the CONNECT HTTP verb (because that's how a proxy works for HTTPS). In this case, the proxy server simply connects to the target server and relays whatever is sent by the server back to the client's socket (and vice versa). There's no caching involved in this case (but you might be able to log the hosts you're connecting to).
The exchange will look like this (client to proxy):
C->P: CONNECT target.host:443 HTTP/1.0
C->P:
P->C: 200 OK
P->C:
After this, the proxy simply opens a plain socket to the target server (no HTTP or SSL/TLS yet) and relays everything between the initial client and the target server (including the TLS handshake that the client initiates). The client upgrades the existing socket it has to the proxy to use TLS/SSL (by starting the SSL/TLS handshake). Once the client has read the '200' status line, as far as the client is concerned, it's as if it had made the connection to the target server directly.
I'm not sure about twisted, but I want to warn you that if you implement a HTTPS proxy, a web browser will expect the server's SSL certificate to match the domain name in the URL (address bar). The web browser will issue security warnings otherwise.
There are ways around this, such as generating certificates on the fly, but you'd need the root certificate to be trusted on the browser.