I want to use twisted for some basic FTP server, just like this example:
from twisted.protocols.ftp import FTPFactory, FTPRealm
from twisted.cred.portal import Portal
from twisted.cred.checkers import AllowAnonymousAccess, FilePasswordDB
from twisted.internet import reactor
#pass.dat looks like this:
# jeff:bozo
# grimmtooth:bozo2
p = Portal(FTPRealm('./'), (AllowAnonymousAccess(), FilePasswordDB("pass.dat")))
f = FTPFactory(p)
reactor.listenTCP(21, f)
reactor.run()
...with one simple customization: I want to fire an event when a file upload (STOR) is completed successfully, so that my custom code can adequately handle this file.
I found no documentation for FTPFactory or FTP that helps me doing this. Should I overload the FTP object or some other object? How to wire everything up?
I have done simple custom HTTP servers with twisted in the past and it was pleasantly easy, but I can find nearly no material about FTP.
First off, this is just a modification of Rakis' answer. Without his answer this would not exist. His one just wouldn't work on my setup. It also may just be that the API has changed, since this is 5 years later.
class MyFTP (ftp.FTP):
def ftp_STOR(self, path):
d = super(MyFTP, self).ftp_STOR(path)
def onStorComplete(d):
print 'STORED', repr(d), path
return d
d.addCallback(onStorComplete)
return d
f = ftp.FTPFactory( some_portal_object )
f.protocol = MyFTP
It looks like the following may do the trick
from twisted.protocols import ftp
class MyFTP (ftp.FTP):
def ftp_STOR(self, path):
d = super(MyFTP, self).ftp_STOR(path)
d.addCallback( lambda _: self.onStorComplete(path) )
return d
def onStorComplete(self, path):
# XXX your code here
f = ftp.FTPFactory( some_portal_object )
f.protocol = MyFTP
Related
I am writing a simple commandline application - transfer.py - to allow for uploading and downloading files from the transfer.sh service as a learning exercise, using the 'requests' library for HTTP. Thanks to some answers on here, I was able to implement a progress bar using python-clint and python-requests for monitoring the file download - said functionality being seen here.
Anyway, I got very, very lost when trying to implement the same kind of progress bar to monitor the upload - which uses HTTP PUT. I understand conceptually it should be very similar, but cannot for some reason figure it out, and would be very thankful if someone could point me in the right direction on this. I tried a few methods using multipart encoders and suchlike, but those lead to the file being mangled on the way up (the service accepts raw PUT requests, and multipart encoding messes it up seemingly).
The end goal is to write a script to AES encrypt the file to be uploaded with a random key, upload it to the service, and print a link + encryption key that can be used by a friend to download/decrypt the file, mostly for fun and to fill in some knowledge-gaps in my python.
I recommend you use the requests_toolbelt with the clint.textui.progress module. I found this code which will do.
from clint.textui.progress import Bar as ProgressBar
from requests_toolbelt import MultipartEncoder, MultipartEncoderMonitor
import requests
def create_callback(encoder):
encoder_len = encoder.len
bar = ProgressBar(expected_size=encoder_len, filled_char='=')
def callback(monitor):
bar.show(monitor.bytes_read)
return callback
def create_upload():
return MultipartEncoder({
'form_field': 'value',
'another_form_field': 'another value',
'first_file': ('progress_bar.py', open(__file__, 'rb'), 'text/plain'),
'second_file': ('progress_bar.py', open(__file__, 'rb'),
'text/plain'),
})
if __name__ == '__main__':
encoder = create_upload()
callback = create_callback(encoder)
monitor = MultipartEncoderMonitor(encoder, callback)
r = requests.post('https://httpbin.org/post', data=monitor,
headers={'Content-Type': monitor.content_type})
print('\nUpload finished! (Returned status {0} {1})'.format(
r.status_code, r.reason
))
The following code should work for you:
import requests
import os
from tqdm import tqdm
from tqdm.utils import CallbackIOWrapper
def upload_from_file(src, dst):
file_size = os.path.getsize(src)
with open(src, "rb") as fd:
with tqdm(desc=f"Uploading", total=file_size, unit="B", unit_scale=True, unit_divisor=1024) as t:
reader_wrapper = CallbackIOWrapper(t.update, fd, "read")
response = requests.put(dst, data=reader_wrapper)
response.raise_for_status()
SRC = '/path/to/file'
DST = '/url/to/upload'
upload_from_file(SRC, DST)
Just define your own SRC and DST variables.
Then you can just copy and paste the code.
You can try to use DST='http://httpbin.org/put' for testing.
Enjoy!
Following advice from Jean-Paul Calderone here on SO, I'm trying to modify the twisted "starttls_server" sample below to support the use of ssl.ClientCertificateOptions, to allow me to specify my private key, certificate, and trusted roots, as per http://twistedmatrix.com/documents/14.0.0/api/twisted.internet.ssl.CertificateOptions.html
from twisted.internet import ssl, protocol, defer, task, endpoints
from twisted.protocols.basic import LineReceiver
from twisted.python.modules import getModule
class TLSServer(LineReceiver):
def lineReceived(self, line):
print("received: " + line)
if line == "STARTTLS":
print("-- Switching to TLS")
self.sendLine('READY')
self.transport.startTLS(self.factory.options)
def main(reactor):
certData = getModule(__name__).filePath.sibling('server.pem').getContent()
cert = ssl.PrivateCertificate.loadPEM(certData)
factory = protocol.Factory.forProtocol(TLSServer)
factory.options = cert.options()
endpoint = endpoints.TCP4ServerEndpoint(reactor, 8000)
endpoint.listen(factory)
return defer.Deferred()
if __name__ == '__main__':
import starttls_server
task.react(starttls_server.main)
My understanding is that I effectively need to replace the cert = ssl.PrivateCertificate... and cert.options = ssl.PrivateCertificate.... lines with something like certopts = ssl.CertificateOptions(privateKey=pKeyData, certificate=certData, trustRoot=caCertsData) (having read the appropriate files in to certData, caCertsData, and pKeyData) and then pass this in to factory.options - but without pasting every variant of code I've tried, I've yet to work this out correctly - my efforts have produced varying results from the classic "OpenSSL.crypto.Error: []" - through to seemingly just dumping the contents of my 3 PEM files to screen and exiting!
Can anyone enlighten me? Thank you :)
cert.options() is already returning a CertificateOptions. The problem is that options takes authorities (as Certificate objects) as positional args, and doesn't let you pass through all the other configuration values through, so you probably want to construct a CertificateOptions directly.
Just change the factory.options = cert.options() line to factory.options = ssl.CertificateOptions(...).
However, CertificateOptions takes a pyOpenSSL PKey object as its privateKey, not the key data. So you'll need to use OpenSSL APIs to load that key, or you can extract it from a PrivateCertificate.
If you read the signature of CertificateOptions very carefully, the required types should be fairly clear. You may also need to consult the pyOpenSSL documentation as well.
PS: I have a similar question with Requests HTTP library here.
I am using python v2.7 on windows 7 OS. I am using urllib2 module. I have two code snippets. One file is named as myServer.py The server class has 2 methods named as getName(self,code) and getValue(self).
The other script named as testServer.py simply calls the methods from the server class to retrieve the values and prints them. The server class basically retrieves the values from a Server in my local network. So, unfortunately I can't provide you the access for testing the code.
Problem: When I execute my testServer.py file, I observed in the task manager that the memory consumption keeps increasing. Why is it increasing and how to avoid it? If I comment out the following line
print serverObj.getName(1234)
in testServer.py then there is no increase in memory consumption.
I am sure that the problem is with the getName(self,code) of the server class. But unfortunately, I couldn't figure out what the problem is.
Code: Please find the code snippets below:
#This is the myServer.py file
import urllib2
import json
import random
class server():
def __init__(self):
url1 = 'https://10.0.0.1/'
username = 'user'
password = 'passw0rd'
passwrdmgr = urllib2.HTTPPasswordMgrWithDefaultRealm()
passwrdmgr.add_password(None, url1, username, password)
authhandler = urllib2.HTTPBasicAuthHandler(passwrdmgr)
opener = urllib2.build_opener(authhandler)
urllib2.install_opener(opener)
def getName(self, code):
code = str(code)
url = 'https://10.0.0.1/' + code
response = urllib2.urlopen(url)
data = response.read()
name = str(data).strip()
return name
def getValue(self):
value = random.randrange(0,11)
return value
The following is the testServer.py snippet
from myServer import server
import time
serverObj = server()
while True:
time.sleep(1)
print serverObj.getName(1234)
print serverObj.getValue()
Thank you for your time!
This is question is quite similar to my other question. So I think the answer is also quite similar. The answer can be found here https://stackoverflow.com/a/23172330/2382792
a fews days ago, i tried to learn the python twisted..
and this is how i make my webserver :
from twisted.application import internet, service
from twisted.web import static, server, script
from twisted.web.resource import Resource
import os
class NotFound(Resource):
isLeaf=True
def render(self, request):
return "Sorry... the page you're requesting is not found / forbidden"
class myStaticFile(static.File):
def directoryListing(self):
return self.childNotFound
#root=static.file(os.getcwd()+"/www")
root=myStaticFile(os.getcwd()+"/www")
root.indexNames=['index.py']
root.ignoreExt(".py")
root.processors = {'.py': script.ResourceScript}
root.childNotFound=NotFound()
application = service.Application('web')
sc = service.IServiceCollection(application)
i = internet.TCPServer(8080, server.Site(root))##UndefinedVariable
i.setServiceParent(sc)
in my code, i make an instance class for twisted.web.static.File and override the directoryListing.
so when user try to access my resource folder (http://localhost:8080/resource/ or http://localhost:8080/resource/css), it will return a notFound page.
but he can still open/read the http://localhost:8080/resource/css/style.css.
it works...
what i want to know is.. is this the correct way to do that???
is there another 'perfect' way ?
i was looking for a config that disable directoryListing like root.dirListing=False. but no luck...
Yes, that's a reasonable way to do it. You can also use twisted.web.resource.NoResource or twisted.web.resource.Forbidden instead of defining your own NotFound.
I want to change the env.hosts dynamically because sometimes I want to deploy to one machine first, check if ok then deploy to many machines.
Currently I need to set env.hosts first, how could I set the env.hosts in a method and not in global at script start?
Yes you can set env.hosts dynamically. One common pattern we use is:
from fabric.api import env
def staging():
env.hosts = ['XXX.XXX.XXX.XXX', ]
def production():
env.hosts = ['YYY.YYY.YYY.YYY', 'ZZZ.ZZZ.ZZZ.ZZZ', ]
def deploy():
# Do something...
You would use this to chain the tasks such as fab staging deploy or fab production deploy.
Kind of late to the party, but I achieved this with ec2 like so (note in EC2 you do not know what the ip/hostname may be, generally speaking - so you almost have to go dynamic to really account for how the environment/systems could come up - another option would be to use dyndns, but then this would still be useful):
from fabric.api import *
import datetime
import time
import urllib2
import ConfigParser
from platform_util import *
config = ConfigParser.RawConfigParser()
#task
def load_config(configfile=None):
'''
***REQUIRED*** Pass in the configuration to use - usage load_config:</path/to/config.cfg>
'''
if configfile != None:
# Load up our config file
config.read(configfile)
# Key/secret needed for aws interaction with boto
# (anyone help figure out a better way to do this with sub modules, please don't say classes :-) )
global aws_key
global aws_sec
aws_key = config.get("main","aws_key")
aws_sec = config.get("main","aws_sec")
# Stuff for fabric
env.user = config.get("main","fabric_ssh_user")
env.key_filename = config.get("main","fabric_ssh_key_filename")
env.parallel = config.get("main","fabric_default_parallel")
# Load our role definitions for fabric
for i in config.sections():
if i != "main":
hostlist = []
if config.get(i,"use-regex") == 'yes':
for x in get_running_instances_by_regex(aws_key,aws_sec,config.get(i,"security-group"),config.get(i,"pattern")):
hostlist.append(x.private_ip_address)
env.roledefs[i] = hostlist
else:
for x in get_running_instances(aws_key,aws_sec,config.get(i,"security-group")):
hostlist.append(x.private_ip_address)
env.roledefs[i] = hostlist
if config.has_option(i,"base-group"):
if config.get(i,"base-group") == 'yes':
print "%s is a base group" % i
print env.roledefs[i]
# env["basegroups"][i] = True
where get_running_instances and get_running_instances_by_regex are utility functions that make use of boto (http://code.google.com/p/boto/)
ex:
import logging
import re
from boto.ec2.connection import EC2Connection
from boto.ec2.securitygroup import SecurityGroup
from boto.ec2.instance import Instance
from boto.s3.key import Key
########################################
# B-O-F get_instances
########################################
def get_instances(access_key=None, secret_key=None, security_group=None):
'''
Get all instances. Only within a security group if specified., doesnt' matter their state (running/stopped/etc)
'''
logging.debug('get_instances()')
conn = EC2Connection(aws_access_key_id=access_key, aws_secret_access_key=secret_key)
if security_group:
sg = SecurityGroup(connection=conn, name=security_group)
instances = sg.instances()
return instances
else:
instances = conn.get_all_instances()
return instances
Here is a sample of what my config looked like:
# Config file for fabric toolset
#
# This specific configuration is for <whatever> related hosts
#
#
[main]
aws_key = <key>
aws_sec = <secret>
fabric_ssh_user = <your_user>
fabric_ssh_key_filename = /path/to/your/.ssh/<whatever>.pem
fabric_default_parallel = 1
#
# Groupings - Fabric knows them as roledefs (check env dict)
#
# Production groupings
[app-prod]
security-group = app-prod
use-regex = no
pattern =
[db-prod]
security-group = db-prod
use-regex = no
pattern =
[db-prod-masters]
security-group = db-prod
use-regex = yes
pattern = mysql-[d-s]01
Yet another new answer to an old question. :) But I just recently found myself attempting to dynamically set hosts, and really have to disagree with the main answer. My idea of dynamic, or at least what I was attempting to do, was take an instance DNS-name that was just created by boto, and access that instance with a fab command. I couldn't do fab staging deploy, because the instance doesn't exist at fabfile-editing time.
Fortunately, fabric does support a truly dynamic host-assignment with execute. (It's possible this didn't exist when the question was first asked, of course, but now it does). Execute allows you to define both a function to be called, and the env.hosts it should use for that command. For example:
def create_EC2_box(data=fab_base_data):
conn = boto.ec2.connect_to_region(region)
reservations = conn.run_instances(image_id=image_id, ...)
...
return instance.public_dns_name
def _ping_box():
run('uname -a')
run('tail /var/log/cloud-init-output.log')
def build_box():
box_name = create_EC2_box(fab_base_data)
new_hosts = [box_name]
# new_hosts = ['ec2-54-152-152-123.compute-1.amazonaws.com'] # testing
execute(_ping_box, hosts=new_hosts)
Now I can do fab build_box, and it will fire one boto call that creates an instance, and another fabric call that runs on the new instance - without having to define the instance-name at edit-time.