Wildcard support was removed from the giampaolo/pyftpdlib library. Globbing functionality was available until r358. We could go back to r357 to see what was removed and reimplement it. https://github.com/giampaolo/pyftpdlib/issues/106#issuecomment-44425620
Globbing of wildcards is not supported (for example, NLST *.txt will not work). ” - From the compliance doc https://github.com/giampaolo/pyftpdlib/blob/64d629aa480be4340cbab4ced1213211b4eee8db/docs/rfc-compliance.rst
Furthermore, he suggests using MLSD over NLST as a possible solution. https://github.com/giampaolo/pyftpdlib/issues/106#issuecomment-44425620 . My guess is that with modifications to the library the wildcard functionality can be restored.
Does anyone have any clue how to achieve this?
ADDITIONAL DETAILS:
We followed this tutorial and created an FTP server on Ubuntu 14.04 using the awesome pyftpdlib Python library. It was easy to setup the server, add users, and create hooks.
Some clients communicate with our FTP server directly from their server using mget. The mget command works if you specify the filename directly, but passing a wildcard fails. Here is a sample response:
ftp> dir
229 Entering extended passive mode (|||26607|).
125 Data connection already open. Transfer starting.
-rw-r--r-- 1 serverpilot serverpilot 914 Oct 06 19:05 index.php
226 Transfer complete.
ftp> mget *.php
No such file or directory.
ftp> glob
Globbing off.
ftp> mget *.php
mget *.php [anpqy?]? y
229 Entering extended passive mode (|||60975|).
550 No such file or directory.
ftp> mget index.php
mget index.php [anpqy?]? y
229 Entering extended passive mode (|||17945|).
125 Data connection already open. Transfer starting.
100% |***************************************************************************************************************************************************************| 914 763.53 KiB/s 00:00 ETA
226 Transfer complete.
914 bytes received in 00:00 (692.99 KiB/s)
Our script looks like this:
from pyftpdlib.authorizers import DummyAuthorizer
from pyftpdlib.handlers import FTPHandler
from pyftpdlib.servers import FTPServer
# The port the FTP server will listen on.
# This must be greater than 1023 unless you run this script as root.
FTP_PORT = 2121
# The name of the FTP user that can log in.
FTP_USER = "myuser"
# The FTP user's password.
FTP_PASSWORD = "change_this_password"
# The directory the FTP user will have full read/write access to.
FTP_DIRECTORY = "/srv/users/serverpilot/apps/APPNAME/public/"
def main():
authorizer = DummyAuthorizer()
# Define a new user having full r/w permissions.
authorizer.add_user(FTP_USER, FTP_PASSWORD, FTP_DIRECTORY, perm='elradfmw')
handler = FTPHandler
handler.authorizer = authorizer
# Define a customized banner (string returned when client connects)
handler.banner = "pyftpdlib based ftpd ready."
# Optionally specify range of ports to use for passive connections.
#handler.passive_ports = range(60000, 65535)
address = ('', FTP_PORT)
server = FTPServer(address, handler)
server.max_cons = 256
server.max_cons_per_ip = 5
server.serve_forever()
if __name__ == '__main__':
main()
It looks like wildcards were intentionally left out of the library based upon this issue. For reference, here is my own issue as well.
Can anyone provide more insight or guide me to reenabling wildcards?
Related
I have a web user that can read my cheroot + Flask webserver files and can bind to ports 80 and 443 (through authbind). The systemd service is configured to run python through authbind as the web user. Problem is, cheroot takes both the SSL certificate and and key file paths to load them, and the only way I can think to make this work is to make them both readable by the web user. This, and the fact that the web user can bind to the two ports, rings alarms bells in me.
From what I understand after some research, the appropriate way to handle this would be to have another user (let's say web-starter for example) that can bind to the ports and read my private key file start the webserver and then "hand over" control to the web user.
How, and at what point do I do this? Can I do this in Python?
Example code:
webserver.service
...
User=web
Environment="FLASK_ENV=production"
ExecStart=/usr/bin/authbind --deep /usr/bin/python3 [PATH_TO_WEBAPP]/main.py
...
main.py
#! /usr/bin/python3
from cheroot.wsgi import Server as WSGIServer
from cheroot.ssl.builtin import BuiltinSSLAdapter as SSLAdapter
from ssl import TLSVersion
from myflaskapp import create_app
from os import environ
# if environment is 'development', use port 5000 and no ssl
DEVEL = environ.get('FLASK_ENV') == 'development'
if __name__ == '__main__':
port = 5000 if DEVEL else 443
server = WSGIServer(('0.0.0.0', port), create_app())
if not DEVEL:
server.ssl_adapter = SSLAdapter('ssl.crt', 'ssl.key')
server.ssl_adapter.context.minimum_version = TLSVersion.TLSv1_2
try:
server.start()
except KeyboardInterrupt:
server.stop()
PS: Also, is the use of environmental variables safe here? In other words, can a compromised web user affect them, or will systemd manage them?
ssl.key
jk ;)
Also, is there a way to protect writable files, such as SQLite databases?
I'm downloading two files from a C-More industrial HMI FTP server. I don't know what OS the HMI is running but I suspect that its FTP server has some quirks. Using Jython 2.7, one file can be read without difficulty but the other has a space in the file name and the normal wrap-in-quotes solution doesn't work - yet.
The following works in the Windows 10 FTP client.
ftp> get NO_SPACES.csv
200 PORT command successful.
150 Opening ASCII mode data connection for NO_SPACES.csv.
226 Transfer complete.
ftp: 12774 bytes received in 0.27Seconds 47.66Kbytes/sec.
ftp> get "WITH SPACE.csv"
200 PORT command successful.
150 Opening ASCII mode data connection for WITH SPACE.csv.
226 Transfer complete.
ftp: 6328 bytes received in 0.02Seconds 316.40Kbytes/sec.
So far, so good. Now try it in Python:
ftp = FTP(myIP) # Connect.
ftp.login(userName, password) # Login.
ftp.set_pasv(False) # Required by the C-More panel for some reason.
with io.BytesIO() as binary_buffer:
# read all of products into a binary buffer
# ftp.retrbinary("RETR NO_SPACES.csv", binary_buffer.write) # This line works.
ftp.retrbinary('RETR "WITH SPACE.csv"', binary_buffer.write) # This one doesn't.
The script console in my development system reports:
ftplib.error_perm: 550 "WITH SPACE.csv": Requested action not taken.
Filenames have been changed to protect the innocent.
Windows FTP likes the get command. Python seems to favour RETR.
I've tried 'RETR "WITH SPACE.csv"' and "RETR 'WITH SPACE.csv'". Same result.
If I have to I can rename the files in the HMI but that will require some validation and paperwork and that's no fun.
I'm developing this on the latest version of Inductive Automation's Ignition! SCADA system which uses Jython 2.7.
Has anyone got any ideas for me to try?
The ftplib has no issue with spaces. The problem are the quotes you add to the RETR command. There should be no quotes:
ftp.retrbinary('RETR WITH SPACE.csv', binary_buffer.write)
If you enable the debug mode in ftp using the -d switch, you will see that it also sends no quotes to the FTP server in the RETR command:
ftp> get "WITH SPACE.csv"
---> PORT 127,0,0,1,15,145
200 Port command successful
---> RETR WITH SPACE.csv
150 Opening data channel for file download from server of "/WITH SPACE.csv"
226 Successfully transferred "/WITH SPACE.csv"
ftp: 12 bytes received in 0.00Seconds 12000.00Kbytes/sec.
Note that the get is a commandline ftp client user command that translates to the FTP protocol RETR command.
At the moment, my networking team is downloading a firmware version using an FTP server (Filezilla).
We create an account on the server, upload the firmware to the server and then on the cisco device we use the command copy ftp://username:password#server_ip/file_name storage_device: and the device is downloading the firmware from the ftp server.
Now, i have a website created with Flask in python, that has some scripts that my team uses, and i want to add a script that allows the user to upload a file to the website, supply the ip of the device, and the script will connect to the device and pull the file from the server.
While creating an ftp server using pyftpdlib, i have encounterd some issues, as the device is logging in with the supplied user, but does not download the file.
If i log in to the server using an FTP client (FileZilla client) i can download the file separately.
I guess the issue is with the cisco device trying to download the firmware directly while connecting.
The code I used to create the python server:
import os
from pyftpdlib.authorizers import DummyAuthorizer
from pyftpdlib.handlers import FTPHandler
from pyftpdlib.servers import FTPServer
def main():
authorizer = DummyAuthorizer()
authorizer.add_user('username', 'password', '/ftp', perm='elradfmwMT')
handler = FTPHandler
handler.authorizer = authorizer
handler.banner = "Test FTP server"
address = ('0.0.0.0', 21)
server = FTPServer(address, handler)
server.max_cons = 256
server.max_cons_per_ip = 10
server.serve_forever()
if __name__ == '__main__':
main()
If anyone has encounterd any issue like that, help would be appreciated.
Thank you!
Two possible issues:
1. Watch the diskspace on the IOS device. Some platforms have very little space for extra images, so good image hygiene is essential.
2. In my various experiments, I've found that pushing the image to the device is generally more reliable than asking the device to pull the image from elsewhere. You may need to add a configuration entry "ip scp server enable". Then you can write your script to push the images diretly.
I am trying to automate to collect the logs from the Cisco Call Manager via CLI by using the from paramiko_expect import SSHClientInteraction where I am not able to send the interactive command to the server.
While trying to download the logs, it will ask information like SFTP IP address, username, password and directory which needs to send an interactive command.
whenever the code runs, it stops at the interactive command section where its not sending the command to the server because of which python script stops here. need to know is there any other way to code these requirements.
for example
Below section is interactive shell where I have to type y/xx.xx.xx.xx/22/User ID/Password/Directory but I can't do the same.
I need help here.. to send the command
+++++++++++++++++++++++++++++++++
Would you like to proceed [y/n]? y
SFTP server IP: xx.xx.xx.xx
SFTP server port [22]: 22
User ID: *****
Password: *****
Download directory: /
+++++++++++++++++++++++++++++++++
Command Line Interface is starting up, please wait ...
Welcome to the Platform Command Line Interface
VMware Installation:
4 vCPU: Intel(R) Xeon(R) Platinum 8180 CPU # 2.50GHz
Disk 1: 110GB, Partitions aligned
6144 Mbytes RAM
admin:file get activelog /syslog/AlternateSyslog
Please wait while the system is gathering files info ...
Get file: active/syslog/AlternateSyslog
done.
Sub-directories were not traversed.
Number of files affected: 5
Total size in Bytes: 23354752
Total size in Kbytes: 22807.375
Would you like to proceed [y/n]? y
SFTP server IP: xx.xx.xx.xx
SFTP server port [22]:
User ID: *****
Password: *****
Download directory: /
The authenticity of host 'xx.xx.xx.xx (xx.xx.xx.xx)' can't be established.
Are you sure you want to continue connecting (yes/no)? yes
.....
Transfer completed.
admin:
I am able to get the show command output but not able to download the logs.
#!/usr/bin/python
# PSFL license
# Importing SSHClientInteraction from paramiko
import paramiko
from paramiko_expect import SSHClientInteraction
import threading
# Specify connection info for each node in square brackets: ["IP ADDRESS", "USERNAME", "PASSWORD"]
connection = [["xx.xx.xx.xx", "userid", "password"]]
# Define function which is responsible for opening SSH connection and running specified commands
def cucm(ip, username, password):
sshsession = paramiko.SSHClient()
sshsession.set_missing_host_key_policy(paramiko.AutoAddPolicy())
sshsession.connect(ip, username=username, password=password)
# "display=True" is just to show you what script does in real time. While in production you can set it to False
interact = SSHClientInteraction(ssh, timeout=600, display=True)
# program will wait till session is established and CUCM returns admin prompt
interact.expect('admin:')
# program runs show status command
interact.send('show status')
# program waits for show status command to finish (this happen when CUCM returns admin prompt)
interact.except('admin:')
# program sends syslog to download the file
interact.send('file get activelog /syslog/AlternateSyslog')
if interact.last_match == 'Would you like to proceed [y/n]? ': # program matches prompted command by using if command and will send interact command to it.
interact.send('y')
if interact.last_match == 'SFTP server IP:':
interact.send('xx.xx.xx.xx')
if interact.last_match == 'SFTP server port [22]:':
interact.send('22')
if interact.last_match == 'User ID:':
interact.send('userid')
if interact.last_match == 'Password:':
interact.send('password')
if interact.last_match == 'Download directory:':
interact.send('/')
interact.expect('admin:')
output = interact.current_output_clean # program saves output of show status command to the "output" variable
sshsession.close()
# Run loop which will open separate thread for each node specified in the connection list. This targets "session" function defined at the beginning
for i in connection:
t = threading.Thread(target = cucm, args = (i[0], i[1], i[2]))
t.daemon = True
t.start()
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Below is the output for the python script.
there is no error message but it stops at Would you like to proceed [y/n]? here
Command Line Interface is starting up, please wait ...
Welcome to the Platform Command Line Interface
VMware Installation:
4 vCPU: Intel(R) Xeon(R) Platinum 8180 CPU # 2.50GHz
Disk 1: 110GB, Partitions aligned
6144 Mbytes RAM
admin:file get activelog /syslog/AlternateSyslog
Please wait while the system is gathering files info ...
Get file: active/syslog/AlternateSyslog
done.
Sub-directories were not traversed.
Number of files affected: 1
Total size in Bytes: 2261400
Total size in Kbytes: 2208.3984
Would you like to proceed [y/n]?
You could try adding the global configuration command "file prompt quiet" at the beginning of your program before any other commands are sent. This will suppress any yes/no questions and auto them to the default. Just make sure that at the end of the code you turn it back off to prevent any later nasty surprises using "file prompt alert".
This works in most Cisco IOS platforms, if the command is different in CUCM I'm sure there will be an equivalent to do the same thing.
maybe you already sorted this out, but I see, that you have there one small type, which could stop that script of moving forward:
you have there:
interact.except('admin:')
instead of:
interact.expect('admin:')
I need to create a test case for a FTP client that involves connecting to a server that only accepts 'active' FTP connections. For other cases I am using pyftpdlib, and it works like charm, but I can't see an easy way to configure it to behave just in FTP active mode, and not passive.
Thanks.
If by what you wrote in the title you mean "literaly disable PASV (passive)" mode you can just tell pyftpdlib to not interpret that command. Not tested:
from pyftpdlib.ftpserver import FTPHandler
handler = FTPHandler
del handler.proto_cmds['PASV']
del handler.proto_cmds['EPSV']
...
This way pyftpdlib will reject any PASV/EPSV request with "550 Command PASV not understood.".