I have a remote server with some files.
smb://ftpsrv/public/
I can be authorized there as an anonymous user. In java I could simply write this code:
SmbFile root = new SmbFile(SMB_ROOT);
And get the ability to work with files inside (it is all I need, one row!), but I can't find how to manage with this task in Python 3, there are a lot of resources, but I think they are not relevant to my problem, because they are frequently tailored for Python 2, and old other approaches. Is there some simple way, similar to Java code above?
Or can somebody provide a real working solution if, for example, I want to access file fgg.txt in smb://ftpsrv/public/ folder. Is there really a handy lib to tackle this problem?
For example on site:
import tempfile
from smb.SMBConnection import SMBConnection
# There will be some mechanism to capture userID, password, client_machine_name, server_name and server_ip
# client_machine_name can be an arbitary ASCII string
# server_name should match the remote machine name, or else the connection will be rejected
conn = SMBConnection(userID, password, client_machine_name, server_name, use_ntlm_v2 = True)
assert conn.connect(server_ip, 139)
file_obj = tempfile.NamedTemporaryFile()
file_attributes, filesize = conn.retrieveFile('smbtest', '/rfc1001.txt', file_obj)
# Retrieved file contents are inside file_obj
# Do what you need with the file_obj and then close it
# Note that the file obj is positioned at the end-of-file,
# so you might need to perform a file_obj.seek() if you need
# to read from the beginning
file_obj.close()
Do I seriously need to provide all of these details: conn = SMBConnection(userID, password, client_machine_name, server_name, use_ntlm_v2 = True)?
A simple example of opening a file using urllib and pysmb in Python 3
import urllib
from smb.SMBHandler import SMBHandler
opener = urllib.request.build_opener(SMBHandler)
fh = opener.open('smb://host/share/file.txt')
data = fh.read()
fh.close()
I haven't got an anonymous SMB share ready to test it with, but this code should work.
urllib2 is the python 2 package, in python 3 it was renamed to just urllib and some stuff got moved around.
I think you were asking for Linux, but for completeness I'll share how it works on Windows.
On Windows, it seems that Samba access is supported out of the box with Python's standard library functions:
import glob, os
with open(r'\\USER1-PC\Users\Public\test.txt', 'w') as f:
f.write('hello') # write a file on a distant Samba share
for f in glob.glob(r'\\USER1-PC\Users\**\*', recursive=True):
print(f) # glob works too
if os.path.isfile(f):
print(os.path.getmtime(f)) # we can get filesystem information
Related
I have a small but mysterious and unsolvable problem using python to open a password protected file in an AWS S3 bucket.
The password I have been given is definitely correct and I can download the zip to Windows and extract it to reveal the csv data I need.
However I need to code up a process to load this data into a database regularly.
The password has a pattern like this (includes mixed case letters, numbers and a single "#"):-
ABCD#Efghi12324567890
The code below works with other zip files I place in the location with the same password:-
import boto3
import pyzipper
from io import BytesIO
s3_resource = boto3.resource('s3', aws_access_key_id=aws_access_key_id, aws_secret_access_key=aws_secret_access_key)
zip_obj = s3_resource.Object(bucket_name=my_bucket, key=my_folder + my_zip)
buffer = BytesIO(zip_obj.get()["Body"].read())
z = pyzipper.ZipFile(buffer)
my_newfile=z.namelist()[0]
s3_resource.meta.client.upload_fileobj(
z.open(my_newfile, pwd=b"ABCD#Efghi12324567890"), #HERE IS THE OPEN COMMAND
Bucket=my_bucket,
Key=my_folder + my_newfile)
I am told the password is incorrect:-
RuntimeError: Bad password for file 'ThisIsTheFileName.csv'
I resorted to using pyzipper rather than zipfile, since zipfile didn't support the compression method of the file in question:-
That compression method is not supported
In 7-zip I can see the following for the zip file:-
Method: AES-256 Deflate
Characteristics: WzAES: Encrypt
Host OS: FAT
So to confirm:-
-The password is definitely correct (can open it manually)
-The code seems ok - it opens my zip files with the same password
What is the issue here please and how do I fix it?
You would have my sincere thanks!
Phil
With some help from a colleague and a useful article, I now have this working.
Firstly as per the compression type, I have found it necessary to use the AESZipFile() method of pyzipper (although this method also seemed to work on other compression types).
Secondly the AESZipFile() method apparently accepts a BytesIO object as well as a file path, presumably because this is what it sees when it opens the file.
Therefore the zip file can be extracted in situ without having to download it first.
This method creates the pyzipper object which you can then read by specifying the file name and the password.
The final code looks like this:-
import pyzipper
import boto3
from io import BytesIO
my_bucket = ''
my_folder = ''
my_zip = ''
my_password = b''
aws_access_key_id=''
aws_secret_access_key=''
s3 = boto3.client('s3', aws_access_key_id=aws_access_key_id, aws_secret_access_key=aws_secret_access_key)
s3_file = s3.get_object(Bucket=my_bucket, Key=my_folder + my_zip)
s3_iodata = BytesIO(s3_file['Body'].read())
f = pyzipper.AESZipFile(s3_iodata)
my_file = f.namelist()[0]
file_content = f.read(my_file, pwd = my_password)
response = s3.put_object(Body=file_content, Bucket=my_bucket, Key=my_folder + my_file)
Here is an article that was useful:-
https://www.linkedin.com/pulse/extract-files-from-zip-archives-in-situ-aws-s3-using-python-tom-reid
I hope this is helpful to someone,
Phil
Have a peculiar issue that I can't seem to fix on my own..
I'm attempting to FTP a list of files in a directory over to an iSeries IFS using Python's ftplib library.
Note, the files are in a single subdirectory down from the python script.
Below is an excerpt of the code that is giving me trouble:
from ftplib import FTP
import os
localpath = os.getcwd() + '/Files/'
def putFiles():
hostname = 'host.name.com'
username = 'myuser'
password = 'mypassword'
myftp = FTP(hostname)
myftp.login(username, password)
myftp.cwd('/STUFF/HERE/')
for file in os.listdir(localpath):
if file.endswith('.csv'):
try:
file = localpath + file
print 'Attempting to move ' + file
myftp.storbinary("STOR " + file, open(file, 'rb'))
except Exception as e:
print(e)
The specific error that I am getting throw is:
Attempting to move /home/doug/Files/FILE.csv
426-Unable to open or create file /home/doug/Files to receive data.
426 Data transfer ended.
What I've done so far to troubleshoot:
Initially I thought this was a permissions issue on the directory containing my files. I used chmod 777 /home/doug/Files and re-ran my script, but the same exception occured.
Next I assumed there was an issue between my machine and the iSeries. I validated that I could indeed put files by using ftp. I was successfully able to put the file on the iSeries IFS using the shell FTP.
Thanks!
Solution
from ftplib import FTP
import os
localpath = os.getcwd() + '/Files/'
def putFiles():
hostname = 'host.name.com'
username = 'myuser'
password = 'mypassword'
myftp = FTP(hostname)
myftp.login(username, password)
myftp.cwd('/STUFF/HERE/')
for csv in os.listdir(localpath):
if csv.endswith('.csv'):
try:
myftp.storbinary("STOR " + csv, open(localpath + csv, 'rb'))
except Exception as e:
print(e)
As written, your code is trying to execute the following FTP command:
STOR /home/doug/Files/FILE.csv
Meaning it is trying to create /home/doug/Files/FILE.csv on the IFS. Is this what you want? I suspect that it isn't, given that you bothered to change the remote directory to /STUFF/HERE/.
If you are trying to issue the command
STOR FILE.csv
then you have to be careful how you deal with the Python variable that you've named file. In general, it's not recommended that you reassign a variable that is the target of a for loop, precisely because this type of confusion can occur. Choose a different variable name for localpath + file, and use that in your open(..., 'rb').
Incidentally, it looks like you're using Python 2, since there is a bare print statement with no parentheses. I'm sure you're aware that Python 3 is recommended by now, but if you do stick to Python 2, it's recommended that you avoid using file as a variable name, because it actually means something in Python 2 (it's the name of a type; specifically, the return type of the open function).
I have a problem connecting to Excel API in windows 10. I use Office365 and with it Excel2016. My goal is: to download CSV file from a client FTPS Server, merge it with the existing files,perfom some action on it(with pandas) and then load the whole data into excel and do reporting with it... Up to the point of loading it into Excel everything is fine.I managed to do all steps automatically with Python (sorry if my code looks a little cluttered - I am new to Python)
import subprocess
import os
import ftplib
import fnmatch
import sys
from ftplib import FTP_TLS
from win32com.client import Dispatch
import pandas as pd
filematch = '*.csv'
target_dir = 'cannot tell you the path :-) '
def loginftps(servername,user,passwort):
ftps = FTP_TLS(servername)
ftps.login(user=user,passwd=passwort)
ftps.prot_p()
ftps.cwd('/changes to some directory')
for filename in ftps.nlst(filematch):
target_file_name = os.path.join(target_dir,os.path.basename(filename))
with open(target_file_name,'wb') as fhandle:
ftps.retrbinary('RETR %s' %filename, fhandle.write)
def openExcelApplication():
xl = Dispatch("Excel.Application")
xl.Visible = True # otherwise excel is hidden
def mergeallFilestoOneFile():
subprocess.call(['prepareData_executable.bat'])
def deletezerorows():
rohdaten = pd.read_csv("merged.csv",engine="python",index_col=False,encoding='Latin-1',delimiter=";", quoting = 3)
rohdaten = rohdaten.convert_objects(convert_numeric=True)
rohdaten = rohdaten[rohdaten.UN_PY > 0]
del rohdaten['deletes something']
del rohdaten['deletes something']
rohdaten.to_csv('merged_angepasst.csv',index=False,sep=";")
def rohdatenExcelAuswertung():
csvdaten = pd.csv_read("merged.csv")
servername = input("please enter FTPS serveradress:")
user = input("Loginname:")
passwort = input("Password:")
loginftps(servername,user,passwort)
mergeallFilestoOneFile()
deletezerorows()
And here I am stuck somehow,.. I did extensive google research but somehow nobody has ever tried to perform Excel tasks from within Python??
I found this stackoverflow discussion: Opening/running Excel file from python but I somehow cannot figure out where my Excel-Application is stored to run code mentioned in this thread.
What I have is an Excel-Workbook which has a data connection to my CSV file. I want Python to open MS-Excel, refresh data connection and refresh a PivoTable & then save and close the file.
Has anybody here ever tried to to something similar and can provide some code to get me started?
Thanks
A small snippet of code that should work for opening an excel file, updating linked data, saving it, and finally closing it:
from win32com.client import Dispatch
xl = Dispatch("Excel.Application")
xl.Workbooks.Open(Filename='C:\\Users\\Xukrao\\Desktop\\workbook.xlsx', UpdateLinks=3)
xl.ActiveWorkbook.Close(SaveChanges=True)
xl.Quit()
I have my NAS server located and i am able to get/upload files on it. Now i have suituation where i need to read .png files location from server and pass it on UI thread to show the image. Right now i am only aware of method get which needs local location to save. I don't want file to be save on my local machine but just i shall be able to show that image on my application.
I have gone through this http://docs.paramiko.org/en/2.1/api/sftp.html but didn't found relevant method to use
Code is :-
import paramiko
paramiko.util.log_to_file(r'D:\TechnoThrone\Builds\paramiko.log')
# Open a transport
host = "stedgela01.TechnoThrone.com"
port = 2222
transport = paramiko.Transport((host, port))
# Auth
password = "xxx"
username = "xxxx"
transport.connect(username = username, password = password)
# Go!
sftp = paramiko.SFTPClient.from_transport(transport)
# Download
filepath = '/A/B/C/pic_ex.png'
localpath = r'D:\picfolder\pic_ex.png'
sftp.get(filepath, localpath)
I don't quite get the problem, so I will try to guess a bit.
You should be able to look at content in paths in the remote server without need to download locally the file.
get is not the right method if you don't want to download because as per documentation:
get(remotepath, localpath, callback=None) Copy a remote file
(remotepath) from the SFTP server to the local host as localpath. Any
exception raised by operations will be passed through. This method is
primarily provided as a convenience.
Parameters: remotepath (str) – the remote file to copy localpath
(str) – the destination path on the local host callback (callable) –
optional callback function (form: func(int, int)) that accepts the
bytes transferred so far and the total bytes to be transferred
There are other methods which can get filenames in a remote directory and attributes of those without the need to download.
These are listdir, listdir_attr and listdir_iter for example.
For example, listdir_attr will
Return a list containing SFTPAttributes objects corresponding to files
in the given path. The list is in arbitrary order. It does not include
the special entries '.' and '..' even if they are present in the
folder.
The returned SFTPAttributes objects will each have an additional
field: longname, which may contain a formatted string of the file’s
attributes, in unix format. The content of this string will probably
depend on the SFTP server implementation.
Parameters: path (str) – path to list (defaults to '.')
Returns: list
of SFTPAttributes objects
You could use something along these lines:
list_png_files = []
for file_attributes in sftp.listdir_attr("remote_path"):
if file_attributes.filename.endswith('.png'):
list_png_files.append(file_attributes.filename)
Check if it will give you relative or absolute path of course.
Similar you could try with listdir, etc.
def _list_remote_dir(self, dir="."):
sftp_session = self.ssh_session.open_sftp() # ssh_session is paramiko.SSHClient()
sftp_session.chdir(".")
cwd = sftp_session.getcwd()
print(cwd)
remote_file_attrs = sftp_session.listdir_attr()
for i in remote_file_attrs:
logging.info(i.filename)
this snipet may help, you got the current working directory, and you got the file name, you can get the abs-path of files on server.
Im writing a python scrpit to upload files to my file server:
host = "myhost.dev"
tn = telnetlib.Telnet()
tn.open(host, 5202)
print tn.read_until("\n")
fp = "./output"
f = open(fp, "r")
f_body = f.read()
tn.write(f_body)
tn.write("\n")
f.close()
If file has a new line character- 0a, and it is part of a binary data in gzip file, what should I do to escape it ? Can python telnetlib do it by itself ? or should I do it ?
best regards
I think that telnet is not the best option for transfering files, but if you still want to use it for uploading files. You may try to do the following (haven't tried, but I think should work)
#On client side
...
import base64
with open('test.gz','rb') as f:
content = f.read()
content_serialized = base64.b64encode(content)+'\n'
...
#On server side
...
import base64
content = base64.b64decode(content_serialized.rstrip('\n'))
with open('test.gz','wb') as f:
f.write(content)
...
Maybe telnet is not the best solution for this, it is better to use FTP o HTTP.
The telnet protocol is not suitable for transmiting files, and it has some control flow processes that make if dificult to transmit files and special chars.
If you want to use a non standard protocol, it is better so use the socket module, with sockets you don't have this problems with 0a.