We Keep Coding

Need a workaround for known issue with os.replace() function on Windows

I'm working with a script that downloads files from cloud firebase storage, stores them locally in a folder "assets", and renames the references to those files internally on each target .md note file.
The issue is just with the last step, where the temp file with the new reference needs to replace the old file. I receive the error "PermissionError: [WinError 32] The process cannot access the file because it is being used by another process"
I tried a few things, but eventually found this source: https://bugs.python.org/issue46003
It seems there is a persistent bug with the os.replace() function on Windows, and I don't know how to implement a workaround in my context. I considered attempting to add delay as suggested in the thread, but this doesn't seem like the best option since there are hundreds of references I need to modify.
Here is the code:
(the area in question is at the end "if os.path.exists(fullTempFilePath)..."
import re
import glob
import os
import requests
import calendar
import time
vaultDir = '/Users/kacar/Firebase Temp/Firebase-Test'
firebaseShort = 'none'
fullRead = 'none'
fileFullPath = ''
fullTempFilePath = ''
i = 0
ext = ''
/# Walk through all files in all directories within the specified vault directory
for subdir, dirs, files in os.walk(vaultDir):
for file in files:
# Open file in directory
fileFullPath = os.path.join(subdir,file)
fhand = open(fileFullPath, errors='ignore')
for line in fhand:
# Download the Firebase file and save it in the assets folder
if 'firebasestorage' in line:
try:
# If it's a PDF, it will be in the format {{pdf: link}}
if '{{pdf:' in line:
link = re.search(r'https://firebasestorage(.*)\?alt(.*)\}', line)
else:
link = re.search(r'https://firebasestorage(.*)\?alt(.*)\)', line)
firebaseShort = 'https://firebasestorage' + link.group(1) # https://firebasestorage.googleapis.com/v0/b/firescript-577a2.appspot.com/o/imgs%2Fapp%2FDownloadMyBrain%2FLy4Wel-rjk.png
firebaseUrl = link.group(0)[:-1] # https://firebasestorage.googleapis.com/v0/b/firescript-577a2.appspot.com/o/imgs%2Fapp%2FDownloadMyBrain%2FLy4Wel-rjk.png?alt=media&token=0fbafc8f-0a47-4720-9e68-88f70803ced6
# Download the file locally
r = requests.get(firebaseUrl)
timestamp = calendar.timegm(time.gmtime())
# Get file extension of file. Ex: .png; .jpeg
reg = re.search(r'(.*)\.(.+)', firebaseShort[-5:]) # a.png / .jpeg
ext = '.' + reg.group(2) # .jpeg
# Create assets folder if it doesn't exist
if not os.path.exists(vaultDir + '/assets'):
os.makedirs(vaultDir + '/assets')
# Create new local file out of downloaded firebase file
newFilePath = 'assets/' + str(timestamp) + '_' + str(i) + ext
# print(firebaseUrl + '>>>' + newFilePath)
with open(vaultDir + '/' + newFilePath,'wb') as output_file:
output_file.write(r.content)
except AttributeError: # This is to prevent the AttributeError exception when no matches are returned
continue
# Save Markdown file with new local file link as a temp file
# If there is already a temp version of a file, open that.
fullTempFilePath = vaultDir + '/temp_' + file
if os.path.exists(fullTempFilePath):
fullRead = open(fullTempFilePath, errors='ignore')
else:
fullRead = open(fileFullPath, errors='ignore')
data = fullRead.read()
data = data.replace(firebaseUrl,newFilePath)
fullRead.close()
with open(fullTempFilePath,'wt') as temp_file:
temp_file.write(data)
i = i + 1
if os.path.exists(fullTempFilePath):
path = os.replace(fullTempFilePath,fileFullPath)
# Close file
fhand.close()

Changing the current directory of ftp client

I have a ftp client developed with python. When I specify a file in the current directory, it is successfully uploading. I want to specify a different directory except the current directory. How could I modify this code?
from ftplib import FTP
ftp = FTP('')
ftp.connect("127.0.0.1", 1026)
ftp.login()
ftp.retrlines('LIST')
def uploadFile():
filename = "f.txt" #replace with your file in your home folder
ftp.storbinary('STOR '+filename, open(filename, 'rb'))
print(ftp.storbinary)
ftp.quit()
print("filename",filename,"uploaded to server")
uploadFile()
Here I want to specify this directory to select files C:\Users\User\Desktop\nnn.
Please any help would be highly appreciated.

Put the directory prefix in the path when calling open():
ftp.storbinary('STOR ' + filename, open(os.path.join(r'C:\Users\User\Desktop\nnn', filename), 'rb'))

You can set filename like this way
ftp.storbinary('STOR {0}.mrss'.format("Your file name"), file)

download remote gz files that reside in a tree like directories does snot work

I have been scratching my head for more than 2 days, but still cannot figure out how to do the following!
I want to download all Geo data sets that are in ftp://ftp.ncbi.nlm.nih.gov and then in each data set, I need to see if they contain the keywords that I am interested in. I was able to manually download one of the data sets and checked the file for the desired keywords. However, since the number of data sets are huge, I cannot do it manually. I want to write a program to do it for me. For the first step, I just tried to see if I can download them.
The structure is as follows:
hots->
/geo/
-> datasets/
-> GDS1nnn/ .... all the way through GDS6nnn and each of them
contain more than 600 directories; ordered by number i.e.
GDS1001. Now, in each of these directories:
---> soft inside this folder there are 2 files that are named
like this: folder name (GDS1001)+_full.soft.gz
this is the file that I think I need to download and then see if the keywords that I am looking for are inside that file.
Here is my code:
ftp = FTP('ftp.ncbi.nlm.nih.gov') # remember that you ONLY need to provide the host name not the complete address!
ftp.login()
#ftp.retrlines('LIST')
ftp.cwd("/geo/datasets/GDS1nnn/")
ftp.retrlines('LIST')
filenames = ftp.nlst()
count = len(filenames)
curr = 0
print ("found {} files".format(count))
for filename in filenames:
first_path=filename+"/soft/"
second_path=first_path+filename+"_full.soft.gz"
#print(second_path)
local_filename = os.path.join(r'full path to a folder that I
created')
file = open(local_filename, 'wb')
ftp.retrbinary('RETR ' + second_path, file.write)
file.close()
ftp.quit()
Output:
file = open(local_filename, 'wb')
PermissionError: [Errno 13] Permission denied: full path to a folder that I created'
However, I have both read and write permission on this folder.
Thanks for your help

The following code shows how you can create a folder for each dataset and save their content into that folder.
import sys, ftplib, os, itertools
from ftplib import FTP
from zipfile import ZipFile
ftp = FTP('ftp.ncbi.nlm.nih.gov')
ftp.login()
#ftp.retrlines('LIST')
ftp.cwd("/geo/datasets/GDS1nnn/")
ftp.retrlines('LIST')
filenames = ftp.nlst()
curr = 0
#print ("found {} files".format(count))
count = 0
for filename in filenames:
array_db=[]
os.mkdir( os.path.join('folder called "output' + filename ) )
first_path=filename+"/soft/"
os.mkdir( os.path.join('folder called "output' + first_path ) )
second_path=first_path+filename+"_full.soft.gz"
array_db.append(second_path)
for array in array_db:
print(array)
local_filename = os.path.join('folder called "output' + array )
file = open(local_filename, 'wb')
ftp.retrbinary('RETR ' + array, file.write)
file.flush()
file.close()
ftp.quit()

Why do I get this IOerror13 ? Using python

import sys
import os
import re
import ftplib
os.system('dir /S "D:\LifeFrame\*.jpg" > "D:\Python\placestogo.txt"') #this is where to search.
dat = open('placestogo.txt','r').read()
drives = re.findall(r'.\:\\.+.+',dat)
for i in range(len(drives)):
path = drives[i]
os.system('dir '+ path +'\*.jpg > D:\python\picplace.txt')
picplace = open('picplace.txt','r').read()
pics = re.findall(r'\w+_\w+.\w+..jpg|IMG.+|\w+.jpg',picplace)
for i in range(len(pics)):
filename = pics[i]
ftp = ftplib.FTP("localhost")
print ftp.login("xxxxxxxx","xxxxxxxx")
ftp.cwd("/folder")
myfile = open(path,"rb")
print ftp.storlines('STOR ' + filename, myfile)
print ftp.quit()
sys.exit()
i am trying to copy all of those files to my ftp server but it gives me that error:
d:\Python>stealerupload.py
230 Logged on
Traceback (most recent call last):
File "D:\Python\stealerupload.py", line 22, in <module>
myfile = open(path,"rb")
IOError: [Errno 22] invalid mode ('rb') or filename: '"D:\\LifeFrame"'
any one knows where is the problem ? I am running as administrator and folder should have permissions

The error seems pretty obvious. You are trying to open a directory path, which is neither possible nor what you actually want to do. This bit:
for i in range(len(drives)):
path = drives[i]
...
for i in range(len(pics)):
...
myfile = open(path,"rb")
Inside the loop you are setting path to be one of your drives elements. each of these items appears to be a directory path. Then you try to open path later, which is a directory path and not a file.

In the error message, it shows '"D:\\LifeFrame"', which to me looks like you have extra quotes in path. Try adding print path to see what its value is.
Maybe you want to upload data from filename to your server, not from path, in which case Python showed in the error message is where the bug is: you should be opening filename instead.

Downloading from an ftp using Python

I have a piece of code in Python to download files from an ftp. The code downloads the very first file in the list of available days but fails to download the second. What could be the problem?
import os, ftplib
destdir='D:\precipitation\dl'
ftp = ftplib.FTP('ftp.itc.nl')
ftp.login('anonymous', '')
ftp.cwd('pub/mpe/msg')
available_days=['summsgmpe_20100101.zip','summsgmpe_20100102.zip', 'summsgmpe_20100103.zip', 'summsgmpe_20100104.zip', 'summsgmpe_20100105.zip', 'summsgmpe_20100106.zip', 'summsgmpe_20100107.zip', 'summsgmpe_20100108.zip']
hdfs = list()
for day in available_days :
file = available_days[available_days.index(day)]
print 'file=', file
local_file = os.path.join(destdir, file)
ftp.retrbinary('RETR %s' %file, open(local_file, 'wb').write)
hdfs.append(os.path.abspath(local_file))
ftp.cwd('..')
ftp.quit()

Remove your call to ftp.cwd(..)
That's moving up a directory for each iteration of the list, instead of staying in the correct folder where the files are.