First of: I know of pyinotify.
What I want is an upload service to my home server using Dropbox.
I will have a Dropbox's shared folder on my home server. Everytime someone else, who is sharing that folder, puts anything into that folder, I want my home server to wait until it is fully uploaded and move all the files to another folder, and removing those files from the Dropbox folder, thus, saving Dropbox space.
The thing here is, I can't just track for changes in the folder and move the files right away, because if someone uploads a large file, Dropbox will already start downloading and therefore showing changes in the folder on my home server.
Is there some workaround? Is that somehow possible with the Dropbox API?
Haven't tried it myself, but the Dropbox CLI version seems to have a 'filestatus' method to check for current file status. Will report back when I have tried it myself.
There is a Python dropbox CLI client, as you mentioned in your question. It returns "Idle..." when it isn't actively processing files. The absolutely simplest mechanism I can imagine for achieving what you want would be a while loop that checked the output of dropbox.py filestatus /home/directory/to/watch and performed an scp of the contents and then a delete on the contents if that suceeded. Then slept for five minutes or so.
Something like:
import time
from subprocess import check_call, check_output
DIR = "/directory/to/watch/"
REMOTE_DIR = "user#my_server.com:/folder"
While True:
if check_output(["dropbox.py", "status", DIR]) == "\nIdle...":
if check_call(["scp", "-r", DIR + "*", REMOTE_DIR]):
check_call(["rm", "-rf", DIR + "*"])
time.sleep(360)
Of course I would be very careful when testing something like this, put the wrong thing in that second check_call and you could lose your filesystem.
You could run incrond and have it wait for IN_CLOSE_WRITE events in your Dropbox folder. Then it would only be triggered when a file transfer completed.
Here is a Ruby version that doesn't wait for Dropbox to be idle, therefore can actually start moving files, while it is still syncing. Also it ignores . and ... It actually checks the filestatus of each file within a given directory.
Then I would run this script either as a cronjob or in a separate screen.
directory = "path/to/dir"
destination = "location/to/move/to"
Dir.foreach(directory) do |item|
next if item == '.' or item == '..'
fileStatus = `~/bin/dropbox.py filestatus #{directory + "/" + item}`
puts "processing " + item
if (fileStatus.include? "up to date")
puts item + " is up to date, starting to move file now."
# cp command here. Something along this line: `cp #{directory + "/" + item + destination}`
# rm command here. Probably you want to confirm that all copied files are correct by comparing md5 or something similar.
else
puts item + " is not up to date, moving on to next file."
end
end
This is the full script, I ended up with:
# runs in Ruby 1.8.x (ftools)
require 'ftools'
directory = "path/to/dir"
destination = "location/to/move/to"
Dir.glob(directory+"/**/*") do |item|
next if item == '.' or item == '..'
fileStatus = `~/bin/dropbox.py filestatus #{item}`
puts "processing " + item
puts "filestatus: " + fileStatus
if (fileStatus.include? "up to date")
puts item.split('/',2)[1] + " is up to date, starting to move file now."
`cp -r #{item + " " + destination + "/" + item.split('/',2)[1]}`
# remove file in Dropbox folder, if current item is not a directory and
# copied file is identical.
if (!File.directory?(item) && File.cmp(item, destination + "/" + item.split('/',2)[1]).to_s)
puts "remove " + item
`rm -rf #{item}`
end
else
puts item + " is not up to date, moving to next file."
end
end
Related
I am a beginner and I have a question how to make a "for" loop to check the following condition for me:
if the folder does not exist, create it
if there is a folder, create a file in it
and check if the folder exists again
a) the folder exists, create a file in it
b) does not exist, create a folder, create a file
import os
import datetime
now = datetime.datetime.now().strftime("%Y-%m-%d")
print(now)
directory = "D:/Dir/Dir_" + now
print(directory) # D:/Dir/Dir_2021-11-12
if not os.path.exists(directory):
os.mkdir(directory)
print("Created folder: " + directory)
else:
print("Folder exists")
now_datetime = datetime.datetime.now().strftime("%Y-%m-%d_%H%M")
print("Current date: " + now_datetime)
full_path = directory + "/file_" + now_datetime + ".txt"
print(full_path) # D:/Dir/Dir_2021-11-12/file_2021-11-12_1000.txt
output = print("Something ...")
print(output)
plik = open(full_path,"w+")
plik.write(output)
plik.close()
Ultimately, I would like to make a process that will run all the time, because I want to read messages from the server. So, for example, when I get messages from the server, it will write them to a file and then the loop will check if there is a folder for a given day. If it exists, new files will be created in this folder. Until a new folder appears. Then the files will start falling into a new folder.
Can anyone help me? Somehow guide as the easiest way to do this?
Maybe use a while-loop, either running infinitely or until a certain condition is meet. See this simple example:
import time
x = 0
while x < 10: # while(true)
x += 1
time.sleep(1)
print(x)
Add the time.sleep to make the loop sleep so that it does not run continuously
A piece code below is part of a larger program which I am running on a remote server via a batch script with #!/bin/bash -l as its first line.
On my local machine it runs normally but on a remote server permission issues arises. What may be wrong?
The description of the code may not important to the problem, but basically the code uses awk in processing the contents of the files based on the names of the files.
Why is awk denied permission to operate on the files? When I run awk directly on a shell prompt of the remote server it works normally.
#!/usr/bin/env python
list_of_files = ["file1", "file2", "file3"]
for file in list_of_files:
awk_cmd = '''awk '/^>/{print ">" substr(FILENAME,1,length(FILENAME)) ++i; next} 1' ''' + file + " > tmp && mv tmp " + file + \
" | cat files > 'pooled_file' "
exitcode = subprocess.call(awk_cmd, shell=True)
Any help would be appreciated.
I am pretty sure it is a permissions issue since when you are landing into remote machine it is NOT landing on directory where your Input_file(s) are present, off course it will land in HOME directory of logged in user at remote server. So it is a good practice to mention file names with complete paths(Make sure file names with path you are giving are present in target location too else you could write a wrapper over it to check either files are present or not too). Could you please try following.
#!/usr/bin/env python
list_of_files = ["/full/path/file1", "/full/path/file2", "/full/path/file3"]
for file in list_of_files:
awk_cmd = '''awk '/^>/{num=split(FILENAME,array,"/");print ">" substr(array[num],1,length(array[num])) ++i; next} 1' ''' + file + " > tmp$$ && mv tmp$$ " + file + \
" | cat files > 'pooled_file' "
exitcode = subprocess.call(awk_cmd, shell=True)
I haven't tested it but I have changed it as per full path, since awk will print complete path with filename so I have changed FILENAME in your code to as per array's place, I also changed tmp temporary file to tmp$$ for safer side.
Hey and thanks for all of your answers. I try to write a piece of python code that only executes once, (first time the program is installed) and copies the program into the windows startup folders.
(C:\Users\ USER \AppData\Roaming\Microsoft\Windows\Start Menu\Programs\Startup)
That's the code i wrote for this. (Please don't judge me. I know it's
very shitty code. But I'm very new to coding. (this is the second
little program i try to write)
import os
import shutil
#get username
user = str(os.getlogin())
user.strip()
file_in = ('C:/Users/')
file_in_2 = ('/Desktop/Py Sandbox/test/program.py')
file_in_com = (file_in + user + file_in_2)
folder_seg_1 = ('C:/Users/')
folder_seg_2 = ('/AppData/Roaming/Microsoft/Windows/Start Menu/Programs/Startup')
#create FolderPath
folder_com = (folder_seg_1 + user + folder_seg_2)
shutil.copy2(file_in_com, folder_com)
Because i got an error, that there is no such internal, external,
command, program or batch file named Installer. I tried to generate a batch file with
nothing in it that executes when the installation process is finished.(But the error is still there.)
save_path = 'C:/Windows/assembly/temp'
name_of_file = str("Installer")
completeName = os.path.join(save_path, name_of_file+".bat")
file1 = open(completeName, "w")
file1.close()
The main idea behind this that there is my main Program, you execute
it it runs the code above and copies itself to the startup folder.
Then the code the whole installer file gets deleted form my main
program.
import Installer
#run Installer File
os.system('Installer')
os.remove('Installer.py')
But maybe there's someone out there who knows the answer to this problem.
And as I said earlier, thanks for all of your answers <3.
BTW I'm currently using Python 3.5.
Okay guys now I finally managed to solve this problem. It's actually not that hard but you need to think from another perspective.
This is now the code i came up with.
import os
import sys
import shutil
# get system boot drive
boot_drive = (os.getenv("SystemDrive"))
# get current Username
user = str(os.getlogin())
user.strip()
# get script path
script_path = (sys.argv[0])
# create FolderPath (Startup Folder)
folder_seg_1 = (boot_drive + '/Users/')
folder_seg_2 = ('/AppData/Roaming/Microsoft/Windows/Start Menu/Programs/Startup')
folder_startup = (folder_seg_1 + user + folder_seg_2)
#check if file exits, if yes copy to Startup Folder
file_name = (ntpath.basename(script_path))
startup_file = (folder_startup + ("/") + file_name)
renamed_file = (folder_startup + ("/") + ("SAMPLE.py"))
# checkfile in Startup Folder
check_path = (os.path.isfile(renamed_file))
if check_path == True:
pass
else:
shutil.copy2(script_path, folder_startup)
os.rename(startup_file, renamed_file)
This script gets your username, your boot drive, the file location of
your file than creates all the paths needed. Like your personal
startup folder. It than checks if there is already a file in the
startup folder if yes it just does nothing and goes on, if not it
copies the file to the startup folder an than renames it (you can use that if you want but you don't need to).
It is not necessary to do an os.getenv("SystemDrive") or os.getlogin(), because os.getenv("AppData") already gets both. So the most direct way of doing it that I know of is this:
path = os.path.join(os.getenv("appdata"),"Microsoft","Windows","Start Menu","Programs","Startup")
I have a python script which is supposed to loop through all files in a directory and set the date of each file to the current time. It seems to have no effect, i.e. the Date column in the file explorer shows no change. I see the code looping through all files, it just appears that the call to utime has no effect.
The problem is not this because most of the dates are months old.
# set file access time to current time
#!/usr/bin/python
import os
import math
import datetime
def convertSize(size):
if (size == 0):
return '0B'
size_name = ("B", "KB", "MB", "GB", "TB", "PB", "EB", "ZB", "YB")
i = int(math.floor(math.log(size,1024)))
p = math.pow(1024,i)
s = round(size/p,2)
return '%s %s' % (s,size_name[i])
# see www.tutorialspoint.com/python/os_utime.htm
def touch(fname, times=None):
fhandle = open(fname, 'a')
try:
os.utime(fname, times)
finally:
fhandle.close()
def main():
print ("*** Touch Files ***");
aml_root_directory_string = "C:\\Documents"
file_count = 0
file_size = 0
# traverse root directory, and list directories as dirs and files as files
for root, dirs, files in os.walk(aml_root_directory_string):
path = root.split('/')
#print((len(path) - 1) * '---', os.path.basename(root))
for file in files:
filename, file_extension = os.path.splitext(file)
print(len(path) * '---', file)
touch(filename, )
#
print ("\n*** Total files: " + str(file_count) + " Total file size: " + convertSize(file_size) + " ***");
print ("*** Done: Time: " + str(datetime.datetime.now()) + " - Touch Files ***");
# main ###############################################################################
if __name__ == "__main__":
# stuff only to run when not called via 'import' here
main()
Edit:
In case anyone reads this in the future, it is also important to note the the file explorer can display more than 1 kind of date
You've got three issues:
You're using the file name, not the full path, when touching, so all the touching occurs in the working directory
You're stripping the file extension too, so the touched files lack extensions
You're touching files to which you have an open file handle, and on Windows, on Python 2.7, this is a problem, because os.utime opens the files with no sharing allowed, which is incompatible with existing open file handles
To fix #3, change your touch method to:
def touch(fname, times=None):
# Open and immediately close file to force existence
with open(fname, 'ab') as f:
pass
# Only alter times when file is closed
os.utime(fname, times)
To fix #1 and #2, change your main method to call touch like so:
touch(os.path.join(root, file))
which uses the original name and joins it with the root directory being traversed, where touch(filename) was touching a file without the extension, in the program's working directory (because you used an unqualified name). If you find your program's working directory (print os.getcmd() will tell you where to look), you'll find a bunch of random empty files there corresponding to the files found in the tree you were traversing, stripped of paths and file extensions.
Side-note: If you can move to Python 3 (it's been a while, and there are a lot of improvements), you can make a slightly safer (race-free) and faster touch thanks to file descriptor support in os.utime:
def touch(fname, times=None):
with open(fname, 'ab') as f:
os.utime(f.fileno(), times)
Not all systems will support file descriptors, so if you need to handle such systems, define touch based on testing for file descriptor support via os.supports_fd.
os.utime does work on Windows but probably you are looking at the wrong date in explorer. os.utime does not modify the creation date (which it looks like is what is used in the date field in explorer). It does update the "Date modified" field. You can see this if you right click on the category bar and check the "date modified" box. Alternatively start a command line and type "dir". The date shown there should reflect the change.
I tested os.utime on python 2.7 where you have to give two arguments:
os.utime("file.txt", None)
and on Python 3 where the second argument defaults to None:
os.utime("file.txt")
I have the following code where as part of the code it tries to create a folder if it doesnt exist, I see the folder created with directoryname as "AYH9P4~H",it should be created whatever is in the text file "data_file.txt" which is "AU_1.2.333",any inputs on why it is created with folder name "AYH9P4~H",BTW , running this script on a linux box
import os
def getAU (aufilename):
file = open(aufilename,'r')
return file.read()
def main ():
aufilename="data_file.txt"
#ROOT_DIR - The reference android build used to make the builds
AU=getAU(aufilename)
if (os.path.isdir("/local/mnt/workspace/builds/" + AU)):
print "Direcotry local/mnt/workspace/builds/" + AU + "exists"
else:
print "Directory doesnt exist,creating directory" + AU + "now"
os.chdir("/local/mnt/workspace/builds")
os.makedirs(AU)
ROOT_DIR= "/local/mnt/workspace/builds/" + AU
print "ROOT_DIR"
print ROOT_DIR
if __name__ == '__main__':
main()
My best guess would be, that file.read() actually returns everything including linefeed, and possible other characters. You need to sanitize the output of file.read() in any case.
Other thought is that if you're trying to create such a folder on a filesystem that doesn't support multiple extensions (you have multiple dots in the name) it will be borked too.