This question already has answers here:
How do I watch a file for changes?
(28 answers)
Closed 5 years ago.
I am relatively new to python, but I am trying to create an automated process where my code will listen for new file entries in a directory. For example, someone can manually copy a zip file into a particular folder, and I want my code to recognize the file once it has completely been copied into the folder. The code can then do some manipulations, but that is irrelevant. I currently have my code just checking for a new file every 5 seconds, but this seems inefficient to me. Can someone suggest something that is more asynchronous?
Checking for new file every several seconds is actually not that bad an approach — in fact, it is the only really portable way to monitor for filesystem changes. If you're running on Linux, answers in the question linked by #sdolan will help you check for new files more efficiently, but they won't help you with the other part of your question.
Detecting that the file has been copied completely is much more difficult than it first appears. Your best best is, when a new file is detected, wait until it hasn't been touched for a while before processing it. The length of the wait period is best determined experimentally. It's a balancing act: make the interval too short, and you're risking operating on incomplete files; make it too long, and the user will notice a delay between the copy operation finishing and your code processing it.
Related
This question already has answers here:
Python: Get a list of selected files in Explorer (WIndows 7)
(4 answers)
Closed 8 months ago.
My problem is that I am making a program that will get file's data as per the folder that the user has opened as a file explorer... and I have surfed the whole internet to find the answer but it seems that this problem is never faced by anybody before. Many answers are related to the current working directory in which project files are situated.
So, I basically want the path for the folder which is currently being viewed by the user in file explorer...
Best way I can imagine doing that is write your Python software to use context menu in Explorer. With quick googling I found a interesting library for that: https://pypi.org/project/context-menu/
I suppose that there is no sensible way to read data from another process. That would cause security problems.
This question already has answers here:
What is the best way to call a script from another script? [closed]
(16 answers)
Closed 3 years ago.
I am a beginner to Python, and I would like a way to run a Python file from Python.
For example, I have a Python file named checker.py. In that file, I would like to iterate over a folder that contains inputs and outputs, and I would like to, using the Python script, give input to a different Python file, and check if it matches the expected output (in a different file). Is there any way to do this in Python?
Here is the GitHub link for the problems I have completed and need to check so far: https://github.com/vishnupsatish/CCC-practice
Try to split your big problem into several smaller ones.
First, try to find all files.
Then execute the Python files ( https://docs.python.org/3/library/subprocess.html )
Then read the output files and compare the results.
As this is a practice for you, I won't deliver all the code. Just try to do one small task after the other. And then ask when you encounter a problem.
This question already has answers here:
Python, want logging with log rotation and compression
(11 answers)
Closed 4 years ago.
I am currently using logging package in Python to log statements to a single file (log.log)
The problem I am facing is if the log file gets too large, it takes a lot of time to open the log file.
I thought after a pre-defined file size or line numbers, I can create a log file called log.log.1, and continue running with a fresh log.log.
Also, after log.log.5, the old logs should get deleted automatically.
How can I achieve this?
Depending on how you want to rotate the logs, you can use either the TimedRotatingFileHandler to rotate files at a given time interval (for example, create a new file ever day, or every hour); or you can use the RotatingFileHandler to create new files after they reach a certain size.
The logging cookbook has examples to help you get started.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
Questions concerning problems with code you've written must describe the specific problem — and include valid code to reproduce it — in the question itself. See SSCCE.org for guidance.
Closed 9 years ago.
Improve this question
I have a Python script which loads binary data from any targeted file and stores in inside
itself, in a list. The problem is that the bigger the stored file is, the longer it takes to open it the next time. Let's say that I want to load a 700 MB movie and store it in my script file. Then imagine that I open it next day with the 700 MB data stored in that script. It takes an eternity to open it!
Here is a simplified layout of how the script file looks.
Line 1: "The 700 MB movie is stored here inside a list."
Everything below : "All functions that the end-user uses."
Before the interpreter reaches the functions that the user is waiting for to be called,
it has to interpret a 700 MB data that is on line 1 first! This is a problem
because who wants to wait for an hour just to open a script?
So, would it help if I changed the layout of the file like this?
First lines: "All functions that the end-user uses." Below : "The 700
MB movie is stored here inside a list."
Would that help? Or would the interpreter have to plow through all the 700 MBs before the functions were called anyways?
Python compiler works in a way that makes what you are looking to do very very hard to say the least.
First, every-time you change the script (by adding the file for example), it will trigger a new cycle of compilation before the execution (turning a .py file in a .pyc one).
Second, every time you import the module, you will have that large block of data loaded into memory (whether it is on import or when you first access the data).
This is not just slow, it's also unsafe and error prone.
I'm guessing that what you intend to do is, distribute one single file with the data in it.
You might be able to do that using this little trick:
Making an executable python package (a zip file basically).
Building the zip file is very easy using the zipfile module.
This question already has answers here:
Python FTP get the most recent file by date
(5 answers)
Closed 4 years ago.
I am using Python to connect to an FTP server that contains a new list of data once every hour. I am only connecting once a day, and I only want to download the newest file in the directory. Is there a way to do this?
Seems like any system that is automatically generating a file once an hour is likely to be using an automated naming scheme. Are you over thinking the problem by asking the server for the newest file instead of more easily parsing the file names?
This wouldn't work in all cases, and if the directory got large it might become time consuming to get the file listing. But it seems likely to work in most cases.
Look at ftplib in your current version of python. You can see a function to handle the result of the LIST command that you would issue to do a dir, if you know a last time that you run a successful script then you can parse the result from the LIST and act on the new files on the directory. See the ftplib for more info on how to do it. The retrlines function is what I would expect to use.