We Keep Coding

Copying Large Amount of File : Permission Error in Python

This is a simple copy file problem and I came here with my solution. There are some questions in Stackoverflow that are not closed as my problem. That's why I opened this.
Explanation:
Python gives errors while copying files, in my case I tried to copy the image data set to another file(318MB). Python gives these errors in random sequence.
Data Set: Data
Python Version: 3.8.13
IDE: VSCode
Code and Error Message:
for jpgfile in glob.iglob(os.path.join(imageData, "*.jpeg")):
shutil.copy(jpgfile, dst_dir)
C:/Users/user1/Desktop/imageData Traceback (most recent call last):
File "c:\Users\user1\Desktop\copyImageJPEG.py", line 35, in
shutil.copy(jpgfile, dst_dir) File "C:\Users\user1\AppData\Local\Programs\Python\Python38\lib\shutil.py",
line 418, in copy
copyfile(src, dst, follow_symlinks=follow_symlinks) File "C:\Users\user1\AppData\Local\Programs\Python\Python38\lib\shutil.py",
line 264, in copyfile
with open(src, 'rb') as fsrc, open(dst, 'wb') as fdst: PermissionError: [Errno 13] Permission denied:
'C:/Users/user1/Desktop/copyFolder\person110_virus_207.jpeg'
What I've done till now,
Gave permission access with chmod
Tried to copy , copyFile, copy2 functions
Also this ref.
Wrote full path for both source and destination
Encoded the path for both source and destination
Run the python code as Admin
Also this ref.
r-prefix
Those solutions did not work for my problem. For those who can't find a solution like me, here is another solution:
Just close the copyFolder, don't disrupt it. In my case, I got the error while trying to see the last copied image, so the error popped up 50th, 100th, 150th, etc., depending on my check time.
Somehow the operating system blocks the code while we are using the folder.
I will update this post if I get another solution or the real reason for this disruption.
Edit: #Barmar Windows locks files while they're open in an editor. You get an error if you try to overwrite the locked file.

open() method not working in JupyterLab ( pyspark kernel )

Getting below error when running the command:
output = open( "C:/Users/TAA3656/mytddutc_nudges_sample.json", 'w') # Update to local path and file name
[Errno 2] No such file or directory: 'C:/Users/TAA3656/mytddutc_nudges_sample.json'
Traceback (most recent call last):
FileNotFoundError: [Errno 2] No such file or directory: 'C:/Users/TAA3656/mytddutc_nudges_sample.json'

I'd guess the path C:/Users/TAA3656 doesn't exist, so it's not possible to create a file in this nonexistent path. For example:
>>> open("nonexistent/thing.json", 'w')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
FileNotFoundError: [Errno 2] No such file or directory: 'nonexistent/thing.json'
One could think that open(..., 'w') should create nonexistent/thing.json if it doesn't exist, but in this case, the directory nonexistent is... non-existent, so open refuses to create the file along with an entire path structure.
You should create the path first:
from pathlib import Path
the_path = Path("C:/Users/TAA3656")
# Create the path if it doesn't exist
the_path.mkdir(parents=True, exist_ok=True)
# Open or create the file
with (the_path / "your_file.json").open("w") as output:
... # run your code

verify the path of the file you are trying to open is correct. to verft the path exist use !pwd to show the path you are in and navegate to the location and !mkdir name to create the folder and the you can acces it.you also need to import sys and import os. so you can navegate through the system files

Do you know if you are running your jupyterlab locally, or in some kind of cloud environment? (Your mention of a pyspark kernel suggests to me it may be the latter.) If it is a cloud platform of some kind, you may not have access to your local hard disk.
Try this to check your platform, and see if it looks local to you.
Or you can try:
import os
os.getcwd()
to give you the path of where you are currently working. If it doesn't look like a directory on your local pc, again, you not be working locally.
Or ask a friendly looking colleague.

Python file error while running the file, no helpful feedback from the terminal as well. Any idea what this means?

The code is absolutely correct, here is the link to the repository from where I cloned it: https://github.com/nft-fun/generate-bitbirds.
I did not use the in-built vs code one-it was having some other issues, used other but already activated using conda init beforehand
The only prerequisite was to install the dependencies and so I did.
After doing that I activated the base conda env then run the py script, and this is what it shows.
C:\Users\Zee\Documents\NFT_Tests\generate-bitbirds (main -> origin)
(base) λ python bitbird_generation_script.py
Traceback (most recent call last):
File "bitbird_generation_script.py", line 244, in <module>
new_image.save(imgname)
File "C:\Users\Zee\anaconda3\lib\site-packages\PIL\Image.py", line 2232, in save
fp = builtins.open(filename, "w+b")
FileNotFoundError: [Errno 2] No such file or directory: '/bird_images/0.png'
Any insights would be helpful :)

You have to find out what is the issue.
Inside the script, check the value of dirname = os.path.dirname(__file__).
The error you encounter seems to be that dirname is the empty string and so the images are trying to get saved in /bird_images which does not exist.
It should try to save the images in the local directory ./bird_images/
You could try :
On Linux :
PYTHONPATH=. python bitbird_generation_script.py
On Windows:
set PYTHONPATH=.
python bitbird_generation_script.py

Python Script can see the Y:/ Drive when run from IDLE but not the command line

I have a python script that builds a file by searching through folders and pulling in a list of files. This file runs fine and works as expected when I open and run it in IDLE, but if when I run the script in a commandline window I get this error:
C:\Windows\system32>python "C:\Users\ntreanor\Documents\RV Scripts\Server RV Sequence.py"
Traceback (most recent call last):
File "C:\Users\ntreanor\Documents\RV Scripts\Server RV Sequence.py", line 69,
in <module>
for foldername in os.listdir(pngFolders):
WindowsError: [Error 3] The system cannot find the path specified:
'Y:/20_temp_script_testing/pr126 movs\\04_comp_pngs/*.*'
In case it's not obvious, yes the path does exist. It not only works in IDLE but I double checked and the path definitely exists.
I also tried to create folders with a script that runs as a daemon and got a similar result
Traceback (most recent call last):
File "D:\shotgun\shotgunEventDaemon.py", line 888, in process
self._callback(self._shotgun, self._logger, event, self._args)
File "D:\shotgun\plugins\CreateAssetFolders.py", line 72, in createAssetFolders
os.makedirs(folder)
File "D:\Python27\Lib\os.py", line 150, in makedirs
makedirs(head, mode)
File "D:\Python27\Lib\os.py", line 150, in makedirs
makedirs(head, mode)
File "D:\Python27\Lib\os.py", line 150, in makedirs
makedirs(head, mode)
File "D:\Python27\Lib\os.py", line 157, in makedirs
mkdir(name, mode)
WindowsError: [Error 3] The system cannot find the path specified: 'Y:/'
This is what the script logged as a folder right before that:
Making folder:
Y:/07_design/04_environmental_elements\eec005-08_insect_ladybird_red_7_spots_wide
(the reason it's saying Y and not the whole path is that it attempts to make each folder back until it can't go back any further and that's when the exception is thrown)
Are the environment variables of the commandline window somehow affecting the drive mapping that should be pointing the script to the right location?

The issue is likely because IDLE and your command line are running with a different level of privileges. Mapped network drives are not automatically available to all user contexts. There is a superuser question on this here and plenty of other resources cover this topic. In short, the mapped network drive is only available to programs running at the level the mapping was made at.
If you have mapped the network drive through the windows UI, then it will be mapped for un-elevated programs. However if it was mapped with net use then it depends on the level of the command prompt when the mapping was made!
Disabling UAC will also affect change this behaviour, as will use of an elevated (or not) command prompt, which may explain why some PCs display different behaviour.

I think your problem is that you're trying to open the * file, which of course doesn't exist. open(path) takes path as a literal string and doesn't translate it in anyway, so it expects that value to be a valid filename. You should change your code to get a directory instead of a file, and then walking that directory.

python 3.4 windows 8.1 unable to create file in C:\

I'm trying to run a script in python3.4 & windows 8 .
It worked well with windows 7 but now, in windows 8.1 it can´t write a file in C: , may be access denied.
If I change the directory, for example User/xxx it works, but I don´t know how to do to 'write' in C:
with open('C:/xxxx.csv', 'r') as csvfile:
Traceback (most recent call last):
File "C:\Python34\Scripts_jmab\csv_to_shp.py", line 12, in <module>
urllib.request.urlretrieve(a,filename)
File "C:\Python34\lib\urllib\request.py", line 188, in urlretrieve
tfp = open(filename, 'wb')
OSError: [Errno 22] Invalid argument: 'C:/xxxxx.csv'
thanks

Due to security settings in Windows, users are not allowed to create files in the root C:\ folder, as some important system files reside there that could be attacked by viruses. While it is possible to disable this feature, the easiest (and safest) solution is to create a directory for your programs to put their output files in. C:\TEMP is a good location (you can create it if it doesn't already exist), or you can make another folder named according to your own choosing.