We Keep Coding

FileNotFoundError error while trying to rename the symlinks using os.rename [duplicate]

I would like to know is it possible to rename a symlink with python.
Already tried os.rename and shutil.move
Any ideas?
os.rename return me this error : OSError: [Errno 18] Cross-device link
>>> import sys, os
>>>
>>> path = '/Library/Application Support/appsolute/MAMP PRO/db/'
>>> job = path + 'mysql-job/'
>>> perso = path + 'mysql-perso/'
>>> mysql = path + 'mysql/'
>>>
>>> os.rename(mysql, job)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
OSError: [Errno 18] Cross-device link
>>> exit()
Danny-Dombrowski:script ddombrowski$ ls -al /Library/Application\ Support/appsolute/MAMP\ PRO/db/
total 24
drwxrwxr-x 5 root admin 170 7 fév 19:29 .
drwxrwxr-x 12 root admin 408 7 fév 17:14 ..
-rw-r--r--# 1 ddombrowski admin 6148 7 fév 19:29 .DS_Store
lrwxr-xr-x 1 ddombrowski admin 46 7 fév 19:29 mysql -> /Volumes/Gestion Portail Sante/Database/mysql/
drwxrwxr-x 11 ddombrowski admin 374 7 fév 19:22 mysql-perso

os.rename should work.
xupeng#xupeng t $ ls -l
total 0
-rw-r--r-- 1 xupeng xupeng 0 Feb 8 08:22 a
lrwxrwxrwx 1 xupeng xupeng 1 Feb 8 08:23 b -> a
xupeng#xupeng t $ python
Python 2.6.5 (release26-maint, Sep 21 2011, 10:32:38)
[GCC 4.3.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.rename('b', 'c')
xupeng#xupeng t $ ls -l
total 0
-rw-r--r-- 1 xupeng xupeng 0 Feb 8 08:22 a
lrwxrwxrwx 1 xupeng xupeng 1 Feb 8 08:23 c -> a

os.rename will work fine:
$ ln -s target link
$ python -c "import os; os.rename('link', 'link.new')"
$ ls -l link.new
lrwxrwxrwx 1 phihag phihag 6 Feb 8 01:25 link.new -> target
Make sure you don't include a / after the symlink: link/ is the same as link/., and not the same as link.

Why does the following code not successfully move desired files to the desired directories?

import os
import shutil
from collections import Counter as cnt
curr_path = os.getcwd()
ls = lambda path: [x for x in os.listdir(path) if x[0] != '.']
raw_files = [x for x in ls(curr_path) if '_' not in x]
filenames = []
folders = []
for f in raw_files: # sweeping out the '.', only having filenames
try:
i = f.index('.')
f_new = f[0:i]
filenames.append(f_new)
except ValueError:
filenames.append(f)
fname_freq = cnt(filenames)
for fname, freq in fname_freq.items():
if freq > 1:
folders.append(fname)
for fldr in folders:
print(fldr+'\n')
try:
os.makedirs(fldr+'_')
# adding a '_' to make the true folder name, i.e. foldername_ instead of foldername
except OSError:
pass
for f in raw_files:
print("File being analyzed is: {fn} \n".format(fn=f))
for fldr in folders:
print("Folder to move stuff to is {fold}\n".format(fold=fldr))
print("File {f} being checked . . . ".format(f=f))
if fldr in f and f[-1] != '_':
print("\t Moving file {fn} \n".format(fn=f))
shutil.move(f, fldr+'_')
The above is a program FileGrouper.py, which will look inside a directory for whether there are multiple files with a shared name (i.e. random.java, random.class). If there are, then they will be moved to a directory named [insert shared name]_ to organize them.
This code in particular refuses to work for a specific set of files.
Note: I replaced my actual Username, as well as the Timestamps with placeholders
-rwxr-xr-x 1 UserName staff 8480 Month 20 Time:Time another_ptr_func
-rw-r--r--# 1 UserName staff 324 Month 20 Time:Time another_ptr_func.c
-rwxr-xr-x 1 UserName staff 8572 Month 20 Time:Time arrays1
-rw-r--r-- 1 UserName staff 321 Month 20 Time:Time arrays1.c
-rw-r--r--# 1 UserName staff 2058 Month 20 Time:Time file_grouper.py
-rwxr-xr-x 1 UserName staff 8432 Month 20 Time:Time forloop
-rw-r--r-- 1 UserName staff 119 Month 20 Time:Time forloop.c
-rwxr-xr-x 1 UserName staff 4248 Month 20 Time:Time gen_a
-rw-r--r-- 1 UserName staff 53 Month 20 Time:Time gen_a.c
-rwxr-xr-x 1 UserName staff 8432 Month 20 Time:Time hello
-rw-r--r-- 1 UserName staff 65 Month 20 Time:Time hello.c
-rwxr-xr-x 1 UserName staff 8432 Month 20 Time:Time hello2
-rw-r--r-- 1 UserName staff 140 Month 20 Time:Time hello2.c
-rw-r--r--# 1 UserName staff 343 Month 20 Time:Time pointer.c
-rwxr-xr-x 1 UserName staff 8472 Month 20 Time:Time pointer_func
-rw-r--r-- 1 UserName staff 352 Month 20 Time:Time pointer_func.c
-rwxr-xr-x# 1 UserName staff 19340 Month 20 Time:Time pointer_hex
-rwxr-xr-x 1 UserName staff 8432 Month 20 Time:Time switch
-rw-r--r--# 1 UserName staff 219 Month 20 Time:Time switch.c
-rwxr-xr-x 1 UserName staff 8480 Month 20 Time:Time void_another_ptr_func
-rw-r--r-- 1 UserName staff 335 Month 20 Time:Time void_another_ptr_func.c
The above is the specific set of files that this code refuses to work for. I have tested this same code on the following, consisting of dummy files with zero size (used touch to create them):
-rw-r--r-- 1 UserName staff 1183 Month 20 Time:Time README.md
-rw-r--r--# 1 UserName staff 2067 Month 20 Time:Time file_grouper.py
-rw-r--r-- 1 UserName staff 0 Month 20 Time:Time random1
-rw-r--r-- 1 UserName staff 0 Month 20 Time:Time random1.txt
-rw-r--r-- 1 UserName staff 0 Month 20 Time:Time random2
-rw-r--r-- 1 UserName staff 0 Month 20 Time:Time random2.txt
-rw-r--r-- 1 UserName staff 0 Month 20 Time:Time random3
-rw-r--r-- 1 UserName staff 0 Month 20 Time:Time random3.txt
-rw-r--r-- 1 UserName staff 0 Month 20 Time:Time random4
-rw-r--r-- 1 UserName staff 0 Month 20 Time:Time random4.txt
-rw-r--r-- 1 UserName staff 0 Month 20 Time:Time random5.txt
The program worked successfully on these files, and generated the following:
.
├── README.md
├── file_grouper.py
├── random1_
│   ├── random1
│   └── random1.txt
├── random2_
│   ├── random2
│   └── random2.txt
├── random3_
│   ├── random3
│   └── random3.txt
├── random4_
│   ├── random4
│   └── random4.txt
└── random5.txt
As you can see, the program successfully ignored the lone random5.txt, as no other files with a shared name existed. However, the files where others with a shared name did exist were successfully grouped into a folder of [insert shared name]_

If by "refusing to work" you mean certain files are skipped, then that could be because of this line:
raw_files = [x for x in ls(curr_path) if '_' not in x]
All files with an underscore are removed.
Another issue is that file systems will not allow files and directories with the same name, which you cover by appending an underscore.
I'd strongly suggest to use the pathlib library, it makes your code more maintainable and readable:
from pathlib import Path
def file_grouper():
# Path to group
path = Path('.')
# List all files
ls = path.glob('*')
# Map all stems (file names without extension) to their file names
names = {}
for x in ls:
if x.is_file():
if x.stem not in names:
names[x.stem] = []
names[x.stem].append(x)
# Create and move
for stem, values in names.items():
if len(values) > 1:
(path / (stem + '_')).mkdir(exist_ok=True)
for value in values:
value.rename(path / (stem + '_') / value.name)
file_grouper()

python compressed 4Gb bz2 EOFError: end of stream was already found nested subfolders

I'm trying to read a specific file from a compressed file bz2 using python.
tar = tarfile.open(filename, "r|bz2", bufsize=57860311)
for tarinfo in tar:
print tarinfo.name, "is", tarinfo.size, "bytes in size and is",
if tarinfo.isreg():
print "a regular file."
# read the file
f = tar.extractfile(tarinfo)
#print f.read()
elif tarinfo.isdir():
print "a directory."
else:
print "something else."
tar.close()
But at the end I got the error:
/usr/local/Cellar/python#2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/tarfile.pyc in read(self, size)
577 buf = "".join(t)
578 else:
--> 579 buf = self._read(size)
580 self.pos += len(buf)
581 return buf
/usr/local/Cellar/python#2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/tarfile.pyc in _read(self, size)
594 break
595 try:
--> 596 buf = self.cmp.decompress(buf)
597 except IOError:
598 raise ReadError("invalid compressed data")
EOFError: end of stream was already found
I also tried to list the files within the tar through 'tar.list()' and again ...
-rwxr-xr-x lindauer/or3uunp 0 2013-05-21 00:58:36 r3.2/
-rw-r--r-- lindauer/or3uunp 6057 2012-01-05 14:41:00 r3.2/readme.txt
-rw-r--r-- lindauer/or3uunp 44732 2012-01-04 10:08:54 r3.2/psychometric.csv
-rw-r--r-- lindauer/or3uunp 57860309 2012-01-04 09:58:20 r3.2/logon.csv
/usr/local/Cellar/python#2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/tarfile.pyc in _read(self, size)
594 break
595 try:
--> 596 buf = self.cmp.decompress(buf)
597 except IOError:
598 raise ReadError("invalid compressed data")
EOFError: end of stream was already found
I listed the files inside the archive using the tar command. Here is the result:
tar -tvf r3.2.tar.bz2
drwxr-xr-x 0 lindauer or3uunp 0 May 21 2013 r3.2/
-rw-r--r-- 0 lindauer or3uunp 6057 Jan 5 2012 r3.2/readme.txt
-rw-r--r-- 0 lindauer or3uunp 44732 Jan 4 2012 r3.2/psychometric.csv
-rw-r--r-- 0 lindauer or3uunp 57860309 Jan 4 2012 r3.2/logon.csv
-rw-r--r-- 0 lindauer or3uunp 12494829865 Jan 5 2012 r3.2/http.csv
-rw-r--r-- 0 lindauer or3uunp 1066622500 Jan 5 2012 r3.2/email.csv
-rw-r--r-- 0 lindauer or3uunp 218962503 Jan 5 2012 r3.2/file.csv
-rw-r--r-- 0 lindauer or3uunp 29156988 Jan 4 2012 r3.2/device.csv
drwxr-xr-x 0 lindauer or3uunp 0 May 20 2013 r3.2/LDAP/
-rw-r--r-- 0 lindauer or3uunp 140956 Jan 4 2012 r3.2/LDAP/2011-01.csv
-rw-r--r-- 0 lindauer or3uunp 147370 Jan 4 2012 r3.2/LDAP/2010-05.csv
-rw-r--r-- 0 lindauer or3uunp 149221 Jan 4 2012 r3.2/LDAP/2010-02.csv
-rw-r--r-- 0 lindauer or3uunp 141717 Jan 4 2012 r3.2/LDAP/2010-12.csv
-rw-r--r-- 0 lindauer or3uunp 148931 Jan 4 2012 r3.2/LDAP/2010-03.csv
-rw-r--r-- 0 lindauer or3uunp 147370 Jan 4 2012 r3.2/LDAP/2010-04.csv
-rw-r--r-- 0 lindauer or3uunp 149793 Jan 4 2012 r3.2/LDAP/2009-12.csv
-rw-r--r-- 0 lindauer or3uunp 143979 Jan 4 2012 r3.2/LDAP/2010-09.csv
-rw-r--r-- 0 lindauer or3uunp 145591 Jan 4 2012 r3.2/LDAP/2010-07.csv
-rw-r--r-- 0 lindauer or3uunp 139444 Jan 4 2012 r3.2/LDAP/2011-03.csv
-rw-r--r-- 0 lindauer or3uunp 142347 Jan 4 2012 r3.2/LDAP/2010-11.csv
-rw-r--r-- 0 lindauer or3uunp 138285 Jan 4 2012 r3.2/LDAP/2011-04.csv
-rw-r--r-- 0 lindauer or3uunp 149793 Jan 4 2012 r3.2/LDAP/2010-01.csv
-rw-r--r-- 0 lindauer or3uunp 146008 Jan 4 2012 r3.2/LDAP/2010-06.csv
-rw-r--r-- 0 lindauer or3uunp 144711 Jan 4 2012 r3.2/LDAP/2010-08.csv
-rw-r--r-- 0 lindauer or3uunp 137967 Jan 4 2012 r3.2/LDAP/2011-05.csv
-rw-r--r-- 0 lindauer or3uunp 140085 Jan 4 2012 r3.2/LDAP/2011-02.csv
-rw-r--r-- 0 lindauer or3uunp 143420 Jan 4 2012 r3.2/LDAP/2010-10.csv
-r--r--r-- 0 lindauer or3uunp 3923 Jan 4 2012 r3.2/license.txt
I think this is due to the fact the archive has subfolders and for some reason python libraries have problems in dealing with subfolders extractions?
I also tried to open the tar file manually and I have no problems so I don't think the file is corrupted. Any help appreciated.

Comment: I tried the debug=3 and I get : ReadError: bad checksum
Found the following related Infos:
tar: directory checksum error
Cause
This error message from tar(1) indicates that the checksum of the directory and the files it has read from tape does not match the checksum advertised in the header block. Usually this message indicates the wrong blocking factor, although it could indicate corrupt data on tape.
Action
To resolve this problem, make certain that the blocking factor you specify on the command line (after -b) matches the blocking factor originally specified. If in doubt, leave out the block size and let tar(1) determine it automatically. If that remedy does not help, the tape data could be corrupted.
SE:tar-ignore-or-fix-checksum
I'd try the -i switch to see if you can just ignore and messages regarding EOF.
-i, --ignore-zeros ignore zeroed blocks in archive (means EOF)
Example
$ tar xivf backup.tar
bugs.python.org:tarfile-headererror
The comment in tarfile.py reads (Don't know the date of the file!):
- # We shouldn't rely on this checksum, because some tar programs
- # calculate it differently and it is merely validating the
- # header block.
ReadError: unexpected end of data
From the tarfile Documentation
The tarfile module defines the following exceptions:
exception tarfile.ReadError
Is raised when a tar archive is opened, that either cannot be handled by the tarfile module or is somehow invalid.
First, try with another tar archiv file to verify your python environent.
Second, check if your tar archiv file match the following format:
tarfile.DEFAULT_FORMAT
The default format for creating archives. This is currently GNU_FORMAT.
Third, instead of using tarfile.open(...), to create a tarfile instance, try to use the following, to set debug=3.
tar = tarfile.TarFile(name=filename, debug=3)
tar.open()
...
class tarfile.TarFile(name=None, mode='r', fileobj=None, format=DEFAULT_FORMAT, tarinfo=TarInfo, dereference=False, ignore_zeros=False, encoding=ENCODING, errors='surrogateescape', pax_headers=None, debug=0, errorlevel=0)

String formatting in a loop

I've got a directory full of JPG photographs. I want to take the file names of those photographs and end up with the following being printed:
<description>Test. <![CDATA[<img src='.
/files/fantaWP.jpg]>]]></description>
The file name is a variable. I've tried my very best below and i'm nearly there, but I end up with the following output:
<description>Test. <![CDATA[<img src='.
/files/['fantaWP.jpg', 'icon', 'p1.JPG', 'p2.JPG', 'p3.jpg', 'p4.jpg']>]]></description>
Here is my code:
photofileName = []
path='C:\Users\Simon\Desktop\Dir\pics'
dirList=os.listdir(path)
for fname in dirList:
photofileName.append(fname)
print photofileName
photoVar = [x for x in photofileName]
itemsInListOne = 3
iterations = itemsInListOne
num = 0
while num < iterations:
num = num+1
print ("\<description>Test. <![CDATA[<img src='./files/{}'>]]></description>\n".format(photoVar))
Thank you in advance.

The following should be enough if I understand you correctly.
for fname in os.listdir(path):
print("\<description>Test. <![CDATA[<img src='./files/{}'>]]>=</description>\n".format(fname))
Example:
>>> path = "/home/msvalkon/Pictures/Sample Album"
>>> for fname in os.listdir(path):
... print("\<description>Test. <![CDATA[<img src='./files/{}'>]]>=</description>\n".format(fname))
...
...
\<description>Test. <![CDATA[<img src='./files/Costa Rican Frog.jpg'>]]>=</description>
\<description>Test. <![CDATA[<img src='./files/Pensive Parakeet.jpg'>]]>=</description>
\<description>Test. <![CDATA[<img src='./files/Boston City Flow.jpg'>]]>=</description>
>>>
And the content of the path..
msvalkon#Lunkwill:~/Pictures/Sample Album$ ll
total 1208
drwxrwxr-x 2 msvalkon msvalkon 4096 Apr 19 2012 ./
drwxr-xr-x 7 msvalkon msvalkon 28672 Jan 3 18:27 ../
-rw-rw-r-- 1 msvalkon msvalkon 339773 Dec 13 2009 Boston City Flow.jpg
-rw-rw-r-- 1 msvalkon msvalkon 354633 Dec 13 2009 Costa Rican Frog.jpg
-rw-rw-r-- 1 msvalkon msvalkon 480098 Dec 13 2009 Pensive Parakeet.jpg

Cant import psycopg2

I am trying to install postgresql_python.
I downloaded the tarball and installed it using:
python setup.py build
python setup.py install
I got /usr/lib64/python2.4/site-packages/psycopg2/ with
> total 836
> -rw-r--r-- 1 root root 12759 Dec 11 18:18 errorcodes.py
> -rw-r--r-- 1 root root 14584 Dec 12 13:49 errorcodes.pyc
> -rw-r--r-- 1 root root 14584 Dec 12 13:49 errorcodes.pyo
> -rw-r--r-- 1 root root 5807 Dec 11 18:18 extensions.py
> -rw-r--r-- 1 root root 7298 Dec 12 13:49 extensions.pyc
> -rw-r--r-- 1 root root 7298 Dec 12 13:49 extensions.pyo
> -rw-r--r-- 1 root root 31495 Dec 11 18:18 extras.py
> -rw-r--r-- 1 root root 35124 Dec 12 13:49 extras.pyc
> -rw-r--r-- 1 root root 35124 Dec 12 13:49 extras.pyo
> -rw-r--r-- 1 root root 6177 Dec 11 18:18 __init__.py
> -rw-r--r-- 1 root root 5740 Dec 12 13:49 __init__.pyc
> -rw-r--r-- 1 root root 5740 Dec 12 13:49 __init__.pyo
> -rw-r--r-- 1 root root 8855 Dec 11 18:18 pool.py
> -rw-r--r-- 1 root root 8343 Dec 12 13:49 pool.pyc
> -rw-r--r-- 1 root root 8343 Dec 12 13:49 pool.pyo
> -rw-r--r-- 1 root root 3389 Dec 21 11:17 psycopg1.py
> -rw-r--r-- 1 root root 3182 Dec 21 11:22 psycopg1.pyc
> -rw-r--r-- 1 root root 3167 Dec 12 13:49 psycopg1.pyo
> -rwxr-xr-x 1 root root 572648 Dec 21 11:22 _psycopg.so drwxr-xr-x 2 root root 4096 Dec 21 10:38 tests
> -rw-r--r-- 1 root root 4427 Dec 11 18:18 tz.py
> -rw-r--r-- 1 root root 4325 Dec 12 13:49 tz.pyc
> -rw-r--r-- 1 root root 4325 Dec 12 13:49 tz.pyo
But in python shell when I am trying to import library, I got error:
>>> import psycopg2
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "/usr/lib64/python2.4/site-packages/psycopg2/__init__.py", line 76, in ?
from psycopg2._psycopg import _connect, apilevel, threadsafety, paramstyle
ImportError: cannot import name _connect
I am running with Postgresql 9.2.
What am I missing here?
Please let me know.
Thanks.

You most likely have to remove some existing packages related to psycopg2 within your root. Some common locations:
rm -r /usr/lib/python2.4/site-packages/psycopg2*
rm -r /usr/local/lib/python2.6/dist-packages/psycopg2*
However, I recommend setting up a virtualenv to house the packages for your Python app.
Check out virtualenv. It's easy to use once installed:
virtualenv myapp
. myapp/bin/activate
cd ~/your/postgres_lib/download
python setup.py install
This will install postgres libraries into your virtualenv (located under the myapp) folder. Then, whenever you want to run your app, you just need to activate the environment via
. myapp/bin/activate
Adjusting the path to myapp when necessary. There are helpers, like virtualenvwrapper to streamline this process.