We Keep Coding

FileNotFoundError error while trying to rename the symlinks using os.rename [duplicate]

I would like to know is it possible to rename a symlink with python.
Already tried os.rename and shutil.move
Any ideas?
os.rename return me this error : OSError: [Errno 18] Cross-device link
>>> import sys, os
>>>
>>> path = '/Library/Application Support/appsolute/MAMP PRO/db/'
>>> job = path + 'mysql-job/'
>>> perso = path + 'mysql-perso/'
>>> mysql = path + 'mysql/'
>>>
>>> os.rename(mysql, job)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
OSError: [Errno 18] Cross-device link
>>> exit()
Danny-Dombrowski:script ddombrowski$ ls -al /Library/Application\ Support/appsolute/MAMP\ PRO/db/
total 24
drwxrwxr-x 5 root admin 170 7 fév 19:29 .
drwxrwxr-x 12 root admin 408 7 fév 17:14 ..
-rw-r--r--# 1 ddombrowski admin 6148 7 fév 19:29 .DS_Store
lrwxr-xr-x 1 ddombrowski admin 46 7 fév 19:29 mysql -> /Volumes/Gestion Portail Sante/Database/mysql/
drwxrwxr-x 11 ddombrowski admin 374 7 fév 19:22 mysql-perso

os.rename should work.
xupeng#xupeng t $ ls -l
total 0
-rw-r--r-- 1 xupeng xupeng 0 Feb 8 08:22 a
lrwxrwxrwx 1 xupeng xupeng 1 Feb 8 08:23 b -> a
xupeng#xupeng t $ python
Python 2.6.5 (release26-maint, Sep 21 2011, 10:32:38)
[GCC 4.3.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.rename('b', 'c')
xupeng#xupeng t $ ls -l
total 0
-rw-r--r-- 1 xupeng xupeng 0 Feb 8 08:22 a
lrwxrwxrwx 1 xupeng xupeng 1 Feb 8 08:23 c -> a

os.rename will work fine:
$ ln -s target link
$ python -c "import os; os.rename('link', 'link.new')"
$ ls -l link.new
lrwxrwxrwx 1 phihag phihag 6 Feb 8 01:25 link.new -> target
Make sure you don't include a / after the symlink: link/ is the same as link/., and not the same as link.

python compressed 4Gb bz2 EOFError: end of stream was already found nested subfolders

I'm trying to read a specific file from a compressed file bz2 using python.
tar = tarfile.open(filename, "r|bz2", bufsize=57860311)
for tarinfo in tar:
print tarinfo.name, "is", tarinfo.size, "bytes in size and is",
if tarinfo.isreg():
print "a regular file."
# read the file
f = tar.extractfile(tarinfo)
#print f.read()
elif tarinfo.isdir():
print "a directory."
else:
print "something else."
tar.close()
But at the end I got the error:
/usr/local/Cellar/python#2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/tarfile.pyc in read(self, size)
577 buf = "".join(t)
578 else:
--> 579 buf = self._read(size)
580 self.pos += len(buf)
581 return buf
/usr/local/Cellar/python#2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/tarfile.pyc in _read(self, size)
594 break
595 try:
--> 596 buf = self.cmp.decompress(buf)
597 except IOError:
598 raise ReadError("invalid compressed data")
EOFError: end of stream was already found
I also tried to list the files within the tar through 'tar.list()' and again ...
-rwxr-xr-x lindauer/or3uunp 0 2013-05-21 00:58:36 r3.2/
-rw-r--r-- lindauer/or3uunp 6057 2012-01-05 14:41:00 r3.2/readme.txt
-rw-r--r-- lindauer/or3uunp 44732 2012-01-04 10:08:54 r3.2/psychometric.csv
-rw-r--r-- lindauer/or3uunp 57860309 2012-01-04 09:58:20 r3.2/logon.csv
/usr/local/Cellar/python#2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/tarfile.pyc in _read(self, size)
594 break
595 try:
--> 596 buf = self.cmp.decompress(buf)
597 except IOError:
598 raise ReadError("invalid compressed data")
EOFError: end of stream was already found
I listed the files inside the archive using the tar command. Here is the result:
tar -tvf r3.2.tar.bz2
drwxr-xr-x 0 lindauer or3uunp 0 May 21 2013 r3.2/
-rw-r--r-- 0 lindauer or3uunp 6057 Jan 5 2012 r3.2/readme.txt
-rw-r--r-- 0 lindauer or3uunp 44732 Jan 4 2012 r3.2/psychometric.csv
-rw-r--r-- 0 lindauer or3uunp 57860309 Jan 4 2012 r3.2/logon.csv
-rw-r--r-- 0 lindauer or3uunp 12494829865 Jan 5 2012 r3.2/http.csv
-rw-r--r-- 0 lindauer or3uunp 1066622500 Jan 5 2012 r3.2/email.csv
-rw-r--r-- 0 lindauer or3uunp 218962503 Jan 5 2012 r3.2/file.csv
-rw-r--r-- 0 lindauer or3uunp 29156988 Jan 4 2012 r3.2/device.csv
drwxr-xr-x 0 lindauer or3uunp 0 May 20 2013 r3.2/LDAP/
-rw-r--r-- 0 lindauer or3uunp 140956 Jan 4 2012 r3.2/LDAP/2011-01.csv
-rw-r--r-- 0 lindauer or3uunp 147370 Jan 4 2012 r3.2/LDAP/2010-05.csv
-rw-r--r-- 0 lindauer or3uunp 149221 Jan 4 2012 r3.2/LDAP/2010-02.csv
-rw-r--r-- 0 lindauer or3uunp 141717 Jan 4 2012 r3.2/LDAP/2010-12.csv
-rw-r--r-- 0 lindauer or3uunp 148931 Jan 4 2012 r3.2/LDAP/2010-03.csv
-rw-r--r-- 0 lindauer or3uunp 147370 Jan 4 2012 r3.2/LDAP/2010-04.csv
-rw-r--r-- 0 lindauer or3uunp 149793 Jan 4 2012 r3.2/LDAP/2009-12.csv
-rw-r--r-- 0 lindauer or3uunp 143979 Jan 4 2012 r3.2/LDAP/2010-09.csv
-rw-r--r-- 0 lindauer or3uunp 145591 Jan 4 2012 r3.2/LDAP/2010-07.csv
-rw-r--r-- 0 lindauer or3uunp 139444 Jan 4 2012 r3.2/LDAP/2011-03.csv
-rw-r--r-- 0 lindauer or3uunp 142347 Jan 4 2012 r3.2/LDAP/2010-11.csv
-rw-r--r-- 0 lindauer or3uunp 138285 Jan 4 2012 r3.2/LDAP/2011-04.csv
-rw-r--r-- 0 lindauer or3uunp 149793 Jan 4 2012 r3.2/LDAP/2010-01.csv
-rw-r--r-- 0 lindauer or3uunp 146008 Jan 4 2012 r3.2/LDAP/2010-06.csv
-rw-r--r-- 0 lindauer or3uunp 144711 Jan 4 2012 r3.2/LDAP/2010-08.csv
-rw-r--r-- 0 lindauer or3uunp 137967 Jan 4 2012 r3.2/LDAP/2011-05.csv
-rw-r--r-- 0 lindauer or3uunp 140085 Jan 4 2012 r3.2/LDAP/2011-02.csv
-rw-r--r-- 0 lindauer or3uunp 143420 Jan 4 2012 r3.2/LDAP/2010-10.csv
-r--r--r-- 0 lindauer or3uunp 3923 Jan 4 2012 r3.2/license.txt
I think this is due to the fact the archive has subfolders and for some reason python libraries have problems in dealing with subfolders extractions?
I also tried to open the tar file manually and I have no problems so I don't think the file is corrupted. Any help appreciated.

Comment: I tried the debug=3 and I get : ReadError: bad checksum
Found the following related Infos:
tar: directory checksum error
Cause
This error message from tar(1) indicates that the checksum of the directory and the files it has read from tape does not match the checksum advertised in the header block. Usually this message indicates the wrong blocking factor, although it could indicate corrupt data on tape.
Action
To resolve this problem, make certain that the blocking factor you specify on the command line (after -b) matches the blocking factor originally specified. If in doubt, leave out the block size and let tar(1) determine it automatically. If that remedy does not help, the tape data could be corrupted.
SE:tar-ignore-or-fix-checksum
I'd try the -i switch to see if you can just ignore and messages regarding EOF.
-i, --ignore-zeros ignore zeroed blocks in archive (means EOF)
Example
$ tar xivf backup.tar
bugs.python.org:tarfile-headererror
The comment in tarfile.py reads (Don't know the date of the file!):
- # We shouldn't rely on this checksum, because some tar programs
- # calculate it differently and it is merely validating the
- # header block.
ReadError: unexpected end of data
From the tarfile Documentation
The tarfile module defines the following exceptions:
exception tarfile.ReadError
Is raised when a tar archive is opened, that either cannot be handled by the tarfile module or is somehow invalid.
First, try with another tar archiv file to verify your python environent.
Second, check if your tar archiv file match the following format:
tarfile.DEFAULT_FORMAT
The default format for creating archives. This is currently GNU_FORMAT.
Third, instead of using tarfile.open(...), to create a tarfile instance, try to use the following, to set debug=3.
tar = tarfile.TarFile(name=filename, debug=3)
tar.open()
...
class tarfile.TarFile(name=None, mode='r', fileobj=None, format=DEFAULT_FORMAT, tarinfo=TarInfo, dereference=False, ignore_zeros=False, encoding=ENCODING, errors='surrogateescape', pax_headers=None, debug=0, errorlevel=0)

increase paramiko.SSHClient.exec_command() width

I'm using paramiko.SSHClient.exec_command() to run commands on remote servers.
Does someone know if it's possible to change the width, like with invoke_shell(width=150) ?
When I exec_command("ls -la") I get :
drwx------. 6 myuser myuser 4096 25 avril 15:59
.
drwxr-xr-x. 14 root root 4096 5 mai 15:05
..
-rw-------. 1 myuser myuser 2818 28 avril 11:09
.bash_history
-rw-r--r--. 1 myuser myuser 340 14 avril 14:16
.bashrc
and I want :
drwx------. 6 myuser myuser 4096 25 avril 15:59 .
drwxr-xr-x. 14 root root 4096 5 mai 15:05 ..
-rw-------. 1 myuser myuser 2818 28 avril 11:09 .bash_history
-rw-r--r--. 1 myuser myuser 340 14 avril 14:16 .bashrc
(using exec_command, not invoke_shell)
My code :
ssh_client = paramiko.SSHClient()
mykey = paramiko.RSAKey.from_private_key_file("/path/to/my/key", password="passphrase")
ssh_client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh_client.connect("myserver.mydomain.com", username="myuser", pkey=mykey)
transport = ssh_client.get_transport()
agent_channel = transport.open_session()
agent_handler = paramiko.agent.AgentRequestHandler(agent_channel)
stdin, stdout, stderr = ssh_client.exec_command("ls -la")

Ok, found it: it was just a pprint strange behavior. I put the received lines in a list, and then pprint this list. If I do :
for line in received:
print(line)
Then It's ok.
pprint printed :
['-rw-r----- 1 myuser mygroup 23228063744 06 mai 11:41 '
'my_file.txt-rw-r----- 1 '
...
I don't know why. Well, I'll stop using pprint.

Korn Shell Script on Linux Red Hat

My default shell is Bash and I am on directory /opt/app/IRMS and execute a script /opt/app/depl/bin/runjob.sh. Runall exists and is a soft link.
I get the following error :
[tidal#atlesapp002 IRMS]$ ksh -x /opt/app/depl/bin/runjob.sh execute
ls irms_inq.env
+ myenv=irms_inq.env
+ . ./irms_inq.env
+ . /home/tidal/cfg/setenv
+ export PATH
+ export LD_LIBRARY_PATH
+ export FPATH
+ export SQLPATH
+ PATH=/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/tidal/bin:.
+ . /home/tidal/cfg/python.env
+ PY=/opt/python
+ PATH=/opt/python/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/tidal/bin:.
+ LD_LIBRARY_PATH=:/opt/python/lib
+ . /home/tidal/cfg/svn.env
+ SVN=/opt/app/svn
+ PATH=/opt/app/svn/bin:/opt/python/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/tidal/bin:.
+ LD_LIBRARY_PATH=/opt/app/svn/lib::/opt/python/lib
+ . /home/tidal/cfg/ora.env
+ CLI=/opt/app/instantclient_11_2
+ NLS_LANG=AMERICAN_AMERICA.UTF8
+ export NLS_LANG
+ LD_LIBRARY_PATH=/opt/app/instantclient_11_2:/opt/app/svn/lib::/opt/python/lib
+ export LD_LIBRARY_PATH
+ TNS_ADMIN=/opt/app/instantclient_11_2
+ export TNS_ADMIN
+ PATH=/opt/app/svn/bin:/opt/python/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/tidal/bin:.:/opt/app/instantclient_11_2
+ export PATH
+ . /home/tidal/cfg/depl.env
+ DEPL=/opt/app/depl
+ PATH=/opt/app/svn/bin:/opt/python/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/tidal/bin:.:/opt/app/instantclient_11_2:/opt/app/depl/bin
+ FPATH=:/opt/app/depl/lib
+ SQLPATH=:/opt/app/depl/lib
+ . /home/tidal/cfg/putils.env
+ PATH=/opt/app/svn/bin:/opt/python/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/tidal/bin:.:/opt/app/instantclient_11_2:/opt/app/depl/bin:/opt/app/putils/bin
+ SQLPATH=:/opt/app/depl/lib:/opt/app/putils/lib
+ pwd
+ irmspwd=/opt/app/IRMS
+ PATH=/opt/app/svn/bin:/opt/python/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/tidal/bin:.:/opt/app/instantclient_11_2:/opt/app/depl/bin:/opt/app/putils/bin:/opt/app/IRMS/bin
+ SQLPATH=:/opt/app/depl/lib:/opt/app/putils/lib:/opt/app/IRMS/lib
+ SQLPATH=:/opt/app/depl/lib:/opt/app/putils/lib:/opt/app/IRMS/lib:/opt/app/IRMS/src
+ runall execute
/opt/app/depl/bin/runjob.sh[7]: runall: not found [No such file or directory]
[tidal#atlesapp002 IRMS]$
====
[tidal#atlesapp002 bin]$ ls -al
total 96
drwxrwxr-x 3 tidal tidal 4096 Jun 9 13:29 .
drwxrwxr-x 4 tidal tidal 4096 May 19 16:38 ..
lrwxrwxrwx 1 tidal tidal 8 Jun 8 12:56 runall -> setup.sh
-rwxr-xr-x 1 tidal tidal 2319 May 19 16:38 runall.save
-rwxr-xr-x 1 tidal tidal 831 May 19 16:38 runcp.sh
-rwxr-xr-x 1 tidal tidal 54 Jun 9 13:29 runjob.sh
-rwxr-xr-x 1 tidal tidal 2242 May 19 16:38 runsockets.pl
lrwxrwxrwx 1 tidal tidal 9 Jun 9 00:01 runsql -> runsql.sh
-rwxr-xr-x 1 tidal tidal 1877 May 19 16:38 runsql.save
-rwxr-xr-x 1 tidal tidal 1877 May 19 16:38 runsql.sh
-rwxr-xr-x 1 tidal tidal 5590 May 19 16:38 setParm.pl
-rwxr-xr-x 1 tidal tidal 2319 May 19 16:38 setup.sh

It seems runall started from $PATH and not from current directory ./runall, since /opt/app/IRMS/bin was added to the $PATH.
Try the following:
start ksh [tidal#atlesapp002 IRMS]$ ksh
run /opt/app/depl/bin/runjob.sh execute
If this does not help try adding /opt/app/IRMS/bin to PATH and then export PATH

Yes when I added ksh to the runall script it worked.
like ksh runall
Why ?

Cant import psycopg2

I am trying to install postgresql_python.
I downloaded the tarball and installed it using:
python setup.py build
python setup.py install
I got /usr/lib64/python2.4/site-packages/psycopg2/ with
> total 836
> -rw-r--r-- 1 root root 12759 Dec 11 18:18 errorcodes.py
> -rw-r--r-- 1 root root 14584 Dec 12 13:49 errorcodes.pyc
> -rw-r--r-- 1 root root 14584 Dec 12 13:49 errorcodes.pyo
> -rw-r--r-- 1 root root 5807 Dec 11 18:18 extensions.py
> -rw-r--r-- 1 root root 7298 Dec 12 13:49 extensions.pyc
> -rw-r--r-- 1 root root 7298 Dec 12 13:49 extensions.pyo
> -rw-r--r-- 1 root root 31495 Dec 11 18:18 extras.py
> -rw-r--r-- 1 root root 35124 Dec 12 13:49 extras.pyc
> -rw-r--r-- 1 root root 35124 Dec 12 13:49 extras.pyo
> -rw-r--r-- 1 root root 6177 Dec 11 18:18 __init__.py
> -rw-r--r-- 1 root root 5740 Dec 12 13:49 __init__.pyc
> -rw-r--r-- 1 root root 5740 Dec 12 13:49 __init__.pyo
> -rw-r--r-- 1 root root 8855 Dec 11 18:18 pool.py
> -rw-r--r-- 1 root root 8343 Dec 12 13:49 pool.pyc
> -rw-r--r-- 1 root root 8343 Dec 12 13:49 pool.pyo
> -rw-r--r-- 1 root root 3389 Dec 21 11:17 psycopg1.py
> -rw-r--r-- 1 root root 3182 Dec 21 11:22 psycopg1.pyc
> -rw-r--r-- 1 root root 3167 Dec 12 13:49 psycopg1.pyo
> -rwxr-xr-x 1 root root 572648 Dec 21 11:22 _psycopg.so drwxr-xr-x 2 root root 4096 Dec 21 10:38 tests
> -rw-r--r-- 1 root root 4427 Dec 11 18:18 tz.py
> -rw-r--r-- 1 root root 4325 Dec 12 13:49 tz.pyc
> -rw-r--r-- 1 root root 4325 Dec 12 13:49 tz.pyo
But in python shell when I am trying to import library, I got error:
>>> import psycopg2
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "/usr/lib64/python2.4/site-packages/psycopg2/__init__.py", line 76, in ?
from psycopg2._psycopg import _connect, apilevel, threadsafety, paramstyle
ImportError: cannot import name _connect
I am running with Postgresql 9.2.
What am I missing here?
Please let me know.
Thanks.

You most likely have to remove some existing packages related to psycopg2 within your root. Some common locations:
rm -r /usr/lib/python2.4/site-packages/psycopg2*
rm -r /usr/local/lib/python2.6/dist-packages/psycopg2*
However, I recommend setting up a virtualenv to house the packages for your Python app.
Check out virtualenv. It's easy to use once installed:
virtualenv myapp
. myapp/bin/activate
cd ~/your/postgres_lib/download
python setup.py install
This will install postgres libraries into your virtualenv (located under the myapp) folder. Then, whenever you want to run your app, you just need to activate the environment via
. myapp/bin/activate
Adjusting the path to myapp when necessary. There are helpers, like virtualenvwrapper to streamline this process.