We Keep Coding

Not able to run python file in Cron Job in putty

I am using Putty and it does not have Python3 it has python2 or python so tried both to run python file by using command
55 08 * * * /usr/bin/python2 /home/user/file.py
and couple of other commands BUT nothing is working.
The python file I have runs totally fine with spark2-submit command. This is a pyspark file converted to python. When I use /usr/bin/python2 I get error for line "from pyspark.sql import sparksession" - error-> No module named pyspark.sql.
I think spark2-submit is not supported in corntab job. and /usr/bin/python2 is giving error for pyspark convered python file.
Can anyone please help me out here.

Cronjob commands should contain minute(m), hour(h), day of month(dom), month(mon) and day of week(dow).
You should write them using the format below :
# (m) (h) (dom) (mon) (dow) command
You're missing one parameter in your case.
For example, if you want to run your code at 08:55 everyday, you can use :
55 08 * * * /usr/bin/python2 /home/user/file.py

how to use the same environment in terminal with what's in jupyter notebook

I'm a greener in anaconda. Forgive me if I ask a stupid question. The following is the issue I faced. And thanks a lot in advance to take your time.
I installed anaconda2 and used jupyter notebook to run the python program.
One program used opencv. When I run the problem in notebook, there's no problem at all.
While when I run the program with terminal, there's following error:
Traceback (most recent call last):
File "classify.py", line 9, in <module>
import imutils
File "/home/oakcloud/anaconda2/lib/python2.7/site-packages/imutils/__init__.py", line 8, in <module>
from .convenience import translate
File "/home/oakcloud/anaconda2/lib/python2.7/site-packages/imutils/convenience.py", line 6, in <module>
import cv2
ImportError: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.22' not found (required by /home/oakcloud/anaconda2/lib/python2.7/site-packages/../../libopencv_objdetect.so.3.3)
There's only one conda env named base in my PC, and I have such problem after I enter the env with source activate base command. I had thought notebook should use the same env.
The issue above complained the lib dismatch. I checked the lib in sys directory:
oakcloud#ubuntu:/usr/lib/x86_64-linux-gnu$ pwd
/usr/lib/x86_64-linux-gnu
oakcloud#ubuntu:/usr/lib/x86_64-linux-gnu$ ll libstdc++.so.*
lrwxrwxrwx 1 root root 19 Feb 6 12:07 libstdc++.so.6 -> libstdc++.so.6.0.21
-rw-r--r-- 1 root root 1566440 Feb 6 12:21 libstdc++.so.6.0.21
To fix the problem, I think I can just replace the lib with a new version.
While I found in anaconda, the lib is already installed：
oakcloud#ubuntu:~/anaconda2$ ll ./pkgs/libstdcxx-ng-7.2.0-h7a57d05_2/x86_64-conda_cos6-linux-gnu/sysroot/lib/libstdc++.so.6
lrwxrwxrwx 1 oakcloud oakcloud 32 Oct 6 2017 ./pkgs/libstdcxx-ng-7.2.0-h7a57d05_2/x86_64-conda_cos6-linux-gnu/sysroot/lib/libstdc++.so.6 -> ../../../lib/libstdc++.so.6.0.24*
So my question is how can I use the same setting in shell with what's in notebook?
cheers,
Julian.

Scheduling cronjob using python cannot find module bs4

I wrote a script to scrape some web data that I would like to run every 10 minutes. The script runs fine when I do it from the command line but I get an error when I try to run it using a cronjob every 10 minutes because it can't import bs4'. I get this error:
No module named bs4
My crontab:
*/10 * * * * PYTHONPATH=/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages python /Users/{username}/Documents/scripts/cot.py
I got the PYTHONPATH variable by finding where 'bs4' was downloaded using this command:
python -c 'import bs4; print(bs4.__file__)'
Output in /var/{username}/mail:
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME={username}>
X-Cron-Env: <USER={username}>
X-Cron-Env: <HOME=/Users/{username}>
Message-Id: <20171223194001.0783F8639E1#mbp-0500.localdomain>
Date: Sat, 23 Dec 2017 13:40:00 -0600 (CST)
Traceback (most recent call last):
File "/Users/{username}/Documents/scripts/cot.py", line 3, in <module>
from bs4 import BeautifulSoup
ImportError: No module named bs4

Had the same problem. In my case the job was executed by root and not in user mode. Be sure to execute the cronjob in user mode. The right solution was answered here

I was having the same issue with bs4 and python.
I eventually solved it by creating a shell script (making it executable with chmod +x) then executing that script from the command line (./scraper.sh)
bs4 was installed but it threw an error so I downloaded it again with pip using the --user flag (pip3 install bs4 --user), and it worked fine.
Then I set the crontab to run the shell script (I think running it as a shell script preserves some of the environmental variables that python or conda needs, read that in a different question)
Hope this helps!

Permission denied on dl.open() with ipython but not with python

My initial goal is to open a dll file on Cygwin using ctypes. However I found some issues with it. I dug up to sys.dl which returns an unknown Permission denied only on IPython.
With python everything looks fine:
$ ls
my.dll
$ python
Python 2.7.8 (default, Jul 28 2014, 01:34:03)
[GCC 4.8.3] on cygwin
>>> import dl
>>> dl.open('my.dll')
<dl.dl object at 0xfffaa0c0>
With ipython I get the error:
$ ipython
Python 2.7.8 (default, Jul 28 2014, 01:34:03)
In [1]: import dl
In [2]: dl.open('my.dll')
---------------------------------------------------------------------------
error Traceback (most recent call last)
<ipython-input-2-c681630fa713> in <module>()
----> 1 dl.open('my.dll')
error: Permission denied
I investigated on this using strace. The output log for `IPython is huge, more than 4MB. Fortunately, I identified some weird things:
symlink.check(C:\Users\user\Home\projects\foo\my.dll, 0x28AB88) (0x4022)
35 2705178 [main] python2.7 16924 path_conv::check: this->path(C:\Users\user\Home\projects\foo\my.dll), has_acls(1)
37 2705215 [main] python2.7 16924 cwdstuff::get: posix /cygdrive/c/Users/user/Home/projects/foo
32 2705247 [main] python2.7 16924 cwdstuff::get: (C:\Users\user\Home\projects\foo) = cwdstuff::get (0x8006ECF0, 32768, 0, 0), errno 11
--- Process 14376, exception c0000138 at 7726163E
3286 2708533 [main] python2.7 16924 seterrno_from_win_error: /home/corinna/src/cygwin/cygwin-1.7.35/cygwin-1.7.35-1.i686/src/src/winsup/cygwin/dlfcn.cc:174 windows error 182
42 2708575 [main] python2.7 16924 geterrno_from_win_error: unknown windows error 182, setting errno to 13
36 2708611 [main] python2.7 16924 dlopen: ret 0x0
Who is /home/corinna? I have no corinna user in my installation, neither on my Windows. Corinna does not come from my installation. Is it some hard-coded stuff?
Now, here is what I get from strace for python:
symlink.check(C:\Users\user\Home\projects\foo\my.dll, 0x28B728) (0x4022)
26 10440048 [main] python 12604 path_conv::check: this->path(C:\Users\user\Home\projects\foo\my.dll), has_acls(1)
23 10440071 [main] python 12604 cwdstuff::get: posix /cygdrive/c/Users/user/Home/projects/foo
25 10440096 [main] python 12604 cwdstuff::get: (C:\Users\user\Home\projects\foo) = cwdstuff::get (0x8006ECF0, 32768, 0, 0), errno 0
3405 10443501 [main] python 12604 dlopen: ret 0x5B9C0000
dlopen is returning 0x0 in IPython while it is returning 0x5B9C0000 for python. I notice that cwdstuff::get is raising an error before dlopen is called.
EDIT
I sent a message to Cygwin's mailing list and the answer of Corinna regarding this issue is:
This is not Cygwin's fault, AFAICS. Cygwin never loads functions by
ordinal. This is also a bit on the lean side as far as information is
concerned. One can't see how the process calls dlopen, for instance.
Corinna
How to solve this issue?
My earlier tests using ctypes
Initially when I asked my question I was just playing with ctypes.
I am working on Cygwin 32-bit and Windows 7.
With IPython I got an OSError when I tried to load a dll using cdll.LoadLibrary.

Two ideas:
1) in the next cell, type %pdb, and then interactively "print self._name" to see what it is.
2) Use a full path to cdll.LoadLibrary("foo.dll") to see if that works.
Once you know what the issue is, then you can decide whose bug it is, and report it (could be a ctypes issue, but probably ipython)

More ideas:
Check that the user accessing the DLL is the same. You do that like this:
import getpass
print(getpass.getuser())
Check what's the current process actually doing. I haven't used cygwin but in the linux shell the executable strace should show you this.
Usage: get the PID of your current process: import os; os.getpid()
After this, you can use (from outside the python/ipython console the command strace -p <the pid> -e file. After this setup, you can try to load your DLL.
Remarks: the -e file flag should be written exactly like that. The word file tells strace to report all file operations that the process makes. If no differences occur when running on python/ ipython, you can try dropping the -e file flag. Then you will see all system calls that the process makes. I haven't worked on windows like that, so this might not work at all there, but on linux at least, this should report to you everything that the process did. You could see there at least all the files that got opened, but more interesting things could be found there as well. If the outputs are identical, then the problem can be debugged further in python/ ipython. This would require basically what #Doug Blank suggested, but I'd also recommend investigating every name (variable) that gets touched. The self, _dlopen and mode names also sound like they might contain useful information.
Otherwise, do a dir(self) and dir(_dlopen) to see what other properties you could find there that might have gotten modified by IPYthon.
Try these first, and after that we can help you dig further.

Perhaps Python executable and IPython kernel use different manifest files which define loading policy?
Try appending the DLL path to sys.path in both cases.
Check admin rights (UAC) in both cases.
Use dependency walker to figure out dependencies of this DLL. Maybe the problem comes from dependencies?
Possibly your machine has multiple copies of this DLL?
Finally you can use Process Explorer to see the list of DLLs loaded in both cases and see any differences.
I'm working on a very similar issue:
ipython notebook & script difference - loading DLLs

Can't import the cx_Oracle module unless I'm using an interactive shell

When using Python on an interactive shell I'm able to import the cx_Oracle file with no problem. Ex:
me#server~/ $ python
Python 2.7.6 (default, Mar 22 2014, 22:59:56)
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import cx_Oracle
>>>
As you can see, importing works without a hitch. However, when I try to run a Python script doing the same thing, I get an error:
me#server~/ $ sudo script.py
Traceback (most recent call last):
File "/usr/local/bin/script.py", line 19, in <module>
import cx_Oracle
ImportError: No module named "cx_Oracle'
Here is the important section from script.py:
# 16 other lines above here
# Imports
import sys
import cx_Oracle
import psycopg2
...
I'm befuddled here. Other pertinent information is the server I'm running is Ubuntu 14.04.1 LTS (upgraded from 12.04) 64bit. which python and sudo which python both point to the same location. Also, doing this as root via sudo su - gets the same results; import OK from interactive but error from script.
Nothing other than the OS upgrade happened between when this worked and when it stopped working.

Sorry, all. This was a silly on my part. Turns out the script in question was using Python3, and when the server upgraded, Python3 went from being 3.2 version to being 3.4 version.
Once the cx_Oracle module was set up in the 3.4 version, everything worked as expected.
Phil, your final note talking about the shebang was what lead me to discover this, so kudos to you! The reason I didn't mark your response as the answer was because technically it wasn't but led me on the right path.
Cheers!

sudo starts a new bash environment which is then pointing to a different python executable (different installed modules).
You can verify this with which python and sudo which python
EDIT: so if they point to the same executable, then you should look at sys.path to find differences. In both environemnts you can:
python -c "import sys; print('\n'.join(sys.path))"
sudo python -c "import sys; print('\n'.join(sys.path))"
Look for differences. If there are none:
A common error in import situations like this is that python will first look at the local dir. So if you happen to be running python and importing something what is found locally (i.e. cx_Oracle is a subdir of your current location), you will get an import error if you change directories.
Final note: I have assumed here that the shbang of the script.py points to the same executable as which python. That is, that python script.py and script.py return the same error.