I have a python3.2 script that's supposed to delete a folder after everything is done:
def perforce_backup(
source,
destination,
tmp_location,
zip_tmp_loc,
):
logger.info('--------------------Perforce Backup--------------------'
)
logger.info('--- Check integrity of perforce depot (p4 verify)')
p4verify(source, 'user', 'password')
logger.info('--- Create a checkpoint (p4 admin checkpoint)')
p4checkpoint(source, 'user', 'password')
logger.info('--- Do the backup locally')
rsync(source, tmp_location)
logger.info('--- Zip perforce db and depot locally')
zipdir(tmp_location, zip_tmp_loc)
logger.info('--- Remove file from last folder on backup FTP')
shutil.rmtree(destination.path)
makedir(destination.path)
logger.info('--- Move zip to backup FTP')
cp(zip_tmp_loc.path + '/*', destination.path)
logger.info('--- Remove tmp_file locally - raw copy and archive')
shutil.rmtree(tmp_location.path)
logger.info('--- Remove tmp_file locally - raw copy and archive2')
shutil.rmtree(zip_tmp_loc.path)
logger.info('--- Remove tmp_file locally - raw copy and archive3')
When I run the script manually, using the "vbackup" user, it works. I defined a task in my "user" crontab with this syntax (i am executing crontab -e as "vbackup" by using "su vbackup":
00 22 * * * python3.2 /opt/valibackup/main.py
When I use the above, the script runs every day at 22:00. The problem is that it seems to run without the needed privileges and the shutil.rmtree() doesn't work, when it does when I run the script manually.
I tried the following syntax that I found here to be sure that it was ran with "vbackup" rights, but it doesn't even start.
*/30 * * * * vbackup python3.2 /opt/valibackup/main.py
If I edit by using "sudo crontab -e" instead, the rmtree works, but not the rsync sends a Permission denied error.
Any idea?
It sounds like you need to be a privileged user to remove the folders and need to run rsync as you local user.
Modify your script to work as below and try.
Just switch to sudo user before the delete operation and then switch back to your normal user.
I think this answer in stackoverflow will help you
switching user in python
Related
I would like to use a cron task in order to delete media files if the condition is True.
Users generate export files stored in the Media folder. In order to clean export files in the background, I have a Cron task which loops over each file and looks if the expiry delay is passed or not.
I used django-cron library
Example:
File in Media Folder : Final_Products___2019-04-01_17:50:43.487845.xlsx
My Cron task looks like this :
class MyCronExportJob(CronJobBase):
""" Cron job which removes expired files at 18:30 """
RUN_AT_TIMES = ['18:30']
schedule = Schedule(run_at_times=RUN_AT_TIMES)
code = 'app.export_cron_job'
def do(self):
now = datetime.datetime.now()
media_folder = os.listdir(os.path.join(settings.MEDIA_ROOT, 'exports'))
for files in media_folder:
file = os.path.splitext(files.split(settings.EXPORT_TITLE_SEPARATOR, 1)[1])[0]
if datetime.datetime.strptime(file, '%Y-%m-%d_%H:%M:%S.%f') + timedelta(minutes=settings.EXPORT_TOKEN_DELAY) < now:
os.remove(os.path.join(os.path.join(settings.MEDIA_ROOT, 'exports'), files))
# settings.EXPORT_TOKEN_DELAY = 60 * 24
I edited my crontab -e :
30 18 * * * source /home/user/Bureau/Projets/app/venv/bin/activate.csh && python /home/user/Bureau/Projets/app/src/manage.py runcrons --force app.cron.MyCronExportJob
Then I launched service cron restart
But nothing as changed. My file is still there. However, it should be removed because his date is greater than now + settings.EXPORT_TOKEN_DELAY
I'm using Ubuntu to local dev and FreeBSD as a production server environment.
EDIT:
I tried some things but crontab doesn't work for the moment.
1) * * * * * /bin/date >> /home/user/Bureau/Projets/app/cron_output
==> It works, so crontab works
2) I ran : python manage.py runcrons in my console
==> It works
3) I ran this script (cron.sh):
source /home/user/.bashrc
cd /home/user/Bureau/Projets/app
pyenv activate app
python src/manage.py runcrons --force
deactivate
==> It works
4) I ran this crontab line :
35 10 * * * /home/user/Bureau/Projets/app/utility/cron.sh
==> Service restarted at 10h32, I waited until 10h38 : nothing !
When I run this os.walk code from my PyScripter IDE, it works just fine. The os.walk was able to traverse the remote unix path and print the directory and file names.
However, when I run this same script from the cgi-bin of my Apache server, I get no os.walk (path, dirs, files) output. I also don't get any error messages recorded in the Apache error.log file.
Note: Python and my cgi-bin are on the same Windows machine, and the remote_path is on Unix.
Why does this same code work from the console but not from the cgi-bin and what can I do to resolve this?
#!C:\Python27\python.exe -u
import os
print "Content-type: text/html\n\n";
print "<html><head>"
print "<font size=+2><B>os.walk test</B></font><br><br>";
# Remote path
remote_path = r'\\unix_server\path\2015\q1\files\na\canada'
i = 0
for (path, dirs, files) in os.walk(remote_path):
print "Path", path
print "<BR><BR>Dirs", dirs
print "<BR><BR>Files", files
i += 1
if i >= 1:
break
Thanks to kindall, I found out the problem was indeed that Apache didn't have the appropriate permissions.
To solve the issue, I did the following (Windows 7 OS):
Opened services.msc. Start -> Run -> "services.msc"
Located my Apache service in the list "Apache2". Double click it.
In the Properties dialog for the Apache2 service, switch to the "Log On" tab. The radio button for "Local System account" was checked. I switched it to "This account" and clicked Browse.
In the "Enter object name to select" text box, I entered the name of my network + backslash + my user name. For example NETWORK\username. Clicked OK.
Then on the Log On tab, I saw that my full network email address was populated. I entered my password and password confirmation and clicked Apply.
The final steps were to stop the Apache2 service and then restart the service. Once that was done, the problem was solved and my script started working from the cgi-bin. :)
I wrote the following script, which sends an email to a specific email address, and saved it inside the .openshift/cron/minutely directory:
import smtplib
g = smtplib.SMTP('smtp.gmail.com:587')
g.ehlo()
g.starttls()
g.ehlo()
g.login('myusername','mypassword')
g.sendmail('myemail','otheremail','message')
I then pushed the script to the server.
I expected the program to run once every minute, and receive an email every minute. However, there is no evidence indicating that my code is being run. Any idea what might be causing the problem? Did I forget a step while setting up my application?
Note: I've checked that the email address and password I provided were correct, and that cron is installed.
EDIT: It seems that the problem is originating from the server:
I deleted the original contents of the file, created 'testfile.txt', and wrote this code instead:
a = open('testfile.txt','r+')
if not a.read():
a.write('Test writing')
a.close()
after waiting for the code to run and ssh-ing into the server, I changed to the directory named app-root/logs and displayed the contents of cron.log, which looked something like this:
Sat Nov 8 11:01:11 EST 2014: START minutely cron run
__________________________________________________________________________
/var/lib/openshift/545a6ac550044652510001d3/app-root/runtime/repo//.openshift/cron/minutely/test_openshift.py:
/var/lib/openshift/545a6ac550044652510001d3/app-root/runtime/repo//.openshift/cron/minutely/test_openshift.py: line 1: syntax error near unexpected token `('
/var/lib/openshift/545a6ac550044652510001d3/app-root/runtime/repo//.openshift/cron/minutely/test_openshift.py: line 1: `a = open('testfile.txt','r+')'
__________________________________________________________________________
Sat Nov 8 11:01:11 EST 2014: END minutely cron run - status=0
__________________________________________________________________________
Could it be that the server is not interpreting the code in my file as python code? Any suggestions welcome.
connect to openshift console
rhc ssh app_name
Change to a directory to have permission to create script:
cd $OPENSHIFT_DATA_DIR
create test01.py script
touch test01.py
Give executing permission to test01.py
chmod +x test01.py
Edit script
nano test01.py
Add a simple code like
print("Hello")
run script:
./test01.py
Error:
./test01.py: line 1: syntax error near unexpected token `"Hello"'
./test01.py: line 1: `print("Hello")'
Now check python path
which python
Output
/var/lib/openshift/your-sesseion-id/python/virtenv/venv/bin/python
Now add a she bang to test01.py
#!/var/lib/openshift/your-sesseion-id/python/virtenv/venv/bin/python
print("Hello")
Now Execute it
./test01.py
Output:
Hello
Conclusion:
Your script should know how to run and where is python path, so add it at the first line of your script
Original Question
I've got some python scripts which have been using Amazon S3 to upload screenshots taken following Selenium tests within the script.
Now we're moving from S3 to use GitHub so I've found GitPython but can't see how you use it to actually commit to the local repo and push to the server.
My script builds a directory structure similar to \images\228M\View_Use_Case\1.png in the workspace and when uploading to S3 it was a simple process;
for root, dirs, files in os.walk(imagesPath):
for name in files:
filename = os.path.join(root, name)
k = bucket.new_key('{0}/{1}/{2}'.format(revisionNumber, images_process, name)) # returns a new key object
k.set_contents_from_filename(filename, policy='public-read') # opens local file buffers to key on S3
k.set_metadata('Content-Type', 'image/png')
Is there something similar for this or is there something as simple as a bash type git add images command in GitPython that I've completely missed?
Updated with Fabric
So I've installed Fabric on kracekumar's recommendation but I can't find docs on how to define the (GitHub) hosts.
My script is pretty simple to just try and get the upload to work;
from __future__ import with_statement
from fabric.api import *
from fabric.contrib.console import confirm
import os
def git_server():
env.hosts = ['github.com']
env.user = 'git'
env.passowrd = 'password'
def test():
process = 'View Employee'
os.chdir('\Work\BPTRTI\main\employer_toolkit')
with cd('\Work\BPTRTI\main\employer_toolkit'):
result = local('ant viewEmployee_git')
if result.failed and not confirm("Tests failed. Continue anyway?"):
abort("Aborting at user request.")
def deploy():
process = "View Employee"
os.chdir('\Documents and Settings\markw\GitTest')
with cd('\Documents and Settings\markw\GitTest'):
local('git add images')
local('git commit -m "Latest Selenium screenshots for %s"' % (process))
local('git push -u origin master')
def viewEmployee():
#test()
deploy()
It Works \o/ Hurrah.
You should look into Fabric. http://docs.fabfile.org/en/1.4.1/index.html. Automated server deployment tool. I have been using this quite some time, it works pretty fine.
Here is my one of the application which uses it, https://github.com/kracekumar/sachintweets/blob/master/fabfile.py
It looks like you can do this:
index = repo.index
index.add(['images'])
new_commit = index.commit("my commit message")
and then, assuming you have origin as the default remote:
origin = repo.remotes.origin
origin.push()
How do i get a basic web2py server up and running on
PythonAnywhere?
[update - 29/05] We now have a big button on the web tab that will do all this stuff for you. Just click where it says Web2Py, fill in your admin password, and you're good to go.
Here's the old stuff for historical interest...
I'm a PythonAnywhere developer. We're not massive web2py experts (yet?) but I've managed to get web2py up and running like this:
First download and unpack web2py:
wget http://www.web2py.com/examples/static/web2py_src.zip
unzip web2py_src.zip
Go to the PythonAnywhere "Web" panel and edit your wsgi.py. Add these lines:
import os
import sys
path = '/home/my_username/web2py'
if path not in sys.path:
sys.path.append(path)
from wsgihandler import application
replacing my_username with your username.
You will also need to comment out the last two lines in wsgi.py, where we have the default hello world web.py application...
# comment out these two lines if you want to use another framework
#app = web.application(urls, globals())
#application = app.wsgifunc()
Thanks to Juan Martinez for his instructions on this part, which you can view here:
http://web2py.pythonanywhere.com/
then open a Bash console, and cd into the main web2py folder, then run
python web2py.py --port=80
enter admin password
press ctrl-c
(this will generate the parameters_80.py config file)
then go to your Web panel on PythonAnywhere, click reload web app,
and things should work!
You can also simply run this bash script:
http://pastebin.com/zcA5A89k
admin will be disabled because of no HTTPS unless you bypass it as in the previous post. It will create a security vulnerability.
Pastebin was down, I retrieved this from the cache.
cd ~
wget -O web2py_srz.zip http://web2py.com/examples/static/web2py_src.zip
unzip web2py_src.zip
echo "
PATH = '/home/"`whoami`"/web2py'
import os
import sys
sys.stdout = sys.stderr
os.chdir(PATH)
if not './' in sys.path[:1]: sys.path.insert(0,'./')
from gluon.main import wsgibase as application
" > /var/www/wsgi.py
cd web2py
python -c "from gluon.main import save_password; save_password(raw_input('admin password: '),433)"
I have recently summarized my experience with deployment of Web2Py on PythonAnywhere here
Hope it helps
NeoToren
I'll try to add something new to the discussion. The EASIEST way I've found is to go here when you aren't logged in. This makes it so you don't have to mess around with the terminal:
https://www.pythonanywhere.com/try-web2py
Come up with a domain name, then you'll get redirected to a page showing your login information and created dashboard for that domain. From there just create an account so your app isn't erased after 24 hours. When you sign up, your app has a 3 month expiry date (if you're not paying). I believe this is a new policy. Then simply go to https://appname.pythonanywhere.com/admin and then enter the password you were given and then upload your Web2Py file into the dashboard and then visit the page.
I'm not sure how to upload a Web2Py app on PythonAnywhere for an existing account, but that's the easiest method I've found.