I am using the following code to call two different scripts:
soup=BeautifulSoup(htmltext)
title=soup.find('title')
os.system("python scraper/updatedb.py %s" % (title))
os.system("python scraper/insertlinks.py %s" % (tag['href']))
The second one is running but the first one is not running. Please help.
updatedb.py and insertlinks.py work fine when run individually.
updatedb.py is as follows:
import sqlite3 as db
import sys
print "inserted"
key=sys.argv[1]
id="1"
conn=db.connect('store.db')
cursor=conn.cursor()
with conn:
cursor.execute('insert into records (id,keyword) values(?,?)',(id,key))
conn.close()
Os.system always returns the code, error code if not executed the command and 0 if it is executed successfully.
result_code = os.system("your command")
you can check the error message on google by using python error code
Related
I'am trying to write a script that is waiting for a database to be online.
The script that checks the database connection is written in python and should be excited every 5 seconds. If the script returns 0 the main script should continue.
I have never written in shell, so i can only guess how to get the python script into the condition of the while loop. Here is my attempt, but i have no idea how to get it right.
The python script is working well.
psqltest.py:
#!/usr/bin/python2.4
#
import psycopg2
try:
db = psycopg2.connect("dbname='postgis' user='postgres' host='db' password='postgres'")
except:
exit(1)
exit(0)
my main shell script:
echo waiting for database...
while [ python /root/psqltest.py && echo 0 || echo 1 ]
do
sleep 5
done
the error message:
1 ]
Thanks for helping me out
The shell script would simply be
while ! python /root/psqltest.py; do
sleep 5
done
or the lesser known
until python /root/psqltest.py; do
sleep 5
done
I have improved my Scripts, now the bash is only calling the python script and waits for it to finish, as Kristof mentioned. This is working so far, but i try to improve it further more.
#!/usr/bin/python3
#
import psycopg2
import time
import sys
db = None
sys.stdout.write('waiting ')
while db is None:
try:
db = psycopg2.connect("dbname='postgis' user='postgres' host='db' password='...'")
db.close()
except:
sys.stdout.write('.')
sys.stdout.flush()
time.sleep( 5 )
print('')
exit(0)
I am trying to import data into MongoDB through Python envoy. However it keeps showing "permission denied". After I have changed the permission of the data file, it keeps showing "Exec format error".
However, when I ran the same command on MongoDB Shell, everything works well.
Do you know how can I run the commands through Python envoy?
Below is my code:
def load_data():
data_file = os.path.join(os.getcwd(), 'enron.mbox.json')
print data_file
r = envoy.run('mongoimport --db enron --collection mbox --drop --file %s' % data_file)
print r.std_out
print sys.stderr.write(r.std_err)
I would like to execute the MySQL query source FileName.sql in a Python script on Linux.
I am able to execute other queries like SELECT * FROM table_name but this one is giving an error. I am executing this on a Linux server with a MySQL database using Python. The frontend I am using is Putty.
The Python script I have used is:
import MySQLdb
db = MySQLdb.connect("hostname","username","pswrd","dbname")
cursor = db.cursor()
cursor.execute("source FileName.sql")
db.close()
How can I execute the query source FileName.sql on the location where this file-> FileName.sql is located?
source is not a SQL command. It's a MySQL CLI command, it only exists in the console application mysql (and wherever else implemented). All it does is to read the contents of FileName.sql and issue the SQL commands inside.
To do this in python, you can use something like
Edit: This assumes you have 1 query per line! If you have multi-line queries, you'll have to find the means to extract each query from the file.
import MySQLdb
db = MySQLdb.connect("hostname","user","pass","db")
cursor = db.cursor()
for line in open("FileName.sql"):
cursor.execute(line)
db.close()
You can execute a bash command with Python and import your SQL file.
This exemple is for MySQL
import subprocess
command = "mysql -u username --password=p#55W0rD database_name < file.sql".split()
p = subprocess.Popen(command, stdout=subprocess.PIPE)
p.communicate() # you can see if errors are returned
if your SQL file creates a database, remove database_name.
sources:
https://docs.python.org/3/library/subprocess.html#popen-constructor
https://dev.mysql.com/doc/refman/8.0/en/mysql-batch-commands.html
Separate the scripts in SQL file in python using ";" as delimiter
Execute each command iteratively.
awesome5team had developed a nice solution in https://github.com/awesome5team/General-Resources-Box/issues/7
Code snippet from the same:
import mysql.connector
cnx = mysql.connector.connect(user='root',
password='YOUR-PASSWORD-FOR-MYSQL',
host='localhost',
database='YOUR-DATABASE-NAME')
cursor =cnx.cursor()
def executeScriptsFromFile(filename):
fd = open(filename, 'r')
sqlFile = fd.read()
fd.close()
sqlCommands = sqlFile.split(';')
for command in sqlCommands:
try:
if command.strip() != '':
cursor.execute(command)
except IOError, msg:
print "Command skipped: ", msg
executeScriptsFromFile('SQL-FILE-LOCATION')
cnx.commit()
The code is running and inserting but I get errors in command prompt saying 'tab' is not recognised as an internal or external command,operable program or batch file.
What is the mistake i have done and How can i fix it ?
Here is the python code :
updatedb.py
import sqlite3 as db
import urllib
import re
import sys
url=sys.argv[1]
htmltext=urllib.urlopen(url).read()
regex='<title>(.+?)</title>'
pattern=re.compile(regex)
title= re.findall(pattern,htmltext)
print title[0]
id="1"
conn=db.connect('insertlinks.db')
cursor=conn.cursor()
with conn:
cursor.execute('insert into records (id,keyword) values(?,?)',(id,title[0]))
#print "inserted"
#conn.close()
The above code is called as follows:
import urlparse
import os
import urllib
from bs4 import BeautifulSoup
url="http://www.google.com"
urls=[url]
visited=[url]
try:
while len(urls)>0:
htmltext=urllib.urlopen(urls[0]).read()
soup=BeautifulSoup(htmltext)
urls.pop(0)
for tag in soup.findAll('a',href=True):
tag['href']=urlparse.urljoin(url,tag['href'])
if tag['href'] not in urls and tag['href'] not in visited:
os.system("python scraper/insertlinks.py %s" % (tag['href']))
os.system("python scraper/updatedb.py %s" % (tag['href']))
urls.append(tag['href'])
visited.append(tag['href'])
except:
print 'error in 1'
EDIT:
The problem is in tag['href']. Its value is http://maps.google.co.in/maps?hl=en&tab=il. The tab in the url is creating a problem. How do i solve it?
Use the subprocess.call() method instead of os.system()
The & in the url is what is causing the problem.
On Windows:
Command1 & Command2
Means run Command1 then run Command2
The error you are getting is a Windows error, not a Python error. Somehow one or both of your os.system calls are passing "tab" as a command to the Windows command line.
I suspect this is because many of the urls on the google.com page have ?tab=Wx or &tab=wT or other similar arguments tacked on to the url. The ? shouldn't hurt anything, but the & may be interpreted as the start of another command. (If that is the case, I would expect you to receive errors about a lot more than just 'tab' though.)
I am having trouble trying to get this script to work. When I debug this code it will not read into the class or functions. The code will not execute properly. Has anyone know the problem here, Thanks
#!/home/build/test/Python-2.6.4
import os, subprocess
class mks_function:
sandbox="new_sandbox"
def mks_create_sandbox():
try:
retcode=call("si createsandbox" + "--no --hostname=bel --port=70 --user=user --password=1234 --populate --project=e:/project.pj --lineTerminator=lf new_sandbox", shell=True)
if retcode < 0:
print >>sys.stderr, "Child was terminated by signal", -retcode
else:
print >>sys.stderr, "Child returned", retcode
except OSError, e:
print >>sys.stderr, "Execution failed:", e
print "sandbox retVal="+retcode
print "Creating a new sandbox called "+sandbox+" "
###############################################################
Few things to check your code
call should be subprocess.call
better use full path when you call for example, /usr/bin/si createsandbox, you can check with which si in shell
instead of concatenating the commands "si createsandbox" + "--no ...", please use list ["/usr/bin/si","createsandbox --no ..."]
you didn't import sys, but using it
sandbox should be self.sandbox and def mks_create_sandbox(): should be def mks_create_sandbox(self):
Use an IDE for example Ulipad.
Try put as the first line:
#!/usr/bin/env python
If you really need specific version of Python, setup your environment before running.
Possible problems:
your code is never executed (it's like you define the class only). Use it in the file (names are misleading):
if __name__ == '__main__':
myObject = mks_function()
show us how are you executing the code? Have you changed the permissions to be able to run the script?
chmod +x filename.py
or are you trying to start it as:
python filename.py