Im trying to use a dictionary to check a given number of servers listed for a particular SQL backup success or fail. My problem so far is when I run this code:
for serverChk in srvrDict['Server']:
it returns the server name as single characters on each new line like:
S
E
R
V
E
R
So in my trial I see this "Error connecting to T to check OS version" where T is the fist character of the servername. I can't seem to put my finger on it and all the searching I've done has lead me to asking.
Thanks!
class checkstatus:
#def getServers(self):
chkbkpstats = csv.reader(file('c://temp//networkerservers.csv'))
for row in chkbkpstats:
srvrDict = {}
srvrDict['Server'] = row[0]
srvrDict['Instance'] = row[1]
print srvrDict
for serverChk in srvrDict['Server']:
try:
c = wmi.WMI(server)
for os in c.Win32_OperatingSystem():
osVer = os.caption
except:
print 'Error connecting to %s to check OS version' % serverChk
if '2003' in osVer:
print 'w2k3'
if '2008' in osVer:
print 'w2k8'
I suppose you have stored a string in your dictionary. So the line for serverChk in srvrDict['Server'] translates to for serverChk in yourSavedString. This is why you are getting individual characters. To access individual dictionary items you should do for k,v in srvrDict.iteritems() where k is the key and v is the value.
You are overwriting the Server and Instance values in srvrDict each iteration of your loop through chkbkpstats, not actually generating a sequence of data with an entry for each item in your log file as it appears you expect. You need to make that a list containing dictionaries, which you append to each iteration. You are probably looking for something more like:
chkbkpstats = csv.reader(file('c://temp//networkerservers.csv'))
srvrs = []
for for row in chkbkpstats:
srvrs.append({'Name' : row[0], 'Instance' : row[1]})
for srvr in srvrs:
try:
c = wmi.WMI(srvr['Instance'])
except:
print 'Error connecting to %s to check OS version' % srvr['Name']
else:
osVer = c.Win32_OperatingSystem()[0].Caption
if '2003' in osVer:
print 'w2k3'
elif '2008' in osVer:
print 'w2k8'
There are a few problems with your code.
First, you create a new srvrDict each time you go through the first for loop, overwriting the value that was stored in this variable the last time. I think, what you actually intended to do is the following:
srvrDict = {}
for row in chkbkpstats:
srvrDict[row[0]] = row[1]
Now, srvrDict will contain an entry like {'P1RT04': ['THP06ASU']} for each row in chkbkpstats, mapping server names to lists of instances running on that server.
Then, in the second loop, use for serverChk in srvrDict: to iterate over all the entries in the dictionary. However, I'm not sure where the variable server in c = wmi.WMI(server) comes from. If this is what has been row[1] in the first loop, then you should use srcvDict[serverChk] to retrieve the value from the dictionary.
This way, the whole procedure would look something like this:
chkbkpstats = csv.reader(file('c://temp//networkerservers.csv'))
srvrDict = {}
for row in chkbkpstats:
name, instance = row[0], row[1]
if name not in srvrDict:
srvrDict[name] = []
srvrDict[name].append(instance)
for server in srvrDict:
for instance in srvrDict[server]:
try:
c = wmi.WMI(instance)
except:
print 'Error connecting to %s to check OS version' % server
else:
osVer = c.Win32_OperatingSystem()[0].caption
if '2003' in osVer:
print 'w2k3'
elif '2008' in osVer:
print 'w2k8'
else:
print 'unknown OS'
PS.: I'm not sure what's the return value of c.Win32_OperatingSystem(). [...] Update: Thanks to sr2222 for pointing this out. Code fixed.
Update: Edited the code to allow for one server hosting multiple instances.
Related
I have this yaml:
global:
environment:
DURATION: 3599
BLA: 1234
dev1:
BASEPATH: bla.co.uk
LOGGING: ERROR
dev2:
BASEPATH: bla.co.uk
LOGGING: ERROR
Using the below will retrieve the dict for dev1.
import yaml
if __name__ == '__main__':
stream = open("devtest.yaml", 'r')
dictionary = yaml.load(stream)
for key, value in dictionary["global"]["environment"]["dev1"].items():
print (key + " : " + str(value))
I would like to get the environment: key + values too on the way. Not just dev1.
I can do it separately but then everything gets included under environment such as dev1 and dev2.
I always want environment and either dev1 or dev2 upon choice.
Thanks
You can go through the environment and check if the values are dictionaries or just simple values. Then you can only print the dictionary ones if they are the dev you want. This will, however, not work properly if there are other dict values that you would like to print.
def print_values(dictionary, dev: str):
# dev is either "dev1" or "dev2"
for key, value in dictionary["global"]["environment"].items():
if not isinstance(value, dict):
# if the value is not a dict, print it
print (key + " : " + str(value))
continue
# if it is a dict, check if it is the dev you want,
# print values inside if it is
if key == dev:
for dev_key, dev_value in value.items():
print (dev_key + " : " + str(dev_value))
You could write a method that accepts an environment of your choice and retrieves the properties from the yaml file. Then you could print the properties while also printing out the environment name.
Also, it is better to open files using with so that you don't need to remember to close them.
import yaml
def retrieve_env_properties(env_name):
with open("devtest.yaml", 'r') as stream:
dictionary = yaml.load(stream, Loader=yaml.Loader)
return dictionary["global"]["environment"][env_name]
def print_env_properties(dictionary, env_choice):
for key, value in dictionary.items():
print (f"{env_choice}: {key} : {str(value)}")
if __name__ == '__main__':
env_choice = "dev1"
dictionary = retrieve_env_properties(env_choice)
print_env_properties(dictionary, env_choice)
I´m trying to save a SQLAlchemy query result on a python variable (in case there is one result) in order to insert it with other data later. Is there any chance to do something like that?
Things you may want to know:
I'm trying to connect to an Oracle DB
I'm using python 3.7.1
Till now I´ve been able to do this:**
try:
map_projn_id_search = Session.execute("""
SELECT GEOFRAME_MAP_PROJN_ID
FROM V_GEOFRAME_MAP_PROJN
WHERE COORD_REF_SYS_CODE = :crs AND TRNSFRM = :tfm"""
, {"crs" : crs , "tfm" : tfm} )
map_projn_id_search.fetchone()
except Exception as e:
print("error")
else:
#If it doesn´t exist I need to insert it into the V_GEOFRAME_MAP_PROJN
if map_projn_id_search.rowcount == 0:
instances_with_unkown = ['SAMSDB', 'CASSDB', 'CCUSDB', 'EMASDB', 'EMLSDB', 'HNOSDB', 'KULSDB', 'NAFSDB', 'RUSSDB', 'YUZSDB', 'AMESDB', 'LAGSDB']
instances_with_UNKNWON =[ 'USASDB']
if emsdb_instance in instances_with_unkown:
projn_id = "Unknown"
print(f'\n The EPSG code (ProjCRS:GeogCRS:Tfm) does not exist on EMSDB. Set {projn_id} as CRS. \n')
elif emsdb_instance in instances_with_UNKNWON:
projn_id = "UNKNOWN"
print(f'\n The EPSG code (ProjCRS:GeogCRS:Tfm) does not exist on EMSDB. Set {projn_id} as CRS. \n')
else:
#If it exists then print that it exists so the user knows
print("\n Cannot insert the EPSG code (ProjCRS:GeogCRS:Tfm) on that database. \n")
elif map_projn_id_search.rowcount != 0:
#If it exists then print that it exists so the user knows
projn_id = map_projn_id_search.fetchone()
print(f"\n Set {projn_id} as the CRS of the survey. \n")
So, what I needed to do is:
Add a variable like
a = map_projn_id_search.fetchone()
And then make another who could fetch the first result as a string instead of a list
b = a[0]
The fields that I have in Mongoldb are;
id, website_url, status.
I need to find the website_url and update its status to 3 and add a new field called err_desc.
I have a list of website_urls, its status and its err_desc.
Below is my code.
client = MongoClient('localhost', 9000)
db1 = client['Company_Website_Crawl']
collection1 = db1['All']
posts1 = collection1.posts
bulk = posts1.initialize_ordered_bulk_op()
website_url = ["http://www.example.com","http://example2.com/"]
err_desc = ["error1","error2"]
for i in website_url:
parsed_uri = urlparse(i)
domain = '{uri.scheme}://{uri.netloc}/'.format(uri=parsed_uri)
final_url = domain
final_url_strip = domain.rstrip("/")
print i,final_url,final_url_strip,"\n"
try:
k = bulk.find({'website_url':i}).upsert().update({'$push':{'err_desc':err_desc,'status':3}})
k = bulk.execute()
print k
except Exception as e:
print "fail"
print e
Error
fail batch op errors occurred
fail Bulk operations can only be executed once.
Initially I used
k = posts1.update({'website_url':final_url_strip},{'$set':{'err_desc':err_desc,'status':3}},multi=True)
It was too slow for 5M records. So I wanted to use bulk update option. Kindly help me to use bulk upsert for this scenario.
The error message is telling you that you need to re-initialize the batch writes operation after calling execute(). But the thing is, you are doing it wrong. In you case, you need to call execute at the end of the for loop like this:
from itertools import count
ct = count()
for url in website_url:
...
try:
bulk.find({'website_url':i}).upsert().update({'$push':{'err_desc':err_desc,'status':3}})
val = next(ct)
except Exception as e:
...
if val > 0:
bulk.execute()
Also note that Bulk() is now deprecated and replaced with bulkwrite
I am using ConfigObj and Validator to parse a configuration file in python. While I like this tool a lot, I am having trouble with validation using a configSpec file. I am using the option() configSpec type that forces the value to be chosen from a controlled vocabulary:
output_mode = option("Verbose", "Terse", "Silent")
I want my code to know when the user enters an option that's not in the CV. From what I have fond, Validator only seems to say which config key failed validation, but not why it failed:
from configobj import ConfigObj, flatten_errors
from validate import Validator
config = ConfigObj('config.ini', configspec='configspec.ini')
validator = Validator()
results = config.validate(validator)
if results != True:
for (section_list, key, _) in flatten_errors(config, results):
if key is not None:
print 'The "%s" key in the section "%s" failed validation' % (key, ', '.join(section_list))
else:
print 'The following section was missing:%s ' % ', '.join(section_list)
That code snippet works but there are any number of reasons why a key might have failed validation, from not being in an integer range to not being in the CV. I don't want to have to interrogate the key name and raise a different kind of exception depending on the failure cases for that key. Is there a cleaner way to handle specific types of validation errors?
Long time stackoverflow reader, first time poster :-)
Update: I think this does what I want to do. The key is that config obj stores errors as Exceptions which can then be checked against those that subclass ValidateError. Then you just have to do one check per subclass rather than one check per parameter value. It might be nicer if validate just threw an exception if validation failed but maybe you would lose other functionality.
self.config = configobj.ConfigObj(configFile, configspec=self.getConfigSpecFile())
validator = Validator()
results = self.config.validate(validator, preserve_errors=True)
for entry in flatten_errors(self.config, results):
[sectionList, key, error] = entry
if error == False:
msg = "The parameter %s was not in the config file\n" % key
msg += "Please check to make sure this parameter is present and there are no mis-spellings."
raise ConfigException(msg)
if key is not None:
if isinstance(error, VdtValueError):
optionString = self.config.configspec[key]
msg = "The parameter %s was set to %s which is not one of the allowed values\n" % (key, self.config[key])
msg += "Please set the value to be in %s" % optionString
raise ConfigException(msg)
OptionString is just a string of the form option("option 1", "option 2") rather than a list so to get this to look nice, you need to grab the substring in the ()'s.
For future reference for anyone interested, you could also check for extraneous data. This can be handled with the get_extra_values function. The complete example shown below hence does:
load the configuration with validator
look for all the validated errors
verify extra values
from configobj import ConfigObj, ConfigObjError, flatten_errors, get_extra_values
from validate import Validator, VdtValueError
def load_config(configfile, configspec, raise_exception=True):
"Load and check configvale acccording to spec"
config = ConfigObj(configfile, file_error=True, configspec=configspec)
validator = Validator()
results = config.validate(validator, preserve_errors=True)
msg = ""
fatalerr = False
for entry in flatten_errors(config, results):
[sectionList, key, error] = entry
if error is False:
msg += f"\n{key:>30s} missing in section [{']['.join(sectionList)}]"
fatalerr = True
if key is not None:
if isinstance(error, VdtValueError):
optionString = config.configspec[key]
msg += f"\nThe parameter {key} was set to {[config[s][key] for s in sectionList]} which is not one of the allowed values\n"
msg += " Please set the value to be in %s" % optionString
fatalerr = True
# verifying extra values below
wmsg = ""
for sections, name in get_extra_values(config):
# this code gets the extra values themselves
the_section = config
for section in sections:
the_section = the_section[section]
# the_value may be a section or a value
the_value = the_section[name]
section_or_value = 'value'
if isinstance(the_value, dict):
# Sections are subclasses of dict
section_or_value = 'section'
section_string = '[' + (']['.join(sections) or "TOP LEVEL") + ']'
wmsg += f"\n{name:>30s}: Extra {section_or_value} on section {section_string}"
if wmsg != "":
print(f"\nWARNINGS found in configuration file {configfile}")
print(wmsg)
if fatalerr:
print(f"\nERRORS found in configuration file {configfile}")
if raise_exception:
raise RuntimeError(msg)
else:
print("Fatal errors found, but no exception raised, as requested")
print(msg)
print(f'Configuration {configfile} validated successfully')
return config
if __name__ == "__main__":
configfile="xt_default.cfg"
configspec="xt_default_spec.cfg"
config = load_config(configfile, configspec)
Example - I have the following dictionary...
URLDict = {'OTX2':'http://lsdb.hgu.mrc.ac.uk/variants.php?select_db=OTX2&action=view_all',
'RAB3GAP':'http://lsdb.hgu.mrc.ac.uk/variants.php?select_db=RAB3GAP1&action=view_all',
'SOX2':'http://lsdb.hgu.mrc.ac.uk/variants.php?select_db=SOX2&action=view_all',
'STRA6':'http://lsdb.hgu.mrc.ac.uk/variants.php?select_db=STRA6&action=view_all',
'MLYCD':'http://lsdb.hgu.mrc.ac.uk/variants.php?select_db=MLYCD&action=view_all'}
I would like to use urllib to call each url in a for loop, how can this be done?
I have successfully done this with with the urls in a list format like this...
OTX2 = 'http://lsdb.hgu.mrc.ac.uk/variants.php?select_db=OTX2&action=view_all'
RAB3GAP = 'http://lsdb.hgu.mrc.ac.uk/variants.php?select_db=RAB3GAP1&action=view_all'
SOX2 = 'http://lsdb.hgu.mrc.ac.uk/variants.php?select_db=SOX2&action=view_all'
STRA6 = 'http://lsdb.hgu.mrc.ac.uk/variants.php?select_db=STRA6&action=view_all'
MLYCD = 'http://lsdb.hgu.mrc.ac.uk/variants.php?select_db=MLYCD&action=view_all'
URLList = [OTX2,RAB3GAP,SOX2,STRA6,PAX6,MLYCD]
for URL in URLList:
sourcepage = urllib.urlopen(URL)
sourcetext = sourcepage.read()
but I want to also be able to print the key later when returning data. Using a list format the key would be a variable and thus not able to access it for printing, I would lonly be able to print the value.
Thanks for any help.
Tom
Have you tried (as a simple example):
for key, value in URLDict.iteritems():
print key, value
Doesn't look like a dictionary is even necessary.
dbs = ['OTX2', 'RAB3GAP', 'SOX2', 'STRA6', 'PAX6', 'MLYCD']
urlbase = 'http://lsdb.hgu.mrc.ac.uk/variants.php?select_db=%s&action=view_all'
for db in dbs:
sourcepage = urllib.urlopen(urlbase % db)
sourcetext = sourcepage.read()
I would go about it like this:
for url_key in URLDict:
URL = URLDict[url_key]
sourcepage = urllib.urlopen(URL)
sourcetext = sourcepage.read()
The url is obviously URLDict[url_key] and you can retain the key value within the name url_key. For exemple:
print url_key
On the first iteration will printOTX2.