I am using Config Parser to specify a list of variables, and the values for those variables are then pulled from a larger file. The variables/lines in the larger file all look like this:
callCount.1.cell=2
callCount.2.cell=10
callCount.3.cell=12
Rather than listing all these variables specifically, would I be able to use an '*' as a wildcard character, in place of the number, like this:
[variablesToPull]
callCount.*.cell
I can't change the formatting of the larger file I'm pulling values from, and I don't always know what the numbers that are apart of the variables will be.
EDIT: I'm using Python 2.7 to do all my Config Parsing
After looking around for a while I don't think it's possible. I ended up using just the first part of the desired variable name in the config file
[variablesToPull]
callCount
Then I created a dictionary from the file I was storing the full list of variable names and values in (All the following code is for Python 2.7, it probably works for Python 3, but might need some syntax changes)
f = open(variable_names_and_values_file, 'r')
dict_of_vars= {}
for line in f:
k, v = line.strip().split('=')
dict_of_vars[k.strip()] = v.strip()
f.close()
Then I looped over this dictionary and created a new list of the specific variables to pull (like callCount.1.cell, callCount.2.cell, ect...)
new_variables= []
for var in partial_variable_names_from_config_file:
for data in dict_of_vars:
if var in data:
new_variables.append(data)
Initially I didn't want to do so much looping for fear of performance decrease (my file of variables has like 10000 lines), but it doesn't seem to have slowed down my script by a noticeable amount.
Hope this helps someone out there
Related
I am writing a minor OP5 plugin in Python 2.7 (version is out of my hands) that iterates over a multidimensional list that verifies fallback zip downloads have gone as they should.
Up until now I have put each host with their IP address in a multidimensional list looking like (cut short for brevity):
fallback = [
["host1", "192.168.1.3"],
["host2", "192.168.15.59"]
]
...and so on.
This lets me iterate through fallback[i] and use that along with fallback[i][1] for the IP address, the rest of the script uses both of these informations for various tasks and string manipulations. The script as it is now is mechanically sound but relies on availability of these indexes.
There is however a hidden file (.fallbackinfo) containing the same information for another script but it is written for perl, same as the script that uses that file as a source.
The file looks like this:
#hosts = (
["host1", "192.168.1.3", "type of firmware", "subfolder"],
["host2", "192.168.15.59", "type of firmware", "subfolder"],
);
I wish to import this into an iterable multidimensional list in my Python script, but am getting incredibly stuck.
My current attempt is the closest I have gotten:
with open("/home/runninguser/.fallbackinfo") as f:
lines = []
for line in f:
lines.append(line.rstrip().strip())
fallback = lines[1:len(lines)-1]
This has successfully made the list look as I want it, but all lines get imported as str objects. I have attempted to use list() to force the object to become a list but most of the time, that makes each character in the lines to become a list object instead. The network in question is cut off from internet access so I have to rely on built-in modules. My interpretation is that since it is formatted as a list, it should somehow be able to be interpreted as a list.
Can this be done at all, and if so, how?
You can use the json package (built-in) to achieve this:
import json
with open("/home/runninguser/.fallbackinfo") as f:
# For each line
for line in f:
# If the line starts with a bracket
if line.strip()[0] == "[":
# Print the line after removing spaces in front and the comma in the back
# and converting it into a list
print(json.loads(line.strip().rstrip(",")))
If you now use the type() function, you will see the list-formatted strings are now <class 'list'>
I have to read a file that has always the same format.
As I know it has the same format I can readline() and tokenize. But I guess there is a way to read it more, how to say it, "pretty to the eyes".
The file I have to read has this format :
Nom NMS-01
MAC AAAAAAAAAAA
UDPport 2019
TCPport 9129
I just want a different way to read it without having to tokenize, if that is possbile
Your question seems to imply that "tokenizing" is some kind of mysterious and complicated process. But in fact, the thing you are trying to do is exactly tokenizing.
Here is a perfectly valid way to read the file you show, break it up into tokens, and store it in a data structure:
def read_file_data(data_file_path):
result = {}
with open(data_file_path) as data_file:
for line in data_file:
key, value = line.split(' ', maxsplit=1)
result[key] = value
return result
That wasn't complicated, it wasn't a lot of code, it doesn't need a third-party library, and it's easy to work with:
data = read_file_data('path/to/file')
print(data['Nom']) # prints "NMS-01"
Now, this implementation makes many assumptions about the structure of the file. Among other things, it assumes:
The entire file is structured as key/value pairs
Each key/value pair fits on a single line
Every line in the file is a key/value pair (no comments or blank lines)
The key cannot contain space characters
The value cannot contain newline characters
The same key does not appear multiple times in the file (or, if it does, it is acceptable for the last value given to be the only one returned)
Some of these assumptions may be false, but they are all true for the data sample you provided.
More generally: if you want to parse some kind of structured data, you need to understand the structure of the data and how values are delimited from each other. That's why common structured data formats like XML, JSON, and YAML (among many others!) were invented. Once you know the language you are parsing, tokenization is simply the code you write to match up the language with the text of your input.
Pandas does many magical things, so maybe that is prettier for you?
import pandas as pd
pd.read_csv('input.txt',sep = ' ',header=None,index_col=0)
This gives you a dataframe that you can manipulate further:
0 1
Nom NMS-01
MAC AAAAAAAAAAA
UDPport 2019
TCPport 9129
How to get/extract number of lines added and deleted?
(Just like we do using git diff --numstat).
repo_ = Repo('git-repo-path')
git_ = repo_.git
log_ = g.diff('--numstat','HEAD~1')
print(log_)
prints the entire output (lines added/deleted and file-names) as a single string. Can this output format be modified or changed so as to extract useful information?
Output format: num(added) num(deleted) file-name
For all files modified.
If I understand you correctly, you want to extract data from your log_ variable and then re-format it and print it? If that's the case, then I think the simplest way to fix it, is with a regular expression:
import re
for line in log_.split('\n'):
m = re.match(r"(\d+)\s+(\d+)\s+(.+)", line)
if m:
print("{}: rows added {}, rows deleted {}".format(m[3], m[1], m[2]))
The exact output, you can of course modify any way you want, once you have the data in a match m. Getting the hang of regular expressions may take a while but it can be very helpful for small scripts.
However, be adviced, reg exps tend to be write-only code and can be very hard to debug. However, for extracting small parts like this, it is very helpful.
I try to get all values from section in my ini file (via configparser) as a variable:
hue310section = dict(parser.items('HUE_310'))
for keys, value in hue310section.items():
pairs = keys + ' = ' + value
print(pairs)
it gave me partnewfilepath = http://some_site:PORT/about, but I don't know how to import this output as an python variable, that I can use partnewfilepath somewhere in my code. Of course one section will have more values than only one, and I want to change all that in variable. I trying to find solution but I think I miss something because my knowledge about python is not enough yet. I think I need to rebuilt my for statement but don't have a clue how to do it in this particular problem.
My config.ini file looks like:
[HUE_310]
partNewFilePath = ${common:domain}/about
otherValues = something
nextvalue = another something
UPDATE:
I think I need to elaborate more about what I want to achieve. In other part of my code I check version of site I want to process. If the site has, let say version 3.10 I want to get all values from section HUE_310 from my ini file, and use them as python variable. Rest of my code use those variable and if the site version will change I can get values from other section from my ini file and get those values to python variable and use them. I assume that some variables will change from version to version and that's why I want to prepare my code to check this. Also it gives me some freedom to modify some variable if site will change.
I hope it is now more clear.
You don't need a new variable or a for loop, you already have hue310section dict.
You can just use
hue310section['partNewFilePath']
which will be equal to
"http://some_site:PORT/about"
Note that after hue310section = dict(parser.items('HUE_310'))
, otherValues and nextvalue keys will also be defined.
from configobj import ConfigObj
parser_data = ConfigObj(config_path)
current = parser_data['HUE_310'].get('partNewFilePath', 'http://www.default.com')
config_path is path to the file
http://www.default.com is the default value in case that particular key is not found.
I'm trying to set up an environment variable via Python:
os.environ["myRoot"]="/home/myName"
os.environ["subDir"]="$myRoot/subDir"
I expect the subDir environment variable to hold /home/myname/subDir, however it holds the string '$myRoot/subDir'. How do I get this functionality?
(Bigger picture : I'm reading a json file of environment variables and the ones lower down reference the ones higher up)
Use os.environ to fetch the value, and os.path to correctly put slashes in the right places:
os.environ["myRoot"]="/home/myName"
os.environ["subDir"] = os.path.join(os.environ['myRoot'], "subDir")
You can use os.path.expandvars to expand environment variables like so:
>>> import os
>>> print os.path.expandvars("My home directory is $HOME")
My home director is /home/Majaha
>>>
For your example, you might do:
os.environ["myRoot"] = "/home/myName"
os.environ["subDir"] = os.path.expandvars("$myRoot/subDir")
I think #johntellsall's answer is the better for the specific example you gave, however I don't doubt you'll find this useful for your json work.
Edit: I would now recommend using #johntellsall's answer, as os.path.expandvars() is designed explicitly for use with paths, so using it for arbitrary strings may work but is kinda hacky.
def fix_text(txt,data):
'''txt is the string to fix, data is the dictionary with the variable names/values'''
def fixer(m): #takes a regex match
match = m.groups()[0] #since theres only one thats all we worry about
#return a replacement or the variable name if its not in the dictionary
return data.get(match,"$%s"%match)
return re.sub("$([a-zA-Z]+)",fixer,txt) #regular expression to match a "$" followed by 1 or more letters
with open("some.json") as f: #open the json file to read
file_text= f.read()
data = json.loads(file_text) #load it into a json object
#try to ensure you evaluate them in the order you found them
keys = sorted(data.keys() ,key=file_text.index)
#create a new dictionary by mapping our ordered keys above to "fixed" strings that support simple variables
data2= dict(map(lambda k:(k,fixer(data[k],data)),keys)
#sanity check
print data2
[edited to fix a typo that would cause it not to work]