We Keep Coding

Python Functions: Pass parameter as string and reference?

I have a knot in my head. I wasn't even sure what to google for. (Or how to formulate my title)
I want to do the following: I want to write a function that takes a term that occurs in the name of a .csv, but at the same time I want a df to be named after it.
Like so:
def read_data_into_df(name):
df_{name} = pd.read_csv(f"file_{name}.csv")
Of course the df_{name} part is not working. But I hope you get the idea.
Is this possible without hard coding?
Thanks!

IIUC, you can use globals :
def read_data_into_df(name):
globals()[f"df_{name}"] = pd.read_csv(f"file_{name}.csv")

If I were you I would create a dictionary and create keys with
dictionary = f"df_{name}: {whatever_you_want}"

If there are only a couple of dataframes, just accept the minimal code repetition:
def read_data_into_df(name):
return pd.read_csv(f"file_{name}.csv")
...
df_ham = read_data_into_df('ham')
df_spam = read_data_into_df('spam')
df_bacon = read_data_into_df('bacon')
...
# Use df_ham, df_spam and df_bacon
If there's a lot of them, or the exact data frames are generated, I would use a dictionary to keep track of the dataframes:
dataframes = {}
def read_data_into_df(name):
return pd.read_csv(f"file_{name}.csv")
...
for name in ['ham', 'spam', 'bacon']:
dataframes[name] = read_data_into_df('name')
...
# Use dataframes['ham'], dataframes['spam'] and dataframes['bacon']
# Or iterate over dataframes.values() or dataframes.items()!

Conditionally modify multiple variables

Not quite sure what the correct title should be.
I have a function with 2 inputs def color_matching(color_old, color_new). This function should check the strings in both arguments and assign either a new string if there is a hit.
def color_matching(color_old, color_new):
if ('<color: none' in color_old):
color_old = "NoHighlightColor"
elif ('<color: none' in color_new):
color_new = "NoHighlightColor"
And so forth. The problem is that each of the arguments can be matched to 1 of 14 different categories ("NoHighlightColor" being one of them). I'm sure there is a better way to do this than repeating the if statement 28 times for each mapping but I'm drawing a blank.

You can at first parse your input arguments, if for example it's something like that:
old_color='<color: none attr:ham>'
you can parse it to get only the value of the relevant attribute you need:
_old_color=old_color.split(':')[1].split()[0]
That way _old_color='none'
Then you can use a dictionary where {'none':'NoHighlightColor'}, lets call it colors_dict
old_color=colors_dict.get(_old_color, old_color)
That way if _old_color exists as a key in the dictionary old_color will get the value of that key, otherwise, old_color will remain unchanged
So your final code should look similar to this:
def color_matching(color_old, color_new):
""" Assuming you've predefined colros_dict """
# Parsing to get both colors
_old_color=old_color.split(':')[1].split()[0]
_new_color=new_color.split(':')[1].split()[0]
# Checking if the first one is a hit
_result_color = colors_dict.get(_old_color, None)
# If it was a hit (not None) then assign it to the first argument
if _result_color:
color_old = _result_color
else:
color_new = colors_dict.get(_color_new, color_new)

You can replace conditionals with a data structure:
def match(color):
matches = {'<color: none': 'NoHighlightColor', ... }
for substring, ret in matches.iteritems():
if substring in color:
return ret
But you seems to have a problem that requires a proper parser for the format you are trying to recognize.
You might build one from simple string operations like "<color:none jaja:a>".split(':')
You could maybe hack one with a massive regex.
Or use a powerful parser generated by a library like this one

How to select values of tuples inside tuples?

I have a tuple called params which included two other tuples. The tutorial I got this code from accesses the tuples inside the tuple with self.params.printlog. However, that is not working for me. Is there anything I'm missing?
class TestStrategy():
params = (
('maperiod', 15),
('printlog', False),
)
def log(self, txt, dt=None, doprint=False):
if self.params.printlog or doprint:
dt = dt or self.datas[0].datetime.date(0)
print('%s, %s' % (dt.isoformat(), txt))
Description in the tutorial of what is done:
I thought the same, but it's not a dict and not a named tuple. It
would a bit unpractical to hardcode some of the values in the strategy
and have no chance to change them easily. Parameters come in handy to
help.
Definition of parameters is easy and looks like:
params = (('myparam', 27), ('exitbars', 5),)
Being this a standard Python tuple with some tuples inside it, the
following may look more appealling to some:
params = (
('myparam', 27),
('exitbars', 5),)

Use params[0][0] for accessing maperiod
And similarly, params[1][0] for printlog
Alternatively, you can also use a named tuple

Ok so there are three main data structures at play here.
There is a tuple, which is what you have in the code you've shown us:
params = (('maperiod', 15), ('printlog', False))
You have to use ints to access, like to get the 'printlog' value use params[1][1] and the maperiod value use params[0][1].
params[0][1] == 15
params[1][1] == False
There are dicts which is what is sounds like how you accessed the data
params = {'maperiod': 15, 'printlog': False}
Now we can access the data by key
params['maperiod'] == 15
params['printlog'] == False
Sometimes we want the best of both worlds, both a tuple and we can access by key. This sounds like what the example was using since we access with . notation. For that we use namedtuples.
from collections import namedtuple
params = namedtuple('Params', 'maperiod, printlog')(15, False)
and access by index or by attribute
params[0] == params.maperiod == 15
params[1] == params.printlog == False
What is confusing is that you've mentioned all three in different ways. I'd go back and look at the example to see which one they are using and follow that.
Edit: If what you do have is a tuple then it is very easy to convert into a dict or namedtuple for easier accessing. Just do:
dict_params = dict(params)
nt_params = namedtuple('Params', [p[0] for p in params])(*[p[1] for p in params])

How to pass in a dictionary with additional elements in python?

I have a dictionary:
big_dict = {1:"1",
2:"2",
...
1000:"1000"}
(Note: My dictionary isn't actually numbers to strings)
I am passing this dictionary into a function that calls for it. I use the dictionary often for different functions. However, on occasion I want to send in big_dict with an extra key:item pair such that the dictionary I want to send in would be equivalent to:
big_dict[1001]="1001"
But I don't want to actually add the value to the dictionary. I could make a copy of the dictionary and add it there, but I'd like to avoid the memory + CPU cycles this would consume.
The code I currently have is:
big_dict[1001]="1001"
function_that_uses_dict(big_dict)
del big_dict[1001]
While this works, it seems rather kludgy.
If this were a string I'd do:
function_that_uses_string(myString + 'what I want to add on')
Is there any equivalent way of doing this with a dictionary?

As pointed out by Veedrac in his answer, this problem has already been solved in Python 3.3+ in the form of the ChainMap class:
function_that_uses_dict(ChainMap({1001 : "1001"}, big_dict))
If you don't have Python 3.3 you should use a backport, and if for some reason you don't want to, then below you can see how to implement it by yourself :)
You can create a wrapper, similarly to this:
class DictAdditionalValueWrapper:
def __init__(self, baseDict, specialKey, specialValue):
self.baseDict = baseDict
self.specialKey = specialKey
self.specialValue = specialValue
def __getitem__(self, key):
if key == self.specialKey:
return self.specialValue
return self.baseDict[key]
# ...
You need to supply all other dict method of course, or use the UserDict as a base class, which should simplify this.
and then use it like this:
function_that_uses_dict(DictAdditionalValueWrapper(big_dict, 1001, "1001"))
This can be easily extended to a whole additional dictionary of "special" keys and values, not just single additional element.
You can also extend this approach to reach something similar as in your string example:
class AdditionalKeyValuePair:
def __init__(self, specialKey, specialValue):
self.specialKey = specialKey
self.specialValue = specialValue
def __add__(self, d):
if not isinstance(d, dict):
raise Exception("Not a dict in AdditionalKeyValuePair")
return DictAdditionalValueWrapper(d, self.specialKey, self.specialValue)
and use it like this:
function_that_uses_dict(AdditionalKeyValuePair(1001, "1001") + big_dict)

If you're on 3.3+, just use ChainMap. Otherwise use a backport.
new_dict = ChainMap({1001: "1001"}, old_dict)

You can add the extra key-value pair leaving original dictionary as such like this:
>>> def function_that_uses_bdict(big_dict):
... print big_dict[1001]
...
>>> dct = {1:'1', 2:'2'}
>>> function_that_uses_bdict(dict(dct.items()+[(1001,'1001')]))
1001
>>> dct
{1: '1', 2: '2'} # original unchanged

This is a bit annoying too, but you could just have the function take two parameters, one of them being big_dict, and another being a temporary dictionary, created just for the function (so something like fxn(big_dict, {1001,'1001'}) ). Then you could access both dictionaries without changing your first one, and without copying big_dict.

Is there a better way to create dynamic functions on the fly, without using string formatting and exec?

I have written a little program that parses log files of anywhere between a few thousand lines to a few hundred thousand lines. For this, I have a function in my code which parses every line, looks for keywords, and returns the keywords with the associated values.
These log files contain of little sections. Each section has some values I'm interested in and want to store as a dictionary.
I have simplified the sample below, but the idea is the same.
My original function looked like this, it gets called between 100 and 10000 times per run, so you can understand why I want to optimize it:
def parse_txt(f):
d = {}
for line in f:
if not line:
pass
elif 'apples' in line:
d['apples'] = True
elif 'bananas' in line:
d['bananas'] = True
elif line.startswith('End of section'):
return d
f = open('fruit.txt','r')
d = parse_txt(f)
print d
The problem I run into, is that I have a lot of conditionals in my program, because it checks for a lot of different things and stores the values for it. And when checking every line for anywhere between 0 and 30 keywords, this gets slow fast. I don't want to do that, because, not every time I run the program I'm interested in everything. I'm only ever interested in 5-6 keywords, but I'm parsing every line for 30 or so keywords.
In order to optimize it, I wrote the following by using exec on a string:
def make_func(args):
func_str = """
def parse_txt(f):
d = {}
for line in f:
if not line:
pass
"""
if 'apples' in args:
func_str += """
elif 'apples' in line:
d['apples'] = True
"""
if 'bananas' in args:
func_str += """
elif 'bananas' in line:
d['bananas'] = True
"""
func_str += """
elif line.startswith('End of section'):
return d"""
print func_str
exec(func_str)
return parse_txt
args = ['apples','bananas']
fun = make_func(args)
f = open('fruit.txt','r')
d = fun(f)
print d
This solution works great, because it speeds up the program by an order of magnitude and it is relatively simple. Depending on the arguments I put in, it will give me the first function, but without checking for all the stuff I don't need.
For example, if I give it args=['bananas'], it will not check for 'apples', which is exactly what I want to do.
This makes it much more efficient.
However, I do not like it this solution very much, because it is not very readable, difficult to change something and very error prone whenever I modify something. Besides that, it feels a little bit dirty.
I am looking for alternative or better ways to do this. I have tried using a set of functions to call on every line, and while this worked, it did not offer me the speed increase that my current solution gives me, because it adds a few function calls for every line. My current solution doesn't have this problem, because it only has to be called once at the start of the program. I have read about the security issues with exec and eval, but I do not really care about that, because I'm the only one using it.
EDIT:
I should add that, for the sake of clarity, I have greatly simplified my function. From the answers I understand that I didn't make this clear enough.
I do not check for keywords in a consistent way. Sometimes I need to check for 2 or 3 keywords in a single line, sometimes just for 1. I also do not treat the result in the same way. For example, sometimes I extract a single value from the line I'm on, sometimes I need to parse the next 5 lines.

I would try defining a list of keywords you want to look for ("keywords") and doing this:
for word in keywords:
if word in line:
d[word] = True
Or, using a list comprehension:
dict([(word,True) for word in keywords if word in line])
Unless I'm mistaken this shouldn't be much slower than your version.
No need to use eval here, in my opinion. You're right in that an eval based solution should raise a red flag most of the time.
Edit: as you have to perform a different action depending on the keyword, I would just define function handlers and then use a dictionary like this:
def keyword_handler_word1(line):
(...)
(...)
def keyword_handler_wordN(line):
(...)
keyword_handlers = { 'word1': keyword_handler_word1, (...), 'wordN': keyword_handler_wordN }
Then, in the actual processing code:
for word in keywords:
# keyword_handlers[word] is a function
keyword_handlers[word](line)

Use regular expressions. Something like the next:
>>> lookup = {'a': 'apple', 'b': 'banane'} # keyword: characters to look for
>>> pattern = '|'.join('(?P<%s>%s)' % (key, val) for key, val in lookup.items())
>>> re.search(pattern, 'apple aaa').groupdict()
{'a': 'apple', 'b': None}

def create_parser(fruits):
def parse_txt(f):
d = {}
for line in f:
if not line:
pass
elif line.startswith('End of section'):
return d
else:
for testfruit in fruits:
if testfruit in line:
d[testfruit] = True
This is what you want - create a test function dynamically.
Depending on what you really want to do, it is, of course, possibe to remove one level of complexity and define
def parse_txt(f, fruits):
[...]
or
def parse_txt(fruits, f):
[...]
and work with functools.partial.

You can use set structure, like this:
fruit = set(['cocos', 'apple', 'lime'])
need = set (['cocos', 'pineapple'])
need. intersection(fruit)
return to you 'cocos'.