In dm-haiku, parameters of neural networks are defined in dictionaries where keys are module (and submodule) names. If you would like to traverse through the values, there are multiple ways of doing so as shown in this dm-haiku issue. However, the dictionary doesn't respect the ordering of the modules and makes it hard to parse submodules. For example, if I have 2 linear layers, each followed by a mlp layer, then using hk.data_structures.traverse(params) will (roughly) return:
['linear', 'linear_2', 'mlp/~/1', 'mlp/~/2'].
whereas I would like it to return:
['linear', 'mlp/~/1', 'linear_2', 'mlp/~/2'].
My reason for wanting this form is if creating an invertible neural network and wanting to reverse the order the params are called, isolating substituent parts for other purposes (e.g. transfer learning), or, in general, wanting more control of how and where to (re)use trained parameters.
To deal with this, I've resorted to regex the names and put them in the order that I want, then using hk.data_structures.filter(predicate, params) to filter by the sorted module names. Although, this is quite tedious if I have to remake a regex every time I want to do this.
I'm wondering if there is a way to convert a dm-haiku dictionary of params to something like a pytree with a hierarchy and ordering that makes this easier? I believe equinox handles parameters in this manner (and I'm going to look more into how that is done soon), but wanted to check to see if I'm overlooking a simple method to allow grouping, reversing, and other permutations of the params's dictionary?
According to source code https://github.com/deepmind/dm-haiku/blob/main/haiku/_src/filtering.py#L42#L46 haiku use the sorted function of dict (haiku parameters are vanilla dict since 0.0.6) for hk.data_structures.traverse. Therefore you can't get the result you want without modifying the function itself. By the way, I don't get precisely what do you mean by "to reverse the order the params are called". All parameters are passed together in input and then the only thing that determines the order of use is the architecture of the function itself so you should manually invert the forward pass but you don't need to change something in params.
Related
I need to create a sort of similarity matrix based on user_id values. I am currently using Pandas to store the majority of my data, but I know that iteration is very anti-pattern, so I am considering creating a set/dictionary nest to store the similarities, similar to some of the proposed structures here
I would only be storing N nearest similarities, so it would amount to something like this:
{
'user_1' : {'user_2':0.5, 'user_4':0.9, 'user_3':1.0},
'user_2' : ...
}
It would be allowing me to access a neighbourhood by doing dict_name[user_id] quite easily.
Essentially the outermost dictionary key would hold a user_id which returns another dictionary of its N closest neighbours with user_id- similarity_value key-value sets.
For more context, I'm just writing a simple KNN recommender. I am doing it from scratch as I've tried using Surpriselib and sklearn but they don't have the context-aware flexibility I require.
This seems like a reasonable way to store these values to me, but is it very anti-pythonic, or should I be looking to do this using some other structures (e.g. NumPy or Pandas or something else I don't yet know about)?
As the comment says, there is nothing inherently wrong or anti-pythonic with using (one level of) nested dictionaries and writing everything from scratch.
Performance-wise you can probably beat your self-written solution if you use an existing data structure whose API works well with the transformations/operations you intend to perform on them. Numpy/Pandas only will help if your operations can be expressed as vectorized operations that operate on all (pairs of) elements along a common axis, e.g. all users in your top-level dictionary.
Some methods don't need to make a new variable, i.e. lists.reverse() works like this:
lists = [123, 456, 789]
lists.reverse()
print(lists)
this method make itself reversed (without new variable).
Why there is vary ways to manufacture variable in Python?
Some cases which is like variable.method().method2().method3() are typed continuously but type(variable) and print() are not. Why we can't typing like variable.print() or variable.type()?
Is there any philosophical reasons for Python?
You may be confused by the difference between a function and a method, and by three different purposes to them. As much as I dislike using SO for tutorial purposes, these issues can be hard to grasp from other documentation. You can look up function vs method easily enough -- once you know it's a (slightly) separate issue.
Your first question is a matter of system design. Python merely facilitates what programmers want to do, and the differentiation is common to many (most?) programming languages since ASM and FORTRAN crawled out of the binary slime pools in the days when dinosaurs roamed the earth.
When you design how your application works, you need to make a lot of implementation decisions: individual variables vs a sequence, in-line coding vs functions, separate functions vs encased functions vs classes and methods, etc. Part of this decision making is what each function should do. You've raised three main types:
(1) Process this data -- take the given data and change it, rearrange it, whatever needs doing -- but I don't need the previous version, just the improved version, so just put the new stuff where the old stuff was. This is used almost exclusively when one variable is getting processed; we don't generally take four separate variables and change each of them. In that case, we'd put them all in a list and change the list (a single variable). reverse falls into this class.
One important note is that for such a function, the argument in question must be mutable (capable of change). Python has mutable and immutable types. For instance, a list is mutable; a tuple is immutable. If you wanted to reverse a tuple, you'd need to return a new tuple; you can't change the original.
(2) Tell me something interesting -- take the given data and extract some information. However, I'm going to need the originals, so leave them alone. If I need to remember this cool new insight, I'll put it in a variable of my own. This is a function that returns a value. sqrt is one such function.
(3) Interact with the outside world -- input or output data permanently. For output, nothing in the program changes; we may present the data in an easy-to-read format, but we don't change anything internally. print is such a function.
Much of this decision also depends on the function's designed purpose: is this a "verb" function (do something) or a noun/attribute function (look at this data and tell me what you see)?
Now you get the interesting job for yourself: learn the art of system design. You need to become familiar enough with the available programming tools that you have a feeling for how they can be combined to form useful applications.
See the documentation:
The reverse() method modifies the sequence in place for economy of space when reversing a large sequence. To remind users that it operates by side effect, it does not return the reversed sequence.
I am trying to degenerate my data from already manipulated data. It takes five parameters: target, source, target_key, source_key, transformer
For example:
target = {}
source = {first_name: tom}
target_key = name
source_key = first_name
transformer = lambda value: value.title()
So, currently, I set first_name to name. and response becomes {name: Tom}
Now, I am trying to reverse it. If I get {name: Tom}, it should result in {first_name: tom} using same lambda or function. similarly, there are many other keys with different transformers
Is there any way/keyword to reverse the functionality of a lambda or any function.
Thanks,
Short answer: is if fundamentally impossible to construct an "inverse" function for a given function.
You cannot derive the "reverse" function from a given function (whether it is a lambda expression) is irrelevant. There are several aspects here:
First of all, it is possible that several inputs map on the same output. Take for instance the function lambda x : x.lower(). In that case both 'foo' and 'FOO' map to 'foo'. So even if you somehow could calculate input that maps on a given output, a question would be: "what input do you pick".
Next say we simply state that any input would suffice, one can ask whether it is possible. It still is not since the problem is also undecidable in the sense that if you provide as "expected output" a value that cannot be generated by the function, the hypothetical inverse function cannot know that. One can prove this by using computability theory since it would conflict with the fact that the emptiness problem ETM is undecidable.
Is there a theoretical way to derive an object that maps to a given valid value? Yes: one could enumerate over all possible inputs (it is an infinite, but countable so enumerable), calculate the output and then validate it. Furthermore the evaluation of functions should happen in "parallel" since it is possible one of the function calls results in an infinite loop.
Nevertheless hoping that it is realistic to construct a real function that calculates the inverse is not advisable. In a practical sense the above sketched algorithm is unfeasible. It would require an enormous amount of memory to store all the simulations of these functions. Furthermore it is possible that these have side effects (like writing to a file). As a result you should make copies of everything that might have side effects. Furthermore in practice some side effects cannot be "virtualized" or "undone". If the function for instance communicates with a web server, you cannot "undo" the HTTP request. It can also take ages before a valid input structure is entered and evaluated.
Like #JohnColeman says in his comment the fact that a function is not (feasibly) inverse is sometimes desired behavior. In asymmetrical encryption for instance the public key is usually publicly available. Nevertheless we do not want the message encrypted by the public key to be (efficiently) computable. A lot of todays cryptography and security depends on the fact that it is hard or impossible to perform the inverse operation of a function.
A final note is that of course it can be possible to construct an "inverse constructor" for certain families of functions. But in general (meaning a "inverse generator" that can take any kind of function as input), it is impossible.
To regenerate your data, you need to invert the mappings that were applied to it. There is no general function-inverse operator in Python or any other programming language, for the reasons that #Willem explained, but humans are pretty good at identifying and reversing simple manipulations. With enough work, it is possible to understand and reverse complicated manipulations too. This is part of how hackers reverse-engineer programs and algorithms, and it is what you need to do too if your data is worth the effort. (Of course you can partially
automate the process, especially if you know the kinds of manipulations that have been applied, e.g. if you wrote them yourself.)
If you have the source code, it's relatively easy: Inspect each function, write a suitable inverse (to the extent that hey exist), and write a main loop that somehow determines which inverse to apply. If you don't have the source code but have the compiled program (.pyc or .pyo files), you can still disassemble them and puzzle out what they do. See the dis module (but it's not at at all trivial):
>>> import dis
>>> dis.dis(transformer)
1 0 LOAD_FAST 0 (value)
3 LOAD_ATTR 0 (title)
6 CALL_FUNCTION 0 (0 positional, 0 keyword pair)
9 RETURN_VALUE
So... the bottom line is, you have to do it yourself. Good luck with it.
I'm quite new to python (2.7) and have a question about what's the most Pythonic way to do something; my code (part of a class) Looks like this (a somewhat naive Version):
def calc_pump_height(self):
for i in range(len(self.primary_)):
for j in range(len(self.primary_)):
if self.connections_[i][j].sub_kind_ in [1,4]:
self.calc_spec_pump_height(i,j)
def calc_spec_pump_height(self,i,j):
pass
(obviously pass will be replaced by something else, manipulating attributes of the object of this class, without generating a return value)
I'd like to ask how I should do this: I could avoid the second function and write the extra code directly into the first function, getting rid of one function (Simple is better than complex), but creating a heavily nested function at the same time (Flat is better than nested).
I could also create some sort of list comprehension to avoid using a double Loop, eg:
def calc_pump_height(self):
ra = range(len(self.primary_))
[self.calc_spec_pump_height(i,j) for i,j in zip(ra, ra)]
(I'd have to move the if condition into the 2nd function; this would also create a null-list but I don't care about this, since calc_spec_pump_height is supposed to manipulate the object, not return something useful)
In essence: I'm iterating over a 2D list, testing each object for a certain characteristic and then do something with that object.
Which of the above methods is 'the best'? Or is there another way that I'm missing?
The key thing about functions/methods is that they should do one thing.
calc_pump_height implements two things: It finds elements in a 2D list that match some criteria, and then it calculates a value for each of those elements. It's ok for its purpose to be combining the other two operations, if that makes sense for the object's public API, but its not ok for it to implement either or both.
Finding the elements that match the criteria is a discrete step; that should be a function.
Calculating your value is clearly a discrete step; that should be a function.
I would implement the element matcher as a (private) generator, that takes the test condition as an argument, and yields all matching elements. It's just an iterator over your data structure, masked by the logical test. You can wrap that in a named public method called get_1_4_subkinds() or something that makes more sense in your domain. That generalises the code and gives you the flexibility to implement other conditions in the future. Also, your i and j are tightly coupled, so it makes sense to pass them around as a single concept. Then your code becomes:
def calc_pump_height(self):
for subkind_indices in self.get_1_4_subkinds():
self.calc_pump_spec_height(subkind_indices)
You have misunderstood “simplicity”:
write the extra code directly into the first function, getting rid of one function (Simple is better than complex)
That's not simple. Breaking complex sequences into discrete, focussed functions increases simplicity.
In that light, I would say that yes, you should definitely prefer calc_spec_pump_height as a separate function.
You can eliminate one level of nesting in your first function by using itertools.product to generate your i and j values at the same time (itertools.product(range(len(self.primary_)), repeat=2). The zip you use in the your second version won't work correctly, it will only yield identical pairs, 0,0, 1,1, 2,2, etc.
As for the overall design, you should not use a list comprehension if you don't care about the return value from the function you're calling. Use an explicit loop when it's the looping you want (rather than a list of computed values).
If there's a non-trivial amount of code that will go in calc_spec_pump_height, it makes perfect sense to make it as a separate method. If it's a one or two liner, then it might be OK to inline within calc_pump_height, but that method's loops and condition testing may be complicated enough already to justify factoring out the inner part of the algorithm.
You should usually think about splitting a big function up when it is too long to fit onto a single screen in your editor. That is about the limit of how many details (variable names, etc.) we can keep in our mind simultaneously. On the other hand, you shouldn't waste time (either your own programming time or function call overhead at run time) by factoring out every little piece of every problem. Factor part of a function out if you're using it from more than one place, or if you can't keep the details of the whole function in your head at once otherwise.
So, other than the (marginal) improvement of itertools.product and given the limited information you've provided about what calc_spec_pump_height will do, I think your code is already about as good as it can get!
I have a program (PatchDock), which takes its input from a parameters file, and produces an output file. Running this program is time-intensive, and I'd like to cache results of past runs so that I need not run the same parameters twice.
I'm able to parse the input and output files into appropriate data structures. The input file, for example, is parsed into a dictionary-like object. The input keys are all strings, and the values are primitive data types (ints, strings, and floats).
My approach
My first idea was to use the md5 hash of the input file as the keys in a shelve database. However, this fails to capture cached files with the exact same inputs, but some slight differences in the input files (comments, spacing, order of parameters, et cetera).
Hashing the parsed parameters seems like the best approach to me. But the only way I can think of getting a unique hash from a dictionary is to hash a sorted string representation.
Question
Hashing a string representation of a parameters dictionary seems like a roundabout way of achieving my end goal- keying unique input values to a specified output. Is there a more straightforward way to achieve this caching system?
Ideally, I'm looking to achieve this in Python.
Hashing a sorted representation of the parsed input is actually the most straightforward way of doing this, and the one that makes sense. Your instincts were correct.
Basically, you're normalizing the input (by parsing it and sorting it), and then using that to construct a hash key.
Hashing seems a very viable way, but doing this yourself seems a bit overkill. Why not use the tuple of inputs as key for your dictionary? You wouldn't have to worry about hashing and possible collisions yourself. All you have to do is fix a order for the keyword arguments (and depending on your requirements add a flag-object for keywords that are not set).
You also might find the functools.lru_cache useful, if you are using Python 3.2+.
This is a decorator that will enable caching for the last n calls of the decorated function.
If you are using a older version there are backports of this functionality out there.
Also there seem to be a project with similar goals called FileDict which might be worth looking at.