I want to map a multi-step function onto a set of arrays. The multi-step (parent) function is composed of several helper functions. Currently, I am using map(parentFunction, argArrays**). The parent function and the helper functions all take in individual floats. Would it be more computationally efficient to instead make the parent function take in the arrays, and use map(helperFunction_i, someArgArrays**)?
I'm struggling here because I don't fully understand what's going on under the hood here. And while a fractionally small speed increase may not matter, an order of magnitude speed increase may matter because the parent function here is going to be nested inside a monte carlo loop, that monte carlo loop as a whole may be called thousands of times for a high-level function evaluation of a model I'm building, and I expect to need thousands of those high-level function evaluations for my research purposes.
While I'd most appreciate a direct and unambiguous answer to my question if an unambiguous answer exists, I'd also appreciate any recommended readings that would help me to understand how to write efficient code in python.
Related
I have a Python function where I want to find the global minimum.
I am looking for a resource on how to choose the appropriate algorithm for it.
When looking at scipy.optimize.minimize documentation, I can find some not very specific statements like
has proven good performance even for non-smooth optimizations
Suitable for large-scale problems
recommended for medium and large-scale problems
Apart from that I am unsure whether for my function basinhopping might be better or even bayesian-optimization. For the latter I am unsure whether that is only meant for machine learning cost functions or can be used generally.
I am looking for a cheat sheet like
but just for minimization problems.
Does something like that already exist?
If not, how would I be able to choose the most appropriate algorithm based on my constraints (that are: time consuming to compute, 4 input variables, known boundaries)?
Is there any place with a brief description of each of the algorithms for the parameter method in the minimize function of the lmfit package? Both there and in the documentation of SciPy there is no explanation about the details of each algorithm. Right now I know I can choose between them but I don't know which one to choose...
My current problem
I am using lmfit in Python to minimize a function. I want to minimize the function within a finite and predefined range where the function has the following characteristics:
It is almost zero everywhere, which makes it to be numerically identical to zero almost everywhere.
It has a very, very sharp peak in some point.
The peak can be anywhere within the region.
This makes many minimization algorithms to not work. Right now I am using a combination of the brute force method (method="brute") to find a point close to the peak and then feed this value to the Nelder-Mead algorithm (method="nelder") to finally perform the minimization. It is working approximately 50 % of the times, and the other 50 % of the times it fails to find the minimum. I wonder if there are better algorithms for cases like this one...
I think it is a fair point that docs for lmfit (such as https://lmfit.github.io/lmfit-py/fitting.html#fit-methods-table) and scipy.optimize (such as https://docs.scipy.org/doc/scipy/reference/tutorial/optimize.html#optimization-scipy-optimize) do not give detailed mathematical descriptions of the algorithms.
Then again, most of the docs for scipy, numpy, and related libraries describe how to use the methods, but do not describe in much mathematical detail how the algorithms work.
In fairness, the different optimization algorithms share many features and the differences between them can get pretty technical. All of these methods try to minimize some metric (often called "cost" or "residual") by changing the values of parameters for the supplied function.
It sort of takes a text book (or at least a Wikipedia page) to establish the concepts and mathematical terms used for these methods, and then a paper (or at least a Wikipedia page) to describe how each method differs from the others. So, I think the basic answer would be to look up the different methods.
i am trying to optimize (minimize to be more specific) a function using python. The standard way led me to using scipy.optimize.minimize. However, i have a large number of variables with individual constraints that obey the same function to be minimized (~ 100,000).
The function itself is not fully analytical, as it involves some prediction using neural networks. In order to speed things up and to reduce memory consumption it would be very beneficial if all variables could be optimized in parallel (such that the processing, like the ML prediction is done in parallel). The task is doable with multiprocessing, but i believe it would be much faster if it could be vectorized.
The fact that all variables can be treated independently should make this doable.
Does anyone know if there is a package similar to scipy.optimize.minimize that is able to minimize a vector-output function with a large number of independent variables (ideally with the flexibility to incorporate constraints and bounds for each variable)?
In order to clarify what i am intending:
I want multiple parallel optimizations, where all variables of one optimization are independent of all other optimization. Ideally this would be formulated in a vectorized manner, by means of a multivariate-multiobjective constrained minimization procedure. Imaging the following example:
You have food diaries of ~100,000 people and you use pandas.DataFrame to handle the data and you trained some ML model to predict if the food is good for each individuals. Clearly this task would be faster to operate vectorized than in a for-loop for every individual.
Now assume that you want to vary some of the features in the food diary, e.g., to find the optimum dose or quantity for some of the items (this is what you would like to optimize). And this is clearly independent of all other individuals.
I have a multivariate optimization problem that I want to run. Each evaluation is quite slow. So obviously the ability to thread it out to multiple machines would be quite nice. I have no trouble writing the code to dispatch jobs to other machines. However, scipy.optimize.minimize calls each evaluation call sequentially; it won't give me another set of parameters to evaluate until the previous one returns.
Now, I know that the "easy" solution to this would be "run your evaluation task in a parallel manner - break it up". Indeed, while it is possible to do this to some extent, it only goes so far; the bandwidth rises the more you split it up until splitting it up further actually starts to slow you down. Having another axis in which to parallelize - aka, the minimization function itself - would greatly increase scaleability.
Is there no way to do this with scipy.optimize.minimize? Or any other utility that performs in a roughly functionally equivalent manner (trying to find as low of a minimum as possible)? Surely it's possible for a minimization utility to make use of parallelism, particularly on a multivariate optimization problem where there's many axes whose gradients relative to each other at given datapoints need to be examined.
My programs are growing larger and more sophisticated. As a result I am using more and more functions. My question is, should I "fetch" a value from a function once and then "tote" it around, sending it into other functions as a parameter, or just call, "fetch," the value again from within the other function(s)?
I am sure resources, and speed, are a factor, but what is the general rule, if any?
For example, should I call my sigmoid function, and then use that value as a parameter in a call to the next function that uses it, or just call the sigmoid function again from within that next function?
I know that this question borders on opinion, but I did not attend a CS school, and so find myself wondering what the "norm" for some things are.
Thanks.
You are correct that this question relates more to software engineering theory than just a language (Python). There are programming paradigms which promote one variant over the other but the most general rule of thumb you should aim for is:
High cohesion and low coupling
i.e., within a module of software (which roughly corresponds to a Python module, if you are using them), the functions should have dependence on each other and you should call them to fetch the value. However, across modules, you should not have functional calls and should tie them up at a higher level module (or the main function) by fetching values from one module and passing it to the other.
See also: Memoization.