I recently installed the tsfresh package to extract features of my timeseries data. I tried to run the example in the documentation and got the following error:
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.
I'm a bit confused since it's literally the example code:
from tsfresh.examples import load_robot_execution_failures
from tsfresh import extract_features
df, _ = load_robot_execution_failures()
X = extract_features(df, column_id='id', column_sort='time')
I get the same error when i try the function with my own data.
What am i doing wrong?
I think there is solution to the problem.
You can try to put "if __name__ == __main__:" around function callings.
Related
So I am trying to run LDA mallet on a dataset. It takes in lemma tokens and a bunch of texts which is our dataset. The issue is when we run, a freeze message pops up and all of our old methods that have already ran start running again. It says its due to the multiprocessing starting before the other finished. Not sure how to fix. This is ran on MacOS. Code and output are below.
import gensim
from gensim.models.coherencemodel import CoherenceModel
from gensim.corpora import Dictionary
from gensim.models.ldamodel import LdaModel
import os.path
def optimize_parameters(lemma_tokens, texts):
os.environ['MALLET_HOME'] = '****/mallet-2.0.8'
mallet_path = '****/mallet-2.0.8/bin/mallet'
id2word = Dictionary(lemma_tokens)
# Filtering Extremes
id2word.filter_extremes(no_below=2, no_above=.99)
# Creating a corpus object
corpus = [id2word.doc2bow(d) for d in lemma_tokens]
model = gensim.models.wrappers.LdaMallet(mallet_path, corpus=corpus, num_topics=5, id2word=id2word, workers = 4)
coherencemodel = CoherenceModel(model=model, texts=lemma_tokens, dictionary=id2word, coherence='c_v')
coherence = coherencemodel.get_coherence()
The "****" is the rest of the path that can't be shown due to privacy.
The error output:
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.
<10> LL/token: -6.83952
<20> LL/token: -6.70949
I figure it out. You have to put the entire script in
if __name__ == '__main__':
imports
code
Found solution via an old google chat. Posted link:
https://groups.google.com/g/gensim/c/-gMNdkujR48/m/i4Dn1_bjBQAJ
To summarize what is happening, due to multiprocessing, the other bits of code are run multiple times instead of the once it is supposed to. This is the same case for the actual function itself which runs the same call multiple times. The fix of the if statement checks to see if this is the first run through. If it is, then we do the entire call. If not, we don't run anything at all. This works because it makes sure that we are only running it once.
I'm trying to understand why the following simple example doesn't successfully complete execution and seems to get stuck on the first line of really_simple_func (on Ubuntu machines, but not Windows). The code is:
import torch as t
import numpy as np
import multiprocessing as mp # I've tried both multiprocessing
# import torch.multiprocessing as mp # and torch.multiprocessing
def really_simple_func():
temp_val_2 = t.tensor(np.zeros(425447)[0:400000]) # this is the line that blocks.
return 4.3
if __name__ == "__main__":
print("Run brief starting")
some_zeros = np.zeros(425447)
temp_val = t.tensor(some_zeros[0:400000]) # DELETE THIS LINE TO MAKE IT WORK
pool = mp.Pool(processes=1)
job = pool.apply_async(really_simple_func)
print("just before job.get()")
result = job.get()
print("Run brief completed. Reward = {}".format(result))
I have torch 1.11.0 installed, numpy 1.22.3 and have tried both CPU and GPU versions of Torch. When I run this code on two different Ubuntu machines, I get the following output:
Run brief starting
just before job.get()
However, the code never successfully completes (doesn't print the "Run brief completed" line). (It does complete on a third Windows box).
On the Ubuntu machines, if I delete the line with the comment "#DELETE THIS LINE TO MAKE IT WORK" the execution DOES complete, printing the final line as expected. Similarly, if I leave the line defining temp_val in but delete the line with the comment "This is the line that blocks" it will also complete. Moreover, if I reduce the size of the temp_val tensor (say from 400000 to 4000) it will also complete successfully. Finally, it is worth noting that while I can reproduce this behaviour on two different Ubuntu machines, this code does actually complete on my Windows machine - though, as far as I can tell, the versions of key packages, such as torch, are the same.
I don't understand this behaviour. I suspect it is something to do with the way torch allocates memory or stores information. I've tried calling del temp_val to free up memory, but that doesn't seem to fix things. It seems to me that the async call to t.tensor within really_simple_func is stopped from completing if there has already been a call to t.tensor in the main code block, creating a sufficiently large tensor.
I don't understand why this is happening, or even if that is the correct explanation. In any case, what would be best practice if I do need to do some tensor processing within apply_async as well as in the main thread? More generally, what is Torch waiting on when I make a call to t.tensor?
(Obviously, this is just the simplest version of the real code I'm trying to get to work that reproduced this issue. I realise that calling mp.Pool with only one process doesn't really make sense...nor, indeed, does using apply_async to call a function that returns a constant!)
Unfortunately, I cannot provide any answers to your questions.
I can, however, share experiences with seemingly the same issue. I use a Linux machine with torch 1.8.1 and numpy 1.19.2.
When I run the following code on my machine:
with Pool(max_pool) as p:
pool_outputs = list(
tqdm(
p.imap(lambda f: get_model_results_per_query_file(get_preds, tokenizer, f), query_files),
total=len(query_files)
)
)
For which the function get_model_results_per_query_file contains operations similar to the following:
feats = features.unsqueeze(0).repeat(batch_size, 1, 1).to(device)
(features is a torch tensor)
The first round of jobs automatically fail, and new ones are immediately started (that do not fail for some reason). The whole process never completes though, since the main process still seems to be waiting for the first failed jobs.
If I remove the lines in my code involving the repeat function, no jobs fail.
I managed to solve my issue and preserve the same results by adapting a similar solution to yours:
feats = torch.as_tensor(np.tile(features, (batch_size, 1, 1))).to(device)
I believe as_tensor works in a similar fashion to from_numpy in this case.
I only managed to find this solution thanks to your post and your proposed workaround, so thank you!
After some further exploration, here is a brief answer to my own question.
While I still don't fully understand the blocking behaviour (and would welcome any further explanation), I have just seen that the way I'm generating torch tensors from a numpy array is not correct.
In particular, instead of using torch.tensor(temp_val) where temp_val is a numpy array, I should be using torch.from_numpy(temp_val). Doing this fixes the problem.
Alternatively, I can convert temp_val into a list and then create the tensor via torch.tensor(temp_val_as_list) - which also avoids the issue.
I am trying to parallelise a series of computations that use bigfloat. However, there is the error
Error sending result: '[BigFloat.exact('1.0000000', precision=20)]'. Reason: 'TypeError('self._value cannot be converted to a Python object for pickling')'
I MWE to reproduce the error is
from bigfloat import *
from multiprocessing import Pool
def f(x,a,b,N):
with precision(20):
X=BigFloat(x)
for i in range(N):
X = a*X*X-b
return X
if __name__ == '__main__':
pool = Pool(processes=2)
out1,out2 = pool.starmap(f,[(1,2,1,3),(2,2,2,2)])
(the function itself is not important at all). If I do not use bigfloat, then everything is fine. So, it is definitely some sort of interaction between multiprocessing and bigfloat.
So, I imagine that multiprocessing is having troubles saving the BigFloat object. I do not seem to be able to "extract" only the value thrown by BigFloat. How can I resolve this issue?
apparently bigfloat doesn't support pickling, I get the same error when doing pickle.dumps(BigFloat(1))
https://github.com/mdickinson/bigfloat/issues/106 notes this as needing to be done
as a work around, why not just convert to strings when transferring between processes? i.e. change f to return str(X) and then have other routines parse the strings into BigFloats as needed
otherwise, you could write some code to support this and submit it to the project
I am using joblib to parallel a for loop for my own function.
from joblib import Parallel, delayed
from my_function import my_case_study
result = Parallel(n_jobs=4)(delayed(my_case_study)(i) for i in range(100))
So my_case_study is the only function in my_function.py file, and it takes i as the hyperparameter. my_case_study calls a bunch of different model fitting algorithm contains in the other python files, which are imported at the top of my_function. my_function.py basicly looks like
from anotherfile import fun1
from anotherfile import fun2
def my_case_study(i):
mse1 = fun1(i)
mse2 = fun2(i)
return (mse1,mse2)
But then I get the error message: A task gas failed to un-serialize. Please ensure that the arguments of the function are all picklable.
How to solve this? Thanks!!
I found the solution in the following link helpful in my case:
https://github.com/joblib/joblib/issues/810
Don't know if there is any other better solution. Since the comments at the end mentioned that there might be some issues with that (didn't fully understand).
I've looked around the forum and can't seem to find anything that helps. I'm trying to parallelize some processes but cannot seem to get it to work.
from multiprocessing import *
from numpy import *
x=array([1,2,3,4,5])
def SATSolver(args):
#many_calculations
return result
def main(arg):
new_args=append(x,arg)
return SATSolver(new_args)
y=array([8,9,10,11])
if __name__ == '__main__':
pool=Pool()
results=pool.map(main,y)
print(results)
The SATSolver function is where the bulk of the work happens. Basically, I have an array x and a second array y. I want to add append each value in y to x individually, and then run this new set through my SATSolver function. I would like to use the multiprocessing module so this can run in parallel.
Whenever I try and run this, I don't get an error, but a new interactive window pops up and says "Could not load the file from filepath\-c. Do you want to create a new file?"
Everything works perfectly when it's run without multiprocessing.
Any ideas on how to make this work?
Thanks!