I want to compare multithreading and multiprocessing python program. But, I got interpretation error:
File "./parallelPython.py", line 23
time fornorm(g,range(100))
^
SyntaxError: invalid syntax
The code is as follows:
#!/usr/bin/python -tt
import numpy as np
import math
def f(x):
print x
y = [1]*10000000
[math.exp(i) for i in y]
def g(x):
print x
y = np.ones(10000000)
np.exp(y)
from handythread import foreach
from processing import Pool
from timings import f,g
def fornorm(f,l):
for i in l:
f(i)
time fornorm(g,range(100))
time fornorm(f,range(10))
time foreach(g,range(100),threads=2)
time foreach(f,range(10),threads=2)
p = Pool(2)
time p.map(g,range(100))
time p.map(f,range(100))
I do not why fornorm() has a problem, it has been defined !!!
thanks
It doesn't say fornorm hasn't been defined, it says you have a syntax error on the line where you're calling fornorm. Syntax errors mean Python can't even understand your code: it's as if I say to you "flrk ask web ski ur lkjq", and then ask you to do what I said. An error about fornorm not being defined would happen much later. As it is, Python can't even tell whether you're asking it to call a function, let alone whether you're calling one that is already defined or not.
It looks like your error is this:
time fornorm(g,range(100))
That looks like you're trying to use the shell command time. Shell commands are not Python, and Python doesn't understand it.
However, your code as pasted into SO also has indentation errors, which should have triggered a syntax error earlier than that line, so I suspect what we can see here is not exactly what you were running.
It looks like an Indentation error here :
def fornorm(f,l):
for i in l:
f(i)
After your def python is expecting an indented block.
By the way, time something is an IPython "magic" function and it won't work in a script file. You should import timeit module and use that instead.
Where are you getting "time" from? That's not a valid python statement. It's not like shell scripting.
If you want to time stuff, use the timeit library:
http://docs.python.org/library/timeit.html
Related
I'm trying to run the code below, and the output that I get shows me nothing. My question is: is there a problem with using pandas functions within a def scope?
import numpy as np
import pandas as pd
def readTrainingData():
url = "https://raw.githubusercontent.com/MohammadWasil/Predicting-Titanic-Survivors/master/train.csv"
dadosTreino = pd.read_csv(url)
return(dadosTreino)
def main():
dfTrain = readTrainingData()
dfTrain.head()
main()
Python's interpreter prompt is a REPL, or "read-eval-print loop". The "print" is key here: it will attempt to display the result of each calculation (as long as the "eval"uation produced something other than the special value None).
But this printing is only caused by the interpreter prompt, not by the code itself. So there is no implicit printing when you write code in a source file and run it with e.g. python myfile.py.
dfTrain.head() computes a value, but there is nothing to make it display. It also is not returned from the function, so even if you tried to import the code from the interpreter prompt and call main() from there, it will evaluate to None and not print anything.
I want to pass function definition to a python command line script. What is the best way to do this? I am using python 2. Suppose i have a script like this:
#myscript.py
x = load_some_data()
my_function = load_function_definition_from_command_line()
print my_function(x)
And i want to call it like this: python myscript.py 'def fun(x): return len(x)'
How do i perform the load_function_definition_from_command_line part ?
I imagine a workaround:
get the string function definition from command line
write it to a file with .py extension in some temp directory
load the definition from file using solutions from this question: How to import a module given the full path?
execute
cleanup
But I am sure there must be a better way.
You can use eval to run code defined in a string. Like so:
import sys
x = load_some_data()
function = eval("".join(sys.argv[1:]))
print(function(x))
With your specific example though you might have to use something like lambda x: len(x)
As #Jan-Spurny rightly points out: "Never, never, never use eval unless you're absolutely sure there is no other way. And even then you should stop and think again."
In my mind the better strategy would be to turn the data loader and executor into a module with a method that takes a function as an argument and runs the desired code. The end result something like this:
import data_loader_and_executor
def function(x):
return len(x)
data_loader_and_executor.run(function)
You can use eval or exec to create a function in your current namespace.
exec "somefunc(x): return x * 2"
somefunc(2) # 2
Example within your context
python load_function.py "def f(x): return x * 2"
//load_function.py
import sys
exec sys.argv[1]
print f(2)
Command line output:
4
Edit: Obligatory, "It is not wise to execute user input like this."
Use function exec:
import sys
def load_function_definition_from_command_line():
exec(sys.argv[1])
return locals()['fun']
Of course you have to know, how your function will be named, but this can be done by passing to your argument second argument:
$ python myscript.py 'def fun(x): return len(x)' fun
And then your function will look like:
import sys
def load_function_definition_from_command_line():
exec(sys.argv[1])
return locals()[sys.argv[2]]
!!Remember though, that evaluating user input is very dangerous!!
Edit: Since fun would be the only object defined in locals, you can just return first element in locals():
def load_function_definition_from_command_line():
exec(sys.argv[1])
return locals()[0]
The most obvious source for the correct answer on how to do this is in the timeit python builtin library.
It is invoked like this:
$ python -m timeit '"-".join(str(n) for n in range(100))'
and you can find the source code here, which uses compile and exec to invoke the code from the command line
https://hg.python.org/cpython/file/2.7/Lib/timeit.py#l143
I'm just finding out now that when importing a module, it seems to run through ALL the code, instead of just the one function that I want it to go through. I've been trying to find a way around this, but can't seem to get it. Here is what is happening.
#mainfile.py
from elsewhere import something_else
number = 0
def main():
print('What do you want to do? 1 - something else')
donow = input()
if donow == '1':
something_else()
while 1:
main()
#elsewhere.py
print('I dont know why this prints')
def something_else():
from mainfile import number
print('the variable number is',number)
Now, although this code KIND OF works the way I want it to, the first time when I initiate it, it will go to the main menu twice. For example: I start the program, press one, then it asks me what I want to do again. If I press one again, then it will print "the variable number is 0".
Once I get this working, I would like to be importing a lot of variables back and forth. The only issue is,if I add more import statements to "elsewhere.py" I think it will just initiate the program more and more. If I put "from mainfile import number" on line 1 of "elsewhere.py", I think this raises an error. Are there any workarounds to this? Can I make a different file? What if I made a class to store variables, if that is possible? I'm very new to programming, I would appreciate it if answers are easy to read for beginners. Thank you for the help.
As Jan notes, that's what import does. When you run import, it runs all of the code in the module. You might think: no it doesn't! What about the code inside something_else? That doesn't get run! Right, when the def statement is executed it creates a new function, but it doesn't run it. Basically, it saves the code for later.
The solution is that pretty much all interesting code should be in a function. There are a few cases which make sense to put at the top-level, but if in doubt, put it inside a function. In your particular case, you shouldn't be printing at the top level, if you need to print for some reason, put that into a function and call it when you need it. If you care when something happens, put it in a function.
On a second node, don't import your primary script in other scripts. I.e. if your mainfile.py directly, don't import that in other files. You can but it produces confusing results, and its really best to pretend that it doesn't work.
Don't try to import variables back and forth. Down that path lies only misery. You should only be importing things that don't change. Functions, classes, etc. In any other case, you'll have hard time making it do what you want.
If you want to move variables between places, you have other options:
Pass function arguments
Return values from a function
Use classes
I'll leave it is an exercise to the reader to learn how to do those things.
import executes imported code
import simply takes the Python source file and executes it. This is why it prints, because that instruction is in the code and with import all the instructions get exectued.
To prevent execution of part of imported package/module, you shall use the famous:
if __name__ == "__main__":
print("I do not print with `import`")
Note, that this behaviour is not new in Python 3, it works the same way in Python 2.x too.
Is there any way that I can create a program where it gives input to another file and and collects its output?
The best that google give me is this. And I tried to recreate (read: copying the code in some unknown manner (read: stabbing in the dark))
And I got this
import time
string="file.py"
process=subprocess.Popen(string,stdin=subprocess.PIPE,stdout=subprocess.PIPE);
process.stdin.write("3 5")
time.sleep(1)
print process.stdout.read()
And it gives me error
File "D:\Pekerjaan non website\IO\reader.py", line 3, in <module>
process=subprocess.Popen(string,stdin=subprocess.PIPE,stdout=subprocess.PIPE);
NameError: name 'subprocess' is not defined
Can anybody tell me how to create that program. (
Note: I have no knowledge in this kind of program I/O or subprocess
Note: It's up to you. Where you will explain this from my code or throw my code away and explain this from zero.
Thank you beforehand.
(PS: If my previous statement is confusing. My point is simple: Can you teach me? :))
subprocess is a stdlib module that you need to import (the same way time is) - so you just need to:
import subprocess
some time before you try to use the functions in it (usually, you want to do this near the top of your code, right underneath your current import time).
In addition to lvc's answer you should consider using Popen.communicate(). something like,
import subprocess
string="file.py"
process=subprocess.Popen(string,stdin=subprocess.PIPE,stdout=subprocess.PIPE)
res=process.communicate("3 5")
print res[0]
In this way you don't have to use that sleep command.it'll Wait for process to terminate and return a tuple as (stdoutdata, stderrdata).
Imagine a python script that will take a long time to run, what will happen if I modify it while it's running? Will the result be different?
Nothing, because Python precompiles your script into a PYC file and launches that.
However, if some kind of exception occurs, you may get a slightly misleading explanation, because line X may have different code than before you started the script.
When you run a python program and the interpreter is started up, the first thing that happens is the following:
the module sys and builtins is initialized
the __main__ module is initialized, which is the file you gave as an argument to the interpreter; this causes your code to execute
When a module is initialized, it's code is run, defining classes, variables, and functions in the process. The first step of your module (i.e. main file) will probably be to import other modules, which will again be initialized in just the same way; their resulting namespaces are then made available for your module to use. The result of an importing process is in part a module (python-) object in memory. This object does have fields that point to the .py and .pyc content, but these are not evaluated anymore: module objects are cached and their source never run twice. Hence, modifying the module afterwards on disk has no effect on the execution. It can have an effect when the source is read for introspective purposes, such as when exceptions are thrown, or via the module inspect.
This is why the check if __name__ == "__main__" is necessary when adding code that is not intended to run when the module is imported. Running the file as main is equivalent to that file being imported, with the exception of __name__ having a different value.
Sources:
What happens when a module is imported: The import system
What happens when the interpreter starts: Top Level Components
What's the __main__ module: __main__- Top-level code environment
This is a fun question. The answer is that "it depends".
Consider the following code:
"Example script showing bad things you can do with python."
import os
print('this is a good script')
with open(__file__, 'w') as fdesc:
fdesc.write('print("this is a bad script")')
import bad
Try saving the above as "/tmp/bad.py" then do "cd /tmp" and finally "python3 bad.py" and see what happens.
On my ubuntu 20 system I see the output:
this is a good script
this is a bad script
So again, the answer to your question is "it depends". If you don't do anything funky then the script is in memory and you are fine. But python is a pretty dynamic language so there are a variety of ways to modify your "script" and have it affect the output.
If you aren't trying to do anything funky, then probably one of the things to watch out for are imports inside functions.
Below is another example which illustrates the idea (save as "/tmp/modify.py" and do "cd /tmp" and then "python3 modify.py" to run). The fiddle function defined below simulates you modifying the script while it is running (if desired, you could remove the fiddle function, put in a time.sleep(300) at the second to last line, and modify the file yourself).
The point is that since the show function is doing an import inside the function instead of at the top of the module, the import won't happen until the function is called. If you have modified the script before you call show, then your modified version of the script will be used.
If you are seeing surprising or unexpected behavior from modifying a running script, I would suggest looking for import statements inside functions. There are sometimes good reasons to do that sort of thing so you will see it in people's code as well as some libraries from time to time.
Below is the demonstration of how an import inside a function can cause strange effects. You can try this as is vs commenting out the call to the fiddle function to see the effect of modifying a script while it is running.
"Example showing import in a function"
import time
def yell(msg):
"Yell a msg"
return f'#{msg}#'
def show(msg):
"Print a message nicely"
import modify
print(modify.yell(msg))
def fiddle():
orig = open(__file__).read()
with open(__file__, 'w') as fdesc:
modified = orig.replace('{' + 'msg' + '}', '{msg.upper()}')
fdesc.write(modified)
fiddle()
show('What do you think?')
No, the result will not reflect the changes once saved. The result will not change when running regular python files. You will have to save your changes and re-run your program.
If you run the following script:
from time import sleep
print("Printing hello world in: ")
for i in range(10, 0, -1):
print(f"{i}...")
sleep(1)
print("Hello World!")
Then change "Hello World!" to "Hello StackOverflow!" while it's counting down, it will still output "Hello World".
Nothing, as this answer. Besides, I did experiment when multiprocessing is involved. Save the script below as x.py:
import multiprocessing
import time
def f(x):
print(x)
time.sleep(10)
if __name__ == '__main__':
with multiprocessing.Pool(2) as pool:
for _ in pool.imap(f, ['hello'] * 5):
pass
After python3 x.py and after the first two 'hello' being printed out, I modified ['hello'] to ['world'] and observed what happend. Nothing interesting happened. The result was still:
hello
hello
hello
hello
hello
It happens nothing. Once the script is loaded in memory and running it will keep like this.
An "auto-reloading" feature can be implemented anyway in your code, like Flask and other frameworks does.
This is slightly different from what you describe in your question, but it works:
my_string = "Hello World!"
line = input(">>> ")
exec(line)
print(my_string)
Test run:
>>> print("Hey")
Hey
Hello World!
>>> my_string = "Goodbye, World"
Goodbye, World
See, you can change the behavior of your "loaded" code dynamically.
depending. if a python script links to other modified file, then will load newer version ofcourse. but if source doesnt point to any other file it'll just run all script from cache as long as its run. changes will be visible next time...
and if about auto-applying changes when they're made - yes, #pcbacterio was correct. its possible to do thar but script which does it just remembers last action/thing what was doing and checks when the file is modified to rerun it (so its almost invisible)
=]