from celery.decorators import task
from celery.decorators import task
#task()
def add(x, y):
r = open("./abc.txt","w")
r.write("sdf")
r.close()
return x + y
That's my tasks.py file.
>>> import tasks
>>> r = tasks.add.delay(3,5)
>>> r.result
8
As you can see, the function works. However, the file does not create.
Why?
I've tried changing multiple file paths, due to possible permission issues. but no luck.
If the file was not being written, you would get an exception, so the function cannot possibly complete.
Since the function is returning 8, it follows that the file is being written somewhere.
Perhaps the file is written in a different directory to the one you are expecting
The only other possibility I can think of is that the add function that is being run is not the one that you have shown here
I think the problem is that you're running this by importing a module. The . in the file path is relative to where the module lives, not your current working directory. Try giving it a full path name.
If that doesn't work, show us exactly where you're running the script from and an ls -la on that directory. And if that still doesn't show anything abnormal. Do a
find / -name abc.txt
The working directory may not be the one you expect.
The task may be done on another host, if you use multiple hosts.
Related
I have two Python files (main.py and main_test.py). The file main_test.py is executed within main.py. When I do not use a log file this is what gets printed out:
Main file: 17:41:18
Executed file: 17:41:18
Executed file: 17:41:19
Executed file: 17:41:20
When I use a log file and execute main.py>log, then I get the following:
Executed file: 17:41:18
Executed file: 17:41:19
Executed file: 17:41:20
Main file: 17:41:18
Also, when I use python3 main.py | tee log to print out and log the output, it waits and prints out after finishing everything. In addition, the problem of reversing remains.
Questions
How can I fix the reversed print out?
How can I print out results simultaneously in terminal and log them in a correct order?
Python files for replication
main.py
import os
import time
import datetime
import pytz
python_file_name = 'main_test'+'.py'
time_zone = pytz.timezone('US/Eastern') # Eastern-Time-Zone
curr_time = datetime.datetime.now().replace(microsecond=0).astimezone(time_zone).time()
print(f'Main file: {curr_time}')
cwd = os.path.join(os.getcwd(), python_file_name)
os.system(f'python3 {cwd}')
main_test.py
import pytz
import datetime
import time
time_zone = pytz.timezone('US/Eastern') # Eastern-Time-Zone
for i in range(3):
curr_time = datetime.datetime.now().replace(microsecond=0).astimezone(time_zone).time()
print(f'Executed file: {curr_time}')
time.sleep(1)
When you run a script like this:
python main.py>log
The shell redirects output from the script to a file called log. However, if the script launches other scripts in their own subshell (which is what os.system() does), the output of that does not get captured.
What is surprising about your example is that you'd see anything at all when redirecting, since the output should have been redirected and no longer echo - so perhaps there's something you're leaving out here.
Also, tee waits for EOF on standard in, or for some error to occur, so the behaviour you're seeing there makes sense. This is intended behaviour.
Why bother with shells at all though? Why not write a few functions to call, and import the other Python module to call its functions? Or, if you need things to run in parallel (which they didn't in your example), look at multiprocessing.
In direct response to your questions:
"How can I fix the reversed print out?"
Don't use redirection, and write to file directly from the script, or ensure you use the same redirection when calling other scripts from the first (that will get messy), or capture the output from the subprocesses in the subshell and pipe it to the standard out of your main script.
"How can I print out results simultaneously in terminal and log them in a correct order?"
You should probably just do it in the script, otherwise this is not a really a Python question and you should try SuperUser or similar sites to see if there's some way to have tee or similar tools write through live.
In general though, unless you have really strong reasons to have the other functionality running in other shells, you should look at solving your problems in the Python script. And if you can't, use you can use something like Popen or derivatives to capture the subscript's output and do what you need instead of relying on tools that may or may not be available on the host OS running your script.
For the life of me i can't figure this one out.
I have 2 applications build in python, so 2 projects in different folders, is there a command to say in the first application like run file2 from documents/project2/test2.py ?
i tried something like os.system('') and exec() but that only seems to work if its in the same folder. How can i give a command a path like documents/project2 and then for example:
exec(documents/project2 python test2.py) ?
short version:
Is there a command that runs python test2.py while that test2 is in a completely different file/project?
thnx for all feedback!
There's a number of approaches to take.
1 - Import the .py
If the path to the other Python script can be made relative to your project, you can simply import the .py. This will cause all the code at the 'root' level of the script to be executed and makes functions as well as type and variable definitions available to the script importing it.
Of course, this only works if you control how and where everything is installed. It's the most preferable solution, but only works in limited situations.
import ..other_package.myscript
2 - Evaluate the code
You can load the contents of the Python file like any other text file and execute the contents. This is considered more of a security risk, but given the interpreted nature of Python in normal use not that much worse than an import under normal circumstances.
Here's how:
with open('/path/to/myscript.py', 'r') as f:
exec(f.read())
Note that, if you need to pass values to code inside the script, or out of it, you probably want to use files in this case.
I'd consider this the least preferable solution, due to it being a bit inflexible and not very secure, but it's definitely very easy to set up.
3 - Call it like any other external program
From a Python script, you can call any other executable, that includes Python itself with another script.
Here's how:
from subprocess import run
run('python path/to/myscript.py')
This is generally the preferable way to go about it. You can use the command line to interface with the script, and capture the output.
You can also pipe in text with stdin= or capture the output from the script with stdout=, using subprocess.Popen directly.
For example, take this script, called quote.py
import sys
text = sys.stdin.read()
print(f'In the words of the poet:\n"{text}"')
This takes any text from standard in and prints them with some extra text, to standard out like any Python script. You could call it like this:
dir | python quote.py
To use it from another Python script:
from subprocess import Popen, PIPE
s_in = b'something to say\nright here\non three lines'
p = Popen(['python', 'quote.py'], stdin=PIPE, stdout=PIPE)
s_out, _ = p.communicate(s_in)
print('Here is what the script produced:\n\n', s_out.decode())
Try this:
exec(open("FilePath").read())
It should work if you got the file path correct.
Mac example:
exec(open("/Users/saudalfaris/Desktop/Test.py").read())
Windows example:
exec(open("C:\Projects\Python\Test.py").read())
I have an error that I keep encountering repeatedly, sadly without being able to find solution to at the site.
try:
#create working dir if it doens't exist already
if not os.path.isdir(WORKINGDIR):
print '>>>mdkir ',WORKINGDIR
subprocess.Popen(['mkdir',WORKINGDIR]).wait()
print os.path.isdir(WORKINGDIR)
#create output csv file
outputCSVFile = WORKINGDIR+ '/'+'results.csv'
if not os.path.isfile(outputCSVFile):
print '>>> touch',outputCSVFile
subprocess.check_output(['touch',outputCSVFile])
Although the line: print os.path.isdir(WORKINGDIR) always prints True, subprocess returns this error:
touch: cannot touch
`/nfs/iil/proj/mpgarch/archive_06/CommandsProfiling/fastScriptsOutput190916/results.csv':
No such file or directory
The same error doesn't appear when I use subprocess.checkoutput instead of subprocess.Popen().wait().
I know that this issue can be solved in many ways (such as using os methods to creates directories and files), but I am interested on why my way isn't working.
Thanks in advance.
EDIT: as some suggested, the problem probably lies with the fact that the program continues too fast after the subprocess.Popen, and hence the issue is solved using subprocess.checkoutput which is probably slower (since it has to wait for output). But still - I don't understand exactly what is happening, since os.path.istdir shows that the dir was created, before continuing to the line that performs touch
I suppose you have file permission problems.
In your path it appears that you are using NFS. Did you already try it on the local file system?
Anyway, you should avoid to use sub processes for simple file operations.
To create a directory:
if not os.path.exists(WORKINGDIR):
os.makedirs(WORKINGDIR)
For touch:
import os
def touch(fname, times=None):
with open(fname, 'a'):
os.utime(fname, times)
touch(WORKINGDIR+ '/'+'results.csv')
I've got a bash file that I normally execute using Cygwin.
I need to run this file from my Python code.
I tried this:
for bashfile in files:
p = Popen(bashfile, cwd=dname) #dname is the current directory of the script
stdout, stderr = p.communicate()
I've also seen a similar question here, but when trying to run it that way it says that it can't find the directory of my bash file...
Any ideas? Thanks! :-)
Edit: bashfile has a full path.
Do you need its output to get it directly to Python? If not this may be very fast and easy solution:
os.system("""here some code you use to execute in Terminal""")
You can also try this, though it does (and will no matter what you try) matter where the directory is. This, as far as the output goes, may be a little bit cleaner than the os method.
import commands
cmd="bash ./script.sh"
commands.getoutput(cmd)
If the case is that you need to change the directory:
cmd = "/path/to/your/script/script.sh"
The added benefit of using this method, versus say, os is that you can assign the output to a variable...
fun_times = commands.getoutput("bash ./script.sh")
whereas...
not_fun_times = os.system("./script.sh")
will throw an error.
etc, etc.
I need to create a folder that I use only once, but need to have it exist until the next run. It seems like I should be using the tmp_file module in the standard library, but I'm not sure how to get the behavior that I want.
Currently, I'm doing the following to create the directory:
randName = "temp" + str(random.randint(1000, 9999))
os.makedirs(randName)
And when I want to delete the directory, I just look for a directory with "temp" in it.
This seems like a dirty hack, but I'm not sure of a better way at the moment.
Incidentally, the reason that I need the folder around is that I start a process that uses the folder with the following:
subprocess.Popen([command], shell=True).pid
and then quit my script to let the other process finish the work.
Creating the folder with a 4-digit random number is insecure, and you also need to worry about collisions with other instances of your program.
A much better way is to create the folder using tempfile.mkdtemp, which does exactly what you want (i.e. the folder is not deleted when your script exits). You would then pass the folder name to the second Popen'ed script as an argument, and it would be responsible for deleting it.
What you've suggested is dangerous. You may have race conditions if anyone else is trying to create those directories -- including other instances of your application. Also, deleting anything containing "temp" may result in deleting more than you intended. As others have mentioned, tempfile.mkdtemp is probably the safest way to go. Here is an example of what you've described, including launching a subprocess to use the new directory.
import tempfile
import shutil
import subprocess
d = tempfile.mkdtemp(prefix='tmp')
try:
subprocess.check_call(['/bin/echo', 'Directory:', d])
finally:
shutil.rmtree(d)
"I need to create a folder that I use only once, but need to have it exist until the next run."
"Incidentally, the reason that I need the folder around is that I start a process ..."
Not incidental, at all. Crucial.
It appears you have the following design pattern.
mkdir someDirectory
proc1 -o someDirectory # Write to the directory
proc2 -i someDirectory # Read from the directory
if [ %? == 0 ]
then
rm someDirectory
fi
Is that the kind of thing you'd write at the shell level?
If so, consider breaking your Python application into to several parts.
The parts that do the real work ("proc1" and "proc2")
A Shell which manages the resources and processes; essentially a Python replacement for a bash script.
A temporary file is something that lasts for a single program run.
What you need is not, therefore, a temporary file.
Also, beware of multiple users on a single machine - just deleting anything with the 'temp' pattern could be anti-social, doubly so if the directory is not located securely out of the way.
Also, remember that on some machines, the /tmp file system is rebuilt when the machine reboots.
You can also automatically register an function to completely remove the temporary directory on any exit (with or without error) by doing :
import atexit
import shutil
import tempfile
# create your temporary directory
d = tempfile.mkdtemp()
# suppress it when python will be closed
atexit.register(lambda: shutil.rmtree(d))
# do your stuff...
subprocess.Popen([command], shell=True).pid
tempfile is just fine, but to be on a safe side you'd need to safe a directory name somewhere until the next run, for example pickle it. then read it in the next run and delete directory. and you are not required to have /tmp for the root, tempfile.mkdtemp has an optional dir parameter for that. by and large, though, it won't be different from what you're doing at the moment.
The best way of creating the temporary file name is either using tempName.TemporaryFile(mode='w+b', suffix='.tmp', prifix='someRandomNumber' dir=None)
or u can use mktemp() function.
The mktemp() function will not actually create any file, but will provide a unique filename (actually does not contain PID).