I need to run several python script's, some of those are scrapy projects.
To run a spider I try this:
from subprocess import call
import subprocess
call(["scrapy",'crawl','my_spider','-o output_file.csv'],cwd='/home/luis/Schreibtisch/kukun/bbb_new_pro/scripts/2_Get_links)
I wonder whether is posible to specify the output's file directory, I tried this:
call(["scrapy",'crawl','my_spider','-o folder_1/folder_2/output_file.csv'],cwd='project_folder')
But that only creates a new folders under the project directory I want the file outside tht folder.
The other thing is, Can I specify the name of the ouput file in a variable? something like:
file_name = 'output file.csv'
call(["scrapy",'crawl','my_spider','-o + file_name '],cwd='project_folder')
This worked for me:
from subprocess import call
name = "spider_name"
call(["scrapy", "crawl", "{0}".format(name), "-o {0}.json".format(name)])
Related
I have two files, one bell.mp3 one main.py file, that plays back bell.mp3 via subprocess.
If I do:
pyinstaller main.py
the Dist file ends up correctly, and everything works fine, The program runs in a directory.
This is the code for my file which i call pyinst_tester.py
it creates a text file, and plays a bell.mp3 file
#
from con import * # this is just a configuration file that has g='play' in it.
import subprocess
f=open(r'/home/godzilla/Desktop/Pyinstaller testing/testfile1','w')
f.write('This has worked')
f.close()
file='/home/godzilla/Desktop/Pyinstaller testing/data/bell.mp3'
if 'play' == g:
subprocess.call(['/usr/bin/cvlc',file])
a single file is created, but if I delete the bell.mp3 file it doesn't work. In a single file isn't the bell.mp3 zipped inside the main.exe ? therefore, redundant as a separate file ?
What Is the point having a single file exe, if you need an adjacent file with all the mp3s inside?
Pyinstaller has many features and if you want to include non python files (for example mp3 files) you have to do so explicitly with the --add-binary switch.
In one file mode the executable will be unpacked into a temporary directory prior to execution of the python code.
So how to write your code to access these data files.
You might want to look at the pyinstaller documention at following sections:
https://pyinstaller.readthedocs.io/en/stable/runtime-information.html#run-time-information
https://pyinstaller.readthedocs.io/en/stable/runtime-information.html#using-sys-executable-and-sys-argv-0
I personally place all my files in a separate directory. e.g. data.
If you place the file bell.mp3 in the directory data, then you had to call pyinstaller with the option --add-binary data:data
in the one file mode the executable is extracted into a temporary directory
whose path you get get from the variable sys._MEIPASS
Your data directory will bi in the sub directory data of sys._MEIPASS
In my example I create a function, that will be able to locate the data files in normal python mode and in pyinstaller one file or one directory mode.
Just try it out it should be self explaining
simple example:
minitst.py
import os, sys
import time
is_frozen = getattr(sys, "frozen", False)
MYDIR = os.path.realpath(os.path.dirname(__file__))
def data_fname(fname):
if is_frozen:
return os.path.join(sys._MEIPASS, "data", fname)
else:
return os.path.join(MYDIR, "data", fname)
def main():
print("This application is %s frozen" %
("" if is_frozen else "not"))
print("executable =", sys.executable,
"File =", __file__,
"mydir =", MYDIR)
if is_frozen:
print("MEIPASS", sys._MEIPASS)
fname = data_fname("tst.txt")
print("will open", fname)
with open(fname) as fin:
print(fin.read())
time.sleep(5) # this shall allow to view the console e.g. on windows if clicking on the executable.
if __name__ == "__main__":
main()
now create a directory data and place a file "tst.txt"
data/tst.txt
Hello world
Now call
pyinstaller -F minitst.py --add-binary data:data -c
and call dist/minitst from a console.
The output should look like:
This application is frozen
executable = /home/gelonida/so/pyinst/dist/minitst File = minitst.py mydir = /home/gelonida/so/pyinst
MEIPASS /tmp/_MEIKGqah9
will open /tmp/_MEIKGqah9/data/tst.txt
Hello
Now concerning your code.
I compacted the code to determine the datadir a little, but it is the same logic as in the upper example
import os, sys
from con import * # this is just a configuration file that has g='play' in it.
import subprocess
basedir = getattr(sys, "_MEIPASS", os.path.realpath(os.path.dirname(__file__)))
f=open('testfile1','w')
f.write('This has worked')
f.close()
file=os.path.join(basedir, 'data/bell.mp3')
if 'play' == g:
subprocess.call(['/usr/bin/cvlc',file])
My Python app contains a subfolder called Tests which I use to run unit tests. All of my files are in the parent folder, which I will call App. The Tests folder contains, say, a test.py file. The App folder contains an app.py file and a file.txt text file.
In my test.py file, I can make my imports like this:
import sys
sys.path.append("PATH_TO_PARENT_DIR")
Say my app.py file contains the following:
class Stuff():
def do_stuff():
with open("file.txt") as f:
pass
Now if I run test.py, I get the following error:
FileNotFoundError: [Errno 2] No such file or directory: 'file.txt'
How can I fix this? Many thanks!
Assuming the file is located in the same folder as your script:
import os
parent_dir = os.path.abspath(os.path.dirname(__file__))
class Stuff():
def do_stuff():
with open(os.path.join(parent_dir, "file.txt")) as f:
pass
Explanation:
__file__ is the path to your script
os.path.dirname get's the directory in which your script sits
os.path.abspath makes that path absolute instead of relative (just in case relative paths mess your script up, it's good practice)
Then all we need to do is combine your parent_dir with the file, we do that using os.path.join.
Read the docs on os.path methods here: https://docs.python.org/3/library/os.path.html
A more explicit version of this code can be written like this, if that helps:
import os
script_path = __file__
parent_dir = os.path.dirname(script_path)
parent_dir_absolute = os.path.abspath(parent_dir)
path_to_txt = os.path.join(parent_dir_absolute, 'file.txt')
The open function looks for the file in the same folder as the script that calls the open function. So, your test.py looks in the tests folder, not the app folder. You need to add the full path to the file.
open('app_folder' + 'text.txt')
or move the test.py file in the same folder as text.txt
I have the following python code which tries to copy a file stored in a variable file from a variable directory dir. I also attempting get those variables passed as shell variables, then I can copy using shell command as well.
Is any/both of these possible? Then please amend my code.
import os
import shutil
import subprocess
dir='somedir'
file='somefile'
# Now I want to get $dir as a shell variable and copy the file to
# my present directory
os.system("echo $dir")
os.system("cp $dir/$file .")
# I want to copy dir/file to the present directory
shutil.copy(dir/file,file) # Following shutil.copy(src,dest) format
shutil.copyfile(dir/file,file)
The
You can do it simply in Python itself like below:-
import os
import shutil
dir = 'somedir'
file = 'somefile'
file_to_copy = dir + "/" + file
current_location = os.getcwd()
shutil.copy(file_to_copy, current_location)
Now your file somefile in location somedir will be copied to your current directory location.
Example:
I have my main file, main.py with this code:
import os
os.popen("start folder/subfile.py")
Then i have my other file, subfile.py with this code:
file = open("test.txt", "w")
file.close()
I want my subfile.py to create the test.txt file in its own folder, but it creates it in the main.py's folder.
So my question is, how do i make the subfile.py run from it own folder even though it's started from main.py
main.py folder : C:/users/user/Desktop
subfile.py folder: C:/users/user/Desktop/folder
In subfile.py, change the working directory (os.chdir) to the directory that contains the subfile.py file:
import os
os.chdir(os.path.dirname(__file__))
You can use subprocess instead which has that built in:
subprocess.Popen(sys.executable + ' subfile.py', cwd=os.path.dirname(__file__) + '/folder')
Try This :
import os
import inspect
your_current_folder_path = os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe())))
your_current_file_path = os.path.join(your_current_folder_path, "test.txt")
print(your_current_folder_path)
print(your_current_file_path)
with open(your_current_file_path, "w") as make_file:
make_file.write("")
print("Done.")
Another way: you can change your working directory by os.chdir("your_folder_directory") function.
os.chdir(r"C:/users/user/Desktop/folder")
I have a file ref.py that depends on a text file ex.txt, that is in the same directory \home\ref_dir . So it works normally when I run the file ref.py, but if I try to import ref.py to another file work.py in a different directory \home\work_dir , I do the following
import sys
sys.path.append('\home\ref_dir')
import ref
But then I get an error, that the program cannot find ex.txt
How can I solve this issue without using absolute paths. Any ideas?
Use the os module to get access to the current absolute path of the module that you're in, and then get the dirname from that
You would want to open ex.txt in your file like this.
import os
with open('%s/ex.txt' % os.path.dirname(os.path.abspath(__file__)) as ex:
print ex.read()