We Keep Coding

Is there a way to check if script is running from subprocess?

Let's say I have a python script which reads all the images in a folder and resizes them. The script works all by his own, it takes in two arguments - the input folder and an output folder.
To have a visual response of the progress I'm using a progressbar which is printed out to the console/terminal.
resize.py:
import argparse
import fnmatch
import os
import PIL
from PIL import Image
from progress.bar import Bar
parser = argparse.ArgumentParser(description='Photo resizer.')
parser.add_argument('input_folder', nargs='?', default='', help="Input folder")
parser.add_argument('export_folder', nargs='?', default='', help="Output folder")
args = parser.parse_args()
if args.input_folder:
input_folder = args.input_folder
if args.export_folder:
export_resized_folder = args.export_folder
NEW_SIZE = 2000
inputFiles = []
for root, dirnames, filenames in os.walk(input_folder):
for filename in fnmatch.filter(filenames, '*.jpg'):
inputFiles.append(os.path.join(root, filename))
bar = Bar("Processing photos", max=len(inputFiles), check_tty=False)
for photo in inputFiles:
filename = os.path.basename(photo)
im = Image.open(photo)
im_width, im_height = im.size
if im_width > im_height:
new_width = NEW_SIZE
new_height = int(NEW_SIZE * im_height / im_width)
else:
new_height = NEW_SIZE
new_width = int(NEW_SIZE * im_width / im_height)
new_size = (new_width, new_height)
im_resized = im.resize(new_size, resample=PIL.Image.Resampling.LANCZOS)
im_resized.save(os.path.join(export_resized_folder, filename), quality=70)
bar.next()
bar.finish()
Now I have an another script (main_gui.py) which does some batch processing and one of the jobs is to resize the images. This script provides a simple GUI. When it comes to resizing the images, I use subprocess Popen to execute the script and pass in the input and output folders as args.
So in main_gui.py I start the subprocess:
script_path = "resize.py"
process = subprocess.Popen(["python", script_path, INPUT_FOLDER, OUTPUT_FOLDER], universal_newlines=True, stdout=subprocess.PIPE)
Now I'd like to see the progress in the GUI also. I don't know if I'm doing it correctly (It is a high probability that not, this is just the first thing that came to my mind)...
So in resize.py along with the progressbar I print out information about my progress and then read it in the main_gui.py and based on that information I update a tkinter progressbar.
In resize.py:
bar = Bar("Processing photos", max=len(inputFiles), check_tty=False)
print("**TOTAL** " + str(len(inputFiles)))
...
progressCounter = 1
for photo in inputFiles:
...
bar.next()
print("**PROGRESS** " + str(progressCounter))
progressCounter += 1
...
I read these values in main_gui.py
process = subprocess.Popen(["python", script_path], universal_newlines=True, stdout=subprocess.PIPE)
while process.poll() is None:
data = process.stdout.readline().strip()
print(data)
if "**TOTAL** " in data:
total = int(data.replace("**TOTAL** ", ""))
progressbarWidget['maximum'] = total
if "**PROGRESS** " in data and self.GUI:
progressCounter = int(data.replace("**PROGRESS** ", ""))
progressbarWidget['value'] = progressCounter
progressbarWidget.update_idletasks()
And at this point I'd like in my resize.py check if it is run by itself or by the subprocess, so I don't have the unnecessary print statements.
I tried pass in an env value as Charles suggested in the comments, but couldn't get it done

Trying to detect your parent process is an unnecessary amount of magic for this use case. Making it explicit with an optional argument will let others writing their own GUIs (potentially in non-Python languages) get the machine-readable status output without needing to try to fool the detection.
parser = argparse.ArgumentParser(description='Photo resizer.')
parser.add_argument('--progress', choices=('none', 'human', 'machine-readable'), default='none',
help="Should a progress bar be written to stderr in a human-readable form, to stdout in a machine-readable form, or not at all?")
parser.add_argument('input_folder', nargs='?', default='', help="Input folder")
parser.add_argument('export_folder', nargs='?', default='', help="Output folder")
args = parser.parse_args()
...and then later...
if args.progress == 'machine-readable':
pass # TODO: Write your progress messages for the programmatic consumer to stdout here
elif args.progress == 'human':
pass # TODO: Write your progress bar for a human reader to stderr here
while on the GUI side, adding --progress=human to the argument list:
process = subprocess.Popen([sys.executable, script_path, '--progress=human'],
universal_newlines=True, stdout=subprocess.PIPE)

Where does transformers save microsoft/DialoGPT-small?

when I run the code
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
SML=input("small,medium or large: ").lower()#I pick small
model_name = "microsoft/DialoGPT-"+SML
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)
It downloads microsoft/DialoGPT-small from somewhere on the internet and downloads it to my computer but I don`t know where the saved file is. Anyone have any idea where this file is saved?
#EDIT#
I found the below code in \Lib\site-packages\transformers\models\dialogpt\convert_dialogpt_original_pytorch_checkpoint_to_pytorch.py but have no idea what it means.
def convert_dialogpt_checkpoint(checkpoint_path: str, pytorch_dump_folder_path: str):
d = torch.load(checkpoint_path)
d[NEW_KEY] = d.pop(OLD_KEY)
os.makedirs(pytorch_dump_folder_path, exist_ok=True)
torch.save(d, os.path.join(pytorch_dump_folder_path, WEIGHTS_NAME))
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument("--dialogpt_path", default=".", type=str)
args = parser.parse_args()
for MODEL in DIALOGPT_MODELS:
checkpoint_path = os.path.join(args.dialogpt_path, f"{MODEL}_ft.pkl")
pytorch_dump_folder_path = f"./DialoGPT-{MODEL}"
convert_dialogpt_checkpoint(
checkpoint_path,
pytorch_dump_folder_path,
)

Python script acts differently when ran in console and when ran from other file

I have two python files, when I run "python downloader.py todownload" it downloads images from todownload.txt file, but when I run it indirectly (calling function of downloader.py via other python file) I don't get the images downloaded.
I tried debugging and printing what arguments are sent to function "download_all_images(args)" when I call it both ways, and the object args match.
downloader.py
import argparse
from concurrent import futures
import os
import re
import sys
import boto3
import botocore
import tqdm
BUCKET_NAME = 'open-images-dataset'
REGEX = r'(test|train|validation|challenge2018)/([a-fA-F0-9]*)'
def check_and_homogenize_one_image(image):
split, image_id = re.match(REGEX, image).groups()
yield split, image_id
def check_and_homogenize_image_list(image_list):
for line_number, image in enumerate(image_list):
try:
yield from check_and_homogenize_one_image(image)
except (ValueError, AttributeError):
raise ValueError(
f'ERROR in line {line_number} of the image list. The following image '
f'string is not recognized: "{image}".')
def read_image_list_file(image_list_file):
with open(image_list_file, 'r') as f:
for line in f:
yield line.strip().replace('.jpg', '')
def download_one_image(bucket, split, image_id, download_folder):
try:
bucket.download_file(f'{split}/{image_id}.jpg',
os.path.join(download_folder, f'{image_id}.jpg'))
except botocore.exceptions.ClientError as exception:
sys.exit(
f'ERROR when downloading image `{split}/{image_id}`: {str(exception)}')
def download_all_images(args):
print(type(args))
print(args)
"""Downloads all images specified in the input file."""
bucket = boto3.resource(
's3', config=botocore.config.Config(
signature_version=botocore.UNSIGNED)).Bucket(BUCKET_NAME)
download_folder = args['download_folder'] or os.getcwd()
if not os.path.exists(download_folder):
os.makedirs(download_folder)
try:
image_list = list(
check_and_homogenize_image_list(
read_image_list_file(args['image_list'])))
except ValueError as exception:
sys.exit(exception)
progress_bar = tqdm.tqdm(
total=len(image_list), desc='Downloading images', leave=True)
with futures.ThreadPoolExecutor(
max_workers=args['num_processes']) as executor:
all_futures = [
executor.submit(download_one_image, bucket, split, image_id,
download_folder) for (split, image_id) in image_list
]
for future in futures.as_completed(all_futures):
future.result()
progress_bar.update(1)
progress_bar.close()
if __name__ == '__main__':
parser = argparse.ArgumentParser(
description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
parser.add_argument(
'image_list',
type=str,
default=None,
help=('Filename that contains the split + image IDs of the images to '
'download. Check the document'))
parser.add_argument(
'--num_processes',
type=int,
default=5,
help='Number of parallel processes to use (default is 5).')
parser.add_argument(
'--download_folder',
type=str,
default=None,
help='Folder where to download the images.')
download_all_images(vars(parser.parse_args()))
imagegatherer.py
import pandas as pd
import argparse
from concurrent import futures
import os
import re
import sys
import boto3
import botocore
import tqdm
import downloader
labels = pd.read_csv('class-descriptions-boxable.csv', header=None)
labels.columns = ['id', 'name']
classified = pd.read_csv('train-annotations-human-imagelabels-boxable.csv', nrows=2500)
classes = ['Apple', 'Motorcycle', 'Snowman']
ids = labels[labels['name'].isin(classes)]
for entry in ids.iterrows():
id = entry[1][0]
name = entry[1][1]
good = classified.loc[classified['LabelName'] == id]['ImageID'].tolist()
with open('todownload.txt', 'w') as file:
for img in good:
file.write('train/' + img + '\n')
args = dict()
args['image_list'] = 'todownload.txt'
args['num_processes'] = 5
args['download_folder'] = None
downloader.download_all_images(args)
todownload.txt content that should download 4 images:
train/0000048549557964
train/000023aa04ab09ed
train/00002f4ff380c64c
train/000037c2dd414b46

Append to file succeeding but calling error from another def

When I specify the -a switch to add to config file for some reason this calls the read_config file also. For example I use ./script.py -a newfile.txt and the file is added correctly but then returns "Config File not found".
parser = argparse.ArgumentParser(description='Copy multiple Files from a specified data file')
parser.add_argument('-c', '--configfile', default="config.dat", help='file to read the config from')
parser.add_argument('-l', '--location', default="/home/admin/Documents/backup/",help='Choose location to store files')
parser.add_argument('-a', '--add', help='add new line to config data')
def read_config(data):
try:
dest = '/home/admin/Documents/backup/'
# Read in date from config.dat
data = open(data)
# Interate through list of files '\n'
filelist = data.read().split('\n')
# Copy through interated list and strip white spaces and empty lines
for file in filelist:
if file:
shutil.copy(file.strip(), dest)
except FileNotFoundError:
logger.error("Config file not found")
print ("Config File not found")
Not quite sure why this section below is working but read_config is called and finding the except error. Where am I going wrong?
def addtofile(add):
f = open('config.dat', "a")
f.write(add + '\n')
f.close()
args = vars(parser.parse_args())
read = read_config(args['configfile'])
add = addtofile(args['add'])

parser = argparse.ArgumentParser()
parser.add_argument('-c', action='store_true')
a = parser.parse_args()
if a.c:
{
print("get custom config")
}
else:
{
print("using default config file")
}

I think your solution will be:
parser = argparse.ArgumentParser()
parser.add_argument('-c', action='store_true')
a = parser.parse_args()
if a.c:
{
print("get custom config")
}
else:
{
print("using default config file")
}

Using argparse to create output file

I have been using argparse in a program I am writing however it doesnt seem to create the stated output file.
My code is:
parser.add_argument("-o", "--output", action='store', dest='output', help="Directs the output to a name of your choice")
with open(output, 'w') as output_file:
output_file.write("%s\n" % item)
I have also tried:
parser.add_argument("-o", "--output", action='store', type=argparse.FileType('w'), dest='output', help="Directs the output to a name of your choice")
output_file.write("%s\n" % item)
The error that occurs is :
output_file.write("%s\n" % item)
NameError: name 'output_file' is not defined
Can someone please explain why I am having this error occuring and how I could solve it?
All my code:
from __future__ import print_function
from collections import defaultdict
from itertools import groupby
import argparse #imports the argparse module so it can be used
from itertools import izip
#print = print_function
parser = argparse.ArgumentParser() #simplifys the wording of using argparse as stated in the python tutorial
parser.add_argument("-r1", type=str, action='store', dest='input1', help="input the forward read file") # allows input of the forward read
parser.add_argument("-r2", type=str, action='store', dest='input2', help="input the reverse read file") # allows input of the reverse read
parser.add_argument("-v", "--verbose", action="store_true", help=" Increases the output, only needs to be used to provide feedback to Tom for debugging")
parser.add_argument("-n", action="count", default=0, help="Allows for up to 5 mismatches, however this will reduce accuracy of matching and cause mismatches. Default is 0")
#parser.add_argument("-o", "--output", action='store', type=argparse.FileType('w'), dest='output', help="Directs the output to a name of your choice")
parser.add_argument("-fastq", action="store_true", help=" States your input as fastq format")
parser.add_argument("-fasta", action="store_true", help=" States your input as fasta format")
parser.add_argument("-o", "--output", action='store', dest='output', help="Directs the output to a name of your choice")
args = parser.parse_args()
def class_chars(chrs):
if 'N' in chrs:
return 'unknown'
elif chrs[0] == chrs[1]:
return 'match'
else:
return 'not_match'
with open(output, 'w') as output_file:
s1 = 'aaaaaaaaaaN123bbbbbbbbbbQccc'
s2 = 'aaaaaaaaaaN456bbbbbbbbbbPccc'
n = 0
consec_matches = []
chars = defaultdict(int)
for k, group in groupby(zip(s1, s2), class_chars):
elems = len(list(group))
chars[k] += elems
if k == 'match':
consec_matches.append((n, n+elems-1))
n += elems
print (chars)
print (consec_matches)
print ([x for x in consec_matches if x[1]-x[0] >= 9])
list = [x for x in consec_matches if x[1]-x[0] >= 9]
flatten_list= [x for y in list for x in y]
print (flatten_list)
matching=[y[1] for y in list for x in y if x ==0 ]
print (matching)
magic = lambda matching: int(''.join(str(i) for i in matching)) # Generator exp.
print (magic(matching))
s2_l = s2[magic(matching):]
line3=s1+s2_l
print (line3)
if line3:
output_file.write("%s\n" % item)

You are missing the bit where the arguments are actually parsed:
parser.add_argument("-o", "--output", help="Directs the output to a name of your choice")
args = parser.parse_args()
with open(args.output, 'w') as output_file:
output_file.write("%s\n" % item)
parser.parse_args() will give you an object from which you can access the arguments by name using the long option name bar the dashes.

I think you almost had the most correct answer. The only problem is output_file was not read from the args:
parser.add_argument("-o", "--output", action='store',
type=argparse.FileType('w'), dest='output',
help="Directs the output to a name of your choice")
#output_file is not defined, you want to read args.output to get the output_file
output_file = args.output
#now you can write to it
output_file.write("%s\n" % item)

When I run your script I get:
Traceback (most recent call last):
File "stack23566970.py", line 31, in <module>
with open(output, 'w') as output_file:
NameError: name 'output' is not defined
There's no place in your script that does output = ....
We can correct that with:
with open(args.output, 'w') as output_file:
argparse returns values as attributes of the args object.
Now I get:
Traceback (most recent call last):
File "stack23566970.py", line 62, in <module>
output_file.write("%s\n" % item)
NameError: name 'item' is not defined
Again, there's no item = ... line.
What is item supposed to be?