I have created a python program, see below. It moves files to AWS S3. It it is expecting input. It works well with pycharm but when I call the python script from command prompt, it doesn't take the input value. Here is my code:
import os
import sys
import boto3
from botocore.client import Config
import configparser
import re
import os.path
## Initialize the Parameters
def initconfig(input):
config = configparser.ConfigParser()
config.read_file(open( 'CONFIG_AIRBILLING.conf'))
print('Code Name is :'+ input)
global REMOTE_DIR,ACCESS_KEY_ID,ACCESS_SECRET_KEY,BUCKET_NAME,TARGET_DIR,FILENAME,SRC_DIR,data,File
ACCESS_KEY_ID = config.get('ACCESS', 'ACCESS_KEY_ID')
print('ACCESS_ID_IS:'+ ACCESS_KEY_ID)
ACCESS_SECRET_KEY = config.get('ACCESS', 'ACCESS_SECRET_KEY')
BUCKET_NAME = config.get('ACCESS', 'BUCKET_NAME')
SRC_DIR = config.get(input, 'SRC_DIR')
FILENAME = config.get(input, 'FILENAME')
#LOC="C:\test\demo.txt"
TARGET_DIR = config.get(input, 'TARGET_DIR')
File='demo.txt'
#data = open(File, 'rb') ## This is the filename, need to change it
## This function will make sure file exist in Source directory
def readstatus():
try:
with open(File,'r') as f:
f.closed
result='True'
movefiles(result)
except (Exception,FileNotFoundError) as e:
print('***Error:File Not Found or Accessible***')
result='False*'
raise e
## This function will move the files to AWS S3 bucket
def movefiles(result):
if result=='True':
s3 = boto3.resource(
's3',
aws_access_key_id=ACCESS_KEY_ID,
aws_secret_access_key=ACCESS_SECRET_KEY,
config=Config(signature_version='s3v4')
)
s3.Bucket(BUCKET_NAME).put_object(Key=TARGET_DIR + '/' + File, Body=File)
print('***File Moved***')
print("Done")
initconfig("ABC")
readstatus()
The code above runs fine with pycharm because I can change the value of initconfig function value. But when I run this through command prompt, it doesn't take the parameter values I pass. Here is how I am passing the value, Please help me to fix this.
From Command Prompt
python move_files_to_S3 "TEST"
You want sys.argv, which is a list with all parameters passed through the command line (but notice sys.argv[0] is the name of the script itself).
You'd also do good to check for __name__ == '__main__' to distinguish between when your code is called from the python interpreter through the command line (as in your example at the end of your post) and when it is imported from another module:
if __name__ == '__main__':
initconfig(sys.argv[1])
readstatus()
Related
I have a program that sends some data to GCS in a while loop, which requires a JSON credential file, it roughly looks like the following:
import os
import sys
from pathlib import Path
GOOGLE_SERVICE_ACCOUNT_FP = "pos-service-account.json"
IS_INSTALLER = getattr(sys, "frozen", False) and hasattr(sys, "_MEIPASS")
if IS_INSTALLER:
GOOGLE_SERVICE_ACCOUNT_FP = os.path.join(sys._MEIPASS, GOOGLE_SERVICE_ACCOUNT_FP)
def send_data(data, credential_fp: Path = Path(GOOGLE_SERVICE_ACCOUNT_FP)):
if not credential_fp.is_file():
msg = f"Google service account key json file: {str(credential_fp)!r} is not found!\nPWD files:\n{list(Path.cwd().glob('*'))}"
raise FileNotFoundError(msg)
# some function post data to cloud storage
post_to_gcs(data)
def main():
while True:
data = ...
send_data(data)
if __name__ == '__main__':
main()
I packaged using one file with the following command:
pyinstaller --onefile -w .\main.py --add-data 'pos-service-account.json;.'
Then when I clicked on the .exe file on windows, it was working fine, I could see the data posted by this program. However, I came back after a few days, I got the file not found error:
Google service account key json file: 'C:\\Users\\POS\\AppData\\Local\\Temp\\_MEI30522\\pos-service-account.json' is not found!
this does not make sense to me as the program was working at the beginning, which means it did find the json file, I'm still trying to replicate the error, but so far my suspicion is:
Does the sys._MEIPASS directory ever change? e.g. if the computer goes to sleep and comes back on (I'm going to test it), I could see how my script would fail if the _MEIPASS changes when the program is running.
Yes the MEIPASS folder changes it's name often, so hardcoding that file location is a bad idea. Instead save the file relative to the __file__ path variable so it will always be referring to the currently running programs location.
import os
import sys
from pathlib import Path
GOOGLE_SERVICE_ACCOUNT_FP = "pos-service-account.json"
CREDENTIAL = os.path.join(os.path.dirname(os.path.abspath(__file__)), GOOGLE_SERVICES_ACCOUNT_FP)
def send_data(data, credential_fp=CREDENTIAL):
if not credential_fp.is_file():
msg = f"Google service account key json file: {str(credential_fp)!r} is not found!\nPWD files:\n{list(Path.cwd().glob('*'))}"
raise FileNotFoundError(msg)
# some function post data to cloud storage
post_to_gcs(data)
def main():
while True:
data = ...
send_data(data)
if __name__ == '__main__':
main()
I have written a program that utilizes "InquirerPy" to create and validate a series of prompts for our users.
Problem is I realized that these prompts need to be unit tested in our Continuous Integration setup.
Example:
from InquirerPy import inquirer
def set_path():
config_path = inquirer.filepath(default="C:\\", message="Enter path to config folder", only_directories=True).execute()
path_dict = {"path":config_path}
if not os.path.exists(config_path):
print(config_path)
os.makedirs(config_path)
# add path to json
path = os.path.join(dir_path, "path.json")
with open(path, "w+") as f:
json.dump(path_dict, f)
Now how do I pass an argument "autonomously" in a CI setup using pytest?
An example would be a unittest that reads in the path from "path.json" and checks that it is equal to the "homepath" variable. Something like this (sort of pseudo)
import json
def test_set_path():
path = "C:\\config"
set_path() <- where we somehow pass in the path variable
with open(path, 'r') as f:
data = json.load(f)
json_path = data["path"]
assert path == json_path
If i write this test, how do I pass the homepath variable into the InquirerPy prompt properly?
I'm having a difficult time trying to run a python file within another python file as a module. The program I am trying to run inside the other python file works fine on its own. However, when I import it as a module it does not do anything and does not even give me an error code.
This is the first code. The file name is speech2text.py
def mainprogram():
import os
import sys
os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = r"C:\Users\taiki\PycharmProjects\startup\stelarvision-280712-c709366612cc.json"
filepath = "./"
output_filepath = "./"
bucketname = "stelarvision2020"
sys.path.append("/users/taiki/appdata/local/packages/pythonsoftwarefoundation.python.3.8_qbz5n2kfra8p0/localcache/local-packages/python38/site-packages")
sys.path.append("/Users/taiki/AppData/Local/Programs/Python/Python38-32/Lib/site-packages")
from pydub import AudioSegment
from google.cloud import speech
from google.cloud.speech import enums
from google.cloud.speech import types
import wave
from google.cloud import storage
def stereo_to_mono(audio_file_name):
sound = AudioSegment.from_wav(audio_file_name)
sound = sound.set_channels(1)
sound.export(audio_file_name, format="wav")
def frame_rate_channel(audio_file_name):
with wave.open(audio_file_name, "rb") as wave_file:
frame_rate = wave_file.getframerate()
channels = wave_file.getnchannels()
return frame_rate, channels
def upload_blob(bucket_name, source_file_name, destination_blob_name):
storage_client = storage.Client()
bucket = storage_client.get_bucket(bucket_name)
blob = bucket.blob(destination_blob_name)
blob.upload_from_filename(source_file_name)
def delete_blob(bucket_name, blob_name):
storage_client = storage.Client()
bucket = storage_client.get_bucket(bucket_name)
blob = bucket.blob(blob_name)
blob.delete()
def google_transcribe(audio_file_name):
file_name = filepath + audio_file_name
frame_rate, channels = frame_rate_channel(file_name)
if channels > 1:
stereo_to_mono(file_name)
bucket_name = bucketname
source_file_name = filepath + audio_file_name
destination_blob_name = audio_file_name
upload_blob(bucket_name, source_file_name, destination_blob_name)
gcs_uri = 'gs://' + bucketname + '/' + audio_file_name
transcript = ''
client = speech.SpeechClient()
audio = types.RecognitionAudio(uri=gcs_uri)
config = types.RecognitionConfig(
encoding=enums.RecognitionConfig.AudioEncoding.LINEAR16,
sample_rate_hertz=frame_rate,
language_code='en-US',
enable_automatic_punctuation=True)
operation = client.long_running_recognize(config, audio)
response = operation.result(timeout=10000)
for result in response.results:
transcript += result.alternatives[0].transcript
delete_blob(bucket_name, destination_blob_name)
return transcript
def write_transcripts(transcript_filename, transcript):
f = open(output_filepath + transcript_filename, "w+")
f.write(transcript)
f.close()
if __name__ == "__main__":
audio_file_name = "sample_music.wav"
transcript = google_transcribe(audio_file_name)
transcript_filename = audio_file_name.split('.')[0] + '.txt'
write_transcripts(transcript_filename, transcript)
mainprogram()
This is the second python file
def run():
import speech2text
speech2text.mainprogram()
run()
When I execute the second file it doesn't do anything..
The part if __name__ == "__main__": is only executed when you execute that script directly, i.e. python speech2text.py.
To make your program work, delete that if-statement.
The structure of your python files is a little bit strange. It's better to import sys and import os at the head of the file rather than import something in a class.
Also, your indent has problem. In my opinion, you'd better download a nice IDE like pycharm to automatically check your syntax errors.
First, you should put the import outside the function, as it would be more conventional. You should also check if two files are in the same folder.
Second, this part will not work if you call it outside
if __name__ == "__main__":
audio_file_name = "sample_music.wav"
transcript = google_transcribe(audio_file_name)
transcript_filename = audio_file_name.split('.')[0] + '.txt'
write_transcripts(transcript_filename, transcript)
this one is used to indicate that don't automatically run when got called from another program, which basically you are doing now.
Third, the last line mainprogram() in the first file, not sure what you mean here. You have already have a command to run a function in the second program speech2text.mainprogram() after importing.
Maybe you only have done wrong in one part not all 3 that I am suggesting, so please do try and test!
Suppose you second py file name is two and first one name is one
now to run two in one try this
add this line in one import two
and to run their function two.run()
and your two should be like this.
Two learn basic Python visit my youtube channle TechieBaar
I have written a simple python script to hash a file and output the result. However, when I run the script (python scriptname.py), I don't get any output (expected it to print the checksum). I don't get any errors from the console either.
What am I doing wrong?
#!/usr/bin/env python
import hashlib
import sys
def sha256_checksum(filename, block_size=65536):
sha256 = hashlib.sha256()
filename = '/Desktop/testfile.txt'
with open(filename, 'rb') as f:
for block in iter(lambda: f.read(block_size), b''):
sha256.update(block)
return sha256.hexdigest()
def main():
for f in sys.argv[1:]:
checksum = sha256_checksum(f)
print(f + '\t' + checksum)
if __name__ == '__main__':
main()
def main():
for f in sys.argv[1:]:
The script expected arguments. If you run it without any arguments you don't see any ouput.
The main body suppose that you provide list of files for hashing but in hashing function you hardcoded
filename = '/Desktop/testfile.txt'
So, if you want to pass files for hashing as script arguments remove the line
filename = '/Desktop/testfile.txt'
and run
python scriptname.py '/Desktop/testfile.txt'
I am having trouble creating and writing to a text file in Python. I am running Python 3.5.1 and have the following code to try and create and write to a file:
from os import *
custom_path = "MyDirectory/"
if not path.exists(custom_path)
mkdir(custom_path)
text_path = custom_path + "MyTextFile.txt"
text_file = open(text_path, "w")
text_file.write("my text")
But I get a TypeError saying an integer is required (got type str) at the line text_file = open(text_path, "w").
I don't know what I'm doing wrong as my code is just about identical to that of several tutorial sites showing how to create and write to files.
Also, does the above code create the text file if it doesn't exist, and if not how do I create it?
Please don't import everything from os module:
from os import path, mkdir
custom_path = "MyDirectory/"
if not path.exists(custom_path):
mkdir(custom_path)
text_path = custom_path + "MyTextFile.txt"
text_file = open(text_path, 'w')
text_file.write("my text")
Because there also a "open" method in os module which will overwrite the native file "open" method.