So my question is how do I run a PowerShell script in python, with passable variables from python, then have the python run the PowerShell script, and then from there have specific variables from the PowerShell script pass to the python script.
What I have already is each code working separately.
As you can see I am passing in hidden variables from a .ini file and from there I am trying to pass the compList to Powershell.
I am not confident that I am running the command right to call PowerShell and pass the args in my python script
preferences = configparser.ConfigParser()
preferences.read('config.ini')
hostName = preferences.get('Directory', 'hostName')
port = preferences.get('Directory', 'port')
serviceName = preferences.get('Directory', 'serviceName')
usr = preferences.get('Credentials', 'userName')
pswrd = preferences.get('Credentials', 'password')
compList = preferences.get('ComputerList', 'compList')
compList = list(compList.split(","))
p = subprocess.Popen(["powershell.exe", r"C:\Users\pecorgx\TestPS1.ps1", 'arg1 #("compList")'], stdout=sys.stdout)
p.communicate()
My PowerShell Script:
$ArrComputers = $arg1
Clear-Host
foreach ($Computer in $ArrComputers)
{
$computerSystem = get-wmiobject Win32_ComputerSystem -Computer $Computer
$computerBIOS = get-wmiobject Win32_BIOS -Computer $Computer
$computerOS = get-wmiobject Win32_OperatingSystem -Computer $Computer
$computerCPU = get-wmiobject Win32_Processor -Computer $Computer
$hdds = Get-WmiObject Win32_LogicalDisk -ComputerName $Computer -Filter drivetype=3
write-host "System Information for: " $computerSystem.Name -BackgroundColor DarkCyan
"-------------------------------------------------------"
"Manufacturer: " + $computerSystem.Manufacturer
"Model: " + $computerSystem.Model
"Serial Number: " + $computerBIOS.SerialNumber
"CPU: " + $computerCPU.Name
foreach ($computerhdd in $hdds) { "HDD Drive Letter: " + $computerhdd.DeviceID + "HDD Capacity:" + "{0:N2}" -f ($computerHDD.Size/1GB) + "GB" + " HDD Space: " + "{0:P2}" -f ($computerHDD.FreeSpace/$computerHDD.Size) }
"RAM: " + "{0:N2}" -f ($computerSystem.TotalPhysicalMemory/1GB) + "GB"
"Operating System: " + $computerOS.caption + ", Service Pack: " + $computerOS.ServicePackMajorVersion
"User logged In: " + $computerSystem.UserName
"Last Reboot: " + $computerOS.ConvertToDateTime($computerOS.LastBootUpTime)
""
"-------------------------------------------------------"
}
I think I am passing this correctly but I want the Powershell to pass back to my python script
$computerSystem.TotalPhysicalMemory/1GB
$computerSystem.Manufacturer
$computerSystem.Model
$computerBIOS.SerialNumber
$computerCPU.Name
and I want this var $computerHDD.Size/1GB to be passed into a list to be also passed to my python script
EDIT: I decided to just use .txt to pass commands. I made $myOBJ = "" | Select "computer" then made $myOBJ.computer = [string](....) with (...) being things I want to pass back out after the script has ran
Would love to know a solution to use without having a third party help
Here is simplified example of your use case:
import subprocess;
process=subprocess.Popen(["powershell","Get-Childitem C:\\Windows\\*.log"],stdout=subprocess.PIPE);
result=process.communicate()[0]
print (result)
Of course the above is just printing out a directory list, but you just replace that command with your .ps1 script path.
Also, as noted in my comment, why not just use the ...
# Get specifics for a module, cmdlet, or function
(Get-Command -Name Get-ComputerInfo).Parameters
(Get-Command -Name Get-ComputerInfo).Parameters.Keys
Get-help -Name Get-ComputerInfo -Examples
Get-Help -Name Get-ComputerInfo -Detailed
Get-help -Name Get-ComputerInfo -Full
Get-help -Name Get-ComputerInfo -Online
Get-ComputerInfo -Property '*'
... cmdlet, to collect this info and select the properties you want?
Only running the other collectors for info that Get-ComputerInfo does not provide.
There is also the built-in ...
Get-PhysicalDisk
Get-Volume
... cmdlets to get storage information.
How about constructing your output details this way...
Clear-Host
$env:COMPUTERNAME |
ForEach-Object{
$computerSystem = Get-wmiobject Win32_ComputerSystem -ComputerName $PSItem
$computerBIOS = Get-wmiobject Win32_BIOS -ComputerName $PSItem
$computerOS = Get-wmiobject Win32_OperatingSystem -ComputerName $PSItem
$computerCPU = Get-wmiobject Win32_Processor -ComputerName $PSItem
(
$ComputerInfo = [PSCustomObject]#{
Manufacturer = $computerSystem.Manufacturer
Model = $computerSystem.Model
SerialNumber = $computerBIOS.SerialNumber
CPU = $computerCPU.Name
RAM = [math]::Round($($computerSystem.TotalPhysicalMemory/1GB), 3)
OperatingSystem = "$($computerOS.caption), Service Pack: $($computerOS.ServicePackMajorVersion)"
LoggedOnUser = $env:USERNAME
LastReboot = $computerOS.ConvertToDateTime($computerOS.LastBootUpTime)
}
)
(
$StorageInfo = [PSCustomObject]#{
DeviceID = (Get-Volume).DriveLetter
Size = Get-Volume | ForEach {[math]::Round($($PSitem.Size/1GB), 3)}
SizeRemaining = Get-Volume | ForEach {[math]::Round($($PSitem.SizeRemaining/1GB), 3)}
}
)
}
... or combine and ouput this way
Clear-Host
$env:COMPUTERNAME |
ForEach-Object{
$ComputerSystem = Get-wmiobject Win32_ComputerSystem -ComputerName $PSItem |
Select-Object -Property Manufacturer, Model, UserName
$ComputerBIOS = Get-wmiobject Win32_BIOS -ComputerName $PSItem|
Select-Object -Property SerialNumber
$ComputerOS = Get-wmiobject Win32_OperatingSystem -ComputerName $PSItem |
Select-Object -Property Caption, ServicePackMajorVersion,
#{
Name = 'LastReboot'
Expression = {$PSItem.ConvertToDateTime($PSItem.LastBootUpTime)}
}
$ComputerCPU = Get-wmiobject Win32_Processor -ComputerName $PSItem |
Select-Object -Property Name
$ComputerStorage = Get-WmiObject Win32_LogicalDisk -ComputerName $PSItem -Filter drivetype=3 |
Select-Object -Property DeviceID,
#{
Name = 'Size'
Expression = {[math]::Round($($PSitem.Size/1GB), 3)}
},
#{
Name = 'FreeSpace'
Expression = {[math]::Round($($PSitem.FreeSpace/1GB), 3)}
}
$ComputerDetails = New-Object -Type PSObject
$ComputerSystem, $ComputerBIOS, $ComputerOS, $ComputerCPU, $ComputerStorage |
ForEach-Object {
$CurObj = $PSItem
$PSItem |
Get-Member |
Where-Object {$PSItem.MemberType -match 'NoteProperty'} |
ForEach-Object {
$NewMember = $PSItem.Name
$ComputerDetails |
Add-Member -MemberType NoteProperty -Name $NewMember -Value $CurObj.$NewMember
}
}
}
$ComputerDetails
Related
Please forgive me if this is not the place to ask this question. I'm running python scripts in a Jenkins pipeline from a Jenkinsfile. I am also updating Jira Xray tickets within the Jenkisfile. Behave is being used to validate the test status. If the check fails then the Jenkins build fails without getting the Xray ticket updated with the failure. I've attempted to use "try" to capture the failure but have not succeeded in getting the failure to propagate to the Xray ticket.
Would anyone here know where I might find an answer? I would be in your dept.
Jenkinsfile
node() {
def repoURL = '<GitLab URL>/prod-003.git'
def STC_INSTALL = "/opt/STC_CLIENT/Spirent_TestCenter_5.22/Spirent_TestCenter_Application_Linux/"
try {
stage("Prepare Workspace") {
echo "*** Prepare Workspace ***"
cleanWs()
env.WORKSPACE_LOCAL = sh(returnStdout: true, script: 'pwd').trim()
env.BUILD_TIME = "${BUILD_TIMESTAMP}"
echo "Workspace set to:" + env.WORKSPACE_LOCAL
echo "Build time:" + env.BUILD_TIME
sh """
cd ${env.WORKSPACE_LOCAL}
rm -fr *
"""
}
stage('Checkout Code') {
echo "*** Checking Code Out ***"
git branch: 'master', credentialsId: '', url: repoURL
}
stage('Executing Tests') {
if (env.WanModeCheck == "Yes") {
echo "Executing WAN Mode Change Before FW Upgrade"
sh """
/var/lib/jenkins/.pyenv/shims/python WanMode.py -i $modemIP -m $WanMode
"""
echo "Starting Firmware Upgrade"
sh """
cd ${env.WORKSPACE_LOCAL}
./ModemUpgrade.sh -i $modemIP -f $FW -p2
/var/lib/jenkins/.pyenv/shims/behave -f cucumber -o storetarget-bdd/reporting/cucumber.json --junit --format=json -o target/behave.json --junit ./features/PROD-003.feature
"""
} else {
echo "#######################\n# Skipping WAN Mode Change #\n#######################"
}
if (env.WanModeCheck == "No") {
echo "Starting Firmware Upgrade"
sh """
cd ${env.WORKSPACE_LOCAL}
./ModemUpgrade.sh -i $modemIP -f $FW -p2
/var/lib/jenkins/.pyenv/shims/behave -f cucumber -o storetarget-bdd/reporting/cucumber.json --junit --format=json -o target/behave.json --junit ./features/fwupgrade.feature
"""
}
// Setting variables to use for the Xray Test Execution
res = sh(returnStdout: true, script: 'awk "/##/{f=1;next} /#####/{f=0} f" PROD-003-Out.txt | sed -e "s/#//g" -e "s/^ * //g" | tr "\n" "%" | sed -e "s/^%%%%%%//g" -e "s/%%$//g" -e "s/%/\\\\\\\\Z/g" -e "s/Z/n/g"')
env.STResults = res.strip()
model = sh(returnStdout: true, script: 'grep Model: PROD-003-Out.txt')
env.Model = model.strip()
wanmode = sh(returnStdout: true, script: 'grep CPE PROD-003-Out.txt')
env.WanMode = wanmode.strip()
serialnum = sh(returnStdout: true, script: 'grep Number: PROD-003-Out.txt')
env.SerialNum = serialnum.strip()
echo "End of test phase"
}
stage('Expose report') {
echo "*** Expose Reports ***"
echo "*** Archive Artifacts ***"
archiveArtifacts "**/cucumber.json"
echo "*** cucumber cucumber.json ***"
cucumber '**/cucumber.json'
junit skipPublishingChecks: true, allowEmptyResults: true, keepLongStdio: true, testResults: 'reports/*.xml'
cucumber buildStatus: "UNSTABLE",
fileIncludePattern: "**/cucumber.json",
jsonReportDirectory: 'reports'
}
stage('Import results to Xray') {
echo "*** Import Results to XRAY ***"
def description = "Jenkins Project: ${env.JOB_NAME}\\n\\nCucumber Test Report: [${env.JOB_NAME}-Link|${env.BUILD_URL}/cucumber-html-reports/overview-features.html]\\n\\nJenkins Console Output: [${env.JOB_NAME}-Console-Link|${env.BUILD_URL}/console]\\n\\nCPE IP: ${modemIP}\\n\\nCPE FW File Name: ${FW}\\n\\n${env.STResults}"
def labels = '["regression","automated_regression"]'
def environment = "DEV"
def testExecutionFieldId = 10552
def testEnvironmentFieldName = "customfield_10372"
def projectKey = "AARC"
def projectId = 10608
def xrayConnectorId = "e66d84d8-f978-4af6-9757-93d5804fde1d"
// def xrayConnectorId = "${xrayConnectorId}"
def info = '''{
"fields": {
"project": {
"id": "''' + projectId + '''"
},
"labels":''' + labels + ''',
"description":"''' + description + '''",
"summary": "''' + env.JOB_NAME + ' ' + env.Model + ' ' + env.WanMode + ' ' + env.SerialNum + ''' Test Executed ''' + env.BUILD_TIME + ''' " ,
"issuetype": {
"id": "''' + testExecutionFieldId + '''"
}
}
}'''
echo info
step([$class: 'XrayImportBuilder',
endpointName: '/cucumber/multipart',
importFilePath: 'storetarget-bdd/reporting/cucumber.json',
importInfo: info,
inputInfoSwitcher: 'fileContent',
serverInstance: xrayConnectorId])
}
}
catch(e) {
// If there was an exception thrown, the build failed
currentBuild.result = "FAILED"
throw e
} finally {
// Success or failure, always send notifications
echo "Sending final test status to Slack"
// notifyBuild(currentBuild.result)
}
}
def notifyBuild(String buildStatus = 'STARTED') {
// build status of null means successful
buildStatus = buildStatus ?: 'SUCCESSFUL'
// Default values
def colorName = 'RED'
def colorCode = '#FF0000'
def subject = "${buildStatus}: Job '${env.JOB_NAME} [${env.BUILD_NUMBER}]'"
def summary = "${subject} (${env.BUILD_URL})"
def details = """<p>STARTED: Job '${env.JOB_NAME} [${env.BUILD_NUMBER}]':</p>
<p>Check console output at "<a href='${env.BUILD_URL}'>${env.JOB_NAME} [${env.BUILD_NUMBER}]</a>"</p>"""
// Override default values based on build status
if (buildStatus == 'STARTED') {
color = 'BLUE'
colorCode = '#0000FF'
msg = "Build: ${env.JOB_NAME} has started: ${BUILD_TIMESTAMP}"
} else if (buildStatus == 'UNSTABLE') {
color = 'YELLOW'
colorCode = '#FFFF00'
msg = "Build: ${env.JOB_NAME} was listed as unstable. Look at ${env.BUILD_URL} and Report: ${env.BUILD_URL}/cucumber-html-reports/overview-features.html"
} else if (buildStatus == 'SUCCESSFUL') {
color = 'GREEN'
colorCode = '#00FF00'
msg = "Build: ${env.JOB_NAME} Completed Successfully ${env.BUILD_URL} Report: ${env.BUILD_URL}/cucumber-html-reports/overview-features.html"
} else {
color = 'RED'
colorCode = '#FF0000'
msg = "Build: ${env.JOB_NAME} had an issue ${env.BUILD_URL}/console"
}
// Send notifications
slackSend (color: colorCode, message: summary)
slackSend baseUrl: 'https://hooks.slack.com/services/',
channel: '#wopr-private',
color: colorCode,
message: msg,
teamDomain: '<Slack URL>',
tokenCredentialId: 'Jenkins-Slack-Token',
username: 'JenkinsAutomation'
}
feature file
Feature: SNMP Firmware Upgrade Test
#demo #AARC-3428
Scenario: SNMP Firmware Upgrade Executed against the DUT
Given ModemUpgrade.sh Script Exists
When SNMP Firmware Upgrade Executed
Then I expect Result Pass
step file
from behave import *
import pathlib
from pathlib import Path
#given('ModemUpgrade.sh Script Exists')
def step_impl(context):
STCFile = pathlib.Path('ModemUpgrade.sh')
if STCFile.exists():
print("SNMP Firmware Upgrade file exists")
pass
# else:
# print("SNMP Firmware Upgrade file does not exists")
# assert context.failed
#when('SNMP Firmware Upgrade Executed')
def step_impl(context):
path_to_file = 'PROD-003-Out.txt'
path = Path(path_to_file)
if path.is_file():
print(f'Output file {path_to_file} exists')
else:
print(f'Output file {path_to_file} does not exists')
#then('I expect Result Pass')
def step_impl(context):
Result = False
with open("PROD-003-Out.txt") as FwUpgradeResults:
for line in FwUpgradeResults:
if 'Upgrade Status: Passed'.lower() in line.strip().lower():
Result = True
break
else:
Result = False
break
if Result is False:
print("Error: Upgrade Failed")
assert context.failed
The suggestion of using || /usr/bin/true appears to have worked for the above mentioned code. Now I have a second instance where my Python test is throwing an exception when the DUT fails DHCP bind
def wait_for_dhcp_bind():
try:
stc.perform("Dhcpv4BindWait", objectlist=project)
except Exception:
raise Exception("DHCP Bind Failed")
I attempted to add the same after the Python script but the Jenkins build fails without the Xray test getting updated with a failure.
Here is what this looks like in the Jenkinsfile
echo "Starting Speed Test"
// def ModemMac = sh(returnStdout: true, script: './ModemUpgrade.sh -i ${modemIP} -f mac')
sh """
export STC_PRIVATE_INSTALL_DIR=${STC_INSTALL}
cd ${env.WORKSPACE_LOCAL}
/var/lib/jenkins/.pyenv/shims/python SpeedTest.py -d $dsp -u $usp -i $iterations -x $imix -f $frames -m $ModemMac || /usr/bin/true
/var/lib/jenkins/.pyenv/shims/behave -f cucumber -o storetarget-bdd/reporting/cucumber.json --junit --format=json -o target/behave.json --junit ./features/speedtest.feature || /usr/bin/true
"""
Your case should be easy to fix. Behave utility returns exit code 1 if tests fails..
Just add this to the end of your behave command || /usr/bin/true (please make sure of the path of the "true" command).
This will make your command to always return true even if some problems exist with behave.
So your overall command should be something like:
/var/lib/jenkins/.pyenv/shims/behave -f cucumber -o storetarget-bdd/reporting/cucumber.json --junit --format=json -o target/behave.json --junit ./features/PROD-003.feature || /usr/bin/true
Sorry in advance for silly question.
I am trying to write simple python script that runs virtual machine using PowerShell commands.
I have got a little problem with converting output from PowerShell command to a variable in Python.
The idea is:
I launch virtual machine, after that I check the state of it and, if the state is Running - start all the activity.
It is not a problem to do it in PowerShell, I wrote all the commands (launch VM, check state, if statement etc), but it is a problem to do it from py file.
My script looks like that:
import subprocess
import time
import os
class Utils(object):
def __init__(self):
self.ps_exec = r"C:\path\PsExec.exe"
self.power_shell = r"C:\path\powershell.exe"
def vm(self, apply_vm_script):
subprocess.Popen([self.power_shell, apply_vm_script])
util = Utils()
def turn_on_vm(vm_name, checkpoint_name):
apply_vm_script = 'Invoke-Command -Computername name -ScriptBlock ' \
'{ Get-VM ''"' + vm_name + '"'' | get-vmsnapshot -Name ' + '"' + checkpoint_name + '" | ' \
'Restore-VMSnapshot -Confirm:$false ; Start-VM -Name ''"' + vm_name + '"''}'
util.vm(apply_vm_script)
time.sleep(10)
def check_if_vm_on(vm_name):
check_vm_script = 'Invoke-Command -Computername name -ScriptBlock { Get-VM | where {$_.Name -eq ' + vm_name + ' } | where { $_.State -eq "Running" } | select Name,State}'
util.vm(check_vm_script)
time.sleep(3)
def test():
turn_on_vm('VM_name', 'checkpoint_name')
if(check_if_vm_on('VM_name')):
Do my activity
def main():
test()
if __name__ == '__main__':
main()
Also, I can perform all if actions in PowerShell, but also can't convert bool output into Python:
if($State -like '*Running*') { Write-Output "True" }
State was defined earlier, no problem with variables.
Any ideas how to solve it?
Thank you in advance!!!
You need to get the stdout from the powershell script to your python program. This can be done with Popen.communicate().
def vm(self, apply_vm_script):
p = subprocess.Popen([self.power_shell, apply_vm_script], stdout=subprocess.PIPE)
result = p.communicate()[0]
return str(result)
Also you need to return this value from check_if_vm_on
def check_if_vm_on(vm_name):
check_vm_script = 'Invoke-Command -Computername name -ScriptBlock { Get-VM | where {$_.Name -eq ' + vm_name + ' } | where { $_.State -eq "Running" } | select Name,State}'
result = util.vm(check_vm_script)
time.sleep(3)
return result
Then you will be able to check it with the if statement:
if(check_if_vm_on('VM_name') == "True"):
Do my activity
I want to run following jq command with subprocess.Popen() in python3.
$ jq 'INDEX(.images[]; .id) as $imgs | {
"filename_with_label":[
.annotations[]
| select(.attributes.type=="letter" )
| $imgs[.image_id] + {label:.text}
| {id:.id} + {filename:.file_name} + {label:.label}
]
}' image_data_annotation.json > image_data_annotation_with_label.json
Note that first command line argument contains dot, dollar sign, double quotes within single quote.
FYI, jq is JSON processor utility for processing json files.
I wrote following python3 script for automating JSON file processing with jq utility.
#!python3
# file name: letter_image_tool.py
import os, subprocess
"""
command line example to automate
$ jq 'INDEX(.images[]; .id) as $imgs | {
"filename_with_label":[
.annotations[]
| select(.attributes.type=="letter" )
| $imgs[.image_id] + {label:.text}
| {id:.id} + {filename:.file_name} + {label:.label}
]
}' image_data_annotation.json > image_data_annotation_with_label.json
"""
# define first command line argument
jq_filter='\'INDEX(.images[]; .id) as $imgs | { "filename_with_label" : [ .annotations[] | select(.attributes.type=="letter" ) | $imgs[.image_id] + {label:.text} | {id:.id} + {filename:.file_name} + {label:.label} ] }\''
input_json_files= [ "image_data_annotation.json"]
output_json_files= []
for input_json in input_json_files:
print("Processing %s" %(input_json))
filename, ext = os.path.splitext(input_json)
output_json = filename + "_with_label" + ext
output_json_files.append(output_json)
print("output file is : %s" %(output_json))
#jq_command ='jq' + " " + jq_filter, input_json + ' > ' + output_json
jq_command =['jq', jq_filter, input_json + ' > ' + output_json]
print(jq_command)
subprocess.Popen(jq_command, shell=True)
Running the above python script on bash results in folowing:
$ ./letter_image_tool.py
Processing image_data_annotation.json
output file is : image_data_annotation_with_label.json
['jq', '\'INDEX(.images[]; .id) as $imgs | { "filename_with_label" : [ .annotations[] | select(.attributes.type=="letter" ) | $imgs[.image_id] + {label:.text} | {id:.id} + {filename:.file_name} + {label:.label} ] }\'', 'image_data_annotation.json > image_data_annotation_with_label.json']
jq - commandline JSON processor [version 1.6-124-gccc79e5-dirty]
Usage: jq [options] <jq filter> [file...]
jq [options] --args <jq filter> [strings...]
jq [options] --jsonargs <jq filter> [JSON_TEXTS...]
jq is a tool for processing JSON inputs, applying the given filter to
its JSON text inputs and producing the filter's results as JSON on
standard output.
The simplest filter is ., which copies jq's input to its output
unmodified (except for formatting, but note that IEEE754 is used
for number representation internally, with all that that implies).
For more advanced filters see the jq(1) manpage ("man jq")
and/or https://stedolan.github.io/jq
Example:
$ echo '{"foo": 0}' | jq .
{
"foo": 0
}
For a listing of options, use jq --help.
It does not handle the first argument of jq utility:
'INDEX(.images[]; .id) as $imgs | {
"filename_with_label":[
.annotations[]
| select(.attributes.type=="letter" )
| $imgs[.image_id] + {label:.text}
| {id:.id} + {filename:.file_name} + {label:.label}
]
}'
The first argument should be enclosed with single quote as above snipet but my script does not handle it.
I think the main problems are related to the dot, dollar sign, single quote and double quote used in the first command line argument (jq_filter in the above python script). But I don't know how to treat this kind of complex meta character related to bash.
What should I do to solve above problems?
Thanks for your kind reading.
Update with my solution
With triple quote for jq_filter defintion, and space seprated join as follows
#!python3
# file name: letter_image_tool.py
import os, subprocess
"""
command line example to automate
$ jq 'INDEX(.images[]; .id) as $imgs | {
"filename_with_label":[
.annotations[]
| select(.attributes.type=="letter" )
| $imgs[.image_id] + {label:.text}
| {id:.id} + {filename:.file_name} + {label:.label}
]
}' image_data_annotation.json > image_data_annotation_with_label.json
"""
# define first command line argument with triple quotes
jq_filter=""" 'INDEX(.images[]; .id) as $imgs | {
"filename_with_label" : [
.annotations[]
| select(.attributes.type=="letter" )
| $imgs[.image_id] + {label:.text}
| {id:.id} + {filename:.file_name} + {label:.label} ] } ' """
input_json_files= [ "image_data_annotation.json"]
output_json_files= []
for input_json in input_json_files:
print("Processing %s" %(input_json))
filename, ext = os.path.splitext(input_json)
output_json = filename + "_with_label" + ext
output_json_files.append(output_json)
print("output file is : %s" %(output_json))
#jq_command ='jq' + " " + jq_filter, input_json + ' > ' + output_json
# jq command composed with space separated join
jq_command =' '.join['jq', jq_filter, input_json, ' > ', output_json]
print(jq_command)
# shell keyword argument should be set True
subprocess.Popen(jq_command, shell=True)
With triple double quotes, jq_filter can be more readable using multi-lined definition instead of single line defintion.
The reason you need single quotes is to prevent the shell from doing any expansion of your argument. This is a problem, only when using shell=True. If this is not set, the shell will never touch your arguments and there is no need to "protect" them.
However, the shell is also responsible for the stdout redirect (i.e. [... '>', output_json]). Not using the shell, requires that the redirect is handled in the Python code instead. That, however, is as simple as adding the argument stdout=... to Popen.
All-in-all this means that your code can be rewritten as
import os
import subprocess
# Still define first command line argument with triple quotes for readability
# Note that there are no single quotes though
jq_filter = """INDEX(.images[]; .id) as $imgs | {
"filename_with_label" : [
.annotations[]
| select(.attributes.type=="letter" )
| $imgs[.image_id] + {label:.text}
| {id:.id} + {filename:.file_name} + {label:.label} ] }"""
input_json_files = ["image_data_annotation.json"]
output_json_files = []
for input_json in input_json_files:
print("Processing %s" % (input_json))
filename, ext = os.path.splitext(input_json)
output_json = filename + "_with_label" + ext
output_json_files.append(output_json)
print("output file is : %s" % (output_json))
# Keep command as list, since this is what we need when NOT using shell=True
# Note also that the redirect and the output file are not parts of the argument list
jq_command = ['jq', jq_filter, input_json]
# shell keyword argument should NOT be set True
# Instead redirect stdout to an out_file
# (We must open the file for writing before redirecting)
with open(output_json, "w") as out_file:
subprocess.Popen(jq_command, stdout=out_file)
Generally it is recommended to not use shell=True anyway, since that opens up another vector of attack against the code, since an injection attack can give full access to the shell. Also, another small benefit with not using the shell, is that it will reduce the number of created subprocesses, since no extra shell process is needed.
I have a Backup software made in Python. This backup software lists all the computers on a network and I need to backup the computers to a given directory. My thought was: When the user click on backup on a certain computer, it launches a Thread where a Powershell script is executed and the percentage ((Bytes Copied / Total Bytes to be Copied)*100) is displayed. Everything works fine and I'm able to copy a single computer without problem.
But here's the problem: Whenever I launch a second thread (I click backup on another Computer), it seems like the process from Powershell stops as there's no more output and the only running process is the new Powershell process launched on the second thread.
Thread run method:
def run(self):
'''
Initialise the runner function with passed args, kwargs.
'''
self.setAlive(True)
p = subprocess.Popen(["powershell.exe",".\script.ps1 . .\\Backups"],stdout=subprocess.PIPE)
self.signals.progress.emit(self.CurrentComp['RowIndex'], 0, self)
percentage = 0.0
a = ""
displayPercentage = 0.0
while a != "Done Copying":
a = p.stdout.readline()
a = a.decode("utf-8").strip()
try:
percentage = float(a)
print(percentage)
while displayPercentage != int(percentage):
displayPercentage += 1
sleep(0.03)
self.signals.progress.emit(self.CurrentComp['RowIndex'], displayPercentage ,self)
except ValueError:
pass
self.signals.finished.emit()
Param([String[]]$paths,[String]$Destination)
function Copy-WithProgress {
[CmdletBinding()]
param (
[Parameter(Mandatory = $true)]
[string] $Source
, [Parameter(Mandatory = $true)]
[string] $Destination
, [int] $Gap = 200
, [int] $ReportGap = 2000
)
$RegexBytes = '(?<=\s+)\d+(?=\s+)';
$CommonRobocopyParams = '/MIR /NP /NDL /NC /BYTES /NJH /NJS';
$StagingLogPath = '{0}\temp\{1} robocopy staging.log' -f $env:windir, (Get-Date -Format 'yyyy-MM-dd HH-mm-ss');
$StagingArgumentList = '"{0}" "{1}" /LOG:"{2}" /L {3}' -f $Source, $Destination, $StagingLogPath, $CommonRobocopyParams;
Start-Process -Wait -FilePath robocopy.exe -ArgumentList $StagingArgumentList -NoNewWindow;
$StagingContent = Get-Content -Path $StagingLogPath;
$TotalFileCount = $StagingContent.Count - 1;
[RegEx]::Matches(($StagingContent -join "`n"), $RegexBytes) | % { $BytesTotal = 0; } { $BytesTotal += $_.Value; };
$RobocopyLogPath = '{0}\temp\{1} robocopy.log' -f $env:windir, (Get-Date -Format 'yyyy-MM-dd HH-mm-ss');
$ArgumentList = '"{0}" "{1}" /LOG:"{2}" /ipg:{3} {4}' -f $Source, $Destination, $RobocopyLogPath, $Gap, $CommonRobocopyParams;
$Robocopy = Start-Process -FilePath robocopy.exe -ArgumentList $ArgumentList -Verbose -PassThru -NoNewWindow;
Start-Sleep -Milliseconds 100;
#region Progress bar loop
while (!$Robocopy.HasExited) {
Start-Sleep -Milliseconds $ReportGap;
$BytesCopied = 0;
$LogContent = Get-Content -Path $RobocopyLogPath;
$BytesCopied = [Regex]::Matches($LogContent, $RegexBytes) | ForEach-Object -Process { $BytesCopied += $_.Value; } -End { $BytesCopied; };
#Write-Verbose -Message ('Bytes copied: {0}' -f $BytesCopied);
#Write-Verbose -Message ('Files copied: {0}' -f $LogContent.Count);
$Percentage = 0;
if ($BytesCopied -gt 0) {
$Percentage = (($BytesCopied/$BytesTotal)*100)
}
Write-Host $Percentage
}
Write-Host "Done Copying"
I call the python code from a Powershell script in order to loop over some arguments. Calling the python script from a Powershell is straight forward and works without a hitch:
PS C:\Windows\system32> C:\Users\Administrator\AppData\Local\Programs\Python\Python35-32\python.exe C:\Users\Administrator\AppData\Local\Programs\youtube-upload-master\bin\youtube-upload C:\Users\Administrator\Documents\timelapse\videos\timelapse_10.0.0.51-2016-06-21.mp4 --client-secrets=C:\Users\Administrator\Documents\timelapse\credentials\.yt-ul-ioa-secr.json --credentials-file=C:\Users\Administrator\Documents\timelapse\credentials\.yt-ul-ioa-cred.json --title="Timelapse 21.06.2016" --playlist "Timelapses June 2016"
Then within a script I am changing the parameters inserting variables into the argument strings, and finally calling the whole thing with Invoke-Command:
# yt-ul.ps1
param(
#[switch]$all_cams = $false,
[int]$days = -1,
[string]$cam = "ioa"
)
$cam_ip_hash = #{
"ioa" = "10.0.0.51";
"pam" = "10.0.0.52";
"biz" = "10.0.0.56";
"prz" = "10.160.58.25";
"igu" = "10.160.38.35"}
$cam_ip = $cam_ip_hash[$cam]
$date = (Get-Date).AddDays($days).ToString("yyyy-MM-dd")
$py = "C:\Users\Administrator\AppData\Local\Programs\Python\Python35-32\python.exe"
$yt_ul = "C:\Users\Administrator\AppData\Local\Programs\youtube-upload-master\bin\youtube-upload"
$title_date = (Get-Date).AddDays($days).ToString("dd.MM.yyyy")
$us = New-Object System.Globalization.CultureInfo("en-US")
$playlist_date = (Get-Date).AddDays($days).ToString("Y", $us)
$vid = "C:\Users\Administrator\Documents\timelapse\videos\timelapse_$cam_ip-$date.mp4"
$secr = "--client-secrets=C:\Users\Administrator\Documents\timelapse\credentials\.yt-ul-igu-secr.json"
$cred = "--credentials-file=C:\Users\Administrator\Documents\timelapse\credentials\.yt-ul-igu-cred.json"
$title = "--title=`"Timelapse $title_date`""
$playlist_date = "--playlist `"Timelapses $playlist_date`""
$arg_list = "$yt_ul $vid $secr $cred $title $playlist_date"
Invoke-Command "$py $arg_list"
But actually calling the script fails as follows:
PS C:\Users\Administrator\Documents\scripts> .\yt-ul.ps1
Invoke-Command : Parameter set cannot be resolved using the specified named parameters.
At C:\Users\Administrator\Documents\scripts\yt-ul.ps1:34 char:1
+ Invoke-Command "$py $arg_list"
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidArgument: (:) [Invoke-Command], ParameterBindingException
+ FullyQualifiedErrorId : AmbiguousParameterSet,Microsoft.PowerShell.Commands.InvokeCommandCommand
I assume I am doing something really stupid with the single and double quotes, but I am not sure.
Thanks to JosefZ this works:
& $py $yt_ul $vid $secr $cred --title "Timelapse $title_date" --playlist "Timelapses $playlist_date"