I have python code which calls a shell script (get_list.sh) and this shell script calls one .txt file which is having the entires like :
aaf:hfhfh:notusable:type: - city_name
hhf:hgyt:usable:type: - city_name
llf:hdgt:used:type: - city_name
and when I providing the input like after running the python code :
code for providing the input :
List = str(raw_input('Enter pipe separated list : ')).upper().strip()
hhf|aaf|llf
code for getting the output :
if List:
try:
cmd = "/home/dponnura/get_list.sh -s " + "\"" + List + "\""
selfP = commands.getoutput(cmd).strip()
except OSError:
print bcolors.FAIL + "Could not invoke Pod Details Script. " + bcolors.ENDC
it shows me the output as :
hhf detils : hfhfh:notusable:type: - city_name
aaf details : hgyt:usable:type: - city_name
llf details : hdgt:used:type: - city_name
What my requirnment is, if I passes the input after execution of python code and if my enties are not present in .txt file it should show me the output as :
if I provide the input as :
hhf|aaf|llf|ggg
then for 'ggg' it should show me like :
'ggg' is wrong input
hhf detils : hfhfh:notusable:type: - city_name
aaf details : hgyt:usable:type: - city_name
llf details : hdgt:used:type: - city_name
Could you please let me know how can I do this in python or shell?
Here is this done in Python, without calling get_list.sh
import sys,re
List = str(raw_input('Enter pipe separated list : ')).strip().split('|')
for linE in open(sys.argv[1]):
for l1 in List:
m1 = re.match(l1+':(.+)',linE)
if m1:
print l1,'details :',m1.group(1)
List.remove(l1)
break
for l1 in List : print l1,'is wrong input'
Usage :
python script.py textFile.txt
Your task can ( and I think have to ) be implemented in pure Python. Below is one of possible variants of its solution using just Python, without external scripts or libraries
pipelst = str(raw_input('Enter pipe separated list : ')).split('|')
filepath = 'test.txt' # specify path to your file here
for lns in open(filepath):
split_pipe = lns.split(':', 1)
if split_pipe[0] in pipelst:
print split_pipe[0], ' details : ', split_pipe[1]
pipelst.remove(split_pipe[0])
for lns in pipelst : print lns,' is wrong input'
As you can see, it is a short, simple and clear.
Related
I have written python code to retrieve information about build. I prints a summary of successful and unsuccessful builds.
from prettytable import PrettyTable
t = PrettyTable(['Job name','Successful','Failed','Unstable','Aborted','Total Builds','Failure Rate'])
t1 = PrettyTable(['Status', 'Job name','Build #','Date','Duration','Node','User'])
aggregation ={}
jobs = server.get_all_jobs(folder_depth=None)
for job in jobs:
print(job['fullname'])
aggregation[job['fullname']] = {"success" : 0 , "failure" : 0 , "aborted" : 0, "unstable":0}
info = server.get_job_info(job['fullname'])
# Loop over builds
builds = info['builds']
for build in builds:
information = server.get_build_info(job["fullname"],
build['number'])
if "SUCCESS" in information['result']:
aggregation[job['fullname']]['success'] = str(int(aggregation[job['fullname']]['success']) + 1)
if "FAILURE" in information['result']:
aggregation[job['fullname']]['failure'] = str(int(aggregation[job['fullname']]['failure']) + 1)
if "ABORTED" in information['result']:
aggregation[job['fullname']]['aborted'] = str(int(aggregation[job['fullname']]['aborted']) + 1)
if "UNSTABLE" in information['result']:
aggregation[job['fullname']]['unstable'] = str(int(aggregation[job['fullname']]['unstable']) + 1)
t1.add_row([ information['result'], job['fullname'],information["id"],datetime.fromtimestamp(information['timestamp']/1000),information["duration"],"master",information["actions"][0]["causes"][0]["userName"]])
total_build = int(aggregation[job['fullname']]['success'])+int(aggregation[job['fullname']]['failure'])
t.add_row([job["fullname"], aggregation[job['fullname']]['success'],aggregation[job['fullname']]['failure'],aggregation[job['fullname']]['aborted'],aggregation[job['fullname']]['unstable'],total_build,(float(aggregation[job['fullname']]['failure'])/total_build)*100])
with open('result', 'w') as w:
w.write(str(t1))
w.write(str(t))
This is what the output looks like:
And this is what Windows batch execute command looks like:
cd E:\airflowtmp
conda activate web_scraping
python hello.py
hello.py prints hello world. If I add print counter =100 or something like this then how do I return it and print it in this resultant table.
Edit:
I am trying to get some kind of variable from code to display. For instance if Im scraping pages and scraper ran successfully then I want to know the number of pages that it scraped. You can think of it as a simple counter. Is there any way to return a variable from Jenkins to python
I am new to snakemake and would like to use the following rule :
input_path = config["PATH"]
samples = pd.read_csv(config["METAFILE"], sep = '\t', header = 0)['sample']
rule getPaired:
output:
fwd = temp(tmp_path + "/reads/{sample}_fwd.fastq.gz"),
rev = temp(tmp_path + "/reads/{sample}_rev.fastq.gz")
params:
input_path = input_path
run:
shell("scp -i {params.input_path}/{wildcards.sample}_*1*.f*q.gz {output.fwd}"),
shell("scp -i {params.input_path}/{wildcards.sample}_*2*.f*q.gz {output.rev}")
Input files have different patterns :
{sampleID}_R[1-2]_001.fq.gz (for example : 2160_J15_S480_R1_001.fastq.gz)
{sampleID}_[1-2].fq.gz (for example : SRX000001_1.fq.gz)
The getPaired rule works for input like {sample}_[1-2].fq.gz but not for the second pattern.
What am I doing wrong ?
You should make use of input functions. I made an example which isn't really what you need, but I think it should clearly show what you want to achieve:
paths = {'sample1': '/home/jankees/data',
'sample2': '/mnt/data',
'sample3': '/home/christina/fastq'}
extensions = {'sample1': '.fq.gz',
'sample2': '.fq.gz',
'sample3': '.fastq.gz'}
def get_input(wildcards):
input_file = paths[wildcards.sample] + "/read/" + wildcards.sample + extensions[wildcards.sample]
return input_file
rule all:
input:
["sample1_trimmed.fastq.gz",
"sample2_trimmed.fastq.gz",
"sample3_trimmed.fastq.gz"]
rule trim:
input:
get_input
output:
"{sample}_trimmed.fastq.gz"
shell:
"touch {output}"
I am using Behave to automate the testing of a config file, as part of this test I need to populate various fields in the config file with invalid and blank fields. Where I am entering values I can do this using a Scenario Outline entering the values in the Examples. However when I try entering a blank field using this method Behave does not like the fact there is no value.
Is there an easy way to pass a blank value from the Examples file, or will I need to test these conditions using a separate behave test
feature
Scenario Outline:Misconfigured Identity Listener
Given an already stopped Identity Listener
And parameter <parameter> is configured to value <config_value>
When the Identity Listener is started
Then the identity listener process is not present on the system
And the log contains a <message> showing that the parameter is not configured
Examples: Protocols
|parameter |message |config_value|
|cache_ip_address | cache_ip_address | |
|cache_ip_address | cache_ip_address | 123.123.12 |
the step where I define the config value
#given('parameter {parameter} is configured to value {config_value}')
def step_impl(context, parameter, config_value):
context.parameter = parameter
context.config_value = config_value
context.identity_listener.update_config(parameter, config_value)
changing the config file using sed -i (I am interacting with a linux box in this test)
def update_config(self, param, config_value):
command = 'sudo sh -c "sed -i'
command = command + " '/" + param + "/c\\" + param + "= "+ config_value + " \\' {0}\""
command = command.format(self.config_file)
self.il_ssh.runcmd(command)
Thanks to answer from #Verv i got this working solution below
passed an empty value in for fields where I don't want a value passed
|parameter |message |config_value|
|cache_ip_address | cache_ip_address | empty |
Added an if else statement into my update config step
def update_config(self, param, config_value):
if config_value == "empty":
il_config = ""
else:
il_config = config_value
command = 'sudo sh -c "sed -i'
command = command + " '/" + param + "/c\\" + param + "= " + il_config + " \\' {0}\""
command = command.format(self.config_file)
self.il_ssh.runcmd(command)
You could put something like empty in the field, and tweak your method so that whenever the field's value is empty, you treat it as an actual empty string (i.e. "")
I have a VBA script in Microsoft Access. The VBA script is part of a large project with multiple people, and so it is not possible to leave the VBA environment.
In a section of my script, I need to do complicated linear algebra on a table quickly. So, I move the VBA tables written as recordsets) into Python to do linear algebra, and back into VBA. The matrices in python are represented as numpy arrays.
Some of the linear algebra is proprietary and so we are compiling the proprietary scripts with pyinstaller.
The details of the process are as follows:
The VBA script creates a csv file representing the table input.csv.
The VBA script runs the python script through the command line
The python script loads the csv file input.csv as a numpy matrix, does linear algebra on it, and creates an output csv file output.csv.
VBA waits until python is done, then loads output.csv.
VBA deletes the no-longer-needed input.csv file and output.csv file.
This process is inefficient.
Is there a way to load VBA matrices into Python (and back) without the csv clutter? Do these methods work with compiled python code through pyinstaller?
I have found the following examples on stackoverflow that are relevant. However, they do not address my problem specifically.
Return result from Python to Vba
How to pass Variable from Python to VBA Sub
Solution 1
Either retrieve the COM running instance of Access and get/set the data directly with the python script via the COM API:
VBA:
Private Cache
Public Function GetData()
GetData = Cache
Cache = Empty
End Function
Public Sub SetData(data)
Cache = data
End Sub
Sub Usage()
Dim wshell
Set wshell = VBA.CreateObject("WScript.Shell")
' Make the data available via GetData()'
Cache = Array(4, 6, 8, 9)
' Launch the python script compiled with pylauncher '
Debug.Assert 0 = wshell.Run("C:\dev\myapp.exe", 0, True)
' Handle the returned data '
Debug.Assert Cache(3) = 2
End Sub
Python (myapp.exe):
import win32com.client
if __name__ == "__main__":
# get the running instance of Access
app = win32com.client.GetObject(Class="Access.Application")
# get some data from Access
data = app.run("GetData")
# return some data to Access
app.run("SetData", [1, 2, 3, 4])
Solution 2
Or create a COM server to expose some functions to Access :
VBA:
Sub Usage()
Dim Py As Object
Set Py = CreateObject("Python.MyModule")
Dim result
result = Py.MyFunction(Array(5, 6, 7, 8))
End Sub
Python (myserver.exe or myserver.py):
import sys, os, win32api, win32com.server.localserver, win32com.server.register
class MyModule(object):
_reg_clsid_ = "{5B4A4174-EE23-4B70-99F9-E57958CFE3DF}"
_reg_desc_ = "My Python COM Server"
_reg_progid_ = "Python.MyModule"
_public_methods_ = ['MyFunction']
def MyFunction(self, data) :
return [(1,2), (3, 4)]
def register(*classes) :
regsz = lambda key, val: win32api.RegSetValue(-2147483647, key, 1, val)
isPy = not sys.argv[0].lower().endswith('.exe')
python_path = isPy and win32com.server.register._find_localserver_exe(1)
server_path = isPy and win32com.server.register._find_localserver_module()
for cls in classes :
if isPy :
file_path = sys.modules[cls.__module__].__file__
class_name = '%s.%s' % (os.path.splitext(os.path.basename(file_path))[0], cls.__name__)
command = '"%s" "%s" %s' % (python_path, server_path, cls._reg_clsid_)
else :
file_path = sys.argv[0]
class_name = '%s.%s' % (cls.__module__, cls.__name__)
command = '"%s" %s' % (file_path, cls._reg_clsid_)
regsz("SOFTWARE\\Classes\\" + cls._reg_progid_ + '\\CLSID', cls._reg_clsid_)
regsz("SOFTWARE\\Classes\\AppID\\" + cls._reg_clsid_, cls._reg_progid_)
regsz("SOFTWARE\\Classes\\CLSID\\" + cls._reg_clsid_, cls._reg_desc_)
regsz("SOFTWARE\\Classes\\CLSID\\" + cls._reg_clsid_ + '\\LocalServer32', command)
regsz("SOFTWARE\\Classes\\CLSID\\" + cls._reg_clsid_ + '\\ProgID', cls._reg_progid_)
regsz("SOFTWARE\\Classes\\CLSID\\" + cls._reg_clsid_ + '\\PythonCOM', class_name)
regsz("SOFTWARE\\Classes\\CLSID\\" + cls._reg_clsid_ + '\\PythonCOMPath', os.path.dirname(file_path))
regsz("SOFTWARE\\Classes\\CLSID\\" + cls._reg_clsid_ + '\\Debugging', "0")
print('Registered ' + cls._reg_progid_)
if __name__ == "__main__":
if len(sys.argv) > 1 :
win32com.server.localserver.serve(set([v for v in sys.argv if v[0] == '{']))
else :
register(MyModule)
Note that you'll have to run the script once without any argument to register the class and to make it available to VBA.CreateObject.
Both solutions work with pylauncher and the array received in python can be converted with numpy.array(data).
Dependency :
https://pypi.python.org/pypi/pywin32
You can try loading your record set into an array, dim'ed as Double
Dim arr(1 to 100, 1 to 100) as Double
by looping, then pass the pointer to the first element ptr = VarPtr(arr(1, 1)) to Python, where
arr = numpy.ctypeslib.as_array(ptr, (100 * 100,)) ?
But VBA will still own the array memory
There is a very simple way of doing this with xlwings. See xlwings.org and make sure to follow the instructions to enable macro settings, tick xlwings in VBA references, etc. etc.
The code would then look as simple as the following (a slightly silly block of code that just returns the same dataframe back, but you get the picture):
import xlwings as xw
import numpy as np
import pandas as pd
# the #xw.decorator is to tell xlwings to create an Excel VBA wrapper for this function.
# It has no effect on how the function behaves in python
#xw.func
#xw.arg('pensioner_data', pd.DataFrame, index=False, header=True)
#xw.ret(expand='table', index=False)
def pensioner_CF(pensioner_data, mortality_table = "PA(90)", male_age_adj = 0, male_improv = 0, female_age_adj = 0, female_improv = 0,
years_improv = 0, arrears_advance = 0, discount_rate = 0, qxy_tables=0):
pensioner_data = pensioner_data.replace(np.nan, '', regex=True)
cashflows_df = pd.DataFrame()
return cashflows_df
I'd be interested to hear if this answers the question. It certainly made my VBA / python experience a lot easier.
I am using pyparsing to parse a language called pig. I found some unexpected result when used the function 'lineno' when the input text have some '\t' in it.
For easy to ask, I simplified the code to address the problem:
#/usr/bin/env python
from pyparsing import *
ident = Word(alphas)
statement1 = ident + Literal('=')+ Keyword('GENERATE', caseless = True) + SkipTo(Literal(';'),ignore = nestedExpr())+ Literal(';').suppress()
statement2 = Keyword('STORE',caseless = True) + ident + Literal(';').suppress()
statement = statement1|statement2
text = """
fact = GENERATE
('Exp' :(a
)
) ;
STORE fact ;
"""
all_statements = statement.scanString(text)
for tokens,startloc,endloc in all_statements:
print 'startloc:' + str(startloc) , 'lineno:' + str(lineno(startloc,text))
print 'endloc:' + str(endloc), 'lineno:' + str(lineno(endloc,text))
print tokens
Notice that in the input text, there is more than 3 '\t' in the beginning of the third line
When I run this , the output is :
startloc:1 lineno:2
endloc:66 lineno:10
['fact', '=', 'GENERATE', "('Exp' :(a\n )\n) "]
startloc:68 lineno:10
endloc:80 lineno:10
['STORE', 'fact']
this should be wrong, as there is total 9 line, it said the first statement is from line 2 to line 10.
I happened to found that when I delete those '\t', the input text is :
text = """
fact = GENERATE
('Exp' :(a
)
) ;
STORE fact ;
"""
and I run it again, the result is :
startloc:1 lineno:2
endloc:34 lineno:5
['fact', '=', 'GENERATE', "('Exp' :(a\n)\n) "]
startloc:36 lineno:7
endloc:48 lineno:7
['STORE', 'fact']
And this result seems correct, the first statement is from line 2 to line 5, the second statemenet is from line 7 to line 7 .This is what I expected.
So I think there maybe something wrong in the lineno() function, or maybe scanString.
Or maybe there is something wrong in my code?
Use parseWithTabs before calling scanString.