We Keep Coding

Optional Output from Bazel Action? (SWIG rule for Bazel)

I'm working on a bazel rule (using version 5.2.0) that uses SWIG (version 4.0.1) to make a python library from C++ code, adapted from a rule in the tensorflow library. The problem I've run into is that, depending on the contents of ctx.file.source.path, the swig invocation might produce a necessary .h file. If it does, the rule below works great. If it doesn't, I get:
ERROR: BUILD:31:11: output 'foo_swig_h.h' was not created
ERROR: BUILD:31:11: SWIGing foo.i. failed: not all outputs were created or valid
If the h_out stuff is removed from _py_swig_gen_impl, the rule below works great when swig doesn't produce the .h file. But, if swig does produce one, bazel seems to ignore it and it isn't available for native.cc_binary to compile, resulting in gcc failing with a 'no such file or directory' error on an #include <foo_swig_cc.h> line in foo_swig_cc.cc.
(The presence or absence of the .h file in the output is determined by whether the .i file at ctx.file.source.path uses SWIG's "directors" feature.)
def _include_dirs(deps):
return depset(transitive = [dep[CcInfo].compilation_context.includes for dep in deps]).to_list()
def _headers(deps):
return depset(transitive = [dep[CcInfo].compilation_context.headers for dep in deps]).to_list()
# Bazel rules for building swig files.
def _py_swig_gen_impl(ctx):
module_name = ctx.attr.module_name
cc_out = ctx.actions.declare_file(module_name + "_swig_cc.cc")
h_out = ctx.actions.declare_file(module_name + "_swig_h.h")
py_out = ctx.actions.declare_file(module_name + ".py")
args = ["-c++", "-python", "-py3"]
args += ["-module", module_name]
args += ["-I" + x for x in _include_dirs(ctx.attr.deps)]
args += ["-I" + x.dirname for x in ctx.files.swig_includes]
args += ["-o", cc_out.path]
args += ["-outdir", py_out.dirname]
args += ["-oh", h_out.path]
args.append(ctx.file.source.path)
outputs = [cc_out, h_out, py_out]
ctx.actions.run(
executable = "swig",
arguments = args,
mnemonic = "Swig",
inputs = [ctx.file.source] + _headers(ctx.attr.deps) + ctx.files.swig_includes,
outputs = outputs,
progress_message = "SWIGing %{input}.",
)
return [DefaultInfo(files = depset(direct = [cc_out, py_out]))]
_py_swig_gen = rule(
attrs = {
"source": attr.label(
mandatory = True,
allow_single_file = True,
),
"swig_includes": attr.label_list(
allow_files = [".i"],
),
"deps": attr.label_list(
allow_files = True,
providers = [CcInfo],
),
"module_name": attr.string(mandatory = True),
},
implementation = _py_swig_gen_impl,
)
def py_wrap_cc(name, source, module_name = None, deps = [], copts = [], **kwargs):
if module_name == None:
module_name = name
python_deps = [
"#local_config_python//:python_headers",
"#local_config_python//:python_lib",
]
# First, invoke the _py_wrap_cc rule, which runs swig. This outputs:
# `module_name.cc`, `module_name.py`, and, sometimes, `module_name.h` files.
swig_rule_name = "swig_gen_" + name
_py_swig_gen(
name = swig_rule_name,
source = source,
swig_includes = ["//third_party/swig_rules:swig_includes"],
deps = deps + python_deps,
module_name = module_name,
)
# Next, we need to compile the `module_name.cc` and `module_name.h` files
# from the previous rule. The `module_name.py` file already generated
# expects there to be a `_module_name.so` file, so we name the cc_binary
# rule this way to make sure that's the resulting file name.
cc_lib_name = "_" + module_name + ".so"
native.cc_binary(
name = cc_lib_name,
srcs = [":" + swig_rule_name],
linkopts = ["-dynamic", "-L/usr/local/lib/"],
linkshared = True,
deps = deps + python_deps,
)
# Finally, package everything up as a python library that can be depended
# on. Note that this rule uses the user-given `name`.
native.py_library(
name = name,
srcs = [":" + swig_rule_name],
srcs_version = "PY3",
data = [":" + cc_lib_name],
imports = ["./"],
)
My question, broadly, how I might best handle this with a single rule. I've tried adding a ctx.actions.write before the ctx.actions.run, thinking that I could generate a dummy '.h' file that would be overwritten if needed. That gives me:
ERROR: BUILD:41:11: for foo_swig_h.h, previous action: action 'Writing file foo_swig_h.h', attempted action: action 'SWIGing foo.i.'
My next idea is to remove the h_out stuff and then try to capture the h file for the cc_binary rule with some kind of glob invocation.

I've seen two approaches: add an attribute to indicate whether it applies, or write a wrapper script to generate it unconditionally.
Adding an attribute means something like "has_h": attr.bool(), and then use that in _py_swig_gen_impl to make the ctx.actions.declare_file(module_name + "_swig_h.h") conditional.
The wrapper script option means using something like this for the executable:
#!/bin/bash
set -e
touch the_path_of_the_header
exec swig "$#"
That will unconditionally create the output, and then swig will overwrite it if applicable. If it's not applicable, then passing around an empty header file in the Bazel rules should be harmless.

For posterity, this is what my _py_swig_gen_impl looks like after implementing #Brian's suggestion above:
def _py_swig_gen_impl(ctx):
module_name = ctx.attr.module_name
cc_out = ctx.actions.declare_file(module_name + "_swig_cc.cc")
h_out = ctx.actions.declare_file(module_name + "_swig_h.h")
py_out = ctx.actions.declare_file(module_name + ".py")
include_dirs = _include_dirs(ctx.attr.deps)
headers = _headers(ctx.attr.deps)
args = ["-c++", "-python", "-py3"]
args += ["-module", module_name]
args += ["-I" + x for x in include_dirs]
args += ["-I" + x.dirname for x in ctx.files.swig_includes]
args += ["-o", cc_out.path]
args += ["-outdir", py_out.dirname]
args += ["-oh", h_out.path]
args.append(ctx.file.source.path)
outputs = [cc_out, h_out, py_out]
# Depending on the contents of `ctx.file.source`, swig may or may not
# output a .h file needed by subsequent rules. Bazel doesn't like optional
# outputs, so instead of invoking swig directly we're going to make a
# lightweight executable script that first `touch`es the .h file that may
# get generated, and then execute that. This means we may be propagating
# an empty .h file around as a "dependency" sometimes, but that's okay.
swig_script_file = ctx.actions.declare_file("swig_exec.sh")
ctx.actions.write(
output = swig_script_file,
is_executable = True,
content = "#!/bin/bash\n\nset -e\ntouch " + h_out.path + "\nexec swig \"$#\"",
)
ctx.actions.run(
executable = swig_script_file,
arguments = args,
mnemonic = "Swig",
inputs = [ctx.file.source] + headers + ctx.files.swig_includes,
outputs = outputs,
progress_message = "SWIGing %{input}.",
)
return [
DefaultInfo(files = depset(direct = outputs)),
]
The ctx.actions.write generates the suggested bash script:
#!/bin/bash
set -e
touch %{h_out.path}
exec swig "$#"
Which guarantees that the expected h_out will always be output by ctx.actions.run, whether or not swig generates it.

How can i run a php function in python?

for some reasons, i have to run a php function in python.
However, i realized that it's beyond my limit.
So, i'm asking for help here.
below is the code
function munja_send($mtype, $name, $phone, $msg, $callback, $contents) {
$host = "www.sendgo.co.kr";
$id = ""; // id
$pass = ""; // password
$param = "remote_id=".$id;
$param .= "&remote_pass=".$pass;
$param .= "&remote_name=".$name;
$param .= "&remote_phone=".$phone; //cellphone number
$param .= "&remote_callback=".$callback; // my cellphone number
$param .= "&remote_msg=".$msg; // message
$param .= "&remote_contents=".$contents; // image
if ($mtype == "lms") {
$path = "/Remote/RemoteMms.html";
} else {
$path = "/Remote/RemoteSms.html";
}
$fp = #fsockopen($host,80,$errno,$errstr,30);
$return = "";
if (!$fp) {
echo $errstr."(".$errno.")";
} else {
fputs($fp, "POST ".$path." HTTP/1.1\r\n");
9
fputs($fp, "Host: ".$host."\r\n");
fputs($fp, "Content-type: application/x-www-form-urlencoded\r\n");
fputs($fp, "Content-length: ".strlen($param)."\r\n");
fputs($fp, "Connection: close\r\n\r\n");
fputs($fp, $param."\r\n\r\n");
while(!feof($fp)) $return .= fgets($fp,4096);
}
fclose ($fp);
$_temp_array = explode("\r\n\r\n", $return);
$_temp_array2 = explode("\r\n", $_temp_array[1]);
if (sizeof($_temp_array2) > 1) {
$return_string = $_temp_array2[1];
} else {
$return_string = $_temp_array2[0];
}
return $return_string;
}
i would be glad if anyone can show me a way.
thank you.

I don't know PHP, but based on my understanding, here should be a raw line-for-line translation of the code you provided, from PHP to python. I've preserved your existing comments, and added new ones for clarification in places where I was unsure or where you might want to change.
It should be pretty straightforward to follow - the difference is mostly in syntax (e.g. + for concatenation instead of .), and in converting str to bytes and vice versa.
import socket
def munja_send(mtype, name, phone, msg, callback, contents):
host = "www.sendgo.co.kr"
remote_id = "" # id (changed the variable name, since `id` is also a builtin function)
password = "" # password (`pass` is a reserved keyword in python)
param = "remote_id=" + remote_id
param += "&remote_pass=" + password
param += "&remote_name=" + name
param += "&remote_phone=" + phone # cellphone number
param += "&remote_callback=" + callback # my cellphone number
param += "&remote_msg=" + msg # message
param += "&remote_contents=" + contents # image
if mtype == "lms"
path = "/Remote/RemoteMms.html"
else:
path = "/Remote/RemoteSms.html"
socket.settimeout(30)
# change these parameters as necessary for your desired outcome
fp = socket.socket(family=socket.AF_INET, type=socket.SOCK_STREAM)
errno = fp.connect_ex((host, 80))
if errno != 0:
# I'm not sure where errmsg comes from in php or how to get it in python
# errno should be the same, though, as it refers to the same system call error code
print("Error(" + errno + ")")
else:
returnstr = b""
fp.send("POST " + path + "HTTP/1.1\r\n")
fp.send("Host: " + host + "\r\n")
fp.send("Content-type: application/x-www-form-urlencoded\r\n")
# for accuracy, we convert param to bytes using utf-8 encoding
# before checking its length. Change the encoding as necessary for accuracy
fp.send("Content-length: " + str(len(bytes(param, 'utf-8'))) + "\r\n")
fp.send("Connection: close\r\n\r\n")
fp.send(param + "\r\n\r\n")
while (data := fp.recv(4096)):
# fp.recv() should return an empty string if eof has been hit
returnstr += data
fp.close()
_temp_array = returnstr.split(b'\r\n\r\n')
_temp_array2 = _temp_array[1].split(b'\r\n')
if len(temp_array2) > 1:
return_string = _temp_array2[1]
else:
return_string = _temp_array2[0]
# here I'm converting the raw bytes to a python string, using encoding
# utf-8 by default. Replace with your desired encoding if necessary
# or just remove the `.decode()` call if you're fine with returning a
# bytestring instead of a regular string
return return_string.decode('utf-8')
If possible, you should probably use subprocess to execute your php code directly, as other answers suggest, as straight-up translating code is often error-prone and has slightly different behavior (case in point, the lack of errmsg and probably different error handling in general, and maybe encoding issues in the above snippet). But if that's not possible, then hopefully this will help.

according to the internet, you can use subprocess and then execute the PHP script
import subprocess
# if the script don't need output.
subprocess.call("php /path/to/your/script.php")
# if you want output
proc = subprocess.Popen("php /path/to/your/script.php", shell=True, stdout=subprocess.PIPE)
script_response = proc.stdout.read()

PHP code can be executed in python using libraries subprocess or php.py based on the situation.
Please refer this answer for further details.

Convert PowerShell output to Python and store it in variable

Sorry in advance for silly question.
I am trying to write simple python script that runs virtual machine using PowerShell commands.
I have got a little problem with converting output from PowerShell command to a variable in Python.
The idea is:
I launch virtual machine, after that I check the state of it and, if the state is Running - start all the activity.
It is not a problem to do it in PowerShell, I wrote all the commands (launch VM, check state, if statement etc), but it is a problem to do it from py file.
My script looks like that:
import subprocess
import time
import os
class Utils(object):
def __init__(self):
self.ps_exec = r"C:\path\PsExec.exe"
self.power_shell = r"C:\path\powershell.exe"
def vm(self, apply_vm_script):
subprocess.Popen([self.power_shell, apply_vm_script])
util = Utils()
def turn_on_vm(vm_name, checkpoint_name):
apply_vm_script = 'Invoke-Command -Computername name -ScriptBlock ' \
'{ Get-VM ''"' + vm_name + '"'' | get-vmsnapshot -Name ' + '"' + checkpoint_name + '" | ' \
'Restore-VMSnapshot -Confirm:$false ; Start-VM -Name ''"' + vm_name + '"''}'
util.vm(apply_vm_script)
time.sleep(10)
def check_if_vm_on(vm_name):
check_vm_script = 'Invoke-Command -Computername name -ScriptBlock { Get-VM | where {$_.Name -eq ' + vm_name + ' } | where { $_.State -eq "Running" } | select Name,State}'
util.vm(check_vm_script)
time.sleep(3)
def test():
turn_on_vm('VM_name', 'checkpoint_name')
if(check_if_vm_on('VM_name')):
Do my activity
def main():
test()
if __name__ == '__main__':
main()
Also, I can perform all if actions in PowerShell, but also can't convert bool output into Python:
if($State -like '*Running*') { Write-Output "True" }
State was defined earlier, no problem with variables.
Any ideas how to solve it?
Thank you in advance!!!

You need to get the stdout from the powershell script to your python program. This can be done with Popen.communicate().
def vm(self, apply_vm_script):
p = subprocess.Popen([self.power_shell, apply_vm_script], stdout=subprocess.PIPE)
result = p.communicate()[0]
return str(result)
Also you need to return this value from check_if_vm_on
def check_if_vm_on(vm_name):
check_vm_script = 'Invoke-Command -Computername name -ScriptBlock { Get-VM | where {$_.Name -eq ' + vm_name + ' } | where { $_.State -eq "Running" } | select Name,State}'
result = util.vm(check_vm_script)
time.sleep(3)
return result
Then you will be able to check it with the if statement:
if(check_if_vm_on('VM_name') == "True"):
Do my activity

Behave - Testing using blank Example fields

I am using Behave to automate the testing of a config file, as part of this test I need to populate various fields in the config file with invalid and blank fields. Where I am entering values I can do this using a Scenario Outline entering the values in the Examples. However when I try entering a blank field using this method Behave does not like the fact there is no value.
Is there an easy way to pass a blank value from the Examples file, or will I need to test these conditions using a separate behave test
feature
Scenario Outline:Misconfigured Identity Listener
Given an already stopped Identity Listener
And parameter <parameter> is configured to value <config_value>
When the Identity Listener is started
Then the identity listener process is not present on the system
And the log contains a <message> showing that the parameter is not configured
Examples: Protocols
|parameter |message |config_value|
|cache_ip_address | cache_ip_address | |
|cache_ip_address | cache_ip_address | 123.123.12 |
the step where I define the config value
#given('parameter {parameter} is configured to value {config_value}')
def step_impl(context, parameter, config_value):
context.parameter = parameter
context.config_value = config_value
context.identity_listener.update_config(parameter, config_value)
changing the config file using sed -i (I am interacting with a linux box in this test)
def update_config(self, param, config_value):
command = 'sudo sh -c "sed -i'
command = command + " '/" + param + "/c\\" + param + "= "+ config_value + " \\' {0}\""
command = command.format(self.config_file)
self.il_ssh.runcmd(command)
Thanks to answer from #Verv i got this working solution below
passed an empty value in for fields where I don't want a value passed
|parameter |message |config_value|
|cache_ip_address | cache_ip_address | empty |
Added an if else statement into my update config step
def update_config(self, param, config_value):
if config_value == "empty":
il_config = ""
else:
il_config = config_value
command = 'sudo sh -c "sed -i'
command = command + " '/" + param + "/c\\" + param + "= " + il_config + " \\' {0}\""
command = command.format(self.config_file)
self.il_ssh.runcmd(command)

You could put something like empty in the field, and tweak your method so that whenever the field's value is empty, you treat it as an actual empty string (i.e. "")

Why do I get E127 from this vimscript?

I have the following vimscript .vim/ftplugin dir:
" change to header file from c file or vice versa
function! CppAlter()
python << endpy
import vim
import os
bufferNames = [os.path.basename(b.name) for b in vim.buffers]
currentBufName = vim.eval("expand('%:p:t')")
currentBufStem, currentBufExt = os.path.splitext(currentBufName)
if currentBufExt == ".cpp" or currentBufExt == ".c" or currentBufExt == ".cc":
altBufName1 = currentBufStem + ".h"
altBufName2 = currentBufStem + ".hpp"
if altBufName1 in bufferNames:
vim.command("b " + altBufName1)
elif altBufName2 in bufferNames:
vim.command("b " + altBufName2)
else:
raise ValueError("No header file corresponding to this c file")
elif currentBufExt == ".h" or currentBufExt == ".hpp":
altBufName1 = currentBufStem + ".cpp"
altBufName2 = currentBufStem + ".c"
altBufName3 = currentBufStem + ".cc"
if altBufName1 in bufferNames:
vim.command("b " + altBufName1)
elif altBufName2 in bufferNames:
vim.command("b " + altBufName2)
elif altBufName3 in bufferNames:
vim.command("b " + altBufName3)
else:
raise ValueError("No c file corresponding to this header file")
else:
raise ValueError("This is not a c type file")
endpy
endfunction
nnoremap <leader>vc :call CppAlter()<cr>
inoremap <leader>vc <esc>:call CppAlter()<cr>
When I open vim I get an error:
" vim.error: Vim(function):E127: Cannot redefine function CppAlter: It is in use
But if I save it in /tmp and explicitly :so /tmp/x.vim, there is no error msg.
Wondering what is wrong here.

Inside your function, you're loading another buffer (e.g. vim.command("b " + altBufName1)). When that buffer has the same filetype, the current ftplugin script is sourced again as part of the filetype plugin handling, but the original function hasn't returned yet, so you get the E127.
Solution
I recommend putting the function itself into an autoload script, e.g. in ~/.vim/autoload/ft/cppalter.vim:
function! ft#cppalter#CppAlter()
...
Your ftplugin script becomes much smaller and efficient, as the function is only sourced once:
nnoremap <leader>vc :call ft#cppalter#CppAlter()<cr>
...
(You should probably use :nnoremap <buffer> here to limit the mapping's scope.)
Alternative
If you don't want to break this up, move the function definition(s) to the bottom and add a guard, like:
nnoremap <leader>vc :...
if exists('*CppAlter')
finish
endif
function! CppAlter()
...

I encountered an interesting case of E127, which pretty much sums up why it would occur in almost any situation. Let me explain.
First, let's look at what the docs say.
E127 E122
When a function by this name already exists and [!] is
not used an error message is given. There is one
exception: When sourcing a script again, a function
that was previously defined in that script will be
silently replaced.
When [!] is used, an existing function is silently
replaced. **Unless it is currently being executed, that
is an error.**
For the next part notice what the last line has to say.
Let's understand this by an example. Below is a function that guesses and sources the current script based on what filetype it has. Notice how the exec command will initiate an endless recursive sourcing of the current file on calling this function.
function! s:SourceScriptImplicit()
if !&readonly
w
endif
let l:bin=system("which " . &filetype)[:-2]
let l:sourcecommand=
\ #{
\ vim: "source %",
\ sh: "!source %",
\ javascript: "!node %",
\ python: "!python3 %"
\ }
exec l:sourcecommand[split(l:bin, "/")[-1]]
endfunction
To fix this, simply remove the recursive part out of the function.
function! s:SourceScriptImplicit()
if !&readonly
w
endif
let l:bin=system("which " . &filetype)[:-2]
let l:sourcecommand=
\ #{
\ vim: "source %",
\ sh: "!source %",
\ javascript: "!node %",
\ python: "!python3 %"
\ }
return l:sourcecommand[split(l:bin, "/")[-1]]
endfunction
nn <leader>so :exec <SID>SourceScriptImplicit()<cr>
Now it works perfectly!

I think you should notice that | can be useful for execute command in the bottom command line,such as :source expand("%") | source ./awesome.vim
Here is my .init.vim(or .vimrc) snippet whose func is sourcing all .vim files in /home/zarkli/.config/nvim/myInitCustom/ directory:
function SourceVimScripts()
let l:command = ""
let l:files = split(globpath('/home/zarkli/.config/nvim/myInitCustom/','*.vim'),'\n') " use absolute path to avoid problems when opening an non-nvim_init file
for l:file in l:files
let l:command .= "source ".l:file." |"
endfor
" the end of the l:command should be '|',but that doesn't matter
return l:command
endfunction
exec SourceVimScripts()
It can perfectly deals with your problem.