I am building a tool in Go which needs to provide a way to resolve variables declared in the global scope of Python scripts. In the future I would like to extend this to Node.js as well. It needs to be cross-platform.
Basically, if someone were to have the following Python code:
#!/usr/bin/env python
hello = "world"
some_var = "one"
another_var = "two"
var_three = some_func()
I would like to have access to these variable keys and values in my Golang code. In case of the function, I would like to have access to the value it returns.
My current idea is to run the script with the Golang exec.Command function and have the variables printed to its stdout in some format (e.g. JSON), which in turn can be parsed with Golang. Thoughts?
They are of different runtime environments. Golang cannot directly access variables in Python's runtime. Vica versa. You can, however, program them to pass on variable values through standard I/O or environment variables. The key is to determine the proper format for information exchanges.
For example, if the python script takes arguments as input and print the result, encoded as JSON, to the stdout. Then you can call the script with proper arguments, and decode the stdout as JSON.
Such as:
range.py
import json
import sys
def toNum(str):
return int(str)
def main(argv):
# Basically called range() with all the arguments from script call
print(json.dumps(list(range(*map(toNum, argv)))))
if __name__ == '__main__':
main(sys.argv[1:])
main.go
package main
import (
"encoding/json"
"fmt"
"log"
"os/exec"
)
func pythonRange(start, stop, step int) (c []byte, err error) {
return exec.Command(
"python3",
"./range.py",
fmt.Sprintf("%d", start),
fmt.Sprintf("%d", stop),
fmt.Sprintf("%d", step),
).Output()
}
func main() {
var arr []int
// get the output of the python script
result, err := pythonRange(1, 10, 1)
if err != nil {
log.Fatal(err)
}
// decode the stdout of the python script
// as a json array of integer
err = json.Unmarshal(result, &arr)
if err != nil {
log.Fatal(err)
}
// show the result with log.Printf
log.Printf("%#v", arr)
}
Output global variables
To output global variables in Python as JSON object:
import json
def dump_globals():
# Basically called range() with all the arguments from script call
vars = dict()
for (key, value) in globals().items():
if key.startswith("__") and key.endswith("__"):
continue # skip __varname__ variables
try:
json.dumps(value) # test if value is json serializable
vars[key] = value
except:
continue
print(json.dumps(vars))
foo = "foo"
bar = "bar"
dump_globals()
Output:
{"foo": "foo", "bar": "bar"}
You can use a main() similar to the last one for this script:
import (
"encoding/json"
"fmt"
"log"
"os/exec"
)
func pythonGetVars() (c []byte, err error) {
return exec.Command(
"python3",
"./dump_globals.py",
).Output()
}
func main() {
var vars map[string]interface{}
// get the output of the python script
result, err := pythonGetVars()
if err != nil {
log.Fatal(err)
}
// decode the json object
err = json.Unmarshal(result, &vars)
if err != nil {
log.Fatal(err)
}
// show the result with log.Printf
fmt.Printf("%#v", vars)
}
Output:
map[string]interface {}{"bar":"bar", "foo":"foo"}
Related
I would like to call a python script in my C# project , I'm using this function to do the job but unfortunately I didn't get any result and the result variable shows always an empty output. I would like to know what's the reason of this
public string RunFromCmd(string rCodeFilePath, string args)
{
string file = rCodeFilePath;
string result = string.Empty;
try
{
var info = new ProcessStartInfo(pythonPath);
info.Arguments = #"C:\Users\MyPc\ExternalScripts\HelloWorld.py" + " " + args;
info.RedirectStandardInput = false;
info.RedirectStandardOutput = true;
info.UseShellExecute = false;
info.CreateNoWindow = true;
using (var proc = new Process())
{
proc.StartInfo = info;
proc.Start();
proc.WaitForExit();
if (proc.ExitCode == 0)
{
result = proc.StandardOutput.ReadToEnd();
}
}
return result;
}
catch (Exception ex)
{
throw new Exception("R Script failed: " + result, ex);
}
}
Click Event ( Calling funtion )
private void Button1_Click(object sender, RoutedEventArgs e)
{
pythonPath = Environment.GetEnvironmentVariable("PYTHON_PATH");
RunFromCmd(pythonPath, "");
}
Python Script :
import sys
def main():
text = "Hello World"
return text
result = main()
I've fixed the issue by setting Copy if newer instead of Do Not Copy to HelloWorld.py Script
I have the following example where I want to output a variable in the same json format with os.system (the idea is to output with a system command) but the double quotes are ignored in output.
import json
import os
import requests
PAYLOAD_CONF = {
"cluster": {
"ldap": "string",
"processes": 22
}
}
paystr = (str(PAYLOAD_CONF))
paydic = (json.dumps(PAYLOAD_CONF))
os.system("echo "+paystr+"")
os.system("echo "+paydic+"")
Output:
{cluster: {processes: 22, ldap: string}}
{cluster: {processes: 22, ldap: string}}
Can you help me in a workaround where I can output this with double quotes? It's very important to output with system command.
For cases like this, where you embed variables into a shell command you should use shlex.quote. Using this (and with minor cleanup), the code can be written as:
import json
import os
import shlex
PAYLOAD_CONF = {
"cluster": {
"ldap": "string",
"processes": 22
}
}
paydic = json.dumps(PAYLOAD_CONF)
os.system("echo " + shlex.quote(paydic))
Output:
{"cluster": {"ldap": "string", "processes": 22}}
Using subprocess
The subprocess module contains a lot of helper functions for calling external applications. These functions are generally preferrable to use than os.system for various security reasons.
If there is no hard dependency of os.system you can also use one of depending on your needs:
subprocess.call -- This will return even if the subprocess fails. If this returns a non-zero value, the process exited abnormally.
subprocess.check_call -- This will raise an exception if the process exits abnormally.
subprocess.check_output -- This will return stdout from the subprocess and raise an exception if it exits abnormally.
... The module contains many other helpful functions for interacting with subprocess which you should check out if the above don't suit your needs.
using check_output, the code becomes:
from subprocess import check_call
import json
import os
import shlex
PAYLOAD_CONF = {
"cluster": {
"ldap": "string",
"processes": 22
}
}
paydic = json.dumps(PAYLOAD_CONF)
check_call(["echo", paydic])
You're not adding quotes; you're adding an empty string to the end.
Also, echo is going to interpret the first set of double quotes as a wrapper around the argument – not as part of the string itself. In the echo command itself, you need to escape double quotes with a backslash, e.g. echo \"hello\" will output "hello", whereas echo "hello" will output hello.
In a Python string, you're going to have to escape the literal backslash in the echo command with another backslash, e.g. os.system('echo \\"hello\\"') for output "hello".
Applying this to your case and using format to make it easy:
import json
import os
import requests
PAYLOAD_CONF = {
"cluster": {
"ldap": "string",
"processes": 22
}
}
paystr = (str(PAYLOAD_CONF))
paydic = (json.dumps(PAYLOAD_CONF))
os.system('echo \\"{}\\"'.format(paystr))
os.system('echo \\"{}\\"'.format(paydic))
Output:
"{cluster: {ldap: string, processes: 22}}"
"{cluster: {ldap: string, processes: 22}}"
Your paystr variable is also unnecessary, since all objects are automatically converted to strings by print and format via their inherited or overridden __str__ methods.
EDIT:
To output the variable as it appears in Python you just need to iterate through the payload dict and render each key-value pair in a type-sensitive way.
import os
import requests
def make_payload_str(payload, nested=1):
payload_str = "{\n" + "\t" * nested
for i, k in enumerate(payload.keys()):
v = payload[k]
if type(k) is str:
payload_str += '\\"{}\\"'.format(k)
else:
payload_str += str(k)
payload_str += ": "
if type(v) is str:
payload_str += '\\"{}\\"'.format(v)
elif type(v) is dict:
payload_str += make_payload_str(v, nested=nested + 1)
else:
payload_str += str(v)
# Only add comma if not last element
if i < len(payload) - 1:
payload_str += ",\n" + "\t" * nested
else:
payload_str += "\n"
return payload_str + "\n" + "\t" * (nested - 1) + "}"
PAYLOAD_CONF = {
"cluster": {
"ldap": "string",
"processes": 22
}
}
paystr = make_payload_str(PAYLOAD_CONF)
os.system('echo "{}"'.format(paystr))
Output:
{
"cluster": {
"ldap": "string",
"processes": 22
}
}
If the payload contains a dictionary, as it does in the example you provided, the function calls itself to produce the string for that dictionary, indented the right number of tabs using the nested parameter.
If the payload is also allowed to have lists and other more complex types, you'll have to include cases that account for those, but it's just more of the same.
I was considering to adapt go for my future projects for performance reason but there is a big surprise: go's running time is 13.974427s while pythons elapsed time is just 6.593028783798218s
Less then a half!
The XML file size is over 300 MB.
Here is the pythons's code:
from lxml import objectify
import time
most = time.time()
root = objectify.parse(open(r"c:\temp\myfile.xml", 'rb')).getroot()
if hasattr(root, 'BaseData'):
if hasattr(root.BaseData, 'SzTTs'):
total_records = 0
for sztt in root.BaseData.SzTTs.sztt:
total_records += 1
print("total_records", total_records)
print("Time elapsed: ", time.time()-most)
and here is the simplified go code:
package main
// An example streaming XML parser.
import (
"encoding/xml"
"fmt"
"io"
"os"
"time"
)
var inputFile = "c:\\temp\\myfile.xml"
type SzTTs struct {
Sztt []sztt
}
type sztt struct {
}
func main() {
xmlFile, err := os.Open(inputFile)
if err != nil {
fmt.Println("Error opening file:", err)
return
}
defer xmlFile.Close()
d := xml.NewDecoder(xmlFile)
total1 := 0
total2 := 0
start := time.Now()
for {
// Read tokens from the XML document in a stream.
t, err := d.Token()
// If we are at the end of the file, we are done
if t == nil || err == io.EOF {
fmt.Println("The end")
break
} else if err != nil {
log.Fatalf("Error decoding token: %s", err)
}
// Inspect the type of the token just read.
switch se := t.(type) {
case xml.StartElement:
if se.Name.Local == "SzTTs" {
var p SzTTs
// decode a whole chunk of following XML into the
// variable p which is a Page (se above)
if err = d.DecodeElement(&p, &se); err != nil {
log.Fatalf("Error decoding item: %s", err)
}
for i := range p.Sztt {
total1 = i
}
}
default:
}
}
fmt.Printf("Total sztt: %d \n", total1)
fmt.Printf("Elapsed time %s", time.Since(start))
}
Why is this big difference?
To expand upon my comment - this is expected.
Go's encoding/xml is written in pure Go, whereas lxml is written in C, essentially. It's using Cython, which generates C code from a Python-like DSL.
2x is not a huge performance difference IMHO, but if every last drop of performance matters to you, consider using another Go package - one that wraps an optimized C implementation.
For example, libxml (one of the most popular C implementations) has several wrappers:
https://github.com/moovweb/gokogiri
https://github.com/lestrrat-go/libxml2
I expect these will be much faster than encoding/xml.
Edid (2019-07-22): This question inspired me to write more about streaming XML performance in Go, and a new wrapper for the libxml SAX interface.
I have just started learning Uber's Tchannel. I'm trying to run the code from tchannel documentation in python and nodejs. In both the cases I am not able to connect the client to the server.
This is how my code looks like in nodejs, which i followed from http://tchannel-node.readthedocs.org/en/latest/GUIDE/:
var TChannel = require('tchannel');
var myLocalIp = require('my-local-ip');
var rootChannel = TChannel();
rootChannel.listen(0,myLocalIp());
rootChannel.on('listening', function onListen() {
console.log('got a server', rootChannel.address());
});
var TChannelThrift = rootChannel.TChannelAsThrift;
var keyChan = rootChannel.makeSubChannel({
serviceName: process.env.USER || 'keyvalue'
});
var fs = require('fs');
var keyThrift = TChannelThrift({
source: fs.readFileSync('./keyvalue.thrift', 'utf8')
});
var ctx = {
store: {}
};
keyThrift.register(keyChan, 'KeyValue::get_v1', ctx, get);
keyThrift.register(keyChan, 'KeyValue::put_v1', ctx, put);
function get(context, req, head, body, cb) {
cb(null, {
ok: true,
body: {
value: context.store[body.key]
}
});
}
function put(context, req, head, body, cb) {
context.store[body.key] = body.value;
cb(null, {
ok: true,
body: null
});
}
When i run this code i get this error:
node sever.js
assert.js:93
throw new assert.AssertionError({
^
AssertionError: every field must be marked optional, required, or have a default value on GetResult including "value" in strict mode
at ThriftStruct.link (/home/bhaskar/node_modules/thriftrw/struct.js:154:13)
at Thrift.link (/home/bhaskar/node_modules/thriftrw/thrift.js:199:32)
at new Thrift (/home/bhaskar/node_modules/thriftrw/thrift.js:69:10)
at new TChannelAsThrift (/home/bhaskar/node_modules/tchannel/as/thrift.js:46:17)
at TChannelAsThrift (/home/bhaskar/node_modules/tchannel/as/thrift.js:38:16)
at Object.<anonymous> (/home/bhaskar/uber/tchannel/thrift/sever.js:16:17)
at Module._compile (module.js:456:26)
at Object.Module._extensions..js (module.js:474:10)
at Module.load (module.js:356:32)
at Function.Module._load (module.js:312:12)
Similar, I have also tried the same thing in python by following http://tchannel.readthedocs.org/projects/tchannel-python/en/latest/guide.html link.
code in python looks like this:
from __future__ import absolute_import
from tornado import ioloop
from tornado import gen
from service import KeyValue
from tchannel import TChannel
tchannel = TChannel('keyvalue-server')
values={}
#tchannel.thrift.register(KeyValue)
def getValue(request):
key = request.body.key
value = values.get(key)
if value is None:
raise KeyValue.NotFoundError(key)
return value
#tchannel.thrift.register(KeyValue)
def setValue(request):
key = request.body.key
value = request.body.value
values[key] = value
def run():
tchannel.listen()
print('Listening on %s' % tchannel.hostport)
if __name__ == '__main__':
run()
ioloop.IOLoop.current().start()
when i run this with python server.py command i get ` Listening on my-local-ip:58092
`
but when i try to connect it with the client using tcurl as:
tcurl -p localhost:58092 -t ~/keyvalue/thrift/service.thrift service KeyValue::setValue -3 '{"key": "hello", "value": "world"}'
I get this:
assert.js:93
throw new assert.AssertionError({
^
AssertionError: every field must be marked optional, required, or have a default value on NotFoundError including "key" in strict mode
at ThriftException.link (/usr/lib/node_modules/tcurl/node_modules/thriftrw/struct.js:154:13)
at Thrift.link (/usr/lib/node_modules/tcurl/node_modules/thriftrw/thrift.js:199:32)
at new Thrift (/usr/lib/node_modules/tcurl/node_modules/thriftrw/thrift.js:69:10)
at new TChannelAsThrift (/usr/lib/node_modules/tcurl/node_modules/tchannel/as/thrift.js:46:17)
at asThrift (/usr/lib/node_modules/tcurl/index.js:324:18)
at onIdentified (/usr/lib/node_modules/tcurl/index.js:278:13)
at finish (/usr/lib/node_modules/tcurl/node_modules/tchannel/peer.js:266:9)
at Array.onIdentified [as 1] (/usr/lib/node_modules/tcurl/node_modules/tchannel/peer.js:257:9)
at DefinedEvent.emit (/usr/lib/node_modules/tcurl/node_modules/tchannel/lib/event_emitter.js:90:25)
at TChannelConnection.onOutIdentified (/usr/lib/node_modules/tcurl/node_modules/tchannel/connection.js:383:26)
Can anyone tell me what is the mistake?
for the node example, the thrift file in the guide needs to be updated. try to use the following (i just added a required keyword for every field):
struct GetResult {
1: required string value
}
service KeyValue {
GetResult get_v1(
1: required string key
)
void put_v1(
1: required string key,
2: required string value
)
}
I am trying to use a dll in Python source using the ctypes. I began reading it from Python documentation as I am new to this. After successfully loading the dll in Python, when I try to pass a string to a function which expects a char *, it gives me
"ValueError: Procedure probably called with too many arguments (4
bytes in excess)".
I also tried looking at other posts but couldn't resolve the problem.
I tried different approaches to pas this string such as using byref() and pointer() but it didn't change the outcome. I also tried with WINFUNCTYPE but failed. The dll that I'm using is windll.
Here's a test program I wrote in python:
from ctypes import *
lpDLL=WinDLL("C:\some_path\myDll.dll")
print lpDLL
IP_ADDR = create_string_buffer('192.168.100.228')
#IP_ADDR = "192.168.100.228"
#IP_ADDR = c_char_p("192.168.100.228")
print IP_ADDR, IP_ADDR.value
D_Init=lpDLL.D_Init
D_InitTester=lpDLL.D_InitTester
#D_InitTesterPrototype = WINFUNCTYPE(c_int, c_char_p)
#D_InitTesterParamFlags = ((1, "ipAddress", None),)
#D_InitTester = d_InitTesterPrototype(("D_InitTester", lpDLL), D_InitTesterParamFlags)
try:
D_Init()
D_InitTester("192.168.100.254")
except ValueError, msg:
print "Init_Tester Failed"
print msg
Here's how the D_InitTester is implemented in a cpp file which is available in dll exports,
D_API int D_InitTester(char *ipAddress)
{
int err = ERR_OK;
if (LibsInitialized)
{
...
some code;
...
else
{
err = hndl->ConInit(ipAddress);
}
if ( 0 < err )
{
err = ERR_NO_CONNECTION;
}
else
{
nTesters = 1;
InstantiateAnalysisClasses();
InitializeTesterSettings();
if(NULL != hndl->hndlFm)
{
FmInitialized = true;
}
}
}
else
{
err = ERR_NOT_INITIALIZED;
}
return err;
}
Your help is greatly appreciated.
Most likely the cause of the error is a mismatch of calling conventions. I'm guessing your C++ DLL exports functions with cdecl convention but your use of WinDLL implies stdcall.
Create your library like this to use cdecl:
lpDLL=CDLL("C:\some_path\myDll.dll")