We Keep Coding

Python not printing output

I am learning to use electron js with python and I am using python-shell so I have the following simple python script:
import sys, json
# simple JSON echo script
for line in sys.stdin:
print(json.dumps(json.loads(line)))
and in my main.js:
let {PythonShell} = require('python-shell')
let pyshell = new PythonShell('/home/bassel/electron_app/pyapp/name.py', {mode : 'json'});
pyshell.send({name:"mark"})
pyshell.on('message', function (message) {
// received a message sent from the Python script (a simple "print" statement)
console.log("hi");
});
but the hi is not getting printed, what is wrong?
This problem can also occur when trying to suppress the newline from the end of print output. See Why doesn't print output show up immediately in the terminal when there is no newline at the end?.

Output is often buffered in order to preserve system resources. This means that in this case, the system holds back the Python output until there's enough to release together.
To overcome this, you can explicitly "flush" the output:
import sys, json
# simple JSON echo script
for line in sys.stdin:
print(json.dumps(json.loads(line)))
sys.stdout.flush() # <--- added line to flush output
If you're using Python 3.3 or higher, you may alternatively use:
import sys, json
# simple JSON echo script
for line in sys.stdin:
print(json.dumps(json.loads(line)), flush=True) # <--- added keyword

same string gives different result in Python

So I'm using approach in this post
to extract a double quoted string from a string. If the input string comes from terminal argument, it works fine. But if the input string comes from a txt file like the following, it gives nontype error. I tried to get the hash code for two strings(one from file and one from terminal) with identical txt content, and turns out they are different. I'm curious if anyone knows how to solve this?(in Python 3.x)
That said, I have set the default encoding to "utf-8" in my code.
python filename.py < input.txt

If you are using command python, the command recognize it to python 2.x.
If you want python 3.x, just change the command to python3
like this
python3 filename.py < input.txt

Two things, if you want to ingest a txt file into a python script, you need to specify it. Add these two lines
import sys
text = str(sys.argv[1])
this mean text would be your 'input.txt'.
Second, if your script has only a function, it would not know what you want to do with the function, you have to either, tell the script explicity to execute the function through the entry main
import re
import sys
def doit(text):
matches=re.findall(r'\"(.+?)\"',text)
# matches is now ['String 1', 'String 2', 'String3']
return ",".join(matches)
if __name__ == '__main__':
text_file = str(sys.argv[1])
text = open(text_file).read()
print(doit(text))
Alternately, you can just execute line by line without wrapping the re in a function, since it is only one line.

I just figure it out, the bug doesn't come from my code. I had the "smart quotes" enabled on my Mac so whenever it reads a quote, it's identified as a special character. Disable this under keyboard setting would do the trick.
LOL what a "bug".

Using Makefile bash to save the contents of a python file

For those who are curious as to why I'm doing this: I need specific files in a tar ball - no more, no less. I have to write unit tests for make check, but since I'm constrained to having "no more" files, I have to write the check within the make check. In this way, I have to write bash(but I don't want to).
I dislike using bash for unit testing(sorry to all those who like bash. I just dislike it so much that I would rather go with an extremely hacky approach than to write many lines of bash code), so I wrote a python file. I later learned that I have to use bash because of some unknown strict rule. I figured that there was a way to cache the entire content of the python file into a single string in the bash file, so I could take the string literal in bash and write to a python file and then execute it.
I tried the following attempt (in the following script and result, I used another python file that's not unit_test.py, so don't worry if it doesn't actually look like a unit test):
toStr.py:
import re
with open("unit_test.py", 'r+') as f:
s = f.read()
s = s.replace("\n", "\\n")
print(s)
And then I piped the results out using:
python toStr.py > temp.txt
It looked something like:
#!/usr/bin/env python\n\nimport os\nimport sys\n\n#create number of bytes as specified in the args:\nif len(sys.argv) != 3:\n print("We need a correct number of args : 2 [NUM_BYTES][FILE_NAME].")\n exit(1)\nn = -1\ntry:\n n = int(sys.argv[1])\nexcept:\n print("Error casting number : " + sys.argv[1])\n exit(1)\n\nrand_string = os.urandom(n)\n\nwith open(sys.argv[2], 'wb+') as f:\n f.write(rand_string)\n f.flush()\n f.close()\n\n
I tried taking this as a string literal and echoing it into a new file and see whether I could run it as a python file but it failed.
echo '{insert that giant string above here}' > new_unit_test.py
I wanted to take this statement above and copy it into my "bash unit test" file so I can just execute the python file within the bash script.
The resulting file looked exactly like {insert giant string here}. What am I doing wrong in my attempt? Are there other, much easier ways where I can hold a python file as a string literal in a bash script?

the easiest way is to only use double-quotes in your python code, then, in your bash script, wrap all of your python code in one pair of single-quotes, e.g.,
#!/bin/bash
python -c 'import os
import sys
#create number of bytes as specified in the args:
if len(sys.argv) != 3:
print("We need a correct number of args : 2 [NUM_BYTES][FILE_NAME].")
exit(1)
n = -1
try:
n = int(sys.argv[1])
except:
print("Error casting number : " + sys.argv[1])
exit(1)
rand_string = os.urandom(n)
# i changed ""s to ''s below -webb
with open(sys.argv[2], "wb+") as f:
f.write(rand_string)
f.flush()
f.close()'

How to pass collection data from Python to bash?

I am writing a dev ops kind of a bash script that is used for running an application in a local development environment under configuration as similar to production as possible. To eliminate duplicating some code/data which is already in a Python script, I would like my bash script to invoke a Python call to retrieve data that is hard coded in that Python script. The data structure in Python is a dict but I really only care about the keys so I can just return an array of keys. The Python script is used in production and I want to use it and not duplicate the data in my shell script to avoid having to follow on any modification in the production script with parallel changes in the local environment shell script.
Is there any way I can invoke a Python function from bash and retrieve this collection of values? If not, should I just have the Python function print to STDOUT and have the shell script parse the result?

Yes, that is best and almost only way to pass data from python to bash.
Also your function can write to file, which would be read by bash script.

To write a Python dictionary from a module out to a NUL-delimited key/value stream (which is the preferred serialization format if you want to represent the full range of values bash is capable of handling):
#!/usr/bin/env python
import sys, yourmodule
saw_errors = 0
for k, v in yourmodule.data.iteritems():
if '\0' in k or '\0' in v:
saw_errors = 1 # setting exit status is nice-to-have but not essential
continue # ...but skipping invalid content is important; otherwise,
# we'd corrupt the output stream.
sys.stdout.write('%s\0%s\0' % (k, v))
sys.exit(saw_errors)
...and to read that stream into an associative array:
# this is bash 4.x's equivalent to a Python dict
declare -A items=()
while IFS= read -r -d '' key && IFS= read -r -d '' value; do
items[$key]=$value
done < <(python_script) # where 'python_script' behaves as given above
...whereafter you can access the items from your Python script:
echo "Value for hello is: ${items[hello]}"
...or iterate over the keys:
printf 'Received key: %q\n' "${!items[#]}"
...or iterate over the values:
printf 'Received value: %q\n' "${items[#]}"
Caveat: Python bytestrings (regular strings, in Python 2.x) are Pascal-style; they have an explicit length stored, so they can contain any raw binary data whatsoever. (Python 3.x character strings are also Pascal-style, and can also contain NULs, but the aforementioned sentence doesn't quite apply as they don't contain raw binary content -- while the next one still does). Bash strings are C strings; they're NUL-terminated, so they can't contain raw NUL characters.
Thus, some data which can be represented in Python cannot be represented in bash.

As an alternative, you could make a python script that prints out a bash array.
bashify.py
#! /usr/bin/python
from sys import argv
from importlib import import_module
def as_bash_array(mapping):
return " ".join("[{!r}]={!r}".format(*item) for item in mapping.items())
def get_mapping(name):
module, var = name.rsplit(".", 1)
return getattr(import_module(module), var)
executable, mapping_name = argv
mapping = get_mapping(mapping_name)
print "(", as_bash_array(mapping), ")"
usage:
declare -A my_arr="`./bashify.py my_module.my_dict`"
Using !r in the format string means non-printing characters such as NUL will be escaped ("\x00" for NUL). It also means that string values will be quoted -- allowing characters that would otherwise break the array declaration syntax.

using a python list as input for linux command that uses stdin as input

I am using python scripts to load data to a database bulk loader.
The input to the loader is stdin. I have been unable to get the correct syntax to call the unix based bulk loader passing the contents of a python list to be loaded.
I have been reading about Popen and PIPE but they have not been behaving as i expect.
The python list contains database records to be bulkloaded. In linux it would look similar to this:
echo "this is the string being written to the DB" | sql -c "COPY table FROM stdin"
What would be the correct way replace the echo statement with a python list to be used with this command ?
I do not have sample code for this process as i have been experimenting with the features of Popen and PIPE with some very simple syntax and not obtaining the desired result.
Any help would be very much appreciated.
Thanks

If your data is short and simple, you could preformat the entire list and do it simple with subprocess like this:
import subprocess
data = ["list", "of", "stuff"]
proc = subprocess.Popen(["sql", "-c", "COPY table FROM stdin"], stdin=subprocess.PIPE)
proc.communicate("\n".join(data))
If the data is too big to preformat like this, then you can attempt to use the stdin pipe directly, though subprocess module is flaky when using the pipes if you need to read from stdout/stderr too.
for line in data:
print >>proc.stdin, line