We Keep Coding

How to generate a BinaryField in a mongo init shell script

In my Python code I have a UserDocument which looks like this ->
class UserDocument(DynamicDocument):
user_id = IntField()
email = StringField(unique=True)
password = BinaryField()
salt = BinaryField()
I would like to populate my database with a new user during the instantiation of my Docker container. Related to the document it's pretty easy, I just have create a shell script and copy it into docker-entrypoint-initdb.d.
Here is the script for the user creation ->
#!/bin/sh
# This is supposed to generate the BinaryField
SALT=$(xxd -u -l 64 -p /dev/urandom)
BINARY_SALT=$(echo -n $SALT | xxd -r -p)
mongo -- "inup" <<EOF
var rootUser = '$MONGO_INITDB_ROOT_USERNAME';
var rootPassword = '$MONGO_INITDB_ROOT_PASSWORD';
var inup = db.getSiblingDB('inup');
inup.auth(rootUser, rootPassword);
db.user_document.insert({
user_id: 1,
email: 'toto#gmail.com',
password: 'password123',
salt: $BINARY_SALT # This doesn't work
});
EOF
This works but I have a problem. How can I generate a BinaryField for the attribut salt ?
Have a nice days and thanks for the replies !

Binary data in MongoDB are created from Base64 string, e.g.
SALT=$(xxd -u -l 64 -p /dev/urandom)
BASE64_SALT=$(echo -n $SALT | base64)
db.user_document.insertOne({"salt" : BinData(0, "$BASE64_SALT")});
See Binary
Or you store the Base64 as string.
db.user_document.insertOne({"salt" : "$BASE64_SALT"});

Running Python Script Inside Bash Not Working When In Crontab

Hi guys I would like to ask for some help with my bash script.
I am running 2 python script inside my bash script and it is working when I'm running it manually but when I'm using cron only the commands in the .sh file is working not on .py
Please take note that I already install necessary utils and packages for python3.
This is the script:
#!/usr/bin/env bash
# list.tmp path directory
fileLoc="/home/ec2-user/PushNotification/Incoming34days/list34days.tmp"
# URL to POST request
refLink='http link'
# Title of Push Notification
title='34th day: Grace Period is about to end'
# curl type
type='Notification'
# curl action_type
actionType='NotificationActivity'
# Get the current date and time
now=$(date '+%b %d %Y %H:%M:%S')
# Message to the user
body="Subscribe to the Philippine Mobile Number plan now to continue receiving calls and texts and sending text messages to the Philippines."
# Logs location
logsLoc="/home/ec2-user/PushNotification/Incoming34days/logs.tmp"
# current number
currentNumLoc="/home/ec2-user/PushNotification/Incoming34days/currentNum.tmp"
echo "[$now] Sending notifications to mobile numbers advising today is the last day of grace period..." > $logsLoc
# Python file to SELECT all id who has 34 days counter
python3 select34days.py
# psql -d $database -t -c "SELECT id FROM svn WHERE current_date - expiry_date::DATE = 4" psql must be setup using .pgpass for postgresql authentication, please indicated database
# name and query list directory. Deleting the last line from list.txt
# This is to read the textfile list.txt line per line
while IFS='' read -r list;
# for list in `cat list.txt`;
do
# curl POST request
response=$(curl --location --request POST $refLink \
--header 'Authorization: Basic YXBwdm5vdXNlcjphcHB2bm9wYXNz' \
--header 'Content-Type: application/json' \
--data-raw '{
"title":"'"$title"'",
"body":"'"$body"'",
"min" :[{"mobileNumber" : "'"$list"'"}],
"type" : "'"$type"'",
"action_type" : "'"$actionType"'"}')
# Echo mobile number
echo "[$now] Mobile Number: $list" >> $logsLoc
# Echo response from curl
echo "Response: '$response'"
echo "[$now] Result: '$response'" >> $logsLoc
# Update the current number of the list
echo $list > $currentNumLoc
echo "[$now] Updating $list status into EXPIRED" >> $logsLoc
# Updating status into EXPIRED
python3 updateQuery34.py
done < "$fileLoc"
# end of script
The select34days.py and updateQuery34.py is not running.
I have a log.tmp to check regarding this situation and only displaying commands inside my .sh file
Inside my cron are
SHELL=/bin/bash
PATH=/sbin:/bin:/usr/bin:/usr/bin
MAILTO=root

Your PATH looks wrong:
PATH=/sbin:/bin:/usr/bin:/usr/bin
This includes /usr/bin twice which isn't necessary, but hints that something else should have been there.
Depending on how you've installed it, python might be in /usr/bin/ or /usr/local/bin or even somewhere in /opt.
At the commandline you can find python's directory using:
dirname $(which python3)
This directory needs to be added to your path in your crontab.

Just declare the specific directory with script name (eg. /etc/test.sh)every time you are using bash scripts and adding it to a cron job since the cron doesn't know where is the specific script within the the server.

Converting a BASH script to Python problem

I have the following bash script I've been trying to convert to python. I have the main part but I'm having some trouble on how to translate these two lines cut=${auth#*<ChallengeCode>} and authStr=${cut%</ChallengeCode>*}. The first request returns XML that contains <ChallengeCode> , I need to be able to extract that code and store it for future use.
BASH CODE:
#!/bin/bash
IP=IP_ADDR
USER=USERNAME
PASS=PASSWORD
auth=$(curl -ks -H "Content-Type: text/xml" -c "cookies.txt" "https://${IP}/goform/login?cmd=login&user=admin&type=1")
cut=${auth#*<ChallengeCode>}
authStr=${cut%</ChallengeCode>*}
hash=$(echo -n ${authStr}:GDS3710lZpRsFzCbM:${PASS} | md5sum | tr -d '/n')
hash=$(echo $hash | cut -d' ' -f1 | tr -d '/n')
curl -ks -H "Content-Type: text/xml" -c "cookies.txt" "https://${IP}/goform/login?cmd=login&user=admin&authcode=${hash}&type=1"
curl -ks -H "Content-Type: image/jpeg" --cookie "cookies.txt" "https://${IP}/snapshot/view0.jpg" >> snapshot.jpg
PYTHON CODE:
import requests
import hashlib
hmd5 = hashlib.md5()
ip = "192.168.100.178"
user = "admin"
password = "Password1"
auth = requests.get('https://{0}/goform/login', headers=headers, params=params, verify=False).format(ip)
chcode = (This is where I want to put the challenge code i get back from the previous request)
hstring = "{0}:GDS3710lZpRsFzCbM:{1}".format(chcode,password).encode()
hmd5.update(hstring)
hashauth = hmd5.hexdigest()
response = requests.get('https://{0}/snapshot/view0.jpg', headers=headers, cookies=cookies, verify=False).format(ip)
Any advice on how I could better improve the code would also be appreciated.

If your request returns XML, it'd be suitable to use a XML parser. Presuming you've imported xml.etree.ElementTree perhaps with:
import xml.etree.ElementTree as ET
You can have it parse your response:
root_el = ET.fromstring(auth.text)
And then use XPath (might be different depending on structure of your XML) to find your element and get value of text it contains:
chcode = root_el.find("./ChallengeCode").text

While one of the virtues of a real programming language is the availability of powerful libraries (e.g., for parsing XML), there is a direct analog of your Bash substring operations that is particularly simple given the limited use of wildcards:
${a#*foo} — a.partition("foo")[0]
${a%foo*} — a.rpartition("foo")[-1]
${a##*foo} — a.rpartition("foo")[0]
${a%%foo*} — a.partition("foo")[-1]

Using gpg --search-keys in --batch mode

I'm working on an application that will eventually graph the gpg signature connections between a predefined set of email addresses. I need it to programmatically collect the public keys from a key server. I have a working model that will use the --search-keys option to gpg. However, when run with the --batch flag, I get the error "gpg: Sorry, we are in batchmode - can't get input". When I run with out the --batch flag, gpg expects input.
I'm hoping there is some flag to gpg that I've missed. Alternatively, a library (preferably python) that will interact with a key server would do.

Use
gpg --batch --keyserver hkp://pool.sks-keyservers.net --search-keys ...
and parse the output to get key IDs.
After that
gpg --batch --keyserver hkp://pool.sks-keyservers.net --recv-keys key-id key-id ..
should work

GnuPG is not performing very well anyway when you import very large portions of the web of trust, especially during the import phase.
I'd go for setting up a local keyserver, just dumping all the keys in there (less than 10GB of download size in 2014) and directly querying your own, local keyserver.
Hockeypuck is rather easy to setup and especially query, as it stores the data in a PostgreSQL database.

Use --recv-keys to get the keys without prompting.

In the case of a hkps server the following would work :
gpg --keyserver hkps://***HKPSDOMAIN*** --recv-keys \
$(curl -s "https://***HKPSDOMAIN***/?op=index&options=mr&search=***SEARCHSTRING***"\
|grep pub|awk -F ":" '{print $2}')

We can store the std and err output of the gpg --search-keys commands into variables by specifying 2>&1, then do work on those variables. For example, get the public key ids or those with *.amazon.com email addresses:
pubkeyids=$(gpg --batch --keyserver hkp://keyserver.ubuntu.com --search-keys amazon.com 2>&1 | grep -Po '\d+\s*bit\s*\S+\s*key\s*[^,]+' | cut -d' ' -f5)
The regular expression is fully explained on regex101.com. We can automate searching for keys by their IDs and add them to the keyring using bash by parsing that output. As an illustration, I created the following GitHub gist to host the code below.
Example address list example.csv:
First Name
Last Name
Email Address
Hi
Bye
hi#bye.com
Yes
No
yes#no.com
Why
Not
why#not.com
Then we can pass the csv path to a bash script which will add all keys belonging to the email addresses in the csv:
$ getPubKeysFromCSV.sh ~/example.csv
Here is an implementation of the above idea, getPubKeysFromCSV.sh:
# CSV of email address
csv=$1
# Get headers from CSV
headers=$(head -1 $csv)
# Find the column number of the email address
emailCol=$(echo $headers | tr ',' '\n' | grep -n "Email Address" | cut -d':' -f1)
# Content of the CSV at emailCol column, skipping the first line
emailAddrs=$(tail -n +2 $csv | cut -d',' -f$emailCol)
gpgListPatrn='(?<entropy>\d+)\s*bit\s*(?<algo>\S+)\s*key\s*(?<pubkeyid>[^,]+)'
# Loop through the array and get the public keys
for email in "${emailAddrs[#]}"
do
# Get the public key ids for the email address by matching the regex gpgListPatt
pubkeyids=$(gpg --batch --keyserver hkp://keyserver.ubuntu.com --search-keys $email 2>&1 | grep -Po $gpgListPatrn | cut -d' ' -f5)
# For each public key id, get the public key
for pubkeyid in $pubkeyids
do
# Add the public key to the local keyring
recvr=$(gpg --keyserver hkp://keyserver.ubuntu.com --recv-keys $pubkeyids 2>&1)
# Check exit code to see if the key was added
if [ $? -eq 0 ]; then
# If the public key is added, do some extra work with it
# [do stuff]
fi
done
done
If we wanted, we could make getPubKeysFromCSV.sh more complex by verifying a file signature in the body of the loop, after successfully adding the public key. In addition to the CSV path, we will pass the signature path and file path as arguments two and three respectively:
$ getPubKeysFromCSV.sh ~/example.csv ./example.file.sig ./example.file
Here is the updated script difference as a diff:
--- original.sh
+++ updated.sh
## -1,6 +1,12 ##
# CSV of email address
csv=$1
+# signature file
+sig=$2
+
+# file to verify
+file=$3
+
# Get headers from CSV
headers=$(head -1 $csv)
## -22,5 +28,17 ##
recvr=$(gpg --keyserver hkp://keyserver.ubuntu.com --recv-keys $pubkeyids 2>&1)
# Check exit code to see if the key was added
+ if [ $? -eq 0 ]; then
+ verify=$(gpg --batch --verify $sig $file 2>&1)
+ # If the signature is verified, announce it was verified
+ # else, print error not verified and exit
+ if [[ $verify =~ "^gpg: Good signature from" ]]; then
+ echo "$file was verified by $email using $pubkeyid"
+ else
+ printf '%s\n' "$file was unable to be verified" >&2
+ exit 1
+ fi
+ fi
done
done

Run shell commands from python and string concatenation

I am trying to adjust the "post to dropbox" services for Snow loepard (http://dl.dropbox.com/u/1144075/Post%20to%20Dropbox.zip).
I dont want the public URL, but a shortened one from goo.gl
Therefor I am using these shell commands:
curl -s --data-urlencode "url=http://link.com" http://googl/action/shorten | grep "googl" | awk -F\" '{print $(NF-1)}' | awk 'BEGIN { FS = "=" } ; { print $2}' | pbcopy
Now the python script does this to copy a dropbox URL for all the files it just copies in the public folder to the clipboard:
pasteURLs = []
for file in copied_files: # for all elements in our list
components = file.split(os.sep) # seperate the path
local_dir = os.sep.join(components[5:]) # cut off the beginning
local_dir = urllib.quote(local_dir) # convert it to a URL (' ' -> '%20', etc.)
#construct the URL
finalURL = 'http://dl.dropbox.com/u/%s/%s' % ( dropbox_id, local_dir )
pasteURLs.append(finalURL) # add the current URL to the string
copy_string = "\n".join(pasteURLs)
os.system( "echo '%s' | pbcopy" % (copy_string) ) # put the string into clipboard
I have to admit I dont know anything about python, but from what it looks like, I need to change the last two lines with this:
shortURL = []
for thisURL in pasteURLs:
shortURL = os.system( curl -s --data-urlencode "url=http://link.com" http://googl/action/shorten | grep "goo.gl" | awk -F\" '{print $(NF-1)}' | awk 'BEGIN { FS = "=" } ; { print $2}' | pbcopy )
shortURLs.append(shortURL)
copy_string = "\n".join(shortURLs)
os.system( "echo '%s' | pbcopy" % (copy_string) ) # put the string into clipboard
But my problem is, how to put the correct URL in the command? As u can see it says http://link.com But it should use thisURL instead.
Any ideas? Thanks in advance!

I think your os.system call should look something like this:
os.system("curl -s --data-urlencode \"url=%s\" http://goo.gl/action/shorten | grep \"goo.gl\" | awk -F\\\" '{print $(NF-1)}' | awk 'BEGIN { FS = \"=\" } ; { print $2}' | pbcopy " % thisURL)

UPDATE I wrote the script for you and used a much simpler command pipeline. Not that the entire thing could be done in python without curl, but here it is.
import subprocess
thisURL = 'http://whatever.com'
pipeline = []
pipeline.append('curl -s -i --data-urlencode "url=%s" ' % thisURL +
'http://goo.gl/action/shorten')
pipeline.append('grep Location:')
pipeline.append('cut -d = -f 2-')
#pipeline.append('pbcopy')
command = "|".join(pipeline)
link, ignore = subprocess.Popen(command, stdout=subprocess.PIPE,
shell=True).communicate()
print link

Other answers have already provided the core of this: use quotation marks around your command, use a format string to insert the value and consider using subprocess in order to actually get the output from the command.
However, if you, like me, think this is getting a bit too convoluted, go have a look at this example on how to do the actual shortening in python. If you're new to python, this might mean you'll need to read up on your exception handling to understand it. (It also looks like you might need a custom module, but then again it appears to only be used if you get an exception...)