I'm currently writing a dockerfile, in which the project's lines of code shall be counted via cloc, and sent to a server. The server is providing a RESTful API. From this API I want to get a Data-Transfer-Object, modify it and send it back to the same API to update it.
Since the runner does not memorize variables over more than one line, I (probably) have to put everything into one line (except the static API):
ENV API=http://example-url.com/api/datapoints/
My curl command that I go so far is this:
RUN curl
-d (curl -s ${API} | python3 -c "import sys, json;
newDatapointDTO=json.load(sys.stdin);
newDatapointDTO['metric']="`cloc ./src/ --json`";
print(newDatapointDTO)")
-H "Content-Type: application/json"
-X PUT ${API}
The problem here is that the runner does not like the interruption of the python command in order to set in a variable from bash.. So how do I insert cloc ./src/ --json then into Python?
I also think the command is a bit too complicated - there should be probably a better solution for this...
Converting your command to JSON using jq -Rs . <<'EOF', pasting your command, and then EOF gives us:
RUN ["bash", "-c", "curl \n -d (curl -s ${API} | python3 -c \"import sys, json; \n newDatapointDTO=json.load(sys.stdin); \n newDatapointDTO['metric']=\"`cloc ./src/ --json`\"; \n print(newDatapointDTO)\") \n -H \"Content-Type: application/json\" \n -X PUT ${API}\n"]
Taking out the extra whitespace and fixing up the quoting makes that:
RUN ["bash", "-c", "curl -d (curl -s ${API} | python3 -c \"import sys, json;\nnewDatapointDTO=json.load(sys.stdin);\nnewDatapointDTO['metric']=$(cloc ./src/ --json);print(newDatapointDTO)\") -H \"Content-Type: application/json\" -X PUT ${API}"]
However, because that code is using shell injection to insert data into Python, it's still a Really Bad Idea. A saner version of your Python code might instead look like:
import sys, json, subprocess
newDatapointDTO=json.load(sys.stdin)
newDatapointDTO['metric'] = subprocess.run(['cloc', './src/', '--json'], check=True, stdout=subprocess.PIPE).stdout
print(newDatapointDTO)
...so wrapping that in a shell command might look like:
pythonScript="import sys, json, subprocess
newDatapointDTO=json.load(sys.stdin)
newDatapointDTO['metric'] = subprocess.run(['cloc', './src/', '--json'], check=True, stdout=subprocess.PIPE).stdout
print(newDatapointDTO)"
curl -d "$(curl -s "$API" | python -c "$pythonScript")" \
-H "Content-Type: application/json" \
-X PUT "$API"
...so, generating a JSON encoding of a command line that calls that would look like:
jq -Rs . <<'EOF'
pythonScript="import sys, json, subprocess
newDatapointDTO=json.load(sys.stdin)
newDatapointDTO['metric'] = subprocess.run(['cloc', './src/', '--json'], check=True, stdout=subprocess.PIPE).stdout
print(newDatapointDTO)"
curl -d "$(curl -s "$API" | python -c "$pythonScript")" \
-H "Content-Type: application/json" \
-X PUT "$API"
EOF
...which gives us the output:
"pythonScript=\"import sys, json, subprocess\nnewDatapointDTO=json.load(sys.stdin)\nnewDatapointDTO['metric'] = subprocess.run(['cloc', './src/', '--json'], check=True, stdout=subprocess.PIPE).stdout\nprint(newDatapointDTO)\"\n\ncurl -d \"$(curl -s \"$API\" | python -c \"$pythonScript\")\" \\\n -H \"Content-Type: application/json\" \\\n -X PUT \"$API\"\n"
...so we know you can put in your Dockerfile the following:
RUN ["bash", "-c", "pythonScript=\"import sys, json, subprocess\nnewDatapointDTO=json.load(sys.stdin)\nnewDatapointDTO['metric'] = subprocess.run(['cloc', './src/', '--json'], check=True, stdout=subprocess.PIPE).stdout\nprint(newDatapointDTO)\"\n\ncurl -d \"$(curl -s \"$API\" | python -c \"$pythonScript\")\" \\\n -H \"Content-Type: application/json\" \\\n -X PUT \"$API\"\n"]
Related
I am trying to replicate shell command:
curl -u [staff_email]:[api_key] -F "case[attachments][0]=#/path/to/file1.ext" -F "case[attachments][1]=#/path/to/file2.ext" -F "case[content]=I need help" -F "case[subject]=I need help" -F "case[user_email]=user#domain.ru" -F "case[user_full_name]=FullName" -F "case[language_id]=2" -F "case[custom_fields][cf_44]=3" -X POST https://[domain].omnidesk.ru/api/cases.json
into Python code using Requests library:
import requests
What is a proper syntax for this example (curl -X POST)?
You can use requests.post, as explained here
for the attachments part (your first -F), read here
for the auth part (-u): here
I have a curl command, Which works on my UNIX server.I want to execute it using python script.
curl comma:
curl -X POST -H "Content-type: application/vnd.appd.cntrl+json;v=1" -d '{"name":"Suppression_Testing","timeRange": {"startTimeMillis": "2017-12-25T04:16:30+0000","endTimeMillis": "2017-12-26T06:16:30+0000"},"affects": {"type": "APP"}}' --user api#adgm-nonprod:ac#123
https://apcn.adm.com/controller/api/accounts/3/applications/61/actionsuppressions
I tried import os and executed curl butI am getting error as POST invalid syntax.
Also I tried using subprocess
subprocess.Popen(curl 0) but no luck. Can anyone help me?
Many Thanks,
Apurva Acharya
you can use python , subprocess module to do this
from subprocess import Popen
command='''curl -X POST -H "Content-type: application/vnd.appd.cntrl+json;v=1" -d '{"name":"Suppression_Testing","timeRange": {"startTimeMillis": "2017-12-25T04:16:30+0000","endTimeMillis": "2017-12-26T06:16:30+0000"},"affects": {"type": "APP"}}' --user api#adgm-nonprod:ac#123'''
#use shell=True , this will allow you to run the command in a single string on a shell like environment
proc=Popen(command,shell=True)
I have this command
curl http://localhost:6800/schedule.json | python -c "import json; print (' '.join(['-d '+key+'='+word for key,word in json.load(open('cron_parameters_testing.json')).items() ] ))"
actually the python command reads parameters from JSON file like this
{
'project' : 'default'
}
and returns the output as -d project=default
I have separately tested python -c "import json; print (' '.join(['-d '+key+'='+word for key,word in json.load(open('cron_parameters_testing.json')).items() ] )) command it works perfectly but it does work with the cURL
my final command that I want to run is
curl http://localhost:6800/schedule.json -d project=default
of course I want -d project=default to be generated from command I mentioned
| is a pipe, in your case it sends standard output from curl to become the standard input to python. This seems unrelated to your task. Try:
curl http://localhost:6800/schedule.json `python -c "import json; print (' '.join(['-d '+key+'='+word for key,word in json.load(open('cron_parameters_testing.json')).items() ] ))"`
The backticks (`...`) evaluate the command inside and get substituted with the output of the command. Alternatively you can use $(...), especially if you need to nest commands like this.
I've got the following bash script to parse endpoints JSON:
echo curl -s -H "X-Auth-Token: my_access_token" -X GET "https://api.selvpc.ru/identity/v3/endpoints?interface=public" | python -mjson.tool | grep -Pi '^\s*"url":\s*".*",?$' | awk '{print $2}' | tr -d '"' | sed "s/[%\\\$](tenant_id)s/my_project_id/g")
But bash says:
-bash: syntax error near unexpected token `)'
My hoster says, this script works well on Linux-based OS, but no guarantee to work on OS X. What can be the syntax issue?
EDIT:
If i use the following:
curl -s -H "X-Auth-Token: my_access_token" -X GET "https://api.selvpc.ru/identity/v3/endpoints?interface=public" | python -mjson.tool
JSON parses as expected. But with grep -Pi '^\s*"url":\s*".*",?$' I guess it causes grep warning
usage: grep [-abcDEFGHhIiJLlmnOoqRSsUVvwxZ] [-A num] [-B num] [-C[num]]
[-e pattern] [-f file] [--binary-files=value] [--color=when]
[--context[=num]] [--directories=action] [--label] [--line-buffered]
[--null] [pattern] [file ...]
I guess the first problem is grep error?
As #4ae1e1 suggested, please use a JSON processor for the job. jq is great and it's worthwhile investing your time to learn it.
wget https://github.com/stedolan/jq/releases/download/jq-1.5/jq-osx-amd64
mv jq-osx-amd64 jq
chmod u+x jq
curl -s -H "X-Auth-Token: $TOKEN" https://api.selvpc.ru/identity/v3/endpoints?interface=public | \
./jq -r .endpoints[].url
That will get you a list of OpenStack API endpoints.
I think a python script using python-keystoneclient can be easier to understand and maintain
Python newb here, here's my current script:
#!/usr/bin/env python
import os
import time
import datetime
import subprocess
ts = time.time()
st = datetime.datetime.fromtimestamp(ts).strftime(%Y%m%d)
My curl command:
curl -i -k -H -o <timestamp>.txt "Accept: application/json" -H "Content-Type: application/json" -H "appID:******" -H "appKey:*******" -X GET https://*****.com/csOpen/workplace/hr/v1/employees_s?type=EMP&location=******&term_from=<timestamp>
The dynamic aspect of this curl request comes from the python portion of my script. I need the output file to be the $currentTime.txt and i need the php variable $term_from to also be the timestamp.
So far ive tried invoking the curl command using
os.system('curl -i -k -H -o' + %st + '.txt "Accept: application/json" -H "Content-Type: application/json" -H "appID:arcsght_app" -H "appKey:*****" -X GET https://csopen.teamaol.com/csOpen/workplace/hr/v1/employees_s?type=EMP&location=Dulles&term_from=' + %st)
That didn't work, then i tried using
subprocess.call(<same curl command as above>)
and that didnt work.
Ive tried my curl command from bash and it works, and i can get my timestamp to show how i need it. I just cant figure out how to tie everything together. Before i posted this i did try to figure it out on my own, but this is my first real adventure into python so my knowledge of what works and what doesn't is pretty slim. Looking for help! Thanks.
my_command = 'curl -i -k -H -o {timestamp}.txt "Accept: application/json" -H "Content-Type: application/json" -H "appID:******" -H "appKey:*******" -X GET https://*****.com/csOpen/workplace/hr/v1/employees_s?type=EMP&location=******&term_from={timestamp}'.format(timestamp=123456)
os.system(my_command)
should work fine ... Im not entirely sure why you want to do this in python... but this should allow you to no problem