Everything ok with curl:
curl -v "http://user:password#localhost/_control.html" -d $'data1=1\r\n'
I tried this way in python:
url = "http://localhost/_control.html"
payload = {'data1': '1\r\n'}
headers = {"Content-type": "application/x-www-form-urlencoded", "Accept": "text/plain"}
r = requests.post(url, data=payload, headers=headers, auth=('user', 'password'))
But it doesn't work. Content-length in this case is 13 instead of 9 (with curl request)
Is it possible to send same data (with \r\n at the end) using python requests?
The \r and \n characters are being URL-encoded, as they should be, because application/x-www-form-urlencoded data cannot contain those characters directly:
Non-alphanumeric characters are replaced by %HH, a percent sign and two hexadecimal digits representing the ASCII code of the character. Line breaks are represented as "CR LF" pairs (i.e., %0D%0A).
Logging the request sent from Python using this technique, we can see that it correctly sends these 13 bytes:
data1=1%0D%0A
In the case of curl, its manual page makes no mention of encoding of whatever you pass to -d/--data, so presumably you're expected to encode the string yourself before passing it to curl. We can confirm this with --trace-ascii -:
=> Send data, 9 bytes (0x9)
0000: data1=1
The \r\n pair doesn't show up clearly here, but we can infer that it's not encoded because of the byte count.
In short, the request you are sending with that curl command is not valid to begin with.
Related
I am trying to convert this curl command in python requests, but I am unsure about how I should pass the data:
curl -X POST -u "apikey:{apikey}" \
--header "Content-Type: text/plain" \
--data "some plain text data" \
"{url}"
I have tried to pass the string directly and to encode it with str.encode('utf-8') but I get an error 415 Unsupported Media Type
This is my code:
text = "some random text"
resp = requests.post(url, data=text, headers={'Content-Type': 'text/plain'}, auth=('apikey', self.apikey))
When using requests library, it is usually a good idea not to set Content-Type header manually using headers= keyword.
requests will set this header for you if it is needed (for example posting JSON will always result in Content-Type: application/json header).
Another reason for not setting this type of header manually is encoding, because sometimes you should specify something like Content-Type: text/plain; charset=utf-8.
One more important thing about Content-Type is that this header is not required for making POST requests. RFC 2616:
Any HTTP/1.1 message containing an entity-body SHOULD include a
Content-Type header field defining the media type of that body. If
and only if the media type is not given by a Content-Type field, the
recipient MAY attempt to guess the media type via inspection of its
content and/or the name extension(s) of the URI used to identify the
resource. If the media type remains unknown, the recipient SHOULD
treat it as type "application/octet-stream".
So depending on the server type you're making request to, this header may be left empty.
Sorry for this explanation being a bit vague. I cannot give you an exact explanation why this approach worked for you unless you provide target URL.
I'm catching HTTP requests made by Selenium using Browser Mob Proxy (using the browsermob-proxy Python package). In my HAR file, I see this (it should be a Javascript file):
"content": {
"comment": "",
"size": 10908,
"mimeType": "application/x-javascript; charset=utf-8",
"encoding": "base64",
"text": "4qF2FJIUvnlRlDNSpoB6KeBkiFADdf4D8QYKthNRVc+tMqHv1vuipJbkMKsgPuuM4/FG3RVcLlYhxHmH35SDP/NTPTs+PWuP0OSUmpep2uur7Gg0QAyJLhUO46Fajm7qKW2HIkAKFgUwAKh2pplqVvWqnKqQtoMInI5jpiEhh7QO8b+zQ+HSllatf4Oq8hdgAzWVjJFBy4K7J02g0CWJaE3ikUbWh96mImD0tVD/hsgTm03zkZSKPTbyuux6ZjVB07R8fS9NIHHcSSsOkdejy1htoCuWa6qNMM7z2VERAsm2YnW3oRpFsJkUvrT3/99X7Z1ffUj6En9wyp2bLmQNl4tmOqemFs7Z517zBcDzAJBLIEGuERUmkNLE8M4NjyCCDEDgMIy0VvwpaBxCrmLponHp1adYxq6kOjeldTMD6cmsD03id+swTblb2+2X1oTXgDEGDCHbH9Pv+feptbckYYUdiFQCJAkgFKl3f8tnz3Np6clo7Ew75vzG+LaQOB1e21ynJe35eaQA+I2sEYlCgm6h+3cbWLVx4P2jO4754x/xT0lPemYveuX+jAjSCQeYXsCHqHK+xS9AIgSCvgDnhNzk3IcLxJEmwqIAJz+ccdWDC0AiQxA8QvMlzrniwQWgl6P//xfAEdO2ruZucLVYmUp6L8E5KE8gPqpjGVdkWCnJMJDqyq3bEkxdVBdt/dWpFB/n7JTjKAWquRsjYY49sj6MU0mu8+tOomC5okbz1zk+mvUs80FeweRRKRYW4FyjaRrQBbAOCQRMnIgzT/64QJrkePBXCl7JmhzbXBUevKs1h14bjs1sUmtKqH+rLB0fQUqwhDTrVBagWlKH5wlnIY2dCSIkOfKSIVexLI+XuLxVOn/IghTLTaLnbwpql4VSlGrJbyLmJMdECxS2wSLYQVOELiAoMBGGbBd9sD0GepfYYnCfAeElLGvqTq0ShuoNNuhNfKW/rO/0l/Wl/rC+xRoqZxy6pA2MmmIat8dp0tUbzfRBSHijlOJhbcHmNh9+x7v4mozNGIBkYoahaA2zpiBeNhj0Ej0RA6wKLIDyB4EPPygG2w9rPs1RumMd5jI8FKD9dFvMjNo0aIbaV0L8GeSZbYL3U6tOHJDs6m8CTv3fJO83U2oqb0Z/OqF3s6YaofjpKWJodgmTDYQoe34GirhT48/d6KtVqYeopRxkma1jkwh3VvyLbZKt7EiC218Pgzhd8S9dRwZyWuVy7DZNVTfnhuAdstuEMJE3buPKTWXxh2gHe3vMOMAu/UgDJEI2dWlPZY9lMxAbZhX6mmZS4nBJ6pmMsC399L5U8P5wXF52C/gLHJyh0gG/tEYaCnTYOkGBMxeKvdVCCzhjF9F2N0FEqpda6AbF5dqEcqA9LeVWfVB593KW5d0UNzLZfKA7KkyIaxv0frIe3pArWTII45gE6bKRfKOds+qY2Rw7d25jzdM+izE3iwAV/yVQjfsqQyzYUD9og43+lKQ31pvqh2OPUusJtMwgwRwzgsrNJV6+EfV6wxTI5gpTLWtDOBlwqzc9mDSPP8s8V9Bk9K9AZEDwc4liGpTxJORUggJzI8wi63Ef9VnABBvbgqTMVRCaits9Ct36UWYhFA+E9JUzaxjRYdE5lB+fFUHiMGpmLtBIQ7tMvtgYZ4wOTYE6mrr1+HEAouFCGmM+U+YiYDMYFfyAPIxNYPLnAYMo6ris9QAh/MsP5ZHJFuCOaq1CCFVqX/8omOL/63CN95GEbmXfoi3yOx75uOmMBbrSIFimVA93Eu0ybKo4B2l1+W5LnpfwQizbfODnn9wJqKHB+N5bOXAGzdBZySrgfmmAToVL1Ia3phOZmSD+pP0TT7HF9vsBiwI6iAFgWFvCCBEz/kApOkCJOdrn9ddVSBTu54yYizFgKXyX1qfvgvEn9vd2O/dQ85tAZwdLyvRwYFKNYFvUUx4XcRXt0GF/iscrGnN2lNeMHZm5UYa8lz55jWX2bwKLeRl4GmDa20GCk0USvplft02D68R422UEzqQSzNPyB0GtKGa2ZR9/OfSG1V39yuRrs5xAC00z6N39/A7IEoAIJKtzM2cmKGwNFEjArbJxhZmdT/+8PD3d/8JNVVuuEQCt+dDQMQx2FBnw8zxHvKW822DM8mAS5686v5o9TqXd92/4VPmJzhg4lZCLFswnkLBMzBHm9LxosFcZ+APh2LdZWAJ2ppaUN2t1Jh3SbSmCl+N5sskINxsLepYv+f6uDVUPgPaL99WtdmH4oR08VL4fbth53mUoc4/KdzjMRUfTZJSowxll0JOkO+Dc+qZtwstk8ctkObmNPWQpnJ4QCn1USRhHKI2kWMyNEur9puecpciC8oBl4inEpZf2dMfJwUkIs9RfGLXXPQzdd0iBdr3txSXhLvCHNQmmY6MSNyZJ/VyMkBpNgmra2XPFCydXP705ekqDEtUDEmirRFjydKAabA/xAtWPRSV+CqNMhh9JlCWN9yDMgsIGEeZoUySQrWfDbl58t0TbLZGoVe6/uCSv1Lndd1cJVUxXZXPA9w5ygglMnFt7XjidPxOkxBnsYzV0iMN4mxImbYy5Bp9rAUG5tKLty1c9r1aez3lWU+aQy55P+bPkyfawyx6tP0imJ84v+fnxaZy18/YgpjL4t1UAYIvEEiDkGsHIWCaW2XOJRufXTqqkUJmSgl7eCQt/dDn8dtdEaqlHqVPZAa7ggcsMsB5LmEPXp2ZqTz60HOzikHF0ShLoV5+Kz96bvEZclCTy+TMw6Q9PzqiFHgd/MHXHeDrUomtAMwc4J7uOHg4nwH/q3PAEaJPHZOHjL2D8OuRNvXTAxfQZp39ezxP8NfzKkdXCkAwrh+jnzNRtF/uFX67Oloz6Dve6ygxHsCTozqZM+YRmEXNerv6AkP7JX8zBDET+21359HP9+yIuhuoQ+wr7Zhj7RV2clQ5Kn3Zfkd864Kc66/PiUK5KfbOFqJmDgeKYdnGbOU7BjZgepm7fouQvbZ7KBXN/eXt1yorXHflfXsriP1mU0rrOXCs2HakKCRhiu3b/HDRsStjQ26Wy6FEV7d7kqbYVxs66jBHlp7DGfpiChZgJotfZ40nzGUHurzkDovnN1bKv7no//OvPf/+G3dXd9XTv5dv+Zvt63d2vx3K8j3tf12uZRzj6duh/6cv+3c8vP6/b/c234/XLza3+se6tsvW5jM6+2bXW8/nXN+BvcvcH8DaMkaoQpW6OPFBwyCFo0JjMyidiKJb6of+K4PNcyZrDdNLifm82ZykFi6td9itE3YLj1DZ/Q0C6ya4aKKfbP799TndvoBrhcp2JOz3HozuP4k3G1643eVR3vz19AfoQ5/mEPIXOLkxJH/DujJn1AmV0v7K/d7mpso+q00ortiQzb7bHjViDHnfV6uXRHcCSA9yVf40X0UFGxEKHQ40dhc6yKXW3RlpaAQyRs4nCeCUIrH49IFWahFKASuTvKfPAJpfM1I2uucwYD2ONHgYQbv8YeFwdQS4HPoeYTuTAiIF6QOqGmwlcaN31n2CfU54BZ89vTs9OgYIJ46kNLhQ4mLh68uZjd7n7DAzIggN59bpzqJ3gCk0AjVOzAamu0fRHZSu1R9MwXdfd5lESwTHrm22W9YY3q6CFhuBt1Nser1+9T5WlueKoXyUJUk87x8r2WJ62ziNvWFaDiRDYCm5VUOPqodwcpVWdlpToxyuQdio/0MgNbpv6oXMgjmAzWERu03C7Mlq4lCG4eRktXaK3hXAqdauh+f+4d2IKcKumRjBldAviNKpToVYjNwMg4QyFlI8Jk0CCnAvK6zSM3w998HDMQL0fO7g8ZromatNA2ln1RvMouneLySCJyeKepSokdJjlAm1Eq3+jd+8G7/Qu1LZpFp5ZUsopHynPYoaTFDlB1IeG8K8a2r9qTKBO4pQgL9MYQKOr2Wx8bRmcWxJYFbSJe1ne8lao1QPkN7RG4F/9yPzRhcIYHGvEo401qcUdZ+ZeLLx5DkIb/rWnNLLYKUKJR5p5WIjGnPmQ4UWvI0A3Rh0PhiPxQweDgRhDYGkLQRDiUeF+Ivc7ohJnFPbI6pwS91ShkndwYFVCAWCvqOSJ2vMK8SNSmG8zo0TUMFfWowKrhw8Fqu4N6O8Wi/fKIF05x60uVBbocUxnEznftHrr7XrTCzE/6WiKWKB0DCWq/PGSQPijJEBrWgeIJnOoc6PVuHTOTOfJnLkxZz6MJgjmSWeIc08fNneB2wDxCubVxfMMyOOu4WargYXqx1ibPkZVih9cLdkPLkoh5dZfXKBXTm7MBSaMwRfQN2R9BLw7CdbWcKfc1j5+eGtc/2LN7V+so/gXj+twGPAYmHdlUgVhjaiWGPdU9ON3SkVT0MsLD/3CH9Cbp6rmznYyA6umTokDBMQfPfsgrS6cb/TrxA+/bhrUryMwhOoqgUhriFXppQH82ugySbPEiLi1GX77rSdaYAIDtNe0oUiCICN0D9dVh4EgNDoo+LHZ2NKPxUaKfuhQV4QXBDqg9gc/qE99mE4hOgS6kzfP78Phdh2VmgTZSGfXcehsuj20Ynvop/PQYM5DQ5tDQ+Fwqt5LqeNYmZWJmJWqtWuprrTWioJrmd1fnwuKOqwYXgx/1T/j91XXh6t4K6/e3LwQUdY+jnNVsel2DHHZCVp1PZgZ0/gqg1Fdbbn/+2nb1FjsQ6gsLPGL0Xiijoskvo7OOyY9phuYgh+AB1Cl/ocksDyFm1/OF+/KmiiI4L+zHoNxgZSHNxpSVb769HNJGPVswULjoMq7yYGUTnYbs3edpBFXXSobnChhCQS8CYf/oYFtPHwi+Pl0osZ6hEK/piiJa1CeB2eYCet/BErJFYh9p95g8noP8wy4lQEGdpDihpbAIl7svJnOJbDSDmhmJsdY3ff1cZ/kOOUOGXJjS6DNpcWfbLzKtawjugTJnTYyjtkFnKgk4JOza5atw+sqtx/nNR8GzeTRLPSq3q3gXB1xzr50d228USc4h0nZtPH+pKScoqO4SOvDN79jTA6+aoHHcdtRiEtD9/Zzzq8FBtvNkF+QvrrTL/X6URw0vu0J8D+dDPDWLprBrlRWChvXP05Oljv4Y0K08RswhwU263RbNE0XnRUPJ80v0OBc/BToU1/iQmO3lwz4CBzGQX0pDo1h20b8eac++cdnAAFJaT6db0Gt/XPdVTYpiAgwLAOsQuG2WOtHtx+LV23ryDeis7rTX/07n9X7KaWjkp1oFtuiTVs38p1Gl07qUOHf6aReayaAdmVjLd61PSI3CpH1j/7q3/is3k45HdiuW5vsaUTx/Yj7TTUUd/GUPU1kKVk8ihlNwQwZQcq3rXEoDdCPNSAEQ+8dsEyJWBb3DcYPJYVMjmalq6Z0qiPNAoFR2UF6ugE5V0nzDY14cv6SuiXiKca1MxENfGCzXRb1w+YzIh9afI7+Hm9YHEmSfSrwnyHjQenFCcsrdkWnqkBmHKFIz0mWb7e2v1oUSS6BJPWgaayGAoJ9LMGL3Wia9gYCUQ1AWXGKk0MVy0VAptiP1B5BZoNxOPsWQ25cqFALAOHscU/tY54SycZiGMtrYPkB3BOYRtP/HLqF0nnJhCBXVbme2zOl/T9JGwEOBWbgfd+XNQHryTIMhjs11K33nvGmLH5YYIDjv6mvcUw2XRcXjnXxqZGqJVkGLLpCFaVc/eG0m4IR0nR3Gjis3sfh9Dl6CuqbZGueIxd68231G/C/l5tGdOOmdP5ysFU09UtG19hSHztMueofy9/+g8a6HGpa0lzBi3UieTSCKfIKjqfTOceb6b7AitUjyqJ+ihb04lBcgzsim7vmM+VDGfPfAN61BtQInLAh0In0W1EKPLLgy9CtsZv75Y/z4Fp/AvM33/gmfrPdfZNm275ajxpcK4uXqjTv+ZfmHKOfNUBKYIIxaEnicq70/V+lHCpeXKyRZt3DlRVfv4dWUXs6nbeE8uMPf3BRw5psP9ORlCuDxsbrgQa797iGo+XzHu+RAvEUzL/FABuS1il/YW56WWMhtVeZkpZLZKvUXu21tmUzgNfeCdCq5a8gXB98hOoqbZtXyhuZ7uKh7DDb5KiuMllHeLOeFkQOoCFdoFV1ymsKFIvjmglDO9qj/KcbdVl4nMnt+ACFutVWXcvSMDfvCeTfBXKQwqhlCVH5/Xx/lC81sqJc+EsPNsar1OSJdgQju1emUqO8XEm+nkHvG4LpbLDd+cPFbFRRcZBk0KNQC9Q+yRtVtjNKakmGDippvthQ6VgQkDFSq7NpZnuvvLZU+pw/LcnAhV1Hox6hKSIRuSR1pTy4EMF2XN4i8fAv/dop23jw+9Bxe0E9bvw/rwsTp3qucHj9AAAzSAFC2nzS4bqK2V+X52YoUyUP8KT0UDNat8ob1yVten0lqKzADssBzjlIr4BXMhbwUf5QbCcSkmrCqWTUKxoYDECJIWUCiU44SeDZ14HZ1HscCgGUYSQRWnZ57n0sq6HP3om+AG61pZHFKOigM0zJuPeEmB+Ny7tCFp1fcI6Ol9YY/zRitkMUh8uIfTjZTT6IfMaC1Yk/EiKGZzqsM4xt5Tg7KveS2zx/fcSKqAT1eNNxP9q7VZiWz5xcLop84PP383gTmCrRej0oakCJI9UONyGNyyMa66bh/Q51lfFW+dA0K8lfGx2CIVQ7bVbHI7GJkLKjKY2piO8LGuAaTVSLBcT34mYElI1OMT9O4W0/m8V8SVy7KkuusNMrdWSZTQJ3rvUye/klnA0CTXGqT7oRk5XcIZV7SdoD46KmAkDO0oq1FC6DKBqXc/b/2AwXvDSEjn4AJWq8osW8xQSqqDshRi8PECN+I7afn12BXQlM8Wv+Q1hR/pRT2CxafbTQQEoNRb5VNs4+zx2S63o0x7knuPgzUz0E2YfPVxlE0tAWbS9UFzV+Mj/99F48SecvgmqrdFm0or4oKQj1U6vScX8CjsT3XWbyP7vsqeJTD1U5kBSte6N8i9aV3suqXbjzk8j4RrZbJ9tviiLDTquaDyC1OJzzy1ijqrGZW2UFCjgsiOSLJgtd0TVhdklgaWzg95tMzs0OC4qpb7Et6KAq8vu6hbQFtv7vu+K5+FMJ+izEegQCluSpoBk0E5ZNWhXmQROrjy8ANtpKeD2CYmay3NhLJaFXKE1JMXMngbMFvoqDceEFo3BRLiDUQGIKAVlX/o2hg9oODkiCVQmRTZ7irz+tJ4K2MKq4Yw2VRrpsO1t4wnbO85xXUWdSHpX1HHaGHNoJkdtVlthFL6yqY5R+VUSXipc4D+O1MoldJHmRlLt/6iN0E31M+juTl9eyRy9VacGL60LW4dBeF6eJmqkOV9Gsj6tF0MD1JpjgIEIyR6GfoVrv1h+U+H73rGh1f87riN8GMxCpXC5MWYqLUYDLYJbiqhV0OMisTFFOJSRk5iiWQBsTJExRK5yaDh7z2fKQtsxyfeCsyDqRF+zXII6WT787owTh4fqclRsz7JrJlCwpovU6oCUh6oCXuWkOo5KH63IQIRq46gvLHOS/2v8OOZP2EBzcBpx27WdQWs6TekuSECWiJqw6sK2RQc0QMvAOWDRvk0o1a5wVNUmu3ciVY1USIwq5sSKo22JAsNrO92azjMXZGuS3Nk7sz/lppW1ZXYY+rrS3MtMwz2fseKv8kqmw3RLjZtXQePWbxCu4pskpV7G4XsJ4vzva6eIn2PboWe0n/YK6deaUDl0sCNvuGfdcs7DQbmtRDTI6iM4lZ84Z0cNdDaD8hGSl85NRGCuFbtb6gzqK6XbSGlvhsb/QTIZPsiBbAfY15vLv7LQknib98/orlv9m+5su18nLOH2FyUbCE9Ki/XDm8x4h7o5f2EwlBK7NGLY2MpaJ8Gl8cpwFltCEFtt6GZCDkCigwxRV/iCrKiQH7gMoKwmRJqMd3kS1lJubP8tN87l5cxNmuFAhBwqrSPmJomU/Sl3pYKEGidQFhx7N82KyZ2mGJyJQS/iiGSTWjZoGAlIFEYUotWjKTOpErqm4kfqU2yGE5DqGstBivNDUjqS5FwlIwhMZpIFMvnn5AEJCHGKFaUx3yjLZ/S+MENml8rSLIKaZAkoIzXOBLUIhHivbOJ5qcuOZm81OIb3MBN84qFbCaI2Ryy/AwEl7Nptd2VLz19fKkzN62vqDnN5H2JZwQDStGGJi7FErsOJWeQJwcha8loUUV6KP1KdO2HAluuhf5jIsLdaBnP2C7rbHTlnSEecx6n0ONlHTO56kKNQgZT/geSquw5Z1fLTNPvadd7NO7rraah0PWKtrNCwmgp09BlHlo/hugan06Qx/goZxmrSqaZO4i6/YBVHKh3hyFdyLymEv3mQXfI9gN5WUNKlS0KX2rAJZ7K7/319NDmUVzVbfNvniLuVpdeCm/w0u6Afl3xGzUPbWKncB98+w2cdefYphjY5c6eo0CslH+ZY7F6h5XEHoUVpt9ubEncjLtC/IXHSB5KZLJEgSxkwuvlIiUKfIsnrUbWJcpPILwoGmnMBTUXAFSCpgkqqCNHO6U2PsGzDnamcryfepJ1DzXapv775We9k9/z9nXpYFcxJ13wkGS6m9Uh3o7j0GT2AxHAh205rya/X3aqxVyVvLTZWBAez8Xkc1m7L1nm5BULFQqS5PMcgsb2fGWVui9wRaFjoJxgp1LxCwyjmc+cQJrUnarR52mYZ9sKGoqrmt19yWQy9CbF5jJoVegT2B2V54Q8XW6TLWk+ZQunhud4kIN4K21zRWqOGt5KP8grVoo6123719p8/+pAe3cKtK19LG/Xt2vgYOJ7vfkn3ae/rAp2vfkz70nDx3CIkr2EWoOCMV+BFvjvsknLrDqcjh7Y9Wa9ZWFblOFckaySluExovHbSknP3UeiSisN5X8h0/6f8wHNfOcYlialcRiTaebzLR2LBJTl4Rr34zISN5GanxJVif+SRx9lKTw8rlb4VaVwg709EEUwAXIS8jhJyBVhzC6HbPDo0Cz63sbaYO3J6412KORjiia7NGX3gTgEfwFjsTluqoxoqOytOUsQTK2tQ+Wyxut3iyxXyUmnfLBertuM+Lgq1eJzUXvawE53K1CeRBLG7dwc/ZImC+Ltb7Xsa22epRJdxUi/epmlIMKAdQO29dYpVdV0r9R0T9rcyv2OV1HrP2/+YD1kgb6Y++9F0kqTlZ5AH25TdhfQb+nog1RcaHdlct9ZI/T/HzusYUBOTDpshOWe+bbQKaQfHvWg4Jv8yWxMrblzgwl+FlfEW0zzneppc3lqbojoi/wTXdV+GxtwkRQb5dQGWtof86MoJikRaBJ/c4M3zTuu+ncB/bo+Y/tvaCvCUCb9OBgO/JVa+kV18pei5TDOTxKbXLIo336oAtufmFUsWo/YIBX/gE/+6Iij/DP5sDQXP0JBdyzeFONJZ/0TebUVzExh6EuViAq7B58mORXHqort/lb9+smlsd1tEN+EKDcKxjYKClXLkX4Qf+SxTOmlJcpBx+pcJzSW0D0ltUt8oauqRcAphz+/us9SxNYbGtMAwtNkwP6mFVCp0xXJLe2p/XVMau3ksXek5uZov6uFow5/ja/GaS2yZ2A6lLchcUFLY45JbXS3IDHLfYbmm3yQyAYXh7XZ4uUj9uZW53MK47ZYPrA3MRDpcO8k66LJDqOV1EyCeY6oLV6gkpV9K3vKyWTJPQ73pbWJnfnIsVuD3p1YU59OAj2gF7WtdCx/CqxCRcuBlclkrDpOxzMP4Gwxweey1ygNrOSy35ciTMcVsumg0kNQGmrx/baEmRgmG32BYAKVdBhhjhf7I2kOW21Uu8RLavVlV3EWkO2bDOBZf5begnlT0Z/3cnIamoDQhRfQgqn4Tsy7aInkjcEi5t43/RfFej9pPFYEOKBxqWDEXUUjmmM8FuvIBv5+/g5JlgXDPX+iCeqU0Cp314nv0LTi3KNi8dbJT30o4wwHPTKanehsdS5vKSNfuGsfrqi9rgkr2BQN4Rlsz9CLvcru8kaW/8GPFV6Pyo1fsnsETaQdffsM/cg+B4K2chq7rV6zDNIT13bs2f6J4SIoIpAuSxDioSpGpTCZ7mFnwRYerGJfO/RYN9rMlBZp2HxeYIEC3zBq98gq+Clh/SxgNwhxibg9ChQ6VaHVepvQdpGe/CLbQXuhW5KuNiIJBXm2rMq20Wx58Xp8+tXLRaioxIFc6ER/lQOy9RTIcE5+DXK1UPfb5XNiaOT0TjrBV0gL9mISWPA3YbdFWjglacGhNP108T5ndyA85Tu7KtQVYqYQxYwWBceiszLKflPMaZJQD5ItIrMLjQFRNxBAiNooRBjm15rzxpJORrqZ+ttN1PRQlmUg7xS5WC93rxZxVcw+Y3rAdcd4TG3eop4hW/bdRPPoRyYN7jIsR2L+mzq+G/ilhfqddmUchicQ2KLmWBGKS7bkzRiRQadBCGXYKCblVEiWir8mdcrIA646hwymIpXTYMdR5rbUpNI3qvddVMMxQgJBJ4X/0eSYrMA6dSlOR0iYYBJNYRAsEQvPlMpt41IuCIKIY0omLF9mVHgwd1C7MkKsEcg1VpBmZMj+ks6eqlHvMW2oT7Wfc4nAu9hzMAEmoLilwHDFXCOxs5Mxyrc9Orbhwhlv0Ci7gxtadBDlutwD7VGhta3u6fheojSSj7w3tKJXDCdh2SVvNrn45tfz/xil/fvv52FymbPEQau7iKoZFRtmLr7pHrv2zk73UZmwbcj0ypCD7JLBI3KGZKsUyGxhXnW4IdUPJLZYXTpKzhmBHbR21QWdmGEBKYvdPvY5N3HZrCFv1e2QIXDt+xRn/owBz1DpVVPaLcF0vA8u72ITcqpnHls5vEmM6suhagpZwv7BnTBtLvIoFORqtYBBwaepUMV7/RhFYeyocIkPU/hDQezgUtpf5FqOshXCA213GoCLC0X1/6lHbYGMWasWCnrKSl9O6BJiGUWmDTD32GRmvCjelwFcTdgVuKuLlF5qq0hKSDJHV+vHEUEG4XaaY8nnQiBKItZ42LXMPDQLEODtNoP/7t4z6hyHolGGu0Q1mOUKkS3C0NRag0rIJXi2SjdiMtfsm6j8aMoDvDGSovLI6B2skEORRlJ4XqAdGjocXoAAmbqj2zKnKoJttaGdlo6eqVxAoDJT44oSMhHnOB5GZnv2fuJnK4zSRqxZ6mgsTU5yiZaZKvypJocEkc6kFIe6B3DhA4knx0wcJ2ddCI4DPmLtVRCZ94T2XaJs14EneJKRMfFE2IlgijkP62miXUmbklRuPZakLe640dYDgWjsLyH6erpNCaROnBbtLF0Ovk78MZC/SITS+kuOJtDuLArnOSHOkG4LBzfa8LidDAeFVg6mueRUCOk8oYmaMTF6cwcwiZtIUMsafbn1ZX5RCOhjhPFCFOayF1u/pOff46oeznLA49sDoh/6OwgxOl0bo2AlzihhODgGLq9Kqcop3rNplCbZT7Fnd9ADOkPZUfuge9dBovleLcqBJ9abnG/C6j1ACYqRbu4egKY93gEr0WBG5FjkZwaEObVYliSDQsyiEC5RXHp/2AZeFJuREigIedpTBP8z93ONvmgC7j1OtEPabCNRyIpC1oqwwQWv0oNFJG9/14wzd1TKxYxFBZch6Fl52tBiB2M8HFdct4faBHgPlZMIsOmoUrGLcdUD8tj9t3n5KQ2dhSXue3oBeCN18IENU8QWq85xqo1kG73JMYE+zKcAsJ/rJx7bpZA7gwPdy+CxhXtFFP4LkH+kmp/XOVAkrY7v5+K81N9njXjKdXS1ewvnWC5zxKkrqanFtjfSqmTchNJroRXHEOc3SduUtd6X1RkwUozpOJPztqWPOihljW3b4xKKXEQzqH2c1cGOmFjkzVj9uPlDeMeZ/fj84yERTK22KmBerxHgdoQF9LbXlzn5ES/phmxn7gKRXXNSYm9N164ME16yHOx6UxS0fdobHXFTFlhO0E5Q0YPLKB0mFL8cS0Q+x7WGdadruMd0pO4UuAmLJJmIsM+YDIuzDHlwUITctR4CbDpRWq/YkSfC7G6Ve3IwMZAqp2t48mL//HEMMP3Ne0v1dYSxUWhRLPmm2xeCZ+3MnX9FXIl3bWN6yyi/HWHB8SQOOIB0pUPTlSIB7oex2LYUD4t8X0t528aQT5nZwOjTOLwNeHa+NXciPncsdvQ7a3s1v/LEVU67745O/6KNs/qJeX/9+wjvDQEzsHqGaPYDmOjqUA3cHN4rzARIGijh+xJCgTMj5B1NgAsZRAc7w2ieH+zR917EfsPT7S1A6Gst3k5eO10ebatOxTvqaEbpvC82dorJ3sVOZpvML4nWa3xIjYEjmRWeiYiQ4NbyZHqeEgSA3kZDmu3i1ZVMQZQbpeUd71x5Cwsllju+x4DOEc3CHfWR8ySavjSasbvsypoB2UVqKDVty/jdfFjdhNDXYRDxBGa1fh2/2CJs2MXthrkPUOgW1/tyAZUP5erkFmApF0f24Yos3SHKHT1kydbmdyToC+2gQ6UFt310qVrVBbVGGHNfNAlZ337y8sXSqYjArSm4N4zbAu+rs+Y63TeC5yC2SsaGCwRODd4e6SnRsjnUoRcMpidsNs5nv4E+cEKMSpw2owfzvJIQAi4AMBDSrYdSNWDIqjNn8jr7B4MoORnfPZ7JOZj+mbw2/kE7uFB0Yk7lcu0fj/5ipzfFALZP3hCf4ZtMyyofe3BXHwxyqGddOg1nYgYkXHITeHc9gqOk7R3e40zmmRudTYKSsI7/tns7kAjzITWTMCx2PhF5rbLvgd5lrpV9QfKFiUA2Q9AK2/e3nh1aNPm2TaweRGByOilyt/Otn57+w98oMlxvs63D2+Pkl7pVMkWm4Syq+RZZnYpmIjRU5JvMwJ+e4l1lcCabRRvfHSNhjbuhkWbQboptviqp7ymDoeowyoce1Xmt097KFJwiSsXdJkU8PUD+z7MlMn5EeCY0IQjs3nvtauJg3imri2T92s2Su3qpw2mEhiG9jJbSRz8j10ANqPsPj0FsUqaUMtmhyOd4ZYsJGacoHpzPa3TfM/rezGKhUndEJMau6Hz5IThhaqTm1jX2PM4HtJcA5Zc87WJbsckAGrAEcCIBhxPIChPLSnZoMT5sFYfKC3V1qoBvMcJI781PT5x0hjZYmQMwDpItB4EOHEDmn0m75OzdBhNrp7WuOAQSSqGbFLKZhxY2cW9birzyRMOqUL738d4K5EFty75UL2sX+ZFtAN48zF4BktRmhIjTbD7OX+VqJhDFKUEbtbUzmnQ1PV+bDUPDQOtreC7KbqIEJlT8jZoeLkTB9pZ6bujxzqL+KIpKMKPe0kxqqf+twexA8rOsLmWO4qziEo3YSwnYvmwr4mZ3RO4nIcopEdDOtKSVNtwtlkuvcDxFQ05zSkU/NhTVkUR9JtsfG+OvKtmmpaYsU8Z1gcTOWfHqBC4qe/uHf8o2J8UdNdT3KIxB8OCfnmKpslCDY+u5lGIR9iK/jt2ak8Oflt6vMz40LdFMSK5kn7mDke7Z36y/JC9Va5Y8gsvSF55k3lvmTth4xO6TppRTrG4x3Cnh+POmLSjzNe1zmQF0N1+ZiOYKfdEmKxWSC3E3aMt7AUX3SMXNmF0/SBWvwbapStJIGIJMjQeSpRtoNJkZocWdwpsje1qGE1jkuljLHXd9CW7AIk6ZoiZ/PyeDwLqw9OjeGNZTqHsYynauqJekM8njWKPHskYiTfZw7EhVNl3agJDwp0ViubpbA9m8Y6V9XlqnTWLvcqpB6R4AiEMySjWmYE/UYmfWouWttF59/BoiOcxxgbWTU4mR1CohX54TbWnLXoDGdPNtiAvTNkK1WPJoHb12tSRP91kIbFAd0l/yemKfGwTJHSJTiroKw7jMq4nnggOmxskDKIzvH9vcdHlEKhrzKhQUVvecVsscqrfCp4zyytVUmE6nsTH/sYf0ZSZmLjCpGuBu7wB438OGMq5d/aT4syPdqdgbPaYbt7NyWymaOsVEW6xSH3UUaRo/VqTq4IZ0htmfeatxMr2JSRhbTFNEUZ17DEQS0UL+jWvrii1f67NdpzFHUMUBdU4v+dWIC6Xh1BXXMtwqkVwMKchBGNDKJZWhQB+CSVqkiMa3nxZ3WNKFddMiSk5enWGBI3hnEfxIYU7j3JkGCEaemaFPh9rV895sJhdxfTL1XHrWSkCMVRkd6VU5A/dCo2qSIm6g/UeH1aB2OyCjMNmPlMPn0vZuxI+pWJ/bJMKYZadfX8QdEuD8SSj0g4AnW1QYjPHfWaiAKxkbhJVmdxMiG0QU1DWz4DeZptmkHDQhRW5idMqzifRbpkqGdOvx08wiE6Kz6dSvkK7x5p4zrH9GeE8eLFuRbwq28v2kiT6jskPS7okJj4VkziTQSommnr0xScVQ5bN2R3/ZxU99rnZOfPbZByxGCTfmSXk7aRQ5I8d0ncXbzEsW0Lkk4tLVWuUNsgatviVbiTbhOL6LKq0RN7no5kNlUjbLwubt584R4nF/ZRFC+DnORNTSD75rukAMq35s3RPSlvVeK703y3z5d8erCRa6543oNbPg6qtyN8O7LOPvFAUR0AhCyTP4q+A64qO4R1ludQXWImoSjdNMwgsiZHgHQG"
}
However, when decoding the text of the content using
base64.b64decode(my_coded_text).decode("UTF-8")
I invariably receive the following error:
UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 0-1: invalid continuation byte
What's the proper way to decode such strings?
EDITED TO ADD
I'm assuming the issue comes from this line from rfc1341: "A CRLF sequence in base64 data should be converted to a quoted-printable line break, but ONLY when converting text data" because decoding works fine with images.
However, I've yet to understand what's a CRLF sequence in base64 data and how it can be converted.
I'm not sure what kind of data is supposed to be in the "text" property, but it sure isn't text. You'll need to analyze the application that made this request to figure this out.
The decoded text starts out with the following data, which doesn't contain any readable text, and doesn't match with any known file format:
00000000: e2a1 7614 9214 be79 5194 3352 a680 7a29 ..v....yQ.3R..z)
00000010: e064 8850 0375 fe03 f106 0ab6 1351 55cf .d.P.u.......QU.
00000020: ad32 a1ef d6fb a2a4 96e4 30ab 203e eb8c .2........0. >..
00000030: e3f1 46dd 155c 2e56 21c4 7987 df94 833f ..F..\.V!.y....?
An analysis of the data shows that it's virtually incompressible. This means that it may be encrypted, or compressed with an unknown algorithm.
I couldn't find a way to decode the base64 but I could find a workaround: sending the same request using selenium-requests and accessing the response which is, this time, not base64-encoded.
Code now looks like this:
for http_req in proxy.har["log"]["entries"]:
req_url = http_req["request"]["url"]
if url in req_url:
# Captures the POST data
payload = {}
for post_param in http_req["request"]["postData"]["params"]:
param_name = post_param["name"]
param_value = post_param["value"]
payload[param_name] = param_value
# Resends the post request
r = driver.request('POST', url, data=payload)
I'm calling the google API myself instead of using their python Library because I'm behind an inconvenient corporate proxy which kills their library, so I have to do it all myself.
This works fine:
requests.get('https://www.googleapis.com/webmasters/v3/sites', params =
{'access_token':'my_access_token_here'})
This on the other hand, doesn't:
site = https://www.my_website_from_the_above_function.com
site = urllib.parse.quote_plus(site)
def get_website_info():
url = 'https://www.googleapis.com/webmasters/v3/sites/{}/searchAnalytics/query'.format(site)
params = {
"endDate": "2017-12-10",
"startDate": "2017-12-01",
"access_token": my_access_token
}
r = requests.post(url, params = params)
return r
x = get_website_info().json()
All I get is this error code:
{'error': {'code': 500,
'errors': [{'domain': 'global',
'message': 'Backend Error',
'reason': 'backendError'}],
'message': 'Backend Error'}}
Even with the reccomended 'Exponential backoff'
Using googles API explorer seems to work fine:
Aditionally: This also seems give similar errors:
r = requests.post(url, params = auth_params, data = json.dumps(params))
and finally:
r = requests.post(url, params = auth_params, data = params)
just gives
{'error': {'code': 400,
'errors': [{'domain': 'global',
'message': 'This API does not support parsing form-encoded input.',
'reason': 'parseError'}],
'message': 'This API does not support parsing form-encoded input.'}}
So, you can think of the contents of a request as just text, right? Not only text, but text that accepts a relatively limited number of characters.
With that in mind, it all boils down on how to serialize "complex" data structures into text. I recently answered another question about files that is kinddddaaa similar idea.
If you have a bunch of key=value parameters, you could use a simple "trick":
Control names and values are escaped. Space characters are replaced by
+, and then reserved characters are escaped as described in
[RFC1738], section 2.2: Non-alphanumeric characters are replaced by
%HH, a percent sign and two hexadecimal digits representing the
ASCII code of the character. Line breaks are represented as "CR LF"
pairs (i.e., %0D%0A).
The control names/values are listed in the
order they appear in the document. The name is separated from the
value by = and name/value pairs are separated from each other by
&.
So this data:
{a="foo", b="bar baz"}
Could be serialized into text following the specification above like: a=foo&b=bar+baz
That serialization format is identified as application/x-www-form-urlencoded in the Content-type request's header. That request's header is telling the server that receives it something like "Hey! The data that is coming in my body is serialized following that convention that separates keys from values using the = symbol and splits key/value pairs using &, changes whitespaces by +... and so on"
(!) Very important: That is the format used by the requests module on a POST unless told otherwise.
Another format, which allows more flexibility (such as maintaining basic types or nesting structures) is JSON. That is the format that the Google server "wants", and in order to tell servers that the "text" contained in the request's body follows the Json standard (or convention), the Content-Type header must be set to 'application/json'.
What appears that your Google server was doing upon receiving a request was checking the Content-type header and if it wasn't Json, it gave you a 400 error to indicate "Oh, I don't understand this format... I want Json!"
That's why you have to specify the Json header.
There's an example comparing both formats here.
You can also see it more clearly since the latest versions of requests module can do the JSON parsing for you. Since the JSON format has become so common, you can pass data provided in a Python structure (a dict, for instance) through the json= argument, and the module will do the json.dumps and set the header for you. This also allows you to "introspect" a little how the body will look like (to see the differences maybe more clearly).
Check this out:
from requests import Request
data = {
'a': 'foo-1 baz',
'b': 5,
'c': [1, 2, 3],
'd': '6'
}
req = Request('POST', 'http://foo.bar', data=data)
prepped = req.prepare()
print("Normal headers: %s" % prepped.headers)
print("Normal body: %s" % prepped.body)
req = Request('POST', 'http://foo.bar', json=data)
prepped = req.prepare()
print("Json headers: %s" % prepped.headers)
print("Json body: %s" % prepped.body)
Outputs:
Normal headers: {'Content-Length': '31', 'Content-Type': 'application/x-www-form-urlencoded'}
Normal body: d=6&a=foo-1+baz&c=1&c=2&c=3&b=5
Json headers: {'Content-Length': '52', 'Content-Type': 'application/json'}
Json body: b'{"d": "6", "a": "foo-1 baz", "c": [1, 2, 3], "b": 5}'
See the difference? JSON is capable of making a difference between the strings foo-1 or 6 (using ") as opposed to 5 being an integer, while the x-www-form can't (see how the form encoding doesn't differentiate between the integer 5 or the string 6). Same with the list. By using the character [, the server will be able to tell that c is a list (and of integers)
I got it! The solution:
was to pass in header information with:
headers = {'Content-type': 'application/json',
'Authorization' : 'Bearer %s' % access_token}
and make sure the json data was dumped to a string:
r = requests.post(url,data = json.dumps(params), headers = headers)
If someone could explain the reason behind my answer, that would be great.
I'm having trouble converting curl code to python in order to access a token to an API.
The given code is:
curl -k -d "grant_type=client_credentials&scope=PRODUCTION" -H "Authorization :Basic <long base64 value>, Content-Type: application/x-www-form-urlencoded" https://api-km.it.umich.edu/token
I know that -H indicates a header, however Im not sure what to do with -d. So far I have:
authorizationcode = 'username:password'
authorizationcode = base64.standard_b64encode(authorizationcode)
header = {'Authorization ': 'Basic ' + authorizationcode, 'Content-Type': 'application/x-www-form-' + authorizationcode}
r = requests.post('https://api-km.it.umich.edu/token',
data = 'grant_type=client_credentials&scope=PRODUCTION',
headers = header)
Also, these are the instructions:
Obtain your consumer key and consumer secret from the API Directory. These are generated on the Subscriptions page after an application is successfully subscribed an API.
Combine the consumer key and consumer secret keys in the format: consumer-key:consumer-secret. Encode the combined string using base64. Most programming languages have a method to base64 encode a string. For an example of encoding to base64. Visit the base64encode site for more information.
Execute a POST call to the token API to get an access token.
Our data is correct however we are getting a 415 error from the server.
Assistance would be greatly appreciated.
A 415 Error is described in http://www.checkupdown.com/status/E415.html as "Unsupported media type"
As #krock mentioned, the content-type is not specified as application/x-www-form-urlencoded, rather it is being set to x-www-form- + your auth code.
You are setting an incorrect Content-Type header:
'Content-Type': 'application/x-www-form-' + authorizationcode
That should be 'application/x-www-form-urlencode'. You do not, however, have to set it at all as requests does this for you automatically if you pass in a dictionary to the data argument.
requests will also handle the Authorization header for you; pass in the username and password to the auth argument as a tuple:
auth = ('username', 'password')
params = {'grant_type': 'client_credentials', 'scope': 'PRODUCTION'}
r = requests.post('https://api-km.it.umich.edu/token', data=params, auth=auth)
where user and password are the parts before and after the colon. requests will produce the correct Basic base64-encoded header for you from those two strings.
I cannot for the life of me figure out how to perform an HTTP PUT request with verbatim binary data in Python 2.7 with the standard Python libraries.
I thought I could do it with urllib2, but that fails because urllib2.Request expects its data in application/x-www-form-urlencoded format. I do not want to encode the binary data, I just want to transmit it verbatim, after the headers that include
Content-Type: application/octet-stream
Content-Length: (whatever my binary data length is)
This seems so simple, but I keep going round in circles and can't seem to figure out how.
How can I do this? (aside from open up a raw binary socket and write to it)
I found out my problem. It seems there is some obscure behavior in urllib2.Request / urllib2.urlopen() (at least in Python 2.7)
The urllib2.Request(url, data, headers) constructor seems to expect the same type of string in its url and data parameters.
I was giving the data parameter raw data from a file read() call (which in Python 2.7 returns it in the form of a 'plain' string), but my url was accidentally Unicode because I concatenated a portion of the URL from the result of another function which returned Unicode strings.
Rather than trying to "downcast" url from Unicode -> plain strings, it tried to "upcast" the data parameter to Unicode, and it gave me a codec error. (oddly enough, this happens on the urllib2.urlopen() function call, not the urllib2.Request constructor)
When I changed my function call to
# headers contains `{'Content-Type': 'application/octet-stream'}`
r = urllib2.Request(url.encode('utf-8'), data, headers)
it worked fine.
You're misreading the documentation: urllib2.Request expects the data already encoded, and for POST that usually means the application/x-www-form-urlencoded format. You are free to associate any other, binary data, like this:
import urllib2
data = b'binary-data'
r = urllib2.Request('http://example.net/put', data,
{'Content-Type': 'application/octet-stream'})
r.get_method = lambda: 'PUT'
urllib2.urlopen(r)
This will produce the request you want:
PUT /put HTTP/1.1
Accept-Encoding: identity
Content-Length: 11
Host: example.net
Content-Type: application/octet-stream
Connection: close
User-Agent: Python-urllib/2.7
binary-data
Have you considered/tried using httplib?
HTTPConnection.request(method, url[, body[, headers]])
This will send a request to the server using the HTTP request method
method and the selector url. If the body argument is present, it
should be a string of data to send after the headers are finished.
Alternatively, it may be an open file object, in which case the
contents of the file is sent; this file object should support fileno()
and read() methods. The header Content-Length is automatically set to
the correct value. The headers argument should be a mapping of extra
HTTP headers to send with the request.
This snipped worked for me to PUT an image:
on HTTPS site. If you don't need HTTPS, use
httplib.HTTPConnection(URL) instead.
import httplib
import ssl
API_URL="api-mysight.com"
TOKEN="myDummyToken"
IMAGE_FILE="myimage.jpg"
imageID="myImageID"
URL_PATH_2_USE="/My/image/" + imageID +"?objectId=AAA"
headers = {"Content-Type":"application/octet-stream", "X-Access-Token": TOKEN}
imgData = open(IMAGE_FILE, "rb")
REQUEST="PUT"
conn = httplib.HTTPSConnection(API_URL, context=ssl.SSLContext(ssl.PROTOCOL_TLSv1))
conn.request(REQUEST, URL_PATH_2_USE, imgData, headers)
response = conn.getresponse()
result = response.read()