I know this is a duplicate of multiple questions, but for some reason I've not been able to figure out how to apply those solutions to my problem. The function works fine in Lambda tests, but fails when testing it via API Gateway.
import boto3
import json
import decimal
from boto3.dynamodb.conditions import Key, Attr
def lambda_handler(event, context):
dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('some_table')
result = table.scan()
response = {
"status code": 200,
"body": result["Items"]
}
return response
Here's the error log
Execution log for request
Tue Dec 31 22:47:10 UTC 2019 : Starting execution for request:
Tue Dec 31 22:47:10 UTC 2019 : HTTP Method: GET, Resource Path: /notes
Tue Dec 31 22:47:10 UTC 2019 : Method request path: {}
Tue Dec 31 22:47:10 UTC 2019 : Method request query string: {}
Tue Dec 31 22:47:10 UTC 2019 : Method request headers: {}
Tue Dec 31 22:47:10 UTC 2019 : Method request body before transformations:
Tue Dec 31 22:47:10 UTC 2019 : Endpoint request URI:
Tue Dec 31 22:47:10 UTC 2019 : Endpoint request headers: {x-amzn-lambda-integration-tag=1c231f4e-97e9-405a-aadf-ce37b34ccccd, Authorization=*****************************************************************************************************************************************************************************************************************************************************************************************************************************25519d, X-Amz-Date=20191231T224710Z, x-amzn-apigateway-api-id=4tjnqn8083, X-Amz-Source-Arn=arn:aws:execute-api:561581028295:4tjnqn8083/test-invoke-stage/GET/notes, Accept=application/json, User-Agent=AmazonAPIGateway_4tjnqn8083, X-Amz-Security-Token=IQoJb3JpZ2luX2VjEI7//////////wEaDmFwLXNvdXRoZWFzdC0yIkcwRQIgWl5Cw0aOXcxA4tBC8730wNLqnDVeo98T4+nu23F0CH8CIQCfqC5gJ6U4/UaXtHMOc1riROnwTj7AbYIKs/PCGam00irHAwj3//////////8BEAIaDDc5ODM3NjExMzg1MyIM3wb8dOuNeahpJ6o1KpsDbq4XLSkUYzoiplWuxXWlXvC3sTNceGepB4Gzgwzq8Aw4KO4tcI0GXDBjaNDCTDUpI3HMfxboA6r4v2H84VJ6YiSyIfpqRrv/2DiBortTr4iTARMBIVQb+Nc1v [TRUNCATED]
Tue Dec 31 22:47:10 UTC 2019 : Endpoint request body after transformations: {"resource":"/notes","path":"/notes","httpMethod":"GET","headers":null,"multiValueHeaders":null,"queryStringParameters":null,"multiValueQueryStringParameters":null,"pathParameters":null,"stageVariables":null,"requestContext":{"resourceId":"wb2eow","resourcePath":"/notes","httpMethod":"GET","extendedRequestId":"Fl1tQG5sywMF1tg=","requestTime":"31/Dec/2019:22:47:10 +0000","path":"/notes","accountId":"34523452346","protocol":"HTTP/1.1","stage":"test-invoke-stage","domainPrefix":"testPrefix","requestTimeEpoch":1577832430388,"requestId":"1c231f4e-97e9-405a-aadf-ce37b34ccccd","identity":{"cognitoIdentityPoolId":null,"cognitoIdentityId":null,"apiKey":"test-invoke-api-key","principalOrgId":null,"cognitoAuthenticationType":null,"userArn":"arn:aws:iam::561581028295:user/sanjay","apiKeyId":"test-invoke-api-key-id","userAgent":"aws-internal/3 aws-sdk-java/1.11.690 Linux/4.9.184-0.1.ac.235.83.329.metal1.x86_64 OpenJDK_64-Bit_Server_VM/25.232-b09 java/1.8.0_232 vendor/Oracle_Co [TRUNCATED]
Tue Dec 31 22:47:10 UTC 2019 : Sending request to https://lambda.amazonaws.com/2015-03-31/functions/arn:aws:lambda:2:562534523452345:function:listMyNote/invocations
Tue Dec 31 22:47:11 UTC 2019 : Received response. Status: 200, Integration latency: 1393 ms
Tue Dec 31 22:47:11 UTC 2019 : Endpoint response headers: {Date=Tue, 31 Dec 2019 22:47:11 GMT, Content-Type=application/json, Content-Length=118, Connection=keep-alive, x-amzn-RequestId=ac66aba1-d4c3-45ec-add3-f436cf177da9, x-amzn-Remapped-Content-Length=0, X-Amz-Executed-Version=$LATEST, X-Amzn-Trace-Id=root=1-5e0bcfee-7c8dc2fff64742d811635106;sampled=0}
Tue Dec 31 22:47:11 UTC 2019 : Endpoint response body before transformations: {"status code": 200, "body": "[{'id': '00f5fe2a-2c17-11ea-b5d9-dda84499b43e', 'text': 'Hello from the other side!'}]"}
Tue Dec 31 22:47:11 UTC 2019 : Execution failed due to configuration error: Malformed Lambda proxy response
Tue Dec 31 22:47:11 UTC 2019 : Method completed with status: 502
I've read multiple posts talking about the response body needing to be a string and to follow some kind of predefined format, but I'm not sure what I'm missing. Any help would be greatly appreciated.
It's statusCode. And you should return a string for body.
Here you go:
import json
response = {
"statusCode": 200,
"body": json.dumps(result["Items"])
}
Related
import paramiko
import ssh
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect(ssh_host, username=ssh_user, password=ssh_passwd)
stdin, stdout, stderr = ssh.exec_command('time', timeout=300)
start = time.time()
stdout.readlines()
print(f'elapsed: {time.time() - start}'
OUTPUT:
elapsed: 1.0629479885101318
vs running locally:
$ time time
real 0m0.000s
user 0m0.000s
sys 0m0.000s
vs running python locally
test.py
import time
start=time.time()
print(f'elapsed: {time.time() - start}')
OUTPUT:
$ python3 test.py
elapsed: 7.152557373046875e-07
I'm only checking the time for when the command is running, and time returns almost immediately, but it still takes >1s to run. That seems extremely long. This is also exclusive of the ssh steup/connection.
EDIT:
Metrics on running ssh command directly and RTT:
$ ssh remote time 2>&1 | while read line; do echo "`date +'%a %b %d %H:%M:%S.%N %Y'` $line"; done
Thu Sep 30 12:10:17.045624550 2021
Thu Sep 30 12:10:17.046640382 2021 real 0m0.000s
Thu Sep 30 12:10:17.047518094 2021 user 0m0.000s
Thu Sep 30 12:10:17.048373649 2021 sys 0m0.000s
$ time ssh remote time 2>&1 | while read line; do echo "`date +'%a %b %d %H:%M:%S.%N %Y'` $line"; done
Thu Sep 30 12:10:36.694376506 2021
Thu Sep 30 12:10:36.695425964 2021 real 0m0.000s
Thu Sep 30 12:10:36.696347822 2021 user 0m0.000s
Thu Sep 30 12:10:36.697234893 2021 sys 0m0.000s
real 0m0.423s
user 0m0.013s
sys 0m0.005s
RTT:
rtt min/avg/max/mdev = 0.401/0.405/0.422/0.026 ms
ssh -v
$ ssh -v remote time 2>&1 | while read line; do echo "`date +'%a %b %d %H:%M:%S.%N %Y'` $line"; done
Thu Sep 30 12:21:21.463403647 2021 OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017
Thu Sep 30 12:21:21.464389238 2021 debug1: Reading configuration data /home/user_name/.ssh/config
Thu Sep 30 12:21:21.465492970 2021 debug1: /home/user_name/.ssh/config line 11: Applying options for remote
Thu Sep 30 12:21:21.466583875 2021 debug1: Reading configuration data /etc/ssh/ssh_config
Thu Sep 30 12:21:21.467567989 2021 debug1: /etc/ssh/ssh_config line 58: Applying options for *
Thu Sep 30 12:21:21.468557630 2021 debug1: Connecting to remote [remote_ip] port 22.
Thu Sep 30 12:21:21.469518951 2021 debug1: Connection established.
Thu Sep 30 12:21:21.470447401 2021 debug1: identity file /home/user_name/.ssh/id_rsa_dev type 1
Thu Sep 30 12:21:21.471434362 2021 debug1: key_load_public: No such file or directory
Thu Sep 30 12:21:21.472354732 2021 debug1: identity file /home/user_name/.ssh/id_rsa_dev-cert type -1
Thu Sep 30 12:21:21.473283013 2021 debug1: Enabling compatibility mode for protocol 2.0
Thu Sep 30 12:21:21.474141590 2021 debug1: Local version string SSH-2.0-OpenSSH_7.4
Thu Sep 30 12:21:21.475110801 2021 debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4
Thu Sep 30 12:21:21.476174471 2021 debug1: match: OpenSSH_7.4 pat OpenSSH* compat 0x04000000
Thu Sep 30 12:21:21.477111235 2021 debug1: Authenticating to remote:22 as 'remote_user'
Thu Sep 30 12:21:21.478265855 2021 debug1: SSH2_MSG_KEXINIT sent
Thu Sep 30 12:21:21.479124130 2021 debug1: SSH2_MSG_KEXINIT received
Thu Sep 30 12:21:21.480019158 2021 debug1: kex: algorithm: curve25519-sha256
Thu Sep 30 12:21:21.480923242 2021 debug1: kex: host key algorithm: ecdsa-sha2-nistp256
Thu Sep 30 12:21:21.481849840 2021 debug1: kex: server->client cipher: chacha20-poly1305#openssh.com MAC: <implicit> compression: none
Thu Sep 30 12:21:21.482724093 2021 debug1: kex: client->server cipher: chacha20-poly1305#openssh.com MAC: <implicit> compression: none
Thu Sep 30 12:21:21.483566928 2021 debug1: kex: curve25519-sha256 need=64 dh_need=64
Thu Sep 30 12:21:21.484479465 2021 debug1: kex: curve25519-sha256 need=64 dh_need=64
Thu Sep 30 12:21:21.485411507 2021 debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
Thu Sep 30 12:21:21.486294850 2021 debug1: Server host key: ecdsa-sha2-nistp256 SHA256:bycBupsIKGtJN4ACVfYRMBl4mahOUFs1Z1tz/V0f71w
Thu Sep 30 12:21:21.487238710 2021 debug1: Host 'remote' is known and matches the ECDSA host key.
Thu Sep 30 12:21:21.488108414 2021 debug1: Found key in /home/user_name/.ssh/known_hosts:6
Thu Sep 30 12:21:21.489027882 2021 debug1: rekey after 134217728 blocks
Thu Sep 30 12:21:21.489877711 2021 debug1: SSH2_MSG_NEWKEYS sent
Thu Sep 30 12:21:21.490768567 2021 debug1: expecting SSH2_MSG_NEWKEYS
Thu Sep 30 12:21:21.491718078 2021 debug1: SSH2_MSG_NEWKEYS received
Thu Sep 30 12:21:21.492650467 2021 debug1: rekey after 134217728 blocks
Thu Sep 30 12:21:21.493520701 2021 debug1: SSH2_MSG_EXT_INFO received
Thu Sep 30 12:21:21.494433476 2021 debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
Thu Sep 30 12:21:21.524232376 2021 debug1: SSH2_MSG_SERVICE_ACCEPT received
Thu Sep 30 12:21:21.525490270 2021 debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
Thu Sep 30 12:21:21.526420265 2021 debug1: Next authentication method: gssapi-keyex
Thu Sep 30 12:21:21.527334026 2021 debug1: No valid Key exchange context
Thu Sep 30 12:21:21.528265130 2021 debug1: Next authentication method: gssapi-with-mic
Thu Sep 30 12:21:21.529169019 2021 debug1: Unspecified GSS failure. Minor code may provide more information
Thu Sep 30 12:21:21.530080064 2021 No Kerberos credentials available (default cache: KEYRING:persistent:1075)
Thu Sep 30 12:21:21.531015618 2021
Thu Sep 30 12:21:21.531907775 2021 debug1: Unspecified GSS failure. Minor code may provide more information
Thu Sep 30 12:21:21.532835737 2021 No Kerberos credentials available (default cache: KEYRING:persistent:1075)
Thu Sep 30 12:21:21.533666147 2021
Thu Sep 30 12:21:21.534487462 2021 debug1: Next authentication method: publickey
Thu Sep 30 12:21:21.535315203 2021 debug1: Offering RSA public key: /home/user_name/.ssh/id_rsa_dev
Thu Sep 30 12:21:21.536140209 2021 debug1: Server accepts key: pkalg rsa-sha2-512 blen 279
Thu Sep 30 12:21:21.537081945 2021 debug1: Authentication succeeded (publickey).
Thu Sep 30 12:21:21.538008014 2021 Authenticated to remote ([remote_ip]:22).
Thu Sep 30 12:21:21.538923798 2021 debug1: channel 0: new [client-session]
Thu Sep 30 12:21:21.539804699 2021 debug1: Requesting no-more-sessions#openssh.com
Thu Sep 30 12:21:21.540647666 2021 debug1: Entering interactive session.
Thu Sep 30 12:21:21.541469077 2021 debug1: pledge: network
Thu Sep 30 12:21:21.548486172 2021 debug1: client_input_global_request: rtype hostkeys-00#openssh.com want_reply 0
Thu Sep 30 12:21:21.588863420 2021 debug1: Sending environment.
Thu Sep 30 12:21:21.590085121 2021 debug1: Sending env LANG = en_US.UTF-8
Thu Sep 30 12:21:21.590987535 2021 debug1: Sending command: time
Thu Sep 30 12:21:21.879118311 2021
Thu Sep 30 12:21:21.880089083 2021 real 0m0.000s
Thu Sep 30 12:21:21.880993237 2021 user 0m0.000s
Thu Sep 30 12:21:21.881840176 2021 sys 0m0.000s
Thu Sep 30 12:21:21.882675893 2021 debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
Thu Sep 30 12:21:21.883488013 2021 debug1: client_input_channel_req: channel 0 rtype eow#openssh.com reply 0
Thu Sep 30 12:21:21.884312564 2021 debug1: channel 0: free: client-session, nchannels 1
Thu Sep 30 12:21:21.885128132 2021 debug1: fd 1 clearing O_NONBLOCK
Thu Sep 30 12:21:21.886031729 2021 Transferred: sent 2864, received 2784 bytes, in 0.3 seconds
Thu Sep 30 12:21:21.886861936 2021 Bytes per second: sent 8309.0, received 8076.9
Thu Sep 30 12:21:21.887709496 2021 debug1: Exit status 0
EDIT 2:
I ran ssh with subprocess, which isn't quite the same but I think it's a decent comparison. It's much slower than the pure shell methods but still about 2x as fast as paramiko:
test2.py
import subprocess
import time
host = 'remote'
cmd = 'time'
start = time.time()
p = subprocess.Popen(f"ssh {host} {cmd}", shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()
print(p)
print(f'elapsed: {time.time() - start}')
$ time python3 test2.py
(b'', b'\nreal\t0m0.000s\nuser\t0m0.000s\nsys\t0m0.000s\n')
elapsed: 0.4182932376861572
real 0m0.444s
user 0m0.029s
sys 0m0.011s
I am trying to create java jar with the files in AWS S3 by Lambda function using Python
I have tried in both ways
import os
def lambda_handler(event, context):
os.system('jar cf abc.jar, file1.txt, file2.txt')
return "Success"
and
import subprocess
def lambda_handler(event, context):
subprocess.call('jar cf abc.jar, file1.txt, file2.txt')
return "Success"
Remove commas
import subprocess
subprocess.call('jar cf abc.jar *.py')
# import os
# os.system('jar cf abc.jar *.py')
print("Done")
Output :
Done
Process finished with exit code 0
Verifying :
import subprocess
subprocess.call('jar tvf abc.jar')
# import os
# os.system('jar tvf abc.jar')
Output :
0 Wed Feb 28 18:22:32 IST 2018 META-INF/
68 Wed Feb 28 18:22:32 IST 2018 META-INF/MANIFEST.MF
196 Wed Jan 24 20:27:32 IST 2018 dirTut.py
438 Tue Jan 09 20:44:00 IST 2018 epochtodatetime.py
1540 Sun Jan 07 15:30:42 IST 2018 firstServer.py
164 Sun Jan 14 19:34:20 IST 2018 flaskTut.py
295 Tue Jan 09 17:59:20 IST 2018 funcOverriding.py
390 Tue Jan 09 16:55:24 IST 2018 underscore.py
I'm trying to parse some log files in Python, but my responses always return only null bytes.
I've confirmed that the file in question does contain data:
$ zcat Events.log.gz | wc -c
188371128
$ zcat Events.log.gz | head
17 Jan 2018 08:10:35,863: {"deviceType":"A16ZV8BU3SN1N3",[REDACTED]}
17 Jan 2018 08:10:35,878: {"deviceType":"A1CTGXB4BA274T",[REDACTED]}
17 Jan 2018 08:10:35,886: {"deviceType":"A1DL2DVDQVK3Q",[REDACTED]}
17 Jan 2018 08:10:35,911: {"deviceType":"A2CZFJ2RKY7SE2",[REDACTED]}
17 Jan 2018 08:10:35,937: {"deviceType":"A2JTEGS8GUPDOF",[REDACTED]}
17 Jan 2018 08:10:35,963: {"appOtaState":"ota",[REDACTED]}
17 Jan 2018 08:10:35,971: {"deviceType":"A1DL2DVDQVK3Q",[REDACTED]}
17 Jan 2018 08:10:36,006: {"deviceType":"A2JTEGS8GUPDOF",[REDACTED]}
17 Jan 2018 08:10:36,013: {"deviceType":"A1CTGXB4BA274T",[REDACTED]}
17 Jan 2018 08:10:36,041: {"deviceType":"A1DL2DVDQVK3Q",[REDACTED]}
But attempting to read it in Python gives only null bytes:
$ python
Python 2.6.9 (unknown, Sep 14 2016, 17:46:59)
[GCC 4.4.6 20110731 (Red Hat 4.4.6-3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> filename = 'Events.log.gz'
>>> import gzip
>>> content = gzip.open(filename).read()
>>> len(content)
188371128
>>> for i in range(10):
... content[i*10000:(i*10000)+10]
...
'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
I've tried explicitly setting 'mode' to either 'r' or 'rb', with no difference in result.
I've also tried subprocess.Popen(['zcat', filename], stdout=subprocess.PIPE).stdout.read(), with the same response.
Perhaps relevantly, when I tried to zcat the file to another file, the output was a binary file:
$ zcat Events.log.gz > /tmp/logoutput
$ less /tmp/logoutput
"/tmp/logoutput" may be a binary file. See it anyway?
[y]
^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#^#...
$ head /tmp/logoutput
17 Jan 2018 08:10:35,863: {"deviceType":"A16ZV8BU3SN1N3",[REDACTED]}
17 Jan 2018 08:10:35,878: {"deviceType":"A1CTGXB4BA274T",[REDACTED]}
17 Jan 2018 08:10:35,886: {"deviceType":"A1DL2DVDQVK3Q",[REDACTED]}
17 Jan 2018 08:10:35,911: {"deviceType":"A2CZFJ2RKY7SE2",[REDACTED]}
17 Jan 2018 08:10:35,937: {"deviceType":"A2JTEGS8GUPDOF",[REDACTED]}
17 Jan 2018 08:10:35,963: {"appOtaState":"ota",[REDACTED]}
17 Jan 2018 08:10:35,971: {"deviceType":"A1DL2DVDQVK3Q",[REDACTED]}
17 Jan 2018 08:10:36,006: {"deviceType":"A2JTEGS8GUPDOF",[REDACTED]}
17 Jan 2018 08:10:36,013: {"deviceType":"A1CTGXB4BA274T",[REDACTED]}
17 Jan 2018 08:10:36,041: {"deviceType":"A1DL2DVDQVK3Q",[REDACTED]}
i have a list which goes as follows:
-------------------------------------------------------------------------------------------
www.mydomain.de UP Thu May 8 09:10:57 2014
HTTPS OK Thu May 8 09:10:08 2014
HTTPS-Cert OK Thu May 8 09:10:55 2014
-------------------------------------------------------------------------------------------
www.someotherdomain.de UP Thu May 8 09:09:17 2014
HTTPS OK Thu May 8 09:09:30 2014
HTTPS-Cert OK Thu May 8 09:11:10 2014
-------------------------------------------------------------------------------------------
www.somedifferentdomain.at UP Thu May 8 09:08:47 2014
HTTPS OK Thu May 8 09:10:26 2014
HTTPS-Cert OK Thu May 8 09:11:13 2014
-------------------------------------------------------------------------------------------
www.foobladomain.de UP Thu May 8 09:09:17 2014
HTTPS OK Thu May 8 09:09:30 2014
HTTPS-Cert OK Thu May 8 09:11:08 2014
-------------------------------------------------------------------------------------------
www.snafudomain.at UP Thu May 8 09:09:17 2014
HTTP OK Thu May 8 09:09:42 2014
HTTPS OK Thu May 8 09:10:10 2014
HTTPS-Cert OK Thu May 8 09:10:09 2014
-------------------------------------------------------------------------------------------
www.lolnotanotherdomain.de UP Thu May 8 09:06:57 2014
HTTP OK Thu May 8 09:11:10 2014
HTTPS OK Thu May 8 09:11:16 2014
HTTPS-Cert OK Thu May 8 09:11:10 2014
and i have a function which takes the hostname as parameter and prints it out:
please enter hostname to search for: www.snafudomain.at
www.snafudomain.at UP Thu May 8 09:09:17 2014
but what i want to archive is that the following lines after the hostname are printed out until the delimiter line "-----" the function i right now looks like this:
def getChecks(self,hostname):
re0 = "%s" % hostname
mylist = open('myhostlist', 'r')
for i in mylist:
if re.findall("^%s" % re0, str(i)):
print i
else:
continue
is there some easy way to do this? If something is unclear please comment. Thanks in advance
edit
to clarify the output should look like this:
www.mydomain.de UP Thu May 8 09:10:57 2014
HTTPS OK Thu May 8 09:10:08 2014
HTTPS-Cert OK Thu May 8 09:10:55 2014
-------------------------------------------------------------------------------------
just want to print out the lines from the searched domain name till the line with only minuses.
How about not using regex at all?
def get_checks(self, hostname):
record = False
with open('myhostlist', 'r') as file_h:
for line in file_h:
if line.startswith(hostname):
record = True
print(line)
elif line.startswith("---"):
record = False
print(line)
elif record:
print(line)
import re
def get_checks(hostname):
pattern = re.compile(r"{}.*?(?=---)".format(re.escape(hostname)), re.S)
with open("Input.txt") as in_file:
return re.search(pattern, in_file.read())
print get_checks("www.snafudomain.at").group()
This will returns all the lines starting with www.snafudomain.at till it finds ---. The pattern generated will be like this
www\.snafudomain\.at.*?(?=---)
Online Demo
We use re.escape because your hostname has . in it. Since . has special meaning in the Regular Expressions, we just want the RegEx engine to treat . as literal dot.
I have a script that is parsing out fields within email headers that represent dates and times. Some examples of these strings are as follows:
Fri, 10 Jun 2011 11:04:17 +0200 (CEST)
Tue, 1 Jun 2011 11:04:17 +0200
Wed, 8 Jul 1992 4:23:11 -0200
Wed, 8 Jul 1992 4:23:11 -0200 EST
Before I was confronted with the CEST/EST portions at the ends of some the strings I had things working pretty well just using datetime.datetime.strptime like this:
msg['date'] = 'Wed, 8 Jul 1992 4:23:11 -0200'
mail_date = datetime.datetime.strptime(msg['date'][:-6], '%a, %d %b %Y %H:%M:%S')
I tried to put a regex together to match the date portions of the string while excluding the timezone information at the end, but I was having issues with the regex (I couldn't match a colon).
Is using a regex the best way to parse all of the examples above? If so, could someone share a regex that would match these examples? In the end I am looking to have a datetime object.
From python time to age part 2, timezones:
from email import utils
utils.parsedate_tz('Fri, 10 Jun 2011 11:04:17 +0200 (CEST)')
utils.parsedate_tz('Fri, 10 Jun 2011 11:04:17 +0200')
utils.parsedate_tz('Fri, 10 Jun 2011 11:04:17')
The output is:
(2011, 6, 10, 11, 4, 17, 0, 1, -1, 7200)
(2011, 6, 10, 11, 4, 17, 0, 1, -1, 7200)
(2011, 6, 10, 11, 4, 17, 0, 1, -1, None)
Perhaps I misunderstood your question, but wont a simple split suffice?
#!/usr/bin/python
d = ["Fri, 10 Jun 2011 11:04:17 +0200 (CEST)", "Tue, 1 Jun 2011 11:04:17 +0200",
"Wed, 8 Jul 1992 4:23:11 -0200", "Wed, 8 Jul 1992 4:23:11 -0200 EST"]
for i in d:
print " ".join(i.split()[0:5])
Fri, 10 Jun 2011 11:04:17
Tue, 1 Jun 2011 11:04:17
Wed, 8 Jul 1992 4:23:11
Wed, 8 Jul 1992 4:23:11