We Keep Coding

Assistance debuging imaplib errors

Want to search folder "SPAM", for specific_user#any domain, and delete found mail.
Code below ...
import imaplib
box = imaplib.IMAP4_SSL('imap.mail.yahoo.com', 993)
box.login("xxxxxxxx#yahoo.com","xxxxxxxxxx")
box.select('SPAM')
typ, data = box.search(None, 'from','name#*.*')
for num in data[0].split():
box.store(num, '+FLAGS', '\\Deleted')
box.expunge()
box.close()
box.logout()
... is generating these errors below, please assist in debugging, thanks.
Traceback (most recent call last):
File "C:\Users\Desktop\Desktop\Python Spam Buster\test.py", line 6, in <module>
typ, data = box.search(None, 'from','name#*.*')
File "C:\Users\Desktop\AppData\Local\Programs\Python\Python310\lib\imaplib.py", line 734, in search
typ, dat = self._simple_command(name, *criteria)
File "C:\Users\Desktop\AppData\Local\Programs\Python\Python310\lib\imaplib.py", line 1230, in _simple_command
return self._command_complete(name, self._command(name, *args))
File "C:\Users\Desktop\AppData\Local\Programs\Python\Python310\lib\imaplib.py", line 968, in _command
raise self.error("command %s illegal in state %s, "
imaplib.IMAP4.error: command SEARCH illegal in state AUTH, only allowed in states SELECTED
Issue is how to "search" folder using wildcard for domain.
Another issue is how to "select" any folder other then INBOX with yahoo mail!
For example, Gmail you would select SPAM folder as such '[Gmail]/Spam', but Yahoo mail is a mystery how you can access any folder other then INBOX using python imaplib.

SOLUTION:
Got off the phone with Yahoo Premium Support Engineer.
Yahoo mail has a unique way of generating the "Spam" folder filtering system, thus, folder scan would not display it!
Due to the above mentioned, 3rd party mailbox managers, can't access it by it's name, SO, an identical box called "Bulk" is generated, giving you access to all the contents of the "Spam" folder via "Bulk".
When I ran a scan for the folders/boxes, I didn't notice that I had an extra folder called "Bulk", that only appears via the scan, and can't be seen via the web browser.
I'm paraphrasing the above info from the engineer's explanation, hope it makes sense for anyone having the same issue.

interaction with rinkeby smart contract using web3.py python

Here is my piece of web3.py code. I have implemented the smart contract on rinkeby testnet using remix. I am able to call other functions, but when I am calling the transact function I am getting following error.
CODE:
web3 = Web3(Web3.HTTPProvider(url))
web3.middleware_onion.inject(geth_poa_middleware, layer=0)
print(web3.isConnected())
class SendCoin:
def send_coin_on_reps(self, reps):
print(web3.isConnected())
# web3.eth.defaultAccount = web3.eth.accounts[-1]
# INTRACTING WITH REMIX CONTRACT
abi = json.load()
deployed_contract_address = web3.toChecksumAddress('0x40eab3d93EFE536560Ad5802B15EAb56203c3A48')
contract = web3.eth.contract(address = deployed_contract_address, abi = abi)
print(contract)
reward = contract.functions.getReward().call()
print("reward = ", reward)
tx_hash = contract.functions.setReward(reps).transact()
ERROR:
File "/home/sohail/Blockchain/local_ganache_network_web3_app.py", line 48, in send_coin_on_reps
tx_hash = contract.functions.setReward(reps).transact()
File "/home/sohail/anaconda3/lib/python3.9/site-packages/web3/contract.py", line 997, in transact
return transact_with_contract_function(
File "/home/sohail/anaconda3/lib/python3.9/site-packages/web3/contract.py", line 1590, in transact_with_contract_function
txn_hash = web3.eth.send_transaction(transact_transaction)
File "/home/sohail/anaconda3/lib/python3.9/site-packages/web3/eth.py", line 815, in send_transaction
return self._send_transaction(transaction)
File "/home/sohail/anaconda3/lib/python3.9/site-packages/web3/module.py", line 57, in caller
result = w3.manager.request_blocking(method_str,
File "/home/sohail/anaconda3/lib/python3.9/site-packages/web3/manager.py", line 198, in request_blocking
return self.formatted_response(response,
File "/home/sohail/anaconda3/lib/python3.9/site-packages/web3/manager.py", line 171, in formatted_response
raise ValueError(response["error"])
ValueError: {'code': -32601, 'message': 'The method eth_sendTransaction does not exist/is not available'}

It looks to me like you're trying to use a hosted node as if it were a local node. You can read more about the difference in the web3.py docs.
In short: there is no eth_sendTransaction on a hosted node (like Infura, Alchemy, etc), because hosted nodes don't have access to your private keys.
In order to transact(), you need an account funded with ether. Where is the private key for that account? If you have it in python, then you'll want to use the API for signing a contract transaction with local keys.
Otherwise, if the private key is in a local node, like geth, then you'll need to connect to that correctly, probably using an IPC connection. Then a simple transact() invocation should run fine.

Python: Uploading video to youtube using google provided code snippet does not work

I'm trying to upload a video to Youtube using a python script.
So the code given here (upload_video.py) is supposed to work and I've followed the set up which includes enabling the Youtube API and getting OAuth secret keys and what not. You may notice that the code is in Python 2 so I used 2to3 to make it run with python3.7. The issue is that for some reason, I'm asked to login when I execute upload_video.py:
Now this should not be occuring as that's the whole point of having a client_secrets.json file, that you don't need to explicitly login. So once I exit this in-shell browser, Here's what I see:
Here's the first line:
/usr/lib/python3.7/site-packages/oauth2client/_helpers.py:255: UserWarning: Cannot access upload_video.py-oauth2.json: No such file or directory
warnings.warn(_MISSING_FILE_MESSAGE.format(filename))
Now I don't understand why upload_video.py-oauth2.json is needed as in the upload_video.py file, the oauth2 secret file is set as "client_secrets.json".
Anyways, I created the file upload_video.py-oauth2.json and copied the contents of client_secrets.json to it. I didn't get the weird login then but I got another error:
Traceback (most recent call last):
File "upload_video.py", line 177, in <module>
youtube = get_authenticated_service(args)
File "upload_video.py", line 80, in get_authenticated_service
credentials = storage.get()
File "/usr/lib/python3.7/site-packages/oauth2client/client.py", line 407, in get
return self.locked_get()
File "/usr/lib/python3.7/site-packages/oauth2client/file.py", line 54, in locked_get
credentials = client.Credentials.new_from_json(content)
File "/usr/lib/python3.7/site-packages/oauth2client/client.py", line 302, in new_from_json
module_name = data['_module']
KeyError: '_module'
So basically now I've hit a dead end. Any ideas about what to do now?

See the code of function get_authenticated_service in upload_video.py: you should not create the file upload_video.py-oauth2.json by yourself! This file is created upon the completion of the OAuth2 flow via the call to run_flow within get_authenticated_service.
Also you may read the doc OAuth 2.0 for Mobile & Desktop Apps for thorough info about the authorization flow on standalone computers.

Using Python to Manage AWS

I’m trying to use Python to create EC2 instances but I keep getting these errors.
Here is my code:
#!/usr/bin/env python
import boto3
ec2 = boto3.resource('ec2')
instance = ec2.create_instances(
ImageId='ami-0922553b7b0369273',
MinCount=1,
MaxCount=1,
InstanceType='t2.micro')
print instance[0].id
Here are the errors I'm getting
Traceback (most recent call last):
File "./createinstance.py", line 8, in <module>
InstanceType='t2.micro')
File "/usr/lib/python2.7/site-packages/boto3/resources/factory.py", line 520, in do_action
response = action(self, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/boto3/resources/action.py", line 83, in __call__
response = getattr(parent.meta.client, operation_name)(**params)
File "/usr/lib/python2.7/site-packages/botocore/client.py", line 320, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/lib/python2.7/site-packages/botocore/client.py", line 623, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidAMIID.NotFound) when calling the RunInstances operation: The image id '[ami-0922553b7b0369273]' does not exist
I also get an error when trying to create a key pair
Here's my code for creating the keypair
import boto3
ec2 = boto3.resource('ec2')
# create a file to store the key locally
outfile = open('ec2-keypair.pem','w')
# call the boto ec2 function to create a key pair
key_pair = ec2.create_key_pair(KeyName='ec2-keypair')
# capture the key and store it in a file
KeyPairOut = str(key_pair.key_material)
print(KeyPairOut)
outfile.write(KeyPairOut)
response = ec2.instance-describe()
print response
Here's are the error messages
./createkey.py: line 1: import: command not found
./createkey.py: line 2: syntax error near unexpected token `('
./createkey.py: line 2: `ec2 = boto3.resource('ec2')'
What I am I missing?

For your first script, one of two possibilities could be occurring:
1. The AMI you are referencing by the ID is not available because the key is incorrect or the AMI doesn't exist
2. AMI is unavailable in the region that your machine is setup for
You most likely are running your script from a machine that is not configured for the correct region. If you are running your script locally or on a server that does not have roles configured, and you are using the aws-cli, you can run the aws configure command to configure your access keys and region appropriately. If you are running your instance on a server with roles configured, your server needs to be ran in the correct region, and your roles need to allow access to EC2 AMI's.
For your second question (which in the future should probably be posted separate), your syntax error in your script is a side effect of not following the same format for how you wrote your first script. It is most likely that your python script is not in fact being interpreted as a python script. You should add the shebang at the top of the file and remove the spacing preceding your import boto3 statement.
#!/usr/bin/env python
import boto3
# create a file to store the key locally
outfile = open('ec2-keypair.pem','w')
# call the boto ec2 function to create a key pair
key_pair = ec2.create_key_pair(KeyName='ec2-keypair')
# capture the key and store it in a file
KeyPairOut = str(key_pair.key_material)
print(KeyPairOut)
outfile.write(KeyPairOut)
response = ec2.instance-describe()
print response

AWS - OS Error permission denied Lambda Script

I'm trying to execute a Lambda Script in Python with an imported library, however I'm getting permission errors.
I am also getting some alerts about the database, but database queries are called after the subprocess so I don't think they are related. Could someone explain why do I get error?
Alert information
Alarm:Database-WriteCapacityUnitsLimit-BasicAlarm
State changed to INSUFFICIENT_DATA at 2016/08/16. Reason: Unchecked: Initial alarm creation
Lambda Error
[Errno 13] Permission denied: OSError Traceback (most recent call last):File "/var/task/lambda_function.py", line 36, in lambda_handler
xml_output = subprocess.check_output(["./mediainfo", "--full", "--output=XML", signed_url])
File "/usr/lib64/python2.7/subprocess.py", line 566, in check_output process = Popen(stdout=PIPE, *popenargs, **kwargs)
File "/usr/lib64/python2.7/subprocess.py", line 710, in __init__ errread, errwrite) File "/usr/lib64/python2.7/subprocess.py", line 1335, in _execute_child raise child_exception
OSError: [Errno 13] Permission denied
Lambda code
import logging
import subprocess
import boto3
SIGNED_URL_EXPIRATION = 300 # The number of seconds that the Signed URL is valid
DYNAMODB_TABLE_NAME = "TechnicalMetadata"
DYNAMO = boto3.resource("dynamodb")
TABLE = DYNAMO.Table(DYNAMODB_TABLE_NAME)
logger = logging.getLogger('boto3')
logger.setLevel(logging.INFO)
def lambda_handler(event, context):
"""
:param event:
:param context:
"""
# Loop through records provided by S3 Event trigger
for s3_record in event['Records']:
logger.info("Working on new s3_record...")
# Extract the Key and Bucket names for the asset uploaded to S3
key = s3_record['s3']['object']['key']
bucket = s3_record['s3']['bucket']['name']
logger.info("Bucket: {} \t Key: {}".format(bucket, key))
# Generate a signed URL for the uploaded asset
signed_url = get_signed_url(SIGNED_URL_EXPIRATION, bucket, key)
logger.info("Signed URL: {}".format(signed_url))
# Launch MediaInfo
# Pass the signed URL of the uploaded asset to MediaInfo as an input
# MediaInfo will extract the technical metadata from the asset
# The extracted metadata will be outputted in XML format and
# stored in the variable xml_output
xml_output = subprocess.check_output(["./mediainfo", "--full", "--output=XML", signed_url])
logger.info("Output: {}".format(xml_output))
save_record(key, xml_output)
def save_record(key, xml_output):
"""
Save record to DynamoDB
:param key: S3 Key Name
:param xml_output: Technical Metadata in XML Format
:return:
"""
logger.info("Saving record to DynamoDB...")
TABLE.put_item(
Item={
'keyName': key,
'technicalMetadata': xml_output
}
)
logger.info("Saved record to DynamoDB")
def get_signed_url(expires_in, bucket, obj):
"""
Generate a signed URL
:param expires_in: URL Expiration time in seconds
:param bucket:
:param obj: S3 Key name
:return: Signed URL
"""
s3_cli = boto3.client("s3")
presigned_url = s3_cli.generate_presigned_url('get_object', Params={'Bucket': bucket, 'Key': obj},
ExpiresIn=expires_in)
return presigned_url

I'm fairly certain that this is a restriction imposed by the lambda execution environment, but it can be worked around by executing the script through the shell.
Try providing shell=True to your subprocess call:
xml_output = subprocess.check_output(["./mediainfo", "--full", "--output=XML", signed_url], shell=True)

I encountered a similar situation. I was receiving the error:
2016-11-28T01:49:01.304Z d4505c71-b50c-11e6-b0a1-65eecf2623cd Error: Command failed: /var/task/node_modules/youtube-dl/bin/youtube-dl --dump-json -f best https://soundcloud.com/bla/blabla
python: can't open file '/var/task/node_modules/youtube-dl/bin/youtube-dl': [Errno 13] Permission denied
For my (and every other) Node Lambda project containing third party libraries, there will be a directory called "node_modules" (most tutorials, such as this one, will detail how this directory is created) that has all the third party packages and their dependencies. The same principles apply to the other supported languages (currently Python and Java). THESE ARE THE FILES THAT AMAZON IS ACTUALLY PUTTING ON THE LAMBDA AMIS AND ATTEMPTING TO USE. So, to fix the issue, run this on the node_modules directory (or whatever directory your third party libraries live in):
chmod -R 777 /Users/bla/bla/bla/lambdaproject/node_modules
This command means making the file readable, writable and executable by all users. Which is apparently what the servers that execute Lambda functions need, in order to work. Hopefully this helps!