We Keep Coding

Assistance debuging imaplib errors

Want to search folder "SPAM", for specific_user#any domain, and delete found mail.
Code below ...
import imaplib
box = imaplib.IMAP4_SSL('imap.mail.yahoo.com', 993)
box.login("xxxxxxxx#yahoo.com","xxxxxxxxxx")
box.select('SPAM')
typ, data = box.search(None, 'from','name#*.*')
for num in data[0].split():
box.store(num, '+FLAGS', '\\Deleted')
box.expunge()
box.close()
box.logout()
... is generating these errors below, please assist in debugging, thanks.
Traceback (most recent call last):
File "C:\Users\Desktop\Desktop\Python Spam Buster\test.py", line 6, in <module>
typ, data = box.search(None, 'from','name#*.*')
File "C:\Users\Desktop\AppData\Local\Programs\Python\Python310\lib\imaplib.py", line 734, in search
typ, dat = self._simple_command(name, *criteria)
File "C:\Users\Desktop\AppData\Local\Programs\Python\Python310\lib\imaplib.py", line 1230, in _simple_command
return self._command_complete(name, self._command(name, *args))
File "C:\Users\Desktop\AppData\Local\Programs\Python\Python310\lib\imaplib.py", line 968, in _command
raise self.error("command %s illegal in state %s, "
imaplib.IMAP4.error: command SEARCH illegal in state AUTH, only allowed in states SELECTED
Issue is how to "search" folder using wildcard for domain.
Another issue is how to "select" any folder other then INBOX with yahoo mail!
For example, Gmail you would select SPAM folder as such '[Gmail]/Spam', but Yahoo mail is a mystery how you can access any folder other then INBOX using python imaplib.

SOLUTION:
Got off the phone with Yahoo Premium Support Engineer.
Yahoo mail has a unique way of generating the "Spam" folder filtering system, thus, folder scan would not display it!
Due to the above mentioned, 3rd party mailbox managers, can't access it by it's name, SO, an identical box called "Bulk" is generated, giving you access to all the contents of the "Spam" folder via "Bulk".
When I ran a scan for the folders/boxes, I didn't notice that I had an extra folder called "Bulk", that only appears via the scan, and can't be seen via the web browser.
I'm paraphrasing the above info from the engineer's explanation, hope it makes sense for anyone having the same issue.

Spotipy - Bad Request following doc example

Following the docs here for Auth Code Flow, I can't seem to get the example to work.
import apis
import spotipy
import spotipy.util as util
username = input("Enter username: ")
scope = "user-library-read"
token = util.prompt_for_user_token(username, scope,
client_id=apis.SPOTIFY_CLIENT,
client_secret=apis.SPOTIFY_SECRET,
redirect_uri="http://localhost")
if token:
sp = spotipy.Spotify(auth=token)
results = sp.current_user_saved_tracks()
for item in results['items']:
track = item['track']
print(track['name'] + ' - ' + track['artists'][0]['name'])
else:
print("Can't get token for", username)
I get a 400 Bad Request error:
Traceback (most recent call last):
File "/home/termozour/.local/share/JetBrains/Toolbox/apps/PyCharm-P/ch-0/193.6494.30/plugins/python/helpers/pydev/pydevd.py", line 1434, in _exec
pydev_imports.execfile(file, globals, locals) # execute the script
File "/home/termozour/.local/share/JetBrains/Toolbox/apps/PyCharm-P/ch-0/193.6494.30/plugins/python/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "/media/termozour/linux_main/PycharmProjects/SpotiDowner/main.py", line 27, in <module>
sp = spotipy.Spotify(auth=token)
File "/media/termozour/linux_main/PycharmProjects/SpotiDowner/venv/lib/python3.7/site-packages/spotipy/util.py", line 92, in prompt_for_user_token
token = sp_oauth.get_access_token(code, as_dict=False)
File "/media/termozour/linux_main/PycharmProjects/SpotiDowner/venv/lib/python3.7/site-packages/spotipy/oauth2.py", line 382, in get_access_token
raise SpotifyOauthError(response.reason)
spotipy.oauth2.SpotifyOauthError: Bad Request
Yes, I've seen the other ideas people had to fix this issue but none worked for me:
I tried to reset the client secret, change my URI to a website or localhost, check my client ID and secret ID, and they are all fine.
What I also tried was to go all the way to spotipy/oauth2.py where I get the traceback, added a neat little print(response.text) and got a marvelous {"error":"invalid_grant","error_description":"Invalid authorization code"}
Any ideas or insight?

I have to specify that I ran this code from PyCharm Professional(2019.3.3) and the issue came up from a trailing space.
When the code is run, spotipy asks for that confirmation URL you get with the code based on the redirect URI you set in the app and the code.
The issue was how PyCharm handles URLs in it's terminal window when you run the project. When you put a URL in, and you press enter, PyCharm opens the URL in the browser (for some reason) so the workaround for that is to add a space after the URL and then press enter. That works for PyCharm, but sometimes screws up the code. In this case, it did.
I tried to run the code from PyCharm terminal (python3 ), pasted the URL and directly pressed enter. Not only it didn't open the browser window, it also accepted the URL and let me get my info from Spotify.
In any case, the code is fine, it was the IDE that was breaking it all - maybe a good suggestion would be to remove trailing spaces after the url, if any when parsed from the library itself.
This 'bug' has been reported here and has been marked as fixed (spoiler alert: not fixed) and will be fixed in PyCharm 2020.1
I tried running the code normally from PyCharm 2020.1 and it all works fine, so I can confirm: in my case it was an IDE issue.

SMC-Python Adding and Removing Blacklisted IP's

I'm trying to programmatically add a blacklisted IP to the firewall. I try this but get an error. I'm not that new to python, but I'm not all that proficient in reading the documentation, so here is that if it helps.
https://media.readthedocs.org/pdf/smc-python/latest/smc-python.pdf
https://smc-python.readthedocs.io/en/latest/index.html
from smc import session
from smc_monitoring.monitors.blacklist import BlacklistQuery
from smc.core.engines import Engine
from smc.administration.system import System
session.login(url='http://nope', api_key='supersecret')
print("logged in")
# # Method 1 ERROR
system = System()
print(system.smc_version)
system.blacklist(src='1.1.1.1/32', dst='2.2.2.2/32', duration=3600)
session.logout()
Traceback (most recent call last): File
"/home/matthew/PycharmProjects/GitSMC/BlacklistTest.py", line 12, in
system.blacklist(src='1.1.1.1/32', dst='2.2.2.2/32', duration=3600)
File
"/home/matthew/PycharmProjects/GitSMC/venv/lib/python3.7/site-packages/smc/administration/system.py",
line 159, in blacklist
json=prepare_blacklist(src, dst, duration, **kw))
File
"/home/matthew/PycharmProjects/GitSMC/venv/lib/python3.7/site-packages/smc/base/mixins.py",
line 32, in make_request
result = getattr(request, method)()
File
"/home/matthew/PycharmProjects/GitSMC/venv/lib/python3.7/site-packages/smc/api/common.py",
line 66, in create
return self._make_request(method='POST')
File
"/home/matthew/PycharmProjects/GitSMC/venv/lib/python3.7/site-packages/smc/api/common.py",
line 101, in _make_request
raise err
smc.api.exceptions.ActionCommandFailed: Invalid JSON format: At line 1
and column 17, end_point1 is not recognized as JSON attribute.

There are multiple ways to blacklist, either through the System entry point like you have above, or individually against a single firewall/cluster.
If using the System entry point, the blacklist entry will go to all SMC managed firewalls.
Based on the message, it appears you might be using a newer version of smc-python (i.e. >6.5.x).
In that case it's best to use the engine level blacklisting:
from smc.elements.other import Blacklist
engine = Engine('myfw')
blacklist = Blacklist()
blacklist.add_entry(src='1.1.1.1/32', dst='2.2.2.2/32')
engine.blacklist_bulk(blacklist)
I just noticed that the System entry point does not have a blacklist function for SMC 6.5 (which hasn't technically been fully certified for this library yet), but I will add to the develop branch as 6.5.x will be officially supported in the next couple of weeks.
If you are using SMC version <= 6.4.x, you can use the engine.blacklist, or System.blacklist commands.
DLP

How to extract variable from within a function in Python?

I'm making a program which scrapes bus information from a server and sends it to a user via Facebook messenger. It works fine, but I'm trying to add functionality which splits really long timetables into separate messages. To do this, I had to make an if statement that detects really long timetables, splits them and calls the send_message function from my main file, app.py
Here is the part of the function in app with the variable I need to extract:
for messaging_event in entry["messaging"]:
if messaging_event.get("message"): # someone sent us a message
sender_id = messaging_event["sender"]["id"] # the facebook ID of the person sending you the message
recipient_id = messaging_event["recipient"]["id"] # the recipient's ID, which should be your page's facebook ID
message_text = messaging_event["message"]["text"] # the message's text
tobesent = messaging_event["message"]["text"]
send_message(sender_id, fetch.fetchtime(tobesent))
and here is the if statement in fetch which detects long messages, splits them and calls the send_message function from the other file, app.py:
if len(info["results"]) > 5:
for i, chunk in enumerate(chunks(info, 5), 1):
app.send_message((USER ID SHOULD BE HERE, 'Listing part: {}\n \n{}'.format(i, chunk)))
I'm trying to call the send_message function from app.py, but it requires two arguments, sender_id and the message text. How can I go about getting the sender_id variable from this function and using it in fetch? I've tried returning it and calling the function, but it doesn't work for me.
EDIT:error
Traceback (most recent call last):
File "bus.py", line 7, in <module>
print fetch.fetchtime(stopnum)
File "/home/ryan/fb-messenger-bot-master/fetch.py", line 17, in fetchtime
send_message((webhook(),'Listing part: {}\n \n{}'.format(i, chunk)))
File "/home/ryan/fb-messenger-bot-master/app.py", line 30, in webhook
data = request.get_json()
File "/usr/local/lib/python2.7/dist-packages/werkzeug/local.py", line 343, in __getattr__
return getattr(self._get_current_object(), name)
File "/usr/local/lib/python2.7/dist-packages/werkzeug/local.py", line 302, in _get_current_object
return self.__local()
File "/usr/local/lib/python2.7/dist-packages/flask/globals.py", line 37, in _lookup_req_object
raise RuntimeError(_request_ctx_err_msg)
RuntimeError: Working outside of request context.
This typically means that you attempted to use functionality that needed
an active HTTP request. Consult the documentation on testing for
information about how to avoid this problem.

Shelve module in python not working: "db type cannot be determined"

I am trying to make a simple password-storing program in Python, and it seems pretty simple so I am wondering if I am using shelve wrong.
I have the main .py file:
import shelve
passwords = shelve.open('./passwords_dict.py')
choice = raw_input("Add password (a) or choose site (c)?")
if choice[0] == 'a':
site_key = raw_input("Add for which site? ").lower()
userpass = raw_input("Add any info such as username, email, or passwords: ")
passwords[site_key] = userpass
else:
site = raw_input("Which site? ").lower()
if site in passwords:
print "Info for " + site + ": " + passwords[site]
else:
print site, "doesn't seem to exist!"
print "Done!"
passwords.close()
And the other file, passwords_dict.py, is just an empty dictionary.
But when I try to run the program, I get this error:
Traceback (most recent call last):
File "passwords.py", line 3, in <module>
passwords = shelve.open('passwords_dict.py')
File "/usr/lib/python2.7/shelve.py", line 239, in open
return DbfilenameShelf(filename, flag, protocol, writeback)
File "/usr/lib/python2.7/shelve.py", line 223, in __init__
Shelf.__init__(self, anydbm.open(filename, flag), protocol, writeback)
File "/usr/lib/python2.7/anydbm.py", line 82, in open
raise error, "db type could not be determined"
anydbm.error: db type could not be determined
When I try to use anydbm instead, I get this error:
Traceback (most recent call last):
File "passwords.py", line 3, in <module>
passwords = anydbm.open('passwords_dict.py')
File "/usr/lib/python2.7/anydbm.py", line 82, in open
raise error, "db type could not be determined"
anydbm.error: db type could not be determined
And when I try to use dbm instead, I get this error:
Traceback (most recent call last):
File "passwords.py", line 3, in <module>
passwords = dbm.open('./passwords_dict.py')
dbm.error: (2, 'No such file or directory')
What am I doing wrong? Is there another way to store a dictionary and still be able to extract keys using user input (rather than the entire dictionary, which I suppose is what pickle does)?

I think you're misunderstanding how the shelve module works. It opens a database file. When you try and open an existing file that contains a Python script, it's trying to detect what database type the file contains (since shelve supports multiple backend databases).
I think instead you want something like this:
import os
import shelve
curdir = os.path.dirname(__file__)
passwords = shelve.open(os.path.join(curdir, 'password_db'))
This will create a new file in the same directory as your script called password_db.<db> where <db> is an implementation-specific database file extension.

I've experienced this issue as well. It seems to be related to undocumented conditions for the filename argument of shelve.open. It is currently very intransparent (e.g. shelve.open("/tmp/tmphTTQLda") works while shelve.open("/tmp/tmphTTQLd") doesn't). Failure and success of variable filenames are hard to predict. I requested an explanation in form a documentation enhancement at http://bugs.python.org/issue23174.
In my case opening a persistent dict outside shelve and passing it to shelve.Shelve works, e.g. code
a = dumbdbm.open(tempfile.mkstemp()[1])
b = shelve.Shelf(dict=a)
and do with b what you'd have done with the return value of shelve.open.

There is one bug with anydb https://bugs.python.org/issue13007 that could not use the right identification for gdbm files.
So if you are trying to open a valid gdbm file with shelve and is thorwing that error use this instead:
mod = __import__("gdbm")
file = shelve.Shelf(mod.open(filename, flag))
More information on this question: shelve db type could not be determined, whichdb is not recognizing gdb