We Keep Coding

pysftp throwing paramiko.ssh_exception.SSHException even though known_hosts file provided?

Getting error
paramiko.ssh_exception.SSHException: No hostkey for host target.org found.
when using pysftp (for a connection that requires a specific port), even though I am providing the same known_hosts file that was initially used to connect to that location (following the post here). Ie. did...
[airflow#airflowetl reporting]$ sftp -oPort=15259 my_user#target.org
The authenticity of host '[target.org]:15259 ([205.172.2.88]:15259)' can't be established.
RSA key fingerprint is SHA256:UM6OflG0rkcYohes7qOlYoJZ4TIqVd0JQSh7HXYZQVA.
RSA key fingerprint is MD5:33:c2:30:22:57:5b:57:98:2f:11:07:4d:a3:4a:10:0f.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '[target.org]:15259,[205.172.2.88]:15259' (RSA) to the list of known hosts.
password
Enter password for my_user
Password:
Connected to target.org.
sftp> ls
csv_drop test_results
sftp> exit
and then copied the /home/me/.ssh/known_hosts to another location
[airflow#airflowetl .ssh]$ ls -lha
total 8.0K
drwx------ 2 airflow airflows 25 Oct 19 17:31 .
drwx------. 32 airflow airflows 4.0K Oct 19 17:31 ..
-rw-r--r-- 1 airflow airflows 777 Oct 19 17:32 known_hosts
[airflow#airflowetl .ssh]$ ls -lha /home/airflow/projects/backups/reporting/configs/known_hosts
-rw-r--r-- 1 airflow airflows 777 Oct 19 17:34 /home/airflow/projects/backups/reporting/configs/known_hosts
(noting that the permissions are the name for the original and copy) that gets used like...
# connect to ftp location
assert os.path.isfile(os.path.join(PROJECT_HOME, "configs", "known_hosts"))
cnopts = pysftp.CnOpts(knownhosts=os.path.join(PROJECT_HOME, "configs", "known_hosts"))
#cnopts = pysftp.CnOpts(knownhosts="/home/airflow/.ssh/known_hosts")
HOSTNAME = "target.org"
PORT = 15259
sftp = pysftp.Connection(HOSTNAME,
port=PORT,
username=CREDS["sink"]["ftp_creds"]["username"],
password=CREDS["sink"]["ftp_creds"]["password"],
cnopts=cnopts)
print(sftp.pwd())
print(sftp.listdir())
sftp.cwd(CONF["reporting_configs"]["to"]["share_folder_path"])
print(sftp.pwd())
print(sftp.listdir())
Yet, when running the script I get the error saying it could not find the hostkey (Note that even using the original known_hosts file path throws the same error (and happens whether using hostname or the IP)). What could be causing this? Any more debugging steps I could try to get more info? Don't have much experience with this kind of thing.
Looking in the debugger while running, I see that the cnopts hostkeys entries does seem to contain the right hostkey...
<HostKeyEntry ['[target.org]:15259', '[205.172.2.88]:15259']: <paramiko.rsakey.RSAKey object at 0x7f8d752d4208>>
Note that the hostkey entry is '[target.org]:15259' (ie. it is combining the specified port), even though the name provided as the service name to the connection function is just 'target.org'
The full traceback looks like...
Traceback (most recent call last):
File "./source/local2ftp.2.py", line 52, in <module>
cnopts=cnopts)
File "/home/airflow/projects/backups/reporting/venv/lib/python3.6/site-packages/pysftp/__init__.py", line 132, in __init__
self._tconnect['hostkey'] = self._cnopts.get_hostkey(host)
File "/home/airflow/projects/backups/reporting/venv/lib/python3.6/site-packages/pysftp/__init__.py", line 71, in get_hostkey
raise SSHException("No hostkey for host %s found." % host)
paramiko.ssh_exception.SSHException: No hostkey for host target.org found.
Exception ignored in: <bound method Connection.__del__ of <pysftp.Connection object at 0x7f1a61df6f98>>
Traceback (most recent call last):
File "/home/airflow/projects/backups/reporting/venv/lib/python3.6/site-packages/pysftp/__init__.py", line 1013, in __del__
self.close()
File "/home/airflow/projects/backups/reporting/venv/lib/python3.6/site-packages/pysftp/__init__.py", line 784, in close
if self._sftp_live:
AttributeError: 'Connection' object has no attribute '_sftp_live'
Note that the user I used to initially connect via sftp in the command line and am using in the pysftp script is not the user running the script (the user I used to connect to the sftp server is a special set of credentials for connecting). I don't know if this is relevant.

The pysftp does not support the host key entries with the port.
Either skip pysftp and use Paramiko directly.
Or hack it by replacing the [target.org]:15259 with target.org in the known_hosts file.

TLDR: The code expects the pystfp.Connection hostname to match that in the hostkey file (eg. ~/.ssh/known_hosts), but if the rsa entry in the hostkey file entry was created with a connection that specified the port and thus caused the rsa entry in known_hosts to be formatted in a way that pysftp could not understand / handle,
like...
[airflow#airflowetl reporting]$ sftp -oPort=15259 my_user#target.org
The rsa entry that gets created looks like...
'[target.org]:15259,[205.172.2.88]:15259 ssh-rsa AAAAB3HJGVJGCTCKHGVYTVKUH===...'
so when you use this hostkey file for the knownhosts in...
cnopts = pysftp.CnOpts(knownhosts=os.path.join(path, to, hostkey, or, known_hosts, file))
hostname = "target.org"
port = 15259
sftp = pysftp.Connection(hostname, port, username=user, password=pass, cnopts=cnopts)
the code ends up checking the hostkey file for an entry that looks like
"target.org"
(what you entered as the hostname) but only finds
"[target.org]:15259"
(the entry in the hostkey / known_hosts file formatted for the specific port you connected on) so throws an error.
Note also that you can't just use "[target.org]:15259" as the hostname for the pysftp.Connection() function either just to get it to match when the pysftp code does this internal check because it will say that it does not recognize the service name since (apparently) that's not a valid servicename representation (ie. it won't just know to separate that string into the hostname and port components).
What I had to do was...
Copy the known_hosts file that was created when I initially connected to the target.org host
then in that file, manually edit it to look like
'target.org,205.172.2.88 ssh-rsa AAAAB3HJGVJGCTCKHGVYTVKUH===...'
Specifically in the code, the pysftp.Connection() function...
Tries to get the hostkey
and uses the underlying Paraminko module to try to find the hostkey entry from the knownhosts file given in the cnopt arg
which uses the function here to match the string literal entries from the hostkey file with the hostname that you entered as the pysftp.Connection arg (which, if you connected to using a specific port, created an rsa entry that when taken as a literal is not formatted the way you likely are passing in the hostname arg and is not even a valid host name), thus confusing the matching logic and causing it to not find any matches
thus causing an exception to be raised, here back in the pysftp code

Error sending email in django through gmail

I am trying to send a mail using django through gmail.
Following is my settings.py code:
EMAIL_HOST = 'smtp.gmail.com'
EMAIL_PORT = 587
EMAIL_HOST_USER = 'me#mycompany.com'
EMAIL_HOST_PASSWORD = 'mypassword'
EMAIL_USE_TLS = True
view.py I am using method send_mail():
send_mail(subject, contact_message, emailfrom, emailto)
When I execute the code & enter details in my contact form & hit submit.
I get the below error:
smtplib.SMTPNotSupportedError: STARTTLS extension not supported by server.
Please suggest!

I don't know Gmail, and I don't know Django, but I hope the following explanations will help you.
A quick analysis shows that your situation is kind of odd. There is a simple method to test what capabilities / extension an SMTP server provides: Get a command line telnet client, connect to the server in question and query its capabilities. The following shows how this is done in Linux, but it's basically the same with Windows:
root#spock:~# telnet smtp.gmail.com 587 <-- Type this on the command line
Trying 74.125.71.109... <-- This is output
Connected to gmail-smtp-msa.l.google.com. <-- This is output
Escape character is '^]'. <-- This is output
220 smtp.gmail.com ESMTP s196sm2489285wmb.6 - gsmtp <-- This is output
EHLO localhost.com <-- **YOU** must type this
250-smtp.gmail.com at your service, [46.83.27.246] <-- The following lines are output
250-SIZE 35882577
250-8BITMIME
250-STARTTLS <-- NOTE THIS LINE
250-ENHANCEDSTATUSCODES
250-PIPELINING
250-CHUNKING
250 SMTPUTF8
The key points here are the second command line parameter to the telnet command which designates which port to use, and the EHLO command you type (make sure you really type EHLO and not HELO which is another command).
As you can see from the example, smtp.gmail.com definitely supports the STARTTLS command / extension. I have tested this in a normal environment; my Linux box is behind a NAT router which is the usual setup for private households and small companies. No port forwarding or other special configuration is in place.
So the first thing I would recommend is that you repeat this simple test at your place. Then,
If you get the same result as me, especially if the output you get also contains the line 250-STARTTLS, there is absolutely no explanation what happens here besides errors in Django itself. Are you using the most recent version? I can't help you here, because I never have used it and I don't know anything about it.
If the line 250-STARTTLS is not in the output you get, something completely weird must go on. As my example shows, STARTTLS is supported at least for clients at my place, and there is no reason why it shouldn't be supported for clients at your place.
Exceptions might be China or similar countries where governments try to prevent the usage of encryption. So I could imagine that Google turns off the STARTTLS extension based on geo-blocking when a client from such a country connects. But I really don't know! People who do are encouraged to leave a comment ...
Perhaps you are using a proxy which disturbs the communication between your client (Django) and the SMTP server.
I you still haven't found the problem, you could do the following:
You could try port 25 instead of port 587. I have verified that smtp.gmail.com supports STARTTLS on port 25 as well. With most email clients, it doesn't matter if you use port 25 or port 587. You should be able to make Django use port 25 by saying EMAIL_PORT = 25 in your configuration file.
You could try to use implicit TLS (most often called SSL) instead of the explicit STARTTLS. The port which is usually used for this is 465. To implement this, say EMAIL_PORT = 465, EMAIL_USE_TLS = False and EMAIL_USE_SSL = True in your configuration file.

Sending email using smtplib in ubuntu docker throws "421 4.3.2 Connection rate limit exceeded"

I am using smtplib to send emails with my SMTP server as localhost and port number 25 in a ubuntu machine. It was working fine until I got connection limit exceeded error. The stack trace is as follows:
File "/src/tasks/driver/sendEmail.py", line 57, in run
s = smtplib.SMTP(smtp)
File "/root/anaconda/lib/python2.7/smtplib.py", line 258, in __init__
raise SMTPConnectError(code, msg)
SMTPConnectError: (421, '4.3.2 Connection rate limit exceeded.')
People suggested to edit /etc/mail/sendmail.mc file to change the window size and throttle.
define(confCONNECTION_RATE_THROTTLE',15')dnl
define(confCONNECTION_RATE_WINDOW_SIZE',60s')dnl
The real problem is I don't see any such configuration file. No traces of sendmail installation in the docker image or the host machine. How does smtplib send emails with localhost as the smtp server? How to increase the throttle if I don't have the configuration file in the machines? Is it because my company manages the SMTP server for all the machines in a separate file?

Send email via Python/django on 1&1

I am developing a website with django framework. And I recently tried to release it on a 1&1 shared hosting.
I managed to make the project run well, except for one last “detail”: I can't send email from django.
I tried almost everything in settings (different emails, ports, etc.), but each time I got a beautiful '500 Internal Server Error' =/ (whereas it went fine on a free alwaysdata server)
In order to find the origin of this issue, I tried different things:
to test send_mail and EmailMessage via the python interpreter:
>>> send_mail('a subject', 'a test message', 'mymail#gmail.com', ['mymail#gmail.com'])
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Subject: a subject
From: mymail#gmail.com
To: mymail#gmail.com
Date: Tue, 14 Jan 2014 22:12:48 -0000
Message-ID: <20140114221248.6718.86150#infong-fr25.kundenserver.de>
a test message
-------------------------------------------------------------------------------
1
But... I got nothing in my inbox.
My settings being:
EMAIL_HOST_PASSWORD = 'mypassword'
EMAIL_USE_TLS = True
EMAIL_HOST = 'smtp.gmail.com'
DEFAULT_FROM_EMAIL = 'mymail#gmail.com'
EMAIL_HOST_USER = 'mymail#gmail.com'
EMAIL_PORT = 587
to connect to an SMTP server via the python interpreter:
./manage.py shell
>>> from smtplib import SMTP
>>> smtp_conn = SMTP()
>>> smtp_conn.connect('smtp.gmail.com', 25)
And nothing happens (I tried it with auth.smtp.1and1.fr, or port 587...); when I interrupt the process, it tells me:
^CTraceback (most recent call last):
File "<console>", line 1, in <module>
File "/usr/lib/python2.6/smtplib.py", line 295, in connect
self.sock = self._get_socket(host, port, self.timeout)
File "/usr/lib/python2.6/smtplib.py", line 273, in _get_socket
return socket.create_connection((port, host), timeout)
File "/usr/lib/python2.6/socket.py", line 554, in create_connection
sock.connect(sa)
File "<string>", line 1, in connect
KeyboardInterrupt
to test with a php file:
With a basic:
<?php
if (mail("mymail#gmail.com", "a subject", "a test")) {
echo("<p>Email successfully sent!</p>");
}
else {
echo("<p>Email delivery failed…</p>");
}
?>
And the mail was sent! So, it seems that without SMTP connection, it is possible (at least, via php).
Conclusion
So, my questions are: do you think I can manage to correct this bug, and otherwise is it possible to send mail without smtp connection ("like php")?

1and1 does not allow SMTP within 1and1. To send an email when running on 1and1...
#!/usr/bin/python
import os
mail_to = "sam#comcast.net"
mail_from = "steve#example.com"
subject = "test message"
header = """From: {0}
To: {1}
Subject: {2}
""".format(mail_from, mail_to, subject)
msg = header + "a test from me"
sendmail = os.popen("/usr/lib/sendmail -t", "w")
sendmail.write(msg)
sendmail.close()
The blog post linked shows an example that sends mail using SMTP through 1and1 via an external machine. Perhaps you can override Django's send_mail (or maybe this gives you [or someone else] a hint as to why it doesn't work.) This should allow you to send email in the Python interpreter, anyway.
Source: https://flenniken.net/blog/send-email-using-python-and-1and1/

"getaddrinfo() argument 1 must be string or None" error starting dev_appserver.py on Linux

I'm getting re-started with App Engine after not having used it in a while. I'm using the 64-bit Linux Go runtime, version 1.8.1.
I believe I'm following the steps from the documentation correctly, and I believe I'm doing what's worked correctly in the past, but I'm getting this error when attempting to launch dev_appserver.py:
$ dev_appserver.py .
INFO 2013-07-11 07:24:45,919 sdk_update_checker.py:244] Checking for updates to the SDK.
INFO 2013-07-11 07:24:46,230 sdk_update_checker.py:288] This SDK release is newer than the advertised release.
WARNING 2013-07-11 07:24:46,443 simple_search_stub.py:955] Could not read search indexes from /tmp/appengine.batterybotinfo.darshan/search_indexes
Traceback (most recent call last):
File "/home/darshan/bin/dev_appserver.py", line 182, in
_run_file(__file__, globals())
File "/home/darshan/bin/dev_appserver.py", line 178, in _run_file
execfile(script_path, globals_)
File "/home/darshan/software/google_appengine/google/appengine/tools/devappserver2/devappserver2.py", line 695, in
main()
File "/home/darshan/software/google_appengine/google/appengine/tools/devappserver2/devappserver2.py", line 688, in main
dev_server.start(options)
File "/home/darshan/software/google_appengine/google/appengine/tools/devappserver2/devappserver2.py", line 659, in start
apis.start()
File "/home/darshan/software/google_appengine/google/appengine/tools/devappserver2/api_server.py", line 137, in start
super(APIServer, self).start()
File "/home/darshan/software/google_appengine/google/appengine/tools/devappserver2/wsgi_server.py", line 295, in start
if self._start_all_dynamic_port(host_ports):
File "/home/darshan/software/google_appengine/google/appengine/tools/devappserver2/wsgi_server.py", line 348, in _start_all_dynamic_port
server.start()
File "/home/darshan/software/google_appengine/google/appengine/tools/devappserver2/wsgi_server.py", line 194, in start
socket.SOCK_STREAM, 0, socket.AI_PASSIVE)
TypeError: getaddrinfo() argument 1 must be string or None
My first thought was that I might be using an incorrect version of Python. Sure enough, I'm using 2.7.5, and the documentation clearly states that 2.5 is necessary. However, the documentation seems to be outdated, because after installing 2.5 and setting my system to use it, I got this error:
Error: Python 2.5 is not supported. Please use version 2.7.
Okay, so back to 2.7.5 and my initial error.
I'm not sure if this is a bug in the dev_appserver.py Python code (I'm guessing not, as it's been out for a month), an issue with my Python installation, or something else about my system that isn't configured according to Google's expectations.
I'd rather not mess with the dev_appserver.py code unless necessary, but I'm happy to poke at it to help figure out what's going wrong. The error is on line 194; here are lines 190-195:
# AF_INET or AF_INET6 socket
# Get the correct address family for our host (allows IPv6 addresses)
host, port = self.bind_addr
try:
info = socket.getaddrinfo(host, port, socket.AF_UNSPEC,
socket.SOCK_STREAM, 0, socket.AI_PASSIVE)
I've determined that the containing method is called twice. The first time host is always "127.0.0.1" and port is 0. The second time is the one that crashes; host is always 10 (an int, not a string), and port is a seemingly-random five-digit int.
I've tried hard-coding host to "127.0.0.1" and port to either 8080 or 0, but then I get another error. I feel in over my head, and I suspect I'm not going to resolve the real issue by changing things I don't really understand. Googling for the error message hasn't helped.

Persistent Googling eventually paid off. Despite this question having a very different (and much more informative) error message, the solution turned out to be the same: ensure that /etc/hosts contains only a single entry for localhost.
Notably, my system contained both of these lines:
127.0.0.1 localhost
::1 localhost
Commenting out the second (and adding a comment to document why) solved my issue:
127.0.0.1 localhost
# Having multiple localhost entries causes App Enginge dev_appserver.py to fail.
# IPv6 not currently needed, and the dev server IS needed, so commenting out.
#::1 localhost
dev_appserver.py now starts and works properly.

If you edit lines 189-194 to this, it should work until Google releases an update. This is basically just checking the type of host and returning early if it isn't a string.
189 # AF_INET or AF_INET6 socket
190 # Get the correct address family for our host (allows IPv6 addresses)
191 host, port = self.bind_addr
192 try:
193 if type(host) is not str:
194 return
195 info = socket.getaddrinfo(host, port, socket.AF_UNSPEC,
196 socket.SOCK_STREAM, 0, socket.AI_PASSIVE)