Paramiko holds on put command - python

I'm trying to send a zip file to a remote server via sftp but it keeps holding up the script. On windows with a decent connection the file is sent without a problem but on a linux it doesn't work. I've tried raising an exception when the upload finishes but with no luck.
These are the log messages:
Jul 28 10:24:40 igt22-dev mqtt.sh[1353]: 46.21875 46.258296966552734
Jul 28 10:24:40 igt22-dev mqtt.sh[1353]: 46.25 46.258296966552734
Jul 28 10:24:40 igt22-dev mqtt.sh[1353]: 46.258296966552734 46.258296966552734
Jul 28 10:24:40 igt22-dev mqtt.sh[1353]: finished
and this is the code:
ftp_client = server_client.open_sftp()
source = "/usr/local/bin/log_history.zip"
destination = "/home/ubuntu/log_history.zip"
def callback_fun(current, maximum):
print(current / 1048576, maximum / 1048576)
sys.stdout.flush()
if current == maximum:
print("finished")
sys.stdout.flush()
raise Exception("finished")
ftp_client.put(source, destination, callback=callback_fun)
ftp_client.close()
print("exited")
sys.stdout.flush()

Not a fix but I managed to find a workaround. I run the code in a separate thread and when it finishes i just kill it

Related

Trying to open an existing named pipe in Windows 10 - errno 22, Invalid Argument, but why?

I'm writing/experimenting with a python script that needs to connect to an existing named pipe in Windows 10. I'm using Python 3.8.
The Named Pipe
Specifically, I'm trying to connect to a pipe named SendPipe1 and write 14 ascii-encoded characters to it.
The pipe was created by some other software (driver?) for a device connected to my PC. I am connecting to it to try to interact with the device in the same way the manufacturer-provided software does. The manufacturer-provided software connects to this same pipe and sends a coded message to it, which commands the device to do something. This is what I'd like to do.
I know the pipe already exists from running the PipeList utility:
PipeList v1.02 - Lists open named pipes
Copyright (C) 2005-2016 Mark Russinovich
Sysinternals - www.sysinternals.com
Pipe Name Instances Max Instances
--------- --------- -------------
[...]
chrome.blah.foo 1 1
SendPipe1 1 -1
ReceivePipe1 1 -1
chrome.blah.bar 1 1
[...]
The Code
Here is what I have right now:
ACTIVE_DISP_ID = 1
msg = f'6;16;100;{ACTIVE_DISP_ID};1\r\n'
pipe_path = f'\\\\.\\PIPE\\SendPipe{ACTIVE_DISP_ID}'
try:
f = open(pipe_path, 'wb+', buffering=0)
f.write(msg.encode(encoding='ascii'))
except OSError as e:
raise
which gives me
The Error(s)
Traceback (most recent call last):
File "C:/Users/Spooqi/PycharmProjects/Spooqi/radiant/pipe-test.py", line 7, in <module>
f = open(pipe_path, 'wb+', buffering=0)
OSError: [Errno 22] Invalid argument: '\\\\.\\PIPE\\SendPipe1'
I am fairly certain the path '\\\\.\\PIPE\\SendPipe1' is correct, because if I change it to something like '\\\\.\\PIPE\\FooPipeBar', I get a different Exception:
FileNotFoundError: [Errno 2] No such file or directory: '\\\\.\\PIPE\\FooPipeBar'
and if I write to a regular file that doesn't (yet) exist with relative path foobar it will just produce a text file with that message normally.
The Question(s)
In this context, what does errno 22 mean? Why is this argument invalid? Why can't I connect to this pipe and write to it?
Is there some other process already connected to the pipe? How would I check that? If there is, how can I also connect to the device with a pipe at the same time?
I don't have a source for this, but errno 22 when opening a pipe on Windows from Python seems to indicates the pipe is already in use by another process.
For instance:
Hyper-V opens \\.\pipe\pipe1 for writing as a ComPort for VM-A
Hyper-V can't also open \\.\pipe\pipe1 for writing as a ComPort for VM-B
or:
Hyper-V opens a ComPort \\.\pipe\pipe1 for writing
Program-A opens ComPort \\.\pipe\pipe1 for reading
Program-B can't also open \\.\pipe\pipe1 for reading
There are probably additional caveats based on the type of pipe, but this has been my observations for Error 22 when using single reader/writer named pipes with Hyper-V
When using Powershell, you see this error
PS C:\> $pipe = new-object System.IO.Pipes.NamedPipeClientStream \\.\pipe\pipe1
PS C:\> $pipe.Connect(1)
Exception calling "Connect" with "1" argument(s): "The semaphore timeout period has expired.
"
At line:1 char:1
+ $pipe.Connect(1)
+ ~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [], MethodInvocationException
+ FullyQualifiedErrorId : IOException

SysLogHandler messages grouped on one line on remote server

I am trying to use python logging module to log messages to a remote rsyslog server. The messages are received, but its concatenating the messages together on one line for each message. Here is an example of my code:
to_syslog_priority: dict = {
Level.EMERGENCY: 'emerg',
Level.ALERT: 'alert',
Level.CRITICAL: 'crit',
Level.ERROR: 'err',
Level.NOTICE: 'notice',
Level.WARNING: 'warning',
Level.INFO: 'info',
Level.DEBUG: 'debug',
Level.PERF: 'info',
Level.AUDIT: 'info'
}
#staticmethod
def make_logger(*, name: str, log_level: Level, rsyslog_address: Tuple[str, int], syslog_facility: int) -> Logger:
"""Initialize the logger with the given attributes"""
logger = logging.getLogger(name)
num_handlers = len(logger.handlers)
for i in range(0, num_handlers):
logger.removeHandler(logger.handlers[0])
logger.setLevel(log_level.value)
syslog_priority = Log.to_syslog_priority[log_level]
with Timeout(seconds=RSYSLOG_TIMEOUT, timeout_message="Cannot reach {}".format(rsyslog_address)):
sys_log_handler = handlers.SysLogHandler(rsyslog_address, syslog_facility, socket.SOCK_STREAM)
# There is a bug in the python implementation that prevents custom log levels
# See /usr/lib/python3.6/logging/handlers.SysLogHandler.priority_map on line 789. It can only map
# to 5 log levels instead of the 8 we've implemented.
sys_log_handler.mapPriority = lambda *args: syslog_priority
logger.addHandler(sys_log_handler)
stdout_handler = logging.StreamHandler(sys.stdout)
logger.addHandler(stdout_handler)
return logger
if __name__ == '__main__':
app_logger = Log.make_logger(name='APP', log_level=Log.Level.INFO, rsyslog_address=('localhost', 514),
syslog_facility=SysLogHandler.LOG_USER)
audit_logger = Log.make_logger(name='PERF', log_level=Log.Level.INFO, rsyslog_address=('localhost', 514),
syslog_facility=SysLogHandler.LOG_LOCAL0)
perf_logger = Log.make_logger(name='AUDIT', log_level=Log.Level.INFO, rsyslog_address=('localhost', 514),
syslog_facility=SysLogHandler.LOG_LOCAL1)
log = Log(log_level=Log.Level.WARNING, component='testing', worker='tester', version='1.0', rsyslog_srv='localhost',
rsyslog_port=30514)
app_logger.warning("Testing warning logging")
perf_logger.info("Testing performance logging1")
audit_logger.info("Testing aduit logging1")
audit_logger.info("Testing audit logging2")
app_logger.critical("Testing critical logging")
perf_logger.info("Testing performance logging2")
audit_logger.info("Testing audit logging3")
app_logger.error("Testing error logging")
On the server side, I'm added the following the following line to the /etc/rsyslog.d/50-default.conf to disable /var/log/syslog logging for USER, LOCAL0 and LOCAL1 facilities (which I use for app, perf, and audit logging).
*.*;user,local0,local1,auth,authpriv.none -/var/log/syslog
And I update the to the /etc/rsyslog.config:
# /etc/rsyslog.conf Configuration file for rsyslog.
#
# For more information see
# /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html
#
# Default logging rules can be found in /etc/rsyslog.d/50-default.conf
#################
#### MODULES ####
#################
module(load="imuxsock") # provides support for local system logging
#module(load="immark") # provides --MARK-- message capability
# provides UDP syslog reception
#module(load="imudp")
#input(type="imudp" port="514")
# provides TCP syslog reception
module(load="imtcp")
input(type="imtcp" port="514")
# provides kernel logging support and enable non-kernel klog messages
module(load="imklog" permitnonkernelfacility="on")
###########################
#### GLOBAL DIRECTIVES ####
###########################
#
# Use traditional timestamp format.
# To enable high precision timestamps, comment out the following line.
#
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
# Filter duplicated messages
$RepeatedMsgReduction on
#
# Set the default permissions for all log files.
#
$FileOwner syslog
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022
$PrivDropToUser syslog
$PrivDropToGroup syslog
#
# Where to place spool and state files
#
$WorkDirectory /var/spool/rsyslog
#
# Include all config files in /etc/rsyslog.d/
#
$IncludeConfig /etc/rsyslog.d/*.conf
user.* -/log/app.log
local0.* -/log/audit.log
local1.* -/log/perf.log
So after doing all that when I run the python code (listed above) these are these are the messages I'm seeing on the remote server:
for log in /log/*.log; do echo "${log} >>>"; cat ${log}; echo "<<< ${log}"; echo; done
/log/app.log >>>
Oct 23 13:00:23 de4bba6ac1dd rsyslogd: imklog: cannot open kernel log (/proc/kmsg): Operation not permitted.
Oct 23 13:01:34 Testing warning logging#000<14>Testing critical logging#000<14>Testing error logging
<<< /log/app.log
/log/audit.log >>>
Oct 23 13:01:34 Testing aduit logging1#000<134>Testing audit logging2#000<134>Testing audit logging3
<<< /log/audit.log
/log/perf.log >>>
Oct 23 13:01:34 Testing performance logging1#000<142>Testing performance logging2
<<< /log/perf.log
As you can see the messages are being filtered to the proper log file, but they're being concatenated onto one line. I'm guessing that its doing it because they arrive at the same time, but I'd like the messages to be split onto separate lines.
In addition, I've tried adding a formatter to the SysLogHandler so that it inserts a line break to the message like this:
sys_log_handler.setFormatter(logging.Formatter('%(message)s\n'))
However, this really screws it up:
for log in /log/*.log; do echo "${log} >>>"; cat ${log}; echo "<<< ${log}"; echo; done
/log/app.log >>>
Oct 23 13:00:23 de4bba6ac1dd rsyslogd: imklog: cannot open kernel log (/proc/kmsg): Operation not permitted.
Oct 23 13:01:34 Testing warning logging#000<14>Testing critical logging#000<14>Testing error logging
Oct 23 13:12:00 Testing warning logging
Oct 23 13:12:00 172.17.0.1 #000<134>Testing audit logging2
Oct 23 13:12:00 172.17.0.1 #000<14>Testing critical logging
Oct 23 13:12:00 172.17.0.1 #000<142>Testing performance logging2
Oct 23 13:12:00 172.17.0.1 #000<134>Testing audit logging3
Oct 23 13:12:00 172.17.0.1 #000<14>Testing error logging
Oct 23 13:12:00 172.17.0.1
<<< /log/app.log
/log/audit.log >>>
Oct 23 13:01:34 Testing aduit logging1#000<134>Testing audit logging2#000<134>Testing audit logging3
Oct 23 13:12:00 Testing aduit logging1
<<< /log/audit.log
/log/perf.log >>>
Oct 23 13:01:34 Testing performance logging1#000<142>Testing performance logging2
Oct 23 13:12:00 Testing performance logging1
<<< /log/perf.log
As you can see the first message gets put into the right file for the audit and performance logs, but then all the other messages get put into the application log file. However, there is definitely a line break now.
My question is, I want to filter the messages based on facility, but with each message on a seperate line. How can I do this using the python logging library? I guess I could take a look at the syslog library?
So I came across this python bug:
https://bugs.python.org/issue28404
So I took a look at the source code (nice thing about python), specifically the SysLogHander.emit() method:
def emit(self, record):
"""
Emit a record.
The record is formatted, and then sent to the syslog server. If
exception information is present, it is NOT sent to the server.
"""
try:
msg = self.format(record)
if self.ident:
msg = self.ident + msg
if self.append_nul:
# Next line is always added by default
msg += '\000'
# We need to convert record level to lowercase, maybe this will
# change in the future.
prio = '<%d>' % self.encodePriority(self.facility,
self.mapPriority(record.levelname))
prio = prio.encode('utf-8')
# Message is a string. Convert to bytes as required by RFC 5424
msg = msg.encode('utf-8')
msg = prio + msg
if self.unixsocket:
try:
self.socket.send(msg)
except OSError:
self.socket.close()
self._connect_unixsocket(self.address)
self.socket.send(msg)
elif self.socktype == socket.SOCK_DGRAM:
self.socket.sendto(msg, self.address)
else:
self.socket.sendall(msg)
except Exception:
self.handleError(record)
As you can see it adds a '\000' to the end of the message by default, so if I set this to False and then set a Formatter that adds a line break, then things work the way I expect. Like this:
sys_log_handler.mapPriority = lambda *args: syslog_priority
# This will add a line break to the message before it is 'emitted' which ensures that the messages are
# split up over multiple lines, see https://bugs.python.org/issue28404
sys_log_handler.setFormatter(logging.Formatter('%(message)s\n'))
# In order for the above to work, then we need to ensure that the null terminator is not included
sys_log_handler.append_nul = False
Thanks for your help #Sraw, I tried to use UDP, but never got the message. After applying these changes this is what I see in my log files:
$ for log in /tmp/logging_test/*.log; do echo "${log} >>>"; cat ${log}; echo "<<< ${log}"; echo; done
/tmp/logging_test/app.log >>>
Oct 23 21:06:40 083c9501574d rsyslogd: imklog: cannot open kernel log (/proc/kmsg): Operation not permitted.
Oct 23 21:06:45 Testing warning logging
Oct 23 21:06:45 Testing critical logging
Oct 23 21:06:45 Testing error logging
<<< /tmp/logging_test/app.log
/tmp/logging_test/audit.log >>>
Oct 23 21:06:45 Testing audit logging1
Oct 23 21:06:45 Testing audit logging2
Oct 23 21:06:45 Testing audit logging3
<<< /tmp/logging_test/audit.log
/tmp/logging_test/perf.log >>>
Oct 23 21:06:45 Testing performance logging1
Oct 23 21:06:45 Testing performance logging2
<<< /tmp/logging_test/perf.log
I believe tcp stream makes it more complicated. When you are using tcp stream, rsyslog won't help you to split the message, it is all on your own.
Why not use udp protocol? In this case, every single message will be treated as a single log. So you don't need to add \n manually. And manually adding \n will makes you impossible to log multiple line logs correctly.
So my suggestion is changing to udp protocol and:
# Disable escaping to accept multiple line log
$EscapeControlCharactersOnReceive off
I came accross the same issue.
capture multiline events with rsyslog and storing them to file
The cause is related to the question above.
NULL character is not handled as delimilter in imtcp.
My solution was setting AddtlFrameDelimiter="0", something like below.
module(load="imtcp" AddtlFrameDelimiter="0")
input(type="imtcp" port="514")
reference:
AddtlFrameDelimiter
Syslog Message Format
related rsyslog code
EDITED 2022/10/04
The previous example was using input parameter.
And AddtlFrameDelimiter's input paramter has been newly supported since v8.2106.0.
Therefore, I changed the input paramter's example into old style module golbal parameter's one.
# module golbal parameter case, supported since v6.XXXX
module(load="imtcp" AddtlFrameDelimiter="0")
input(type="imtcp" port="514")
# input paramter case, supported since v8.2106.0
module(load="imtcp")
input(type="imtcp" port="514" AddtlFrameDelimiter="0")

accessing files via python using a service account

I am playing with windows server 2012 r2.
I have some files on the server. I have a separate service account which has the read access to the files. What i want to do is using python access the files by network share(any other suggestions welcomed) but only through the service account.
PS: i cannot use RDP.
The underlying WINAPIs for this task are part of [MS.Docs]: WNetAddConnection2W function family.
The [GitHub]: mhammond/pywin32 - Python for Windows (pywin32) Extensions wrapper is [ActiveState]: Module win32wnet (it's not the official doc (I couldn't find any at this point) - I don't know for how long the URL will be valid, but it's the best I could find).
I've prepared a trivial example.
code00.py:
#!/usr/bin/env python3
import sys
import os
import pywintypes
import win32wnet
CONNECT_INTERACTIVE = 0x00000008
HOST_NAME = "192.168.1.3"
SHARE_NAME = "Work"
SHARE_FULL_NAME = os.path.sep * 2 + os.path.sep.join((HOST_NAME, SHARE_NAME))
SHARE_USER = "cfati"
SHARE_PWD = "********"
def main():
net_resource = win32wnet.NETRESOURCE()
net_resource.lpRemoteName = SHARE_FULL_NAME
flags = 0
#flags |= CONNECT_INTERACTIVE
print("Trying to create connection to: {:s}".format(SHARE_FULL_NAME))
try:
win32wnet.WNetAddConnection2(net_resource, SHARE_PWD, SHARE_USER, flags)
except pywintypes.error as e:
print(e)
else:
print("Success!")
if __name__ == "__main__":
print("Python {:s} on {:s}\n".format(sys.version, sys.platform))
main()
Notes:
The password is obfuscated (obviously)
This is the simplest functionality (the equivalent of your command), however the function can do much more:
One thing that I want to point out. If you:
Input some invalid credentials, and
Decomment the flags |= CONNECT_INTERACTIVE line
A credentials dialog box will then pop up
Output:
(py35x64_test) e:\Work\Dev\StackOverflow\q050602112>net use
New connections will be remembered.
There are no entries in the list.
(py35x64_test) e:\Work\Dev\StackOverflow\q050602112>"e:\Work\Dev\VEnvs\py35x64_test\Scripts\python.exe" code00.py
Python 3.5.4 (v3.5.4:3f56838, Aug 8 2017, 02:17:05) [MSC v.1900 64 bit (AMD64)] on win32
Trying to create connection to: \\192.168.1.3\Work
Success!
(py35x64_test) e:\Work\Dev\StackOverflow\q050602112>net use
New connections will be remembered.
Status Local Remote Network
-------------------------------------------------------------------------------
OK \\192.168.1.3\Work Microsoft Windows Network
The command completed successfully.
(py35x64_test) e:\Work\Dev\StackOverflow\q050602112>net use * /delete /y
You have these remote connections:
\\192.168.1.3\Work
Continuing will cancel the connections.
The command completed successfully.
(py35x64_test) e:\Work\Dev\StackOverflow\q050602112>net use
New connections will be remembered.
There are no entries in the list.

Raspberry Pi (Raspbian Linux flavor) run script on wifi up

Currently I am experience issues with the script automatic run after wifi adapter connects to a network.
After ridiculously extended research, I've made several attempts to add script to a /etc/network/if-up.d/. Manually my script works; however it does not automatically.
User permissions:
ls -al /etc/network/if-up.d/*
-rwxr-xr-x 1 root root 703 Jul 25 2011 /etc/network/if-up.d/000resolvconf
-rwxr-xr-x 1 root root 484 Apr 13 2015 /etc/network/if-up.d/avahi-daemon
-rwxr-xr-x 1 root root 4958 Apr 6 2015 /etc/network/if-up.d/mountnfs
-rwxr-xr-x 1 root root 945 Apr 14 2016 /etc/network/if-up.d/openssh-server
-rwxr-xr-x 1 root root 48 Apr 26 03:21 /etc/network/if-up.d/sendemail
-rwxr-xr-x 1 root root 1483 Jan 6 2013 /etc/network/if-up.d/upstart
lrwxrwxrwx 1 root root 32 Sep 17 2016 /etc/network/if-up.d/wpasupplicant -> ../../wpa_supplicant/ifupdown.sh
Also, I've tried to push the command directly in /etc/network/interfaces
by adding a row
post-up /home/pi/r/sendemail.sh
Contents of sendemail.sh:
#!/bin/sh
python /home/pi/r/pip.py
After the reboot, nothing actually happen. I've even tried sudo in front
I assume that wpasupplicant is the thing which causes that, but I cannot get how to run my script in ifupdown.sh script under /etc/wpa_supplicant.
Appreciate your help!
If you have no connectivity prior to initializing the wifi interface, I would suggest adding a cron job of a bash or python script that checks for connectivity every X minutes.
Ping (host);
If host is up then run python commands or external command.
This is rather ambiguous but hopefully is of some help.
Here is an example of a script that will check if a host is alive;
import re,commands
class CheckAlive:
def __init__(self):
myCommand = commands.getstatusoutput('ping ' + 'google.com)
searchString = r'ping: unknown host'
match = re.search(searchString,str(myCommand))
if match:
# host is not alive
print 'no alive, don't do stuff';
else:
# host is alive
print 'alive, time do stuff';

Python script not executed from Cron

I am trying to run a Python script from Cron but it does not work. I have already tried everything I have seen in multiple Stackoverflow questions.The machine is a Raspberry running Raspbian. The following piece of code is the edition of crontab:
PATH=/usr/sbin:/usr/bin:/sbin/bin:/sbin:/bin:/home/pi/miniconda/bin:/usr/local/bin:/usr/local/sbin
*/5 * * * * rsync -az --timeout=10 --progress pritms#bigdata.trainhealthmanagement.com:Upload/*.csv /home/pi/PAD-S100/PAD-S100-Bloque_Motor/from_repo/ | /bin/sh /home/pi/PAD-S100/PAD-S100-Bloque_Motor/adddate_to_logs.sh >> /home/pi/PAD-S100/PAD-S100-Bloque_Motor/log.log 2>&1
*/5 * * * * /bin/sh /home/pi/PAD-S100/PAD-S100-Bloque_Motor/from_repo/launcher.sh | /bin/sh /home/pi/PAD-S100/PAD-S100-Bloque_Motor/adddate_to_logs.sh >> /home/pi/PAD-S100/PAD-S100-Bloque_Motor/log.log 2>&1
*/30 * * * * rm /home/pi/PAD-S100/PAD-S100-Bloque_Motor/log.log
* * * * * /usr/bin/python /home/pi/PAD-S100/PAD-S100-Bloque_Motor/from_repo/event_management.py | /bin/sh /home/pi/PAD-S100/PAD-S100-Bloque_Motor/adddate_to_logs.sh >> home/pi/PAD-S100/PAD-S100-Bloque_Motor/log.log 2>&1
0 0 * * * rm /home/pi/PAD-S100/PAD-S100-Bloque_Motor/from_repo/*.csv | /bin/sh /home/pi/PAD-S100/PAD-S100-Bloque_Motor/adddate_to_logs.sh >> /home/pi/PAD-S100/PAD-S100-Bloque_Motor/log.log 2>&1
Crontab Observations:
The Path obtained from echo $PATH is included.
launcher.sh, addddate_to_logs.sh and
event_management are executables using the command sudo chmod a+x <file_name>.
The log.log file does not show anything strange.
The system log file /var/log/syslog has the following logs:
Feb 27 15:11:08 raspberrypi cron[21814]: sendmail: Cannot open :25
Feb 27 15:12:01 raspberrypi rsyslogd-2007: action 'action 17' suspended, next retry is Mon Feb 27 15:13:31 2017 [try http://www.rsyslog.com/e/2007 ]
Feb 27 15:12:01 raspberrypi CRON[22209]: (pi) CMD (/usr/bin/python /home/pi/PAD-S100/PAD-S100-Bloque_Motor/from_repo/event_management.py | /bin/sh /home/pi/PAD-S100/PAD-S100-Bloque_Motor/adddate_to_logs.sh >> home/pi/PAD-S100/PAD-S100-Bloque_Motor/log.log 2>&1)
Feb 27 15:12:09 raspberrypi sSMTP[22212]: Unable to set UsesSTARTTILS=""
Feb 27 15:12:09 raspberrypi sSMTP[22212]: Unable to locate
Feb 27 15:12:09 raspberrypi cron[21814]: sendmail: Cannot open :25
Feb 27 15:12:09 raspberrypi sSMTP[22212]: Cannot open :25
Feb 27 15:12:09 raspberrypi CRON[22205]: (pi) MAIL (mailed 178 bytes of output but got status 0x0001 from MTA#012)
We can observe that it is probable that the failing crontab line is the one of the python script. As I am not an expert in Linux I believe it may be something related to the sSMTP. The same kind of error log appears after every call of the cron python script. But I have no idea of how to fix it or configure the local email.
Here is the piece of code of event_management.py file :
#!/usr/bin/python
# -*- coding: utf-8 -*-
import imaplib
import email
import csv
import datetime
EMAIL = <email_user>
FROM_PWD = <password>
SMTP_SERVER = 'mail.o365.alstom.com'
datum = dict()
translate = {'#09': 1, '#0A': 2, '#0B': 3, '#0C': 4}
def connect_imap():
mail = imaplib.IMAP4_SSL(SMTP_SERVER)
mail.login(EMAIL, FROM_PWD)
return mail
def read_email_from_gmail(writer, mail):
mail.select('BRMS')
kind, data = mail.search(None, 'ALL')
mail_ids = data[0]
id_list = mail_ids.split()
first_email_id = int(id_list[0])
latest_email_id = int(id_list[-1])
for i in range(latest_email_id, first_email_id, -1):
typ, data = mail.fetch(i, '(RFC822)')
for response_part in data:
if isinstance(response_part, tuple):
msg = email.message_from_string(response_part[1])
for part in msg.walk():
if part.get_content_type() == 'text/html':
content = part.get_payload()
manage_email_content(content, writer)
return 0
def manage_email_content(content, writer):
content = content.split('\n')
for i, line in enumerate(content):
if 'Alert description' in line:
line = line.split()
datum['Event code'] = line[-1][4:]
if line[-1][:3] in translate:
datum['Motor block num'] = translate[line[-1][:3]]
else:
datum['Motor block num'] = 'Defecto ajeno al bloque motor'
elif 'Alert condition' in line:
line = line.split()
datum['Code description'] = ' '.join(line[4:])
elif 'Unit id' in line:
line = line.split()
datum['Train num'] = line[3][3:]
elif 'Alert raised' in line:
line = line.split()
datum['Date'] = line[4][:10]
datum['Time'] = line[4][11:]
writer.writerow(datum)
print datum
return 0
def move_to_trash_before_date(mail, folder, days_before):
# required to perform search, m.list() for all lables, '[Gmail]/Sent Mail'
no_of_msgs = int(mail.select(folder)[1][0])
print("- Found a total of {1} messages in '{0}'.".format(folder, no_of_msgs))
before_date = (datetime.date.today() - datetime.timedelta(days_before)).strftime("%d-%b-%Y")
typ, data = mail.search(None, '(BEFORE {0})'.format(before_date)) # search pointer for msgs before before_date
if data != ['']: # if not empty list means messages exist
no_msgs_del = data[0].split()[-1] # last msg id in the list
print("- Marked {0} messages for removal with dates before {1} in '{2}'.".format(no_msgs_del, before_date, folder))
mail.store("1:{0}".format(no_msgs_del), '+X-GM-LABELS', '\\Trash') # move to trash
empty_folder(mail, 'Elementos eliminados', do_expunge=True) # can send do_expunge=False, default True
else:
print("- Nothing to remove.")
return 0
def empty_folder(mail, folder, do_expunge=True):
mail.select(folder) # select all trash
mail.store("1:*", '+FLAGS', '\\Deleted') # Flag all Trash as Deleted
if do_expunge: # See Gmail Settings -> Forwarding and POP/IMAP -> Auto-Expunge
mail.expunge() # not need if auto-expunge enabled
else:
print("Expunge was skipped.")
return 0
def disconnect_imap(mail):
mail.close()
mail.logout()
return 0
def main():
with open('email_data.csv', 'w') as f:
writer = csv.DictWriter(f, fieldnames=['Time', 'Date', 'Train num', 'Motor block num',
'Event code', 'Code description'], delimiter=';')
try:
m = connect_imap()
writer.writeheader()
read_email_from_gmail(writer, m)
move_to_trash_before_date(m, 'BRMS', 15) # inbox cleanup, before 15 days
disconnect_imap(m)
except Exception, e:
print str(e)
if __name__ == "__main__":
main()
event_management file connects to an Outlook email folder, reads the emails and builds a CSV file with data extracted from the the emails' contents. This file works properly, it is already tested; and it works fine when executed manually (not using Cron). So I not sure it is related to the sSMTP issue appearing in the system log.
I will apreciate every kind of help or suggestions!
After some test and reading other users' answers, I have found the problem. It is a combination of two different issues that are not directly related, but together made this problem a pain in the ass to debug.
First Problem:
log.log file contains logs and errors from three different executables, hence I did not notice that evet_management file did not have the correct permissions. I did not apply chmod command well, and I have not notice it as it contained a lot of data.
Conclussion 1: One cronjob, one log file.
Conclussion 2: /var/log/syslog contains a lot of data, from various resources, hence it may confuse you when trying to debug. Better to produce log files apart.
Second Problem:
I have two Python distributions installed in my machine. When I execute manually the script, one is used. When Cron executes the script the other one is used. Furthermore I noticed it when first problem was fixed. I got an error of module not found when running the Python script by Cron in the log file, but perfectly worked when manually executed. Hence I have seen that when using pip install <module-name>, it is just for one distro. To check wich version of Python I was using:
which Python
Conclussion: Be smart, don't be like me, don't mess with multiple Python distributions.
Bonus: Always use full paths to be clear. Cron has different env than yours.

Categories