I have a script to read messages on a mail server and save them in specific folders based on the content of the message bodies. Intermittently, usually about once or twice a day, it fails while executing this part of the code:
if not os.path.isfile(att_path) :
# finally write the stuff
fp = open(att_path, 'wb')
fp.write(part.get_payload(decode=True))
fp.close()
ext = att_path.split(".")[-1]
print "att_path",att_path
f = open(att_path.replace("."+ext,".txt"),'wb')
f.write(headers)
f.write("\n\n\n")
f.write(body)
f.close()
filelist.append(vdir+"/"+filename)
messageReceived = True
else:
noErrors = False
errFiles.append(vdir+"/"+filename)
It saves the actual attachment in the expected directory, but not the subsequent text file with the headers and body information. Because an exception is thrown ("[Errno 9] Bad file descriptor"), the email is not marked for deletion and stays on the server until the saved attachment is either deleted or moved, at which point both files will be saved without any errors.
I'm stumped at what could be causing it, since it processes several hundred emails every day without any problems, except for this intermittent issue.
I encountered intermittent bad descriptor error in a script run with pywin32 (running python as Windows service). A near identical script (sans the pywin32 boilerplate) runs without issues in the cmd. The module traceback also points to various print statements, thus I commented out all the print statements, and it works!
Please correct me if I'm wrong, I suspect this is something to do with the lack of stdout. I used to use print statements to debug but switched to the logging module after this.
Related
import requests
res = requests.get("https://google.com")
res.raise_for_status()
print(len(res.text))
print(res.text)
with open('mygoogle.html',"w",encoding=('utf-8')) as f :
f.write(res.text)
The with open as f part doesn't work.
If it does, it has to open new file, but it doesn't.
I have tested this code and it works properly - mygoogle.html gets written and contains the expected content. The issue is likely outside of the script then. Since it is using a relative path, the issue could be that the file is getting written somewhere where you don't expect it due to the working directory not being set how you expect - Windows often defaults to C:\Windows\system32 or your user profile directory (%userprofile% aka C:\users\<username>) for new cmd instances.
I have a python script that takes a long time to run.
I placed print-outs throughout the script to observe its progress.
As this script different programs, some of whom print many messages, it is unfeasible to print directly to the screen.
Therefore, I am using a report file
f_report = open(os.path.join("//shared_directory/projects/work_area/", 'report.txt'), 'w')
To which I print my massages:
f_report.write(" "+current_image+"\n")
However, when I look at the file while the script is running, I do not see the messages. They appear only when the program finishes and closes the file, making my approach useless for monitoring on-going progress.
What should I do in order to make python output the messages to the report file in real time?
Many thanks.
You should use flush() function to write immediately to the file.
f_report.write(" "+current_image+"\n")
f_report.flush()
try this:
newbuffer = 0
f_report = open(os.path.join("//shared_directory/projects/work_area/", 'report.txt'), 'w', newbuffer)
it sets up a 0 buffer which will push OS to write content to file "immediately". well, different OS may behavior differently but in general content will be flushed out right away.
I am trying to use Pastebin to host two text files for me to allow any copy of my script to update itself through the internet. My code is working, but the resultant .py file has a blank line added between each line. Here is my script...
import os, inspect, urllib2
runningVersion = "1.00.0v"
versionUrl = "http://pastebin.com/raw.php?i=3JqJtUiX"
codeUrl = "http://pastebin.com/raw.php?i=GWqAQ0Xj"
scriptFilePath = (os.path.abspath(inspect.getfile(inspect.currentframe()))).replace("\\", "/")
def checkUpdate(silent=1):
# silently attempt to update the script file by default, post messages if silent==0
# never update if "No_Update.txt" exists in the same folder
if os.path.exists(os.path.dirname(scriptFilePath)+"/No_Update.txt"):
return
try:
versionData = urllib2.urlopen(versionUrl)
except urllib2.URLError:
if silent==0:
print "Connection failed"
return
currentVersion = versionData.read()
if runningVersion!=currentVersion:
if silent==0:
print "There has been an update.\nWould you like to download it?"
try:
codeData = urllib2.urlopen(codeUrl)
except urllib2.URLError:
if silent==0:
print "Connection failed"
return
currentCode = codeData.read()
with open(scriptFilePath.replace(".py","_UPDATED.py"), mode="w") as scriptFile:
scriptFile.write(currentCode)
if silent==0:
print "Your program has been updated.\nChanges will take effect after you restart"
elif silent==0:
print "Your program is up to date"
checkUpdate()
I stripped the GUI (wxpython) and set the script to update another file instead of the actual running one. The "No_Update" bit is for convenience while working.
I noticed that opening the resultant file with Notepad does not show the skipped lines, opening with Wordpad gives a jumbled mess, and opening with Idle shows the skipped lines. Based on that, this seems to be a formatting problem even though the "raw" Pastebin file does not appear to have any formatting.
EDIT: I could just strip all blank lines or leave it as is without any problems, (that I've noticed) but that would greatly reduce readability.
Try adding the binary qualifier in your open():
with open(scriptFilePath.replace(".py","_UPDATED.py"), mode="wb") as scriptFile:
I notice that your file on pastebin is in DOS format, so it has \r\n in it. When you call scriptFile.write(), it translates \r\n to \r\r\n, which is terribly confusing.
Specifying "b" in the open() will cause scriptfile to skip that translate and write the file is DOS format.
In the alternative, you could ensure that the pastebin file has only \n in it, and use mode="w" in your script.
I have a server/client socket pair in Python. The server receives specific commands, then prepares the response and send it to the client.
In this question, my concern is just about possible injections in the code: if it could be possible to ask the server doing something weird with the 2nd parameter -- if the control on the command contents is not sufficient to avoid undesired behaviour.
EDIT:
according to advices received
added parameter shell=True when calling check_output on windows. Should not be dangerous since the command is a plain 'dir'.
.
self.client, address = self.sock.accept()
...
cmd = bytes.decode(self.client.recv(4096))
ls: executes a system command but only reads the content of a directory.
if cmd == 'ls':
if self.linux:
output = subprocess.check_output(['ls', '-l'])
else:
output = subprocess.check_output('dir', shell=True)
self.client.send(output)
cd: just calls os.chdir.
elif cmd.startswith('cd '):
path = cmd.split(' ')[1].strip()
if not os.path.isdir(path):
self.client.send(b'is not path')
else:
os.chdir(path)
self.client.send( os.getcwd().encode() )
get: send the content of a file to the client.
elif cmd.startswith('get '):
file = cmd.split(' ')[1].strip()
if not os.path.isfile(file):
self.client.send(b'ERR: is not a file')
else:
try:
with open(file) as f: contents = f.read()
except IOError as er:
res = "ERR: " + er.strerror
self.client.send(res.encode())
continue
... (send the file contents)
Except in implementation details, I cannot see any possibilities of direct injection of arbitrary code because you do not use received parameters in the only commands you use (ls -l and dir).
But you may still have some security problems :
you locate commands through the path instead of using absolute locations. If somebody could change the path environment variable what could happen ... => I advice you to use directly os.listdir('.') which is portable and has less risks.
you seem to have no control on allowed files. If I correctly remember reading CON: or other special files on older Windows version gave weird results. And you should never give any access to sensible files, configuration, ...
you could have control on length of asked files to avoid users to try to break the server with abnormally long file names.
Typical issues in a client-server scenario are:
Tricking the server into running a command that is determined by the client. In the most obvious form this happens if the server allows the client to run commands (yes, stupid). However, this can also happen if the client can supply only command parameters but shell=True is used. E.g. using subprocess.check_output('dir %s' % dir, shell=True) with a client-supplied dir variable would be a security issue, dir could have a value like c:\ && deltree c:\windows (a second command has been added thanks to the flexibility of the shell's command line interpreter). A relatively rare variation of this attack is the client being able to influence environment variables like PATH to trick the server into running a different command than intended.
Using unexpected functionality of built-in programming language functions. For example, fopen() in PHP won't just open files but fetch URLs as well. This allows passing URLs to functionality expecting file names and playing all kinds of tricks with the server software. Fortunately, Python is a sane language - open() works on files and nothing else. Still, database commands for example can be problematic if the SQL query is generated dynamically using client-supplied information (SQL Injection).
Reading data outside the allowed area. Typical scenario is a server that is supposed to allow only reading files from a particular directory, yet by passing in ../../../etc/passwd as parameter you can read any file. Another typical scenario is a server that allows reading only files with a particular file extension (e.g. .png) but passing in something like passwords.txt\0harmless.png still allows reading files of other types.
Out of these issues only the last one seems present in your code. In fact, your server doesn't check at all which directories and files the client should be allowed to read - this is a potential issue, a client might be able to read confidential files.
[Errno 2] No such file or directory: 'http://api.nytimes.com/svc/search/v1/article?query=title:music&api-key=**********'
This is the error that I get from urllib.urlopen when I try to open the above address (I've starred out the API key for obvious reasons, but in the actual error message it is present.)
The code is:
print url
print type(url)
f = urllib.urlopen(url)
The first two lines I've added to diagnose the issue. url does in fact contain exactly the string returned by the error message, and it is in fact a string object (str).
If I try the same URL on my Mac and it works (this code is running on a cloud server somewhere.) If I try the same URL on that server, in the Python terminal, it works. But for some reason, when that line of code gets executed in that script, it doesn't. Any ideas?