I am downloading dynamic data from api server and writing it to file by means of an endless loop in python. For whatever reason the program stops writing to file after couple thousand lines, while the program seems to be running. I am not sure where the problem is. The program does not give error, ie. it is not that the API server is refusing response. When restarted it continues as planned. I am using python 3.6 and win10.
Simplified version of the code looks something like:
import requests, json, time
while True:
try:
r = requests.get('https://someapiserver.com/data/')
line = r.json()
with open('file.txt', 'a') as f:
f.write(line)
time.sleep(5)
except:
print('error')
time.sleep(10)
Try opening the file first and keeping the lock on it like so:
import requests, json, time
f = open('file.txt', 'a')
while True:
try:
r = requests.get('https://someapiserver.com/data/')
line = r.json()
f.write(line)
f.flush()
time.sleep(5)
except:
print('error')
time.sleep(10)
f.close() # remember to close the file
The solution is ugly but it will do.
Related
What I need to do is to write some messages on a .txt file, close it and send it to a server. This happens in a infinite loop, so the code should look more or less like this:
from requests_toolbelt.multipart.encoder import MultipartEncoder
num = 0
while True:
num += 1
filename = f"example{num}.txt"
with open(filename, "w") as f:
f.write("Hello")
f.close()
mp_encoder = MultipartEncoder(
fields={
'file': ("file", open(filename, 'rb'), 'text/plain')
}
)
r = requests.post("my_url/save_file", data=mp_encoder, headers=my_headers)
time.sleep(10)
The post works if the file is created manually inside my working directory, but if I try to create it and write on it through code, I receive this response message:
500 - Internal Server Error
System.IO.IOException: Unexpected end of Stream, the content may have already been read by another component.
I don't see the file appearing in the project window of PyCharm...I even used time.sleep(10) because at first, I thought it could be a time-related problem, but I didn't solve the problem. In fact, the file appears in my working directory only when I stop the code, so it seems the file is held by the program even after I explicitly called f.close(): I know the with function should take care of closing files, but it didn't look like that so I tried to add a close() to understand if that was the problem (spoiler: it was not)
I solved the problem by using another file
with open(filename, "r") as firstfile, open("new.txt", "a+") as secondfile:
secondfile.write(firstfile.read())
with open(filename, 'w'):
pass
r = requests.post("my_url/save_file", data=mp_encoder, headers=my_headers)
if r.status_code == requests.codes.ok:
os.remove("new.txt")
else:
print("File not saved")
I make a copy of the file, empty the original file to save space and send the copy to the server (and then delete the copy). Looks like the problem was that the original file was held open by the Python logging module
Firstly, can you change open(f, 'rb') to open("example.txt", 'rb'). In open, you should be passing file name not a closed file pointer.
Also, you can use os.path.abspath to show the location to know where file is written.
import os
os.path.abspath('.')
Third point, when you are using with context manager to open a file, you don't close the file. The context manger supposed to do it.
with open("example.txt", "w") as f:
f.write("Hello")
I'm trying to send a file over a socket. Everything seems to be working properly except for the file not writing properly. I snipped the code down to the major issue but can send the full server and client code if necessary.
if inst == "send":
try:
print ("Receiving...")
l = s.recv(1024)
with open('torecv.py', 'wb') as f:
print ("Writing...")
newFile = l.decode("UTF-8")
f.write(newFile)
f.close()
print ("Done Receiving")
except:
pass
The output returns:
Receiving...
Writing...
and newFile saves the correct data which says to me that it is running f.write is the problem because "torecv.py" is empty.
I'm pretty new to python so go easy on me. Thanks!
I am browsing URL using txt file follow.txt and doing click on specific button in website.But the problem is that sometime I am getting error of unable to locate element and unable to click button.
I want that if that error come, it should read second line of txt file and ignore the error.I also tried the code to overcome the problem.But it is still not working properly.I think my code have some problem.How i can solve this problem.Here is my code that i used for error handling.
try:
f = open('follow.txt', 'r', encoding='UTF-8', errors='ignore')
line = f.readline()
while line:
line = f.readline()
browser.get(line)
browser.find_element_by_xpath(""".//*[#id='react-root']/section/main/article/header/div[2]/div[1]/span/button""").click()
time.sleep(50)
f.close();
except Exception as e:
f = open('follow.txt', 'r', encoding='UTF-8', errors='ignore')
line = f.readline()
while line:
line = f.readline()
browser.get(line)
browser.find_element_by_xpath(""".//*[#id='react-root']/section/main/article/header/div[2]/div[1]/span/button""").click()
time.sleep(20)
browser.find_element_by_tag_name("body").send_keys(Keys.ALT + Keys.NUMPAD2)
browser.switch_to_window(main_window)
time.sleep(10)
f.close();
In the way you have written the answer to a question like... "What happens when there is an error even in the second line?" would be scary. You definitely do NOT want to write as many nested try except blocks as the number of lines in the file.
So, you will need to have the try except on the statement where you would expect an error, which will allow you to use the opened file object without the necessity to reopen the file. Something similar to the following:
f = open('follow.txt', 'r', encoding='UTF-8', errors='ignore')
line = f.readline()
while line:
line = f.readline()
browser.get(line)
try:
browser.find_element_by_xpath(""".//*[#id='react-root']/section/main/article/header/div[2]/div[1]/span/button""").click()
except Exception as e:
print e # Or better log the error
time.sleep(50)
browser.find_element_by_tag_name("body").send_keys(Keys.ALT + Keys.NUMPAD2)
browser.switch_to_window(main_window)
time.sleep(10)
f.close();
This should let you to continue with the next line even though there is an error at the time of ".click()". Note that you do not want to close the file when you are not done with reading all that you want from file.
My intention of moving "try except" deep into the logic doesn't mean that you shouldn't use "try except" else where for example while opening file. The more better way is to use 'with' in which case you don't even need to worry about closing the file and handling exceptions while opening the file.
with open('follow.txt', 'r', encoding='UTF-8', errors='ignore') as f:
....
My client requests a page from a server written in python 3.
The server return an html page that is presented by client.
Therefore, I did a dummy.html page and when client asks for it, my python reads it and returns it to the client:
filename = "dummy.html"
fh = open(filename, 'rt')
line = fh.readline()
while line:
print(line)
line = fh.readline()
fh.close()
However, this code does not read the <!DOCTYPE html> that is placed in the top of my dummy.html file (and thus, things like bootstrap don't work for me...).
I also tried printing it manually print('<!DOCTYPE html>') but that also does not work.
print('<!DOCTYPE html>') <---- IT IS PRINTED TO SDOUT BUT WHEN PRINTED TO CLIENT, THE PAGE DOES NOT HAVE THIS LINE ....
filename = CURRENTPATH+"\\..\\su.html"
fh = open(filename, 'rt')
line = fh.readline()
print('hello')
print('<'+'!'+'DOCTYPE html>')
while line:
print(line)
line = fh.readline()
fh.close()
How can I fix it?
It looks like you're trying to reimplement a web server in Python. Please consider using an existing web framework, such as Django (https://www.djangoproject.com/), Flask (http://flask.pocoo.org/) or Pyramid (http://www.pylonsproject.org/), which will do most of the work for you (including built-in support for a wide variety of HTML templating libraries, and actual performance).
As for your actual answer, a bare print statement prints to stdout, as expected. You need, instead, to write to the file-like object whose contents will be sent to the client (is it a socket? a file? who knows? stop reinventing the wheel).
Ok, so I'm trying to do a sentimental analysis of twitter tweets and all my code works perfect to get a response of live tweets. However the shell deletes all the tweets after a certain amount was reached. I have been messing around with my code to try and write all the tweets to a text file but for the last 5 hours of my struggles I can not figure it out. Where the comment symbol # is code I added to try and write the information to my text file. I'm fairly new to python so if someone can help me out I would very much appreciate it.
I would use Git because I know how to write all the data to a text file in that program but I can't figure out how to get it to run my python files.
def twitterreq(url, method, parameters):
req = oauth.Request.from_consumer_and_token(oauth_consumer,
token=oauth_token,
http_method=http_method,
http_url=url,
parameters=parameters)
req.sign_request(signature_method_hmac_sha1, oauth_consumer, oauth_token)
headers = req.to_header()
if http_method == "POST":
encoded_post_data = req.to_postdata()
else:
encoded_post_data = None
url = req.to_url()
opener = urllib.OpenerDirector()
opener.add_handler(http_handler)
opener.add_handler(https_handler)
response = opener.open(url, encoded_post_data)
return response
def fetchsamples():
url = "https://stream.twitter.com/1/statuses/sample.json"
parameters = []
response = twitterreq(url, "GET", parameters)
f=open("C:\\Users\\name\\Desktop\\datasci_course_materials\\assignment1", "w") # my attempt
for line in response:
f.write(str(line) + "\n") # 100% sure im not using this command properly
print line.strip()
if __name__ == '__main__':
fetchsamples()
I have left out the top of my code because we shouldn't need my access and consumer keys to answer this question. This code is in Python 2.7
Could try something along the lines of.
try:
with open("filename.txt", "a") as f:
for n in response:
f.write(n + "\n")
f.close()
except IOError as e:
print e
except TypeError as t:
print t
This will attempt to open filename.txt and append each item in "response" to a new line. It will capture IO errors and Type errors.
The line f=open("<filename>", "w") # my attempt means that if it stops, your file will just be lost completely and erased. Every time that your program runs this line it erases the file and then opens it.
Try changing the mode "a", which means that each subsequent call will just add data to the end.
f = open("<filename>", "a") # Appending instead of overwriting.
Extra information: https://docs.python.org/2/library/functions.html#open