I am simply trying to add a counter around my for loop, to count how many files are in my directory per qualification of the contains 'VCCS'... the logic is working for the iterating, as it's iterating through my directory the amount of times I have a file in... however my counter keeps reporting 1. Relevant lines are files_in_directory market with comment # here.
I am getting this warning in PyLint: Constant name "files_in_directory" doesn't conform to UPPER_CASE naming stylepylint(invalid-name) , on files_in_directory = 0
I've tried moving the set 0, above the for and try, any thoughts?
if __name__ == "__main__":
try:
currentDT = datetime.datetime.now()
files_in_directory = 0 # here
for filename in os.listdir(config.DIRECTORY_LOCATION):
if filename.__contains__('VCCS'):
old_stdout = sys.stdout
log_file = open("./logs/metrics.log","w")
sys.stdout = log_file
files_in_directory += 1 # here
PENDING_RECORDS = FindPendingRecords().get_excel_data()
# Do operations on PENDING_RECORDS
# Reads excel to map data from excel to vital
MAP_DATA = FindPendingRecords().get_mapping_data()
# Configures Driver
VITAL_ENTRY = VitalEntry()
# Start chrome and navigate to vital website
VITAL_ENTRY.instantiate_chrome()
# Begin processing Records
VITAL_ENTRY.process_records(PENDING_RECORDS, MAP_DATA)
print(f"Date: ")
print (str(currentDT))
print(f"Files in Directory #{files_in_directory}") # here
sys.stdout = old_stdout
log_file.close()
except Exception as exc:
# print(exc)
raise
Note: This is to take place of many comments in the interest of readability
Your question as it sits is not an MCVE. To make it more succinct and to identify the exact cause:
import os
if __name__ == "__main__":
# Remove try block, just let it raise an error
my_counter = 0
for file in os.listdir("some_directory"):
# you don't need to call the __contains__ method
# as the 'in' keyword will invoke that for you
if "VCCS" in file:
# increment your counter first
my_counter += 1
print(file, my_counter)
Now there is no question what is modifying my_counter, and this will print out the file you are looking at, alongside the counter.
Once you've ironed out that behavior, you can start adding in your other functions
import os
if __name__ == "__main__":
# Remove try block, just let it raise an error
my_counter = 0
for file in os.listdir("some_directory"):
if 'VCCS' in file:
my_counter += 1
print(my_counter, file)
# Add functions back in one by one
PENDING_RECORDS = FindPendingRecords().get_excel_data()
Continue this process until you identify what is causing your behavior. As it stands, I don't see anything explicit that could be overwriting that counter variable, so I suspect that either A) the code you posted does not reflect what is being run or B) you are modifying/resetting files_in_directory somewhere else in the module.
Suggested Edit:
I would recommend you add in the other code from your module to see what's going on. That way we have a clearer picture of what's happening as your code runs
So first thing first: you're sure you're passing the right directory, and that this directory contains indeed more than one file containing VCCS in their filename ?
I would also try to run this code without the Try/Except block, to see if you don't get some error after your first increment.
LMK what you get, I hope this helps.
Related
i have a irrational error,
code:
#====================================================#
#X Programming Language 2022(for Jadi) License: GPL #
#====================================================#
from os import system #importing system func. for cmd
code_location = "" #saving code location in this
code = "" #saving code in this
def main(): #main func.
code_location = input() #get code location from user
code = get_code() #cal get_code and save res to code var
print(code) #print code
end() #calling end
def get_code(): #get_code func. for getting code from file
code_file = open(code_location, 'r') #openning file with method read and saving to code_file
res = code_file.readlines() #saving the content from file to res
return res #returning res
def compiler(): #compiler func. for compiling code
pass
def end(): #when program end...
input("Press Enter to Countinue...")
if __name__ == "__main__":
main()
and this is code directory:
enter image description here
running:
enter image description here
Short answer: Your two code_location variables are not the same thing.
Variable scopes
Variables have a property called a scope, which is essentially which context they belong to. Unless specified otherwise, variables within a function exist only within that function. Consider:
a = 0
def set_a():
a = 1
set_a()
print(a)
This will print 0. This is because the a variable within the function set_a is actually a different variable to the a defined in line 1. Although they have the same name, they point to different places in memory.
Solutions
There are a few ways to do this:
Defining scope
Either, you can set the scope of a within the function to global (instead of local). What this does is now, instead of a within the function pointing to a different memory location, it points to the same memory location as a outside the variable. This is done like so:
a = 0
def set_a():
global a
a = 1
set_a()
print(a)
In this case, a will be set to 1, and "1" will be printed
Passing as an argument
Another way to do this, and may be more relevant in your circumstance, is to pass the value as a variable to the function. In your case, you are using code_location as the file path, so therefore what you want to pass code_location into the function. You would then have to define your function like this:
def get_code(code_location):
and call the function (from your main function) like this:
code = get_code(code_location)
Notes
When operating on files, it is best practice to use a with block. This handles closing your file when you are done with it, and can prevent corruption of files in the rare case that something goes wrong with your code. This can be done like this:
with open(code_location, 'r') as code_file:
res = code_file.readlines()
return res
Python global variables are read-only in local scopes (e.g. your function's scope) by default.
So in the line code_location = input() you are essentially creating a new new local variable of the same name and assigning the input to it.
In order to write to your global variable instead, you first have to declare your intention:
def main(): #main func.
global code_location # DECLARE WRITE INTENTION FOR GLOBAL
code_location = input() #get code location from user
code = get_code() #cal get_code and save res to code var
print(code) #print code
end() #calling end
You don't have to do the same thing in get_code() since you are only reading from code_location there.
PS:
As was alluded to in the comment, it's good practice to close opened files after you've finished with them:
def get_code(): #get_code func. for getting code from file
code_file = open(code_location, 'r') #openning file with method read and saving to code_file
res = code_file.readlines() #saving the content from file to res
code_file.close() # CLOSE FILE
return res #returning res
Or have it done automatically by a context manager:
def get_code(): #get_code func. for getting code from file
with open(code_location, 'r') as code_file: #openning file with method read and saving to code_file
res = code_file.readlines() #saving the content from file to res
return res #returning res
I have a data logger to record the temperature. I want to save these data and epoch time in a csv file. I tried the following code, there is no error reporting but the csv file is empty. Can anyone help me to figure out the problem?
import board
import busio
import adafruit_mcp9600
import time
i2c = busio.I2C(board.SCL,board.SDA,frequency = 100000)
mcp = adafruit_mcp9600.MCP9600(i2c, 0x60, tctype = "J")
with open ("/home/pi/Documents/test.csv", "a") as log:
while True:
temp = mcp.temperature
temptime = time.time()
log.write("{0},{1}\n".format(str(temptime),str(temp)))
time.sleep(1)
Assuming those libraries you're importing are working correctly, I think this is because the writer is not flushing the buffer, so it appears like nothing is being written.
The solution would be to flush with log.flush() after each time you write a log.
Try a simpler example:
A)
import time
def go():
i = 0
with open("/home/some/dir/test.csv", "a") as nice:
while True:
nice.write(f"hello,{i},{time.time()}\n")
i += 1
time.sleep(5)
if __name__ == "__main__":
go()
versus
B)
import time
def go():
i = 0
while True:
with open("/home/some/dir/test.csv", "a") as nice:
nice.write(f"hello,{i},{time.time()}\n")
i += 1
time.sleep(5)
if __name__ == "__main__":
go()
When I refresh the file in case A, new rows do not appear to be written. They are in case B, though.
If I modify case A) and add nice.flush() after each write, it fixes the issue.
The above two blocks are just to demonstrate what you're seeing. I'm not suggesting you do one or the other. Ultimately, I would not suggest doing anything like this, and I would instead use the logging package and configure a proper logger if you're indeed trying to create log files.
I am trying to test this demo program from lynda using Python 3. I am using Pycharm as my IDE. I already added and installed the request package, but when I run the program, it runs cleanly and shows a message "Process finished with exit code 0", but does not show any output from print statement. Where am I going wrong ?
import urllib.request # instead of urllib2 like in Python 2.7
import json
def printResults(data):
# Use the json module to load the string data into a dictionary
theJSON = json.loads(data)
# now we can access the contents of the JSON like any other Python object
if "title" in theJSON["metadata"]:
print(theJSON["metadata"]["title"])
# output the number of events, plus the magnitude and each event name
count = theJSON["metadata"]["count"];
print(str(count) + " events recorded")
# for each event, print the place where it occurred
for i in theJSON["features"]:
print(i["properties"]["place"])
# print the events that only have a magnitude greater than 4
for i in theJSON["features"]:
if i["properties"]["mag"] >= 4.0:
print("%2.1f" % i["properties"]["mag"], i["properties"]["place"])
# print only the events where at least 1 person reported feeling something
print("Events that were felt:")
for i in theJSON["features"]:
feltReports = i["properties"]["felt"]
if feltReports != None:
if feltReports > 0:
print("%2.1f" % i["properties"]["mag"], i["properties"]["place"], " reported " + str(feltReports) + " times")
# Open the URL and read the data
urlData = "http://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/2.5_day.geojson"
webUrl = urllib.request.urlopen(urlData)
print(webUrl.getcode())
if webUrl.getcode() == 200:
data = webUrl.read()
data = data.decode("utf-8") # in Python 3.x we need to explicitly decode the response to a string
# print out our customized results
printResults(data)
else:
print("Received an error from server, cannot retrieve results " + str(webUrl.getcode()))
Not sure if you left this out on purpose, but this script isn't actually executing any code beyond the imports and function definition. Assuming you didn't leave it out on purpose, you would need the following at the end of your file.
if __name__ == '__main__':
data = "" # your data
printResults(data)
The check on __name__ equaling "__main__" is just so your code is only executing when the file is explicitly run. To always run your printResults(data) function when the file is accessed (like, say, if its imported into another module) you could just call it at the bottom of your file like so:
data = "" # your data
printResults(data)
I had to restart the IDE after installing the module. I just realized and tried it now with "Run as Admin". Strangely seems to work now.But not sure if it was a temp error, since even without restart, it was able to detect the module and its methods.
Your comments re: having to restart your IDE makes me think that pycharm might not automatically detect newly installed python packages. This SO answer seems to offer a solution.
SO answer
I am new to Python. I need to create a door.lock file that contains the current date and time. Also, I need to overwrite this file every x minutes with a new file containing the current date and time. I'm using this as a pseudo lock file to allow me to test on run of the software whether or not the software crashed and how long ago it crashed. My issue is I can't seem to overwrite the file. I've only failed at creating and/or appending the file. I created the following as a test:
from datetime import datetime, timedelta
ending = False
LOCK_FILENAME = "door.lock" # The lock file
LOCK_FILE_UPDATE = True
MINS_LOCK_FILE_UPDATE = 1 # the (x) time in minutes to write to lock file
NEXT_LOCK_FILE_UPDATE = datetime.now()
lock_file = open(LOCK_FILENAME, "w")
now = datetime.now()
NOW_STRING1 = str(now.strftime("%Y-%m-%d_%a_%H:%M"))
lock_file.write(NOW_STRING1)
print "First Now String"
print NOW_STRING1
# ==============================================================================
#Main Loop:
while ending is False:
# ==============================================================================
# Check if it is time to do a LOCK FILE time update
now = datetime.now()
NOW_STRING1 = str(now.strftime("%Y-%m-%d_%a_%H:%M"))
if LOCK_FILE_UPDATE: # if LOCK_FILE_UPDATE is set to True in DM settings
if NEXT_LOCK_FILE_UPDATE <= datetime.now():
lock_file.write(NOW_STRING1)
print NOW_STRING1
NEXT_LOCK_FILE_UPDATE = datetime.now() + timedelta(minutes=MINS_LOCK_FILE_UPDATE)
Will someone pinpoint my error(s) for me? TIA
When I cat the above file, door.lock, it is empty.
You need to push buffer to file. You can do it with a close() and re-open for next write.
lock_file.close()
...
lock_file = open(LOCK_FILENAME, "a")
If you are logging events you'd be better using a logger instead of a plain text file.
Solution from #MAC will work except it will append and seems that you don't want to do that so just open again with the 'w' option or yet better, use the 'w+' option so it can be truncated (which for what I get it is what you want to do) and read.
Also, consider your changes won't get written down until you close the file (having said that, consider open/close inside your loop instead).
lock_file = open(LOCK_FILENAME, "w+")
now = datetime.now()
NOW_STRING1 = str(now.strftime("%Y-%m-%d_%a_%H:%M"))
lock_file.write(NOW_STRING1)
# your loop and so on ...
lock_file.close()
So I have two Python3.2 processes that need to communicate with each other. Most of the information that needs to be communicated are standard dictionaries. Named pipes seemed like the way to go so I made a pipe class that can be instantiated in both processes. this class implements a very basic protocol for getting information around.
My problem is that sometimes it works, sometimes it doesn't. There seems to be no pattern to this behavior except the place where the code fails.
Here are the bits of the Pipe class that matter. Shout if you want more code:
class Pipe:
"""
there are a bunch of constants set up here. I dont think it would be useful to include them. Just think like this: Pipe.WHATEVER = 'WHATEVER'
"""
def __init__(self,sPath):
"""
create the fifo. if it already exists just associate with it
"""
self.sPath = sPath
if not os.path.exists(sPath):
os.mkfifo(sPath)
self.iFH = os.open(sPath,os.O_RDWR | os.O_NONBLOCK)
self.iFHBlocking = os.open(sPath,os.O_RDWR)
def write(self,dMessage):
"""
write the dict to the fifo
if dMessage is not a dictionary then there will be an exception here. There never is
"""
self.writeln(Pipe.MESSAGE_START)
for k in dMessage:
self.writeln(Pipe.KEY)
self.writeln(k)
self.writeln(Pipe.VALUE)
self.writeln(dMessage[k])
self.writeln(Pipe.MESSAGE_END)
def writeln(self,s):
os.write(self.iFH,bytes('{0} : {1}\n'.format(Pipe.LINE_START,len(s)+1),'utf-8'))
os.write(self.iFH,bytes('{0}\n'.format(s), 'utf-8'))
os.write(self.iFH,bytes(Pipe.LINE_END+'\n','utf-8'))
def readln(self):
"""
look for LINE_START, get line size
read until LINE_END
clean up
return string
"""
iLineStartBaseLength = len(self.LINE_START)+3 #'{0} : '
try:
s = os.read(self.iFH,iLineStartBaseLength).decode('utf-8')
except:
return Pipe.READLINE_FAIL
if Pipe.LINE_START in s:
#get the length of the line
sLineLen = ''
while True:
try:
sCurrent = os.read(self.iFH,1).decode('utf-8')
except:
return Pipe.READLINE_FAIL
if sCurrent == '\n':
break
sLineLen += sCurrent
try:
iLineLen = int(sLineLen.strip(string.punctuation+string.whitespace))
except:
raise Exception('Not a valid line length: "{0}"'.format(sLineLen))
#read the line
sLine = os.read(self.iFHBlocking,iLineLen).decode('utf-8')
#read the line terminator
sTerm = os.read(self.iFH,len(Pipe.LINE_END+'\n')).decode('utf-8')
if sTerm == Pipe.LINE_END+'\n':
return sLine
return Pipe.READLINE_FAIL
else:
return Pipe.READLINE_FAIL
def read(self):
"""
read from the fifo, make a dict
"""
dRet = {}
sKey = ''
sValue = ''
sCurrent = None
def value_flush():
nonlocal dRet, sKey, sValue, sCurrent
if sKey:
dRet[sKey.strip()] = sValue.strip()
sKey = ''
sValue = ''
sCurrent = ''
if self.message_start():
while True:
sLine = self.readln()
if Pipe.MESSAGE_END in sLine:
value_flush()
return dRet
elif Pipe.KEY in sLine:
value_flush()
sCurrent = Pipe.KEY
elif Pipe.VALUE in sLine:
sCurrent = Pipe.VALUE
else:
if sCurrent == Pipe.VALUE:
sValue += sLine
elif sCurrent == Pipe.KEY:
sKey += sLine
else:
return Pipe.NO_MESSAGE
It sometimes fails here (in readln):
try:
iLineLen = int(sLineLen.strip(string.punctuation+string.whitespace))
except:
raise Exception('Not a valid line length: "{0}"'.format(sLineLen))
It doesn't fail anywhere else.
An example error is:
Not a valid line length: "KE 17"
The fact that it's intermittent says to me that it's due to some kind of race condition, I'm just struggling to figure out what it might be. Any ideas?
EDIT added stuff about calling processes
How the Pipe is used is it is instantiated in processA and ProcessB by calling the constructor with the same path. Process A will then intermittently write to the Pipe and processB will try to read from it. At no point do I ever try to get the thing acting as a two way.
Here is a more long winded explanation of the situation. I've been trying to keep the question short but I think it's about time I give up on that. Anyhoo, I have a daemon and a Pyramid process that need to play nice. There are two Pipe instances in use: One that only Pyramid writes to, and one that only the daemon writes to. The stuff Pyramid writes is really short, I have experienced no errors on this pipe. The stuff that the daemon writes is much longer, this is the pipe that's giving me grief. Both pipes are implemented in the same way. Both processes only write dictionaries to their respective Pipes (if this were not the case then there would be an exception in Pipe.write).
The basic algorithm is: Pyramid spawns the daemon, the daemon loads craze object hierarchy of doom and vast ram consumption. Pyramid sends POST requests to the daemon which then does a whole bunch of calculations and sends data to Pyramid so that a human-friendly page can be rendered. the human can then respond to what's in the hierarchy by filling in HTML forms and suchlike thus causing pyramid to send another dictionary to the daemon, and the daemon sending back a dictionary response.
So: only one pipe has exhibited any problems, the problem pipe has a lot more traffic than the other one, and it is a guarentee that only dictionaries are written to either
EDIT as response to question and comment
Before you tell me to take out the try...except stuff read on.
The fact that the exception gets raised at all is what is bothering me. iLineLengh = int(stuff) looks to me like it should always be passed a string that looks like an integer. This is the case only most of the time, not all of it. So if you feel the urge to comment about how it's probably not an integer please please don't.
To paraphrase my question: Spot the race condition and you will be my hero.
EDIT a little example:
process_1.py:
oP = Pipe(some_path)
while 1:
oP.write({'a':'foo','b':'bar','c':'erm...','d':'plop!','e':'etc'})
process_2.py:
oP = Pipe(same_path_as_before)
while 1:
print(oP.read())
After playing around with the code, I suspect the problem is coming from how you are reading the file.
Specifically, lines like this:
os.read(self.iFH, iLineStartBaseLength)
That call doesn't necessarily return iLineStartBaseLength bytes - it might consume "LI" , then return READLINE_FAIL and retry. On the second attempt, it will get the remainder of the line, and somehow end up giving the non-numeric string to the int() call
The unpredictability likely comes from how the fifo is being flushed - if it happens to flush when the complete line is written, all is fine. If it flushes when the line is half-written, weirdness.
At least in the hacked-up version of the script I ended up with, the oP.read() call in process_2.py often got a different dict to the one sent (where the KEY might bleed into the previous VALUE and other strangeness).
I might be mistaken, as I had to make a bunch of changes to get the code running on OS X, and further while experimenting. My modified code here
Not sure exactly how to fix it, but.. with the json module or similar, the protocol/parsing can be greatly simplified - newline separated JSON data is much easier to parse:
import os
import time
import json
import errno
def retry_write(*args, **kwargs):
"""Like os.write, but retries until EAGAIN stops appearing
"""
while True:
try:
return os.write(*args, **kwargs)
except OSError as e:
if e.errno == errno.EAGAIN:
time.sleep(0.5)
else:
raise
class Pipe(object):
"""FIFO based IPC based on newline-separated JSON
"""
ENCODING = 'utf-8'
def __init__(self,sPath):
self.sPath = sPath
if not os.path.exists(sPath):
os.mkfifo(sPath)
self.fd = os.open(sPath,os.O_RDWR | os.O_NONBLOCK)
self.file_blocking = open(sPath, "r", encoding=self.ENCODING)
def write(self, dmsg):
serialised = json.dumps(dmsg) + "\n"
dat = bytes(serialised.encode(self.ENCODING))
# This blocks until data can be read by other process.
# Can just use os.write and ignore EAGAIN if you want
# to drop the data
retry_write(self.fd, dat)
def read(self):
serialised = self.file_blocking.readline()
return json.loads(serialised)
Try getting rid of the try:, except: blocks and seeing what exception is actually being thrown.
So replace your sample with just:
iLineLen = int(sLineLen.strip(string.punctuation+string.whitespace))
I bet it'll now throw a ValueError, and it's because you're trying to cast "KE 17" to an int.
You'll need to strip more than string.whitespace and string.punctuation if you're going to cast the string to an int.