I'm currently generating three different xml files, and I would like to have the second and third file have the same date/time as the first file.
In the first file, I do
import datetime
time = datetime.datetime.utcnow().strftime('%y%m%d%H%M%S')
This gives me the format I would like. I've tried multiple approaches such as storing it in a different variable and importing it to the second and third files, but it seems that it'll always keep the actual current time and not the time of the first file. I don't know if there's a solution to my problem using the datetime module but if anyone has any ideas that would be wonderful.
Whenever you call that function, whether directly or through import it will always run again and give a new "now".
If the same program just uses that string for 3 times there shouldn't be a problem, but if you're running 3 different scripts you will get 3 different dates!
To avoid this, I would save the first generated string to a file:
with open('.tmpdate') as f:
f.write(time)
And read it in the next to files:
with open('.tmpdate') as f:
time = f.read()
And finally, just to clean up after yourself, you can delete that file after it was used for the 3rd time with os.remove('.tmpdate') (you need to import os before that, of course)
Related
This seems like a pretty obvious/dumb question, but there are a few specifications that make this a bit harder.
Let's say I have a program that takes 3 numbers from a user and does mathematical processes to them to get outputs. Then I open("file", "r") to write those variables to a file.
Then, let's say another program then imports them and uses them for other processes. I need to be able to import that file as Python code. To be clear: I am not saving text, I am saving python code to a file that is not a .py file.
Is there any way to save and import Python code to and from a non-.py file? And how?
EDIT: In the file I'm saving and importing, I'm also saving Python functions. I cannot simply save the variables themselves; I need the variable names, values, and python functions to be saved as normal text in a file, but when I import the file, it should be parsed as Python code.
Probably not a good idea to store computation result as code & then import it from elsewhere. You should use a proper data format to store the results - and import it as data. Use JSON or pickle etc.
However, if you do want to shoot yourself in the foot, Python gives you the tools to do that:
Let's say i have some code in a file temp.txt
number3=30
def f():
return 'method'
Then you can do this:
with open('temp.txt') as f:
code = f.read()
exec(code)
print(number3)
print(f())
Which outputs:
30
method
If i got this right, this might be done via eval function e.g. you save all code to be executed into a string and then save into a file.
When you need that executed read the file, tke the string and eval it
I must say however that using eval is a bad (very bad) practice and i would advice against it unless there is no other solution that you can find
let's assume I have some code which creates some data, which I want to put in a csv-file. Therefore I'm using "to_csv". Now, I want to run the code several times, and want to have several csv files.
for i in xrange(1000):
code
to_csv("csv[i].csv")
How can I do something like that?
This doesn't really have anything to do with pandas. You simply use string formatting to generate a different file name.
for i in xrange(1000): # should be xrange, not xrang
# code
df.to_csv('csv{}.csv'.format(i))
I am trying to make a script that requires a user to input at least 12 different values in order to function. I thought this was somewhat impractical, so I decided to make a function that would generate dict from a .csv file that was designed with two columns– variables and their respective values. The user could use a provided .csv file as a template and then fill it in with all their necessary values, save it as their own .csv file, and then run it with the script.
Although this sounds simple in theory, I have found that is not working quite so well in practice. Because some of the inputs values will be text with a lot of periods in them ("..."), they are sometimes converted into the unicode representing horizontal ellipses (xe2\x80\xa6). Also, a UTF-8 mark will occur at the beginning of the first column and row (which can be designated by the code codecs.BOM_UTF8), and must be removed. In other cases, the delimiter of the .csv file was changed so that tabs were recognized as separating cells, or the contents of each row were converted from two to one cell.
I have no experience with the different forms of encoding or what any of them entail, but from what I have tested, it seems that opening the .csv template file in Excel or using different settings when opening your .csv file causes such problems. It's also possible that copying and pasting the values from other places brings hidden characters with them. I have been trying to fix the problems but then new problems keep springing up, and I feel like it's possible that my current approach is just wrong.
Can anybody recommend me a different, more efficient approach for allowing a user to enter in multiple inputs in one go? Or should I stick to my original approach and figure out how to keep the .csv formatting as rigorous as possible?
You can always use the csv module to abstract away most of the CSV oddities (although you will have to enforce the basic format):
import csv
import sys
def main(argv):
if len(argv) < 2:
print("Please provide path to your CSV template as the first argument.")
return 1
with open(argv[1], "r") as f:
reader = csv.DictReader(f)
your_vars = next(reader)
print(your_vars) # prints a dictionary of all CSV vars
return 0
if __name__ == "main":
sys.exit(main(sys.argv))
NOTE: This requires the first row to hold the variables, while the second holds their values.
So all users have to do is call the script with: python your_script.py their_file.csv and in most cases it will print out a dict with the values... However, Excel is notoriously bad in handling unicode CSVs and if your users use it as their primary spreadsheet app they're likely to encounter issues. Some of that can be rectified by installing the unicodecsv module and using it as a drop-in replacement (import unicodecsv as csv) but if your users start going wild with the format eventually it will break.
If you're looking for suggestions on formats, one of the most user-friendly formats you can use is YAML and there are several parsers available for Python - they largely work the same for the simple stuff like this but I'd recommend using the ruamel.yaml module as it's actively maintained.
Then you can create a YAML template like:
----
var1: value1
var2: value2
var3: value3
etc: add as many as you want
And your users can fill in the values in a simple text editor, then to replicate the above CSV behavior all you need is:
import yaml
import sys
def main(argv):
if len(argv) < 2:
print("Please provide path to your YAML template as the first argument.")
return 1
with open(argv[1], "r") as f:
your_vars = yaml.load(f)
print(your_vars) # prints a dictionary of all YAML vars
return 0
if __name__ == "main":
sys.exit(main(sys.argv))
Bonus is that YAML is plain-text format so your users don't need fancy editors and therefore they have a lesser chance to screw up. Of course, while YAML is permissive it still requires modicum of well-formedness so be sure to include the usual checks (if the file exists, can it be open, can it be parsed etc.)
I'm an extreme noob to python, so If there's a better way to do what I'm asking please let me know.
I have one file, which works with flask to create markers on a map. It has an array which stores these said markers. I'm starting the file through command prompt, and opening said file multiple times. Basically, how would one open a file multiple times, and have them share a variable (Not the same as having a subfile that shares variables with a superfile.) I'm okay with creating another file that starts the instances if needed, but I'm not sure how I'd do that.
Here is an example of what I'd like to accomplish. I have a file called, let's
say, test.py:
global number
number += 1
print(number)
I'd like it so that when I start this through command prompt (python test.py) multiple times, it'd print the following:
1
2
3
4
5
The only difference between above and what I have, is that what I have will be non-terminating and continuously running
What you seem to be looking for is some form of inter-process communication. In terms of python, each process has its own memory space and its own variables meaning that if I ran.
number += 1
print(number)
Multiple times then I would get 1,2..5 on a new line. No matter how many times I start the script, number would be a global.
There are a few ways where you can keep consistency.
Writing To A File (named pipe)
One of your scripts can have (generator.py)
import os
num = 1
try:
os.mkfifo("temp.txt")
except:
pass # In case one of your other files already started
while True:
file = open("temp.txt", "w")
file.write(num)
file.close() # Important because if you don't close the file
# The operating system will lock your file and your other scripts
# Won't have access
sleep(# seconds)
In your other scripts (consumer.py)
while True:
file = open("temp.txt", "r")
number = int(file.read())
print(number)
sleep(# seconds)
You would start 1 or so generator and as many consumers as you want. Note: this does have a race condition that can't really be avoided. When you write to the file, you should use a serializer like pickler or json to properly encode and decode your array object.
Other Ways
You can also look up how to use pipes (both named and unnamed), databases, ampq (IMHO the best way to do it but there is a learning curve and added dependencies), and if you are feeling bold use mmap.
Design Change
If you are willing to listen to a design change, Since you are making a flask application that has the variable in memory why don't you just make an endpoint to serve up your array and check the endpoint every so often?
import json # or pickle
import flask
app = Flask(__name__)
array = [objects]
converted = method_to_convert_to_array_of_dicts(array)
#app.route("/array")
def hello():
return json.dumps(array)
You will need to convert but then the web server can be hosted and your clients would just need something like
import requests
import json
while True:
result = requests.get('localhost/array')
array = json.loads(str(result.body)) # or some string form of result
sleep(...)
Your description is kind of confusing, but if I understand you correctly, one way of doing this would be to keep the value of the variable in a separate file.
When a script needs the value, read the value from the file and add one to it. If the file doesn't exist, use a default value of 1. Finally, rewrite the file with the new value.
However you said that this value would be shared among two python scripts, so you'd have to be careful that both scripts don't try to access the file at the same time.
I think you could use pickle.dump(your array, file) to serie the data(your array) intoto a file. And at next time running the script, you could just load the data back with pickle.dump(your array, file)
I have a file that has over 200 lines in this format:
name old_id new_id
The name is useless for what I'm trying to do currently, but I still want it there because it may become useful for debugging later.
Now I need to go through every file in a folder and find all the instances of old_id and replace them with new_id. The files I'm scanning are code files that could be thousands of lines long. I need to scan every file with each of the 200+ ids that I have, because some may be used in more than one file, and multiple times per file.
What is the best way to go about doing this? So far I've been creating python scripts to figure out the list of old ids and new ids and which ones match up with each other, but I've been doing it very inefficient because I basically scanned the first file line by line and got the current id of the current line, then I would scan the second file line by line until I found a match. Then I did this over again for each line in the first file, which ended up with my reading the second file a lot. I didn't mind doing this inefficiently because they were small files.
Now that I'm searching probably somewhere around 30-50 files that can have thousands of line of code in it, I want it to be a little more efficient. This is just a hobbyist project, so it doesn't need to be super good, I just don't want it to take more than 5 minutes to find and replace everything, then look at the result and see that I made a little mistake and need to do it all over again. Taking a few minutes is fine(although I'm sure with computers nowadays they can do this almost instantly still) but I just don't want it to be ridiculous.
So what's the best way to go about doing this? So far I've been using python but it doesn't need to be a python script. I don't care about elegance in the code or way I do it or anything, I just want an easy way to replace all of my old ids with my new ids using whatever tool is easiest to use or implement.
Examples:
Here is a line from the list of ids. The first part is the name and can be ignored, the second part is the old id, and the third part is the new id that needs to replace the old id.
unlock_music_play_grid_thumb_01 0x108043c 0x10804f0
Here is an example line in one of the files to be replaced:
const v1, 0x108043c
I need to be able to replace that id with the new id so it looks like this:
const v1, 0x10804f0
Use something like multiwordReplace (I've edited it for your situation) with mmap.
import os
import os.path
import re
from mmap import mmap
from contextlib import closing
id_filename = 'path/to/id/file'
directory_name = 'directory/to/replace/in'
# read the ids into a dictionary mapping old to new
with open(id_filename) as id_file:
ids = dict(line.split()[1:] for line in id_file)
# compile a regex to do the replacement
id_regex = re.compile('|'.join(map(re.escape, ids)))
def translate(match):
return ids[match.group(0)]
def multiwordReplace(text):
return id_regex.sub(translate, text)
for code_filename in os.listdir(directory_name):
with open(os.path.join(directory, code_filename), 'r+') as code_file:
with closing(mmap(code_file.fileno(), 0)) as code_map:
new_file = multiword_replace(code_map)
with open(os.path.join(directory, code_filename), 'w') as code_file:
code_file.write(new_file)