How do I import in django_cities_light? - python

I noticed I could not find Newport, Oregon in my django_cities_light django application. It is a small city with population slightly above 10k, so I downloaded cities1000.zip which contains cities with a population higher than 1k. I unzipped this file and started searching for Newport's id and indeed it is there:
5742750 Newport Newport N'juport,Newport (Oregon),Njuport,ONP,nyupoteu,nyupoto,nywbwrt,nywpwrt awrgn,Њупорт,Ньюпорт,Нюпорт,نيوبورت,نیوپورت، اورگن,نیوپورٹ، اوریگون,ニューポート,뉴포트 44.63678 -124.05345 P PPLA2 US OR ...
Now, I have in my myapp/settings/development.py the following:
CITIES_LIGHT_TRANSLATION_LANGUAGES = ['en']
CITIES_LIGHT_INCLUDE_CITY_TYPES = ['PPL', 'PPLA', 'PPLA2', 'PPLA3', 'PPLA4', 'PPLC', 'PPLF', 'PPLG', 'PPLL', 'PPLR', 'PPLS', 'STLMT',]
CITIES_LIGHT_APP_NAME = 'jobs'
CITIES_LIGHT_CITY_SOURCES = ['http://download.geonames.org/export/dump/cities1000.zip'] # <-- this added as part of this task
I added CITIES_LIGHT_CITY_SOURCES following this post and the information here.
I then tried to import from using the following command, which I understand downloads the cities1000 file specified in myapp.settings.development:
python manage.py cities_light --settings=myapp.settings.development --force-all --progress
Newport, Oregon, with id 5742750 is not found in my database. I also cannot see from the command that my settings file is used and that the value of CITIES_LIGHT_CITY_SOURCES is overridden properly.
Does anyone know what I'm doing wrong and how to properly add from the source files? Thx!
EDIT: I added DJANGO_SETTINGS_MODULE explicity as an env var, went into the cities_light directory in my virtual environment and added a print that checks the value of DJANGO_SETTINGS_MODULE. It points to my settings file. I also added a print that prints the value of CITIES_LIGHT_CITY_SOURCES and it works. I also went into the cities_light/data and saw that both cities1000.zip and cities1000.txt were there. I deleted them, ran the command again, and they were there again. Still no success in having Newport, Oregon in my database.
EDIT2: It seems that I either was doing something wrong (likely) or there is a bug in cities_light (less likely). If I started with a database previously populated, lets say with cities15000.zip, and then try to populate it like I have tried before, it wont work. If I start from a completely empty database, and then run as I mentioned, it works. One thing I did notice is that I made a script to manually insert the cities of cities1000.txt. The insert for Newport worked successfully, so I then went to check the database and now ALL the cities were there. I am not sure how this happened, maybe they had been there for a while and I just missed it.
EDIT 3: Important to not confuse id and geoname_id.

Related

Anylogic, parameter variation experiment pipeline source error

I am trying to create a parameter variation based on a single python script run using the pipeline jar. I manually added a variable called 'pyCommunicator ' of the type other 'PyCommunicator' with initial value new 'PyCommunicator()'.
In the 'initial experiment set' block of the properties section, I import the python file via 'pyCommunicator.run("import GA");'
However, when I try to run the parameter variation I get the error code "PyCommunicator.java:669' indicating that the source code is not present. It also states '669 is not a valid line number in com.anylogic.libraries.pypeline.PyCommunicator'
When I add the jar file at the 'Change Attached Source' option the error does not change. The jar file is in the same folder as the anylogic and python files. I am not sure why the error occurs now. Has anybody experienced this issue before?
This is what it shows if you click on the error:

How to update environment variables through python code on Heroku?

What I'm trying to do is:
I'm trying to make a Telegram-bot that sends me a message when there's a new post updated on a specific web page.
I made the code and uploaded it on Heroku.
The bot is set to keep starting every 10 minutes using Heroku Scheduler so that it would detect any new post updated every 10 minutes.
Now the problem is:
The code is set to remember the latest post number and not to make any alarm if there's nothing updated between the previous bot run and the current run.
If the saved post number in the previous run matches the latest post number in the present run, the bot should not alarm me and keep doing the scheduled process (keep checking new posts every 10 minutes).
This is what I made to make this work
import os
latest_num = os.environ.get("POST_ID")
post_num = posts.find("td", {"class" : "no"}).text.strip()
if latest_num != post_num :
latest_num = post_num
os.environ["POST_ID"] = latest_num
I assume that if the latest post number from the previous run is saved as "POST_ID" through environment variables on Heroku, it should appear in the present run and be the value of latest_num when using os.environ to read "POST_ID" from the environment variable.
But the problem is, it seems like os.environ["POST_ID"] doesn't overwrite its value after the current run is done. Every time the Heroku scheduler starts the program, the 'latest_num' value is 0, the same as the default value of "POST_ID" on Heroku's settings.
So, even though there's no new post, the bot keeps sending me a message because 'latest_num' doesn't match 'post_num' all the time.
How can I fix this? Actually, I don't know whether setting environment variables through python code is possible or not. Please tell me if there's something better to make this work.
You can save those previous post id either in text file or in python variable. I prefer to save it in text file because if you restart or close your program then it forget all previous stuff but it isn't for text file.
Here's the code: At first while saving do this:
post_num = posts.find("td", {"class" : "no"}).text.strip()
with open("pre.txt","wb") as f:
f.write(post_num)
And now your scrape scrape second time then do this:
post_num = posts.find("td", {"class" : "no"}).text.strip()
with open("pre.txt","rb") as f:
pre_num=f.read()
if post_num!=pre_num:
#Do Something
with open("pre.txt","wb") as f:
f.write(post_num)
else:
#Do something

ValueError: need more than 0 values to unpack (Python 2)

I am trying to replicate another researcher's findings by using the Python file that he added as a supplement to his paper. It is the first time I am diving into Python, so the error might be extremely simple to fix, yet after two days I haven't still. For context, in the Readme file there's the following instruction:
"To run the script, make sure Python2 is installed. Put all files into one folder designated as “cf_dir”.
In the script I get an error at the following lines:
if __name__ == '__main__':
cf_dir, cf_file, cf_phys_file = sys.argv[1:4]
os.chdir(cf_dir)
cf = pd.read_csv(cf_file)
cf_phys = pd.read_csv(cf_phys_file)
ValueError: need more than 0 values to unpack
The "cf_file" and "cf_phys_file" are two major components of all files that are in the one folder named "cf_dir". The "cf_phys_file" relates only to two survey question's (Q22 and Q23), and the "cf_file" includes all other questions 1-21. Now it seems that the code is meant to retrieve those two files from the directory? Only for the "cf_phys_file" the columns 1:4 are needed. The current working directory is already set at the right location.
The path where I located "cf_dir" is as follows:
C:\Users\Marc-Marijn Ossel\Documents\RSM\Thesis\Data\Suitable for ML\Data en Artikelen\Per task Suitability for Machine Learning score readme\cf_dir
Alternative option in readme file,
In the readme file there's this option, but also here I cannot understand how to direct the path to the right location:
"Run the following command in an open terminal (substituting for file names
below): python cfProcessor_AEAPnP.py cf_dir cf_file cf_phys_file task_file jobTaskRatingFile
jobDataFile OESfile
This should generate the data and plots as necessary."
When I run that in "Command Prompt", I get the following error, and I am not sure how to set the working directory correctly.
- python: can't open file 'cfProcessor_AEAPnP.py': [Errno 2] No such file or directory
Thanks for the reading, and I hope there's someone who could help me!
Best regards & stay safe out there during Corona!!
Marc
cf_dir, cf_file, cf_phys_file = sys.argv[1:4]
means, the python file expects few arguments when called.
In order to run
python cfProcessor_AEAPnP.py cf_dir cf_file cf_phys_file task_file jobTaskRatingFile jobDataFile OESfile
the command prompt should be in that folder.
So, open command prompt and type
cd path_to_the_folder_where_ur_python_file_is_located
Now, you would have reached the path of the python file.
Also, make sure you give full path in double quotes for the arguments.

Error: IntegerField should be set to int, not float

I am relatively new to all of this but I have created an IntegerField(). After some time, I noticed that I need a FloatField() for that. However, when I change the IntegerField() to FloatField(), I get the Error: "IntegerField should be set to int, not float".
I try to sum up the number of trees (=no_trees) for every round with:
self.player.cumulative_donated_trees = sum([p.no_trees for p in self.player.in_all_rounds()])
When I leave the no_trees field as an IntegerField() and type in something like 0.9, I obviously get 0 in return. But I need to get 0.9.
self.player.cumulative_donated_trees = float(self.player.cumulative_donated_trees) is giving me the same Error message btw.
thanks in advance!
I think they deleted my comment under your other screenshot, so here is the procedure.
You look like you are using Pycharm. In the very bottom left, there should be a button which says “Terminal.”
Click on it.
Now go to your file tree (the list of files typically on the left side of Pycharm), and locate the directory where manage.py is.
Now, check that the path of that directory matches the path in your terminal.
If it does, skip the next step.
If it doesn’t, you need to navigate your Terminal to that directory. The most probable thing you will need to enter into your terminal is
cd YourProjectName
Once you are in the correct directory, enter into your terminal:
python manage.py makemigrations
If that works fine without errors, do:
python manage.py migrate
That should resolve your issue.

Subversion Hook Script WIndows, Python, pysvn

I'm trying to create a hook script for subversion on windows, I have a bat file that calls my python script but getting the log/comments seems to be beyond me.
I have pysvn installed and can get the transaction like this:
repos_path = sys.argv[1]
transaction_name = sys.argv[2]
transaction = pysvn.Transaction( repos_path, transaction_name)
I can also list what has changed:
transaction.changed(0)
What I cannot figure out is how to get the log/comment for the transaction. I realize that in pysvn there is a command similar to:
transaction.propget(propname,path)
But cannot for the life of me get it to return anything. I assume propname should be "svn:log", for path I have tried the fiel name, the repo path, null but all get are errors.
AT the end of the day I need to validate the comment, there will be matching against external data that will evolve, hence why I want to do it in python rather than the bat file, plus it may move to a linux server later.
AM I missing something obvious? How do I get the log/comment as a string?
Thanks, Chris.
After a great deal of trial and error and better searching after a day of frustration I found that I need to use the revision property, not a straight property, for a given transaction this will return the user submitted comment:
transaction.revpropget("svn:log")
There are other useful properties, this will return a list of all revision properties:
transaction.revproplist()
for example:
{'svn:log': 'qqqqqqq', 'svn:txn-client-compat-version': '1.9.7', 'svn:txn-user-agent': 'SVN/1.9.7 (x64-microsoft-windows) TortoiseSVN-1.9.7.27907', 'svn:author': 'harry', 'svn:date': '2017-12-14T16:13:52.361605Z'}

Categories