Is there a way to get the exact date/time from the web rather than taking the PC date/time?
I am creating a website where the answer is time relevant. But i don't want someone cheating by putting their pc clock back. When i do:
today = datetime.datetime.today()
or
now = datetime.datetime.utcnow().replace(tzinfo=utc)
I still get whatever time my pc is set to.
Is there a way to get the correct date/time.
datetime.today() takes its time information from the server your application is running on. If you currently run your application with python manage.py localhost:8000, the server is your local PC. In this scenario, you can tamper with the time setting of your PC and see different results.
But in production environment, your hosting server will provide the time information. Unless you have a security issue, no unauthorized user should be able to change that.
Related
I am trying to get DNS records of particular domain. So, I found dnspython package where it could be easily done. It works fine when I run it from my computer. However, when I call it from Django views it shows the previous records (old records) which means it's not updating.
Is it some kind of caching in OS level? Note that, I am also using Docker. Restarting docker and clearing cache in Django didn't help, still shows old records.
Here's the sample code for checking records:
import dns.resolver
result = dns.resolver.resolve("domain.com", "TXT")[0].to_text()
The code snippet above works and shows any update in TXT record, when I run it from my computer. However, in Django it's stuck in old records and not updating.
In Django views:
def verify_dns_view(request, domain_id):
domain = get_object_or_404(models.Domain, id=domain_id)
mx_record = dns.resolver.resolve(domain.name, "MX")[0].to_text()
txt_record_spf = dns.resolver.resolve(domain.name, "TXT")[0].to_text()
...
There always might be a different DNS server that your app and PC are connecting to. In your case app server is "further away" from the server where the actual domain is registered so it did not update the record yet.
I apologize ahead of time, as this may seem like a ridiculous question. I have used Python for the purposes of regex, automation and leveraging pandas for work. I would now like to run a web scraping script (nothing special, < 50 lines, 24 hours a day, which i think i will need server access for to do so(can't run on laptop if laptop is shut while i am sleeping) for personal use, but I have no idea how to go about this?
Would i need to rent a virtual machine, physical server space? My intent is to run the script and send an email with the results, its all very low tech and low capacity, more of a notification. How does one go about modifying their code to do something like this and then what kind of server access would i need? any suggestions would be helpful!!
UNIX style approach to the problem:
Rent a server ( VPS ) - you can find something for $2.5-$5
Create and upload your scraping script to the server
Create crontab ( tool that's responsible for executing your script on regular basic - say once per hour or whatever you want )
Use https://docs.python.org/3/library/email.html package for sending emails from Python
I have been working on a website for over a year now, using Django and Python3 primarily. A few of my buddies and I built a front end where a user enters some parameters and submits, this goes to the GAE to run the job and return the results.
In my local dev environment, everything works well. I have two separate dev environments. One builds the entire service up in a docker container. This produces the desired results in roughly 11 seconds. The other environment runs the source files locally on my computer and connects to the Postgres database hosted in Google Cloud. The Python application runs locally. It takes roughly 2 minutes for it to run locally, a lot of latency between the cloud and the post/gets from my local machine.
Once I perform the Gcloud app deploy and attempt to run in production, it never finishes. I have some print statements built into the code, I know it gets to the part where the submitted parameters go to the Python code. I monitor via this command on my local computer: gcloud app logs read.
I suspect that since my local computer is a beast (i7-7770 processor with 64 GB of RAM), it runs the whole thing no problem. But in the GAE, I don't think it's providing the proper machines to do the job efficiently (not enough compute, not enough RAM). That's my guess.
So, I need help in how to troubleshoot this. I tried changing my app.yaml file so that resources would scale to 16 GB of memory, but it would never deploy. I received an error 13.
One other note, after it spins around trying to run the job for 60 minutes, the website crashes and displays this message:
502 Server Error
Error: Server Error
The server encountered a temporary error and could not complete your request.
Please try again in 30 seconds.
OK, so just in case anybody in the future is having a similar problem...the constant crashing of my Google App Engine workers was because of using Pandas dataframes in the production environment. I don't know exactly what Pandas was doing, but I kept getting Memory Errors, it would crash the site...and it didn't appear to be occurring in a single line of code. That is, it randomly happened somewhere in a Pandas Dataframe operation.
I am still using a Pandas Dataframe simply to read in a csv file. I then use
data_I_care_about = dict(zip(df.col1, df.col2))
#or
other_data = df.col3.values.tolist()
and then go to town with processing. As a note, on my local machine (my development environment basically) - it took 6 seconds to run from start to finish . That's a long time for a web request but I was in a hurry, thus why I used Pandas to begin with.
After refactoring, the same job completed in roughly 200ms using python lists and dicts (again, in my dev environment). The website is up and running very smoothly now. It takes a maximum of 7 seconds after pressing "Submit" for the back-end to return the data sets and render on the web page. Thanks for the help peeps!
Newbie on appengine and I really don't know how to phrase the question which sadly results in me not knowing what keywords to google and I hope that i really do get help other than the bashing that a lot of people do.
I'm confused between the behavior of appengine online and the appengine on the local server.
Background info:
Btw this is in Python
Initially i assumed that , when needed or as authored
an instance of the app or module will be created.
And that instance will be the one serving multiple requests from different clients.
In this behavior any initialization code will only be run once.
But in the local development server.
Every time i add something new, specially in the main.py,
the server is able to catch the new changes,
then on browser-refresh be able to run it.
This made me think, wait...
Does it run the entire script over and over again
on every request?
Question:
Does an instance/module run the entire code on every request or is this just an added behavior to the dev server to make development easier?
Both your assumptions - about behaviour in production and development - are wrong.
In production, GAE spins up instances as required. This may be in response to increased load, or the host may simply decide after a certain amount of time to recycle an instance by killing it and starting a new one. Initialization code will always be run whenever a new instance is started.
In development, you only get a single instance. However, the server watches your file system for changes. If it detects a change to the code itself, it will restart itself, and therefore re-run the initialization code. But if you don't make any code changes between requests, the existing process continues indefinitely, and init code will not be re-run.
I have some simple python code running in Google App Engine such as this:
types = memcache.get('types')
if types is None:
# do something, creating a 'types' object
memcache.set('types', types, 36000000)
Whenever I run this on the local development server, memcache.get('types') always returns None. It is not the same live on App Engine, the memcache calls work correctly.
Is it necessary to install a separate package along with the GAE development server locally?
The time argument to memcache.set can be a maximum of one month to indicate a relative lifetime, otherwise it is interpreted as an absolute unix timestamp (seconds since 1970). 36000000 is much more than a month and so it's setting the entry to expire in February 1971.
If you want something to stay in cache for as long as possible, then leave out the time argument.