I wrote a simple python script that authenticates to a website, gets the cookie write it to a file and do some scrapping at the website. I'm writting the cookie to a file so, I can reuse it and don't need to authenticate my self over and over.
At my personal computer the script works fine. Although when I upload it to my server it refuse to work.
The most strange part is if I upload the cookie created at my personal computer to the my server it will work fine. Of course, I have some issues at the function that saves the cookie...
As far I as know if I have library issues Python would warm me about it, so I guess my problem is much more complex.
I also tried to run as a root, but no lucky.
What do you think may be causing this stuff?
BTW: All pythons are 2.7
Refer to tags to more infos
Related
I have written a python code that I can run it on my computer. It gets stock data from a website by GET request. Then it does some calculations on upon these online data and decides to send a buy/sell request to my broker's account.
All of these works fine on my computer, but I like to upload my code on a web server to preventing electricity/internet disconnection. I think I should upload my code on a virtual server, but this is all the thing I know and I don't have more experience or knowledge about it. I don't know if it is what I need, how can I do that?
In my experience of making some simple web pages, we should put our main code inside the public.html file? So should I buy a web host and wrap my python code inside a .html file?
Sorry if they are simple/stupid questions but I couldn't find any answer or guidence for doung what I want, so I asked here!
With the info you provide I can't really be sure how your script is ran.
I presume you run it by shell in that case a simple vps should suffice.
No need to mess around with a webserver.
Maybe look at Digital Ocean. They are a market leader and the simple $5 option probably enough for your purpose.
When you set it up you can connect to it with ssh (preferably with a ssh key because you are working with financial data) and setup your python script as if it were your own computer.
Ps. I don't work for digital ocean or get money from the link. It is just an example, any vps will work.
Scenario:
I am working an an auto whatsapp responder using whatsapp web.
I log in via chromedriver on selenium with python 3.
I run a function that does some stuff inside a while True.
Problem:
Sometimes, due to a lack of conectivity with the phone, or whatever other problems, the program just does not keep running the right way.
There are a lot of factors that might cause the whole thing to lose the right flow. I am analyzing them all and fixing them as best as I can.
Question:
I came up with the idea that maybe if I restart the whole thing every hour (or every whatever-thousands iterations) it would become more solid. As it will refind the flow no matter what happens, if I did not catch the bug yet.
Is it possible to restart the whole thing, without losing the browser session? Whatsapp web requires a QR scan, but it allows a "keep session alive in further connections" (which I do not really know how it works... if cookies or something else.)
Note: I know that a python script can be rebooted, but the bigger problem here is to reuse the browser session. Of course I am doing my research. None of what I read so far made me come with a solid solution, and that is why I ask to all the super cool brains out there.
Whatsapp stores session in localStorage of the browser.
You can extract the localStorage and save to a file on closing of a session.
Upon instantiating a session check if this file exists, then parse the file and update localStorage with saved values before opening a URL.
I am trying to invoke a python code for screen scraping (using Beautiful Soup) from my jsp servlet. Or it would also work if it can be directly invoked from the HTML.
Looked through few threads but couldn't get any solution.
What I want is to give the python program some arguments and want it to do some screen scrapping and return the result to jsp somehow.
I assume you are talking about web scraping which is pulling information from other websites.
You are not going to be able to do something like this in somebody's browser because it violates Javascript's same origin policy, AND there is no way a browser is going to let you download and execute a script on a client's computer.
You could just write a python script to do this for you and execute it yourself on your machine however.
Just be sure you are not violating web site's terms of service.
EDIT:
In that case I would recommend running the script on the command line, and then using the output of the program in the servlet to generate the responses you want.
I'm working on a web interface which currently runs using PHP and communicates locally to a python script.
I'm moving the web side to appengine, which so far is going well when being used locally, I'm currently communicating from the appengine app to the python app via get requests that are handled by the python script.
The problem is, that obviously the machine running the python script will be behind a firewall, I've never needed to do this before and am not sure on how to implement this best.
The only idea I have so far is for the python script to send post requests to the appengine with some data and then as a response, send back some other data. The only problem with this is that the web interface should update the client quite fast.
Any ideas?
Take a look at ProtoRPC Python API: https://developers.google.com/appengine/docs/python/tools/protorpc/overview
Though it is still marked as experimental, it seems to be a decent framework for what you are trying to do - send messages back and forth between the apps.
Since you said your local app runs behind a firewall, I'm assuming you cannot open up an endpoint and protect it with some form of authentication.
Once you have messages flowing, you can either use Channel API to keep the front-end updated: https://developers.google.com/appengine/docs/python/channel/overview
Or if you want to go more basic, just implement long/short polling through AJAX.
Sorry with the limited amount of info you have provided, that's all I can think of right now. Please feel free to post more details and I'll try to help further.
Here is what I would like to do, and I want to know how some people with experience in this field do this:
With three POST requests I get from the http server:
widgets and layout
and then app logic (minimal)
data
Or maybe it's better to combine the first two or all three. I'm thinking of using pyqt. I think I can load .ui files. I can parse json data. I just think it would be rather dangerous to pass code over a network to be executed on the client. If someone can hijack the connection, or can change the apps setting to access a bogus server, that is nasty.
I want to do it this way because it keeps all the clients up-to-date. It's sort of like a webapp but simpler because of Qt. Essentially the "thin" app is just a minimal compiled python file that loads data from a server.
How can I do this without introducing security issues on the client? Is https good enough? Is there a way to get pyqt to run in a sandbox of sorts?
PS. I'm not stuck on Qt or python. I do like the concept though. I don't really want to use Java - server or client side.
Your desire to send "app logic" from the server to the client without sending "code" is inherently self-contradictory, though you may not realize that yet -- even if the "logic" you're sending is in some simplified ad-hoc "language" (which you don't even think of as a language;-), to all intents and purposes your Python code will be interpreting that language and thereby execute that code. You may "sandbox" things to some extent, but in the end, that's what you're doing.
To avoid hijackings and other tricks, instead, use HTTPS and validate the server's cert in your client: that will protect you from all the problems you're worrying about (if somebody can edit the app enough to defeat the HTTPS cert validation, they can edit it enough to make it run whatever code they want, without any need to send that code from a server;-).
Once you're using https, having the server send Python modules (in source form if you need to support multiple Python versions on the clients, else bytecode is fine) and the client thereby save them to disk and import / reload them, will be just fine. You'll basically be doing a variant of the classic "plugins architecture" where the "plugins" are in fact being sent from the server (instead of being found on disk in a given location).
Use a web-browser it is a well documented system that does everything you want. It is also relatively fast to create simple graphical applications in a browser. Examples for my reasoning:
The Sage math environment has built their graphical client as an application that runs in a browser, together with a local web-server.
There is the Pyjamas project that compiles Python to Javascript. This is IMHO worth a try.
Edit:
You could try PyPy's sandbox interpreter, as a secure Python interpreter for the code that was transferred over a network.
An then there is the most simple solution: Simply send Python modules over the network, but sign and/or encrypt them. This is the way all Linux distributions work. You store a cryptographic token on the local computer. The server signs/encrypts the code before it sends it, with a matching token. GPG should be able to do it.