Ok, so I have used this code to setup a simple program. Basically it just sets points on a map in python. Very basic. The problem now is I have a condition where when that condition happens it refreshes the page with a simple QWebView.setHtml(). What I want to do now is find a way to get the current zoom and center information of the map and save it. Is there anyway to go into the javascript and grab that information, store it and then write it into the html the same way this code already does before i do the refresh?
Sorry if this is confusing or broad, I just cant think of a way to get the information from the map using python.
There are a number of ways to interact with client-side code from the server (Python). You could set a cookie on the client, send an AJAX request to the server from Javascript, submit a form, or go a more exotic route like WebSockets. Without more information, I'm not sure we can tell you which is the best.
What web framework are you using?
EDIT:
Oh, I see- you're using a scriptable web view... maybe something like zoom = view.evaluateJavaScript('map.getZoom();')? From what I can see of the library, the difficult part might be getting a reference to the map var in JS.
EDIT:
I don't think this is possible without modifying or extending pymaps, since it scopes the GMap locally in JS and doesn't expose it anywhere. I've done just that in a gist. You can then access the zoom with something similar to the above- maybe zoom = view.evaluateJavaScript("PyMaps[0].gmap.getZoom();").
EDIT:
In case this wasn't clear- the gist I included requires that you use MyPyMap instead of PyMap.
From another StackOverflow question, I realized you can't do evaluteJavaScript straight on a view. The subsequent code would look more like this
doc = view.page().mainFrame().documentElement()
zoom_level = doc.evaluateJavaScript("PyMaps[0].gmap.getZoom();")
Related
Goal: I want to send the result of a <script> tag in HTML to a server of any kind.
I am working with the Spotify authorization API for a project and one of the required keys is contained in a query string at the end of my app’s redirect URI. I need to get this key.
My solution is to set the redirect URI to a redirect page. Then, on the page, automatically run a script that gets the current URL and send it to my Python script for use. If there’s another way, please tell me, because I’m pretty stuck.
However, to send variables between HTML and Python, I have found I need to use a simple server. Setting this up is the hard part. I’ve made Java, Node.js, and Python servers, but nothing seems to want to work with the <string> tag, and I’m doubtful something like this would even get the actual output of the script.
Is there a way to do this? This is a pretty long question and I apologize, and it should probably be noted I’m a beginner so an explanation and code examples would be nice. Thank you to anyone who reads this!
I'm trying to get a specific data from a website, but this is a little bit complicated to understand so here is some images.
So, first, I'm on this page,
Image1
then I click on the icon in the middle and something pop,
popup
then I have to click on this,
almost there
And finally I land here
arrival
And I want to get all the names of the people here
So, my question is, is there a way to get directly this list with a requests ?
If yes, how do i have do to ? I can't find the URL of this kind of pop up and I'm a complete beginner with requests and all this kind of things..
(To get the name, I have to be connected on my account by the way)
So, since I don't know how to access to the pop-up windows, this is the only code I got :
import requests
x = requests.get('https://www.tiktok.com/#programm___r?lang=en', headers={'User-Agent':'test'})
print(x.text)
I checked what it prints, and i didn't see a sign of the pop-up window
you can get some sort of network interception tool like Burpsuite and watch the network traffic that comes through each time you click on each link along the way to your final destination, this should give you an endpoint you may be able to send your request too. I think this network information should also be available in the browser tools but I'm not sure. A potential issue here is that usually tokens and other information has to be passed down the chain along the way, which might make scripting something like this too hard.
So aside from that, with browser automation software like selenium, you could automate the process of getting to that point on the page, and be able to pull out the list you want once you're there. I've used selenium myself and it's really usable and well documented!
I'm currently trying to design a filter that with it I can block certain URLs and also
block based on keywords that may be at the data that came back from the http response.
Just for clarification I'm working on a Windows 10 x64 machine for this project.
In order to be able to do such thing, I quickly understood that I would need a web proxy.
I checked out about 6 proxies written in python that I found on github.
These are the project I tried to use (some are Python3 some 2):
https://github.com/abhinavsingh/proxy.py/blob/develop/proxy.py
https://github.com/inaz2/proxy2/blob/master/proxy2.py
https://github.com/inaz2/SimpleHTTPProxy - this one is the earlier version of the top one
https://github.com/FeeiCN/WebProxy
Abhinavsingh's Proxy (first in the list):
what I want to happen
I want the proxy to be able to block sites based on the requests and the content came back, I also need the filter to be in a separated file and to be generic
so I can apply it on every site and every request/response.
I'd like understand where is the correct place to put a filter on this proxy
and how to do redirect or just send a block page back when the client tries to access
sites that has specific urls, or when the response is a page that contains some keywords.
what I tried
I enabled the proxy on google chrome's 'Open proxy settings'
and executed the script. It looked pretty promising, and I noticed that I can insert a filterer's function call in line 383 at the _process_request function so that I can return
to it maybe another host to redirect to or just block. It worked partially for me.
The problems
First of all, I couldn't fully redirect/block the sites. Sometimes it worked, Sometimes it
didn't.
Another problem I ran into was that I realized that I can't access the content of site that returned, if it is https.
Also, filter the response was unfortunately not clear for me.
I noticed also that proxy2 (the second in the list) can solve that problem I had
of filtering https page's content, but I couldn't find out how to make this feature work (and also I think it requires linux utilities anyhow).
The process I described above was pretty much the one that I tried to work on every proxy in the list. At some proxies, like proxy2.py I couldn't understand at all what I needed to do.
If anybody managed to make a filter on top of this proxy or any other from this list and can help me understand how to do it, I'll be grateful if you'll comment down below.
Thank you.
I am currently pulling data from a public series data from https://www3.bcb.gov.br/expectativas/publico/en/serieestatisticas
This is a public page that uses apache wicket I believe.
I usually am ok with scraping, whether GET or POST. Here I and my colleagues are stuck. Can anyone help understand what URL needs to be used to actually make the request. Here's what I've got so far:
The form with inputs:
The Fiddler capture manually executed:
Text View:
form19_hf_0=&indicador=0&calculo=0&linhaPeriodicidade%3Aperiodicidade=0&tfDataInicial=11%2F10%2F2015&tfDataFinal=11%2F24%2F2015&divPeriodoRefereEstatisticas%3AgrupoAnoReferencia%3AanoReferenciaInicial=16&divPeriodoRefereEstatisticas%3AgrupoAnoReferencia%3AanoReferenciaFinal=16&btnCSV=Generate+CSV
Form data I'm passing in the request:
Summary:
I need some help, I can't seem to get the POST working correctly, it takes me to a different page, and I'm not sure of how to work through this one.
NB: I'm trying to grab back a CSV.
The libraries I'm using are primarily Requests (I was going to use LXML but I don't think its going to be applicable here).
I've been trying to figure out the right form with Postman and Fiddler to understand what the request needs to be.
So,
The solution to this was somewhat indirect. We were not able to do a straight POST because the the page incremented the actual POST url in a way that was generally impossible to predict.
The solution that we used was installing Selenium web driver and using that to simulate the dropdown visible values and button clicks.
This worked out very cleanly.
Thanks and HTH anyone else who might have a similar problem.
I want to create a special wiki page on my local Redmine server. It should contain an inventory of some executables from my server. My goal is a script which scans certain folders on my server for these files and put them (with some additional information) in a nice Redmine wiki page.
My first thought was to traverse my server's file system with a simple batch file and to create a SQL expression for putting the results directly into the underlying mySQL database (which contains Redmine's wiki pages). But I consider this too risky and too error-prone.
Then I had the idea to use a script language like python (which I always wanted to learn) to retrieve the information and send it back to the Redmine server, like a web browser would do. This should be a much safer way. But this doesn't seems to be an easy beginner's task when just starting with python - I fail to authenticate myself on the Redmine server.
My last idea was to create a HTML page with python, which could be displayed within a Redmine wiki page with the plugin 'Redmine Wiki Extensions'. But I consider this only as a solution light, because it's not very elegant.
So what I seek is either a new idea to solve this problem or some clues on how to do a proper authentification with python on my Redmine server - maybe I could use a cookie for easier authentification...
I'm not familiar with redmine, but if you are looking for something like having a script that performs some actions the same way you would do in a browser, then mechanize is a library that might be helpful for you unless there's some javascript involved. In that case, then I'd look into something like windmill or selenium to let you drive the web browser.
However, please note using web scraping is also error-prone since any change in the design of the web pages involved might break your scripts.
Regarding the option of using an API as pointed out by the comment from AdamKG, that would be a good option, since there's a REST API that you can use from python if you like. Unfortunately, I don't see anything to let you do what you're looking for and it seems it hasn't yet reached the stable status yet. Anyway, as I said, it's still a good option to consider in the future.