I'm coding some automation tool for a specific web site, and having some problems. The web site needs to access (such as pushing buttons) within a browser to get JSON format responses.
(I'm familiar with Python but not regarding network traffic such things. and sorry for my poor explanation, English is not my first language)
i have to listen to JSON format response from the web site, AFAIK, local proxy(127.0.0.1) is needed to fetch the traffic. I've found a code(http://luugiathuy.com/2011/03/simple-web-proxy-python/) that fetches data from port 80 (HTTP). however, the code needs to change network setting of my PC. is there any way not to change whole pc's proxy setting to get traffic data? it slows my pc. i want to run this code "independently?". i've tried to emulate independent web browser to handle this but i had hard time to figure out setting independent local proxy.
followed by question #1, the website needs some button action triggered by mouse. as i mentioned above(independently operatable), it has not to interfere actual mouse. is there any library i can use for this purpose? i've tried to create some "virtual mouse" to achieve this goal but sadly failed.
i have more detail questions but shorten down to most crucial ones.
Related
I was making some test http requests using Python's request library. When searching for Walmart's Canadian site (www.walmart.ca), I got this:
How do servers like Walmart's detect that my request is being made programatically? I understand browsers send all sorts of metadata to the server. I was hoping to get a few specific examples of how this is commonly done. I've found a similar question, albeit related to Selenium Web Driver, here where it claims that there are some vendors that provide this service but I was hoping to get something a bit more specific.
Appreciate any insights, thanks.
As mentioned in the comments, a real browsers send many different values - headers, cookies, data. It reads from server not only HTML but also images, CSS, JS, fonts. Browser can also run JavaScript which can get other information about browser - version, extensions, data in local storage, etc (i.e how you move mouse). And real human loads/visits pages with random delays and in rather in random order. And all these elements can be used to detect a script. Servers may use very complex systems even Machine Learning (Artificial Intelligence) and use data from few mintues or hours to compare your behavior.
tl;dr -- how do I combine a pair of SDK projects together and have the "menuconfig" system work properly?
I have a simple project in mind for an ESP32 module: Have the ESP32 provide a new dedicated WiFi network that allows a smart phone to connect directly to the ESP32. Once connected, the phone can point its web browser to the IP address of the ESP32 (hard-coded as 192.168.1.1) and the ESP32 will accept HTTP connections from the phone. Thus the user will have a simple interface through which to interact with the ESP32 using HTML data-entry forms, buttons and so on.
Note, this connection goes no further than the ESP32, no internet connection is required at all for this project.
Luckily, the SDK for the module includes lots of examples, and I have managed to get the web server working, and also the WiFi access point, but not both at the same time. These are two example projects ("... examples/protocols/http_server/simple" and "... examples/wifi/getting_started/softAP").
So far I've managed to discover that the WiFi access point software runs in the "background" and continues even after the main() function has ended. This means that I should be able to bring in the web-server code from the other example and execute it in the main() function. I'm pretty sure that will be feasible (please correct me if I'm mistaken).
The Question:
Each of these two example projects contain a "idf.py menuconfig" command that lets the user quickly change any of the important settings such as the WiFi SSID or the HTTP basic authentication password.
I'd like to retain this menu system after the two projects have been combined because it's extremely useful and somewhat reduces the human error involved in altering the code before recompiling.
Is there a simple way to combine the menuconfig systems from two SDK example projects into a single one that can be used to fully configure a project that is a combination of both code-bases?
Further info: I'm using the ESP-IDF dev kit on an Ubuntu 20 virtual machine, connecting to the ESP32 via USB.
It's very simple. The menuconfig items are described in the component's Kconfig.projbuild file. If you want add those items from one project to the other, you can simply copy-paste the content of one main/Kconfig.projbuild file to the other one and that's it. See ESP IDF documentation for details.
I'm attempting to automate tests of Adobe Analytics (aka Omniture) instrumentation of a web app by implementing test scripts with the Selenium Python package.
If correctly instrumented, HTTP requests are made from the browser with certain expected query parameters. Is there a Python package that would allow me to capture those outgoing HTTP requests? Right now, we do it manually with the Chrome dev tools in the Network -> Images section.
This application is also available as a native app across nearly twenty other platforms (including Smart TVs and game consoles), and I'll need to perform similar tests across those. Although, unfortunately, I won't be able to automate the script, I'd still like to capture and store the HTTP calls. I'm currently using HTTPScoop to do this manually.
I'm most comfortable with Python, but if there's a simple way of doing this in another language, I'm all ears.
I was recently working on a similar task so I can share my experience and what I've learnt on the way (rather than give you the solution).
First you need to run a proxy on your machine (e.g. http://bmp.lightbody.net/). Then I needed to run manually a few commands ( https://github.com/lightbody/browsermob-proxy#rest-api). Once the proxy was running I wrote a small script following example here https://github.com/lightbody/browsermob-proxy#using-with-selenium. Finally you simply loop over the har entries as captured on the proxy and check if an analytics request is present (you can check for URL params if needed).
I have this ready in form of a unit test for FF and Chrome (for a given URL). To be able to run this test on different devices/OS/platforms one would probably need to run the code through selenium remote webdriver https://code.google.com/p/selenium/wiki/RemoteWebDriver using service like https://www.browserstack.com/ in the cloud. I contacted them but they don't have any documentation ready but suggested I refer to online resources. That's where I am now.
Hope it helps
I have developed a python program that parses a webpage and creates a new text document with the parsed data. I want to deliver this new information to the web. I have no idea where to start with something like this. Are there any free options where I can have a site automatically call this python code upon request and update the new data to its page? Or is the only feasible solution here to have my own website/server that uses my code? I'm honestly pretty overwhelmed with many of the options when I try to begin doing a web-search for a solution like this. I have done a decent amount of application programming before so i'm confident in my ability to learn new things, but web protocols are all new to me so its hard to find a starting point.
Ultimately I want this python code to run automatically, or per request of a user, and deliver to the data to them. It could even be through an email, although that is probably less practical.
I personally have good experience using Google Appengine (and its free for a limited amount of requests). The downside is that it does not allow C-extensions or Python3.
If you want to host your own server, tornado is a good option I think. Tornado supports both Python2 and Python3.
There are a great deal of options available.. from 'traditional' virtual server or website hosts like a2hosting or godaddy to 'Cloud Application Hosts' such as Amazon EC2, Heroku or OpenShift.
For your case, and without knowing more, I would suggest that an application hosting is more appropriate, and that you should take a look at Heroku and Openshift in particular.
Define carefully what you want to achieve (how the users access your application, what they see, how they interact with it... etc..) and then evaluate these options based on those requirements.
Most offer a free trial, or even free services, depending on what you need! Good luck
If you've never worked with web technologies before this will be a overwhelming task, since there's a lot of different technologies involved, and many possible ways to combine them.
You'll probably want to start by familiarizing yourself with the very basics of the HTTP protocol.
Then you should read a bit on CGI server-side programming (the article also has a quick overview on HTTP).
Python can run both on CGI and WSGI (if the server provider allows such access), so you may also want to read about WSGI.
Once you grasp all these concepts, you should check this question for actual python techniques.
Also, since you seem to be under the impression you must pay to have a website/app deployed, you should know there are companies that host python apps for free
I'm working on a web interface which currently runs using PHP and communicates locally to a python script.
I'm moving the web side to appengine, which so far is going well when being used locally, I'm currently communicating from the appengine app to the python app via get requests that are handled by the python script.
The problem is, that obviously the machine running the python script will be behind a firewall, I've never needed to do this before and am not sure on how to implement this best.
The only idea I have so far is for the python script to send post requests to the appengine with some data and then as a response, send back some other data. The only problem with this is that the web interface should update the client quite fast.
Any ideas?
Take a look at ProtoRPC Python API: https://developers.google.com/appengine/docs/python/tools/protorpc/overview
Though it is still marked as experimental, it seems to be a decent framework for what you are trying to do - send messages back and forth between the apps.
Since you said your local app runs behind a firewall, I'm assuming you cannot open up an endpoint and protect it with some form of authentication.
Once you have messages flowing, you can either use Channel API to keep the front-end updated: https://developers.google.com/appengine/docs/python/channel/overview
Or if you want to go more basic, just implement long/short polling through AJAX.
Sorry with the limited amount of info you have provided, that's all I can think of right now. Please feel free to post more details and I'll try to help further.