Avoid running GeoIP on every page - python

This is the module I am working with: http://wiki.nginx.org/HttpGeoipModule
From what I can see, since it is configured on the nginx config and uwsgi it looks like there is no choice but to have it run the geoip on every page and then only collect and use the variable when needed.
From a performance point of view I would rather have it so I request the geoip ONLY when needed, cache it in a cookie or session and then not request it again to speed up the site.
Is anyone able to tell me if this is possible?

From a performance point of view I would rather have it so I request the geoip ONLY when needed, cache it in a cookie or session and then not request it again to speed up the site.
Is anyone able to tell me if this is possible?`
Yes, it's possible. But from a performance point of view, you should not worry, as geoip database are stored in memory (at the reading configuration phase) and nginx doing lookups very fast.
Anyway if you want, you can use something like:
set $country $cookie_country;
if ($country == '') {
set $country $geoip_country_code;
add_header Set-Cookie country=$geoip_country_code;
}
uwsgi_param GEOIP_COUNTRY $country;

No, you can't make nginx to perform GeoIP lookup on demand only. Since you define a geoip_country or geoip_city directive, nginx will request data from GeoIP database, whether the answer is used later or not. But you can fetch GeoIP data without nginx at all, i.e. directly with your application. Take a look for python geoip lib: http://dev.maxmind.com/geoip/downloadable#Python-5

Related

has been blocked by CORS policy: Request header field access-control-allow-origin is not allowed by Access-Control-Allow-Headers in preflight response [duplicate]

I am working on an app using Vue js.
According to my setting I need to pass to a variable to my URL when setting change.
<!-- language: lang-js -->
$.get('http://172.16.1.157:8002/firstcolumn/' + c1v + '/' + c1b, function (data) {
// some code...
});
But when my app hit on URL, it shows the following message.
Failed to load http://172.16.1.157:8002/firstcolumn/2017-03-01/2017-10-26: Redirect from 'http://172.16.1.157:8002/firstcolumn/2017-03-01/2017-10-26' to 'http://172.16.1.157:8002/firstcolumn/2017-03-01/2017-10-26/' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://localhost:8080' is therefore not allowed access.
In addition to what awd mentioned about getting the person responsible for the server to reconfigure (an impractical solution for local development) I use a change-origin chrome plugin like this:
Moesif Orign & CORS Changer (use to be free but now wants a work email address >_>)
Allow CORS: Access-Control-Allow-Origin
You can make your local dev server (ex: localhost:8080) to appear to be coming from 172.16.1.157:8002 or any other domain.
In case the 2nd plugin link breaks in the future or the plugin writer decides to capitalize off the fame of this thread, open your browser's
plugin marketplace and search "allow cors", there's going to be a
bunch of them.
Thanks all, I solved by this extension on chrome.
Allow CORS: Access-Control-Allow-Origin
If you have control over your server, you can use PHP:
<?PHP
header('Access-Control-Allow-Origin: *');
?>
Ask the person maintaining the server at http://172.16.1.157:8002/ to add your hostname to Access-Control-Allow-Origin hosts, the server should return a header similar to the following with the response-
Access-Control-Allow-Origin: yourhostname:port
Using npm:
To allow cross-origin requests install 'cors':
npm i cors
Add this in the server-side:
let cors = require("cors");
app.use(cors());
When you have this problem with Chrome, you don't need an Extension.
Start Chrome from the Console:
chrome.exe --user-data-dir="C:/Chrome dev session" --disable-web-security
Maybe you have to close all Tabs in Chrome and restart it.
I will assume that you're a front-end developer only and that you don't have access to the backend of the application (regarding the tags of the question).
Short answer on how to properly solve this in your case? You can't, you'll need somebody else.
What is this about?
You need to understand that CORS is a security thing, it's not just here to annoy you just for fun.
It's purpose is to mainly prevent the usage of a (malicious) HTTP call from a non-whitelisted frontend to your backend with some critical mutation.
You could give a look to this YouTube video or any other one really, but I recommend a visual video because text-based explanation can be quite hard to understand.
You also need to understand that if you use Postman or any other tool to try your API call, you will not get the CORS issue. The reason being that those tools are not Web frontends but rather some server-based tools.
Hence, don't be surprised if something is working there but not in your Vue app, the context is different.
Now, how to solve this?
Depending of the framework used by your backend team, the syntax may be quite different but overall, you'll need to tell them to provide something like Access-Control-Allow-Origin: http://localhost:3000 (or any other port you'll be using).
PS: Using Access-Control-Allow-Origin: * would be quite risky because it would allow anybody to access it, hence why a stricter rule is recommended.
If you're using a service, like an API to send SMS, payment, some Google console or something else really, you'll need to allow your localhost in the dashboard of the service. Ask for credentials to your manager or Tech Lead.
If you have access to the backend, you could it yourself as shown here (ExpressJS in this example): https://flaviocopes.com/cors/
How to hack it in a dirty way?
If you're in a damn hurry and want to get something really dirty, you could use a lot of various hacks a listed in the other answers, here's a quick list:
use any extension who is able to create a middleware and forward the request to the backend (it will work because it's not directly coming from your frontend)
force your browser to disable CORS, not sure how this would actually solve the issue
use a proxy, if you're using Nuxt2, #nuxtjs/proxy is a popular one but any kind of proxy (even a real backend will do the job)
any other hack related somehow to the 3 listed above...
At the end, solving the CORS issue can be done quite fast and easily. You only need to communicate with your team or find something on your side (if you have access to the backend/admin dashboard of some service).
I heavily do recommend trying get it right from the beginning because it's related to security and that it may be forgotten down the road...
The approved answer to this question is not valid.
You need to set headers on your server-side code
app.use((req,res,next)=>{
res.setHeader('Access-Control-Allow-Origin','*');
res.setHeader('Access-Control-Allow-Methods','GET,POST,PUT,PATCH,DELETE');
res.setHeader('Access-Control-Allow-Methods','Content-Type','Authorization');
next();
})
You can also try a chrome extension to add these headers automatically.
Hello If I understood it right you are doing an XMLHttpRequest to a different domain than your page is on. So the browser is blocking it as it usually allows a request in the same origin for security reasons. You need to do something different when you want to do a cross-domain request. A tutorial about how to achieve that is Using CORS.
When you are using postman they are not restricted by this policy. Quoted from Cross-Origin XMLHttpRequest:
Regular web pages can use the XMLHttpRequest object to send and receive data from remote servers, but they're limited by the same origin policy. Extensions aren't so limited. An extension can talk to remote servers outside of its origin, as long as it first requests cross-origin permissions.
To add the CORS authorization to the header using Apache, simply add the following line inside either the <Directory>, <Location>, <Files> or <VirtualHost> sections of your server config (usually located in a *.conf file, such as httpd.conf or apache.conf), or within a .htaccess file:
Header set Access-Control-Allow-Origin "*"
And then restart apache.
Altering headers requires the use of mod_headers. Mod_headers is enabled by default in Apache, however, you may want to ensure it's enabled.
I had the same problem in my Vue.js and SpringBoot projects. If somebody work with spring you can add this code:
#Bean
public FilterRegistrationBean simpleCorsFilter() {
UrlBasedCorsConfigurationSource source = new UrlBasedCorsConfigurationSource();
CorsConfiguration config = new CorsConfiguration();
config.setAllowCredentials(true);
// *** URL below needs to match the Vue client URL and port ***
config.setAllowedOrigins(Collections.singletonList("http://localhost:8080"));
config.setAllowedMethods(Collections.singletonList("*"));
config.setAllowedHeaders(Collections.singletonList("*"));
source.registerCorsConfiguration("/**", config);
FilterRegistrationBean bean = new FilterRegistrationBean<>(new CorsFilter(source));
bean.setOrder(Ordered.HIGHEST_PRECEDENCE);
return bean;
}
I found solution in this article Build a Simple CRUD App with Spring Boot and Vue.js
You are making a request to external domain 172.16.1.157:8002/ from your local development server that is why it is giving cross origin exception.
Either you have to allow headers Access-Control-Allow-Origin:* in both frontend and backend or alternatively use this extension cors header toggle - chrome extension unless you host backend and frontend on the same domain.
Try running this command in your terminal and then test it again.
curl -H "origin: originHost" -v "RequestedResource"
Eg:
If my originHost equals https://localhost:8081/ and my RequestedResource equals https://example.com/
My command would be as below:
curl -H "origin: https://localhost:8081/" -v "https://example.com/"
If you can notice the following line then it should work for you.
< access-control-allow-origin: *
Hope this helps.
Do specify #CrossOrigin(origins = "http://localhost:8081")
in Controller class.
You can solve this temporarily by using the Firefox add-on, CORS Everywhere. Just open Firefox, press Ctrl+Shift+A , search the add-on and add it!
You won't believe this,
Make sure to add "." at the end of the "url"
I got a similar error with this code:
fetch(https://itunes.apple.com/search?term=jack+johnson)
.then( response => {
return response.json();
})
.then(data => {
console.log(data.results);
}).catch(error => console.log('Request failed:', error))
The error I got:
Access to fetch at 'https://itunes.apple.com/search?term=jack+johnson'
from origin 'http://127.0.0.1:5500' has been blocked by CORS policy:
No 'Access-Control-Allow-Origin' header is present on the requested
resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.
But I realized after a lot of research that the problem was that I did not copy the
right URL address from the iTunes API documentation.
It should have been
https://itunes.apple.com/search?term=jack+johnson.
not
https://itunes.apple.com/search?term=jack+johnson
Notice the dot at the end
There is a huge explanation about why the dot is important quoting issues about DNS and character encoding but the truth is you probably do not care. Try adding the dot it might work for you too.
When I added the "." everything worked like a charm.
I hope it works for you too.
install:
npm i cors
Then include cors():
app.get("/list",cors(),(req,res) =>{
});
In addition to the Berke Kaan Cetinkaya's answer.
If you have control over your server, you can do the following in ExpressJs:
app.use(function(req, res, next) {
// update to match the domain you will make the request from
res.header("Access-Control-Allow-Origin", "YOUR-DOMAIN.TLD");
res.header("Access-Control-Allow-Methods", "GET,HEAD,OPTIONS,POST,PUT");
res.header("Access-Control-Allow-Headers", "Origin, X-Requested-With, Content-Type, Accept");
next();
});
https://enable-cors.org/server_expressjs.html
I tried this code,and that works for me.You can see the documentation in this link
var io = require("socket.io")(http, {
cors: {
origin: "*",
methods: ["GET", "POST"]
}
})
The reason that I came across this error was that I hadn't updated the path for different environments.
you have to customize security for your browser or allow permission through customizing security. (it is impractical for your local testing)
to know more about please go through the link.
https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS
These errors may be caused due to follow reasons, ensure the following steps are followed. To connect the local host with the local virtual machine(host). Here, I'am connecting http://localhost:3001/ to the http://abc.test Steps to be followed:
1.We have to allow CORS, placing Access-Control-Allow-Origin: in header of request
may not work. Install a google extension which enables a CORS request.*
2.Make sure the credentials you provide in the request are valid.
3.Make sure the vagrant has been provisioned. Try vagrant up --provision this make the localhost connect to db of the homestead.
Try changing the content type of the header. header:{ 'Content-Type' : 'application/x-www-form-urlencoded; charset=UTF-8;application/json' }
this point is very important.
Another solution to this problem in a specific scenario :
If
AWS APIGW is your backend with authentication enabled and
authentication fails,
your browser may end up complaining about CORS even if CORS is enabled in APIGW. You also need to enable CORS for 4XX as follows
API:YourAPI > Resources > /YourResource > Actions > Enable CORS > Gateway Responses for yourAPI check Default 4XX
Authentication will still fail but it won't look like CORS is the root cause
$.get('https://172.16.1.157:8002/firstcolumn/' + c1v + '/' + c1b, function (data) {
// some code...
});
Just put "https" .

Does python with wsgi (uwsgi) under nginx have some small default cache?

In my small web-site I feel need to make some data widely available, to avoid exchanging with database for every request made. E.g. this could be the list of current users show in the bottom of every page or the time of last update of ranking.
The stuff works in Python (Flask) running upon nginx + uwsgi (this docker image).
I wonder, do I have some small cache or shared memory for keeping such information "out of the box", or I need to take care of explicitly setting up some dedicated cache? Or perhaps some thing like this is provided by nginx?
alternatively I still can use database for it has its own cache I think, anyway
Sorry if question seems to be naive/silly - for I come from java world (where things a bit different as we serve all requests with one fat instance of java application) - and have some difficulty grasping what powers does wsgi/uwsgi provide. Thanks in advance!
Firstly, nginx has cache:
https://www.nginx.com/blog/nginx-caching-guide/
But for flask cacheing you also have options:
https://pythonhosted.org/Flask-Cache/
http://flask.pocoo.org/docs/1.0/patterns/caching/
Did you have a look at caching section from Flask docs?
It literally says:
Flask itself does not provide caching for you, but Werkzeug, one of the libraries it is based on, has some very basic cache support
You create a cache object once and keep it around, similar to how Flask objects are created. If you are using the development server you can create a SimpleCache object, that one is a simple cache that keeps the item stored in the memory of the Python interpreter:
from werkzeug.contrib.cache import SimpleCache
cache = SimpleCache()
-- UPDATE --
Or you could solve on the frontend side storing data in the web browser local storage.
If there's nothing in the local storage you call the DB, else you use the information from local storage rather than making db call.
Hope it helps.

How to configure NGINX to start a cacheRefresher program on server startup

Is there any way i can configure a python module to be invoked as part of the startup,when ever the NGINX server starts.
The role of the python module is to query database and cache it in memory database like redis. It also has to refresh the cache periodically every 5 mins for example.
I am using NGINX as reverse proxy server and uWSGI as the application server. The idea of caching is to reduce the application server response time to milliseconds from mins
Originally your question sounded like you an nginx caching proxy...
https://www.nginx.com/resources/wiki/start/topics/examples/reverseproxycachingexample/
Preloading an application cache like Redis is a different problem. What goes in the cache and in what format is hard to generalize ie you may want to cache html fragments and I may want to cache JSON data from an API, someone else may want to cache queries from a database.
Personally if you're serving requests and your applications uses the cache then use the application to preload the cache. A concrete example that I've used with Squid but this will also work with nginx.
Parse logs and gets stats for most used URL's. This is your heatmap.
Start Nginx.
Hit those URL's.
This loads the recently used items into the cache. I've also used the following method ie get id's from database and then hit the app... (forgive my python)
import grequests
base_url = "http://localhost:70/?id="
big_list = ['1', '2', '3', '4', '5']
for i in range(len(big_list)):
big_list[i] = base_url + big_list[i]
rs = (grequests.get(u) for u in big_list)
grequests.map(rs)
This loops over a list of id's and hits the URL. The app then caches everything it needs.
You can use any combination of the above ie I'd rather load only what's needed to avoid caching rubbish that's rarely requested.
The old answer follows...
Your question doesn't read very well but anytime I hear nginx and caching it's normally about how to configure it to cache pages. Some more details here...
https://serversforhackers.com/nginx-caching/
Have a look here for nginx caching with timeouts ie look for
http://nginx.org/en/docs/http/ngx_http_proxy_module.html
look for proxy_cache_valid on that page.

Django's urlresolvers.reverse isn't enough, how to get the rest?

I'm sending a callback URL to a remote widely API over which I have no control.
I've written my callback view and it's properly named (say, myapp_callback) in my urls.py, so all that I have to do is to call reverse('myapp_callback'), right? That's what it says in the manual.
Well, not so much. The result is /myapp/callback. Where's my protocol and hostname? The remote service I'm sending these API calls to has no idea. How can I detect it while maybe behind an Apache reverse proxy?
I'm working around this problem by putting the full URL into the settings file, but I'd love to provide a more "turnkey" solution.
Try out the request.build_absolute_uri(reverse('myapp_callback')).
Returns the absolute URI form of location. If no location is provided, the location will be set to request.get_full_path().
If the location is already an absolute URI, it will not be altered. Otherwise the absolute URI is built using the server variables available in this request.
Example: "http://example.com/music/bands/the_beatles/?print=true"

How to use cache-control with python in GAE?

I'm choosing whether to enable cache header and what difference it will make.
The current code is this and I wonder whether I should enable the caching and what it will do?
#seconds_valid = 8600
#self.response.headers['Cache-Control'] = "public, max-age=%d" % seconds_valid
self.response.headers['Cache-Control'] = 'no-cache'
Can I test what the difference is if I change the code to this
seconds_valid = 8600
self.response.headers['Cache-Control'] = "public, max-age=%d" % seconds_valid
Am I doing it the right way? What exactly is going to happen when I enable the cache? Will I still be able to update the page?
Thank you
There is also AppEngine's caching reverse proxy / edge cache which may pick up your Cache-Control header if given a max-age and set to public like in your example. The edge cache is "best effort", meaning it is not 100% certain it will cache your response.
More information can be found here and here.
Setting Cache-Control will make no difference to your application. This value is only used by web browser, caching is done only on client side, not on the server. Correct values for Cache-Control can reduce your server loads and save bandwidth because user agents will try to cache content locally but it has nothing to do with appengine.
If you are looking for server-side caching to improve response time and decrease database reads have a look at memcached. To use memcache optimally you might also need to search the internet for cache-invalidation strategies.

Categories