We have a Python Linux azure function that is connected to a custom oidc provider and azure ad to provide authentication to the HTTP triggered functions using Microsofts easyauth.
After the initial setup, the azure function was working and has been working for the last few months.
In the last 2 days, our application suddenly started to error out on our custom provider, the azure ad authentication is still working, after checking the easyauth logs, we see the error
System.PlatformNotSupportedException: Windows Cryptography Next Generation (CNG) is not supported on this platform.
No changes were made on either the custom oidc provider or the azure function in the last 2 days.
We suspect that maybe the base easyauth docker image (mcr.microsoft.com/appsvc/middleware:stage2) got updated and that broke the authentication.
Any ideas or suggestions on possible fixes or even related problems?
Could it be due to this: https://github.com/Azure/app-service-announcements/issues/404
Use RSACNG when validating tokens to add PS256 support
EDIT: Also experiencing this issue as of this morning. I'm currently trying to manually downgrade the version using this command az webapp auth update --name xxx --resource-group xxx --runtime-version "1.5.1" but my Azure credentials don't have enough power to run that so I can't validate if it works or not.
EDIT2: Doesn't work if you are using auth v2.
EDIT3: It actually does work if you are using auth v2. You just have to check the help options of the command to realize that for auth v2 you have to install a CLI extension with command az extension add --name authV2. After that you can run the commands. I downgraded the version to 1.5.1 but nothing changed. I'm not sure if it has something to do with the fact that we are deploying to a slot first which probably had the new version still. I have also created an Azure support ticket about this.
EDIT4: Got in to a support call with Azure yesterday. They fixed the issue during the night. A restart of the application is required. I'm still baffled by the fact that the documentation shows that you can pinpoint the version of Easy Auth / Authentication/Authorization middleware but when I go to troubleshoot my AppService and select Easy Auth it actually shows that the pinpointed version is 1.5.1 and the running version is 1.6.2. So it just totally ignores the whole configuration. Fun, right?
we have started to see this as well on some of our instances, the worrying thing is that we have multiple running instances and it is working in some and not in some.
we "solved" the issue on one production instance by redeploying the function app, it is setup through terraform and a destroy of the function app and then a create made it work again.
Exact same issue there.
2 app services (one for prod and one for dev located in France central region) using an Azure AD app in an other Azure B2C tenant for authentication (https://learn.microsoft.com/en-us/azure/app-service/configure-authentication-provider-aad#-option-2-use-an-existing-registration-created-separately) were working for about 1 year.
Then after the deployment of a new container version of our app in the "dev" app service, the authentication broke en DEV only and we started receiving ERROR 500 message when we are being redirected to the /.auth/login/aad/callback endpoint after the authentication is done in Azure B2C.
By inspecting the app service log we have these logs :
2022-11-08T08:47:28.449645417Z [41m[30mfail[39m[22m[49m: Microsoft.AspNetCore.Server.Kestrel[13]
**2022-11-08T08:47:28.449692217Z Connection id "0HMM1CIPP8I5M", Request id "0HMM1CIPP8I5M:00000004": An unhandled exception was thrown by the application**.
2022-11-08T08:47:28.450647224Z System.PlatformNotSupportedException: Windows Cryptography Next Generation (CNG) is not supported on this platform.
2022-11-08T08:47:28.451187128Z at System.Security.Cryptography.RSACng..ctor()
2022-11-08T08:47:28.451205328Z at Microsoft.Azure.AppService.Middleware.JsonWebKey.GetSecurityKeys() in /EasyAuth/Microsoft.Azure.AppService.Middleware.Modules/JsonWebKey.cs:line 100
2022-11-08T08:47:28.451422129Z at Microsoft.Azure.AppService.Middleware.OpenIdConnectConfiguration.GetJwtValidationParameters(String siteName, String clientId, String authenticationType, String allowedAudiences) in /EasyAuth/Microsoft.Azure.AppService.Middleware.Modules/OpenIdConnectConfiguration.cs:line 114
2022-11-08T08:47:28.457668471Z at Microsoft.Azure.AppService.Middleware.AzureActiveDirectoryProvider.GetOpenIdConnectValidationParameters(ConfigManager oidcConfigManager, Boolean forceRefresh) in /EasyAuth/Microsoft.Azure.AppService.Middleware.Modules/IdentityProviders/AzureActiveDirectoryProvider.cs:line 1131
2022-11-08T08:47:28.457685071Z at Microsoft.Azure.AppService.Middleware.AzureActiveDirectoryProvider.HandleServerDirectedLoginAsync(HttpContextBase context) in /EasyAuth/Microsoft.Azure.AppService.Middleware.Modules/IdentityProviders/AzureActiveDirectoryProvider.cs:line 518
2022-11-08T08:47:28.457689872Z at Microsoft.Azure.AppService.Middleware.IdentityProviderBase.OnCompleteServerDirectedLoginAsync(HttpContextBase context) in /EasyAuth/Microsoft.Azure.AppService.Middleware.Modules/IdentityProviders/IdentityProviderBase.cs:line 655
2022-11-08T08:47:28.457693772Z at Microsoft.Azure.AppService.Middleware.IdentityProviderBase.TryHandleProtocolRequestAsync(HttpContextBase context) in /EasyAuth/Microsoft.Azure.AppService.Middleware.Modules/IdentityProviders/IdentityProviderBase.cs:line 185
2022-11-08T08:47:28.457697572Z at Microsoft.Azure.AppService.Middleware.EasyAuthModule.OnBeginRequestAsync(HttpContextBase context) in /EasyAuth/Microsoft.Azure.AppService.Middleware.Modules/EasyAuthModule.cs:line 220
2022-11-08T08:47:28.457818072Z at Microsoft.Azure.AppService.Middleware.NetCore.AppServiceMiddleware.InvokeAsync(HttpContext context) in /EasyAuth/Microsoft.Azure.AppService.Middleware.NetCore/AppServiceMiddleware.cs:line 102
2022-11-08T08:47:28.457928173Z at Microsoft.Azure.AppService.MiddlewareShim.AutoHealing.AutoHealingMiddleware.Invoke(HttpContext context) in /EasyAuth/Middleware.Host/AutoHealing/AutoHealingMiddleware.cs:line 55
2022-11-08T08:47:28.457939473Z at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)
Creating a new app in an other app service plan did not improve the situation so we have opened a support ticket/case at Microsoft. This issue has nothing to do with our application.This issue is 100% related to a change that might happened at Microsoft.
Let's keep in touch on this thread to share knowledge about this issue.
issue is solved after restarting the azure app services
Related
I am using supertokens to build an authentication system and using fastApi as backend but while using their prebuild UI and already setup backend code in python, I am not able to access the 127.0.0.0:3000/docs endpoint. It is showing only a blank page.
Also, the custom routes that I have built in my API are not working and accessible.
Here is the code that I have written
#app.get("/sessioninfo")
async def secure_api(s: SessionContainer = Depends(verify_session())):
return {
"sessionHandle": s.get_handle(),
"userId": s.get_user_id(),
"accessTokenPayload": s.get_access_token_payload(),
}
Here is the app_info part of init function in supertokens
app_info = InputAppInfo(
app_name="demoApp",
api_domain="http://localhost:3001",
website_domain="http://localhost:3000",
)
After hitting the API with port 3000 and endpoint /sessioninfo I am getting blankpage
localhost:3000/session_info
And for localhost:3001/session_info I am getting an internal server error.
localhost:3001/session_info
localhost:3000/session_info won't respond with session info because it's being handled by the front end.
But localhost:3001/session_info should work. This error is likely to be happening because of python/dependency versions.
Update: As per this GH comment, this error is likely to be coming from Python 3.11. So maybe try a different version of python
I hope this helps :)
In short:
I have a Django application being served up by Apache on a Google Compute Engine VM.
I want to access a secret from Google Secret Manager in my Python code (when the Django app is initialising).
When I do 'python manage.py runserver', the secret is successfully retrieved. However, when I get Apache to run my application, it hangs when it sends a request to the secret manager.
Too much detail:
I followed the answer to this question GCP VM Instance is not able to access secrets from Secret Manager despite of appropriate Roles. I have created a service account (not the default), and have given it the 'cloud-platform' scope. I also gave it the 'Secret Manager Admin' role in the web console.
After initially running into trouble, I downloaded the a json key for the service account from the web console, and set the GOOGLE_APPLICATION_CREDENTIALS env-var to point to it.
When I run the django server directly on the VM, everything works fine. When I let Apache run the application, I can see from the logs that the service account credential json is loaded successfully.
However, when I make my first API call, via google.cloud.secretmanager.SecretManagerServiceClient.list_secret_versions , the application hangs. I don't even get a 500 error in my browser, just an eternal loading icon. I traced the execution as far as:
grpc._channel._UnaryUnaryMultiCallable._blocking, line 926 : 'call = self._channel.segregated_call(...'
It never gets past that line. I couldn't figure out where that call goes so I couldnt inspect it any further than that.
Thoughts
I don't understand GCP service accounts / API access very well. I can't understand why this difference is occurring between the django dev server and apache, given that they're both using the same service account credentials from json. I'm also surprised that the application just hangs in the google library rather than throwing an exception. There's even a timeout option when sending a request, but changing this doesn't make any difference.
I wonder if it's somehow related to the fact that I'm running the django server under my own account, but apache is using whatever user account it uses?
Update
I tried changing the user/group that apache runs as to match my own. No change.
I enabled logging for gRPC itself. There is a clear difference between when I run with apache vs the django dev server.
On Django:
secure_channel_create.cc:178] grpc_secure_channel_create(creds=0x17cfda0, target=secretmanager.googleapis.com:443, args=0x7fe254620f20, reserved=(nil))
init.cc:167] grpc_init(void)
client_channel.cc:1099] chand=0x2299b88: creating client_channel for channel stack 0x2299b18
...
timer_manager.cc:188] sleep for a 1001 milliseconds
...
client_channel.cc:1879] chand=0x2299b88 calld=0x229e440: created call
...
call.cc:1980] grpc_call_start_batch(call=0x229daa0, ops=0x20cfe70, nops=6, tag=0x7fe25463c680, reserved=(nil))
call.cc:1573] ops[0]: SEND_INITIAL_METADATA...
call.cc:1573] ops[1]: SEND_MESSAGE ptr=0x21f7a20
...
So, a channel is created, then a call is created, and then we see gRPC start to execute the operations for that call (as far as I read it).
On Apache:
secure_channel_create.cc:178] grpc_secure_channel_create(creds=0x7fd5bc850f70, target=secretmanager.googleapis.com:443, args=0x7fd583065c50, reserved=(nil))
init.cc:167] grpc_init(void)
client_channel.cc:1099] chand=0x7fd5bca91bb8: creating client_channel for channel stack 0x7fd5bca91b48
...
timer_manager.cc:188] sleep for a 1001 milliseconds
...
timer_manager.cc:188] sleep for a 1001 milliseconds
...
So, we a channel is created... and then nothing. No call, no operations. So the python code is sitting there waiting for gRPC to make this call, which it never does.
The problem appears to be that the forking behaviour of Apache breaks gRPC somehow. I couldn't nail down the precise cause, but after I began to suspect that forking was the issue, I found this old gRPC issue that indicates that forking is a bit of a tricky area.
I tried to reconfigure Apache to use a different 'Multi-processing Module', but as my experience in this is limited, I couldn't get gRPC to work under any of them.
In the end, I switched to using nginx/uwsgi instead of Apache/mod_wsgi, and I did not have the same issue. If you're trying to solve a problem like this and you have to use Apache, I'd advice further investigating Apache forking, how gRPC handles forking, and the different MPMs available for Apache.
I'm facing a similar issue. When running my Flask Application with eventlet==0.33.0 and gunicorn https://github.com/benoitc/gunicorn/archive/ff58e0c6da83d5520916bc4cc109a529258d76e1.zip#egg=gunicorn==20.1.0. When calling secret_client.access_secret_version it hangs forever.
It used to work fine with an older eventlet version, but we needed to upgrade to the latest version of eventlet due to security reasons.
I experienced a similar issue and I was able to solve with the following:
import grpc.experimental.gevent as grpc_gevent
from gevent import monkey
from google.cloud import secretmanager
monkey.patch_all()
grpc_gevent.init_gevent()
client = secretmanager.SecretManagerServiceClient()
I am getting the error while deploying the Azure function from the local system.
I wen through some blogs and it is stating that my function is unable to connect with the Azure storage account which has the functions meta data.
Also, The function on the portal is showing the error as: Azure Functions runtime is unreachable
Earlier my function was running but after integrating the function with a Azure premium App service plan it has stooped working. My assumption is that my app service plan having some restriction for the inbound/outbound traffic rule and Due to this it is unable to establish the connection with the function's associated storage account.
Also, I would like to highlight that if a function is using the premium plan then we have to add few other configuration properties.
WEBSITE_CONTENTAZUREFILECONNECTIONSTRING = "DefaultEndpointsProtocol=https;AccountName=blob_container_storage_acc;AccountKey=dummy_value==;EndpointSuffix=core.windows.net"
WEBSITE_CONTENTSHARE = "my-function-name"
For the WEBSITE_CONTENTSHARE property I have added the function app name but I am not sure with the value.
Following is the Microsoft document reference for the function properties
Microsoft Function configuration properties link
Can you please help me to resolve the issue.
Note: I am using python for the Azure functions.
I have created a new function app with Premium plan and selected the interpreter as Python. When we select Python, OS will be automatically Linux.
Below is the message we get to create functions for Premium plan function App:
Your app is currently in read only mode because Elastic Premium on Linux requires running from a package.
PortalScreenshot
We need to create, deploy and run function apps from a package, refer to the documentation on how we can run functions from package.
Documentation
Make sure to add all your local.settings.json configurations to Application Settings in function app.
Not sure of what kind of Azure Function you are using but usually when there is a Storage Account associated, we need to specify the AzureWebJobsStorage field in the serviceDependencies.json file inside Properties folder. And when I had faced the same error, the cause was that while publishing the azure function from local, some settings from the local.settings.json were missing in the Application Settings of the app service under Configuration blade.
There can be few more things which you can recheck:
Does the storage account you are trying to use existing still or is deleted by any chance.
While publishing the application from local, using the web deploy method, the publish profile is correct or has any issues.
Disabling the function app and then stopping the app service before redeploying it.
Hope any of the above mentions help you diagnose and solve the issue.
The thing is that there is a difference in how the function deployed using Consumption vs Premium service plan.
Consumption - working out of the box.
Premium - need to add the WEBSITE_RUN_FROM_PACKAGE = 1 in the function Application settings. (see https://learn.microsoft.com/en-us/azure/azure-functions/run-functions-from-deployment-package for full details)
I've set up a Google Cloud BigTable cluster on my project. The main codebase for the project runs within a standard Python App Engine environment, which can't use the gcloud-python library because of the reliance on grpcio. To get around this, I've set up a Python App Engine Flexible Environment service within the same project and written a very simple Flask server to run on it, which I can then hit from my standard environment. The code looks something like this:
from gcloud import bigtable
app = Flask(__name__)
client = bigtable.Client(project=bigtable_config.PROJECT_ID, read_only=True)
cluster = client.cluster(bigtable_config.ZONE_ID, bigtable_config.CLUSTER_ID)
table = cluster.table(bigtable_config.TABLE_ID)
#app.route("/query/<start_key>/<end_key>")
def run_query(start_key, end_key):
if not client.is_started():
client.start()
row_data = table.read_rows(start_key=start_key, end_key=end_key)
row_data.consume_all()
// do some stuff to the row data here, get results
return jsonify(results)
I can run this code locally and it works great. I can deploy it to my service and it continues to work great. However, if the service sits idle for some period of time (I've typically noticed it after about an hour), then every request I make starts failing with this error:
NetworkError(code=StatusCode.UNAUTHENTICATED, details="Request had invalid authentication credentials.")
If I redeploy the service, it starts working again. I do not observe this behavior when I'm running the service locally.
What am I doing wrong? I'm assuming I'm making some mistake in my setup of the client, where it's not properly using the app engine credentials. Do I need to kill the client and restart it when I encounter this error?
This issue is being tracked at github.
I am trying to develop a Facebook application using PyFacebook (hosted on Google App Engine). It's an FBML application (runs in a Facebook canvas instead of an iframe). I'm having problems getting any API calls to function. The sequence looks like this:
fb = facebook.Faceboook(api_key, secret_key)
fb.session_key = cherrypy.request.params['fb_sig_session_key']
fb.uid =cherrypy.request.params['fb_sig_user']
Then if I try, for example:
user = fb.users.getLoggedInUser()
This will fail with:
Error 104: Incorrect signature
I'm not sure where I'm going wrong. I've verified that the api_key and secret_key are correct (many, many times). The application has been added to my account. I've even been able to get a desktop application working; my problem is only with this web-hosted version.
Thanks for your help!
Deleting the Facebook application and re-creating it, then configuring my code with the new API key and secret key, resolved this problem.