OAuth2 Token Generation issue at High Traffic Time and Solutions
Problem Overview
In client-server environments with low TPS (Transactions Per Second), resource invocation is like that,
According to the above diagram, after the expiration time of the access token, we have to call server-side 3 times to get the expected response which means single invocation takes 3 times of normal invocation time. By maintaining the expiration time in client-side and checking it before invoking the resource, invocation time can be reduced to 2 times of normal invocation time approximately. This mechanism is fine for low TPS client-server environment.
If we have a production-grade middleware environment having a high amount of traffic and handling transactions like payments, we are in a problem. Let’s see them one by one,
Let’s think there are setup timeouts at the client-side,
At the expiration time, the transaction journey time will double and there is a possibility to timeout the particular transaction. Also at high TPS time, multiple transactions are going to refresh the token parallelly and all token generation requests will fail except one. Hence hundreds of transactions may fail due to client-side timeout. At the token generation time, if a single transaction generates the token while others are waiting for the token, then also hundreds of transactions will fail due to client-side timeout.
If the client-side timeouts are ignored, token generation failures due to parallel invocations, performance impacts since waiting for token generation by a single transaction, can be encountered.
If we think that the expiration time of the access token is 1 hour, then hundreds of transactions will fail hourly.
So like that above failures and performance impacts may lead to huge revenue losses.
My opinion is, OAuth2 must give some extra time like 10 seconds for old access token even after generate new access token from the particular refresh token.
Possible Solutions
There are a couple of solutions to mitigate the raised issue,
Token Pool Mechanism
According to this solution, multiple applications must have been given and multiple tokens(2 or more) must be maintained in the token pool (one token per application). In here, when a token is going to expire, remove the token from the pool and refresh the token before actual expire time (before 5-10 min). Hence requests always use valid tokens to invoke the backend resources and also nothing to worry about network delay of the token generation since requests use different application token while new access token for the application that has access token which is going to expire is being generated.
Problems with Token Pool Mechanism
Having multiple tokens per application is kind of a security vulnerability. However, the system maintaining multiple application that having a single token. But it’s also kind of a security vulnerability since multiple tokens are used to invoke the same resources by the same environment.
I came up with the below solution due to the above mention vulnerabilities. I called it Grant Type Mechanism.
Grant Type Mechanism
This solution is kind of a bit tricky solution. According to my perspective, this solution addresses all things.
Normally OAuth2 supports password, refresh_token and client_credentials grant types. So access token can be generated using the password and refresh token grant types and also using the client credentials grant type. In here, password and refresh token together generate an access token with an expiration time and client credentials grant type generates different access token without revoking the previously generated access tokens. So now multiple access tokens (2 access tokens) are maintained per application without having multiple applications.
In here also like token pool, there is a security vulnerability since having multiple tokens per application. To avoid this vulnerability,
When an access token is going to expire, another access token can be generated using the client credentials grant type and maintaining kind of a flag to indicate the currently using access token, switch the flag values of generated access token with the access token that is going to expire. After that new access token can be generated using refresh token grant type (if it’s not working, password grant type can be used) while requests are using the access token generated from client credential grant type. Once the new access token is generated from the refresh token grat type, we can switch the flag values of access tokens again and revoke the access token which was generated using client credentials. (or using client credential grant type, the access token can be generated with custom validity time then nothing to revoke the access token)
When this mechanism is used, backend side access token count will be doubled and end of the day, invalid access token count will also be doubled. If the backend deals with a high amount of transactions, invalid tokens must be cleaned early since the allocated database size will exceed.
Moreover, to improve the performance of the environment, this token generation process must be separated from the transaction process. In here, a dynamic scheduler can be used to manage the above-mentioned solution. By using the dynamic scheduler, the next token generation time can be decided using the current expiration time of the generated access token. Also by maintaining a multi-value flag for access tokens, this dynamic scheduler can detect backend token failures and notify the transactions before invoking the backend resources.
Like that transactions don’t want to worry about tokens, they only have to care about the flag of the token.
Conclusion
This solution improves the performance, zero token failures mostly, zero timeouts mostly (assume that no backend or no dynamic scheduler failures), release the additional load of the transactions.
Currently, I have implemented a draft solution by integrating Token Pool Mechanism and Grant Type Mechanism together. If backend-side has no concerns about having multiple applications, Token Pool Mechanism can be used, else Grant Type Mechanism can be used. So I called it the Token Management System. I have triggered some scenarios on this solution and able to get results as expected. Next time I’ll come up with a full implemented solution and results.
