One of the SQL servers reported high CPU utilization during the evening/night of November 5th. This affected a number of tenants currently residing on that server, in the form of longer response times for different API endpoints. Checking DPA suggested a query most likely to be the culprit. The issue resolved itself after some time, most likely due a decrease in the number of API calls. The next day, a missing database index was identified as the probable root cause.
All tenants on the affected SQL would in theory have experienced decreased performance for all operations toward the database, including API calls, messages, automations. However, since the incident happened when there was low activity in the system, this didn’t result in any major problems for the end users.
An increase in queries against the Contact table, using a where-clause which was not included in any index. A missing index on the affected table was identified and added.
We will look into how much the database performance has been improved by the added index as well as investigating if there is a need for creating additional indexes. The API usage for tenants on the affected server will be evaluated to see if there is an inefficient implementation causing unnecessary load.