At around 18.52 on Sunday evening we received a handful of reports of users being unable to open the various web apps, including Manager and Communicator, if not already loaded. After some investigation we discovered the Branding API, from which the customer-facing web apps fetch their branding, colours, fonts etc, was taking too long to respond in some cases, causing the requests from affected the users’ browsers to time out. By 19.48 the affected worker nodes had been identified and removed, restoring full service.
Since the Branding API is an ancillary component used only by the various web apps for the purpose of displaying the appropriate partner brand, the core platform, calls, and all other services remained unaffected throughout. Further, where the page was already loaded, or users were hitting unaffected nodes, users were similarly unaffected.
This incident has enabled us to identify the Branding API as a point of failure within the web apps, and we have been working since to eliminate any dependency on it in order for them to load. We expect these changes to be rolled out by the end of March and while incidents like this are extremely rare, the changes will ensure this particular one cannot reoccur.