Post Mortem 3.63.1
Description of the Security misconfiguration

In release 3.51.0 the Workspace package (now WebApp) the "Active directory" settings page was replaced by the "User provisioning" settings page. The reason behind this change was the release of SCIM API support, i.e. instead of only being able to configure "(Azure) Active Directory sync" it became possible to instead configure SCIM. In this transition, the admin authorization policy was not applied to the new endpoints of/used by this page. This policy denies endpoint access to environment users who are not assigned the admin role. When such a user tries to utilize an endpoint with this policy configured, the user will receive a 403 (Forbidden) response instead of the normal response. One thing to note here is that the Settings sidebar, listing the available pages to the user, only ever listed the "User provisioning" option to environment admin users. This means that if an environment user would want to access the "User provisioning" settings page, the user would have to be aware of (or guess) the URL of this page. Please do note that the other authorization policies were correctly configured, i.e. only an active, none deleted, authenticated environment user could abuse the absence of the admin authorization policy and access these endpoints.

Timeline

Release 3.51.0 (12-08-2022 - 01-09-2022)

- The developers working on this change forgot to apply the admin authorization policy to the endpoints (this is an attribute on the group of endpoints)

- The developer doing the Code Review did not notice the absence of the admin authorization policy attribute

- The quality assurance engineer did not notice that an environment user without admin role could access the settings page (it was not visible in the Settings Sidebar, nor was it part of our testing checklist)

11-07-2023 (times are in CET)

[09:00] A quality assurance engineer noticed that an environment user without admin role could access the "User provisioning" settings page and registered a bug on our backlog, and the initial priority was assigned to be fixed for Release 3.64.0 (the release we would start preparing 2 days later)

[09:39] A developer noticed the newly registered bug on the backlog and discussed it with the Product Owner, resulting in the priority being upgrade to Hotfix (3.63.1)

[09:53] The code change containing the missing admin authorization policy attribute was submitted for Code Review and approved

[10:00] The ISO team was notified of the Security vulnerability and the impact/scope

[10:05] Emergency CAB (ECAB) is arranged by the ISO team and kept informed by development.

[10:40] A quality assurance engineer tested and approved the code change

[10:57] Release 3.63.1 was created and informed support to make that the download was available for all partners. All the partners were also informed via the announcements from the Support Portal.

[11:21] The code changes were back ported to 3.62 (release 3.62.1)

[13:51] Download available for the self-hosted partners. Communication via the support portal and informed the self-hosted partners via a separate email which was sent by the change manager.

[22:00] The Operations team deployed release 3.63.1 on our hosted platform

21-07-2023

- The Post Mortem blog post was updated with some more information and advice to regenerate API key/token

02-08-2023

Things we are changing

- We are extending our endpoint (route) tests for the WebApp to include the configured authorization policies. This means that instead of only having the policy defined in a single place, it is now in 2 places. This should make it easier for developers to detect incorrect configuration, both during development as during Code Review.

- We are including the checking of hidden pages (e.g. when not available in the Settings Sidebar) for the WebApp to our quality assurance testing checklist

- We are extending our automated UI tests for the WebApp with the checking of admin authorization policy for settings pages

- Process optimisation, lessons learned, within the ISO process