Partial outage

Incident Report for Archbee

Postmortem

We use a hosted/managed database service from Azure.

Because we are careful with customer data, and want to fulfill our end to end encryption promise, we use SSL when connecting to any database, this one included.

A few weeks ago, we saw a notification about a SSL certificate change in our of our test databases, and we made the SSL certificate change for that specific database.

Because we were 100% sure that the same thing will happen soon with our main database, we made a change preemptively while making sure multiple times that the change is backwards compatible, using both certificates in the certificate chain.

After deploying the change, our systems went on undisturbed for over two weeks.

Today, while having a planned database update from Azure, the system suddenly stopped working due to the new change we had made.

Posted Apr 20, 2025 - 17:52 EEST

Resolved

We have resolved the issue. We are sorry for the inconvenience and we're taking steps to prevent this from ever happening again. We are writing a postmortem and releasing the notes soon.
Posted Apr 20, 2025 - 17:47 EEST

Update

We have determined the root cause of the issue and we're deploying a fix.
Posted Apr 20, 2025 - 16:18 EEST

Investigating

We are currently investigating the issue with EU servers and potentially system wide.
Posted Apr 20, 2025 - 15:36 EEST
This incident affected: API, Content Management System, and Hosted Documentation Sites.