Everything has been working correctly since last update. System is back to its nominal state. We will continue monitoring until incident is officially closed upstream.

Issue seems fixed upstream. We are gradually re-enabling object storage in API and workers, while monitoring the situation. Webhook delivery is still stable.

Fix was deployed on PG-based workers a few minutes ago: webhook delivery has recovered and queue is now empty.

Our PG-based workers are having issues delivering webhooks. Issue was identified and a fix is being written. Pulsar-based workers are not affected.

Our provider acknowledged the issue: https://www.clevercloudstatus.com/incidents/1086

We temporarily disabled writing incoming events to object storage; they will be written to database instead. This should improve event ingestion.

We are seing transient issues with our object storage. Object storage is used to store incoming event payloads and responses to outgoing webhooks. Webhook delivery is still working, but in a degraded state.