Server-side generated delta (beta) are failing randomly

Incident Report for Hosted Mender

Resolved

The metrics indicate no further issues; for now, the situation appears stable with the applied workaround. In the coming days, however, we will implement additional updates.

Posted Apr 24, 2024 - 20:14 UTC

Monitoring

As a workaround, we applied a Pod Antiaffinity Policy, which avoids server-side Generated Delta pods to be scheduled on the nodes with others worker pods. This allows them to have more storage available

Posted Apr 24, 2024 - 16:43 UTC

Identified

The issue has been identified: the worker Kubernetes pods that are running the server-side Generated Delta, in some cases get ephemeral storage exhaustion and the job won't run successfully. A fix is being implemented.

Posted Apr 24, 2024 - 16:24 UTC

Investigating

Some of the server-side Generated Delta, which is a beta feature, are not working for some customers in the European cluster. We're investigating the issue.

Posted Apr 24, 2024 - 15:16 UTC

This incident affected: Hosted Mender EU.