Resolved -
The metrics indicate no further issues; for now, the situation appears stable with the applied workaround. In the coming days, however, we will implement additional updates.
Apr 24, 20:14 UTC
Monitoring -
As a workaround, we applied a Pod Antiaffinity Policy, which avoids server-side Generated Delta pods to be scheduled on the nodes with others worker pods. This allows them to have more storage available
Apr 24, 16:43 UTC
Identified -
The issue has been identified: the worker Kubernetes pods that are running the server-side Generated Delta, in some cases get ephemeral storage exhaustion and the job won't run successfully. A fix is being implemented.
Apr 24, 16:24 UTC
Investigating -
Some of the server-side Generated Delta, which is a beta feature, are not working for some customers in the European cluster. We're investigating the issue.
Apr 24, 15:16 UTC