Degraded Performance: Mobaro Backend, Mobile App, and RideOps Application

Incident Report for Mobaro

Postmortem

Summary

Today at approximately 08:45 UTC, our system experienced performance degradation due to an automated database backup procedure. The issue has been identified and resolved, with additional measures being implemented to prevent recurrence.

Incident Details

A full database cluster backup was initiated at 08:45 UTC as part of our regular maintenance schedule. While normally routine, these full backups have been causing cluster instability since a database update was applied on March 10th.

Due to previous incidents, we had temporarily paused the full backup procedure while our database vendor developed a solution. Our system continued to perform frequent incremental backups during this time, maintaining data security without compromising system performance.

Yesterday, however, an internal reassignment within our cluster logic inadvertently reactivated the full backup task, resulting in the system instability experienced today.

Resolution

Our team has restored system stability by halting the problematic backup process. Additionally, our database vendor will be applying a permanent patch later today to address the underlying issue.

Moving Forward

Our technical team will be monitoring the system closely following the vendor's patch deployment. We remain committed to maintaining the highest levels of service reliability and data security.

We sincerely apologize for any inconvenience this disruption may have caused and appreciate your understanding.

For any specific concerns or questions, please contact our support team at support@mobaro.com.

Posted Mar 18, 2025 - 20:14 CET

Resolved

The result backlog has now been processed, and all service are back to normal operation.

Thank you for your patience and understanding. If you have any further questions, please contact our support team via support@mobaro.com.
We apologize for any inconvenience this may have caused and will be looking into how we can prevent this in the future.
Posted Mar 18, 2025 - 11:46 CET

Monitoring

The issue has been resolved but we are currently working through a backlog of queued checklist results.
Delays might still occur when handing in checklist results until the backlog has been fully processed.

We will post an update when the backlog has been emptied after which all services should be fully operational.
Posted Mar 18, 2025 - 11:15 CET

Identified

We are working in collaboration with our database provider to mitigate this issue.
Posted Mar 18, 2025 - 10:33 CET

Investigating

We are experiencing issues with our database provider causing performance degradation across most of the platform.
We are working in collaboration with our database provider to mitigate this issue.
Posted Mar 18, 2025 - 10:29 CET
This incident affected: Mobile App, Backend Application, RideOps, and Public API.