I was in the ordering team of Ula and was on-call when this incident happened. The warehouse teams used to generate daily order reports with crucial item-level details, including shipping locations. On this day, we received back-to-back oncalls from all the warehouses reporting missing orders in the reports, blocking the warehouse team from processing those orders and halting all downstream operations.
Root Cause and Fixes:
We unblocked the warehouse teams as soon as we could. After several hours of digging, we finally figured out the root cause: perfectly camouflaged whitespaces.
Years ago, serviceable locations were manually added to the location database via file uploads. Several locations in these files had trailing whitespaces, which went unhandled. One of my colleagues had noticed this and notified the senior engineering leadership, leading to the removal of these whitespaces after discussions.
This change was communicated to the other teams, but unfortunately, the ordering team wasn't aware of it. So, while the whitespaces in the locations in the location database (MySQL) were removed by the location team, BigQuery order data still had the locations with whitespaces.
Order report generation involved fetching the locations from the location database and adding them to the BigQuery query. Unlike MySQL, BigQuery does not ignore whitespaces when querying for data, and hence, none of the orders belonging to the updated locations were fetched on query execution. We ended up updating the BigQuery data and also trimmed all the strings we were dealing with.
There is a saying in a popular Malayalam movie, which roughly translates to: "When a bomb is dropped in America, the blast will be felt in India". And that is what happened here. A bug fix in the location service broke the order service. All due to whitespaces.