Field Note · Small business · Sri Lanka · 2023 · Case study
Turning a Production Data Loss Incident into a Practical Backup Strategy
An online business directory stored thousands of company listings and uploaded logo files. During routine development work, a production uploads directory was accidentally deleted.
The incident exposed a serious business risk. The platform depended on data that could not simply be recreated from the source-code repository, yet no documented backup policy protected it.
Identify what cannot be rebuilt
The system contained approximately 20 GB of data. Backing up every application file in the same way would have added cost and complexity without providing equal value.
We separated recoverable assets from irreplaceable business data. Application code and related files could be restored from version history. The database and customer-uploaded files changed continuously and therefore required regular, independent backups.
This distinction gave the business a clear answer to a basic recovery question: what would we be unable to recreate if the production server disappeared today?
Design the policy before the scripts
We documented a backup architecture using Amazon S3, with older recovery points moving to S3 Glacier storage classes for longer-term retention.
The policy defined:
- Daily, weekly, monthly, and yearly recovery points
- How long each backup should be retained
- When expired backups should be removed
- When backups should run
- Which database and uploaded-file data should be included
- How a rollback or restoration would begin
Expected monthly and annual storage costs were calculated using the backup size, frequency, and retention plan. The architecture, assumptions, and estimates were shared with the client for feedback and approval.
Implement and observe
After approval, shell scripts were developed to create and transfer the backups. They were tested before being deployed to production.
The process was observed closely for the first month. The client was also asked to compare the actual S3 charges with the estimate so that unexpected storage growth or lifecycle costs could be identified early.
The control we missed
The first implementation still had an important weakness: it did not send an alert when a scheduled backup failed.
Manual monitoring during the first month provided some visibility, but it was not a dependable long-term control. A failed script could go unnoticed and create a false sense of protection.
The corrective lesson was clear. Backup monitoring must notify a responsible person when a run fails or does not occur. Periodic restore tests are also needed to prove that stored data can actually be recovered within an acceptable period.
The practical lesson is that backup is a complete operating process—not simply a copy command. Identify the data the business cannot recreate, define its retention and cost, automate the work, monitor every run, and regularly prove that restoration works.