Home » Backups

Category: Backups

Keep your site up to date – good advice, but only in theory?

“Keep your software up to date!”

I am sure you have heard this saying many times, and in general, it is good advice.

However, let’s take a WordPress website, for example.

A WordPress site is made from the WordPress core and usually many plugins. You have many pieces in your puzzle. Not all of them are updated at the same time or in the same way.

In my experience, it happened more than once that an updated piece no longer fit with the rest of the puzzle. Auch!

Most update processes do a good job warning you that you need to do a backup first and to ensure the other plugins (components of the puzzle) are compatible with the update.

This approach puts the responsibility of “making sure that things still work” with the user. And not everyone can make that assessment. Also, let’s admit it, sometimes we are in a hurry or just plain lazy :). Ideally, a piece of software should not rely on a human to do the right thing.

I too used to be overconfident in the automatic updates process, and I would apply those every time there was a new update. Click, click, and I was done! What could possibly go wrong?

One time, I updated the store. We had a newsletter scheduled that we expected to generate much interest and I thought I wanted to offer the best and latest shopping experience for our customers. So I updated the store and was on my way. The next day I opened the email to a ton of complaints from our subscribers that the checkout is not working! A full email campaign wasted, not the mention that we looked totally unprofessional — that hurt both my ego and the sales.

What did I learn from it?

1. Updates can potentially be very painful

2. You never change the system right before a big promo campaign. (This feels like common sense now.)

3. You need to be extra careful when you update the part that generates income: the store, the “pay now buttons,” and the subscribe boxes.

4. After you update you need to test at least the critical functionality: add to cart, checkout, subscribe.

5. You better have good backups, in case you need to roll back.

It was not all bad, because I did have backups, so it was relatively easy to go back to the previous version. Moreover, I sent another email campaign with an apology and a second invitation to check out the offer. If I hadn’t had backups that would not have been possible to do.

Another thing I decided to implement is to write automatic tests for the website that can run in the background and make sure the critical processes are still working.

In conclusion: stay up to date, but be smart about it 🙂

Versioned Backups – A form of Insurance

What are version backups and why should your online business care?

Allow me to share a story with you. One morning I get a call for a client of mine. A website they were maintaining had bad been hacked. The service they were providing was not working anymore and they were losing the trust of their customers. They were asking me to fix it for them.

Fixing a hacked website is very difficult and time-consuming and it can mean a lot of downtime. The better option was to restore the site to a previous state when everything was working.

Thankfully this client understood the value of backups so he had one. Took me an hour to restore the backup. And when we checked the site…

Surprise!

It was still looking bad and the browser was still issuing security complaints. Auch!!

It became clear that the hack had happened a more than a week ago, so restoring the most recent backup did us no good.

And here come the versioned backups. Which is a fancy name for backups that go back in time. You don’t have only the latest backups, you have a daily backup for the last 30 days or a weekly backup for the last 10 weeks.

Because we had those I was able to discover when the hack took place and restore the backup before this. Another 2 hours spent, but now the website was working again.

After one more hour, I discovered that one of the plugins installed had a security flaw that had been exploited. I had to disable and delete that plugin or the site would have been hacked again shortly.

Versioned backups are snapshots of your website across time where you keep more than just the last one.

As you can see, this allows you to reach back in time to when “things were working” and restore your data in case of trouble, even if you discover the issue a few days after the fact.

Why should your online business care about versioned backups?

If your website is mostly static and you don’t offer any services online then you don’t need versioned backups. Just an old backup from last year will do the job.

But let’s be honest. Most websites are in fact web-applications. Meaning they are not just static pages. There is content that is updated, products that are promoted, customer lists, fulfilled orders, and invoices. And if you are doing well, these get updated at least once a day. So a backup from last year will help, but you will still lose a lot of your data.

Depending on how you run your online business and the amount of online activity you will have to decide how often to backup and for how long to keep a backup history.

In my experience so far, with small and medium-sized businesses, doing weekly backups and keeping only the last 4 works very well. This means that in the worst-case scenario you can go back a month, and in the best-case scenario you lose a week of your data: new posts, customers and sales.

But I am paranoid and what I usually do is daily backups that I keep for 2 or 3 months.

Lots of backups and a long history sounds good a reassuring. But there is a cost to that in time and resources. Your server needs to work (sometimes hard) to generate the backup, and then you need the storage space to keep al that history. That is why you need to strike a balance between your real business needs and your peace of mind.

The Take-Away

Versioned backups are a good form of insurance because sometimes the ‘latest backup’ is just as bad as the live website.

To Backup or not to Backup

Some years ago I had the opportunity to work alongside a veteran software developer. That was a treat for me and also a way to learn big lessons fast.

I remember being overconfident in my abilities, fresh out of school, and making silly mistakes when all that knowledge had to be put into practice.

I wanted to be quick, and agile, and free! I wanted to get in, fix the problem and move on!

But there was an incident that taught me a valuable lesson.

The server we were managing got hacked and crashed.

Working alongside the Veteran we managed to identify the security vulnerability, fix it and then restore the website within 6 hours. This was a big and popular forum. 6 hours recovery time was much shorter than the couple of days that this usually takes.

Shortly after restoring access, I heard from one of the members saying: “The way you recovered from this and the speed at which you did it is nothing short of impressive. In my career, I have worked for big software companies and none of them have in place such a good recovery plan.”

I could not take much credit for that, so I decided to pay attention to “the Old Veteran” because it was clear now he knew was he was doing :).

The Importance of Backups

We were able to bounce back so quickly because we had backups. Now only that, but we had versioned backups. Meaning we could go “back in time” to before the problem, see what changed and fix it. And then restore almost all of the user data, with minimal loss. Without versioned backups, this process would have been long and tedious and I do not know if we would have been able to spot the point of entry.

This is a happy ending story and here is what I have learned:

1. You always do backups – even if you think you don’t need them.

2. You test your backups – an untested backup is no backup. I have a story here where a client was paying their hosting company for a remote backup system and when the time came to use it, the backups were corrupted and so not usable.

3. You never delete things – you rename them and then archive them – this way you can always retrace your steps back to something that was working

4. When writing software you always, always use source control – which is basically a system that does smart backups of your work that allow you to “go back in time” and fix problems.

A beginner’s mistake- “I am too good for Backups”

As I have said, fresh out of school, I had bright ideas and I wanted to move very fast, but I did not ever have to deliver work that was used by real people, in a real situation, facing potential attacks from real online threats.

When you are prototyping and testing out an idea, it is OK to be quick, because if the idea is bad or not useful, you need to find out fast. But once you have something that you want to build out for the long term, then you need to switch gears and sacrifice reaction speed for being more organized.

I confess that this did not make sense to me for a long time. But as I worked in bigger and bigger projects it became obvious how the “slow work” of thinking of a structure to organize your code, setting up source control and doing backups was actually the fast lane. Why? Because it reduces risk and allows you to easily maintain the project as you move forward.

The opposite of this is working at neck-breaking speed, not “wasting time” with backups or source control, in order to put something on the market quickly. All the projects that I managed or I was a part of, that did not put in the time to be organized, eventually ground to a halt and had to be abandoned or rewritten.

I have done this mistake enough times to learn my lesson: for quality and sustainable work always do backups and use source control.

Client’s point of view – Do backups make business sense?

It is now obvious for me that backups are not just a good idea. But why should you care about them?

It depends on how well you can manage risk and how important is your data and your customers’ data to you and your business.

If you can afford to lose it all, then you don’t need backups.

If you can afford the downtime of having to rebuild your application from scratch, then you don’t need backups.

But in my opinion, good backups are a cost-effective way to mitigate the security and data loss risks associated with running an online business.

Do you have a backup policy in place? And if you do, have you tested your backups lately to make sure that you will find in there what you expect to find?