All your Eggs in One Giant Humungous Basket

October 31, 2014

RAM (Random Access Memory) is the short-term very fast memory your computer uses, not to be confused with a hard-disk, which generally are larger, but much slower.

If you buy the cheapest available computer from Dell, today, it will ship with at least 1 Gigabyte of RAM, and you will be very strongly advised to double that for merely a few bucks more.

Those of you old enough to remember the Apple IIe, will recall it shipped with 64 kilobytes of RAM, or roughly 0.0000009 the amount of RAM in today's Dell.

There have been plenty of predictions that computing power and storage size will grow exponentially. Most famous of these is Moore's Law, which (more or less) claims that advances in technology mean the number of transistors able to fit on a silicon chip (and consequently its computing power) will double every 2 years. A related argument, Kryder's Law suggests roughly the same growth in computer storage. A casual examination of any computer catalogue will easily show the massive hard disks being sold for around $100, which were unheard of a few years ago at any price. Terabytes, anyone?

On the other hand, one of the most popular quotes amongst anti-Microsoft tech-heads is wrongly attributed to Bill Gates, who is said to have suggested, in reference to computer RAM that "640 Kilobytes should be enough for anyone".

As if.

For the record, Gates never actually said it (and cringes when he hears it) however it makes the point that people cannot predict the effect of Moore's Law on their computing requirements.

Curiously, in a similar way to the observation that a gas always expands to fill the volume available, it seems the larger our storage media grow, the more stuff we find to fill it. Put another way - if your attic doubled in size every year, you would probably never throw anything out.

The word processor which ran on the Apple IIe came on a single 64kb floppy disk. The system requirements for Microsoft Word 2007 are approximately 1.5Gb. Admittedly today's Word Processor is significantly more capable than its predecessor, but when you actually look at what the vast majority of people do with their word processor - type stuff, it immediately becomes obvious most of the features, bells and whistles, are unused. Do they really justify 1.5Gb for most users? In short, no.

An 'easter egg', or hidden feature placed as a joke by programmers, in a 1997 version of Microsoft Excel featured a flight simulator game. Although certainly entertaining, it surely translated to a mass of storage wasted. Cool? Totally. Efficient programming? Err... Next question.

So in a Catch-22 situation, storage becomes greater to support requirements thrown against it. Meanwhile, increasingly larger storage is taken for granted by software developers. The relatively low expense means this is probably something we can deal with. But there's one issue people should really pay more attention to - Backing up. Traditionally, backup media such as tapes, was much slower than hard-drives and not suited for real-time use, but massive storage sizes (relative to hard-drives) meant they were ideal for backup. So, your 500Mb hard drive would be backed up daily, weekly and monthly to an 8Gb tape drive.

Today, an external USB Terabyte hard disk can be had for a few hundred bucks. What can one back that up to? Usually it's another hard disk. However this often creates problems such as storing a copy off-site (in case your building is destroyed). Unlike durable tapes, these things don't like being schlepped around. Dropping hard-drive (or in some cases merely moving them) can also cause data loss. So much for your "backup".

A typical backup regime for a 9-5 O'clock company meant a daily backup on Monday, Tuesday, Wednesday, Thursday and Friday, with a weekly backup each Saturday for a month, and a monthly backup retained. That meant after any given disaster, minor loss or data corruption, the organisation can refer to daily backups for the last week, and weekly backups for the last month and so on. It also meant a collection of at least 7 tapes assuming everything could fit on one tape. When you start talking about hard drives however, this becomes a significant expense.

Sure there are devices such as Network Applicance Storage (NAS) and huge Storage Area Networks (SAN) but what exactly are you archiving to justify this expense? To further complicate matters, users are becoming used to virtually unlimited storage. Google's Gmail email system effectively has no limit on storage. Never again will users have to delete old emails. As such, those who have been so tempted, will very shortly afterward demand the same from their IT Department who once would have capped their quota at 100Mb (if that). Newer regulatory compliance programs, such as the American Sarbanes-Oxley Act and some of our own recordkeeping requirements mean lots of electronic data cannot be disposed of for years.

Organisations need to give very detailed thought to a few issues:

  • How much storage space the organisation and its users really require.
  • How this information is to be backed up
  • If this information is to be retained, how it will be migrated to newer or different storage requirements.

To put point 3 in perspective, consider how many people could today play an LP record if they found an old box full of them upstairs in their Magic Attic. Now think about their chances 10 years from now. There's no point hanging onto data if there is no means to recover it. While offices are trying to go paperless, at least paper can be read hundreds of years later. Electronic records often can't. Anyone burying a time-capsule in their backyard is unlikely to put a USB-thumbdrive of photos in there, as it's a pretty safe bet our descendants won't have anywhere (useful) to stick it.

Having these huge amounts of storage also create a significant security threat - imagine how much sensitive information can be stored on the drive in a single laptop, or pocket-sized USB drive. Now imagine that being left in a taxi.

In 2005 Wilcox Memorial Hospital in Hawaii had to inform 120,000 past and present patients that their private information had been lost. Each of their names, addresses, Social Security numbers and medical record numbers had been placed on a tiny USB thumb-drive — and now, according to the letter, the drive was missing. Oops.

Business Information Agency Lexis-Nexis had a data leak involving about 32,000 customers. American hardware outlet Home Depot lost a laptop containing personal information belonging to 10,000 employees.

American examples are easy to find, as their legal system in many cases forces companies to publicly admit when they've had a breach. Many Australian companies might prefer to keep their mouths shut (and their clients none the wiser) but a casual Google search for "data leak" will quickly demonstrate the extent of the problem. The quick answer to the problem of backup media being lost, stolen or misplaced, is to encrypt it. However this creates a significant problem of its own, called 'what to do when nobody remembers the password'. If there is a disaster, which destroys your building and results in loss of life, it is possible the content of encrypted backups could literally be taken to someone's grave. As with all encryption, it's therefore important that 'key management' issues be considered to address problems of forgotten passwords, or as was recently the case in a San Francisco Government office, former IT staff members holding you to ransom for your data.

Companies need to address these issues proactively, rather than stare at a pile of data and wonder how to preserve, archive and (eventually) dispose of it, as well as manage ongoing capacity issues.

We are becoming addicted to storage, and like any addiction need to give thought to where it can take us.

Organisational Information Security Policies should clearly address all of these issues, and then it is simply a matter of technical implementation of your clearly defined policy.

For further enquiries or reprint information, click here to contact Calamity.

Have a question?
We'll answer your enquiry within 24 hours.