Dark Data: How to Audit What Your Business Is Already Storing

Fewer than 15% of organisations have catalogued even half their data. A practical one-week audit you can run yourself, and what to do with what you find.

Dark data is the industry term for information an organisation stores and never uses. Estimates put it at around 55 per cent of everything held, and fewer than 15 per cent of organisations have catalogued more than half their data assets.

Those are enterprise figures. In a business of thirty to three hundred people the proportion is usually worse, because nobody has ever been given the job.

This is a one-week audit you can run yourself, with no tooling and no supplier. At the end you will know what you have, what it is costing, what could be valuable and what should have been deleted years ago.

This article is part of our guide to the data you already own.

Before you start: why do it yourself

Suppliers will happily run this for you, and the output will be thorough and less useful than the version you produce, for one reason. The value of the audit is in the judgement calls about what matters, and you are the only person who can make those.

Do the week yourself. Bring someone in afterwards for the parts that turn out to be worth pursuing.

Day one: list the stores

Every place information lives. Walk the building if you have one.

Every business system: accounts, CRM, job management, payroll, HR
Shared drives, including the ones nobody uses any more
Cloud storage accounts, including personal ones used for work
Email archives, per mailbox and per shared mailbox
Backups, and where they are
Physical storage: filing cabinets, boxes, the loft, the unit you still rent
Anything on a machine under someone's desk
Departed staff's accounts, if they still exist

That last one is where the uncomfortable discoveries usually are. So is the rented storage unit.

For each, record roughly how much, covering which years, and who owns it.

Day two: follow the money

Get the actual costs. Most businesses are surprised.

Cloud storage bills, per service
Any physical storage rental
Licences for systems retained only to read old data
Backup services
The server nobody has switched off

Add it up. A figure between £2,000 and £20,000 a year is typical for an established SME, and a good proportion of it will be for material nobody has opened this decade.

Day three: classify

Three buckets, and be decisive. Anything you cannot classify in thirty seconds goes in a fourth pile to revisit.

Asset. Could support a decision, a price or a product. Quotes and outcomes, job records, customer histories, correspondence explaining decisions, drawings, case files. See where your intellectual property actually lives.

Ballast. Must keep, will never use. Statutory records, old accounts, expired contracts within retention.

Liability. Should not still have it. Personal data past retention, records of people who exercised erasure rights, client material you agreed to destroy, ex-employee mailboxes.

The typical split is around 10 per cent asset, 60 per cent ballast, and more liability than anyone expects.

Day four: find the liability properly

This is the day that saves you from something, and it is the one people skip because it is uncomfortable.

Retention. Do you have a retention policy? Is it followed? Under UK GDPR, keeping personal data with no current purpose is a breach, not a neutral state.

Erasure requests. If someone asked to be forgotten in 2022, are they gone from backups, the old CRM and the shared drive, or only from the live system?

Departed staff. Mailboxes and personal drives of people who left. Often full of personal data, often still accessible, often forgotten entirely.

Client obligations. Contracts frequently require destruction at the end of an engagement. Check whether it happened.

Special category data. Health, ethnicity, union membership. Frequently present in HR folders and subject to much stricter rules than businesses realise.

Anything found here gets a decision this week, not a project next quarter.

Day five: score the assets

Only the asset pile. Four questions per item, yes or no.

Specific? About your business rather than your sector. Verified? Records outcomes, not just intentions. Exclusive? A competitor could not buy the equivalent. Connected? Joined, or joinable, to something else.

Four yeses is rare and valuable. Three is worth pursuing. Two or fewer is interesting rather than useful.

Then for each of the strong ones, write the sentence: "If this were easy to query, we would be able to decide ___." If you cannot finish the sentence, it is not an asset yet.

What to do with the result

Delete the liability. This week, with a record of what was deleted and when. It is the only part of the audit that is urgent.

Price the ballast. If it costs £6,000 a year to store material you must keep, check whether it is on the right tier of storage. Archive tiers cost a fraction of live storage and most businesses have never moved anything.

Pick one asset and test it. One question, one year of data, three to six weeks. Do not commission a full extraction on the strength of an audit. See scanned, filed and forgotten. Test the slice first.

Change what happens next. The audit is a snapshot. If nothing changes about how records are captured, you will be in the same position in three years. Usually this means adding one or two fields to something people already fill in.

What this typically finds

In a professional services firm of about eighty people, a real example of the shape of it:

4.2TB across six stores, costing roughly £9,400 a year
Two ex-employee mailboxes still live, three years after departure
A storage unit costing £2,280 a year containing files superseded by a scanning project in 2021
An asset: eleven years of proposals with outcomes, spread across email and a spreadsheet, never compared
A liability: candidate CVs from a discontinued recruitment arm, retained with no basis

The deletions took a fortnight. The storage saved about £4,000 a year. The proposals turned into a pricing project that was worth considerably more than both.

Key Takeaways

Around 55 per cent of stored data is never used. In most SMEs the split is roughly 10 per cent asset, 60 per cent ballast, and an uncomfortable amount of liability.
Run the audit yourself over a week. The value is in judgement calls only you can make.
Day four is the one that matters most and the one people skip. Retained personal data with no purpose is a breach, not a neutral state.
Score assets on specific, verified, exclusive, connected. Then finish the sentence "if this were queryable we could decide ___". If you cannot, it is not an asset.
Deleting the liability is urgent. Everything else is a project. Test one asset on one year before commissioning anything.

Frequently Asked Questions

Do we need a tool to do this?

No. A spreadsheet and a week. Data discovery tools are useful at much larger scale and they will not tell you which of the findings matters, which is the entire point of the exercise.

What if we find something bad?

Deal with it, record what you did and when, and do not delete evidence of the problem itself. A business that finds a retention failure and fixes it with a documented decision is in a materially better position than one that never looked. "We did not know" is not the mitigation people assume it is.

How often should we repeat this?

A full audit every two to three years. The retention and departed-staff checks are worth doing annually, and they take an afternoon once the first full pass is done.

Our data is all in one system. Is this still worth it?

Almost certainly yes, because it very rarely is. The audit's most common finding is stores nobody had counted: a shared drive, a departed mailbox, a storage unit, a laptop. If you believe everything is in one system, day one will be the interesting day.

Want a second pair of eyes on what your audit turned up? Talk to Halo Technology Lab. Our strategy and scoping service picks up where the week ends, and we will tell you which findings are worth money and which are just tidying.