Dark Data

Dark data is information an organisation collects and stores but never uses for analysis or decisions. Estimates put it at around 55 per cent of everything businesses hold, and fewer than 15 per cent of organisations have catalogued even half their data assets.

Those are enterprise figures. In a business of thirty to three hundred people the proportion is usually worse, because nobody has ever been given the job.

The three things it actually splits into

Calling it all "dark data" is less useful than sorting it, because the three parts need completely different responses.

Asset. Information that could support a decision, a price or a product if it were findable and reliable. Quotes and their outcomes. Job records with what actually went wrong. Correspondence explaining why something was decided. Typically around 10 per cent of the total.

Ballast. Information you must keep for legal or regulatory reasons and will never otherwise use. Statutory records, expired contracts within retention, backups of backups. It costs money and creates risk, and that is the whole of its contribution. Usually the majority.

Liability. Information you should not still have. Personal data past its retention period, records of people who exercised erasure rights, client material you agreed to destroy, the mailbox of someone who left in 2022. Under UK GDPR, holding personal data with no current purpose is a breach rather than a neutral state.

Most businesses running this exercise for the first time find more liability than they expected, and it is the only part that is genuinely urgent.

What makes the asset portion valuable

Four tests, and most archives fail the fourth.

Specific to your business rather than your sector. Verified, recording what happened rather than what was planned. Exclusive, so a competitor could not simply buy the equivalent. And connected, joined to something else.

The joins are where both the value and the work sit. A customer list is mildly useful. A customer list joined to purchase history, price paid and what went wrong is a different asset entirely, and in most businesses those facts live in three systems and have never been put in the same place.

Why it is worth more than it was

The economics changed over the last two years. Falling processing costs made it viable to read a whole archive rather than a sample, which means extraction projects that did not pay for themselves in 2024 now do.

That said, the test before starting has not changed: name the decision that would be different. If nobody can finish the sentence "if this were queryable, we could decide ___", the answer is to leave it dark and reduce the storage bill instead.