Public Data Is Everywhere. Usable Data Is Another Story From broadband coverage to hospital equipment, court records and shipping timelines, information can be public and still be remarkably difficult to use.

At first glance, the project briefs do not belong in the same article.

One involves checking broadband availability across roughly 1.8 million UK postcodes. Another is about finding evidence of new surgical equipment on hospital websites across five countries. A third deals with court records spread across different jurisdictions. There is an airline pricing project and, elsewhere, the problem of turning two shipping carriers' tracking information into one understandable timeline.

They come from different industries and answer different business questions.

They also came through the same company, Datamam.

That gave us a reason to be cautious about drawing too broad a conclusion from them. A company that specializes in external data is naturally going to encounter external-data problems. Five examples from one provider do not prove that the entire business world has the same problem.

So we looked outside those projects.

The same tension appears in places where organizations have already made serious attempts to solve it.

US healthcare regulators require insurers and hospitals to publish large quantities of pricing information in machine-readable form. Yet in a 2025 proposal, federal departments identified file size, missing context and difficulty comparing different disclosure systems as barriers to actually using that information. CMS is still working on standardization and comparability requirements in 2026.

Public procurement has arrived at a similar conclusion from another direction. The Open Contracting Data Standard exists specifically to give contracting information common structures, definitions and identifiers so that records published by different governments can actually be compared. It has been implemented by more than 50 national and subnational governments.

In other words, putting information online is one problem.

Making it useful together is another.

That distinction became the more interesting story.

A Very Simple Question Can Become a Very Large System

Broadband is probably the clearest example.

If I want to know whether fiber internet is available at my home, I can open a provider's website, enter my postcode, select an address and get an answer.

There is nothing particularly sophisticated about that.

Now try asking the same question across roughly 1.8 million UK postcodes and close to 100 internet providers.

The original question suddenly contains dozens of smaller ones.

Which providers serve the postcode? Does availability change by address? Which packages are offered? What is the promotional price and what is the normal one? Is there an upfront cost? What kind of connection is it? What are the advertised speeds? How long does the contract last?

And, perhaps most importantly, when was all of that true?

This was one of the Datamam projects we reviewed. The scope was not simply to find broadband coverage. The aim was to turn hundreds of thousands of individual availability checks into something that could describe the market consistently.

The websites themselves were never designed for that purpose.

They were designed for individual customers.

That difference seems obvious once you see it, but it has large consequences. A website can work perfectly well for the person it was built for and still be a terrible source for systematic analysis.

There is another complication: the answer does not stay still.

Ofcom said in March 2026 that full-fibre broadband had reached 78% of UK homes, compared with less than a quarter five years earlier.

That makes broadband data a moving target.

A map can be completely accurate when it is built and still become increasingly wrong without anyone making a mistake.

This is where Sandro Shubladze, Datamam's founder and CEO, makes a distinction worth keeping.

"One postcode can produce a completely different commercial picture from the postcode next to it," he says. "Once you multiply that by millions of locations, dozens of providers and repeated updates, you are no longer doing market research. You are operating a data system."

There is a less technical way to put it.

Sometimes scale does not simply make the same job bigger.

It changes what the job is.

Machine-Readable Does Not Necessarily Mean Usable

Healthcare regulators have provided a useful real-world test of that idea.

Under US transparency rules, enormous quantities of healthcare pricing information are now required to be published as machine-readable files.

In theory, this is exactly what data users ask for. No PDFs to interpret. No need to manually read a webpage. Structured information, published digitally.

And still, problems emerged.

When federal agencies proposed changes to the Transparency in Coverage rules in 2025, they described three major barriers they had observed: some machine-readable files were so large that they were difficult to access, raw data often lacked enough context to interpret it properly, and hospital and insurer disclosures were not aligned well enough to make comparison easy.

CMS has spent several years introducing templates, data dictionaries and more standardized reporting requirements for hospital pricing. Its current rules specify what hospitals must publish and how those files should be structured, while 2026 changes go further in trying to improve accuracy and comparability.

There is something useful in this example because no scraping company is required to make the argument.

The government already mandated machine-readable data.

The next problem was making the machine-readable data understandable and comparable.

That is a much broader lesson.

A CSV does not settle what a field means.

JSON does not guarantee that two organizations use the same definition.

An API does not make two databases conceptually compatible.

The technical format matters. It just does not finish the job.

Sometimes There Is No Dataset to Begin With

The problem becomes even less tidy when the desired information was never published as structured data at all.

One Datamam healthcare project involved hospitals across China, South Korea, Japan, India and Australia.

The goal was to find public evidence that hospitals had acquired or installed surgical equipment, then extract details such as the hospital, equipment brand, quantity where available, and acquisition or installation date. The search was intended to be repeated quarterly.

There was no master field called "new surgical equipment."

The evidence could be buried in a hospital announcement or press release.

This looks less like downloading data and more like research.

Suppose a hospital says it has started performing a procedure using a new technology. Does that mean it bought a machine?

Possibly.

Does it prove it?

Not necessarily.

A manufacturer announcement may say an installation took place but omit the number of units. A hospital may announce a new capability without explaining whether the equipment is owned, leased or shared. A local article might mention the technology while leaving the date unclear.

Those distinctions matter if the result eventually becomes a neat row in a spreadsheet.

The row can look much more certain than the source ever was.

This was one place where Shubladze's explanation was useful because it described a problem journalists encounter as well.

"Finding the keyword is the easy part," he says. "The harder part is determining what that occurrence actually means and structuring enough evidence around it that somebody can make a business decision."

It is essentially a source problem.

What does the evidence actually establish?

And how much of that evidence survives once an article, document or webpage becomes four database fields?

There is a temptation in automation to treat uncertainty as something that should disappear.

Sometimes it should remain visible.

Court Records Show Why Definitions Matter

Courts provide almost the opposite situation.

The information is highly formal. The concepts are familiar. Cases have parties, hearings, dates, judges and documents.

You might expect standardization to be easier.

In practice, one Datamam court-intelligence project involved nine court systems across several US jurisdictions, and those systems represented similar concepts in quite different ways. Some records appeared as HTML, others as PDFs or calendars, while some required searches before the relevant information appeared. Even labels for similar concepts varied between systems.

At that point, retrieving every record successfully is not the end of the work.

It creates nine datasets.

The business wants one.

Someone has to decide whether "hearing purpose" in one system is equivalent to a differently named field somewhere else. A date may refer to filing in one system and activity in another. Two court identifiers that look similar may not describe the same thing.

This is not unusual enough to be considered an edge case.

Entire international standards have been created to address essentially the same problem elsewhere.

Public procurement is one example.

Government contracting information covers tenders, awards, suppliers, contract values, amendments and implementation. Different governments naturally built different systems for publishing it.

The Open Contracting Data Standard tries to make that information comparable by supplying a common data model, shared definitions, identifiers and codelists. Its documentation makes an important point: a data standard defines both the structure and meaning of information so that people and systems can interpret it without the same ambiguity.

That wording matters.

Structure and meaning are not the same thing.

A database can be beautifully structured and still represent the wrong concept.

Not All Data Ages at the Same Speed

The broadband example raised another question that kept returning as we looked through the projects:

How current does information actually need to be?

The fashionable answer is "real time."

The practical answer is more complicated.

One Datamam travel project compared fares for the same airline across four booking platforms. The comparison had to take route, departure date and point of sale into account, and the relevant number was the complete fare rather than the most visible promotional price. Dates extended roughly 90 days into the future and the comparison was refreshed daily.

Four websites sounds small.

But the unit being compared is not "website."

It is a particular route, on a particular departure date, in a particular market, viewed through a particular booking channel, with several components contributing to the final fare.

Change one variable and the answer can change too.

There is no obvious reason that the same refresh schedule should apply to every other kind of information.

A hospital address can remain useful for years.

A corporate filing may matter for months.

A tender can lose much of its competitive value well before its official deadline.

An airfare may change today.

I find it more useful to think about this as the economic half-life of information.

How quickly does a correct observation become too old to support the decision it was collected for?

That question is more disciplined than simply asking for real-time data.

If information remains useful for six months, collecting it every five minutes does not make the business smarter.

If the commercial situation changes every few hours, a monthly dataset can be technically accurate and operationally irrelevant.

Freshness is therefore not an abstract measure of quality.

It depends on what somebody plans to do with the answer.

Matching May Matter More Than Collection

There is one more problem that is easy to underestimate because it happens after the information has already been found.

Two sources can describe the same real-world thing differently.

In ecommerce, the same product appears under slightly different titles.

A property is reposted across real-estate websites with a shortened address.

A hotel has an English name, a local name and sometimes an old brand name.

A company changes its domain.

Shipping systems have their own terminology for movements and milestones.

The records can all be individually correct.

They are still difficult to compare until somebody establishes which records belong together.

One logistics project Datamam worked on involved two major ocean carriers, with documented populations of roughly 10,000 containers in one system and 30,000 in the other. The carriers represented shipment activity differently, while the desired output was one understandable operational timeline.

That required reconciling booking and bill-of-lading identifiers, vessels, voyages, ports, transshipment events and estimated versus actual movements.

The customer does not particularly care that two carriers have different data models.

The customer wants to know where the container is.

This is where the word "matching" undersells the work.

The raw information exists before matching.

The useful business object often does not.

Three records become one shipment.

Two listings become one property.

Several product descriptions become one product.

The act of deciding that these things are the same is part of the business logic, not housekeeping performed after the real work.

There Is a Point Where This Argument Can Go Too Far

After looking through enough examples, it is easy to arrive at a seductive conclusion: external information is valuable, therefore companies should collect more of it.

I do not think the examples support that.

More data creates its own problems.

Every source costs something to integrate and maintain.

Every additional update consumes resources.

Matching can produce false positives.

Public information can involve privacy, licensing, contractual or legal considerations.

An automated conclusion can create more confidence than its underlying evidence deserves.

And sometimes the data is simply not valuable enough.

This is where the technology conversation tends to run ahead of the business question.

A company can build an extremely sophisticated system for monitoring something that rarely changes a decision.

That is still a bad investment.

There is another issue too.

The internet is not a neutral database.

Websites publish information for different reasons. A hospital announcement is written partly to communicate. A retailer wants to sell a product. A government portal has legal publication requirements. A marketplace determines how offers are ranked and presented.

Context does not disappear simply because the information is converted into rows.

This is why I would be cautious about describing every external signal as "truth."

It is evidence produced in a particular context.

Good systems preserve enough of that context to make the evidence useful without pretending it is more definitive than it is.

Shubladze makes a narrower argument.

"The mistake would be thinking the lesson is that every company needs more external data," he says. "The useful question is which external signal changes a decision, and then what infrastructure is required to make that signal dependable."

That is harder to turn into a technology slogan.

It is probably closer to the real problem.

The Internet Was Never Meant to Be One Database

The independent examples changed how I looked at the Datamam projects.

If the problem appeared only in five projects handled by one external-data company, it would be easy to explain as a feature of that company's particular line of work.

But healthcare regulators are trying to standardize machine-readable pricing files because accessibility alone did not make the information sufficiently comparable. Public-procurement organizations created a common international data model because governments could publish information openly and still describe it differently. OECD work on data-sharing frameworks similarly treats interoperability and reusability as problems requiring technical, organizational and governance solutions, not merely access.

The Datamam projects are not proof of a universal law.

They are useful illustrations of a problem that appears elsewhere independently.

And perhaps there is a straightforward reason for it.

The internet was never designed to function as one enormous company database.

An ISP availability checker exists to answer a prospective customer's question.

A hospital press release was written for patients, journalists or the local community.

A court portal exists to make legal records available.

A travel site exists to sell travel.

A carrier portal exists so customers can track shipments.

Each source makes sense on its own terms.

The difficulty appears when another organization wants thousands or millions of those observations to behave like one coherent internal system.

Then the work changes.

A webpage has to become a record without losing what the page meant.

A public statement has to become a signal without becoming more certain than the evidence allows.

Different definitions have to be reconciled.

Records describing the same thing have to be connected.

And information has to be collected again before it gets too old to matter.

That is why the most interesting pattern across these projects is not really web scraping, automation or even scale.

It is translation.

Companies are trying to translate an external world they do not control into information precise enough to make decisions with.

Shubladze sees the same problem from the infrastructure side.

"The real challenge with external data is that you are not just collecting facts, you are collecting somebody else's representation of reality," Shubladze says. "A reliable system has to distinguish between a change in the real world and a change in how the source describes it."

There is already more public information available than most organizations could ever consume.

The scarce part is not always the data.

It is knowing what the data means, how long it remains true, what it corresponds to elsewhere, and whether the answer is reliable enough to act on.