Data Sources

Data sources are the evidence layer behind research, software, search, maps, dashboards, knowledge graphs, and data visualization. A useful source is not merely a downloadable file. It is a bundle of provenance, access method, schema, identifiers, license, update behavior, documentation, and interpretation rules.

The practical question is not "where can I get data?" but "can this source be trusted, cited, joined, refreshed, stored, and explained after the original collection context has faded?"

This page treats sources as durable records. It is a wiki-style checklist for deciding whether a dataset, catalog, API, archive, web capture, document collection, field note, or local enrichment belongs in a serious knowledge system.

A data source is any identifiable origin from which records, observations, files, facts, metadata, media, or claims are collected for later use. The source may be a government portal, research repository, archive, book catalog, API endpoint, web crawl, database dump, instrument feed, spreadsheet, field notebook, photo set, map layer, command transcript, or physical object record.

A source becomes dependable when it can answer six questions:

• who created it, and for what purpose;
• what it contains, excludes, and transforms;
• how it can be accessed without violating terms or context;
• which identifiers, schemas, units, and vocabularies give it meaning;
• how it changes over time;
• which claims come directly from the source and which are local interpretation.

That last boundary is central. A catalog record can say that a book has a publication date; a local enrichment can infer that it belongs near books, print, or history. Those are different evidence layers. A good source workflow keeps them connected without flattening them into one fact.

Most useful sources fall into overlapping categories:

• Primary records: original observations, measurements, images, scans, filings, logs, transcripts, field notes, code commits, and official records.
• Curated datasets: cleaned tables, research data releases, statistical series, geospatial layers, bibliographic indexes, and benchmark corpora.
• Catalogs and registries: descriptions of other objects or datasets, such as library catalogs, DOI registries, agency catalogs, vocabulary registries, or collection inventories.
• APIs and data services: request-time access to records, subsets, search results, graph queries, or derived representations.
• Archives and snapshots: preserved copies of pages, files, media, software, and metadata that may no longer match the live source.
• Derived sources: embeddings, OCR, labels, classifications, summaries, reconciled entities, and other computed interpretations.

The category matters because it changes the trust question. A primary measurement may be noisy but close to the event. A catalog may be authoritative about holdings but incomplete about meaning. An API may be current but hard to reproduce later. An archive may preserve evidence but lose live context. A derived source may be useful while still requiring model version, prompt, confidence, and input provenance.

A dataset is a collection of records intended for analysis, reuse, or reference. A catalog describes datasets or objects so they can be discovered. An API provides an access path. An archive preserves a state of material across time. Confusing these layers causes many source failures.

For example, Data.gov (opens in new tab) is a catalog for United States government data. A Census API endpoint is a data service. A CSV downloaded from an agency is a distribution. A local copy with checksum, retrieval date, and transformation log becomes a managed source record. A web capture of the agency page is supporting evidence, not the dataset itself.

The same distinction matters for semantic web work. A Wikidata item is a graph entity with statements and references. A Wikidata dump is a dataset distribution. The SPARQL endpoint is a query service. A local reconciliation table is a derived source. Keeping those layers explicit makes graph edges and search results auditable.

Provenance records where a source came from, who handled it, and how it changed. It should capture publisher, creator, maintainer, retrieval date, collection method, version, transformation steps, and the difference between source claims and editorial claims.

For OSINT, provenance is chain of custody: URL, query, timestamp, screenshot or archive URL, language, translation, geospatial context, and uncertainty. For photography, it includes camera metadata, location policy, edits, derivatives, and rights. For bookbinding, it may include work, edition, physical copy, repair history, conservation photo, catalog record, and auction record as related but separate sources.

Provenance is also a graph design problem. A graph should be able to say that a claim was asserted by a specific source, imported on a date, transformed by a script, reconciled against an identifier, and displayed in a local article. Without that, a graph can look authoritative while quietly mixing evidence, inference, and presentation.

Open access does not always mean open reuse. Before storing or publishing a source, record the license, terms of service, attribution requirement, redistribution rule, privacy boundary, rate limit, and commercial-use restriction. Government works, public-domain datasets, CC0 data, CC BY data, database-right-protected material, and proprietary API outputs all have different reuse surfaces.

Respect access intent. Prefer official downloads, bulk files, APIs, documented exports, sitemaps, and data packages over scraping visible pages. When a site publishes a rate limit or automated-access policy, treat that as part of the source. A technically reachable page can still be the wrong source if redistribution would violate rights, expose private context, or misrepresent the publisher's intended access path.

For web material, a wget capture should name its scope, command, tool version, retrieval date, status codes, redirects, exclusions, robots behavior, storage path, and publication policy. The capture is only source-backed if future readers can tell what was collected and why.

Sources decay. URLs move, schemas change, identifiers merge, institutions rename collections, APIs deprecate fields, and once-complete datasets stop receiving updates. A source that looked stable at ingestion can become misleading if the update cadence is unknown.

Record whether a source is static, append-only, periodically revised, rolling, or query-time dynamic. Preserve version labels, release dates, checksums, changelogs, API documentation versions, and local fetch timestamps. If a dataset is silently overwritten, keep raw snapshots when licensing allows. If an API is live-only, store enough response examples and query parameters to explain what the system saw.

Dedupe is part of freshness. Bibliographic data may distinguish work, edition, translation, manifestation, holding, and physical copy. Place data may contain a modern municipality, an ancient settlement, a protected area, and an archaeological site with overlapping names. People and organizations may appear through aliases, initials, transliteration, merger history, and name changes. Treat identifiers and aliases as first-class records, not cleanup trivia.

When a source is new, triage it before building on it:

• Authority: Is the publisher close to the underlying event, measurement, collection, or standard?
• Documentation: Are fields, units, coverage, missing values, identifiers, and update rules explained?
• Licensing: Is reuse allowed for the intended public, private, commercial, or research context?
• Completeness: What populations, places, languages, dates, formats, or edge cases are missing?
• Stability: Are URLs, IDs, releases, and schemas durable enough to cite?
• Bias: What collection incentives shaped the data?
• Joinability: Does it expose stable IDs, coordinates, dates, names, or external references?
• Auditability: Can a reader reproduce the access path and separate raw records from local enrichment?

Rules of thumb help here: prefer stable identifiers, explicit licenses, documented schemas, primary publishers, clear update histories, and independent cross-checks. Those shortcuts should lead into a source record, not replace one.

A minimal source record should include:

• title and short description;
• canonical URL and access URL;
• publisher, creator, maintainer, and contact page when available;
• retrieval date, release date, version, and update cadence;
• license, terms, attribution text, and redistribution limits;
• access method: file, API, database, web page, catalog, archive, or physical object;
• schema notes, field definitions, units, formats, and controlled vocabularies;
• primary identifiers and external reconciliation targets;
• geographic, temporal, linguistic, and domain coverage;
• known exclusions, quality warnings, and privacy notes;
• storage location, checksum, raw snapshot policy, and transformation scripts;
• derived claims, confidence, model versions, and human editorial notes.

That schema links directly to data storage: raw source, manifest, transformations, outputs, and citations should be recoverable as separate layers. It also links to type theory: fields become stronger when identifiers, units, dates, states, and missing values have explicit types instead of ad hoc strings.

Book and text sources need object boundaries. A work, edition, copy, binding, digitized scan, catalog record, OCR layer, rights note, and sales record should remain connected without becoming one flattened source. Library of Congress, Internet Archive, Open Library, HathiTrust, WorldCat, and national archives all expose useful but different parts of that stack.

Scholarly sources connect works, authors, institutions, funders, citations, datasets, software, preprints, corrections, and persistent identifiers. Crossref, DataCite, ORCID, ROR, OpenAlex, PubMed, arXiv, and Zenodo are valuable because they expose identifiers and metadata relationships that can be joined into research graphs.

Government and civic sources often have high authority but uneven usability. Data.gov, Census, FRED, World Bank, Eurostat, OECD, UN Data, and agency-specific catalogs are best used with attention to release schedule, geography, methodology, survey design, and revision history.

Geospatial sources need coordinate reference systems, scale, geometry type, topology, accuracy, licensing, and boundary date. Maps and topology matter because roads, parcels, watersheds, administrative boundaries, meshes, and terrain surfaces encode relationships beyond latitude and longitude.

Language, script, and standards sources need vocabulary stability. Unicode, CLDR, Glottolog, Ethnologue, Omniglot, schema.org, Dublin Core, DCAT, PROV-O, SKOS, and ISO-style standards are often reference sources for names, encodings, categories, and interoperability.

Technical artifacts from hacker culture can also be sources: repositories, issues, package manifests, mailing-list archives, command transcripts, releases, advisories, and reproducible build logs. Preserve author or maintainer context, commit or release identifiers, timestamps, and the difference between observed evidence and local interpretation.

A source that cannot be refreshed may still be valuable, but the record should say what kind of source it is. Static archives, historical scans, government statistical releases, live APIs, scraped pages, package registries, and personal spreadsheets all age differently. A quarterly API refresh and a one-time archival capture should not share the same freshness label.

The minimum refresh policy names the owner, access method, expected update cadence, last retrieval date, schema version, license state, and known breaking-change signals. For APIs, preserve endpoint paths, query parameters, pagination behavior, authentication requirements, rate limits, and sample responses. For file releases, preserve checksum, file size, format, compression, and mirror location. For web captures, preserve final URL, status code, timestamp, crawl scope, and robots or terms constraints.

Deprecation is just as important as freshness. A source may be retired, replaced, split, merged, paywalled, rate-limited, legally constrained, or superseded by a better maintained catalog. The graph should preserve supersedes, superseded_by, archived_at, unavailable_since, and use_with_caution edges rather than simply deleting the older source. That helps readers understand why an older data visualization, map, or OSINT note cited a source that no longer looks current.

Many source errors appear during joins. Two datasets can share a name, place, person, organization, ISBN, DOI, coordinate, or URL while still meaning different things. A responsible join record should name the key, normalization rule, match confidence, collision handling, and manual review state. Fuzzy matches should be treated as hypotheses until reviewed.

This is where semantics, semantic web, language, and standards become practical rather than abstract. A join between a Wikidata entity, an archival catalog record, a government dataset, and a local note needs identifier provenance, not just similar labels. Good source work keeps the original identifiers visible so local enrichment can be corrected later.

Common source failures include:

• orphaned CSVs with no publisher, date, license, or schema;
• APIs used without preserving query parameters or example responses;
• screenshots treated as data without source URL or timestamp;
• archived pages mistaken for the current live source;
• scraped pages that ignore terms, robots, pagination, redirects, or hidden context;
• identifiers merged by fuzzy matching without review;
• model labels, OCR, captions, or embeddings stored as if they were primary evidence;
• dashboards that cite transformed aggregates but not the raw source;
• public pages that expose personal, sensitive, or location data beyond the original context.

Most of these failures are preventable with a manifest. The manifest does not need to be elaborate. It needs to be honest enough that a later reader can retrace the path from published claim back to source material.

Useful source records should carry a visible status: live, archived, deprecated, refreshed, transformed, sampled, private, sensitive, incomplete, or disputed. Status labels prevent old data from looking current and transformed data from looking raw. They also help a reader decide whether a source can support a claim, a visualization, a model, or only a lead for further inspection.

For graph use, status belongs beside provenance. A DOI, API endpoint, archive URL, local file, scraped page, or uploaded CSV should say when it was observed, what license or access condition applies, and whether a later process changed it. That keeps knowledge graphs, semantic web, and data visualization honest.

A source-backed knowledge graph should model sources as entities, not footnotes. A graph can then connect people, places, works, datasets, claims, identifiers, citations, licenses, files, and transformations while retaining provenance.

Useful graph predicates include: created by, published by, retrieved from, derived from, cites, same as, has license, has distribution, covers place, covers time span, uses vocabulary, has checksum, supersedes, invalidates, and supports claim. These relationships make it possible to answer questions like "which claims depend on this API?", "which articles cite stale URLs?", "which sources are CC BY?", or "which datasets can be refreshed automatically?"

For multimodal AI, this distinction is critical. A photo, scan, audio clip, video, chart, or UI screenshot can be a source. OCR, captions, embeddings, object labels, and generated summaries are derived interpretations that need their own timestamps, model versions, prompts, and confidence notes. For training neural networks, add label provenance, split rules, deduplication choices, leakage checks, and deletion policy.

• Data Storage for manifests, checksums, raw snapshots, transformations, backups, and retention.
• Semantic Web for identifiers, linked data, provenance, vocabularies, and queryable meaning.
• Graphs for source-backed claims, evidence paths, entity reconciliation, and traversal.
• Standards for technical definitions, metadata vocabularies, and interoperability.
• OSINT for public evidence chains, timestamps, uncertainty, and ethical collection.
• Maps for projection, scale, boundary, and coordinate-system concerns.
• wget for Web Archival for scoped captures, retrieval logs, and preservation policy.
• Software Libraries for clients, parsers, validators, normalizers, and dependency trust.

General open data and public statistics:

• Data.gov (opens in new tab) and Catalog.data.gov (opens in new tab) for United States government open-data discovery.
• World Bank Open Data (opens in new tab) for global development indicators and Data360 resources.
• U.S. Census Data API (opens in new tab) for Census Bureau datasets and documented API access.
• Federal Reserve FRED and DDP API information (opens in new tab) for programmatic access notes around Federal Reserve economic data series.
• Eurostat Data Browser (opens in new tab) for European Union statistics.
• OECD Data Explorer (opens in new tab) for OECD statistical datasets.
• UNdata (opens in new tab) for United Nations statistical databases.

Archives, libraries, and cultural collections:

• Internet Archive (opens in new tab) and the Internet Archive Developer Portal (opens in new tab) for item metadata, media collections, and Wayback access paths.
• Library of Congress APIs (opens in new tab) for structured access to Library of Congress collections.
• National Archives Catalog API (opens in new tab) for NARA archival descriptions, authorities, OCR text, digital-object metadata, and public contributions.
• Open Library APIs (opens in new tab) for book metadata and monthly dumps.
• WorldCat (opens in new tab) for global library holdings discovery.

Scholarly and persistent-identifier infrastructure:

• Crossref REST API (opens in new tab) for DOI metadata, funding data, license data, ORCID/ROR links, references, and updates deposited by members.
• DataCite REST API (opens in new tab) for DataCite DOI metadata retrieval, creation, and update workflows.
• OpenAlex API (opens in new tab) for open scholarly works, authors, sources, institutions, topics, funders, and snapshots.
• Zenodo Developers (opens in new tab) for records, files, deposits, and published research outputs.
• PubMed (opens in new tab) for biomedical literature discovery.
• arXiv (opens in new tab) for open preprints in physics, mathematics, computer science, and related fields.
• ORCID Public API (opens in new tab) and ROR API (opens in new tab) for researcher and organization identifiers.

Web, graph, and machine-readable knowledge:

• Wikidata (opens in new tab), Wikidata data access (opens in new tab), and Wikidata database downloads (opens in new tab) for open linked data, SPARQL, dumps, and entity reconciliation.
• Common Crawl (opens in new tab) for large-scale open web crawl data.
• DBpedia (opens in new tab) for structured data extracted from Wikipedia.
• Schema.org Dataset (opens in new tab) for web-visible dataset metadata.
• W3C DCAT 3 (opens in new tab) for dataset catalogs, distributions, and data services.
• W3C PROV-O (opens in new tab) for provenance modeling.
• Dublin Core Terms (opens in new tab) for general metadata properties.
• FAIR Principles (opens in new tab) for findable, accessible, interoperable, and reusable data.

Geospatial, environmental, and scientific data:

• OpenStreetMap (opens in new tab) for collaboratively maintained map data.
• Natural Earth (opens in new tab) for public-domain cartographic base layers.
• USGS National Map (opens in new tab) for United States elevation, hydrography, imagery, and topographic data.
• OpenTopography (opens in new tab) for terrain, lidar, and point-cloud access.
• NASA Earthdata (opens in new tab) for NASA Earth observation datasets and discovery tools.
• NOAA NCEI Search (opens in new tab) for weather, climate, ocean, and geophysical data.
• GBIF API (opens in new tab) for biodiversity occurrence, species, registry, maps, literature, and validation data.
• NIST Chemistry WebBook (opens in new tab) for chemical and thermophysical reference data.