wget for Web Archival

wget is a command-line tool for retrieving files over HTTP, HTTPS, FTP, and related protocols. It is useful for reproducible downloads, archival work, dataset collection, link-preserving mirrors, and scripted retrieval where a browser is the wrong tool.

This page is deliberately about responsible archival, not indiscriminate scraping. Respect site terms, robots directives, rate limits, authentication boundaries, and copyright. For OSINT or research workflows, pair wget with source notes and provenance so downloaded material remains interpretable later. See also data sources, OSINT, domains, blogs, data storage, SEO, and standards.

wget is best understood as a retrieval instrument. It does not decide whether a capture is allowed, whether a page is authoritative, whether a local mirror is publishable, or whether a downloaded file still means what the source meant. Those are editorial, legal, and provenance questions. The tool becomes valuable when the human workflow answers those questions explicitly.

A good wget note is therefore half command and half evidence record. It should preserve the URL list, scope decision, command, logs, timestamps, status codes, checksums when needed, and a plain-language reason for the capture. That turns a local folder into something the data sources, data storage, GitHub, and semantic web parts of the compendium can reason about.

Use wget when the important thing is reproducible retrieval from known URLs. It is a good fit for downloading public files, preserving a small documentation subtree, collecting an approved list of PDFs, saving a source page before it changes, or building a local mirror for offline reading. It is especially useful when the capture can be described before execution: URL list, accepted domains, recursion depth, file types, delays, output location, and purpose.

It is a poor fit when the page depends heavily on JavaScript, user gestures, authenticated state, anti-abuse challenges, live APIs, or browser-only rendering. In those cases a browser automation tool, API export, dataset dump, official archive, or manual citation may produce a more honest artifact. A command-line mirror that silently misses the actual content is worse than no mirror because it looks complete.

The practical rule is simple: wget should make a narrow retrieval legible. It should not turn uncertainty about permission, scope, or content meaning into a large folder of files.

Start with a single file:

SOURCE_FILE_URL="<approved-file-url>"
wget "$SOURCE_FILE_URL"

Mirror an allowed section gently:

CAPTURE_ROOT="<approved-section-url>"
wget --mirror --page-requisites --convert-links --adjust-extension --no-parent --wait=1 --random-wait "$CAPTURE_ROOT"

This keeps the mirror scoped below the provided path, downloads page requisites, converts links for local browsing, and adds delay between requests.

The command should reveal the intent. A single approved file download, a URL-list refresh, a bounded documentation mirror, and a WARC preservation run are different operations even when they all use wget.

• Use a direct URL when the target file is known and the goal is retrieval.
• Use --input-file when the scope has been reviewed separately.
• Use --mirror --no-parent only when the starting path is the boundary.
• Use --domains when redirects or embedded assets might otherwise expand the crawl.
• Use --warc-file when preservation context matters more than a convenient local folder.

This distinction is part of hacker culture: the elegant command is not the shortest command, but the one whose assumptions can be inspected later.

The scope of a wget run should be written before the command starts. Name the starting URL, allowed domains, accepted file types, maximum recursion depth, excluded paths, rate limits, and whether redirects should be followed. A capture without scope is hard to defend later because every missing file looks accidental and every extra file looks intentional.

For public research, use the smallest capture that answers the question. If the goal is to preserve one policy page, do not mirror the whole domain. If the goal is to preserve a documentation set, start from the documentation root and use --no-parent. If the goal is to preserve evidence around a claim, keep the source URL list and retrieval log beside the files so the collection remains explainable.

Scope should also name what is out of bounds. Exclude login paths, account pages, cart flows, search result permutations, calendars, generated archives, infinite URL spaces, private files, and endpoints whose URLs encode personal data. A capture that is technically public can still be invasive or misleading when it amplifies material outside the original context.

When the source offers a sitemap, index page, downloadable archive, or API endpoint, prefer that deliberate surface over blind recursion. The source's own export structure often preserves more intent than a crawler can infer from links alone.

wget supports robots exclusion behavior, but the existence of a flag is not permission to bypass a site owner's stated preferences. For compendium work, the default stance is conservative: honor robots directives, follow published terms, identify the capture purpose in notes, and avoid authenticated or personalized pages unless the user is explicitly preserving their own data.

There are cases where a public-interest archivist may preserve material that later disappears, but that is a decision with context, not a convenience setting. If a capture turns into evidence for history, OSINT, or public reporting, record why the capture was necessary, what was excluded, whether redistribution is allowed, and how readers can reach the live source or recognized archive.

Rate limiting is part of consent in practice. Small sites, personal blogs, public agencies, and volunteer-maintained documentation can be fragile. Use delays, small scopes, and off-peak timing; do not make another person's server absorb your archival ambition.

A useful wget capture should have a manifest before it has a folder full of files. Record the purpose, source URLs, allowed hostnames, excluded paths, command, tool version, start time, timezone, requester, storage location, and publication policy. If the capture will support a public claim, also record whether robots directives were checked, whether the source required authentication, and whether the output is safe to redistribute.

The manifest turns a download into a data source. It also makes the capture auditable in data storage: a future reader can distinguish the raw mirror, a cleaned derivative, a cited excerpt, and a public screenshot without guessing which copy is authoritative.

For overlanding, a scoped capture can preserve a closure notice, route advisory, forum thread, or public land-manager page exactly as it appeared when a trip decision was made. The capture should stay tied to its route note and publication policy so sensitive coordinates are not accidentally amplified.

At minimum, a manifest should answer:

• what question the capture was meant to answer;
• which URLs were intentionally in scope;
• which domains and paths were excluded;
• what command and tool version produced the files;
• when the run started and ended;
• which status codes, redirects, and failures occurred;
• whether the output may be quoted, redistributed, indexed, or only kept privately.

That last distinction matters. The right private archive can be an excellent memory aid while still being inappropriate for public search indexing.

If you only want specific file types:

CAPTURE_ROOT="<approved-archive-url>"
wget --recursive --no-parent --wait=1 --random-wait --accept jpg,jpeg,png,webp,pdf "$CAPTURE_ROOT"

This command will only download files with the specified extensions. You can add or remove extensions as needed.

For repeatable file-list captures, keep the URL list separate from the command:

wget --input-file=urls.txt --continue --timestamping --wait=1 --random-wait --output-file=wget.log

urls.txt becomes the intentional scope. wget.log becomes the retrieval history. Together they are easier to review than a shell command that discovers URLs recursively and leaves no independent list of what it was supposed to touch.

A useful mirror is not just a pile of files. Before starting, write down the source URL, date, purpose, scope, license or terms, and the exact command used. Keep the command in a shell history note, README, or capture log so the mirror can be audited later.

For research collections, preserve three layers:

• the raw URL list that defined the scope;
• the downloaded files exactly as retrieved;
• a short provenance note explaining why the material was collected.

That habit matters for OSINT, data sources, and personal web archiving because a local copy can easily outlive the context that made it meaningful.

Treat every serious wget run like a small research artifact. Keep urls.txt, wget.log, the command, the working directory, checksums for important outputs, and a short README explaining scope and exclusions. If the capture supports a public claim, record the date and timezone.

This connects wget to data storage, blogs, domains, and history. Web captures become historical evidence quickly, and the missing context is often the part that future readers need most.

A web capture becomes a dataset only after it gains structure. The raw files are one layer. A manifest of URLs, status codes, content types, timestamps, checksums, and local paths is another. Derived text, screenshots, OCR, embeddings, or extracted tables are additional layers and should not overwrite the raw capture.

Keep those layers separate in storage and prose. A quoted page, a local mirror, a parsed CSV, and a graph claim have different authority. The graph claim may say that a source page mentioned a project on a date; the mirror proves only what was retrieved; the log explains how the retrieval happened. This separation keeps OSINT, SEO, and data visualization work from accidentally treating a convenience copy as a primary source.

Use the smallest scope that answers the question. A single PDF, CSV, or documentation folder is usually better than a full domain mirror. Prefer --no-parent, --domains, --accept, --reject, --wait, and --random-wait over broad recursive defaults. When a site offers an official export, API, sitemap, dataset download, or archive bundle, use that instead of crawling page by page.

For long-running jobs, combine --continue, --timestamping, and a log file:

wget --timestamping --continue --wait=1 --random-wait --output-file=wget.log --input-file=urls.txt

The log becomes part of the artifact. It can explain missing files, redirects, failed status codes, and retry behavior.

Wget can write WARC files with --warc-file, which is useful when the goal is web preservation rather than ordinary file download. WARC captures HTTP request and response records in a format used by web archives. It is heavier than a simple folder mirror, but it preserves more evidence about what the server returned.

Use WARC when you care about replay, provenance, or long-term preservation. Use ordinary downloads when the goal is a local copy of stable files.

WARC also changes the review habit. Instead of asking only "did the file download?", ask whether the capture has enough request and response context to explain itself later: target URI, date, headers, payload, redirects, content type, and record relationship. For a serious preservation run, pair the WARC with a plain-language README so the archive is not intelligible only to replay software.

If a capture will be cited publicly, keep a stable citation path separate from the raw WARC path. The raw archive is evidence; the citation is the reader-facing pointer. They can refer to the same source event without being the same artifact.

Web retrieval is more than bytes on disk. Status codes, redirects, content types, ETags, last-modified headers, canonical links, robots directives, and server timestamps all help explain what was retrieved. When a capture is evidence, save headers or WARC records. When a capture is only a convenience copy, a log file and checksums may be enough.

This connects wget to standards and domains. HTTP behavior, DNS resolution, canonical URLs, and redirects can all change the meaning of a downloaded file. A copy made after a domain migration may not represent the same source as a copy made before it.

Metadata also protects against false precision. A successful 200 response proves that a server returned something, not that the document is current, canonical, authorized, complete, or unchanged. A 301 may be benign migration or evidence that an old citation now points somewhere else. A 404 may be deletion, temporary routing, localization, anti-abuse behavior, or a stale URL. Treat status as a clue, then compare content.

The point of web archival is not to freeze everything forever. It is to know what was available at a particular time and to keep enough evidence to compare that state with the current web. A refreshed capture should preserve what changed: new redirects, missing files, altered canonical links, updated headers, removed pages, and changed checksums.

This is where wget intersects with SEO. Broken outbound references, moved documentation, stale sitemaps, and changed canonical URLs are editorial problems as much as technical ones. A link audit should say whether a URL is still live, whether the content still means the same thing, and whether the local article should point to a newer canonical source.

wget makes retrieval easy, which is exactly why boundaries matter. Check terms, robots policy, copyright, authentication state, rate limits, and the sensitivity of the material before collecting. A public URL can still point to material that should not be republished. A signed URL, account-specific export, private map, temporary download, or leaked file is not safe just because it responds to HTTP.

Operationally, keep captures small enough to review. Use delays, respect error rates, stop on unexpected scope expansion, and avoid running broad mirrors from shared networks or production environments. If a capture is for private evidence, label it private. If it is for public citation, prefer stable canonical URLs, official archives, or institutional sources when they exist.

• --mirror: enables recursive mirroring with timestamping.
• --no-parent: prevents climbing above the starting directory.
• --domains: restricts retrieval to named hostnames.
• --page-requisites: fetches assets needed to display HTML pages locally.
• --convert-links: rewrites links for local browsing after download.
• --adjust-extension: gives saved HTML/CSS files useful extensions.
• --level: caps recursion depth.
• --reject-regex: excludes URL patterns that should not be fetched.
• --robots=on: keeps robot-exclusion behavior explicit in command notes.
• --wait and --random-wait: reduce request pressure.
• --accept and --reject: filter by extension or pattern.
• --input-file: download URLs listed in a file.
• --continue: resume an interrupted download.
• --timestamping: avoid refetching unchanged resources in refresh runs.
• --output-file: save a log as part of the capture artifact.
• --server-response: show response headers when headers are the evidence.
• --warc-file: write a WARC capture for preservation-oriented runs.

A serious web capture can become a graph node. Useful fields include source URL, canonical URL, retrieval time, command, status code, content type, checksum, license note, referring page, redirect chain, and local storage path. Those fields let a graph connect a captured PDF to a data source, a blog, a domain, or an OSINT case file without pretending that a downloaded copy is the original authority.

For this compendium, wget belongs near the practical edge of the Semantic Web: it helps preserve human-readable pages, but the capture becomes truly useful only when metadata, provenance, and links are kept with the files.

Useful graph edges include retrieved_from, redirected_to, captured_at, saved_as, checksummed_as, governed_by, cited_by, superseded_by, refreshed_by, excluded_path, and derived_from. Those edges let a later reader distinguish a source page, a capture event, a local file, a cleaned dataset, a quotation, and a claim. Without those distinctions, a graph can accidentally launder a local mirror into an authority record.

The graph should also preserve absences. Failed URLs, excluded paths, robots-blocked regions, missing assets, and JavaScript-only content are not just errors; they are evidence about what the capture does not contain. That makes refreshes honest and prevents a mirror from looking more complete than it is.

Do not use wget as a shortcut around a better source interface. Prefer official exports for social platforms, package registries, government data portals, and documentation sites that publish versioned archives. Prefer APIs when freshness, pagination, or structured fields matter. Prefer a browser-based capture when the content exists only after client-side rendering. Prefer a citation or screenshot when the goal is to quote one page, not preserve a subtree.

Also avoid wget for ambiguous private/public boundaries. If a URL includes a token, account identifier, session parameter, private map link, signed download, or personalized view, stop and decide whether the capture belongs in a private evidence folder rather than a compendium note. A clean command is not the same as a clean boundary.

• Accidentally crawling outside the intended path.
• Downloading dynamic pages that require JavaScript and do not replay well.
• Hammering a small site because no delay was configured.
• Preserving files without source dates, commands, or provenance.
• Treating a local mirror as permission to republish copyrighted material.
• Assuming that a successful HTTP response means the content is authoritative.
• Losing the retrieval command, which makes the mirror impossible to audit or refresh.
• Capturing a logged-in or personalized view and mistaking it for a public source.

• Data Sources for source quality, freshness, and provenance.
• OSINT for careful public-source collection.
• Blogs and domains for fragile personal-web material.
• Hackers for command-line craft and practical tool use.
• Data Storage for manifests, checksums, raw mirrors, and derived archives.
• SEO for link audits, canonical URLs, redirects, and crawl health.
• Standards for HTTP, robots, WARC, and interoperable retrieval metadata.
• Overlanding for preserving route advisories, source pages, and field-report provenance.
• Rules of Thumb for keeping capture scope smaller than curiosity.
• Python and Good Libraries for post-processing, manifests, and repeatable data pipelines.
• GitHub for versioning capture scripts, manifests, and source lists without committing bulky private mirrors.

• GNU Wget for the project homepage.
• GNU Wget Manual for command behavior and options.
• GNU Wget robot exclusion documentation for robot-exclusion behavior.
• GNU Wget WARC options for preservation-oriented capture options.
• GNU Wget FAQ for practical usage notes.
• GNU Wget2 repository for the newer Wget-family implementation.
• RFC 9110: HTTP Semantics for HTTP method, status, and representation semantics.
• RFC 9309: Robots Exclusion Protocol for the standardized robots.txt protocol.
• RFC 9564: The WARC Format for web-archive record structure.
• WARC 1.1 specification mirror for archive-community access to the WARC format.
• Library of Congress WARC format description for preservation-format context.
• Library of Congress Sustainability of Digital Formats for preservation-format evaluation.
• DBpedia: Wget for linked-data context.
• Wikidata: GNU Wget for the public entity identifier used by this page.