Prepare URLs for a Sitemap

Clean the inventory before building XML

URL inventories often arrive as copied navigation links, analytics exports, crawl results, CMS records, or comma-separated notes. Before those addresses enter a sitemap generator, they need a consistent one-URL-per-line shape and a deliberate canonical policy.

This workflow is for site owners, SEO practitioners, developers, and content teams preparing a bounded URL list. The Tools.Town steps perform list cleanup only. They do not generate sitemap XML, validate XML syntax, crawl pages, check HTTP status codes, test indexability, or submit a sitemap to a search engine.

Establish the canonical form first

Text deduplication can remove identical strings, but these strings are all different:

http://example.com/guides
https://example.com/guides
https://www.example.com/guides
https://example.com/guides/
https://example.com/guides?utm_source=newsletter

Your site’s canonical URL policy must determine which form belongs in the sitemap. Review:

• Protocol: normally the canonical production https:// URL, not an http:// variant.
• Host: use the canonical www or non-www hostname, not both.
• Trailing slash: follow the URLs emitted by canonical tags and routing redirects.
• Parameters: exclude tracking parameters and usually exclude filter, sort, session, and preview parameters unless a parameterized page is intentionally canonical and indexable.
• Fragments: remove #section fragments; fragments identify locations within a document, not separate sitemap pages.
• Case and encoding: preserve the actual canonical path. Do not lowercase paths or rewrite percent-encoding unless the application guarantees equivalence.

These are editorial and technical decisions, not capabilities of a list formatter.

Example cleanup target

Raw input:

https://example.com/about, https://example.com/guides/,https://example.com/contact
https://example.com/about
https://example.com/guides/?utm_source=email

After applying the site’s policy—HTTPS, no tracking parameters, canonical trailing slash for directory routes—the desired list might be:

https://example.com/about/
https://example.com/contact/
https://example.com/guides/

The tools can convert delimiters, remove exact duplicate lines, format, and sort. They cannot infer that /about should redirect to /about/ or that the utm_source version canonically resolves to the clean URL. Make those corrections from routing and canonical evidence.

A precise URL-list workflow

1. Keep the raw export. Record its source and date. Different inputs may contain URLs outside the intended sitemap scope.
2. Define inclusion rules. Include canonical, indexable pages that you want search engines to discover. Exclude admin, login, cart, internal search, staging, preview, duplicate, redirected, error, and non-canonical URLs as appropriate for the site.
3. Convert a flat comma-separated dump. If the input is simply URLs separated by commas, use Comma Separated to List to place one value on each line. Do not use it as a complete CSV parser when rows contain quoted fields or multiple columns.
4. Apply canonical corrections outside blind list operations. Fix protocol, hostname, path, trailing slash, fragments, and parameters according to the documented site policy.
5. Remove exact repeats with Remove Duplicates from List. Near-duplicates remain until their text is made canonical.
6. Use List Formatter if you need a clean one-line-per-item representation. Check that it has not introduced bullets, numbering, quotes, or delimiters that your sitemap generator will treat as URL characters.
7. Optionally use Alphabetize List. Sorting makes manual review and diffing easier, but XML sitemaps do not require alphabetical URL order.
8. Pass the approved list to your real sitemap generator. Use the CMS, framework integration, or dedicated XML process responsible for <urlset>, namespaces, escaping, optional metadata, and file splitting.
9. Validate and test separately. Validate generated XML, fetch sample URLs, review redirects and canonical tags, and confirm sitemap accessibility. Then submit or reference it through the site’s normal search-engine workflow.

Pre-generator verification checklist

• Every line is an absolute root URL with the canonical protocol and hostname.
• www and non-www variants are not mixed.
• Trailing slashes match the site’s canonical and redirect policy.
• Tracking parameters, fragments, preview tokens, and session values are absent.
• No staging, localhost, development, admin, account, or internal-search URLs remain.
• Exact duplicates are removed and near-duplicate forms were reviewed.
• The list includes only the intended locale, subdomain, and content scope.
• Redirected, missing, blocked, noindex, and non-canonical pages were checked using appropriate crawling or site tools.
• The cleaned URL count is plausible compared with the CMS or source inventory.

Edge cases and common sitemap mistakes

International sites may use locale paths, subdomains, or alternate hosts. Do not collapse them simply because titles are similar. Large sites may need multiple sitemap files and a sitemap index; list chunking and XML size limits belong in the generator workflow, not an improvised text split.

Images, videos, and news content have specialized sitemap requirements. A plain URL list does not create their required XML fields. Likewise, adding lastmod dates requires trustworthy source data and valid formatting; a text list cannot invent or verify modification timestamps.

Do not assume that a URL’s presence means it is reachable or indexable. Cleanup never checks robots.txt, meta robots, authentication, server responses, canonical tags, or rendering.

Privacy, scope, and next steps

URL exports can expose unpublished slugs, preview tokens, internal paths, customer identifiers, or private document names. Remove secrets and restricted URLs before using a web tool. Never place authenticated or tokenized URLs in a public sitemap.

After cleanup, generate and validate XML with the site’s actual sitemap system. Use List Ops for other bounded text-list tasks, but use a crawler, framework, CMS, XML validator, and search-engine tooling for sitemap correctness and coverage.

Related next steps

• List Ops hub