Sitemaps for a site in two languages

What a sitemap is and is not

It is a list of the URLs you want crawled, in a schema Google and Bing both read. It does not make anything rank. What it does is make pages findable — and that matters most exactly where it is least visible: deep pages, new pages, pages nothing links to.

Without alternates, your languages compete

Publish the same page in English and Arabic with no signal connecting them and Google sees two documents covering the same subject on one domain. It picks one. Which one is not up to you, and the other largely disappears.

The fix is an xhtml:link for every language a URL exists in — including itself:

<url>
  <loc>https://example.com/page/</loc>
  <xhtml:link rel="alternate" hreflang="en"
              href="https://example.com/page/"/>
  <xhtml:link rel="alternate" hreflang="ar"
              href="https://example.com/ar/page/"/>
  <xhtml:link rel="alternate" hreflang="x-default"
              href="https://example.com/page/"/>
</url>

And the same three lines appear again under the Arabic <loc>, unchanged. That repetition is not waste — it is the reciprocity the whole mechanism depends on, and a cluster missing it is discarded whole.

Absolute, always

href="/ar/page/" parses. It validates. It is thrown away.

This site shipped 936 of them — 312 pages, three alternates each — and Search Console reported a number with no indication of which part was at fault. The number went to zero the day they were made absolute. That is the entire diagnostic trail available, which is why it is worth knowing the rule rather than discovering it.

The <urlset> element also needs the namespace declared, or the alternates are ignored as unknown elements:

<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
        xmlns:xhtml="http://www.w3.org/1999/xhtml">

The fields that do nothing

changefreq and priority have been ignored by Google for years. Emit them if you like; nothing reads them, and a file full of fields nothing reads is a file nobody maintains.

lastmod is different — Google does use it, but only from sites where it is honest. Stamp every URL with today's date on every build and it is discounted entirely, including for the pages that genuinely did change.

What not to include

Pages carrying noindex, and pages blocked in robots.txt. Both send two opposite instructions at once, and Search Console files the result as a coverage error rather than guessing which one you meant.

The URL has to match exactly

A sitemap entry is compared against the URL Google already knows, character for character. Four mismatches account for most of the errors people see in the sitemap report, and all four look identical to the eye:

• http:// in the sitemap when the site redirects to https://
• www. in one and not the other
• a trailing slash in one and not the other
• a URL on a different host entirely, which is simply ignored

Each of these turns an entry into a redirect rather than a page, and Search Console reports it as one. The fix is always the same: list the URL exactly as the canonical tag on the page states it. If the two disagree, the canonical is right and the sitemap is wrong.

Past fifty thousand: the index file

A single sitemap holds 50,000 URLs or 50MB uncompressed. Beyond that you split it and list the parts in a sitemap index — a different file with a different root element, <sitemapindex> rather than <urlset>, and entries that point at sitemaps rather than at pages.

Splitting is worth doing before you have to. Sitemaps divided by section — one for products, one for articles — turn the Search Console coverage report from a single number into a diagnosis, because you can see which part of the site is not being indexed rather than only that some of it is not.

What to check after you submit it

The sitemap report in Search Console gives a read count and a discovery date. A count far below what you submitted means URLs were rejected — usually one of the four mismatches above.

The other thing to watch is "Discovered — currently not indexed". That is Google saying it found the URL in your sitemap and chose not to fetch it, which is a statement about the page rather than about the file. A sitemap gets a page looked at; it cannot make a page worth keeping.

After it is written

Reference it from robots.txt and submit it in Search Console — both, not either. The robots.txt line is what every other crawler reads.

The sitemap generator takes a list of URLs, pairs each with its translation from the path prefix, and writes the full cluster including the self-reference. It refuses relative URLs rather than emitting them.