improve robots.txt, update sitemap.xml

Search engines become confused when they discover pages such
as under `inc/head.default.html` that are **NOT** real pages,
but rather quasi-templates that are used by bin tools in the
generation of HTML content.  So we "_Disallow_" search engine
access to URLs under `inc/` except for the two "safe" pages.

Updated `sitemap.xml` as per recent modifications.
This commit is contained in:
Landon Curt Noll committed 2025-04-06 18:13:50 -07:00
1 parent 49068e4c5b
commit 84017fb0c7
2 files changed
+45 -43

No files matched your search

+2
View File
@@ -1,3 +1,5 @@
User-Agent: *
Allow: /inc/index.html
Disallow: /inc/
Allow: /
Sitemap: https://www.ioccc.org/sitemap.xml