opendev/system-config
Code Issues Proposed changes

Merge "Update our Gitea robots.txt from gitea.com's"

Browse Source
This commit is contained in:
Zuul 2024-01-18 19:27:16 +00:00 committed by Gerrit Code Review
commit 2d27aa98a8
1 changed files with 65 additions and 8 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

73
docker/gitea/custom/robots.txt
View File

@ -3,6 +3,7 @@
# and # and
# https://github.com/robots.txt # https://github.com/robots.txt
# at 2020-07-01 # at 2020-07-01
# and https://gitea.com/robots.txt on 2024-01-05
# #
# Some commented out items are left to indicate we have considered # Some commented out items are left to indicate we have considered
# them and would like to explicitly allow them for indexing while they # them and would like to explicitly allow them for indexing while they
@ -10,26 +11,82 @@
User-agent: * User-agent: *
# Disallow: /avatars Disallow: /api/*
# Disallow: /user/* Disallow: /avatars
Disallow: /user/*
# Disallow: /*/*/src/commit/* # Disallow: /*/*/src/commit/*
# Disallow: /*/*/commit/* # Disallow: /*/*/commit/*
# Disallow: /*/*/*/refs/*
Disallow: /*/*/*/star
Disallow: /*/*/*/watch
Disallow: /*/*/labels
Disallow: /*/*/activity/* Disallow: /*/*/activity/*
Disallow: /vendor/librejs.html Disallow: /vendor/*
Disallow: /api/swagger
Disallow: /swagger.*.json Disallow: /swagger.*.json
# Language spam # Language spam
Disallow: /*?lang= Disallow: /*?lang=
# From github # from Github, to be cleaned
Disallow: */archive/ Allow: /*/*/tree/master
Disallow: */blame/ Allow: /*/*/blob/master
Disallow: /*/*/pulse
Disallow: /*/*/tree/*
Disallow: /*/*/blob/*
Disallow: /*/*/wiki/*/*
Disallow: /gist/*/*/*
Disallow: /oembed
Disallow: /*/forks
Disallow: /*/stars
Disallow: /*/download
Disallow: /*/revisions
Disallow: /*/*/issues/new
Disallow: /*/*/issues/search
Disallow: /*/*/commits/*/*
Disallow: /*/*/commits/*?author
Disallow: /*/*/commits/*?path
Disallow: /*/*/branches
Disallow: /*/*/tags
Disallow: /*/*/contributors
Disallow: /*/*/comments
Disallow: /*/*/stargazers
Disallow: /*/*/search
Disallow: /*/tarball/
Disallow: /*/zipball/
Disallow: /*/*/archive/
# Disallow: /raw/* # Disallow: /raw/*
Disallow: /*/followers
Disallow: /*/following
Disallow: /stars/*
Disallow: /*/blame/
Disallow: /*/watchers
Disallow: /*/network
Disallow: /*/graphs
# Disallow: /*/raw/
Disallow: /*/compare/
Disallow: /*/cache/
Disallow: /*/*/blame/
Disallow: /*/*/watchers
Disallow: /*/*/network
Disallow: /*/*/graphs
# Disallow: /*/*/raw/
Disallow: /*/*/compare/
Disallow: /*/*/cache/
Disallow: /.git/ Disallow: /.git/
Disallow: */.git/ Disallow: /*/.git/
Disallow: /*.git$ Disallow: /*.git$
Disallow: /*/sitemap.xml
Disallow: /search/advanced
Disallow: /search
Disallow: /*q= Disallow: /*q=
Disallow: /*.atom
Crawl-delay: 2 Crawl-delay: 2