Browse Source
Change robots.txt to exclude some URLs (#10037)
- Exclude static assets
- Exclude uploaded files
- Exclude alternate versions of the profile page
- Exclude media proxy URLs
pull/4/head
Eugen Rochko
5 years ago
committed by
GitHub
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with
13 additions and
5 deletions
-
public/robots.txt
|
|
@ -1,5 +1,13 @@ |
|
|
|
# See http://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file |
|
|
|
# |
|
|
|
# To ban all spiders from the entire site uncomment the next two lines: |
|
|
|
# User-agent: * |
|
|
|
# Disallow: / |
|
|
|
User-Agent: * |
|
|
|
Disallow: /users/*/followers |
|
|
|
Disallow: /users/*/following |
|
|
|
Disallow: /@*/media |
|
|
|
Disallow: /@*/with_replies |
|
|
|
Disallow: /@*/tagged/* |
|
|
|
Disallow: /media_proxy/* |
|
|
|
Disallow: /emoji/* |
|
|
|
Disallow: /packs/* |
|
|
|
Disallow: /sounds/* |
|
|
|
Disallow: /system/* |
|
|
|
Disallow: /avatars/* |
|
|
|
Disallow: /headers/* |