closed-social
/
mastodon
3
0
Fork 2
Code Issues 5 Pull Requests 0 Projects 0 Releases 3 Wiki Activity
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
10161 Commits
4 Branches
567 MiB
Tree: 5614e6724e
Branches Tags
${ item.name }
Create branch ${ searchTerm }
from '5614e6724e'
${ noResults }
mastodon/public/robots.txt

5 lines
158 B
Raw Normal View History

Change robots.txt to exclude only media proxy URLs (#10038) * Revert "Change robots.txt to exclude some URLs (#10037)" This reverts commit 80161f43510ad9316c60c9b50dd5c09c2dae4d54. * Let's block media_proxy /media_proxy/ is a dynamic route used for requesting uncached media, so it's probably bad to let crawlers use it * misleading comment
5 years ago
​
​
​
Disallow robots from indexing /interact/ (#10666) This does not provide any new information and may just triple the number of crawled pages
5 years ago
 
  1. # See http://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file
  2. 

  3. User-agent: *
  4. Disallow: /media_proxy/
  5. Disallow: /interact/