* Revert "Change robots.txt to exclude some URLs (#10037)"
This reverts commit 80161f4351.
* Let's block media_proxy
/media_proxy/ is a dynamic route used for requesting uncached media, so it's
probably bad to let crawlers use it
* misleading comment