fix: introduce measures to avoid bots crawling and indexing activities (#5728)

* fix: 404 for robots.txt and meta robots tags

* fix: add unit tests for robots txt and tag

* fix: add meta tag robots none for login pages

* fix: weird format issue in header.go

* fix: add x-robots-tag=none to grpcwebserver

* fix linting

---------

Co-authored-by: Silvan <silvan.reusser@gmail.com>
Co-authored-by: Livio Spring <livio.a@gmail.com>

internal/api/ui/login/static/templates/main.html

@@ -23,6 +23,7 @@
 
     <title>{{ .Title }}</title>
     <meta name="description" content="{{ .Description }}"/>
+    <meta name="robots" content="none" />
 
     <script src="{{ resourceUrl "scripts/theme.js" }}"></script>
 </head>