<!doctype html><htmllang=enclass=no-js><head><metacharset=utf-8><metaname=viewportcontent="width=device-width,initial-scale=1"><metaname=descriptioncontent="An open source, self-hosted implementation of the Tailscale control server."><metaname=authorcontent="Headscale authors"><linkhref=https://juanfont.github.io/headscale/development/ref/debug/rel=canonical><linkhref=../remote-cli/rel=prev><linkhref=../integration/reverse-proxy/rel=next><linkrel=iconhref=../../assets/favicon.png><metaname=generatorcontent="mkdocs-1.6.1, mkdocs-material-9.6.16"><title>Debug - Headscale</title><linkrel=stylesheethref=../../assets/stylesheets/main.7e37652d.min.css><linkrel=stylesheethref=../../assets/stylesheets/palette.06af60db.min.css><linkrel=preconnecthref=https://fonts.gstatic.comcrossorigin><linkrel=stylesheethref="https://fonts.googleapis.com/css?family=Roboto:300,300i,400,400i,700,700i%7CRoboto+Mono:400,400i,700,700i&display=fallback"><style>:root{--md-text-font:"Roboto";--md-code-font:"Roboto Mono"}</style><script>__md_scope=newURL("../..",location),__md_hash=e=>[...e].reduce(((e,_)=>(e<<5)-e+_.charCodeAt(0)),0),__md_get=(e,_=localStorage,t=__md_scope)=>JSON.parse(_.getItem(t.pathname+"."+e)),__md_set=(e,_,t=localStorage,a=__md_scope)=>{try{t.setItem(a.pathname+"."+e,JSON.stringify(_))}catch(e){}}</script><metaproperty=og:typecontent=website><metaproperty=og:titlecontent="Debug - Headscale"><metaproperty=og:descriptioncontent="An open source, self-hosted implementation of the Tailscale control server."><metaproperty=og:imagecontent=https://juanfont.github.io/headscale/development/assets/images/social/ref/debug.png><metaproperty=og:image:typecontent=image/png><metaproperty=og:image:widthcontent=1200><metaproperty=og:image:heightcontent=630><metacontent=https://juanfont.github.io/headscale/development/ref/debug/property=og:url><metaname=twitter:cardcontent=summary_large_image><metaname=twitter:titlecontent="Debug - Headscale"><metaname=twitter:descriptioncontent="An open source, self-hosted implementation of the Tailscale control server."><metaname=twitter:imagecontent=https://juanfont.github.io/headscale/development/assets/images/social/ref/debug.png></head><bodydir=ltrdata-md-color-scheme=defaultdata-md-color-primary=whitedata-md-color-accent=indigo><inputclass=md-toggledata-md-toggle=drawertype=checkboxid=__drawerautocomplete=off><inputclass=md-toggledata-md-toggle=searchtype=checkboxid=__searchautocomplete=off><labelclass=md-overlayfor=__drawer></label><divdata-md-component=skip><ahref=#debugging-and-troubleshootingclass=md-skip> Skip to content </a></div><divdata-md-component=announce></div><divdata-md-color-scheme=defaultdata-md-component=outdatedhidden></div><headerclass=md-headerdata-md-component=header><navclass="md-header__inner md-grid"aria-label=Header><ahref=../..title=Headscaleclass="md-header__button md-logo"aria-label=Headscaledata-md-component=logo><imgsrc=../../logo/headscale3-dots.svgalt=logo></a><labelclass="md-header__button md-icon"for=__drawer><svgxmlns=http://www.w3.org/2000/svgviewbox="0 0 24 24"><pathd="M3 6h18v2H3zm0 5h18v2H3zm0 5h18v2H3z"/></svg></label><divclass=md-header__titledata-md-component=header-title><divclass=md-header__ellipsis><divclass=md-header__topic><spanclass=md-ellipsis> Headscale </span></div><divclass=md-header__topicdata-md-component=header-topic><spanclass=md-ellipsis> Debug </span></div></div></div><formclass=md-header__optiondata-md-component=palette><inputclass=md-optiondata-md-color-mediadata-md-color-scheme=defaultdata-md-color-primary=whitedata-md-color-accent=indigoaria-label="Switch to dark mode"type=radioname=__paletteid=__palette_0><labelclass="md-header__button md-icon"title="Switch to dark mode"for=__palette_1hidden><svgxmlns=http://www.w3.org/2000/svgviewbox="0 0 24 24"><pathd="M128a44000-444400044440004-444000-4-4m010a66001-6-6660016-6660016666001-66m8-9.31V4h-4.69L12.69
</span></span></code></pre></div><h3id=database-logging>Database logging<aclass=headerlinkhref=#database-loggingtitle="Permanent link">¶</a></h3><p>The database debug mode logs all database queries. Enable it to see how Headscale interacts with its database. This also requires the application log level to be set to either <code>debug</code> or <code>trace</code>.</p><divclass="language-yaml highlight"><pre><span></span><code><spanid=__span-1-1><aid=__codelineno-1-1name=__codelineno-1-1href=#__codelineno-1-1></a><spanclass=nt>database</span><spanclass=p>:</span>
</span><spanid=__span-1-2><aid=__codelineno-1-2name=__codelineno-1-2href=#__codelineno-1-2></a><spanclass=w></span><spanclass=c1># Enable debug mode. This setting requires the log.level to be set to "debug" or "trace".</span>
</span></span></code></pre></div><h3id=metrics-and-debug-endpoint>Metrics and debug endpoint<aclass=headerlinkhref=#metrics-and-debug-endpointtitle="Permanent link">¶</a></h3><p>Headscale provides a metrics and debug endpoint. It allows to introspect different aspects such as:</p><ul><li>Information about the Go runtime, memory usage and statistics</li><li>Connected nodes and pending registrations</li><li>Active ACLs, filters and SSH policy</li><li>Current DERPMap</li><li>Prometheus metrics</li></ul><divclass="admonition warning"><pclass=admonition-title>Keep the metrics and debug endpoint private</p><p>The listen address and port can be configured with the <code>metrics_listen_addr</code> variable in the <ahref=../configuration/>configuration file</a>. By default it listens on localhost, port 9090.</p><p>Keep the metrics and debug endpoint private to your internal network and don't expose it to the Internet.</p></div><p>Query metrics via <ahref=http://localhost:9090/metrics>http://localhost:9090/metrics</a> and get an overview of available debug information via <ahref=http://localhost:9090/debug/>http://localhost:9090/debug/</a>. Metrics may be queried from outside localhost but the debug interface is subject to additional protection despite listening on all interfaces.</p><divclass="tabbed-set tabbed-alternate"data-tabs=1:4><inputchecked=checkedid=__tabbed_1_1name=__tabbed_1type=radio><inputid=__tabbed_1_2name=__tabbed_1type=radio><inputid=__tabbed_1_3name=__tabbed_1type=radio><inputid=__tabbed_1_4name=__tabbed_1type=radio><divclass=tabbed-labels><labelfor=__tabbed_1_1>Direct access</label><labelfor=__tabbed_1_2>SSH port forwarding</label><labelfor=__tabbed_1_3>Via debug key</label><labelfor=__tabbed_1_4>Via debug IP address</label></div><divclass=tabbed-content><divclass=tabbed-block><p>Access the debug interface directly on the server where Headscale is installed.</p><divclass="language-console highlight"><pre><span></span><code><spanid=__span-2-1><aid=__codelineno-2-1name=__codelineno-2-1href=#__codelineno-2-1></a><spanclass=go>curl http://localhost:9090/debug/</span>
</span></code></pre></div></div><divclass=tabbed-block><p>Use SSH port forwarding to forward Headscale's metrics and debug port to your device.</p><divclass="language-console highlight"><pre><span></span><code><spanid=__span-3-1><aid=__codelineno-3-1name=__codelineno-3-1href=#__codelineno-3-1></a><spanclass=go>ssh <HEADSCALE_SERVER> -L 9090:localhost:9090</span>
</span></code></pre></div><p>Access the debug interface on your device by opening <ahref=http://localhost:9090/debug/>http://localhost:9090/debug/</a> in your web browser.</p></div><divclass=tabbed-block><p>The access control of the debug interface supports the use of a debug key. Traffic is accepted if the path to a debug key is set via the environment variable <code>TS_DEBUG_KEY_PATH</code> and the debug key sent as value for <code>debugkey</code> parameter with each request.</p><divclass="language-console highlight"><pre><span></span><code><spanid=__span-4-1><aid=__codelineno-4-1name=__codelineno-4-1href=#__codelineno-4-1></a><spanclass=go>openssl rand -hex 32 | tee debugkey.txt</span>
</span></code></pre></div><p>Access the debug interface on your device by opening <code>http://<IP_OF_HEADSCALE>:9090/debug/?debugkey=<DEBUG_KEY></code> in your web browser. The <code>debugkey</code> parameter must be sent with every request.</p></div><divclass=tabbed-block><p>The debug endpoint expects traffic from localhost. A different debug IP address may be configured by setting the <code>TS_ALLOW_DEBUG_IP</code> environment variable before starting Headscale. The debug IP address is ignored when the HTTP header <code>X-Forwarded-For</code> is present.</p><divclass="language-console highlight"><pre><span></span><code><spanid=__span-5-1><aid=__codelineno-5-1name=__codelineno-5-1href=#__codelineno-5-1></a><spanclass=go>export TS_ALLOW_DEBUG_IP=192.168.0.10 # IP address of your device</span>
