In Part 1 of this blog, I demonstrated how easy it is to deploy Prometheus and Grafrana to a cluster using Portainer.
But what if you have multiple clusters? Do you want an instance per cluster? Wouldn't you rather visualize all of them in a single Grafana dashboard?
Let me show you how.
Login to Portainer, and choose the SECOND cluster (the one that is NOT running Prometheus stack).
Click on 'Namespaces', and create a namespace called "Prometheus"
Go into Helm, select the "kube-prometheus-stack", edit the custom values, find line 641, and change true to false.
Also edit lines 2427 - 2436 if you want Prometheus persistent.
Click "Install"
In the application list, click on the prometheus application
Click "Edit this application"
Add a new "NodePort", with 9090 exposed as 32767 (or any free port), then click "update application"
NOTE THIS EXPOSES YOUR PROMETHEUS INSTANCE TO THE PUBLIC. I STRONGLY ADVOCATE ONLY DOING THIS FOR LAN ATTACHED CLUSTERS. SEE HERE FOR MORE INFO
Login to your Grafana instance, then click on "Configuration" then "Data Sources", then click "add data source"
Choose Prometheus..
Add in the IP address of one of your nodes, and the NodePort you set previously.
Click "save and test" and make sure you get the green tick.
Go to dashboards, select a dashboard, and change the datasource to the one you just added.
Success. You are now viewing the remote Prometheus instance from your central Grafana instance.