#Grafana

Grafana is a web-based UI for visualizing data. It can consume data from many different sources, and it's particularly well matched with Prometheus. You might use the Prometheus UI to fine-tune your queries, and then you'd take the PromQL and use it in Grafana.

The screenshots and the written steps in this lab were both taken on Grafana 13.2.4, the version the containers run. Menu labels do move between Grafana releases - if something has shifted again, the Grafana docs are authoritative.

#Reference

• Using Prometheus as a data source
• Panels - individual query visualizations
• Dashboards - combining multiple panels

#Run Grafana

Grafana is a cross-platform app, written in Go. You can download Grafana for different platforms - but we'll run in a Docker container:

• grafana.yml - specifies containers for Prometheus and Grafana
• prometheus.yml - the Prometheus config we'll be using; sets up scraping for various apps, and adds the tier label

Run the metrics components:

docker compose -f labs/grafana/grafana.yml up -d

Check the Prometheus status at http://localhost:9090/targets

You'll see all the targets are down, but the tier label is set along with instance and job. You can add to target labels like this to include infrastructure-specific details; you might include the region for cloud services, or the operating system and CPU architecture.

Browse to Grafana at http://localhost:3000

Sign in with the default credentials - you'll be asked to create a new password, but you can skip that step:

• username: admin
• password: admin

In current Grafana you add a data source from Connections > Data sources:

📋 Add a Prometheus data source, with the URL http://prometheus:9090 and test it.

That's all the setup you need - now you can create a dashboard with panels to query Prometheus.

#Create an uptime panel

Open Dashboards from the left nav, then New > New dashboard. You'll see an empty dashboard like this:

Every visualization is a panel within a dashboard.

📋 Add a new panel to show the uptime of Prometheus targets using the up metric.

The panel designer is in three sections:

• visualization preview at the top
• data on the bottom, including the query and transforms
• display options on the right, including visualization type

The defaults are to use a Time series visualization, representing a range vector from Prometheus.

Now it's time to find your way around 🙂

📋 Change the panel setup so Grafana treats this as a table showing an instant vector, using a table visualization. Set up the table so the instance name and status are the only fields, showing ok for instances which are up and OFFLINE for others.

Here's what you're aiming for:

Grafana dashboards can be set to auto-refresh. Select to refresh every 5 seconds (top-right of the dashboard), so when we start the apps you'll see the table update as they come online.

#Visualization Types

Start the apps, so Prometheus will have some metrics to collect:

• apps.yml - specifies the node exporter, document API and three document processors; one of the processors is configured to keep exiting and restarting.

Run the apps:

docker compose -f labs/grafana/apps.yml up -d

Switch back to Grafana - you'll see the table update as instances come online.

If you watch long enough you'll see processor instance 1 flicks between ok and OFFLINE. The table we have shows the current status, and it would also be useful to see the history of each component's uptime.

Click Add in the top menu and choose Panel to add a second panel.

Last time we used a transform to hide labels, but we can do that in the query too:

• set the query to be sum without(job, tier) (up)
• change the visualization type to State timeline
• add the same value mapping to show 0 as OFFLINE and 1 as ok
• change the time range to (e.g.) last 15 minutes

We're getting there, just some tidying up to do.

📋 Edit the query and display options so the legend on the y-axis is more useful, hide the x-axis legend, and show OFFLINE as a red bar and ok as green.

This is the goal:

That's better - the dashboard now shows current and historical status for all the instances. We can see there's a recurring problem with processor 1, which isn't clear from the table alone.

#A Real Dashboard

Grafana can be fiddly to work with - but once you have your dashboard designed, you can load it into any Grafana server.

In the lab we'll carry on with a custom dashboard, but so you can see what a real dashboard looks like, there's one stored in the file labs/grafana/dashboards/app.json you can load:

• open Dashboards from the left nav, then New > Import
• click Upload dashboard JSON file and navigate to the path for the

  app.json file

• select your Prometheus data source if you are prompted, then click Import

These visualizations are all powered by the metrics from the container apps you have running. You can reduce the timescale from the dropdown in the top-right and show metrics from the last 30 minutes to see how Grafana shows trends.

#Lab

You'll have several dashboards for your apps, showing information at different levels.

For this lab we'll stick with instance uptime, but your job is to split the dashboard into multiple rows, one for each tier of the application:

• an infrastructure row with the node exporter instance
• a web row with the document API instance
• a backend row with the processor instances

Each row should have a current status table and an timeline for the instances in that tier, something like this:

Save your dashboard as JSON so you can load it into another Grafana instance later.

Stuck? Try hints or check the solution.

#Cleanup

Clean up by removing this lab's containers:

docker compose -f labs/grafana/grafana.yml down
docker compose -f labs/grafana/apps.yml down

This removes only the containers this lab started. If you want a clean slate, docker ps will show anything still running.