#Grafana

Grafana is a web-based UI for visualizing data. It can consume data from many different sources, and it's particularly well matched with Prometheus. You might use the Prometheus UI to fine-tune your queries, and then you'd take the PromQL and use it in Grafana.

#Reference

• Using Prometheus as a data source
• Panels - individual query visualizations
• Dashboards - combining multiple panels

#Run Grafana

Grafana is a cross-platform app, written in Go. You can download Grafana for different platforms - but we'll run in a Docker container:

• grafana.yml - spcifies containers for Prometheus and Grafana
• prometheus.yml - the Prometheus config we'll be using; sets up scraping for various apps, and adds the tier label

Run the metrics components:

docker compose -f labs/grafana/grafana.yml up -d

Check the Prometheus status at http://localhost:9090/targets

You'll see all the targets are down, but the tier label is set along with instance and job. You can add to target labels like this to include infrastructure-specific details; you might include the region for cloud services, or the operating system and CPU architecture.

Browse to Grafana at http://localhost:3000

Sign in with the default credentials - you'll be asked to create a new password, but you can skip that step:

• username: admin
• password: admin

The homepage has a link to create a data source:

📋 Add a Prometheus data source, with the URL http://prometheus:9090 and test it.

That's all the setup you need - now you can create a dashboard with panels to query Prometheus.

#Create an uptime panel

Click on the plus icon from the left menu and select Create...Dashboard. You'll see a screen like this:

Every visualization is a panel within a dashboard.

📋 Add a new panel to show the uptime of Prometheus targets using the up metric.

The panel designer is in three sections:

• visualization preview at the top
• data on the bottom, including the query and transforms
• display options on the right, including visualization type

The defaults are to use a Time series visualization, representing a range vector from Prometheus.

Now it's time to find your way around 🙂

📋 Change the panel setup so Grafana treats this as a table showing an instant vector, using a table visualization. Set up the table so the instance name and status are the only fields, showing ok for instances which are up and OFFLINE for others.

Here's what you're aiming for:

Grafana dashboards can be set to auto-refresh. Select to refresh every 5 seconds (top-right of the dashboard), so when we start the apps you'll see the table update as they come online.

#Visualization Types

Start the apps, so Prometheus will have some metrics to collect:

• apps.yml - specifies the node exporter, document API and three document processors; one of the processors is configured to keep exiting and restarting.

Run the apps:

docker compose -f labs/grafana/apps.yml up -d

Switch back to Grafana - you'll see the table update as instances come online.

If you watch long enough you'll see processor instance 1 flicks between ok and OFFLINE. The table we have shows the current status, and it would also be useful to see the history of each component's uptime.

Click the add panel icon in the top menu (it's the bar chart with a plus sign), and select Add an empty panel.

Last time we used a transform to hide labels, but we can do that in the query too:

• set the query to be sum without(job, tier) (up)
• change the visualization type to State timeline
• add the same value mapping to show 0 as OFFLINE and 1 as ok
• change the time range to (e.g.) last 15 minutes

We're getting there, just some tidying up to do.

📋 Edit the query and display options so the legend on the y-axis is more useful, hide the x-axis legend, and show OFFLINE as a red bar and ok as green.

This is the goal:

That's better - the dashboard now shows current and historical status for all the instances. We can see there's a recurring problem with processor 1, which isn't clear from the table alone.

#Lab

You'll have several dashboards for your apps, showing information at different levels.

For this lab we'll stick with instance uptime, but your job is to split the dashboard into multiple rows, one for each tier of the application:

• an infrastructure row with the node exporter instance
• a web row with the docunent API instance
• a backend row with the processor instances

Each row should have a current status table and an timeline for the instances in that tier, something like this:

Save your dashboard as JSON so you can load it into another Grafana instance later.

Stuck? Try hints or check the solution.

#Cleanup

Clean up by removing this lab's containers:

docker compose -f labs/grafana/grafana.yml down
docker compose -f labs/grafana/apps.yml down

This removes only the containers this lab started. If you want a clean slate, docker ps will show anything still running.