Browse Source

New blog post about the healthchecks project

master
Elia el Lazkani 4 months ago
parent
commit
48de646c64
Signed by: Elia El Lazkani <[email protected]> GPG Key ID: FBD81F2B1F488C2B
3 changed files with 154 additions and 0 deletions
  1. BIN
      images/simple_cron_monitoring_with_healthchecks/borgbackup_healthchecks.png
  2. BIN
      images/simple_cron_monitoring_with_healthchecks/borgbackup_healthchecks_logs.png
  3. +154
    -0
      posts/monitoring/simple_cron_monitoring_with_healthchecks.rst

BIN
images/simple_cron_monitoring_with_healthchecks/borgbackup_healthchecks.png View File

Before After
Width: 1794  |  Height: 514  |  Size: 72KB

BIN
images/simple_cron_monitoring_with_healthchecks/borgbackup_healthchecks_logs.png View File

Before After
Width: 1521  |  Height: 486  |  Size: 130KB

+ 154
- 0
posts/monitoring/simple_cron_monitoring_with_healthchecks.rst View File

@@ -0,0 +1,154 @@
.. title: Simple cron monitoring with HealthChecks
.. date: 2020-02-09
.. slug: simple_cron_monitoring_with_healthchecks
.. updated: 2020-02-09
.. status: published
.. tags: monitoring, healthchecks, cron
.. category: monitoring
.. authors: Elia el Lazkani
.. description: Ever needed to monitor simple things ? Well, HealthChecks seems perfect for that.
.. type: text

In a :doc:`previous post <automating-borg>`, I showed you how you can automate your **borg** backups with **borgmatic**.

After I started using **borgmatic** for my backups and hooked it to a *cron* running every 2 hours, I got interested into knowing what's happening to my backups at all times.

My experience comes handy in here, I know I need a monitoring system.
I also know that traditional monitoring systems are too complex for my use case.

I need something simple. I need something I can deploy myself.

.. TEASER_END

Choosing a monitoring system
============================

I already know I don't want a traditional monitoring system like *nagios* or *sensu* or *prometheus*.
It is not needed, it's an overkill.

I went through the list of hooks that **borgmatic** offers out of the box and checked each project.

I came across `HealthChecks <https://healthchecks.io/>`_.

HealthChecks
============

The `HealthChecks <https://healthchecks.io/>`_ project works in a simple manner.
It simply offers syou an endpoint which you need to ping within a certain period, otherwise you get paged.

It has a lot of integrations from simple emails to other third party services that will call or message you or even trigger push notifications to your phone.

In my case, a simple email is enough. After all, they are simply backups and if they failed now, they will work when cron runs again in 2 hours.

Deploy
======

Let's create a `docker-compose` service configuration that looks like the following:

.. code:: yaml

healthchecks:
container_name: healthchecks
image: linuxserver/healthchecks:v1.12.0-ls48
restart: unless-stopped
ports:
- "127.0.0.1:8000:8000"
volumes:
- "./healthchecks/data:/config"
environment:
PUID: "5000"
PGID: "5000"
SECRET_KEY: "super-secret-key"
ALLOWED_HOSTS: '["*"]'
DEBUG: "False"
DEFAULT_FROM_EMAIL: "[email protected]"
USE_PAYMENTS: "False"
REGISTRATION_OPEN: "False"
EMAIL_HOST: "smtp.example.com"
EMAIL_PORT: "587"
EMAIL_HOST_USER: "[email protected]"
EMAIL_HOST_PASSWORD: "super-secret-password"
EMAIL_USE_TLS: "True"
SITE_ROOT: "https://healthchecks.example.com"
SITE_NAME: "HealthChecks"
MASTER_BADGE_LABEL: "HealthChecks"
PING_ENDPOINT: "https://healthchecks.example.com/ping/"
PING_EMAIL_DOMAIN: "healthchecks.example.com"
TWILIO_ACCOUNT: "None"
TWILIO_AUTH: "None"
TWILIO_FROM: "None"
PD_VENDOR_KEY: "None"
TRELLO_APP_KEY: "None"

This will create a docker container exposing it locally on ``127.0.0.1:8000``.
Let's point nginx to it and expose it using something similar to the following.

.. code:: text

server {
listen 443 ssl;
server_name healthchecks.example.com;

ssl_certificate /path/to/the/fullchain.pem;
ssl_certificate_key /path/to/the/privkey.pem;

location / {
proxy_pass http://127.0.0.1:8000;
add_header X-Frame-Options SAMEORIGIN;
add_header X-XSS-Protection "1; mode=block";
proxy_redirect off;
proxy_buffering off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Port $server_port;
proxy_read_timeout 90;
}

}

This should do nicely.


Usage
=====

Now it's a simple matter of creating a checks.

.. thumbnail:: /images/simple_cron_monitoring_with_healthchecks/borgbackup_healthchecks.png
:align: center
:alt: HealthChecks monitoring for BorgBackup


This will give you a link that looks like the following ``https://healthchecks.example.com/ping/84b2a834-02f5-524f-4c27-a2f24562b219``

Let's feed it to **borgmatic**.

.. code:: yaml

hooks:
healthchecks: https://healthchecks.example.com/ping/84b2a834-02f5-524f-4c27-a2f24562b219


After you configure the **borgmatic** hook to keep *HealthChecks* in the know of what's going on.
We can take a look at the log to see what happened and when.

.. thumbnail:: /images/simple_cron_monitoring_with_healthchecks/borgbackup_healthchecks_logs.png
:align: center
:alt: HealthChecks monitoring log for BorgBackup



Conclusion
==========

As we saw in the blog post, now I am always in the know about my backups.
If my backup fails, I get an email to notify me of a failure.
I can also monitor how much time it takes my backups to run.
This is a very important feature for me to have.

The question of deploying one's own monitoring system is a personal choice.
After all, one can use free third party services if they would like.
The correct answer though is to always monitor.

Loading…
Cancel
Save