I usually lurk around *Freenode* in a few projects that I use, can learn from and/or help with. This is a great opportunity to learn new things /all the time/.
This story is familiar in that manner, but that's where similarities diverge. Someone asked around =#Weechat= a question that caught my attention because it was, sort of, out of topic. The question was around how do you backup your stuff ?
#+hugo: more
I mean if I were asked that, I would've mentioned revision controlled off-site repositories for the code that I have.
For the personal stuff on the other hand, I would've admitted simple rudimentary solutions like =rsync=, =tar= and external drives.
So I was sort of happy with my backup solution, it has worked. Plain and simple.
I have to admit that, by modern standards it might not offer the ability to go back in time to a certain point.
But I use /file systems/ that offer /snapshot/ capabilities. I can recover from previous snapshots and send them somewhere safe.
Archiving and encrypting those is not a simple process, wish it was. That limits storage possibilities if you care to keep your data private.
But if you know me, you'd know that I'm always open to new ways of doing things.
I can't remember exactly the conversation but the name *BorgBackup* was mentioned (thank you however you are). That's when things changed.
**** BorgBackup
[[https://www.borgbackup.org/][Borg]] is defined as a
#+BEGIN_QUOTE
Deduplicating archiver with compression and encryption
#+END_QUOTE
Although this is a very accurate and encompassing definition, it doesn't really show you how /AWESOME/ this thing is.
I had to go to the docs first before I stumbled upon this video.
It can be a bit difficult to follow the video, I understand.
This is why I decided to write this post, to sort of explain to you how *Borg* can backup your stuff.
**** Encryption
Oh yeah, that's the *first* thing I look at when I consider any suggested backup solution. *Borg* offers built-in /encryption/ and /authentication/. You can read about it in details in the [[https://borgbackup.readthedocs.io/en/stable/usage/init.html#encryption-modes][docs]].
So that's a check.
**** Compression
This is another thing I look for in a suggested backup solution. And I'm happy to report that *Borg* has this under the belt as well.
*Borg* currently supports /LZ4/, /zlib/, /LZMA/ and /zstd/. You can also tune the level of compression. Pretty neat !
**** Full Backup
I've watched a few videos and read a bit of their documentation and they talk about *FULL BACKUP*.
Which means every time you run *Borg*, it will take a full backup of your stuff. A full backup at that point in time, don't forget.
The implication of this is that you have a versioned list of your backups, and you can go back in time to any of them.
Yes, you read that right. *Borg* does a full backup every time you run it. That's a pretty neat feature.
If you're a bit ahead of me, you were gonna say woooow there bud ! I have *Gigabytes* of data, what do you mean *FULL BACKUP*, you keep saying *FULL BACKUP*.
I mean *FULL BACKUP*, wait until you hear about the next feature.
**** Deduplication
Booyah ! It has deduplication. Ain't that awesome. I've watched a presentation by the project's original maintainer explain this.
I have one thing to say. It's pretty good. How good, you may ask ?
My answer would be, good enough to fool me into thinking that it was taking snapshots of my data.
It wasn't until I dug in deeper into the matter that I understood that it was a full backup and the deduping taking care of the rest.
**** Check
*Borg* offers a way to vefiry the consistency of the repository and the archives within. This way, you can make sure that your backups haven't been corrupted.
This is a very good feature, and a must in my opinion from a backup solution. *Borg* has /YOU/ covered.
**** Restore
A backup solution is nothing if you can't get your data backup.
*Borg* has a few ways for you to get your data.
You can either create an /archive/ file out of a backup. You can export a file, a directory or the whole directory tree from a backup.
You can also, if you like, mount a backup and get stuff out.
*Borg* is a great tool for backup. It comes in an easily installable self-contained binary so you can use it, pretty much, anywhere giving you no excuse /whatsoever/ not to use it.
Their documentation is very good, and *Borg* is easy to use.
It offers you all the features you need to do off-site and on-site backups of all your important data.
I'll be testing *Borg* moving forward for my data. I'll make sure to report back anything I find, in the future, related to the subject.
*** DONE Automating Borg :borgmatic:borgbackup:borg:
In the previous blog post entitle [[#borgbackup]], I talked about *borg*.
If you read that post, you would've noticed that *borg* has a lot of features.
With a lot of features come a lot of automation.
If you were thinking about using *borg*, you should either make a /simple cron/ or you're gonna have to write an elaborate script to take care of all the different steps.
What if I told you there's another way ? An easier way ! The *Borgmatic* way... What would you say ?
#+hugo: more
**** Borgmatic
*Borgmatic* is defined on their [[https://torsion.org/borgmatic/][website]] as follows.
#+BEGIN_QUOTE
borgmatic is simple, configuration-driven backup software for servers
and workstations. Protect your files with client-side encryption.
Backup your databases too. Monitor it all with integrated third-party
services.
#+END_QUOTE
If you go down to it, *borgmatic* uses *borg*'s /API/ to automate a list of configurable /tasks/.
This way, it saves you the trouble of writing your own scripts to automate these steps.
*Borgmatic* uses a /YAML/ configuration file. Let's configure a few tasks.
**** Location
First, let's start by configuring the locations that *borg* is going to be working with.
#+BEGIN_SRC yaml
location:
source_directories:
- /home/
repositories:
- user@backupserver:sourcehostname.borg
one_file_system: true
exclude_patterns:
- /home/*/.cache
- '*.pyc'
#+END_SRC
This tells *borg* that we need to backup our =/home= directories excluding a few patterns.
Let's not forget that we told *borg* where the repository is located at.
A few months ago, I went on a search for a solution for my /dotfiles/.
I tried projects likes [[https://www.gnu.org/software/stow/][GNU Stow]], [[https://github.com/anishathalye/dotbot][dotbot]] and a [[https://www.atlassian.com/git/tutorials/dotfiles][bare /git/ repository]].
Each one of these solutions has its advantages and its advantages, but I found mine in [[https://www.chezmoi.io/][/Chezmoi/]].
/Chezmoi/ ? That's *French* right ? How is learning *French* going to help me ?
#+hugo: more
**** Introduction
On a /*nix/ system, whether /Linux/, /BSD/ or even /Mac OS/ now, the applications one uses have their configuration saved in the user's home directory. These files are called /configuration/ files. Usually, these configuration files start with a =.= which on these systems designate hidden files (they do not show up with a simple =ls=). Due their names, these /configuration/ files are also referred to as /dotfiles/.
I will be using /dotfiles/ and /configuration files/ interchangeably in this article, and they can be thought as such.
#+BEGIN_EXPORT html
</div>
#+END_EXPORT
One example of such files is the =.bashrc= file found in the user's /home directory/. It allows the user to configure /bash/ and change some behaviours.
Now that we understand what /dotfiles/ are, let's talk a little bit about the /previously mentioned/ solutions.
They deserve mentioning, especially if you're looking for such solution.
***** GNU Stow
/GNU Stow/ leverages the power of /symlinks/ to keep your /configuration/ in a *centralized* location.
Wherever your repository lives, /GNU Stow/ will mimic the internal structure of said repository in your *home directory* by /smartly symlinking/ everything.
I said /smartly/ because it tries to *minimize* the amount of /symlinks/ created by /symlinking/ to common root directories if possible.
By having all your configuration files under one directory structure, it is easier to push it to any public repository and share it with others.
The downsize is, you end-up with a lot of /symlinks/. It is also worth mentioning that not all applications behave well when their /configuration directories/ are /symlinked/. Otherwise, /GNU Stow/ is a great project.
***** Dotbot
/Dotbot/ is a /Python/ project that *aims* at automating your /dotfiles/. It gives you great control over what and how to manage your /dotfiles/.
Having it written in /Python/ means it is very easy to install; =pip=. It also means that it /should/ be easy to migrate it to different systems.
/Dotbot/ has a lot going for it. If the idea of having control over every aspect of your /dotfiles/, including the /possibility/ of the setup of the environment along with it, then /dotbot/ is for you.
Well, it's not for *me*.
***** Bare /Git/ Repository
This is arguably the /most elegant/ solution of them all.
The nice thing about this solution is its /simplicity/ and /cleanliness/. It is /essentially/ creating a /bare git/ repository /somewhere/ in your /home directory/ specifying the /home directory/ itself to be the /working directory/.
If you are wondering where one would use a /bare git/ repository in real life other than this use case.
Well, you have no other place to turn than any /git server/. On the server, /Gitea/ for example, your repository is only a /bare/ repository. One has to clone it to get the /working directory/ along with it.
Anyway, back to our topic. This is a great solution if you don't have to worry about things you would like to hide.
By hide, I mean things like /credentials/, /keys/ or /passwords/ which *never* belong in a /repository/.
You will need to find solutions for these types of files. I was looking for something /less involving/ and /more involved/.
**** /Chezmoi/ to the rescue ?
Isn't that what they *all* say ?
I like how the creator(s) defines [[https://www.chezmoi.io/][/Chezmoi/]]
#+BEGIN_QUOTE
Manage your dotfiles across multiple machines, securely.
#+END_QUOTE
Pretty basic, straight to the point. Unfortunately, it's a little bit harder to grasp the concept of how it works.
/Chezmoi/ basically /generates/ the /dotfiles/ from the /local repository/. These /dotfiles/ are saved in different forms in the /repository/ but they *always* generate the same output; the /dotfiles/. Think of /Chezmoi/ as a /dotfiles/ templating engine, at its basic form it saves your /dotfiles/ as is and /deploys/ them in *any* machine.
**** Working with /Chezmoi/
I think we should take a /quick/ look at /Chezmoi/ to see how it works.
/Chezmoi/ is written /Golang/ making it /fairly/ easy to [[https://www.chezmoi.io/docs/install/][install]] so I will forgo that boring part.
***** First run
To start using /Chezmoi/, one has to *initialize* a new /Chezmoi repository/.
#+BEGIN_SRC bash
chezmoi init
#+END_SRC
This will create a *new*/git repository/ in =~/.local/share/chezmoi=. This is now the *source state*, where /Chezmoi/ will get your /dotfiles/.
***** Plain /dotfiles/ management with /Chezmoi/
Now that we have a /Chezmoi/ repository. We can start to /populate/ it with /dotfiles/.
Let's assume that we would like to start managing one of our /dotfiles/ with /Chezmoi/.
I'm going with an /imaginary application/'s configuration directory.
This directory will hold different files with /versatile/ content types.
This is going to showcase some of /Chezmoi/'s capabilities.
Always use /Chezmoi/ to edit your managed /dotfiles/. Do *NOT* edit them directly.
*ALWAYS* use =chezmoi diff= before every /applying/.
#+BEGIN_EXPORT html
</div>
#+END_EXPORT
To /edit/ your managed /dotfile/, simply tell /Chezmoi/ about it.
#+BEGIN_SRC bash
chezmoi edit ~/.ds9/config
#+END_SRC
/Chezmoi/ will use your =$EDITOR= to open the file for you to edit. Once saved, it's saved in the /repository database/.
Be aware, at this point the changes are not reflected in your /home/ directory, *only* in the /Chezmoi source state/. Make sure you *diff* and then *apply* to make the changes in your /home/.
***** /Chezmoi/ repository management
As mentioned previously, the repository is found in =~/.local/share/chezmoi=.
I *always* forget where it is, luckily /Chezmoi/ has a solution for that.
#+BEGIN_SRC bash
chezmoi cd
#+END_SRC
Now, we are in the repository. We can work with it as a /regultar//git/ repository.
When you're done, don't forget to =exit=.
***** Other features
It is worth mentioning at this point that /Chezmoi/ offers a few more integrations.
****** Templating
Due to the fact that /Chezmoi/ is written in /Golang/, it can leverage the power of the /Golang [[https://www.chezmoi.io/docs/how-to/#use-templates-to-manage-files-that-vary-from-machine-to-machine][templating]]/ system.
One can replace /repeatable/ values like *email* or *name* with a template like ={{ .email }}= or ={{ .name }}=.
This will result in a replacement of these /templated variables/ with their real values in the resulting /dotfile/.
This is another reason why you should *always* edit your managed /dotfiles/ through /Chezmoi/.
Our /previous/ example would look a bit different.
Follow the [[https://www.chezmoi.io/docs/how-to/#use-templates-to-manage-files-that-vary-from-machine-to-machine][documentation]] to /configure/ the *values*.
#+BEGIN_EXPORT html
</div>
#+END_EXPORT
****** Password manager integration
Once you have the power of /templating/ on your side, you can always take it one step further.
/Chezmoi/ has integration with a big list of [[https://www.chezmoi.io/docs/how-to/#keep-data-private][password managers]]. These can be used directly into the /configuration files/.
In our /hypothetical/ example, we can think of the /credentials/ file (=~/.ds9/sisko-creds.cred=).
Do not /forget/ that this is also using the /templating/ engine. So you need to add as a /template/.
#+BEGIN_SRC bash
chezmoi add --template ~/.ds9/sisko-creds.cred
#+END_SRC
****** File encryption
Wait, what ! You almost slipped away right there old fellow.
We have our /Mastodon/*API* key in the =sisko-api.mastodon= file. The whole file cannot be pushed to a repository.
It turns out that /Chezmoi/ can use /gpg/ to [[https://www.chezmoi.io/docs/how-to/#use-gpg-to-keep-your-secrets][encrypt your files]] making it possible for you to push them.
To add a file encrypted to the /Chezmoi/ repository, use the following command.
#+BEGIN_SRC bash
chezmoi add --encrypt ~/.ds9/sisko-api.mastodon
#+END_SRC
****** Misc
There is a list of other features that /Chezmoi/ supports that I did not mention.
I did not use all the /features/ offered yet. You should check the [[https://www.chezmoi.io/][website]] for the full documentation.
**** Conclusion
I am fully migrated into /Chezmoi/ so far. I have used all the features above, and it has worked flawlessly so far.
I like the idea that it offers *all* the features I need while at the same time staying out of the way.
I find myself, often, editing the /dotfiles/ in my /home/ directory as a /dev/ version. Once I get to a configuration I like, I add it to /Chezmoi/. If I ever mess up badly, I ask /Chezmoi/ to override my changes.
I understand it adds a little bit of /overhead/ with the use of =chezmoi= commands, which I aliased to =cm=. But the end result is a /home/ directory which seems untouched by any tools (no symlinks, no copies, etc...) making it easier to migrate /out/ of /Chezmoi/ as a solution and into another one if I ever choose in the future.
When I first started using [[https://www.ansible.com/][ansible]], I did not know about [[https://molecule.readthedocs.io/en/latest/][molecule]]. It was a bit daunting to start a /role/ from scratch and trying to develop it without having the ability to test it. Then a co-worker of mine told me about molecule and everything changed.
#+hugo: more
I do not have any of the tools I need installed on this machine, so I will go through, step by step, how I set up ansible and molecule on any new machine I come across for writing ansible roles.
**** Requirements
What we are trying to achieve in this post, is a working ansible role that can be tested inside a docker container. To be able to achieve that, we need to install docker on the system. Follow the instructions on [[https://docs.docker.com/install/][installing docker]] found on the docker website.
**** Good Practices
First thing's first. Let's start by making sure that we have python installed properly on the system.
#+BEGIN_EXAMPLE
$ python --version
Python 3.7.1
#+END_EXAMPLE
Because in this case I have /python3/ installed, I can create a /virtualenv/ easier without the use of external tools.
Once all the steps above are complete, we can start by creating our first ansible role.
#+BEGIN_EXAMPLE
$ molecule init role -r example-role
--> Initializing new role example-role...
Initialized role in /home/elijah/sandbox/test-roles/example-role successfully.
$ tree example-role/
example-role/
├── defaults
│ └── main.yml
├── handlers
│ └── main.yml
├── meta
│ └── main.yml
├── molecule
│ └── default
│ ├── Dockerfile.j2
│ ├── INSTALL.rst
│ ├── molecule.yml
│ ├── playbook.yml
│ └── tests
│ ├── __pycache__
│ │ └── test_default.cpython-37.pyc
│ └── test_default.py
├── README.md
├── tasks
│ └── main.yml
└── vars
└── main.yml
9 directories, 12 files
#+END_EXAMPLE
You can find what each directory is for and how ansible works by visiting [[https://docs.ansible.com][docs.ansible.com]].
***** =meta/main.yml=
The meta file needs to modified and filled with information about the role. This is not a required file to modify if you are keeping this for yourself, for example. But it is a good idea to have as much information as possible if this is going to be released. In my case, I don't need any fanciness as this is just sample code.
#+BEGIN_SRC yaml
---
galaxy_info:
author: Elia el Lazkani
description: This is an example ansible role to showcase molecule at work
license: license (BDS-2)
min_ansible_version: 2.7
galaxy_tags: []
dependencies: []
#+END_SRC
***** =tasks/main.yml=
This is where the magic is set in motion. Tasks are the smallest entities in a role that do small and idempotent actions. Let's write a few simple tasks to create a user and install a service.
#+BEGIN_SRC yaml
---
# Create the user example
- name: Create 'example' user
user:
name: example
comment: Example user
shell: /bin/bash
state: present
create_home: yes
home: /home/example
# Install nginx
- name: Install nginx
apt:
name: nginx
state: present
update_cache: yes
notify: Restart nginx
#+END_SRC
***** =handlers/main.yml=
If you noticed, we are notifying a handler to be called after installing /nginx/. All handlers notified will run after all the tasks complete and each handler will only run once. This is a good way to make sure that you don't restart /nginx/ multiple times if you call the handler more than once.
#+BEGIN_SRC yaml
---
# Handler to restart nginx
- name: Restart nginx
service:
name: nginx
state: restarted
#+END_SRC
***** =molecule/default/molecule.yml=
It's time to configure molecule to do what we need. We need to start an ubuntu docker container, so we need to specify that in the molecule YAML file. All we need to do is change the image line to specify that we want an =ubuntu:bionic= image.
#+BEGIN_SRC yaml
---
dependency:
name: galaxy
driver:
name: docker
lint:
name: yamllint
platforms:
- name: instance
image: ubuntu:bionic
provisioner:
name: ansible
lint:
name: ansible-lint
scenario:
name: default
verifier:
name: testinfra
lint:
name: flake8
#+END_SRC
***** =molecule/default/playbook.yml=
This is the playbook that molecule will run. Make sure that you have all the steps that you need here. I will keep this as is.
#+BEGIN_SRC yaml
---
- name: Converge
hosts: all
roles:
- role: example-role
#+END_SRC
**** First Role Pass
This is time to test our role and see what's going on.
PLAY RECAP *********************************************************************
instance : ok=4 changed=3 unreachable=0 failed=0
#+END_EXAMPLE
It looks like the *converge* step succeeded.
**** Writing Tests
It is always a good practice to write unittests when you're writing code. Ansible roles should not be an exception. Molecule offers a way to run tests, which you can think of as unittest, to make sure that what the role gives you is what you were expecting. This helps future development of the role and keeps you from falling in previously solved traps.
***** =molecule/default/tests/test_default.py=
Molecule leverages the [[https://testinfra.readthedocs.io/en/latest/][testinfra]] project to run its tests. You can use other tools if you so wish, and there are many. In this example we will be using /testinfra/.
