The DevOps Blog https://blog.lazkani.io
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

8111 lines
325 KiB

#+STARTUP: content
#+AUTHOR: Elia el Lazkani
#+HUGO_BASE_DIR: ../.
#+HUGO_AUTO_SET_LASTMOD: t
* Custom Pages
:PROPERTIES:
:EXPORT_HUGO_CUSTOM_FRONT_MATTER: :noauthor true :nocomment true :nodate true :nopaging true :noread true
:EXPORT_HUGO_MENU: :menu false
:EXPORT_HUGO_SECTION: .
:EXPORT_HUGO_WEIGHT: auto
:END:
** Not Found
:PROPERTIES:
:EXPORT_FILE_NAME: not-found
:EXPORT_HUGO_LASTMOD: 2021-07-04
:EXPORT_DATE: 2020-02-08
:CUSTOM_ID: not-found
:END:
*** 404 Not Found
Oops... We don't know how you ended up here.
There is nothing here to look at...
Head back over [[/][home]].
** Forbidden
:PROPERTIES:
:EXPORT_FILE_NAME: forbidden
:EXPORT_HUGO_LASTMOD: 2021-07-04
:EXPORT_DATE: 2020-06-05
:CUSTOM_ID: forbidden
:END:
*** 403 Forbidden
Naughty naughty !
What brought you to a forbidden page ?
Take this =403 Forbidden= and head over the [[/][main site]].
* Pages
:PROPERTIES:
:EXPORT_HUGO_CUSTOM_FRONT_MATTER: :noauthor true :nocomment true :nodate true :nopaging true :noread true
:EXPORT_HUGO_MENU: :menu main
:EXPORT_HUGO_SECTION: pages
:EXPORT_HUGO_WEIGHT: auto
:END:
** About
:PROPERTIES:
:EXPORT_FILE_NAME: about
:EXPORT_HUGO_LASTMOD: 2021-07-04
:EXPORT_DATE: 2019-06-21
:CUSTOM_ID: about
:END:
*** Who am I ?
I am a DevOps cloud engineer with a passion for technology, automation, Linux and OpenSource.
I've been on Linux since the /early/ 2000's and have contributed, in some small capacity, to some open source projects along the way.
I dabble in this space and I blog about it. This is how I learn, this is how I evolve.
*** Contact Me
If, for some reason, you'd like to get in touch you have sevaral options.
- Find me on [[https://libera.chat/][libera]] in ~#LearnAndTeach~.
- Email me at ~blog[at]lazkani[dot]io~
If you use /GPG/ and you should, my public key is ~2383 8945 E07E 670A 4BFE 39E6 FBD8 1F2B 1F48 8C2B~
** FAQ
:PROPERTIES:
:EXPORT_FILE_NAME: faq
:EXPORT_HUGO_LASTMOD: 2021-07-04
:EXPORT_DATE: 2021-07-04
:CUSTOM_ID: faq
:END:
*** What is this ?
This is my humble blog where I post things related to DevOps in hope that I or someone else might benefit from it.
*** Wait what ? What is DevOps ?
[[https://duckduckgo.com/?q=what+is+devops+%3F&t=ffab&ia=web&iax=about][Duckduckgo]] defines DevOps as:
#+BEGIN_QUOTE
DevOps is a software engineering culture and practice that aims at unifying
software development and software operation. The main characteristic of the
DevOps movement is to strongly advocate automation and monitoring at all
steps of software construction, from integration, testing, releasing to
deployment and infrastructure management. DevOps aims at shorter development
cycles, increased deployment frequency, and more dependable releases,
in close alignment with business objectives.
#+END_QUOTE
In short, we build an infrastructure that is easily deployable, maintainable and, in all forms, makes the lives of the developers a breeze.
*** What do you blog about ?
Anything and everything related to DevOps. The field is very big and complex with a lot of different tools and technologies implemented.
I try to blog about interesting and new things as much as possible, when time permits.
*** Does this blog have *RSS* ?
Yup, here's the [[/posts/index.xml][link]].
* Posts
:PROPERTIES:
:EXPORT_HUGO_SECTION: posts
:END:
** Backup :@backup:
*** DONE BorgBackup :borg:borgbackup:
:PROPERTIES:
:EXPORT_HUGO_LASTMOD: 2020-01-30
:EXPORT_DATE: 2020-01-30
:EXPORT_FILE_NAME: borgbackup
:CUSTOM_ID: borgbackup
:END:
I usually lurk around *Freenode* in a few projects that I use, can learn from and/or help with. This is a great opportunity to learn new things /all the time/.
This story is familiar in that manner, but that's where similarities diverge. Someone asked around =#Weechat= a question that caught my attention because it was, sort of, out of topic. The question was around how do you backup your stuff ?
#+hugo: more
I mean if I were asked that, I would've mentioned revision controlled off-site repositories for the code that I have.
For the personal stuff on the other hand, I would've admitted simple rudimentary solutions like =rsync=, =tar= and external drives.
So I was sort of happy with my backup solution, it has worked. Plain and simple.
I have to admit that, by modern standards it might not offer the ability to go back in time to a certain point.
But I use /file systems/ that offer /snapshot/ capabilities. I can recover from previous snapshots and send them somewhere safe.
Archiving and encrypting those is not a simple process, wish it was. That limits storage possibilities if you care to keep your data private.
But if you know me, you'd know that I'm always open to new ways of doing things.
I can't remember exactly the conversation but the name *BorgBackup* was mentioned (thank you however you are). That's when things changed.
**** BorgBackup
[[https://www.borgbackup.org/][Borg]] is defined as a
#+BEGIN_QUOTE
Deduplicating archiver with compression and encryption
#+END_QUOTE
Although this is a very accurate and encompassing definition, it doesn't really show you how /AWESOME/ this thing is.
I had to go to the docs first before I stumbled upon this video.
#+BEGIN_EXPORT md
[![asciicast](https://asciinema.org/a/133292.svg)](https://asciinema.org/a/133292)
#+END_EXPORT
It can be a bit difficult to follow the video, I understand.
This is why I decided to write this post, to sort of explain to you how *Borg* can backup your stuff.
**** Encryption
Oh yeah, that's the *first* thing I look at when I consider any suggested backup solution. *Borg* offers built-in /encryption/ and /authentication/. You can read about it in details in the [[https://borgbackup.readthedocs.io/en/stable/usage/init.html#encryption-modes][docs]].
So that's a check.
**** Compression
This is another thing I look for in a suggested backup solution. And I'm happy to report that *Borg* has this under the belt as well.
*Borg* currently supports /LZ4/, /zlib/, /LZMA/ and /zstd/. You can also tune the level of compression. Pretty neat !
**** Full Backup
I've watched a few videos and read a bit of their documentation and they talk about *FULL BACKUP*.
Which means every time you run *Borg*, it will take a full backup of your stuff. A full backup at that point in time, don't forget.
The implication of this is that you have a versioned list of your backups, and you can go back in time to any of them.
Yes, you read that right. *Borg* does a full backup every time you run it. That's a pretty neat feature.
If you're a bit ahead of me, you were gonna say woooow there bud ! I have *Gigabytes* of data, what do you mean *FULL BACKUP*, you keep saying *FULL BACKUP*.
I mean *FULL BACKUP*, wait until you hear about the next feature.
**** Deduplication
Booyah ! It has deduplication. Ain't that awesome. I've watched a presentation by the project's original maintainer explain this.
I have one thing to say. It's pretty good. How good, you may ask ?
My answer would be, good enough to fool me into thinking that it was taking snapshots of my data.
#+BEGIN_EXAMPLE
-----------------------------------------------------------------------------
Original size Compressed size Deduplicated size
All archives: 34.59 GB 9.63 GB 1.28 GB
Unique chunks Total chunks
Chunk index: 47772 469277
#+END_EXAMPLE
It wasn't until I dug in deeper into the matter that I understood that it was a full backup and the deduping taking care of the rest.
**** Check
*Borg* offers a way to vefiry the consistency of the repository and the archives within. This way, you can make sure that your backups haven't been corrupted.
This is a very good feature, and a must in my opinion from a backup solution. *Borg* has /YOU/ covered.
**** Restore
A backup solution is nothing if you can't get your data backup.
*Borg* has a few ways for you to get your data.
You can either create an /archive/ file out of a backup. You can export a file, a directory or the whole directory tree from a backup.
You can also, if you like, mount a backup and get stuff out.
#+BEGIN_EXPORT html
<div class="admonition warning">
<p class="admonition-title"><b>Warning</b></p>
#+END_EXPORT
Mounting a *Borg* backup is done using /fuse/
#+BEGIN_EXPORT html
</div>
#+END_EXPORT
**** Conclusion
*Borg* is a great tool for backup. It comes in an easily installable self-contained binary so you can use it, pretty much, anywhere giving you no excuse /whatsoever/ not to use it.
Their documentation is very good, and *Borg* is easy to use.
It offers you all the features you need to do off-site and on-site backups of all your important data.
I'll be testing *Borg* moving forward for my data. I'll make sure to report back anything I find, in the future, related to the subject.
*** DONE Automating Borg :borgmatic:borgbackup:borg:
:PROPERTIES:
:EXPORT_HUGO_LASTMOD: 2020-02-02
:EXPORT_DATE: 2020-02-02
:EXPORT_FILE_NAME: automating-borg
:CUSTOM_ID: automating-borg
:END:
In the previous blog post entitle [[#borgbackup]], I talked about *borg*.
If you read that post, you would've noticed that *borg* has a lot of features.
With a lot of features come a lot of automation.
If you were thinking about using *borg*, you should either make a /simple cron/ or you're gonna have to write an elaborate script to take care of all the different steps.
What if I told you there's another way ? An easier way ! The *Borgmatic* way... What would you say ?
#+hugo: more
**** Borgmatic
*Borgmatic* is defined on their [[https://torsion.org/borgmatic/][website]] as follows.
#+BEGIN_QUOTE
borgmatic is simple, configuration-driven backup software for servers
and workstations. Protect your files with client-side encryption.
Backup your databases too. Monitor it all with integrated third-party
services.
#+END_QUOTE
If you go down to it, *borgmatic* uses *borg*'s /API/ to automate a list of configurable /tasks/.
This way, it saves you the trouble of writing your own scripts to automate these steps.
*Borgmatic* uses a /YAML/ configuration file. Let's configure a few tasks.
**** Location
First, let's start by configuring the locations that *borg* is going to be working with.
#+BEGIN_SRC yaml
location:
source_directories:
- /home/
repositories:
- user@backupserver:sourcehostname.borg
one_file_system: true
exclude_patterns:
- /home/*/.cache
- '*.pyc'
#+END_SRC
This tells *borg* that we need to backup our =/home= directories excluding a few patterns.
Let's not forget that we told *borg* where the repository is located at.
**** Storage
We need to configure the storage next.
#+BEGIN_SRC yaml
storage:
# Recommended
# encryption_passcommand: secret-tool lookup borg-repository repo-name
encryption_passphrase: "ReallyStrongPassphrase"
compression: zstd,15
ssh_command: ssh -i /path/to/private/key
borg_security_directory: /path/to/base/config/security
archive_name_format: 'borgmatic-{hostname}-{now}'
#+END_SRC
In this section, we tell borg a little big of information about our repository.
What are the credentials, where it can find them, etc.
The easy way is to go with a =passphrase=, but I recommend using an =encryption_passcommand= instead.
I also use =zstd= for encryption instead of =lz4=, you better do your research before you change the default.
I also recommend, just as they do, the use of a security directory as well.
**** Retention
We can configure a retention for our backups, if we like.
#+BEGIN_SRC yaml
retention:
keep_hourly: 7
keep_daily: 7
keep_weekly: 4
keep_monthly: 6
keep_yearly: 2
prefix: "borgmatic-"
#+END_SRC
The part of what to keep from /hourly/ to /daily/ is self explanatory.
I would like to point out the =prefix= part as it is important.
This is the /prefix/ that *borgmatic* uses to consider backups for *pruning*.
#+BEGIN_EXPORT html
<div class="admonition warning">
<p class="admonition-title"><b>Warning</b></p>
#+END_EXPORT
Watch out for the retention =prefix=
#+BEGIN_EXPORT html
</div>
#+END_EXPORT
**** Consistency
After the updates, we'd like to check our backups.
#+BEGIN_SRC yaml
consistency:
checks:
- repository
- archives
check_last: 3
prefix: "borgmatic-"
#+END_SRC
#+BEGIN_EXPORT html
<div class="admonition warning">
<p class="admonition-title"><b>Warning</b></p>
#+END_EXPORT
Watch out, again, for the consistency =prefix=
#+BEGIN_EXPORT html
</div>
#+END_EXPORT
**** Hooks
Finally, hooks.
I'm going to talk about hooks a bit. Hooks can be used to backup *MySQL*, *PostgreSQL* or *MariaDB*.
They can also be hooks for =on_error=, =before_backup=, =after_backup=, =before_everything= and =after_everything=.
You can also hook to third party services which you can check on their webpage.
I deployed my own, so I configured my own.
**** Borgmatic Configuration
Let's put everything together now.
#+BEGIN_SRC yaml
location:
source_directories:
- /home/
repositories:
- user@backupserver:sourcehostname.borg
one_file_system: true
exclude_patterns:
- /home/*/.cache
- '*.pyc'
storage:
# Recommended
# encryption_passcommand: secret-tool lookup borg-repository repo-name
encryption_passphrase: "ReallyStrongPassphrase"
compression: zstd,15
ssh_command: ssh -i /path/to/private/key
borg_security_directory: /path/to/base/config/security
archive_name_format: 'borgmatic-{hostname}-{now}'
retention:
keep_hourly: 7
keep_daily: 7
keep_weekly: 4
keep_monthly: 6
keep_yearly: 2
prefix: "borgmatic-"
consistency:
checks:
- repository
- archives
check_last: 3
prefix: "borgmatic-"
#+END_SRC
Now that we have everything together, let's save it in =/etc/borgmatic.d/home.yaml=.
**** Usage
If you have *borg* and *borgmatic* already installed on your system and the *borgmatic* configuration file in place, you can test it out.
You can create the repository.
#+BEGIN_EXAMPLE
# borgmatic init -v 2
#+END_EXAMPLE
You can list the backups for the repository.
#+BEGIN_EXAMPLE
# borgmatic list --last 5
borgmatic-home-2020-01-30T22:01:30 Thu, 2020-01-30 22:01:42 [0000000000000000000000000000000000000000000000000000000000000000]
borgmatic-home-2020-01-31T22:02:12 Fri, 2020-01-31 22:02:24 [0000000000000000000000000000000000000000000000000000000000000000]
borgmatic-home-2020-02-01T22:01:34 Sat, 2020-02-01 22:01:45 [0000000000000000000000000000000000000000000000000000000000000000]
borgmatic-home-2020-02-02T16:01:22 Sun, 2020-02-02 16:01:32 [0000000000000000000000000000000000000000000000000000000000000000]
borgmatic-home-2020-02-02T18:01:36 Sun, 2020-02-02 18:01:47 [0000000000000000000000000000000000000000000000000000000000000000]
#+END_EXAMPLE
You could run a check.
#+BEGIN_EXAMPLE
# borgmatic check -v 1
/etc/borgmatic.d/home.yaml: Pinging Healthchecks start
/borg/home: Running consistency checks
Remote: Starting repository check
Remote: Starting repository index check
Remote: Completed repository check, no problems found.
Starting archive consistency check...
Analyzing archive borgmatic-home-2020-02-01T22:01:34 (1/3)
Analyzing archive borgmatic-home-2020-02-02T16:01:22 (2/3)
Analyzing archive borgmatic-home-2020-02-02T18:01:36 (3/3)
Orphaned objects check skipped (needs all archives checked).
Archive consistency check complete, no problems found.
summary:
/etc/borgmatic.d/home.yaml: Successfully ran configuration file
#+END_EXAMPLE
But most of all, if you simply run =borgmatic= without any parameters, it will run through the whole configuration and apply all the steps.
At this point, you can simply add the =borgmatic= command in a *cron* to run on an interval.
The other options would be to configure a =systemd= *timer* and *service* to run this on an interval.
The latter is usually provided to you if you used your *package manager* to install *borgmatic*.
**** Conclusion
If you've checked *borg* and found it too much work to script, give *borgmatic* a try.
I've been using borgmatic for few weeks now with no issues at all.
I recently hooked it to a monitoring system so I will have a better view on when it runs, how much time each run takes.
Also, if any of my backups fail I get notified by email. I hope you enjoy *borg* and *borgmatic* as much as I am.
*** DONE Dotfiles with /Chezmoi/ :dotfiles:chezmoi:encryption:templates:
:PROPERTIES:
:EXPORT_HUGO_LASTMOD: 2020-10-05
:EXPORT_DATE: 2020-10-05
:EXPORT_FILE_NAME: dotfiles-with-chezmoi
:CUSTOM_ID: dotfiles-with-chezmoi
:END:
A few months ago, I went on a search for a solution for my /dotfiles/.
I tried projects likes [[https://www.gnu.org/software/stow/][GNU Stow]], [[https://github.com/anishathalye/dotbot][dotbot]] and a [[https://www.atlassian.com/git/tutorials/dotfiles][bare /git/ repository]].
Each one of these solutions has its advantages and its advantages, but I found mine in [[https://www.chezmoi.io/][/Chezmoi/]].
/Chezmoi/ ? That's *French* right ? How is learning *French* going to help me ?
#+hugo: more
**** Introduction
On a /*nix/ system, whether /Linux/, /BSD/ or even /Mac OS/ now, the applications one uses have their configuration saved in the user's home directory. These files are called /configuration/ files. Usually, these configuration files start with a =.= which on these systems designate hidden files (they do not show up with a simple =ls=). Due their names, these /configuration/ files are also referred to as /dotfiles/.
#+BEGIN_EXPORT html
<div class="admonition note">
<p class="admonition-title"><b>Note</b></p>
#+END_EXPORT
I will be using /dotfiles/ and /configuration files/ interchangeably in this article, and they can be thought as such.
#+BEGIN_EXPORT html
</div>
#+END_EXPORT
One example of such files is the =.bashrc= file found in the user's /home directory/. It allows the user to configure /bash/ and change some behaviours.
Now that we understand what /dotfiles/ are, let's talk a little bit about the /previously mentioned/ solutions.
They deserve mentioning, especially if you're looking for such solution.
***** GNU Stow
/GNU Stow/ leverages the power of /symlinks/ to keep your /configuration/ in a *centralized* location.
Wherever your repository lives, /GNU Stow/ will mimic the internal structure of said repository in your *home directory* by /smartly symlinking/ everything.
I said /smartly/ because it tries to *minimize* the amount of /symlinks/ created by /symlinking/ to common root directories if possible.
By having all your configuration files under one directory structure, it is easier to push it to any public repository and share it with others.
The downsize is, you end-up with a lot of /symlinks/. It is also worth mentioning that not all applications behave well when their /configuration directories/ are /symlinked/. Otherwise, /GNU Stow/ is a great project.
***** Dotbot
/Dotbot/ is a /Python/ project that *aims* at automating your /dotfiles/. It gives you great control over what and how to manage your /dotfiles/.
Having it written in /Python/ means it is very easy to install; =pip=. It also means that it /should/ be easy to migrate it to different systems.
/Dotbot/ has a lot going for it. If the idea of having control over every aspect of your /dotfiles/, including the /possibility/ of the setup of the environment along with it, then /dotbot/ is for you.
Well, it's not for *me*.
***** Bare /Git/ Repository
This is arguably the /most elegant/ solution of them all.
The nice thing about this solution is its /simplicity/ and /cleanliness/. It is /essentially/ creating a /bare git/ repository /somewhere/ in your /home directory/ specifying the /home directory/ itself to be the /working directory/.
If you are wondering where one would use a /bare git/ repository in real life other than this use case.
Well, you have no other place to turn than any /git server/. On the server, /Gitea/ for example, your repository is only a /bare/ repository. One has to clone it to get the /working directory/ along with it.
Anyway, back to our topic. This is a great solution if you don't have to worry about things you would like to hide.
By hide, I mean things like /credentials/, /keys/ or /passwords/ which *never* belong in a /repository/.
You will need to find solutions for these types of files. I was looking for something /less involving/ and /more involved/.
**** /Chezmoi/ to the rescue ?
Isn't that what they *all* say ?
I like how the creator(s) defines [[https://www.chezmoi.io/][/Chezmoi/]]
#+BEGIN_QUOTE
Manage your dotfiles across multiple machines, securely.
#+END_QUOTE
Pretty basic, straight to the point. Unfortunately, it's a little bit harder to grasp the concept of how it works.
/Chezmoi/ basically /generates/ the /dotfiles/ from the /local repository/. These /dotfiles/ are saved in different forms in the /repository/ but they *always* generate the same output; the /dotfiles/. Think of /Chezmoi/ as a /dotfiles/ templating engine, at its basic form it saves your /dotfiles/ as is and /deploys/ them in *any* machine.
**** Working with /Chezmoi/
I think we should take a /quick/ look at /Chezmoi/ to see how it works.
/Chezmoi/ is written /Golang/ making it /fairly/ easy to [[https://www.chezmoi.io/docs/install/][install]] so I will forgo that boring part.
***** First run
To start using /Chezmoi/, one has to *initialize* a new /Chezmoi repository/.
#+BEGIN_SRC bash
chezmoi init
#+END_SRC
This will create a *new* /git repository/ in =~/.local/share/chezmoi=. This is now the *source state*, where /Chezmoi/ will get your /dotfiles/.
***** Plain /dotfiles/ management with /Chezmoi/
Now that we have a /Chezmoi/ repository. We can start to /populate/ it with /dotfiles/.
Let's assume that we would like to start managing one of our /dotfiles/ with /Chezmoi/.
I'm going with an /imaginary application/'s configuration directory.
This directory will hold different files with /versatile/ content types.
This is going to showcase some of /Chezmoi/'s capabilities.
#+BEGIN_EXPORT html
<div class="admonition note">
<p class="admonition-title"><b>Note</b></p>
#+END_EXPORT
This is how I use /Chezmoi/. If you have a better way to do things, I'd like to hear about it!
#+BEGIN_EXPORT html
</div>
#+END_EXPORT
****** Adding a /dotfile/
This *DS9* application has its directory configuration in =~/.ds9/= where we find the =config=.
The configuration looks like any /generic/ /ini/ configuration.
#+BEGIN_SRC ini :tangle ~/.ds9/config
[character/sisko]
Name = Benjamin
Rank = Captain
Credentials = sisko-creds.cred
Mastodon = sisko-api.mastodon
#+END_SRC
/Nothing/ special about this file, let's add it to /Chezmoi/
#+BEGIN_SRC bash
chezmoi add ~/.ds9/config
#+END_SRC
****** Listing /dotfiles/
And /nothing/ happened... Hmm...
#+BEGIN_SRC bash
chezmoi managed
#+END_SRC
#+BEGIN_EXAMPLE
/home/user/.ds9
/home/user/.ds9/config
#+END_EXAMPLE
Okay, it seems that it is being managed.
****** Diffing /dotfiles/
We can /test/ it out by doing something like this.
#+BEGIN_SRC bash
mv ~/.ds9/config ~/.ds9/config.old
chezmoi diff
#+END_SRC
#+BEGIN_EXAMPLE
install -m 644 /dev/null /home/user/.ds9/config
--- a/home/user/.ds9/config
+++ b/home/user/.ds9/config
@@ -0,0 +1,5 @@
+[character/sisko]
+Name = Benjamin
+Rank = Captain
+Credentials = sisko-creds.cred
+Mastodon = sisko-api.mastodon
#+END_EXAMPLE
Alright, everything looks as it should be.
****** Apply /dotfiles/
But that's only a /diff/, how do I make /Chezmoi/ apply the changes because my /dotfile/ is still =config.old=.
Okay, we can actually get rid of the =config.old= file and make /Chezmoi/ regenerate the configuration.
#+BEGIN_SRC bash
rm ~/.ds9/config ~/.ds9/config.old
chezmoi -v apply
#+END_SRC
#+BEGIN_EXPORT html
<div class="admonition note">
<p class="admonition-title"><b>Note</b></p>
#+END_EXPORT
I like to use the =-v= flag to check what is *actually* being applied.
#+BEGIN_EXPORT html
</div>
#+END_EXPORT
#+BEGIN_EXAMPLE
install -m 644 /dev/null /home/user/.ds9/config
--- a/home/user/.ds9/config
+++ b/home/user/.ds9/config
@@ -0,0 +1,5 @@
+[character/sisko]
+Name = Benjamin
+Rank = Captain
+Credentials = sisko-creds.cred
+Mastodon = sisko-api.mastodon
#+END_EXAMPLE
And we get the same output as the =diff=. Nice!
The configuration file was also recreated, that's awesome.
****** Editing /dotfiles/
If you've followed so far, you might have wondered... If I edit =~/.ds9/config=, then /Chezmoi/ is going to *override* it!
*YES*, *yes* it will.
#+BEGIN_EXPORT html
<div class="admonition warning">
<p class="admonition-title"><b>Warning</b></p>
#+END_EXPORT
Always use /Chezmoi/ to edit your managed /dotfiles/. Do *NOT* edit them directly.
*ALWAYS* use =chezmoi diff= before every /applying/.
#+BEGIN_EXPORT html
</div>
#+END_EXPORT
To /edit/ your managed /dotfile/, simply tell /Chezmoi/ about it.
#+BEGIN_SRC bash
chezmoi edit ~/.ds9/config
#+END_SRC
/Chezmoi/ will use your =$EDITOR= to open the file for you to edit. Once saved, it's saved in the /repository database/.
Be aware, at this point the changes are not reflected in your /home/ directory, *only* in the /Chezmoi source state/. Make sure you *diff* and then *apply* to make the changes in your /home/.
***** /Chezmoi/ repository management
As mentioned previously, the repository is found in =~/.local/share/chezmoi=.
I *always* forget where it is, luckily /Chezmoi/ has a solution for that.
#+BEGIN_SRC bash
chezmoi cd
#+END_SRC
Now, we are in the repository. We can work with it as a /regultar/ /git/ repository.
When you're done, don't forget to =exit=.
***** Other features
It is worth mentioning at this point that /Chezmoi/ offers a few more integrations.
****** Templating
Due to the fact that /Chezmoi/ is written in /Golang/, it can leverage the power of the /Golang [[https://www.chezmoi.io/docs/how-to/#use-templates-to-manage-files-that-vary-from-machine-to-machine][templating]]/ system.
One can replace /repeatable/ values like *email* or *name* with a template like ={{ .email }}= or ={{ .name }}=.
This will result in a replacement of these /templated variables/ with their real values in the resulting /dotfile/.
This is another reason why you should *always* edit your managed /dotfiles/ through /Chezmoi/.
Our /previous/ example would look a bit different.
#+BEGIN_SRC ini :tangle ~/.ds9/config
[character/sisko]
Name = {{ .sisko.name }}
Rank = {{ .sisko.rank }}
Credentials = sisko-creds.cred
Mastodon = sisko-api.mastodon
#+END_SRC
And we would add it a bit differently now.
#+BEGIN_SRC bash
chezmoi add --template ~/.ds9/config
#+END_SRC
#+BEGIN_EXPORT html
<div class="admonition warning">
<p class="admonition-title"><b>Warning</b></p>
#+END_EXPORT
Follow the [[https://www.chezmoi.io/docs/how-to/#use-templates-to-manage-files-that-vary-from-machine-to-machine][documentation]] to /configure/ the *values*.
#+BEGIN_EXPORT html
</div>
#+END_EXPORT
****** Password manager integration
Once you have the power of /templating/ on your side, you can always take it one step further.
/Chezmoi/ has integration with a big list of [[https://www.chezmoi.io/docs/how-to/#keep-data-private][password managers]]. These can be used directly into the /configuration files/.
In our /hypothetical/ example, we can think of the /credentials/ file (=~/.ds9/sisko-creds.cred=).
#+BEGIN_SRC init :tangle ~/.ds9/sisko-creds.cred
Name = {{ (keepassxc "sisko.ds9").Name }}
Rank = {{ (keepassxc "sisko.ds9").Rank }}
Access_Code = {{ (keepassxc "sisko.ds9").AccessCode }}
#+END_SRC
Do not /forget/ that this is also using the /templating/ engine. So you need to add as a /template/.
#+BEGIN_SRC bash
chezmoi add --template ~/.ds9/sisko-creds.cred
#+END_SRC
****** File encryption
Wait, what ! You almost slipped away right there old fellow.
We have our /Mastodon/ *API* key in the =sisko-api.mastodon= file. The whole file cannot be pushed to a repository.
It turns out that /Chezmoi/ can use /gpg/ to [[https://www.chezmoi.io/docs/how-to/#use-gpg-to-keep-your-secrets][encrypt your files]] making it possible for you to push them.
To add a file encrypted to the /Chezmoi/ repository, use the following command.
#+BEGIN_SRC bash
chezmoi add --encrypt ~/.ds9/sisko-api.mastodon
#+END_SRC
****** Misc
There is a list of other features that /Chezmoi/ supports that I did not mention.
I did not use all the /features/ offered yet. You should check the [[https://www.chezmoi.io/][website]] for the full documentation.
**** Conclusion
I am fully migrated into /Chezmoi/ so far. I have used all the features above, and it has worked flawlessly so far.
I like the idea that it offers *all* the features I need while at the same time staying out of the way.
I find myself, often, editing the /dotfiles/ in my /home/ directory as a /dev/ version. Once I get to a configuration I like, I add it to /Chezmoi/. If I ever mess up badly, I ask /Chezmoi/ to override my changes.
I understand it adds a little bit of /overhead/ with the use of =chezmoi= commands, which I aliased to =cm=. But the end result is a /home/ directory which seems untouched by any tools (no symlinks, no copies, etc...) making it easier to migrate /out/ of /Chezmoi/ as a solution and into another one if I ever choose in the future.
** Configuration Management :@configuration_management:
*** DONE Ansible testing with Molecule :ansible:molecule:
:PROPERTIES:
:EXPORT_HUGO_LASTMOD: 2019-06-21
:EXPORT_DATE: 2019-01-11
:EXPORT_FILE_NAME: ansible-testing-with-molecule
:CUSTOM_ID: ansible-testing-with-molecule
:END:
When I first started using [[https://www.ansible.com/][ansible]], I did not know about [[https://molecule.readthedocs.io/en/latest/][molecule]]. It was a bit daunting to start a /role/ from scratch and trying to develop it without having the ability to test it. Then a co-worker of mine told me about molecule and everything changed.
#+hugo: more
I do not have any of the tools I need installed on this machine, so I will go through, step by step, how I set up ansible and molecule on any new machine I come across for writing ansible roles.
**** Requirements
What we are trying to achieve in this post, is a working ansible role that can be tested inside a docker container. To be able to achieve that, we need to install docker on the system. Follow the instructions on [[https://docs.docker.com/install/][installing docker]] found on the docker website.
**** Good Practices
First thing's first. Let's start by making sure that we have python installed properly on the system.
#+BEGIN_EXAMPLE
$ python --version
Python 3.7.1
#+END_EXAMPLE
Because in this case I have /python3/ installed, I can create a /virtualenv/ easier without the use of external tools.
#+BEGIN_EXAMPLE
# Create the directory to work with
$ mkdir -p sandbox/test-roles
# Navigate to the directory
$ cd sandbox/test-roles/
# Create the virtualenv
~/sandbox/test-roles $ python -m venv .ansible-venv
# Activate the virtualenv
~/sandbox/test-roles $ source .ansible-venv/bin/activate
# Check that your virtualenv activated properly
(.ansible-venv) ~/sandbox/test-roles $ which python
/home/elijah/sandbox/test-roles/.ansible-venv/bin/python
#+END_EXAMPLE
At this point, we can install the required dependencies.
#+BEGIN_EXAMPLE
$ pip install ansible molecule docker
Collecting ansible
Downloading https://files.pythonhosted.org/packages/56/fb/b661ae256c5e4a5c42859860f59f9a1a0b82fbc481306b30e3c5159d519d/ansible-2.7.5.tar.gz (11.8MB)
100% |████████████████████████████████| 11.8MB 3.8MB/s
Collecting molecule
Downloading https://files.pythonhosted.org/packages/84/97/e5764079cb7942d0fa68b832cb9948274abb42b72d9b7fe4a214e7943786/molecule-2.19.0-py3-none-any.whl (180kB)
100% |████████████████████████████████| 184kB 2.2MB/s
...
Successfully built ansible ansible-lint anyconfig cerberus psutil click-completion tabulate tree-format pathspec future pycparser arrow
Installing collected packages: MarkupSafe, jinja2, PyYAML, six, pycparser, cffi, pynacl, idna, asn1crypto, cryptography, bcrypt, paramiko, ansible, pbr, git-url-parse, monotonic, fasteners, click, colorama, sh, python-gilt, ansible-lint, pathspec, yamllint, anyconfig, cerberus, psutil, more-itertools, py, attrs, pluggy, atomicwrites, pytest, testinfra, ptyprocess, pexpect, click-completion, tabulate, future, chardet, binaryornot, poyo, urllib3, certifi, requests, python-dateutil, arrow, jinja2-time, whichcraft, cookiecutter, tree-format, molecule, docker-pycreds, websocket-client, docker
Successfully installed MarkupSafe-1.1.0 PyYAML-3.13 ansible-2.7.5 ansible-lint-3.4.23 anyconfig-0.9.7 arrow-0.13.0 asn1crypto-0.24.0 atomicwrites-1.2.1 attrs-18.2.0 bcrypt-3.1.5 binaryornot-0.4.4 cerberus-1.2 certifi-2018.11.29 cffi-1.11.5 chardet-3.0.4 click-6.7 click-completion-0.3.1 colorama-0.3.9 cookiecutter-1.6.0 cryptography-2.4.2 docker-3.7.0 docker-pycreds-0.4.0 fasteners-0.14.1 future-0.17.1 git-url-parse-1.1.0 idna-2.8 jinja2-2.10 jinja2-time-0.2.0 molecule-2.19.0 monotonic-1.5 more-itertools-5.0.0 paramiko-2.4.2 pathspec-0.5.9 pbr-4.1.0 pexpect-4.6.0 pluggy-0.8.1 poyo-0.4.2 psutil-5.4.6 ptyprocess-0.6.0 py-1.7.0 pycparser-2.19 pynacl-1.3.0 pytest-4.1.0 python-dateutil-2.7.5 python-gilt-1.2.1 requests-2.21.0 sh-1.12.14 six-1.11.0 tabulate-0.8.2 testinfra-1.16.0 tree-format-0.1.2 urllib3-1.24.1 websocket-client-0.54.0 whichcraft-0.5.2 yamllint-1.11.1
#+END_EXAMPLE
**** Creating your first ansible role
Once all the steps above are complete, we can start by creating our first ansible role.
#+BEGIN_EXAMPLE
$ molecule init role -r example-role
--> Initializing new role example-role...
Initialized role in /home/elijah/sandbox/test-roles/example-role successfully.
$ tree example-role/
example-role/
├── defaults
│ └── main.yml
├── handlers
│ └── main.yml
├── meta
│ └── main.yml
├── molecule
│ └── default
│ ├── Dockerfile.j2
│ ├── INSTALL.rst
│ ├── molecule.yml
│ ├── playbook.yml
│ └── tests
│ ├── __pycache__
│ │ └── test_default.cpython-37.pyc
│ └── test_default.py
├── README.md
├── tasks
│ └── main.yml
└── vars
└── main.yml
9 directories, 12 files
#+END_EXAMPLE
You can find what each directory is for and how ansible works by visiting [[https://docs.ansible.com][docs.ansible.com]].
***** =meta/main.yml=
The meta file needs to modified and filled with information about the role. This is not a required file to modify if you are keeping this for yourself, for example. But it is a good idea to have as much information as possible if this is going to be released. In my case, I don't need any fanciness as this is just sample code.
#+BEGIN_SRC yaml
---
galaxy_info:
author: Elia el Lazkani
description: This is an example ansible role to showcase molecule at work
license: license (BDS-2)
min_ansible_version: 2.7
galaxy_tags: []
dependencies: []
#+END_SRC
***** =tasks/main.yml=
This is where the magic is set in motion. Tasks are the smallest entities in a role that do small and idempotent actions. Let's write a few simple tasks to create a user and install a service.
#+BEGIN_SRC yaml
---
# Create the user example
- name: Create 'example' user
user:
name: example
comment: Example user
shell: /bin/bash
state: present
create_home: yes
home: /home/example
# Install nginx
- name: Install nginx
apt:
name: nginx
state: present
update_cache: yes
notify: Restart nginx
#+END_SRC
***** =handlers/main.yml=
If you noticed, we are notifying a handler to be called after installing /nginx/. All handlers notified will run after all the tasks complete and each handler will only run once. This is a good way to make sure that you don't restart /nginx/ multiple times if you call the handler more than once.
#+BEGIN_SRC yaml
---
# Handler to restart nginx
- name: Restart nginx
service:
name: nginx
state: restarted
#+END_SRC
***** =molecule/default/molecule.yml=
It's time to configure molecule to do what we need. We need to start an ubuntu docker container, so we need to specify that in the molecule YAML file. All we need to do is change the image line to specify that we want an =ubuntu:bionic= image.
#+BEGIN_SRC yaml
---
dependency:
name: galaxy
driver:
name: docker
lint:
name: yamllint
platforms:
- name: instance
image: ubuntu:bionic
provisioner:
name: ansible
lint:
name: ansible-lint
scenario:
name: default
verifier:
name: testinfra
lint:
name: flake8
#+END_SRC
***** =molecule/default/playbook.yml=
This is the playbook that molecule will run. Make sure that you have all the steps that you need here. I will keep this as is.
#+BEGIN_SRC yaml
---
- name: Converge
hosts: all
roles:
- role: example-role
#+END_SRC
**** First Role Pass
This is time to test our role and see what's going on.
#+BEGIN_EXAMPLE
(.ansible-role) ~/sandbox/test-roles/example-role/ $ molecule converge
--> Validating schema /home/elijah/sandbox/test-roles/example-role/molecule/default/molecule.yml.
Validation completed successfully.
--> Test matrix
└── default
├── dependency
├── create
├── prepare
└── converge
--> Scenario: 'default'
--> Action: 'dependency'
Skipping, missing the requirements file.
--> Scenario: 'default'
--> Action: 'create'
PLAY [Create] ******************************************************************
TASK [Log into a Docker registry] **********************************************
skipping: [localhost] => (item=None)
TASK [Create Dockerfiles from image names] *************************************
changed: [localhost] => (item=None)
changed: [localhost]
TASK [Discover local Docker images] ********************************************
ok: [localhost] => (item=None)
ok: [localhost]
TASK [Build an Ansible compatible image] ***************************************
changed: [localhost] => (item=None)
changed: [localhost]
TASK [Create docker network(s)] ************************************************
TASK [Create molecule instance(s)] *********************************************
changed: [localhost] => (item=None)
changed: [localhost]
TASK [Wait for instance(s) creation to complete] *******************************
changed: [localhost] => (item=None)
changed: [localhost]
PLAY RECAP *********************************************************************
localhost : ok=5 changed=4 unreachable=0 failed=0
--> Scenario: 'default'
--> Action: 'prepare'
Skipping, prepare playbook not configured.
--> Scenario: 'default'
--> Action: 'converge'
PLAY [Converge] ****************************************************************
TASK [Gathering Facts] *********************************************************
ok: [instance]
TASK [example-role : Create 'example' user] ************************************
changed: [instance]
TASK [example-role : Install nginx] ********************************************
changed: [instance]
RUNNING HANDLER [example-role : Restart nginx] *********************************
changed: [instance]
PLAY RECAP *********************************************************************
instance : ok=4 changed=3 unreachable=0 failed=0
#+END_EXAMPLE
It looks like the *converge* step succeeded.
**** Writing Tests
It is always a good practice to write unittests when you're writing code. Ansible roles should not be an exception. Molecule offers a way to run tests, which you can think of as unittest, to make sure that what the role gives you is what you were expecting. This helps future development of the role and keeps you from falling in previously solved traps.
***** =molecule/default/tests/test_default.py=
Molecule leverages the [[https://testinfra.readthedocs.io/en/latest/][testinfra]] project to run its tests. You can use other tools if you so wish, and there are many. In this example we will be using /testinfra/.
#+BEGIN_SRC python
import os
import testinfra.utils.ansible_runner
testinfra_hosts = testinfra.utils.ansible_runner.AnsibleRunner(
os.environ['MOLECULE_INVENTORY_FILE']).get_hosts('all')
def test_hosts_file(host):
f = host.file('/etc/hosts')
assert f.exists
assert f.user == 'root'
assert f.group == 'root'
def test_user_created(host):
user = host.user("example")
assert user.name == "example"
assert user.home == "/home/example"
def test_user_home_exists(host):
user_home = host.file("/home/example")
assert user_home.exists
assert user_home.is_directory
def test_nginx_is_installed(host):
nginx = host.package("nginx")
assert nginx.is_installed
def test_nginx_running_and_enabled(host):
nginx = host.service("nginx")
assert nginx.is_running
#+END_SRC
#+BEGIN_EXPORT html
<div class="admonition warning">
<p class="admonition-title"><b>Warning</b></p>
#+END_EXPORT
Uncomment =truthy: disable= in =.yamllint= found at the base of the role.
#+BEGIN_EXPORT html
</div>
#+END_EXPORT
#+BEGIN_EXAMPLE
(.ansible_venv) ~/sandbox/test-roles/example-role $ molecule test
--> Validating schema /home/elijah/sandbox/test-roles/example-role/molecule/default/molecule.yml.
Validation completed successfully.
--> Test matrix
└── default
├── lint
├── destroy
├── dependency
├── syntax
├── create
├── prepare
├── converge
├── idempotence
├── side_effect
├── verify
└── destroy
--> Scenario: 'default'
--> Action: 'lint'
--> Executing Yamllint on files found in /home/elijah/sandbox/test-roles/example-role/...
Lint completed successfully.
--> Executing Flake8 on files found in /home/elijah/sandbox/test-roles/example-role/molecule/default/tests/...
/home/elijah/.virtualenvs/world/lib/python3.7/site-packages/pycodestyle.py:113: FutureWarning: Possible nested set at position 1
EXTRANEOUS_WHITESPACE_REGEX = re.compile(r'[[({] | []}),;:]')
Lint completed successfully.
--> Executing Ansible Lint on /home/elijah/sandbox/test-roles/example-role/molecule/default/playbook.yml...
Lint completed successfully.
--> Scenario: 'default'
--> Action: 'destroy'
PLAY [Destroy] *****************************************************************
TASK [Destroy molecule instance(s)] ********************************************
changed: [localhost] => (item=None)
changed: [localhost]
TASK [Wait for instance(s) deletion to complete] *******************************
ok: [localhost] => (item=None)
ok: [localhost]
TASK [Delete docker network(s)] ************************************************
PLAY RECAP *********************************************************************
localhost : ok=2 changed=1 unreachable=0 failed=0
--> Scenario: 'default'
--> Action: 'dependency'
Skipping, missing the requirements file.
--> Scenario: 'default'
--> Action: 'syntax'
playbook: /home/elijah/sandbox/test-roles/example-role/molecule/default/playbook.yml
--> Scenario: 'default'
--> Action: 'create'
PLAY [Create] ******************************************************************
TASK [Log into a Docker registry] **********************************************
skipping: [localhost] => (item=None)
TASK [Create Dockerfiles from image names] *************************************
changed: [localhost] => (item=None)
changed: [localhost]
TASK [Discover local Docker images] ********************************************
ok: [localhost] => (item=None)
ok: [localhost]
TASK [Build an Ansible compatible image] ***************************************
changed: [localhost] => (item=None)
changed: [localhost]
TASK [Create docker network(s)] ************************************************
TASK [Create molecule instance(s)] *********************************************
changed: [localhost] => (item=None)
changed: [localhost]
TASK [Wait for instance(s) creation to complete] *******************************
changed: [localhost] => (item=None)
changed: [localhost]
PLAY RECAP *********************************************************************
localhost : ok=5 changed=4 unreachable=0 failed=0
--> Scenario: 'default'
--> Action: 'prepare'
Skipping, prepare playbook not configured.
--> Scenario: 'default'
--> Action: 'converge'
PLAY [Converge] ****************************************************************
TASK [Gathering Facts] *********************************************************
ok: [instance]
TASK [example-role : Create 'example' user] ************************************
changed: [instance]
TASK [example-role : Install nginx] ********************************************
changed: [instance]
RUNNING HANDLER [example-role : Restart nginx] *********************************
changed: [instance]
PLAY RECAP *********************************************************************
instance : ok=4 changed=3 unreachable=0 failed=0
--> Scenario: 'default'
--> Action: 'idempotence'
Idempotence completed successfully.
--> Scenario: 'default'
--> Action: 'side_effect'
Skipping, side effect playbook not configured.