If you have used Docker on a Mac
laptop, you have likely ran into the "crossing file systems"
issue that can lead to really poor performance in your containers.
This becomes frustrating if you are recreating a development environment
for applications that rely on a lot of services (micro or otherwise)
to run. If you are a Windows or Linux user, this sort of thing is pretty much
a non-issue.
It seemed like I had two choices on how to potentially deal with this.
Choice number one was to switch to a different operating system (I have been looking
at Framework laptops) but I was (LOL) grumpy about
getting a new laptop when the 2020 13" MacBook pro I have as I write this is only slightly
more than a year old. Okay, I could sell it. But that seems like a bunch
of busy work.
The second choice was to see if I could create a development environment
that was faster and still allowed me to use Docker, which is 100% a requirement
for the client work I am currently doing.
In the past, other people online had hinted at the possibility of doing the development
work inside a virtual machine. Basically like connecting to another machine
via SSH and then doing my work in there. Good thing I am already a member of the
Cult of Vim.
Okay, I think I could work with this. The next issue was HOW to do this all. I've played
around enough with Vagrant, VMWare and
Virtual Private Servers to know that the steepest part of the curve with using
this solution is getting configured and in a state where it useful rather than
a time-sink or a toy.
I am always one to extol the virtues of "better lucky than good" and I happened to
stumble across a tweet from Mitchell Hashimoto (one of the creators of Vagrant)
where he talked about how he does all his development work inside a VM running
on a Mac laptop. I can't find the original tweet, but I made a note about it
and then he did another tweet about it:
My NixOS configurations for my dev VM setup are finally sanitized and open source. They work for both Intel and Apple Silicon. And I put together a video showing how I setup a new machine!
(originally posted at https://twitter.com/mitchellh/status/1452721115009191938)
In that tweet he also shared a link to a YouTube video and
the GitHub repo that he was publicly sharing
that you could use as a template.
Again, in a big coincidence, Mr. Hashimoto was also using NixOS as his
VM's operating system. NixOS is a Linux distribution based on Nix package manager.
Which doesn't mean much if you're not a fan of their approach.
The reason to like Nix is that it bills itself as a purely functional package manager.
If you've ever come across the "functional style" of programming then you might know
how powerful it is to have code that has no "side effects". This means that your
code is not doing things like writing to the file system, or updating records in a database.
In a functional world (unless the code is supposed return something random) it is easy
to test code because the same input should always generate the same output.
If you apply this to package management, it means your are choosing and installing dependencies
in such a way that you can isolate them from each other. It can be difficult to have, for example,
two different versions of PHP installed for your use unless you are relying on third party tools
to keep track of what version should be active.
Nix handles this by forcing you to declare any dependencies explicitly. No more
worrying about globally-installed libraries causing incompatibility problems.
Want to see if your PHP project runs on 7.4, 8.0, and 8.1? You can do that easily
with NixOS and it's tooling.
I guess you can tell I am a fan of NixOS and look forward to using it a lot more.
The idea from a high level is this: given a NixOS VM running in VMWare (I am using
VMWare Fusion) it should take less than 10 minutes to create a development
environment configured with my preferred tools installed from scratch.
The repo he provides is definitely not ready to go as-is. You will need to modify
a lot of the things in there -- I know I did. It took about a week of poking at it,
creating and destroying lots of VM's, and learning how NixOS wants to do things to
get it to the point where I could get it up and running and actually use it.
I ended up removing a bunch of tools that are related to Mitchell's work on Docker
and added a few things I knew I was going to need for my work with this client. It
also took me a while to figure out how to generate a hash for the password for the
user account the build-and-configure process can create for you. But in the
end I had a VM up and running (that I could also SSH into if I wanted to)
By default, the VM uses a graphical interface with a tiling window manager
and you type Command-N on your Mac and it opens up a terminal session in
Kitty and you are ready to go!.
I also had modified the configuration to install Docker and the related
command-line tools. Once I cloned the client repo all I had to do was
make build to create the Docker containers the development environment
needed and make unit to run the unit test suite in less than half the time.
So what is the point of doing all this work? Let's go back to my original
problem. Running a test suite that used a development environment consisting
of multiple Docker containers was incredibly slow. Running on my Mac (and giving
Docker half the cores and half the available memory) it takes about 80 seconds.
Running inside a VM that has access to the same resources takes about 35 seconds.
If you've never done a development work flow of "make a change, run a process to
verify the change works as expected" for a large chunk of your work day then perhaps
you don't think this is a big deal. Every loop also has come context switching
as you try and figure out what happened. If you do this 100 times in a day, you
probably want this loop to run as quickly as possible.
I don't know if there is a phrase or concept or "law" about this sort of perception-versus-reality
issue, but I find myself wanting to get things done FASTER when the process by
which results are determined gets SLOWER. When my test suite runs in 30 seconds,
I feel like I have lots of time to solve the problem. When it takes a minute-and-a-half
I get...anxious? Maybe that's the wrong emotion. I know something takes too long
when I start muttering to myself "this is taking too long.".
With the "happens faster than before" issue solved, I find the next benefit to be
as I learn how to use Nix to build repeatable environments, a major mistake can be
solved by deleting the VM and trying again. Maybe 20 minutes tops to get back to
where I was. When I mess up my development environment
on my MacBook (OS update or maybe Homebrew updates an underlying
dependency) it can be a whole afternoon spent trying to "fix whatever I broke."
It's not clear there will ever be any kind of solution for the "crossing file systems"
issue that leads to Docker performing so poorly. My fellow programmers who are
running the newer MacBooks that use Apple's new chips tell me performance is
quite good. Spending another CAD$3k so Docker runs faster seems like a waste of
money to me, but that is just a personal opinion.
I used to be someone who lived on the bleeding edge when it came to their software.
As I got older and grumpier I started to value stability and repeatability in my
software more. A development environment that can be built using Make
and a VM is one I can rely on to start me off at a known point, exhibiting
behaviour I am expecting. Almost like the benefits of a test suite!
Doing this NixOS-in-a-VM stuff relies on you having some experience with Linux
environments. I did run Linux as my desktop environment for several years
before I started buying Apple hardware (which I've done since 2002) and
MacOS's "BSD with a pretty window manager" approach also let me use those
command-line skills.
I highly recommend watching Mitchell's YouTube video as he explains how the
whole process works. I found it useful because he explains the philosophy of
his approach. Understanding WHY someone does things can often lead you to
quicker insights as to what needs to change to fit your needs.
If you do get it all set up, let me know your experiences. With some effort
I can probably create a more generic version of my set up and create a GitHub
repo with all the files in it.