If you have used Docker on a Mac laptop, you have likely ran into the "crossing file systems" issue that can lead to really poor performance in your containers. This becomes frustrating if you are recreating a development environment for applications that rely on a lot of services (micro or otherwise) to run. If you are a Windows or Linux user, this sort of thing is pretty much a non-issue.

It seemed like I had two choices on how to potentially deal with this. Choice number one was to switch to a different operating system (I have been looking at Framework laptops) but I was (LOL) grumpy about getting a new laptop when the 2020 13" MacBook pro I have as I write this is only slightly more than a year old. Okay, I could sell it. But that seems like a bunch of busy work.

The second choice was to see if I could create a development environment that was faster and still allowed me to use Docker, which is 100% a requirement for the client work I am currently doing.

In the past, other people online had hinted at the possibility of doing the development work inside a virtual machine. Basically like connecting to another machine via SSH and then doing my work in there. Good thing I am already a member of the Cult of Vim.

Okay, I think I could work with this. The next issue was HOW to do this all. I've played around enough with Vagrant, VMWare and Virtual Private Servers to know that the steepest part of the curve with using this solution is getting configured and in a state where it useful rather than a time-sink or a toy.

I am always one to extol the virtues of "better lucky than good" and I happened to stumble across a tweet from Mitchell Hashimoto (one of the creators of Vagrant) where he talked about how he does all his development work inside a VM running on a Mac laptop. I can't find the original tweet, but I made a note about it and then he did another tweet about it:

My NixOS configurations for my dev VM setup are finally sanitized and open source. They work for both Intel and Apple Silicon. And I put together a video showing how I setup a new machine! (originally posted at https://twitter.com/mitchellh/status/1452721115009191938)

In that tweet he also shared a link to a YouTube video and the GitHub repo that he was publicly sharing that you could use as a template.

Again, in a big coincidence, Mr. Hashimoto was also using NixOS as his VM's operating system. NixOS is a Linux distribution based on Nix package manager. Which doesn't mean much if you're not a fan of their approach.

The reason to like Nix is that it bills itself as a purely functional package manager. If you've ever come across the "functional style" of programming then you might know how powerful it is to have code that has no "side effects". This means that your code is not doing things like writing to the file system, or updating records in a database. In a functional world (unless the code is supposed return something random) it is easy to test code because the same input should always generate the same output.

If you apply this to package management, it means your are choosing and installing dependencies in such a way that you can isolate them from each other. It can be difficult to have, for example, two different versions of PHP installed for your use unless you are relying on third party tools to keep track of what version should be active.

Nix handles this by forcing you to declare any dependencies explicitly. No more worrying about globally-installed libraries causing incompatibility problems. Want to see if your PHP project runs on 7.4, 8.0, and 8.1? You can do that easily with NixOS and it's tooling.

I guess you can tell I am a fan of NixOS and look forward to using it a lot more.

The idea from a high level is this: given a NixOS VM running in VMWare (I am using VMWare Fusion) it should take less than 10 minutes to create a development environment configured with my preferred tools installed from scratch.

The repo he provides is definitely not ready to go as-is. You will need to modify a lot of the things in there -- I know I did. It took about a week of poking at it, creating and destroying lots of VM's, and learning how NixOS wants to do things to get it to the point where I could get it up and running and actually use it.

I ended up removing a bunch of tools that are related to Mitchell's work on Docker and added a few things I knew I was going to need for my work with this client. It also took me a while to figure out how to generate a hash for the password for the user account the build-and-configure process can create for you. But in the end I had a VM up and running (that I could also SSH into if I wanted to)

By default, the VM uses a graphical interface with a tiling window manager and you type Command-N on your Mac and it opens up a terminal session in Kitty and you are ready to go!.

I also had modified the configuration to install Docker and the related command-line tools. Once I cloned the client repo all I had to do was make build to create the Docker containers the development environment needed and make unit to run the unit test suite in less than half the time.

So what is the point of doing all this work? Let's go back to my original problem. Running a test suite that used a development environment consisting of multiple Docker containers was incredibly slow. Running on my Mac (and giving Docker half the cores and half the available memory) it takes about 80 seconds. Running inside a VM that has access to the same resources takes about 35 seconds.

If you've never done a development work flow of "make a change, run a process to verify the change works as expected" for a large chunk of your work day then perhaps you don't think this is a big deal. Every loop also has come context switching as you try and figure out what happened. If you do this 100 times in a day, you probably want this loop to run as quickly as possible.

I don't know if there is a phrase or concept or "law" about this sort of perception-versus-reality issue, but I find myself wanting to get things done FASTER when the process by which results are determined gets SLOWER. When my test suite runs in 30 seconds, I feel like I have lots of time to solve the problem. When it takes a minute-and-a-half I get...anxious? Maybe that's the wrong emotion. I know something takes too long when I start muttering to myself "this is taking too long.".

With the "happens faster than before" issue solved, I find the next benefit to be as I learn how to use Nix to build repeatable environments, a major mistake can be solved by deleting the VM and trying again. Maybe 20 minutes tops to get back to where I was. When I mess up my development environment on my MacBook (OS update or maybe Homebrew updates an underlying dependency) it can be a whole afternoon spent trying to "fix whatever I broke."

It's not clear there will ever be any kind of solution for the "crossing file systems" issue that leads to Docker performing so poorly. My fellow programmers who are running the newer MacBooks that use Apple's new chips tell me performance is quite good. Spending another CAD$3k so Docker runs faster seems like a waste of money to me, but that is just a personal opinion.

I used to be someone who lived on the bleeding edge when it came to their software. As I got older and grumpier I started to value stability and repeatability in my software more. A development environment that can be built using Make and a VM is one I can rely on to start me off at a known point, exhibiting behaviour I am expecting. Almost like the benefits of a test suite!

Doing this NixOS-in-a-VM stuff relies on you having some experience with Linux environments. I did run Linux as my desktop environment for several years before I started buying Apple hardware (which I've done since 2002) and MacOS's "BSD with a pretty window manager" approach also let me use those command-line skills.

I highly recommend watching Mitchell's YouTube video as he explains how the whole process works. I found it useful because he explains the philosophy of his approach. Understanding WHY someone does things can often lead you to quicker insights as to what needs to change to fit your needs.

If you do get it all set up, let me know your experiences. With some effort I can probably create a more generic version of my set up and create a GitHub repo with all the files in it.