Bundling Common Rake Tasks into a Gem
Note: The code snippets in this post are licensed as Apache-2.0 and available at jamietanna/example-cookbook-helper-gem.
Foreword
As the number of cookbooks I'm working on grows, there are a number of common tasks that I need to have automated. These include, but are not limited to, linting, unit testing and generating documentation. For cookbooks I test with GitLab CI, I also autogenerate the .gitlab-ci.yml file, creating a separate phase per test suite. These tasks are performed using Rake, the Ruby task runner.
Having multiple copies of the same code was starting to grind on me, and as part of my Chef 13 upgrades, I've found there are a number of style changes to take effect in each Rakefile. Instead of making the changes in each repo, I bit the bullet and learned how to create my own Gem which would contain these tasks and make it easier to upgrade all Gems in one fell swoop. I've written this post to document it for posterity, and to show how easy it is to do.
Note: All the below commands are prefaced with chef exec, to tie the version of Ruby, RubyGems and Bundler with the version in the ChefDK, making sure all dependencies will be aligned.
Determining Dependency Versions
Now, this seems like quite an easy task, but to throw a slight spanner in the works, we'll also require that the Gem tests against the exact versions of tools as in the ChefDK at version 2.4.17, which is the version I'm currently using. This ensures that across any machine, they'll only ever use the right versions, instead of potentially polluting our installation with incorrect Gem versions.
To check the versions of each of the dependencies we have, I ran the following:
$ docker run --rm chef/chefdk:2.4.17 chef gem list |\
grep -e rubocop -e chef -e berkshelf -e chefspec -e foodcritic
berkshelf (6.3.1)
chef (13.6.4)
chef-api (0.7.1)
chef-config (13.6.4)
chef-dk (2.4.17)
chef-provisioning (2.6.0)
chef-provisioning-aws (3.0.0)
chef-provisioning-azure (0.6.0)
chef-provisioning-fog (0.26.0)
chef-sugar (3.6.0)
chef-vault (3.3.0)
chef-zero (13.1.0)
cheffish (13.1.0)
chefspec (7.1.1)
chefstyle (0.6.0)
foodcritic (12.2.1)
rubocop (0.49.1)
Creating our Gem
For example, we'll call this Gem cookbook_helper, following the naming scheme as defined in RubyGems' docs.
We can create our Gem boilerplate by running the handy chef exec bundle gem cookbook_helper:
$ chef exec bundle gem cookbook_helper
Creating gem 'cookbook_helper'...
create cookbook_helper/Gemfile
create cookbook_helper/lib/cookbook_helper.rb
create cookbook_helper/lib/cookbook_helper/version.rb
create cookbook_helper/cookbook_helper.gemspec
create cookbook_helper/Rakefile
create cookbook_helper/README.md
create cookbook_helper/bin/console
create cookbook_helper/bin/setup
create cookbook_helper/.gitignore
create cookbook_helper/.travis.yml
create cookbook_helper/.rspec
create cookbook_helper/spec/spec_helper.rb
create cookbook_helper/spec/cookbook_helper_spec.rb
Initializing git repo in /path/to/cookbook_helper
Let's start by filling in some of the information in the cookbook_helper.gemspec:
Gem::Specification.new do |spec|
spec.name = "cookbook_helper"
spec.version = CookbookHelper::VERSION
spec.authors = ["Jamie Tanna"]
spec.email = ["..."]
spec.summary = %q{Opinionated Rake tasks for aiding with building a cookbook in line with ChefDK 2.4.17}
...
Code Style Tasks
The quickest tasks we can bring in are the code style tasks, Rubocop and FoodCritic.
Rubocop
Starting with Rubocop, we need to first add the dependency to our cookbook_helper.gemspec:
Gem::Specification.new do |spec|
# ...
spec.add_runtime_dependency 'rubocop', '= 0.49.1'
We then follow the example of other Gems, and we create the file lib/cookbook_helper/rake_task.rb:
require 'rubocop/rake_task'
RuboCop::RakeTask.new(:rubocop)
FoodCritic
Again, we simply need to version pin FoodCritic:
Gem::Specification.new do |spec|
# ...
spec.add_runtime_dependency 'foodcritic', '= 12.2.1'
And again add the Rake task to lib/cookbook_helper/rake_task.rb:
require 'foodcritic'
# ...
FoodCritic::Rake::LintTask.new(:foodcritic)
However, this doesn't quite mirror the functionality of the FoodCritic CLI. For instance, when the CLI finds any FoodCritic violations, it returns an error code, whereas the Rake task only fails on correctness tags, whereas the CLI returns an error code if any tags fail.
To preserve the functionality of the foodcritic, we can add the following config to the Rake task:
require 'foodcritic'
# ...
-FoodCritic::Rake::LintTask.new(:foodcritic)
+FoodCritic::Rake::LintTask.new(:foodcritic) do |t|
+ # As with the FoodCritic CLI, fail if there are any tags that fail
+ t.options = {
+ fail_tags: ['any']
+ }
+end
Unit Testing Tasks
Cookbook unit testing is traditionally done using ChefSpec, which wraps some Chef-iness around RSpec:
Gem::Specification.new do |spec|
# ...
spec.add_runtime_dependency 'chefspec', '= 7.1.1'
Adding the Rake task to lib/cookbook_helper_gem/rake_task.rb:
require 'rspec/core/rake_task'
# ...
RSpec::Core::RakeTask.new(:spec)
If you're using Berkshelf for dependency management, you will also need to pull it in, so ChefSpec can download dependent cookbooks:
Gem::Specification.new do |spec|
# ...
spec.add_runtime_dependency 'berkshelf', '= 6.3.1'
Documentation Rake Tasks
As I've mentioned before, I use knife-cookbook-doc as a means to pull inline documentation into the README.md.
Generating Documentation
For pulling in knife-cookbook-doc, we also need to specify the version of chef, which matches the version ChefDK 2.4.17 provides:
Gem::Specification.new do |spec|
# ...
spec.add_runtime_dependency 'chef', '= 13.6.4'
spec.add_runtime_dependency 'knife-cookbook-doc', '~> 0.25'
Note that we need a minimum of 0.25.0 to support generating documentation for Chef 13, but this could be anything, really. In this case, we'll pull in anything matching Semantic Versioning minor bumps.
Testing Documentation
It may be useful to be able to generate documentation, but it'd also be pretty useful to be able to confirm whether the README.md has been updated.
require 'knife_cookbook_doc/rake_task'
# ...
KnifeCookbookDoc::RakeTask.new(:readme)
KnifeCookbookDoc::RakeTask.new(:readme_test) do |t|
t.options[:output_file] = 'tmp/README.md'
end
task :readme_test_dir do
FileUtils.mkdir_p 'tmp'
end
task doc_test: [:readme_test_dir, :readme_test] do
unless FileUtils.identical?('README.md', 'tmp/README.md')
$stderr.puts "Generated file is not identical to the README.md in the repo. Please update it:"
# the command will fail with an error code, therefore failing the Rake task
sh 'diff -aur --color README.md tmp/README.md'
end
end
We utilise Rake's dependencies functionality to ensure that doc_test isn't run until we have readme_test_dir and readme_test, in that order. Note that we use FileUtils.mkdir_p to silence errors if the tmp directory already exists.
When the files aren't identical, we output a friendly error message to inform the user that they need to update the documentation, as well as outputting the raw diff between the files. The diff itself is to make it easier to debug any cases where they're different.
Cleaning up our Rakefile
Wrapping Related Blocks in namespaces
Rake's namespaces functionality allows us to place common code into blocks, such as:
namespace :style do
RuboCop::RakeTask.new(:rubocop)
FoodCritic::Rake::LintTask.new(:foodcritic) do |t|
# As with the FoodCritic CLI, fail if there are any tags that fail
t.options = {
fail_tags: ['any']
}
end
end
namespace :unit do
RSpec::Core::RakeTask.new(:spec)
end
namespace :doc do
KnifeCookbookDoc::RakeTask.new(:readme)
KnifeCookbookDoc::RakeTask.new(:readme_test) do |t|
t.options[:output_file] = 'tmp/README.md'
end
task :test_dir do
FileUtils.mkdir_p 'tmp'
end
task test: [:test_dir, :readme_test] do
unless FileUtils.identical?('README.md', 'tmp/README.md')
# the command will fail with an error code, therefore failing the Rake task
$stderr.puts "Generated file is not identical to the README.md in the repo. Please update it:"
sh 'diff -aur --color README.md tmp/README.md'
end
end
end
This makes it much clearer when reading through the source, as well as being able to see what each task corresponds with. For instance, before:
$ chef exec rake -T
rake foodcritic # Lint Chef cookbooks
rake readme # Generate cookbook documentation
rake readme_test # Generate cookbook documentation
rake rubocop # Run RuboCop
rake rubocop:auto_correct # Auto-correct RuboCop offenses
rake spec # Run RSpec code examples
And after:
$ chef exec rake -T
rake doc:readme # Generate cookbook documentation
rake doc:readme_test # Generate cookbook documentation
rake style:foodcritic # Lint Chef cookbooks
rake style:rubocop # Run RuboCop
rake style:rubocop:auto_correct # Auto-correct RuboCop offenses
rake unit:spec # Run RSpec code examples
However, note that we're missing our doc:test_dir and doc:test tasks!
This is because they don't have corresponding documentation lines, i.e.
namespace :doc do
# ...
+ desc 'Verify that documentation has been generated from latest source code'
task test: [:test_dir, :readme_test] do
unless FileUtils.identical?('README.md', 'tmp/README.md')
$stderr.puts "Generated file is not identical to the README.md in the repo. Please update it:"
sh 'diff -au --color README.md tmp/README.md'
end
end
end
This then makes the doc:test task appear:
$ chef exec rake -T
rake doc:readme # Generate cookbook documentation
rake doc:readme_test # Generate cookbook documentation
rake doc:test # Verify that documentation has been generated from latest source code
rake style:foodcritic # Lint Chef cookbooks
rake style:rubocop # Run RuboCop
rake style:rubocop:auto_correct # Auto-correct RuboCop offenses
rake unit:spec # Run RSpec code examples
Creating Helper tasks
Now we're using namespaces, it's more complicated to call certain tasks, such as rake style:rubocop && rake style:foodcritic to run our style checks. By combining these in a top-level task, we can prevent this:
# ...
task style: ['style:rubocop', 'style:foodcritic']
task unit: ['unit:spec']
task doc: ['doc:readme']
We can also add in a default task to allow us to simply run rake, and call the helper tasks:
task default: ['style', 'doc:test', 'unit']
Integrating into Cookbooks
Now we've set up our Rake tasks, we need to actually integrate into another cookbook. However, we don't need to push it up to RubyGems quite yet, as we can test it all locally.
First, we need a cookbook. For convenience, we'll create a fresh cookbook:
$ chef generate cookbook test-cookbook
We create a Rakefile to include our Gem's Rake tasks:
require 'cookbook_helper/rake_task'
As well as making sure that the Gem is included in the cookbook via the Gemfile, taking care to reference the path on disk to the cookbook_helper folder:
source 'https://rubygems.org'
gem 'cookbook_helper', path: '/path/to/cookbook_helper'
To verify this has worked, we can run the following:
$ chef exec bundle exec rake -T
rake doc:readme # Generate cookbook documentation
rake doc:readme_test # Generate cookbook documentation
rake style:foodcritic # Lint Chef cookbooks
rake style:rubocop # Run RuboCop
rake style:rubocop:auto_correct # Auto-correct RuboCop offenses
rake unit:spec # Run RSpec code examples
At this point, we've now got a Gem ready to release to RubyGems which pins exactly to a given version of the ChefDK.
There will be a follow-up post to describe how I build out my GitLab CI configuration for cookbooks.
