Review: Chef 12 fires up devops

Two of the most important tasks in a datacenter are server provisioning and configuration management. At one time, administrators spent a significant amount of time physically deploying servers and network infrastructure, followed by even more minutes and hours manually configuring hundreds or even thousands of nodes. Then they spent an additional chunk of the day troubleshooting and fixing the errors they introduced by configuring all of these systems with their fat fingers.

Today, thanks to automation software such as Chef, administrators are able to provision servers, deploy software, and set a nearly endless number of configuration items in a blink, without lifting a finger. And they do so in a way that is more scalable, more repeatable, and more immune to human error.

Although widely considered a configuration management product, Chef seems to think bigger than comparable tools on the market. Chef was designed with a devops culture in mind, and in addition to automating the provisioning of server infrastructure, Chef can automate the provisioning of runtime environments, applications, and containers. This is a comprehensive tool that provides not only configuration management but end-to-end IT automation.

At its most basic (the free open source version), Chef combines a development kit for creating “cookbooks” (think “recipes” for system configurations), a client that runs on managed nodes, and the Chef server, a central repository for cookbooks and the means of applying the recipes in cookbooks to nodes. Chef's open source community marketplace provides a vast repository of cookbooks that can speed up delivery time. Advanced features -- including high availability, a web-based management console, and reporting and analytics -- are free for up to 25 nodes.

After that you have two supported commercial options to choose from: Hosted Chef and Chef Automate, which cost $72 and $137 per node, respectively. Chef Automate comes complete with team workflow capabilities that extend the automation across the entire application lifecycle, from development and testing through staging and production.

A Chef overview

Chef comes with an array of features that provide you with fine-grained control over your infrastructure. Let’s take a closer look at each component.

Chef DK. Chef DK is a downloadable software development toolkit that contains all of the best-of-breed tools developed by the Chef community. It enables you to quickly set up and upgrade the Chef Client on workstations and get control of the cookbook dependencies. Chef DK includes built-in testing tools like RuboCop and Foodcritic, the unit testing framework known as ChefSpec, and Test Kitchen, an integration tool for testing coded infrastructure. Chef DK offers a streamlined workflow for new users along with plenty of flexibility for advanced users.

Test Kitchen. Test Kitchen is a powerful tool that lets you run configured code on various platforms in isolation. The Test Kitchen is not only an isolated environment, but a pristine one -- it ensures there are no traces of a prior state. A .kitchen.yml file, which contains the static declarative configuration, is typically stored in the root of the Chef repository. With a driver plugin architecture, Test Kitchen allows you to test cookbook code across various cloud providers and virtualization technologies, including Microsoft Azure, Amazon EC2, Apache CloudStack, Blue Box, DigitalOcean, Vagrant, Rackspace, Docker, and Linux Containers. Test Kitchen supports testing frameworks like Inspec, and it’s compatible with dependency management tools such as Berkshelf. To date, many Chef community cookbooks have already included support for Test Kitchen.

Chef Supermarket. Chef's open source community site lets users browse, download, and share cookbooks. Initially, Supermarket existed only as a public community, allowing users to upload and download cookbooks from anywhere. Today, Supermarket is offered in two forms: public and private, with the private Supermarket deployed on-premises and accessed via users’ Chef Server credentials. In both the public and private Supermarkets, you can quickly search the cookbook repository via web GUI or command line. In addition to Linux, Chef Supermarket offers cookbooks that provide plenty of Windows-specific resources.

Chef Automation. Chef Automation is a premium feature of Chef that facilitates continuous delivery of infrastructure and runtime environments, including applications and containers, and provides visibility into those workflows. To meet changing business needs, big companies have to make an ongoing series of changes to networks, applications, platforms, and software. Chef Automation provides this flexibility with a unified process that automates changes to applications, infrastructure, and runtime environments at the same time.

The shared pipeline provided by Chef Automation allows you to safely take code from the developers through the build, test, and production stages, while Chef's collaborative platform brings visibility to each step of the process. IT architects, engineers, developers, and security and compliance staff receive prompt alerts on proposed changes. Everyone can see the flow of changes through the pipeline.

Chef Analytics. Chef Analytics, another premium feature, provides precise and real-time visibility into everything that’s happening in the managed estate. From the changes made to who made them, when, and how, Chef Analytics enables you to carefully monitor the modifications happening in the infrastructure. For example, compliance against internal controls can be easily verified using Analytics, and changes immediately communicated to users in real time through email and webhooks. Also, you can integrate Chef Analytics with HipChat or Slack to send real-time notifications to corresponding chat rooms.

Version control. IT environments are quick to embrace changing technologies to meet growing business demands. For instance, version control is an essential requirement when multiple people work together on a single software development project. This is true even when the software being developed is infrastructure code. Chef recognizes this, and versioning is a core principle that’s the basis of Chef today.

Chef uses Git for version control of cookbooks. That means each cookbook is kept cohesive because all changes are merged into a single updated version, and you can view the older versions or a graphical display of the entire project at any time. This minimizes data storage and reduces the complexity of manually storing each version with a different name. Also, you can easily see what happened with the cookbook, retrieve old cookbooks, and automatically back up cookbooks. A new developer can simply download the current version of the cookbook to the local environment and keep track of it.

Windows support. With the increasing use of Chef in enterprise environments, adding enhanced support for Windows operating systems has become a priority. Chef has answered the call, allowing you to automate several key tasks that would be cumbersome and time-consuming without this enhanced support. Let's take a quick look:

• Installation of Windows features: The installation of Windows features can be achieved via Chef, allowing you to enable features such as Internet Information Services (IIS) or the .Net Framework for application servers. Performing this task with the UI is complicated and time-consuming, so having this built into the platform is a big plus.
• Windows Installers: The windows_package resource is a useful feature that's built into Chef. It enables you to manage and automate the installation of most installer types, such as InstallShield and the Windows Installer Package (MSI). If the installer needs to be upgraded, this will happen automatically.
• IIS Cookbook: For those deploying an application into IIS, a popular choice on the Chef Supermarket is the IIS cookbook. This allows Chef to create and manage IIS sites, apps, application pools, and related configurations.
• Windows cookbook: The Windows cookbook provides valuable resources to the Windows ecosystem such as the ability to create scheduled tasks, manage digital certificates, and even install fonts. It also offers tools to manage the system path.

Evaluating Chef

Capability. Chef uses the server-client model, which means the Chef server works with the Chef client to apply configurations to managed nodes. To streamline this process, the Chef server provides the latest set of recipes and instructions for each node based on its role. For instance, a load balancer receives one set of instructions while the web server receives another set of instructions. Each time the Chef client runs, it queries the server for the latest set of recipes, and the Chef client applies these directions in the same order every time so that consistent and repeatable results are seen.

Chef uses the pull method whereby nodes periodically contact the server to receive their instructions. (It should be noted that push functionality is not well supported.) By default, the Chef client pulls the latest instructions from the Chef server every 30 minutes.

If there are any configuration drifts, the Chef client immediately changes the state of the node to the desired state, but these changes are made when configurations are found to diverge from the desired state. In most cases, nothing happens, which means you can run scripts anytime without any issues. When you instruct the client to install a particular service on a node, it only performs this task if the service is not already installed on the node or if it is outdated. By using a convergent mode of configuration, Chef moves the network as a whole to the desired.

In addition to ensuring desired state, Chef allows you to search and report on node configurations across the network. The Ohai tool that is part of the Chef client installation detects the attributes of the node at the start of each Chef client run. These characteristics include platform details, network usage, memory usage, kernel data, CPU data, host names, fully qualified domain names, and other configuration details.

Scalability. Chef is designed to provide high-level configuration management for large enterprises, and it can easily scale to thousands of nodes. With Chef, the entire environment is easily searchable, and these searches are available to the recipes -- making the system highly scalable by incorporating various load balancing features. As new web servers are added to the network, they are automatically noticed by the load balancers and requests are instantly routed to them. This automation is key to scaling well.

Designed as a distributed system, Chef allows you to run components on separate nodes to distribute the load. Big companies like Facebook have successfully managed thousands of servers, configurations, and administrative access policies in dynamic computing environments by deploying Chef.

Starting with Chef 11, the product was completely rewritten using the Erlang programming language. The core server API was also renamed Erchef. The CouchDB database was replaced with a PostgresSQL database, and this combination provides greater scalability. While memory consumption was reduced by a factor of 10, only a fraction of CPU resources is used. Due to these changes, a single Chef 11 or Chef 12 server can easily manage 10,000 clients -- a four-fold increase over what the Chef 10 server could handle. As the scale increases, the system expands into a tiered front-end and back-end architecture.

The scaling of the Chef infrastructure is measured in terms of chef-client runs per minute (CCRs/min). For instance, if 200 nodes are set to check in with the server every 30 minutes, the CCRs/min is 6.66. The Chef server can easily handle up to 333 CCRs/min without a tiered topology or high-availability setup. This is equivalent to 10,000 nodes. For more details about scaling, see Chef's documentation.

Operating system and application coverage. Chef supports multiple platforms and integrates well with cloud services such as Microsoft Azure. Initially, Chef was designed to support Linux. However, support for Windows has increased dramatically in recent years, and the latest releases support PowerShell DSC, IIS, and SQL Server. Support for 64-bit Windows has also been added recently, which means Chef can be used to manage the upcoming Windows Nano Server.