Piecing things together (Posts about infrastructure)

Mesos container log forwarding with Filebeat

Joel Wilsson — Sun, 02 Oct 2016 08:17:00 GMT

The next level

In the previous post we managed to forward container logs to journald, but we could have done the same thing and much more if we could have forwarded the logs to systemd-cat instead. We wouldn't have had to write any code at all if that was possible.

So we're going to make that possible by backporting an upcoming "external" container logger, being developed for the next release of Mesos, as a module compiled for the currently stable release. That's version 1.0.1 at the time of writing.

Then we'll use this module to set up log forwarding from Mesos containers to Graylog using Filebeat 1.3 and include additional fields from the Mesos executor's environment.

Mesos and external container loggers

Apache Mesos has had support for a LogrotateContainerLogger since version 0.27, but if you look at what that container logger you'll find that all it does is receive input from standard input and standard error, counts the number of bytes received, and writes to the stdout and stdout files in the sandbox directory.

It calls logrotate once a configurable number of bytes has been received. This logic is implemented by mesos-logrotate-logger which is a stand-alone program. Why do we need a separate module to feed logs to this specific external program?

MESOS-6003 was created to add a logging module for logging to any external program. Thanks to the hard work of Will Rouesnel, shepherded by Joseph Wu, this is almost completed.

What if we can't wait for the next release of Mesos? Maybe we're stuck on 1.0.1 for a while for reasons beyond our control, but we'd love to use Filebeat for log forwarding. Unfortunately we can't, because the recommended setup doesn't work.

Even if it did we'd still be missing out on important context, like what the Marathon app ID and the Mesos task ID are. Maybe there's a specific version of the Marathon app which is having issues. It would be nice if we could find all log messages from that application and version with a simple search.

We'll make that possible by backporting the external container logging module to Mesos 1.0.1 and setting it up to use a small wrapper around Filebeat to create configuration which includes the task's environment variables and forwards the log messages to Graylog. It will also work if you are using Logstash.

A Mesos logging module for journald

Joel Wilsson — Thu, 29 Sep 2016 19:23:00 GMT

A short introduction to Mesos

We're going to send logs from Mesos containers to systemd-journald by writing a container logging module in C++. You can skip this section if you already know what Mesos is.

Apache Mesos describes itself as a distributed systems kernel, which is appropriate in a lot of ways. For some it may sound a bit intimidating and complicated, and I think that's a bit unfortunate, because it can be explained very simply without losing too much.

Mesos offers resources in a cluster. Resources types include CPU cores, memory, disk space, network ports, GPUs, etc. Let's say I'm a developer and I want to run my application on the cluster. I get an offer from Mesos of 4 CPU cores, 8 GBs of memory, 40 GBs of disk space and a range of ports from 10000-20000.

I don't need all of it, and reply that accept 1 CPU core, 2 GBs of memory, 200 MBs of disk space and one port, port 10000, and I want to fetch https://wjoel.com/foo-standalone.jar (a self-contained "fat JAR" with no external dependencies) and run it with the command java -jar foo-standalone.jar. Mesos will create a container using cgroups (if running on Linux) to enforce limits based on the resource constraints I accepted. The container is also known as a sandbox, and we get to play in it as long as we stay within the resource limits.

Developers typically don't want to bother with resource offers from Mesos. Programs that respond to resource offers from Mesos are called frameworks. One such framework is Mesosphere's Marathon, and its application specifications are essentially lists of resources and the command to run. Marathon can also ensure that applications are restarted if they die for any reason, do rolling updates, and many other useful things that developers like to have.

You may have noticed that I told Mesos to run my JAR file using Java, but didn't specify that I wanted Java to be downloaded. Hence, my application will only run if Mesos decides to run it somewhere where Java is already installed.

I could create a Docker image which includes foo-standalone.jar, Java, and any other dependencies I might need. Mesos can run Docker containers as well, either on its own or using by Docker for isolation. Alternatively, I could have included an additional URL in my reply, containing the location of an archive with a full Java installation and used that instead, all from within the container.

Creating Jenkins pipelines with Ansible, part 2

Joel Wilsson — Wed, 10 Aug 2016 19:52:00 GMT

The job-dsl and Pipeline plugins

This is a continuation of the previous post. For each project there are only two things we want to do.

Check out the source code.
Run the pipeline in the Jenkinsfile at the root of the repository.

This can be accomplished with the job-dsl plugin. It takes a definition of Jenkins jobs we want, and creates or updates them, as necessary. Somewhat confusingly, job-dsl itself needs to be run from a Jenkins job, known as a seed job.

The only reason for this job to exist is so that job-dsl can run, and every other job should be created with job-dsl. Yes, it is a bit mind bending, but if you don't think of it as a job but as a script which you run from Jenkins it gets easier.

In our case, every other job will look the same. They'll all be Pipeline jobs which run a Jenkinsfile after checking out a git repository from the same git host.

{% for repository in jenkins_git_repositories %}
pipelineJob('{{ repository }}') {
  definition {
    cpsScm {
      scm {
        git {
          remote {
            url('{{ jenkins_git_user }}@{{ jenkins_git_host }}:' +
                '{{ jenkins_git_path }}/{{ repository }}.git')
          }
          branch('master')
        }
      }
      scriptPath('Jenkinsfile')
    }
  }
}
{% endfor %}

Creating Jenkins pipelines with Ansible, part 1

Joel Wilsson — Sat, 06 Aug 2016 18:49:00 GMT

Introduction

Infrastructure as code, continuous integration (CI), continuous delivery (CD), and version control (also known as source control management, or SCM) are good things. In this post and the next we'll automate the deployment of a system to do CI/CD in what I consider a proper way, where the definition of your build and deployment pipelines live alongside your code in git repositories on some server you have SSH access to.

We'll use Jenkins for CI/CD and Ansible to install it. Jenkins has been around for more than a decade, created before anyone had ever uttered the word "devops". Since it's old, Jenkins has a lot of features, most of them provided as plugins. The core itself isn't bad, and is stable and relatively free from bugs because it has been in use by a lot of people for such a long time. All those years have created a graveyard of plugins, though, yet some plugins are essential if you want to use Jenkins "properly". Want a timestamp for your builds? There's a plugin for that. Actually, there's more than one plugin for that.

Jenkins comes from a time when people thought it was great to have easy to use web interfaces to configure their build systems. The web interface gets in the way of infrastructure as code, where configuration is version controlled and changed by editing text files, not by clicking buttons.

Pipeline is a "new" feature of Jenkins 2.0, but it is based on a plugin which used to be known as Workflow. Using Pipeline we can describe how to test, build and deploy our project using text. That description is typically saved in a file named Jenkinsfile. We'll also use the job-dsl plugin to create the Pipeline jobs, but everything else will be done with shell scripts and kept as simple as possible.