How to take over the computer of a Jenkins user

08/14/2014

I recently began using Jenkins and found quite a bit of security indifference. This is unfortunate because Jenkins is the world’s leading continuous integration server used for testing, building, and deploying code. According to RebelLabs, Jenkins has 70% market share, with the next closest competitor having only 9%. I’ve raised these issues with the Jenkins team and have received only dismissive responses thus far. The response I’ve received and the fact that Jenkins has over 50 open bugs filed against it which are categorized as critical security issues and leaves me with little confidence that the team will move on these issues unless attention is drawn to them, which is why I’ve written this post.

Unsecure installation

Let’s start at the beginning and walk through the install instructions. The very first step on Ubuntu is:

wget -q -O - http://pkg.jenkins-ci.org/debian/jenkins-ci.org.key | sudo apt-key add -

Here are the first two steps on Redhat:

sudo wget -O /etc/yum.repos.d/jenkins.repo http://pkg.jenkins-ci.org/redhat-stable/jenkins.repo
sudo rpm --import http://pkg.jenkins-ci.org/redhat-stable/jenkins-ci.org.key

If you haven’t noticed anything wrong yet, you’re not alone. I didn’t either the first time I followed these instructions. The issue here is the http://. When you download software from a Linux repository, the system verifies downloaded packages against a gpg signature. Debian has been using strong crypto to validate downloaded packages since 2005, so this is a long standing best practice. However, if you download this signature over an insecure channel, then there is little point because anyone who could deliver a malicious package could also deliver a malicious signature. For this reason, you should only use https with “apt-key add” or else you are rendering void any security it provides. Indeed if you Google “apt-key add” the very first result you get is a StackOverflow post which says “adding keys you fetch over non-HTTPS breaks any security that signing packages added. Wherever possible, you should download keys over a secure channel (https://)”. If only Jenkins would properly configure their SSL certificate for downloading this file and update their docs to suggest https!

Unsecure updates

Jenkins by default loads the URLs to use for updating plugins from http://updates.jenkins-ci.org/update-center.json. This is a problem because Jenkins will download and install whatever package URLs are listed in this file, so if an attacker can modify this file they can install whatever malicious plugins they want. I attempted to remedy this with a one-character pull request to change http to https which was rejected as being too load intensive upon Jenkins servers. I was told on the bug that I filed for the issue that there’s a signature embedded within the file which makes it secure. The problem here is that you need a key which you received securely to check that signature. Because the key is delivered over HTTP as already discussed, much of its value is lost.

Unsecure plugins

A response I’ve gotten to the preceding issue is “You realize that anyone with a Jenkins-ci.org account can release updates to any plugin, right?” So why bother delivering widely used plugins securely when they could be malicious before they ever leave the Jenkins servers? I could update all the most popular Jenkins plugins with malicious code and no doubt thousands of people would update their plugins and find themselves running malicious code. The plugins are all open source, but I have no idea if I’m running the code that I see open sourced. An attacker could download the code for a plugin, modify it in an evil manner, and release an update to that plugin and there’s no way to know whether the code downloaded matches what is in the open source repository.

The irony here is almost killing me. Using Jenkins to build the plugins instead of letting “anyone with a Jenkins-ci.org account” build them would be a great solution to this problem. I was told that fixing this problem would violate “Jenkins project core principles, so you should probably build a better case than ‘this is wrong’ before you bring it up on the dev list.” Without further explanation I’m left wondering why closing security holes would violate Jenkins project core principles. Looking at the core principles only seem to reinforce the idea that these problems should be fixed. It would lower the barrier to entry by making it such that plugin developers don’t need to figure out how to publish them since a continuous integration server could do it. It seems meritocratic to fix security issues raised by the community. It would increase transparency to know that you’re running the code you see available on GitHub and not some attacker’s code. It would not affect compatibility or code licensing. It certainly would be a more automated solution (someone get Alanis Morissette on the phone before I die).

Unsecure for contributors

You can’t even work on Jenkins without facing security problems. If you try to write a plugin for Jenkins, for example, the docs suggest you add the following to your Maven settings:

      <repositories>
        <repository>
          <id>repo.jenkins-ci.org</id>
          <url>http://repo.jenkins-ci.org/public/</url>
        </repository>
      </repositories>
      <pluginRepositories>
        <pluginRepository>
          <id>repo.jenkins-ci.org</id>
          <url>http://repo.jenkins-ci.org/public/</url>
        </pluginRepository>
      </pluginRepositories>

Again, downloading software over http is not secure. I was told this is a “cosmetic issue” when I filed a bug though I’m hoping the engineer that the bug is assigned to will see that telling users to connect to http is a bit more than that. To help demonstrate this point, I linked to an article which shows how to exploit exactly this problem in my bug report. As a result of that article, Sonatype (who host the most popular Maven repository) is turning on SSL for all users. It is not yet apparent that this will sway anyone working on Jenkins.

Consequences

So what can you do by getting someone to install a malicious version of a Jenkins server or plugin and how hard is it? Well, there’s already a proof-of-concept for launching a Man-in-the-middle attack against a Maven repository http download and it’s pretty basic code, so I think it’s fair to say that it can be done. If you go to a Jenkins Meetup there’s a chance you’ll be able to snag someone downloading some Jenkins-related software over an unsecured wi-fi connection and be able to infect them. The types of folks who would install Jenkins on their laptops are also somewhat likely to have access to production systems at their companies. And because Jenkins is used to build software that means a malicious version could potentially inject further maliciousness into the software that it’s building or leak the source code of that software to an attacker.

If you care about building secure software, I hope that you’ll ask the Jenkins team to fix these issues and make sure other Jenkins users are familiar with these holes until then. You can also check out https://www.connectifier.com/careers.