Cloud-hosted continuous integration solutions 2015

For a new project at work, we needed to setup a continuous integration routine. As we don’t really have the capacity to host it on our own, I decided to make a brief market analysis of the available solutions in late 2015.

While my research was done with a certain project configuration in mind (a web platform running Node.js with Angular), I’m sure it can be useful for other types of projects as well.

Travis CI

Extremely popular as a lot of open-source projects hosted on Github use their free plan. The documentation is comprehensive and clean [1] plus there are tons of related posts on stackoverflow. I also found a dedicated section on browser testing. They even offer to run your script on Mac hardware [2] allowing to test on the most recent versions of Safari. The downside however is the price. Their plans start from 129$ excl. VAT per month for 2 concurrent build jobs. [3] Also, your sources must be hosted on GitHub as there is no support for any other type of repository.

Codeship

Codeship is definitely worth a look for small companies as they offer a free plan with 100 builds per month, as well as an unlimited personal plan for 49$ per month. However, this does not allow to reflect organizational hierarchies in terms of account management. The organizational plan is 99$ per month. [4] Codeship supports both Github and Bitbucket, however there is no way to use any other type of repository. [5] They also do not offer builds on Mac hardware. I skimmed over their documentation and it looked clearly arranged and pretty extensive. [6]

Circle CI

Circle CI offers the most generous free plan of all solutions. You get unlimited builds for one repository for free. Any additional container is 50$ [7]. Their documentation looks good with lots of real-world examples and technology-specific tutorials. I discovered a section on iOS builds on Mac hardware. [8] While this feature seems to be experimental at the moment, it might indicate that Circle CI is planning to offer Mac hardware for regular build jobs in the near future. As for SCM, only Github is supported. [9]

Jenkins CI on AWS

This section is hard to evaluate since I don’t have any experience how much resources Jenkins will actually use. However it is possible to answer a few obvious questions. You will have to spend a lot of time setting up and configuring Jenkins, even more so if you plan to use its EC2 features allowing it to scale by launching additional instances on demand. There is clearly no way to use Mac hardware and although there is no actual limit on build jobs, make sure you have set AWS billing alerts.

Conclusion

Since pricing policy differs from company to company, I decided to compare the first plans that offer unlimited builds on 2 concurrent jobs.

Price and feature comparison
Travis CI Codeship Circle CI Jenkins CI on EC2
Github yes yes yes yes
Bitbucket no yes no yes
Private SCM no no no yes
Mac Hardware yes no iOS no
Concurrent Jobs 2 1(2)* 2 ultd.
Ultd. Plans from 129$ 49$(99$**) 50$ ?
Setup easy easy easy hard
SSH Debugging no*** yes yes yes

* Only one concurrent job but 2 parallel test pipelines
** Organization plans including account management from 99$
*** Various blog posts indicate that they will launch a VM for you to debug your scripts

References

[1] http://docs.travis-ci.com
[2] http://docs.travis-ci.com/user/workers/os-x-infrastructure
[3] https://travis-ci.com/plans
[4] https://codeship.com/pricing
[5] https://codeship.com/documentation/faq/other-scm
[6] https://codeship.com/documentation
[7] https://codeship.com/pricing
[8] https://circleci.com/docs/ios
[9] https://circleci.com/docs/faq#do-you-support-bitbucket-or-gitlab-