For a new project at work, we needed to setup a continuous integration routine. As we don’t really have the capacity to host it on our own, I decided to make a brief market analysis of the available solutions in late 2015.
While my research was done with a certain project configuration in mind (a web platform running Node.js with Angular), I’m sure it can be useful for other types of projects as well.
Travis CI
Extremely popular as a lot of open-source projects hosted on Github use their free plan. The documentation is comprehensive and clean [1] plus there are tons of related posts on stackoverflow. I also found a dedicated section on browser testing. They even offer to run your script on Mac hardware [2] allowing to test on the most recent versions of Safari. The downside however is the price. Their plans start from 129$ excl. VAT per month for 2 concurrent build jobs. [3] Also, your sources must be hosted on GitHub as there is no support for any other type of repository.
Codeship
Codeship is definitely worth a look for small companies as they offer a free plan with 100 builds per month, as well as an unlimited personal plan for 49$ per month. However, this does not allow to reflect organizational hierarchies in terms of account management. The organizational plan is 99$ per month. [4] Codeship supports both Github and Bitbucket, however there is no way to use any other type of repository. [5] They also do not offer builds on Mac hardware. I skimmed over their documentation and it looked clearly arranged and pretty extensive. [6]
Circle CI
Circle CI offers the most generous free plan of all solutions. You get unlimited builds for one repository for free. Any additional container is 50$ [7]. Their documentation looks good with lots of real-world examples and technology-specific tutorials. I discovered a section on iOS builds on Mac hardware. [8] While this feature seems to be experimental at the moment, it might indicate that Circle CI is planning to offer Mac hardware for regular build jobs in the near future. As for SCM, only Github is supported. [9]
Jenkins CI on AWS
This section is hard to evaluate since I don’t have any experience how much resources Jenkins will actually use. However it is possible to answer a few obvious questions. You will have to spend a lot of time setting up and configuring Jenkins, even more so if you plan to use its EC2 features allowing it to scale by launching additional instances on demand. There is clearly no way to use Mac hardware and although there is no actual limit on build jobs, make sure you have set AWS billing alerts.
Conclusion
Since pricing policy differs from company to company, I decided to compare the first plans that offer unlimited builds on 2 concurrent jobs.
| Travis CI | Codeship | Circle CI | Jenkins CI on EC2 | |
|---|---|---|---|---|
| Github | yes | yes | yes | yes |
| Bitbucket | no | yes | no | yes |
| Private SCM | no | no | no | yes |
| Mac Hardware | yes | no | iOS | no |
| Concurrent Jobs | 2 | 1(2)* | 2 | ultd. |
| Ultd. Plans from | 129$ | 49$(99$**) | 50$ | ? |
| Setup | easy | easy | easy | hard |
| SSH Debugging | no*** | yes | yes | yes |
* Only one concurrent job but 2 parallel test pipelines
** Organization plans including account management from 99$
*** Various blog posts indicate that they will launch a VM for you to debug your scripts
References
[1] http://docs.travis-ci.com
[2] http://docs.travis-ci.com/user/workers/os-x-infrastructure
[3] https://travis-ci.com/plans
[4] https://codeship.com/pricing
[5] https://codeship.com/documentation/faq/other-scm
[6] https://codeship.com/documentation
[7] https://codeship.com/pricing
[8] https://circleci.com/docs/ios
[9] https://circleci.com/docs/faq#do-you-support-bitbucket-or-gitlab-