On a quest of reducing Jenkins CI build time

In my organization we are using Jenkins as our CI tool. The core build is followed by multiple jobs consisting of unit tests, integration tests, SAS integration tests, PMD, all running in parallel, running over 3000+ tests which took the entire build pipeline to run over 1 hour 30 minutes to produce the build artifacts. The amount of time taken was too high and it was very frustrating specially when the tests failed, as multiple developers would check in files while the earlier build was in progress, the next job would start and by the time issue was identified and fixed, it would take more than 4/5 hours to get a stable build. QA would not get build artifacts on time. Valuable development time was getting lost. There were frustrations all around.

This persistent issue pushed me on a quest to reduce the CI build time.

The problem with duplication

The discovery of Ant JUnit task options

The assumptions around IO and SSD

The alternative for SSD – in-memory/in-process db

The “eureka” moment – discovery of RAM Disk Drives

The excitement and the disappointment

Test on smaller data set

CPU profiling for rescue

Today Ajay moved our Jenkins VM to a box having hybrid disk and now the build pipeline time has reduced from 25 minutes to 15 minutes and all the test jobs run in less than 10 minutes!!! And I am feeling very happy and satisfied on my quest of reducing the build time. This journey took more than two months, during which I have learnt a lot.

 On a quest of reducing Jenkins build time – Part 2

Resources

Advertisements

13 comments

  1. Excellent post Ashish. Lots of gems.
    Another idea that can surely bring down your test time is to run them in parallel. JUnit tests can be run in parallel using @RunWith(ConcurrentTestRunner). Also on many projects, we’ve separated slow running tests from fast running tests. And then we run these 2 categories of tests in parallel as well.

    Like

    1. Thanks Naresh! I will try out @RunWith(ConcurrentTestRunner) option. We have separated our slow and fast running tests into parallel jobs. Since we are using too may db operations where most tests create their own data and truncate data prior to running tests, concurrent running tests on the same db tend to conflict with each other. Thus we create multiple empty dbs each for individual job running in parallel.

      Like

  2. Good post. Ashish, Naresh already suggested parallelzation of test. Also you can experiment with Sequence of tests. Sometimes change in sequence also give good gians in terms of time.

    Like

  3. Good summarization of the quest Ashish!
    Apparantly I had once tried running @RunWith(ConcurrentTestRunner) but the results were not encouraging may be now it would make significant difference.

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s