In my organization we are using Jenkins as our CI tool. The core build is followed by multiple jobs consisting of unit tests, integration tests, SAS integration tests, PMD, all running in parallel, running over 3000+ tests which took the entire build pipeline to run over 1 hour 30 minutes to produce the build artifacts. The amount of time taken was too high and it was very frustrating specially when the tests failed, as multiple developers would check in files while the earlier build was in progress, the next job would start and by the time issue was identified and fixed, it would take more than 4/5 hours to get a stable build. QA would not get build artifacts on time. Valuable development time was getting lost. There were frustrations all around.
This persistent issue pushed me on a quest to reduce the CI build time.
The discovery of Ant JUnit task options
The assumptions around IO and SSD
The alternative for SSD – in-memory/in-process db
The “eureka” moment – discovery of RAM Disk Drives
The excitement and the disappointment
Today Ajay moved our Jenkins VM to a box having hybrid disk and now the build pipeline time has reduced from 25 minutes to 15 minutes and all the test jobs run in less than 10 minutes!!! And I am feeling very happy and satisfied on my quest of reducing the build time. This journey took more than two months, during which I have learnt a lot.
On a quest of reducing Jenkins build time – Part 2
Resources
- Jenkins – http://jenkins-ci.org/
- CI – http://en.wikipedia.org/wiki/Continuous_integration
- Mklink – http://technet.microsoft.com/en-us/library/cc753194.aspx
- MySQL – http://www.mysql.com/
- http://ant.apache.org/manual/Tasks/junit.html
- http://java.dzone.com/articles/javalangoutofmemory-permgen
- SSD – http://en.wikipedia.org/wiki/Solid-state_drive
- Hybrid disk – http://en.wikipedia.org/wiki/Hybrid_drive
- HSQL – http://hsqldb.org/
- H2 – http://www.h2database.com/html/main.html
- Memsql – http://www.memsql.com/
- MySQL is bazillion times faster than MemSQL
- Tmpfs – http://en.wikipedia.org/wiki/Tmpfs
- http://blog.laptopmag.com/faster-than-an-ssd-how-to-turn-extra-memory-into-a-ram-disk
- RAM Disk Software Benchmarked
- http://jvmmonitor.org/
Ashish Parkhi 
Excellent post Ashish. Lots of gems.
Another idea that can surely bring down your test time is to run them in parallel. JUnit tests can be run in parallel using @RunWith(ConcurrentTestRunner). Also on many projects, we’ve separated slow running tests from fast running tests. And then we run these 2 categories of tests in parallel as well.
LikeLike
Thanks Naresh! I will try out @RunWith(ConcurrentTestRunner) option. We have separated our slow and fast running tests into parallel jobs. Since we are using too may db operations where most tests create their own data and truncate data prior to running tests, concurrent running tests on the same db tend to conflict with each other. Thus we create multiple empty dbs each for individual job running in parallel.
LikeLike
Well done Ashish! Thanks for sharing.
LikeLike
This is really a great post Ashish! Congratulations on getting build time under 20 min
LikeLike
Congrats on the improvement and sharing!!!
LikeLike
Good post. Ashish, Naresh already suggested parallelzation of test. Also you can experiment with Sequence of tests. Sometimes change in sequence also give good gians in terms of time.
LikeLike
Good Sharing Ashish!
LikeLike
Good summarization of the quest Ashish!
Apparantly I had once tried running @RunWith(ConcurrentTestRunner) but the results were not encouraging may be now it would make significant difference.
LikeLike
Good post Ashish !
LikeLike
QA benefited due to this change. Thanks Ashish!
LikeLike