Prev – The “eureka” moment – discovery of RAM Disk Drives
Now there was no stopping, I did not need an SSD, all I had to do was install the RAM drive on our actual Jenkins windows machine, point SAS temporary data set creation to tmpfs and get rid of all the database IO issues and bring the build time on the Jenkins machine from 1+ hour to 25/30 minutes. Ajay allotted more RAM to both the Jenkins and the SAS Linux VMs. I did the needful and triggered the build process and to my astonishment, the build timings did not reduce much on our actual Jenkins and SAS Linux machines. I thought I did not do something right. I reconfirmed the changes on both the machines, re triggered the build and the result was same. I was completely shocked and disappointed. Though it was supposed to work, my hypothesis had failed and I was wondering why. The only reason I could imagine for the hypothesis to not work was that probably the SAS data set operations and the MySQL database operations that we were performing via tests were not as IO intensive as I had assumed earlier and there was something else in play. This was my new hypothesis.
Next – Test on smaller data set
Prev – The alternative for SSD, in-memory/in-process db
I was feeling very impatient and was wondering what to do. Was there an alternative to SSD? While thinking over it and with the word “memsql”, I had an “eureka” moment. I was wondering what if I was able to create a drive in RAM and have the entire MySQL db on it. Now, though it won’t be in-process, it would be completely in-memory and without any code changes it would be faster than any hard disk, hybrid disk, SSD. That was just a hypothesis and to my surprise I found that there are software’s that would convert extra RAM into RAM drive. I also came to know that Linux (any Unix flavor) already has this built-in with its tmpfs. I also came across these two blogs “RAMDisks Roundup and testing”, “RAM Disk Software Benchmarked”. Based on available benchmarks I shortlisted “SoftPerfect RAM Disk”. I was super excited and I wanted to immediately try it on the test VM servers and the build time did came down further!!!. The hypothesis had worked, the build time on the test VM Server came down to 25 minutes.
Next – The excitement and the disappointment
In my organization we are using Jenkins as our CI tool. The core build is followed by multiple jobs consisting of unit tests, integration tests, SAS integration tests, PMD, all running in parallel, running over 3000+ tests which took the entire build pipeline to run over 1 hour 30 minutes to produce the build artifacts. The amount of time taken was too high and it was very frustrating specially when the tests failed, as multiple developers would check in files while the earlier build was in progress, the next job would start and by the time issue was identified and fixed, it would take more than 4/5 hours to get a stable build. QA would not get build artifacts on time. Valuable development time was getting lost. There were frustrations all around.
This persistent issue pushed me on a quest to reduce the CI build time.
The problem with duplication
The discovery of Ant JUnit task options
The assumptions around IO and SSD
The alternative for SSD – in-memory/in-process db
The “eureka” moment – discovery of RAM Disk Drives
The excitement and the disappointment
Test on smaller data set
CPU profiling for rescue
Today Ajay moved our Jenkins VM to a box having hybrid disk and now the build pipeline time has reduced from 25 minutes to 15 minutes and all the test jobs run in less than 10 minutes!!! And I am feeling very happy and satisfied on my quest of reducing the build time. This journey took more than two months, during which I have learnt a lot.
On a quest of reducing Jenkins build time – Part 2