Ember Parallel Testing & Code Coverage Reports at ActiveCampaign

ActiveCampaign has experienced an explosion of growth in the past few years, and with that growth came the pressures of scalability. Two years ago we decided to adopt the use of Ember.js, a frontend Javascript framework, to help speed up development and keep our code clean and more organized.

As we transitioned from legacy PHP pages to Ember, we wrote acceptance, integration, and unit tests to ensure the product was stable and working as intended. Unfortunately, as our ember-app repository grew to include thousands of tests, so did the length of time it took to run our test suite and deploy our code.

In this article I’ll outline the methods we took to help speed up our testing situation & generate code coverage reports for our application.

CircleCI

We practice continuous integration at ActiveCampaign and use CircleCI to run our automated tests and deploy our latest changes. This service will trigger a build based off of a pull request, run tests, and display the results at the bottom of the pull request for a lead to review before merging and deploying.

CircleCI has plenty of tools for us to use at its disposal, and one that got our attention was the ability to run tests in parallel across multiple containers. By default, Ember doesn’t know how to break its tests into partitions, so we found an addon that did.

Ember Exam

Ember Exam is a fantastic addon that gives us an incredible amount of control over how our test suite is run:

$ ember exam --split=6 --partition=1

This command tells Ember that we want to run tests as normal, but split the tests into 6 pieces and only run tests on the first piece. This is useful because when working with CircleCI, we can create a second job that specifies the second partition, and so on and so forth until we have our entire suite running in parallel. Each job will run in its own container at the same time:

You’ll notice that some tests complete about a minute faster than others. This is due to how tests are split across the containers — by file — and can result in an uneven distribution of tests across containers. We can optimize this later — for now, this is a good start!

Code Coverage Reports

With Ember Exam, we experienced a pretty significant speed increase — we could run our entire suite of tests in about 12 minutes, down from a previous 26 minutes. However, one thing we lacked was coverage reports for the application. Coverage reports tell us how much of the application is covered by tests, and they’re useful because they can let us know if a pull request we make is about to increase or decrease the overall test coverage of the application. This kind of data is also useful for making a case to focus on test coverage for previous features rather than new features, for the sake of stability for the end user.

We found an addon called ember-cli-code-coverage that could generate coverage reports for the app. They’d even taken into consideration tests running in parallel — setting parallel to true in the configuration file for the addon generates code coverage folders with random hashes at the end of the folder names so that the tests run in parallel don’t overwrite each other.

Our config.yml file ends up looking a little something like this:

- run:
   name: Run Ember tests
   command: 'ember exam --split=6 --partition=1'
  # Rename coverage report to match partition
  - run: mv ~/ember-app/coverage/report_* ~/ember-app/coverage/report_1
  # Save the coverage report for processing later
  - persist_to_workspace:
   root: ~/ember-app/coverage
   paths:
    - report_1

This logic is rinsed & repeated for each of the other 5 partitions.

The mv command above is used to take the generated folder, which might look something like report_hZsScA4c, and rename it to something we know so that we can use the persist_to_workspace key.

The persist_to_workspace key afterwards tells CircleCI that we’d like to use what was stored in the folder specified and save it for use in a separate job. We’ll use its complement, attach_workspace, to pull in all of the generated folders across our 6 partitions once we get to our last step:

- attach_workspace:
      at: ~/ember-app/coverage

This will attach each of our folders from the previous steps — report_1report_2, etc. and make them available for us to modify. We still need to merge all of the report folders into one, so we’ll use a command provided by the ember-cli-code-coverage addon:

  # Run ember coverage-merge to merge all coverage_* directories into one final report
  - run:
    name: Merge Ember Tests
    command: 'ember coverage-merge'
  # Archive test results
  - store_test_results:
    path: ~/ember-app/coverage/report
  # Upload test coverage artifacts to S3
  - store_artifacts:
    path: ~/ember-app/coverage/report

The ember coverage-merge command looks for all folders matching report_* then merges them into one single report.

Afterwards you can use store_artifacts to upload any artifacts to S3 and view them in the CircleCI summary.

Looks great! We can dive down into index.html to see what our code coverage report looks like for either the entire application or a particular route, controller, template, or component:

Next Steps

After parallelization was introduced, the next big timesink was how long it takes Ember to build before the test suite is run. We’re on an older version of Ember and still using some Bower components, but as we upgrade to later versions of Ember and remove those Bower dependencies we should see a bit of speed improvement there.

Until we can knock out some deprecations and start the Ember upgrade process, this is a bottleneck — but cutting our testing time in half and generating code coverage reports is a great start!

Follow me on Twitter: @thoughtfulleaf

Special thanks to Tomasz Nieżurawski for his article on Ember testing, which was incredibly helpful while researching this work. Thanks to Tim Rourke as well, who helped in setting up our parallel testing configuration.

A trial is worth a thousand words.

Get started today, no credit card required.