Tuesday, August 26, 2014

Subresults logged after Simpletest.finish() are now causing failures in mochitest

Yesterday, finally the patch for bug 1032878 went in. This makes mochitests fail when subresults are being reported after SimpleTest.finish() has been called. It took quite a while, because I had to fix all the test failures beforehand, that were happening with this patch applied. After the fix went in, it caused 2 new failures that weren't caught by my tryserver runs. Luckily, it was easy to fix those.

Friday, August 22, 2014

Mochitests that rely on bfcache

A lot of the intermittent failures ("oranges") that are happening on the test machines, are race conditions, like low memory conditions/high disk usage/slow web server/etc.

When trying to fix one of those intermittent failures, I noticed how that test is able to fail: it is relying on bfcache to work.
Especially on mobile, that is a bad assumption.

One thing that could be improved is to clear the cache after each mochitest. But I also think the tests themselves shouldn't rely on bfache to work (unless of course, they are testing bfcache). So I filed bug 1057499 and pushed a tryserver run there to see how things go with bfcache turned off. Could be interesting.

Thursday, August 21, 2014

Status 2014-08-21

  • Patch for bug 1051791, makes errors in mochitest popup windows also report test failures automatically
  • All dependancies are fixed now, patch for bug 1032878 ready for review, enforces subresults to be called before SimpleTest.finish() (sheriffing-p1)
  • Initial patch for bug 1025055 - Steeplechase should take app packages as arguments, unpack them on clients. Only works on Linux, uses hardcoded stuff (need to talk with drno, geo or syd if this is the way to go, actually)
  • One And Done task created, I have ideas for 3 more (bug 744387, bug 951272, 1056851)
  • Send some rough instructions (unfortunately, the problems with setting up, I didn't write down, because I was too busy figuring out how to fix those, at the time) on how to set up MarionetteJS tests and Gaia UI tests to Geo and Richard. MarionetteJS tests can only be run on desktop, atm, btw. We need better documentation about this on MDN.
  • On request of developer, filed bug 1055579 and tried to get a minimized mochitest for him.
  • Volunteered for work on migrating QMO docs to MDN, see etherpad QMO-Docs-ChrisMills and MDN documentation plan, will attend meeting at Thursday, August 21 - 11 AM Pacific. 

Wednesday, August 20, 2014

Documentation about mozbase

Some good documentation about mozbase (a set of Python packages for automated testing of Mozilla products) can be found at readthedocs.org. I'm a little sad this can't be found at MDN, though.

JSON logs for mochitests

I'm pretty happy with having the ability to output results as JSON files. This could be used to track (amount of) JS/chrome errors, warnings, all kinds of things from inside an HTML page.
Usage example: ./mach mochitest-plain --log-mach=- --log-raw=mochi.json testing/mochitest/tests/Harness_sanity/
I just noticed, it didn't log some system JS error, so it's not perfect yet, see bug 1055863.

Tuesday, August 19, 2014

Automated testing overview

Here is a very good overview of all the automated testing that is being done at Mozilla: https://developer.mozilla.org/en-US/docs/Mozilla/QA/Automated_testing.
It's not complete and still needs some corrections and info perhaps, but it is already pretty useful (thank you Paolo Amadini!).

Tryserver logs and where are the subtest results logged?

Any Mozilla developer will know about Tryserver, which is super-convenient to test whether patches don't cause test failures in some of the automated tests.
However, there is a caveat, normally when you run a mochitest:
./mach mochitest-plain content/base/test/test_bug339494.xhtml
You get something like this as a result:

96 INFO TEST-START | /tests/content/base/test/test_bug339494.xhtml
102 INFO TEST-PASS | /tests/content/base/test/test_bug339494.xhtml | Value check 1 - There should be no value
103 INFO TEST-PASS | /tests/content/base/test/test_bug339494.xhtml | Value check 2 
104 INFO TEST-PASS | /tests/content/base/test/test_bug339494.xhtml | Reachability - We shouldn't have crashed
105 INFO TEST-PASS | /tests/content/base/test/test_bug339494.xhtml | Value check 3 - There should be a value
106 INFO TEST-PASS | /tests/content/base/test/test_bug339494.xhtml | Value check 4 
107 INFO TEST-PASS | /tests/content/base/test/test_bug339494.xhtml | Value check 5 
114 INFO TEST-OK | /tests/content/base/test/test_bug339494.xhtml | took 1230ms

In the tryserver logs, you would only see something like this:

10:07:12     INFO -  2260 INFO TEST-START | /tests/content/base/test/test_bug339494.xhtml
10:07:13     INFO -  2261 INFO ++DOMWINDOW == 23 (0x9af4ec30) [pid = 1874] [serial = 586] [outer = 0xa3992a30]
10:07:13     INFO -  2262 INFO TEST-OK | /tests/content/base/test/test_bug339494.xhtml | took 312ms

Where are all the subresults shown?
On tryserver, those aren't shown at all anymore, see this dev.platform message.
This was done to reduce the runtime of the tests and to reduce the size of the logs.
However, when a test failure happens, the last 100 messages of that test are logged on tryserver
Sometimes, that may not be enough, though, in that case you can write this in your mochitest:

SimpleTest.requestCompleteLog()

This will make sure that everything will be logged on tryserver (or any other tbpl machine).
This was handy for me to find out what subtest wasn't logged in bug 928678.