SimpleTest.requestFlakyTimeout

Interesting: http://mxr.mozilla.org/mozilla-central/source/testing/mochitest/tests/SimpleTest/SimpleTest.js#656

/**
 * Request the framework to allow usage of setTimeout(func, timeout)
 * where |timeout > 0|.  This is required to note that the author of
 * the test is aware of the inherent flakiness in the test caused by
 * that, and asserts that there is no way around using the magic timeout
 * value number for some reason.
 *
 * The reason parameter should be a string representation of the
 * reason why using such flaky timeouts.
 *
 * Use of this function is STRONGLY discouraged.  Think twice before
 * using it.  Such magic timeout values could result in intermittent
 * failures in your test, and are almost never necessary!
 */

That means that the current usage of SimpleTest.requestFlakyTimeout should be removed as much as possible. That sound like an interesting little project to work on, there are currently 586 instances of this function in mxr: http://mxr.mozilla.org/mozilla-central/search?string=SimpleTest.requestFlakyTimeout

Subresults logged after Simpletest.finish() are now causing failures in mochitest

Yesterday, finally the patch for bug 1032878 went in. This makes mochitests fail when subresults are being reported after SimpleTest.finish() has been called. It took quite a while, because I had to fix all the test failures beforehand, that were happening with this patch applied. After the fix went in, it caused 2 new failures that weren't caught by my tryserver runs. Luckily, it was easy to fix those.

Mochitests that rely on bfcache

A lot of the intermittent failures ("oranges") that are happening on the test machines, are race conditions, like low memory conditions/high disk usage/slow web server/etc.

When trying to fix one of those intermittent failures, I noticed how that test is able to fail: it is relying on bfcache to work.
Especially on mobile, that is a bad assumption.

One thing that could be improved is to clear the cache after each mochitest. But I also think the tests themselves shouldn't rely on bfache to work (unless of course, they are testing bfcache). So I filed bug 1057499 and pushed a tryserver run there to see how things go with bfcache turned off. Could be interesting.

Tryserver logs and where are the subtest results logged?

Any Mozilla developer will know about Tryserver, which is super-convenient to test whether patches don't cause test failures in some of the automated tests.
However, there is a caveat, normally when you run a mochitest:

./mach mochitest-plain content/base/test/test_bug339494.xhtml

You get something like this as a result:

96 INFO TEST-START | /tests/content/base/test/test_bug339494.xhtml
102 INFO TEST-PASS | /tests/content/base/test/test_bug339494.xhtml | Value check 1 - There should be no value
103 INFO TEST-PASS | /tests/content/base/test/test_bug339494.xhtml | Value check 2 
104 INFO TEST-PASS | /tests/content/base/test/test_bug339494.xhtml | Reachability - We shouldn't have crashed
105 INFO TEST-PASS | /tests/content/base/test/test_bug339494.xhtml | Value check 3 - There should be a value
106 INFO TEST-PASS | /tests/content/base/test/test_bug339494.xhtml | Value check 4 
107 INFO TEST-PASS | /tests/content/base/test/test_bug339494.xhtml | Value check 5 
114 INFO TEST-OK | /tests/content/base/test/test_bug339494.xhtml | took 1230ms

In the tryserver logs, you would only see something like this:

10:07:12     INFO -  2260 INFO TEST-START | /tests/content/base/test/test_bug339494.xhtml
10:07:13     INFO -  2261 INFO ++DOMWINDOW == 23 (0x9af4ec30) [pid = 1874] [serial = 586] [outer = 0xa3992a30]
10:07:13     INFO -  2262 INFO TEST-OK | /tests/content/base/test/test_bug339494.xhtml | took 312ms

Where are all the subresults shown?
On tryserver, those aren't shown at all anymore, see this dev.platform message.
This was done to reduce the runtime of the tests and to reduce the size of the logs.
However, when a test failure happens, the last 100 messages of that test are logged on tryserver
Sometimes, that may not be enough, though, in that case you can write this in your mochitest:

SimpleTest.requestCompleteLog()

This will make sure that everything will be logged on tryserver (or any other tbpl machine).
This was handy for me to find out what subtest wasn't logged in bug 928678.