Tuesday, August 19, 2014

Tryserver logs and where are the subtest results logged?

Any Mozilla developer will know about Tryserver, which is super-convenient to test whether patches don't cause test failures in some of the automated tests.
However, there is a caveat, normally when you run a mochitest:
./mach mochitest-plain content/base/test/test_bug339494.xhtml
You get something like this as a result:

96 INFO TEST-START | /tests/content/base/test/test_bug339494.xhtml
102 INFO TEST-PASS | /tests/content/base/test/test_bug339494.xhtml | Value check 1 - There should be no value
103 INFO TEST-PASS | /tests/content/base/test/test_bug339494.xhtml | Value check 2 
104 INFO TEST-PASS | /tests/content/base/test/test_bug339494.xhtml | Reachability - We shouldn't have crashed
105 INFO TEST-PASS | /tests/content/base/test/test_bug339494.xhtml | Value check 3 - There should be a value
106 INFO TEST-PASS | /tests/content/base/test/test_bug339494.xhtml | Value check 4 
107 INFO TEST-PASS | /tests/content/base/test/test_bug339494.xhtml | Value check 5 
114 INFO TEST-OK | /tests/content/base/test/test_bug339494.xhtml | took 1230ms

In the tryserver logs, you would only see something like this:

10:07:12     INFO -  2260 INFO TEST-START | /tests/content/base/test/test_bug339494.xhtml
10:07:13     INFO -  2261 INFO ++DOMWINDOW == 23 (0x9af4ec30) [pid = 1874] [serial = 586] [outer = 0xa3992a30]
10:07:13     INFO -  2262 INFO TEST-OK | /tests/content/base/test/test_bug339494.xhtml | took 312ms

Where are all the subresults shown?
On tryserver, those aren't shown at all anymore, see this dev.platform message.
This was done to reduce the runtime of the tests and to reduce the size of the logs.
However, when a test failure happens, the last 100 messages of that test are logged on tryserver
Sometimes, that may not be enough, though, in that case you can write this in your mochitest:


This will make sure that everything will be logged on tryserver (or any other tbpl machine).
This was handy for me to find out what subtest wasn't logged in bug 928678.

No comments:

Post a Comment