SimpleTest.requestFlakyTimeout

Interesting: http://mxr.mozilla.org/mozilla-central/source/testing/mochitest/tests/SimpleTest/SimpleTest.js#656

/**
 * Request the framework to allow usage of setTimeout(func, timeout)
 * where |timeout > 0|.  This is required to note that the author of
 * the test is aware of the inherent flakiness in the test caused by
 * that, and asserts that there is no way around using the magic timeout
 * value number for some reason.
 *
 * The reason parameter should be a string representation of the
 * reason why using such flaky timeouts.
 *
 * Use of this function is STRONGLY discouraged.  Think twice before
 * using it.  Such magic timeout values could result in intermittent
 * failures in your test, and are almost never necessary!
 */

That means that the current usage of SimpleTest.requestFlakyTimeout should be removed as much as possible. That sound like an interesting little project to work on, there are currently 586 instances of this function in mxr: http://mxr.mozilla.org/mozilla-central/search?string=SimpleTest.requestFlakyTimeout

The power of HTML5

The infinite is possible with HTML, as you can see at https://html5zombo.com/.

Issues for a first time user, using the Flame phone in the Netherlands

While I attended the T-Dose event, one guy approached me, who had bought the Flame device. He had some issues with the phone.

First, he tried to update the device to Firefox OS 2.1, but he got stuck in the boot loader. We managed to update the base image to 1.80 and then we updated the phone to Firefox OS 2.1.

Then, we advised him to use English in the setup process, because with the Dutch lanaguage the virtual keyboard doesn't work currently.

After going through the First Time User Wizard, he noticed how the time was off a couple of hours and the date with a couple of days. Apparently, this is bug 1032101 and bug 1048154.

Even though his phone's date was only a few days off, this caused somehow the Marketplace app to not load at all. He would only get a blank page, wit a "x" in there. It took us a while to find out that the wrong time and date was the cause of that issue. There is bug 1077628 that seems to handle about this.

T-Dose event in Eindhoven

Last Saturday and Sunday, I went to the T-Dose event, manning a booth with my good friend Tim Maks van den Broek.

I met a lot of of smart and interesting people:

• Someone with an NFC chip in his hand! I was able to import his contact details into my FirefoxOS phone by keeping it close to his hand. Very funny to see, but it also felt weird.
• Bram Moolenaar (VIM author! Note to self, never complain about software in front of people you don't know)
• Some crazy enthusiastic people about Perl and more specifically, Perl 6. They even brought a camel! (ok, not a live one)

Often asked questions during the event

Where can I phone with FirefoxOS? What price? 

The only phone that you can buy online in the Netherlands, that is getting the latest updates, is the Flame device. You can order it on everbuying.com. More information on Mozilla Developer Network. Because the phone will be shipped outside of the EU, import duties will be charged, which are probably somewhere around €40.

Can I put Firefox OS on my Android device?

On MDN, there are a couple of devices mentioned, but I wouldn't know how well that works.

Is there a backup tool to backup on your desktop computer?

As far as I know, there is no backup software on the desktop computer, that would be able to make a backup of your Firefox OS phone. I know there is a command line tool from Taiwan QA that allows you to backup  and restore your profile.

Does Whatsapp work on it?

There is no official Whatsapp, but there is an app called ConnectA2, that allows you to connect with your Whatsapp contacts. I haven't used it, but Tim Maks told me it works reasonably well.

Broken adapter cable

Oops, I didn't mean to put this in my main blog (the previous post either). I keep another blog, where I have shorter notes about commands I use and some handy tricks and websites.
They aren't really meant for general consumption.

 But anyway, since I managed to post this image anyway, I might as well tell you what's going on with my macbook chargers.
It looks really broken at the base and not only it looks broken, but it is nearly broken, because when in some positions, the charger won't charge anymore.

I guess I should just buy a new one or is this why tape is invented for?

Requirements file for virtualenv for python ui tests

ManifestDestiny==0.6
blessings==1.5.1
boto==2.32.1
certifi==14.05.14
gaiatest==0.28
httplib2==0.9
lightblue==0.4
manifestparser==0.6
marionette-client==0.8.3
marionette-transport==0.3
mozcrash==0.13
mozdevice==0.40
mozfile==1.1
mozhttpd==0.7
mozinfo==0.7
mozlog==2.6
moznetwork==0.24
mozprocess==0.21
mozprofile==0.22
mozrunner==6.2
moztest==0.7
mozversion==0.6
oauth2==1.5.211
plivo==0.9.6
requests==2.4.0
treeherder-client==1.0
wsgiref==0.1.2

Software Freedom Day Utrecht

Last Saturday, I attended the Software Freedom Day gathering in Utrecht
Me and Tim-Maks van den Broek went there and gave some demos with the FirefoxOS Flame devices. It was really nice to meet other people from other open source projects and to speak with some open source enthusiasts, in general.

How to apply a patch in git e.g. merge a pull request

git remote add zacc https://github.com/zacc/gaia.git
git branch zacc
git checkout zacc
git fetch zacc
git checkout -b bug_1060381 zacc/bug_1060381

git status
# On branch zacc
# Untracked files:
#   (use "git add ..." to include in what will be committed)
#
# .idea/
# tests/python/gaia-ui-tests/gaiatest/gecko.log
# tests/python/gaia-ui-tests/gaiatest/testvars.json
# xulrunner-sdk-30/
# xulrunner-sdk-33/
nothing added to commit but untracked files present (use "git add" to track)

git remote -v
origin https://github.com/mozilla-b2g/gaia.git (fetch)
origin https://github.com/mozilla-b2g/gaia.git (push)
zacc https://github.com/zacc/gaia.git (fetch)
zacc https://github.com/zacc/gaia.git (push)

Automated testing in FirefoxOS

Here on MDN the various automated testing methods for FirefoxOS are explained, but it can still be confusing. So let me write down what I know about it.
This is the list that is given on MDN:

• Gaia UI tests Python tests for Gaia UI interactions and features.
• Gaia integration tests JavaScript integration tests for Gaia, based on Marionette.
• Gaia unit tests Gaia unit tests with no UI interaction; written in JavaScript, not Marionette-based.
• Gaia performance tests Measures Gaia app performance based on internal instrumentation. The testing harness is in-tree.

Some random notes from me:

• Marionette tests == MarionetteJS tests == integration tests
• Gaia UI tests and MarionetteJS tests are doing the same kind of testing. But Gaia UI tests also run on device (on tbpl), while MarionetteJS test only run on desktop currently. The goals is to get rid of Gaia UI tests and only use MarionetteJS tests, eventually.
• Invoke MarionetteJS test: make test-integration-test TEST_FILES=./apps/clock/test/marionette/timer_test.js
• Invoke Gaia UI test on desktop: GAIATEST_SKIP_WARNING=1 gaiatest --binary $HOME/B2G/gaia/B2G/Contents/MacOS/b2g-bin --profile $HOME/B2G/gaia/profile --testvars /Users/mwargers/B2G/gaia/tests/python/gaia-ui-tests/gaiatest/testvars.json --restart --type b2g /Users/mwargers/B2G/gaia/tests/python/gaia-ui-tests/gaiatest/tests/accessibility/phone/test_a11y_phone_edit_contact.py

MSISDN api and testing

The MSISDN api can be pretty difficult to understand. There are a whole bunch of different pieces (server-side and client-side) to the puzzle.

https://github.com/mozilla-services/msisdn-gateway/blob/master/API.md is currently the best documentation where this api and its flow is described.
This all ties in with the Firefox Loop project.

Some other links surrounding MSISDN and its testing:

• https://wiki.mozilla.org/WebAPI/MobileIdentity
• https://bugzilla.mozilla.org/show_bug.cgi?id=988469 - (loop_msisdn_verific) MSISDN verification API for privileged apps
• https://bugzilla.mozilla.org/show_bug.cgi?id=1021594 - (mobileid) [meta] Mobile ID WebAPI
• MSISDN server gateway source code: https://github.com/mozilla-services/msisdn-gateway
• Mozilla MSISDN gateway: https://msisdn.services.mozilla.com/
• Mochitests were made in: https://bugzilla.mozilla.org/show_bug.cgi?id=1022181
• Tracking bug for writing more automated mobile ID tests: https://bugzilla.mozilla.org/show_bug.cgi?id=1044048 - (mobileid-tests) [meta] Mobile ID WebAPI tests
• Preferences for debugging: https://wiki.mozilla.org/WebAPI/MobileIdentity#Debugging
• MSISDN verifier client for testing Mozilla's MSISDN server API:
  • https://github.com/mozilla-services/msisdn-verifier-client
  • http://mozilla-services.github.io/msisdn-verifier-client/
  • https://bugzilla.mozilla.org/show_bug.cgi?id=1042866 MSISDN cli (command line interface) to test the MSISDN Mozilla server API: https://github.com/mozilla-services/msisdn-cli
• https://wiki.mozilla.org/Loop/QA/MSISDN_testplan
• https://wiki.mozilla.org/Loop/QA#MSISDN_Support
• Mobile country codes can be found here: http://mxr.mozilla.org/gaia/source/shared/resources/mcc.json

Subresults logged after Simpletest.finish() are now causing failures in mochitest

Yesterday, finally the patch for bug 1032878 went in. This makes mochitests fail when subresults are being reported after SimpleTest.finish() has been called. It took quite a while, because I had to fix all the test failures beforehand, that were happening with this patch applied. After the fix went in, it caused 2 new failures that weren't caught by my tryserver runs. Luckily, it was easy to fix those.

Mochitests that rely on bfcache

A lot of the intermittent failures ("oranges") that are happening on the test machines, are race conditions, like low memory conditions/high disk usage/slow web server/etc.

When trying to fix one of those intermittent failures, I noticed how that test is able to fail: it is relying on bfcache to work.
Especially on mobile, that is a bad assumption.

One thing that could be improved is to clear the cache after each mochitest. But I also think the tests themselves shouldn't rely on bfache to work (unless of course, they are testing bfcache). So I filed bug 1057499 and pushed a tryserver run there to see how things go with bfcache turned off. Could be interesting.

Status 2014-08-21

• Patch for bug 1051791, makes errors in mochitest popup windows also report test failures automatically
• All dependancies are fixed now, patch for bug 1032878 ready for review, enforces subresults to be called before SimpleTest.finish() (sheriffing-p1)
• Initial patch for bug 1025055 - Steeplechase should take app packages as arguments, unpack them on clients. Only works on Linux, uses hardcoded stuff (need to talk with drno, geo or syd if this is the way to go, actually)
• One And Done task created, I have ideas for 3 more (bug 744387, bug 951272, 1056851)
• Send some rough instructions (unfortunately, the problems with setting up, I didn't write down, because I was too busy figuring out how to fix those, at the time) on how to set up MarionetteJS tests and Gaia UI tests to Geo and Richard. MarionetteJS tests can only be run on desktop, atm, btw. We need better documentation about this on MDN.
• On request of developer, filed bug 1055579 and tried to get a minimized mochitest for him.
• Volunteered for work on migrating QMO docs to MDN, see etherpad QMO-Docs-ChrisMills and MDN documentation plan, will attend meeting at Thursday, August 21 - 11 AM Pacific. 

Documentation about mozbase

Some good documentation about mozbase (a set of Python packages for automated testing of Mozilla products) can be found at readthedocs.org. I'm a little sad this can't be found at MDN, though.

JSON logs for mochitests

I'm pretty happy with having the ability to output results as JSON files. This could be used to track (amount of) JS/chrome errors, warnings, all kinds of things from inside an HTML page.
Usage example: ./mach mochitest-plain --log-mach=- --log-raw=mochi.json testing/mochitest/tests/Harness_sanity/
I just noticed, it didn't log some system JS error, so it's not perfect yet, see bug 1055863.

Automated testing overview

Here is a very good overview of all the automated testing that is being done at Mozilla: https://developer.mozilla.org/en-US/docs/Mozilla/QA/Automated_testing.
It's not complete and still needs some corrections and info perhaps, but it is already pretty useful (thank you Paolo Amadini!).

Tryserver logs and where are the subtest results logged?

Any Mozilla developer will know about Tryserver, which is super-convenient to test whether patches don't cause test failures in some of the automated tests.
However, there is a caveat, normally when you run a mochitest:

./mach mochitest-plain content/base/test/test_bug339494.xhtml

You get something like this as a result:

96 INFO TEST-START | /tests/content/base/test/test_bug339494.xhtml
102 INFO TEST-PASS | /tests/content/base/test/test_bug339494.xhtml | Value check 1 - There should be no value
103 INFO TEST-PASS | /tests/content/base/test/test_bug339494.xhtml | Value check 2 
104 INFO TEST-PASS | /tests/content/base/test/test_bug339494.xhtml | Reachability - We shouldn't have crashed
105 INFO TEST-PASS | /tests/content/base/test/test_bug339494.xhtml | Value check 3 - There should be a value
106 INFO TEST-PASS | /tests/content/base/test/test_bug339494.xhtml | Value check 4 
107 INFO TEST-PASS | /tests/content/base/test/test_bug339494.xhtml | Value check 5 
114 INFO TEST-OK | /tests/content/base/test/test_bug339494.xhtml | took 1230ms

In the tryserver logs, you would only see something like this:

10:07:12     INFO -  2260 INFO TEST-START | /tests/content/base/test/test_bug339494.xhtml
10:07:13     INFO -  2261 INFO ++DOMWINDOW == 23 (0x9af4ec30) [pid = 1874] [serial = 586] [outer = 0xa3992a30]
10:07:13     INFO -  2262 INFO TEST-OK | /tests/content/base/test/test_bug339494.xhtml | took 312ms

Where are all the subresults shown?
On tryserver, those aren't shown at all anymore, see this dev.platform message.
This was done to reduce the runtime of the tests and to reduce the size of the logs.
However, when a test failure happens, the last 100 messages of that test are logged on tryserver
Sometimes, that may not be enough, though, in that case you can write this in your mochitest:

SimpleTest.requestCompleteLog()

This will make sure that everything will be logged on tryserver (or any other tbpl machine).
This was handy for me to find out what subtest wasn't logged in bug 928678.