No special permission is required. The Telemetry dashboard is open to all. Don’t have a Persona account? No problem. Click the sign in button to be prompted to create a free account.

This change is now live. You can try it yourself by visiting the Telemetry dashboard at http://mzl.la/telemetrydash.

Before I get into the details about the data issues, I think a little background is in order for those who don’t have it.

Telemetry sends a daily ping with the information collected by the Telemetry probes. This data is aggregated and presented in the Telemetry Dashboard and Telemetry Evolution (http://mzl.la/telemetrydash). The Telemetry data is stored in three separate data repositories:

1. daily ping (raw) data is stored in hbase
2. aggregated data for use in the Telemetry Dashboard is stored in elastic search
3. trend data for use in Telemetry Evolution is stored in a relational database

A job is run to aggregate the raw data (1) into the data for the Telemetry Dashboard (2). A second job is then run to create trend data from the aggregate data (2) for Telemetry Evolution (3).

Several weeks ago cww discovered that the Telemetry dashboard was not displaying the volume of data that is expected from the Nightly channel. The source of the problem was determined to be an issue with the newly landed persistent Telemetry feature that caused the Telemetry ping to fail. However, once this change was backed out a second issue was discovered with the data job that processes the raw data from the daily Telemetry ping (in hbase) and aggregates it into a form that can be consumed by the Telemetry Dashboard (elastic search). As the trend data for Telemetry Evolution is generated from the elastic search data it was also affected.

The status and summary of these two issues is:
1. The persistent Telemetry issue has been resolved but has resulted in a gap in Nightly Telemetry data submissions from Mar 1-17, 2012.
2. As the issue with the data job was with the job that aggregates the raw data into elastic search there was no issue with data loss of any of the raw data. The job has been corrected and Telemetry data is now available from Jan 1, 2012, onwards. The metrics team is running the jobs to repopulate the trend data from prior to Jan 1, and expect the full data set to be back online shortly. (Bug 731662.)

Our next steps are to put data validation processes in place to catch data issues early. Discussion is taking place in bug 742883, bug 742897, and bug 742903. These issues also surfaced a need to discuss the Telemetry data retention policy, which is being done in bug 742496.

• Vlad released a new version of about:telemetry that shows full slow SQL data on Nightly builds. (He’s also working on bundling about:telemetry in product.)
• Vlad also received more super slow start-up data from the metrics team and is continuing his investigation.
• The profiler add-on can now toggle the Jank/Stackwalking feature.
• Vlad integrated Benoit’s SPS extension with the Symbolication Server.
• Vlad wrote a blog post about setting up a local symbolication server.
• Peptest results from try, mozilla-inbound, and mozilla-central are now being reported to http://mrcote.info/peptest/.
• Mark added Peptest documentation to MDN.
• mccr8 has continued his attack on the CC graph landing patches for bug 736563 and bug 712735 and is working to land even more patches (bug 737075 and bug 740185).
• Rafael implemented write poisoning, which detects late disk writes that are occurring when we’re trying to shut down.
• Rafael discovered that xul <query> tags don’t get finalized (the connections remain open) until the last cycle collector run and is working on a patch.
• Felix continued working to make form history use the asynchronous storage API. Patch is getting close to r+.
• Progress was made on replacing synchronous favicons calls in the codebase with async calls. Not quite ready to land but moving along.

Last week Taras started his post with a summary of the work that went into Firefox 13. I’m pleased that Vlad has resolved the first Snappy bug for Firefox 14. (It’s a Telemetry probe but definitely a righteous change as it’s measuring slow SQL.)

Here’s a rundown of what happened in Snappy this week.

Slow SQL

As I said above, Vlad landed a Telemetry probe for slow dynamic SQL queries. He also updated about:telemetry to show the data.

Profiler

The profiler saw progress this week. I hope this will continue as Benoit has a bit of time to devote to Snappy again.

• Ehsan landed Android stackwalking support.
• Markus Stange (mstange) has been rocking the entire UI. See the new front end screenshot. (bug 719530)
• Front-end now using chrome and web worker. Improved the performance of operations in the UI to support a larger profile.
• Investigated adding an option to IonMonkey to generate JIT with frame pointers on profiling builds, which will let us use our current infrastructure. (bug 736299)
• Working towards multiple thread profiling and investigating OpenGL profiling with fence.
• Vlad and Benoit are working to modify the SPS profiler extension to work with the Snappy Symbolication Server. (See Chromehang below.)

Chromehang

Updates courtesy of Vlad.

• Mark added date controls and a selector for period of moving average.
• Mark is working on figuring out how to detect and report changes.
• Peptest is enabled on try. Submit your test cases (mcote can review) for changes for which you want to measure the effect on responsiveness.

Cycle Collector

• Looking at speeding up CCs with two tabs open (Gmail+mozilla Wiki) by removing more stuff from the CC graph. Landed bugs 735342, 736763, 735550, 737060.
• mccr8 has a preliminary patch for delaying slow CCs. (bug 710496)
• More preliminary patches from mccr8 for removing more things from the CC graph (bug 736563, bug 737075)
• Next will investigate CCs with 6 or so tabs open and reported slowdowns from add-ons.

Shutdown

Rafael identified new issues related to shutdown and has some further paths to check.

• The switch to private browsing doesn’t close the db. (bug 735697)
• Checking that Connection::internalClose is called on every connection we called Connection::initialize on. (bug 711076)
• Have the basics for write poisoning working. (bug 732173)
• Patch under review to decide if we should take the fast exit path or not. (bug 731741)

Thank you too everyone who provided their expertise in bug 731672 in order to get this extension back online quickly.

You can follow the work in bug 731672.

I’m excited, nay thrilled, to announce that you can now embed Bugzilla data in wiki pages on wiki.mozilla.org via the mediawiki-bugzilla extension. Let me jump into two samples to show you the value:

• Keep your feature pages up-to-date by embedding live bug data to show real time status of related bugs. See the development sections of the Silent Update: Removal of OS security dialog for Windows and Silent Update: Background updates feature pages for live samples.
  
• Keep project pages current by automatically pulling bug lists and show a rolled up view of status via charts. See the project Snappy page for an example use of charts.
  

You can also setup personal and team work pages with custom views of bug data and create kanban style and other dashboards.

To embed bug data in your own pages use the following markup:

<bugzilla>
    {
        json query using Bugzilla REST API
    }
</bugzilla>

Here are some examples to get you started.

Show a listing of 3 specific bugs:

<bugzilla>
    {
        "id": "691829,706936,721366"
    }
</bugzilla>

Show all new bugs with the whiteboard value [telemetry:p1]:

<bugzilla>
    {
        "whiteboard": "[telemetry:p1]",
        "status": "new"
    }
</bugzilla>

Show a chart of the telemetry:p1 bugs by status

<bugzilla type="count" display="bar">
    {
        "whiteboard": "[telemetry:p1]",
        "x_axis_field": "status"
    }
</bugzilla>

Further information and examples are available in the readme.

There are some issues that you should be aware of when using this extension:

1. Data is pulled anonymously. There is no way to pull private bugzilla data including security and other confidential bugs.
2. There is no current way to specify multiple values for parameters (like status) that require multiple query parameters in the bugzilla api (bug 729259).
3. Only bar charts are currently supported.
4. Data that exceeds the wikimedia memory limit will result in a blank page. The readme has instructions on recovering from this scenario.

Thanks to Christian Legnitto for kicking off this project last summer and Brandon Savage and Jake Maul for getting the code into a production ready state and deployed on wikimo.

Bugs can be filed here.

If you’re interested in contributing to this extension or using it with your own mediawiki installation, you can find the source on github.

Mobile

Back in October it was decided to rewrite the Firefox mobile UI as a native Android application. This is no small feat. Erin stepped up as the project manager for this effort and has been doing an amazing job. If you haven’t seen the progress that this team has made in 3 months check out the latest Nightly or Aurora builds. I can’t wait to see this effort reach the first release of the new native Android application early in 2012.

Silent Update

Transforming the Firefox update experience into a silent update experience is another big task that requires changes to the way the browser handles update related prompting including restart, Windows user account control (UAC), and information pages, the update operation itself, and add-on compatibility. Lawrence has been managing this effort, which landed changes for the restart prompt, the what’s new page, and add-ons default to compatible in Q4. The remaining changes are slated to land in 1Q 2012 with all features being released in the 1H 2012.

Bugzilla Anthropology

In an effort to better understand our project’s use of bugzilla, Martin conducted a series of interviews with Mozilla contributors on various teams. The goal of this project is to understand how people use bugzilla. Martin put together a good blog post that provides details about this important project.

Crashkill

Sheila continued to run the ever important Crashkill program that monitors and acts on Firefox stability issues. Along with supporting Firefox desktop, the Crashkill program has been actively contributing crash reports to the new mobile native UI rewrite.

Telemetry

Lawrence picked up management of Telemetry with the immediate goals of improving the Telemetry adoption rate, improving the consumability of the Telemetry dashboard, and increasing adoption of the framework by the implementation of more probes. 4Q saw the release of v2 of the Telemetry dashboard and the initial efforts to improve adoption of this data collection framework among the users on the various Firefox channels. 1Q 2012 will see an improved Telemetry adoption rate, continued work on the dashboard, and even more probes.

Snappy

In October Firefox project leadership made a decision to put the electrolysis effort on hold to focus resources on short term responsiveness gains. The result of this is project Snappy, which kicked off late in 4Q. Lawrence is managing Snappy, which includes broad membership from various front-end and platform teams. In 1Q we plan to move from investigation to solutions. Look for changes to start landing as early as 1Q.

HTML5 Games

Martin kicked off a new effort late in the 4Q to make Firefox an awesome platform for HTML5 based game development. There is lots more coming in 1Q on this active effort. You can follow along on the HTML5 games wiki page.

The goal of Electrolysis is to separate the user interface process from the content processes (the portion of the browser that renders Web pages for display) with an aim to improve responsiveness in the user interface. There are a number of secondary program goals such as crash protection, support for multi-core machines, and sandboxing (isolating) content to improve security. (See Chris Blizzard’s goals for multi-process Firefox blog post for more details about the goals.) All of these items have technical merit and it was not the technical merit of the initiative that was under review.

Electrolysis is a huge undertaking. I can’t emphasize that point enough. Converting an established product, like Firefox, from a single- to multi-process architecture requires the involvement and coordination of many teams. Most recently I have been working with the accessibility, add-ons, front-end, graphics, and release engineering teams on various projects. Electrolysis requires a large investment of resources and time and has a long timeline for completion. How long? At this point we do not have a definitive answer as there are many unknowns that need to be investigated and addressed, such as how to ensure that add-ons can function in this new multi-process environment.

The list of responsiveness initiatives includes a number of smaller initiatives, such as out of process plug-ins (OOPP), Places-optimization, and incremental garbage collection. When looking at the investment in Electrolysis, it is quickly apparent that the talented people working on this program could instead be working on some of these smaller initiatives. We reviewed a short list of these initiatives and concluded that the results of working on the short term initiatives should provide significant responsiveness wins delivered in a shorter (known) time frame than Electrolysis. This means a more responsive Firefox will be available on your desktop sooner.