App Assay Methodology (master)

Top-level questions for the research

For a given App, the top-level questions for our research on this App are:

  • How effective is the App against COVID-19? Can it be quantified?

  • Are there downsides, or risks, of using the App, and if so, what are they? Can they be quantified? Are there particular population groups of App Users that may be more at risk than others?

  • Does the App, based the its current implementation status and usage that we find, have unique opportunities to enlarge its impact on the pandemic or further reduce downsides or risks? What recommendations can we make to the App Creator in the context of current world-wide best practices?

Target audiences and their needs

Our research is targeted towards the following audiences:

  • The general public wants to know which App they should use, why they should use it, and that it is safe for them to do so. Should there be risks or downsides: what are they?

  • Public health authorities want to know the best practices for Apps helping with the pandemic and understand how they have been implemented around the world, with the goal to adapt them to local circumstances.

  • App developers and operators may be unfamiliar with some approaches (technologies, operations or governance) that have been successful for other App deployments around the world, and would like to learn how to adapt them for their App.

Information sources for an App

Our research into an App is based on the following information sources, to the extent they are available:

  • Information self-asserted by App Creators, such as in public communications (e.g. press releases, website, app store listing) describing their App.

  • Information inferred by third parties, such as by direct examination of the running App, review of publicly available code, reverse engineering, or review of assertions by App Creators.

  • Information from third-party audits, in which a qualified independent third-party entity has the full cooperation of the App Creators, and full access to all relevant people and information, in order to assess all aspects of the App.

We consider third-party audits the best source for reliable information about an App. Relevant parties include experts in epidemiology, public health, technology, operations, governance and civil rights.

Caveats

Completeness and correctness

Analyzing an App is a complex undertaking. As for any complex undertaking, unfortunately mistakes not only can happen, but are virtually certain to happen. So regardless how diligent a job we try to do, we must work under the assumption that our analysis will be incomplete and incorrect in part.

When this happens, we apologize, and would like to invite you to tell us by filing an issue in our issue tracker. We will review the issue, and depending on whether we concur with your assessment or not, make suitable corrections to this site.

Even after we close the issue in the issue tracker, it and any resulting discussion will remain available there for review by you and others.

General disclaimer

Note the site’s general disclaimer.

Questionnaire

We use this Questionnaire for an App.

Technical assay of an App

We perform a technical assay if enough of the source code of the App is available, and the cost/benefit ratio appears beneficial related to our available resources.

The technical assay is directed by the questionnaire that we attempt to answer.

The details of the technical assay highly depend on the features, architecture, and availability of information about the App.

Summary of the assay for an App

We publish a summary of an App’s assay on the top-level questions as follows:

  • Impact on the pandemic.
  • Avoidance of downsides or risks.

These are separately rated.

Symbols explained

For both ratings, the same symbols are used.

  • Independent assessment by third parties, such as App Assay or other independent organizations or individuals, without enough help from the App Creators to qualify as an Audit, is shown in the bottom-left corner. The color of the arc can be red, yellow, green, or gray, dependent on the third party/parties' external assessment (see below).
  • If an App has been audited, a single colored circle is used for the rating. The color of the circle can be red, yellow or green, dependent on the result of the Audit (see below).

Colors

  • The App is implementing world-wide best practices on this subject, or is close to them. We have not found major concerns.
  • The App implements a set of tradeoffs on this subject that are debatable. Reasonable people may disagree on them. For example, is it better to take an extra month of development time for an App that makes a significant positive impact on COVID-19, if during that time much better privacy protections can be implemented?
  • At App Assay, we do not make tradeoffs like this; we only provide the information to enable others to make this tradeoff.
  • Insufficient information is available to ascertain whether the App is effective, or what risks and harms may exist when using the App.

Hypothetical example

For example, a hypothetical App may have the following summary:

Impact on the pandemic

  • While the App traces contacts, it encourages large crowds to congregate for prolonged periods of time. As a result, its net-impact on COVID-19 is likely to be negative.

which should be interpreted as:

  • In the view of the App Creators, the App has a significantly positive impact on fighting back the pandemic (green arc upper-left),
  • while a 3rd-party assessment disagrees (red arc lower-left), with an explanation that is given in text to the right.

Avoidance of downsides or risks

  • Privacy is substantially below best practices

which should be interpreted as:

  • An audit on potential risks and harms was conducted, which found a substandard result (full circle in yellow), with an explanation that is given in text to the right.

Sources for the methodology