WP1: Framework for traceable computation in metrology

The aim of this work package is to create the general framework for a system of traceability in computationally-intensive metrology.

Areas of metrology were reviewed in which computation plays a critical part and - with the involvement of stakeholders from those metrology domains - validation and traceability requirements were set out for those areas. The requirements for reference data generation and the logical and organisational mechanisms of data flow as well as the issue of online certificates via the internet are specified. The outcome of the work package provides the general framework for validating metrological software within the TraCIM network. The TraCIM network is a loose alliance of NMIs and Dis which offer software validation. The network currently comprises the JRP-Partners, but it is open for other NMIs and Dis. In order to maximise the impact of this JRP in terms of delivering services to stakeholder communities, it is necessary that the operational principles of the TraCIM network for software validation fully take into account commercial and legal issues. Finally, essential and basic documents have been prepared that describe new terms and definitions necessary for the unambiguous specification of procedures for establishing traceability in computationally-intensive metrology. These documents shall provide a platform for national and international standardisation. They will provide a unified concept for the technical aspects of the JRP covering the core interests of software validation, complementing and compatible with existing methodologies and protocols for ensuring traceability of physical artefacts.

The current state of this work package is as follows.

The review and prioritisation of those areas of metrology, in which computations have a significant impact on measurement uncertainty and traceability has been done and a list of possible metrology applications was compiled (list of potential applications). Then, the Length, Chemistry, Electrical & Magnetism and Interdisciplinary Metrology domains have been selected as the domains in which computation has dominating impact on measurement uncertainty and traceability.

The requirements for the ICT infrastructure of the TraCIM network have been investigated, evaluated and specified starting with the hardware and software, information security up to data transmission, test data generation and data archiving.

A glossary of terms and definitions for computational metrology has also been created.

In order to promote european and international scientific cooperation in the field of software validation beyond the lifetime of the project general rules and principles for collaboration have been developed, which are laid down in the statutes of the TraCIM association. The main task of the association is the further development and assurance of uniform quality standards for validation of metrological evaluation algorithms. The statutes of the TraCIM association provides the framework, within which the individual member institutions (national metrological institutes and designated institutes) act legally and economically independently.

Leading NMI: CMI

Partners: NPL, PTB, UM, VSL, Hexagon, Mitutoyu, Werth, Zeiss, REG(HUD), REG(WHZ)

WP2: Formal statement of computational aim

It is only possible to verify and validate software when it is known what problem the software is intended to solve or task the software is intended to execute. A statement of the computational aim of the software is used to set the user and functional requirements for the software developer, that is, to specify what is required of software to be a conforming product, and to provide a basis for the verification and validation of a software implementation.

A computational aim is required to state what problem the software is intended to solve or task the software is intended to execute and how the problem is to be solved or task is to be executed. Decisions about how the task is to be executed, such as the choice of algorithm, are the concern of the software developer whose responsibility it is to implement the computational aim, and can depend, for example, on the environment in which the algorithm is to be implemented and used.

The specification of the computational aim should be unambiguous, complete, free from contradictions, and independent of the environment, such as hardware and software configurations, in which it is to be implemented.

The work package has developed a generic procedure for specifying computational aims that is applicable to all metrology domains. The procedure is based on using the abstract and universal language of mathematics provide specifications of computational aims. Furthermore, it distinguishes between the purely mathematical problem that is to be solved, which involves operations on numerical values, and an instantiation of the mathematical problem within a metrology area, measuring system or instrument, for which the numerical values will be associated with quantities with given dimensions, measurement units and (possibly) sets of typical values.

The work package has also developed a searchable database Opens external link in new windowhttp://www.tracim-cadb.npl.co.uk that is used to store the specifications of computational aims, and to facilitate the submission, modification, approval and examination of such specifications. It is essentially a system for storing and managing documents that constitute the specifications of computational aims with those specifications then cited by other parts of the TraCIM system. Approved specifications are in the public domain, and can be examined by users of the TraCIM system. A Opens internal link in current windowtutorial is available to support the use of the database. The database contains specifications of the computational aims for the ten priority metrology applications identified in WP1.

The database is also used to store the public domain benchmarking suites of reference data sets and reference results generated in WP3. Such data sets and results can be used to help software developers to test their software implementations, in regression testing to assess the performance of a new release of a software component against a previous version, and as a pre-cursor to certification of the software.

The aim of the research undertaken by REG(UoY) is to add rigour to the expression of computational aims and demonstrate how to gain confidence in their validity via formal analysis. The formal language of Z is being used to express important features of computational aims in metrology, including constraints and properties on the inputs and outputs of mathematical models. The language will be matched against various state-of-the-art technologies available for analysis, and techniques will be proposed for increasing assurance in the validity of computational aims. The proposed techniques will be evaluated on representative examples and recommendations made for future development of the approach.

Leading NMI: NPL

Partners:CMI, PTB, VSL, REG(HUD), REG(WHZ),REG(UoY)

WP3: Generation of reference data

The aim of this work package is to develop novel reference data generators. These are being used to generate reference data and results with proven performance. Given a well-posed statement of the computational aim (WP2) and reference input data, the solution output data is determined mathematically. If this solution is known, then software under test can be applied to the reference input data and the results output by the test software compared with the solution output data (a type of black box test of the software). To implement this approach it is necessary to have available both the reference input data and the reference output data, sometimes known as a reference pair or reference data set. The term data generator is used to represent the process of developing both the reference input data and the reference output data.

There are two general approaches to developing data generators:

  1. The first involves the provision of reference software that takes input data and produces an output solution – forward data generation. The disadvantages of this approach are:
    • The effort required to produce the reference software implementation.
    • The problem of how to validate the reference software.
  2. The second approach involves the analysis of the computational aim in order to produce input data for a given set of outputs – reverse data generation [1-5]. This approach depends on being able to construct a data generation algorithm from the mathematical analysis of the computational aim. The advantages of this approach are:
    • It is generally much easier to implement and circumvents the necessity of providing reference software.
    • The validity and performance of the data generation algorithms can often be established from first principles (just as it is always possible to verify if a set of parameter values satisfies a system of equations).

This WP has developed these data generation technologies for the priority metrology applications and computational aims defined in WPs 1 and 2 in the form of automated, self-validating data generators (ASVDGs) with reliable, robust and accurate performance.

The data generators are being used to provide different types of reference data suites:

  • Public domain benchmarking suites with both the reference input data and reference output data made available. These benchmarking suites will help software developers test their software implementations. They can also be used to test uncertified software or in regression tests to assess the performance of a new release of a software component against previous versions. The benchmarking suites will be soon available in the following database: http://www.tracim-cadb.npl.co.uk/.
  • Confidential validation suites with both the reference input data and reference output data known only to the JRP-Consortium, to be used in formal software certification.
  • Customised benchmarking suites developed for individual users (or communities) to reflect their particular requirements in a test and to demonstrate to quality engineers, etc., that the software being used (certified or not) is fit for their particular requirements, requirements that could differ significantly from standard applications. Some of the customised benchmarking data suites developed in this project are based on real world artifacts and measurement data.

Leading NMI: VSL

Partners: CMI, NPL, PTB, UM, Hexagon, Mitutoyu, Werth, Zeiss, REG(HUD), REG(WHZ)


WP4: Performance metrics

The aim of this workpackage is to provide metrics to evaluate the performance of software under test, taking into account:

  • The numerical uncertainty associated with the reference data, i.e., how close the reference input and output data is to the true mathematical solution;
  • Characteristics of the measurement data likely to arise in practice, such as the simulated measurement uncertainty associated with the reference input and/or output data;
  • A maximum permissible error (MPE), or other relevant specification, that applies in the relevant metrology domain.

The metrics will assess two features of the software under test: the numerical accuracy and its fitness for purpose. These metrics can be expressed in terms of forward and inverse error. The forward error is a measure of how far the computed solution is from the reference solution and can be related directly to any specified MPE, for example. The inverse error is a measure of how much the input data would have to be perturbed in order for the computed solution to be the exact mathematical solution for the perturbed data. If the size of the perturbation of the input data is small compared to the simulated measurement uncertainty associated with the input data, then the software may be regarded as fit for purpose.

In many practical situations, complex computational tasks are replaced by simpler ones that will only provide an approximate solution (e.g., see [2,3,6] for examples). Generally, the difference between the approximate solution and the reference solution will be much larger than the numerical uncertainty (since the software under test does not address the same computational aim). However, if the difference between the approximate solution and the reference solution is smaller than the measurement uncertainty, the approximate solution may be judged to be sufficiently accurate.

The metrics should ideally be invariant with respect to change of units of the input and outputs.

Leading NMI: NPL

WP5: Launch of TraCIM System

One of the main objectives of the TraCIM project (Traceability of Compu­ta­tionally-Intensive Metrology) is to develop new technology that allows users to validate their software directly at the point of use (e.g. on the measuring instru­ment it­self) and at any time. To achieve this goal, a new technical infrastructure is provided. It comprises a client-server architecture which allows a direct link between NMIs and service users. It is a fundamental principle that the TraCIM service is provided and hosted only by national metrology institutes (NMI) or other authorized organizations.

TraCIM’s IT architecture consists of four central modules (Fig. 1):

(a) The TraCIM core application software is a JavaEE application running on a JBoss server which processes requests from a client software (such as ordering of tests, sending test data and re­ceiving test data results, sending reports and certificates). In addition the application stores customer and order data as well and communicates with a Web shop and the expert modules.
(b) The so-called expert modules are specialised program modules, which pro­vide specific test data on demand, compare reference data to data calculated by the software under test and issue test certificates and test reports. Each ex­pert module operates basically autonomously.
(c) Since individual tests may vary significantly from one test application to another, only few input/output parameters have been specified for the data exchange between expert modules and core application. This applies, for instance, to the support of a software interface in JAVA which allows the expert module to be linked to the server system. A further requirement is that user and core application exchange data via a REST interface and XML messages whereby task-specific formatting rules have to be met.
(d) The Web shop provides methods for user registration and for ordering tests.
(e) The TraCIM client is an interface software on the computer of the user, which is preferably smoothly integrated in the software under test. It is re­sponsible for connecting the software under test with the TraCIM server. The client interface allows the retrieval of test data and – after processing them – the sending of the calculated results directly to the TraCIM core application. The client-server commu­ni­ca­tion runs via a REST interface. Hereby, the data are embed­ded into XML structures which can contain test data and calculated results in a specific format (such as binary formats, existing test data structures or newly specified formats) de­pending on the application.

Within the scope of the research project an online demonstrator of the TraCIM system was installed at PTB that allows running the validation service in order to verify the practicality of the system (see (https://tracim.ptb.de/tracim/index.jsf).

New users of PTB’s TraCIM system need to register to receive a username  and a password. For testing and trying out the communication with the TraCIM server, a manual, an example client and public data are provided (see for instance  the technical notes for the Gauss test at https://tracim.ptb.de/tracim/customer/service_gauss.jsf). The Gauss test allows users to verify the computational accuracy of Gaussian minimization algorithms.


Fig. 1: TraCIM System

 Leading NMI: PTB

 Partners: CMI, NPL, UM, VSL, Hexagon, Mitutoyo, Werth, Zeiss, REG(WHZ), REG(Ostfalia)


WP6: Creating Impact

The aim of this JRP is to provide the capability to validate metrology software at the point of use. This will be achieved by the unique and innovative internet-based software validation service that will be available worldwide for the first time and demonstrated using priority metrology applications relevant to the coordinate metrology, manufacturing and machine tools industries, measuring machines manufacturers, and software developers in the field of precision engineering. The service will be founded on exemplary science in mathematics, numerical analysis, ICT, as well as metrology domain expertise.

World-leading instrument suppliers Hexagon, Mitutoyo, Werth and Zeiss, as JRP-Partners, will help the JRP deliver this capability to thousands of users worldwide. In general, the underlying concepts of this service can be applied to the verification of complex calculations in any field of measurement science and related disciplines.

This WP has three objectives:

  • Ensuring stakeholder requirements are identified, captured and considered for incorporation into the TRACIM system design.
  • Bringing the complete spectrum of outputs of the JRP (mathematical and statistical theory, algorithms, software, reference data suites, databases, web pages, specifications, protocols, guidelines and tutorial documentation) to the stakeholder community (e.g. standard bodies, metrology organisations, instrument suppliers and TRACIM system users) in a timely manner using the most effective knowledge services.
  • Ensuring that all TraCIM system users can interact with the system effectively.

To promote the European scientific cooperation in the field of traceability of mathematical evaluation algorithms in metrology, TraCIM Association (TraCIM e.V.) was established on 12th March. The non-profit association TraCIM e.V. (German law) pursues the objective to validate numerical evaluation algorithms in the field of metrology and to trace them back to reference results of the participating metrology institutes. To achieve this, the association has taken on the task of further enhancing and implementation of consistent quality standards and takes care of the software infrastructure, which is a prerequisite for the online-validation. The collaboration is carried out on a European platform within a network of multiple national metrology institutes. As members of technical committees, industrial partners as well will have the chance to contribute to specifications, content and design of validation of algorithms.

Contact: Opens window for sending emailinfo@tracim.eu

Leading NMI: PTB

WP7: JRP Management and Coordination

The project will be managed and coordinated by NPL within the framework of its ISO 9001 accredited
management system.

Leading NMI: NPL