Creatis- Univesity of Lyon || NTNU || Technical University of Denmark


Evaluation framework
Sending your code

To participate in the challenge, and be included in the article under preparation, you must submit your beamformer code to the PICMUS organizers through the following email:

Both P-code and M-code (matlab source code) will be accepted. Contact the organizers in case your algorithm uses other programing languages different from MATLAB. All sub-functions and data needed to launch your beamforming algorithm must be included in the submission. In case you submission exceeds 25 MB, you can contact the organizers to arrange an alternative submission method.

Some additional data must be provided with your submission,
  • Name of the authors
  • Affiliations
  • Method's name
  • Brief description of the method
  • Data format the method can run on (RF, IQ, or both)
We set the 10th of November 2017 as deadline for code submission. If you haven't submitted your code by then your algorithm will not be included in the publication.

What do we do with your code

The code will be submitted to a number of blind tests to estimate
  • speckle quality,
  • geometrical distortion,
  • linearity of output dynamic range,
  • contrast,
  • full width half maximum, and
  • axial/lateral resolution.
For more details check the page on metrics. For internal testing a evaluation code is provided to the participants in the script script_evaluate_image.m in the examples folder.

A number of datasets, different from the ones provided to the participants, will be used to evaluate the performance of the algorithms. This is to avoid the overespecialization of beamforming algorithms to particular datasets. These datasets will be similar to those provided for testing, but the locations and number of testing structures will vary. This way participants cannot tune their algorithms for a particular dataset.

In order to produce more comprehensible figures, two different transmission schemes will be tested. This means that the methods should be able to work with an arbitrary set of steered plane-waves. The authors can still concentrate on optimize their method for a specific transmission scheme (for instance 1 plane-wave, or 3 plane-wave) but categories will not be defined a priory. See Transmission sub-menu for more details on this.

Article preparation

A manuscript is currently under preparation with the results of the challenge. We aim to submit this manuscript to IEEE TUFFC the 1st of May 2018. The results of the evaluation tests will be included and we will provide an analysis of performance of the participant algorithms based on the obtained metrics.

All the participants will be included as coauthors in the manuscript.