As the presence and role of computer systems in our daily life increases, we rely more and more on the services that are provided by software. On the one hand, more tasks and functions are delegated to software systems (e.g., in the automotive domain), and on the other hand, the expectations and demands on the variety of services these systems provide are dramatically growing (e.g., in mobile phones). In this context, the success of a software product may not only be dependent on the logical correctness of its functions, but also on their quality characteristics and how they perform. Such system characteristics, referred to and captured as Extra-Functional Properties (EFPs), or Non-Functional Properties, have determinant importance, particularly in resource-constrained systems. For instance, in the real-time embedded domain there can be limitations on available memory, CPU and processing capability, power consumption, and so on, that need to be considered along with timing requirements of an application. EFPs can span different aspect of the system such as security, safety, robustness, performance, power consumption, etc. With the increasing complexity of modern software systems, the EFPs need to be considered and continuously tested from early stages of the system development until the system is deployed in production, inclusively. Furthermore, considering the rapid development towards increased integration of software with the social and physical world that we see today, quality aspects become more important in an increasing number of the systems and devices we use and depend on. These systems, therefore, need to be tested with a special attention to EFPs such as safety, security, performance and robustness. Testing a system with respect to its EFPs, however, poses specific challenges and traditional functional testing methods and approaches may not simply be applicable. Examples of such challenges are: fault localization, the need to have appropriate techniques for different types of EFPs, the role and impact of the environment in testing EFPs, observability and testability issues, coverage and test-stop criteria, modeling EFPs and generating meaningful test cases, mutation operators for EFPs, etc.

Topics of Interest

This special issue focuses on all topics relevant to testing extra-functional properties. In particular, the topics of interest include, but are not limited to:

  • Model-based testing of EFPs; e.g., choice of modeling languages to capture EFPs and their role on testability, model-based test case generation, etc.
  • Use of machine learning and artificial intelligence for testing EFPs
  • Search-based testing techniques for EFPs
  • Mutation-based testing for EFPs; e.g., application of mutation techniques for testing of EFPs particularly introduction of EFP-specific mutation operators
  • Testing of AI and Machine Learning based solutions with respect to quality attributes; such as safety, security, and ethical concerns
  • Testability, observability, and the role of the platform; e.g., how choosing an operating system can impact testability of EFPs, for instance, a real-time operating system, introducing testability mechanisms into a platform, designing middlewares for testing of EFPs
  • Empirical studies and experience reports; e.g., on the importance of testing EFPs, evaluation of testing methods, case-study and reports on project failures due to EFPs, comparison of methods and techniques
  • Quality assurance, standards, and their impact on testing EFPs
  • Requirements and testing EFPs; e.g., identification and generation of test oracles for EFPs from requirements, requirements for testability, traceability
  • Coverage criteria in testing EFPs
  • Processes and their role in testing EFPs; e.g., agile and TDD
  • Fault localization for EFPs and debugging
  • Formal methods, model-checking, and reasoning about EFPs
  • Parallelism, Concurrency, and Testing of multicore applications
  • Performance, Robustness, and Security Testing
  • Testing real-time, embedded, and cyber-physical systems, and their challenges
  • Testing quality characteristics of distributed, mobile, and cloud applications


Extended versions of workshop/conference papers should contain at least 30% new material and should explain clearly the additional contribution. It also should have a different abstract and should cite the original workshop/conference paper. Please submit your paper electronically using the Software Testing, Verification & Reliability manuscript submission site ( Select "Special Issue Paper" and also specify in the cover letter that you are submitting to the "Special Issue on Testing Extra-Functional Properties - 2nd Edition". It is also encouraged that you email and notify us when you submit your paper.

Important Dates

Papers undergo the review process upon arrival. Notifications will be sent out to the authors as soon as a decision is reached for an article, but latest by Oct 31, 2023

  • Manuscript submission - open: October 1, 2022
  • Manuscript submission - deadline: Feb 28, 2023
  • First notification: 3 months after submission
  • (Last) revised manuscript submission: Aug 15, 2023
  • (Last) notification of the final decision: Oct 31, 2023

Special Issue Editors