title

Continuous validation of performance test workloads | Automated Software Engineering

charset

UTF-8

X-UA-Compatible

IE=edge

applicable-device

pc,mobile

viewport

width=device-width, initial-scale=1

og:url

https://link.springer.com/article/10.1007/s10515-016-0196-8

og:type

article

og:site_name

SpringerLink

og:title

Continuous validation of performance test workloads - Automated Software Engineering

og:description

The rise of large-scale software systems poses many new challenges for the software performance engineering field. Failures in these systems are often associated with performance issues, rather than with feature bugs. Therefore, performance testing has become essential to ensuring the problem-free operation of these systems. However, the performance testing process is faced with a major challenge: evolving field workloads, in terms of evolving feature sets and usage patterns, often lead to “outdated” tests that are not reflective of the field. Hence performance analysts must continually validate whether their tests are still reflective of the field. Such validation may be performed by comparing execution logs from the test and the field. However, the size and unstructured nature of execution logs makes such a comparison unfeasible without automated support. In this paper, we propose an automated approach to validate whether a performance test resembles the field workload and, if not, determines how they differ. Performance analysts can then update their tests to eliminate such differences, hence creating more realistic tests. We perform six case studies on two large systems: one open-source system and one enterprise system. Our approach identifies differences between performance tests and the field with a precision of 92 % compared to only 61 % for the state-of-the-practice and 19 % for a conventional statistical comparison.

twitter:site

@SpringerLink

twitter:card

summary_large_image

twitter:image:alt

Content cover image

twitter:title

Continuous validation of performance test workloads

twitter:description

Automated Software Engineering - The rise of large-scale software systems poses many new challenges for the software performance engineering field. Failures in these systems are often associated...

position

1

position

2

position

3

apple-touch-icon

/oscar-static/img/favicons/darwin/apple-touch-icon-6ef0829b9c.png

canonical

https://link.springer.com/article/10.1007/s10515-016-0196-8

icon

/oscar-static/img/favicons/darwin/android-chrome-192x192.png

icon

/oscar-static/img/favicons/darwin/favicon-32x32.png

icon

/oscar-static/img/favicons/darwin/favicon-16x16.png