Performance Testing the DDF family of Systems

Goal:

Goal

We need a test driver that can be used to see how well DDF can meet performance expectations.
These performance tests should be able to be ran as Integration Tests as well as be driven manually. 

They should be parametrizable (N queries over X time), (K simultaneous queries), and answer questions such as

- "can DDF handle the load given?"
- "how many queries can be made simultaneously before failure?"
- "how long does a query take to respond when N queries are issued simultaneously?"

Past performance tests

jmeter tests
- must be ran manually each time.
SoapUI tests
- Docker to create the base image for running tests, SoapUI Used for the actual test definition.
- SoapUI was previously used for testing but is not preferred anymore over unit/integration tests.

Performance Testing Frameworks

jmeter
- Open Source
- can be run distributed
- Heavily reliant on IDE to generate tests
- generates xml test plans that are then ran
- http://jmeter.apache.org/
gatling
- Has a DSL based on scala
- can be run distributed
- http://gatling.io/#/
grinder
- Open Source
- Allows test scripting in java, jython, and clojure
- can be run distributed
- No recent maintainers
- Would be great base for our own frameowork in some future project
- https://sourceforge.net/p/grinder/code/ci/master/tree/
spf4j
- (simple performance framework for java)
- This is more of a Performance monitoring/metrics library but could be useful
- http://zolyfarkas.github.io/spf4j/
locust.io
- Open Source
- Python for writing tests
- can be run distributed
- http://locust.io/
tsung
- xml based
- can be run distributed
- (written in erlang–just a side note)

Also I think its important to mention a large reason for preference of locust is the rapid speed at which we can prototype/develop tests. The minimal investment means that down the road when we either invest in creating our own performance testing framework and/or find a suitable long term solution these can be easily ported and thrown away.

An example of locust which I personally would use: in ~30 lines of code

intrigue-load-test.py

import locust
from locust import task


class IntrigueTasks(locust.TaskSet):
    def on_start(self):
        self.client.verify = False
        
    @task
    def index(self):
        self.client.get('/search/catalog/')

    @task
    def sources(self):
        self.client.get('/services/catalog/sources')

    @task
    def catalogid(self):
        self.client.get('/search/catalog/internal/localcatalogid')

    @task
    def workspace(self):
        self.client.get('/search/catalog/internal/enumerations/metacardtype/workspace')


class IntrigueUser(locust.HttpLocust):
    task_set = IntrigueTasks
    min_wait = 5000
    max_wait = 10000
    host = "https://example.com:8993"

You get robust load testing with a simple UI for statistics and results.

With static resources emulating most of the Intrigue network requests:

Also provides completed within stats:

Percentage Completed Within

Percentage of the requests completed within given times
 Name                                                           # reqs    50%    66%    75%    80%    90%    95%    98%    99%   100%
--------------------------------------------------------------------------------------------------------------------------------------------
 GET /catalog/cesium/Widgets/widgets.css                            64      7      7      7      7      8      8      9    260    262
 GET /search/catalog/                                               73      7      8      8      8      9     10     12     64     64
 GET /search/catalog/1.bundle.js                                    71     40     42     43     44     50    230    900   1100   1052
 GET /search/catalog/2.bundle.js                                    62     49     50     51     51     53     55     64     69     69
 GET /search/catalog/blank.html                                     62      7      7      7      8      9    100    290    490    486
 GET /search/catalog/bundle.js                                      66      7      7      8      8      9     12    140   1000   1018
 GET /search/catalog/css/fonts/open-sans/open-sans-v13-latin-300.woff2       69      9     10     10     11     11     12     15   1000   1014
 GET /search/catalog/css/fonts/open-sans/open-sans-v13-latin-regular.woff2       57      9      9     10     10     10     19     22     45     45
 GET /search/catalog/css/index.css                                  80     13     14     15     15     17     20     23     51     51
 POST /search/catalog/internal/cql                                2148     14     15     15     15     17     24    280    680   1036
 GET /search/catalog/internal/enumerations/metacardtype/workspace     1071      7      7      7      8      8     14    440    770   1025
 GET /search/catalog/internal/localcatalogid                        87      7      7      7      8      8      9    530    560    560
 GET /search/catalog/lib/bootstrap/dist/css/bootstrap.min.css       66     13     13     14     14     15     24    470    750    753
 GET /search/catalog/lib/cesium-drawhelper/DrawHelper.css           60      7      7      8      8      9     10     29    870    874
 GET /search/catalog/lib/eonasdan-bootstrap-datetimepicker/css/bootstrap-datetimepicker.css       93      7      8      8      8     10    180   1000   1000   1014
 GET /search/catalog/lib/font-awesome/css/font-awesome.min.css       75      8      9      9     10     11     15    400   1000   1011
 GET /search/catalog/lib/jquery-ui-multiselect-widget/jquery.multiselect.css       61      7      7      7      8      9      9     11    930    927
 GET /services/catalog/sources                                     761      8      8      8      8      9     12    490    940   1004
--------------------------------------------------------------------------------------------------------------------------------------------