ActiveGate plugin module performance

The ActiveGate plugin module enables you to execute custom plugins that can run any Python code and feed DESK with data. This page describes the constraints of the module, such as the execution time, CPU and memory usage.

Test setup

The data we present was gathered on two different hosts - a virtual machine with 2.5GHz 2 core 4GB and 16 core server 72GB RAM. We've tested 400 and 4000 endpoints on the virtual machine and 4000 on server. Each endpoint performs a single HTTP request to service on a local network, that takes 300ms to process. Memory and CPU usage was gathered based on top UNIX data.

Plugin execution time.

The plugin module works in one minute loops. It means that each custom plugin should execute in less than a minute - which should include all network connections to service and parsing response. If a minute limit is exceeded, the plugin will not gather data in a next loop iteration, which will cause data discontinuity.

Multiple endpoints with one ActiveGate plugin module

You can run multiple endpoints using a single plugin module, and they will be executed in parallel. It's achieved using the Python concurrency feature, which has its limitations. For more information, see Global Interpreter Lock.

Network plugin

Important concept in GIL is that only one thread can process API functions, but the lock is released on potentially blocking I/O operations, such as network requests. It means that plugin instances that make network calls can be run in parallel. See the chart presenting total execution time of 4000 endpoints, where each one performs a request to the service on local network, where the service takes 300 ms to process the request.

As you can see, despite large amount of endpoints, the plugin engine was able to successfully run them in parallel.

Network overhead

In the next scenario, 400 endpoints run one host, but each performs a request to a service on an external network. Similarly, the service takes 300 ms to process the request, but the network adds 150 ms of time.

As you can see, the real network adds a lot of overhead. The main issue here is the saturation of network interface - it is simply not able to handle 4000 requests running one after another.

CPU Heavy plugin.

In the next scenario, GIL is unable to switch between threads to handle concurrency. The plugin uses the whole CPU, and so GIL is unable to switch between plugin instances

As you can see, Python is unable to run those tasks in parallel - execution of two tasks is twice as long as one. CPU is usage also doubled.

Memory usage

The chart below is based on the UNIX top, and presents data of 4000 instances of the network plugin and 5 instances of the CPU plugin.

Summary

You have to consider many variables when planning the maximum number of endpoints for a single plugin module. If you want to run plugins that require much of the CPU usage, your execution time must stay within one minute. If you want to run plugins that use network to gather data, make sure you provide a stable connection to you services, however you still should be able to run many instances. See ActiveGate plugins limitations to learn about limitations of a single ActiveGate plugin.