github twitter linkedin
Some API Framework Benchmarks
Sep 25, 2018
3 minutes read

I was wondering the other day what framework I wanted to write some sort of API in. I enjoy challenging myself, and while I know Python decently, I thought it would be fun to learn something new.

When evaluating what framework I would like to learn, one of the criteria I wanted to explore was performance of framework. Another one is how quickly I would be able to learn the framework and have a functional API.

I decided to write some quick benchmarks in a few different frameworks. You can find the code at this GitLab project.

The Frameworks

The frameworks I decided to try were:

  • .NET Core, using C#
  • Mux using Go
  • Node.js
  • Falcon and Python
  • Flask and Python

For the Python apps, I also wanted to look at wether Gunicorn or uWSGI performed better.

The Apps

The apps all do something very simple. Generate a data structure, convert it to JSON, and return it when receiving a GET request to /hello. The data structure is as simple as:

message = { hello: "world" };

or:

var message = new Dictionary<string, string>();
message["hello"] = "world";

I think you get the gist. The apps should all return the same result, which can be tested using curl, e.g.:

curl http://<hostname>:8000/hello
{"hello":"world"}

Methodology

The apps were placed in a docker container and ran individually on a VM running in my basement. The VM has 2 CPUs and 4GB of RAM with nothing else running on it.

From another server, I then used the modern HTTP benchmarking tool, wrk.

wrk was ran with 4 threads and 100 connections for 1 minute, like this:

$ ./wrk -c 100 -d 1m -t 4 http://docker01:8000/hello
Running 1m test @ http://docker01:8000/hello
  4 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     6.00ms    3.22ms  63.54ms   92.53%
    Req/Sec     4.28k   665.29     7.36k    83.24%
  1022904 requests in 1.00m, 171.69MB read
Requests/sec:  17020.51
Transfer/sec:      2.86MB

The data were then compiled here.

The Results

Requests per second

Average latency

Discussion

Looking at the data, it’s clear from this simple test, that the Go app outperforms the rest. It had the highest number of requests per second and the lowest average latency per thread. Both the .NET and Node.js apps showed respectable results as well. Disappointingly, the Python apps just couldn’t keep up.

Of the Python apps, the Falcon framework outperformed Flask. Also, running the apps with uWSGI was considerably faster than Gunicorn. In fact, Gunicorn seemed to be a terrible performer. I should note a disclaimer here, both uWSGI and Gunicorn have a lot of parameters that can be tuned. The only thing I did for both was to run 4 threads. I’m sure someone who knows more about these servers can tune them to run faster.

One last thing for me to consider is not just the raw performance of the API frameworks, but how quickly I can prototype an API and get something ready to use. When considering that metric, the Python APIs are by far the best choice as Python is the language I’m most familiar with. The next would be Go, followed by Node.js and .NET w/ C#.

When looking at these numbers, it’s also important to consider the limitations of my benchmarks. You should do your own tests with your own application and your own environments to determine what framework will work best for you.

What will I go with? I don’t know. Probably Python w/ Falcon or Go w/ mux. Why Python? To be honest, I’ll never reach the scale with anything that I write in which the performance is really going to matter.



Back to posts