2020-03-14 20:21:20 -07:00
# Benchmarks
Huma makes use of some reflection. Reflection is slow. It's natural to think that this may make servers built with Huma slow.
2020-03-18 08:58:47 -07:00
This folder contains four implementations of the same service using the following libraries:
2020-03-14 20:21:20 -07:00
- [FastAPI ](https://github.com/tiangolo/fastapi ) (Python, popular framework)
- [Gin ](https://github.com/gin-gonic/gin ) (Go, popular framework)
2020-03-18 08:58:47 -07:00
- [Echo ](https://echo.labstack.com/ ) (Go, popular framework)
2020-03-14 20:21:20 -07:00
- [Huma ](https://github.com/danielgtaylor/huma ) (Go, this project)
2020-03-18 08:58:47 -07:00
The [wrk ](https://github.com/wg/wrk ) benchmarking tool is used to make requests against each implementation for 10 seconds with 10 concurrent workers. The results on a 2017 MacBook Pro are shown below:
2020-03-14 20:21:20 -07:00
2020-03-18 08:58:47 -07:00
| Implementation | Req/s | Avg Req (ms) | Transfer | Relative Speed |
| -------------- | -----: | -----------: | --------: | -------------: |
| FastAPI | 7,052 | 1.440 | 1.35MB/s | 8% |
| Gin | 85,892 | 0.118 | 14.91MB/s | 100% |
| Echo | 84,532 | 0.120 | 14.67MB/s | 98% |
| Huma | 80,478 | 0.136 | 13.97MB/s | 94% |
2020-03-14 20:21:20 -07:00
< details >
< summary > Benchmark Details< / summary >
2020-03-18 08:58:47 -07:00
First install the benchmark tool, e.g:
2020-03-14 20:21:20 -07:00
```sh
2020-03-18 08:58:47 -07:00
$ brew install wrk
2020-03-14 20:21:20 -07:00
```
2020-03-18 08:58:47 -07:00
For Python, install [Pipenv ](https://pipenv.pypa.io/en/latest/ ). Then:
2020-03-14 20:21:20 -07:00
```sh
$ cd benchmark/fastapi
$ pipenv install
$ pipenv run uvicorn --workers=8 --no-access-log main:app
```
In another tab:
```sh
2020-03-18 08:58:47 -07:00
$ wrk -t 10 -d 10 -H "Authorization: bearer 123" http://localhost:8000/items/123
2020-03-14 20:21:20 -07:00
Running 10s test @ http://localhost:8000/items/123
2020-03-18 08:58:47 -07:00
10 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 1.44ms 748.88us 25.13ms 93.33%
Req/Sec 713.35 232.62 1.28k 59.02%
71237 requests in 10.10s, 13.59MB read
Requests/sec: 7051.84
Transfer/sec: 1.35MB
2020-03-14 20:21:20 -07:00
```
For the Gin benchmark:
```sh
$ go run ./benchmark/gin/main.go
```
In another tab:
```sh
2020-03-18 08:58:47 -07:00
$ wrk -t 10 -d 10 -H "Authorization: bearer 123" http://localhost:8888/items/123
2020-03-14 20:21:20 -07:00
Running 10s test @ http://localhost:8888/items/123
2020-03-18 08:58:47 -07:00
10 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 118.42us 101.92us 5.00ms 96.21%
Req/Sec 8.65k 411.81 11.28k 84.23%
867478 requests in 10.10s, 150.57MB read
Requests/sec: 85892.15
Transfer/sec: 14.91MB
```
For the Echo benchmark:
```sh
$ go run ./benchmark/echo/main.go
```
In another tab:
```sh
$ wrk -t 10 -d 10 -H "Authorization: bearer 123" http://localhost:1323/items/123
Running 10s test @ http://localhost:1323/items/123
10 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 120.17us 94.26us 5.66ms 95.71%
Req/Sec 8.50k 496.66 10.33k 78.71%
853731 requests in 10.10s, 148.18MB read
Requests/sec: 84532.91
Transfer/sec: 14.67MB
2020-03-14 20:21:20 -07:00
```
For Huma:
```sh
$ go run ./benchmarks/huma/main.go
```
In another tab:
```sh
2020-03-18 08:58:47 -07:00
$ wrk -t 10 -d 10 -H "Authorization: bearer 123" http://localhost:8888/items/123
2020-03-14 20:21:20 -07:00
Running 10s test @ http://localhost:8888/items/123
2020-03-18 08:58:47 -07:00
10 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 136.38us 163.89us 8.27ms 94.67%
Req/Sec 8.09k 456.92 10.07k 79.60%
812830 requests in 10.10s, 141.08MB read
Requests/sec: 80478.11
Transfer/sec: 13.97MB
2020-03-14 20:21:20 -07:00
```
< / details >
Does reflection slow Huma down? Yes. Does it matter for 99% of use-cases? Probably not. For that small slowdown you get a lot of nice features built-in.
2020-03-18 08:58:47 -07:00
Also, if you like FastAPI, you can get a **massive** speedup (over 10x) by switching to Huma while keeping most of the same feature set.