Sérgio Cipriano's Website

How to view the Debian Upload Queue

2026-04-21T12:16:58-03:00

How to view the Debian Upload Queue

Some people may not know this, but the Debian Upload Queue is public and very easy to access:

$ curl ftp://ftp.upload.debian.org/pub/UploadQueue/
drwxr-sr-x   18 1518     1281         4096 Jun 26  2019 DELAYED
-rw-r--r--    1 1518     1281         3442 Jul 14  2025 README
-rw-r-----    1 117      1281         3052 Apr 20 21:32 neovim-tokyonight_4.14.1-1.debian.tar.xz
-rw-r-----    1 117      1281         2119 Apr 20 21:32 neovim-tokyonight_4.14.1-1.dsc
-rw-r-----    1 117      1281         5533 Apr 20 21:32 neovim-tokyonight_4.14.1-1_amd64.buildinfo
-rw-r-----    1 117      1281         2637 Apr 20 21:32 neovim-tokyonight_4.14.1-1_source.changes
-rw-r-----    1 117      1281       197584 Apr 20 21:32 neovim-tokyonight_4.14.1.orig.tar.gz

Zero-Code Instrumentation of an Envoy TCP Proxy using eBPF

2025-12-31T11:15:25-03:00

Zero-Code Instrumentation of an Envoy TCP Proxy using eBPF

I recently had to debug an Envoy Network Load Balancer, and the options Envoy provides just weren't enough. We were seeing a small number of HTTP 499 errors caused by latency somewhere in our cloud, but it wasn't clear what the bottleneck was. As a result, each team had to set up additional instrumentation to catch latency spikes and figure out what was going on.

My team is responsible for the LBaaS product (Load Balancer as a Service) and, of course, we are the first suspects when this kind of problem appear.

Before going for the current solution, I read a lot of Envoy's documentation.

It is possible to enable access logs for Envoy, but they don't provide the information required for this debug. This is an example of the output:

[2025-12-08T20:44:49.918Z] "- - -" 0 - 78 223 1 - "-" "-" "-" "-" "172.18.0.2:8080"

I won't go into detail about the line above, since it's not possible to trace the request using access logs alone.

Envoy also has OpenTelemetry tracing, which is perfect for understanding sources of latency. Unfortunatly, it is only available for Application Load Balancers.

Most of the HTTP 499 were happening every 10 minutes, so we managed to get some of the requests with tcpdump, Wireshark and using http headers to filter the requests.

This approach helped us reproduce and track down the problem, but it wasn't a great solution. We clearly needed better tools to catch this kind of issue the next time it happened.

Therefore, I decided to try out OpenTelemetry eBPF Instrumentation, also referred to as OBI.

I saw the announcement of Grafana Beyla before it was renamed to OBI, but I didn't have the time or a strong reason to try it out until now. Even then, I really liked the idea, and the possibility of using eBPF to solve this instrumentation problem had been in the back of my mind.

OBI promises zero-code automatic instrumentation for Linux services using eBPF, so I put together a minimal setup to see how well it works.

Reproducible setup

I used the following tools:

Setting up a TCP Proxy with Envoy was straightforward:

static_resources:
  listeners:
  - name: go_server_listener
    address:
      socket_address:
        address: 0.0.0.0
        port_value: 8000
    filter_chains:
    - filters:
      - name: envoy.filters.network.tcp_proxy
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
          stat_prefix: go_server_tcp
          cluster: go_server_cluster
  clusters:
  - name: go_server_cluster
    connect_timeout: 1s
    type: LOGICAL_DNS
    load_assignment:
      cluster_name: go_server_cluster
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: target-backend
                port_value: 8080

This is the simplest Envoy TCP proxy configuration: a listener on port 8000 forwarding traffic to a backend running on port 8080.

For the backend, I used a basic Go HTTP server:

package main

import (
    "fmt"
    "net/http"
)

func main() {
    http.Handle("/", http.FileServer(http.Dir(".")))

    server := http.Server{Addr: ":8080"}

    fmt.Println("Starting server on :8080")
    panic(server.ListenAndServe())
}

Finally, I wrapped everything together with Docker Compose:

services:
  autoinstrumenter:
    image: otel/ebpf-instrument:main
    pid: "service:envoy"
    privileged: true
    environment:
      OTEL_EBPF_TRACE_PRINTER: text
      OTEL_EBPF_OPEN_PORT: 8000

  envoy:
    image: envoyproxy/envoy:v1.33-latest
    ports:
      - 8000:8000
    volumes:
      - ./envoy.yaml:/etc/envoy/envoy.yaml 
    depends_on:
      - target-backend
  
  target-backend:
    image: golang:1.22-alpine
    command: go run /app/backend.go
    volumes:
      - ./backend.go:/app/backend.go:ro
    expose:
      - 8080

OBI should output traces to the standard output similar to this when a HTTP request is made to Envoy:

2025-12-08 20:44:49.12884449 (305.572µs[305.572µs]) HTTPClient 200 GET /(/) [172.18.0.3 as envoy:36832]->[172.18.0.2 as localhost:8080] contentLen:78B responseLen:0B svc=[envoy generic] traceparent=[00-529458a2be271956134872668dc5ee47-6dba451ec8935e3e[06c7f817e6a5dae2]-01]
2025-12-08 20:44:49.12884449 (1.260901ms[366.65µs]) HTTP 200 GET /(/) [172.18.0.1 as 172.18.0.1:36282]->[172.18.0.3 as envoy:8000] contentLen:78B responseLen:223B svc=[envoy generic] traceparent=[00-529458a2be271956134872668dc5ee47-06c7f817e6a5dae2[0000000000000000]-01]

This is exactly what we needed, with zero-code. The above trace shows:

2025-12-08 20:44:49.12884449: time of the trace.
(1.260901ms[366.65µs]): total response time for the request, with the actual internal execution time of the request (not counting the request enqueuing time).
HTTP 200 GET /: protocol, response code, HTTP method, and URL path.
[172.18.0.1 as 172.18.0.1:36282]->[172.18.0.3 as envoy:8000]: source and destination host:port. The initial request originates from my machine through the gateway (172.18.0.1), hits the Envoy (172.23.0.3), the proxy then forwards it to the backend application (172.23.0.2).
contentLen:78B: HTTP Content-Length. I used curl and the default request size for it is 78B.
responseLen:223B: Size of the response body.
svc=[envoy generic]: traced service.
traceparent: ids to trace the parent request. We can see that the Envoy makes a request to the target and this request has the other one as parent.

Let's add one more Envoy to show that it's also possible to track multiple services.

  envoy1:
    image: envoyproxy/envoy:v1.33-latest
    ports:
      - 9000:9000
    volumes:
      - ./envoy1.yaml:/etc/envoy/envoy.yaml
    depends_on:
      - envoy

The new Envoy will listen on port 9000 and forward the request to the other Envoy listening on port 8000. Now we just need to change OBI open port variable to look at a range:

OTEL_EBPF_OPEN_PORT: 8000-9000

And change the pid field of the autoinstrumenter service to use the host's PID namespace inside the container:

pid: host

This is the output I got after one curl:

2025-12-09 12:28:05.12912285 (2.202041ms[1.524713ms]) HTTP 200 GET /(/) [172.19.0.1 as 172.19.0.1:59030]->[172.19.0.5 as envoy:9000] contentLen:78B responseLen:223B svc=[envoy generic] traceparent=[00-69977bee0c2964b8fe53cdd16f8a9d19-856c9f700e73bf0d[0000000000000000]-01]
2025-12-09 12:28:05.12912285 (1.389336ms[1.389336ms]) HTTPClient 200 GET /(/) [172.19.0.5 as envoy:59806]->[172.19.0.4 as localhost:8000] contentLen:78B responseLen:0B svc=[envoy generic] traceparent=[00-69977bee0c2964b8fe53cdd16f8a9d19-caa7f1ad1c68fa77[856c9f700e73bf0d]-01]
2025-12-09 12:28:05.12912285 (1.5431ms[848.574µs]) HTTP 200 GET /(/) [172.19.0.5 as 172.19.0.5:59806]->[172.19.0.4 as envoy:8000] contentLen:78B responseLen:223B svc=[envoy generic] traceparent=[00-69977bee0c2964b8fe53cdd16f8a9d19-cbca9d64d3d26b40[caa7f1ad1c68fa77]-01]
2025-12-09 12:28:05.12912285 (690.217µs[690.217µs]) HTTPClient 200 GET /(/) [172.19.0.4 as envoy:34256]->[172.19.0.3 as localhost:8080] contentLen:78B responseLen:0B svc=[envoy generic] traceparent=[00-69977bee0c2964b8fe53cdd16f8a9d19-5502f7760ed77b5b[cbca9d64d3d26b40]-01]
2025-12-09 12:28:05.12912285 (267.9µs[238.737µs]) HTTP 200 GET /(/) [172.19.0.4 as 172.19.0.4:34256]->[172.19.0.3 as backend:8080] contentLen:0B responseLen:0B svc=[backend go] traceparent=[00-69977bee0c2964b8fe53cdd16f8a9d19-ac05c7ebe26f2530[5502f7760ed77b5b]-01]

Each log line represents a span belonging to the same trace (69977bee0c2964b8fe53cdd16f8a9d19). For readability, I ordered the spans by their traceparent relationship, showing the request's path as it moves through the system: from the client-facing Envoy, through the internal Envoy hop, and finally to the Go backend. You can see both server-side (HTTP) and client-side (HTTPClient) spans at each hop, along with per-span latency, source and destination addresses, and response sizes, making it easy to pinpoint where time is spent along the request chain.

The log lines are helpful, but we need better ways to visualize the traces and the metrics generated by OBI. I'll share another setup that more closely reflects what we actually use.

Production setup

I'll be using the following tools this time:

The goal of this setup is to mirror an environment similar to what I used in production. This time, I've omitted the load balancer and shifted the emphasis to observability instead.

I will run three HTTP servers on port 8080: two inside Incus containers and one on the host machine. The OBI process will export metrics and traces to an OpenTelemetry Collector, which will forward traces to Jaeger and expose a metrics endpoint for Prometheus to scrape. Grafana will also be added to visualize the collected metrics using dashboards.

The aim of this approach is to instrument only one of the HTTP servers while ignoring the others. This simulates an environment with hundreds of Incus containers, where the objective is to debug a single container without being overwhelmed by excessive and irrelevant telemetry data from the rest of the system.

OBI can filter metrics and traces based on attribute values, but I was not able to filter by process PID. This is where the OBI Collector comes into play, it allows me to use a processor to filter telemetry data by the PID of the process being instrumented.

These are the steps to reproduce this setup:

Create the incus containers.

$ incus launch images:debian/trixie server01
Launching server01
$ incus launch images:debian/trixie server02
Launching server02

Start the HTTP server on each container.

$ apt install python3 --update -y
$ tee /etc/systemd/system/server.service > /dev/null <<'EOF'
[Unit]
Description=Python HTTP server
After=network.target

[Service]
User=root
Group=root
Type=simple
ExecStart=/usr/bin/python3 -m http.server 8080
Restart=always
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target
EOF
$ systemctl start server.service

Start the HTTP server on the host.

$ python3 -m http.server 8080

Start the Docker compose.

services:
  autoinstrumenter:
    image: otel/ebpf-instrument:main
    pid: host
    privileged: true
    environment:
      OTEL_EBPF_CONFIG_PATH: /etc/obi/obi.yml 
    volumes:
      - ./obi.yml:/etc/obi/obi.yml

  otel-collector:
    image: otel/opentelemetry-collector-contrib:0.98.0
    command: ["--config=/etc/otel-collector-config.yml"]
    volumes:
      - ./otel-collector-config.yml:/etc/otel-collector-config.yml
    ports:
      - "4318:4318" # Otel Receiver
      - "8889:8889" # Prometheus Scrape
    depends_on:
      - autoinstrumenter
      - jaeger
      - prometheus

  prometheus:
    image: prom/prometheus
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090" # Prometheus UI

  grafana:
    image: grafana/grafana
    restart: always
    environment:
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD=RandomString123!
    volumes:
      - ./grafana-ds.yml:/etc/grafana/provisioning/datasources/datasource.yml
    ports:
      - "3000:3000" # Grafana UI

  jaeger:
    image: jaegertracing/all-in-one
    container_name: jaeger
    ports:
      - "16686:16686" # Jaeger UI
      - "4317:4317"  # Jaeger OTLP/gRPC Collector

Here's what the configuration files look like:

obi.yml:

log_level: INFO
trace_printer: text

discovery:
  instrument:
    - open_ports: 8080

otel_metrics_export:
  endpoint: http://otel-collector:4318
otel_traces_export:
  endpoint: http://otel-collector:4318

prometheus.yml:

global:
  scrape_interval: 5s

scrape_configs:
  - job_name: 'otel-collector'
    static_configs:
      - targets: ['otel-collector:8889']

grafana-ds.yml:

apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true

otel-collector-config.yml:

receivers:
  otlp:
    protocols:
      http:
        endpoint: otel-collector:4318

exporters:
  otlp/jaeger:
    endpoint: jaeger:4317
    tls:
      insecure: true

  prometheus:
    endpoint: 0.0.0.0:8889
    namespace: default

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [otlp/jaeger]

    metrics:
      receivers: [otlp] 
      exporters: [prometheus]

We're almost there, the OpenTelemetry Collector is just missing a processor. To create the processor filter, we can look at the OBI logs to find the PID of the HTTP server being instrumented:

autoinstrumenter-1  | time=2025-12-30T19:57:17.593Z level=INFO msg="instrumenting process" component=discover.traceAttacher cmd=/usr/bin/python3.13 pid=297514 ino=460310 type=python service=""
autoinstrumenter-1  | time=2025-12-30T19:57:18.320Z level=INFO msg="instrumenting process" component=discover.traceAttacher cmd=/usr/bin/python3.13 pid=310288 ino=722998 type=python service=""
autoinstrumenter-1  | time=2025-12-30T19:57:18.512Z level=INFO msg="instrumenting process" component=discover.traceAttacher cmd=/usr/bin/python3.13 pid=315183 ino=2888480 type=python service=""

Which can also be obtained using standard GNU/Linux utilities:

$ cat /sys/fs/cgroup/lxc.payload.server01/system.slice/server.service/cgroup.procs 
297514
$ cat /sys/fs/cgroup/lxc.payload.server02/system.slice/server.service/cgroup.procs 
310288
$ ps -aux | grep http.server
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
1000000   297514  0.0  0.1  32120 14856 ?        Ss   16:03   0:00 /usr/bin/python3 -m http.server 8080
1000000   310288  0.0  0.1  32120 10616 ?        Ss   16:09   0:00 /usr/bin/python3 -m http.server 8080
cipriano  315183  0.0  0.1 103476 11480 pts/3    S+   16:17   0:00 python -m http.server 8080

If we search for the PID in the OpenTelemetry Collector endpoint where Prometheus metrics are exposed, we can find the attribute values to filter on.

$ curl http://localhost:8889/metrics | rg 297514
default_target_info{host_id="148f400ad3ea",host_name="148f400ad3ea",instance="148f400ad3ea:297514",job="python3.13",os_type="linux",service_instance_id="148f400ad3ea:297514",service_name="python3.13",telemetry_sdk_language="python",telemetry_sdk_name="opentelemetry-ebpf-instrumentation",telemetry_sdk_version="main"} 1

Now we just need to add the processor to the collector configuration:

processors: # <--- NEW BLOCK
  filter/host_id:
    traces:
      span:
        - 'resource.attributes["service.instance.id"] == "148f400ad3ea:297514"'

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [filter/host_id] # <--- NEW LINE
      exporters: [otlp/jaeger]

    metrics:
      receivers: [otlp] 
      processors:  # <--- NEW BLOCK
        - filter/host_id
      exporters: [prometheus]

That's it! The processor will handle the filtering for us, and we'll only see traces and metrics from the HTTP server running in the server01 container. Below are some screenshots from Jaeger and Grafana:

Closing Notes

I am still amazed at how powerful OBI can be.

For those curious about the debug, we found out that a service responsible for the network orchestration of the Envoy containers was running netplan apply every 10 minutes because of a bug. Netplan apply causes interfaces to go down temporarily and this made the latency go above 500ms which caused the 499s.

Avoiding 5XX errors by adjusting Load Balancer Idle Timeout

2025-10-10T11:56:37-03:00

Avoiding 5XX errors by adjusting Load Balancer Idle Timeout

Recently I faced a problem in production where a client was running a RabbitMQ server behind the Load Balancers we provisioned and the TCP connections were closed every minute.

My team is responsible for the LBaaS (Load Balancer as a Service) product and this Load Balancer was an Envoy proxy provisioned by our control plane.

The error was similar to this:

[2025-10-03 12:37:17,525 - pika.adapters.utils.connection_workflow - ERROR] AMQPConnector - reporting failure: AMQPConnectorSocketConnectError: timeout("TCP connection attempt timed out: ''/(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('<IP>', 5672))")
[2025-10-03 12:37:17,526 - pika.adapters.utils.connection_workflow - ERROR] AMQP connection workflow failed: AMQPConnectionWorkflowFailed: 1 exceptions in all; last exception - AMQPConnectorSocketConnectError: timeout("TCP connection attempt timed out: ''/(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('<IP>', 5672))"); first exception - None.
[2025-10-03 12:37:17,526 - pika.adapters.utils.connection_workflow - ERROR] AMQPConnectionWorkflow - reporting failure: AMQPConnectionWorkflowFailed: 1 exceptions in all; last exception - AMQPConnectorSocketConnectError: timeout("TCP connection attempt timed out: ''/(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('<IP>', 5672))"); first exception - None

At first glance, the issue is simple: the Load Balancer's idle timeout is shorter than the RabbitMQ heartbeat interval.

The idle timeout is the time at which a downstream or upstream connection will be terminated if there are no active streams. Heartbeats generate periodic network traffic to prevent idle TCP connections from closing prematurely.

Adjusting these timeout settings to align properly solved the issue.

However, what I want to explore in this post are other similar scenarios where it's not so obvious that the idle timeout is the problem. Introducing an extra network layer, such as an Envoy proxy, can introduce unpredictable behavior across your services, like intermittent 5XX errors.

To make this issue more concrete, let's look at a minimal, reproducible setup that demonstrates how adding an Envoy proxy can lead to sporadic errors.

Reproducible setup

I'll be using the following tools:

This setup is based on what Kai Burjack presented in his article.

Setting up Envoy with Docker is straightforward:

$ docker run \
    --name envoy --rm \
    --network host \
    -v $(pwd)/envoy.yaml:/etc/envoy/envoy.yaml \
    envoyproxy/envoy:v1.33-latest

I'll be running experiments with two different envoy.yaml configurations: one that uses Envoy's TCP proxy, and another that uses Envoy's HTTP connection manager.

Here's the simplest Envoy TCP proxy setup: a listener on port 8000 forwarding traffic to a backend running on port 8080.

static_resources:
  listeners:
  - name: go_server_listener
    address:
      socket_address:
        address: 0.0.0.0
        port_value: 8000
    filter_chains:
    - filters:
      - name: envoy.filters.network.tcp_proxy
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
          stat_prefix: go_server_tcp
          cluster: go_server_cluster
  clusters:
  - name: go_server_cluster
    connect_timeout: 1s
    type: static
    load_assignment:
      cluster_name: go_server_cluster
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: 127.0.0.1
                port_value: 8080

The default idle timeout if not otherwise specified is 1 hour, which is the case here.

The backend setup is simple as well:

package main

import (
    "fmt"
    "net/http"
    "time"
)

func helloHandler(w http.ResponseWriter, r *http.Request) {
  w.Write([]byte("Hello from Go!"))
}

func main() {
    http.HandleFunc("/", helloHandler)

    server := http.Server{
        Addr:        ":8080",
        IdleTimeout: 3 * time.Second,
    }

    fmt.Println("Starting server on :8080")
    panic(server.ListenAndServe())
}

The IdleTimeout is set to 3 seconds to make it easier to test.

Now, oha is the perfect tool to generate the HTTP requests for this test. The Load test is not meant to stress this setup, the idea is to wait long enough so that some requests are closed. The burst-delay feature will help with that:

$ oha -z 30s -w --burst-delay 3s --burst-rate 100 http://localhost:8000

I'm running the Load test for 30 seconds, sending 100 requests at three-second intervals. I also use the -w option to wait for ongoing requests when the duration is reached. The result looks like this:

We had 886 responses with status code 200 and 64 connections closed. The backend terminated 64 connections while the load balancer still had active requests directed to it.

Let's change the Load Balancer idle_timeout to two seconds.

filter_chains:
- filters:
  - name: envoy.filters.network.tcp_proxy
    typed_config:
      "@type": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
      stat_prefix: go_server_tcp
      cluster: go_server_cluster
      idle_timeout: 2s # <--- NEW LINE

Run the same test again.

Great! Now all the requests worked.

This is a common issue, not specific to Envoy Proxy or the setup shown earlier. Major cloud providers have all documented it.

AWS troubleshoot guide for Application Load Balancers says this:

The target closed the connection with a TCP RST or a TCP FIN while the load balancer had an outstanding request to the target. Check whether the keep-alive duration of the target is shorter than the idle timeout value of the load balancer.

Google troubleshoot guide for Application Load Balancers mention this as well:

Verify that the keepalive configuration parameter for the HTTP server software running on the backend instance is not less than the keepalive timeout of the load balancer, whose value is fixed at 10 minutes (600 seconds) and is not configurable.

The load balancer generates an HTTP 5XX response code when the connection to the backend has unexpectedly closed while sending the HTTP request or before the complete HTTP response has been received. This can happen because the keepalive configuration parameter for the web server software running on the backend instance is less than the fixed keepalive timeout of the load balancer. Ensure that the keepalive timeout configuration for HTTP server software on each backend is set to slightly greater than 10 minutes (the recommended value is 620 seconds).

RabbitMQ docs also warn about this:

Certain networking tools (HAproxy, AWS ELB) and equipment (hardware load balancers) may terminate "idle" TCP connections when there is no activity on them for a certain period of time. Most of the time it is not desirable.

Most of them are talking about Application Load Balancers and the test I did was using a Network Load Balancer. For the sake of completeness, I will do the same test but using Envoy's HTTP connection manager.

The updated envoy.yaml:

static_resources:
  listeners:
  - name: listener
    address:
      socket_address:
        address: 0.0.0.0
        port_value: 8000
    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          stat_prefix: go_server_http
          access_log:
          - name: envoy.access_loggers.stdout
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog
          http_filters:
          - name: envoy.filters.http.router
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
          route_config:
            name: http_route
            virtual_hosts:
            - name: local_service
              domains: ["*"]
              routes:
              - match:
                  prefix: "/"
                route:
                  cluster: go_server_cluster
  clusters:
  - name: go_server_cluster
    type: STATIC
    load_assignment:
      cluster_name: go_server_cluster
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: 0.0.0.0
                port_value: 8080

The yaml above is an example of a service proxying HTTP from 0.0.0.0:8000 to 0.0.0.0:8080. The only difference from a minimal configuration is that I enabled access logs.

Let's run the same tests with oha.

Even thought the success rate is 100%, the status code distribution show some responses with status code 503. This is the case where is not that obvious that the problem is related to idle timeout.

However, it's clear when we look the Envoy access logs:

[2025-10-10T13:32:26.617Z] "GET / HTTP/1.1" 503 UC 0 95 0 - "-" "oha/1.10.0" "9b1cb963-449b-41d7-b614-f851ced92c3b" "localhost:8000" "0.0.0.0:8080"

UC is the short name for UpstreamConnectionTermination. This means the upstream, which is the golang server, terminated the connection.

To fix this once again, the Load Balancer idle timeout needs to change:

  clusters:
  - name: go_server_cluster
    type: STATIC
    typed_extension_protocol_options: # <--- NEW BLOCK
      envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
        "@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
        common_http_protocol_options:
          idle_timeout: 2s # <--- NEW VALUE
        explicit_http_config:
          http_protocol_options: {}

Finally, the sporadic 503 errors are over:

To Sum Up

Here's an example of the values my team recommends to our clients:

Key Takeaways:

The Load Balancer idle timeout should be less than the backend (upstream) idle/keepalive timeout.
When we are working with long lived connections, the client (downstream) should use a keepalive smaller than the LB idle timeout.

Running Docker (OCI) Images in Incus

2025-08-11T23:18:00-03:00

Running Docker (OCI) Images in Incus

Incus 6.15 released with a lot of cool features, my favorite so far is the authentication support for OCI registries.

Here's an example:

$ incus remote add docker https://docker.io --protocol=oci
$ incus launch docker:debian:sid sid
$ incus shell sid
root@sid:~# apt update && apt upgrade -y
root@sid:~# cat /etc/os-release 
PRETTY_NAME="Debian GNU/Linux forky/sid"
NAME="Debian GNU/Linux"
VERSION_CODENAME=forky
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

This has been really useful for creating containers to test packages, much better than launching the official Debian stable Incus images and then manually changing the sources list.

Query Debian changelogs by keyword with the FTP-Master API

2025-08-03T16:52:06-03:00

Query Debian changelogs by keyword with the FTP-Master API

In my post about tracking my Debian uploads, I used the ProjectB database directly to retrieve how many uploads I had so far.

I was pleasantly surprised to receive a message from Joerg Jaspert, who introduced me to the Debian Archive Kit web API (dak), also known as the FTP-Master API.

Joerg gave the idea of integrating the query I had written into the dak API, so that anyone could obtain the same results without needing to use the mirror host, with a simple http request.

I liked the idea and I decided to work on it. The endpoint is already available and you can try by yourself by doing something like this:

$ curl https://api.ftp-master.debian.org/changelogs?search_term=almeida+cipriano

⚠️ WARNING: Check v2: https://people.debian.org/~gladk/blog/posts/202601_ftp-master-changelog-v2/

The query provides a way to search through the changelogs of all Debian packages currently published. The source code is available at Salsa.

I'm already using it to track my uploads, I made this page that updates every day. If you want to setup something similar, you can use my script and just change the search_term to the name you use in your changelog entries.

I’m running it using a systemd timer. Here’s what I’ve got:

# .config/systemd/user/track-uploads.service
[Unit]
Description=Track my uploads using the dak API
StopWhenUnneeded=yes

[Service]
Type=oneshot
WorkingDirectory=/home/cipriano/public_html/uploads
ExecStart=/usr/bin/python3 generate.py

# .config/systemd/user/track-uploads.timer
[Unit]
Description=Run track-uploads script daily

[Timer]
OnCalendar=daily
Persistent=true

[Install]
WantedBy=timers.target

After placing every file in the right place you just need to run:

$ systemctl --user daemon-reload
$ systemctl --user enable --now track-uploads.timer
$ systemctl --user start track-uploads.service # generates the html now

If you want to get a bit fancier, I’m also using an Ansible playbook for that. The source code is available on my GitLab repository.

If you want to learn more about dak, there is a web docs available.

I’d like to thank Joerg once again for suggesting the idea and for reviewing and merging the change so quickly.

Handling malicious requests with fail2ban

2025-07-28T12:36:55-03:00

Handling malicious requests with fail2ban

I've been receiving a lot of malicious requests for a while now, so I decided to try out fail2ban as a possible solution.

I see fail2ban as nice to have tool that is useful to keep down the "noise", but I wouldn't rely on it for security. If you need a tool to block unauthorized attempts or monitor log files excessively, you are probably doing something wrong.

I'm currently using fail2ban 1.0.2-2 from Debian Bookworm. Unfortunatly, I quickly ran into a problem, fail2ban doesn't work out of the box with this version:

systemd[1]: Started fail2ban.service - Fail2Ban Service.
fail2ban-server[2840]: 2025-07-28 14:40:13,450 fail2ban.configreader   [2840]: WARNING 'allowipv6' not defined in 'Definition'. Using default one: 'auto'
fail2ban-server[2840]: 2025-07-28 14:40:13,456 fail2ban                [2840]: ERROR   Failed during configuration: Have not found an y log file for sshd jail
fail2ban-server[2840]: 2025-07-28 14:40:13,456 fail2ban                [2840]: ERROR   Async configuration of server failed
systemd[1]: fail2ban.service: Main process exited, code=exited, status=255/EXCEPTION
systemd[1]: fail2ban.service: Failed with result 'exit-code'.

The good news is that this issue has already been addressed for Debian Trixie.

Since I prefer to manage my own configuration, I removed the default file at /etc/fail2ban/jail.d/defaults-debian.conf and replaced it with a custom setup. To fix the earlier issue, I also added a systemd backend to the sshd jail so it would stop expecting a logpath.

Here's the configuration I'm using:

$ cat /etc/fail2ban/jail.d/custom.conf 
[DEFAULT]
maxretry = 3
findtime = 24h
bantime  = 24h

[nginx-bad-request]
enabled  = true
port     = http,https
filter   = nginx-bad-request
logpath  = /var/log/nginx/access.log

[nginx-botsearch]
enabled  = true
port     = http,https
filter   = nginx-botsearch
logpath  = /var/log/nginx/access.log

[sshd]
enabled  = true
port     = ssh
filter   = sshd
backend  = systemd

I like to make things explicit, so I did repeat some lines from the default jail.conf file. In the end, I'm quite happy with it so far. Soon after I set it up, fail2ban was already banning a few hosts.

$ sudo fail2ban-client status nginx-bad-request
Status for the jail: nginx-bad-request
|- Filter
|  |- Currently failed: 42
|  |- Total failed: 454
`- Actions
   |- Currently banned: 12
   |- Total banned: 39

How I finally tracked my Debian uploads correctly

2025-07-05T10:28:37-03:00

How I finally tracked my Debian uploads correctly

A long time ago, I became aware of UDD (Ultimate Debian Database), which gathers various Debian data into a single SQL database.

At that time, we were trying to do something simple: list the contributions (package uploads) of our local community, Debian Brasília. We ended up with a script that counted uploads to unstable and experimental.

I was never satisfied with the final result because some uploads were always missing. Here is an example:

debci (3.0) experimental; urgency=medium
...
   [ Sergio de almeida cipriano Junior ]
   * Fix Style/GlovalVars issue
   * Rename blacklist to rejectlist
...

I made changes in debci 3.0, but the upload was done by someone else. This kind of contribution cannot be tracked by that script.

Then, a few years ago, I learned about Minechangelogs, which allows us to search through the changelogs of all Debian packages currently published.

Today, I decided to explore how this was done, since I couldn't find anything useful for that kind of query in UDD's tables.

That's when I came across ProjectB. It was my first time hearing about it. ProjectB is a database that stores all the metadata about the packages in the Debian archive, including the changelogs of those packages.

Now that I'm a Debian Developer, I have access to this database. If you also have access and want to try some queries, you can do this:

$ ssh mirror.ftp-master.debian.org -N -L 15434:danzi.debian.org:5435
$ psql postgresql://guest@localhost:15434/projectb?sslmode=allow

In the end, it finally solved my problem.

Using the code below, with UDD, I get 38 uploads:

import psycopg2

contributor = 'almeida cipriano'

try:
    connection = psycopg2.connect(
        user="udd-mirror",
        password="udd-mirror",
        host="udd-mirror.debian.net",
        port="5432",
        database="udd"
    )

    cursor = connection.cursor()

    query = f"SELECT source,version,date,distribution,signed_by_name \
FROM public.upload_history \
WHERE changed_by_name ILIKE '%{contributor}%' \
ORDER BY date;"

    cursor.execute(query)
    records = cursor.fetchall()

    print(f"I have {len(records)} uploads.")

    cursor.close()
    connection.close()

except (Exception, psycopg2.Error) as error:
    print("Error while fetching data from PostgreSQL", error)

Using the code bellow, with ProjectB, I get 43 uploads (the correct amount):

import psycopg2

contributor = 'almeida cipriano'

try:
    # SSH tunnel is required to access the database:
    # ssh <username>@mirror.ftp-master.debian.org -N -L 15434:danzi.debian.org:5435
    connection = psycopg2.connect(
        user="guest",
        host="localhost",
        port="15434",
        database="projectb",
        sslmode="allow"
    )
    connection.set_client_encoding('UTF8')

    cursor = connection.cursor()

    query = f"SELECT c.source, c.version, c.changedby \
FROM changes c \
JOIN changelogs ch ON ch.id = c.changelog_id \
WHERE c.source != 'debian-keyring' \
  AND (\
    ch.changelog ILIKE '%{contributor}%' \
    OR c.changedby ILIKE '%{contributor}%' \
  )\
ORDER BY c.seen;"

    cursor.execute(query)
    records = cursor.fetchall()

    print(f"I have {len(records)} uploads.")

    cursor.close()
    connection.close()

except (Exception, psycopg2.Error) as error:
    print("Error while fetching data from PostgreSQL", error)

It feels good to finally solve this itch I've had for years.

How I deployed this Website

2025-06-29T15:57:07-03:00

How I deployed this Website

I will describe the step-by-step process I followed to make this static website accessible on the Internet.

DNS

I bought this domain on NameCheap and am using their DNS for now, where I created these records:

Record Type	Host	Value
A	sergiocipriano.com	201.54.0.17
CNAME	www	sergiocipriano.com

Virtual Machine

I am using Magalu Cloud for hosting my VM, since employees have free credits.

Besides creating a VM with a public IP, I only needed to set up a Security Group with the following rules:

Type	Protocol	Port	Direction	CIDR
IPv4 / IPv6	TCP	80	IN	Any IP
IPv4 / IPv6	TCP	443	IN	Any IP

Firewall

The first thing I did in the VM was enabling ufw (Uncomplicated Firewall).

Enabling ufw without pre-allowing SSH is a common pitfall and can lock you out of your VM. I did this once :)

A safe way to enable ufw:

$ sudo ufw allow OpenSSH      # or: sudo ufw allow 22/tcp
$ sudo ufw allow 'Nginx Full' # or: sudo ufw allow 80,443/tcp
$ sudo ufw enable

To check if everything is ok, run:

$ sudo ufw status verbose
Status: active
Logging: on (low)
Default: deny (incoming), allow (outgoing), disabled (routed)
New profiles: skip

To                           Action      From
--                           ------      ----
22/tcp (OpenSSH)             ALLOW IN    Anywhere                  
80,443/tcp (Nginx Full)      ALLOW IN    Anywhere                  
22/tcp (OpenSSH (v6))        ALLOW IN    Anywhere (v6)             
80,443/tcp (Nginx Full (v6)) ALLOW IN    Anywhere (v6)

Reverse Proxy

I'm using Nginx as the reverse proxy. Since I use the Debian package, I just needed to add this file:

/etc/nginx/sites-enabled/sergiocipriano.com

with this content:

server {
    listen 443 ssl;      # IPv4
    listen [::]:443 ssl; # IPv6

    server_name sergiocipriano.com www.sergiocipriano.com;

    root /path/to/website/sergiocipriano.com;
    index index.html;

    location / {
        try_files $uri /index.html;
    }
}

server {
    listen 80;
    listen [::]:80;

    server_name sergiocipriano.com www.sergiocipriano.com;

    # Redirect all HTTP traffic to HTTPS
    return 301 https://$host$request_uri;
}

TLS

It's really easy to setup TLS thanks to Let's Encrypt:

$ sudo apt-get install certbot python3-certbot-nginx
$ sudo certbot install --cert-name sergiocipriano.com
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Deploying certificate
Successfully deployed certificate for sergiocipriano.com to /etc/nginx/sites-enabled/sergiocipriano.com
Successfully deployed certificate for www.sergiocipriano.com to /etc/nginx/sites-enabled/sergiocipriano.com

Certbot will edit the nginx configuration with the path to the certificate.

HTTP Security Headers

I decided to use wapiti, which is a web application vulnerability scanner, and the report found this problems:

CSP is not set
X-Frame-Options is not set
X-XSS-Protection is not set
X-Content-Type-Options is not set
Strict-Transport-Security is not set

I'll explain one by one:

The Content-Security-Policy header prevents XSS and data injection by restricting sources of scripts, images, styles, etc.
The X-Frame-Options header prevents a website from being embedded in iframes (clickjacking).
The X-XSS-Protection header is deprecated. It is recommended that CSP is used instead of XSS filtering.
The X-Content-Type-Options header stops MIME-type sniffing to prevent certain attacks.
The Strict-Transport-Security header informs browsers that the host should only be accessed using HTTPS, and that any future attempts to access it using HTTP should automatically be upgraded to HTTPS. Additionally, on future connections to the host, the browser will not allow the user to bypass secure connection errors, such as an invalid certificate. HSTS identifies a host by its domain name only.

I added this security headers inside the HTTPS and HTTP server block, outside the location block, so they apply globally to all responses. Here's how the Nginx config look like:

add_header Content-Security-Policy "default-src 'self'; style-src 'self';" always;
add_header X-Frame-Options "DENY" always;
add_header X-Content-Type-Options "nosniff" always;
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;

I added always to ensure that nginx sends the header regardless of the response code.

To add Content-Security-Policy header I had to move the css to a separate file, because browsers block inline styles under strict CSP unless you allow them explicitly. They're considered unsafe inline unless you move to a separate file and link it like this:

<link rel="stylesheet" href="./resources/header.css">

Why package X is installed in your Debian

2025-06-05T17:08:32-03:00

Why package X is installed in your Debian

When I was looking for alternatives to mount a USB flash drive without sudo, I came across udisks2. To my surprise, this package was already installed on my machine.

I found these two methods to understand why:

$ apt rdepends udisks2 --installed
udisks2
Reverse Depends:
  Depends: gnome-disk-utility (>= 2.7.6)
  Depends: gvfs-daemons
  Recommends: fwupd

$ aptitude why udisks2
i   task-gnome-desktop Depends gnome-core                
i A gnome-core         Depends gnome-disk-utility (>= 46)
i A gnome-disk-utility Depends udisks2 (>= 2.7.6)

The i marker means installed packages.

The A marker means the package was installed automatically.

Disable sleep on lid close

2025-06-04T11:15:04-03:00

Disable sleep on lid close

I am using an old laptop in my homelab, but I want to do everything from my personal computer, with ssh. The default behavior in Debian is to suspend when the laptop lid is closed, but it's easy to change that, just edit

/etc/systemd/logind.conf

and change the line

#HandleLidSwitch=suspend

HandleLidSwitch=ignore

then

$ sudo systemctl restart systemd-logind

That's it.

Why this website exists

2025-06-03T23:01:05-03:00

Why this website exists

I believe I can learn more by writing.

Besides, I have a terrible memory. This is an attempt to document what I'm learning so that I can refer to it in the future.

There are also many concepts that I don't fully understand yet, even the ones I use daily. My idea is to write about these things and explain how they work. This will force me to study them.

Lastly, I don't like relying on LinkedIn to share my resume.