HAProxy Ingress – Examples

Docs: Blue/green

Mon, 01 Jan 0001 00:00:00 +0000

This example demonstrates how to configure blue/green deployment on HAProxy Ingress controller, in order to route requests based on distict weight on deployment groups as well as selecting a group based on http header or cookie value.

Prerequisites

This document has the following prerequisite:

A Kubernetes cluster with a running HAProxy Ingress controller v0.6 or above. See the five minutes deployment or the deployment example

Deploying applications

In order to the configuration have effect, at least two deployments, or daemon sets, or replication controllers should be used with at least two pairs of label name/value.

The following instructions create two deployment objects using run label as the service selector and group label as the blue/green deployment selector:

$ kubectl run blue \
  --image=jcmoraisjr/whoami \
  --port=8000 --labels=run=bluegreen,group=blue
deployment "blue" created

$ kubectl run green \
  --image=jcmoraisjr/whoami \
  --port=8000 --labels=run=bluegreen,group=green
deployment "green" created

Certify that the pods are running and have the correct labels. Note that both group and run labels were applied:

$ kubectl get pod -lrun=bluegreen --show-labels
NAME                     READY     STATUS    RESTARTS   AGE       LABELS
blue-79c9b67d5b-5hd2r    1/1       Running   0          35s       group=blue,pod-template-hash=3575623816,run=bluegreen
green-7546d648c4-p7pmz   1/1       Running   0          28s       group=green,pod-template-hash=3102820470,run=bluegreen

Configure

Create a service that bind both deployments together using the run label. The expose command need a deployment object, take anyone, we will override it’s selector:

$ kubectl expose deploy blue --name bluegreen --selector=run=bluegreen
service "bluegreen" exposed

$ kubectl get svc bluegreen -otemplate --template '{{.spec.selector}}'
map[run:bluegreen]

Check also the endpoints, it should list both blue and green pods:

$ kubectl get ep bluegreen
NAME         ENDPOINTS                           AGE
bluegreen    172.17.0.11:8000,172.17.0.19:8000   2m

$ kubectl get pod -lrun=bluegreen -owide
NAME                     READY     STATUS    RESTARTS   AGE       IP            NODE
blue-79c9b67d5b-5hd2r    1/1       Running   0          2m        172.17.0.11   192.168.100.99
green-7546d648c4-p7pmz   1/1       Running   0          2m        172.17.0.19   192.168.100.99

Configure the ingress resource. No need to change the host below, bluegreen.example.com is fine:

$ kubectl create -f - <<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    haproxy-ingress.github.io/balance-algorithm: roundrobin
    haproxy-ingress.github.io/blue-green-deploy: group=blue=1,group=green=1
    haproxy-ingress.github.io/blue-green-mode: pod
    haproxy-ingress.github.io/ssl-redirect: "false"
  name: bluegreen
spec:
  rules:
  - host: bluegreen.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: bluegreen
            port:
              number: 8000
EOF

$ kubectl get ing
NAME        HOSTS                   ADDRESS   PORTS     AGE
bluegreen   bluegreen.example.com             80        11s

Test blue/green balance

Lets test! The following snippets use an alias hareq declared below. Change IP to your HAProxy Ingress controller IP address:

$ IP=192.168.100.99
$ alias hareq='echo Running 100 requests...; for i in `seq 1 100`; do
    curl -fsS $IP -H "Host: bluegreen.example.com" | cut -d- -f1
  done | sort | uniq -c'

BG Mode: pod
BG Balance: blue=1, green=1

Replicas: blue=1, green=1

$ hareq
Running 100 requests...
50 blue
50 green

Now changing green replicas to 3 and wait all the replicas to be running. BG Mode is pod, so the number of replicas will increase the load of the green deployment.

$ kubectl scale deploy green --replicas=3
$ kubectl get pod -w

BG Mode: pod
BG Balance: blue=1, green=1

Replicas: blue=1, green=3

$ hareq
Running 100 requests...
25 blue
75 green

Changing to deploy mode. This mode targets the balance config to the whole deployment instead of single pods.

Note: BG mode was added on v0.7. On v0.6, the only supported mode is pod.

$ kubectl annotate --overwrite ingress bluegreen \
  haproxy-ingress.github.io/blue-green-mode=deploy

BG Mode: deploy
BG Balance: blue=1, green=1

Replicas: blue=1, green=3

$ hareq
Running 100 requests...
50 blue
50 green

Changing now the balance to ¹⁄₃ blue and ²⁄₃ green:

$ kubectl annotate --overwrite ingress bluegreen \
  haproxy-ingress.github.io/blue-green-deploy=group=blue=1,group=green=2

BG Mode: deploy
BG Balance: blue=1, green=2

Replicas: blue=1, green=3

$ hareq
Running 100 requests...
33 blue
67 green

The balance will be the same despite the number of replicas:

$ kubectl scale deploy green --replicas=6
$ kubectl get pod -w

BG Mode: deploy
BG Balance: blue=1, green=2

Replicas: blue=1, green=6

$ hareq
Running 100 requests...
33 blue
67 green

Test blue/green selector

Blue/green selector requires HAProxy Ingress controller v0.9 or above.

Follow the deployment and configuration instructions to deploy the sample application.

After that, add the following annotation:

$ kubectl annotate --overwrite ingress bluegreen \
  haproxy-ingress.github.io/blue-green-header=x-server:group

Create (or update) the hareq alias. Change IP to your HAProxy Ingress controller IP address:

$ IP=192.168.100.99
$ alias hareq='echo Running 100 requests...; for i in `seq 1 100`; do
    curl -fsS $IP -H "Host: bluegreen.example.com" -H "X-Server: $GROUP" | cut -d- -f1
  done | sort | uniq -c'

Choose blue group:

$ GROUP=blue

The envvar GROUP will populate the X-Server header with the value blue.

Run the requests:

$ hareq
Running 100 requests...
 100 blue

Choose green group:

$ GROUP=blue
$ hareq
Running 100 requests...
 100 green

Choose an invalid group, the configured blue/green balance will be used:

$ kubectl annotate --overwrite ingress bluegreen \
  haproxy-ingress.github.io/blue-green-deploy=group=blue=1,group=green=3
$ GROUP=invalid
$ hareq
Running 100 requests...
  25 blue
  75 green

Docs: External haproxy

Mon, 01 Jan 0001 00:00:00 +0000

This example demonstrates how to configure HAProxy Ingress to manage an external haproxy instance deployed as a sidecar container. This approach decouple the controller and the running haproxy version, allowing the sysadmin to update any of them independently of the other.

Prerequisites

This document requires only a Kubernetes cluster. HAProxy Ingress doesn’t need to be installed, and if so, the installation process should use the Helm chart.

Configure the controller

The easiest and recommended way to configure an external haproxy is using the Helm chart with a customized values file. Create the haproxy-ingress-values.yaml file with the following content:

controller:
  config:
    syslog-endpoint: stdout
    syslog-format: raw
  haproxy:
    enabled: true
    securityContext:
      runAsUser: 0

These parameters are configuring an external haproxy and configuring haproxy to log to stdout. Also, haproxy is configured as root so it has permission to bind ports :80 and :443. By default haproxy container is started as UID 99.

A word about security

haproxy historically started as root so it has the permissions needed to bind to privileged ports, configure chroot, configure file descriptor limits, and other administrative tasks. haproxy then drops its own privileges just before starting its event loop. See Security Considerations from the documentation.

Since 2.4, haproxy container has been started as UID 99. There are a few ways to give it permissions to bind privileged port, none of them is provided by default by HAProxy Ingress Helm chart because all of them has some sort of limitation. Choose one of the options below that best suits the needs of your environment:

Configure haproxy to start as root, This is the configuration provided above, but it will not work if cluster policies deny containers running as root.
Some container runtime engines, like Docker 20.10 or newer, or Containerd embedded in k3s, reconfigure the starting of unprivileged ports so haproxy should work out of the box listening to :80 and :443 without the need to run as root. Give it a try by removing the securityContext configuration altogether:
```
controller:
  config:
    syslog-endpoint: stdout
    syslog-format: raw
  haproxy:
    enabled: true
```

Change haproxy listening port to unprivileged ports, like 8080 and 8443:

Note that, if exposing haproxy via hostNetwork, end users would need to connect to :8443 instead of the well known :443, so this is only an option if the cluster provides LoadBalancer services

controller:
  config:
    syslog-endpoint: stdout
    syslog-format: raw
    http-port: "8080"
    https-port: "8443"
  service:
    httpPorts:
    - port: 80
      targetPort: 8080
    httpsPorts:
    - port: 443
      targetPort: 8443
    type: LoadBalancer
  haproxy:
    enabled: true

Change the haproxy image by adding the NET_BIND_SERVICE capability to the haproxy binary:

FROM haproxy:X.X-alpine
USER root
RUN apk add -U libcap-utils
RUN setcap 'cap_net_bind_service=+ep' /usr/local/sbin/haproxy
USER haproxy

Reconfigure the start of unprivileged port to 80 or below using the following configuration:

This configuration does not work if hostNetwork is configured as true, and does not work on Kernel versions older than 4.11.

controller:
  config:
    syslog-endpoint: stdout
    syslog-format: raw
  haproxy:
    enabled: true
    securityContext:
      sysctls:
        name: net.ipv4.ip_unprivileged_port_start
        value: "1"

Install the controller

Add the HAProxy Ingress Helm repository if using HAProxy Ingress’ chart for the first time:

$ helm repo add haproxy-ingress https://haproxy-ingress.github.io/charts

Install or upgrade HAProxy Ingress using the haproxy-ingress-values.yaml parameters:

$ helm upgrade haproxy-ingress haproxy-ingress/haproxy-ingress\
  --install --create-namespace --namespace=ingress-controller\
  -f haproxy-ingress-values.yaml

Check if the controller successfully starts or restarts:

$ kubectl --namespace ingress-controller get pod -w

Test

Open two distinct terminals to follow haproxy-ingress and haproxy logs:

$ kubectl --namespace ingress-controller get pod
NAME                               READY   STATUS    RESTARTS   AGE
haproxy-ingress-6f8848d6fb-gxmrk   2/2     Running   0          13s

$ kubectl --namespace ingress-controller logs -f haproxy-ingress-6f8848d6fb-gxmrk -c haproxy-ingress

and

$ kubectl --namespace ingress-controller logs -f haproxy-ingress-6f8848d6fb-gxmrk -c haproxy

Do some curl to any exposed application, or just use the controller or service loadbalancer IP like the example below:

$ curl 192.168.1.11

HAProxy Ingress and the external haproxy should be logging their own events:

haproxy-ingress container:

...
I0117 17:30:27.282701       6 controller.go:87] HAProxy Ingress successfully initialized
I0117 17:30:27.282743       6 leaderelection.go:243] attempting to acquire leader lease  ingress-controller/ingress-controller-leader-haproxy...
I0117 17:30:27.335674       6 status.go:177] new leader elected: haproxy-ingress-6f8848d6fb-cxb6w
I0117 17:30:27.392372       6 controller.go:321] starting haproxy update id=1
I0117 17:30:27.392463       6 ingress.go:153] using auto generated fake certificate
I0117 17:30:27.437047       6 instance.go:309] haproxy successfully reloaded (external)
I0117 17:30:27.437217       6 controller.go:353] finish haproxy update id=1: parse_ingress=0.143483ms write_maps=0.149637ms write_config=0.971026ms reload_haproxy=43.498718ms total=44.762864ms
I0117 17:30:58.066768       6 leaderelection.go:253] successfully acquired lease ingress-controller/ingress-controller-leader-haproxy
I0117 17:30:58.066867       6 status.go:177] new leader elected: haproxy-ingress-6f8848d6fb-gxmrk

haproxy container:

...
192.168.1.10:61116 [17/Jan/2021:17:32:36.050] _front_http _error404/<lua.send-404> 0/0/0/0/0 404 190 - - LR-- 1/1/0/0/0 0/0 "GET / HTTP/1.1"

What was changed?

The sections below have details of what changed in the deployment compared with a default installation.

Sidecar

This example configures 2 (two) new containers in the controllers’ pod:

haproxy is the external haproxy deployment with two mandatory arguments: -S with the master CLI unix socket, and -f with the configuration files path
init, a Kubernetes’ initContainer used to create an initial and valid haproxy configuration.

The haproxy container references the official Alpine based image haproxy:2.3.12-alpine, but can be any other, provided that it isn’t older than the minimal required version. The minimal version for the external HAProxy can be found in the README file.

The init container just copy a minimum and valid haproxy.cfg. This file is used to properly starts haproxy and configures its master CLI that HAProxy Ingress uses to manage the instance.

A new command-line --master-socket was also added to the HAProxy Ingress container. This option enables an external haproxy instance, pointing to the unix socket path of its master CLI.

Shared filesystem

HAProxy Ingress sends configuration files to the haproxy instance using a shared filesystem. A Kubernetes’ emptyDir works well.

The following directories must be shared:

/etc/haproxy: configuration and map files - init and haproxy-ingress need write access, haproxy need read access.
/var/lib/haproxy: mostly ssl related files - haproxy-ingress need write access, haproxy need read access.
/var/run/haproxy: unix sockets - haproxy-ingress and haproxy need write access.

Liveness probe

Default HAProxy Ingress deployment has a liveness probe to an haproxy’s health check URI. This example changes the liveness probe from the HAProxy Ingress container to the haproxy one.

Docs: Metrics

Mon, 01 Jan 0001 00:00:00 +0000

This example demonstrates how to configure Prometheus and Grafana to collect and expose HAProxy and HAProxy Ingress metrics using Prometheus Operator.

Prerequisites

This document requires only a Kubernetes cluster. HAProxy Ingress doesn’t need to be installed, and if so, the installation process should use the Helm chart.

Configure Prometheus Operator

This section can be skipped if the Kubernetes cluster has already a running Prometheus Operator.

HAProxy Ingress installation configures Prometheus using a ServiceMonitor custom resource. This resource is used by Prometheus Operator to configure Prometheus instances. The following steps deploy Prometheus Operator via kube-prometheus-stack Helm chart.

Create a file named prometheus-operator-values.yaml - change both hostnames with a name that resolves to the Kubernetes cluster:

grafana:
  enabled: true
  ingress:
    enabled: true
    annotations:
      kubernetes.io/ingress.class: haproxy
    hosts:
    - grafana.192.168.0.11.nip.io
    tls:
    - hosts:
      - grafana.192.168.0.11.nip.io

Add kube-prometheus-stack helm repo:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

Install the chart:

helm install prometheus prometheus-community/kube-prometheus-stack\
  --create-namespace --namespace monitoring\
  -f prometheus-operator-values.yaml

Note

Bitnami has also a Prometheus Operator helm chart and it’s also a good option. Note however that the values file has a different syntax.

Configure HAProxy Ingress

The steps below configures HAProxy Ingress’ Helm chart to add a new ServiceMonitor custom resource. This resource will be responsible for HAProxy and HAProxy Ingress metrics scrape.

Merge the content below to the actual haproxy-ingress-values.yaml file:

controller:
  stats:
    enabled: true
  metrics:
    enabled: true
  serviceMonitor:
    enabled: true
    labels:
      release: prometheus
    metrics:
      relabelings:
      - replacement: cl1
        targetLabel: cluster
      - sourceLabels: [__meta_kubernetes_pod_node_name]
        targetLabel: hostname
    ctrlMetrics:
      relabelings:
      - replacement: cl1
        targetLabel: cluster
      - sourceLabels: [__meta_kubernetes_pod_node_name]
        targetLabel: hostname

There are two important configurations in the snippet above:

Added a label release: prometheus in the ServiceMonitor. HAProxy Ingress metrics will share the same Prometheus instance installed by Prometheus Operator. This can be changed to another dedicated instance, and must be checked if using another customized Prometheus Operator deployment.
Added relabels to HAProxy and HAProxy Ingress metrics. The HAProxy Ingress dashboard uses hostname label as a way to distinguish two controller instances, and also cluster label to distinguish controllers running on distinct clusters. The source of the name can be adjusted but the label name should be the same.

Now upgrade the chart - change upgrade to install if HAProxy Ingress isn’t installed yet:

helm upgrade haproxy-ingress haproxy-ingress/haproxy-ingress\
  --create-namespace --namespace ingress-controller\
  -f haproxy-ingress-values.yaml

Compatibility

This dashboard works with HAProxy’s internal Prometheus exporter. Follow these steps to adjust the scrape config and the dashboard if using Prometheus’ HAProxy Exporter:

Change the metric name of “Backend status / Top 5 max/avg connection time” to haproxy_backend_http_connect_time_average_seconds

Add this relabel configuration in the haproxy-ingress-values.yaml file

controller:
  ...
  serviceMonitor:
    ...
    metrics:
      ...
      metricRelabelings:
      - sourceLabels: [frontend]
        targetLabel: proxy
      - sourceLabels: [backend]
        targetLabel: proxy

Configure the dashboard

Import this Grafana dashboard. If Grafana was deployed using the steps provided in this walktrought:

Open Grafana page - the URL is the same provided in the prometheus-operator-values.yaml file and should resolve to the ingress deployment
Log in to Grafana, user is admin and the first password is prom-operator
Click the big plus + sign in the left side, Import, type 12056 as the Grafana.com ID, Load, select a Prometheus datasource, Import

If everything worked as expected, the dashboard should look like this:

Test

Lets make some noise and see what the dashboard tell us about our HAProxy Ingress cluster.

Deploy a demo application and a custom (self-signed) certificate:

openssl req -x509 -subj "/CN=dory.localdomain" -nodes -days 30 -newkey rsa:2048 -keyout /tmp/h.key -out /tmp/h.crt
kubectl --namespace default create secret tls dory --cert /tmp/h.crt --key /tmp/h.key
rm -fv /tmp/h.crt /tmp/h.key
kubectl --namespace default create deploy dory --image jcmoraisjr/dory
kubectl --namespace default scale deploy dory --replicas=4
kubectl --namespace default expose deploy dory --port 8000
kubectl --namespace default create ingress dory\
  --annotation kubernetes.io/ingress.class=haproxy\
  --annotation haproxy-ingress.github.io/ssl-redirect=false\
  --rule="dory.localdomain/*=dory:8000,tls=dory"

Check if the app is up and running:

kubectl --namespace default get pod -lapp=dory -w

Download vegeta and place it in the path.

Make a test and check if everything is working as expected. Change IP below to the IP of a HAProxy Ingress node:

IP=192.168.0.11
## Using Fish?
# set IP 192.168.0.11
echo "GET http://$IP" |\
  vegeta attack -duration=1s -rate=1 -header "Host: dory.localdomain" -keepalive=true |\
  vegeta report

The output should look like this. The most important part is Success ratio=100% and an Error Set empty:

...
Success       [ratio]                    100.00%
Status Codes  [code:count]               200:1
Error Set:

Now the real test. Adjust the duration and rate (number of requests per second) if needed. A dual core VM dedicated to HAProxy Ingress should accept a few thousands requests per second. Lets configure 200 which should move some lines in the dashoard:

IP=192.168.0.11
echo "GET http://$IP" |\
  vegeta attack -duration=5m -rate=200 -header "Host: dory.localdomain" -keepalive=true |\
  vegeta report

Follow the dashboard while the test is running. Most metrics have its resolution in 1m (one minute) so wait at least this amount of time to see the correct conns/s, rps, proc use and so on.

What’s the impact of not using keepalive? Try the same test, changing only -keepalive to false:

IP=192.168.0.11
ulimit -n 2048 # avoids 'too many open files' error in the client side
echo "GET http://$IP" |\
  vegeta attack -duration=5m -rate=200 -header "Host: dory.localdomain" -keepalive=false |\
  vegeta report

Last test: what about TLS connections without keepalive? Change http to https and add -insecure command-line option to Vegeta:

IP=192.168.0.11
ulimit -n 2048
echo "GET https://$IP" |\
  vegeta attack -insecure -duration=5m -rate=200 -header "Host: dory.localdomain" -keepalive=false |\
  vegeta report

Docs: ModSecurity

Mon, 01 Jan 0001 00:00:00 +0000

This example demonstrates how to configure ModSecurity web application firewall on HAProxy Ingress controller.

Prerequisites

This document has the following prerequisites:

A Kubernetes cluster with a running HAProxy Ingress controller. See the five minutes deployment or the deployment example
ingress-controller namespace, the default of the five minutes deployment

Deploying agent

A ModSecurity agent can be deployed in a number of ways: as a sidecar container in the same HAProxy Ingress deployment/daemonset resource, as a standalone container in the same host of ingress, or in dedicated host(s), inside or outside a k8s cluster. The steps below will deploy ModSecurity in some dedicated hosts of a k8s cluster, adjust the steps to fit your need.

The ModSecurity agent used is jcmoraisjr/modsecurity-spoa.

Create the ModSecurity agent deployment with 3 running pods:

$ kubectl create -f https://haproxy-ingress.github.io/resources/modsecurity-deployment.yaml
deployment.apps/modsecurity-spoa created

Note

This deployment configures a small amount of requests and limits resources, remember to adjust them before moving to production.

Check if the agent is up and running:

$ kubectl -n ingress-controller get deployment modsecurity-spoa
NAME                     READY     UP-TO-DATE   AVAILABLE  AGE  
modsecurity-spoa         3/3       3            3          7s

You can now create the service that provides a ClusterIP address for the HAProxy ConfigMap.

$ kubectl -n ingress-controller expose deployment modsecurity-spoa --port=12345 --type=ClusterIP
service/modsecurity-spoa exposed

Once the service is created, you can obtain the ClusterIP address to be used later in the ConfigMap.

$ kubectl -n ingress-controller get service modsecurity-spoa
NAME                     TYPE       CLUSTERIP        EXTERNAL-IP  PORT(S)     AGE
modsecurity-spoa         ClusterIP  172.20.216.246   <none>       12345/TCP   7m

Configuring HAProxy Ingress

Add the ConfigMap key modsecurity-endpoints with a comma-separated list of IP:port of the ModSecurity agent server(s). The default port number of the agent is 12345. A kubectl -n ingress-controller edit configmap haproxy-ingress should work.

Example of a ConfigMap content if the ModSecurity service has a ClusterIP of 172.20.216.246:

apiVersion: v1
data:
  modsecurity-endpoints: 172.20.216.246:12345
  ...
kind: ConfigMap

Test

Deploy any application:

$ kubectl run echo \
  --image=gcr.io/google_containers/echoserver:1.3 \
  --port=8080 \
  --expose

… and create its ingress resource. Remember to annotate waf as modsecurity. No need to use a valid domain, echo.domain below is fine:

$ kubectl create -f - <<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    haproxy-ingress.github.io/ssl-redirect: "false"
    haproxy-ingress.github.io/waf: "modsecurity"
  name: echo
spec:
  rules:
  - host: echo.domain
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: echo
            port:
              number: 8080
EOF

Test with a simple request. Change the IP below to the IP of your Ingress controller:

$ curl -I 192.168.100.99 -H 'Host: echo.domain'
HTTP/1.1 200 OK
Server: nginx/1.9.11
Date: Sun, 27 May 2018 23:28:58 GMT
Content-Type: text/plain

Test now with a malicious request:

curl -i '192.168.100.99?p=/etc/passwd' -H 'Host: echo.domain'
HTTP/1.0 403 Forbidden
Cache-Control: no-cache
Connection: close
Content-Type: text/html

<html><body><h1>403 Forbidden</h1>
Request forbidden by administrative rules.
</body></html>

Check the agent logs:

$ kubectl -n ingress-controller get pod -lrun=modsecurity-spoa
NAME                                READY   STATUS    RESTARTS   AGE
modsecurity-spoa-6f757ffd88-9qt2f   1/1     Running   0          11m
modsecurity-spoa-6f757ffd88-vwtzr   1/1     Running   0          11m
modsecurity-spoa-6f757ffd88-q4rvm   1/1     Running   0          11m
...

$ kubectl -n ingress-controller logs --tail=10 modsecurity-spoa-6f757ffd88-9qt2f
...
1527464273.942819 [00] [client 127.0.0.1] ModSecurity: Access denied with code 403 (phase 2). Matche
d phrase "etc/passwd" at ARGS:p. [file "/etc/modsecurity/owasp-modsecurity-crs/rules/REQUEST-930-APP
LICATION-ATTACK-LFI.conf"] [line "108"] [id "930120"] [rev "4"] [msg "OS File Access Attempt"] [data
 "Matched Data: etc/passwd found within ARGS:p: /etc/passwd"] [severity "CRITICAL"] [ver "OWASP_CRS/
3.0.0"] [maturity "9"] [accuracy "9"] [tag "application-multi"] [tag "language-multi"] [tag "platfor
m-multi"] [tag "attack-lfi"] [tag "OWASP_CRS/WEB_ATTACK/FILE_INJECTION"] [tag "WASCTC/WASC-33"] [tag
 "OWASP_TOP_10/A4"] [tag "PCI/6.5.4"] [hostname "ingress.localdomain"] [uri "http://echo.domain/"] [
unique_id ""]
...

Deploy ModSecurity Agent With a Sidecar Container for Audit Logs

A ModSecurity agent can be deployed with an additional sidecar container so you can have access to the logs stored in the AuditLog file. If you are using the default configuration of the ModSecurity agent, the logs written to the AuditLog specified are not reachable in the agent container’s STDOUT.

In order to read information written to that file, you must add a sidecar container to the method of deployment of the ModSecurity agent in Kubernetes. This is especially useful if you set the SecRuleEngine configuration to DetectionOnly.

Update the ModSecurity agent deployment to have a sidecar container to read the audit log file to STDOUT

$ kubectl apply -f https://haproxy-ingress.github.io/resources/modsecurity-deployment-auditlog-sidecar.yaml
deployment "modsecurity-spoa" configured

Now the ModSecurity agent pods will have two containers to get logs from: one for the traditional ModSecurity logs and one for the logs written to the AuditLog file.

$ kubectl -n ingress-controller get pod -lrun=modsecurity-spoa
NAME                                READY   STATUS    RESTARTS   AGE
modsecurity-spoa-6596c6b444-cht27   2/2     Running   0          14m
modsecurity-spoa-6596c6b444-kw2tr   2/2     Running   0          14m
modsecurity-spoa-6596c6b444-mkndw   2/2     Running   0          14m