parsing k8s ingress-nginx logs with fluentbit

in a nutshell

• split up stdout and stderr using fluentbit streams
• alternate path to conf/ as it points to helm’s mounted volume
• use debug image to add command-lines at startup
• grab only ingress logs
• target matches an existing template hence will be created on-the-fly
• we have a containerd runtime hence using cri input parser

parsing input

if not using custom json log format, there is k8s-nginx-ingress parser – note the parser name has inverted nginx-ingress naming, just to make things easier grasp.

    [FILTER]
        name parser
        match kube
        #key_name log
        key_name message
        parser k8s-nginx-ingress
        reserve_data true

if you have some nodes dedicated to ingress and kyverno in da place

nodeSelector:
  app_type: ingress

tolerations:
  - key: app_type
    operator: Equal
    value: ingress
    effect: NoSchedule

if you have request field but cannot tune the nginx log format as you wish, here’s a workaround

    [FILTER]
        name parser
        match source.stdout
        key_name request
        # provides method path http_version
        parser split_request
        reserve_data true

    [FILTER]
        name parser
        match source.stdout
        key_name path
        # provides page
        parser strip_querystr
        reserve_data true
        preserve_key true

with those custom parsers

    # split-up the request field
    [PARSER]
        name split_request
        format regex
        regex ^(?<method>[^ ]*) (?<path>[^ ]*) HTTP/(?<http_version>[^ ]*)

    # help differenciate web pages - strip out the query string
    [PARSER]
        name strip_querystr
        format regex
        regex ^(?<page>[^?]*)

if there are some irrelevant logs you want to avoid

    [FILTER]
        name grep
        match source.stdout
        exclude host some.domain.tld
        exclude upstream_status 2[0-9][0-9]
        logical_op AND