vector logging

 

vector docs

https://vector.dev/docs/about/vector/

Vector

 

EFK 구성

log pipeline

• vector -> data-prepper -> opensearch

role

• vector:  log 수집
• data-prepper: log 가공
• opensearch: 저장 및 시각화

YAML

vector

kind: Deployment
apiVersion: apps/v1
metadata:
  name: tomcat
  labels:
    app: tomcat
spec:
  replicas: 1
  selector:
    matchLabels:
      app: tomcat
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: tomcat
        deployment: tomcat
    spec:
      volumes:
        - name: app-logs
          emptyDir: {} # 로그 볼륨은 emptyDir로 한다.
        - name: vector-config
          configMap:
            name: vector-config
            defaultMode: 420
      containers:
        - name: tomcat
          image: >-
            docker.io/mooneyred/tomcat:latest
          ports:
            - containerPort: 8080
              protocol: TCP
          resources: {}
          volumeMounts:
            - name: app-logs # vector와 공유할 로그 볼륨
              mountPath: /usr/local/tomcat/logs
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          imagePullPolicy: IfNotPresent
        - resources: {}
          terminationMessagePath: /dev/termination-log
          name: vector
          env:
            - name: VECTOR_LOG
              value: info
          imagePullPolicy: Always
          volumeMounts:
            - name: app-logs
              mountPath: /usr/local/tomcat/logs
            - name: vector-config
              readOnly: true
              mountPath: /etc/vector
          terminationMessagePolicy: File
          image: 'timberio/vector:0.43.1-alpine'
          args:
            - '--config'
            - /etc/vector/vector.toml
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 25%
      maxSurge: 25%

 

kind: ConfigMap
apiVersion: v1
metadata:
  name: vector-config
data:
  vector.toml: |
    [sources.tomcat_logs]
    type = "file"
    include = ["/usr/local/tomcat/logs/*.log"] 
    ignore_older = 86400 
    fingerprint.strategy = "checksum"

    [transforms.json_parser]
    type = "remap"
    inputs = ["tomcat_logs"]
    source = '''
    . = parse_json!(.message)
    '''

    [sinks.data-prepper]
    inputs = ["json_parser"]
    type = "http"
    encoding.codec = "json"
    uri = "http://data-prepper-headless:21891/vector/ingest"

 

data-prepper

data-prepper configmap 에서 pipelines.yaml 에 아래내용 추가

vector-pipeline:
  source:
    http:
      path: "/vector/ingest"
      port: 21891

  sink:
    - opensearch:
        hosts: ["http://opensearch:9200"]
        index: vector
        index_type: custom
        bulk_size: 200
    - stdout: