传统日志与kubernetes日志对比
- 传统服务
- 容器服务
k8s中的日志处理
常见日志采集方案：
1.远程日志 将容器中日志直接写入远程kafka、es等，再由logstash等处理。劣势就是需要改造服务由写入本地的就要进行修改 传输到远端存储。
2.sidecar模式，在每个pod中运行一个filebeat，logstash等pod共享一个valume，由采集工具将日志内容采集发送。缺点每个pod都需要额外增加一个日志采集工具，对pod有侵入。
3.logagent模式，在node中运行一个filebeat，logstash等，通过将pod日志目录固定事先约定好，直接采集对应目录内容传输至远端。优点节约资源，对pod无侵入。缺点：文件后缀名尽量统一否则维护较为困难。目录预先定义好无法判断来源于哪个pod，只能区分node。
实践：
采用logagent模式
日志采集工具使用阿里开源logpilot（优点与docker主进程进行交互能够通过docker动态识别各个pod日志目录，底层采用filebeat）
常见日志采集工具 logpilot filebeat logstash Fluentd Logtail Flume（filebeat与flentd在容器环境中应用最多）缺点：都是采集监测静态目录。
搭建es：
--- apiVersion: v1 kind: Service Metadata: name: elasticsearch-api namespace: kube-system labels: name: elasticsearch spec: selector: app: es ports: - name: transport port: 9200 protocol: TCP --- apiVersion: v1 kind: Service Metadata: name: elasticsearch-discovery namespace: kube-system labels: name: elasticsearch spec: selector: app: es ports: - name: transport port: 9300 protocol: TCP --- apiVersion: apps/v1beta1 kind: StatefulSet Metadata: name: elasticsearch namespace: kube-system labels: kubernetes.io/cluster-service: "true" spec: replicas: 3 serviceName: "elasticsearch-service" selector: matchLabels: app: es template: Metadata: labels: app: es spec: tolerations: - effect: NoSchedule key: node-role.kubernetes.io/master serviceAccountName: dashboard-admin initContainers: - name: init-sysctl image: busyBox:1.27 command: - sysctl - -w - vm.max_map_count=262144 securityContext: privileged: true containers: - name: elasticsearch image: registry.cn-hangzhou.aliyuncs.com/imooc/elasticsearch:5.5.1 ports: - containerPort: 9200 protocol: TCP - containerPort: 9300 protocol: TCP securityContext: capabilities: add: - IPC_LOCK - SYS_RESOURCE resources: limits: memory: 4000Mi requests: cpu: 100m memory: 2000Mi env: - name: "http.host" value: "0.0.0.0" - name: "network.host" value: "_eth0_" - name: "cluster.name" value: "docker-cluster" - name: "bootstrap.memory_lock" value: "false" - name: "discovery.zen.ping.unicast.hosts" value: "elasticsearch-discovery" - name: "discovery.zen.ping.unicast.hosts.resolve_timeout" value: "10s" - name: "discovery.zen.ping_timeout" value: "6s" - name: "discovery.zen.minimum_master_nodes" value: "2" - name: "discovery.zen.fd.ping_interval" value: "2s" - name: "discovery.zen.no_master_block" value: "write" - name: "gateway.expected_nodes" value: "2" - name: "gateway.expected_master_nodes" value: "1" - name: "transport.tcp.connect_timeout" value: "60s" - name: "ES_JAVA_OPTS" value: "-xms2g -Xmx2g" livenessProbe: tcpsocket: port: transport initialDelaySeconds: 20 periodSeconds: 10 volumeMounts: - name: es-data mountPath: /data terminationGracePeriodSeconds: 30 volumes: - name: es-data hostPath: path: /es-dataes.yaml
搭建logpilot：
--- apiVersion: extensions/v1beta1 kind: DaemonSet Metadata: name: log-pilot namespace: kube-system labels: k8s-app: log-pilot kubernetes.io/cluster-service: "true" spec: template: Metadata: labels: k8s-app: log-es kubernetes.io/cluster-service: "true" version: v1.22 spec: tolerations: - key: node-role.kubernetes.io/master effect: NoSchedule serviceAccountName: dashboard-admin containers: - name: log-pilot image: registry.cn-hangzhou.aliyuncs.com/imooc/log-pilot:0.9-filebeat resources: limits: memory: 200Mi requests: cpu: 100m memory: 200Mi env: - name: "FILEBEAT_OUTPUT" value: "elasticsearch" - name: "ELASTICSEARCH_HOST" value: "elasticsearch-api" - name: "ELASTICSEARCH_PORT" value: "9200" - name: "ELASTICSEARCH_USER" value: "elastic" - name: "ELASTICSEARCH_PASSWORD" value: "changeme" volumeMounts: - name: sock mountPath: /var/run/docker.sock - name: root mountPath: /host readOnly: true - name: varlib mountPath: /var/lib/filebeat - name: varlog mountPath: /var/log/filebeat securityContext: capabilities: add: - SYS_ADMIN terminationGracePeriodSeconds: 30 volumes: - name: sock hostPath: path: /var/run/docker.sock - name: root hostPath: path: / - name: varlib hostPath: path: /var/lib/filebeat type: DirectoryOrCreate - name: varlog hostPath: path: /var/log/filebeat type: DirectoryOrCreatelogpilot.yaml
搭建kibana：
--- apiVersion: v1 kind: Service Metadata: name: kibana namespace: kube-system labels: component: kibana spec: selector: component: kibana ports: - name: http port: 80 targetPort: http --- #ingress apiVersion: extensions/v1beta1 kind: Ingress Metadata: name: kibana namespace: kube-system spec: rules: - host: kibana.mooc.com http: paths: - path: / backend: serviceName: kibana servicePort: 80 --- apiVersion: apps/v1beta1 kind: Deployment Metadata: name: kibana namespace: kube-system labels: component: kibana spec: replicas: 1 selector: matchLabels: component: kibana template: Metadata: labels: component: kibana spec: containers: - name: kibana image: registry.cn-hangzhou.aliyuncs.com/imooc/kibana:5.5.1 env: - name: CLUSTER_NAME value: docker-cluster - name: ELASTICSEARCH_URL value: http://elasticsearch-api:9200/ resources: limits: cpu: 1000m requests: cpu: 100m ports: - containerPort: 5601 name: httpkibana.yaml
版权声明：本文内容由互联网用户自发贡献，该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容， 请发送邮件至 [email protected] 举报，一经查实，本站将立刻删除。
