节点日志无法被抓取
问题描述:
某个节点的容器日志无法被正常抓取;
环境信息:
Kubernetes v1.22.15, Promtail 5.0.0, Loki 2.5.0, Grafana 8.5.3
原因分析:
Bug: Promtail log is reporting error for non-existent symlink · Issue #2513 · grafana/loki
promtail logs no such file or directory · Issue #429 · grafana/loki
解决方案:
重启该节点的 Promtail 服务;
… HTTP status 429 Too Many Requests …
how can I increase the maximum active stream limit? · Issue #3335 · grafana/loki · GitHub
问题描述:
level=warn ts=2022-11-10T05:10:49.533015596Z caller=client.go:349 component=client host=loki-loki-distributed-gateway.loki.svc.cluster.local msg="error sending batch, will retry" status=429 error="server returned HTTP status 429 Too Many Requests (429): Maximum active stream limit exceeded, reduce the number of active streams (reduce labels or reduce label values), or contact your Loki administrator to see if the limit can be increased"
解决方案:
修改 limits_config.max_streams_per_user: 0 配置;