nginx日志的loki面板

nginx-ingress-controller loki grafana dashboard：

https://grafana.com/grafana/dashboards/12559

在kubernetes的环境中安装如下：

安装loki：

helm repo add loki https://grafana.github.io/loki/charts
kubectl create ns loki
helm upgrade --install loki loki/loki-stack -n loki

配置nginx的日志格式：

kubectl edit configmaps -n kube-system nginx-configuration

内容如下：

apiVersion: v1
data:
  log-format-upstream: '{"msec": "$msec", "connection": "$connection", "connection_requests":
    "$connection_requests", "pid": "$pid","request_id": "$request_id", "request_length":
    "$request_length", "remote_addr": "$remote_addr","remote_user": "$remote_user","remote_port":
    "$remote_port","time_local": "$time_local","time_iso8601": "$time_iso8601","request":
    "$request","request_uri": "$request_uri","args": "$args","status": "$status","body_bytes_sent":
    "$body_bytes_sent","bytes_sent": "$bytes_sent","http_referer": "$http_referer","http_user_agent":
    "$http_user_agent","http_x_forwarded_for": "$http_x_forwarded_for","http_host":
    "$http_host","server_name": "$server_name", "request_time": "$request_time","upstream":
    "$upstream_addr","upstream_connect_time": "$upstream_connect_time","upstream_header_time":
    "$upstream_header_time","upstream_response_time": "$upstream_response_time","upstream_response_length":
    "$upstream_response_length","upstream_cache_status": "$upstream_cache_status","ssl_protocol":
    "$ssl_protocol","ssl_cipher": "$ssl_cipher", "scheme": "$scheme","request_method":
    "$request_method","server_protocol": "$server_protocol","pipe": "$pipe","gzip_ratio":
    "$gzip_ratio","http_cf_ray": "$http_cf_ray","geoip_country_code": "$geoip_country_code"
    }'
kind: ConfigMap
metadata:
  labels:
    app: ingress-nginx
  name: nginx-configuration
  namespace: kube-system

grafana 8.0：

安装 worldmap panel 插件：

grafana-cli plugins install grafana-worldmap-panel

重启grafana

systemctl restart grafana-server.service

地图显示不出来解决方法：

到grafana安装的机器：

cd /var/lib/grafana/plugins/grafana-worldmap-panel/

将：

https://cartodb-basemaps-{s}.global.ssl.fastly.net/light_all/{z}/{x}/{y}.png

替换成：

http://{s}.basemaps.cartocdn.com/light_all/{z}/{x}/{y}.png

将：

https://cartodb-basemaps-{s}.global.ssl.fastly.net/dark_all/{z}/{x}/{y}.png

替换成：

http://{s}.basemaps.cartocdn.com/dark_all/{z}/{x}/{y}.png

然后在grafana中可以看到地图可以正常加载。

在普通nginx中使用此面板操作如下：

首先安装loki服务：scriptjc.com/article/1277

这里用到了Geo模块：

cat > /etc/yum.repos.d/nginx.repo <<EOF
[nginx]
name=nginx repo
baseurl=http://nginx.org/packages/centos/7/$basearch/
gpgcheck=0
enabled=1
EOF

yum -y install geoip-devel # 安装Geo库
yum install nginx-r1.20.1  nginx-module-geoip # nginx和Geo模块可能

在nginx.conf中加载geo模块和引用Geo库：

# 加载模块配置放在配置文件开头位置
load_module "modules/ngx_http_geoip_module.so";
load_module "modules/ngx_stream_geoip_module.so";

http {
    geoip_country /usr/share/GeoIP/GeoIP.dat;
}

Geo nginx官方文档：

http://nginx.org/en/docs/http/ngx_http_geoip_module.html

修改nginx的日志格式：

log_format json_analytics escape=json '{'
	'"msec": "$msec", '
	'"connection": "$connection", '
	'"connection_requests": "$connection_requests", '
	'"pid": "$pid", '
	'"request_id": "$request_id", '
	'"request_length": "$request_length", '
	'"remote_addr": "$remote_addr", '
	'"remote_user": "$remote_user", '
	'"remote_port": "$remote_port", '
	'"time_local": "$time_local", '
	'"time_iso8601": "$time_iso8601", '
	'"request": "$request", '
	'"request_uri": "$request_uri", '
	'"args": "$args", '
	'"status": "$status", '
	'"body_bytes_sent": "$body_bytes_sent", '
	'"bytes_sent": "$bytes_sent", '
	'"http_referer": "$http_referer", '
	'"http_user_agent": "$http_user_agent", '
	'"http_x_forwarded_for": "$http_x_forwarded_for", '
	'"http_host": "$http_host", '
	'"server_name": "$server_name", '
	'"request_time": "$request_time", '
	'"upstream": "$upstream_addr", '
	'"upstream_connect_time": "$upstream_connect_time", '
	'"upstream_header_time": "$upstream_header_time", '
	'"upstream_response_time": "$upstream_response_time", '
	'"upstream_response_length": "$upstream_response_length", '
	'"upstream_cache_status": "$upstream_cache_status", '
	'"ssl_protocol": "$ssl_protocol", '
	'"ssl_cipher": "$ssl_cipher", '
	'"scheme": "$scheme", '
	'"request_method": "$request_method", '
	'"server_protocol": "$server_protocol", '
	'"pipe": "$pipe", '
	'"gzip_ratio": "$gzip_ratio", '
	'"http_cf_ray": "$http_cf_ray",'
	'"geoip_country_code": "$geoip_country_code"'
	'}';

    access_log /var/log/nginx/json_access.log json_analytics;

配置promtail来读取nginx日志：

server:
  http_listen_port: 9080
  grpc_listen_port: 0
 
positions:
  filename: /tmp/positions.yaml
 
clients:
  - url: http://192.168.199.20:3100/loki/api/v1/push  # loki服务端地址
 
scrape_configs:
- job_name: system
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      __path__: /var/log/nginx/*log

在grafana中添加数据源，导入面板即可。