'Zabbix Performance tunning (with Proxies)

We setup a distributed monitoring cluster with one Zabbix (3.4.7) Server and 8 Proxies:

Zabbix:

OS: Debian Stretch
CPU: 16*2.27GHz
RAM: 48GB
Disk: Raid1 10K (Non-SSD)


LogFile=/var/log/zabbix/zabbix_server.log
PidFile=/var/run/zabbix/zabbix_server.pid
DBName=zabbix
DBUser=zabbix
DBHost=127.0.0.1
DBPort=3307
LogFileSize=0
DBPassword=****
Timeout=4
AlertScriptsPath=/etc/zabbix/alert.d/
FpingLocation=/usr/bin/fping
LogSlowQueries=3000
Include=/etc/zabbix/zabbix_server.conf.d/*.conf
StartAlerters=10
StartPollers=80
StartPollersUnreachable=80
StartTrappers=20
StartPingers=30
StartEscalators=5
CacheSize=8G
StartDBSyncers=16
HistoryCacheSize=2048M
TrendCacheSize=256M
ValueCacheSize=10G
HistoryIndexCacheSize=2G
ExternalScripts=/etc/zabbix/alert.d/
SSHKeyLocation=/nonexistent/.ssh

Proxy:

OS: Debian Stretch
CPU: 15*2.5GHz
RAM: 6GB
Disk: Raid1 10K (Non-SSD)


Server=XXXX
Hostname=zbx-lte
LogFile=/var/log/zabbix/zabbix_proxy.log
LogFileSize=0
PidFile=/var/run/zabbix/zabbix_proxy.pid
SocketDir=/var/run/zabbix
DBName=zabbix
DBUser=zabbix
DBPassword=159753
ConfigFrequency=600
DataSenderFrequency=1
StartPollers=240
StartPollersUnreachable=80
StartTrappers=20
StartPingers=80
SNMPTrapperFile=/var/log/snmptrap/snmptrap.log
CacheSize=1G
StartDBSyncers=16
HistoryCacheSize=2048M
HistoryIndexCacheSize=2G
Timeout=6
ExternalScripts=/usr/lib/zabbix/externalscripts
FpingLocation=/usr/bin/fping
LogSlowQueries=3000

We're monitoring near 1650 nodes (snmp, icmp, agent, ssh ,external scripts and external applications) by Zabbix

Since (about ) 2 month ago , we saw so many lags in non-icmp graphs (Pic: 5.png) on specific zabbix proxy (zbx-lte in pics)

_{(source: sassan.co)}

this graph is related to a device (with snmp v2). (there are many other graph like this on this proxy) I capture network traffic of relevant zabbix proxy , For one of items , zabbix proxy sends only 16 queries instead of 60 queries (with interval 1m for 1 hour) .

_{(source: sassan.co)}