Just a quick post for anyone using the TICK stack and finding Kapacitor is generating duplicate alerts.
To allow Kapacitor to get a stream of events from InfluxDB it creates a subscription for itself in InfluxDB with it’s URL i.e. http://kapacitor:9092. The subscriptions include it’s ServerID or ClusterID. This happens first time Kapacitor starts where no existing subscription is found with it’s Server/ClusterID.
What can happen is that if the configuration of Kapacitor changes, causing it to have a new ServerID or ClusterID. Then Kapacitor will create another subscription for itself with it’s new ID. Resulting in InfluxDB containing two (or more) records for the same Kapacitor URL. This can cause each event to be sent to Kapacitor multiple times.
This commonly occurs with Kapacitor under Docker, if for example you switch to another Kapacitor repo.
To check if you have multiple subscriptions in InfluxDB pointing to the same Kapacitor URL login into Chronograf and run the command “show subscriptions”
The solution is to remove the duplicate subscriptions from the InfluxDB using the steps here.
https://docs.influxdata.com/influxdb/v1.8/administration/subscription-management/#remove-subscriptions
For example
DROP SUBSCRIPTION “kapacitor-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx” ON “telegraf”.”autogen”
There have been feature requests and issues raised around this, particularly for Docker. Ideally it looks like Kapacitor removes its subscriptions in InfluxDB when it shutsdown.
https://github.com/influxdata/kapacitor/issues/870
Peter C's Tech. 