Follow up on incident yesterday evening

Resolved
Resolved

Last night there was a bad actor sending massive amounts of traffic our way that was originating from Amsterdam, Netherlands.

The sources have been blocked, and everything is running smoothly again. Not all Kiva Logic customers were affected. If you were- I'm sorry for the trouble! Some changes have been made to help in the future.

At the end of October I started deploying our new bad bot blacklist feature, which will help prevent things like from happening. Automatically.

The mode we used to block bad IP addresses is called a "managed_block". Cloudflare tries to determine what sort of block to use, and we leave the decision in their hands. HOWEVER, this method was not suitable for last night's attack, so I have updated the block mode that we use to straight "block".

Things learned:

  • Changed the mode we used to block IP addresses via the Cloudflare API
  • The alert on 'high traffic' threshold was set WAY to high, so I didn't see this incident until it was too late to prevent
  • Also I'm sorry if I confused you with the multiple updates here on the status page, I haven't used it in quite some time and created a new incident when one already existed

If you have any questions please don't hesitate to send an email over to hello@kivalogic.com. Thank you for your patience and understanding, and have a great day!

Wayne

Avatar for
Began at: