Teamwork Desk 2 US region emergency maintenance

Incident Report for Teamwork.com

Postmortem

Shortly after at 2:15pm GMT on Friday the 29th of November, A Teamwork Desk database server in our US region began to reboot itself over and over.

Our Desk team and Sysops team worked on the database and from reviewing error logs found a single table was corrupt. When the app attempted to read from the table, it caused the database server to reboot.

The table in question is used by our Helpdocs product to record basic Helpdoc page view numbers for owners to see in their Reports.

The damaged table was recreated to solve the problem and backups were used to retore data in to the table.

The damaged table was specific to 1 database. The table was recreated and data is being restored.

Posted Dec 02, 2019 - 10:59 UTC

Resolved

This issue is now resolved, the database changes have been stable for a number of hours.

Posted Nov 30, 2019 - 15:25 UTC

Monitoring

We have identified a corrupt table causing the instability in one of our databases. This has been resolved and we are monitoring the database for any further issues.

Posted Nov 30, 2019 - 11:42 UTC

Investigating

We are currently replacing our Teamwork Desk 2 US region database due to recurring issues with the database. There will be some momentary disruption to a subset of customers while we perform this maintenance.

Posted Nov 29, 2019 - 18:01 UTC

This incident affected: Teamwork Desk (Teamwork Desk - US Region).