Phoenix Protocol RP - Timeouts – Incident details

HoneyBadger API experiencing degraded performance

Timeouts

Resolved
Partial outage 30 %
Started 2 months agoLasted 27 days

Affected

HostHavoc

Degraded performance from 12:00 AM to 12:42 AM, Operational from 12:00 AM to 1:52 AM, Degraded performance from 12:42 AM to 12:58 AM, Partial outage from 12:58 AM to 1:33 AM, Degraded performance from 12:58 AM to 6:52 PM, Operational from 6:52 PM to 7:19 PM, Degraded performance from 7:19 PM to 12:39 PM, Operational from 12:39 PM to 11:47 AM

Phoenix Protocol RP Main (Monitored)

Degraded performance from 12:00 AM to 12:58 AM, Partial outage from 12:58 AM to 1:33 AM, Degraded performance from 1:33 AM to 6:52 PM, Operational from 6:52 PM to 7:19 PM, Degraded performance from 7:19 PM to 12:39 PM, Operational from 12:39 PM to 11:47 AM

Phoenix Protocol Staging (Monitored)

Operational from 12:00 AM to 1:52 AM, Degraded performance from 1:52 AM to 6:52 PM, Operational from 6:52 PM to 7:19 PM, Degraded performance from 7:19 PM to 12:39 PM, Operational from 12:39 PM to 11:47 AM

Updates
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Update
    UTC
    Update

    We are continuing to monitor the fix. Some outlying users may still experience timeouts

  • Monitoring
    UTC
    Monitoring
    We implemented a fix and are currently monitoring the result.
  • Identified
    UTC
    Identified

    Still working with our host on this.

  • Monitoring
    UTC
    Monitoring
    We implemented a fix and are currently monitoring the result.
  • Update
    UTC
    Update

    We have temporarily reverted the changes while we await answers from the vendor. The server is joinable, but timeouts are likely still

  • Update
    UTC
    Update

    We are implementing some changes that may temporarily cause connection lost until complete.

  • Update
    UTC
    Update

    We are still working with HostHavoc in trying to mitigate this. Due to security policies, updates are limited.

  • Update
    UTC
    Update

    Phoenix Infra and HostHavoc have identified UDP Flooding against our FiveM services. The flooding is only causing randomly occurring timeouts. We are working on correcting this issue. Updates will be limited per our security policy.

  • Identified
    UTC
    Identified

    We are continuing to work on a fix for this incident. Updates will be limited for security purposes.

  • Investigating
    UTC
    Investigating

    We are currently investigating reoccuring timeouts