Phoenix Protocol RP - DDOS 2: Electric Boogaloo – Incident details

All systems operational

DDOS 2: Electric Boogaloo

Resolved
Major outage
Started about 2 months agoLasted 27 days

Affected

HostHavoc

Major outage from 12:56 AM to 1:18 AM, Degraded performance from 1:18 AM to 4:13 AM, Major outage from 4:13 AM to 4:32 AM, Degraded performance from 4:32 AM to 4:52 AM, Major outage from 4:52 AM to 5:16 AM, Degraded performance from 5:16 AM to 5:38 AM, Major outage from 5:38 AM to 7:12 AM, Degraded performance from 7:12 AM to 6:08 PM, Partial outage from 6:08 PM to 8:09 PM, Degraded performance from 8:09 PM to 9:58 AM, Partial outage from 9:58 AM to 10:41 AM, Degraded performance from 10:41 AM to 10:31 PM, Operational from 10:31 PM to 2:58 AM, Degraded performance from 2:58 AM to 1:00 PM, Operational from 1:00 PM to 5:56 AM, Under maintenance from 5:56 AM to 10:13 AM, Operational from 10:13 AM to 2:33 PM

Back End Panel

Major outage from 12:56 AM to 1:18 AM, Degraded performance from 1:18 AM to 4:13 AM, Major outage from 4:13 AM to 4:32 AM, Degraded performance from 4:32 AM to 4:52 AM, Major outage from 4:52 AM to 5:16 AM, Degraded performance from 5:16 AM to 5:38 AM, Major outage from 5:38 AM to 7:12 AM, Degraded performance from 7:12 AM to 6:08 PM, Partial outage from 6:08 PM to 8:09 PM, Degraded performance from 8:09 PM to 9:58 AM, Partial outage from 9:58 AM to 10:41 AM, Degraded performance from 10:41 AM to 10:31 PM, Operational from 10:31 PM to 2:58 AM, Degraded performance from 2:58 AM to 1:00 PM, Operational from 1:00 PM to 5:56 AM, Under maintenance from 5:56 AM to 10:13 AM, Operational from 10:13 AM to 2:33 PM

Phoenix Protocol RP Main (Monitored)

Major outage from 12:56 AM to 1:18 AM, Degraded performance from 1:18 AM to 4:13 AM, Major outage from 4:13 AM to 4:32 AM, Degraded performance from 4:32 AM to 4:52 AM, Major outage from 4:52 AM to 5:16 AM, Degraded performance from 5:16 AM to 5:38 AM, Major outage from 5:38 AM to 7:12 AM, Degraded performance from 7:12 AM to 6:08 PM, Partial outage from 6:08 PM to 8:09 PM, Degraded performance from 8:09 PM to 9:58 AM, Partial outage from 9:58 AM to 10:41 AM, Degraded performance from 10:41 AM to 10:31 PM, Operational from 10:31 PM to 2:58 AM, Degraded performance from 2:58 AM to 1:00 PM, Operational from 1:00 PM to 5:56 AM, Under maintenance from 5:56 AM to 10:13 AM, Operational from 10:13 AM to 2:33 PM

Phoenix Protocol Staging (Monitored)

Major outage from 12:56 AM to 1:18 AM, Degraded performance from 1:18 AM to 4:13 AM, Major outage from 4:13 AM to 4:32 AM, Degraded performance from 4:32 AM to 4:52 AM, Major outage from 4:52 AM to 5:16 AM, Degraded performance from 5:16 AM to 5:38 AM, Major outage from 5:38 AM to 7:12 AM, Degraded performance from 7:12 AM to 6:08 PM, Partial outage from 6:08 PM to 8:09 PM, Degraded performance from 8:09 PM to 9:58 AM, Partial outage from 9:58 AM to 10:41 AM, Degraded performance from 10:41 AM to 10:31 PM, Operational from 10:31 PM to 2:58 AM, Degraded performance from 2:58 AM to 1:00 PM, Operational from 1:00 PM to 5:56 AM, Under maintenance from 5:56 AM to 10:13 AM, Operational from 10:13 AM to 2:33 PM

HoneyBadger API

Major outage from 12:56 AM to 1:18 AM, Degraded performance from 1:18 AM to 4:13 AM, Major outage from 4:13 AM to 4:32 AM, Degraded performance from 4:32 AM to 4:52 AM, Major outage from 4:52 AM to 5:16 AM, Degraded performance from 5:16 AM to 5:38 AM, Major outage from 5:38 AM to 7:12 AM, Degraded performance from 7:12 AM to 6:08 PM, Partial outage from 6:08 PM to 8:09 PM, Degraded performance from 8:09 PM to 9:58 AM, Partial outage from 9:58 AM to 10:41 AM, Degraded performance from 10:41 AM to 10:31 PM, Operational from 10:31 PM to 2:58 AM, Degraded performance from 2:58 AM to 1:00 PM, Operational from 1:00 PM to 5:56 AM, Under maintenance from 5:56 AM to 10:13 AM, Operational from 10:13 AM to 2:33 PM

Internal Gitlab

Major outage from 12:56 AM to 1:18 AM, Degraded performance from 1:18 AM to 4:13 AM, Major outage from 4:13 AM to 4:32 AM, Degraded performance from 4:32 AM to 4:52 AM, Major outage from 4:52 AM to 5:16 AM, Degraded performance from 5:16 AM to 5:38 AM, Major outage from 5:38 AM to 7:12 AM, Degraded performance from 7:12 AM to 6:08 PM, Partial outage from 6:08 PM to 8:09 PM, Degraded performance from 8:09 PM to 9:58 AM, Partial outage from 9:58 AM to 10:41 AM, Degraded performance from 10:41 AM to 10:31 PM, Operational from 10:31 PM to 2:58 AM, Degraded performance from 2:58 AM to 1:00 PM, Operational from 1:00 PM to 5:56 AM, Under maintenance from 5:56 AM to 10:13 AM, Operational from 10:13 AM to 2:33 PM

Updates
  • Postmortem
    Postmortem

    After nearly two months of a long term DDOS attacks, we believe the situation has been mitigated. HostHavoc reported this as one of the largest attacks they have encountered. The initial attack was 1.4 Terabits a second (Tbps) For simplicity, that would be the equivalent of steaming all 62 episodes of Breaking Bad in 4k at the same time 1,129 times a second. The attacks were taking their entire NYC center offline. To protect their network, they would nullroute our server. The practice of nullrouting is basically disconnecting our server from their network and shunting all that data into the void for a predetermined time.

    During this, HostHavoc and ourselves increased protections to mitigate the attack, especially once the attacks started to escalate into a cyber attack with attempted brute forcing into our servers. We currently do not believe that there was any unauthorized access into our systems and player data is safe.

    We will continue to monitor our network and work with HostHavoc and partners to continue strengthening our security.

    We do apologize for the inconvenience over the period of the incident and thank you for sticking it out with us.

  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    We are aware of several security concerns that we are addressing, including that the attacker may be monitoring our lines of communications. There is currently no threat to users. We will be limiting our updates to very generic updates and may omit updates to maintain operational security.

  • Identified
    Identified

    We are continuing to work with our service provider on the attacks.

  • Update
    Update

    We have been tracking our data along with HH and believe that our defense structure has mitigated any further attacks over the past several days. We will continue to monitor.

  • Monitoring
    Monitoring

    Null routing removed

  • Identified
    Identified

    Null routing in effect

  • Update
    Update

    Null routing removed. Periodic outages are very likely.

  • Update
    Update

    Another attack is occurring, null routing in effect.

  • Monitoring
    Monitoring

    Null routing removed. Periodic outages are very likely.

  • Identified
    Identified

    Null route in effect

  • Monitoring
    Monitoring

    HostHavoc has notified us that the FiveM ports appears to be the primary target in the attack. They are working on strengthening their infrastructure to continue mitigation of the attacks on a daily basis. Phoenix Infrastructure and HostHavoc believe we will soon be at a point where things will return back to normal operations.

  • Identified
    Identified

    Frequency of the attacks resulting in null routing is increasing. Periodic outages are very likely.

  • Monitoring
    Monitoring

    Attack mitigated. Periodic outages are still likely.

  • Identified
    Identified

    Toaster brigade is attacking again

  • Monitoring
    Monitoring

    Attack mitigated. Outages are still likely.

  • Identified
    Identified

    Attack occuring

  • Monitoring
    Monitoring

    Null routing removed. Outages are still likely.

  • Identified
    Identified

    Another attack occuring

  • Monitoring
    Monitoring

    Null routing removed. Outages are still likely.

  • Identified
    Identified

    Null routed due to a 200Gbps attack

  • Investigating
    Investigating
    We are currently investigating this incident.