So, you are wondering what happened to this guy? Why has there is no single activity on social media for the past 8 days? Is he alive or dead because of his ridiculous (NOT reckless) car driving? Well, I’m alive and ready to drive with no destination. Yep! I’m gonna make you all jealous again by publishing selfies! So, get ready to unfollow or block me from your channels while you are stuck with the workload.
No more guesses! Let me tell you what exactly happened in the last 8 days, which is one of the most expensive lessons and real-time experience that I gained it from a stranger! Cut the crap and tell me what that is? Kind of book? PDF or hardcopy? C’mon man…! it’s a massive DDoS (distributed denial of service attack) attack against one of our dedicated servers which are hosted under Hivelocity Networks!
Layer 4 ( Syn Flood) Attack
I noticed there was a huge spike in our server CPU, RAM, and the networks and the upstream provider confirmed it’s a Layer 4 SYN flood-based attack with a much higher amount of TCP connections. We started getting DDoS alert notifications from Splunk, and their report confirmed its TCP-based SYN Flood. After talking with Hivelocity, NOC engineers started deploying the DDoS mitigation through their upstream provider! I assume they were using Corero for such emergency DDoS mitigation activities!
Splunk: Report:
Time:01/18/2017 12:12 ESTnn Host: 162.216.4.78nn Impact: 251 Mbps / 403115 pps nn Attacks: Service Flood to http at 251 Mbps / 403115 ppsnn Sflow stats: Unblocked:75 EP:75 Total:288
Even, it was an unmanaged server, Hivelocity NOC engineers were really helpful to mitigate the situation for hardening the CSF and configuring custom rules within the firewall! However, it didn’t help me to stop the situation! Hivelocity Tech support asked me to contact the sales team or my account manager in order to obtain one of their DDoS mitigation packages!
I tried to contact Hivelocity sales via email, the Support center, and Livechat! But, unfortunately, they started up selling their hardware’s and service saying The RAM is full, HD is FULL upgrading is the right choice which is absolutely a pure lie, and it won’t help in this situation.
Then I send a mail to my Hivelocity Account Manager Lee Linton; It’s confirmed he opened my email several times, but he didn’t reply either assign this to any other manager to outreach! I’m a bit disappointed about the outcome, Then I contacted Hivelocity Livechat, and the account Manager Drew Adams helped me to select the right mitigation package in order to stabilize the situation and the mitigation officially started.
Layer 3 (Network Layer) Attack
Based on Hivelocity DDoS Filtering, I thought it was under control! But my assumption is wrong; we started getting attacks from ports, mail server IPs, WHM, etc.! After deep investigation, we found it’s a layer 3 based network layer attack and checking pcap file, Again there is a huge packet loss from the server main IP! Our Server management company helped in this stage for blocking IPs, uRPFing IP’s and hardening the source guard to limit IPs on unverified layers. It started from 256 Mbps and reached 512 Mbps
The server Management Company and Hivelocity NOC engineers worked very hard to mitigate the situation, and yes they did! It was under control from 01/18/2017 12:48 EST from 01/18/2017 14:24 EST.
At this stage, our newly appointed CTO Saruban. B started collaborating with the NOC and the server management company! Saruban is a kid and very new to this industry neither me, But he is very aggressive a fast learner and a positive thinker! So, we both started investigating the root causes for this issue and we ended up without any clue and then started learning about the OSI model, DDoS mitigation strategies to prevent it in the future.
Layer 7 (Web Application layer) Attack
People who know me are aware I’m an avid traveler and even I don’t know where I will be tomorrow, especially on weekends, I will be somewhere near to the beach or hiking mountains.
Okay, Fine! I thought it’s all done and started digging booking.com to find out the best weekend deal to chill with my family! That time Saruban B said, he isn’t able to access the web and mail servers! I told him it’s a simple IP block just restart the router or check VPN; he said he tried all these basic DNS flushing activities, but it didn’t help him.
Same time, I received a mail from our server management company saying; the server is getting a Level 7 Web Application Layer (WAF) attack! I just replied to them saying “you know how to mitigate it right? Please do the needful “because I need to plan my weekend adventure! I received the reply saying this is a kind of application layer level attack and the server is not responding!
This is something weird! I just tried to access the server via IPMI and it is not responding at all and even the DC isn’t able to find it! They did a hard reboot and the server back online after 7 minutes and then found there is a massive flood in consuming the CPU and the RAM.
The server management company said nothing they can do until we get a professional DDoS mitigation plan! Seriously? Oh, then my weekend plans? Yep, it’s ruined!
The situation is now getting heated! We started getting advice from the industry experts and as per their advice decided to move this data to another server with a proper mitigation plan and we did it!
The (Simple, Yet Powerful) Action Plan
We sent a Global email to all our hosting customers who are hosted on the Layer24 server about the current situation (Didn’t hide anything) and our action plan. We already have few servers from multiple providers like SingleHop, Softlayer, Rackspace and Limestonenetworks, and OVH. We started collecting quotes from them and selected one. Data migration had been started, but the server was responding 600 times slower! This means 1GB CPmove is taking 2 hours to move another Cpanel server! So, we decided to move based on the priority and used different types of migration strategies, and we made it. Collected quotes from major DDoS mitigation providers Like Incapsula, Arbor Networks, Cloudflare Enterprise and shortlisted one and negotiated with them about the features and protocols. Now, all our dedicated servers are under one of the world’s Top Rated DDoS protection company surveillance.
When there is a Layer 7,4 Attack in place; your firewall, load balancer, and the fancy routers will not help you! It doesn’t matter how techie you are or you are a doctorate in Cyber Security or ethical hacking! The only way to resolve this situation is to find out the best DDoS mitigation provider and let them handle the situation! If you choose the cheap one, you will get what you paid!
Privilegeserver Technologies isn’t a typical shared hosting, nor it’s not for anyone! I started this company because there is a need for my Digital marketing clients that’s “High performance” and optimize and manage their accounts by ourselves while they focus on growing their business. As per my business plan, this year I decided to focus on Extreme SEO business activities to avoid distractions.
Conclusion
Within the last 6 days, there is NO sleep, NO Trips, NO meetings at all! But all I learned is one of the most expensive and precious lessons, which I “CAN’T” afford or cannot learn from the universities! Yes, that’s a real-time experience! We didn’t give UP, We have mitigated the situation, maybe it will happen again, and the server may rarely go offline! But, I will find a way to mitigate the solution rather than giving up!!! Because I’m still alive and breathing!
Now It’s Your Turn, That’s all about our DDoS attack nightmare and mitigation strategies that we applied. And now I’d like to hear from you:
- Do you have any questions about this post or any better suggestions?
- Maybe you have a great tip that I didn’t emphasize here.
- Either way, let me know by leaving a comment below right now.
