T O P
  • By - Kovis

MyNameIsGriffon

I'm beginning to think that letting the entire internet run on three guys' computers might have been a mistake.


HeliNinja

Ugh, fine, I’ll fire up my Commodore 64 and help out.


TackleballShootyhoop

Based on https://downdetector.com/, there were a lot of outages for other things that I wouldn't expect to be using AWS in any way, like Azure. It seems like it is some other issue, unless I am underestimating how widely used AWS is.


ziggy473

I think AWS is used by much more programs than you think—my digitally paid laundry machines in my apartment building were not connecting to servers both times the AWS outages happened. It seems AWS is used for much more than one would think.


TackleballShootyhoop

I get that, but Azure is Microsoft’s direct competitor to AWS, I can’t imagine they have any of their stuff hosted on there. I could be wrong, though, just seems like a bad business plan lol


dantheman91

I wouldn't be super surprised. In large companies these groups can act pretty independently, especially before Azure was more mature, AWS may have had an offering Azure didn't.


Dredly

Depends on what services and how you are getting to them. a lot of the DNS and routing services run on AWS


chillin_themost_

microsoft has their own datacenters


cr125rider

Not Azure though ;) Looks like a backbone connectivity/Level 3 issue not just AWS


zxern

Level 3 again? Lol


department_g33k

>It seems AWS is used for much more than one would think. Agreed, but Azure?


Ollep7

Zoom too. Had a hybrid event going and everything crashed 4x.


uzlonewolf

Duo MFA was taken down in this outage, so my assumption is people blamed Azure for not being able to log in.


Toad32

I highly suspect Log4J vulnerability related. We just patched out systems and the patch broke things.


jimbo831

AWS had been a mess for over a year now. Companies should start diversifying their cloud providers. I wish mine would, but our services are all built so specifically around AWS. I suspect so many other companies have this problem as well.


drmariopepper

I’m a firm believer that it’s because of how they treat their employees. The tech pool is only so deep, and Amazon is *desperate* for talent right now. It took a while to reach the tipping point, but I think it’s going to keep getting worse until they make some major cultural changes


jimbo831

Holy shit I get 2-3 recruiting emails from Amazon every single week. It’s insane how desperate they are. No amount of money would make me work there.


drmariopepper

Same, I’m a former employee that left because the culture was so toxic


DarkSkyForever

We decided to go with containerization instead of AWS native solutions and we're still patting ourselves on the back for that decision.


jimbo831

I used to work for a Fortune 100 retail company. They decided it was bad business to pay millions of dollars to their biggest competitor every year, so they spent a ton of money making all of their software not dependent on AWS. It's been great. They use Google Cloud now, but everything is built to work independent of the specific cloud provider and they could switch to Azure (or even AWS) within weeks if they wanted. More companies should do this. Good on yours.


ironichaos

Also it’s a nice negotiating tool. We will move to X cloud provider unless you lower our costs.


bigkoi

No really. Enterprises are risk averse. Why risk the business and pull resources away from money making features to save a few points? That tactic can work for the digital natives. No Enterprise is going to play ping pong with their ERP.


[deleted]

What do you mean by "AWS Native Solutions"? Wording has me thinking "serverless" vs. "distributed VM instances in an Auto Scaling group behind a load balancer" kind of difference, but I feel like that's not what you meant.


qwerty12qwerty

Think he was talking about whatever containerization tools they use (non docker)


FingolfinTEK

Could mean deploying a containerised message broker instead of using SQS, Kafka instead of Kinesis etc


cbftw

We're in the process of migrating to AWS and the plan is to use ECS instead of EC2. EC2 would be easier for a lift and shift but containers are just a better plan in the long run


gefunkt

Your plan is incomplete, you need to be thinking about your ECS exit strategy as well.


cbftw

I may be mistaken and we may be using EKS. I'm not in the decision making meetings on this.


gefunkt

EKS is a good k8s and where you should probably be heading, but if you're coming from a bare metal world I can see how ECS would look like a good intermediate step. In some ways it is. In other ways it's yet another half baked AWS minimum viable product full of hidden dependencies and inconsistent behaviors. Tell your deciders that some random guy on the internet said so.


cbftw

Our current webservers are baremetal. And it drives me mad. Virtually everything else we have is a VM with hourly, daily, weekly, and monthly snapshots.


bigkoi

ECS? Ughh.


cbftw

Please elaborate


ChinesePropagandaBot

Congratulations on paying out the ass, and still needing weeks to switch to another provider because of IAM and other dependencies.


rostasan

It's not just that, using multiple cloud partners is a good thing, but most of the failures I've looked into were due to not just AWS failing, but the lack of proper DR with the companies using AWS. For god sakes at least have a multi region failover if you have \* a business with thousands of customers. \*edit for clarity


asshatastic

Seems to be back up now


Kvothin2

Not for me unfortunately.


wobblydavid

Not working for the software I use yet.


makemejelly49

Okay, real talk. What would I need to make a serious competitor to AWS, aside from the obvious metric fucktonnes of cash?


BobDaBilda

Even more money. Money will buy or rent most of what you need; Start with a metric fuck ton of both well-qualified employees, and server hardware. The moment you started hiring, Amazon, Microsoft, and Google would all know what you were going for and send corporate spies to figure out how far along you are, and possibly to quietly sabotage you. The hardware at this point would be near impossible to get at the scale you'd need to compete. Then you'd need ways to deal with all the issues. Customer support. Hardware failures. Hardware allocation and de-allocation to clients. Compatibility issues. Worldwide availability. 24/7 availability. Real estate. Construction. A name that would make it so people would trust you with their data. HR (Remember how I mentioned corporate spies? If you let them in here, you're fucked). Lawyers (Ditto corporate spies). ISP connections (both corporate connections, and literal wire to wire and IP connections). Advertising. Corporations to use you. Better prices than the other guys while doing the same, or similar prices while doing more. Political favors (do you think Amazon won't call the mayor of wherever you're buying real estate and offer them money hand over fist? Think again. Now replace "mayor" with Governor, State Senator, State House member, US Senator, US House member, etc.). 3 types of security that I can think of, and probably more besides; 1) Physical security: Doors and Locks and Keys and ways of revoking keys if people get fired. 2) Software security: If someone can break into your servers, they can break into all your customers' servers. 3) Personal security: You're on the level of Satya Nadella / Jeff Bezos / Bill Gates / any other huge person in the industry, and going somewhere without a security detail is no longer an option. Then, after nearly infinite money, time. However many hours of work per day it'll take. 20-hour days should sound like a fun proposal for the next few years. Every problem will come to you, in one way or another. Delegation will be necessary. But will you delegate the right problems to the right people? And there are dozens more problems I won't even think of. I'm mostly thinking of tech solutions, and people solutions. But what about financial decisions, and who makes them? What happens if your bank turns off your cards for making sudden, unexpected purchases? What about lunch for your employees? Employee Benefits? Will your company support unions? Will your company do illegal things to make ends meet, or to keep from losing to your competitors? What will your company culture be like? What will the people around you expect from the company? What will they do in the company's name? Will you agree with what your company becomes? Will you stay in control of your company? You would have very little time to figure all this out and get off the ground profitably or your company would either be absorbed by a larger one or be strangled out of the market.


perplexedtriangle

Was about to come and say something similar, until I realised your answer was far more comprehensive. But I just really wanna stress this; however much money OP is thinking, double it then multiple it by 100 and double it again.


Tech_Headache

3 words... Middle out algorithm ... If you know you know. Solves all problems.


bigkoi

Ha! Funny question Mr. Ellison.


realbrownsugar

Depends on how deep you want to go down the stack? Application Platforms - Serverless - Database APIs (RDS/NoSQL/DynamoDB) Container/Kubernetes cluster management stack. VM Instance - Server hardware VPC / Software Networking Appliances - Hardware Networking Switch on Server Rack Availability Zones - Data Center Aisle + Gigabit Networking Switch Region - Data Center build out. Inter region link... Ingress/Egress data piping. Edge Networking (Akamai or Amazon Cloudfront) and Peering networks. ... could go even further all the way to controlling home networking and prioritizing traffic to stuff hosted on your network. Similar to how Netflix pays Comcast to have their content servers hosted inside their local switching centers. And all of this is outside of the application development and deployment tools that these cloud providers offer. i.e, DynamoDB / Firebase, IAM for security, Google Cloud Repos for hosting and building your code repos, etc. Best way to go about it is to think old school and build a server on a rack to host a service and start scaling it out to server your needs. Eventually you would have built a cloud infrastructure that serves your needs, and hopefully others with similar needs can pay for it. GCP is an offshoot of Google Search's serving needs. AWS is an offshoot of [Amazon.com](https://Amazon.com)'s serving needs.


MarkusBerkel

It’s good you took a shot, b/c the asker seems pretty ignorant of what hyperscale computing looks like. Even still, this list is laughable. EC2 instances can do 25gb or 40gb. So, 1/25th of an instance is gonna saturate your entire switch. Also, we’re talking about minimum of 2 60 MW data centers at each location. Plus, AWS doesn’t even buy off the shelf CPUs. They have agreements with Intel to custom make chips with different power envelopes. And I’m sure the other bigs (MS, Goog, hell, even FB) have similar arrangements. Plus, they have their own networking hardware which allows them to have multiple tenants on each machine, and still be able to bill appropriately for usage—while being secure. And, as just another example of how it’s fucking impossible to compete—at least directly—AWS also puts down its own undersea fiber optics. This isn’t some home lab. And, unlike other providers, an availability zone in AWS is not a data center aisle. It’s an entire 30-60 MW DC.


yiannistheman

Money. We can talk about everything else, and all of the stack from top to bottom and the enhancements that Google, Amazon and Microsoft bring to their cloud services are definitely worthwhile - but if you wanted to build a competitor at this point, you'd have to have a seemingly endless pile of cash to build out an enormous, geographically distributed compute presence at scale - one that would rival those three. Add in the time to get the people you need hired and ramped up before you could build it out, and coming up with a competitor in the short term (less than 5 years) is pretty much a non-starter.


old_adage

A huge team with specific expertise in multiple areas. Building and running infrastructure at this scale, with massive regulatory, privacy, and security constraints, at competitive cost requires a massive team of highly skilled individuals.


Ok_Maybe_5302

A business line with local ISP


MarkusBerkel

Make sure you get ***at least*** a T-1.


mcrissjr

A few thousand top quality developers.


InsaneVanity

Twitch.tv was effected by the outage this time.


Mccobsta

More hosts the better especially with aws going down twice recently


WhatTheZuck420

jandy assy can't keep it up.


thepwnydanza

Every time this happens Amazon’s internal systems fuck up and packages get delayed. This happened right before Christmas is gonna hurt them a tiny bit.


Hsensei

5 9s my ass


hideyshole

Since so much of the nation is ran off of those servers, why aren’t they nationalized in the name of national security?


Kovis

They’re protected by laws already. When I managed a datacenter AZ for AWS, someone cut one of our dark fiber lines. They caught the person and charged them with domestic terrorism. It’s all protected by anti-terrorism laws


hideyshole

I’m talking to make sure they stay functional, not just due to sabotage. Why are important national functions being left in the hands of for-profit corporations?