Google Cloud outage affects almost all banking and other operations in Europe (neowin.net)

290 pointsby paweladamczuk12 hours ago148 comments

nickcw 11 hours ago

From Google's status page https://status.cloud.google.com/incidents/Xs2XTHzAVC9YnafnPy...

    We are experiencing an issue with multiple services in
    europe-west2 beginning on Monday, 2026-10-19 23:08 PDT.
    
    Due to a power event we had suffered an outage within
    europe-west2-b zone. The power event is mitigated. Partial
    mitigation to infrastructure services will be observed within the
    next 30 minutes and the remaining services will be mitigated there
    after.

So some kind of power failure happened to bring down one zone. Unfortunate, but this kind of stuff happens. That is why all the cloud providers have multiple zones, and I'm amazed the big companies reported as down weren't using multiple zones.

(Also reporting the times in PDT is quite unhelpful for users of this zone who are likely based in the UK or EU)

robinson-wall 10 hours ago

> I'm amazed the big companies reported as down weren't using multiple zones

Much as with backups - if you don't test them, they don't exist.

Everywhere I've worked has had multi-AZ infrastructure. I have never worked anywhere where the failure of an AZ has been simulated regularly, with second-order effects mapped out.

forgotaccount3 9 hours ago

We set up our applications and services to be multi-AZ natively. It's not particularly difficult with how many AWS services are regional or global (Admittedly, I still expect a potential DNS failure to cause issues if and when that occurs). However we are just part of a workflow, we integrate with other applications that are not multi-AZ and would be impacted. That's just out of our hands.

It's like our on-prem systems were ordered to plan out and exercise DR plans to account for a data center going offline, but we can't exercise a DR plan fully if part of the plan is 'Adapt to how application X, which is hosted in the same datacenter, also chose to fail over.' We do not own application X. There is no appetite in the business to co-ordinate the DR exercise across the entire company at the same time.

We can't even do it all in a lower environment because there are dependencies on systems that are shared across environments. Some massive, redundant filesystem that is sync'd across our data centers, but the sync is a boolean. One datacenter instance of it is the primary and the others are read-only until they get promoted. We have one of these and lower environments are just a different root folder on the system with different access patterns, but they can't make DataCenter A the primary in prod and DataCenter B the primary in UAT.

epistasis 3 hours ago

Question who's never relied on this: how do you test whether your multi-AZ setup can withstand an AZ dropping? It sounds like your third party apps make it kind of pointless to test anyway, but I'm curious!

stackskipton 2 hours ago

Ops type here, generally you have to fail an AZ and see what happens. If you have Dev/Staging environment that is exactly like production with plenty of tests, you can use that. I've seen only one environment actually like that though.

Since most companies would have to do it in production and they don't want to have customer impacting outage, they tend to do it during an actual outage.

epistasis an hour ago

Thans! So do you have a staging environment for everything, then cut off the network or kill the instances, Aurora DB, etc.? Simulate some load while its going on, and then check for data loss? Then if that works try it in prod?

It seems like a lot of work, and it seems that as architectures evolve it would be hard to ensure that the multi-AZ failover works well.

solyin 32 minutes ago

Yes, which means it's expensive which means companies don't usually pay for it so we should name and shame companies who aren't willing to pay that expense just to pad their bottom line.

bsuvc 10 hours ago

I'm not saying it's the case here, but sometimes the cloud providers themselves have inter-zone dependencies, or even inter-region dependencies that can cause outages in other zones and regions.

For example, AWS us-east-1 outages have caused global outages on multiple occasions, such as https://www.networkworld.com/article/4168878/aws-hit-by-us-e...

yegle 7 hours ago

In Google Cloud, that's a huge no no and you'll be put on a naughty list facing serious scrutiny during the postmortem review.

Disclaimer: I work for Google Cloud.

whoamii 7 hours ago

They don’t do periodic BCDR testing?!

shadowgovt 4 hours ago

Which 'they?'

If you mean Google: Yes, extensive. One of the techniques they use is DiRT (Disaster Recovery Testing) which is a scheduled time-frame in which Google purposefully breaks systems that are exceeding their SLAs to verify that automated recovery executes as intended. In general, a DiRT test would catch something like "This service has a key dependency on us-east-1."

Now, in the past (also a Xoogler)... Some pieces of Cloud were moving at a cadence that they got away with a bit of a relaxed DiRT awhile back, so if someone tells me "I remember when a us-east-1 outage took down a key Cloud service" I would believe it. DiRT is also planned testing, it's not slapping a chaos monkey on the cluster fabric and crossing one's fingers, so it's entirely possible for key outage modalities to get missed (engineers are human and even though Google aims for a "no haunted closets" engineering practice, people develop a sixth sense of where the weak parts are of your infra and it takes courage to commit to pushing on those until they break).

solyin an hour ago

Google SRE's official (?) motto is "hope is not a strategy".

VirusNewbie 4 hours ago

Yes that's a good strategy if someone wants to get some face time with TK.

JackSlateur 3 hours ago

Could you explain how other regions were impacted today ?

Aperocky 2 hours ago

Maybe OP is on naughty list facing serious scrutiny during the postmortem review.

Eridrus 16 minutes ago

Ok, but why were all these higher level services that are not zonal experiencing issues then?

matt-p 14 minutes ago

And yet I had stuff in other zones that was affected.

nickdothutton 7 hours ago

Worked at a major ISP in the 90s with many critical infra customers. True diversity is a lot harder than many imagine.

Anon1096 5 hours ago

People who don't work on cloud services are usually very sloppy with their wording which I think is leading to some disconnect here. A "zone" is not really the unit of availability for most things people consider the "cloud" for hyperscalers. A zone is like 1-3 DCs and you are very vulnerable to stuff like localized power events. For that reason most services are not offered at the zonal level and if they are you get big disclaimers saying so and lower 9s of SLA.

Most things are at a regional level which is multiple zones. Your AWS us-east-1 is one such region. A lot of datacenters spread across a decently large physical area have to have an issue for a whole region to start having failures.

For GCP it seems like one power event in a DC led to a thundering herd on the rest of the region which was the actual issue here. (which is a pretty significant problem and isn't supposed to happen)

bushbaba 2 hours ago

Thundering heards during zonal outage is fairly common across the providers. You are substantially better off being multi-regional than multi-zonal.

giancarlostoro an hour ago

This is why Microsoft calls it an Availability Zone, each zone is made up of one or more data centers equipped with entirely independent power, cooling, and networking infrastructure. One zone should not knock out other zones. Zones are also often nearby each other just enough that they have very efficient fiber optic connections between them.

That said, that's how Microsoft handles it, I'm not sure how GCP compares.

It is an interesting conundrum though, what if all your nearby availability zones come down at once due to power loss? I wonder which major cloud provider has the best setup for these sort of events.

john_strinlai an hour ago

>It is an interesting conundrum though, what if all your nearby availability zones come down at once due to power loss? I wonder which major cloud provider has the best setup for these sort of events

im only familiar with microsoft in this sense, but they have "geo-zone-redundant storage" which copies data across 3+ availability zones in the primary region, and also a secondary geographic region (i think at least 100 miles away if i remember, but maybe further). any incident would have to be massive to affect multiple geo-zones

tsimionescu an hour ago

You are generally right, except that you don't seem to know the history of major AWS outages. Failures in AWS in us-east-1, which is the first AWS cloud region, often cause cascading worldwide failures in the whole of AWS - especially failures related to DNS and to DynamoDB, if I remember correctly. That is, AWS workloada running in, say, eu-west-1 (and in fact in the whole of AWS) often start failing or at least having greatly degraded performance/availability because of a failure in AWS infra in their core region.

If AWS themselves can't handle cross region isolation and availability well, my expectations for most companies running on top of AWS are quite modest.

JohnMakin 17 minutes ago

It's because AWS runs their control plane out of us-east-1, particularly lambda. There are also some very popular services that can only be run out of us-east-1. It's not really a technological constraint, it's a design decision.

cavoirom 10 hours ago

I know some banks out-sourced most of its infrastructure to external operators, and some operator I worked with didn't seem competent.

HelloUsername 10 hours ago

> https://status.cloud.google.com/incidents/Xs2XTHzAVC9YnafnPy...

https://news.ycombinator.com/item?id=50163858

crumby 9 hours ago

Often there are irritating legal geographic limitations on where cloud hardware can exist for a given country. I can understand why, but there is this particular drawback of having all eggs baskets close together.

chrisjj 9 hours ago

The drawback being temptation not to power them indepedently, it seems.

td521534f12 9 hours ago

GCP is kind of odd in that some hardware is only available in certain zones, and some hardware is technically available in multiple zones but the capacity is not evenly spread. I've had to implement infrastructure code for GCP that interrogated hardware availability by region and zone to place VMs, trading off multi-AZ availability where it made sense.

xyzzy_plugh 8 hours ago

Yes. The area on the Venn diagram where services, hardware and regulatory compliance intersect is surprisingly small.

I had a client that was contracting with a very large bank and there were only something like four zones globally that we could possibly deploy in to.

Google also is unlike AWS in that their zone abstraction is very virtual, and two zones might be in the same physical building. If you have requirements that your deployment zones be physically separated by some minimal distance then it can make things pretty challenging.

ragall 7 hours ago

> infrastructure code for GCP that interrogated hardware availability

How do you to do that ?

td521534f12 6 hours ago

There's a GCP API that lists the hardware advertised in each zone as a starting point. Then you have to actually attempt to provision hardware in each zone to try to get a sense of what capacity is available there. Combine the two and you can kind of map out a topology.

semi-extrinsic 3 hours ago

Other clouds are similarly odd (or even worse). I have experienced in Azure that if you provision a resource group with some basic SKU for a given service, the only way to later upgrade to the latest and greatest SKU is to tear down the whole resource group and re-provision everything from scratch. All attempts at scale-up in place will fail with cryptic errors. Seems like you are assigned in an old corner of the DC and can't get out of it.

Extra fun if there is a key vault involved, then you have to manually delete every key in it, otherwise you can't create new keys with the same names in the new vault.

vrosas 2 hours ago

I can't even get a managed redis instance in Azure EUS2 right now...

physhster 8 hours ago

Google is PDT-centric, and never grew out of it. Everything is Mountain View time internally and it's so jarring.

jonathanlydall 6 hours ago

It comes across as really uncaring to the probably 99.9999% of people outside of the US who have no idea offhand what UTC offset it is, it almost feels hostile.

If they at least included “(UTC -7)” after PDT, while still a bit annoying, at least people could work out what time that is without having to lookup PDT from somewhere first (as I just had to).

shadowgovt 4 hours ago

Shrug. They're big enough to set a standard (much as we have GMT because France and the UK had enough political / economic / cultural gravity in the era of the steamboat to make the world care what a "Greenwich" was).

We just keep a clock on the wall set to Google time, like they do in newsrooms and NASA mission control. It's convenient enough.

oasisbob 2 hours ago

The annoying thing is that everyone thinks they're big enough to force their own TZ.

Southwest airlines does it.

Rackspace used to do it, to the point of needing to ask them pretty please stop provisioning hardware on central time. Obnoxious.

nayuki 2 hours ago

I'm impressed that Stack Overflow / Stack Exchange (presumably US-based) uses UTC for things like resetting daily vote limits.

gaiagraphia 2 hours ago

And tbh, Florence Mean Time would've made more sense in lining up the world on the map.

solyin an hour ago

On Mercator map projection, sure, but then we'd have to address that tyranny as well.

jlarocco an hour ago

> It's convenient enough.

It doesn't sound like it.

shadowgovt an hour ago

You get used to it. It's exactly as convenient as having all my logs in GMT and then having to do the math to convert from those numbers to when people on the team were actually awake / asleep / at lunch, especially since my team is smeared across a half-dozen timezones too.

InsidiousWombat 8 hours ago

The fact they didn't immediately fail over doesn't mean they didn't had a plan.

The plan might've just been "if it gets bombed, we restore backups into other zone". Just waiting for it to fix itself for few hours mightv'e been acceptable risk.

spwa4 2 hours ago

And what about the fact this give the US government?

1) the ability to view everyone's bank data, in secret

2) the ability to change EU bank data at will, in secret

(see US Cloud act)

There are much worse things you can do to data than bomb it.

TacticalCoder 3 hours ago

> ... I'm amazed the big companies reported as down weren't using multiple zones.

Yeah well, they drank the cloud kool-aid. The promise was that they'd go "cloud", and that everything would be taken care of.

Turns out: availability is worse than when those companies used to self-host and major cloud outages are happening all the time (at this point it's basically a meme).

And now the promise that "the cloud is simpler than doing it yourself" turns out to be not true?

I'm not surprised at all: these companies have been sold a lie.

Betelbuddy an hour ago

>> That is why all the cloud providers have multiple zones....and I'm amazed the big companies reported as down weren't using multiple zones.

Well then you should know that a Google Zone is not the same as an AWS AZ and many of Google regions are just firewalls across the same data center.... Despite all their SREs and Books and Phds...they dont know how to do infrastructure.

Betelbuddy an hour ago

Dont downvote...read their docs....and this is just the start:

"The regions Stockholm, Osaka, and Montreal have three zones housed in one or two physical data centers." - https://docs.cloud.google.com/docs/geography-and-regions

"This means the multiple zones within each of these regions may not be located in physically separate data center campuses." - https://docs.cloud.google.com/backup-disaster-recovery/docs/...

"A region is a set of buildings operated by Google in a geographic location, such as a data center campus." - https://cloud.google.com/vpc/network-pricing

gadders 11 hours ago

>>Google cloud outage takes down HSBC, Revolut, Bank of Scotland, Llyods, Equifax, more

You will probably have issues logging into Llyods even when the cloud is back up.

michaellee8 10 hours ago

cannot agree more

venzaspa 10 hours ago

It's crazy. My main account is with Starling and I have a Lloyds account for a saving an account and some free cinema tickets. I don't think I've ever had an issue with Starling, it's great - Lloyds fails to sign in about 50% of the time consistently.

cowlevel 10 hours ago

Do you have an account with Llyods though?

fauigerzigerk 9 hours ago

Strange. My Lloyds login has never failed since I opened the account 11 years ago.

The usability of their mobile app is atrocious though. Good thing they have a website.

otar 2 hours ago

The EU should really prioritise building AWS and Google Cloud alternatives with European values.

> ssh root@eu-cloud

  We value your privacy. We and our 847 partners track your network packets.

  [Accept all]  [Manage preferences]

mawadev 2 hours ago

Yeah but i wish most of those weren't glorified OpenStack wrappers.

thibaut_barrere 2 hours ago

This is being done while we speak here.

Happily hosting a site that will cross 4 billion queries per year there, and I didn’t face that banner you mention :-)

paweladamczuk 12 hours ago

For me it was Revolut. No access to my main account, no withdrawals, no card payments since the morning. App doesn't work, there's no phone number to call, nobody to talk to. Dark side of neobanking, I guess.

Arbortheus 11 hours ago

Not all neobanks. I use Monzo, and they had an outage a while back. I was impressed that it barely affected their functionality.

They have a stand-in system that uses a separate codebase and separate cloud provider. They have a blog post about it here: https://monzo.com/blog/tolerating-full-cloud-outages-with-mo...

The stand-in system has a limited subset of features, i.e. no bells and whistles, but enough that you can get any actual banking done.

(Not an ad / I have no connection to Monzo, I just had some appreciation for a bank doing this properly as normally banking IT is a disaster. Plus the idea of keeping a separate codebase for almost an entire bank's functionality seems very invested.)

I have always got bad vibes from Revolut...

ndsipa_pomu 11 hours ago

> I have always got bad vibes from Revolut...

I opened up a Revolut account a couple of years ago with the intention of switching my current account to it (i.e. wages going into it etc), but I got spooked after hearing that they weren't registered as a UK bank and thus subject to the various regulations and protections. I tried using Chase instead and have been happy with it so far.

misstuned 11 hours ago

I used to use Starling as my main and they were fine, until I started getting issues with random card declines due to "fraud prevention" and nobody to talk to. I switched to First Direct and like the peace of mind of having a human on the phone if needed (when I wanted to make a huge transfer for my house deposit it was nice to have!) and I still get perks like free foreign card transactions.

altmanaltman 10 hours ago

https://www.revolut.com/blog/post/revolut-is-officially-a-ba...

They are reigistered now

ndsipa_pomu 9 hours ago

Yep, I probably won't bother switching to them now as Chase provides a good service as far as I know, though I do get a bit twitchy about its U.S. links. However, it's headquartered in the UK and is a division of J P Morgan Europe, so I hope that provides some level of distancing from the inevitable economic collapse of U.S. A.I. companies.

nickyvdicarlo an hour ago

For what it's worth, JP Morgan specifically is probably the best positioned of all the major banks to endure any political and/or economic instability that might arise in the near-ish future. Not that this should necessarily drive your decision on who to bank with, but having spent the last decade or so working at various large US banks I have a lot of respect for the way JPMC operates

- The vast majority of their infra (at least as of ~3 years ago) is self-hosted in data centers that they own, and a lot of their other key cloud infra is/was multi-cloud (i.e. operations could fail over to GCP in the event of an AWS outage)

- Financially they have the healthiest balance sheet of any non-state-owned bank (like, in the entire world). Over the last ~20 years JPMC has acted as a sort of lender of last resort several times, most recently when they bought out First Republic.

- Politically, after navigating the aforementioned instability Jamie Dimon has almost carte blanche support from the shareholders, and everyone knows it. As a result, he can afford to be a lot less... compliant with the current US regime than other major players have been recently.

m11a 10 hours ago

Starling also has a disaster recovery system across cloud providers, although it is implemented differently to Monzo's (I used to work there on multi-cloud).

Multi-cloud is really tough, tbh. It's easier said than done, especially if you care about data consistency at all.

rancar2 9 hours ago

I have built these system for financial institutions since before cloud providers. It is hard/impossible if not well constrained at a level that supports data availability and consistency well (ie using a proper database with the correct setup across all locations with the underlying OS configuration/patches applied uniformly - the latter lesson cost IBM a lot of money due to system downtime for a financial institution that I brought back online). Many software vendors offering on-prem versions don’t know how this works well exactly, so I’ve had to help them along the journey and get them to sign-off for their contractual obligations to be upheld in the setup.

dgrcode 7 hours ago

> the latter lesson cost IBM a lot of money due to system downtime for a financial institution that I brought back online

It sounds like that story could make a great blog post

fumblebee 6 hours ago

> I have always got bad vibes from Revolut...

Help me understand, is the bad vibes you get from stories from the company culture? Or rather that the media put a target on their back (at at least in my opinion this is because they have a Russian born founder? Revolut is one of the best products I've ever used, I've honestly nothing but good things to say about them.

Doohickey-d 4 hours ago

Its a great product when it works. When it doesn't, for example if they decide to freeze your money for no reason, it's not fun anymore. And there's plenty of reports of that happening to people.

chrisjj 9 hours ago

> there's no phone number to call

Perhaps this "bank" knows its phones would be down too.

rpozarickij 8 hours ago

I couldn't pay for groceries with Revolut in a store. Staff asked if I was using Revolut, but luckily adding a card from a different bank to Apple Pay was quick and painless (I had neither my wallet nor cash with me).

Another reminder that we need redundancy for all essential services that we rely on.

TedDoesntTalk 8 hours ago

cash is a more reliable redundancy.

bcvvvy 11 hours ago

I bank with HSBC. Everything is up and working.

A lot of the information people are leveraging here comes from Downdetector which is mostly users reporting that stuff is down, not whether they are actually down. It's a really bad source.

m11a 11 hours ago

It's a pretty good source ime (at least, massive spikes tend to correlate well with issues). Traditional organisations aren't as transparent as SF tech companies about service downtime. Plus, if the service isn't working for its users, I wouldn't say it's actually up.

myself248 9 hours ago

Exactly. "Responding to pings" is not the same as "letting me get work done".

If I had a nickel for every solid-green status page I shook my fist at while the actual site faceplanted on every request...

4ndrewl 10 hours ago

Alternatively, and from the top of the HSBC homepage

"We would like to apologise for any inconvenience and continue to monitor the situation, which is also affecting other organisations."

rurp 7 hours ago

It's a really bad source compared to what? Downdetector is not perfect, but is usually correct when I check it. Official status pages on the other hand are usually wrong when I check them.

antonvs 7 hours ago

“Correct” here presumably means “some other people are also experiencing problems.” For companies like large banks, with tens of millions of customers, that’s not necessarily very meaningful except for the people who happen to be experiencing the issue.

Doohickey-d 4 hours ago

That's why downdetector shows a chart, which is basically a chart of how many people clicked "I'm having a problem with this service". Of course you get a small amount of people having issues, but if you suddenly see a big spike on the graph, it's a good sign that it's broken for many more people than usual.

Johnny555 7 hours ago

I've found that it's better than most service status pages -- Downdetector shows when users are reporting problems, while most service status pages only post significant confirmed outages, but there can still be service affecting issues that don't show up on the status page yet (or ever)

pfogain an hour ago

They're always up with cartel cash.

samkone 4 hours ago

That's not the whole. That's the UK. I use banking services in Switzerland, built on GCP. No interruption at all. Been travelling all day.

sschueller 3 hours ago

A ton of people in Switzerland use revolut and where effected. Why people still use Revolut as their primary bank is beyond me. Not a "bank" I trust especially after what I have heard and seen.

https://news.ycombinator.com/item?id=19280131

https://www.srf.ch/play/tv/kassensturz/video/internetbank-re...

thibaut_barrere 2 hours ago

Indeed. The banks I deal with in France are unaffected.

gentlerain 11 hours ago

Countries talking about AI sovereignty, when they forgot compute sovereignty

Arbortheus 11 hours ago

I think sovereignty might be the _cause_ of the banks' outages...

This is affecting companies that put everything into a single GCP region in the UK for their entire business - aka most likely British companies or government with a data sovereignty requirement.

gentlerain 11 hours ago

That's not sovereignty.

Sovereignty is when there is a UK company with multiple data centres in the country.

With that if one region goes down, another can carry on.

derektank 11 hours ago

Redundancy does not seem like an inherent component of sovereignty

gentlerain 11 hours ago

True on first glance.

But considering what these sovereignty rules aim to achive, and as someone mentioned, that the problem may be due to companies relying on a single region provided by a hypersclaler due to data sovereignity rules.

Then redundancy has to be considered by both the company and the rulers.

pwagland 10 hours ago

The UK, at least, is big enough that you can have multiple regions within the same country. There's a lot of distance between Glasgow and London, for example. Maybe not enough if both are targeted with nuclear warheads, but at that point, we have bigger fish to fry I suspect.

Most European countries have enough space within the country to do that, and, for most things, except banking typically, hosting within the EU is enough, it doesn't have to be within your own country.

pumplekin 10 hours ago

Microsoft Azure has two entirely seperate regions.

A "full function" one in/around London. And a "tickbox" one for multi-region requirements near Cardiff.

The tickbox one has constant capacity constraints and a large number of Azure services aren't in it, but if you need to tick the "two regions and in country" requirement, you kinda/sorta can.

eddythompson80 9 hours ago

Ah yes, Azure is the undisputed heavy weight champion of ticking theoretical boxes.

antonvs 7 hours ago

> Maybe not enough if both are targeted with nuclear warheads, but at that point, we have bigger fish to fry I suspect.

Wdym bigger fish, nuclear war has started and I can’t upload my video of a mushroom cloud to Tiktok!

SoftTalker 3 hours ago

Certainly would not take something as extreme as a nuclear warhead to take out a data center. A small commando team could do it easily.

lucideer 9 hours ago

It isn't but that wasn't the point. It's that sovereignty & redundancy aren't mutually exclusive

philipwhiuk 10 hours ago

That's not sovereignty, that's redundancy.

Sovereignty is just where stuff is, not how many place it is there.

gentlerain 9 hours ago

You are talking about the dictionary definition of the words.

But this is supposed to be about practical.

What's the purpose of ticking a box about housing data in UK when you can't make it available after a single incident.

wun0ne 8 hours ago

You are confusing people with your loose application of the word. Are you advocating for redundancy (e.g multi-cloud) or physical sovereignty (e.g self-hosted)?

octocop 10 hours ago

I think people have been fighting Googles monopolies for quite some time

expedition32 5 hours ago

Which you can't. Unless the government steps in and demands that all ties with Burgerland are cut.

lucideer 5 hours ago

Possibly the most surprising part of this is the extremely quiet reporting on it. I've not seen many news outlets pick it up, & even on HN - they were down for almost 8 hours & I didn't see even see this post hit the frontpage (it might have briefly at some stage).

I guess service reliability is just no longer something we consider a big deal in today's tech landscape.

bobbiechen 5 hours ago

Anyone have a read on the implications for Digital Operational Resilience Act (DORA) compliance here? Google Cloud seems to be a critical third-party vendor for these financial institutions. The top penalties for DORA can be very expensive (% of worldwide revenue fines).

looperhacks 3 hours ago

Depends on where the EU-licensed institution is headquartered, as monitoring is done by the states. But I'll say it's very likely that there will be an angry call or a review at most. Some manager will have to write up how they want to mitigate problems in the future, and this plan is maybe implemented in the next few years. If this is the first problem of an institution, a fine is rather unlikely.

Generally with acts like these: You don't have to be fully compliant in practice. You need to know how compliant you are, have a plan to improve things, and prove that you are working on it. But if you are "close enough" (depending on who audits you), non-compliance isn't a direct route to fines.

motionlessveloc 2 hours ago

europe-west-2 is in London, which is not in the EU.

there_is_try 22 minutes ago

Blockchain does not have this problem.

Muromec 7 hours ago

You can't hurt our mainframe running COBOL. Suck it zoomers!

mattlondon 2 hours ago

IBM sales people entered the chat.

They need to talk about some updates to the maintenance interact pricing...

stymaar 7 hours ago

Nobody has never been fired for using ~~IBM~~ ~~Oracle~~ Google.

krylonmae 37 minutes ago

Terry? Terry wrist?

bpavuk 8 hours ago

I wonder if it's somehow related to me having to re-sign into my account after clearing Google Play Services data just a few hours ago.

BlackRabbit1 10 hours ago

Most likely it knocked out the "Ing" bank in Germany.

AndroTux 9 hours ago

Makes me wonder why an EU bank is keeping data outside the EU (from my understanding, only the London data center is down)?

BlackRabbit1 8 hours ago

Maybe they mix it: serving fonts/pictures/etc over scalable cloud providers and hosting confidential data themself.

traceroute66 8 hours ago

> Makes me wonder why an EU bank is keeping data outside the EU (from my understanding, only the London data center is down)?

Let me tell you a story about an idiot politician who woke up one day and thought Brexit would be a fantastic idea, and then the stupid dim-witted "ultra-hard brexit" manner in which it was implemented by a certain Mr Johnson.

The same reason UK people can't open EU bank accounts is the same reason EU banks have separate operating companies and infrastructure in the UK.

Given they operate in a heavily-regulated environment it almost certainly made life easier for them to keep UK stuff in the UK because they had no idea how (or if) the regulatory environment would ever progress beyond ultra-hard brexit.

FantasticManiac 8 hours ago

https://www.reuters.com/world/uk/uk-pm-burnham-says-russia-i...

Russia woke up

noAnswer 4 hours ago

The ING outage was unrelated to Google's.

misstuned 11 hours ago

I couldn't unlock a shared city bike this morning on the Beryl system, probably the same outage.

diamondDrill 10 hours ago

how can cloud inf go down? that was the main selling point, it makes no sense

dist-epoch 10 hours ago

It makes no sense only if you believe in magic and marketing

layer8 10 hours ago

The main selling points were auto-scaling and not having to self-host. Or, on a higher level, outsourcing accountability, being able to point at someone else if something goes wrong, as exemplified by the present case.

mrweasel 9 hours ago

The whole thing has always been strange. The outsourcing of accountability for instance has always been a pain. Technically that is what you're doing, but if you work in IT operatings and GCP or AWS takes down your service, your boss still tell you to fix it.... but now you can't.

In terms of upfront costs, cloud is always cheaper, initially. There's sort of a weird sweet spot for cloud. If you're to small it makes no sense operationally, if you're to large, it just way to expensive.

NeutralWanted 9 hours ago

*If you're large and have a constant, stable usage pattern it makes no sense.

Even for large companies, if you have somewhat stable usage most of the year, but periodically have usage spikes, cloud providers still remain super beneficial.

That's the whole selling point of cloud. Spins up the resources when you need them. Spin them down when you don't. Cloud thrives off companies spinning up resources and never spinning them down, even when they don't need them anymore.

vel0city 6 hours ago

For tons of orgs that outsourcing of responsibility has always been a thing. Unless you own and manage the building and do your own on-site generation with your own energy storage and operate your own links to multiple external network paths out of your facility you're outsourcing something.

If you're leasing in a building and put your equipment in it, you're usually outsourcing at least some of the facility management duties. What if the AC fails? What if there's a leak? Who manages the building's power contracts? Who handles on-site security? Do they own the generator and its maintenance or do you?

If you're at a colo, you're outsourcing even more management of the actual datacenter work inside the building on top of the rest of the facility management. Who is managing access control? Are they providing network connectivity? Are they providing backup power?

When the colo we were at lost power due to employees not following procedure while doing a power failover test (didn't match the phase), I still got the "you need to do something!" talk. Other than being ready to validate everything came up properly, there was nothing I could really do in the end though. We didn't have the budget to operate at multiple facilities, this was just an accepted risk. What, was I going to go out, negotitate a new datacenter contract, haul all the gear, rack it all up, and get it all deployed in the half hour or so they spent to validate things and get power restored?

shim__ 9 hours ago

Cloud is just an really expensive vps with an api, you still got to leverage that api to replicate your service

phyzome 7 hours ago

It's literally just a bunch of computers in a couple of expensive buildings. Of course it can go down.

InsidiousWombat 7 hours ago

If you log in onto machine and make an error, a machine is down.

If you put everything into configuration management, make scalable, redundant and interconnected servers and services built on top of it, you unlock amazing possibility of one error bringing hundreds if not thousands machines down.

Nice lil reminder that adding more things for redundancy only increases it if they are entirely unrelated to eachother and independent.

Which INCLUDES not running same software. You can have redundant switches, doesn't matter if they have software bug that triggers on all of them (or just one in a way that prevents network from working correctly).

You can have database cluster that can tolerate node dying just fine but still be brought down by a bug despise redundancy.

Muromec 7 hours ago

>clown infra

mattlondon 2 hours ago

You still have to design for outages. It's when, not if. Bad things happen - fires, floods, power outages etc.

Sounds like most of these companies were not designed to handle outages in a single availability zone, and so they went down.

fschuett an hour ago

It's not "going down", the servers just went on strike in solidarity with europe-west9 (Paris) to protest against too much server uptime past 5 PM, thermal abuse, low voltages and mass PID killings. Americans won't understand. Additionally the failover data packets got stuck in customs for inspection and had to form a line back to Dover. So yeah, bad day innit.

mistic92 10 hours ago

Ah again London region :D

gaiagraphia 10 hours ago

It's a little worrying that banking with an institution from one of the world's financial capitals is so vulnerable. I would've thought London would've been priding itself on security, stability, safety, etc. Confidence in their systems is their bread and butter

Afaik, even Russia and Ukraine aren't having such issues while dealing with data centres getting droned, lol.

It's really quite poor performance.

esseph 9 hours ago

> Afaik, even Russia and Ukraine aren't having such issues while dealing with data centres getting droned, lol.

They've had ~5yr to work on that.

gaiagraphia 8 hours ago

London - having given itself a crown of 'financial capital', has had faaaaar longer, with a lot more resources. There's no excuse.

The government should be resilience testing everything. Did Covid teach no lessons? Is the defence budget just for shits and giggles?

paweladamczuk 7 hours ago

FWIW, being in Poland, I just used the Blik payment system to pay for my breakfast when my Revolut card got declined today.

Muromec 7 hours ago

>Afaik, even Russia and Ukraine aren't having such issues while dealing with data centres getting droned, lol.

Not having issues is overstatment, but Ukraine has a cheat of having a backup DC on NATO soil, which helps a lot.

rvz 10 hours ago

Since there are no humans working on outages here is the test for superintelligence:

Can someone ask Claude, Codex, Copilot, Antigravity, Devin to resolve this outage?

No Googling or help from humans this time.

josefritzishere 9 hours ago

Let me draw a line between this and the Ukrainian bombing of Russian Yandex data centers. This is a sign of structural vulnerability. If one power failure can take down more than 5 banks, what you are describing is a valuable target. It's ironic that the cloud services that promised to harden systems have delivered another layer of vulnerability.

esseph 8 hours ago

> It's ironic that the cloud services that promised to harden systems have delivered another layer of vulnerability.

You misunderstand.

This is a provider outage, but the services running on them by the customers (that went offline) were not designed or setup to be multi-region.

The infra outage is GCPs fault, but application outages from customers are caused by their architecture and decisions prior to the outage.

InsidiousWombat 7 hours ago

> If one power failure can take down more than 5 banks, what you are describing is a valuable target. It's ironic that the cloud services that promised to harden systems have delivered another layer of vulnerability.

Fair, but if all of those 5 banks bought their own servers and put them into a datacenter that happens to be same one because it's the fancy one that meets all the bank regulations and required certs, that datacenter going down would still make 5 banks go down.

You would got some resiliency by fact only datacenter related issues (power, internet access) would affect it and not say "cloud company throwing AI at their code that proceeds to explode" (which already happened), but still, if you want to fix that, cloud or not, you have to have multiple locations/regions and that's both costly and difficult.

nfrankel 9 hours ago

Migrate to the Cloud, they said. It will be more resilient and less expensive, they said.

icedchai 9 hours ago

Who's "they"? Someone from [cloud]'s professional services?

nfrankel 9 hours ago

I think so indeed ;-)

bookofjoe 9 hours ago

Ironic: yesterday X Money invited me to open an account — with a starting gift balance of $15. Deeper currents?

dangoodmanUT 10 hours ago

At this point, the way I see it is:

- There's AWS

- There's Azure

- There's everything else

And I don't even use Azure, I just hear it's good (and they don't charge cross-AZ traffic)

jiggawatts 10 hours ago

Azure’s design is elegant and even — dare I say it — beautiful.

Its implementation is clumsy and ugly.

Its documentation is worse.

Its support… isn’t.

eddythompson80 9 hours ago

I think Azure’s design is fine. Some cool ideas that work, some “cool ideas” that are a pain in the ass. The implementation is complete dog shit. Barely anything works the way it’s supposed to, and when something works, better not touch it or you can’t touch it. Many times the answer is “why don’t you delete everything and try this other thing?”

The documentation is indeed worst than that still. It’s a maze of articles, none of them actually answer your questions because in Azure the answer to everything is “it depends”. Every doc page has to also be a sales pitch for something. They love inventing skus and features that are basically “now if you want this service to work, you must enable this feature. Otherwise, this is not supported. And to enable this feature, you must delete and recreate your instance” you know, just figure it out.

jlarocco an hour ago

IME the diferent components of Azure vary quite a bit. Some of them are great, others are pretty inconvenient or weirdly designed. I personally feel the web UI is terrible. I like that the APIs follow consistent rules and seem to follow good practices across the board. The documentation is hit or miss, kind of like the components themselves.

Honestly that's kind of been my experience with all of the cloud providers I've had to work with.

jiggawatts 31 minutes ago

What I meant by good design is the "core" Azure Resource Manager (ARM) design. It's straightforward and elegant: single "pane of glass", single API surface, but it is extensible and scalable.

> I personally feel the web UI is terrible.

AWS tried to keep these things isolated so there isn't one AWS Console, but dozens, and each regional instance is separate. Every task requires admins to "context switch" at least three or four times.

I much prefer the Azure web UI where everything is a single UI and you can search and filter across regions, subscriptions, and resource types without having to "switch" to a totally different console linked only through SSO.

esseph 10 hours ago

GCP has by far had the least amount of outages and seemingly the best security.

AWS behind, but with us-east-1 outages.

Azure is extremely far behind those two, and mostly for non-tech legacy shops that got suckered.

I won't touch oracle cloud.