Supermarket freezers across Europe hit by second suspected cyber-attack (theguardian.com)

It was a better deal before AI, but now the economics of a lot of things are changing rapidly.

"That will never happen to us" - until it does.

Or netbird. I was exactly thinking the same btw clearly we're missing out something. or maybe cybersecurity is a cow ready to be milked?

Did Tailscale not have a major vulnerability just a few days ago?

That's my thought too.

These fridges may need more complex control for a variety of reasons, but that doesn't mean they have to be connected to _the internet_. To a local network for control, sure.

Put your security measures into the network itself, but I really do feel fridges are another thing which do not need to be IoT devices.

  > I still don't see why the expert has to be remote

Because there are 1000 store locations. You're either then paying for 1000 mostly idle engineers, or somebody not really qualified has to don that hat, or you're flying your engineers around constantly. Or just connect the devices to the Wild Wild Web and be done with it.

> Plus there’s no way this remote management stuff doesn’t cost at least $500/store in operating costs.

Going by the number of companies that still can't grasp having to pay for ongoing maintenance for and IoT system, even with CRA and NIS2, some still selling stuff running on 12+ year old software, I wouldn't expect them to pay much for this service or to even properly maintain it.

> Why would there be accidental port mappings on their router though?

That's the point of NAT-PMP / UPnP, it just works (for game servers, torrent clients, industrial control systems etc.)

My guess is setting up a port forward in the gateway router is part of the freezer install process. The customer probably had to give the tech the router credentials and the tech sets up the forward themselves.

Litigation is not how we fix things in Sweden, the system is very different from the US. The companies will have to take the losses or try to recoup from insurance and update their systems. They might be able to negotiate deals w the fridge install companies.

Yeah

And it would be (mostly) fine if they were on the net, with read only access

Thanks!

The comments in the Aug 26 thread aged quite badly.

Multiple sources confirm that it's the same Danfoss problem this time, for example, https://www.hln.be/binnenland/centrum-voor-cybersecurity-bev... (in Dutch)

Somebody needs to run Windows Update :-)

> the hackers pirated controllers mostly supplied by the Danish company Danfoss, which said subsequently that it had identified some security flaws last year and had “long recommended” customers install updates.

Sounds like other retail systems where lock screen passwords are the store number or similar conventions

Similar to the 'S' in VPN

Good link, thanks.

Was a bit surprised (well, no, not really) to see that the linked XWEB bulletin in that alert mentions a backdoor which can be used to help customers regain access to breached systems. Wondering how likely it is that this backdoor algo has also been leaked or reverse-engineered too:

(translated from French to English with Google Translate)

> If your password has been changed and you are unable to log in, please contact Copeland Technical Support, available 24/7, at [...] and ask for instructions on how to obtain a temporary password. Copeland technical support will need the current date of the XWEB EVO and the MAC address displayed on the XWEB login page.

Depends. Lidl wants to be a tech company.

https://en.wikipedia.org/wiki/Schwarz_Digits

It really depends on the grocery. Kroger gets a big chunk of their profits from their data stack.

Not many, probably, but their cold chain people definitely would have known. Instrumented and remote monitored cold chains are standard these days because keeping stuff refridgerated end to end is a pretty big deal.

I bet most managers don't even know you can even connect a fridge to the internet.

Monitoring, alerting, remote diagnostics and trouble shooting.

Commercial cold chains are expensive, high value collections of equipment that even small errors at the edges can cause huge monetary losses. And grocery stores are in an extremely low margin business.

Brainstorming here; monitoring, logging, etc. The fact the store managers were called out of bed when there was an issue indicates it works. If a cooler broke down overnight or its temperature was too high for X amount of time, they have to discard the products. Without monitoring they likely wouldn't know until customers got sick.

I wish commenters (in general) would do a bit more "steelmanning" before asking questions about why something is (in this case) connected to the internet.

No, and I get that, but working in the industry, he can't have been in the dark about the events in France. If he doesn't deal with the monitoring of the fridges himself, he should be equally upset that the chain has failed to act.

Likely so that stores can be monitored remotely. A lot of supermarkets have a centralized orchestration somewhere, especially big chains.

For commercial uses some kind of global monitoring is not super crazy. Regular retail freezers being on the internet, that's the crazy talk.

Completely speculating as I don't know the situation in Europe, but some possible reasons are:

- Remote alerting for the store manager/head office to quickly respond to a refrigerator failure after hours, providing options for saving valuable stock (e.g. move stock to another refrigerator).

- Maintaining an audit trail for cold chain management (e.g. regulator says you must keep an audit trail proving that a frozen fish has been kept below -30oC since it was processed on a ship all the way up to point of sale in a supermarket).

- Refrigerator manufacturer says a condition of warranty is that the refrigerator must provide remote telemetry proving some warranty condition has been met (e.g. regular cleaning cycle operated).

Note: all these speculated reasons are unidirectional data flows. So even if a valid reason exists for Internet connectivity, why can't it be limited to unidirectional data flows?

On top of the monitoring mentioned by others, it's also to create a centralized log of actual temperatures. This then lets you show inspectors that you are keeping up with refrigeration requirements.

What I read from the last wave is that supermarket changes how cold freezers will run depending on how much customers they expect that will open and close the doors. Just having the freezers adapt based on the temperature of every single moment cost too much electricity or can result in unsafe temperatures.

I don’t know about supermarkets but for medical freezers it’s useful to be online for monitoring and calibration.

My org has lots of freezers that need to be at precise temperatures. Sometimes they drift so you have to check them a few times per day, log the check, and adjust as necessary. Doing this for a thousand freezers is tedious and error prone. Networking helps. “Cloud” lets it all be done remotely without getting IT very involved for vpns to be set up with specific vendors. It also helps with patching and auditing security so seems odd that these supermarket ones were out of date.

Some large supermarkets (in the UK at least) sell their capacity to the National Grid so it can be balanced in times of demand. They agree to turn off their freezers for a set amount of time to create a "virtual battery".

Not just that. But why the fuck is the control software so weak? Why does it allow remotely bumping temperatures to near boiling without any checks? Which idiot designed this system? Why aren't they out of business? Same with the people who installed these systems. Why on earth didn't they think to secure it?

Supermarket freezers are a bit more complicated than that. They can run at different temperatures and can consume a lot of power, so are part of a bigger store HVAC and electrical system as they may load-balance. Stores do not have qualified engineers on site to do anything with refrigeration equipment and it is better to have minimum wage offsite monitoring than expect a minimum wage shelf packer who only gets one toilet break a day to care about the fridges.

I hear you're passionate about this, but why a fine? They were the victims, in our society the victim of a crime doesn't need to be penalized further, moreover, the only damage was to themselves.

Yes, VPNs would be the correct solution.

However, we're talking about companies that put unprotected systems on the public network and did nothing about it even after getting three weeks advance warning.

The skill, and time, required to "take it offline" is far lower than any solution mentioning VPN. And given that, more likely to succeed while a better solution is found.

I dont think VPNs are the correct solution. They are one solution, sure, but if it is the correct solution depends on many things.

For example, where does the VPN connect to? The vendor? Do you trust the vendor more than the internet. Sure. Do you trust the vendor, that does not manage the firmware of their devices and uses inbound connections in the first place, enough to put them on a VPN?

How many devices and vendors do you have? How many VPNs can you establish with the existing networking hardware, and how many can you manage without IP clashes etc. IP clashes work both ways, btw. The vendor has to manage this themselves also if all their clients want VPNs.

Retailers are enterprises with 100s of vendors that run on a shoestring. On top, they use a lot of legacy architecture and hardware. They have teams that have a hard time making an architectural decision in the best of times, but even less so under stress. It is probably the same on the vendors side.

In my mind, the correct solution is indeed to deny any inbound connection to the network, now, to prevent further damage.

Incident response is, in order:

    1. preparation (failed)
    2. identification (done)
    3. containment <--- we are here (take offline, put manual rota in place to manage fridge defrost cycles) 
    4. eradication (VPN, etc)
    5. recovery (take it back online, relieve manual rota)
    6. postmortem etc

>>Or to take the freezers offline if an update isn't available. > > Why the heck would this be necessary? We have VPNs holy crap

Each to their own… but why do fridges need to be connected to the internet in the first place?

Sounds like they might have to pay experts to do it.

The question is whether this is attack has cost them more than that would have.

I worked in IT for a major retail chain in the US. I was on the team in charge of store systems. We knew what devices were connected to the network, and if we had a vulberable device like this, we would have sent an urgent message to store managers to disconnect themm immediatly, and we would have followed up with scans over several days to ensure it happened. There is no excuse. (If we really needed the devices connected, we would have put them on a firewalled VLAN, which we could have pushed out in hours and had across the nation within days at most.). Granted, ideally we would have already had them VLANed.

It is incompetence to not have this capabilty.

The internet seems like the easy solution, not the obvious solution. For years, we have used the internet for things that didn't need it because it was easier and it seemed safe enough. I think that the coming months and years are going to see a lot of companies burned by that decision and scrambling to undo it.

The fact was that it was never safe, it was never secure, but that taking advantage of those problems was hard, and generally not monetarily worth it, meant we ignored the issues. That's no longer the case.

As for low quality software, the floor for the quality that internet connected software needs to have is now dramatically higher than the quality that non-connected software needs. I think that in many applications, it may no longer be economically worth it, or maybe even possible at all, to hit the necessary quality.

So then you have a connection going one way (freezer to data-logging endpoint) and not the other.

Have the freezer connected to an air gapped grocery local server. Fixed

> it lies with low quality software.

Nearly all software is low quality software. The programmers at the fridge company are likely 4 underpaid engineers in Pennsylvania or some other equivalent.

Saying the problem here is bad code is like saying the problem with gun violence is humans are too squishy to deflect bullets.

These need to have an audit trail. Making it centralized is an addition, no?

The audit trail (status in general) could be read only. Enabling remote defrost (control in general) is an optional convenience which requires better quality software.

What if high quality software exists, but the supermarkets did not pay to install the last 10 years of updates?

Which is easier, getting the audit trail from non-internet-connected devices, or keeping all the devices patched?

The right answer isn't forever patching freezer software. The answer is either to not have them on the internet, or to have something in front of them, and keep that thing very up-to-date. (Or maybe several layers of "something" in front of them.)

You should only ever have hardware visible in the internal network, no need to put them on the Internet at all.

Those regulations need an update.

And the simplest solution is to air gap these systems, then figure out how everything else should work.

In NL we use paper and pencils for elections because it makes fraud incredibly hard. It makes the process harder as well but the baseline is that it breaks nation-scale attacks. Why this level of thinking isn't applied to *our foodchain* is beyond me.

Monitoring doesn't require anything more than read-only access to fridge data.

Ironically, were the monitoring system competently made, not only would it have been unable to cause this problem, but it would have also been able to alert about it in time to prevent significant loss.

Why do they need to change the temperature? I thought it was supposed to be constant.

> What is the store to do, employ full-time knob-turners?

How often do you think the knobs would need to be turned? You think someone or something is constantly fiddling with the temperature knobs of each individual freezer?

The freezer itself doesn't need to connect to a SCADA. You can network power and temperature monitors, but those monitors don't need the ability to shut off the freezer.

These freezers are manually stocked and cleaned. If they need to be powered down it's a simple matter of a hardware switch that you flip when the unit is empty.

That would be cool, do you have any source that confirms they used this for taking advantage of price fluctuations?

>The markets hit here in Sweden are all in rural communities with very low number of tech workers.

Hej! You are wrong, here in the middle of Stockholm got hit.

https://www.mitti.se/nyheter/icabutikens-frysar-hackades--fi...

ICA Baronen on Odengatan is one of several stores in Sweden that have suffered a major hacker attack. On Monday, their freezers were cancelled at 50 degrees.

This is in Stockholm - https://maps.app.goo.gl/BwbDST4dPVkjcdgA8

So no lack of IT workers/tech workers... Nor is it very rural.

>They likely order their system from a supplier, that should have alerted them and helped them, but apparently did nothing.

The first thing to do when you get access to these as a hacker? You disable the alarms...!

Now, the supplier should have set them up correctly over VPN, good passwords, Zero Trust etc but these an OT/SCADA are usually just open to the internet due to the simplicity of use.

But then the supplier is responsible, why would you ever put a fridge online if you are are tech noob. Someone told these people to put these fridges online, they didn't put an Ethernet cable in the back by themselves for the fun of it.

Some of them (Coop) got new equipment after what happened in France and where gonna upgrade them in two days.

ICA is franchised, so stores are owned and run by operators. The central company offers some services, but a lot of the decisions comes down to the individual stores.

Imagine developing a control system for a professional freezer and designing it so it can be directly exposed to internet due to "simplicity".

Want something simple? Don't connect it.