Today we announce that we have completely removed all traces of disks being used by our VPN infrastructure!
Full article:
We have successfully completed our migration to RAM-only VPN infrastructure
20 September 2023 NEWS SYSTEM TRANSPARENCY
Today we announce that we have completely removed all traces of disks being used by our VPN infrastructure!
In early 2022 we announced the beginning of our migration to using diskless infrastructure with our bootloader known as “stboot”. Completing the transition to diskless infrastructure
Our VPN infrastructure has since been audited with this configuration twice (2023, 2022), and all future audits of our VPN servers will focus solely on RAM-only deployments.
All of our VPN servers continue to use our custom and extensively slimmed down Linux kernel, where we follow the mainline branch of kernel development. This has allowed us to pull in the latest version so that we can stay up to date with new features and performance improvements, as well as tune and completely remove unnecessary bloat in the kernel.
The result is that the operating system that we boot, prior to being deployed weighs in at just over 200MB. When servers are rebooted or provisioned for the first time, we can be safe in the knowledge that we get a freshly built kernel, no traces of any log files, and a fully patched OS.
It’s a good day to be a Mullvad user. Switched over from Surfshark a while ago, and I love it.
no port forwarding. No business from me
Why not host your own on-site VPN server if you want to be able to dial in/allow others to dial in?
Still can’t torrent then
Is it noticeably faster than Surfshark for you?
I haven’t noticed a difference but this company is significantly more trust worthy IMO
Funny thing is I started using Surfshark just before they started all the YouTube sponsorships. Them doing so many sponsorships actually made me trust them less somehow, if that makes sense.
Mullvad “appears” to be more trustworthy but maybe they are just better at marketing that image. They still cost twice as much as Surfshark.
The best piece of marketing Mullvad ever got was when the Swedish police raided them and Mullvad literally had zero data to turn over to them.
Usually when a company throws buckoos of money into advertisements, that’s where the money that could’ve been spent on a better product went. I’ve found products that were advertised so heavily, almost always have dog shit quality.
You don’t use Mullvad for their performance, you use them for their insanely paranoid security and privacy practices.
And for the record, I was never impressed with Surfshark speeds. I dropped them when they bundled a virus scanner into their VPN client, that’s sketchy as hell. I don’t want my VPN provider scanning my files.
Yes I agree Surfshark has done some weird things. I find it weird that it’s actually the same company now as NordVPN, but they don’t make it clear.
Regarding performance, Surfshark is decent speed but still slower than not using a VPN. The more annoying thing is that I get a lot more captchas when using Surfshark. I think these issues are common for all VPNs, though I haven’t tried Mullvad yet (I will when my Surfshark subscription ends).
Yeah a lot of these little VPN companies are getting bought up by larger companies with unknown investors, it’s kinda worrying. There’s one company that owns ExpressVPN, PIA, and CyberGhost now: https://www.kape.com/our-brands/
Kape Technologies (previously named CrossRider) has a pretty sketchy history of making adware: https://www.forbes.com/sites/thomasbrewster/2015/06/09/from-israel-unit-8200-to-ad-men/?sh=7c46d70e26e2
Mullvad has excellent performance though. I got nearly gigabit, I, unfortunately, had to switch when they removed port forwarding.
I never had any real issues with speed using Surfshark, the reason I made the switch was largely about trust. As another user said, as soon as I saw Surfshark start their YouTube advertising spree, and start to bloat their client with unnecessary features, I started looking for alternatives.
I’m iffy about any VPN company that uses YouTuber marketing as it is, and while my threat model isn’t overly paranoid, I believe the VPN company someone chooses to use should have paranoid business practices. After I saw the news on Mullvad’s raid, the authorities subsequently finding nothing, and the fact that a user’s account is merely a string of numbers, I decided it was the VPN for me.
Mullvad compared to PIA, Google annoys me less with recaptures. I know it doesn’t answer your question but thought I’d throw my 2 cents in since PIA was quite a popular choice with their YouTube sponsor slots and cheap prices
It’s good to hear you get less captchas with Mullvad. At least for me, when using Surfshark + Private Browsing, I am basically guaranteed to get a ton of captchas on any Google searches.
Why is their logo a Mole when Mullvad is The Goat
In case someone actually ponders this, Mullvad means mole in Swedish.
VPN = tunnel
Nice.
Wow, that is very impressive. I’ve been a subscriber for a few years and I couldn’t be happier with their service.
Except with the removal of port forwarding
That didn’t effect me much personally and I could understand their reasoning. Still, it’s understandable that it lead to some frustration among other users.
I‘m not really informed on this. What is the reasoning to remove port forwarding?
Ah man, that was the one reason I was going to switch to them !!
Someone in this thread mentioned that was abused so much that hosting providers cancelled them. So they needed to remove it to be able to continue to operate.
I’ve been a subscriber for 5+ years and have zero issue with the loss of port forwarding. I use my devices for everything from gaming to torrenting, and haven’t run into something cause a problem that required me to use port forwarding on mullvad.
what has been an incredible source of frustration as a user of Mullvad tho is when websites block me or hit me with repeating captchas. I’ve also had a huge uptick of spam coming in from weird domains. Obviously not sure if thats mullvad-related, but sounds like the issue of “individuals have frequently used this feature to host undesirable content and malicious services from ports that are forwarded from our VPN servers”.
The removal of this feature seems to be a better of two difficult options.
Torrenting works better and faster with port forwarding.
dogs need about one ounce of water per pound of body weight per day
Der Zuhausi hat Geschmack!
Mullvad is good, definitely my go-to VPN these days.
Interesting what’s going happening with mullvad. For the best part of 10’years, you hear nothing.
Does anyone know why they are recently noisy?
Going by rate of blog posts by year they don’t seem any noisier than usual. The opposite if anything. 18 this year and there’s only 3 and a bit months left of the year whereas in 2018 they made 60.
You are incorrect. Look through their blog archive (scroll to the bottom): https://mullvad.net/en/blog/
They’ve been posting steadily for over a decade, maybe the posts just got more popular this year on whatever sites you browse
They dropped port forwarding and likely lost a lot of business related to this. They are trying to compensate for the loss I think which is great.
Noisy? Bit odd to call it that. Also, from my perception they were always present with regularly published news about how they improve or just update/change their service.
They’ve been relatively quiet on their blogs actually, its just that its growing in popularity, so more people are talking about it.
Imo what makes it so good is their pricing scheme. You put however much money you want in your account, and you get an equivalent amount of time. All you need is an account number, no email no contracts no hassle
I like your username in the context of your comment.
Not sure what you mean, they’ve been posting fairly regular updates on software and infrastructure improvements and security audit responses on their blog for the entire time I’ve been a customer (6 years).
Great question. I’m curious too
Mullvad was always the most straightforward privacy centric vpn with a very long and uneventful history.
They used to offer port forwarding on top of all that. People use port forwarding to do torrents, run internet facing services from home, and share csam.
The authorities could never go through mullvad to get the identities of csam users because of mullvads infrastructure (according to them it’s not stored so it’s impossible for a raid to turn up identifying data).
The authorities switched tactics and convinced a bunch of websites, dns services and other stuff to block mullvads public facing IPs. For a while there in march or so you couldn’t browse shit from a mullvad ip.
The goal was to force csam people on to other services that are softer targets for law enforcement.
It worked. Mullvad dropped port forwarding and all the torrenters, selfhosters and csam traders left.
Mullvad has been working to be a better vpn provider ever since because the raids themselves scare users off, the dropped services lost them some users and the awareness of international law enforcement cooperation scared some users off using vpn services in fourteen eyes countries (mullvad is in Sweden).
At the time all this went down I only had mullvad but now I use another vpn for port forwarding and mullvad for everything that doesn’t need that.
deleted by creator
I find the “Mullvad VPN scratch cards” interesting. If a store near you has them you could buy one and be totally anonymous. What I find a bit odd is that you can buy them on amazon as well but sold directly by mullvad. Doesn’t that defeat the purpose? The idea of the card is a decoupling of your real identity from the vpn user but when you buy the card in their store doesn’t it negate that?
I am probably just missing something here. Does anyone have more insight?
The code on the card is covered so Amazon might know you use Mullvad but they have no way of knowing what your account is.
Mullvad know your acct but they have no way of knowing how it is you paid other than maybe it being a scratchcard which they don’t track anyway.
I am not talking about amazon knowing it. Amazon offers shops for businesses, where a business directly sells goods to their customers using amazon as a transaction platform. Those shops send the goods directly to their customers (Sometimes it comes from an amazon warehouse as well tho). If the first case is true, mullvad would send me the card directly, so they would know I bought it, which makes the card obsolete in my view.
But maybe they don’t send it themself and the cards are all just sitting in a big warehouse. Either way, to me it’s not 100% a given that they couldn’t at least in theory know who bought it.
I am just playing devils advocate here btw, I am not really concerned about it.
You are buying access to a VPN not a nuclear warhead for the black market. The link between buying a VPN card and the code used in that card to link to said vpn activity which is also pretty well protected on Mullvad is not easily discoverable. Seems like a pretty reasonable privacy gap to me.
If you’re a Mullvad customer then they already know your IP and from that they could identify you pretty easily. But that’s true of all VPN providers, but they claim they don’t log and I seem to recall them saying they don’t keep a record of scratch off card numbers (why would they?). Either way you have to trust them and based on the fact they’re totally open I do.
deleted by creator
All those things you listed simply confirm that a particular person bought a Mullvad scratch card. There’s literally no way to associate that data with a particular Mullvad account. To do that they’d have to have a record of the card batch number and somehow have accessed the code underneath the scratch-off panel and then find a way to match those numbers against your Mullvad acct.
You’re not wrong, BUT that’s why Mullvad offers other forms of anonymous payment, the flexibility lets you be as paranoid as YOU want to be. You can pay in Bitcoin, or you can literally mail them an envelope of cash with no return address. Amazon scratch cards are just the most convenient option, and as always, you trade security for convenience.
Well amazon can tell youve bought a card
But not which code you recieved, on the physical card…
Better yet, they employ a guy you can find in an alley who has a bunch of redemption cards in his trench coat. He takes cash or crack.
I think I’ll stick with Monero…
Honestly, privacy utopia.
Probably not because they still dont know who bought that card since the scratch card is linked to the money but that card could be used by anyone. Nothing stop you from buying them and giving them to a friend
Just use cash.
Well the biggest selling point of VPNs is easier piracy not privacy. Most VPN customers just want to protect themselves from anyone watching their downloading habits. Yeah technically there would be a trail but no one is going to follow it to catch someone downloading inception.
Just use monero
Mullvad is such a good company. I just bought another month yesterday, but guess I’ll go and add another year to that!
Of only they’d kept port forwarding.
Didn’t really have a choice:
…Regrettably individuals have frequently used this feature to host undesirable content and malicious services from ports that are forwarded from our VPN servers. This has led to law enforcement contacting us, our IPs getting blacklisted, and hosting providers cancelling us.
Big issue there is hosting providers cancelling them. Can’t operate a business without that.
Short of getting their own servers of course. This update seems to be a step forward in that direction
Even if you own your own servers you still need somewhere to host them; ISPs / colocation providers are going to have the same issues with abuse.
They already own a lot of their own servers, or at least in the nordics.
Agreed. Seems like they were in a super tough spot with that and kind of had to drop it. All the sudden they seem to be doing some new cool stuff to try to keep their edge which I really appreciate / respect. That being said, I’ve dumped them and switched to a service that still port forwards as it gives me better torrenting throughput. Sorry Mullvad.
Only reason I switched away :(
Oh, they ditched it? I was about to switch from Windscribe but I need port forwarding for all sorts of stuff every day. Oh well :-(
They’re amazing. I don’t torrent anymore so I’ll definitely be renewing.
What speaks against torrents with mullvad?
No port forwarding.
What do you mean? Is that needed for torrenting?
I have been using mullvad for a month and have 2 tb of Linux isos, should I expect a call from someone?
The BitTorrent protocol basically works like this when you download a torrent:
- a tracker has a list of clients that have some data of a torrent
- you want to download that torrent, so you ask the tracker for this list
- after you receive this list, you ask the clients on this list to upload their data to you
- repeat 3. until you have the whole torrent
As soon as you have something downloaded, you become a client on the list of the tracker that theoretically has the torrent available for others. So you would become the “client being asked” of step 3 as well.
But how can you be asked? In a P2P networking context, you can only “be asked” if you have a port open that allows connections to it. Otherwise it’s as if you gave people your home adress but your mailbox has a hole on the bottom that leads directly to the garbage can beneath it, so all mail is immediately lost. Completely unusable.
In other words, it’s (basically) impossible for you to send the torrent data to someone else. You’re a leecher, someone that doesn’t give back to others. If everyone would act like you, torrents wouldn’t work at all.
I have been seeding until I reach a 3 ratio, so am I doing so through my real ip?
Is it visible to copyright holders?
I did bind qbittorrent to the mullvad interface so there is no doubt I am using the VPN, the question is how is it possible that I am reaching a 3 ratio when I shouldn’t be able to seed at all?
Found this apparently other people are seeding as well
I’ve been downloading and seeding ever since too. People are dismissing / moving away from mullvad for nothing.
That’s why I said (basically). If another user has a port open and you connect to them through their open port, a bidirectional connection gets established and then you can also upload. But if the other user also didn’t have a port open, then BitTorrent wouldn’t work. You rely on other people to have ports open, if everyone was using mullvad, then it would stop working.
In other words, it’s (basically) impossible for you to send the torrent data to someone else.
I don’t know how (in)accurate this description is, but I’ve been seeding hundreds of GB since Mullvad dropped port fwd. Same for the old times when I didn’t bother using a VPN, I never had to enable port forward in my router for it to work.
That’s why I said (basically). If another user has a port open and you connect to them through their open port, a bidirectional connection gets established and then you can also upload. But if the other user also didn’t have a port open, then BitTorrent wouldn’t work. You rely on other people to have ports open, if everyone was using mullvad, then it would stop working.
that’s so weird, that only one of two people need to have an open port. So the person with the closed port can still establish a connection, both upload AND download. However, this process stops the moment the other person closes his port, then NO connection whatsoever can be established? So we’re gonna pretend this makes sense? Or is it about initiating a connection which requires an open port? That’d make a bit more sense
Isn’t this what STUN is for?
It’s not that simple. Who hosts that server? Which torrent clients implemented support for it? What about symmetric NATs?
In short: no. In long: read up on it yourself.
It’s not needed to leech but it’s pretty important if you want to seed more anonymously
Port forwarding with torrent clients, which they disabled because a bunch of pedophiles were misusing it. Even IVPN disabled it, and AirVPN and Windscribe are the only ones left that are clean and provide this feature.
From what I read in the article, there is still one part of the boot sequence that does require some sort of storage: the part where the bootloader fetches the network boot image and verifies it against the checksum signature. But I think that can be performed by booting from a pendrive and then removing it. The problem will come if law enforcement gets a hold of said pendrive…
Why would that be a problem? A boot image should only contain the commands to get the main system started after POST. It shouldn’t contain any kind of logs, traffic data, or user data. In fact it should be read-only.
PXE boot will TFTP the boot image into RAM and carry on from there. You shouldn’t need any storage on your device.
I’m aware of PXE, but in order to do so you need either of:
- the boot image supplying server being in the same intranet as the rest of the other servers, or
- some sort of method to point the diskless server to the correct external IP address to listen to
Since the first mode is probably too unsafe, that leaves us with the second mode. Either the operator memorizes a specific IP address and types it into the BIOS each time the server is rebooted, or the IP address (and possibly the checksum of the image) are stored in a single-use pendrive that the operator carries. I wonder which of these two methods is used in this case.
PXE is automagic being basically kind of hacky extension to DHCP stuff.
If PXE is enabled the machine will automatically find it via a DHCP relay on the network.
the boot image supplying server being in the same intranet as the rest of the other servers … Since the first mode is probably too unsafe, that leaves us with the second mode.
Why do you think that’s unsafe?
I’ve never done this, but I believe server network cards can be configured for PXE automatically so
Bios -> network card -> PXE over network
So the storage is in the bios config, and then I guess the network card has its own kind of bios?
So there is still one single damning piece of information stored in the servers after all - the IP address to fetch the PXE boot image from. But hey, if Mullvad finds a way to strip even that out of the servers, that’d be great
Why is it damning?
Because by knowing which IP is the boot image stored from, law enforcement can locate the source of the unencrypted image, thus making the scheme lose its privacy. The only way to bypass the issue is by manually configuring the IP after every reboot and keeping it a secret.
Why does being able to access the unencrypted image pose such a problem?
Boot Drive could be immutable and not contain any form of log?
Destroy the drive. That’s what Apple does and how they get around the whole “we need a backdoor” problem. When no one can access the server, no more problems.
Something tells me that they have a stack of single-use drives so that each time a server needs to reboot for some reason, they write a boot loader in one from their central headquarters, walk back to the server room, use the device to boot the server, and finally hammer the everliving bejeezus out of the thumb drive juuuuust in case. Hopefully they don’t have to reboot that often!
What tells you that?
something
So now all that needs to happen is a freak solar flare or electrical surge to completely destroy their business?
Not at all. Of course their operating system has to be booted from some kind of solid state disk, but all actual operations are carried out in the RAM, meaning that nothing is ever written to the disk. Since the RAM is periodically overwritten and doesn’t hold any data in case of a reboot or power failure, they are de-facto not logging a single thing.
I believe the edge devices would use netboot, so they load the kernel and user land into RAM over a network and have no disk in them at all.
Here’s hoping that image stays clean😉
Right, I mean that the boot image is stored somewhere in a solid state. Whether that’s on a centralized server or locally, no clue. But makes sense to load it remotely, easier to maintain proper versioning that way and ensure cross-compatibility.
I think you’re misunderstanding.
One big enough to fry running RAM, especially if they’re using real servers with ECC: RAM that can fix data corruption.
If a solar flare is frying server RAM, the power grid itself is going to be on fire. All of it.
It’d turn the servers off obviously, but that’d be true if it was on disk or not. The source code is stored elsewhere probably on multiple data servers they access with git.
It’s a valid question, even if your scenario isn’t plausible. The very point is that all data is ephemeral - there is no “data at rest” to be compromised. But the problem is that this data is very, very important. It would include (among other things) account information. If all of the servers power off simultaneously (for whatever reason), then yes, it would likely destroy them. More likely is a software fault that causes each system to crash, or lose/corrupt that data.
But there are ways around this, too. I have no idea which (if any) of these they are doing, just that these are options. They already probably sync data among running servers, it will just now be done exclusively in RAM. They can have “seed” distributed servers, running an entirely different codebase, simply to house this data. They would also be diskless, but mostly unconnected to the standard operational servers. From an architecture and design standpoint, these would work a lot like disks.
Distributed is also a key word - it wouldn’t be a single server, rack, or even datacenter that would need to collapse. It would be to be all of them, or at least sever their connections to each other.
(Side note: Going diskless addresses concerns about data security for data at rest. It does nothing about data in motion)
TL;DR: Theoretically yes, but it would take a lot more than that.
Have they ever been audited like PIA?
The article is five very short paragraphs. The third one is:
Our VPN infrastructure has since been audited with this configuration twice (2023, 2022), and all future audits of our VPN servers will focus solely on RAM-only deployments.
Few times, yeah. Last time was August this year
Thank you, a real answer that compares to PIA
Have you read the article?
I did read the article. It does not state if the audit was internal or external “like PIA.”
Guys I know we all hate it when people don’t read the article but did they really deserve the condescension and the down votes? They’re dumb not evil.
I wish votes were like how Reddit originally intended their voting system to be and still states it to be. Upvote if it’s relevant and adds to the conversation, otherwise downvote. But people use it as an agree/disagree system.
I have a counter-argument for that: if votes don’t represent agree/disagree, then your only way of signalling agreement is a reply. Votes != agreement leads to a bunch of one word “This” replies.
Eliminating voting doesn’t eliminate the popularity contest; it just shifts the voting to a more noisy mechanism. You can’t eliminate Popularity; it’s a core function of society. I am not arguing that it’s a good thing, only that it’s going to take more than trying to squash the desire.
If someone states a valid argument that you disagree with, you believe it should be sent to the bottom of the comments? Sounds like the reinforcement of a hive mind, and therefore Lemmy will become just another echo chamber.
Should there be another mechanism to show both? Probably.
I thought that default ranking in Lemmy was based on activity, not votes. I could have misunderstood that, though.
I don’t know, man. Denying the hive it’s just delusional. The best honey is right in the middle; rogue drones just die early. Come, join the beautiful hexagonal map-dance and partake in the nectar.
Anyway, Lemmy’s already an echo chamber; it’s just enforced at the instance level.
Hexagon is the bestagon. Hahaha…I like your hive response.
I’m using the Sync app for Lemmy. It looks like comments are ordered by points/voting and there doesn’t appear to be a way to sort by activity. Hot/new/top/old are the options.
I don’t deny that there will always be an element of echo chamber but I’d prefer it to be as balanced as possible.
Maybe because I’m using an app instead of visiting an instance website and have my default view set to subscribed communities that I’m not seeing an instance echo chamber. Lemmy just appears to be more programming/Linux/self host/home lab than anything else at the moment.
I’m using Voyager, but that’s a good point: clients are free to sort the data however they like.
To be accurate, I think what I heard was that the “front page” feed was populated by activity, not votes.
As per the echo chamber, consider hosts like hexabear, or even my own instance, which has a strong left-leaning flavor. I joined my instance because it’s regionally local, not because of the politics, but I think it’s run by - and, at least initially, populated by - 2A socialists: most members appeared to members of a socialist gun club in Wisconsin (or Illinois?). And Democrat members that try to post on Hexabear get the shit beaten out of them, metaphorically. Votes, IMO, are a much gentler tool for voicing approval or disapproval than vitriolic abuse in comments.
I did read the article. It does not state if the audit was internal or external “like PIA.”
Fair enough, I was more making a point to the people down voting you
Yeah they are pretty transparent about them. The audits will typically find security issues and potential privacy leaks like they ideally should so Mullvad can go and fix them.
Interesting, will this affect performance at all?
I think (disclaimer: not an expert at all) that RAM is much faster to access than a hard drive so if anything it should improve.
Yea. Also a reboot is enough to wipe anything of the face of the globe. So I that can only improvement
It will make no difference to the performance of a VPN; nothing that those nodes are doing is IO bound.
It’s unlikely to have any noticeable impact. This is more about verifiably and categorically not having any traces of logging or cached state.
Both caching and logging should be independent of the direct usage performance anyway. And service startup happens only once - not during its usage.
Doesnt Matter, if the police wants the data, they come with Auto Batteries and an usp and make User of the multiple Power supplies of modern servers.
They will carry the whole rack in one piece if they can.
And if they don’t have anything recent in persistent RAM storage…?
