>No “vulnerabilities” in Tailscale were found or exploited, and that might make it even more uncomfortable for us. [...] But, we're a security tool. Their intrusion is our intrusion, and it's our job to take it seriously.
im a happy customer of tailscale, so i am obviously biased, but i have a lot of respect for this. they could have just stayed quiet and i dont think anyone would have bat an eye.
I don't think anyone is faulting Tailscale for using their blog to advertise, and no one is faulting Tailscale for how they handled this situation in general, or their response to it.
The issue is acting like their response is 'brave' in anyway, or altruistic. It can be considered admirable only to the extent any company doing a good job running its business can be.
This is a good, smart response to what happened. They are approaching this incident as a way to improve their product and offer a better service to their customers. That is good, and it is fine to reward them with your business in response. There is nothing wrong with making good business decisions, but it isn't something that we need to unduly respect.
i said i have a lot of respect for this. whether you do or not is up to you.
anything a company writes is an advertisement by the nature of being written by a company. i dont think that means anything a company writes is bad by default. there are many corporate blogs i enjoy reading, or learn from, etc., despite the fact that they are all technically advertisements.
in this case, tailscale is setting a higher expectation for themselves when no one asked for it. i find that respectable.
> tailscale is setting a higher expectation for themselves
What exactly is the higher expectation? As someone with little expertise and no stake in any of this, the blog reads as "our products are great and could have solved this problem if they were being used correctly, so it's not our fault" with a few vague proclamations about how they will improve their UX. This isn't at all a bad thing, it just isn't very notable in my opinion.
better defaults, better documentation, better UX, and "But, we didn't stop it. Next time, we will." are all commitments that they didn't need to make, but now they need to follow through with or lose face.
>it just isn't very notable in my opinion.
i agree that this seems to be getting way more attention than i would have expected.
I have recently noticed that the words "ad" or "marketing" have become, in and of themselves, with no additional information or context, slurs or dismissals.
I understand why. The modern internet has turned advertising into a morass of constant bombardment and the only sane response is to block as much as possible and ignore as much else as possible.
But it's unfortunate because, in some sense, ever single thing that a company every says that is not legally mandated in some way is a form of advertising.
And in many cases, that "advertising" contains true, useful information that can be helpful.
What is important isn't whether or not something is an "ad", but instead, whether or not it contains true information that is helpful in some way.
Many ads don't reach this bar. They are either misleading, straight up lying, or information that is almost completely useless.
But when I'm searching for a particular product, about the only source of information at all is some form of advertising, and I almost always find at least some amount of it to be helpful in making a product decision.
Ads are more often than not polluting to the informational ecosystem, but that's not because they are ads.
Fun corollary: Self-promotion is only considered to be advertising, marketing, or spam when it’s describing someone else’s self-promotion. When it’s describing one’s own, it’s not a perjorative :)
Tailscale as a company reminds me of Valve and other good old tech-oriented people that I can "trust" that they know what they're doing. I'm a happy customer too and I hope they retain the essence of what distinguishes Tailscale.
(Tailscale CEO) You have posted here multiple times that "none of the code has had a security audit" and that the SOC2 audit "is not the same thing."
It's true that those two audits aren't the same thing. However, the SOC2 auditor confirms, in the published report, that Tailscale has regular and ongoing security audits including penetration tests and many kinds of code reviews.
The security audit report, which you perhaps imagine to be a long list of vulnerabilities... doesn't look like that. It says we don't have a long list of vulnerabilities. The security bulletins are all here: https://tailscale.com/security-bulletins
The majority of your security bulletins are as the result of third-party reports to you.
Which, by definition, means they are not done by you, which means you don't know when they will be done or how much of your code base they are looking at.
I think you know full well what I mean by a security audit. If you don't, go look at, for example, the ones that Mullvad publish for their software https://mullvad.net/en/blog/tag/audits.
Please do not try to portray SOC2 as being the same thing as a code audit.
And IF you have regular code audits, then please publish suitably redacted reports in public on your website. Just like everyone else does !
(Tailscale CEO) I don't know what to tell you. The problems that are found internally, or via security reviews and pentests we pay for, are ones that we fix before releasing. They don't need bulletins.
Bugs that are found by other people are found, by definition, after release. They are therefore more likely to need a bulletin.
Is your theory that "any competent security review" will find every security hole in a product? Because that sure would be great if it were true. Unfortunately it does not match my experience.
The post you replied you clearly asked why insecure argument handling should be discovered after release. You widened that to mean “every security hole must be discovered prior to release” which made it a much easier argument to attack.
The point stands that the insecure argument handling could’ve been discovered by a security audit even when every security hole may not have been found.
i think its probably okay to drop it when you are this far deep into a conversation and the affiliation has already been stated multiple times in the existing comment chain.
But things like insecure argument handling are low-hanging fruit for security auditors.
Insecure argument handling is not like the more advanced subtle vulnerabilities that we are seeing in some LLM-assisted reports these days. Insecure argument handling is 1990's security.
The fundamental problem remains that Tailscale has too many new "features" being added to it the whole time. New features means a whole bunch new code. Which increases bloat and exponentially increases the attack surface.
It would be really nice if you could stop shoehorning in every new feature you can think of. Remove some of the existing ones that don't really need to be there. And get your codebase back to a more focused state, get back to your roots as a VPN product.
Stop trying to be all things to all men, as the old saying goes.
It appears that you have a major bone to pick with Tailscale and direct replies to your concerns do not seem to matter. They have a good security track record and are extremely widely deployed. I don’t see the evidence for your assertions that it is bloated and insecure.
<s>Read</s> Skimmed the report (and cheers for not gating it behind request-to-obtain) and I'm a bit surprised to not see any mentions of pentests which I'd expect given the large surface you host. What gives?
The large attack surface is a good point. I started using it initially and the ease of setting up a vpn was nice, but then I came across few security vulnerability postings which led to concern so I went to Wireguard. I think they should reign in the features and treat it as a secure vpn first and foremost and remove unnecessary features to minimize the attack surface.
If you want a hard-to-use VPN with minimal features and minimal surface area, as you said, Wireguard is right there. Tailscale is convenient Wireguard.
I use Wireguard for several site-to-site VPNs. It just works. I never have to worry about it, and there are very few configuration settings to mess up (unlike, say, IPsec, which is a nightmare.)
You do realise that of all the security tools on the planet, plain wireguard most likely has the smallest attack surface of them all, right ?
The problem here is as the other poster said. Tailscale is a security tool and yet the guys at Tailscale seem to be insistent on dumping everything INCLUDING the kitchen sink into it as a "feature".
That sort of attitude is not going to end well. You end up with a large bloated code base, which equals large attack surface.
Agreed its a tool, and it does not looks like any vulnerability was exploited on their end. However part of the blog is confusing to me. Long lived keys vs short lived have their own space in discussion, there are use cases, pros & and cons for the same. However it is not very clear how a short lived credential would have prevented the exploit in this case?
I read it as I should add new nodes with short lived keys and avoid long lived ones.
Yes, this doesn’t solve the issue, but also adds one more step to the chain.
The AI didn’t crack the encryption or managed to get arbitrary access. And also Tailscale makes things easier to manage than simple VPNs and firewall rules. But it still requires a decent amount of attention and careful configuration to make a perfect system.
It would not have prevented the initial exploit. The agent gained access to the K8s cluster and read the Tailscale Auth Key from Cluster Secrets. A short-lived credential would reduce the risk that the key is still valid when an agent gains access.
Thats the point, the short lived credentials would have done much other than adding more step of getting the secrets again which is why I am questioning the discussion of that in a root cause analysis itself. Adding one more layer does not increases the security by default in every case.
A short-lived credential would reduce the risk that the key is still valid when an agent gains access
How does that work? If the agent is reading the credential from the live configuration, how short does the lifespan of a key need to be to prevent it from being used by an unauthorized process?
That's why I said "intent" as opposed to "perception". I'm implying that it's unfalsifiable not due to faulty construction but rather due to being inherently true. (Of course I realize that's technically an overgeneralization.)
Tailscale is responsible for designing a system whose convenient defaults allowed a stolen credential to have a very large blast radius. A marketing blog is not changing that.
im a happy customer of tailscale, so i am obviously biased, but i have a lot of respect for this. they could have just stayed quiet and i dont think anyone would have bat an eye.