Reflections on Trusting Trust

Submitted by tytso on Thu, 10/23/2008 - 04:45.

Printer-friendly version There's been some discussions on the weekly LSB call about how far we should go in terms using a very careful system of protecting the OpenPGP keys that we use for signing the packages that we release. Some have gone as far as to say that we should use a hardware solution, much like Red Hat has done for signing their packages. For more information about what they did, it's instructive to read Mark Cox's description of what he implemented, here: http://www.awe.com/mark/blog/200701300906.html What did he do, specifically? He created a hardware signing solution where "authorized package signers" could submit a kerberos authenticated request to server which then talked to a FIPS 140-2 certified hardware module which held the private key componets of Red Hat's package signing key. Why did he do this? To quote from Mark's blog posting: So the authorised signers not only had the ability to sign with the key, but they also have the ability to read the key material. In theory this means that a malicious internal signer could copy the key, take it away with them, and sign whatever and whenever they wanted. Or, more likely, a clever intruder who gained access to our internal network could perhaps capture the key and passphrase, compromising the key. The risks mean we've had to be really careful who has signing privileges with the legacy key and how the key signing is handled. So there are really two fundamental threats: (a) the malicious insider retaining a copy of the key (b) an attacker who manages to compromise the machine containing the key, installing a keyboard sniffer and then grabing the key. In addition, one thing to keep in mind is that Red Hat (and any other distribution) has to sign hundreds of packages per distribution, and so they need a solution which can scale to handle signing a *very* large number of packages every day, by multiple people managing the release process. First of all, I'll note that a malicious insider can do plenty of damage without having any access to the key. Anyone who can get root on one of our build engines can build a trapdoor into the C compiler's crt0 file, at which point every single binary that we build could be a trojan horse. And, of course, there is alwys the techique which Ken Thompson demonstrted in his classic paper, "Reflections on Trusting Trust". So the real concern is the attacker who manages to compromise a machine that contains the keying material, and who manages to somehow gain access to the passphrase. And to counter that threat, there's something we can do that wouldn't be practical for a Red Hat or any other distribution, but which should be good enough for us given very small number of packages that we need to sign for releases, beta releases, updates, etc. What's my proposed solution? For our most secure keys (the release key and annual key) the private component is only kept on a bootable USB stick; the OS on this USB stick will have no network daemons, and any new keys or packages to be signed are stored on a second USB stick. Once created, the bootable USB stick never gets inserted into a running system; the procedure to be followed is to reset the system, insert the bootable USB stick with the private key components, and then boot into the OS stored on the bootable USB stick. So even if an attacker manages to compromise the laptop where the packages are signed, the OS USB stick should be safe since the attacker won't have access to it. Yes, it's not as convenient as the solution devised by Mark Cox, but it's much simpler to implement, and we would probably need to boot into the USB environment no more than once every 2 or 3 months. - Ted

Reflections on Trusting Trust
Submitted by LSB List on Thu, 10/23/2008 - 07:15.

Posted to the lsb-discuss mailing list by: Nelson B Bolyard

Theodore Ts'o wrote, On 2008-10-22 20:39:

> What did he do, specifically? He created a hardware signing solution
> where "authorized package signers" could submit a kerberos authenticated
> request to server which then talked to a FIPS 140-2 certified hardware
> module which held the private key componets of Red Hat's package signing
> key.

I have a FIPS compliant hardware security module (HSM) in my pocket,
on my key ring, actually. It's the size of a USB stick. It has a
USB interface, but it is NOT a USB memory device. It is a real HSM.
The private key on there won't come off due to any clever software.

I paid about USD $50 for it, quantity 1, if I recall correctly.
I've had it for a couple of years now. It holds several private keys
and related certs. It takes about 1-2 seconds to sign with a 2k-bit
RSA private key.

Known as an Aladdin eToken, drivers are available for it for Linux and
Windows. It has a PKCS#11 module, so it works with NSS-based applications
such as all the Mozilla browsers and email clients (Firefox, Thunderbird).

Also, a growing number of X86 PC motherboards now have suitable crypto
HSMs built onto the motherboards. Known as a "trusted platform module"
(TPM), they are capable of storing private keys and doing signing.
Some of these have PKCS#11 modules available for them, and work with NSS.
They are available even for laptops and tablet PCs.
See http://download.intel.com/design/motherbd/articles/TPMFlyer.pdf and
http://www.tonymcfadden.net/tpmvendors_arc.html for more info.

My point is not to sell eTokens or Intel motherboards. :) It is that
good strong crypto solutions that are easy to use are inexpensive and
readily available. I'd encourage people to consider them over protocols
involving memory sticks and rebooting computers to use them.

Reflections on Trusting Trust
Submitted by tytso on Thu, 10/23/2008 - 14:15.

On Thu, Oct 23, 2008 at 12:02:24AM -0700, Nelson B Bolyard wrote:
>
> I have a FIPS compliant hardware security module (HSM) in my pocket,
> on my key ring, actually. It's the size of a USB stick. It has a
> USB interface, but it is NOT a USB memory device. It is a real HSM.
> The private key on there won't come off due to any clever software.

I looked at the Aladdin eToken, actually. It's now about $35 dollars,
and it's OK, BUT:

*) The web pages implied that you had to patch software to make it all
work, and the software stack required is rather large, and I fear,
fragile. There was also some question about possibly losing
functionality if you used the open source OpenSC stack as opposed to
the proprietary binary one provided by the Aladdin.

*) The Aladdin eToken was clearly optimized for the case of storing an
individual's key, where it's no big deal if if the token gets
destroyed or the passphrase gets lost --- you just generate a new
key. It doesn't work so well if you are storing some key where you
want more than one person to be able to sign on behalf of that key,
or where you want to have backups since revoking a key high in the
trust/signing hierarchy is troublesome and might cause conspiracy
theorists to claim that may be there was some security breach you're
not telling them about. :-)

*) There is some hint that it's possible to upload an SSH key (and
thus any private key) to the eToken, which would be one way of
solving the above problem --- just generate the key outside of the
device and then upload it to multiple etokens --- but there are more
web pages who insist just as stridently that You Can't Do This with
the eToken. (This may be where use of the proprietary Aladin stack
does something OpenSC does not; I'm not sure.)

As far as TPM is concerned, I've also spent several hours in the past
trying to get the TPM module in my Lenovo laptop to work, and at least
when the TPM was previously initialized under Windows (because I
wanted to use some of the security features for XP), I was never able
to get it to work. Some web searches claimed it was because the
string-to-key algorithm was never standardized, or the Thinkpads used
a non-standard string-to-key algorithm (something about Unicode 16 if
memory serves correctly). I tried patching the Trousers source code
to get around it, but that was also something that I could never get
working. I believe I also tried initializing the TPM from scratch and
then trying to make it work, but at least at the time, it was highly
frustrating, and I never was able to make it do anything useful.

In addition, the caveats about needing the ability to back
up the private key apply just as much to a TPM based solution, since
it's much easier to lock up USB-stick-sized device except when we need
it (which is only once every 2-3 months) where as a laptop tends to be
taken out and about, and you have to worry more about what happens if
laptop gets stolen at a conference, or it gets dropped and the
motherboard gets toasted.

I know that TPM's do have the ability (in theory) to export their keys
so they can be migrated to another device, but I'm not sure how well
those features are tested, and I'm not sure how it works if you are
migrating between two different TPM made by two different platform
manufacturers (i.e., Lenovo vs. Dell). The fact that none of this
technology (eTokens, smartcards, TPM) has gotten widespread use makes
me nervous that there aren't Stupid Stuff (i.e., incompatible
string-to-key algorithms, and another really stupid, novice-grade
interoperability mistakes) hiding here.

I used to be a security geek for a living (dev lead for MIT Kerberos,
IPSEC working group chair), and if I had two weeks where I could do
nothing but spend time hacking these hardware tokens, I'm sure I could
do it --- I spent two hours going through web pages learning about
this stuff, and it was a very enjoyable two hours, too.

But because I've done enough professional security work, I started
thinking about our threat environments, and realistically, we have
enough ways that a malicious insider could sneak untrusted code into
the software packages that we export; this is a generalized problem of
all open source software, and the reality is that someone with root on
our build boxes can get access to the key used for daily builds, yes
--- but it would also be trivial enough to gimmick the bzr binary to
insert unstrustworthy code, or the crt0 file, never mind Ken
Thompson's oldie-but-goodie of hacking the C compilers, which is
perhaps the hardest way of doing the deed.

So it would be hugely entertaining for me to implement some hardware
based solution, and one of these days I'll probably get me one of the
Aladdin eToken devices and see which of the conflicting web pages
regarding whether you can upload ssh keys or not is in fact supplying
the correct information. I could easily see my taking a vacation for
2-3 weeks and trying to make all of this stuff work.

But implementing some solution such as what Mark Cox described would
be for us, the same as the TSA collecting water bottles and throwing
them in a huge barrel right by the X-ray machines, or comparing the
names on photo ID against paper boarding passes that can be trivially
forged if you know how to edit PDF files or know how to use
gimp/photoshop. That is, it would be Security Theatre; looks good,
yes, but whether it actually improves net security is highly
debatable.

The main reason for not doing something like a bootable USB stick and
protocols that require rebooting into a trusted environment while the
laptop is disconnected from the network, etc., is the convenience
factor. This sort of solution certainly wouldn't work for a distro
who is signing hundreds or thousands of packages a week during their
release cycle, but we don't have that problem.

If someone can point me at a FIPS-140 solution where the software
stack is known to work out of the box on multiple distributions
(ideally, Debian/Ubuntu, Fedora/RHEL, and OpenSuSE/SLES) --- no
patching or use of proprietary object-code-only software from the
manufacturers --- and which supports key backup/migration, and
seamless integration with GPG, all without needing any custom patches,
or replacing random daemons in some other security's software stack to
fake them out, I don't have an objection to using a hardware solution.
Or if someone else wants to volunteer the time to get it all working,
and can convince me that the any patches to make it all work will
either make its way upstream to the distro's in fairly short order, or
that it is possible for us to support said patches ourselves
indefinitely, I'm not *against* a hardware solution; in fact, I would
think that would be pretty cool.

However, it's hard for me to justify a huge amount of my personal
time, or paid staff time, to work on such a solution, especially if
the end result might be hard to support and maintain over time. It
would be great from the convenience point of view, and from a
gee-we-have-this-cool-hardware-solution-geek-street-cred point of
view, but I haven't been convinced that from a security perspective
only, the benefits outweigh the likely costs to implement and maintain
such a solution. I'd love to be proven wrong....

Regards,

- Ted

Reflections on Trusting Trust
Submitted by Wichmann Mats D on Thu, 10/23/2008 - 15:45.

Theodore Tso wrote:
> On Thu, Oct 23, 2008 at 12:02:24AM -0700, Nelson B Bolyard wrote:
>>
>> I have a FIPS compliant hardware security module (HSM) in my pocket,
>> on my key ring, actually. It's the size of a USB stick. It has a
>> USB interface, but it is NOT a USB memory device. It is a real HSM.
>> The private key on there won't come off due to any clever software.
>
> I looked at the Aladdin eToken, actually. It's now about $35 dollars,
> and it's OK, BUT:
>
> *) The web pages implied that you had to patch software to make it all
> work, and the software stack required is rather large, and I fear,
> fragile. There was also some question about possibly losing
> functionality if you used the open source OpenSC stack as opposed to
> the proprietary binary one provided by the Aladdin.
>
> *) The Aladdin eToken was clearly optimized for the case of storing an
> individual's key, where it's no big deal if if the token gets
> destroyed or the passphrase gets lost --- you just generate a new
> key. It doesn't work so well if you are storing some key where you
> want more than one person to be able to sign on behalf of that key,
> or where you want to have backups since revoking a key high in the
> trust/signing hierarchy is troublesome and might cause conspiracy
> theorists to claim that may be there was some security breach you're
> not telling them about. :-)
>
> *) There is some hint that it's possible to upload an SSH key (and
> thus any private key) to the eToken, which would be one way of
> solving the above problem --- just generate the key outside of the
> device and then upload it to multiple etokens --- but there are more
> web pages who insist just as stridently that You Can't Do This with
> the eToken. (This may be where use of the proprietary Aladin stack
> does something OpenSC does not; I'm not sure.)
>
> As far as TPM is concerned, I've also spent several hours in the past
> trying to get the TPM module in my Lenovo laptop to work, and at least
> when the TPM was previously initialized under Windows (because I
> wanted to use some of the security features for XP), I was never able
> to get it to work. Some web searches claimed it was because the
> string-to-key algorithm was never standardized, or the Thinkpads used
> a non-standard string-to-key algorithm (something about Unicode 16 if
> memory serves correctly). I tried patching the Trousers source code
> to get around it, but that was also something that I could never get
> working. I believe I also tried initializing the TPM from scratch and
> then trying to make it work, but at least at the time, it was highly
> frustrating, and I never was able to make it do anything useful.

yes, let's stay away from the TPM-in-a-laptop situation. besides the
fragility of the software - I'm being driven batty by a Windows-only
implementation on HP (what you'd expect to be the best supported),
where the system just loses its brain and can't talk to the TPM any more,
and then the NT Security Service shuts down the system, so it's like
my notebook crashing slowly (I get 60 seconds warning) - but when the
notebook hardware breaks, as they generally do, you've lost whatever.

There are other hardware-ish solutions that do more - take a look
at the Inaura K9 UMAK (the device formerly known as a Blackdog): this is
actually a full system on a USB token, with an embedded PPC chip,
and you need do nothing to your host to get it running.