Back to qualys.com
74 posts

Qualys SSL Labs releases raw data from the Internet SSL survey

About two months ago, Qualys SSL Labs published the results of an Internet-wide SSL survey.  We said that we would make the raw data available, and today we are  following up on that promise. (By the way, we realize that two months is  a long time, but we couldn’t complete the process faster on this occasion. We hope to make future releases pretty much as soon as we obtain the data. As you may remember, our plan it to make our survey a quarterly event from 2011.)

The raw data contains the SSL assessment results of about 850,000 domain names (out of about 120M we inspected). The main file (1.2 GB 120 MB compressed, 3.5 GB 800 MB uncompressed) is a dump of our PostgreSQL database in CSV format. We include in the download a simple PHP script that iterates through all the rows, which means that you can consume the data directly. Alternatively, you can put the data back into the database and use SQL to run ad-hoc queries (we provide the schema along with the import instructions).

The database schema contains 63 fields that generally parallel the information you would obtain from the SSL Labs online test. The complete original certificate chain is included, which is handy if you want to look into the aspects we didn’t. We chose not to release certain sensitive data: the information on the low entropy private keys, renegotiation support, and HTTP server signatures was removed.

This is what you need to do to obtain the data:

  1. First, make sure that our terms and conditions are acceptable to you. At the core, we use the Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported licence, but there are a few additional requirements. For example, we ask the obvious — that you don’t use the data for illegal activities. The other requirements are just common sense. (Please do read the entire file, however.)
  2. Second, send us an email (username "ivanr"; domain name "webkreator.com"), introduce yourself, and tell us how you intend to use the data. We will then send you back the download instructions. We need this second step to give us an idea if the data is used, and how.

Update: We are removing the certificate chain data from the database until we confirm that we are legally allowed to redistribute it. If you need such data in the meantime, retrieve it directly from the servers.

Podcast: Ivan talks about the Qualys SSL Labs Internet-wide SSL survey and the recent release of the raw data from the survey.

Disabling SSL renegotiation is a crutch, not a fix

In the days that followed the discovery of SSL/TLS Authentication Gap, some sites (those that did not need renegotiation) were able to deal with the problem by disabling renegotiation in server code. With no support for renegotiation, gone was the danger of exploitation. Good for them.

The sites that did need renegotiation had to wait, first for the TLS working group to solve the issue on the protocol level, and then for their SSL library (or web server) vendors to support the enhancement. The TLS working group did a great job negotiating the fix. As for the vendors, some implemented the new feature quickly, some dragged their feet a little, and some (Debian) seem to refuse to fix the problem, leaving their users vulnerable.

To sum it up, today, almost a year after the initial public discovery, we have some servers that are still vulnerable, some that refuse to support renegotiation, and some that support the new standard for secure renegotiation.

So where is the problem, you might ask? If disabling renegotiation prevents exploitation, that’s surely a good thing? Well, it depends on how you look at things. Try to look at the problem through the eyes of a browser developer. I was actually prompted to write about this problem by Yngve Nysæter Pettersen, who’s part of Opera’s security group. Opera wants to protect its users, and for that to be possible they need to know if a particular server supports secure renegotiation. If a server does, Opera can happily renegotiate whenever necessary. But if a server does not support secure renegotiation, you can make an argument that Opera should refuse any renegotiation attempts.

The servers that support secure renegotiation indicate so during the SSL handshake phase, and everyone’s happy and secure. The issue is with the servers that disable renegotiation, because they provide no indication of their security status. Some are insecure, while some aren’t. Without knowing, browsers can’t do anything. They can perhaps only inconvenience users and force them to manually configure protection levels.

While it is possible to test for insecure renegotiation (SSL Labs does it), the test is indicative but not conclusive — there is no way to test for server-initiated renegotiation. Besides, it’s not reasonable to expect browsers to test every SSL site they encounter.

My point is those that disabled SSL renegotiation must nevertheless implement the proper fix as soon as it becomes available for their platform. Patching is slow enough as it is, and we don’t need any further distractions to slow us down.

Internet SSL Survey 2010 Results

We are happy to announce the results of the first Internet SSL Survey conducted by SSL Labs. The 2010 version is a result of several months of full time work, resulting with a detailed analysis of how SSL is deployed on the Internet.

Attachments

Internet SSL Survey 2010 3.0 M

SSL and TLS Authentication Gap vulnerability discovered

A serious vulnerability has been discovered in the way web servers utilise SSL (and TLS, up to the most recent version, 1.2), effectively allowing an active man-in-the-middle attacker to inject arbitrary content into an encrypted data stream. Both the Apache web server and the IIS have been found to be vulnerable.

The problem is with the renegotiation feature, which allows one part of an encrypted connection (the one taking place before renegotiation) to be controlled by one party with the other part (the one taking place after renegotiation) to be controlled by another. A MITM attacker can open a connection to an SSL server, send some data, request renegotiation and, from that point on, continue to forward to the SSL server the data coming from a genuine user. One could argue that this is not a fault in the protocols, but it is certainly a severe usability issue. The protocols do not ensure continuity before and after negotiation.

To make things worse, web servers will combine the data they receive prior to renegotiation (which is coming from an attacker) with the data they receive after renegotiation (which is coming from a victim). This issue is the one affecting the majority of SSL users.

The following example demonstrates how the flaw can be exploited by an attacker to send an arbitrary request using the authentication credentials of a victim. The red parts are sent by the attacker and the blue parts are sent by the victim.

GET /path/to/resource.jsp HTTP/1.0
Dummy-Header:
GET /index.jsp HTTP/1.0
Cookie: sessionCookie=Token

The good news is that, although the attacker can execute an arbitrary request, he will not be able to retrieve the corresponding response. On the negative side, the client will see something different from that what she requested.

You can see that GET attacks are essentially trivial to execute. To date, no one has claimed a successful execution of a POST request using this flaw. Until someone does, that means that an application that only makes changes in response to POST requests will probably not be vulnerable. Further, an application not vulnerable to CSRF attacks will probably be safe too, because the attacker won’t be able to generate or predict the token required for the request to go through.

Mitigation options:

  1. If you can, disable renegotiation. There isn’t normally a configuration option to do this, but patches are being developed and will be available soon. The majority of web sites do not use renegotiation so disabling it won’t be a problem. Those that do will need to make changes to their sites to make them work without it.
  2. Use a web application firewall to monitor the contents of all request headers to spot what seems like an embedded HTTP request line. The good news is that the embedded request line will not be obfuscated, making it easier to detect. I do not believe that this advice can help the bypass of the client certificate authentication, though.
  3. If you can, monitor all connections that make use of the renegotiation feature. That won’t help you if renegotiation is an integral feature of your web site, but it may do if it is rarely used.

Further information: