CPDoS can be used to attack content delivery networks (CDNs) to serve error pages instead of legitimate sites through caching.
Freshly termed as a Cache Poisoned Denial of Service (CPDoS) attack, two academics from the Cologne University of Applied Sciences in Germany have discovered [PDF] how content delivery networks can be attacked to serve error pages instead of legitimate sites through caching.
In normal circumstances, when the user re-visits a website, they are served a cached version of it through a content delivery network such as Cloudflare.
On the other hand, during the poisoning process, the attacker visits a website with the intention of generating a request for the web page from the CDN.
However, the request, in this case, contains a malformed header which can be from the following 3 options:
- An HTTP Header Oversize (HHO)
- An HTTP Meta Character (HMC)
- An HTTP Method Override (HMO)
HTTP Header Oversize (HHO)
An HTTP request header contains vital information for intermediate systems and web servers. This includes cache-related header fields or meta data on client supported media types, languages and encodings.
The HTTP standard does not define any size limit for HTTP request headers. As a consequence, intermediate systems, web servers, and web frameworks define limits by their own. Most web servers and proxies such as Apache HTTPD provide a request header size limit of around 8,192 bytes to mitigate, e.g., Request Header Buffer Overflow or ReDoS attacks. However, there are also intermediate systems that specify limits larger than 8,192 bytes.
For instance, the Amazon Cloudfront CDN allows up to 20,480 bytes. This semantic gap in terms of request header size limits can be exploited to conduct a cache poisoning attack which can lead to a denial of service.
HHO CPDoS attacks work in scenarios where a web application uses a cache that accepts a larger header size limit than the origin server. To attack such a web application, a malicious client sends a HTTP GET request including a header larger than the size supported by the origin server but smaller than the size supported by the cache. To do so, an attacker has two options. First, she crafts a request header with many malicious headers as shown in the following Ruby code snippet. The other option is to include one single header with an oversized key or value.
require 'net/http'
uri = URI("https://example.org/index.html")
req = Net::HTTP::Get.new(uri)
num = 200
i = 0
# Setting malicious and irrelevant headers fields for creating an oversized header
until i > num do
req["X-Oversized-Header-#{i}"] = "Big-Value-0000000000000000000000000000000000"
i +=1;
end
res = Net::HTTP.start(uri.hostname, uri.port, :use_ssl => uri.scheme == 'https') {|http|
http.request(req)
}
HTTP Meta Character (HMC)
The HTTP Meta Character (HMC) CPDoS attack works similar to the HHO CPDoS attack. Instead of sending an oversized header, this attack tries to bypass a cache with a request header containing a harmful meta character. Meta characters can be, e.g., control characters such as line break/carriage return (\n), line feed (\r) or bell (\a).
An unaware cache forwards such a request to the origin server without blocking the message or sanitizing the meta characters. The origin server, however, may classify such a request as malicious as it contains harmful meta characters. As a consequence, the origin server returns an error message which is stored and reused by the cache.
HTTP Method Override Attack (HMO)
The HTTP standard provides several HTTP methods for web servers and clients for performing transactions on the web. GET, POST, DELETE and PUT are arguably the most used HTTP methods in web applications and REST-based web services. Many intermediate systems such as proxies, load balancers, caches, and firewalls, however, do only support GET and POST. This means that HTTP requests with DELETE and PUT are simply blocked. To circumvent this restriction many REST-based APIs or web frameworks such as the Play Framework 1, provide headers such as X-HTTP-Method-Override, X-HTTP-Method or X-Method-Override for tunnel blocked HTTP methods. Once the request reaches the server, the header instructs the web application to override the HTTP method in the request line with the one in the corresponding header value.
POST /items/1 HTTP/1.1
Host: example.org
X-HTTP-Method-Override: DELETE
HTTP/1.1 200 OK
Content-Type: text/plain
Content-Length: 62
Resource has been successfully removed with the DELETE method.
The code snippet shows a request that can bypass a security policy that prohibits DELETE requests by using the X-HTTP-Method-Override header. On the server-side this POST request will be interpreted as a DELETE request.
These method overriding headers are very useful in scenarios when intermediate systems block distinct HTTP methods. However, if a web application supports such a header and also uses a web caching system like a reverse proxy cache or CDN for optimizing performance, a malicious client can exploit this constellation to conduct a CPDoS attack. The figure below illustrates the principle flow of an HTTP Method Override Attack (HMO) CPDoS attack using the X-HTTP-Method-Override header.
Here, the attacker sends a GET request with an X-HTTP-Method-Override header containing POST. A vulnerable cache interprets this request as a benign GET request targeting the resource https://example.org/index.html.
The web application, however, will interpret this request as a POST request, since the X-HTTP-Method-Override header instructs the server to replace the HTTP method in the request line. Accordingly, the web application returns a response based on POST.
Let’s assume that the target web application doesn’t implement any business logic for POST on /index.html. In such cases, web frameworks like the Play Framework 1 return an error message with the status code 404 Not Found.
The cache assumes that the returned response with the error code is the result of the GET request targeting https://example.org/index.html. Since the status code 404 Not Found is allowed to be cached according to the HTTP Caching RFC 7231, caches store and reuse this error response for recurring requests. Each benign client making a subsequent GET request to https://example.org/index.html will receive a stored error message with status code 404 Not Found instead of the genuine web application’s start page.
Any of these, in turn, cause an error on the webserver. As a part of regular operations, the error page is then cached on the CDN and in turn served to legitimate users. Slowly, it spreads to other nodes of the CDN as they are updated hence giving the impression of the site being unavailable globally successfully completing the attack. This could cause both a reputational loss and financial losses as well for mission-critical websites.
This was further confirmed by the researchers themselves along the lines of,
To get a clearer picture on the real life impact of CPDoS attacks, we took some samples based on the URLs from the Alexa Top 500, DoD, and HTTP Archive data sets.
Overall, we found twelve vulnerable resources within a few days.
These also include mission-critical websites such as ethereum.org, marines.com, and nasa.gov which use CloudFront as CDN.
At all these websites, we were able to block multiple resources including scripts, style sheets, images, and even dynamic content such as the start page.
Example screenshot on Marines.com website:
An example showcased by the researchers of a before and after scenario of the attack on Marines.com.
For those of you wondering when your site is getting poisoned, you can relax and luckily take a series of precautions.
The first one is that you could turn off the caching of HTTP error pages by default on your CDN service through predefined options or by editing your server’s configuration files. Hence, the error page will never be distributed by the CDN en masse.
Secondly, you could make sure your CDN provider is complying with standard caching protocols.
One of these involves caching only certain error pages such as the “404 Not Found” but not others such as the “400 Bad Request” one which is generated because of such an attack.
A table allowing one to analyze the errors different CDN providers cache.
Continuing with the often seen pattern in the cybersecurity world, these revelations were also followed with responsible disclosure leading to companies like Microsoft and Amazon taking action to fix the problem at hand.
However, some companies like Flask have still not responded according to the researchers and so it is yet to be seen how the customers of such firms will react. For users looking to do further research on the subject in more detail, they can visit a recently released website by the team at CPDOS.
The map below shows the impact of CPDoS attacks on CDNs. Once the error page is injected, the CDN distributes it to many other edge cache server locations around the world. The map illustrates how far the error page is distributed to several edge locations within the CDN. The icons show the affected locations displaying the error page. Fortunately, not all edge servers are infected by this attack which is shown by the icons. This icon denotes the locations where clients receive the genuine page. The icon shows the location of the origin server and the icon displays the attacker’s locations.
The first figure shows the affected regions in Europe and some parts of Asia when sending a CPDoS attack from Frankfurt, Germany to a victim origin server in Cologne, Germany. The second one illustrates the poisoned regions in the USA when executing a CPDoS attack from Northern Virginia, USA to the same victim origin server in Cologne, Germany.
This analysis has been conducted with TurboBytes Pulse and the speed testing tool of KeyCDN. Both services provide a testing environment covering a lot of test agents scattered around the world.
CPDoS vulnerability overview
This overview summarizes what pair of web caching system and HTTP implementation is vulnerable to what CPDoS attack. More details are described in the paper which can be downloaded below.
Mitigations
One of the main reasons for HHO and HMC CPDoS attacks lies in the fact that a vulnerable cache illicitly stores responses containing error codes such as 400 Bad Request by default. This is not allowed according to the HTTP standard. The web caching standard only allows to cache the error codes 404 Not Found, 405 Method Not Allowed, 410 Gone and 501 Not Implemented. Hence, caching error pages according to the policies of the HTTP standard is the first step to avoid CPDoS attacks.
Content providers must also use the appropriate status code for the corresponding error case. For instance, 400 Bad Request which is used by many HTTP implementations for declaring an oversized header is not the suitable status code. IIS even uses 404 Not Found when a specific header is exceeded. The right error code for an oversized request header is 431 Request Header Fields Too Large. According to our analysis, this error message is not cached by any web caching systems.
Another effective countermeasure against HHO and HMC CPDoS attacks is to exclude error pages from caching. One approach is to add the header Cache-Control: no-store to each error page. The other option is to disable error page caching in the cache configuration. CDNs like CloudFront or Akamai provide configuration settings to do so.
A Web Application Firewalls (WAF) can also be deployed to mitigate CPDoS attacks. However, WAFs must be placed in front of the cache in order to block malicious content before they reach the origin server. WAFs that are placed in front of the origin server can be exploited to provoke error pages that get cached either.
For more details on possible mitigations and countermeasures, please read our paper.
