Cloudfare, an internet service provider of hosting, DNS, etc were informed last week by a Google employee that their servers were returning web pages that seemed to contain sensitive information dumped into them.
It turns out the issue was a result of a bug in the code that modifies web pages as they move through the Cloudfare system. They use this to provide services to customers that redirect requests, inject code into pages etc - it's all very useful, until it goes wrong.
The bug was a simple missing instruction in the C program that runs this service. The bug was only exposed when a web page moving through the code was malformed - for example a web page with an unclosed script tag.
This led to the program getting confused about where it should be reading from and it 'jumped' out of the memory where the HTML was kept and into memory from other requests/programs that were also running. This means that a broken web page could display a piece of authorisation header information, api keys, content etc from a totally unconnected website, they just happen to both use Cloudfare's services.
The Impact
The impact of this bug should not be underestimated. Some huge brands and services use Cloudfare services and as a result data from those services has been injected into badly written web pages. That in itself isn't a huge concern until you think about two things.... search engine spiders and browser caches. Lots of this data has been cached by search engine crawlers, you can still find it now (I won't post a link). There's also the issue that a lot of it is also stored locally in people's browser caches. That's a huge potential security breach with all kinds of personal and transaction data being dumped out.
What to do
The best thing to do is check whether your are using any of the websites and services listed here: https://github.com/pirate/sites-using-cloudflare/blob/master/README.md - if you are then you should probably change your passwords, API keys etc as a precaution.
Below is a detailed timeline of the incident from Cloudfare themselves.
Detailed Timeline
We are very grateful to our colleagues at Google for contacting us about the problem and working closely with us through its resolution. All of which occurred without any reports that outside parties had identified the issue or exploited it. All times are UTC. 2017-02-18 0011 Tweet from Tavis Ormandy asking for Cloudflare contact information 2017-02-18 0032 Cloudflare receives details of bug from Google 2017-02-18 0040 Cross functional team assembles in San Francisco 2017-02-18 0119 Email Obfuscation disabled worldwide 2017-02-18 0122 London team joins 2017-02-18 0424 Automatic HTTPS Rewrites disabled worldwide 2017-02-18 0722 Patch implementing kill switch for cf-html parser deployed worldwide 2017-02-20 2159 SAFE_CHAR fix deployed globally 2017-02-21 1803 Automatic HTTPS Rewrites, Server-Side Excludes and Email Obfuscation re-enabled worldwide
Report from Cloudfare here: https://blog.cloudflare.com/incident-report-on-memory-leak-caused-by-cloudflare-parser-bug/
Cloudfare seem to have been completely transparent about this and have sorted it very quickly. Hopefully lessons will be learned here by everyone and the result will be better systems and checks going forward......hopefully.
