Google App Engine – Dealing with large number of files

One of the most common problem that one has to deal while deploying a large scale application at Google App Engine is the file limit – in terms of number of files.

As per the issue reported at http://code.google.com/p/googleappengine/issues/detail?id=161, the limit has been increased from 1000 to 3000, but there’s catch. Any static file (non-JSP files outside the WEB-INF folder, unless explicitly excluded in appengine-web.xml) are counted twice. Why? They’re pushed into a special folder __static__ and all such files are counted twice.

I looked at deploying the SmartGWT showcase on GAE and the “appcfg upload” gave an error about having over 9000 files. I was shocked! Literally.

And then I came across a simple yet powerful idea – zip all the static files and use a servlet to extract and serve on the fly. The solution servlet was uploaded by sindhom at http://code.google.com/p/googleappengine/issues/detail?id=161#c68.

Take that as a base, I made an update – again simple yet probably very useful. Look at the “If-Modified-Since” header and respond with with a 304 or 200 depending upon if the entry in the zip file is updated or an old one. There’s one thing hardcoded in the my solution – the Cache-Control / Expires timeline is hardcoded to 365 days, and it can be easily moved into the configuration.

The solution is mentioned at http://code.google.com/p/googleappengine/issues/detail?id=161#c88. The SmartGWT showcase is available at http://1.latest.mastergaurav.appspot.com, along with the complete code of the servlet at http://1.latest.mastergaurav.appspot.com/UnzipperServlet.java.txt.

Notice: This work is licensed under a BY-NC-SA. Permalink: Google App Engine – Dealing with large number of files

Leave a Reply

Your email address will not be published. Required fields are marked *

question razz sad evil exclaim smile redface biggrin surprised eek confused cool lol mad twisted rolleyes wink idea arrow neutral cry mrgreen

*