Tag Archives: GAE

Writing Files to GAE Blobstore

FileSystem API was introduced in AppEngine last year. A great feature. I tried using it more than a couple of times and ended up with one problem or the other. Some of the notable issues include:

  • What should be the filename during create and save? The name given during create was only name-part and cannot be used during read. Read must use fullpath.
  • If you use fullpath, you cannot use it to delete the file.

After a bunch of experiments, I concluded that it is best to use BlobKey.

Read more …

Processing Shortcodes in a post in Minow

Minow with Shortcode processing

Minow can now process short-codes. Well, it started doing so almost a month back… just that I never got time to update my blog.

As of now, the short-code processor has been encapsulated in a simple interface that does the processing of a short-code with attributes. At http://minow-web.appspot.com/, you may notice YouTube videos embedded in the posts. They are not hard-coded HTML but processed using the YouTube short-code processor.

Read more …

Update on Minow Progress

Update on Minow Progress - A Closeup

Minow (Minow Is Not WordPress) Engine can now read data from data-store. It’s not generic enough though – right now hard-coded to Google App Engine, but nevertheless.

Here’s what I’ve done since last push:

  • Model for localized content is ready
  • Basic logic to retrieve content for a given locale (not parent, no fallback) is ready. For example, if the language is “hi_IN” (Hindi-India), it – as of now – can get you the content for “hi” and/or fallback to “en_US” / “en”.
    As of now, the language is a query-parameter… will enhance to parse “accept-languages” header with priority (q=0.3 etc)
  • Nest theme has been updated to use the “real” data
  • Implementation exists to lazy load data from the store – don’t load the posts until they’re really required. A lot of WordPress templates do not have blog on their landing page
  • And of course, a piece of code generate sample data 😉

Read more …

WordPress on Google App Engine

Minow is not Wordpress - Wordpress on Google App Engine

I think this is a dream for almost every blogger… well, at least my and at least hundreds of thousands of others.

Most of the other results that you’ll find on search engines relating to WordPress and Google App Engine are about running WordPress on GAE/J by using Caucho Quercus – a bridge between PHP and Java. However, if running / simulating PHP on Java was that easy, PHP on GAE would not have been most starred issue.

And that made me think… how about creating, may be, yet another blog engine for GAE. And what better than taking clues from WordPress.

So, that gave birth of Minow, acronym for Minow Is Not WordPress. Yes, it’s a recursive acronym – see http://en.wikipedia.org/wiki/Recursive_acronym for what I mean.

Read more …

WordPress for Google App Engine?

I love WordPress. A big fan. But then I once-upon-a-time used to be a huge fan of PHP, and then I’ve grown up. I know people will kill me for that statement. 😉

But the fact of the matter is PHP still has serious issues. And most recently, I can’t use it to run WordPress at Google App Engine, something that’s amazingly great for the starters who want to setup quick-and-not-so-dirty app.

I thought… would it be good to have WordPress for GAE? I came across Vosao – Open Source CMS for GAE/J. And I loved it immediately. Has a long way to go, and I do not see any active development on the same… surprised, what happened to the team.

StackOverflow has a question – What CMS runs on Google App Engine, but I personally didn’t like any of them.

And then I thought again… how about writing my own CMS with persistence in a Non-Relational Store with plug-in model so that I can run it on any store, be it GAE’s Data Store, MongoDB or Amazon DynamoDB or whatever.

Let me see if I can hold on my thoughts and really get on with it… At the moment, no promises! :-)

User Experience and Application on Google App Engine

So, we have been working on this application… web based, data driven, to be used in “social context”. I think today there’s nothing sexier than using the word social.

We designed this application nicely, fantastic by many ways. We were sure of putting this up on a public cloud, just that we were not sure of which one: EC2 or GAE. One of the most riskiest questions to keep open… but really, we were not sure. The only thing we were sure of was Java + jQuery + HTML5 (as much as possible).

EC2 gives you IaaS but then you must manage all the servers, load balancing, failovers. Even if you happen to choose BeanStalk, a lot of work has to be done, specifically database. Thankfully, we now have ElastiCache (launched around August 2011, IIRC) otherwise managing a cluster of Memcache was yet another task for the admins.

GAE provides you PaaS, so you don’t need to worry about doing all the stuff in previous paragraph yourself. Except that you have strong restrictions… a very small subset of Java SE library – specifically, not threads or direct sockets, no JDBC and Datastore, though amazing, is not an easy meat to digest for those not used to. And above all, the server side execution must complete in 30 seconds (now on what are known are front-end servers).

So, we designed our own data-access abstraction so that we can switch EC2 <=> GAE without any hassles. It took a while, but it’s a great thing that we have.

After writing core entities and business logic layer, it was time to start pushing it all up to the user interface. The dashboard – the landing page after login – has to get updates from various “components” in the system. All components worked great, except that the dashboard took somewhere around 1200ms to load for the first time. Note that most of the public resources (images, JavaScript, CSS) were loaded from GAE CDNs.

Just not acceptable!

Read more …

Google App Engine and GWT – Multi-Project Application

While working with Google Web Toolkit (GWT), it is a very common scenario to split the main application into multiple sub-projects and then use IDE to do a combined build and compile. However, the problem arises in scenarios where the GWT-RPC services and their implementations are kept in separate projects.

Recently working on an application, I structured it as follows:

  • MainApp: The final GWT+GAE application that will be deployed
  • AppCommon: Common classes to be used across projects
  • AppServices: Service definitions. It comprises of the GWT-RPC service definitions and their Async counterparts.
  • MainApp project contains the implementations of the services defined in the AppServices project.

When I “GWT Compile Project” or “Deploy App Engine Project“, the “shared” or “server” code of the dependencies is never copied to the war/WEB-INF/classes folder. Only the compiled JavaScript code is compiled.

So, I thought of automating all the stuff using Ant files.

GWT app-creator tool creates the “build.xml” for Ant and for GAE, there are guidelines on the web, for example here.

At an outline, the following needs to be done during the project build:

  • Enlist all the dependencies in order.
    If there are five projects – P1, P2, P3, P4, P5 – my script cannot identify which one depends on the other. It’s the developer’s responsibility to write them in the order so that independent or least dependent projects are listed and hence compiled first.
  • Compile all the projects in order with appropriate class-path…
  • Make a jar comprising of all the compiled class files and push it to war/WEB-INF/lib folder.
  • Compile the GWT module, as usual
  • Publish the application to GAE, as usual

I have made used of the following Ant tasks:

  • foreach from ant-contrib. You can download the jar from Sourceforge. Project home page is at http://ant-contrib.sourceforge.net.
  • Standard property, mkdir, delete, javac, copy etc tasks
  • App Engine task appcfg.

First step is to define a property, I named it projects, enlisting all projects in order.

<property name="projects" value="AppCore,AppServices,AppOne,AppTwo" />

Next is to expand the default task javac to compile all the dependencies as well. To take care of dynamic classpath entries arising because of the multiple projects, I copy all the class files to the <main-app>/war/WEB-INF/classes folder and that’s my standard classpath.

<target name="javac" depends="libs" description="Compile java source">
  <foreach list='${projects}' param="project.dir" target='compileDeps' />
  <mkdir dir="war/WEB-INF/classes" />
  <javac srcdir="src" includes="**" encoding="utf-8" destdir="war/WEB-INF/classes"
    source="1.5" target="1.5" nowarn="true" debug="true" debuglevel="lines,vars,source">
    <classpath refid="project.class.path" />
  <copy todir="war/WEB-INF/classes">
    <fileset dir="src" excludes="**/*.java" />

What this target does is compile all the dependencies before compiling the main application project.

Defining the compileDeps target is more straightforward now, noting that we have a helping hand from foreach task. Here’s how it looks like.

<target name='compileDeps' description='Compiles dependencies'>
  <property name="pdir" location="../${project.dir}" />
  <property name="cdir" location="." />
  <mkdir dir="${pdir}/war/WEB-INF/classes" />
  <javac srcdir="${pdir}/src" includes="**" encoding="utf-8" destdir="${pdir}/war/WEB-INF/classes"
    source="1.5" target="1.5" nowarn="true" debug="true"
    <classpath refid="project.class.path" />
  <copy todir="${pdir}/war/WEB-INF/classes">
    <fileset dir="${pdir}/src" excludes="**/*.java" />
  <copy todir="${cdir}/war/WEB-INF/classes">
    <fileset dir="${pdir}/war/WEB-INF/classes" excludes="**/*.java" />

This task compiles the dependencies, and copies all resources (non-java files) from the corresponding src folder and all binaries from the corresponding war/WEB-INF/classes folder to the <main-app>/war/WEB-INF/classes folder.

I further expanded the gwtc target to make a jar file rather than individual class files in the WEB-INF folder. Here’s the code for that.

<target name="gwtc" depends="javac" description="GWT compile to JavaScript">
  <jar destfile="war/WEB-INF/lib/main-app.jar">
    <fileset dir='war/WEB-INF/classes' />
  <delete dir="war/WEB-INF/classes" failonerror="false" />
  <java failonerror="true" fork="true" classname="com.google.gwt.dev.Compiler">...</java>

And that’s it!

Publishing the application to GAE requires no special change. All files are right there! Just use the <appcfg> task.

Google App Engine – Dealing with large number of files

One of the most common problem that one has to deal while deploying a large scale application at Google App Engine is the file limit – in terms of number of files.

As per the issue reported at http://code.google.com/p/googleappengine/issues/detail?id=161, the limit has been increased from 1000 to 3000, but there’s catch. Any static file (non-JSP files outside the WEB-INF folder, unless explicitly excluded in appengine-web.xml) are counted twice. Why? They’re pushed into a special folder __static__ and all such files are counted twice.

I looked at deploying the SmartGWT showcase on GAE and the “appcfg upload” gave an error about having over 9000 files. I was shocked! Literally.

And then I came across a simple yet powerful idea – zip all the static files and use a servlet to extract and serve on the fly. The solution servlet was uploaded by sindhom at http://code.google.com/p/googleappengine/issues/detail?id=161#c68.

Take that as a base, I made an update – again simple yet probably very useful. Look at the “If-Modified-Since” header and respond with with a 304 or 200 depending upon if the entry in the zip file is updated or an old one. There’s one thing hardcoded in the my solution – the Cache-Control / Expires timeline is hardcoded to 365 days, and it can be easily moved into the configuration.

The solution is mentioned at http://code.google.com/p/googleappengine/issues/detail?id=161#c88. The SmartGWT showcase is available at http://1.latest.mastergaurav.appspot.com, along with the complete code of the servlet at http://1.latest.mastergaurav.appspot.com/UnzipperServlet.java.txt.