Category Archives: Google App Engine

User Experience and Application on Google App Engine

So, we have been working on this application… web based, data driven, to be used in “social context”. I think today there’s nothing sexier than using the word social.

We designed this application nicely, fantastic by many ways. We were sure of putting this up on a public cloud, just that we were not sure of which one: EC2 or GAE. One of the most riskiest questions to keep open… but really, we were not sure. The only thing we were sure of was Java + jQuery + HTML5 (as much as possible).

EC2 gives you IaaS but then you must manage all the servers, load balancing, failovers. Even if you happen to choose BeanStalk, a lot of work has to be done, specifically database. Thankfully, we now have ElastiCache (launched around August 2011, IIRC) otherwise managing a cluster of Memcache was yet another task for the admins.

GAE provides you PaaS, so you don’t need to worry about doing all the stuff in previous paragraph yourself. Except that you have strong restrictions… a very small subset of Java SE library – specifically, not threads or direct sockets, no JDBC and Datastore, though amazing, is not an easy meat to digest for those not used to. And above all, the server side execution must complete in 30 seconds (now on what are known are front-end servers).

So, we designed our own data-access abstraction so that we can switch EC2 <=> GAE without any hassles. It took a while, but it’s a great thing that we have.

After writing core entities and business logic layer, it was time to start pushing it all up to the user interface. The dashboard – the landing page after login – has to get updates from various “components” in the system. All components worked great, except that the dashboard took somewhere around 1200ms to load for the first time. Note that most of the public resources (images, JavaScript, CSS) were loaded from GAE CDNs.

Just not acceptable!

Read more …

Google App Engine and GWT – Multi-Project Application

While working with Google Web Toolkit (GWT), it is a very common scenario to split the main application into multiple sub-projects and then use IDE to do a combined build and compile. However, the problem arises in scenarios where the GWT-RPC services and their implementations are kept in separate projects.

Recently working on an application, I structured it as follows:

  • MainApp: The final GWT+GAE application that will be deployed
  • AppCommon: Common classes to be used across projects
  • AppServices: Service definitions. It comprises of the GWT-RPC service definitions and their Async counterparts.
  • MainApp project contains the implementations of the services defined in the AppServices project.

When I “GWT Compile Project” or “Deploy App Engine Project“, the “shared” or “server” code of the dependencies is never copied to the war/WEB-INF/classes folder. Only the compiled JavaScript code is compiled.

So, I thought of automating all the stuff using Ant files.

GWT app-creator tool creates the “build.xml” for Ant and for GAE, there are guidelines on the web, for example here.

At an outline, the following needs to be done during the project build:

  • Enlist all the dependencies in order.
    If there are five projects – P1, P2, P3, P4, P5 – my script cannot identify which one depends on the other. It’s the developer’s responsibility to write them in the order so that independent or least dependent projects are listed and hence compiled first.
  • Compile all the projects in order with appropriate class-path…
  • Make a jar comprising of all the compiled class files and push it to war/WEB-INF/lib folder.
  • Compile the GWT module, as usual
  • Publish the application to GAE, as usual

I have made used of the following Ant tasks:

  • foreach from ant-contrib. You can download the jar from Sourceforge. Project home page is at http://ant-contrib.sourceforge.net.
  • Standard property, mkdir, delete, javac, copy etc tasks
  • App Engine task appcfg.

First step is to define a property, I named it projects, enlisting all projects in order.

<property name="projects" value="AppCore,AppServices,AppOne,AppTwo" />

Next is to expand the default task javac to compile all the dependencies as well. To take care of dynamic classpath entries arising because of the multiple projects, I copy all the class files to the <main-app>/war/WEB-INF/classes folder and that’s my standard classpath.

<target name="javac" depends="libs" description="Compile java source">
  <foreach list='${projects}' param="project.dir" target='compileDeps' />
  <mkdir dir="war/WEB-INF/classes" />
  <javac srcdir="src" includes="**" encoding="utf-8" destdir="war/WEB-INF/classes"
    source="1.5" target="1.5" nowarn="true" debug="true" debuglevel="lines,vars,source">
    <classpath refid="project.class.path" />
  </javac>
  <copy todir="war/WEB-INF/classes">
    <fileset dir="src" excludes="**/*.java" />
  </copy>
</target>

What this target does is compile all the dependencies before compiling the main application project.

Defining the compileDeps target is more straightforward now, noting that we have a helping hand from foreach task. Here’s how it looks like.

<target name='compileDeps' description='Compiles dependencies'>
  <property name="pdir" location="../${project.dir}" />
  <property name="cdir" location="." />
  <mkdir dir="${pdir}/war/WEB-INF/classes" />
  <javac srcdir="${pdir}/src" includes="**" encoding="utf-8" destdir="${pdir}/war/WEB-INF/classes"
    source="1.5" target="1.5" nowarn="true" debug="true"
    debuglevel="lines,vars,source">
    <classpath refid="project.class.path" />
  </javac>
  <copy todir="${pdir}/war/WEB-INF/classes">
    <fileset dir="${pdir}/src" excludes="**/*.java" />
  </copy>
  <copy todir="${cdir}/war/WEB-INF/classes">
    <fileset dir="${pdir}/war/WEB-INF/classes" excludes="**/*.java" />
  </copy>
</target>

This task compiles the dependencies, and copies all resources (non-java files) from the corresponding src folder and all binaries from the corresponding war/WEB-INF/classes folder to the <main-app>/war/WEB-INF/classes folder.

I further expanded the gwtc target to make a jar file rather than individual class files in the WEB-INF folder. Here’s the code for that.

<target name="gwtc" depends="javac" description="GWT compile to JavaScript">
  <jar destfile="war/WEB-INF/lib/main-app.jar">
    <fileset dir='war/WEB-INF/classes' />
  </jar>
  <delete dir="war/WEB-INF/classes" failonerror="false" />
  <java failonerror="true" fork="true" classname="com.google.gwt.dev.Compiler">...</java>
</target>

And that’s it!

Publishing the application to GAE requires no special change. All files are right there! Just use the <appcfg> task.

Google App Engine – Dealing with large number of files

One of the most common problem that one has to deal while deploying a large scale application at Google App Engine is the file limit – in terms of number of files.

As per the issue reported at http://code.google.com/p/googleappengine/issues/detail?id=161, the limit has been increased from 1000 to 3000, but there’s catch. Any static file (non-JSP files outside the WEB-INF folder, unless explicitly excluded in appengine-web.xml) are counted twice. Why? They’re pushed into a special folder __static__ and all such files are counted twice.

I looked at deploying the SmartGWT showcase on GAE and the “appcfg upload” gave an error about having over 9000 files. I was shocked! Literally.

And then I came across a simple yet powerful idea – zip all the static files and use a servlet to extract and serve on the fly. The solution servlet was uploaded by sindhom at http://code.google.com/p/googleappengine/issues/detail?id=161#c68.

Take that as a base, I made an update – again simple yet probably very useful. Look at the “If-Modified-Since” header and respond with with a 304 or 200 depending upon if the entry in the zip file is updated or an old one. There’s one thing hardcoded in the my solution – the Cache-Control / Expires timeline is hardcoded to 365 days, and it can be easily moved into the configuration.

The solution is mentioned at http://code.google.com/p/googleappengine/issues/detail?id=161#c88. The SmartGWT showcase is available at http://1.latest.mastergaurav.appspot.com, along with the complete code of the servlet at http://1.latest.mastergaurav.appspot.com/UnzipperServlet.java.txt.