In December Amazon launched a new A/B testing service for Android apps on the Amazon Appstore. Integrating A/B testing, particularly for in-app purchase related events, in the store portal is a welcome addition. Slightly disappointing considering this comes from a store provider is that the A/B testing service does not support testing different copy or icons on the storefront itself, purely in-app A/B tests, for which there are already third-party alternatives.

Not a “leaner” alternative store

When we looked at the developer platform “lean factor” for Android previously, we were only considering Google Play. Android came out on top due to the ability to publish new versions to the store almost immediately, enabling much faster iteration. However, the Amazon Appstore does have a review process which introduces significant publishing delay. As such even with the convenient addition of A/B testing baked into Amazon’s store, Google Play still offers a better environment for implementing lean development.

Testing in a smaller market

That said, there is at least one interesting use of this new service. Amazon’s Appstore has historically had a much higher proportion of users paying for downloads and in-app purchases than on Google Play – so much so that in early 2012 many developers made more money on Amazon’s store despite the much smaller audience. A smaller market with a high proportion of paying users is ideal for A/B testing improvements with the most important user group, particularly improvements to in-app purchase flows. Testing new variations on Amazon’s store and then rolling the successful ones out to a Google Play version might be a useful strategy for increasing revenues.

But how much smaller?

A final question is how much smaller is Amazon’s store in terms of downloads? They are notoriously silent when it comes to absolute numbers and even the relative numbers they give are ambiguous. In the press release for the A/B testing service linked above they say that app downloads have grown 500% over the previous year. This was in a release at the beginning of December with no fixed reference point given for the comparison. As can be seen from the relative growth chart produced by Distimo at the same time last year below, the launch of the original Kindle Fire at that time produced similarly massive growth.

Dependending on where the comparison is made, a 500% increase could be anywhere from slightly behind January’s download level to 6 times the December 2011 peak. Until they provide some concrete numbers the best advice is to try it and see what your own results are like. The only major difference we’re certain of between the app buying users on the two stores is a higher proportion of tablet users at Amazon’s.