Author Archives: Simon

My $18,534.25 Amazon Bill!

Over the past couple of weeks, we’ve seen some aggressive pricing cuts for cloud services. First, Google announced that they were going to slash prices of compute and storage by around 60%. Amazon quickly followed matching or beating the reductions, and Microsoft announced a similar move just last week.

Imagine my horror then, when I logged into my account on March 25th and saw the following bill:

Aws bill

After more investigation, I found that someone (or some group) had hacked into my account and instantiated a large number of extra large EC2 images across multiple regions. It was a pretty clever hack – they had left my existing images untouched (otherwise I would have likely discovered the issue sooner), changed many of my existing security groups, and had been able to script multiple EC2 nodes running across all zones, each of which was racking up the best part of $500 per hour. Had I not had sticker shock, I would have likely have taken a snapshot of one of the instances to see what these machines were doing – although having read reports of similar hacks, and looking at the types of machines, I’m guessing that they were probably being used for some kind of Bitcoin/Litecoin mining.

The real question however was whether I was going to be liable for the charges, which came out at $19,438.95 by the end of the billing cycle. There’s a chance that the result of the hack was due to one of my secret keys being leaked through a Github repo, and I didn’t have any of the alarms setup on my account that would have notified me sooner – so there was definitely a sense of responsibility that I could have done more to prevent this.

It did take a couple of weeks to reach to right support team in Amazon, but once I did, the results were phenomenal. Not only did the team work with me to prevent this happening in the future, but they also reversed all of the charges within a day, wrote off the entire month’s bill (which included my valid charges), and gave me a $500 credit for any charges on next month’s bill.

My three takeaways from this story:

1. Cloud credentials matter! I’m still not 100% sure how my account was compromised, but I now have a new level of respect for the importance of securing my account, especially when it comes to symmetric keys that have access to unbounded resources.

2. Setup alarms! I had always taken a passive approach to setting any types of alarms on my account – the importance of knowing when things are going wrong is paramount for any cloud service, no matter how small the deployment.

3. Amazon’s support rocks! I was very impressed with how the AWS team went above and beyond in order to put things right on my account – echoing the same level of customer service that I’ve become accustomed to as a retail customer.

  • Facebook
  • Twitter
  • Delicious
  • LinkedIn
  • StumbleUpon
  • Add to favorites
  • Email
  • RSS

Bluetooth Issues on Mavericks

I’ve been having some issues recently with the Bluetooth stack on my Mac acting strangely – causing Bluetooth devices to drop at random times and on occasions, the default mouse and keyboard will stop working even though the machine is running fine. One of the symptoms of this has been the bluetooth preferences pane showing “do not localize” when things become corrupt.

I think (well, I hope) I’ve solved it by performing an SMC and NVRAM reset on the Mac. In case other are seeing similar symptoms, I wanted to share how you do these resets:

SMC Reset: Power down the Mac. Hold down Shift-Control-Option and the power button for a few seconds. Not much will happen. Release all of the keys, and power on normally.

NVRAM Reset: Power down the Mac. Hold Command, P and R down and power on the Mac. The Mac will boot for a few seconds and then reboot. Release all of the keys at this point, and boot normally.

After doing these resets, the Bluetooth stack seems a lot more stable.

  • Facebook
  • Twitter
  • Delicious
  • LinkedIn
  • StumbleUpon
  • Add to favorites
  • Email
  • RSS

Automated Web Testing using JavaScript

The process of creating automated tests for Web applications has been around for many years. Vendors such as HP, Parasoft, QFS, and even Microsoft have test software that can be used to create and run automated tests. Over the past couple of years however, we’ve seen an industry-trend towards open source Web testing solutions based on JavaScript. Such solutions have the advantage of being easily modified, free to download, very modular, supported by a vibrant community, and (given the popularity of client-side JavaScript) tests can often be written using the same language as the Web application. 

For this blog post, I wanted to share some of our observations at Neudesic, and some of the tools that we’ve had success with during recent projects.

Unit Tests or Assertions?

Before considering any unit test, one of the questions we try to ask is whether the same functionality can be provided by an assertion. An assertion is, in its simplest form, code in your application that asserts that some condition is true – and if it’s false, stops the application and/or reports an error message.

The easiest way to perform an assertion in JavaScript is with the console.assert function. This is supported on the latest version of most major browsers (Chrome, Safari, and IE 10 upwards). A simple use of console.assert might look as follows:

var myMethod = function(x){
  console.assert(x, 'X should not be undefined');
  /* rest of method goes here */
};

As shown above, this is the simplest form of an assertion, yet quite effective. More importantly, it also negates the need to have a redundant unit test that checks whether the value of x is null, resulting in fewer tests that simply do assertion checking. There are many developers who use assertions during development and then strip out the assertions for production. While this has some merit, a more productive approach can be to create your own assertion function. This might look something like the following:

var myMethod = function(x){
  assert(x, 'X should not be undefined');
  /* rest of method goes here */
};

As shown above, instead of calling console.assert we call a custom function. This has the advantage that in development mode we can inject an assert function that writes to the console or adds a breakpoint in code, whereas in production mode we might choose to display a more friendly error message to the user and potentially log the crash via an analytics service.

QUnit

Assertions are useful for performing simple checks on methods, but there will likely come a time where you need to perform more rigorous tests of logic, using a unit test. QUnit is a popular JavaScript framework for unit testing, whose origins come from jQuery and jQuery UI. QUnit works by defining a set of tests that are run within the QUnit test runner (which itself is just a Web page). Unit tests are simple, and follow a format similar to the following:

test('a basic test example', function() {
  var value = 'hello';
  equal(value, ‘hello', 'We expect value to be hello');
})

If you are new to unit testing in JavaScript, and starting out by writing simple single page tests, QUnit is a good way to get started.

Jasmine and Karma

While QUnit provides a good introduction to unit testing, it is easy to quickly reach a limit, especially if your application has a lot of tests nested in a hierarchy. In addition, QUnit does not typically provide any CI (Continuous Integration) functionality out of the box. As you increase your familiarity with unit testing using JavaScript, it is well worth checking out Jasmine and Karma.

Jasmine is a BDD-style framework for writing tests. While similar to the QUnit construct, the syntax tends to be cleaner and more widely adopted in the JavaScript community:

describe('a basic test example', function() {
  it('tests that true is always the truth', function() {
  expect(true).toBe(true);
  });
});

As can be shown in the example above, the describe, it, and expect keywords work with unit tests, but also apply to BDD style tests also – making Jasmine a good candidate for writing tests of both types. Jasmine is just a language syntax and by design doesn’t offer any test runner implementation. Fortunately, there are a number of test runners available that can run Jasmine tests. One such framework gaining adoption is Karma, a test framework developed by the AngularJS team at Google.

Karma, which used to be called Testacular (insert your joke here about why the name changed!), is a flexible test framework that can be used to call unit tests written in Jasmine and other frameworks. Karma is very lightweight, which means it works well in CI (Continuous Integration) settings, even to the point where it’s possible to have Karma invoke a series of tests after each file save on your development machine. While there are many different test runners capable of executing unit tests, I believe Karma adoption will continue to grow, and this should be something that you should look at for your own testing needs.

Selenium

Unit testing is useful for testing discreet logic in your application, but falls short of telling you whether the application is working correctly for the user in their environment – i.e. through their browser. To do this we need to turn to integration testing – testing the end to end operation of your Web application. Selenium is an open source project that has been in development for the past 10 years, originally developed by ThoughtWorks as a replacement for Mercurial (hence the name, as Selenium is often used to cure Mercury poisoning!).

Today, Selenium has three major components: Selenium IDE, Selenium WebDriver, and Selenium Grid.

Selenium IDE is a browser extension for Firefox that enables you to record a series of actions and test assertions via a “recording function” in the browser. By default, the test gets written to a HTML-based test suite, and can be replayed at any time. While Selenium IDE is a useful tool for investigating the underlying operations of the testing platform, it is not well suited for use in production – the tests don’t support inheritance, which makes management of the tests difficult, especially when things change in your application. Moreover, the tests have to be run through the Firefox browser, which makes true automation difficult.

Selenium WebDriver (which used to be Selenium RC) is a server-side version of the Selenium testing platform that does support scripted tests. WebDriver (based on the emerging WebDriver spec, hence the name), runs as a Java service and accepts incoming TCP socket connections from test clients. Upon receiving a connection, the service invokes a browser, runs the tests, and reports status back to the client via the socket. Any test platform that implements the WebDriver specification can issue tests to a Selenium server.

Selenium Grid is a service where multiple Selenium servers can be clustered to handle a high volume of tests. It is often used by service providers (such as SauceLabs) who are offering testing services to multiple clients.

For many projects, Selenium WebDriver makes the most sense. In order to use WebDriver with Jasmine however, you still need to have a test runner (remember that Jasmine doesn’t provide any test platform). Karma is not a fit here as it is best suited for unit tests, not end-to-end tests (as it happens, the documentation actually discourages the use of Karma for integration testing). To solve this, we turn to another testing framework from Google, Protractor:

Protractor is an end-to-end test framework, typically used to test AngularJS applications – although any Web application can be tested as protractor contains methods to call the underlying WebDriver implementation. It is built upon a library called WebDriverJS, a JavaScript-implementation of the WebDriver client, which makes it wire compatible with Selenium WebDriver. Protractor is simple to install, and supports Jasmine as it’s primary test language. A test using protractor might look as follows:

describe('test hello world home page', function() {
  it('should greet the named user', function() {
    browser.get('http://www.angularjs.org');
    element(by.model('yourName')).sendKeys('Simon');
    var greeting = element(by.binding('yourName'));
    expect(greeting.getText()).toEqual('Hello Simon!');
  });
});

As you can see above, this simple test will open a Web page, look for a data binding called ‘yourName’, send a name to it, and then assert that the greeting replied is correct. You should note that the model and binding above are specific to AngularJS, and (if you are not using AngularJS) can be replaced with similar lookups based on HTML ID, CSS, or XPATH query. Using the protractor test runner, this test can be sent to an instance of Selenium, which in turn will invoke a new browser, run the test, and report back the results to the client.

IDEs for Creating Protractor Tests. While creating tests is a manual process today, there are several open source projects with a goal of being able to generate protractor-based tests from a more visual interface. One such tool is Selenium Builder, which while still in Alpha, looks very promising.

Using Page Objects. While Jasmine and Protractor offer a very quick way of creating and running tests, you often need to be careful to manage the tests that you create. If you make a simple change in the UI, you want to avoid a situation where you need to change many tests. One way of overcoming this is to use Page Objects when using Protractor. A Page Object is, by definition, a JavaScript class that defines methods on a page.

These methods might be AddCustomer, DeleteCustomer, ModifyCustomer, etc. The tests then call these page objects to perform the function. For example, AddCustomer(valid), AddCustomer(invalid), AddCustomer(invalid2), etc. This approach offers many advantages, especially if the page changes. If a change to page occurs, only the page object has to be changed once, which makes maintenance and re-use throughout the tests more predictable.

PhantomJS. Finally, as we discussed previously, Selenium works by invoking a new browser process in order to run tests. While this is useful on your development machine, there are times where you may want to invoke tests that do not have access to a fully installed browser – for example, if you are doing server-side tests on a Linux or a Docker instance without a UI installed. To accomplish this, PhantomJS can be used. PhantomJS is a headless version of WebKit written in Qt, and can act as a browser without any of the UI overhead associated with actually launching a browser.

PhantomJS can be invoked in one of two ways – either plugged directly into Selenium as an alternative browser process – or, because PhantomJS implements GhostDriver (wire level compatibility with WebDriver), it can be called directly from Protractor. PhantomJS is still in early development stages (and at the time of writing being ported to Qt5), but looks to offer a lot of potential for developers who are looking to do integration testing in a headless environment.

  • Facebook
  • Twitter
  • Delicious
  • LinkedIn
  • StumbleUpon
  • Add to favorites
  • Email
  • RSS

File > New > Presentation

About a year ago, I set out on a journey to write a book on presentation skills, aimed primarily towards software developers and other technical professionals. My reasoning is that too many presentations in our industry still fail to connect with their audience, don’t tell a story, and end up as a set of mind-numbing slides of bullet points that put people to sleep.

The book, aptly titled ‘File > New > Presentation’, was released last November, and reviews have started to appear in the New Year – a few on Amazon, and this one by Josh Holmes, who presented about the subject at CodeMash earlier this month.

Book Cover

If you are a developer (or other technical professional), looking to improve their presentation skills, I hope you’ll check out the book. You can find it on Amazon in both Kindle and paperback format, iTunes, and Google Play.

  • Facebook
  • Twitter
  • Delicious
  • LinkedIn
  • StumbleUpon
  • Add to favorites
  • Email
  • RSS

iBeacon Demo at Gartner AADI

Last week, I joined my colleagues from Neudesic at the Gartner AADI (Application Architecture Development & Integration) Summit in Las Vegas. One of the neat things that we were showing was a demo of iBeacon technology, in order to help locate and pinpoint individuals within indoor environments where GPS is not an option. 

To achieve this, we placed a selection of iBeacons around the expo floor (for the iBeacons we used RedBear’s BLE Mini boards, flashed with their iBeacon firmware).

Red Bear BLE Mini

Using Bluetooth LE (a.k.a. Bluetooth Smart), each beacon transmits a unique ID (made up of a major and minor value) on a particular “frequency” (known as a proximity UUID). iBeacon supported mobile applications are then able to “tune in” to this proximity UUID, and based on the power signal for each beacon, determine which area of the floor a user is located closest to.

 

Gartner AADI screenshot

Using this information we created a mobile app that reports on all of the beacons within range on the expo floor. As you can see in the above screenshot, we have 5 iBeacons in range, listed in order of proximity, with an estimated distance calculated by the power signal. As you can likely gather from the data, we were located in the Neudesic booth at the time this screenshot was taken, with the theater, and a selection of other booths in range.

For the show, we developed two versions of the application – one for iOS and one for Android. Both are native mobile applications written using Xamarin, using CLLocationManager for iOS, and Radius Network’s binding for supporting iBeacon on Android. The Radius implementation is especially interesting in that the beacon detection runs as a background service, polling for new iBeacons and raising intents to the UI as necessary (even though Bluetooth LE is a lot more responsible with power vs. regular Bluetooth, we still need to be careful not to kill the battery when using this however).

While this is neat to show just as a mobile application, we wanted to take it one step further and demonstrate how this could be applied in the real world with Neuron, an ESB product from Neudesic. For our expo demo, Neuron provided a backbone infrastructure to allow the mobile app to resolve a name for a beacon ID (think of his like DNS for iBeacons!) and also give a scalable way for users to “check in” to an iBeacon the same way that they would check in via Foursquare or Facebook.

Neuron process flow

As shown above, we developed a process flow using Neuron to accept incoming messages from the mobile application, and then provided logic to determine whether the device was trying to acquire a name for a beacon, or whether the user had walked in to the area of a beacon and wanted to “check in”. The benefit for using Neuron in this situation vs. just propping up a single web service is that our application can now scale to hundreds of thousands of concurrent mobile clients without needing to make any major adjustments on the server-side.

If you were able to stop by the booth at the Gartner summit, I hope you enjoyed the demo and the conversation. If you weren’t able to attend, but would like to know more about how Neudesic is working with various organizations on iBeacon technology and/or Neuron, feel free to drop me a line.

  • Facebook
  • Twitter
  • Delicious
  • LinkedIn
  • StumbleUpon
  • Add to favorites
  • Email
  • RSS

Building a WebRTC Client for Android

*** Update:  As a few readers have pointed out, the libjingle source has now been merged into the main WebRTC branch (https://code.google.com/p/webrtc/source/browse/trunk/talk). As a result, some of the instructions here will need to be adjusted.  I’m going to leave the post as is, but bear this in mind if you are following each of the steps.  Thanks.  ***

If you’ve been following any of the recent developments with WebRTC, you’ll know that the majority of samples and example code available today target the Web browser (typically Chrome or Firefox).  While this is useful to get up to speed, IMO one of the most powerful applications of WebRTC will be for mobile devices.  Despite this, getting WebRTC running on mobile can be somewhat challenging.

While Chrome Beta for Android supports WebRTC, any true mobile client will always need some kind of native application (in order to receive incoming calls in the background).  In this post, I’ll be showing how to walk through the minefield of compiling WebRTC for Android in order to build your own demo app that you can expand upon.

NewImage

Configuring your Build Environment

In order to build Android WebRTC, you are going to need a machine running Linux.  For the purposes of this article, I would recommend some kind of virtual image (VMWare or VirtualBox) running Ubuntu Server 12.04.2 LTS.  When you build your VM image, ensure that you use the 64bit version of Ubuntu, and that you allocate a minimum of 2Gb RAM to the image (any less and you may well see the compiler running out of memory).

After you get your Linux VM up, the first thing is to install the required build tools.  First, the git and subversion clients:

sudo apt-get install git git-svn subversion

Then, you’ll need the Chromium depot tools – these tools are required to build the WebRTC bits, much of which derives from various parts of the Chromium project.  To install:

cd ~
git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git

After the tools have been brought down, you’ll also need to add them to your path:

echo export PATH="$PATH":`pwd`/depot_tools >> ~/.bashrc

Once this is done, log out and log back in to the server.  From the command line run:

gclient

If all is working, you should see a list of available client commands.  If you see “command not found”, edit the ~/.bashrc file and double check your path is set correctly.

The final part of configuring your build environment involves installing some required libraries and compilers.  To perform this, use the following command:

sudo apt-get install g++ pkg-config gtk+-2.0 libnss3-dev

Once this is complete your machine should be in a position to start pulling down the WebRTC code.

Obtaining libjingle Source Code

In order to build the Android WebRTC client, we are going to pull down and compile the “libjingle” project.  Libjingle is a set of components to interoperate with Google Talk, but also contains (at the time of writing) the most complete and functional Android client.

To pull down the source, follow these instructions:

cd ~
mkdir libjingle
cd libjingle
gclient config http://libjingle.googlecode.com/svn/trunk

After you’ve run the config command you should notice a new file called .gclient in the ~/libjingle directory.  Edit this file (using VIM or nano) and append the following to the end of the file:

target_os=['android','unix']

This will instruct gclient to pull down the required third party libraries and other tools for an Android build of libjingle.

With this file modified, run the following command from the ~/linjingle directory:

gclient sync --nohooks

This will start a large sync of the liblibjingle source code, which will likely take some time.  A great opportunity for a coffee or break at this point!

Installing the Oracle JDK

Before we can compile libjingle, we’ll need to install the Oracle JDK.  Although the majority of the library is native (and uses the Android NDK) this is required to build the JAR and APK file at the end of the build.

To install, go to the Oracle JDK downloads page (http://www.oracle.com/technetwork/java/javase/downloads/index.html) and download the latest JDK 1.6.  Note that JDK 7 will not work and you will get errors later on.  For the purpose of this tutorial I have been using Java SE Development Kit 6u45.

As you are running server, you may need to download the JDK on another machine (where you will need to accept the license agreement) and then copy the .bin file to your Linux server (using either curl, wget, or VMWare shared folders).

Assuming the .bin file is in your home (~) directory, to install the JDK, perform the following:

cd /usr/lib/jvm && sudo /bin/sh ~/jdk-6u45-linux-x64.bin -noregister

This will extract the JDK to the /usr/lib/jvm directory.  After this is complete, we need to set the defaults to use the Oracle VM as opposed to the OpenJDK VM (which is installed by default).

sudo update-alternatives --install /usr/bin/javac javac /usr/lib/jvm/jdk1.6.0_45/bin/javac 50000
sudo update-alternatives --install /usr/bin/java java /usr/lib/jvm/jdk1.6.0_45/bin/java 50000
sudo update-alternatives --config javac
sudo update-alternatives --config java

Finally, to overcome a small bug in the libjingle gwp compile scripts, we need to create a symbolic link:

cd /usr/lib
ln -s jdk1.6.0_45 java-6-sun

After you’ve done this, run the following command:

java -version

If you see “Java HotSpot build 1.6.0_45″ instead of “OpenJDK”, the correct version of Java is install and you should be in good shape.

Preparing for Compilation

Before we compile, there are a few things that are required.  Firstly, we need to install some 32 bit compatibility libraries so that aapt and other Android compilation tools will function correctly.  To install, run the following command:

sudo apt-get install ia32-libs

Next, edit ~/libjingle/trunk/third_party/webrtc/build/common.gypi

Navigate to the line that contains:

enable_tracing==0

and replace with:

enable_tracing==1

This will prevent the Android client from crashing (when it realizes it doesn’t have a default trace file) and will make debugging the Android application a whole lot easier!

Finally, perform the following commands to complete preparation:

cd ~/libjingle/trunk
./build/install-build-deps-android.sh
. ./build/android/envsetup.sh

This will correctly setup the android dependencies.  Note the leading period on the envsetup script – this is very important as we need the variables to be set for the rest of the session.

Assuming these commands ran without errors, now run:

gclient runhooks

This command will generate the required ninja gwp compilation scripts.  After this, run:

android_gyp

You may get an error indicating that content.gyp could not be found – it is fine to ignore this for now.

Compiling!

If you’ve reached this stage, congratulations!  We can now try to compile!  To do this, run the following command from the ~/libjingle/trunk directory:

ninja -C out/Debug -j 10 AppRTCDemo

This command is instructing the ninja compile tool to generate a debug version of the build, using a maximum of 10 concurrent build threads.

Depending on your machine, compilation will likely take about 10-15 minutes (time for that other coffee!).  Assuming everything went well, you should see some output in the ~/libjingle/trunk/out/Debug folder:

AppRTCDemo-debug.apk

This is the main demo APK.

libjingle_peerconnection.jar

This is the JAR for libjingle.

libjingle_peerconnection_so.so

This is the ARM-based native library.

Running the APK on your device

To run the APK on your device, copy the file to a machine with ADB installed (and your device connected) and run the following command:

adb -d install AppRTCDemo-debug.apk

If all goes well, you should see a new application called “AppRTC”.  To test the application, launch a new browser on your desktop (not on the phone!) and navigate to http://apprtc.appspot.com.  Enable access to your camera and microphone and note the room number of the conference window.

Now, launch the AppRTC application on your device and enter the room number as part of the URL (ensure that there are no spaces).  If all goes well you should have an established connection with video and audio between the browser and the mobile device!

If you run into issues (and this is quite complex, so it’s usual) check ADB logcat for any errors and exceptions that are being produced by the AppRTC application.  Although functionally, this code is working, at the time of writing the TURN server being hosted on Google’s Compute Cloud is having some issues – so I had to dive into the code and configure the application to use an alternative – but that’s probably a good excuse for another post sometime in the future!

Either way, I hope this was useful and good luck!

  • Facebook
  • Twitter
  • Delicious
  • LinkedIn
  • StumbleUpon
  • Add to favorites
  • Email
  • RSS

Designing a Web API for Mobile Apps

At Neudesic, almost every mobile project that we’ve been involved in has had some dependency on a Web API.  This has included APIs that we’ve had to create from scratch, as well as ones that already exist that the app has to consume.

As you can imagine, over the past few years, we’ve seen a fair share of good and bad API design.  In this blog post, I wanted to share some of my observations, thoughts, and questions that I ask of a well designed Web API, especially one that will be called from a mobile app.

1.  Are the basics being met?

First, the basics.  Without question, every Web API should be stateless.  There should be no session state, cookies, or server-side values used to hold any state of any kind.  Adding state adds complexity, and limits the ability for the API to scale, which for a mobile application that could reach millions of users is something that we want to avoid.

Also, endpoints of the API should be exposed through SSL by default.  SSL is easy to setup, performant, and should be enabled for any API that we either consume or create.  As you may have observed, using SSL as default seems to be a direction in which many other APIs are heading.

2.  How do we authenticate with the API?

Authentication is critical to a Web API, and (in my opinion) is one of the common pitfalls.  As a rule, user credentials should never be passed as part an API call.  Take the following example:

GET /accounts?username=simon&password=simon

Doing this is bad for three reasons:

Firstly, although the URL gets encapsulated as part of the HTTPS session, it is likely still visible in any logs on the Web server.  This often also applies to user credentials passed as HTTP headers.

Secondly, it makes debugging with users and other developers awkward (because you often need to ask for their username and password).  This is especially bad if the credentials are corporate accounts used for other systems.

Finally, and most importantly, these types of credentials typically have a long shelf life.  If the call is ever compromised, there’s a good chance that it can be replayed back to the service up until the password is changed, which could be many months (if ever at all).

To overcome this, some APIs use an application key or some other token derived from a HMAC algorithm.  This may work for some scenarios, but unfortunately if the key is exposed, it can be difficult to revoke.  This is especially true if the key has been embedded in a mobile app running on thousands of devices.

Fortunately, to overcome both of these issues, there is OAuth 2.0.  OAuth 2.0 works by having the user pass a set of credentials (typically a username and password) to an authentication service.  Assuming the credentials are valid, the user/application/consumer receives back an access token.  This access token is then passed as a HTTP Authorization Header to the Web API to verify the authenticity of the request.  Moreover, this access token has an expiry (I find an hour to be a good time frame) so that if someone were to get hold of the token, their usage of the API is limited to this timeframe.  (You can read up much more about OAuth 2.0 here)

There’s no doubt that implementing OAuth 2.0 involves more work, including setting up an authentication API and handing the lifetime of the token, but the end result is an API that is more secure and will also reflect the security model used by many others (e.g. Facebook, Google) – which means that you can even use these third parties as identity providers if you so choose.

3.  Is the API really using REST?

I’ve seen many examples of people thinking that they have a REST API when really they don’t.  Correct use of REST is about nouns, not verbs.  Let’s take this URL for example:

GET /GetAccountBalance?account_id=1234

Although the above URL is accessed over HTTP, it’s really not a REST API.

Using REST to it’s true intention means combining HTTP VERBS together with nouns or entities in the URL that represent the data you are exposing.  Instead of the previous URL, a correct REST syntax would be the following:

GET /accounts/1234

To create a new account we would use a HTTP PUT (together with a payload with the new account information)

PUT /accounts

To delete an account, we would use:

DELETE /accounts/1234

This noun-based approach can also work with hierarchical data.  For example:

GET /accounts/1234/transactions/50

Will return the a transaction (with id of 50) for Account 1234.

Overall, a good understanding of REST, together with a focus on exposing nouns instead of functional methods will go a long way to create a more more usable and accepted API.

4. How should we consume this API?

If you are dealing with an API that exposes a lot of entities, in addition to exposing generic REST endpoints, there are typically six things also worth considering:  Sorting, Searching, Filtering, Pagination, Helpers, and Partial Responses.

Sorting.  Should the API return a sorted list of data to the consumer/application?  If so, a sort parameter on the noun can be useful:

GET /accounts?sort=id

GET /accounts?sort=-id

As shown above, a leading hyphen to indicate ascending or descending sort order can be a great timesaver (and often negates another query string parameter for sort order).

Searching.  Similar to sorting, providing a way for consumers to search entities can be useful.  The use of the “q” parameter for search is somewhat standard:

GET /api/accounts?q=acme

Filtering.  Another useful pivot for any REST based API is the ability to filter on a particular noun.

GET /accounts?filter=balance%3E500

(You can choose to use the URL encoded character for > as shown above, or I’ve seen many other APIs use gt, lt query parameters).

Pagination.  A type of filtering, especially useful for large datasets.

GET /accounts?limit=50&offset=25

This above call will get the next 50 accounts, starting at the 25th entry.

Helpers.  With many APIs there are a number of common requests.  Maybe it’s the list of top ten customers, or the best sales people of the month. Putting these common requests as “helpers” into the API can be very useful for consumers, and can also help reduce the “chattiness” of the API by reducing the number of repeat requests.

GET /accounts/top-ten

Partial responses.  Finally, many consumers of the API (especially mobile applications) will want only a summary set of data.  This can be useful to build a list of items (in a master/detail view), without having to send the entire details for each item.

GET /accounts?fields=id,name,balance

Of course all of the above parameters can be combined as required.

GET /accounts?fields=id,name&sort=id&limit=100&offset=50

5.   What will the API return?

For the majority of APIs, especially those that will be consumed from a mobile application over a potentially slow connection, returning JSON is always good practice.  Compared to XML, data returned in JSON format will likely be more lightweight, and will require less parsing and processor overhead on the device.

With that said, there are cases where other formats might be required – for example, a legacy system that is already expecting data in XML.  In which case, you might want to consider allowing the consumer to specify what type of data to return either through the HTTP Accept header or through a URL action (useful if you anticipate doing a lot of debugging).

GET /accounts?format=xml

There has also been a lot of talk recently about HATEOAS (Hypermedia As The Engine Of Application State), a term coined by Roy Fielding.  While there are many articles and presentations that explain Roy’s initial intentions, for the purpose of this blog post (and my own sanity), HATEOAS in a Web API referring to the use of links that instruct the consumer where to go for related information.

For example, let’s imagine we made the following API call:

GET /accounts/1234

We might receive the following response:

{ "account_id" : "1234", "balance" : "100.90" }

With a “HATEOAS-compliant” Web API, we may also receive embedded links.  For example:

{ "account_id" : "1234", "balance" : "100.90", { "_links" : { "transactions" : { "href" : "/accounts/1234/transactions" } } } }

As you can see above, the API returns the data for the account, but also returns a link to the API call that will return all of the transactions for that account.  Think of these links as helping the consumer navigate to other related API calls.  (Incidentally there are a number of JSON formats for doing this, although I would recommend JSON HAL

6.  Are the methods of the API consistent? 

While it’s difficult to recommend what you should name your methods and other parts of your API, the key to success is often consistency.  For example, if you have the endpoints for your accounts here:

GET /accounts

For your invoices, it would be silly to have them here:

GET /order_entry/ledger/invoices_full

In an ideal world (and even one without HATEOAS!), a user should be able to guess what the API should be based on previous usage.  Keeping the paths and names consistent are key to making this happen.

Related to this, choosing the right case for APIs can be very important.  Having these two apis:

GET /accounts

GET /Invoices

will likely lead to issues because of the case mismatch on the entity name.  My recommendation is to use lowercase throughout (then there is no ambiguity) and to use snake case to conjoin words.  For example:

GET /customer_details

Spinal case (using hyphens) is also acceptable, but if you are doing a lot of descending sorting, you may want to be careful.

Finally, in terms of consistency, it’s always nice to be consistent with pluralization:

GET /accounts/1234/invoice

Assuming there are more than one invoice per account, this could also run people into trouble.  I would recommend deferring everything to plural to ensure consistency.

GET /accounts/1234/invoices

7.  How is the API versioned?

Versioning is important, especially if there are breaking changes in production environments.  There are a couple of ways to achieve this:

For large volume APIs where version consistency is critical, I would recommend placing the version information as part of the API call.

GET /v1.0.0/accounts

Versioning by using the URL makes it very explicit as to the version that the consumer is using.

For less critical systems, or for APIs where breaking changes are going to be rare, I would recommend that consumers pass an optional version number as part of the HTTP header.  If the version number is passed as part of the post, the consumer gets a specific versioned response, otherwise they’ll be receiving the latest version.

In addition to version numbers, I always like to see specific environments affiliated with the URL.  This is most easily done as part of the host subdomain, as it will likely correspond with the physical or virtual machine that the API is hosted from:

GET https://dev.example.org/accounts

GET https://uat.example.org/accounts

GET https://prod.example.org/accounts

The above makes it very clear whether I’m hitting the development, UAT, or production version of the APIs when I make my calls.

8.  How is the API documented?

If you have a well designed API, you do not need to spend hours of time documenting the API in a Word document.  If anything you are going to end up with a Word document that will become quickly out of date.  In terms of documentation, there are two things that I find invaluable:

Firstly, mandatory and optional methods and parameters should be documented.  It’s important that consumers understand what they can and cannot pass to the API.  What’s nice is that this documentation can typically be auto generated from the method signatures or comment blocks (which will keep your documentation in sync with your code).

Secondly, sample code to show how to call the API.  A few sample calls for each method can be a life saver and much more valuable than reams of documents.  In these samples, show how the request should be formatted and what the response looks like.

9.  What does the API return when things go wrong?

Returning useful error messages to consumers of your API is really important.  You should think about two types of error messages – ones that can be expressed with HTTP result codes, and ones that cannot.

For the ones that can be expressed through a result code, simply return the result code with an optional body of information (in JSON format).  For example, if the access token has expired, return a 401 HTTP error code, and maybe some JSON payload to help debugging.  Also, if any part of the system is down (e.g. the database connection can’t be established), I would recommend returning a 500 for clarity.  With any HTTP result code, remember to pass the right one.  A good rule of thumb is that result codes in the 400′s typically indicate an error with the client, whereas codes in the 500′s means that something has gone wrong on the server.

For errors that can’t be expressed through a HTTP result code, you should be returning a JSON payload that contains a minimum of two pieces of data – a unique error code for the issue, and a message aimed for the consumer/developer of the application to help them debug.  For example, if the consumer tried to create a new account without including all of the mandatory fields, this would be a useful error to return:

{ "error" : 16, "debug" : "The mandatory field for balance was missing in the request" }

Some recommend returning a user message also, which can be useful.  Others use the error code to create their own message to pass to the user.

10.  Finally, what else should we be thinking about for using the API in production?

There are many considerations for using APIs in production – here are just a few:

How are you going to make money from your API?  Are you thinking about a transaction cost per call, freemium, capped model, or something else?  Many of these systems are going to require some kind of API metering – which isn’t necessarily hard, but is definitely something else to consider.

Are you going to rate limit your API?  How are you going to prevent a rogue customer, application, or process, who wishes to call your API hundreds of thousands of times?  Fortunately, there are answers to this – including RFC6585 which specifically deals with rate limiting – but again, something that you should be considering.

Should your API provide cached results? Is this something that can improve the performance for your consumers, and also prevent unnecessary calls to back end databases and other systems?

How is your API going to work over a low bandwidth connection?  Your API might work great on your FIOS line here in the US, but do consumers get the same experience when calling the API from a J2ME application from a cell phone in the middle of Africa?  There are many ways to simulate throttled connections and this should be something that is definitely worth testing for.

Finally, and arguably most importantly, how can you get everything to go through your API?  Instead of thinking of an API as a companion to your existing Web-based applications, what would it take to push everything through this API – treating the API as a single source of truth?  It might sound scary, and undoubtedly it’s additional work to have everything using the API – but a single API that every application uses has the potential to offer a lot of re-use and sharing as you develop your API over time.

  • Facebook
  • Twitter
  • Delicious
  • LinkedIn
  • StumbleUpon
  • Add to favorites
  • Email
  • RSS

Persistent Logging in Android

No code is perfect, but the nasty types of bugs are the ones that happen randomly and infrequently.  On a couple of recent Android projects we’ve been experiencing such bugs, and as you can guess they can be difficult to track them down.

While Android’s logcat can help, using the tool often means that you need the device connected through ADB and a USB cable in order to track the events.  While this is useful in development, this doesn’t work so well when debugging applications that are running on devices in the field.

Rather than setting up your own logging application or service, it’s possible to setup logcat to persistently log to a file on the device or SD card.  To do this, simply open an ADB Shell and run logcat as a background process with the -f option:

adb shell

# logcat -f /mnt/sdcard/extsd/logcat.txt &

As you can see in the above statement, this will create a new logcat process to persistently log every event to a file called logcat.txt on the SD card.  To kill logcat, simply find the PID through the ps tool, and kill the process.

In case you are worried about the size of the log files, logcat also has the ability to do log recycling:

adb shell

# logcat -r 1024 -n 10 -f /mnt/sdcard/extsd/logcat.txt &

This command will create up to 10 logcat.txt files on the SD card (named logcat.txt.1, logcat.txt.2, logcat.txt.3, etc.) up to 1Mb in size.  Once the end of the 10th file is reached, it will overwrite the first.

Finally, this is all great – but if the user reboots the device, the logcat process will be terminated.  To overcome this, simply edit the /init.rc file and add the following:

service logcat -r 1024 -n 10 -f /mnt/sdcard/extsd/logcat.txt

    oneshot

Hope this helps, and happy logging!

  • Facebook
  • Twitter
  • Delicious
  • LinkedIn
  • StumbleUpon
  • Add to favorites
  • Email
  • RSS

Transferring Data via Bluetooth on Android (android-btxfr)

Recently, I’ve been working on some code to transfer images and other data between Android devices using Bluetooth.  While I could have used the Basic Imaging Profile (BIP) of the Bluetooth 4.0 specification, this particular application has specific requirements that would have been difficult to implement using BIP.  To overcome this, I ended up building an application that instead relies on the Serial Port Profile (SPP), exchanging data using the RFCOMM protocol.  

The SPP implementation on Android is nice, but only offers the fundamentals of an InputStream and OutputStream.  While this provides the basics of sending and receiving bytes, it doesn’t handle data integrity, progress updates, threading, or any of the other requirements often needed when sending data between two devices.

The result from my recent work is an Android library called android-btxfr, which I’m happy to share through my Github repository.  android-btxfr is lightweight library designed to send and receive any type of data between Android (API 15 and higher).  It can be used to exchange text, files, photos, videos, sounds, and literally any other type of binary data.  The library supports anything that can be put into a byte stream and includes digest checking to ensure data integrity.

The library exposes two thread types (ClientThread and ServerThread) depending on whether you are sending or receiving data.

Receiving data is easy. Simply run the server thread, passing the paired bluetooth device and handler. The handler will be then called with the following messages:

DATA_PROGRESS_UPDATE – Data is being received by the other device. The message contains the progress of the data being received.
DATA_RECEIVED – Data has been fully received from the other device. The message will contain the actual payload (a byte stream of the image, video, etc.)

There are other message types to handle failure conditions.

The client thread works in a similar fashion. To send data, invoke the client thread, passing the paired bluetooth device and handler. The handler will be called with the following messages:

READY_FOR_DATA – Indicates that the connection has been established, and data can be sent.
SENDING_DATA - Indicates that data is being sent to the other device.
DATA_SENT_OK – Indicates that the recipient received the payload.

Again, there are other message types to handle failures.

To show all of this in action, I’ve put together a sample application, which can be found here.  This app uses the android-btxfr library to send captured photos between devices. 

NewImageNewImage

  • Facebook
  • Twitter
  • Delicious
  • LinkedIn
  • StumbleUpon
  • Add to favorites
  • Email
  • RSS

Enabling x86 Android Emulation

If you’ve undertaken any Android development, you’ll have likely found that the Android emulator can be painfully slow – to the point where the majority of developers I know use a physical device to do any development at all.  The primary reason behind this is that the default emulator is emulating an ARM based chipset on x86 hardware, and the translation between these two architectures is of course costly.  What many of these developers may not realize (and I didn’t until very recently) is that Intel have released a x86 Android emulator.  The x86 emulator has a few caveats (and of course cannot run any ARM-only libraries, such as the Google Play APIs), but overall can be used to speed up the performance of testing apps in the emulator by leaps and bounds.
This is how to get it working:
1.  In the Android SDK Manager, ensure you have API Level 15 or higher installed.  Under the API of your choice, install the Intel x86 Atom System Image:
NewImage
2.  Under the Extras folder, install the Intel x86 Emulator Accelerator (HAXM) module:
NewImage
3.  After you add the HAXM support through the SDK manager, you still need to install the HAXM package.  To do this, navigate to the Extras/Intel/Hardware_Accelerated_Execution_Manager folder under your installed Android SDK path.  Run the .dmg in this folder to install the support.  As part of the installation, you’ll be asked how much RAM to allocate/partition.  On my 16Gb MBP, I have chosen a 2Gb allocation.
4.  This is really important (for Mac OS X Mountain Lion users).  Go to this knowledge library page on Intel’s HAXM site and install the 1.0.4 hotfix.  If you do not install this hotfix, you will likely get a kernel panic when you try to start the AVD.
5.  Create a new AVD, or edit an existing AVD.  Ensure that the CPU/ABI is set to Intel Atom (x86).
NewImage
You can also increase the RAM up to the limit that you set during the HAXM installation.  Also, you should ensure that the “Use Host GPU” check box is enabled as this will allow the emulator to use OpenGL ES on the host for increased performance.
 
With that you should be set!  If you are editing an existing AVD, I recommend checking the “Wipe user data” checkbox before starting the image.  (Otherwise you might find that the emulator will hang during start up).  
 
NewImage
If everything is working OK, you should receive a message that HAXM has been enabled, and your AVD should boot in several seconds (as opposed to several minutes) and be lightning quick to use!
  • Facebook
  • Twitter
  • Delicious
  • LinkedIn
  • StumbleUpon
  • Add to favorites
  • Email
  • RSS