Around Christmas 2010, we released a free iPhone game called Tiny Terrors into the App Store. This was written as a native Objective-C, iOS app.

Since then we have rewritten the app using open web technologies i.e. HTML5, CSS3 and JavaScript and with the aid of Adobe’s PhoneGap, we have wrapped it up for distribution via the App Store.

In this post I’ll explain why we did this, how we built it and what our next steps are likely to be.

Where do I get it?

Tiny Terrors is available for iOS devices via the App Store.

You can also play a cut down version of Tiny Terrors directly in your web browser if you are a Chrome or Firefox user at www.tinyterrorsapp.com/play. At the moment, the desktop version doesn’t allow adding yourself to the game and creating your own “tiny terrors”.

Why do this?

The purpose was simply to prove that we could develop effective, cross platform mobile apps that could be distributed by the various app stores using open web technologies.

Throughout 2011, we investigated technologies that could help reduce the overall cost and time associated with developing cross platform mobile applications. Building bespoke native apps for each of the major platforms can be time consuming and expensive. We needed an alternative solution that would allow our clients to reach the maximum number of platforms and devices as quickly and as cost effective as possible.

There were a number of important considerations that we held in mind during this research:

• It was crucial that the final results matched or exceeded what was expected from a typical native app user experience.
• It would be great if we could harness our existing mobile web skills to help build mobile apps. 2010 and 2011 was a busy period for Code and mobile websites. The skills we refined on these sites covering technical, creative and user experience would serve as invaluable tools for mobile app development.
• We understood that cross platform technologies would not replace the need for native app development or remove the need to have native app expertise within the agency. As an agency, we understand the need to pick the best technology for the job. We know there will be times when true native apps will be more suitable.

How did we do it?

We investigated a number of cross platform technologies in addition to the HTML/CSS/JavaScript stack including Adobe’s Flash, Appcelerator’s Titanium Platform and Ansca Mobile’s Corona.

These are fantastic products and each in their own way take the pain out of developing cross platform apps. However, one major concern was that we would become tied in to yet another proprietary platform. We wanted something open, something we had control off, something forward thinking and something universal that would provide us with greater long term flexibility.

During early 2011, we released the open source library PhotoSwipe. This library made it simple for developers and designers to integrate touch based image galleries into mobile websites. Written in HTML, CSS and JavaScript, PhotoSwipe was purposely designed to mimic as close as possible the experience and functionality of the photo app that is bundled with iOS.

What we discovered when building PhotoSwipe was that with well thought out and tightly written JavaScript and clever use of CSS3, we could mimic very closely the appearance of a native app. Subtle tricks to trigger hardware acceleration in animation helped massively and we achieved impressive speeds even on older iOS devices like the iPhone 3G.

Enter PhoneGap

On the back of PhotoSwipe and other mobile web based prototypes that we developed (which included a touch based 3D product viewer!), we set out to look at the best way of wrapping HTML based apps in such a way that we could get them submitted and approved into the various mobile app stores. It’s true that you can write mobile web apps without the need of an app store presence, the Financial Times mobile web app being a fantastic example of this. The truth of the matter is however that you simply can’t ignore these app stores as an important distribution channel for your apps.

In addition, we also needed a way of allowing our HTML apps to use device hardware and features, for example camera and media access, push notifications etc. Certainly you can access a lot of features from mobile websites and on iOS devices you can query geo-location data, touch events and even the accelerometer. However, what you can access at the current time of writing is relatively limited compared to a native app.

To help with this we selected the open source project PhoneGap. PhoneGap makes it incredibly simple to package up our HTML based mobile apps into a native wrapper that then allows us to distribute our apps successfully into the various app stores. PhoneGap also comes with a number of features that makes it simple for our apps to access various hardware features of a mobile device in a uniform way. For instance, using the camera is made accessible via a JavaScript interface that is identical for iOS and Android devices. Additionally, the open nature of the project meant it fitted in perfectly with our research objectives.

However, the major plus point for PhoneGap is the ease at which it can be extended if a feature does not exist out of the box. Using both native language and JavaScript skills, it’s very simple to build your own plugins to access any feature of the device that your HTML based apps require. We successfully developed a number of plugins including overlaying a web app with native user interface components that blurred the line as to what was and wasn’t web platform based. We’ve developed other plugins to undertake work behind the scenes on behalf of a HTML based app – such as requesting content from another website or RSS feed. We also successfully developed a push notification plugin that allow our HTML based apps to respond intelligently to push notifications sent to a device.

What we discovered by joining HTML/CSS and JavaScript with PhoneGap was that we could write 80-90% (if not more) of an app using open web technologies and then using PhoneGap, fill in the the remaining functionality by using it’s built in features and developing our own plugins.

Other frameworks and libraries we used

In addition to PhoneGap, the Tiny Terrors app was built on a number of existing frameworks and libraries, including our own bespoke JavaScript app platform.

JavaScript libraries used

Tiny Terrors is based on a JavaScript framework built by ourselves. The framework was initially designed as a rapid prototyping tool for mobile apps and grew out of the work undertaken with PhotoSwipe etc. It wasn’t long however before it quickly grew into a production ready framework. This library is heavily events driven, extendable and out of the box makes it simple to handle transitions, page states and effectively “knits” an app’s components together.

Our framework is also built on a solid stack of proven 3rd party open source libraries including Underscore.js, Backbone.js, Lawnchair and Zepto.js. As well as Zepto.js, the framework works equally as well with jQuery for supporting non WebKit / Mozilla based browsers.

We did investigate using existing mobile web frameworks such as jQuery Mobile and Sencha Touch and while we highly rate these, we found both a bit too large for what we needed and in the case of jQuery Mobile, the fact that it is tightly coupled to jQuery provided us with less flexibility than we would have liked.

iOS libraries used

In addition to PhoneGap, Tiny Terrors also uses the Facebook iOS SDK, the excellent ObjectAL library for audio and MBProgressHUD for displaying a native “loader / progress” indicator.

Conclusion

Currently Tiny Terrors works on the desktop (Chrome and Firefox currently) and iOS devices. Our next steps will be to integrate this into Android and Windows Phone 7 Mango devices.

If anything, rewriting Tiny Terrors has demonstrated that it is possible to write an app using open web technologies to maximise device and platform reach. It also shows that you can obtain a near native like experience. But as with any approach, you need to think carefully about whether the technology is right for your solution and be open and flexible to both native and “non-native” approaches. At the end of the day, if the end user receives an app experience that is on par with or exceeds what they expect from a device, then that’s the decisive factor.

Ste Brennan

So the highlights of the day for us were…

The Lessons learned from theexpressiveweb.com – Mike Chambers

Mike gave a very good insight into the way any company (not just Adobe) should evaluate the launch of a site, be it successful or not.
Kicking off the talk with a quote, simply stating:

“Is HTML5 ready for prime time?”

And without ever actually giving a definitive answer to this, he seemed to allude that if you can use it, then why not?
His presentation not only gave a broad overview of some of the issue Adobe had to contend with when producing theexpressiveweb.com, but also their outcomes/recommendations as well as a list of resources they used.

Hand Covers Bruise – Joel Gethin Lewis

“If you can code you can create value out of nothing”

Joel Gethin Lewis one half of Hellicar & Lewis – http://www.hellicarandlewis.com/ gave possibly the most inspiring talk of the day.

The most powerful part of his talk focused on a project that he’s been working on for the last 4 years to help Autistic children – ReacTickles and Somantics. This opensource project was split into two halves, one side focused on the proven therapeutic effects that interactive software can have on autism, the other half focused on enabling and encouraging self expression.

Reacticles was an interactive installation utilising simple shapes and colours with noise detection. Some of the kids that normally were catatonic actually began to interacted and some even managed maintained eye contact, in some cases this proved to be very therapeutic and allow the kids to self-medicate and lower their doses.

Somanickles was in a similar vein to ReacTickles but much more playful. Joel was told that people effected by the condition struggle with self expression and that trying to encourage this behaviour would be nigh on impossible, but he belived that this was not the case and he was later proven right. In a series of videos shown towards the end of the talk, kids are seen actively engaging with the experiments on both an individual and collaborative level.

Meta Process: Making It up As I Go Along – Hilman Curtis

One of the last talks of the day was from Hillman Curtis – a truly inspiring character. Hillman’s talk was full of great insights, stories and advice, anyone unfamiliar with Hillman’s work and bio should make sure to look it up. http://hillmancurtis.com/about/

The main section of the talk focused around the creative process he defined 10 years ago in his book MTIV:Process, Practice and Inspiration for the New Media Designer (http://www.amazon.co.uk/Process-Inspiration-Practice-Media-Designer/dp/0735711658). Hillman reflected on the original 7 principles that were at the core of his process back then and went on to define 7 new principles which have formed from the re-invention of himself and his work.

And as per usual Seb Lee-Delisle blew everyone away with his interactive installation experiment, but we could probably write and entire post just on this session but we won’t bore you right now.

Be sure to check back tomorrow to see our highlights on day 3.

- Chris, Daniel, Jono & Luke

This years FOTB opener saw a break from the traditional Keynote, and was kicked off by various product managers within the Adobe teams. As much as some areas of the web community enjoy slating Adobe and some of their products (rightly so in some cases, but this is isn’t the post to get into that now) it’s always nice to see some of the new bits & bobs Adobe are planning.

The way they are helping the new ‘expressive web‘, along with the likes of Google, Microsoft et al, as well as helping too support and push forward new technologies and standards (such as CSS regions) is great to see, as are some of the new tools (Edge) and improvements to existing products (CSS3/HTML5 support in DW). Having heard about Adobe Edge, but never actually having seen it running (it brought a wry smile to our faces and that ‘oh crap this could be terrible in the wrong hands’ thought instantly flashed through our minds) the demo given on stage was relatively simple enough too suitably impress the audience with the ease at how simple animations could be achieved.

Overall it was a good kick off to the day, but it was maybe bogged down with a bit too much development for some of the sore design heads at this early hour.

Ideas to interface – Aarron Walter

“When you work for the man, you become so focussed on your day-to-day tasks; meeting clients; and urgent deadlines, we lose creative thinking”

The message from this is quite a important, you need to give yourself permission and make time to instigate independant project and get over the fear of faliure.

Aarron made reference to the fledgling idea which soon became Twitter. The original idea (stat.us) was developed around 2000 but Jack Dorsey decided the time wasn’t right so he put it on the shelf until 206 and well you know the rest.

“If you have a 1000 ideas in the shelf you will never fail.”

An important part of MailChimps UX process is ‘Sketchboarding’, these enable the team (and the rest of the company) to ideate quickly, incubate ideas on the walls and also to easily evaluate ideas with colleagues. It is important in this process to keep things pubilc so everyone can give opinons not matter what thier role

Sketchboards: A Technique for better & faster UX solutions – http://www.youtube.com/watch?v=iVFTBj_BYy0

Tron GFX – GMunk

Gmunk – a motion graphics director, talking mainly about his involvement with some of the special effects and interface designs for the feature film TRON: Legacy.

Set up a team of 7 guys ranging from Graphic designers to Code artists. It was great to get an insight into the workflow of the team and the intricacy of the graphics that they were creating.

Josh Nimoy created a whole bunch of Graphic applications which allowed the guys to create complex and organic animations.

The overall design process was quite organic and backed up by a great reference library. He really stressed the importance of having a strong and wealthy bank of references for any creative projects.

The talk gave a great insight into the tremendous amount of detail that was put into the scenes for the film, each decision had been meticulously crafted and was backed up with reasonings for it looking or moving a certain way. The Solar Sailor sequence was a fine example of this: http://work.gmunk.com/#1190816/TRON-Solar-Sailor. Although just over two minutes in length the scene was crammed with smart particles systems and organic flowing forms.

Lettering is not Fonts – Jessica Hishche

Jessica’s talk centred mainly around her background and how she got into the world of lettering and fonts.

She gave an great overview of the lettering and type design industries, discussed why we’re all responsible for keeping specialized industries alive, showed a fair amount of pretty pictures, and cursed like a sailor.

It was a great talk and was delivered in a personal and very frank manner, helped along by a massive talent!

“Graphic designers don’t understand type design because if they did they wouldn’t complain about how expensive fonts are”

Keep your eyes posted for tomorrows highlights.

- Daniel, Luke, Chris & Jono

Code Computerlove’s mobile technology expert Ste Brennan explores cross platform app development and offers advice on approaches to consider for cost effective mobile campaigns that can reach the mobile masses.

A Diverse Mobile Market

For apps, Android and Apple dominate the market, but these are not the only players. There’s also Blackberry and Windows Phone 7, which is set to experience significant market growth with Nokia’s move away from Symbian to the Microsoft platform.

To consumers this offers great choice, but for marketers it makes it time consuming and costly to maximise reach and target all devices. Typically, each operating system requires bespoke app development. To complicate this further, devices running on the same operating have different capabilities or require a different end user experiences.

Mobile Websites vs Native Apps

Current mobile app development eschews the principles of the web to build once and deploy anywhere. With a well written website, you can be confident that it will work on various platforms and web browsers. So in the same vein, maximising the potential reach of your mobile presence is a primary goal for our mobile development work.

Mobile websites offer many advantages. They embrace the cross platform spirit of the web, they don’t tie you to an app store eco-system and allow you to be more responsive to issues and user demands, which can be especially difficult in a tightly managed environment like Apple’s App Store.

However, there are cases where native apps offer opportunities for brands that simply cannot be achieved with mobile websites. Arguably delivering better usability, they also allow access to features on mobile devices such as a camera, notifications, compass and lower level hardware access. Plus they offer convenient monetisation opportunities like “in app” purchasing, and despite certain drawbacks, visibility in the App Store and Android Market etc.

What Approach Should I Take?

Firstly, when weighing up what approach to take, it’s vital to have a holistic mobile and digital strategy and not view apps in isolation. Building an app just to tick a box is obviously the wrong approach. Mobile is set to become the primary digital marketing channel between businesses, brands and consumers, and how an app and website integrate with an overall platform and digital strategy is critical.

Secondly, it’s important not to get too hung up on the technology and general perceptions as to the ‘right or wrong way’ of developing apps. A good digital partner will ensure your solutions continue to push the boundaries of what can be achieved and which is why we put so much emphasis on R&D, to develop new ways of achieving client’s mobile objectives.

For example, you can build mobile apps utilising one of the many cross platform development tools available such as Adobe’s Flash platform, Appcelerator’s Titanium X and PhoneGap. All of these can ease the investment and time required to hit as many devices as possible, leverage features not available to mobile websites and enable distribution via mobile app stores.
 
It’s true that these solutions may not offer all the latest ‘whizz bang’ features of a device at the rate that true native development does. However, to get hung up on this is missing the point.

Remember, technology isn’t the only driver, the customer experience is. If the end user doesn’t see the difference between an app written in Flash for instance versus a “true” native app, then who is to say that this is not a valid mobile app experience?

At Code, we recognise and apply technologies best suited to client’s needs. For example, to a recent brief our suggestion of developing a core application that can be re-purposed for multi-platform use proved a cost effect and wide-reaching solution. In this instance, our advice has been to build an application in Flash that can then be deployed across many devices and channels from App Store mobile apps (smartphones and tablets) to traditional desktop websites with the core reusable code working in tandem with the company’s back-end e-commerce systems.
 
For another client, it was clear their target audience consisted of Apple users and therefore a true “native” iOS app was suggested coupled with a strong mobile web presence.

Flexibility in approach is key

So, apps are blossoming (especially in retail), brands are embracing mobile websites and consumers are using both. Both are an important part of a mobile strategy.

Our advice is not to get pre-occupied with apps in isolation or be single minded in your approach. Take a holistic view on your mobile marketing strategy and consider how it fits into a business and its strategy.
 
To the end user, it doesn’t matter what technology is being used as long as it meets their expectations and needs. Certainly consider native apps, but don’t be afraid to run with cross platform tools to help you reach a wider audience in a timely and cost effect manner.

Digital agencies that can deliver this in the most cost-effective way are surely where the sensible marketer takes their marketing spend?

• Message threading across a number of platforms – for instance start a conversation with contact across SMS, continue on Facebook and see the thread in one place on your phone!
• Deeper social network integration with The People Hub integrated with Twitter and Linked in
• Dynamic tile widgets for real-time information e.g. flight times without having to open a dedicated app
• Location and context aware search and results. This is really exciting and yet again highlights how important context aware searching is in the mobile arena.

For a demo of the update, check out:

http://www.bbc.co.uk/news/technology-13525442

It’s also great to see that the developer Mango beta tools have been launched today (May 25th 2011), giving developers plenty of time to play about with the over 500 new features!

http://windowsteamblog.com/windows_phone/b/wpdev/archive/2011/05/24/developer-news-beta-mango-tools-available-today.aspx

Ste

Last month we were asked by our friends at Cake Solutions if we wanted to contribute to a guideline they were working on for the National Computing Centre. Obviously we jumped at the chance!

The title was “Java vs .NET”, here’s the summary of what we were looking to provide:

“In these Guidelines, we compare the features of the Java platform with the features of the .net platform. they are written with most of the comparison using the prominent language of each platform: Java on the Java platform and C# on the .net platform.

The aim is to give you fair and unbiased assessment of each platform as a whole. Consequently we have considered the languages, development tools, continuous integration and quality assurance tools, the wider ecosystem of the platforms and finally the platforms’ infrastructural impact. We also included guidance on the future of your favourite platform. these Guidelines cover enterprise systems: that is systems with complex domains that store the information in a relational database, systems that must comply with strict transactional behaviour, security and auditing rules. We also explore the world outside the typical enterprise – the smart devices are growing more and more popular and it would be foolish to ignore them.

Before we begin, we must stress that we do not wish to convert you from one platform to the other. But, simply to make the Java developers see that .net is not just a collection of coding horrors in VB.net just as much as we wish to show to the C# developers that Java has moved on from the days of unmaintainable JSPs.”

You can view the Java vs .NET guideline here. Let us know what you think and if you work in the Java or .NET space, please share your views/experiences.

We’d like to thank Cake Solutions for giving us the opportunity to be part of their guideline.

@timpeel

The introduction of mobile handsets many moons ago has changed the way we view web content. The switch was a slow and steady pace at first, with little capacity to produce rich content for mobile devices. The introduction of smart phones proved to be a turning point, but still its been a slow graduation into mobile device specific design and content.

The introduction of the app store and apps, specific to smart phones, not just apple but android and later windows phone 7 changed the way designers, UX architects and developers alike looked at the mobile content. The app store in particular has been the reason for businesses seeking an app for their brand or product. In most cases the option for a app is misguided and irrelevant to their business, but the pressure of ‘having an app’ has proved too strong.

Gradually we have seen businesses offering mobile versions of their web content. In the past 6 months this offering has increased ten fold.

In the past businesses were happy to serve web content in the same way as a desktop website, letting the mobile device shrink the existing layout to fit. With small screens and limited real estate for accessible content, it has become clear that the user wants easy access to the content they desire. Pinch and zoom, although an ingenious way to cope with desktop websites on a hand held device has become a chore, and an inconvenience.

Faster, snappier, and more easily accessible content is what the user wants. Click and scroll is how we are used to using websites, and this is how it should be on a mobile device. Click has been replaced with touch, and scroll with swipe. Pinch and zoom should not become a primary function when viewing rich web content.

So what does this mean for designers and developers?

Well first of all, more work. Great :). Yes it’s great that there will be no shortage of work for the industry. But at what cost?

Designing and developing mobile websites, takes time and as such will ultimately cost. Large companies should have no problem paying for a mobile offering, but smaller companies however might be stretched to find the budget for both. At the pace things are moving it won’t be long before we see the desktop website playing second fiddle to the mobile.

I have been involved in many mobile design and development projects, and from what I have seen so far, after a while the process becomes very linear, and can leave little room for creativity in design. Screen size on most current mobile devices (I am talking mobile phones here) are very small with the average around 320 x 640(iPhone). From this observation questions can be raised. Will the designer become bored of designing for mobile? If mobile becomes the primary medium for accessing web content, what room for creativity is there, without compromising usability?

Here at Code, being creative with mobile design and development is seen as a massive opportunity, and will require every discipline to think outside the realms of their regular website approach. The exploration into new and innovative ways to deliver mobile content is one that is both fast paced and exciting.

I think if you are a designer that also develops the HTML and CSS (front end) then you will have your work cut out with the multitude of handsets and rendering engines. If you are purely design, should you be worried of getting bored purely designing for mobile? Personally I think while mobile handsets are at the size they are the tablet will be your saviour, which poses the question… what purpose will the desktop website have? Will it become the designers playground?

We have not really seen such a dramatic shift in platforms for websites before, so who knows which way things will go, and what will become of the desktop website, but for now everything seems to be pointing to mobile.

Are you a designer / developer? What are your thoughts?

Andrew Brandwood

That was the response of most people when we (myself and Ste Brennan) said we were going on a physical computing course at Manchester’s MadLab in association with the region’s Omniversity. Robots were a small spec on the horizon compared to the level we dabbled, but it was fun and inspirational all the same.

The course was run by the Omniversity of Manchester and held at Manchester’s Digital community space MadLab, and was an introduction to Arduino and physical computing.

Attending the course was a real mix of people from all different backgrounds and walks of life. Among the attendees were art students, programmers and inventors alike. The course ran at a pace that was comfortable for everyone. It didn’t leave anyone behind, nor did it leave people sat waiting for the next step.

Getting started really took me back to school and electronics classes, with a plethora of wires, LED lights and resistors, which I immediately started sticking into my circuit board. This obviously (as it had done in school) returned no results. Nothing lit up, made a sound or even gave me (or my fellow attendees) an electric shock. At this point I thought I’d sit back and listen for a while.

We started off with an interesting and knowledgeable introduction to Arduino and its creators, along with a glimpse of the weird and whacky things people had been creating. After that we were all ready to go.

As I say, robots were a far cry from what we were about to embark on, but that did not detract from the sense of excitement and achievement we attained after managing to light up an LED on a physical circuit (Arduino) board with a few lines of code. The lights progressed on to two lights, and then on to physical controls to dim them, make them flash, and pretty much anything you can do with an LED, to be honest.

Arduino is a great technology to get into and the limits are pretty much endless with what you can do. Not only can you program something on a computer to have a physical reaction, you can flip it around and interact with a program called Processing.

With Processing you can introduce a physical action which in turns tells the computer to do something. As part of the course we also had a quick play with the Processing side, using analogue dials to bend and pitch sound frequencies through our computer speakers.

The Beginner’s Guide to Arduino and Physical Computing was an excellent course, the tutor was really helpful and knowledgable and I would recommend anyone who is even mildly interested in this sort of thing to look out for the next course and get involved. The guys at MadLab have a great space in the Norther Quarter and are more than accommodating.

The next course is on this Saturday (March 12th 2011). Get yourselves down there and be inspired.

http://madlab.org.uk/content/beginners-guide-to-arduino-and-physical-computing-2/

Links:

http://madlab.org.uk/
http://omniversity.madlab.org.uk/

Andrew Brandwood (Branny)

To add my own thoughts on app development and the opportunities mobile applications offer to marketers; yes, apps can offer a great way to build an audience and generate interest in a brand or product if part of an overall digital strategy.

But the market is experiencing an explosion of start-up companies offering standalone ‘out of the box’ apps, luring in marketers keen to get an app out there with low prices and fast turnaround. As a result, I’m seeing too many brands developing low quality apps that don’t really do a very good job, and which are scoring a bit of an own goal, gaining low ratings and negative feedback from users who want and expect more.

Firms are tending to make the same mistakes they did with their early websites and shop for their apps in isolation like they were some shiny thing they could buy off the shelf, and we’re actually getting calls from companies coming to us asking us to re-do apps that they’ve launched because they’re damaging the brand.

For us, it’s not about just producing an app ‘because an app is the holy grail’, but it’s about where the app fits into the customer journey, adds value and how it’s managed longer term. There is a consumer expectation with an app that it is relevant, useful and that the brand will continue to invest in them.

Apps can be a key consumer interface so some careful planning needs to go into their design and ongoing promotion. Remember, you’re not buying an app, you’re building an audience.

It’s also about the idea behind the app and being able to see that the app is going to get used, and that it is not just a quirky app idea that in fact no one will really want. Focus needs to be on “how can I use this technology/platform to really help my consumers and support the all-important value exchange that customers are looking for; how to build an extension of a product, and develop individual and niche apps that support the products when consumers are using them.

Another word of caution to marketers who’ve read about ‘the power of the app’ and are now rushing out to buy one, is not to forget the role of the mobile website in all of this. It’s business critical in today’s mobile world.

- Tony

As everyone familiarises themselves with smart phones, touch screens and various interactive mediums which do not require a traditional user interface such as a mouse or keyboard. Technology providers are quickly establishing a ubiquitous standard for controlling, navigating and requesting actions with our finger tips. These gestures are practically the same on each device, bar a few minor differences in performance perhaps, but nonetheless the same gestures are used to interact and navigate with most applications and games.

Gestural technology in the Gaming arena is no new kid on the block – first implemented by the likes of Sony and Nintendo, with their respective consoles. Microsoft have now waged in with their version of gestural interfacing – The ‘Kinect’. In a nutshell the Kinect provides multiple users the ability to interact with the X-Box 360 without the need for a hand held device to accurately pin point their location in 3D space. Using an array of complex cameras and microphones the Kinect is able to recognize user’s faces, use a skeletal tracking system which can track up to 6 people – as well as 2 active users, each having 20 of their joints tracked at a cool 30/fps. The Kinect also boasts an impressive depth of field, tracking users up to 3.5m away from the sensor. According to Steve ‘The Ape‘ Ballmer – CEO of Microsoft, the Kinect has sold over 3 million more units than its originally modest projections of 5 million, almost double the sale of the Playstation’s counterpart ‘Move’ motion technology.

With the Kinect showing the potential to dominate the ‘Controller-less’ gaming market, hacking the Kinect has become the new black, so much so that Microsoft has unveiled that they will be creating its own binaries and drivers to enable third party developers to explore the capabilities and create new content for the Kinect on non X-Box 360 platforms. According to reghardware.com the SDKs should be released “in the coming months”. Unfortunately the SDKs will only be available on the Windows platform, though we can’t really blame Microsoft for being ‘choosey’ about what they will and won’t allow to run on their systems, unlike some turtleneck wearing people I could mention. At the moment this isn’t very good news for Mac developers – but that hasn’t been stopping us up to now.
There have been many ‘Kinect Hacks’ released recently here are some of my personal favorites:

These applications are effectively interpreting the raw data that the Kinect is seeing and making that data available through particular middle-wear to a wide range of programming languages. As an ActionScript3 developer there are a few choices of SDK and various types of interface; OpenKinect, FLKinect, and TuioKinect. I have played around on all three and found TuioKinect to be the easiest to set up and get going with for ‘Blob Tracking’ (Basic hand tracking). Yet it does have some draw backs, one being the user must be standing a certain distance from the sensor and is fairly limited in terms on gaining access to some of the Kinect’s motor features or microphone accessibility. This was disappointing as our concept would require a much more stable solution. Then I discovered OpenNI which probably the most extensive collection of  libraries – but be prepared for a long slog installing it on Mac OSX, as there are no Binaries available at the moment, but users are assured they are on the way. None-the-less OpenNI would provide me with the stability and speed which I would need to create a proof of concept for commercial viability of the Kinect.

After doing a few experiments with the Kinect’s and OpenNI depth measuring capabilities and seeing if the experiments I had created could effectively be used through that transparent stuff we call glass, particularly double glazing. A few experiments later, I discovered it works like a charm! So I began to focus the experimentation on more commercially viable experiments (rather than continuing to recreate arcade classics like Pong, Space Invaders and Breakout). These experiments on a basic level are aimed at promoting simple gestures to iterate through certain levels of information displayed on a rear projection screen or large scale monitor. The demo should give some idea of the viability of what gestural technology can offer to a commercial or retail environment.

A little bit about the technology used in the demo: The front end is built for Adobe AIR (Adobe Integrated Runtime) in pure AS3, using assets from a Flash’s shared library to make it look sexy and a little less Flex-like. I used the Flex’s socket classes to connect to a simple JavaScript socket server running locally on my machine. As for the middleware I decided to use a combination of OpenNI and slightly modified PrimeSense C++ application to produce the example. The modifications were fairly simple, but a little daunting to start with as C++ isn’t my first language or second, third, etc.. But using some examples and samples provided by Microsoft, I scraped together a bridge application to connect to a Node.js socket server. Now the Kinect was sending data through to the socket, which basically consisted of x, y and z coordinates in rapid succession. This only happens once the user has performed the initiation gesture.

Without a doubt the most challenging aspect of creating a gesture based UI for the general public to effectively engage with, is ‘ease of use’. With our little experiment being a relatively/totally unique user interface, we have to rely on user’s knowledge of current gestural technology on other devices in order to interact successfully with ours, minus an instruction manual. The standard gestures like “Pinch” to zoom, “Expand” –  to zoom out, “Swipe” left or right‚ could be used to navigate your way through content, and should be second nature to the majority of smart phone, iPad or next generation mice users. These are simply replicated by gesturing the hands in front of the display rather than on it.

Businesses which choose to utilise advertising or display information within its own commercial or retail space, will be notifying potential customers of their presence within this environment. These businesses can implement bespoke interactive experiences to suit their customer’s needs for information. Window real estate is a valuable commodity for many businesses, but is restricted to the size of their premises or windows within their premises. An interactive solution provides consumers with the ability to attain large amounts of information night or day, upon request.

A typical example of a suitable business looking to implement such technology would be an estate agent. Only displaying a few properties within each price band to entice a maximum range or potential buyers simply cannot address all of their customers needs. Using an interactive system such as this, potential buyers can ‘gesture’ through the entire back catalogue of properties, price ranges, and retrieve detailed information that a paper based or static digital solution just simply cannot provide. The basic concept behind the solution can be rolled out to pretty much every business with a window.

The Kinect is a powerful and complex bit of kit, but is rather limited in understanding its environment outside of the visual. This is something that is essential for commercial sales and with the addition of third party input devices, could create a intuitive sales devices;  A clothing retailer for example can tailor its digital advertising strategy based on the time of day or current weather trends. To start advertising umbrellas when it is raining, or bobble hats & mittens when it drops below 0 degrees C. This provides the retailer with an ‘intelligent’ salesman who never sleeps. These environmental considerations can all be taken into account when a user is interacting with a digital display. Yet these are not the only considerations that the system could be aware of. The user’s physical state could also change the information they are being shown. The Kinect can read exactly where you are in 3D space, this being said it could ‘virtually’ size you up and let you know roughly whether there is currently any stock in your size or make an informed decision on the colour scheme you are currently wearing and suggest garments with matching colours. As we demonstrated in the demo, users and businesses could take advantage of QR technology, for example a user has found something that they desire and want to order or get more detailed information on the product, but it is outside of opening hours. A QR code could be shown to the user which would provide them with a simple reference point or method of ordering the item.

Like any new technology there are some potential drawbacks to implementing something like this into a business advertising or sales strategy. The Kinect at the moment has the ability to support up to 6 users. Problems could arise supporting multiple users all trying to use the technology at once, this would be problematic and confusing as everyone walking past is going to want have a go. Yet this could be overcome by registering one user at a time, for example if a user stands in a certain spot. The system will be able to detect their presence and only react to their gestures. Windows would also need to be regularly cleaned or use self cleaning glass for the ideal installation. This can also be said for window displays using traditional (not modern) glazing as this will have a slight effect on user’s gesture recognition. There will also be a problem if there is condensation on glass or large amounts of moisture or rain on the glass – as this will block the effective view and range of the cameras. A potential solution, buy an awning! Minimum lighting levels was an issue which arose while testing the prototype. Minimum levels will need to be matched, or artificial lighting may need to be implemented if the effective 3.5m range is to be achievable in the pitch black.

Even without a legitimate SDK provided by Microsoft, the possibilities of hacking the Kinect to utilise its power for advanced user interactivity are immense, accessible and exciting. It is also intriguing, designing and prototyping interfaces for this not too distant technology. It is only a matter of time before we start seeing this type of installation on our high streets and we start living in ‘Minority Report’ style world – minus the creepy metal spider robots and precognitives.

- Jono