In this post I’ll describe the basic operation of this feature and describe a quick example of its use.

PHP 5.4 File Upload Progress Demo

How it Works

The upload progress functionality stores the current progress in a session variable which can then be queried as required to give the current progress, it requires the use of PHP native sessions. The $_SESSION key is set by the form name and a prefix defined in php.ini

An example of the data stored is shown below:

<?php $_SESSION["upload_progress_123"] = array(  "start_time" = 1234567890,
 "content_length" => 57343257,
 "bytes_processed" => 453489,
 "done" => false,
 "files" => array(
  0 => array(
   "field_name" => "file1",
   // The following 3 elements equals those in $_FILES
   "name" => "foo.avi",
   "tmp_name" => "/tmp/phpxxxxxx",
   "error" => 0,
   "done" => true,
   "start_time" => 1234567890,
   "bytes_processed" => 57343250,
  ),
  // An other file, not finished uploading, in the same request
  1 => array(
   "field_name" => "file2",
   "name" => "bar.avi",
   "tmp_name" => NULL,
   "error" => 0,
   "done" => false,
   "start_time" => 1234567899,
   "bytes_processed" => 54554,
  ),
 )
);

It is up to the developer how they wish to present this data to the user.

Setup

This functionality is included as standard in PHP 5.4 which can be simply installed from an ubuntu package as discussed in my previous post: http://chemicaloliver.net/internet/installing-php-5-4-in-ubuntu/

This feature should be enabled by default.

Example

Complete code can be found on github.

Upload Form

The upload form is standard html file upload form apart from an additional hidden value defining the progress name attribute so the upload can be identified in the session variables later:

<form id="upload" action="/progress/upload.php" method="POST" enctype="multipart/form-data">
 <input type="hidden" name="" value="upload" />

 <input id="file1" type="file" name="file1" />
 <input id="file2" type="file" name="file2" />

 <input class="btn primary" type="submit" value="Upload" /></form>

In my example I’ve used the jquery form plugin to make submitting the form using AJAX more simple.

Recieving Script

In this example the receiving script needs to do nothing except start a session, I added a var_dump for debugging initially:

<?php session_start(); var_dump($_SESSION); var_dump($_FILES); ??

Monitoring Progress

This is performed through a combination of javascript and PHP, the client polls a server page which echos the current progress as JSON:

Client:

The client uses an interval time to get the progress every 200ms and pass it to an HTML5 progress bar.

interval_id = setInterval(function() {
$.getJSON('/progress/progress.php', function(data){
    //if there is some progress then update
    if(data)
    {
        $('#progress').val(data.bytes_processed / data.content_length);
        $('#progress-txt').html('Uploading '+ Math.round((data.bytes_processed / data.content_length)*100) + '%');
    }

    //When there is no data the upload is complete
    else
    {
        $('#progress').val('1');
        $('#progress-txt').html('Complete');
        stopProgress();
    }
})
}, 200);

Server Side:

The server side just echos a json encoded version of the session variable:

<?php
session_start();
echo json_encode( $_SESSION['upload_progress_upload']);
?>

I recommend using a VM so you don’t mess up any stable PHP install – I only tested this on Ubuntu 11.04 as that’s what was recommended and I already had a VM with it installed.

Installation

Installation is a very simple matter of adding the gpg key:

curl http://apt.damz.org/key.gpg | sudo apt-key add -

Adding the repository to /etc/apt/sources.list:

deb http://apt.damz.org/ubuntu natty php54

Then installing as normal

sudo apt-get install php5 libapache2-mod-php5

There is however one important change to note which will stop old tests working, that is the way that test classes define a label, these are optional so this only applies if you use them in your project. The name of the test is now passed through the simpletest constructor otherwise the default is to use the classname of the test.

Before

class test_users_model extends CodeIgniterUnitTestCase
{
	public function __construct()
	{
		parent::__construct();

		$this->UnitTestCase('Users Model');

		$this->load->model('users/users_model');
	}

etc...

After

class test_users_model extends CodeIgniterUnitTestCase
{
	public function __construct()
	{
		parent::__construct('Users Model');

		$this->load->model('users/users_model');
	}
etc...

I’m also working on integration with CI systems, Jenkins in particular based on this rather cool presentation: Setting up a Continuous Integration Server for Ubuntu with Codeigniter and Github

Talks (Selected)

PHP Tester’s Toolbox – Sebastian Bergmann

A quick run through different testing frameworks for use with PHP, as Sebastian is the creator of PHPUnit this talk had an obvious bias but I didn’t feel this overwhelmed the talk. For me who has used phpunit and simpletest for unit testing there was nothing ground breaking here but gave me an introduction to BDD and reassured me I was doing the existing bits right.

Deploying PHP Applications with Phing – Michiel Rook

Phing is one of those things I keep meaning to get around to, I’ve used various methods for deployment from mercurial to FTP and ant (when using html5 boilerplate) but Phing seems like a logical thing to use for PHP deployment, as with the previous talk this just provided reassurance that I’d understood it correctly and didn’t really provide anything new. It also only half filled the slot which felt like a bit of a waste.

Estimation or ‘How to dig your own grave’ – Rowan Merewood

Without a doubt this was the most entertaining talk of the day, plagued by early technical issues Rowan breezed through it all with a casual air of confidence and a good measure of humour. The content of the talk was also pretty spot on discussing common sense ways to improve estimating how long your projects will take, much of it rang true for me and will be bearing it in mind in the future, if only my boss got to watch it he might understand that actually my estimations are not so long after all.

Are your tests really helping – Laura Beth Denker

Another entertaining talk centring on best practice when testing with anecdotes from LB’s experience at Google and Etsy liberally mixed in. For me this was a talk which described a way of working to aspire to; Etsy have over 100 developers and can deploy new code up to every 20 minutes due to the efficiency of the test and deployment process. The approach LB presented was a nice common sense approach of testing what needs to be tested rather than every little thing.

Introducing Pyri the new aggregator for Pear – Jeremy Coates

The only talk I’m going to mention here that I watched from the unconference track given by the organiser of the event (or at least one of them). It described an aggregator and search engine for the PEAR PHP package system which is notoriously fragmented. The system, which is only in its early stages of development at the moment  (humorously described as Conference Driven Development), crawls submitted PEAR repos and grabs the XML data on the packages and indexes it. The data is then accessed via an API. The eventual aim is to have a one stop reference site for all PEAR packages available. Development is set to continue on the project over the coming months.

Organisation

In general the organisation of the conference seemed great pretty much everything ran smoothly and there were no real frustrations.

I’m still not sure in my mind if three tracks + an unconference track are too much, there were odd things in the unconference track I would liked to have seen but clashed with one or more things on the schedule. However I did appreciate the unconference track as it filled in a few gaps when other talks finished early. It’s probably better this way than having times when there was nothing I wished to see.

Venue

The physical location for the conference was great, the Ramada Hotel in Manchester is about 10 minutes walk from the station, however the hotel itself wasn’t amazing; the chairs were the most uncomfortable I’ve ever sat on at a conference and some of the hotel looked a bit tired.When it came to lunchtime for some (me included) the queue was unbearable and lack of space to be seated was a disappointment so I retreated to the local Pret a Manger which also allowed me to get some welcome fresh air.

Overall

All in all I had an enjoyable day and will probably return next year, however I didn’t leave inspired to go home an instantly start a new exciting project to make use of something I’d learnt, more a host of useful little things to integrate into existing projects and ways to improve what I’m already doing, so still good but not mind blowing.

This code uses the PCNTL extension which was installed as standard on my Ubuntu 11.04 PHP 5.3 install.

Conventionally to generate a couple of different sizes of images from a large JPEG image one might use something like this:

<?php

/**
 * Create a thumbnail image from the specified file
 *
 * @param int $height output image height
 * @param int $width  output image width
 */

function create_preview($height, $width)
{
    //Input image
    $img = 'test.jpg';

    //Output image
    $new_img = 'test -'.$width.'x'.$height.'.jpg';

    //get the current image size
    list($full_width, $full_height) = getimagesize($img) ;

    //Create a blank image to load in the new resized image
    $tn = imagecreatetruecolor($width, $height) ;

    //Load the original file
    $image = imagecreatefromjpeg($img) ;

    //Do the crop
    imagecopyresampled($tn, $image, 0, 0, 0, 0, $width, $height, $full_width, $full_height) ;

    //Save the file
    imagejpeg($tn, $new_img, 100);
}        

// The thumbnail sizes to be generated
$image_sizes = array(
    array(100,100),
    array(200, 200),
    array(300, 300),
    array(400, 400),
    array(500, 500),
    array(1000, 1000)
);

// Loop through the images sequentially
foreach($image_sizes as $image_size)
{
    create_preview($image_size[0], $image_size[1]);
}

?>

To get an idea of the speed of this script we can run it from the command line:

time php single_process.php

This takes couple of seconds, as you might expect, in my case:

real	0m2.325s
user	0m2.210s
sys	0m0.100s

However rather than performing the resizing sequentially it can be performed in parallel using the same create_preview() function as before:

// The thumbnail sizes to be generated
$image_sizes = array(
    array(100,100),
    array(200, 200),
    array(300, 300),
    array(400, 400),
    array(500, 500),
    array(1000, 1000)
);

//Counter for number of processes
$i = 1;

//Loop through the image sizes but this time fork a process for each size.
foreach($image_sizes as $image_size)
{
    //Fork a process
    $pid = pcntl_fork();

    //if we're in a child thread then grab an image size and process it.
    if (!$pid)
    {
        echo 'starting child ', $i, PHP_EOL;
        create_preview($image_size[0], $image_size[1]);

        //Die otherwise the process will continue to loop and each process will create all the thumbnails
        die();
    }

    $i++;
}

//Wait for all the subprocesses to complete to avoid zombie processes
foreach($image_sizes as $image_size)
{
    pcntl_wait($status);
}

Now running the script using the same command line again the advantage is instantly noticeable:

time php multi_process.php

starting child 1
starting child 2
starting child 3
starting child 4
starting child 5
starting child 6

real	0m0.693s
user	0m3.030s
sys	0m0.140s

As you can see from this very unscientific test running the processes in parallel increases the speed by over 3.5x. As long as you’re careful to watch for threads terminating and not just leaving them unattended this seems like a great way to speed up simple repetitive tasks, I’d certainly hope to find a place for it in some of my applications in the future I’ll probably not use this in a production web application due to the warning that Paul kindly posted below, I guess should read the manual before getting excited…

UK Caller ID Protocol

The caller id system in the UK transmits the callers telephone number (and date and time) as a burst of 1200 baud serial between the first and second ring, this is then decoded and displayed.

Hardware

The hardware is very simple as the CD50 has an internal serial port so all that’s required is to safely isolate the phone line from any sensitive computer systems with an optoisolator.

CD50

The CD50 has been hacked to provide serial output in a couple of projects before, however all the pages documenting this have long gone (the unit is around 10yrs old). Internally the unit uses two EXAR XR-2211 FSK Demodulator/ Tone Decoders, one to detect the presence of the signal and one to decode it (allegedly). In this configuration this means serial data is outputted on pin 7 of the left chip (see below)

If you wish to power the unit from a source other than the internal batteries this can be easily achieved by connecting to the points below the LCD as shown, this must be done with caution as the unit will not work if connected to mains earth as used by many PSUs it must be fully isolated.

Bridge

The optoisolating bridge protects connected equipment from high voltages which can occur in telephone lines, here I used the Vishay 6N137 device as shown (I chose the Vishay as it requires very low current). I powered the CD50 side using the power supply on pin 1 and the arduino side using the internal power supply from the arduino. The 6N137 was sourced from Farnell.

Arduino

The arduino simply receives the serial data using its hardware serial port (I had mixed results with a software serial port), parses the telephone number from the output and then sends it in an MQTT message it would be simple to pass it using over usb back to the computer if you are not using MQTT too.

Software

The serial output from the CD50 has some apparently (but probably not) random letters appended to the beginning of each transmission and the data and time directly in front of the telephone number so once messages have been received this must be parsed at some stage, I choose to do it on the arduino.

All code is also on my github.

Arduino

I wrote this simple sketch to parse and send the data, it requires the Arduino MQTT Library - http://knolleary.net/arduino-client-for-mqtt/.

/*
 * Quick arduino sketch to recieve a telephone number over
 * serial (1200 baud using hardware serial) and broadcast it
 * using MQTT.
 *
 * This sketch is part of a project to read caller id data from
 * a BT Caller Display 50.
 *
 * @author Oliver Smith
 *
 */

#include <SPI.h>
#include <Ethernet.h>
#include <PubSubClient.h>

byte mac[]    = { 0xDE, 0xED, 0xBA, 0xFE, 0xFE, 0xED };
byte server[] = { 192, 168, 1, 2 };
byte ip[]     = { 192, 168, 1, 8 };

PubSubClient client(server, 1883, callback);

const int bufferLength = 19;      //Length of buffer
char stringBuffer[bufferLength];  //Buffer to hold input
int stringIndex = 0;              //Index of next free element in buffer

void setup()
{
  Serial.begin(1200);

  Serial.println("Serial Connected");

  Ethernet.begin(mac, ip);
  Serial.println("Network Connected");

  if(client.connect("arduino"))
  {
    Serial.println("MQTT Connected");
  }

}

void loop()
{
  while(Serial.available() && stringIndex < bufferLength)
  {
    char inputChar = Serial.read();

    //Check if the char read is a number
    if(inputChar  >= '0' && inputChar <= '9' )
    {

        //Add number to buffer
        stringBuffer[stringIndex] = inputChar;

      //Serial.println(stringBuffer);
      stringIndex++;
    }
  }

  //If the buffer is long enough
  if(stringIndex == bufferLength)
  {
    Serial.println("Buffer full");
    //remove initial irrelevant digits

    String output = String(stringBuffer);
    String telephoneNumberString = output.substring(8);

    char charBuf[12];
    telephoneNumberString.toCharArray(charBuf, 12);

    //Send the telephone number back over serial for debugging
    Serial.println(charBuf);
    client.publish("topic",charBuf);

    stringIndex = 0;
  }
}

void callback(char* topic, byte* payload,int length) {
  // no action required
}

Example Python Client

I used a simple MQTT subscribe script with pynotify in ubuntu to display messages on the desktop, details are available in my previous post: First steps using Python and MQTT

I’d rather not show it working as that would give away my telephone number! But it does work, so far 100% of the time.

Getting There

Initially there was lots of head scratching, probing and logic analysing to check the information I’d read:

History

The circuit used seems to be a duplicate of a previous project which was documented at:

http://www.amarok.demon.co.uk/dl/cd50_mod/

and

http://www.photon.dyndns.org/projects/btcd50conversion/old_index.html

However these pages are long gone and the wayback machine only has the text not the images detailing the hardware. I never managed to find a copy of them so I built my own version.

A simple example of it in use is shown below:

//if you're using sparks
$this->load->spark('codeigniter-sitemaps/0.0.1')

$this->load->library('sitemaps');

//assuming a hypothetical posts_model
$posts = $this->posts_model->get_posts();

foreach ($posts AS $post)
{
$item = array(
"loc" => site_url("blog/" . $post->slug),
"lastmod" => date("c", strtotime($post->last_modified)),
"changefreq" => "hourly",
"priority" => "0.8"
);

$this->sitemaps->add_item($item);
}

// file name may change due to compression
$file_name = $this->sitemaps->build("sitemap_blog.xml");

$reponses = $this->sitemaps->ping(site_url($file_name));

You can download the spark from: http://getsparks.org/packages/codeigniter-sitemaps

Or get the code from my github: http://github.com/chemicaloliver/codeigniter-sitemaps

Simple documentation is included and should be self explanatory.

This library is based on work by Philipp Dörner 2010 http://signalkraft.com/google-sitemaps-for-codeigniter

This is the video that gave me the inspiration (my code replicates this exactly so I didn’t see the need to recreate the video too), the only difference is I used linux all round rather than windows:

Code

The code can be found on my github.

System Structure

The system consists of three main elements, a client which streams video, a client for viewing the video and a server to route the video via websockets and host the web pages.

Camera Client

The camera is created using getUserMedia to send a video stream from a devices camera to the html5 video tag which is displayed on the page. Capture happens by pulling frames onto a canvas and then saving the canvas as a base 64 encoded image and transmitting it over websockets to the server. Repeated frames are captured using setInterval. I’ve tested this demo using the custom Opera Mobile for Android build with getUserMedia support on my HTC Desire Z (T mobile G2) about 2fps is achievable due to the limitations of processing the large amount of data. On faster devices this should improve. I intend to produce a second version using a desktop client to test this.

Viewer Client

The viewer is very simple, it receives frames over a websocket connection as base64 encoded images and displays them in as standard images in html.

Server

The server uses node.js to receive frames via a websocket and then resends them back out to all clients, it is also used to host the static files for the clients.

In this post I will give a simple example of generating the menu below using the FuelCMS Menu Class in Codeigniter.

Before (Plain HTML):

<nav id="main">
    <ul class="clearfix">
        <li><a href="/">home</a></li>
        <li><a href="/customers">customers</a></li>
        <li><a href="/jobs">jobs / orders</a></li>
        <li><a href="/basket">basket</a></li>
    </ul>
</nav>

After (Generated):

<nav id="main">
<ul>
    <li class="first active">
        <a href="/" title="Home">Home</a>
    </li>
    <li>
        <a href="/customers" title="Customers">Customers</a>
    </li>
    <li>
        <a href="/jobs" title="Jobs">Jobs</a>
    </li>
    <li class="last">
        <a href="/basket" title="Basket">Basket</a>
    </li>
</ul>
</nav>

FuelCMS is a CMS based on Codeigniter, it also contains many libraries which can be used with codeigniter independently of FuelCMS in general Codeigniter applications. One of which is the Menu Class. It does many things other than just basic menus like breadcrumb trails and collapsible menus and page titles so serves my purposes very well.

Example Usage

Add the setup code in the constructor of each controller where $this->data is the data sent to the view. In the view $menu contains the prepared html menu.

function __construct()
{
    //Array of pages 'name of controller' => 'Displayed text'
    $this->menu_pages = array(
                    '' => 'Home',
                    'customers' => 'Customers',
                    'jobs' => 'Jobs',
                    'basket' => 'Basket'
                );

    //get the name of the active page
    $this->CI->load->library('uri');
    $this->active = $this->uri->segment(1);

    //setup the menu and load it ready for display in the view
    $this->data['menu'] = $this->menu->render($this->menu_pages, $this->active);
}

In this post I intend to summarise recent developments surrounding using a webcam within web applications and go on to discuss current implementations available for using a webcam with javascript.

Buzz about html5 <device>

Last year at Full Frontal 2010 Raul Rouget gave a talk entitled ‘Batshit crazy stuff you’ll be able to do in browsers’, he basically stole the show with a series of demos of webgl, audio api and file api developments in firefox, however the parts I was most intrigued by involved the use of the now defunct device API – using a webcam or other similar input devices directly in the browser.

Later that evening I sat in may hotel room, after a good amount of beer, and attempted to compile a patched version firefox from source. After downloading the whole lot over a tethered mobile connection and merging in some patches attached to a bug report I got a binary which crashed on startup. I tried again several times later and still didn’t get the desired result, having seen no other blog posts about it I assume very few other people got it to work!

getUserMedia()

Now, 8 months on, the former HTML5 <device> API has been depreciated in favour of a new API based around getUserMedia() this is part of a larger API for peer to peer, in browser  video communication. It allows html5 <video> elements to have their source defined as a stream from a device:

<!DOCTYPE html>

<h1>Simple web camera display demo</h1>
<video autoplay></video>

<script>
//Grab the elements
var video = document.getElementsByTagName('video')[0],
heading = document.getElementsByTagName('h1')[0];

//test for getUserMedia
if(navigator.getUserMedia) {
  //setup callbacks
  navigator.getUserMedia('video', successCallback, errorCallback);

  //if everything if good then set the source of the video element to the mediastream
  function successCallback( stream ) {
    video.src = stream;
  }

  //If everything isn't ok then say so
  function errorCallback( error ) {
    heading.textContent =
        "An error occurred: [CODE " + error.code + "]";
  }
}
else {
  //show no support for getUserMedia
  heading.textContent =
      "Native web camera streaming is not supported in this browser!";
}
</script>
</html>

Code sample taken from http://my.opera.com/core/blog/2011/03/23/webcam-orientation-preview and commented by me

That code produces this output when viewed in an opera mobile development build:

Implementations

At present implementions of this API are very thin on the ground:

• Support is available in webkit on 32 bit ubuntu if a custom version of libwebkitgtk is used.
• There are experimental builds of opera mobile which support some aspects of the API

Mobile

Opera Mobile

Opera have released a build of opera mobile which has support for use of a mobile phone webcam using the new API on android phones. I have successfully tested this on my HTC Desire Z (see screenshots) the support for the full API for peer to peer communication is still in development but this has support for the basics of video capture. This implementation appears to follow the published specification fairly closely

Desktop

Webkit (Ericsson)

An even more experimental implementation of the API has been written by Ericsson which uses a modified version of the libwebkitgtk library which allows use of the new API with compatible desktop browsers (supported with Epiphany). The installation is simple as they’ve provided a PPA:

#Install the Ericsson Labs public GPG key used to verify the package signatures
wget -O- --quiet https://labs.ericsson.com/files/gpg/public.key | sudo apt-key add -

#Add the Ericsson Labs PPA
sudo add-apt-repository http://files.labs.ericsson.net/ubuntu
sudo apt-get update

# Upgrade to the Ericsson Labs modified libwebkitgtk packages
sudo apt-get -y install libwebkitgtk-1.0-0

# (optional) Install Epiphany from the default Ubuntu repository
sudo apt-get install epiphany-browser

Once this installation is complete epiphany (and other browsers configured to use the modified library will have support for the API. It does however support a great deal more functionality compared with the Opera implementation including peer to peer functionality to enable serverless webcam chat type applications. Full documentation is available here

Sadly this implementation of the API differs from the standard specification and uses a prefixed version using webkitGetUserMedia() along with various other annoying differences. A comparable example  to the simple webcam view above could be written  thus:

<html>
<body>

<h1>Simple web camera display demo</h1>
<video id="selfView" autoplay audio=muted></video>

<script>
//Grab the elements
var video = document.getElementsByTagName('video')[0],
heading = document.getElementsByTagName('h1')[0];

//test for getUserMedia
if(navigator.webkitGetUserMedia) {
  //setup callbacks
  navigator.webkitGetUserMedia('video', successCallback);

  //if everything if good then set the source of the video element to the mediastream
  function successCallback( stream ) {
    video.src = webkitURL.createObjectURL(stream);
  }
}
else {
  //show no support for getUserMedia
  heading.textContent =
      "Native web camera streaming is not supported in this browser!";
}
</script>
</body>
</html>

Annoyances

This is not an API ready for use in any kind of real work application as there is no implementation in any mainstream browser and the specification is still in a state of flux. There are also functions which need to be worked out to handle multiple input sources and choosing the correct one, selecting the input size and many other bits and pieces. It could be many months or years before there is a reliable implementation  across browsers.

The Future

This API could be the basis for many cool web applications and potentially replace applications like skype but there is a long way to go before we see standard implementations. However many parts of the API may be able to be replicated in flash and, for the simple parts at least, this may be a realistic step on the ladder for those wishing to get going with the API. A polyfill for the simple features is on my list of things to try…get hold your breath though!

My Plans

Recently I’ve been playing with the API on Opera Mobile in conjunction with jquery mobile, the results can be seen on github. At present it contains a simple camera app which can post a snapshot of the video stream to the sever as a base64 encoded image via the canvas.