One point to note, there is a Symfony2 bundle that could be easily integrated with Silex which aims to integrate php-resque but it’s a lot more complicated than I require and in most cases is just not necessary.

The key to using php-resque is to use the registerPrefix method of the symfony classloader (included with Silex) which uses PEAR naming conventions load libraries.

To configure php-resque first clone a copy of the library from github into the vendors folder (or wherever you like to store your external libraries):

git clone https://github.com/chrisboulton/php-resque.git vendors/php-resque

Now we have the library in place where ever you configure your application, in my case /src/bootstrap.php which is included in the main application file point the Synfony class loader to the place for classes beginning with the prefix Resque, all the relevant php-resque files are in the lib subdirectory:

$app['autoloader']->registerPrefix('Resque', __DIR__ . '/../vendor/php-resque/lib');
$app['autoloader']->register();

It does need to be before the $app['autoloader']->register();

The library can then be used in your application at will without requiring or manually including it:

Resque::setBackend('localhost:6379');

$args = array();

Resque::enqueue('default', 'My_Job', $args);

Once you get your head around the different options the synfony2 classloader (and the rest of the components) are wonderful.

HTTP basic authentication is very simple and just passes a username and password in the headers, PHP has built in functionality to extract these values which can be used in the Silex before hook to ensure it happens before every request is fulfilled, my example is for an API which returns JSON but it would work equally well for a conventional website:

$app->before(function() use ($app)
{
    if (!isset($_SERVER['PHP_AUTH_USER']))
    {
        header('WWW-Authenticate: Basic realm="<website name>"');
        return $app->json(array('Message' => 'Not Authorised'), 401);
    }
    else
    {
        //once the user has provided some details, check them
        $users = array(
            'workflow' => 'password'
            );

        if($users[$_SERVER['PHP_AUTH_USER']] !== $_SERVER['PHP_AUTH_PW'])
        {
            //If the password for this user is not correct then resond as such
            return $app->json(array('Message' => 'Forbidden'), 403);
        }

        //If everything is fine then the application will carry on as normal
    }
});

Full details of implementing HTTP auth in PHP can be found in the PHP manual, this includes how to implement HTTP digest auth.

One of my recent projects required me to build a quick JSON only API to abstract interaction with multiple databases for multiple web applications, as I’d already got some of the logic in Codeigniter I just added Phil Sturgeon’s Codeigniter REST Library. However while this handles all method not found errors when URL routing gets as far as the controller all other errors still appear as HTML, when using curl and attempting to parse as JSON this isn’t helpful.

The Solution

My solution was to extend the core Exceptions class which normally deals with these errors. This is done by creating MY_Exceptions.php in application/core and using something similar to the code below:

<?php defined('BASEPATH') OR exit('No direct script access allowed');

/**
 * Extending the default errors to always give JSON errors
 *
 * @author Oliver Smith
 */

class MY_Exceptions extends CI_Exceptions
{
	function __construct()
	{
		parent::__construct();
	}

	/**
	 * 404 Page Not Found Handler
	 *
	 * @param	string	the page
	 * @param 	bool	log error yes/no
	 * @return	string
	 */
	function show_404($page = '', $log_error = TRUE)
	{
		// By default we log this, but allow a dev to skip it
		if ($log_error)
		{
			log_message('error', '404 Page Not Found --> '.$page);
		}

		header('Cache-Control: no-cache, must-revalidate');
		header('Content-type: application/json');
		header("HTTP/1.1 404 Not Found");

		echo json_encode(
			array(
				'status' => FALSE,
				'error' => 'Unknown method',
			)
		);

		exit;
	}

	/**
	 * General Error Page
	 *
	 * This function takes an error message as input
	 * (either as a string or an array) and displays
	 * it using the specified template.
	 *
	 * @access	private
	 * @param	string	the heading
	 * @param	string	the message
	 * @param	string	the template name
	 * @param 	int		the status code
	 * @return	string
	 */
	function show_error($heading, $message, $template = 'error_general', $status_code = 500)
	{
		header('Cache-Control: no-cache, must-revalidate');
		header('Content-type: application/json');
		header("HTTP/1.1 500 Internal Server Error");

		echo json_encode(
			array(
				'status' => FALSE,
				'error' => 'Internal Server Error',
			)
		);

		exit;
	}

	/**
	 * Native PHP error handler
	 *
	 * @access	private
	 * @param	string	the error severity
	 * @param	string	the error string
	 * @param	string	the error filepath
	 * @param	string	the error line number
	 * @return	string
	 */
	function show_php_error($severity, $message, $filepath, $line)
	{
		header('Cache-Control: no-cache, must-revalidate');
		header('Content-type: application/json');
		header("HTTP/1.1 500 Internal Server Error");

		echo json_encode(
			array(
				'status' => FALSE,
				'error' => 'Internal Server Error',
			)
		);

		exit;
	}
}

?>

Limitations

This code will only output a basic JSON formatted message, it will not vary the http error codes based on conditions, nor will it take into account accepts headers or similar. Right now this class meets my needs but features such as other formats could easily be added.

There was a live demo of this but I couldn’t justify the cost of a whole ec2 instance just for this!

The way I have implemented this may be problematic in Chrome/Safari due to Webkit Bug 23933,  ”XMLHttpRequest doesn’t work while submitting a form (useful for progress tracking)” I tested this in firefox (the latest version at the time) and it worked fine. Chrome may or may not work. Let me know how you get on.

Full details of the upload progress implementation can be found the official RFC: https://wiki.php.net/rfc/session_upload_progress

Developers who work with PHP applications that upload files commonly struggle with providing user feedback on the upload progress, usually using flash and javascript solutions like uploadify. In PHP 5.4 there is now integrated functionality to allow file upload progress to be passed back to the browser.

In this post I’ll describe the basic operation of this feature and describe a quick example of its use.

How it Works

The upload progress functionality stores the current progress in a session variable which can then be queried as required to give the current progress, it requires the use of PHP native sessions. The $_SESSION key is set by the form name and a prefix defined in php.ini

An example of the data stored is shown below:

<?php $_SESSION["upload_progress_123"] = array(  "start_time" = 1234567890,  "content_length" = 57343257,
 "bytes_processed" => 453489,
 "done" => false,
 "files" => array(
  0 => array(
   "field_name" => "file1",
   // The following 3 elements equals those in $_FILES
   "name" => "foo.avi",
   "tmp_name" => "/tmp/phpxxxxxx",
   "error" => 0,
   "done" => true,
   "start_time" => 1234567890,
   "bytes_processed" => 57343250,
  ),
  // An other file, not finished uploading, in the same request
  1 => array(
   "field_name" => "file2",
   "name" => "bar.avi",
   "tmp_name" => NULL,
   "error" => 0,
   "done" => false,
   "start_time" => 1234567899,
   "bytes_processed" => 54554,
  ),
 )
);

It is up to the developer how they wish to present this data to the user.

Setup

This functionality is included as standard in PHP 5.4 which can be simply installed from an ubuntu package as discussed in my previous post: http://chemicaloliver.net/internet/installing-php-5-4-in-ubuntu/

This feature should be enabled by default.

Example

Complete code can be found on github.

Upload Form

The upload form is standard html file upload form apart from an additional hidden value defining the progress name attribute so the upload can be identified in the session variables later:

<form id="upload" action="/progress/upload.php" method="POST" enctype="multipart/form-data"><input type="hidden" name="" value="upload" />

 <input id="file1" type="file" name="file1" />
 <input id="file2" type="file" name="file2" />

 <input class="btn primary" type="submit" value="Upload" /></form>

In my example I’ve used the jquery form plugin to make submitting the form using AJAX more simple.

Recieving Script

In this example the receiving script needs to do nothing except start a session, I added a var_dump for debugging initially:

<?php session_start(); var_dump($_SESSION); var_dump($_FILES);?>

Monitoring Progress

This is performed through a combination of javascript and PHP, the client polls a server page which echos the current progress as JSON:

Client:

The client uses an interval time to get the progress every 200ms and pass it to an HTML5 progress bar.

interval_id = setInterval(function() {
$.getJSON('/progress/progress.php', function(data){
    //if there is some progress then update
    if(data)
    {
        $('#progress').val(data.bytes_processed / data.content_length);
        $('#progress-txt').html('Uploading '+ Math.round((data.bytes_processed / data.content_length)*100) + '%');
    }

    //When there is no data the upload is complete
    else
    {
        $('#progress').val('1');
        $('#progress-txt').html('Complete');
        stopProgress();
    }
})
}, 200);

Server Side:

The server side just echos a json encoded version of the session variable:

<?php session_start(); echo json_encode( $_SESSION['upload_progress_upload']); ?>

I recommend using a VM so you don’t mess up any stable PHP install.

Update

Since writing this post php 5.4 stable has been released and a more up to date package is available in this ppa: https://launchpad.net/~ondrej/+archive/php5 I haven’t tested this personally but should work as described below.

Installation

The ppa can be simply added to ubuntu:

sudo add-apt-repository ppa:ondrej/php5

Then installing as normal

sudo apt-get update
sudo apt-get install php5 libapache2-mod-php5

There is however one important change to note which will stop old tests working, that is the way that test classes define a label, these are optional so this only applies if you use them in your project. The name of the test is now passed through the simpletest constructor otherwise the default is to use the classname of the test.

Before

class test_users_model extends CodeIgniterUnitTestCase
{
	public function __construct()
	{
		parent::__construct();

		$this->UnitTestCase('Users Model');

		$this->load->model('users/users_model');
	}

etc...

After

class test_users_model extends CodeIgniterUnitTestCase
{
	public function __construct()
	{
		parent::__construct('Users Model');

		$this->load->model('users/users_model');
	}
etc...

I’m also working on integration with CI systems, Jenkins in particular based on this rather cool presentation: Setting up a Continuous Integration Server for Ubuntu with Codeigniter and Github

Talks (Selected)

PHP Tester’s Toolbox – Sebastian Bergmann

A quick run through different testing frameworks for use with PHP, as Sebastian is the creator of PHPUnit this talk had an obvious bias but I didn’t feel this overwhelmed the talk. For me who has used phpunit and simpletest for unit testing there was nothing ground breaking here but gave me an introduction to BDD and reassured me I was doing the existing bits right.

Deploying PHP Applications with Phing – Michiel Rook

Phing is one of those things I keep meaning to get around to, I’ve used various methods for deployment from mercurial to FTP and ant (when using html5 boilerplate) but Phing seems like a logical thing to use for PHP deployment, as with the previous talk this just provided reassurance that I’d understood it correctly and didn’t really provide anything new. It also only half filled the slot which felt like a bit of a waste.

Estimation or ‘How to dig your own grave’ – Rowan Merewood

Without a doubt this was the most entertaining talk of the day, plagued by early technical issues Rowan breezed through it all with a casual air of confidence and a good measure of humour. The content of the talk was also pretty spot on discussing common sense ways to improve estimating how long your projects will take, much of it rang true for me and will be bearing it in mind in the future, if only my boss got to watch it he might understand that actually my estimations are not so long after all.

Are your tests really helping – Laura Beth Denker

Another entertaining talk centring on best practice when testing with anecdotes from LB’s experience at Google and Etsy liberally mixed in. For me this was a talk which described a way of working to aspire to; Etsy have over 100 developers and can deploy new code up to every 20 minutes due to the efficiency of the test and deployment process. The approach LB presented was a nice common sense approach of testing what needs to be tested rather than every little thing.

Introducing Pyri the new aggregator for Pear – Jeremy Coates

The only talk I’m going to mention here that I watched from the unconference track given by the organiser of the event (or at least one of them). It described an aggregator and search engine for the PEAR PHP package system which is notoriously fragmented. The system, which is only in its early stages of development at the moment  (humorously described as Conference Driven Development), crawls submitted PEAR repos and grabs the XML data on the packages and indexes it. The data is then accessed via an API. The eventual aim is to have a one stop reference site for all PEAR packages available. Development is set to continue on the project over the coming months.

Organisation

In general the organisation of the conference seemed great pretty much everything ran smoothly and there were no real frustrations.

I’m still not sure in my mind if three tracks + an unconference track are too much, there were odd things in the unconference track I would liked to have seen but clashed with one or more things on the schedule. However I did appreciate the unconference track as it filled in a few gaps when other talks finished early. It’s probably better this way than having times when there was nothing I wished to see.

Venue

The physical location for the conference was great, the Ramada Hotel in Manchester is about 10 minutes walk from the station, however the hotel itself wasn’t amazing; the chairs were the most uncomfortable I’ve ever sat on at a conference and some of the hotel looked a bit tired.When it came to lunchtime for some (me included) the queue was unbearable and lack of space to be seated was a disappointment so I retreated to the local Pret a Manger which also allowed me to get some welcome fresh air.

Overall

All in all I had an enjoyable day and will probably return next year, however I didn’t leave inspired to go home an instantly start a new exciting project to make use of something I’d learnt, more a host of useful little things to integrate into existing projects and ways to improve what I’m already doing, so still good but not mind blowing.

This code uses the PCNTL extension which was installed as standard on my Ubuntu 11.04 PHP 5.3 install.

Conventionally to generate a couple of different sizes of images from a large JPEG image one might use something like this:

<?php

/**
 * Create a thumbnail image from the specified file
 *
 * @param int $height output image height
 * @param int $width  output image width
 */

function create_preview($height, $width)
{
    //Input image
    $img = 'test.jpg';

    //Output image
    $new_img = 'test -'.$width.'x'.$height.'.jpg';

    //get the current image size
    list($full_width, $full_height) = getimagesize($img) ;

    //Create a blank image to load in the new resized image
    $tn = imagecreatetruecolor($width, $height) ;

    //Load the original file
    $image = imagecreatefromjpeg($img) ;

    //Do the crop
    imagecopyresampled($tn, $image, 0, 0, 0, 0, $width, $height, $full_width, $full_height) ;

    //Save the file
    imagejpeg($tn, $new_img, 100);
}        

// The thumbnail sizes to be generated
$image_sizes = array(
    array(100,100),
    array(200, 200),
    array(300, 300),
    array(400, 400),
    array(500, 500),
    array(1000, 1000)
);

// Loop through the images sequentially
foreach($image_sizes as $image_size)
{
    create_preview($image_size[0], $image_size[1]);
}

?>

To get an idea of the speed of this script we can run it from the command line:

time php single_process.php

This takes couple of seconds, as you might expect, in my case:

real	0m2.325s
user	0m2.210s
sys	0m0.100s

However rather than performing the resizing sequentially it can be performed in parallel using the same create_preview() function as before:

// The thumbnail sizes to be generated
$image_sizes = array(
    array(100,100),
    array(200, 200),
    array(300, 300),
    array(400, 400),
    array(500, 500),
    array(1000, 1000)
);

//Counter for number of processes
$i = 1;

//Loop through the image sizes but this time fork a process for each size.
foreach($image_sizes as $image_size)
{
    //Fork a process
    $pid = pcntl_fork();

    //if we're in a child thread then grab an image size and process it.
    if (!$pid)
    {
        echo 'starting child ', $i, PHP_EOL;
        create_preview($image_size[0], $image_size[1]);

        //Die otherwise the process will continue to loop and each process will create all the thumbnails
        die();
    }

    $i++;
}

//Wait for all the subprocesses to complete to avoid zombie processes
foreach($image_sizes as $image_size)
{
    pcntl_wait($status);
}

Now running the script using the same command line again the advantage is instantly noticeable:

time php multi_process.php

starting child 1
starting child 2
starting child 3
starting child 4
starting child 5
starting child 6

real	0m0.693s
user	0m3.030s
sys	0m0.140s

As you can see from this very unscientific test running the processes in parallel increases the speed by over 3.5x. As long as you’re careful to watch for threads terminating and not just leaving them unattended this seems like a great way to speed up simple repetitive tasks, I’d certainly hope to find a place for it in some of my applications in the future I’ll probably not use this in a production web application due to the warning that Paul kindly posted below, I guess should read the manual before getting excited…

UK Caller ID Protocol

The caller id system in the UK transmits the callers telephone number (and date and time) as a burst of 1200 baud serial between the first and second ring, this is then decoded and displayed.

Hardware

The hardware is very simple as the CD50 has an internal serial port so all that’s required is to safely isolate the phone line from any sensitive computer systems with an optoisolator.

CD50

The CD50 has been hacked to provide serial output in a couple of projects before, however all the pages documenting this have long gone (the unit is around 10yrs old). Internally the unit uses two EXAR XR-2211 FSK Demodulator/ Tone Decoders, one to detect the presence of the signal and one to decode it (allegedly). In this configuration this means serial data is outputted on pin 7 of the left chip (see below)

If you wish to power the unit from a source other than the internal batteries this can be easily achieved by connecting to the points below the LCD as shown, this must be done with caution as the unit will not work if connected to mains earth as used by many PSUs it must be fully isolated.

Bridge

The optoisolating bridge protects connected equipment from high voltages which can occur in telephone lines, here I used the Vishay 6N137 device as shown (I chose the Vishay as it requires very low current). I powered the CD50 side using the power supply on pin 1 and the arduino side using the internal power supply from the arduino. The 6N137 was sourced from Farnell.

Arduino

The arduino simply receives the serial data using its hardware serial port (I had mixed results with a software serial port), parses the telephone number from the output and then sends it in an MQTT message it would be simple to pass it using over usb back to the computer if you are not using MQTT too.

Software

The serial output from the CD50 has some apparently (but probably not) random letters appended to the beginning of each transmission and the data and time directly in front of the telephone number so once messages have been received this must be parsed at some stage, I choose to do it on the arduino.

All code is also on my github.

Arduino

I wrote this simple sketch to parse and send the data, it requires the Arduino MQTT Library - http://knolleary.net/arduino-client-for-mqtt/.

/*
 * Quick arduino sketch to recieve a telephone number over
 * serial (1200 baud using hardware serial) and broadcast it
 * using MQTT.
 *
 * This sketch is part of a project to read caller id data from
 * a BT Caller Display 50.
 *
 * @author Oliver Smith
 *
 */

#include <SPI.h>
#include <Ethernet.h>
#include <PubSubClient.h>

byte mac[]    = { 0xDE, 0xED, 0xBA, 0xFE, 0xFE, 0xED };
byte server[] = { 192, 168, 1, 2 };
byte ip[]     = { 192, 168, 1, 8 };

PubSubClient client(server, 1883, callback);

const int bufferLength = 19;      //Length of buffer
char stringBuffer[bufferLength];  //Buffer to hold input
int stringIndex = 0;              //Index of next free element in buffer

void setup()
{
  Serial.begin(1200);

  Serial.println("Serial Connected");

  Ethernet.begin(mac, ip);
  Serial.println("Network Connected");

  if(client.connect("arduino"))
  {
    Serial.println("MQTT Connected");
  }

}

void loop()
{
  while(Serial.available() && stringIndex < bufferLength)
  {
    char inputChar = Serial.read();

    //Check if the char read is a number
    if(inputChar  >= '0' && inputChar <= '9' )
    {

        //Add number to buffer
        stringBuffer[stringIndex] = inputChar;

      //Serial.println(stringBuffer);
      stringIndex++;
    }
  }

  //If the buffer is long enough
  if(stringIndex == bufferLength)
  {
    Serial.println("Buffer full");
    //remove initial irrelevant digits

    String output = String(stringBuffer);
    String telephoneNumberString = output.substring(8);

    char charBuf[12];
    telephoneNumberString.toCharArray(charBuf, 12);

    //Send the telephone number back over serial for debugging
    Serial.println(charBuf);
    client.publish("topic",charBuf);

    stringIndex = 0;
  }
}

void callback(char* topic, byte* payload,int length) {
  // no action required
}

Example Python Client

I used a simple MQTT subscribe script with pynotify in ubuntu to display messages on the desktop, details are available in my previous post: First steps using Python and MQTT

I’d rather not show it working as that would give away my telephone number! But it does work, so far 100% of the time.

Getting There

Initially there was lots of head scratching, probing and logic analysing to check the information I’d read:

History

The circuit used seems to be a duplicate of a previous project which was documented at:

http://www.amarok.demon.co.uk/dl/cd50_mod/

and

http://www.photon.dyndns.org/projects/btcd50conversion/old_index.html

However these pages are long gone and the wayback machine only has the text not the images detailing the hardware. I never managed to find a copy of them so I built my own version.

A simple example of it in use is shown below:

//if you're using sparks
$this->load->spark('codeigniter-sitemaps/0.0.1')

$this->load->library('sitemaps');

//assuming a hypothetical posts_model
$posts = $this->posts_model->get_posts();

foreach ($posts AS $post)
{
$item = array(
"loc" => site_url("blog/" . $post->slug),
"lastmod" => date("c", strtotime($post->last_modified)),
"changefreq" => "hourly",
"priority" => "0.8"
);

$this->sitemaps->add_item($item);
}

// file name may change due to compression
$file_name = $this->sitemaps->build("sitemap_blog.xml");

$reponses = $this->sitemaps->ping(site_url($file_name));

You can download the spark from: http://getsparks.org/packages/codeigniter-sitemaps

Or get the code from my github: http://github.com/chemicaloliver/codeigniter-sitemaps

Simple documentation is included and should be self explanatory.

This library is based on work by Philipp Dörner 2010 http://signalkraft.com/google-sitemaps-for-codeigniter