Archive for the ‘Projects’ Category.

How to stop the autoplaying videos on Fairfax websites

Summary: Install this Greasemonkey script for Firefox and the videos will stop autoplaying.

Some Fairfax news websites (including The Sydney Morning Herald, The Age and The Brisbane Times) inflict videos on their users which begin playing after 5 seconds. For instance, this story: Pakistan turn the screws on luckless Australia.

You’ll notice a couple of obnoxious things. First, you need to interrupt your browsing to hit “stop” on the video player. Second, you have to watch a thirty-second ad before the real content starts. And third, the video doesn’t start automatically if you remove “autostart=1″ from the end of that URL.

After getting fed up with this behaviour, I wrote a script to do something about it: Remove Fairfax autoplay video links. You’ll need Firefox with the Greasemonkey extension installed. Afterwards, the script will do two things without your interaction. On any of smh.com.au, theage.com.au and brisbanetimes.com.au, it will:

  • Check whether the current page URL contains “autostart=1″ and, if so, reload the page without that element in the URL.
  • Go through all of the hyperlinks on the current page and remove “autostart=1″, so any subsequent pages visited will be free of autoplaying videos.

This script is free and cobbled together from various tutorials around the web. If you like it, please share the link with your friends. In a perfect world, Fairfax would get the picture and put an end to autoplaying videos. However, I think this tactic probably makes money for them, and so they’ll continue to sell out their user experience for more money.

UNSW Library Lawn in Photosynth

Today I went out onto the Library Lawn at work and took 75 images, which I then dropped into Microsoft’s rather excellent Photosynth tool, building the panorama you’ll see embedded below (or here) if you’ve got Silverlight plugin installed (unfortunately this excludes OSX):

I really was quite surprised at how quick and easy it was to build this panorama. True, walking around taking lots of photos is a bit of a drag, but the whole process took less than half an hour.

HOWTO: Track user activity on your PHPBB forum

Introduction

I run a music forum called Small Night In using PHPBB forum software. Like any website owner, I’m interested in the user activity on my site. That’s why I run Google Analytics, which covers visitor numbers, referring sites, and a wealth of other useful data. Additionally, PHPBB reports the number of users, topics and posts in total and on average.

However, what PHPBB doesn’t report is potentially more useful: long-term trends in growth or decline of active users and new posts. Tracking visits in Google Analytics isn’t sufficient, because some visitors are casual and don’t return – active, registered users are the most valuable to a forum. Tracking the average daily number of posts is no good either, because it gives an average and not an ongoing series of activity snapshots.

To address both these issues, I’ve written a useful script which will allow someone with a minimal amount of PHP/MySQL knowledge to track the long-term number of active users and daily posts on their website.

The script, in brief:

  • Get the time and date 24 hours ago
  • Count how many users last visited later than that time
  • Count how many posts were created later than that time
  • Output the results

The script output is (according to the user’s needs) either a simple PHP echo() of the number to a webpage, or written to a .CSV file for easy long-term tracking and charting.

The script

The first part of the code is just database settings, which you should personalise as needed:

// Change these to your MySQL details
$host = "localhost"; // you probably won't need to change this setting
$user = "your_php_username";
$pass = "your_php_password";
$database = "your_phpbb_database_name";

The next section grabs the current timestamp and subtracts one day’s worth of seconds, as well as yesterday’s date in YYYY-MM-DD format:


$time24hoursago = time() - 86400; // Get the timestamp for a day ago by subtracting 24*60*60 seconds
$date = date('Y-m-d', $time24hoursago);

In the next section, we get the number of entries (users) in `phpbb_users` where the timestamp in `user_lastvisit` is more recent than the $time24hoursago timestamp:


// Database query, recent users
$conn = mysql_connect( $host, $user, $pass ) or die ("Could not connect to MySQL");
$rs = @mysql_select_db($database, $conn) or die("Could not connect to database");
$sql="SELECT COUNT(user_id) FROM `phpbb_users` WHERE `user_lastvisit` > '$time24hoursago'";
$rs = mysql_query( $sql,$conn );
$activeusers = mysql_fetch_array($rs);

Then we do the same thing for recent posts:


// Database query, recent posts
$sql="SELECT COUNT(`post_id`) FROM `phpbb_posts` WHERE `post_time` > '$time24hoursago'";
$rs = mysql_query( $sql,$conn );
$posts = mysql_fetch_array($rs);

Now the arrays $activeusers and $posts contain the number of users and posts. If you just want to see the results, a simple echo command will do the trick:


echo $date . ',' . $activeusers["0"] . ',' . $posts["0"];

Of course, you might be like me, and want to automate the entire process using cron. This next bit of script will write the date and number to a .CSV file called “forum_activity.csv” in the format YYYY_MM_DD,users,posts. For the following process, I’ve modified the file append tutorial on Tizag.com:


$myFile = "public_html/scripts/forum_activity.csv"; // this is the file we'll put the results in
$fh = fopen($myFile, 'a') or die("can't open file");
$stringData = $date . ',' . $activeusers["0"] . ',' . $posts["0"] . "\n";
fwrite($fh, $stringData);
fclose($fh);

Mind the file path $myFile. If you’re running this script through a web browser, you’ll want to make the reference local – that is, have the file path as just “forum_activity.csv” and place the file in the same folder as the script. But if you run this script through cron – for instance daily at midnight, as I do – you’ll want to put the full path to the .csv file or cron won’t be able to find it. Always test before leaving this running for a week and finding it doesn’t work!

Small Night In – a replacement for the Big Day Out forum

In brief: Much of the Big Day Out forum community has relocated to Small Night In after the BDO forum shutdown.

The popular Big Day Out forum was shut down early last week after the death from ecstacy overdose of Gemma Thoms at the Perth Big Day Out.

I’ve got several friends on the BDO forum and they lost a part of their community. Since I’ve got some 50GB of bandwidth, which my other sites barely use a fraction of, I set up a replacement forum called Small Night In. The site has been received enthusiastically and will remain online for as long as people find it useful.

Whether or not the BDO forum ever comes back is unknown, with complete silence on the matter issuing forth from the organisers. I’m enjoying the technical challenges of running a busy forum (and they are many) and learning a bit about community management at the same time.

The forum is an out-of-the-box installation of PHPBB3 with a couple of modifications installed. I prefer to keep things fairly simple and rely on free, community-supported software, so PHPBB is a perfect solution (as well as being the same system used on the BDO forum, giving users some continuity of experience).

HOWTO: Lojack your iPhone and track its location

Update: After further testing, it seems there are a couple of major problems with this setup. The first is that the iPhone doesn’t appear to update if it goes to sleep and isn’t plugged into mains power. The second is that the GPS coordinates I’m getting are wildly inaccurate for my location (varying by a few hundred kilometres) so I figure there’s something wrong with the “findme” application called below. I’ll keep looking for a better way, but for now take this article with a grain of salt.

In this article I’ll explain how to program your jailbroken iPhone to automatically upload its location to a server of your choosing at regular intervals. This will work whenever your iPhone is switched on and has a network connect (3G or Wifi).

As I’m going on holidays tomorrow, I’m going to be in situations where theft of personal belongings is a lot more likely. So after coming up with the idea this afternoon, I spent the last few hours hacking together a tracking solution for my phone. It idles silently in the background, popping up every fifteen minutes to write a latitude/longitude pair to a timestamped database on my server.

This tutorial is going to be a whirlwind because it’s 11pm and there’s so much to explain. And I have a holiday to go to.

You will need:

  • One jailbroken iPhone: I used an iPhone 3G, not sure if this is possible with an original iPhone. To jailbreak your phone, use QuickPWN. Disclaimer, warranty, own risk etc etc.
  • MobileTerminal, OpenSSH, Netatalk and cURL, installed as packages from within Cydia
  • findme, a script written by Erica Sadun and available in a binaries package on her site
  • access to a server with PHP5 & MySQL (possible with other configuration but outside the scope of this article)
  • a moderate level of coding chops – I rate this project “moderately difficult” because of the diverse areas of skill required.

The basic steps involved are:

  1. Preparing your iPhone to be messed with
  2. Teaching your iPhone to query its location using GPS
  3. Constructing a server-side script to record location data
  4. Teaching your iPhone to run the location script at regular intervals
  5. Outputting the data in a readable format

Let’s get started.

The Setup

After jailbreaking your iPhone you will install MobileTerminal, OpenSSH, Netatalk and cURL through Cydia. Then you’ll need to connect to your phone to drop files onto its disk. The easiest way is to connect over Appletalk, but if you’re handy with a command line (or don’t have a Mac) you can ssh root@10.1.1.1 with default password ‘alpine’ and your iPhone’s actual IP address (get this in settings -> network) from your PC.

Drop the findme script into /bin. Now make it executable using chmod 775 findme. When you invoke findme from the command line, it will return some XML containing your phone’s latitude and longitude, like this:

<?xml version=”1.0″?><SearchResults><Success>true</Success><Latitude>-33.887242</Latitude><Longitude>151.256718</Longitude><Method>Skyhook WiFi Location</Method></SearchResults>

(Eagle-eyed readers will noticed I’ve given myself a fancier address in this example.)

The Storage

So now we’ve got a process on your iPhone that delivers GPS over XML. The next step is to transfer that information to a database. For this part, I’m relying heavily on Erica Sadun’s iPhone Lojack article on TUAW, which goes over the same stuff as this article but using Twitter as the output channel.

What we want to do is post the output of findme to a PHP script which then posts the data. You’ll need to create a shell script and upload it to the same folder as findme. Here’s a template:

#! /bin/sh
curl –basic –url http://www.yourserver.com/lojack/index.php5 \
–data status=”`findme`” \

Save this file as ‘loc’ (no extension) and copy it to the same directory as ‘findme’. Don’t forget to make it executable: chmod 755 loc

Now, on your server at the path specified above, create a PHP file that grabs the posted status message and writes it to a database. This script assumes you have a database with a table called ‘locs’ containing two DECIMAL(9,6) fields and a TIMESTAMP field with the current time as its default entry:

< ?php

$username=”username”;
$password=”password”;
$server=”localhost”;
$database=”mylocations”;

$status = $_POST['status'];
$status = stripslashes($status);

$xml = simplexml_load_string($status);

//echo “Status: ” . $status;
$lat = $xml -> Latitude;
$lon = $xml -> Longitude;
echo $lat . ” ” . $lon;

$conn = mysql_connect( $server, $username, $password ) or die(“Err:conn”); //connect
$rs = mysql_select_db( $database, $conn ) or die(“Err:db”); //select db
$sql = “insert into `locs` (lat, lon) values ($lat, $lon)”; //the query
$rs = mysql_query( $sql,$conn );

? >

Now you should be able to invoke loc from the command line (shell, whatever) and it will send the XML to your server script, which will parse out the GPS coordinates and save them to a database. If it’s not working, try removing the comment slashes from //echo “Status: ” . $status; to see what results cURL is getting.

Timing is Everything

Great, so that script sorts out a single location upload. To be useful as a lojack, we want this upload to occur at a regular interval. For that, we’ll use the iPhone’s own LaunchDaemon service. Modify the instructions found in step 4 of the TUAW article to suit your setup. Here’s the script I’m using, com.apple.lojack.plist:

<?xml version=”1.0″ encoding=”UTF-8″?>
<!DOCTYPE plist PUBLIC “-//Apple//DTD PLIST 1.0//EN” “http://www.apple.com/DTDs/PropertyList-1.0.dtd”>
<plist version=”1.0″>
<dict>
<key>Label</key>
<string>com.lojack.upload</string>
<key>ProgramArguments</key>
<array>
<string>/bin/loc</string>
</array>
<key>StartInterval</key>
<integer>900</integer>
</dict>
</plist>

The 900 in the script above specifies this script will run every 900 seconds (15 minutes). You’ll need to restart your iPhone before the LaunchDaemon will work.

Output your results

It’s no good storing all this away on a server somewhere. You need to make it user-friendly. This script will query the database and print the output according to your choice: vanilla, or in tables with links to Google Maps:

< ?php

$username=”username”;
$password=”password”;
$server=”localhost”;
$database=”mylocations”;
$layout = $_POST['layout'];
$conn = mysql_connect( $server, $username, $password ) or die(“Err:conn”); //connect
$rs = mysql_select_db( $database, $conn ) or die(“Err:db”); //select db
$query1 = mysql_query(“SELECT * FROM locs”);

echo ‘<form action=”where.php5″ method=”post”>’;
echo ‘<input type=”radio” name=”layout” value=”plain”>Plain<br>’;
echo ‘<input type=”radio” name=”layout” value=”table”>table<br>’;
echo ‘<input type=”submit”></form>’;

if ($layout == “plain”) {
while ($table1 = mysql_fetch_array($query1))
{
echo $table1[0] . “,” . $table1[1] . “,” . $table1[2] . “,” . $table1[3] . “<br>”;
}
}

if ($layout == “table”) {
echo “<table border=1><tr><td>Timestamp</td><td>Latitude</td><td>Longitude</td><td>id</td><td>Link</td></tr>”;
while ($table1 = mysql_fetch_array($query1))
{
echo “<tr><td>$table1[0]</td><td>$table1[1]</td><td>$table1[2]</td><td>$table1[3]</td><td><a href=\”http://maps.google.com/maps?q=$table1[1],$table1[2]\”>Map</a></td></tr>”;
}
echo “</table>”;
}

? >

While I’d love to show you the results, I’m not quite ready to share the intimate details of my physical location with all of you. Hopefully this article can act as a guide for those with a little knowledge in all of these areas – let me know in the comments if there’s anything that’s unclear. In the future I’d like to build a way to render the data collected as a heat map, or a series of paths – but for now, I’ve got a beach to go and lie on.

Mashup: Dirty restaurants on Google Maps

Recently the NSW Food Authority started publishing data on fines levied against food establishments. (See Register of penalty notices.) Seeing an opening for a useful mashup, I figured out how to scrape their website data, geolocate the addresses, and plot them onto a map:

Center of map
markers

I’ll be maintaining the page properly at http://electronsoup.net/?page_id=148. Tools used were:

Weather Watch Radar on your iPhone

Weather Watch Radar shows rainfall in your locality

Weather Watch Radar shows rainfall in your locality

Last Sunday the sky grew very grey at 4pm, but checking the Bureau of Meteorology for storm info was a hard task on my iPhone. So I created a better way to do it: Weather Watch Radar for iPhone.

This mobile-formatted website gives you the choice of 128km radar images from fifty locations around Australia. It even comes with a nice little icon when you add it to your iPhone’s “Home” screen.

If you have an iPhone, compare my site with the BoM’s Australian Weather Watch Radar Network page, and with the page recommended by the Bureau for viewing on a mobile phone. I hope you’ll find mine much easier to use, but if you don’t, please give me some feedback as to why.

I would like to incorporate animation (to show which direction weather is moving), but that’s a little harder to do; I have a method for grabbing recent images, but without an internal piece of Bureau code it only works for about 75% of locations. Moreover I don’t quite know how to render an animation even if I do have the images.

This whole process – including learning the basics of how to format a site for iPhones – took me just a couple of hours. Imagine what the Bureau of Meteorology could do if it turned its development resources towards building a mobile weather portal for Australians, and opening up access to its data via public APIs.

Update: BOMRadar, a native iPhone application which adds some more functionality than what’s in my page, can be downloaded here. I am darkly amused that this application should be released the day AFTER I invest some programming time into my own solution, instead of before I’d gone to the trouble, because it’s obviously been in development for some time.

‘Tim Bennett project’ update

Three days, and a couple of changes, after my initial post on increasing my Google rank, I’ve gone up from #9 to #4 in Google for “Tim Bennett”. So how did I improve my Google rank five places in three days?

What I’ve done so far

  • Moved my website tagline (“Tim Bennett’s bits and pieces”) into the <h1> section of the header
  • Added meta tags to my Wordpress template:

<meta name="description" content="Tim Bennett's personal site, including blog, resume, links and portfolio.">
<meta name="keywords" content="tim, bennett, tim bennett, electron, soup, electron soup, blog, biography, information, resume, portfolio">

  • Claimed my site on Google Webmaster Tools and filled in some extra information about it (preferred domain, geographic target, robots.txt)
  • Installed the Google XML Sitemaps plugin and submitted a sitemap to Google (making sure my pages are all catalogued)

That’s pretty much all I’ve done so far. I think the first two are the most important (thanks for the suggestion, Kunaal), because they put my key search phrases in places where Google’s likely to look for them, and assign them importance.

So what’s next?

There are some other challenges ahead. Many of the inbound links to my page have the anchor text “Electron Soup” or “Flashman” (the latter, as most of you will know, being my online pseudonym these past six years). Not surprisingly I rank first for “Electron Soup”, though I expected to rank a little higher than 24th for “Flashman”. Even my Twitter account and previous blog rank higher than this site for “Flashman”! (No, you can’t read my old blog. It’s too embarrassing to contemplate.)

The competition

Unfortunately the assault on the top three positions is going to be difficult. I am up against:

  1. A programmer’s website with over 2.2 million page views
  2. A YouTube video (Google properties mysteriously rise to the top of search rankings)
  3. A UK government website (the .gov extension carries a lot of influence with search engines)

I’m not saying it’s impossible, but these three may take a bit of beating.

The Tim Bennett Project

Have you ever met somebody with the same name as you? Through the internet, I’ve become well aware that I’m not the only Tim Bennett. There are a lot of other people out there who share my name. There’s a cyclist, a software developer, six guys in Hollywood, a farming union boss, an ‘equine consultant‘, and a musician, and that’s just on the first two pages of Google. I wonder if we have anything in common besides our name.

I’m thinking of starting the ‘Tim Bennett Project’, to find out more about these people. I’d also like to be higher in the Google search rankings – some days I crack the top ten, but I’d like to be reliably within the top five. I want to be THE Tim Bennett!

Learning page scraping and mashups

As a side project, I’ve been extending my programming skills with a novel mashup project.

“Mashing up” is the art of taking two elements and combining them to produce something new. A musical mashup can create something as interesting as Bootystition (Destiny’s Child vs Stevie Wonder). But mashups of online data exist, too, and can be very useful or beautiful.

Some examples of data mashups:

  • Visual Headlines grabs Flickr images related to keywords in CNN headlines
  • BBC News Map, another headlines service, plots news headlines on a map
  • Housing Maps, another map service, plots Craigslist listings on a map

In some cases these mashups are created using API technology, which supports data interoperability. But sometimes, where websites aren’t set up to export data in easy-to-manipulate formats, other methods must be used.

Page scraping is one such method. Many programming languages support methods of downloading a web page’s contents and extracting particular data from it. The ways in which this can be done are too numerous to go into here, but the upshot is that page scraping allows a programmer to acquire structured data for the purposes of integrating it with another system.

My expedition into page scraping and mashups was inspired by my discovery of the recently-released NSW Food Authority Register of Penalty Notices, a list of penalty notices served to food establishments for breaches of food safety. I want to take this data – which includes business names and addresses – and plot it on a Google map, to allow consumers to see which of their neighbourhood restaurants have been issued a penalty notice.

Using the instructions in the article “PHP: Write a Web Page Scraper” I have been able to grab a link to each penalty notice’s details page. I can then grab the contents that page using cURL, and then use a regular expression search (thanks for the suggestion, Kunaal) to pick out the offence details. (I love government websites; they’re so structured, orderly, and predictable.) At the moment, after a few hours’ programming, that’s all I have achieved. Here is an example of the script output. The next steps involved will be:

  • Saving the information to a database
  • Setting up periodic crawling for new information
  • Building an interface to attach the penalty notice data to labels on a Google map

What other potential uses does this data have? I am not sure, yet. A running tally of the amount of fines issued is one idea. Perhaps a tag cloud of common keywords in the offence descriptions. Chronic Infoholic suggested a possible domain name, which I’ll keep between us for now, but I’m not sure whether I want to develop this into more than a curiosity at the moment. We’ll see.