Free Trial

Safari Books Online is a digital library providing on-demand subscription access to thousands of learning resources.


  • Create BookmarkCreate Bookmark
  • Create Note or TagCreate Note or Tag
  • PrintPrint
Share this Page URL
Help

4. Look, Ma, No Computer! Microcontrolle... > An Embedded Network Client Applicati...

An Embedded Network Client Application

Now that you’ve made your first server, it’s time to make a client. This project is an embedded web scraper. It takes data from an existing website and uses it to affect a physical output. It’s conceptually similar to devices made by Ambient Devices, Nabaztag, and others—but it’s all yours.

Project 7: Networked Air-Quality Meter

In this project, you’ll make a networked air-quality meter. You’ll need an analog panel meter, like the kind you find in speedometers and audio VU meters. I got mine at a yard sale, but you can often find them in electronics surplus stores or junk shops. The model recommended in the parts list is less picturesque than mine, but it will do for a placeholder until you find one you love.

Figure 4-8 shows how it works: the microcontroller makes a network connection to a PHP script through the Ethernet shield. The PHP script connects to another web page, reads a number from that page, and sends the number back to the microcontroller. The microcontroller uses that number to set the level of the meter. The web page in question is AIRNow, www.airnow.gov, the U.S. Environmental Protection Agency’s site for reporting air quality. It reports hourly air quality status for many U.S. cities, listed by ZIP code. When you’re done, you can set a meter from your home or office to see the current air quality in your city (assuming you live in the U.S.).

MATERIALS
  • 1 Arduino Ethernet or

  • 1 Arduino Ethernet shield and 1 Arduino microcontroller module

  • 1 Ethernet connection to the Internet

  • 1 solderless breadboard

  • 1 voltmeter

  • 4 LEDs

  • 4 220-ohm resistors

Control the Meter Using the Microcontroller

First, you need to generate a changing voltage from the microcontroller to control the meter. Microcontrollers can’t output analog voltages, but they can generate a series of very rapid on-and-off pulses that can be filtered to give an average voltage. The higher the ratio of on-time to off-time in each pulse, the higher the average voltage. This technique is called pulse-width modulation (PWM). In order for a PWM signal to appear as an analog voltage, the circuit receiving the pulses has to react much more slowly than the rate of the pulses. For example, if you pulse-width modulate an LED, it will seem to be dimming because your eye can’t detect the on-off transitions when they come faster than about 30 times per second. Analog voltmeters are very slow to react to changing voltages, so PWM works well as a way to control these meters. By connecting the positive terminal of the meter to an output pin of the microcontroller, and the negative pin to ground, and pulse-width modulating the output pin, you can easily control the position of the meter. Figure 4-9 shows the whole circuit for the project.

The networked air-quality meter.
Figure 4-8. The networked air-quality meter.
The circuit for a networked meter. The Ethernet controller shown in the schematic is on the shield or the Arduino Ethernet board.
The circuit for a networked meter. The Ethernet controller shown in the schematic is on the shield or the Arduino Ethernet board.
Figure 4-9. The circuit for a networked meter. The Ethernet controller shown in the schematic is on the shield or the Arduino Ethernet board.
Test It

The program to the right tests whether you can control the meter.

You will need to adjust the range of pwmValue depending on your meter’s sensitivity. The meters used to design this project had different ranges. The meter in the parts list responds to a 0- to 5-volt range, so the preceding program moves it from its bottom to its top. The antique meter, on the other hand, responds to 0 to 3 volts, so it was necessary to limit the range of pwmValue to 0–165. When it was at 165, the meter reached its maximum. Note your meter’s minimum and maximum values. You’ll use them later to scale the air-quality reading to the meter’s range.

/*
    Voltmeter Tester
    Uses analogWrite() to control a voltmeter.
    Context: Arduino
*/
const int meterPin = 9;

int pwmValue = 0;  // the value used to set the meter

void setup() {
  Serial.begin(9600);
}

void loop() {
  // move the meter from lowest to highest values:
  for (pwmValue = 0; pwmValue < 255; pwmValue ++) {
    analogWrite(meterPin, pwmValue);
    Serial.println(pwmValue);
    delay(10);
  }
  delay(1000);
  // reset the meter to zero and pause:
  analogWrite(meterPin, 0);
  delay(1000);
}

Write a PHP Script to Read the Web Page

Next, you need to get the data from AIRNow’s site in a form the microcontroller can read. The microcontroller can read in short strings serially, and converting those ASCII strings to a binary number is fairly simple. Using a microcontroller to parse through all the text of a web page is possible, but a bit complicated. However, it’s the kind of task for which PHP was made. The program that follows reads the AIRNow page, extracts the current air-quality index (AQI) reading, and makes a simpler summary page that’s easy to read with the microcontroller. The Ethernet controller is the microcontroller’s gateway to the Internet, allowing it to open a TCP connection to your web host, where you will install this PHP script.

Note

You could also run this script on one of the computers on your local network. As long as the microcontroller is connected to the same network, you’ll be able to connect to it and request the PHP page. For information on installing PHP or finding a web-hosting provider that supports PHP, see www.php.net/manual/en/tutorial.php#tutorial.requirements.

Figure 4-10 shows AIRNow’s page for New York City (http://airnow.gov/?action=airnow.local_city&zipcode=10003&submit=Go). AIRNow’s page is formatted well for extracting the data. The AQI number is clearly shown in text, and if you remove all the HTML tags, it appears on a line by itself, always following the line Current Conditions.

Note

One of the most difficult things about maintaining applications like this, which scrape data from an existing website, is the probability that the designers of the website could change the format of their page. If that happens, your application could stop working, and you’ll need to rewrite your code. In fact, it happened between the first and second editions of this book. This is a case where it’s useful to have the PHP script do the scraping of the remote site. It’s more convenient to rewrite the PHP than it is to reprogram the microcontroller once it’s in place.

AIRNow’s page is nicely laid out for scraping. The PHP program used in this project ignores the ozone level.
Figure 4-10. AIRNow’s page is nicely laid out for scraping. The PHP program used in this project ignores the ozone level.
Fetch It

This PHP script opens the AIRNow web page and prints it line by line. The fgetss() command reads a line of text and removes any HTML tags.

When you save this file on your web server and open it in a browser, you should get the text of the AIRNow page without any HTML markup or images. It’s not very readable in the browser window, but if you view the source code (use the View Source option in your web browser), it looks a bit better. Scroll down and you’ll find some lines like this:

Current Conditions
Air Quality Index (AQI)
observed at 17:00 EST
45

These are the only lines you care about.

<?php
/*
    AIRNow Web Page Scraper
    Context: PHP
*/
    // Define variables:
    // url of the page with the air quality index data for New York City:
    $url =
      'http://airnow.gov/index.cfm?action=airnow.showlocal&cityid=164';

    // open the file at the URL for reading:
    $filePath = fopen ($url, "r");

    // as long as you haven't reached the end of the file:
    while (!feof($filePath))
    {
        // read one line at a time, and strip all HTML and
        // PHP tags from the line:
        $line = fgetss($filePath, 4096);
        echo $line;
    }
    // close the file at the URL, you're done:
    fclose($filePath);
?>
Scrape It

To extract the data you need from those lines, you’ll need a couple more variables. Add this code before the fopen() command.

// whether you should check for a value
       // on the line you're reading:
       $checkForValue = false;

       // value of the Air Quality reading:
       $airQuality = −1;

Replace the command echo $line; in the program with the block of code at right.

This block uses the preg_match() command to look for a string of text matching a pattern you give it. In this case, it looks for the pattern Current Conditions. When you see that line, you know the next line is the number you want. When the PHP script finds that line, it sets the variable $checkForValue to true.

// if the current line contains the substring "Current Conditions"
        // then the next line with an integer is the air quality:
        if (preg_match('/Current Conditions/', $line)) {
           $checkForValue = true;
 }

Now, add the following block of code after the one you just added. This code checks to see whether $checkForValue is true, and whether the line contains an integer and nothing else. If so, the program reads the next line of text and converts it from a string to an integer value. It will only get a valid integer when it reaches the line with the AQI value.

if ($checkForValue == true && (int)$line > 0){
                         $airQuality = (int)$line;
                         $checkForValue = false;
        }

Finally, add the following lines at the end of the script, after the while loop. This prints out the air-quality reading and closes the connection to the remote site.

The result in your web browser should look like this:

Air Quality: 43

Now you’ve got a short string of text that your microcontroller can read. Even if the script got no result, the value −1 will tell you that there was no connection.

echo "Air Quality:". $airQuality;
    // close the file at the URL, you're done:
    fclose($filePath);

Read the PHP Script Using the Microcontroller

Next, it’s time to connect to the PHP script through the Net using the Ethernet module. This time, you’ll use the shield as a client, not a server. Before you start programming, plan the sequence of messages. Using the Ethernet module as a network client is very similar to using Processing as a network client. In both cases, you have to know the correct sequence of messages to send and how the responses will be formatted. You also have to write a program to manage the exchange of messages. Whether you’re writing that program in Processing, in Arduino, or in another language on another microcontroller, the steps are still the same:

  1. Open a connection to the web server.

  2. Send an HTTP GET request.

  3. Wait for a response.

  4. Process the response.

  5. Wait an appropriate interval and do it all again.

A flowchart of the Arduino program for making and processing an HTTP GET request.
Figure 4-11. A flowchart of the Arduino program for making and processing an HTTP GET request.

Figure 4-11 is a flowchart of what happens in the microcontroller program. The major decisions (if statements in your code) are marked by diamonds; the methods are marked by rectangles. Laying out the whole program in a flowchart like this will help you keep track of what’s going on at any given point. It also helps you to see what methods depend on a particular condition being true or not.

The circuit for this project also uses LEDs to keep track of the state of the program. LEDs attached to I/O pins will indicate the state. There’s an LED to indicate that it’s connected, another to indicate that it’s disconnected, a third to indicate if it got a valid reading, and a fourth to indicate that the microcontroller is resetting.

This program will check the PHP script every two minutes. If there’s a new value for the air quality, it’ll read it and set the meter. If it can’t get a connection, it will try again two minutes later. Because it’s a client and not a server, there’ s no web interface to the project, only the meter.

TextFinder Library

For this sketch, you’re going to need Michael Margolis’ TextFinder library for Arduino. Download it from www.arduino.cc/playground/Code/TextFinder, unzip it, and save the TextFinder folder to the libraries folder of your Arduino sketches directory (the default location is Documents/Arduino/libraries/ on OS X, My Documents\Arduino\libraries\ on Windows 7, and ~/Documents/Arduino/libraries/ on Ubuntu Linux. If the libraries directory doesn’t exist, create it and put TextFinder inside. Restart Arduino, and the TextFinder library should show up in the Sketch→Import Library menu. TextFinder lets you find a substring of text from the incoming stream of bytes. It’s useful for both Ethernet and serial applications, as you’ll see.

TextFinder was modified and included in version 1.0.1 of Arduino, after this edition was published. You can find its methods in the Stream class, from which Client, Server, and Serial inherit methods. For alternate versions of these sketches that use Stream instead of TextFinder, see the GitHub repository for this book’s code, at https://github.com/tigoe/MakingThingsTalk2. Check the Reference section of www.arduino.cc for details of the Stream class.

Connect It

The program starts out by including the SPI and Ethernet libraries, just as in the previous project. Then define the output pins, the minimum and maximum values for the meter that you determined earlier, and the time between HTTP requests using constant integers.

/*
  AirNow Web Scraper
 Context: Arduino
 */
#include <SPI.h>
#include <Ethernet.h>
#include <TextFinder.h>

const int connectedLED = 2;          // indicates a TCP connection
const int successLED = 3;            // indicates a successful read
const int resetLED = 4;              // indicates reset of Arduino
const int disconnectedLED = 5;       // indicates connection to server
const int meterPin = 9;              // controls VU meter
const int meterMin = 0;              // minimum level for the meter
const int meterMax = 200;            // maximum level for the meter
const int AQIMax = 200;              // maximum level for air quality
const int requestInterval = 10000;   // delay between updates to the server
Finding a Host’s IP Address

This project uses numeric IP addresses rather than names. If you need to find a host’s IP address, use the ping command mentioned in Chapter 3. For example, if you open a command prompt and type ping -c 1 www.oreillynet.com, (use -n instead of -c on Windows), you will get the following response, listing the IP address you need:

PING www.oreillynet.com (208.201.239.37): 56 data bytes
64 bytes from 208.201.239.37: icmp_seq=0 ttl=45 time=97.832 ms

If you can’t use ping (some service providers block it, since it can be used for nefarious purposes), you can also use nslookup. For example, nslookup google.com will return the following:

Server: 8.8.8.8

Address: 8.8.8.8#53

nslookup returns the Domain Name Server it used to do the lookup as well.

Non-authoritative answer:
Name: google.com
Address: 173.194.33.104

Any one of the addresses listed will point to it’s associated name, so you can use any of them.

Next, initialize a few array variables with the network configuration for the module.

byte mac[] = { 0x00, 0xAA, 0xBB, 0xCC, 0xDE, 0x01 };
IPAddress ip(192,168,1,20);
IPAddress server(208,201,239,101 );

Change these to match your own device and server.

The last global variables you need to add are for the EthernetClient class, and a few variables relating to the transaction with the server.

// Initialize the Ethernet client library
EthernetClient client;

boolean requested;           // whether you've made a request
long lastAttemptTime = 0;    // last time you connected to the server
int airQuality = 0;          // AQI value
boolean meterIsSet = false;  // whether the meter is set

setup() starts the serial and Ethernet connections, sets all the LED pins to be outputs, blinks the reset LED, and makes an initial attempt to connect to the server.

void setup() {
  // start the Ethernet connection:
  Ethernet.begin(mac, ip);
  // start the serial library:
  Serial.begin(9600);
  // set all status LED pins:
  pinMode(connectedLED, OUTPUT);
  pinMode(successLED, OUTPUT);
  pinMode(resetLED, OUTPUT);
  pinMode(disconnectedLED, OUTPUT);
  pinMode(meterPin, OUTPUT);

  // give the Ethernet shield a second to initialize:
  delay(1000);
  // blink the reset LED:
  blink(resetLED, 3);
  // attempt to connect:
  connectToServer();
}

The blink() method called in the setup blinks the reset LED so you know the microcontroller’s main loop is about to begin.

void blink(int thisPin, int howManyTimes) {
  //     Blink the reset LED:
  for (int blinks=0; blinks< howManyTimes; blinks++) {
    digitalWrite(thisPin, HIGH);
    delay(200);
    digitalWrite(thisPin, LOW);
    delay(200);
  }
}

The loop() contains all the logic laid out in Figure 4-11. If the client is connected to the server, it makes an HTTP GET request. If it’s made a request, it uses TextFinder to search the response for the air-quality string and then sets the meter. If the client isn’t connected to the server, it waits another two minutes until the request interval’s passed, and tries to connect again.

void loop()
{
  // if you're connected, save any incoming bytes
  // to the input string:
  if (client.connected()) {
    if (!requested) {
      requested = makeRequest();
    }
    else {
      // make an instance of TextFinder to search the response:
      TextFinder response(client);
      // see if the response from the server contains the AQI value:
      if(response.find("Air Quality:"))  {
        // convert the remaining part into an integer:
        airQuality = response.getValue();
        // set the meter:
        meterIsSet = setMeter(airQuality);
      }
    }
  }
  else if (millis() - lastAttemptTime > requestInterval) {
    // if you're not connected, and two minutes have passed since
    // your last connection, then attempt to connect again:
    client.stop();
    connectToServer();
  }

  // set the status LEDs:
  setLeds();
}

Both setup() and loop() attempt to connect to the server using the connectToServer() method. If it gets a connection, it resets the requested variable so the main loop knows it can make a request.

void connectToServer() {
  // clear the state of the meter:
  meterIsSet = false;

  // attempt to connect, and wait a millisecond:
  Serial.println("connecting...");
  if (client.connect(server, 80)) {
    requested = false;
  }
  // note the time of this connect attempt:
  lastAttemptTime = millis();
}

Once the client’s connected, it sends an HTTP GET request. Here’s the makeRequest() method. It returns a true value to set the requesting variable in the main loop, so the client doesn’t request twice while it’s connected.

The server replies to makeRequest() like so:

HTTP/1.1 200 OK
Date: Fri, 14 Nov 2010 21:31:37 GMT
Server: Apache/2.0.52 (Red Hat)
Content-Length: 10
Connection: close
Content-Type: text/html; charset=UTF-8

 Air Quality: 65
boolean makeRequest() {
  // make HTTP GET request and fill in the path to
  // the PHP script on your server:
  client.println("GET /~myaccount/scraper.php HTTP/1.1\n");
  // fill in your server's name:
  client.print("HOST:example.com\n\n");
  // update the state of the program:
  client.println();
  return true;
}

Change these to match your own server name and path.

Once the client’s found the air-quality string and converted the bytes that follow into an integer, it calls setMeter() to set the meter with the result it obtained. You might want to adjust the AQIMax value to reflect the typical maximum for your geographic area.

boolean setMeter(int thisLevel) {
  Serial.println("setting meter...");
  boolean result = false;
  // map the result to a range the meter can use:
  int meterSetting = map(thisLevel, 0, AQIMax, meterMin, meterMax);
  // set the meter:
  analogWrite(meterPin, meterSetting);
  if (meterSetting > 0) {
    result = true;
  }
  return result;
}

The last thing you need to do in the main loop is set the indicator LEDs so that you know where you are in the program. You can use the client.connected() status and the meterIsSet variable to set each of the LEDs.

That’s the whole program. Once you’ve got this program working on the microcontroller, the controller will make the HTTP GET request once every ten minutes and set the meter accordingly.

void setLeds() {
  // connected LED and disconnected LED can just use
  // the client's connected() status:
  digitalWrite(connectedLED, client.connected());
  digitalWrite(disconnectedLED, !client.connected());
  // success LED depends on reading being successful:
  digitalWrite(successLED, meterIsSet);
}