allthingsbecause.org

Arduino: Standalone wifi http messenger message parsing

2010-05-18T05:11:00.001+08:00

I've been asked a few times how I do message parsing in my sketch (as in how to I get the message from the http server into the code). I was going to edit the previous post but figured with it's compexity I may as well post another. I have to start by saying that this sketch does the most basic method of message capture you can. It works for a small number of cases, but the chances that people are going to get it wrong are slim because it only does one thing. It may help in reading this post to follow along with the code here.

The sketch is based around the simpleserver example that handles all the web server grunt work. It works like this: in the loop area of the arduino there is a line "WiServer.server_task();" that would be a function to run all of the server functions, upkeep the wireless board, and respond to page requests on the line. When a page request is found on the line the function


boolean SendPage(char* URL)

is run. At it's heart this function just tells the server what message to return as html as the page is requested. There is no fancy hard drive access to get web pages here though, just a URL passed in and a function "WiServer.print()" to print output. Because the URL is passed in however we can roll over it to do things by accessing the URL variable. This might respond to simple URL page requests:


//something like
if (substr(URL, 1, 5) == "test") //if characters 2 to 6 were "test"
{

WiServer.print("test was entered"); //this is written as page output and displayed by the browser

In it's most simple form the specific characters in the URL can change something in the arduino. Say we are controlling three lights. We can check if the first character of the URL is '1' turn the first light on; '0' for off. The same occurs with the second character for the second light, third character for the third, etc. If you are interested, I have basically done this here but the crux of the code would be


if (URL[1] == '1')

digitalWrite(ledPin1, true);


if (URL[1] == '0')
digitalWrite(ledPin1, false);


if (URL[2] == '1')
digitalWrite(ledPin2, true);

if (URL[2] == '0')
digitalWrite(ledPin2, false);

Now before I move on we'll have to talk a little bit about the URL array. It starts at 0, ends at (I'm guessing) 255. We don't know how long it is in the code but being a C string it's end will be denoted by '%5C0'. We can't really use the 0th element however because this will generally be taken up by a / to finish the server part of the URL eg the / at the end of "http://www.google.com/". When we are doing proper processing there will also be a ?variable= sort of set up.

So on to my text parsing. First we need to check if the URL being requested is to generate the index page (URL will be basically empty), or if the user is trying to add a new message from the form. This is done in the line


"if (StrCmp(URL, "?MESSAGE=", 1, 9) == true)"

Basically if the characters 1-9 in the URL are "?MESSAGE=" then we are expecting the user is trying to add a message. And why this specific string? because I said so on the form in the index page:


<form action='/' method='get'>

<input type='text' name='MESSAGE' size='15' maxlength='40'>
<input type='submit' value='submit'>



</form>

This just tells the browser to set up a visual form. The page that will process the form is '/' (nothing), the method is get so print the form input in the URL. Then the variable name will be 'message' and print a submit button. The process takes this a converts it to the form /?MESSAGE=bleargh. Its a bit more complicated than that though.

The browser can't send everything exactly as it is or the browser would get confused. How would it know the difference between the column in test-server:8080/stuff.htm and if someone had just written a column after it? It fixes this by just changing anything that could be a problem (basically anything that isn't a letter or number) to a hex representation of it's ascii code. I'll slow down a bit here, so if you know about ascii and hex skip to the next paragraph. So ASCII is the American Standard Code for Information Interchange. Computers don't talk in text; they talk in numbers. We make them talk in text by simply asssigned each letter (upper, lower), number, punctuation and a few other things to a specific number. View them all at http://www.asciitable.com/. Hex is the base 16 representation of that number. We count decimally (by 10) but computers count in twos (binary 0s and 1s). Because of this it is more efficient to count in a numbers of twos. Base 16 counts to 16 rather than 10 (0 1 2 3 4 5 6 7 8 9 A B C D E F), so decimal 16 is hex 10. The easiest way to figure it is just to look at the web site above and look at the dec and Hx columns. When the browser finds a character it doesn't like it swaps it for %Hex ie ' ' normally changes to %20, '(' to %28. Space is different; if the browser is working correctly it should be written as a '+'. Of course they don't have to be bad characters. http://www.%67%6F%6f%67%6c%65.com should take you to google.com. Edit: the browser seems to convert automatically if you look at the status bar at the bottom of he screen. Know that the above link is pointing to the exact web address displayed.

But I digress. Having figured that a URL does contain a message the sketch needs to convert the above hex representations and strange characters back to characters that we can display. It does this by looping through each letter and looking for the % sign that would say the character needs to be converted.

do
{ 
    loopLetterFound = false;
    if (SERIALOUT) Serial.print(urlIndex);
    if (urlIndex < (urlLen - 2))
    {
        //begin quick and dirty encoding parser
        if (URL[urlIndex] == '%') //encoding makes non viable characters into %hh where hh is 
        {                        //the characters hex value.
            if (SERIALOUT) Serial.println("Will test hex");
            tmpChar = HexToInt(URL[urlIndex + 1], URL[urlIndex + 2]);
            if (tmpChar != -1)
            {
                messages[0][messageIndex] = tmpChar;
                loopLetterFound = true;
                urlIndex += 2;
            }
            else //problem occured with hex conversion.
            {
                messages[0][messageIndex] = '%';
                loopLetterFound = true;
                urlIndex += 2;
            }
        }
    }
}

So:

If it is a '+' add a ' ' to the message.
If it is a % convert the next two characters to a character and add it to the message
Otherwise just add the character to the message.

HexToInt just plays with the maths behind base 16 so that it can be converted back to base 10. From here we can cast as a char (character) if it isn't one already to turn it into the corresponding character. If the base 16 stuff still hurts or you want to better understand the hex2int function this website explains base 16 reasonably well.

Arduino: Standalone wifi http messenger (expanded version)

2009-12-14T10:23:00.000+08:00

I've Had some interest in this post, so I'm writing up a longer definition detailing some of the ins and out of the arduino, where you can buy specific parts and how the parts fit together. Note if you already know about the Arduino and shields, you should jump straight to the "On the Code" section below.

Some hardware stuff:

The arduino comes in a variety of forms with the arduino duemilanove being a very popular one. This is the one that I use for my projects and includes the chip itself (an Atmiga 328) along with a usb port for transferring programs / power and an actual power plug. The Atmega 328 itself has a bootloader flashed onto it (a small program that handles uploading "sketches", which are programs). It has 2k of RAM on the chip and 32k of ROM in which to install programs (although ~2k of ROM is used by the boot loader). In function, the chip supports 14 digital I/O pins and 6 analogue I/O pins. Of th 14 digital pins, 6 support PWM for a form of analogue output. For an introduction to the arduino I would suggest starting at http://www.ladyada.net/learn/arduino/index.html that goes through some simple programs accessing the pins and setting up the software to create programs. Basically on the software side the arudino uses a java IDE that works with "sketches" in c like syntax. These are compiled and uploaded quite simply by the program. Also, being built in java there is a version available for windows, mac and linux.

In terms of buying the hardware to start with, there are a number of online stores that sell both the arduino and the wishield. A good place to start is adafruit (http://www.adafruit.com/index.php? main_page=product_info&cPath=17&products_id=68) or sparkfun (http://www.sparkfun.com/commerce/product_info.php?products_id=666). The adafruit link there is to a pack that includes a charger, lights, resistors and a heap of other stuff that makes it easy to start off with the arduino. If you do a google search though it is quite possible to find a vendor closer to your area. From these sites you can also buy a range of LCD screens. A lot of character driven displays (ones with eg 20x4 characters) are the same and are based on the HD44780 chip which makes them very easy to set up and use. For reference, I got mine from here: http://toysdownunder.com/arduino/lcd/lcd-5v-blue-20x4.html. Unfortunately I am have a hard time finding the wishield on adafruit or sparkfun, but I got mine from here: http://toysdownunder.com/arduino/arduino-shields/arduino-wifi-shield.html. Also unfortunately this is in Australia (where I'm from) so you might have to email the owner of the site if you go through him to sort out sending overseas. You could also buy from asynclabs direct at their store: http://asynclabs.com/store?page=shop.browse&category_id=6.

For the wishield, this is an attachment (a "shield" as they are called) that plugs into the top of the arduino to extend it's features. Basically, the 20 pins out from the arduino are aligned on either side of the board as female plugs. Shields extend the arduino's features by plugging into these with a male on the bottom of the board and extending through to a female at the top. It may be easier to explain this by looking at pictures of the arduino and wishield. The pins on the bottom of the wishield plug into the top of the arduino. This can cause some problems though as while all the pins are extended through to the top of the shield (so you can plug other things in), some are used by the wishield itself and cannot be used. On this build I had to take care not to use specific pins used by the wishield with the LCD (because both used quite a lot). In hindsite it probably would have been better to use just an ethernet shield rather than a wifi one. It would probably use less pins and the wifi adds another possible problem to development. An ethernet shield would also be much cheaper.

For further reading on all of these topics I would suggest:

1. The arduino tutorial site I talked bout above (http://www.ladyada.net/learn/arduino/index.html)
2. After installing the arduino software it comes with a large range of examples programs to look at, pull apart etc.
3. The arduino site also has a lot of tutorials about all sorts of arduino activities (http://arduino.cc/en/Tutorial/HomePage)
4. For the LCD specifically the arudino site has a good writeup on how to talk to it (http://arduino.cc/en/Tutorial/LiquidCrystal) via the liquidcrystal library. (http://www.arduino.cc/en/Reference/LiquidCrystal)
5. For the wishield specifically the asynclabs wiki will explain how to set it up along with sample programs (http://asynclabs.com/wiki/index.php?title=AsyncLabsWiki)
6. The arduino.cc forum is a great place for more advanced stuff (http://www.arduino.cc/cgi-bin/yabb2/YaBB.pl).

On to the code:

The server is very simple and is implemented in code as a header file; this is because of the relatively low power of the arduino. To explain how it works I will first explain a normal web server. In a normal web server everything is separate and largely reliant on operating system functions. The operating would handle the network interface via a driver to control it. It would accept packets sent to it, deconstruct them and send the information within them onto the server (eg Apache web server) that would act upon them. In general this means to check the URI (Uniform Resource Identifier) for the page required. The server program gets that page from the hard drive (again via operating system calls), compiles it as required and sends the information back to the OS to be put into packets and sent. Most of the functionality therefore is performed by the operating system with the server only required to generate the page. The page itself can be written in a multitude of web languages (in plain text) because the web browser compiles and maintains the pages automatically. In terms of the OSI Model the operating system performs all functions up to session (the sixth), with the web server only fulfilling the application layer.

The arduino + wishield handles serving web pages at a much lower level. The wishield itself has a driver that is written in c and compiled into the program when you create it. This handles the set up / configuration (which pins to use etc.) as well as general running of the wishield. The device as a whole splits the network stack between the arduino and the wishield. The wishield handles the OSI model up to around session before handing data off to the arduino for processing. The reason for this of course is that the arduino itself only has 2k of RAM; a full size Ethernet packet (1492 bytes) would largely fill this. The wishield however is a dump chip that requires a reasonable amount of arduino side programming to make it work.

The server itself is also written in c and loaded as a header. I used the SimpleServer application as a basis; It is available in File -> Examples -> WiShield -> SimpleServer once you have installed the wishield drivers and examples. This server handles the sockets and connections, accepting requests for a web page (GET) and returning status 200 (OK) along with a coded web page. With the use of constructs, variables, etc you can then check the requested URL and send whichever page is required. For a much simpler project involving this webserver you should check out my led server writeup at http://asynclabs.com/forums/viewtopic.php?f=18&t=104. This checks the url and turns a set of leds on or off as required. For a step in the other direction, you could check out tjk's wifi SMS gateway project at http://asynclabs.com/forums/viewtopic.php?f=18&t=89. For more generic information on the server itself you should check out the asynclabs example wiki page at http://asynclabs.com/wiki/index.php?title=AsyncLabsWiki, or a short page on the webserver itself at http://asynclabs.com/wiki/index.php?title=WebServer_sketch.

In terms of setting up the wishield on a network, this is hard coded into the software before it is uploaded. It may be possible to set up the network after the setup of the device, but I didn't have to go into it. The specific section of the code if you are interested comes looks like this:

// Wireless configuration parameters ----------------------------------------

unsigned char local_ip[] = {10,1,1,5};    // IP address of WiShield

unsigned char gateway_ip[] = {192,168,1,1}; // router or gateway IP address

unsigned char subnet_mask[] = {255,0,0,0};   // subnet mask for the local network

const prog_char ssid[] PROGMEM = {"DLINK"};             // max 32 bytes

unsigned char security_type = 0;        // 0 - open; 1 - WEP; 2 - WPA; 3 - WPA2

// WPA/WPA2 passphrase

const prog_char security_passphrase[] PROGMEM = {"12345678"};   // max 64 characters

// WEP 128-bit keys

// sample HEX keys

prog_uchar wep_keys[] PROGMEM = {0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d,     // Key 0

0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,     // Key 1

0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,     // Key 2

0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00      // Key 3

};

// setup the wireless mode

// infrastructure - connect to AP

// adhoc - connect to another WiFi device

Here I set up a simple connection without encryption on the 10.1.1.5 network. The default gateway is left at 192.168.1.1 because my build did not need client access to the internet. It uses the 255.0.0.0 subnet and will automatically try to connect to the "DLINK" network.

This will do for now. If anyone else has any questions post them here and I will update this page. Also if you spot anything wrong or worthy of improvement drop me a line at kimbecause -a-t- gmail dot com.

Arduino: Standalone wifi http messenger

2009-10-01T15:23:00.001+08:00

So I got an arduino and wishield the other day and was wondering what cool stuff I could do with it. Coupled with the lcd that I had as well I figured I may as well just combine everything into a funky little message screen. I started off by getting the lcd working on the breadboard. This took a while in itself and I think by the end I must have known every pin on the board. This was pretty messy I was using wires I had cut out of a network cable and they kept breaking and pulling out of their sockets. A trip to my local jaycar electronics was in order.

[edit] Did have a link to it here but I turned it off.

So I bought a new soldering iron kit, a bitchload of resistors and some prototyping boards. I soldered some headers up to the lcd to make it neater and wired this onto a board:

I also bent the pins on some male to male cables so the 5v and ground cable could be easily plugged back into the arduino. The soldering is a bit messay this is the first major soldering job I've done. It's been a long time since I was a kid ripping resistors out of old electronics to use later. So I tested it, stuck it all together and ended up something like this:

I had to stuff some paper down between my soldered board and the lcd because something was shorting down there and nothing was coming up. So at this stage I had done most of the coding (mostly between getting the LCD working and getting the soldered board together). I noticed however that if a message came in and you missed it you wouldn't know, and that there was no way to show the message on screen. Basically there was no input or output apart from the web and lcd. So anyway ... I went back and added a button and led to a breadboard on the side. At some stage I'll put these on a proper board but cbf for the moment. So now we are up to

Finally here is a little video of it in action:

When the arduino receives a message it blinks the light until you press the button, which also lights up the display for the normal time. All In all I think it worked out fairly well. I have it plugged in to a 12v notebook adapter at the moment. One day I'll get a smaller wall wart because this brick isn't very efficient. My friends that I've told have had a little too much fun sending me messages. Things to do next would include as above to put the led ad switch on their own board. Change it over to battery (wouldn't last very long) or a smaller power supply. Also as it stands while the WiShield does support wep, wpa and psk2 the arduino couldn't hack the psk2 that runs on my network. At some later stage I will rectify this.

For those that are interested in creating your own or something like it, here is the schematic:

I must warn you this is the first schematic I have ever done, and is of the first major electronic build I have ever done. It will quite possibly have errors either within the electrical circuit, or within how I have done the schematic itself.

And finally here is the code I am running on the arduino to catch URLs and server the web page:

Edit:

I did the schematic up from my working implementation, did the switch implementation wrong when I put it into the drawing program.

It should go like this:

5v -------switch ----------- 10k ---------- ground
                         |
                         |  100 ohm
                         |
                       pin 1

This means that normally the pin is grounded through the 100ohm and 10kohm line (pin low). When you press the switch down almost all the 5v goes through the 100 ohm to the pin (the easier path). Some leaks through the 10k ohm to ground but enough gets through to register pin high. If the ground wasn't there the pin would get floating voltage which is invalid input.

In the schematic I have online we have

gnd --------switch ------------ 10k ------------5v
                           |
                           | 100 ohm
                           |
                         pin 1

This means that the pin always reads high through from the 5v through the 10k and 100 to pin1. When you press the switch down most of the power goes through the 10k ohm, through the switch to ground that makes a short circuit. The arduino should crash or at the very least the switch will do nothing.

Edit 2:
There was a link above to the source code that I made to run on the arduino. In moving the blog to blogger years later I lost this, but I don't know if anyone is interested enough in this project now to bother trying to find a backup. If you are reading through this and want the source post a comment and I'll go to the effort of digging through years of backups to find it.

Linux: Logitech G15 with Ubuntu 9.04 Jaunty

2009-09-23T07:04:00.000+08:00

This is another of those things that used to be quite difficult but due to the ongoing diligence of the coders is now awesomely easy.
Let's begin.

First of all use the ubuntu repository to get g15daemon and g15composer. These are the programs that actually control the lcd at the lowest level:

sudo aptitude install g15daemon g15composer

So once these scripts start up you should see a giant clock on your lcd screen - success at mark one. The buttons also work: the circle on the left does nothing yet (it does, but nothing useful yet). The first button changes between 12 and 24 hour, thesecond changes options on the screen. The third changes the visible screen and the fourth does nothing. From here you can also get the audacious plugin to use these programs directly:

sudo aptitude install audacious g15daemon-audacious

Then start audacious and go ctrl + P, then plugins -> visualization and click G15daemon Visualization Plugin. This screen is pretty cool so if a clock and audacious spectrum is enough for you, I would stop here. If you want more we will move on to LCDproc. I found it easier here to get the main program from the ubuntu repositories:

sudo aptitude install lcdproc

This will get the program and install everything inthe right places, unfortunately however it doesn't seem to like installing the g15 driver when it does it. To get this we will download the source and build the driver, and then copy it to the normal place that the lcdproc drivers live. First we will install some more supporting programs:

sudo aptitude install libg15render-dev libg15daemon-client-dev libg15-dev

Also for reference everything above I found on this blog. Below however I will deviate a bit to make it a bit easier. We'll get the source the same as normal:

cd ~/Downloads/
sudo aptitude source lcdproc
tar -xvzf lcdproc*

Now we have the source in a folder call lcdproc-0.5.x - for me this is 0.5.2 so I'll use that from here in. Note you may need to change the tar line above to point to the file you downloaded. Now on to compiling:

cd lcdproc-0.5.2
./configure --enable-drivers=g15
make

This will tell the source to specifically build the g15 driver along with the server. Now we'll just set up the lcdproc file and check the paths. Stay in this folder but execute:

sudo nano /etc/LCDd.conf

This is the config file for the server that connects to the low level G15 hardware. From here clients connect to this and send it what to display. So firstly, in the [server] section, go down and find

Driver=curses

Change this to

Driver=g15

Skip down to the section that is talking about serverscreen and uncomment the line that says

ServerScreen=no

This will stop the server information (that you will see in just a second) from fighting with the proper screens that you want to see. Finally skip down to a line that looks something like

DriverPath=/usr/lib/lcdproc

(etc. - this is what mine says). Copy this to gedit or whatever and hit ctrl-X and save. You can edit the rest of that file if you want it is very well documented; just remember to back it up first.

So now we will copy the driver we compiled to this directory. For me this was of course done with

sudo cp server/drivers/g15.so /usr/lib/lcdproc/

From where I was, or to use the full paths:

sudo cp /home/kim/Downloads/lcdproc-0.5.2/server/drivers/g15.so /usr/lib/lcdproc/

Yours may be slightly different for different lcdproc versions / different configurations of where lcdproc is looking for drivers. Now everything is set up we can restart the lcd daemon with the new settings:

sudo /etc/init.d/LCDd restart

If everything worked out you should now see a screen that says "LCDproc Server" across the top and Clients / Screens down the side. This meens the server is successfully talking to the g15 daemon, which is successfully talking to the lcd. But this is more boring than the last thing it was doing you may say. But of course, we have set up the server but not a client to connect to it and tell it what to do. Run

lcdproc

as any user and you will see ... the time. Then it will switch to some nice system stats, and continue to cycle through random screens and the lcdproc server screen. Now the left and right square buttons move more quickly between screens. The large circle button changes between the default g15composer program and the new lcdproc daemon client program. Note also: this last little bit I got from reading between the lines of http://linuxgazette.net/issue77/taneja.html.

You'll probably want to kill it for a sec so you can configure it and restart it:

killall lcdproc

From here it is up to you as to what you want displayed on the screen. I think it is generally easier to have only one screen active at once, switching between them is kinda distracting. The easiest way to set screens to on or off seems to be to edit the default configuration file:

sudo nano /etc/lcdproc.conf

The file seems well commented and fairly easy to follow. If you set all the screens to off then you can also switch them back on manually using eg

lcdproc C

for the cpu screen. When using this screen you may want to set foreground=true in lcdproc.conf to make it easier to test the different screens. You can see a full list of these commands for lcdproc by typing

lcdproc -h

Alternatively a lot of programs can access LCDProc directly - a major one is xbmc (post more in the comments - what do you want to use the lcd for?). That is about it though. So

FIN

Ubuntu: Wireless on the MSI U100 (wind) With Ubuntu 9.04 Jaunty

2009-09-23T06:58:00.000+08:00

It used to be fairly difficult to get the wireless working on any flavour of linux with this laptop. I bought mine with grand visions of wireless n and stealing the uni's wireless. This was not the case, with the 8187se chipset from realtek working badly in Windows and worse on linux. Thankfully going back to it now there is an easier way. NOTE BUT: This should only work with jaunty jackalope (9.04) that has the source driver in it's repository.

So after updating your repositories with
sudo aptitude update

we can run
sudo aptitude install rtl8187se-source

to download the source and then
sudo m-a a-i rtl8187se

to compile the module. From here just edit /etc/modules so it is loaded at startup:
echo "rtl8187se" >> /etc/modules

Reboot the computer to load the module and set everything up and it should all be good. Run
lsmod | grep rtl

to make sure the module is loaded and you should be able to connect normally using wireless manager.

ANOTHER NOTE BUT: You have to make sure the wireless adapter is actually enabled, as it seems to be cut off in the laptop in hardware if it is turned off. Just press function + f11 until there is a yellow light under the sattelite icon. If you don't everything will work as normal, but you will never see any networks.