Building vtk User Interfaces: Part 4 – LCM Integration

With the changes to the project over the last few months, I totally forgot to post this! LCM is pretty awesome and was used in the example with the Wrap 920 glasses, so here we go…

LCM (Lightweight Communications and Marshalling) is a communication framework that allows you to transfer messages between machines on a network. This post provides a quick overview of LCM and then shows how to integrate a few messages into a Python example project.

Introduction to LCM

Rather than running away with a big discussion on it, let me hit a few questions right off:

  • Why do we want to use it?
    • So up to now, you have this fancy 3D UI. However, the aim of this project is to couple the UI to a real system (even though we’ll fake a networked robot in these posts)
    • How do you pass messages from the robot/device to the UI? What if there are multiple robots? What if I’m swapping between languages and operating systems, how do I handle the interpretation of the messages?
    • A lot of people address this differently, which in my experience seems to be:
      • IT background – Let’s use a REST service/a webpage or an XML message that can be requested (pull architecture)
      • Engineering background – Serial, everything will be over serial comms and if the project goes beyond the POC stage we can use a serial->WiFi converter
      • Both are fair answers, but the first adds a lot of overhead and both are a lot of effort because they require translating data into message frames (especially the serial comms case)
    • LCM is a simpler answer to these questions, and allows you to do all that with minimal effort – it’s a soft real-time communication framework that allows complex messages to be produced and consumed across a network in various languages (C, Java, C#, Python, etc.)
  • What is it built on?
    • LCM uses UDP broadcasting to pass messages around – everyone gets the messages, some choose to consume it
    • Note that because it’s based on UDP:
      • It’s applicable to soft real-time applications (as opposed to, say, RTNet in Xenomai/RTAI) so it’s great for supervisory systems but probably not suitable to networked real-time control systems. We are working towards a visualization platform for a robot (or multiple robots) so it’s an awesome fit
      • It’s a best-effort system, because UDP doesn’t make sure the message reached the destination. This isn’t really a downside – it’s good for performance – but it means that you probably should consider integrating your own heartbeat in complex, time-dependent systems
  • What languages and platforms is it applicable to?
    • Oh, it can run on anything with minimal installation. That’s the really great thing about LCM, the guys that wrote it really put some effort into making it simple to integrate and use
    • Yup, anything from Ubuntu to Windows…
    • And you can talk in soft realtime between C, C++, C#, Java, Python, Julia, etc.
    • …Getting excited? 🙂
  • Who built it?
    • LCM was built by the MIT DARPA Urban Challenge Team
  • Aren’t there similar technologies?
    • Wow, difficult question. Well, yes, you could use ROS (Robotic Operating System), Oxford’s MOOS, or write your own XML message passing, but LCM has a few unique features that make it appealing (just post a comment if you’d like to discuss this is more detail)
  • Where do I read up on it?

If you haven’t installed LCM yet (part of the first series of posts), you can follow the steps on the GitHub page, or take a quick look at Part 2: Setting Up Your Dev Environment.

The remainder of the post will:

  1. Create a few LCM types that we can use
  2. Integrate the new types into two sandbox Python scripts to make sure everything’s copacetic (hello world!)
  3. Test the new types by running the scripts in parallel

Creating LCM Types

The first thing I think everyone asks (including me) when looking at LCM is: “What data can it transfer and how does it look?”.

LCM allows you to specify complex structs that can be passed around, very similar to C-style structs. It handles all the encoding and framing, you just need to tell it what to send and receive. That means that you can transfer anything from simple strings and vectors all the way to camera video feeds and point-clouds. Yup, it can handle multi-dimensional arrays and hierarchical data types…

multidimensional_arrays

Discussing the frames and what can be included is akin to jumping head first down the rabbit-hole (at least in this post). More information about the frames and how they are constructed can be found at LCM Type Specification Language. For this projects, we’re going to run with what we have and build a few LCM types that are suitable for what we need. The types are derived from the LCM tutorials, which can be found at Defining a data type – example_t.

For this project, we’re going to define 3 LCM structs which will be passed around over the network:

  1. A struct that updates the vtkGUI as to the new bot’s information – called ‘bot_info’
  2. A struct that updates the bot with new references, or set-points, as to where we want it to go – called ‘bot_reference_update’
  3. A struct that tells the bot to change how it behaves (set it active, to idle, etc.,) – called ‘bot_set_status’

LCM Message: Bot_info

The first message is one that is transferred from the bot to the vtkGUI. This contains information about the bot’s position, orientation, and status. In LCM code, this is defined as:

//A frame for the bot info which will be transferred to the vtkGUI.
//Frame adapted from: http://lcm.googlecode.com/svn/www/reference/lcm/tut_lcmgen.html
struct bot_info
{
	int64_t timestamp; //The timestamp of the current frame in ms
	
	//The current position/orientation
	double position[3]; //The current position
	double orientation[4]; //The orientation as a quaternion
	
	//The current reference (set-point) position/orientation
	double ref_position[3]; //The current position
	double ref_orientation[4]; //The orientation as a quaternion

	//A set of constants that can be used as a status enumeration.
	const int32_t STATUS_OFFLINE=1, STATUS_STARTINGUP=2, STATUS_ACTIVEIDLE=3, STATUS_ACTIVEREFS=4, STATUS_ERROR=5;
	int16_t status; //The bot status
	string name; //The name of the bot
}

Most of these members are self-explanatory, but a few points of interest:

  • The frame contains the position as a 3D vector, and the orientation of the bot as a quaternion. It’s just an example frame, so don’t stress too much about the quaternion and the definition of the components – the power is in the simple definition of a complex message
  • The frame also contains a set of feedback references (set-points) that indicate what the bot thinks its reference are (where it wants to be and the desired orientation) – this is important as it allows the bot to acknowledge that it is tracking the requested references
  • Finally the frame contains the current status of the bot and it’s name. We include the name in the event that we have a few of them and want to address each individually on the same channel

LCM Message: Bot_reference_update

//A frame that updates the references (set-points) for the bot. This is sent from the vtkGUI to the bot.
struct bot_reference_update
{
	int64_t timestamp; //The timestamp of the current frame in ms

	//The reference (set-point) position/orientation
	double ref_position[3]; //The current position
	double ref_orientation[4]; //The orientation as a quaternion

	//The name of the bot
	string bot_name;
}

This is similar to the ‘bot_info’ message, except that it is sent to the bot, and contains the desired position and orientation for that specific bot. The GUI can send this down to indicate that the bot needs to change where it is and where it is looking.

LCM Message: Bot_set_status

//A frame that requests the bot to change state
struct bot_set_status
{
	int64_t timestamp; //The timestamp of the current frame in ms

	//A set of constants that can be used as a status enumeration.
	const int32_t STATUS_OFFLINE=1, STATUS_STARTINGUP=2, STATUS_ACTIVEIDLE=3, STATUS_ACTIVEREFS=4, STATUS_ERROR=5;
	int16_t request_status; //The requested bot status

	//The name of the bot
	string bot_name;
}

Finally we have a message that sets the bot’s status. It’s sent from the GUI down to the bot and is used to tell it to change from, say idling around to actively chasing the reference values.

Building the LCM Types

So now that these LCM types are defined, we need to put them in a file and compile them for LCM. This is straightforward and done with the following steps:

  • Create a new text file in the root of your project folder called ‘vtkGui_messages.lcm’. GEdit is good for doing this, just open it in the Start menu of Ubuntu
  • Paste in the following code, which was discussed above:
package vtkgui;

//A frame for the bot info which will be transferred to the vtkGUI.
//Frame adapted from: http://lcm.googlecode.com/svn/www/reference/lcm/tut_lcmgen.html
struct bot_info
{
	int64_t timestamp; //The timestamp of the current frame in ms
	
	//The current position/orientation
	double position[3]; //The current position
	double orientation[4]; //The orientation as a quaternion
	
	//The current reference (set-point) position/orientation
	double ref_position[3]; //The current position
	double ref_orientation[4]; //The orientation as a quaternion

	//A set of constants that can be used as a status enumeration.
	const int32_t STATUS_OFFLINE=1, STATUS_STARTINGUP=2, STATUS_ACTIVEIDLE=3, STATUS_ACTIVEREFS=4, STATUS_ERROR=5;
	int16_t status; //The bot status
	string name; //The name of the bot
}

//A frame that updates the references (set-points) for the bot. This is sent from the vtkGUI to the bot.
struct bot_reference_update
{
	int64_t timestamp; //The timestamp of the current frame in ms

	//The reference (set-point) position/orientation
	double ref_position[3]; //The current position
	double ref_orientation[4]; //The orientation as a quaternion

	//The name of the bot
	string bot_name;
}

//A frame that requests the bot to change state
struct bot_set_status
{
	int64_t timestamp; //The timestamp of the current frame in ms

	//A set of constants that can be used as a status enumeration.
	const int32_t STATUS_OFFLINE=1, STATUS_STARTINGUP=2, STATUS_ACTIVEIDLE=3, STATUS_ACTIVEREFS=4, STATUS_ERROR=5;
	int16_t request_status; //The requested bot status

	//The name of the bot
	string bot_name;
}
  • This code defines an LCM package called ‘vtkgui’ which will contain the discussed types
  • Open a command prompt and navigate to the project folder
  • Run the following code to compile the LCM types into Python-friendly code:
lcm-gen -p vtkGui_messages.lcm
  • You shouldn’t see any errors, if you do please let me know in a comment on this post
  • That’s it! We’ve now got Python code for accessing these types, which can be found in the project folder under ‘./vtkgui’. We’ll use the files as-is and just import them to describe the custom types to LCM when we pass them around in the program

Testing Types in Python – Hello World!

Now that we have LCM types and the Python code to use them, we need to test it all out before integrating it into the existing project. The pickle with testing this code is that we need two programs: one to act like the bot and the other to act like the vtkGUI. For the test, the “bot” is going to produce a ‘bot_info’ frame and broadcast it; the other part of the test-code is going to wait for messages and print them when they become available. Both will run on the local machine to make testing simpler.

In this section, I’m going to assume that you’ve read a bit on LCM and know what it’s expected. If you haven’t, no worries, but you might want to review the Python Tutorial of the LCM tutorials, which discusses the imports etc. in mode detail than I can fit in a post (…I’m getting hand cramps as we speak). This code is largely derived from it, so it should explain how the code below works.

Hello World “Bot” Code

</pre>
<pre>### --- Program to produce a simple message [GearsAD] ###
### Ref: http://lcm.googlecode.com/svn/www/reference/lcm/tut_python.html ###
import lcm
import time
#Import all our message types
from vtkgui import *

#Create a new test bot_info message that we are going to send.
msg = bot_info()
msg.timestamp = 0
msg.position = (5, 5, 5)
msg.orientation = (1, 0, 0, 0)
msg.ref_position = (5, 5, 5)
msg.ref_orientation = (1, 0, 0, 0)
msg.status = msg.STATUS_ACTIVEIDLE
msg.name = "testbot"

#Push the message out to a channel called "vtkgui_bot_1"
lc = lcm.LCM()

try:
    while True:
        print "Sending a message on channel 'vtkgui_bot_1'..."
        #Push a bot_info message onto the channel
        lc.publish("vtkgui_channel_bot_1", msg.encode())
        #Wait 2 seconds
        time.sleep(2)
        
except KeyboardInterrupt:
    pass #Finished

If you run this code:

hello_world_bot

Hello World “vtkGUI” Code

<pre>import lcm

from vtkgui import *

def my_handler(channel, data):
    msg = bot_info.decode(data)
    print("Received message on channel \"%s\"" % channel)
    print(" timestamp = %s" % str(msg.timestamp))
    print(" position = %s" % str(msg.position))
    print(" orientation = %s" % str(msg.orientation))
    print(" Reference position = %s" % str(msg.ref_position))
    print(" Reference orientation = %s" % str(msg.ref_orientation))
    print(" Status = %s" % str(msg.status))
    print(" name = '%s'" % msg.name)
    print("")

lc = lcm.LCM()
subscription = lc.subscribe("vtkgui_channel_bot_1", my_handler)
try:
    while True:
        lc.handle()
except KeyboardInterrupt:
    pass

If you run this code (with the ‘hello_world_writer.py’ running in parallel):

hello_world_vtkGUI

Things to Note About LCM

Padrissimo, how cool was that when they start talking!

There are a few things to note when starting with LCM:

  1. If you don’t call ‘lc.handle()’ for you subscribed objects in your reader code, you won’t catch any messages. This seems a little obvious, but I’ve stared at the screen for a few hours today, scratching my head and cursing in 14 different tongues:
    1. Best practice? Wrap that in a thread and let it run indefinitely – we’ll do t this in the project code. If you want to try that with the sandbox code, a good place to read up on threading in Python is the tutorial on TutorialPoint
  2. If you have hierarchical LCM types (say you define a custom ‘sensor_info’ LCM type, and then use it in the ‘bot_info’ frame), you have to initialize the value every time you want to send it. If it isn’t set, you’ll see a strange message (‘AttributeError: ‘NoneType’ object has no attribute ‘_get_packed_fingerprint”) that is actually related to the fact that your child ‘sensor_info’ object is null
  3. In some instances you may send large messages (e.g. camera data) – LCM may complain that your kernel UDP buffer is small. If you are considering large messages, you may want to update the buffer size before testing, which is well documented at LCM: UDP Multicast Setup

Summary

This post has been in my drafts for a few months – sorry about the late post! I’m not sure I will have time to show how this is integrated into the visualization platform (working on another system at the moment) but, if you look at the Wrap920 example, it has a fully-working project that uses LCM directly.

– Sam

Advertisements

One thought on “Building vtk User Interfaces: Part 4 – LCM Integration

  1. Pingback: New Arrival! Eye-tracking… | Semi-Sorted

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s