Aim of Project
The point of this hobby project is to build a neat user-interface (UI) for an application. The idea for the UI is:
- It will show both 2D graphs+information, as well as contain a 3D representation of an environment
- It will allow a user to interact naturally in both the 2D layer (say with buttons, list boxes, etc.) as well as via the 3D layer (clicking and ray-tracing)
- It will be communicating with an external real-time data source
- It has to run cross-platform
- It should be flexible because the requirements can change at short notice (…show me a project that doesn’t have this clause :p)
This is very similar to a number of projects that we’ve worked on in the past. I doubt I’m the only one. I mean, let’s be honest, who doesn’t want a really fancy UI for their application?
“…No no no! None of that fancy graphics crap, I want a terminal with that psychedelic-green command prompt… 32-bit my technicolour butt… Great, make it print the position coordinates and debug information line by line! … Yes, now make it print faster so we just can’t quite read it! YES! Like the Matrix, that’s right! Hell yes! Omg the iPhone users are going to love this!!!!”
And we’ve attacked these requirements in a bunch of ways. Personally, off the top of my head, I’ve built projects using:
- MatLab graphing integrated into XPDS for communications and control – plane autopilots
- A pretty WPF UI wired up to a custom communication backbone – data analysis and perimeter monitoring (kudos to Alucard for the UI)
- DirectX shaders drawing Kinect and AR Drone data – flying drones with a VR headset (ARX)
Well here is another way to do this. But, this is one that really quite blew my kilt up a bit. It’s elegant without being overbearing and quick without being, well, hacky. It has it’s downsides, mostly that there’s not a lot of documentation, which is why I thought I’d try string a few pages together.
If you want a feel of the technique and haven’t watched the videos of MIT’s entry into the DARPA Urban Challenge, I’d recommend you have a look at ‘A Summary of Team MIT’s Approach to the Virtual Robotics Challenge‘ to get an idea (shown below). That’s a great example of what you can do with this type of visualization+communication backbone. That blew me away – it’s like 3D Max, but for robotics!
The original idea actually stemmed from talking to some of the people on that project. We just started another visualization+communication hobby project and some of the guys that worked on that recommended a similar software structure. Heck, I originally was going to go with a clone of the DirectX method (ARX video, which is shown below), but the VTK/LCM flexibility alone closed the deal, even if you want to take the coolness factor out of the considerations.
So what will be used in this set of articles? The following technologies will be used as the baseline, but they are subject to change as I find weaknesses or something neat. Please let me know if you have any recommendations or know of any potential landmines.
What does it do?
- Python is a coding language and will be used to weld all the independent software components together into a functional UI
- No memory management
- Cross-platform with no serious complications
- No compiling, it’s a scripting language
- I was using it for a data analysis project and developed a liking for it (it’s an upcoming MTA-subway data analysis post)
Daniel posted a good question in the previous article – ‘Why not just C/C++, VTK, and Qt?’ – which is something I didn’t think to address. Honestly, it comes down to development time. Although C would be faster at runtime:
- Python does the memory management (no chasing segfaults – hey it happens to the best of us, me more than usual though)
- What you build will run cross-platform by just copying the files across. If the Python import statement works, the program should run. C/C++ cross-platform is a little more tricky sometimes
Visualization ToolKit (VTK)
What does it do?
- VTK is an OpenGL-based visualization framework from Kitware that’s used for scientific plotting and data visualization. It specializes in 3D, so it’s a little different from matplotlib or ggplot
- Powerful for rendering large complex 3D scenes with point clouds, meshes, and models
- Hardware accelerated – performance is not a problem
- VTK has been shown to be suitable for such applications from the case studies
What does it do?
- PyQt is a wrapper for the Qt UI library. It allows you to build forms and dialogs, similar to Microsoft’s WPF or Java’s SWING
- All the buttons, list boxes, combo boxes, text boxes, browsers, multimedia player etc. you can handle
- Neatly integrates on top of VTK to allow you to build 2D UI’s on top of your 3D environment
- It’s possible to design a GUI in Qt Designer and export it for use in Python via PyQt (fancy GUI’s without fiddling in code – win!)
LCM (Lightweight Communications and Marshalling)
What does it do?
- LCM allows you to publish and consume real-time data over a network between different machines and in different languages
- Skip all the painful network socket code in a networking application
- It reminds me of the XPDS networking layer so the design is familiar (but LCM is way more extensive)
- Think ROS (Robotic Operating System) but… well… lighter, from what I’ve heard. I don’t have too much information on this, but it was recommended by a really good team so that’s good enough for me (proof by association, lol)
Great, so with all the technologies laid out, the next article will discuss the steps required to set up a development environment for this project. Here is a quick summary of the discussed components:
- LCM will handle the communication between the source(s) and the UI
- Python will be used as the backbone language
- The 3D portion of the UI will be built in VTK
- The 2D portion of the UI will be built in PyQt (either programmatically or with PyQt Designer)