System Infrastructure

NOTE: This page is rather hopelessly out of date; for a current discussion of the system design, please look HERE.

Despite severe space and weight constraints, the Microships carry an extensive network of electronic systems. As with BEHEMOTH, all of them can be loosely grouped into two broad overlapping categories: infrastructure and applications.

At the infrastructure level, the system has to deal with power management and distribution, thruster control, routing of audio/video/serial channels on demand, ship security, internal performance tracking and diagnostics, console pressure and temperature monitoring, dynamic network configuration including remote wireless control, and so on... the low-level tools that keep the boats working, provide access to resources, and are always on and thus must therefore use power as efficiently as possible. In this category, we are jealous of every milliwatt, and this affects the design profoundly.

The applications layer includes obvious nautical systems such as navigation and communication, live satellite weather image display, scanning sonar imaging, digital video production and camera management, music composition and MIDI control, environmental data collection with hourly posting of telemetry blocks to the Net, general Internet access for email and web use, ham radio, manpack tracking, the whole suite of productivity tools accessible while sailing or off-boat, and ongoing software development.

It thus became obvious early in the development of the Microship system that some unique design challenges were present:

• central server-based multitasking environment on each boat that can accept asynchronous data from a variety of sources and stream it to a variety of clients, including real time displays and a scriptable database...
• random software-controlled interconnection of analog and serial channels...
• distribution of independent control resources across a low-level network to simplify development and minimize power drain...
• graphic front-end access via web browsers, whether local to a boat or via wireless remote connections...

Let’s take these key features one at a time...

On-Board Server

I should have expected this. When I started working on the Microship project back in 1993, uncertainty about how to proceed with the nautical substrate induced me to focus instead on the on-board control network. With the help of UCSD students and industry volunteers, a system took shape: a FORTH 68HC11-based multidrop network. It actually worked very well, and we started writing front-end tools – briefly in C on a PC, then in HyperCard on a Mac, then in NewtonScript to support wireless handheld computers. I vaguely recall joking about how it would all be obsolete by the time I finally got the boat done, and, well, that turned out to be the case.

Obsolescence is a touchy subject... some people abandon old systems just because marginally faster new ones are available; others hold on to once-whizzy clunkers until antique value causes the price to start back up. In our case, the deciding factor was a sea change in networking technology and the availability of single board computers so robust that we could run a full-scale linux server with modern tools... for about the power cost of a typical nightlight. What does that buy us? Nothing less than the ability to see the Microship as a web server, with all control and monitoring widgets implemented as a website, available both locally on the boat’s console screen or from any computer within a few miles... or anywhere in the world, for that matter, given our satellite Internet link.

Phew.

Using a single-board Pentium-class computer from Octagon, we brought up an implementation of Debian GNU/linux and went to work. Throughout this process, we redefined the intent of this whole system... eventually viewing it as a set of processes clustered around a central database and communicating through sockets. Each physical resource is "owned" by a single process, such as the Hub server that controls the connection with the long-established New Micros FORTH board that (together with a boatload of interface hardware) manages the crossbar networks and handles all the low-level I/O on the ship. Any process that wants to talk to hardware, whether to schedule an audio crossbar connection or snag a snippet of environmental data from a sensor, thus becomes a client of the Hub server, which in turn handles the gritty details of interacting with the FORTH interpreter and other low-level tools.

One of the ongoing objectives here is the collection of many flavors of data pouring in through serial, analog, and digital channels. Readings are time-stamped and dropped into a database, multichannel snapshots are transmitted to the public web server on an hourly basis, equipment-performance datasets are archived for sponsors, and – depending on the current situation – any arbitrary set of inputs can be displayed live on the console browser as stripcharts or simulated instruments. A key part of all this is the Realtime Data Broadcaster, which provides a publish/subscribe service for any client interested in receiving realtime data.

The fun part about all this, of course, is the variety of applications that now become possible. An example is the virtual console that will run on the canoe's browser: it's a configurable screenload of applets including a scrolling chart segment with a current location icon, basic nautical navigation tools, relevant live sensor data, any subset of eight moving strip charts vertically arranged on the same timebase, power system performance instrumentation, and links to everything from extensive online documentation to close-up windows focused on individual subsystems. Bear in mind that this architecture will be replicated on Natasha’s boat, and possibly even in the manpack computers... allowing any browser in our wireless network to connect to any of the local URLs just like surfing to a website on the Internet, yielding a fully capable front-end control panel complete with online documentation, deep archives, image feeds, live audio, real-time displays, messaging, and more.

Most of this high-level stuff is written in Object Perl, and takes advantage of the whole industry that has grown up around the Web as well as the open-source linux community. But the real beauty of this application depends on interactions among distributed resources, requiring quite a bit of low-level code and hardware hackage...

Grand Central Station

One of the most troublesome problems facing this design was the sheer number of hardware devices that have to interact. We’re not talking bus-compliant systems speaking a standard interopable data format here; we’re talking about celphones, marine radios, video cameras, sensors spewing NMEA or RS-232 datastreams, frame grabbers, tiny dedicated control interfaces and translators, and so on. From bike experience, I know that any “complete set” of anticipated interconnects will become obsolete the first time a seductive new gadget arrives – the Winnebiko II, by the end of its era in 1988, was a study in reassigned console switches and multiple layers of annotation scrawled upon already lousy documentation.

For BEHEMOTH, we developed an audio crossbar around the Mitel 8816 chip, vastly simplifying those spur-of-the moment connections that are irresistibly associated with having lots of devices. It took only a few lines of FORTH code, for example, to respond to a lat-long change without the right password by dialing 911 on the cellular phone and having the speech synthesizer intone, “I am a bicycle and I am being stolen. My present coordinates are...”

This sold me on the crossbar architecture for like-flavored signal paths, and the “Grand Central Station” region of the Microship system includes three of them... a cabling-intensive zone in the console with 244 connectors.

The audio crossbar (Auxbar) allows up to 8 simultaneous line-level connections among any of 32 sources and 32 destinations. This consists of a stack of two boards bristling with gold-plated RCA jacks, so all we have to do to add an audio device is run a cable and add a line to a config file to allow reference by name. Natasha’s boat joins the network via a short range stereo full-duplex transceiver for quality audio and a 70-cm radio link for voice-grade exchanges... devices which are, of course, merely addresses on the Auxbar. At her end, a scaled-down version of the same design allows the addition of multiple devices as well.

The video crossbar (Vixbar) is similar, but optimized for higher-bandwidth signals and the need to channel switch during retrace intervals to prevent glitches. This system can handle 8 simultaneous links among any of 16 sources and 8 sinks, and, since it is less likely to be on call 24 hours a day, is supported by a dedicated FORTH node and software-controlled power switch. This network is also expanded to the other boat via a video transceiver, since Natasha runs the Draco Casablanca video editing system and needs to select camera channels on the fly; like audio, this system appears as an array of RCA jacks and is trivial to expand or edit. One-line commands make or break any link, add a receiver to an existing connection, or clear all of them.

The serial crossbar (Sexbar, of course), provides up to 4 simultaneous bidirectional connections among any of 32 serial channels – 28 of which appear as a dense 4x7 array of DB-9 connectors pop-riveted together. This one is particularly amusing, since it also has to solve the annoying problem of RS-232 polarity, those eternally reversed pins 2 and 3. The code in the Sexbar controller, whenever asked to establish a connection, first zips through the involved pins of both channels, connects them one at a time to a window comparator, sees who’s receiving and who’s transmitting, then assembles a virtual straight cable or a virtual null modem cable as needed. The process is so quick and painless that we often link serial gadgets together just for the fun of it, a statement I haven’t been able to make since, oh, about 1973 when the process of making machines communicate was still intrinsically astonishing and UAR/Ts were radical new alternatives do doing it with TTL shift registers and control logic <creak>.

I should address one issue before continuing: I occasionally take flak in these Internet-savvy days for using such primitive networking tools as vanilla serial channels loping along at a poky 9600 baud via hardware-intensive crossbar switches. Why not just give everything a TCP/IP stack and do it right? Well, if the involved devices were all “real computers” that would be sensible, but most of the serial widgets we’re using are small, low-power, dedicated controllers... even PIC processors acting as translators between RS-232 and proprietary protocols like Sony LANC and Dallas MicroLAN. Sticking them all on ethernet would be power-hungry overkill, adding far more hardware overhead than the devices themselves!

The power-control system deserves a quick comment in this section even though it’s not a network, per se. In keeping with the spirit of maximum flexibility, all devices that are even remotely power hungry are hanging on solid-state relays, in turn wired to Phoenix Contact blocks that carry a few dozen bits of parallel I/O attached to the Hub processor. It is a trivial process (running one wire) to add a power-control channel with its own status LED, fuse, and pair of switched screw terminals; it takes just another one-line command to turn anything on either boat on or off.

That takes care of the interconnect layer of our Microship network: with very few exceptions, all random pieces of communication, entertainment, data collection, security, and control hardware appear to the server as nothing more than some subset of audio, video, serial, and power channel names.

The Hub and OtherStray Processors

An interesting problem confronts the designer of a power-limited portable system – even with 480 watts of solar panels and a big honkin’ marine deep-cycle battery, we still care deeply about how many milliwatts are being continuously converted into heat. The linux server is designed to go into a succession of power-saving modes (slow-clock, disk-off, suspend, shutdown, etc.) depending on activity, but even when we’re away from the boats and in no need of graphic front-end tools we still want a faithful electronic watchdog keeping watch over power, security, bilge pump activity, and the like. Besides, some system has to be alive enough to respond to a call, transmit location beacons... and boot up the big iron if necessary.

In addition, there are so many little distributed tasks around the ship that trying to do it all with one central computer would create an I/O nightmare, a software quagmire, and catastrophic single-point failure potential. Even with Mimsy (the web server) shut down, the Microship needs to be capable of taking care of herself in a surprising variety of ways.

The solution to all this is a distributed network of processors engaged in a variety of tasks – autonomous control systems that are happy on their own but can be queried or controlled by server-level code when the big system is alive.

The most central of these “little guys” is the Hub, which was once in the exalted role of running a whole multidrop network of nodes in the early years of Microship design. This folding board stack is based on a New Micros 68HC11 with FORTH in ROM, with loads of added I/O including 64 bits each of individually addressable input and output bits along with analog and serial ports. The packaging includes a hinged “nexus board” that simplifies random little interface hacks, and the system is also the controller of the Auxbar and Sexbar subsystems. A custom multitasker adds considerable utility, as monitoring or control tasks can be handed off to the board and then forgotten, with the results accessible through an easy textual exchange of variables via the Hub’s console port.

(As mentioned earlier, there is a Hub server process that runs under linux, converting FORTH’s vanilla serial console port into an internet-domain socket with full resource locking. We can open multiple telnet sessions to the little 8-bitter, for example, and bang away asynchronously without difficulty. It’s very strange, telnetting to a chip via ethernet...)

Anyway, the Hub’s role in the Microship is both central and subservient: it handles all the signal routing on demand while keeping an eye on all the security and low-level monitoring jobs, reporting to Mimsy as needed (even yanking hardware lines to boot it up if something needs high-level attention, like sending an email message or going into full-scale alert mode). This turns out to be quite convenient, for one of the Hub’s many local I/O devices is a DTMF (Touch-Tone) decoder that sits on the audio crossbar. With the boat completely shut down and parked miles away, I can still send a sequence of tone commands from a ham radio transceiver or cellular phone, causing the Hub to wake up the server so I can surf my way over to the full-scale user interface. And the Hub is empowered to beep me and sound basic security alerts even if it’s all alone on watch.

There are other baby micros scattered around the boats as well, of varying robustness. All of them communicate via vanilla serial ports through the Sexbar, not only with the server but with each other or even remote users... useful, for example, when one of our software developers wants to log in and hack <shudder>. Following is a sampling of embedded processors aboard the Microships, some within commercial products, others developed to solve specific problems. From the system perspective, these are all simply serial devices that either spew data continuously or respond to queries with requested variables or some kind of action...

• Peak Power Trackers, continuously optimizing eight 60-watt “channels” of incoming solar power and delivering detailed performance data on demand.

• Thruster and Power Manager, which keeps an eye on the battery and all system loads, tells the motor’s PWM controller how much free power it can have, and takes care of software-controlled thruster steering with Hall-effect position feedback.

• Video Turret Controller, which aims either the zoomable color or low-light monochrome camera at any specified angle, or scans at any rate between any pair of angles. This takes care of camera power control, lens shield servo retraction, and turret internal environmental monitoring.

• Video Crossbar Controller, which for historical reasons happened to end up in a dedicated processor.

• Wind Sensor, an ultrasonic marvel from Handar on the stern that decuces wind speed and direction by resolving differences in the speed of sound propagation among three transducers.

• Electronic Compass, continuously transmitting heading information at the rate of one sample/second.

• GPS Receiver, which provides a continuous feed of latitude, longitude, and course data.

• APRS and Packet Nodes, which take care of location telemetry among our wandering group as well as email with the ham radio community and the transmission of periodic tracking updates to public map servers.

• Virtual Front Panel, a little PIC-based circuit that maps 48 arbitrary bit conditions onto a front-panel LED matrix.

• HF Radio Controller, responding to serial commands by tuning and otherwise tweaking the all-band amateur radio transceiver.

• Water Quality Analyzer, a remarkable multichannel probe that returns massively detailed reports about the stuff we’re floating in.

• Random Controllers, which translate serial commands into the proprietary interface protocols embedded in the DAT, VCR, and MD devices.

• Speech Synthesizer, a standalone hardware text-to-speech board that lets any device in the system have its say (the linux box has its own software synthesis using ViaVoice).

• Wee Embedded Sensors scattered all over the boat, single-chip parts that input analog values representing temperatures, voltages, stresses, or other parameters and translate them into continuous serial data streams. Others of the same class convert serial commands into 5 bits of local control

And so on. As you can imagine, these are all so thoroughly different (and from a networking perspective, so wimpy) that we’d have a real mess on our hands without the Sexbar. With it, putting this wide range of resources to work is a relatively simple matter of writing drivers and filtering incoming data streams (with parsing hacks and regular expressions) into values that can be retransmitted by the realtime data broadcaster to webpages, databases, navigation software, control loops, monitoring processes, or even little daemons that periodically pack up a telemetry block and email it via satellite to somewhere far, far away...

Front End and Mimsy

So how does this all translate into the day-to-day life of traveling on tiny boatlets? If I’m strolling around town with Microship Io hauled up on a beach, it’s nice to be able to be notified of security alerts, check power system status, confirm my location relative to the boat, and so on. Suppose I want an offshore weather update, delivered to my pocket walkie-talkie in a synthesized voice? The ease of interconnecting resources turns the following into a one-click operation in a web browser or even a spoken command: power up the HF single-sideband radio, tune it to the coast guard NAVTEX frequency for digital weather report transmission, pipe receive audio to a packet radio TNC, send the resulting data stream to a speech synthesizer, then direct its voice output to a radio transmitter. This was one of the first tests of the completed crosspoint network, in fact, and writing the software to do it took about 5 minutes (most of which was looking up or assigning crossbar channels!). After that, it was a single click on a button, displayed on a wireless handheld computer.

This top level server running the network is Mimsy – the Microship Information Management System. Seen by the user as an on-board web site, Mimsy pulls together everything into a single browsable environment: our huge library of system hardware and software documentation, all databases, navigation and charting tools, calculation and engineering widgets, reference material, a copy of our publicly accessible www.microship.com web server, text archives, photo libraries, video channels, and the interactive control network itself.

The control system is certainly the most amusing part of all this, as it presents live data from the low-level network while allowing direct intervention in system activity. Surfed locally through Netscape or Internet Explorer, there are tools for everything: setting up communication events, collecting and graphically presenting environmental data, monitoring or overriding the thruster management software, controlling the video turret, setting up custom security or watch functions, programming the autopilot, or whatever. All this takes place via any web browser on the ethernet... and since the two boats and two backpacks are all linked via high-speed wireless networks, this means that all resources are available all the time (even from thousands of miles away via password-protected cellular or satellite link).

In addition to presenting all this in a single smooth environment, Mimsy busies itself with collecting both internal and environmental data and sending telemetry blocks via email to our land-based server on a scheduled basis. It continuously monitors the health of all low-level systems and restarts crashed nodes, reporting such events to me via the console display but otherwise relieving me of the need to keep track of gritty internal details. It archives everything in a sprawling database, yielding an annotated breadcrumb trail of the entire adventure with attached photos, time- and location-stamps, and full system telemetry. Mimsy is the central tool that is used to manage the whole project (and, as such, our lives).

Of course, this system can’t quite handle everything – certain functions must exist in parallel with Mimsy and its subordinates. Navigation and charting tools, in particular, require a certain degree of autonomy, and after much agonizing we decided to install a separate nav PC in the form of a wearable computer (the heads-up display is particularly well suited to navigating in the fog, especially if a head-mounted compass is used to re-orient the display to overlay a computer-generated wireframe model of reality onto the soup). Since so many excellent off-the-shelf nautical applications are PC-based, this is also the machine that handles live weather satellite images, phased-array sonar displays, tide and current info, and so on.

In addition to those system displays, the console hosts a suite of dedicated instruments: marine VHF, electronic compass, standalone GPS, battery monitor, and so on. As much as possible, other display-intensive subsystems are integrated completely into Mimsy and the Nav PC. After all, this is still a canoe... we’re hurting for space!

Finally, the user interface to all this is of critical interest, since we wish to avoid the clutter of hard-to-seal and uneditable dedicated switches and controls. The VHF, compass, navlights, and GPS are standalone for good reason, since if all the fancy stuff crashes we still need those basics to make landfall safely, but for the most part everything else is integrated into the high-level system – controlled by a chord keyboard on the right-hand hydraulic rudder control, a floating sealed QWERTY unit that tucks behind the daggerboard trunk, and a sealed torque-sensing pointing device for mouse functions. The left rudder control is integrated with a throttle for the electric thruster as well as push-to-talk for the currently selected radio, and a small switch panel on the left “wing” of the console controls navlights and other hardwired basics.