\documentclass[twocolumn]{article} \usepackage{times} \usepackage{alltt} \title{Porting a Network Cryptographic Service to the RMC2000: A Case Study in Embedded Software Development} \makeatletter \def\section{% \@startsection{section}{1}{\z@}{-0.47\baselineskip}% {0.47\baselineskip}{\bfseries}} \def\subsection{% \@startsection{subsection}{2}{\z@}{-0.5\baselineskip plus .5ex minus .2ex}% {0.5\baselineskip plus .2ex}{\itshape}} \makeatother \author{ \begin{tabular}{ccc} Stephen Jan & Paolo de Dios & Stephen A. Edwards \\ \multicolumn{3}{c}{Dept. of Computer Science} \\ \multicolumn{3}{c}{Columbia University, New York, NY} \\ \multicolumn{3}{c}{\{sj178, pd119\}@columbia.edu\ \ sedwards@cs.columbia.edu} \\ \end{tabular}} % For blind review \author{} \date{} \textwidth 7in \oddsidemargin -0.25in \evensidemargin -0.25in \topmargin -1in \textheight 9.5in \parindent 1em \frenchspacing \begin{document} \maketitle \begin{abstract} This paper describes our experience porting a transport-layer cryptography service to an embedded microcontroller. We describe some key development issues and techniques involved in porting networked software to a connected, limited resource device such as the Rabbit RMC2000 we chose for this case study. We examine the effectiveness of a few proposed porting strategies by examining important program and run-time characteristics. \end{abstract} \section{Introduction} Embedded systems present a different software engineering problem. These systems are unique in that the hardware and the software are tightly integrated. The limited nature of an embedded systems operating environment require a different approach to developing and porting software. In this paper, we discuss the key issues in developing and porting a Unix system level transport-level security (TLS) service to an embedded microcontroller. We discuss our design decisions and experience porting this service using Dynamic~C, a C variant, on the RMC2000 microcontroller from Rabbit Semiconductor\footnote{www.rabbitsemiconductor.com}. The main challenges came when APIs for operating-system services such as networking were either substantially different or simply absent. Porting software across platforms is such a common and varied software engineering exercise that much commercial and academic research has been dedicated to identifying pitfalls, techniques, and component analogues for it. Porting software has been addressed by high level languages~\cite{brown1972levels,kernighan1988c}, modular programming~\cite{hanson1997c}, and component based abstraction, analysis and design techniques~\cite{vinoski1997corba}. Despite the popularity of these techniques, they are of limited use when dealing with the limited and rather raw resources of a typical embedded system. In fact, these abstraction mechanisms tend to consume more resources, especially memory, making them impractical for microcontrollers. Though some have tried to migrate some of these abstractions to the world of embedded systems~\cite{gokhale1999techniques}, porting applications in a resource-constrained system still requires much reengineering. This paper presents our experiences porting a small networking service to an embedded microcontroller with an eye toward illustrating what the main problems actually are. Section~\ref{sec:crypto} introduces the network cryptographic service we ported. Section~\ref{sec:related} describes some relevant related work, and Section~\ref{sec:rmc2k} describes the target of our porting efforts, the RMC~2000 development board. Section~\ref{sec:port} describes the issues we encountered while porting the cryptographic network service to the development board, Section~\ref{sec:experiments} describes the performance experiments we conducted, and finally Section~\ref{sec:conclusions} summarizes our findings. \section{Network Cryptographic Services} \label{sec:crypto} For our case study, we ported ``iSSL,''\footnote{http://sourceforge.net/projects/issl} a public-domain implementation of the Secure Sockets Layer (SSL) protocol~\cite{freier1996ssl}, a Transport Layer Security (TLS) standard proposed by the IETF~\cite{dierks1997tls}. SSL is a protocol that layers on top of TCP/IP to provide secure communications, e.g., to encrypt web pages with sensitive information. Security, unfortunately, is not cheap. Establishing and maintaining a secure connection is a computationally-intensive task; negotiating a SSL session can signifigantly degrade server performance, potentially limiting the number of simultaneous sessions. Goldberg et al.~\cite{goldberg1998secure} observed it reducing throughput by an order of magnitude. iSSL is a cryptographic library that layers on top of the Unix sockets layer to provide secure point-to-point communications. After a normal unencrypted socket is created, the iSSL API allows a user to bind to the socket and then do secure read/writes on it. To gain experience using the library, we first implemented a simple Unix service that used the iSSL library to establish a secure redirector. Later, we ported this service to the RMC2000. Because SSL forms a transparent layer on top of TCP, it can be easily factored out of the server and placed in separate hardware. Many commercial systems use coprocessor cards that perform SSL functions to offload work from the server's main processor. Our case study implements a similar service. The iSSL package uses the RSA and AES cipher algorithms and can generate session keys and exchange public keys. Because the RSA algorithm uses a difficult-to-port ``bignum'' package, we only ported the AES cipher, which uses the Rijndael algorithm~\cite{daemen1998block}. By default, iSSL supports key lengths of 128, 192, or 256 bits and block lengths of 128, 192, and 256 bits, but to keep our implementation simple, we only implemented 128-bit keys and blocks. During porting, we also referred to the AESCrypt implementation developed by Eric Green and Randy Kaelber\footnote{http://aescrypt.sourceforge.net}. \section{Related Work} \label{sec:related} Cryptographic services for transport layer security (TLS) have long been available as operating system and application server services~\cite{engelschall2000mod}. The concept of an embedded TLS service or custom ASIC for stream ciphering are commercially available as SSL/TLS accelerator products from vendors such as Sun Microsystems and Cisco. They operate as black boxes and the development issues to make these services available to embedded devices have been rarely discussed. Though the performance of various cryptographic algorithms such as AES and DES have been examined on many systems~\cite{schneier1999performance}, including embedded devices~\cite{yang2001performance}, a discussion on the challenges of porting complete services to a device have not received such a treatment. The scope of embedded systems development has been covered in a number of books and articles~\cite{gassle1992art,gassle1997dumb}. Optimization techniques at the hardware design level and at the pre-processor and compiler level are well-researched and benchmarked topics~\cite{gassle1992art,leupers2000code,zivojnovic1995dspstone}. Embedded programming guidelines for optimization, style and robustness are outlined for specific languages such as ANSI~C~\cite{barr1999programming}. Design patterns have even been proposed to increase portability and leverage reuse among device configurations for embedded software~\cite{champlain1996patterns}. Overall, we found the issues involved in porting software to the embedded world have not been written about extensively, and are largely considered ``just engineering'' doomed to be periodically reinvented. Our hope is that this paper will help engineers be more prepared in the future. \section{The RMC2000 Environment} \label{sec:rmc2k} Typical for a small embedded system, the RMC2000 TCP/IP Development Kit includes 512k of flash RAM, 128k SRAM, and runs a 30~MHz, 8-bit Z80-based microcontroller (a Rabbit 2000). While the Rabbit~2000, like the Z80's, manipulates 16-bit addresses, it can access up to 1~MB through bank switching. It includes a 10Base-T network interface and comes with software implementing TCP/IP, UDP and ICMP. The development environment includes compilers and diagnostic tools, and the board has a 10-pin programming port to interface with the development environment. \subsection{Dynamic C} The Dynamic~C language, developed in concert with Rabbit microcontrollers, is an ANSI~C variant with many extensions to support the Rabbit~2000 in embedded system applications. For example, the language directly supports cooperative and preemptive multitasking, battery-backed variables, and atomicity guarantees for shared multibyte variables. Unlike ANSI, Dynamic~C's function-local variables are \texttt{static} by default. While this can be overriden with a directive, ignoring it can dramatically change program behavior. Dynamic~C does not support the \texttt{\#include} directive, using instead \texttt{\#use}, which gathers precompiled function prototypes from libraries. It took some effort to decide which \texttt{\#use} directives should replace the many \texttt{\#include} directives in the source files. Dynamic~C omits and modifies some ANSI~C behavior. Bit fields and enumerated types are not supported. There are also minor differences in the \texttt{extern} and \texttt{register} keywords. As mentioned earlier, the default storage class for variables is \texttt{static}, not \texttt{auto}, which can dramatically change the behavior of recursively-called functions. Variables initialized in a declaration are stored in flash memory and cannot be changed. Dynamic~C's support for inline assembly is more comprehensive than most~C implementations, and it can also integrate~C into assembly code, as in the following: \begin{alltt}\small #asm nodebug InitValues:: ld hl,0xa0; c start_time = 0; \texttt{// Inline C} c counter = 256; \texttt{// Inline C} ret #endasm \end{alltt} We used the inline assembly feature in the error handling routines that caught exceptions thrown by the hardware or libraries, such as divide-by-zero. We could not rely on an operating system to handle these errors, so instead we specified an error handler using the \texttt{defineErrorHandler(void *errfcn)} system call. Whenever the system encounters an error, the hardware passes information about the source and type of error on the stack and calls this user-defined error handler. In our implementation, we used (simple) inline assembly statements to retrieve this information. Because our application was not designed for high reliability, we simply ignored most errors. \subsection{Multitasking in Dynamic C} Dynamic~C provides both cooperative multitasking, through costatements and cofunctions, and preemptive multitasking through either the \texttt{slice} statement or a port of Labrosse's $\mu$C/OS-II real-time operating system~\cite{labrosse1998microc-os-ii}. Dynamic~C's costatements provide multiple threads of control through independent program counters that may be switched among explicitly, such as in this example: \begin{alltt}\small for (;;) \{ costate \{ waitfor( tcp_packet_port_21() ); \textrm{// handle FTP connection} yield(); \textrm{// Force context switch} \} costate \{ waitfor( tcp_packet_port_23() ); \textrm{// handle telnet connection} \} \} \end{alltt} The \texttt{yield} statement immediately passes control to another costatement. When control returns to the costatement that has yielded, it resumes at the statement following the \texttt{yield}. The statement \texttt{waitfor(\emph{expr})}, which provides a convenient mechanism for waiting for a condition to hold, is equivalent to \texttt{while (!\emph{expr}) yield();}. Cofunctions are similar, but also take arguments and may return a result. In our port, we used costatements to handle multiple connections with multiple processes. We did not use $\mu$C/OS-II. \subsection{Storage Class Specifiers} To avoid certain race conditions, Dynamic~C generates code that disables interrupts while multibyte variables marked \texttt{shared} are being changed, guaranteeing atomic updates. For variables marked \texttt{protected}, Dynamic~C generates extra code that copies their value to battery-backed RAM before every modification. Backup values are copied to main memory when when system is restarted or when \texttt{\_sysIsSoftReset()} is called. We did not need this feature in this port. The Rabbit 2000 microcontroller has a 64K address space but uses bank-switching to access 1M of total memory. The lower 49.5K is fixed, ``root'' memory, and the top 8K is bank-switched access to the remaining memory. A user can explicitly request a function to be located in either root or extended memory using the storage class specifiers \texttt{root} and \texttt{xmem} (Figure~\ref{fig:specifiers}). We had to explicitly request certain functions, such the error handler, to be located in root memory, but we let the compiler place all the other functions. \begin{figure} \begin{alltt}\small \textrm{// Interrupts disabled during changes to a, b, and c} \textrm{// Updates guaranteed atomic} shared float a, b, c; main() \{ protected int state1; \textrm{// Battery-backed} ... \textrm{// restore protected variables} _sysIfSoftReset() \} \textrm{// place func1 in root memory} root int func1() \{ ... \} \textrm{// Place following assembly code in root memory} #memmap root #asm root ... #endasm \textrm{// place func2 in extended memory} xmem int func2() \{ ... \} \end{alltt} \caption{Fragment illustrating various Dynamic-C-specific storage class specifiers.} \label{fig:specifiers} \end{figure} \subsection{Function Chaining} Dynamic C provides function chaining, which allows segments of code to be embedded within one or more functions. Invoking a named function chain causes all the segments belonging to that chain to execute. Such chains enable initialization, data recovery, or other kinds of tasks on request. Our port did not use this feature. \begin{alltt}\small \textrm{// Create a chain named ``recover'' and add three functions} #makechain recover #funcchain recover free_memory #funcchain recover declare_memory #funcchain recover initialize \textrm{// Invoke all three functions in the chain in some sequence} recover(); \end{alltt} % Nice tutorial at http://www.ecst.csuchico.edu/~beej/guide/net/html/ \begin{figure*} \begin{tabular}{@{}c@{}c@{}} \begin{minipage}[t]{0.5\textwidth} \scriptsize \begin{verbatim} int echo_server() { int sock, newsock, len; struct sockaddr_in addr; char buf[LEN]; if ((sock = socket(AF_INET, SOCK_STREAM, 0)) < 0) return -1; memset(&addr, 0, sizeof(addr)); addr.sin_family = AF_INET; addr.sin_addr.s_addr = htonl(INADDR_ANY); addr.sin_port = htons(MYPORT); if ( bind(sock, (struct sockaddr *) &addr, sizeof(struct sockaddr_in)) < 0 ) return -1; if ( listen(sock, LISTENQ) < 0 ) return -1; for (;;) { if ((newsock = accept(sock, NULL, NULL) ) < 0 ) return -1; if ((len = recv(newsock, buf, LEN, 0)) < 0) return -1; if (send(newsock, buf, len, 0) < 0) return -1; close(conn_s); } } \end{verbatim} \end{minipage} & \begin{minipage}[t]{0.5\textwidth} \scriptsize \begin{verbatim} int echo_server() { tcp_Socket sock; int status; char buf[LEN]; sock_init(); for (;;) { tcp_listen(&sock, PORT, 0, 0, NULL, 0); sock_wait_established(&sock, 0, NULL, &status); sock_mode(&sock, TCP_MODE_ASCII); while (tcp_tick(&sock)) { sock_wait_input(&sock, 0, NULL, &status); if (sock_gets(&sock, buf, LEN)) sock_puts(&sock, buf); } } } \end{verbatim} \end{minipage} \\ (a) & (b)\\ \end{tabular} \caption{A comparison of (a) traditional BSD sockets-based code and (b) equivalent code in the Dynamic C environment illustrating the significant differences in API.} \label{fig:sockets} \end{figure*} \section{Porting and Development Issues} \label{sec:port} A program rarely runs unchanged on a dramatically different platform; something always has to change. The fundamental question is, then, how much must be changed or rewritten, and how difficult these rewrites will be. We encountered three broad classes of porting problems that demanded code rewrites. The first, and most common, was the absence of certain libraries and operating system facilities. This ranged from fairly simple (e.g., Dynamic~C does not provide the standard \texttt{random} function), to fairly difficult (e.g., the protocols include timeouts, but Dynamic~C does not have a timer), to virtually impossible (e.g., the iSSL library makes some use of a filesystem, something not provided by the RMC2000 environment). Our solutions to these ranged from creating a new implementation of the library function (e.g., writing a \texttt{random} function) to working around the problem (e.g., changing the program logic so it no longer read a hash value from a file) to abandoning functionality altogether (e.g., our final port did not implement the RSA cipher because it relied on a fairly complex ``bignum'' library that we considered to complicated to rework). A second class of problem stemmed from differing APIs with similar functionality. For example, the protocol for accessing the RMC2000's TCP/IP stack differs quite a bit from the BSD sockets used within iSSL. Figure~\ref{fig:sockets} illustrates some of these differences. While solving such problems is generally much easier than, say, porting a whole library, reworking the code is tedious. A third class of problem required the most thought. Often, fundamental assumptions made in code designed to run on workstations or servers, such as the existence of a filesystem with nearly unlimited capacity (e.g., for keeping a log), are impractical in an embedded systems. Logging and somewhat sloppy memory management that assumes the program will be restarted occasionally to cure memory leaks are examples of this. The solutions to such problems are either to remove the offending functionality at the expense of features (e.g., remove logging altogether), or a serious reworking of the code (e.g., to make logging write to a circular buffer rather than a file). \subsection{Interrupts} We used the serial port on the RMC2000 board for debugging. We configured the serial interface to interrupt the processor when a character arrived and then either prompt the system to send a status message back out the port, reset the application, or reset the application maintaining program state. A Unix operating environment provides a high-level mechanism for handling software interrupts: \begingroup\small \begin{alltt} main() \{ signal(SIGINT, sigproc); \textrm{// Register signal handler} \} void sigproc() \{ \textrm{/* Handle the signal */} \} \end{alltt} \endgroup By contrast in Dynamic~C, we had to handle all the details ourselves. For example, to set up the interrupt from the serial port, we had to enable the serial port to generate interrupts, register the interrupt routine in the vector table, and finally enable the interrupt receiver. \begin{alltt}\small main() \{ \textrm{// Set serial port A as input interrupt} \textrm{// SADRShadow is pointer to value at register} WrPortI(SADR, &SADRShadow, 0x00); \textrm{// Register interrupt service routine} SetVectExtern2000(1, my_isr); \textrm{// Enable external INT0 on SA4, rising edge} WrPortI(I0CR, NULL, 0x2B); \textrm{/* Main program */} \textrm{// Disable interrupt 0} WrPortI(I0CR, NULL, 0x00); \} nodebug root interrupt void my_isr() \{ \textrm{/* handle interrupt */} \} \end{alltt} We could have avoided the use of interrupts if we designed the application to open another ``debugging'' network connection, but this would have made it impossible to debug a system that had a networking-related bug. \subsection{Memory} A significant difference between general platform development and embedded system development is memory. Most embedded devices have very little memory in comparison to a full fledged computer. Expecting to run into memory issues, we used a well-defined taxonomy~\cite{zurell2000c} to plan out memory requirements. This turned out to be unnecessary, however, because out application had very modest memory requirements. Dynamic~C does not support the standard library functions \texttt{malloc} and \texttt{free}. Instead, it provides the \texttt{xalloc} function that allocates extended memory only (arithmetic, therefore, cannot be performed on the returned pointer). More seriously, there is no analogue to \texttt{free}; allocated memory cannot be returned to a pool. Instead of implementing our own memory management system (which would have been awkward given the Rabbit's bank-switched memory map), we chose to remove all references to \texttt{malloc} and statically allocate all variables. This prompted us to drop support of the variable key and block sizes in the iSSL library. \subsection{Program Structure} As we often found during the porting process, the original implementation made use of high-level operating system functions such as \texttt{fork} that were not provided by the rudimentary environment on the RMC2000. This forced us to restructure the program significantly. The original TLS implementation handles an arbitrary number of connections using the typical BSD sockets approach shown below. It first calls \texttt{listen} to begin listening for incoming connections, then calls \texttt{accept} to wait for a new incoming connection. Each request returns a new file descriptor passed to a newly-\texttt{fork}ed process that handles the request. Meanwhile, the main loop immediately calls \texttt{accept} to get the next request. \begin{alltt}\small listen(listen_fd) for (;;) \{ accept_fd = accept(listen_fd); if ((childpid = fork()) == 0) \{ \textrm{// process request on accept\_fd} exit(0); \textrm{// terminate process} \} \} \end{alltt} Unfortunately, the Dynamic~C environment provides neither the standard Unix \texttt{fork} nor an equivalent of \texttt{accept}. In the RMC~2000's TCP implementation, the socket bound to the port also handles the request, so each connection is required to have a corresponding call to \texttt{tcp\_listen}. Furthermore, although Dynamic~C provides concurrent processes in the form of costatements, their number is fixed at program compile-time. Thus, to handle multiple connections and processes, we split the application into four processes: three processes to handle requests (allowing a maximum of three connections), and one to drive the TCP stack (Figure~\ref{fig:requests}). We could easily increase the number of processes (and hence simultaneous connections) by adding more costatements, but the program would have to be re-compiled. \begin{figure} \begin{alltt}\scriptsize for (;;) \{ costate \{ tcp_listen(socket1,TLS_PORT, ...); while (sock_established(socket1) == 0) yield; \textrm{// handle request} \} costate \{ tcp_listen(socket2,TLS_PORT, ...); while((0 == sock_established(socket2))) yield; \textrm{// handle request} \} costate \{ tcp_listen(socket2,TLS_PORT, ...); while((0 == sock_established(socket2))) yield; \textrm{// handle request} \} costate \{ \textrm{// drive TCP stack} tcp_tick(NULL); \} \} \end{alltt} \caption{The structure of the main loop of the TLS server, which can handle at most three requests because it is limited to four processes.} \label{fig:requests} \end{figure} \section{Experimental Results} \label{sec:experiments} To estimate which optimization techniques were worthwhile, we compared the~C implementation of the AES algorithm (Rijndael) included with the iSSL library with a hand-coded assembly version supplied by Rabbit Semiconductor. A testbench that pumped keys through the two implementations of the AES cipher showed that the handcrafted assembly implementation ran faster than the~C port by a factor of 15--20. We tried a variety of optimizations on the~C code (e.g., moved data to root memory, unrolled some loops, turned off debugging, enabled compiler optimization), but this only improved its run time by perhaps 20\%. Code size appeared uncorrelated to execution speed. The assembly implementation was 9\% smaller than the~C, but ran more than an order of magnitude faster. Debugging and testing consumed the majority of the development time. Many of these problems came from our lack of experience with Dynamic~C and the RMC2000 platform, but unexpected, undocumented, or simply contradictory behavior of the hardware or software and its specifications also presented challenges. \section{Conclusions} \label{sec:conclusions} We described our experiences porting a library and server for transport-level security protocol (specifically iSSL) onto a small embedded development board (the RMC 2000, based on the Z80-inspired Rabbit 2000 microcontroller). While the Dynamic~C development environment supplied with the board gave useful, necessary support for some hardware idiosyncrasies (e.g., its bank-switched memory architecture), support for concurrent programming (through language-level support for cooperative multitasking in the form of costatements and cofunctions), and a TCP/IP stack, porting the service, originally designed for a Unix-like environment, was not trivial. The biggest challenges revolved around different or missing APIs, such as the substantial difference between BSD-like sockets and provided TCP/IP implementation or the simple absence of a filesystem. Our solutions to these problems involved either writing substantial amounts of additional code to implement the missing library functions or reworking the original code to use (or simply avoid) the API. Finally, we compared the speed of our direct port of a C implementation of the RSA (Rijndael) ciper with a hand-optimized assembly version and found a disturbing factor of 15--20 in performance in favor of the assembly. From all of this, we conclude that there must be a better way. Understanding and dealing with differences in operating environment (effectively, the API) is a tedious, error-prone task that should be automated, yet we know of no work beyond high-level language compilers that confront this problem directly. % Applies to the next page \advance\textheight -5.25in \bibliographystyle{plain} \bibliography{date2003} % Applies to the current page \advance\footskip 5.25in \end{document}