Inside RTI’s IIoT Lab: From Hardware to Advanced Build & Test Systems

“Make sure you tell them about the software systems operating the lab,” my team reminded me as I prepared to lead a tour of RTI’s Industrial IoT (IIoT) Lab during the Silicon Valley edition of the Connext Conference. While the lab’s walls are lined with servers, Raspberry Pis, and other hardware, it is the underlying software that transforms the space into a fully functional test environment. Our in‑house orchestration tools integrate tightly with RTI’s internal build, test, and support pipelines.

Before diving into details, consider the breadth of software development the lab supports. RTI Connext software runs on more than 100 processor/OS/toolchain combinations—from mainstream Linux on Intel x86 to niche real‑time operating systems on custom embedded platforms. It supports multiple networking transports and interfaces with various databases. Our library suite covers C, C++, Java, C#, Ada, Lua, Python, JavaScript, and Go, and our product portfolio spans over 20 individual offerings. To keep legacy customers happy, we ship patches back to software versions that are a decade old.

At RTI, we operate our own build and test infrastructure. While some workloads could migrate to cloud providers like AWS, Azure, or Atlassian Cloud, the bulk of our lab’s capabilities remain on‑premises. Here’s a snapshot of what we run.

The RTI IIoT lab consists of three large groups of systems:

1. The build and test cluster – running various x86/x64 virtual machines
2. The enterprise and embedded test target machines – a combination of special hardware, real‑time operating systems, and less commonly used systems
3. The scalability and performance machines – a set of powerful test machines and a large bank of Raspberry Pi boards

The Build and Test Cluster

Every day, developers and our Continuous Integration (CI) system trigger dozens of builds and test runs on our cluster. Built atop OpenStack and Ceph, the cluster virtualizes compute and storage resources. We currently use Atlassian Bamboo for CI, but are transitioning to Jenkins to take advantage of modern CI/CD tooling.

On a weekly basis, we launch a comprehensive build and regression test cycle that spans all supported architectures, including virtual machines and real‑world enterprise and embedded targets.

We employ custom scripts to automate the weekly build. These scripts initiate native and cross‑compilation, reset embedded boards, execute tests on both enterprise and embedded targets, aggregate results, and surface them on a developer‑friendly dashboard. Achieving consistent results across heterogeneous platforms demands deep expertise in porting software, understanding OS nuances, and mastering embedded automation.

The Enterprise and Embedded Test Target Machines

Certain test targets cannot be virtualized—our lab hosts AIX on IBM Power, Solaris on SPARC, and even openVMS on DEC Alpha systems. In one unique case, we repurposed a PlayStation 3 console to run Linux, offering a cost‑effective alternative to custom Cell‑processor rigs. We also run multiple macOS versions and hardened Redhawk Linux deployments from Concurrent.

The majority of our targets are embedded, many as SoCs. We support x86, PowerPC, MIPS, ARM (including Zynq and NVIDIA Tegra X2) and a wide array of real‑time operating systems—VxWorks, VxWorks 653, Integrity, LynxOS, QNX, Nucleus, FreeRTOS, real‑time Linux, DEOS, and Intime. When customers provide proprietary hardware, OS, and BSPs, our platform team integrates the stack, often being the first to validate new hardware or BSPs.

The Scalability and Performance Machines

The third group includes high‑performance x64 machines isolated for latency and throughput measurement, as well as a Raspberry Pi cluster for network topology tests. Using our custom “Polygraph” framework, we verify discovery protocols at scale, measuring CPU, memory, bandwidth, and discovery latency. Our research team built a lightweight system to deploy test applications and collect results across large clusters.

Networking

We validate middleware against diverse networking technologies—10/100 Mbps, Gigabit, 10 GbE, Wi‑Fi, InfiniBand, and even satellite links—while simulating packet loss and corruption. Our robust cable management and wireless testbed allow us to stress‑test Transport Mobility features. Remote access is facilitated via networked power switches and serial servers, enabling engineers worldwide to control every system from anywhere.

The Next Generation Build System

While our current infrastructure has served us well, we’re developing a next‑generation build system to boost productivity: faster builds, modern toolchains like CMake, Conan.io, JFrog Artifactory, Docker, and Jenkins; parallelized job execution; and simplified machine reservation. This new system will further empower our engineering and release teams.

The RTI IIoT Lab operates 24/7, 365 days a year, ensuring that our customers’ mission‑critical distributed systems remain reliable and performant. Stay tuned for more updates on the upcoming build system.