Smart Pen: Real‑Time Handwriting & Movement Tracking with Raspberry Pi and IMU

Introduction

In today’s digital era, capturing handwritten input usually requires capacitive touchscreens or stylus pens. While precise, these solutions are limited by surface requirements and lack true spatial freedom. Our Smart Pen eliminates those constraints by combining a Raspberry Pi with an inertial measurement unit (IMU) to track motion and reconstruct trajectories instantly. The system relies solely on the IMU’s gyroscope and accelerometer, transmitting data wirelessly to the Pi. Custom algorithms then recover the device’s orientation and position, store the data, and render it in real time on the Pi’s TFT display via pygame. A single push‑button starts and stops recording, enabling flexible use on pens, robots, or wearable applications. This approach delivers a versatile, surface‑agnostic solution for handwriting capture, motion analysis, and beyond.

Objective

The core aim is to develop a modular tracking unit that can be affixed to pens, robots, or even human limbs. When the host moves, the module records its trajectory and projects the path onto a horizontal world‑frame on screen. Unlike conventional stylus systems, our design operates independently of any predefined plane, allowing it to track movement in free air or on irregular surfaces. The module’s orientation‑insensitive performance ensures consistent results regardless of tilt or placement.

IMU Calibration

Manufacturing tolerances often misalign the IMU’s accelerometer and gyroscope axes, leading to discrepancies in calculated Euler angles. We employ a six‑position calibration routine to quantify and correct these offsets. A custom calibration jig (Figure XX) holds the IMU in each orientation, recording raw sensor readings. The calibration matrix derived from these six samples is then applied to every accelerometer measurement:

\[\mathbf{A}_{\text{cal}} = \begin{bmatrix}A_x\\A_y\\A_z\end{bmatrix}, \quad \mathbf{a}_{\text{raw}} = \begin{bmatrix}a_x\\a_y\\a_z\end{bmatrix}\]

After calibration, the corrected acceleration vector \(\mathbf{A}_{\text{cal}}\) replaces the raw data for subsequent processing.

Sensor Fusion for Raw Data

To obtain a robust orientation estimate, we fuse gyroscope and accelerometer outputs using quaternion mathematics. Quaternions offer singularity‑free rotation representation and simplify vector transformations. The conversion between quaternion \(q\) and Euler angles \((\alpha,\beta,\gamma)\) is given by:

\[\displaystyle q = \cos\frac{\theta}{2} + \sin\frac{\theta}{2}\bigl(\cos\alpha\,\mathbf{i} + \cos\beta\,\mathbf{j} + \cos\gamma\,\mathbf{k}\bigr)\]

For a fixed vector \(\mathbf{V} = V_x\mathbf{i} + V_y\mathbf{j} + V_z\mathbf{k}\), quaternion rotation is performed as \(\mathbf{V}' = q \circ \mathbf{V} \circ q^{-1}\).

Because the gyroscope alone drifts over time, we continually correct its bias by comparing the measured gravity vector from the accelerometer with the gyroscope‑derived gravity estimate. The UpdateIMU routine executes 2,000 iterations while the sensor is stationary, building a proportional‑integral (PI) feedback loop that minimizes angular error. The core of the algorithm is shown below:

def UpdateIMU(self, Gyr, Acc):
    if np.linalg.norm(Acc) == 0:
        warnings.warn("Accelerometer magnitude is zero.  Algorithm update aborted.")
        return
    Acc = np.array(Acc / np.linalg.norm(Acc))
    v = np.array([[2*(self.q[1]*self.q[3] - self.q[0]*self.q[2])],
                  [2*(self.q[0]*self.q[1] + self.q[2]*self.q[3])],
                  [self.q[0]**2 - self.q[1]**2 - self.q[2]**2 + self.q[3]**2]])
    error = np.cross(v, np.transpose([Acc]), axis=0)
    self.IntError += error
    Ref = Gyr - np.transpose(self.Kp*error + self.Ki*self.IntError)
    pDot = 0.5 * self.quaternProd_single(self.q, [0, Ref[0,0], Ref[0,1], Ref[0,2]])
    self.q += pDot * self.SamplePeriod
    self.q /= np.linalg.norm(self.q)
    self.Quaternion = self.quaternConj(self.q)

The corrected gyroscope data (Ref) feeds into the quaternion update, yielding a stable orientation estimate that powers the trajectory reconstruction displayed on the Pi.

Read more: Smart Pen: Final Project for ECE5725