Switching surgical instruments during surgery is both necessary and common. Depending on the type of procedure, up to four instruments may be exchanged on a single arm in rapid succession to perform a task and these cycles occur multiple times over a given procedure. These exchanges have been demonstrated to contribute to 10 to 30% of total operative time, increasing patient exposure to anesthesia. Additionally, each instrument is only permitted to be used for 10 operations regardless of the operation length due to potential instrument degradation and even within this permitted-use window instruments frequently fail. Moreover, between patients, instruments must undergo high pressure, high heat sterilization that further degrades the instrument. Finally, instrument collisions during a procedure are common and can alter the cabling properties of the arm, necessitating re-calibration in the case of automated surgery. Sophisticated, instrument-specific calibration techniques require many long trajectories of data, which further increases the wear on the instrument, reduces its lifespan, and can require time during or before a surgical procedure to collect data.

Using teleoperation, we collect 15 corrective trajectories on each of the 12 pegs for both the pick and place task, resulting in 180 expert transfers, and 360 corrective expert trajectories. We preprocess the images in two ways: (1) crop a 150x150 image around the center of the target peg, and (2) color-crop out all red pixels outside of a block-sized radius from the center of the peg.

The extracted action, indicated by the arrows, is a vector of length λ in the direction towards the next waypoint on the expert trajectory that is at least λ from the current waypoint. The extracted termination label, indicated by the color of the waypoint, is 1 if the distance to the final position in the trajectory is less than ν, and 0 otherwise. Each waypoint corresponds to an image, and each image receives both a corrective action label and a termination signal label.

Preprocessed images from a corrective place trajectory with labels extracted using method described above.

Uncalibrated Baseline (UNCAL)

This is a coarse open-loop policy, implemented using the default dVRK controller with no modifications. The trajectories are tracked in closed-loop with respect to the robot's odometry, but open-loop with respect to vision. Due to cable-related effects such as backlash, hysteresis, and cable tension, this method often struggles to consistently complete high precision tasks because its odometry will erroneously signal that it has reached the target position.

Calibrated Baseline (CAL)

This is a calibrated open-loop policy implemented directly from Hwang et al. by using the same source code as used by the authors. This is the current state-of-the-art method for automating peg transfer. In order to correct for backlash, hysteresis, and cable tension, the authors train a recurrent dynamics model to estimate the true position of the robot based on prior commands. Similar to the uncalibrated baseline, the robot tracks reference trajectories to target grasp and place locations in closed loop with respect to the position estimated by the recurrent model, but open loop with respect to visual inputs.

Benchmark comparing peg transfer performance of the uncalibrated baseline, calibrated baseline, and IVS across 3 different surgical instruments with unique cabling characteristics. To conduct instrument transfer evaluation, we experiment using 3 different surgical instruments. Each instrument has inconsistent cabling characteristics resulting in various success rates. We investigate whether models learned from data using one instrument can transfer to another instrument without modification. This is challenging, because different surgical robotic instruments, even of the same type, have different cabling properties due to differences in wear and tear. We hypothesize that errors in executing the corrective motion can be mitigated over time by executing additional correct motions, as long as the cumulative error is decreasing. However, the calibrated baseline uses an observer model that explicitly predicts the motion of the robot based on prior commands, which requires learning the dynamics of the specific instrument used in training which may not be sufficiently accurate on a new instrument. We observe that the IVS model trained on instrument A does not decrease in performance on different instruments, while the calibrated baselines suffer significantly on different instruments.

Benchmark analyzing efficiency of IVS on various instruments. IVS adds about 1.22 seconds per transfer. The goal is to produce higher success rates, rather than to reduce the timing. However, we find that the proposed method does not use too much extra time compared to the baselines. Due to fast frame capture and continuous servoing via RGB imaging, we are able to update the robot's velocity and check for termination 10 times per second, minimizing additions to the mean transfer time. As a result, the mean transfer time is only 1.5s slower than the uncalibrated baseline, and 0.7s slower than the calibrated baseline.