CloudPendulum — Benchmarking Interface

Benchmarking

Here is a quick introduction on how to use the benchmarking interface for the Cloud Pendulum system. The benchmarking interface is used during competition so that we have a unified way of running your controllers using the same control loop. This is necessary in order for us to judge your submissions fairly.

Controller Base

The only difference you have to make to your controller in order to conform to the benchmarking interface is that your controller should implement the following abstract class:

from dataclasses import dataclass
from abc import ABC, abstractmethod
import struct

@dataclass
class MotorState:
    position: float
    velocity: float

class ControllerBase(ABC):
    
    @abstractmethod
    def __init__(self):
        pass

    @abstractmethod
    def get_control_output(
        self,
        state: list[MotorState],
    ) -> list[float]:
        pass

This class is available in the cloudpendulumclient package and the documentation can be found here.

You can use the __init__ method to initialize your controller before running the control loop.

The get_control_output method receives the state of the system from the control loop in the form of a list of MotorStates, each corresponding to an actuator in the cell running the experiment. For example, if you were running a DoublePendulum experiment the state parameter would contain two elements; one for the shoulder, and one for the elbow actuator.

The result of the get_control_output method should be a list of torques which should be applied to the actuators. For a DoublePendulum experiment this list should contain two elements, but the length of this list does not necessarily have to match the length of the state parameter. For example, the Pendbot and Acrobot experiments expects only one torque to be returned from this function.

Control loop

When developing your controller you can of course define your control loop however you want, but the cloudpendulumclient package comes with a basic control loop to use when benchmarking. The documentation can be found here.

Evaluator

Every competition uses some evaluation metric for determining the performance of your controller when benchmarking. This evaluation procedure is defined in a class derived from the Evaluator base class, which is documented here.

One such evaluator is the GoalHeightEvaluator (documentation) which evaluates the controller by counting the time that the pendulum is above some height threshold.

Example

Below is a simple example showing how to define a controller, and how to benchmark it using the ControlLoop and GoalHeightEvaluator.

A more complete example can be found in this Github repository.

from cloudpendulumclient.benchmark.controller import ControllerBase, MotorState
from cloudpendulumclient.benchmark.control_loop import ControlLoop
from cloudpendulumclient.benchmark.evaluators.goal_height_evaluator import GoalHeightEvaluator

class Controller(ControllerBase):

    def __init__(self):
        pass

    def get_control_output(
        self,
        state: list[MotorState],
    ) -> list[float]:
        return [0.0]

if __name__ == "__main__":
    controller = Controller()
    evaluator = GoalHeightEvaluator()
    control_loop = ControlLoop(
        user_token = "USER TOKEN",
        experiment_time = 20.0,
        experiment_type = "Pendubot",
    )
    control_loop.start()

    while not control_loop.finished():
        state = control_loop.get_state()
        control_loop_time = control_loop.time()

        controller_output = controller.get_control_output(state)
        evaluator.evaluate(state, controller_output, control_loop_time)

        control_loop.step(controller_output)

    video_url, logs = control_loop.stop()

    score = evaluator.get_score()
    print("Score:", score)