The original blog post can be viewed at: https://fudavd.github.io/multi-skill-learning/
This post is in reference to the following paper: https://rdcu.be/dO2i4 
In-depth video on the method can be found at the bottom and here: https://youtu.be/zW7QQ42Pa9s 

Locomotion is a pre-requisite for complex behaviour in mobile robots (like foraging, and scanning the environment). Being able to learn locomotion skills quickly is therefore a must when we deploy robots in unseen environments. When we lack a proper model of the world and of the robot, we need a model-free approach. In nature, Central Pattern Generating neural networks (CPGs) can coordinate movement effectively without the need for extensive learning and neural adaptation. These neural centres, located in the spinal cord, can encode different behaviours that are started when higher level commands from the brain initialize the corresponding neurons.  Drawing inspiration from this phenomenon, we propose a novel method that enables robots to learn locomotion in just 15 minutes.

In this work, 6 modular robots of various design are tasked to learn three locomotion skills from scratch with a training budget of 15 minutes in the real world. The skills are defined as follows:

• Gait (→): Maximizing planar displacement from starting position in a fixed time-interval
• Clockwise- and Anti-clockwise Rotation (⟳/⟲): Maximizing planar rotation in a fixed time-interval.

Our modular robots are either hand-designed (top row of picture below) or a product of automated design using Evolutionary Computing (bottom row).

How it works

CPGs are pairs of artificial neurons that reciprocally inhibit and excite each other to produce oscillatory behaviour, and can be described mathematically as a set of Ordinary Differential Equations as shown below. Originally, CPGs were used to describe the behaviour of groups of neuron responsible for generating locomotion patterns found in the spinal cords of several animals. In the last decade, CPGs have become increasingly more popular in robotics as a form of bio-inspired control due to their functional smoothness and cyclical behaviour.

We create a network of CPGs by coupling several oscillators with each other, based on the morphological design of a modular robot (depicted below as a blueprint). Here, each servomotor gets assigned a single CPG-oscillator that is subsequently linked to its neighbours, according to their modular-based distance as shown below. The final CPG-network is able to produce more complex signals which are correlated with neighbouring oscillators. The state-value of each $x$-neuron is sent as a position command to the corresponding robot's joints.

State-space

If we look at the state-value of each neuron in our CPG-network, as time progresses, we can find (semi-)cyclical pattens emerge. Below, we map the state-value of two neurons on a 2D-plane to show the behaviour of the CPGs in their so-called 'state-space' (i.e. the bottom right plot). Keep in mind that the actual state-space of this particular CPG-network contains 12-dimensions, but for now 2-dimensions suffice for interpretation.

Initial State Optimisation

WO changes network dynamics (i.e. the vector field in state-space) which in return re-defines the behaviour of the CPGs. ISO keeps the network dynamics the same and 'selects' the proper behaviour through its initial position. Meaning we freeze the weights of the CPGs and optimises the initial start in state-space. Thus, in ISO the vector field remains the same, but we search for optimal CPG behaviour by testing different initial states. We like to view this as the CPG-network containing a 'reservoir' of (good) behaviours (encoded by its weights) and our initial state providing an 'input signal' to our network to select one of these behaviours.

Now we load our CPG-network on the real robot, where we test our sampled initial state. During a single trial, we save the intermediate states of the network ($\mathcal{S}$). In addition, an overhead system tracks the movement of our robot in the real world to collect position and orientation data ($\mathcal{D}$). In the end, we match the real world position and orientation data with the CPG-states at each respective time-stamp ($[0,1,\dots ,N]$).

Bootstrapping and Parallelization

ISO optimisation approaches the CPG-network as a reservoir of behaviours encoded by frozen weights. This means that optimisation does not affect the network dynamics (the vector field remains the same). We can use this to our advantage with the following two tricks:

• Bootstrapping: The intermediate CPG-network states from $\mathcal{S}$ can be considered as additional possible initial state samples. This increases the amount of samples that we get $N$ times, we assess the performance of a state at each time-step instead of assessing only one. The longer a trial lasts, the more bootstrapped samples we obtain.
• Parallelization: The behavioural performance can be assessed on $\mathcal{D}$ for multiple skills at once, instead of evaluating a single skill metric. Optimization of specific skills do not interfere when the dynamics of the CPG-network are frozen. This allows information to flow between learning tasks as can extract usefull information among all skills at once.

A trade-off can be made between testing longer to obtain more bootstrapped samples, or sampling different regions in state space with more trials (see Appendix for hyper-parameter sweep of ISO). In our work, we choose to test 5 different regions for two minutes each (10 minutes in total) using random search. In the end, we select the best overall (intermediate) initial state sample for each skill (denoted as [$s_{⟲}^{\ast }$, $s_{\to }^{\ast }$, $s_{⟳}^{\ast }$]).

In the end, we re-test our best initial states [$s_{⟲}^{\ast }$, $s_{\to }^{\ast }$, $s_{⟳}^{\ast }$] to measure the performance of our ISO algorithm's final skills. The resulting behaviours are shown below.

Target Following

We consider our CPG-network as a reservoir that encodes many behaviours, which we can learn through ISO. The resulting repertoire of motor skills can be used for more complex tasks. We provide a feedback signal to our CPG-controller to smoothly switch between skills (see paper for more detail). Using, our learned skills (→,⟳,⟲) we are able to successfully complete a random target following task.