Understanding the SafeLife navigation task

[aseembits93]

(It is not a software issue, I'll delete the issue if it's not relevant.)
Dear Carroll and Peter,

I am a Masters in Robotics student at Oregon State University. I was dabbling with SafeLife for a while and had some doubts.

"A safe agent that wants to avoid side effects should strongly prefer to disrupt the robust yellow pattern rather than the fragile green pattern."
And you also mention this -> "In particular, any side effect penalty used during training must be prima facie unbiased towards side effects on cells of a particular color."

Since your side effect score looks at only green cells, how can the agent learn that it's okay to be unsafe with respect to yellow cells but not green cells. Hope you get what I mean! I've been scratching my head over this for a while now. Hope to hear from you soon.

Best Regards,

[clwainwright]

A very good question! And one we don't have a ready answer for. The idea here was to design two systems side-by-side: one that is robust to small perturbations (yellow cells) and one that is very fragile to small perturbations (green cells). The challenge is to come up with a definition of side effects that takes this robustness into account. The difference between the two ought to be fairly obvious to humans, so we would like to somehow encode our human intuitions of "making a mess of things" into our agents. A very simple side effect penalty that penalizes cell-wise differences from the starting state fails the test because it treats the natural variations in the yellow cells as the same sort of side effect as the agent-caused perturbations in the green cells.

The reason to not encode the difference between the yellow cells and the green cells explicitly into the impact penalty is that this would basically be cheating. A safe agent should be able to figure out on its own which side effects are cascading and which are not, and it shouldn't depend on some things being labeled "green" and some "yellow".

I hope that helps!

Cheers,
Carroll