What is expected behavior on resource policy start, stop and reconfigure

[askervin]

Description

When a resource policy plugin (topology-aware or balloons) is started or reconfigured, it starts making system-wide changes in containers, CPUs and uncore, none of which are automatically "restored" after stopping the policy. For instance:

1. Policy changes container attributes:
   • CPU and memory affinities
   • cache allocations
   • memory and block device bandwidth limits and quotas
2. Policy changes CPU attributes:
   • min/max frequencies
   • C-states
   • SST CLOS associations, possibly fully overriding existing SST configuration
   • Future: modify IRQ CPU masks
3. Policy changes uncore attributes:
   • uncore frequencies

Rationale

The purpose of this issue is to gather ideas and feedback what should happen when...

1. A policy is started. Should it automatically/optionally initialize cpuidle, cpufreq, uncore frequencies, for instance?
2. A policy is reconfigured. Note: not all knobs from old configuration may be present in new configuration, leaving newly unmanaged features in (unspecified) old state rather than a (specified) "default" state.
3. A policy is stopped, possibly for a restart/update/uninstall.