Order of device suspend/resume

[Justin Hibbits]

On Dec 15, 2016, at 3:38 PM, John Baldwin wrote:

> On Thursday, December 15, 2016 11:40:33 AM Roger Pau Monné wrote:
>> Hello,
>>
>> I'm currently dealing with a bug in the Xen suspend/resume  
>> sequence, and I've
>> found that lacking a way to order device priority during suspend/ 
>> resume is
>> proving quite harmful for Xen (and maybe other systems too). The  
>> current
>> suspend/resume code simply scans the root bus, and suspends/resumes  
>> every device
>> based on the order they are attached to their parents. The problem  
>> here is that
>> there's no way to tell that some devices should be resumed before  
>> others, for
>> example the event timers/time counters/uarts should definitely be  
>> resume before
>> other devices, but that's seems to happens mostly out of chance.
>>
>> Currently most time related devices are attached directly to the  
>> nexus, which
>> means they will get resumed first, but for example the uart is  
>> currently
>> attached to the pci bus IIRC, which means it gets resumed quite  
>> late. On Xen
>> systems, this is even worse. The Xen PV bus (that contains all Xen- 
>> related
>> devices) is attached the last one (because it tends to pick up  
>> unused memory
>> regions for it's own usage) and this bus also contains the PV  
>> timecounter which
>> should be resumed _before_ other devices, or else timecounting will  
>> be
>> completely screwed and things can get stuck in indefinitely long  
>> loops (due to
>> the fact that the timecounter is implemented based on the uptime of  
>> the host,
>> and that changes from host-to-host).
>>
>> In order to solve this I could add a hack to the Xen resume process  
>> (which is
>> already different from the ACPI one), but this looks gross. I could  
>> also attach
>> the Xen PV timer to the nexus directly (as it was done before), but  
>> I also
>> prefer to keep all Xen-related devices in the same bus for  
>> coherency. Last
>> option would be to add some kind of suspend/resume priorities to  
>> the devices,
>> and do more than one suspend/resume pass. This is more complex and  
>> requires more
>> changes, so I would like to know if it would be helpful for other  
>> systems, or if
>> someone has already attempted to do it.
>
> I think Justin Hibbits had some patches to make use of the boot-time  
> new-bus
> passes for suspend and resume which I think would help with this.   
> You suspend
> things in the reverse order of boot and resume operates in the same  
> order as
> boot.
>
> -- 
> John Baldwin

John is right.  I have a (somewhat abandoned due to time and focus)  
branch, https://svnweb.freebsd.org/base/projects/pmac_pmu/ which has  
the necessary code working mostly on PowerPC.  The diff can be found  
at https://reviews.freebsd.org/D203 too.

- Justin