Stale memory during post fork cow pmap update

[Elliott.Rabe at dell.com]

On 02/13/2018 07:30 AM, Konstantin Belousov wrote:
> Apparently Ryzen CPUs are able to demonstrate it quite reliably with the
> python driver for the samba build.  It was very surprising to me, esp.
> because I tried to understand the Ryzen bug for the whole last week and
> thought that it is more likely CPU store/load inconsistency than a software
> thing.  See https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=225584.
Cool, it is good to have other validations.  As for the goose-chase, 
it's totally understandable; the subtle nature of the corruptions (ie. 
stale reads) often causes the symptoms to manifest as an 
MP-caching-store/load problem.
>> I am also attaching a patch that I have been using to prevent the
>> problem.  I was looking at things with a much narrower view and made the
>> changes directly in pmap_enter.  I suspect the internal
>> double-update-invalidate is slightly better performance wise then taking
>> two whole faults, but I haven't benchmarked it, it probably doesn't
>> matter much compared to the cost and frequency of the actual copies, and
>> it also has the disadvantage of being architecture specific.  I also
>> don't feel like I have enough experience with the vm fault code in
>> general for my commentary to be very valuable here.  However, I do
>> wonder: 1) if there are any other scenarios where a potentially
>> accessible page might be undergoing an [address+writable] change in the
>> same way (this sort of thing seems hard to read out of code), and 2) if
>> there is ever any legal reason why an accessible page should be
>> undergoing such a change?  If not, perhaps we could come up with an
>> appropriate sanity-check condition to guard against any cases of this
>> sort of thing accidentally slipping in the future.
> I am not sure how to formulate such check. I believe there is now other
> place in kernel, except the COW handling, which can create similar
> situation, but I cannot guarantee it.
>
> I also believe that the problem exists for all SMP hardware, so fixing
> pmaps would create a lot of work, which is also quite hard to validate.
> Note that system generally considers the faults which do not cause disk
> io as 'easy', they are accounted as the 'soft faults'.  Sure, not causing
> the second fault for write would be more efficient than arranging for it,
> if only because we would not need to lookup and lock the map and shadow
> chain of vm objects and pages, but fork is quite costly already and code
> simplicity there is also important.
Fair enough, thanks for this perspective.  I now plan to apply just your 
patch back in the original environment and confirm stability follows.
> I put the review with the patch at https://reviews.freebsd.org/D14347.
> You have to add yourself as subsriber or reviewer.
Done, thanks again!