I did some testing with Automation Scripts having a Script Variable with Variable Type set to INOUT and Binding Type set to SYSPROP and it seems that INOUT doesn't work with SYSPROP (as if SYSPROP can only be readonly [IN]), even though it is allows to be configured that way.
Alternatively, I used the MBO framework (MboSet, Mbo, etc) to write code that retrieves the MAXPROPVALUE Mbo via MboSet.getMbo(0) with an appropriate WHERE clause. I was able to write code to set the PROPVALUE on the MAXPROPVALUE Mbo and saved via MboSet.Save(). Now, I've also figured out that I can use MXServer instance to force the reloading of the MAXPROP cache for the specific System Property that I've updated. I hope you find this to be helpful.