I feel I ought to contribute here but if I were going to build my ultimate 500 Dominator engine for NZ classic racing the first thing I would do is a lot of research and thinking. So I cannot say what I think right now. Plus I will probabely buy the software metioned by Snotzo to try a few options out.
Much could also be learnt from the best 250s of the past. Aermacchi, NSU, Moto Guzzi, Mondial, Morini etc. There are several good books on classsic motorcycle engines which deserve careful study.
So I dont really know what I would build but there are a few practical issues that come to mind. Unless you are really rich you will have to use major components that are already available.
Obtaining a head. You would probably go the Fullauto route and perhaps try to get one which could be machined and ported for a 500. Some authors would suggest the standard Dommie needs a larger diameter inlet valve and a small diameter exhaust valve. Its unlikely to be the same as a Commando.
My ideas about rpm (bore and stroke) would be constrained around how well you could make a two valve engine breath at high rpm and how to control the valve train. These are probably your limiting factors. With the right pistons rods etc they shouldnt be a limiting factor. Snotz has already pointed out a standard Dommie can go to 8500 using the old 4000 ft/min rule of thumb My Dommie has a one piece Nourish crank and has reved to 8500 as recorded by the onboard video - but I know from the dyno power and torque is dropping at that point. Breathing could be simulated on the computer but simulating a valve train is only possible on very expensive software. As this point you might be best to steal ideas from those who have gone before 
Beehive valve springs, diferent followers etc. Jim Schmidt would be the sort of man to talk to !
I would try to build an engine that minimised vibration and was very strong and reliable. An ES2 ws mentioned above. There is a ES2 in NZ that beats most Manx Nortons. It won races in the US this year. But even the owners would recognise that it is fragile. It is much much easier to make a machine orignally designed for racing - Manx, G50, KTT etc go faster for longer than the very best modified or developed production engine. But the twin does have an advantage. Its a four valve engine by design. Only a couple of Manx Nortons can match that
