The big problem with miniaturisation remains power, both the supply and the dissipation. Smaller processor feature size helps, but there are physical limits there (which you hit long before you hit atomic scales, which are about 0.1nm). So can we reach that type of processing power and size combination by just extrapolating our current technology down to the physical limits what can be fabricated? I'm honestly not sure. It's not even a matter of how long you want it to be able to run for, if you wear it round your neck it may well be how hot you can tolerate it getting that's the limit. Maybe there will be breakthroughs in processor architecture and battery technology which will render this post moot, and I hope so, but by definition that's unknowable.
Of course with a personal area network (remember that term? Very fashionable about 6 years ago, gone very quiet lately) you could carry a bulkier processor/power/WAN unit on your person (belt, handbag, pocket) and interact with it via wearable peripherals. It seems to me that the interfaces are a bigger problem there than the unit itself: glasses for the display, some faster/more precise/less intrusive input than speech recognition, a way of interacting with elements on the display, all of it needing power, needing to work together in a convenient, natural and non-intrusive way. The biggest impediment to all of this might be the smartphone itself: you need the result to be more convenient or offer significant advantages over what we already have for a wide range of uses. Half-baked prototypes like Google Glass may have set this type of thing back rather than forward in terms of public acceptability.
That said, if I had to guess I'd say that if it happens the first step in this direction will be a "smart glasses" peripheral, with the smartphone acting as the hub. That is, if the manufacturers can convince enough people that an AR interface is useful enough for enough people to buy it, and if it can be made acceptable socially (which is an intersting problem: the camera is obviously a selling point, and has real uses as well as just social media narcissism, and would be necessary for AR functionality, but it was also one of the biggest barriers to acceptance). If you can get that accepted and widespread then perhaps other interfaces, and less need for the phone itself to be a user interface, can follow. But if you are going to rely on speech they had better develop a sub-vocal interface, because widespread use of speech interfaces would render any public space unbearable.
(I'm not mentioning quantum computing because that remains a lab demonstration even after decades of study, and the states that quantum computers rely on are very delicate. I think it will be a long time before we see that stuff in portable devices).
