Switching between apps all willy-nilly might be a problem if the Xbox One had to quit an app every time you leave it, but thankfully that's not the case.
Here's a hypothetical: you're playing Ryse, or Forza, or whatever. You get a Skype call and answer it (with a voice command, of course). When the call is over, you forget what you were doing and start watching TV. While you're watching TV, you remember that you were stuck in the game, so you pin Internet Explorer up on one side of the screen ("Xbox, snap Internet Explorer") and look up what to do.
When you go back to the game—again, with voice commands, by saying "Xbox, go to Forza Motorsport 5" or "Xbox, go to Ryse: Son of Rome"—it will pick up literally exactly where you left off, like it was just minimized the whole time.