Most games featuring star ship combat try to turn it into a shooter
Indeed, but it doesn't really work for providing that large ship feeling, does it? That I said that it would be merely a stylistic choice doesn't mean that it wouldn't be powerful stylistics or that it couldn't lead to interesting and unconventional approach regarding other aspects.
For example, it won't necessarily have to involve dogfights at almost melee range to remain aesthetically appealing.
As for movement, I was originally thinking of continuing to ape Star Trek and basically you just choose what planet you want to warp to, and from there any movement would actually be one dimensional (how far away from the enemy ship you are). However, the idea above about incorporating nebulae and other space "terrain" sounds fun, so my thinking has changed.
Well, while I don't really like "Sci-Fi" show nebulae (I hate with passion, actually), you'll usually have more than just two ships duking it out in the middle of nowhere (and why exactly would two starships be duking it out in the middle of nowhere to begin with?). You'll typically have some planet nearby, stations, moons, other combatants and so on.
Hell, even with just two ships you can still have stuff like drones and munitions flying around and adding spatial complexity to the battle - if, as it was said in ST, 2D thinking is already a liability in space combat, then why the fuck would you want your combat to be 1D?
You could represent this with a 3D grid. I wasn't thinking of anything like step based movement though. If a grid was used, the speed of the ship would determine how many spaces it could move. I am definitely thinking turn based because I like the tension of wondering if I have made the correct choice(s) as I hit end turn and see what happens.
But why bother with a grid?
You won't have dungeon here, combat will either be TB or slow paced, and you won't have much range limitations.
Again, why the fuck ape anything?
OTOH with "smooth" space you will be able to accommodate wider range of ranges - if you want long range combat instead of close up dogfight (remember, you don't want space shootan and you can always "put it on screen" to avoid zapping some invisible dot in the sky) you'll either have to deal with millions of grid blocks, defeating the purpose of having a grid, or grid blocks that can easily fit an entire starfleet or two, plus a planet.
With smooth space you can also adopt more or less 'realistic' mechanics more easily, from just flying in straight lines and zapping each other, to newtonian with orbits, and I believe you could push some sort of orbital navigation menu on tactical RPG crowd easier than you could teach twitchers rocket science (and RPG crowd is generally used to stuff that looks abstract - as in doesn't make sense at first glance).
There would be some other differences too - you'd generally try to fit crew to different stations depending on ship's loadout and type, and crew's proficiencies, rather than allocating equipment to the party based on their abilities and roles, so you'd have a kind of an inversion between the roles of party and gear.
This is true, and it appeals to me.
Great! So that's how I'd see it:
Your ship generally plays the role party plays in in terms of mechanics. It is divided into sections arranged in fixed 3D "formation" (we need to determine occlusion for hits, firing arcs shield/point defense coverage and so on), each having its own damage system, fixed equipment and equipment slots. Some sections (generally those that actively do something), or perhaps entire systems across sections will also have crew slots (even though the crewman in question will generally be physically on bridge, he will have his battle station linked to it). Crewmen would generally have various stats determining their abilities with different types of stations and actions.
Bridge crew would generally not be directly damaged in any way, depending on setting you could also have auxiliary crew - repair teams, boarding/security teams, fighter pilots, medbay, and stuff like that, or have the entire crew be composed of only people on bridge, with fighter drones being remotely operated or AI controlled, repairs being done by repair bots supervised by engineer and so forth - in such case you avoid division into category a and category b crew, and you may have singular crew members in harm's way sometimes (if you have to send an away team and your ship has no other crew to begin with). You can also mix approaches depending on size and type of the ship and mechanization factor.
So, for example you'll have someone on sensors station (which will be linked to sensors packs installed in appropriate slots on the ship - with different possible sensor packs and, if you have many, possibly requiring multiple sensors operators), and depending on their skills they may be excellent at planetary scans involving scientific work, but moderately good at mining scans and shitty at both planetary bombardment scans and combat scans. You'll have someone at drone station, launching and commanding drones or even remotely piloting one, you will have navigators and gunners firing all sorts of armaments. You'll have engineers overseeing system operation and repairs, defense operators overseeing point defenses, countermeasures and shields (if present), computer ops capable of decrypting communications and possibly taking over insufficiently protected enemy drones and so on. You might also have ship's computer, which, depending on power and capacity, could house software allowing for boosting particular abilities of people at given stations, provide unique functionalities, or multiplex multiple systems of given type to a single station.
You'd generally always have first person view (often zoomed in) from either your own ship, allied ship or own/allied/captured drone/probe, although you should also have tactical map (information you actually have only). TPP "vanity mode" is also allowed given that due to distances involved you wouldn't really have actual use for it (but hey, pretty screenies).
I would propose real time with variable game speed, due to the fact that combat would be really slow-paced at real speed anyway (so no real advantage TB could provide, unlike conventional blobber) and RT would be advantageous for both mechanical reasons and the fact that game's speed would vary drastically, making fixed length time-steps ill suited.