... what would be the best thing to do in that circumstance I was in?
Okay, from looking at your pictures, I see a couple of problems:
1. You only have one liquid fuel tank left. Ideally, you want about 2 in order to decelerate, land and then take off again and make it back to Kerbin. But that all depends on -
2. The engine you have looks like one of the more underpowered ones, that doesn't provide a good thrust/fuel consumption ratio.
The thing with KSP is that because you build your own space craft, yours will have a different weight and power profile than mine. The trick is to experiment with different numbers, types and configurations of stages so that you can get a craft into orbit that is powerful enough to do what you need it to. Unless you're using mods, the torrus (i think that's what it's called) or one of the mid-range liquid rockets usually has enough 'umph' to do the job, but the hard part is getting all that equipment into orbit in the first place.
Also, be sure you have landing gear equipped that extends below the base of your descent engine, so you don't break it off or blow it up on touchdown. If you're anything like me, you'll want to use it again to take back off. (fun fact: the apollo program lunar lander descent stage stayed on the moon, and only the top half of the lander took back off to rendezvous with the command module in lunar orbit. I haven't been able to do anything similar in KSP, because unless you burn all of the fuel in the descent stage that you plan on leaving behind first, the engine exhaust from the ascent stage causes it to explode, killing everyone.)
With regard to burning prograde and retrograde only, the exception to that rule is when doing mid-course corrections. Depending on what exactly is 'off' about your approach, short bursts at other directions, either close to prograde or retrograde, or closer to perpendicular to your direction of movement can be very effective, and help save on fuel. You will need to experiment and intuit a bit to figure out where the third axis needs to be - to do this, pick a direction and press 't' to hold your craft there, (i recommend not using rcs thrusters, because the autopilot burns through those way too quickly. instead, leave it on atmospheric control, and it will still do a relatively good job of holding your attitude - a glitch, to be sure, since there is no air in space and this should not technically work, but i digress...) then, add a very small amount of thrust and go to the map view and see how your burn is affecting your orbit. Switch back and forth between the map and main views, changing your orientation each time until you find the direction that seems to be giving you the best results. Then you can increase thrust a bit, comfortable knowing you aren't wasting fuel.
Let me know if you have any questions, and once you get a good lunar orbit we can focus on actually landing the thing. The idea is to start with a lunar orbit then lower it until it shows you will intercept the planet (although not at too high a speed), then as you get closer to touchdown, you burn retrograde to decrease both your horizontal speed and your rate of descent, slowly 'pitching over' to follow the retrograde indicator, which you want to be in the center of the sky on your gimball when you are 30ish feet from the ground, and at that point have zero horizontal acceleration and zero vertical acceleration (ie, you're not moving at all). Then you decrease thrust to softly touch down. I'm oversimplifying it, but learning to fly by your instruments will help immensely, and you will start to notice that you can 'push' the retrograde indicator across the gimball by firing in a direction a little off from the retrograde indicator, but in a direction 180 degrees opposite from the direction you want it to go. This is how you 'push' the retrograde indicator to the center of the sky on your descent burn, an indication that corresponds with decreasing speed, while at the same time increasing and decreasing thrust to slow your descent. Practice will allow you to perfect the technique. The first time I zeroed out my horizontal speed I was 3,000 feet high, so there's a lot to be said for trial and error.
As for reading, I believe there are some good tutorials somewhere floating around for both KSP and Orbiter that explain all of the above much better than I can. Da google, I guess.