You just have to continue preaching the Word in and out of season as the scripture says. I know how you feel, b/c when I first go saved, it seemed like the people whom I was drawn by the Holy Spirit to minister to were either agnostic or atheists. And at the time, it seemed really tough, b/c they would always want to draw me into some philosophical debate, and that actually drew me away from what I was trying to present, which was the Word of God. So when people try to bring you into vain babblings, you need to re-direct to the Word of God and realize that though a conversion might not take place that day, realize that you have either planted a seed or watered it, so that God can get the increase (1 Cor 3:7).
Another key point as you mentioned would be prayer, and ask the Holy Spirit for guidance as to how to proceed in ministering and prayer. One thing I learned from an elder at my old church was that we in the body needed to pray effectively. He used an example of how he was talking to a man about Christ at his apt. complex, and asked the man could he pray for him. The man said yes, and he proceed to pray something along the lines of this, "Lord, do whatever you need to do to get this young man to the throne of grace where he can give his heart to you, do it, whether it be using his money, family, etc. to do so." The man immediately responded that isn't the type of prayer he wanted. But guess what, it was exactly what he needed.
Also, remember that your walk in Christ is a huge testament in itself. There is always someone watching, and even though the person you could be personally ministering to you doesn't seem to be hearing you, there is a good chance that someone who is maybe 10ft away is listening and watching.
