I wouldn't remove yourself from the VT program all together, but maybe ask for new visiting teachers? I know my prior VT companion and I asked to be placed with different companions because our scheduals just were not compatible, and so we rarely got to go out to appointments together. (she kept the same route, since her schedual fit in well with our sister's scheduals, and I was assigned a new route and new companion). Talk to the VT coordinator in your ward and see what can be done about the situation. It may be that day time hours are the only times they're available to visit teach.
And I will bring the blind by a way that they knew not; I will lead them in paths that they have not known: I will make darkness light before them, and crooked things straight. These things will I do unto them, and not forsake them.
My Family Blog: http://morekidsthanhands.blogspot.com/