How are you making your probes? If you are cutting out bands derived from
plasmid vetors, your "gel-purified" inserts are almost certainly contaminated
with plasmid vector which might hybridize to the vector as you describe.
Perhaps you should amplify your probe DNA by PCR? Or purify DNA from an
alternate vector (a phage?) that has no homology with the library vector?