The stock standard answer to "May I ask you to give me a referral?" is usually, "I don't know anybody that is currently in the market."
The scope of a referral base is massive - immediate work environment, previous companies worked for, industry contacts, academic alumni networks, current community you reside in, previous neighbours, and extra-curricular clubs (sports, culture, social).
As a candidate, whether you are active or passive, providing a referral does not mean you stamped a guaranteed sense of approval onto a particular person for a specific job. It is all about opening the door for someone else for a potential opportunity in future, as well as for the contacts in their networks to receive information about available positions.
As recruiters, phrase the referral question differently. We are not actually expecting the candidate to know which of the contacts in their networks are active job seekers. However, this is the perception we create when asking for referrals and the reason why typically zero to none are given.
The benefit of referral exchange positively contributes to the giver and the receiver; connecting people with jobs and companies with future employees.