Doctors told the BBC that the way surgeries are funded favours practices in towns and cities – and the situation is getting worse.
They said it was a struggle to recruit family doctors in remote areas.
The claim comes as the Royal College of GPs (RCGP) said more than one in 10 family doctor roles in England are vacant – with many practices having to rely on locum doctors.
An RCGP survey of 549 practices in England found that 10.2% of full time equivalent positions are vacant – 61% of which are currently filled by locum or agency staff.
Of the practices surveyed, 64% reported that finding enough locum doctors to meet growing patient demand was either difficult or very difficult.
Just 8% reported that this was easy.
The RCGP said its findings cast doubt on the feasibility of government plans to introduce a seven-day working week in general practice.
The college is calling on the government and NHS England to concentrate its efforts on strengthening the GP workforce to ensure that the current five-day service is robust.
Currently, only 45% of patients know how to access out of hours GP services, according to the latest GP Patient Survey, conducted by Ipsos Mori on behalf of NHS England.
The college is also calling for steps to be taken to promote practice-based working as a secure and satisfying career for doctors entering the profession.
RCGP chair Maureen Baker said: "We are in dire straits if we do not act to address the GP recruitment crisis immediately."
Dr Baker said it was vital that there were enough GPs in the system so that practices did not have to run with a substantial number of vacant positions.
"Our new research brings home how difficult GP practices are finding it to recruit new doctors and retain existing ones.
"It is simply unrealistic to be thinking about seven-day working when our existing five-day service and out of hours GP services are under so much pressure.
"Many GP services are already offering extended hours.
"However, opening for extended periods is pie-in-the-sky for many family doctors who are already working exceptionally long hours in clinic to cope with demand."
The government has pledged to create 5,000 more GPs over the course of this parliament.
But the RCGP estimates that around 3,300 additional GPs are needed now – just to meet current demand.
This doesn't take into account growing patient demand over the next five years, and those GPs who will leave the profession due to retirement.
The RCGP believes that England will need 8,000 additional GPs by 2020 to provide existing five day and OOH services for the changing needs of the population.
The college's research was based on an online survey with more than 1,200 of its members, during August, which were representative of 549 practices in England.