Factionalism was a greater threat to Labor's future than links to unions, an ALP powerbroker said last night as he called for rank-and-file supporters to regain control of the party.

Wayne Swan, a long-time factional operator and the opposition's family services spokesman, reignited debate about Labor's structure by suggesting it take the radical step of adopting American-style primary voting.

As Labor figures Bob Hawke and Neville Wran begin an audit of Labor's rules and relationship with voters, Mr Swan called for ALP supporters to have the power to choose election candidates and to vote for the party's national president.

The plan to restrict the power of factions would mean ordinary voters who were not Labor members could be able to vote in preselection battles that in recent years have sometimes become embroiled in allegations of branch-stacking and back-room factional deals.

Labor has around 50,000 members. But under the proposed overhaul of party rules, the 4.3 million people who cast their primary vote for Labor in the last election could be eligible to participate.

Mr Swan, who was a key strategist advising former Labor leader Kim Beazley, said Labor could adopt the United States' primary voting system "with some unique Australian features" in a speech last night to the Fabian Society in Melbourne.

"Some aspects of factionalism are a far bigger impediment than any concerns about unions," he said.

Changes must be considered to restore balance within the ALP and a "connection" between voters and politicians, he said. "Too much of the party's precious talent and energy has been diverted away from electing candidates and into just advancing factional interest ... the factions are not seen to be driven by altruism or ideas but just number-crunching. Factions are perceived to have hijacked the ideas.

"Many in the party who could have made a contribution, but don't belong to a faction, feel they are ignored and so drift away."

Opposition Leader Simon Crean's office said last night he had not seen the speech and declined to comment.

Under Mr Swan's proposal, Labor would select a short-list of potential candidates for parliament and registered Labor supporters living in that electorate would vote for the preferred candidate, taking the decision away from factions and paid-up party members.

The plan would require changes to the electoral system to allow the Electoral Commission to conduct the primary elections. People would be able to nominate themselves as Labor supporters when they enrolled to vote so they could participate in the preselection ballot, but would not be required to vote Labor in elections.

Preselection rules vary in each state. In Victoria, 50 per cent of the votes for a candidate are lodged by between 300 and 700 local branch members and 50 per cent are made by a central panel, of which unions have half the votes.

Mr Swan said the system could also be used to choose some members of Labor's powerful national executive. He also put forward a compromise option to split voting between Labor supporters and union members.

"Part of the alienation from politics is the lack of connection between voters and their elected representatives," he said.

The plan is expected to face criticism within Labor ranks, with senior sources last night warning it would encourage big-spending campaigns to win preselection battles and import the worst aspects of American politics.

Others welcomed Mr Swan's contribution as brave thinking to confront factions.

Mr Swan also delivered a sober assessment of the November 10 election, revealing that Labor's private polling showed in April that while it was ahead in the polls, the swing was being driven by preferences from minor parties and not a groundswell of support for Labor.

"Before Tampa and September 11 there was a certain loss of impetus in our drive towards victory," he said.

He said Labor failed to convince the "missing middle" of outer suburban people "who have done well in life and want their children to do even better".

"This middle-class, suburban voter is Labor's greatest challenge. It's no good thinking they will come along for the ride out of curiosity - they have to be convinced."