Representative Steve Scalise of Louisiana withdrew on Thursday from consideration for the speakership he was on the cusp of claiming after hard-line Republicans balked at rallying around their party’s chosen candidate, leaving the House leaderless and the G.O.P. in chaos.

After being narrowly nominated for speaker during a Wednesday closed-door secret-ballot contest among House Republicans, Mr. Scalise, their No. 2 leader, found himself far from the 217 votes needed to be elected on the House floor. Many supporters of his challenger, Representative Jim Jordan of Ohio, the right-wing Republican endorsed by former President Donald J. Trump, refused to switch their allegiance.

With no clear end in sight to the G.O.P. infighting that has left one chamber of Congress paralyzed at a time of challenges at home and abroad, Mr. Scalise said he would step aside in hopes that someone else could unite the fractious party.

“I just shared with my colleagues that I was withdrawing my name as a candidate for speaker-designee,” Mr. Scalise said. “If you look at where our conference is, there’s still work to be done. Our conference still has to come together, and it’s not there. There are still some people that have their own agendas.”

His abrupt exit left Republicans back at square one, as fractured as ever over who should lead them and trading recriminations about the disarray in which they found themselves.

They planned a Friday morning meeting to discuss how to move forward.

“Steve won fair and square, and yet we had people who refused to vote for him,” said Representative Don Bacon of Nebraska, adding, “If you reward bad behavior, you’re going to get more of it.”

Mr. Scalise’s downfall came after an extraordinary few days on Capitol Hill that put Republican divisions on vivid display. Mr. Scalise surpassed Mr. Jordan during the internal party contest by just 14 votes. But rather than consolidating his narrow base of backers, Mr. Scalise almost immediately began hemorrhaging supporters, as lawmakers from several factions publicized that they did not intend to fall into line behind him.

Then Mr. Trump weighed in on Thursday against Mr. Scalise, arguing that the Louisianian was unfit for the speakership because he has blood cancer.

“Steve is a man that is in serious trouble, from the standpoint of his cancer,” Mr. Trump said on Fox News Radio, adding, “I just don’t know how you can do the job when you have such a serious problem.”

Other top House Republicans refrained from publicly rallying around Mr. Scalise, allowing the resistance to fester. Mr. Jordan never made a full-throated endorsement of Mr. Scalise, despite indicating his support. And Representative Kevin McCarthy of California, the ousted former speaker who has an icy relationship with Mr. Scalise, said the Louisiana Republican had overestimated his backing and might be unable to recover.

After Mr. Scalise’s withdrawal, Mr. Jordan’s supporters immediately began endorsing him as next in line, and he was expected to pursue the speakership on Friday. But he is likely to encounter opposition from the party’s more moderate members.

“I hope now he’s the obvious choice,” said Representative Jim Banks of Indiana, who backs Mr. Jordan. “He barely came in second place to Steve Scalise.”

But Representative Mike Garcia of California said the way Mr. Jordan’s supporters acted after he lost would make it hard for some in the conference to back the Ohioan.

“There’s an academic debate about whether we reward the tyranny of the minority in this case,” Mr. Garcia said, adding, “The problem is, I think there’s enough people that would see what has happened and transpired over the last 40 hours and not support him; that we’re going to have the same problem with Jordan that we had with Scalise.”

Mr. McCarthy did not rule out a return to the speakership as well, saying he would “let the conference decide” whether to reinstate him to the job from which he was ousted just last week.

And Representative Patrick T. McHenry of North Carolina, who was named the interim speaker after Mr. McCarthy’s removal, was also being talked about as a potential candidate.

Foreseeing a fight that could last for weeks, some members were discussing how they might give Mr. McHenry, whose role is primarily to hold an election for a speaker, more power to carry out the chamber’s work until the conflict could be resolved.

Mr. Scalise’s exit was the latest remarkable turn in a saga that has been marked by whiplash, shifting alliances and petty grudges. The situation has highlighted major changes in the nature of the House Republican conference, whose members once dutifully lined up in support of their chosen leaders but increasingly appeared to be pursuing a strategy of every member for themselves.

The range of complaints against Mr. Scalise ran the gamut, crossing ideological and regional lines and reflecting the many competing factions among House Republicans. But some were merely petty and personal.

“There’s some folks that really need to look in the mirror over the next couple of days and decide: Are we going to get back on track, or are they going to try to pursue their own agenda?” Mr. Scalise said. “You can’t do both.”

Even though the votes on Thursday had clearly stacked up against Mr. Scalise, some of his allies were still surprised by his withdrawal announcement in a closed-door meeting. Several openly wept.

The uncertainty has hobbled the House, as it confronts multiple crises, with U.S. allies Israel and Ukraine at war and a government shutdown looming next month without a congressional spending agreement.

Representative Mark Alford of Missouri said the conference was in disarray: “There is some deep mistrust. There’s some communication problems. Some things are jacked up.”

But he believed that fellow Republicans would work out their differences. “We will get to 217,” he said of the votes needed. “Who that is, I don’t know.”

Mr. Scalise has served in House leadership since 2014, and overcame great personal hardship to become the choice of a majority of Republicans to lead the chamber.

He is undergoing intense treatment for his blood cancer, which has prompted him to wear a mask to vote on the House floor and at news conferences. And in 2017, during a practice for a congressional baseball game, an anti-Trump extremist shot and seriously wounded Mr. Scalise. He still walks with a limp from the incident.

It was not clear whether Mr. Scalise could keep his post as majority leader after his failed attempt to win the top job, though Mr. McCarthy expressed confidence that he could. And Mr. Scalise indicated he would try.

“I’m the majority leader of the House. I love the job I have,” Mr. Scalise said. “I’ve had many challenges in my life. I’ve been tested in ways that really put perspective on life.”