Prime Minister Theresa May's office said late Sunday that May had accepted the resignation of Home Secretary Amber Rudd.
The scandal has prompted a wider debate about the government's harsh methods against illegal immigration.
She told Theresa May in a telephone call on Sunday evening that she had made a decision to quit for "inadvertently" misleading MPs over the Home Office's use of deportation targets.
"I don't like the term hostile and I won't be using it", he vowed.
But Labour lawmaker David Lammy, a strong critic of the government's immigration policies, said the scandal would cause European Union citizens to think, "My God, if this can happen to Windrush, of course it can happen to us".
The appointment was made after Amber Rudd quit the position over an illegal immigration policy controversy.
Highlighting his status as a second-generation migrant, the Home Secretary described how the row has "affected me greatly", adding: "I thought that that could be my mum, my brother, my uncle or even me".
Ms Rudd was forced from office after telling lawmakers last week that there were no targets for the removal of people deemed to be in the country illegally.
But she felt it "necessary" to tender her resignation after the emergence of documents, addressed to her office, showing those goals were in place.
Javid is the son of a Pakistani bus driver who arrived in Britain in 1961 with one pound in his pocket. The 48-year old, who previously served as business and culture secretaries, led the government's response to last year's Grenfell Tower fire disaster.
Explaining the thinking behind Mr Javid's appointment, Mrs May's spokesman said he was "one of the most experienced ministers" in Cabinet who had "proved his drive, his ambition and his determination to get to grips with hard subjects". Mr Javid was a lukewarm campaigner to remain in the bloc and has said the 2016 referendum result meant "in some ways, we're all Brexiteers now".
Rudd's departure from the government is a massive blow to May, who was her immediate predecessor in the Home Office.
Ms Rudd was the fifth person to quit the cabinet since the June 2017 snap general election, called by Mrs May, but which cost the Conservatives their majority in parliament.
He was replaced as Secretary of State for Housing, Communities and Local Government by former Northern Ireland secretary James Brokenshire, who has recently returned to Westminster after treatment for cancer.
The furor has grown since the Guardian newspaper reported that some people who came to the United Kingdom from the Caribbean in the decades after World War II had recently been refused medical care in Britain or threatened with deportation because they could not produce paperwork proving their right to reside in the country.
An inspection of removals by the borders and immigration watchdog said targets were set in 2014/15 and 2015/16, which were split between 19 Immigration Compliance and Enforcement (Ice) teams across the UK.
Known as the "Windrush generation" because one of the first ships to bring them was the HMT Empire Windrush, hundreds.
However, many never formalized their status, often because they were children who came over on their parents' passports and then never applied for their own.
Ms Rudd wrote in her resignation letter that she "should have been aware" of the targets, adding that Britons "want people who have a right to live here to be treated fairly and humanely, which has sometimes not been the case" - a criticism of her own ministry, and possibly of Mrs May.