Police said Abul Hashem and four of his sons were among eight people detained over the inferno, which broke out Thursday and raged for more than a day.
A separate inquiry has been launched into the use of child labour at the food factory.
Jayedul Alam, police chief for Narayanganj district where the factory is located, said the entrance had been padlocked at the time of the blaze and the factory breached multiple safety regulations.
“It was a deliberate murder,” the police chief told AFP. All eight people detained face murder charges.
Before his arrest, Hashem told the Daily Star newspaper the fire “may have been a result of workers’ carelessness”, suggesting a discarded cigarette could have sparked the carnage.
Emergency services found 48 of the bodies on the third floor of the Hashem Food and Beverage factory in Rupganj, an industrial town outside Dhaka.
The fire department said the exit door to the main staircase had been padlocked and highly flammable chemicals and plastics had been stored in the building.
Monnujan Sufian, state minister for labour, said inquiries had begun into the use of child workers at the factory.
Child victims mourned
Laizu Begum, who spent hours waiting outside the factory, said her 11-year-old nephew had been working on the third floor and was now missing.
“We heard the door of the floor where my nephew worked was padlocked. Then we realised, after seeing how big the fire was, that he is probably dead,” said the woman, who broke into tears.
Outside the burned shell of the factory and at the hospital morgue, AFP spoke to 30 survivors and relatives of the dead, who said child workers at the factory were paid just 20 taka (24 US cents) per hour.
Bilal Hossain, father of 14-year-old Mitu Akter, who was among the missing, went to the Dhaka Medical College Hospital morgue with a photo of his daughter, begging police and doctors to hand over her remains.
“I sent my baby girl to die,” said Hossain, who added that the company even owed the girl back wages. “How will I tell her mother?”
At the hospital, Chandu Mia held a photo of his 15-year-old daughter. “I am not sure if she is alive,” he said, holding back tears.
State minister Sufian told AFP she had spoken to two survivors aged 14.
“I went to the hospital and I asked them how old they are. The youngest were 14,” she said.
Sufian said some children aged as young as 14 were allowed to work in non-hazardous jobs, but that the Hashem factory was considered hazardous.
“If child labour is proved, we will take action against the owner and the inspectors,” she said.
Bangladesh pledged safety reforms after the Rana Plaza disaster in 2013, when a nine-storey complex collapsed killing more than 1,100 people.
But there has been a series of fires and other disasters since then. In February 2019 at least 70 people died when a fire ripped through Dhaka apartments where chemicals were illegally stored.