These include the prevention of killing, and "causing serious bodily or mental harm" to members of the group, as well as preserving evidence of possible genocide that has already occurred.
Presiding judge Abdulqawi Ahmed Yusuf said Myanmar should report back within four months on how it was implementing the ruling.
The measures are binding and not subject to appeal, but the court has no means of enforcing them.
Media captionRohingya refugee: "They killed my family in front of me"
What now for Aung San Suu Kyi?
By Nick Beake, Myanmar correspondent, BBC News
This judgment has surely obliterated any remnants of Aung San Suu Kyi's international reputation.
Remember, she didn't have to go to The Hague and become the embodiment of Myanmar's defence. She chose to argue, in person, there was no mass murder, rape or arson.
Even her biggest critics used to acknowledge she doesn't control the still powerful Burmese army, but now she has destroyed the firewall between her and the generals by trying - and failing - to justify their actions.
So far, Myanmar has played by the rules of the International Court of Justice. But will it abide by these emergency measures?
Writing today in a British newspaper, Aung San Suu Kyi questioned whether the international justice system was capable of ignoring "unsubstantiated narratives" told by human rights groups and UN investigators against her country.
So, after initially engaging with the UN's top court, will a defeated Aung San Suu Kyi retreat now into isolation?
What is Myanmar's position?
During hearings at the court in December, Ms Suu Kyi asked the ICJ to drop the case, describing it as "incomplete and incorrect".
And in an article for the Financial Times published shortly before the court's judgement she said human rights groups had condemned Myanmar based on "unproven statements without the due process of criminal investigation".
Media captionThe BBC saw the remains of burned villages on a tightly controlled government trip to Rakhine state
"The international condemnation has had a negative effect on Myanmar's endeavours to bring stability and progress to Rakhine," she said.
"It has undermined painstaking domestic efforts to establish co-operation between the military and the civilian government."
Her remarks also appeared to echo a statement by a government-appointed panel earlier this week which accepted that war crimes may have been committed by individuals but said there was no indication of an intent to commit genocide.
However, the BBC's Anna Holligan, who is in The Hague, says that by coming to the court in December, Ms Suu Kyi had in effect recognised its legitimacy and it will now be difficult for Myanmar to ignore its judgement.
What has been the reaction?
Rohingya groups have welcomed the decision.
"Today's ruling by the ICJ is a crucial moment for Rohingya justice, and vindication for those of us who have lived through this genocide for decades," tweeted Tun Khin, president of the Burmese Rohingya Organisation UK.
"The court's decision clearly shows that it takes the allegations of genocide seriously, and that Myanmar's hollow attempts to deny these have fallen on deaf ears."
Human rights organisation Amnesty International said the decision sent a message that the world would not tolerate Myanmar's "atrocities".
Gambian Justice Minister Abubacarr Tambadou, who led the prosecution, said he was "very, very pleased".
"I think this represents a triumph of international law and international justice. And it is the international community - as represented by the ICJ - saying in the strongest of terms that genocide will not be accepted under any circumstances by any perpetrators," he told the BBC's Focus on Africa programme.
But some Burmese people responding to a BBC Facebook live broadcast were scathing of the court and its judges.
"This is not a fair and just ruling. I would like to speak on behalf of the Myanmar people that those judges are blind. They are deaf. They do not know the real situation in the country," said Nu Yimwin.
Kyaw Myint Oo described the ruling as a tragic day for the country: "Our situation is like being a prey gradually strangled by a python and eventually we will be forced to give in to all their demands."
Bangladesh Foreign Minister AK Abdul Momen said: "We hope good sense will prevail in Myanmar and they will take back all the Rohingya refugees and provide them security."
What is the background to the case?
The Rohingya, who numbered around one million in Myanmar at the start of 2017, are one of the many ethnic minorities in the country. Rohingya Muslims are the largest community of Muslims in Myanmar, with the majority living in Rakhine state.
But Myanmar's government denies them citizenship, refusing to recognise them as a people and seeing them as illegal immigrants from Bangladesh.
Waves of Rohingya refugees have fled Myanmar for Bangladesh over the decades but their latest exodus began on 25 August 2017 after militants from a Rohingya insurgent group called Arsa launched deadly attacks on more than 30 police posts.
Rohingyas arriving in Bangladesh said they fled after troops, backed by local Buddhist mobs, responded by burning their villages and attacking and killing civilians.
The government claims that "clearance operations" against the militants ended on 5 September 2017, but analysis of satellite imagery by Human Rights Watch suggests hundreds of villages were destroyed after August that year.