Amazon EMR - Map Reduce error - # of failed Map Tasks exceeded allowed limit

Amazon EMR - Map Reduce error:

Job not successful. Error: # of failed Map Tasks exceeded allowed limit

All this error message means, is that an unacceptable number of Map tasks have failed.
I guess normally, this is if ANY Map tasks fail.

So to solve the issue, we need to dig into the logs, to find why the individual tasks failed:

In EMR web console:


click on the job flow.
select Debug.
beside the failed step: select View Jobs.
Beside the failed job, select View Tasks.

You can now see a more detailed breakdown of the individual map - reduce tasks.

Select a task that failed, and select View Attempts.

Select View stderr.

You should now be able to diagnose the reason for the failure.

note: in my case, I had silly mistakes, such as:

- incorrectly setting the path to the Map script

- my Map script was in Python, and I forgot to add the bit of text at the top of the script, to tell Unix that this is a Python script (and not a shell script)

#!/usr/bin/python