In Deuteronomy 5:9, it talks about God saying that he punishes the children for the sin of their parents… How is this fair? Doesn’t this violate Gods quality of being a just God? 
Even in genesis! When ham committed sin, Canaan was punished. (I think Canaan is the descendants of ham) How is this fair? Isn’t it ham who should have been punished?


Deuteronomy 5     ‘You shall not make for yourself a carved image—any likeness of anything that is in heaven above, or that is in the earth beneath, or that is in the water under the earth; you shall not bow down to them nor serve them. For I, the Lord your God, am a jealous God, visiting the iniquity of the fathers upon the children to the third and fourth generations of those who hate Me, 10 but showing mercy to thousands, to those who love Me and keep My commandments.

As you said, context is important, so let’s look at the whole second commandment. The Bible says that God is love, that He is just and fair, that each person will die for his own sin, and that nobody will die for another’s sin (2 Chronicles 25:4). So, either the Bible contradicts itself, or this passage has another meaning. And it has. The context is idol worship. What it is saying is that whatever the parents do as an example for their children (worship idols), is likely to be followed by the children, or even worse than the parents. And by the third and fourth generations God will be totally out of the picture, and idol worship will be the only worship left. Don’t ever think that God is unfair, harsh, partial or anything that is contrary to what the Bible tells us. That is exactly what the devil has been telling the world since Adam and Eve, which is all lies.

John 8:44    You are of your father the devil, and the desires of your father you want to do. He was a murderer from the beginning, and does not stand in the truth, because there is no truth in him. When he speaks a lie, he speaks from his own resources, for he is a liar and the father of it.”

When we look at Ham, who disrespected his father Noah and laughed at his nakedness, we see what he passed on to his son Canaan. It was not God who spoke the curse, but Noah was the one that spoke the words about Canaan. What Noah saw, was a disrespectful father pass-ing these traits along to his son Canaan, leading to the characteristics that would make Ham’s descendants fit only for serving the others.