There are horror stories about trying to work with him. Apparently he had a terrible attitude and never remembered his lines.
There’s a scene in season 2 (I think) where he is sitting in a room with Lucky/Meyer and I think Capone, and they are plotting against Nucky. Apparently they couldn’t do the scene as intended because Pitt didn’t know his lines. Much of the scene is filmed with the camera solely on him because they had to stay late after everyone left and shoot his dialogue scenes individually to get his lines right. Things like this were allegedly a regular occurrence.