Michael Nelson cut a frustrated figure after the game at Roots Hall on Friday evening.
The defender, who turns 29 on Saturday, felt the team were hard done by in defeat but admitted Southend's third goal just before the break made it extremely difficult,
"Obviously the penalty decision and the sending off killed us early on and there were a lot of little things that went on after that which sort of changed the course of the game.
"You know you see decisions being contested and then accepted but sometimes the referees have to take responsibility for decisions made.
"Joel has kicked it and then stumbled over and lost the ball and the referee has played on. Ben Clark then did exactly the same to their lad and the referee played on but came back to book Ben.
"I had a go at him about the inconsistency and raised my voice and the referee has pointed his finger in my face and told me to 'f****** shut up'. Where the respect campaign goes there, I don't know but I think if I had pointed my finger in his face and said the same, I'd have ended up watching the last half an hour in the changing room.
"We got ourselves back into it with the penalty and that was another one of those incidences. The lad has brought Joel down inside the area and the referee has not even had a word with him never mind booking him.
"Their third goal just before half time though was a bad one for us because going in at 2-1 would have given us more of a chance.
"We soaked it up in the second half and could have snatched it. We got the second goal and made it 3-2 and i think their lads started to panic a bit."
And now Nelson insists full attention will turn to the next game as Pools fight to secure their League One status,
"We're alright now but we need to make sure we are still there come the end of the season. We need to pick some points up and we need them fast, starting against Hereford next Saturday."