It’s been three weeks since the Boston Red Sox made headlines by firing manager Alex Cora and overhauling much of their coaching staff on April 25. At that time, the team sat at 10-17, dead last in the AL East. But has the shakeup sparked a turnaround?

In the 16 games since the changes, Boston has gone 8-8—a modest improvement, but hardly a dramatic rescue. While the record shows slight progress, the underlying numbers tell a more complex story.

The offense, despite new hitting coaches, continues to sputter, especially at Fenway Park. Before the staff overhaul, the Red Sox scored 73 runs over 16 games with a .665 OPS. In the 16 games since, they’ve managed just 49 runs with a nearly identical .668 OPS. That averages out to a drop from 4.15 runs per game to 3.06, suggesting the team’s hitting struggles may run deeper than coaching.

On the bright side, Boston has found other ways to compete. The team is stealing more bases, the defense has been sharp, and the pitching staff has made significant strides. After posting a 4.35 ERA through April, the staff has improved to 2.56 in May—a clear win for general manager Craig Breslow’s decision to keep the pitching coaches in place.

Still, strong pitching can only carry a team so far. The Red Sox need to ignite their offense if they hope to climb out of the AL East basement and salvage their season. For now, the jury is still out on whether this coaching shakeup will deliver the lasting change Boston desperately needs.