I have successfully set up a billboard shader that works, it can take in a quad and rotate it so it always points toward the screen. I am using this vertex-shader:

void main(){
vec4 tmpPos = (MVP * bufferMatrix * vec4(0.0, 0.0, 0.0, 1.0)) + (MV * vec4(
    vertexPosition.x * 1.0 * bufferMatrix[0][0], 
    vertexPosition.y * 1.0 * bufferMatrix[1][1], 
    vertexPosition.z * 1.0 * bufferMatrix[2][2], 
    0.0)
);

UV = UVOffset + vertexUV * UVScale;
gl_Position = tmpPos;

BufferMatrix is the model-matrix, it is an attribute to support Instance-drawing.

The problem is best explained through pictures:

This is the start position of the camera:

And this is the position, looking in from 45 degree to the right:

Obviously, as each character is it's own quad, the shader rotates each one around their own center towards the camera. What I in fact want is for them to rotate around a shared center, how would I do this?

What I have been trying to do this far is:

 mat4 translation = mat4(1.0);
    translation = glm::translate(translation, vec3(pos)*1.f * 2.f);
    translation = glm::scale(translation, vec3(scale, 1.f));
    translation = glm::translate(translation, vec3(anchorPoint - pos) / vec3(scale, 1.f));

Where the translation is the bufferMatrix sent to the shader. What I am trying to do is offset the center, but this might not be possible with a single matrix..?

I am interested in a solution that doesn't require CPU calculations each frame, but rather set it up once and then let the shader do the billboard rotation. I realize there's many different solutions, like merging all the quads together, but I would first like to know if the approach with offsetting the center is possible.

If it all seems a bit confusing, it's because I'm a little confused myself.