lnd: remove peer from peerChanInfo when necessary

We now remove the peer from `peerChanInfo` if this peer doesn't have
channels with us. Also patched a unit test for `removePeerAccess`.

server.go

@@ -4964,7 +4964,8 @@ func (s *server) removePeer(p *peer.Brontide) {
 	}
 
 	// Remove the peer's access permission from the access manager.
-	s.peerAccessMan.removePeerAccess(p.IdentityKey())
+	peerPubStr := string(p.IdentityKey().SerializeCompressed())
+	s.peerAccessMan.removePeerAccess(peerPubStr)
 
 	// Copy the peer's error buffer across to the server if it has any items
 	// in it so that we can restore peer errors across connections.