Short version: 
Found the culprit. Seems to be working. Not sure who's to blame. 
It was missing route to container IP subnet on KVM virtualization host.

Long version. 

All Kubernetes hosts are VM's on my workstation, deployed via Juju on MaaS. So networking was "automagically" created by MaaS and it seems to be working. I can communicate between nodes and maas/juju deployments works as expected. 

sysctl net.ipv4.ip_forward shows 1 as expected. 

Juju cluster machines:

Machine  State    DNS             Inst id       Series  AZ       Message
0        started  192.168.101.8   tight-gar     bionic  default  Deployed
1        started  192.168.101.9   modest-aphid  bionic  default  Deployed
2        started  192.168.101.10  live-hound    bionic  default  Deployed
3        started  192.168.101.19  super-cub     bionic  default  Deployed
4        started  192.168.101.17  fair-fish     bionic  default  Deployed
5        started  192.168.101.18  happy-viper   bionic  default  Deployed

Test Kubernetes pod (I have deployed a generic deployment in case metrics server being some kind of special).
bacila@juodas ~/Documents/Work/Kubernetes $ kubectl  get po -o wide
NAME                                READY   STATUS    RESTARTS   AGE   IP              NODE          NOMINATED NODE   READINESS GATES
nginx-deployment-574b87c764-z2trs   1/1     Running   0          19s   10.100.69.132   happy-viper   <none>           <none>

Regarding firewall, that was an idea worth checking. I was just not sure which corner should I kick into, and which corner should I listen for an echo to identify problem. ip a l listed 37 interfaces, so there were quite a few of them there :) So I started from good old tcpdump. 

* SSH to worker node and run ping metrics_server_IP 
* On the workstation run "tcpdump -i any icmp"

22:17:03.045019 IP 192.168.101.17 > 10.100.69.132: ICMP echo request, id 9430, seq 1, length 64
22:17:03.045067 IP 192.168.101.1 > 10.100.69.132: ICMP echo request, id 9430, seq 1, length 64
22:17:03.045299 IP 10.100.69.132 > 192.168.101.1: ICMP echo reply, id 9430, seq 1, length 64

So I see that ping reaches test pod and it sends reply to Juju network gateway but it does not reach back origin. 

I have looked at main host route list and it lacked container network IP range, which means that ICMP response is sent to default GW. 

root@juodas:/home/bacila# ip  r l
default via 192.168.4.254 dev enp8s0 proto dhcp metric 100 
169.254.0.0/16 dev enp8s0 scope link metric 1000 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 
172.18.0.0/16 dev br-03ebb1003d68 proto kernel scope link src 172.18.0.1 linkdown 
192.168.4.0/24 dev enp8s0 proto kernel scope link src 192.168.4.2 metric 100 
192.168.100.0/24 dev virbr1 proto kernel scope link src 192.168.100.1 linkdown 
192.168.101.0/24 dev virbr2 proto kernel scope link src 192.168.101.1 
192.168.123.0/24 dev virbr0 proto kernel scope link src 192.168.123.1 linkdown 

So I have added a route to container IP range via juju cluster GW. 
root@juodas:/home/bacila# ip r a 10.100.0.0/16 via 192.168.101.1

Voila! ping responds and metrics apiservice no longer shows an error and I have my metrics. 

I have also added a service IP range so requests to service IP now works
ip r a 10.152.0.0/16 via 192.168.101.1

So it was missing routes on virtualization host. Why they are missing it is a question. Juju is run via non privileged user, so juju cannot update host level routing. So should it be done somehow via MaaS? Or simply install instructions updated with some manual post deployment tasks.