When I have message-router not running and send event through VES, VES collector tries to send event to mesage-router in infinity loop.

From logs I see that there is threshold of retries = 5, hovewer threshold is reached but retring is not stopped. Current number of retries exceeded 38 and time when event is on the queue reached almost 5M of ms.

I suspect that when number of events will be higher VES can have problems with memory as a "snowball" effect.

2020-09-15 08:23:22 WARN Error executing HTTP request. Connect to message-router:3904 [message-router/10.43.212.244] failed: Connection timed out (Connection timed out); blacklisting for 2 minutes
2020-09-15 08:23:22 WARN Connect to message-router:3904 [message-router/10.43.212.244] failed: Connection timed out (Connection timed out)
org.apache.http.conn.HttpHostConnectException: Connect to message-router:3904 [message-router/10.43.212.244] failed: Connection timed out (Connection timed out)
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) ~[httpclient-4.5.10.jar:4.5.10]
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:374) ~[httpclient-4.5.10.jar:4.5.10]
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient-4.5.10.jar:4.5.10]
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.10.jar:4.5.10]
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.10.jar:4.5.10]
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.10.jar:4.5.10]
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.10.jar:4.5.10]
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.10.jar:4.5.10]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient-4.5.10.jar:4.5.10]
at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:622) ~[saClientLibrary-0.0.1.jar:?]
at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:452) ~[saClientLibrary-0.0.1.jar:?]
at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.sendBatch(CambriaSimplerBatchPublisher.java:315) [cambriaClient-0.0.1.jar:?]
at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.send(CambriaSimplerBatchPublisher.java:232) [cambriaClient-0.0.1.jar:?]
at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher.access$100(CambriaSimplerBatchPublisher.java:30) [cambriaClient-0.0.1.jar:?]
at com.att.nsa.cambria.client.impl.CambriaSimplerBatchPublisher$1.run(CambriaSimplerBatchPublisher.java:384) [cambriaClient-0.0.1.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [?:?]
at java.util.concurrent.FutureTask.runAndReset(Unknown Source) [?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
at java.lang.Thread.run(Unknown Source) [?:?]
Caused by: java.net.ConnectException: Connection timed out (Connection timed out)
at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:?]
at java.net.AbstractPlainSocketImpl.doConnect(Unknown Source) ~[?:?]
at java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source) ~[?:?]
at java.net.AbstractPlainSocketImpl.connect(Unknown Source) ~[?:?]
at java.net.SocksSocketImpl.connect(Unknown Source) ~[?:?]
at java.net.Socket.connect(Unknown Source) ~[?:?]
at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) ~[httpclient-4.5.10.jar:4.5.10]
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[httpclient-4.5.10.jar:4.5.10]
... 20 more
2020-09-15 08:23:22 WARN Send failed, 1 message to send.
2020-09-15 08:23:22 ERROR PUB_CHRONIC_FAILURE: Send failure count is 38, above threshold 5.
2020-09-15 08:23:23 INFO sending 1 msgs to /events/unauthenticated.SEC_FAULT_OUTPUT. Oldest: 4981432 ms
2020-09-15 08:23:23 WARN All hosts were blacklisted; reverting to full set of hosts.
2020-09-15 08:23:23 INFO POST http://message-router:3904/events/unauthenticated.SEC_FAULT_OUTPUT (anonymous) ...