Database connection fails then node startup crashes

Hi,

I was forging when a database connection error happened. After this, the node was still up, but stopped forging.

uns-sandbox | [2020-02-21 05:44:42.219] INFO : Received new block at height 175,010 with 0 transactions from 127.0.0.1
uns-sandbox | [2020-02-21 05:44:42.219] INFO : Previous block 175,009 pinged blockchain 8 times
uns-sandbox | [2020-02-21 05:44:42.220] DEBUG: event 'NEWBLOCK': "idle" -> "newBlock"
uns-sandbox | [2020-02-21 05:44:42.222] DEBUG: Delegate b2c0b3cfd7d9b2b8f849b71996911afe596d8719ead0e28d7818cd1002c5fd2f (036efe3c6c6a9568ba425f5655749eed30f7f5814e7baeb620b1ce525d12f549d2) allowed to forge block 175,010
uns-sandbox | [2020-02-21 05:44:42.625] DEBUG: event 'PROCESSFINISHED': "newBlock" -> "idle" -> actions: [checkLater, blockchainReady]
uns-sandbox | [2020-02-21 05:44:43.125] INFO : Broadcasting block 175,010 to 11 peers
uns-sandbox | [2020-02-21 05:44:50.883] INFO : Received new block at height 175,011 with 0 transactions from 51.91.252.109
uns-sandbox | [2020-02-21 05:44:50.883] INFO : Previous block 175,010 pinged blockchain 10 times
uns-sandbox | [2020-02-21 05:44:50.883] DEBUG: event 'NEWBLOCK': "idle" -> "newBlock"
uns-sandbox | [2020-02-21 05:44:50.885] DEBUG: Delegate fb460b6fe4c6fcc5488cab08778fb8e44eac124296ebe68ce659da0c178558c0 (0209d018a06f9db7bf6833325b1b046be57593dbf969c490f5e930fa8179affe1c) allowed to forge block 175,011
uns-sandbox | [2020-02-21 05:45:04.774] INFO : Received new block at height 175,012 with 0 transactions from 51.15.218.206
uns-sandbox | [2020-02-21 05:45:04.776] INFO : Previous block 175,011 pinged blockchain 8 times
uns-sandbox | [2020-02-21 05:45:04.777] DEBUG: event 'NEWBLOCK': "newBlock" -> "newBlock"
uns-sandbox | [2020-02-21 05:45:14.827] INFO : Received new block at height 175,013 with 0 transactions from 51.15.218.206
uns-sandbox | [2020-02-21 05:45:14.828] INFO : Previous block 175,012 pinged blockchain 8 times
uns-sandbox | [2020-02-21 05:45:14.829] DEBUG: event 'NEWBLOCK': "newBlock" -> "newBlock"
uns-sandbox | [2020-02-21 05:45:22.703] INFO : Received new block at height 175,014 with 0 transactions from 51.158.99.6
uns-sandbox | [2020-02-21 05:45:22.704] INFO : Previous block 175,013 pinged blockchain 9 times
uns-sandbox | [2020-02-21 05:45:22.705] DEBUG: event 'NEWBLOCK': "newBlock" -> "newBlock"
uns-sandbox | [2020-02-21 05:45:36.711] ERROR: read ECONNRESET
uns-sandbox | [2020-02-21 05:45:36.712] ERROR: Could not save 1 blocks to database : Error: read ECONNRESET
uns-sandbox |     at TLSWrap.onStreamRead (internal/stream_base_commons.js:205:27)
uns-sandbox | (node:123) UnhandledPromiseRejectionWarning: Error: Client has encountered a connection error and is not queryable
uns-sandbox |     at /opt/uns/node_modules/pg/lib/client.js:500:25
uns-sandbox |     at processTicksAndRejections (internal/process/task_queues.js:79:11)
uns-sandbox |     at runNextTicks (internal/process/task_queues.js:66:3)
uns-sandbox |     at processImmediate (internal/timers.js:429:9)
uns-sandbox | (node:123) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 380)
uns-sandbox | (node:123) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
uns-sandbox | [2020-02-21 05:45:36.755] INFO : Received new block at height 175,015 with 0 transactions from 51.15.239.15
uns-sandbox | [2020-02-21 05:45:36.756] INFO : Previous block 175,014 pinged blockchain 6 times
uns-sandbox | [2020-02-21 05:45:36.757] DEBUG: event 'NEWBLOCK': "newBlock" -> "newBlock"
uns-sandbox | [2020-02-21 05:45:50.886] DEBUG: event 'WAKEUP': "newBlock" -> "newBlock"
uns-sandbox | [2020-02-21 05:46:00.330] INFO : Received new block at height 175,016 with 0 transactions from 82.64.214.110
uns-sandbox | [2020-02-21 05:46:00.331] INFO : Previous block 175,015 pinged blockchain 8 times
uns-sandbox | [2020-02-21 05:46:00.332] DEBUG: event 'NEWBLOCK': "newBlock" -> "newBlock"
uns-sandbox | [2020-02-21 05:46:10.903] INFO : Received new block at height 175,017 with 0 transactions from 82.64.214.110
uns-sandbox | [2020-02-21 05:46:10.904] INFO : Previous block 175,016 pinged blockchain 8 times
uns-sandbox | [2020-02-21 05:46:10.905] DEBUG: event 'NEWBLOCK': "newBlock" -> "newBlock"
uns-sandbox | [2020-02-21 05:46:18.823] INFO : Received new block at height 175,018 with 0 transactions from 51.158.113.20

When I get to restart the node manually, I get the following error.

uns-sandbox | [2020-02-21 12:06:28.614] DEBUG: Connecting to database
uns-sandbox | [2020-02-21 12:06:32.542] DEBUG: Connected to database.
uns-sandbox | [2020-02-21 12:06:33.408] INFO : Connecting to transaction pool
uns-sandbox | [2020-02-21 12:06:33.618] INFO : Starting Blockchain Manager :chains:
uns-sandbox | [2020-02-21 12:06:33.623] DEBUG: event 'START': "uninitialised" -> "init" -> actions: [init]
uns-sandbox | [2020-02-21 12:06:33.625] INFO : Verifying database integrity
uns-sandbox | [2020-02-21 12:06:34.156] INFO : Verified database integrity
uns-sandbox | [2020-02-21 12:06:34.290] INFO : Last block in database: 175,010
uns-sandbox | [2020-02-21 12:06:34.298] INFO : State Generation - Step 1 of 27: Block Rewards
uns-sandbox | [2020-02-21 12:06:34.484] INFO : State Generation - Step 2 of 27: Fees & Nonces
uns-sandbox | [2020-02-21 12:06:34.627] INFO : State Generation - Step 3 of 27: Transfer
uns-sandbox | [2020-02-21 12:06:34.759] INFO : State Generation - Step 4 of 27: SecondSignature
uns-sandbox | [2020-02-21 12:06:35.024] ERROR: TypeError: Cannot read property 'signature' of null
uns-sandbox |     at SecondSignatureTransactionHandler.bootstrap (/opt/uns/packages/core-transactions/dist/handlers/second-signature.js:23:74)
uns-sandbox |     at async StateBuilder.run (/opt/uns/packages/core-database-postgres/dist/state-builder.js:25:13)
uns-sandbox |     at async PostgresConnection.buildWallets (/opt/uns/packages/core-database-postgres/dist/postgres-connection.js:92:9)
uns-sandbox |     at async DatabaseService.buildWallets (/opt/uns/packages/core-database/dist/database-service.js:104:9)
uns-sandbox |     at async init (/opt/uns/packages/core-blockchain/dist/state-machine.js:146:13)
uns-sandbox | [2020-02-21 12:06:35.027] DEBUG: event 'FAILURE': "init" -> "exit" -> actions: [exitApp]
uns-sandbox | [2020-02-21 12:06:35.030] ERROR: Failed to startup blockchain. Exiting ARK Core!

EDIT moderator: moved to #uns-network:support category

Hi,

Can you give more information about the environment of your node?
Architecture, OS, Docker, Docker-compose, node version …

Thank you!

2x2 GHz CPU
2 GB RAM
1 Gbps

PostgreSQL 11 hosted on IBM Cloud (London).

$ uname -a
Linux 4.15.0-88-generic #88-Ubuntu SMP Tue Feb 11 20:11:34 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux


$ docker version
Client:
 Version:           18.09.0
 API version:       1.39
 Go version:        go1.10.4
 Git commit:        4d60db4
 Built:             Wed Nov  7 00:48:57 2018
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.0
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.4
  Git commit:       4d60db4
  Built:            Wed Nov  7 00:16:44 2018
  OS/Arch:          linux/amd64
  Experimental:     false


$ docker-compose version
docker-compose version 1.17.1, build unknown
docker-py version: 2.5.1
CPython version: 2.7.17
OpenSSL version: OpenSSL 1.1.1  11 Sep 2018

Latest uns core node version.
I changed the database location to Frankfurt. Latency seems to be lower. The problem may come from there after all.

It is probably better to host the UNS node and the DB in the same location.

How do you measure the latency between the node and the DB?

A simple ping!

I think this post can be closed.