CM Handle State: Watchdog-process that syncs 'ADVISED' CM Handles

Team discussion on approaches:

• consider kubernetes job Pros
  1. pro: schedule configurable
  2. pro: possible retry mechanism
  3. multiple instances ?
  4. pro: monitoring
  5. Con: tight to kubernetes
• Java/Spring schedule (This is what is being implemented)
  1. pro: schedule using springboot parameters
  2. Retry mechanism?
  3. pro: more control
  4. pro: instances follows NCMP instances
  5. pro: simpler
  6. pro: independent of kubernetes
  7. pro: function part of NCMP feature

steps, each NCMP will have 1 watchdog process that:

1. Wake up every minute(configurable) using springboot schedule with a fixed-delay
2. query top 1 where state='ADVISED'
3. execute model-sync (on watch dog thread)
   1. If model sync fails, cm handle state = "LOCKED"
4. set state to 'READY'  (consider state engine, see ref below)
5. Go back to step 2 and repeat

References:

1. https://wiki.onap.org/display/DW/CPS-799+Spike%3A+Define+states+and+state+handling+for+CM+handle
2. https://wiki.onap.org/display/DW/CPS-872+CM+Handle+State%3A+define+and+agree+new+dmi-registry+yang+model+supporting+States
3. https://www.baeldung.com/java-enum-simple-state-machine
4. https://reflectoring.io/spring-scheduler/

Notes

1. Refactor existing model sync code into separate package and classes so it can be used by both sync(current) service and new watchdog process (subtask, separate commit!) Has been done as part of https://jira.onap.org/browse/CPS-950
2. (Re-)use model-sync code to separate (watchdog) process which regularly checks DB for 'ADVISED' CM handles and will attempt to perform modelsync on those
   1. Agree timing with team
   2. Single CM-Handle to be worked on at a time.
   3. Use DB Locking (timestamp field) to prevent multiple instances attempting to sync the same CM Handle (Working on single cm-handle at a time, multiple instances shouldn't sync together)
   4. Update State to READY when model sync completed (and remove timestamp)
   5. Update State to LOCKED_MISBEHAVING when sync fails (use timeout - decision hasn't been on timeout duration) 
3. Consider what to do when still busy with previous sync (should NOT start new sync, can only sync one cm-handle at a time)
4. Ensure central (single class) handler for state-transitions 

Testing

• code is not 'really active' will need to be triggered by setting state on a cm-handle using DB interaction

A/C

1. Demo watchdog sync using 'sql-trigger' when change is detected on a node, or time based mechanism which runs on a yet to be determined schedule
   1. success
   2. fail
2. Demo timer configuration for watchdog
3. (snapshot) documentation of relevant configuration