Update README.md

65e328cd · thl-cmk · 5ea9aed7 · 65e328cd
Commit 65e328cd authored 4 months ago by thl-cmk
--- a/README.md
+++ b/README.md
-[PACKAGE]: ../../raw/master/mkp/nvdct-0.9.2-20241116.mkp "nvdct-0.9.2-20241116.mkp"
+[PACKAGE]: ../../raw/master/mkp/nvdct-0.9.2-20241117.mkp "nvdct-0.9.2-20241117.mkp"
 # Network Visualization Data Creation Tool (NVDCT)

 This script creates the topology data file needed for the [Checkmk Exchange Network visualization](https://exchange.checkmk.com/p/network-visualization) plugin.\
@@ -79,27 +79,23 @@ and let Checkmk run the HW/SW inventory for each device at least once

 Usage with CDP and STATIC (this is the default):
 ```
-~/local/bin/nvdct/nvdct.py -s Core01 -d
+~/local/bin/nvdct/nvdct.py -u ~/local/bin/conf/my_nvdct.toml -d
 ```
 Usage with LLDP and STATIC:
 ```
-~/local/bin/nvdct/nvdct.py -s Core01 -d -l LLDP STATIC
+~/local/bin/nvdct/nvdct.py -u ~/local/bin/conf/my_nvdct.toml -d -l LLDP STATIC
 ```

 <details><summary>Help output from nvdct.py --help</summary>

 ```
 OMD[build]:~$ ~/local/bin/nvdct/nvdct.py -h
-usage: nvdct.py [-h] [-b {LIVESTATUS,MULTISITE,RESTAPI}] [-d] [-o OUTPUT_DIRECTORY]
-                [-s SEED_DEVICES [SEED_DEVICES ...]] [-p PREFIX]
-                [-l {CDP,CUSTOM,LLDP,STATIC,L3v4} [{CDP,CUSTOM,LLDP,STATIC,L3v4} ...]]
-                [-u USER_DATA_FILE] [-v] [--api-port API_PORT] [--case {LOWER,UPPER}]
-                [--check-user-data-only] [--log-file LOG_FILE]
-                [--log-level {CRITICAL,FATAL,ERROR,WARNING,INFO,DEBUG,OFF}] [--log-to-stdout]
-                [--dont-compare] [--filter-customers {INCLUDE,EXCLUDE}]
-                [--filter-sites {INCLUDE,EXCLUDE}] [--keep-domain] [--keep KEEP]
-                [--min-age MIN_AGE] [--pre-fetch] [--quiet] [--skip-l3-if] [--skip-l3-ip]
-                [--time-format TIME_FORMAT]
+usage: nvdct.py [-h] [-b {LIVESTATUS,MULTISITE,RESTAPI}] [-d] [-o OUTPUT_DIRECTORY] [-p PREFIX]
+                [-l {CDP,CUSTOM,LLDP,STATIC,L3v4} [{CDP,CUSTOM,LLDP,STATIC,L3v4} ...]] [-u USER_DATA_FILE] [-v]
+                [--api-port API_PORT] [--case {LOWER,UPPER}] [--check-user-data-only] [--log-file LOG_FILE]
+                [--log-level {CRITICAL,FATAL,ERROR,WARNING,INFO,DEBUG,OFF}] [--log-to-stdout] [--dont-compare]
+                [--filter-customers {INCLUDE,EXCLUDE}] [--filter-sites {INCLUDE,EXCLUDE}] [--remove-domain] [--keep KEEP]
+                [--min-age MIN_AGE] [--pre-fetch] [--quiet] [--skip-l3-if] [--skip-l3-ip] [--time-format TIME_FORMAT]

 This script creates the topology data file needed for the Checkmk "network_visualization"
 plugin by Andreas Boesl and schnetz. For more information see
@@ -111,7 +107,7 @@ CDP: https://thl-cmk.hopto.org/gitlab/checkmk/vendor-independent/inventory/inv_c
 LLDP: https://thl-cmk.hopto.org/gitlab/checkmk/vendor-independent/inventory/inv_lldp_cache
 L3v4: https://thl-cmk.hopto.org/gitlab/checkmk/vendor-independent/inventory/inv_ipv4_addresses

-Version: 0.9.0-20240923 | Written by: thl-cmk
+Version: 0.9.2-20241117 | Written by: thl-cmk
 for more information see: https://thl-cmk.hopto.org/gitlab/checkmk/vendor-independent/nvdct

 options:
@@ -128,19 +124,16 @@ options:
                        I.e.: my_topology. Default is the actual date/time
                        in "--time-format" format.
                        NOTE: the directory is a sub directory under "~/var/check_mk/topology/data/"
-  -s SEED_DEVICES [SEED_DEVICES ...], --seed-devices SEED_DEVICES [SEED_DEVICES ...]
-                        List of devices to start the topology discovery from.
-                        I.e. Core01 Core02
  -p PREFIX, --prefix PREFIX
                        Prepends each host with the prefix. (Needs testing)
  -l {CDP,CUSTOM,LLDP,STATIC,L3v4} [{CDP,CUSTOM,LLDP,STATIC,L3v4} ...], --layers {CDP,CUSTOM,LLDP,STATIC,L3v4} [{CDP,CUSTOM,LLDP,STATIC,L3v4} ...]
                         - CDP  : needs inv_cdp_cache package at least in version 0.7.1-20240320
                         - LLDP : needs inv_lldp_cache package at least in version 0.9.3-20240320
                         - L3v4 : needs inv_ipv4_addresses package at least in version 0.0.3-20240320
-                                  adds, layer 3 topology fpr IPv4 
+                                  adds, layer 3 topology fpr IPv4
  -u USER_DATA_FILE, --user-data-file USER_DATA_FILE
                        Set the name uf the user provided data file
-                        Default is ~/local/bin/nvdct/conf/conf/nvdct.toml
+                        Default is ~/local/bin/nvdct/conf/nvdct.toml
  -v, --version         Print version of this script and exit
  --api-port API_PORT   TCP Port to access the REST API. Default is 80. NVDCT will try to automatically
                        detect the site apache port.
@@ -162,7 +155,7 @@ options:
  --filter-sites {INCLUDE,EXCLUDE}
                        INCLUDE/EXCLUDE site list from config file.
                        Note: MULTISITE backend only.
-  --keep-domain         Do not remove the domain name from the neighbor name
+  --remove-domain       Remove the domain name from the neighbor name
  --keep KEEP           Number of topologies to keep. The oldest topologies above keep
                        max will be deleted.
                        NOTE: The default topologies will be always kept.
@@ -182,7 +175,7 @@ Exit codes:
 4 - Automation secret not found

 Usage:
-~/local/bin/nvdct/nvdct.py -s Core01 Core02 -d
+~/local/bin/nvdct/nvdct.py -u ~/local/bin/nvdct/conf/nvdct.toml

 OMD[build]:~$ 
 ```
@@ -192,7 +185,6 @@ OMD[build]:~$
 ---
 ### Known limitations

- the backend `FILESYSTEM` can only acces the local autochheck files, this might lead in ditributed environments to incomplete data, use `MUTLISITE` or `RESTAPI` instead

 ---
 ### FAQ
@@ -203,7 +195,7 @@ OMD[build]:~$
 To make the topology created by this tool the default use the option `-d` / `--default`

 ```
-$ ~/local/bin/nvdct/nvdct.py -s Core01 -d
+$ ~/local/bin/nvdct/nvdct.py -u ~/local/bin/conf/my_nvdct.toml -d
 ```

 </details>
@@ -214,7 +206,7 @@ $ ~/local/bin/nvdct/nvdct.py -s Core01 -d
 Yes. Add a cron job for the site user in `~/etc/cron.d`. I.e. to create a topology every hour create the file `nvdct` in `~/etc/cron.d`

 ```
-0 * * * * $OMD_ROOT/local/bin/nvdct/nvdct.py -s Core01 -d --keep 10 --min-age 14
+0 * * * * $OMD_ROOT/local/bin/nvdct/nvdct.py $OMD_ROOT/local/bin/nvdct/conf_my_nvdct.toml -d --keep 10 --min-age 14
 ```

 This will run the tool every hour and create a new topology if there was a changee. It keeps 10 topologies and deletes old topologies only if they are older than 14 days.
@@ -235,7 +227,7 @@ With crontab -l you get the list of active cron jobs
 OMD[build]:~$ crontab -l | grep -A 3 -B 1  nvdct 
 # ------------------------------------------------------------
 # /omd/sites/build/etc/cron.d/nvdct
-0 * * * * $OMD_ROOT/local/bin/nvdct/nvdct.py -s Core01 d --keep 10
+0 * * * * $OMD_ROOT/local/bin/nvdct/nvdct.py $OMD_ROOT/local/bin/nvdct/conf_my_nvdct.toml d --keep 10

 # ------------------------------------------------------------
 ```
@@ -250,7 +242,7 @@ OMD[build]:~$ crontab -l | grep -A 3 -B 1  nvdct
 Yes. If you use the option --keep <number of topologies to keep> the tool will delete all topologies over this number. I.e.

 ```
-$ ~/local/bin/nvdct/nvdct.py -s Core01 --keep 10
+$ ~/local/bin/nvdct/nvdct.py -u ~/local/bin/conf/my_nvdct.toml --keep 10
 ```
 will delete the oldest topologies until there only 10 topologies left.

@@ -262,7 +254,7 @@ will delete the oldest topologies until there only 10 topologies left.
 \
 No problem. Just use the option `--min-age` togeher with `--keep`. I.e. 
 ```
-$ ~/local/bin/nvdct/nvdct.py -s Core01 --keep 10 --min-age 30
+$ ~/local/bin/nvdct/nvdct.py -u ~/local/bin/conf/my_nvdct.toml --keep 10 --min-age 30
 ``` 
 will only delete a topology if it is older than 30 days.
 </details>
@@ -270,12 +262,12 @@ will only delete a topology if it is older than 30 days.
 <details><summary>Can the topology data creation be customized</summary>

 \
-Yes. You can customize the topology data creation by modifying the  `~/local/bin/nvdct/conf/nvdct.toml` file. Or provide your own data file (see Option -u, --user-data-file)
+Yes you can and you sould. Create a copy of the default `~/local/bin/nvdct/conf/nvdct.toml` file. Or provide your own data file (see Option -u, --user-data-file).

 **Note**: This file uses [Tom's Obvious Minimal Language](https://toml.io/en/)

 ```
-~/local/bin/nvdct/nvdct.py -u ~/local/bin/nvdct/conf/my.toml
+~/local/bin/nvdct/nvdct.py -u ~/local/bin/nvdct/conf/my_nvdct.toml
 ```

 </details>
@@ -287,28 +279,35 @@ If the neighbor names from the inventory and the checkmk host names do not match

 ```
 # map inventory neighbour name to Checkmk host name
-[HOST_MAP]
+[L2_HOST_MAP]
 inventory_neighbour1 = "cmk_host1"
 inventory_neighbour2 = "cmk_host2"
 inventory_neighbour3 = "cmk_host3"
 ```
-**NOTE**: The script will automatically remove the domain name form the neighbour names. Use the option `--keep-domain` if you want to keep the domain name.

 </details>

 <details><summary>In my inventory are invalid neighbor names</summary>

 \
-If you have invalid neighbor names in the inventory (i.e. "not advertised"), you can drop them by adding them to the **DROP_HOSTS** list in the data file (by default `nvdct.toml`). I.e..
+If you have invalid neighbor names in the inventory (i.e. "not advertised"), you can drop them by adding them to the **L2_DROP_HOSTS** list in the data file (by default `nvdct.toml`). I.e..

 ```
 # drop neighbours with invalid names
-DROP_HOSTS = [
+L2_DROP_HOSTS = [
    "not advertised",
    "a nother invalid name",
 ]
 ```

+If you need a more advanced option to remove neighbours from the topology, you can also use the `L2_NEIGHBOUR_REPLACE_REGEX` section.
+Ie. to remove all neighbours advertised as MAC-address only use:
+
+```
+[L2_NEIGHBOUR_REPLACE_REGEX]
+"^(([0-9a-fA-F]){2}[:.-]?){5}([0-9a-fA-F]){2}$" = ""
+```
+
 </details>


@@ -341,7 +340,7 @@ CUSTOM_LAYERS = [
 ```
 then run the tool with the option `-l CUSTOM`
 ```
-~/local/bin/network-topology/nvdct.py -s Core01 -l CUSTOM
+~/local/bin/network-topology/nvdct.py -u ~/local/bin/conf/my_nvdct.toml -l CUSTOM
 ```

 </details>
@@ -362,30 +361,29 @@ First check your HW/SW inventory if it not contains the data needed to create th
                        Sets the loglevel. The default is "WARN"
  * --log-to-stdout       Send log to stdout.

+For more troubleshooting see [TROUBLESHOOTING.md](/TROUBLESHOOTING.md)
+
 </details>

 ---
 ### Sample Output

-Sample output of NVDCT.
-
 ```
-OMD[build]:~$ ~/local/bin/nvdct/nvdct.py  -u ~/local/bin/nvdct/conf/my.toml 
+OMD[build]:~$ ~/local/bin/nvdct/nvdct.py -u ~/local/bin/nvdct/conf/lrm.toml -d

 Network Visualisation Data Creation Tool (NVDCT)
-by thl-cmk[at]outlook[dot]com, version 0.8.7-20240430
+by thl-cmk[at]outlook[dot]com, version 0.9.2-20241117
 see https://thl-cmk.hopto.org/gitlab/checkmk/vendor-independent/nvdct

-Start time....: 2024-04-30T14:17:02.04
-Devices added.: 16, source lldp
-Devices added.: 72, source cdp
-Devices added.: 29, source STATIC
-Devices added.: 59, source L3v4
-Time taken....: 3.225333135/s
-End time......: 2024-04-30T14:17:05.04
+Start time....: 2024-11-17T17:20:54.11
+Source lldp...: Devices/Objects/Connections added 8/18/15
+Source cdp....: Devices/Objects/Connections added 28/144/174
+Source static.: Devices/Objects/Connections added 29/85/84
+Source l3v4...: Devices/Objects/Connections added 2/141/140
+Time taken....: 13.360647338/s
+End time......: 2024-11-17T17:21:08.11

 OMD[build]:~$ 
-
 ```

 Sample network topology CDP/LLDP