Newer
Older
[PACKAGE]: ../../raw/master/mkp/nvdct-0.8.7-20240430.mkp "nvdct-0.8.7-20240430.mkp"
This script creates the topology data file needed for the [Checkmk Exchange Network visualization](https://exchange.checkmk.com/p/network-visualization) plugin by Andreas Boesl and [schnetz](https://exchange.checkmk.com/u/schnetz).\
For more information see [Checkmk forum Network Visualization](https://forum.checkmk.com/t/network-visualization/41680).
- Create Layer 2 (CDP/LLDP) topology data
- Create Layer 3 (IPv4) topology data -> CMK2.3.0 only
- Highlight connetion issues (Speed/Duplex/Half duplex/Native VLAN)
- Reading connection data from the checkmk HW/SW inventory.
- Works in standalone and distributed monitoring environments
- Add custom connections (STATIC) for connections that are not in the inventory
- Optimized for my [CDP](https://thl-cmk.hopto.org/gitlab/checkmk/vendor-independent/inventory/inv_cdp_cache/), [LLDP](https://thl-cmk.hopto.org/gitlab/checkmk/vendor-independent/inventory/inv_lldp_cache/) and [IPv4](https://thl-cmk.hopto.org/gitlab/checkmk/vendor-independent/inventory/inv_ipv4_addresses) inventory plugins
- Merge CDP, LLDP, STATIC topologies (CMK 2.2.0 only)
The necessary HW/SW inventory data can be created with my CDP/LLDP inventory plugins.
- [CDP inventory plugin](https://thl-cmk.hopto.org/gitlab/checkmk/vendor-independent/inventory/inv_cdp_cache/)
- [LLDP inventory plugin](https://thl-cmk.hopto.org/gitlab/checkmk/vendor-independent/inventory/inv_lldp_cache/)
- [IPv4 inventory plugin](https://thl-cmk.hopto.org/gitlab/checkmk/vendor-independent/inventory/inv_ipv4_addresses)
- [Interface name inventory plugin (optional but recommended)](https://thl-cmk.hopto.org/gitlab/checkmk/vendor-independent/inventory/inv_ifname)
Andreas Boesl and schnetz for the Network visualization plugin\
detlev[dot]gerke[at]siegfried[dot]ch, who was my beta tester ;-)
**Note**: this is sill work in progress, so **before** updating read the [CHANGELOG](/CHANGELOG "CHANGELOG") please.
**Note**: before updating to a newer version check the [CHANGELOG](/CHANGELOG "CHANGELOG"). There could be always incompatible changes.
---
### Installation
You can install the package by uploading it to your CheckMK site and as site user run
```
mkp add PAKAGE_NAME.mkp
mkp enable PAKAGE_NAME VERSION
```
In the non RAW editions of CheckMK you can use the GUI to install the package (_Setup_ -> _Extension Packages_ -> _Upload package_)
---
### Want to Contribute?
Nice ;-) Have a look at the [contribution guidelines](CONTRIBUTING.md "Contributing")
---
### Requirements
To use this tool:
- your Checkmk Site must be running at least CMK version 2.2.0p1
- the HW/SW inventory must be enabled
- you need to install/enable an inventory plugin to collect the connection data I.e. my [CDP cache](https://thl-cmk.hopto.org/gitlab/checkmk/vendor-independent/inventory/inv_cdp_cache/) or [LLDP cache](https://thl-cmk.hopto.org/gitlab/checkmk/vendor-independent/inventory/inv_lldp_cache/) plugins
- check if the Checkmk HW/SW inventory finds the required data (CDP/LLDP neighbors)
- the [Network visualization plugin](https://exchange.checkmk.com/p/network-visualization) must be installed (CMK 2.2.0 only, for CMK2.3.0 this is bildin)
**Note**: if you are using my CDP/LLDP/IPv4 plugins update at least to version
- 0.7.6-20240414 for CDP
- 0.9.7-20240414 for LLDP
- 0.0.4-20240407 for IPv4
and let Checkmk run the HW/SW inventory for each device at least once
usage: nvdct.py [-h] [-b {FILESYSTEM,LIVESTATUS,MULTISITE,RESTAPI}] [-d] [-o OUTPUT_DIRECTORY]
[-s SEED_DEVICES [SEED_DEVICES ...]] [-p PREFIX]
[-l {CDP,CUSTOM,LLDP,STATIC,L3v4} [{CDP,CUSTOM,LLDP,STATIC,L3v4} ...]]
[-u USER_DATA_FILE] [-v] [--api-port API_PORT] [--check-user-data-only]
[--log-file LOG_FILE] [--log-level {CRITICAL,FATAL,ERROR,WARNING,INFO,DEBUG,OFF}]
[--log-to-stdout] [--dont-compare] [--keep-domain] [--keep KEEP] [--lowercase]
[--min-age MIN_AGE] [--new-format] [--pre-fetch] [--quiet] [--skip-l3-if]
[--skip-l3-ip] [--time-format TIME_FORMAT] [--uppercase]
This script creates the topology data file needed for the Checkmk "network_visualization"
plugin by Andreas Boesl and schnetz. For more information see
the announcement from schnetz: https://forum.checkmk.com/t/network-visualization/41680
and the plugin on the Exchange: https://exchange.checkmk.com/p/network-visualization .
The required inventory data can be created with my inventory plugins:
CDP: https://thl-cmk.hopto.org/gitlab/checkmk/vendor-independent/inventory/inv_cdp_cache
LLDP: https://thl-cmk.hopto.org/gitlab/checkmk/vendor-independent/inventory/inv_lldp_cache
L3v4: https://thl-cmk.hopto.org/gitlab/checkmk/vendor-independent/inventory/inv_ipv4_addresses
for more information see: https://thl-cmk.hopto.org/gitlab/checkmk/vendor-independent/nvdct
-b {FILESYSTEM,LIVESTATUS,MULTISITE,RESTAPI}, --backend {FILESYSTEM,LIVESTATUS,MULTISITE,RESTAPI}
- FILESYSTEM : fetches the data directly form the inventory files (deprecatred)
- LIVESTATUS : fetches data via local Livestatus (local site only)
- MULTISITE : like LIVESTATUS but for distributed environments (default)
- RESTAPI : uses the CMK REST API.
-d, --default Set the created topology data as default. Will be created automatically
if it doesn't exists.
Directory name where to save the topology data.
I.e.: my_topology. Default is the actual date/time
in "--time-format" format.
NOTE: the directory is a sub directory under "~/var/topology_data/" (CMK2.2.0)
For CMK 2.3.0 the path is under "~/var/check_mk/topology/data/".
-s SEED_DEVICES [SEED_DEVICES ...], --seed-devices SEED_DEVICES [SEED_DEVICES ...]
List of devices to start the topology discovery from.
I.e. Core01 Core02
-p PREFIX, --prefix PREFIX
Prepends each host with the prefix. (Needs testing)
-l {CDP,CUSTOM,LLDP,STATIC,L3v4} [{CDP,CUSTOM,LLDP,STATIC,L3v4} ...], --layers {CDP,CUSTOM,LLDP,STATIC,L3v4} [{CDP,CUSTOM,LLDP,STATIC,L3v4} ...]
will be merged automatically (CMK2.2 only).
- CDP : needs inv_cdp_cache package at least in version 0.7.1-20240320
- LLDP : needs inv_lldp_cache package at least in version 0.9.3-20240320
- L3v4 : needs inv_ipv4_addresses package at least in version 0.0.3-20240320
adds, layer 3 topology fpr IPv4 (CMK 2.3.0 only)
-u USER_DATA_FILE, --user-data-file USER_DATA_FILE
Set the name uf the user provided data file
--api-port API_PORT TCP Port to access the REST API. Default is 80. NVDCT will try to automatically
detect the site apache port.
Only tries to read/parse the user data from nvdct.toml and exits.
--log-file LOG_FILE Set the log file. Default is ~/var/log/nvdct.log
--log-level {CRITICAL,FATAL,ERROR,WARNING,INFO,DEBUG,OFF}
Sets the log level. The default is "WARNING"
--dont-compare Do not compare the actual topology data with the default topology
data. By default, the actual topology is compared with the default
topology. If the data matches, the actual topology is not saved.
So, if you run this tool in a cron job, a new topology will be
created only if there was a change, unless you use "--dont-compare".
--keep-domain Do not remove the domain name from the neighbor name
--keep KEEP Number of topologies to keep. The oldest topologies above keep
--min-age MIN_AGE The minimum number of days before a topology is deleted by "--keep".
--new-format Save data in new format. Use for CMK 2.3.x
NVDCT will try to automatically detect the correct format. (deprecated)
--pre-fetch Try to fetch host data, with less API calls. Can improve RESTAPI backend
performance
--skip-l3-if Skip interface in layer 3 topology
--skip-l3-ip Skip ip-address in layer 3 topology
--time-format TIME_FORMAT
Format string to render the time. (default: %Y-%m-%dT%H:%M:%S.%m)
--uppercase Change neighbour names to all upper case
Exit codes:
0 - No error
1 - Bad options list
3 - Backend not implemented
4 - Automation secret not found
- the backend `FILESYSTEM` can only acces the local autochheck files, this might lead in ditributed environments to incomplete data, use `MUTLISITE` or `RESTAPI` instead
<details><summary>If i open the Network visualization, there is no default topology</summary>
\
To make the topology created by this tool the default use the option `-d` / `--default`
```
</details>
<details><summary>Can i automate the creation of the topology data</summary>
\
Yes. Add a cron job for the site user in `~/etc/cron.d`. I.e. to create a topology every hour create the file `nvdct` in `~/etc/cron.d`
0 * * * * $OMD_ROOT/local/bin/nvdct/nvdct.py -s Core01 -d --keep 10 --min-age 14
```
This will run the tool every hour and create a new topology if there was a changee. It keeps 10 topologies and deletes old topologies only if they are older than 14 days.
Don't forget to restart the crontab service after creating the cron job.
```
OMD[build]:~/etc/cron.d$ omd restart crontab
Temporary filesystem already mounted
Removing Crontab...OK
Initializing Crontab...OK
OMD[build]:~/etc/cron.d$
With crontab -l you get the list of active cron jobs
```
# ------------------------------------------------------------
# /omd/sites/build/etc/cron.d/nvdct
0 * * * * $OMD_ROOT/local/bin/nvdct/nvdct.py -s Core01 d --keep 10
# ------------------------------------------------------------
**Note**: By default the tool will only create a new topology if the new topology differs from the default topology. The name for the new topology is by default the creation date and time.
<details><summary>Can I limit the number of topologies saved</summary>
Yes. If you use the option --keep <number of topologies to keep> the tool will delete all topologies over this number. I.e.
```
will delete the oldest topologies until there only 10 topologies left.
**Note**: the default topology will never be deleted.
</details>
<details><summary>I will not delete topologies from the last X days</summary>
No problem. Just use the option `--min-age` togeher with `--keep`. I.e.
$ ~/local/bin/nvdct/nvdct.py -s Core01 --keep 10 --min-age 30
<details><summary>Can the topology data creation be customized</summary>
\
Yes. You can customize the topology data creation by modifying the `~/local/bin/nvdct/nvdct.toml` file. Or provide your own data file (see Option -u, --user-data-file)
**Note**: This file uses [Tom's Obvious Minimal Language](https://toml.io/en/)
~/local/bin/topology_data/nvdct.py -u ~/local/bin/topology_data/my.toml
<details><summary>My Neighbor names from the inventory and checkmk host names do not match</summary>
If the neighbor names from the inventory and the checkmk host names do not match, the topology does not work. To fix this, add a mapping between the neighbor name and the checkmk host name to the **HOST_MAP** in the data file (by default `nvdct.toml`). I.e.
# map inventory neighbour name to Checkmk host name
[HOST_MAP]
inventory_neighbour1 = "cmk_host1"
inventory_neighbour2 = "cmk_host2"
inventory_neighbour3 = "cmk_host3"
**NOTE**: The script will automatically remove the domain name form the neighbour names. Use the option `--keep-domain` if you want to keep the domain name.
<details><summary>In my inventory are invalid neighbor names</summary>
If you have invalid neighbor names in the inventory (i.e. "not advertised"), you can drop them by adding them to the **DROP_HOSTS** list in the data file (by default `nvdct.toml`). I.e..
"not advertised",
"a nother invalid name",
]
```
</details>
<details><summary>Not all connections are included in the inventory</summary>
\
If your inventory data does not contain all the required topology information, you can add static connections in the data file (by default `nvdct.toml`). This may be the case if not all of your devices use CDP/LLDP. The connections defined in **STATIC_CONNECTIONS** are added to the topology from host to neighbor and vice versa. I.e..
```
# user defined static connections
# connections will be added from host to neighbour and in reverese
STATIC_CONNECTIONS = [
["host1", "local-port1", "neighbour-port1", "neighbour1", "label"],
["host1", "local-port2", "neighbour-port2", "neighbour2", "label"],
<details><summary>Can I use the tool with other inventory data plugins</summary>
\
Yes. If you don\`t use my plugins you can still use this tool to create your topology data, for this you need the path to your topology data in the inventory. For example for my CDP plugin this is _Networking_ -> _CDP Cache_ -> _CDP neighbours_ internally known as `networking` -> `cdp_cache` -> `neighbours`. The data must form a table in the inventory that contains at least the three columns for the neighbor name, neighbor port, and local port. For these columns you need the internal names. I.e. for my CDP plugin the columns _Neighbor name_, _Neighbor port_ and _Local port_ are known internally as `neighbour_name`, `local_port` and `neighbour_port`. With this information, you can edit the CUSTOM_LAYERS section in the user data file like this
# optional custom layers use option -l/--layers CUSTOM to include this layers
CUSTOM_LAYERS = [
{ path = "networking,lldp_cache.neighbours", columns = "neighbour_name,local_port_num,neighbour_port", label = "custom_LLDP" },
{ path = "networking,cdp_cache.neighbours", columns = "neighbour_name,local_port,neighbour_port", label = "custom_CDP" },
]
```
then run the tool with the option `-l CUSTOM`
```
<details><summary>How are port names mapped to interface service names?</summary>
\
When mapping Layer 2 port names, the first step is to check whether the port name is in the host's list of interface items; if it is, the port name is used; if it is not, the port name is searched for in the inventory interface table. If the interface table is found it tries to find the correct item by comparing the list of items with the fields index, description, alias, physical address (MAC) and (if available) [name](https://thl-cmk.hopto.org/gitlab/checkmk/vendor-independent/inventory/inv_ifname). If we still can't find the item, we pad the index and try the search again.
So most of the time we should find the correct interface service for a port.
</details>
First check your HW/SW inventory if it not contains the data needed to create the network topology data you need first to troubleshoot the inventory. If the inventory is ok you can use the buildin logging to troubleshoot the data creation. By default the tool will create the log file `~/var/log/nvdct.log`. The default log level set is to **WARNING**, so most information for everything thats not working should go to the log. If you want to change the logging see the options:
* --log-file LOG_FILE Set the log file. Default is ~/var/log/nvdct.log
* --log-level {CRITICAL,ERROR,WARN,INFO,DEBUG}
Sets the loglevel. The default is "WARN"
* --log-to-stdout Send log to stdout.
</details>
OMD[build]:~$ ~/local/bin/nvdct/nvdct.py -u ~/local/bin/nvdct/nvdct.toml
see https://thl-cmk.hopto.org/gitlab/checkmk/vendor-independent/nvdct
Start time....: 2024-04-30T14:17:02.04
Devices added.: 16, source lldp
Devices added.: 72, source cdp
Devices added.: 29, source STATIC
Devices added.: 59, source L3v4
Time taken....: 3.225333135/s
End time......: 2024-04-30T14:17:05.04
Sample network topology CDP/LLDP CMK 2.2.0
Sample network topology CDP/LLDP CMK 2.2.0
Sample network topology IPv4 CMK 2.3.0
