README.md

[PACKAGE]: ../../raw/master/create_topology_data-0.0.5-202310116.mkp "create_topology_data-0.0.5-202310116.mkp"
# PoC for creating topology_data.json from inventory data

This script creates the topology data file needed for the [Checkmk Exchange Network visualization](https://forum.checkmk.com/u/schnetz) plugin by Andreas Boesl and [schnetz](https://exchange.checkmk.com/u/schnetz). For more information see [Checkmk forum Network Visualization](https://forum.checkmk.com/t/network-visualization/41680). 

The network topology data is read from the Checkmk HW/SW inventory. The necessary inventory data can be created with my CDP/LLDP inventory plugins.

CDP: [CDP inventory plugin](https://thl-cmk.hopto.org/gitlab/checkmk/vendor-independent/inventory/inv_cdp_cache/)\
CDP path-in-inventory: `"networking,cdp_cache"`\
CDP inventory-columns: `"device_id,local_port,device_port"`

LLDP: [LLDP inventory plugin](https://thl-cmk.hopto.org/gitlab/checkmk/vendor-independent/inventory/inv_lldp_cache/)\
LLDP path-in-inventory: `"networking,lldp_cache"`\
LLDP inventory-columns: `"system_name,local_port_num,port_id"`

Usage with CDP (this is the default):
```
~/local/lib/topology_data/create_topology_data.py -s CORE01 -m
```

Usage with LLDP:
```
~/local/lib/topology_data/create_topology_data.py -s CORE01 -m --lldp

```

---
### Download
* [Download latest mkp file][PACKAGE]
                        
---
### Installation

You can install the package by uploading it to your CheckMK site and as site user run 
```
mkp add PAKAGE_NAME.mkp
mkp enable PAKAGE_NAME VERSION
```

In the Enterprise/Free/Cloud edition of CheckMK you can use the GUI to install the package (_Setup_ -> _Extension Packages_ -> _Upload package_)

---
### Want to Contribute?

Nice ;-) Have a look at the [contribution guidelines](CONTRIBUTING.md "Contributing")

---
### Requirements

To use this tool:
- your Checkmk Site must be running at least CMK version 2.2.0p1
- the HW/SW inventory must be enabled
- you need to install/enable an inventory plugin to collect the connection data I.e. my [CDP cache](https://thl-cmk.hopto.org/gitlab/checkmk/vendor-independent/inventory/inv_cdp_cache/) plugin
- the [Network visualization](https://exchange.checkmk.com/p/network-visualization) must be installed

---
### Usage

<details><summary>Help output from create_topology_data.py -h</summary>

```
OMD[build]:~$ local/bin/topology_data/create_topology_data.py -h
usage: create_topology_data.py [-h] [-c INVENTORY_COLUMNS] [-d DATA_SOURCE] [-m] [-o OUTPUT_DIRECTORY] [-p PATH_IN_INVENTORY] [-s SEED_DEVICES [SEED_DEVICES ...]] [-v] [--debug] [--dont-compare] [--keep-domain] [--keep-max KEEP_MAX] [--lldp] [--lowercase] [--min-age MIN_AGE] [--time-format TIME_FORMAT]
                               [--uppercase]

This script creates the topology data file needed for the Checkmk "network_visualization" plugin by Andreas Boesl and schnetz.
For more information see https://forum.checkmk.com/t/network-visualization/41680 and https://exchange.checkmk.com/p/network-visualization

The inventory data could be created with my inventory plugins:
CDP: https://thl-cmk.hopto.org/gitlab/checkmk/vendor-independent/inventory/inv_cdp_cache
LLDP: https://thl-cmk.hopto.org/gitlab/checkmk/vendor-independent/inventory/inv_lldp_cache

Version: 0.0.5-202310116 | Written by: thl-cmk, for more information see: https://thl-cmk.hopto.org

options:
  -h, --help            show this help message and exit
  -c INVENTORY_COLUMNS, --inventory-columns INVENTORY_COLUMNS
                        Columns used from the inventory data. I.e. "device_id,local_port,device_port"
                        NOTE: the columns must be in the order: neighbour, local_port, neighbour_port
  -d DATA_SOURCE, --data-source DATA_SOURCE
                        The source from which the topology data originates. I.e. inv_CDP for CDP data from the inventory. NOTE: right now this only an unused label.
  -m, --make-default    Set the created topology data as default
  -o OUTPUT_DIRECTORY, --output-directory OUTPUT_DIRECTORY
                        Directory name where to save the topology data. I.e.: my_topology. Default is the actual date/time in "--time-format" format.
                        NOTE: the directory is a sub directory under "~/var/topology_data/"
  -p PATH_IN_INVENTORY, --path-in-inventory PATH_IN_INVENTORY
                        Checkmk inventory path to the topology data. I.e. "networking,cdp_cache"
  -s SEED_DEVICES [SEED_DEVICES ...], --seed-devices SEED_DEVICES [SEED_DEVICES ...]
                        List of devices to start the topology discovery from. I.e. Core01 Core02
  -v, --version         Print version of this script and exit
  --debug               Print debug information
  --dont-compare        Do not compare the actual topology data with the default topology data. By default, the actual
                        topology is compared with the default topology. If the data matches, the actual topology is not saved.
                        So, if you run this tool in a cron job, a new topology will be created only if there was a change,
                        unless you use "--dont-compare".
  --keep-domain         Do not remove the domain name from the neighbor name
  --keep-max KEEP_MAX   Number of topologies to keep. The oldest topologies above keep max will be deleted. The last 5 topologies will never be deleted.
  --lldp                Set data source to inv_LLDP, inventory path to "networking,lldp_cache" and columns to "system_name,local_port_num,port_id"
  --lowercase           Change neighbour names to all lower case
  --min-age MIN_AGE     The minimum number of days before a topology is deleted by "--keep-max". Topologies that are not older than 14 days are always kept.
  --time-format TIME_FORMAT
                        Format string to render the time. (default: %Y-%m-%dT%H:%M:%S.%m)
  --uppercase           Change neighbour names to all upper case

Usage:
for CDP (the default):
~/local/bin/network-topology/create_topology_data.py -s Core01 Core02 -m
or
~/local/bin/network-topology/create_topology_data.py -s Core01 Core02 -m -p "networking,cdp_cache" -c "device_id,local_port,device_port" -d inv_CDP

for LLDP:
~/local/bin/network-topology/create_topology_data.py -s Core01 Core02 -m --lldp
or
~/local/bin/network-topology/create_topology_data.py -s Core01 Core02 -m -p "networking,lldp_cache" -c "system_name,local_port_num,port_id" -d inv_LLDP

```

</details>

---
### Limitations

- local/neighbour interface names have to match the service names in Checkmk

---
### FAQ

<details><summary>If i open the Network visualization, there is no default topology</summary>

\
To make the topology created by this tool the default use the option `-m` / `--make-default`

```
$ ~/local/bin/topology_data/create_topology_data.py -s CORE01 -m
```

</details>

<details><summary>Can i automate the creation of the topology data</summary>

\
Yes. Add a cron job for the site user in `~etc/check_mk/conf.d/`. The tool can than run in regular intervals. By default it will only create a new topology if the new topology differs from the default topology. So by running
```
$ ~/local/bin/topology_data/create_topology_data.py -s CORE01 -m
```
you get each time a new default topology if ther was a change. The name for the new topology is by default the creation date and time.

</details>

<details><summary>Can I limit the number of topology changes saved</summary>

\
Yes. If you use the option --keep-max <number of topologies to keep> the tool will delete all topologies over this number. I.e.

```
$ ~/local/bin/topology_data/create_topology_data.py -s CORE01 --keep-max 10
```
will delete the oldest topologies until there only 10 topologies left.

**Note**: the default topology will never be deleted.
**Note**: the minimal `--keep-max` is _1_
</details>

<details><summary>I will not delete topologies from the last X days</summary>

No problem. Just use the option `--min-age` togeher with `--keep-max`. I.e. 

```
$ ~/local/bin/topology_data/create_topology_data.py -s CORE01 --keep-max 10 --min-age 30
``` 

will only delete a topology if it older than 30 days.

**Note**: the minimum value for `--min-age` is _1_ day.
</details>

<details><summary>Can the topology data creation be customized</summary>

\
Yes. You can customize the topology data creation by modifying the  `~/local/bin/topology_data/create_topology_data.toml` file. Use `~/local/bin/topology_data/create_topology_data.toml_clean` as a template.

**Note**: This file uses [Tom's Obvious Minimal Language](https://toml.io/en/)

</details>

<details><summary>My Neighbor names from the inventory and checkmk host names do not match</summary>

\
If the neighbor names from the inventory and the checkmk host names do not match, the topology does not work. To fix this, add a mapping between the neighbor name and the checkmk host name to the **HOST_MAP** in the `create_topology_data.toml` file. I.e.

```
# map inventory neighbour name to Checkmk host name
[HOST_MAP]
inventory_neighbour1 = "cmk_host1"
inventory_neighbour2 = "cmk_host2"
inventory_neighbour3 = "cmk_host3"
```
**NOTE**: The script will automatically remove the domain name form the neighbour names. Use the option `--keep-domain` if you want to keep the domain name.

</details>

<details><summary>In my the inventory are invalid neighbor names</summary>

\
If you have invalid neighbor names in the inventory (i.e. "not advertised"), you can drop them by adding them to the **DROP_HOSTS** list in the `create_topology_data.toml` file. I.e..

```
# drop neighbours with invalid names
DROP_HOSTS = [
    "not advertised",
    "a nother invalid name",
]
```

</details>


<details><summary>Not all connections are included in the inventory</summary>

\
If your inventory data does not contain all the required topology information, you can add static connections in the `create_topology_data.toml` file. This may be the case if not all of your devices use CDP/LLDP. The connections defined in **STATIC_CONNECTIONS** are added to the topology from host to neighbor and vice versa. I.e..

```
# user defined static connections
# connections will be added from host to neighbour and in reverese
STATIC_CONNECTIONS = [
    ["host1", "local-port1", "neighbour-port1", "neighbour1", "label"],
    ["host1", "local-port2", "neighbour-port2", "neighbour2", "label"],
]
```

</details>

<details><summary>Can I use the tool with other inventory data plugins</summary>

\
Yes. If you don\`t use my plugins you can still use this tool to create your topology data, for this you need the path to your topology data in the inventory. For example for my CDP plugin this is _Networking_ -> _CDP Cache_ internally known as `networking` -> `cdp_cache`. The data must form a table in the inventory that contains at least the three columns for the neighbor name, neighbor port, and local port. For these columns you need the internal names. I.e. for my CDP plugin the columns _Neighbor device_, _Neighbor port_ and _Local port_ are known internally as `device_id`, `local_port` and `device_port`. With this information, you can run the tool like this

```
~/local/bin/network-topology/create_topology_data.py -s Core01 -p "networking,cdp_cache" -c "device_id,local_port,device_port"
```

Wehre `-p` is the path to your inventory data and `-c` are the columns used in the order _neighbor_, _local port_, _neighbor port_.

</details>

---
### Sample Output

Sample output

```
OMD[build]:~$ ~/local/bin/topology_data/create_topology_data.py -s CORE01
Start time: 2023-10-16T21:44:53.10
Devices added: 66, source inv_CDP
time taken: 1.035935568/s
End time: 2023-10-16T21:44:54.10

```

Sample network topology

![sample output details](/img/sample-details.png?raw=true "sample output details")