Compliance Tests
Tests are the main thing Validity was created for.
In general, Tests are Python expressions that are bound to Selectors. Inside test expression you can use:
- Python built-ins like
str()
,round()
,map()
and so on device
variable. This variable represents NetBoxDevice
instance plus some additional properties: (see below)jq.all()
andjq.first()
functions. These functions allow you to use the full power of JQ expressions- additional functions and classes from bound Name Sets
Test Expressions
You can create tests using Python expression syntax. It means that you can't write random python code inside the test, the code you've written must form an expression.
Some examples:
Expression | NOT an expression |
---|---|
a == 10 + 20 |
a = 10 + 20 |
[i * 2 for i in range(5)] |
for i in range(5): print(i) |
'ODD' if x % 2 else 'EVEN' |
if x % 2: res = 'ODD' |
If you're still unsure, you always can use python built-in function eval()
to check your expression. eval()
works with expressions only and raises SyntaxError
on all other syntax.
Note
Your expression SHOULD return boolean value as a result. If it doesn't, the result will be converted to boolean using Python built-in function bool()
Expressions Context
built-ins
The most of the Python built-ins are available inside the expression.
Introspection or other possibly dangerous functions like type()
, dir()
, open()
, eval()
are disabled.
You can find the full list of available built-ins in this source file.
jq.all(), jq.first()
jq.all(expression: str, json: dict | list | str | int | float) -> list
This function applies expression
to json
and return the list of results.
If you want to get only the first result (e.g. your expression produces only one result), you may use
jq.first(expression: str, json: dict | list | str | int | float) -> Any
device
This is the representation of the current device the test is running against. The instance of NetBox Device
model
device.state - special dictionary which contains all device State. Keys of the dictionary are either command labels or "config". Allows getting access to its values by attribute name, e.g. device.state.show_version
is equal to device.state['show_version']
device.state.config, device.config - the serialized config of the Device
device.dynamic_pair - dynamic pair for this device, it is also Device
instance. May be equal to None
Device has multiple built-in fields and methods that can be used in test expressions.
It's not the purpose of Validity to list all these attributes. If you want you can discover them by yourself using django shell.
cd netbox_folder/netbox
./manage.py shell
from dcim.models import Device
device = Device.objects.first()
dir(device)
This is how you can discover the whole list of device fields and methods.
config()
config(device: Device) -> dict | list
config(device)
does exactly the same thing as device.config
, it returns serialized config for the device.
When you extract a particular device instance from the ORM, .config
attribute is unreachable, because device from the ORM is just regular NetBox Device, it has no extra properties. Therefore, the only option to extract config from the ORM device is to use config()
.
Example:
Let's suppose that 2 switches are connected to each other using interface ge0/0/1 on both sides. Let's check these interfaces have exactly the same configuration.
jq.first(".interfaces.ge0/0/1", device.config) == jq.first(
".interfaces.ge0/0/1",
config(device.interfaces.get(name='ge0/0/1').connected_endpoints[0].device)
)
Here we used config()
function, because the peering device obtained from ORM has no .config
property
state()
state(device: Device) -> dict
Function to get device state (the same as device.state) if .state property is not available (see previous example with config())
Expression restrictions
In test expressions you CANNOT USE:
- the fields or methods that start with underscore (e.g.
device._meta
) .delete()
,.save()
,.update()
,.bulk_update()
,.bulk_create()
,.mro()
,.format()
,.format_map()
methods- introspection or other dangerous built-ins like
type()
,open()
oreval()
- Creation of very long strings (over 1 000 000 characters) or very long comprehensions (over 100 000 items)
Warning
As you may notice, execution of the test with the above defaults is RELATIVELY safe. It means that you can't break anything with some obvious way. However:
- The users can bring their own functions into the tests via Name Sets. No one besides the author can guarantee that these functions are safe
- it's still random code execution and no one can guarantee that it is safe
So, you must assign the permissions to add/modify the tests with cautions.
Fields
Name
The name of the test. Must be unique.
Severity
One of the LOW
, MIDDLE
or HIGH
. This param influences the future analysis of the Test Results and the appearance of Reports (test results inside the report will be grouped by test severity).
Selectors
The list of selectors this test is bound to.
Description
Description of the test is mandatory. Writing meaningful description for each of your tests may facilitate the future usage/modification of this test and is considered a best practice.
Expression
Inside this field at the test page you can view your expression defined either via DB or via Data Source.
At the add/edit form this field is used to store an expression code inside the DB. This option fits well when you want to quickly check your test or just don't want to make things complex with Git-based storage.
Data Source and Data File
Info
You can use only one option per one Test instance: you either define your expression via DB (Expression field) or via Data Source (Data Source and Data File fields). You can't use both approaches at the same time for the same Test.
This pair of fields allows you to store test expression as a file in a Data Source (likely pointing to a git repository).
This is the best option if you have plenty of complex Tests and want to get all the benefits from storing them under version control.