Enable Creation of Individual Files for Translated Rules

[Mat0vu]

Hi everyone,

this PR adds two new optional parameters to the sigma-cli command sigma convert and would close #58

• --output-dir (-od)
• --nesting-level (-nl)

With output_dir it is possible to specify a directory in which all rules from the input are stored. The difference to the existing --output is that not all rules are written in a single file, instead within the output_dir there will be one file created for each translated rule.

The nesting-level parameter defaults to 1. With this value, when one calls sigma convert rules/* with the following file hierarchy, the output_directory will contain all files on the same level. With a nesting_level of 2, the original structure with the parent directories windows and linux would be transferred to the output_directory.

rules/
├── windows/
│   └── rule_1.yml
├── linux/
│   └── rule_2.yml

I hope the names and descriptions of the two operators are clear. I also added two test cases. In my opinion this can be helpful for some people. Please let me know what you think and feel free to make changes.

[thomaspatzke]

Hi! Thanks for the pull request! I've identified an issue while code review: It's possible that this change misses to output converted queries because there can be more queries returned by the backend than input rules, see the corresponding comment for details.

Another issue I've discovered was while I tried to change to pyTest's tmp_path way to manage temporary outputs while testing. This has various advantages like output retention, automatic handling etc. Using tmp_path causes that an absolute path is provided and this caused the code to fail in creation of the output directory. It appears that ensure_dir_exists() has issues with drive letters in paths, which I recommend to replace anyways with pathlib.Path.mkdir() that basically does the same.

[Mat0vu]

Hi Thomas,
thanks for the feedback and the tips, I´m not so familiar with pytest and didn´t know about the tmp_path possibility, but that´s a nice solution. I´ve changed the path handling now to use this and fixed also the ensure_dir_exists().

I can´t see any comments of you in the code review part at the moment but I also realized the issue now. I added a new test test_convert_correlation_rule_to_output_dir with a correlation rule and the generate: true attribute in the rule so that for one rule, two results are returned by the convert function. Currently the code would only write the first part of the result into the file.

While for only a single translated rule this can be handled easily, I´m wondering how this is solved best in general, because if a complete directory of rules is traversed and some of them generate more than one output, how can these outputs be mapped to the corresponding paths of the input rules?

[thomaspatzke]

Currently I don't see a clean way to do this in a generic way for all backends that emit single queries as the link between rule and query is not maintained in the conversion process. I see two possibilities:

• If this is specific to an output format, the directory structure could be build by the backend and output as ZIP archive in a byte stream. Sigma CLI then enforces from the user that the output is written to a file.
• If this should be made generic for backends that return a list of queries then this code in pySigma must be changed in a way that the information about the converted rule. To my knowledge, attaching additional attributes to builtin objects as strings is not possible in Python. I think the only way to overcome this would be to add some result container object that contains this information, which would possibly break lot of stuff based on pySigma.
• Another possibility could be to track the rules in a separate array in pySigma and then allow to (optionally to maintain downward compatibility. Return this array too from Backend.convert().