The folder structures I normally use depend on whether the plugin is simple or complex.ĬRMProject/crm_plugin/operators/_init_.pyĬRMProject/crm_plugin/operators/customer_operator.pyĬRMProject/crm_plugin/operators/sync_operator.py I prefer to use a simple Python package approach where the package folder gets copied to the Airflow plugins folder, just like you would if you used the single module approach. It specifies that you need to use the setuptools entrypoint system. The Airflow documentation has a short section on creating plugins via a Python package.
#Airflow operators code#
This makes it easier to extend the plugin in future and also keeps the code neat and tidy. It allows me to separate the objects neatly into their own files and folders. My personal preference is to use a python package for plugins. The second option is to create a python package that contains modules and/or sub packages for the different objects.The first option is to simply create a python module that contains all of the objects.If you have looked at the Airflow documentation you will see that there are two options structuring your code. Here is an example of what that would mean for a fictitious CRM plugin. It should never communicate with the service, API or system interface directly but should always do that through the hook.īelow is a simple diagram of the logical components.īy keeping the logic separated into these two components you will be able to build multiple Operators that all use the same Hook without duplicating the code needed to interact with the source system. This is the logic that deals with how your Task will behave and what actions it will perform. OperatorĪn Operator contains all of the “business” logic of what will be a Task in one of your DAGs. It should not contain the logic of the operator itself but should expose methods that will then be used by the Operator in order to perform the actions it requires against the API, service or sustem interface that the operator is targeting. HookĪ Hook contains the logic required to perform actions against an API, service or interface of some system. The two main parts of a custom operator is the Hook and the Operator. We will only be focusing on using them to build custom operators that you can then use as Tasks in your DAGs. The Airflow documentation for plugins show that they can be used to do all sorts of customisation of Airflow. Logical components of a custom operator plugin You will be able to remove your operator’s logic from the code that defines your DAG, ensuring you do not mix the logic of your data pipeline with the logic of the components that form its building blocks.įinally, it allows you to create a redistributable component that can be used by all of the Airflow instances across your organisation.
It allows you to make use of all of the functionality Airflow provides.
Of the three methods only option 3 integrates into Airflow’s core.
0 kommentar(er)
