2.1 Adding Model Descriptions

Every object of class TextEmbeddingModel comes with several methods allowing you to provide important information for potential users of your model.

First, every model needs a clear description how it was developed, modified and how it can be used. You can add a description via the method set_model_description.

example_model$set_model_description(
  eng=NULL,
  native=NULL,
  abstract_eng=NULL,
  abstract_native=NULL,
  keywords_eng=NULL,
  keywords_native=NULL)

This method allows you to provide a description in English and in the native language of your model to make the distribution of your model easier. You can write your description in HTML which allows you to add links to other sources or publications, to add tables or to highlight important aspects of your model.

We would like to recommend that you write at least an English description to allow a wider community to recognize your work. Furthermore, your description should include:

• Which kind of data was used to create the model.
• How much data was used to create the model.
• Which steps were performed and which method was used.
• For which kinds of tasks or materials the model can be used.

With abstract_eng and abstract_native you can provide a summary of your description. This is very important if you would like to share your work on a repository. With keywords_eng and keywords_native you can set a vector of keywords which help to find your work with search engines. We would like to recommend that you at least provide this information in English.

You can access a model’s description by using the method get_model_description

example_model$get_model_description()

Besides a description of your work it is necessary to provide information about other people who were involved in creating a model. This can be done with the method set_publication_info.

example_model$set_publication_info(
  type,
  authors,
  citation,
  url=NULL)

First of all you have to decide the type of information you would like to add. You have two choices: “developer”, and “modifier”, which you set with type.

type="developer" stores all information about the people involved and the process of developing the model. If you use a transformer model from Hugging Face, the people and their description of the model should be entered as developers. In all other cases you can use this type for providing a description of how you developed the model.

In some cases you might wish to modify an existing model. This might be the case if you use a transformer model and you adapt the model to a specific domain or task. In this case you rely on the work of other people and modify their work. In these cases you can describe your modifications by setting type=modifier.

For every type of person you can add the relevant individuals via authors. Please use the R’s function personList() for this. With citationyou can provide a free text how to cite the work of the different persons. With url you can provide a link to relevant sites of the model.

You can access the information by using get_publication_info.

example_model$get_publication_info()

Finally, you must provide a license for using your model. This can be done with set_software_license and get_software_license.

example_model$set_software_license("GPL-3")

Please note, that in most cases the license for your model must be “GPL-3” since some of the software used to create your model are licensed under “GPL-3”. Thus, derivative work must also be licensed under “GPL-3”.

The documentation of your work is not part of the software. Here you can set other licenses such as Creative Common (CC) or Free Documentation License (FDL). You can set the license for your documentation by using the method ‘set_documentation_license’.

example_model$set_documentation_license("CC BY-SA")

Now you are able to share your work. Please remember to save your now fully described object as described in the following section 2.2.

2.2 Saving and Loading

Saving a created text embedding model is very easy by using the function save_ai_model. This function provides a unique interface for all text embedding models. For saving your work you can pass your model to model and the directory where to save the model to model_dir. Please do only pass the path of a directory and not the path of a file to this function. Internally the function creates a new folder in the directory where all files belonging to a model are stored.

save_ai_model(
  model=topic_modeling, 
  model_dir="text_embedding_models",
  append_ID=FALSE)

save_ai_model(
  model=global_vector_clusters_modeling, 
  model_dir="text_embedding_models",
  append_ID=FALSE)

save_ai_model(
  model=bert_modeling, 
  model_dir="text_embedding_models",
  append_ID=FALSE)

As you can see all three text embedding models are saved within the same directory named “text_embedding_models”. Within this directory the function creates a unique folder for every model. The name of this folder is specified with dir_name.

If you set dir_name=NULL and append_ID=FALSE the the name of the folder is created by using the models’ names. If you change the argument append_ID to append_ID=TRUE and set dir_name=NULL the unique ID of the model is added to the directory. The ID is added automatically to ensure that every model has a unique name. This is important if you would like to share your work with other persons.

If you want to load your model, just call the function load_ai_model and you can continue using your model. The following code assumes that you have specified dir_name manually.

topic_modeling<-load_ai_model(
  model_dir="text_embedding_models/model_topic_modeling",
  ml_framework=aifeducation_config$get_framework())

global_vector_clusters_modeling<-load_ai_model(
  model_dir="text_embedding_models/model_global_vectors",
  ml_framework=aifeducation_config$get_framework())

bert_modeling<-load_ai_model(
  model_dir="text_embedding_models/model_transformer_bert",
  ml_framework=aifeducation_config$get_framework())

In the case you set dir_name=NULL and append_ID=TRUE loading the models may look as shown in the following code snippet:

topic_modeling<-load_ai_model(
  model_dir="text_embedding_models/topic_model_embedding_ID_DfO25E1Guuaqw7tM")

global_vector_clusters_modeling<-load_ai_model(
  model_dir="text_embedding_models/global_vector_clusters_embedding_ID_5Tu8HFHegIuoW14l")

bert_modeling<-load_ai_model(
  model_dir="text_embedding_models/bert_embedding_ID_CmyAQKtts5RdlLaS")

Please note that you have to add the name of the model to the directory path. In our example we have stored three models in the directory “text_embedding_models”. Each model is saved within its own folder. The folder’s name is created automatically with the help of the name of the model. Thus, for loading a model you must specify which model you want to load by adding the model’s name to the directory path as shown above.

At this point you may wonder why there is an ID in model’s name although you did not enter an ID on model’s creation. The ID is added automatically to ensure that every model has a unique name. This is important if you would like to share your work with other persons. During saving the ID is appended automatically by setting append_ID=TRUE.

Now you are ready to share your work. Just provide all files within your model folder. For the BERT model in the example above this would be the folder "text_embedding_models/model_transformer_bert" or "text_embedding_models/bert_embedding_ID_CmyAQKtts5RdlLaS" depending on how you saved the model.

3.1 Adding Model Descriptions

Adding the model description of a classifier is similar to TextEmbeddingModels. With the methods set_model_description and get_model_description you can provide a detailed description (parameter eng and native) of your classifier in English and the native language of your classifier. With abstract_eng and abstract_native you can provide the corresponding abstract of your descriptions while keywords_eng and keywords_native take a vector with the corresponding keywords.

example_classifier$set_model_description(
eng="This classifier targets the realization of the need for competence from 
  the self-determination theory of motivation by Deci and Ryan in lesson plans 
  and materials. It describes a learner’s need to perceive themselves as capable. 
  In this classifier, the need for competence can take on the values 0 to 2. 
  A value of 0 indicates that the learners have no space in the lesson plan to 
  perceive their own learning progress and that there is no possibility for 
  self-comparison. At level 1, competence growth is made visible implicitly, 
  e.g. by demonstrating the ability to carry out complex exercises or peer 
  control. At level 2, the increase in competence is made explicit by giving 
  each learner insights into their progress towards the competence goal. For 
  example, a comparison between the target vs. actual development towards the 
  learning objectives of the lesson can be made, or the learners receive 
  explicit feedback on their competence growth from the teacher. Self-assessment 
  is also possible. The classifier was trained using 790 lesson plans, 298 
  materials and up to 1,400 textbook tasks. Two people who received coding 
  training were involved in the coding and the inter-coder reliability for the 
  need for competence increased from a dynamic iota value of 0.615 to 0.646 over 
  two rounds of training. The Krippendorffs alpha value, on the other hand, 
  decreased from 0.516 to 0.484. The classifier is suitable for use in all 
  settings in which lesson plans and materials are to be reviewed with regard 
  to their implementation of the need for competence.",
native="Dieser Classifier bewertet Unterrichtsentwürfe und Lernmaterial danach, 
  ob sie das Bedürfnis nach Kompetenzerleben aus der Selbstbestimmungstheorie 
  der Motivation nach Deci und Ryan unterstützen. Das Kompetenzerleben stellt 
  das Bedürfnis dar, sich als wirksam zu erleben. Der Classifer unterteilt es 
  in drei Stufen, wobei 0 bedeutet, dass die Lernenden im Unterrichtsentwurf 
  bzw. Material keinen Raum haben, ihren eigenen Lernfortschritt wahrzunehmen 
  und auch keine Möglichkeit zum Selbstvergleich besteht. Bei einer Ausprägung 
  von 1 wird der Kompetenzzuwachs implizit, also z.B. durch die Durchführung 
  komplexer Übungen oder einer Peer-Kontrolle ermöglicht. Auf Stufe 2 wird der 
  Kompetenzzuwachs explizit aufgezeigt, indem jede:r Lernende einen objektiven 
  Einblick erhält. So kann hier bspw. ein Soll-Ist-Vergleich mit den Lernzielen 
  der Stunde erfolgen oder die Lernenden erhalten dezidiertes Feedback zu ihrem 
  Kompetenzzuwachs durch die Lehrkraft. Auch eine Selbstbewertung ist möglich.
  Der Classifier wurde anhand von 790 Unterrichtsentwürfen, 298 Materialien und 
  bis zu 1400 Schulbuchaufgaben traniert. Es waren an der Kodierung zwei Personen 
  beteiligt, die eine Kodierschulung erhalten haben und die 
  Inter-Coder-Reliabilität für das Kompetenzerleben würde über zwei 
  Trainingsrunden von einem dynamischen Iota-Wert von 0,615 auf 0,646 gesteigert. 
  Der Krippendorffs Alpha-Wert sank hingegen von 0,516 auf 0,484. Er eignet sich 
  zum Einsatz in allen Settings, in denen Unterrichtsentwürfe und Lernmaterial 
  hinsichtlich ihrer Umsetzung des Kompetenzerlebens überprüft werden sollen.",
abstract_eng="This classifier targets the realization of the need for 
  competence from Deci and Ryan’s self-determination theory of motivation in l
  esson plans and materials. It describes a learner’s need to perceive themselves 
  as capable. The variable need for competence is assessed by a scale of 0-2. 
  The classifier was developed using 790 lesson plans, 298 materials and up to 
  1,400 textbook tasks. A coding training was conducted and the inter-coder 
  reliabilities of different measures 
  (i.e. Krippendorff’s Alpha and Dynamic Iota Index) of the individual categories 
  were calculated at different points in time.",
abstract_native="Dieser Classifier bewertet Unterrichtsentwürfe und 
  Lernmaterial danach, ob sie das Bedürfnis nach Kompetenzerleben aus der 
  Selbstbestimmungstheorie der Motivation nach Deci & Ryan unterstützen. Das 
  Kompetenzerleben stellt das Bedürfnis dar, sich als wirksam zu erleben. Der 
  Classifer unterteilt es in drei Stufen und wurde anhand von 790 
  Unterrichtsentwürfen, 298 Materialien und bis zu 1400 Schulbuchaufgaben 
  entwickelt. Es wurden stets Kodierschulungen durchgeführt und die 
  Inter-Coder-Reliabilitäten der einzelnen Kategorien zu verschiedenen 
  Zeitpunkten berechnet.",
keywords_eng=c("Self-determination theory", "motivation", "lesson planning", "business didactics"),
keywords_native=c("Selbstbestimmungstheorie", "Motivation", "Unterrichtsplanung", "Wirtschaftsdidaktik")

In the case of a classifier, the description should include:

• A short reference to the theoretical models that guided the development.
• A clear and detailed description of every single category/class.
• A short statement where the classifier can be used.
• A description of the kind and quantity of data used for training.
• Information on potential bias in the data.
• If possible, information about the inter-coder-reliability of the coding process providing the data.
• If possible, provide a link to the corresponding text embedding model.

Again, you can provide your description in HTML to include tables (e.g. for reporting the reliability of the initial coding process) or links to other sources and publications.

Please do not report the performance values of your classifier in the description. These can be accessed directly via example_classifier$reliability$test_metric_mean.

With the methods set_publication_info and get_publication_info you can provide the bibliographic information of your classifier.

example_classifier$set_publication_info(
  authors,
  citation,
  url=NULL)

In contrast to TextEmbeddingModels there are not different types of author groups.

Finally, you can manage the license for using your classifier via set_software_license and get_software_license.

example_classifier$set_software_license("GPL-3")

Similar to TextEmbeddingModels your classifier has to be licensed via “GPL-3” since some of the software used for creating a classifier applies this license. For the documentation you can choose between further license since the documentation is not part of the software. For setting and receivinf the license you can call the methods ‘set_documentation_license’ and ‘get_documentation_license’

example_classifier$set_documentation_license("CC BY-SA")

Now you are ready for sharing your classifier. Please remember to save your changes as described in the following section 3.2.

3.2 Saving and Loading

If you have created a classifier, saving and loading is very easy due to the functions save_ai_model and load_ai_model. The process for saving a model is similar to the process for text embedding models. You only have to pass the model and a directory path to the function save_ai_model. The folder name is set with dir_name.

save_ai_model(
  model=classifier,
  model_dir="classifiers",
  dir_name="movie_review_classifier",
  save_format = "default",
  append_ID = FALSE)

In contrast to text embedding models you can specify the additional argument save_format. In the case of pytorch models this arguments allows you to choose between save_format = "safetensors" and save_format = "pt". We recommend to chose save_format = "safetensors" since this is a safer method to save your models. In the case of tensorflow models this arguments allows you to choose between save_format = "keras", save_format = "tf" and save_format = "h5". We recommend to chose save_format = "keras" since this is the recommended format by keras. If you set save_format = "default" .safetensors is used for pytorch models and .keras is used for tensorflow models.

If you would like to load a model you can call the function load_ai_model.

classifier<-load_ai_model(
  model_dir="classifiers/movie_review_classifier")

Note: Classifiers depend on the framework which was used during creation. Thus, a classifier is always initalized with its original framework. The argument ml_framework has no effect.

In the case you would like to share your classifier with a broader audience we recommend to set dir_name=NULL and append_ID = TRUE. This will create a folder name automatically using the classifier’s name and unique ID. Similar to text embedding models ID is added to the name during the creation of the classifier ensuring a unique name for your model. With this options the folder name may look like "movie_review_classifier_ID_oWsaNEB7b09A1pPB".

If you would like to share your classifier with other persons you have to provide all files within this folder "classifiers/movie_review_classifier_ID_oWsaNEB7b09A1pPB". Since all files are stored with a specific structure do not change or edit the files manually.

Please note that you need the TextEmbeddingModel that was used for training in order to predict new data with the classifier. You can request the name, label, and configuration of that model with example_classifier$get_text_embedding_model()$model. Thus, if you would like to share your classifier, ensure that you also share the corresponding text embedding model.

If you would like to apply your classifier to new data, two steps are necessary. First, you must transform the raw text into a numerical expression by using exactly the same text embedding model that was used to train your classifier. The resulting object can be passed to the method predict and you will receive the predictions together with an estimate of certainty for each class/category.

More information can be found in the vignette 02a Using Aifeducation Studio or in 02 classification tasks.