How to Use & Create Cosmic Frog Model Utilities

Introducing Utilities

Utilities enable powerful modelling capabilities for use cases like integration to other services or data sources, repeatable data transformation or anything that can be supported by Python! System Utilities are available as a core capability in Cosmic Frog for use cases like LTL rate lookups, TransitMatrix time & distance generation, and copying items like Maps and Dashboards from one model to another. More useful System Utilities will become available in Cosmic Frog over time. Some of these System Utilities are also available in the Resource Library where they can be downloaded from, and then customized and made available to modelers for specific projects or models. In this Help Article we will cover both how to use use System Utilities as well as how to customize and deploy Custom Utilities.

The “Using and Customizing Utilities” resource in the Resource Library includes a helpful 15-minute video on Cosmic Frog Model Utilities and users are encouraged to watch this.

In this Help Article, System Utilities will be covered first, before discussing the specifics of creating one’s own Utilities. Finally, how to use and share Custom Utilities will be explained as well.

System Utilities

Users can access utilities within Cosmic Frog by going to the Utilities section via the Module Menu drop-down:

1. Click on the icon with 3 horizontal bars to open the Module Menu drop-down.
2. Choose Utilities to go to the Utilities section of Cosmic Frog.

Once in the Utilities section, user will see the list of available utilities:

1. We are in the Utilities section of Cosmic Frog.
2. At the top, the System Utilities are listed. They have categories (e.g. Test, Transportation, etc.) which can be expanded and collapsed by clicking on the arrowhead icons to the left of the category names.
3. At the bottom, the user’s custom utilities are listed in the My Utilities list.
4. Users can search for specific utilities in their list by free typing a search term and the list will automatically be filtered for utilities that contain the search term in their name.
5. The utilities list can be collapsed by clicking on the icon with the 2 arrowheads pointing left. Once collapsed, the pane can be expanded again by clicking on the icon with 2 arrowheads pointing right that will now be showing.

Utilities vary in complexity by how many input parameters a user can configure and range from those where no parameters need to be set at all to those where many can be set. Following screenshot shows the Orders to Demand utility which does not require any input parameters to be set by the user:

1. In the Data Transformation category there is a utility called Orders to Demand, which is selected here.
2. The name of the utility is repeated at the top of the tab where it is open.
3. Each utility has a short description which explains briefly what the utility will do when it is run.
4. Since there are no input parameters that user needs to configure for this utility, user can simply click the Run Utility button when ready to run.

The Copy map to a model utility shown in the next screenshot does require several parameters to be set by the user:

1. The Copy map to a model utility from the Copy to a Model category is selected.
2. This utility has 4 parameters which can be configured by the user.
3. All parameters in all System Utilities have a blue question mark to the right of them, hovering over these will show a text bubble with a short explanation of the value to set for the parameter.
4. Different parameters may have different configuration options. This one for example has a drop-down list to select the parameter’s value from a set of possible values. Here, the options are to either Append or Replace.
5. Some parameters will have a free text field for the user to type the value of the parameter into, like the one shown here. Other parameter types are for example Booleans and integers.
6. If a utility uses 1 or more parameters, it has a Reset button which can be used in case user wants to revert the parameters to their original values.
7. Once user has configured all parameters as desired, the Run Utility button can be clicked to start performing the actions of the utility.

When the Run Utility button has been clicked, a message appears beneath it briefly:

Clicking on this message will open the Model Activity pane to the right of the tab(s) with open utilities:

1. The Model Activity pane can also be opened by clicking on the dark blue speech bubble icon at the top right of the Cosmic Frog application. Clicking the same icon again will close the pane.
2. The name and icon of the pane.
3. There are 2 buttons here with options to filter the Model Activity list:
   1. The crossed-out eye icon lets a user toggle between seeing the full Model Activity history or only those that have not yet been marked as read. When this icon is blue, activities that have been marked as read are not shown. When it is black, all activities are shown, including those that have been marked as read.
   2. The filter icon with the 3 horizontal bars gives the user the option to filter out activities by their status: started, completed, failed, and pending. Any combination of these can be selected by enabling/disabling the checkbox for each status individually.
4. User can mark a Model Activity as read by clicking on the cross icon at the right top of the activity. This icon then changes to a blue eye icon.
5. User can click on the expand icon to see the details of the activity:

1. User clicked on the expand icon for the model activity listed at the top of the list, which was running the Copy Scenarios to a model utility. It has finished successfully.
2. The log of the Copy Scenarios to a model utility activity is shown.
3. User has the option to download this log as a text file by clicking on the icon with the arrow pointing downwards or copying the text of the log to the clipboard by clicking on the icon with the 2 squares.

Users will not only see activities related to running utilities in the Model Activity list. Other actions that are executed within Cosmic Frog will be listed here too, like for example when user has geocoded locations by using the Geocode tool on the Customers / Facilities / Suppliers tables or when user makes a change in a master table and chooses to cascade these changes to other tables.

Please note that the following 2 System Utilities have a separate Help Article where they are explained in more detail:

1. The SMC3 RateWare utility is explained in this article: How to Use SMC³’s RateWare XL LTL Rating Engine in Cosmic Frog
2. The Distance Lookup utility is explained towards the end of this article: Geo Providers for Geocoding and Distance and Time Calculations

Customizing Existing & Creating New Utilities

The utilities that are available in the Resource Library can be downloaded by users and then customized to fit the user’s specific needs. Examples are to change the logic of a data transformation, apply similar logic but to a different table, etc. Or users may even build their own utilities entirely. If a user updates a utility or creates a new one, they can share these back with other users so they can benefit from them as well.

Utilities are Python scripts that contain a specific structure which will be explained in this section. They can be edited directly in the Atlas application on the Optilogic platform or users can download the Python file that is being used as a starting point and edit it using an IDE (Integrated Development Environment) installed on their computer. A rich text editor geared towards coding, like for example Visual Studio Code, will work fine too for most. An advantage of working locally is that user can take advantage of code completion features (auto-completion while typing, showing what arguments functions need, catch incorrect syntax/names, etc.) by installing an extension package like for example IntelliSense (for Visual Studio Code). The screenshots of the Python files underlying the utilities that follow in this documentation are taken while working with them in Visual Studio Code locally and on a machine that has the IntelliSense extension package installed.

A great resource on how to write Python scripts for Cosmic Frog models is this “Scripting with Cosmic Frog” video. In this video, the cosmicfrog Python library, which adds specific functionality to the existing Python features to work with Cosmic Frog models, is covered in some detail.

We will start by looking at the Python file of the very simple Hello World utility. In this first screenshot, the parts that can stay the same for all utilities are outlined in green:

1. The Hello World.py file which underlies the Hello World System Utility in Cosmic Frog has been downloaded from the Using and Customizing Utilities resource in the Resource Library and is opened locally in Visual Studio Code.
2. These lines are the same for all utilities:
   1. Line 1: several modules from the cosmicfrog library are imported so they can be used by the script.
   2. Line 3: a function named details is defined. It will be modified (between lines 5 and 12) based on what the utility needs to look like in Cosmic Frog (e.g. description, number and type of input parameters, etc.).
   3. Line 5: a variable named util is defined and set to the UtilityDetails module for ease of use, so now util can be typed to easily access any of the functions contained in UtilityDetails.
3. These lines can also remain the same across utilities:
   1. Line 12: “return util” wraps up the definition of the details function.
   2. Line 14: a new function named run is defined. It will be modified (between lines 14 and 18) based on what the utility needs to do.
4. Finally, these last few lines of the script remain unchanged across utilities too:
   1. Line 18: “return” wraps up the definition of the run function.
   2. Lines 20-21: this if statement executes the code on line 21 (=run the utility) when the Python file is executed as a script, which is what Cosmic Frog does for you when clicking the Run Utility button within a utility.

Next, onto the parts of the utility’s Python script that users will want to update when customizing / creating their own scripts:

1. The category and description of the utility can be set by the user using this syntax:
   1. category: this is a string variable which sets the category of the utility. Here it is set to Test. The utility will be added to this category and show up under its name in the list of utilities.
   2. description: this is also a string variable; it specifies the description of the utility. This description is shown in the top part of the utility when it is open in Cosmic Frog. It is recommended to provide a brief explanation of what the utility will do in this description.
2. The “util.params = Params()” specifies the parameters for the utility. This line stays the same for all utilities as well, and as we will see further below, individual parameters are added after this first initialization of them here. This Hello World utility does not have any input parameters for the user to configure, so for this utility just this line initializing parameters suffices.
3. This is the part where the actions the utility performs are specified. In this simple script, the only thing that will happen is that “Hello World” will be printed on the screen/in the utility’s log.

Now, we will discuss how input parameters, which users can then set in Cosmic Frog, can be added to the details function. After that we will cover different actions that can be added to the run function.

Adding Parameters

If a utility needs to be able to take any inputs from a user before running it, these are created by adding parameters in the details function of the utility’s Python script:

1. To add a new parameter, type util.params.add. Then once an opening parenthesis is typed, context-sensitive help explaining the add function and its arguments opens up.
2. The arguments of the add function and their types are listed at the top. The argument in blue font is the one that user is currently on. The arguments are also described further down in the help window, outlined in green with the number 5.
3. The description of the argument that the user is currently on is shown here. User is on the name argument (the first argument) which is also the argument that is showing in blue font above.
4. A description of the add function is given here.
5. This section gives a short description of each argument of the add function:
   1. Name: this name will be showing in Cosmic Frog as the parameter name.
   2. Description: this will be the tooltip information user sees in Cosmic Frog when hovering over the blue question mark at the right of the parameter.
   3. Default: if a default is specified here, it will be filled out as the value for the parameter in Cosmic Frog.
   4. Param_type: this can for example be text where a user can free type the value of the parameter, a list of strings which will show as a drop-down in the parameter’s value field for a user to choose one from, a Boolean, etc.
6. If the function returns anything, the data type of the result will be listed here (e.g. string or integer, etc). In this case the add function does not return anything (“None” in the green box with number 2).

We will take a closer look at a utility that uses parameters and map the arguments of the parameters back to what the user sees when the utility is open in Cosmic Frog, see the next 2 screenshots: the numbers in the script screenshot are matched to those in the Cosmic Frog screenshot to indicate what code leads to what part of the utility when looking at it in Cosmic Frog. These screenshots use the Copy dashboard to a model utility of which the Python script (Copy dashboard to a model.py) was downloaded from the Resource Library.

1. The name of the Python script, this also becomes the name of the utility in Cosmic Frog.
2. In addition to importing modules from the cosmicfrog library (line 2 here), a specific Python module named datetime is imported from the datetime library in this script (line 1). This is so files can be time-stamped later in the script as part of the utility’s actions.
3. The utility’s category and description are specified here:
   1. The category of this utility is Copy to a model, so in Cosmic Frog, we can look under System Utilities > Copy to a model to find this Copy dashboard to a model utility.
   2. The description that will be listed at the beginning of the utility is set here by util.description.
4. The first input parameter of the utility is added here:
   1. The name of the parameter is “Destination Model”.
   2. The description of the parameter which will show up as the tooltip when hovering over the blue question mark is “Enter the name of the model that you want to copy the dashboard into.”
   3. The default of the parameter is blank.
   4. The data_type of the parameter is string, meaning it will be a free type text field.
5. The “Replace or Append dashboards” parameter is added.
6. The “Copy All or a Specific dashboard” parameter is added.
7. The “Specific dashboard to copy” parameter is added.

Note that Python lists are 0-indexed, meaning that the first parameter (Destination Model in this example) is referenced by typing params[0], the second parameter (Replace of Append dashboards) by typing params[1], etc. We will see this in the code when adding actions to the run function below too.

Now let’s have a look at how the above code translates to what a user sees in the Cosmic Frog user interface for the Copy dashboard to a model System Utility (note that the numbers in this screenshot match with those in the above screenshot):

1. The name of the utility, which is set by the name of the utility’s Python script. It is shown in the list of utilities, as the name of the tab when it is open, and finally repeated at the top of the utility.
3. The category and description of the utility:
   1. The category (util.category variable) that was defined for the utility (Copy to a model); it sits under the System Utilities.
   2. The description that was entered as the value for the util.description variable is shown in the top part of the utility.
4. The first input parameter:
   1. The name of the parameter, which is the first argument of the add function (Destination Model).
   2. The text in the tooltip that shows when hovering over the blue question mark was set as the second argument (description) for this first parameter.
   3. The default value of this parameter was set to “” by the third argument of the add function, which means the default is blank.
   4. The data type of the parameter type is set to “string” by the fourth argument of the add functoin, so the user can free type the value for the Destination Model into the text box.
5. The second input parameter: Replace or Append Dashboards, which has a drop-down with 2 values (Append & Replace).
6. The third input parameter: Copy All or a Specific Dashboard, which has a drop-down with 2 values (All & Specific).
7. The fourth parameter: Specific Dashboard To Copy, which is a free type text field.

Adding Actions

The actions a utility needs to perform are added to the run function of the Python script. These will be different for different types of utilities. We will cover the actions the Copy dashboard to a model utility uses at a high level and refer to Python documentation if user is interested in understanding all the details. There are a lot of helpful resources and communities online where users can learn everything there is to know about using & writing Python code. A great place to start is on the Python for Beginners page on python.org. This page also mentions how more experienced coders can get started with Python. Also note that text in green font that follows a hash sign are comments to add context to code.

1. The first part of the run function of the “Copy dashboard to a model” utility’s script sets the values of 3 variables to the user-entered values for input parameters 2, 3, and 4.
   1. The copy_type variable is set to the value of the second user input parameter, which is accessed by typing params[1] (since Python lists are 0-indexed). So, the value will be either Replace or Append, based on what the user chose in the drop-down list for this input parameter.
   2. The all_specific variable is set to the value of the third user input parameter. This value will be either All or Specific, again based on what the user chose in the drop-down list for this input parameter.
   3. The specific_name variable is used if the user chose to only copy a specific dashboard, i.e. the all_specific variable is set to Specific, and then the value is set to what the user typed into the text box for the fourth input parameter.
2. The second part of the run function sets 2 fixed parameters:
   1. The variable table_name is set to ‘analytics’ which is the SQL table name of Cosmic Frog models where dashboards are saved.
   2. The variable column_name is set to ‘dashboardname’ which is the column in the analytics table which contains the names of all the dashboards present in Cosmic Frog models.
3. Here the script connects to the destination model specified by the user in the first input parameter:
   1. It tries to connect by using the FrogModel function, using the user’s first input parameter (referenced by params[0]) as the model name. If this is successful, the script continues.
   2. If it fails to connect, the script prints “Destination model does not exist” to the utility’s log and exits the run function, which also exits the entire script.

1. This line of code creates a data frame named df_CopyData that contains the whole analytics table of the Cosmic Frog source model (the model from within user runs the Copy dashboard to a model utility).
2. Here an if statement is used to reduce the df_CopyData data frame to only the record where the dashboardname column’s value matches the specific dashboard name that user has specified (as the fourth input parameter, which the specific_name variable is set to). This is only done if the user has not chosen All as the value for the “Copy All or a Specific Dashboard” input parameter (the all_specific variable).
3. It is possible that at this stage the df_CopyData data frame is empty. This can be the case when there are no dashboards specified in the model at all or when a specific dashboard is being copied and the name of it (the specific_name variable) does not match a name in the dashboardname column of the analytics table. If the data frame is empty, a message “No dashboards have been copied. …” is printed to the utility’s log and the run function is exited, which also exits the entire script. If the data frame is not empty, a message “Dashboard(s) are being copied” is written to the utility’s log and the script continues.
4. A data frame named df_CheckData is created from the entire analytics table of the destination model (the first user input parameter); this will be used to check if dashboard(s) of the same name already exist in the destination model if the user chose to Append and not Replace dashboards (the copy_type variable).

1. This for loop checks for each row in the df_CopyData data frame (from the source model) if the value of the dashboardname column is equal to a value in the dashboardname column in the df_CheckData data frame (from the destination model). If so, this means a dashboard of that name already exists in the destination model. In this case how the script continues depends on if the user has chosen to either Append or Replace the dashboard(s) in the destination model, which is set in the second user input parameter, and the variable copy_type has been set to this value.
2. If user has chosen to Append dashboards (the copy_type variable is set to ‘Append’), then this piece of code changes the name of the dashboard that is being copied by adding a timestamp to it.
3. If user has chosen to Replace dashboards (the copy_type variable is set to ‘Replace’), then this piece of code deletes the dashboard from the destination model, since it will be replaced by the one of the same name from the source model.
4. This line writes the dashboard from the source model (the df_CopyData dataframe) into the analytics table of the destination model.
5. Finally, the script writes a message that dashboard(s) have been copied to the destination model to the utility’s log.

Using & Sharing Custom Utilities

For a custom utility to be showing in the My Utilities category of the utilities list in Cosmic Frog, it needs to be saved under My Files > My Utilities in the user’s Optilogic account:

1. While logged into your Optilogic account on cosmicfrog.com, open up the Explorer by clicking on the > icon at the left top of the screen, if not open already.
2. Expand the My Files folder.
3. Expand the My Utilities folder. If you do not have a My Utilities folder yet, you can create it by right-clicking on the My Files folder and selecting the Create New Folder option.
4. Right-click on My Utilities and select Upload Files if a Python utility file is to be uploaded from user’s local machine.

Note that if a Python utility file is already in user’s Optilogic account, but in a different folder, user can click on it and drag it to the My Utilities folder.

For utilities to work, a requirements.txt file which only contains the text cosmicfrog needs to be placed in the same My Files > My Utilities folder (if not there already):

A customized version of the Copy dashboard to a model utility was uploaded here, and a requirements.txt file is present in the same folder too.

Once a Python utility file is uploaded to My Files > My Utilities, it can be accessed from within Cosmic Frog:

1. Expand the My Utilities category when in the Utilities module of Cosmic Frog. Click on the refresh icon (2 arrows pointing in a circle) to refresh the list, so any newly added utilities will be shown if they are not already.
2. The “Copy dashboard to a model Custom” utility is now showing in the My Utilities list and can be opened by clicking on it.

If users want to share custom utilities with other users, they can do so by right-clicking on it and choosing the “Send Copy of File” option:

The following form then opens:

1. The Directory Path is listed, which user cannot edit as it is taken from where the file user wants to share is located in user’s account.
2. The File Name is listed, which user cannot edit either as it is the file user right-clicked on.
3. Enter the email address(es) of the Cosmic Frog user(s) you want to share the custom utility with. If multiple, use commas in between the email addresses.
4. Click on Share File to send a copy of the file to the other user(s).

When a custom utility has been shared with you by another user, it will be saved under the Sent To Me folder in your Optilogic account:

1. Expand the Sent To Me folder to find the custom utility that was shared with you. It will be in a sub-folder with the name of the person/account who shared it with you.
2. Expand the My Files folder and look for the My Utilities folder. Then click on the custom utility file (in a sub-folder under the Sent To Me folder), and drag it on top of the My Utilitie. Now the custom utility should be visible from within Cosmic Frog. (If you do not have a My Utilities folder yet, you can create it by right-clicking on the My Files folder and selecting the Create New Folder option.)

Should you have created a custom utility that you feel a lot of other users can benefit from and you are allowed to share outside of your organization, then we encourage you to submit it into Optilogic’s Resource Library. Click on the Contribute button at the left top of the Resource Library and then follow the steps as outlined in the “How can I add Python Modules to the Resource Library?” section towards the end of the “How to use the Resource Library” help article.