Introduction

Welcome to GuardOps!

GuardOps is an open source project developed to make AI usage easier and safer through extensive monitoring and collaboration. It has been built primarily with LLMs in mind but is being extended toward other AI models in the future. GuardOps consists of three main components:

  1. Backend - A FastAPI based backend that stores data in a MongoDB database.
  2. Frontend - A Next.js based webapp that provides usage for all the backend components.
  3. Python Library - A python library that provides the key features of the backend for usage in your code

Many features are built with collaboration and cooperation in mind and the benefits of these features will only be noticed when used by multiple people. With this in mind, user management as well as RBAC is done using auth0. GuardOps can be run with just a single user or unprotected but this is not recommended.

How to Run

To provide usability for a wide range of usecases, GuardOps can be deployed in different ways. The default deployment is what it was initially built for and is recommended. However, options exist to deploy GuardOps for just one user or in an unprotected state. Learn more about the unprotected state in the Security section of this page.

⚠️

Make sure you have replaced all the placeholders of the .env files with your actual parameters before starting the containers!

Default Deployment

GuardOps can be run by using the provided docker-compose file in the root of this (opens in a new tab) repository using

docker-compose up -d

However, this implies that the default ports used by the seperate services are not in use already. If you run into any port conflicts, you first need to change the specified ports according to your available allocation.

Deployment for just one user

Running GuardOps for one user eliminates a lot of the needed preparation since there is no more user management or RBAC needed.

💡

Single user deployment eliminates all user management and RBAC requirements as well as Backend API endpoint protection. All data created is linked to the user_id = 0. Switching between single user deployment and default deployment is not possible without extensive database migration.

Unprotected Deployment

The Unprotected Deployment eliminates Backend API Protection that would be done using auth0 M2M APIs. The User Management and RBAC are still done using auth0.

Environment variables

GuardOps uses .env files to manage environment variables. Depending on which option you want to deploy, different parameters need to be set

💡

There are multiple auth0 variables needed that can be confusing. The Backend needs variables for the M2M API. The frontend needs variables for the Application, the User Management as well as the M2M API (backend API for token creation). The frontend .env file's variables can be split in the following categories:

  • Application: AUTH0_BASE_URL, AUTH0_CLIENT_ID, AUTH0_CLIENT_SECRET, AUTH0_CLIENT_SECRET
  • API: AUTH0_TRACEAPI, AUTH0_TRACEAPI_CLIENT_ID, AUTH0_TRACEAPI_CLIENT_SECRET, TRACEAPI_AUDIENCE
  • User Management: AUTH0_CLIENT_SECRET, AUTH0_CLIENT_SECRET, MANAGEMENT_AUDIENCE

1. Backend

Adjust the variable placeholders in the .env file of the backend located at the following path

      • .env
  • MONGO_URI=mongodb+srv://<your_username>:<your_password>@guardops_mongo_db/?retryWrites=true&w=majority
    BASE_KEY=<your_desired_prefix_for_python_library_key_generation>
    AUTH0_DOMAIN=<your_auth0_domain>
    AUTH0_API_AUDIENCE=<your_auth0_api_audience>
    AUTH0_ISSUER=<your_auth0_issuer>
    AUTH0_ALGORITHMS = RS256

    2. Frontend

    Adjust the variable placeholders in the .env file of the frontend located at the following path

      • .env
  • AUTH0_BASE_URL=<your_auth0_base_url>
    AUTH0_CLIENT_ID=<your_auth0_client_id>
    AUTH0_CLIENT_SECRET=<your_auth0_client_secret>
    AUTH0_ISSUER_BASE_URL=<your_auth0_issuer_base_url>
    AUTH0_MANAGEMENT_CLIENT_ID=<your_auth0_management_client_id>
    AUTH0_MANAGEMENT_CLIENT_SECRET=<your_auth0_management_client_secret>
    AUTH0_SECRET=<your_auth0_secret>
    AUTH0_TRACEAPI=<your_auth0_issuer_base_url>/oauth/token
    AUTH0_TRACEAPI_CLIENT_ID=<your_auth0_m2m_client_id_for_backend_api>
    AUTH0_TRACEAPI_CLIENT_SECRET=<your_auth0_m2m_client_secret_for_backend_api>
    BackendBaseUrl=https://<your_backend_api_container_ip_adress>/
    MANAGEMENT_AUDIENCE=<your_auth0_management_audience>
    NPM_CONFIG_PRODUCTIO=false
    TRACEAPI_AUDIENCE=<identical_to_AUTH0_API_AUDIENCE_of_backend>
    UPSTASH_REDIS_REST_TOKEN=<your_upstash_redis_rest_token_for_m2m_token_caching>
    UPSTASH_REDIS_REST_URL=<your_upstash_redis_rest_url_for_m2m_token_caching>

    Port Configuration

    GuardOps deployments always create a docker network with multiple containers running the different components. This means that the default ports can be used within the containers but need to be mapped to any available free port on your host machine. By default the following ports get mapped to the following containers:

    componentdefault port
    Frontend3001
    Backend API8001
    Backend DB27018
    Redis M2M Token Cache6380

    You can adjust these ports within the docker-compose file in the root directory.

    Additional requirements

    Retention is discussed in the concepts section. Making this feature work requires an automatic trigger that runs in consistent intervals. The trigger needs to be set up. MongoDB provides functionality for triggers natively if you use Atlas (opens in a new tab). If you do however use the provided docker-compose, you will have to set this step up manually. Below is the code needed to recreate the retention feature either for MongoDB Atlas or local docker containers

    1. Atlas

    Create a new scheduled Trigger that runs periodically every 6/12/24 hours or however often you want it to run and include the bottom code as the function itself. Remember to replace the placeholder with your actual service name in line 2.

    exports = async function() {
      const db = context.services.get("<YOUR_SERVICE_NAME>").db("Prod");
      
      const pipeline = [
    {
        "$addFields": {
          "projects": {
            "$filter": {
              "input": "$projects",
              "as": "proj",
              "cond": { "$gt": ["$$proj.retention", 0] }
            }
          }
        }
      },
      {
        "$unwind": "$projects"
      },
      {
        "$lookup": {
          "from": "Traces",
          "localField": "projects.project_id",
          "foreignField": "project_id",
          "as": "matched_traces"
        }
      },
      {
        "$match": {
          "matched_traces": { "$ne": [] } 
        }
      },
      {
        "$unwind": "$matched_traces"
      },
      {
        "$lookup": {
          "from": "Spans",
          "localField": "matched_traces.trace_id",
          "foreignField": "context.trace_id",
          "as": "joined_spans"
        }
      },
      {
        "$unwind": "$joined_spans"
      },
      {
        "$addFields": {
          "time": {
            "$toDate": "$joined_spans.start_time" 
          },
          "current_timestamp": new Date(), 
          "retention_limit": {
            "$add": [
              { "$toDate": "$joined_spans.start_time" }, 
              { "$multiply": ["$projects.retention", 24 * 60 * 60 * 1000] }
            ]
          }
        }
      },
      {
        "$match": {
          "$expr": {
            "$lte": ["$retention_limit", "$current_timestamp"]
          }
        }
      },
      {
        "$group": {
          "_id": "$projects.project_id",
          "matched_traces": { "$addToSet": "$matched_traces" },
          "joined_spans": { "$push": "$joined_spans" },
          "projects": { "$first": "$projects" },
        }
      },
      {
        "$project": {
          "_id": 0,
          "trace_id": "$matched_traces.trace_id",
          "project_id": "$projects.project_id",
          "project_retention": "$projects.retention",
          "span_id": "$joined_spans.context.span_id",
          "time": 1,
          "current_timestamp": 1,
          "retention_limit": 1
        }
      }  ];
      
      const results = await db.collection("Users").aggregate(pipeline).toArray();
      
      let spanIds = [];
      let traceIds = [];
      
      results.forEach(result => {
        if (result.span_id) spanIds = spanIds.concat(result.span_id);
        if (result.trace_id) traceIds = traceIds.concat(result.trace_id);
      });
     
      await db.collection("Spans").deleteMany({ "context.span_id": { $in: spanIds } });
      await db.collection("Traces").deleteMany({ trace_id: { $in: traceIds } });
     
      const cleanupDocument = {
        Execution_time: new Date(),
        deleted_traces: traceIds.length,
        traces: traceIds,
        deleted_spans: spanIds.length,
        spans: spanIds
      };
     
      await db.collection("Cleanup").insertOne(cleanupDocument);
    };
     

    2. Container

    Triggers as explained in the Atlas solution are exclusive to atlas. This means that you need to implement the logic to periodically run the script yourself. You will need a scheduler that can be implemented differently depending on how you are hosting GuardOps. Remember to create your .env file as well or just replace the variable value for client in the highlighted line 7 with your actual connection string.

    from pymongo import MongoClient
    from datetime import datetime, timedelta
    from dotenv import load_dotenv
    import os
    load_dotenv(override=True)
     
    client = MongoClient(os.getenv("MONGO_URI"))
    db = client["Prod"]
     
    pipeline = [{
        "$addFields": {
          "projects": {
            "$filter": {
              "input": "$projects",
              "as": "proj",
              "cond": { "$gt": ["$$proj.retention", 0] }
            }
          }
        }
      },
      {
        "$unwind": "$projects"
      },
      {
        "$lookup": {
          "from": "Traces",
          "localField": "projects.project_id",
          "foreignField": "project_id",
          "as": "matched_traces"
        }
      },
      {
        "$match": {
          "matched_traces": { "$ne": [] } 
        }
      },
      {
        "$unwind": "$matched_traces"
      },
      {
        "$lookup": {
          "from": "Spans",
          "localField": "matched_traces.trace_id",
          "foreignField": "context.trace_id",
          "as": "joined_spans"
        }
      },
      {
        "$unwind": "$joined_spans"
      },
      {
        "$addFields": {
          "time": {
            "$toDate": "$joined_spans.start_time" 
          },
          "current_timestamp": datetime.now(), 
          "retention_limit": {
            "$add": [
              { "$toDate": "$joined_spans.start_time" }, 
              { "$multiply": ["$projects.retention", 24 * 60 * 60 * 1000] }
            ]
          }
        }
      },
      {
        "$match": {
          "$expr": {
            "$lte": ["$retention_limit", "$current_timestamp"]
          }
        }
      },
      {
        "$group": {
          "_id": "$projects.project_id",
          "matched_traces": { "$addToSet": "$matched_traces" },
          "joined_spans": { "$push": "$joined_spans" },
          "projects": { "$first": "$projects" },
        }
      },
      {
        "$project": {
          "_id": 0,
          "trace_id": "$matched_traces.trace_id",
          "project_id": "$projects.project_id",
          "project_retention": "$projects.retention",
          "span_id": "$joined_spans.context.span_id",
          "time": 1,
          "current_timestamp": 1,
          "retention_limit": 1
        }
      }
    ]
     
    # Execute the aggregation pipeline to get the results
    results = list(db.Users.aggregate(pipeline))
     
    # Gather span_id and trace_id from the results
    span_ids = []
    trace_ids = []
     
    for result in results:
        if "span_id" in result:
            span_ids.extend(result["span_id"])
        if "trace_id" in result:
            trace_ids.extend(result["trace_id"])
     
     
    # Delete documents from Spans collection
    db.Spans.delete_many({"context.span_id": {"$in": span_ids}})
     
    # Delete documents from Traces collection
    db.Traces.delete_many({"trace_id": {"$in": trace_ids}})
     
    cleanup_document = {
        "Execution_time": datetime.utcnow(),
        "deleted_traces": len(trace_ids),
        "traces": trace_ids,
        "deleted_spans": len(span_ids),
        "spans": span_ids
    }
     
    db.Cleanup.insert_one(cleanup_document)

    Security

    💡

    Using the unprotected deployment eliminates the need for the auth0 API but still requires auth0 for User Management whereas using the single user deployment eliminates the need for using auth0 ENTIRELY.

    GuardOps is built to secure Backend API interactions as well as RBAC and User Management with auth0. This means that - if you want to use auth0 which is recommended - you need to setup your own auth0 environment. The following steps need to be done within auth0

    Application

    A Regular Web Application (opens in a new tab) is needed for user management. This provides the usual login and logout functionalities.

    API

    A Machine to Machine API (opens in a new tab) is needed for the Backend API endpoint protection. M2M tokens are expensive. By using a redis cache deployed either with the backend or using cloud services like upstash (opens in a new tab), only one M2M token per day is used.

    Roles

    The Frontend implements RBAC using auth0 roles (opens in a new tab). The following roles are needed to recreate the intended RBAC concept:

    • Datasets
    • Evaluation
    • Monitoring
    • Playground
    • Projects
    • Full_Access
    💡

    Roles have been created to manage access for different user types. For example, if your organization only has one person that should handle Evaluations or Monitoring, only they can be assigned the specific roles. If you do not need RBAC and want to provide the full functionality to all users, you can edit the save_logged_user Action in the next section to automatically assign every user to the Full_Access role.

    Actions

    GuardOps uses auth0 Actions (opens in a new tab) to automatically store user_ids in the Backend Database as well as assign new users a role.

    The code below is for an implementation that uses an upstash redis cache for M2M token caching. If you use your own local redis cache, you need to expose it publicly or allow connections from auth0 (opens in a new tab).

    You need to create two actions that each have secrets that correspond to the variables you have to fill in as described in Environment Variables

    Backend user_id linking

    This Action does the following: After every login, the ID assigned to the user by auth0 is sent to the Backend API and a default role is assigned to the user.

    This is done after every login since in some edge cases a post registration action that only executes after account registration can get lost/not executed resulting in a broken account. The user also gets assigned a role automatically. For this to work, the ID of the role needs to be changed. This can be changed to "Full_Access" by replacing the ID in the highlighted line 99 accordingly, otherwise the role Playground is suggested for default role allocation.

    Remember to replace the placeholders in the highlighted lines with your actual values or include them as a new secret.

    In addition to the code below, the action also needs secrets and dependencies. You can extract the secrets from your adjusted .env files.

    • Secrets: UPSTASH_REDIS_REST_URL, UPSTASH_REDIS_REST_TOKEN, AUTH0_TRACEAPI_CLIENT_ID, AUTH0_TRACEAPI_CLIENT_SECRET, TRACEAPI_AUDIENCE, AUTH0_TRACEAPI, AUTH0_DOMAIN, AUTH0_MANAGEMENT_CLIENT_ID, AUTH0_MANAGEMENT_CLIENT_SECRET, MANAGEMENT_AUDIENCE, AUTH0_MANAGEMENT_API
    • Dependencies: axios@1.6.7, @upstash/redis@1.28.4
    const axios = require('axios');
    const { Redis } = require('@upstash/redis');
     
     
    exports.onExecutePostLogin = async (event, api) => {
      const redis = new Redis({
        url: event.secrets.UPSTASH_REDIS_REST_URL,
        token: event.secrets.UPSTASH_REDIS_REST_TOKEN,
      });
     
      const getToken = async () => {
        const redisToken = "access_token";
        try {
          const cachedToken = await redis?.get(redisToken);
          if (cachedToken) {
            return cachedToken;
          }
          const options = {
            method: "POST",
            headers: { "Content-Type": "application/json" },
            body: JSON.stringify({
              client_id: event.secrets.AUTH0_TRACEAPI_CLIENT_ID,
              client_secret: event.secrets.AUTH0_TRACEAPI_CLIENT_SECRET,
              audience: event.secrets.TRACEAPI_AUDIENCE,
              grant_type: "client_credentials",
            }),
          };
     
          try {
            const response = await fetch(
              `${event.secrets.AUTH0_TRACEAPI}`,
              options
            );
            const data = await response.json();
            const newToken = data.access_token;
            if (newToken) {
              await redis.set(redisToken, newToken, { ex: 86400 });
              return newToken;
            } else {
              throw new Error("Failed to fetch new token");
            }
          } catch (error) {
            console.error("Error during API request:", error);
          }
        } catch (error) {
          throw error;
        }
      };
     
      const getRoleToken = async () => {
        const redisToken = "access_role_token";
        try {
          const cachedToken = await redis?.get(redisToken);
          if (cachedToken) {
            return cachedToken;
          }
          const options = {
            method: "POST",
            headers: { "Content-Type": "application/json" },
            body: JSON.stringify({
              client_id: event.secrets.AUTH0_MANAGEMENT_CLIENT_ID,
              client_secret: event.secrets.AUTH0_MANAGEMENT_CLIENT_SECRET,
              audience: event.secrets.MANAGEMENT_AUDIENCE,
              grant_type: "client_credentials",
            }),
          };
     
          try {
            const response = await fetch(
              `${event.secrets.AUTH0_MANAGEMENT_API}`,
              options
            );
            const data = await response.json();
            const newToken = data.access_token;
            if (newToken) {
              await redis.set(redisToken, newToken, { ex: 86400 });
              return newToken;
            } else {
              throw new Error("Failed to fetch new token");
            }
          } catch (error) {
            console.error("Error during API request:", error);
          }
        } catch (error) {
          throw error;
        }
      };
      const userId = event.user.user_id;
     
      const token = await getToken()
     
      const config = {
        headers: { Authorization: `Bearer ${token}` }
      };
      const assignRole = async () => {
        const roleToken = await getRoleToken()
        try {
          const body = {
            roles: ['<ROLE_ID>'] // Replace with the actual role_id of your desired starter role for every new user 
          };
          const url = `https://${event.secrets.AUTH0_DOMAIN}/api/v2/users/${userId}/roles`;
          const response = await axios.post(url, body, {
            headers: {
              'Content-Type': 'application/json',
              Authorization: `Bearer ${roleToken}`,
              'cache-control': 'no-cache'
            }
          });
     
          console.log(`Role "Playground" assigned to user ${userId}. Response:`, response.data);
        } catch (error) {
          console.error(`Error assigning role "Playground" to user ${userId}:`, error.response.data || error.message);
        }
      };
      try {
        // Check if userId is already in the list
        const url = '<YOUR_API_URL>/api/get_userids';
        const response = await axios.get(url, config);
     
        if (response.statusText === 'OK') {
          const userList = response.data;
          if (!userList.includes(userId)) {
     
            const userData = {
              user_id: userId,
              // Include other user information if needed
            };
     
            // Call the API to store the user ID using POST method with query parameter
            const apiUrl = '<YOUR_API_URL>/api/create_user';
            const saveResponse = await axios.post(apiUrl, userData, {
              headers: config.headers,
              params: {
                user_id: userId,
              }
            });
            await assignRole();
            console.log(`User ID ${userId} saved successfully. Response:`, saveResponse.data);
          } else {
            console.log(`User ID ${userId} already exists.`);
          }
        }
      } catch (error) {
        console.error('Error fetching or saving user ID:', error);
      }
    };
     

    Default project creation

    This Action does the following: After every login, a default project is created for the user if it does not exist already. It checks for the name "Default Project" and does not create it if it exists. A default project makes sense since this allows users without the projects role to be able to use the full playground functionality.

    Remember to replace the placeholders in the highlighted lines with your actual url or include your url as a new secret.

    In addition to the code below, the action also needs secrets and dependencies. You can extract the secrets from your adjusted .env files.

    • Secrets: UPSTASH_REDIS_REST_URL, UPSTASH_REDIS_REST_TOKEN, AUTH0_TRACEAPI_CLIENT_ID, AUTH0_TRACEAPI_CLIENT_SECRET, TRACEAPI_AUDIENCE, AUTH0_TRACEAPI
    • Dependencies: node-fetch@3.3.2, axios@1.6.8, @upstash/redis@1.29.0
    const axios = require('axios');
    const { Redis } = require('@upstash/redis');
     
    exports.onExecutePostLogin = async (event, api) => {
      const redis = new Redis({
        url: event.secrets.UPSTASH_REDIS_REST_URL,
        token: event.secrets.UPSTASH_REDIS_REST_TOKEN,
      });
     
      const getToken = async () => {
        const redisToken = "access_token";
        try {
          const cachedToken = await redis?.get(redisToken);
          if (cachedToken) {
            return cachedToken;
          }
          const options = {
            method: "POST",
            headers: { "Content-Type": "application/json" },
            body: JSON.stringify({
              client_id: event.secrets.AUTH0_TRACEAPI_CLIENT_ID,
              client_secret: event.secrets.AUTH0_TRACEAPI_CLIENT_SECRET,
              audience: event.secrets.TRACEAPI_AUDIENCE,
              grant_type: "client_credentials",
            }),
          };
     
          try {
            const response = await fetch(
              `${event.secrets.AUTH0_TRACEAPI}`,
              options
            );
            const data = await response.json();
            const newToken = data.access_token;
            if (newToken) {
              await redis.set(redisToken, newToken, { ex: 86400 });
              return newToken;
            } else {
              throw new Error("Failed to fetch new token");
            }
          } catch (error) {
            console.error("Error during API request:", error);
          }
        } catch (error) {
          throw error;
        }
      };
      const userId = event.user.user_id;
     
      const token = await getToken()
     
      const config = {
        headers: { Authorization: `Bearer ${token}` }
      };
      try {
        // Check if userId is already in the list
        const url = '<YOUR_API_URL>/api/get_userids';
        const response = await axios.get(url, config);
     
        if (response.statusText === 'OK') {
          const userList = response.data;
          if (userList.includes(userId)) {
     
     
            const projectsUrl = '<YOUR_API_URL>/api/get_projects';
            const projectsListResponse = await axios.get(projectsUrl, {
              params: { user_id: userId }, // Pass user_id as a query parameter
              headers: config.headers
            });
            if (projectsListResponse.statusText === "OK") {
              const projectsList = projectsListResponse.data.projects;
              let projectExists = false;
              for (let i = 0; i < projectsList.length; i++) {
                if (projectsList[i].name === "Default Project") {
                  projectExists = true;
                  console.log("Skipping default project creation")
                  break;
                }
              }
     
              if (!projectExists) {
                // Define the payload data
                const payload = {
                  user_id: userId,
                  project_name: 'Default Project',
                  project_description: 'This is the coai default project created after registration',
                  project_tags: 'coai',
                  project_retention: 0
                };
     
                const postUrl = '<YOUR_API_URL>/api/create_project';
     
                // Extract token from config
                const token = config.headers['Authorization'];
     
                const queryString = Object.keys(payload)
                  .map(key => encodeURIComponent(key) + '=' + encodeURIComponent(payload[key]))
                  .join('&');
     
                const urlWithParams = `${postUrl}?${queryString}`;
     
                // Make the POST request with payload as query parameters
                await fetch(urlWithParams, {
                  method: 'POST',
                  headers: {
                    'Content-Type': 'application/json',
                    'Authorization': token
                  }
                })
                  .then(async response => {
                    // Check if the response was successful
                    if (!response.ok) {
                      const responseData = await response.json();
                      throw new Error(JSON.stringify(responseData));
                    }
                    // Parse the JSON response
                    return response.json();
                  })
                  .then(data => {
                    console.log('Response:', data);
                  })
                  .catch(error => {
                    console.error('Error:', error);
                  });
     
              }
     
            }
          } else {
            console.log(`User does not exist.`);
          }
        }
      } catch (error) {
        console.error('Error fetching or saving user ID:', error);
      }
    };
     

    Refer to the auth0 Docs (opens in a new tab) for detailed instructions.

    Alternatively, you can disabled auth0 protection entirely by using the unprotected branch (opens in a new tab).