TGNG Server/Datacenter and TGE Server/Datacenter to TGNG Cloud migration

Owned by Dam
Last updated: Nov 04, 2024 by Hung Techlead
22 min read

This documentation describe a solution to migration from TGNG Server / Datacenter to TGNG Cloud.

In order to migrate from TGNG Server / Datacenter, we build a script to help migrate issues data.

We’re excited to introduce a brand new in-app feature for migrating your grid data to the Atlassian Cloud—no scripts or advanced technical skills required!

For more details, please visit Migration Feature - v2.3.0 .

 

TGNG Server/Datacenter → TGNG Cloud migration

Groovy Script

import com.atlassian.jira.issue.customfields.CustomFieldType import com.atlassian.jira.bc.issue.search.SearchService; import com.atlassian.jira.component.ComponentAccessor; import com.atlassian.jira.jql.parser.JqlQueryParser; import com.atlassian.jira.issue.CustomFieldManager; import com.atlassian.jira.issue.fields.CustomField; import com.atlassian.jira.user.util.UserManager; import com.atlassian.jira.web.bean.PagerFilter; import groovy.json.StreamingJsonBuilder; import com.atlassian.jira.issue.Issue; import groovy.json.JsonSlurper; import groovy.json.*; //////////////////////////////////////////// // ***** CHANGE VARIABLES HERE ***** //////////////////////////////////////////// // Authorization String authorizationStringServer = "Basic ===" String authorizationStringCloud = "Basic ===" String authorizationStringCloudAccountID = """Basic ===""" //Jira BASE URL String jiraBaseURLServer = "https://===" String cloudInstanceName = "===" String jqlQuery = "===" //////////////////////////////////////////// // ***** CHANGE VARIABLES HERE ***** //////////////////////////////////////////// String baseURL = "https://databuckets.net/rest/tgc/api/v1/grids" CustomFieldManager customFieldManager = ComponentAccessor.getOSGiComponentInstanceOfType(CustomFieldManager.class) def tgngFieldType = customFieldManager.getCustomFieldType("tge.cloud:idalko-table-grid-custom-field-key") def tgngFieldList = customFieldManager.getCustomFieldObjects().findAll {it.getCustomFieldType() == tgngFieldType} // Cloud URL def jsonSlurperAccount = new JsonSlurper() StringBuilder responses = new StringBuilder() // Get all Cloud Grids URL getGridsURL; getGridsURL = new URL(baseURL); URLConnection getGridsConnection = getGridsURL.openConnection(); getGridsConnection.setRequestMethod("GET") getGridsConnection.doOutput = true getGridsConnection.setRequestProperty("Authorization", authorizationStringCloud) getGridsConnection.connect(); def getGridsResponseCode = getGridsConnection.getResponseCode() def gridsList if(getGridsResponseCode != 200){ responses.append("Error while getting the list of Grids in Cloud") }else{ gridsList = jsonSlurperAccount.parseText(getGridsConnection.getInputStream().getText()); } Map<Long,String> gridCorrespondance = [:] for(grid in tgngFieldList){ if(gridsList.find { it.name == grid.toString() } != null) { gridCorrespondance.putAll([(grid): (gridsList.find { it.name == grid.toString() }.id)]) } } def searchService = ComponentAccessor.getComponent(SearchService) def loggedInUser = ComponentAccessor.getJiraAuthenticationContext().getLoggedInUser() def queryParser = ComponentAccessor.getComponent(JqlQueryParser) def sb = new StringBuffer() //your query goes here def query = queryParser.parseQuery(jqlQuery) //gets results of query def search = searchService.search(loggedInUser, query, PagerFilter.getUnlimitedFilter()) //iterate over each returned issue Collection<Issue> issues = search.results.key for (String issue : issues){ for (tgngCustomField : gridCorrespondance) { String issueKey = issue // Grid IDs String gridCustomFieldIdServer = tgngCustomField.getKey().getIdAsLong() String gridIdCloud = tgngCustomField.getValue() def gridDataDCURL = jiraBaseURLServer + "/rest/idalko-grid/1.0/api/grid/" + gridCustomFieldIdServer + "/issue/" + issueKey; def gridDataCloudURL = "https://databuckets.net/rest/tgc/api/v1/grids/" + gridIdCloud + "/issue/" + issueKey; //Getting the Grid information from the Data Center Jira instance URL url; url = new URL(gridDataDCURL); URLConnection connection = url.openConnection(); connection.setRequestMethod("GET") connection.doOutput = true connection.setRequestProperty("Authorization", authorizationStringServer) connection.connect(); String serverData = connection.getInputStream().getText() def jsonSlurper = new JsonSlurper() def object = jsonSlurper.parseText(serverData) Long gridRowsinCloud = -1; //create variable with invalid value just to signal exception and skip condition if failed try { // Geting Grid info from Cloud to see if it was already uploaded (avoid duplicates) URL url3; url3 = new URL(gridDataCloudURL); URLConnection connection3 = url3.openConnection(); connection3.setRequestMethod("GET") connection3.doOutput = true connection3.setRequestProperty("Authorization", authorizationStringCloud) connection3.connect(); responseCode = connection3.getResponseCode() if(responseCode != 200){ responseMessage = connection3.getErrorStream().text; }else{ String cloudData = connection3.getInputStream().getText(); def cloudjson = jsonSlurperAccount.parseText(cloudData) //Long gridRowsinServer = object.rows.size(); gridRowsinCloud = cloudjson.rows.size(); //if it works, change value of variable } } catch(java.io.IOException ex) { responses.append("!! exception while getting Grid data !!"); } // check if data was already uploaded if(gridRowsinCloud <= 0){ // Formatting the output so that it fits the POST request body formatting requirements object.remove("customFieldId") //Removing Grid ID since does not need to be passed in the request body object.rows.each { e -> //Going through all of the rows from the response e.remove("order") // Removing order index e.remove("rowId") // Removing Row ID since it will be automatically assigned e.columns.each{ key, value -> // Unpacking the "columns" section into the previous level of JSON structure if(value != null || !value.toString().equals("{}") || value != ""){ if(value instanceof String){ e.put(key, value) }else if(value instanceof org.apache.groovy.json.internal.LazyMap){ if(value.keySet().contains("username")){ // Get user email to match with user in Cloud def userManager = ComponentAccessor.getUserManager() def user = userManager.getUserByName(value.username) if(user != null){ String email = user.getEmailAddress() // Getting new user info from Cloud based on email URL urlAccountID urlAccountID = new URL("https://${instanceName}.atlassian.net/rest/api/3/user/search?query=" + email.replace(" ","%20")); URLConnection connection4 = urlAccountID.openConnection(); connection4.setRequestMethod("GET") connection4.setRequestProperty("Authorization", authorizationStringCloudAccountID) connection4.doOutput = true connection4.connect(); String cloudData1 = connection4.getInputStream().getText() def jsonSlurperAccount1 = new JsonSlurper() def objectAccount1 = jsonSlurperAccount1.parseText(cloudData1) if(objectAccount1[0] != null){ e.put(key, jsonSlurperAccount1.parseText('{"accountId":"' + objectAccount1[0].accountId+'"}')) } } } if(value.keySet().contains("label")){ e.put(key, value.label) } }else if(value.toString().length() != 2){ e.put(key, value.value) } } } e.remove("columns") // Removing the packed "columns" section as it is redundant and breaks the formatting e.remove("Item Number") // Remove SEQUENCE type column from the data load } try { // Uploading the grid data to the Cloud URL url2; url2 = new URL(gridDataCloudURL); URLConnection connection2 = url2.openConnection(); connection2.setRequestMethod("POST") connection2.doOutput = true connection2.setRequestProperty("Authorization", authorizationStringCloud) connection2.setRequestProperty("Content-Type", "application/json;charset=UTF-8") connection2.outputStream.withWriter("UTF-8") { new StreamingJsonBuilder(it, object) } connection2.connect(); responseCode = connection2.getResponseCode() if(responseCode != 201){ responseMessage = connection2.getErrorStream().text + JsonOutput.toJson(object); }else{ responseMessage = connection2.getResponseMessage(); } responses.append( " || IssueKey: " + issueKey + " Grid Field Name: " + tgngCustomField.getKey() + " ResponseCode: " + responseCode + " ResponseMessage: " + responseMessage) } catch(java.io.IOException ex) { responses.append("!! exception !!"); } responses.append("\n") } } } return responses

Prerequisites

  • A plugin/tool for running Groovy code is present on the instance from which the script is being run

  • TGNG apps installed on both Server and Cloud instances

  • The project from the source instance was already migrated to the target instance

  • All Columns in the Grid Config are made editable before running the script (can be changed after)

  • Please note that the script was written to be used as a sample and requires you to change the variables' values

  • None of the Target Grids have default rows (they can be re-added after the data is migrated)

Disclaimers

  • The script was developed to be run from the source Server instance.

  • The script is not an official solution for migration but it’s a workaround solution and should be used with care and precaution.

Guide

  1. Make sure that all of the prerequisites above are met

  2. Verify whether the grid fields on your Server instance have a multi-context setup

    1. If not, configure identical Grids on the Cloud instance (+ Scopes)

    2. If so, create an identical Grid configuration for each distinct context

  3. Obtain valid authentication strings from both instances

  4. Replace the placeholder authentication values as well as instance URLs with valid ones

  5. Replace the gridCustomFieldIdServer and gridIdCloud for the values corresponding to the first grid or Grid context that is being migrated

  6. Prepare the script to verification by adding a logic gate in the "for" loop or by hard-coding a single issue KEY into the issueKey variable

  7. Change projectId to a project that is relevant to the current grid field

  8. Verify whether object.rows.each { } loop does not contain any mentions of columns that are not present in the Grid configuration and add e.remove("COLUMN_NAME") for all sequence columns

  9. Run the script and verify data, for each Grid or Grid context:

    1. Change the gridCustomFieldIdServer, gridIdCloud, projected or any custom issue-pulling logic to relevant values

    2. Re-do step 11

    3. Run the script

    4. Verify whether the data was migrated correctly

 

TGE Server/Datacenter → TGNG Cloud migration

Groovy Script

 Expand to see the script....

import groovy.json.JsonSlurper; import groovy.json.StreamingJsonBuilder; import com.atlassian.jira.issue.fields.CustomField; import com.atlassian.jira.issue.CustomFieldManager; import com.atlassian.jira.component.ComponentAccessor; import com.atlassian.jira.issue.Issue; import com.atlassian.jira.bc.issue.search.SearchService; import com.atlassian.jira.jql.parser.JqlQueryParser; import com.atlassian.jira.web.bean.PagerFilter; import groovy.json.*; import com.atlassian.crowd.embedded.api.User import com.atlassian.jira.security.JiraAuthenticationContext import com.atlassian.plugin.PluginAccessor import com.atlassian.jira.user.ApplicationUser //////////////////////////////////////////// // ***** CHANGE VARIABLES HERE ***** //////////////////////////////////////////// String authorizationStringCloud = "Basic ===" String jqlQuery = '====' //////////////////////////////////////////// // ***** CHANGE VARIABLES HERE ***** //////////////////////////////////////////// // find TGE custom fields PluginAccessor pluginAccessor = ComponentAccessor.getPluginAccessor(); CustomFieldManager customFieldManager = ComponentAccessor.getComponent(CustomFieldManager); Class tgeConfigManagerClass = pluginAccessor.getClassLoader().findClass("com.idalko.jira.plugins.igrid.api.config.grid.TGEGridConfigManager"); def tgeConfigManager = ComponentAccessor.getOSGiComponentInstanceOfType(tgeConfigManagerClass); List<Long> tgeCustomFieldIds = tgeConfigManager.getGridCustomFieldIds(); List<Map<Long,String>> tgeCustomFieldNames = tgeCustomFieldIds.collect { [(it) : customFieldManager.getCustomFieldObject(it).name] } // read the grid data Class dataManagerClass = pluginAccessor.getClassLoader().findClass("com.idalko.jira.plugins.igrid.api.data.TGEGridTableDataManager"); def tgeGridDataManager = ComponentAccessor.getOSGiComponentInstanceOfType(dataManagerClass); // get current user JiraAuthenticationContext jiraAuthenticationContext = ComponentAccessor.getOSGiComponentInstanceOfType(JiraAuthenticationContext.class); Object userObject = jiraAuthenticationContext.getLoggedInUser(); User user = userObject instanceof ApplicationUser ? ((ApplicationUser) userObject).getDirectoryUser() : (User) userObject StringBuilder responses = new StringBuilder() def searchService = ComponentAccessor.getComponent(SearchService) def loggedInUser = ComponentAccessor.getJiraAuthenticationContext().getLoggedInUser() def queryParser = ComponentAccessor.getComponent(JqlQueryParser) //your query goes here def query = queryParser.parseQuery(jqlQuery) //gets results of query def search = searchService.search(loggedInUser, query, PagerFilter.getUnlimitedFilter()) //iterate over each returned issue Collection<Issue> issues = search.results // Cloud URL def baseURL = "https://databuckets.net/rest/tgc/api/v1/grids" // Get all Cloud Grids def jsonSlurperAccount = new JsonSlurper() URL getGridsURL; getGridsURL = new URL(baseURL); URLConnection getGridsConnection = getGridsURL.openConnection(); getGridsConnection.setRequestMethod("GET") getGridsConnection.doOutput = true getGridsConnection.setRequestProperty("Authorization", authorizationStringCloud) getGridsConnection.connect(); def getGridsResponseCode = getGridsConnection.getResponseCode() def gridsList if(getGridsResponseCode != 200){ responses.append("Error while getting the list of Grids in Cloud") }else{ gridsList = jsonSlurperAccount.parseText(getGridsConnection.getInputStream().getText()); } Map<Long,String> gridCorrespondance = [:] for(grid in tgeCustomFieldNames){ if(gridsList.find { it.name == grid.values()[0].toString() } != null) { gridCorrespondance.putAll([(grid.keySet()[0]): (gridsList.find { it.name == grid.values()[0].toString() }.id)]) } } for (Issue issue : issues){ for (Long tgeCustomFieldId : tgeCustomFieldIds) { String issueKey = issue.getKey() def issueIdServer = issue.getId() def gridIdCloud = gridCorrespondance[tgeCustomFieldId] def baseURL2 = baseURL + "/" + gridIdCloud + "/issue/" + issueKey try { def callResult = tgeGridDataManager.readGridData(issue.getId(), tgeCustomFieldId, null, null, 0, 10, user); def serverDataObject = callResult.getValues() // responses.append("Grid ID=" + tgeCustomFieldId + " content: " + callResult.getValues() + "\n"); def rowsToAdd = [] if(serverDataObject.size > 0){ for(row in serverDataObject){ rowsToAdd.add(JsonOutput.toJson(row.collectEntries { [it.key.toString(), (it.value instanceof HashMap) ? it.value.name : it.value] })) } // Generating the JSON body to pass onto the Cloud API and removing all irrelevant data def cloudJsonObject = jsonSlurperAccount.parseText( '{ "rows":' + rowsToAdd.toString() + '}') cloudJsonObject.rows.each { e -> ArrayList<Object> toRemove = new ArrayList<>(); e.remove("issueId") e.remove("modified") e.remove("id") e.each{ key, value -> if(!value || value === null || value === 'null'){ toRemove.add(key) } } for (o in toRemove) { e.remove((String)o); } } // Geting Grid info from Cloud to see if it was already uploaded (avoid duplicates) def gridRowsinCloud = -1 try { URL url3; url3 = new URL(baseURL2); URLConnection connection3 = url3.openConnection(); connection3.setRequestMethod("GET") connection3.doOutput = true connection3.setRequestProperty("Authorization", authorizationStringCloud) connection3.connect(); responseCode = connection3.getResponseCode() if(responseCode != 200){ responseMessage = connection3.getErrorStream().text; }else{ String cloudData = connection3.getInputStream().getText(); def cloudjson = jsonSlurperAccount.parseText(cloudData) //Long gridRowsinServer = object.rows.size(); gridRowsinCloud = cloudjson.rows.size(); //if it works, change value of variable } } catch(java.io.IOException ex) { responses.append("!! exception !!"); } // Uploading the grid data to the Cloud if the target Grid is empty (non-empty grids are assumed to have been already migrated) if(gridRowsinCloud <= 0){ try { URL url2; url2 = new URL(baseURL2); URLConnection connection2 = url2.openConnection(); connection2.setRequestMethod("POST") connection2.doOutput = true connection2.setRequestProperty("Authorization", authorizationStringCloud) connection2.setRequestProperty("Content-Type", "application/json;charset=UTF-8") connection2.outputStream.withWriter("UTF-8") { new StreamingJsonBuilder(it, cloudJsonObject) } connection2.connect(); responseCode = connection2.getResponseCode() if(responseCode != 201){ responseMessage = connection2.getErrorStream().text + JsonOutput.toJson(cloudJsonObject); }else{ responseMessage = connection2.getResponseMessage(); } responses.append( " || IssueKey: " + issueKey + " ResponseCode:" + responseCode + " ResponseMessage:" + responseMessage) } catch(java.io.IOException ex) { responses.append("!! exception !!"); } responses.append("\n") } } } catch (Exception e) { responses.append("Grid ID=" + tgeCustomFieldId + " data cannot be retrieved: " + e + "\n"); } } } return responses

Prerequisites

  • A plugin/tool for running Groovy code is present on the instance from which the script is being run

  • TGE app is installed and enabled on Server/Datacenter, TGNG Cloud is installed and enabled on Cloud

  • The project from the source instance was already migrated to the target instance

  • All Columns in the Grid Config are made editable before running the script (can be changed after)

  • Please note that the script was written to be used as a sample and requires you to change the variables' values

  • None of the Target Grids have default rows (they can be re-added after the data is migrated)

Disclaimers

  • The script was developed to be run from the source Server/Datacenter instance.

  • The script is not an official solution for migration but it’s a workaround solution and should be used with care and precaution.

Guide

  1. Make sure that all of the prerequisites above are met

  2. Verify whether the grid fields on your Server instance have a multi-context setup

    1. If not, configure identical Grids on the Cloud instance (+ Scopes)

    2. If so, create an identical Grid configuration for each distinct context

  3. Obtain a valid authentication string from the Cloud instance and replace the placeholder value in the script with it

  4. Prepare the script for verification by adding a logic gate in the "for" loop or by setting a JQL string so that it only returns a single issue

  5. Verify whether object.rows.each { } loop does not contain any mentions of columns that are not present in the Grid configuration and add e.remove("COLUMN_NAME") for all sequence columns

  6. Run the script and verify data, for each Grid or Grid context

  7. After the finalised script has been validated on one of a few (up to 10) issues, proceed to remove the logic gate added in step 4 or expand the JQL string to an expression covering all relevant issues